From 12ef5d7eee9c1a9564e96d8ef8ea8b2faf254346 Mon Sep 17 00:00:00 2001 From: Mathew Sir Guest the best Date: Tue, 7 Jul 2026 20:07:07 -0600 Subject: [PATCH] docs: add CLAUDE.md and modernization design spec Design approved in brainstorming session: uv packaging, Selenium 4 migration, session persistence rebuilt on JSON + session-ID reattach (replacing pickle/dill), configurable pacing, pytest unit+integration tiers, Sphinx docs. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 34 ++++ .../2026-07-07-wabot-modernization-design.md | 156 ++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 CLAUDE.md create mode 100644 docs/superpowers/specs/2026-07-07-wabot-modernization-design.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..c901b99 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,34 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +WABot is a Python library wrapping Selenium for stateful browser automation (login flows, multi-page form filling, scraping). Its two distinguishing features: + +1. **Page-object automation with state carried between pages** — consumers model each website page as a `Page` subclass and drive them through a single `BrowserProxy`. +2. **Browser sessions that survive the Python process** — a remote webdriver reference is persisted to disk so a *different* Python process can reattach to the same still-open browser. + +There is currently no test suite, linter, or CI. Packaging is a legacy `setup.py` (`pip install -e .` to develop). The code is pinned to Selenium 3-era APIs (`desired_capabilities`, `browser_profile`, `switch_to_alert`); it does not run against Selenium 4+ without migration. + +## Architecture + +Consumer code talks to one object, `BrowserProxy` (`wabot/api.py`), which composes everything else: + +- **`BrowserProxy`** holds the webdriver plus a *current page* object and acts as a facade. `__getattr__` delegates unknown attributes to the current page, so `bot.login()` invokes `bot.page.login()`; `bot[key]` resolves page elements. `set_page(PageClass)` swaps the current page and calls its `verify()` gate. `perform(name, ...)` invokes a page method with exception trapping — any failure flips `self.good = False`, after which all further actions are refused (`REFUSE_AFTER_EXCEPTION`), forcing an explicit `reset_good_status()`. +- **`Page`** (`wabot/page.py`) is the base class consumers subclass. Pages declare a class-level `elements = {key: (type, accessors)}` map; lookup walks the MRO, so subclasses inherit and override parent element maps. `page[key]` returns a typed field wrapper, never a raw WebElement. +- **Field wrappers** (`wabot/fields.py`): `TextField`, `SelectField`, `CheckField` wrap elements with `get_value`/`set_value`; unknown attribute access falls through to the underlying selenium element. `NullField` is a falsy null-object returned for missing elements so callers can `if el:` instead of catching exceptions. +- **`CreateBrowser`** (`wabot/create_browser.py`) is the driver factory. Local drivers (Chrome/Firefox) are plain. Remote drivers (Selenium server expected at `localhost:4444`) are serialized with `dill` to `appdirs.user_data_dir('wabot')/saved_browser_instances.pickle`, keyed by session name with a timestamp (refs older than 3 days are purged). On the next request for the same session name, the pickled driver is loaded and validated by touching `driver.current_url` — that is the reattach mechanism. + +**Human-like behavior is intentional, not accidental slowness**: clicks land at Gaussian-random coordinates within the element instead of Selenium's default (0,0), and field setters `sleep(random.uniform(...))` between actions. Chrome full-page screenshots are stitched from viewport tiles with PIL to work around chromedriver capturing only the visible area. Don't "optimize" these away. + +## Gotchas + +- `LOGGER.trace(...)` is called throughout, but stdlib `logging` has no `trace` method — the consuming application is expected to register a custom TRACE level; without it those calls raise. +- `appdirs` and `Pillow` are imported but missing from `install_requires` in `setup.py` (only `dill` and `selenium` are declared). +- `wabot/__init__.py` star-exports `api`, `page`, and `fields`, so anything public in those modules is public API. +- Remote driver types require an external Selenium server already running on port 4444; nothing in this repo starts one. + +## Git + +Remote is Gitea (`gitea@git.zavage.net:Zavage-Software/wabot.git`); default branch is `master`. Do feature work on branches — pushing is done manually by the user (no SSH auth from inside the Claude shell). diff --git a/docs/superpowers/specs/2026-07-07-wabot-modernization-design.md b/docs/superpowers/specs/2026-07-07-wabot-modernization-design.md new file mode 100644 index 0000000..c02af96 --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-wabot-modernization-design.md @@ -0,0 +1,156 @@ +# WABot Modernization — Design + +**Date:** 2026-07-07 +**Status:** Approved (brainstorming session with Mathew) +**Approach:** Rebuild the core on the existing design (approach B) — port proven logic into cleanly separated modules; redesign the session-persistence layer; free API redesign was explicitly approved (no downstream consumers to protect). + +## Context & goals + +WABot is a Selenium-wrapping browser-automation library (page-object pattern, stateful multi-page flows, anti-bot stealth behavior). It is stuck on Selenium 3-era APIs because its flagship feature — saving a browser so a *different* Python process can pick it up — was implemented by pickling live webdriver objects with `dill`, which broke on Selenium 4. + +Goals, as requested: + +1. Modern packaging with **uv** (`pyproject.toml`, lockfile). +2. **Sphinx + sphinx-rtd-theme** documentation. +3. **pytest** suite (unit + real-browser integration). +4. Migrate to **latest Selenium 4**, keeping core functionality — especially cross-process browser pickup, rebuilt on a sound mechanism. +5. General code quality: fix latent bugs, drop dead code, keep the behavioral subtleties that make the library effective. + +### Decisions from the brainstorming Q&A + +| Question | Decision | +|---|---| +| API compatibility | Free to redesign; keep the concepts, not the exact signatures | +| Browser host for persistence | **Both**: external Selenium server URL *and* wabot-spawned detached driver service | +| Test depth | Unit (mocked) + integration (real headless browsers, local HTML fixtures, cross-process reattach e2e) | +| Stealth pacing | Configurable policy; human-like behavior stays the **default** | + +### Key insight (why persistence broke, and the fix) + +A `Remote` webdriver is a thin HTTP client around `(command_executor_url, session_id)`. Pickling the whole object was always fragile and is unworkable on Selenium 4. The robust mechanism is to persist just the URL + session ID as JSON and **reattach** by constructing a `Remote` subclass whose `start_session` adopts the saved session ID instead of creating a new session. This works against any WebDriver-protocol server: a Selenium Grid *or* a bare chromedriver/geckodriver. `dill` is removed entirely. The browser must be hosted by a process that outlives Python — same constraint as the old design, where only `remote_*` drivers were persistable. + +## Tooling & layout + +- `pyproject.toml` managed by uv, `uv_build` backend, `requires-python = ">=3.10"`. +- Runtime deps: `selenium>=4` (latest), `platformdirs` (replaces dead `appdirs`), `Pillow`. **Removed:** `dill`, `appdirs`. +- Dev group: `pytest`, `pytest-cov`, `ruff` (lint + format), `sphinx`, `sphinx-rtd-theme`. +- Commands: `uv sync`, `uv run pytest`, `uv run pytest -m integration`, `uv run ruff check`, `uv run sphinx-build docs docs/_build/html`. + +``` +wabot/ +├── pyproject.toml +├── uv.lock +├── src/wabot/ +│ ├── __init__.py # explicit public API (no star exports); wabot.browser() entry point +│ ├── browser.py # Browser (was BrowserProxy) — facade over driver + current page +│ ├── page.py # Page base class + element resolution +│ ├── fields.py # TextField / SelectField / CheckField / NullField +│ ├── pacing.py # HumanPacing / NoPacing policies +│ ├── sessions.py # SessionStore — JSON persistence + stale eviction +│ ├── hosts.py # ExternalServer / ManagedService driver hosts +│ └── screenshot.py # Chromium full-page stitching (PIL) +├── tests/ +│ ├── unit/ +│ ├── integration/ +│ └── fixtures/ # static HTML pages (login form, multi-page flow) +└── docs/ # Sphinx +``` + +The legacy `setup.py` and top-level `wabot/` package are removed once the `src/` layout lands. + +## Session persistence layer + +Three cooperating pieces (this replaces all pickle logic in `create_browser.py`): + +**`SessionStore`** (`sessions.py`) +- JSON file at `platformdirs.user_data_dir('wabot')/sessions.json`. +- Schema: `{name: {executor_url, session_id, browser, created_at, service_pid?, service_port?}}`. +- Entries older than a configurable max age (default 3 days, matching current behavior) are evicted on load. +- A corrupt file is renamed aside (`sessions.json.bad`) and recreated — never a crash. + +**`DriverHost`** (`hosts.py`) — small interface: `ensure_running() -> executor_url`. +- `ExternalServer(url)`: a Selenium server/Grid the user runs; wabot just connects. Docs must note Grid 4 reaps idle sessions after 5 minutes unless `--session-timeout` is raised. +- `ManagedService(browser)`: wabot spawns `chromedriver`/`geckodriver` as a **detached** process (new process session, output redirected to a log file in the wabot data dir) so it outlives Python; records port + pid in the store. Driver binaries are found on `PATH` first, falling back to Selenium Manager's downloaded copies. On reattach the service's `/status` endpoint is health-checked before use. + +**Reattach** +- A private `Remote` subclass overrides `start_session` to adopt the saved `session_id` (no new session created). +- Liveness validation: touch `driver.current_url`. On failure: log a warning, prune the store entry, fall through to fresh creation. + +**Public flow** + +```python +import wabot + +bot = wabot.browser(session="scraper1", browser="firefox") # create + persist +# process exits; browser stays open +bot = wabot.browser(session="scraper1", browser="firefox") # second process reattaches +``` + +- `session=None` (default): ephemeral — the browser is not persisted and the store is untouched. With no `host`, this is a plain local Selenium driver that dies with the process; with a `host`, it connects to that server but the session is still not saved. +- `session="name"` + `host="http://..."` → persist via `ExternalServer`; `session="name"` without `host` → persist via `ManagedService`. +- Housekeeping: `wabot.sessions()` lists saved sessions; `wabot.destroy(name)` quits the browser, kills a managed service, removes the entry. + +## Core API + +**`Browser`** (`browser.py`, rename of `BrowserProxy`) +- Facade: holds driver + *current page*; `set_page(PageClass)` instantiates and runs the page's `verify()` gate; `__getattr__` delegates unknown attributes to the current page (`bot.login()` → `bot.page.login()`); `bot[key]` resolves elements on the current page. +- `good` flag properly initialized to `True` (today it is never set, so the refuse-after-exception guard is broken); `perform()` keeps trap-and-flip behavior; `reset_good_status()` retained. +- Dropped: unused `switch_after` decorator, NHSN-specific `RC` enum values, hardcoded `/home/mathew/firefox_prof` profile path, PhantomJS remnants, `firefox1/firefox2/chromium1/chromium2` naming (now just `firefox` / `chromium` — with `chrome` accepted as an alias, since chromedriver drives both — plus the optional `host`). +- User-agent becomes an optional parameter defaulting to the browser's own UA (the hardcoded 2006 MSIE 7 string is an anti-stealth beacon today). +- Logging: stdlib only — `LOGGER.trace` → `debug`, `LOGGER.warn` → `warning` (the `trace` calls crash under stdlib logging unless the host app registers a custom level). + +**`Page` + fields** — declarative design preserved +- Class-level `elements = {key: (type, accessors)}`; lookup walks the MRO so subclasses inherit/override element maps; `page[key]` returns typed wrappers; `NullField` stays the falsy null-object for missing elements. +- Consolidate the duplicated find/wait logic; replace bare `except:` with specific Selenium exception types. +- Selenium 4 port subtlety: `move_to_element_with_offset` now measures from the element **center** (was top-left in Selenium 3), so the Gaussian click math is recomputed relative to center — otherwise human-like clicks land outside small elements. + +**`pacing.py`** — stealth extracted into a policy +- `HumanPacing` (default): current behavior — random 3–11 s delays per field type, Gaussian click offsets. +- `NoPacing`: instant actions, exact clicks (used by tests and trusted-site automation). +- `Browser` holds the policy; fields and `Page.click` consult it instead of calling `time.sleep` directly. + +**Screenshots** (`screenshot.py`) +- Firefox: Selenium 4 native `get_full_page_screenshot_as_file`. +- Chromium: keep the PIL viewport-stitching workaround, isolated in this module. + +## Error handling + +Philosophy stays *log and refuse, don't crash the scraper*: +- Missing elements → `NullField`; failed actions → `False`; exceptions in `perform()` → logged with traceback, `good = False`. +- Dead reattach → warn, prune, create fresh (never crash on a stale session). +- Corrupt store → rename aside, rebuild. +- Dead managed service → detected by `/status` health check before driving. +- No bare `except:` — catch specific Selenium exceptions so real bugs surface in logs. + +## Testing + +Two pytest tiers split by the `integration` marker: + +**Unit** (default, fast, no browsers): element resolution through the MRO; field wrappers against a mocked driver; pacing policies; `SessionStore` round-trip, eviction, and corruption handling on `tmp_path`; reattach logic against a mocked executor. + +**Integration** (`-m integration`): real headless Firefox + Chromium driving static HTML fixtures (login form, multi-page flow) served locally — no external network. Flagship test: create a session via `ManagedService`, then a **second Python process** (via `subprocess`) reattaches by name and asserts browser state (URL, filled field value) survived — proving cross-process pickup end to end. `ExternalServer` mode is tested against a locally spawned chromedriver used as the "external" URL (no Java Grid required). Integration tests always run with `NoPacing`. + +## Documentation + +Sphinx in `docs/`, `sphinx-rtd-theme`, autodoc + napoleon (Google-style docstrings on the public API). Pages: +1. Quickstart — install, first scrape. +2. Persistent sessions guide — the flagship feature; managed services vs external servers; the Grid `--session-timeout` caveat. +3. Page-objects guide — modeling a site with `Page` + `elements`. +4. API reference (autodoc). + +Local build only (`uv run sphinx-build docs docs/_build/html`); no hosting setup for now. + +## Out of scope + +- CI pipeline (can be added later on Gitea). +- Read the Docs hosting. +- New automation features beyond parity (e.g., async API, non-Selenium backends). +- PhantomJS and the legacy `firefox1`-style driver taxonomy (deliberately removed). + +## Known latent bugs fixed by this work + +- `LOGGER.trace()` calls crash under stdlib logging. +- `_clean_old_pickles()` references undefined `final_name`. +- `BrowserProxy.get_driver()` calls `self._create_driver_chromium()` which doesn't exist on that class. +- `self.good` never initialized, breaking the refuse-after-exception guard. +- `appdirs` and `Pillow` used but undeclared in `setup.py`.