docs: add CLAUDE.md and modernization design spec

Design approved in brainstorming session: uv packaging, Selenium 4
migration, session persistence rebuilt on JSON + session-ID reattach
(replacing pickle/dill), configurable pacing, pytest unit+integration
tiers, Sphinx docs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CLAUDE.md

@@ -0,0 +1,34 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+WABot is a Python library wrapping Selenium for stateful browser automation (login flows, multi-page form filling, scraping). Its two distinguishing features:
+
+1. **Page-object automation with state carried between pages** — consumers model each website page as a `Page` subclass and drive them through a single `BrowserProxy`.
+2. **Browser sessions that survive the Python process** — a remote webdriver reference is persisted to disk so a *different* Python process can reattach to the same still-open browser.
+
+There is currently no test suite, linter, or CI. Packaging is a legacy `setup.py` (`pip install -e .` to develop). The code is pinned to Selenium 3-era APIs (`desired_capabilities`, `browser_profile`, `switch_to_alert`); it does not run against Selenium 4+ without migration.
+
+## Architecture
+
+Consumer code talks to one object, `BrowserProxy` (`wabot/api.py`), which composes everything else:
+
+- **`BrowserProxy`** holds the webdriver plus a *current page* object and acts as a facade. `__getattr__` delegates unknown attributes to the current page, so `bot.login()` invokes `bot.page.login()`; `bot[key]` resolves page elements. `set_page(PageClass)` swaps the current page and calls its `verify()` gate. `perform(name, ...)` invokes a page method with exception trapping — any failure flips `self.good = False`, after which all further actions are refused (`REFUSE_AFTER_EXCEPTION`), forcing an explicit `reset_good_status()`.
+- **`Page`** (`wabot/page.py`) is the base class consumers subclass. Pages declare a class-level `elements = {key: (type, accessors)}` map; lookup walks the MRO, so subclasses inherit and override parent element maps. `page[key]` returns a typed field wrapper, never a raw WebElement.
+- **Field wrappers** (`wabot/fields.py`): `TextField`, `SelectField`, `CheckField` wrap elements with `get_value`/`set_value`; unknown attribute access falls through to the underlying selenium element. `NullField` is a falsy null-object returned for missing elements so callers can `if el:` instead of catching exceptions.
+- **`CreateBrowser`** (`wabot/create_browser.py`) is the driver factory. Local drivers (Chrome/Firefox) are plain. Remote drivers (Selenium server expected at `localhost:4444`) are serialized with `dill` to `appdirs.user_data_dir('wabot')/saved_browser_instances.pickle`, keyed by session name with a timestamp (refs older than 3 days are purged). On the next request for the same session name, the pickled driver is loaded and validated by touching `driver.current_url` — that is the reattach mechanism.
+
+**Human-like behavior is intentional, not accidental slowness**: clicks land at Gaussian-random coordinates within the element instead of Selenium's default (0,0), and field setters `sleep(random.uniform(...))` between actions. Chrome full-page screenshots are stitched from viewport tiles with PIL to work around chromedriver capturing only the visible area. Don't "optimize" these away.
+
+## Gotchas
+
+- `LOGGER.trace(...)` is called throughout, but stdlib `logging` has no `trace` method — the consuming application is expected to register a custom TRACE level; without it those calls raise.
+- `appdirs` and `Pillow` are imported but missing from `install_requires` in `setup.py` (only `dill` and `selenium` are declared).
+- `wabot/__init__.py` star-exports `api`, `page`, and `fields`, so anything public in those modules is public API.
+- Remote driver types require an external Selenium server already running on port 4444; nothing in this repo starts one.
+
+## Git
+
+Remote is Gitea (`gitea@git.zavage.net:Zavage-Software/wabot.git`); default branch is `master`. Do feature work on branches — pushing is done manually by the user (no SSH auth from inside the Claude shell).

docs/superpowers/specs/2026-07-07-wabot-modernization-design.md

@@ -0,0 +1,156 @@
+# WABot Modernization — Design
+
+**Date:** 2026-07-07
+**Status:** Approved (brainstorming session with Mathew)
+**Approach:** Rebuild the core on the existing design (approach B) — port proven logic into cleanly separated modules; redesign the session-persistence layer; free API redesign was explicitly approved (no downstream consumers to protect).
+
+## Context & goals
+
+WABot is a Selenium-wrapping browser-automation library (page-object pattern, stateful multi-page flows, anti-bot stealth behavior). It is stuck on Selenium 3-era APIs because its flagship feature — saving a browser so a *different* Python process can pick it up — was implemented by pickling live webdriver objects with `dill`, which broke on Selenium 4.
+
+Goals, as requested:
+
+1. Modern packaging with **uv** (`pyproject.toml`, lockfile).
+2. **Sphinx + sphinx-rtd-theme** documentation.
+3. **pytest** suite (unit + real-browser integration).
+4. Migrate to **latest Selenium 4**, keeping core functionality — especially cross-process browser pickup, rebuilt on a sound mechanism.
+5. General code quality: fix latent bugs, drop dead code, keep the behavioral subtleties that make the library effective.
+
+### Decisions from the brainstorming Q&A
+
+| Question | Decision |
+|---|---|
+| API compatibility | Free to redesign; keep the concepts, not the exact signatures |
+| Browser host for persistence | **Both**: external Selenium server URL *and* wabot-spawned detached driver service |
+| Test depth | Unit (mocked) + integration (real headless browsers, local HTML fixtures, cross-process reattach e2e) |
+| Stealth pacing | Configurable policy; human-like behavior stays the **default** |
+
+### Key insight (why persistence broke, and the fix)
+
+A `Remote` webdriver is a thin HTTP client around `(command_executor_url, session_id)`. Pickling the whole object was always fragile and is unworkable on Selenium 4. The robust mechanism is to persist just the URL + session ID as JSON and **reattach** by constructing a `Remote` subclass whose `start_session` adopts the saved session ID instead of creating a new session. This works against any WebDriver-protocol server: a Selenium Grid *or* a bare chromedriver/geckodriver. `dill` is removed entirely. The browser must be hosted by a process that outlives Python — same constraint as the old design, where only `remote_*` drivers were persistable.
+
+## Tooling & layout
+
+- `pyproject.toml` managed by uv, `uv_build` backend, `requires-python = ">=3.10"`.
+- Runtime deps: `selenium>=4` (latest), `platformdirs` (replaces dead `appdirs`), `Pillow`. **Removed:** `dill`, `appdirs`.
+- Dev group: `pytest`, `pytest-cov`, `ruff` (lint + format), `sphinx`, `sphinx-rtd-theme`.
+- Commands: `uv sync`, `uv run pytest`, `uv run pytest -m integration`, `uv run ruff check`, `uv run sphinx-build docs docs/_build/html`.
+
+```
+wabot/
+├── pyproject.toml
+├── uv.lock
+├── src/wabot/
+│   ├── __init__.py         # explicit public API (no star exports); wabot.browser() entry point
+│   ├── browser.py          # Browser (was BrowserProxy) — facade over driver + current page
+│   ├── page.py             # Page base class + element resolution
+│   ├── fields.py           # TextField / SelectField / CheckField / NullField
+│   ├── pacing.py           # HumanPacing / NoPacing policies
+│   ├── sessions.py         # SessionStore — JSON persistence + stale eviction
+│   ├── hosts.py            # ExternalServer / ManagedService driver hosts
+│   └── screenshot.py       # Chromium full-page stitching (PIL)
+├── tests/
+│   ├── unit/
+│   ├── integration/
+│   └── fixtures/           # static HTML pages (login form, multi-page flow)
+└── docs/                   # Sphinx
+```
+
+The legacy `setup.py` and top-level `wabot/` package are removed once the `src/` layout lands.
+
+## Session persistence layer
+
+Three cooperating pieces (this replaces all pickle logic in `create_browser.py`):
+
+**`SessionStore`** (`sessions.py`)
+- JSON file at `platformdirs.user_data_dir('wabot')/sessions.json`.
+- Schema: `{name: {executor_url, session_id, browser, created_at, service_pid?, service_port?}}`.
+- Entries older than a configurable max age (default 3 days, matching current behavior) are evicted on load.
+- A corrupt file is renamed aside (`sessions.json.bad`) and recreated — never a crash.
+
+**`DriverHost`** (`hosts.py`) — small interface: `ensure_running() -> executor_url`.
+- `ExternalServer(url)`: a Selenium server/Grid the user runs; wabot just connects. Docs must note Grid 4 reaps idle sessions after 5 minutes unless `--session-timeout` is raised.
+- `ManagedService(browser)`: wabot spawns `chromedriver`/`geckodriver` as a **detached** process (new process session, output redirected to a log file in the wabot data dir) so it outlives Python; records port + pid in the store. Driver binaries are found on `PATH` first, falling back to Selenium Manager's downloaded copies. On reattach the service's `/status` endpoint is health-checked before use.
+
+**Reattach**
+- A private `Remote` subclass overrides `start_session` to adopt the saved `session_id` (no new session created).
+- Liveness validation: touch `driver.current_url`. On failure: log a warning, prune the store entry, fall through to fresh creation.
+
+**Public flow**
+
+```python
+import wabot
+
+bot = wabot.browser(session="scraper1", browser="firefox")   # create + persist
+# process exits; browser stays open
+bot = wabot.browser(session="scraper1", browser="firefox")   # second process reattaches
+```
+
+- `session=None` (default): ephemeral — the browser is not persisted and the store is untouched. With no `host`, this is a plain local Selenium driver that dies with the process; with a `host`, it connects to that server but the session is still not saved.
+- `session="name"` + `host="http://..."` → persist via `ExternalServer`; `session="name"` without `host` → persist via `ManagedService`.
+- Housekeeping: `wabot.sessions()` lists saved sessions; `wabot.destroy(name)` quits the browser, kills a managed service, removes the entry.
+
+## Core API
+
+**`Browser`** (`browser.py`, rename of `BrowserProxy`)
+- Facade: holds driver + *current page*; `set_page(PageClass)` instantiates and runs the page's `verify()` gate; `__getattr__` delegates unknown attributes to the current page (`bot.login()` → `bot.page.login()`); `bot[key]` resolves elements on the current page.
+- `good` flag properly initialized to `True` (today it is never set, so the refuse-after-exception guard is broken); `perform()` keeps trap-and-flip behavior; `reset_good_status()` retained.
+- Dropped: unused `switch_after` decorator, NHSN-specific `RC` enum values, hardcoded `/home/mathew/firefox_prof` profile path, PhantomJS remnants, `firefox1/firefox2/chromium1/chromium2` naming (now just `firefox` / `chromium` — with `chrome` accepted as an alias, since chromedriver drives both — plus the optional `host`).
+- User-agent becomes an optional parameter defaulting to the browser's own UA (the hardcoded 2006 MSIE 7 string is an anti-stealth beacon today).
+- Logging: stdlib only — `LOGGER.trace` → `debug`, `LOGGER.warn` → `warning` (the `trace` calls crash under stdlib logging unless the host app registers a custom level).
+
+**`Page` + fields** — declarative design preserved
+- Class-level `elements = {key: (type, accessors)}`; lookup walks the MRO so subclasses inherit/override element maps; `page[key]` returns typed wrappers; `NullField` stays the falsy null-object for missing elements.
+- Consolidate the duplicated find/wait logic; replace bare `except:` with specific Selenium exception types.
+- Selenium 4 port subtlety: `move_to_element_with_offset` now measures from the element **center** (was top-left in Selenium 3), so the Gaussian click math is recomputed relative to center — otherwise human-like clicks land outside small elements.
+
+**`pacing.py`** — stealth extracted into a policy
+- `HumanPacing` (default): current behavior — random 3–11 s delays per field type, Gaussian click offsets.
+- `NoPacing`: instant actions, exact clicks (used by tests and trusted-site automation).
+- `Browser` holds the policy; fields and `Page.click` consult it instead of calling `time.sleep` directly.
+
+**Screenshots** (`screenshot.py`)
+- Firefox: Selenium 4 native `get_full_page_screenshot_as_file`.
+- Chromium: keep the PIL viewport-stitching workaround, isolated in this module.
+
+## Error handling
+
+Philosophy stays *log and refuse, don't crash the scraper*:
+- Missing elements → `NullField`; failed actions → `False`; exceptions in `perform()` → logged with traceback, `good = False`.
+- Dead reattach → warn, prune, create fresh (never crash on a stale session).
+- Corrupt store → rename aside, rebuild.
+- Dead managed service → detected by `/status` health check before driving.
+- No bare `except:` — catch specific Selenium exceptions so real bugs surface in logs.
+
+## Testing
+
+Two pytest tiers split by the `integration` marker:
+
+**Unit** (default, fast, no browsers): element resolution through the MRO; field wrappers against a mocked driver; pacing policies; `SessionStore` round-trip, eviction, and corruption handling on `tmp_path`; reattach logic against a mocked executor.
+
+**Integration** (`-m integration`): real headless Firefox + Chromium driving static HTML fixtures (login form, multi-page flow) served locally — no external network. Flagship test: create a session via `ManagedService`, then a **second Python process** (via `subprocess`) reattaches by name and asserts browser state (URL, filled field value) survived — proving cross-process pickup end to end. `ExternalServer` mode is tested against a locally spawned chromedriver used as the "external" URL (no Java Grid required). Integration tests always run with `NoPacing`.
+
+## Documentation
+
+Sphinx in `docs/`, `sphinx-rtd-theme`, autodoc + napoleon (Google-style docstrings on the public API). Pages:
+1. Quickstart — install, first scrape.
+2. Persistent sessions guide — the flagship feature; managed services vs external servers; the Grid `--session-timeout` caveat.
+3. Page-objects guide — modeling a site with `Page` + `elements`.
+4. API reference (autodoc).
+
+Local build only (`uv run sphinx-build docs docs/_build/html`); no hosting setup for now.
+
+## Out of scope
+
+- CI pipeline (can be added later on Gitea).
+- Read the Docs hosting.
+- New automation features beyond parity (e.g., async API, non-Selenium backends).
+- PhantomJS and the legacy `firefox1`-style driver taxonomy (deliberately removed).
+
+## Known latent bugs fixed by this work
+
+- `LOGGER.trace()` calls crash under stdlib logging.
+- `_clean_old_pickles()` references undefined `final_name`.
+- `BrowserProxy.get_driver()` calls `self._create_driver_chromium()` which doesn't exist on that class.
+- `self.good` never initialized, breaking the refuse-after-exception guard.
+- `appdirs` and `Pillow` used but undeclared in `setup.py`.