wabot/docs/superpowers/specs/2026-07-07-wabot-modernization-design.md

# WABot Modernization — Design

**Date:** 2026-07-07
**Status:** Approved (brainstorming session with Mathew)
**Approach:** Rebuild the core on the existing design (approach B) — port proven logic into cleanly separated modules; redesign the session-persistence layer; free API redesign was explicitly approved (no downstream consumers to protect).

## Context & goals

WABot is a Selenium-wrapping browser-automation library (page-object pattern, stateful multi-page flows, anti-bot stealth behavior). It is stuck on Selenium 3-era APIs because its flagship feature — saving a browser so a *different* Python process can pick it up — was implemented by pickling live webdriver objects with `dill`, which broke on Selenium 4.

Goals, as requested:

1. Modern packaging with **uv** (`pyproject.toml`, lockfile).
2. **Sphinx + sphinx-rtd-theme** documentation.
3. **pytest** suite (unit + real-browser integration).
4. Migrate to **latest Selenium 4**, keeping core functionality — especially cross-process browser pickup, rebuilt on a sound mechanism.
5. General code quality: fix latent bugs, drop dead code, keep the behavioral subtleties that make the library effective.

### Decisions from the brainstorming Q&A

| Question | Decision |
|---|---|
| API compatibility | Free to redesign; keep the concepts, not the exact signatures |
| Browser host for persistence | **Both**: external Selenium server URL *and* wabot-spawned detached driver service |
| Test depth | Unit (mocked) + integration (real headless browsers, local HTML fixtures, cross-process reattach e2e) |
| Stealth pacing | Configurable policy; human-like behavior stays the **default** |

### Key insight (why persistence broke, and the fix)

A `Remote` webdriver is a thin HTTP client around `(command_executor_url, session_id)`. Pickling the whole object was always fragile and is unworkable on Selenium 4. The robust mechanism is to persist just the URL + session ID as JSON and **reattach** by constructing a `Remote` subclass whose `start_session` adopts the saved session ID instead of creating a new session. This works against any WebDriver-protocol server: a Selenium Grid *or* a bare chromedriver/geckodriver. `dill` is removed entirely. The browser must be hosted by a process that outlives Python — same constraint as the old design, where only `remote_*` drivers were persistable.

## Tooling & layout

- `pyproject.toml` managed by uv, `uv_build` backend, `requires-python = ">=3.10"`.
- Runtime deps: `selenium>=4` (latest), `platformdirs` (replaces dead `appdirs`), `Pillow`. **Removed:** `dill`, `appdirs`.
- Dev group: `pytest`, `pytest-cov`, `ruff` (lint + format), `sphinx`, `sphinx-rtd-theme`.
- Commands: `uv sync`, `uv run pytest`, `uv run pytest -m integration`, `uv run ruff check`, `uv run sphinx-build docs docs/_build/html`.

```
wabot/
├── pyproject.toml
├── uv.lock
├── src/wabot/
│   ├── __init__.py         # explicit public API (no star exports); wabot.browser() entry point
│   ├── browser.py          # Browser (was BrowserProxy) — facade over driver + current page
│   ├── page.py             # Page base class + element resolution
│   ├── fields.py           # TextField / SelectField / CheckField / NullField
│   ├── pacing.py           # HumanPacing / NoPacing policies
│   ├── sessions.py         # SessionStore — JSON persistence + stale eviction
│   ├── hosts.py            # ExternalServer / ManagedService driver hosts
│   └── screenshot.py       # Chromium full-page stitching (PIL)
├── tests/
│   ├── unit/
│   ├── integration/
│   └── fixtures/           # static HTML pages (login form, multi-page flow)
└── docs/                   # Sphinx
```

The legacy `setup.py` and top-level `wabot/` package are removed once the `src/` layout lands.

## Session persistence layer

Three cooperating pieces (this replaces all pickle logic in `create_browser.py`):

**`SessionStore`** (`sessions.py`)
- JSON file at `platformdirs.user_data_dir('wabot')/sessions.json`.
- Schema: `{name: {executor_url, session_id, browser, created_at, service_pid?, service_port?}}`.
- Entries older than a configurable max age (default 3 days, matching current behavior) are evicted on load.
- A corrupt file is renamed aside (`sessions.json.bad`) and recreated — never a crash.

**`DriverHost`** (`hosts.py`) — small interface: `ensure_running() -> executor_url`.
- `ExternalServer(url)`: a Selenium server/Grid the user runs; wabot just connects. Docs must note Grid 4 reaps idle sessions after 5 minutes unless `--session-timeout` is raised.
- `ManagedService(browser)`: wabot spawns `chromedriver`/`geckodriver` as a **detached** process (new process session, output redirected to a log file in the wabot data dir) so it outlives Python; records port + pid in the store. Driver binaries are found on `PATH` first, falling back to Selenium Manager's downloaded copies. On reattach the service's `/status` endpoint is health-checked before use.

**Reattach**
- A private `Remote` subclass overrides `start_session` to adopt the saved `session_id` (no new session created).
- Liveness validation: touch `driver.current_url`. On failure: log a warning, prune the store entry, fall through to fresh creation.

**Public flow**

```python
import wabot

bot = wabot.browser(session="scraper1", browser="firefox")   # create + persist
# process exits; browser stays open
bot = wabot.browser(session="scraper1", browser="firefox")   # second process reattaches
```

- `session=None` (default): ephemeral — the browser is not persisted and the store is untouched. With no `host`, this is a plain local Selenium driver that dies with the process; with a `host`, it connects to that server but the session is still not saved.
- `session="name"` + `host="http://..."` → persist via `ExternalServer`; `session="name"` without `host` → persist via `ManagedService`.
- Housekeeping: `wabot.sessions()` lists saved sessions; `wabot.destroy(name)` quits the browser, kills a managed service, removes the entry.

## Core API

**`Browser`** (`browser.py`, rename of `BrowserProxy`)
- Facade: holds driver + *current page*; `set_page(PageClass)` instantiates and runs the page's `verify()` gate; `__getattr__` delegates unknown attributes to the current page (`bot.login()` → `bot.page.login()`); `bot[key]` resolves elements on the current page.
- `good` flag properly initialized to `True` (today it is never set, so the refuse-after-exception guard is broken); `perform()` keeps trap-and-flip behavior; `reset_good_status()` retained.
- Dropped: unused `switch_after` decorator, NHSN-specific `RC` enum values, hardcoded `/home/mathew/firefox_prof` profile path, PhantomJS remnants, `firefox1/firefox2/chromium1/chromium2` naming (now just `firefox` / `chromium` — with `chrome` accepted as an alias, since chromedriver drives both — plus the optional `host`).
- User-agent becomes an optional parameter defaulting to the browser's own UA (the hardcoded 2006 MSIE 7 string is an anti-stealth beacon today).
- Logging: stdlib only — `LOGGER.trace` → `debug`, `LOGGER.warn` → `warning` (the `trace` calls crash under stdlib logging unless the host app registers a custom level).

**`Page` + fields** — declarative design preserved
- Class-level `elements = {key: (type, accessors)}`; lookup walks the MRO so subclasses inherit/override element maps; `page[key]` returns typed wrappers; `NullField` stays the falsy null-object for missing elements.
- Consolidate the duplicated find/wait logic; replace bare `except:` with specific Selenium exception types.
- Selenium 4 port subtlety: `move_to_element_with_offset` now measures from the element **center** (was top-left in Selenium 3), so the Gaussian click math is recomputed relative to center — otherwise human-like clicks land outside small elements.

**`pacing.py`** — stealth extracted into a policy
- `HumanPacing` (default): current behavior — random 3–11 s delays per field type, Gaussian click offsets.
- `NoPacing`: instant actions, exact clicks (used by tests and trusted-site automation).
- `Browser` holds the policy; fields and `Page.click` consult it instead of calling `time.sleep` directly.

**Screenshots** (`screenshot.py`)
- Firefox: Selenium 4 native `get_full_page_screenshot_as_file`.
- Chromium: keep the PIL viewport-stitching workaround, isolated in this module.

## Error handling

Philosophy stays *log and refuse, don't crash the scraper*:
- Missing elements → `NullField`; failed actions → `False`; exceptions in `perform()` → logged with traceback, `good = False`.
- Dead reattach → warn, prune, create fresh (never crash on a stale session).
- Corrupt store → rename aside, rebuild.
- Dead managed service → detected by `/status` health check before driving.
- No bare `except:` — catch specific Selenium exceptions so real bugs surface in logs.

## Testing

Two pytest tiers split by the `integration` marker:

**Unit** (default, fast, no browsers): element resolution through the MRO; field wrappers against a mocked driver; pacing policies; `SessionStore` round-trip, eviction, and corruption handling on `tmp_path`; reattach logic against a mocked executor.

**Integration** (`-m integration`): real headless Firefox + Chromium driving static HTML fixtures (login form, multi-page flow) served locally — no external network. Flagship test: create a session via `ManagedService`, then a **second Python process** (via `subprocess`) reattaches by name and asserts browser state (URL, filled field value) survived — proving cross-process pickup end to end. `ExternalServer` mode is tested against a locally spawned chromedriver used as the "external" URL (no Java Grid required). Integration tests always run with `NoPacing`.

## Documentation

Sphinx in `docs/`, `sphinx-rtd-theme`, autodoc + napoleon (Google-style docstrings on the public API). Pages:
1. Quickstart — install, first scrape.
2. Persistent sessions guide — the flagship feature; managed services vs external servers; the Grid `--session-timeout` caveat.
3. Page-objects guide — modeling a site with `Page` + `elements`.
4. API reference (autodoc).

Local build only (`uv run sphinx-build docs docs/_build/html`); no hosting setup for now.

## Out of scope

- CI pipeline (can be added later on Gitea).
- Read the Docs hosting.
- New automation features beyond parity (e.g., async API, non-Selenium backends).
- PhantomJS and the legacy `firefox1`-style driver taxonomy (deliberately removed).

## Known latent bugs fixed by this work

- `LOGGER.trace()` calls crash under stdlib logging.
- `_clean_old_pickles()` references undefined `final_name`.
- `BrowserProxy.get_driver()` calls `self._create_driver_chromium()` which doesn't exist on that class.
- `self.good` never initialized, breaking the refuse-after-exception guard.
- `appdirs` and `Pillow` used but undeclared in `setup.py`.