wabot/docs/superpowers/specs/2026-07-07-wabot-modernization-design.md

WABot Modernization — Design

Date: 2026-07-07 Status: Approved (brainstorming session with Mathew) Approach: Rebuild the core on the existing design (approach B) — port proven logic into cleanly separated modules; redesign the session-persistence layer; free API redesign was explicitly approved (no downstream consumers to protect).

Context & goals

WABot is a Selenium-wrapping browser-automation library (page-object pattern, stateful multi-page flows, anti-bot stealth behavior). It is stuck on Selenium 3-era APIs because its flagship feature — saving a browser so a different Python process can pick it up — was implemented by pickling live webdriver objects with dill, which broke on Selenium 4.

Goals, as requested:

1. Modern packaging with uv (pyproject.toml, lockfile).
2. Sphinx + sphinx-rtd-theme documentation.
3. pytest suite (unit + real-browser integration).
4. Migrate to latest Selenium 4, keeping core functionality — especially cross-process browser pickup, rebuilt on a sound mechanism.
5. General code quality: fix latent bugs, drop dead code, keep the behavioral subtleties that make the library effective.

Decisions from the brainstorming Q&A

Question	Decision
API compatibility	Free to redesign; keep the concepts, not the exact signatures
Browser host for persistence	Both: external Selenium server URL and wabot-spawned detached driver service
Test depth	Unit (mocked) + integration (real headless browsers, local HTML fixtures, cross-process reattach e2e)
Stealth pacing	Configurable policy; human-like behavior stays the default

Key insight (why persistence broke, and the fix)

A Remote webdriver is a thin HTTP client around (command_executor_url, session_id). Pickling the whole object was always fragile and is unworkable on Selenium 4. The robust mechanism is to persist just the URL + session ID as JSON and reattach by constructing a Remote subclass whose start_session adopts the saved session ID instead of creating a new session. This works against any WebDriver-protocol server: a Selenium Grid or a bare chromedriver/geckodriver. dill is removed entirely. The browser must be hosted by a process that outlives Python — same constraint as the old design, where only remote_* drivers were persistable.

Tooling & layout

• pyproject.toml managed by uv, uv_build backend, requires-python = ">=3.10".
• Runtime deps: selenium>=4 (latest), platformdirs (replaces dead appdirs), Pillow. Removed: dill, appdirs.
• Dev group: pytest, pytest-cov, ruff (lint + format), sphinx, sphinx-rtd-theme.
• Commands: uv sync, uv run pytest, uv run pytest -m integration, uv run ruff check, uv run sphinx-build docs docs/_build/html.

wabot/
├── pyproject.toml
├── uv.lock
├── src/wabot/
│   ├── __init__.py         # explicit public API (no star exports); wabot.browser() entry point
│   ├── browser.py          # Browser (was BrowserProxy) — facade over driver + current page
│   ├── page.py             # Page base class + element resolution
│   ├── fields.py           # TextField / SelectField / CheckField / NullField
│   ├── pacing.py           # HumanPacing / NoPacing policies
│   ├── sessions.py         # SessionStore — JSON persistence + stale eviction
│   ├── hosts.py            # ExternalServer / ManagedService driver hosts
│   └── screenshot.py       # Chromium full-page stitching (PIL)
├── tests/
│   ├── unit/
│   ├── integration/
│   └── fixtures/           # static HTML pages (login form, multi-page flow)
└── docs/                   # Sphinx

The legacy setup.py and top-level wabot/ package are removed once the src/ layout lands.

Session persistence layer

Three cooperating pieces (this replaces all pickle logic in create_browser.py):

SessionStore (sessions.py)

• JSON file at platformdirs.user_data_dir('wabot')/sessions.json.
• Schema: {name: {executor_url, session_id, browser, created_at, service_pid?, service_port?}}.
• Entries older than a configurable max age (default 3 days, matching current behavior) are evicted on load.
• A corrupt file is renamed aside (sessions.json.bad) and recreated — never a crash.

DriverHost (hosts.py) — small interface: ensure_running() -> executor_url.

• ExternalServer(url): a Selenium server/Grid the user runs; wabot just connects. Docs must note Grid 4 reaps idle sessions after 5 minutes unless --session-timeout is raised.
• ManagedService(browser): wabot spawns chromedriver/geckodriver as a detached process (new process session, output redirected to a log file in the wabot data dir) so it outlives Python; records port + pid in the store. Driver binaries are found on PATH first, falling back to Selenium Manager's downloaded copies. On reattach the service's /status endpoint is health-checked before use.

Reattach

• A private Remote subclass overrides start_session to adopt the saved session_id (no new session created).
• Liveness validation: touch driver.current_url. On failure: log a warning, prune the store entry, fall through to fresh creation.

Public flow

import wabot

bot = wabot.browser(session="scraper1", browser="firefox")   # create + persist
# process exits; browser stays open
bot = wabot.browser(session="scraper1", browser="firefox")   # second process reattaches

• session=None (default): ephemeral — the browser is not persisted and the store is untouched. With no host, this is a plain local Selenium driver that dies with the process; with a host, it connects to that server but the session is still not saved.
• session="name" + host="http://..." → persist via ExternalServer; session="name" without host → persist via ManagedService.
• Housekeeping: wabot.sessions() lists saved sessions; wabot.destroy(name) quits the browser, kills a managed service, removes the entry.

Core API

Browser (browser.py, rename of BrowserProxy)

• Facade: holds driver + current page; set_page(PageClass) instantiates and runs the page's verify() gate; __getattr__ delegates unknown attributes to the current page (bot.login() → bot.page.login()); bot[key] resolves elements on the current page.
• good flag properly initialized to True (today it is never set, so the refuse-after-exception guard is broken); perform() keeps trap-and-flip behavior; reset_good_status() retained.
• Dropped: unused switch_after decorator, NHSN-specific RC enum values, hardcoded /home/mathew/firefox_prof profile path, PhantomJS remnants, firefox1/firefox2/chromium1/chromium2 naming (now just firefox / chromium — with chrome accepted as an alias, since chromedriver drives both — plus the optional host).
• User-agent becomes an optional parameter defaulting to the browser's own UA (the hardcoded 2006 MSIE 7 string is an anti-stealth beacon today).
• Logging: stdlib only — LOGGER.trace → debug, LOGGER.warn → warning (the trace calls crash under stdlib logging unless the host app registers a custom level).

Page + fields — declarative design preserved

• Class-level elements = {key: (type, accessors)}; lookup walks the MRO so subclasses inherit/override element maps; page[key] returns typed wrappers; NullField stays the falsy null-object for missing elements.
• Consolidate the duplicated find/wait logic; replace bare except: with specific Selenium exception types.
• Selenium 4 port subtlety: move_to_element_with_offset now measures from the element center (was top-left in Selenium 3), so the Gaussian click math is recomputed relative to center — otherwise human-like clicks land outside small elements.

pacing.py — stealth extracted into a policy

• HumanPacing (default): current behavior — random 3–11 s delays per field type, Gaussian click offsets.
• NoPacing: instant actions, exact clicks (used by tests and trusted-site automation).
• Browser holds the policy; fields and Page.click consult it instead of calling time.sleep directly.

Screenshots (screenshot.py)

• Firefox: Selenium 4 native get_full_page_screenshot_as_file.
• Chromium: keep the PIL viewport-stitching workaround, isolated in this module.

Error handling

Philosophy stays log and refuse, don't crash the scraper:

• Missing elements → NullField; failed actions → False; exceptions in perform() → logged with traceback, good = False.
• Dead reattach → warn, prune, create fresh (never crash on a stale session).
• Corrupt store → rename aside, rebuild.
• Dead managed service → detected by /status health check before driving.
• No bare except: — catch specific Selenium exceptions so real bugs surface in logs.

Testing

Two pytest tiers split by the integration marker:

Unit (default, fast, no browsers): element resolution through the MRO; field wrappers against a mocked driver; pacing policies; SessionStore round-trip, eviction, and corruption handling on tmp_path; reattach logic against a mocked executor.

Integration (-m integration): real headless Firefox + Chromium driving static HTML fixtures (login form, multi-page flow) served locally — no external network. Flagship test: create a session via ManagedService, then a second Python process (via subprocess) reattaches by name and asserts browser state (URL, filled field value) survived — proving cross-process pickup end to end. ExternalServer mode is tested against a locally spawned chromedriver used as the "external" URL (no Java Grid required). Integration tests always run with NoPacing.

Documentation

Sphinx in docs/, sphinx-rtd-theme, autodoc + napoleon (Google-style docstrings on the public API). Pages:

1. Quickstart — install, first scrape.
2. Persistent sessions guide — the flagship feature; managed services vs external servers; the Grid --session-timeout caveat.
3. Page-objects guide — modeling a site with Page + elements.
4. API reference (autodoc).

Local build only (uv run sphinx-build docs docs/_build/html); no hosting setup for now.

Out of scope

• CI pipeline (can be added later on Gitea).
• Read the Docs hosting.
• New automation features beyond parity (e.g., async API, non-Selenium backends).
• PhantomJS and the legacy firefox1-style driver taxonomy (deliberately removed).

Known latent bugs fixed by this work

• LOGGER.trace() calls crash under stdlib logging.
• _clean_old_pickles() references undefined final_name.
• BrowserProxy.get_driver() calls self._create_driver_chromium() which doesn't exist on that class.
• self.good never initialized, breaking the refuse-after-exception guard.
• appdirs and Pillow used but undeclared in setup.py.