docs: rewrite README and CLAUDE.md for the modernized library

Reflect the final architecture (Browser facade in _browser.py, JSON
session persistence, ManagedService/ExternalServer hosts, pacing
policies), correct the stale "no test suite" claim now that uv +
pytest (140 unit + 9 integration) + ruff + Sphinx are in place, and
round out .gitignore with .venv/, .pytest_cache/, .ruff_cache/.
This commit is contained in:
Mathew Sir Guest the best 2026-07-08 23:10:45 -06:00
parent 163f71e51b
commit 5ec5ad615a
3 changed files with 54 additions and 20 deletions

5
.gitignore vendored

@ -39,6 +39,11 @@ htmlcov/
.cache
nosetests.xml
coverage.xml
.pytest_cache/
# uv / ruff
.venv/
.ruff_cache/
# Translations
*.mo

@ -6,28 +6,33 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
WABot is a Python library wrapping Selenium for stateful browser automation (login flows, multi-page form filling, scraping). Its two distinguishing features:
1. **Page-object automation with state carried between pages** — consumers model each website page as a `Page` subclass and drive them through a single `BrowserProxy`.
2. **Browser sessions that survive the Python process**a remote webdriver reference is persisted to disk so a *different* Python process can reattach to the same still-open browser.
1. **Page-object automation with state carried between pages** — consumers model each website page as a `Page` subclass and drive them through a single `Browser` facade.
2. **Browser sessions that survive the Python process**session identity (executor URL + session id) is persisted to disk so a *different* Python process can reattach to the same still-open browser, whether it's running under an external WebDriver server or a wabot-managed detached driver service.
There is currently no test suite, linter, or CI. Packaging is a legacy `setup.py` (`pip install -e .` to develop). The code is pinned to Selenium 3-era APIs (`desired_capabilities`, `browser_profile`, `switch_to_alert`); it does not run against Selenium 4+ without migration.
Tooling: uv (`uv sync`), pytest (`uv run pytest`; real-browser suite via
`uv run pytest -m integration`), ruff, Sphinx (`uv run sphinx-build -M html docs docs/_build`).
Selenium 4 (pinned <5: session reattach relies on stable 4.x internals).
## Architecture
Consumer code talks to one object, `BrowserProxy` (`wabot/api.py`), which composes everything else:
Consumer code talks to one object, `Browser` (`src/wabot/_browser.py` — underscored so the module doesn't shadow the `wabot.browser()` factory function), which composes everything else:
- **`BrowserProxy`** holds the webdriver plus a *current page* object and acts as a facade. `__getattr__` delegates unknown attributes to the current page, so `bot.login()` invokes `bot.page.login()`; `bot[key]` resolves page elements. `set_page(PageClass)` swaps the current page and calls its `verify()` gate. `perform(name, ...)` invokes a page method with exception trapping — any failure flips `self.good = False`, after which all further actions are refused (`REFUSE_AFTER_EXCEPTION`), forcing an explicit `reset_good_status()`.
- **`Page`** (`wabot/page.py`) is the base class consumers subclass. Pages declare a class-level `elements = {key: (type, accessors)}` map; lookup walks the MRO, so subclasses inherit and override parent element maps. `page[key]` returns a typed field wrapper, never a raw WebElement.
- **Field wrappers** (`wabot/fields.py`): `TextField`, `SelectField`, `CheckField` wrap elements with `get_value`/`set_value`; unknown attribute access falls through to the underlying selenium element. `NullField` is a falsy null-object returned for missing elements so callers can `if el:` instead of catching exceptions.
- **`CreateBrowser`** (`wabot/create_browser.py`) is the driver factory. Local drivers (Chrome/Firefox) are plain. Remote drivers (Selenium server expected at `localhost:4444`) are serialized with `dill` to `appdirs.user_data_dir('wabot')/saved_browser_instances.pickle`, keyed by session name with a timestamp (refs older than 3 days are purged). On the next request for the same session name, the pickled driver is loaded and validated by touching `driver.current_url` — that is the reattach mechanism.
- **`Browser`** holds the webdriver plus a *current page* object and acts as a facade. `__getattr__` delegates unknown attributes to the current page, so `bot.login()` invokes `bot.page.login()`; `bot[key]` resolves page elements. `set_page(PageClass)` swaps the current page and calls its `verify()` gate. `perform(name, ...)` invokes a page method with exception trapping — any failure flips `self.good = False`, after which all further delegated actions are refused, forcing an explicit `reset()`. `quit()` tears down the driver and, for a wabot-managed service, stops the detached process too (a plain `driver.quit()` would leave it running); external servers are left alone.
- **`Page`** (`src/wabot/page.py`) is the base class consumers subclass. Pages declare a class-level `elements = {key: (type, accessors)}` map; lookup walks the MRO, so subclasses inherit and override parent element maps. `page[key]` returns a typed field wrapper, never a raw WebElement.
- **Field wrappers** (`src/wabot/fields.py`): `TextField`, `SelectField`, `CheckField` wrap elements with `get_value`/`set_value`; unknown attribute access falls through to the underlying selenium element. `NullField` is a falsy null-object returned for missing elements so callers can `if el:` instead of catching exceptions.
- **Pacing policies** (`src/wabot/pacing.py`): `HumanPacing` (default) and `NoPacing`. Field wrappers and `Page` ask the active policy for a per-action delay and a click offset instead of hardcoding stealth behavior, so speed/stealth is swappable per `Browser` instance.
- **Sessions** (`src/wabot/sessions.py`): `SessionRecord`/`SessionStore` persist only what's needed to reattach — `executor_url` and `session_id`, plus `service_pid`/`service_port` bookkeeping for managed services — as JSON at `platformdirs.user_data_dir("wabot")/sessions.json`, keyed by session name. Records older than `max_age` (default 3 days) are evicted on load; a corrupt store is quarantined to `sessions.json.bad` rather than crashing the caller.
- **Driver hosts** (`src/wabot/hosts.py`): two ways to obtain a WebDriver server URL. `ExternalServer` points at a Selenium Grid or bare driver the user already runs (validated via `GET /status`). `ManagedService` spawns `chromedriver`/`geckodriver` itself, detached (`start_new_session=True`) so it survives this Python process exiting; `stop_service(pid)` is how `Browser.quit()`/`destroy()` shut it back down. `ReattachingRemote.start_session` skips `NEW_SESSION` entirely and adopts a saved `session_id`; `attach()` builds one and probes `driver.current_url` to confirm the session is still live before handing it back.
- **`wabot.browser()`** (`src/wabot/__init__.py`) is the public entry point tying it together: no `session=` gives an ephemeral local-or-remote browser; a named session looks up its saved record, reattaches if the host answers, and otherwise spins up a host (external or managed) and saves a fresh record. `wabot.sessions()`/`wabot.destroy()` round out the public API.
- **Screenshots** (`src/wabot/screenshot.py`): Firefox exposes a raw `moz/screenshot/full` command over Remote for full-page capture; Chromium has no Remote equivalent, so viewport tiles are captured and stitched with Pillow instead.
**Human-like behavior is intentional, not accidental slowness**: clicks land at Gaussian-random coordinates within the element instead of Selenium's default (0,0), and field setters `sleep(random.uniform(...))` between actions. Chrome full-page screenshots are stitched from viewport tiles with PIL to work around chromedriver capturing only the visible area. Don't "optimize" these away.
**Human-like behavior is intentional, not accidental slowness**: with the default `HumanPacing`, clicks land at Gaussian-random offsets from the element's in-view center (Selenium 4 measures click offsets from the center, not top-left, unlike Selenium 3) instead of a plain click, and field setters sleep a random interval between actions. Pass `pacing=wabot.NoPacing()` for instant, exact behavior (what the test suite uses). Don't "optimize" these away.
## Gotchas
- `LOGGER.trace(...)` is called throughout, but stdlib `logging` has no `trace` method — the consuming application is expected to register a custom TRACE level; without it those calls raise.
- `appdirs` and `Pillow` are imported but missing from `install_requires` in `setup.py` (only `dill` and `selenium` are declared).
- `wabot/__init__.py` star-exports `api`, `page`, and `fields`, so anything public in those modules is public API.
- Remote driver types require an external Selenium server already running on port 4444; nothing in this repo starts one.
- External hosts (`ExternalServer`) must outlive the Python process — wabot never starts or stops them. Only `ManagedService`-spawned drivers are wabot's to stop.
- geckodriver reports `ready: false` on `GET /status` whenever its single session is in use; chromedriver reports `ready: true` even with an active session. Liveness checks (`service_alive`) therefore accept any HTTP 200 with valid JSON and deliberately ignore the `ready` flag — requiring it would work for chromedriver and break for geckodriver.
- Selenium is pinned `<5`: `ReattachingRemote` relies on internals that have been stable across the 4.x line but aren't a public API contract (that `start_session` is the only place a new session gets created, and that `session_id` is injected into every subsequent command). `ReattachingRemote.__init__` raises `RuntimeError` on adoption mismatch as a tripwire if a future selenium release changes this.
## Git

@ -1,9 +1,33 @@
# WABot - Web Automator Bot (python module)
# WABot — Web Automator Bot
Module provides a class which builds on selenium automated-browser control. What it adds
is the ability to carry and maintain-state between web pages. So for e.g. you can automate
a web browser to log-in, navigate through the menus, and fill out a form-- or scrape data.
Stateful Selenium browser automation for Python. Model pages as classes,
drive them through one facade, and — the flagship — keep the browser
alive after your process exits so another Python process can pick it up
by name.
Probably pretty awful code, I'd write it a lot better if I had more time on it. Though I've
used and continue to use to scrape 100,000s of data points with zero human time, performing
complex interactions with websites.k
```python
import wabot
bot = wabot.browser(session="scraper1", browser="firefox")
bot.driver.get("https://example.com/login")
# ... log in; process exits; browser stays open ...
# a different process, later:
bot = wabot.browser(session="scraper1", browser="firefox") # reattaches
```
Human-like pacing (random delays, randomized click positions) is on by
default; pass `pacing=wabot.NoPacing()` for full speed.
## Development
```bash
uv sync # install everything
uv run pytest # unit tests (fast, no browsers)
uv run pytest -m integration # real headless-browser tests
uv run ruff check . && uv run ruff format --check .
uv run sphinx-build -M html docs docs/_build # docs
```
Docs: quickstart, persistent-sessions guide, and API reference under
`docs/` (build locally with the command above).