From 5ec5ad615a35cf40c1c6db6e7a599c116c384219 Mon Sep 17 00:00:00 2001 From: Mathew Sir Guest the best Date: Wed, 8 Jul 2026 23:10:45 -0600 Subject: [PATCH] docs: rewrite README and CLAUDE.md for the modernized library Reflect the final architecture (Browser facade in _browser.py, JSON session persistence, ManagedService/ExternalServer hosts, pacing policies), correct the stale "no test suite" claim now that uv + pytest (140 unit + 9 integration) + ruff + Sphinx are in place, and round out .gitignore with .venv/, .pytest_cache/, .ruff_cache/. --- .gitignore | 5 +++++ CLAUDE.md | 31 ++++++++++++++++++------------- README.md | 38 +++++++++++++++++++++++++++++++------- 3 files changed, 54 insertions(+), 20 deletions(-) diff --git a/.gitignore b/.gitignore index 4ffc73e..71e897b 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,11 @@ htmlcov/ .cache nosetests.xml coverage.xml +.pytest_cache/ + +# uv / ruff +.venv/ +.ruff_cache/ # Translations *.mo diff --git a/CLAUDE.md b/CLAUDE.md index c901b99..14ac831 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,28 +6,33 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co WABot is a Python library wrapping Selenium for stateful browser automation (login flows, multi-page form filling, scraping). Its two distinguishing features: -1. **Page-object automation with state carried between pages** — consumers model each website page as a `Page` subclass and drive them through a single `BrowserProxy`. -2. **Browser sessions that survive the Python process** — a remote webdriver reference is persisted to disk so a *different* Python process can reattach to the same still-open browser. +1. **Page-object automation with state carried between pages** — consumers model each website page as a `Page` subclass and drive them through a single `Browser` facade. +2. **Browser sessions that survive the Python process** — session identity (executor URL + session id) is persisted to disk so a *different* Python process can reattach to the same still-open browser, whether it's running under an external WebDriver server or a wabot-managed detached driver service. -There is currently no test suite, linter, or CI. Packaging is a legacy `setup.py` (`pip install -e .` to develop). The code is pinned to Selenium 3-era APIs (`desired_capabilities`, `browser_profile`, `switch_to_alert`); it does not run against Selenium 4+ without migration. +Tooling: uv (`uv sync`), pytest (`uv run pytest`; real-browser suite via +`uv run pytest -m integration`), ruff, Sphinx (`uv run sphinx-build -M html docs docs/_build`). +Selenium 4 (pinned <5: session reattach relies on stable 4.x internals). ## Architecture -Consumer code talks to one object, `BrowserProxy` (`wabot/api.py`), which composes everything else: +Consumer code talks to one object, `Browser` (`src/wabot/_browser.py` — underscored so the module doesn't shadow the `wabot.browser()` factory function), which composes everything else: -- **`BrowserProxy`** holds the webdriver plus a *current page* object and acts as a facade. `__getattr__` delegates unknown attributes to the current page, so `bot.login()` invokes `bot.page.login()`; `bot[key]` resolves page elements. `set_page(PageClass)` swaps the current page and calls its `verify()` gate. `perform(name, ...)` invokes a page method with exception trapping — any failure flips `self.good = False`, after which all further actions are refused (`REFUSE_AFTER_EXCEPTION`), forcing an explicit `reset_good_status()`. -- **`Page`** (`wabot/page.py`) is the base class consumers subclass. Pages declare a class-level `elements = {key: (type, accessors)}` map; lookup walks the MRO, so subclasses inherit and override parent element maps. `page[key]` returns a typed field wrapper, never a raw WebElement. -- **Field wrappers** (`wabot/fields.py`): `TextField`, `SelectField`, `CheckField` wrap elements with `get_value`/`set_value`; unknown attribute access falls through to the underlying selenium element. `NullField` is a falsy null-object returned for missing elements so callers can `if el:` instead of catching exceptions. -- **`CreateBrowser`** (`wabot/create_browser.py`) is the driver factory. Local drivers (Chrome/Firefox) are plain. Remote drivers (Selenium server expected at `localhost:4444`) are serialized with `dill` to `appdirs.user_data_dir('wabot')/saved_browser_instances.pickle`, keyed by session name with a timestamp (refs older than 3 days are purged). On the next request for the same session name, the pickled driver is loaded and validated by touching `driver.current_url` — that is the reattach mechanism. +- **`Browser`** holds the webdriver plus a *current page* object and acts as a facade. `__getattr__` delegates unknown attributes to the current page, so `bot.login()` invokes `bot.page.login()`; `bot[key]` resolves page elements. `set_page(PageClass)` swaps the current page and calls its `verify()` gate. `perform(name, ...)` invokes a page method with exception trapping — any failure flips `self.good = False`, after which all further delegated actions are refused, forcing an explicit `reset()`. `quit()` tears down the driver and, for a wabot-managed service, stops the detached process too (a plain `driver.quit()` would leave it running); external servers are left alone. +- **`Page`** (`src/wabot/page.py`) is the base class consumers subclass. Pages declare a class-level `elements = {key: (type, accessors)}` map; lookup walks the MRO, so subclasses inherit and override parent element maps. `page[key]` returns a typed field wrapper, never a raw WebElement. +- **Field wrappers** (`src/wabot/fields.py`): `TextField`, `SelectField`, `CheckField` wrap elements with `get_value`/`set_value`; unknown attribute access falls through to the underlying selenium element. `NullField` is a falsy null-object returned for missing elements so callers can `if el:` instead of catching exceptions. +- **Pacing policies** (`src/wabot/pacing.py`): `HumanPacing` (default) and `NoPacing`. Field wrappers and `Page` ask the active policy for a per-action delay and a click offset instead of hardcoding stealth behavior, so speed/stealth is swappable per `Browser` instance. +- **Sessions** (`src/wabot/sessions.py`): `SessionRecord`/`SessionStore` persist only what's needed to reattach — `executor_url` and `session_id`, plus `service_pid`/`service_port` bookkeeping for managed services — as JSON at `platformdirs.user_data_dir("wabot")/sessions.json`, keyed by session name. Records older than `max_age` (default 3 days) are evicted on load; a corrupt store is quarantined to `sessions.json.bad` rather than crashing the caller. +- **Driver hosts** (`src/wabot/hosts.py`): two ways to obtain a WebDriver server URL. `ExternalServer` points at a Selenium Grid or bare driver the user already runs (validated via `GET /status`). `ManagedService` spawns `chromedriver`/`geckodriver` itself, detached (`start_new_session=True`) so it survives this Python process exiting; `stop_service(pid)` is how `Browser.quit()`/`destroy()` shut it back down. `ReattachingRemote.start_session` skips `NEW_SESSION` entirely and adopts a saved `session_id`; `attach()` builds one and probes `driver.current_url` to confirm the session is still live before handing it back. +- **`wabot.browser()`** (`src/wabot/__init__.py`) is the public entry point tying it together: no `session=` gives an ephemeral local-or-remote browser; a named session looks up its saved record, reattaches if the host answers, and otherwise spins up a host (external or managed) and saves a fresh record. `wabot.sessions()`/`wabot.destroy()` round out the public API. +- **Screenshots** (`src/wabot/screenshot.py`): Firefox exposes a raw `moz/screenshot/full` command over Remote for full-page capture; Chromium has no Remote equivalent, so viewport tiles are captured and stitched with Pillow instead. -**Human-like behavior is intentional, not accidental slowness**: clicks land at Gaussian-random coordinates within the element instead of Selenium's default (0,0), and field setters `sleep(random.uniform(...))` between actions. Chrome full-page screenshots are stitched from viewport tiles with PIL to work around chromedriver capturing only the visible area. Don't "optimize" these away. +**Human-like behavior is intentional, not accidental slowness**: with the default `HumanPacing`, clicks land at Gaussian-random offsets from the element's in-view center (Selenium 4 measures click offsets from the center, not top-left, unlike Selenium 3) instead of a plain click, and field setters sleep a random interval between actions. Pass `pacing=wabot.NoPacing()` for instant, exact behavior (what the test suite uses). Don't "optimize" these away. ## Gotchas -- `LOGGER.trace(...)` is called throughout, but stdlib `logging` has no `trace` method — the consuming application is expected to register a custom TRACE level; without it those calls raise. -- `appdirs` and `Pillow` are imported but missing from `install_requires` in `setup.py` (only `dill` and `selenium` are declared). -- `wabot/__init__.py` star-exports `api`, `page`, and `fields`, so anything public in those modules is public API. -- Remote driver types require an external Selenium server already running on port 4444; nothing in this repo starts one. +- External hosts (`ExternalServer`) must outlive the Python process — wabot never starts or stops them. Only `ManagedService`-spawned drivers are wabot's to stop. +- geckodriver reports `ready: false` on `GET /status` whenever its single session is in use; chromedriver reports `ready: true` even with an active session. Liveness checks (`service_alive`) therefore accept any HTTP 200 with valid JSON and deliberately ignore the `ready` flag — requiring it would work for chromedriver and break for geckodriver. +- Selenium is pinned `<5`: `ReattachingRemote` relies on internals that have been stable across the 4.x line but aren't a public API contract (that `start_session` is the only place a new session gets created, and that `session_id` is injected into every subsequent command). `ReattachingRemote.__init__` raises `RuntimeError` on adoption mismatch as a tripwire if a future selenium release changes this. ## Git diff --git a/README.md b/README.md index eec5eaf..6426a0b 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,33 @@ -# WABot - Web Automator Bot (python module) +# WABot — Web Automator Bot -Module provides a class which builds on selenium automated-browser control. What it adds -is the ability to carry and maintain-state between web pages. So for e.g. you can automate -a web browser to log-in, navigate through the menus, and fill out a form-- or scrape data. +Stateful Selenium browser automation for Python. Model pages as classes, +drive them through one facade, and — the flagship — keep the browser +alive after your process exits so another Python process can pick it up +by name. -Probably pretty awful code, I'd write it a lot better if I had more time on it. Though I've -used and continue to use to scrape 100,000s of data points with zero human time, performing -complex interactions with websites.k +```python +import wabot + +bot = wabot.browser(session="scraper1", browser="firefox") +bot.driver.get("https://example.com/login") +# ... log in; process exits; browser stays open ... + +# a different process, later: +bot = wabot.browser(session="scraper1", browser="firefox") # reattaches +``` + +Human-like pacing (random delays, randomized click positions) is on by +default; pass `pacing=wabot.NoPacing()` for full speed. + +## Development + +```bash +uv sync # install everything +uv run pytest # unit tests (fast, no browsers) +uv run pytest -m integration # real headless-browser tests +uv run ruff check . && uv run ruff format --check . +uv run sphinx-build -M html docs docs/_build # docs +``` + +Docs: quickstart, persistent-sessions guide, and API reference under +`docs/` (build locally with the command above).