browser-automation

---
name: browser-automation
description: Use when automating browser tasks: scraping sites, filling forms, capturing screenshots/PDFs, extracting structured data, or building Playwright workflows. Not for writing browser tests.
category: Engineering
version: 1.0.0
tools: []
---

# Browser Automation

Build production-grade web automation and scraping workflows with Playwright — data extraction, form filling, screenshot/PDF capture, session management, and anti-detection patterns. Not for writing browser tests, API testing, or load testing.

**Use for:** scraping structured data (tables, listings, search results) from websites, automating multi-step browser workflows (login, form fill, download), capturing screenshots/PDFs, extracting data from JavaScript-heavy SPAs, building repeatable browser-based data pipelines.

**Why Playwright over Selenium/Puppeteer:**
- Auto-wait built in — no explicit sleep()/waitForElement() for most actions
- Multi-browser from one API — Chromium, Firefox, WebKit, zero config changes
- Native network interception — block ads, mock responses, capture API calls
- Browser contexts give isolated sessions without spinning up new browser instances
- Codegen records actions and generates scripts
- Async-first: Python async/await for high-throughput scraping

## Selector Strategy (priority order)

1. `data-testid` / `data-id` / custom data attributes — stable across redesigns
2. `#id` selectors — unique but may change between deploys
3. Semantic selectors (`article`, `nav`, `main`, `section`) — resilient to CSS changes
4. Class-based (`.product-card`, `.price`) — brittle if classes are generated (CSS modules)
5. Positional (`nth-child()`, `nth-of-type()`) — last resort, breaks on layout changes

Use XPath only when CSS cannot express the relationship (ancestor traversal, text-based selection). Pagination handling covers next-button, URL-based (`?page=N`), infinite scroll, and load-more patterns.

## Core Techniques

**Form filling & multi-step workflows** — break multi-step forms into one function per step: fill fields, click Next/Continue, wait for the next step (URL change or DOM element). Covers login flows, multi-page forms, file uploads (including drag-and-drop), and native/custom dropdown handling.

**Screenshots & PDF** — full-page (`page.screenshot(full_page=True)`), single-element, and PDF export (Chromium only, `page.pdf(format="A4", print_background=True)`). For visual regression, store baselines with a `{page}_{viewport}_{state}.png` naming convention.

**Structured data extraction** — tables to JSON (map `<thead>`/`<tbody>` into dictionaries), listings to arrays (field-selector maps supporting `::attr()` for attributes), and recursive extraction for nested/threaded data (comments with replies, category trees).

**Cookie & session management** — save/restore cookies via `context.cookies()` / `context.add_cookies()`; save full storage state (cookies + localStorage) with `context.storage_state()` and restore via `browser.new_context(storage_state=...)`. Save state after login and reuse across sessions; verify session validity with a lightweight request to a protected page before starting a long job.

**Anti-detection** (apply in priority order):
1. WebDriver flag removal — strip `navigator.webdriver` via an init script (critical)
2. Custom, rotating user agents — never the default headless UA
3. Realistic viewport (1920×1080) — the default 800×600 headless viewport is a red flag
4. Request throttling — randomized delays between actions
5. Proxy support, per-browser or per-context

Full stealth stack also covers navigator property hardening, WebGL/canvas fingerprint evasion, behavioral simulation (mouse movement, typing speed, scroll patterns), and proxy rotation.

**Dynamic content** — wait on content selectors for SPAs, not the load event; use `page.expect_response("**/api/data*")` to intercept and wait on specific API calls; Playwright pierces open Shadow DOM with the `>>` operator (`page.locator("custom-element >> .inner-class")`); trigger lazy-loaded images with `scroll_into_view_if_needed()`.

**Error handling & retry** — wrap interactions in retry logic with exponential backoff (1s, 2s, 4s); fall back to alternative selectors on `TimeoutError` before failing; capture error-state screenshots on unexpected failures; detect HTTP 429 and respect `Retry-After` headers.

## Workflows

**Single-page extraction** — launch headed during development, headless in production; navigate and wait for the content selector; extract with a field-selector map; validate for nulls/expected types; return as JSON.

**Multi-page scraping with pagination** — launch with anti-detection settings; extract the current page; check whether Next exists and is enabled; click and wait for new content (not just navigation, since SPAs swap content without a full nav event); repeat until no next page or a max-pages cap; deduplicate by unique key; stream output incrementally rather than holding everything in memory.

**Authenticated workflow automation** — check for existing session state; if none, log in and save `storage_state`; navigate using the saved session; fill multi-step form data step by step; use `page.expect_download()` around the triggering click to capture file downloads.

## Anti-Patterns

- **Hardcoded waits** (`wait_for_timeout(5000)`) — flaky and slow; use `wait_for_selector`, `wait_for_url`, `expect_response`, or `wait_for_load_state` instead.
- **No error recovery** — wrap every page interaction in try/except, capture error-state screenshots, retry with exponential backoff.
- **Ignoring robots.txt** — fetch and parse it first; respect `Crawl-delay`; skip disallowed paths; identify your bot in the User-Agent at scale.
- **Storing credentials in scripts** — use environment variables or a secrets manager, never hardcoded usernames/passwords.
- **No rate limiting** — add 1–3s randomized delays for polite scraping; watch for 429s; back off exponentially.
- **Selector fragility** — avoid auto-generated class names (`.css-1a2b3c`) and deep nesting; prefer data attributes, semantic HTML, or text-based locators.
- **Not cleaning up browser instances** — always close the browser via try/finally or an async context manager.
- **Running headed in production** — develop headed, deploy headless.

**Cross-references:** for public APIs, hit them directly instead of scraping rendered pages — faster, more reliable, less detectable. Credential storage for authenticated workflows belongs to an env/secrets-management skill.