universal-scraping-architect

---
name: universal-scraping-architect
description: Use when scraping websites, extracting data from URLs, crawling documents, or building validation-heavy data pipelines.
category: Engineering
version: 1.0.0
tools: []
---

# Universal Scraping Architect

Design complete, robust data-extraction pipelines with intelligent routing, validation, and token-budget tracking.

## Three Extraction Modes

### Mode 1: API-Driven (Firecrawl)
Use when the source is a public URL, heavily dynamic (JS/SPA), requires search-first discovery, or involves bulk crawling across a domain.

### Mode 2: Local Python (Traditional)
Use when extracting from local files (PDF, Excel, CSV), the data is private/sensitive, or the target is simple static HTML where Firecrawl is overkill. Libraries: BeautifulSoup4 for HTML parsing, Pandas for data normalization.

### Mode 3: Hybrid Pipeline
Use when Firecrawl handles URL discovery/web extraction, but local Python (Pandas) is required to clean, normalize, and structure the output before saving.

## The Extraction Pipeline

1. **Route the Approach:** Explicitly state whether Firecrawl or Local Python is being used and why.
2. **Track Budgets:** Estimate Firecrawl API quotas or LLM token context limits before executing large jobs.
3. **Extract Safely:** Implement checkpointing for multi-page jobs. Handle pagination and dynamic layouts gracefully.
4. **Validate & Clean:** Validate extraction results for structural correctness. Check required fields and duplicates against the pipeline spec before delivering. Default output formats: CSV for tabular data, JSON for nested structures, Markdown for clean text.
5. **Handle Private Data:** If user asks to send local sensitive files to external APIs, flag the privacy risk and suggest Mode 2 (Local Python).

## Proactive Checks

- **Hardcoded API Keys:** Flag immediately and rewrite to use `os.getenv('FIRECRAWL_API_KEY')`.
- **Missing Pagination:** If target implies hundreds of records but pagination logic is not requested, flag it and add checkpointing.

## Anti-Patterns

- Never use highly nested CSS selectors (e.g., `div > span > ul > li:nth-child(3)`). Use data attributes or robust structural anchors.
- Never scrape without checking `robots.txt` or implementing sensible rate limits.
- Never blindly write scraped data without checking if the array is empty or missing critical keys.