ProScraper

A personal-use CLI scraping platform powered by SerpAPI for structured data extraction. Ships 4 scrapers with SQLite persistence, response caching, batch mode, and LLM-agent integration.

Scrapers

Command	Source	Description
google-hotels	Google Hotels	Accommodation search by city/dates/guests
google-flights	Google Flights	Flight search by airports/dates, one-way or round-trip
tripadvisor	TripAdvisor	Places, restaurants, hotels, attractions
google-scholar	Google Scholar	Academic papers with citations and year filtering

Setup

Requires Python 3.11+ and a SerpAPI key.

git clone <repo-url> && cd proscraper
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
export SERPAPI_API_KEY=your_key_here

Usage

# Hotels
proscraper google-hotels "hotels in NYC" --check-in-date 2026-03-01 --check-out-date 2026-03-03

# Flights
proscraper google-flights JFK LAX --outbound-date 2026-03-01

# TripAdvisor
proscraper tripadvisor "restaurants in Paris" --category restaurants

# Scholar
proscraper google-scholar "machine learning" --year-from 2023 --num 20

All commands output structured JSON to stdout.

Common flags

Flag	Description
-q / --quiet	Suppress progress output
--retries N	Max retry attempts (default: 3)
--no-cache	Bypass response cache, always fetch fresh
--no-store	Skip persisting results to SQLite
--batch FILE	JSON file (or - for stdin) with array of query objects

Batch mode

# From file
echo '[{"query": "hotels in NYC"}, {"query": "hotels in London"}]' > queries.json
proscraper google-hotels --batch queries.json

# From stdin
echo '[{"query": "machine learning"}, {"query": "deep learning"}]' | proscraper google-scholar --batch -

History

Query past scrape results stored in SQLite:

proscraper history
proscraper history --scraper google-hotels --since 2026-02-01
proscraper history --search "NYC" --full

Schema introspection

proscraper schema google-hotels    # JSON schema for input/output models
proscraper schema google-flights

LLM agent integration

proscraper --prompt    # Outputs structured guide for LLM system prompts

Storage

Results are persisted to SQLite at ~/.proscraper/proscraper.db. SerpAPI responses are cached (SHA256-keyed with TTL) to avoid redundant API calls.

Override the DB location:

export PROSCRAPER_DB=/path/to/custom.db

Updating

cd proscraper
git pull
# editable install picks up changes automatically

Development

pip install -e ".[dev]"
pytest tests/ -v

License

Personal use.