Skip to content
74 changes: 50 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ When using a custom API URL (anything other than `https://api.firecrawl.dev`), a

### `scrape` - Scrape URLs

Extract content from any webpage. Pass multiple URLs to scrape them concurrently -- each result is saved to `.firecrawl/` automatically.
Extract content from any webpage. A single URL writes to stdout unless `-o` is supplied. Pass multiple URLs to scrape them concurrently; without output flags, successful results are saved to `.firecrawl/`, while failed URLs are reported on stderr without per-URL files. With `--json` or `-o <path>`, results form one JSON array in input order, written to stdout or the requested file. Each item contains `url`, `success`, and the full `data` (including metadata) or an `error`. `--pretty` indents the array. Any failed URL makes the command exit nonzero, after successful results and errors have been saved.

```bash
# Basic usage (outputs markdown)
Expand All @@ -234,29 +234,30 @@ firecrawl scrape https://firecrawl.dev https://firecrawl.dev/blog https://docs.f

#### Scrape Options

| Option | Description |
| -------------------------- | ------------------------------------------------------- |
| `-f, --format <formats>` | Output format(s), comma-separated |
| `-H, --html` | Shortcut for `--format html` |
| `-S, --summary` | Shortcut for `--format summary` |
| `--only-main-content` | Extract only main content (removes navs, footers, etc.) |
| `--wait-for <ms>` | Wait time before scraping (for JS-rendered content) |
| `--screenshot` | Take a screenshot |
| `--full-page-screenshot` | Take a full page screenshot |
| `--include-tags <tags>` | Only include specific HTML tags |
| `--exclude-tags <tags>` | Exclude specific HTML tags |
| `--max-age <milliseconds>` | Maximum age of cached content in milliseconds |
| `--lockdown` | Enable lockdown mode for the scrape |
| `--redact-pii` | Redact personally identifiable information from output |
| `--schema <json>` | JSON schema for structured extraction |
| `--schema-file <path>` | Path to JSON schema file for structured extraction |
| `--actions <json>` | JSON actions array to run during scrape |
| `--actions-file <path>` | Path to JSON actions file |
| `--proxy <proxy>` | Proxy mode for scraping (for example, `auto`, `basic`) |
| `-o, --output <path>` | Save output to file |
| `--json` | Output as JSON format |
| `--pretty` | Pretty print JSON output |
| `--timing` | Show request timing info |
| Option | Description |
| -------------------------- | ------------------------------------------------------------ |
| `-f, --format <formats>` | Output format(s), comma-separated |
| `-H, --html` | Shortcut for `--format html` |
| `-S, --summary` | Shortcut for `--format summary` |
| `--only-main-content` | Extract only main content (removes navs, footers, etc.) |
| `--wait-for <ms>` | Wait time before scraping (for JS-rendered content) |
| `--timeout <ms>` | Request timeout in milliseconds |
| `--screenshot` | Take a screenshot |
| `--full-page-screenshot` | Take a full page screenshot |
| `--include-tags <tags>` | Only include specific HTML tags |
| `--exclude-tags <tags>` | Exclude specific HTML tags |
| `--max-age <milliseconds>` | Maximum cached-content age; use `0` to request fresh content |
| `--lockdown` | Enable lockdown mode for the scrape |
| `--redact-pii` | Redact personally identifiable information from output |
| `--schema <json>` | JSON schema for structured extraction |
| `--schema-file <path>` | Path to JSON schema file for structured extraction |
| `--actions <json>` | JSON actions array to run during scrape |
| `--actions-file <path>` | Path to JSON actions file |
| `--proxy <proxy>` | Proxy mode for scraping (for example, `auto`, `basic`) |
| `-o, --output <path>` | Save output to file |
| `--json` | Output as JSON format |
| `--pretty` | Pretty print JSON output |
| `--timing` | Show request timing info |

#### Available Formats

Expand Down Expand Up @@ -345,6 +346,8 @@ firecrawl search "API documentation" --scrape --scrape-formats markdown,links
firecrawl search "AI data tools"
```

Search prints the returned Search ID and credit usage to stderr when available, keeping stdout suitable for piping. `--json` and `--pretty` preserve the response metadata even when no results match; `-o` also saves empty results.

#### Search Options

| Option | Description |
Expand All @@ -364,6 +367,7 @@ firecrawl search "AI data tools"
| `--only-main-content` | Include only main content when scraping (default: true) |
| `-o, --output <path>` | Save to file |
| `--json` | Output as compact JSON |
| `--pretty` | Output as pretty-printed JSON |

#### Examples

Expand Down Expand Up @@ -914,6 +918,22 @@ firecrawl https://example.com -o output.md
firecrawl https://example.com --format links --pretty
```

### Receipts, failures, and retries

Search and URL scrape JSON output add a root `receipt` while preserving existing response fields; multi-URL JSON has a `receipt` on each result item. Raw text output stays unchanged. Available receipt fields are:

- `creditsUsed`: actual credits reported by the response, including zero; absent means unknown, not free. A tool's catalog price is not a charge receipt.
- `requestId`: client idempotency ID for Alexandria calls.
- `operationId` and `operationType` (`search` or `scrape`): the returned server operation ID and its kind, for tracing the operation.

These identifiers serve different purposes. `--request-id` controls Alexandria retry identity; it is not supported for ordinary URL scrape. Diagnostics print available IDs, credits, and retry timing to stderr. Keep stderr separate from JSON stdout; `2>&1` combines them and is not parseable JSON.

With `--json` or `-o`, failed search/scrape calls write structured error output before exiting nonzero. A single scrape failure writes JSON even when the requested filename ends in `.md`. Multi-URL explicit output preserves successful results and failures in input order. Inspect the exit code and error fields before treating a saved file as usable content. Missing receipt fields do not establish whether a timed-out operation was billed.

On a rate limit, wait at least the returned retry delay when available; otherwise use bounded exponential backoff. API keys on the same team share limits, which vary by plan and endpoint. See the Alexandria retry guidance below for provider executions.

For URL scrape, `--timeout <milliseconds>` sets the server-side scrape timeout (the SDK allows transport overhead) and `--max-age 0` requests fresh content. Freshness does not guarantee the source returns a successful page, and a timeout is not proof that upstream work stopped. Inspect returned status/error metadata as well as content.

### Format Behavior

- **Single format**: Outputs raw content (markdown text, HTML, etc.)
Expand Down Expand Up @@ -1039,3 +1059,9 @@ https://www.firecrawl.dev/app/settings?tab=data-sources. Never infer consent fro
failed lookup or automatically retry an acceptance. The API remains authoritative
for organization access and acceptance authority.
After confirmed success, rerun the original provider command; its normal credits apply.

### Alexandria receipts and retries

Alexandria execution JSON includes an additive `receipt`: `creditsUsed` is actual reported usage (missing means unknown), `requestId` is the client idempotency identity, and `operationId`/`operationType` identify the server scrape. Existing response fields remain available. IDs, reported credits, and available retry delays print to stderr.

Use the same request ID to recover pending or uncertain execution. Completed results, including failures, replay under the same ID; a deliberate new execution needs a new ID and may charge again. Never automatically rotate an uncertain ID. Structured failures preserve available status, code, action and retry metadata.
15 changes: 13 additions & 2 deletions skills/firecrawl-scrape/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ firecrawl scrape "<url>" --only-main-content -o .firecrawl/page.md
# Wait for JS to render, then scrape
firecrawl scrape "<url>" --wait-for 3000 -o .firecrawl/page.md

# Multiple URLs (markdown only; each saved to .firecrawl/; -o is ignored)
# Multiple URLs (successful results saved to .firecrawl/; failures reported)
firecrawl scrape https://example.com https://example.com/blog https://example.com/docs

# Get markdown and links together
Expand All @@ -47,11 +47,22 @@ firecrawl scrape "https://example.com/report.pdf" --max-pages 5 --json -o .firec

The cap applies to each PDF, not the whole command or total credits. Extra formats and options can add charges. The CLI does not quote page counts or costs before execution. Use JSON output to inspect the returned `metadata.numPages` (parsed), `metadata.totalPages` (document total), and `metadata.creditsUsed` when present; a smaller parsed count means the result is partial.

## Receipts and recovery

Use `--json` to preserve metadata and the additive `receipt` (at the root for a single scrape, on each result item for multiple URLs). `receipt.creditsUsed` is actual returned usage, including zero; missing means unknown. Existing `metadata.creditsUsed` remains available when returned. `receipt.operationId` identifies the server scrape; Alexandria `receipt.requestId` is a separate client idempotency ID. Available IDs, credit usage, and retry timing print to stderr. Keep stderr separate from JSON stdout.

Failures with `--json` or `-o` write structured errors before exiting nonzero, even if the filename ends in `.md`. Inspect the exit code and saved error/status fields before using the content. A successful transport response can still contain a refused or unsuccessful page; do not treat it as task completion or infer a refund.

- Use `--timeout <milliseconds>` to set the server-side scrape timeout; the SDK allows transport overhead. A timeout does not prove the operation stopped or cost zero credits.
- Use `--max-age 0` when fresh URL content is required. This does not guarantee the source page succeeds.
- On rate limits, honor the returned retry delay when available, otherwise use bounded exponential backoff. Limits are shared across a team's keys and depend on plan and endpoint.
- For Alexandria, keep the same `--request-id` while an operation is unresolved. A completed failure can replay under the same ID; starting a new attempt requires a new ID and may charge again. Never rotate IDs automatically. Ordinary URL scrape does not support `--request-id`.

## Tips

- **Prefer plain scrape over `--query`.** Scrape to a file, then use `grep`, `head`, or read the markdown directly — you can search and reason over the full content yourself. Use `--query` only when you want a single targeted answer without saving the page (costs 5 extra credits).
- **Scrape handles static pages and JS-rendered SPAs.** Escalate to `interact` when the page needs interaction (clicks, form fills, pagination) or scrape misses content.
- Multiple URLs are scraped concurrently — check `firecrawl --status` for your concurrency limit. This mode saves markdown only and ignores `-o`; other requested formats are dropped. If markdown wasn't requested, the whole JSON response is written into the `.md` file.
- Multiple URLs are scraped concurrently. Use `--json` for an ordered JSON array on stdout or `-o results.json` to save it. Each item contains `url`, `success`, and full `data` with metadata or an `error`; any failed URL makes the command exit nonzero. Without either flag, successful results are saved under `.firecrawl/` as markdown when available, otherwise JSON in a `.md` file. Failed URLs are reported on stderr without creating per-URL files. Check `firecrawl --status` for your concurrency limit.
- Single format outputs raw content. Multiple formats (e.g., `--format markdown,links`) output JSON.
- Always quote URLs — shell interprets `?` and `&` as special characters.
- Naming convention: `.firecrawl/{site}-{path}.md`
Expand Down
14 changes: 11 additions & 3 deletions skills/firecrawl-search/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Run `firecrawl search --help` for the full option list.

`--categories developer` weighs the developer index beside ordinary web results in this same call (no passage control, no index filters). `--categories research` is a website filter, not the paper index. Dedicated skills: [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md) and [firecrawl-research-index](../firecrawl-research-index/SKILL.md).

**Done when:** results are saved under `.firecrawl/`, verified non-empty, processed for the request, and one feedback event is sent within the time window (unless opted out).
**Done when:** the response is saved under `.firecrawl/`, results or an empty result set have been inspected and handled for the request, and eligible feedback is sent within the time window (unless opted out).

## Alexandria in normal search

Expand All @@ -40,6 +40,14 @@ Use `find-tools` only for an explicitly requested tool set or a missing contract

If no returned tool covers the country/market/segment or required inputs, continue with ordinary web results. Do not exhaust the catalogue or pay for adjacent tools just to probe coverage. `--sources web` explicitly opts out of Alexandria; `--sources web --domain-tools` retains domain matches only.

## Receipts and failures

JSON output preserves the response and additive `receipt`, including empty results. Read `receipt.creditsUsed` for actual reported usage (zero is valid; missing means unknown) and `receipt.operationId` with `operationType: "search"` for the server search ID. Existing `id` and `creditsUsed` fields remain available. Search IDs identify results; they are not Alexandria client idempotency IDs.

Available IDs, credits, and retry timing print to stderr. Keep stderr separate from JSON stdout. Failed calls with `--json` or `-o` write structured errors before exiting nonzero; check the exit code and `success` before using the file. Empty successful searches still write the requested output and retain their metadata.

On rate limits, wait at least the returned retry delay when available; otherwise use bounded exponential backoff. API keys on one team share limits, which vary by plan and endpoint. Do not invent a universal requests-per-minute quota or infer zero billing from a missing receipt.

## Tips

- **`--highlights` on by default:** results are query-relevant excerpts, not full-page snippets. Use `--no-highlights` for the original snippets.
Expand All @@ -66,13 +74,13 @@ Search costs 2 credits. After you've actually used the results (or decided they
- **Idempotent:** re-submitting for the same search id returns success but no extra refund.
- **`--silent &`** is the right pattern — exit code 0 even on failure, so a rejected/expired call never crashes your pipeline.

Verify the search returned results before reading its `id`. Zero-result searches write no output file, so the file may be missing — or left over from an earlier search. The guard below skips feedback when the file is missing or has zero results; call `search-feedback` only inside it:
Verify the search succeeded before reading its `id`. Empty successful searches preserve their JSON output; failed searches contain error output. The guard below only sends feedback for a successful response with an ID and nonempty results; call `search-feedback` only inside it:

```bash
# Send once per search. Rate honestly and replace the placeholder with the
# rating that matches what actually happened. The two fields shown
# satisfy the substantive-content rule for every rating.
if SEARCH_ID=$(jq -er 'select(any(.data[]; length > 0)) | .id' .firecrawl/search-react-hooks.json); then
if SEARCH_ID=$(jq -er 'select(.success == true and any(.data[]; length > 0)) | .id' .firecrawl/search-react-hooks.json); then
firecrawl search-feedback "$SEARCH_ID" \
--rating "<good|partial|bad>" \
--valuable-sources '[{"url":"https://react.dev/reference/react/hooks","reason":"Most authoritative"}]' \
Expand Down
Loading
Loading