Parallel probes, result caching, cost tracking, and scale benchmarks

[cursor]

Summary

Performance harness for large catalogs, optional parallel probe workers, model-result caching, and optional cost tracking.

Milestone

Performance / cost (P3)

Suggested labels

enhancement, priority:p3, area:probes, tests

Acceptance criteria

• Benchmark script/docs for snapshot/diff/probe times at 10/100/500/1000/5000 tools
• --workers N (or config) for parallel probe execution with deterministic ordering in reports
• Cache key: model + prompt/tools hash + snapshot hash + temperature + seed
• Optional cost/token summary when provider metadata present
• Unit tests for cache hit/miss; perf numbers not gated flaky in PR CI

Related

• Discovery testing; multi-model matrix