Optional LLM semantic judge for ambiguous tool comparisons

[cursor]

Summary

Optional LLM semantic judge for hard cases (“are these tools the same user task?”) that never overrides deterministic findings and is always labeled MODEL-BASED.

Milestone

Milestone 9 — Semantic Intelligence

Suggested labels

enhancement, priority:p2, area:probes, research

Acceptance criteria

• Opt-in flag/config only; default eval path unchanged
• Uses existing runner.py interface (Design a provider-neutral model runner #45/Expand provider adapters and packaging for the model runner #60)
• Output separated as MODEL-BASED in reports
• Cannot clear a deterministic breaking change
• Fake-runner unit tests; no live calls in CI
• Docs: when to use vs deterministic/embedding layers

Related

• Expand provider adapters and packaging for the model runner #60 provider adapters; Harden model-backed probes against untrusted tool metadata in prompts #66 secret-safe prompts