ai(observability): add a code-based (Hog) evaluation for review output quality

[JSONbored]

Part of epic #8286 (Phase 3 — AI observability). Follow-on to #10218 (content capture) and #10226 ($ai_metric).

Why Hog, and why not the other two

PostHog offers three evaluation types. Only one fits this deployment:

Type	Cost	Verdict
Code-based (Hog)	Free — no LLM, runs in milliseconds	Use this
Sentiment	Free — local ML model on PostHog's side	Skip — no value here
LLM-as-a-judge	One LLM API call per evaluation, billed to a pinned cloud key	Skip — see below

• LLM-judge is structurally wrong for this stack. Its provider enum is openai | anthropic | gemini | openrouter | fireworks | azure_openai | together_ai | minimax | zeabur — there is no ollama or self-hosted option. Enabling it means a new cloud bill on a deployment that is deliberately 100% self-hosted, and 7,434 of the last 13,997 AI events are free local ollama embeddings, so a paid cloud judge would mostly be grading calls that cost nothing.
• Sentiment classifies user-message sentiment. The "user messages" here are PR diffs and review prompts. There is no human emotion in a diff to measure.

Hog is free, deterministic, needs no model, and checks the things that actually matter for a review engine.

Event-budget constraint (this is the real cost)

Evaluations are free to run, but each one emits an $ai_evaluation event, and PostHog's LLM-analytics free tier is 100K events/month. Current run rate is ~2,300 AI events/day (~70K/month). Evaluating all 6,563 generations at 100% rollout would add ~33K/month and push the project over the free tier.

Mitigations, both first-class in the API:

• conditions[].rollout_percentage (0–100) samples matching events.
• target: "trace" evaluates once per trace instead of once per generation. With trace grouping now live (~7 generations/trace), this is roughly a 7× reduction on its own.

Start sampled and trace-targeted; widen only if the signal proves useful.

What Hog can read — verified, not assumed

Probed live with llma-evaluation-test-hog (3 recent generations, target: generation):

source: return not properties.$ai_is_error
→ 3/3 samples returned true, no errors

So generation properties ($ai_is_error, $ai_model, $ai_provider, $ai_latency, $ai_total_cost_usd, repo, pullNumber) are readable today.

The same probe returned empty input_preview / output_preview, confirming content is not yet flowing — #10218 is merged but has not reached the ORB image. Any check that inspects the prompt or completion is blocked until it does.

Candidate checks

Available now (properties only):

• A review generation that succeeded but reported zero cost and zero latency — a shape that suggests a degenerate or short-circuited call rather than a real one.
• A claude-code generation whose latency exceeds the configured effort tier's timeout, which should be impossible and indicates the ladder is misconfigured.

Blocked on content capture reaching the ORB (#10218):

• The review returned parseable JSON matching the reviewer contract (assessment, blockers, nits, suggestions, confidence). A malformed envelope is the single highest-value deterministic check — it is exactly what extractCliText / the JSON-mode fallback exist to survive, and today a malformed response degrades silently.
• The output is non-empty and not truncated — catches claude_code_empty_output and clipped completions as a quality signal rather than only as an error.
• A blocker was asserted with supporting evidence (a file path or line reference present in the blocker text). A blocker with no locus is the shape that produces a bad auto-close.

Pick the JSON-contract check first; it is the one with a real failure mode already documented in the codebase.

Deliverables

• Validate the Hog source with llma-evaluation-test-hog (sample_count up to 10) before creating anything — the tool exists precisely so the code is proven against real samples first.
• Create one hog evaluation via llma-evaluation-create with evaluation_type: "hog", output_type: "boolean", and output_config.allows_na: true so non-review generations (embeddings, vision, advisory) return N/A instead of a fabricated fail.
• Scope it with conditions[].properties so it only runs on review generations, not on the free ollama embedding traffic.
• Set target: "trace" and a starting rollout_percentage well under 100, then re-check the monthly $ai_evaluation volume against the 100K free tier before widening.
• Create it with enabled: false first, confirm the sampled results read correctly in the Traces view, then enable.
• Add the resulting pass rate to the AI observability dashboard (id 1934420) alongside the existing error-rate and spend panels.

Blocked by

The content-inspecting checks need #10218 live on the ORB. The property-only checks can ship immediately.

Note on $ai_evaluation

$ai_evaluation is PostHog's output, not ours — the docs are explicit that evaluations "run on new $ai_generation events; results are $ai_evaluation events". There is no SDK helper because clients are not meant to emit it. This is why #10226 deliberately excluded it and used $ai_metric for LoopOver-native quality signal instead. The two are complementary: $ai_metric carries what the engine already computes (reviewer votes, judgment agreement, gate/rule precision); $ai_evaluation carries what PostHog derives from the generation itself.

[linear-code]

JSO-7139

[JSONbored]

Shipped, with one correction to the plan.

Correction: target: "trace" cannot express any of the checks in this issue

The issue leans on trace targeting for a ~7x event-budget reduction. Probed with llma-evaluation-test-hog, a trace-target run receives exactly one global:

trace = {'id': '18963820f89cd4e08acd30d5efb1cd3b', 'event_count': 7}

properties, input and output all fail with Global variable not found. So at trace grain a Hog evaluation can count events and nothing else — no property-based check is expressible there, and neither are the content checks once content lands.

The budget concern it was meant to solve turns out not to exist anyway. Scoping conditions[].properties to $ai_provider = claude-code already excludes the free local ollama embedding traffic, which is roughly half of all AI events. That leaves ~390 matching generations/day; at 25% sampling this is ~2.9K $ai_evaluation events/month against the 100K free tier, versus the ~33K the issue projected for an unscoped 100% rollout.

What was created

Evaluation "Review call is non-degenerate" (019fb8c8-d5b7-0000-59c7-e7194af07a48), hog / boolean / allows_na: true, target: generation, scoped to $ai_provider = claude-code at rollout_percentage: 25.

-- N/A the failure path: an errored generation legitimately has no cost, and the AI error-rate alert
-- already owns that signal. This measures whether SUCCESSFUL review calls look like real calls.
if (properties.$ai_is_error) { return null }
let cost := toFloat(properties.$ai_total_cost_usd)
let latency := toFloat(properties.$ai_latency)
-- Unreported cost or latency is not a failed review, it is an unmeasurable one.
if (cost = null or latency = null) { return null }
-- A success that billed nothing and took no time is a short-circuited or degenerate call, not a review.
return not (cost = 0 and latency = 0)

The $ai_is_error guard is an addition to the issue's sketch. Without it the check would double-count the failure path that the error-rate alert already owns, and the pass rate would stop meaning "successful reviews that look real".

Validated before creation against 10 live samples (10 pass, 0 errors, 0 N/A once the condition filter was applied), created enabled: false, then exercised end to end with llma-evaluation-run against a real generation. It emitted a genuine result:

$ai_evaluation_result: true, $ai_evaluation_applicable: true, $ai_evaluation_runtime: hog,
$ai_trace_id: 92bb99f38db520087e35c8c25e22f55b

so it joins to the trace. Now enabled.

Dashboard panel "Evaluation pass rate (30d)" added to AI observability (1934420). Non-applicable results are excluded from the rate and counted in their own column, and nullIf guards the all-N/A day so an undefined rate does not render as 0%.

Baseline, for whoever retunes this later

Over 48h of production traffic: 772 claude-code generations, 585 ollama, and zero with the degenerate shape. Successful claude-code calls run p50 7.9s / p95 61.6s / max 160.8s. The 8 failures all sit at exactly 180.0s, which is the timeout, and those return N/A here by design. So a pass rate below 100% is a genuine regression signal, not noise.

Still blocked

The three content-inspecting checks (JSON envelope conformance, non-empty/non-truncated output, blocker-with-evidence) remain blocked. input_preview / output_preview are still empty in every probe: #10218 is merged but content capture has not reached the ORB image, and LOOPOVER_POSTHOG_AI_CONTENT=1 still has to be set there. The JSON-contract check stays the highest-value one to add first once it does.