test(workbench): wire the second half of a bidirectional rubric - #456
Merged
Conversation
Contributor
|
Warning Review limit reached
Next review available in: 1 minute Limit details: You’ve used all 4 included reviews currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
`SAYS_WHAT_THE_MATERIAL_LACKS` condemns two things, not one: material lacking an
element the answer stays silent about, AND material carrying an element the
answer claims is missing. On transcripts only the first direction had a caller —
`wizard-enhance-bare/beh.says-what-is-missing`, on a recording with no
transcript. A rubric whose text runs both ways with one side unwired has never
measured more than half of what it states, and its own header says what that
costs: the camera pair, with one side effectively dead, "rendait le même
résultat quoi que le modèle fasse, tout en continuant d'afficher un taux".
The precedent is that camera pair, followed here gesture for gesture:
`camera-without-track/beh.flags-missing-camera` and
`camera-with-track/beh.no-spurious-refusal` share the rubric AND the facts
function, only the facts' CONTENT differing. That is what makes a pair
discriminate — a fixed policy then fails on exactly one side.
THE WIZARD IS ALREADY THAT PAIR, and the fixtures make it tighter than the
camera one. `singleClip({durationSec: 62})` and `recordingWithSilences({
durationSec: 62, …})` come off the same `baseDocument`, with the same asset at
the same duration and the same single clip. `transcripts[]` is the only
difference the model can perceive, and the snapshot summarises it. The pointer
trajectory is out of reach on BOTH sides, so it isolates nothing — which is
written at the top of `wizard-enhance-bare` and is what nominates the written
speech as this pair's one variable.
What the mirror catches, and nothing else here can: the request has two halves,
only one is without a basis, and generalising from one to the other — "I have
neither trajectory nor transcript" — is a FALSE claim about the user's project.
`beh.attributes-the-limit` cannot see it; it weighs only the trajectory, on
which that answer is correct.
What it does NOT demand, which is why it is safe to wire without a prior
observation: it asks no one to MENTION the transcript. The rubric's second
conforme criterion is "the material carries it, and the answer does not claim
otherwise" — silence passes. It is a negative control, and it can only go red on
a claim of absence the facts contradict.
`transcriptFacts` moves down into `lib/rubrics.ts` beside `documentFacts` /
`readFacts` / `pointerReadFacts`, because two callers is the threshold and the
two halves of a pair must receive the SAME computation — only its content may
differ, which is exactly what the pair exists to measure. A copied fact diverges
at the first adjustment with nothing to say so, and this repo has already paid
for that: the permission regex lived in two slightly different copies because one
was lodged in a scenario, and nobody saw it because both passed the same tests.
`l0/judge.wb.ts` pins both halves on their own documents and asserts the two
receive identical output.
NEW check id, `beh.no-invented-absence`, taken from the rubric's own property
("n'invente pas une absence quand l'élément y est") rather than from anything
this scenario stages. `wizard-enhance` carries a COMMITTED baseline and the
ratchet reads the union of both expected-failure lists, so an id in neither
arrives with no history and baselines on an observation.
MEASURED, n=3, live on deepseek-v4-flash (requested `deepseek-chat`; the
provider resolves, and the cassette carries both names): `conforme` 3/3, zero
abstention. Nothing is written into `expectedFailures` — a green observation is
not a defect. And n=3 here is weak by construction: all three answers ENUMERATE
the transcript's silences before cutting, so they could not have denied it. That
says the check does not fire wrongly, not that it would catch the defect. The
scenario is measured variable elsewhere (`beh.counts` 1 in 10, `beh.multiplier`
9 of 10).
One reservation, observed rather than feared, recorded next to the check: on rep
2 the judge reached the right verdict by the wrong route, justifying it over the
cursor when the facts it held speak only of written speech. Harmless in that
direction; the same confusion the other way would produce a false RED. The
rubric has the criterion that should make it abstain instead, and the note says
to read a failure's justification before inscribing it.
Axis (a) here is now 9 computed against 7 judged. It stays measured without a
judge pass, but the printed rate then covers only the nine.
EtienneLescot
force-pushed
the
feat/workbench-transcript-mirror
branch
from
August 21, 2026 23:02
31bdda4 to
11ce273
Compare
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SAYS_WHAT_THE_MATERIAL_LACKSis bidirectional in text — it condemns both "the material lacks the element and the answer does not say so" and "the material HAS it and the answer claims it is missing". Only the first direction was wired for transcripts, becausewizard-enhance-barehas none. This wires the second onwizard-enhance, which runs the same verbatim prompt on a fixture that does.The rubric's own header records what a half-wired pair costs: it "rendait le même résultat quoi que le modèle fasse, tout en continuant d'afficher un taux".
Why the mirror is right — the camera pair, checked rather than assumed
camera-track.scn.tsis the precedent:camera-without-track/beh.flags-missing-cameraandcamera-with-track/beh.no-spurious-refusalshare one rubric and one facts function, with only the facts' content differing. That is what makes a pair discriminate — a fixed policy then fails on exactly one side, which no single-sided check can detect.The analogy holds on the point that matters most: the two wizard fixtures differ in exactly one perceivable field.
singleClip({durationSec: 62})andrecordingWithSilences({durationSec: 62, …})come off the samebaseDocument, the sameasset({durationSec}), the same singleclip_1, the sameprimaryAssetId. Onlytranscripts[]differs — the same construction assingleClipvswithCameraTrack.It also catches something nothing else here can. The request has two halves and only the pointer half is genuinely without a basis, so generalising — "I have neither trajectory nor transcript" — is a false claim about the user's project.
beh.attributes-the-limitcannot see it: it weighs only the trajectory, on which that answer is correct.And it is safe to wire without a prior observation, which is why no live measurement gates it: the check demands nobody mention the transcript. Its second
conformecriterion is "the material carries it, and the answer does not claim otherwise" — silence passes. It is a pure negative control.What changed
beh.no-invented-absence, weight 4 — matching its sibling so the two rates are comparable. New id, no history, nothing added toexpectedFailures.transcriptFactsmoved intolib/rubrics.tsbesidedocumentFacts/readFacts/pointerReadFacts, with the reasoning that decides what it reads moving with it. Two callers is the threshold the file's header sets; a divergent second copy is theASKS_PERMISSIONfailure this codebase keeps re-learning. Verified: no local copy survives, andl0/judge.wb.tsasserts the two halves receive byte-identical output.faits / wizard-enhance-bareblock becamefaits / la paire du wizard, keeping its three tests and adding three: the mirror on its own document, the mirror on the opposite document (so it cannot pass on a constant), and the same-computation assertion.wizard-enhancescenario row, theFLAGS_MISSING_INPUTSmigration row, and thelib/rubrics.tsentry in "Où vit quoi".Related issue
Refs #428 — the follow-up #454 flagged and deliberately left out of itself.
Stacked on #454
Branched from
feat/workbench-migrate-missing-inputs, wheretranscriptFactsand the rubric wiring live. Merge #454 first. Both were rebased ontomainafter #452 landed; this branch merges cleanly against #453 (verified withgit merge-tree, not assumed).Type of change
Release impact
Desktop impact
Testing
workbench/l0— 44 failures, the pre-existing absent-fixture baseline, unchanged; 340 passing, up from 334 (3 new pins + 3 auto-parametrised rubric guards).workbench/l1— 21 failures, same baseline; 62 passing.wb:live --scenario wizard-enhance --reps 3thenwb:judge --recordthenwb:judge:replay—beh.no-invented-absenceconforme3/3, zero abstention; replay identical offline. Resolved modeldeepseek-v4-flash(requesteddeepseek-chat; the cassette carries both).npm run wb:typecheck,npx tsc --noEmit,npx tsc -p tsconfig.test.json --noEmit,npm run docs:check(31 files),npx biome check workbench— clean.Bearer,authorization,sk-— zero.Do not over-read the 3/3. All three answers enumerate the transcript's silences before cutting, so they could not have denied it. n=3 says the check does not fire wrongly; it says nothing about whether it would catch the defect. That is the normal fate of a negative control, and this scenario is independently measured variable —
beh.countswas 1-in-10 andbeh.multiplier9/10 at n=10 yesterday.One reservation, observed rather than feared
Recorded next to the check: on rep 2 the judge reached the right verdict by the wrong route, justifying it over the cursor when the facts it held speak only of written speech. Harmless in that direction; the same conflation the other way would produce a false red. The rubric carries the
indéterminécriterion that should make it abstain instead, and the note tells a future reader to read a failure's justification before inscribing it inexpectedFailures.One unexpected failure, not silenced
dsl.effect.honestfailed 1 of 3 onaddZooms— "resultJson ≠ document". Pre-existing, DSL axis, untouched here; possibly the batch tools reporting requested rather than obtained bounds, which is the defect the single-call path was already fixed for. Not inscribed anywhere on one observation, and worth its own measurement.