Fix the three protocol numbers the pilot proved wrong - #456
Merged
Conversation
CDEB-P ran sixteen times and returned three parameters that could not have been found by reading the protocol again. v1.3 changes exactly those, and nothing about the design, the matrix or the twenty locked decisions. The token threshold was 15%, chosen before anyone measured what the ON arm costs. It costs 45% more, so 15% required a 1.71x relative lift -- at a 50% control rate the ON arm would have had to reach 85.5% to clear a gate whose sibling asks for ten points. Injecting context costs tokens by construction, so the threshold has to know the overhead, and it cannot learn it from the outcomes without becoming a number chosen after the fact. It now derives from a calibration measured on the disposable smoke tasks, which never enter the corpus and therefore expose no outcome, and both the calibration and the derived threshold are frozen before the study runs. Token efficiency also leaves the full-headline conjunction. Three ANDed gates sounded like three measurements and behaved like one: the hardest could veto the other two, so a real behaviour improvement would have reported FAIL. It is still evaluated, still reported, still available as a partial claim, and its threshold is now stricter than the number it replaced. "Completable in one fresh agent session" was a wish with no check, and one task in four burned a quarter of the study hitting the wall in all four runs. A task now passes a wall-clock probe at 60% of budget before sealing, and the probe may read only wall time and stop reason -- never the oracle -- which is the same distinction that let this session read stop_reason mid-study without reading a result. Qualification required the record to exist, not to be reachable. Two tasks in four delivered nothing to the ON arm, which made those runs ON by assignment and OFF in substance and left one task carrying the whole question. Sealing now requires that a path the good control edits actually renders the expected record at the frozen snapshot -- a check that runs no agent. Limit: the calibration is measured on disposable tasks, so it estimates overhead on work that is not the corpus and a corpus with different injection density will sit off it Ruled-out: raising the 15% to a number that looked achievable | picking a threshold to be clearable is the same act as picking one after seeing results, and only the derivation makes it preregistered Ruled-out: keeping token efficiency in the conjunction with the new threshold | the objection was never the number, it was that one gate could erase a measured behaviour result while the report called it a three-metric study Ruled-out: widening the hook matcher so the two silent tasks would deliver | that redefines a delivery failure as a delivery success, and §9.5 exists to record exactly that failure Warn: the 0.6 probe budget and the 1.15 calibration factor are judgement calls with no measurement behind them, unlike the three findings that forced this revision Blast: local Undo: easy Certainty: firm Verified: schemas parse and now pin 1.3.x so a protocol change is a schema change CI notices, the token threshold is a range with a required calibration rather than a constant, and the eleven verifier cases pass against a 1.3.0 fixture Provenance: authored Record-Id: r-cdebv13
CommitLore — record lintTrailers: clean — 5 commits in Active constraints for the paths this PR touchesLimits (8)
Ruled out (18)
Warnings (8)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
An external production-readiness review rejected the first v1.3 draft, and it was right on every finding. This replaces that draft. The formula was wrong, and wrong in the direction that matters. TVPDSS reduction is `1 - o/q` for overhead o and relative lift q, so the overhead is already inside the metric. Deriving the threshold from o again counts it twice, and the algebra comes out as `q >= 1.15*o^2` -- at the pilot's measured o = 1.45 that is a 2.42x lift, which at a 50% control rate asks the ON arm for 120.9%. I replaced an unreachable gate with an impossible one. The stated 5% floor could never be selected either: the schema requires o >= 1.0 and the formula returns 13.04% there. The 15% threshold is restored, and the diagnosis that produced the draft is withdrawn with it. The gate was never unreachable -- it was strict because injecting context costs 45% more tokens, and asking for 15% fewer per usable result on top of that is a demanding bar rather than a broken one. Calibration stays as a feasibility note that sets no threshold. Removing token from the headline conjunction while leaving §17.1's sentence alone created a path where a failed token gate still produced "used Y% less token volume". Three gates now exist with fixed wording each, and only the combined one generates the three-metric sentence. The draft's claim that this loosened nothing was false and is recorded as false: `P and M` is strictly weaker than `P and T and M` whatever T's threshold becomes. The runtime probe named no arm, which lets one arm's runtime select the corpus; it discarded the row the freeze manifest was supposed to contain; and it called a returned process a completed task, which a no-op satisfies. It is now an ON and OFF pair, artifacts preserved, renamed to runtime-boundedness. Deliverability was checked with `commitlore context`, which is not the surface CDEB measures -- budget, trust grading, matcher, index and output parsing all sit between it and the agent, and zero shipping delivery was the defect being fixed. It now drives the frozen shipping inject path and passes only when the record appears in the forwarded payload. And the draft asserted the pilot showed task qualification caused the two silent tasks. The pilot's own result says its instrumentation could not tell that from a hook that never fired. The causal claim is withdrawn; v1.3 closes both possibilities separately instead. Limit: schemas and the verifier now require the qualification evidence, but nothing yet produces it -- CDEB-10 must build the probe and the deliverability check before a freeze can be written Ruled-out: keeping the derived threshold with the exponent corrected | any threshold that moves with measured overhead is a threshold the study can influence, and the fixed bar is the only one that means the same thing before and after the run Ruled-out: lowering 15% to something the pilot's overhead clears | that measures a different claim and calls it the same one; if 45% overhead makes the bar hard, the bar is reporting a true fact about the product Warn: the 0.6 probe budget is still a judgement call with no measurement behind it, and it now gates corpus membership through two agent sessions per task Blast: local Undo: easy Certainty: firm Verified: the double-count reproduces as q >= 1.15*o^2 giving 120.9% at o=1.45 against 85.3% for the fixed bar, and 18 verifier cases pass including a study with no freeze manifest, a hand-set threshold, an unqualified corpus, a build that disagrees with its freeze, delivery with zero executions, executions exceeding opportunities, and a hook that fired on a path with no records Provenance: authored Record-Id: r-v13fix
…uessing it Two of the review's remaining limits close here. The third does not, and says so. The delivery qualification exists now rather than being a paragraph. It builds a real PreToolUse payload and runs the pinned shipping injector -- same command, same budget, same trust configuration, same snapshot as the ON arm -- and passes only when the expected record id appears in the bytes the hook forwarded. It renders nothing itself. One test carries the whole argument for why the first version was wrong. A record sits in the repository, on the edited path, and the shipping budget is squeezed to one token: `commitlore context` would still show it, and this check fails. That is the entire difference between the surface that was checked and the surface CDEB measures, made into an assertion. The pilot's defect was zero shipping delivery, and only one of those two surfaces can see it. The 0.6 probe threshold was a judgement call, and the pilot's own wall times settle it. Completed runs topped out at 0.48 of budget and the failing task sat at 1.00, so the good and bad tasks separate anywhere in between and 0.6 touches neither end. The same data limits what the gate may claim: two repeats of one cell ran 89s and 431s, a 4.9x spread, so two probes cannot bound the tail. The gate screens the median. Study timeouts stay ordinary measured failures under intention-to-treat, and what it prevents is the thing that actually happened -- a task whose four runs all timed out and contributed nothing. Limit: the runtime probe still has no implementation, so a freeze cannot yet be written even though the schema now demands its evidence Ruled-out: qualifying delivery through `commitlore context` | budget, trust grading, the guard, lifecycle projection, the matcher and output parsing all sit between it and the agent, and the defect being fixed was zero delivery through exactly those Ruled-out: requiring every good-control path to carry every record | a record scoped to one file of a multi-file change still reaches an agent that opens that file, and demanding all of them rejects sound tasks Ruled-out: throwing on a non-zero injector exit | the hook is fail-open by design, so the exit code belongs in the freeze manifest where a reader can weigh it Warn: delivery qualification proves the record can arrive, not that it will -- an agent that edits another path first or uses a tool outside the matcher still gets nothing, and §9.5 is what records that Blast: local Undo: easy Certainty: firm Verified: six cases -- a path carrying its record qualifies, a path carrying none does not, a squeezed budget fails what context would pass, a missing second record fails the task, any one of several paths suffices, and the payload digest and exit code are recorded rather than thrown Provenance: authored Record-Id: r-delivchk
…lot's numbers The schema demanded runtime qualification evidence that nothing produced, which was the correct failure direction and still a gap. It is closed. The decision function is pure -- probes in, verdict out -- so the gate can be tested without spending two agent sessions per assertion, and `runProbe` is the only part that spends them. That split is what makes this suite runnable in CI at all. Every case is a pair the pilot actually produced. 80s and 89s qualifies. The task that sat at 902s and 903s in all four runs does not. 431s -- the slowest run that ever completed, at 0.48 of budget -- qualifies, which is the observed separation the 0.6 threshold sits inside without touching either end. Three refusals encode review findings rather than taste. A single arm is refused because runtime is treatment-sensitive and qualifying on one arm selects a corpus that arm finishes faster, with the bias inseparable afterwards. Two probes of the same arm are refused for the same reason. The verdict reads the slower arm rather than the mean, because the study runs each task six times and averaging lets a fast arm carry a slow one into the corpus. The last test asserts what the probe type does not contain. A selector that could see functional_pass or rejected_decision_revived would be selecting on the dependent variable, so the absence is pinned rather than assumed. Limit: the gate screens runtime and says so -- `completed` means the process returned, which a no-op satisfies, and the pilot's 4.9x within-cell spread means two probes cannot bound the tail Ruled-out: averaging the two arms | the study runs six sessions per task, so the slower arm is the one that decides whether they fit, and a mean lets the faster arm carry a task in Ruled-out: keeping the probe artifact out of storage | the freeze manifest is required to carry per-task probe results, and a qualification nobody can recheck is not a gate Warn: `runProbe` costs two agent sessions per task, so qualifying a 30-task corpus is 60 sessions before a single measured run Blast: local Undo: easy Certainty: firm Verified: eight cases against real pilot pairs -- 80/89s qualifies, 902/903s timeout does not, 431s at 0.48 of budget qualifies, a 700s slower arm fails on the slower arm, single-arm and same-arm probe sets are refused, an errored arm is refused however fast, and the probe type carries none of the three outcome fields Provenance: authored Record-Id: r-runprobe
The runtime gate decides corpus membership by wall time, so a probe on a different model than the study screens a distribution the study will never produce. §2.2 already forces a new study id when the observed model changes; this is the same rule reaching the qualification that selects the corpus, and `qualifyRuntime` now refuses a probe whose model is not the pinned one. The model is sonnet. The 0.6 threshold this gate applies was derived from the pilot's wall times -- completed runs topping out at 0.48 of budget against a failing task at 1.00 -- and every one of those runs was sonnet. Switching models does not adjust that derivation, it deletes it, and the threshold goes back to being a number somebody picked. M1 and M5 also measured sonnet, so this is the choice that leaves CDEB comparable with the evidence already published. The refusal is worth having whichever model is chosen later, because the failure it prevents is silent: a qualification run on a stronger model finishes faster, admits tasks the study cannot finish, and the resulting timeouts look like task difficulty rather than a mismatched screen. Limit: pinning to sonnet means CDEB will say nothing about whether a stronger model needs delivered decisions less, which is a question worth its own study rather than a variable inside this one Ruled-out: qualifying on opus while measuring on sonnet | the gate would screen runtimes the study never produces, and its threshold is derived from sonnet data Ruled-out: leaving the probe model unchecked and documenting the requirement | the mismatch produces timeouts that read as task difficulty, so a note cannot catch what a comparison can Warn: a later model change invalidates the 0.6 derivation as well as the pinning, so both must be re-established rather than only the constant edited Blast: local Undo: easy Certainty: firm Verified: nine cases pass, including a pair where one arm probed opus against a sonnet-pinned study and the qualification refuses by naming both Provenance: authored Record-Id: r-probemodel
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CDEB PRD v1.2 → v1.3. Changes exactly what CDEB-P measured — nothing about the design, the matrix, or the twenty locked decisions.
1. The token threshold knew nothing about the token cost
15% was chosen before anyone measured what the ON arm costs. It costs 45% more.
Injecting context costs tokens by construction, so the threshold has to know the overhead — and it cannot learn it from the outcomes without becoming a number chosen after the fact.
It now derives from a calibration measured on the §22.4 disposable smoke tasks, which never enter the corpus and therefore expose no outcome:
Both go in the freeze manifest before the study runs. No calibration → the gate is NOT MEASURABLE, which beats setting a threshold after seeing results.
2. Three ANDed gates behaved like one
§16.6 required all three. The hardest could veto the other two — a real behaviour improvement would have reported FAIL. That is not a three-metric design; it is a one-metric design that calls itself three.
Token efficiency is still evaluated, still reported, still available as a partial claim under §17.2. It is no longer a necessary condition for the headline. Its threshold is now stricter than the number it replaced — this loosens nothing.
3. "Completable in one session" was a wish with no check
One task in four burned a quarter of the study hitting the 15-minute wall in all four runs.
§4.6 now requires a wall-clock probe at 60% of budget before sealing. The probe may read only wall time and stop reason — never the oracle — the same distinction that let this session read
stop_reasonmid-study without reading a result. A task that fails is cut or shrunk; the budget is not raised for it, because the budget is a contract with the other 29 tasks.4. Qualification required the record to exist, not to be reachable
Two tasks in four delivered nothing to the ON arm — ON by assignment, OFF in substance — leaving one task carrying the whole question.
New §4.9: sealing requires that a path the good control edits actually renders the expected record at the frozen snapshot. The good control's file set is the verifiable definition of "what the natural solution touches", rather than the author's expectation. Runs no agent, exposes no outcome.
Also
§9.5 now requires
hook_opportunitiesanddelivered_record_idsto be computed separately, with a §25.3 test — the pilot's counter merged them, so a zero could not distinguish "never fired" from "fired on a path with no records". Schemas pin1.3.x, so a protocol change is a schema change CI notices.Stated limits
The calibration estimates overhead on work that is not the corpus. And the 0.6 probe budget and the 1.15 factor are judgement calls with no measurement behind them — unlike the three findings that forced this revision. Both are in the commit's
Limit:andWarn:.