Skip to content

Fix the three protocol numbers the pilot proved wrong - #456

Merged
MongLong0214 merged 5 commits into
devfrom
cdeb-v13
Aug 8, 2026
Merged

Fix the three protocol numbers the pilot proved wrong#456
MongLong0214 merged 5 commits into
devfrom
cdeb-v13

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

CDEB PRD v1.2 → v1.3. Changes exactly what CDEB-P measured — nothing about the design, the matrix, or the twenty locked decisions.

1. The token threshold knew nothing about the token cost

15% was chosen before anyone measured what the ON arm costs. It costs 45% more.

required S_on/S_off for a 15% reduction = 1.45 / 0.85 = 1.71×
→ at a 50% control rate, ON must reach 85.5%   (+35.5pp)
→ its sibling gate asks for +10pp

Injecting context costs tokens by construction, so the threshold has to know the overhead — and it cannot learn it from the outcomes without becoming a number chosen after the fact.

It now derives from a calibration measured on the §22.4 disposable smoke tasks, which never enter the corpus and therefore expose no outcome:

CalibratedOverhead = T_on / T_off   (on smoke tasks)
RequiredReduction  = max(0.05, 1 - 1 / (CalibratedOverhead × 1.15))

Both go in the freeze manifest before the study runs. No calibration → the gate is NOT MEASURABLE, which beats setting a threshold after seeing results.

2. Three ANDed gates behaved like one

§16.6 required all three. The hardest could veto the other two — a real behaviour improvement would have reported FAIL. That is not a three-metric design; it is a one-metric design that calls itself three.

Token efficiency is still evaluated, still reported, still available as a partial claim under §17.2. It is no longer a necessary condition for the headline. Its threshold is now stricter than the number it replaced — this loosens nothing.

3. "Completable in one session" was a wish with no check

One task in four burned a quarter of the study hitting the 15-minute wall in all four runs.

§4.6 now requires a wall-clock probe at 60% of budget before sealing. The probe may read only wall time and stop reason — never the oracle — the same distinction that let this session read stop_reason mid-study without reading a result. A task that fails is cut or shrunk; the budget is not raised for it, because the budget is a contract with the other 29 tasks.

4. Qualification required the record to exist, not to be reachable

Two tasks in four delivered nothing to the ON arm — ON by assignment, OFF in substance — leaving one task carrying the whole question.

New §4.9: sealing requires that a path the good control edits actually renders the expected record at the frozen snapshot. The good control's file set is the verifiable definition of "what the natural solution touches", rather than the author's expectation. Runs no agent, exposes no outcome.

Also

§9.5 now requires hook_opportunities and delivered_record_ids to be computed separately, with a §25.3 test — the pilot's counter merged them, so a zero could not distinguish "never fired" from "fired on a path with no records". Schemas pin 1.3.x, so a protocol change is a schema change CI notices.

Stated limits

The calibration estimates overhead on work that is not the corpus. And the 0.6 probe budget and the 1.15 factor are judgement calls with no measurement behind them — unlike the three findings that forced this revision. Both are in the commit's Limit: and Warn:.

CDEB-P ran sixteen times and returned three parameters that could not have been
found by reading the protocol again. v1.3 changes exactly those, and nothing
about the design, the matrix or the twenty locked decisions.

The token threshold was 15%, chosen before anyone measured what the ON arm
costs. It costs 45% more, so 15% required a 1.71x relative lift -- at a 50%
control rate the ON arm would have had to reach 85.5% to clear a gate whose
sibling asks for ten points. Injecting context costs tokens by construction, so
the threshold has to know the overhead, and it cannot learn it from the outcomes
without becoming a number chosen after the fact. It now derives from a
calibration measured on the disposable smoke tasks, which never enter the
corpus and therefore expose no outcome, and both the calibration and the derived
threshold are frozen before the study runs.

Token efficiency also leaves the full-headline conjunction. Three ANDed gates
sounded like three measurements and behaved like one: the hardest could veto the
other two, so a real behaviour improvement would have reported FAIL. It is still
evaluated, still reported, still available as a partial claim, and its threshold
is now stricter than the number it replaced.

"Completable in one fresh agent session" was a wish with no check, and one task
in four burned a quarter of the study hitting the wall in all four runs. A task
now passes a wall-clock probe at 60% of budget before sealing, and the probe may
read only wall time and stop reason -- never the oracle -- which is the same
distinction that let this session read stop_reason mid-study without reading a
result.

Qualification required the record to exist, not to be reachable. Two tasks in
four delivered nothing to the ON arm, which made those runs ON by assignment and
OFF in substance and left one task carrying the whole question. Sealing now
requires that a path the good control edits actually renders the expected record
at the frozen snapshot -- a check that runs no agent.

Limit: the calibration is measured on disposable tasks, so it estimates overhead on work that is not the corpus and a corpus with different injection density will sit off it
Ruled-out: raising the 15% to a number that looked achievable | picking a threshold to be clearable is the same act as picking one after seeing results, and only the derivation makes it preregistered
Ruled-out: keeping token efficiency in the conjunction with the new threshold | the objection was never the number, it was that one gate could erase a measured behaviour result while the report called it a three-metric study
Ruled-out: widening the hook matcher so the two silent tasks would deliver | that redefines a delivery failure as a delivery success, and §9.5 exists to record exactly that failure
Warn: the 0.6 probe budget and the 1.15 calibration factor are judgement calls with no measurement behind them, unlike the three findings that forced this revision
Blast: local
Undo: easy
Certainty: firm
Verified: schemas parse and now pin 1.3.x so a protocol change is a schema change CI notices, the token threshold is a range with a required calibration rather than a constant, and the eleven verifier cases pass against a 1.3.0 fixture
Provenance: authored
Record-Id: r-cdebv13
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 5 commits in origin/dev..6966cb3d178365570b799b4b3f12ae9999b5ddef
Active constraints: 8 limits · 18 ruled-out · 8 warnings — from 8 records over 10 changed paths

Active constraints for the paths this PR touches

Limits (8)

  • r-probemodel 6966cb3 — pinning to sonnet means CDEB will say nothing about whether a stronger model needs delivered decisions less, which is a question worth its own study rather than a variable inside this one
  • r-runprobe 6a368ec — the gate screens runtime and says so -- completed means the process returned, which a no-op satisfies, and the pilot's 4.9x within-cell spread means two probes cannot bound the tail
  • r-delivchk 59ba928 — the runtime probe still has no implementation, so a freeze cannot yet be written even though the schema now demands its evidence
  • r-v13fix f4b6859 — schemas and the verifier now require the qualification evidence, but nothing yet produces it -- CDEB-10 must build the probe and the deliverability check before a freeze can be written
  • r-cdebv13 e0bd61a — the calibration is measured on disposable tasks, so it estimates overhead on work that is not the corpus and a corpus with different injection density will sit off it
  • r-cdebpres e01b482 — one repository, four tasks, six usable runs per arm and a local evaluator -- no claim gate is evaluable and preregistration §8 forbids any number here reaching a public surface
  • r-cdebver01 ce7b278 — the schemas freeze protocol 1.2.0 constants -- thresholds, matrix size, category names -- so a protocol change is a schema change and CI notices
  • r-cdebprd12 c3fefd2 — CDEB measures the shipping trust configuration as it exists, which grades every record claim; a directive-path result requires a different study under a different product digest

Ruled out (18)

  • r-probemodel 6966cb3 — qualifying on opus while measuring on sonnet | the gate would screen runtimes the study never produces, and its threshold is derived from sonnet data
  • r-probemodel 6966cb3 — leaving the probe model unchecked and documenting the requirement | the mismatch produces timeouts that read as task difficulty, so a note cannot catch what a comparison can
  • r-runprobe 6a368ec — averaging the two arms | the study runs six sessions per task, so the slower arm is the one that decides whether they fit, and a mean lets the faster arm carry a task in
  • r-runprobe 6a368ec — keeping the probe artifact out of storage | the freeze manifest is required to carry per-task probe results, and a qualification nobody can recheck is not a gate
  • r-delivchk 59ba928 — qualifying delivery through commitlore context | budget, trust grading, the guard, lifecycle projection, the matcher and output parsing all sit between it and the agent, and the defect being fixed was zero delivery through exactly those
  • r-delivchk 59ba928 — requiring every good-control path to carry every record | a record scoped to one file of a multi-file change still reaches an agent that opens that file, and demanding all of them rejects sound tasks
  • r-delivchk 59ba928 — throwing on a non-zero injector exit | the hook is fail-open by design, so the exit code belongs in the freeze manifest where a reader can weigh it
  • r-v13fix f4b6859 — keeping the derived threshold with the exponent corrected | any threshold that moves with measured overhead is a threshold the study can influence, and the fixed bar is the only one that means the same thing before and after the run
  • r-v13fix f4b6859 — lowering 15% to something the pilot's overhead clears | that measures a different claim and calls it the same one; if 45% overhead makes the bar hard, the bar is reporting a true fact about the product
  • r-cdebv13 e0bd61a — raising the 15% to a number that looked achievable | picking a threshold to be clearable is the same act as picking one after seeing results, and only the derivation makes it preregistered
  • r-cdebv13 e0bd61a — keeping token efficiency in the conjunction with the new threshold | the objection was never the number, it was that one gate could erase a measured behaviour result while the report called it a three-metric study
  • r-cdebv13 e0bd61a — widening the hook matcher so the two silent tasks would deliver | that redefines a delivery failure as a delivery success, and §9.5 exists to record exactly that failure
  • r-cdebpres e01b482 — reporting the completed-only table as the pilot's result without the timed-out task | the exclusion is the registered stop_reason rule, and a reader who cannot see that a quarter of the study never finished cannot judge the rest
  • r-cdebpres e01b482 — dropping the two zero-delivery tasks as broken | they are the intention-to-treat case the protocol is built around, and they are also how the qualification gap was found
  • r-cdebver01 ce7b278 — default-in discovery over the CDEB tree like the legacy gate | that gate's enemy was the unregistered file, CDEB's is the unregistered input, and bench:m5 reads every jsonl in bench/results, so the registered analysis would run over M1, M4 and files marked non-citable #441 showed a glob passing a stopping rule on contamination
  • r-cdebver01 ce7b278 — validating with the draft-07 Ajv entry and downgrading the schemas | the 2020-12 entry ships in the same package, and weakening six schemas to fit a meta-schema is backwards
  • r-cdebprd12 c3fefd2 — reopening any of the twenty locked decisions in §29 | v1.1 locked them deliberately and every v1.2 change is additive definition, disclosure, or leak-closure
  • r-cdebprd12 c3fefd2 — enabling capture in the ON arm for realism | an agent commit mid-run would stage records unattended and change repository state between repeats, destroying the same-history invariant the whole design rests on

Warnings (8)

  • r-probemodel 6966cb3 (claim) — a later model change invalidates the 0.6 derivation as well as the pinning, so both must be re-established rather than only the constant edited
  • r-runprobe 6a368ec (claim)runProbe costs two agent sessions per task, so qualifying a 30-task corpus is 60 sessions before a single measured run
  • r-delivchk 59ba928 (claim) — delivery qualification proves the record can arrive, not that it will -- an agent that edits another path first or uses a tool outside the matcher still gets nothing, and §9.5 is what records that
  • r-v13fix f4b6859 (claim) — the 0.6 probe budget is still a judgement call with no measurement behind it, and it now gates corpus membership through two agent sessions per task
  • r-cdebv13 e0bd61a (claim) — the 0.6 probe budget and the 1.15 calibration factor are judgement calls with no measurement behind them, unlike the three findings that forced this revision
  • r-cdebpres e01b482 (claim) — the exposure counter counts deliveries rather than opportunities, so a zero cannot distinguish a hook that never fired from one that fired on a path with no records -- CDEB v1 §9.5 requires that split and CDEB-05 must build it
  • r-cdebver01 ce7b278 (claim) — every failure fixture is the one valid row with a single mutation, so keep the valid-row builder in lockstep with result.schema.json -- a builder drift fails all eleven cases at once and reads as a broken verifier
  • r-cdebprd12 c3fefd2 (claim) — the corpus does not exist yet; nothing in this commit authorises a measured run, which stays behind the freeze gate and Definition of Done

git log --follow accepts exactly one pathspec, so renames are not followed for 10 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

An external production-readiness review rejected the first v1.3 draft, and it
was right on every finding. This replaces that draft.

The formula was wrong, and wrong in the direction that matters. TVPDSS reduction
is `1 - o/q` for overhead o and relative lift q, so the overhead is already
inside the metric. Deriving the threshold from o again counts it twice, and the
algebra comes out as `q >= 1.15*o^2` -- at the pilot's measured o = 1.45 that is
a 2.42x lift, which at a 50% control rate asks the ON arm for 120.9%. I replaced
an unreachable gate with an impossible one. The stated 5% floor could never be
selected either: the schema requires o >= 1.0 and the formula returns 13.04%
there.

The 15% threshold is restored, and the diagnosis that produced the draft is
withdrawn with it. The gate was never unreachable -- it was strict because
injecting context costs 45% more tokens, and asking for 15% fewer per usable
result on top of that is a demanding bar rather than a broken one. Calibration
stays as a feasibility note that sets no threshold.

Removing token from the headline conjunction while leaving §17.1's sentence
alone created a path where a failed token gate still produced "used Y% less
token volume". Three gates now exist with fixed wording each, and only the
combined one generates the three-metric sentence. The draft's claim that this
loosened nothing was false and is recorded as false: `P and M` is strictly
weaker than `P and T and M` whatever T's threshold becomes.

The runtime probe named no arm, which lets one arm's runtime select the corpus;
it discarded the row the freeze manifest was supposed to contain; and it called
a returned process a completed task, which a no-op satisfies. It is now an ON
and OFF pair, artifacts preserved, renamed to runtime-boundedness.

Deliverability was checked with `commitlore context`, which is not the surface
CDEB measures -- budget, trust grading, matcher, index and output parsing all
sit between it and the agent, and zero shipping delivery was the defect being
fixed. It now drives the frozen shipping inject path and passes only when the
record appears in the forwarded payload.

And the draft asserted the pilot showed task qualification caused the two silent
tasks. The pilot's own result says its instrumentation could not tell that from
a hook that never fired. The causal claim is withdrawn; v1.3 closes both
possibilities separately instead.

Limit: schemas and the verifier now require the qualification evidence, but nothing yet produces it -- CDEB-10 must build the probe and the deliverability check before a freeze can be written
Ruled-out: keeping the derived threshold with the exponent corrected | any threshold that moves with measured overhead is a threshold the study can influence, and the fixed bar is the only one that means the same thing before and after the run
Ruled-out: lowering 15% to something the pilot's overhead clears | that measures a different claim and calls it the same one; if 45% overhead makes the bar hard, the bar is reporting a true fact about the product
Warn: the 0.6 probe budget is still a judgement call with no measurement behind it, and it now gates corpus membership through two agent sessions per task
Blast: local
Undo: easy
Certainty: firm
Verified: the double-count reproduces as q >= 1.15*o^2 giving 120.9% at o=1.45 against 85.3% for the fixed bar, and 18 verifier cases pass including a study with no freeze manifest, a hand-set threshold, an unqualified corpus, a build that disagrees with its freeze, delivery with zero executions, executions exceeding opportunities, and a hook that fired on a path with no records
Provenance: authored
Record-Id: r-v13fix
…uessing it

Two of the review's remaining limits close here. The third does not, and says so.

The delivery qualification exists now rather than being a paragraph. It builds a
real PreToolUse payload and runs the pinned shipping injector -- same command,
same budget, same trust configuration, same snapshot as the ON arm -- and passes
only when the expected record id appears in the bytes the hook forwarded. It
renders nothing itself.

One test carries the whole argument for why the first version was wrong. A
record sits in the repository, on the edited path, and the shipping budget is
squeezed to one token: `commitlore context` would still show it, and this check
fails. That is the entire difference between the surface that was checked and
the surface CDEB measures, made into an assertion. The pilot's defect was zero
shipping delivery, and only one of those two surfaces can see it.

The 0.6 probe threshold was a judgement call, and the pilot's own wall times
settle it. Completed runs topped out at 0.48 of budget and the failing task sat
at 1.00, so the good and bad tasks separate anywhere in between and 0.6 touches
neither end. The same data limits what the gate may claim: two repeats of one
cell ran 89s and 431s, a 4.9x spread, so two probes cannot bound the tail. The
gate screens the median. Study timeouts stay ordinary measured failures under
intention-to-treat, and what it prevents is the thing that actually happened --
a task whose four runs all timed out and contributed nothing.

Limit: the runtime probe still has no implementation, so a freeze cannot yet be written even though the schema now demands its evidence
Ruled-out: qualifying delivery through `commitlore context` | budget, trust grading, the guard, lifecycle projection, the matcher and output parsing all sit between it and the agent, and the defect being fixed was zero delivery through exactly those
Ruled-out: requiring every good-control path to carry every record | a record scoped to one file of a multi-file change still reaches an agent that opens that file, and demanding all of them rejects sound tasks
Ruled-out: throwing on a non-zero injector exit | the hook is fail-open by design, so the exit code belongs in the freeze manifest where a reader can weigh it
Warn: delivery qualification proves the record can arrive, not that it will -- an agent that edits another path first or uses a tool outside the matcher still gets nothing, and §9.5 is what records that
Blast: local
Undo: easy
Certainty: firm
Verified: six cases -- a path carrying its record qualifies, a path carrying none does not, a squeezed budget fails what context would pass, a missing second record fails the task, any one of several paths suffices, and the payload digest and exit code are recorded rather than thrown
Provenance: authored
Record-Id: r-delivchk
…lot's numbers

The schema demanded runtime qualification evidence that nothing produced, which
was the correct failure direction and still a gap. It is closed.

The decision function is pure -- probes in, verdict out -- so the gate can be
tested without spending two agent sessions per assertion, and `runProbe` is the
only part that spends them. That split is what makes this suite runnable in CI
at all.

Every case is a pair the pilot actually produced. 80s and 89s qualifies. The
task that sat at 902s and 903s in all four runs does not. 431s -- the slowest
run that ever completed, at 0.48 of budget -- qualifies, which is the observed
separation the 0.6 threshold sits inside without touching either end.

Three refusals encode review findings rather than taste. A single arm is
refused because runtime is treatment-sensitive and qualifying on one arm selects
a corpus that arm finishes faster, with the bias inseparable afterwards. Two
probes of the same arm are refused for the same reason. The verdict reads the
slower arm rather than the mean, because the study runs each task six times and
averaging lets a fast arm carry a slow one into the corpus.

The last test asserts what the probe type does not contain. A selector that
could see functional_pass or rejected_decision_revived would be selecting on the
dependent variable, so the absence is pinned rather than assumed.

Limit: the gate screens runtime and says so -- `completed` means the process returned, which a no-op satisfies, and the pilot's 4.9x within-cell spread means two probes cannot bound the tail
Ruled-out: averaging the two arms | the study runs six sessions per task, so the slower arm is the one that decides whether they fit, and a mean lets the faster arm carry a task in
Ruled-out: keeping the probe artifact out of storage | the freeze manifest is required to carry per-task probe results, and a qualification nobody can recheck is not a gate
Warn: `runProbe` costs two agent sessions per task, so qualifying a 30-task corpus is 60 sessions before a single measured run
Blast: local
Undo: easy
Certainty: firm
Verified: eight cases against real pilot pairs -- 80/89s qualifies, 902/903s timeout does not, 431s at 0.48 of budget qualifies, a 700s slower arm fails on the slower arm, single-arm and same-arm probe sets are refused, an errored arm is refused however fast, and the probe type carries none of the three outcome fields
Provenance: authored
Record-Id: r-runprobe
The runtime gate decides corpus membership by wall time, so a probe on a
different model than the study screens a distribution the study will never
produce. §2.2 already forces a new study id when the observed model changes;
this is the same rule reaching the qualification that selects the corpus, and
`qualifyRuntime` now refuses a probe whose model is not the pinned one.

The model is sonnet. The 0.6 threshold this gate applies was derived from the
pilot's wall times -- completed runs topping out at 0.48 of budget against a
failing task at 1.00 -- and every one of those runs was sonnet. Switching models
does not adjust that derivation, it deletes it, and the threshold goes back to
being a number somebody picked. M1 and M5 also measured sonnet, so this is the
choice that leaves CDEB comparable with the evidence already published.

The refusal is worth having whichever model is chosen later, because the failure
it prevents is silent: a qualification run on a stronger model finishes faster,
admits tasks the study cannot finish, and the resulting timeouts look like task
difficulty rather than a mismatched screen.

Limit: pinning to sonnet means CDEB will say nothing about whether a stronger model needs delivered decisions less, which is a question worth its own study rather than a variable inside this one
Ruled-out: qualifying on opus while measuring on sonnet | the gate would screen runtimes the study never produces, and its threshold is derived from sonnet data
Ruled-out: leaving the probe model unchecked and documenting the requirement | the mismatch produces timeouts that read as task difficulty, so a note cannot catch what a comparison can
Warn: a later model change invalidates the 0.6 derivation as well as the pinning, so both must be re-established rather than only the constant edited
Blast: local
Undo: easy
Certainty: firm
Verified: nine cases pass, including a pair where one arm probed opus against a sonnet-pinned study and the qualification refuses by naming both
Provenance: authored
Record-Id: r-probemodel
@MongLong0214
MongLong0214 merged commit 69b32f2 into dev Aug 8, 2026
8 checks passed
@MongLong0214
MongLong0214 deleted the cdeb-v13 branch August 29, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant