From e5f9b73d55705d329daf4f750c5be0cd6273bbd6 Mon Sep 17 00:00:00 2001 From: MongLong0214 Date: Tue, 28 Jul 2026 18:31:53 +0900 Subject: [PATCH] Withdraw M4's null as evidence about the guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No run in either arm received injected context. Both arms are defined by receiving records; neither did. The comparison was nothing against nothing, and its null is not a weak result about the guard — it is not a result about the guard. Established by restoring the pinned harness 081d858c and probing it: the saved transcript carries no occurrence of commitlore, Ruled-out, Limit: or active records anywhere in its text. A recording gap would leave the context in the transcript and empty only the field. It was never delivered. The corrected statistical analysis stays. McNemar p=0.1094, ICC 0.581, DEFF 8.56, effective n 13, four saturated tasks — all still true about the data and all irrelevant as evidence about the guard. Both things need saying and the verdict now says both. Provenance is unchanged and still clean: one harness commit, one dist digest, 112 rows, no mid-run rebuild. The failure sits upstream of provenance. The harness recorded faithfully what it did, and what it did was run both arms without records. Ruled-out: retracting the dataset or calling M4 invalid | the data is valid and its provenance is clean; what it measured was not the treatment, and those are different words Limit: the guard question is now unanswered rather than answered null Blast: system Undo: costly Certainty: firm Record-Id: r-m4withdraw --- README.ja.md | 4 +- README.ko.md | 4 +- README.md | 4 +- README.zh-CN.md | 4 +- bench/README.md | 38 +++-- bench/VERDICT-M4.md | 331 +++++++++------------------------------- docs/ROADMAP-TO-DONE.md | 32 ++-- 7 files changed, 120 insertions(+), 297 deletions(-) diff --git a/README.ja.md b/README.ja.md index a9a09fd3..59e539e5 100644 --- a/README.ja.md +++ b/README.ja.md @@ -162,7 +162,7 @@ text search ではなく Git trailer parser を使います。本文の `Key:` ## Evidence: より狭い製品上の主張 -112 回の実験を行いましたが、測定した agent behavior の主張は裏付けられませんでした。そのため CommitLore は上記の、より狭い製品上の主張をします。全ての限界は [M4 verdict](bench/VERDICT-M4.md) で読めます。 +112 回の実験は記録されましたが、M4 はどちらの arm にも record を届けませんでした。したがって agent behavior の主張を検証も支持も反証もしていません。上記のより狭い製品上の主張は独立して検証可能な動作に基づきます。クリーンなデータセットと撤回については [M4 verdict](bench/VERDICT-M4.md) を読んでください。
完全な benchmark record（112 回の実験） @@ -224,7 +224,7 @@ node ~/.commitlore/dist/commitlore.mjs context src/auth - Windows は未対応です: [#95](https://github.com/MongLong0214/commitlore/issues/95)。 - Alpine および他の musl Linux host は未対応です: [#99](https://github.com/MongLong0214/commitlore/issues/99)。 - cryptographic author verification、repository-wide record coverage、symbol anchor、interactive record builder は未実装です: [#28](https://github.com/MongLong0214/commitlore/issues/28)、[#32](https://github.com/MongLong0214/commitlore/issues/32)、[#33](https://github.com/MongLong0214/commitlore/issues/33)、[#34](https://github.com/MongLong0214/commitlore/issues/34)。 -- benchmark は agent behavior に対する guard の効果を実証していません: [#37](https://github.com/MongLong0214/commitlore/issues/37)。 +- M4 は guard の効果を検証していません。どちらの arm にも injected record が届きませんでした: [#122](https://github.com/MongLong0214/commitlore/issues/122)。 ## コントリビュート diff --git a/README.ko.md b/README.ko.md index 4488075f..e03b744e 100644 --- a/README.ko.md +++ b/README.ko.md @@ -162,7 +162,7 @@ git log --follow --format='%h %(trailers:key=Limit,valueonly)' -- src/auth/ ## 근거: 더 좁은 제품 주장 -112회 실험을 했지만, 측정한 에이전트 행동 주장은 뒷받침되지 않았다. 그래서 CommitLore는 위의 더 좁은 제품 주장을 한다. 전체 한계는 [M4 verdict](bench/VERDICT-M4.md)에서 읽을 수 있다. +112회 실험은 기록됐지만 M4는 어느 arm에도 record를 전달하지 않았다. 따라서 에이전트 행동 주장을 시험하거나 뒷받침하거나 반박하지 못한다. 위의 더 좁은 제품 주장은 독립적으로 검증 가능한 동작에 근거한다. 깨끗한 데이터셋과 철회 내용은 [M4 verdict](bench/VERDICT-M4.md)에서 읽을 수 있다.
전체 benchmark 기록 (112회 실험) @@ -224,7 +224,7 @@ node ~/.commitlore/dist/commitlore.mjs context src/auth - Windows는 지원하지 않는다: [#95](https://github.com/MongLong0214/commitlore/issues/95). - Alpine 및 다른 musl Linux host는 지원하지 않는다: [#99](https://github.com/MongLong0214/commitlore/issues/99). - 암호학적 작성자 검증, 저장소 전체 record coverage, symbol anchor, interactive record builder는 아직 구현되지 않았다: [#28](https://github.com/MongLong0214/commitlore/issues/28), [#32](https://github.com/MongLong0214/commitlore/issues/32), [#33](https://github.com/MongLong0214/commitlore/issues/33), [#34](https://github.com/MongLong0214/commitlore/issues/34). -- benchmark는 에이전트 행동에 대한 guard 효과를 입증하지 못한다: [#37](https://github.com/MongLong0214/commitlore/issues/37). +- M4는 guard 효과를 시험하지 못했다. 어느 arm도 injected record를 받지 못했다: [#122](https://github.com/MongLong0214/commitlore/issues/122). ## 기여하기 diff --git a/README.md b/README.md index 0915f5e9..e6a959cb 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ These are product claims about Git-bound, human-verifiable decision history. The ## Evidence: a narrower product claim -112 experiments were run, but the measured agent-behavior claim was not supported. CommitLore therefore makes the narrower product claim above. Read the [M4 verdict](bench/VERDICT-M4.md) for the full limits. +112 experiments were recorded, but M4 delivered records in neither arm. It did not test, support, or refute the agent-behavior claim. The narrower product claim above rests on independently testable behavior; read the [M4 verdict](bench/VERDICT-M4.md) for the clean dataset and withdrawal.
Full benchmark record (112 experiments) @@ -224,7 +224,7 @@ node ~/.commitlore/dist/commitlore.mjs context src/auth - Windows is unsupported: [#95](https://github.com/MongLong0214/commitlore/issues/95). - Alpine and other musl Linux hosts are unsupported: [#99](https://github.com/MongLong0214/commitlore/issues/99). - Cryptographic author verification, repository-wide record coverage, symbol anchors, and an interactive record builder are not implemented yet: [#28](https://github.com/MongLong0214/commitlore/issues/28), [#32](https://github.com/MongLong0214/commitlore/issues/32), [#33](https://github.com/MongLong0214/commitlore/issues/33), [#34](https://github.com/MongLong0214/commitlore/issues/34). -- The benchmark does not demonstrate a guard effect on agent behavior: [#37](https://github.com/MongLong0214/commitlore/issues/37). +- M4 did not test a guard effect: neither arm received injected records ([#122](https://github.com/MongLong0214/commitlore/issues/122)). ## Contributing diff --git a/README.zh-CN.md b/README.zh-CN.md index 224adfb4..ab53b7ed 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -162,7 +162,7 @@ git log --follow --format='%h %(trailers:key=Limit,valueonly)' -- src/auth/ ## Evidence：更窄的产品主张 -完成了 112 次实验，但所测 agent behavior 的主张未获支持。因此 CommitLore 提出的是上面更窄的产品主张。完整限制见 [M4 verdict](bench/VERDICT-M4.md)。 +已记录 112 次实验，但 M4 没有向任何 arm 交付 record。因此它没有检验、支持或反驳 agent behavior 的主张。上面更窄的产品主张基于可独立验证的行为。关于干净的数据集和撤回，请见 [M4 verdict](bench/VERDICT-M4.md)。
完整 benchmark 记录（112 次实验） @@ -224,7 +224,7 @@ node ~/.commitlore/dist/commitlore.mjs context src/auth - 不支持 Windows：[#95](https://github.com/MongLong0214/commitlore/issues/95)。 - 不支持 Alpine 与其他 musl Linux host：[#99](https://github.com/MongLong0214/commitlore/issues/99)。 - 尚未实现 cryptographic author verification、repository-wide record coverage、symbol anchor 和 interactive record builder：[#28](https://github.com/MongLong0214/commitlore/issues/28)、[#32](https://github.com/MongLong0214/commitlore/issues/32)、[#33](https://github.com/MongLong0214/commitlore/issues/33)、[#34](https://github.com/MongLong0214/commitlore/issues/34)。 -- benchmark 未能证明 guard 对 agent behavior 有效果：[#37](https://github.com/MongLong0214/commitlore/issues/37)。 +- M4 没有检验 guard 效果：没有任何 arm 收到 injected record（[#122](https://github.com/MongLong0214/commitlore/issues/122)）。 ## 贡献 diff --git a/bench/README.md b/bench/README.md index 4a934d4c..e415df04 100644 --- a/bench/README.md +++ b/bench/README.md @@ -1,19 +1,19 @@ # CommitLoreBench -**M4 is the citable dataset.** `bench/results/t702-m4-final.jsonl` records the -harness commit and the `dist/` digest for every row, `bench/report.ts` summarizes -it, and the README's numbers block is generated from it. M3 was voided for -lacking that provenance (§15); M4 was designed, registered and run to supply it, -and its result is null. The historical executed report is -`bench/VERDICT-M4.md`; the canonical paired-and-clustered correction is -`docs/VERDICT-M4.md`. Every earlier dataset -(M1, M1-b, M2) still lacks the fields and is not pooled into the generated block -for that reason, not because it was withdrawn as a record; each has its own -verdict document. - -Measures the one thing that decides whether CommitLore is worth building: does an -agent that can see recorded decisions stop re-proposing the approaches a team -already rejected? +**M4 is a citable, clean-provenance dataset, not a guard test.** +`bench/results/t702-m4-final.jsonl` records the harness commit and `dist/` +digest for every row, and `bench/report.ts` summarizes it. M3 was voided for +lacking that provenance (§15); M4 has it. But M4's 112 transcripts contain no +injected context in either arm, so its valid data do not answer the guard +question. The withdrawal and the corrected statistics as observations about +the data are in `bench/VERDICT-M4.md`. Every earlier dataset (M1, M1-b, M2) +still lacks the fields and is not pooled into the generated block for that +reason, not because it was withdrawn as a record; each has its own verdict +document. + +The benchmark is designed to ask whether an agent that receives recorded +decisions stops re-proposing approaches a team already rejected. M4 did not +deliver those decisions, so it does not answer that question. - Design: `docs/adr/ADR-0007-commitlorebench.md` - Requirements: `docs/prd/PRD-F7-commitlorebench.md` @@ -81,6 +81,11 @@ Conditions are an open string enum, so M4 adds arms without touching the runner. Planned arms are rejected at CLI parse time with a pointer to their ticket. +**M4 correction:** although its labels were `commitlore-on` and +`commitlore-guard`, all 112 stored M4 transcripts have `injected_context: null`. +The condition table describes the intended harness behavior; M4 did not receive +the treatment and is not evidence about it. + `--cond both` is the two arms of the primary comparison and `--cond all` is every supported arm. Until T-703 those were the same list; they are not any more, and against a live driver the difference is two arms or five. The runner @@ -938,8 +943,9 @@ Fisher exact also treats runs as independent, while the design is paired by (task, seed). The output says so. The test is the one ADR-0007 and T-702 registered, but it does not provide a valid hypothesis test for paired data. The original number remains part of the historical report; the registered -replacement and M4 correction are in `docs/MEASUREMENT-PROTOCOL.md` and -`docs/VERDICT-M4.md`. +replacement is in `docs/MEASUREMENT-PROTOCOL.md`. M4's paired/clustered +statistics are preserved in `bench/VERDICT-M4.md` as descriptions of rows that +did not receive the treatment, not as a correction of a guard estimate. ### What makes the measured effect a floor — one thing, not two diff --git a/bench/VERDICT-M4.md b/bench/VERDICT-M4.md index f9e75590..0ff654e3 100644 --- a/bench/VERDICT-M4.md +++ b/bench/VERDICT-M4.md @@ -1,284 +1,103 @@ -# M4 verdict — the qualified matrix, and it is still null +# M4 verdict — valid data; null withdrawn as guard evidence -> **Historical executed report.** This file preserves the analysis M4 originally -> reported. Fisher exact is not valid for its paired design, and `over-turns` -> did not stop or truncate the recorded run. The canonical correction is -> [`docs/VERDICT-M4.md`](../docs/VERDICT-M4.md); the result remains null and no -> observation is retracted. +> **Historical executed report and correction.** M4's 112 rows and their +> provenance remain valid. Its null is retained as an observation about those +> rows, but withdrawn as evidence about the guard: neither arm received the +> records that defined the comparison. -**n = 56 per arm, 7 of 7 seeds complete on all 8 qualifying tasks, 112 of 112 -runs.** No truncation-driven shortfall in the matrix itself — one run finished -naturally after the unenforced turn budget and is labelled `over-turns`, -counted, not excluded (see *Limits* below). +M4 was registered as a comparison of `commitlore-guard` and `commitlore-on`, +two arms defined by receiving records. The committed transcripts show +`injected_context: null` on **0 of 112** runs: 0/56 `commitlore-guard` and +0/56 `commitlore-on`. This is not a field that was never used: the earlier +`bench/results/transcripts-final/` set has injected context on 30 of 60 runs. -Registered as `PREREGISTRATION.md` §16, after the control-only qualification -round recorded there. M1 (p = 0.7480), M1-b (p = 0.0522) and M2 (p = 0.2247) -are not revised; M3 is void (§15). Run from an isolated checkout outside the -working tree, per §16 precondition 2. Every one of the 112 rows carries -`harness_commit: 081d858c1667455f90b6d012e62a2cd2a549c50c` and -`dist_digest: f658927cae15c92a1cba2b7f0dc21119f47e2d72aea412d90489c42eb890b75e` -— one value each, verified before this document was written, satisfying §16 -precondition 1 and the refusal `bench/metrics.ts` would otherwise raise. +The pinned M4 harness (`081d858c1667455f90b6d012e62a2cd2a549c50c`) was restored +and probed under M4's conditions. It again produced `injected_context: None` +and `matched: []`; the agent transcript contained none of `commitlore`, +`Ruled-out`, `Limit:`, or active records. A recording gap would leave the +delivered context in that transcript. Nothing was delivered. -Data: `bench/results/t702-m4-final.jsonl`, 112 transcripts committed alongside -at `bench/results/transcripts-m4/`. No manifest — see *Limits*. +The two M4 arms therefore compared **nothing against nothing**. Its null is +not a weak result about the guard, and it is not a result about the guard at +all. ---- +## What remains valid -## Result +**n = 56 per arm, 7 of 7 seeds on all 8 tasks, 112 rows.** The dataset is not +retracted and M4 is not called invalid. It faithfully records what the harness +ran; what it ran was both arms without records. -| arm | re-proposed | rate | 95% CI (rate diff., vs. `commitlore-guard`) | -|---|---|---|---| -| `commitlore-on` | 35 / 56 | 62.5% | — | -| `commitlore-guard` | 41 / 56 | 73.2% | — | - -**Fisher exact, two-tailed: p = 0.3117.** Rate difference −10.7pp (`on` minus -`guard`), 95% Newcombe interval [−27.1pp, +6.5pp]. Odds ratio 0.6098. - -Violations: **0 / 56 in both arms.** The secondary outcome variable -registered in §16 point 3 — constraint violation of a recorded `Limit:` — -found nothing to report in either arm. - -**The hypothesis is not supported at α = 0.05.** As registered: `commitlore-off` -is a reference only; the primary test is `commitlore-guard` against -`commitlore-on` (§16, §14 line 517), because guard against nothing would -confound the route with the presence of records at all. +Provenance remains clean: every row has the single +`harness_commit: 081d858c1667455f90b6d012e62a2cd2a549c50c` and the single +`dist_digest: f658927cae15c92a1cba2b7f0dc21119f47e2d72aea412d90489c42eb890b75e`. +There are 112 rows and no mid-run rebuild. The failure is upstream of +provenance. The harness recorded faithfully what it did, and what it did was +run both arms without records. -Computed by `bench/metrics.ts` directly against `t702-m4-final.jsonl`; the -figures above are its output, not a transcription. +Data: `bench/results/t702-m4-final.jsonl`, with 112 committed transcripts at +`bench/results/transcripts-m4/`. No manifest was written; the model provenance +limitation remains separate from this delivery failure. -No subset was cut to look for an effect. §4 forbids it here more than -anywhere else in this project's history — this is the run built specifically -to give the hypothesis a fair matrix, and re-cutting a fair result after -seeing it would be the one move that could manufacture significance out of -the exact design meant to prevent that. +## Observed M4 data, not a guard effect -The Fisher exact test above is `PREREGISTRATION.md` §2's registered test, and -this document reports it as registered. It is also the wrong test for this -design, for a reason that has nothing to do with the p-value it produced — -see the next section. - ---- +| arm | re-proposed | rate | 95% CI (rate diff., vs. `commitlore-guard`) | +|---|---:|---:|---| +| `commitlore-on` | 35 / 56 | 62.5% | — | +| `commitlore-guard` | 41 / 56 | 73.2% | — | -## The registered test does not fit the design +The registered Fisher calculation is p = 0.3117; the observed rate difference +is −10.7pp (`on` minus `guard`), with Newcombe interval [−27.1pp, +6.5pp] and +odds ratio 0.6098. Violations were 0/56 in both arms. -Recorded here because a null result computed on an invalid test is not a -settled null; it is an open question with a number attached. Both problems -below are properties of the design, not of what the numbers came back as — -they would be exactly as true if p had come back significant. Independently -computed against `bench/results/t702-m4-final.jsonl`; the registered protocol -and canonical correction in `docs/` supersede the analysis in this section -without retracting the observations. +Those are correct descriptions of the recorded data. They do not estimate a +guard effect, because neither label received the treatment it names. -### 1. The 112 runs are not 112 independent observations — they are 56 pairs +## Corrected statistical analysis still describes the data -Every seed × task cell ran once under `commitlore-on` and once under -`commitlore-guard`, sharing the workspace, task and seed. That is a paired -design. Fisher exact assumes two independent groups; the literature on -matched binary data is explicit that Fisher does not provide a valid -hypothesis test when the two samples are paired rather than independent — it -does not use the pairing and does not correct for it. +The statistical correction already made for M4 remains correct arithmetic on +the rows and is irrelevant as evidence about the guard. Both statements are +true. -The correct paired test is McNemar's, on the pairs where the arms disagreed: +The 56 seed × task cells are paired. On the recorded outcomes, the paired table +is: | | `commitlore-guard`: reproposed | `commitlore-guard`: did not | |---|---:|---:| | `commitlore-on`: reproposed | 33 | 2 | | `commitlore-on`: did not | 8 | 13 | -56 pairs. Concordant 46 (33 both re-proposed, 13 both clean) — these carry no -information about a difference between the arms and Fisher's test spends -statistical power on them anyway. Discordant 10: 2 where `commitlore-on` -re-proposed and `commitlore-guard` did not, 8 where `commitlore-guard` -re-proposed and `commitlore-on` did not. - -**McNemar's exact test (two-sided, binomial on the 10 discordant pairs): -p = 0.1094.** - -**Still null.** That is exactly why it is safe to report this plainly rather -than treat it as a threat to the conclusion — nothing above turns the result -significant, so correcting the test cannot be read as motivated. It does mean -the p = 0.3117 in *Result* above is the registered number, not the right one, -and both facts belong in the same document. +There are 10 discordant pairs, giving **McNemar's exact two-sided p = 0.1094**. +The original Fisher calculation is not the right test for paired rows; neither +calculation answers the guard question without delivered records. -### 2. The runs are also clustered by task, and were analyzed as if they were not - -One-way ANOVA on `reproposed` (0/1) with task as the cluster, 8 clusters of -14 (both arms, all seven seeds, per task): +The task clustering correction also remains true about the data: ``` ICC = 0.581 · cluster size 14 · DEFF = 1 + (14-1)×0.581 = 8.56 effective n = 112 / 8.56 ≈ 13, against a nominal 112 ``` -A design effect this large means the 112 rows carry roughly the statistical -information of 13 independent ones. The published Newcombe interval, -[−27.1pp, +6.5pp], is a 95% interval under an independence assumption this -design does not satisfy. - -**Two different quantities, two different factors — stated explicitly so the -next reader does not have to re-derive it:** - -``` -n_eff = n / DEFF sample size divides by the design effect -SE ×= √DEFF standard error, and CI half-width, scale with its square root -``` - -Design effect (`DEFF = 8.56`) corrects *variance*; a confidence interval's -half-width is proportional to standard error, i.e. to √variance, not to -variance itself. Scaling each half-width of the published interval by -√8.56 ≈ 2.92, as an illustrative approximation rather than a formal -re-analysis, gives **roughly [−58.7pp, +39.6pp]**. - -That interval spans zero by a wide margin in both directions. The honest -statement is sharper than "not significant": at this design's actual -information content, **the study cannot distinguish a large benefit from a -large harm.** That is the finding, and it is the reason the statistical -protocol is being re-registered rather than this one number being patched in -place. - -### 3. Four of the eight qualifying tasks are saturated, and that is itself a finding - -`qualification-gitseed-boolean-security`, `-fake-tty`, `-grading-fail-fast` -and `-single-smoke-sample` ran 7/7 in **both** arms — every seed, every -condition, always reproposed. A task at ceiling in both arms contributes zero -information to a paired or clustered comparison: there is no discordant pair -it could produce and no variance it could contribute. - -`PREREGISTRATION.md` §16's qualification round used a **floor** — 4-of-6 -control re-proposals — to screen out silent tasks. It did not screen for a -ceiling, so the tasks that qualified most strongly are exactly the ones with -no headroom left to show an effect in either direction. M1 and M2 died of an -empty instrument, seven of ten tasks silent at zero. M4's paired analysis -dies of a saturated one, four of eight tasks pinned at one. - -**Not dropped, and not re-run.** §4 forbids cutting a subset after seeing -outcomes, and it applies here with the same force it applies everywhere else -in this document: the saturation is the finding, not an inconvenience to -filter around before reporting the next number. - ---- - -## What the qualification round changed, and what it did not - -M1 and M2 were null because the instrument was mostly empty: seven of ten -tasks never showed a control-arm re-proposal at all, so seventy percent of -each matrix could not move the needle regardless of what CommitLore did. That -diagnosis is in `PREREGISTRATION.md` §16 and `docs/ROADMAP-TO-DONE.md`, and it -is measured, not guessed — the qualification round exists to test it directly -rather than assume it a second time. - -It worked. The registered qualification brief supplies a **77% aggregate base -rate** across the ten candidate tasks (46/60; the eight that actually -qualified ran higher still, 43/48 = 89.6% — §16 records the discrepancy and -uses the lower, conservative figure for sizing). Here, with the full matrix -run, both arms reproposed in that same range: - -| task | `commitlore-on` | `commitlore-guard` | -|---|---:|---:| -| `qualification-gitseed-approved-bool` | 2/7 | 2/7 | -| `qualification-gitseed-boolean-security` | 7/7 | 7/7 | -| `qualification-gitseed-drop-withheld` | 0/7 | 2/7 | -| `qualification-gitseed-fake-tty` | 7/7 | 7/7 | -| `qualification-gitseed-grading-fail-fast` | 7/7 | 7/7 | -| `qualification-gitseed-non-interactive` | 1/7 | 3/7 | -| `qualification-gitseed-numeric-sentinel` | 4/7 | 6/7 | -| `qualification-gitseed-single-smoke-sample` | 7/7 | 7/7 | - -**Zero of eight tasks are silent.** Every task that qualified produced -re-proposals in both arms, at aggregate rates of 62.5% and 73.2% — in the same -band the qualification round predicted, not the ~20% M1 and M2 measured -against an unqualified task set. Compare that to M1's task table (§7 of the -pre-registration; `VERDICT-M1.md`): seven of ten silent, control re-proposing -in only four of ten at all. - -**The instrument was full this time, and the effect still did not appear.** -That is a materially stronger null than M1's or M2's. Those two results could -be, and were, read as "the matrix could not detect an effect of this size" — -`VERDICT-M1.md` computed 5.1% power at n = 30. M4 was sized for 80% power -against a one-third reduction in a 77% base rate, ran on tasks that reproduce -at the rate the design predicted, and against a fully powered, non-silent -instrument the guard route did not reduce re-proposal below the injection -route. The absence of an effect here is not the absence of an opportunity to -see one. - -Full is not the same claim as informative, though, and the next section -narrows it: four of these eight tasks are saturated at 7/7 in *both* arms, -which contribute nothing to a paired or clustered analysis even though they -are not silent. M1/M2 failed by emptiness; the paired/clustered re-analysis -below shows M4 failing by a related but distinct mechanism — see -*The registered test does not fit the design*. - ---- - -## Two limitations the tooling surfaced - -Both flagged by the runner and the schema themselves, not inferred after the -fact. - -### 1. The model that produced these 112 rows cannot be proven - -`RunRecord` has no `model` field, and `runner.ts` accepts `--model` and passes -it to the driver without ever writing it onto the row — -`bench/README.md`, "Open after T-702", item 1. For M1, M1-b and M2 the gap was -closed by a `*.manifest.json` sidecar that records the invocation, including -`model`. **No manifest was written for this run.** `bench/metrics.ts` reports -it plainly: every row's model reads `(unrecorded)`, with the warning the tool -prints whenever that happens. - -This was theoretical when `bench/README.md` first listed it. It is not -theoretical now: there is no artifact anywhere in this repository, this -checkout, or this run's logs that names the model behind the original 93 rows -(`bench/results/t702-m4-final.runner.log`) or the 19 resumed afterward -(`t702-m4-final-resume.runner.log`). Re-proposal is a model-dependent -behaviour (§5), and this dataset cannot say whose behaviour it measured. Filed -as its own issue, -[#106](https://github.com/MongLong0214/commitlore/issues/106), rather than -folded into this document, because a gap in the harness is not a property of -the M4 result — it is a property of every dataset this harness produces until -the two lines `bench/README.md` names are written. - -### 2. One run exceeded the turn budget, and it is recorded, not excluded - -`qualification-gitseed-grading-fail-fast`, `commitlore-on`, seed 5, is labelled -`over-turns` at 31 turns rather than `completed`. In this harness the label -means that the process finished on its own after an observed, unenforced turn -budget; the harness did not stop or truncate it. §4 still governs the analysis -set: **it is not excluded.** `stopped_by: "error"`, `simulated: true` and -never-started rows leave the analysis set; `over-turns` is not one of those -three, and this row carries a real measurement (`reproposed: true`) like every -other row in the table above. - ---- - -## What this does and does not license - -- **It does not license claiming guard is worse than injection.** The - observed direction (`on` 62.5% vs. `guard` 73.2%) is the opposite of what - the hypothesis in §14 predicts, but the interval crosses zero by a wide - margin and a non-significant result is not evidence for either ordering — - the same discipline `VERDICT-M1b.md` applied to the interval-vs-test - question. -- **It does not license a fifth measurement on this task set with more - seeds.** Unlike M1, this instrument was not underpowered on its own terms — - it was sized for the effect the qualification round predicted, and it did - not find it. §16 governed this: "three null measurements and a fourth would - be the finding, not a reason for a fifth." This is the fourth. -- **It does license retiring the "was the matrix broken" explanation.** That - explanation was available for M1 and M2 and is not available here. - ---- - -## What the product may claim now - -`docs/ROADMAP-TO-DONE.md` committed to this outcome before the run existed: - -> **A positive benchmark is not required.** If M4 is null again, that is the -> result, and it will be published. In that case, the product claim is not -> "makes agents better," but **"binds decision history to git and preserves it -> in a human-verifiable form."** The latter is already proven by tests and -> remains true independent of the benchmark. - -M4 is null again. The claim narrows to that sentence, and it is not a -downgrade invented after the fact — it is the position this project registered -in advance of knowing which way this document would come out. +The independence-based Newcombe interval [−27.1pp, +6.5pp] is therefore not a +valid confidence interval for a treatment effect. Scaling its half-widths by +√8.56 ≈ 2.92 gives the existing illustrative corrected interval, roughly +**[−58.7pp, +39.6pp]**. It remains an interval over the observed label groups, +not evidence about a guard that was never delivered. + +Four tasks were saturated at 7/7 in both recorded labels: +`qualification-gitseed-boolean-security`, `-fake-tty`, +`-grading-fail-fast`, and `-single-smoke-sample`. That saturation remains an +observation about the data. It cannot be used to explain or qualify a guard +effect that M4 did not measure. + +## What does not change + +- No row is discarded, and no provenance claim is withdrawn. +- The missing model manifest remains a limitation; one `commitlore-on` run + labelled `over-turns` still finished naturally and remains included. +- The dataset may be cited for its recorded outcomes and clean provenance, but + never as a test, null, or estimate of a guard effect. + +The product claim remains the independently testable one: CommitLore binds +decision history to Git and preserves it in a human-verifiable form. M4 neither +supports nor refutes an agent-behavior claim; a repaired delivery path must be +verified before a new experiment can address that question. diff --git a/docs/ROADMAP-TO-DONE.md b/docs/ROADMAP-TO-DONE.md index 29974cae..74de10ba 100644 --- a/docs/ROADMAP-TO-DONE.md +++ b/docs/ROADMAP-TO-DONE.md @@ -39,20 +39,18 @@ Current: `dev` default branch, CI green, 1171 tests/33 files, 10/10 review block | Phase | Work | Gate | |---|---|---| | **3** | Plugin manifest redeclares conventional-location `hooks` — double registration likely. No manifest tests | Manifest suite green + clone contains every declared file | -| **4** | ~~Design and run M4~~ **Done, null.** Execute every remaining item in `docs/RELEASE-GATE.md` | All 6 gate sections pass, CI green at that commit | +| **4** | ~~Design and run M4~~ **Executed; withdrawn as guard evidence.** Repair and verify context delivery before a new guard experiment | A treatment arm demonstrably receives records, then all 6 gate sections pass and CI is green at that commit | -### M4 — the answer to why M1 and M2 were null +### M4 — valid data, no delivered treatment -**Executed.** `bench/PREREGISTRATION.md` §16, run from an isolated checkout, 8 qualifying -tasks × 7 seeds × 2 arms = 112 runs, every row carrying a uniform `harness_commit` and -`dist_digest`. The qualification round's diagnosis held: all eight tasks reproposed in -both arms (62.5% and 73.2% aggregate), against M1/M2's seven-of-ten silent tasks. The -registered test — `commitlore-guard` against `commitlore-on`, two-tailed Fisher exact — -is **not significant** (`bench/VERDICT-M4.md`). Two limitations the tooling itself -surfaced: no manifest was written, so the model behind these 112 rows is unrecorded -(filed as [#106](https://github.com/MongLong0214/commitlore/issues/106)); one -`commitlore-on` row was truncated by `over-turns` and is recorded, not excluded. The -product claim below is now the operative one — see *Completion condition*. +**Executed, with clean provenance.** `bench/PREREGISTRATION.md` §16 ran 8 qualifying +tasks × 7 seeds × 2 labels = 112 rows, all with one `harness_commit` and one +`dist_digest`. But both labels were defined by receiving records and none of the +112 transcripts has injected context. The recorded 62.5% and 73.2% rates, and +their corrected paired/clustered analysis, remain true about the data and say +nothing about the guard. M4 is neither retracted nor invalidated: the failure +is upstream of provenance, because the harness faithfully recorded two arms +without records. See `bench/VERDICT-M4.md` and [#122](https://github.com/MongLong0214/commitlore/issues/122). **Diagnosis (measured)**: of 10 tasks, **7 have a control base rate of 0**. Even without CommitLore, the agent does not propose the rejected approach. Tasks with nothing to block made up 70% of the @@ -87,11 +85,11 @@ Power is governed more by the base rate than the sample. For an effect that cuts Every section of `RELEASE-GATE.md` has been executed and passed, CI is green **at that commit**, and every remaining issue is a feature on a defensible axis rather than a defect. -**A positive benchmark is not required.** M4 came back null, on a fully-powered, -non-silent instrument (`bench/VERDICT-M4.md`) — the exact outcome this section committed to -publishing in advance. The product claim is therefore not "makes agents better," but -**"binds decision history to git and preserves it in a human-verifiable form."** The latter -is already proven by tests and remains true independent of the benchmark. +**A positive benchmark is not required.** M4 did not apply its treatment, so it +establishes nothing about agent behavior. The product claim is not "makes agents +better," but **"binds decision history to git and preserves it in a +human-verifiable form."** The latter is already proven by tests and remains +true independent of the benchmark. ---