Skip to content

research: full research programme + pre-registered evaluation refuting two reliability mechanisms - #147

Merged
CodeWithJuber merged 3 commits into
masterfrom
research/empirical-refutation
Sep 20, 2026
Merged

CodeWithJuber merged 3 commits into
masterfrom
research/empirical-refutation

Conversation

@CodeWithJuber

@CodeWithJuber CodeWithJuber commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Summary

Uploads the full research programme behind forgekit as a single research/ tree with a master
index, and adds the pre-registered empirical evaluation that refuted two of the programme's own
headline claims
.

That refutation is the contribution. The later work corrects the earlier work, and the corrections
are the most useful part — so research/README.md leads with what was actually measured, not with
what was originally claimed.

What the measurement showed

Claimed (self-built demos) Measured (real data)
Impact oracle recall 1.00 0.022 — 95% CI [0.012, 0.033]
Router/gate F1 1.00 0.37 [0.12, 0.59]
Cost saving +62.1% −20.2% (total pipeline spend)

91% of files got exactly zero recall, and a grep baseline with no dependency graph beat the
graph-based oracle by ~10× on F1. The cost result is the sharpest: routing genuinely saves 59.5% on
first attempts, but almost none of that cheaper output is correct, and once escalation retries are
counted the pipeline costs more than always using the premium tier. The saving did not shrink —
it inverted.

Diagnosis, then repair

Classifying all 18,221 false negatives showed 96.8% were not a ground-truth ceiling: the path
existed in the graph the tool already builds. 94.7% were sibling pairs reachable only by a
mixed-direction path that reverse-only traversal cannot take, plus a construction defect that broke
one of the two standard Python package layouts.

Repairing both, with all eight new parameters fixed on six repositories and frozen before the three
held-out repositories were touched even once
:

Precision Recall F1
As shipped 0.398 0.022 0.042
+ construction fix 0.304 0.242 0.270
+ traversal fix, held-out 0.305 0.653 0.416
grep baseline, held-out 0.269 0.601 0.371

Beats the baseline for the first time. The margin is narrow, and precision generalised distinctly
worse than recall (−0.141 vs −0.021) — visible only because the split was declared in advance.

What is in this PR

research/
  README.md                  master index — leads with the refutation
  cognitive-substrate/       the theory: 48pp whitepaper, literature gap map,
                             graded evidence layer, ecosystem map, the fourteen-mapping
                             design lens, primary sources, figures
  formal-synthesis/          (already in repo) the mathematics
  empirical-refutation/      the pre-registered evaluation, repair, and replication package
  python-prototypes/         (already in repo) the code

Honesty notes a reviewer should check

  • EXECUTIVE_SUMMARY.md carries a status banner saying its own prototype numbers are refuted,
    with a table of what replaced them. Uploading it unmarked would have been misleading.
  • Three widely-repeated industry statistics were caught as misattributed during evidence grading
    and dropped rather than repeated.
  • Priority is conceded. The two-layer composition law is standard protection-layer algebra, and
    two concurrent preprints derive a strictly more general form weeks earlier. What survives is that
    both are simulation-only; this work has real repositories.
  • A theoretical claim is corrected. Perfect recall was inferred from a completeness theorem, but
    such a theorem guarantees completeness only relative to the relation the closure runs over.
  • Our own adversarial review ships with the work, listing four numbers we had stated incorrectly
    before correction — including two cases of reporting the more favourable of several available
    figures, which is exactly the failure the paper indicts.
  • The correctness judge is the same model as the ladder's mid tier, so for 38/64 tasks it grades
    its own model family. That biases toward overstating savings, which makes the reported inversion
    conservative.
  • The freeze was self-administered with no external timestamping authority. A reader can verify
    internal consistency and the append-only amendment trail, but must take the ordering on trust.
  • One figure (eval_precision_recall.png) had a panel title claiming the oracle held the highest F1
    while the same panel plotted grep higher. Corrected and marked superseded.

Review notes

  • No source code changed. The repaired oracle ships inside the replication package rather than
    replacing research/python-prototypes/impact_oracle/, so that swap stays your decision.
  • Opened for review, not for immediate merge.

…ty mechanisms

Adds research/empirical-refutation/: a pre-registered empirical evaluation of the
impact oracle and the complexity-router/assumption-gate against real repositories
and real issues, which REFUTES both of their previously reported headline claims.

Findings:
- Impact oracle recall 1.00 (self-built demo) -> 0.022 (9 OSS repos, 801 files).
  A grep baseline with no dependency graph beats it ~10x on F1.
- Router/gate F1 1.00 -> 0.37 on 80 real issues/PRs, thresholds frozen.
- Cost saving does not shrink but inverts: counting the pipeline's actual
  escalation spend, routing costs 20.2% MORE than always-premium.

Diagnosis and repair:
- 96.8% of missed impact was reachable in the graph the tool already builds:
  94.7% sibling pairs unreachable by reverse-only traversal, plus a construction
  defect affecting src-layout packages. Only 3.2% was a true proxy ceiling.
- Repairing both, with parameters frozen before three held-out repos were touched,
  lifts recall 0.022 -> 0.653 and F1 0.042 -> 0.416, beating grep (0.371) for the
  first time. The margin is narrow and precision generalised worse than recall.

Also corrects a theoretical claim: perfect recall was inferred from Theorem T5,
but T5 guarantees completeness only relative to the relation the closure runs over.
It says nothing about whether that relation contains the edges that matter.

Includes the frozen pre-registration, mined ground truth, held-out task set with
two labelling passes, all results with bootstrap CIs, both prototype versions, a
replication package, and our own adversarial review listing four numbers we had
stated incorrectly before correction.
… retries

The held-out correctness judge is the same model as the tier ladder's mid tier, so
for 38/64 tasks it grades output from its own model family. Evaluators favour their
own generations, which biases the measurement toward OVERSTATING routed savings --
the reported cost inversion is therefore conservative. Also discloses seven
generation calls retried at a raised token budget after returning empty output.
Adds the remaining three quarters of the programme alongside the empirical
refutation already on this branch, and a research/README.md that indexes all four
layers and leads with what the measurement actually showed.

New under research/cognitive-substrate/:
- the Theory -> Evidence -> Build-Map whitepaper (48pp, HTML + PDF)
- EXECUTIVE_SUMMARY.md, carrying a status banner: its prototype numbers are
  refuted by research/empirical-refutation/ and the banner says so up front
- literature/  gap map + 32 graded references behind each faculty claim
- evidence/    twelve industry statistics independently re-grounded and graded
               confirmed/vendor-reported/unverifiable (three widely-repeated
               statistics were caught as misattributed and dropped), plus the
               ecosystem map of what the 2026 stack already solves
- quranic-lens/ the fourteen-mapping ethical-epistemic design lens, with its
               framing caveat: it names which safeguards are obligatory, and is
               never offered as technical authority for an engineering claim
- sources/     the primary documents the evidence layer was graded against
- figures/     architecture schematics and prototype evaluations

Also corrects eval_precision_recall.png, whose panel title claimed the oracle had
the highest F1 while the same panel plotted grep higher, and marks the figure
superseded.

No source code changed.
@CodeWithJuber CodeWithJuber changed the title research(refutation): pre-registered evaluation refutes two reliability mechanisms research: full research programme + pre-registered evaluation refuting two reliability mechanisms Sep 20, 2026
@CodeWithJuber
CodeWithJuber merged commit b9a340a into master Sep 20, 2026
5 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant