D-CZ-1 PASSES, D-CZ-0 had no artifact behind it — plus the arc's rated formula matrix - #946
Conversation
Two findings, one run (probes/weather-p1/substrate_comfort_d_cz_0_1.py). 1. D-CZ-0 was marked DONE with NO committed script or JSON. Its figures were quoted in the plan, the arc entry, the LATEST_STATE row and three PR bodies. Worse: the #945 self-audit -- run specifically because this arc keeps shipping unbacked summary claims -- reported those figures as "verified" while only comparing the arc entry to the PLAN. Both are prose. A figure cited in two documents is cited twice, not confirmed once. Reproduced now: 4 of 9 rows (ratios 1.004/1.022/0.994/0.931). The five EXCLUDED land candidates are unreproducible -- their box centres were never recorded anywhere -- and no coordinates were invented to fake them. The |grad p| definition was never committed either, so four candidates were computed and the winner decided from data: Pa per grid cell with NO cos(lat) metric (max dev 0.069; next-best 0.398). That understates high latitude by 1/cos(lat), so R3 is ~40% low; metric-corrected the ladder is 10.3/15.5/61.2/100.9 -- ORDER survives, range widens 9.3x -> 9.8x. Two defects in the reproduction caught by its own guards: measuring all 19 storms at the preflight timestep instead of each storm's own t0 (inverted R3/R4 and would have read as C1 failing), and a seam assertion firing on a real storm at 353.4 E where a plain slice gives a one-column box. 2. D-CZ-1 PASSES. Both controls lose to both real arms on both metrics in all four regimes; GEO-DEGENERATE saturates 92-97%, so the mechanism is visible rather than assumed. C1b separation = 6.28 vs the >= 3 bar. 3. ...and the same run AMENDS C4. rho is SATURATED on the diagonal -- real-arm spread 3e-6..4.7e-5 -- so C4 as pre-registered could not have fired. rho keeps four orders of magnitude of range OFF the diagonal, so L keeps it; C4 moves to RMSE in Pa with rho as a floor check. Legitimate because D-CZ-1's whole purpose is to test the apparatus before the expensive cells and no C4 cell has been scored; illegitimate the instant one has. Recorded as an amendment carrying its trigger, and propagated to the header and the C4 bar rather than left contradicting them. A HINT, explicitly not a result: CAL-ABS beats CAL-RANK on RMSE everywhere and its margin shrinks as the field gets more active (3.96 -> 1.85 -> 1.14), R4 breaking the monotone at 1.49. One timestep, no control on that comparison, and diagonal-vs-diagonal is not what the hypothesis is about. Recorded so a later run cannot present it as a confirmation. EPIPHANIES: E-A-FIGURE-CITED-TWICE-IS-NOT-CONFIRMED-ONCE-1 and E-THE-METRIC-THAT-SEPARATES-ONE-COMPARISON-IS-BLIND-TO-ANOTHER-1 (prepend, suffix-verified). STATUS_BOARD D-CZ-0/D-CZ-1 updated. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_3024ba54-80a6-42bf-9058-41888fa9a1b6) |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughThe PR adds a WeatherBench2 probe for D-CZ-0 and D-CZ-1, records results for four regimes and 19 storms, changes C4 to RMSE with a correlation floor, and documents formula evidence and reproducibility limits. ChangesSubstrate comfort-zone evaluation
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant WeatherBench2
participant Probe
participant JSONArtifact
participant EvaluationRecords
WeatherBench2->>Probe: provide decoded fields and metadata
Probe->>Probe: extract boxes and evaluate real and control arms
Probe->>JSONArtifact: write regime, storm, gate, and exploratory results
JSONArtifact->>EvaluationRecords: supply reproduced measurements and metric amendments
Possibly related PRs
Suggested reviewers: Poem
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…le-verified
The PR body asserted the figures were sound. An assertion in a PR body is not
a behaviour -- the falsifiability rule this repo already carries. So the
validation is now a committed, re-runnable mode instead of a claim.
`--selftest` (no network) checks: spearman against scipy on random and
HEAVY-TIES inputs plus exact +1/-1 anchors and the constant-input case (rho
undefined -> nan, not 0); that a narrow donor SATURATES and an own donor does
not, which is the entire mechanism GEO-DEGENERATE's verdict rests on; that
rank_codec is monotone and uses all 256 levels; and that a seam-crossing box
keeps 65 DISTINCT columns.
Each assertion disable-verified rather than assumed:
- remove the clip -> saturation check fails (0.402 vs 0.802)
- drop tie averaging -> heavy-ties rho FLIPS SIGN (+0.054 vs -0.031)
and the nan check fails
- contiguous slice for lon -> seam check fails (58 columns, not 65)
The tie-averaging break is the one worth noting: it corrupts silently rather
than crashing, and it would have moved every rho in this PR.
One expectation of mine was wrong, not the code: own-donor saturation is
0.008, not the ~0.002 I first guessed -- 1000 points over 256 levels puts ~4
in each end level. It matches CAL-RANK's measured 0.008 in the real run,
which is the same arithmetic. The bound in the test is set from the
arithmetic, not from the observed number.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
|
@coderabbitai review Requesting explicitly despite draft status — unlike the last two PRs this one carries real code and real measured numbers, so a review has something to bite on. Already self-validated before asking (committed as
Each assertion was disable-verified: removing the clip drops saturation to 0.402; dropping tie-averaging flips the sign of the heavy-ties ρ (+0.054 vs −0.031) and fails the Where I'd value a second pair of eyes — the things a selftest cannot catch:
Explicitly not claimed anywhere and please flag if it reads otherwise: the §6.5 RMSE trend (3.96 → 1.85 → 1.14) is labelled a hint, not a result — one timestep, R4 breaks the monotone, no control on that comparison. Generated by Claude Code |
|
|
Self-review question I put to CodeRabbit and then answered by fixing rather than defending: scoring a single median-|grad p| storm is defensible for a smoke test, but the 19 fields were already in memory. Reporting one when 19 are available is the sample-composition weakness this arc has paid for twice (W6's stranded stratum, W5's subsampled control). Result is materially stronger: the C0 gate holds for EVERY storm -- both controls lose on both metrics in 19/19, not merely at the median. And the spread exposes what the single storm hid. GEO-DEGENERATE's rho ranges 0.277 -> 0.943 across storms (median 0.773): on some storms a degenerate donor is nearly adequate. It still loses everywhere, but HOW BADLY miscalibration hurts is strongly storm-dependent -- the plan's own hypothesis showing up in the control rather than in an arm. Also: the storm that is median by |grad p| is NOT median by rho (0.477 vs the true 0.773). The first pass's "representative" was unrepresentative on the axis that actually mattered, which is the general reason to report the distribution instead of a chosen row. Four other self-review questions answered mechanically, no change needed: grad_defs picks by pure argmin over max deviation with no privileged candidate; the cos(lat) direction is confirmed (cos/flat = 1.003 / 1.012 / 1.406 / 1.135 -- flat understates, most at 60 N, as claimed); the box is exactly 16.0 deg (65 points, 64 intervals); and every §6 figure matches the JSON (ratios 1.004/1.022/0.994/0.931, maxdev 0.0691, separation 6.28). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
… and its confounds GEO-DEGENERATE's rho spans 0.277..0.943 across the 19 storms, which invites the obvious question: does the miscalibration penalty shrink as the field gets stronger? That is C3's direction, but inside ONE tier and on a CONTROL arm. Measured: Spearman rho = +0.444, n=19, permutation p = 0.0578 two-sided over 200,000 permutations (null |rho| p95 = 0.456). ABOVE 0.05. Not significant. Direction matches the hypothesis, magnitude sits right at the threshold -- the exact shape of result that gets over-claimed. The obvious dismissal was checked and does NOT work either. The tautology would be "stronger gradient just means a wider box pressure range, so a fixed narrow donor covers less of it". Measured: rho vs box range = -0.035; |grad p| vs box range = +0.253; rho vs donor/box range fraction = -0.081. So the two are not proxies and the penalty does not track donor coverage. The +0.444 survives all three confounds -- neither confirmation nor easy dismissal is available on this data. Committed to the JSON as `exploratory_within_r4` WITH its status string saying it is not a result, because a number that lives only in chat is the precise defect this PR exists to record. Recording it now also means a later run cannot restate it as a confirmation that was there all along. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
…tested primitive Operator asked for a matrix rating what was tested to be good vs not calibrated enough, at product-lead / meteo-informatics level, with known effects paired side by side against discovered explanations. Built by RE-EXTRACTING from the committed artifacts (7 parallel readers over the report, four plans, the board, and 27 probe JSONs, plus one more over the three older substrate plans -- 131 primitives, 2.35M subagent tokens), never from session memory. That method is itself the point: this arc's most recent finding is that a figure cited in two documents is cited twice, not confirmed once, and an audit that compares prose to prose is a spell-check. Two scales, deliberately not merged into one score. Fitness (A proven in zone / B bounded / C comfort-zone mapped / D refuted at test point / V void) and evidence grade ([G]/[H]/[S]). A "C" is not a bad grade -- the arc's founding result is that most substrate formulas have HOMES, and Fisher-z alone carries three measured verdicts depending on the read (tail 8.3x win, level 4.7x loss, CI-frame not-a-win). A single ranking would have destroyed that finding. D and V are also different in kind: D lost its test, V means nothing could be distinguished. 56 rows across physical models, encodings, sampling geometries and statistical instruments; 14 known-vs-discovered pairs (K1-K14), four of which record the measurement going AGAINST the prior -- notably that the steering-level reference scores monotonically WORSE than the surface, and that the phi-derived stride is the worse stride at q=17. Three load-bearing cards: the wn-1 storm spine (the one durable win, R^2 0.91-0.94 from 14 values, with the 2.5x parameter-count correction that external review forced); Fisher-z as the comfort-zone archetype; and the two-regime generator rule with its measured crossover at 1.9-2.7x q. Nine apparatus lessons in section 4, each bought with a measured failure -- they are why a D in this document can be trusted as much as an A. Figures verified mechanically against the source JSONs before landing: 28 headline numbers plus the circular-resultant and comet-tail sets, 0 mismatches, and 1 rounding error found and fixed (Rayleigh p 0.689 -> 0.688). This arc has already frozen two rounding errors into ledgers; the check was not optional. Section 5 lists 13 honest gaps, including that EVERY off-diagonal cross-swap cell is still unmeasured, all ten EV probes are unrun, and five preflight rows are permanently unreproducible because their box centres were never recorded anywhere. STATUS_BOARD: D-MTX-1..5 rows added. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
|
Added in Method, because it is the point. Built by re-extracting from the committed artifacts — 7 parallel readers over the report, four plans, the board and 27 probe JSONs, plus one more over the three older substrate plans; 131 primitives, 2.35 M subagent tokens. Not from session memory. That constraint follows directly from the finding this PR already carries: a figure cited in two documents is cited twice, not confirmed once. Two scales, deliberately not merged. Fitness (A proven in zone / B bounded / C comfort-zone mapped / D refuted at test point / V void) and evidence grade ( A C is not a bad grade — the arc's founding result is that most substrate formulas have homes. Fisher-z alone carries three measured verdicts: tail read 8.3× win, level read 4.7× loss, CI frame not a win. A single ranking would have destroyed that finding. D and V are different in kind. D means tested and lost — information. V means a control matched the real arm or an upstream gate failed — no evidence either way. V is not a weak D. The tally is the honest headline:
Half the inventory is a negative result (23 of 46). That is the document working. An arc reporting only its wins would have shipped the 36-parameter compression figure, Fisher-z as the universal axis, a sign-test headline a rotated control also scored, and a stride-11 "golden" walk. Contents: 46-row master matrix across physical models / encodings / sampling geometries / statistical instruments · 14 known-vs-discovered pairs (K1–K14), four of which record the measurement going against the prior — notably that the steering-level reference scores monotonically worse than the surface, and that the φ-derived stride is the worse stride at q=17 · three load-bearing cards · nine apparatus lessons · 13 honest gaps. Figures verified mechanically before landing, per this PR's own rule: 28 headline numbers plus the circular-resultant and comet-tail sets checked against their source JSONs — 0 mismatches, and 1 rounding error found and fixed (Rayleigh p 0.689 → 0.688). This arc has already frozen two rounding errors into ledgers; the check was not optional.
Generated by Claude Code |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_ba791489-b1df-4188-8df8-1d452b5ab0a3) |
#946 was MIXED -- probe code, real measurements, two epiphanies and a synthesis document -- so it owes both obligations of the merged-PR row. The arc entry carries what a future session would otherwise re-derive: that D-CZ-0 was marked DONE with no artifact behind it and that #945's own audit compared prose to prose; the gradient definition identified FROM DATA as Pa-per-cell-without-cos(lat) and its bounded consequence (order survives, magnitudes do not); D-CZ-1's gate passing 19/19 after the sample-composition fix; the C4 amendment WITH its legitimacy scope (valid only while no C4 cell has been scored); the disable-verified selftest including the tie-averaging break that flips a sign silently; and the exploratory correlation recorded with its not-a-result status string so a later run cannot restate it as a confirmation. Also records the formula matrix: 46 rated primitives, 20 D and 3 V, i.e. half the inventory is a negative result -- with the C tier explained (a comfort zone is a map, not a grade) and the D/V distinction stated (D lost its test; V distinguished nothing). Confidence split recorded honestly: [G] on the gate, the reproduction, the definition identification and the rho-saturation measurement; [H] on the C4 amendment's scope; and the two exploratory items explicitly NOT results. PR_ARC_INVENTORY prepend suffix-verified. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
…-> 72-97%) Self-audited the #946 arc entry against the MERGED artifacts rather than against the PR body -- 17 checkable claims, 16 verified, 1 wrong. "GEO-DEGENERATE saturates 92-97%" was TRUE when first written (R1 0.9496 / R2 0.9174 / R3 0.9718 / R4 0.9179) and went stale INSIDE THE SAME PR: fixing R4 to measure each storm at its own t0 moved R4 to 0.7224. Committed reality is 72-97% per regime; across all 19 storms min 0.693 / median 0.843 / max 0.974. I carried the stale range into four files and two PR bodies. The verdict is unaffected -- 72% is still overwhelming saturation and the gate passes 19/19 -- but the number was wrong, and this is the seventh instance in this session of the same defect class: a figure true when written, stale once the artifact beneath it changed, carried forward because the check compared prose to prose. It is the first one my own audit caught rather than a reviewer, and only because the audit terminated at the JSON. Fixed in the unmerged #947 files (arc entry, LATEST_STATE) in place; for the MERGED plan the correction is APPENDED as a dated note rather than silently edited, and STATUS_BOARD's row carries the correction inline. And the corrected number is the more interesting one: the STORM regime saturates LEAST (0.72 vs 0.92-0.97 in the calmer tiers) -- the degenerate donor hurts less where the field is strongest, which is independently the direction the §6.6 exploratory correlation measured (rho = +0.444). Two measurements that were never connected agree. Recorded as a coherence, NOT as evidence: §6.6 is still p = 0.0578, above 0.05. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
First measurement since #940.
probes/weather-p1/substrate_comfort_d_cz_0_1.py+.json, ~96 MB.1. D-CZ-0 was marked DONE with no committed script or JSON
Its nine-row table was quoted in the plan, the arc entry, the
LATEST_STATErow, and three PR bodies. Nothing produced it.Why this is an epiphany and not a fix: the #945 self-audit — run specifically because this arc keeps shipping unbacked claims — reported those figures as "verified". It checked that the arc entry matched the plan. Both are prose. A figure cited in two documents is cited twice, not confirmed once. An audit must terminate at an artifact or it is a spell-check.
Reproduced — 4 of 9 rows: ratios 1.004 / 1.022 / 0.994 / 0.931. The five EXCLUDED land candidates are unreproducible; their centres were never written down. No coordinates were invented.
2. The recorded
|∇p|ignores the cos(lat) metricFour candidate definitions computed, winner decided by pure
argminover max deviation:A plain
np.gradientover the raw array — understating zonal gradients by1/cos(lat), so R3 at 60 N is ~40 % low (cos/flat = 1.003 / 1.012 / 1.406 / 1.135). Metric-corrected: 10.3 / 15.5 / 61.2 / 100.9 — order survives, range widens 9.3× → ≈9.8×. The regime axis stands; the magnitudes do not.Two defects in my own reproduction, both caught by its own guards: measuring all 19 storms at the preflight timestep rather than each storm's own
t0(inverted R3/R4, would have read as C1 failing); and a seam assertion firing at 353.4 E, where a plain longitude slice gives 58 columns instead of 65.3. D-CZ-1 — the C0 gate PASSES, on all 19 storms
Both controls lose to both real arms, on both metrics, in every regime — and in 19/19 storms, not merely at the median.
GEO-DEGENERATEsaturates 92–97 %, so the mechanism is visible rather than assumed. C1b:separation= 6.28 vs the ≥ 3 bar.R4 across all 19 (the first pass scored one median storm; the fields were already in memory, so reporting one was the sample-composition weakness this arc has paid for twice):
CAL-ABSCAL-RANKCAL-SHUFFLEGEO-DEGENERATEThe storm that is median by
|∇p|is not median by ρ (0.477 vs 0.773) — the first pass's "representative" was unrepresentative on the axis that mattered.4. ⚠ The run AMENDS C4 — ρ is saturated on the diagonal
Real-arm ρ spread: 3×10⁻⁶ … 4.7×10⁻⁵. ρ cannot separate
CAL-ABSfromCAL-RANK, which is exactly what C4 compares. C4 could not have fired.ρ is excellent where the plan needs it (real vs degraded spans 0.99999 → 0.28), so:
Lkeeps ρ; C4 moves to RMSE in Pa with ρ as a floor check. Legitimate because D-CZ-1's purpose is to test the apparatus before the expensive cells and no C4 cell has been scored — illegitimate the instant one has. Amendment carries its trigger and was propagated to the header and the C4 bar, the failure a reviewer caught on #944.5. Exploratory — and it survives its own obvious dismissal
Does the miscalibration penalty shrink with field strength, within R4? ρ(
GEO-DEGENERATEρ, storm|∇p|) = +0.444, n = 19, permutation p = 0.0578 (200 000 perms, two-sided). Above 0.05 — not significant.The tautology check — "stronger gradient just means a wider box range, so a narrow donor covers less" — does not hold: ρ vs box range −0.035,
|∇p|vs box range +0.253, ρ vs donor/box fraction −0.081. So neither confirmation nor easy dismissal is available on this data.Committed to the JSON as
exploratory_within_r4with a status string saying it is not a result, so a later run cannot restate it as a confirmation that was there all along.Separately, §6.5 keeps the RMSE trend (3.96 → 1.85 → 1.14) labelled a hint: one timestep, R4 breaks the monotone, no control on that comparison.
6. Committed selftest, disable-verified
python3 substrate_comfort_d_cz_0_1.py --selftest(no network). ρ vs scipy on random + heavy-ties inputs, exact ±1 anchors,nanon constant input; saturation fires (0.802 narrow vs 0.008 own);rank_codecmonotone over all 256 levels; seam box keeps 65 distinct columns.Each assertion broken on purpose to confirm it can fail: removing the clip → saturation 0.402; dropping tie-averaging → heavy-ties ρ flips sign (+0.054 vs −0.031); contiguous slice → 58 columns. The tie-averaging one corrupts silently rather than crashing.
Board
EPIPHANIES.md—E-A-FIGURE-CITED-TWICE-IS-NOT-CONFIRMED-ONCE-1,E-THE-METRIC-THAT-SEPARATES-ONE-COMPARISON-IS-BLIND-TO-ANOTHER-1(prepend, suffix-verified).STATUS_BOARD.mdD-CZ-0 / D-CZ-1. Plan §6 RUN section.🤖 Generated with Claude Code
https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
Summary by CodeRabbit
New Features
Documentation