D-WXS-7/D-WXS-8 RUN: the real gate — 12/12 PASS, and a real mixed result on the other - #950
Merged
Merged
Conversation
…ur times The session repeatedly reported the gate as sitting behind D-WXS-4 (the bake) and therefore behind D-WXS-0 (the classid mint). Checked against bar B6 rather than against my own previous sentence, that is wrong. B6 needs real ERA5 values at grid scale, the SHIPPED floor.rs quantiser, a shuffled decode table as the control, a 16/64/256 resolution ladder, and a Spearman. Every one of those exists. It needs no Lance dataset, no NodeRow, and no classid -- because it measures the CODEC, and the codec shipped with D-WXS-3. The mis-read, named: the deliverable line says "representation fidelity on the substrate ... over pairs sampled from the whole grid". "On the substrate" and "at grid scale" describe the SCALE and the SOURCE -- the whole real field through the shipped codec, rather than four hand-picked boxes through a numpy re-derivation. They do not say "persisted to Lance". I read the phrase as implying the bake and then repeated the conclusion without returning to the bar. This is the session's own recurring defect, committed by me, about my own plan: a claim that was plausible, load-bearing, never verified against the artifact, and propagated by repetition. Same shape as the stale saturation figure, the three vacuous disable probes, and the "no external review" statement -- all of which this session also caught. The countermeasure is already on the board: an audit must terminate at an artifact, and a BLOCKER is an artifact-checkable claim like any other. I had been treating blockers as background facts rather than as claims. What IS blocked, precisely: D-WXS-4 and D-WXS-5 write rows and a row needs a routable classid; D-WXS-6 needs datasets to read; D-WXS-9/10 need the neighbour read the bake provides. D-WXS-7 and D-WXS-8 need none of it. Consequence: the gate everything else hangs on has been runnable since D-WXS-3 landed. Re-scoped from Blocked to READY on the board, with the reason recorded in plan section W3 so the next session does not inherit the wrong blocker. Section 4's "W0 -> W1 -> W2 -> W3" is a dependency ordering for the BAKE deliverables, not a licence to treat W3 as unreachable. Doc-only; no bar changed, no verdict claimed, nothing run yet. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
…d result on the other The gate this arc had reported as blocked behind the classid mint four times turned out to need none of it (previous commit). Once corrected, it ran the same day, against REAL live-fetched ARCO-ERA5 grid data, 3 real calendar seasons, computed with jc::reliability::spearman as bar B6 literally requires -- not a Python re-implementation. Method. fidelity_probe_fetch.py extends the proven fetch.py pattern (real HTTPS, real Zarr v2 decode, 404=all-NaN=valid-missing-chunk semantics) to grid scale. The 3 seasons were NOT assumed -- a first attempt at 4 fixed 2021 calendar anchors found only the SAME 3 variables present at every one of them (2m_temperature, 2m_dewpoint_temperature, 10m_u_component_of_wind); 10m_v_component_of_wind / mean_sea_level_pressure / surface_pressure / total_column_water_vapour / total_cloud_cover / sea_surface_temperature were absent at all 4, confirming the README's "sparse by design" finding is not a one-timestep artifact. 10m_wind_speed does not exist in this store at all, confirming the pre-existing weather-normalized-substrate.md finding. A follow-up live HEAD-sweep of 24 candidate timesteps across the whole 1959-2023 archive found winter/spring/summer, each independently meeting bar B7's >=4-variable />=2-unit floor with genuinely present, fully-finite fields. sea_surface_temperature came back PHYSICALLY MASKED (686,364 of 1,038,240 finite -- land is NaN by definition) and was correctly excluded as not-fully-usable rather than force-included. fidelity_probe_prep.py quantises via the exact floor.rs formula (re-expressed in Python for the same reason floor.rs itself is zero-dep -- this stage fetches over HTTP, which the crate deliberately cannot do) and writes 200,000-pair raw (truth, code) f64 arrays per comparison, matching p2_probe.py's own N. crates/weather-poc/examples/fidelity_probe.rs reads them and computes every rho with jc::reliability::spearman, added as a DEV-DEPENDENCY ONLY (Cargo.toml explains why that is safe here and was not safe for helix: dev-only, and jc is itself zero-external-dep by default). Sanity-checked before trusting a number: the shuffled-decode arm is a genuinely different array from the unshuffled one (range [0,4.92] vs [0,202], not a copy), and its rho collapses to 0.02-0.024 -- the pipeline demonstrably discriminates. D-WXS-7 (bar B6): 12/12 PASS, all 3 seasons. rho256 = 0.999909 / 0.999895 / 0.999684. This DOES NOT REPLICATE the plan's own cited near-miss (K x K = 0.999556, below bar, from p2_probe.py's smaller fixture) -- the earlier number is not retracted, it was real on a real smaller fixture, but it no longer licenses "the substrate's fidelity is marginal" at grid scale. D-WXS-8 (bar B7): mixed, reported in full, nothing filtered or softened. Control (per-variable floor must LOSE, the KILL-gated clause) 16/16 PASS -- rho_pervar 0.245-0.939 vs rho_shared 0.9987-0.9999 on identical pairs. The KILL does not fire; the shared-floor design is not refuted. Primary (rho_shared >= 0.9996 exactly) 10/16 PASS, 6 close misses (0.9987-0.9996, skewed toward wind/pressure pairs). Stay-silent twin: the diff<=0.0001 half passes at spring/summer, fails at winter (0.000174); the ZERO EMPTY BUCKETS half fails at ALL THREE seasons (38/39/45 of 256) -- the literal "zero" from the 1-timestep/3-variable fixture this claim was carried from does not hold once the shared floor pools a wider real multi-unit window at grid scale. The reusable shape, recorded on the board: a KILL clause and a primary/stay-silent bar are not the same severity, and collapsing them into one verdict sentence loses the finding. This run had three distinct tiers on ONE deliverable -- a KILL that did not fire (nothing retracted), a strict numeric bar that partly missed (a real, reported gap), and an earlier measurement that did not replicate at the new scale (a correction, not a failure). Every pre-registered verdict applied literally; every one printed, including the FAILs. Data discipline: the 138 MB of raw pair-distance f64 arrays and the store's 48 KB .zmetadata snapshot are gitignored, matching the repo's existing rule for probes/weather-p1/fixture/*.npy (same rationale: re-fetch deterministic from a committed script, do not carry regeneratable bytes). Committed: both Python stages, the Rust example, the small manifest/meta.txt sidecars, and the two measured-results JSONs (the actual receipts). A stale fixture/season_autumn/ directory, leftover from the first (unsuccessful) 4-fixed-season fetch attempt, was deleted rather than left orphaned. Gates: cargo build --example fidelity_probe clean; cargo test 35/35; fmt clean; jc's own pre-existing warnings (not touched) are the sibling crate's, not this crate's. Python syntax-checked. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (14)
📝 WalkthroughWalkthroughThe PR adds a three-season ARCO-ERA5 fidelity probe. Python scripts fetch and prepare weather arrays. A Rust example computes Spearman-based B6/B7 verdicts. Fixtures and planning records capture passing controls and threshold failures. ChangesWeather fidelity validation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant FetchProbe
participant ARCOERA5
participant PrepProbe
participant RustFidelityProbe
participant ResultsFixture
FetchProbe->>ARCOERA5: Fetch seasonal Zarr chunks
ARCOERA5-->>FetchProbe: Return decoded values or missing-chunk fills
FetchProbe->>PrepProbe: Provide usable seasonal arrays
PrepProbe->>PrepProbe: Standardize and quantize fields
PrepProbe->>RustFidelityProbe: Write pair-distance arrays and metadata
RustFidelityProbe->>RustFidelityProbe: Compute Spearman verdicts
RustFidelityProbe->>ResultsFixture: Write seasonal results
Possibly related PRs
Suggested reviewers: Poem
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_84a2c2b3-cb12-42a3-9a6a-4808f1643b84) |
AdaWorldAPI
marked this pull request as ready for review
August 13, 2026 12:06
AdaWorldAPI
pushed a commit
that referenced
this pull request
Aug 13, 2026
…d 16/16
Found while re-counting the committed JSON to write the product-lead update.
Both denominators in D-WXS-8's bar-B7 report were wrong.
written actual
B7 control (per-var must LOSE) 16/16 19/19
B7 primary (rho >= 0.9996) 10/16 (63%) 9/19 (47%)
per season (primary) -- winter 2/9, spring 4/5, summer 3/5
There are 19 cross-unit pairs, not 16 (winter 9 + spring 5 + summer 5); the
same-unit pairs (KxK, m/s x m/s) are informational and correctly excluded
from the bar, and I dropped them from the denominator inconsistently.
The direction matters, not just the arithmetic. 10/16 reads as "mostly
passing, six close misses". 9/19 reads as "the strict bar fails on a MAJORITY
of cross-unit pairs". Both describe the same measurements -- every per-pair
rho was carried correctly -- but only the second is what the data says. The
KILL-gated control is unaffected and in fact better at 19/19, so NO conclusion
reverses; what changes is that the primary bar's shortfall was under-reported.
The failures also concentrate in winter (2/9), the only season carrying
mean_sea_level_pressure -- visible in the corrected per-season split, invisible
in the aggregate I first wrote.
The diagnosis is narrower than "check your figures". Every number the program
COMPUTED AND PRINTED -- 56 verdicts, 42 pass / 14 fail, every individual rho --
was carried into the writeup correctly. The only two wrong numbers are the ones
I produced by counting rows in terminal output by eye. The pipeline was sound;
the tally was not.
A figure you tallied yourself is a DERIVED figure, and a derived figure needs
the artifact exactly as much as a measured one does. "The audit must terminate
at an artifact" is usually applied to CITING a number. This is its narrower and
easier-to-miss form: PRODUCING a number by counting, summing or eyeballing a
listing is itself an unverified derivation, even when every input to it is
verified.
Mechanical fix now in use: any count, rate or ratio that appears in a writeup
is computed from the committed artifact in the same command that prints it --
never transcribed from a previous run's stdout, never tallied by reading a list.
Reach: the error went to main via PR #950 in four places -- the plan's W3 RUN
section, the STATUS_BOARD D-WXS-8 row, the EPIPHANIES entry, and the PR body.
The first two are corrected in place (not append-only ledgers). EPIPHANIES gets
a PREPENDED correction entry per the storno rule, leaving the wrong entry
standing beneath it. The merged PR body cannot be corrected -- #950's body
carries the superseded figures and the new entry is authoritative.
Board: EPIPHANIES prepend E-A-FIGURE-YOU-TALLIED-YOURSELF-IS-A-DERIVED-FIGURE-1
(suffix-verified); plan W3 RUN carries the same correction note inline.
Doc-only. No measurement re-run, no bar changed, no code touched.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #949 (merged). Branch restarted from
main.The gate this arc had reported as blocked behind the classid mint — four times, never checked — turned out to need none of it (previous commit,
8977a11e). Once corrected, it ran the same day, against real live-fetched ARCO-ERA5 grid data, 3 real calendar seasons, computed withjc::reliability::spearmanexactly as bar B6 requires — not a Python re-implementation.Method
fidelity_probe_fetch.pyextends the provenfetch.pypattern (real HTTPS, real Zarr v2 decode, 404=all-NaN=valid-missing-chunk semantics) to grid scale. The 3 seasons were not assumed — a first attempt at 4 fixed 2021 calendar anchors found only the SAME 3 variables present at every one of them; six other candidates were absent at all 4, confirming the README's "sparse by design" finding is not a one-timestep artifact.10m_wind_speeddoes not exist in this store at all, confirming a pre-existing finding. A follow-up live HEAD-sweep of 24 candidate timesteps across the whole 1959–2023 archive found winter/spring/summer, each independently meeting bar B7's ≥4-variable/≥2-unit floor with genuinely present, fully-finite fields.sea_surface_temperaturecame back physically masked (686,364/1,038,240 finite — land is NaN by definition) and was correctly excluded rather than force-included.fidelity_probe_prep.pyquantises via the exactfloor.rsformula and writes 200,000-pair raw(truth, code)f64 arrays per comparison.crates/weather-poc/examples/fidelity_probe.rsreads them and computes every ρ withjc::reliability::spearman,jcadded as a dev-dependency only (Cargo.tomlexplains why that's safe here, unlikehelix). Sanity-checked before trusting a number: the shuffled-decode arm is a genuinely different array from the unshuffled one, and its ρ collapses to 0.02–0.024 — the pipeline demonstrably discriminates.D-WXS-7(bar B6): 12/12 PASS, all 3 seasonsThis does not replicate the plan's own cited near-miss (K×K = 0.999556, below bar, from
p2_probe.py's smaller fixture). The earlier number isn't retracted — it was real, on a real smaller fixture — but it no longer licenses "the substrate's fidelity is marginal" at grid scale.D-WXS-8(bar B7): mixed, reported in full, nothing softened|diff| ≤ 0.0001passes at spring/summer, fails at winter (0.000174). Zero-empty-buckets fails at all 3 seasons (38/39/45 of 256) — the literal "zero" carried from a 1-timestep/3-variable fixture does not hold once the shared floor pools a wider real multi-unit window at grid scale.The reusable shape
A KILL clause and a primary/stay-silent bar are not the same severity, and collapsing them into one verdict sentence loses the finding. This run had three distinct tiers on ONE deliverable: a KILL that did not fire (nothing retracted), a strict numeric bar that partly missed (a real, reported gap), and an earlier measurement that did not replicate at the new scale (a correction, not a failure). Every pre-registered verdict applied literally; every one printed, including the FAILs.
Data discipline
138 MB of raw pair-distance f64 arrays + the store's 48 KB
.zmetadatasnapshot are gitignored, matching the repo's existing rule forfixture/*.npy— re-fetch deterministic from a committed script, don't carry regeneratable bytes. Committed: both Python stages, the Rust example, the small manifest/meta.txt sidecars, and the two measured-results JSONs (the actual receipts). A stalefixture/season_autumn/from the first unsuccessful fetch attempt was deleted rather than left orphaned.Gates:
cargo build --example fidelity_probeclean ·cargo test35/35 · fmt clean · Python syntax-checked.jc's own pre-existing warnings are the sibling crate's, untouched by this PR.Still open
D-WXS-0(classid mint, OGAR-side, operator-gated) remains the sole blocker for the bake (D-WXS-4/D-WXS-5) and everything downstream of it.D-WXS-9/D-WXS-10(ζ) need the bake's neighbour read. Not run this session: the resolution ladder for cross-unit pairs (only computed for K×K); a 4th+ season; the wind/pressure-skew pattern in the primary FAILs was observed, not tested as a hypothesis.🤖 Generated with Claude Code
https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi
Generated by Claude Code
Summary by CodeRabbit
New Features
Documentation