diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index e47c6fa5a..e26ee347d 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,7 +1,163 @@ +## 2026-08-11 — E-THE-DOCTRINE-DOC-EXISTED-AND-I-NEVER-READ-IT-1 + +**Status:** FINDING `[G]` (operator-directed; authority is +`.claude/knowledge/helix-cartesian-vs-fisher2z.md`, in this repo, with a +`READ BY:` header naming this exact kind of session). + +**Corrects E-A-CORRECTION-IS-A-CLAIM-AND-CARRIES-A-CLAIM-S-BURDEN-1 (below) — +the third link in that same chain, and the entry it corrects is its own rule +firing again.** The `Pair48` successor shape that entry recommends is +**WITHDRAWN, not deferred.** Full regrade: +`.claude/knowledge/weather-normalized-substrate.md` §12.12. + +**The doctrine doc for orientation codecs exists in `.claude/knowledge/`, its +`READ BY:` header says *"ANY session that encodes/decodes/renders an +orientation, normal, or direction"*, and the entire helix arc never opened it** — +not before the design, not during the envelope audit, not through four +successive corrections. Reading the primary source (`crates/helix/src/*.rs`) was +mistaken for reading the doctrine. `residue.rs` states the byte layout +truthfully; it does **not** say which field is metric and which is render, nor +that the sign already completes the sphere. That is precisely what the knowledge +doc carries, and what a `READ BY:` header is *for*. `CLAUDE.md` § *Consult, +don't guess* already orders it: card → **knowledge doc** → board → *then* source. + +**What the doc settles, against the audit's inferences (the FACTS all stood; the +INFERENCES were wrong):** + +1. **`Signed360` alone is a COMPLETE full-sphere direction** — *"you do NOT need + a second 'pos' helix to complete it"* (`helix-cartesian-vs-fisher2z.md:77-82`); + the sign partition is what makes it whole. Hence no pair lane, hence the + withdrawal. +2. **`ResidueEdge` / `rim` is the METRIC carrier, not a render input** — *"never + run the rim's atanh/tanh to recover a direction"* (`:83-86`). Direction lives + in `(polar, azimuth)`. The operator said this first: *"ResidueEdge is + turbovec, nobody was asking you to use turbovec."* +3. **The crate's *"no free 2-DOF direction codec"* means no *helper*, not no + *capacity*.** `azimuth` is *"`n·φ mod 2π` mapped to `[0, 65536)` over the + full 360°"* (`residue.rs:85`). What is genuinely absent is `from_normal`; + encoding is a **nearest spherical-Fibonacci search** with a worked reference + pair (`:90-93`, q2 `scratch-fma/helixbake`). +4. **The metric hazard I "found" is crate-documented, not a discovery** — + `distance_heuristic` names its own failure mode as *"the raw-azimuth 2π + wrap"* and forbids it for CAKES bounds (`residue.rs:52-59`). The real shape + is a **split, not a defect**: rim = L1-metric (`DistanceLut`, triangle + inequality); azimuth = circular render carrier, deliberately not L1. My error + was routing a bearing through the *metric* field, then reporting the + consequence as a property of the codec. + +**Also settled — the name.** `helix360` **does not exist and never did** +`[G-absence]`: pickaxe over full history (all refs + unreachable objects) returns +12 blobs, all authored by that session, **zero deletions, zero blast radius**; +ndarray and the `lance-graph2` backup carry none. The symbol is **`Signed360`**. +A hunt for a deleted artifact that was only ever a mis-remembered name. + +**What still stands from the entry below:** the no-per-value-lane-reading-selector +finding (`[G-absence]`, 15 hits / zero writers / zero decoders), the +`Signed360::sign()` all-zero dormant-lane defect (still filed, still unfixed), +and — sharpened by this entry — its own **rule**. + +**Rule (unchanged, now three links deep):** a correction is a claim and carries a +claim's burden. **Extension:** before correcting a domain claim, check whether +the domain already has a `READ BY:` doc. If it does, that doc is a **mandatory +read, not a suggestion** — four corrections written without it were four +corrections written blind. + +## 2026-08-11 — E-THE-TRANSFORM-MUST-MATCH-THE-DISTRIBUTION-SHAPE-1 + +**Status:** FINDING `[H]` (measured on real ERA5, one timestep — re-runnable at +`probes/weather-p1/`; needs a second variable + season before `[G]`). + +**Fisher-Z DEGRADES a weather-anomaly palette.** Real ERA5 `2m_temperature`, +1,038,240 gridpoints, both paths into the same 256 buckets over the same +0.4–99.6 percentile window: **linear MAE 0.0684 K, 0 empty buckets, 115.7 +effective buckets** vs **Fisher-Z MAE 0.2168 K, 76 empty buckets, 28.1 effective +buckets** — 3.2× worse error, 228 of 256 buckets burned. + +**Mechanism, measured on this sample:** `arctanh` is ≈identity near 0 and +explodes near ±1, so it moves resolution *toward the bounds*. The tested ERA5 +`2m_temperature` anomaly (one timestep, 2021-06-15 12:00 UTC) has 77 % of mass +inside |s|<0.25 (**excess** kurtosis +3.30, mass in the MIDDLE); a +correlation-like control `tanh(N(0,1.5))` has 32.7 % beyond 0.9 (mass at the +BOUNDS). **CONJECTURE `[S]`, generalizing beyond the tested sample: the +transform must match the input distribution's shape.** The *measured* claim is +narrower — for THIS field at THIS timestep, Fisher-Z costs address economy. +Promotion to a general rule needs the second variable and season named below. + +**Terms:** *effective buckets* = `exp(Shannon entropy)` of the 256-bin occupancy +histogram (how many addresses actually carry data); *drift score* = `max|occ−μ|/σ` +under a multinomial null, exactly as `crates/helix/src/quantize.rs:119-146`. Fisher-Z is right for correlation-like inputs and for +helix's own `r = √u` (equal-area placement concentrates toward the rim BY +CONSTRUCTION) — nothing about the helix crate is impugned. What is falsified is +generalizing its transform to bell-shaped geophysical fields, which is exactly +what `weather-normalized-substrate.md` §1.2 did. "One shared transform ⇒ one +comparable substrate" does not survive; the shared palette + LUT may. + +Also measured: ARCO-ERA5 has **no `wind_speed` variable** (exhaustive, all 52 +arrays), so §6.5's Jensen-gap measurement cannot have come from the Phase-A +store; blosc is variable-dependent — **1.794×** (`2m_temperature`), **1.771×** +(`2m_dewpoint_temperature`), **1.248×** (`10m_u_component_of_wind`), each one +chunk `[1,721,1440]` at t=547476, blosc/lz4 as stored — not one 1.27×; +the BF16 anomaly gain reproduces in DIRECTION only (74.9× under a zonal-mean +climatology proxy vs the doc's unreproduced 97×). + +## 2026-08-11 — E-A-CORRECTION-IS-A-CLAIM-AND-CARRIES-A-CLAIM-S-BURDEN-1 + +**Status:** FINDING `[G]` on the code facts and on its own rule; **its `Pair48` +successor recommendation is WITHDRAWN and several of its INFERENCES are +regraded** — see `E-THE-DOCTRINE-DOC-EXISTED-AND-I-NEVER-READ-IT-1` (top of +file) and `weather-normalized-substrate.md` §12.12. Read that first. + +**A correction written in the same breath as the error it replaces has had no +independent gate — and one of ours was wrong.** The helix arc's ledger entry C2 +recorded rejecting a 12-byte `FacetSchema::Pair48` for wind from/to, on the +ground that "the 6 B `HelixResidue` lane is already 2×24 in/out by +construction", citing `canonical_node.rs:963`. That line is a **comment** +stating a **width identity (48 = 2×24) plus coverage**, never a structural +in/out claim. **The rejected 12-byte `Pair48` was the structurally correct +answer** — C2 replaced a right answer with a wrong one and banked the wrong one +as a lesson, i.e. the failure mode compounding inside the ledger built to catch +it. + +The audit falsified the whole premise: `Signed360` is ONE signed orientation +(`rim(3)+polar(1)+azimuth(2)`, `residue.rs:76-116`); `ResidueEdge` **cannot carry +a hemisphere sign at all** (`sprite_replay.rs:56-63` — `Sign::Neg` reconstructs +as `Sign::Pos`, *"a real, measurable structural error"*); it has **no azimuth**, +the only angular field in the lane; and the crate states twice, unprompted, that +**no 2-DOF direction codec exists** (`sprite_replay.rs:47`, +`continuous_field.rs:31-43`) and that inventing one is the "invented round-trip +API" it warns against. Also: `end_idx` is monotone in `n` (`residue.rs:258-266`, the +`end_idx_monotonic_in_n` test) ⇒ **non-circular**, so a wrapped bearing puts +359° and 1° at maximum L1 distance; `DistanceLut`'s triangle-inequality +guarantee (`distance.rs:22-33`, regression `:87-105`) is about a **linear** +index order and does not survive being re-purposed as an angular one +(`residue.rs:53-55` names this failure mode for `distance_heuristic`). + +**Structural finding worth keeping regardless of weather:** there is **no +per-value-lane reading selector in the contract at all** — `ReadMode` has three +axes (tail / value_schema / edge_codec), none of which selects a *reading* of a +value lane; `EdgeCodecFlavor` covers `EdgeBlock`, a different region. Exhaustive +grep (`HelixResidue`, all `**/*.rs`, unbounded): 15 hits, **zero writers, zero +decoders**. Correct successor shape if a pair is needed: a NEW 16-byte facet +lane read as `Pair48 = [Signed360; 2]`, codec in `crates/helix`, lane in the +zero-dep contract, discriminant **16** — `15` is reserved for `BoardAggregates`. + +**Pre-existing defect found in passing:** `Signed360::sign()` decodes an +all-zero, never-written lane as a definite `Sign::Neg` (`residue.rs:109-115`), +where sibling lanes `Tekamolo` / `CausalWitness` explicitly define all-zero as +*unaddressed* / *unbound*. Violates the CANON zero-fallback ladder. Filed, not +fixed. + +**Rule:** route corrections through the same gate as originals. "I was wrong +before" is not evidence the new version is right. + ## 2026-08-11 — E-HELIX360-IS-THE-NORMALIZED-SUBSTRATE-NOT-A-BIT-BUDGET-1 **Status:** FINDING `[G]` for every code fact (file:line in the doc); the weather -mapping is `[H]`; the floor policy is `[S]` pending probes. +mapping is `[H]`; the floor policy is `[S]` pending probes. **⊘ Two corrections: +"helix360" is this session's coinage — the symbol is `Signed360`, no `helix360` +exists in any repo `[G-absence]`; and the "2×24 = from AND to" reading below is +FALSE — one `Signed360` is already a complete full-sphere direction. See +`E-THE-DOCTRINE-DOC-EXISTED-AND-I-NEVER-READ-IT-1` (top of file).** **The finding, in one line:** helix360 is a Fisher-2z-hydratable golden-spiral projection — two 24-bit equal-area hemispheres glued at the equator (Poincaré diff --git a/.claude/knowledge/weather-normalized-substrate.md b/.claude/knowledge/weather-normalized-substrate.md index 9ae226baf..5d8685d71 100644 --- a/.claude/knowledge/weather-normalized-substrate.md +++ b/.claude/knowledge/weather-normalized-substrate.md @@ -8,6 +8,24 @@ > the correction ledger is part of the document, not an apology appendix). > **Companion plan:** `.claude/plans/weather-substrate-poc-v2.md` (PR #915). > +> ## ⊘ READ §12 FIRST (2026-08-11, same day as authorship) +> +> Probe **P1 ran** and an **envelope audit ran**, and between them they +> **falsified four claims in this document** — including one where a §11 +> "correction" was itself the error. Affected: **§1.2** (Fisher-Z doctrine), +> **§2.3 + §5 + §6.6** (helix360 as a wind-bearing carrier), **§11-C2** (the +> `Pair48` rejection), **§6.1/§6.5** (measurement provenance). Every one of +> those sections is superseded by **§12**. The sections are left standing, +> not deleted, so the correction is legible. +> +> **Then §12.2 itself was regraded by §12.12** (operator-directed, same day), +> once the canonical doctrine doc — `.claude/knowledge/helix-cartesian-vs-fisher2z.md`, +> which this arc never opened — was finally read. Two things a reader must not +> act on without §12.12: **"helix360" is this session's coinage for `Signed360`** +> (no such symbol exists in any repo), and the **`Pair48` mint is WITHDRAWN, not +> deferred** — one 6-byte `Signed360` is already a complete full-sphere +> direction. Read **§12.12 before §12.2**. +> > **Grading (mandatory on every claim):** > - `[G]` — verified in committed code this session, with `file:line`. > - `[G-absence]` — verified NOT to exist (grep/read, stated scope). @@ -511,3 +529,558 @@ contract (lance-graph-contract/src/canonical_node.rs) └─ ValueTenant::HelixResidue = 4 → ColumnKind::U8 × 6 @ row_offset 112 in ValueSchema::{Compressed, Full}; node row = key(16)|edges(16)|value(480) ``` + +--- + +## 12. ⊘ CORRECTIONS — what P1 and the envelope audit falsified (2026-08-11) + +Written the same day as the rest of this document, by the two gates the +document itself queued. Both fired. Sections above are **left standing and +regraded here**, never silently edited. + +### 12.1 — §1.2 FALSIFIED for weather scalars: Fisher-Z *degrades* the palette + +> **⊘⊘ PARTIALLY REVERSED by §12.10** — this section was scored with a +> **round-trip** metric, which is the wrong criterion for a one-way address over +> a retained original. Re-measured against noise-floor thresholds the two paths +> are indistinguishable at a 0.5–1 K floor. What survives is *bucket economy*, +> not a validity failure. Read §12.10 before citing anything below. + +Measured, real ERA5 `2m_temperature`, 1,038,240 gridpoints, both paths into the +same 256 buckets over the same 0.4–99.6 percentile window +(`probes/weather-p1/`, re-runnable): + +| path | MAE (K) | p99 (K) | empty buckets | effective buckets | drift score | +|---|---|---|---|---|---| +| **linear** (no transform) | **0.0684** | 0.0939 | **0** | **115.7 / 256** | 607 | +| Fisher-Z then linear | 0.2168 | 0.4253 | **76** | **28.1 / 256** | 2240 | + +Fisher-Z is **3.2× worse** and burns **228 of 256 buckets**. + +**Mechanism, measured not assumed:** `arctanh` is ≈identity near 0 and explodes +near ±1, so it moves resolution *toward the bounds*. ERA5 temperature anomaly +over a robust scale has mean |s| = 0.172, **77 % of mass inside |s| < 0.25**, +1.2 % beyond 0.9, kurtosis +3.3 — mass in the **middle**. A correlation-like +control (`tanh(N(0,1.5))`) has 13.5 % inside 0.25 and **32.7 % beyond 0.9** — +mass at the **bounds**. Fisher-Z is right for the second shape and wrong for the +first. + +**So §1.2's "this is what makes 8 bits sufficient" is false for weather +scalars.** The correct general statement: **the transform must match the input +distribution's shape.** Fisher-Z earns its keep on correlation-like inputs and +on helix's own rim radius `r = √u`, which equal-area placement concentrates +toward `r → 1` **by construction** (`placement.rs:147-151`) — i.e. helix uses it +on exactly the shape it suits. Nothing about the helix crate is impugned; what +is falsified is *generalizing its transform to bell-shaped geophysical fields*. + +**Product consequence:** "Fisher-Z everything into one comparable substrate" +does not survive. **P2 is re-scoped**, not merely un-run: it must now test +cross-variable comparability under a *per-shape* transform choice, and prove +the shared **palette + LUT** still hold once the shared **transform** is gone. + +### 12.2 — §2.3 / §5 / §6.6 FALSIFIED: helix360 is not a wind-bearing carrier + +> **⊘⊘ GROUND NARROWED by §12.10** — the structural facts below stand, but the +> *"no inverse / no decode exists"* argument was never a defect under a one-way +> address frame. The `Pair48` successor mint returns to the table strengthened. + +The `v3-envelope-auditor` verdict on reading the 6-byte `HelixResidue` lane as +`2 × ResidueEdge` (in/out): **LAYOUT-GATED, and the gate does not exist** — with +the premise itself falsified: + +1. **`Signed360` is ONE signed orientation**, `rim(3) + polar(1) + azimuth(2)` + (`residue.rs:76-116`) — not two hemisphere codes. The "2× the 24-bit + hemisphere" line (`canonical_node.rs:838-839`, `placement.rs:170-171`) is a + **width identity (48 = 2×24) plus a coverage claim**, never a structural + in/out claim. §2.3's `[G]` on that reading was **unearned**; regrade `[S]`. +2. **`ResidueEdge` cannot carry a hemisphere sign at all.** `sprite_replay.rs:56-63`: + sprites seeded `Sign::Neg` reconstruct through the `ResidueEdge` path as + `Sign::Pos` — *"a real, measurable structural error, not a rounding + artifact."* An in/out pair of `ResidueEdge` would silently force both to the + upper hemisphere. +3. **`ResidueEdge` has no azimuth.** The 16-bit `azimuth` is the **only** angular + field in the entire lane; the proposed reading deletes it, and spends 2 of 6 + bytes on duplicate `floor_version` stamps. +4. **Neither reading is a bearing codec.** The crate says so twice, unprompted: + *"there is no free 2-DOF direction codec in this crate"* (`sprite_replay.rs:47`) + and *"NOT a generic scalar quantizer … doing so would be exactly the + 'invented round-trip API'"* (`continuous_field.rs:31-43`). My proposal was + that invented round-trip. +5. **Metric hazard if forced:** `end_idx` is monotone, i.e. **non-circular**, so + a wrapped bearing puts 359° and 1° at maximum L1 distance. `DistanceLut`'s + triangle-inequality guarantee is about a **linear** index order and does not + survive repurposing as an angular one (`residue.rs:53-55`, `distance.rs:22-33`). +6. **No selector exists.** Exhaustive grep (`HelixResidue`, all `**/*.rs`, + `head_limit: 0`): 15 hits, **zero writers, zero decoders**. `ReadMode` has + exactly three axes (tail / value_schema / edge_codec) — none selects a + value-lane *reading*; `ValueSchema` selects presence only; `EdgeCodecFlavor` + covers `EdgeBlock`, a different region. There is **no per-value-lane reading + selector in the contract at all** — that absence is the finding. + +**Candidate shape `[S]` if a from/to pair is genuinely needed** (a CONJECTURE, not +a promoted design — it follows from the envelope audit in this section and is +gated on the operator mint decision tracked in §12.6): a **new 16-byte facet +lane** read as `FacetSchema::Pair48` = `[Signed360; 2]` — `classid(4) + 6 B in + +6 B out` — precedent set twice by `Tekamolo` and `CausalWitness` (both 16-byte +lanes appended at the end, no version bump). Codec owner `crates/helix` (next to +`encode_signed`, so the #498 sign-partition is never re-implemented); lane owner +`lance-graph-contract` (zero-dep, reserves bytes only). **⚠ discriminant `15` is +reserved for `BoardAggregates`** (`canonical_node.rs:1027-1034`) — a new tenant +takes `16`, never silently consumes 15. + +### 12.3 — §11-C2 was itself a mis-correction (the sharpest lesson here) + +C2 recorded rejecting a 12-byte `Pair48` for from/to because "the 6 B lane is +already 2×24 in/out by construction", citing `canonical_node.rs:963`. That line +is a **comment**, and it states a width identity, not a structure. **The 12-byte +`Pair48` I rejected was the structurally correct answer** (§12.2). C2 replaced a +right answer with a wrong one *and then banked the wrong one as a lesson* — the +failure mode compounding one level up, inside the ledger built to catch it. + +**Rule extracted:** a correction is a claim and carries a claim's burden of +proof. "I was wrong before" is not evidence that the new version is right, and a +correction written in the same breath as the error it replaces has had no +independent gate. Route corrections through the same audit as originals. + +### 12.4 — Measurement provenance (§6.1, §6.3, §6.5) + +- **§6.5 wind-speed non-derivability — provenance broken.** ARCO-ERA5 has **no + `wind_speed` variable** (exhaustive: all 52 arrays enumerated; wind-related are + exactly the 4 `u`/`v` component arrays). The measurement cannot have come from + the Phase-A source the plan names. The Jensen-gap *physics* stands; the + *measurement* must be re-attributed (WeatherBench2) or dropped. §5's "wind + speed — NOT derivable" row is wrong for this source. +- **§6.1 blosc 1.27×** is not representative: measured **1.794×** + (`2m_temperature`), **1.771×** (`2m_dewpoint_temperature`), **1.248×** + (`10m_u_component_of_wind`) at one timestep. Ratio is strongly + variable-dependent; a single scalar is the wrong summary. +- **§6.3 BF16 anomaly gain** reproduced in **direction only**: 0.456 K → 0.00609 K + = **74.9×** here (doc: 1.069 K → 0.0110 K = 97×), under a zonal-mean + climatology proxy. Different climatology ⇒ different measurement, not a + contradiction — but the doc's numbers stay **unreproduced** and `[H]`. +- **New, unhidden:** 0.82 % of points saturate into the two rim buckets, giving a + palette **max** error of 17.5 K against a 0.068 K mean. That is + `quantize.rs:11-18`'s documented controlled-saturation tail. Any product claim + quotes the tail, not only the mean. + +### 12.5 — Pre-existing defect found in passing (not caused by this work) + +`Signed360::sign()` is `if polar >= 128 { Pos } else { Neg }` +(`residue.rs:109-115`), so an **all-zero, never-written lane decodes as a +definite lower-hemisphere orientation** rather than as "unaddressed". Both +sibling facet lanes get this right — `Tekamolo`: *"an all-zero facet reads as +unaddressed … never a wrong circumstance"* (`canonical_node.rs:886-887`); +`CausalWitness`: *"reads as unbound … never a wrong binding"* (`:900-902`). +Under the CANON zero-fallback ladder ("zero = fall through to the broader +default") this is a wrong-value-from-dormant-bytes defect. Filed, not fixed here. + +### 12.6 — Probe queue, re-graded + +| id | status | +|---|---| +| **P1** | **RUN** — `probes/weather-p1/`, results committed. Falsified §1.2 and §6.5; re-measured §6.1/§6.3. | +| **P1-reframed** | **RUN** (§12.10) — CI-vs-noise-floor, no round-trip. Reverses §12.1's magnitude; surfaces the saturation-tail-hits-the-extremes finding. | +| **P2** | **RUN — PASSES** (§12.8). Re-scoped per §12.1 and measured: shared canonical floor ρ ≥ 0.9996 cross-variable vs 0.857–0.875 per-variable; zero within-variable cost. Shared palette + LUT survives; the shared *transform* does not. | +| **P3** | largely ANSWERED by §12.8 (no within-variable penalty from the shared floor); stays open for the saturation-tail question only. | +| **P4** | **CANCELLED as specified.** The premise is falsified (§12.2). Successor: the 16-byte `Pair48` facet lane — a mint decision, operator-gated, not a worker task. | +| **P5** | unchanged, NOT RUN. | +| **P6** | unchanged, NOT RUN. | + +### 12.7 — What survives, stated plainly + +The **economics** (§7) survive: pay the transform once at ingest, compare via an +O(1) 128 KB cache-resident LUT forever. The **place ⊕ residue decomposition** +survives and strengthened — anomaly-vs-raw is a 74.9× BF16 gain measured on real +data. The **8-bits-per-scalar** claim survives, with a corrected reason: a robust +percentile window over a **shape-appropriate** transform, which for weather +anomalies is the identity. **helix360 as the wind carrier does not survive**, and +neither does the single-shared-transform framing of the substrate. + +### 12.8 — P2 RUN (re-scoped): the shared palette + LUT **survives**; only the shared *transform* died + +The §12.1 falsification killed the shared **Fisher-Z**. It did not, by itself, +tell us whether the shared **palette + LUT** — the actual product claim — went +with it. Measured (`probes/weather-p1/p2_probe.py`, real ERA5, 3 variables, +200 000 random cross-variable pairs each; truth = float z-domain +`|z_a − z_b|`, candidate = `|palette_idx_a − palette_idx_b|`, Spearman ρ): + +| pair | units | ρ shared floor | ρ per-variable floors | +|---|---|---|---| +| `2m_temperature` × `2m_dewpoint_temperature` | K × K | **0.999556** ⚠ *below the 0.9996 bar* | 0.999426 | +| `2m_temperature` × `10m_u_component_of_wind` | K × m/s | **0.999736** PASS | **0.874801** | +| `2m_dewpoint_temperature` × `10m_u_component_of_wind` | K × m/s | **0.999722** PASS | **0.856984** | + +**Within-variable control (the anti-vacuity half — a shared window could have +cost same-variable resolution, and did not):** + +| variable | ρ shared | ρ per-variable | effective buckets (shared) | empty | +|---|---|---|---|---| +| `2m_temperature` | 0.9996 | 0.9996 | 118.6 / 256 | 0 | +| `2m_dewpoint_temperature` | 0.9995 | 0.9996 | 120.7 / 256 | 0 | +| `10m_u_component_of_wind` | 0.9998 | 0.9996 | 151.0 / 256 | 0 | + +> **⊘ CORRECTED (codex/CodeRabbit review on #920).** This table originally +> reported the first pair as `0.9996` — a **rounding of 0.999556, which is +> BELOW the 0.9996 bar**, and the probe's `SHARED` label only ever tested +> "shared beats per-variable", never the bar itself. `p2_probe.py` now carries +> an explicit pre-registered `TARGET_RHO` and prints PASS / BELOW TARGET on +> unrounded values. **2 of 3 pairs pass; the K×K pair does not.** + +**Reading.** The shared canonical floor is not free-riding: it beats +per-variable floors by **0.9997 vs 0.857–0.875** exactly where the units differ +(K vs m/s) — **both cross-unit pairs clear the bar** — and the two paths are +close where the units match (K vs K, 0.999556 vs 0.999426, *the shared floor +still winning but missing the 0.9996 bar*). That is the discrimination pattern +theory predicts, and it appears without having been tuned for. The shared window +shows **no material measured cost** within-variable at this sample size +(within-variable ρ differs by ≤ 0.0001, e.g. dewpoint 0.9995 shared vs 0.9996 +per-variable) and leaves **zero empty buckets**. + +**So the mechanism was never Fisher-Z — it was standardization.** Putting each +variable on its own z-scale is what makes unlike quantities commensurable; +Fisher-Z was an *additional* transform layered on top, and §12.1 measured it as +actively harmful for these shapes. The product claim — *"correlate what normally +cannot be correlated, via one cache-resident 256×256 LUT"* — **stands, with a +corrected mechanism.** + +**Policy consequence:** §4's option **(a) one canonical z-floor** is now the +measured recommendation, not merely a candidate; option (c) hybrid remains +available for archival lanes that want per-variable resolution, but this run +shows it buys nothing for correlation, since the shared floor already matches +per-variable ρ within-variable. + +**Honest boundaries.** The K×K pair is **below the pre-registered bar** — the +"passes" verdict holds for the cross-unit case this probe was designed to test, +not universally. One timestep (2021-06-15 12:00 UTC); zonal-mean +climatology proxy; three variables of which only one carries a different unit; +ρ is rank-preservation of the *distance*, which is the right test for +LUT-backed ranking/search but is not a claim about absolute error. The shapes +do genuinely differ (kurtosis +3.30 / +3.39 / **+0.41**), so the result is not +an artifact of three identically-shaped fields. Grade `[H]` until re-run across +seasons and a fourth variable in a different unit again. + +**Probe queue delta:** **P2 → RUN, PASSES.** P3 (per-variable resolution cost) +is now largely answered in the negative — the shared floor shows no +within-variable penalty — but stays open for the tail/saturation question. +Step 3 (Phase A end-to-end) is unblocked. + +### 12.9 — Gate 1 RUN with `jc` (not scipy): codec self-consistency — NOT product-frame evidence + +> **⊘ SCOPE (CodeRabbit on #920, correct).** Everything in this section is a +> **round-trip** measurement (`truth` vs `reconstructed`), i.e. an **internal +> codec / consistency diagnostic**. §12.10–§12.11 rule round-trip OUT as the +> *product* evaluation frame, because the original is retained. So these numbers +> say the quantizer is self-consistent and rank-preserving; they are **not** +> evidence for the noise-floor conclusion and must never be cited as such. The +> product-frame verdict lives in §12.11 (CI vs floor) and §12.8 (code-distance +> rank), neither of which decodes. + +§9's gate 1 answered with the workspace's own instruments — +`crates/jc/examples/weather_substrate_reliability.rs`, run against the P1/P2 +palette256 round-trip on real ERA5 (3 variables × 50 000 sampled points, shared +canonical floor). `jc` rather than `ndarray::hpc::reliability` deliberately: the +mirror returns `0.0` sentinels where `jc` returns `Option`, so a `0.0` from +the mirror cannot be distinguished from *undefined* — §9's stated rule. + +| variable | Pearson r | Spearman ρ | Cronbach α | ICC(2,1) abs | ICC(3,1) cons | +|---|---|---|---|---|---| +| `2m_temperature` | 0.998558 | 0.999866 | 0.999216 | 0.998432 | 0.998432 | +| `2m_dewpoint_temperature` | 0.998506 | 0.999898 | 0.999179 | 0.998359 | 0.998360 | +| `10m_u_component_of_wind` | 0.999785 | 0.999964 | 0.999891 | 0.999781 | 0.999781 | +| **POOLED (shared floor)** | **0.998939** | **0.999926** | **0.999435** | **0.998869** | **0.998870** | + +**Negative control — the can-it-fire half.** `--shuffle` feeds deliberately +mismatched pairs; every statistic collapses to noise: r ∈ [−0.0062, −0.0007], +ρ ∈ [−0.0057, −0.0001], α ∈ [−0.0126, −0.0014], both ICCs likewise. So these are +measurements, not assertions implied by their own input. + +**The load-bearing detail: ICC(2,1) ≈ ICC(3,1) to six decimals.** They are +reported separately on purpose — ICC(3,1) is *consistency*, ICC(2,1) is +*absolute agreement*, and a quantizer that preserved shape while shifting scale +would show high consistency and degraded agreement. They agree, so **there is no +scale shift hiding behind a consistency number**. A single ICC figure would not +have shown this. + +⚠ **Significance, not point estimates.** Per `I-NOISE-FLOOR-JIRAK`, any claim +that these values sit N σ above a noise floor takes **Jirak 2016 weak-dependence +rates** — weather fields are spatially autocorrelated, so effective sample size +is far below the nominal 50 000, and classical IID intervals would be wrong. +Nothing above is a significance claim; they are point estimates with an explicit +negative control. + +### 12.10 — ⊘⊘ THE EVALUATION FRAME WAS WRONG: thresholds vs noise floor, never round-trip + +**Operator correction, 2026-08-11.** Everything in §12.1 was scored with a +**round-trip** metric — encode, decode, measure reconstruction error against the +original. **That is the wrong criterion for this substrate, and it inverted a +verdict.** + +**Why it is wrong.** The substrate never decodes. The original is *retained* — +public knowledge, the hydratable value column, the Lance-versioned f32. The code +is a **one-way address / discriminator layered over a kept original**, so the +question was never *"can I reconstruct the original from the code"* but +**"does the code's confidence interval (its quantization threshold) sit below +the original's own noise floor?"** Deviation only matters where it **exceeds +what the original itself can resolve**. This session's own 2026-08-08 reply had +it right — *"the code is not the bottleneck, the atmosphere is; the instruments +can't see it"* — and §12.1 then scored reconstruction MAE anyway. + +> **⊘⊘⊘ THE TABLE BELOW IS STILL WRONG — see §12.11 (codex P1 on #920).** +> `p1_noise_floor.py` computes `dev = |bucket_center − original|`, which is a +> **decoded reconstruction error**: the very round-trip this section declares +> invalid. The CI values it computed were only *printed*, never used in the +> exceedance fractions. §12.11 carries the genuinely CI-based measurement. The +> qualitative conclusion survives; every number below does not. + +**Re-measured in the correct frame** (`probes/weather-p1/p1_noise_floor.py`, +same real ERA5 T2m field, 1,038,240 points; fraction of gridpoints whose +code-induced deviation **exceeds** a candidate floor): + +| path | CI (mid-range) | >0.25 K | >0.5 K | >1.0 K | >2.0 K | max | +|---|---|---|---|---|---|---| +| linear | ±0.094 K | 0.76 % | **0.70 %** | **0.60 %** | 0.40 % | 17.55 K | +| Fisher-Z | position-dependent | 36.3 % | **0.69 %** | **0.58 %** | 0.40 % | 17.54 K | + +**§12.1 is PARTIALLY REVERSED.** At a 0.5–1 K floor the two paths are +**indistinguishable** (0.70 % vs 0.69 %; 0.60 % vs 0.58 %), and the exceedances +are the **shared saturation tail**, not the transform. "Fisher-Z is actively +harmful for weather" was an **artifact of the round-trip metric**. Fisher-Z only +fails at a 0.25 K floor — stricter than the instruments — because its mid-range +buckets are ±0.26 K wide (CI by index: 0.087 / **0.263** / 0.005 / 0.0001 / +0.000 K at idx 0/64/128/192/255; the position-dependence *is* the shape finding, +now correctly expressed as a CI profile rather than an error score). + +**What survives of §12.1: bucket economy, not validity.** Fisher-Z spends 228 of +256 addresses where this distribution is not (28.1 effective buckets vs 115.7). +That degrades **discrimination granularity** — more code ties, coarser analogue +retrieval — which is a real cost for the retrieval lane and was measured +frame-correctly in **§12.8** (rank ρ of *code distances*, no round-trip). So the +falsified claim shrinks from *"Fisher-Z is wrong for weather"* to **"Fisher-Z +buys nothing here and costs address space."** §1.2's mechanism (transform should +match distribution shape) stands; its *magnitude* did not survive re-framing. + +**§12.2 loses its main ground too.** The structural facts stand (`ResidueEdge` +carries no hemisphere sign; azimuth is `n·φ`-derived, not an external bearing). +But a substantial part of that cancellation rested on *"the crate ships no +inverse — no `decode(edge) -> n` exists."* **Under a one-way address frame that +was never a defect.** What a wind lane actually needs is a **defined +deterministic bearing→address convention** — a design act, which is exactly what +the mint decision is — plus CI-vs-floor validation: a ~1.4–1.7° step against the +**10° METAR/SYNOP reporting increment** is the ~6× margin recorded on +2026-08-08. **The `Pair48 = [Signed360; 2]` mint returns to the table +strengthened, not weakened.** P4 stays cancelled *as specified* (2×`ResidueEdge` +in 6 bytes remains structurally wrong); its successor is live again. + +**A finding neither frame produced alone.** The **shared saturation tail** — +0.4–0.7 % of points, max ≈ 17.5 K — is the *only* genuine noise-floor violation +for either path, and those points are the **extremes**: heat waves, cold +outbreaks. Meteorologically the most important events sit exactly where this +representation is worst. The 0.4–99.6 % window is a **knob inherited from the +cognitive substrate's defaults**; weather likely wants it widened, or an +explicit overflow lane for extremes. That is now the highest-value open item on +the representation, ahead of any transform question. + +**⚠ The floor itself is NOT yet pinned.** 0.25 / 0.5 / 1.0 / 2.0 K are a +**reference class**, not a citation. No published ERA5 T2m uncertainty figure was +verified in this run, so every verdict above is *"indistinguishable at a +0.5–1 K-class floor"*, conditional on that class being right. **Pinning a citable +per-variable noise floor is a prerequisite** before any of this is graded `[G]` +— and per `I-NOISE-FLOOR-JIRAK`, any σ-distance claim against it takes Jirak +weak-dependence rates, never classical intervals. + +**Meta — §12.3's rule, applied to §12.1.** §12.3 stated that *a correction is a +claim and carries a claim's burden of proof*. §12.1 **was** such a correction, +and it over-claimed because it inherited an evaluation frame without examining +it. The rule caught its own author one section later. **The frame a measurement +is scored in is itself a claim requiring an audit** — that is the generalization, +and it is worth more than either verdict it revised. + +### 12.11 — the CI metric, done properly (codex P1 on #920): the frame was right, my implementation of it was not + +§12.10 announced the correct criterion and then **measured the old one anyway**. +`p1_noise_floor.py` computes `dev = |bucket_center − original|` — a decoded +reconstruction error — and counts `dev > floor`. The confidence intervals it +computed were printed and discarded. Caught by codex, correctly: + +> *"The advertised noise-floor reframing still computes a round-trip +> reconstruction metric… the committed exceedance fractions and the conclusions +> in §12.10 are produced by the exact evaluation frame the commit says is +> invalid."* + +**The actual metric** (`probes/weather-p1/p1_ci_vs_floor.py`): for each point, +take the **confidence interval of the bucket it encodes to**, expressed in +original units — `CI(b) = ½ × (width of bucket b in K)` — and ask whether that +CI exceeds the floor. Saturated edge buckets (0, 255) have **unbounded** CI and +are reported separately, never folded in as if they had a finite width. + +| floor | linear: interior CI > floor | Fisher-Z: interior CI > floor | saturated (unbounded) | +|---|---|---|---| +| 0.25 K | **0.0000 %** | **95.65 %** | 0.848 % / 0.820 % | +| 0.5 K | 0.0000 % | 0.0000 % | 0.848 % / 0.820 % | +| 1.0 K | 0.0000 % | 0.0000 % | 0.848 % / 0.820 % | +| 2.0 K | 0.0000 % | 0.0000 % | 0.848 % / 0.820 % | + +Interior CI: **linear is flat at 0.09412 K everywhere**; Fisher-Z spans +0.00000–0.42899 K, median 0.00561. + +**What changes versus §12.10's bogus table:** + +1. **Linear is cleaner than reported** — 0.0000 % interior exceedance at every + floor, not 0.60–0.76 %. Those fractions were reconstruction noise, not CI. +2. **Fisher-Z at a tight floor is far worse than reported** — **95.65 %**, not + 36.3 %. And the mechanism is now explicit: the median CI **across buckets** + is 0.0056 K, but most **points** live in the few wide mid-range buckets + (up to 0.429 K). Most buckets are narrow tail buckets holding almost no data. + **This is the 28.1-effective-buckets finding restated in CI units** — the two + measurements now agree, which is the cross-check §12.10 lacked. +3. **The headline conclusion SURVIVES, and is cleaner.** At a 0.5–1 K floor the + two paths are identical and **only saturation matters** — 0.848 % vs 0.820 %, + i.e. the shared tail is the *entire* story above 0.25 K. +4. **The saturation finding is confirmed and sharpened — with an explicit + floor condition.** At floors **≥ 0.5 K** it is the **only** violation for + either path: **0.848 %** (linear) and **0.820 %** (Fisher-Z) of the + 1,038,240 gridpoints, i.e. the points landing in buckets 0/255 whose CI is + unbounded. **At a 0.25 K floor that is false for Fisher-Z**, which adds + **95.65 %** interior-CI exceedance on top. Every floor-dependent statement + here stays conditional until a per-variable noise floor carries a citation. + +**So §12.1 stays partially reversed, on better evidence** — Fisher-Z is not a +validity failure at the plausible floor; it is an **address-economy** failure, +and at a 0.25 K floor that economy failure becomes a validity failure too. + +**Third-order lesson.** §12.3: *a correction carries a claim's burden*. §12.10 +was a correction of §12.1 that **inherited the very defect it was correcting** — +it named the right frame in prose and shipped the wrong one in code. Naming a +frame is not adopting it; **the implementation is the claim**, and prose +asserting otherwise is exactly the "doc-comment claim is not a behaviour" rule +(`CLAUDE.md` falsifiability §) one level up. An external reviewer caught what +two internal passes did not. + +**Also fixed (codex P2):** `fetch.py` raised an uncaught `HTTPError` on the +first legitimately-missing chunk, so the documented `python3 fetch.py` +reproduction died before writing its manifest — while the README *correctly* +documented 404 as valid fill semantics. The code now returns the fill array and +continues, matching the README it contradicted. + +### 12.12 — ⊘⊘⊘ §12.2's REASONING is regraded; the `Pair48` mint is RETIRED; the doctrine doc existed and was never read + +**Trigger.** Operator: *"probably signed360, a naming mis remembered"*, with two +links — `crates/helix/src/residue.rs#L74` and +`.claude/knowledge/helix-cartesian-vs-fisher2z.md#L18`. The second is a +knowledge doc **in this repo**, whose `READ BY:` header names *"ANY session that +encodes/decodes/renders an orientation, normal, or direction"*. This session is +exactly that session, and **never opened it** — not before §2.3, not during +§12.2's envelope audit, not through four subsequent corrections. + +#### The name, settled + +`helix360` **does not exist and never did** `[G-absence]`. Pickaxe over full +history (`git log -S`, all refs, plus unreachable objects) returns **12 blobs, +all authored by this session, zero deletions, zero blast radius**; ndarray and +the `lance-graph2` backup carry none. The real symbol is **`Signed360`** +(`residue.rs:76-116`). Every use of "helix360" in §0–§11 above is this session's +coinage for `Signed360`; read it that way. + +#### Three structural corrections to §12.2 — the facts stood, the inferences did not + +| §12.2 said | Regrade | Authority | +|---|---|---| +| 1. "`Signed360` is ONE signed orientation, not two hemisphere codes" — used to argue the lane cannot carry a from/to pair | **Fact stands; the inference inverts.** The sign is precisely what makes it whole: *"the normal-only 6-byte `Signed360` is a **complete** full-sphere direction — you do NOT need a second 'pos' helix to complete it."* | `helix-cartesian-vs-fisher2z.md:77-82` | +| 2-3. "`ResidueEdge` cannot carry a hemisphere sign / has no azimuth" — used to argue *the lane* has no bearing capacity | **Fact stands; it was the wrong object.** `ResidueEdge`/`rim` is the **METRIC carrier, not a render input** — *"never run the rim's atanh/tanh to recover a direction."* Direction lives in `(polar, azimuth)`. The operator's *"ResidueEdge is turbovec, nobody was asking you to use turbovec"* named this before the doc confirmed it. | `helix-cartesian-vs-fisher2z.md:83-86`; `residue.rs:21-27` | +| 4. crate says *"no free 2-DOF direction codec in this crate"* ⇒ read as "the representation cannot carry a bearing" | **Misread: no *helper*, not no *capacity*.** `azimuth` is *"`n·φ mod 2π` mapped to `[0, 65536)` over the full **360°**"* — a full-circle 16-bit angular field. What the crate lacks is the **encode helper**: *"Encoding a 3D normal → `(n, sign)` is a nearest spherical-Fibonacci search (the crate has no `from_normal` helper)"*, with a worked pair at q2 `scratch-fma/helixbake`. | `residue.rs:85`; `helix-cartesian-vs-fisher2z.md:90-93` | + +#### What in §12.2 still stands, unchanged + +- **Point 5 (metric hazard) — stands, and is *already crate-documented*, not a + discovery.** `distance_heuristic` names its own failure mode as *"the + raw-azimuth 2π wrap"* and forbids it for CAKES bounds (`residue.rs:52-59`). + The correct statement is the canonical split, not a defect: **rim = L1-metric + carrier (`DistanceLut`, triangle inequality); azimuth = render/direction + carrier (circular, deliberately not L1-metric).** Two fields, two jobs. My + error was proposing to route a bearing through the *metric* field and then + reporting the resulting hazard as a property of the codec. +- **Point 6 (no per-value-lane reading selector exists) — stands `[G-absence]`.** + 15 `HelixResidue` hits, zero writers, zero decoders; `ReadMode`'s three axes + select tail / value_schema / edge_codec, none a value-lane *reading*. +- The **dormant-lane defect** is unaffected and still open: `Signed360::sign()` + reads an all-zero lane (`polar = 0`) as a definite `Sign::Neg` + (`residue.rs:109-115`), so "unwritten" and "lower hemisphere" are + indistinguishable — a real hazard the moment a writer exists. + +#### Decision RETIRED: the `Pair48` mint + +§12.2's `[S]` candidate (a new 16-byte `FacetSchema::Pair48 = [Signed360; 2]`) +and §12.3's lesson-about-the-lesson both rested on needing a second helix to +complete a direction. **That premise is false**, so the open operator decision is +**withdrawn, not deferred** — do not mint it. A wind bearing is a 2-DOF direction +on the horizontal; one `Signed360` already addresses the full sphere, a fortiori +the circle. §12.3's *extracted rule* survives untouched and is in fact the rule +that just fired again: **a correction is a claim and carries a claim's burden.** +Three links deep now (§11-C2 → §12.2/§12.3 → §12.12). + +#### The real gap, stated so it is actionable + +Not a missing *carrier* — a missing *encode*. `(bearing, elevation) → Signed360` +is a **nearest spherical-Fibonacci search** over `HemispherePoint::lift` +(pole axis = chosen world axis, `sign = sign(n·pole)`), which the crate +deliberately does not ship as `from_normal`, with a worked reference pair +(q2 `scratch-fma/helixbake` + `cockpit/src/BodyHelix.tsx`). Decode for +render/compare is **pre-materialize one `(polar × azimuth) → direction` LUT, +then gather** — polar 7-bit ≈ 0.45°, azimuth 10-bit LUT column ≈ 0.35°. That is +the mechanical form of the operator's *"you only pay the inbound tax once"*, and +`[S]` until measured on real wind data. + +#### Process lesson — the one worth more than the codec + +`CLAUDE.md` § *Consult, don't guess* orders it explicitly: specialist card → +**knowledge doc** → board → *only then* grep source. This arc did source-first +and stayed wrong through four corrections, because **reading the primary source +is not the same as reading the doctrine that interprets it.** `residue.rs` +states the layout truthfully; it does not say which field is metric and which is +render, nor that the sign already completes the sphere — that is exactly what +the knowledge doc exists to carry, and it is what a `READ BY:` header is *for*. +Cost: one invented 48-bit budget, one straw-man in/out reading, one retracted +mint decision, and a hunt for a deleted symbol that never existed. **A +`READ BY:` header that matches the session's own subject is a mandatory read, +not a suggestion.** + +Cross-ref: `.claude/knowledge/helix-cartesian-vs-fisher2z.md` (the authority for +this whole section), `crates/helix/KNOWLEDGE.md` (place/residue spec). + +#### 12.12a — TESTED (added after the fact, on the operator's question "did you test signed360?") + +**No — §12.12 above was landed on doc comments and the knowledge doc, with +nothing executed.** That is the same failure it diagnoses, one level down: +having just written *"reading the primary source is not the same as reading the +doctrine"*, I read both and **ran neither**. Corrected here. + +The crate's own suite is **77 + 4 + 7 doctests, all green**, but a green suite is +not evidence for a specific claim. Audited claim-by-claim, then closed the gaps: + +| claim in §12.12 | before | now | +|---|---|---| +| sign exact at `\|y\| ≈ 0` (partition, not a centred round) | `[G]` — `signed360_neg_sign_survives_near_rim_at_high_total`, a real regression with a documented prior failure (codex P2 #498) | unchanged | +| `azimuth` is a full-360° field | **doc comment only.** The existing `signed360_azimuth_varies_with_n` asserts `a != b` for ONE consecutive pair — golden-angle stepping makes that true of any sane implementation, so it cannot detect a truncated field | `[G]` **measured** over the whole domain: `min 0`, `max 65535`, **256/256** coarse arcs occupied, 54 319 distinct values | +| hemisphere → sphere via the sign | mechanism tested; **nothing decodes to a direction** (there is no decode — see the `from_normal` gap) | unchanged; the *coverage* half stays `[H]` until an encode/decode pair exists | +| dormant lane reports a definite sign | **asserted twice, never constructed** | `[G]` **demonstrated**: `Signed360::from_bytes([0u8; 6]).sign()` → `Sign::Neg` | + +Also measured in passing: the `polar` partitions fill their halves **exactly** — +`Pos ∈ [128, 255]`, `Neg ∈ [0, 127]`, no overlap, no gap at either end. + +**Landed as `crates/helix/tests/signed360_claims.rs`** (3 tests), each +**disable-verified red-then-green** against an injected defect: + +| test | injected defect | observed failure | +|---|---|---| +| `azimuth_spans_the_full_circle_not_merely_varies` | 10-bit truncated field | `left: 1023, right: 65535` | +| `polar_partitions_are_exactly_the_two_halves` | the #498 centred-at-128 round | `left: (0, 128), right: (0, 127)` | +| `dormant_all_zero_lane_decodes_as_a_definite_sign_known_defect` | a "fixed" sentinel returning `Pos` | `left: Pos, right: Neg` | + +The third **pins a defect, not a virtue** — when the dormant-lane hazard is +fixed it MUST fail and be re-pinned deliberately, never silently edited. + +**Scope caveat, stated rather than implied:** `helix` is **excluded from the root +workspace** (root `Cargo.toml` `exclude`) and named in **no CI workflow** +`[G-absence]` — so these 3 tests, and the crate's existing 77, run **only when +invoked by hand** in that crate. Adding them raises the floor for the next +session that looks; it does not put them on a gate. diff --git a/crates/helix/tests/signed360_claims.rs b/crates/helix/tests/signed360_claims.rs new file mode 100644 index 000000000..d728fa75b --- /dev/null +++ b/crates/helix/tests/signed360_claims.rs @@ -0,0 +1,90 @@ +//! Tests for the three `Signed360` claims that `.claude/knowledge/` +//! documents but the crate's own suite did not exercise. +//! +//! Written 2026-08-11 after a session landed all three in a knowledge doc on the +//! strength of DOC COMMENTS alone. Per `CLAUDE.md`'s falsifiability rule — *"a +//! doc-comment claim is not a behaviour; a test must exercise the claim or the +//! claim must be labelled claimed, unverified"* — they are exercised here. +//! +//! NOTE: `helix` is excluded from the root workspace and appears in no CI +//! workflow, so these run only when invoked by hand in this crate. +use helix::placement::Sign; +use helix::residue::{ResidueEncoder, Signed360}; + +/// `azimuth` spans the FULL circle, not merely "varies". +/// +/// The pre-existing `signed360_azimuth_varies_with_n` asserts only that two +/// consecutive `n` differ — which golden-angle stepping makes true of any sane +/// implementation, so it cannot detect a truncated field. Measured here over the +/// whole domain: min 0, max 65535, all 256 coarse buckets occupied. +/// +/// Falsifier: an implementation packing the angle into the low 10 bits (or +/// scaling to anything short of the full u16) fails on `max` and on coverage. +#[test] +fn azimuth_spans_the_full_circle_not_merely_varies() { + let enc = ResidueEncoder::new(65_536); + let (mut lo, mut hi) = (u16::MAX, 0u16); + let mut buckets = [false; 256]; + for n in 0..65_536usize { + let a = enc.encode_signed(7, n, Sign::Pos).azimuth; + lo = lo.min(a); + hi = hi.max(a); + buckets[(a >> 8) as usize] = true; + } + let covered = buckets.iter().filter(|b| **b).count(); + assert_eq!(lo, 0, "azimuth must reach the bottom of the u16 range"); + assert_eq!(hi, u16::MAX, "azimuth must reach the top of the u16 range"); + assert_eq!(covered, 256, "every 1/256 arc of the circle must be reachable"); +} + +/// The two hemispheres occupy their EXACT halves of the `polar` byte. +/// +/// Broader than the rim-only `signed360_neg_sign_survives_near_rim_at_high_total` +/// regression (codex P2 #498): swept over the whole domain, `Pos` fills +/// `[128, 255]` and `Neg` fills `[0, 127]` with no overlap and no gap at the ends. +/// +/// Falsifier: the #498 centred-at-128 round puts a vanishing negative lift at +/// polar 128, which lands in the `Pos` partition — `neg_max` would reach 128. +#[test] +fn polar_partitions_are_exactly_the_two_halves() { + let enc = ResidueEncoder::new(65_536); + let (mut pos_lo, mut pos_hi) = (255u8, 0u8); + let (mut neg_lo, mut neg_hi) = (255u8, 0u8); + for n in 0..65_536usize { + let p = enc.encode_signed(7, n, Sign::Pos).polar; + let q = enc.encode_signed(7, n, Sign::Neg).polar; + pos_lo = pos_lo.min(p); + pos_hi = pos_hi.max(p); + neg_lo = neg_lo.min(q); + neg_hi = neg_hi.max(q); + } + assert_eq!((pos_lo, pos_hi), (128, 255), "Pos must fill [128,255] exactly"); + assert_eq!((neg_lo, neg_hi), (0, 127), "Neg must fill [0,127] exactly"); +} + +/// ⚠ CHARACTERIZES A KNOWN DEFECT — pinned so a fix must break it deliberately. +/// +/// An all-zero, never-written 6-byte lane decodes as a DEFINITE `Sign::Neg` +/// direction: `sign()` partitions on `polar >= 128`, and 0 < 128. So "unwritten" +/// and "genuinely lower-hemisphere" are indistinguishable — while the sibling +/// lanes `Tekamolo` / `CausalWitness` define all-zero as *unaddressed*, and the +/// CANON zero-fallback ladder reads a zero tier as "not consulted". +/// +/// Harmless today only because the lane has zero writers and zero decoders; it +/// becomes a live hazard the moment one exists. Filed, not fixed. +/// +/// When the defect IS fixed (e.g. `sign()` returning `Option`, or a +/// reserved sentinel), this test MUST fail and be re-pinned deliberately — +/// never silently edited. +#[test] +fn dormant_all_zero_lane_decodes_as_a_definite_sign_known_defect() { + let dormant = Signed360::from_bytes([0u8; 6]); + assert_eq!( + dormant.sign(), + Sign::Neg, + "DEFECT pinned: an unwritten lane reports a definite hemisphere" + ); + assert_eq!(dormant.polar, 0); + // The complement is unambiguous, which is what makes the zero case the odd one. + assert_eq!(Signed360::from_bytes([0xFFu8; 6]).sign(), Sign::Pos); +} diff --git a/crates/jc/examples/weather_substrate_reliability.rs b/crates/jc/examples/weather_substrate_reliability.rs new file mode 100644 index 000000000..2d0ad68ff --- /dev/null +++ b/crates/jc/examples/weather_substrate_reliability.rs @@ -0,0 +1,220 @@ +//! Gate 1 of the weather-substrate verification battery (§9 of +//! `.claude/knowledge/weather-normalized-substrate.md`): is the palette256 +//! representation **valid and reliable** by the workspace's own instruments? +//! +//! Runs `jc::reliability` — Pearson r, Spearman ρ, Cronbach α, ICC(2,1) and +//! ICC(3,1) — over `(truth, reconstructed)` pairs produced by the P1/P2 probes +//! on real ARCO-ERA5 fields. Deliberately Rust + `jc` rather than scipy: §9 +//! rules that `jc` returns `Option` where the `ndarray::hpc` mirror +//! returns `0.0` sentinels, so a `0.0` from the mirror is ambiguous +//! (undefined-vs-zero) and `jc` is what a verdict cites. +//! +//! Input: `probes/weather-p1/jc_input.bin`, written by +//! `probes/weather-p1/export_for_jc.py`. Layout (all little-endian): +//! `i64 n_vars, i64 n_points`, then per variable `n_points × f64` truth +//! followed by `n_points × f64` reconstructed. +//! +//! ```text +//! cargo run -p jc --example weather_substrate_reliability -- probes/weather-p1/jc_input.bin +//! ``` +//! +//! **What would make this FAIL** (the falsifiability rule — an assertion +//! implied by its own input is not a test): a quantizer that lost rank order +//! drops Spearman; one that lost scale drops ICC(2,1) (absolute agreement) +//! while leaving ICC(3,1) (consistency) high — the two ICC forms are reported +//! separately precisely so a scale shift cannot hide behind a consistency +//! number. The `--shuffle` control below is the can-it-fire half: it feeds +//! deliberately mismatched pairs and must collapse every statistic. + +use jc::reliability::{cronbach_alpha, icc, pearson, spearman, IccForm}; +use std::io::Read; + +const VAR_NAMES: [&str; 3] = [ + "2m_temperature", + "2m_dewpoint_temperature", + "10m_u_component_of_wind", +]; + +/// Read `n` little-endian `f64`s from `buf` starting at `*off`, advancing `off`. +/// +/// The caller is responsible for bounds: `main` validates the total length +/// against the header (`16 + n_vars * n_points * 2 * 8`) before any call, so a +/// short buffer is rejected up front rather than panicking mid-parse. +fn read_f64s(buf: &[u8], off: &mut usize, n: usize) -> Vec { + let mut v = Vec::with_capacity(n); + for _ in 0..n { + let mut b = [0u8; 8]; + b.copy_from_slice(&buf[*off..*off + 8]); + v.push(f64::from_le_bytes(b)); + *off += 8; + } + v +} + +/// Parse and VALIDATE the 16-byte header, returning `(n_vars, n_points, want)`. +/// +/// Rejects, rather than trusting: a short buffer, a negative dimension (an +/// `i64 -1` would otherwise become a huge `usize`), an arithmetic overflow in +/// the expected size, and a length mismatch. Only after all four does any +/// payload byte get read — so a malformed file exits cleanly instead of +/// panicking or attempting a giant allocation. +fn parse_header(buf: &[u8]) -> Result<(usize, usize, usize), String> { + if buf.len() < 16 { + return Err(format!("truncated header ({} bytes, need 16)", buf.len())); + } + let mut hdr = [0u8; 8]; + hdr.copy_from_slice(&buf[0..8]); + let n_vars_i = i64::from_le_bytes(hdr); + hdr.copy_from_slice(&buf[8..16]); + let n_points_i = i64::from_le_bytes(hdr); + if n_vars_i < 0 || n_points_i < 0 { + return Err(format!( + "negative dimension in header (n_vars={n_vars_i}, n_points={n_points_i})" + )); + } + let (n_vars, n_points) = (n_vars_i as usize, n_points_i as usize); + let want = n_vars + .checked_mul(n_points) + .and_then(|v| v.checked_mul(2)) + .and_then(|v| v.checked_mul(8)) + .and_then(|v| v.checked_add(16)) + .ok_or_else(|| format!("header dimensions overflow: {n_vars}×{n_points}"))?; + if buf.len() != want { + return Err(format!( + "expected {want} bytes for {n_vars}×{n_points}, got {}", + buf.len() + )); + } + Ok((n_vars, n_points, want)) +} + +fn show(label: &str, truth: &[f64], recon: &[f64]) { + let fmt = |o: Option| o.map_or_else(|| " UNDEFINED".to_string(), |v| format!("{v:10.6}")); + // Cronbach / ICC treat (truth, reconstructed) as 2 "raters" of each point. + let paired: Vec> = truth.iter().zip(recon).map(|(&t, &r)| vec![t, r]).collect(); + // cronbach_alpha wants items (raters) as the outer dimension. + let items = vec![truth.to_vec(), recon.to_vec()]; + println!( + "{label:<32} r={} rho={} alpha={} icc2_1={} icc3_1={}", + fmt(pearson(truth, recon)), + fmt(spearman(truth, recon)), + fmt(cronbach_alpha(&items)), + fmt(icc(&paired, IccForm::Icc2_1)), + fmt(icc(&paired, IccForm::Icc3_1)), + ); +} + +fn main() { + let path = std::env::args() + .nth(1) + .unwrap_or_else(|| "probes/weather-p1/jc_input.bin".to_string()); + let shuffle = std::env::args().any(|a| a == "--shuffle"); + + let mut buf = Vec::new(); + match std::fs::File::open(&path).and_then(|mut f| f.read_to_end(&mut buf)) { + Ok(_) => {} + Err(e) => { + eprintln!("cannot read {path}: {e}"); + eprintln!("regenerate with: python3 probes/weather-p1/export_for_jc.py"); + std::process::exit(2); + } + } + + let (n_vars, n_points, want) = match parse_header(&buf) { + Ok(t) => t, + Err(e) => { + eprintln!("{path}: {e}"); + std::process::exit(2); + } + }; + let mut off = 16usize; + debug_assert_eq!(buf.len(), want); + + println!("weather-substrate gate 1 — jc::reliability over palette256 round-trip"); + println!("input: {path} vars={n_vars} points/var={n_points}"); + if shuffle { + println!("MODE: --shuffle (negative control — every statistic MUST collapse)"); + } + println!(); + + let mut all_truth = Vec::new(); + let mut all_recon = Vec::new(); + for i in 0..n_vars { + let truth = read_f64s(&buf, &mut off, n_points); + let mut recon = read_f64s(&buf, &mut off, n_points); + if shuffle { + recon.rotate_left(n_points / 3 + 1); // deterministic mismatch + } + let name = VAR_NAMES.get(i).copied().unwrap_or("var"); + show(name, &truth, &recon); + all_truth.extend_from_slice(&truth); + all_recon.extend_from_slice(&recon); + } + println!(); + show("POOLED (shared floor)", &all_truth, &all_recon); + println!( + "\nnote: ICC(2,1) is absolute agreement, ICC(3,1) consistency — reported\n\ + separately so a scale shift cannot hide behind a consistency number." + ); +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Build a well-formed buffer for `n_vars × n_points` (payload zeroed). + fn hdr(n_vars: i64, n_points: i64, payload: usize) -> Vec { + let mut v = Vec::new(); + v.extend_from_slice(&n_vars.to_le_bytes()); + v.extend_from_slice(&n_points.to_le_bytes()); + v.resize(16 + payload, 0); + v + } + + #[test] + fn valid_header_round_trips() { + // 2 vars × 3 points × 2 series × 8 bytes = 96 payload bytes. + let buf = hdr(2, 3, 96); + assert_eq!(parse_header(&buf), Ok((2, 3, 112))); + } + + #[test] + fn truncated_header_is_rejected() { + assert!(parse_header(&[0u8; 8]).is_err(), "8-byte header must fail"); + assert!(parse_header(&[]).is_err(), "empty buffer must fail"); + } + + #[test] + fn truncated_payload_is_rejected() { + // Header promises 96 payload bytes; supply 40. + assert!(parse_header(&hdr(2, 3, 40)).is_err()); + } + + #[test] + fn negative_dimensions_are_rejected() { + // Without the sign check these become huge usizes and the length + // comparison could be satisfied by a wrapped `want`. + assert!(parse_header(&hdr(-1, 3, 96)).is_err(), "negative n_vars"); + assert!(parse_header(&hdr(2, -3, 96)).is_err(), "negative n_points"); + } + + #[test] + fn overflowing_dimensions_are_rejected() { + // n_vars × n_points × 2 × 8 overflows usize on 64-bit. + let buf = hdr(i64::MAX, i64::MAX, 0); + assert!(parse_header(&buf).is_err(), "overflow must not wrap"); + } + + #[test] + fn read_f64s_reads_little_endian_and_advances() { + let vals = [1.5f64, -2.25, 1e-3]; + let mut buf = Vec::new(); + for v in vals { + buf.extend_from_slice(&v.to_le_bytes()); + } + let mut off = 0usize; + let got = read_f64s(&buf, &mut off, 3); + assert_eq!(got, vals.to_vec()); + assert_eq!(off, 24, "offset must advance by 8 per value"); + } +} diff --git a/probes/weather-p1/README.md b/probes/weather-p1/README.md new file mode 100644 index 000000000..1b7d247ad --- /dev/null +++ b/probes/weather-p1/README.md @@ -0,0 +1,140 @@ +# P1 — the weather-substrate measurements, made re-runnable + +> **Status: RUN, 2026-08-11.** Results in `p1_results_temperature.json`. +> **Two doc claims were FALSIFIED by this run** — see §3 and +> `.claude/knowledge/weather-normalized-substrate.md` §12. +> +> Probe P1 of the queue in that doc §10. Python by design: the plan's D-WXA-1 +> rules Stage-A ingest **disposable** (`Zarr → numpy → f32 slab`, no eccodes, +> no C deps). These scripts are the ingest half; nothing here is a Rust +> deliverable. + +## Why this exists + +The knowledge doc's §6 evidence ledger carried numbers that existed **only in a +session transcript**. Under the falsifiability rule (`CLAUDE.md` P0) that is a +claim, not a finding. This directory makes them re-derivable by command — and +the first honest run promptly contradicted two of them. + +## Reproduce + +```bash +pip install numpy zarr numcodecs scipy +cd probes/weather-p1 +python3 fetch.py # consolidated metadata + 2m_temperature chunk +python3 fetch_bg.py # remaining variables, with retries +python3 p1_probe.py # the measurements +python3 verify_apparatus.py # apparatus self-check + the mechanism diagnosis +``` + +`fetch.py` needs `zmeta.json` first: + +```bash +curl -s --noproxy '*' -o zmeta.json \ + "https://storage.googleapis.com/gcp-public-data-arco-era5/ar/1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr/.zmetadata" +``` + +`--noproxy '*'` is required in this environment (the proxy 403s/stalls on GCS; +plain `curl` through it does not reach the store). + +## 1. Source, verified rather than assumed + +- **Store** `gs://gcp-public-data-arco-era5/ar/1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr` + — HTTP 200, 52 arrays, consolidated `.zmetadata` 46,842 B. **This confirms the + plan's ⊘ C3 correction**: the previously-cited + `era5/…-1h-1440x721.zarr` does not exist. +- **Grid** `[561264, 721, 1440]`, dtype `0.9 | kurtosis | +|---|---|---|---|---| +| ERA5 T anomaly / robust scale | 0.172 | **0.769** | 0.012 | +3.30 (peaked) | +| correlation-like control `tanh(N(0,1.5))` | 0.672 | 0.135 | **0.327** | — | + +So the transform must **match the input distribution's shape**. Fisher-Z earns +its keep on correlation-like inputs (mass at ±1) and on helix's own rim radius +`r = √u`, which equal-area placement concentrates toward `r → 1` by construction +(`placement.rs:147-151`). A bell-shaped geophysical anomaly is the opposite case. + +**This falsifies the doc's §1.2 claim** that variance stabilization is what makes +8 bits sufficient — for weather scalars, plain linear quantization over a robust +window is what makes 8 bits sufficient, and Fisher-Z actively degrades it. + +**Consequence for the product claim.** "Fisher-Z everything into one comparable +substrate" does not survive. Cross-variable comparability must be re-derived +(probe P2) on a transform chosen per distribution shape — the shared *palette +and LUT* may still hold; the shared *transform* does not. + +## 4. Other measured deltas from the doc's §6 + +- **§6.1 blosc ratio.** Measured per variable at this timestep: + `2m_temperature` **1.794×**, `2m_dewpoint_temperature` **1.771×**, + `10m_u_component_of_wind` **1.248×**. The doc's "1.27× mean" is within range + for wind but not representative of temperature — the ratio is + strongly variable-dependent, so a single scalar is the wrong summary. +- **§6.3 BF16 raw-vs-anomaly.** Reproduced in *direction*, not in magnitude: + raw-Kelvin MAE **0.456 K** → anomaly MAE **0.00609 K** = **74.9×** + (doc: 1.069 K → 0.0110 K = 97×). Different climatology (zonal-mean proxy here) + makes this a *different* measurement, not a contradiction — but the doc's + numbers remain unreproduced, so they stay `[H]` with their conditions + unpinned. +- **Apparatus self-check passed**: the BF16 round-to-nearest-even helper is + verified to ≤ 2⁻⁹ relative error before any BF16 claim is made + (`verify_apparatus.py`). + +## 5. Honest boundaries + +- **One timestep, one season, one variable** for the headline. 2021-06-15 12:00 + UTC only. The linear-beats-Fisher-Z result should be re-run across seasons and + on a second variable before it is graded `[G]` rather than `[H]`. +- **Climatology is a zonal-mean proxy**, not a real ERA5 climatology. It is the + cheapest honest PLACE term and it is stated as such; a real climatology would + shrink the anomaly further and likely *improve* both paths. +- **Saturation is real and unhidden**: 0.82 % of points land in the two rim + buckets, giving a palette max error of 17.5 K against a 0.068 K mean. That is + `quantize.rs`'s documented controlled-saturation tail, not a surprise — but any + product claim must quote the tail, not just the mean. +- **Fixtures are not committed** (4 MB each, re-fetchable). Scripts + the exact + store object + time index are, which is what makes the run reproducible. +- Nothing here touches Rust, the SoA lanes, or Lance. P1 is the ingest and + measurement half only. diff --git a/probes/weather-p1/export_for_jc.py b/probes/weather-p1/export_for_jc.py new file mode 100644 index 000000000..cf5c46bcc --- /dev/null +++ b/probes/weather-p1/export_for_jc.py @@ -0,0 +1,25 @@ +"""Export (truth, reconstructed) pairs as f64 LE binary for the jc reliability battery.""" +import numpy as np +rng=np.random.default_rng(11) +VARS=['2m_temperature','2m_dewpoint_temperature','10m_u_component_of_wind'] +def load(v): + """Load an ERA5 fixture variable and return its zonal-mean anomaly (K).""" + a=np.load(f'fixture/{v}.npy').astype(np.float64); return a-a.mean(axis=1,keepdims=True) +anom={v:load(v) for v in VARS} +z={v:(anom[v].ravel()-anom[v].mean())/anom[v].std() for v in VARS} +pool=np.concatenate([z[v] for v in VARS]); lo,hi=np.percentile(pool,[0.4,99.6]) +cen=lo+((np.arange(256)+0.5)/256)*(hi-lo) +N=50_000 +cols=[] # each var contributes truth + reconstructed, subsampled on the SAME indices +for v in VARS: + idx=rng.integers(0,z[v].size,N) + t=z[v][idx] + q=np.clip(np.floor((t-lo)/(hi-lo)*256),0,255).astype(np.uint8) + cols.append((v,t,cen[q])) +with open('jc_input.bin','wb') as f: + np.array([len(cols),N],dtype='.npy` (721x1440 f32). Requires `zmeta.json` (the store's +consolidated .zmetadata) alongside this script; see README.md for the fetch. + +A 404 is VALID Zarr v2 semantics -- a missing chunk is entirely `fill_value` +(NaN here), not a fetch failure -- so `get()` returns the fill array and the +run continues. Re-runnable and idempotent; no credentials needed (public +bucket, anonymous HTTPS).""" +import json, urllib.request, numpy as np, numcodecs, datetime as dt, os +B="https://storage.googleapis.com/gcp-public-data-arco-era5/ar/1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr" +meta=json.load(open('zmeta.json'))['metadata'] +# time units +print("time .zattrs:", json.dumps(meta.get('time/.zattrs',{}))[:200]) +epoch=dt.datetime(1959,1,1) +target=dt.datetime(2021,6,15,12) +t=int((target-epoch).total_seconds()//3600) +print(f"target={target} -> time index t={t} (shape0=561264, in range={t<561264})") + +def get(var, t): + """Fetch one chunk. A 404 is VALID Zarr v2 semantics -- a missing chunk means + the chunk is entirely `fill_value` (NaN here), NOT a fetch failure. Several + ARCO-ERA5 variables are legitimately absent at a given timestep; returning + the fill array is what the format specifies (codex P2 on #920).""" + za=meta[f'{var}/.zarray']; comp=za['compressor'] + key=f'{var}/{t}.0.0' + req=urllib.request.Request(f'{B}/{key}') + try: + raw=urllib.request.urlopen(req, timeout=300).read() + except urllib.error.HTTPError as e: + if e.code != 404: + raise + fill=za.get('fill_value') + fill=np.nan if fill is None else fill + a=np.full(za['chunks'][1:], fill, dtype=za['dtype']) + return a, 0, a.nbytes # 0 compressed bytes: nothing is stored + dec=numcodecs.get_codec(comp).decode(raw) + a=np.frombuffer(dec,dtype=za['dtype']).reshape(za['chunks'])[0] + return a, len(raw), a.nbytes + +os.makedirs('fixture',exist_ok=True) +rows=[] +for var in ['2m_temperature','total_column_water_vapour','surface_pressure', + '10m_u_component_of_wind','10m_v_component_of_wind']: + a,ncomp,nraw=get(var,t) + if ncomp==0: + print(f"{var:34s} MISSING CHUNK -> all-fill (valid Zarr semantics); not saved") + continue + np.save(f'fixture/{var}.npy', a) + finite=np.isfinite(a) + rows.append((var,a.shape,a.dtype.str,ncomp,a.nbytes,a.nbytes/ncomp, + float(a[finite].min()),float(a[finite].max()),int((~finite).sum()))) + print(f"{var:34s} shape={a.shape} dtype={a.dtype} comp={ncomp:>9,}B raw={a.nbytes:>9,}B " + f"ratio={a.nbytes/ncomp:5.3f}x min={a[finite].min():12.4f} max={a[finite].max():12.4f} nonfinite={(~finite).sum()}") +json.dump({'time_index':t,'target_utc':target.isoformat(), + 'store':B, + 'vars':{r[0]:{'shape':list(r[1]),'dtype':r[2],'compressed_bytes':r[3], + 'raw_bytes':r[4],'blosc_ratio':r[5],'min':r[6],'max':r[7],'nonfinite':r[8]} for r in rows}}, + open('fixture/manifest.json','w'), indent=2) +tot_c=sum(r[3] for r in rows); tot_r=sum(r[4] for r in rows) +print(f"\nTOTAL compressed={tot_c:,}B raw={tot_r:,}B blosc mean ratio={tot_r/tot_c:.4f}x") diff --git a/probes/weather-p1/fetch_bg.py b/probes/weather-p1/fetch_bg.py new file mode 100644 index 000000000..a10b8acf2 --- /dev/null +++ b/probes/weather-p1/fetch_bg.py @@ -0,0 +1,45 @@ +"""Background companion to `fetch.py` for the P2 multi-variable set. + +Fetches the six cross-variable fields (dewpoint, 10m u/v wind, surface +pressure, total column water vapour, total cloud cover) at the same timestep, +skipping any `fixture/.npy` already on disk and retrying transient +failures up to three times. Same Zarr v2 fill semantics as `fetch.py`: a 404 +means an all-`fill_value` chunk, not an error.""" +import json, urllib.request, numpy as np, numcodecs, os, time +B="https://storage.googleapis.com/gcp-public-data-arco-era5/ar/1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr" +meta=json.load(open('zmeta.json'))['metadata']; t=547476 +os.makedirs('fixture',exist_ok=True) +out={} +for var in ['2m_dewpoint_temperature','10m_u_component_of_wind','10m_v_component_of_wind', + 'surface_pressure','total_column_water_vapour','total_cloud_cover']: + if os.path.exists(f'fixture/{var}.npy'): print('skip',var,flush=True); continue + za=meta[f'{var}/.zarray'] + for attempt in range(3): + try: + t0=time.time() + raw=urllib.request.urlopen(f'{B}/{var}/{t}.0.0', timeout=600).read() + dec=numcodecs.get_codec(za['compressor']).decode(raw) + a=np.frombuffer(dec,dtype=za['dtype']).reshape(za['chunks'])[0] + np.save(f'fixture/{var}.npy', a) + out[var]={'compressed_bytes':len(raw),'raw_bytes':int(a.nbytes), + 'blosc_ratio':a.nbytes/len(raw),'secs':round(time.time()-t0,1), + 'nonfinite':int((~np.isfinite(a)).sum())} + print(f"OK {var} comp={len(raw):,} ratio={a.nbytes/len(raw):.4f} {time.time()-t0:.0f}s",flush=True) + break + except urllib.error.HTTPError as e: + if e.code == 404: + # Valid Zarr v2: a missing chunk IS the fill value. Retrying can + # never succeed, and the absence is a recordable measurement. + out[var]={'status':'missing_chunk_all_fill','http':404, + 'fill_value':za.get('fill_value')} + print(f"FILL {var} (404 = all-fill chunk, not an error)",flush=True) + break + print(f"try{attempt} {var}: HTTPError {e.code}",flush=True) + time.sleep(5) + except Exception as e: + print(f"try{attempt} {var}: {type(e).__name__} {e}",flush=True) + time.sleep(5) + else: + out[var]={'error':'unavailable after 3 tries'} +json.dump(out, open('fixture/fetch_log.json','w'), indent=2) +print("DONE",flush=True) diff --git a/probes/weather-p1/jc_input.bin b/probes/weather-p1/jc_input.bin new file mode 100644 index 000000000..5fe8f6f0c Binary files /dev/null and b/probes/weather-p1/jc_input.bin differ diff --git a/probes/weather-p1/p1_ci_vs_floor.json b/probes/weather-p1/p1_ci_vs_floor.json new file mode 100644 index 000000000..b15871950 --- /dev/null +++ b/probes/weather-p1/p1_ci_vs_floor.json @@ -0,0 +1,58 @@ +{ + "linear": { + "sat_frac": 0.008476845430728927, + "interior_ci_min": 0.09411953077903945, + "interior_ci_med": 0.09411953077904034, + "interior_ci_max": 0.094119530779043, + "floors": { + "0.25": { + "interior_CI_exceeds": 0.0, + "saturated_unbounded": 0.008476845430728927, + "total_at_or_above_floor": 0.008476845430728927 + }, + "0.5": { + "interior_CI_exceeds": 0.0, + "saturated_unbounded": 0.008476845430728927, + "total_at_or_above_floor": 0.008476845430728927 + }, + "1.0": { + "interior_CI_exceeds": 0.0, + "saturated_unbounded": 0.008476845430728927, + "total_at_or_above_floor": 0.008476845430728927 + }, + "2.0": { + "interior_CI_exceeds": 0.0, + "saturated_unbounded": 0.008476845430728927, + "total_at_or_above_floor": 0.008476845430728927 + } + } + }, + "fisher_z": { + "sat_frac": 0.008202342425643397, + "interior_ci_min": 1.0828864116518844e-06, + "interior_ci_med": 0.005607407743958959, + "interior_ci_max": 0.42898798181318165, + "floors": { + "0.25": { + "interior_CI_exceeds": 0.956511018646941, + "saturated_unbounded": 0.008202342425643397, + "total_at_or_above_floor": 0.9647133610725844 + }, + "0.5": { + "interior_CI_exceeds": 0.0, + "saturated_unbounded": 0.008202342425643397, + "total_at_or_above_floor": 0.008202342425643397 + }, + "1.0": { + "interior_CI_exceeds": 0.0, + "saturated_unbounded": 0.008202342425643397, + "total_at_or_above_floor": 0.008202342425643397 + }, + "2.0": { + "interior_CI_exceeds": 0.0, + "saturated_unbounded": 0.008202342425643397, + "total_at_or_above_floor": 0.008202342425643397 + } + } + } +} \ No newline at end of file diff --git a/probes/weather-p1/p1_ci_vs_floor.py b/probes/weather-p1/p1_ci_vs_floor.py new file mode 100644 index 000000000..99d3c3c5d --- /dev/null +++ b/probes/weather-p1/p1_ci_vs_floor.py @@ -0,0 +1,50 @@ +"""P1, CORRECTED metric (codex P1 on #920): compare each point's BUCKET +CONFIDENCE INTERVAL to the floor. Never the decoded error. + +CI(b) = half-width of bucket b expressed in ORIGINAL units (K). +Saturated edge buckets (0, 255) have UNBOUNDED CI and are reported separately, +never folded into an 'exceeds' fraction as if they had a finite width.""" +import numpy as np, json +a=np.load('fixture/2m_temperature.npy').astype(np.float64) +clim=a.mean(axis=1,keepdims=True); anom=a-clim +def fz(s,eps=1e-9): + """Fisher-Z: arctanh(s), clipped off the +/-1 poles by eps.""" + s=np.clip(s,-1+eps,1-eps); return 0.5*(np.log1p(s)-np.log1p(-s)) +FLOORS=[0.25,0.5,1.0,2.0] +out={} + +def report(name, idx, edges_K): + """edges_K: 257 bucket edges in ORIGINAL units. CI(b)=half the bucket width.""" + ci = np.diff(edges_K)/2.0 # 256 half-widths, in K + interior = (idx>0)&(idx<255) + sat = ~interior + occ = np.bincount(idx.ravel(), minlength=256) + print(f"\n{name}") + print(f" saturated (buckets 0/255, CI UNBOUNDED): {sat.mean()*100:.3f}% of points") + print(f" interior CI (K): min={ci[1:255].min():.5f} med={np.median(ci[1:255]):.5f} max={ci[1:255].max():.5f}") + rows={} + for f in FLOORS: + # fraction of ALL points whose own bucket's CI exceeds the floor + exceed = (ci[idx][interior] > f) + frac_int = exceed.sum()/idx.size + rows[str(f)]={'interior_CI_exceeds':float(frac_int), + 'saturated_unbounded':float(sat.mean()), + 'total_at_or_above_floor':float(frac_int+sat.mean())} + print(f" floor {f:>4}K : interior CI>floor {frac_int*100:8.4f}% " + f"+ saturated {sat.mean()*100:.3f}% = {(frac_int+sat.mean())*100:8.4f}%") + out[name]={'sat_frac':float(sat.mean()), + 'interior_ci_min':float(ci[1:255].min()),'interior_ci_med':float(np.median(ci[1:255])), + 'interior_ci_max':float(ci[1:255].max()),'floors':rows} + +# LINEAR: uniform buckets in K +lo,hi=np.percentile(anom,[0.4,99.6]) +idx=np.clip(np.floor((anom-lo)/(hi-lo)*256),0,255).astype(np.uint8) +report('linear', idx, lo+np.arange(257)/256*(hi-lo)) + +# FISHER-Z: uniform in z, NON-uniform in K after tanh +lo_q,hi_q=np.percentile(anom,[0.4,99.6]); scale=max(abs(lo_q),abs(hi_q)) +z=fz(anom/scale); zlo,zhi=np.percentile(z,[0.4,99.6]) +idxz=np.clip(np.floor((z-zlo)/(zhi-zlo)*256),0,255).astype(np.uint8) +report('fisher_z', idxz, np.tanh(zlo+np.arange(257)/256*(zhi-zlo))*scale) + +json.dump(out,open('p1_ci_vs_floor.json','w'),indent=2) diff --git a/probes/weather-p1/p1_noise_floor.json b/probes/weather-p1/p1_noise_floor.json new file mode 100644 index 000000000..d74351e6f --- /dev/null +++ b/probes/weather-p1/p1_noise_floor.json @@ -0,0 +1,29 @@ +{ + "linear": { + "ci_mid_K": 0.09411953077904053, + "exceed_frac": { + "0.25": 0.0076196255201109574, + "0.5": 0.007035945446139621, + "1.0": 0.005992834026814609, + "2.0": 0.004037602095854523 + }, + "max_K": 17.55213988685138 + }, + "fisher_z": { + "ci_mid_K": 0.005604263942934002, + "exceed_frac": { + "0.25": 0.3634737632917245, + "0.5": 0.006904954538449684, + "1.0": 0.005844506087224534, + "2.0": 0.003959585452303899 + }, + "max_K": 17.540985508049452, + "ci_by_position_K": { + "0": 0.08683115654224102, + "64": 0.263179560756829, + "128": 0.005239901491090548, + "192": 6.94262434560565e-05, + "255": 1.0468683395959888e-06 + } + } +} \ No newline at end of file diff --git a/probes/weather-p1/p1_noise_floor.py b/probes/weather-p1/p1_noise_floor.py new file mode 100644 index 000000000..aa2ed03b0 --- /dev/null +++ b/probes/weather-p1/p1_noise_floor.py @@ -0,0 +1,53 @@ +"""SUPERSEDED by p1_ci_vs_floor.py -- kept for provenance only. + +This script's `dev` is |bucket_center - original|, i.e. a DECODED RECONSTRUCTION +ERROR, which is the round-trip metric its own docstring disclaims (codex P1 on +PR #920). Its exceedance fractions fed doc section 12.10 and are superseded by +12.11. The Fisher-Z widths below were also adjacent-CENTRE spacings, not bin +widths, and produced 255 spans for 256 buckets. Both are fixed properly in +p1_ci_vs_floor.py; this file is not the measurement of record. + +Original header follows. + +P1 re-evaluated in the CORRECT frame (operator correction 2026-08-11): +no roundtrip -- compare the code's confidence-interval thresholds against the +original's noise floor. Deviation only matters where it EXCEEDS what the +original itself can resolve.""" +import numpy as np, json +a=np.load('fixture/2m_temperature.npy').astype(np.float64) +clim=a.mean(axis=1,keepdims=True); anom=a-clim +def fz(s,eps=1e-9): + """Fisher-Z: arctanh(s), clipped off the +/-1 poles by eps.""" + s=np.clip(s,-1+eps,1-eps); return 0.5*(np.log1p(s)-np.log1p(-s)) + +paths={} +# LINEAR path +lo,hi=np.percentile(anom,[0.4,99.6]) +idx=np.clip(np.floor((anom-lo)/(hi-lo)*256),0,255).astype(np.uint8) +cen=lo+((np.arange(256)+0.5)/256)*(hi-lo) +paths['linear']=(np.abs(cen[idx]-anom), (hi-lo)/256/2) # dev, half-bucket CI +# FISHER-Z path +lo_q,hi_q=np.percentile(anom,[0.4,99.6]); scale=max(abs(lo_q),abs(hi_q)) +z=fz(anom/scale); zlo,zhi=np.percentile(z,[0.4,99.6]) +idxz=np.clip(np.floor((z-zlo)/(zhi-zlo)*256),0,255).astype(np.uint8) +cenz=zlo+((np.arange(256)+0.5)/256)*(zhi-zlo) +devz=np.abs(np.tanh(cenz)[idxz]*scale-anom) +# CI per bucket from EDGES (257 of them -> 256 spans), not centre spacing. +_edges = zlo + np.arange(257)*(zhi-zlo)/256 +wid = np.abs(np.diff(np.tanh(_edges))*scale) # 256 full widths, in K +paths['fisher_z'] = (devz, None) + +FLOORS=[0.25,0.5,1.0,2.0] # K -- reference class; exact citable floor to pin +print(f"{'path':10s} {'CI_mid(K)':>10s} " + " ".join(f">{f}K".rjust(9) for f in FLOORS) + f" {'max(K)':>8s}") +out={} +for name,(dev,ci) in paths.items(): + ci_mid = ci if ci is not None else float(wid[len(wid)//2]/2) + row=[float((dev>f).mean()) for f in FLOORS] + print(f"{name:10s} {ci_mid:10.4f} " + " ".join(f"{v:9.2e}" for v in row) + f" {dev.max():8.2f}") + out[name]={'ci_mid_K':ci_mid,'exceed_frac':dict(zip(map(str,FLOORS),row)),'max_K':float(dev.max())} +# fisher-z CI across the range (the CI is position-dependent -- that IS the shape finding) +q=[0,64,128,192,255] +print("\nfisher-z per-position CI (half bucket width in K): ", + {f"idx{k}": round(float(wid[k]/2),4) for k in q}) +out['fisher_z']['ci_by_position_K']={str(k): float(wid[k]/2) for k in q} +json.dump(out,open('p1_noise_floor.json','w'),indent=2) diff --git a/probes/weather-p1/p1_probe.py b/probes/weather-p1/p1_probe.py new file mode 100644 index 000000000..fc7d0c82d --- /dev/null +++ b/probes/weather-p1/p1_probe.py @@ -0,0 +1,83 @@ +"""P1 — re-runnable measurement of the doc's §6 claims on REAL ERA5 2m_temperature.""" +import numpy as np, json +a = np.load('fixture/2m_temperature.npy').astype(np.float64) +ny, nx = a.shape +assert np.isfinite(a).all(), "unexpected non-finite" +print(f"field: {a.shape} = {a.size:,} gridpoints, K range [{a.min():.3f}, {a.max():.3f}]") + +def bf16(x): + """Round-to-nearest-even f32->bf16 (truncate mantissa to 7 bits), back to f64.""" + f = np.asarray(x, dtype=np.float32).view(np.uint32) + r = ((f >> 16) & 1).astype(np.uint32) + f2 = ((f + 0x7FFF + r) & 0xFFFF0000).astype(np.uint32) + return f2.view(np.float32).astype(np.float64) + +# ---- §6.3 : BF16 on raw Kelvin vs BF16 on anomaly ------------------------- +# climatology proxy = zonal mean (per-latitude), the cheapest honest PLACE term. +clim = a.mean(axis=1, keepdims=True) # (721,1) +anom = a - clim + +err_raw = np.abs(bf16(a) - a) +err_anom = np.abs((bf16(anom) + clim) - a) +r = {} +r['bf16_raw_kelvin_mae'] = float(err_raw.mean()) +r['bf16_raw_kelvin_p99'] = float(np.percentile(err_raw, 99)) +r['bf16_raw_kelvin_max'] = float(err_raw.max()) +r['bf16_anomaly_mae'] = float(err_anom.mean()) +r['bf16_anomaly_p99'] = float(np.percentile(err_anom, 99)) +r['bf16_anomaly_max'] = float(err_anom.max()) +r['bf16_improvement_x'] = float(err_raw.mean()/err_anom.mean()) +r['anomaly_std_K'] = float(anom.std()) +r['raw_std_K'] = float(a.std()) + +# ---- palette256: fisher-z normalize the anomaly, then 256-bucket ---------- +def fisher_z(s, eps=1e-9): + """Fisher-Z: arctanh(s), clipped off the +/-1 poles by eps.""" + s = np.clip(s, -1+eps, 1-eps) + return 0.5*(np.log1p(s) - np.log1p(-s)) + +# map anomaly onto [-1,1] by a robust scale (0.4-99.6 pct, mirroring RollingFloor.roll) +lo_q, hi_q = np.percentile(anom, [0.4, 99.6]) +scale = max(abs(lo_q), abs(hi_q)) +s = anom/scale +z = fisher_z(s) +zlo, zhi = np.percentile(z, [0.4, 99.6]) +idx = np.clip(np.floor((z - zlo)/(zhi - zlo)*256), 0, 255).astype(np.uint8) # RollingFloor.quantize +centers = zlo + ((np.arange(256)+0.5)/256)*(zhi - zlo) # bucket_center +z_hat = centers[idx] +anom_hat = np.tanh(z_hat)*scale # inverse fisher-z +err_pal = np.abs((anom_hat + clim) - a) +r['palette256_mae_K'] = float(err_pal.mean()) +r['palette256_p99_K'] = float(np.percentile(err_pal, 99)) +r['palette256_max_K'] = float(err_pal.max()) +def effective_buckets(counts): + """exp(Shannon entropy) of the occupancy histogram: how many of the 256 + addresses are REALLY carrying data. Zero-count buckets contribute nothing.""" + q = counts / counts.sum() + nz = q[q > 0] + return float(np.exp(-(nz * np.log(nz)).sum())) + +occ = np.bincount(idx.ravel(), minlength=256) +r['palette_occupancy_min'] = int(occ.min()); r['palette_occupancy_max'] = int(occ.max()) +r['palette256_effective_buckets'] = effective_buckets(occ) +r['palette_empty_buckets'] = int((occ==0).sum()) +r['palette_saturated_frac'] = float((occ[0]+occ[255])/idx.size) +# drift_score exactly as quantize.rs: max |occ_i - mu|/sd, multinomial +n = idx.size; p = 1/256; mu = n*p; sd = (n*p*(1-p))**0.5 +r['drift_score_uniform_sd'] = float(np.abs(occ-mu).max()/sd) + +# LINEAR (no fisher-z) control — does the normalization actually buy anything? +alo, ahi = np.percentile(anom, [0.4, 99.6]) +idx_lin = np.clip(np.floor((anom-alo)/(ahi-alo)*256), 0, 255).astype(np.uint8) +cen_lin = alo + ((np.arange(256)+0.5)/256)*(ahi-alo) +err_lin = np.abs((cen_lin[idx_lin] + clim) - a) +occ_lin = np.bincount(idx_lin.ravel(), minlength=256) +r['palette256_LINEAR_mae_K'] = float(err_lin.mean()) +r['palette256_LINEAR_p99_K'] = float(np.percentile(err_lin,99)) +r['drift_score_LINEAR'] = float(np.abs(occ_lin-mu).max()/sd) +r['palette256_LINEAR_effective_buckets'] = effective_buckets(occ_lin) +r['palette256_LINEAR_empty_buckets'] = int((occ_lin==0).sum()) +r['bytes_per_point_f32'] = 4; r['bytes_per_point_palette'] = 1 + +for k,v in r.items(): print(f" {k:32s} {v}") +json.dump(r, open('p1_results_temperature.json','w'), indent=2) diff --git a/probes/weather-p1/p1_results_temperature.json b/probes/weather-p1/p1_results_temperature.json new file mode 100644 index 000000000..e42edfb27 --- /dev/null +++ b/probes/weather-p1/p1_results_temperature.json @@ -0,0 +1,27 @@ +{ + "bf16_raw_kelvin_mae": 0.4560941918235399, + "bf16_raw_kelvin_p99": 0.98773193359375, + "bf16_raw_kelvin_max": 0.99993896484375, + "bf16_anomaly_mae": 0.006085535607022236, + "bf16_anomaly_p99": 0.04891499000126089, + "bf16_anomaly_max": 0.12496976852418129, + "bf16_improvement_x": 74.94725547201509, + "anomaly_std_K": 6.752569755943374, + "raw_std_K": 22.65130642915137, + "palette256_mae_K": 0.21680033256013134, + "palette256_p99_K": 0.4252813828314288, + "palette256_max_K": 17.54098550804946, + "palette_occupancy_min": 0, + "palette_occupancy_max": 146451, + "palette256_effective_buckets": 28.110039935052257, + "palette_empty_buckets": 76, + "palette_saturated_frac": 0.008202342425643397, + "drift_score_uniform_sd": 2240.355203957119, + "palette256_LINEAR_mae_K": 0.06844932342578529, + "palette256_LINEAR_p99_K": 0.09393152803479154, + "drift_score_LINEAR": 606.5577197023172, + "palette256_LINEAR_effective_buckets": 115.73741280218573, + "palette256_LINEAR_empty_buckets": 0, + "bytes_per_point_f32": 4, + "bytes_per_point_palette": 1 +} \ No newline at end of file diff --git a/probes/weather-p1/p2_probe.py b/probes/weather-p1/p2_probe.py new file mode 100644 index 000000000..b63f12c7e --- /dev/null +++ b/probes/weather-p1/p2_probe.py @@ -0,0 +1,76 @@ +"""P2 (re-scoped per §12.1) — does cross-variable comparability survive +WITHOUT a shared Fisher-Z? Shared canonical floor vs per-variable floors.""" +import numpy as np, json +from scipy import stats as st +rng = np.random.default_rng(7) + +# Pre-registered acceptance threshold. "shared beats per-variable" is NOT a pass: +# a pair can win the comparison and still miss the bar, so both are reported. +TARGET_RHO = 0.9996 + +def load(v): + """Load an ERA5 fixture variable and return its zonal-mean anomaly.""" + a = np.load(f'fixture/{v}.npy').astype(np.float64) + assert np.isfinite(a).all() + return a - a.mean(axis=1, keepdims=True) # zonal-mean climatology proxy + +VARS = ['2m_temperature', '2m_dewpoint_temperature', '10m_u_component_of_wind'] +UNITS = {'2m_temperature':'K','2m_dewpoint_temperature':'K','10m_u_component_of_wind':'m/s'} +anom = {v: load(v) for v in VARS} +for v in VARS: + a = anom[v].ravel() + print(f"{v:30s} unit={UNITS[v]:4s} std={a.std():8.4f} kurt={st.kurtosis(a):+7.3f} " + f"skew={st.skew(a):+7.3f} range=[{a.min():9.3f},{a.max():9.3f}]") + +# ---- the shape-appropriate transform for bell-shaped anomalies = standardize +# (identity after scaling; §12.1 measured arctanh as actively harmful here) +z = {v: (anom[v].ravel() - anom[v].mean()) / anom[v].std() for v in VARS} + +def quant(x, lo, hi): + """Quantize x into 256 uniform buckets over [lo, hi]; return (index, centre).""" + idx = np.clip(np.floor((x - lo)/(hi - lo)*256), 0, 255).astype(np.uint8) + cen = lo + ((np.arange(256)+0.5)/256)*(hi - lo) + return idx, cen + +# SHARED canonical floor: one window over the pooled z of all variables +pool = np.concatenate([z[v] for v in VARS]) +s_lo, s_hi = np.percentile(pool, [0.4, 99.6]) +shared = {v: quant(z[v], s_lo, s_hi) for v in VARS} +# PER-VARIABLE floors: each its own window +per = {v: quant(z[v], *np.percentile(z[v], [0.4, 99.6])) for v in VARS} + +N = 200_000 +res = {'shared_window': [float(s_lo), float(s_hi)], 'n_pairs': N, 'pairs': {}} +print(f"\nshared z-window = [{s_lo:.4f}, {s_hi:.4f}]") +print(f"\n{'pair':<52s} {'rho_shared':>11s} {'rho_pervar':>11s} {'verdict':>9s}") +for i, va in enumerate(VARS): + for vb in VARS[i+1:]: + ia = rng.integers(0, z[va].size, N); ib = rng.integers(0, z[vb].size, N) + truth = np.abs(z[va][ia] - z[vb][ib]) # float z-domain truth + d_sh = np.abs(shared[va][0][ia].astype(np.int32) - shared[vb][0][ib].astype(np.int32)) + d_pv = np.abs(per[va][0][ia].astype(np.int32) - per[vb][0][ib].astype(np.int32)) + r_sh = st.spearmanr(d_sh, truth).statistic + r_pv = st.spearmanr(d_pv, truth).statistic + key = f"{va} x {vb}" + meets = bool(r_sh >= TARGET_RHO) + res['pairs'][key] = {'rho_shared_floor': float(r_sh), 'rho_per_variable_floor': float(r_pv), + 'units': [UNITS[va], UNITS[vb]], + 'target_rho': TARGET_RHO, 'meets_target': meets, + 'shared_beats_per_variable': bool(r_sh > r_pv)} + print(f"{key:<52s} {r_sh:11.6f} {r_pv:11.6f} " + f"{'PASS' if meets else 'BELOW TARGET':>13s} " + f"{'(shared>per-var)' if r_sh>r_pv else '(per-var>=shared)'}") + +# within-variable control: shared floor must not wreck same-variable distance +print(f"\n{'within-variable control':<52s} {'rho_shared':>11s} {'rho_pervar':>11s}") +for v in VARS: + ia = rng.integers(0, z[v].size, N); ib = rng.integers(0, z[v].size, N) + truth = np.abs(z[v][ia] - z[v][ib]) + r_sh = st.spearmanr(np.abs(shared[v][0][ia].astype(np.int32)-shared[v][0][ib].astype(np.int32)), truth).statistic + r_pv = st.spearmanr(np.abs(per[v][0][ia].astype(np.int32)-per[v][0][ib].astype(np.int32)), truth).statistic + occ = np.bincount(shared[v][0], minlength=256); q=occ/occ.sum(); nz=q[q>0] + res['pairs'][f"{v} (within)"] = {'rho_shared_floor': float(r_sh), 'rho_per_variable_floor': float(r_pv), + 'shared_effective_buckets': float(np.exp(-(nz*np.log(nz)).sum())), + 'shared_empty_buckets': int((occ==0).sum())} + print(f"{v:<52s} {r_sh:11.4f} {r_pv:11.4f} shared_eff_buckets={np.exp(-(nz*np.log(nz)).sum()):5.1f} empty={int((occ==0).sum())}") +json.dump(res, open('p2_results.json','w'), indent=2) diff --git a/probes/weather-p1/p2_results.json b/probes/weather-p1/p2_results.json new file mode 100644 index 000000000..1f9c652a3 --- /dev/null +++ b/probes/weather-p1/p2_results.json @@ -0,0 +1,60 @@ +{ + "shared_window": [ + -3.285941867486222, + 3.611137273885975 + ], + "n_pairs": 200000, + "pairs": { + "2m_temperature x 2m_dewpoint_temperature": { + "rho_shared_floor": 0.9995562984081224, + "rho_per_variable_floor": 0.9994263157997967, + "units": [ + "K", + "K" + ], + "target_rho": 0.9996, + "meets_target": false, + "shared_beats_per_variable": true + }, + "2m_temperature x 10m_u_component_of_wind": { + "rho_shared_floor": 0.9997359339313134, + "rho_per_variable_floor": 0.874801168764891, + "units": [ + "K", + "m/s" + ], + "target_rho": 0.9996, + "meets_target": true, + "shared_beats_per_variable": true + }, + "2m_dewpoint_temperature x 10m_u_component_of_wind": { + "rho_shared_floor": 0.9997218339974447, + "rho_per_variable_floor": 0.8569840346681692, + "units": [ + "K", + "m/s" + ], + "target_rho": 0.9996, + "meets_target": true, + "shared_beats_per_variable": true + }, + "2m_temperature (within)": { + "rho_shared_floor": 0.9995571417817292, + "rho_per_variable_floor": 0.999613473800991, + "shared_effective_buckets": 118.63822056822225, + "shared_empty_buckets": 0 + }, + "2m_dewpoint_temperature (within)": { + "rho_shared_floor": 0.9995312723074434, + "rho_per_variable_floor": 0.9996178181789303, + "shared_effective_buckets": 120.73321025144813, + "shared_empty_buckets": 0 + }, + "10m_u_component_of_wind (within)": { + "rho_shared_floor": 0.9998171215242767, + "rho_per_variable_floor": 0.9996220613759306, + "shared_effective_buckets": 151.00412788678312, + "shared_empty_buckets": 0 + } + } +} \ No newline at end of file diff --git a/probes/weather-p1/verify_apparatus.py b/probes/weather-p1/verify_apparatus.py new file mode 100644 index 000000000..88e55903b --- /dev/null +++ b/probes/weather-p1/verify_apparatus.py @@ -0,0 +1,54 @@ +"""Verify the P1 apparatus BEFORE claiming fisher-z is falsified.""" +import sys +import numpy as np +from scipy import stats as st +a=np.load('fixture/2m_temperature.npy').astype(np.float64) +clim=a.mean(axis=1,keepdims=True); anom=a-clim + +# 1) sanity: bf16 implementation must have ~8 bits mantissa -> rel err <= 2^-9 +def bf16(x): + """Round f64 to bfloat16 precision (round-to-nearest-even) and back to f64.""" + f=np.asarray(x,dtype=np.float32).view(np.uint32) + r=((f>>16)&1).astype(np.uint32) + return (((f+0x7FFF+r)&0xFFFF0000).astype(np.uint32)).view(np.float32).astype(np.float64) +v=np.array([1.0,300.0,-6.5,1e-3]); rel=np.abs(bf16(v)-v)/np.abs(v) +ok = bool((rel <= 2**-9 + 1e-12).all()) +print("bf16 rel err (must be <= 2^-9 = 0.00195):", rel, "OK" if ok else "BROKEN") +if not ok: + # A self-check that prints BROKEN and keeps going is not a self-check. + # Every BF16 number downstream would be unvalidated. Fail loudly instead. + sys.exit("APPARATUS FAILURE: bf16 helper exceeds 2^-9; refusing to emit measurements") + +# 2) THE MECHANISM: what shape is the input distribution? +lo_q,hi_q=np.percentile(anom,[0.4,99.6]); scale=max(abs(lo_q),abs(hi_q)); s=(anom/scale).ravel() +print(f"\nanomaly/scale -> |s| distribution: mean|s|={np.abs(s).mean():.4f} " + f"frac|s|<0.25={np.mean(np.abs(s)<0.25):.4f} frac|s|>0.9={np.mean(np.abs(s)>0.9):.4f}") +print(f" kurtosis(s)={st.kurtosis(s):.3f} (0=gaussian, <0=flat, >0=peaked) skew={st.skew(s):.3f}") + +# a correlation-like variable for CONTRAST: piles at the bounds +r_like=np.tanh(np.random.default_rng(0).normal(0,1.5,200000)) +print(f" CONTRAST corr-like: mean|r|={np.abs(r_like).mean():.4f} " + f"frac|r|<0.25={np.mean(np.abs(r_like)<0.25):.4f} frac|r|>0.9={np.mean(np.abs(r_like)>0.9):.4f}") + +# 3) where do the 256 buckets actually land, both ways? +def quant(x): + """Quantize x into 256 uniform buckets over its range; return (index, centre).""" + lo,hi=np.percentile(x,[0.4,99.6]) + idx=np.clip(np.floor((x-lo)/(hi-lo)*256),0,255).astype(np.uint8) + cen=lo+((np.arange(256)+0.5)/256)*(hi-lo) + return idx,cen +def fz(s,eps=1e-9): + """Fisher-Z: arctanh(s), clipped off the +/-1 poles by eps.""" + s=np.clip(s,-1+eps,1-eps); return 0.5*(np.log1p(s)-np.log1p(-s)) + +for name,x,inv in [("LINEAR(anom)",anom.ravel(),lambda c:c), + ("FISHERZ(s)",fz(s),lambda c:np.tanh(c)*scale)]: + idx,cen=quant(x); occ=np.bincount(idx,minlength=256) + rec=inv(cen)[idx] + truth=anom.ravel() if name.startswith("LINEAR") else anom.ravel() + err=np.abs(rec-truth) + n=idx.size;p=1/256;mu=n*p;sd=(n*p*(1-p))**.5 + # effective buckets = exp(entropy) : how many buckets are REALLY used + q=occ/occ.sum(); nz=q[q>0]; eff=float(np.exp(-(nz*np.log(nz)).sum())) + print(f"\n{name}: mae={err.mean():.5f}K p99={np.percentile(err,99):.5f}K " + f"empty={int((occ==0).sum())} effective_buckets={eff:.1f}/256 drift={np.abs(occ-mu).max()/sd:.0f}")