From 5ab462642efeb3fb17ecd6a6f54645c0be9d13f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 10:00:13 +0000 Subject: [PATCH 1/5] D-WXS-3b SHIPPED: the L4 lane -- and a disable probe that can itself be vacuous The lane: pack/unpack ONE 16-byte facet under layout L4 -- a 4-byte classid prefix plus 6 pairs of two SEPARATE bytes (u8:u8, never widened), each byte one ERA5 field quantised by a CalibratedFloor. 33/33 crate-wide, fmt clean, clippy -D warnings clean. The property the file exists to hold is le-contract section 2: labels and positions come from the ClassView, never from a slot in the payload. So the lane names no ERA5 variable in its own source -- it asks the manifest for the slot and hands the entry back out through a caller-owned closure, which is where variable knowledge is allowed to live. A `match` on a variable name inside lane.rs would be the defect the file was written to prevent. Reserved slots are zero on pack and ABSENT on unpack -- not decoded to bucket_center(0). That distinction is the point: a reserved slot must not read back as a plausible number. A plan gap found while filing: the lane had a worker in section 6.2 but NO D-id in section 4's ladder, which jumped D-WXS-3 to D-WXS-4. Added as D-WXS-3b rather than folded silently into a neighbour. D-WXS-4 (the bake) is annotated as blocked behind D-WXS-0, the classid mint. Four disables, each verified to kill the expected tests and no others: lo/hi swapped -> the swap test; slot hard-coded -> 3 tests including the manifest-load-bearing one; version guard bypassed -> the version test; unmapped slots made to emit values -> the reserved-slot test. EPIPHANIES E-A-DISABLE-PROBE-CAN-ITSELF-BE-VACUOUS-1 records the method finding underneath that list. Three of my probes this session did not probe: one searched for a variant name that does not exist (ManifestError::DuplicateSlot; the real one is SlotCollision) so the substitution aborted and the run that followed executed unmodified code and reported 25 passed; one inserted dead code that applied cleanly and bound nothing; one targeted a code path the loop can never reach for the case under test. The workspace already holds "an assertion implied by the code it tests is not a test" and "turning a knob that does not bind is not a disable" -- both stated about TESTS. The same failure applies to the VERIFICATION PROBE, and is harder to see there, because a broken probe and a passing suite look identical. The signature that separates them: a correct disable kills at least one test, usually a small nameable set. A disable that kills ZERO tests is more likely a broken probe than a missing guard -- treat zero as re-check-the-probe, never as verified. Mechanical fix now in use: every substitution asserts its pattern was found and the file actually changed, so a wrong symbol name is an error instead of a green run. This matters because the whole disable-the-fix discipline exists to establish that a passing test COULD fail. An unchecked probe gives that discipline an unverified root and hands back exactly the confidence it was built to withdraw. Board: STATUS_BOARD gains D-WXS-3b and the D-WXS-4 blocker annotation; EPIPHANIES prepended, suffix-verified. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/EPIPHANIES.md | 57 +++ .claude/board/STATUS_BOARD.md | 3 +- crates/weather-poc/src/lane.rs | 834 +++++++++++++++++++++++++++++++++ crates/weather-poc/src/lib.rs | 1 + 4 files changed, 894 insertions(+), 1 deletion(-) create mode 100644 crates/weather-poc/src/lane.rs diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 64213ed3..63cbf836 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,60 @@ +## 2026-08-13 — E-A-DISABLE-PROBE-CAN-ITSELF-BE-VACUOUS-1 + +**Status:** FINDING `[G]` — three measured instances in one session, all mine. +**Confidence:** High. Method-level; no code claim. + +**The known rule it extends.** This workspace already holds *"an assertion +implied by the code it tests is not a test"* (`CLAUDE.md` § falsifiability rule) +and, in the sibling repo's words, *"turning a knob that does not bind is not a +disable."* Both are stated about **tests**. This entry records that the same +failure applies one level up — to the **verification probe** that is supposed to +prove a test can fail — and that it is harder to spot there, because a broken +probe and a passing suite look identical. + +**The three instances, same session, gating `crates/weather-poc`.** + +1. **Wrong symbol name.** A probe searched for `ManifestError::DuplicateSlot`; + the real variant is `SlotCollision`. The substitution script aborted, the + test run afterwards executed **unmodified code**, and reported `25 passed`. + Read casually, that is a passing disable-verification of a guard that was + never touched. +2. **Dead code.** A probe inserted an `if` block computing `lo`/`hi` and + discarding both (`let _ = (lo, hi);`). It applied cleanly and changed + nothing. `25 passed` again. +3. **Wrong target.** A probe changed `raw` to `raw.max(1)` intending to make + reserved slots decode — but the unpack loop only visits **manifest-resolved** + slots, so the edit could never reach a reserved one. `33 passed`. + +Instance 1 is the dangerous one: 2 and 3 at least ran, while 1 silently did not. + +**The signature that separates the two causes.** A disable run that stays green +has two possible explanations — *the guard is absent* or *the probe never +touched it* — and greenness alone does not distinguish them. What does: a +correct disable kills **at least one** test, and usually a small, nameable set. + +> **A disable that kills ZERO tests is more likely a broken probe than a missing +> guard.** Treat zero as "re-check the probe", never as "verified". + +Corollary, the mechanical fix now in use: **the probe must assert that it +applied.** Every substitution asserts its pattern was found and the file +actually changed, and fails loudly otherwise — so instance 1 becomes an error +instead of a green run. + +**Why this is worth a board entry rather than a shrug.** The whole +disable-the-fix discipline exists because a passing test proves nothing about +whether it *could* fail. If the probe that establishes that is itself unchecked, +the discipline has an unverified root and inherits exactly the confidence it was +built to withdraw. Three instances in one session, by an operator applying the +rule deliberately, is the measured argument that the root needs checking too. + +**Cross-ref:** `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1`, +`E-A-CONTROL-THAT-CANNOT-LOSE-IS-NO-CONTROL-1`, +`E-ANTI-EIGENVALUE-MACHINERY-CAN-ITSELF-BECOME-THE-EIGENVALUE-1` (the same +one-level-up move, applied there to guards rather than probes); +`CLAUDE.md` § The falsifiability rule. + +--- + ## 2026-08-12 — E-THE-REGIME-LADDER-MEASURED-RANGE-NOT-TURBULENCE-1 **Status:** FINDING `[G]` — measured, same run, found by an operator diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index c302ba1b..26eec430 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -14,7 +14,8 @@ with a control that can lose and a stay-silent twin. | D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | **SHIPPED 2026-08-13** — `key.rs`, 5/5 green; exhaustive 1,038,240-cell round-trip + collision-free; both bar-B1 halves **disable-verified** by the orchestrator (zeroing the HIP lat byte kills 3 tests incl. collision + ragged; removing the seam split kills the wrap twin while the non-wrap twin stays green) | a 16° box becomes a HEEL-prefix scan; bar B1 | | D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | Queued | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | | D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`, 7/7; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted | bar B2 | -| D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued | bar B3; the missing path | +| D-WXS-3b | **NEW — the L4 lane (pack/unpack ONE 16-byte facet).** The plan gave the lane a worker in §6.2 but **no D-id in §4's ladder** — it jumped D-WXS-3 → D-WXS-4. Added here as the pack/unpack half the bake will call | 1 | **SHIPPED 2026-08-13** — `lane.rs`, 33/33 crate-wide; 4 disables verified by the orchestrator (lo/hi swap → the swap test; hard-coded slot → 3 tests incl. manifest-load-bearing; version guard bypassed → the version test; unmapped slots emitting values → the reserved-slot test). The lane names no ERA5 variable in its own source — the caller's closure owns that | precursor to bar B3; §2.6 slot purity as code | +| D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued — **blocked behind D-WXS-0** (must refuse to write without a minted classid) | bar B3; the missing path | | D-WXS-5 | statics bake — separate classid, separate dataset, exactly ONE version | 1 | Queued | bar B4; avoids ~1.3 PB of rewritten constants | | D-WXS-6 | version-range read (`QueryReference::at(v,rung)` + `deinterlace`) + version-count scaling measurement | 2 | Queued | bar B5; KILL if growth is superlinear at 92,044 versions | | D-WXS-7 | **D-WXA-5 re-homed and RE-SPECIFIED** — ρ(code_dist, field_dist) via `jc::reliability::spearman` over whole-grid pairs. (a) ρ ≥ 0.9996 (the bar a real pair FAILED at 0.999556); (b) shuffled-codebook control < 0.98 (measured losable at 0.003–0.159); (c) 16/64/256-level ladder must be MONOTONE before any verdict | 3 | Queued | ⚠ poc-v2's ρ ≥ 0.98 is at risk of being vacuous — D-CZ-1 §6.4 measured real-arm ρ spread 3e-6…4.7e-5 | diff --git a/crates/weather-poc/src/lane.rs b/crates/weather-poc/src/lane.rs new file mode 100644 index 00000000..36bd1fd8 --- /dev/null +++ b/crates/weather-poc/src/lane.rs @@ -0,0 +1,834 @@ +//! The L4 lane: pack and unpack ONE 16-byte facet — 4-byte classid prefix + +//! 12-byte payload as `6 x (8:8)` palette pairs +//! (`.claude/v3/soa_layout/le-contract.md` §1/§2/§3's L4 row). +//! +//! Owner: the lane worker (plan `.claude/plans/weather-soa-bake-v1.md` §6.2, +//! deliverable — this file, `D-WXS-4`'s row-assembly companion). Consumes +//! [`crate::manifest`] (the ClassView-side field manifest) and +//! [`crate::floor`] (the calibrated linear quantiser); invents nothing new +//! architecturally. +//! +//! # Slot purity (le-contract.md §2) — restated in this module's own words +//! +//! **Labels and positions come from the ClassView, NEVER from a slot in the +//! payload.** A `NodeRow`'s value slab is dumb bytes; the class makes them +//! meaningful. This module is the code half of that split: it moves bytes +//! according to a [`crate::manifest::FieldManifest`] it was HANDED — it does +//! not know, and must never come to know, which ERA5 variable rides which +//! byte. There is no `match` on a variable name anywhere in this file, and +//! there must never be one. If a future change to this module needs to name +//! a specific ERA5 field to work correctly, that is the exact defect this +//! module exists to prevent — the mapping belongs in the manifest, not here. +//! +//! **The consequence, stated plainly: nothing this file writes ever encodes +//! what a byte means.** [`pack_facet`] and [`unpack_facet`] both take the +//! manifest and the calibrated floors as opaque parameters and follow +//! whatever they say. A caller who wants `v_component_of_wind @ 850 hPa` in +//! byte 5 today and byte 9 tomorrow only ever has to edit +//! `data/field_manifest_v1.tsv` — this module's behaviour follows without a +//! single line changing here. [`crate::manifest`]'s own bar B0 (mutating one +//! manifest entry must change the bytes the bake writes for at least one +//! cell) is exercised end-to-end through this module's tests, closing the +//! half `manifest.rs`'s own module docs left open ("the bake does not exist +//! yet"). +//! +//! # The layout, byte-for-byte +//! +//! | bytes | content | +//! |---|---| +//! | `0..4` | `classid: u32`, little-endian — a caller-supplied parameter, never composed or bit-sliced here | +//! | `4..16` | the 96-bit payload: 6 pairs, `payload[2p]` = pair `p`'s LOW byte, `payload[2p+1]` = pair `p`'s HIGH byte | +//! +//! `u8:u8` means **two separate bytes** — a pair is never combined into a +//! `u16`, never byte-swapped, and the low/high assignment is fixed by the +//! manifest's own `PairByte::{Lo, Hi}` convention (`manifest.rs`'s "lo/hi +//! convention" doc section), never inferred from magnitude at pack or +//! unpack time. +//! +//! # A reserved slot is a slot the manifest has no row for +//! +//! `crate::manifest`'s own docs: "reserved slots are NOT rows... their +//! absence IS their meaning: dormant, expandable later without a layout +//! change." This module enforces the two halves of that on the wire: +//! [`pack_facet`] leaves an unresolved `(facet, pair, byte)` slot at its +//! initial `0`, and [`unpack_facet`] never emits an [`UnpackedField`] for +//! one — a reserved slot is *absent* from the unpacked result, never +//! present with the decoded value of a stray `0` byte (`bucket_center(0)` +//! is a real, plausible-looking number for most floors; reporting it for an +//! unoccupied slot would be exactly the corruption this split prevents). +//! +//! # Floor-version provenance is external, per le-contract.md §2.6 point 2 +//! +//! The plan is explicit that a floor's `(lo, hi, floor_version)` is +//! **dataset metadata, aligned to the Lance version boundary** — never a +//! per-row byte, which would itself be a slot-purity break. This module +//! therefore does not stamp a version into the row. [`stamped_floor_versions`] +//! stands in for "what a reader would find in that dataset metadata": the +//! `floor_version` each occupied slot's floor carried at the moment the +//! facet was packed. [`unpack_facet`] takes that map as an explicit +//! parameter and refuses (via [`crate::floor::CalibratedFloor::decode`]) to +//! resolve any slot whose stamped version has drifted from the floor it was +//! actually handed — a version mismatch is a reported error, never a +//! silently wrong decoded value. +//! +//! # Scope — what this module is NOT +//! +//! This is one 16-byte facet. It is not the bake (`D-WXS-4`, blocked on the +//! `D-WXS-0` classid mint) and not the 512-byte row assembly (three facets +//! plus the key and edge block). Both are deferred to their own +//! deliverables; this module's job ends at one facet in, one facet out. + +use std::collections::HashMap; + +use crate::floor::CalibratedFloor; +use crate::manifest::{FieldManifest, ManifestEntry, PairByte}; + +/// Length in bytes of one L4 facet: 4-byte classid prefix + 12-byte payload +/// (le-contract.md §1). +pub const FACET_LEN: usize = 16; + +/// Length in bytes of the classid prefix (le-contract.md §1, bytes `0..4`). +const CLASSID_LEN: usize = 4; + +/// Number of `(8:8)` pairs in the L4 payload (le-contract.md §3, "6 x +/// (8:8)"). +pub const PAIR_COUNT: u8 = 6; + +const _: () = assert!(FACET_LEN == CLASSID_LEN + PAIR_COUNT as usize * 2); + +/// The byte offset **within a 16-byte facet** that a given `(pair, byte)` +/// slot occupies. +/// +/// This is a purely structural fact about the L4 layout (le-contract.md +/// §3: "every layout is exactly 12 bytes — 6x2... layouts differ only in +/// how the 96 payload bits subdivide") — it is not a label, a variable +/// name, or a display position, so exposing it does not break slot +/// purity: it answers "where is pair `p`'s low/high byte", never "what is +/// stored there". +/// +/// # Panics +/// +/// Panics if `pair >= `[`PAIR_COUNT`]. Every caller in this module reaches +/// this function only through a `0..PAIR_COUNT` loop, so the panic path is +/// unreachable in practice; it exists as a loud guard against a future +/// caller passing an out-of-range pair directly. +pub fn slot_offset(pair: u8, byte: PairByte) -> usize { + assert!( + pair < PAIR_COUNT, + "pair {pair} out of range (0..{PAIR_COUNT})" + ); + let byte_idx = match byte { + PairByte::Lo => 0, + PairByte::Hi => 1, + }; + CLASSID_LEN + (pair as usize) * 2 + byte_idx +} + +/// One resolved field read back out of a packed facet: the slot it +/// occupied, its manifest identity, and its dequantised value. +/// +/// Only ever produced by [`unpack_facet`] for a slot the manifest actually +/// resolves — a reserved-zero slot never produces one of these (see the +/// module docs). +#[derive(Debug, Clone, PartialEq)] +pub struct UnpackedField { + /// The facet index this field was read from. + pub facet: u8, + /// The pair index within the facet. + pub pair: u8, + /// Which byte of the pair. + pub byte: PairByte, + /// The ERA5 variable name, copied from the manifest entry that + /// resolved this slot (never inferred from the byte value itself). + pub variable: String, + /// The pressure level, copied from the manifest entry. + pub level_hpa: Option, + /// The dequantised value (`CalibratedFloor::bucket_center` of the raw + /// byte, under the floor whose stamped version matched). + pub value: f64, +} + +/// Everything that can go wrong packing or unpacking an L4 facet. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum LaneError { + /// An occupied manifest slot names a `floor_id` that is not present in + /// the `floors` map handed to [`pack_facet`], [`unpack_facet`], or + /// [`stamped_floor_versions`]. + UnknownFloor { + /// The facet the missing floor was needed for. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The `floor_id` the manifest entry named. + floor_id: String, + }, + /// [`pack_facet`]'s `value_of` closure returned `None` for an occupied + /// slot — every occupied slot in the target facet must have a value; a + /// caller with a genuinely missing reading should not call `pack_facet` + /// for that facet at all, or should thread a documented sentinel + /// through its own value source rather than lean on the lane to invent + /// a default. + MissingValue { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The variable the manifest names for this slot. + variable: String, + /// The level the manifest names for this slot. + level_hpa: Option, + }, + /// [`unpack_facet`]'s `stamped_versions` map has no entry for a + /// `floor_id` an occupied slot needs — the caller's "dataset metadata" + /// is incomplete for this facet. + MissingStampedVersion { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The `floor_id` with no stamped version. + floor_id: String, + }, + /// A slot's stamped `floor_version` (the version recorded at pack time, + /// per the module docs' "dataset metadata" framing) does not match the + /// `floor_version` the [`CalibratedFloor`] handed to [`unpack_facet`] + /// actually carries. Detected via + /// [`CalibratedFloor::decode`](crate::floor::CalibratedFloor::decode) + /// returning `None` — never silently dequantised under the wrong + /// `[lo, hi]` window. + FloorVersionMismatch { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The `floor_id` whose version drifted. + floor_id: String, + /// The version stamped in `stamped_versions`. + expected: u64, + /// The version the floor handed to `unpack_facet` actually has. + found: u64, + }, +} + +impl std::fmt::Display for LaneError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: no floor for floor_id {floor_id:?}" + ), + LaneError::MissingValue { + facet, + pair, + byte, + variable, + level_hpa, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: no value supplied for {variable:?} at level {level_hpa:?}" + ), + LaneError::MissingStampedVersion { + facet, + pair, + byte, + floor_id, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: no stamped floor_version for floor_id {floor_id:?}" + ), + LaneError::FloorVersionMismatch { + facet, + pair, + byte, + floor_id, + expected, + found, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: floor_id {floor_id:?} stamped version {expected} does not match floor's actual version {found}" + ), + } + } +} + +impl std::error::Error for LaneError {} + +/// The six `(pair, byte)` slots of one facet, in a fixed, deterministic +/// order (`pair` ascending, `Lo` before `Hi` within a pair). Both +/// [`pack_facet`] and [`unpack_facet`] walk this same order, which is what +/// makes the manifest — not iteration order — the only thing either +/// function's output depends on. +fn all_slots() -> impl Iterator { + (0..PAIR_COUNT).flat_map(|pair| [(pair, PairByte::Lo), (pair, PairByte::Hi)]) +} + +/// Packs ONE facet: for every occupied `(pair, byte)` slot the manifest +/// resolves against `facet`, quantises the value `value_of` supplies +/// (through the floor the manifest names) into that slot's byte; every +/// unresolved slot stays `0` (reserved-zero, see the module docs). +/// +/// `classid` is written verbatim as little-endian bytes into `0..4` — this +/// function never composes, mints, or bit-slices it. +/// +/// `value_of` is called once per occupied slot, in the fixed order +/// [`all_slots`] walks, with the [`ManifestEntry`] that slot resolved to. +/// It returns the raw `f64` reading for that field, or `None` if none is +/// available — which is reported as [`LaneError::MissingValue`], not +/// silently treated as zero (a `0.0` reading and a missing reading are +/// different facts, and conflating them would corrupt the bucket the +/// floor quantises `0.0` to). +/// +/// # Errors +/// +/// Returns [`LaneError::UnknownFloor`] if an occupied slot's `floor_id` is +/// not in `floors`, or [`LaneError::MissingValue`] if `value_of` returns +/// `None` for an occupied slot. Fails on the first such slot encountered +/// (in [`all_slots`] order); a facet with several bad slots reports the +/// first one, not all of them. +pub fn pack_facet( + classid: u32, + facet: u8, + manifest: &FieldManifest, + floors: &HashMap, + mut value_of: F, +) -> Result<[u8; FACET_LEN], LaneError> +where + F: FnMut(&ManifestEntry) -> Option, +{ + let mut bytes = [0u8; FACET_LEN]; + bytes[0..CLASSID_LEN].copy_from_slice(&classid.to_le_bytes()); + + for (pair, byte) in all_slots() { + let Some(entry) = manifest.resolve_slot(facet, pair, byte) else { + // Reserved-zero slot: no manifest entry, leave the byte at its + // initial 0. This IS the slot's meaning (dormant, expandable), + // never an omission to fill in. + continue; + }; + + let floor = floors + .get(&entry.floor_id) + .ok_or_else(|| LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + let value = value_of(entry).ok_or_else(|| LaneError::MissingValue { + facet, + pair, + byte, + variable: entry.variable.clone(), + level_hpa: entry.level_hpa, + })?; + + bytes[slot_offset(pair, byte)] = floor.quantize(value); + } + + Ok(bytes) +} + +/// Unpacks ONE facet: for every `(pair, byte)` slot `manifest` resolves +/// against `facet`, dequantises the raw byte through the matching floor — +/// gated on `stamped_versions` agreeing with that floor's actual +/// `floor_version` — and returns it as an [`UnpackedField`]. A slot the +/// manifest does not resolve is skipped entirely: it produces no +/// `UnpackedField`, reserved or not, whatever raw byte value it holds. +/// +/// # Errors +/// +/// Returns [`LaneError::UnknownFloor`] if an occupied slot's `floor_id` is +/// not in `floors`, [`LaneError::MissingStampedVersion`] if it is not in +/// `stamped_versions`, or [`LaneError::FloorVersionMismatch`] if the +/// stamped version does not match the floor's own +/// [`CalibratedFloor::floor_version`](crate::floor::CalibratedFloor::floor_version) +/// (checked via [`CalibratedFloor::decode`](crate::floor::CalibratedFloor::decode), +/// never bypassed). Fails on the first such slot, in [`all_slots`] order. +pub fn unpack_facet( + bytes: &[u8; FACET_LEN], + facet: u8, + manifest: &FieldManifest, + floors: &HashMap, + stamped_versions: &HashMap, +) -> Result, LaneError> { + let mut out = Vec::new(); + + for (pair, byte) in all_slots() { + let Some(entry) = manifest.resolve_slot(facet, pair, byte) else { + continue; + }; + + let floor = floors + .get(&entry.floor_id) + .ok_or_else(|| LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + let expected_version = stamped_versions + .get(&entry.floor_id) + .copied() + .ok_or_else(|| LaneError::MissingStampedVersion { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + let raw = bytes[slot_offset(pair, byte)]; + let value = + floor + .decode(raw, expected_version) + .ok_or_else(|| LaneError::FloorVersionMismatch { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + expected: expected_version, + found: floor.floor_version(), + })?; + + out.push(UnpackedField { + facet, + pair, + byte, + variable: entry.variable.clone(), + level_hpa: entry.level_hpa, + value, + }); + } + + Ok(out) +} + +/// Collects the `floor_version` every occupied slot of `facet` actually +/// used, keyed by `floor_id`. +/// +/// This stands in for the "dataset metadata, aligned to the Lance version +/// boundary" the plan describes (module docs, "Floor-version provenance is +/// external") — in the real bake it would be read from that metadata; here +/// it is derived directly from the same `floors` map [`pack_facet`] was +/// given, so a caller can produce it immediately after packing and hand it +/// to [`unpack_facet`] as the "what was this written under" record. +/// +/// # Errors +/// +/// Returns [`LaneError::UnknownFloor`] under the same condition +/// [`pack_facet`] would have failed under — an occupied slot names a +/// `floor_id` not present in `floors`. +pub fn stamped_floor_versions( + manifest: &FieldManifest, + facet: u8, + floors: &HashMap, +) -> Result, LaneError> { + let mut out = HashMap::new(); + + for (pair, byte) in all_slots() { + let Some(entry) = manifest.resolve_slot(facet, pair, byte) else { + continue; + }; + + let floor = floors + .get(&entry.floor_id) + .ok_or_else(|| LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + out.insert(entry.floor_id.clone(), floor.floor_version()); + } + + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::floor::calibrate; + + /// The exact manifest header, spelled literally rather than imported — + /// `manifest::HEADER` is a private constant of that module (by design: + /// it is validated internally, not a public API surface). + const HEADER: &str = "facet\tpair\tbyte\tvariable\tlevel_hpa\tunit\tfloor_id"; + + /// A small, self-contained fixture manifest for this module's tests: + /// facet 0 only, three occupied slots spread across two pairs, leaving + /// both a fully-reserved pair (2) and a half-reserved pair (1, hi byte) + /// to exercise the reserved-slot behaviour without depending on the + /// full committed W1 manifest. + /// + /// | facet | pair | byte | variable | floor_id | + /// |---|---|---|---|---| + /// | 0 | 0 | lo | fieldA | fa | + /// | 0 | 0 | hi | fieldB | fb | + /// | 0 | 1 | lo | fieldC | fa | + /// | 0 | 1 | hi | *(reserved)* | | + /// | 0 | 2..5 | * | *(fully reserved)* | | + fn fixture_manifest() -> FieldManifest { + let text = format!( + "{HEADER}\n\ + 0\t0\tlo\tfieldA\tsurface\tK\tfa\n\ + 0\t0\thi\tfieldB\tsurface\tPa\tfb\n\ + 0\t1\tlo\tfieldC\tsurface\tK\tfa\n" + ); + FieldManifest::parse(&text).expect("fixture manifest must parse") + } + + /// Deterministic evenly-spaced sample, mirroring `floor.rs`'s own test + /// helper (private to that module, so re-declared here). + fn linspace(lo: f64, hi: f64, n: usize) -> Vec { + assert!(n >= 2, "linspace needs at least 2 points"); + (0..n) + .map(|i| lo + (hi - lo) * i as f64 / (n as f64 - 1.0)) + .collect() + } + + /// Two distinctly-ranged floors, keyed exactly as [`fixture_manifest`]'s + /// entries name them (`"fa"`, `"fb"`). + fn fixture_floors() -> HashMap { + let mut floors = HashMap::new(); + floors.insert( + "fa".to_string(), + calibrate(&linspace(-100.0, 100.0, 10_000)).expect("fa calibrates"), + ); + floors.insert( + "fb".to_string(), + calibrate(&linspace(900.0, 1_100.0, 10_000)).expect("fb calibrates"), + ); + floors + } + + /// A `value_of` closure over the three fixture fields, matched by + /// variable name (the closure — supplied by the TEST, not by this + /// module — is where variable-name knowledge is allowed to live; the + /// module under test never inspects `entry.variable` itself to decide + /// what to do, only to report it back to the caller). + fn fixture_values( + field_a: f64, + field_b: f64, + field_c: f64, + ) -> impl FnMut(&ManifestEntry) -> Option { + move |entry: &ManifestEntry| match entry.variable.as_str() { + "fieldA" => Some(field_a), + "fieldB" => Some(field_b), + "fieldC" => Some(field_c), + _ => None, + } + } + + // ── round-trip ─────────────────────────────────────────────────────── + + #[test] + fn round_trip_recovers_every_occupied_field_within_half_a_bucket() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + let field_a = 12.3_f64; + let field_b = 1_005.7_f64; + let field_c = -42.0_f64; + + let bytes = pack_facet( + 0xDEAD_BEEF, + 0, + &manifest, + &floors, + fixture_values(field_a, field_b, field_c), + ) + .expect("pack must succeed"); + + let stamped = stamped_floor_versions(&manifest, 0, &floors).expect("stamping must succeed"); + let unpacked = + unpack_facet(&bytes, 0, &manifest, &floors, &stamped).expect("unpack must succeed"); + + assert_eq!(unpacked.len(), 3, "exactly the three occupied slots"); + + let expect_value = |variable: &str| match variable { + "fieldA" => field_a, + "fieldB" => field_b, + "fieldC" => field_c, + other => panic!("unexpected variable in unpacked output: {other}"), + }; + + for field in &unpacked { + let entry = manifest + .resolve_slot(field.facet, field.pair, field.byte) + .expect("unpacked field must resolve back to a manifest entry"); + let floor = floors + .get(&entry.floor_id) + .expect("resolved entry's floor_id must be in the fixture floors map"); + let (lo, hi) = floor.bounds(); + let half_bucket = (hi - lo) / (2.0 * 256.0); + let original = expect_value(&field.variable); + let delta = (field.value - original).abs(); + assert!( + delta <= half_bucket + 1e-9, + "{}: round-trip error {delta} exceeds the +/-half-bucket bound {half_bucket}", + field.variable + ); + } + } + + // ── reserved slots stay zero AND read as absent ───────────────────── + + #[test] + fn reserved_slots_are_byte_zero_after_pack_and_absent_after_unpack() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + let bytes = pack_facet(0x1, 0, &manifest, &floors, fixture_values(1.0, 2.0, 3.0)) + .expect("pack must succeed"); + + // Fully-reserved pair (pair 2, both bytes) is byte-zero. + assert_eq!(bytes[slot_offset(2, PairByte::Lo)], 0); + assert_eq!(bytes[slot_offset(2, PairByte::Hi)], 0); + // Half-reserved pair (pair 1, hi byte only) is also byte-zero. + assert_eq!(bytes[slot_offset(1, PairByte::Hi)], 0); + + let stamped = stamped_floor_versions(&manifest, 0, &floors).expect("stamping must succeed"); + let unpacked = + unpack_facet(&bytes, 0, &manifest, &floors, &stamped).expect("unpack must succeed"); + + // Absence, not a decoded-0.0 entry: no UnpackedField at any + // reserved slot. + let has_slot = + |pair: u8, byte: PairByte| unpacked.iter().any(|f| f.pair == pair && f.byte == byte); + assert!(!has_slot(2, PairByte::Lo), "pair 2 lo must be absent"); + assert!(!has_slot(2, PairByte::Hi), "pair 2 hi must be absent"); + assert!(!has_slot(1, PairByte::Hi), "pair 1 hi must be absent"); + + // The occupied slots ARE present, so the absence above is really a + // filtered set and not an empty-everything bug. + assert!(has_slot(0, PairByte::Lo), "pair 0 lo must be present"); + assert!(has_slot(0, PairByte::Hi), "pair 0 hi must be present"); + assert!(has_slot(1, PairByte::Lo), "pair 1 lo must be present"); + } + + // ── manifest is load-bearing (can-it-fire) ────────────────────────── + + #[test] + fn moving_a_field_to_a_different_slot_changes_the_packed_bytes() { + let manifest_a = fixture_manifest(); + // Same fields, but `fieldA` moves from (pair 0, lo) to the + // previously fully-reserved (pair 2, lo) — a genuinely different + // manifest, not a relabelling. + let manifest_b_text = format!( + "{HEADER}\n\ + 0\t2\tlo\tfieldA\tsurface\tK\tfa\n\ + 0\t0\thi\tfieldB\tsurface\tPa\tfb\n\ + 0\t1\tlo\tfieldC\tsurface\tK\tfa\n" + ); + let manifest_b = + FieldManifest::parse(&manifest_b_text).expect("moved-slot manifest must parse"); + + let floors = fixture_floors(); + let value_of = || fixture_values(12.3, 1_005.7, -42.0); + + let bytes_a = pack_facet(0x1, 0, &manifest_a, &floors, value_of()) + .expect("pack under manifest_a must succeed"); + let bytes_b = pack_facet(0x1, 0, &manifest_b, &floors, value_of()) + .expect("pack under manifest_b must succeed"); + + assert_ne!( + bytes_a, bytes_b, + "moving fieldA to a different slot must change the packed bytes" + ); + // Specifically: fieldA's quantised byte now lives at pair 2 lo + // (manifest_a leaves that slot reserved-zero, manifest_b occupies + // it), and the pair-0-lo slot it vacated is back to reserved-zero + // under manifest_b. + assert_eq!(bytes_a[slot_offset(2, PairByte::Lo)], 0); + assert_eq!(bytes_b[slot_offset(0, PairByte::Lo)], 0); + // Anti-vacuity: the relocated byte is a real, non-zero quantised + // value -- not a coincidental all-zero match between the two facets. + assert_ne!(bytes_a[slot_offset(0, PairByte::Lo)], 0); + assert_eq!( + bytes_b[slot_offset(2, PairByte::Lo)], + bytes_a[slot_offset(0, PairByte::Lo)], + "fieldA's quantised byte value should be identical, just relocated" + ); + } + + // ── stay-silent twin: reordered-but-identical manifest ────────────── + + #[test] + fn reordering_manifest_rows_does_not_change_the_packed_bytes() { + let manifest_forward = fixture_manifest(); + + let reordered_text = format!( + "{HEADER}\n\ + 0\t1\tlo\tfieldC\tsurface\tK\tfa\n\ + 0\t0\thi\tfieldB\tsurface\tPa\tfb\n\ + 0\t0\tlo\tfieldA\tsurface\tK\tfa\n" + ); + let manifest_reordered = + FieldManifest::parse(&reordered_text).expect("reordered manifest must parse"); + + let floors = fixture_floors(); + let value_of = || fixture_values(12.3, 1_005.7, -42.0); + + let bytes_forward = pack_facet(0x1, 0, &manifest_forward, &floors, value_of()) + .expect("pack under forward-order manifest must succeed"); + let bytes_reordered = pack_facet(0x1, 0, &manifest_reordered, &floors, value_of()) + .expect("pack under reordered manifest must succeed"); + + assert_eq!( + bytes_forward, bytes_reordered, + "a byte-different-but-semantically-identical manifest must pack identically" + ); + } + + // ── the pair is two bytes, never combined or swapped ──────────────── + + #[test] + fn lo_and_hi_bytes_of_a_pair_land_at_their_own_index_never_swapped() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + // fieldA (lo, floor "fa" over [-100, 100]) near its floor's low + // bound; fieldB (hi, floor "fb" over [900, 1100]) near its floor's + // high bound -- deliberately far apart so a swap or a u16-combine + // is impossible to miss. + let field_a = -95.0_f64; + let field_b = 1_095.0_f64; + let field_c = 0.0_f64; + + let bytes = pack_facet( + 0x1, + 0, + &manifest, + &floors, + fixture_values(field_a, field_b, field_c), + ) + .expect("pack must succeed"); + + let expected_lo = floors["fa"].quantize(field_a); + let expected_hi = floors["fb"].quantize(field_b); + + assert_eq!( + bytes[slot_offset(0, PairByte::Lo)], + expected_lo, + "the lo byte must hold fieldA's own quantised value" + ); + assert_eq!( + bytes[slot_offset(0, PairByte::Hi)], + expected_hi, + "the hi byte must hold fieldB's own quantised value, not fieldA's" + ); + // Anti-vacuity: the two really are different bytes at different + // indices -- a swap-bug (writing lo into hi's slot or vice versa) + // would make this assertion pass for the wrong reason if the two + // expected values happened to coincide. + assert_ne!( + expected_lo, expected_hi, + "fixture must produce genuinely distinct lo/hi bytes to be a real test" + ); + assert_eq!( + slot_offset(0, PairByte::Lo) + 1, + slot_offset(0, PairByte::Hi) + ); + } + + // ── floor-version mismatch is detected, never silently mis-decoded ── + + #[test] + fn a_stamped_floor_version_mismatch_is_reported_not_silently_mis_decoded() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + let bytes = pack_facet( + 0x1, + 0, + &manifest, + &floors, + fixture_values(12.3, 1_005.7, -42.0), + ) + .expect("pack must succeed"); + + let mut stamped = + stamped_floor_versions(&manifest, 0, &floors).expect("stamping must succeed"); + + // Corrupt only floor "fa"'s stamped version (shared by fieldA and + // fieldC); floor "fb" (fieldB) is untouched. + let real_fa_version = stamped["fa"]; + let bogus_fa_version = real_fa_version.wrapping_add(1); + assert_ne!( + bogus_fa_version, real_fa_version, + "fixture must construct a genuinely different version" + ); + stamped.insert("fa".to_string(), bogus_fa_version); + + let err = unpack_facet(&bytes, 0, &manifest, &floors, &stamped) + .expect_err("a stamped version drift must be reported, not silently decoded"); + + match err { + LaneError::FloorVersionMismatch { + floor_id, + expected, + found, + .. + } => { + assert_eq!(floor_id, "fa"); + assert_eq!(expected, bogus_fa_version); + assert_eq!(found, real_fa_version); + } + other => panic!("expected FloorVersionMismatch, got {other:?}"), + } + + // Positive control: restoring the correct stamped version decodes + // cleanly again, proving the failure above was really about the + // version and not some other defect in the fixture. + stamped.insert("fa".to_string(), real_fa_version); + let unpacked = unpack_facet(&bytes, 0, &manifest, &floors, &stamped) + .expect("unpack with the correct stamped version must succeed"); + assert_eq!(unpacked.len(), 3); + } + + // ── error paths this module owns ───────────────────────────────────── + + #[test] + fn pack_reports_a_missing_value_rather_than_defaulting_to_zero() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + // A value_of closure that always returns None -- every occupied + // slot is "missing". + let err = pack_facet(0x1, 0, &manifest, &floors, |_entry| None) + .expect_err("a missing value must be reported, not silently zeroed"); + + assert!(matches!(err, LaneError::MissingValue { pair: 0, .. })); + } + + #[test] + fn pack_reports_an_unknown_floor_rather_than_skipping_it() { + let manifest = fixture_manifest(); + // Deliberately omit "fa" from the floors map. + let mut floors = fixture_floors(); + floors.remove("fa"); + + let err = pack_facet(0x1, 0, &manifest, &floors, fixture_values(1.0, 2.0, 3.0)) + .expect_err("an unresolvable floor_id must be reported"); + + assert!(matches!( + err, + LaneError::UnknownFloor { floor_id, .. } if floor_id == "fa" + )); + } +} diff --git a/crates/weather-poc/src/lib.rs b/crates/weather-poc/src/lib.rs index 40d5801e..42546f38 100644 --- a/crates/weather-poc/src/lib.rs +++ b/crates/weather-poc/src/lib.rs @@ -38,4 +38,5 @@ pub mod floor; pub mod key; +pub mod lane; pub mod manifest; From d83b4d3e3859fec59d55b62c70b52d2647dbfc3b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 10:02:46 +0000 Subject: [PATCH 2/5] PRE-REGISTER D-WXS-2a: the row-major-vs-Morton bar, written before the run The bar, not the result. Committed first because that is the rule -- and because this deliverable exists to settle a deviation I introduced, which makes writing the bar afterwards exactly the failure mode the rule guards. Split in two, because only one half is runnable today. Half A is pure key space -- no ERA5 data, no classid, no bake -- and is what is pre-registered here. Half B is the zeta stencil under each layout, gated on D-WXS-9 and therefore on the classid mint; it inherits these metrics when it runs. A correction to section 1.3a's own wording lands with it: I called the shipped layout "row-major", which is imprecise. The key orders bytes as [lat_tile, lon_tile, lat_hip, lon_hip], so lexicographic order is tile-row-major and then row-major WITHIN a tile -- a two-level blocked order that already has better locality than a flat row-major over 721x1440. Recorded before measuring, so the comparison runs against what is actually shipped rather than against the looser word I used. Two metrics, both over key-order index: range count to cover a box EXACTLY with no false positives (an over-reading scan that filters afterwards is a weaker thing and does not count), and median neighbour distance over the 4-neighbourhood with longitude wrapping. Three arms: SHIPPED, MORTON (the OGAR-canon nibble-interleave), and CONTROL-BAD (axis bytes reversed, deliberately locality-destroying). The bar has both halves and a kill. Primary: MORTON must beat SHIPPED on BOTH metrics. Control that can lose: CONTROL-BAD must be worse than both -- if a deliberately bad order scores like the good ones, the metric is not measuring locality and no verdict may be read off it. Stay-silent twin: on a TILE-ALIGNED box, SHIPPED and MORTON must give exactly one range each, identical -- that is section 1.2's actual load-bearing claim and it must show NO difference where the plan claims none; a comparison that finds MORTON better everywhere, including here, is measuring something other than what it says. KILL: if MORTON does not win on both metrics, the deviation is harmless for this workload, section 1.3a downgrades from "stated deviation owing a decision" to a recorded note, and D-WXS-2a closes with no code change. A negative result is the cheaper outcome -- it retires an open question instead of opening a migration. Stated now so a green half A is not later misread as a mandate: half A can only show whether a difference exists in key space at all. The stencil is where locality is actually spent. If half A kills, half B is moot; if half A confirms, half B still has to run before any migration. Doc-only. No probe written yet, deliberately. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/STATUS_BOARD.md | 2 +- .claude/plans/weather-soa-bake-v1.md | 63 ++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+), 1 deletion(-) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 26eec430..7fb0270d 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -12,7 +12,7 @@ with a control that can lose and a stay-silent twin. | D-WXS-1 | field manifest v1 — (facet, pair, byte) → (variable, level, unit, floor id), a committed data artifact, ClassView-side | 0 | **SHIPPED 2026-08-13** — `data/field_manifest_v1.tsv` (22 rows = F0 5 pairs + F1/F2 3 pairs each, reserved slots emit NO row) + `manifest.rs`, 13/13; collision guard **disable-verified** (removed → only `colliding_entries_are_rejected` fails, BOTH stay-silent twins stay green). Bar B0's end-to-end half (mutating an entry changes written bytes) DEFERRED — the bake does not exist yet | slot purity §2; bar B0 | | D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | **SHIPPED 2026-08-13** — `era5_variable_census.py` + `.json`; `--selftest` PASS on all 10 constants, orchestrator-rerun independently; guard disable-verified (one constant broken → exit 1, correct message) | ends the chat-only-figure defect for the census | | D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | **SHIPPED 2026-08-13** — `key.rs`, 5/5 green; exhaustive 1,038,240-cell round-trip + collision-free; both bar-B1 halves **disable-verified** by the orchestrator (zeroing the HIP lat byte kills 3 tests incl. collision + ragged; removing the seam split kills the wrap twin while the non-wrap twin stays green) | a 16° box becomes a HEEL-prefix scan; bar B1 | -| D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | Queued | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | +| D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | **Bar PRE-REGISTERED 2026-08-13** (plan §1.3b, committed before the run). Half A (pure key-space: range count + neighbour locality; arms SHIPPED/MORTON/CONTROL-BAD) is runnable NOW. Half B (the ζ stencil) is gated on D-WXS-9 → D-WXS-0 | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | | D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`, 7/7; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted | bar B2 | | D-WXS-3b | **NEW — the L4 lane (pack/unpack ONE 16-byte facet).** The plan gave the lane a worker in §6.2 but **no D-id in §4's ladder** — it jumped D-WXS-3 → D-WXS-4. Added here as the pack/unpack half the bake will call | 1 | **SHIPPED 2026-08-13** — `lane.rs`, 33/33 crate-wide; 4 disables verified by the orchestrator (lo/hi swap → the swap test; hard-coded slot → 3 tests incl. manifest-load-bearing; version guard bypassed → the version test; unmapped slots emitting values → the reserved-slot test). The lane names no ERA5 variable in its own source — the caller's closure owns that | precursor to bar B3; §2.6 slot purity as code | | D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued — **blocked behind D-WXS-0** (must refuse to write without a minted classid) | bar B3; the missing path | diff --git a/.claude/plans/weather-soa-bake-v1.md b/.claude/plans/weather-soa-bake-v1.md index 89b2f24f..6a45d06d 100644 --- a/.claude/plans/weather-soa-bake-v1.md +++ b/.claude/plans/weather-soa-bake-v1.md @@ -198,6 +198,69 @@ test:** `box_ranges` treats `lon_lo == lon_hi` as *wrap the whole circle*, not a *empty box*. Neither reading is forced by the spec. It needs a test pinning the chosen one, or an API that makes the ambiguity unrepresentable. +### §1.3b `D-WXS-2a` — the row-major-vs-Morton bar, PRE-REGISTERED 2026-08-13 (written and committed BEFORE the run) + +§1.3a leaves the layout a *stated deviation, not a ruling*. This section is the +bar that resolves it. **Split in two, because only one half is runnable today:** + +- **Half A — pure key-space. Runnable now**, needs no ERA5 data, no classid, + no bake. This is what is pre-registered here. +- **Half B — the ζ stencil under each layout.** Gated on `D-WXS-9`, itself gated + on the bake and therefore on `D-WXS-0`. Not pre-registered here; it inherits + this section's metrics when it runs. + +**First, a correction to §1.3a's own wording.** §1.3a called the shipped layout +"row-major". That is imprecise. The key orders bytes +`[lat_tile, lon_tile, lat_hip, lon_hip]`, so lexicographic order is +**tile-row-major, then row-major *within* a tile** — a two-level blocked order, +which already has better locality than a flat row-major over 721×1440 would. +Recording this before measuring, so the comparison is against what is actually +shipped rather than against the looser word. + +#### The two metrics (both computed over key-order index, not raw bytes) + +1. **Range count** — the number of maximal contiguous runs in key order needed + to cover **exactly** a given box, with **no false positives** (a scan that + over-reads and filters is a different, weaker thing and does not count). +2. **Neighbour locality** — the median `|key_index(a) − key_index(b)|` over the + 4-neighbourhood (`lat ± 1`, `lon ± 1`, longitude wrapping), across a + deterministic sample of cells. + +#### Arms + +| arm | layout | +|---|---| +| **SHIPPED** | `[lat_tile, lon_tile, lat_hip, lon_hip]` — what `key.rs` emits today | +| **MORTON** | the OGAR-canon reading: the two axis bytes of a tier nibble-interleaved | +| **CONTROL-BAD** | a deliberately locality-destroying order (axis bytes byte-reversed, i.e. `lon_hip` most significant) | + +#### The bar, with both halves and a kill + +- **Primary:** MORTON beats SHIPPED on **both** metrics, over a box set that + includes tile-aligned, non-tile-aligned, seam-crossing and pole-adjacent + boxes. "Beats" is stated before the run as: strictly fewer ranges on the + **median** non-tile-aligned box, **and** strictly smaller median neighbour + distance. +- **Control that can lose:** **CONTROL-BAD must be worse than both** on both + metrics. If a deliberately bad order scores like the good ones, the metric is + not measuring locality and no verdict may be read off it. +- **Stay-silent twin (non-trivial):** on a **tile-aligned** box, SHIPPED and + MORTON must produce **exactly one range each** — identical. This is §1.2's + actual load-bearing claim, and it must show **no difference** where the plan + claims none. A comparison that reports MORTON better *everywhere*, including + here, is measuring something other than what it says. +- **KILL:** if MORTON does **not** win on both metrics, the deviation is + **harmless for this workload**, §1.3a downgrades from "stated deviation owing a + decision" to a recorded note, and `D-WXS-2a` closes without a code change. + A negative result here is a real result and is the cheaper outcome — it retires + an open question rather than opening a migration. + +**Discipline note.** Half A cannot settle the *whole* question, because the +stencil (half B) is where locality is actually spent. Half A can only show +whether a difference exists **in key space at all**. If half A kills, half B is +moot; if half A confirms, half B still has to run before any migration. Stated +now so a green half A is not later read as a mandate. + ### §1.4 classid — a mint decision, NOT taken here `0x0F = Geo` already exists in the OGAR domain table; free domains are `0x03–0x06` From 03a1d0c4b0236df0212aaa5bab9c018e1ee7a54d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 10:16:16 +0000 Subject: [PATCH 3/5] P1 (codex #948): reject non-finite readings before quantizing Codex found a real silent-corruption path in pack_facet. Verified on every link before fixing, and it is worse than the review comment states. The store: probes/weather-p1/README.md section 1 records ARCO-ERA5's fill_value: NaN, that several variables 404 at the arc's OWN fixture timestep, and that in Zarr v2 a missing chunk means all-fill_value -- so a 404 is valid store semantics, not a fetch failure, and "any ingest must treat 404 as data". The overlap: the 404-ing list at that timestep includes mean_sea_level_pressure, 10m_v_component_of_wind, surface_pressure, total_column_water_vapour and total_cloud_cover -- FIVE variables the W1 field set actually packs (F0 pairs 0, 1, 3, 4). Not hypothetical. The mechanism, measured rather than assumed: CalibratedFloor::quantize maps non-finite input to a valid-looking bucket, silently. NaN -> 0, -inf -> 0, +inf -> 255. Rust's float-to-int cast saturates and sends NaN to zero, and f64::clamp PROPAGATES NaN rather than clamping it, so the clamp does not catch it either. Beyond the review comment: the guard must cover ALL non-finite, not only NaN. +/-inf land on the rim buckets, which are legitimate saturation values and therefore exactly as indistinguishable from a real reading. Without the guard an entire missing field would be written as plausible low-bucket measurements and read back through bucket_center as ordinary numbers -- the same failure the reserved-slot rule ("a reserved slot must not read back as a plausible number") exists to prevent, one level deeper and harder to see. Fix: LaneError::NonFiniteValue, and pack_facet refuses before quantize. The guard sits in the lane because that is the boundary where an external value enters the byte register. quantize itself is left alone -- it is a hot primitive, changing its signature ripples, and calibrate already filters is_finite (checked: it is clean, the hole was only in the pack path). The test's third block is what makes it a test rather than a restatement of the guard: it quantises the same non-finite values directly and asserts the decoded result lands INSIDE [lo, hi] -- demonstrating in the suite what the guard prevents. Can-fire covers NaN/+inf/-inf with only ONE field poisoned per run, so a guard that fired on everything would fail the paired stay-silent half, which packs the same slots with finite values. Disable-verified: neutralising the guard fails exactly this test and no other. 34/34, clippy -D warnings clean, fmt clean. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- crates/weather-poc/src/lane.rs | 144 +++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/crates/weather-poc/src/lane.rs b/crates/weather-poc/src/lane.rs index 36bd1fd8..054420ce 100644 --- a/crates/weather-poc/src/lane.rs +++ b/crates/weather-poc/src/lane.rs @@ -182,6 +182,52 @@ pub enum LaneError { /// The level the manifest names for this slot. level_hpa: Option, }, + /// [`pack_facet`]'s `value_of` closure returned a **non-finite** reading + /// (`NaN`, `+inf` or `-inf`) for an occupied slot. + /// + /// # Why this is a hard error and not a silent bucket + /// + /// This is **not** a defensive nicety — it closes a measured + /// silent-corruption path, found by review on PR #948: + /// + /// * ARCO-ERA5 is **sparse by design**. `probes/weather-p1/README.md` §1 + /// records the store's `fill_value: NaN`, that several variables 404 at + /// the arc's own fixture timestep, and that in Zarr v2 *a missing chunk + /// means all-`fill_value`* — so **a 404 is valid store semantics, not a + /// fetch failure**, and an all-`NaN` field is ordinary data an ingest + /// must expect. + /// * The 404-ing list at that timestep includes `mean_sea_level_pressure`, + /// `10m_v_component_of_wind`, `surface_pressure`, + /// `total_column_water_vapour` and `total_cloud_cover` — **five + /// variables the W1 field set actually packs** (F0 pairs 0, 1, 3, 4). + /// * [`CalibratedFloor::quantize`](crate::floor::CalibratedFloor::quantize) + /// maps non-finite input to a **valid-looking bucket, silently** + /// (measured: `NaN` → `0`, `-inf` → `0`, `+inf` → `255`; Rust's + /// float→int cast saturates and sends `NaN` to zero, and `f64::clamp` + /// propagates `NaN` rather than clamping it). + /// + /// Without this guard an entire missing field would be written as + /// plausible low-bucket measurements and read back through + /// [`CalibratedFloor::bucket_center`](crate::floor::CalibratedFloor::bucket_center) + /// as ordinary numbers — the exact failure the reserved-slot rule + /// (*"a reserved slot must not read back as a plausible number"*) exists + /// to prevent, one level deeper and harder to see. + /// + /// The guard covers **all** non-finite values, not only `NaN`: `±inf` + /// land on the rim buckets, which are legitimate saturation values and + /// therefore just as indistinguishable from a real reading. + NonFiniteValue { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The variable the manifest names for this slot. + variable: String, + /// The level the manifest names for this slot. + level_hpa: Option, + }, /// [`unpack_facet`]'s `stamped_versions` map has no entry for a /// `floor_id` an occupied slot needs — the caller's "dataset metadata" /// is incomplete for this facet. @@ -240,6 +286,17 @@ impl std::fmt::Display for LaneError { f, "facet={facet} pair={pair} byte={byte}: no value supplied for {variable:?} at level {level_hpa:?}" ), + LaneError::NonFiniteValue { + facet, + pair, + byte, + variable, + level_hpa, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: non-finite reading for {variable:?} at level {level_hpa:?} \ + (an all-NaN field is valid ARCO-ERA5 store semantics for a missing chunk, never a bucket)" + ), LaneError::MissingStampedVersion { facet, pair, @@ -336,6 +393,19 @@ where level_hpa: entry.level_hpa, })?; + // A non-finite reading must NEVER reach `quantize`. See + // `LaneError::NonFiniteValue` for the measured silent-corruption + // path this guard closes. + if !value.is_finite() { + return Err(LaneError::NonFiniteValue { + facet, + pair, + byte, + variable: entry.variable.clone(), + level_hpa: entry.level_hpa, + }); + } + bytes[slot_offset(pair, byte)] = floor.quantize(value); } @@ -816,6 +886,80 @@ mod tests { assert!(matches!(err, LaneError::MissingValue { pair: 0, .. })); } + /// A non-finite reading is REJECTED, and the paired half proves the same + /// slot packs fine when the reading is finite. + /// + /// This is the regression for the PR #948 review finding. ARCO-ERA5 is + /// sparse by design: `probes/weather-p1/README.md` §1 records + /// `fill_value: NaN` and that a 404 chunk means an all-`NaN` field is + /// **valid store semantics**, at the arc's own fixture timestep, for five + /// variables the W1 field set actually packs. + /// + /// The third assertion block is what makes this a real test rather than a + /// restatement of the guard: it shows what the guard PREVENTS, by + /// quantising the same non-finite values directly and observing that they + /// land on ordinary, plausible buckets. + #[test] + fn pack_rejects_non_finite_readings_which_would_otherwise_become_plausible_buckets() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + // ── can-fire: every flavour of non-finite is refused ── + // Only ONE field is poisoned per run; the others stay finite, so a + // guard that fired on everything would fail the twin below. + for (name, bad) in [ + ("NaN", f64::NAN), + ("+inf", f64::INFINITY), + ("-inf", f64::NEG_INFINITY), + ] { + let err = pack_facet(0x1, 0, &manifest, &floors, move |entry| { + if entry.variable == "fieldA" { + Some(bad) + } else { + Some(1.0) + } + }) + .expect_err("a non-finite reading must be refused"); + assert!( + matches!(err, LaneError::NonFiniteValue { ref variable, .. } if variable == "fieldA"), + "{name} must be reported as NonFiniteValue for fieldA, got {err:?}" + ); + } + + // ── stay-silent twin (non-trivial): the SAME slot, finite, packs ── + let ok = pack_facet( + 0x1, + 0, + &manifest, + &floors, + fixture_values(12.3, 1_005.7, -42.0), + ) + .expect("finite readings at the same slots must pack cleanly"); + assert_ne!( + ok, [0u8; FACET_LEN], + "the finite pack must actually write something" + ); + + // ── what the guard PREVENTS (the anti-vacuity half) ── + // Without the guard these values reach `quantize`, which maps them to + // ordinary buckets with no signal that anything was wrong. + let floor = floors.get("fa").expect("fixture floor fa"); + let (lo, hi) = floor.bounds(); + for (name, bad) in [ + ("NaN", f64::NAN), + ("+inf", f64::INFINITY), + ("-inf", f64::NEG_INFINITY), + ] { + let bucket = floor.quantize(bad); + let decoded = floor.bucket_center(bucket); + assert!( + decoded >= lo && decoded <= hi, + "{name} quantised to bucket {bucket}, decoding to {decoded} — inside [{lo}, {hi}], \ + i.e. indistinguishable from a real reading. This is why the guard exists." + ); + } + } + #[test] fn pack_reports_an_unknown_floor_rather_than_skipping_it() { let manifest = fixture_manifest(); From 53e09143b9e43a1b06bd72adf0a553144a2f7da5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 10:19:42 +0000 Subject: [PATCH 4/5] The same P1, one level worse: saturation_of folded non-finite into the metric Following codex's lane.rs finding to its class rather than stopping at the instance. The question after a review catch is never "is it fixed" but "was that the only instance or the first one found". It was the first. CalibratedFloor::saturation_of takes -- by its own doc -- "an ARBITRARY external population" and quantised every value directly. quantize maps non-finite to a RIM bucket silently (NaN -> 0, -inf -> 0, +inf -> 255), and rim is exactly what saturation_of counts. So a NaN read as saturation. This is worse than the lane hole, not equal to it. saturation_of is bar B2's INSTRUMENT. A corrupted stored value is bad; a corrupted measurement is worse, because every conclusion downstream inherits it and nothing looks wrong. An all-NaN population -- valid ARCO-ERA5 store semantics for a missing chunk -- would have scored 1.0: "completely saturated" when the truth is "no data at all". Those are opposite findings and the bare fraction could not tell them apart. Silently DROPPING the non-finite values would have been the other half of the same mistake: the caller would never learn the population was partly or wholly absent. So the fix reports rather than folds or drops, matching this crate's standing shape (calibrate and decode both return None on a degenerate case rather than inventing a number). saturation_of now returns SaturationScore { fraction, finite, non_finite }. The fraction is over the finite subset; non_finite is a fact about the DATA, not about the floor, and the caller sees it. Both existing call sites updated to assert non_finite == 0 on their all-finite fixtures, which also makes them say out loud that their populations are clean. The test's sharp block is the all-NaN case that would have scored 1.0. The mixed block proves absent values do not move a genuinely low fraction AT ALL (asserted equal, not merely close). +/-inf are covered separately because they land on the rim buckets rather than bucket 0, so a NaN-only guard would have missed them. The stay-silent twin holds clean data unaffected. Disable-verified: folding non-finite back in -- the pre-fix behaviour -- fails exactly this test and no other. Also checked and CLEAN: calibrate already filters is_finite, so the hole never reached the calibration path. Every quantize call site in the crate is now either guarded or provably finite. 35/35, clippy -D warnings clean (two repeat().take() nits fixed), fmt clean. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- crates/weather-poc/src/floor.rs | 182 +++++++++++++++++++++++++++++--- 1 file changed, 166 insertions(+), 16 deletions(-) diff --git a/crates/weather-poc/src/floor.rs b/crates/weather-poc/src/floor.rs index 03347f72..cd773e78 100644 --- a/crates/weather-poc/src/floor.rs +++ b/crates/weather-poc/src/floor.rs @@ -46,6 +46,26 @@ pub const LO_PERCENTILE: f64 = 0.4; /// See [`LO_PERCENTILE`]. pub const HI_PERCENTILE: f64 = 99.6; +/// The result of scoring an external population against a floor +/// ([`CalibratedFloor::saturation_of`]). +/// +/// Carries the non-finite count alongside the fraction **by design**: see +/// [`CalibratedFloor::saturation_of`] for why folding non-finite values into +/// the fraction turns "no data at all" into "completely saturated", and why +/// silently dropping them is the other half of the same mistake. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct SaturationScore { + /// Fraction of the **finite** values landing in the two rim buckets. + /// `0.0` when `finite == 0` — check `finite` before reading this. + pub fraction: f64, + /// How many values were finite, i.e. the denominator of `fraction`. + pub finite: usize, + /// How many values were non-finite (`NaN`, `±inf`) and therefore + /// excluded. Non-zero means the population was partly or wholly absent + /// — a fact about the data, not about the floor. + pub non_finite: usize, +} + /// A calibrated, versioned linear 256-bucket floor over `[lo, hi]`. /// /// Produced only by [`calibrate`] — there is no public constructor that @@ -179,19 +199,63 @@ impl CalibratedFloor { /// anything. Both directions bar B2 needs are "this floor scored /// against a population it was not calibrated on": a narrow floor /// scored globally, and a global floor scored on one narrow box. - /// Returns `0.0` on an empty `values` slice. - pub fn saturation_of(&self, values: &[f64]) -> f64 { - if values.is_empty() { - return 0.0; + /// + /// # Non-finite values are EXCLUDED and COUNTED, never folded in + /// + /// The `fraction` is computed over the **finite** subset only, and + /// [`SaturationScore::non_finite`] reports how many were set aside. This + /// is not fussiness — it closes the sibling of the PR #948 `lane.rs` + /// finding, in the more dangerous position: + /// + /// * [`Self::quantize`] maps non-finite input to a rim bucket silently + /// (`NaN` → `0`, `-inf` → `0`, `+inf` → `255`; see its own docs). Rim + /// is exactly what this function counts. + /// * So a folded-in `NaN` reads as **saturation**, and an all-`NaN` + /// population — *valid ARCO-ERA5 store semantics for a missing chunk*, + /// per `probes/weather-p1/README.md` §1 — would score `1.0`: + /// "completely saturated" when the truth is "no data at all". Those + /// are opposite findings and the bare fraction cannot tell them apart. + /// * This function is bar B2's **instrument**. A corrupted stored value + /// is bad; a corrupted measurement is worse, because every conclusion + /// downstream inherits it silently. + /// + /// Silently *dropping* the non-finite values would be the other half of + /// the same mistake — the caller would not learn that a population was + /// partly or wholly absent. Hence a reported count, matching this + /// crate's standing rule that a degenerate case is **reported**, never + /// folded (the same shape as [`calibrate`] and [`Self::decode`] + /// returning `None`). + /// + /// On an empty slice, or one with no finite values at all, `fraction` is + /// `0.0` and `finite` is `0` — check `finite` before reading `fraction`. + pub fn saturation_of(&self, values: &[f64]) -> SaturationScore { + let mut finite = 0usize; + let mut non_finite = 0usize; + let mut rim = 0usize; + + for &v in values { + if !v.is_finite() { + non_finite += 1; + continue; + } + finite += 1; + let b = self.quantize(v); + if b == 0 || b == (BUCKETS - 1) as u8 { + rim += 1; + } + } + + let fraction = if finite == 0 { + 0.0 + } else { + rim as f64 / finite as f64 + }; + + SaturationScore { + fraction, + finite, + non_finite, } - let rim = values - .iter() - .filter(|&&v| { - let b = self.quantize(v); - b == 0 || b == (BUCKETS - 1) as u8 - }) - .count(); - rim as f64 / values.len() as f64 } } @@ -353,9 +417,12 @@ mod tests { let wide_population = linspace(-1000.0, 1000.0, 102_400); let sat = narrow_floor.saturation_of(&wide_population); + assert_eq!(sat.non_finite, 0, "fixture population is all finite"); + assert_eq!(sat.finite, wide_population.len()); assert!( - sat > 0.9, - "expected high saturation applying a narrow floor globally, got {sat}" + sat.fraction > 0.9, + "expected high saturation applying a narrow floor globally, got {}", + sat.fraction ); } @@ -382,10 +449,93 @@ mod tests { ); let sat = global_floor.saturation_of(sub_window); + assert_eq!(sat.non_finite, 0, "fixture sub-window is all finite"); + assert_eq!(sat.finite, sub_window.len()); + assert!( + sat.fraction < 0.1, + "global floor should not saturate heavily on an interior-adjacent sub-window, got {}", + sat.fraction + ); + } + + // ── non-finite values are excluded and counted, never folded ───────── + + /// A non-finite value must NOT be counted as saturation. + /// + /// This is the sibling of the PR #948 `lane.rs` finding, in the more + /// dangerous position: `saturation_of` is bar B2's INSTRUMENT, so folding + /// `NaN` into the fraction corrupts a measurement rather than a stored + /// value, and every conclusion downstream inherits it silently. + /// + /// The first block is the sharp case: an ALL-`NaN` population is valid + /// ARCO-ERA5 store semantics for a missing chunk + /// (`probes/weather-p1/README.md` §1), and the pre-fix code would have + /// scored it `1.0` — "completely saturated" — when the truth is "no data + /// at all". + #[test] + fn non_finite_values_are_excluded_and_counted_never_scored_as_saturation() { + let population = linspace(-1000.0, 1000.0, 10_240); + let floor = calibrate(&population).expect("non-empty sample calibrates"); + + // ── the sharp case: an entirely absent field ── + let all_nan = vec![f64::NAN; 4_096]; + let sat = floor.saturation_of(&all_nan); + assert_eq!( + sat.non_finite, 4_096, + "every value must be counted as absent" + ); + assert_eq!(sat.finite, 0, "none of them is a measurement"); + assert_eq!( + sat.fraction, 0.0, + "an absent field must not read as saturation — pre-fix this was 1.0" + ); + + // ── mixed: NaN must not inflate a genuinely low fraction ── + // Interior values (the population's middle third) do not saturate. + let n = population.len(); + let interior = &population[(n / 3)..(2 * n / 3)]; + let clean = floor.saturation_of(interior); + assert_eq!(clean.non_finite, 0); assert!( - sat < 0.1, - "global floor should not saturate heavily on an interior-adjacent sub-window, got {sat}" + clean.fraction < 0.1, + "interior values must not saturate, got {}", + clean.fraction + ); + + let mut poisoned: Vec = interior.to_vec(); + poisoned.extend(std::iter::repeat_n(f64::NAN, interior.len())); + let mixed = floor.saturation_of(&poisoned); + assert_eq!(mixed.non_finite, interior.len(), "the NaN half is counted"); + assert_eq!( + mixed.finite, + interior.len(), + "the real half is the denominator" ); + assert_eq!( + mixed.fraction, clean.fraction, + "adding absent values must not move the fraction at all" + ); + + // ── +/-inf too, not just NaN: they land on the RIM buckets ── + for (name, bad) in [("+inf", f64::INFINITY), ("-inf", f64::NEG_INFINITY)] { + let mut with_inf: Vec = interior.to_vec(); + with_inf.extend(std::iter::repeat_n(bad, interior.len())); + let scored = floor.saturation_of(&with_inf); + assert_eq!( + scored.non_finite, + interior.len(), + "{name} must be counted as absent" + ); + assert_eq!( + scored.fraction, clean.fraction, + "{name} must not inflate saturation (it quantises to a rim bucket)" + ); + } + + // ── stay-silent twin: an all-finite population is unaffected ── + let sat_finite = floor.saturation_of(&population); + assert_eq!(sat_finite.non_finite, 0, "no false positives on clean data"); + assert_eq!(sat_finite.finite, population.len()); } // ── round-trip bound ───────────────────────────────────────────────── From 5169ff86901b41db98c956324e8f372e6964ba01 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 10:21:43 +0000 Subject: [PATCH 5/5] board: E-A-TOTAL-FUNCTION-THAT-CANNOT-REFUSE-IS-A-CORRUPTION-PATH-1 Records the class behind the two NaN fixes, because the transferable part is not the fix -- it is the shape. CalibratedFloor::quantize is TOTAL: every input returns a valid-looking bucket. It has no way to say "that was not a measurement". Measured: NaN -> 0 (f64::clamp PROPAGATES NaN rather than clamping it, then the float-to-int cast saturates to zero), -inf -> 0, +inf -> 255. All three are legitimate buckets; 0 and 255 are the ordinary saturation values and bucket_center(0) is a real number near lo. Nothing downstream can tell the result from a genuine reading. Live, not theoretical: ARCO-ERA5 is sparse by design, a 404 chunk means all-NaN and is valid store semantics, and five of the variables that 404 at the arc's own fixture timestep are in the W1 field set. Instance 1 (codex P1) was the STORE path. Instance 2, found by following the first to its class, was the INSTRUMENT path -- and the sharpening is the entry's point: a corrupted stored value is one wrong row; a corrupted instrument is every conclusion drawn with it, each looking sound and carrying no trace of the defect. When a finding lands on a total function, check its measurement call sites before its storage call sites. Also recorded: silently DROPPING invalid input is the other half of the same mistake, so the fix reports (SaturationScore carries finite/non_finite), matching calibrate and decode returning None rather than inventing a number. The generalizable check the entry leaves behind: for every total function mapping a wider domain onto a narrower one -- quantisers, clamps, `as` casts, unwrap_or, saturating arithmetic -- ask what an invalid input returns and whether that is distinguishable from a valid one. If it is a valid-looking value, the function cannot refuse, and every call site is a corruption path until one of them does. STATUS_BOARD D-WXS-3 and D-WXS-3b annotated with their amendments; both are Status-field rewrites of the same rows, no deletions (audited: 4 changed lines, 2 removals, each the row it replaced). Doc-only. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CcpLeEC3XK8Eye53GKBVvi --- .claude/board/EPIPHANIES.md | 78 +++++++++++++++++++++++++++++++++++ .claude/board/STATUS_BOARD.md | 4 +- 2 files changed, 80 insertions(+), 2 deletions(-) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 63cbf836..4ea125c1 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,81 @@ +## 2026-08-13 — E-A-TOTAL-FUNCTION-THAT-CANNOT-REFUSE-IS-A-CORRUPTION-PATH-1 + +**Status:** FINDING `[G]` — two measured instances, one crate, one hour. +Found by codex review on PR #948; the second by following the first to its +class. **Confidence:** High. Mechanism measured, not inferred. + +**The shape.** `CalibratedFloor::quantize(f64) -> u8` is **total**: every input +returns a valid-looking bucket. It has no way to say *"that was not a +measurement."* Measured behaviour on non-finite input: + +| input | bucket | why | +|---|---|---| +| `NaN` | **0** | `f64::clamp` **propagates** `NaN` rather than clamping it, then Rust's float→int cast saturates and sends `NaN` to zero | +| `-inf` | **0** | clamps to the low rim | +| `+inf` | **255** | clamps to the high rim | + +Every one of those is a **legitimate** bucket. `0` and `255` are the ordinary +saturation values; `bucket_center(0)` is a real number near `lo`. Nothing +downstream can distinguish the result from a genuine reading. + +**Why this was live and not theoretical.** ARCO-ERA5 is sparse *by design*: +`probes/weather-p1/README.md` §1 records `fill_value: NaN`, that several +variables 404 at the arc's **own fixture timestep**, and that in Zarr v2 a +missing chunk means all-`fill_value` — so **a 404 is valid store semantics, not +a fetch failure**, and *"any ingest must treat 404 as data."* The 404-ing list +at that timestep includes five variables the W1 field set actually packs. + +**Instance 1 — the store path** (`lane.rs::pack_facet`, codex P1). An absent +field would have been written as plausible low-bucket measurements and read +back through `bucket_center` as ordinary numbers. This is the reserved-slot +rule — *"a reserved slot must not read back as a plausible number"* — defeated +one level deeper, where the existing guard could not see it. + +**Instance 2 — the INSTRUMENT path, and it is worse** +(`floor.rs::saturation_of`). That function scores *"an ARBITRARY external +population"* by counting rim buckets — and non-finite input lands **on the +rim**. An all-`NaN` population would have scored `1.0`: **"completely +saturated" when the truth is "no data at all."** Those are opposite findings +and the bare fraction could not tell them apart. It is bar B2's instrument, so +the corruption would have propagated into a *measurement* rather than a stored +value. + +> **The sharpening worth keeping: a corrupted stored value is bad; a corrupted +> INSTRUMENT is worse.** A bad value is one wrong row. A bad instrument is +> every conclusion drawn with it, each of which looks sound and carries no +> trace of the defect. When a finding lands on a total function, check its +> *measurement* call sites before its storage call sites. + +**The other half of the same mistake — do not silently drop.** Skipping +non-finite values without reporting them is equally wrong: the caller never +learns the population was partly or wholly absent. The fix therefore +**reports**: `saturation_of` returns `SaturationScore { fraction, finite, +non_finite }`, matching this crate's standing shape (`calibrate` and `decode` +return `None` on a degenerate case rather than inventing a number) and the +`D-WXS-12` rule that *the degenerate case must be reported, never folded as +`0.0`*. + +**Where the guard belongs.** At the boundary where an external value enters +the register — not inside the hot primitive. `quantize` keeps its signature +(changing it ripples through every call site); `pack_facet` refuses, and +`saturation_of` excludes-and-counts. `calibrate` was checked and is **clean** +— it already filters `is_finite`, so the hole never reached calibration. Every +`quantize` call site in the crate is now either guarded or provably finite. + +**Generalizable check, cheap to run:** for every total function that maps a +wider domain onto a narrower one — quantisers, clamps, `as` casts, +`unwrap_or`, saturating arithmetic — ask *what does an invalid input return, +and is that return distinguishable from a valid one?* If the answer is "a +valid-looking value", the function cannot refuse, and every call site is a +corruption path until one of them does. + +**Cross-ref:** `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1`; +`E-A-DISABLE-PROBE-CAN-ITSELF-BE-VACUOUS-1` (same session, the verification +layer); `.claude/plans/weather-soa-bake-v1.md` §4 bar B2 (the instrument); +`probes/weather-p1/README.md` §1 (the store semantics); PR #948. + +--- + ## 2026-08-13 — E-A-DISABLE-PROBE-CAN-ITSELF-BE-VACUOUS-1 **Status:** FINDING `[G]` — three measured instances in one session, all mine. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 7fb0270d..8a90446e 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -13,8 +13,8 @@ with a control that can lose and a stay-silent twin. | D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | **SHIPPED 2026-08-13** — `era5_variable_census.py` + `.json`; `--selftest` PASS on all 10 constants, orchestrator-rerun independently; guard disable-verified (one constant broken → exit 1, correct message) | ends the chat-only-figure defect for the census | | D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | **SHIPPED 2026-08-13** — `key.rs`, 5/5 green; exhaustive 1,038,240-cell round-trip + collision-free; both bar-B1 halves **disable-verified** by the orchestrator (zeroing the HIP lat byte kills 3 tests incl. collision + ragged; removing the seam split kills the wrap twin while the non-wrap twin stays green) | a 16° box becomes a HEEL-prefix scan; bar B1 | | D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | **Bar PRE-REGISTERED 2026-08-13** (plan §1.3b, committed before the run). Half A (pure key-space: range count + neighbour locality; arms SHIPPED/MORTON/CONTROL-BAD) is runnable NOW. Half B (the ζ stencil) is gated on D-WXS-9 → D-WXS-0 | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | -| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`, 7/7; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted | bar B2 | -| D-WXS-3b | **NEW — the L4 lane (pack/unpack ONE 16-byte facet).** The plan gave the lane a worker in §6.2 but **no D-id in §4's ladder** — it jumped D-WXS-3 → D-WXS-4. Added here as the pack/unpack half the bake will call | 1 | **SHIPPED 2026-08-13** — `lane.rs`, 33/33 crate-wide; 4 disables verified by the orchestrator (lo/hi swap → the swap test; hard-coded slot → 3 tests incl. manifest-load-bearing; version guard bypassed → the version test; unmapped slots emitting values → the reserved-slot test). The lane names no ERA5 variable in its own source — the caller's closure owns that | precursor to bar B3; §2.6 slot purity as code | +| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted. **AMENDED same day** (`E-A-TOTAL-FUNCTION-THAT-CANNOT-REFUSE-IS-A-CORRUPTION-PATH-1`): `saturation_of` folded non-finite input into the metric — `quantize` sends `NaN`/`-inf`→0 and `+inf`→255, all **rim** buckets, so an all-`NaN` population scored **1.0** ("fully saturated") where the truth is "no data at all". Now returns `SaturationScore {fraction, finite, non_finite}` — reported, never folded and never silently dropped. `calibrate` checked **CLEAN** (already filters `is_finite`) | bar B2 — this is the bar's own INSTRUMENT, so the defect would have corrupted a measurement, not a value | +| D-WXS-3b | **NEW — the L4 lane (pack/unpack ONE 16-byte facet).** The plan gave the lane a worker in §6.2 but **no D-id in §4's ladder** — it jumped D-WXS-3 → D-WXS-4. Added here as the pack/unpack half the bake will call | 1 | **SHIPPED 2026-08-13** — `lane.rs`, 33/33 crate-wide; 4 disables verified by the orchestrator (lo/hi swap → the swap test; hard-coded slot → 3 tests incl. manifest-load-bearing; version guard bypassed → the version test; unmapped slots emitting values → the reserved-slot test). The lane names no ERA5 variable in its own source — the caller's closure owns that **AMENDED same day (codex P1, PR #948):** `pack_facet` accepted non-finite readings; `quantize` maps them to valid-looking buckets, so a missing ARCO-ERA5 chunk (all-`NaN` — **valid store semantics**, and five W1 variables 404 at the arc's own fixture timestep) would have been stored as plausible low-bucket measurements. Now `LaneError::NonFiniteValue`, covering `±inf` too since they land on the rim. Disable-verified | precursor to bar B3; §2.6 slot purity as code | | D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued — **blocked behind D-WXS-0** (must refuse to write without a minted classid) | bar B3; the missing path | | D-WXS-5 | statics bake — separate classid, separate dataset, exactly ONE version | 1 | Queued | bar B4; avoids ~1.3 PB of rewritten constants | | D-WXS-6 | version-range read (`QueryReference::at(v,rung)` + `deinterlace`) + version-count scaling measurement | 2 | Queued | bar B5; KILL if growth is superlinear at 92,044 versions |