diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 64213ed3..4ea125c1 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,138 @@ +## 2026-08-13 — E-A-TOTAL-FUNCTION-THAT-CANNOT-REFUSE-IS-A-CORRUPTION-PATH-1 + +**Status:** FINDING `[G]` — two measured instances, one crate, one hour. +Found by codex review on PR #948; the second by following the first to its +class. **Confidence:** High. Mechanism measured, not inferred. + +**The shape.** `CalibratedFloor::quantize(f64) -> u8` is **total**: every input +returns a valid-looking bucket. It has no way to say *"that was not a +measurement."* Measured behaviour on non-finite input: + +| input | bucket | why | +|---|---|---| +| `NaN` | **0** | `f64::clamp` **propagates** `NaN` rather than clamping it, then Rust's float→int cast saturates and sends `NaN` to zero | +| `-inf` | **0** | clamps to the low rim | +| `+inf` | **255** | clamps to the high rim | + +Every one of those is a **legitimate** bucket. `0` and `255` are the ordinary +saturation values; `bucket_center(0)` is a real number near `lo`. Nothing +downstream can distinguish the result from a genuine reading. + +**Why this was live and not theoretical.** ARCO-ERA5 is sparse *by design*: +`probes/weather-p1/README.md` §1 records `fill_value: NaN`, that several +variables 404 at the arc's **own fixture timestep**, and that in Zarr v2 a +missing chunk means all-`fill_value` — so **a 404 is valid store semantics, not +a fetch failure**, and *"any ingest must treat 404 as data."* The 404-ing list +at that timestep includes five variables the W1 field set actually packs. + +**Instance 1 — the store path** (`lane.rs::pack_facet`, codex P1). An absent +field would have been written as plausible low-bucket measurements and read +back through `bucket_center` as ordinary numbers. This is the reserved-slot +rule — *"a reserved slot must not read back as a plausible number"* — defeated +one level deeper, where the existing guard could not see it. + +**Instance 2 — the INSTRUMENT path, and it is worse** +(`floor.rs::saturation_of`). That function scores *"an ARBITRARY external +population"* by counting rim buckets — and non-finite input lands **on the +rim**. An all-`NaN` population would have scored `1.0`: **"completely +saturated" when the truth is "no data at all."** Those are opposite findings +and the bare fraction could not tell them apart. It is bar B2's instrument, so +the corruption would have propagated into a *measurement* rather than a stored +value. + +> **The sharpening worth keeping: a corrupted stored value is bad; a corrupted +> INSTRUMENT is worse.** A bad value is one wrong row. A bad instrument is +> every conclusion drawn with it, each of which looks sound and carries no +> trace of the defect. When a finding lands on a total function, check its +> *measurement* call sites before its storage call sites. + +**The other half of the same mistake — do not silently drop.** Skipping +non-finite values without reporting them is equally wrong: the caller never +learns the population was partly or wholly absent. The fix therefore +**reports**: `saturation_of` returns `SaturationScore { fraction, finite, +non_finite }`, matching this crate's standing shape (`calibrate` and `decode` +return `None` on a degenerate case rather than inventing a number) and the +`D-WXS-12` rule that *the degenerate case must be reported, never folded as +`0.0`*. + +**Where the guard belongs.** At the boundary where an external value enters +the register — not inside the hot primitive. `quantize` keeps its signature +(changing it ripples through every call site); `pack_facet` refuses, and +`saturation_of` excludes-and-counts. `calibrate` was checked and is **clean** +— it already filters `is_finite`, so the hole never reached calibration. Every +`quantize` call site in the crate is now either guarded or provably finite. + +**Generalizable check, cheap to run:** for every total function that maps a +wider domain onto a narrower one — quantisers, clamps, `as` casts, +`unwrap_or`, saturating arithmetic — ask *what does an invalid input return, +and is that return distinguishable from a valid one?* If the answer is "a +valid-looking value", the function cannot refuse, and every call site is a +corruption path until one of them does. + +**Cross-ref:** `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1`; +`E-A-DISABLE-PROBE-CAN-ITSELF-BE-VACUOUS-1` (same session, the verification +layer); `.claude/plans/weather-soa-bake-v1.md` §4 bar B2 (the instrument); +`probes/weather-p1/README.md` §1 (the store semantics); PR #948. + +--- + +## 2026-08-13 — E-A-DISABLE-PROBE-CAN-ITSELF-BE-VACUOUS-1 + +**Status:** FINDING `[G]` — three measured instances in one session, all mine. +**Confidence:** High. Method-level; no code claim. + +**The known rule it extends.** This workspace already holds *"an assertion +implied by the code it tests is not a test"* (`CLAUDE.md` § falsifiability rule) +and, in the sibling repo's words, *"turning a knob that does not bind is not a +disable."* Both are stated about **tests**. This entry records that the same +failure applies one level up — to the **verification probe** that is supposed to +prove a test can fail — and that it is harder to spot there, because a broken +probe and a passing suite look identical. + +**The three instances, same session, gating `crates/weather-poc`.** + +1. **Wrong symbol name.** A probe searched for `ManifestError::DuplicateSlot`; + the real variant is `SlotCollision`. The substitution script aborted, the + test run afterwards executed **unmodified code**, and reported `25 passed`. + Read casually, that is a passing disable-verification of a guard that was + never touched. +2. **Dead code.** A probe inserted an `if` block computing `lo`/`hi` and + discarding both (`let _ = (lo, hi);`). It applied cleanly and changed + nothing. `25 passed` again. +3. **Wrong target.** A probe changed `raw` to `raw.max(1)` intending to make + reserved slots decode — but the unpack loop only visits **manifest-resolved** + slots, so the edit could never reach a reserved one. `33 passed`. + +Instance 1 is the dangerous one: 2 and 3 at least ran, while 1 silently did not. + +**The signature that separates the two causes.** A disable run that stays green +has two possible explanations — *the guard is absent* or *the probe never +touched it* — and greenness alone does not distinguish them. What does: a +correct disable kills **at least one** test, and usually a small, nameable set. + +> **A disable that kills ZERO tests is more likely a broken probe than a missing +> guard.** Treat zero as "re-check the probe", never as "verified". + +Corollary, the mechanical fix now in use: **the probe must assert that it +applied.** Every substitution asserts its pattern was found and the file +actually changed, and fails loudly otherwise — so instance 1 becomes an error +instead of a green run. + +**Why this is worth a board entry rather than a shrug.** The whole +disable-the-fix discipline exists because a passing test proves nothing about +whether it *could* fail. If the probe that establishes that is itself unchecked, +the discipline has an unverified root and inherits exactly the confidence it was +built to withdraw. Three instances in one session, by an operator applying the +rule deliberately, is the measured argument that the root needs checking too. + +**Cross-ref:** `E-VACUOUS-ASSERTION-IS-THE-HOUSE-STYLE-1`, +`E-A-CONTROL-THAT-CANNOT-LOSE-IS-NO-CONTROL-1`, +`E-ANTI-EIGENVALUE-MACHINERY-CAN-ITSELF-BECOME-THE-EIGENVALUE-1` (the same +one-level-up move, applied there to guards rather than probes); +`CLAUDE.md` § The falsifiability rule. + +--- + ## 2026-08-12 — E-THE-REGIME-LADDER-MEASURED-RANGE-NOT-TURBULENCE-1 **Status:** FINDING `[G]` — measured, same run, found by an operator diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index c302ba1b..8a90446e 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -12,9 +12,10 @@ with a control that can lose and a stay-silent twin. | D-WXS-1 | field manifest v1 — (facet, pair, byte) → (variable, level, unit, floor id), a committed data artifact, ClassView-side | 0 | **SHIPPED 2026-08-13** — `data/field_manifest_v1.tsv` (22 rows = F0 5 pairs + F1/F2 3 pairs each, reserved slots emit NO row) + `manifest.rs`, 13/13; collision guard **disable-verified** (removed → only `colliding_entries_are_rejected` fails, BOTH stay-silent twins stay green). Bar B0's end-to-end half (mutating an entry changes written bytes) DEFERRED — the bake does not exist yet | slot purity §2; bar B0 | | D-WXS-1a | variable census as a committed re-runnable probe (17 surface + 91 upper-air + 14 static = 122 fields; 92,044 six-hourly steps) | 0 | **SHIPPED 2026-08-13** — `era5_variable_census.py` + `.json`; `--selftest` PASS on all 10 constants, orchestrator-rerun independently; guard disable-verified (one constant broken → exit 1, correct message) | ends the chat-only-figure defect for the census | | D-WXS-2 | key codec `(lat,lon) ↔ NodeGuid` — HEEL 16° tile / HIP within-tile / TWIG dormant; ragged tiles; lon-wrap range-SET | 1 | **SHIPPED 2026-08-13** — `key.rs`, 5/5 green; exhaustive 1,038,240-cell round-trip + collision-free; both bar-B1 halves **disable-verified** by the orchestrator (zeroing the HIP lat byte kills 3 tests incl. collision + ragged; removing the seam split kills the wrap twin while the non-wrap twin stays green) | a 16° box becomes a HEEL-prefix scan; bar B1 | -| D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | Queued | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | -| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`, 7/7; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted | bar B2 | -| D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued | bar B3; the missing path | +| D-WXS-2a | **NEW — row-major vs Morton, pre-registered comparison.** The shipped key assigns one WHOLE byte per axis; OGAR's cascade doctrine specifies the axis bytes **nibble-interleaved** (Morton). §1.2 deviated from the canon it cites and did not say so — now recorded as plan §1.3a. The prefix-scan claim holds under both; what differs is neighbour locality (`lat ± 1` is 1440 cells away under row-major) and how many ranges a non-tile-aligned box needs | 1 | **Bar PRE-REGISTERED 2026-08-13** (plan §1.3b, committed before the run). Half A (pure key-space: range count + neighbour locality; arms SHIPPED/MORTON/CONTROL-BAD) is runnable NOW. Half B (the ζ stencil) is gated on D-WXS-9 → D-WXS-0 | gates any downstream assumption of Morton locality — measured against the ζ stencil (D-WXS-9), metric stated before the run | +| D-WXS-3 | shared canonical floor calibration (global 0.4–99.6 pct, frozen per epoch, stamped in dataset metadata) | 1 | **SHIPPED 2026-08-13** — `floor.rs`; bar B2 **disable-verified** (widening the "narrow" control floor kills only the control, twin stays green); version-stamp mismatch detected, ±½-bucket round-trip asserted. **AMENDED same day** (`E-A-TOTAL-FUNCTION-THAT-CANNOT-REFUSE-IS-A-CORRUPTION-PATH-1`): `saturation_of` folded non-finite input into the metric — `quantize` sends `NaN`/`-inf`→0 and `+inf`→255, all **rim** buckets, so an all-`NaN` population scored **1.0** ("fully saturated") where the truth is "no data at all". Now returns `SaturationScore {fraction, finite, non_finite}` — reported, never folded and never silently dropped. `calibrate` checked **CLEAN** (already filters `is_finite`) | bar B2 — this is the bar's own INSTRUMENT, so the defect would have corrupted a measurement, not a value | +| D-WXS-3b | **NEW — the L4 lane (pack/unpack ONE 16-byte facet).** The plan gave the lane a worker in §6.2 but **no D-id in §4's ladder** — it jumped D-WXS-3 → D-WXS-4. Added here as the pack/unpack half the bake will call | 1 | **SHIPPED 2026-08-13** — `lane.rs`, 33/33 crate-wide; 4 disables verified by the orchestrator (lo/hi swap → the swap test; hard-coded slot → 3 tests incl. manifest-load-bearing; version guard bypassed → the version test; unmapped slots emitting values → the reserved-slot test). The lane names no ERA5 variable in its own source — the caller's closure owns that **AMENDED same day (codex P1, PR #948):** `pack_facet` accepted non-finite readings; `quantize` maps them to valid-looking buckets, so a missing ARCO-ERA5 chunk (all-`NaN` — **valid store semantics**, and five W1 variables 404 at the arc's own fixture timestep) would have been stored as plausible low-bucket measurements. Now `LaneError::NonFiniteValue`, covering `±inf` too since they land on the rim. Disable-verified | precursor to bar B3; §2.6 slot purity as code | +| D-WXS-4 | the bake: one timestep → 1,038,240 NodeRows → ONE Lance version | 1 | Queued — **blocked behind D-WXS-0** (must refuse to write without a minted classid) | bar B3; the missing path | | D-WXS-5 | statics bake — separate classid, separate dataset, exactly ONE version | 1 | Queued | bar B4; avoids ~1.3 PB of rewritten constants | | D-WXS-6 | version-range read (`QueryReference::at(v,rung)` + `deinterlace`) + version-count scaling measurement | 2 | Queued | bar B5; KILL if growth is superlinear at 92,044 versions | | D-WXS-7 | **D-WXA-5 re-homed and RE-SPECIFIED** — ρ(code_dist, field_dist) via `jc::reliability::spearman` over whole-grid pairs. (a) ρ ≥ 0.9996 (the bar a real pair FAILED at 0.999556); (b) shuffled-codebook control < 0.98 (measured losable at 0.003–0.159); (c) 16/64/256-level ladder must be MONOTONE before any verdict | 3 | Queued | ⚠ poc-v2's ρ ≥ 0.98 is at risk of being vacuous — D-CZ-1 §6.4 measured real-arm ρ spread 3e-6…4.7e-5 | diff --git a/.claude/plans/weather-soa-bake-v1.md b/.claude/plans/weather-soa-bake-v1.md index 89b2f24f..6a45d06d 100644 --- a/.claude/plans/weather-soa-bake-v1.md +++ b/.claude/plans/weather-soa-bake-v1.md @@ -198,6 +198,69 @@ test:** `box_ranges` treats `lon_lo == lon_hi` as *wrap the whole circle*, not a *empty box*. Neither reading is forced by the spec. It needs a test pinning the chosen one, or an API that makes the ambiguity unrepresentable. +### §1.3b `D-WXS-2a` — the row-major-vs-Morton bar, PRE-REGISTERED 2026-08-13 (written and committed BEFORE the run) + +§1.3a leaves the layout a *stated deviation, not a ruling*. This section is the +bar that resolves it. **Split in two, because only one half is runnable today:** + +- **Half A — pure key-space. Runnable now**, needs no ERA5 data, no classid, + no bake. This is what is pre-registered here. +- **Half B — the ζ stencil under each layout.** Gated on `D-WXS-9`, itself gated + on the bake and therefore on `D-WXS-0`. Not pre-registered here; it inherits + this section's metrics when it runs. + +**First, a correction to §1.3a's own wording.** §1.3a called the shipped layout +"row-major". That is imprecise. The key orders bytes +`[lat_tile, lon_tile, lat_hip, lon_hip]`, so lexicographic order is +**tile-row-major, then row-major *within* a tile** — a two-level blocked order, +which already has better locality than a flat row-major over 721×1440 would. +Recording this before measuring, so the comparison is against what is actually +shipped rather than against the looser word. + +#### The two metrics (both computed over key-order index, not raw bytes) + +1. **Range count** — the number of maximal contiguous runs in key order needed + to cover **exactly** a given box, with **no false positives** (a scan that + over-reads and filters is a different, weaker thing and does not count). +2. **Neighbour locality** — the median `|key_index(a) − key_index(b)|` over the + 4-neighbourhood (`lat ± 1`, `lon ± 1`, longitude wrapping), across a + deterministic sample of cells. + +#### Arms + +| arm | layout | +|---|---| +| **SHIPPED** | `[lat_tile, lon_tile, lat_hip, lon_hip]` — what `key.rs` emits today | +| **MORTON** | the OGAR-canon reading: the two axis bytes of a tier nibble-interleaved | +| **CONTROL-BAD** | a deliberately locality-destroying order (axis bytes byte-reversed, i.e. `lon_hip` most significant) | + +#### The bar, with both halves and a kill + +- **Primary:** MORTON beats SHIPPED on **both** metrics, over a box set that + includes tile-aligned, non-tile-aligned, seam-crossing and pole-adjacent + boxes. "Beats" is stated before the run as: strictly fewer ranges on the + **median** non-tile-aligned box, **and** strictly smaller median neighbour + distance. +- **Control that can lose:** **CONTROL-BAD must be worse than both** on both + metrics. If a deliberately bad order scores like the good ones, the metric is + not measuring locality and no verdict may be read off it. +- **Stay-silent twin (non-trivial):** on a **tile-aligned** box, SHIPPED and + MORTON must produce **exactly one range each** — identical. This is §1.2's + actual load-bearing claim, and it must show **no difference** where the plan + claims none. A comparison that reports MORTON better *everywhere*, including + here, is measuring something other than what it says. +- **KILL:** if MORTON does **not** win on both metrics, the deviation is + **harmless for this workload**, §1.3a downgrades from "stated deviation owing a + decision" to a recorded note, and `D-WXS-2a` closes without a code change. + A negative result here is a real result and is the cheaper outcome — it retires + an open question rather than opening a migration. + +**Discipline note.** Half A cannot settle the *whole* question, because the +stencil (half B) is where locality is actually spent. Half A can only show +whether a difference exists **in key space at all**. If half A kills, half B is +moot; if half A confirms, half B still has to run before any migration. Stated +now so a green half A is not later read as a mandate. + ### §1.4 classid — a mint decision, NOT taken here `0x0F = Geo` already exists in the OGAR domain table; free domains are `0x03–0x06` diff --git a/crates/weather-poc/src/floor.rs b/crates/weather-poc/src/floor.rs index 03347f72..cd773e78 100644 --- a/crates/weather-poc/src/floor.rs +++ b/crates/weather-poc/src/floor.rs @@ -46,6 +46,26 @@ pub const LO_PERCENTILE: f64 = 0.4; /// See [`LO_PERCENTILE`]. pub const HI_PERCENTILE: f64 = 99.6; +/// The result of scoring an external population against a floor +/// ([`CalibratedFloor::saturation_of`]). +/// +/// Carries the non-finite count alongside the fraction **by design**: see +/// [`CalibratedFloor::saturation_of`] for why folding non-finite values into +/// the fraction turns "no data at all" into "completely saturated", and why +/// silently dropping them is the other half of the same mistake. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct SaturationScore { + /// Fraction of the **finite** values landing in the two rim buckets. + /// `0.0` when `finite == 0` — check `finite` before reading this. + pub fraction: f64, + /// How many values were finite, i.e. the denominator of `fraction`. + pub finite: usize, + /// How many values were non-finite (`NaN`, `±inf`) and therefore + /// excluded. Non-zero means the population was partly or wholly absent + /// — a fact about the data, not about the floor. + pub non_finite: usize, +} + /// A calibrated, versioned linear 256-bucket floor over `[lo, hi]`. /// /// Produced only by [`calibrate`] — there is no public constructor that @@ -179,19 +199,63 @@ impl CalibratedFloor { /// anything. Both directions bar B2 needs are "this floor scored /// against a population it was not calibrated on": a narrow floor /// scored globally, and a global floor scored on one narrow box. - /// Returns `0.0` on an empty `values` slice. - pub fn saturation_of(&self, values: &[f64]) -> f64 { - if values.is_empty() { - return 0.0; + /// + /// # Non-finite values are EXCLUDED and COUNTED, never folded in + /// + /// The `fraction` is computed over the **finite** subset only, and + /// [`SaturationScore::non_finite`] reports how many were set aside. This + /// is not fussiness — it closes the sibling of the PR #948 `lane.rs` + /// finding, in the more dangerous position: + /// + /// * [`Self::quantize`] maps non-finite input to a rim bucket silently + /// (`NaN` → `0`, `-inf` → `0`, `+inf` → `255`; see its own docs). Rim + /// is exactly what this function counts. + /// * So a folded-in `NaN` reads as **saturation**, and an all-`NaN` + /// population — *valid ARCO-ERA5 store semantics for a missing chunk*, + /// per `probes/weather-p1/README.md` §1 — would score `1.0`: + /// "completely saturated" when the truth is "no data at all". Those + /// are opposite findings and the bare fraction cannot tell them apart. + /// * This function is bar B2's **instrument**. A corrupted stored value + /// is bad; a corrupted measurement is worse, because every conclusion + /// downstream inherits it silently. + /// + /// Silently *dropping* the non-finite values would be the other half of + /// the same mistake — the caller would not learn that a population was + /// partly or wholly absent. Hence a reported count, matching this + /// crate's standing rule that a degenerate case is **reported**, never + /// folded (the same shape as [`calibrate`] and [`Self::decode`] + /// returning `None`). + /// + /// On an empty slice, or one with no finite values at all, `fraction` is + /// `0.0` and `finite` is `0` — check `finite` before reading `fraction`. + pub fn saturation_of(&self, values: &[f64]) -> SaturationScore { + let mut finite = 0usize; + let mut non_finite = 0usize; + let mut rim = 0usize; + + for &v in values { + if !v.is_finite() { + non_finite += 1; + continue; + } + finite += 1; + let b = self.quantize(v); + if b == 0 || b == (BUCKETS - 1) as u8 { + rim += 1; + } + } + + let fraction = if finite == 0 { + 0.0 + } else { + rim as f64 / finite as f64 + }; + + SaturationScore { + fraction, + finite, + non_finite, } - let rim = values - .iter() - .filter(|&&v| { - let b = self.quantize(v); - b == 0 || b == (BUCKETS - 1) as u8 - }) - .count(); - rim as f64 / values.len() as f64 } } @@ -353,9 +417,12 @@ mod tests { let wide_population = linspace(-1000.0, 1000.0, 102_400); let sat = narrow_floor.saturation_of(&wide_population); + assert_eq!(sat.non_finite, 0, "fixture population is all finite"); + assert_eq!(sat.finite, wide_population.len()); assert!( - sat > 0.9, - "expected high saturation applying a narrow floor globally, got {sat}" + sat.fraction > 0.9, + "expected high saturation applying a narrow floor globally, got {}", + sat.fraction ); } @@ -382,10 +449,93 @@ mod tests { ); let sat = global_floor.saturation_of(sub_window); + assert_eq!(sat.non_finite, 0, "fixture sub-window is all finite"); + assert_eq!(sat.finite, sub_window.len()); + assert!( + sat.fraction < 0.1, + "global floor should not saturate heavily on an interior-adjacent sub-window, got {}", + sat.fraction + ); + } + + // ── non-finite values are excluded and counted, never folded ───────── + + /// A non-finite value must NOT be counted as saturation. + /// + /// This is the sibling of the PR #948 `lane.rs` finding, in the more + /// dangerous position: `saturation_of` is bar B2's INSTRUMENT, so folding + /// `NaN` into the fraction corrupts a measurement rather than a stored + /// value, and every conclusion downstream inherits it silently. + /// + /// The first block is the sharp case: an ALL-`NaN` population is valid + /// ARCO-ERA5 store semantics for a missing chunk + /// (`probes/weather-p1/README.md` §1), and the pre-fix code would have + /// scored it `1.0` — "completely saturated" — when the truth is "no data + /// at all". + #[test] + fn non_finite_values_are_excluded_and_counted_never_scored_as_saturation() { + let population = linspace(-1000.0, 1000.0, 10_240); + let floor = calibrate(&population).expect("non-empty sample calibrates"); + + // ── the sharp case: an entirely absent field ── + let all_nan = vec![f64::NAN; 4_096]; + let sat = floor.saturation_of(&all_nan); + assert_eq!( + sat.non_finite, 4_096, + "every value must be counted as absent" + ); + assert_eq!(sat.finite, 0, "none of them is a measurement"); + assert_eq!( + sat.fraction, 0.0, + "an absent field must not read as saturation — pre-fix this was 1.0" + ); + + // ── mixed: NaN must not inflate a genuinely low fraction ── + // Interior values (the population's middle third) do not saturate. + let n = population.len(); + let interior = &population[(n / 3)..(2 * n / 3)]; + let clean = floor.saturation_of(interior); + assert_eq!(clean.non_finite, 0); assert!( - sat < 0.1, - "global floor should not saturate heavily on an interior-adjacent sub-window, got {sat}" + clean.fraction < 0.1, + "interior values must not saturate, got {}", + clean.fraction + ); + + let mut poisoned: Vec = interior.to_vec(); + poisoned.extend(std::iter::repeat_n(f64::NAN, interior.len())); + let mixed = floor.saturation_of(&poisoned); + assert_eq!(mixed.non_finite, interior.len(), "the NaN half is counted"); + assert_eq!( + mixed.finite, + interior.len(), + "the real half is the denominator" ); + assert_eq!( + mixed.fraction, clean.fraction, + "adding absent values must not move the fraction at all" + ); + + // ── +/-inf too, not just NaN: they land on the RIM buckets ── + for (name, bad) in [("+inf", f64::INFINITY), ("-inf", f64::NEG_INFINITY)] { + let mut with_inf: Vec = interior.to_vec(); + with_inf.extend(std::iter::repeat_n(bad, interior.len())); + let scored = floor.saturation_of(&with_inf); + assert_eq!( + scored.non_finite, + interior.len(), + "{name} must be counted as absent" + ); + assert_eq!( + scored.fraction, clean.fraction, + "{name} must not inflate saturation (it quantises to a rim bucket)" + ); + } + + // ── stay-silent twin: an all-finite population is unaffected ── + let sat_finite = floor.saturation_of(&population); + assert_eq!(sat_finite.non_finite, 0, "no false positives on clean data"); + assert_eq!(sat_finite.finite, population.len()); } // ── round-trip bound ───────────────────────────────────────────────── diff --git a/crates/weather-poc/src/lane.rs b/crates/weather-poc/src/lane.rs new file mode 100644 index 00000000..054420ce --- /dev/null +++ b/crates/weather-poc/src/lane.rs @@ -0,0 +1,978 @@ +//! The L4 lane: pack and unpack ONE 16-byte facet — 4-byte classid prefix + +//! 12-byte payload as `6 x (8:8)` palette pairs +//! (`.claude/v3/soa_layout/le-contract.md` §1/§2/§3's L4 row). +//! +//! Owner: the lane worker (plan `.claude/plans/weather-soa-bake-v1.md` §6.2, +//! deliverable — this file, `D-WXS-4`'s row-assembly companion). Consumes +//! [`crate::manifest`] (the ClassView-side field manifest) and +//! [`crate::floor`] (the calibrated linear quantiser); invents nothing new +//! architecturally. +//! +//! # Slot purity (le-contract.md §2) — restated in this module's own words +//! +//! **Labels and positions come from the ClassView, NEVER from a slot in the +//! payload.** A `NodeRow`'s value slab is dumb bytes; the class makes them +//! meaningful. This module is the code half of that split: it moves bytes +//! according to a [`crate::manifest::FieldManifest`] it was HANDED — it does +//! not know, and must never come to know, which ERA5 variable rides which +//! byte. There is no `match` on a variable name anywhere in this file, and +//! there must never be one. If a future change to this module needs to name +//! a specific ERA5 field to work correctly, that is the exact defect this +//! module exists to prevent — the mapping belongs in the manifest, not here. +//! +//! **The consequence, stated plainly: nothing this file writes ever encodes +//! what a byte means.** [`pack_facet`] and [`unpack_facet`] both take the +//! manifest and the calibrated floors as opaque parameters and follow +//! whatever they say. A caller who wants `v_component_of_wind @ 850 hPa` in +//! byte 5 today and byte 9 tomorrow only ever has to edit +//! `data/field_manifest_v1.tsv` — this module's behaviour follows without a +//! single line changing here. [`crate::manifest`]'s own bar B0 (mutating one +//! manifest entry must change the bytes the bake writes for at least one +//! cell) is exercised end-to-end through this module's tests, closing the +//! half `manifest.rs`'s own module docs left open ("the bake does not exist +//! yet"). +//! +//! # The layout, byte-for-byte +//! +//! | bytes | content | +//! |---|---| +//! | `0..4` | `classid: u32`, little-endian — a caller-supplied parameter, never composed or bit-sliced here | +//! | `4..16` | the 96-bit payload: 6 pairs, `payload[2p]` = pair `p`'s LOW byte, `payload[2p+1]` = pair `p`'s HIGH byte | +//! +//! `u8:u8` means **two separate bytes** — a pair is never combined into a +//! `u16`, never byte-swapped, and the low/high assignment is fixed by the +//! manifest's own `PairByte::{Lo, Hi}` convention (`manifest.rs`'s "lo/hi +//! convention" doc section), never inferred from magnitude at pack or +//! unpack time. +//! +//! # A reserved slot is a slot the manifest has no row for +//! +//! `crate::manifest`'s own docs: "reserved slots are NOT rows... their +//! absence IS their meaning: dormant, expandable later without a layout +//! change." This module enforces the two halves of that on the wire: +//! [`pack_facet`] leaves an unresolved `(facet, pair, byte)` slot at its +//! initial `0`, and [`unpack_facet`] never emits an [`UnpackedField`] for +//! one — a reserved slot is *absent* from the unpacked result, never +//! present with the decoded value of a stray `0` byte (`bucket_center(0)` +//! is a real, plausible-looking number for most floors; reporting it for an +//! unoccupied slot would be exactly the corruption this split prevents). +//! +//! # Floor-version provenance is external, per le-contract.md §2.6 point 2 +//! +//! The plan is explicit that a floor's `(lo, hi, floor_version)` is +//! **dataset metadata, aligned to the Lance version boundary** — never a +//! per-row byte, which would itself be a slot-purity break. This module +//! therefore does not stamp a version into the row. [`stamped_floor_versions`] +//! stands in for "what a reader would find in that dataset metadata": the +//! `floor_version` each occupied slot's floor carried at the moment the +//! facet was packed. [`unpack_facet`] takes that map as an explicit +//! parameter and refuses (via [`crate::floor::CalibratedFloor::decode`]) to +//! resolve any slot whose stamped version has drifted from the floor it was +//! actually handed — a version mismatch is a reported error, never a +//! silently wrong decoded value. +//! +//! # Scope — what this module is NOT +//! +//! This is one 16-byte facet. It is not the bake (`D-WXS-4`, blocked on the +//! `D-WXS-0` classid mint) and not the 512-byte row assembly (three facets +//! plus the key and edge block). Both are deferred to their own +//! deliverables; this module's job ends at one facet in, one facet out. + +use std::collections::HashMap; + +use crate::floor::CalibratedFloor; +use crate::manifest::{FieldManifest, ManifestEntry, PairByte}; + +/// Length in bytes of one L4 facet: 4-byte classid prefix + 12-byte payload +/// (le-contract.md §1). +pub const FACET_LEN: usize = 16; + +/// Length in bytes of the classid prefix (le-contract.md §1, bytes `0..4`). +const CLASSID_LEN: usize = 4; + +/// Number of `(8:8)` pairs in the L4 payload (le-contract.md §3, "6 x +/// (8:8)"). +pub const PAIR_COUNT: u8 = 6; + +const _: () = assert!(FACET_LEN == CLASSID_LEN + PAIR_COUNT as usize * 2); + +/// The byte offset **within a 16-byte facet** that a given `(pair, byte)` +/// slot occupies. +/// +/// This is a purely structural fact about the L4 layout (le-contract.md +/// §3: "every layout is exactly 12 bytes — 6x2... layouts differ only in +/// how the 96 payload bits subdivide") — it is not a label, a variable +/// name, or a display position, so exposing it does not break slot +/// purity: it answers "where is pair `p`'s low/high byte", never "what is +/// stored there". +/// +/// # Panics +/// +/// Panics if `pair >= `[`PAIR_COUNT`]. Every caller in this module reaches +/// this function only through a `0..PAIR_COUNT` loop, so the panic path is +/// unreachable in practice; it exists as a loud guard against a future +/// caller passing an out-of-range pair directly. +pub fn slot_offset(pair: u8, byte: PairByte) -> usize { + assert!( + pair < PAIR_COUNT, + "pair {pair} out of range (0..{PAIR_COUNT})" + ); + let byte_idx = match byte { + PairByte::Lo => 0, + PairByte::Hi => 1, + }; + CLASSID_LEN + (pair as usize) * 2 + byte_idx +} + +/// One resolved field read back out of a packed facet: the slot it +/// occupied, its manifest identity, and its dequantised value. +/// +/// Only ever produced by [`unpack_facet`] for a slot the manifest actually +/// resolves — a reserved-zero slot never produces one of these (see the +/// module docs). +#[derive(Debug, Clone, PartialEq)] +pub struct UnpackedField { + /// The facet index this field was read from. + pub facet: u8, + /// The pair index within the facet. + pub pair: u8, + /// Which byte of the pair. + pub byte: PairByte, + /// The ERA5 variable name, copied from the manifest entry that + /// resolved this slot (never inferred from the byte value itself). + pub variable: String, + /// The pressure level, copied from the manifest entry. + pub level_hpa: Option, + /// The dequantised value (`CalibratedFloor::bucket_center` of the raw + /// byte, under the floor whose stamped version matched). + pub value: f64, +} + +/// Everything that can go wrong packing or unpacking an L4 facet. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum LaneError { + /// An occupied manifest slot names a `floor_id` that is not present in + /// the `floors` map handed to [`pack_facet`], [`unpack_facet`], or + /// [`stamped_floor_versions`]. + UnknownFloor { + /// The facet the missing floor was needed for. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The `floor_id` the manifest entry named. + floor_id: String, + }, + /// [`pack_facet`]'s `value_of` closure returned `None` for an occupied + /// slot — every occupied slot in the target facet must have a value; a + /// caller with a genuinely missing reading should not call `pack_facet` + /// for that facet at all, or should thread a documented sentinel + /// through its own value source rather than lean on the lane to invent + /// a default. + MissingValue { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The variable the manifest names for this slot. + variable: String, + /// The level the manifest names for this slot. + level_hpa: Option, + }, + /// [`pack_facet`]'s `value_of` closure returned a **non-finite** reading + /// (`NaN`, `+inf` or `-inf`) for an occupied slot. + /// + /// # Why this is a hard error and not a silent bucket + /// + /// This is **not** a defensive nicety — it closes a measured + /// silent-corruption path, found by review on PR #948: + /// + /// * ARCO-ERA5 is **sparse by design**. `probes/weather-p1/README.md` §1 + /// records the store's `fill_value: NaN`, that several variables 404 at + /// the arc's own fixture timestep, and that in Zarr v2 *a missing chunk + /// means all-`fill_value`* — so **a 404 is valid store semantics, not a + /// fetch failure**, and an all-`NaN` field is ordinary data an ingest + /// must expect. + /// * The 404-ing list at that timestep includes `mean_sea_level_pressure`, + /// `10m_v_component_of_wind`, `surface_pressure`, + /// `total_column_water_vapour` and `total_cloud_cover` — **five + /// variables the W1 field set actually packs** (F0 pairs 0, 1, 3, 4). + /// * [`CalibratedFloor::quantize`](crate::floor::CalibratedFloor::quantize) + /// maps non-finite input to a **valid-looking bucket, silently** + /// (measured: `NaN` → `0`, `-inf` → `0`, `+inf` → `255`; Rust's + /// float→int cast saturates and sends `NaN` to zero, and `f64::clamp` + /// propagates `NaN` rather than clamping it). + /// + /// Without this guard an entire missing field would be written as + /// plausible low-bucket measurements and read back through + /// [`CalibratedFloor::bucket_center`](crate::floor::CalibratedFloor::bucket_center) + /// as ordinary numbers — the exact failure the reserved-slot rule + /// (*"a reserved slot must not read back as a plausible number"*) exists + /// to prevent, one level deeper and harder to see. + /// + /// The guard covers **all** non-finite values, not only `NaN`: `±inf` + /// land on the rim buckets, which are legitimate saturation values and + /// therefore just as indistinguishable from a real reading. + NonFiniteValue { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The variable the manifest names for this slot. + variable: String, + /// The level the manifest names for this slot. + level_hpa: Option, + }, + /// [`unpack_facet`]'s `stamped_versions` map has no entry for a + /// `floor_id` an occupied slot needs — the caller's "dataset metadata" + /// is incomplete for this facet. + MissingStampedVersion { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The `floor_id` with no stamped version. + floor_id: String, + }, + /// A slot's stamped `floor_version` (the version recorded at pack time, + /// per the module docs' "dataset metadata" framing) does not match the + /// `floor_version` the [`CalibratedFloor`] handed to [`unpack_facet`] + /// actually carries. Detected via + /// [`CalibratedFloor::decode`](crate::floor::CalibratedFloor::decode) + /// returning `None` — never silently dequantised under the wrong + /// `[lo, hi]` window. + FloorVersionMismatch { + /// The facet. + facet: u8, + /// The pair. + pair: u8, + /// The byte. + byte: PairByte, + /// The `floor_id` whose version drifted. + floor_id: String, + /// The version stamped in `stamped_versions`. + expected: u64, + /// The version the floor handed to `unpack_facet` actually has. + found: u64, + }, +} + +impl std::fmt::Display for LaneError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: no floor for floor_id {floor_id:?}" + ), + LaneError::MissingValue { + facet, + pair, + byte, + variable, + level_hpa, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: no value supplied for {variable:?} at level {level_hpa:?}" + ), + LaneError::NonFiniteValue { + facet, + pair, + byte, + variable, + level_hpa, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: non-finite reading for {variable:?} at level {level_hpa:?} \ + (an all-NaN field is valid ARCO-ERA5 store semantics for a missing chunk, never a bucket)" + ), + LaneError::MissingStampedVersion { + facet, + pair, + byte, + floor_id, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: no stamped floor_version for floor_id {floor_id:?}" + ), + LaneError::FloorVersionMismatch { + facet, + pair, + byte, + floor_id, + expected, + found, + } => write!( + f, + "facet={facet} pair={pair} byte={byte}: floor_id {floor_id:?} stamped version {expected} does not match floor's actual version {found}" + ), + } + } +} + +impl std::error::Error for LaneError {} + +/// The six `(pair, byte)` slots of one facet, in a fixed, deterministic +/// order (`pair` ascending, `Lo` before `Hi` within a pair). Both +/// [`pack_facet`] and [`unpack_facet`] walk this same order, which is what +/// makes the manifest — not iteration order — the only thing either +/// function's output depends on. +fn all_slots() -> impl Iterator { + (0..PAIR_COUNT).flat_map(|pair| [(pair, PairByte::Lo), (pair, PairByte::Hi)]) +} + +/// Packs ONE facet: for every occupied `(pair, byte)` slot the manifest +/// resolves against `facet`, quantises the value `value_of` supplies +/// (through the floor the manifest names) into that slot's byte; every +/// unresolved slot stays `0` (reserved-zero, see the module docs). +/// +/// `classid` is written verbatim as little-endian bytes into `0..4` — this +/// function never composes, mints, or bit-slices it. +/// +/// `value_of` is called once per occupied slot, in the fixed order +/// [`all_slots`] walks, with the [`ManifestEntry`] that slot resolved to. +/// It returns the raw `f64` reading for that field, or `None` if none is +/// available — which is reported as [`LaneError::MissingValue`], not +/// silently treated as zero (a `0.0` reading and a missing reading are +/// different facts, and conflating them would corrupt the bucket the +/// floor quantises `0.0` to). +/// +/// # Errors +/// +/// Returns [`LaneError::UnknownFloor`] if an occupied slot's `floor_id` is +/// not in `floors`, or [`LaneError::MissingValue`] if `value_of` returns +/// `None` for an occupied slot. Fails on the first such slot encountered +/// (in [`all_slots`] order); a facet with several bad slots reports the +/// first one, not all of them. +pub fn pack_facet( + classid: u32, + facet: u8, + manifest: &FieldManifest, + floors: &HashMap, + mut value_of: F, +) -> Result<[u8; FACET_LEN], LaneError> +where + F: FnMut(&ManifestEntry) -> Option, +{ + let mut bytes = [0u8; FACET_LEN]; + bytes[0..CLASSID_LEN].copy_from_slice(&classid.to_le_bytes()); + + for (pair, byte) in all_slots() { + let Some(entry) = manifest.resolve_slot(facet, pair, byte) else { + // Reserved-zero slot: no manifest entry, leave the byte at its + // initial 0. This IS the slot's meaning (dormant, expandable), + // never an omission to fill in. + continue; + }; + + let floor = floors + .get(&entry.floor_id) + .ok_or_else(|| LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + let value = value_of(entry).ok_or_else(|| LaneError::MissingValue { + facet, + pair, + byte, + variable: entry.variable.clone(), + level_hpa: entry.level_hpa, + })?; + + // A non-finite reading must NEVER reach `quantize`. See + // `LaneError::NonFiniteValue` for the measured silent-corruption + // path this guard closes. + if !value.is_finite() { + return Err(LaneError::NonFiniteValue { + facet, + pair, + byte, + variable: entry.variable.clone(), + level_hpa: entry.level_hpa, + }); + } + + bytes[slot_offset(pair, byte)] = floor.quantize(value); + } + + Ok(bytes) +} + +/// Unpacks ONE facet: for every `(pair, byte)` slot `manifest` resolves +/// against `facet`, dequantises the raw byte through the matching floor — +/// gated on `stamped_versions` agreeing with that floor's actual +/// `floor_version` — and returns it as an [`UnpackedField`]. A slot the +/// manifest does not resolve is skipped entirely: it produces no +/// `UnpackedField`, reserved or not, whatever raw byte value it holds. +/// +/// # Errors +/// +/// Returns [`LaneError::UnknownFloor`] if an occupied slot's `floor_id` is +/// not in `floors`, [`LaneError::MissingStampedVersion`] if it is not in +/// `stamped_versions`, or [`LaneError::FloorVersionMismatch`] if the +/// stamped version does not match the floor's own +/// [`CalibratedFloor::floor_version`](crate::floor::CalibratedFloor::floor_version) +/// (checked via [`CalibratedFloor::decode`](crate::floor::CalibratedFloor::decode), +/// never bypassed). Fails on the first such slot, in [`all_slots`] order. +pub fn unpack_facet( + bytes: &[u8; FACET_LEN], + facet: u8, + manifest: &FieldManifest, + floors: &HashMap, + stamped_versions: &HashMap, +) -> Result, LaneError> { + let mut out = Vec::new(); + + for (pair, byte) in all_slots() { + let Some(entry) = manifest.resolve_slot(facet, pair, byte) else { + continue; + }; + + let floor = floors + .get(&entry.floor_id) + .ok_or_else(|| LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + let expected_version = stamped_versions + .get(&entry.floor_id) + .copied() + .ok_or_else(|| LaneError::MissingStampedVersion { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + let raw = bytes[slot_offset(pair, byte)]; + let value = + floor + .decode(raw, expected_version) + .ok_or_else(|| LaneError::FloorVersionMismatch { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + expected: expected_version, + found: floor.floor_version(), + })?; + + out.push(UnpackedField { + facet, + pair, + byte, + variable: entry.variable.clone(), + level_hpa: entry.level_hpa, + value, + }); + } + + Ok(out) +} + +/// Collects the `floor_version` every occupied slot of `facet` actually +/// used, keyed by `floor_id`. +/// +/// This stands in for the "dataset metadata, aligned to the Lance version +/// boundary" the plan describes (module docs, "Floor-version provenance is +/// external") — in the real bake it would be read from that metadata; here +/// it is derived directly from the same `floors` map [`pack_facet`] was +/// given, so a caller can produce it immediately after packing and hand it +/// to [`unpack_facet`] as the "what was this written under" record. +/// +/// # Errors +/// +/// Returns [`LaneError::UnknownFloor`] under the same condition +/// [`pack_facet`] would have failed under — an occupied slot names a +/// `floor_id` not present in `floors`. +pub fn stamped_floor_versions( + manifest: &FieldManifest, + facet: u8, + floors: &HashMap, +) -> Result, LaneError> { + let mut out = HashMap::new(); + + for (pair, byte) in all_slots() { + let Some(entry) = manifest.resolve_slot(facet, pair, byte) else { + continue; + }; + + let floor = floors + .get(&entry.floor_id) + .ok_or_else(|| LaneError::UnknownFloor { + facet, + pair, + byte, + floor_id: entry.floor_id.clone(), + })?; + + out.insert(entry.floor_id.clone(), floor.floor_version()); + } + + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::floor::calibrate; + + /// The exact manifest header, spelled literally rather than imported — + /// `manifest::HEADER` is a private constant of that module (by design: + /// it is validated internally, not a public API surface). + const HEADER: &str = "facet\tpair\tbyte\tvariable\tlevel_hpa\tunit\tfloor_id"; + + /// A small, self-contained fixture manifest for this module's tests: + /// facet 0 only, three occupied slots spread across two pairs, leaving + /// both a fully-reserved pair (2) and a half-reserved pair (1, hi byte) + /// to exercise the reserved-slot behaviour without depending on the + /// full committed W1 manifest. + /// + /// | facet | pair | byte | variable | floor_id | + /// |---|---|---|---|---| + /// | 0 | 0 | lo | fieldA | fa | + /// | 0 | 0 | hi | fieldB | fb | + /// | 0 | 1 | lo | fieldC | fa | + /// | 0 | 1 | hi | *(reserved)* | | + /// | 0 | 2..5 | * | *(fully reserved)* | | + fn fixture_manifest() -> FieldManifest { + let text = format!( + "{HEADER}\n\ + 0\t0\tlo\tfieldA\tsurface\tK\tfa\n\ + 0\t0\thi\tfieldB\tsurface\tPa\tfb\n\ + 0\t1\tlo\tfieldC\tsurface\tK\tfa\n" + ); + FieldManifest::parse(&text).expect("fixture manifest must parse") + } + + /// Deterministic evenly-spaced sample, mirroring `floor.rs`'s own test + /// helper (private to that module, so re-declared here). + fn linspace(lo: f64, hi: f64, n: usize) -> Vec { + assert!(n >= 2, "linspace needs at least 2 points"); + (0..n) + .map(|i| lo + (hi - lo) * i as f64 / (n as f64 - 1.0)) + .collect() + } + + /// Two distinctly-ranged floors, keyed exactly as [`fixture_manifest`]'s + /// entries name them (`"fa"`, `"fb"`). + fn fixture_floors() -> HashMap { + let mut floors = HashMap::new(); + floors.insert( + "fa".to_string(), + calibrate(&linspace(-100.0, 100.0, 10_000)).expect("fa calibrates"), + ); + floors.insert( + "fb".to_string(), + calibrate(&linspace(900.0, 1_100.0, 10_000)).expect("fb calibrates"), + ); + floors + } + + /// A `value_of` closure over the three fixture fields, matched by + /// variable name (the closure — supplied by the TEST, not by this + /// module — is where variable-name knowledge is allowed to live; the + /// module under test never inspects `entry.variable` itself to decide + /// what to do, only to report it back to the caller). + fn fixture_values( + field_a: f64, + field_b: f64, + field_c: f64, + ) -> impl FnMut(&ManifestEntry) -> Option { + move |entry: &ManifestEntry| match entry.variable.as_str() { + "fieldA" => Some(field_a), + "fieldB" => Some(field_b), + "fieldC" => Some(field_c), + _ => None, + } + } + + // ── round-trip ─────────────────────────────────────────────────────── + + #[test] + fn round_trip_recovers_every_occupied_field_within_half_a_bucket() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + let field_a = 12.3_f64; + let field_b = 1_005.7_f64; + let field_c = -42.0_f64; + + let bytes = pack_facet( + 0xDEAD_BEEF, + 0, + &manifest, + &floors, + fixture_values(field_a, field_b, field_c), + ) + .expect("pack must succeed"); + + let stamped = stamped_floor_versions(&manifest, 0, &floors).expect("stamping must succeed"); + let unpacked = + unpack_facet(&bytes, 0, &manifest, &floors, &stamped).expect("unpack must succeed"); + + assert_eq!(unpacked.len(), 3, "exactly the three occupied slots"); + + let expect_value = |variable: &str| match variable { + "fieldA" => field_a, + "fieldB" => field_b, + "fieldC" => field_c, + other => panic!("unexpected variable in unpacked output: {other}"), + }; + + for field in &unpacked { + let entry = manifest + .resolve_slot(field.facet, field.pair, field.byte) + .expect("unpacked field must resolve back to a manifest entry"); + let floor = floors + .get(&entry.floor_id) + .expect("resolved entry's floor_id must be in the fixture floors map"); + let (lo, hi) = floor.bounds(); + let half_bucket = (hi - lo) / (2.0 * 256.0); + let original = expect_value(&field.variable); + let delta = (field.value - original).abs(); + assert!( + delta <= half_bucket + 1e-9, + "{}: round-trip error {delta} exceeds the +/-half-bucket bound {half_bucket}", + field.variable + ); + } + } + + // ── reserved slots stay zero AND read as absent ───────────────────── + + #[test] + fn reserved_slots_are_byte_zero_after_pack_and_absent_after_unpack() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + let bytes = pack_facet(0x1, 0, &manifest, &floors, fixture_values(1.0, 2.0, 3.0)) + .expect("pack must succeed"); + + // Fully-reserved pair (pair 2, both bytes) is byte-zero. + assert_eq!(bytes[slot_offset(2, PairByte::Lo)], 0); + assert_eq!(bytes[slot_offset(2, PairByte::Hi)], 0); + // Half-reserved pair (pair 1, hi byte only) is also byte-zero. + assert_eq!(bytes[slot_offset(1, PairByte::Hi)], 0); + + let stamped = stamped_floor_versions(&manifest, 0, &floors).expect("stamping must succeed"); + let unpacked = + unpack_facet(&bytes, 0, &manifest, &floors, &stamped).expect("unpack must succeed"); + + // Absence, not a decoded-0.0 entry: no UnpackedField at any + // reserved slot. + let has_slot = + |pair: u8, byte: PairByte| unpacked.iter().any(|f| f.pair == pair && f.byte == byte); + assert!(!has_slot(2, PairByte::Lo), "pair 2 lo must be absent"); + assert!(!has_slot(2, PairByte::Hi), "pair 2 hi must be absent"); + assert!(!has_slot(1, PairByte::Hi), "pair 1 hi must be absent"); + + // The occupied slots ARE present, so the absence above is really a + // filtered set and not an empty-everything bug. + assert!(has_slot(0, PairByte::Lo), "pair 0 lo must be present"); + assert!(has_slot(0, PairByte::Hi), "pair 0 hi must be present"); + assert!(has_slot(1, PairByte::Lo), "pair 1 lo must be present"); + } + + // ── manifest is load-bearing (can-it-fire) ────────────────────────── + + #[test] + fn moving_a_field_to_a_different_slot_changes_the_packed_bytes() { + let manifest_a = fixture_manifest(); + // Same fields, but `fieldA` moves from (pair 0, lo) to the + // previously fully-reserved (pair 2, lo) — a genuinely different + // manifest, not a relabelling. + let manifest_b_text = format!( + "{HEADER}\n\ + 0\t2\tlo\tfieldA\tsurface\tK\tfa\n\ + 0\t0\thi\tfieldB\tsurface\tPa\tfb\n\ + 0\t1\tlo\tfieldC\tsurface\tK\tfa\n" + ); + let manifest_b = + FieldManifest::parse(&manifest_b_text).expect("moved-slot manifest must parse"); + + let floors = fixture_floors(); + let value_of = || fixture_values(12.3, 1_005.7, -42.0); + + let bytes_a = pack_facet(0x1, 0, &manifest_a, &floors, value_of()) + .expect("pack under manifest_a must succeed"); + let bytes_b = pack_facet(0x1, 0, &manifest_b, &floors, value_of()) + .expect("pack under manifest_b must succeed"); + + assert_ne!( + bytes_a, bytes_b, + "moving fieldA to a different slot must change the packed bytes" + ); + // Specifically: fieldA's quantised byte now lives at pair 2 lo + // (manifest_a leaves that slot reserved-zero, manifest_b occupies + // it), and the pair-0-lo slot it vacated is back to reserved-zero + // under manifest_b. + assert_eq!(bytes_a[slot_offset(2, PairByte::Lo)], 0); + assert_eq!(bytes_b[slot_offset(0, PairByte::Lo)], 0); + // Anti-vacuity: the relocated byte is a real, non-zero quantised + // value -- not a coincidental all-zero match between the two facets. + assert_ne!(bytes_a[slot_offset(0, PairByte::Lo)], 0); + assert_eq!( + bytes_b[slot_offset(2, PairByte::Lo)], + bytes_a[slot_offset(0, PairByte::Lo)], + "fieldA's quantised byte value should be identical, just relocated" + ); + } + + // ── stay-silent twin: reordered-but-identical manifest ────────────── + + #[test] + fn reordering_manifest_rows_does_not_change_the_packed_bytes() { + let manifest_forward = fixture_manifest(); + + let reordered_text = format!( + "{HEADER}\n\ + 0\t1\tlo\tfieldC\tsurface\tK\tfa\n\ + 0\t0\thi\tfieldB\tsurface\tPa\tfb\n\ + 0\t0\tlo\tfieldA\tsurface\tK\tfa\n" + ); + let manifest_reordered = + FieldManifest::parse(&reordered_text).expect("reordered manifest must parse"); + + let floors = fixture_floors(); + let value_of = || fixture_values(12.3, 1_005.7, -42.0); + + let bytes_forward = pack_facet(0x1, 0, &manifest_forward, &floors, value_of()) + .expect("pack under forward-order manifest must succeed"); + let bytes_reordered = pack_facet(0x1, 0, &manifest_reordered, &floors, value_of()) + .expect("pack under reordered manifest must succeed"); + + assert_eq!( + bytes_forward, bytes_reordered, + "a byte-different-but-semantically-identical manifest must pack identically" + ); + } + + // ── the pair is two bytes, never combined or swapped ──────────────── + + #[test] + fn lo_and_hi_bytes_of_a_pair_land_at_their_own_index_never_swapped() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + // fieldA (lo, floor "fa" over [-100, 100]) near its floor's low + // bound; fieldB (hi, floor "fb" over [900, 1100]) near its floor's + // high bound -- deliberately far apart so a swap or a u16-combine + // is impossible to miss. + let field_a = -95.0_f64; + let field_b = 1_095.0_f64; + let field_c = 0.0_f64; + + let bytes = pack_facet( + 0x1, + 0, + &manifest, + &floors, + fixture_values(field_a, field_b, field_c), + ) + .expect("pack must succeed"); + + let expected_lo = floors["fa"].quantize(field_a); + let expected_hi = floors["fb"].quantize(field_b); + + assert_eq!( + bytes[slot_offset(0, PairByte::Lo)], + expected_lo, + "the lo byte must hold fieldA's own quantised value" + ); + assert_eq!( + bytes[slot_offset(0, PairByte::Hi)], + expected_hi, + "the hi byte must hold fieldB's own quantised value, not fieldA's" + ); + // Anti-vacuity: the two really are different bytes at different + // indices -- a swap-bug (writing lo into hi's slot or vice versa) + // would make this assertion pass for the wrong reason if the two + // expected values happened to coincide. + assert_ne!( + expected_lo, expected_hi, + "fixture must produce genuinely distinct lo/hi bytes to be a real test" + ); + assert_eq!( + slot_offset(0, PairByte::Lo) + 1, + slot_offset(0, PairByte::Hi) + ); + } + + // ── floor-version mismatch is detected, never silently mis-decoded ── + + #[test] + fn a_stamped_floor_version_mismatch_is_reported_not_silently_mis_decoded() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + let bytes = pack_facet( + 0x1, + 0, + &manifest, + &floors, + fixture_values(12.3, 1_005.7, -42.0), + ) + .expect("pack must succeed"); + + let mut stamped = + stamped_floor_versions(&manifest, 0, &floors).expect("stamping must succeed"); + + // Corrupt only floor "fa"'s stamped version (shared by fieldA and + // fieldC); floor "fb" (fieldB) is untouched. + let real_fa_version = stamped["fa"]; + let bogus_fa_version = real_fa_version.wrapping_add(1); + assert_ne!( + bogus_fa_version, real_fa_version, + "fixture must construct a genuinely different version" + ); + stamped.insert("fa".to_string(), bogus_fa_version); + + let err = unpack_facet(&bytes, 0, &manifest, &floors, &stamped) + .expect_err("a stamped version drift must be reported, not silently decoded"); + + match err { + LaneError::FloorVersionMismatch { + floor_id, + expected, + found, + .. + } => { + assert_eq!(floor_id, "fa"); + assert_eq!(expected, bogus_fa_version); + assert_eq!(found, real_fa_version); + } + other => panic!("expected FloorVersionMismatch, got {other:?}"), + } + + // Positive control: restoring the correct stamped version decodes + // cleanly again, proving the failure above was really about the + // version and not some other defect in the fixture. + stamped.insert("fa".to_string(), real_fa_version); + let unpacked = unpack_facet(&bytes, 0, &manifest, &floors, &stamped) + .expect("unpack with the correct stamped version must succeed"); + assert_eq!(unpacked.len(), 3); + } + + // ── error paths this module owns ───────────────────────────────────── + + #[test] + fn pack_reports_a_missing_value_rather_than_defaulting_to_zero() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + // A value_of closure that always returns None -- every occupied + // slot is "missing". + let err = pack_facet(0x1, 0, &manifest, &floors, |_entry| None) + .expect_err("a missing value must be reported, not silently zeroed"); + + assert!(matches!(err, LaneError::MissingValue { pair: 0, .. })); + } + + /// A non-finite reading is REJECTED, and the paired half proves the same + /// slot packs fine when the reading is finite. + /// + /// This is the regression for the PR #948 review finding. ARCO-ERA5 is + /// sparse by design: `probes/weather-p1/README.md` §1 records + /// `fill_value: NaN` and that a 404 chunk means an all-`NaN` field is + /// **valid store semantics**, at the arc's own fixture timestep, for five + /// variables the W1 field set actually packs. + /// + /// The third assertion block is what makes this a real test rather than a + /// restatement of the guard: it shows what the guard PREVENTS, by + /// quantising the same non-finite values directly and observing that they + /// land on ordinary, plausible buckets. + #[test] + fn pack_rejects_non_finite_readings_which_would_otherwise_become_plausible_buckets() { + let manifest = fixture_manifest(); + let floors = fixture_floors(); + + // ── can-fire: every flavour of non-finite is refused ── + // Only ONE field is poisoned per run; the others stay finite, so a + // guard that fired on everything would fail the twin below. + for (name, bad) in [ + ("NaN", f64::NAN), + ("+inf", f64::INFINITY), + ("-inf", f64::NEG_INFINITY), + ] { + let err = pack_facet(0x1, 0, &manifest, &floors, move |entry| { + if entry.variable == "fieldA" { + Some(bad) + } else { + Some(1.0) + } + }) + .expect_err("a non-finite reading must be refused"); + assert!( + matches!(err, LaneError::NonFiniteValue { ref variable, .. } if variable == "fieldA"), + "{name} must be reported as NonFiniteValue for fieldA, got {err:?}" + ); + } + + // ── stay-silent twin (non-trivial): the SAME slot, finite, packs ── + let ok = pack_facet( + 0x1, + 0, + &manifest, + &floors, + fixture_values(12.3, 1_005.7, -42.0), + ) + .expect("finite readings at the same slots must pack cleanly"); + assert_ne!( + ok, [0u8; FACET_LEN], + "the finite pack must actually write something" + ); + + // ── what the guard PREVENTS (the anti-vacuity half) ── + // Without the guard these values reach `quantize`, which maps them to + // ordinary buckets with no signal that anything was wrong. + let floor = floors.get("fa").expect("fixture floor fa"); + let (lo, hi) = floor.bounds(); + for (name, bad) in [ + ("NaN", f64::NAN), + ("+inf", f64::INFINITY), + ("-inf", f64::NEG_INFINITY), + ] { + let bucket = floor.quantize(bad); + let decoded = floor.bucket_center(bucket); + assert!( + decoded >= lo && decoded <= hi, + "{name} quantised to bucket {bucket}, decoding to {decoded} — inside [{lo}, {hi}], \ + i.e. indistinguishable from a real reading. This is why the guard exists." + ); + } + } + + #[test] + fn pack_reports_an_unknown_floor_rather_than_skipping_it() { + let manifest = fixture_manifest(); + // Deliberately omit "fa" from the floors map. + let mut floors = fixture_floors(); + floors.remove("fa"); + + let err = pack_facet(0x1, 0, &manifest, &floors, fixture_values(1.0, 2.0, 3.0)) + .expect_err("an unresolvable floor_id must be reported"); + + assert!(matches!( + err, + LaneError::UnknownFloor { floor_id, .. } if floor_id == "fa" + )); + } +} diff --git a/crates/weather-poc/src/lib.rs b/crates/weather-poc/src/lib.rs index 40d5801e..42546f38 100644 --- a/crates/weather-poc/src/lib.rs +++ b/crates/weather-poc/src/lib.rs @@ -38,4 +38,5 @@ pub mod floor; pub mod key; +pub mod lane; pub mod manifest;