diff --git a/claude-notes/plans/2026-08-08-osm-soa-cockpit-wiring.md b/claude-notes/plans/2026-08-08-osm-soa-cockpit-wiring.md index 39726fa80..889f41e05 100644 --- a/claude-notes/plans/2026-08-08-osm-soa-cockpit-wiring.md +++ b/claude-notes/plans/2026-08-08-osm-soa-cockpit-wiring.md @@ -1140,3 +1140,92 @@ Two verification-method notes, both cost a run: about visibility**. The probe picks by measured `getBoundingClientRect`. - [x] Points-only → click any dot for its real OSM identity and tags. + +## Phase 8 — rehydration: the decode half of the codec (2026-08-11) + +Operator framing, which reshaped this phase before it was built: *"I'm not +talking about a rebake to optimize — I'm talking about necessary rehydration +rules from the same business logic you already had in mind when you encoded +it."* And the acceptance criterion: *"and if it has any gaps the POC can show +it."* So: no new design decisions, no probes — the encode side already fixed +the rules; this phase writes them down as the paired decode, and the POC is +the falsifier surface. + +The confirming discovery: `read.rs` was **already computing every way's full +vertex chain** (`cells: Vec`, z=32) and dropping it after `mean_cell`. +The chain was never missing — it was discarded. Rehydration is un-discarding +it. + +### The `.chains` sidecar (osm-soa-bake PR #23, merged branch pending) + +`src/chains.rs` — encode AND decode in ONE module, same crate as the bake, so +a consumer can never re-interpret bytes (the osm_tiles V1/V3 drift lesson: +Berlin HEEL `0x624b` vs `0xc8e1`). + +- Format: magic `OSMCHNS1` · `slab_digest: u64` (pin) · count · blob_len · + ordinal-sorted index (`ordinal/offset/len` u32×3) · blob = n varint + + absolute first vertex + zigzag-varint `(dx,dy)` deltas. +- Delta-position varints per areal_probe P6's measured verdict ("what the PBF + already spends"). Cells are z=32 **integers** ⇒ the roundtrip falsifier is + exact equality, not epsilon. +- The digest pin is enforced at open: chains from one bake against a different + slab are refused loudly, not served. +- Bake emission: tagged ways only, deduped by ordinal (continuation rows share + one), written in the pre-rename window so a slab publishes only alongside + BOTH sidecars. + +Berlin: `berlin.chains` = 63,777,240 B, sha256 +`276253f00503d4171c66947c5abda48627667aca51d7e0a6b891f32798350aae`, uploaded +to `s3://$AWS_S3_BUCKET_NAME/q2/bakes/berlin-v1/` with an additive 3-line +`SHA256SUMS` (old binaries reading only soa+books still verify). The bake is +byte-deterministic (slab digest `8ec93a6ee63e89d2` across two runs), which is +what allowed pinning new sidecars to the already-published slab instead of a +`berlin-v2`. + +### cockpit-server: `GET /api/osm/geometry/:idx` + the shape layer + +- `osm_features.rs`: `open_chains()` (OnceLock, digest-verified), + `query_geometry` (row → identity ordinal → chain → `tile_to_lonlat` + points), `osm_geometry_handler` — **404 when no chain**, never + 200-with-empty, so "node/relation, no shape stored" and "empty shape" can + never be confused. +- `osm.rs`: an SVG `shapeLayer` inside `#tiles` (inherits the map transform), + `classFor(tags)` — water `#2b6cb088`, building `#8fa0b888`, wood/green + fills, highway = stroke-only `#ffd166` — and `showShape(idx)` wired into + the existing click detail. `vector-effect:non-scaling-stroke`. +- `osm_slab_hydrate.rs`: `ARTIFACTS` grew to + `["berlin.soa", "berlin.books", "berlin.chains"]` — a deploy now hydrates + the geometry sidecar too (~1.42 GB cold total). Volume sizing: ≥ 2 GB. + +### Measured, on the real Berlin bake (browser, hermetic) + +A REAL click (not a scripted lookup) on a harbour dot resolved **"Westhafen I"** +(`natural=water`, `water=harbour`) and drew its shore ring filled blue — +literally the operator's See/Ufer model on a water body: the stored edge is +the Ufer, the fill classification is the rehydration rule. Also drawn via the +page's own `showFeature`: an 8-pt building ring (`#8fa0b888`), a 48-pt +landuse ring (`#2f6b4a66`), a highway open polyline; a node correctly 404s. +Zero page errors. + +### Gaps the POC now SHOWS (the operator's criterion, working as intended) + +1. **Shapes are click-only.** The operator's target is "fläche und darüber + die Straßen nodes als zusatz overlay" — area fill as a BASE layer with the + street/node overlay above it. The decode + classification exist; what's + missing is bulk retrieval (a per-tile geometry endpoint) and z-ordering. +2. **Multipolygon relations are unassembled.** A lake with an island is a + relation of ways; chains store per-way rings only, so the island hole is + not subtracted. Relation assembly is a bake-side concern (the encode knows + the member roles), not a client heuristic. +3. **Small rings are sub-pixel at overview zoom.** An 8-vertex apartment ring + is meters wide — invisible at z12, correct at city zoom. A base layer + would want the cascade-cell representative form here, not per-feature + rings. + +- [x] `.chains` codec, encode+decode paired in osm-soa-bake (PR #23) +- [x] Bake emits sidecar; S3 upload + additive SHA256SUMS +- [x] `GET /api/osm/geometry/:idx` + digest-pinned open +- [x] Click → classified shape in the browser, verified on real data +- [x] Hydrate downloads `berlin.chains` +- [ ] Base fill layer + node/street overlay (gap 1) +- [ ] Relation assembly at bake time (gap 2) diff --git a/crates/cockpit-server/src/main.rs b/crates/cockpit-server/src/main.rs index 8d6644214..984624246 100644 --- a/crates/cockpit-server/src/main.rs +++ b/crates/cockpit-server/src/main.rs @@ -273,6 +273,7 @@ async fn main() { .route("/api/osm/tile/:z/:x/:y", get(osm_tiles::osm_tile_meta_handler)) .route("/api/osm/features/:z/:x/:y", get(osm_features::osm_features_handler)) .route("/api/osm/feature/:idx", get(osm_features::osm_feature_handler)) + .route("/api/osm/geometry/:idx", get(osm_features::osm_geometry_handler)) // The /OSM cockpit page — a slippy map over the tile material, HHTL live. .route("/osm", get(osm::osm_page_handler)) // Garmin terrain scenes, mod-rewrite style: /api/garmin/:location resolves the diff --git a/crates/cockpit-server/src/osm.rs b/crates/cockpit-server/src/osm.rs index 72d7570fc..683fdc1b2 100644 --- a/crates/cockpit-server/src/osm.rs +++ b/crates/cockpit-server/src/osm.rs @@ -190,6 +190,47 @@ function visibleGrid(){ // each of ~49 arrivals. Painting only what is missing makes it 1x. let drawnCells=new Set(); +// The clicked feature's rehydrated SHAPE — the decode half of the .chains +// codec, drawn. A closed ring carrying an areal tag is FILLED (the operator's +// edge model: the ring is the shore; the tag says what is inside); an open +// chain is stroked. Classification is from tags the click already fetched. +function shapeLayer(){ + let svg=document.getElementById('shape'); + if(!svg){ + svg=document.createElementNS('http://www.w3.org/2000/svg','svg'); + svg.id='shape'; + svg.setAttribute('width','1'); svg.setAttribute('height','1'); + svg.style.cssText='position:absolute;left:0;top:0;overflow:visible;pointer-events:none;z-index:3'; + } + if(svg.parentElement!==tilesEl) tilesEl.appendChild(svg); // render() clears #tiles + return svg; +} +function classFor(tags){ + const t=tags||{}; + if(t.natural==='water'||t.waterway) return {fill:'#2b6cb088',stroke:'#7db3ff',w:1.5}; + if(t.building) return {fill:'#8fa0b888',stroke:'#c9d3e0',w:1}; + if(t.natural==='wood'||t.landuse==='forest') return {fill:'#1d4d2b88',stroke:'#4a8f63',w:1}; + if(t.landuse||t.leisure||t.natural) return {fill:'#2f6b4a66',stroke:'#5aa87a',w:1}; + if(t.highway) return {fill:'none',stroke:'#ffd166',w:2.5}; + return {fill:null,stroke:'#ffb454',w:1.5}; +} +async function showShape(idx,tags){ + const svg=shapeLayer(); svg.innerHTML=''; + const r=await fetch(`/api/osm/geometry/${idx}`); + if(!r.ok) return false; // a node: no chain is the answer + const g=await r.json(); + const pts=g.points.map(([lon,lat])=>`${(lon2x(lon,z)*256).toFixed(1)},${(lat2y(lat,z)*256).toFixed(1)}`).join(' '); + const c=classFor(tags); + const el=document.createElementNS('http://www.w3.org/2000/svg', g.closed?'polygon':'polyline'); + el.setAttribute('points',pts); + el.setAttribute('fill', g.closed && c.fill ? c.fill : 'none'); + el.setAttribute('stroke',c.stroke); + el.setAttribute('stroke-width',c.w); + el.setAttribute('vector-effect','non-scaling-stroke'); + svg.appendChild(el); + return true; +} + function paintFeatures(){ const visibleKeys=[]; if(showFeatures){ @@ -277,9 +318,11 @@ async function showFeature(idx, el){ if(d.error){ box.innerHTML='

'+d.error+'

'; return; } const rows=Object.entries(d.tags||{}) .map(([k,v])=>`
${k}${v}
`).join(''); + const drawn=await showShape(idx, d.tags); box.innerHTML = `
osm key${d.osm_key||'—'}
` - + (rows || '

no tags on this element

'); + + (rows || '

no tags on this element

') + + (drawn ? '' : '

point feature — no shape chain

'); }catch(err){ box.innerHTML='

lookup failed: '+err+'

'; } } diff --git a/crates/cockpit-server/src/osm_features.rs b/crates/cockpit-server/src/osm_features.rs index 14d96240a..7d10172a4 100644 --- a/crates/cockpit-server/src/osm_features.rs +++ b/crates/cockpit-server/src/osm_features.rs @@ -540,6 +540,126 @@ pub async fn osm_feature_handler(Path(idx): Path) -> Response { } } +/// The `.chains` sidecar — vertex chains for tagged ways, opened once. +/// +/// **The digest pin is enforced, not decorative:** a chains file whose +/// `slab_digest` does not match the mapped slab is refused entirely, because +/// serving ring geometry from one bake against identities of another is +/// silent cross-bake corruption — the exact drift the pin exists to make loud. +static CHAINS: OnceLock> = OnceLock::new(); + +fn open_chains() -> Option<&'static osm_soa_bake::chains::Chains> { + CHAINS + .get_or_init(|| { + let slab = open_slab()?; + let path = std::path::PathBuf::from(std::env::var("OSM_SLAB_PATH").ok()?) + .with_extension("chains"); + let bytes = std::fs::read(&path).ok()?; + match osm_soa_bake::chains::Chains::from_bytes(bytes) { + Ok(ch) => { + let want = osm_soa_bake::codebook::hash_slab(slab); + if ch.slab_digest != want { + tracing::error!( + got = format_args!("{:016x}", ch.slab_digest), + want = format_args!("{want:016x}"), + "osm chains: sidecar pinned to a DIFFERENT slab; refusing" + ); + return None; + } + tracing::info!(path = %path.display(), ways = ch.len(), "osm chains: loaded"); + Some(ch) + } + Err(e) => { + tracing::warn!(path = %path.display(), error = ?e, "osm chains: unreadable; geometry unavailable"); + None + } + } + }) + .as_ref() +} + +/// One feature's rehydrated shape — the decode half of the chains codec. +#[derive(Debug, Serialize, PartialEq)] +pub struct FeatureGeometryOut { + pub idx: usize, + pub ordinal: Option, + /// First vertex == last vertex and at least 4 vertices — a ring. The + /// renderer fills a ring carrying an areal tag and strokes everything else; + /// that CLASSIFICATION is the client's, the SHAPE is the codec's. + pub closed: bool, + /// `[lon, lat]` pairs in way order, decoded from z=32 cells — the same + /// grid the anchors live on, so a way's first vertex and a node at the + /// same position agree exactly. + pub points: Vec<[f64; 2]>, +} + +fn query_geometry(bytes: &[u8], idx: usize) -> Result, String> { + let slab = RowSlab::new(bytes).map_err(|e| format!("slab bytes not row-aligned: {e:?}"))?; + if idx >= slab.len() { + return Err(format!( + "row {idx} is past the end of the slab ({})", + slab.len() + )); + } + let Some(rows) = slab.rows() else { + return Ok(None); + }; + let Some((_, ordinal)) = read_identity(&rows[idx]) else { + return Ok(None); + }; + let Some(chains) = open_chains() else { + return Ok(None); + }; + let chain = chains + .get(ordinal) + .map_err(|e| format!("chain record for ordinal {ordinal} is malformed: {e:?}"))?; + let Some(chain) = chain else { + return Ok(None); // a node or relation: no chain is a real answer + }; + let closed = chain.len() >= 4 && chain.first() == chain.last(); + let points = chain + .iter() + .map(|c| { + let (lon, lat) = osm_soa_bake::tms::tile_to_lonlat(c.x, c.y_xyz); + [lon, lat] + }) + .collect(); + Ok(Some(FeatureGeometryOut { + idx, + ordinal: Some(ordinal), + closed, + points, + })) +} + +/// `GET /api/osm/geometry/:idx` — the dot's SHAPE. 404 (not 200-with-empty) +/// when the feature has no chain, so "no geometry stored" and "empty geometry" +/// can never be confused. +pub async fn osm_geometry_handler(Path(idx): Path) -> Response { + let Some(bytes) = open_slab() else { + return ( + StatusCode::SERVICE_UNAVAILABLE, + Json(serde_json::json!({ + "error": "OSM_SLAB_PATH is not set or the baked slab could not be opened", + })), + ) + .into_response(); + }; + match query_geometry(bytes, idx) { + Ok(Some(out)) => Json(out).into_response(), + Ok(None) => ( + StatusCode::NOT_FOUND, + Json(serde_json::json!({ "error": "no chain stored for this feature" })), + ) + .into_response(), + Err(e) => ( + StatusCode::BAD_REQUEST, + Json(serde_json::json!({ "error": e })), + ) + .into_response(), + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/cockpit-server/src/osm_slab_hydrate.rs b/crates/cockpit-server/src/osm_slab_hydrate.rs index ff92e3d94..4ac933196 100644 --- a/crates/cockpit-server/src/osm_slab_hydrate.rs +++ b/crates/cockpit-server/src/osm_slab_hydrate.rs @@ -4,10 +4,10 @@ //! //! ```text //! S3 (durable source of truth) -//! │ s3://$AWS_S3_BUCKET_NAME//{berlin.soa, berlin.books, SHA256SUMS} +//! │ s3://$AWS_S3_BUCKET_NAME//{berlin.soa, berlin.books, berlin.chains, SHA256SUMS} //! ▼ //! $RAILWAY_VOL (persistence across container rebuilds — a CACHE, not truth) -//! │ /osm/{berlin.soa, berlin.books} +//! │ /osm/{berlin.soa, berlin.books, berlin.chains} //! ▼ //! mmap ([`crate::osm_features::open_slab`], unchanged) //! ``` @@ -56,10 +56,13 @@ use sha2::{Digest, Sha256}; /// Default S3 prefix holding the bake. Overridable with `OSM_SLAB_S3_PREFIX`. const DEFAULT_PREFIX: &str = "q2/bakes/berlin-v1"; -/// The slab and its codebook sidecar. Both are required: `RowSlab` can read -/// positions without the books, but identity resolution needs them, and a slab -/// without its books is what the bake itself calls unreadable. -const ARTIFACTS: [&str; 2] = ["berlin.soa", "berlin.books"]; +/// The slab and its sidecars. All three are required: `RowSlab` can read +/// positions without the books, but identity resolution needs them, and the +/// `.chains` geometry sidecar is what turns a clicked feature back into its +/// vertex chain (`/api/osm/geometry/:idx`). The bake publishes all three +/// atomically (the slab renames into place only after both sidecars exist), +/// so a prefix with a slab but no chains is a stale bake, not a valid state. +const ARTIFACTS: [&str; 3] = ["berlin.soa", "berlin.books", "berlin.chains"]; /// Where the hydrated copy lives, given the volume root. fn cache_dir(vol: &str) -> PathBuf { @@ -99,14 +102,14 @@ pub async fn ensure_slab_local() -> Option { } // Announce BEFORE the transfer, not after. This call blocks the listener - // bind, and a cold boot moves ~1.35 GB — so without a line here the boot + // bind, and a cold boot moves ~1.42 GB — so without a line here the boot // log is silent for 60-90s, which is indistinguishable from a hang for // whoever is watching a deploy. Naming the bucket and destination also // makes a misconfigured prefix obvious from the first line rather than // from a later "not readable" error. tracing::info!( %bucket, %prefix, dir = %dir.display(), - "osm slab: resolving from S3 (cold boot transfers ~1.35 GB and delays the listener; \ + "osm slab: resolving from S3 (cold boot transfers ~1.42 GB and delays the listener; \ a warm volume re-verifies in ~1s)" );