From c0e8a076bf80b72dcbe11bcf7997fa9d7ebbf1ff Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 09:35:03 +0000 Subject: [PATCH] osm basemap: split the green bucket, texture the ground cover, and make a region a config value MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three things, all found from the live deploy. 1. THE FLAT GREEN WASH. `class_for_tags` collapsed landuse|leisure|natural into one `Green` bucket, so `landuse=residential` — which covers most of a suburb — rendered as vegetation. A whole z16 viewport read as meadow. Split into `Meadow` (grass/farmland/heath), `Park` (leisure/cemetery) and `Built` (residential/industrial/commercial, grey and NOT green), with `Green` kept as the fallback so widening the specific set can never demote something to `Other`. Wire codes are APPENDED (7/8/9), never inserted: 0-6 stay a contract with the client's CLASS_ORDER, so an older client renders a new class as unknown rather than as the wrong one. One pre-existing assertion moved: `leisure=park` asserted `Green` because that WAS the collapsed behaviour. It is re-pinned to `Park` with a comment saying the behaviour changed deliberately — not a test edited for convenience. 2. MEADOWS HAD NO TEXTURE (operator). A flat fill cannot tell a meadow from a park from a lawn, and at z16 those are most of the viewport. Canvas gives this cheaply where SVG would not have: one 8x8 offscreen tile per texture, built once, handed to createPattern and reused for every polygon of that class — one pattern object per session, not per shape. Stipple for meadow, canopy strokes for wood. The pattern is deliberately NOT transform- corrected: it is created in canvas space and we translate() per tile, so the texture rides with the map instead of crawling under it while panning. 3. A REGION IS NOW CONFIG, NOT CODE. `ARTIFACTS` was hardcoded to berlin.* and the prefix to q2/bakes/berlin-v1, so a second bake needed a code change. `OSM_BAKE_REGION` now drives artifact names, S3 prefix and the volume path. The name is validated to [a-z0-9-]: it is interpolated into an S3 key AND joined onto a filesystem path, so `..` or `/` would traverse — an invalid value warns and falls back rather than building that path. Because the cache dir is region-keyed, two regions coexist on one volume and flipping between them costs no re-download. Also: the books header's `slab` digest — the same cross-bake pin `.chains` already enforces — was being DISCARDED (`Ok((_header, books))`). A codebook from a different bake was accepted silently, resolving every ordinal to some other bake's string. Now verified, and a magic mismatch is reported as an OUTDATED bake format naming expected vs found, instead of a generic "unreadable" after which every shape silently falls to `ShapeClass::Other`. Verified in headless Chromium against the shipped page, bake endpoints stubbed with the new classes: 2,698 distinct painted colours (was a flat wash), textures visible, 0 SVG nodes, 0 external requests, pan repaints, and the binary wire still 4.7x smaller than JSON on the same shapes. Docs: claude-notes/instructions/osm-bake-new-region.md — the BW runbook with measured sizing (645 MB PBF vs Berlin's 99 MB; ~9.1 GB output; budget 32 GB RAM, which is why BW cannot be baked in the dev container). 103 passed, 0 failed. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw --- .../instructions/osm-bake-new-region.md | 110 ++++++++++++++ crates/cockpit-server/src/osm.rs | 61 ++++++-- crates/cockpit-server/src/osm_features.rs | 132 ++++++++++++++++- crates/cockpit-server/src/osm_slab_hydrate.rs | 134 ++++++++++++++++-- 4 files changed, 417 insertions(+), 20 deletions(-) create mode 100644 claude-notes/instructions/osm-bake-new-region.md diff --git a/claude-notes/instructions/osm-bake-new-region.md b/claude-notes/instructions/osm-bake-new-region.md new file mode 100644 index 000000000..e564e1125 --- /dev/null +++ b/claude-notes/instructions/osm-bake-new-region.md @@ -0,0 +1,110 @@ +# Baking a new OSM region (worked example: Baden-Württemberg) + +Serving a second region is **config plus a bake**, not a code change. The +baker was already region-agnostic; as of this change the consumer is too +(`OSM_BAKE_REGION`). What follows is the whole procedure with measured +numbers, because the sizing is the part that bites. + +## What the region name controls + +One env var drives every name (`osm_slab_hydrate.rs`): + +| | value for `OSM_BAKE_REGION=baden-wuerttemberg` | +|---|---| +| S3 prefix | `q2/bakes/baden-wuerttemberg-v1` (override: `OSM_SLAB_S3_PREFIX`) | +| artifacts | `baden-wuerttemberg.{soa,books,chains}` + `SHA256SUMS` | +| volume cache | `$RAILWAY_VOL/osm/baden-wuerttemberg.soa` (+ sidecars) | + +The name is validated to `[a-z0-9-]`, ≤64 chars — it is interpolated into an +S3 key *and* joined onto a filesystem path, so `..` or `/` would traverse. +An invalid value logs a warning and falls back to `berlin` rather than +building that path. + +Because the cache dir is region-keyed, **two regions coexist on one volume**: +flipping `OSM_BAKE_REGION` and restarting switches maps, and the other +region's files stay warm (no re-download when you flip back). Size the volume +for the sum if you intend to keep both. + +## Sizing — measured, not estimated + +| | Berlin (shipped) | Baden-Württemberg (projected) | +|---|---|---| +| Geofabrik `.osm.pbf` | **99 MB** (measured) | **645 MB** (measured) | +| ratio to Berlin | 1× | **6.5×** | +| `.soa` slab | **1.35 GB** = 2.64 M rows × 512 B | **~8.7 GB** | +| `.chains` | 64 MB | ~0.4 GB | +| all three | ~1.42 GB | **~9.1 GB** | + +The slab is one 512-byte row per kept feature, so it scales with feature +count rather than with PBF bytes — the 6.5× is the best available proxy and +the projection is therefore **approximate**. Treat ~9 GB as the planning +figure and read the real size off the first bake. + +**Machine requirements for the bake step:** + +- **Disk ≥ 12 GB free** — 645 MB PBF + ~9.1 GB output + headroom for the + `.part` files the baker renames into place. +- **RAM: budget 32 GB.** `read::read_features_with_chains` reads *all* + features plus every way's vertex chain into memory before sorting + (`Vec` + the chain map), so peak RSS scales with the region. This is + the constraint that actually decides where you can run it — not disk. +- Runtime: Berlin bakes in minutes; expect roughly 6-7× that. + +**Railway volume ≥ 12 GB** for serving (the container only needs the output, +not the PBF). + +## Procedure + +```bash +# 1. The extract (Geofabrik publishes daily; ~645 MB) +curl -fSL -o bw.osm.pbf \ + https://download.geofabrik.de/europe/germany/baden-wuerttemberg-latest.osm.pbf + +# 2. Bake. Region-agnostic: PBF in, slab out. Sidecars land beside the slab. +cargo run --release -p osm-soa-bake --bin bake -- \ + bw.osm.pbf baden-wuerttemberg.soa + +# 3. Pin the bytes. The hydrator verifies every file against this on download +# AND on a warm cache hit — there is no unverified path. +sha256sum baden-wuerttemberg.soa \ + baden-wuerttemberg.books \ + baden-wuerttemberg.chains > SHA256SUMS + +# 4. Publish all four together, into the region's own prefix. +aws s3 cp --recursive . "s3://$AWS_S3_BUCKET_NAME/q2/bakes/baden-wuerttemberg-v1/" \ + --exclude '*' --include 'baden-wuerttemberg.*' --include 'SHA256SUMS' + +# 5. Serve it: set OSM_BAKE_REGION=baden-wuerttemberg on the service, redeploy. +``` + +Publish the sidecars **before or with** the slab, never after: the hydrator +requires all three, and a prefix holding a slab without its chains is a stale +bake rather than a valid state. + +## Verifying the switch + +The boot log names the region it resolved, so one line settles which bake is +live: + +``` +INFO osm slab: resolving from S3 region=baden-wuerttemberg bucket=… prefix=q2/bakes/baden-wuerttemberg-v1 dir=… +INFO osm slab: hydrated and verified path=/volume01/osm/baden-wuerttemberg.soa +``` + +Then, from the outside: + +- `GET /api/osm/geometry/tile-bin/12/2133/1420` (Stuttgart-ish) returns + `application/octet-stream` starting with the `OSM1` magic, not a 503. +- The cockpit's status line reports a shape/row count for a BW viewport. +- The `.chains` digest pin is enforced against the slab, so a mismatched pair + refuses loudly (`osm chains: sidecar pinned to a DIFFERENT slab`) instead + of drawing one bake's geometry against another's identities. + +## Why this was not baked in the dev container + +Recorded so the next session does not retry it: the container had **9.8 GB +free disk and 15 GB RAM**, against ~9.1 GB of output plus a 645 MB input and +a peak RSS that scales 6.5× off Berlin's — no headroom on either axis. More +decisively, `AWS_S3_BUCKET_NAME` is not set there, so the result could not be +published and would have died with the container. The bake belongs on a box +with the bucket credentials and real memory. diff --git a/crates/cockpit-server/src/osm.rs b/crates/cockpit-server/src/osm.rs index eda6d98a9..b73f7edb4 100644 --- a/crates/cockpit-server/src/osm.rs +++ b/crates/cockpit-server/src/osm.rs @@ -273,14 +273,57 @@ function shapeLayer(){ const CLASS_STYLE={ water: {fill:'#2b6cb088', stroke:'#4a7fa8', w:0.8}, building: {fill:'#39445580', stroke:'#5b6b7f', w:0.5}, - wood: {fill:'#1d4d2b88', stroke:'#316b41', w:0.5}, + wood: {fill:'#1d4d2b88', stroke:'#316b41', w:0.5, texture:'canopy'}, green: {fill:'#2a5a4055', stroke:'#3f7a58', w:0.5}, rail: {fill:'none', stroke:'#7a8598', w:0.9}, road: {fill:'none', stroke:'#6b6250', w:1.1}, other: {fill:'none', stroke:'#39424f', w:0.7}, + // Split out of `green` so a suburb stops reading as vegetation, and so the + // kinds that SHOULD look like ground cover can carry a texture. + meadow: {fill:'#2f5c3a4d', stroke:'#43764f', w:0.4, texture:'stipple'}, + park: {fill:'#27553866', stroke:'#3d7a54', w:0.5}, + built: {fill:'#2a303b66', stroke:'#3f4756', w:0.4}, }; function styleForClass(c){ return CLASS_STYLE[c] || CLASS_STYLE.other; } +// ── textures ───────────────────────────────────────────────────────────── +// A flat fill cannot tell a meadow from a park from a lawn, and at z16 those +// are most of the viewport — the operator's "meadows don't have a texture yet". +// Canvas gives this cheaply and SVG would not have: one small offscreen tile +// per texture, built once, handed to `createPattern` and reused for every +// polygon of that class. Cost is one pattern object per session, not per shape. +// +// The pattern is NOT transform-corrected on purpose: it is created in canvas +// space and we `translate()` per tile before filling, so the texture rides +// along with the map instead of crawling underneath it while panning. +const patternCache=new Map(); +function texture(ctx,name,style){ + let p=patternCache.get(name); + if(p!==undefined) return p; + const s=8, off=document.createElement('canvas'); + off.width=s; off.height=s; + const o=off.getContext('2d'); + o.fillStyle=style.fill; o.fillRect(0,0,s,s); + o.strokeStyle=style.stroke; o.globalAlpha=0.5; + if(name==='stipple'){ + // Sparse dots on the diagonal — grass, at any zoom, without banding. + o.fillStyle=style.stroke; + for(const [x,y] of [[1,1],[5,3],[3,6],[7,5]]) o.fillRect(x,y,1,1); + } else if(name==='canopy'){ + // Short crossing strokes read as tree cover at a distance. + o.lineWidth=0.7; o.beginPath(); + o.moveTo(0,6); o.lineTo(3,3); o.moveTo(4,8); o.lineTo(8,4); + o.stroke(); + } + p=ctx.createPattern(off,'repeat'); + patternCache.set(name,p); + return p; +} +function fillStyleFor(ctx,cls){ + const st=CLASS_STYLE[cls] || CLASS_STYLE.other; + return st.texture ? (texture(ctx,st.texture,st) || st.fill) : st.fill; +} + // World-pixel point list for an SVG element, at the current zoom. `offset` is // the repeated-world-copy shift the tile images already use. function svgPoints(points, offset){ @@ -333,11 +376,13 @@ let geomZoom=null; // paths are built in this zoom's pixel space // Wire codes, pinned to ShapeClass::wire_code on the server — the Rust test // `wire_codes_are_pinned` and this array must agree, and the array index IS // the wire byte. -const CLASS_ORDER=['water','building','wood','green','rail','road','other']; -// Areas back-to-front (a lake sits on its meadow, a building on its block), -// then lines with roads on top. -const FILL_PASS=[2,3,0,1]; // wood, green, water, building -const STROKE_PASS=[2,3,0,1,6,4,5]; // area outlines, then other, rail, road +const CLASS_ORDER=['water','building','wood','green','rail','road','other', + 'meadow','park','built']; +// Areas back-to-front: the broad ground cover first, then the specific kinds +// on top of it, then water, then buildings — a lake sits on its meadow, a +// building on its block. Lines last, roads on top of everything. +const FILL_PASS=[9,7,3,8,2,0,1]; // built, meadow, green, park, wood, water, building +const STROKE_PASS=[9,7,3,8,2,0,1,6,4,5];// area outlines in the same order, then other, rail, road function parseTileBin(buf){ const dv=new DataView(buf); @@ -431,11 +476,11 @@ function drawBase(){ ctx.translate(tx*256+ox, ty*256+oy); for(const c of FILL_PASS){ const p=d.fills.get(c); if(!p) continue; - ctx.fillStyle=CLASS_STYLE[CLASS_ORDER[c]].fill; ctx.fill(p); + ctx.fillStyle=fillStyleFor(ctx,CLASS_ORDER[c]); ctx.fill(p); } for(const c of STROKE_PASS){ const p=d.strokes.get(c); if(!p) continue; - const st=CLASS_STYLE[CLASS_ORDER[c]]; + const st=styleForClass(CLASS_ORDER[c]); // unknown byte -> `other`, never undefined ctx.strokeStyle=st.stroke; ctx.lineWidth=st.w; ctx.stroke(p); } ctx.restore(); diff --git a/crates/cockpit-server/src/osm_features.rs b/crates/cockpit-server/src/osm_features.rs index 3ccce7cd7..894ba22c9 100644 --- a/crates/cockpit-server/src/osm_features.rs +++ b/crates/cockpit-server/src/osm_features.rs @@ -440,10 +440,42 @@ fn open_books() -> Option<&'static osm_soa_bake::codebook::Books> { let file = std::fs::File::open(&path).ok()?; let mut r = std::io::BufReader::new(file); match osm_soa_bake::codebook::read_books(&mut r) { - Ok((_header, books)) => { - tracing::info!(path = %path.display(), "osm books: codebook loaded"); + Ok((header, books)) => { + // The header's `slab` is a `hash_slab` of the rows the + // codebook was built from — the SAME cross-bake pin + // `.chains` enforces, and it was being discarded here. + // Accepting a foreign codebook is worse than rejecting it: + // every ordinal resolves to some OTHER bake's string, so + // the map answers confidently and wrongly. + let want = slab_digest()?; + if header.slab != want { + tracing::error!( + got = format_args!("{:016x}", header.slab), + want = format_args!("{want:016x}"), + path = %path.display(), + "osm books: codebook pinned to a DIFFERENT slab — re-bake this \ + region so slab and sidecars come from one run; tags unavailable" + ); + return None; + } + tracing::info!(path = %path.display(), rows = header.rows, "osm books: codebook loaded"); Some(books) } + // A magic mismatch is not "corrupt" — it is an OUTDATED (or + // foreign) bake, and saying so is the difference between a + // one-line fix and diagnosing a grey map by eye. Everything + // downstream falls back to `ShapeClass::Other` without tags, + // so the symptom is a map that draws but says nothing. + Err(osm_soa_bake::codebook::BookError::Magic(found)) => { + tracing::error!( + path = %path.display(), + found = %String::from_utf8_lossy(&found).escape_debug().to_string(), + expected = %String::from_utf8_lossy(&osm_soa_bake::codebook::MAGIC).escape_debug().to_string(), + "osm books: OUTDATED bake format — this build does not speak that \ + codebook version; re-bake the region with the current baker" + ); + None + } Err(e) => { tracing::warn!(path = %path.display(), error = ?e, "osm books: unreadable; tags unavailable"); None @@ -572,6 +604,15 @@ fn open_chains() -> Option<&'static osm_soa_bake::chains::Chains> { tracing::info!(path = %path.display(), ways = ch.len(), "osm chains: loaded"); Some(ch) } + Err(osm_soa_bake::chains::ChainError::BadMagic) => { + tracing::error!( + path = %path.display(), + expected = %String::from_utf8_lossy(&osm_soa_bake::chains::MAGIC).escape_debug().to_string(), + "osm chains: OUTDATED bake format — this build does not speak that \ + chains layout; re-bake the region with the current baker" + ); + None + } Err(e) => { tracing::warn!(path = %path.display(), error = ?e, "osm chains: unreadable; geometry unavailable"); None @@ -686,10 +727,26 @@ pub enum ShapeClass { Water, Building, Wood, + /// Whatever carries a green-ish tag but is not one of the specific kinds + /// below. Kept as the FALLBACK so widening the specific set never demotes + /// something to `Other`. Green, Rail, Road, Other, + // Appended, never inserted: 0-6 are a wire contract with the client's + // CLASS_ORDER array (see `wire_code`). + /// Grass, meadow, farmland, heath — the textured kinds. Split out because + /// they are the majority of a rural or suburban viewport and reading them + /// as one flat wash with everything else is what made a whole z16 view + /// uniformly green. + Meadow, + /// Managed green: park, garden, pitch, cemetery. + Park, + /// `landuse=residential` and its industrial/commercial siblings. NOT + /// green — this is the tag that covers most of a suburb, and colouring it + /// as vegetation is why Zehlendorf rendered as a meadow. + Built, } impl ShapeClass { @@ -706,6 +763,9 @@ impl ShapeClass { ShapeClass::Rail => 4, ShapeClass::Road => 5, ShapeClass::Other => 6, + ShapeClass::Meadow => 7, + ShapeClass::Park => 8, + ShapeClass::Built => 9, } } } @@ -743,6 +803,32 @@ where if natural == Some("wood") || landuse == Some("forest") { return ShapeClass::Wood; } + // The three specific kinds, ahead of the generic green fallback. Order + // among THEM does not matter (the tag sets are disjoint); order relative + // to the fallback is everything. + if matches!( + landuse, + Some("grass" | "meadow" | "farmland" | "farmyard" | "orchard" | "vineyard" + | "allotments" | "village_green" | "greenfield" | "flowerbed") + ) || matches!(natural, Some("grassland" | "heath" | "scrub" | "moor" | "fell")) + { + return ShapeClass::Meadow; + } + if matches!( + leisure, + Some("park" | "garden" | "pitch" | "playground" | "recreation_ground" + | "golf_course" | "dog_park" | "common") + ) || matches!(landuse, Some("cemetery" | "grave_yard" | "recreation_ground")) + { + return ShapeClass::Park; + } + if matches!( + landuse, + Some("residential" | "industrial" | "commercial" | "retail" | "construction" + | "garages" | "railway" | "quarry" | "brownfield" | "military") + ) { + return ShapeClass::Built; + } if landuse.is_some() || leisure.is_some() || natural.is_some() { return ShapeClass::Green; } @@ -1211,7 +1297,11 @@ mod tests { assert_eq!(class_for_tags([("building", "yes")]), ShapeClass::Building); assert_eq!(class_for_tags([("natural", "wood")]), ShapeClass::Wood); assert_eq!(class_for_tags([("landuse", "forest")]), ShapeClass::Wood); - assert_eq!(class_for_tags([("leisure", "park")]), ShapeClass::Green); + // Re-pinned: this asserted `Green` when every green-ish tag collapsed + // into one bucket. A park is now its own class — the assertion moved + // because the BEHAVIOUR deliberately changed, and the split is what + // `specific_land_kinds_beat_the_generic_green_fallback` covers. + assert_eq!(class_for_tags([("leisure", "park")]), ShapeClass::Park); assert_eq!(class_for_tags([("railway", "rail")]), ShapeClass::Rail); assert_eq!(class_for_tags([("highway", "primary")]), ShapeClass::Road); @@ -1336,6 +1426,42 @@ mod tests { assert_eq!(ShapeClass::Rail.wire_code(), 4); assert_eq!(ShapeClass::Road.wire_code(), 5); assert_eq!(ShapeClass::Other.wire_code(), 6); + // Appended after the first shipped wire. A client built against the + // 7-class table renders these as unknown, never as the WRONG class — + // which is only true while 0-6 keep their meaning. + assert_eq!(ShapeClass::Meadow.wire_code(), 7); + assert_eq!(ShapeClass::Park.wire_code(), 8); + assert_eq!(ShapeClass::Built.wire_code(), 9); + } + + /// The flat-green-wash defect, pinned. A whole suburban z16 viewport read + /// as meadow because `landuse=residential` fell into the generic `Green` + /// fallback along with actual vegetation. Two-sided: the specific kinds + /// resolve to themselves AND residential is provably not green. + #[test] + fn specific_land_kinds_beat_the_generic_green_fallback() { + assert_eq!(class_for_tags([("landuse", "meadow")]), ShapeClass::Meadow); + assert_eq!(class_for_tags([("landuse", "farmland")]), ShapeClass::Meadow); + assert_eq!(class_for_tags([("natural", "heath")]), ShapeClass::Meadow); + assert_eq!(class_for_tags([("leisure", "park")]), ShapeClass::Park); + assert_eq!(class_for_tags([("landuse", "cemetery")]), ShapeClass::Park); + + let residential = class_for_tags([("landuse", "residential")]); + assert_eq!(residential, ShapeClass::Built); + assert_ne!( + residential, + ShapeClass::Green, + "a suburb is not vegetation — this is the whole defect" + ); + + // The specific kinds must not have eaten the ones that already worked. + assert_eq!(class_for_tags([("natural", "wood")]), ShapeClass::Wood); + assert_eq!(class_for_tags([("natural", "water")]), ShapeClass::Water); + assert_eq!(class_for_tags([("building", "yes")]), ShapeClass::Building); + // …and the fallback still catches a green-ish tag none of them name, + // so widening the specific set can never demote one to `Other`. + assert_eq!(class_for_tags([("landuse", "some_new_tag")]), ShapeClass::Green); + assert_eq!(class_for_tags([("leisure", "marina")]), ShapeClass::Green); } /// Decode `OSM1` bytes back into shapes — an independent reader for the diff --git a/crates/cockpit-server/src/osm_slab_hydrate.rs b/crates/cockpit-server/src/osm_slab_hydrate.rs index 5ffc961a8..a07887b77 100644 --- a/crates/cockpit-server/src/osm_slab_hydrate.rs +++ b/crates/cockpit-server/src/osm_slab_hydrate.rs @@ -72,8 +72,50 @@ use object_store::aws::AmazonS3Builder; use object_store::{ObjectStore, ObjectStoreExt}; use sha2::{Digest, Sha256}; -/// Default S3 prefix holding the bake. Overridable with `OSM_SLAB_S3_PREFIX`. -const DEFAULT_PREFIX: &str = "q2/bakes/berlin-v1"; +/// The region baked by default. `OSM_BAKE_REGION` selects another one. +/// +/// The region is the ONLY thing that differs between bakes: the baker +/// (`osm-soa-bake`'s `bake `) is already region-agnostic, +/// and the sidecars are found by extension from the slab's own stem — so +/// serving Baden-Württemberg instead of Berlin is this string plus a bake in +/// the bucket, not a code change. +const DEFAULT_REGION: &str = "berlin"; + +/// The region name, validated. +/// +/// **The validation is not decoration.** This string is interpolated into an +/// S3 object key AND joined onto a filesystem path, so `..`, `/`, or a +/// backslash would let a stray environment variable read a different prefix or +/// write outside the cache dir. Restricting it to `[a-z0-9-]` makes both uses +/// safe by construction rather than by careful escaping at each site. A +/// rejected value falls back to the default and says so. +fn bake_region() -> String { + match env_var_nonempty("OSM_BAKE_REGION") { + None => DEFAULT_REGION.to_string(), + Some(r) if is_valid_region(&r) => r, + Some(bad) => { + tracing::warn!( + rejected = %bad, using = DEFAULT_REGION, + "osm slab: OSM_BAKE_REGION must be lowercase [a-z0-9-]; ignoring it" + ); + DEFAULT_REGION.to_string() + } + } +} + +fn is_valid_region(r: &str) -> bool { + !r.is_empty() + && r.len() <= 64 + && r.bytes() + .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-') +} + +/// Default S3 prefix holding the bake. Overridable with `OSM_SLAB_S3_PREFIX`, +/// which wins outright — that is the escape hatch for a layout this naming +/// convention does not cover. +fn default_prefix(region: &str) -> String { + format!("q2/bakes/{region}-v1") +} /// The slab and its sidecars. All three are required: `RowSlab` can read /// positions without the books, but identity resolution needs them, and the @@ -81,7 +123,16 @@ const DEFAULT_PREFIX: &str = "q2/bakes/berlin-v1"; /// vertex chain (`/api/osm/geometry/:idx`). The bake publishes all three /// atomically (the slab renames into place only after both sidecars exist), /// so a prefix with a slab but no chains is a stale bake, not a valid state. -const ARTIFACTS: [&str; 3] = ["berlin.soa", "berlin.books", "berlin.chains"]; +/// +/// The slab is FIRST and stays first: [`ensure_slab_local`] returns +/// `artifacts[0]` as the path to mmap. +fn artifacts(region: &str) -> [String; 3] { + [ + format!("{region}.soa"), + format!("{region}.books"), + format!("{region}.chains"), + ] +} /// Where the hydrated copy lives, given the volume root. fn cache_dir(vol: &str) -> PathBuf { @@ -155,7 +206,10 @@ pub async fn ensure_slab_local() -> Option { } let bucket = bucket_env.expect("checked non-empty above"); let vol = vol_env.expect("checked non-empty above"); - let prefix = env_var_nonempty("OSM_SLAB_S3_PREFIX").unwrap_or_else(|| DEFAULT_PREFIX.to_string()); + let region = bake_region(); + let artifacts = artifacts(®ion); + let prefix = + env_var_nonempty("OSM_SLAB_S3_PREFIX").unwrap_or_else(|| default_prefix(®ion)); let dir = cache_dir(&vol); if let Err(e) = std::fs::create_dir_all(&dir) { @@ -170,9 +224,9 @@ pub async fn ensure_slab_local() -> Option { // makes a misconfigured prefix obvious from the first line rather than // from a later "not readable" error. tracing::info!( - %bucket, %prefix, dir = %dir.display(), - "osm slab: resolving from S3 (cold boot transfers ~1.42 GB and delays the listener; \ - a warm volume re-verifies in ~1s)" + %region, %bucket, %prefix, dir = %dir.display(), + "osm slab: resolving from S3 (a cold boot transfers the whole bake and delays the \ + listener — Berlin is ~1.42 GB; a warm volume re-verifies in ~1s)" ); // `from_env()` reads AWS_ENDPOINT_URL / AWS_ACCESS_KEY_ID / @@ -195,7 +249,7 @@ pub async fn ensure_slab_local() -> Option { None => return None, }; - for name in ARTIFACTS { + for name in artifacts.iter().map(String::as_str) { let want = match sums.iter().find(|(k, _)| k == name).map(|(_, h)| h.clone()) { Some(h) => h, None => { @@ -227,7 +281,7 @@ pub async fn ensure_slab_local() -> Option { } } - let slab = dir.join(ARTIFACTS[0]); + let slab = dir.join(&artifacts[0]); tracing::info!(path = %slab.display(), "osm slab: hydrated and verified"); Some(slab) } @@ -445,4 +499,66 @@ not-a-hash junk.txt fn missing_inputs_is_empty_when_both_are_present() { assert!(missing_inputs(Some("my-bucket"), Some("/volume01")).is_empty()); } + + /// Every artifact name and the prefix derive from ONE region string, so a + /// second bake is config. The slab must stay at index 0 — + /// `ensure_slab_local` returns `artifacts[0]` as the path to mmap, so a + /// reorder would hand the books file to `RowSlab`. + #[test] + fn artifacts_and_prefix_follow_the_region() { + assert_eq!( + artifacts("berlin"), + ["berlin.soa", "berlin.books", "berlin.chains"], + "the default region must reproduce the names the Berlin bake already \ + published — this is a wire/bucket contract, not a naming preference" + ); + assert_eq!( + artifacts("baden-wuerttemberg"), + [ + "baden-wuerttemberg.soa", + "baden-wuerttemberg.books", + "baden-wuerttemberg.chains" + ] + ); + assert!(artifacts("bw")[0].ends_with(".soa"), "slab stays at index 0"); + assert_eq!(default_prefix("berlin"), "q2/bakes/berlin-v1"); + assert_eq!( + default_prefix("baden-wuerttemberg"), + "q2/bakes/baden-wuerttemberg-v1" + ); + } + + /// **The guard that matters.** The region is interpolated into an S3 key + /// AND joined onto a filesystem path, so anything that can traverse must + /// be rejected before it reaches either. Two-sided: the shapes a real + /// Geofabrik region name takes are all accepted. + #[test] + fn region_validation_rejects_traversal_and_accepts_real_names() { + for good in [ + "berlin", + "baden-wuerttemberg", + "nordrhein-westfalen", + "bw2", + ] { + assert!(is_valid_region(good), "{good} is a legitimate region name"); + } + for bad in [ + "../secrets", // parent escape + "a/b", // sub-prefix + "a\\b", // Windows separator + "Berlin", // uppercase: S3 keys are case-sensitive + "baden_wuerttemberg", // underscore is not in the allowed set + "baden würt", // non-ASCII + space + "", // empty + ] { + assert!(!is_valid_region(bad), "{bad:?} must be rejected"); + } + // A path built from a rejected name would have escaped the cache dir — + // proving the guard is load-bearing rather than cosmetic. + assert_eq!( + cache_dir("/volume01").join(format!("{}.soa", "../..")), + PathBuf::from("/volume01/osm/../...soa"), + "documents WHY the guard exists: this is the path we refuse to build" + ); + } }