Skip to content

osm basemap: split the green bucket, texture the ground cover, region as config #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AdaWorldAPI merged 1 commit into from
Aug 12, 2026
Merged

osm basemap: split the green bucket, texture the ground cover, region as config #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions claude-notes/instructions/osm-bake-new-region.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Baking a new OSM region (worked example: Baden-Württemberg)

Serving a second region is **config plus a bake**, not a code change. The
baker was already region-agnostic; as of this change the consumer is too
(`OSM_BAKE_REGION`). What follows is the whole procedure with measured
numbers, because the sizing is the part that bites.

## What the region name controls

One env var drives every name (`osm_slab_hydrate.rs`):

| | value for `OSM_BAKE_REGION=baden-wuerttemberg` |
|---|---|
| S3 prefix | `q2/bakes/baden-wuerttemberg-v1` (override: `OSM_SLAB_S3_PREFIX`) |
| artifacts | `baden-wuerttemberg.{soa,books,chains}` + `SHA256SUMS` |
| volume cache | `$RAILWAY_VOL/osm/baden-wuerttemberg.soa` (+ sidecars) |

The name is validated to `[a-z0-9-]`, ≤64 chars — it is interpolated into an
S3 key *and* joined onto a filesystem path, so `..` or `/` would traverse.
An invalid value logs a warning and falls back to `berlin` rather than
building that path.

Because the cache dir is region-keyed, **two regions coexist on one volume**:
flipping `OSM_BAKE_REGION` and restarting switches maps, and the other
region's files stay warm (no re-download when you flip back). Size the volume
for the sum if you intend to keep both.

## Sizing — measured, not estimated

| | Berlin (shipped) | Baden-Württemberg (projected) |
|---|---|---|
| Geofabrik `.osm.pbf` | **99 MB** (measured) | **645 MB** (measured) |
| ratio to Berlin | 1× | **6.5×** |
| `.soa` slab | **1.35 GB** = 2.64 M rows × 512 B | **~8.7 GB** |
| `.chains` | 64 MB | ~0.4 GB |
| all three | ~1.42 GB | **~9.1 GB** |

The slab is one 512-byte row per kept feature, so it scales with feature
count rather than with PBF bytes — the 6.5× is the best available proxy and
the projection is therefore **approximate**. Treat ~9 GB as the planning
figure and read the real size off the first bake.

**Machine requirements for the bake step:**

- **Disk ≥ 12 GB free** — 645 MB PBF + ~9.1 GB output + headroom for the
`.part` files the baker renames into place.
- **RAM: budget 32 GB.** `read::read_features_with_chains` reads *all*
features plus every way's vertex chain into memory before sorting
(`Vec<Keyed>` + the chain map), so peak RSS scales with the region. This is
the constraint that actually decides where you can run it — not disk.
- Runtime: Berlin bakes in minutes; expect roughly 6-7× that.

**Railway volume ≥ 12 GB** for serving (the container only needs the output,
not the PBF).

## Procedure

```bash
# 1. The extract (Geofabrik publishes daily; ~645 MB)
curl -fSL -o bw.osm.pbf \
https://download.geofabrik.de/europe/germany/baden-wuerttemberg-latest.osm.pbf

# 2. Bake. Region-agnostic: PBF in, slab out. Sidecars land beside the slab.
cargo run --release -p osm-soa-bake --bin bake -- \
bw.osm.pbf baden-wuerttemberg.soa

# 3. Pin the bytes. The hydrator verifies every file against this on download
# AND on a warm cache hit — there is no unverified path.
sha256sum baden-wuerttemberg.soa \
baden-wuerttemberg.books \
baden-wuerttemberg.chains > SHA256SUMS

# 4. Publish all four together, into the region's own prefix.
aws s3 cp --recursive . "s3://$AWS_S3_BUCKET_NAME/q2/bakes/baden-wuerttemberg-v1/" \
--exclude '*' --include 'baden-wuerttemberg.*' --include 'SHA256SUMS'

# 5. Serve it: set OSM_BAKE_REGION=baden-wuerttemberg on the service, redeploy.
```

Publish the sidecars **before or with** the slab, never after: the hydrator
requires all three, and a prefix holding a slab without its chains is a stale
bake rather than a valid state.

## Verifying the switch

The boot log names the region it resolved, so one line settles which bake is
live:

```
INFO osm slab: resolving from S3 region=baden-wuerttemberg bucket=… prefix=q2/bakes/baden-wuerttemberg-v1 dir=…
INFO osm slab: hydrated and verified path=/volume01/osm/baden-wuerttemberg.soa
```

Then, from the outside:

- `GET /api/osm/geometry/tile-bin/12/2133/1420` (Stuttgart-ish) returns
`application/octet-stream` starting with the `OSM1` magic, not a 503.
- The cockpit's status line reports a shape/row count for a BW viewport.
- The `.chains` digest pin is enforced against the slab, so a mismatched pair
refuses loudly (`osm chains: sidecar pinned to a DIFFERENT slab`) instead
of drawing one bake's geometry against another's identities.

## Why this was not baked in the dev container

Recorded so the next session does not retry it: the container had **9.8 GB
free disk and 15 GB RAM**, against ~9.1 GB of output plus a 645 MB input and
a peak RSS that scales 6.5× off Berlin's — no headroom on either axis. More
decisively, `AWS_S3_BUCKET_NAME` is not set there, so the result could not be
published and would have died with the container. The bake belongs on a box
with the bucket credentials and real memory.
61 changes: 53 additions & 8 deletions crates/cockpit-server/src/osm.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,14 +273,57 @@ function shapeLayer(){
const CLASS_STYLE={
water: {fill:'#2b6cb088', stroke:'#4a7fa8', w:0.8},
building: {fill:'#39445580', stroke:'#5b6b7f', w:0.5},
wood: {fill:'#1d4d2b88', stroke:'#316b41', w:0.5},
wood: {fill:'#1d4d2b88', stroke:'#316b41', w:0.5, texture:'canopy'},
green: {fill:'#2a5a4055', stroke:'#3f7a58', w:0.5},
rail: {fill:'none', stroke:'#7a8598', w:0.9},
road: {fill:'none', stroke:'#6b6250', w:1.1},
other: {fill:'none', stroke:'#39424f', w:0.7},
// Split out of `green` so a suburb stops reading as vegetation, and so the
// kinds that SHOULD look like ground cover can carry a texture.
meadow: {fill:'#2f5c3a4d', stroke:'#43764f', w:0.4, texture:'stipple'},
park: {fill:'#27553866', stroke:'#3d7a54', w:0.5},
built: {fill:'#2a303b66', stroke:'#3f4756', w:0.4},
};
function styleForClass(c){ return CLASS_STYLE[c] || CLASS_STYLE.other; }

// ── textures ─────────────────────────────────────────────────────────────
// A flat fill cannot tell a meadow from a park from a lawn, and at z16 those
// are most of the viewport — the operator's "meadows don't have a texture yet".
// Canvas gives this cheaply and SVG would not have: one small offscreen tile
// per texture, built once, handed to `createPattern` and reused for every
// polygon of that class. Cost is one pattern object per session, not per shape.
//
// The pattern is NOT transform-corrected on purpose: it is created in canvas
// space and we `translate()` per tile before filling, so the texture rides
// along with the map instead of crawling underneath it while panning.
const patternCache=new Map();
function texture(ctx,name,style){
let p=patternCache.get(name);
if(p!==undefined) return p;
const s=8, off=document.createElement('canvas');
off.width=s; off.height=s;
const o=off.getContext('2d');
o.fillStyle=style.fill; o.fillRect(0,0,s,s);
o.strokeStyle=style.stroke; o.globalAlpha=0.5;
if(name==='stipple'){
// Sparse dots on the diagonal — grass, at any zoom, without banding.
o.fillStyle=style.stroke;
for(const [x,y] of [[1,1],[5,3],[3,6],[7,5]]) o.fillRect(x,y,1,1);
} else if(name==='canopy'){
// Short crossing strokes read as tree cover at a distance.
o.lineWidth=0.7; o.beginPath();
o.moveTo(0,6); o.lineTo(3,3); o.moveTo(4,8); o.lineTo(8,4);
o.stroke();
}
p=ctx.createPattern(off,'repeat');
patternCache.set(name,p);
return p;
}
function fillStyleFor(ctx,cls){
const st=CLASS_STYLE[cls] || CLASS_STYLE.other;
return st.texture ? (texture(ctx,st.texture,st) || st.fill) : st.fill;
}

// World-pixel point list for an SVG element, at the current zoom. `offset` is
// the repeated-world-copy shift the tile images already use.
function svgPoints(points, offset){
Expand Down Expand Up @@ -333,11 +376,13 @@ let geomZoom=null; // paths are built in this zoom's pixel space
// Wire codes, pinned to ShapeClass::wire_code on the server — the Rust test
// `wire_codes_are_pinned` and this array must agree, and the array index IS
// the wire byte.
const CLASS_ORDER=['water','building','wood','green','rail','road','other'];
// Areas back-to-front (a lake sits on its meadow, a building on its block),
// then lines with roads on top.
const FILL_PASS=[2,3,0,1]; // wood, green, water, building
const STROKE_PASS=[2,3,0,1,6,4,5]; // area outlines, then other, rail, road
const CLASS_ORDER=['water','building','wood','green','rail','road','other',
'meadow','park','built'];
// Areas back-to-front: the broad ground cover first, then the specific kinds
// on top of it, then water, then buildings — a lake sits on its meadow, a
// building on its block. Lines last, roads on top of everything.
const FILL_PASS=[9,7,3,8,2,0,1]; // built, meadow, green, park, wood, water, building
const STROKE_PASS=[9,7,3,8,2,0,1,6,4,5];// area outlines in the same order, then other, rail, road

function parseTileBin(buf){
const dv=new DataView(buf);
Expand Down Expand Up @@ -431,11 +476,11 @@ function drawBase(){
ctx.translate(tx*256+ox, ty*256+oy);
for(const c of FILL_PASS){
const p=d.fills.get(c); if(!p) continue;
ctx.fillStyle=CLASS_STYLE[CLASS_ORDER[c]].fill; ctx.fill(p);
ctx.fillStyle=fillStyleFor(ctx,CLASS_ORDER[c]); ctx.fill(p);
}
for(const c of STROKE_PASS){
const p=d.strokes.get(c); if(!p) continue;
const st=CLASS_STYLE[CLASS_ORDER[c]];
const st=styleForClass(CLASS_ORDER[c]); // unknown byte -> `other`, never undefined
ctx.strokeStyle=st.stroke; ctx.lineWidth=st.w; ctx.stroke(p);
}
ctx.restore();
Expand Down
132 changes: 129 additions & 3 deletions crates/cockpit-server/src/osm_features.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -440,10 +440,42 @@ fn open_books() -> Option<&'static osm_soa_bake::codebook::Books> {
let file = std::fs::File::open(&path).ok()?;
let mut r = std::io::BufReader::new(file);
match osm_soa_bake::codebook::read_books(&mut r) {
Ok((_header, books)) => {
tracing::info!(path = %path.display(), "osm books: codebook loaded");
Ok((header, books)) => {
// The header's `slab` is a `hash_slab` of the rows the
// codebook was built from — the SAME cross-bake pin
// `.chains` enforces, and it was being discarded here.
// Accepting a foreign codebook is worse than rejecting it:
// every ordinal resolves to some OTHER bake's string, so
// the map answers confidently and wrongly.
let want = slab_digest()?;
if header.slab != want {
tracing::error!(
got = format_args!("{:016x}", header.slab),
want = format_args!("{want:016x}"),
path = %path.display(),
"osm books: codebook pinned to a DIFFERENT slab — re-bake this \
region so slab and sidecars come from one run; tags unavailable"
);
return None;
}
tracing::info!(path = %path.display(), rows = header.rows, "osm books: codebook loaded");
Some(books)
}
// A magic mismatch is not "corrupt" — it is an OUTDATED (or
// foreign) bake, and saying so is the difference between a
// one-line fix and diagnosing a grey map by eye. Everything
// downstream falls back to `ShapeClass::Other` without tags,
// so the symptom is a map that draws but says nothing.
Err(osm_soa_bake::codebook::BookError::Magic(found)) => {
tracing::error!(
path = %path.display(),
found = %String::from_utf8_lossy(&found).escape_debug().to_string(),
expected = %String::from_utf8_lossy(&osm_soa_bake::codebook::MAGIC).escape_debug().to_string(),
"osm books: OUTDATED bake format — this build does not speak that \
codebook version; re-bake the region with the current baker"
);
None
}
Err(e) => {
tracing::warn!(path = %path.display(), error = ?e, "osm books: unreadable; tags unavailable");
None
Expand Down Expand Up @@ -572,6 +604,15 @@ fn open_chains() -> Option<&'static osm_soa_bake::chains::Chains> {
tracing::info!(path = %path.display(), ways = ch.len(), "osm chains: loaded");
Some(ch)
}
Err(osm_soa_bake::chains::ChainError::BadMagic) => {
tracing::error!(
path = %path.display(),
expected = %String::from_utf8_lossy(&osm_soa_bake::chains::MAGIC).escape_debug().to_string(),
"osm chains: OUTDATED bake format — this build does not speak that \
chains layout; re-bake the region with the current baker"
);
None
}
Err(e) => {
tracing::warn!(path = %path.display(), error = ?e, "osm chains: unreadable; geometry unavailable");
None
Expand Down Expand Up @@ -686,10 +727,26 @@ pub enum ShapeClass {
Water,
Building,
Wood,
/// Whatever carries a green-ish tag but is not one of the specific kinds
/// below. Kept as the FALLBACK so widening the specific set never demotes
/// something to `Other`.
Green,
Rail,
Road,
Other,
// Appended, never inserted: 0-6 are a wire contract with the client's
// CLASS_ORDER array (see `wire_code`).
/// Grass, meadow, farmland, heath — the textured kinds. Split out because
/// they are the majority of a rural or suburban viewport and reading them
/// as one flat wash with everything else is what made a whole z16 view
/// uniformly green.
Meadow,
/// Managed green: park, garden, pitch, cemetery.
Park,
/// `landuse=residential` and its industrial/commercial siblings. NOT
/// green — this is the tag that covers most of a suburb, and colouring it
/// as vegetation is why Zehlendorf rendered as a meadow.
Built,
}

impl ShapeClass {
Expand All @@ -706,6 +763,9 @@ impl ShapeClass {
ShapeClass::Rail => 4,
ShapeClass::Road => 5,
ShapeClass::Other => 6,
ShapeClass::Meadow => 7,
ShapeClass::Park => 8,
ShapeClass::Built => 9,
}
}
}
Expand Down Expand Up @@ -743,6 +803,32 @@ where
if natural == Some("wood") || landuse == Some("forest") {
return ShapeClass::Wood;
}
// The three specific kinds, ahead of the generic green fallback. Order
// among THEM does not matter (the tag sets are disjoint); order relative
// to the fallback is everything.
if matches!(
landuse,
Some("grass" | "meadow" | "farmland" | "farmyard" | "orchard" | "vineyard"
| "allotments" | "village_green" | "greenfield" | "flowerbed")
) || matches!(natural, Some("grassland" | "heath" | "scrub" | "moor" | "fell"))
{
return ShapeClass::Meadow;
}
if matches!(
leisure,
Some("park" | "garden" | "pitch" | "playground" | "recreation_ground"
| "golf_course" | "dog_park" | "common")
) || matches!(landuse, Some("cemetery" | "grave_yard" | "recreation_ground"))
{
return ShapeClass::Park;
}
if matches!(
landuse,
Some("residential" | "industrial" | "commercial" | "retail" | "construction"
| "garages" | "railway" | "quarry" | "brownfield" | "military")
) {
return ShapeClass::Built;
}
if landuse.is_some() || leisure.is_some() || natural.is_some() {
return ShapeClass::Green;
}
Expand Down Expand Up @@ -1211,7 +1297,11 @@ mod tests {
assert_eq!(class_for_tags([("building", "yes")]), ShapeClass::Building);
assert_eq!(class_for_tags([("natural", "wood")]), ShapeClass::Wood);
assert_eq!(class_for_tags([("landuse", "forest")]), ShapeClass::Wood);
assert_eq!(class_for_tags([("leisure", "park")]), ShapeClass::Green);
// Re-pinned: this asserted `Green` when every green-ish tag collapsed
// into one bucket. A park is now its own class — the assertion moved
// because the BEHAVIOUR deliberately changed, and the split is what
// `specific_land_kinds_beat_the_generic_green_fallback` covers.
assert_eq!(class_for_tags([("leisure", "park")]), ShapeClass::Park);
assert_eq!(class_for_tags([("railway", "rail")]), ShapeClass::Rail);
assert_eq!(class_for_tags([("highway", "primary")]), ShapeClass::Road);

Expand Down Expand Up @@ -1336,6 +1426,42 @@ mod tests {
assert_eq!(ShapeClass::Rail.wire_code(), 4);
assert_eq!(ShapeClass::Road.wire_code(), 5);
assert_eq!(ShapeClass::Other.wire_code(), 6);
// Appended after the first shipped wire. A client built against the
// 7-class table renders these as unknown, never as the WRONG class —
// which is only true while 0-6 keep their meaning.
assert_eq!(ShapeClass::Meadow.wire_code(), 7);
assert_eq!(ShapeClass::Park.wire_code(), 8);
assert_eq!(ShapeClass::Built.wire_code(), 9);
}

/// The flat-green-wash defect, pinned. A whole suburban z16 viewport read
/// as meadow because `landuse=residential` fell into the generic `Green`
/// fallback along with actual vegetation. Two-sided: the specific kinds
/// resolve to themselves AND residential is provably not green.
#[test]
fn specific_land_kinds_beat_the_generic_green_fallback() {
assert_eq!(class_for_tags([("landuse", "meadow")]), ShapeClass::Meadow);
assert_eq!(class_for_tags([("landuse", "farmland")]), ShapeClass::Meadow);
assert_eq!(class_for_tags([("natural", "heath")]), ShapeClass::Meadow);
assert_eq!(class_for_tags([("leisure", "park")]), ShapeClass::Park);
assert_eq!(class_for_tags([("landuse", "cemetery")]), ShapeClass::Park);

let residential = class_for_tags([("landuse", "residential")]);
assert_eq!(residential, ShapeClass::Built);
assert_ne!(
residential,
ShapeClass::Green,
"a suburb is not vegetation — this is the whole defect"
);

// The specific kinds must not have eaten the ones that already worked.
assert_eq!(class_for_tags([("natural", "wood")]), ShapeClass::Wood);
assert_eq!(class_for_tags([("natural", "water")]), ShapeClass::Water);
assert_eq!(class_for_tags([("building", "yes")]), ShapeClass::Building);
// …and the fallback still catches a green-ish tag none of them name,
// so widening the specific set can never demote one to `Other`.
assert_eq!(class_for_tags([("landuse", "some_new_tag")]), ShapeClass::Green);
assert_eq!(class_for_tags([("leisure", "marina")]), ShapeClass::Green);
}

/// Decode `OSM1` bytes back into shapes — an independent reader for the
Expand Down
Loading