cockpit-server: convert .chains/.books sidecars to Lance datasets at boot - #138
Conversation
…ts at boot Extends osm_lance.rs's established pattern (lance::dataset::Dataset + lance_graph_contract, already used for the row slab) to the .chains and .books sidecars: converted once at boot into Lance datasets on the same local volume, mirroring the row slab's own fix in #135/#136/#137. New osm_chains_books_lance.rs: - write_ordinal_blob_dataset / OrdinalIndex / take_by_row_index / take_by_ordinal_sparse: the write/read primitives for variable-length per-ordinal blobs (ordinal: UInt32Array, value: LargeBinaryArray). - Found via TDD: Books ordinals are dense (row position == ordinal, no index needed) but Chains ordinals are sparse (only tagged ways get an entry) and need the OrdinalIndex translation. A gapped fixture (7/42/1000/1001) is the regression test for this. - ensure_chains_lance_local / ensure_books_lance_local: the actual boot-time conversion, reading the already S3-hydrated local sidecars. Books split into 4 sibling datasets (identities/tag_keys/tag_values/ labels), one per dense codebook. main.rs: wires both conversions right after the existing slab conversion, same fail-open contract — any failure or skip falls back to the existing eager-Vec read path unaffected. Companion sibling-repo change (openstreetmap-website-rs, committed separately): Chains::iter() — a zero-copy borrowing iterator over (ordinal, raw bytes) — and decode_chain promoted to pub, so raw bytes read back from Lance decode through the same function Chains::get() uses internally. NOT yet done: osm_features.rs's open_chains()/open_books() still do the eager Vec read this whole plan exists to fix — this commit builds the Lance datasets at boot but nothing reads from them yet. See claude-notes/plans/2026-08-16-chains-books-lancedb-blob.md for the remaining checklist. Verified: cargo check -p cockpit-server (clean) and cargo nextest run -p cockpit-server --bin q2-cockpit osm_chains_books_lance (7/7 passing, including two real end-to-end round-trip tests). Full workspace verification (cargo build --workspace / cargo nextest run --workspace / cargo xtask verify) has NOT been run yet this session due to severe disk-space constraints in this environment (see the plan file's disk-constraint note) — required before push per CLAUDE.md.
|
Important Review available on request
Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_440e47c4-2647-47f3-b9f0-60928321cbc2) |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Summary
Extends
osm_lance.rs's established pattern (lance::dataset::Dataset+lance_graph_contract, already used for the row slab) to the.chainsand.bookssidecars: converted once at boot into Lance datasets on the same local volume, mirroring the row slab's own fix in #135/#136/#137.osm_chains_books_lance.rs:write_ordinal_blob_dataset/OrdinalIndex/take_by_row_index/take_by_ordinal_sparse— the write/read primitives for variable-length per-ordinal blobs (ordinal: UInt32Array,value: LargeBinaryArray).OrdinalIndextranslation. A deliberately gapped fixture (ordinals 7/42/1000/1001) is the regression test for this.ensure_chains_lance_local/ensure_books_lance_local— the actual boot-time conversion, reading the already S3-hydrated local sidecars. Books split into 4 sibling datasets (identities/tag_keys/tag_values/labels), one per dense codebook.main.rswires both conversions right after the existing slab conversion, same fail-open contract — any failure or skip falls back to the existing eager-Vecread path unaffected.AdaWorldAPI/openstreetmap-website-rs#30):Chains::iter()— a zero-copy borrowing iterator over(ordinal, raw bytes)— anddecode_chainpromoted topub, so raw bytes read back from Lance decode through the same functionChains::get()uses internally.NOT yet done
osm_features.rs'sopen_chains()/open_books()still do the eagerVecread this whole plan exists to fix — this PR builds the Lance datasets at boot but nothing reads from them yet. That's the batched-take_rows-per-request rewrite described inclaude-notes/plans/2026-08-16-chains-books-lancedb-blob.md, tracked as the remaining checklist item there.Test plan
cargo check -p cockpit-server— clean.cargo nextest run -p cockpit-server --bin q2-cockpit osm_chains_books_lance— 7/7 passing, including two real end-to-end round-trip tests (ensure_chains_lance_local_round_trips_a_real_chains_sidecar,ensure_books_lance_local_round_trips_all_four_dense_codebooks).cargo build --workspace/cargo nextest run --workspace/cargo xtask verify) has not been run this session — this environment hit severe disk-space exhaustion twice while working on this change (details in the plan file's disk-constraint note), and full verification was explicitly deferred by request given that history. Scoped verification above is what's been confirmed.See
claude-notes/plans/2026-08-16-chains-books-lancedb-blob.mdfor the full checklist and disk-constraint notes.Generated by Claude Code