osm lance: build the dataset over the slab mapping, then evict it

[AdaWorldAPI]

bake → lance → evict the bake from RAM immediately. One file.

RAM is the dominant line item — $10/GB-month against $0.15 for volume and $0.015 for bucket, so a GB held in memory costs 667× a GB in the bucket. The conversion was the worst offender in the process.

The allocation

write_lance took an owned Vec<u8>, and ensure_lance_local filled it with:

let bytes = std::fs::read(slab_path)?;   // the WHOLE slab, on the heap

1.4 GB for Berlin, 3.75 GB for Brandenburg. Anonymous memory — no amount of pressure can reclaim it — sitting on top of an mmap of the same file taken 80 lines earlier for the digest. That is how 1.4 GB of data becomes a multi-gigabyte plateau on the memory graph while CPU reads 0.

Arrow never needed to own it. The mapping is now kept as an Arc and adopted directly:

let buffer = unsafe {
    Buffer::from_custom_allocation(NonNull::new_unchecked(map.as_ptr() as *mut u8),
                                   map.len(),
                                   map.clone())   // owner: the pages can't vanish under Arrow
};

The pre-existing pointer-identity assertion still guards the zero-copy claim — it now proves the buffer addresses the mapping.

The eviction

Once the dataset is on disk, the slab's pages have done their job. Left alone they simply stay: a 24 GB limit over a 1.4 GB file means nothing ever creates the pressure that would reclaim them, so the conversion's footprint stays in the bill until the process exits.

release_after_write advises MADV_DONTNEED — #[cfg(unix)], a documented no-op elsewhere per the cross-platform rule — but only when it holds the last reference. A surviving reference means something downstream is still reading those pages; it logs and leaves them alone rather than yanking memory from under a live reader.

The doc comment is explicit about the limit of the guarantee: the advice stops this process holding the pages so they can be reclaimed. It does not promise they leave the page cache the instant it returns.

The test is two-sided on the property that matters

A release_after_write that always answered "no" — Lance still holding the buffer, say — would be completely inert and look identical from outside. So the test asserts both directions: a mapping something else still holds must NOT be released, and a sole holder MUST be.

Gates

15/15 osm_lance tests, 53/53 across the osm modules, cargo check -p cockpit-server clean. Scoped to that crate — this touches one leaf binary and nothing depends on it.

Not measured: the actual resident-set change on the deploy. The mechanism is right and the figures above are arithmetic, not observation. The honest next number is what the container is charged for while serving tiles after this lands.

Not in this PR

The steady-state ~2 GB baseline is the serving mmap, not this path — it needs either reclaim pressure (lowering the memory limit, which costs nothing and needs no deploy) or the columnar reader through lance-graph. Separate changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw

---

Generated by Claude Code

[coderabbitai]

Important

Review available on request

• 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 711c64ed-f9cc-4142-99fb-9113c1b6288b

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_b172b134-00bc-4fb6-a1ea-6b6f14a44fdb)