Skip to content

feat: add POST /append-batch and Store::append_batch - #158

Open
kiil wants to merge 3 commits into
cablehead:mainfrom
kiil:feat/append-batch
Open

kiil wants to merge 3 commits into
cablehead:mainfrom
kiil:feat/append-batch

Conversation

@kiil

@kiil kiil commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #156 and #157. This branch contains their commits too. Only the last commit (feat: add POST /append-batch …) is new here, and I'll rebase once those land. It depends on #156 because a batch larger than the broadcast channel overruns every live follower on its own. Without that fix, followers silently drop most of the batch.

What it adds

POST /append-batch takes an NDJSON body with one frame per line (topic, optional meta, ttl, hash). It returns the stored frames as NDJSON, in order.

printf '%s\n' \
    '{"topic":"clip.add","meta":{"n":1}}' \
    '{"topic":"clip.add","meta":{"n":2},"ttl":"last:100"}' |
  curl --unix-socket ./store/sock --data-binary @- http://localhost/append-batch

Ordering and atomicity

Store::append_batch takes the append lock once for the whole batch. Under that lock it:

  • assigns consecutive ids,
  • stages every frame (stream plus indexes) in one fjall write batch,
  • commits it,
  • broadcasts the frames in id order.

So a batch is contiguous in the stream, no other append can interleave, and a reader sees all of it or none of it. A malformed line returns 400 and an invalid topic returns an error; either way nothing is stored.

Other details:

  • ttl defaults to forever, as for a single append.
  • last:n trims are scheduled once per topic per batch, not once per frame.
  • insert_frame is split into add_frame_to_batch + note_indexed, so the single-frame path and the batch path share the same code.

Numbers (macOS, aarch64; 500k meta-only frames; 4 pipelined connections; includes #157)

frames per request frames/s
1 (/append/{topic}) 50k
10 229k
100 340k
1000 379k

For comparison, the store alone does about 230k/s with one append per frame, and main over HTTP does about 16k/s.

Checks

  • New tests: test_append_batch (store level: ids, order, atomic reject) and test_handle_stream_append_batch (route, handler, ttl default, 400 on a bad line). test_follow_replays_frames_dropped_by_lag is extended with a 3000-frame batch.
  • cargo test --release: 230 passed. Integration: 12 + 12 passed. tests/test_xs_nu.nu: passed.
  • cargo clippy -- -D warnings and cargo fmt --check: clean.
  • Live cat -f during load at batch 1, 100 and 1000 (up to 371k/s): 200,000 / 200,000 frames, no dupes, same order as the store.
  • The docs example above was run against this build.

Open questions

kiil and others added 3 commits September 29, 2026 10:38
A follow that falls more than the broadcast channel's capacity (1024)
behind got RecvError::Lagged, logged a warning and carried on from the
channel's new head, silently dropping every frame in between. Any slow
reader hits this: an `xs cat -f` whose consumer pauses for 3s while 20k
frames are appended received 1,673 of them.

Every broadcast frame is committed before it is sent, so the gap is
always in the store. On Lagged, read the stored frames after the last one
delivered (in chunks of 4096), then resume the channel and drop its
backlog by id. The subscription is now taken under the append lock
together with a floor id, so a follow that lags before delivering
anything knows where to replay from. Ephemeral frames are never stored
and can still be lost to a lag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /append/<topic> opened a cacache writer, which creates a temp file,
before reading the body, then dropped it unused when the body was empty.
A meta-only append paid a file create and unlink it never needed.

Open the writer on the first non-empty chunk instead. Meta-only appends
(4 pipelined connections, macOS): 16.1k/s -> 47.3k/s. Appends with a body
are unchanged (~8.2k/s either way).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Append many frames in one request. The body is NDJSON, one frame per line
({topic, meta?, ttl?, hash?}); the response is the stored frames as
NDJSON, in order.

Store::append_batch takes the append lock once, assigns consecutive ids,
stages every frame in a single write batch and commits it, then
broadcasts in id order. So a batch is contiguous in the stream and
atomic: a malformed line (400) or an invalid topic stores nothing.
last:n trims are scheduled once per topic per batch. insert_frame is
split into add_frame_to_batch + note_indexed so both paths share it.

Meta-only frames, 4 pipelined connections, macOS: 50k/s one per request,
229k/s at 10 per batch, 379k/s at 1000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant