Repository navigation
Conversation
A follow that falls more than the broadcast channel's capacity (1024) behind got RecvError::Lagged, logged a warning and carried on from the channel's new head, silently dropping every frame in between. Any slow reader hits this: an `xs cat -f` whose consumer pauses for 3s while 20k frames are appended received 1,673 of them. Every broadcast frame is committed before it is sent, so the gap is always in the store. On Lagged, read the stored frames after the last one delivered (in chunks of 4096), then resume the channel and drop its backlog by id. The subscription is now taken under the append lock together with a floor id, so a follow that lags before delivering anything knows where to replay from. Ephemeral frames are never stored and can still be lost to a lag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /append/<topic> opened a cacache writer, which creates a temp file, before reading the body, then dropped it unused when the body was empty. A meta-only append paid a file create and unlink it never needed. Open the writer on the first non-empty chunk instead. Meta-only appends (4 pipelined connections, macOS): 16.1k/s -> 47.3k/s. Appends with a body are unchanged (~8.2k/s either way). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Append many frames in one request. The body is NDJSON, one frame per line
({topic, meta?, ttl?, hash?}); the response is the stored frames as
NDJSON, in order.
Store::append_batch takes the append lock once, assigns consecutive ids,
stages every frame in a single write batch and commits it, then
broadcasts in id order. So a batch is contiguous in the stream and
atomic: a malformed line (400) or an invalid topic stores nothing.
last:n trims are scheduled once per topic per batch. insert_frame is
split into add_frame_to_batch + note_indexed so both paths share it.
Meta-only frames, 4 pipelined connections, macOS: 50k/s one per request,
229k/s at 10 per batch, 379k/s at 1000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #156 and #157. This branch contains their commits too. Only the last commit (
feat: add POST /append-batch …) is new here, and I'll rebase once those land. It depends on #156 because a batch larger than the broadcast channel overruns every live follower on its own. Without that fix, followers silently drop most of the batch.What it adds
POST /append-batchtakes an NDJSON body with one frame per line (topic, optionalmeta,ttl,hash). It returns the stored frames as NDJSON, in order.Ordering and atomicity
Store::append_batchtakes the append lock once for the whole batch. Under that lock it:So a batch is contiguous in the stream, no other append can interleave, and a reader sees all of it or none of it. A malformed line returns 400 and an invalid topic returns an error; either way nothing is stored.
Other details:
ttldefaults toforever, as for a single append.last:ntrims are scheduled once per topic per batch, not once per frame.insert_frameis split intoadd_frame_to_batch+note_indexed, so the single-frame path and the batch path share the same code.Numbers (macOS, aarch64; 500k meta-only frames; 4 pipelined connections; includes #157)
/append/{topic})For comparison, the store alone does about 230k/s with one
appendper frame, and main over HTTP does about 16k/s.Checks
test_append_batch(store level: ids, order, atomic reject) andtest_handle_stream_append_batch(route, handler, ttl default, 400 on a bad line).test_follow_replays_frames_dropped_by_lagis extended with a 3000-frame batch.cargo test --release: 230 passed. Integration: 12 + 12 passed.tests/test_xs_nu.nu: passed.cargo clippy -- -D warningsandcargo fmt --check: clean.cat -fduring load at batch 1, 100 and 1000 (up to 371k/s): 200,000 / 200,000 frames, no dupes, same order as the store.Open questions
/append-batch. Content negotiation on/appendis an alternative.xs.nucommand yet. Happy to add one if you want the endpoint.