perf(fusillade): archive large batches in resumable chunks

[fergusfinn]

Stacked on #1878. It uses #1878's maintenance deadline (maintenance_deadline(), begin_maintenance_write_until(), bound_to_deadline()), so merge #1878 first; this PR's base then moves to main.

Problem

A batch moved into the archive in one transaction. It copied every row with INSERT ... SELECT ... ON CONFLICT DO NOTHING, deleted the originals, then stamped location = 'archive'.

On a large batch that transaction outlives the daemon's maintenance timeout. It rolls back, and the next pass starts again from the first row. So the copy is repeated on every pass and never finishes. Each attempt leaves all of its copied rows, and their out-of-line (TOAST) storage, behind as dead tuples in the archive partition. On a large deployment the partition's heap and TOAST storage fill with discarded copies and grow far larger than the live data. Autovacuum then spends hours on it, while the batch itself stays live.

With #1878 a failing move is cancelled on the server at the call's deadline instead of running on. It is still retried, and still discarded, on every pass.

Change

A batch now moves in bounded chunks, one transaction per chunk (fusillade-arsenal/src/postgres/batch_archive_move.rs):

• Checks: each chunk locks the batch row with SKIP LOCKED, the same as before. It re-checks the same preconditions (not deleted, not already archived, counts frozen), share-locks the week's bucket row and checks the partition exists.
• Move: it then copies up to 1,000 of the batch's remaining live rows into the archive and deletes exactly those rows from requests. It verifies that every row it selected was copied and deleted, or aborts.
• Location: it sets location to split while rows remain and to archive when the last row moves. The stamp uses the same retry_version compare-and-swap as before.
• Deadline:
  • One archive_batch call takes a single maintenance deadline, which fix(fusillade): bound maintenance transactions on the server #1878 derives from the daemon's configured query timeout.
  • Each chunk begins with begin_maintenance_write_until(deadline), and every later statement in the chunk is bound with bound_to_deadline.
  • New chunks start only during the first half of the call's budget, so the last chunk started has at least half the budget to finish instead of being cancelled part-way.
  • A call that stops early returns a new ArchiveOutcome::Progressed { rows }. The batch stays split and remains a candidate, so the next pass resumes from what is left. The daemon counts Progressed as a move (outcome="progressed") and adds its rows to fusillade_archive_moved_rows_total.
• Stops: if a later chunk finds the batch no longer a candidate (a retry, a cancel, a concurrent mover), the call reports the progress so far. If the chunk finds the partition missing or its week fenced, the call returns SkippedNoPartition even after progress, so the daemon still raises archive_partition_missing. In both cases the batch stays split.
• Small batches fit in one chunk and still move in a single transaction.
• Downloads: the output, error and results download streams now read the batch's bucket stamp in each page's own statement, (SELECT archive_bucket FROM batches WHERE id = $1), the same way get_batch_requests already does. Previously they resolved it once when the stream started, falling back to the week the batch was created in. A move stamps the bucket in the same transaction as the first rows it moves, so every page sees the partition of every row its snapshot sees archived. That holds whichever week the mover chose: its own week, or the current week when its own week has been retired.

Every chunk deletes what it copies, so no row is ever copied twice. The ON CONFLICT arm stays only as a guard and does not fire on the normal path. An abandoned chunk discards only its own rows; earlier chunks stay committed.

Safety argument

• Exactly one table: every committed state keeps each row in exactly one of requests and batch_requests_archive. The copy and delete of a chunk commit together, and the delete removes only rows verified present in the archive.
• split already works: it is the state a retry of an archived batch leaves, with some rows archived and the rest live. Every path that reads or changes a batch's rows already handles it:
  • get_batch_requests, get_requests and the download streams read the union of live and bucket-pruned archive rows, whatever the location.
  • retry_failed_requests_for_batch and the per-request retry move failed or cancelled archived rows back when location is not live, and re-pend live ones in place.
  • The freeze, finalizer and cancelled-batch counters add archived counts for split batches. Candidate listing already includes split.
• Frozen rows don't change: only frozen batches move, and any retry un-freezes the batch and bumps retry_version under the batch-row lock. Each chunk re-checks both under that lock, so a retry between chunks stops the move. The batch then stays split and is resumed after it re-freezes.
• Retention waits for the move: partition retirement drops a week only when every batch in it has location = 'archive', so a part-moved batch keeps its week alive. A fenced week stops further chunks and is reported, as above.
• Deleted batches: the purge already clears both tables for a soft-deleted batch, and the move skips soft-deleted batches.

Testing

New tests:

• test_archive_moves_large_batch_in_resumable_chunks: 7 rows moved in chunks of 2, one chunk per call. After every call it asserts:

  • no row is in both tables and none is lost;
  • get_batch_requests and get_requests return every row exactly once;
  • the frozen counts are unchanged;
  • the batch is split and still listed as a candidate until the last chunk stamps archive.

  It takes 4 transactions, and the archive ends up holding exactly the batch's ids, each once.

• test_archive_finishes_large_batch_within_one_call_budget: one call with the normal deadline moves all 7 rows chunk by chunk and returns Archived { rows: 7 }.

• test_retry_of_part_moved_batch_resumes_and_rearchives: a batch is retried after two chunks have moved. Failed rows on both sides become pending live rows, only completions stay archived, and the counts still include the archived completions. After re-freezing it finishes into the same single bucket.

• test_part_moved_batch_reports_a_fenced_partition: after one chunk the batch's week is fenced. The next call returns SkippedNoPartition, and the batch stays split with every row in exactly one table and readable.

• test_download_across_a_move_into_the_current_week: the batch's creation week is retired, so the mover routes it into the current week. An output download and a results download each start while the batch is live. With a one-item download buffer, each stream has read its first 1,000-row page but not its second when the batch moves, so the move lands exactly between pages. Both return all 1,005 rows exactly once. As a negative control, restoring the old creation-week bucket in the output stream makes this test fail with 1,000 of 1,005 rows.

• Unit tests for the outcome of a call that stops after progress.

Existing coverage and checks:

• All existing archive, retry, freeze, download and purge tests pass unchanged: all of cargo test -p fusillade-arsenal --tests and cargo test -p fusillade.
• cargo fmt --check and cargo clippy --all-features --no-deps -D warnings pass for fusillade-arsenal, fusillade and fusillade-core. An offline build (SQLX_OFFLINE=true) of those crates and dwctl also passes.
• scripts/check_query_projections.py passes. The existing reviewed exemption for the archive INSERT ... SELECT r.* moves to the new file with the query.

The .sqlx offline data is updated for the new and changed statements only.

https://claude.ai/code/session_01X51YHjxkWDVMWSdBkLj3ti

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	be350bf
Status:	✅  Deploy successful!
Preview URL:	https://5a012a67.control-layer.pages.dev
Branch Preview URL:	https://perf-archive-moves-in-chunks.control-layer.pages.dev

View logs

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[cubic-dev-ai]

All reported issues were addressed across 17 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>