fix(hotblocks): enforce finalized fork floor - #96
Merged
Conversation
mo4islona
force-pushed
the
fix/hotblocks-finalized-fork-floor
branch
4 times, most recently
from
July 16, 2026 12:06
45bf5ea to
fd80630
Compare
mo4islona
force-pushed
the
fix/hotblocks-finalized-fork-floor
branch
14 times, most recently
from
July 30, 2026 07:27
ca92e7d to
406bfee
Compare
mo4islona
force-pushed
the
fix/hotblocks-finalized-fork-floor
branch
2 times, most recently
from
July 30, 2026 08:41
446aba8 to
86d20de
Compare
A fork whose common ancestor lies inside a finality-straddling chunk cannot resume at fin+1 — insert_fork replaces whole chunks, so a mid-chunk position wedges the dataset on a 60s restart loop — and resuming below fin instead silently rewrites the finalized prefix (GAP-22). Resolution now stops at the stored chunk boundary and the write path verifies the finalized region: a replacement reaching that low must span fin and reproduce every block up to it, hash for hash. Comparing only fin's own hash would admit a source rewriting the interior, since hashes come from the source and nothing ties a reproduced boundary to what leads to it. Payload equality is out of reach by construction (INV-13 Scope). The guard only recovers if the replay reaches fin, which was left as an assumption about the flush triggers. It does not hold: the 200k-row bound can cut the replayed chunk one block below fin, the write path refuses it, and every retry repeats the identical cut — an indefinite park, observed. Fork resolution now carries fin to the ingest, which withholds a chunk until it covers that block. A retention trim can also land inside a fork replay, leaving a rollback resolved against a window that no longer exists; insert_fork then drops every surviving chunk and the head ends up under the floor with the trimmed blocks back. Whether the ingest may keep running is read off the trim's own result: predicting it from block numbers covered only a floor moving past the head, while the window is also cleared on an empty dataset, on a parent-hash mismatch and on a gap — none of which move the floor past the head, so a stale ingest outlived all three, and with nothing left stored it committed below the floor against a guard that had no bound to enforce. A surviving head still keeps the ingest — the floor advances continuously through the Api path, so restarting on every trim reconnected the source for nothing — and that replay is refused at commit instead. The bound is the first stored chunk, not first(D): retention trims whole chunks, so its logical floor can sit inside the surviving one, which a fork legitimately resumes at; with nothing stored it falls back to first(D), which is where resolution resumes anyway. The check precedes the finality guard because a trim above fin clears fin, and then it is the only guard left. A push repeating the current floor is answered before the trim runs: nothing below it is left to delete, and reading the verdict off an empty window reports a lost head and restarts an ingest the call never touched — once per push, on a dataset that has not committed its first chunk yet. The report that becomes fin was itself unchecked, and it anchors both that guard and every later rollback. It is a header the source hands over, not a block it served, so where it names a height the committed chunk carries, the chunk now decides (WP-8); where it names a height the chunk skips there is no hash to record, so it is dropped rather than refused — refusing would park a slot-numbered dataset for as long as an honest source keeps reporting a height its own stream skipped. Every refusal shape shared one metric bucket, so a stale-ingest refusal that clears itself on the next epoch read exactly like a source rewriting finalized history; UnapplicableFork::reason is now a slug on a cause label. And a withheld replay is counted only where a flush is attempted, which a source that stops below the floor never reaches — the pending floor is a level-readable gauge, set at fork resolution. Unrelated but folded in: the query slot is released before its caller is woken. The pool thread sent the result and dropped the slot afterwards, so a caller woken in between could be refused admission for a query that had already finished — and the panic-path test flaked on it, 3 failures in 200 local runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FD1RAN4sYMB1mgTVMEoASi Co-Authored-By: Codex <codex@openai.com>
mo4islona
force-pushed
the
fix/hotblocks-finalized-fork-floor
branch
from
July 30, 2026 12:26
86d20de to
c672856
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes GAP-22 and GAP-42 — four ways a fork could replace the finalized prefix or park the dataset on the 60-second retry loop for good.
Resolution position.
insert_forkreplaces whole chunks, sofin + 1is not a representable position when finality falls inside one: clamping there wedges the commit. Resolution now stops at the stored chunk boundary, which may sit at or belowfin, and the finalized prefix is guarded on the write path instead — a replacement reaching that low is admitted only if it spansfinand reproduces every block up to it, hash for hash (Tx::validate_finalized_prefix). Comparing onlyfin's own hash admits a source that rewrites the interior; payload equality is out of reach by construction (INV-13 Scope).Reaching
finat all. That guard only recovers if the replay gets back up tofin, which was left as an assumption about the flush triggers. The 200k-row bound can cut the replayed chunk one block short, and every retry repeats the identical cut.Rollback::reach_at_leastnow carriesfinto the ingest, which withholds a chunk until it covers that block.A trim inside a replay. A retention trim landing between fork resolution and its commit leaves a rollback aimed at a window that no longer exists.
retainnow reports whether the head survived instead of leaving the caller to predict it, the ingest restarts when it did not, and a replacement starting below the first stored chunk is refused at commit. A push repeating the current floor is answered before the trim runs — reading the verdict off an empty window would restart an ingest the call never touched, once per push.The finality report itself. It is a header the source hands over, not a block it served, and it anchors both the guard above and every later
compute_rollback. Where it names a height the committed chunk carries, the chunk decides (WP-8); a height the chunk skips is ignored rather than refused — refusing would park a slot-numbered dataset on an honest report.Observability
UnapplicableFork::reasonis now a metric slug on acauselabel: the class alone did not say which refusal fired, and a stale-ingest refusal that clears itself on the next epoch read exactly like a source rewriting finalized history. A withheld replay is counted only where a flush is attempted, which a source stopping below the floor never reaches, so the pending floor is also level-readable (ingest_flush_floor, set at fork resolution).A compaction-invalidated physical rollback boundary is now classified as
stale_rollback_boundaryinstead ofother/unspecified. The stale replacement remains atomic, and resolution against the current layout goes one chunk deeper.Also in here
Known limits
Test plan
Write-controller and ingest regressions cover finalized-prefix integrity, a second fork while replay is withheld, finality advancing before the first replay flush, and compaction consuming a resolved boundary. The five black-box
ct4_finalityscenarios now assert an actual new/streamrequest and the exact stored boundary used asresume_from, so a silent full-window replay no longer passes as recovery.cargo test -p sqd-hotblocks -p sqd-storage -p sqd-hotblocks-harnesscargo clippy -p sqd-hotblocks -p sqd-storage --all-targets -- -D clippy::correctnesscargo +nightly fmt --all -- --check