Skip to content

fix(hotblocks): enforce finalized fork floor - #96

Merged
mo4islona merged 1 commit into
masterfrom
fix/hotblocks-finalized-fork-floor
Jul 30, 2026
Merged

fix(hotblocks): enforce finalized fork floor#96
mo4islona merged 1 commit into
masterfrom
fix/hotblocks-finalized-fork-floor

Conversation

@mo4islona

@mo4islona mo4islona commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes GAP-22 and GAP-42 — four ways a fork could replace the finalized prefix or park the dataset on the 60-second retry loop for good.

Resolution position. insert_fork replaces whole chunks, so fin + 1 is not a representable position when finality falls inside one: clamping there wedges the commit. Resolution now stops at the stored chunk boundary, which may sit at or below fin, and the finalized prefix is guarded on the write path instead — a replacement reaching that low is admitted only if it spans fin and reproduces every block up to it, hash for hash (Tx::validate_finalized_prefix). Comparing only fin's own hash admits a source that rewrites the interior; payload equality is out of reach by construction (INV-13 Scope).

Reaching fin at all. That guard only recovers if the replay gets back up to fin, which was left as an assumption about the flush triggers. The 200k-row bound can cut the replayed chunk one block short, and every retry repeats the identical cut. Rollback::reach_at_least now carries fin to the ingest, which withholds a chunk until it covers that block.

A trim inside a replay. A retention trim landing between fork resolution and its commit leaves a rollback aimed at a window that no longer exists. retain now reports whether the head survived instead of leaving the caller to predict it, the ingest restarts when it did not, and a replacement starting below the first stored chunk is refused at commit. A push repeating the current floor is answered before the trim runs — reading the verdict off an empty window would restart an ingest the call never touched, once per push.

The finality report itself. It is a header the source hands over, not a block it served, and it anchors both the guard above and every later compute_rollback. Where it names a height the committed chunk carries, the chunk decides (WP-8); a height the chunk skips is ignored rather than refused — refusing would park a slot-numbered dataset on an honest report.

Observability

UnapplicableFork::reason is now a metric slug on a cause label: the class alone did not say which refusal fired, and a stale-ingest refusal that clears itself on the next epoch read exactly like a source rewriting finalized history. A withheld replay is counted only where a flush is attempted, which a source stopping below the floor never reaches, so the pending floor is also level-readable (ingest_flush_floor, set at fork resolution).

A compaction-invalidated physical rollback boundary is now classified as stale_rollback_boundary instead of other / unspecified. The stale replacement remains atomic, and resolution against the current layout goes one chunk deeper.

Also in here

  • Unrelated, folded in: the query slot is released before its caller is woken, so a caller woken in between can no longer be refused admission for a query that already finished.
  • Normative spec changes are in the diff: WP-6/WP-8, INV-13 Scope, INV-14, FM-OP-6, OB-9, and the living conformance matrix (GAP-22/42 closed, GAP-43 filed).

Known limits

  • Nothing caps how long a replay may withhold (GAP-43). The pending floor is visible; the escape is not automatic.
  • A data-availability mask change inside a replayed finalized range stops the ingest loudly (FM-OP-6: out of scope, operator-sequenced).
  • A compaction-invalidated rollback boundary self-heals on the next epoch; immediate in-epoch re-resolution remains future work.
  • The trim-inside-replay race has no black-box script — driving SET-RETENTION into the replay window needs a harness retention client and deterministic control of the replay's duration. Both damaging outcomes are covered by unit tests.

Test plan

Write-controller and ingest regressions cover finalized-prefix integrity, a second fork while replay is withheld, finality advancing before the first replay flush, and compaction consuming a resolved boundary. The five black-box ct4_finality scenarios now assert an actual new /stream request and the exact stored boundary used as resume_from, so a silent full-window replay no longer passes as recovery.

  • cargo test -p sqd-hotblocks -p sqd-storage -p sqd-hotblocks-harness
  • cargo clippy -p sqd-hotblocks -p sqd-storage --all-targets -- -D clippy::correctness
  • cargo +nightly fmt --all -- --check

@mo4islona
mo4islona force-pushed the fix/hotblocks-finalized-fork-floor branch 4 times, most recently from 45bf5ea to fd80630 Compare July 16, 2026 12:06
@mo4islona
mo4islona force-pushed the fix/hotblocks-finalized-fork-floor branch 14 times, most recently from ca92e7d to 406bfee Compare July 30, 2026 07:27
@mo4islona
mo4islona force-pushed the fix/hotblocks-finalized-fork-floor branch 2 times, most recently from 446aba8 to 86d20de Compare July 30, 2026 08:41
A fork whose common ancestor lies inside a finality-straddling chunk cannot resume
at fin+1 — insert_fork replaces whole chunks, so a mid-chunk position wedges the
dataset on a 60s restart loop — and resuming below fin instead silently rewrites
the finalized prefix (GAP-22). Resolution now stops at the stored chunk boundary
and the write path verifies the finalized region: a replacement reaching that low
must span fin and reproduce every block up to it, hash for hash. Comparing only
fin's own hash would admit a source rewriting the interior, since hashes come from
the source and nothing ties a reproduced boundary to what leads to it. Payload
equality is out of reach by construction (INV-13 Scope).

The guard only recovers if the replay reaches fin, which was left as an assumption
about the flush triggers. It does not hold: the 200k-row bound can cut the replayed
chunk one block below fin, the write path refuses it, and every retry repeats the
identical cut — an indefinite park, observed. Fork resolution now carries fin to the
ingest, which withholds a chunk until it covers that block.

A retention trim can also land inside a fork replay, leaving a rollback resolved
against a window that no longer exists; insert_fork then drops every surviving
chunk and the head ends up under the floor with the trimmed blocks back. Whether
the ingest may keep running is read off the trim's own result: predicting it from
block numbers covered only a floor moving past the head, while the window is also
cleared on an empty dataset, on a parent-hash mismatch and on a gap — none of which
move the floor past the head, so a stale ingest outlived all three, and with
nothing left stored it committed below the floor against a guard that had no bound
to enforce.

A surviving head still keeps the ingest — the floor advances continuously through
the Api path, so restarting on every trim reconnected the source for nothing — and
that replay is refused at commit instead. The bound is the first stored chunk, not
first(D): retention trims whole chunks, so its logical floor can sit inside the
surviving one, which a fork legitimately resumes at; with nothing stored it falls
back to first(D), which is where resolution resumes anyway. The check precedes the
finality guard because a trim above fin clears fin, and then it is the only guard
left. A push repeating the current floor is answered before the trim runs: nothing
below it is left to delete, and reading the verdict off an empty window reports a
lost head and restarts an ingest the call never touched — once per push, on a
dataset that has not committed its first chunk yet.

The report that becomes fin was itself unchecked, and it anchors both that guard and
every later rollback. It is a header the source hands over, not a block it served, so
where it names a height the committed chunk carries, the chunk now decides (WP-8);
where it names a height the chunk skips there is no hash to record, so it is dropped
rather than refused — refusing would park a slot-numbered dataset for as long as an
honest source keeps reporting a height its own stream skipped.

Every refusal shape shared one metric bucket, so a stale-ingest refusal that clears
itself on the next epoch read exactly like a source rewriting finalized history;
UnapplicableFork::reason is now a slug on a cause label. And a withheld replay is
counted only where a flush is attempted, which a source that stops below the floor
never reaches — the pending floor is a level-readable gauge, set at fork resolution.

Unrelated but folded in: the query slot is released before its caller is woken. The
pool thread sent the result and dropped the slot afterwards, so a caller woken in
between could be refused admission for a query that had already finished — and the
panic-path test flaked on it, 3 failures in 200 local runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FD1RAN4sYMB1mgTVMEoASi
Co-Authored-By: Codex <codex@openai.com>
@mo4islona
mo4islona force-pushed the fix/hotblocks-finalized-fork-floor branch from 86d20de to c672856 Compare July 30, 2026 12:26
@mo4islona
mo4islona merged commit 2a80c8e into master Jul 30, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant