Skip to content

fix(cluster): serve metadata reads at or above the client's own writes - #4024

Open
numinnex wants to merge 1 commit into
masterfrom
peer_shard_read_stale_state
Open

fix(cluster): serve metadata reads at or above the client's own writes#4024
numinnex wants to merge 1 commit into
masterfrom
peer_shard_read_stale_state

Conversation

@numinnex

@numinnex numinnex commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

A client that commits a metadata write and then re-homes its session onto a backup can be served the pre-write state. auth.rs already documents the gap: the session epoch is the register's commit op, and on a backup that forwarded the proposal the local applied commit still lags it. A backup applies committed ops only as its commit walk advances, and nothing tied a read to the op the client's own write committed.

Metadata reads now gate on the connection's committed watermark. One applied-frontier counter per process advances after every metadata apply and is shared with every shard, which for the first time gives a shard without consensus the applied position it had no way to observe. The watermark comes from the commit field replies already carry, seeded at bind from the session epoch. The fast path is a single atomic load with no awaits, so the shared-nothing read path is unchanged; a lagging node parks briefly, then fails the read retryable rather than answering stale.

Over HTTP this closes the forwarded-register case, where a healthy backup forwards the register so the bound epoch can exceed the local frontier. It does not close the case where forwarding is active: the follower relays the write, its handler never runs, so the node that later serves the read holds no session and no watermark. Closing that needs the serving primary's commit op to travel back to the reading node, for instance a response header beside the view the forward middleware already relays. That is additive but touches every control-plane write response, so it is left out here and documented at the gate.

The BDD delete-then-get steps now assert "not the stream we deleted" instead of "nothing at this id". The server hands a deleted stream's numeric id straight to the next create, so once scenarios share a server the old assertion cannot hold, and removing the polling loop without this would have left the spec flaky for an unrelated reason.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Sep 1, 2026
@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.45283% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.05%. Comparing base (328b289) to head (9d2e8ee).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
core/simulator/src/lib.rs 90.10% 15 Missing and 3 partials ⚠️
core/server/src/dispatch.rs 78.57% 11 Missing and 1 partial ⚠️
core/server/src/http/reads.rs 93.75% 3 Missing and 2 partials ⚠️
core/server/src/http/submit.rs 80.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4024      +/-   ##
============================================
+ Coverage     85.00%   85.05%   +0.04%     
  Complexity     1402     1402              
============================================
  Files          1225     1225              
  Lines        180283   180752     +469     
  Branches     146587   147056     +469     
============================================
+ Hits         153248   153735     +487     
+ Misses        22993    22952      -41     
- Partials       4042     4065      +23     
Components Coverage Δ
Rust Core 85.95% <90.90%> (+0.05%) ⬆️
Java SDK 67.29% <ø> (ø)
C# SDK 75.44% <ø> (+0.07%) ⬆️
Python SDK 90.06% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 96.24% <ø> (ø)
Go SDK 69.35% <ø> (+0.03%) ⬆️
Files with missing lines Coverage Δ
core/metadata/src/impls/metadata.rs 88.19% <100.00%> (+0.10%) ⬆️
core/server/src/bootstrap.rs 80.77% <100.00%> (+0.28%) ⬆️
core/server/src/http.rs 92.96% <ø> (ø)
core/server/src/http/error.rs 87.31% <100.00%> (+0.34%) ⬆️
core/server/src/http/extractor.rs 85.71% <100.00%> (+0.46%) ⬆️
core/server/src/http/reply.rs 90.03% <100.00%> (ø)
core/server/src/http/session.rs 96.11% <100.00%> (+0.43%) ⬆️
core/server/src/http/state.rs 88.54% <100.00%> (+0.12%) ⬆️
core/server/src/session_manager.rs 91.41% <100.00%> (+1.03%) ⬆️
core/simulator/src/client.rs 96.53% <100.00%> (+0.18%) ⬆️
... and 5 more

... and 32 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant