Skip to content

fix(shard): drive partition repair and the commit walk from the tick - #4006

Open
numinnex wants to merge 4 commits into
masterfrom
partition_repair_and_commit_from_tick
Open

fix(shard): drive partition repair and the commit walk from the tick#4006
numinnex wants to merge 4 commits into
masterfrom
partition_repair_and_commit_from_tick

Conversation

@numinnex

Copy link
Copy Markdown
Contributor

A partition backup that fell behind had no way back on its own. Every repair arming site is edge triggered, and the edge is starvable: a follower advances commit_max from every prepare header in replicate_preflight, before the gap check drops the prepare, so the commit heartbeat lands as Accepted rather than Advanced and the backstop inside that branch never fires. Under sustained produce the gap wedged until an unrelated view change happened along. The same starvation left a second state stuck, a backup holding resident committed ops that nothing re-drove, because the commit walk is driven by the same branch.

▎ The partition tick now evaluates two level-triggered checks off one probe. A hole below commit_max arms the existing repair against the primary, debounced and capped at three arms per tick. Resident committed ops re-drive commit_journal directly, undebounced and uncapped, since that is local work already owed. The two predicates are mutually exclusive by construction, so the arms cannot fight over a partition. Gap-check drops were log only and now increment a counter.

@numinnex

Copy link
Copy Markdown
Contributor Author

Merge after #4005

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 31, 2026
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.08816% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.99%. Comparing base (c8ef3b9) to head (3f741c4).
⚠️ Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
core/simulator/src/lib.rs 92.85% 19 Missing and 6 partials ⚠️
core/shard/src/lib.rs 95.82% 14 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4006      +/-   ##
============================================
+ Coverage     84.91%   84.99%   +0.07%     
  Complexity     1405     1405              
============================================
  Files          1224     1225       +1     
  Lines        179327   180588    +1261     
  Branches     145615   146878    +1263     
============================================
+ Hits         152273   153488    +1215     
- Misses        23026    23035       +9     
- Partials       4028     4065      +37     
Components Coverage Δ
Rust Core 85.89% <95.08%> (+0.08%) ⬆️
Java SDK 67.35% <ø> (ø)
C# SDK 75.32% <ø> (-0.08%) ⬇️
Python SDK 90.06% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 96.22% <ø> (+0.08%) ⬆️
Go SDK 69.36% <ø> (+0.03%) ⬆️
Files with missing lines Coverage Δ
core/partitions/src/iggy_partition.rs 89.68% <100.00%> (+0.08%) ⬆️
core/server/src/partition_reconciler.rs 97.86% <100.00%> (-0.10%) ⬇️
core/shard/src/metrics.rs 87.17% <100.00%> (+0.62%) ⬆️
core/shard/src/router.rs 78.34% <100.00%> (+0.07%) ⬆️
core/shard/src/lib.rs 83.07% <95.82%> (+0.88%) ⬆️
core/simulator/src/lib.rs 96.45% <92.85%> (-0.50%) ⬇️

... and 48 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@numinnex
numinnex force-pushed the partition_repair_and_commit_from_tick branch from 4e9d294 to 3f741c4 Compare August 31, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant