Skip to content

Serialize InFlightScriptStore reads on the per-task lock stripe#430

Merged
ppXD merged 1 commit into
mainfrom
fix/inflight-script-store-read-concurrency
Jun 11, 2026
Merged

Serialize InFlightScriptStore reads on the per-task lock stripe#430
ppXD merged 1 commit into
mainfrom
fix/inflight-script-store-read-concurrency

Conversation

@ppXD

@ppXD ppXD commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • InFlightScriptStore.TryGetTicketAsync (the crash-resume re-attach probe) queried the shared scoped IRepository/DbContext without taking the per-serverTaskId lock stripe that the writes (RecordDispatchedAsync/ClearAsync) already use.
  • A parallel batch runs every target's re-attach probe on the one DbContext the Hangfire worker owns for that task. Two concurrent probes — or a probe racing a record/clear — drove two operations onto the same context, and EF threw A second operation was started on this context instance. Intermittent crash in multi-target deployments using crash-resume re-attach.
  • Fix: gate the read with the same stripe as the writes, so all of a task's DbContext access serialises onto one writer.

Why this surfaced now

The KubernetesResumeCheckpointE2ETests/TentacleMixedModeE2ETests.MixedMode_SingleDeployment_BothTargetsExecute E2E (two targets in parallel) hits this reattach path and exposed the race intermittently — it's a narrow timing window, so most CI runs win it. Pre-existing on main; not introduced by any recent change.

Non-breaking + generic

  • Same method signature and return; the read just serialises on the existing stripe mechanism. No API/behavior change for callers — only correctness under concurrency.
  • The write path (MutateAsync) is untouched; the fix mirrors its existing inline gate pattern.
  • The stripe is bounded (64, no per-task growth) and a task always maps to the same stripe, so a task's reads + writes fully serialise; distinct tasks colliding on a stripe serialise harmlessly.

Test plan

  • New integration regression InFlightScriptStoreTests.ConcurrentReadsAndWrites_OnOneSharedContext_DoNotThrow — resolves one store (one DbContext) and fires concurrent reads+writes. Verified red→green: without the fix it fails with the exact production error (A second operation was started on this context instance); with the fix it passes. (The other tests here use a fresh Run scope per call and so never share a context — this one deliberately doesn't.)
  • Checkpoint + resume-reattach integration sweep: 11/11. InFlightScript unit: 11/11.
  • dotnet build — 0 errors.

TryGetTicketAsync queried the shared scoped DbContext without taking the
per-task lock stripe the writes already use. A parallel batch runs every
target's re-attach probe on the one DbContext the Hangfire worker owns
for the task, so two concurrent probes (or a probe racing a record/clear)
drove two operations onto the same context and EF threw "a second
operation was started on this context instance" — an intermittent crash
in multi-target deployments that use crash-resume re-attach.

Gate the read with the same stripe as the writes so all of a task's
DbContext access serialises onto one writer. Adds an integration
regression test that resolves one store (one DbContext) and fires
concurrent reads+writes: red before the fix with the exact EF error,
green after.
@ppXD ppXD added this to the 1.8.18 milestone Jun 11, 2026
@ppXD
ppXD merged commit a57f758 into main Jun 11, 2026
15 checks passed
@ppXD
ppXD deleted the fix/inflight-script-store-read-concurrency branch June 11, 2026 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant