Skip to content

stop forced coroutine unloading from emitting durable work - #164

Merged
Tomer Rosenthal (torosent) merged 1 commit into
mainfrom
torosent-coroutine-teardown-safety
Sep 29, 2026
Merged

Tomer Rosenthal (torosent) merged 1 commit into
mainfrom
torosent-coroutine-teardown-safety

Conversation

@torosent

@torosent Tomer Rosenthal (torosent) commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

The problem

The SDK stops waiting coroutines between execution turns and when an orchestration ends. Go runs their defer functions during that shutdown—even when the workflow has not finished normally.

For example, a child coroutine waiting for an event could have deferred a cleanup activity. When the parent finished, shutting down the child could schedule that activity after the orchestration was already marked complete. Defers could also overwrite custom status or consume events that should be kept for the next execution.

We reproduced the extra activity schedule on both the emulator and live DTS. Both accepted and stored it after completion. We did not observe activity dispatch during the test window; that does not prove it can never happen.

The fix

  • Stop durable SDK calls during shutdown with the existing ErrTaskBlocked signal, before they change workflow state.
  • Suppress ctx.Logger() output during shutdown.
  • Stop long timers from scheduling another chunk after the orchestration finishes.

What this means for cleanup

Normal-return defers still work: they can schedule and await durable work, including across replay.

During forced shutdown, a durable call stops the rest of that deferred function. Other defers, such as metrics or span.End, still run on each unload.

Required cleanup must finish before the parent returns. Cancel the child's work, keep its cleanup in an uncanceled scope, and wait for the child to finish. The README includes this pattern.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@torosent
Tomer Rosenthal (torosent) marked this pull request as ready for review September 29, 2026 15:21
Copilot AI balanced review requested due to automatic review settings September 29, 2026 15:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The teardown guards are consistently applied and backed by comprehensive unit and DTS integration coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Prevents forced coroutine unloading from emitting durable work or logs while preserving normal deferred cleanup behavior.

Changes:

  • Guards durable APIs during forced unwind with ErrTaskBlocked.
  • Prevents late long-timer chunks and suppresses unwind logging.
  • Adds unit/E2E coverage and cleanup guidance.
File Description
README.md Documents deferred cleanup patterns.
CHANGELOG.md Records teardown fixes.
task/​context.go Suppresses unwind logging.
task/​eventchannel.go Guards event-channel operations.
task/​orchestrator.go Adds unwind guards across durable APIs.
task/​scheduler.go Tracks forced stack unwinding.
task/​select.go Guards selection during teardown.
task/​select_test.go Supports executor options in tests.
task/​task.go Guards task creation and awaiting.
task/​teardown_test.go Tests teardown behavior comprehensively.
task/​waitgroup.go Guards waits and documents cancellation.
tests/​durabletaskscheduler/​dts_teardown_test.go Verifies behavior against DTS.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@torosent
Tomer Rosenthal (torosent) merged commit 623a1cb into main Sep 29, 2026
9 checks passed
@torosent
Tomer Rosenthal (torosent) deleted the torosent-coroutine-teardown-safety branch September 29, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants