Skip to content

fix: align long durable timers and cancellation with Python - #366

Merged
wangbill (YunchuWang) merged 7 commits into
mainfrom
yunchuwang-long-timer-support
Sep 21, 2026
Merged

wangbill (YunchuWang) merged 7 commits into
mainfrom
yunchuwang-long-timer-support

Conversation

@YunchuWang

@YunchuWang wangbill (YunchuWang) commented Sep 11, 2026 •

Copy link
Copy Markdown
Member

Summary

What changed?

Align durable timer behavior with microsoft/durabletask-python at a6c18f24, without adding a provider configuration framework.

Contract Implementation
Generic core / test workers Three-day backend segments by default, including retry delays
Core interval override maximumTimerIntervalMs; null, zero, or negative disables segmentation
AzureManaged builder Explicit null, matching Python's native DTS timers
Functions worker / runOrchestrator Inherits the core three-day default; no Functions interval setting
Replay Same logical timer, fresh backend IDs, recorded TimerFired.fireAt, normal completion only at the final deadline
Cancellation Handler first, canceled + complete but not failed, parent notification; true once, false when already terminal
Timer results getResult() and result throw TaskCancelledError when canceled; timer result also throws while pending/failed

isCanceled retains its existing JavaScript spelling. whenAny can return a canceled timer as its winner. whenAll counts cancellation as terminal and can throw while collecting final child results. Executor resumption now reads getResult() rather than the raw result field, including for custom Task subclasses; such accessors must be replay-safe.

Representation difference: Python timedelta supports microseconds; JavaScript Date supports milliseconds. Positive fractional intervals round up to whole milliseconds (minimum 1 ms). Non-finite values are rejected; no positive-only/safe-integer restriction is imposed.

No total business-timer duration cap, app setup API, provider boolean hooks, private Functions test-worker subclass, or automatic backend detection. Functions connected to DTS still segments. The separate Node timeout re-arm change was removed; native in-memory timers retain the pre-existing Node timeout limit when segmentation is disabled.

Why is this change needed?

The Functions protobuf/gRPC route bypasses legacy host-side long-timer splitting. A single 21/30-day timer can exceed Azure Storage's seven-day queue visibility-delay limit. SDK segments address that constraint without changing the logical deadline or task identity.

Regression found and fixed during adversarial audit

This workflow produced a false success in the initial implementation:

const timer = ctx.createTimer(deadline);
const all = whenAll([timer]);
try { timer.cancel(); } catch (error) {
  if (!(error instanceof TaskCancelledError)) throw error;
}
const result = yield all; // Previously resumed successfully with undefined.

Direct execution confirmed: baseline remained pending, the initial PR completed with "FALSE_SUCCESS", and pinned Python rejected its uninitialized result. The fix is a nine-line WhenAllTask.getResult() guard: perform existing pending/failure checks first, then reject an uninitialized result. Valid arrays, including [] and [undefined], remain valid. It preserves the original synchronous cancellation exception, timer state, failure aggregation, and parent-notification behavior. Three focused assertions were observed failing before the fix and passing afterward.

The inherited exceptional-parent limitation remains: after a completion callback throws, do not catch it and reuse that group or its parents. A group may be marked complete before its result/parent notification finishes. The guard prevents the false-success read; it does not resume missing notifications or introduce a cancellation framework. Python also leaves the outer parent pending on this path; exact post-exception error types are not claimed identical.

Issues / work items

Related: Azure/azure-functions-durable-extension#1910. This addresses the JavaScript provider gap, not the entire cross-language umbrella issue.

Minimal CI bootstrap repair

Run 34900479259 failed before tests because durabletask-go#158 removed the root sidecar executable. The workflow install and matching local reproduction comment now pin 3fe35d93fe1d2bdab21a3d85c14867532adef0b0, the exact revision from the last passing run. No assertions or skips were changed to resolve that infrastructure failure.


Project checklist

  • Release notes added to the directly affected package changelogs
  • Backport is not required
  • Timer, replay, provider, and confirmed regression tests added
  • Intentional breaking changes and limitations documented

Migration: core timers now default to three-day segments and canceled timers are terminal. Timer result no longer silently returns undefined while pending/canceled/failed. Inspect isCanceled before reading a race winner. Explicit result reads in orchestration code can catch TaskCancelledError; yielding a canceled task fails through the executor's result-read boundary.

Old single native timers replay at their recorded deadlines in the tested histories, including overlapping timers and interleaved activity/entity IDs. This is not a universal migration guarantee: changed cancellation branching, mixed versions, or changing intervals can affect replay. Drain affected instances or use a new task hub before rollback, interval changes, or switching providers. Never replay already-segmented histories with native timers.


AI-assisted code disclosure (required)

Was an AI tool used? (select one)

  • No

  • Yes, AI helped write parts of this PR

  • Yes, an AI agent generated most of this PR

  • Tool: GitHub Copilot App / CLI

  • Areas: implementation, regressions, directly related docs/changelogs, and the two-file CI bootstrap repair

  • Revisions: aligned defaults/cancellation to pinned Python; removed provider-only policy machinery and unrelated Node timeout work; reproduced and fixed the caught-cancellation false-success regression.

Agent verification, separate from author review:

  • Read and executed relevant pinned Python task/resume code
  • Compared baseline/head outputs for unchanged workflows
  • Exercised replay, interval limits, cancellation, retry, lifecycle, and custom result accessors
  • Verified actual local Functions host/Azurite history
  • Ran full unit suites, builds, lint, and fresh CI

Testing

No real Azure cloud resources or production services were provisioned or tested. Local Functions host/Azurite and Docker DTS emulator runs are not Azure-cloud validation, and no test waited 30 days.

Fresh CI on 7f2a478edfe1374f4cdaafb092a930a1affd0cc7

Validation Result / evidence
Full lint and workspace units 35635847905: 111 suites / 1,987 tests passed
Go-sidecar E2E, Node 22 and 24 Same run: each 13 passed / 1 existing skip
Real Functions Core Tools host + Azurite 35635848070: 14 suites / 42 passed / 10 existing skips
Docker DTS emulator, Node 22 and 24 35635847829: all 10 matrix jobs passed
Samples 35635847776: passed

Local audit

  • npm run test:unit: 111 suites / 1,987 passed after the guard fix (core 1,519; Functions 175; AzureManaged 227; history export 66).
  • npm run lint, git diff --check, commit hooks, and all three package builds passed.
  • npm run build:core && npm run build:azuremanaged && npm run build -w durable-functions.
  • Actual local Functions Core Tools 4.10.0 + Azurite 3.37.0 / Node 24.14.0: full host suite 14 suites / 42 passed / 10 existing skips. Owned host/emulator processes and temporary storage were cleaned up.

Long timer host proof

TimeoutOrchestrator is exercised with 2-second, 10-second, and 30-day timeouts. For 30 days, the test queries actual host history and verifies exactly one TimerCreated at orchestration-start + 3 days, no TimerFired, the preserved logical deadline at + 30 days, and isCanceled=true / isCompleted=true after the roughly five-second activity wins. This verifies initial long-timer scheduling and cancellation against the Azure Storage host path, not a full multi-day wait.

Additional adversarial regressions

long-timer-replay-regressions.spec.ts contains 33 cases, including:

  • preserves overlapping timer/activity/entity IDs through wakeups and nested composites;
  • cancellation in either same-batch event order, both intermediate and final segments;
  • three-day boundary ±1 ms, Date limits, overflow rejection, UTC offsets, and monotonic final deadlines;
  • nested/shared timer races, failed siblings, and the exact caught-cancellation false-success workflow;
  • suspended timer firing, continue-as-new version/carryover/ID reset;
  • delayed retry firing, exhausted retries, stale segment events, and remaining retry timeout;
  • custom result accessor exceptions and completed-task reuse.

The narrow guard also has explicit pending, valid [undefined] (before/after construction), empty-array, and ordinary failure coverage.

Eight baseline/head workflows produced byte-identical protobuf actions without changing the primary checkout: short timer + activity; overlapping native timers; caught activity failure; policy retry; nested composite success; canceled losing timeout followed by an activity; suspend/resume of a short timer; continue-as-new carryover/version. These checks provide evidence for the exercised flows, not a proof of zero regressions.


Notes for reviewers

Pinned references: task/cancellation/composites, timer segmentation, AzureManaged native override, Functions inherited default.

App registration, the in-memory backend, base Task, and WhenAny implementations remain unchanged. WhenAll has only the result-read guard described above. No npm dependency/protocol/generated-source changes; upstream history streaming and response cancellation from #363 remain intact.

Add configurable deterministic timer segments while preserving logical task identity and cancellation. Enable safe Functions defaults, retain native DTS timers, and bound in-memory Node timeouts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 11, 2026 18:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Broad cross-package behavior changes and missing live backend validation require final human review.

Pull request overview

Adds backend-aware segmentation for long durable timers and retry delays across core, Functions, testing, and Azure Managed integrations.

Changes:

  • Adds configurable timer intervals and provider-specific defaults.
  • Preserves replay, cancellation, and logical task identity across segments.
  • Bounds native in-memory timeouts and adds tests, documentation, and changelogs.
File summaries
File Description
README.md Documents timer policies and rollout guidance.
packages/durabletask-js/test/long-timers.spec.ts Tests segmented timers, retries, cancellation, replay, and validation.
packages/durabletask-js/test/in-memory-long-timers.spec.ts Tests bounded native timeouts.
packages/durabletask-js/src/worker/task-hub-grpc-worker.ts Propagates timer options to execution.
packages/durabletask-js/src/worker/runtime-orchestration-context.ts Schedules timer and retry segments.
packages/durabletask-js/src/worker/orchestration-executor.ts Advances segments during replay.
packages/durabletask-js/src/worker/durable-timer-options.ts Defines and validates timer configuration.
packages/durabletask-js/src/testing/test-worker.ts Supports timer options in test workers.
packages/durabletask-js/src/testing/in-memory-backend.ts Re-arms oversized native timeouts.
packages/durabletask-js/src/index.ts Exports timer options.
packages/durabletask-js-azuremanaged/test/unit/worker-builder.spec.ts Verifies native DTS timers.
packages/durabletask-js-azuremanaged/src/worker-builder.ts Retains native timers for DTS.
packages/durabletask-js-azuremanaged/CHANGELOG.md Documents Azure Managed behavior.
packages/azure-functions-durable/test/unit/worker.spec.ts Tests Functions timer policies.
packages/azure-functions-durable/test/unit/testing.spec.ts Tests testing-helper timer propagation.
packages/azure-functions-durable/test/unit/app.spec.ts Tests app setup and registration behavior.
packages/azure-functions-durable/src/worker.ts Applies the Functions timer default.
packages/azure-functions-durable/src/testing/index.ts Propagates timer options to test runs.
packages/azure-functions-durable/src/app.ts Adds configurable shared-worker setup.
packages/azure-functions-durable/README.md Documents Functions timer configuration.
packages/azure-functions-durable/CHANGELOG.md Records provider timer changes.
CHANGELOG.md Records core timer and timeout changes.
Review details
  • Files reviewed: 22/22 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Retain both timer and history-streaming release notes and preserve upstream worker shutdown behavior. Add streamed-history coverage for deterministic timer segments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecc7780b-7549-48d5-99df-42ba05748db8
Use fixed three-day segments in Functions and its test helper through protected provider hooks. Remove public timer configuration and restore eager app registration while preserving native core/DTS behavior and replay semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecc7780b-7549-48d5-99df-42ba05748db8
@YunchuWang wangbill (YunchuWang) changed the title fix: add backend-aware long durable timer support fix: automatically split long Functions durable timers Sep 14, 2026
Default core and test workers to three-day segments, support the Python-equivalent interval override, and keep AzureManaged native. Make timer cancellation terminal with boolean return and TaskCancelledError results. Remove provider hooks and the separate Node timeout re-arm change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecc7780b-7549-48d5-99df-42ba05748db8
@YunchuWang wangbill (YunchuWang) changed the title fix: automatically split long Functions durable timers fix: align long durable timers and cancellation with Python Sep 14, 2026
Use the exact durabletask-go revision from the last successful Node 22/24 E2E run. Upstream main removed the root sidecar executable, breaking bootstrap before tests could start.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecc7780b-7549-48d5-99df-42ba05748db8
Cover interleaved legacy histories, boundaries, cancellation ordering, nested composites, lifecycle and retry replay. Assert a 30-day logical timer produces a three-day Azure Storage host history segment and cancels on activity completion. Document exceptional composite reuse and custom result-accessor limits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecc7780b-7549-48d5-99df-42ba05748db8
Prevent a caught timer-cancellation callback followed by yielding whenAll from completing with an undefined result. Preserve pending/failure checks, synchronous callback errors, normal result arrays, and existing parent notification semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecc7780b-7549-48d5-99df-42ba05748db8
@YunchuWang
wangbill (YunchuWang) merged commit ecddf16 into main Sep 21, 2026
31 checks passed
@YunchuWang
wangbill (YunchuWang) deleted the yunchuwang-long-timer-support branch September 21, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants