Python Durable Functions on DTS: oversized orchestrator completion fails the orchestration invisibly, and large-payload externalization can't be enabled

[nachogarcia]

Environment

• Azure Functions, Python 3.14, Elastic Premium, storageProvider.type: "azureManaged" (Durable Task Scheduler)
• azure-functions-durable==2.0.0rc1 (on durabletask==1.10.1), azure-functions==2.3.0
• Durable extension 3.14.0 via Microsoft.Azure.Functions.ExtensionBundle.Preview [4.29.0, 5.0.0)

We have an instance ID and task-hub details for a production occurrence and can share them privately.

What happened

An orchestrator fanned out 121 activities; each result was 2–48 KB. When the last one completed, the orchestrator scheduled one more activity whose input was the fanned-in result list. The scheduler rejected that episode and marked the orchestration Failed:

runtimeStatus:            FAILED
failureDetails.errorType: InvalidOperationException
failureDetails.message:   The complete orchestration work item request of size 11.05MB exceeds the 3.90MB limit.
                          Enable large-payload externalization to Azure Blob Storage to support oversized completion requests.
ExecutionCompletedEvent:  orchestration_status 3, isNonRetriable: true

(Retrieved with DurableTaskSchedulerClient.get_orchestration_state(fetch_payloads=True) and get_orchestration_history().)

Problems

1. The failure never reaches the app. No exception surfaces in orchestrator code, so our try/except compensation (refund, user notification) never ran. The host and Application Insights log nothing. The last host line for the instance is Orchestrator orchestrate_process_document yielded with 1 task(s) and 0 event(s) outstanding. The failure exists only in scheduler state, and our users see a review stuck in "processing".
2. The rejected episode's triggering completion isn't in history. History has 121 TaskScheduled events but only 120 TaskCompleted. The missing completion is the activity whose result triggered the rejected episode, and the host logged that activity as Completed. We assume this follows from the work item being rejected as a unit, but it means history and host logs disagree.
3. The 3.90 MB limit isn't documented. The published DTS limits describe 1 MB per activity input/output. The cap on the orchestrator completion request, and the fact that scheduled-activity inputs count toward it, aren't documented anywhere we found. Our code had assumed the orchestrator→activity direction was safe.
4. The suggested fix isn't available to us. The error message recommends large-payload externalization. durabletask 1.10.1 has it (durabletask[azure-blob-payloads]), and its worker externalizes OrchestratorResponse payloads when given a payload_store. But azure-functions-durable 2.0.0rc1 constructs DurableFunctionsWorker and DurableFunctionsClient without one, and exposes no way to pass one. The large payload docs confirm Durable Functions support is .NET-isolated only.
5. Serialization inflates payloads for non-Latin text. The stored activity results were 2.55 MB of UTF-8 (mostly Cyrillic), but the request was 11.05 MB. azure.functions._durable_functions.df_dumps calls json.dumps with the default ensure_ascii=True, so each Cyrillic character becomes a 6-byte \uXXXX escape instead of 2 bytes of UTF-8.

Asks

1. Deliver an oversized orchestrator completion as a failure the orchestrator can catch, or at least log it on the Functions host with the instance ID.
2. Document the completion-request limit, and that scheduled-activity inputs count toward it.
3. Is large-payload support planned for azure-functions-durable (Python) on DTS? We'd be glad to contribute a PR. Our rough plan:
   • accept a PayloadStore in DFApp or via app settings, and pass it to DurableFunctionsWorker and DurableFunctionsClient. The orchestrator and entity paths then externalize through the existing durabletask hooks, which alone would have prevented the failure above.
   • in the activity_trigger wrapper, resolve token inputs before calling the user function, and externalize outputs over the threshold. The open question is how the token should survive df_dumps JSON-quoting on the output side, so that deexternalize_payloads recognizes it in TaskCompleted.result. We'd like your preference before writing that part.
4. Would you accept ensure_ascii=False in df_dumps? It's valid JSON and cuts non-Latin payloads about 3×, but it changes the bytes on the wire, so we'd rather ask first.