Python: Add durable Foundry-hosted workflow recovery#7134
Draft
alliscode wants to merge 25 commits into
Draft
Conversation
- azure-ai-agentserver-core: 2.0.0b5 -> 2.0.0b7 (resilient task primitive, event streaming API) - azure-ai-agentserver-invocations: 1.0.0b3 -> 1.0.0b6 (bug fixes, requires core >=b7) - azure-ai-agentserver-responses: 1.0.0b7 -> 1.0.0b8 (resilient background responses, steerable conversations, developer checkpoints) New versions are unreleased on PyPI; source them via direct URL to the pinned commit in azure-sdk-for-python via [tool.uv.sources]. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix resilient_background rename: ResponsesServerOptions no longer accepts durable_background (renamed to resilient_background in b8). The old name caused a silent TypeError that disabled crash recovery in hosted environments. - Add steerable_conversations parameter to ResponsesHostServer.__init__. When True, concurrent turns on the same conversation chain are queued rather than rejected with 409 conversation_locked. The flag is threaded into ResponsesServerOptions and composed with resilient_background in the auto-enable path for hosted environments. - Fix streaming cancellation terminal: both _handle_inner_agent and _handle_inner_workflow now emit response.completed before returning when cancellation_signal fires. Per spec §10, the handler must emit a terminal event; the framework overrides completed → cancelled when context.client_cancelled=True (real client cancel), and leaves it as completed for steering pressure (client_cancelled=False). Previously returning without a terminal caused the framework to force failed, which broke the steered-turn chain. - Add comment in _handle_inner_workflow explaining the relationship between previous_response_id and conversation_chain_id for steerable turns (they resolve to the same key, so the existing restore logic is already correct). - Update tests: rename resilient_background tests, update cancellation test to assert response.completed is emitted, add three new tests for steerable_conversations threading and precedence rules. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix async generator handler signature (b8 compatibility): _handle_response was async def returning AsyncIterable. The b8 SDK passes the raw handler result directly to _intercept_checkpoints which does 'async for raw in handler_iterator' — this fails on a coroutine with TypeError (caught as B8 pre-creation error), silently producing empty responses. Changed to async generator (using yield) so calling it returns an AsyncGenerator immediately iterable without awaiting. Update 01_basic sample (durable_background -> resilient_background): main.py comments and README referenced the old durable_background field name (renamed to resilient_background in b8). Updated both to use the correct API and added steerable_conversations documentation. Add steering integration test (test_steering.py): Self-contained end-to-end test using Hypercorn on a loopback port and concurrent asyncio tasks. Demonstrates the full steering flow: turn 1 streams, turn 2 arrives while turn 1 is in_progress and gets status=queued, turn 1 emits response.completed (cancellation terminal fix), turn 2 runs and completes. Requires clean ~/.agentserver state (no stale task files) on first run; uses unique UUID per test run. Also add pre-existing sample exclusions to pyrightconfig.samples.json for monty/tools samples not installed in the dev environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replaces the mock slow agent with a real FoundryChatClient + Agent using FOUNDRY_PROJECT_ENDPOINT and AZURE_AI_MODEL_DEPLOYMENT_NAME env vars, consistent with the 01_basic sample. Loads .env via python-dotenv. Asks the agent to count to 100 (produces a long streaming response), then steers with a joke request after 2s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
resilient_background=True was previously auto-enabled when running in a
hosted Foundry environment. This is now opt-in: callers must pass it
explicitly via ResponsesServerOptions.
server = ResponsesHostServer(
agent,
options=ResponsesServerOptions(resilient_background=True),
)
Reasoning: auto-enabling crash recovery silently can cause unexpected
behaviour for agents with non-idempotent side effects. Requiring explicit
opt-in makes the behaviour transparent and intentional.
Updates:
- _responses.py: remove auto-enable logic and AgentConfig check
- test_responses.py: replace auto-enable tests with a single test
asserting resilient_background is never auto-enabled
- 01_basic/main.py: rewrite comment to describe opt-in pattern
- 01_basic/README.md: rewrite durability section as opt-in
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add client.py — a two-terminal steering demonstration. Run main.py in one terminal and client.py in a second to see steering in action with a real Foundry agent: - Turn 1 streams the agent counting to 50 - 2 s later, turn 2 arrives on the same conversation with a different question - Turn 2 is accepted with status=queued (not HTTP 409) - Turn 1's handler is cancelled and emits response.completed with partial output - Turn 2 runs and its answer is printed client.py accepts an optional SERVER_URL argument so it can be pointed at a deployed instance as well as localhost. Also enable steerable_conversations=True in main.py and update README with the two-terminal usage instructions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- 01_basic: restore to minimal server-only sample; remove client.py, test_steering.py, and all steering/resilience content from README. Add brief 'Next steps' links to 13 and 14. - 13_steering: new sample demonstrating steerable_conversations=True. main.py starts the server; client.py is an interactive two-terminal demo that sends two concurrent turns and shows the queued/steered flow. - 14_resilience: new sample demonstrating resilient_background=True. main.py starts the server with crash recovery enabled. README walks through testing crash recovery locally (kill the server mid-response, restart, and poll the response to see it recover). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ence The previous README asked users to Ctrl+C the server at the right moment while counting to 1000, which was unreliable and the model often refused the prompt anyway. demo.py orchestrates everything automatically: 1. Starts main.py as a subprocess 2. Sends a background request (returns in_progress immediately) 3. Kills the server after 2s (no timing pressure) 4. Restarts the server (recovery scanner fires) 5. Polls until the response completes and prints it Uses a substantive prose prompt (explain how the internet works) that reliably generates a long response without model refusals. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Switch agentserver wheel sources from GitHub raw URLs to find-links at local long-running-agents-private-preview checkout; remove [tool.uv.sources] URL entries from python/pyproject.toml - Re-lock uv.lock for find-links path sources - Guard tracker.close() in cancellation path to avoid ValueError when the SDK state machine is in not_started state - Fix test assertions to access _ContextAwareCheckpointStorage._inner.storage_path instead of .storage_path (which is not exposed directly on the wrapper) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
Store hosted workflow checkpoints and approval state in the persistent session home, and add a deployable resilience sample that proves recovery across container crashes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
…ke the point better.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands the Python Foundry-hosting experience by adding new “responses” samples (steerable conversations and durable workflow recovery) and by enhancing ResponsesHostServer to better support resilient background execution, workflow checkpoint persistence/recovery, and updated AgentServer/Responses SDK behaviors (notably around streaming and MCP output item shapes).
Changes:
- Add new samples under
python/samples/04-hosting/foundry-hosted-agents/responses/13_steering,15_workflow_resilience, and16_foundry_workflow_resilience(including local run + deployment guidance and helper tooling). - Update
agent_framework_foundry_hosting/_responses.pyto seed/persist exact workflow checkpoint references, support recovery semantics, and adjust streaming/output-item tracking (including MCP call/result identity handling). - Update
python/packages/foundry_hosting/tests/test_responses.pyto reflect the new streaming-first handling and the updated MCP / checkpoint / recovery behavior.
Show a summary per file
| File | Description |
|---|---|
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/wheelhouse/.gitignore | Ignore copied private preview wheels in the wheelhouse build context. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/requirements.txt | Install required private preview wheels by local file path plus public dependencies. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/main.py | Implement a multi-stage translation workflow configured for resilient background execution. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/Dockerfile | Containerize the resilient translation workflow and validate the preview API is present. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/agent.yaml | Define an Agent Manifest for Foundry hosting (responses protocol). |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/.env.example | Document required/optional environment variables for the sample. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/.dockerignore | Avoid bundling local env/build artifacts into container builds. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/.azdignore | Ensure wheels are included in azd packaging while ignoring local artifacts. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/src/resilient-translation-workflow/.agentignore | Ensure wheels are included in Foundry code deployment packages. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/README.md | Document the resilient translation workflow scenario and deployment steps. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/prepare_wheels.py | Copy and validate private preview wheels for the sample build context. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/durability_client.py | Provide a client that creates/polls a stored background response through host replacements. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/azure.yaml | Define an azd project for provisioning + deploying the hosted resilient workflow. |
| python/samples/04-hosting/foundry-hosted-agents/responses/16_foundry_workflow_resilience/.gitignore | Ignore locally-generated durability state file. |
| python/samples/04-hosting/foundry-hosted-agents/responses/15_workflow_resilience/requirements.txt | Update sample requirements and explain private preview wheel install expectations. |
| python/samples/04-hosting/foundry-hosted-agents/responses/15_workflow_resilience/README.md | Add end-to-end documentation for durable workflow recovery and preview setup. |
| python/samples/04-hosting/foundry-hosted-agents/responses/15_workflow_resilience/main.py | Implement deterministic multi-stage workflow used to demonstrate durability and recovery. |
| python/samples/04-hosting/foundry-hosted-agents/responses/15_workflow_resilience/Dockerfile | Containerize the durability demo server. |
| python/samples/04-hosting/foundry-hosted-agents/responses/15_workflow_resilience/demo.py | Provide a local, automated interruption/recovery demonstration harness. |
| python/samples/04-hosting/foundry-hosted-agents/responses/15_workflow_resilience/.env.example | Document optional env vars for the durability demo server. |
| python/samples/04-hosting/foundry-hosted-agents/responses/13_steering/requirements.txt | Add sample dependencies for steerable conversations demo. |
| python/samples/04-hosting/foundry-hosted-agents/responses/13_steering/README.md | Document steerable conversation behavior and how to run the demo. |
| python/samples/04-hosting/foundry-hosted-agents/responses/13_steering/main.py | Implement a server configured with steerable_conversations=True. |
| python/samples/04-hosting/foundry-hosted-agents/responses/13_steering/Dockerfile | Containerize the steering demo server. |
| python/samples/04-hosting/foundry-hosted-agents/responses/13_steering/client.py | Provide an interactive client that demonstrates steering with concurrent turns. |
| python/samples/04-hosting/foundry-hosted-agents/responses/13_steering/.env.example | Provide env var placeholders for the steering sample. |
| python/samples/04-hosting/foundry-hosted-agents/responses/01_basic/README.md | Link to the new steering and durability samples as “Next steps”. |
| python/pyrightconfig.samples.json | Expand sample exclusions for pyright typechecking. |
| python/packages/foundry_hosting/tests/test_responses.py | Align tests with updated streaming, checkpointing, recovery, and MCP output behavior. |
| python/packages/foundry_hosting/agent_framework_foundry_hosting/_responses.py | Implement resilient background support enhancements, checkpoint reference persistence, and output tracking updates. |
Review details
- Files reviewed: 30/31 changed files
- Comments generated: 3
- Review effort level: Low
Comment on lines
+858
to
+861
| if are_options_set and not isinstance(self._agent, RawAgent): | ||
| logger.warning("Agent doesn't support runtime options. They will be ignored.") | ||
| else: | ||
| run_kwargs["options"] = chat_options |
Member
Author
There was a problem hiding this comment.
This is a valid concern, but the options behavior predates this changeset and is unchanged by the durability work. Per our review discussion, I am intentionally leaving it out of this PR to avoid an unrelated behavior change.
Contributor
There was a problem hiding this comment.
Automated Code Review
Reviewers: 5 | Confidence: 74%
✓ Correctness
No actionable issues found in this dimension.
✓ Security Reliability
No actionable issues found in this dimension.
✓ Test Coverage
This PR introduces substantial new functionality for workflow checkpoint-based crash recovery, cancellation handling, and MCP item ID tracking. The test coverage is generally quite thorough — new classes (_ContextAwareCheckpointStorage, _WorkflowCheckpointPublisher, _WorkflowCheckpointExecution) and major code paths (recovery, cancellation, multi-turn checkpoint seding) all have dedicated tests. The primary test coverage gap is that text_reasoning content with text=None is now silently dropped (previously fell back to empty string), and this behavioral change has no dedicated test. Minor: the _delete_checkpoints_except method is only tested indirectly via _prepare_workflow_checkpoint_execution.
✓ Failure Modes
No actionable issues found in this dimension.
✓ Design Approach
I found one blocking design regression in the hosted response conversion path: the new reasoning handling drops valid
text_reasoningcontent when it carries only protected/encrypted state and no display text, which breaks the repo’s existing reasoning-state continuity contract. I found two design-level issues in the new samples. The local workflow-resilience Dockerfile is wired to install package names even though the sample’s own docs and code require private preview wheels forresilient_background, so the built image can resolve the wrong AgentServer artifacts and fail at startup. I also found an internal protocol-version split in the Foundry resilience sample: itsagent.yamladvertisesresponsesv1while the same sample’sazure.yamland the repo’s other Responses manifests use2.0.0, so local/code-deployment runs exercise a different contract than provisioned deployments. I found one design issue in the new resilient translation sample: its crash markers are persisted globally per stage, so the durability demo only injects failures the first time a session home sees each stage. After one successful run, later responses in the same hosted session silently skip the intended restart path and no longer exercise the resilience behavior the sample is meant to prove.
Automated review by alliscode's agents
alliscode
commented
Jul 15, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1674d491-af7d-4f9e-99aa-5a829db81591
Member
Author
|
The remaining automated review-summary items are also addressed in b9f23c7: |
Comment on lines
+81
to
+85
| requirements = destination / GENERATED_REQUIREMENTS | ||
| requirements.write_text( | ||
| "".join(f"./wheelhouse/{source.name}\n" for source in sources), | ||
| encoding="utf-8", | ||
| ) |
Comment on lines
+894
to
+908
| async for update in self._agent.run(stream=True, **run_kwargs): # type: ignore[reportUnknownMemberType] | ||
| for content in update.contents: | ||
| for event in tracker.handle(content): | ||
| yield event | ||
| if tracker.needs_async: | ||
| async for item in _to_outputs( | ||
| response_event_stream, | ||
| content, | ||
| approval_storage=approval_storage, | ||
| call_id_override=tracker.take_async_call_id_override(), | ||
| ): | ||
| yield item | ||
| tracker.needs_async = False | ||
| if cancellation_signal.is_set() or context.shutdown.is_set(): | ||
| return |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation & Context
Foundry Hosted Agents can restart a running background response on a replacement host, while Agent Framework workflows checkpoint at internal superstep boundaries. Without explicit synchronization, the response can recover from a checkpoint that does not match the workflow's durable state, lose emitted outputs, or persist state outside the hosted session's durable filesystem.
This change integrates workflow checkpointing with response-level durability so one stored background response can continue correctly across multiple host crashes and restarts.
Description & Review Guide
Related Issue
Fixes #7136
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.