Python: fix: parse Responses function_call_output so hosted tool results reach transports

[Manjunath Janardhan (manjunathshiva)]

Motivation & Context

A Foundry hosted agent using a Foundry Toolbox dispatches its knowledge-base tool through the toolbox's generic call_tool wrapper. The Toolbox executes the inner tool server-side and ResponsesHostServer serializes the result as a standard function_call_output Responses item. None of the three parse dispatch sites in agent_framework_openai handled that item type, so it fell into case _: / the end of the elif chain, was logged as Unparsed event of type: ... at debug level, and discarded.

No Content.from_function_result(...) was produced, so agent_framework_ag_ui emitted TOOL_CALL_END with no matching TOOL_CALL_RESULT and its declaration-only fallback took over (_agent_run.py:3100-3108). The model still received the real output — only the client lost the structured result. The same tool called directly is an mcp_call and works, which is what isolates this to the client parser.

Description & Review Guide

• What are the major changes?

  A function_call_output branch at all three dispatch sites — the non-streaming _parse_response_from_openai and both streaming response.output_item.added / .done handlers — sharing one _parse_function_call_output_content helper and one _pairable_function_call_output_call_id gate, so the three item-type lists cannot drift apart on this type again. The gate returns the validated call_id rather than a bool, so the parse is handed the narrowed value instead of re-deriving one the SDK types as Optional[str].

  output is str | list[ResponseInputText|Image|File]. The list form maps to canonical Content items -- text to text, image and file URLs to from_uri with the media type inferred, file_id to from_hosted_file, inline base64 to from_data -- so a returned image stays addressable instead of arriving as JSON inside a text item. A hosted reference is kept as a reference because only the issuing provider can resolve it and text would destroy that. Parts are read field-by-field rather than by isinstance, because transports and test doubles deliver plain mappings as well as SDK models. Each part is mapped defensively: a content factory rejecting provider data degrades that part to JSON rather than raising out of a streaming parse and failing the request.

  One behaviour change to weigh: for an output with no text parts the flat result is now empty, because from_function_result derives it from the text items. A consumer reading only result and not items previously saw JSON there.

  Per-request dedup state stays off the signature. _parse_chunk_from_openai is byte-identical to the signature released agent-framework-foundry wheels override; the seen-id set travels in the parse options mapping under a private key. options there is the validated parse context, not the outbound request body -- that is built from a separate run_options dict -- so the key cannot reach the service. Two tests pin the contract: one structural on the parameter list, one behavioural through a subclass that forwards only the four arguments it knows about.

  Re-derived against the current dependency state, since that was the ask. Foundry's floor rose from agent-framework-openai>=1.10.0,<2 to >=1.14.2,<2 in Python: Bump package versions for 1.18.0 release #8222, which does not close the break: agent-framework-foundry 1.13.0 still overrides the method with the old signature, and >=1.14.2,<2 still admits a later 1.14.x carrying this change. A published constraint cannot be retracted, so the signature cannot grow. All four Foundry changes this PR previously made to accommodate a new parameter are reverted.

  The two streaming handlers defer outputs explicitly marked in_progress, including empty or partial .added placeholders, without claiming their item IDs. The first eligible output is emitted and its item ID is recorded in a per-request set so the other event cannot emit a second result. Completed empty outputs remain valid. This does not depend on a provider guarantee that empty in-progress placeholders never occur. Deduplication is keyed on item ID rather than call_id, which the function-calling loop contract says must not be assumed unique forever.

  Two follow-up commits are separated deliberately so they can be read on their own: forwarding the new parameter through the two RawFoundryChatClient / RawFoundryAgentChatClient overrides, and rejecting a blank call_id so no unpairable result is emitted.

• What is the impact of these changes?

  Hosted-toolbox tool results now reach transports as function_result content, so AG-UI emits the full TOOL_CALL_START → TOOL_CALL_END → TOOL_CALL_RESULT lifecycle. No AG-UI change was needed: _emit_tool_result already handles function_result.

  Rich-content scope: preserving image/file parts in framework Content.items and OpenAI replay does not make them visible as frontend attachments. The ordinary AG-UI result emitter reads only the text-only Content.result: mixed outputs emit their text, while image/file-only outputs emit empty result content. Parser-to-AG-UI regression cases cover this boundary. Frontend attachment display is a separate transport/frontend capability and remains outside this PR's scope.

  Reviewed against docs/specs/004-python-function-calling-loop.md, which covers provider serialization of function calls and results. No result is orphaned or duplicated, and the streaming and non-streaming paths agree. I have not edited the spec — its checklist asks that the matrix name a regression test for each affected scenario, and I would rather you decide whether this warrants a row than edit a cross-package contract in a bug fix. Glad to add one.

  Verified against a live Foundry Responses endpoint in addition to the unit tests. Two things that measurement settled:

  1. Every output item fires both .added and .done (confirmed for function_call, message, reasoning). Emitting from both handlers without the seen-id set would have produced duplicate results; emitting from only one would have been a guess about which event carries the payload.
  2. A stored function_call_output is re-sent inline on the next turn under previous_response_id, and the service accepts it. Since the outbound serializer emits only call_id / type / output and no item id, a server-generated result is indistinguishable on the wire from a locally-executed one, so this needs no outbound companion change.

• What do you want reviewers to focus on?

  Three things I could not settle myself, all raised deliberately rather than left for you to find:

  1. The file_id representation. An input_image/input_file part carrying a file_id becomes from_hosted_file rather than text. I chose addressable over readable: only the issuing provider can resolve it, but flattening it to text destroys the information entirely, whereas a reference lets a consumer decide. Say if you would rather have text and I will change it -- it is one branch.
  2. The empty result above. It follows from the shape you asked for, but it is caller-visible, so it is your call whether it needs a release note or a text summary of the non-text parts.
  3. packages/foundry is missing from spec 004's minimum validation commands (docs/specs/004-python-function-calling-loop.md), even though it subclasses the OpenAI Responses client the spec governs. That omission is exactly why my first sweep missed the regression above; uv run poe test -P foundry reproduces it. Worth adding for the next contributor.

  Separately, and not proposed here: the case _: default at all three sites drops any unknown item type at debug severity, which is why the reporter hit three such types in one session (function_call_output plus two SharePoint preview ones). Silently discarding a tool result is a correctness event rather than a diagnostic one. I deliberately kept this PR to the reproduced type — the SharePoint types are preview and I have no repro, so guessing their shape risks a wrong parser. If useful I will open a separate issue proposing either a shared dispatch table across the three sites or a warning for unknown *_output items.

Related Issue

Fixes #8068

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and the title prefix in sync automatically.

[Copilot]

🟡 Changes recommended

Supported older SDK versions can crash, and rich output parts are not serialized into usable results.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds parsing for hosted function_call_output items so tool results reach downstream transports.

Changes:

• Parses streaming and non-streaming function outputs.
• Deduplicates streaming results by item ID.
• Propagates parser state through Foundry clients and adds tests.

File summaries

File	Description
python/packages/openai/agent_framework_openai/_chat_client.py	Implements output parsing and deduplication.
python/packages/openai/tests/openai/test_openai_chat_client.py	Tests parsing scenarios.
python/packages/foundry/agent_framework_foundry/_chat_client.py	Forwards deduplication state.
python/packages/foundry/agent_framework_foundry/_agent.py	Forwards deduplication state.
python/packages/foundry/tests/foundry/test_foundry_chat_client.py	Updates delegation assertion.
python/packages/foundry/tests/foundry/test_foundry_agent.py	Updates delegation assertion.

Review details

Suppressed comments (1)

python/packages/openai/agent_framework_openai/_chat_client.py:2692

• For list output, the SDK supplies ResponseInputText/ResponseInputImage/ResponseInputFile model instances, not the dictionaries used in the new test. Text happens to work via .text, but image/file parts fall through to json.dumps(..., default=str), producing quoted Pydantic reprs (and concatenating multiple reprs) rather than preserving usable rich output. Map these provider parts to Content items, or serialize the full list with _serialize_provider_payload while retaining its boundaries before creating the function result.

        return Content.from_function_result(
            call_id=item.call_id,
            result=self._stringify_mcp_output(item.output),
            additional_properties=additional_properties,

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Manjunath Janardhan (manjunathshiva)]

Reproduced end-to-end on real Foundry infrastructure, not just in unit tests, so the before/after is observable rather than inferred.

Setup

• Azure AI Search (basic tier) with an index of three plain-text docs, one containing a distinctive sentinel string.
• A project connection (CognitiveSearch / ApiKey) to that Search service.
• A Foundry toolbox whose default version carries an azure_ai_search tool bound to that connection and index.
• A Foundry hosted agent built exactly as samples/04-hosting/foundry-hosted-agents/responses/foundry_toolbox/main.py does — Agent(client=FoundryChatClient(...), tools=FoundryToolbox(credential), default_options={"store": False}) behind ResponsesHostServer.
• Driven through agent_framework_ag_ui's AgentFrameworkAgent.run(), with an OpenAIChatClient pointed at the local host.

The host emits the item as reported

function_call        name=azure_ai_search  call_id=call_KWrwLEyAGQN83s9XyQCgC9I2
function_call_output call_id=call_KWrwLEyAGQN83s9XyQCgC9I2  output=AG-UI snapshot codename
                     The internal codename for the AG-UI thread snapshot subsystem is BLUEHERON. ...
ANSWER: The internal codename is BLUEHERON.

Matching call_id, populated output, real retrieval from the index.

AG-UI events, same request, same host, only the parser differing

Event	main	this PR
TOOL_CALL_START	1	1
TOOL_CALL_END	1	1
TOOL_CALL_RESULT	absent	1

On main the result never materializes — the reported "TOOL_CALL_END but no matching TOOL_CALL_RESULT". With the change, TOOL_CALL_RESULT carries the retrieved KB text.

Two incidental notes from the exercise, neither part of this change

1. The host serializes any hosted-agent function_result as a function_call_output output item (foundry_hosting/_responses.py, the content.type == "function_result" branch), so the gap is not specific to a toolbox or to a knowledge-base tool — a toolbox is just one way to get a server-side tool execution. That widens the blast radius of the original report.
2. The toolbox's azure_ai_search tool defaults to query_type=vector_semantic_hybrid, which fails with "requires a vector field with integrated vectorizer" against a plain-text index. Setting query_type on the index resource fixes it. Unrelated to this PR, noted only in case it saves someone else the detour.

[Manjunath Janardhan (manjunathshiva)]

Flagging a process point on myself before anyone spends review time here.

docs/specs/004-python-function-calling-loop.md has a Contribution ownership clause I missed on my first read, even though I cited the spec in the PR description:

Issues involving this code must not be picked up by external contributors without first checking with the Agent Framework core team. The core team must confirm the intended behavior, affected scenario-matrix rows, ownership across core/providers/transports, and the required validation scope before implementation starts.

Read together with "Any change to the function-calling loop or its approval/history/serialization paths", this PR is inside that scope — it changes how a provider serializer parses a function-call result — and I did not check first. That's on me.

Rather than just flag it, here is where the change stands against the seven requirements in the same section, so you can judge cheaply whether it is worth continuing or whether you would rather own it:

#	Requirement	Status
1	Identify every affected scenario-matrix row	Not done — see the ask below
2	Add or update the corresponding regression tests	13 tests across the three dispatch sites, including a stub shaped like the openai==2.25.0 field set
3	Validate streaming updates, streaming finalization, and non-streaming	All three. .added, .done, and _parse_response_from_openai each covered; confirmed the streaming finalizer aggregates already-yielded updates rather than re-parsing, so there is no double-emit path
4	Validate model-bound history and caller-visible responses	Both, on a live Foundry endpoint. Outbound, a re-sent function_call_output carries only call_id/type/output and the service accepts it under previous_response_id; caller-visible, AG-UI emits the paired TOOL_CALL_RESULT
5	Full core tests plus every affected provider/transport package	core 4321, openai 488, foundry 388, ag-ui 1149, declarative 989
6	Source typing, test typing, syntax for every affected package	openai and foundry (the two changed): pyright strict 0 errors, plus ty/pyrefly/mypy/zuban
7	Extra review on call/result pairing, exactly-once execution, history replay	Partly self-addressed — the per-request seen-id set gives exactly-once across .added/.done, and a blank-call_id guard prevents an orphaned result — but the review itself is yours to give

The ask. On requirement 1 I would rather not guess at the matrix rows unilaterally, since that is exactly the judgment the clause reserves for you. My reading is that this touches "History and provider serialization" and nothing else, but I may be wrong about whether it also implicates the streaming/non-streaming agreement rows.

So: would you prefer to

1. proceed with this PR and tell me which matrix rows to add, or
2. have me close it and pick the fix up yourselves — the diagnosis in the description and the live-repro comment above should transfer directly, and I would not consider that wasted, or
3. something else?

Happy with any of those. One note in case it affects the call: the underlying gap is not specific to a toolbox — foundry_hosting serializes any hosted-agent function_result as a function_call_output output item, so every Foundry hosted agent with a tool is affected, not just the reporter's setup.

[Evan Mattson (moonbox3)]

tagging Eduard van Valkenburg (@eavanvalkenburg) here to look at this.

[Manjunath Janardhan (manjunathshiva)]

Reworked and pushed as 2bc2a04f3. All four threads addressed; the description is updated, including
two reviewer-focus items that argued for the signature change this reverts.

Eduard van Valkenburg (@eavanvalkenburg) — "there have been updates to the dependencies now, so please have another look."
Re-derived, and the answer is that the break is still open. Foundry's floor did rise, from
agent-framework-openai>=1.10.0,<2 to >=1.14.2,<2 in #8222 — which was Evan Mattson (@moonbox3)'s own second
suggestion — but it does not close it: agent-framework-foundry 1.13.0 still overrides
_parse_chunk_from_openai with the signature ending at seen_reasoning_delta_item_ids, and
>=1.14.2,<2 still admits a later 1.14.x carrying this change. A published constraint cannot be
retracted, so the signature cannot grow.

So I took Evan Mattson (@moonbox3)'s first option. The per-request seen-id set now travels in the parse options
mapping under a private key, leaving the signature byte-identical, and all four Foundry changes
this PR made to accommodate a new parameter are reverted. One thing worth stating because it is the
reason this is safe: options there is the validated parse context, not the outbound request body
— that is built from a separate run_options dict — so a private key cannot reach the service. Two
tests pin the contract: one structural on the parameter list, one behavioural through a subclass that
forwards only the four arguments it knows about.

I did not take the .done-only variant. It would drop the dedup state entirely, but it rests on
.done always carrying a populated output, which I never verified — and I would rather not bet the
fix on an unverified provider behaviour.

Eduard van Valkenburg (@eavanvalkenburg) — duplicate serialization handling. You were right, and it was worse than
duplication: _serialize_provider_payload already did what my helper hand-rolled, and did it
properly — model_dump(mode="json", exclude_none=True) with recursion — where mine walked one level
via getattr with no mode="json". _plain_function_call_output is deleted rather than relocated;
the degrade path reuses your two existing helpers, and the mapper now lives beside them in the class
instead of being a fourth mechanism.

Eduard van Valkenburg (@eavanvalkenburg) — Any on those functions. Concrete types throughout, including a named
ResponseFunctionCallOutputPart union. Typing them is what surfaced the Package Checks failure:
call_id became Optional[str] in the SDK bump, so the gate now returns the validated call_id
and the parse is handed the narrowed value rather than re-deriving one it cannot narrow. pyright
strict is at 0 errors.

Evan Mattson (@moonbox3) — list-shaped output as canonical Content.items. Done: text parts to text, image and
file URLs to from_uri with the media type inferred, inline base64 to from_data. On the question I
asked you and then had to answer myself — a part carrying a file_id becomes from_hosted_file
rather than text. I chose addressable over readable: only the issuing provider can resolve it, but
flattening to text destroys the information outright, whereas a reference lets a consumer decide.
It is one branch if you disagree.

Two things I would rather you heard from me than found:

• For an output with no text parts, the flat result is now empty, because from_function_result
  derives it from the text items. A consumer reading only result and not items previously saw
  JSON there. It follows from the shape you asked for, but it is caller-visible.

• The part-list branch cannot be reached from this host, which is worth knowing. I went back to
  the Foundry project from the original repro to exercise it, then found in the code why that could
  never work: ResponsesHostServer emits the result as
  output_item_function_call_output(call_id, _json_safe_to_str(content.result))
  (_responses.py:1237), and _json_safe_to_str returns "", the string itself, or JSON --
  always a str. The input_text/input_image/input_file handling in the same module
  (:2003-2025) is in _convert_message_content, on the inbound request path. So MAF's own hosting
  never emits a part list in function_call_output, and no deployment would have exercised it.

  That does not make the mapping wrong -- ResponseFunctionToolCallOutputItem.output is declared
  str | list[ResponseInputText|Image|File], so another Responses producer can send parts, and that
  is the case you were pointing at.

  To be precise about what is and is not verified: no function_call_output item was produced in
  either environment I could reach, so neither branch is validated on live traffic. What is live
  is that the reworked parse handles an ordinary streaming tool loop unchanged, and the echo-back
  finding below. Both branches are validated against real ResponseFunctionToolCallOutputItem
  instances -- every shape in the declared union, all seven mapping correctly -- which is a step up
  from MagicMock but is still the declared shape rather than one I have seen on a wire.

  If you know of a producer that does emit parts here, say so and I will point the probe at it.

Live testing did settle one thing worth passing on. I flagged in the description that a locally
executed result is serialized outbound as function_call_output, so if the service echoed it back
in response.output we would emit a second result — the duplication spec-004 forbids. A real
two-round-trip streaming tool loop on gpt-5-mini sends the result in history on turn two and
nothing comes back: the only inbound items are reasoning, function_call and message. So
that path does not exist on this service. The same run also confirms the reworked parse handles
ordinary streaming tool calls unchanged.

Because the mapping now calls content factories that validate their input, each part is mapped
defensively: one that trips a factory degrades to JSON instead of raising out of a streaming parse
and failing the request. The old flattening path could not realistically raise, so that is a
robustness property the rework had to add back rather than inherit.

Validation: openai 502, foundry 391, core 5110, ag-ui 1284, declarative 1016, foundry_hosting 292;
poe check -P openai clean across all five checkers. The one thing I could not reproduce locally is
the OpenAI integration 2.x floor job — running pyright against openai 2.25 outside the package's
own config produced thousands of unrelated errors. I confirmed the part types and the Optional[str]
call_id do exist at 2.25, but that job is the verdict, and the workflows on this head are sitting
at action_required. Could you release them?

The ownership question is still open, and it is the one thing I have not been able to answer
myself. Requirement 1 in the spec-004 table — identify every affected scenario-matrix row — needs
your input, and everything else in that table is done. A one-word "continue" is enough.

[Eduard van Valkenburg (eavanvalkenburg)]

Thanks for the contribution. This branch currently conflicts with main. Could you please resolve the merge conflicts and update the PR? Once that is done, the approval-gated workflows can be run and the PR can be reviewed.

[Eduard van Valkenburg (eavanvalkenburg)]

Two remaining rich-output round-trip cases need covering before this mapping is complete. Both reproduce with real OpenAI SDK content models on this head; details inline.

[Eduard van Valkenburg (eavanvalkenburg)]

Pushed follow-up fixes in 5b6ce00 and merged current main in 8cadaa4. The merge conflict is resolved and both rich-output review threads are addressed, with 60 parse-to-replay regression cases.

The OpenAI 2.25.0 and 3.0.0 dependency-bound jobs now pass, as do code quality, source/test typing, samples/markdown, and coverage. I retried only the Windows 3.12 matrix job after an unrelated workflow-shutdown test hit its one-second timeout.

There is a separate base-workflow blocker: Public API Compatibility is stopped at checkout by the fork/pull_request_target safety guard, before the compatibility checker runs. The same checker passes locally against current main with no published API breakages. This requires a safe correction to the base workflow; I have not opted into unsafe checkout or bypassed the check. The merge gate currently reports that failure.

[Jose Alvarez (jpalvarezl)]

Two questions on the latest revision:

1. Rich results reaching AG-UI

The parser now preserves image/file parts in Content.items, but AG-UI's ordinary result emitter reads the text-only Content.result. An image-only result therefore appears to still produce an empty frontend result through that path.

Could we add a parser-to-AG-UI regression case and clarify whether displaying attachments is outside this PR's scope? That would distinguish "preserved in framework history" from "visible to the frontend." This is separate from the original text-result fix and the already-addressed OpenAI replay cases.

2. Streaming placeholders and deduplication

Can a response.output_item.added event contain status="in_progress" and output="", followed by the real output on response.output_item.done for the same item ID? The new guard and deduplication helper skip None, but accept an empty string; that would claim the item ID before the final answer arrives and suppress the later result.

Could we cover that sequence, or document the provider guarantee that rules it out? This is a contract question, not a claim that this sequence has been observed on live traffic; a genuinely completed empty result should still remain valid.

[Eduard van Valkenburg (eavanvalkenburg)]

Jose Alvarez (@jpalvarezl) addressed both points in c4194e4.

1. Added real-SDK stream-parser → ordinary AG-UI result-emitter regression cases for string results, image-only/file-only results, and mixed text/attachment results. They assert that Content.items retains the attachment while TOOL_CALL_RESULT.content contains only the text (empty for attachment-only results). Updated the README, helper documentation, and PR description to state this boundary explicitly: frontend attachment display is outside this PR; retaining framework items and OpenAI replay is not a claim that the frontend displays attachments.

2. The sequence you described reproduced with constructed SDK events. The guard now defers explicitly in_progress outputs before claiming their item ID, including empty strings, empty lists, and partial text. A subsequent completed .done emits the final result once. Completed empty results remain valid, including when first delivered on .added. Regression cases cover these combinations and repeated .done delivery. This avoids relying on an unverified provider guarantee; no live observation of that sequence is claimed.

The affected core/OpenAI/AG-UI/Foundry suites pass (7,934 tests), and the OpenAI suite plus source typing pass at both SDK 2.25.0 and 3.0.0 (603 tests each).

[Manjunath Janardhan (manjunathshiva)]

Thanks for finishing this off, Eduard van Valkenburg (@eavanvalkenburg) — and for pushing the fix rather than sending it back
for another round. Both findings were real and both were in the direction I had not tested.

I had validated the mapping thoroughly inbound: unit tests first, then real
ResponseFunctionToolCallOutputItem instances instead of MagicMock so the SDK's own validation was
in the loop. All of it parse-side. I never asked what _prepare_content_for_openai would do with
what I produced, which is exactly where both bugs were. Choosing from_hosted_file over text is the
one that stings: I defended it as "addressable beats readable", and because the outbound serializer
ignored hosted_file items it actually destroyed the reference on replay — strictly worse than the
text I had rejected. Reasoning about a mapping in one direction is not reasoning about a mapping.

I went through your commit before this merged and checked the round trips independently rather than
taking the message on trust. All five shapes survive now, and each keeps the provider's declared
type: file-only and text-plus-file retain file_id, the extensionless image_url no longer raises
ContentError, and an input_file named scan.jpg replays as input_file rather than flipping to
input_image. The openai_content_type fallback to the old media-type heuristic when the key is
absent is the part I would have missed — it keeps content that predates this change working.

Also noted that you hit the Public API Compatibility fork guard and left it alone rather than
opting into unsafe checkout. #8419 landing separately was the right resolution, and it is worth
knowing for other fork PRs.

Two takeaways I am keeping: a parse change is half a change, so round-trip it; and inference that
describes content -- the media type I derived from a filename or URL -- must never stand in for a
discriminator the provider actually sent.