fix(realtime): support externally transcribed user turns

[TonyG-FWE]

Fixes #5408

Summary

Realtime sessions using external VAD/STT could let provider audio and finalized STT independently represent the same user turn. This caused duplicate input, dropped turns, incorrect interruption behavior, and lifecycle races.

This PR makes turn ownership explicit through TurnHandlingOptions.realtime_input_mode:

Mode	Turn owner	Provider input
"audio" (default)	Realtime provider	Raw audio, preserving existing behavior
"text"	External STT pipeline	Finalized text exactly once; raw audio is not submitted

Text mode applies on_user_turn_completed edits before submitting the finalized message.

Design

• An immutable resolved policy determines turn detection, input ownership, interruption ownership, and empty-transcript handling before live state changes.
• Framework-owned audio uses FIFO turn transactions from capture through endpointing, provider submission, generation, cancellation, and cleanup.
• Provider-owned turns remain outside framework transaction and deferred-input machinery.
• Finalized-message synchronization distinguishes accepted, rejected, and unknown outcomes without relying on provider chat-history equality.
• Google input sequences, provider epochs, deferred media, reconnects, and tool-result replay retain explicit ownership.
• Shutdown settles pending commit and end-of-turn work without swallowing caller cancellation or recording a trailing transcript twice.
• Fallback replacement avoids holding synchronization locks across interruption, child closure, or provider setup.

The default remains backward compatible. Provider-specific types and provider-name checks are not introduced into AgentActivity.

Review guide

Suggested review order:

1. Resolved policy and framework turn ownership in turn.py, agent_activity.py, and audio_recognition.py.
2. Finalized-message and fallback synchronization in llm/realtime.py and realtime_fallback_adapter.py.
3. Google provider lifecycle and reconnect ownership in realtime_api.py.
4. External-input, close, fallback, and provider regression tests.

Validation

• GitHub Linux unit suite: 2,400 passed, 5 skipped
• Focused close, external-input, fallback, and mocked Google/OpenAI/xAI suites passed
• Ruff formatting and lint passed
• Python 3.10 and 3.13 typing passed
• BlockGuard passed on Ubuntu, macOS, and Windows
• Release gate and CLA passed

The local Windows full-unit run encountered the existing closed-event-loop teardown cascade; GitHub Linux completed the authoritative suite.

[CLAassistant]

All committers have signed the CLA.

[TonyG-FWE]

@codex review

[TonyG-FWE]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 90f92f31db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[TonyG-FWE]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 9dcf6ebf94

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[TonyG-FWE]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 8c249b8ce6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[TonyG-FWE]

Fresh Devin shutdown finding on 744b9370c was split by lifecycle evidence:

• Confirmed: an empty realtime-audio candidate could finish its hook after close blocked new work and then be appended as ChatMessage(content=[""]). test_close_does_not_commit_empty_bounded_realtime_audio_turn in 4dfcadf09 fails on the untouched reviewed head and also verifies no generation starts and owned audio is cleared.
• Not reproducible under valid ownership: the duplicate-provider-transcript half. Server-detected and manually submitted audio turns exit before this callback path. In the close race, provider audio is still unsubmitted; the existing regression verifies commit_audio_calls == 0, so retaining non-empty externally bounded text prevents data loss rather than duplicating a provider item.
• Fix: a53327677 routes every shutdown-bounded local commit through one idempotent helper that rejects empty/whitespace candidates while retaining a non-empty unsubmitted turn exactly once.

Post-fix evidence: the close module passes 9 tests; close + external-input + reply-context + AgentSession pass 189 tests; full Ruff and strict package typing pass.

[TonyG-FWE]

Fresh Devin follow-up (e890e9dc2)

I rechecked all 2 bug cards and 12 investigation flags against the exact prior head before changing production code.

Confirmed and fixed:

• Provider-owned shutdown duplication: a server-detected audio turn could be followed by a second, externally recognized item because close-time local retention ran before the provider-ownership check. The fail-before regression in 6ade6f5d8 produced two user items; e890e9dc2 now keeps the provider-owned item only. The complete close/fallback/late-final set is 59/59.
• Zero-cooldown fallback cycling: a replacement failure could immediately re-enable and retry the same model, bypassing a later healthy backup. The fail-before regression in 6ade6f5d8 landed back on model 0; 479d228c0 attempts each model once per swap and lands on model 2.

Not production defects:

• The late-final/empty-bounce card does not occur through legal recognition events. A VAD late final replaces the pending bounce; an STT final synchronously schedules its replacement before the old bounce can clean up. Both lifecycle tests pass.
• Mixed Gemini user/tool updates exclude function-call items before computing the trailing provider turn; fresh-session tool results are intentionally rejected, while resumable restarts retain FIFO ownership.
• Complete deferred audio is intentionally retained; capping it would truncate a genuine next turn. Swap-time audio is intentionally dropped rather than replayed with permanent latency.
• The STT reconnect remains the guarded segment-boundary compatibility workaround. The id(event) and replay-exclusion observations remain bounded implementation debt, not reproduced lifecycle failures.
• Preemptive realtime generation remains excluded by the existing llm.LLM guard. Exhausted fallback replacement emits a terminal error and immediately starts AgentSession close; no continuing child is selected.

Validation on this head: 260 affected ownership tests passed (the two remaining AudioRecognition failures reproduce on exact upstream 49bfd8b31), 187 hermetic provider tests passed, Ruff is clean across 943 files, and Python 3.13 mypy is clean across 206 files. GitHub Linux CI is now authoritative for the full unit and Python 3.10 lanes.

[TonyG-FWE]

Final Devin ownership-boundary follow-up (946be984e)

The remaining findings were confirmed with deterministic fail-before coverage in 7ec384a51 and fixed without changing the established turn-policy, FIFO transaction, reconnect, or replay architecture:

• 1245db908 delegates explicit clear/activity controls to provider-owned sessions without creating framework ownership, and restores per-reply interruption overrides when the session default is disabled.
• 946be984e observes provider-close failures that outlive caller-cancelled teardown and emits one terminal error for an unexpected automatic fallback-swap failure while preserving normal cancellation and explicitly awaited restart() propagation.

Post-fix evidence: all five exact regressions pass; the four directly affected modules pass 145/145; Google realtime passes 120/120; OpenAI/xAI realtime passes 62/62. GitHub is green with 2,228 unit tests passed and 5 skipped, native Python 3.10/3.13 typing, Ruff, BlockGuard on all three operating systems, aggregate dumps, release gate, and CLA.

The six documented compatibility/debt flags listed in the PR description remain unchanged because no focused lifecycle reproduction established a defect.

[TonyG-FWE]

Integrated upstream 53c77a6b3 via merge commit cd2abdd3c.

The conflict resolution composes upstream's cancellation isolation, Gemini generation timestamps, and lk.pii.* log classification with this branch's bounded-close, FIFO turn, provider-epoch, reconnect, quarantine, and replay ownership. In particular, a cancelled predecessor no longer cancels its successor, non-cancellation failures remain observable, and both interim/final Gemini input transcripts retain the generation timestamp.

Validation: focused conflict/close/telemetry 47 passed; recognition/external-input 171 passed; session/fallback/close 192 passed; Google realtime 121 passed; OpenAI/xAI realtime 58 passed; Ruff and core Python 3.13/Linux typing passed. GitHub Linux unit: 2,299 passed, 5 skipped; Ruff, Python 3.10/3.13 typing, BlockGuard on all three platforms, release gate, and CLA are green.

The full local Windows run ended at 4 failed, 1,886 passed, 7 skipped, 11 errors: one exact-upstream scheduling assertion, three missing optional-Rime imports (the Rime source run passed 3/3), two sandbox temp-directory errors, then the known closed-event-loop cascade.

[TonyG-FWE]

Resolved the two upstream conflicts by merging da6af86ac in 568feba5e. The composition preserves this PR's turn-ownership and shutdown architecture while adding serialized inline-task handoffs, correct tool/away state transitions, a single speech-end notification after interruption, and OpenAI capability synchronization alongside transcription timestamps.

Validation: focused conflict suites 157 passed; adjacent ownership/session suites 217 passed, 44 deselected; close/fallback/provider suites 268 passed; Rime/PII assertions 7 passed; Ruff format/lint and Linux-platform Python 3.13 mypy passed. GitHub Linux unit: 2,326 passed, 5 skipped, 32 warnings; Python 3.10/3.13 typing, Ruff, BlockGuard on Ubuntu/macOS/Windows, release gate, CLA, and the automatic Devin status all passed. The Windows full-unit run reached 1,913 passed, 7 skipped before the known event-loop/environment cascade (6 failed, 9 errors); the relevant failures reproduced on exact upstream.

[TonyG-FWE]

Integrated upstream 08b8dfb13 via merge commit 685de9104.

The conflict resolution composes #6962's still-generating-response cancellation with this branch's exact generation/turn ownership. On the untouched branch, upstream's six-scenario regression had 2 failures and a leaked-task teardown error; the resolved module passes 7/7, including an overlapped-response case that protects newer output. Current upstream 04853a1d3 adds only a non-overlapping Sarvam default-model update and auto-merges cleanly.

GitHub validation is green: 2,400 passed and 5 skipped on Linux, with Ruff, Python 3.10/3.13 typing, BlockGuard on Ubuntu/macOS/Windows, release gate, and CLA all passing.

@davidzhao, since you reviewed the overlapping realtime change in #6962, a maintainer review when you have bandwidth would be appreciated.