Skip to content

simulation: participant-sync race, fake-job metadata fallback, no audio recording #6069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
theomonnom merged 6 commits into from
Jun 12, 2026
Merged

simulation: participant-sync race, fake-job metadata fallback, no audio recording #6069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9420e2c
simulation: don't cache a context miss before the participant list syncs
theomonnom Jun 12, 2026
654da62
simulation: job-metadata dispatch fallback only for fake jobs
theomonnom Jun 12, 2026
db3db53
voice: no audio recording under text simulation
theomonnom Jun 12, 2026
99c9e3f
trim comments
theomonnom Jun 12, 2026
195d743
simulation: drop the job-metadata dispatch fallback
theomonnom Jun 12, 2026
138a8b3
simulation: register the simulator disconnect handler unconditionally
theomonnom Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions livekit-agents/livekit/agents/job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,14 +461,11 @@ def simulation_context(self) -> SimulationContext | None:
metadata = dispatch_json
break
if not metadata:
# older servers sent the dispatch in the job metadata;
# fake_job_context places it there too
metadata = self._info.job.metadata
if not metadata:
# The simulator participant is only visible once the room is
# connected; a miss before then (AgentSession.start consults
# _text_only pre-connect) must not be cached.
self._simulation_resolved = self._room.isconnected()
# The simulator joins before the agent, so a miss is only final
# once the room is connected and a remote participant is visible.
self._simulation_resolved = (
self._room.isconnected() and len(self._room.remote_participants) > 0
)
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
return None
Comment on lines 463 to 469

@devin-ai-integration devin-ai-integration Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Removal of job-metadata fallback breaks pre-connect simulation detection for all _text_only callers in start()

The removal of the self._info.job.metadata fallback at job.py:463-469 is intentional per the commit message, but it has a broader impact than just the new line 670-671 code. The pre-existing _text_only checks at agent_session.py:696 (log message) and agent_session.py:749 (disabling audio I/O in room options) also relied on the fallback to work before connect(). Without it, text simulations will set up audio I/O via RoomIO unnecessarily. The AgentActivity checks (agent_activity.py:3903, 3961, 3971) work correctly because they run after the room connects, so STT/TTS/VAD are properly disabled — but audio tracks are already published/subscribed.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@theomonnom theomonnom Jun 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fallback was already inert for real jobs: the server stopped populating job metadata with the dispatch when it moved to participant attributes (production runs detect via attributes with empty metadata). Pre-connect _text_only resolution therefore did not work through metadata before this PR either; connect-first entrypoints are unaffected.


self._simulation_resolved = True
Expand Down Expand Up @@ -597,8 +594,10 @@ async def connect(
await self._room.connect(self._info.url, self._info.token, options=room_options)
self._on_connect()

if self.simulation_context() is not None:
self._room.on("participant_disconnected", self._on_simulator_disconnected)
# Always registered: the callback ignores participants without the
# simulator attribute, and gating on simulation_context() here would
# race the participant-list sync.
self._room.on("participant_disconnected", self._on_simulator_disconnected)

for p in self._room.remote_participants.values():
self._participant_available(p)
Expand Down
7 changes: 3 additions & 4 deletions livekit-agents/livekit/agents/testing.py
View file
Open in desktop

@devin-ai-integration devin-ai-integration Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 job_metadata parameter remains in fake_job_context but is no longer consumed for simulation dispatch

The job_metadata parameter is still accepted by fake_job_context at testing.py:31 and still assigned to the job proto at testing.py:66. However, since the job-metadata fallback was removed from simulation_context(), passing a SimulationDispatch JSON via job_metadata no longer triggers simulation detection. The docstring was updated to remove the mention of this use case (testing.py:39-41), which is consistent. Any existing test or dev code that relied on fake_job_context(job_metadata=<SimulationDispatch JSON>) to trigger text-only mode would silently break. The parameter could be kept for other metadata use cases, but its simulation use case is dead.

(Refers to line 31)

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@theomonnom theomonnom Jun 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional: no deployed server sends the dispatch via job metadata, so the simulation use of job_metadata was already dead in practice. The parameter stays for generic ctx.job.metadata consumers; in-process simulation tests can set the simulation context on the yielded JobContext directly.

Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,9 @@ def fake_job_context(
agent in-process (tests/dev) — no worker, no AgentServer.

The context is a real ``fake_job`` :class:`JobContext` (so ``get_job_context()``
and its access points behave normally). ``job_metadata`` is placed on the job —
e.g. a protojson ``SimulationDispatch``, which ``JobContext.simulation_context()``
reads. The :class:`JobContext` is yielded so callers can tweak it (set
``_simulation_end_fnc``, etc.). Wrap ``session.start(room=room)`` with it::
and its access points behave normally). The :class:`JobContext` is yielded so
callers can tweak it (set ``_simulation_end_fnc``, etc.). Wrap
``session.start(room=room)`` with it::

async with rtc.Room() as room:
await room.connect(url, token)
Expand Down
2 changes: 2 additions & 0 deletions livekit-agents/livekit/agents/voice/agent_session.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -667,6 +667,8 @@ async def start(
record = job_ctx.job.enable_recording if job_ctx else False

self._recording_options = _resolve_recording_options(record) # type: ignore[arg-type]
if self._text_only:
self._recording_options["audio"] = False
Comment on lines +670 to +671

@devin-ai-integration devin-ai-integration Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 New _text_only check at line 670 is ineffective for real (non-fake) simulation jobs

The new if self._text_only: self._recording_options["audio"] = False check at line 670 calls simulation_context() which requires the room to be connected and a simulator participant to be visible. However, at this point in start(), the room has not yet been connected — job_ctx.connect() is scheduled later as a task at line 812-813 and awaited at line 836. For real simulation jobs, _text_only will return False here, making this check a no-op. It only works for fake jobs where fake_job_context pre-sets jc._connected = True and provides simulation dispatch via job_metadata.

This is consistent with the existing _text_only checks at lines 696 and 749 which have the same limitation. The practical impact is minimal: even when _recording_options["audio"] stays True, the RecorderIO at line 789-791 is only created when self.input.audio and self.output.audio are both set, and the AgentActivity properties (stt, tts, vad at agent_activity.py:3897-3973) independently check _text_only at access time to disable audio models.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@theomonnom theomonnom Jun 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the thread below: effective in the connect-first flow every simulation entrypoint uses; the auto-connect ordering limitation predates this PR.

Comment on lines +670 to +671

@devin-ai-integration devin-ai-integration Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 New text-simulation audio recording disable is ineffective because room is not yet connected

The newly added code at lines 670-671 attempts to disable audio recording for text simulations by checking self._text_only, which calls job_ctx.simulation_context(). However, simulation_context() (livekit-agents/livekit/agents/job.py:463-469) can only detect the simulation after the room is connected and remote participants are visible. At line 670, the room connection hasn't happened yet — it's scheduled later as a task at agent_session.py:813 (asyncio.create_task(job_ctx.connect())), which runs during asyncio.gather(*tasks) at line 836. Since the room is not connected, _text_only returns False and self._recording_options["audio"] is never set to False.

The same issue affects the pre-existing _text_only check at line 749 (which disables audio I/O in room options) — but that check previously worked because the old code had a fallback to self._info.job.metadata (removed by this PR). Without the fallback, both audio I/O setup and audio recording proceed normally in text simulations, and the RecorderIO is created at line 791 to record audio that shouldn't exist. The AgentActivity checks for _text_only (e.g., agent_activity.py:3903, 3961, 3971) happen later when the room IS connected, so STT/TTS/VAD are correctly disabled — but by then audio I/O and recording infrastructure have already been set up.

Prompt for agents 
The _text_only property relies on simulation_context() which requires the room to be connected and remote participants to be visible. At line 670 in start(), the room has not yet connected (connection is scheduled as a task at line 813 and runs during asyncio.gather at line 836). This means _text_only always returns False at this point, and the audio recording disable never fires.

The same timing issue affects the _text_only check at line 749 (disabling audio I/O in room options). Previously, the job-metadata fallback in simulation_context() allowed these early checks to work, but that fallback was removed in this PR.

Possible approaches:
1. Move the audio recording and room options adjustments for text simulation to after the room connects (e.g., after asyncio.gather completes at line 836), and retroactively disable the recorder if _text_only is True.
2. Re-introduce a lightweight pre-connect detection mechanism for text simulation mode (e.g., a flag on the job or RunningJobInfo) that doesn't depend on participant attributes.
3. Check _text_only after the gather and tear down the recorder if it was unnecessarily created.

Files involved: livekit-agents/livekit/agents/voice/agent_session.py (start method, lines 656-850), livekit-agents/livekit/agents/job.py (simulation_context method, lines 442-491).
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@theomonnom theomonnom Jun 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially right, but the impactful flow works: every simulation entrypoint awaits ctx.connect() before session.start() (e.g. examples/frontdesk), so at line 670 the room is connected and participants are synced — with this PR's no-cache guard the check is effective there. The auto-connect flow (start() scheduling connect at line 813) indeed decides before connect; that limitation predates this PR and is unchanged by it. Worth noting the metadata fallback never helped here in practice: the server stopped sending the dispatch via job metadata when it moved to participant attributes, and last night's production runs confirm detection succeeded via attributes with empty job metadata.


is_primary = True
if job_ctx:
Expand Down
Loading