Skip to content

[fix] the recording survives leaving the foreground, and one tap means one recording - #272

Merged
YJack0000 merged 2 commits into
mainfrom
fix/recording-pipeline
Aug 19, 2026
Merged

YJack0000 merged 2 commits into
mainfrom
fix/recording-pipeline

Conversation

@YJack0000

Copy link
Copy Markdown
Contributor

What this changes

The phone stops losing the microphone when the meeting is not on screen, one tap stops being able to start two recordings, and recording begins when the button is pressed instead of one network round trip later. Android gets the two robustness gaps it shared with iOS, plus the same latency fix.

Why

Three reports from dogfooding, which turned out to be three faces of one thing: nothing owned the recording. LiveView kept the capture, the relay and the uploader in @State and only set isRecording once all of them were up.

1. The meeting died after switching apps

UIBackgroundModes: audio keeps the process alive, but it says nothing about iOS taking the microphone back — and there are four ways it does, every one of them routine while the user is in another app. AudioCapture listened for none.

What happens Notification Result before
Call, Siri, alarm, another app opening a recording session interruptionNotification engine stops; nothing restarts it
Headphones/Bluetooth in or out, speaker override routeChangeNotification a tap built for the old format keeps feeding a dead node
The engine's IO format changes underneath it .AVAudioEngineConfigurationChange AVAudioEngine tears the tap down itself
mediaserverd restarts mediaServicesWereResetNotification every audio object in the process is dead

All four look identical from outside: the app is still up, isRecording is still true, the level meter sits at zero for the rest of the meeting, and about a minute later the relay idle-closes for want of audio and the screen says "Relay closed". It reads as a dropped connection. The network was never the problem — the microphone died first.

AudioCapture now treats capture as something to hold: all four notifications observed, engine + tap + converter rebuilt on each, backoff retry, and a two-second watchdog for what iOS announces to nobody (a route that never came back, an interruption that never posted .ended). Transitions surface through onStatus, so the screen says the microphone is paused rather than going quietly silent; a microphone that is genuinely gone ends the meeting properly instead of recording nothing.

2. Two sockets transcribing one room

Not a subtle race — a window you can hit on purpose:

sequenceDiagram
    participant U as User
    participant V as LiveView
    participant R as Relay
    U->>V: tap → consent → start()
    Note over V: isRecording is still false
    V->>R: await client.start() — TLS + HTTP upgrade
    U->>V: button still says "Start recording" → tap again
    Note over V: second start()
    V->>R: a second socket, a second AVAudioEngine, a second uploader
    R-->>V: both legs write segments into one transcript
    Note over V: self.relay holds only the last one;<br/>the first keeps streaming, unreferenced
Loading

MeetingRecorder makes the state machine the type. phase moves to .starting synchronously on the tap, start()/stop() are guarded by it, and the button reads it rather than a success flag. No amount of care inside start() fixes this while "am I recording?" is a flag set at the end.

3. Seconds before recording began

The relay handshake blocked the microphone, and setActive(true) — which negotiates the route, and takes its time when a Bluetooth device is around — ran on the main actor.

before after
tap → visible feedback after everything succeeds same tick
tap → microphone open after the handshake, on the main thread immediately, off the main actor
handshake blocking alongside; the client buffers what the mic gives it until the socket is up

Two more the same code hid

  • Chunk order was not guaranteed. Every chunk was its own Task { await client.send(pcm:) } — one unstructured task per 85 ms of audio, with no ordering between them. The provider could be handed speech with two chunks swapped and would faithfully transcribe the garble. A single writer now drains an AsyncStream, which also bounds the queue and is what makes the connect window safe to record through.
  • A closed socket ended live transcription for good. It now reconnects with backoff while the microphone keeps running. Each leg needs its own segment-id prefix and time offset: the provider restarts its numbering and its clock at zero, so the second leg's mix-0 would otherwise overwrite the opening of the meeting. Covered by a new test on both platforms.

Nothing about the audio file depends on the socket, so a dropped relay costs live transcript, not the recording — and the cloud transcribes the uploaded audio anyway. That is why this reconnects quietly instead of failing the meeting.

Dictation shares AudioCapture and the relay client, so it picks up the interruption recovery, the off-main-actor start and the ordered writer for free.

Android

Already the healthier pipeline — foreground service, explicit state machine, ordered chunks, double-start guarded — so this is the gaps it shared, plus the latency win:

  • A silenced microphone was invisible. MicCapture documents the case and leaves it: from Android 10, when another app takes the microphone the framework does not fail, it feeds silence, and AudioRecord.read keeps returning success. A meeting could record forty minutes of nothing and report no problem. MeetingSession now registers an AudioRecordingCallback and publishes micSilenced; the screen shows it in error colour. The recording deliberately keeps running — the takeover usually ends by itself and what follows is still worth having.
  • A dropped relay never came back — same reconnect, same per-leg id prefix and time offset.
  • The microphone waited on the handshake. connect() splits into open() + awaitOpen(). The config frame is queued as soon as the socket is created; OkHttp buffers frames until the upgrade lands and writes them in order, so it is guaranteed to reach the relay ahead of any audio.

Related: #109 is the same class of bug (never more than one active listener) on desktop voice typing. This PR does not touch that path, so it does not close it.

How it was verified

  • bunx tsc --noEmit passes
  • bunx vitest run passes (231 tests) — no TypeScript changed, run to confirm the base is clean
  • Added or updated tests — reconnected-leg ids and rebased timestamps, SegmentBuilderTests.swift and SegmentBuilderTest.kt
  • Added new user-facing strings to both zh-Hant and en — 10 in ios/App/Parley/Localizable.xcstrings, 1 in android/.../res/values*/strings.xml (this PR adds no src/i18n/messages.ts strings)
  • swift test — 20 tests
  • xcodebuild Debug and Release
  • ./gradlew :app:compileDebugKotlin :app:testDebugUnitTest :parleykit:test
  • Ran the app — the interruption and route paths cannot be exercised in the simulator. Device checks below.

Device checks before this goes in a build

Do this mid-recording Expect
Switch to another app playing audio / open Voice Memos "Microphone paused by the system", then "Microphone is back" on release; transcript resumes
Take a call and hang up same; recovers within ~0.25 s of the call ending
Plug and unplug AirPods brief gap, then the level meter moves again
Airplane mode for 30 s, then off "Transcription dropped — reconnecting…" → "Transcribing live", and the opening of the meeting is not overwritten
Press record five times fast exactly one meeting; from the second press the button already reads "End meeting"

Known limitation, deliberately not addressed

The record button is inert while a finished meeting uploads ("Wrapping up…"). A long meeting on cellular can hold that for minutes. Fixing it means splitting MeetingUploader.finishAndUpload into "persist to the pending queue" and "upload", and releasing the button after the first. Left out on purpose: closing the two-meetings-at-once hole first is the safer order.

Three reports, one cause: nothing owned the recording. The live screen kept
the capture, the relay and the uploader in `@State` and only set
`isRecording` once every one of them was up.

**The meeting died after switching apps.** `UIBackgroundModes: audio` keeps
the process alive, but iOS has four ways of taking the microphone back —
`interruptionNotification` (call, Siri, another app opening a recording
session), `routeChangeNotification`, `.AVAudioEngineConfigurationChange`, and
`mediaServicesWereResetNotification` — and `AudioCapture` listened for none of
them. All four look identical from the outside: the app is still up,
`isRecording` is still true, the level meter sits at zero, and a minute later
the relay idle-closes for want of audio and reports "Relay closed". It reads
as a dropped connection; the network was never the problem.

`AudioCapture` now treats capture as something to hold rather than start. It
observes all four notifications, rebuilds engine, tap and converter on each,
retries with backoff, and polls every two seconds for what iOS announces to
nobody (a route that never came back, an interruption with no `.ended`). Each
transition is reported through `onStatus`, so the screen says the microphone
is paused instead of going quietly silent, and a microphone that is genuinely
gone ends the meeting properly rather than recording nothing.

**Two sockets transcribing one room.** Between the tap and `isRecording`
going true sat a WebSocket handshake. A second tap in that window ran
`start()` again: two `AVAudioEngine`s on the microphone, two relay sockets
streaming the same room into one transcript, two uploaders of which only the
last stayed referenced. `MeetingRecorder` makes the state machine the type —
`phase` moves to `.starting` synchronously on the tap, every entry point is
guarded by it, and the button reads it rather than a success flag.

**Seconds before recording began.** The relay handshake blocked the
microphone, and `setActive(true)` ran on the main actor. Now the client is
built before it is connected and buffers what the microphone gives it, so the
microphone opens first and the handshake runs alongside it;
`AudioCapture.start()` is `async` and does its work off the main actor.

Two more the same code hid:

- Every chunk was its own `Task { await client.send(pcm:) }` — one
  unstructured task per 85 ms of audio, with no ordering between them, so the
  provider could be handed speech with two chunks swapped. A single writer
  now drains an `AsyncStream`, which also bounds the queue and is what makes
  the connect window safe to record through.
- A closed socket ended live transcription for good. It now reconnects with
  backoff while the microphone keeps running. Each leg needs its own id
  prefix and time offset, because the provider restarts its numbering and its
  clock at zero and the second leg's `mix-0` would otherwise overwrite the
  opening of the meeting.

Dictation shares `AudioCapture` and the relay client, so it gets the
interruption recovery, the off-main-actor start and the ordered writer too.

Verified: `swift test` (20), Debug and Release builds, `bunx tsc --noEmit`,
`bunx vitest run`. The interruption and route paths need hardware — see the
PR for the device checks.
The Android pipeline was already the healthier of the two — foreground
service, explicit state machine, ordered chunks, double-start guarded — so
this is the two gaps it shared with iOS, plus the latency win.

**A silenced microphone was invisible.** `MicCapture` documents the case and
leaves it: from Android 10, when another app takes the microphone the
framework does not fail, it feeds silence, and `AudioRecord.read` keeps
returning success. A meeting could record forty minutes of nothing and report
no problem. `MeetingSession` now registers an `AudioRecordingCallback` and
publishes `micSilenced`; the screen shows it in error colour. The recording
deliberately keeps running, because the takeover usually ends by itself and
what follows is still worth having.

**A dropped relay never came back.** `Closed` and `Error` mid-meeting only
raised a banner and the transcript stopped for good. They now reconnect with
backoff, capped per meeting, with the same per-leg id prefix and time offset
the Swift side needed — the provider restarts its numbering and its clock at
zero, so without them the second leg's `mix-0` overwrites the first minute of
the transcript.

**The microphone waited on the handshake.** `connect()` splits into `open()`
and `awaitOpen()`. The config frame is queued as soon as the socket is
created; OkHttp buffers frames until the upgrade lands and writes them in
order, so it is guaranteed to reach the relay ahead of any audio and the
microphone can open at the same time as the socket.

Verified: `:parleykit:test`, `:app:testDebugUnitTest`, `:app:compileDebugKotlin`.
@YJack0000
YJack0000 merged commit 743c902 into main Aug 19, 2026
@sonarqubecloud

Copy link
Copy Markdown

@YJack0000 YJack0000 mentioned this pull request Aug 20, 2026
5 tasks
YJack0000 added a commit that referenced this pull request Aug 20, 2026
App Store Connect refuses a `CFBundleVersion` it has already seen, and 1.3 (9)
is on TestFlight, so the tag `ios-release.yml` checks against has to move with
it. Both targets carry the number and both move together; `Info.plist` is
regenerated from `project.yml` rather than edited, since the next
`xcodegen generate` would otherwise overwrite it.

1.4 rather than 1.3 (10) because what it carries is behavioural, not a rebuild:
the microphone now survives the app leaving the foreground, one tap can no
longer start two recordings, and recording begins on the tap instead of after
the relay handshake (#272).
Lanznx added a commit that referenced this pull request Aug 20, 2026
`project.yml` reached 1.4.1 and both tags went out, but `AppStore/metadata`
still stopped at 1.3 — so the version about to be submitted has no What's New
copy in either locale, and nothing tells a user why the globe key moved.

1.3 is what the App Store is serving, so these notes cover everything since it:
the voice pane becoming a control panel (#266), the recording holding on to the
microphone across foreground changes and the double-recording fix (#272), and
the permission prompt surviving the keyboard hand-off (#281).

The globe paragraph is written to be read by someone who read 1.3's. 1.3
announced the key on every device; this took it back where iOS already draws
one, and saying so plainly is cheaper than letting people think it vanished.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lanznx added a commit that referenced this pull request Aug 20, 2026
`project.yml` reached 1.4.1 and both tags went out, but `AppStore/metadata`
still stopped at 1.3 — so the version about to be submitted has no What's New
copy in either locale, and nothing tells a user why the globe key moved.

1.3 is what the App Store is serving, so these notes cover everything since it:
the voice pane becoming a control panel (#266), the recording holding on to the
microphone across foreground changes and the double-recording fix (#272), and
the permission prompt surviving the keyboard hand-off (#281).

The globe paragraph is written to be read by someone who read 1.3's. 1.3
announced the key on every device; this took it back where iOS already draws
one, and saying so plainly is cheaper than letting people think it vanished.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Lanznx added a commit that referenced this pull request Aug 20, 2026
…he end of it (#285)

A phone call, a Wi-Fi→LTE handover or a relay restart drops the STT
WebSocket, and until now that cost either the words spoken during the gap
(meetings) or the whole session (dictation, where any `.closed`/`.error`
ended it and an error surfaced raw wire text to the user).

The relay cannot resume a session — one socket carries one Soniox session,
and a reconnect is necessarily a fresh leg whose clock restarts at zero.
`SttRelayClient.Options.idPrefix`/`timeOffsetMs` already anticipated that;
what was missing was anything to hold the audio in between, and a clock
honest about where that audio belongs.

ParleyKit gains the two testable pieces:

- `RelayAudioBridge` — the microphone's only counterparty. It forwards to
  the current leg and holds chunks while there is none, bounded (45 s for
  meetings, 20 s for dictation) with drop-oldest, and hands the next leg
  both the held audio and the offset of its *first held sample*. Offsetting
  a leg to "now" instead would file a gap's worth of speech after the words
  that followed it.
- `ReconnectPolicy` — the 1, 2, 4, 8 … capped ladder and the attempt
  ceiling, in one place instead of a `pow()` inline per recorder.

`MeetingRecorder` moves onto both, so the meeting reconnect that landed in
 #272 no longer throws the gap away. `DictationCoordinator` gets reconnect
at all: a dropped socket redials on the shorter dictation ladder with the
microphone still open, the tentative tail is folded into the settled text
rather than lost with the socket, and only an exhausted ladder ends the
session — keeping every word the keyboard already typed.

Reconnecting now looks different from failed, in both languages: an amber
spinner on the meeting screen, "Reconnecting…" plus "keep talking" on the
dictation screen and in the keyboard, against full-weight ink for a live
transcript that is genuinely over. The keyboard learns a `reconnecting`
downlink state so it stops going quiet mid-session.

One failure is deliberately not retried: a 402 from the relay means the
account is out of quota, and the next handshake is refused the same way.
`SttRelayEvent.error` now carries the code so both paths can say that
instead of spending the ladder to arrive at "lost the connection".

Not covered here: Android has no audio-focus or interruption handling at
all, and `MeetingSession` drops gap audio the same way this fixes on iOS.
Tracked separately.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant