Skip to content

[fix] a dropped relay socket is a pause in the live transcript, not the end of it - #285

Merged
Lanznx merged 1 commit into
mainfrom
fix/relay-reconnect
Aug 20, 2026
Merged

Lanznx merged 1 commit into
mainfrom
fix/relay-reconnect

Conversation

@Lanznx

@Lanznx Lanznx commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Part 1 of #283 — the relay half, on iOS.

What was actually broken

The report is "a phone call ends the recording", and reading the code it was three things. Only one of them still is.

The relay never reconnected on the dictation path, and threw the gap away on the meeting path. DictationCoordinator turned any .closed into finishUp() and any .error into fail(message) — with the raw wire string (relay error 402: …) shown to the user. MeetingRecorder did reconnect (that landed in #272) but dropped the sink the moment the socket died, so everything said during the backoff was gone from the live transcript, and the new leg's offset was systemUptime - captureStartedAt — "now" — which files a gap's worth of speech after the words that followed it.

Can the relay resume a session? No. One socket carries one Soniox session; there is no resume token in the protocol (SonioxProtocol.swift — config frame, keepalive, finalize, and that is the whole upstream vocabulary), and SttRelayClient is explicitly single-use. So a reconnect is necessarily a fresh Soniox leg whose token clock restarts at zero. That is exactly what Options.idPrefix and Options.timeOffsetMs were put there for; what was missing was something to hold the audio across the gap and a clock honest about where that audio belongs. This PR does not paper over the lack of resume — it builds on the per-leg offset the protocol already assumed.

iOS audio interruptions: already fixed, left alone. AudioCapture (#272) observes interruptionNotification, routeChangeNotification, .AVAudioEngineConfigurationChange and mediaServicesWereResetNotification, rebuilds the engine rather than leaving a tap on a dead node, deliberately ignores the absent .shouldResume (which is exactly the "other party hung up" case), and runs a two-second watchdog for what iOS announces to nobody. That covers the reported case. I did not touch it.

What this PR adds

RelayAudioBridge (ParleyKit). The microphone's only counterparty. It forwards chunks to the current leg and holds them while there is none, bounded (45 s meetings, 20 s dictation) with drop-oldest, and on the next attach it hands the new leg both the held audio, in order, and the offset of its first held sample. Overflow advances that offset with the front of the buffer, so a leg is never told it is being fed audio it is not. attach takes a factory closure so the offset, flush and swap are one step the audio thread cannot land in the middle of.

ReconnectPolicy (ParleyKit). The 1, 2, 4, 8 … capped ladder and the attempt ceiling, in one place instead of a pow() inline per recorder. .dictation redials faster and gives up sooner — a two-minute session with someone standing there is not a meeting.

MeetingRecorder moves onto both, so the gap is kept instead of dropped and the offset comes from the bridge.

DictationCoordinator gets reconnect at all: a dropped socket redials with the microphone still open, each leg gets its own id prefix so leg 2's first sentence cannot overwrite leg 1's, the tentative tail is folded into the settled text rather than dying with the socket, and only an exhausted ladder ends the session — keeping every word the keyboard already typed into the user's document.

Reconnecting no longer looks like failed, in both languages:

reconnecting live transcript over
meeting screen amber spinner + "Transcription dropped — reconnecting…" / 「轉錄連線中斷,正在重新連線…」 full-weight ink + icon, "…the recording is still running and will be transcribed after it syncs."
dictation screen "Reconnecting…" / 「重新連線中…」 with "keep talking, what you say is kept" error state, "Lost the connection. What you already said has been typed…"
keyboard amber "Reconnecting… keep talking" / 「重新連線中…可以繼續說」 above the transcript, which stays put red error caption

The keyboard learns a reconnecting downlink state, so it stops going quiet mid-session. Red stays reserved for a session that actually ended; the meeting's "stopped" line is ink rather than red, because the recording is still running and colouring it like a failure would say the opposite of what the sentence says.

One failure is deliberately not retried. A 402 means the account is out of quota and the next handshake is refused identically, so SttRelayEvent.error now carries the relay's code and both paths say what happened instead of spending the whole ladder to arrive at "lost the connection".

Verified

  • swift test --package-path ios/ParleyKit — 37 tests, 16 new: hold/flush ordering, the bound and its drop-oldest, timestamp continuity across a simulated reconnect leg (10 s of meeting + 3 s into the gap ⇒ the new leg is offset to 10 000 ms, not 13 000), discard-keeps-the-clock, a concurrent-sender test that no chunk is lost or reordered while a leg is swapped underneath, the backoff ladder and ceiling, and the quota code the parser and the event have to agree on.
  • xcodebuild -project ios/App/Parley.xcodeproj -scheme Parley -destination 'generic/platform=iOS Simulator' CODE_SIGNING_ALLOWED=NO build — clean (after xcodegen generate).
  • bunx tsc --noEmit and bunx vitest run (231 tests) — clean. Untouched by this change; run because CLAUDE.md asks for it.
  • Every new string has both en and zh-Hant entries in the app and keyboard catalogs.

Not verified — needs a device

None of the below is reachable from a simulator or a unit test, and I have not run any of it. From the checklist in #283, for iOS:

  • Incoming call, answered, then ended → recording continues, transcript continues, no gap beyond the call.
  • Incoming call, declined → same.
  • Wi-Fi → cellular handover mid-recording.
  • Airplane mode for ~10 s mid-recording → reconnects, and the words spoken during the gap actually arrive (they are inside the 45 s hold, so they should — this is the specific claim most worth checking).
  • Siri, an alarm, and another app opening a recording session.
  • The relay restarted under a live session.
  • The same six against dictation, where the ladder is shorter (0.5, 1, 2, 4 s) and the session is capped at two minutes — including that the keyboard shows "Reconnecting… keep talking" rather than going quiet, and that nothing is double-inserted across a leg change.
  • A held gap longer than the bound (>45 s meeting / >20 s dictation) → the oldest audio is dropped and the surviving audio still lands at the right timestamp.

Two behaviours worth a specific eye on a device: the relay-side timestamps of a reconnected leg (the unit test proves the offset the client is given, not what the relay does with it), and whether a 402 arrives as an in-band error frame (handled) or as a socket close with code 1011 (falls through to the ordinary reconnect ladder, ending in the honest "live transcription stopped" line 75 s later).

Not in this PR

Android — no audio focus handling at all, and MeetingSession throws the reconnect gap away exactly as iOS did. It is a separate PR by someone who can build it: there is no Android SDK on the machine this was written on and CI skips android/** on pull requests, so shipping unbuildable Kotlin here would have been worse than not shipping it. Filed with the full design and the device checklist as #284.

Closes nothing on its own — #283 stays open until #284 lands.

🤖 Generated with Claude Code

…he end of it

A phone call, a Wi-Fi→LTE handover or a relay restart drops the STT
WebSocket, and until now that cost either the words spoken during the gap
(meetings) or the whole session (dictation, where any `.closed`/`.error`
ended it and an error surfaced raw wire text to the user).

The relay cannot resume a session — one socket carries one Soniox session,
and a reconnect is necessarily a fresh leg whose clock restarts at zero.
`SttRelayClient.Options.idPrefix`/`timeOffsetMs` already anticipated that;
what was missing was anything to hold the audio in between, and a clock
honest about where that audio belongs.

ParleyKit gains the two testable pieces:

- `RelayAudioBridge` — the microphone's only counterparty. It forwards to
  the current leg and holds chunks while there is none, bounded (45 s for
  meetings, 20 s for dictation) with drop-oldest, and hands the next leg
  both the held audio and the offset of its *first held sample*. Offsetting
  a leg to "now" instead would file a gap's worth of speech after the words
  that followed it.
- `ReconnectPolicy` — the 1, 2, 4, 8 … capped ladder and the attempt
  ceiling, in one place instead of a `pow()` inline per recorder.

`MeetingRecorder` moves onto both, so the meeting reconnect that landed in
 #272 no longer throws the gap away. `DictationCoordinator` gets reconnect
at all: a dropped socket redials on the shorter dictation ladder with the
microphone still open, the tentative tail is folded into the settled text
rather than lost with the socket, and only an exhausted ladder ends the
session — keeping every word the keyboard already typed.

Reconnecting now looks different from failed, in both languages: an amber
spinner on the meeting screen, "Reconnecting…" plus "keep talking" on the
dictation screen and in the keyboard, against full-weight ink for a live
transcript that is genuinely over. The keyboard learns a `reconnecting`
downlink state so it stops going quiet mid-session.

One failure is deliberately not retried: a 402 from the relay means the
account is out of quota, and the next handshake is refused the same way.
`SttRelayEvent.error` now carries the code so both paths can say that
instead of spending the ladder to arrive at "lost the connection".

Not covered here: Android has no audio-focus or interruption handling at
all, and `MeetingSession` drops gap audio the same way this fixes on iOS.
Tracked separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@Lanznx
Lanznx merged commit 02ae447 into main Aug 20, 2026
1 check passed
Lanznx added a commit that referenced this pull request Aug 20, 2026
Carries the reconnect (#285) and the microphone window (#289) to TestFlight,
which is the only place the device checklists on #283 and #286 can be run —
neither PR could verify its own load-bearing assumption from a simulator.

The App Store is still serving 1.3 (checked across three storefronts; an earlier
reading of 1.2 was a stale cache). 1.4 and 1.4.1 were tagged and built but never
released, so their What's New copy has never been read. Rather than leave a user
updating from 1.3 to be told about only half of what changed, the 1.4.1 section
becomes 1.4.2 and gains this release's two entries — the window and the
reconnect — on top of the copy it already carried forward.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
YJack0000 added a commit that referenced this pull request Sep 23, 2026
…ictation (#418)

* [fix] a relay client cancelled during its handshake still opened a socket (failing test)

`SttRelayClient.cancel()` runs `tearDown` on a detached task, and `start()`
never checks whether that already happened. A client cancelled while its
owner was still waiting on the microphone went on to connect, send the
config frame and start its keepalive loop, and nothing ever closed it.

The test asserts the refusal `Spent`, which this commit declares and does not
yet throw: today the attempt reaches the network and fails with a transport
error instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [fix] a relay client refuses to start once it has been finished or cancelled

`start()` now throws `Spent` when `finish()` or `cancel()` came first, and
again if a cancel lands during the handshake. The flag is a lock rather than
actor state because `cancel()` is synchronous: an owner that cancels a client
whose `start()` is still in flight needs the refusal to hold from the moment
`cancel()` returns, not from whenever the detached `tearDown` gets the actor.

Before this, a dictation stopped while `launch` was still waiting on the
microphone had its client cancelled by `finishUp`, and that client then
finished connecting anyway: config frame sent, keepalive running, socket open
until the relay idled it out. That is a second live socket on the next
dictation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [fix] a dictation that ended mid-launch could take over the next one

Every relay event and every redial already checks `leg` before it touches
the session (#285, #396). Nothing else in the coordinator did. `launch()`
resumes after the permission prompt, the microphone start and the socket
handshake; `stop()` resumes after the finalize; `begin()` resumes after the
stop it ran for the previous session. Each of them went on to write `relay`,
`capture`, `state` and the downlink as if it still owned the session, and a
session that had ended or been replaced during the wait paid for it: a
`.listening` published over a `done`, a `Connection failed` published under
the next session's id, the next session's relay cancelled and its transcript
settled as `done`, or a second `AudioCapture` feeding the bridge alongside
the first.

Now every continuation checks `owns(owner)`, the same `leg` test the relay
path uses, before it touches anything. `dismiss()` loses its unconditional
`active = false`, which could have flipped a newer session's flag.

The three microphone openers collapse into `openMicrophone()`, which joins
an open already in flight instead of starting a second one. That is the
overlap #404 documented and left to `begin(session:)`: the keyboard's URL
fallback fires 700 ms into a background microphone start, and the foreground
`launch` it triggers found `capture == nil` and opened another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Lanznx added a commit that referenced this pull request Sep 28, 2026
…quota, and the live screen looks like iOS (#486)

* [fix] Android: a reconnect says it is reconnecting, a 402 says it is quota, and the live screen looks like iOS

A dropped relay socket set RELAY_ERROR/RELAY_CLOSED and the screen said, in
error red, that transcription had stopped unexpectedly — while the session was
already redialling and the words were being held for the next leg. The live
transcript now has iOS's split (MeetingRecorder.TranscriptionHealth, #285):
RECONNECTING is an amber spinner and "Transcription dropped — reconnecting…",
cleared when a leg comes back up or words arrive; only a real end (out of
quota, reconnect budget spent, signed out) says live transcription stopped,
drawn in ink with a bolt, because the recording itself is fine. The collapsed
panel's health mark shows the same glyph as the line it stands for.

A finished meeting whose upload hit 402 said "It uploads as soon as the network
is back". Finished now carries waitingForQuota (the same cause the importer
already reads off DrainResult.quotaExhausted) and says iOS's quota sentence; the
recording still stays queued. The uploader reports which recordings it copied
into an org, so the status can say "Synced, and shared to “Org”"; the other
endings take iOS's copy too ("Synced to the cloud", "Sync failed for now…",
"That recording was too short to keep", "Wrapping up…").

The screen:
- the 12-segment level meter is replaced by iOS's scrolling capsule waveform
  (#371): pale bars, newest brightest, gliding between samples, in the full
  column and the compact row;
- Discard is an outlined capsule with a trash glyph, 44dp tall, with a TalkBack
  description that says it asks first (#381);
- transcript lines lose the blue dot on every line; only the turn still being
  spoken has a blue label; the words are selectable, tail included, and a long
  press offers Copy of the whole turn;
- after Stop the transcript, outcome and filing card stay until Done or Back,
  as on iOS, instead of closing after 1.2 s; a settled session left behind is
  cleared before the next recording's consent prompt;
- before the microphone opens the empty transcript says iOS's "Hit record and
  put the phone on the table to catch the whole room."

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* [refactor] LiveWaveform splits its clock and its drawing into helpers

Sonar kotlin:S3776 flagged the composable at cognitive complexity 21 (max 15). The sampling/frame effect moves into rememberWaveformClock and a WaveformClock holder, and the per-frame drawing into DrawScope.drawWaveform / drawBar. Behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: YJack0000 <jack@pathors.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant