Skip to content

[feature] voice-typing keyboard: dictate into any app - #210

Merged
YJack0000 merged 1 commit into
mainfrom
claude/ios-keyboard-voice-transcription-d3712b
Aug 9, 2026
Merged

YJack0000 merged 1 commit into
mainfrom
claude/ios-keyboard-voice-transcription-d3712b

Conversation

@YJack0000

Copy link
Copy Markdown
Contributor

What

A dictation keyboard for Parley: tap the mic in any text field and speak; the words land in the field. The phone's version of the desktop's voice_typing.rs.

The constraint

A keyboard extension cannot open the microphone (iOS forbids it since iOS 8, Full Access included). So the keyboard doesn't record — it bounces to the container app, the app records, and the transcript is handed back. Same pattern every dictation keyboard on the App Store uses.

Recording reuses the existing pipeline: AudioCapture (16 kHz mono) → SttRelayClient (hosted relay, billed under feature: "dictation", no API key on the phone). New parts are the keyboard, the App Group hand-off, and the in-app dictation session.

Flow

  1. Keyboard mic tap — mints a session, captures the host app's bundle id (best-effort), writes the uplink, opens parley://dictate via SwiftUI openURL (the responder-chain openURL: walk was disabled for keyboards in iOS 18; openURL is the path that still works, walk kept as older-system fallback).
  2. App records — onOpenURL → DictationCoordinator starts mic + relay, mirrors the growing transcript into the downlink, posts a Darwin note per update.
  3. Return to host app — see below.
  4. Keyboard inserts — on the Darwin note (and every viewWillAppear, in case it was suspended through the notification) it inserts whatever is new past its insertedCount high-water mark, so a killed-and-relaunched keyboard never double-inserts.

App Group channel (DictationChannel, in ParleyKit)

Two single-writer mailboxes (dictation-down.json app→keyboard, dictation-up.json keyboard→app) plus two Darwin notifications. The files are the source of truth; the notes are pure "go re-read" signals — which is what makes the hand-off robust to the keyboard being suspended/killed while the app is foregrounded.

The "jump back to the previous app" problem

Typeless's auto-return was never a public API. Apple closed it in iOS 26.4 — the private host-bundle-id getters return nil and the private launch lands on the Home Screen. Wispr Flow's own docs confirm they switched to a manual swipe on 26.4.

Decision (obfuscation + version gate):

  • iOS < 26.4 — auto-return via the private LSApplicationWorkspace launch. Every private symbol is assembled from string fragments at runtime and reached through responds(to:)/perform, so no literal private symbol sits in the binary (verified: strings finds none of _hostBundleID, LSApplicationWorkspace, openApplicationWithBundleID) and a missing getter is a nil, never a crash.
  • iOS 26.4+ — no private call attempted; the dictation screen teaches the swipe-right gesture with a first-run guide and keeps recording alive in the background (UIBackgroundModes: audio).

Action Button / Control Center

StartDictationIntent (AudioRecordingIntent, iOS 18+) starts dictation without leaving the current app at all — the lowest-friction trigger, sidestepping the auto-return problem entirely.

App Review

  • 4.4.1 — the keyboard always shows a minimal key row (globe/space/return/delete), functional without Full Access; dictation itself needs Full Access and says so with a jump to Settings.
  • 2.5.1 — the auto-return path is private and version-gated to where it works; everything else (openURL, App Group, insertText, App Intents) is public.
  • Memory — the keyboard process holds no audio, no model, no transcript history, staying under the jetsam limit.

Verification

  • xcodebuild app + keyboard extension → BUILD SUCCEEDED (iPhone 17 Pro sim).
  • ParleyKit: 19 tests pass.
  • Bundle integration: ParleyKeyboard.appex embedded, extension point com.apple.keyboard-service, RequestsOpenAccess true, App Group entitlement on both targets.
  • Obfuscation confirmed at the binary level (no literal private symbols).

Not verified here (needs manual device QA)

The full keyboard round trip (switch keyboard → jump → return → text insertion) needs a signed-in account, the keyboard enabled in system Settings, and a host app; auto-return only fires on iOS < 26.4. Logic/build/integration are verified; the live path needs a device pass.

Before a real App Store build

Register in the Apple Developer portal, or archive signing will fail:

  • App Group group.com.pathors.parley.ios (on both App IDs)
  • Keyboard bundle id com.pathors.parley.ios.keyboard

Design doc: docs/design/ios-voice-keyboard.md. Bumps iOS build to 5.

A keyboard extension can't open the microphone, so the Parley keyboard's mic
button opens the container app (parley://dictate); the app records through the
existing AudioCapture + SttRelayClient pipeline (billed feature: "dictation")
and streams the transcript back through an App Group, which the keyboard inserts
via textDocumentProxy. New DictationChannel (shared in ParleyKit) is two
single-writer mailboxes plus Darwin notifications, robust to the keyboard being
suspended while the app is foregrounded.

Return-to-host is version-gated: on iOS < 26.4 the app auto-returns via the
private LSApplicationWorkspace launch (every private symbol assembled at runtime
so none appears literally in the binary); on 26.4+, where Apple closed that
path, the dictation screen teaches the swipe-right gesture instead. A
StartDictationIntent (AudioRecordingIntent, iOS 18+) offers an Action Button /
Control Center trigger that records without leaving the current app at all.

The keyboard keeps a minimal key row so it stays functional without Full Access
(App Review 4.4.1); Settings gains a setup guide. Design doc:
docs/design/ios-voice-keyboard.md. Bumps iOS build to 5.
@sonarqubecloud

sonarqubecloud Bot commented Aug 8, 2026

Copy link
Copy Markdown

@YJack0000
YJack0000 merged commit 979765c into main Aug 9, 2026
1 check passed
@YJack0000
YJack0000 deleted the claude/ios-keyboard-voice-transcription-d3712b branch August 9, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant