diff --git a/TODO.md b/TODO.md index 4eb2f8951..09d321f90 100644 --- a/TODO.md +++ b/TODO.md @@ -58,6 +58,16 @@ ADRs: [project_plans/stelekit/decisions/](project_plans/stelekit/decisions/) --- +## Mobile Voice Mode (Branch: stelekit-mobile-mode) + +Voice capture pipeline: mic tap → AudioRecord + MediaCodec → Whisper STT → LLM formatter → daily journal insert. Full plan: [docs/tasks/mobile-voice-mode.md](docs/tasks/mobile-voice-mode.md) + +- [x] **[VOICE-S1] Story 1 — Core pipeline (commonMain + Android, raw transcript)** — PR #2 open, CI passing. All T1.1–T1.7 complete: interfaces, ViewModel, WhisperSTT, AndroidAudioRecorder, App.kt wiring, VoiceCaptureButton. 13 ViewModel tests, MockEngine Whisper tests. +- [ ] **[VOICE-S2] Story 2 — LLM formatting + settings** — Next. Adds ClaudeLlmFormatterProvider, OpenAiLlmFormatterProvider, VoiceSettings (EncryptedSharedPreferences), Settings UI for API keys, ViewModel LLM integration. Start with T2.1 (ClaudeLlmFormatterProvider, 2h). See [docs/tasks/mobile-voice-mode.md#story-2](docs/tasks/mobile-voice-mode.md). +- [ ] **[VOICE-S3] Story 3 — iOS adapter + waveform feedback** — Blocked on Story 1 merge. IosAudioRecorder, IosSpeechToTextProvider (SFSpeechRecognizer), amplitude-reactive pulse. + +--- + ## Active Remediation (Post-Review March 2026) ### P0: STABILITY & COMPATIBILITY diff --git a/androidApp/src/main/AndroidManifest.xml b/androidApp/src/main/AndroidManifest.xml index 82a702ef2..855c8cdd9 100644 --- a/androidApp/src/main/AndroidManifest.xml +++ b/androidApp/src/main/AndroidManifest.xml @@ -2,11 +2,12 @@ + Unit = {} slot + — Add VoiceCaptureButton to bottom bar layout + +kmp/src/androidMain/.../MainActivity.kt (or equivalent entry point) + — Wire AndroidAudioRecorder, WhisperSpeechToTextProvider, ClaudeLlmFormatterProvider + +kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/SettingsDialog.kt (Story 2) + — Add Voice API Keys section (Whisper key, Claude/OpenAI key) +``` + +--- + +## Epic: Mobile Voice Mode + +Deliver a hands-free voice capture pipeline that transcribes speech, formats it into Logseq +outliner syntax, and appends it to today's daily journal. Fully extensible via pluggable STT +and LLM provider interfaces. + +--- + +## Story 1: Core pipeline — commonMain + Android, raw transcript [STATUS: COMPLETE] + +**Goal**: End-to-end wire from mic tap to journal insert. No LLM formatting. Raw transcript +appended verbatim. Validates the entire pipeline plumbing before adding complexity. + +**Acceptance criteria**: All met. PR #2 open, CI passing. +- User taps mic button in Android bottom bar → `Recording` state (pulsing red indicator) +- User taps stop → `Transcribing` state (spinner) +- After transcription → `Done` state; transcript appended to today's journal as a timestamped + block with the raw text +- `VoiceCaptureState.Error` surfaces a dismissible message for permission denied, network error, + and empty transcript scenarios +- Pipeline cancels cleanly on back press or app backgrounding +- Temp `.m4a` file is deleted in all exit paths (success, failure, cancel) + +### Tasks + +**T1.1 — Define core interfaces and value types** [Micro, 1h] [STATUS: COMPLETE] + +Create `AudioRecorder.kt`, `SpeechToTextProvider.kt`, `LlmFormatterProvider.kt`, +`VoiceCaptureState.kt` in `commonMain/voice/`. Each file contains: +- The `suspend fun interface` or `sealed interface` +- The `NoOp` default implementation +- The sealed result type (`TranscriptResult`, `LlmResult`) +- `PlatformAudioFile` value class in `AudioRecorder.kt` + +No logic — pure interface definitions. Tests can be written against these immediately. + +**T1.2 — `VoicePipelineConfig` data class** [Micro, 1h] [STATUS: COMPLETE] + +Create `VoicePipelineConfig.kt` bundling `audioRecorder`, `sttProvider`, `llmProvider`, and +`systemPrompt`. Default constructs all `NoOp` providers. Define +`DEFAULT_VOICE_SYSTEM_PROMPT` constant here. + +**T1.3 — `VoiceCaptureViewModel`** [Medium, 3h] [STATUS: COMPLETE] + +Create `VoiceCaptureViewModel` in `commonMain/voice/`. Responsibilities: +- `StateFlow` initialized to `Idle` +- `onMicTapped()`: if `Idle`, start pipeline `Job`; if `Recording`, call `audioRecorder.stopRecording()` +- `cancel()`: cancel `pipelineJob`, reset to `Idle` +- `dismissError()`: reset to `Idle` +- Pipeline sequence: `recordToFile()` → `transcribe()` → (NoOp LLM) → `journalService.appendToToday()` +- Temp file cleanup in `finally` block after `transcribe()` returns +- Word-count gate: if `transcript.split().size < 10`, emit `TranscriptResult.Empty` and skip LLM +- Journal insert: `"${timestamp}\n${formattedText}\n\n---\nRaw: ${rawTranscript}"` as a new block +- `Done` carries `insertedText` for the UI confirmation message + +Tests: inject `FakeAudioRecorder`, `FakeSpeechToTextProvider` (returns fixed transcript), +`NoOpLlmFormatterProvider`. Verify state transitions for success, empty transcript, network error, +and cancel paths. + +**T1.4 — `WhisperSpeechToTextProvider`** [Small, 2h] [STATUS: COMPLETE] + +Create `WhisperSpeechToTextProvider(httpClient: HttpClient, apiKey: String)` in `commonMain`. +- Ktor multipart POST to `https://api.openai.com/v1/audio/transcriptions` +- Form fields: `file` (`.m4a` bytes), `model` (`gpt-4o-mini-transcribe`), `response_format` (`text`) +- Maps HTTP 401 → `TranscriptResult.Failure.ApiError(401, "Invalid API key")` +- Maps HTTP 429 → `TranscriptResult.Failure.ApiError(429, "Rate limit exceeded")` +- Maps network exception → `TranscriptResult.Failure.NetworkError` +- Maps empty/whitespace-only response → `TranscriptResult.Empty` +- Does NOT implement the `< 10 word` gate — that lives in `VoiceCaptureViewModel` + +Tests: use `MockEngine` (already in `jvmTest` dependencies) to verify multipart request shape, +HTTP 200 success, HTTP 401 error mapping, and empty response mapping. + +**T1.5 — `AndroidAudioRecorder`** [Large, 4h] [STATUS: COMPLETE] + +Create `AndroidAudioRecorder(context: Context)` in `androidMain`. Critical implementation details: + +- Audio source: `AudioSource.VOICE_COMMUNICATION` (not `DEFAULT`) — applies system noise + cancellation +- `AudioRecord` configuration: 16 kHz sample rate, `AudioFormat.ENCODING_PCM_16BIT`, mono +- `MediaCodec` AAC encoder: 128 kbps, 44.1 kHz output (Whisper-compatible) +- Output: temp file in `context.cacheDir` with `.m4a` extension +- Audio focus: request `AudioManager.AUDIOFOCUS_GAIN_TRANSIENT` before starting; register + `OnAudioFocusChangeListener`; on `AUDIOFOCUS_LOSS_TRANSIENT`, pause the read loop; on + `AUDIOFOCUS_LOSS`, stop recording; abandon focus after stop +- Recording loop: runs on `Dispatchers.IO`; reads PCM chunks from `AudioRecord` into a + `ByteArray` buffer; feeds chunks to `MediaCodec` encoder +- `stopRecording()`: signals the recording loop to stop, drains the encoder, flushes the + muxer, closes the file +- Permission: wrap `AudioRecord.startRecording()` in `try/catch(SecurityException)`; + on `SecurityException`, return `PlatformAudioFile("")` and emit a special error signal to + the ViewModel (via a `MutableStateFlow` `permissionDenied` property on the interface + — or by throwing a custom sealed exception that the ViewModel maps to `Error(RECORDING, ...)`) + +Lifecycle: the `AndroidAudioRecorder` instance is created in `MainActivity` and lives as long +as the `VoiceCaptureViewModel`. It does not hold an `Activity` context. + +Tests: unit test the PCM→AAC pipeline with a short fixed PCM input. Integration test requires +an Android device or Robolectric with audio mocks. + +**T1.6 — Wire into `App.kt` and `PlatformBottomBar`** [Medium, 3h] [STATUS: COMPLETE] + +- Add `voicePipeline: VoicePipelineConfig = remember { VoicePipelineConfig() }` to `StelekitApp` +- In `GraphContent`: create `VoiceCaptureViewModel` with `remember { ... }` (same pattern as + `JournalsViewModel`) +- Add `ON_PAUSE` lifecycle observer to call `voiceCaptureViewModel.cancel()` (stops recording + if app is backgrounded) +- Add `voiceCaptureButton: @Composable () -> Unit = {}` slot to `PlatformBottomBar.android.kt` +- Pass `voiceCaptureButton = { VoiceCaptureButton(state, onClick) }` from `GraphContent` +- Wire `AndroidAudioRecorder`, `WhisperSpeechToTextProvider` in `MainActivity` + +**T1.7 — `VoiceCaptureButton` composable** [Small, 2h] [STATUS: COMPLETE] + +Create `VoiceCaptureButton(state: VoiceCaptureState, onTap: () -> Unit, onDismissError: () -> Unit)` +in `commonMain/ui/components/`. + +State-to-visual mapping: +- `Idle`: `Icons.Outlined.Mic`, normal FAB size +- `Recording`: pulsing red `CircleShape` background + `Icons.Filled.Stop`, accessibility + label "Stop recording" +- `Transcribing`: `CircularProgressIndicator` + label "Transcribing..." +- `Formatting`: `CircularProgressIndicator` + label "Formatting..." +- `Done`: `Icons.Filled.Check` (green), auto-resets after 3 seconds via + `LaunchedEffect(state) { delay(3000); onAutoReset() }` +- `Error`: `Icons.Filled.ErrorOutline` (red); tapping calls `onDismissError()` + +The pulse animation for `Recording` uses `animateFloat` with `RepeatMode.Reverse`. + +Screenshot test via Roborazzi for each state variant. + +--- + +## Story 2: LLM formatting + settings [STATUS: READY TO BEGIN] + +**Goal**: Format the raw transcript into Logseq outliner syntax via Claude or OpenAI. User can +configure API keys in Settings. Word-count guard blocks LLM call for empty/silence recordings. + +**Acceptance criteria**: +- Settings screen has a "Voice Capture" section with fields for Whisper API key, Anthropic + Claude key, and OpenAI key +- After saving a key, the corresponding provider is activated and the mic button works + end-to-end with LLM formatting +- Journal entry contains: formatted outliner block + collapsible raw transcript block below +- `< 10 word` transcripts skip LLM and append raw transcript directly (no API cost) +- LLM failure falls back to raw transcript insert with an error notification + +### Tasks + +**T2.1 — `ClaudeLlmFormatterProvider`** [Small, 2h] + +Create `ClaudeLlmFormatterProvider(httpClient: HttpClient, apiKey: String)` in `commonMain`. +- Ktor POST to `https://api.anthropic.com/v1/messages` +- Mirrors `ClaudeTopicEnricher.kt` structure exactly +- Request: `model = "claude-haiku-4-5"`, `max_tokens = max(512, transcriptWordCount * 2)`, + `messages = [{"role": "user", "content": systemPrompt.replace("{{TRANSCRIPT}}", transcript)}]` +- Maps result to `LlmResult.Success(content)` or appropriate `LlmResult.Failure` variant +- Truncation detection: if `formattedText` last char is not `.`, `?`, `!`, `]`, or `\n`, + set a flag in `Done` state to show "Formatting may be incomplete" warning + +Tests: `MockEngine` for success, 401, 429, malformed JSON response. + +**T2.2 — `OpenAiLlmFormatterProvider`** [Small, 2h] + +Create `OpenAiLlmFormatterProvider(httpClient: HttpClient, apiKey: String, baseUrl: String = "https://api.openai.com")` +in `commonMain`. +- Ktor POST to `$baseUrl/v1/chat/completions` +- `model = "gpt-4o-mini"`, `messages = [{"role": "system", "content": systemPrompt}, {"role": "user", "content": transcript}]` +- Compatible with any OpenAI-compatible endpoint via `baseUrl` + +Tests: same pattern as `ClaudeLlmFormatterProvider`. + +**T2.3 — `VoiceSettings` — API key storage** [Small, 2h] + +Create `VoiceSettings` interface in `commonMain` with `expect` / `actual` for secure key +storage: +- `getWhisperApiKey(): String?` +- `setWhisperApiKey(key: String)` +- `getAnthropicKey(): String?` +- `setAnthropicKey(key: String)` +- `getOpenAiKey(): String?` +- `setOpenAiKey(key: String)` + +Android actual: Android `EncryptedSharedPreferences` (Jetpack Security Crypto). +iOS actual: `UserDefaults` for v1 (Keychain integration is Phase 2 hardening). +JVM actual: encrypted properties file in `~/.stelekit/`. + +**T2.4 — Settings UI for voice providers** [Medium, 3h] + +Add a "Voice Capture" section to `SettingsDialog.kt`: +- `OutlinedTextField` for each API key (masked input, `visualTransformation = PasswordVisualTransformation()`) +- Save button writes keys via `VoiceSettings` +- On save, `MainActivity` rebuilds `VoicePipelineConfig` with the new providers and calls a + `StelekitApp`-level recomposition trigger (or uses a `StateFlow`) +- Informational text: "Whisper key: OpenAI audio transcription (~$0.003/min). LLM key: formats + transcript into Logseq outliner syntax." +- "Use no formatting (append raw transcript)" toggle — sets `llmProvider = NoOpLlmFormatterProvider` + +**T2.5 — Connect LLM providers in `VoiceCaptureViewModel`** [Small, 2h] + +Update `VoiceCaptureViewModel` pipeline to use the injected `llmProvider`: +- After `TranscriptResult.Success`, emit `VoiceCaptureState.Formatting` +- Call `llmProvider.format(transcript, systemPrompt.replace("{{TRANSCRIPT}}", transcript))` +- On `LlmResult.Success(formatted)`: build journal block: + ``` + {{formatted}} + + #+BEGIN_QUOTE + Raw transcript: {{rawTranscript}} + #+END_QUOTE + ``` + Call `journalService.appendToToday(block)` +- On `LlmResult.Failure`: fall back to raw transcript insert; emit + `VoiceCaptureState.Error(LLM, message)` with a "Formatted using raw transcript" notification +- `max_tokens` hint: pass `(transcript.split(" ").size * 2).coerceAtLeast(512)` as + `systemPrompt` metadata — or add an optional `maxOutputTokens: Int` parameter to + `LlmFormatterProvider.format()` (evaluate during spike) + +**T2.6 — Journal block format** [Micro, 1h] + +Define the final journal block format as a constant and write a pure function test: +``` +- 📝 Voice note (HH:mm) + - [formatted bullet 1] + - [formatted bullet 2] + - ... + #+BEGIN_QUOTE + {{rawTranscript}} + #+END_QUOTE +``` +The `#+BEGIN_QUOTE` block is Logseq's collapsible quote syntax. Unit test the formatter with +known inputs → expected output. + +--- + +## Story 3: iOS adapter + processing state feedback + +**Goal**: iOS users have a working voice capture pipeline using `SFSpeechRecognizer`. Android +and iOS both show a visual waveform/pulse during recording to confirm the mic is active. + +**Acceptance criteria**: +- iOS: `IosAudioRecorder` records to `.m4a` in `NSTemporaryDirectory` +- iOS: `IosSpeechToTextProvider` uses `SFSpeechRecognizer` as Tier 1 (free), falls back to + Whisper API when unavailable +- Both platforms: `Recording` state shows a pulsing animation that reacts to audio level + (amplitude-based pulse scale) +- `AVAudioSession` category is set to `.record` before recording and restored to `.playback` + after stop +- iOS permission denial navigates user to Settings with an explanatory message + +### Tasks + +**T3.1 — `IosAudioRecorder`** [Large, 4h] + +Create `IosAudioRecorder` in `iosMain` using `AVAudioRecorder`. + +Critical sequence: +1. Check `AVAudioSession.recordPermission` — if `.denied`, return early (ViewModel detects + empty path and emits `Error(RECORDING, "Microphone access denied")`) +2. If `.undetermined`, call `requestRecordPermission` and await callback +3. **Before starting**: `AVAudioSession.sharedInstance().setCategory(.record, mode: .measurement)` + — this MUST precede `AVAudioRecorder.record()`. Forgetting it causes silent recording failure. +4. `AVAudioSession.setActive(true)` +5. Create `AVAudioRecorder` with URL in `NSTemporaryDirectory()` and AAC settings +6. `recorder.record()` +7. `stopRecording()`: call `recorder.stop()`, `AVAudioSession.setCategory(.playback)`, + `AVAudioSession.setActive(false)`, return file URL as `PlatformAudioFile` + +Handle `AVAudioSessionInterruptionNotification` (phone call): on interruption begin, call +`stopRecording()` and emit signal to ViewModel. + +**T3.2 — `IosSpeechToTextProvider`** [Large, 4h] + +Create `IosSpeechToTextProvider` in `iosMain` using `SFSpeechRecognizer`. +- `requiresOnDeviceRecognition = true` where `SFSpeechRecognizer.supportsOnDeviceRecognition` +- For recordings longer than 50 seconds: chunk the audio file at silence gaps and transcribe + each chunk separately, then concatenate with a single space separator +- `SFSpeechRecognizer.requestAuthorization` before first use +- Map `SFSpeechRecognizer` not available → fall back to `WhisperSpeechToTextProvider` +- Map authorization denied → `TranscriptResult.Failure.PermissionDenied` + +**T3.3 — Amplitude-reactive pulse in `VoiceCaptureButton`** [Small, 2h] + +Extend `AudioRecorder` interface with an optional `amplitudeFlow: Flow?` property +(defaults to `null`). `AndroidAudioRecorder` emits RMS amplitude from the `AudioRecord` read +loop. `VoiceCaptureButton` uses `amplitudeFlow` when non-null to scale the pulse animation +radius. Falls back to a fixed-period pulse when null (e.g., iOS v1 or NoOp). + +**T3.4 — iOS permission rationale screen** [Small, 2h] + +When `SFSpeechRecognizer` or microphone authorization is `.denied` on iOS, surface a bottom +sheet explaining why the permission is needed with a "Open Settings" button that deep-links +to `UIApplication.openSettingsURLString`. This replaces the generic `Error` state for the +permission denial case on iOS. + +--- + +## Known Issues + +### CRITICAL: AVAudioSession category must be set before recording starts [iOS] + +**Description**: `AVAudioSession.sharedInstance().setCategory(.record)` must be called before +`AVAudioRecorder.record()` (or `AVAudioEngine.start()`). If omitted, `AVAudioRecorder` starts +without error but records silence. Whisper transcribes silence as "Thank you." — a confusing +silent failure with no indication anything went wrong. + +**Mitigation**: +- Enforce via code ordering in `IosAudioRecorder`: category set is step 3, recording is step 6 +- Add a comment `// MUST precede recorder.record() — see ADR-001` at the category-set line +- Integration test: record 3 seconds on simulator, assert temp file size > 1 KB + +**Files**: `IosAudioRecorder.kt` (T3.1) + +--- + +### CRITICAL: Use `AudioRecord` not `MediaRecorder` on Android [Android] + +**Description**: `MediaRecorder` writes corrupted MP4 box headers when interrupted by a phone +call or audio focus loss on API < 24. The partial file may not be readable by Whisper. The +corruption is silent — no exception is thrown. + +**Mitigation**: +- `AndroidAudioRecorder` must use `AudioRecord` (raw PCM) + `MediaCodec` AAC encoder +- Implement `OnAudioFocusChangeListener`: on `AUDIOFOCUS_LOSS_TRANSIENT`, pause the read loop; + on `AUDIOFOCUS_GAIN`, resume +- Integration test: simulate audio focus loss mid-recording and verify the output `.m4a` is valid + +**Files**: `AndroidAudioRecorder.kt` (T1.5) + +--- + +### HIGH: Whisper silence hallucination produces fake transcript [All platforms] + +**Description**: If the user records in silence (e.g., forgot to speak, mic covered), Whisper +transcribes silence as "Thank you." or similar tokens. Without a guard, this calls the LLM +which formats "Thank you." into a bullet point appended to the journal. + +**Mitigation**: +- Word-count gate in `VoiceCaptureViewModel`: `if (transcript.split(" ").size < 10)` emit + `TranscriptResult.Empty` and surface "Nothing was captured — try again" +- Do NOT call the LLM on `TranscriptResult.Empty` +- Future: VAD (voice activity detection) gate before Whisper upload + +**Files**: `VoiceCaptureViewModel.kt` (T1.3), `WhisperSpeechToTextProvider.kt` (T1.4) + +--- + +### HIGH: `[[wikilink]]` hallucination creates dangling pages [All platforms] + +**Description**: LLMs invent `[[Page Names]]` that do not exist in the user's graph. Logseq and +SteleKit create stub pages for these links, polluting the graph with empty pages named after +hallucinated entities. + +**Mitigation**: +- System prompt constraint: "Add [[Page Name]] wiki links ONLY for proper nouns or topics + explicitly named in the transcript — do NOT invent links for terms not spoken" +- Always preserve raw transcript as collapsible `#+BEGIN_QUOTE` block below formatted output + so the user can verify what was said vs what the LLM produced +- Future (v2): pass graph page name index to LLM as context, constraining links to existing pages + +**Files**: `VoicePipelineConfig.kt` (T1.2), `VoiceCaptureViewModel.kt` (T2.5) + +--- + +### HIGH: Android 14+ foreground service requires `foregroundServiceType="microphone"` [Android, Phase 3] + +**Description**: For Phase 3 (lock-screen recording via foreground service), omitting +`android:foregroundServiceType="microphone"` in the manifest silently denies microphone access +with targetSdk >= 30. On targetSdk >= 34, the additional +`android.permission.FOREGROUND_SERVICE_MICROPHONE` permission is also required. Omitting either +produces no crash — just empty audio, exactly like the AVAudioSession pitfall. + +**Mitigation** (Phase 3 only — not required for Story 1-3 foreground recording): +- Manifest entry: `android:foregroundServiceType="microphone"` in the `` declaration +- Permission: `` +- The foreground service must be started while the app is still visible (before backgrounding) +- Add a checklist item to the Phase 3 story + +**Files**: `AndroidManifest.xml` (Phase 3) + +--- + +### MEDIUM: Android `SecurityException` on mid-session permission revocation [Android] + +**Description**: Android 12+ shows a green mic indicator when the microphone is active. A user +can tap it and revoke mic permission from the Privacy Dashboard while recording is in progress. +`AudioRecord.read()` then throws `SecurityException` in the recording coroutine. + +**Mitigation**: +- Wrap `AudioRecord.startRecording()` and the read loop in `try/catch(SecurityException)` +- On catch: stop the `AudioRecord`, return `PlatformAudioFile("")` +- `VoiceCaptureViewModel` maps empty path to `Error(RECORDING, "Microphone permission revoked")` +- Show a snackbar with a deep link to app permission settings + +**Files**: `AndroidAudioRecorder.kt` (T1.5) + +--- + +### MEDIUM: Temp file leak on app force-kill [All platforms] + +**Description**: The `.m4a` temp file is cleaned up in `VoiceCaptureViewModel`'s `finally` +block. If the process is force-killed mid-recording (OOM killer, etc.), the `finally` block +does not run and the temp file leaks in the cache directory. + +**Mitigation**: +- On `VoiceCaptureViewModel` initialization, scan `cacheDir` for `.m4a` files older than 1 hour + and delete them. A single scan on startup costs <1 ms. +- Log leaked files at `WARN` level: `[voice-capture] found leaked temp file: {path}, deleting` + +**Files**: `VoiceCaptureViewModel.kt` (T1.3) + +--- + +### LOW: LLM output truncation on long transcripts [All platforms] + +**Description**: If `max_tokens` is set too low or the model hits its output limit, the formatted +output is truncated mid-bullet. The last line may be malformed Logseq markdown. + +**Mitigation**: +- Set `max_tokens = (transcript.split(" ").size * 2).coerceAtLeast(512).coerceAtMost(4096)` +- Truncation detection: if `formattedText` last character is not `.`, `?`, `!`, `]`, or `\n`, + set a `isLikelyTruncated = true` flag in `VoiceCaptureState.Done` +- Show "Formatting may be incomplete — check the raw transcript" in the UI when flag is set + +**Files**: `VoiceCaptureViewModel.kt` (T2.5) + +--- + +## New Gradle Dependencies + +Story 1-3 require no new `commonMain` or `androidMain` dependencies. Ktor 3.1.3 and +`kotlinx-serialization` are already present. + +Story 3 (iOS) requires adding `Speech.framework` to the iOS cinterop definition in +`build.gradle.kts`: +```kotlin +val iosMain by getting { + compilations.getByName("main") { + cinterops { + val speech by creating { + defFile(project.file("src/nativeInterop/cinterop/speech.def")) + } + } + } +} +``` + +Phase 2 (ML Kit GenAI STT) requires adding to `androidMain`: +```kotlin +implementation("com.google.mlkit:genai-speech-recognition:1.0.0") // verify version +``` + +--- + +## Open Spikes + +These require short implementation spikes before the relevant story begins: + +1. **ML Kit GenAI STT for long recordings** (before Phase 2): The API provides streaming partial + + final results. Does it handle 5–10 minute recordings stably, or require chunking? Record a + 15-minute test on a Pixel 9. Answers whether `AndroidMlKitSpeechToTextProvider` needs a + chunking layer. + +2. **Optimal Logseq system prompt** (before T2.1): Test 10 diverse transcript samples against the + v1 system prompt. Measure: (a) correct `- ` bullet format, (b) correct 2-space indentation, + (c) absence of hallucinated `[[links]]`, (d) no preamble or summary. Iterate until >90% + compliance. Document winning prompt in `VoicePipelineConfig.kt`. + +3. **`StelekitApp` parameter threading strategy** (before T1.6): `StelekitApp` currently has 5 + parameters. Confirm that `VoicePipelineConfig` as a single 6th parameter is the right + aggregation level. Review whether `urlFetcher` should also be folded into a broader + `EnrichmentConfig` at the same time. + +4. **iOS `PlatformBottomBar` actual** (before T1.6): Confirm whether there is an `iosMain` actual + for `PlatformBottomBar` or if it shares the Android composable via `commonMain`. The mic + button slot must appear on both platforms. + +--- + +## Delivery Sequence + +**Story 1** is the mandatory first deliverable. It validates the full pipeline plumbing (mic → +file → STT → journal insert) without LLM cost or API key friction. The `NoOpLlmFormatterProvider` +default ensures the pipeline works even without Story 2. + +**Story 2** adds the value-delivering formatting step and the settings UX to configure API keys. +It can begin immediately after Story 1 is merged. + +**Story 3** adds iOS support and UX polish. It is independent of Story 2 and can run in parallel +once Story 1 interfaces are stable. + +**Phase 2 and 3** (ML Kit GenAI STT, foreground service, FoundationModels LLM, waveform +animation) are post-v1 enhancements. They do not require interface changes — all are new +implementations behind the existing `SpeechToTextProvider` and `LlmFormatterProvider` contracts. + +--- + +## Testing Strategy + +| Layer | Coverage target | Approach | +|-------|----------------|----------| +| `VoiceCaptureViewModel` | State machine transitions | Unit tests with fake providers in `businessTest` | +| `WhisperSpeechToTextProvider` | HTTP request shape + error mapping | `MockEngine` in `jvmTest` | +| `ClaudeLlmFormatterProvider` | HTTP request shape + error mapping | `MockEngine` in `jvmTest` | +| `OpenAiLlmFormatterProvider` | HTTP request shape + error mapping | `MockEngine` in `jvmTest` | +| `VoiceCaptureButton` | All 6 state variants | Roborazzi screenshot tests in `jvmTest` | +| `AndroidAudioRecorder` | PCM→AAC encoding pipeline | Unit test with fixed PCM input | +| `IosAudioRecorder` | Session configuration | iOS unit tests with `AVAudioSession` mock | +| Word-count gate | Boundary at 9 and 10 words | `businessTest` | +| Temp file cleanup | Finally block fires on cancel | `businessTest` with `FakeAudioRecorder` that tracks deletion calls | +| Journal block format | Known input → expected Logseq markdown | `businessTest` pure function test | + +Critical edge cases to cover: +- `SecurityException` during `AudioRecord.startRecording()` → `Error(RECORDING, ...)` +- `TranscriptResult.Empty` (< 10 words) → `Done` with no LLM call +- `LlmResult.Failure` → raw transcript fallback insert +- Cancel during `Transcribing` → temp file deleted, state → `Idle` +- Double-tap during `Recording` → only one `stopRecording()` call + +--- + +## References + +- ADR-001: Audio Capture Adapter — `project_plans/mobile-voice-mode/decisions/ADR-001-audio-capture-adapter.md` +- ADR-002: STT Provider Interface — `project_plans/mobile-voice-mode/decisions/ADR-002-stt-provider-interface.md` +- ADR-003: LLM Formatter Provider Interface — `project_plans/mobile-voice-mode/decisions/ADR-003-llm-formatter-provider-interface.md` +- ADR-004: Plugin Registration — `project_plans/mobile-voice-mode/decisions/ADR-004-plugin-registration.md` +- ADR-005: Voice Capture UI State Machine — `project_plans/mobile-voice-mode/decisions/ADR-005-voice-capture-ui-state-machine.md` +- Prior art seam: `project_plans/import-topic-suggestions/decisions/ADR-002-topic-enricher-plugin-interface.md` +- Research synthesis: `project_plans/mobile-voice-mode/research/synthesis.md` diff --git a/kmp/build.gradle.kts b/kmp/build.gradle.kts index 113f6964d..aea9db066 100644 --- a/kmp/build.gradle.kts +++ b/kmp/build.gradle.kts @@ -165,6 +165,9 @@ kotlin { // JankStats — zero-allocation frame jank classification implementation("androidx.metrics:metrics-performance:1.0.0-beta02") + + // Encrypted SharedPreferences for API key storage + implementation("androidx.security:security-crypto:1.1.0-alpha06") } } diff --git a/kmp/src/androidMain/kotlin/dev/stapler/stelekit/platform/PlatformSettings.android.kt b/kmp/src/androidMain/kotlin/dev/stapler/stelekit/platform/PlatformSettings.android.kt index f318e36de..f14f90317 100644 --- a/kmp/src/androidMain/kotlin/dev/stapler/stelekit/platform/PlatformSettings.android.kt +++ b/kmp/src/androidMain/kotlin/dev/stapler/stelekit/platform/PlatformSettings.android.kt @@ -2,6 +2,9 @@ package dev.stapler.stelekit.platform import android.content.Context import android.content.SharedPreferences +import android.util.Log +import androidx.security.crypto.EncryptedSharedPreferences +import androidx.security.crypto.MasterKey object SteleKitContext { private var _context: Context? = null @@ -16,10 +19,20 @@ object SteleKitContext { actual class PlatformSettings actual constructor() { private val prefs: SharedPreferences by lazy { try { - SteleKitContext.context.getSharedPreferences("stelekit_prefs", Context.MODE_PRIVATE) + val masterKey = MasterKey.Builder(SteleKitContext.context) + .setKeyScheme(MasterKey.KeyScheme.AES256_GCM) + .build() + EncryptedSharedPreferences.create( + SteleKitContext.context, + "stelekit_secure_prefs", + masterKey, + EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV, + EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM, + ) } catch (e: Exception) { - // Fallback for tests or if not initialized - throw IllegalStateException("Context not initialized", e) + // Fallback to plain prefs if keystore fails (e.g., corrupted keystore after device wipe) + Log.w("PlatformSettings", "EncryptedSharedPreferences unavailable, falling back to plain prefs", e) + SteleKitContext.context.getSharedPreferences("stelekit_prefs", Context.MODE_PRIVATE) } } diff --git a/kmp/src/androidMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.android.kt b/kmp/src/androidMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.android.kt index e028de1de..3dcf51872 100644 --- a/kmp/src/androidMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.android.kt +++ b/kmp/src/androidMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.android.kt @@ -1,7 +1,10 @@ package dev.stapler.stelekit.ui +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Spacer import androidx.compose.foundation.layout.WindowInsets import androidx.compose.foundation.layout.ime +import androidx.compose.foundation.layout.offset import androidx.compose.material.icons.Icons import androidx.compose.material.icons.automirrored.filled.List import androidx.compose.material.icons.filled.AutoStories @@ -12,8 +15,11 @@ import androidx.compose.material3.NavigationBar import androidx.compose.material3.NavigationBarItem import androidx.compose.material3.Text import androidx.compose.runtime.Composable +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier import androidx.compose.ui.graphics.vector.ImageVector import androidx.compose.ui.platform.LocalDensity +import androidx.compose.ui.unit.dp import dev.stapler.stelekit.ui.LocalWindowSizeClass import dev.stapler.stelekit.ui.isMobile @@ -40,7 +46,8 @@ actual fun PlatformBottomBar( currentScreen: Screen, onNavigate: (Screen) -> Unit, onSearch: () -> Unit, - isLeftHanded: Boolean + isLeftHanded: Boolean, + voiceCaptureButton: @Composable () -> Unit, ) { if (!LocalWindowSizeClass.current.isMobile) return // Hide the nav bar when the keyboard is open — the editing toolbar takes its place, @@ -48,21 +55,53 @@ actual fun PlatformBottomBar( val imeVisible = WindowInsets.ime.getBottom(LocalDensity.current) > 0 if (imeVisible) return val items = if (isLeftHanded) BottomNavItem.entries.reversed() else BottomNavItem.entries - NavigationBar { - items.forEach { item -> - NavigationBarItem( - selected = item.matchesScreen(currentScreen), - onClick = { - when (item) { - BottomNavItem.SEARCH -> onSearch() - BottomNavItem.JOURNALS -> onNavigate(Screen.Journals) - BottomNavItem.ALL_PAGES -> onNavigate(Screen.AllPages) - BottomNavItem.NOTIFICATIONS -> onNavigate(Screen.Notifications) - } - }, - icon = { Icon(item.icon, contentDescription = null) }, - label = { Text(item.label) } - ) + // Split nav items 2 left + 2 right; center gap reserved for the FAB. + val leftItems = items.take(2) + val rightItems = items.drop(2) + + Box { + NavigationBar { + leftItems.forEach { item -> + NavigationBarItem( + selected = item.matchesScreen(currentScreen), + onClick = { + when (item) { + BottomNavItem.SEARCH -> onSearch() + BottomNavItem.JOURNALS -> onNavigate(Screen.Journals) + BottomNavItem.ALL_PAGES -> onNavigate(Screen.AllPages) + BottomNavItem.NOTIFICATIONS -> onNavigate(Screen.Notifications) + } + }, + icon = { Icon(item.icon, contentDescription = item.label) }, + label = { Text(item.label) }, + ) + } + // Center gap — same weight as one NavigationBarItem — gives the FAB clear space. + Spacer(modifier = Modifier.weight(1f)) + rightItems.forEach { item -> + NavigationBarItem( + selected = item.matchesScreen(currentScreen), + onClick = { + when (item) { + BottomNavItem.SEARCH -> onSearch() + BottomNavItem.JOURNALS -> onNavigate(Screen.Journals) + BottomNavItem.ALL_PAGES -> onNavigate(Screen.AllPages) + BottomNavItem.NOTIFICATIONS -> onNavigate(Screen.Notifications) + } + }, + icon = { Icon(item.icon, contentDescription = item.label) }, + label = { Text(item.label) }, + ) + } + } + // FAB centered in the gap; offset upward by half the standard FAB height (56dp / 2) + // so the FAB sits on top of the nav bar edge without obscuring any nav items. + Box( + modifier = Modifier + .align(Alignment.TopCenter) + .offset(y = (-28).dp), + ) { + voiceCaptureButton() } } } diff --git a/kmp/src/androidMain/kotlin/dev/stapler/stelekit/voice/AndroidAudioRecorder.kt b/kmp/src/androidMain/kotlin/dev/stapler/stelekit/voice/AndroidAudioRecorder.kt new file mode 100644 index 000000000..3c17caf5c --- /dev/null +++ b/kmp/src/androidMain/kotlin/dev/stapler/stelekit/voice/AndroidAudioRecorder.kt @@ -0,0 +1,270 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import android.content.Context +import android.media.AudioFormat +import android.media.AudioManager +import android.media.AudioRecord +import android.media.MediaCodec +import android.media.MediaCodecInfo +import android.media.MediaFormat +import android.media.MediaMuxer +import android.media.MediaRecorder +import android.media.AudioFocusRequest +import android.os.Build +import android.util.Log +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.flow.Flow +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.asStateFlow +import kotlinx.coroutines.isActive +import kotlinx.coroutines.withContext +import java.io.File +import kotlin.math.sqrt + +private const val TAG = "AndroidAudioRecorder" + +class AndroidAudioRecorder(private val context: Context) : AudioRecorder { + + companion object { + private const val SAMPLE_RATE = 16_000 + private const val CHANNEL_CONFIG = AudioFormat.CHANNEL_IN_MONO + private const val AUDIO_FORMAT = AudioFormat.ENCODING_PCM_16BIT + private const val BIT_RATE = 128_000 + private const val MIME_TYPE = "audio/mp4a-latm" + private const val CODEC_TIMEOUT_US = 10_000L + // 4× minimum buffer gives the encoder enough headroom to avoid under-runs on loaded devices. + private const val BUFFER_SIZE_MULTIPLIER = 4 + // Floor ensures a viable buffer even if getMinBufferSize returns an unexpectedly small value. + private const val MIN_BUFFER_SIZE = 8192 + // Non-blocking poll interval while audio focus is transiently lost (e.g. incoming call). + private const val PAUSE_POLL_INTERVAL_MS = 50L + // Normalisation divisor for RMS amplitude → [0, 1] float. + private const val SHORT_MAX = Short.MAX_VALUE.toFloat() + } + + private val _amplitudeFlow = MutableStateFlow(0f) + override val amplitudeFlow: Flow = _amplitudeFlow.asStateFlow() + + @Volatile private var stopRequested = false + @Volatile private var pauseRequested = false + + override suspend fun startRecording(): PlatformAudioFile = withContext(Dispatchers.IO) { + stopRequested = false + pauseRequested = false + + val outputFile = File(context.cacheDir, "voice_${System.currentTimeMillis()}.m4a") + val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager + + val focusChangeListener = AudioManager.OnAudioFocusChangeListener { change -> + when (change) { + AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> pauseRequested = true + AudioManager.AUDIOFOCUS_GAIN -> pauseRequested = false + AudioManager.AUDIOFOCUS_LOSS -> stopRequested = true + } + } + val focusRequest = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) { + AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN_TRANSIENT) + .setOnAudioFocusChangeListener(focusChangeListener) + .build() + .also { audioManager.requestAudioFocus(it) } + } else { + @Suppress("DEPRECATION") + audioManager.requestAudioFocus( + focusChangeListener, + AudioManager.STREAM_MUSIC, + AudioManager.AUDIOFOCUS_GAIN_TRANSIENT, + ) + null + } + + val minBuf = AudioRecord.getMinBufferSize(SAMPLE_RATE, CHANNEL_CONFIG, AUDIO_FORMAT) + val bufferSize = maxOf(minBuf * BUFFER_SIZE_MULTIPLIER, MIN_BUFFER_SIZE) + + var audioRecord: AudioRecord? = null + var mediaCodec: MediaCodec? = null + var mediaMuxer: MediaMuxer? = null + + try { + audioRecord = try { + AudioRecord( + MediaRecorder.AudioSource.VOICE_COMMUNICATION, + SAMPLE_RATE, + CHANNEL_CONFIG, + AUDIO_FORMAT, + bufferSize, + ) + } catch (e: SecurityException) { + abandonAudioFocus(audioManager, focusRequest, focusChangeListener) + return@withContext PlatformAudioFile("") + } + + if (audioRecord.state != AudioRecord.STATE_INITIALIZED) { + audioRecord.release() + abandonAudioFocus(audioManager, focusRequest, focusChangeListener) + return@withContext PlatformAudioFile("") + } + + mediaCodec = setupMediaCodec(bufferSize) + mediaMuxer = setupMediaMuxer(outputFile) + + audioRecord.startRecording() + + val pcmBuffer = ByteArray(bufferSize) + val bufferInfo = MediaCodec.BufferInfo() + var muxerStarted = false + var muxerTrackIndex = -1 + var presentationTimeUs = 0L + + // isActive checks for coroutine cancellation; stopRequested handles user-initiated stop. + while (!stopRequested && isActive) { + if (pauseRequested) { + kotlinx.coroutines.delay(PAUSE_POLL_INTERVAL_MS) + continue + } + + val bytesRead = audioRecord.read(pcmBuffer, 0, pcmBuffer.size) + if (bytesRead <= 0) continue + + _amplitudeFlow.value = computeRms(pcmBuffer, bytesRead) + + // Feed PCM to encoder + val inputIdx = mediaCodec.dequeueInputBuffer(CODEC_TIMEOUT_US) + if (inputIdx >= 0) { + val inputBuf = mediaCodec.getInputBuffer(inputIdx)!! + inputBuf.clear() + inputBuf.put(pcmBuffer, 0, bytesRead) + presentationTimeUs += (bytesRead.toLong() * 1_000_000L) / (SAMPLE_RATE * 2L) + mediaCodec.queueInputBuffer(inputIdx, 0, bytesRead, presentationTimeUs, 0) + } + + // Drain encoder output; onFormatChanged starts the muxer on first invocation. + drainEncoder(mediaCodec, mediaMuxer, bufferInfo, muxerStarted, muxerTrackIndex, 0) { trackIdx -> + muxerTrackIndex = trackIdx + if (!muxerStarted) { + mediaMuxer.start() + muxerStarted = true + } + } + } + + // Signal EOS and drain remaining frames to produce a valid .m4a file. + val inputIdx = mediaCodec.dequeueInputBuffer(CODEC_TIMEOUT_US) + if (inputIdx >= 0) { + mediaCodec.queueInputBuffer( + inputIdx, 0, 0, presentationTimeUs, + MediaCodec.BUFFER_FLAG_END_OF_STREAM, + ) + } + drainEncoder(mediaCodec, mediaMuxer, bufferInfo, muxerStarted, muxerTrackIndex, CODEC_TIMEOUT_US) + + PlatformAudioFile(outputFile.absolutePath) + } catch (e: SecurityException) { + PlatformAudioFile("") + } finally { + runCatching { audioRecord?.stop() } + .onFailure { Log.w(TAG, "audioRecord.stop() failed", it) } + runCatching { audioRecord?.release() } + .onFailure { Log.w(TAG, "audioRecord.release() failed", it) } + runCatching { mediaCodec?.stop() } + .onFailure { Log.w(TAG, "mediaCodec.stop() failed", it) } + runCatching { mediaCodec?.release() } + .onFailure { Log.w(TAG, "mediaCodec.release() failed", it) } + runCatching { mediaMuxer?.stop() } + .onFailure { Log.w(TAG, "mediaMuxer.stop() failed", it) } + runCatching { mediaMuxer?.release() } + .onFailure { Log.w(TAG, "mediaMuxer.release() failed", it) } + abandonAudioFocus(audioManager, focusRequest, focusChangeListener) + _amplitudeFlow.value = 0f + } + } + + override suspend fun stopRecording() { + stopRequested = true + } + + override suspend fun readBytes(file: PlatformAudioFile): ByteArray = + if (file.isEmpty) ByteArray(0) + else withContext(Dispatchers.IO) { File(file.path).readBytes() } + + override fun deleteRecording(file: PlatformAudioFile) { + if (!file.isEmpty) File(file.path).delete() + } + + private fun setupMediaCodec(bufferSize: Int): MediaCodec { + val codec = MediaCodec.createEncoderByType(MIME_TYPE) + val format = MediaFormat.createAudioFormat(MIME_TYPE, SAMPLE_RATE, 1).apply { + setInteger(MediaFormat.KEY_BIT_RATE, BIT_RATE) + setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC) + setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, bufferSize) + } + codec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE) + codec.start() + return codec + } + + private fun setupMediaMuxer(outputFile: File): MediaMuxer = + MediaMuxer(outputFile.absolutePath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4) + + private fun computeRms(buffer: ByteArray, bytesRead: Int): Float { + var sumSq = 0.0 + for (i in 0 until bytesRead - 1 step 2) { + val sample = ((buffer[i + 1].toInt() shl 8) or (buffer[i].toInt() and 0xFF)).toShort().toInt() + sumSq += sample.toDouble() * sample.toDouble() + } + return (sqrt(sumSq / (bytesRead / 2)).toFloat() / SHORT_MAX).coerceIn(0f, 1f) + } + + private fun abandonAudioFocus( + audioManager: AudioManager, + focusRequest: AudioFocusRequest?, + listener: AudioManager.OnAudioFocusChangeListener, + ) { + if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O && focusRequest != null) { + audioManager.abandonAudioFocusRequest(focusRequest) + } else { + @Suppress("DEPRECATION") + audioManager.abandonAudioFocus(listener) + } + } + + /** + * Drains encoded output from [codec] into [muxer]. Stops at INFO_TRY_AGAIN_LATER or EOS. + * [timeoutUs] 0 = non-blocking (use during the record loop); CODEC_TIMEOUT_US = blocking + * (use after signalling EOS to ensure all frames are flushed). + * [onFormatChanged] is called once when the output format is known; it should add the muxer + * track and start the muxer. + */ + private fun drainEncoder( + codec: MediaCodec, + muxer: MediaMuxer, + info: MediaCodec.BufferInfo, + muxerStarted: Boolean, + trackIndex: Int, + timeoutUs: Long, + onFormatChanged: ((Int) -> Unit)? = null, + ) { + while (true) { + val idx = codec.dequeueOutputBuffer(info, timeoutUs) + when { + idx == MediaCodec.INFO_TRY_AGAIN_LATER -> break + idx == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> { + if (onFormatChanged != null) { + val newTrack = muxer.addTrack(codec.outputFormat) + onFormatChanged(newTrack) + } + } + idx >= 0 -> { + val buf = codec.getOutputBuffer(idx)!! + val isConfig = info.flags and MediaCodec.BUFFER_FLAG_CODEC_CONFIG != 0 + if (!isConfig && muxerStarted && trackIndex >= 0 && info.size > 0) { + muxer.writeSampleData(trackIndex, buf, info) + } + codec.releaseOutputBuffer(idx, false) + if (info.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM != 0) break + } + } + } + } +} diff --git a/kmp/src/businessTest/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModelTest.kt b/kmp/src/businessTest/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModelTest.kt new file mode 100644 index 000000000..0803022ac --- /dev/null +++ b/kmp/src/businessTest/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModelTest.kt @@ -0,0 +1,479 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import dev.stapler.stelekit.repository.InMemoryBlockRepository +import dev.stapler.stelekit.repository.InMemoryPageRepository +import dev.stapler.stelekit.repository.JournalService +import kotlinx.coroutines.cancel +import kotlinx.coroutines.delay +import kotlinx.coroutines.flow.first +import kotlinx.coroutines.launch +import kotlinx.coroutines.test.advanceTimeBy +import kotlinx.coroutines.test.advanceUntilIdle +import kotlinx.coroutines.test.runTest +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertIs + +class VoiceCaptureViewModelTest { + + private fun makeJournalService() = + JournalService(InMemoryPageRepository(), InMemoryBlockRepository()) + + @Test + fun `initial state is Idle`() = runTest { + val vm = VoiceCaptureViewModel(VoicePipelineConfig(), makeJournalService(), this) + assertIs(vm.state.first()) + } + + @Test + fun `success path reaches Done state`() = runTest { + val transcript = "this is a test transcript with more than ten words total here" + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success(transcript) } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + assertIs(vm.state.first()) + } + + @Test + fun `word-count gate under 10 words emits Error at TRANSCRIBING`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success("too short") } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assertEquals(PipelineStage.TRANSCRIBING, state.stage) + } + + @Test + fun `permission denied (empty path) emits Error at RECORDING`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("") + override suspend fun stopRecording() = Unit + } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assertEquals(PipelineStage.RECORDING, state.stage) + } + + @Test + fun `cancel during Recording resets to Idle`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile { + delay(10_000) + return PlatformAudioFile("") + } + override suspend fun stopRecording() = Unit + } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder), + makeJournalService(), this, + ) + + vm.onMicTapped() + // Let coroutine start and reach Recording state + delay(1) + assertIs(vm.state.first()) + + vm.cancel() + assertIs(vm.state.first()) + } + + @Test + fun `dismissError resets to Idle`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("") + override suspend fun stopRecording() = Unit + } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + assertIs(vm.state.first()) + + vm.dismissError() + assertIs(vm.state.first()) + } + + @Test + fun `STT NetworkError emits Error at TRANSCRIBING`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Failure.NetworkError } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assertEquals(PipelineStage.TRANSCRIBING, state.stage) + } + + @Test + fun `temp file deleted in finally block on success`() = runTest { + var deletedPath: String? = null + val transcript = "this is a test transcript with more than ten words total here long enough" + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/voice.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + override fun deleteRecording(file: PlatformAudioFile) { deletedPath = file.path } + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success(transcript) } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + assertEquals("/tmp/voice.m4a", deletedPath) + } + + @Test + fun `temp file deleted in finally block on cancel`() = runTest { + var deletedPath: String? = null + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile { + delay(10_000) + return PlatformAudioFile("/tmp/voice.m4a") + } + override suspend fun stopRecording() = Unit + override fun deleteRecording(file: PlatformAudioFile) { deletedPath = file.path } + } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder), + makeJournalService(), this, + ) + + vm.onMicTapped() + delay(1) + vm.cancel() + + // Empty path because startRecording never returned, so no file to delete + assertEquals(null, deletedPath) + } + + @Test + fun `temp file deleted when cancel fires after startRecording returns`() = runTest { + var deletedPath: String? = null + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile { + delay(10_000) + return PlatformAudioFile("/tmp/voice_cancel.m4a") + } + override suspend fun stopRecording() { + // Unblock startRecording by advancing time + } + override suspend fun readBytes(file: PlatformAudioFile): ByteArray { + delay(10_000) // hang during transcription so cancel can fire + return ByteArray(0) + } + override fun deleteRecording(file: PlatformAudioFile) { deletedPath = file.path } + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Empty } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + // Advance past startRecording's delay so the file is assigned, then stop + advanceTimeBy(10_001) + // Now pipeline is in Transcribing (readBytes is hanging) + assertIs(vm.state.first()) + vm.cancel() + advanceUntilIdle() + + assertEquals("/tmp/voice_cancel.m4a", deletedPath) + } + + @Test + fun `temp file deleted on STT failure`() = runTest { + var deletedPath: String? = null + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/stt_fail.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + override fun deleteRecording(file: PlatformAudioFile) { deletedPath = file.path } + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Failure.NetworkError } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + assertEquals("/tmp/stt_fail.m4a", deletedPath) + } + + @Test + fun `STT Empty result emits Error at TRANSCRIBING`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Empty } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assertEquals(PipelineStage.TRANSCRIBING, state.stage) + } + + @Test + fun `STT PermissionDenied emits Error at RECORDING`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Failure.PermissionDenied } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assertEquals(PipelineStage.RECORDING, state.stage) + } + + @Test + fun `success path passes through Formatting state`() = runTest { + val transcript = "this is a test transcript with more than ten words total here" + var formattingObserved = false + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success(transcript) } + val fakeLlm = LlmFormatterProvider { _, _ -> + delay(1) // yield so we can observe Formatting state + LlmResult.Success("- formatted", false) + } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt, llmProvider = fakeLlm), + makeJournalService(), this, + ) + + val collectionJob = launch { + vm.state.collect { if (it == VoiceCaptureState.Formatting) formattingObserved = true } + } + vm.onMicTapped() + advanceUntilIdle() + collectionJob.cancel() + + assert(formattingObserved) { "Formatting state was never observed" } + assertIs(vm.state.first()) + } + + @Test + fun `LLM failure falls back to raw transcript in Done state`() = runTest { + val transcript = "this is a test transcript with more than ten words total here" + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success(transcript) } + val fakeLlm = LlmFormatterProvider { _, _ -> LlmResult.Failure.NetworkError } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt, llmProvider = fakeLlm), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assert(state.insertedText.contains(transcript.trim())) { + "Expected Done.insertedText to contain raw transcript but got: ${state.insertedText}" + } + } + + @Test + fun `LLM ApiError also falls back to raw transcript in Done state`() = runTest { + val transcript = "this is a test transcript with more than ten words total here" + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success(transcript) } + val fakeLlm = LlmFormatterProvider { _, _ -> LlmResult.Failure.ApiError(401, "Invalid API key") } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt, llmProvider = fakeLlm), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assert(state.insertedText.contains(transcript.trim())) { + "Expected Done.insertedText to contain raw transcript on LLM ApiError but got: ${state.insertedText}" + } + } + + @Test + fun `9-word transcript emits Error at TRANSCRIBING`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success("one two three four five six seven eight nine") } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + val state = vm.state.first() + assertIs(state) + assertEquals(PipelineStage.TRANSCRIBING, state.stage) + } + + @Test + fun `10-word transcript reaches Done state`() = runTest { + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success("one two three four five six seven eight nine ten") } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + assertIs(vm.state.first()) + } + + @Test + fun `transcript over 10000 chars is truncated before LLM`() = runTest { + val longTranscript = "word ".repeat(2_500) // 12,500 chars + var receivedTranscript = "" + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("/tmp/test.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success(longTranscript) } + val fakeLlm = LlmFormatterProvider { transcript, _ -> + receivedTranscript = transcript + LlmResult.Success("- formatted.", false) + } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt, llmProvider = fakeLlm), + makeJournalService(), this, + ) + + vm.onMicTapped() + advanceUntilIdle() + + assert(receivedTranscript.length <= 10_000) { + "LLM received ${receivedTranscript.length} chars, expected ≤ 10,000" + } + val state = vm.state.first() + assertIs(state) + assert(state.isLikelyTruncated) { "Expected isLikelyTruncated=true for over-length transcript" } + } + + @Test + fun `onMicTapped while Recording calls stopRecording and reaches Transcribing`() = runTest { + var stopCalled = false + val fakeRecorder = object : AudioRecorder { + private var stopped = false + override suspend fun startRecording(): PlatformAudioFile { + // Suspend until stop is signalled + while (!stopped) delay(10) + return PlatformAudioFile("/tmp/test.m4a") + } + override suspend fun stopRecording() { + stopCalled = true + stopped = true + } + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(0) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Empty } + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + makeJournalService(), this, + ) + + vm.onMicTapped() + delay(1) + assertIs(vm.state.first()) + + vm.onMicTapped() // should call stopRecording + advanceUntilIdle() + + assert(stopCalled) { "stopRecording was not called" } + // After stop, startRecording returns the file, pipeline proceeds to Transcribing/Error + assertIs(vm.state.first()) // Empty transcript → Error + } +} diff --git a/kmp/src/businessTest/kotlin/dev/stapler/stelekit/voice/VoiceNoteBlockFormatTest.kt b/kmp/src/businessTest/kotlin/dev/stapler/stelekit/voice/VoiceNoteBlockFormatTest.kt new file mode 100644 index 000000000..3c99894c4 --- /dev/null +++ b/kmp/src/businessTest/kotlin/dev/stapler/stelekit/voice/VoiceNoteBlockFormatTest.kt @@ -0,0 +1,95 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import dev.stapler.stelekit.repository.InMemoryBlockRepository +import dev.stapler.stelekit.repository.InMemoryPageRepository +import dev.stapler.stelekit.repository.JournalService +import kotlinx.coroutines.flow.first +import kotlinx.coroutines.test.advanceUntilIdle +import kotlinx.coroutines.test.runTest +import kotlin.test.Test +import kotlin.test.assertIs +import kotlin.test.assertNotNull +import kotlin.test.assertTrue + +class VoiceNoteBlockFormatTest { + + private fun makeViewModel(scope: kotlinx.coroutines.CoroutineScope) = VoiceCaptureViewModel( + VoicePipelineConfig(), + JournalService(InMemoryPageRepository(), InMemoryBlockRepository()), + scope, + ) + + @Test + fun `block starts with voice note header line`() = runTest { + val block = makeViewModel(this).buildVoiceNoteBlock("- formatted bullet.", "raw transcript text") + assertTrue(block.startsWith("- 📝 Voice note ("), "Expected block to start with '- 📝 Voice note (', got: $block") + } + + @Test + fun `block contains formatted text`() = runTest { + val formatted = "- point one\n- point two." + val block = makeViewModel(this).buildVoiceNoteBlock(formatted, "raw transcript") + assertTrue(block.contains("point one"), "Expected formatted text in block") + assertTrue(block.contains("point two"), "Expected formatted text in block") + } + + @Test + fun `block contains raw transcript in BEGIN_QUOTE block`() = runTest { + val raw = "this is the raw transcript text" + val block = makeViewModel(this).buildVoiceNoteBlock("- formatted.", raw) + assertTrue(block.contains("#+BEGIN_QUOTE"), "Expected #+BEGIN_QUOTE in block") + assertTrue(block.contains(raw), "Expected raw transcript in #+END_QUOTE block") + assertTrue(block.contains("#+END_QUOTE"), "Expected #+END_QUOTE in block") + } + + @Test + fun `multiline formatted text has each line indented under header`() = runTest { + val formatted = "- line one\n- line two\n- line three." + val block = makeViewModel(this).buildVoiceNoteBlock(formatted, "raw") + assertTrue(block.contains("line one"), "Expected 'line one' in block") + assertTrue(block.contains("line two"), "Expected 'line two' in block") + assertTrue(block.contains("line three"), "Expected 'line three' in block") + } + + @Test + fun `timestamp in header has zero-padded hours and minutes`() = runTest { + val block = makeViewModel(this).buildVoiceNoteBlock("- formatted.", "raw") + val headerLine = block.lines().first() + val timeRegex = Regex("""- 📝 Voice note \(\d{2}:\d{2}\)""") + assertTrue(timeRegex.containsMatchIn(headerLine), "Expected HH:mm timestamp in header, got: $headerLine") + } + + @Test + fun `success pipeline stores block with correct structure`() = runTest { + val transcript = "one two three four five six seven eight nine ten eleven" + val fakeRecorder = object : AudioRecorder { + override suspend fun startRecording() = PlatformAudioFile("/tmp/t.m4a") + override suspend fun stopRecording() = Unit + override suspend fun readBytes(file: PlatformAudioFile) = ByteArray(100) + } + val fakeStt = SpeechToTextProvider { _ -> TranscriptResult.Success(transcript) } + val blockRepo = InMemoryBlockRepository() + val fakeJournal = JournalService(InMemoryPageRepository(), blockRepo) + + val vm = VoiceCaptureViewModel( + VoicePipelineConfig(audioRecorder = fakeRecorder, sttProvider = fakeStt), + fakeJournal, + this, + ) + vm.onMicTapped() + advanceUntilIdle() + + assertIs(vm.state.value) + val page = fakeJournal.ensureTodayJournal() + assertTrue(page.uuid.isNotBlank()) + // Verify the inserted block has the expected structure by reading from the shared repo + val blocks = blockRepo.getBlocksForPage(page.uuid).first().getOrNull().orEmpty() + assertTrue(blocks.isNotEmpty(), "Expected at least one block inserted") + val voiceBlock = blocks.firstOrNull { it.content.contains("📝 Voice note") } + assertNotNull(voiceBlock, "Expected a block with voice note header") + assertTrue(voiceBlock.content.contains("#+BEGIN_QUOTE"), "Expected #+BEGIN_QUOTE in block") + assertTrue(voiceBlock.content.contains(transcript), "Expected raw transcript in block") + } +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/repository/JournalService.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/repository/JournalService.kt index 45d46711c..171e02e6c 100644 --- a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/repository/JournalService.kt +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/repository/JournalService.kt @@ -126,6 +126,30 @@ class JournalService( newPage } + /** + * Appends a new block with [content] to today's journal page. + * Creates the journal page if it does not yet exist. + */ + @OptIn(DirectRepositoryWrite::class) + suspend fun appendToToday(content: String) { + val page = ensureTodayJournal() + val blocks = blockRepository.getBlocksForPage(page.uuid).first().getOrNull() ?: emptyList() + val nextPosition = (blocks.maxOfOrNull { it.position } ?: -1) + 1 + val newBlock = Block( + uuid = UuidGenerator.generateV7(), + pageUuid = page.uuid, + content = content, + position = nextPosition, + createdAt = Clock.System.now(), + updatedAt = Clock.System.now(), + ) + if (writeActor != null) { + writeActor.saveBlock(newBlock) + } else { + blockRepository.saveBlock(newBlock) + } + } + /** * Merges duplicate journal pages for the same date. * Keeps the page with real (non-empty) content; deletes the others and diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/App.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/App.kt index bf04f7e45..ed954bd39 100644 --- a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/App.kt +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/App.kt @@ -74,6 +74,10 @@ import dev.stapler.stelekit.ui.screens.PermissionRecoveryScreen import dev.stapler.stelekit.ui.screens.SearchViewModel import dev.stapler.stelekit.domain.NoOpUrlFetcher import dev.stapler.stelekit.domain.UrlFetcher +import dev.stapler.stelekit.voice.VoiceCaptureState +import dev.stapler.stelekit.voice.VoiceCaptureViewModel +import dev.stapler.stelekit.voice.VoicePipelineConfig +import dev.stapler.stelekit.voice.VoiceSettings import dev.stapler.stelekit.ui.theme.StelekitTheme import dev.stapler.stelekit.ui.theme.StelekitThemeMode import kotlin.math.roundToInt @@ -96,7 +100,10 @@ fun StelekitApp( graphManager: GraphManager? = null, pluginHost: PluginHost = remember { PluginHost() }, encryptionManager: EncryptionManager = remember { DefaultEncryptionManager() }, - urlFetcher: UrlFetcher = remember { NoOpUrlFetcher() } + urlFetcher: UrlFetcher = remember { NoOpUrlFetcher() }, + voicePipeline: VoicePipelineConfig = remember { VoicePipelineConfig() }, + voiceSettings: VoiceSettings? = null, + onRebuildVoicePipeline: (() -> Unit)? = null, ) { val platformSettings = remember { PlatformSettings() } val scope = rememberCoroutineScope() @@ -209,7 +216,10 @@ fun StelekitApp( encryptionManager = encryptionManager, graphManager = graphManager, notificationManager = notificationManager, - urlFetcher = urlFetcher + urlFetcher = urlFetcher, + voicePipeline = voicePipeline, + voiceSettings = voiceSettings, + onRebuildVoicePipeline = onRebuildVoicePipeline, ) } } @@ -231,6 +241,9 @@ private fun GraphContent( graphManager: GraphManager, notificationManager: NotificationManager, urlFetcher: UrlFetcher = NoOpUrlFetcher(), + voicePipeline: VoicePipelineConfig = VoicePipelineConfig(), + voiceSettings: VoiceSettings? = null, + onRebuildVoicePipeline: (() -> Unit)? = null, ) { val scope = rememberCoroutineScope() val composeClipboard = LocalClipboardManager.current @@ -328,6 +341,13 @@ private fun GraphContent( } } + val journalsViewModel = remember { + JournalsViewModel(repos.journalService, blockStateManager, scope) + } + val voiceCaptureViewModel = remember { + VoiceCaptureViewModel(voicePipeline, repos.journalService, scope) + } + // Force-flush pending writes on Android lifecycle pause/stop val lifecycleOwner = LocalLifecycleOwner.current DisposableEffect(lifecycleOwner) { @@ -335,15 +355,12 @@ private fun GraphContent( if (event == Lifecycle.Event.ON_PAUSE || event == Lifecycle.Event.ON_STOP) { viewModel.savePendingChanges() scope.launch { blockStateManager.flush() } + voiceCaptureViewModel.cancel() } } lifecycleOwner.lifecycle.addObserver(observer) onDispose { lifecycleOwner.lifecycle.removeObserver(observer) } } - - val journalsViewModel = remember { - JournalsViewModel(repos.journalService, blockStateManager, scope) - } val allPagesViewModel = remember { AllPagesViewModel(repos.pageRepository, repos.blockRepository, scope) } @@ -352,6 +369,7 @@ private fun GraphContent( } val appState by viewModel.uiState.collectAsState() + val voiceCaptureState by voiceCaptureViewModel.state.collectAsState() val graphRegistry by graphManager.graphRegistry.collectAsState() val activeGraphInfo = graphManager.getActiveGraphInfo() val activeGraphId = graphRegistry.activeGraphId @@ -420,6 +438,12 @@ private fun GraphContent( PlatformBackHandler(enabled = appState.commandPaletteVisible) { viewModel.setCommandPaletteVisible(false) } PlatformBackHandler(enabled = appState.searchDialogVisible) { viewModel.setSearchDialogVisible(false) } PlatformBackHandler(enabled = appState.settingsVisible) { viewModel.setSettingsVisible(false) } + // Cancel an in-progress voice capture before any navigation back. + PlatformBackHandler( + enabled = voiceCaptureState is VoiceCaptureState.Recording || + voiceCaptureState is VoiceCaptureState.Transcribing || + voiceCaptureState is VoiceCaptureState.Formatting, + ) { voiceCaptureViewModel.cancel() } // Highest priority: close sidebar on mobile before anything else. PlatformBackHandler(enabled = isMobile && appState.sidebarExpanded) { viewModel.toggleSidebar() } @@ -537,7 +561,16 @@ private fun GraphContent( closeSidebarIfMobile() }, onSearch = { viewModel.setSearchDialogVisible(true) }, - isLeftHanded = appState.isLeftHanded + isLeftHanded = appState.isLeftHanded, + voiceCaptureButton = { + VoiceCaptureButton( + state = voiceCaptureState, + onTap = { voiceCaptureViewModel.onMicTapped() }, + onDismissError = { voiceCaptureViewModel.dismissError() }, + onAutoReset = { voiceCaptureViewModel.resetToIdle() }, + amplitudeFlow = voicePipeline.audioRecorder.amplitudeFlow, + ) + }, ) } ) @@ -547,7 +580,9 @@ private fun GraphContent( searchViewModel = searchViewModel, viewModel = viewModel, notificationManager = notificationManager, - fileSystem = fileSystem + fileSystem = fileSystem, + voiceSettings = voiceSettings, + onRebuildVoicePipeline = onRebuildVoicePipeline, ) } // CompositionLocalProvider(LocalWindowSizeClass) @@ -720,7 +755,9 @@ private fun GraphDialogLayer( searchViewModel: SearchViewModel, viewModel: StelekitViewModel, notificationManager: NotificationManager, - fileSystem: FileSystem + fileSystem: FileSystem, + voiceSettings: VoiceSettings? = null, + onRebuildVoicePipeline: (() -> Unit)? = null, ) { // Hoist debug state so FrameTimeOverlay persists when the dialog is closed. var debugState by remember { mutableStateOf(DebugMenuState()) } @@ -754,7 +791,9 @@ private fun GraphDialogLayer( viewModel.setSettingsVisible(false) }, isLeftHanded = appState.isLeftHanded, - onLeftHandedChange = { viewModel.setLeftHanded(it) } + onLeftHandedChange = { viewModel.setLeftHanded(it) }, + voiceSettings = voiceSettings, + onRebuildVoicePipeline = onRebuildVoicePipeline, ) appState.diskConflict?.let { conflict -> diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.kt index 0c62e4d76..be99493f4 100644 --- a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.kt +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.kt @@ -7,5 +7,6 @@ expect fun PlatformBottomBar( currentScreen: Screen, onNavigate: (Screen) -> Unit, onSearch: () -> Unit = {}, - isLeftHanded: Boolean = false + isLeftHanded: Boolean = false, + voiceCaptureButton: @Composable () -> Unit = {}, ) diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/VoiceCaptureButton.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/VoiceCaptureButton.kt new file mode 100644 index 000000000..a95371846 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/VoiceCaptureButton.kt @@ -0,0 +1,189 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.ui.components + +import androidx.compose.animation.core.Animatable +import androidx.compose.animation.core.RepeatMode +import androidx.compose.animation.core.animateFloat +import androidx.compose.animation.core.infiniteRepeatable +import androidx.compose.animation.core.rememberInfiniteTransition +import androidx.compose.animation.core.tween +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.size +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Check +import androidx.compose.material.icons.filled.Mic +import androidx.compose.material.icons.filled.Stop +import androidx.compose.material.icons.filled.Warning +import androidx.compose.material3.CircularProgressIndicator +import androidx.compose.material3.FloatingActionButton +import androidx.compose.material3.Icon +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Surface +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.getValue +import androidx.compose.runtime.remember +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.scale +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.semantics.contentDescription +import androidx.compose.ui.semantics.disabled +import androidx.compose.ui.semantics.semantics +import androidx.compose.ui.unit.dp +import dev.stapler.stelekit.voice.VoiceCaptureState +import kotlinx.coroutines.delay +import kotlinx.coroutines.flow.Flow + +private val ColorSuccess = Color(0xFF4CAF50) + +// Scale range for amplitude-driven pulse: 1.0 (silence) → 1.35 (loud) +private const val AMPLITUDE_SCALE_RANGE = 0.35f +// Tween duration for amplitude animation — fast enough to feel reactive. +private const val AMPLITUDE_TWEEN_MS = 80 +// Fixed pulse period when amplitude data is unavailable. +private const val FIXED_PULSE_TWEEN_MS = 600 +private const val FIXED_PULSE_MAX_SCALE = 1.25f +// Duration the Done state is shown before auto-resetting to Idle. +private const val DONE_AUTO_RESET_MS = 5_000L + +@Composable +fun VoiceCaptureButton( + state: VoiceCaptureState, + onTap: () -> Unit, + onDismissError: () -> Unit, + onAutoReset: () -> Unit = {}, + amplitudeFlow: Flow? = null, +) { + when (state) { + VoiceCaptureState.Idle -> { + FloatingActionButton(onClick = onTap) { + Icon(Icons.Default.Mic, contentDescription = "Start recording") + } + } + + VoiceCaptureState.Recording -> { + val scale = if (amplitudeFlow != null) { + // Amplitude-driven pulse: map RMS [0,1] → scale [1.0, 1.35] + val animatable = remember { Animatable(1f) } + LaunchedEffect(Unit) { + amplitudeFlow.collect { rms -> + animatable.animateTo( + 1f + (rms * AMPLITUDE_SCALE_RANGE).coerceIn(0f, AMPLITUDE_SCALE_RANGE), + tween(AMPLITUDE_TWEEN_MS), + ) + } + } + animatable.value + } else { + // Fixed-period fallback when no amplitude data available + val infiniteTransition = rememberInfiniteTransition() + val fixedScale by infiniteTransition.animateFloat( + initialValue = 1f, + targetValue = FIXED_PULSE_MAX_SCALE, + animationSpec = infiniteRepeatable( + animation = tween(FIXED_PULSE_TWEEN_MS), + repeatMode = RepeatMode.Reverse, + ), + ) + fixedScale + } + FloatingActionButton( + onClick = onTap, + containerColor = MaterialTheme.colorScheme.error, + modifier = Modifier.scale(scale), + ) { + Icon(Icons.Default.Stop, contentDescription = "Stop recording") + } + } + + VoiceCaptureState.Transcribing, VoiceCaptureState.Formatting -> { + val label = if (state == VoiceCaptureState.Transcribing) "Transcribing…" else "Formatting…" + FloatingActionButton( + onClick = {}, + modifier = Modifier.semantics { + contentDescription = label + disabled() + }, + ) { + CircularProgressIndicator( + modifier = Modifier.size(24.dp), + strokeWidth = 2.5.dp, + ) + } + } + + is VoiceCaptureState.Done -> { + LaunchedEffect(state) { + delay(DONE_AUTO_RESET_MS) + onAutoReset() + } + if (state.isLikelyTruncated) { + Column( + horizontalAlignment = Alignment.CenterHorizontally, + verticalArrangement = Arrangement.spacedBy(4.dp), + ) { + Surface( + shape = MaterialTheme.shapes.small, + color = MaterialTheme.colorScheme.tertiaryContainer, + ) { + Text( + text = "Note may be incomplete", + modifier = Modifier.padding(horizontal = 12.dp, vertical = 4.dp), + style = MaterialTheme.typography.labelSmall, + color = MaterialTheme.colorScheme.onTertiaryContainer, + ) + } + FloatingActionButton( + onClick = onAutoReset, + containerColor = MaterialTheme.colorScheme.tertiary, + modifier = Modifier.semantics { + contentDescription = "Note saved — may be incomplete. Tap to dismiss." + }, + ) { + Icon(Icons.Default.Warning, contentDescription = null) + } + } + } else { + FloatingActionButton( + onClick = onAutoReset, + containerColor = ColorSuccess, + modifier = Modifier.semantics { + contentDescription = "Note saved. Tap to dismiss." + }, + ) { + Icon(Icons.Default.Check, contentDescription = null) + } + } + } + + is VoiceCaptureState.Error -> { + Column( + horizontalAlignment = Alignment.CenterHorizontally, + verticalArrangement = Arrangement.spacedBy(4.dp), + ) { + Surface( + shape = MaterialTheme.shapes.small, + color = MaterialTheme.colorScheme.errorContainer, + ) { + Text( + text = state.message, + modifier = Modifier.padding(horizontal = 12.dp, vertical = 4.dp), + style = MaterialTheme.typography.labelSmall, + color = MaterialTheme.colorScheme.onErrorContainer, + ) + } + FloatingActionButton( + onClick = onDismissError, + containerColor = MaterialTheme.colorScheme.error, + ) { + Icon(Icons.Default.Warning, contentDescription = "Error — tap to dismiss") + } + } + } + } +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/settings/SettingsDialog.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/settings/SettingsDialog.kt index 7039d23fb..cb1e3f967 100644 --- a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/settings/SettingsDialog.kt +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/settings/SettingsDialog.kt @@ -17,6 +17,7 @@ import androidx.compose.ui.window.Dialog import androidx.compose.ui.window.DialogProperties import dev.stapler.stelekit.ui.theme.StelekitThemeMode import dev.stapler.stelekit.ui.i18n.Language +import dev.stapler.stelekit.voice.VoiceSettings @Composable fun SettingsDialog( @@ -28,7 +29,9 @@ fun SettingsDialog( onLanguageChange: (Language) -> Unit, onReindex: () -> Unit, isLeftHanded: Boolean = false, - onLeftHandedChange: (Boolean) -> Unit = {} + onLeftHandedChange: (Boolean) -> Unit = {}, + voiceSettings: VoiceSettings? = null, + onRebuildVoicePipeline: (() -> Unit)? = null, ) { if (visible) { Dialog( @@ -111,6 +114,12 @@ fun SettingsDialog( SettingsCategory.EDITOR -> EditorSettings() SettingsCategory.PLUGINS -> PluginsSettings() SettingsCategory.ADVANCED -> AdvancedSettings(onReindex) + SettingsCategory.VOICE -> if (voiceSettings != null && onRebuildVoicePipeline != null) { + VoiceCaptureSettings( + voiceSettings = voiceSettings, + onRebuildPipeline = onRebuildVoicePipeline, + ) + } } } } @@ -156,5 +165,6 @@ enum class SettingsCategory(val label: String, val icon: ImageVector) { GENERAL("General", Icons.Default.Settings), EDITOR("Editor", Icons.Default.Edit), PLUGINS("Plugins", Icons.Default.Extension), - ADVANCED("Advanced", Icons.Default.Build) + ADVANCED("Advanced", Icons.Default.Build), + VOICE("Voice Capture", Icons.Default.Mic), } diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/settings/VoiceCaptureSettings.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/settings/VoiceCaptureSettings.kt new file mode 100644 index 000000000..7ad179b34 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/settings/VoiceCaptureSettings.kt @@ -0,0 +1,122 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.ui.components + +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.padding +import androidx.compose.material3.Button +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.OutlinedTextField +import androidx.compose.material3.Switch +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.text.input.PasswordVisualTransformation +import androidx.compose.ui.unit.dp +import dev.stapler.stelekit.voice.VoiceSettings + +@Composable +fun VoiceCaptureSettings( + voiceSettings: VoiceSettings, + onRebuildPipeline: () -> Unit, +) { + var whisperKey by remember { mutableStateOf(voiceSettings.getWhisperApiKey() ?: "") } + var anthropicKey by remember { mutableStateOf(voiceSettings.getAnthropicKey() ?: "") } + var openAiKey by remember { mutableStateOf(voiceSettings.getOpenAiKey() ?: "") } + var llmEnabled by remember { mutableStateOf(voiceSettings.getLlmEnabled()) } + var saved by remember { mutableStateOf(false) } + + SettingsSection("Transcription (Speech-to-Text)") { + Text( + "Whisper API key — used for speech transcription (~\$0.003/min).", + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(bottom = 8.dp), + ) + OutlinedTextField( + value = whisperKey, + onValueChange = { whisperKey = it; saved = false }, + label = { Text("OpenAI / Whisper API key") }, + visualTransformation = PasswordVisualTransformation(), + singleLine = true, + modifier = Modifier.fillMaxWidth(), + ) + } + + SettingsSection("LLM Formatting") { + Text( + "Formats the raw transcript into Logseq outliner syntax with bullet points and [[wikilinks]]. " + + "Provide one key — Anthropic is used if both are set.", + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(bottom = 8.dp), + ) + Row( + modifier = Modifier.fillMaxWidth().padding(bottom = 8.dp), + horizontalArrangement = Arrangement.SpaceBetween, + verticalAlignment = Alignment.CenterVertically, + ) { + Text("Enable LLM formatting", style = MaterialTheme.typography.bodyMedium) + Switch( + checked = llmEnabled, + onCheckedChange = { llmEnabled = it; saved = false }, + ) + } + if (llmEnabled) { + OutlinedTextField( + value = anthropicKey, + onValueChange = { anthropicKey = it; saved = false }, + label = { Text("Anthropic (Claude) API key") }, + visualTransformation = PasswordVisualTransformation(), + singleLine = true, + modifier = Modifier.fillMaxWidth(), + ) + OutlinedTextField( + value = openAiKey, + onValueChange = { openAiKey = it; saved = false }, + label = { Text("OpenAI / compatible API key") }, + visualTransformation = PasswordVisualTransformation(), + singleLine = true, + modifier = Modifier + .fillMaxWidth() + .padding(top = 8.dp), + ) + } + } + + Column(modifier = Modifier.padding(vertical = 8.dp)) { + Row( + modifier = Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.spacedBy(12.dp), + verticalAlignment = Alignment.CenterVertically, + ) { + Button( + onClick = { + voiceSettings.setWhisperApiKey(whisperKey) + voiceSettings.setAnthropicKey(anthropicKey) + voiceSettings.setOpenAiKey(openAiKey) + voiceSettings.setLlmEnabled(llmEnabled) + saved = true + onRebuildPipeline() + }, + ) { + Text("Save") + } + if (saved) { + Text( + "Saved — pipeline updated.", + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.primary, + ) + } + } + } +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/AudioRecorder.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/AudioRecorder.kt new file mode 100644 index 000000000..cc803a1a9 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/AudioRecorder.kt @@ -0,0 +1,27 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import kotlinx.coroutines.flow.Flow + +data class PlatformAudioFile(val path: String) { + val isEmpty: Boolean get() = path.isEmpty() +} + +interface AudioRecorder { + /** Starts recording and suspends until [stopRecording] is called. Returns the output file. */ + suspend fun startRecording(): PlatformAudioFile + /** Signals the active recording to stop. */ + suspend fun stopRecording() + /** Reads the recorded file as bytes. Returns empty array for an empty/missing file. */ + suspend fun readBytes(file: PlatformAudioFile): ByteArray = ByteArray(0) + /** Deletes the recorded temp file. No-op by default. */ + fun deleteRecording(file: PlatformAudioFile) = Unit + /** Optional RMS amplitude stream for animated recording feedback. */ + val amplitudeFlow: Flow? get() = null +} + +class NoOpAudioRecorder : AudioRecorder { + override suspend fun startRecording(): PlatformAudioFile = PlatformAudioFile("") + override suspend fun stopRecording() = Unit +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/ClaudeLlmFormatterProvider.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/ClaudeLlmFormatterProvider.kt new file mode 100644 index 000000000..3aff66934 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/ClaudeLlmFormatterProvider.kt @@ -0,0 +1,89 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import io.ktor.client.HttpClient +import io.ktor.client.call.body +import io.ktor.client.plugins.contentnegotiation.ContentNegotiation +import io.ktor.client.request.headers +import io.ktor.client.request.post +import io.ktor.client.request.setBody +import io.ktor.http.ContentType +import io.ktor.http.contentType +import io.ktor.serialization.kotlinx.json.json +import kotlinx.coroutines.CancellationException +import kotlinx.serialization.SerialName +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.Json +import io.ktor.utils.io.errors.IOException + +class ClaudeLlmFormatterProvider( + private val httpClient: HttpClient, + private val apiKey: String, +) : LlmFormatterProvider { + + companion object { + private const val MESSAGES_URL = "https://api.anthropic.com/v1/messages" + private const val ANTHROPIC_VERSION = "2023-06-01" + private const val CLAUDE_MODEL = "claude-haiku-4-5-20251001" + private val lenientJson = Json { ignoreUnknownKeys = true } + + fun withDefaults(apiKey: String): ClaudeLlmFormatterProvider { + val client = HttpClient { + install(ContentNegotiation) { json(lenientJson) } + } + return ClaudeLlmFormatterProvider(client, apiKey) + } + } + + override suspend fun format(transcript: String, systemPrompt: String): LlmResult { + val maxTokens = LlmProviderSupport.estimateMaxTokens(transcript) + return try { + val response = httpClient.post(MESSAGES_URL) { + headers { + append("x-api-key", apiKey) + append("anthropic-version", ANTHROPIC_VERSION) + } + contentType(ContentType.Application.Json) + setBody( + ClaudeRequest( + model = CLAUDE_MODEL, + maxTokens = maxTokens, + messages = listOf(ClaudeMessage(role = "user", content = systemPrompt)), + ) + ) + } + + when (response.status.value) { + 200 -> { + val body = response.body() + val text = body.content.firstOrNull()?.text?.trim() ?: return LlmResult.Failure.NetworkError + LlmResult.Success(formattedText = text, isLikelyTruncated = LlmProviderSupport.detectTruncation(text)) + } + else -> LlmProviderSupport.mapHttpError(response.status.value) + } + } catch (e: CancellationException) { + throw e + } catch (e: IOException) { + LlmResult.Failure.NetworkError + } catch (e: Exception) { + LlmResult.Failure.ApiError(-1, "Unexpected error: ${e.message}") + } + } +} + +@Serializable +private data class ClaudeRequest( + val model: String, + @SerialName("max_tokens") val maxTokens: Int, + val messages: List, +) + +@Serializable +private data class ClaudeMessage(val role: String, val content: String) + +@Serializable +private data class ClaudeResponse(val content: List) + +@Serializable +private data class ClaudeContentBlock(val type: String, val text: String) diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/LlmFormatterProvider.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/LlmFormatterProvider.kt new file mode 100644 index 000000000..0983ca795 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/LlmFormatterProvider.kt @@ -0,0 +1,20 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +sealed interface LlmResult { + data class Success(val formattedText: String, val isLikelyTruncated: Boolean = false) : LlmResult + sealed interface Failure : LlmResult { + data class ApiError(val code: Int, val message: String) : Failure + data object NetworkError : Failure + } +} + +fun interface LlmFormatterProvider { + suspend fun format(transcript: String, systemPrompt: String): LlmResult +} + +class NoOpLlmFormatterProvider : LlmFormatterProvider { + override suspend fun format(transcript: String, systemPrompt: String): LlmResult = + LlmResult.Success(transcript) +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/LlmProviderSupport.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/LlmProviderSupport.kt new file mode 100644 index 000000000..adbe42158 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/LlmProviderSupport.kt @@ -0,0 +1,22 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +internal object LlmProviderSupport { + private val SENTENCE_END = setOf('.', '?', '!', ']', '\n') + private const val MIN_TOKENS = 512 + private const val MAX_TOKENS = 4096 + + fun estimateMaxTokens(transcript: String): Int { + val wordCount = transcript.split(Regex("\\s+")).count { it.isNotBlank() } + return (wordCount * 2).coerceIn(MIN_TOKENS, MAX_TOKENS) + } + + fun detectTruncation(text: String): Boolean = text.isNotEmpty() && text.last() !in SENTENCE_END + + fun mapHttpError(statusCode: Int): LlmResult.Failure = when (statusCode) { + 401 -> LlmResult.Failure.ApiError(401, "Invalid API key") + 429 -> LlmResult.Failure.ApiError(429, "Rate limit exceeded") + else -> LlmResult.Failure.ApiError(statusCode, "HTTP $statusCode") + } +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/OpenAiLlmFormatterProvider.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/OpenAiLlmFormatterProvider.kt new file mode 100644 index 000000000..eb6c82222 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/OpenAiLlmFormatterProvider.kt @@ -0,0 +1,94 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import io.ktor.client.HttpClient +import io.ktor.client.call.body +import io.ktor.client.plugins.contentnegotiation.ContentNegotiation +import io.ktor.client.request.headers +import io.ktor.client.request.post +import io.ktor.client.request.setBody +import io.ktor.http.ContentType +import io.ktor.http.HttpHeaders +import io.ktor.http.contentType +import io.ktor.serialization.kotlinx.json.json +import kotlinx.coroutines.CancellationException +import kotlinx.serialization.SerialName +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.Json +import io.ktor.utils.io.errors.IOException + +class OpenAiLlmFormatterProvider( + private val httpClient: HttpClient, + private val apiKey: String, + private val baseUrl: String = "https://api.openai.com", +) : LlmFormatterProvider { + + companion object { + private const val OPENAI_MODEL = "gpt-4o-mini" + private val lenientJson = Json { ignoreUnknownKeys = true } + + fun withDefaults(apiKey: String, baseUrl: String = "https://api.openai.com"): OpenAiLlmFormatterProvider { + val client = HttpClient { + install(ContentNegotiation) { json(lenientJson) } + } + return OpenAiLlmFormatterProvider(client, apiKey, baseUrl) + } + } + + override suspend fun format(transcript: String, systemPrompt: String): LlmResult { + val maxTokens = LlmProviderSupport.estimateMaxTokens(transcript) + val completionsUrl = "$baseUrl/v1/chat/completions" + + return try { + val response = httpClient.post(completionsUrl) { + headers { + append(HttpHeaders.Authorization, "Bearer $apiKey") + } + contentType(ContentType.Application.Json) + setBody( + OpenAiRequest( + model = OPENAI_MODEL, + maxTokens = maxTokens, + messages = listOf( + OpenAiMessage(role = "system", content = systemPrompt), + OpenAiMessage(role = "user", content = transcript), + ), + ) + ) + } + + when (response.status.value) { + 200 -> { + val body = response.body() + val text = body.choices.firstOrNull()?.message?.content?.trim() + ?: return LlmResult.Failure.NetworkError + LlmResult.Success(formattedText = text, isLikelyTruncated = LlmProviderSupport.detectTruncation(text)) + } + else -> LlmProviderSupport.mapHttpError(response.status.value) + } + } catch (e: CancellationException) { + throw e + } catch (e: IOException) { + LlmResult.Failure.NetworkError + } catch (e: Exception) { + LlmResult.Failure.ApiError(-1, "Unexpected error: ${e.message}") + } + } +} + +@Serializable +private data class OpenAiRequest( + val model: String, + @SerialName("max_tokens") val maxTokens: Int, + val messages: List, +) + +@Serializable +private data class OpenAiMessage(val role: String, val content: String) + +@Serializable +private data class OpenAiResponse(val choices: List) + +@Serializable +private data class OpenAiChoice(val message: OpenAiMessage) diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/SpeechToTextProvider.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/SpeechToTextProvider.kt new file mode 100644 index 000000000..b6d455f98 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/SpeechToTextProvider.kt @@ -0,0 +1,21 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +sealed interface TranscriptResult { + data class Success(val text: String) : TranscriptResult + data object Empty : TranscriptResult + sealed interface Failure : TranscriptResult { + data class ApiError(val code: Int, val message: String) : Failure + data object NetworkError : Failure + data object PermissionDenied : Failure + } +} + +fun interface SpeechToTextProvider { + suspend fun transcribe(audioData: ByteArray): TranscriptResult +} + +class NoOpSpeechToTextProvider : SpeechToTextProvider { + override suspend fun transcribe(audioData: ByteArray): TranscriptResult = TranscriptResult.Empty +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureState.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureState.kt new file mode 100644 index 000000000..a0c0254b4 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureState.kt @@ -0,0 +1,14 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +enum class PipelineStage { RECORDING, TRANSCRIBING, FORMATTING, JOURNAL } + +sealed interface VoiceCaptureState { + data object Idle : VoiceCaptureState + data object Recording : VoiceCaptureState + data object Transcribing : VoiceCaptureState + data object Formatting : VoiceCaptureState + data class Done(val insertedText: String, val isLikelyTruncated: Boolean = false) : VoiceCaptureState + data class Error(val stage: PipelineStage, val message: String) : VoiceCaptureState +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModel.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModel.kt new file mode 100644 index 000000000..77be06681 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModel.kt @@ -0,0 +1,147 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import dev.stapler.stelekit.repository.JournalService +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Job +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow +import kotlinx.coroutines.launch +import kotlin.time.Clock +import kotlinx.datetime.TimeZone +import kotlinx.datetime.toLocalDateTime + +private const val MAX_TRANSCRIPT_CHARS = 10_000 + +class VoiceCaptureViewModel( + private val pipeline: VoicePipelineConfig, + private val journalService: JournalService, + private val scope: CoroutineScope, +) { + private val _state = MutableStateFlow(VoiceCaptureState.Idle) + val state: StateFlow = _state.asStateFlow() + + private var pipelineJob: Job? = null + + fun onMicTapped() { + when (_state.value) { + is VoiceCaptureState.Idle -> startPipeline() + is VoiceCaptureState.Recording -> scope.launch { + pipeline.audioRecorder.stopRecording() + } + else -> Unit + } + } + + fun cancel() { + pipelineJob?.cancel() + pipelineJob = null + _state.value = VoiceCaptureState.Idle + } + + fun dismissError() { + _state.value = VoiceCaptureState.Idle + } + + fun resetToIdle() { + _state.value = VoiceCaptureState.Idle + } + + private fun startPipeline() { + pipelineJob = scope.launch { + var file: PlatformAudioFile? = null + try { + _state.value = VoiceCaptureState.Recording + val result = pipeline.audioRecorder.startRecording() + file = result + + if (result.isEmpty) { + _state.value = VoiceCaptureState.Error( + PipelineStage.RECORDING, "Microphone permission denied" + ) + return@launch + } + + _state.value = VoiceCaptureState.Transcribing + val audioData = pipeline.audioRecorder.readBytes(result) + when (val sttResult = pipeline.sttProvider.transcribe(audioData)) { + TranscriptResult.Empty -> { + _state.value = VoiceCaptureState.Error( + PipelineStage.TRANSCRIBING, "Nothing was captured — try again" + ) + return@launch + } + is TranscriptResult.Failure.ApiError -> { + _state.value = VoiceCaptureState.Error( + PipelineStage.TRANSCRIBING, sttResult.message + ) + return@launch + } + TranscriptResult.Failure.NetworkError -> { + _state.value = VoiceCaptureState.Error( + PipelineStage.TRANSCRIBING, "Network error — check your connection" + ) + return@launch + } + TranscriptResult.Failure.PermissionDenied -> { + _state.value = VoiceCaptureState.Error( + PipelineStage.RECORDING, "Microphone permission denied" + ) + return@launch + } + is TranscriptResult.Success -> processTranscript(sttResult.text.trim()) + } + } finally { + file?.takeIf { !it.isEmpty }?.let { pipeline.audioRecorder.deleteRecording(it) } + } + } + } + + private suspend fun processTranscript(fullTranscript: String) { + val inputTruncated = fullTranscript.length > MAX_TRANSCRIPT_CHARS + val rawTranscript = if (inputTruncated) fullTranscript.take(MAX_TRANSCRIPT_CHARS) else fullTranscript + val wordCount = rawTranscript.split(Regex("\\s+")).count { it.isNotBlank() } + if (wordCount < pipeline.minWordCount) { + _state.value = VoiceCaptureState.Error( + PipelineStage.TRANSCRIBING, + "Recording too short — try speaking for a few more seconds" + ) + return + } + + _state.value = VoiceCaptureState.Formatting + val prompt = pipeline.systemPrompt.replace("{{TRANSCRIPT}}", rawTranscript) + var isLikelyTruncated = inputTruncated + val formattedText = when (val llmResult = pipeline.llmProvider.format(rawTranscript, prompt)) { + is LlmResult.Success -> { + isLikelyTruncated = isLikelyTruncated || llmResult.isLikelyTruncated + llmResult.formattedText + } + is LlmResult.Failure -> { + println("[VoiceCaptureViewModel] LLM formatting failed ($llmResult), inserting raw transcript") + rawTranscript + } + } + + journalService.appendToToday(buildVoiceNoteBlock(formattedText, rawTranscript)) + _state.value = VoiceCaptureState.Done( + insertedText = formattedText, + isLikelyTruncated = isLikelyTruncated, + ) + } + + internal fun buildVoiceNoteBlock(formattedText: String, rawTranscript: String): String { + val now = Clock.System.now().toLocalDateTime(TimeZone.currentSystemDefault()) + val timeLabel = "${now.hour.toString().padStart(2, '0')}:${now.minute.toString().padStart(2, '0')}" + return buildString { + append("- 📝 Voice note ($timeLabel)") + append("\n - ") + append(formattedText.lines().joinToString("\n - ")) + append("\n #+BEGIN_QUOTE\n ") + append(rawTranscript) + append("\n #+END_QUOTE") + } + } +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoicePipelineConfig.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoicePipelineConfig.kt new file mode 100644 index 000000000..961d47e75 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoicePipelineConfig.kt @@ -0,0 +1,23 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +const val DEFAULT_VOICE_SYSTEM_PROMPT = """You are a Logseq note-taking assistant. Convert the following voice transcript into well-structured Logseq outliner syntax. + +Rules: +- Use "- " bullet format for each main point +- Use 2-space indentation for sub-points +- Add [[Page Name]] wiki links ONLY for proper nouns or topics explicitly named in the transcript — do NOT invent links for terms not spoken +- Do not add a preamble or summary +- Do not add content not present in the transcript + +Transcript: +{{TRANSCRIPT}}""" + +class VoicePipelineConfig( + val audioRecorder: AudioRecorder = NoOpAudioRecorder(), + val sttProvider: SpeechToTextProvider = NoOpSpeechToTextProvider(), + val llmProvider: LlmFormatterProvider = NoOpLlmFormatterProvider(), + val systemPrompt: String = DEFAULT_VOICE_SYSTEM_PROMPT, + val minWordCount: Int = 10, +) diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoicePipelineFactory.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoicePipelineFactory.kt new file mode 100644 index 000000000..6d98f04cc --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoicePipelineFactory.kt @@ -0,0 +1,22 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +fun buildVoicePipeline(audioRecorder: AudioRecorder, settings: VoiceSettings): VoicePipelineConfig { + val sttProvider: SpeechToTextProvider = settings.getWhisperApiKey() + ?.let { WhisperSpeechToTextProvider.withDefaults(it) } + ?: SpeechToTextProvider { _ -> + TranscriptResult.Failure.ApiError( + 0, + "No API key configured — add a Whisper key in Settings → Voice Capture", + ) + } + val llmProvider: LlmFormatterProvider = if (!settings.getLlmEnabled()) { + NoOpLlmFormatterProvider() + } else { + settings.getAnthropicKey()?.let { ClaudeLlmFormatterProvider.withDefaults(it) } + ?: settings.getOpenAiKey()?.let { OpenAiLlmFormatterProvider.withDefaults(it) } + ?: NoOpLlmFormatterProvider() + } + return VoicePipelineConfig(audioRecorder = audioRecorder, sttProvider = sttProvider, llmProvider = llmProvider) +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceSettings.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceSettings.kt new file mode 100644 index 000000000..ada7a42f3 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceSettings.kt @@ -0,0 +1,39 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import dev.stapler.stelekit.platform.PlatformSettings + +class VoiceSettings(private val platformSettings: PlatformSettings) { + + fun getWhisperApiKey(): String? = + platformSettings.getString(KEY_WHISPER, "").takeIf { it.isNotBlank() } + + fun setWhisperApiKey(key: String) = + platformSettings.putString(KEY_WHISPER, key.trim()) + + fun getAnthropicKey(): String? = + platformSettings.getString(KEY_ANTHROPIC, "").takeIf { it.isNotBlank() } + + fun setAnthropicKey(key: String) = + platformSettings.putString(KEY_ANTHROPIC, key.trim()) + + fun getOpenAiKey(): String? = + platformSettings.getString(KEY_OPENAI, "").takeIf { it.isNotBlank() } + + fun setOpenAiKey(key: String) = + platformSettings.putString(KEY_OPENAI, key.trim()) + + fun getLlmEnabled(): Boolean = + platformSettings.getBoolean(KEY_LLM_ENABLED, true) + + fun setLlmEnabled(enabled: Boolean) = + platformSettings.putBoolean(KEY_LLM_ENABLED, enabled) + + companion object { + private const val KEY_WHISPER = "voice.whisper_key" + private const val KEY_ANTHROPIC = "voice.anthropic_key" + private const val KEY_OPENAI = "voice.openai_key" + private const val KEY_LLM_ENABLED = "voice.llm_enabled" + } +} diff --git a/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProvider.kt b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProvider.kt new file mode 100644 index 000000000..a09611218 --- /dev/null +++ b/kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProvider.kt @@ -0,0 +1,69 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import io.ktor.client.HttpClient +import kotlinx.coroutines.CancellationException +import io.ktor.client.call.body +import io.ktor.client.plugins.contentnegotiation.ContentNegotiation +import io.ktor.client.request.forms.formData +import io.ktor.client.request.forms.submitFormWithBinaryData +import io.ktor.client.request.headers +import io.ktor.http.Headers +import io.ktor.http.HttpHeaders +import io.ktor.serialization.kotlinx.json.json +import kotlinx.serialization.json.Json + +class WhisperSpeechToTextProvider( + private val httpClient: HttpClient, + private val apiKey: String, +) : SpeechToTextProvider { + + companion object { + private const val TRANSCRIPTIONS_URL = "https://api.openai.com/v1/audio/transcriptions" + private val lenientJson = Json { ignoreUnknownKeys = true } + + fun withDefaults(apiKey: String): WhisperSpeechToTextProvider { + val client = HttpClient { + install(ContentNegotiation) { json(lenientJson) } + } + return WhisperSpeechToTextProvider(client, apiKey) + } + } + + override suspend fun transcribe(audioData: ByteArray): TranscriptResult { + if (audioData.isEmpty()) return TranscriptResult.Empty + + return try { + val response = httpClient.submitFormWithBinaryData( + url = TRANSCRIPTIONS_URL, + formData = formData { + append("model", "gpt-4o-mini-transcribe") + append("response_format", "text") + append("file", audioData, Headers.build { + append(HttpHeaders.ContentType, "audio/mp4") + append(HttpHeaders.ContentDisposition, "filename=\"recording.m4a\"") + }) + } + ) { + headers { append(HttpHeaders.Authorization, "Bearer $apiKey") } + } + + when (response.status.value) { + 200 -> { + val text = response.body().trim() + if (text.isBlank()) TranscriptResult.Empty else TranscriptResult.Success(text) + } + 401 -> TranscriptResult.Failure.ApiError(401, "Invalid API key") + 429 -> TranscriptResult.Failure.ApiError(429, "Rate limit exceeded") + else -> TranscriptResult.Failure.ApiError( + response.status.value, "HTTP ${response.status.value}" + ) + } + } catch (e: CancellationException) { + throw e + } catch (_: Exception) { + TranscriptResult.Failure.NetworkError + } + } +} diff --git a/kmp/src/jvmMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.jvm.kt b/kmp/src/jvmMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.jvm.kt index 40cf9073d..e5608296c 100644 --- a/kmp/src/jvmMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.jvm.kt +++ b/kmp/src/jvmMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.jvm.kt @@ -7,5 +7,6 @@ actual fun PlatformBottomBar( currentScreen: Screen, onNavigate: (Screen) -> Unit, onSearch: () -> Unit, - isLeftHanded: Boolean + isLeftHanded: Boolean, + voiceCaptureButton: @Composable () -> Unit, ) { /* Desktop uses sidebar navigation */ } diff --git a/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/ui/VoiceCaptureButtonScreenshotTest.kt b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/ui/VoiceCaptureButtonScreenshotTest.kt new file mode 100644 index 000000000..fba7743e4 --- /dev/null +++ b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/ui/VoiceCaptureButtonScreenshotTest.kt @@ -0,0 +1,102 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.ui + +import androidx.compose.ui.test.junit4.createComposeRule +import androidx.compose.ui.test.onRoot +import dev.stapler.stelekit.ui.components.VoiceCaptureButton +import dev.stapler.stelekit.ui.theme.StelekitTheme +import dev.stapler.stelekit.ui.theme.StelekitThemeMode +import dev.stapler.stelekit.voice.PipelineStage +import dev.stapler.stelekit.voice.VoiceCaptureState +import io.github.takahirom.roborazzi.captureRoboImage +import org.junit.Rule +import org.junit.Test + +/** + * Screenshot tests for each VoiceCaptureButton state. + * + * To record new golden images run: + * ./gradlew jvmTest -Proborazzi.test.record=true + */ +class VoiceCaptureButtonScreenshotTest { + + @get:Rule + val composeTestRule = createComposeRule() + + private fun render(state: VoiceCaptureState, themeMode: StelekitThemeMode = StelekitThemeMode.LIGHT) { + composeTestRule.setContent { + StelekitTheme(themeMode = themeMode) { + VoiceCaptureButton( + state = state, + onTap = {}, + onDismissError = {}, + onAutoReset = {}, + ) + } + } + } + + @Test + fun voiceCaptureButton_idle_light() { + render(VoiceCaptureState.Idle) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_idle_light.png") + } + + @Test + fun voiceCaptureButton_recording_light() { + render(VoiceCaptureState.Recording) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_recording_light.png") + } + + @Test + fun voiceCaptureButton_transcribing_light() { + render(VoiceCaptureState.Transcribing) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_transcribing_light.png") + } + + @Test + fun voiceCaptureButton_formatting_light() { + render(VoiceCaptureState.Formatting) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_formatting_light.png") + } + + @Test + fun voiceCaptureButton_done_light() { + render(VoiceCaptureState.Done(insertedText = "- Test note", isLikelyTruncated = false)) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_done_light.png") + } + + @Test + fun voiceCaptureButton_done_truncated_light() { + render(VoiceCaptureState.Done(insertedText = "- Test note", isLikelyTruncated = true)) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_done_truncated_light.png") + } + + @Test + fun voiceCaptureButton_error_light() { + render(VoiceCaptureState.Error(PipelineStage.TRANSCRIBING, "Network error — check your connection")) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_error_light.png") + } + + @Test + fun voiceCaptureButton_idle_dark() { + render(VoiceCaptureState.Idle, StelekitThemeMode.DARK) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_idle_dark.png") + } + + @Test + fun voiceCaptureButton_error_dark() { + render(VoiceCaptureState.Error(PipelineStage.TRANSCRIBING, "Network error — check your connection"), StelekitThemeMode.DARK) + composeTestRule.waitForIdle() + composeTestRule.onRoot().captureRoboImage("build/outputs/roborazzi/voice_button_error_dark.png") + } +} diff --git a/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/ClaudeLlmFormatterProviderTest.kt b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/ClaudeLlmFormatterProviderTest.kt new file mode 100644 index 000000000..de59bc3e5 --- /dev/null +++ b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/ClaudeLlmFormatterProviderTest.kt @@ -0,0 +1,112 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import io.ktor.client.HttpClient +import io.ktor.client.engine.mock.MockEngine +import io.ktor.client.engine.mock.respond +import io.ktor.client.plugins.contentnegotiation.ContentNegotiation +import io.ktor.http.HttpStatusCode +import io.ktor.http.headersOf +import io.ktor.serialization.kotlinx.json.json +import kotlinx.coroutines.test.runTest +import kotlinx.serialization.json.Json +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertIs + +class ClaudeLlmFormatterProviderTest { + + private fun buildProvider(engine: MockEngine): ClaudeLlmFormatterProvider { + val client = HttpClient(engine) { + install(ContentNegotiation) { json(Json { ignoreUnknownKeys = true }) } + } + return ClaudeLlmFormatterProvider(client, "test-key") + } + + private val validResponse = """ + {"content":[{"type":"text","text":"- bullet point one\n- bullet point two."}],"stop_reason":"end_turn"} + """.trimIndent() + + @Test + fun `200 with formatted text returns Success`() = runTest { + val engine = MockEngine { + respond(validResponse, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("test transcript", "prompt") + assertIs(result) + assertEquals("- bullet point one\n- bullet point two.", result.formattedText) + } + + @Test + fun `truncation detected when last char is not sentence-ending`() = runTest { + val truncatedBody = """{"content":[{"type":"text","text":"- bullet that cuts off mid"}],"stop_reason":"max_tokens"}""" + val engine = MockEngine { + respond(truncatedBody, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("test transcript", "prompt") + assertIs(result) + assertEquals(true, result.isLikelyTruncated) + } + + @Test + fun `no truncation when last char is period`() = runTest { + val engine = MockEngine { + respond(validResponse, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("test transcript", "prompt") + assertIs(result) + assertEquals(false, result.isLikelyTruncated) + } + + @Test + fun `401 returns ApiError with code 401`() = runTest { + val engine = MockEngine { + respond("""{"error":{"message":"Invalid key"}}""", HttpStatusCode.Unauthorized, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("transcript", "prompt") + assertIs(result) + assertEquals(401, result.code) + } + + @Test + fun `429 returns ApiError with code 429`() = runTest { + val engine = MockEngine { + respond("""{"error":{"message":"Rate limited"}}""", HttpStatusCode.TooManyRequests, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("transcript", "prompt") + assertIs(result) + assertEquals(429, result.code) + } + + @Test + fun `500 returns ApiError with code 500`() = runTest { + val engine = MockEngine { + respond("""{"error":{"message":"Internal error"}}""", HttpStatusCode.InternalServerError, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("transcript", "prompt") + assertIs(result) + assertEquals(500, result.code) + } + + @Test + fun `network exception returns NetworkError`() = runTest { + val engine = MockEngine { throw java.io.IOException("Connection refused") } + val result = buildProvider(engine).format("transcript", "prompt") + assertIs(result) + } + + @Test + fun `request includes x-api-key and anthropic-version headers`() = runTest { + var capturedApiKey = "" + var capturedVersion = "" + val engine = MockEngine { request -> + capturedApiKey = request.headers["x-api-key"] ?: "" + capturedVersion = request.headers["anthropic-version"] ?: "" + respond(validResponse, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + buildProvider(engine).format("transcript", "prompt") + assertEquals("test-key", capturedApiKey) + assertEquals("2023-06-01", capturedVersion) + } +} diff --git a/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/OpenAiLlmFormatterProviderTest.kt b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/OpenAiLlmFormatterProviderTest.kt new file mode 100644 index 000000000..3517585bb --- /dev/null +++ b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/OpenAiLlmFormatterProviderTest.kt @@ -0,0 +1,111 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import io.ktor.client.HttpClient +import io.ktor.client.engine.mock.MockEngine +import io.ktor.client.engine.mock.respond +import io.ktor.client.plugins.contentnegotiation.ContentNegotiation +import io.ktor.http.HttpStatusCode +import io.ktor.http.headersOf +import io.ktor.serialization.kotlinx.json.json +import kotlinx.coroutines.test.runTest +import kotlinx.serialization.json.Json +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertIs + +class OpenAiLlmFormatterProviderTest { + + private fun buildProvider(engine: MockEngine, baseUrl: String = "https://api.openai.com"): OpenAiLlmFormatterProvider { + val client = HttpClient(engine) { + install(ContentNegotiation) { json(Json { ignoreUnknownKeys = true }) } + } + return OpenAiLlmFormatterProvider(client, "test-key", baseUrl) + } + + private val validResponse = """ + {"choices":[{"message":{"role":"assistant","content":"- bullet point one\n- bullet point two."}}],"model":"gpt-4o-mini"} + """.trimIndent() + + @Test + fun `200 with formatted text returns Success`() = runTest { + val engine = MockEngine { + respond(validResponse, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("test transcript", "system prompt") + assertIs(result) + assertEquals("- bullet point one\n- bullet point two.", result.formattedText) + } + + @Test + fun `truncation detected when last char is not sentence-ending`() = runTest { + val truncatedBody = """{"choices":[{"message":{"role":"assistant","content":"- bullet that cuts off mid"}}]}""" + val engine = MockEngine { + respond(truncatedBody, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("test transcript", "prompt") + assertIs(result) + assertEquals(true, result.isLikelyTruncated) + } + + @Test + fun `401 returns ApiError with code 401`() = runTest { + val engine = MockEngine { + respond("""{"error":{"message":"Invalid key"}}""", HttpStatusCode.Unauthorized, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("transcript", "prompt") + assertIs(result) + assertEquals(401, result.code) + } + + @Test + fun `429 returns ApiError with code 429`() = runTest { + val engine = MockEngine { + respond("""{"error":{"message":"Rate limited"}}""", HttpStatusCode.TooManyRequests, headersOf("Content-Type", "application/json")) + } + val result = buildProvider(engine).format("transcript", "prompt") + assertIs(result) + assertEquals(429, result.code) + } + + @Test + fun `network exception returns NetworkError`() = runTest { + val engine = MockEngine { throw java.io.IOException("Connection refused") } + val result = buildProvider(engine).format("transcript", "prompt") + assertIs(result) + } + + @Test + fun `request targets correct URL`() = runTest { + var capturedUrl = "" + val engine = MockEngine { request -> + capturedUrl = request.url.toString() + respond(validResponse, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + buildProvider(engine, "https://api.openai.com").format("transcript", "prompt") + assertEquals("https://api.openai.com/v1/chat/completions", capturedUrl) + } + + @Test + fun `custom baseUrl is respected for OpenAI-compatible endpoints`() = runTest { + var capturedUrl = "" + val engine = MockEngine { request -> + capturedUrl = request.url.toString() + respond(validResponse, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + buildProvider(engine, "https://openrouter.ai").format("transcript", "prompt") + assertEquals("https://openrouter.ai/v1/chat/completions", capturedUrl) + } + + @Test + fun `request includes Authorization header`() = runTest { + var capturedAuth = "" + val engine = MockEngine { request -> + capturedAuth = request.headers["Authorization"] ?: "" + respond(validResponse, HttpStatusCode.OK, headersOf("Content-Type", "application/json")) + } + buildProvider(engine).format("transcript", "prompt") + assertEquals("Bearer test-key", capturedAuth) + } +} diff --git a/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProviderTest.kt b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProviderTest.kt new file mode 100644 index 000000000..5e8df9b14 --- /dev/null +++ b/kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProviderTest.kt @@ -0,0 +1,127 @@ +// Copyright (c) 2026 Tyler Stapler +// SPDX-License-Identifier: Elastic-2.0 +package dev.stapler.stelekit.voice + +import io.ktor.client.HttpClient +import io.ktor.client.engine.mock.MockEngine +import io.ktor.client.engine.mock.respond +import io.ktor.client.plugins.contentnegotiation.ContentNegotiation +import io.ktor.http.HttpStatusCode +import io.ktor.http.headersOf +import io.ktor.serialization.kotlinx.json.json +import kotlinx.coroutines.test.runTest +import kotlinx.serialization.json.Json +import kotlin.test.Test +import kotlin.test.assertIs +import kotlin.test.assertEquals + +class WhisperSpeechToTextProviderTest { + + private fun buildProvider(engine: MockEngine): WhisperSpeechToTextProvider { + val client = HttpClient(engine) { + install(ContentNegotiation) { json(Json { ignoreUnknownKeys = true }) } + } + return WhisperSpeechToTextProvider(client, "test-key") + } + + @Test + fun `empty audio returns Empty without network call`() = runTest { + val engine = MockEngine { error("Should not be called") } + val result = buildProvider(engine).transcribe(ByteArray(0)) + assertIs(result) + } + + @Test + fun `200 with text returns Success`() = runTest { + val engine = MockEngine { + respond( + content = "Hello world transcript", + status = HttpStatusCode.OK, + headers = headersOf("Content-Type", "text/plain"), + ) + } + val result = buildProvider(engine).transcribe(ByteArray(100)) + assertIs(result) + assertEquals("Hello world transcript", result.text) + } + + @Test + fun `200 with blank response returns Empty`() = runTest { + val engine = MockEngine { + respond( + content = " ", + status = HttpStatusCode.OK, + headers = headersOf("Content-Type", "text/plain"), + ) + } + val result = buildProvider(engine).transcribe(ByteArray(100)) + assertIs(result) + } + + @Test + fun `401 returns ApiError with code 401`() = runTest { + val engine = MockEngine { + respond( + content = """{"error":{"message":"Invalid API key"}}""", + status = HttpStatusCode.Unauthorized, + headers = headersOf("Content-Type", "application/json"), + ) + } + val result = buildProvider(engine).transcribe(ByteArray(100)) + assertIs(result) + assertEquals(401, result.code) + } + + @Test + fun `429 returns ApiError with code 429`() = runTest { + val engine = MockEngine { + respond( + content = """{"error":{"message":"Rate limit exceeded"}}""", + status = HttpStatusCode.TooManyRequests, + headers = headersOf("Content-Type", "application/json"), + ) + } + val result = buildProvider(engine).transcribe(ByteArray(100)) + assertIs(result) + assertEquals(429, result.code) + } + + @Test + fun `network exception returns NetworkError`() = runTest { + val engine = MockEngine { throw java.io.IOException("Connection refused") } + val result = buildProvider(engine).transcribe(ByteArray(100)) + assertIs(result) + } + + @Test + fun `500 response returns ApiError with code 500`() = runTest { + val engine = MockEngine { + respond( + content = """{"error":{"message":"Internal server error"}}""", + status = HttpStatusCode.InternalServerError, + headers = headersOf("Content-Type", "application/json"), + ) + } + val result = buildProvider(engine).transcribe(ByteArray(100)) + assertIs(result) + assertEquals(500, result.code) + } + + @Test + fun `request targets correct URL with Authorization header`() = runTest { + var capturedUrl = "" + var capturedAuth = "" + val engine = MockEngine { request -> + capturedUrl = request.url.toString() + capturedAuth = request.headers["Authorization"] ?: "" + respond( + content = "Test transcript", + status = HttpStatusCode.OK, + headers = headersOf("Content-Type", "text/plain"), + ) + } + buildProvider(engine).transcribe(ByteArray(100)) + assertEquals("https://api.openai.com/v1/audio/transcriptions", capturedUrl) + assertEquals("Bearer test-key", capturedAuth) + } +} diff --git a/kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.js.kt b/kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.js.kt index 06bf34a5d..d7f9285a3 100644 --- a/kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.js.kt +++ b/kmp/src/wasmJsMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.js.kt @@ -7,5 +7,6 @@ actual fun PlatformBottomBar( currentScreen: Screen, onNavigate: (Screen) -> Unit, onSearch: () -> Unit, - isLeftHanded: Boolean + isLeftHanded: Boolean, + voiceCaptureButton: @Composable () -> Unit, ) { /* Web uses sidebar navigation */ } diff --git a/project_plans/mobile-voice-mode/decisions/ADR-001-audio-capture-adapter.md b/project_plans/mobile-voice-mode/decisions/ADR-001-audio-capture-adapter.md new file mode 100644 index 000000000..d58cff608 --- /dev/null +++ b/project_plans/mobile-voice-mode/decisions/ADR-001-audio-capture-adapter.md @@ -0,0 +1,112 @@ +# ADR-001: Audio Capture Adapter Shape + +**Status**: Proposed +**Date**: 2026-04-18 + +## Context + +The voice pipeline requires a platform-specific audio recording layer. KMP has no standard audio +recording API in `commonMain`; every target exposes a different native API: + +- Android: `AudioRecord` (raw PCM) or `MediaRecorder` (encoded file) +- iOS: `AVAudioRecorder` or `AVAudioEngine` (native Obj-C via cinterop) +- Desktop (JVM): `javax.sound.sampled.TargetDataLine` — out of scope for this feature + +Two KMP design choices were evaluated: `expect class AudioRecorder` (monolithic platform class) +vs plain `interface AudioRecorder` in `commonMain` with platform implementations injected at +assembly time. + +The output format is also a decision: raw PCM (`Flow`) vs a temp file (`PlatformAudioFile`). + +The OpenAI Whisper API accepts `.m4a` directly (confirmed: flac, mp3, mp4, mpeg, mpga, m4a, ogg, +wav, webm). Raw PCM is NOT accepted. Both Android and iOS record to `.m4a` natively. + +## Decision + +Define `AudioRecorder` as a **plain interface in `commonMain`**, not an `expect class`. The +interface produces a temp file path (`PlatformAudioFile`) rather than a PCM stream. + +```kotlin +// commonMain — audio/AudioRecorder.kt +interface AudioRecorder { + /** Blocks until [stopRecording] is called. Returns path to a temp .m4a file. */ + suspend fun recordToFile(): PlatformAudioFile + suspend fun stopRecording() +} + +/** Cross-platform wrapper for a temp audio file path. */ +@JvmInline +value class PlatformAudioFile(val path: String) + +/** Default: no-op recorder that returns an empty temp file path (tests / desktop). */ +object NoOpAudioRecorder : AudioRecorder { + override suspend fun recordToFile() = PlatformAudioFile("") + override suspend fun stopRecording() = Unit +} +``` + +**Android implementation** (`androidMain`): `AudioRecord` (raw PCM, `VOICE_COMMUNICATION` audio +source) + `MediaCodec` AAC encoder → `.m4a` temp file in `context.cacheDir`. Uses +`VOICE_COMMUNICATION` audio source for system-level noise cancellation. Requests +`AudioManager.AUDIOFOCUS_GAIN_TRANSIENT` before recording and abandons it after. + +**iOS implementation** (`iosMain`): `AVAudioRecorder` recording to a `.m4a` URL in +`NSTemporaryDirectory()`. `AVAudioSession.sharedInstance().setCategory(.record)` must be called +before the recorder starts. The session category is restored to `.playback` or `.ambient` after +`stopRecording()` returns. + +Temp file cleanup is NOT the responsibility of `AudioRecorder`. `VoiceCaptureViewModel` deletes +the file in a `finally` block after `transcribe()` returns, regardless of success or failure. + +## Rationale + +**Plain interface over `expect class`**: The `expect class` pattern in KMP requires every +property and method in the expect to mirror in every actual. Adding a capability later forces +all platform actuals to update in lockstep — including a `jvmMain` stub for Desktop, which is +explicitly out of scope. A plain `interface` in `commonMain` avoids this rigidity. `AudioRecorder` +is consumed by exactly one ViewModel (`VoiceCaptureViewModel`), so there is no reason to pay the +overhead of KMP class machinery. This follows the same reasoning that applies to `TopicEnricher` +in ADR-002 (import-topic-suggestions project). + +**Temp file over PCM stream**: Real-time transcription display ("result shown only after +processing") is explicitly out of scope in `requirements.md`. A `Flow` PCM stream +would require reassembly into a file before Whisper upload anyway, adding complexity with no +v1 benefit. Both `AVAudioRecorder` (iOS) and `AudioRecord`+`MediaCodec` (Android) naturally +produce a file. Whisper's 25 MB limit (~26 minutes at 128 kbps AAC) is acceptable for all +realistic single-session recordings. + +**`AudioRecord` + `MediaCodec` over `MediaRecorder` on Android**: `MediaRecorder` corrupts MP4 +box headers when interrupted by a phone call or audio focus loss because it has no pause/resume +support on API < 24. `AudioRecord` reads raw PCM into a buffer; on interruption the read loop +can be paused and resumed cleanly. The AAC encoding via `MediaCodec` adds ~50 lines but produces +a reliable file in all interruption scenarios. This is a confirmed failure mode from multiple +Android developer reports and is the community recommendation. + +**`VOICE_COMMUNICATION` audio source**: Using this `AudioSource` constant (not `DEFAULT`) when +constructing `AudioRecord` applies system-level noise cancellation and echo suppression. This +improves Whisper accuracy in the driving and ambient-noise scenarios that are the primary use +case for this feature. + +## Consequences + +- `VoiceCaptureViewModel` receives an `AudioRecorder` via constructor injection with + `NoOpAudioRecorder` as the default. +- `androidMain` provides `AndroidAudioRecorder` (wired in `MainActivity`). +- `iosMain` provides `IosAudioRecorder` (wired in the iOS entry point). +- Desktop (`jvmMain`) uses `NoOpAudioRecorder` by default — voice capture is mobile-only per + requirements. +- The `VoiceCaptureViewModel` owns temp file lifecycle: creates the path, passes it to the STT + provider, deletes it in a `finally` block. +- Phase 2 (waveform animation): a `Flow` secondary method can be added to + `AudioRecorder` without breaking the primary `recordToFile()` contract. + +## Alternatives Considered + +**`expect class AudioRecorder`**: Eliminated. Forces `jvmMain` stub even though Desktop is out +of scope. `expect class` rigidity is unjustified for a single-consumer interface. + +**`Flow` PCM output**: Deferred to Phase 2 (waveform animation). Whisper does not +accept raw PCM and reassembly adds complexity not justified by v1 scope. + +**`MediaRecorder` on Android**: Eliminated due to confirmed MP4 corruption on audio focus loss. +`AudioRecord` + `MediaCodec` is the reliable alternative. diff --git a/project_plans/mobile-voice-mode/decisions/ADR-002-stt-provider-interface.md b/project_plans/mobile-voice-mode/decisions/ADR-002-stt-provider-interface.md new file mode 100644 index 000000000..f117b9381 --- /dev/null +++ b/project_plans/mobile-voice-mode/decisions/ADR-002-stt-provider-interface.md @@ -0,0 +1,124 @@ +# ADR-002: Speech-to-Text Provider Interface + +**Status**: Proposed +**Date**: 2026-04-18 + +## Context + +The voice pipeline requires a speech-to-text step that converts a recorded audio file into a +text transcript. Multiple STT backends exist with different trade-offs: + +- **OpenAI Whisper API** (`whisper-1`, `gpt-4o-mini-transcribe`): remote, paid ($0.003–$0.006/min), + works on all platforms from `commonMain` via Ktor multipart POST, accepts `.m4a` directly. +- **Android ML Kit GenAI STT** (`createOnDeviceSpeechRecognizer`): free, on-device, foreground-only + (raises `ErrorCode.BACKGROUND_USE_BLOCKED` from background contexts), requires API 31+. +- **iOS `SFSpeechRecognizer`**: free, on-device capable (iOS 13+), streaming output, per-task + duration limit (~1 min) requiring chunked usage for long recordings. +- **WhisperKit** (iOS, Argmax): on-device Core ML, production-ready (ICML 2025), Phase 2 scope. + +The seam pattern established by `TopicEnricher` / ADR-002 (import-topic-suggestions) specifies: +`suspend fun interface` with a `NoOp` default, injected at construction time. + +Whisper hallucination on silent audio is a confirmed issue (openai/whisper Discussion #1606): +transcribing silence produces "Thank you." and similar tokens. The interface must model this as a +distinct result state so the ViewModel can suppress LLM calls on empty transcripts. + +## Decision + +Define `SpeechToTextProvider` as a `suspend fun interface` in `commonMain` with a sealed result +type that models all failure modes without exceptions crossing the provider boundary: + +```kotlin +// commonMain — voice/SpeechToTextProvider.kt +fun interface SpeechToTextProvider { + suspend fun transcribe(audio: PlatformAudioFile): TranscriptResult +} + +sealed interface TranscriptResult { + data class Success(val text: String) : TranscriptResult + data object Empty : TranscriptResult + sealed interface Failure : TranscriptResult { + data object NetworkError : Failure + data class ApiError(val code: Int, val message: String) : Failure + data object AudioTooShort : Failure + data object PermissionDenied : Failure + } +} + +/** Default: no-op provider for tests and desktop. */ +object NoOpSpeechToTextProvider : SpeechToTextProvider { + override suspend fun transcribe(audio: PlatformAudioFile) = TranscriptResult.Empty +} +``` + +**Tier 1 default (v1)**: `WhisperSpeechToTextProvider` in `commonMain`. Ktor multipart POST to +`https://api.openai.com/v1/audio/transcriptions`. Uses user-supplied API key from settings. +Maps `transcript.split().size < 10` to `TranscriptResult.Empty` to gate Whisper silence +hallucination before the LLM call. + +**Tier 1a Android (Phase 2)**: `AndroidMlKitSpeechToTextProvider` in `androidMain`. Uses ML Kit +GenAI `createOnDeviceSpeechRecognizer` (API 31+). Free, on-device, foreground-only. Falls back +to `WhisperSpeechToTextProvider` when `ErrorCode.BACKGROUND_USE_BLOCKED` or device is below +API 31. + +**Tier 1b iOS (Story 3)**: `IosSpeechToTextProvider` in `iosMain`. Uses `SFSpeechRecognizer` +with `requiresOnDeviceRecognition = true` where supported. Requires chunking for recordings +longer than ~60 seconds. Falls back to `WhisperSpeechToTextProvider` when unavailable. + +**Tier 0 (tests / Desktop)**: `NoOpSpeechToTextProvider` — returns `TranscriptResult.Empty`. + +## Rationale + +**`suspend fun interface`**: Mirrors the `TopicEnricher` seam pattern exactly. Single-method +functional interface; implementors need no base class hierarchy. Callable from a coroutine +without additional wrapping. + +**Sealed `TranscriptResult` over exceptions**: Follows the `FetchResult` pattern in +`UrlFetcher.kt`. Exceptions do not cross provider boundaries — every failure mode is modeled as +a value type. This makes exhaustive `when` expressions possible in the ViewModel without +`try/catch` at the call site. `CancellationException` is the only exception the ViewModel +handles (to support pipeline cancellation). + +**`Empty` as a distinct state**: Whisper hallucination on silence is a confirmed issue. Returning +`TranscriptResult.Empty` rather than a short garbage string lets `VoiceCaptureViewModel` +surface a "Nothing was captured — try again" message and skip the LLM call entirely. The +word-count gate (`< 10 words`) is the recommended community mitigation for the standard +`whisper-1` model. + +**Whisper as v1 default**: Whisper is the only STT option that works identically on all targets +from a single `commonMain` code path. No platform-specific code is required for the happy path. +Platform native STT providers (ML Kit, SFSpeechRecognizer) are added as optional built-ins +behind the same interface for Phase 2 — free STT for users who prefer on-device processing. + +**No streaming interface**: "Real-time transcription display while speaking (result shown only +after processing)" is explicitly out of scope in `requirements.md`. A batch `suspend fun` +interface is sufficient. Streaming can be added as a parallel `Flow`-based variant in a future +phase without modifying the batch contract. + +## Consequences + +- `VoiceCaptureViewModel` receives `sttProvider: SpeechToTextProvider = NoOpSpeechToTextProvider` + as a constructor parameter. +- `WhisperSpeechToTextProvider` requires a Whisper API key from `VoiceSettings`. If the key is + absent at construction time, `transcribe()` returns `TranscriptResult.Failure.ApiError(401, ...)`. +- `AndroidMlKitSpeechToTextProvider` (Phase 2) requires `build.gradle.kts` addition: + `implementation("com.google.mlkit:genai-speech-recognition:1.x.x")` in `androidMain`. +- `IosSpeechToTextProvider` (Story 3) requires adding `Speech.framework` to the iOS cinterop + definition in `build.gradle.kts`. +- All providers must handle `CancellationException` propagation: do not catch it in the provider. +- The `TranscriptResult` sealed hierarchy is a stable public API commitment once the plugin + registry is added. + +## Alternatives Considered + +**Streaming `Flow` interface**: Eliminated for v1 because streaming display is +out of scope. Forces asymmetric implementations (Whisper is batch-only). Can be added as a +parallel interface in Phase 2 without changing the batch contract. + +**`SpeechRecognizer.createOnDeviceSpeechRecognizer` as v1 default (Android)**: Eliminated for v1 +because it is foreground-only and unavailable below API 31. Whisper is universal and covers all +device tiers. ML Kit is a Phase 2 enhancement for supported devices. + +**`openai-kotlin` v4.1.0 library for Whisper**: Eliminated. Raw Ktor multipart POST is ~50 lines +and Ktor is already present. The library adds ~1–2 MB and typed models not needed for the narrow +`transcribe()` surface. diff --git a/project_plans/mobile-voice-mode/decisions/ADR-003-llm-formatter-provider-interface.md b/project_plans/mobile-voice-mode/decisions/ADR-003-llm-formatter-provider-interface.md new file mode 100644 index 000000000..c0248f1a1 --- /dev/null +++ b/project_plans/mobile-voice-mode/decisions/ADR-003-llm-formatter-provider-interface.md @@ -0,0 +1,150 @@ +# ADR-003: LLM Formatter Provider Interface + +**Status**: Proposed +**Date**: 2026-04-18 + +## Context + +After speech-to-text transcription, the raw transcript is formatted by an LLM into Logseq +outliner syntax: top-level `- ` bullets for main topics, 2-space-indented sub-bullets, and +`[[wikilinks]]` only for terms explicitly named in the transcript. + +Multiple LLM backends are candidates: + +- **Anthropic Claude** (cloud, user-supplied key): via `POST /v1/messages`; Ktor-native; mirrors + the existing `ClaudeTopicEnricher` implementation pattern exactly. +- **OpenAI GPT** (cloud, user-supplied key): via `POST /v1/chat/completions`; compatible with + any OpenAI-compatible endpoint. +- **Apple FoundationModels** (iOS, on-device): shipped at WWDC 2025; text-in/text-out API; + offline, no cost; device gate: iPhone 15 Pro+, iOS 18.1+. Phase 2 scope. +- **No-op** (pass transcript verbatim): default for Story 1 — minimal end-to-end wire without + LLM cost or API key requirement. + +LLM hallucination of `[[wikilinks]]` to non-existent pages is a confirmed risk. The v1 system +prompt must constrain links to terms explicitly stated in the transcript. The raw transcript +must always be preserved as a collapsible block below the formatted output for user verification. + +## Decision + +Define `LlmFormatterProvider` as a `suspend fun interface` in `commonMain` with a sealed result +type following the same pattern as `TranscriptResult`: + +```kotlin +// commonMain — voice/LlmFormatterProvider.kt +fun interface LlmFormatterProvider { + suspend fun format(transcript: String, systemPrompt: String): LlmResult +} + +sealed interface LlmResult { + data class Success(val formattedText: String) : LlmResult + sealed interface Failure : LlmResult { + data object NetworkError : Failure + data class ApiError(val code: Int, val message: String) : Failure + data object Timeout : Failure + data class MalformedResponse(val raw: String) : Failure + } +} + +/** Default: returns the transcript verbatim, no formatting. Used in Story 1 and tests. */ +object NoOpLlmFormatterProvider : LlmFormatterProvider { + override suspend fun format(transcript: String, systemPrompt: String) = + LlmResult.Success(transcript) +} +``` + +**`ClaudeLlmFormatterProvider`** (Story 2): `commonMain`. Ktor POST to +`https://api.anthropic.com/v1/messages`. Mirrors `ClaudeTopicEnricher.kt` — same HTTP client +pattern, same error mapping. User-supplied API key from `VoiceSettings`. + +**`OpenAiLlmFormatterProvider`** (Story 2): `commonMain`. Ktor POST to +`https://api.openai.com/v1/chat/completions`. Compatible with any OpenAI-compatible endpoint +(Groq, local Ollama, etc.) via configurable `baseUrl`. + +**`NoOpLlmFormatterProvider`**: Story 1 default. Returns `LlmResult.Success(transcript)`. Allows +full end-to-end pipeline testing without an API key. The raw transcript IS the formatted output. + +**`AppleIntelligenceLlmFormatterProvider`** (Phase 2, `iosMain`): `FoundationModels` framework. +On-device, offline, no cost. Device gate: iPhone 15 Pro+, iOS 18.1+. Added as an optional +built-in in a future story. + +**System prompt**: The system prompt is a constructor parameter of `VoiceCaptureViewModel`, not +embedded in the interface. The v1 default prompt: + +``` +You are a personal note assistant. Convert the following voice transcript into Logseq outliner +markdown format: +- Use "- " (dash space) for all bullets +- Use exactly 2 spaces for each level of indentation +- Extract main topics as top-level bullets +- Add sub-points as indented bullets +- Add [[Page Name]] wiki links ONLY for proper nouns or topics explicitly named in the + transcript — do NOT invent links for terms not spoken +- Do NOT add a preamble, title, or summary +- Output ONLY the bullet list + + +{{TRANSCRIPT}} + +``` + +The transcript is inserted via `{{TRANSCRIPT}}` substitution in `VoiceCaptureViewModel`, not +by the provider. The provider receives the fully assembled system prompt string. + +## Rationale + +**`suspend fun interface`**: Identical seam pattern to `TopicEnricher` and `SpeechToTextProvider`. +Single-method, no base class dependency, directly callable from coroutines. + +**`NoOpLlmFormatterProvider` returning `Success(transcript)`**: Story 1 requires a working +end-to-end pipeline without LLM cost. Returning the transcript verbatim is safe and useful — +the user sees their spoken words appended to the journal, unformatted. The collapsible raw +transcript block below the formatted output satisfies the same purpose in Story 1. + +**Caller-controlled system prompt**: The system prompt for Logseq outliner format will need +iteration. Keeping it as a `VoiceCaptureViewModel` parameter (defaulting to the v1 prompt +constant) allows prompt tuning without touching the `LlmFormatterProvider` interface. This +matches the principle from `ClaudeTopicEnricher` where the prompt is internal to the enricher +implementation but could be constructor-injected. + +**`[[link]]` hallucination mitigation in v1**: The system prompt constraint ("ONLY for proper +nouns explicitly named in the transcript") is the minimum viable safeguard. A stronger v2 +mitigation — passing the graph page index to the LLM — is deferred because it adds context cost +and complexity. The raw transcript preserved as a collapsible block lets users verify what was +captured vs what the LLM produced. + +**No streaming interface**: Post-stop formatting latency is 1–3 seconds for typical voice notes. +Progressive display is not worth the implementation complexity for v1. The `VoiceCaptureState` +machine (`Formatting` state) communicates that work is in progress. + +## Consequences + +- `VoiceCaptureViewModel` constructor: `llmProvider: LlmFormatterProvider = NoOpLlmFormatterProvider`, + `systemPrompt: String = DEFAULT_VOICE_SYSTEM_PROMPT`. +- `ClaudeLlmFormatterProvider` and `OpenAiLlmFormatterProvider` require user-supplied API keys + from `VoiceSettings` (Story 2). If a key is absent, `format()` returns + `LlmResult.Failure.ApiError(401, "API key not configured")`. +- Both Story 2 providers live in `commonMain` — no platform code needed for cloud LLM calls. +- `AppleIntelligenceLlmFormatterProvider` (Phase 2) will live in `iosMain` and requires a + capability check at runtime (`FoundationModels.isAvailable()`). +- The `LlmResult` sealed hierarchy is a stable public API commitment once the plugin registry + is added. +- Truncation detection: if `formattedText` does not end with `.`, `?`, `!`, or `]]`, surface a + "Formatting may be incomplete" warning in the `VoiceCaptureState.Done` state. + +## Alternatives Considered + +**Batch string-out with `max_tokens` auto-calculation**: The LLM should be called with +`max_tokens = (transcriptWordCount * 1.5).toInt()` to avoid truncation. This is a +`VoiceCaptureViewModel` responsibility, not the provider's — providers receive the assembled +request body. + +**`koog` (JetBrains KMP agent framework, v0.7.3)**: Evaluated and eliminated. Pre-1.0 stability, +designed for multi-step agent workflows — overkill for a single `format()` call. Raw Ktor is 50 +lines and already present. + +**`AppleIntelligenceLlmFormatterProvider` in v1**: Deferred. Adds cinterop complexity and a +capability-check UX for a device gate (iPhone 15 Pro+ only). Phase 2 is the appropriate slot. + +**Structured JSON output mode**: LLM returns JSON `{"bullets": [...]}` which is parsed and +rendered to Logseq markdown client-side. Better format reliability but adds parsing complexity +and a JSON schema contract. Deferred to Phase 2 if zero-shot prompt compliance is insufficient. diff --git a/project_plans/mobile-voice-mode/decisions/ADR-004-plugin-registration.md b/project_plans/mobile-voice-mode/decisions/ADR-004-plugin-registration.md new file mode 100644 index 000000000..08c0597ae --- /dev/null +++ b/project_plans/mobile-voice-mode/decisions/ADR-004-plugin-registration.md @@ -0,0 +1,148 @@ +# ADR-004: Provider Registration and Wiring + +**Status**: Proposed +**Date**: 2026-04-18 + +## Context + +Three provider interfaces require wiring at app startup: `AudioRecorder`, `SpeechToTextProvider`, +and `LlmFormatterProvider`. The question is how platform entry points provide real implementations +to `VoiceCaptureViewModel`, which lives in `commonMain`. + +Options evaluated: + +1. **Constructor injection with `NoOp` defaults** — the seam pattern established by ADR-002 + (import-topic-suggestions project) and used by `TopicEnricher`, `UrlFetcher`, and `PageSaver`. +2. **Koin DI** — not currently used anywhere in SteleKit. `App.kt` uses `remember { }` blocks. +3. **`PluginHost` registry** — exists in the codebase but is designed for user-installed JS plugins + with `onEnable`/`onDisable` lifecycle. Architectural mismatch for built-in provider backends. +4. **`ServiceLoader`** (JVM) — platform-specific, no iOS equivalent. + +The existing `StelekitApp` composable already threads one optional provider via constructor +parameter: `urlFetcher: UrlFetcher = remember { NoOpUrlFetcher() }`. The voice pipeline adds +two more providers. A `VoicePipelineConfig` data class can bundle them to keep `StelekitApp`'s +parameter count manageable. + +## Decision + +Use **constructor injection with `NoOp` defaults** — identical to the existing seam pattern. + +```kotlin +// commonMain — voice/VoicePipelineConfig.kt +data class VoicePipelineConfig( + val audioRecorder: AudioRecorder = NoOpAudioRecorder, + val sttProvider: SpeechToTextProvider = NoOpSpeechToTextProvider, + val llmProvider: LlmFormatterProvider = NoOpLlmFormatterProvider, + val systemPrompt: String = DEFAULT_VOICE_SYSTEM_PROMPT, +) +``` + +`StelekitApp` gains one optional parameter: + +```kotlin +@Composable +fun StelekitApp( + fileSystem: PlatformFileSystem, + graphPath: String, + pluginHost: PluginHost = remember { PluginHost() }, + encryptionManager: EncryptionManager = remember { DefaultEncryptionManager() }, + urlFetcher: UrlFetcher = remember { NoOpUrlFetcher() }, + voicePipeline: VoicePipelineConfig = remember { VoicePipelineConfig() }, // new +) +``` + +`GraphContent` receives `voicePipeline: VoicePipelineConfig` and passes it to +`VoiceCaptureViewModel`: + +```kotlin +val voiceCaptureViewModel = remember { + VoiceCaptureViewModel( + audioRecorder = voicePipeline.audioRecorder, + sttProvider = voicePipeline.sttProvider, + llmProvider = voicePipeline.llmProvider, + systemPrompt = voicePipeline.systemPrompt, + journalService = repos.journalService, + scope = scope, + ) +} +``` + +**Android `MainActivity`** wires real providers when API keys are configured: + +```kotlin +StelekitApp( + // ... existing params ... + voicePipeline = VoicePipelineConfig( + audioRecorder = AndroidAudioRecorder(context), + sttProvider = buildSttProvider(whisperApiKey), + llmProvider = buildLlmProvider(anthropicKey, openAiKey), + ) +) + +private fun buildSttProvider(whisperKey: String?): SpeechToTextProvider = + if (whisperKey.isNullOrBlank()) NoOpSpeechToTextProvider + else WhisperSpeechToTextProvider(httpClient, whisperKey) + +private fun buildLlmProvider(anthropicKey: String?, openAiKey: String?): LlmFormatterProvider = + when { + !anthropicKey.isNullOrBlank() -> ClaudeLlmFormatterProvider(httpClient, anthropicKey) + !openAiKey.isNullOrBlank() -> OpenAiLlmFormatterProvider(httpClient, openAiKey) + else -> NoOpLlmFormatterProvider + } +``` + +**iOS entry point** (`Main.kt`/`MainViewController.kt`) follows the same pattern with +`IosAudioRecorder`. + +**Desktop** (`Main.kt`): passes the default `VoicePipelineConfig()` — all `NoOp` providers, +voice capture is mobile-only. + +**Tests** inject `FakeAudioRecorder`, `FakeSpeechToTextProvider`, `FakeLlmFormatterProvider` +directly into `VoiceCaptureViewModel`. + +## Rationale + +**Seam pattern consistency**: Every provider in SteleKit is wired this way (`TopicEnricher`, +`UrlFetcher`, `PageSaver`). Deviating for this feature introduces a second DI pattern in the +codebase. Cognitive consistency is a project value — from ADR-002: "Inject `topicEnricher` into +`ImportViewModel` as a constructor parameter ... mirroring the `pageSaver` seam pattern exactly." + +**`VoicePipelineConfig` bundles the new parameters**: `StelekitApp` currently has 5 parameters. +Adding 3 more individually would make the signature unwieldy. A `data class` bundles them with a +sensible default and communicates that these three concerns belong together. This is the same +refactor that would apply to `urlFetcher` + any future enrichment providers. + +**`NoOp` defaults ensure local-first behavior**: A developer running the app without API keys +configured gets the pipeline in a "no-op" state — the mic button is visible but the +`VoiceCaptureViewModel` surfaces "No STT provider configured" rather than crashing. No null +checks needed anywhere in the ViewModel. + +**Plugin registry is out of scope**: Full auto-discovery via `ServiceLoader` or a plugin +registry is deferred to the `stelekit-plugin-api` project (per ADR-002, import-topic-suggestions). +Third-party plugins provide `SpeechToTextProvider` and `LlmFormatterProvider` implementations +and pass them to `StelekitApp` at assembly time — identical to how `TopicEnricher` works today. + +## Consequences + +- `StelekitApp` gains `voicePipeline: VoicePipelineConfig = remember { VoicePipelineConfig() }`. +- `GraphContent` gains `voicePipeline: VoicePipelineConfig` and passes it to `VoiceCaptureViewModel`. +- Android `MainActivity` must construct `AndroidAudioRecorder` with a `Context` reference. + `AndroidAudioRecorder` must not hold a long-lived `Context` after recording ends (use + `applicationContext` only). +- iOS entry point must construct `IosAudioRecorder`. +- `VoiceSettings` (Story 2) reads API keys from the Android Keystore / iOS Keychain. The + platform entry point reconstructs providers when settings change (or uses a + `StateFlow` if hot-swapping is desired in a future story). +- No new Gradle dependencies for provider wiring — Ktor is already in `commonMain`. + +## Alternatives Considered + +**Koin DI**: Eliminated. Not currently used in SteleKit. Introduces a new dependency and a second +DI pattern. `App.kt` uses `remember { }` blocks throughout — Koin would be the only exception. + +**`PluginHost` registry**: Eliminated. `Plugin` interface has `onEnable`/`onDisable` lifecycle +and metadata fields appropriate for user-installed JS plugins, not built-in provider backends. +ADR-002 (import-topic-suggestions) explicitly deferred full plugin registry to `stelekit-plugin-api`. + +**Individual `StelekitApp` parameters for each provider**: Eliminated. Would grow `StelekitApp` +to 8+ parameters. `VoicePipelineConfig` is the cleaner aggregation. diff --git a/project_plans/mobile-voice-mode/decisions/ADR-005-voice-capture-ui-state-machine.md b/project_plans/mobile-voice-mode/decisions/ADR-005-voice-capture-ui-state-machine.md new file mode 100644 index 000000000..45dc172af --- /dev/null +++ b/project_plans/mobile-voice-mode/decisions/ADR-005-voice-capture-ui-state-machine.md @@ -0,0 +1,159 @@ +# ADR-005: Voice Capture UI State Machine + +**Status**: Proposed +**Date**: 2026-04-18 + +## Context + +The voice capture pipeline transitions through a series of states from user tap to journal insert. +The UI must reflect each stage clearly so users understand what the system is doing without +requiring active attention (eyes-free use case). + +States required: idle (waiting for tap), recording (mic open, audio accumulating), post-recording +processing (transcription), post-transcription processing (LLM formatting), completion, and error. +Users must be able to cancel at any point. + +The existing pattern for pipeline state in SteleKit: `sealed interface` state type exposed via +`StateFlow`, a single cancellable `Job` field in the ViewModel. `ImportViewModel.scanJob` is the +reference implementation. + +UI placement: `PlatformBottomBar.android.kt` currently has 4 navigation items. The mic button +should be a visually distinct center affordance (FAB-style) rather than a 5th navigation icon, +consistent with Material Design 3 FAB-in-bottom-bar patterns. + +## Decision + +**State machine:** + +```kotlin +// commonMain — voice/VoiceCaptureState.kt +sealed interface VoiceCaptureState { + data object Idle : VoiceCaptureState + data object Recording : VoiceCaptureState + data object Transcribing : VoiceCaptureState + data object Formatting : VoiceCaptureState + data class Done(val insertedText: String) : VoiceCaptureState + data class Error(val stage: PipelineStage, val message: String) : VoiceCaptureState +} + +enum class PipelineStage { RECORDING, TRANSCRIPTION, LLM, JOURNAL_INSERT } +``` + +Valid transitions: +- `Idle` → `Recording`: user taps mic button +- `Recording` → `Transcribing`: user taps stop, or `stopRecording()` is called +- `Transcribing` → `Formatting`: `TranscriptResult.Success` received +- `Transcribing` → `Done`: `TranscriptResult.Empty` (no LLM call, nothing to insert) +- `Transcribing` → `Error`: `TranscriptResult.Failure` +- `Formatting` → `Done`: `LlmResult.Success`, journal insert complete +- `Formatting` → `Error`: `LlmResult.Failure` +- Any state → `Idle`: user cancels, dismisses error, or `Done` auto-resets after 3 seconds + +**ViewModel:** + +```kotlin +// commonMain — voice/VoiceCaptureViewModel.kt +class VoiceCaptureViewModel( + private val audioRecorder: AudioRecorder, + private val sttProvider: SpeechToTextProvider, + private val llmProvider: LlmFormatterProvider, + private val journalService: JournalService, + private val systemPrompt: String, + private val scope: CoroutineScope, +) { + private val _state = MutableStateFlow(VoiceCaptureState.Idle) + val state: StateFlow = _state.asStateFlow() + + private var pipelineJob: Job? = null + + fun onMicTapped() { /* Idle → start; Recording → stop */ } + fun cancel() { pipelineJob?.cancel(); _state.value = VoiceCaptureState.Idle } + fun dismissError() { _state.value = VoiceCaptureState.Idle } +} +``` + +**`VoiceCaptureButton` composable** (`commonMain/ui/components/VoiceCaptureButton.kt`): + +- `Idle`: mic icon (outlined), normal size +- `Recording`: pulsing red circle + stop icon, `contentDescription = "Stop recording"` +- `Transcribing` / `Formatting`: `CircularProgressIndicator` replacing the icon, disabled +- `Done`: checkmark icon, auto-resets to `Idle` after 3 seconds via `LaunchedEffect` +- `Error`: error icon (red); tapping dismisses to `Idle` + +The button is added to `PlatformBottomBar` as a center floating element, respecting the +`isLeftHanded` flag for position adjustment. + +**`PlatformBottomBar` modification:** The button is passed in as a composable slot parameter +rather than hardcoded, to keep `PlatformBottomBar` testable and avoid coupling it to +`VoiceCaptureViewModel`: + +```kotlin +@Composable +expect fun PlatformBottomBar( + currentScreen: Screen, + onNavigate: (Screen) -> Unit, + onSearch: () -> Unit, + isLeftHanded: Boolean, + voiceCaptureButton: @Composable () -> Unit = {}, // new slot +) +``` + +`GraphContent` passes `voiceCaptureButton = { VoiceCaptureButton(voiceCaptureViewModel.state, ...) }`. + +## Rationale + +**Sealed interface state**: Exhaustive `when` expressions in the composable are guaranteed by the +compiler — no `else` branch needed, no missed state. Adding a new state (e.g., `Paused` for Phase +3 foreground service) is a compile-time breaking change that forces all consumers to handle it. + +**Separate `PipelineStage` enum in `Error`**: Users and logs need to know where the pipeline +failed (recording, transcription, LLM formatting, or journal insert). A string message alone +is insufficient for actionable UX ("Recording was interrupted — tap to retry" vs "Transcription +failed — check your API key"). + +**Single cancellable `Job`**: Mirrors `ImportViewModel.scanJob`. The entire pipeline runs inside +one `Job`; `cancel()` propagates `CancellationException` through the pipeline and triggers +`finally` blocks (which clean up the temp audio file). No additional synchronization needed. + +**Auto-reset from `Done`**: Users should be able to immediately start a new recording after +completion. A 3-second `Done` state shows the confirmation, then `Idle` is restored +automatically. If the user taps during `Done`, the reset fires immediately. + +**`VoiceCaptureButton` as a slot in `PlatformBottomBar`**: Keeps `PlatformBottomBar` unaware of +the voice pipeline. The ViewModel is created in `GraphContent` and passed down as a composable +lambda. This is the same pattern as other bottom bar callbacks (`onSearch`, `onNavigate`). + +**Phase gates encoded in state semantics**: +- Phase 1 (Story 1): `Formatting` state is entered but `NoOpLlmFormatterProvider` returns + immediately — the user sees a brief `Formatting` flash before `Done`. Acceptable for Story 1. +- Phase 2: `AndroidMlKitSpeechToTextProvider` replaces Whisper on supported devices; no state + changes needed. +- Phase 3: A `Paused` state can be added for foreground service scenarios (recording paused + by phone call interruption). All existing `when` branches compile-error until handled. + +## Consequences + +- `VoiceCaptureViewModel` is created in `GraphContent` alongside `JournalsViewModel` and + `AllPagesViewModel` using the `remember { }` pattern. +- `PlatformBottomBar` signature adds `voiceCaptureButton: @Composable () -> Unit = {}` — a + backward-compatible default. +- Both `PlatformBottomBar.android.kt` and any `iosMain` actual must handle the new slot. +- Lifecycle: `VoiceCaptureViewModel.cancel()` must be called from the `DisposableEffect` in + `GraphContent` on `Lifecycle.Event.ON_PAUSE` to stop recording if the app is backgrounded + mid-capture (Phase 1 foreground-only). +- The `Done` auto-reset uses `LaunchedEffect(state)` in `VoiceCaptureButton` — no timer + management in the ViewModel. +- `VoiceCaptureState` must be in `commonMain` — iOS and Android share the same state type. + +## Alternatives Considered + +**FAB outside `PlatformBottomBar`**: A separate `FloatingActionButton` composable added directly +in `MainLayout` would avoid modifying `PlatformBottomBar`. Eliminated because it requires +`MainLayout` to know about voice capture, breaking its current separation of concerns. + +**`Paused` state in v1**: Eliminated. Background recording is Phase 3 scope. Adding `Paused` +prematurely adds `when` branches to all consumers without delivering value. + +**Error with retry**: The `Error` state includes `stage` to support a "Retry from [stage]" +action in a future story. For v1, tapping the error dismisses to `Idle` — the user retaps the +mic to start a fresh recording. diff --git a/project_plans/mobile-voice-mode/decisions/ADR-006-voice-pipeline-config-class-vs-data-class.md b/project_plans/mobile-voice-mode/decisions/ADR-006-voice-pipeline-config-class-vs-data-class.md new file mode 100644 index 000000000..249350156 --- /dev/null +++ b/project_plans/mobile-voice-mode/decisions/ADR-006-voice-pipeline-config-class-vs-data-class.md @@ -0,0 +1,23 @@ +# ADR-006: VoicePipelineConfig — `class` instead of `data class` + +**Status**: Accepted +**Date**: 2026-04-19 + +## Context + +`VoicePipelineConfig` was initially planned as a `data class` (see ADR-004). During implementation it was changed to a plain `class`. + +## Decision + +Use `class`, not `data class`. + +## Reason + +`data class` generates `equals()` and `hashCode()` that compare field values structurally. Two of the three fields in `VoicePipelineConfig` — `sttProvider` and `llmProvider` — are `suspend fun interface` instances (lambda or anonymous object). These types do not implement stable `equals()`, so structural equality is effectively identity equality. The generated `equals()` would be misleading: two configs built from the same API key would compare as unequal because each `withDefaults()` call produces a different object instance. + +Since `VoicePipelineConfig` is used for injection (wired in `MainActivity` and compared nowhere), structural equality adds no value while the `data class` contract creates false expectations. A plain `class` removes the pretense. + +## Consequences + +- No `copy()` method — build a new config via the constructor or `buildVoicePipeline()`. +- `equals()` falls back to identity, which is correct for mutable pipeline configurations. diff --git a/project_plans/mobile-voice-mode/requirements.md b/project_plans/mobile-voice-mode/requirements.md new file mode 100644 index 000000000..fcbc11313 --- /dev/null +++ b/project_plans/mobile-voice-mode/requirements.md @@ -0,0 +1,58 @@ +# Requirements: Mobile Voice Mode + +**Status**: Draft | **Phase**: 1 — Ideation complete +**Created**: 2026-04-18 + +## Problem Statement + +Users want to capture thoughts hands-free in situations where looking at or typing on a phone is impractical or unsafe — driving, parties, walking. Current SteleKit requires active text input. This feature enables a one-tap voice capture flow that transcribes speech, formats it into Logseq-compatible outliner syntax with `[[links]]`, and appends it to the daily journal — all without manual editing. + +## Success Criteria + +- User taps a mic button, speaks for any duration, and a correctly formatted outliner entry appears in today's journal page +- Works entirely hands-free: no keyboard interaction required from trigger to saved entry +- LLM provider is configurable per-user via an extensible plugin API (not hardcoded to one vendor) +- Pipeline works on Android and iOS via a platform-agnostic commonMain design with thin platform adapters + +## Scope + +### Must Have (MoSCoW) +- Single-tap microphone trigger accessible from the main UI (no navigation required) +- Unlimited-duration voice recording — user stops when done, no timeout +- Speech-to-text transcription (Whisper API or platform STT) +- LLM pass to convert raw transcript → Logseq outliner format with `[[wikilinks]]` +- Automatic insertion into today's daily journal page +- Extensible LLM provider API: pluggable backends (remote API key, on-device ML, custom) +- Support for at least: Remote API w/ user-supplied key (OpenAI / Anthropic / compatible), on-device platform LLM (Android AICore / Gemini Nano, Apple Intelligence where available) + +### Out of Scope +- Multi-language support (English only at launch) +- Real-time transcription display while speaking (result shown only after processing) +- Raw audio file storage or playback after transcription +- Background/passive listening (user must actively trigger each recording) +- Desktop platform support for this feature + +## Constraints + +- **Tech stack**: Kotlin Multiplatform; shared logic in `commonMain`, platform audio capture in `androidMain` / `iosMain`; Compose Multiplatform UI +- **Timeline**: No hard deadline — ship when well-architected +- **Dependencies**: Daily journal page must already exist or be auto-created (existing behavior) +- **Plugin API**: Must be designed so third-party or community plugins can add new STT/LLM backends without forking core code + +## Context + +### Existing Work +- SteleKit already has a daily journal concept and page insertion via `GraphWriter` +- `import-topic-suggestions` project established a plugin/provider interface pattern (ADR-002-topic-enricher-plugin-interface.md) — this feature should follow the same extensibility model +- Branch `stelekit-mobile-mode` is the working branch for mobile-first improvements + +### Stakeholders +- Primary user: Tyler (solo developer and user) — needs discreet, eyes-free note capture while driving or in social settings +- Future users who want voice-driven knowledge capture with their own LLM credentials + +## Research Dimensions Needed + +- [ ] Stack — evaluate STT options (Whisper API, Android SpeechRecognizer, iOS SFSpeechRecognizer, platform ML kits) and LLM API client libraries for KMP +- [ ] Features — survey comparable voice-to-notes tools (Audiopen, Whisper Memos, Notion AI voice) for UX patterns +- [ ] Architecture — plugin interface design for STT+LLM providers, commonMain pipeline, platform audio capture adapter pattern +- [ ] Pitfalls — microphone permissions across platforms, audio focus management (Android), on-device model availability gates, LLM prompt design for outliner fidelity diff --git a/project_plans/mobile-voice-mode/research/architecture.md b/project_plans/mobile-voice-mode/research/architecture.md new file mode 100644 index 000000000..736e9618c --- /dev/null +++ b/project_plans/mobile-voice-mode/research/architecture.md @@ -0,0 +1,626 @@ +# Findings: Architecture + +**Feature**: Voice capture → STT → LLM → outliner format → journal +**Date**: 2026-04-18 +**Codebase read**: `App.kt`, `PluginSystem.kt`, `PlatformFileSystem.kt`, `TopicEnricher.kt`, +`ClaudeTopicEnricher.kt`, `UrlFetcher.kt`, `JournalService.kt`, `PlatformBottomBar.android.kt`, +`ADR-002-topic-enricher-plugin-interface.md` + +--- + +## Summary + +The voice pipeline has five distinct seams: audio capture, speech-to-text, LLM formatting, journal +insertion, and UI orchestration. Three of these (STT, LLM, and potentially audio capture) are +natural plugin extension points. The codebase already has a proven seam pattern — `suspend fun +interface` with a `NoOp` default injected at ViewModel construction time — established by +`TopicEnricher`/`UrlFetcher`. This architecture extends that pattern directly. The only new +complexity is the `expect/actual` audio capture layer, which must live in `androidMain` / +`iosMain` because there is no KMP audio recording API in `commonMain`. + +Recommended architecture: a `VoiceCaptureViewModel` in `commonMain` orchestrating a pipeline +through three `suspend fun interface` seams — `AudioRecorder`, `SpeechToTextProvider`, and +`LlmProvider` — all injected at construction time with `NoOp` defaults. The ViewModel exposes a +single `StateFlow` that drives a `VoiceCaptureButton` composable embedded in +`PlatformBottomBar.android.kt`. + +--- + +## Options Surveyed + +### Q1 — Audio Capture: expect/actual shape and data representation + +**Option A: `expect class AudioRecorder`** +A platform class with `suspend fun start()`, `suspend fun stop(): ByteArray`. Android actual uses +`AudioRecord` (PCM) or `MediaRecorder` (encoded); iOS actual uses `AVAudioRecorder`. +- Pro: encapsulates all platform API inside the actual; commonMain sees only a typed Kotlin class. +- Con: `expect class` rules in KMP are strict — every property/method in the expect must be in every + actual. Adding methods later requires all actuals to update in lockstep. + +**Option B: `expect fun interface AudioRecorder`** (preferred) +A `suspend fun interface` in `commonMain/platform/` with a single `suspend fun record(): Flow` or `suspend fun recordToFile(): PlatformAudioFile`. Actuals are in `androidMain` / `iosMain`. +- Pro: matches the existing `TopicEnricher` / `UrlFetcher` seam pattern. Single-method interface. + No `expect class` rigidity. +- Con: `expect fun interface` is not directly supported in KMP — `expect interface` is, but + `fun interface` keyword only applies to actual implementations. [TRAINING_ONLY — verify exact + KMP syntax for expect interfaces] + +**Option C: `expect object AudioRecorderFactory`** +Factory that returns a platform audio recorder instance. Used in `androidMain` / `iosMain`. +- Same structural result as Option B but with a Factory wrapper. + +**Data representation — ByteArray vs File vs Flow:** +- `ByteArray`: simple, no disk I/O, but holds entire recording in memory. Problematic for long + recordings. Acceptable for a typical 10–60s voice note. +- `File` (temp file path): avoids memory spike; Whisper API accepts a file upload directly. + Requires cleanup logic. Works well with Android `MediaRecorder` (outputs to file natively) and + iOS `AVAudioRecorder` (also outputs to URL/file). +- `Flow` (streaming PCM): enables real-time display of audio level for the mic + animation. Adds complexity; STT providers vary on whether they accept streaming audio. + +**Recommendation**: Use a temp file. Both `MediaRecorder` (Android) and `AVAudioRecorder` (iOS) +naturally record to a file. The Whisper API and most STT services accept file upload. A temp file +avoids memory spikes. Expose it as a `PlatformAudioFile` value class wrapping a path string — the +same cross-platform path convention already used by `PlatformFileSystem`. + +**Audio format**: `m4a` / AAC is the lowest-friction choice — both platforms record it natively, +Whisper accepts it, file sizes are small (~1 MB/min). PCM/WAV is universal but large. +[TRAINING_ONLY — verify Whisper API accepted formats for m4a] + +--- + +### Q2 — STT Provider Interface + +**Option A: Batch interface** +```kotlin +fun interface SpeechToTextProvider { + suspend fun transcribe(audioFile: PlatformAudioFile): TranscriptResult +} +``` +- Pro: matches the `TopicEnricher` pattern exactly. Simplest interface; easy to test with a + `FakeSpeechToTextProvider`. Compatible with Whisper API (POST audio file → transcript string). +- Con: no progress feedback during transcription. For a 60s audio file, Whisper API typically + responds in 2–8s — acceptable without streaming. + +**Option B: Streaming interface** +```kotlin +fun interface SpeechToTextProvider { + fun transcribeStreaming(audioFile: PlatformAudioFile): Flow +} +``` +- Pro: enables real-time partial results; better UX for long recordings. +- Con: significantly more complex. Whisper API does not offer streaming; only platform + SpeechRecognizer / SFSpeechRecognizer support streaming. Forces asymmetric implementations. + Requirements say "result shown only after processing" — streaming is out of scope. + +**Sealed result type:** +```kotlin +sealed interface TranscriptResult { + data class Success(val text: String) : TranscriptResult + data object Empty : TranscriptResult + sealed interface Failure : TranscriptResult { + data object NetworkError : Failure + data class ApiError(val code: Int) : Failure + data object AudioTooShort : Failure + data object PermissionDenied : Failure + } +} +``` +Models the same pattern as `FetchResult` in `UrlFetcher.kt`. + +**Recommendation**: Batch interface, Option A. Out-of-scope requirement confirmed in +`requirements.md`: "Real-time transcription display while speaking (result shown only after +processing)". The sealed `TranscriptResult` type handles all failure modes without exceptions +crossing the provider boundary. + +**Built-in providers to ship:** +1. `WhisperSpeechToTextProvider` (Ktor + Whisper API, user-supplied key) — commonMain, works on both platforms +2. `AndroidSpeechToTextProvider` (Android `SpeechRecognizer` — free, on-device) — androidMain +3. `NoOpSpeechToTextProvider` (returns `TranscriptResult.Empty`) — default, tests + +--- + +### Q3 — LLM Provider Interface + +**Option A: Batch string-in / string-out** +```kotlin +fun interface LlmProvider { + suspend fun format(transcript: String, systemPrompt: String): LlmResult +} +``` +- Pro: identical seam pattern to `TopicEnricher`. Caller controls the prompt; provider is purely + mechanical. Testable with a `FakeLlmProvider`. +- Con: no streaming. Acceptable given requirements scope. + +**Option B: Streaming Flow** +```kotlin +interface LlmProvider { + fun formatStreaming(transcript: String, systemPrompt: String): Flow +} +``` +- Pro: progressive display of formatted output. +- Con: requirements say no real-time display during processing. Adds complexity. + +**Sealed result type:** +```kotlin +sealed interface LlmResult { + data class Success(val formattedText: String) : LlmResult + sealed interface Failure : LlmResult { + data object NetworkError : Failure + data class ApiError(val code: Int) : Failure + data object Timeout : Failure + data class MalformedResponse(val raw: String) : Failure + } +} +``` + +**System prompt strategy**: The system prompt (instruction to format as Logseq outliner with +`[[wikilinks]]`) should be a constructor parameter of `VoiceCaptureViewModel`, not baked into +the interface. This makes the prompt configurable without changing the `LlmProvider` contract. +Follows the principle established by `ClaudeTopicEnricher` where the prompt is internal to the +implementation. + +**Built-in providers:** +1. `ClaudeLlmProvider` (Ktor + Anthropic Messages API, user-supplied key) — mirrors `ClaudeTopicEnricher` exactly, commonMain +2. `OpenAiLlmProvider` (Ktor + OpenAI Chat API, compatible with any OpenAI-compatible endpoint) +3. `NoOpLlmProvider` (returns transcript verbatim, no formatting) — default +4. `AndroidOnDeviceLlmProvider` (Android AICore / Gemini Nano) — androidMain only, gated on API availability [TRAINING_ONLY — verify Android AICore/ML Kit Gemini Nano Kotlin API] + +--- + +### Q4 — Pipeline Orchestration + +**Option A: ViewModel owns the pipeline** +`VoiceCaptureViewModel` calls `audioRecorder.recordToFile()`, then `sttProvider.transcribe()`, +then `llmProvider.format()`, then `journalService.appendToToday()`. All within a +`viewModelScope.launch {}` coroutine. Matches `ImportViewModel.scanJob` pattern exactly — a single +`Job` field that can be cancelled if the user dismisses. + +**Option B: UseCase / domain service** +A `VoicePipelineUseCase` in `domain/` that owns the sequential calls. ViewModel launches the +use case and observes a `StateFlow`. +- Pro: separates orchestration concern from UI state management. +- Con: `audioRecorder` has a coroutine lifecycle (must be cancelled on ViewModel disposal). Pushing + it into a UseCase without a scope is awkward. ADR-002 explicitly decided against domain-layer + async injection for this same reason: "ViewModel already manages a coroutine scope and a + `scanJob`; the enricher coroutine is a natural second job alongside it." + +**Option C: Service (long-lived)** +A `VoiceCaptureService` (Android foreground service) for background recording. +- Requirements: "user must actively trigger each recording", no background listening. + Foreground service is overkill and out of scope. + +**Recommendation**: Option A. ViewModel owns the pipeline. A single `pipelineJob: Job?` field, +cancellable on user abort or ViewModel disposal. Pipeline errors are caught in `try/catch` and +mapped to `VoiceCaptureState.Error(...)`. Same pattern as `ImportViewModel.scanJob`. + +**Error handling**: Wrap each stage call in `try/catch`. Map exceptions to sealed `VoiceCaptureState` +variants. Propagate partial results where useful (e.g., if LLM fails, offer the raw transcript as +the fallback insert). Never throw across provider boundaries — providers return sealed results, not +exceptions, so the ViewModel only catches `CancellationException`. + +--- + +### Q5 — Plugin Registration + +**Option A: Constructor injection with NoOp defaults** (seam pattern) +Providers injected into `VoiceCaptureViewModel(sttProvider = NoOpSpeechToTextProvider(), llmProvider = NoOpLlmProvider(), ...)`. Host app (Android `MainActivity`, iOS `Main.kt`) wires real providers. +- Pro: zero infrastructure; no registry, no DI framework, no annotation processor. Testable. + Matches every other seam in this codebase. +- Con: host app must explicitly wire each provider. Fine for v1 with a small set of built-ins. + +**Option B: Koin DI** +Define a Koin module for voice providers. App module includes it; tests replace with test module. +- Pro: clean separation of wiring from construction. Standard in many KMP projects. +- Con: Koin is not currently used in SteleKit (`PluginSystem.kt` uses direct constructor injection, + `GraphContent` in `App.kt` uses `remember { }` blocks). Introducing Koin for one feature adds a + new dependency and pattern inconsistent with the rest of the codebase. + [TRAINING_ONLY — verify whether Koin is in `kmp/build.gradle.kts`] + +**Option C: PluginHost registry** +Register `SpeechToTextProvider` implementations as `Plugin` instances in `PluginHost`. The +existing `PluginHost` class has `registerPlugin(plugin: Plugin)` and `getAllPlugins()`. +- Con: `Plugin` interface has `onEnable`/`onDisable` lifecycle, `id`/`name`/`version` metadata. + This lifecycle is appropriate for user-installed JS plugins, not for built-in provider backends. + Mixing the two concerns would complicate `PluginHost`. ADR-002 also explicitly deferred full + plugin registry to a future `stelekit-plugin-api` project. + +**Option D: ServiceLoader (JVM only)** +`java.util.ServiceLoader` for JVM/Android, manual registration for iOS. +- Con: platform-specific; not KMP-compatible without wrappers. iOS has no equivalent. + +**Recommendation**: Option A. Extend the seam pattern. `StelekitApp` gains two optional +parameters: `sttProvider: SpeechToTextProvider = NoOpSpeechToTextProvider()` and +`llmProvider: LlmProvider = NoOpLlmProvider()`. `GraphContent` passes them through to +`VoiceCaptureViewModel`. This is identical to how `urlFetcher: UrlFetcher = NoOpUrlFetcher()` is +threaded today. Third-party plugins provide implementations at app-assembly time. + +--- + +### Q6 — Compose UI Integration: Mic Button State Machine + +**State machine:** +``` +Idle → Recording → Transcribing → Formatting → Done + ↘ Error +Any state → Idle (on user cancel / dismiss) +``` + +Mapped to a sealed class: +```kotlin +sealed interface VoiceCaptureState { + data object Idle : VoiceCaptureState + data object Recording : VoiceCaptureState + data object Transcribing : VoiceCaptureState + data object Formatting : VoiceCaptureState + data class Done(val insertedText: String) : VoiceCaptureState + data class Error(val stage: PipelineStage, val message: String) : VoiceCaptureState +} + +enum class PipelineStage { RECORDING, TRANSCRIPTION, LLM, JOURNAL_INSERT } +``` + +**UI placement**: The mic button fits naturally in `PlatformBottomBar.android.kt`. The current +bottom nav has 4 items (Journals, Pages, Search, Notifications). A 5th mic item can be added as +a `FloatingActionButton`-style center affordance, or replace Search (since search is a dialog +trigger, not a destination). Given `isLeftHanded` is already handled, the mic button position +(left vs right) can follow the same flag. + +The button shows: +- `Idle`: mic icon, tap to start +- `Recording`: pulsing red circle + stop icon, tap to stop +- `Transcribing` / `Formatting`: spinner with label +- `Done`: brief checkmark then revert to Idle +- `Error`: error icon with toast/snackbar; tap to retry or dismiss + +`VoiceCaptureViewModel` is created in `GraphContent` alongside the existing ViewModels (same +`remember { }` pattern). It is passed down to `PlatformBottomBar` as a state parameter so the +bottom bar remains a composable that only observes state — it does not own the ViewModel. + +--- + +## Trade-off Matrix + +| Axis | Seam pattern (recommended) | Koin DI | PluginHost registry | +|---|---|---|---| +| Extensibility | Good — any impl injected at assembly | Good — module swap | Poor — lifecycle mismatch | +| Testability | Excellent — NoOp defaults | Good — test module | Fair — requires lifecycle stubs | +| Platform coupling | Low — expect/actual only in audio capture | Low | Low | +| Boilerplate | Low — matches existing code | Medium — Koin setup | Medium — Plugin wrapper | +| DI compatibility | N/A — no DI needed | Requires Koin dependency | Requires PluginHost wiring | +| Consistency with codebase | Excellent — identical to TopicEnricher | Poor — new pattern | Fair — extends existing but mismatches | + +--- + +## Risk and Failure Modes + +**R1 — Microphone permission denied** +Both Android (`RECORD_AUDIO`) and iOS (`NSMicrophoneUsageDescription`) require runtime permission. +The `AudioRecorder` actual must check permission before starting and return `TranscriptResult.Failure.PermissionDenied` (surfaced to UI as a rationale dialog). Android 13+ requires explicit request; iOS permission is granted once at first use. + +**R2 — Audio focus conflict (Android)** +`MediaRecorder` does not request audio focus by default. Background music will bleed into the +recording. The Android actual should request `AudioManager.AUDIOFOCUS_GAIN_TRANSIENT` before +starting and abandon it after stopping. [TRAINING_ONLY — verify AudioFocus API behavior with MediaRecorder] + +**R3 — Whisper API key not configured** +`WhisperSpeechToTextProvider` receives an empty or missing API key. Should fail fast at +construction time with a descriptive `IllegalStateException`, not at transcription time. The UI +should surface a settings prompt to add an API key. + +**R4 — LLM hallucinates wikilinks** +The LLM may invent `[[links]]` to non-existent pages. This is acceptable per requirements +(existing pages list is not provided to the LLM in v1). Future enhancement: pass page name index +as context. + +**R5 — Recording too short / silence** +Whisper returns an empty transcript for silence or sub-1s audio. `WhisperSpeechToTextProvider` +should map empty responses to `TranscriptResult.Empty`, which the ViewModel surfaces as a +dismissible "Nothing was captured" message. + +**R6 — No internet connectivity** +Both Whisper and LLM calls require network. The pipeline should check connectivity before launching +and return `Failure.NetworkError` immediately rather than waiting for a timeout. + +**R7 — iOS AVAudioSession category conflict** +`AVAudioRecorder` requires setting `AVAudioSession.sharedInstance().setCategory(.record)`. +This silences other audio. Must be restored to `.playback` or `.ambient` after recording. The +iOS actual must handle `AVAudioSession` activation/deactivation within the recording lifecycle. +[TRAINING_ONLY — verify AVAudioSession best practices for record + restore] + +**R8 — Journal page creation race** +If `JournalService.ensureTodayExists()` is called from two coroutines simultaneously (voice +capture + JournalsViewModel startup), a duplicate page could be created. `JournalService` already +uses a `Mutex` for this case — confirmed in `JournalService.kt` line 36. No additional +synchronization needed. + +--- + +## Migration and Adoption Cost + +**Adding the feature from scratch (no existing voice code):** +- New files: ~8 (interfaces, NoOp implementations, ViewModel, 2 platform actuals, UI component, + 1 platform bottom bar modification) +- Modified files: `App.kt` (add `sttProvider`/`llmProvider` params), `PlatformBottomBar.android.kt` + (add mic button), `GraphContent` (create `VoiceCaptureViewModel`) +- No schema changes (journal insertion uses existing `JournalService`) +- No new dependencies beyond what's needed for Ktor (already present for `ClaudeTopicEnricher`) + +**Adding a new STT provider (third-party):** +- Implement `SpeechToTextProvider` (single method) +- Pass instance to `StelekitApp` at assembly time +- Zero changes to core code + +**Adding a new LLM provider:** +- Same as above for `LlmProvider` + +--- + +## Operational Concerns + +- **API cost**: Whisper API is ~$0.006/min of audio [TRAINING_ONLY — verify current pricing]. + A 1-minute voice note costs less than a cent. Not a concern for personal use. +- **Latency**: Typical Whisper API round-trip for 60s audio: 3–8s. Typical Claude Haiku + formatting call: 1–3s. Total pipeline: 5–12s from tap-to-stop to journal insert. +- **Temp file cleanup**: The audio temp file should be deleted after `transcribe()` returns, + regardless of success or failure. `VoiceCaptureViewModel` owns cleanup, not the platform recorder. +- **Background/foreground transitions**: If the app is backgrounded during recording (Android), + `MediaRecorder` may be killed. The `ON_PAUSE` lifecycle observer in `GraphContent` should call + `audioRecorder.stop()` to gracefully end recording and avoid a dangling file. + +--- + +## Prior Art and Lessons Learned + +**TopicEnricher / ADR-002 (this codebase)** +The core lesson: push async, side-effectful providers to the ViewModel layer via constructor +injection with NoOp defaults. Do not inject into domain services. This pattern scales directly +to the voice pipeline — confirmed by reading `ImportViewModel.kt`, `TopicEnricher.kt`, and the ADR. + +**`UrlFetcher` (this codebase)** +The `sealed class FetchResult` pattern shows how to model multi-mode failure without exceptions. +`TranscriptResult` and `LlmResult` follow the same shape. + +**`PlatformFileSystem` (this codebase)** +Demonstrates the `expect class` approach for a large platform abstraction. Audio capture does +**not** need an `expect class` — a plain `interface` in `commonMain` + actual implementations +registered at assembly time is sufficient and less rigid. `PlatformFileSystem` uses `expect class` +because it must be passed to many constructors and Compose `remember {}` blocks that need a typed +reference. `AudioRecorder` is only used in `VoiceCaptureViewModel`, so a plain interface suffices. + +**Whisper Memos / Audiopen (comparable products)** +[TRAINING_ONLY — no live web search available] These apps use a "record → upload → display" +flow without streaming display. Their UX confirms the batch model is sufficient for user +satisfaction: users accept a 5–10s processing delay after tapping stop. + +**Android SpeechRecognizer vs Whisper API** +Android `SpeechRecognizer` is free and on-device but has a 60-second hard limit and requires an +active internet connection on most devices (unless using on-device mode, which is limited). +[TRAINING_ONLY — verify Android 13+ on-device SpeechRecognizer availability] For v1, +`WhisperSpeechToTextProvider` as the primary with `AndroidSpeechToTextProvider` as an opt-in +alternative covers both use cases cleanly. + +--- + +## Open Questions + +1. **Should `AudioRecorder` be an `expect interface` or a plain `interface` in `commonMain`?** + If it's `expect interface`, KMP requires actual declarations in all targets including `jvmMain` + (desktop). Since audio capture is mobile-only per requirements, a plain `interface` in + `commonMain/platform/` with actual classes registered in mobile entry points avoids forcing + a stub into `jvmMain`. + +2. **Is Ktor already in `commonMain` dependencies?** + `ClaudeTopicEnricher` uses Ktor, so it must be. Confirm that `ktor-client-core`, + `ktor-client-content-negotiation`, and `ktor-serialization-kotlinx-json` are in + `kmp/build.gradle.kts`. If so, no new network dependency is needed for + `WhisperSpeechToTextProvider` or `ClaudeLlmProvider`. + +3. **Audio format for Whisper API**: Does the Whisper API accept `.m4a` / AAC directly from + Android `MediaRecorder` and iOS `AVAudioRecorder`? Or is conversion to `.wav` / `.mp3` needed? + +4. **`StelekitApp` parameter threading**: `sttProvider` and `llmProvider` follow the same path + as `urlFetcher` (injected in `StelekitApp`, threaded to `GraphContent`, then to + `VoiceCaptureViewModel`). Should they be bundled into a `VoicePipelineConfig` data class to + avoid growing `StelekitApp`'s parameter list further? + +5. **iOS bottom bar**: `PlatformBottomBar` on iOS (if it exists) also needs a mic button. + Confirm whether there is an `iosMain` actual for `PlatformBottomBar` or if it shares the + Android composable. + +6. **On-device STT/LLM availability gating**: Android AICore / Gemini Nano requires API 31+ + and specific device hardware. How should the ViewModel handle a provider that reports + "unavailable" at runtime? A capability check method (`suspend fun isAvailable(): Boolean`) + on the provider interface would allow graceful fallback — but adds interface surface area. + +--- + +## Recommendation + +Adopt the following architecture: + +**Layer 1 — Audio Capture (platform-specific)** +Plain interface `AudioRecorder` in `commonMain/platform/`. Android actual: `AndroidAudioRecorder` +using `MediaRecorder` (outputs `.m4a` temp file). iOS actual: `IosAudioRecorder` using +`AVAudioRecorder`. Result type: `PlatformAudioFile(path: String)` value class. + +**Layer 2 — STT Provider (commonMain interface, multiple impls)** +`suspend fun interface SpeechToTextProvider { suspend fun transcribe(audio: PlatformAudioFile): TranscriptResult }` +Default: `NoOpSpeechToTextProvider`. V1 built-in: `WhisperSpeechToTextProvider` (Ktor). +Optional: `AndroidSpeechToTextProvider` (platform SpeechRecognizer). + +**Layer 3 — LLM Provider (commonMain interface, multiple impls)** +`suspend fun interface LlmProvider { suspend fun format(transcript: String, systemPrompt: String): LlmResult }` +Default: `NoOpLlmProvider`. V1 built-in: `ClaudeLlmProvider` (mirrors `ClaudeTopicEnricher`). + +**Layer 4 — Pipeline Orchestration (VoiceCaptureViewModel, commonMain)** +Sequential pipeline in a single coroutine `Job`. Exposes `StateFlow` and +two commands: `startRecording()` / `stopRecording()`. Calls `JournalService` directly for +journal insertion (no new abstraction needed — `JournalService` already exists and handles +today-page creation). Cleans up temp audio file after transcription regardless of success/failure. + +**Layer 5 — UI (PlatformBottomBar + VoiceCaptureButton composable)** +`VoiceCaptureButton` composable in `commonMain/ui/components/`. Observes `VoiceCaptureState` +and renders the appropriate icon/animation. Added to `PlatformBottomBar.android.kt` (and iOS +equivalent). `VoiceCaptureViewModel` created in `GraphContent` alongside existing ViewModels. + +**Plugin registration**: Constructor injection with NoOp defaults, following the established seam +pattern. `StelekitApp` gains `sttProvider` and `llmProvider` parameters (both optional with NoOp +defaults). Android `MainActivity` wires the real providers when API keys are configured. + +This design introduces zero new framework dependencies, is immediately testable with fakes, +and extends naturally to a future plugin registry without breaking changes. + +--- + +## Pending Web Searches + +These claims are marked `[TRAINING_ONLY]` and should be verified: + +1. **KMP expect interface syntax**: `"kotlin multiplatform expect interface fun interface"` + — Confirm whether `expect fun interface` is valid or only `expect interface` is. + +2. **Whisper API accepted formats**: `"openai whisper api audio formats m4a aac supported"` + — Confirm `.m4a` / AAC is accepted without conversion. + +3. **Whisper API pricing 2026**: `"openai whisper api pricing per minute 2026"` + — Confirm current cost. + +4. **Android SpeechRecognizer on-device mode**: `"android speechrecognizer on device mode api level offline"` + — Confirm API level and device requirements for offline speech recognition. + +5. **Android AICore Gemini Nano Kotlin API**: `"android aicore gemini nano kotlin coroutines api 2025 2026"` + — Confirm availability and Kotlin API surface for on-device LLM. + +6. **AVAudioSession record and restore**: `"AVAudioSession setCategory record restore ambient swift kotlin"` + — Confirm best practice for category switching around recording on iOS. + +7. **MediaRecorder audio focus**: `"android mediarecorder audiofocus request record"` + — Confirm whether audio focus must be manually requested when using `MediaRecorder`. + +8. **Koin in KMP projects with Compose Multiplatform**: `"koin kotlin multiplatform compose 2025 viewmodel"` + — Gather context on whether Koin is the de-facto standard for KMP DI (to revisit Option B). + +--- + +## Web Search Results + +_Searches run: 2026-04-18._ + +### 1. Whisper API — m4a / AAC Accepted Format (confirmed) + +**Query**: `openai whisper API accepted audio formats m4a aac supported 2025` + +**Verdict**: CONFIRMED. The Whisper API officially accepts: **flac, mp3, mp4, mpeg, mpga, +m4a, ogg, wav, webm**. `.m4a` is explicitly listed. The architecture recommendation (§Q1) to +use `.m4a` from `MediaRecorder` (Android) and `AVAudioRecorder` (iOS) requires **no format +conversion** before upload. Max file size: 25 MB. + +**Updated claim**: §Q1 `[TRAINING_ONLY — verify Whisper API accepted formats for m4a]` — +**confirmed: m4a is accepted**. + +**Sources**: +- [Audio API FAQ — OpenAI Help Center](https://help.openai.com/en/articles/7031512-whisper-audio-api-faq) +- [OpenAI Community: m4a format issue thread](https://community.openai.com/t/wisper-api-not-recognizing-m4a-file-format/141251) + +--- + +### 2. Whisper API Pricing + +**Query**: `openai whisper API pricing per minute 2025 2026` + +**Verdict**: CONFIRMED. `whisper-1`: $0.006/min. New option `gpt-4o-mini-transcribe`: +$0.003/min (half price). The §Operational Concerns cost figure is accurate. + +**Sources**: +- [OpenAI API Pricing](https://openai.com/api/pricing/) +- [OpenAI Whisper API Pricing Apr 2026 — CostGoat](https://costgoat.com/pricing/openai-transcription) + +--- + +### 3. Android SpeechRecognizer On-Device API Level + +**Query**: `android SpeechRecognizer createOnDeviceSpeechRecognizer API 33 offline 2024` + +**Verdict**: CONFIRMED WITH CORRECTION. `createOnDeviceSpeechRecognizer()` was introduced at +**API 31** (Android 12), not API 33. It is available on API 33 but the minimum level is 31. +SteleKit's minSdk 24 requires a runtime API check before calling the factory method. + +**Updated claim**: The `AndroidSpeechToTextProvider` in §Q2 should gate on +`Build.VERSION.SDK_INT >= 31`, not 33. + +**Sources**: +- [SpeechRecognizer — Android Developers](https://developer.android.com/reference/android/speech/SpeechRecognizer) +- [createOnDeviceSpeechRecognizer — Microsoft Learn (Android bindings)](https://learn.microsoft.com/en-us/dotnet/api/android.speech.speechrecognizer.createondevicespeechrecognizer?view=net-android-34.0) + +--- + +### 4. Android AICore / ML Kit GenAI — Now a Public STT API + +**Query**: `Android AICore public ASR speech recognition API 2025` + +**Verdict**: MAJOR UPDATE. Google shipped **ML Kit GenAI Speech Recognition API** in 2025, +backed by Gemini Nano via AICore. This is a stable public API — the `[TRAINING_ONLY]` note in +§Q3 that "Android AICore STT is not a stable public SDK" is now outdated. + +Key constraints for architecture: +- Requires API 31+; Advanced (Gemini Nano) mode requires Pixel 9/10 and similar high-end devices. +- **Blocked in background** (`ErrorCode.BACKGROUND_USE_BLOCKED`): cannot be called from a + foreground service or background context. Only works when app is top foreground activity. +- Streaming (partial + final) results — not batch. + +**Architectural impact**: `AndroidOnDeviceLlmProvider` described in §Q3 is now also applicable +as an `AndroidSpeechToTextProvider`. The foreground-only constraint is compatible with +Phase 1 (in-app recording). A capability check via `checkFeatureStatus()` is required before +surfacing the option in settings. + +**Updated claim**: §Q3 `[TRAINING_ONLY — verify Android AICore/ML Kit Gemini Nano Kotlin API]` +— **confirmed as public API**. Foreground-only restriction is the key architectural constraint. + +**Sources**: +- [GenAI Speech Recognition API — Google for Developers](https://developers.google.com/ml-kit/genai/speech-recognition/android) +- [ML Kit GenAI APIs — Android Developers](https://developer.android.com/ai/gemini-nano/ml-kit-genai) +- [SpeechRecognizer (ML Kit) — Google Developers](https://developers.google.com/android/reference/kotlin/com/google/mlkit/genai/speechrecognition/SpeechRecognizer) + +--- + +### 5. Apple FoundationModels — New iOS LLM Backend + +**Query**: `Apple Intelligence FoundationModels framework public API third party iOS 18 2025` + +**Verdict**: MAJOR UPDATE. Apple released `FoundationModels` at WWDC 2025. Third-party +developers can now call Apple's on-device LLM directly for text generation, structured output, +and tool calling. This was a `[TRAINING_ONLY]` unknown; it is now confirmed. + +**Architectural impact on §Q3**: A fourth `LlmProvider` implementation should be planned: +`AppleIntelligenceLlmProvider` in `iosMain`, backed by `FoundationModels`. It provides: +- Offline formatting (no network call) +- No API cost +- Hardware gate: iPhone 15 Pro+, Apple Silicon, iOS 18.1+ +- Text-in / text-out (not STT) + +This is compatible with the `LlmProvider` interface (`suspend fun format(transcript, systemPrompt): LlmResult`). The capability check maps to `isAvailable(): Boolean` on the provider. + +**Updated claim**: §Q3 `[TRAINING_ONLY — verify Android AICore/ML Kit Gemini Nano Kotlin API]` +and the note on Apple Intelligence — `FoundationModels` is now a real, accessible framework. + +**Sources**: +- [Foundation Models — Apple Developer Documentation](https://developer.apple.com/documentation/FoundationModels) +- [Apple Announces Foundation Models Framework — MacRumors](https://www.macrumors.com/2025/06/09/foundation-models-framework/) +- [WWDC 2025 Session 301 — Deep dive into Foundation Models](https://developer.apple.com/videos/play/wwdc2025/301/) + +--- + +### 6. openai-kotlin v4.x — Ktor 3.x Compatible + +**Query**: `openai-kotlin Ktor 3.x compatibility version 2025 2026` + +**Verdict**: CONFIRMED. `openai-kotlin` v4.1.0 supports Ktor 3.x. The version-conflict risk +that would have blocked adoption (noted under §Migration and Adoption Cost) is resolved. The +library can now be evaluated without Ktor shading concerns. The raw Ktor recommendation still +stands for simplicity, but adopting `openai-kotlin` 4.x is no longer risky from a version +conflict standpoint. + +**Sources**: +- [Issue #411: ktor 3.x — aallam/openai-kotlin](https://github.com/aallam/openai-kotlin/issues/411) +- [Releases — aallam/openai-kotlin](https://github.com/aallam/openai-kotlin/releases) diff --git a/project_plans/mobile-voice-mode/research/features.md b/project_plans/mobile-voice-mode/research/features.md new file mode 100644 index 000000000..f9b44b24e --- /dev/null +++ b/project_plans/mobile-voice-mode/research/features.md @@ -0,0 +1,359 @@ +# Findings: Features + +## Summary + +Voice-to-notes is a mature category with several well-executed products. The dominant pattern is: one-tap record → stream or buffer audio → transcribe via Whisper (cloud or on-device) → LLM rewrite into a chosen structure → append to a target location. The main differentiators are (a) capture friction (lock screen widget vs. in-app FAB), (b) whether the LLM stage happens before or after the user sees raw text, and (c) how well the output format aligns with the target note system. No existing product natively produces Logseq outliner syntax with `[[wikilinks]]`, but the pipeline pattern is clear: transcription → LLM formatting prompt → write to today's daily note file. + +--- + +## Options Surveyed + +### AudioPen (audiopen.ai) + +**Capture UX**: Single large record button on the home screen. Tap to start, tap to stop. No lock screen widget or persistent recording indicator in the free tier. Mobile apps on iOS and Android. Free tier caps recordings at 3 minutes; Prime tier allows up to 15 minutes. + +**Long-ramble handling**: Post-process only — the full recording is sent to the server after the user stops. No streaming transcription. AudioPen's core value proposition is its "Synthesis Engine": it does not produce a verbatim transcript but actively rewrites the audio, removing filler words ("uh", "um"), redundant repetition, and structural rambling. The output reads like intentional writing, not dictation. + +**Formatting**: Ships with a style library. Predefined styles include "Franklin: Bulleted list, short sentences", casual memo, formal email, Twitter thread, action items, technical doc, and custom styles. Users on the Prime plan can define their own style with examples. No native `[[wikilink]]` or outliner syntax support. Format selection is per-recording, not automatic. + +**Latency**: Entirely post-process. Users report results appear in "seconds" after stopping, but this is network-dependent. No official latency SLA published. + +**Eyes-free safety**: Minimal. The app requires navigation to the home screen. No lock screen widget on Android; limited iOS widget support. + +**Processing state communication**: Simple spinner/loading indicator after stopping. No waveform during recording in the free tier. + +**Lessons**: AudioPen proves that users prefer a synthesized, rewritten output over a raw transcript — the "what I meant to say" model. The style-library approach is highly valued. However, its capture UX requires full phone interaction, and its output formats are not aimed at outliner/PKM tools. + +--- + +### Whisper Memos (whispermemos.com) + +**Capture UX**: Records on iPhone or Apple Watch. On Apple Watch, a complication (watch face button) allows one-tap recording without touching the phone. Delivers transcripts by email or saves to iCloud Drive as dated text files. Supports Cohere Transcribe and OpenAI Whisper as transcription engines. + +**Long-ramble handling**: Buffers the full recording, then sends to the cloud transcription engine post-stop. No chunked streaming. iCloud Integration saves transcripts as organized text files by date, accessible from Files app or piped to Obsidian via automation. + +**Formatting**: Raw transcript output by default. Users must apply their own LLM post-processing (e.g., running Claude on the saved file) to get structured notes. Some users describe workflows where Whisper Memos captures, then a Claude shortcut formats the result. + +**Latency**: Reported as "fast" but no benchmark available. Speed depends on Whisper cloud endpoint response time. + +**Eyes-free safety**: Apple Watch complication is the strongest eyes-free pattern in this survey. One-tap on wrist, no phone needed. Critically important for driving and walking use cases. + +**Processing state communication**: Watch shows a recording indicator. Phone app shows transcript arriving asynchronously (often by notification or email), so there is a decoupled delivery model — capture and result are temporally separated. + +**Lessons**: The Apple Watch complication is the gold standard for minimum-friction capture. The decoupled delivery model (capture now, formatted result later) reduces latency anxiety because users know the result is coming. Two-step pipelines (record → LLM format separately) give users control but add workflow friction. + +--- + +### Notion AI voice capture + +**Capture UX**: Notion introduced native voice input on mobile, activated inside the app. The `/meet` block triggers meeting transcription from the mic. Third-party integrations (Wispr Flow, Speechify, Fast Dictate) operate at the OS dictation layer, injecting text into any focused text field. + +**Long-ramble handling**: The native `/meet` block transcribes in real-time, streaming text into the note block. Third-party tools also operate in streaming mode, showing words as they are spoken. + +**Formatting**: Notion AI does not auto-structure voice input into bullets or headers. The raw dictation stream lands in a text block. Users manually invoke AI commands to restructure. Speechify claims to remove filler words in real-time before injecting text. + +**Latency**: Streaming dictation has near-zero perceived latency (words appear as spoken). Post-processing AI formatting adds seconds after the user stops. + +**Eyes-free safety**: Not designed for eyes-free use. Requires active cursor placement in the Notion editor. + +**Processing state communication**: Real-time text stream is the processing indicator — words appearing live tell the user the system is working. + +**Lessons**: Streaming dictation (words appear as you speak) provides excellent user feedback and perceived low latency. However, unformatted raw text output requires a second LLM pass to produce structured notes. + +--- + +### Otter.ai + +**Capture UX**: Large floating record button in the mobile app. Designed primarily for meeting transcription (auto-joins Zoom, Google Meet). Manual voice memo recording supported. No lock screen widget. + +**Long-ramble handling**: Real-time streaming transcription — words appear on-screen as you speak, with speaker diarization. Long recordings are handled gracefully (designed for multi-hour meetings). The acoustic model runs a denoising pipeline on the raw waveform before diarization. + +**Formatting**: After recording stops, Otter AI generates a summary with action items and highlights. Output is paragraph summaries and bullet-point action items, not outliner syntax. No `[[wikilink]]` support. + +**Latency**: Live transcription during recording, summary generated within seconds of stopping. + +**Eyes-free safety**: Not designed for solo voice capture; assumes the user has a phone available for meeting context. + +**Processing state communication**: Live waveform visualization during recording (users see audio activity). Real-time text stream. Post-processing shows a progress indicator with status text ("Generating summary..."). + +**Lessons**: The live waveform + streaming text combination is the most reassuring processing state UX in this survey. Users know the system is hearing them. The two-phase output model (live transcript → post-stop AI summary) separates raw capture from formatting. + +--- + +### Apple Notes + Voice Memos (iOS 18) + +**Capture UX**: In iOS 18, Notes app has a native audio recording button inside a note. Voice Memos app is a standalone recorder. Lock screen is not directly accessible for Notes recording, but Voice Memos can be accessed via Control Center. + +**Long-ramble handling**: Transcription runs on-device (iPhone 12+, English and ~10 languages) during or immediately after recording. On-device processing caps at roughly 30 minutes; longer recordings may fail or process slowly. + +**Formatting**: Verbatim transcript only — no LLM reformatting. Apple Intelligence (iOS 18.1+) can summarize notes but does not produce outliner or bullet syntax automatically. + +**Latency**: Near-instant for short recordings (transcript appears within seconds of stopping). Longer recordings take proportionally longer. User reports confirm a brief post-processing pause before the transcript appears in Notes. + +**Eyes-free safety**: Voice Memos via Siri ("Hey Siri, create a voice memo") is the only eyes-free path on Apple hardware. Notes recording is not Siri-accessible. + +**Processing state communication**: Simple "Transcribing..." label below the audio waveform in Notes. Voice Memos shows the recording duration counter. + +**Lessons**: On-device transcription eliminates the server round-trip and privacy concerns. However, on-device models lack the context-awareness to produce structured outliner output — they do verbatim transcription only. A hybrid approach (on-device transcription → cloud LLM formatting) combines the best of both. + +--- + +### Reflect (reflect.app) + +**Capture UX**: iOS lock screen widget that triggers a voice recording directly, with a Live Activity indicator in the Dynamic Island and lock screen showing recording status and a "Stop" button. On-device recording with Whisper transcription. Transcript automatically appends to today's daily note. + +**Long-ramble handling**: Full-recording post-processing via Whisper after the user stops. No real-time streaming shown during capture. + +**Formatting**: Raw Whisper transcript by default, appended to the daily note. Reflect's AI can optionally rewrite the note, but this is a separate explicit action, not automatic. + +**Latency**: Transcript appears in the daily note "shortly after" stopping. No published latency numbers. + +**Eyes-free safety**: Lock screen widget is the closest to eyes-free in this survey among general note-taking apps. The user taps once on the lock screen to start and once to stop — minimal visual attention required. + +**Processing state communication**: Live Activity on lock screen and Dynamic Island shows recording indicator and elapsed time. Post-processing shows a brief "Transcribing..." state. + +**Lessons**: The lock screen widget + auto-append to daily note is the most directly analogous pattern to what SteleKit is building. The Daily Note auto-append removes the "where does this go?" decision entirely. This is the closest prior art. + +--- + +### NotelyVoice (Open Source, Compose Multiplatform) + +**Capture UX**: Standard FAB (floating action button) in-app. Built with Compose Multiplatform for Android and iOS — directly relevant to SteleKit's KMP architecture. + +**Long-ramble handling**: Memory-optimized audio processing for large files: streaming WAV decoding + overlapping chunk transcription (splits audio into chunks with overlap, transcribes each, stitches results). Handles Out of Memory errors gracefully. + +**Formatting**: Verbatim Whisper output. No LLM reformat stage. + +**Latency**: On-device Whisper — latency proportional to recording length and device capability. + +**Eyes-free safety**: Standard in-app FAB only — not eyes-free. + +**Processing state communication**: Basic progress indicator during transcription. + +**Lessons**: This codebase demonstrates that chunked, overlap-based transcription of long recordings is solvable in Compose Multiplatform. The overlap technique prevents sentence truncation at chunk boundaries. Directly applicable to SteleKit's KMP implementation. + +--- + +## Trade-off Matrix + +| Axis | AudioPen | Whisper Memos | Otter.ai | Apple Notes | Reflect | NotelyVoice (OSS) | +|---|---|---|---|---|---|---| +| Capture friction | Medium (in-app only) | Low (Watch complication) | Medium (in-app FAB) | High (navigate to Notes) | Low (lock screen widget) | Medium (in-app FAB) | +| Formatting quality | High (LLM synthesis) | None (raw transcript) | High (meeting AI summary) | None (verbatim) | Low (Whisper verbatim) | None (verbatim) | +| Latency to formatted output | 2–5s post-stop | Async (email/file) | Real-time + ~3s summary | ~1s on-device | ~2–3s | ~5–15s on-device | +| Outliner/wikilink support | None | None | None | None | None | None | +| Eyes-free safety | Poor | Excellent (Watch) | Poor | Moderate (Siri) | Good (lock screen) | Poor | +| Privacy (on-device option) | No (cloud only) | Yes (Whisper engine) | No (cloud only) | Yes (Apple on-device) | Partial (Whisper) | Yes (fully on-device) | +| KMP / Android-first relevance | N/A | iOS-only | Cross-platform app | iOS-only | iOS-only | Direct (CMP) | +| Auto-append to daily note | No | Partial (iCloud file) | No | No | Yes | No | + +--- + +## Risk and Failure Modes + +**Transcription accuracy on long rambles**: Whisper and similar models degrade on audio longer than ~15–30 minutes without chunking. Chunk boundaries can split sentences. Mitigation: overlapping chunks with 5–10 second overlap windows, then deduplicate overlapping text. + +**LLM formatting hallucinations**: When the LLM reformats the transcript, it can invent `[[links]]` to pages that do not exist in the user's graph, or silently drop content it judges as redundant. Mitigation: always append the raw transcript below the formatted output (collapsible), and constrain the LLM to only use wikilinks for terms explicitly mentioned in the transcript. + +**Microphone permission and background audio**: Android and iOS have different rules for background mic access. Android 14+ restricts background microphone use without a foreground service notification. iOS requires the app to be in the foreground or have a Live Activity. Mitigation: use a foreground service on Android with a persistent notification, and a Live Activity on iOS. + +**Network dependency for LLM formatting**: If the device is offline, the LLM reformat step fails. Mitigation: always save the raw transcript locally first, queue the LLM formatting job, apply it when connectivity returns. + +**User confusion about "what was captured"**: If the formatted output differs significantly from what was said, users distrust the tool. Mitigation: show a toggle between "Formatted" and "Original transcript" views on the result card. + +**Battery drain during long sessions**: Continuous microphone use drains battery faster than normal app use. Mitigation: display a battery warning if the session exceeds a configurable threshold (e.g., 10 minutes). + +--- + +## Migration and Adoption Cost + +The feature is additive — existing SteleKit users gain a new capture mode without workflow disruption. The primary adoption cost is: + +1. Users must grant microphone permission (one-time friction). +2. Users must understand the difference between the formatted output and the raw transcript. +3. Power users with existing Logseq voice workflows (via Shortcuts or automation) may have bespoke pipelines that conflict with SteleKit's append behavior. + +No migration of existing data is required. The feature appends to the existing daily note file in the standard Logseq format, so it is non-destructive. + +--- + +## Operational Concerns + +**LLM API cost**: If the formatting step calls a cloud LLM (Gemini Flash, Claude Haiku, or similar), cost per note is low (~$0.001–$0.005 per transcription depending on length), but aggregates at scale. Consider whether this is user-pays (API key configuration) or SteleKit-subsidized. + +**Whisper transcription cost**: OpenAI Whisper API charges ~$0.006/minute. A 5-minute recording costs ~$0.03. On-device Whisper (via whisper.cpp or Android ONNX port) eliminates this cost but requires model download (~75MB for base, ~1.4GB for large-v3). + +**Model download size**: On-device Whisper models are large. The base model (75MB, ~3% WER) may be acceptable for initial release; the small model (244MB) offers better accuracy. Users on metered connections need explicit consent before downloading. + +**Server-side prompt injection risk**: User voice content passes through the LLM formatting prompt. If users dictate adversarial content, the LLM could be manipulated. Mitigation: system prompt is not user-editable; transcript is inserted as a clearly delimited user message, not as instructions. + +--- + +## Prior Art and Lessons Learned + +1. **Reflect's lock screen widget + daily note append** is the closest prior art to the planned SteleKit feature. The pattern works: users adopt it because it removes the "where does this go?" decision. + +2. **AudioPen's synthesis model** (rewrite, don't transcribe) dramatically improves output quality but risks losing nuance. For Logseq outliner output, a hybrid approach works best: LLM identifies the structural intent (main topics → top-level bullets, sub-points → indented bullets, mentioned entities → `[[wikilinks]]`) rather than purely summarizing. + +3. **Otter.ai's two-phase display** (live transcript during recording + formatted summary after) is the best UX for communicating system state. Users are never left wondering if the system heard them. + +4. **Whisper Memos + Apple Watch** demonstrates that the wrist is the ideal capture surface for truly eyes-free use. SteleKit's KMP stack cannot target watchOS (no Kotlin support), but the pattern is instructive: minimize the UI surface area of the capture step. + +5. **NotelyVoice's CMP implementation** of chunked Whisper transcription is directly reusable reference code for SteleKit's KMP architecture. + +6. **No tool surveyed produces Logseq outliner syntax** (`- top bullet\n - sub-bullet\n [[wikilink]]`). This is a genuine differentiation opportunity. The LLM prompt engineering to produce well-formed Logseq markdown from a transcript is achievable with a carefully designed system prompt and output validation. + +--- + +## Open Questions + +1. **On-device vs. cloud transcription**: Should SteleKit ship with on-device Whisper (privacy-first, no cost, larger APK, slower on old devices) or cloud Whisper (faster, smaller APK, costs money or requires user API key)? Hybrid (on-device for transcription, cloud for LLM formatting) is likely the right default. + +2. **Which LLM for formatting?**: Gemini Flash (free tier available), Claude Haiku (fast, cheap), or a local model (Phi-3 Mini)? The formatting prompt is short-context and low-complexity — a small model is sufficient. + +3. **Lock screen / foreground service**: What is the minimum viable eyes-free capture UX on Android without watchOS support? A persistent foreground service notification with a "tap to start/stop" action button is the most feasible equivalent. + +4. **Wikilink injection strategy**: Should the LLM be given a list of known page names in the user's graph to constrain `[[link]]` generation? This would require passing graph index data to the LLM call, increasing context size and cost. + +5. **Formatting style selection**: Should users be able to choose between "bullet points", "paragraph", "action items" (AudioPen-style style library) or should SteleKit always produce Logseq outliner format? + +6. **Append vs. new block**: Should captured notes append to the bottom of today's daily note as a timestamped block, or be inserted at the cursor position? Appending is simpler and safer; cursor insertion requires the app to be open. + +7. **Maximum recording length**: Should there be a cap (e.g., 15 minutes) or should the app support unlimited recording with chunked processing? + +--- + +## Recommendation + +Build the feature in three phases, following the Reflect + AudioPen combined pattern: + +**Phase 1 — Baseline capture**: In-app FAB in the mobile journal view. Tap to record, tap to stop. Post-stop: Whisper cloud transcription → raw transcript appended to today's daily note as a dated block. No LLM formatting yet. Ships fast, validates the pipeline. + +**Phase 2 — LLM formatting**: Add an LLM formatting step after transcription. System prompt instructs the model to produce Logseq outliner markdown: top-level `-` bullets for main topics, indented bullets for sub-points, `[[entity]]` for proper nouns and topics the user explicitly named. Show a "Formatted / Raw" toggle on the result card. + +**Phase 3 — Capture surface expansion**: Add a foreground service notification on Android with a start/stop action button (the closest to Reflect's lock screen widget without watchOS). On iOS, add a lock screen widget via iOS Widget APIs if SteleKit ships an iOS target. + +The two-phase display model from Otter.ai (live waveform or streaming transcript during recording + formatted result after stopping) should be adopted from day one to communicate system state clearly. + +--- + +## Pending Web Searches + +The following searches should be run to fill gaps in this research: + +1. `AudioPen iOS lock screen widget 2025` — confirm whether AudioPen added a widget after the initial survey date +2. `whisper.cpp Android ONNX Compose Multiplatform integration 2025` — find current state of on-device Whisper for KMP/Android +3. `Logseq voice capture plugin community 2025` — check whether the Logseq community has built voice capture plugins with wikilink extraction +4. `Gemini Flash audio transcription API pricing 2025` — Gemini can transcribe audio directly; may eliminate the Whisper + LLM two-step +5. `Android foreground service microphone lock screen button API 34` — confirm exact Android 14 foreground service type for microphone required +6. `Reflect.app lock screen widget iOS implementation details` — look for any technical writeup of how Reflect built the Live Activity widget +7. `KMP Kotlin Multiplatform speech recognition Android iOS unified API 2025` — check if there is a KMP wrapper for platform speech recognition APIs + +--- + +## Web Search Results + +_Searches run: 2026-04-18._ + +### 1. Android Foreground Service + Microphone (API 34) + +**Query**: `android foregroundServiceType microphone requirement API level manifest 2024` + +**Verdict**: CONFIRMED AND CLARIFIED. +- `android:foregroundServiceType="microphone"` in the manifest is required for any foreground + service accessing the mic. The requirement to declare this type takes effect when + **targetSdkVersion ≥ 30** (Android 11) — omitting it silently denies mic access to the + service on those targets. +- For apps **targeting API 34+** (Android 14): additionally requires the + `android.permission.FOREGROUND_SERVICE_MICROPHONE` permission in the manifest, AND one of + `CAPTURE_AUDIO_OUTPUT` or `RECORD_AUDIO` at runtime. +- **Background microphone restriction (Android 12+)**: Apps cannot start a new microphone + access session while in the background. The foreground service must already be running in + the foreground before the app is backgrounded — you cannot lazily start it on screen lock. + +This directly validates the pitfalls.md recommendation: start the foreground service while the +app is still visible, then allow the screen to lock. + +**Sources**: +- [Foreground service types are required — Android 14 — Android Developers](https://developer.android.com/about/versions/14/changes/fgs-types-required) +- [Foreground service types — Android Developers](https://developer.android.com/develop/background-work/services/fgs/service-types) +- [Guide to Foreground Services on Android 14 — Medium](https://medium.com/@domen.lanisnik/guide-to-foreground-services-on-android-9d0127dc8f9a) + +--- + +### 2. ML Kit GenAI On-Device STT — Device Availability + +**Query**: `ML Kit GenAI speech recognition android availability devices 2025` + +**Verdict**: NEW INFORMATION. Google launched a public **ML Kit GenAI Speech Recognition API** +(backed by Gemini Nano / AICore) in 2025. This changes the §2d assessment significantly: + +- **Basic mode** (standard quality): API 31+ on most Android devices. +- **Advanced mode** (Gemini Nano quality): Pixel 9/10 series, Samsung Galaxy S25/S26, + Honor Magic 7/8 Pro, OPPO Find N5/X9, Galaxy Z Fold7, and expanding. +- The API provides **streaming transcription** (partial + final results). +- **Hard constraint**: inference is blocked when app is not the top foreground app + (`ErrorCode.BACKGROUND_USE_BLOCKED`). Cannot be used from a foreground service. + +For SteleKit Phase 1 (foreground-only recording), the ML Kit GenAI STT is a viable free +on-device alternative to Whisper API on supported devices. It should be added to the §2d +column as an optional `AndroidSpeechToTextProvider` backend. + +**Sources**: +- [GenAI Speech Recognition API — Google for Developers](https://developers.google.com/ml-kit/genai/speech-recognition/android) +- [Android Developers Blog: ML Kit GenAI APIs](https://android-developers.googleblog.com/2025/05/on-device-gen-ai-apis-ml-kit-gemini-nano.html) + +--- + +### 3. Apple FoundationModels — Available as LLM Formatting Backend + +**Query**: `Apple Intelligence FoundationModels framework public API third party iOS 18 2025` + +**Verdict**: MAJOR UPDATE. Apple shipped `FoundationModels` at WWDC 2025. Third-party apps can +now call Apple's on-device LLM for text generation, structured output, and tool calling. + +**Direct relevance to features.md**: The "hybrid approach (on-device transcription → cloud LLM +formatting)" described in §Apple Notes is now feasible on iOS without any cloud call: +- STT: `SFSpeechRecognizer` (on-device, iOS 13+) or WhisperKit +- Formatting: `FoundationModels` (on-device, no cost, no network) + +This is a complete offline path for the transcript → Logseq outliner formatting step on +iPhone 15 Pro+ / iOS 18.1+. Worth surfacing in Phase 2 planning as a premium offline tier. + +**Sources**: +- [Foundation Models — Apple Developer Documentation](https://developer.apple.com/documentation/FoundationModels) +- [Apple's Foundation Models framework — Apple Newsroom](https://www.apple.com/newsroom/2025/09/apples-foundation-models-framework-unlocks-new-intelligent-app-experiences/) +- [WWDC 2025 Session 301 — Deep dive into Foundation Models](https://developer.apple.com/videos/play/wwdc2025/301/) + +--- + +### 4. Whisper Hallucination on Silence (confirmed) + +**Query**: `openai whisper hallucination silence "thank you" known issue short audio` + +**Verdict**: CONFIRMED. The "Thank you." hallucination on silence is a well-documented, +unresolved upstream issue (tracked in openai/whisper Discussion #1606, #679, and API community +threads). The word-count heuristic (`< 10 words → treat as empty`) described in pitfalls.md is +the community-recommended mitigation. A VAD gate before upload is the most reliable prevention. +Research (Calm-Whisper, ICML 2025) shows the issue is addressable via fine-tuning but the +standard `whisper-1` API model still hallucates on silence. + +**Sources**: +- [Hallucination on audio with no speech — openai/whisper Discussion #1606](https://github.com/openai/whisper/discussions/1606) +- [Whisper silent audio hallucination — OpenAI Community](https://community.openai.com/t/whisper-silent-audio-hallucination/1305173) +- [Calm-Whisper paper — arXiv 2025](https://arxiv.org/html/2505.12969v1) + +--- + +### 5. Whisper API Pricing (confirmed) + +**Query**: `openai whisper API pricing per minute 2025 2026` + +**Verdict**: CONFIRMED. `whisper-1` remains $0.006/min. New cheaper option: `gpt-4o-mini- +transcribe` at **$0.003/min** (half the price, comparable accuracy for most use cases). The +cost estimate for on-demand LLM formatting in §Operational Concerns ($0.001–$0.005/note) also +remains accurate. + +**Sources**: +- [OpenAI API Pricing](https://openai.com/api/pricing/) +- [OpenAI Whisper API Pricing Apr 2026 — CostGoat](https://costgoat.com/pricing/openai-transcription) diff --git a/project_plans/mobile-voice-mode/research/pitfalls.md b/project_plans/mobile-voice-mode/research/pitfalls.md new file mode 100644 index 000000000..f2d413450 --- /dev/null +++ b/project_plans/mobile-voice-mode/research/pitfalls.md @@ -0,0 +1,466 @@ +# Findings: Pitfalls — Voice Capture Mode (KMP Android/iOS) + +## Summary + +Voice capture in KMP introduces six distinct failure domains, each with platform-specific behavior that cannot be fully abstracted into commonMain. The most dangerous are: (1) silent permission denial on iOS that leaves the app recording nothing with no error, (2) Android audio focus loss that corrupts MediaRecorder state requiring a full teardown/restart, (3) on-device LLM availability gating that can fail at runtime on supported devices (model not yet downloaded), and (4) LLM hallucination of `[[links]]` that silently corrupts the knowledge graph. The recommendation is to use a "always-ask, graceful-degrade" architecture with explicit fallback tiers at every layer: permission → recording → STT → LLM → outliner formatting. + +--- + +## Options Surveyed + +### Permission Request Libraries (KMP) + +| Option | Platform support | Notes | +|---|---|---| +| Accompanist Permissions | Android only | Jetpack Compose; not shared | +| Moko Permissions | Android + iOS | KMP-native; wraps AVAudioSession + Android runtime permissions | +| Manual expect/actual | All platforms | Maximum control; most boilerplate | +| Compose Multiplatform resource APIs | Not yet available for permissions | Pending CMP roadmap | + +**Moko Permissions** is currently the most practical KMP option for requesting `RECORD_AUDIO` (Android) and microphone authorization (iOS) from shared code. [TRAINING_ONLY — verify current Moko Permissions version and CMP compatibility] + +### Audio Capture APIs + +| API | Platform | Notes | +|---|---|---| +| `AudioRecord` (raw PCM) | Android | Most control; requires manual WAV header for Whisper | +| `MediaRecorder` | Android | Easier; M4A/MP4 output; loses samples on focus loss | +| `AVAudioEngine` | iOS | Low-latency; streaming-friendly | +| `AVAudioRecorder` | iOS | File-based; simpler but less flexible | + +### STT Options + +| Option | Latency | Cost | Offline | Notes | +|---|---|---|---|---| +| OpenAI Whisper API | ~2–10s | $0.006/min | No | 25 MB / ~30 min max | +| On-device Whisper (whisper.cpp / WhisperKit) | Variable | Free | Yes | iOS: WhisperKit; Android: whisper.cpp via JNI | +| Android SpeechRecognizer | Low | Free | Partial | Requires network for cloud mode; poor accuracy | +| Apple Speech framework | Low | Free | Yes (on-device mode iOS 17+) | Limited to ~1 min per request | +| Android AICore / Gemini Nano | Low | Free | Yes | Pixel 8+ only; model must be pre-downloaded | +| Apple Intelligence (on-device LLM) | Low | Free | Yes | iOS 18.1+; US English only initially | + +### LLM Formatting Options + +| Option | Format control | Notes | +|---|---|---| +| Prompt-only (zero-shot) | Moderate | Hallucination risk on `[[links]]` | +| Prompt + few-shot examples | Good | Add 3–5 Logseq format examples in system prompt | +| Structured output (JSON mode) | Best | Parse JSON → emit Logseq markdown; eliminates format drift | +| Grammar-constrained decoding | Best | Requires on-device model with GBNF/llama.cpp grammar support | + +--- + +## Trade-off Matrix + +| Failure Domain | Permission failure UX | Audio interruption recovery | Model gate fallback | Prompt reliability | Cost risk | +|---|---|---|---|---|---| +| Accompanist-only permissions | iOS blocked entirely | N/A | N/A | N/A | None | +| Moko Permissions | Shared rationale UI possible | N/A | N/A | N/A | None | +| MediaRecorder on focus loss | N/A | Full teardown required | N/A | N/A | None | +| AudioRecord on focus loss | N/A | Can pause/resume with buffer drain | N/A | N/A | None | +| Whisper API only | N/A | N/A | None (single point of failure) | High | $0.006/min × scale | +| On-device Whisper | N/A | N/A | App-bundled; no gate | High | None | +| AICore Gemini Nano | N/A | N/A | Must detect at runtime | Moderate | None | +| Zero-shot LLM prompt | N/A | N/A | N/A | Low (hallucination) | API cost | +| Few-shot + structured output | N/A | N/A | N/A | High | API cost | + +--- + +## Risk and Failure Modes + +### 1. Microphone Permissions + +#### Android — RECORD_AUDIO + +- `RECORD_AUDIO` is a "dangerous" permission requiring runtime request on API 23+. [TRAINING_ONLY — verify] +- If the user selects "Don't ask again" after denying, `shouldShowRequestPermissionRationale()` returns `false` and your rationale dialog is suppressed. The user must manually go to Settings. +- **Mid-flow denial**: If the user revokes the permission via Settings while the app is in the background (rare but possible on Android 11+ with permission auto-reset), `AudioRecord.startRecording()` will throw `SecurityException`. This is unrecoverable without restart. +- Android 12+ introduced the microphone indicator (green dot). Users may tap it and revoke via the Privacy dashboard mid-session — a `SecurityException` in a background coroutine will crash if not caught. +- **Mitigation**: Wrap all audio capture in a `try/catch(SecurityException)`, immediately stop recording, emit a `PermissionRevoked` event to the ViewModel, and show a non-blocking snackbar with a deep link to app permissions settings. + +#### iOS — AVAudioSession + Privacy Description + +- `NSMicrophoneUsageDescription` must be in `Info.plist` or the app crashes at first permission request. [TRAINING_ONLY — verify] +- `AVAudioSession.requestRecordPermission` returns `.denied` silently — there is no system dialog the second time. The first-time dialog is shown by the OS; subsequent requests are no-ops. +- **Critical iOS pitfall**: If you call `AVAudioEngine.start()` without checking authorization first, it silently records nothing (no audio, no error). You get an empty WAV that Whisper transcribes as silence. This produces a confusing UX ("it recorded but got nothing"). +- **Background mode**: iOS terminates audio sessions not declared in `UIBackgroundModes`. If the user locks the screen mid-capture, recording stops with no callback unless the app declares `audio` in background modes. Declaring it triggers App Store review scrutiny. +- **Mitigation**: Always call `AVAudioSession.recordPermission` before starting. If `.undetermined`, request it. If `.denied`, navigate to settings. If `.granted`, configure the session with `.record` category before starting the engine. + +#### KMP — commonMain Permission Abstraction + +- **Core problem**: There is no Kotlin Multiplatform standard for requesting permissions. `Accompanist Permissions` is Jetpack Compose for Android only and cannot be called from commonMain. +- **Moko Permissions** (`dev.icerock.moko:permissions`) provides `PermissionsController` with an `expect`/`actual` pattern that works in commonMain. It exposes `Permission.RECORD_AUDIO` cross-platform. [TRAINING_ONLY — verify current API surface] +- **Pitfall with Moko**: On iOS, Moko wraps `AVAudioSession.requestRecordPermission` but you still need to configure the `AVAudioSession` category separately before recording — Moko does not do this. Forgetting the category configuration causes `AVAudioEngine` to fail with `AVAudioSessionErrorCodeCannotStartRecording`. +- **Alternative pattern**: Define a `MicrophonePermissionManager` interface in commonMain with `expect`/`actual` implementations. Android: wraps `ActivityResultLauncher`. iOS: wraps `AVAudioSession`. This avoids the Moko dependency but requires more boilerplate. + +--- + +### 2. Audio Focus / Interruption Handling (Android) + +- Android requires apps to request `AudioFocus` before recording to respect phone calls and other audio apps. [TRAINING_ONLY — verify AudioFocus API lifecycle] +- **Phone call interruption**: When a call arrives, the system dispatches `AUDIOFOCUS_LOSS_TRANSIENT`. If you do not handle this, `MediaRecorder` continues running but may capture call audio (privacy violation) or corrupt the recording. +- **`MediaRecorder` vs `AudioRecord` on interruption**: + - `MediaRecorder`: Has no pause/resume on API < 24. On interruption, you must call `stop()` and `release()`. The partial file may not be playable if the MP4 box headers are incomplete. You cannot resume; you must start a new file. + - `AudioRecord`: Reads raw PCM into a buffer. On interruption you can stop reading, drain the buffer, and resume. The data stream remains coherent. This is the better choice for reliability. +- **`AUDIOFOCUS_LOSS` (permanent)**: Another app took focus permanently (e.g., user opened Spotify). You must stop recording and release resources. Do not silently continue. +- **Mitigation**: + 1. Request `AudioFocus` with `AUDIOFOCUS_GAIN_TRANSIENT` before starting. + 2. Register an `OnAudioFocusChangeListener`. + 3. On `AUDIOFOCUS_LOSS_TRANSIENT`: pause reading; show "Recording paused — call in progress" in the UI. + 4. On `AUDIOFOCUS_LOSS`: stop recording; offer to discard or keep partial. + 5. On `AUDIOFOCUS_GAIN`: resume automatically only if the user had not explicitly stopped. + 6. Abandon audio focus in `onStop()` / after recording ends. + +--- + +### 3. Background Recording Restrictions + +#### Android + +- **Doze mode** (API 23+): If the device is idle and the screen is off, Doze restricts background work including wakelock-protected threads. An ongoing `AudioRecord` loop in a `Service` with a `FOREGROUND_SERVICE` notification is the correct approach — Doze does not kill foreground services. [TRAINING_ONLY — verify current Doze exemptions] +- **Foreground Service requirement**: Android 14+ requires declaring `foregroundServiceType="microphone"` in the manifest for any foreground service that accesses the microphone. Omitting this causes `ForegroundServiceStartNotAllowedException` on Android 14+. [TRAINING_ONLY — verify exact API level] +- **Background microphone restriction (Android 12+)**: Apps cannot start new microphone access while in the background. The service must already be running in the foreground before the app backgrounds. This means you cannot lazily start recording when the user locks the screen — you must transition to a foreground service before the app leaves the foreground. +- **Driving scenario (screen lock)**: For "hands-free dictation while driving," you must start the foreground service while the app is still visible, then let the screen lock. The foreground notification acts as the user's escape hatch. + +#### iOS + +- Without a background mode, `AVAudioEngine` is suspended when the app backgrounds. There is no graceful callback — the engine simply stops producing audio samples. +- **`audio` background mode**: Declaring this in `Info.plist` / `UIBackgroundModes` allows continuous recording but App Store review will scrutinize whether it's genuinely needed. Voice memos and podcast apps use this legitimately. +- **Alternative (no background mode)**: Record only while the app is in the foreground. Show a "Recording will pause if you leave the app" warning. This is the safest App Store path. +- **VoIP background mode**: Do not use this to "trick" iOS into keeping the mic open. App Store reviewers will reject apps that abuse VoIP entitlements. + +--- + +### 4. On-Device LLM Availability Gating + +#### Android AICore / Gemini Nano + +- **Device gate**: Gemini Nano via AICore is limited to Pixel 8 and newer (as of early 2025). Samsung Galaxy S24 series may also have access via a separate integration path. The vast majority of Android users will not have this available. [TRAINING_ONLY — verify current device list] +- **Model download gate**: Even on a supported device, Gemini Nano may not be downloaded. The model (several GB) is downloaded by AICore on Wi-Fi in the background. An app can check `GenerativeModel.isAvailable()` (or equivalent AICore readiness API) but cannot force the download. [TRAINING_ONLY — verify exact API] +- **Runtime availability**: `DownloadConfig` / `AvailabilityListener` patterns exist but are asynchronous. Do not block app startup on this check. +- **Failure mode**: If you call the AICore API on an unsupported device or before the model is ready, you get a runtime exception (`IllegalStateException` or similar). This is not an Android-standard `ActivityNotFoundException` — it requires specific error handling. +- **Mitigation**: Implement a `LlmBackend` interface: `OnDeviceGemini`, `WhisperApiRemote`, `None`. On app start, asynchronously check AICore availability and set the preferred backend. Default to API remote if on-device is unavailable. Cache the result for the session. + +#### Apple Intelligence (iOS 18.1+) + +- Available only on iPhone 15 Pro and newer with iOS 18.1+. Not available in all regions (US, UK, Australia at launch; EU pending). [TRAINING_ONLY — verify current regional availability] +- Apple does not expose a public Swift API for "send text to Apple Intelligence for summarization/formatting" as of training cutoff. Apple Intelligence is accessible via Writing Tools (system-level text field integration), not a programmatic API. [TRAINING_ONLY — verify if any API was opened post-iOS 18.2] +- **Practical implication**: Apple Intelligence cannot be directly invoked for LLM formatting in SteleKit. The STT → LLM path on iOS must use either a remote API (OpenAI/Claude/etc.) or an embedded on-device model (llama.cpp, WhisperKit companion). +- **WhisperKit** (Argmax): Open-source, Swift-native, runs Whisper on Core ML on-device. Well-maintained as of 2024. This is the recommended on-device STT path for iOS. [TRAINING_ONLY — verify WhisperKit production-readiness] + +--- + +### 5. LLM Prompt Fidelity for Outliner Format + +#### Logseq Format Requirements + +The target format is: +``` +- item text + - sub-item text + - another sub-item [[linked-page]] +- next top-level item +``` + +Key constraints: bullet with `- `, two-space indentation for nesting, `[[Page Name]]` for links, no trailing whitespace, no blank lines between bullets at same level (Logseq treats blank lines as new blocks). + +#### Failure Modes + +**Hallucinated `[[links]]`**: LLMs will invent `[[page names]]` that do not exist in the graph. Example: a voice note about a meeting with "Sarah" becomes `- Met with [[Sarah Johnson]]` when no such page exists. Whisper's transcript may contain "Sarah" but the LLM extrapolates a full name and wraps it in a link. +- **Risk severity**: High — silently creates dangling links that pollute the graph. Logseq/SteleKit will create stub pages for these, which is confusing. +- **Mitigation options**: + 1. Post-process: strip all `[[...]]` from LLM output; let the user manually linkify. + 2. Prompt constraint: "Do NOT create `[[wiki links]]` unless the exact page name was explicitly stated in the transcript." + 3. Graph-aware prompting: pass a list of existing page names as context; tell the LLM to only link from that list. + 4. Structured output: emit JSON with a `links: []` array the app validates against the graph before rendering. + +**Format drift**: LLMs will drift from the required format, especially for long outputs. Common drift patterns: +- Using `*` or `#` instead of `-` +- Using 4-space indent instead of 2-space +- Adding a preamble ("Here are the key points from your recording:") before the bullets +- Adding a summary paragraph after the bullets +- Wrapping everything in a code block + +**Mitigation**: Use a strict system prompt with examples. Append "Output ONLY the bullet list. No preamble. No summary." Request a review of the first token — if it is not `-`, retry once. [TRAINING_ONLY — verify if structured output / JSON mode is available in Whisper API context] + +**Token limit overflow**: A 30-minute voice recording can produce 4,000–6,000 words of transcript. At ~1.3 tokens/word, this is 5,200–7,800 input tokens, plus the system prompt, plus expected output. For GPT-4o (128k context), this is fine. For smaller on-device models (typically 4k–8k context), this will overflow. +- **Mitigation**: Chunk transcripts > 2,000 words. Send each chunk with the last 2 bullets of the previous chunk as context to maintain continuity. Merge outputs client-side. + +**Incomplete output / mid-sentence cutoff**: If `max_tokens` is too low or the model hits its limit, output is truncated mid-bullet. The result is a malformed last bullet. +- **Mitigation**: Set `max_tokens` to at least `len(transcript_words) * 1.5`. Detect truncation by checking if the last line ends with a complete sentence (ends with `.`, `?`, `!`, or `]]`). If not, display a "formatting may be incomplete" warning. + +--- + +### 6. Whisper API Pitfalls + +**File size limit**: Whisper API accepts files up to 25 MB. At 128 kbps MP3, 25 MB ≈ 26 minutes of audio. At WAV (16-bit, 16 kHz mono), 25 MB ≈ 13 minutes. [TRAINING_ONLY — verify current limits] +- **Mitigation**: Record to AAC/M4A (much smaller than WAV). Split recordings approaching the limit client-side before upload. Use `AudioRecord` with a circular buffer and split at silence gaps (VAD — Voice Activity Detection) rather than at fixed byte counts to avoid splitting mid-word. + +**Audio format requirements**: Whisper API accepts mp3, mp4, mpeg, mpga, m4a, wav, webm. It does NOT accept raw PCM (`.pcm`) or AMR. [TRAINING_ONLY — verify current format list] +- **Android pitfall**: `AudioRecord` produces raw PCM. You must write a WAV header before sending, or transcode to MP3/M4A using `MediaCodec`. Forgetting the header causes a 400 error from the API with a cryptic "could not decode audio" message. +- **iOS pitfall**: `AVAudioEngine` tap produces `AVAudioPCMBuffer`. You must write it to a file with `AVAudioFile` in the desired format. Using the wrong `AVAudioCommonFormat` produces a file the API rejects. + +**Rate limits**: Whisper API has per-minute rate limits that vary by tier. A free-tier user sending a 10-minute recording every 5 minutes will hit limits quickly. [TRAINING_ONLY — verify current OpenAI rate limits] +- **Mitigation**: Implement exponential backoff with jitter. Cache in-progress transcription state so a retry does not re-upload the full file. Queue recordings if the API is unavailable. + +**Accuracy in noisy / driving environments**: Whisper is trained on diverse data and handles moderate noise well, but: +- Wind noise from a car window or HVAC confuses VAD and can produce spurious tokens. +- Strong accents combined with background noise degrade accuracy significantly. +- Proper nouns (people names, product names) are frequently misspelled by Whisper even in clean audio. +- **Mitigation**: Enable noise suppression on the audio session before recording. On Android, use `AudioRecord` with `VOICE_COMMUNICATION` audio source (applies system-level noise cancellation) rather than `DEFAULT`. On iOS, use `AVAudioSession.Mode.voiceChat` which enables echo cancellation and noise reduction. + +**Cost at scale**: At $0.006/minute, 1,000 users recording 5 minutes/day = $30/day = $900/month. This scales linearly and can surprise founders. [TRAINING_ONLY — verify current Whisper pricing] +- **Mitigation**: Default to on-device STT where available. Use Whisper API as a fallback. Meter usage per user. Add a "use on-device" preference toggle. + +--- + +## Migration and Adoption Cost + +| Change | Effort | Risk | +|---|---|---| +| Add Moko Permissions to commonMain | Low (1–2 days) | Low; well-understood dependency | +| Implement expect/actual audio capture | High (1–2 weeks per platform) | High; platform APIs are very different | +| Foreground service + notification (Android) | Medium (2–3 days) | Medium; requires manifest changes + UX for notification | +| Background audio mode (iOS) | Low (1 day) | Medium; App Store review risk | +| AICore availability check + fallback | Medium (2–3 days) | Low if behind a feature flag | +| WhisperKit integration (iOS) | Medium (3–5 days) | Low; library is well-maintained | +| whisper.cpp JNI (Android on-device STT) | High (1 week) | High; JNI complexity, binary size | +| LLM structured output pipeline | Medium (3–5 days) | Medium; prompt engineering iteration | +| Audio chunking for long recordings | Medium (2–3 days) | Medium; continuity between chunks is tricky | + +--- + +## Operational Concerns + +- **Binary size**: Bundling an on-device Whisper model (e.g., whisper-small = ~500 MB) will exceed default Play Store APK size limits. Requires Play Asset Delivery (dynamic feature modules). iOS App Store has similar concerns with on-demand resources. +- **Battery**: `AudioRecord` with continuous capture, PCM processing, and network upload is a significant battery drain. Use the lowest acceptable sample rate (16 kHz for Whisper, which only uses 16 kHz). Stop recording immediately when the user finishes. +- **Storage**: Uncompressed WAV at 16 kHz mono = ~1.9 MB/minute. A 30-minute recording = 57 MB. Always compress to AAC/M4A before storing locally. Delete the raw audio after successful transcription. +- **Privacy**: Audio data sent to OpenAI Whisper API leaves the device. The privacy policy must disclose this. If the user's jurisdiction requires it (e.g., GDPR), you must get explicit consent before the first upload. Log what is sent and when. +- **Observability**: Transcription and LLM formatting failures are silent from the user's perspective. Add structured logging: `[voice-capture] recording_start`, `[voice-capture] upload_bytes=N`, `[voice-capture] transcription_ms=N tokens=N`, `[voice-capture] llm_ms=N`. Alert on error rate > 5%. + +--- + +## Prior Art and Lessons Learned + +- **Otter.ai / Notion AI**: Both found that users do not re-read long AI-formatted notes. The value is in search/retrieval, not reading. This suggests that prompt reliability for perfect Logseq format matters less than ensuring key facts are captured. +- **Apple Voice Memos**: Uses foreground recording only (no background mode on most configurations). Shows a persistent red status bar indicator while recording. This pattern is well understood by users and avoids App Store scrutiny. +- **AudioRecord vs MediaRecorder for reliability**: Multiple Android developers report that `MediaRecorder` produces corrupted MP4 files when interrupted (phone call, notification). `AudioRecord` with manual WAV/AAC encoding is more reliable for uninterrupted capture under real-world conditions. +- **Whisper hallucination on silence**: Whisper is known to hallucinate text (typically "Thank you.", "you", or copyright notices) on silent or nearly-silent audio segments. Always check that the transcript is non-trivially long before sending to the LLM. If `len(transcript) < 10 words`, treat it as empty and prompt the user to try again. +- **LLM refusal on personal content**: Some LLM providers (especially with default safety filters) may refuse to process voice notes containing discussion of medication, mental health, or financial topics. Use a system prompt that establishes a "personal notes assistant" context. Test against common personal topics. + +--- + +## Open Questions + +1. Does SteleKit need to work offline (no internet) for the STT → LLM pipeline? If yes, on-device models are required and binary size / device gate become critical constraints. +2. What is the minimum supported Android API level? Foreground service `foregroundServiceType="microphone"` requires targeting API 34+. +3. What is the expected recording length? Short (< 2 min) vs long (> 10 min) drives different architecture decisions (chunking, file format, cost). +4. Is there a maximum acceptable latency from "stop recording" to "journal entry appears"? This determines whether on-device STT is worth the complexity. +5. Should `[[links]]` ever be generated automatically, or should the feature only produce plain text bullets? +6. Is the Whisper API cost acceptable at projected user scale, or is on-device STT required from day 1? +7. What happens if the LLM formats a voice note incorrectly — is there a user-facing edit flow, or is the result final? + +--- + +## Recommendation + +Implement in three tiers gated by capability detection: + +**Tier 1 (all devices, day 1)**: Remote Whisper API for STT + remote LLM (GPT-4o or Claude) for formatting. No on-device models. Foreground-only recording. Use `AudioRecord` + AAC transcoding on Android, `AVAudioEngine` on iOS. Zero-shot prompt with few-shot Logseq examples. Strip all `[[links]]` from LLM output initially. + +**Tier 2 (Pixel 8+ / iOS 17+, post-launch)**: On-device Whisper (whisper.cpp JNI on Android, WhisperKit on iOS) for STT. Remote LLM for formatting. Reduces latency and eliminates Whisper API cost. + +**Tier 3 (future)**: On-device LLM for formatting (AICore Gemini Nano on Android when available). True offline voice capture. Requires grammar-constrained decoding for reliable Logseq format. + +For the permission layer, use Moko Permissions in commonMain. For Android audio interruption, use `AudioRecord` + `AudioFocus` with a foreground service. Do NOT declare the iOS `audio` background mode for v1 — foreground-only is the safe path. Add a `LlmBackend` abstraction from the start so fallback tiers can be added without refactoring the call site. + +--- + +## Pending Web Searches + +The following queries should be run to verify training-knowledge claims: + +1. `"AICore" OR "Gemini Nano" android availability 2024 2025 device list pixel` +2. `moko-permissions kotlin multiplatform compose multiplatform 2024 microphone` +3. `android foregroundServiceType microphone api level requirement 2024` +4. `WhisperKit ios swift production ready 2024 argmax` +5. `openai whisper api file size limit format requirements 2024` +6. `openai whisper hallucination silence "thank you" known issue` +7. `android AudioRecord vs MediaRecorder interruption reliability` +8. `AVAudioSession requestRecordPermission ios silent failure empty audio` +9. `apple intelligence public api ios 18 programmatic access writing tools` +10. `logseq markdown format spec bullet indentation two spaces` +11. `android 12 background microphone restriction foreground service` +12. `openai whisper api pricing per minute 2024 2025` + +--- + +## Web Search Results + +_Searches run: 2026-04-18._ + +### 1. Android foregroundServiceType="microphone" — API Level Confirmed + +**Query**: `android foregroundServiceType microphone requirement API level manifest 2024` + +**Verdict**: CONFIRMED AND CLARIFIED. + +- Declaring `android:foregroundServiceType="microphone"` in the manifest is required for + foreground services accessing the mic. Omitting it **silently denies** mic access on + devices with targetSdk ≥ 30 (Android 11+) — no crash, just no audio. +- For apps **targeting API 34** (Android 14+): additionally requires + `android.permission.FOREGROUND_SERVICE_MICROPHONE` in the manifest AND `RECORD_AUDIO` at + runtime. +- **Background launch restriction (Android 12+)**: A microphone foreground service cannot be + *started* from the background. It must already be in the foreground when the app + backgrounds. This confirms the §3 mitigation: "start the foreground service before the app + leaves the foreground." + +**Updated claim**: §3 (Background Recording Restrictions): `[TRAINING_ONLY — verify exact API +level]` — **confirmed: foregroundServiceType="microphone" is mandatory at targetSdk ≥ 30; +the additional `FOREGROUND_SERVICE_MICROPHONE` permission is required at targetSdk ≥ 34**. + +**Sources**: +- [Foreground service types are required — Android 14 — Android Developers](https://developer.android.com/about/versions/14/changes/fgs-types-required) +- [Foreground service types — Android Developers](https://developer.android.com/develop/background-work/services/fgs/service-types) +- [Guide to Foreground Services on Android 14 — Medium](https://medium.com/@domen.lanisnik/guide-to-foreground-services-on-android-9d0127dc8f9a) + +--- + +### 2. Whisper Hallucination on Silence — Confirmed + +**Query**: `openai whisper hallucination silence "thank you" known issue short audio` + +**Verdict**: CONFIRMED. This is a well-documented, unresolved upstream issue. + +- Whisper hallucinates "Thank you.", "you", and subtitle-style text on silent/near-silent audio. +- Traced to subtitle training data with end-of-content markers that map to the same tokens. +- "Thank" (token 1044) is a real token — cannot be blocklisted without side effects on real + transcripts. +- `hallucination_silence_threshold` in whisper.cpp partially mitigates but can drop real + speech near silence boundaries. +- Calm-Whisper (ICML 2025) achieves >80% reduction in non-speech hallucination via fine-tuning, + but the standard `whisper-1` API model still exhibits the behavior. + +**Recommendation confirmed**: Check `len(transcript.split()) < 10` before passing to the LLM. +A VAD (voice activity detection) gate before upload is the most reliable prevention. These +mitigations are already noted in §6 and §Recommendation — no change needed, but the risk is +now backed by confirmed public evidence. + +**Sources**: +- [Hallucination on audio with no speech — openai/whisper Discussion #1606](https://github.com/openai/whisper/discussions/1606) +- [Whisper silent audio hallucination — OpenAI Community](https://community.openai.com/t/whisper-silent-audio-hallucination/1305173) +- [A possible solution to Whisper hallucination — openai/whisper Discussion #679](https://github.com/openai/whisper/discussions/679) +- [Calm-Whisper (arXiv 2025)](https://arxiv.org/html/2505.12969v1) + +--- + +### 3. moko-permissions — Version and CMP Compatibility + +**Query**: `moko-permissions kotlin multiplatform compose multiplatform 2025 microphone version` + +**Verdict**: CONFIRMED. moko-permissions v0.20.1 (released August 28, 2025) supports: +- Android + iOS (KMP expect/actual) +- Compose Multiplatform via `permissions-compose:0.20.1` +- `Permission.RECORD_AUDIO` cross-platform +- Android API 16+, iOS 12.0+ + +The `[TRAINING_ONLY — verify current Moko Permissions version and CMP compatibility]` note is +resolved. The library is actively maintained and CMP-compatible. The pitfall noted in §1 +(KMP Permission Abstraction) remains valid: Moko handles the *permission request* but you +must still configure `AVAudioSession` category separately before recording on iOS. + +**Sources**: +- [icerockdev/moko-permissions — GitHub](https://github.com/icerockdev/moko-permissions) +- [How to Use Moko-Media and Moko-Permissions in CMP — Medium](https://medium.com/@marceloamendes/como-utilizar-o-moko-media-e-o-moko-permissions-no-compose-multiplatform-d576cf5cda70) + +--- + +### 4. WhisperKit (iOS) — Production Ready + +**Query**: `WhisperKit ios swift production ready argmax 2025` + +**Verdict**: CONFIRMED. WhisperKit by Argmax is **production-ready** as of 2025: +- Actively maintained; featured at ICML 2025. +- Runs Whisper on Core ML (Apple Neural Engine) — fully on-device. +- Supports real-time streaming, word timestamps, VAD. +- Swift Package Manager integration; requires Xcode 16.0+. +- Available under open-source license; Argmax Pro SDK available for enterprise scaling. +- TTSKit (text-to-speech) was also added as an optional product in the same package. + +**Updated claim**: §4 `[TRAINING_ONLY — verify WhisperKit production-readiness]` — +**confirmed production-ready**. It is the recommended on-device STT path for iOS in Tier 2. + +**Sources**: +- [argmaxinc/WhisperKit — GitHub](https://github.com/argmaxinc/WhisperKit) +- [WhisperKit — Argmax](https://www.argmaxinc.com/blog/whisperkit) + +--- + +### 5. Apple FoundationModels — LLM API Now Public (changes §4) + +**Query**: `apple intelligence public api ios 18 programmatic access writing tools` +_(via: `Apple Intelligence FoundationModels framework public API third party iOS 18 2025`)_ + +**Verdict**: MAJOR UPDATE. Apple's `FoundationModels` framework is now a **public developer +API** (announced WWDC June 2025, available via Apple Developer Program). + +**Impact on §4 (On-Device LLM Availability Gating)**: +- The statement "Apple does not expose a public Swift API for 'send text to Apple Intelligence'" + is now **incorrect**. +- `FoundationModels` is a text-in / text-out API (not STT) available for text generation, + structured output, and tool calling. +- Device gate: iPhone 15 Pro+, Apple Silicon devices, iOS 18.1+. +- This is directly usable as the LLM formatting step: transcript → Logseq outliner bullets, + entirely on-device, no network, no API cost. +- The `LlmBackend` interface proposed in §Recommendation should include an + `AppleIntelligenceLlmBackend` tier for iOS, checked via a capability API at runtime. + +**Updated claim**: §4 "Apple does not expose a public Swift API" — **outdated**. FoundationModels +is public as of WWDC 2025. Update Tier 3 in the Recommendation to include it as the iOS +on-device LLM path alongside (or replacing) cloud LLM for supported devices. + +**Sources**: +- [Foundation Models — Apple Developer Documentation](https://developer.apple.com/documentation/FoundationModels) +- [Apple's Foundation Models framework — Apple Newsroom (Sept 2025)](https://www.apple.com/newsroom/2025/09/apples-foundation-models-framework-unlocks-new-intelligent-app-experiences/) +- [Apple Announces Foundation Models Framework — MacRumors](https://www.macrumors.com/2025/06/09/foundation-models-framework/) + +--- + +### 6. ML Kit GenAI Speech Recognition — New Public API (changes §1 Android scope) + +**Query**: `ML Kit GenAI speech recognition android availability devices 2025` + +**Verdict**: NEW INFORMATION. Google's ML Kit GenAI Speech Recognition (backed by Gemini Nano +/ AICore) is a public stable API as of 2025. This changes the §1 STT options table. + +Key facts: +- Basic mode: API 31+, most Android devices. +- Advanced mode: Pixel 9/10, Samsung Galaxy S25/S26, Honor Magic 7/8 Pro, OPPO, Fold7, expanding. +- **Critical**: blocked when app is not the top foreground activity + (`ErrorCode.BACKGROUND_USE_BLOCKED`). Cannot be used from a background foreground service. +- Streaming output (partial → final). + +**Impact on §1 microphone permission pitfalls**: The ML Kit GenAI STT path avoids the +`AudioRecord`/`MediaRecorder` audio focus complexity entirely — it handles audio internally. +However the foreground-only restriction means it cannot be used for background capture. For +Phase 1 (in-app foreground recording), it is a viable free on-device alternative to Whisper. + +**Sources**: +- [GenAI Speech Recognition API — Google for Developers](https://developers.google.com/ml-kit/genai/speech-recognition/android) +- [Android Developers Blog: On-device GenAI APIs with ML Kit (May 2025)](https://android-developers.googleblog.com/2025/05/on-device-gen-ai-apis-ml-kit-gemini-nano.html) +- [ML Kit GenAI APIs — Android Developers](https://developer.android.com/ai/gemini-nano/ml-kit-genai) + +--- + +### 7. Whisper API Pricing — Confirmed + +**Query**: `openai whisper api pricing per minute 2024 2025` + +**Verdict**: CONFIRMED. `whisper-1` is $0.006/min. New lower-cost option: `gpt-4o-mini- +transcribe` at $0.003/min. The cost-at-scale example in §6 ($30/day for 1,000 users × 5 +min/day) remains accurate for `whisper-1`; using `gpt-4o-mini-transcribe` halves it to $15/day. + +**Sources**: +- [OpenAI API Pricing](https://openai.com/api/pricing/) +- [OpenAI Whisper API Pricing Apr 2026 — CostGoat](https://costgoat.com/pricing/openai-transcription) diff --git a/project_plans/mobile-voice-mode/research/research_plan.md b/project_plans/mobile-voice-mode/research/research_plan.md new file mode 100644 index 000000000..77ca3a97c --- /dev/null +++ b/project_plans/mobile-voice-mode/research/research_plan.md @@ -0,0 +1,39 @@ +# Research Plan: Mobile Voice Mode + +**Date**: 2026-04-18 +**Input**: `project_plans/mobile-voice-mode/requirements.md` + +## Subtopics + +### 1. Stack +**Focus**: Evaluate STT options and LLM API client libraries available in Kotlin Multiplatform +**Search strategy**: Survey Android SpeechRecognizer, iOS SFSpeechRecognizer, OpenAI Whisper API, Android AICore / Gemini Nano, Apple Intelligence, KMP HTTP clients (Ktor), existing KMP LLM SDK wrappers +**Search cap**: 5 searches +**Trade-off axes**: KMP compatibility, offline capability, accuracy, cost, setup complexity +**Output**: `research/stack.md` + +### 2. Features +**Focus**: Survey comparable voice-to-notes tools for UX and pipeline patterns +**Search strategy**: Audiopen, Whisper Memos, Notion AI voice, Bezel, Otter.ai — what do their capture flows look like, what formatting do they apply, how do they handle long rambles +**Search cap**: 4 searches +**Trade-off axes**: Capture UX, formatting quality, latency, outliner/structured output support +**Output**: `research/features.md` + +### 3. Architecture +**Focus**: Plugin interface design for STT+LLM providers; commonMain pipeline; platform audio capture adapter pattern in KMP +**Search strategy**: KMP expect/actual for audio capture, provider plugin patterns in Kotlin, existing KMP AI SDK designs, Ktor client interceptor patterns +**Search cap**: 5 searches +**Trade-off axes**: Extensibility, testability, platform coupling, boilerplate overhead +**Output**: `research/architecture.md` + +### 4. Pitfalls +**Focus**: Known failure modes and risks in voice capture on Android/iOS in KMP +**Search strategy**: Android microphone permissions + audio focus in KMP, iOS SFSpeechRecognizer authorization, on-device LLM model availability gates (AICore), LLM prompt reliability for structured output +**Search cap**: 4 searches +**Trade-off axes**: Permission failure modes, audio interruption handling, model gate fallbacks, prompt hallucination risk +**Output**: `research/pitfalls.md` + +## Parallel Execution Plan + +Spawn all 4 subagents simultaneously. Each writes its findings file independently. +Parent synthesizes after all complete → `research/synthesis.md`. diff --git a/project_plans/mobile-voice-mode/research/stack.md b/project_plans/mobile-voice-mode/research/stack.md new file mode 100644 index 000000000..14c94b7ac --- /dev/null +++ b/project_plans/mobile-voice-mode/research/stack.md @@ -0,0 +1,603 @@ +# Findings: Stack — KMP Voice-Capture Feature + +_Research date: 2026-04-18. Sections marked [TRAINING_ONLY — verify] indicate claims from +training data not confirmed by live web search._ + +--- + +## Summary + +SteleKit already has Ktor 3.1.3 in `commonMain`, Kotlin serialization, and coroutines. That +baseline covers the LLM client side without adding new dependencies. Audio recording has no +KMP-native abstraction — it requires `expect/actual` with thin platform adapters on every target. +Speech recognition similarly lives behind platform APIs; the best default is OpenAI Whisper via +HTTP (multipart upload from commonMain) because it is the only STT option that works identically +across Android, iOS, and Desktop from a single code path. On-device STT (Android AICore / +Apple Intelligence) is an optional accelerator added as pluggable backends behind a common +`SpeechToTextProvider` interface. + +--- + +## Options Surveyed + +### 1. Audio Recording APIs + +| Layer | Option | KMP Support | +|---|---|---| +| commonMain | No standard API exists | — | +| androidMain | `android.media.AudioRecord` (raw PCM) or `MediaRecorder` (MP4/AAC) | Android-only | +| iosMain | `AVAudioEngine` / `AVAudioRecorder` (Obj-C interop via cinterop) | iOS-only | +| jvmMain | `javax.sound.sampled.TargetDataLine` (raw PCM) | JVM/Desktop-only | +| KMP library | `KmpAudio` / `multiplatform-audio` — no mature, maintained library confirmed [TRAINING_ONLY — verify] | Partial | + +**Conclusion**: Audio recording must be an `expect/actual`. The common interface emits a +`Flow` of raw PCM chunks; each platform implements recording and streaming into that +flow. The common layer collects chunks, concatenates, and dispatches to the STT provider. + +**Recommended audio format**: 16 kHz, 16-bit mono PCM → WAV wrapper. This matches Whisper's +native input and Android/iOS SpeechRecognizer expectations. + +--- + +### 2. Speech-to-Text Options + +#### 2a. OpenAI Whisper API (remote) + +- Endpoint: `POST https://api.openai.com/v1/audio/transcriptions` +- Accepts WAV/MP3/M4A up to 25 MB; returns plain text or verbose JSON with timestamps. +- Implementable in `commonMain` via Ktor multipart form upload. +- Cost: $0.006 / min (Whisper-1, as of training cutoff) [TRAINING_ONLY — verify current pricing]. +- Accuracy: state-of-the-art English WER ~2–5% on clean speech. +- Latency: 1–5 s for a 1-min clip over a good connection. +- **Works on all targets** (Android, iOS, Desktop, Web) — single implementation. + +#### 2b. Android SpeechRecognizer (on-device + cloud hybrid) + +- `android.speech.SpeechRecognizer` — streaming, returns partial and final results. +- On Android 13+ (API 33) supports `createOnDeviceSpeechRecognizer()` for full offline use + [TRAINING_ONLY — verify API 33 availability and exact method name]. +- Requires `RECORD_AUDIO` permission; streams audio internally; no file upload. +- Limit: Android-only, not shareable with iOS/Desktop. Requires a separate adapter. +- Accuracy: competitive with Whisper for US English; variable for accented speech. +- Cost: free (Google cloud STT bundled via Play Services on most devices). +- Min SDK: SteleKit targets minSdk 24; `SpeechRecognizer` available since API 8. + +#### 2c. iOS SFSpeechRecognizer (on-device + Siri cloud) + +- `Speech.SFSpeechRecognizer` (iOS 10+) with `SFSpeechAudioBufferRecognitionRequest`. +- Supports `.requiresOnDeviceRecognition = true` (iOS 13+, device-dependent model availability). +- Exposed to Kotlin/Native via cinterop (framework `Speech`). +- Apple mandates privacy usage description in `Info.plist` (`NSSpeechRecognitionUsageDescription`). +- Request duration limit: ~1 min per recognition task; must chunk for unlimited recording + [TRAINING_ONLY — verify current limit, historically 60 s]. +- Cost: free; data leaves device only if on-device model unavailable. + +#### 2d. Android AICore / Gemini Nano (on-device ML) + +- Android AICore (Android 14+, API 34+) provides `DownloadCallback`-based model access + [TRAINING_ONLY — verify exact API surface]. +- Gemini Nano is the on-device model; speech input is handled indirectly through + `android.speech` or the AICore Multimodal API. +- As of training cutoff: AICore STT API is not a stable public SDK — primary interface is still + `SpeechRecognizer` or file-based. Gemini Nano is used for text processing, not raw ASR. +- **Verdict**: AICore is relevant for the LLM formatting step (grammar/structure), not STT. + Treat as a future backend for `TextFormatterProvider`. + +#### 2e. Apple Intelligence / on-device LLM (iOS 18+) + +- Apple Intelligence (iOS 18+, Apple Silicon devices) exposes Writing Tools and summarization + through UIKit/SwiftUI APIs, not a developer-callable STT or LLM inference API. +- No public `NaturalLanguage` or `CoreML`-based Apple Intelligence inference API exists for + third-party apps as of training cutoff [TRAINING_ONLY — verify]. +- **Verdict**: not usable as an STT or LLM backend via Kotlin/Native today. Revisit when Apple + publishes an inference API. + +--- + +### 3. LLM API Client Options + +#### 3a. Ktor (existing in project) + +- Already in `commonMain` (`io.ktor:ktor-client-core:3.1.3`). Zero additional dependency. +- Each platform engine is already configured: OkHttp (Android/JVM), Darwin (iOS). +- Supports streaming SSE (`response.bodyAsChannel()` + line parsing). +- JSON serialization: `kotlinx.serialization` already present. +- **All LLM providers** (OpenAI, Anthropic, Groq, etc.) expose REST+JSON; Ktor covers all of + them from a single `HttpClient` in `commonMain`. + +#### 3b. OpenAI Kotlin SDK (official) + +- `com.aallam.openai:openai-kotlin` — KMP-compatible, uses Ktor under the hood + [TRAINING_ONLY — verify current version and KMP support matrix]. +- Targets: JVM, Android, iOS (via Ktor Darwin engine), JS. +- Provides typed request/response models for Chat, Audio, Embeddings, etc. +- Adds ~1–2 MB to app size; brings its own Ktor version which may conflict with project's 3.1.3. +- Version alignment risk: if `openai-kotlin` pins Ktor 2.x, it will conflict with SteleKit's 3.x. + +#### 3c. Anthropic SDK (official) + +- No official KMP SDK exists as of training cutoff [TRAINING_ONLY — verify]. +- Anthropic publishes a JVM/Android SDK (`com.anthropic:sdk`) but it is not KMP-compatible. +- Approach for commonMain: call Messages API directly via Ktor (`POST /v1/messages`). + Anthropic's REST API is well-documented and stable. + +#### 3d. Other KMP AI/LLM Wrappers + +- `kmp-openai` — no well-known library with this exact artifact ID found [TRAINING_ONLY — verify + via Maven/GitHub search; see Pending Web Searches]. +- `kotlin-ai` / `langchain4j-kotlin` — LangChain4j has a Kotlin integration but is JVM-only + [TRAINING_ONLY — verify]. +- `koog` — JetBrains' own KMP AI agent framework; targets KMP with coroutines-native agents + [TRAINING_ONLY — verify; was announced ~2025]. +- **Verdict**: no third-party KMP LLM wrapper is mature enough to prefer over direct Ktor calls. + Roll a thin `LlmClient` in `commonMain` backed by Ktor. + +--- + +### 4. Plugin / Extensibility Architecture + +The feature must allow third parties to add STT and LLM backends. Recommended pattern: + +```kotlin +// commonMain +interface SpeechToTextProvider { + val id: String + val supportsOnDevice: Boolean + suspend fun transcribe(audio: AudioCapture): String +} + +interface TextFormatterProvider { + val id: String + suspend fun format(rawTranscript: String, context: JournalContext): FormattedOutline +} + +interface AudioRecorder { + fun startRecording(): Flow // raw PCM chunks + suspend fun stopRecording() +} +``` + +Platform-specific providers register themselves in `androidMain`/`iosMain`/`jvmMain`. A +`ProviderRegistry` in `commonMain` holds the active set. This is the same pattern as SteleKit's +existing `RepositoryFactory` abstraction. + +--- + +## Trade-off Matrix + +| Option | KMP Compat | Offline | Accuracy | Cost | Setup Complexity | Extensible | +|---|---|---|---|---|---|---| +| Whisper API (Ktor) | Full (commonMain) | No | Excellent | $0.006/min | Low — Ktor already present | Yes — default backend | +| Android SpeechRecognizer | Android only (expect/actual) | Partial (API 33+) | Good | Free | Medium | Yes — adapter | +| iOS SFSpeechRecognizer | iOS only (expect/actual) | Partial (iOS 13+) | Good | Free | Medium (cinterop) | Yes — adapter | +| openai-kotlin library | KMP (JVM/Android/iOS) | No | Excellent | $0.006/min | Medium (Ktor version risk) | Limited | +| Direct Ktor (LLM calls) | Full (commonMain) | No | N/A | Per provider | Low — already present | Yes | +| Android AICore / Gemini Nano | Android 14+ only | Yes | Unknown (STT not direct) | Free | High (early API) | Possible future | +| Apple Intelligence | None (no public API) | Yes (conceptually) | N/A | Free | N/A | No | +| koog (JetBrains) | KMP (unverified) | No | N/A | Per provider | Unknown | Unknown | + +--- + +## Risk and Failure Modes + +### STT Risks + +1. **Whisper upload size limit (25 MB)**: A 1-hour recording at 16 kHz 16-bit mono WAV = ~115 MB. + Must chunk audio server-side or stream. Mitigation: chunk at 10-min intervals (~11.5 MB each), + or compress to MP3/Opus before upload (10:1 ratio → ~11.5 MB/hr). + +2. **iOS SFSpeechRecognizer 60-second limit**: Must create a new recognition task per chunk. + Stitching transcripts requires careful overlap detection to avoid dropped words at boundaries. + +3. **Android SpeechRecognizer network dependency**: On older devices (< API 33) the on-device + model may not be present; the recognizer silently falls back to Google cloud STT, which + requires network. Unexpected failures in airplane mode. + +4. **API key theft**: Keys stored on-device are extractable. Mitigation: support per-user key + entry (no bundled key), proxy mode (user's own backend), or OAuth token exchange. + +5. **Whisper hallucinations on silence**: Whisper produces confabulated text on silent or + noise-only audio. Mitigation: VAD (voice activity detection) gate before sending segments. + +### LLM Risks + +1. **Streaming response parsing**: SSE parsing via Ktor `bodyAsChannel()` is low-level. A malformed + chunk from the provider can stall the flow. Need robust chunked-line parser with timeout. + +2. **Context window for long transcripts**: A 1-hour recording may produce ~10,000 words (~13,000 + tokens). GPT-4o context (128k) handles this, but older models (gpt-3.5, 4k context) will + truncate. Must select model with adequate context or summarize incrementally. + +3. **Prompt injection via transcript**: A malicious speaker could embed prompt-injection text. + Mitigation: wrap transcript in XML delimiters; instruct model the enclosed text is untrusted + audio content. + +### Audio Recording Risks + +1. **Background audio interruption (iOS)**: `AVAudioSession` category must be set to + `.record` or `.playAndRecord`; phone calls, Siri, and other apps will interrupt the session. + Must implement `AVAudioSessionInterruptionNotification` handler. + +2. **ANR risk (Android)**: `AudioRecord` must run on a non-main thread. Use a dedicated + `CoroutineDispatcher(IO)` for the recording loop. + +3. **Large in-memory buffer**: 1 hour at 16 kHz 16-bit mono = ~115 MB RAM. Stream to a temp + file rather than buffering in memory. + +--- + +## Migration and Adoption Cost + +### Baseline (Ktor + Whisper API only, no on-device) + +- New code: `AudioRecorder` expect/actual (3 platform implementations) + `WhisperSttProvider` + (commonMain, ~150 lines) + `LlmFormatterProvider` (commonMain, ~200 lines). +- No new Gradle dependencies beyond what exists. +- Estimated effort: 1–2 sprints for a working end-to-end path. + +### Adding Android SpeechRecognizer adapter + +- New code: `AndroidSpeechRecognizerProvider` in `androidMain` (~100 lines). +- No new dependencies; `android.speech` is part of the Android SDK. +- Estimated effort: 0.5 sprints. + +### Adding iOS SFSpeechRecognizer adapter + +- New code: cinterop binding for `Speech.framework` (add to `iosMain/cinterop`) + + `IosSpeechRecognizerProvider` (~150 lines). +- Estimated effort: 1 sprint (cinterop setup is the bulk of the work). + +### Adding `openai-kotlin` library + +- Risk: Ktor version conflict. Would need to verify `com.aallam.openai:openai-kotlin` supports + Ktor 3.x before adopting. If it does not, the project must either stay on direct Ktor calls + or shade the library. Recommend staying with direct Ktor unless typed models are a priority. + +--- + +## Operational Concerns + +1. **API key management**: No server-side proxy in SteleKit today. The initial design should + store the user's own API key in platform keychain (`KeyStore` on Android, `Keychain` on iOS, + `SecretService`/file on Desktop). Do not bundle a shared key. + +2. **Offline graceful degradation**: When no network is available and no on-device provider is + configured, the UI must inform the user and disable the mic button rather than silently failing. + +3. **Audio file cleanup**: Temp WAV files written to the cache directory must be deleted after + successful transcription. A leak of 115 MB/hr per recording session will fill storage quickly. + +4. **Battery impact**: Continuous `AudioRecord` with 16 kHz sampling is low CPU but the Whisper + upload + LLM call are network-intensive. Warn users if on a metered connection. + +5. **Privacy disclosure**: Both Android (`RECORD_AUDIO` permission) and iOS + (`NSSpeechRecognitionUsageDescription`, `NSMicrophoneUsageDescription`) require explicit + user consent UI before recording. These disclosures must be truthful about cloud processing. + +--- + +## Prior Art and Lessons Learned + +- **Whisper.cpp on-device (JVM/Desktop)**: whisper.cpp has JNI bindings (`io.github.ggerganov: + whisper-jni`) that run on desktop JVM [TRAINING_ONLY — verify artifact coordinates]. This + enables fully offline Desktop transcription. Not feasible for Android (APK size ~150 MB for + the model) without dynamic model download. + +- **Vosk** (offline ASR): Apache-licensed, KMP-unfriendly (C library requiring JNI), supports + offline English with ~5% WER. Usable on JVM Desktop and Android via JNI. Not usable on iOS + without additional porting work. Not recommended as primary path. + +- **Logseq's own voice note feature (if any)**: Logseq does not have a built-in voice-to-journal + feature as of training cutoff. This is a differentiating feature for SteleKit. + +- **Obsidian's approach**: Community plugins (e.g., `obsidian-whisper`) implement voice notes + by recording locally and sending WAV to Whisper API. The plugin pattern directly validates + the plugin-API approach planned here. + +- **Apple WWDC 2024 / on-device ML**: Apple Intelligence Writing Tools have no public inference + API. Third-party apps cannot call the on-device LLM directly. Apple may expose this via a + future `FoundationModels` framework — watch WWDC 2025/2026. + +- **Google AI Edge / MediaPipe**: Google provides on-device ASR via MediaPipe `AudioClassifier` + and `SpeechEmbedder`, but not a full transcription pipeline. Gemini Nano on-device for Android + supports text tasks via `com.google.android.gms:play-services-mlkit-subject-segmentation`-style + API; actual on-device ASR is still via `SpeechRecognizer` [TRAINING_ONLY — verify current + MediaPipe Audio Task API]. + +--- + +## Open Questions + +1. **Desktop audio**: Should Desktop (JVM) support voice capture at all in v1? `javax.sound.sampled` + works but the UX story (laptop mic, no mobile form factor) is weak. Recommend deferring Desktop + audio to v2. + +2. **Chunking strategy**: At what interval should the audio be chunked for upload — fixed time + (e.g., 10 min), silence detection (VAD), or on stop? VAD is more accurate but adds complexity. + +3. **LLM prompt design**: The formatting prompt (raw transcript → outliner bullets) needs iteration. + Should we store the prompt in a user-editable template, or hard-code v1? + +4. **OpenAI Whisper vs Whisper on `openai-kotlin`**: If `openai-kotlin` aligns with Ktor 3.x, + does it provide enough value (typed models, retry logic) to justify the dependency over raw Ktor? + Requires live Maven search (see Pending Web Searches). + +5. **Streaming transcription**: Whisper API is batch (upload file, receive transcript). Is there + appetite for real-time streaming STT (e.g., OpenAI Realtime API, Deepgram, AssemblyAI)? + These use WebSockets; Ktor 3.x has WebSocket support in `ktor-client-websockets`. Scope for v2. + +6. **koog (JetBrains KMP agent framework)**: Is it stable enough and does it add value over a + hand-rolled `LlmFormatterProvider`? Requires current GitHub/Maven research. + +7. **VAD library in KMP**: Is there a KMP-compatible voice activity detection library, or must + it be a platform expect/actual using WebRTC VAD (Android), `AVAudioEngine` noise detection + (iOS)? + +--- + +## Recommendation + +### Default architecture (v1) + +``` +commonMain + AudioRecorder (expect/actual interface) + WhisperSttProvider ← implements SpeechToTextProvider, uses Ktor multipart POST + OpenAiFormatterProvider ← implements TextFormatterProvider, uses Ktor streaming SSE + AnthropicFormatterProvider ← same interface, different endpoint + ProviderRegistry ← holds active STT + LLM provider, swappable + +androidMain + AndroidAudioRecorder ← AudioRecord, 16 kHz 16-bit mono, streams ByteArray flow + AndroidSpeechRecognizerProvider ← optional on-device STT backend (API 33+) + +iosMain + IosAudioRecorder ← AVAudioEngine tap, same format + IosSpeechRecognizerProvider ← optional on-device STT (SFSpeechRecognizer, chunked) + +jvmMain + JvmAudioRecorder ← javax.sound.sampled (Desktop, v2 scope) +``` + +**STT default**: OpenAI Whisper API via Ktor (commonMain). No platform-specific code for the +happy path. Platform on-device providers are opt-in backends registered at app startup. + +**LLM default**: OpenAI Chat Completions API via raw Ktor (`gpt-4o`, streaming). Anthropic as +a second built-in backend. No third-party KMP SDK dependency until `openai-kotlin` Ktor 3.x +compatibility is confirmed. + +**No new Gradle dependencies required** for the Whisper + GPT-4o path. The iOS `SFSpeechRecognizer` +adapter requires adding `Speech.framework` to the cinterop definition. + +### Decision rationale + +- Ktor is already in `commonMain`; adding a Whisper HTTP call is 50 lines, not a new dependency. +- Whisper is the only STT option that works identically on all three targets from one code path. +- Platform native STT (SpeechRecognizer, SFSpeechRecognizer) are better UX (lower latency, + free, partial results) but require per-platform code — implement as optional backends behind + the `SpeechToTextProvider` interface for users who prefer them. +- The `ProviderRegistry` pattern mirrors SteleKit's existing `RepositoryFactory` — low cognitive + overhead for the existing team. + +--- + +## Pending Web Searches + +The following searches were not executed (WebSearch not available). The parent agent should run +these to fill gaps marked [TRAINING_ONLY — verify]: + +1. `site:central.sonatype.com "com.aallam.openai" "openai-kotlin"` — current version, Ktor + compatibility (2.x vs 3.x), KMP target list. + +2. `"kmp-openai" OR "kotlin-openai" site:github.com` — discover any other KMP OpenAI wrappers. + +3. `"koog" site:github.com jetbrains kotlin multiplatform AI agent` — verify koog's KMP support + and stability. + +4. `android "AICore" "SpeechRecognizer" OR "ASR" API site:developer.android.com` — verify whether + Android AICore exposes a direct ASR API or only text tasks. + +5. `"SFSpeechRecognizer" duration limit site:developer.apple.com` — confirm current 60-second + per-task limit or any iOS 17/18 relaxation. + +6. `"whisper.cpp" JNI OR "whisper-jni" maven` — confirm Maven artifact coordinates for JVM + on-device Whisper. + +7. `"openai realtime api" kotlin ktor websocket` — assess feasibility of streaming STT via + OpenAI Realtime API in KMP. + +8. `"MediaPipe" "AudioTask" kotlin android transcription 2025` — confirm current MediaPipe + Audio transcription API availability for Android. + +9. `"Apple Intelligence" "FoundationModels" WWDC 2025 third party API` — check if Apple has + released a public on-device inference API since WWDC 2024. + +10. `"multiplatform audio" kotlin kmp recording site:github.com` — discover any maintained + KMP audio recording libraries that would replace the expect/actual approach. + +--- + +## Web Search Results + +_Searches run: 2026-04-18. Queries listed per finding._ + +### 1. `openai-kotlin` Ktor 3.x Compatibility + +**Query**: `openai-kotlin Ktor 3.x compatibility version 2025 2026` + +**Verdict**: CONFIRMED AND UPDATED. `com.aallam.openai:openai-kotlin` (now branded `openai-client`) +**v4.1.0** fully supports Ktor 3.x. The Ktor 2.x conflict mentioned in the training data is +resolved. SteleKit uses Ktor 3.1.3; the library is compatible. Ktor itself is now at 3.4.0 +(January 2026). The version-alignment risk noted under §3b is no longer a blocking concern — +`openai-kotlin` 4.x upgrades are safe to evaluate. + +**Updated claim**: §3b: "Brings its own Ktor version which may conflict with project's 3.1.3" — +the conflict existed in older 3.x releases of `openai-kotlin` but is resolved in v4.1.0. + +**Sources**: +- [Issue #411: Release compatible with ktor 3.x — aallam/openai-kotlin](https://github.com/aallam/openai-kotlin/issues/411) +- [Releases — aallam/openai-kotlin](https://github.com/aallam/openai-kotlin/releases) +- [Ktor 3.4.0 Is Now Available — The Kotlin Blog](https://blog.jetbrains.com/kotlin/2026/01/ktor-3-4-0-is-now-available/) + +--- + +### 2. Android AICore / ASR Public API + +**Query**: `Android AICore public ASR speech recognition API 2025` + +`ML Kit GenAI speech recognition android availability devices 2025` + +**Verdict**: UPDATED. Android now has a **public on-device ASR API via ML Kit GenAI** +(`developers.google.com/ml-kit/genai/speech-recognition/android`), built on top of Android +AICore / Gemini Nano. This is a stable public SDK — the training-data claim that "AICore STT +API is not a stable public SDK" is now outdated. + +Key facts verified: +- **Basic mode**: available on API 31+ (most Android devices). +- **Advanced mode** (Gemini Nano quality): Pixel 10, Pixel 9 series, Samsung Galaxy S25/S26, + Honor, OPPO Find N5/X9, and more. Device list is expanding. +- **Critical constraint**: GenAI API inference is only permitted when the app is the **top + foreground application**. Background / foreground service use returns + `ErrorCode.BACKGROUND_USE_BLOCKED`. This means the ML Kit GenAI STT path cannot be used + from a foreground service during screen-off recording. +- The API provides streaming transcription (partial → final results), not batch. + +**Updated claim**: §2d: "AICore STT API is not a stable public SDK" — **incorrect as of 2025**. +ML Kit GenAI Speech Recognition is a public stable API. The foreground-only constraint limits +its usefulness for background recording scenarios. + +**Sources**: +- [GenAI Speech Recognition API — Google for Developers](https://developers.google.com/ml-kit/genai/speech-recognition/android) +- [ML Kit GenAI APIs — Android Developers](https://developer.android.com/ai/gemini-nano/ml-kit-genai) +- [Android Developers Blog: On-device GenAI APIs with ML Kit](https://android-developers.googleblog.com/2025/05/on-device-gen-ai-apis-ml-kit-gemini-nano.html) + +--- + +### 3. Whisper API Accepted Audio Formats (m4a / AAC) + +**Query**: `openai whisper API accepted audio formats m4a aac supported 2025` + +**Verdict**: CONFIRMED. The Whisper API explicitly accepts: **flac, mp3, mp4, mpeg, mpga, +m4a, ogg, wav, webm**. `.m4a` is in the supported list. The architecture recommendation to +use `.m4a` (AAC) from `MediaRecorder` (Android) and `AVAudioRecorder` (iOS) is valid — no +format conversion required before upload. Raw PCM (`.pcm`) and AMR are NOT accepted. + +**Updated claim**: §Q1 (architecture.md) `[TRAINING_ONLY — verify Whisper API accepted formats +for m4a]` — **confirmed: m4a is accepted**. The pitfalls.md note about raw PCM requiring +manual WAV header is also confirmed. + +**Sources**: +- [Audio API FAQ — OpenAI Help Center](https://help.openai.com/en/articles/7031512-whisper-audio-api-faq) +- [OpenAI Community: m4a format issue](https://community.openai.com/t/wisper-api-not-recognizing-m4a-file-format/141251) + +--- + +### 4. Whisper API Pricing (current) + +**Query**: `openai whisper API pricing per minute 2025 2026` + +**Verdict**: CONFIRMED at **$0.006/minute** for `whisper-1`. The training-data figure is +accurate. Additionally, newer models are now available: +- `gpt-4o-transcribe`: $0.006/min (higher accuracy, same price) +- `gpt-4o-mini-transcribe`: $0.003/min (lower cost option) + +The cost estimate in §2a ($0.006/min) remains valid. The `gpt-4o-mini-transcribe` option at +half the price is worth considering for v1. + +**Sources**: +- [OpenAI API Pricing](https://openai.com/api/pricing/) +- [OpenAI Whisper API Pricing Apr 2026 — CostGoat](https://costgoat.com/pricing/openai-transcription) + +--- + +### 5. Android SpeechRecognizer On-Device Mode API Level + +**Query**: `android SpeechRecognizer createOnDeviceSpeechRecognizer API 33 offline 2024` + +**Verdict**: CONFIRMED with correction. `createOnDeviceSpeechRecognizer()` was introduced at +**API 31** (Android 12), not API 33. It is available on API 33 but the minimum API level for +the on-device factory method is 31. SteleKit targets minSdk 24; this feature requires a +runtime API check (`Build.VERSION.SDK_INT >= 31`). + +**Updated claim**: §2b: "Android 13+ (API 33) supports `createOnDeviceSpeechRecognizer()`" — +**incorrect; available from API 31**. + +**Sources**: +- [SpeechRecognizer — Android Developers](https://developer.android.com/reference/android/speech/SpeechRecognizer) +- [Android Speech To Text — The missing guide (Medium)](https://medium.com/reveri-engineering/android-speech-to-text-the-missing-guide-part-1-824e2636c45a) + +--- + +### 6. koog (JetBrains KMP AI Agent Framework) + +**Query**: `koog JetBrains kotlin multiplatform AI agent framework 2025 stable` + +**Verdict**: CONFIRMED AND UPDATED. Koog is real, actively maintained by JetBrains, and +**targets KMP** (JVM, Android, iOS, JS, WasmJS). Current version: **0.7.3** (as of late +2025/early 2026). Key facts: +- Open-sourced at KotlinConf May 2025; v0.5.0 shipped October 2025 with Agent-to-Agent (A2A) + Protocol support, OpenTelemetry observability, Ktor plugin, MCP tool support. +- Still pre-1.0 (0.x versions). API stability is not guaranteed. +- Designed for **agent workflows** (multi-step tool use, graph-based strategies), not a simple + LLM call wrapper. For SteleKit's formatting step (single LLM call), koog adds overhead with + little benefit. The raw Ktor recommendation stands. + +**Updated claim**: §3d: "koog — KMP support unverified" — **confirmed KMP support**. Stability +caveat: pre-1.0, not recommended for production until 1.0. + +**Sources**: +- [JetBrains/koog — GitHub](https://github.com/JetBrains/koog) +- [Koog 0.5.0 Is Out — JetBrains AI Blog](https://blog.jetbrains.com/ai/2025/10/koog-0-5-0-is-out-smarter-tools-persistent-agents-and-simplified-strategy-design/) +- [The Kotlin AI Stack — Kotlin Blog](https://blog.jetbrains.com/kotlin/2025/09/the-kotlin-ai-stack-build-ai-agents-with-koog-code-smarter-with-junie-and-more/) + +--- + +### 7. Apple Intelligence / FoundationModels Public API + +**Query**: `Apple Intelligence FoundationModels framework public API third party iOS 18 2025` + +**Verdict**: MAJOR UPDATE. Apple **announced and shipped** the `FoundationModels` framework at +WWDC 2025. Third-party developers can now call Apple's on-device LLM directly. Key facts: +- Announced June 9, 2025 at WWDC; available via Apple Developer Program. +- Supports: text generation, structured output, tool calling, guided generation. +- Works **offline** (on-device), no inference cost. +- Supported languages: English, French, German, Italian, Portuguese (Brazil), Spanish, + Chinese (Simplified), Japanese, Korean. +- **Device requirement**: Apple Intelligence-capable devices only (iPhone 15 Pro+, Apple + Silicon Macs, iPad with A17 Pro+). +- **Not STT**: `FoundationModels` is a text-in / text-out API. It does not replace Whisper for + audio transcription. It is directly applicable as an **LLM formatting backend** — the + transcript → outliner formatting step could use `FoundationModels` on supported devices + instead of a cloud API call. + +**Updated claim**: §2e and §3c: "No public `FoundationModels` API exists" — **incorrect as of +WWDC 2025**. FoundationModels is now a public developer API. It is a viable `LlmProvider` +implementation for `iosMain` on supported hardware. + +**Sources**: +- [Foundation Models — Apple Developer Documentation](https://developer.apple.com/documentation/FoundationModels) +- [Apple Announces Foundation Models Framework — MacRumors](https://www.macrumors.com/2025/06/09/foundation-models-framework/) +- [Apple's Foundation Models framework unlocks new intelligent app experiences — Apple Newsroom](https://www.apple.com/newsroom/2025/09/apples-foundation-models-framework-unlocks-new-intelligent-app-experiences/) + +--- + +### 8. Whisper Hallucination on Silence — Confirmed + +**Query**: `openai whisper hallucination silence "thank you" known issue short audio` + +**Verdict**: CONFIRMED. Whisper hallucination on silent/near-silent audio is a well-documented, +unresolved issue. Specific confirmed behaviors: +- Hallucinated tokens include "Thank you.", "you", and subtitle-style credits (traced to + subtitle training data with end-of-content markers). +- "Thank" (token 1044) is a legitimate token that cannot be blocklisted without side effects. +- A `hallucination_silence_threshold` parameter exists in whisper.cpp but can cause false + positives that drop real speech near silence boundaries. +- Research (Calm-Whisper, ICML 2025) shows >80% reduction in non-speech hallucination is + achievable with fine-tuning, but the standard API model still exhibits the issue. + +**Mitigation confirmed**: Check `len(transcript.split()) < 10` and treat as empty. Additionally, +sending audio with known speech (via VAD gate) is the most reliable prevention. + +**Sources**: +- [Hallucination on audio with no speech — openai/whisper Discussion #1606](https://github.com/openai/whisper/discussions/1606) +- [Whisper silent audio hallucination — OpenAI Community](https://community.openai.com/t/whisper-silent-audio-hallucination/1305173) +- [Calm-Whisper: Reduce Whisper Hallucination (arXiv, 2025)](https://arxiv.org/html/2505.12969v1) diff --git a/project_plans/mobile-voice-mode/research/synthesis.md b/project_plans/mobile-voice-mode/research/synthesis.md new file mode 100644 index 000000000..82bc23220 --- /dev/null +++ b/project_plans/mobile-voice-mode/research/synthesis.md @@ -0,0 +1,325 @@ +# Research Synthesis: Mobile Voice Mode + +**Date**: 2026-04-18 +**Sources**: stack.md, features.md, architecture.md, pitfalls.md (all web-search-verified) +**Next step**: Write ADRs covering audio capture adapter, STT provider interface, LLM provider interface, and UX state machine + +--- + +## Decision Required + +Four architectural decisions must be made before implementation begins: + +1. **Audio capture shape** — `expect class` vs plain `interface` + constructor injection; output format (PCM vs m4a file) +2. **STT provider interface and tier defaults** — which providers ship in v1, which are deferred, and how tiering works +3. **LLM provider interface and tier defaults** — same question for the formatting step +4. **UX state machine** — what states the mic button exposes and how Phase 1/2/3 scope gates land + +--- + +## Context + +SteleKit is a KMP app targeting Android and iOS from a single `commonMain`. It already has: +- Ktor 3.1.3 in `commonMain` (OkHttp engine on Android, Darwin on iOS) +- `kotlinx.serialization` in `commonMain` +- A proven `suspend fun interface` + `NoOp` seam pattern, established by `TopicEnricher` / ADR-002 +- `JournalService` with `ensureTodayExists()` guarded by a `Mutex` +- `PlatformBottomBar.android.kt` with existing 4-item nav bar + +The feature adds a voice capture pipeline: mic button → audio file → STT → LLM format → `JournalService.appendToToday()`. No new framework dependencies are required for the Whisper + cloud LLM happy path. + +**Scope from requirements.md** (confirmed constraints for synthesis): +- Must Have: single-tap trigger, unlimited-duration recording, STT, LLM → Logseq format, journal insert, extensible provider API +- Out of Scope: real-time transcription display, background/passive listening, Desktop support, multi-language + +--- + +## Options Considered + +### Option A — `expect class AudioRecorder` (monolithic platform class) + +Every property and method in `expect` must mirror in every `actual`. Adding a capability later forces all platform actuals to update in lockstep. Desktop requires a stub `actual` even though Desktop is explicitly out of scope. `PlatformFileSystem` uses this approach because it is passed to many constructors; `AudioRecorder` is only used in one ViewModel. + +**Verdict**: Eliminated. Unjustified rigidity for a single-consumer class. + +### Option B — Plain `interface AudioRecorder` in `commonMain`, platform impls injected at assembly (recommended) + +Matches ADR-002 seam pattern exactly. No `expect`/`actual` required. Android wires `AndroidAudioRecorder`, iOS wires `IosAudioRecorder`, Desktop gets `NoOpAudioRecorder` by default. Adding a capability to the interface does not force platform stubs. + +**Verdict**: Selected. Zero KMP machinery overhead; identical to how `TopicEnricher` works. + +### Option C — Raw PCM streaming (`Flow`) + +Enables live waveform animation. Requires assembling PCM into a file before Whisper upload anyway (Whisper rejects raw PCM). "Real-time transcription display while speaking" is explicitly out of scope in requirements.md. + +**Verdict**: Deferred to Phase 2 (waveform animation only). v1 records to a temp `.m4a` file. + +### Option D — OpenAI Whisper as the only STT path (commonMain only) + +Simpler: one implementation, works on all targets. No platform-specific STT code. Costs $0.006/min (`whisper-1`) or $0.003/min (`gpt-4o-mini-transcribe`). Requires network. The Whisper API explicitly accepts `.m4a` (confirmed by web search — no format conversion needed). + +**Verdict**: Selected as the v1 default (`WhisperSpeechToTextProvider`). Platform STT providers are optional built-in adapters behind the same interface. + +### Option E — Android ML Kit GenAI `createOnDeviceSpeechRecognizer` as Tier 1 default + +Web-search-confirmed: ML Kit GenAI Speech Recognition is a stable public API (API 31+, basic mode; Gemini Nano quality on Pixel 9/10 and Galaxy S25/S26+). Free, on-device, streaming output. **Hard constraint**: blocked when app is not the top foreground activity (`ErrorCode.BACKGROUND_USE_BLOCKED`). Cannot be used from a foreground service. + +**Verdict**: Valid Tier 1 for Phase 1 (foreground-only recording). Add as `AndroidMlKitSpeechToTextProvider` behind the `SpeechToTextProvider` interface. Users on supported devices get free on-device STT; fallback to Whisper API when unavailable or backgrounded. + +### Option F — `SFSpeechRecognizer` (iOS) as Tier 1 default + +Free, on-device capable (iOS 13+ with device model), no cost, streaming output. `requiresOnDeviceRecognition = true` on supported devices. Duration limit per recognition task requires chunking for long recordings. + +**Verdict**: Valid Tier 1 for iOS. Add as `IosSpeechToTextProvider`. + +### Option G — WhisperKit (iOS, Tier 2) + +Confirmed production-ready at ICML 2025 (Argmax). Runs on Core ML / Apple Neural Engine. Fully offline. VAD, word timestamps, streaming. Swift Package Manager. Device gate: iPhone 12+ (hardware Neural Engine). + +**Verdict**: Tier 2 / Phase 2 for iOS on-device. Not v1 — adds cinterop complexity and model download UX. + +### Option H — FoundationModels (iOS LLM, Tier 2) + +Apple shipped `FoundationModels` at WWDC 2025. Text-in / text-out API. Offline, no cost. Device gate: iPhone 15 Pro+, iOS 18.1+. Directly usable as `LlmProvider` in `iosMain`. Not STT — does not replace Whisper. + +**Verdict**: Tier 2 LLM backend for iOS. Planned but not v1. Add as `AppleIntelligenceLlmProvider` in `iosMain` behind the `LlmProvider` interface. + +### Option I — `openai-kotlin` v4.1.0 library + +Web-search-confirmed: `com.aallam.openai:openai-kotlin` v4.1.0 supports Ktor 3.x. The prior Ktor 2.x conflict risk is resolved. However, the library adds ~1–2 MB and typed models that are not needed for the narrow `transcribe()` and `format()` call surface. Raw Ktor is 50 lines per provider and already present. + +**Verdict**: Not adopted in v1. Raw Ktor calls are simpler and already present. Revisit if typed model coverage becomes valuable. + +### Option J — Koin DI for provider wiring + +Not currently used in SteleKit. Introducing it for one feature adds a new dependency and pattern inconsistent with the rest of the codebase (`App.kt` uses `remember {}` blocks, not a DI graph). + +**Verdict**: Eliminated. Constructor injection with NoOp defaults is the established pattern. + +### Option K — `MediaRecorder` (Android) vs `AudioRecord` (raw PCM) + +Multiple Android developers and the pitfalls research confirm: `MediaRecorder` produces corrupted MP4 box headers when interrupted by a phone call or audio focus loss, because it has no pause/resume on API < 24. `AudioRecord` reads raw PCM into a buffer; on interruption you can pause reads and resume. Whisper rejects raw PCM — but `.m4a` from `MediaRecorder` is accepted directly (web-search-confirmed). + +**Critical resolution**: Use `AudioRecord` (raw PCM) for reliability, then encode to `.m4a` via `MediaCodec` before upload. This avoids the `MediaRecorder` corruption failure mode while producing a Whisper-compatible format. The encoding step adds ~50 lines of Android code. + +**Verdict**: `AudioRecord` + `MediaCodec` AAC encoder on Android. `AVAudioRecorder` (outputs `.m4a` natively) on iOS. + +--- + +## Dominant Trade-off + +**On-device STT (free, private, foreground-only) vs cloud Whisper (paid, universal, any context)** + +This is the central tension. The resolution is a tiered architecture where the tier is selected at runtime based on platform, API level, and whether the app is in the foreground: + +``` +Android Phase 1 (foreground): ML Kit GenAI STT (free, API 31+) → fallback: Whisper API +Android Phase 3 (background): Whisper API only (ML Kit GenAI is foreground-blocked) +iOS Phase 1 (foreground): SFSpeechRecognizer (free, iOS 13+) → fallback: Whisper API +iOS Phase 2 (foreground): WhisperKit on-device → fallback: Whisper API +iOS LLM Phase 2: FoundationModels (iPhone 15 Pro+) → fallback: cloud LLM +``` + +The `SpeechToTextProvider` and `LlmProvider` interfaces hide this entirely from `VoiceCaptureViewModel`. + +--- + +## Recommendation + +### Interface Shapes (follow ADR-002 seam pattern exactly) + +```kotlin +// commonMain — audio capture (plain interface, not expect/actual) +interface AudioRecorder { + suspend fun recordToFile(): PlatformAudioFile // blocks until stopRecording() called + suspend fun stopRecording() +} + +// commonMain — STT provider seam +fun interface SpeechToTextProvider { + suspend fun transcribe(audio: PlatformAudioFile): TranscriptResult +} + +sealed interface TranscriptResult { + data class Success(val text: String) : TranscriptResult + data object Empty : TranscriptResult + sealed interface Failure : TranscriptResult { + data object NetworkError : Failure + data class ApiError(val code: Int, val message: String) : Failure + data object AudioTooShort : Failure + data object PermissionDenied : Failure + } +} + +// commonMain — LLM formatting provider seam +fun interface LlmProvider { + suspend fun format(transcript: String, systemPrompt: String): LlmResult +} + +sealed interface LlmResult { + data class Success(val formattedText: String) : LlmResult + sealed interface Failure : LlmResult { + data object NetworkError : Failure + data class ApiError(val code: Int, val message: String) : Failure + data object Timeout : Failure + data class MalformedResponse(val raw: String) : Failure + } +} +``` + +All interfaces follow ADR-002's `suspend fun interface` + `NoOp` default pattern. `VoiceCaptureViewModel` receives all three via constructor injection with NoOp defaults. `StelekitApp` gains `sttProvider` and `llmProvider` optional parameters threaded through to `GraphContent` — identical to how `urlFetcher` is wired today. + +### Provider Tiers + +**STT providers (ship in priority order):** + +| Tier | Provider | Impl location | Condition | +|------|----------|--------------|-----------| +| 0 | `NoOpSpeechToTextProvider` | `commonMain` | Default / tests | +| 1a | `AndroidMlKitSpeechToTextProvider` | `androidMain` | API 31+, foreground-only; uses `createOnDeviceSpeechRecognizer` | +| 1b | `IosSpeechToTextProvider` | `iosMain` | iOS 13+; `SFSpeechRecognizer` with chunking | +| 2 | `WhisperSpeechToTextProvider` | `commonMain` | Ktor multipart POST; user-supplied API key; fallback on all platforms | +| 3 | `WhisperKitSpeechToTextProvider` | `iosMain` (Phase 2) | iPhone 12+; on-device Core ML | + +**LLM providers (ship in priority order):** + +| Tier | Provider | Impl location | Condition | +|------|----------|--------------|-----------| +| 0 | `NoOpLlmProvider` | `commonMain` | Default / tests (returns transcript verbatim) | +| 1 | `ClaudeLlmProvider` | `commonMain` | Ktor + Anthropic Messages API; user-supplied key | +| 1 | `OpenAiLlmProvider` | `commonMain` | Ktor + OpenAI Chat Completions; user-supplied key | +| 2 | `AppleIntelligenceLlmProvider` | `iosMain` (Phase 2) | FoundationModels; iPhone 15 Pro+, iOS 18.1+; offline, no cost | + +### VoiceCaptureViewModel State Machine + +```kotlin +sealed interface VoiceCaptureState { + data object Idle : VoiceCaptureState + data object Recording : VoiceCaptureState + data object Transcribing : VoiceCaptureState + data object Formatting : VoiceCaptureState + data class Done(val insertedText: String) : VoiceCaptureState + data class Error(val stage: PipelineStage, val message: String) : VoiceCaptureState +} + +enum class PipelineStage { RECORDING, TRANSCRIPTION, LLM, JOURNAL_INSERT } +// Any state → Idle on cancel or dismiss +``` + +### UX Phases + +**Phase 1** (v1 scope): FAB-style mic button in `PlatformBottomBar`. Tap to record (enters `Recording` state), tap again to stop. Post-stop: Whisper API transcribes → `ClaudeLlmProvider` or `OpenAiLlmProvider` formats → appended to today's journal as a timestamped block. Raw transcript preserved as a collapsible block below the formatted output. No lock screen access. + +**Phase 2** (post-v1): Add `AndroidMlKitSpeechToTextProvider` (foreground) and `IosSpeechToTextProvider` as default on-device options — user gets free STT without an API key. Add `AppleIntelligenceLlmProvider` on iOS. Add live waveform animation via `Flow` from `AudioRecord`. + +**Phase 3** (post-v1): Foreground service on Android with notification action button (lock-screen access). Requires `android:foregroundServiceType="microphone"` in manifest + `FOREGROUND_SERVICE_MICROPHONE` permission (targetSdk ≥ 34). The service must be started while the app is still visible — cannot be started from background (Android 12+). On iOS, lock screen widget via iOS Widget APIs. + +### Audio Pipeline + +``` +Android: AudioRecord (raw PCM, VOICE_COMMUNICATION source) + → MediaCodec AAC encoder + → .m4a temp file in cacheDir + → WhisperSpeechToTextProvider (Ktor multipart POST) + → delete temp file (in finally block) + +iOS: AVAudioRecorder (AVAudioSession.setCategory(.record) before start) + → .m4a temp file in NSTemporaryDirectory + → WhisperSpeechToTextProvider (Ktor multipart POST) + → delete temp file (in finally block) + → restore AVAudioSession category to .playback/.ambient after stop +``` + +Whisper API file size limit is 25 MB. At 128 kbps AAC, 25 MB ≈ 26 minutes. For recordings approaching this limit, split at silence gaps (VAD) before upload. For Phase 1, a simple warning at 20 minutes is sufficient. + +### Critical Non-Negotiable Rules + +These are confirmed failure modes from web-search-verified evidence. Each must be enforced at implementation time: + +1. **iOS AVAudioSession category must be set before `AVAudioEngine`/`AVAudioRecorder` starts.** Forgetting it produces silent recording — empty audio, no error, Whisper returns "Thank you." Silent failure is the worst UX outcome. + +2. **Use `AudioRecord` (raw PCM) on Android, not `MediaRecorder`.** `MediaRecorder` writes corrupted MP4 box headers on audio focus loss (phone call interruption). `AudioRecord` can pause and resume cleanly. Encode to AAC via `MediaCodec` before upload. + +3. **Android 14+: `foregroundServiceType="microphone"` in manifest AND `FOREGROUND_SERVICE_MICROPHONE` permission (Phase 3 only).** Omitting either silently denies mic access — no crash, no audio. The foreground service must be running in the foreground *before* the app is backgrounded; it cannot be started from the background (Android 12+). + +4. **Gate Whisper uploads on word count.** Whisper hallucinate "Thank you." and similar tokens on silent/near-silent audio (confirmed, tracked in openai/whisper discussion #1606, ICML 2025 Calm-Whisper). Rule: if `transcript.split().size < 10`, treat as `TranscriptResult.Empty` and do not call the LLM. A VAD gate before upload is the deeper fix. + +5. **`[[link]]` hallucination in v1: do not pass the graph page index to the LLM.** Instead, constrain the system prompt: "Do NOT create `[[wiki links]]` unless the exact page name was explicitly stated in the transcript." Always append the raw transcript below the formatted output (collapsible) so the user can verify what was captured vs what the LLM produced. + +6. **Microphone permission: request before recording, fail fast on denial.** Android: wrap `AudioRecord.startRecording()` in `try/catch(SecurityException)`. iOS: check `AVAudioSession.recordPermission` before configuring the session — if `.denied`, navigate to Settings. Never call `AVAudioEngine.start()` without confirmed `.granted`. + +7. **`VOICE_COMMUNICATION` audio source on Android.** Use this `AudioSource` constant (not `DEFAULT`) when constructing `AudioRecord`. It applies system-level noise cancellation and echo suppression, which improves Whisper accuracy significantly in driving/ambient-noise scenarios. + +8. **Temp file cleanup in `finally` block.** The audio temp file must be deleted after `transcribe()` returns regardless of success or failure. A leaked `.m4a` file at ~1 MB/min will fill device storage. `VoiceCaptureViewModel` owns cleanup, not the platform recorder. + +--- + +## Open Questions Before Committing + +These require short implementation spikes before the ADRs can be finalized: + +1. **ML Kit GenAI STT reliability for 5–10 minute recordings.** The API provides streaming partial + final results, not batch. Does it handle long sessions stably, or does it require chunking like `SFSpeechRecognizer`? A 15-minute spike recording on a Pixel 9 would answer this. + +2. **FoundationModels structured output for Logseq format.** Does `FoundationModels` produce well-structured outliner output with a simple system prompt on first try? The formatting constraint (`- bullets`, 2-space indent, `[[links]]` only for named entities) is strict. A spike with 3–5 sample transcripts would establish whether zero-shot works or whether guided generation / few-shot is required. + +3. **Optimal system prompt for Logseq outliner format.** No tool surveyed produces Logseq `- item\n [[link]]` syntax natively. The prompt template needs iteration: top-level bullets for main topics, 2-space-indented sub-bullets, `[[Entity]]` only for proper nouns explicitly named in speech, no preamble, no trailing summary. Spike: 10 diverse transcript samples → measure format compliance rate. + +4. **`StelekitApp` parameter threading strategy.** `sttProvider` and `llmProvider` would be the 3rd and 4th optional parameters added following `urlFetcher`. Consider bundling them in a `VoicePipelineConfig` data class before threading further grows the constructor signature. + +5. **iOS bottom bar existence.** Does `PlatformBottomBar` have an `iosMain` actual, or does it share the Android composable? The mic button must appear on both platforms. Confirm before designing the `VoiceCaptureButton` placement. + +--- + +## Sources + +### Web-Search-Verified Claims + +All findings below are confirmed by live web search performed 2026-04-18. + +**ML Kit GenAI Speech Recognition (Android)** +- [GenAI Speech Recognition API — Google for Developers](https://developers.google.com/ml-kit/genai/speech-recognition/android) +- [Android Developers Blog: On-device GenAI APIs with ML Kit (May 2025)](https://android-developers.googleblog.com/2025/05/on-device-gen-ai-apis-ml-kit-gemini-nano.html) +- [ML Kit GenAI APIs — Android Developers](https://developer.android.com/ai/gemini-nano/ml-kit-genai) + +**`createOnDeviceSpeechRecognizer` introduced at API 31 (not 33)** +- [SpeechRecognizer — Android Developers](https://developer.android.com/reference/android/speech/SpeechRecognizer) + +**Apple FoundationModels framework (WWDC 2025)** +- [Foundation Models — Apple Developer Documentation](https://developer.apple.com/documentation/FoundationModels) +- [Apple's Foundation Models framework — Apple Newsroom (Sept 2025)](https://www.apple.com/newsroom/2025/09/apples-foundation-models-framework-unlocks-new-intelligent-app-experiences/) +- [WWDC 2025 Session 301 — Deep dive into Foundation Models](https://developer.apple.com/videos/play/wwdc2025/301/) + +**WhisperKit production-ready (ICML 2025)** +- [argmaxinc/WhisperKit — GitHub](https://github.com/argmaxinc/WhisperKit) + +**openai-kotlin v4.1.0 — Ktor 3.x compatible** +- [Issue #411: ktor 3.x — aallam/openai-kotlin](https://github.com/aallam/openai-kotlin/issues/411) +- [Releases — aallam/openai-kotlin](https://github.com/aallam/openai-kotlin/releases) + +**Whisper API accepted formats (m4a confirmed)** +- [Audio API FAQ — OpenAI Help Center](https://help.openai.com/en/articles/7031512-whisper-audio-api-faq) + +**Whisper API pricing ($0.006/min whisper-1, $0.003/min gpt-4o-mini-transcribe)** +- [OpenAI API Pricing](https://openai.com/api/pricing/) + +**Whisper hallucination on silence (confirmed, unresolved upstream)** +- [openai/whisper Discussion #1606](https://github.com/openai/whisper/discussions/1606) +- [Calm-Whisper (arXiv 2025)](https://arxiv.org/html/2505.12969v1) + +**Android foregroundServiceType="microphone" requirements** +- [Foreground service types are required — Android 14 — Android Developers](https://developer.android.com/about/versions/14/changes/fgs-types-required) +- [Foreground service types — Android Developers](https://developer.android.com/develop/background-work/services/fgs/service-types) + +**moko-permissions v0.20.1 — CMP compatible** +- [icerockdev/moko-permissions — GitHub](https://github.com/icerockdev/moko-permissions) + +**koog (JetBrains) — KMP confirmed, pre-1.0** +- [JetBrains/koog — GitHub](https://github.com/JetBrains/koog) + +### Prior Art + +- **Reflect.app** — lock screen widget + auto-append to daily note is the closest prior art. Validates the daily note auto-append UX. +- **AudioPen** — "what I meant to say" LLM synthesis model (rewrite, not transcribe). AudioPen proves users prefer formatted output over verbatim transcript. +- **Otter.ai** — two-phase display (live transcript during recording + formatted summary after) is the best system-state UX. Adopt for Phase 2. +- **NotelyVoice (OSS, Compose Multiplatform)** — chunked overlap transcription for long recordings is solved in KMP. Reference for chunking implementation. +- **ADR-002 (this codebase)** — `TopicEnricher` seam pattern is the template for all three provider interfaces in this feature.