feat(voice): Story 1 — Android voice capture → Whisper STT → journal insert - #2
Merged
Merged
Conversation
…insert End-to-end voice pipeline for Android: tap mic → record .m4a via AudioRecord + MediaCodec → transcribe via Whisper API → append timestamped block to today's journal. All three pipeline seams (AudioRecorder, SpeechToTextProvider, LlmFormatterProvider) are injected via VoicePipelineConfig with NoOp defaults, matching the existing UrlFetcher/TopicEnricher pattern. Key changes: - commonMain/voice/: AudioRecorder, SpeechToTextProvider, LlmFormatterProvider interfaces + sealed result types, VoicePipelineConfig, VoiceCaptureState, VoiceCaptureViewModel (coroutine state machine), WhisperSpeechToTextProvider - androidMain/voice/: AndroidAudioRecorder (AudioRecord + MediaCodec AAC + MediaMuxer + AudioFocus; unified drainEncoder; coroutine-cancellation-aware record loop) - commonMain/ui/components/: VoiceCaptureButton FAB with per-state UX (pulse animation, spinner + contentDescription, error message chip, truncation warning) - PlatformBottomBar.android.kt: split nav items 2+2 with center Spacer gap; FAB offset y=-28dp to sit on nav bar edge without overlapping any nav item - App.kt: VoiceCaptureViewModel wired into GraphContent; PlatformBackHandler for Recording/Transcribing/Formatting states calls cancel() - JournalService: appendToToday() for direct block append - androidApp: BuildConfig.WHISPER_API_KEY wired from gradle.properties/CI secret - 20 tests: 13 VoiceCaptureViewModel (state machine, cancel, cleanup) + 7 WhisperSpeechToTextProvider (MockEngine HTTP scenarios) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces the first end-to-end “voice capture → STT → journal append” pipeline for the Android app, implemented as injected KMP seams (AudioRecorder, SpeechToTextProvider, LlmFormatterProvider) and driven from a new bottom-bar voice FAB state machine.
Changes:
- Add KMP voice pipeline interfaces/config +
VoiceCaptureViewModel+ UI (VoiceCaptureButton) and wire them intoStelekitApp/GraphContent. - Implement Android recording to AAC-in-MP4 (
.m4a) viaAudioRecord+MediaCodec+MediaMuxer, and add a Whisper STT provider using Ktor multipart upload. - Add
JournalService.appendToToday()for appending a new block to today’s journal; add Android build config + permissions for microphone access; add unit tests for STT + ViewModel behavior.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| project_plans/mobile-voice-mode/research/synthesis.md | Research synthesis and recommended architecture for mobile voice mode. |
| project_plans/mobile-voice-mode/research/stack.md | Stack survey and updated web-search-verified findings for STT/LLM options. |
| project_plans/mobile-voice-mode/research/research_plan.md | Research execution plan for stack/features/architecture/pitfalls. |
| project_plans/mobile-voice-mode/research/pitfalls.md | Consolidated pitfalls/failure modes and mitigations for voice capture. |
| project_plans/mobile-voice-mode/research/features.md | UX/product survey and lessons learned for voice-to-notes workflows. |
| project_plans/mobile-voice-mode/research/architecture.md | Proposed KMP seam patterns and wiring strategy aligned with existing app patterns. |
| project_plans/mobile-voice-mode/requirements.md | Draft requirements and scope constraints for voice mode. |
| project_plans/mobile-voice-mode/decisions/ADR-005-voice-capture-ui-state-machine.md | Proposed UI state machine and bottom bar slot-based integration approach. |
| project_plans/mobile-voice-mode/decisions/ADR-004-plugin-registration.md | Proposed provider wiring via VoicePipelineConfig with NoOp defaults. |
| project_plans/mobile-voice-mode/decisions/ADR-003-llm-formatter-provider-interface.md | Proposed LLM formatting interface + result types and prompt strategy. |
| project_plans/mobile-voice-mode/decisions/ADR-002-stt-provider-interface.md | Proposed STT interface + result types and tiering strategy. |
| project_plans/mobile-voice-mode/decisions/ADR-001-audio-capture-adapter.md | Proposed audio capture interface shape and Android/iOS implementation notes. |
| docs/tasks/mobile-voice-mode.md | Implementation plan and story breakdown for the feature rollout. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/AudioRecorder.kt | New audio recorder interface + temp file wrapper and NoOp implementation. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/SpeechToTextProvider.kt | New STT provider interface + sealed result types + NoOp provider. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/LlmFormatterProvider.kt | New LLM formatter interface + sealed result types + NoOp provider. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureState.kt | New VoiceCaptureState + pipeline stage enum for UI orchestration. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoicePipelineConfig.kt | Bundles pipeline seams, default prompt, and word-count guard threshold. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModel.kt | Orchestrates recording → STT → (LLM) → journal insertion with cancellation/cleanup. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProvider.kt | Ktor-based Whisper STT implementation (multipart upload + status mapping). |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/components/VoiceCaptureButton.kt | New FAB UI that reflects capture/transcription/formatting/done/error states. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.kt | Adds a voiceCaptureButton composable slot to the bottom bar expect API. |
| kmp/src/androidMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.android.kt | Renders a center FAB gap and positions the voice capture button above the nav bar. |
| kmp/src/jvmMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.jvm.kt | Updates actual signature to accept the new voiceCaptureButton slot. |
| kmp/src/jsMain/kotlin/dev/stapler/stelekit/ui/PlatformBottomBar.js.kt | Updates actual signature to accept the new voiceCaptureButton slot. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/ui/App.kt | Wires VoiceCaptureViewModel into GraphContent, lifecycle cancel, and bottom bar slot. |
| kmp/src/commonMain/kotlin/dev/stapler/stelekit/repository/JournalService.kt | Adds appendToToday() helper for appending a new block to today’s journal. |
| kmp/src/androidMain/kotlin/dev/stapler/stelekit/voice/AndroidAudioRecorder.kt | Android recorder implementation using AudioRecord + MediaCodec + MediaMuxer. |
| kmp/src/jvmTest/kotlin/dev/stapler/stelekit/voice/WhisperSpeechToTextProviderTest.kt | New tests validating Whisper provider HTTP behavior and error mapping. |
| kmp/src/businessTest/kotlin/dev/stapler/stelekit/voice/VoiceCaptureViewModelTest.kt | New tests validating ViewModel state transitions, cancellation, and cleanup behavior. |
| androidApp/src/main/kotlin/dev/stapler/stelekit/MainActivity.kt | Wires Android providers via VoicePipelineConfig using BuildConfig.WHISPER_API_KEY. |
| androidApp/src/main/AndroidManifest.xml | Adds RECORD_AUDIO permission required for microphone capture. |
| androidApp/build.gradle.kts | Adds BuildConfig.WHISPER_API_KEY injection and enables BuildConfig generation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+48
to
+58
| val focusRequest = AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN_TRANSIENT) | ||
| .setOnAudioFocusChangeListener { change -> | ||
| when (change) { | ||
| AudioManager.AUDIOFOCUS_LOSS_TRANSIENT -> pauseRequested = true | ||
| AudioManager.AUDIOFOCUS_GAIN -> pauseRequested = false | ||
| AudioManager.AUDIOFOCUS_LOSS -> stopRequested = true | ||
| } | ||
| } | ||
| .build() | ||
| audioManager.requestAudioFocus(focusRequest) | ||
|
|
There was a problem hiding this comment.
AudioFocusRequest is API 26+, but androidApp minSdk is 24. As written, constructing AudioFocusRequest.Builder(...) will crash on API 24/25 devices. Use the pre-26 AudioManager.requestAudioFocus(listener, streamType, gain) / abandonAudioFocus(listener) path when Build.VERSION.SDK_INT < 26 (or gate the whole recorder behind API 26+ if that’s acceptable).
Owner
Author
There was a problem hiding this comment.
Fixed in the latest commit — added an API level check: AudioFocusRequest.Builder is only used on API 26+; API 24/25 falls back to the deprecated requestAudioFocus(listener, stream, hint) overload. A shared abandonAudioFocus() helper handles both paths across all three abandon sites.
…ailure `import androidx.compose.foundation.layout.weight` accesses an internal Kotlin property. `Modifier.weight()` inside NavigationBar is available via RowScope receiver without an explicit import. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AudioFocusRequest.Builder requires API 26 (O) but minSdk is 24. Use the deprecated requestAudioFocus(listener, stream, hint) overload on API < 26 to avoid a crash on Android 7.x devices. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Transcribing/Formatting FAB now uses enabled=false so it is clearly non-interactive (cancel is available via the back button) - Error FAB contentDescription now includes state.message so screen reader users hear what went wrong - Fix fully-qualified Block reference in JournalService.appendToToday - Rethrow CancellationException in WhisperSpeechToTextProvider so coroutine cancellation is not swallowed as NetworkError Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… field Material 3 FloatingActionButton does not have an `enabled` parameter. The spinner icon and disabled-looking appearance already communicate that the button is non-interactive during processing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- VoicePipelineConfig: data class → class (interface fields break structural equality) - VoiceCaptureViewModel: add resetToIdle() for Done→Idle auto-reset; add 10k-char transcript guard that propagates isLikelyTruncated to Done state - App: wire onAutoReset to resetToIdle() instead of dismissError() - PlatformBottomBar: nav icon contentDescription = item.label (was null) - Tests: 4 new ViewModel cases (Formatting state, LLM fallback, 9-word/10-word boundary); HTTP 500 case for WhisperSpeechToTextProvider; Roborazzi screenshot tests for all 6 VoiceCaptureButton states Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Updates TODO.md with a Mobile Voice Mode section reflecting PR #2 status (Story 1 all tasks done, CI passing). Updates docs/tasks/mobile-voice-mode.md with [STATUS: COMPLETE] on all T1.1–T1.7 tasks and [STATUS: READY TO BEGIN] on Story 2. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
T2.1 ClaudeLlmFormatterProvider: Ktor POST to Anthropic Messages API,
model claude-haiku-4-5, dynamic max_tokens, truncation detection,
CancellationException propagation, withDefaults() factory.
T2.2 OpenAiLlmFormatterProvider: Ktor POST to OpenAI chat completions,
configurable baseUrl for OpenAI-compatible endpoints (OpenRouter etc),
same error mapping and truncation detection as Claude provider.
T2.3 VoiceSettings: wraps PlatformSettings for whisper/anthropic/openai
API key storage. buildVoicePipeline() helper assembles VoicePipelineConfig
from stored keys — Anthropic preferred over OpenAI when both present.
T2.4 VoiceCaptureSettings composable: masked API key fields, Save button,
inline confirmation. SettingsDialog gains VOICE category (Mic icon).
StelekitApp/GraphContent/GraphDialogLayer threaded with voiceSettings
and onRebuildVoicePipeline params.
T2.5 MainActivity: removes build-time WHISPER_API_KEY wiring; pipeline is
now built from VoiceSettings at runtime and rebuilt on settings save.
T2.6 VoiceNoteBlockFormatTest: pure function tests for buildVoiceNoteBlock
(header format, formatted text content, BEGIN_QUOTE/END_QUOTE structure,
multiline indentation, HH:mm timestamp regex). buildVoiceNoteBlock is
now internal.
Tests: 8 new Claude provider tests, 8 new OpenAI provider tests, 6 new
block format tests. All 895+ JVM tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Recording state FAB now scales to the real-time RMS amplitude emitted by AndroidAudioRecorder.amplitudeFlow. Amplitude [0,1] is mapped to scale [1.0, 1.35] via an Animatable with an 80ms spring so the visual response is smooth rather than jittery. Falls back to the existing fixed 600ms sinusoidal pulse when amplitudeFlow is null (NoOpAudioRecorder, future IoS recorder until T3.2 lands). VoiceCaptureButton gains an optional amplitudeFlow: Flow<Float>? = null parameter. App.kt passes voicePipeline.audioRecorder.amplitudeFlow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Security: - Encrypt API keys with EncryptedSharedPreferences (AES256-GCM via Android Keystore) - Remove hardcoded WHISPER_API_KEY BuildConfig field - Set android:allowBackup=false to prevent key extraction via adb backup Architecture / DRY: - Extract LlmProviderSupport object (estimateMaxTokens, detectTruncation, mapHttpError) - Move buildVoicePipeline to VoicePipelineFactory; show actionable first-run guidance - Extract processTranscript() from startPipeline() in VoiceCaptureViewModel - Extract setupMediaCodec/setupMediaMuxer/computeRms helpers in AndroidAudioRecorder Code quality: - Replace all magic literals with named constants (AMPLITUDE_SCALE_RANGE, DONE_AUTO_RESET_MS, etc.) - Split IOException vs Exception catch in LLM providers (network vs parse errors) - Add Log.w in AndroidAudioRecorder finally runCatching blocks - Add println observability when LLM formatting falls back to raw transcript UX: - Mark Transcribing/Formatting FABs as semantically disabled - Done FAB is now tappable (user-dismissible); auto-resets after 5 s - Add LLM enable/disable toggle to VoiceCaptureSettings Tests: - Add temp-file-deleted-on-STT-failure test - Add LLM ApiError fallback-to-raw-transcript test - Add transcript-truncation-before-LLM test - Fix VoiceNoteBlockFormatTest to share InMemoryBlockRepository; use firstOrNull to skip blank initialBlock Docs: - ADR-006: VoicePipelineConfig is class not data class (interface fields break structural equality) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GraphDialogLayer signature: use FileSystem interface (from main) while keeping voiceSettings and onRebuildVoicePipeline params (from this branch). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…commonMain - PlatformAudioFile: @JvmInline value class → data class (annotation is JVM-only) - Claude/OpenAiLlmFormatterProvider: java.io.IOException → io.ktor.utils.io.errors.IOException (Ktor's KMP-compatible IOException typealias; java.io is not available in commonMain) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tstapler
added a commit
that referenced
this pull request
Sep 1, 2026
Closes Epic 8.2 of the android-git-saf-shadow-worktree plan: zero test coverage of AndroidGitRepository against a real Robolectric shadow worktree (finding #8). - AndroidGitRepositoryShadowWorktreeTest: init/stageSubdir/commit/status end-to-end; merge()'s conflict-path SAF mapping (regression guard for commit 61b689fa61); checkoutFile() SAF write-back round trip; resolveForJGit() literal regression test (validation.md Gap #2). - GitPathResolverChainTest: shadowWorktreeFor() direct-access-wins/ caching/per-subpath-key decision logic; sweepOrphans() stale-vs-fresh- vs-markerless orphan sweep. - AndroidGitRepositoryStorageGuardTest: StatFs pre-clone storage guard, error and happy path (validation.md Gap #3), using Robolectric's ShadowStatFs (confirmed via javap against shadows-framework:4.16 — registerStats(path, totalBlocks, freeBlocks, availableBlocks), BLOCK_SIZE=4096). Marks AndroidGitRepository.shadowWorktreeFor/resolveForJGit internal (was private) for direct test access — no other production logic changed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYKvFLTDH53wzEkXzaaajF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.m4arecording (AudioRecord + MediaCodec AAC) → Whisper STT → timestamped block appended to today's journalAudioRecorder,SpeechToTextProvider,LlmFormatterProvider) are injected viaVoicePipelineConfigwithNoOpdefaults — same pattern asUrlFetcher/TopicEnricher; desktop/web are unaffectedVoiceCaptureButtonFAB in the Android bottom bar drives the full state machine (Idle → Recording → Transcribing → Formatting → Done/Error)What's in scope (Story 1)
commonMain/voice/: interfaces, sealed result types,VoiceCaptureViewModel,WhisperSpeechToTextProviderandroidMain/voice/:AndroidAudioRecorder(AudioRecord + MediaCodec + MediaMuxer + AudioFocus)VoiceCaptureButtoncomposable: pulse animation, spinner with a11y labels, error message chip, truncation warningPlatformBottomBar.android.kt: nav items split 2+2 with a centre gap; FAB offsety = -28dpso it sits above the bar without overlapping any nav itemApp.kt:VoiceCaptureViewModelwired intoGraphContent;PlatformBackHandlercancels recording on back press; lifecycle observer cancels on pause/stopJournalService.appendToToday()for block appendBuildConfig.WHISPER_API_KEYsourced fromgradle.properties/ CI secret (replaces reflection hack)VoiceCaptureViewModelTest+ 7WhisperSpeechToTextProviderTestOut of scope (Stories 2 & 3)
ClaudeLlmFormatterProvider,OpenAiLlmFormatterProvider) — Story 2Test plan
./gradlew :kmp:jvmTest— all 20 voice tests + full suite passON_PAUSEcancels cleanlyWHISPER_API_KEYingradle.properties→NoOpSpeechToTextProviderused (no crash)🤖 Generated with Claude Code