refactor(memory): migrate vector storage to resilient v2 - #1960
Merged
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughThe PR introduces Memory Vector Store Format v2 with exact scans, staged publication, legacy migration, quarantine recovery, and native crash validation. Cleanup APIs now return restart-pending status, which propagates through routes, clients, UI notifications, and localized strings. ChangesMemory vector store v2 and migration
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/main/presenter/memoryPresenter/infra/embeddingPipeline.ts (1)
843-861: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winRemove abandoned agents from
embeddingWarmupAgents.
abandonAgentclears the warmup promise map but never removesagentIdfromembeddingWarmupAgents.getAgentInFlightcan therefore continue discovering the shared warmup as work belonging to the deleted agent, defeating the abandon path and retaining stale agent associations. Remove the ID from each set and delete empty sets.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/presenter/memoryPresenter/infra/embeddingPipeline.ts` around lines 843 - 861, Update abandonAgent to remove agentId from every set in embeddingWarmupAgents, deleting any sets left empty after removal. Keep the existing cleanup behavior intact and ensure no stale agent association remains discoverable through getAgentInFlight.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/architecture/memory-vector-store-v2/plan.md`:
- Line 34: In the authoritative store description, replace “cleanup afterwards”
with the American-English “cleanup afterward,” preserving the surrounding
wording.
In `@docs/issues/memory-vector-store-corruption-hang/spec.md`:
- Around line 30-34: Update the fenced log block in the issue specification to
declare the text language on its opening fence, preserving the existing log
content unchanged.
In `@src/main/presenter/memoryPresenter/infra/embeddingPipeline.ts`:
- Line 48: Update runReindex to inspect the VectorStoreCleanupDisposition
returned by resetAgentStore and immediately stop when it is 'pending-restart'.
Ensure no embeddings are drained, emitted, or used to warm the store after a
deferred reset, while preserving normal reindexing for other dispositions.
In `@src/main/presenter/memoryPresenter/ports.ts`:
- Line 378: The markVectorStoreQuarantined contract in
src/main/presenter/memoryPresenter/ports.ts at lines 378-378 must remain
required. Update the default factory wiring in
src/main/presenter/memoryPresenter/index.ts at lines 142-143 to always provide
this callback, using an explicit no-op for test adapters that do not persist
quarantine markers.
In `@src/main/presenter/memoryPresenter/services/managementService.ts`:
- Around line 645-670: Update clearMemories so VectorStoreQuarantineMarkerError
from resetAgentStore does not bypass post-clear bookkeeping: after clearByAgent
commits removals, still emit the change and clear consolidation cooldown when
applicable, then rethrow the marker error. Ensure the caller’s diagnostics
cleanup also executes before propagation, while preserving existing handling for
non-marker reset failures and return behavior for successful clears.
---
Outside diff comments:
In `@src/main/presenter/memoryPresenter/infra/embeddingPipeline.ts`:
- Around line 843-861: Update abandonAgent to remove agentId from every set in
embeddingWarmupAgents, deleting any sets left empty after removal. Keep the
existing cleanup behavior intact and ensure no stale agent association remains
discoverable through getAgentInFlight.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 04832f2d-04b2-4ffc-aa6f-3edbcdb2851c
📒 Files selected for processing (65)
.github/workflows/prcheck.yml.github/workflows/windows-arm64-e2e.ymldocs/architecture/memory-vector-store-v2/plan.mddocs/architecture/memory-vector-store-v2/spec.mddocs/architecture/memory-vector-store-v2/tasks.mddocs/issues/memory-vector-store-corruption-hang/spec.mdsrc/main/presenter/configPresenter/index.tssrc/main/presenter/index.tssrc/main/presenter/memoryPresenter/domain/types.tssrc/main/presenter/memoryPresenter/index.tssrc/main/presenter/memoryPresenter/infra/embeddingPipeline.tssrc/main/presenter/memoryPresenter/infra/legacyV1Reader.tssrc/main/presenter/memoryPresenter/infra/legacyVssLoader.tssrc/main/presenter/memoryPresenter/infra/memoryVectorStore.tssrc/main/presenter/memoryPresenter/infra/memoryVectorStoreFormat.tssrc/main/presenter/memoryPresenter/infra/vectorStoreErrors.tssrc/main/presenter/memoryPresenter/infra/vectorStoreManager.tssrc/main/presenter/memoryPresenter/ports.tssrc/main/presenter/memoryPresenter/runtimeConstants.tssrc/main/presenter/memoryPresenter/services/managementService.tssrc/main/presenter/memoryPresenter/types.tssrc/main/routes/config/configRouteHandler.tssrc/main/routes/index.tssrc/renderer/api/ConfigClient.tssrc/renderer/api/MemoryClient.tssrc/renderer/settings/components/DeepChatAgentsSettings.vuesrc/renderer/settings/components/MemoryDiagnosticsPanel.vuesrc/renderer/src/i18n/da-DK/settings.jsonsrc/renderer/src/i18n/de-DE/settings.jsonsrc/renderer/src/i18n/en-US/settings.jsonsrc/renderer/src/i18n/es-ES/settings.jsonsrc/renderer/src/i18n/fa-IR/settings.jsonsrc/renderer/src/i18n/fr-FR/settings.jsonsrc/renderer/src/i18n/he-IL/settings.jsonsrc/renderer/src/i18n/id-ID/settings.jsonsrc/renderer/src/i18n/it-IT/settings.jsonsrc/renderer/src/i18n/ja-JP/settings.jsonsrc/renderer/src/i18n/ko-KR/settings.jsonsrc/renderer/src/i18n/ms-MY/settings.jsonsrc/renderer/src/i18n/pl-PL/settings.jsonsrc/renderer/src/i18n/pt-BR/settings.jsonsrc/renderer/src/i18n/ru-RU/settings.jsonsrc/renderer/src/i18n/tr-TR/settings.jsonsrc/renderer/src/i18n/vi-VN/settings.jsonsrc/renderer/src/i18n/zh-CN/settings.jsonsrc/renderer/src/i18n/zh-HK/settings.jsonsrc/renderer/src/i18n/zh-TW/settings.jsonsrc/shared/contracts/routes/config.routes.tssrc/shared/contracts/routes/memory.routes.tssrc/shared/types/presenters/core.presenter.d.tstest/fixtures/memory-vector-store-v2/crash-worker.mjstest/main/presenter/configPresenter/deprecatedProviderCleanup.test.tstest/main/presenter/fakes/memoryFakes.tstest/main/presenter/fakes/memoryPresenterTestAdapter.tstest/main/presenter/memory/legacyV1Reader.test.tstest/main/presenter/memory/managementService.test.tstest/main/presenter/memory/serviceTestSupport.tstest/main/presenter/memory/vectorStoreManager.test.tstest/main/presenter/memoryVectorStore.test.tstest/main/presenter/memoryVectorStoreV2Native.test.tstest/main/routes/dispatcher.test.tstest/memory-test-scope.jsontest/renderer/api/clients.test.tstest/renderer/components/DeepChatAgentsSettings.test.tstest/renderer/components/MemoryDiagnosticsPanel.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces Memory Vector Store V2 to eliminate persistent HNSW corruption risks and make migration, publication, recovery, and quarantined cleanup converge safely across restarts.
Changes
format_version, embedding identity, dimensions, and row-count validation.EBUSY, orphan WAL, post-commit failure, and cleanup failure without retry loops.cleanupPendingRestartin the Settings UI.Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests