search overhaul stage 1: SQLite FTS query parser, bm25 weights, diacritics, pagination - #29
Merged
Merged
Conversation
This was referenced Jul 24, 2026
…nation Replace the OR-of-prefixes FTS builder with a real query parser: AND by default, "quoted phrases", prefix*, explicit OR, and NOT/leading-dash negation. AND-first / OR-fallback keeps precise multi-term queries precise while natural-language questions (ask/context.pack) retain recall instead of returning nothing. Malformed MATCH expressions degrade to zero keyword hits rather than throwing. Weight bm25 columns so title/source_uri outrank body. Rebuild chunks_fts under a new schema v9 migration with the diacritic-folding tokenizer (porter unicode61 remove_diacritics 2), backfilled losslessly from the existing stored FTS rows; the rebuild runs in a transaction so a crash mid-migration rolls back to the old index instead of leaving an empty one. Add offset pagination to hybridSearch/hybridSearchItems with a widened fetch window for stable, non-overlapping pages. Includes the shared parity fixtures corpus and flips the search-parity suite to assert AND/phrase/NOT/prefix/bm25/diacritic/pagination behavior (fails on the old builder, passes now); schema-version expectations bumped 8 -> 9 across affected suites; regenerated bin/dist artifacts.
andrei-hasna
force-pushed
the
feat/search-stage-1-sqlite-fts
branch
from
July 24, 2026 15:10
1f357e3 to
2ced352
Compare
andrei-hasna
changed the base branch from
feat/search-stage-0-parity-scaffolding
to
main
July 24, 2026 15:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 of the staged search overhaul. Stacked on #28 (review/merge that first).
What
src/search.ts): AND by default,"quoted phrases",prefix*, explicitOR,NOT/ leading--negation — replacesterms.map(t=>+"${t}*"+).join(' OR ').ask()/context.pack()questions) the query degrades to OR recall instead of returning nothing.title/source_urioutrank body.src/knowledge-db.ts): rebuildchunks_ftswith the diacritic-folding tokenizer (porter unicode61 remove_diacritics 2), backfilled losslessly from the existing stored FTS rows. Idempotent, guarded by live-tokenizer inspection.hybridSearchwith a widened fetch window (stable, non-overlapping pages). MalformedMATCHdegrades to zero keyword hits instead of throwing.Tests
tests/search-parity.test.tsflips the Stage 0 characterization assertions to AND / phrase / NOT / prefix / bm25 / diacritic / pagination — these fail on the old builder and pass now. Schema-version expectations bumped 8 -> 9 across affected suites.Risk / compatibility
sdk.test.ts).🤖 Generated with Claude Code