search overhaul stage 2: Postgres full-text parity (cloud 'returns nothing' fix) - #30
Closed
andrei-hasna wants to merge 3 commits into
Closed
Conversation
…ion reconcile Add a shared backend-agnostic search corpus and characterization tests that pin the current OR-of-prefixes / no-phrase SQLite keyword behavior, so the follow-up stages produce a reviewable behavior diff. Reconcile the package version to 0.2.87 (ahead of npm latest 0.2.86; main had drifted to 0.2.81) and add a CHANGELOG documenting the staged plan. No runtime behavior change.
…tics, pagination Replace the OR-of-prefixes FTS builder with a real query parser: AND by default, "quoted phrases", prefix*, explicit OR, and NOT/leading-dash negation. Add an AND-first / OR-fallback strategy so precise multi-term queries stay precise while natural-language questions (ask/context.pack) retain recall instead of returning nothing. Weight bm25 columns so title/source_uri outrank body. Rebuild chunks_fts under a new schema v9 migration with the diacritic-folding tokenizer (porter unicode61 remove_diacritics 2), backfilled losslessly from the existing stored FTS rows. Add offset pagination to hybridSearch with a widened fetch window for stable, non-overlapping pages. Malformed MATCH expressions now degrade to zero keyword hits rather than throwing. Tests: flip the Stage 0 characterization suite to assert AND/phrase/NOT/ prefix/bm25/diacritic/pagination behavior (fail on the old builder, pass now); bump schema-version expectations 8 -> 9 across affected suites.
Replace the cloud NoteRepo.list ILIKE-substring + created_at-recency path with weighted full-text search: a tsvector generated column (title=A, content=B) plus a GIN index, queried via websearch_to_tsquery and ranked by ts_rank_cd (created_at tiebreak). This fixes the 'cloud returns nothing' divergence where multi-term / word-order-varying queries matched no substring and results were ordered by recency instead of relevance. Postgres migrations are appended to PG_MIGRATIONS (ids are index-derived, so never inserted mid-array) and are idempotent. Add an in-process Postgres parity suite (@electric-sql/pglite devDependency) that runs the real NoteRepo against the real migrations and asserts word-order independence, relevance-over-recency, phrase adjacency, correct totals, and sqlite-vs-pg equivalence over the shared corpus.
andrei-hasna
force-pushed
the
feat/search-stage-1-sqlite-fts
branch
from
July 24, 2026 15:10
1f357e3 to
2ced352
Compare
Contributor
Author
|
Resolved via #32 (merged to This PR was auto-closed by GitHub when its base branch I rebased the identical Stage-2 change fresh onto current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 2 — the top-priority correctness fix. Stacked on #29 (which is on #28).
Problem
Cloud
NoteRepo.listmatched withtitle/content ILIKE '%q%'and ordered bycreated_at DESC. That substring path misses multi-term / word-order-varying queries (returning near-empty results) and never ranks by relevance — so hosted search diverges materially from local.Fix (
src/serve.ts,src/db/pg-migrations.ts)tsvectorgenerated column onknowledge_items(title = A, content = B) + GIN index.NoteRepo.listnow filters withsearch_vector @@ websearch_to_tsquery('english', $n)and orders byts_rank_cd(...)(created_at tiebreak).websearch_to_tsquerygives users implicit-AND, phrases, and OR/negation.PG_MIGRATIONS(ids are index-derived — never insert mid-array) and are idempotent.Tests
tests/search-pg-parity.test.tsruns the realNoteRepoagainst an in-process Postgres (@electric-sql/pglite, devDependency) with the real migrations applied. It asserts:"beta alpha"findsalpha beta ...— empty under the old ILIKE),totalreflects the FTS predicate,These fail on the pre-Stage-2 implementation and pass now.
🤖 Generated with Claude Code