feat(search): index prose content for BM25 full-text search

[DeusData]

Closes #518. Closes #519.

Distilled in-house from #617 / #1778, with Co-authored-by: credit to @ShauryaaSharma, who designed this and carried it for two months. The reason for taking it in-house was scheduling on my side, not the quality of the work — explained on both threads.

What

nodes_fts gains a fifth column, body, backfilled from each node's existing properties.docstring. Two node kinds that never had a docstring now get one: markdown section bodies (#518) and top-level YAML/JSON description/summary/purpose promoted onto the Module node (#519). Section and Module stop being excluded from BM25 results.

No reindex

CBM_INDEX_FORMAT_VERSION is unchanged. nodes_fts is contentless — a derived index, not a source of truth — and nodes/edges gain no table and no column. Legacy 4-column databases keep working: the body column is detected at runtime and queries degrade to "no prose" rather than erroring. There are tests for exactly that.

Column weights: bm25(nodes_fts, 1.0, 1.0, 1.0, 1.0, 0.3)

The four identifier columns stay at parity; body sits at 0.3.

FTS5 computes f(qi,D) = Σ_c w_c · n(qi,c) — it applies column weights to per-column term frequency before the tf-saturation term. That is what Robertson, Zaragoza & Taylor (CIKM 2004) prescribe for multi-field BM25; the failure mode they warn about is per-field scores combined linearly afterwards, which FTS5 does not do. So weighting here is principled rather than a knob. The field-weighting literature is also consistent that body text takes the lowest field weight (typical title:body 3:1–10:1); 0.3 is the findability-favouring end of that range.

Known and accepted: FTS5 uses a single global |D|/avgdl rather than BM25F's per-field length normalisation, and b=0.75 is hardcoded. So populating body shifts avgdl and therefore moves every row's score, including bodiless ones — no weight setting can undo that, because w_c acts on tf, never on |D|. Measured: cbm's own index moves avgdl 22.68 → 28.76 tokens (+26.8%, 29.2% of nodes carry a docstring); a Kotlin corpus moves +4.9%. That works out to roughly +9.5% score for a bodiless row at average length. Only a separate prose table would avoid it, and that machinery was judged not worth it here.

The third write site

Main had grown a third INSERT INTO nodes_fts at pipeline_delta.c:528 that #617 predates. Left alone it would have written four columns into a five-column table, so body would land NULL for every node arriving via delta merge — the warm path most users hit — while a full reindex looked perfect.

Rather than edit three literals and hope, all write sites now route through one parameterized helper (store.c:421), and both bm25() query sites share a single BM25_WEIGHTS macro so ranking and reported score cannot desync.

Verification

• 967 passed, 6 skipped, 0 failed across store_search store_nodes store_edges store_bulk store_pragmas mcp pipeline extraction complexity. The 6 skips are pre-existing and identical before and after.
• Revert-check: forcing the body column off makes 9 tests fail, including the delta-path test at test_pipeline.c:12587. The green means the fix works, not that the tests are asleep.
• Covered: prose findable end-to-end; results and totals stay consistent across both query sites; an identifier match still outranks a prose-only match (i.e. the 0.3 weighting behaves); UTF-8-safe 500-byte cap; malformed properties_json survived; legacy 4-column table tolerated; and body indexed raw, not camelCase-split (cbm_camel_split applies to name only — splitting prose would mangle it).