memU: index-sargable content-hash dedup + hot-path indexes

[pufit]

Problem

SQLiteMemoryItemRepo.create_item_reinforce (memu-py 1.4.0) deduplicates every item the memorize pipeline persists with:

SELECT ... WHERE json_extract(extra, '$.content_hash') = :hash AND <scope> LIMIT 1

No index can serve this, so it is a full scan of the items table — and the table carries inline embedding JSON, so a scan reads essentially the whole DB file.

On a large corpus (26 GB / 728K items) this only held up while the file was in page cache. On 2026-08-07 other workloads evicted the cache and each lookup became ~90 minutes of serial 4 KiB reads at EBS latency (~1,200 IOPS, queue depth 1). The scan runs synchronously on the dedicated memU loop thread — asyncio cancellation cannot interrupt a running statement — so one cold lookup wedged the entire memorize lane: every memorize call and conversation sweep queued behind it, timed out at 300 s, and retried into the same wall. Symptom signature: memorize_file starting lines with no subsequent LLM steps, and 100% timed out after 300s failures with no "stuck on" step info.

Even in the healthy warm-cache state this scan is the dominant per-item cost of memorize.

Why not just CREATE INDEX

SQLAlchemy compiles func.json_extract(col, "$.content_hash") with the JSON path as a bound parameter (json_extract(extra, ?)), and SQLite never matches a parameterized expression against an expression index — the plan stays SCAN even with the index present (proved by test_bound_parameter_path_cannot_use_index). The query itself has to render the expression literally.

Fix

• _content_hash_reinforce — line-for-line port of the pinned upstream create_item_reinforce with one change: the dedup predicate is rendered as a literal SQL expression (literal_column("json_extract(extra, '$.content_hash')")); only the compared value stays a parameter. The _semantic_sqlite_reinforce fallthrough now calls the port, with a snapshot-protected fallback to the original on any exception (if the pinned upstream ever shifts, memorize degrades to the scan instead of failing).

• MemUBridge._ensure_sqlite_indexes — idempotent DDL after MemoryService init (tables exist by then):

  • ix_memu_memory_items_content_hash on json_extract(extra, '$.content_hash') → the dedup lookup becomes an index seek, immune to page-cache state.
  • ix_memu_memory_items_created_at → serves the event-date sweep (created_at >= :cutoff ORDER BY created_at DESC LIMIT :n in _resolve_event_dates_sync), which otherwise also scans.

  The first startup on an existing large corpus pays a one-time index build (logged with duration).

Tests

9 new tests: index DDL creation / idempotency / missing-table robustness; EXPLAIN QUERY PLAN proof that the literal path uses the index while the parameterized path scans; behavior + parity tests of the port against the pinned upstream implementation on a real repo (duplicate → reinforce, scope isolation, type/text separation, identical resulting rows).

tests/test_memu_bridge.py + tests/test_memorize_background.py + tests/test_xmemory_bridge.py: 151 passed.

Generated by Nerve