Rehash only the samples an older smda escaper hashed (#142)

[r0ny123]

Closes #142 (the "escaper change" gaps). Gaps 1 and 3 of that issue are already handled by mcrit 1.7.0 and maintainer PR #143; this covers the remaining one: a full recalculateMinHashes drops every band collection and rehashes the whole corpus, so an escaper change costs a full outage of the index.

What changes

• Each sample now records which smda version escaped its minhashes (minhash_smda_version, set by updateMinHashesForSample and by the full recalculation).
• /status reports minhash_compatibility_threshold (smda's ESCAPER_DOWNWARD_COMPATIBILITY, falling back to its version) and num_samples_with_stale_minhashes. Samples with no recorded version count as stale, so an existing database shows its whole corpus as stale until one repair or full recalculation has run.
• POST /repair_minhashes (McritClient repairMinHashes()) schedules a job that walks only the stale samples, pulls their band entries, rehashes them one sample at a time and records the running version. The index keeps serving throughout, and a killed run costs one sample. The job result is a report with the threshold, smda version, samples stale/repaired and functions dropped/rehashed.
• Storage: deleteMinHashesForSample, setMinHashVersionForSamples, getSamplesWithStaleMinHashes, countSamplesWithStaleMinHashes on both backends. deleteSample shares the new _pullBandEntries helper. The Mongo stale query compares the few distinct recorded versions instead of scanning every sample document. MemoryStorage gains the deleteAllMinHashes the interface declares (its absence made the local-queue full recalculation fail silently).

Verification

• New tests/testMinHashRepair.py (worker job on the local queue, route, client) and storage tests for both backends; full suite: 211 passed.
• ruff check/format and ty clean.
• Live verification against a running MongoDB instance is in the comment below.

[r0ny123]

Live verification on a MongoDB 7 instance (4 samples, 418 hashed functions, 8360 band entries) with the branch deployed to server and worker:

step	status stale count	job report
before (no version recorded on any sample)	4
repairMinHashes()	0	stale 4, repaired 4, dropped 418, rehashed 418 (7.9 s)
plant minhash_smda_version: 4.0.0 on sample 3	1
repairMinHashes()	0	stale 1, repaired 1, dropped 24, rehashed 24
repairMinHashes() again	0	stale 0, repaired 0

After the first repair every sample carries 4.5.0, the hashed function count and the band entry count are unchanged (same escaper gives the same index back), /status reports minhash_compatibility_threshold: 4.4.5, and no errors were logged.

[danielplohmann]

Rebased onto main (maintainer edit) now that #171/#172/#173/#174 have landed — it had conflicted with #174 in four files. Force-pushed with --force-with-lease; the three commits are unchanged in content, only replayed.

How the conflicts were resolved, since two of them were not purely additive:

• MongoDbStorage.py — the new stale-minhash methods and Keep the family counters consistent with the collections (#151) #174's _deleteFamilyIfEmpty/_ensureFamilyDocument landed at the same insertion point. Both kept. deleteSample needed a real decision: this PR refactors the band-pull into _pullBandEntries, while Keep the family counters consistent with the collections (#151) #174 changed the same method to decrement family stats by what delete_many/delete_one actually removed. It now does both — calls _pullBandEntries and keeps Keep the family counters consistent with the collections (#151) #174's actual-count decrement and _deleteFamilyIfEmpty.
• McritClient.py — repairMinHashes inserted whole, next to Keep the family counters consistent with the collections (#151) #174's recomputeFamilyStats.
• MemoryStorage.py, tests/testStorage.py — additive, both sides kept.
• Added from packaging import version, which _staleMinHashVersionQuery needs and which arrived with the rebase context.

A welcome consequence of the ordering: since #171 landed first, _updateBands no longer upserts on the pull path and deletes the posting lists a pull empties — so the selective repair inherits that fix and does not accumulate tombstones per repair cycle, which is the interaction your PR body and #149 both flagged. Verified in the merged tree, not assumed.

Verified after the rebase:

246 passed, 36 subtests passed
ruff check: All checks passed!
ruff format --check: 138 files already formatted
ty check (pinned 0.0.74): All checks passed!

Worth a glance at the deleteSample resolution in particular, since that is the one place where a judgement was made rather than two additions kept.

🤖 Generated with Claude Code