Skip to content

Clean up the query data nothing refers to, and survive jobs without a result (#68) - #176

Open
r0ny123 wants to merge 2 commits into
danielplohmann:mainfrom
r0ny123:fix/68-cleanup-orphans
Open

Clean up the query data nothing refers to, and survive jobs without a result (#68)#176
r0ny123 wants to merge 2 commits into
danielplohmann:mainfrom
r0ny123:fix/68-cleanup-orphans

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Closes #68 (considerations for DbCleanup: orphan query samples and functions without a job, a compact afterwards).

What the cleanup did, and the bug in it

doDbCleanup deleted query samples and query jobs older than STORAGE_MONGODB_CLEANUP_TTL, which already covers query samples whose job is gone (they age out by their own timestamp). It read result["info"]["sample"] off every finished and failed query job, though, and a failed job that died before matching has no result, so the first such job took the whole cleanup down with a TypeError. On an instance with STORAGE_MONGODB_ENABLE_CLEANUP that means the cleanup never completes again once one query job has failed early.

Changes

  • A job without a result is nothing to protect and nothing to collect: it is deleted once older than the TTL and kept otherwise. Finished jobs are dated by finished_at, failed ones by started_at, a job that never started counts as old.
  • After the TTL pass, deleteOrphanedQueryData() on the storage removes the query functions whose query sample is gone and the query disassembly (query_xcfg) whose function is gone: what an interrupted deletion or a job deleted without its sample leaves behind. The query collections are bounded by the TTL, so listing their ids is affordable; the memory backend drops orphaned query functions the same way.
  • STORAGE_MONGODB_COMPACT_AFTER_CLEANUP (default off) runs MongoDB's compact on query_samples, query_functions, query_xcfg, fs.files and fs.chunks afterwards, so the freed space goes back to the file system. It needs the compact privilege; a refusal is reported per collection, not raised, since the cleanup itself has succeeded by then.
  • The job now answers a report: samples and jobs deleted, orphans removed, and the compact outcome when it ran.

Verification

  • tests/testWorkerCleanup.py (3 tests, worker with fakes): an old failed job without a result is deleted and a recent one kept, a job that never started counts as old; a recent job protects its sample while an old job takes its old sample along and a recent sample is kept; orphan removal always runs and compaction only when configured, with both reflected in the report.
  • tests/testStorage.py (Mongo): orphaned query functions and disassembly are deleted while the query sample's own are kept, and a second run finds nothing; compact answers per collection against the real MongoDB.
  • Full suite: 208 passed; ruff check, ruff format --check, ty check clean.
  • Live verification on the MongoDB-backed instance follows in a comment.

… result

doDbCleanup deleted query samples and query jobs older than the TTL, and
nothing else (danielplohmann#68). It also read result["info"]["sample"] off every finished
and failed query job, so the first failed job without a result (one that died
before matching) took the whole cleanup down with a TypeError.

The cleanup now treats a job without a result as nothing to protect - it is
deleted once older than the TTL, kept otherwise - and after the TTL pass
removes the query functions whose query sample is gone and the query
disassembly whose function is gone, through deleteOrphanedQueryData() on the
storage. With STORAGE_MONGODB_COMPACT_AFTER_CLEANUP (default off, it needs
the compact privilege) it then runs MongoDB's compact on the collections the
deleted query data and results lived in. The job answers a report of what it
deleted and freed.
…t the compaction knob

The cleanup took a client-side snapshot of the query samples and then
deleted every query function outside it, so a query being inserted on
another worker could lose its functions. It now takes the newest query
function id first and judges only records older than that; a query
inserted afterwards is never looked at. The ids are walked in batches
of 5000 instead of one distinct() and one $nin over the whole
collection. MemoryStorage looked for orphans in the wrong collection.
STORAGE_MONGODB_COMPACT_AFTER_CLEANUP is described in docs/TUNING.md.
@r0ny123

r0ny123 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Live verification on the MongoDB-backed instance (server and worker restarted on this change):

Planted the residue the issue is about: two query functions whose query sample does not exist, two query disassembly documents (one for an orphan function, one for no function at all), and a failed getMatchesForUnmappedBinary job from nine days ago with attempts_left: 0 and no result, which is the shape that used to take the cleanup down.

Scheduled doDbCleanup through the index. The job finished and answered:

{'num_query_samples_deleted': 0, 'num_query_jobs_deleted': 1, 'orphans': {'query_functions': 2, 'query_xcfg': 2}}

Afterwards query_functions and query_xcfg were empty, the failed job was gone, and the four corpus samples with their 1158 functions were untouched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Considerations for DbCleanup

1 participant