Clean up the query data nothing refers to, and survive jobs without a result (#68) - #176
Open
r0ny123 wants to merge 2 commits into
Open
Clean up the query data nothing refers to, and survive jobs without a result (#68)#176r0ny123 wants to merge 2 commits into
r0ny123 wants to merge 2 commits into
Conversation
… result doDbCleanup deleted query samples and query jobs older than the TTL, and nothing else (danielplohmann#68). It also read result["info"]["sample"] off every finished and failed query job, so the first failed job without a result (one that died before matching) took the whole cleanup down with a TypeError. The cleanup now treats a job without a result as nothing to protect - it is deleted once older than the TTL, kept otherwise - and after the TTL pass removes the query functions whose query sample is gone and the query disassembly whose function is gone, through deleteOrphanedQueryData() on the storage. With STORAGE_MONGODB_COMPACT_AFTER_CLEANUP (default off, it needs the compact privilege) it then runs MongoDB's compact on the collections the deleted query data and results lived in. The job answers a report of what it deleted and freed.
…t the compaction knob The cleanup took a client-side snapshot of the query samples and then deleted every query function outside it, so a query being inserted on another worker could lose its functions. It now takes the newest query function id first and judges only records older than that; a query inserted afterwards is never looked at. The ids are walked in batches of 5000 instead of one distinct() and one $nin over the whole collection. MemoryStorage looked for orphans in the wrong collection. STORAGE_MONGODB_COMPACT_AFTER_CLEANUP is described in docs/TUNING.md.
Contributor
Author
|
Live verification on the MongoDB-backed instance (server and worker restarted on this change): Planted the residue the issue is about: two query functions whose query sample does not exist, two query disassembly documents (one for an orphan function, one for no function at all), and a failed Scheduled Afterwards |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #68 (considerations for DbCleanup: orphan query samples and functions without a job, a compact afterwards).
What the cleanup did, and the bug in it
doDbCleanupdeleted query samples and query jobs older thanSTORAGE_MONGODB_CLEANUP_TTL, which already covers query samples whose job is gone (they age out by their own timestamp). It readresult["info"]["sample"]off every finished and failed query job, though, and a failed job that died before matching has no result, so the first such job took the whole cleanup down with aTypeError. On an instance withSTORAGE_MONGODB_ENABLE_CLEANUPthat means the cleanup never completes again once one query job has failed early.Changes
finished_at, failed ones bystarted_at, a job that never started counts as old.deleteOrphanedQueryData()on the storage removes the query functions whose query sample is gone and the query disassembly (query_xcfg) whose function is gone: what an interrupted deletion or a job deleted without its sample leaves behind. The query collections are bounded by the TTL, so listing their ids is affordable; the memory backend drops orphaned query functions the same way.STORAGE_MONGODB_COMPACT_AFTER_CLEANUP(default off) runs MongoDB'scompactonquery_samples,query_functions,query_xcfg,fs.filesandfs.chunksafterwards, so the freed space goes back to the file system. It needs the compact privilege; a refusal is reported per collection, not raised, since the cleanup itself has succeeded by then.Verification
tests/testWorkerCleanup.py(3 tests, worker with fakes): an old failed job without a result is deleted and a recent one kept, a job that never started counts as old; a recent job protects its sample while an old job takes its old sample along and a recent sample is kept; orphan removal always runs and compaction only when configured, with both reflected in the report.tests/testStorage.py(Mongo): orphaned query functions and disassembly are deleted while the query sample's own are kept, and a second run finds nothing; compact answers per collection against the real MongoDB.ruff check,ruff format --check,ty checkclean.