Serve repeated requests from the newest finished job, never a failed one (fkie-cad/mcritweb#47) - #160
Open
r0ny123 wants to merge 2 commits into
Open
Conversation
get_cached_job_id picked the newest job with the same descriptor that had attempts left and was not terminated, whatever its state. A force rematch that was still queued or running therefore shadowed the finished job whose result the next plain request could actually use, and cancelling the forced job was the only way to get the old result back (fkie-cad/mcritweb#47). Both queues now prefer a finished job over an unfinished one and the newest within each group; jobs without attempts left (failed) or terminated ones are never reused. On MongoDB this is one sort - finished_at descending puts every date before null - so no second query is needed. LocalQueue drops its descriptor-to-job map, which could only ever remember the last job. test_job_cache_prio encoded the old rule (a queued forced job wins over the finished one) and is updated to the new one.
LocalQueue.get_job() indexed its defaultdict, so a lookup of an unknown or already-cleaned id inserted a None entry that every later scan of the jobs tripped over - the new cache scan included. The lookup uses .get() now, and the cache test covers a lookup of an unknown id. Also narrows the queue type in the mongo test helper for ty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes fkie-cad/mcritweb#47 (labelled
mcritthere). The cache selection was wrong: a force rematch that was still queued or running shadowed the finished job whose result a repeated plain request could actually use.What changed
get_cached_job_idpicked the newest job with the same descriptor that had attempts left and was not terminated, whatever its state. Both queues now prefer a finished job over an unfinished one and the newest within each group; jobs without attempts left (failed) or terminated ones are never reused, exactly the rule the issue proposes.MongoQueue: one query, sortedfinished_atdescending thencreated_atdescending. A descending sort places every date beforenull, so the finished jobs come first without a second query.LocalQueue: the same rule over its jobs; the_descriptor_to_jobmap, which could only ever remember the last job, is gone.test_job_cache_prioencoded the old rule (a queued forced job wins over the finished one) and is updated; new tests cover finished-over-running, newest-finished, and that failed/terminated jobs are skipped, on both queues.Verified against a running instance
MongoDB 7 + this branch as server and worker, four samples, a finished 1vs1 job between two of them:
ruff,tyand the full suite (mongo tests included) pass.