Answer function name searches from the distinct names - #170
Open
r0ny123 wants to merge 2 commits into
Open
Conversation
A free-text function search is an unanchored, case-insensitive regex on function_name. MongoDB cannot bound an index with it, so when nothing matches it examines every function document before answering the empty page: seconds on a million functions, the ~30 s of fkie-cad/mcritweb#76 on larger databases. With a non-default sort it examines every document even when there are hits. Function names repeat heavily (most are empty, the rest are symbols shared across samples), so the set of distinct names is small where the collection is large. MongoDbStorage now lists the distinct names with one DISTINCT_SCAN over the function_name index (a $group with a limit, tens of milliseconds for millions of functions), evaluates the substring condition against that list in Python, and hands MongoDB an $in (or $nin for "!?") of the matching names, which it answers from the index. Nothing matching becomes an empty $in, so the empty page costs nothing. Above 10000 distinct names the search keeps the regex, unbounded as before; the listing only runs when the query actually holds a substring condition on function_name. On two million synthetic functions with 5000 distinct names (MongoDB 7): no-result search 4.2 s -> 22 ms; a name held by 19 functions 4.2 s -> 2 ms; 'beacon' sorted by num_blocks 1.5 s -> 62 ms; the common term 'main' with the default sort 15 ms -> 78 ms.
The count cap alone does not bound the $in the matching names go into: thousands of long mangled symbols fit under 10000 names but could push the find command past MongoDB's 16 MiB limit, where the regex it replaces was valid. The listing now also stops at 1 MiB of names in total and the search keeps the regex beyond it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes fkie-cad/mcritweb#76 (searching for functions is painfully slow, ~30 s, on larger databases if no results are found).
Cause
A free-text function search becomes an unanchored, case-insensitive regex on
function_name. MongoDB cannot bound an index with that, so when nothing matches it examines every function document before it can answer the empty page: seconds on a million functions, the reported 30 s on larger databases. With a non-default sort (sort_by=num_blocksetc.) it examines every document even when there are hits, since it cannot stop early.Fix
Function names repeat heavily: most are empty, the rest are symbols shared across samples (the live test database holds 9 distinct names for 1158 functions). So the set of distinct names is small where the collection is large.
MongoDbStorage.findFunctionByStringnow:$group+$limitaggregation, which MongoDB runs as aDISTINCT_SCANover the existingfunction_nameindex (tens of milliseconds for millions of functions, verified withexplain()). Only when the query actually holds a substring condition onfunction_name;sample_id:0orfunction_name:main(equality) do not pay for it.$in(or$ninfor!?) of the matching names, which it answers from the index. Nothing matching becomes$in: [], so the empty page costs nothing._DISTINCT_VALUES_CAP(10000) distinct names the search keeps the regex, unbounded as before. That is the one case this PR does not speed up; it needs an n-gram index, which is out of scope.MongoSearchTranspilertakes an optionalknown_valuesmap for this; an empty search term keeps the regex (it matches everything), other fields and operators are untouched.Measurements
Two million synthetic functions, MongoDB 7, page of 101, from the storage's own
_get_search_query+find:beacon(515 names), sorted by num_blocksmain(986 names), default sorte(4014 names), default sortThe common-term default-sort case is the only one that gets slower, by tens of milliseconds, because the regex scan in id order finds its 101 hits almost immediately there.
Verification
tests/testMongoSearchTranspiler.py: matching values become$in/$nin, case-insensitive like the regex, metacharacters literal, nothing matching is an empty list, empty term and other fields keep their form.tests/testStorage.py(Mongo): the search answers the same entries through the rewrite and through the regex fallback (cap patched to 2), combined with other conditions and negation, and the listing runs only for substring conditions.ruff check,ruff format --check,ty checkclean./search/functionsfor a no-result term, a name in both cases,sample_id:0 decrypt,sample_id:1 decrypt(no result),function_name:!?decrypt, the empty term and a term with regex metacharacters all answer as before; MCRITweb's function listing, sample page and search page render the same rows.