Skip to content

Answer function name searches from the distinct names - #170

Open
r0ny123 wants to merge 2 commits into
danielplohmann:mainfrom
r0ny123:fix/76-fast-no-result-function-search
Open

Answer function name searches from the distinct names#170
r0ny123 wants to merge 2 commits into
danielplohmann:mainfrom
r0ny123:fix/76-fast-no-result-function-search

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Closes fkie-cad/mcritweb#76 (searching for functions is painfully slow, ~30 s, on larger databases if no results are found).

Cause

A free-text function search becomes an unanchored, case-insensitive regex on function_name. MongoDB cannot bound an index with that, so when nothing matches it examines every function document before it can answer the empty page: seconds on a million functions, the reported 30 s on larger databases. With a non-default sort (sort_by=num_blocks etc.) it examines every document even when there are hits, since it cannot stop early.

Fix

Function names repeat heavily: most are empty, the rest are symbols shared across samples (the live test database holds 9 distinct names for 1158 functions). So the set of distinct names is small where the collection is large.

MongoDbStorage.findFunctionByString now:

  1. Lists the distinct names with one $group + $limit aggregation, which MongoDB runs as a DISTINCT_SCAN over the existing function_name index (tens of milliseconds for millions of functions, verified with explain()). Only when the query actually holds a substring condition on function_name; sample_id:0 or function_name:main (equality) do not pay for it.
  2. Evaluates the substring condition against that list in Python with the same escaped, case-insensitive pattern, and hands MongoDB an $in (or $nin for !?) of the matching names, which it answers from the index. Nothing matching becomes $in: [], so the empty page costs nothing.
  3. Above _DISTINCT_VALUES_CAP (10000) distinct names the search keeps the regex, unbounded as before. That is the one case this PR does not speed up; it needs an n-gram index, which is out of scope.

MongoSearchTranspiler takes an optional known_values map for this; an empty search term keeps the regex (it matches everything), other fields and operators are untouched.

Measurements

Two million synthetic functions, MongoDB 7, page of 101, from the storage's own _get_search_query + find:

collection search before after
5000 distinct names no result, default sort 4.2 s (2M docs) 22 ms (0 docs)
5000 distinct names name held by 19 functions 4.2 s 2 ms
5000 distinct names beacon (515 names), sorted by num_blocks 1.5 s 62 ms
5000 distinct names main (986 names), default sort 15 ms 78 ms
5000 distinct names e (4014 names), default sort 5 ms 27 ms
1.94M distinct names (above cap) no result 4.3 s 4.1 s (regex fallback, +70 ms listing)

The common-term default-sort case is the only one that gets slower, by tens of milliseconds, because the regex scan in id order finds its 101 hits almost immediately there.

Verification

  • tests/testMongoSearchTranspiler.py: matching values become $in / $nin, case-insensitive like the regex, metacharacters literal, nothing matching is an empty list, empty term and other fields keep their form.
  • tests/testStorage.py (Mongo): the search answers the same entries through the rewrite and through the regex fallback (cap patched to 2), combined with other conditions and negation, and the listing runs only for substring conditions.
  • Full suite: 214 passed, ruff check, ruff format --check, ty check clean.
  • Live: server restarted with the change, /search/functions for a no-result term, a name in both cases, sample_id:0 decrypt, sample_id:1 decrypt (no result), function_name:!?decrypt, the empty term and a term with regex metacharacters all answer as before; MCRITweb's function listing, sample page and search page render the same rows.

A free-text function search is an unanchored, case-insensitive regex on
function_name. MongoDB cannot bound an index with it, so when nothing matches
it examines every function document before answering the empty page: seconds
on a million functions, the ~30 s of fkie-cad/mcritweb#76 on larger databases.
With a non-default sort it examines every document even when there are hits.

Function names repeat heavily (most are empty, the rest are symbols shared
across samples), so the set of distinct names is small where the collection is
large. MongoDbStorage now lists the distinct names with one DISTINCT_SCAN over
the function_name index (a $group with a limit, tens of milliseconds for
millions of functions), evaluates the substring condition against that list in
Python, and hands MongoDB an $in (or $nin for "!?") of the matching names,
which it answers from the index. Nothing matching becomes an empty $in, so the
empty page costs nothing. Above 10000 distinct names the search keeps the
regex, unbounded as before; the listing only runs when the query actually
holds a substring condition on function_name.

On two million synthetic functions with 5000 distinct names (MongoDB 7):
no-result search 4.2 s -> 22 ms; a name held by 19 functions 4.2 s -> 2 ms;
'beacon' sorted by num_blocks 1.5 s -> 62 ms; the common term 'main' with the
default sort 15 ms -> 78 ms.
The count cap alone does not bound the $in the matching names go into:
thousands of long mangled symbols fit under 10000 names but could push the
find command past MongoDB's 16 MiB limit, where the regex it replaces was
valid. The listing now also stops at 1 MiB of names in total and the search
keeps the regex beyond it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Searching for functions is painfully slow (~30s) on larger database if no results are found

1 participant