Skip to content

Say which documents are over MongoDB's size limit, and keep the sample (#42) - #175

Open
r0ny123 wants to merge 2 commits into
danielplohmann:mainfrom
r0ny123:fix/42-oversized-documents
Open

Say which documents are over MongoDB's size limit, and keep the sample (#42)#175
r0ny123 wants to merge 2 commits into
danielplohmann:mainfrom
r0ny123:fix/42-oversized-documents

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Closes #42 (Mongo: BSON document might be bigger than 16 MB).

Cause

An insert batch with one document over MongoDB's 16 MiB document limit failed as a bare ValueError("Database insert failed."), with the error log carrying only the traceback, so nothing said which sample or function was responsible, and the sample was lost. The reporter saw it 4 times in 120k files. Since v1.7.0 the disassembly lives in its own xcfg documents, one per function, which is where a single giant function ends up over the limit.

Two shapes of the error exist: pymongo raises DocumentTooLarge before sending a clearly oversized document, and MongoDB answers a write error (code 2, "object to insert too large") for one just over the limit, which pymongo surfaces as BulkWriteError after inserting the documents before it (ordered insert).

Fix

_dbInsertMany recognises both, measures the documents the ordered insert did not get to, and logs the oversized ones with their identifying fields (_id, function_id, sample_id, sha256, offset) and byte sizes.

  • For the blob collections (xcfg, query_xcfg) it drops the oversized blobs with a warning and inserts the rest. The sample and all its functions are stored; only the disassembly of the affected function is missing, which the readers already treat as "no disassembly" (they decode a missing blob to {}). That function then gets no MinHash, as any function without disassembly.
  • For any other collection it raises a ValueError naming the documents, since a function or sample document that cannot be stored cannot be dropped.

Verification

  • tests/testStorage.py (Mongo): a blob one byte over the limit is dropped with a warning naming one function, the other blob is stored, and the error log records the offender with its size and that it was dropped; an oversized blob in the middle of five keeps the other three; an oversized function document fails with a message naming exactly that document and the ordered insert's earlier document stays stored.
  • Storage suite: 52 passed; ruff check, ruff format --check, ty check clean.
  • The tests run against a real MongoDB 7 and exercise the server-side write error path (the one the issue's 60 MB case would take through pymongo's own check).

An insert batch with a document over MongoDB's 16 MiB limit failed as a bare
"Database insert failed." with nothing recording which document, and the
sample was lost (#42; pymongo raises DocumentTooLarge for a clearly oversized
document before sending, MongoDB answers a write error for one just over).

_dbInsertMany now recognises both, measures the documents the ordered insert
did not get to, and logs the oversized ones with their identifying fields and
sizes. For the disassembly blob collections (xcfg, query_xcfg) it drops the
oversized blobs with a warning and inserts the rest, so the sample and its
functions are stored and only the disassembly of the affected function is
missing; for any other collection it raises a ValueError naming the documents,
since a function or sample document that cannot be stored cannot be dropped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mongo: BSON document might be bigger than 16mb

1 participant