Say which documents are over MongoDB's size limit, and keep the sample (#42) - #175
Open
r0ny123 wants to merge 2 commits into
Open
Say which documents are over MongoDB's size limit, and keep the sample (#42)#175r0ny123 wants to merge 2 commits into
r0ny123 wants to merge 2 commits into
Conversation
An insert batch with a document over MongoDB's 16 MiB limit failed as a bare "Database insert failed." with nothing recording which document, and the sample was lost (#42; pymongo raises DocumentTooLarge for a clearly oversized document before sending, MongoDB answers a write error for one just over). _dbInsertMany now recognises both, measures the documents the ordered insert did not get to, and logs the oversized ones with their identifying fields and sizes. For the disassembly blob collections (xcfg, query_xcfg) it drops the oversized blobs with a warning and inserts the rest, so the sample and its functions are stored and only the disassembly of the affected function is missing; for any other collection it raises a ValueError naming the documents, since a function or sample document that cannot be stored cannot be dropped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #42 (Mongo: BSON document might be bigger than 16 MB).
Cause
An insert batch with one document over MongoDB's 16 MiB document limit failed as a bare
ValueError("Database insert failed."), with the error log carrying only the traceback, so nothing said which sample or function was responsible, and the sample was lost. The reporter saw it 4 times in 120k files. Since v1.7.0 the disassembly lives in its ownxcfgdocuments, one per function, which is where a single giant function ends up over the limit.Two shapes of the error exist: pymongo raises
DocumentTooLargebefore sending a clearly oversized document, and MongoDB answers a write error (code 2, "object to insert too large") for one just over the limit, which pymongo surfaces asBulkWriteErrorafter inserting the documents before it (ordered insert).Fix
_dbInsertManyrecognises both, measures the documents the ordered insert did not get to, and logs the oversized ones with their identifying fields (_id,function_id,sample_id,sha256,offset) and byte sizes.xcfg,query_xcfg) it drops the oversized blobs with a warning and inserts the rest. The sample and all its functions are stored; only the disassembly of the affected function is missing, which the readers already treat as "no disassembly" (they decode a missing blob to{}). That function then gets no MinHash, as any function without disassembly.ValueErrornaming the documents, since a function or sample document that cannot be stored cannot be dropped.Verification
tests/testStorage.py(Mongo): a blob one byte over the limit is dropped with a warning naming one function, the other blob is stored, and the error log records the offender with its size and that it was dropped; an oversized blob in the middle of five keeps the other three; an oversized function document fails with a message naming exactly that document and the ordered insert's earlier document stays stored.ruff check,ruff format --check,ty checkclean.