feat(ingestion): reject empty and whitespace-only file uploads - #76
Merged
Conversation
An upload with no ingestible content (zero bytes or only whitespace, e.g. a lone CRLF) parses to zero chunks, so it can never be indexed. Such a document sits in the UI showing a no-op "Index" action that reverts on every click, because BM25 is an auto-maintained generated column and the index task is a no-op — there is nothing to index without chunks. Reject these at the upload boundary instead: - stream_upload_with_size_guard now tracks whether any non-whitespace byte was seen and raises a new EmptyFileError (422) when none was, cleaning up the .tmp so nothing is published to local or S3. This covers every HTTP upload (both storage backends funnel through the guard). - ingest_local_path (master-only MCP path) rejects a zero-byte file with the same EmptyFileError before creating any row or enqueuing. Tests: unit coverage for empty, whitespace-only, and whitespace-surrounded content in the guard; the local-path guard; and endpoint-level 422s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reject file uploads that have no ingestible content — zero bytes or only whitespace (e.g. a lone
\r\n) — at the upload boundary, returning 422.Why
A content-less file parses to zero chunks, so it can never be indexed. The document then sits in the Indexing/Documents UI showing an "Index" action that reverts on every click: BM25 is an auto-maintained Postgres generated column and the
index_documentworker task is a deliberate no-op, so with no chunks there is genuinely nothing to index. This surfaced from a real document whose stored bytes were exactly\r\n(2 bytes) — non-zero, so a naive byte-count check would miss it.Changes
api/services/upload.py—stream_upload_with_size_guardnow tracks whether any non-whitespace byte was streamed and raises a newEmptyFileError(422) when none was. The existingexcept BaseExceptioncleanup unlinks the.tmp, so nothing is published to local or S3. This covers every HTTP upload path — both storage backends funnel through this guard.api/services/documents.py—ingest_local_path(master-only MCP path, which usesput_pathand bypasses the streaming guard) rejects a zero-byte file with the sameEmptyFileErrorbefore any DB row or enqueue.Reviewer notes
bytes.strip()(ASCII whitespace). A flag short-circuits it once content is seen, so a large binary is trimmed at most once. A file containing only a UTF-8 BOM would count as content and pass — intentional, and not a regression.ingest_local_pathguards the 0-byte case only (cheapst_sizecheck) rather than reading a possibly-large local file into memory to whitespace-strip; the whitespace-only case is fully covered on the primary HTTP path where the incident originated.Tests
test_upload.py): empty, whitespace-only (the incident case), and whitespace-surrounded-content all asserted, with.tmpcleanup verified.test_ephemeral_ingest.py): local-path 0-byte rejection returns 422 before any DB/enqueue.test_documents.py): endpoint returns 422 for empty and whitespace-only uploads.Gate: ruff, mypy, and full unit + integration suite (859 passed, 13 skipped) all green.