Skip to content

fix(storage): pad uncompressed flush buffer to allocated page range on disk - #1091

Open
Tyagiquamar wants to merge 2 commits into
LadybugDB:mainfrom
Tyagiquamar:fix-uncheckpointed-csr-scan-eof
Open

Tyagiquamar wants to merge 2 commits into
LadybugDB:mainfrom
Tyagiquamar:fix-uncheckpointed-csr-scan-eof

Conversation

@Tyagiquamar

Copy link
Copy Markdown
Contributor

Problem

When batch-loading rel tables via COPY without intermediate checkpoints, subsequent scans of flushed CSR node groups fail with a read-past-EOF assertion in LocalFileSystem::readFromFile (localFileInfo->getFileSize() >= position + numBytes).

Root cause

ColumnChunkData::flush and Column::flushData allocate page ranges using PageAllocator::allocatePageRange(preScanMetadata.getNumPages()). When flushing uncompressed chunks or ALP floating-point data, uncompressedFlushBuffer previously wrote only the raw unpadded buffer size (buffer.size()) to the data file. For multi-page allocations where the buffer content did not align to whole page multiples, trailing pages within the allocated range were not written on disk, leaving the underlying physical file shorter than the logical page index space. Subsequent scans reading subsequent pages at pageIdx * LBUG_PAGE_SIZE encountered a truncated file size.

Fix

  1. In uncompressedFlushBuffer, when in persistent mode (!dataFH->isInMemoryMode()) and buffer.size_bytes() < entry.numPages * LBUG_PAGE_SIZE, pad the remaining allocated page range with zero bytes on disk.
  2. In flushCompressedFloats, zero-pad any remaining data pages before the ALP exception pages when fewer data pages than allocated are used.
  3. Added RelCopyMultiPageUncheckpointedScan regression test in test/copy/copy_test.cpp to verify that batch COPY-loaded rels spanning multiple pages can be queried without an intermediate checkpoint.

Fixes #1059

@adsharma
adsharma self-requested a review October 4, 2026 17:14

@adsharma adsharma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused vars + clang-format + test coverage for ALP

@Tyagiquamar

Copy link
Copy Markdown
Contributor Author

Addressed maintainer feedback: removed unused startOffset, formatted touched files with clang-format, and added RelCopyMultiPageFloatUncheckpointedScan unit test coverage verifying ALP floating-point uncheckpointed scans.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: scans of COPY-loaded, not-yet-checkpointed CSR rel groups fail with read past EOF

2 participants