Skip to content

[python] Optimize BLOB write performance#8625

Draft
discivigour wants to merge 1 commit into
masterfrom
feat/improveWritePerf
Draft

[python] Optimize BLOB write performance#8625
discivigour wants to merge 1 commit into
masterfrom
feat/improveWritePerf

Conversation

@discivigour

Copy link
Copy Markdown
Contributor

What is the purpose of the change?

Improve Python BLOB write performance by reducing CRC and Arrow row-selection overhead.

Brief change log

  • Use isal_zlib.crc32 when available, with zlib as the fallback.
  • Reuse the original Arrow batch when all rows belong to one partition/bucket.
  • Use zero-copy Arrow slice for contiguous partition/bucket groups.
  • Keep Arrow take for non-contiguous groups.
  • Add coverage for CRC compatibility and all three Arrow grouping paths.

Performance

A 60 GB BLOB write test improved as follows:

Data size Before After Elapsed-time reduction
60 GB 178.984 s 132.871 s 25.76%

The elapsed time decreased by 46.113 seconds, from 178.984 seconds to 132.871 seconds.

Tests

  • Verify the fallback CRC written to BLOB records matches standard zlib.crc32.
  • Verify isal_zlib and zlib produce byte-identical BLOB records.
  • Verify full batches are passed through without Arrow take.
  • Verify contiguous groups use zero-copy Arrow slice.
  • Verify non-contiguous groups continue to use Arrow take.
  • Verify partitioned BLOB data can be written and read end to end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant