Skip to content

Add opt-in RLE v2 run-length widths #7327

Description

@Xuanwo

Summary

Add an opt-in RLE v2 codec path that supports run_lengths = Flat(8|16|32) without changing the Lance data file format or RLE protobuf shape.

Scope

  • Decode RLE run lengths from Flat(8), Flat(16), and Flat(32).
  • Keep Flat(8) as the default compatibility mode.
  • Add strict validation for zero lengths, values/lengths entry mismatches, underflow, and overflow.
  • Add a table-level reader/writer feature flag for datasets that may contain RLE v2 pages.
  • Add an explicit writer opt-in path.

Acceptance Criteria

  • New Lance can write and read opt-in RLE v2 pages.
  • Old Flat(8) RLE data remains readable.
  • Old readers fail closed at the table level when RLE v2 pages may exist.
  • Compatibility mode still writes Flat(8).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions