Skip to content

fix(io): normalize OpenDAL listed paths - #8654

Merged
Xuanwo merged 6 commits into
mainfrom
gatekeeper/fix-8652-1
Aug 24, 2026
Merged

Xuanwo merged 6 commits into
mainfrom
gatekeeper/fix-8652-1

Conversation

@lance-gatefixer

@lance-gatefixer lance-gatefixer Bot commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • route every OpenDAL-backed provider through a Lance adapter that preserves request-compatible listed paths
  • decode recursive, offset, and delimiter listing results only when they mismatch the requested prefix
  • preserve upstream spellings that already match, including literal percent-escape segments
  • retain native start-after pushdown where supported and compare normalized paths in the bridge fallback
  • add in-memory regression cases for both raw reserved characters and literal percent escapes

Root cause

object_store_opendal rebuilds OpenDAL listing entries with Path::from, which percent-encodes reserved characters. Lance constructs dataset bases with Path::from_url_path, so a raw reserved-character base can mismatch the returned listing. Unconditionally decoding the output is also unsafe because a deliberately double-encoded URI segment relies on the existing encoded listing spelling.

The adapter now preserves any upstream location that already prefix-matches the request and decodes only mismatches. This fixes raw reserved-character bases without changing literal percent-escape paths that already worked.

Validation

  • cargo test -p lance-io --no-default-features --features oss object_store::opendal_store::tests::test_list_preserves_request_path_spelling
  • cargo test -p lance-io --lib -- --skip uring::tests (202 passed)
  • cargo check -p lance-io --all-features --tests
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

The complete lance-io library test command cannot finish in this container because io_uring initialization is denied by the environment; the non-io_uring library suite passes.

Fixes #8652

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Aug 20, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the current head is held by an unrelated intermittent macOS BQ test failure that this App cannot rerun.

Remote head 1d7f8e3 contains the current main tip b945b3e. The mac-build (stable) Check failed vector::bq::storage::tests::test_accurate_approx_mode_reduces_binary_lut_quantization_error with accurate_error=0.07851982 and normal_error=0.06913757; this PR changes only lance-io, and the identical base SHA passed the macOS Check.

I fetched and verified base ancestry, inspected the live log, ran both the failed BQ test and the OpenDAL regression successfully, and pushed the assigned branch, which was already up to date. A failed-workflow rerun was requested, but the GitHub App lacks Actions rerun permission.

The smallest unblock is for a maintainer to rerun the failed mac-build (stable) job; alternatively, rerun the full Rust workflow on this head.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 21, 2026
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 21, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 21, 2026
# Conflicts:
#	rust/lance-io/src/object_store/providers/gcp.rs
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 23, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 23, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the current Ready PR is held by an unrelated intermittent Python nearest-neighbor test failure that this App cannot rerun.

Remote head 6b75b1a contains the current main tip cc0f085. The Python Linux 3.13 x86_64 Check (job 97195436446) failed only python/tests/torch_tests/test_bench_utils.py::test_ground_truth with one nearest-neighbor cutoff mismatch; 1 test failed and 1,379 passed. The exact base SHA passed the same Check in job 97193690776, and this PR changes only rust/lance-io with no Python diff.

I fetched and verified base ancestry, inspected the live failure log, and ran the exact test successfully with the repository uv workflow. A failed-workflow rerun was requested, but the GitHub App lacks Actions rerun permission. No attributable repair was found, so no speculative change was made.

The smallest unblock is for a maintainer to rerun the failed Python Linux 3.13 x86_64 job; alternatively, rerun the full Python workflow on this head.

@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 23, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 23, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The conflict-free base merge leaves the reviewed lance-io adapter unchanged. Prefix-aware normalization still preserves raw reserved-character and literal percent-escape paths under the updated workspace version.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 24, 2026
@Xuanwo
Xuanwo merged commit 371da45 into main Aug 24, 2026
39 checks passed
@Xuanwo
Xuanwo deleted the gatekeeper/fix-8652-1 branch August 24, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: OpenDAL-backed stores list percent-encoded paths while dataset.base stays raw

1 participant