fix: error when a path cannot be made relative to the dataset base - #8653
Open
LuciferYang wants to merge 3 commits into
Open
LuciferYang wants to merge 3 commits into
LuciferYang wants to merge 3 commits into
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Contributor
Author
|
cc @wjones127 FYI |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
`remove_prefix` existed twice, behaviourally identical, in `cleanup.rs` and `files.rs`, and both fell back to the input on a non-matching prefix. That fallback returns an absolute path where every caller expects one relative to the dataset root. `tracked_files` and `all_files` both document their `path` column as "Relative to `base_uri`", so the fallback silently breaks a documented output contract: a consumer joining `base_uri` with an absolute `path` gets a path that resolves nowhere. Erroring is the only honest answer, because the correct root-relative path is not recoverable at that point. Keep one copy, in `files.rs` because `scan.rs` is its child and shares the helper, and make it return `Result`. Name it `strip_prefix` to match `std::path::Path::strip_prefix`, whose fallibility is what a reader expects. `Error::internal` rather than `invalid_input`: the helper is private and every input is derived internally, from `data_dir().join(..)`, `deletion_file_path(..)`, `indices_dir().join(uuid)`, or a listing under `base`. No user-supplied path reaches it, so an unmatched prefix is a broken invariant rather than bad input, and the message should ask for a bug report. Seven of the eight production call sites build a keep-set or a reported path, where dropping an entry is what causes damage, so they propagate with `?`. The eighth, `cleanup_file_if_not_referenced`, classifies a listed object as a deletion candidate, and there the safe answer is the opposite: it logs and returns `Ok(None)`, the value the same function already returns for every other path shape it cannot classify. Aborting there would strand a pass that `remove_stream` has already partly executed, leaving old manifests behind while some of the data files they reference are gone. Covered by `cleanup_skips_listed_files_outside_the_dataset_base`, which installs a store wrapper that reports a rewritten location for the data files and asserts the pass skips the unclassifiable entry and still completes. Restoring `?` at that call site makes it fail. `cargo fmt --all --check`, `cargo clippy -p lance --all-targets -- -D warnings`, and `RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps` are clean; `dataset::files` (20) and `dataset::cleanup` (43) pass. The full suite is left to CI.
Make the skipped-file condition visible at the default log level without
flooding it: the per-file detail stays at `debug!`, and
`delete_unreferenced_files` warns once with a count. A store that
misreports one path misreports every path of that kind, so a per-file
`warn!` would print one line per object on an affected dataset.
Strengthen `cleanup_skips_listed_files_outside_the_dataset_base` so it
discriminates the change it is named for. Rewriting listed data-file
locations to `elsewhere/{filename}` did not: with the removed
absolute-path fallback that value fails the `starts_with("data")` check
and lands in the pre-existing "a .lance file outside the data directory
is left alone" arm, so every assertion held either way. It now rewrites
to `data/outside-base/{filename}`, which is outside the base and still
reads as a collectable data path. The rewritten entry is also stamped a
day before the epoch instead of carrying the fixture's own write time,
which had left it sitting exactly on the `unmodified_since` cutoff and
surviving only because the comparison is `<=`.
Add a test that `build_all_files_batch` refuses a location outside the
base rather than emitting an absolute path, and a near-miss sibling case
pinning that the match is per path segment. Document the new failure mode
on `all_files` and `tracked_files`, whose `path` column is documented as
relative to `base_uri`. Correct `listed_metas`'s doc comment, which
described what the cleanup pass classifies rather than what the helper
returns. Restore the `Vec` reservation that the `extend`-to-`push`
conversion dropped.
LuciferYang
force-pushed
the
fix/strip-prefix-fails-closed
branch
from
August 24, 2026 14:30
5c10f1e to
e6f9990
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The merge from main preserves the reviewed patch exactly. The new exact-version cleanup policy still feeds the same manifest-processing and path-classification boundaries, and the merged OpenDAL normalization remains effective, so the fail-closed inventory and cleanup behavior remains sound.
Contributor
Author
|
The red |
Contributor
Author
|
friendly ping @Xuanwo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
remove_prefixexisted twice, identically, indataset/cleanup.rsanddataset/files.rs. Both returned the input unchanged whenPath::prefix_matchfailed, which hands back an absolute path where the caller needs one relative to the dataset root. This replaces both with a singlestrip_prefixinfiles.rsthat returnsResult.The fallback could delete live data. If a misreported location happens to read as base-relative under
data, cleanup's classifier passes it through thestarts_with("data")check, fails to find it in the keep-set, and deletes a file the manifest still references.tracked_filesandall_filesalso document theirpathcolumn as relative tobase_uri, so an absolute value there breaks a documented contract silently.Seven of the eight call sites propagate with
?. Four of them build the path by joining onto the base and cannot reach the error at all. The other three pass a location the store reported from a listing, and there an omission is what does the damage: a missing keep-set entry gets a live file deleted, and a missing inventory row gets it deleted by whatever consumes the inventory.The eighth is
CleanupTask::cleanup_file_if_not_referenced, which skips the object instead. It is the only call site that runs while the pass is already deleting, so propagating would strand a half-finished pass and repeat on every later run. A store that misreports one path misreports every path of that kind, so the per-file detail stays atdebug!and the pass warns once with a count.This is pre-work for #8097, which builds a keep-set for external distributed cleanup out of this helper's output. An absolute path in a keep-set means the real path is absent from it, and a driver that diffs the keep-set against a listing deletes a live file.
#8652 was filed out of this work for a related OpenDAL path-encoding mismatch. #8654 fixed it and this branch is rebased onto that, so the error path is no more reachable on those stores than anywhere else.
Not in scope: two byte-wise prefix strips with the same shape remain in
mem_wal/memtable/flush.rsandlance-table/src/format/index.rs, both unreachable today, andDataFile.path's single-segment invariant is still neither documented nor validated.Test plan
Four
strip_prefixcases: a nested path, a path equal to the base, a path under a different dataset, andbucket/dataset2against basebucket/dataset, which pins that matching is per path segment rather than per byte.build_all_files_batch_rejects_a_path_outside_the_basechecks that the batch fails and names the offending path instead of emitting an absolute one.cleanup_skips_listed_files_outside_the_dataset_basewraps the store so listed data files reportdata/outside-base/{filename}, which is outside the base but still reads as a collectable data path. Both directions were checked by reverting: restore the fallback and the test fails, replace the skip with?and it fails too.cargo fmt --all --check,cargo clippy --all --tests --benches -- -D warnings,RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps, andcargo test -p lance --lib dataset::(2204 tests) are clean.