[core] Keep raw-value pushdown off masked columns in query auth#8582
Closed
plusplusjiajia wants to merge 2 commits into
Closed
[core] Keep raw-value pushdown off masked columns in query auth#8582plusplusjiajia wants to merge 2 commits into
plusplusjiajia wants to merge 2 commits into
Conversation
67f242b to
51f3b9d
Compare
A cross-column mask (e.g. display := concat_ws('-', first, last)) threw
at read time when the query projected the masked target but not the
mask's input columns: "Column masking refers to field 'first' which is
not present in output row type".
Row-filter operands are already added to the read projection and
projected back out (apache#8447); this does the same for column-mask inputs.
TableQueryAuthResult.requiredAuthFields collects filter operands plus,
transitively, the inputs of every mask whose target is readable; the
widening appends the missing table columns as-is (preserving nested
pruning), and the scan pushes the widened read type before planning so
file-level column pruning (e.g. data evolution) keeps the files those
columns live in.
Stale rules (columns absent from the latest schema, e.g. after a rename
or drop) fail closed once at plan time in AbstractDataTableScan.authQuery,
the common path of every scan; re-validated only when the fetched rules
change. Projected system fields (e.g. _ROW_ID) stay valid, unprojected
ones are rejected; rules touching a nested-pruned or unprojected
blob-view column are rejected too.
The read schema is fixed once the first split reader exists (split reads
cache their format readers and ignore later read-type changes); every
split is projected back to the query's read type, and masks apply only
to columns readable from the query, so a column merely retained in a
previously widened schema does not activate a mask after a rules change.
Previously the first createReader call mutated this.readType, so later
splits of the same TableRead leaked the auth-added columns -- also on
the pre-existing row-filter path.
Related re-configuration fixes: MergeFileSplitRead.withReadType and
DataEvolutionFileStoreScan.withReadType reset stale projections, and
IncrementalDiffSplitRead projects from the merge read's actual output
type instead of the full table row type.
Raw-value pushdown safety on masked columns (filter/TopN/limit stats
pruning) is deliberately left to a follow-up PR.
Behavior changes are scoped to query-auth.enabled tables. Tables without
query auth and the write path are unaffected. System tables and
vector/full-text search reads do not consult auth rules (pre-existing;
restrict at the permission layer).
A column mask changes the value domain of its target column. A predicate on a masked column therefore evaluates on the masked value (view semantics): matching the raw value would reveal it. This makes every optimization that consumes the column's raw values or raw statistics unsafe on masked columns -- file/split pruning by min-max or partition value, TopN selection, limit pushdown and reader-level row filtering would drop or leak rows with respect to what the query is entitled to see. Stacks on the projection-augmentation change, and follows the pattern ReadBuilderImpl already uses for read-level TopN on query-auth-enabled tables: never hand unsafe state to the reader, rather than pushing it and retracting it later. Scan side: the query filter is deferred to plan()/listPartitionEntries and pushed once through the two-argument SnapshotReader.withFilter. The full filter marks read-time filtering, and only the conjuncts free of masked columns feed statistics and partition pruning. Since masked conjuncts drop rows only at read time, their presence -- even partition-only -- also keeps limit/TopN split pruning off; TopN split pruning additionally skips masked ordering columns. A mask appearing on an already-pushed filter column fails closed, because raw statistics were already consumed. Read side: on query-auth-enabled tables the filter is stored (for executeFilter) but not forwarded to the reader internals; engines re-evaluate data filters on the masked output. The conjuncts on masked columns are evaluated inside the auth read, post-mask, since engines do not re-evaluate the partition filters they consumed. When the query does not project a masked filter column, the read schema is widened by it (same machinery as mask inputs) and projected back out afterwards. Spark statistics-based aggregate pushdown already degrades for authed reads (auth splits are not DataSplits), anchored by a regression test. Scoped to query-auth.enabled tables; reader-level filter pushdown is conservatively off for them even without mask rules, matching the existing read-level TopN behavior.
51f3b9d to
986fa71
Compare
Member
Author
|
Folded into #8570 per review. Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #8570 — review the top commit only; will be rebased once #8570 merges.
Purpose
A column mask changes the value domain of its target column. A predicate on a masked column therefore evaluates on the masked value (view semantics, as in Snowflake/Ranger): matching the raw value would reveal
it. This makes every optimization that consumes the column's raw values or raw statistics unsafe on masked columns — file/split pruning by min-max or partition value, TopN selection, limit pushdown, and
reader-level row filtering would drop or leak rows relative to what the query is entitled to see. This is a pre-existing issue of the merged masking feature (#8447/#8458), independent of #8570.
The design follows the pattern ReadBuilderImpl already uses for read-level TopN on query-auth.enabled tables: never hand unsafe state to the reader, rather than pushing it and retracting it later.
free of masked columns feed statistics and partition pruning. Masked conjuncts (even partition-only ones) keep limit/TopN split pruning off; TopN split pruning also skips masked ordering columns. A mask
appearing on an already-pushed filter column fails closed, since raw statistics were already consumed.
are evaluated inside the auth read, post-mask — engines do not re-evaluate the partition filters they consumed. A masked filter column the query does not project is widened into the read schema (same machinery
as mask inputs in [WIP][core] Fix column masking correctness in query-auth reads #8570) and projected back out.
Scoped to query-auth.enabled tables: they conservatively lose reader-level filter pushdown even without mask rules, matching the existing read-level TopN behavior; scan-level pruning on safe conjuncts is
retained. Tables without query auth are unaffected.
Tests
partition-only filter, masked pk filter not applied on raw key ranges, mask growth on a pushed filter column fails the scan closed while a fresh scan + existing reader return masked-correct results,
TopN/limit/stats-pruning disablement.