[fix] Apply row filter auth during scan planning by plusplusjiajia · Pull Request #8447 · apache/paimon

plusplusjiajia · 2026-07-03T09:01:16Z

Purpose

Apply row filter auth predicates during scan planning, not only at read time. This lets auth filters participate in partition pruning, non-partition filter pushdown, and limit pushdown decisions, while keeping
reader-side auth filtering as final enforcement.

This fixes cases where a partition row filter allows data in some partitions, but queries without explicit partition predicates may plan splits from unauthorized partitions first and return empty results.

…() calls

JingsongLi · 2026-07-03T12:13:05Z

+1

JingsongLi · 2026-07-03T13:12:54Z

                        predicate, tableSchema.logicalRowType(), tableSchema.partitionKeys());
        if (pair.getLeft().isPresent()) {
-            scan.withPartitionFilter(pair.getLeft().get());
+            scan.withPartitionFilterAnd(pair.getLeft().get());


Do we still need to introduce withPartitionFilterAnd here? Since auth-derived partition predicates are now kept in a separate authPartitionFilter slot and combined with the normal partitionFilter only when computing the effective manifest filter, this PR could keep the existing withPartitionFilter(...) behavior for normal query filters. The repeated withFilter(...) partition-merge semantics feels like a separate behavior change that can be discussed independently if needed.

@JingsongLi Agree. withPartitionFilterAnd is no longer needed for this PR now that authPartitionFilter is separate. I'll revert that part and keep the existing withPartitionFilter(...) behavior for normal query filters.

JingsongLi · 2026-07-03T13:20:38Z

+        snapshotReader.manifestsReader().withAuthPartitionFilter(authPartitionFilter);
+        if (hasNonPartitionPart) {
+            // Non-partition auth removes rows at read time, so limit push down is unsafe.
+            snapshotReader.markHasNonPartitionFilter();


If this marker is only used to make DataTableBatchScan.applyPushDownLimit() return empty, can we keep this state local to DataTableBatchScan instead? For example, when the auth predicate has a non-partition part, set pushDownLimit = null before planning. null already means no limit pushdown there, and this avoids adding markHasNonPartitionFilter() to SnapshotReader for a read-time auth filter that was not actually pushed through withFilter(...).

@JingsongLi good point! dropped it from SnapshotReader, kept the flag scan-local in DataTableBatchScan.

Drop withPartitionFilterAnd; keep the non-partition-auth flag scan-local.

Always call applyAuthFilter so a null auth result clears stale pruning, and clear the file-level limit when auth has a non-partition part so LIMIT reads past unauthorized rows.

Query auth filters rows at read time, so a read-side limit or TopN applied before auth could drop authorized rows. Skip both in ReadBuilder.newRead when query-auth is enabled, and only push the scan file-store limit when no read-time filter is present.

Apply the limit after auth via a LimitTableRead wrapper instead of before, which would drop authorized rows.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. The required columns are computed by the new TableQueryAuthResult.requiredAuthFields: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. The widening no longer goes through withReadType: authedReader now calls applyReadType directly so this.readType keeps the query's projection. Previously the first createReader call mutated this.readType to the widened type, so later splits of the same TableRead saw nothing to add, skipped the back-projection, and leaked the auth-added columns (row arity change plus unprojected data exposure); executeFilter similarly remapped user predicates against the widened type while rows had already been narrowed. Both stay consistent now, also for the pre-existing row-filter path. The parsed filter and masking are cached in transient fields of TableQueryAuthResult, instead of JSON-deserializing every rule twice per split.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. The required columns are computed by the new TableQueryAuthResult.requiredAuthFields: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. The widening no longer goes through withReadType: authedReader now calls applyReadType directly so this.readType keeps the query's projection. Previously the first createReader call mutated this.readType to the widened type, so later splits of the same TableRead saw nothing to add, skipped the back-projection, and leaked the auth-added columns (row arity change plus unprojected data exposure); executeFilter similarly remapped user predicates against the widened type while rows had already been narrowed. Both stay consistent now, also for the pre-existing row-filter path. The parsed filter and masking are cached in transient fields of TableQueryAuthResult, instead of JSON-deserializing every rule twice per split. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the needed columns: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Masks referencing columns absent from the table schema (stale rules after a column rename or drop) fail closed with a clear error instead of silently returning unmasked data. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers and ignore later read-type changes, the widened read schema is frozen once the first split reader exists; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, and rules that change after the freeze are remapped by name against the frozen schema, failing closed if they need more columns. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The parsed filter and masking are cached in transient fields of TableQueryAuthResult and the widening is memoized per TableRead on the auth rules, instead of JSON-deserializing and recomputing per split. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the needed columns: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table schema (stale rules after a column rename or drop) fail closed with a clear error instead of silently returning unmasked data: all rules are checked, because a rule keyed by a since-renamed column is indistinguishable from an unprojected one at read time. Validation runs once at plan time on the coordinator, and again defensively before each reader is created. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader exists; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. withReadType rejects changes once a reader exists, making the invariant total. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the needed columns: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table schema (stale rules after a column rename or drop) fail closed with a clear error instead of silently returning unmasked data: all rules are checked, because a rule keyed by a since-renamed column is indistinguishable from an unprojected one at read time. Validation runs once at plan time on the coordinator, and again defensively before each reader is created. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader exists; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. Validation deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is also rejected, as its transform would silently compute from a partial value. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The scan also pushes the auth-widened read type to the snapshot reader before planning, so file-level column pruning (e.g. data evolution column files) keeps the files of the columns the rules read instead of null-filling them. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The scan also pushes the auth-widened read type to the snapshot reader before planning, so file-level column pruning (e.g. data evolution column files) keeps the files of the columns the rules read instead of null-filling them; DataEvolutionFileStoreScan.withReadType resets its pruning type on a re-configuration whose type carries no user columns, instead of retaining the previous one. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The scan also pushes the auth-widened read type to the snapshot reader before planning, so file-level column pruning (e.g. data evolution column files) keeps the files of the columns the rules read instead of null-filling them; DataEvolutionFileStoreScan.withReadType resets its pruning type on a re-configuration whose type carries no user columns, instead of retaining the previous one, and the scan never narrows its pushed read type within its lifetime, since readers fix their schema on first use. IncrementalDiffSplitRead projects from the merge read's actual output type rather than assuming the full table row type, as the shared merge read may itself be projected. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. Rules reading an unprojected blob-view column are rejected too: auth-added columns bypass the blob-view resolution, which is decided on the query's projection. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The scan also pushes the auth-widened read type to the snapshot reader before planning, so file-level column pruning (e.g. data evolution column files) keeps the files of the columns the rules read instead of null-filling them; DataEvolutionFileStoreScan.withReadType resets its pruning type on a re-configuration whose type carries no user columns, instead of retaining the previous one, and the scan never narrows its pushed read type within its lifetime, since readers fix their schema on first use. IncrementalDiffSplitRead projects from the merge read's actual output type rather than assuming the full table row type, as the shared merge read may itself be projected. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. Rules reading an unprojected blob-view column are rejected too: auth-added columns bypass the blob-view resolution, which is decided on the query's projection. TopN split pruning is skipped when the ordering column is a mask target, since the pruning reads raw statistics that the mask may invalidate. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The scan also pushes the auth-widened read type to the snapshot reader before planning, so file-level column pruning (e.g. data evolution column files) keeps the files of the columns the rules read instead of null-filling them; DataEvolutionFileStoreScan.withReadType resets its pruning type on a re-configuration whose type carries no user columns, instead of retaining the previous one, and the scan never narrows its pushed read type within its lifetime, since readers fix their schema on first use. IncrementalDiffSplitRead projects from the merge read's actual output type rather than assuming the full table row type, as the shared merge read may itself be projected. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. System fields (e.g. _ROW_ID) are readable metadata absent from the table schema and stay valid when the query projects them; unprojected ones are rejected at plan time, since they cannot be widened in. Validation re-runs only when the fetched rules change, so steady-state streaming plans skip the schema listing. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. Rules reading an unprojected blob-view column are rejected too: auth-added columns bypass the blob-view resolution, which is decided on the query's projection. Pushdowns that evaluate raw values are kept off masked columns, since a mask may invalidate them: TopN split pruning is skipped when the ordering column is a mask target, the scan pushes the query filter without the conjuncts on masked columns (the full filter still applies on masked values in executeFilter), and split reads drop a pushed TopN or the masked filter conjuncts before the first reader exists. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The scan also pushes the auth-widened read type to the snapshot reader before planning, so file-level column pruning (e.g. data evolution column files) keeps the files of the columns the rules read instead of null-filling them; DataEvolutionFileStoreScan.withReadType resets its pruning type on a re-configuration whose type carries no user columns, instead of retaining the previous one, and the scan never narrows its pushed read type within its lifetime, since readers fix their schema on first use. IncrementalDiffSplitRead projects from the merge read's actual output type rather than assuming the full table row type, as the shared merge read may itself be projected. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (its transform reads columns other than the masked target) failed with "Column masking refers to field '...' which is not present in output row type" whenever the query projected the masked target but not the mask's input columns. Row-filter operands are already added to the read projection and projected back out afterwards (apache#8447); do the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields computes the columns the rules read: filter operands plus, transitively, the inputs of every mask whose target is readable -- projected, or itself pulled into the read type by the filter or another mask -- so a mask on an unprojected filter operand also works instead of throwing. Rules referencing columns absent from the table's latest schema (stale rules after a column rename or drop) fail closed with a clear error at plan time instead of silently returning unmasked data; the fix for such rules lives on the auth server. The validation sits in AbstractDataTableScan.authQuery, the common path of every scan -- including read-optimized and lookup scans -- so no scan factory can bypass it. It deliberately uses the latest schema, not a time-travel-pinned one: rules are fetched fresh, and a rule on a column added after an old snapshot is valid there yet absent from the pinned schema -- such rules are safely skipped at read time, since the column does not exist in the old output. System fields (e.g. _ROW_ID) are readable metadata absent from the table schema and stay valid when the query projects them; unprojected ones are rejected at plan time, since they cannot be widened in. Validation re-runs only when the fetched rules change, so steady-state streaming plans skip the schema listing. A rule that reads a nested-pruned projected column is rejected, as its transform would silently compute from a partial value, and so is a mask targeting a nested-pruned projected column, whose full-value output would be written into a partial slot. Rules reading an unprojected blob-view column are rejected too: auth-added columns bypass the blob-view resolution, which is decided on the query's projection. Pushdowns that evaluate raw values are kept off masked columns, since a mask may invalidate them: TopN split pruning is skipped when the ordering column is a mask target, the scan pushes the query filter without the conjuncts on masked columns -- via the two-argument SnapshotReader.withFilter, so the full filter still marks the read-time filtering and keeps limit/TopN pruning off, and partition listing (which bypasses plan) pushes it too -- and split reads drop a pushed TopN or the masked filter conjuncts before the first reader exists. The widening appends the missing table columns to the projected fields as-is, preserving any nested pruning the engine pushed into them, and no longer goes through withReadType, so this.readType keeps the query's projection. Since split reads cache their format readers by (schemaId, format) and ignore later read-type changes, the read schema is fixed once the first split reader has been created successfully; every split -- including splits without auth rules, e.g. after the rules were revoked mid-stream -- is projected back to the query's read type, rules that change after that and still fit the fixed schema are remapped by name, and rules that would need more columns fail with an explicit "recreate the reader" error. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns (row arity change plus unprojected data exposure) and executeFilter remapped user predicates against the widened type while rows had already been narrowed. The scan also pushes the auth-widened read type to the snapshot reader before planning, so file-level column pruning (e.g. data evolution column files) keeps the files of the columns the rules read instead of null-filling them; DataEvolutionFileStoreScan.withReadType resets its pruning type on a re-configuration whose type carries no user columns, instead of retaining the previous one, and the scan never narrows its pushed read type within its lifetime, since readers fix their schema on first use. IncrementalDiffSplitRead projects from the merge read's actual output type rather than assuming the full table row type, as the shared merge read may itself be projected. Masks are applied only to columns readable from the query -- the projection plus the fields the rules read -- so a column merely retained in a previously widened read schema does not activate a mask after a rules change. MergeFileSplitRead.withReadType now resets its previous outer projection when a later read type needs no adjustment, so re-configured reads (e.g. lookup) emit the type they advertise. The parsed filter and masking are cached in transient fields of TableQueryAuthResult instead of being JSON-deserialized twice per split; the widening itself is deliberately recomputed per split (a cheap set pass) rather than memoized, so no cached projection can survive a rules change. Flink and Spark ITCases cover the cross-column mask under SQL projection pushdown across multiple splits, where the engines reuse one TableRead per task.

A cross-column mask (e.g. display := concat_ws('-', first, last)) threw at read time when the query projected the masked target but not the mask's input columns: "Column masking refers to field 'first' which is not present in output row type". Row-filter operands are already added to the read projection and projected back out (apache#8447); this does the same for column-mask inputs. TableQueryAuthResult.requiredAuthFields collects filter operands plus, transitively, the inputs of every mask whose target is readable; the widening appends the missing table columns as-is (preserving nested pruning), and the scan pushes the widened read type before planning so file-level column pruning (e.g. data evolution) keeps the files those columns live in. Stale rules (columns absent from the latest schema, e.g. after a rename or drop) fail closed once at plan time in AbstractDataTableScan.authQuery, the common path of every scan; re-validated only when the fetched rules change. Projected system fields (e.g. _ROW_ID) stay valid, unprojected ones are rejected; rules touching a nested-pruned or unprojected blob-view column are rejected too. The read schema is fixed once the first split reader exists (split reads cache their format readers and ignore later read-type changes); every split is projected back to the query's read type, and masks apply only to columns readable from the query, so a column merely retained in a previously widened schema does not activate a mask after a rules change. Previously the first createReader call mutated this.readType, so later splits of the same TableRead leaked the auth-added columns -- also on the pre-existing row-filter path. Related re-configuration fixes: MergeFileSplitRead.withReadType and DataEvolutionFileStoreScan.withReadType reset stale projections, and IncrementalDiffSplitRead projects from the merge read's actual output type instead of the full table row type. Raw-value pushdown safety on masked columns (filter/TopN/limit stats pruning) is deliberately left to a follow-up PR. Behavior changes are scoped to query-auth.enabled tables. Tables without query auth and the write path are unaffected. System tables and vector/full-text search reads do not consult auth rules (pre-existing; restrict at the permission layer).

plusplusjiajia force-pushed the fix-rowfilter branch from beb792b to 4a89e2c Compare July 3, 2026 09:30

[fix] Apply row filter auth during scan planning

88a2703

plusplusjiajia force-pushed the fix-rowfilter branch 2 times, most recently from 7714496 to dbfa05d Compare July 3, 2026 11:52

[fix] Reset stale auth partition filter when auth changes across plan…

f8affc0

…() calls

plusplusjiajia force-pushed the fix-rowfilter branch from dbfa05d to f8affc0 Compare July 3, 2026 12:07

JingsongLi approved these changes Jul 3, 2026

View reviewed changes

JingsongLi reviewed Jul 3, 2026

View reviewed changes

[core] Address review comments

9e4edbe

Drop withPartitionFilterAnd; keep the non-partition-auth flag scan-local.

plusplusjiajia force-pushed the fix-rowfilter branch 2 times, most recently from eb1ea23 to af693f5 Compare July 3, 2026 14:34

[core] Fix auth filter and limit handling in scan planning

ae7c392

Always call applyAuthFilter so a null auth result clears stale pruning, and clear the file-level limit when auth has a non-partition part so LIMIT reads past unauthorized rows.

plusplusjiajia force-pushed the fix-rowfilter branch from af693f5 to ae7c392 Compare July 3, 2026 14:49

plusplusjiajia force-pushed the fix-rowfilter branch from 900e032 to eed3b8d Compare July 3, 2026 17:03

[core] Enforce ReadBuilder limit after query auth

adfe282

Apply the limit after auth via a LimitTableRead wrapper instead of before, which would drop authorized rows.

plusplusjiajia force-pushed the fix-rowfilter branch from eed3b8d to adfe282 Compare July 3, 2026 17:24

JingsongLi merged commit 113b311 into apache:master Jul 4, 2026
12 checks passed

This was referenced Jul 6, 2026

[python] Support query auth (row filter & column masking) for REST catalog #8136

Open

[WIP][core] Fix column masking correctness in query-auth reads #8570

Draft

plusplusjiajia mentioned this pull request Jul 12, 2026

[core] Keep raw-value pushdown off masked columns in query auth #8582

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix] Apply row filter auth during scan planning#8447

[fix] Apply row filter auth during scan planning#8447
JingsongLi merged 6 commits into
apache:masterfrom
plusplusjiajia:fix-rowfilter

plusplusjiajia commented Jul 3, 2026

Uh oh!

JingsongLi commented Jul 3, 2026

Uh oh!

JingsongLi Jul 3, 2026 •

edited

Loading

Uh oh!

plusplusjiajia Jul 3, 2026

Uh oh!

JingsongLi Jul 3, 2026

Uh oh!

plusplusjiajia Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

plusplusjiajia commented Jul 3, 2026

Purpose

Uh oh!

JingsongLi commented Jul 3, 2026

Uh oh!

JingsongLi Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JingsongLi Jul 3, 2026 •

edited

Loading