Skip to content

Add:arrow_metadata() UDF - #19435

Merged
comphead merged 3 commits into
apache:mainfrom
xonx4l:add_arrow_metadata
Dec 23, 2025
Merged

Add:arrow_metadata() UDF#19435
comphead merged 3 commits into
apache:mainfrom
xonx4l:add_arrow_metadata

Conversation

@xonx4l

@xonx4l xonx4l commented Dec 21, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #19356

Rationale for this change

This PR implements the arrow_metadata UDF as requested in issue #19356.

What changes are included in this PR?

Added arrow_metadata UDF
Refactored Tests

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Dec 21, 2025
@Jefffrey Jefffrey mentioned this pull request Dec 21, 2025
Closed
Jefffrey
Jefffrey reviewed Dec 21, 2025
View reviewed changes

@Jefffrey Jefffrey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we just need to add some documentation, see an example from arrow_typeof here:

datafusion/datafusion/functions/src/core/arrowtypeof.rs

Lines 25 to 42 in 2c3566c

#[user_doc(
doc_section(label = "Other Functions"),
description = "Returns the name of the underlying [Arrow data type](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html) of the expression.",
syntax_example = "arrow_typeof(expression)",
sql_example = r#"```sql
> select arrow_typeof('foo'), arrow_typeof(1);
+---------------------------+------------------------+
| arrow_typeof(Utf8("foo")) | arrow_typeof(Int64(1)) |
+---------------------------+------------------------+
| Utf8 | Int64 |
+---------------------------+------------------------+
```
"#,
argument(
name = "expression",
description = "Expression to evaluate. The expression can be a constant, column, or function, and any combination of operators."
)
)]

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Dec 21, 2025
@xonx4l

xonx4l commented Dec 21, 2025

Copy link
Copy Markdown
Contributor Author

@Jefffrey Done.

@comphead
comphead added this pull request to the merge queue Dec 23, 2025
Merged via the queue into apache:main with commit d844f86 Dec 23, 2025
28 checks passed
@adriangb adriangb mentioned this pull request Apr 5, 2026
Merged
github-merge-queue Bot pushed a commit that referenced this pull request Apr 14, 2026
@adriangb @claude @martin-g
Add arrow_field(expr) scalar UDF (#21389)
41dd942 
## Which issue does this PR close?

- Related to #21387 and #19435

## Rationale for this change

DataFusion has individual introspection functions (`arrow_typeof`,
`arrow_metadata`, `is_nullable`) but no single function that returns the
complete Arrow `Field` representation. Having `arrow_field(expr)` that
returns a struct with all field info avoids multiple function calls when
you need the full picture, and provides a natural complement to the
existing introspection suite.

## What changes are included in this PR?

Adds `arrow_field(expr)` scalar UDF that returns a struct:

```sql
> SELECT arrow_field(x) FROM my_table;
{name: x, data_type: Int32, nullable: false, metadata: {}}

> SELECT arrow_field(x)['data_type'] FROM my_table;
Int32
```

The returned struct has four fields:
- `name` (Utf8) — the field name
- `data_type` (Utf8) — the Arrow data type as string
- `nullable` (Boolean) — whether the field is nullable
- `metadata` (Map<Utf8, Utf8>) — the field metadata

Individual fields are accessible via bracket syntax.

**Files:**
- `datafusion/functions/src/core/arrow_field.rs` — new UDF
implementation
- `datafusion/functions/src/core/mod.rs` — registration
- `datafusion/sqllogictest/test_files/arrow_field.slt` — tests

## Are these changes tested?

Yes, sqllogictest covering literals (int, null, bool, string, float,
list), table columns, nullability, and struct field access.

## Are there any user-facing changes?

New SQL function `arrow_field(expr)` is available.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
coderfender pushed a commit to coderfender/datafusion that referenced this pull request Apr 14, 2026
@adriangb @claude
@martin-g B Vadlamani
Add arrow_field(expr) scalar UDF (apache#21389)
d9a5386 
## Which issue does this PR close?

- Related to apache#21387 and apache#19435

## Rationale for this change

DataFusion has individual introspection functions (`arrow_typeof`,
`arrow_metadata`, `is_nullable`) but no single function that returns the
complete Arrow `Field` representation. Having `arrow_field(expr)` that
returns a struct with all field info avoids multiple function calls when
you need the full picture, and provides a natural complement to the
existing introspection suite.

## What changes are included in this PR?

Adds `arrow_field(expr)` scalar UDF that returns a struct:

```sql
> SELECT arrow_field(x) FROM my_table;
{name: x, data_type: Int32, nullable: false, metadata: {}}

> SELECT arrow_field(x)['data_type'] FROM my_table;
Int32
```

The returned struct has four fields:
- `name` (Utf8) — the field name
- `data_type` (Utf8) — the Arrow data type as string
- `nullable` (Boolean) — whether the field is nullable
- `metadata` (Map<Utf8, Utf8>) — the field metadata

Individual fields are accessible via bracket syntax.

**Files:**
- `datafusion/functions/src/core/arrow_field.rs` — new UDF
implementation
- `datafusion/functions/src/core/mod.rs` — registration
- `datafusion/sqllogictest/test_files/arrow_field.slt` — tests

## Are these changes tested?

Yes, sqllogictest covering literals (int, null, bool, string, float,
list), table columns, nullability, and struct field access.

## Are there any user-facing changes?

New SQL function `arrow_field(expr)` is available.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Rich-T-kid pushed a commit to Rich-T-kid/datafusion that referenced this pull request Apr 21, 2026
@adriangb @claude
@martin-g @Rich-T-kid
Add arrow_field(expr) scalar UDF (apache#21389)
a044497 
## Which issue does this PR close?

- Related to apache#21387 and apache#19435

## Rationale for this change

DataFusion has individual introspection functions (`arrow_typeof`,
`arrow_metadata`, `is_nullable`) but no single function that returns the
complete Arrow `Field` representation. Having `arrow_field(expr)` that
returns a struct with all field info avoids multiple function calls when
you need the full picture, and provides a natural complement to the
existing introspection suite.

## What changes are included in this PR?

Adds `arrow_field(expr)` scalar UDF that returns a struct:

```sql
> SELECT arrow_field(x) FROM my_table;
{name: x, data_type: Int32, nullable: false, metadata: {}}

> SELECT arrow_field(x)['data_type'] FROM my_table;
Int32
```

The returned struct has four fields:
- `name` (Utf8) — the field name
- `data_type` (Utf8) — the Arrow data type as string
- `nullable` (Boolean) — whether the field is nullable
- `metadata` (Map<Utf8, Utf8>) — the field metadata

Individual fields are accessible via bracket syntax.

**Files:**
- `datafusion/functions/src/core/arrow_field.rs` — new UDF
implementation
- `datafusion/functions/src/core/mod.rs` — registration
- `datafusion/sqllogictest/test_files/arrow_field.slt` — tests

## Are these changes tested?

Yes, sqllogictest covering literals (int, null, bool, string, float,
list), table columns, nullability, and struct field access.

## Are there any user-facing changes?

New SQL function `arrow_field(expr)` is available.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jefffrey Jefffrey Jefffrey approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add arrow_metadata(<expr>) UDF

3 participants

@xonx4l @Jefffrey @comphead