Skip to content

[GH-3609]: Add new sort order for int96 timestamps#3610

Open
divjotarora wants to merge 6 commits into
apache:masterfrom
divjotarora:int96-stats
Open

[GH-3609]: Add new sort order for int96 timestamps#3610
divjotarora wants to merge 6 commits into
apache:masterfrom
divjotarora:int96-stats

Conversation

@divjotarora

@divjotarora divjotarora commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

See parquet-format proposal for rationale: apache/parquet-format#584

What changes are included in this PR?

Add new INT96_TIMESTAMP_ORDER sort order and related parsing.

Are these changes tested?

Yes, new unit tests.

Are there any user-facing changes?

Two new flags and a Thrift change.

Closes #3609

@divjotarora divjotarora mentioned this pull request Jun 11, 2026
Merged
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/ColumnOrder.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveComparator.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveType.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/InternalParquetRecordWriter.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java Outdated
@divjotarora divjotarora requested a review from rdblue June 15, 2026 19:20
@divjotarora divjotarora mentioned this pull request Jun 23, 2026
Draft
@wgtmac

wgtmac commented Jun 25, 2026

Copy link
Copy Markdown
Member

Please resolve the conflicts.

@divjotarora

divjotarora commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Please resolve the conflicts.

@wgtmac Sorry for the delay, I was on holiday last week. Conflicts are resolved now

rdblue
rdblue reviewed Jul 10, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveType.java Outdated
rdblue
rdblue reviewed Jul 10, 2026
View reviewed changes
Comment thread parquet-column/src/test/java/org/apache/parquet/schema/TestPrimitiveComparator.java Outdated
rdblue
rdblue reviewed Jul 10, 2026
View reviewed changes
Comment thread parquet-column/src/test/java/org/apache/parquet/schema/TestPrimitiveComparator.java Outdated
@rdblue

rdblue commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

I think there are a few things to be fixed, but overall I trust that this works.

@divjotarora divjotarora requested a review from rdblue July 13, 2026 16:51
rdblue
rdblue reviewed Jul 14, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveComparator.java Outdated
rdblue
rdblue reviewed Jul 14, 2026
View reviewed changes
Comment thread parquet-column/src/test/java/org/apache/parquet/schema/TestPrimitiveComparator.java Outdated
rdblue
rdblue reviewed Jul 14, 2026
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
case UNDEFINED:
return false;
default:
throw new IllegalArgumentException("Unknown column order: " + type.columnOrder());

@rdblue rdblue Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I think we would normal throw UnsupportedOperationException since this is a valid argument (it's an enum after all).

@divjotarora divjotarora Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer to leave IllegalArgumentException to match getColumnOrders above

@divjotarora divjotarora requested a review from rdblue July 15, 2026 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rdblue rdblue Awaiting requested review from rdblue

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add new sort order for int96 timestamps

3 participants

@divjotarora @wgtmac @rdblue