GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer by rok · Pull Request #14341 · apache/arrow

rok · 2022-10-07T04:25:43Z

This is to add DELTA_BYTE_ARRAY encoder.

Closes: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

github-actions · 2022-10-07T04:26:03Z

https://issues.apache.org/jira/browse/ARROW-17619

rok · 2022-10-07T04:27:12Z

This is not really review ready yet and needs #14191 and #14293 to merge first.

rok · 2023-01-26T12:35:54Z

note: this should support BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY. PARQUET-2231

github-actions · 2023-02-23T14:15:35Z

Closes: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

mapleFU · 2023-02-28T05:34:01Z

(Can this patch go ahead now?)

mapleFU

Though this patch is still draft, I run it for fun. Seems that here is some issue. May that helps

Co-authored-by: mwish <1506118561@qq.com> Co-authored-by: Gang Wu <ustcwg@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Co-authored-by: mwish <1506118561@qq.com>

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

…tor APIs

pitrou · 2023-08-21T17:02:18Z

I'm going to merge this now.

mapleFU · 2023-08-21T17:12:13Z

Thats a long time, bravo!

rok · 2023-08-21T19:25:04Z

Thanks all for helping this along! I'm very happy we got this in!

anjakefala · 2023-08-21T19:51:46Z

Congratulations, @rok!!!

AlenkaF · 2023-08-22T04:21:38Z

Great work everybody, congrats!!

conbench-apache-arrow · 2023-08-22T04:26:14Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 94bd0d2.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

jorisvandenbossche · 2023-08-22T10:00:35Z

We should probably update the python docstring as well:

arrow/python/pyarrow/parquet/core.py

Lines 822 to 827 in fe750ed

    
           column_encoding : string or dict, default None 
        
               Specify the encoding scheme on a per column basis. 
        
               Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}. 
        
               Certain encodings are only compatible with certain data types. 
        
               Please refer to the encodings section of `Reading and writing Parquet 
        
               files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.

(which was clearly already outdated before this PR as well! ;))

pitrou · 2023-08-22T10:01:24Z

Feel free to open a PR :-)

rok · 2023-08-22T16:13:56Z

I created an issue :) #37312

mapleFU · 2023-11-08T15:00:39Z

-            RETURN_NOT_OK(helper.builder->ReserveData(
-                std::min<int64_t>(len_, helper.chunk_space_remaining)));


Why previously here ReserveData for min(len_, helper.chunk_space_remaining) here, wouldn't len_ be too large @pitrou

I don't understand: are you commented on the removed code? I'd rather not try to understand code that was removed months ago...

See #38437 for context, where this code is being added back partially

I also don't understand them T_T.

Just confused by this change, so I tried to understand the origin code and find out why this cause the regression, what should I do to fix it

github-actions Bot added Component: Documentation Component: C++ Component: Python Component: Parquet labels Oct 7, 2022

rok force-pushed the ARROW-17619 branch from 5ddafef to 8bdb4b9 Compare November 12, 2022 02:16

rok force-pushed the ARROW-17619 branch 2 times, most recently from 37077b0 to 8448815 Compare November 25, 2022 15:27

rok force-pushed the ARROW-17619 branch from 8448815 to 6b0761b Compare December 15, 2022 12:12

asfimport mentioned this pull request Jan 5, 2023

[C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

Closed

rok force-pushed the ARROW-17619 branch from 6b0761b to 2cfa3cc Compare February 10, 2023 19:43

rok force-pushed the ARROW-17619 branch from 2cfa3cc to 7f33ecc Compare February 23, 2023 14:13

github-actions Bot removed the Component: Documentation label Feb 23, 2023

rok changed the title ~~ARROW-17619: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer~~ GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer Feb 23, 2023

mapleFU reviewed Mar 1, 2023

View reviewed changes

Comment thread cpp/src/parquet/encoding.cc Outdated

Comment thread cpp/src/parquet/encoding.cc Outdated

github-actions Bot added awaiting review Awaiting review awaiting committer review Awaiting committer review awaiting changes Awaiting changes and removed awaiting review Awaiting review awaiting committer review Awaiting committer review labels Mar 1, 2023

rok force-pushed the ARROW-17619 branch from 7f33ecc to 75c2fd3 Compare March 1, 2023 17:42

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 1, 2023

rok force-pushed the ARROW-17619 branch from 6062c17 to b677934 Compare March 1, 2023 19:19

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 1, 2023

rok and others added 16 commits August 21, 2023 16:00

random_byte_array etc

74da1b7

Review feedback

92b457a

Apply suggestions from code review

e4b96aa

Co-authored-by: mwish <1506118561@qq.com> Co-authored-by: Gang Wu <ustcwg@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Review feedback

45a6d51

Review feedback

1e10149

Review feedback

6c6fbde

Review feedback

6930a79

Review feedback

409a6ee

Apply suggestions from code review

7dc32e1

Co-authored-by: mwish <1506118561@qq.com>

Switch to resizable buffer

578d7de

Apply suggestions from code review

8167771

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Review feedback

0aaa6b4

Review feedback

9f0cdbb

Review feedback

46aa303

Review feedback

acc40ed

Rewrite Binary/FLBA specializations to correctly account for Accumula…

5039cf9

…tor APIs

pitrou approved these changes Aug 21, 2023

View reviewed changes

mapleFU mentioned this pull request Aug 22, 2023

[C++][Parquet] Encoding: Add Benchmark for DELTA_BYTE_ARRAY #37293

Closed

rok mentioned this pull request Aug 22, 2023

[Python][Docs] Update Python docstrings to reflect new parquet encoding option #37312

Closed

jorisvandenbossche mentioned this pull request Oct 24, 2023

[C++] Parquet reading performance regressions #38432

Closed

mapleFU reviewed Nov 8, 2023

View reviewed changes

		RETURN_NOT_OK(helper.builder->ReserveData(
		std::min<int64_t>(len_, helper.chunk_space_remaining)));

Uh oh!

Conversation

rok commented Oct 7, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2022

Uh oh!

rok commented Oct 7, 2022

Uh oh!

rok commented Jan 26, 2023

Uh oh!

github-actions Bot commented Feb 23, 2023

Uh oh!

mapleFU commented Feb 28, 2023

Uh oh!

mapleFU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pitrou commented Aug 21, 2023

Uh oh!

mapleFU commented Aug 21, 2023

Uh oh!

rok commented Aug 21, 2023

Uh oh!

anjakefala commented Aug 21, 2023

Uh oh!

AlenkaF commented Aug 22, 2023

Uh oh!

conbench-apache-arrow Bot commented Aug 22, 2023

Uh oh!

jorisvandenbossche commented Aug 22, 2023

Uh oh!

pitrou commented Aug 22, 2023

Uh oh!

rok commented Aug 22, 2023

Uh oh!

mapleFU Nov 8, 2023

Choose a reason for hiding this comment

Uh oh!

pitrou Nov 8, 2023

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU Nov 8, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

rok commented Oct 7, 2022 •

edited by github-actions Bot

Loading

jorisvandenbossche Nov 8, 2023 •

edited

Loading