Morsel based parquet loader, part 1

[XiangpengHao]

switch to the new api

[codacy-production]

Up to standards ✅

🟢 Issues 7 medium

Results:
7 new issues

Category	Results
Complexity	7 medium

View in Codacy

🟢 Metrics 100 complexity · 11 duplication

Metric	Results
Complexity	100
Duplication	11

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[codecov]

Codecov Report

❌ Patch coverage is 92.46637% with 84 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.67%. Comparing base (cc9ed21) to head (05e53c4).

Files with missing lines	Patch %	Lines
src/datafusion/src/reader/plantime/morselizer.rs	91.27%	52 Missing and 18 partials ⚠️
src/datafusion/src/reader/runtime/morsel.rs	93.33%	5 Missing and 2 partials ⚠️
src/datafusion/src/reader/plantime/source.rs	93.67%	4 Missing and 1 partial ⚠️
src/datafusion/src/optimizers/mod.rs	98.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #513      +/-   ##
==========================================
+ Coverage   83.37%   83.67%   +0.29%     
==========================================
  Files          86       85       -1     
  Lines       19000    18892     -108     
  Branches    19000    18892     -108     
==========================================
- Hits        15841    15807      -34     
+ Misses       2846     2779      -67     
+ Partials      313      306       -7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

📊 Benchmark Comparison

Current: 4a71869f (Liquid) vs Baseline: 4a71869f (DataFusionDefault)

Query	Cold Time	Δ	Warm Time	Δ	CPU Time	Δ
Q1	2.0ms (2.0ms)	+0.0%	0.500ms (0.000ms)	+inf%	0.000ms (0.000ms)	+0.0%
Q2	15.0ms (8.0ms)	+87.5%	6.0ms (6.5ms)	-7.7%	7.0ms (7.0ms)	+0.0%
Q3	20.0ms (12.0ms)	+66.7%	6.0ms (11.5ms)	-47.8%	2.0ms (23.0ms)	-91.3%
Q4	15.0ms (13.0ms)	+15.4%	3.5ms (11.0ms)	-68.2%	1.5ms (23.5ms)	-93.6%
Q5	64.0ms (49.0ms)	+30.6%	51.5ms (50.0ms)	+3.0%	3.0ms (26.0ms)	-88.5%
Q6	162.0ms (98.0ms)	+65.3%	85.0ms (99.5ms)	-14.6%	28.5ms (77.5ms)	-63.2%
Q7	1.0ms (1.0ms)	+0.0%	1.0ms (0.000ms)	+inf%	0.000ms (0.000ms)	+0.0%
Q8	9.0ms (6.0ms)	+50.0%	5.0ms (6.5ms)	-23.1%	7.0ms (7.0ms)	+0.0%
Q9	109.0ms (85.0ms)	+28.2%	87.0ms (82.5ms)	+5.5%	5.0ms (44.0ms)	-88.6%
Q10	101.0ms (84.0ms)	+20.2%	82.0ms (93.5ms)	-12.3%	4.5ms (68.0ms)	-93.4%
Q11	47.0ms (25.0ms)	+88.0%	21.5ms (25.0ms)	-14.0%	36.0ms (35.5ms)	+1.4%
Q12	52.0ms (31.0ms)	+67.7%	24.5ms (29.0ms)	-15.5%	42.0ms (43.0ms)	-2.3%
Q13	208.0ms (105.0ms)	+98.1%	104.0ms (104.0ms)	+0.0%	68.5ms (80.0ms)	-14.4%
Q14	358.0ms (135.0ms)	+165.2%	144.0ms (134.5ms)	+7.1%	109.0ms (104.5ms)	+4.3%
Q15	236.0ms (104.0ms)	+126.9%	87.0ms (99.0ms)	-12.1%	57.5ms (92.5ms)	-37.8%
Q16	99.0ms (99.0ms)	+0.0%	97.5ms (103.0ms)	-5.3%	3.0ms (25.0ms)	-88.0%
Q17	392.0ms (208.0ms)	+88.5%	222.0ms (205.0ms)	+8.3%	81.0ms (102.5ms)	-21.0%
Q18	376.0ms (198.0ms)	+89.9%	221.0ms (203.0ms)	+8.9%	85.5ms (102.5ms)	-16.6%
Q19	633.0ms (420.0ms)	+50.7%	390.5ms (386.0ms)	+1.2%	101.5ms (148.5ms)	-31.6%
Q20	14.0ms (13.0ms)	+7.7%	3.0ms (12.5ms)	-76.0%	6.0ms (25.5ms)	-76.5%
Q21	782.0ms (172.0ms)	+354.7%	319.0ms (173.0ms)	+84.4%	495.5ms (269.5ms)	+83.9%
Q22	1.17s (166.0ms)	+602.4%	376.5ms (166.0ms)	+126.8%	167.0ms (337.0ms)	-50.4%
Q23	2.33s (454.0ms)	+413.0%	1.06s (459.5ms)	+130.6%	495.0ms (731.5ms)	-32.3%
Q24	21.45s (875.0ms)	+2351.4%	823.5ms (926.0ms)	-11.1%	628.0ms (2.51s)	-75.0%
Q25	202.0ms (72.0ms)	+180.6%	14.5ms (55.5ms)	-73.9%	35.5ms (114.5ms)	-69.0%
Q26	86.0ms (47.0ms)	+83.0%	19.0ms (45.0ms)	-57.8%	48.0ms (81.5ms)	-41.1%
Q27	173.0ms (56.0ms)	+208.9%	26.5ms (58.5ms)	-54.7%	76.0ms (116.0ms)	-34.5%
Q28	975.0ms (214.0ms)	+355.6%	287.0ms (210.5ms)	+36.3%	357.0ms (270.0ms)	+32.2%
Q29	1.81s (962.0ms)	+88.3%	1.04s (971.0ms)	+7.2%	576.0ms (339.0ms)	+69.9%
Q30	28.0ms (26.0ms)	+7.7%	26.0ms (30.5ms)	-14.8%	5.0ms (21.5ms)	-76.7%
Q31	282.0ms (107.0ms)	+163.6%	67.0ms (98.5ms)	-32.0%	41.0ms (143.0ms)	-71.3%
Q32	533.0ms (98.0ms)	+443.9%	89.0ms (96.0ms)	-7.3%	52.5ms (144.5ms)	-63.7%
Q33	313.0ms (306.0ms)	+2.3%	293.0ms (292.5ms)	+0.2%	6.5ms (70.5ms)	-90.8%
Q34	979.0ms (408.0ms)	+140.0%	487.0ms (408.0ms)	+19.4%	347.0ms (268.0ms)	+29.5%
Q35	948.0ms (387.0ms)	+145.0%	452.5ms (390.5ms)	+15.9%	350.5ms (277.0ms)	+26.5%
Q36	96.0ms (95.0ms)	+1.1%	86.0ms (90.5ms)	-5.0%	3.0ms (23.5ms)	-87.2%
Q37	314.0ms (92.0ms)	+241.3%	80.5ms (95.0ms)	-15.3%	44.0ms (70.0ms)	-37.1%
Q38	68.0ms (42.0ms)	+61.9%	27.5ms (42.5ms)	-35.3%	16.0ms (23.0ms)	-30.4%
Q39	267.0ms (46.0ms)	+480.4%	12.5ms (48.0ms)	-74.0%	10.5ms (72.0ms)	-85.4%
Q40	756.0ms (180.0ms)	+320.0%	209.0ms (175.0ms)	+19.4%	83.5ms (122.5ms)	-31.8%
Q41	25.0ms (22.0ms)	+13.6%	11.0ms (20.0ms)	-45.0%	6.0ms (17.5ms)	-65.7%
Q42	23.0ms (18.0ms)	+27.8%	9.5ms (17.5ms)	-45.7%	7.0ms (14.0ms)	-50.0%
Q43	33.0ms (18.0ms)	+83.3%	11.5ms (15.0ms)	-23.3%	7.0ms (10.0ms)	-30.0%

⚠️ LiquidCache is slower on 9 queries (warm)

• Q1: warm +inf% (0.500ms vs 0.000ms)
• Q7: warm +inf% (1.0ms vs 0.000ms)
• Q23: warm +130.6% (1.06s vs 459.5ms)
• Q22: warm +126.8% (376.5ms vs 166.0ms)
• Q21: warm +84.4% (319.0ms vs 173.0ms)
• Q28: warm +36.3% (287.0ms vs 210.5ms)
• Q40: warm +19.4% (209.0ms vs 175.0ms)
• Q34: warm +19.4% (487.0ms vs 408.0ms)
• Q35: warm +15.9% (452.5ms vs 390.5ms)

Compared Liquid vs DataFusionDefault on the same runner
Regressions: warm-time increases of at least 15%. Cold Time: first iteration; Warm Time: median of remaining iterations.