[DISCUSS] Columnar data protocol: Arrow or implement a new one?

[wgtmac]

A columnar data protocol is required in the following cases:

• Reading metadata from snapshot metadata (JSON, Avro)
• Writing metadata to snapshot metadata (JSON, Avro)
• Reading data from data files (Parquet, Avro, ORC)
• Writing data to data files (Parquet, Avro, ORC)

I'd like to use Arrow Array (backed by nanoarrow) for this purpose because

• It is the de-facto standard and accepted by many popular engines.
• nanoarrow library is lightweight and easy to use.

WDYT?

[lidavidm]

I think there's no value to roll-your-own when nanoarrow is small and vendorable. (If arrow-cpp were the only option, then the discussion may be different, but nanoarrow exists & has been adopted in a few places now!)

[pitrou]

There's also sparrow which AFAIK does not implement the full Arrow spec yet, but is actively being developed by QuantStack. cc @JohanMabille @Alex-PLACET

(disclaimer: I work for QuantStack)

[Fokko]

I would be very reluctant to implement the data protocol ourselves. I noticed that nanoarrow is also used by DuckDB, but I don't have enough in-depth knowledge to provide an argument to chose the one over the other :D

[MisterRaindrop]

I am towards those that have already been applied in practical projects. I tend to prefer nanoarrow. However, I want to give some users the option to allow them to customize their own data protocol.

[zeroshade]

I would also be in favor of nanoarrow over rolling your own data protocol. I'll also note that for AI/ML use cases, NVIDIA Rapids (libcudf) uses nanoarrow for importing and exporting data in Arrow format (leveraging the C Device ABIs).

However, I want to give some users the option to allow them to customize their own data protocol.

As long as a user can produce a plan / tasks for a given query, they would be able to use their own customized data protocol for consuming and executing that plan, right? It seems like it would be kinda difficult to really allow full customization of data protocol while still enabling full handling of filtering out deletions, transforms, and so on within this library while still maintaining a sane API for the interfaces internally. At least IMHO

[MisterRaindrop]

I feel this is an attempt to make Iceberg more open and compatible with more scenarios. Of course, this is just my opinion.

[wgtmac]

Thanks for all the suggestions above!

This is my first time to play with nanoarrow. After some research, I found it is super lightweight and convenient to simply use the bundled nanoarrow.hpp and nanoarrow.cc files generated by ci/scripts/bundle.py --cpp --symbol-namespace=iceberg --output-dir /tmp/bundled. I believe we don't need device, ipc and flatcc for iceberg-cpp. However, I'm not sure if this is the recommended approach. Could you please advise? @paleolimbot

I have also checked https://github.com/man-group/sparrow. TBH, I like its design and simplicity but a little bit concerned about its adoption outside of its sponsors. Maybe we need more voices from the project maintainers to make a fair decision? @pitrou

=========

For the columnar data interface of libiceberg, we have two choices:

1. Directly use array API from nanoarrow or sparrow library.
2. Use ArrowArray from the Arrow C data interface and users are free to use their own Arrow implementations.

I prefer option 1 with nanoarrow at this moment. I will create a PR for a PoC.

=========

EDIT: perhaps we need to include --with-device to ci/scripts/bundle.py --with-device --cpp --symbol-namespace=iceberg --output-dir /tmp/bundled if we need to consider GPU? @lidavidm

[pitrou]

I have also checked https://github.com/man-group/sparrow. TBH, I like its design and simplicity but a little bit concerned about its adoption outside of its sponsors. Maybe we need more voices from the project maintainers to make a fair decision? @pitrou

Well, I'm not part of the maintainers so I can't really say anything concrete. Perhaps @JohanMabille wants to shed a bit of light on future prospects for sparrow, and whether it's a good idea to rely on it at this point.

(perhaps sparrow passing the cross-implementation integration tests should be a first milestone before considering it)

It also depends whether Arrow would be a public or private dependency for iceberg-cpp. If a public dependency, going for nanoarrow is certainly safer at this point.

[paleolimbot]

I found it is super lightweight and convenient to simply use the bundled nanoarrow.hpp and nanoarrow.cc files generated by ci/scripts/bundle.py --cpp --symbol-namespace=iceberg --output-dir /tmp/bundled.

That's what we like to hear! We also support CMake via FetchContent and find_package() and meson/wrapdb if any of those are easier 🙂.

(perhaps sparrow passing the cross-implementation integration tests should be a first milestone before considering it)

man-group/sparrow#306 I believe is tracking this one!

All this being said, I think that you want:

2. Use ArrowArray from the Arrow C data interface and users are free to use their own Arrow implementations.

...so that you don't have to force your users to pin a nanoarrow or sparrow version. I don't personally expose nanoarrow.h in any projects I use it in (the interface is always in terms of ArrowArrayStream).

FWIW I also think sparrow is great! We in theory could add more C++ to nanoarrow.hpp too, but there hasn't been much of an appetite to contribute or review (and with some macros and the RAII helpers in nanoarrow.hpp, it's verbose but not bad to just use the C library).

[wgtmac]

It also depends whether Arrow would be a public or private dependency for iceberg-cpp. If a public dependency, going for nanoarrow is certainly safer at this point.

I'd say the ArrowArray C data interface should be public at the minimum (as per the requirement in this issue description). To make it easy to use to the downstream projects, we need a wrapper around the ArrowArray which is exactly the purpose of nanoarrow and sparrow.

All this being said, I think that you want:
Use ArrowArray from the Arrow C data interface and users are free to use their own Arrow implementations.
...so that you don't have to force your users to pin a nanoarrow or sparrow version. I don't personally expose nanoarrow.h in any projects I use it in (the interface is always in terms of ArrowArrayStream).

The downside of this approach (i.e. the option 2) is that downstream projects have to make the decision (choose from nanoarrow, sparrow or bake their own impl) for the Arrow array conversion and manage the dependency by themselves. How about providing two interfaces, one is the ArrowArray C data interface and the other is the nanoarrow::UniqueArray provided by the bundled nanoarrow? Then downstream projects are still free to plugin their own Arrow array implementation to purely use the C data interface.

[paleolimbot]

How about providing two interfaces, one is the ArrowArray C data interface and the other is the nanoarrow::UniqueArray provided by the bundled nanoarrow?

Sure! We do this in ADBC (the API to create a driver is ABI-stable C, but we provide a driver framework where nanoarrow is used to enable more idiomatic C++ implementations). I think it's also safe to expose nanoarrow.h...even though we are pre-1.0.0, we haven't made substantial changes to the API. Feel free to ping me for any reviews/things you need nanoarrow to do 🙂

nanoarrow::UniqueArray

You can also use std::unique_ptr<ArrowArray, (*void)(ArrowArray*)> (i.e., unique pointer with a deleter) if the UniqueXXX bits are all you are exposing in the interface. I believe sparrow and cudf both use this approach.

[wgtmac]

Thanks @paleolimbot! I will keep you updated when we have made progress or need anything from nanoarrow.

[lidavidm]

EDIT: perhaps we need to include --with-device to ci/scripts/bundle.py --with-device --cpp --symbol-namespace=iceberg --output-dir /tmp/bundled if we need to consider GPU? @lidavidm

We can always update the bundling as needed. But yes, I think it would be preferable to use the ArrowDeviceArray interface instead of ArrowArray, at least when dealing with data (I'm not sure metadata needs to go on an accelerator)

[zeroshade]

It also may make sense to consider leveraging the Async C API In some potential APIs.

[lidavidm]

If the core library itself does not do I/O then we can avoid the need to have async/sync copies of everything, IMO (the app would do async I/O then call out to iceberg-core to do the decoding)