Skip to content

List group members level by level instead of enumerating all keys - #92

Open
konstibob wants to merge 1 commit into
zarr-developers:mainfrom
konstibob:fix-group-list-levelwise
Open

List group members level by level instead of enumerating all keys#92
konstibob wants to merge 1 commit into
zarr-developers:mainfrom
konstibob:fix-group-list-levelwise

Conversation

@konstibob

Copy link
Copy Markdown
Contributor

Group.list() called storeHandle.list(), which is fully recursive and returns every key below the group, including all chunk keys. Listing a group with 1M chunks on S3 cost ~1000 paginated requests to find a handful of zarr.json files.

Walk the hierarchy with listChildren() instead, one level at a time, and never descend into arrays. The implementation moves to core.Group, so v2 and v3 share it, and adds members() for the immediate children only.

Also:

  • S3Store.listChildren() used a single listObjectsV2 call, silently truncating groups with more than 1000 children. Use the paginator.
  • FilesystemStore.get() threw instead of returning null when a path component is a file rather than a directory. Probing a plain file next to a group's nodes is normal now, and such a key holds no data.

Group.list() called storeHandle.list(), which is fully recursive and
returns every key below the group, including all chunk keys. Listing a
group with 1M chunks on S3 cost ~1000 paginated requests to find a
handful of zarr.json files.

Walk the hierarchy with listChildren() instead, one level at a time, and
never descend into arrays. The implementation moves to core.Group, so v2
and v3 share it, and adds members() for the immediate children only.

Also:
- S3Store.listChildren() used a single listObjectsV2 call, silently
  truncating groups with more than 1000 children. Use the paginator.
- FilesystemStore.get() threw instead of returning null when a path
  component is a file rather than a directory. Probing a plain file next
  to a group's nodes is normal now, and such a key holds no data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant