List group members level by level instead of enumerating all keys - #92
Open
konstibob wants to merge 1 commit into
Open
List group members level by level instead of enumerating all keys#92konstibob wants to merge 1 commit into
konstibob wants to merge 1 commit into
Conversation
Group.list() called storeHandle.list(), which is fully recursive and returns every key below the group, including all chunk keys. Listing a group with 1M chunks on S3 cost ~1000 paginated requests to find a handful of zarr.json files. Walk the hierarchy with listChildren() instead, one level at a time, and never descend into arrays. The implementation moves to core.Group, so v2 and v3 share it, and adds members() for the immediate children only. Also: - S3Store.listChildren() used a single listObjectsV2 call, silently truncating groups with more than 1000 children. Use the paginator. - FilesystemStore.get() threw instead of returning null when a path component is a file rather than a directory. Probing a plain file next to a group's nodes is normal now, and such a key holds no data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Group.list() called storeHandle.list(), which is fully recursive and returns every key below the group, including all chunk keys. Listing a group with 1M chunks on S3 cost ~1000 paginated requests to find a handful of zarr.json files.
Walk the hierarchy with listChildren() instead, one level at a time, and never descend into arrays. The implementation moves to core.Group, so v2 and v3 share it, and adds members() for the immediate children only.
Also: