Add Internet Archive filesystem implementation - #2121
Draft
lfoppiano wants to merge 2 commits into
Draft
Conversation
This was referenced Sep 8, 2026
Member
|
This looks like fun - please let me know when you are ready. Are you affiliated with IA in any way? If not, should we approach someone for their opinion? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
InternetArchiveFileSystem(fsspec/implementations/ia.py, protocol ia):ia://<identifier>/<filename>for files in archive.org items, plus registry entry, API docs, changelog and tests.It remap to HTTP (HTTPFileSystem) with a path mapping and credentials, from https://archive.org/download//, which redirects to a data node that support HTTP Range queries. That is the same URL the
internetarchivepackage downloads from.S3-like API (s3.us.archive.org, which an S3FileSystem + endpoint_url would use): Internet Archive data nodes ignore Range and answer a ranged GET with 200 and the whole file, so every block read would download the entire object. Measured on a 1.6 GB public item (Range: bytes=0-99 on every request):
IA documents this: "HTTP 1.1 Range headers are ignored" (https://archive.org/developers/ias3.html#how-this-is-different-from-normal-s3). botocore also misreads the 307 as an AWS region redirect and recurses on HeadBucket until RecursionError.
Credentials. Public items need none. For restricted items the class reads the ia.ini written by ia configure, found the way the internetarchive package finds it ($IA_CONFIG_FILE, $XDG_CONFIG_HOME/internetarchive/ia.ini, ~/.config/ia.ini, ~/.ia), with IA_ACCESS_KEY_ID/IA_SECRET_ACCESS_KEY overriding the file.
Cookies go into the session's cookie jar scoped to .archive.org rather than a Cookie header, because every download is a cross-origin redirect and aiohttp drops Cookie/Authorization headers there; the jar re-attaches them to the data node. 401/403 raise PermissionError (otherwise HTTPFileSystem._info reports a 403 as FileNotFoundError).
Tested with
fsspec/implementations/tests/test_ia.py.