Skip to content

Add Internet Archive filesystem implementation - #2121

Draft
lfoppiano wants to merge 2 commits into
fsspec:masterfrom
lfoppiano:ia-filesystem
Draft

Add Internet Archive filesystem implementation#2121
lfoppiano wants to merge 2 commits into
fsspec:masterfrom
lfoppiano:ia-filesystem

Conversation

@lfoppiano

@lfoppiano lfoppiano commented Sep 8, 2026

Copy link
Copy Markdown

Adds InternetArchiveFileSystem (fsspec/implementations/ia.py, protocol ia): ia://<identifier>/<filename> for files in archive.org items, plus registry entry, API docs, changelog and tests.

It remap to HTTP (HTTPFileSystem) with a path mapping and credentials, from https://archive.org/download//, which redirects to a data node that support HTTP Range queries. That is the same URL the internetarchive package downloads from.

S3-like API (s3.us.archive.org, which an S3FileSystem + endpoint_url would use): Internet Archive data nodes ignore Range and answer a ranged GET with 200 and the whole file, so every block read would download the entire object. Measured on a 1.6 GB public item (Range: bytes=0-99 on every request):

Endpoint URL Status Bytes sent
S3-like front end https://s3.us.archive.org// 307 → http://dnNNN.s3dns.us.archive.org:80/… 410 (XML)
S3-like data node (following the 307) http://dnNNN.s3dns.us.archive.org// 200, Accept-Ranges: bytes, no Content-Range 1,689,855,303
Download endpoint https://archive.org/download// 302 → https://dnNNN.ca.archive.org/0/items/… 0
Download data node (following the 302) https://dnNNN.ca.archive.org/0/items// 206, Content-Range: bytes 0-99/1689855303 100

IA documents this: "HTTP 1.1 Range headers are ignored" (https://archive.org/developers/ias3.html#how-this-is-different-from-normal-s3). botocore also misreads the 307 as an AWS region redirect and recurses on HeadBucket until RecursionError.

Credentials. Public items need none. For restricted items the class reads the ia.ini written by ia configure, found the way the internetarchive package finds it ($IA_CONFIG_FILE, $XDG_CONFIG_HOME/internetarchive/ia.ini, ~/.config/ia.ini, ~/.ia), with IA_ACCESS_KEY_ID/IA_SECRET_ACCESS_KEY overriding the file.
Cookies go into the session's cookie jar scoped to .archive.org rather than a Cookie header, because every download is a cross-origin redirect and aiohttp drops Cookie/Authorization headers there; the jar re-attaches them to the data node. 401/403 raise PermissionError (otherwise HTTPFileSystem._info reports a 403 as FileNotFoundError).

Tested with fsspec/implementations/tests/test_ia.py.

@martindurant

Copy link
Copy Markdown
Member

This looks like fun - please let me know when you are ready. Are you affiliated with IA in any way? If not, should we approach someone for their opinion?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants