file_packager.py to accept arguments from a file

[vadimkantorov]

I'm currently generating a command for https://github.com/emscripten-core/emscripten/blob/main/tools/file_packager.py (~1000 --preload arguments), this overflows the command buffer. Is it possible to load the arguments to file_packager.py from a file?

[vadimkantorov]

I could probably write a wrapper patching sys.argv like python -c 'import sys, file_packager; sys.argv += list(open("argv.txt")); file_packager.main()', but it seems a generally useful feature to have

[sbc100]

Sure we could have it read a response file if its needed. Can I ask why you need so many argument? How about moving all the files into a single directory and just recursively including that? Also, is there some reason you are using --preload and not --embed?

[vadimkantorov]

About why so many arguments: I'm packaging really a lot of small files with long file paths: a subset of TexLive directory tree containing thousands of font files

What does embed do when used with file_packager.py? Will it produce a wasm file then? If it places the code adding the files onto the file system together with the data-file, then I still need the separate-metadata mode (be it in the form of js file, or better as json / txt, so that I don't need to parse the js file to get the list of files that it provides). Instead of embedding I would like chunking of data files, so that they could be cached by Chrome. If the embedding produces wasm files instead of js files, I don't mind, but I'd need them to be easily addable in modularized modules dynamically, so that I can have a Promise to file loaded and all FS calls executed.

For my usecase I need to keep the file index separate to be able to load and inspect it dynamically, and only then to decide whether I need to add it to the file system and load the several dozen megabytes data file. Even the index files can be several megabytes, so I'm not even always loading all the index files if all TeX packages were resolved by the preloaded index file.

My other wishes (and more context for feature requests in this message) for file_packager.py refactoring are here: #14385

[sbc100]

About why so many arguments: I'm packaging really a lot of small files with long file paths: a subset of TexLive directory tree containing thousands of font files

Still.. can't you use normal unix tools to contsurt the enture root filesystem and then just use single --preload=./root type thing?

What does embed do when used with file_packager.py? Will it produce a wasm file then?

--embed puts the data directly into the wasm file. It completely avoids the index, and the separate download and saves on space at runtime since the data is used directly in-implace.

Its also more space efficient because it uses direct binary encoding and there is no encdode/decode phase or copying of data at runtime.

[vadimkantorov]

1. I can, as I can write a wrapper patching sys.argv. But it would be nice such things to be supported natively (e.g. by supporting --preload-file-list myfilelist.txt (with every line specifying path@path)). If there was just one preloaded argument allowed, I would be more understanding about this limitation

2. Well, I need the index because I want to decide dynamically if I need to load the heavy data-file or not. Loading and parsing the index file separately saves a lot of time and network saturation. If there's still an index produced in some form + a wasm file whose loading can be awaited, I don't mind (e.g. one can imagine some sort mmap of a trie of files as part of the FS inner structures, and then the FS itself is better handled). But even a regular data-file + many manual calls to FS to register files are okay. Decompression can also be done differently: e.g. online_not_cached (economizing RAM - probably should be default) / online_cached / full pre-decompression (moving all decompression on package loading stage). I guess it would be best to support all such modes, or providing easy hooks / recipes for the users to implement it.

Also, for large files chunking is important for caching. Of course, I can manually split the packages somehow to ensure smaller data file size, but it could be taken care of by the packager as well (and probably a good reason to refactor the data packages in general by moving the loading code into a JS library and make it more modular / callable by the user)