Conversation
Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
Add a component repo for Python distributions installed by pip, uv, or any other installer following the "Recording installed projects" spec. Distributions are discovered via their `*.dist-info/RECORD` files anywhere in the rootfs, so this covers system site-packages as well as virtualenvs. Files listed in RECORD are strongly claimed, along with their parent directories up to site-packages. Bytecode compiled after installation (not in RECORD) is claimed for the component owning the corresponding source file. The repo is below distro package managers in priority so that Python packages installed by the distro package manager stay with their distro package. Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
Install packages with pip into system site-packages and with uv into a venv, and check that they become pip components while rpm-installed Python packages stay with rpm. Covers bytecode both recorded in RECORD (pip) and not (uv --compile-bytecode). Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
…-importable filenames which contain dots Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
Some distributions vendor their dependencies as complete installs, with their own dist-info and RECORD nested inside the outer package (e.g. setuptools/_vendor, see pypa/setuptools#2825). Process dist-infos outermost first and skip any whose RECORD is already claimed, so vendored copies belong to the outer distribution. Assisted-by: AI Signed-off-by: Peter Siegel <psiegel2000@icloud.com>
Author
|
@jlebon if you have a chance to take a look that would be great! I think this could be a big win for people doing ML stuff with lage Python wheels + using containerization :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a component repo for Python distributions installed by pip, uv, or any other installer following the "Recording installed projects" specification (PEP 376).
chunkah is pretty nice for Python-heavy images. Any ML/CUDA venvs can get to be 10s of gigs in size 😅. The classic workaround is to manually split the installation into separate
RUNcommands so large dependencies land in their own layers (see astral-sh/uv#16113 and astral-sh/uv#7712). With this, each distribution becomes its own component and layers can be reused across updates regardless of how the image was built.How it works:
Foo_Pkg-1.2.3.dist-infobecomespip/foo-pkg.--compile-bytecode) is claimed for the component owning the corresponding.py.setuptools/_vendor, see Stop vendoring packages pypa/setuptools#2825) belong to the outer distribution.PipRepopriority below distro package managers so Python packages installed by the distro package manager stay with the distro package.Assisted-by: AI