dev-python/numcodecs: version bump 0.13.0

[mschubert]

This is a draft PR for updating numcodecs that comes bundled with c-blosc and the latter includes zstd (setting aside that this shoud be unbundled)

If I run:

ebuild numcodecs-0.13.0.ebuild manifest clean compile
# python3.12 -m gpep517 build-wheel --prefix=/PREFIX/usr --backend setuptools.build_meta --output-fd 3 \
#    --wheel-dir /PREFIX/var/tmp/portage/dev-python/numcodecs-0.13.0/work/numcodecs-0.13.0-python3_12/wheel

and then, in "${WORKDIR}/numcodecs-0.13.0-python3_12/build/lib.linux-x86_64-cpython-312/numcodecs", I get the following error importing a generated .so:

from ctypes import cdll
cdll.LoadLibrary('./zstd.cpython-312-x86_64-linux-gnu.so')
# OSError: ./zstd.cpython-312-x86_64-linux-gnu.so: undefined symbol: ZSTD_compressBlock_btopt

This is the same error that I get when I install the ebuild and then try to import the package.

Surprisingly, manually cloning and building the package results in a functional .so:

git clone https://github.com/zarr-developers/numcodecs.git
cd numcodecs
git submodule update --init --recursive
git checkout v0.13.0
python3.12 -m gpep517 build-wheel --prefix=/home/schubert/gentoo/usr \
    --backend setuptools.build_meta --output-fd 1 --wheel-dir .
cd build/lib.linux-x86_64-cpython-312/numcodecs
# repeat python import as above -> works

The .so created by ebuild is also much smaller than by the manual build.

Any ideas why this might be? Also tagging @TheChymera who submitted the ebuild initially (likely related: zarr-developers/numcodecs#506, zarr-developers/zarr-python#961)

[Nowa-Ammerlaan]

The reason it works when you do it manually, and not in portage probably has to do with CFLAGS/LDFLAGS. Specifically, I suspect this is your problem: https://wiki.gentoo.org/wiki/Project:Quality_Assurance/As-needed#Failure_in_execution.2C_undefined_symbols

[mschubert]

Thanks! I was able to track it down to issues with CFLAGS=-pipe and have disabled this -> builds and imports fine now

[Nowa-Ammerlaan]

Thanks! I was able to track it down to issues with CFLAGS=-pipe and have disabled this -> builds and imports fine now

I doubt -pipe is the root cause of this issue, it should not affect the built binary, only the build process: https://wiki.gentoo.org/wiki/GCC_optimization#-pipe

[mschubert]

I agree that this shouldn't make a difference, but I get the module import errors only when not filtering out this flag. I also tried changing the other flags (-O2, -Wl,O1, -Wl,--as-needed), but they did not make a difference.

With all (combinations of) flags the package builds and installs fine, but the error comes from:

from numcodecs import zstd # success with filter-flags -pipe
# otherwise: OSError: ./zstd.cpython-312-x86_64-linux-gnu.so: undefined symbol: ZSTD_compressBlock_btopt

[Nowa-Ammerlaan]

Try with LDFLAGS="-Wl,--no-as-needed" to explicitly override the default.

[mschubert]

Exactly the same error with

python_compile() {
    LDFLAGS="-Wl,--no-as-needed" distutils-r1_python_compile
}

[Nowa-Ammerlaan]

This should be conditional to CPU_FLAGS_X86_SSE2 and CPU_FLAGS_X86_AVX2

[TheChymera]

I was also very confused by zarr-developers/zarr-python#961 ... if you know more about it, maybe it would be worth sharing with upstream. They closed it since I stopped nagging them, but I assume it hasn't just evaporated. We can just reopen if it persists with 0.13.0.

I have logs from the various combinations here → zarr-developers/numcodecs#506 (comment) for some reason -flto=auto also seems to fix this.