[Build] Failed to build pulsar-build:manylinux-3.9

[zymap]

Describe the bug
PR #9389 introduces the python 3.9 client wheel supports. But it's will failed to build the base image pulsar-build:manylinux-3.9.

Logs:

Step 23/31 : RUN pip install twine
 ---> Running in 826d0f3ffa14
Collecting twine
  Downloading twine-3.3.0-py3-none-any.whl (34 kB)
Collecting rfc3986>=1.4.0
  Downloading rfc3986-1.4.0-py2.py3-none-any.whl (31 kB)
Collecting requests-toolbelt!=0.9.0,>=0.8.0
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Requirement already satisfied: setuptools>=0.7.0 in /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages (from twine) (54.1.1)
Collecting pkginfo>=1.4.2
  Downloading pkginfo-1.7.0-py2.py3-none-any.whl (25 kB)
Collecting requests>=2.20
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting readme-renderer>=21.0
  Downloading readme_renderer-29.0-py2.py3-none-any.whl (15 kB)
Collecting tqdm>=4.14
  Downloading tqdm-4.59.0-py2.py3-none-any.whl (74 kB)
Collecting keyring>=15.1
  Downloading keyring-23.0.0-py3-none-any.whl (32 kB)
Collecting colorama>=0.4.3
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting SecretStorage>=3.2
  Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting jeepney>=0.4.2
  Downloading jeepney-0.6.0-py3-none-any.whl (45 kB)
Collecting importlib-metadata>=3.6
  Downloading importlib_metadata-3.7.2-py3-none-any.whl (11 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting Pygments>=2.5.1
  Downloading Pygments-2.8.1-py3-none-any.whl (983 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting docutils>=0.13.1
  Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
Collecting bleach>=2.1.0
  Downloading bleach-3.3.0-py2.py3-none-any.whl (283 kB)
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: packaging in /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages (from bleach>=2.1.0->readme-renderer>=21.0->twine) (20.9)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting cryptography>=2.0
  Downloading cryptography-3.4.6.tar.gz (546 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting cffi>=1.12
  Using cached cffi-1.14.5-cp39-cp39-manylinux1_x86_64.whl (406 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages (from packaging->bleach>=2.1.0->readme-renderer>=21.0->twine) (2.4.7)
Building wheels for collected packages: cryptography
  Building wheel for cryptography (PEP 517): started
  Building wheel for cryptography (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/_internal/cpython-3.9.2/bin/python3.9 /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpjpj578gf
       cwd: /tmp/pip-install-vkopdabc/cryptography_145bc764456f4bf5bf46f276ebb0054e
  Complete output (161 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/exceptions.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/fernet.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/utils.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/__init__.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/__about__.py -> build/lib.linux-x86_64-3.9/cryptography
  creating build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/general_name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/base.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/extensions.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/certificate_transparency.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/oid.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/_types.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/_der.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/_oid.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
  copying src/cryptography/hazmat/backends/interfaces.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
  copying src/cryptography/hazmat/backends/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
  copying src/cryptography/hazmat/bindings/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/_cipheralgorithm.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/keywrap.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/_asymmetric.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/constant_time.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/_serialization.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ciphers.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/backend.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x509.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/decode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/encode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/_conditional.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/binding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/kbkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/scrypt.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/hkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/concatkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/x963kdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/pbkdf2.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/hotp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/totp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/algorithms.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/modes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/pkcs12.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/ssh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/pkcs7.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  running egg_info
  writing src/cryptography.egg-info/PKG-INFO
  writing dependency_links to src/cryptography.egg-info/dependency_links.txt
  writing requirements to src/cryptography.egg-info/requires.txt
  writing top-level names to src/cryptography.egg-info/top_level.txt
  reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'docs/_build'
  warning: no previously-included files found matching 'vectors'
  warning: no previously-included files matching '*' found under directory 'vectors'
  warning: no previously-included files matching '*' found under directory '.github'
  warning: no previously-included files found matching 'release.py'
  warning: no previously-included files found matching '.coveragerc'
  warning: no previously-included files found matching 'codecov.yml'
  warning: no previously-included files found matching '.readthedocs.yml'
  warning: no previously-included files found matching 'dev-requirements.txt'
  warning: no previously-included files found matching 'tox.ini'
  warning: no previously-included files found matching 'mypy.ini'
  warning: no previously-included files matching '*' found under directory '.zuul.d'
  warning: no previously-included files matching '*' found under directory '.zuul.playbooks'
  writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
  copying src/cryptography/py.typed -> build/lib.linux-x86_64-3.9/cryptography
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-3.9/_padding.c'
  creating build/temp.linux-x86_64-3.9
  generating cffi module 'build/temp.linux-x86_64-3.9/_openssl.c'
  running build_rust

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed:
         https://cryptography.io/en/latest/installation.html#rust
      5) If you are experiencing issues with Rust for *this release only* you may
         set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
      =============================DEBUG ASSISTANCE=============================

  error: can't find Rust compiler

  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

  To update pip, run:

      pip install --upgrade pip

  and then retry package installation.

  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.

  This package requires Rust >=1.41.0.
  ----------------------------------------
  ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
The command '/bin/sh -c pip install twine' returned a non-zero code: 1

reproduce

1. checkout to the master code,
2. cd pulsar/pulsar-client-cpp/create-images.sh

[zymap]

I removed the python3.9 wheel file build temporarily so we can build the master branch successfully. After the pulsar-build:manylinux-3.9 build out I will add the python 3.9 build back

[alexku7]

@zymap Can we add back the python 3.9 support...please :)
We need it so much 👍

[zymap]

Sure. I will take a look ASAP.

[jeffreytolar]

This appears to be due to cryptography dropping its manylinux1 builds. They do produce manylinux2010 and manylinux2014 wheels.

Per https://github.com/pypa/manylinux,

PEP 513 defined manylinux1_x86_64 and manylinux1_i686 platform tags and the wheels were built on Centos5. Centos5 reached End of Life (EOL) on March 31st, 2017.

The simplest fix is likely to bump

pulsar/pulsar-client-cpp/docker/Dockerfile

Line 21 in f800fde

FROM quay.io/pypa/manylinux1_x86_64

to manylinux2010 (although I haven't tried that personally). However, that image doesn't have any python2 interpreters.

If supporting python2 is still important, one option may be to continue to produce manylinux1 wheels for 2.7, and use manylinux2010 for the newer interpreters