[CI Problem] [Hexagon] Builder failing due to outdated LLVM

[joshherr-quic]

Branch/PR Failing

#12873

Jenkins Link

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-12873/6/pipeline#step-1503-log-710

Flakiness

After implementing a dense op for float16 on hexagon, there is an instruction selection error in LLVM. It has been patched as of llvm 15.0.1. The builder is running 14 I believe. Ideally, I would like to be able to provide a sha from LLVM main which is guaranteed to have all the fixes required for SotA codegen.

cc @mehrdadh

[driazati]

Unfortunately the process for something like this is a little clunky. The LLVM that runs in CI is part of a Docker image defined in docker/, a separate PR to update that to 15 would just need to change the version for hexagon: https://github.com/apache/tvm/blob/main/docker/install/ubuntu_install_hexagon.sh#L22-L29

Once that's done we'll need to wait for an image build and update the tags in the repo, then #12873 can proceed

[joshherr-quic]

Below are the cmake options required to build an llvm cross compiler for hexagon. Some of the options may be superfluous due to these settings coming from a build that I use locally for other dev work.

-DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD:STRING="Hexagon" -DLLVM_BUILD_RUNTIME:BOOL=ON -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_LIBCXX:BOOL=ON -DLLVM_ENABLE_PROJECTS:STRING="clang;llvm;lld" -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" -DTARGET_TRIPLE:STRING=x86_64-unknown-linux-gnu -DLLVM_DEFAULT_TARGET_TRIPLE:STRING=x86_64-unknown-linux-gnu

[joshherr-quic]

Unfortunately the process for something like this is a little clunky. The LLVM that runs in CI is part of a Docker image defined in docker/, a separate PR to update that to 15 would just need to change the version for hexagon: https://github.com/apache/tvm/blob/main/docker/install/ubuntu_install_hexagon.sh#L22-L29

Once that's done we'll need to wait for an image build and update the tags in the repo, then #12873 can proceed

After looking into this, I was unable to find a suitable pre-built binary on the llvm github releases page. The closest build is for LLVM 15.0.2 on SLES clang+llvm-15.0.2-x86_64-unknown-linux-gnu-sles15.tar.xz

[driazati]

@joshherr-quic is the issue that there is not a binary release for 15.02+ on x86_64? we can always build from source (which we already do in some situations: https://github.com/apache/tvm/blob/main/docker/install/ubuntu1804_manual_install_llvm.sh) if necessary so long as there is an actual release backing it

[joshherr-quic]

@driazati ah I see, wonderful. I'll look into it! Thanks

[joshherr-quic]

Builds with new codegen requirements are building successfully.

[mehrdadh]

@joshherr-quic did you update the docker image on CI?

[joshherr-quic]

@joshherr-quic did you update the docker image on CI?

Something prior to the op's codegen must have changed because CI is passing with the old llvm now. However, we still need the updated llvm for other patches.

How do I go about updating the docker image on CI?