Update all ci- containers to reflect main

[areusch]

This is a tracking issue for the process of updating the TVM ci- containers to reflect the following PRs:

• Use new SBT Debian Repo before bintray is shutdown #7926
• [microTVM] Refactor zephyr installation + Update Zephyr RVM doc #7915

It will also unblock PRs:

• [ETHOSN] Removed support for 20.08 version of the driver stack. #7858
• Bumped Ubuntu version to 18.04 for ci_gpu #7970 (it's possible we could just include this PR in this update)

Steps:

• Pick the latest revision of main and use that specific git hash for following steps. Document it here.
• Build docker containers, tagging them as e.g. <username>/ci-<container>:v0.<ver> (we may not need to build all of these if Bumped Ubuntu version to 18.04 for ci_gpu #7970 is not included here)
  • ci-arm
  • ci-cpu
  • ci-gpu
  • ci-i386
  • ci-lint
  • ci-qemu
  • ci-wasm
• Push docker containers to Docker Hub
• Create a draft PR modifying Jenkinsfile to point all containers at <username>/ci-<container>:v0.<ver>
• Force-push the PR to ci-docker-staging branch
• Jenkins will notice the push and start a build here
• Debug the build and repeat these steps until the build passes
• Push the valid containers to tlcpack/ci-<container>:v0.<ver>
• Update the PR to point Jenkinsfile to the new containers.
• Merge the PR.

Let's use this tracking issue to record challenges we face in updating the drivers.

@jroesch can you please note down where you were with this?

cc @tqchen @u99127 @d-smirnov @leandron @tristan-arm

[jroesch]

So last time I tried to update the images but ran into issues where tests no longer passed under updated GPU and CPU images (see docker pull tlcpack/ci-gpu:v0.73-t3).

I had chased down the image building, there are a few important things.

First disable caching in order to make sure you pull the freshest apt as if you have built before you can have a stale layer which pulled an apt-registry which is no longer valid.

Second there were some updates to the drivers which caused breakage, specifically this test no longer works because it incorrectly believes that CUDA is on (even though there is no GPU):

https://github.com/apache/tvm/blob/main/tests/cpp/build_module_test.cc#L84

These needs to be patched to actually check for the GPU's existence.

There were also changes intended to fix Rust CI which should included in master today allowing us to turn Rust CI back on.

[areusch]

@tkonolige and i will attempt to update to 18.04 (e.g. include #7970) with these updates.

[tkonolige]

#8031 contains the updated docker images

[areusch]

@tkonolige is kindly rebuilding the containers. @tkonolige , can you document the main hash used following the steps above?

[areusch]

Testing: https://ci.tlcpack.ai/job/tvm/job/ci-docker-staging/87/

[areusch]

Built from: 55471632d97bb9d0d4535bd977ca4fb7e7cdcc28

[areusch]

retesting: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/88/pipeline

[areusch]

we accidentally tested the wrong thing; so re-re-testing here: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/89/pipeline/6

[areusch]

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/90/pipeline

[areusch]

seems like the last run was unexpectedly unable to access to the gpu in "Frontend : GPU" phase. retrying with printing nvidia-smi before running those tests: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/91/pipeline/

[areusch]

apologies for the light updates here. we determined that the "Frontend : GPU" tests get into a state where either the GPU hardware is inaccessible after a while or TVM's existence check is wrong. Since we didn't change the CUDA version used here--we just updated to 18.04--the theory is that there is some interoperability problem between CUDA running in the containers (at 10.0) and the CUDA driver loaded on the docker host side (either 10.2 or 11.0, depending which CI node you hit).

@tkonolige and I have spent the last couple days running on a test TVM CI cluster using the same AMI (which has only CUDA 11.0). With CUDA 10.0 (ci-gpu) and 11.0 (host), we ran into another similar-looking bug during the GPU unit tests:

[ RUN      ] BuildModule.Heterogeneous
[22:11:58] /workspace/src/target/opt/build_cuda_on.cc:89: Warning: cannot detect compute capability from your device, fall back to compute_30.
unknown file: Failure
C++ exception with description "[22:11:58] /workspace/src/runtime/cuda/cuda_device_api.cc:117: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (e == cudaSuccess || e == cudaErrorCudartUnloading) is false: CUDA: no CUDA-capable device is detected

We then upgraded ci-gpu to use CUDA 11.0, and this test seemed to pass all the way to the end of the GPU integration tests, modulo a tolerance issue:

tests/python/contrib/test_cublas.py::test_batch_matmul FAILED
// ...
>       np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, verbose=True)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-05, atol=1e-07
E       
E       Mismatched elements: 2875175 / 3866624 (74.4%)
E       Max absolute difference: 0.00541687
E       Max relative difference: 0.00015383
E        x: array([[[29.647408, 31.88966 , 33.90233 , ..., 34.673954, 32.908764,
E                31.219051],
E               [30.993076, 30.78019 , 33.67124 , ..., 36.1395  , 29.176218,...
E        y: array([[[29.646427, 31.889557, 33.900528, ..., 34.673126, 32.90791 ,
E                31.21726 ],
E               [30.991737, 30.780437, 33.67001 , ..., 36.139397, 29.174744,...

we'll try and push this CUDA 11.0 ci-gpu container through the test CI cluster to see how far we can get. feel free to comment if there are concerns updating to CUDA 11.0.