CI monitoring rotation schedule

[driazati]

See the CI Monitoring Runbook for context.

This schedule tracks the CI monitoring rotation. If you would like to join, please comment on this issue or, if you are a committer, edit this issue directly.

Week (MM/DD)	On-call
8/1 - 8/5	@driazati
8/8 - 8/12	@gigiblender
8/15 - 8/19	@shingjan
8/22 - 8/26	@cconvey
8/29 - 9/2	@sunggg
9/5 - 9/9	@driazati
9/12 - 9/16	@nverke
9/19 - 9/23	---
9/26 - 9/30	---
10/3 - 10/7	---

[driazati]

This isn't part of the regular rotation but I thought I'd post a summary of my week so everyone has some more visibility into the process. Last week was lots of infra problems which hopefully are fixed now that we've increased capacity limits and fixed our cleanup logic.

• jenkins on c253053 - reverted in [skip ci] Revert "[skip ci][ci][docker] Prune all non-relevant images (#11491)" #11496
• jenkins on 01ee1bc - failed due to a Jenkins reboot
• jenkins on 6f3c8bd - failed due to a Jenkins reboot
• jenkins on 7766ab2 - failed due to a Jenkins reboot
• jenkins on 2a2d910 - failed due to a Jenkins reboot
• jenkins on 6895087 - network problems, filed Use git annex to log large files in CI #11492
• jenkins on cfcca59 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on 2f21698 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on b535e46 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on 52df2e8 - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on a9ece3d - node ran out of disk, fixed in Bump EBS size to 150GB tlc-pack/ci-terraform#41 and [skip ci][ci][docker] Prune all non-relevant images #11491
• jenkins on 8135860 - timed out due to queuing
• jenkins on d519b03 - timed out due to queuing
• jenkins on 92cc5b0 - timed out due to queuing
• jenkins on bbdb656 - network problems and flaky test, filed [ci][network error] tests/python/frontend/tflite/test_forward.py::test_forward_tflite_float16, tests/python/frontend/tflite/test_forward.py::test_forward_qnn_inception_v1_net #11459 and [Flaky Test] tests/python/unittest/test_arith_solve_linear_inequality.py::test_solution_consistency #11458
• jenkins on 6c6dfbc - fixed by [ci] Increase timeout for git submodule update #11457
• jenkins on c247295 - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 014208e - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 3f53e7a - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 7e83c4a - fixed by [skip ci][ci] Fix broken test skips #11456
• jenkins on 7ba8a61 - failed due to a
• jenkins on b141cac - filed [Flaky Test] tests/python/unittest/test_auto_scheduler_evolutionary_search.py::test_mutate_parallel #11440
• gha on 7bab8f7 - filed [ci] MacOS GitHub Actions occasionally dies #11416

[areusch]

Summary from my week: things were fairly quiet, mainly some flaky tests.

• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3414/pipeline) on c1b22ee - flaky test, filed [Flaky Test] tests/python/contrib/test_ethosu/test_codegen.py::test_mean[ifm_shape0-axis0-True-False-ethos-u65-256] #11527
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3415/pipeline) on ac5d781 - continued flaky paddlepaddle test, updated [Flaky] Possible flaky tests from Paddle #9976. better diagnostic submitted today.
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3411/pipeline) on bc14f26 - network flake, filed [CI Problem] from_oneflow can flake due to network dependency #11514
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3444/tests) on 2ae2088 - assert_allclose 2/10000 mismatched (0.02%), filed [Flaky Test] tests/python/frontend/pytorch/test_forward.py::test_argsort #11568

[hpanda-naut]

@areusch @driazati I can take next week

[hpanda-naut]

Summary from 6/13/22 - 6/19/22: mostly infra and flaky test issues

• jenkins on 2df4524 [ci] docker pull network errors #11710
• jenkins on 81cc086 [ci] docker pull failed #11713
• gha on a82d2f5 [ci] andriod_rpc build failed #11738
• jenkins on https://github.com/dos-lab/as-tvm/commit/583696fceb8abd4c3c2f4abb93d0dbce2e79f60f [ci] pytest_wrapper needs guards for invalid JUnits #11749
• jenkins [Flaky Test] tests/python/contrib/test_ethosu/test_codegen.py::test_mean[ifm_shape0-axis0-True-False-ethos-u55-256] #11763
• jenkins on b4a77ac [ci] test wrapper scripts failure #11764
• jenkins on 65d45af [bug][docs] error in "code" directive: maximum 1 argument(s) allowed, 2 supplied. #11804
• gha on 0fdc0ea [ci] Execution failed for task ':app:checkDebugAarMetadata' on android build #11805

[gigiblender]

Happy to sign up for 6/27-7/1 if someone could edit the issue for me :)

[nverke]

Week of 6/20 - 6/27:

• Details on bc75487 [Flaky Test] tests/python/contrib/test_ethosu/test_codegen.py::test_mean[ifm_shape2-axis2-True-False-ethos-u65-256] #11830
• Details on b63801c [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on a363a04 [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on 7b0f791 [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on a5366a7 [ci] [Flaky Test] ai.stanford.edu going down causes CI to break #11811
• Details on 32d16eb [CI Problem] Failure while downloading files from S3. #11917
• Details on 98fb955 [CI Problem] No space left on device #11840
• Details on 6ed3ab3 [CI Problem] No space left on device #11840
• Details on 1e0e954 (Fixed not sure what the issue was)
• Details on 51b0d8c [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #11580
• Details on 410e836 [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #11580
• Details on 12e8744 [CI Problem] Docs failed to build with code 143 #11918
• Details on 5e81067 [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #11580
• Details on 4cb18b4 [CI Problem] /workspace/gallery/how_to/compile_models/from_oneflow.py failed due to url issue.  #11919
• Details on aa66e9f [CI Problem] Docs failed to build with code 143 #11918
• Details on 600a201 [CI Problem] No space left on device #11840
• Details on 98bf40f [CI Problem] Jenkins link 404  #11920

[sunggg]

I can help on 7/11-7/15 :)

[gigiblender]

27/06/2022
jenkins on 1115fd9 aborted due to CI disk space issue

28/06/2022
jenkins on 6c8a353 #11580
jenkins on 688b082 #11580

29/06/2022
jenkins on a17bfc0 #11964

30/06/2022
jenkins on 41c94b2 #11967
jenkins on 898946f #11567
jenkins on 558ba99 #12004 and flaky due to network
jenkins on e7851ed #11967
jenkins on 80a0c6c #11967
jenkins on 915c23b #11967
jenkins on c0f4bf7 #11967

01/07/2022
jenkins on ec39199 #11580
jenkins on 395e91f #11580
jenkins on 9e14509 #11568
jenkins on 50cd4d6 #11568

[areusch]

• [jenkins](https://ci.tlcpack.ai/blue/organizations/[jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3755/pipeline)/tvm/detail/main/3753/pipeline/) on cfe8318 - failed to docker pull from DockerHub
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3755/pipeline) on 9f4bf38 - failed to docker pull from DockerHub
• jenkins on 99d42b2 - test_qlinearadd tolerance, filed [Flaky Test] tests/python/frontend/onnx/test_forward.py::test_qlinearadd[cuda] #12062
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3767/pipeline) on f769f4e - Skipped has no len, added to [ci] pytest_wrapper needs guards for invalid JUnits #11749
• [jenkins](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/3769/pipeline) on a81e69a - http flake

[sunggg]

• jenkins on ef5c3ed - failure on git push to https://github.com/apache/tvm-site.git
• jenkins on 3992d24 - numerical error at test_myloat() in test_custom_datatypes.py, filed [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #12092
• jenkins on b9fa576 - Error response from daemon: Get "https://registry-1.docker.io/v2/tlcpack/ci-i386/manifests/sha256:096c51de2a27fd3e497c17433937a4cd4357fe9623f06f6b2a47e094cf515b14": EOF

[yongwww]

Summary for 7/25/22 - 7/29/22:

jenkins on eada707 - failure on android-rpc build
jenkins on 88bbb40 - failure on git push to https://github.com/apache/tvm-site.git
jenkins on 03aed78 - failure on ios-rpc
jenkins on 195e60b - failure on duplicate global packedfunc
jenkins on 9a4d80c - filed [Flaky Test] tests/python/unittest/test_custom_datatypes.py::test_myfloat #12238
jenkins on aeda760 - filed #12238
jenkins on 578ef03 - filed #12238

[driazati]

Summary for 2022-07-30 to 2022-08-06

Run	Commit	Mitigation
tvm-ci/branch	1f97f1f from #11809	Timeout fixed by #12334
tvm-ci/branch	485bfaf from #12306	Fixed by #12325
tvm-ci/branch	4158738 from #12301	Flaky test, filed #12311 and opened #12312 to improve reporting
tvm-ci/branch	2bfd52f from #12278	Fixed by #12282
tvm-ci/branch	12502cc from #12251	Fixed by #12268
tvm-ci/branch	a231a1d from #12245	Added retries to fix: #12306
tvm-ci/branch	fb87c21 from #12234	Fixed by #12306
tvm-ci/branch	dff5c97 from #11037	Fixed by #12306
tvm-ci/branch	db4380c from #12230	Fixed by #12268

_{Generated by https://gist.github.com/driazati/80cd48e86c6548cd90a6b39be010b921}