[Bug] Some tuning tasks causes device reboot / RPC server termination

[pfk-beta]

Hello,

I'm using heavily RPC tracker and server with TVM. I encounter many times issue, when tuning task cause device reboot or rpc_android_app restart. After that, whole tuning process need to be restarted. Tuning process waits for something - when I turn RPC_APP again, it breaks.

Error from tuning process:

[Task  4/16]  Current/Best:    0.29/   0.96 GFLOPS | Progress: (10/10) | 89.11 s Done.
[Task  5/16]  Current/Best:    0.00/   0.00 GFLOPS | Progress: (0/10) | 0.00 sTraceback (most recent call last):
  File "tests/android_auto.py", line 180, in <module>
    tune_and_evaluate(tuning_option)
  File "tests/android_auto.py", line 142, in tune_and_evaluate
    tune_tasks(tasks, **tuning_opt)
  File "tests/android_auto.py", line 120, in tune_tasks
    autotvm.callback.log_to_file(tmp_log_file),
  File "/workspace/python/tvm/autotvm/tuner/xgboost_tuner.py", line 105, in tune
    super(XGBTuner, self).tune(*args, **kwargs)
  File "/workspace/python/tvm/autotvm/tuner/tuner.py", line 112, in tune
    measure_batch = create_measure_batch(self.task, measure_option)
  File "/workspace/python/tvm/autotvm/measure/measure.py", line 282, in create_measure_batch
    attach_objects = runner.set_task(task)
  File "/workspace/python/tvm/autotvm/measure/measure_methods.py", line 326, in set_task
    "set_task - Cannot get remote devices from the tracker. "
RuntimeError: set_task - Cannot get remote devices from the tracker. Please check the status of tracker by 'python -m tvm.exec.query_rpc_tracker --port [THE PORT YOU USE]' and make sure you have free devices on the queue status.

Ouptput from query_rpc_tracker:

Tracker address 0.0.0.0:9190

Server List
----------------------------
server-address  key
----------------------------
----------------------------

Queue Status
-------------------------------
key       total  free  pending
-------------------------------
android   0      0     1
-------------------------------

[pfk-beta]

Hello,

I think that method check_remote may be working incorrecly: https://github.com/apache/tvm/blob/main/python/tvm/autotvm/measure/measure_methods.py#L781

[masahi]

#10255