[Bug][VM] Performance degradation of non_max_suppression layer on VM

[echuraev]

non_max_suppression works much faster on GraphExecutor in comparison with VirtualMachine.

Expected behavior

I suppose that the performance should be the same for VM and GE.

Actual behavior

On my CPU (Intel Core i7-7700K) non_max_suppression works 3 times slower on VM (1066.29 ms) vs GE (359.79 ms).
Tried to analyze this problem by using VTune Amplifier. And saw that about 70% of the execution time some work was done in lib.so (the name of the compiled model).

In GE we don't have such overhead.

Environment

Linux OS, latest mainline.

Steps to reproduce

You can use the following script to reproduce this problem. I changed its extension to .txt because .py file cannot be uploaded to GitHub.
reproducer.txt

On the top of the source code, you can change the value of variable USE_VM to specify if the layer should be inferred on VM or on GE.

Triage

Please refer to the list of label tags here to find the relevant tags and add them below in a bullet format (example below).

• flow:vm

[twmht]

@echuraev

you can use gist to show the code.

btw, how you analyze the code?

And saw that about 70% of the execution time some work was done in lib.so (the name of the compiled model).

[echuraev]

@twmht I created the gist with the reproducer.

I built TVM with debug symbols and I used Intel VTune Amplifier to analyze the hotspots in the TVM.

[twmht]

@echuraev

I have reproduced the problem. btw, because i never used NMS from TVM, what is the advantage to use nms in TVM? because I thought the benifit of TVM is auto turing, we can auto tune NMS but it's dynamic shape in runtime, not very suit for static shape auto tuning. do you have any idea?

[echuraev]

Nice to hear that @twmht! NMS is used in some topologies. For example, I found this problem after analyzing ssd resnet model from ONNX model zoo. NMS is used in this topology. Also, I saw this layer in onnx-yolov3 and onnx-faster-rcnn.