[Torch] A list of missing op conversion in need of help

[masahi]

Let's keep a list of missing conversion reported so far in one place here. The list will continue to grow as Torch has too many ops. cc @alexwong @jwfromm @pyjhzwh @jjohnson-arm

Implementing conversion should be straightforward following existing implementations. They are up for grab.

• aten::reshape
• aten::split
• aten::max_pool1d
• aten::embedding
• prim::ImplicitTensorToNum
• aten::layer_norm
• aten::arange
• aten::rsub
• aten::gelu
• aten::reciprocal
• aten::repeat
• aten::groupnorm

[megjoshi]

can we add these as well?
'prim::ImplicitTensorToNum', 'aten::layer_norm', 'aten::arange', 'aten::rsub', 'aten::gelu'

[masahi]

@megjoshi Updated. If possible, please show your original PyTorch models where these ops are generated from.

[megjoshi]

https://github.com/huggingface/transformers#quick-tour

this is on which i was trying to optimize using tvm

[NarendraPatwardhan]

Also aten::groupnorm as it is integral part of many new SOTA models and the alternative route pytorch->onnx->tvm introduces inefficient conversion by converting groupnorm to multiple instance norms instead.

[masahi]

@NarendraPatwardhan Group norm is supported in #5358.

[gopinath-r]

@masahi I am trying to use tvm for the following model
[https://github.com/cogaplex-bts/bts/blob/master/pytorch/bts.py]

I am getting the following error while building the relay IR from pytorch scripted model
TypeError: Don't know how to handle type <class 'torch.Tensor'>

After debugging i found that error is thrown at parsing the operation torch.repeat.
and torch.repeat is mapped as aten::repeat in tvm/python/tvm/relay/frontend/pytorch.py

torch.repeat operates on tensor but while parsing torch.repeat it is unable to handle the data type

following is the complete error trace

` in
1 shape_list = [('input0', (1,3,1216,352))]
2 mod, params = relay.frontend.from_pytorch(scripted_model,
----> 3 shape_list)

~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/frontend/pytorch.py in from_pytorch(script_module, input_shapes, custom_convert_map)
2255
2256 ret = convert_operators(_get_operator_nodes(graph.nodes()),
-> 2257 outputs, ret_name, convert_map, prelude)
2258
2259 mod["main"] = tvm.relay.Function(_analysis.free_vars(ret[0]), ret[0])

~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/frontend/pytorch.py in convert_operators(operators, outputs, ret_names, convert_map, prelude)
2169 relay_op = convert_map[operator]
2170 # print("relay_op",relay_op)
-> 2171 relay_out = relay_op(inputs, _get_input_types(op_node))
2172
2173 if isinstance(relay_out, tuple):

~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/frontend/pytorch.py in _impl(inputs, input_types)
322 # print("data",data)
323 # print("reps",reps)
--> 324 return _op.transform.tile(data, reps=reps)
325 return _impl
326

~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/op/transform.py in tile(data, reps)
446 """
447
--> 448 return _make.tile(data, reps)
449
450

~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/_ffi/_ctypes/packed_func.py in call(self, *args)
209 """
210 temp_args = []
--> 211 values, tcodes, num_args = _make_tvm_args(args, temp_args)
212 ret_val = TVMValue()
213 ret_tcode = ctypes.c_int()

~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/_ffi/_ctypes/packed_func.py in _make_tvm_args(args, temp_args)
175 else:
176 print(arg)
--> 177 raise TypeError("Don't know how to handle type %s" % type(arg))
178 return values, type_codes, num_args
179

TypeError: Don't know how to handle type <class 'torch.Tensor'>`

If you know what is the reason and how to resolve this it would be really helpful

Thanks in advance

[siju-samuel]

@gopinath-r can you please share the script to run from bts github or scripted_model?
In repeat, can u please print both inputs, input_types and share the result.

[gopinath-r]

@siju-samuel thanks for the reply

These are the values of input and input_types in repeat
input>>> bound method Kernel.raw_input of <ipykernel.ipkernel.IPythonKernel object at 0x7f1dc147c160>
input_types>>> ['float', 'ListType']
data>>> tensor([[[0., 1., 2., 3., 4., 5., 6., 7.]]])
reps>>> [1, CallNode(Op(multiply), [Constant(152), Constant(8)], (nullptr), []), 44]

Use of torch.repeat can be found at line numbers 141, 144 of bts.py.

I have uploaded the script and necessary files to run bts in gdrive
BTS_TVM

[siju-samuel]

@gopinath-r
Currently tvm-pytorch frontend doesnt support taking each value of reps as another tensor or function. It is expecting as an simple constant and in this case we are getting a multiplication operator.

So as a workaround, you can modify your bts.py
change your local_planar_guidance method
typecast the scalar multiplication to int.

Modify the below two lines and do a simple typecast before multiplication.

    u = self.u.repeat(plane_eq.size(0), plane_eq.size(2) * int(self.upratio), plane_eq.size(3))
    v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), plane_eq.size(3) * int(self.upratio))

to

    u = self.u.repeat(plane_eq.size(0), int(plane_eq.size(2) * int(self.upratio)), plane_eq.size(3))
    v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), int(plane_eq.size(3) * int(self.upratio)))

Hope this will resolve your issue.

Please use https://discuss.tvm.ai/ from now onwards for discussing bugs and issues.

[woniuasd]

my pytorch code is "canvas[:, indices] = voxels"

The following operators are not implemented: ['aten::index_put_']

[woniuasd]

@siju-samuel You mean i can use embedded instead of aten::index_put_ ？

[siju-samuel]

@woniuasd Can you share me the complete script or model where i can simulate this aten::index_put_ operator? Im not sure how to invoke this op.

[masahi]

@siju-samuel @woniuasd Since relay doesn't support in-place op, you cannot use that python idiom.

[tqchen]

@masahi shall we close the PR and open new ones since that all the lists are checked off?