[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
Merged
Conversation
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
tkonolige
requested review from
Huyuwei,
Laurawly,
ZihengJiang,
jcf94,
jwfromm,
kevinthesun,
masahi,
mbrookhart,
vinx13 and
yzhliu
as code owners
January 18, 2022 20:23
masahi
approved these changes
Jan 18, 2022
masahi
left a comment
Member
There was a problem hiding this comment.
makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
michalpiszczek
approved these changes
Jan 18, 2022
Contributor
Author
|
yuanfz98
pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo
pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc
pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
This patch moves the fused ops one more loop nesting inwards to get
On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi @mbrookhart