poor performance on windows

[synodriver]

Why does it so slow compare to the original one, the compress speed is just about 4MB/s, much slower than the origin one. Maybe we should replace bytearray with uint8_t* and manage it by PyMem_Realloc

[synodriver]

replace bytearray with https://github.com/synodriver/c-buffer

[synodriver]

No, I didn't. Compressor is still a problem with a poor performance (about 4Mb/s). I tried to profile it with py-spy but just got some error.

[synodriver]

Maybe we could try to use ringbuffer? But, before we did something, we'd better figure out what's happening inside Compressor. See cython backend for more details.

[synodriver]

Okey, linux version is much more faster than windows one(3x faster). Should be window's problem.

[synodriver]

CFFI vs Cython backend. The efficiency is around 85%. That means when you use this wrapper, 85% of the cpu time is spent on the underlying C library, about 15% performance loss. And the performance on windows should blame microsoft.

[synodriver]

Close this as complete.

[synodriver]

Looks like crc32 is very slow somehow, consuming 30% of CPU time. I don't know if this is a compiler problem or something, but worth improvment.

BTW, I tested it on github codespace which runs the latest Ubuntu version. Not likely to be system's problem. Maybe related to this ?

[synodriver]

With openmp linked to the extension, now python is able to use multicores to encode and decode.