possible Zstandard decompression improvements

[dweiller]

There are a few things that were explicitly not done in the initial Zstandard decompression PR that may be desired improvements. This issue is to collect them (and others that pop up in between) for future reference in a pre-1.0 stdlib review or to list them incase someone to tackle them earlier.

• The current implementation does not decode 4-stream literal sections in parallel in order to avoid memory allocations (or remove a need for an additional 128KiB of memory) in the non-allocating API. This presumably sacrifices decompression performance, but benchmarking may be needed to assess what the impact is in practice. This change would require re-working the block decoding in lib/std/compress/zstandard/decode/block.zig to eagerly decode literals into an extra internal buffer.
• Reuse allocations between frames in std.compress.zstd.decompress.decodeAlloc() API). The frame window could be re-used between frames (provided it is large enough). The relevant allocation is here.
• a few issue we found may result in minor updates to the zstandard spec - issues found while fuzzing to compare behaviour of Zig decoder to this repo facebook/zstd#3508 is the main upstream issue that will hopefully link to any related changes that get made.
  • Check implementation against documentation changes in Clarify zstd specification for Huffman blocks facebook/zstd#3514

[andrewrk]

This is a meta-issue. The next step is to extract from it individual issues and then close this one.

[dweiller]

This is a meta-issue. The next step is to extract from it individual issues and then close this one.

The first point in the list is really reminder/note to consider when work is put into benchmarking and performance optimisation. Is it appropriate to extract a 'performance optimisation and benchmarking' issue from and move that note into it, or is that just another meta-issue and only the latter two points should be extracted to individual issues?

[dweiller]

Extracted the list above into separate issues.