dfast: pipelined extDict block compressor - #4764
Open
gaynor-anthropic wants to merge 4 commits into
Open
Conversation
The fast match finder tests whether a table candidate lies inside the window with either a conditional move or a branch, chosen per block. The branch wins when nearly every candidate is in range; the conditional move wins when many entries are stale and the branch would mispredict. The double-fast noDict loop always used the conditional move, which is the slower form on x86-64 at level 3 on large inputs. Share the helpers between the two match finders (adding 8-byte variants for the long table) and let double-fast choose too, by table freshness rather than window size alone. A slot goes stale only if it is not overwritten for a whole window, which is rare once windowLog >= hashLog + 4 and common when the tables are about as large as the window. So level 3 on large inputs takes the branch; level 4 on large inputs and any small window keep the conditional move. The long candidate at ip+1 goes through the same helper instead of a plain branch. zstd -b on silesia.tar and enwik8: level 3 is 1-4% faster on x86-64 and unchanged on aarch64, where GCC already emitted a branch; level 4 is unchanged; levels 1-2 are not touched. One-shot output is byte-identical. Streaming output can differ by a few bytes once the window slides (at most 0.006%, on silesia and on a synthetic stream of small messages), because the ip+1 long candidate is now accepted at exactly the lowest valid index, as the ip candidates already were. On aarch64 with GCC this change alone costs 1.7-2.5% where the conditional move is used, because GCC turns the ternary in ZSTD_selectAddr() (and so the new ip+1 select) back into a branch. The next commit adds an aarch64 csel there to recover that. Co-authored-by: Claude <noreply@anthropic.com>
ZSTD_selectAddr() exists so that the "is this table index inside the window" test in the fast and double-fast candidate checks compiles to a conditional move, not a branch that mispredicts when many entries are stale, as with a small window. x86-64 forces this with inline asm; elsewhere it is a plain ternary, which GCC 13 and 15 on aarch64 turn back into a branch, though clang does not. Force it on aarch64 too, with an inline-asm csel. Output is unchanged. Neoverse-V2 (Graviton4), GCC 13: double-fast on synthetic small records with a 32 KB window and level-3 tables (wlog=15, hlog=14, clog=13) is 10% faster. zstd -b on silesia.tar and enwik8: level 4 is 2-3% faster, levels 1-2 unchanged; level 3, where nearly all candidates are in range, is unchanged now that double-fast branches when windowLog >= hashLog + 4 (it would be 2.5% slower if double-fast still used the select there). Co-authored-by: Claude <noreply@anthropic.com>
ZSTD_count() and ZSTD_count_2segments() are a handful of instructions on the hottest path of every block compressor and are meant to be inlined. GCC inlines them with default flags. With the hardening flags distributions commonly add (-D_FORTIFY_SOURCE=3, -fstack-protector-strong, -fstack-clash-protection), GCC 13-15 at -O3 stop inlining them in several zstd_double_fast.c specializations once the file gains a few more specializations, as the next commit does; the calls then cost about 2% on match-dense inputs. Add ZSTD_count_inline() and ZSTD_count_2segments_inline(), the same code marked FORCE_INLINE_TEMPLATE, and use them throughout zstd_double_fast.c. The original functions are untouched, so the other block compressors compile to the same code as before (function sizes in zstd_fast.o, zstd_lazy.o, zstd_opt.o and zstd_ldm.o verified identical). Compressed output does not change. Co-authored-by: Claude <noreply@anthropic.com>
When a stream with a small window is flushed every few KB, zstd's input
buffer wraps about once per window's worth of input, so nearly every
block is compressed in extDict mode, and at level 3 the double-fast
extDict loop sets streaming speed. That loop still searched one position
at a time; the double-fast noDict and fast extDict loops had already
been restructured to overlap work across positions and avoid branching
on index validity.
Give that loop the structure of the noDict one. The long-table lookup
for ip+1 is issued before the candidates at ip are resolved. Candidates
are tested with the conditional-move helpers: in extDict mode the lowest
valid index moves with every block, so the tables always hold stale
entries, and the branch form measured slower or equal at every window
size tried (windowLog 15 to 21). Repcodes that cannot be valid anywhere
in the block are disabled up front rather than range-checked at every
position. Step/skip acceleration, prefetch placement and per-minMatch
specialization follow the noDict loop.
Segment handling is unchanged: an index below prefixStartIndex is
addressed through dictBase, match lengths use the two-segment count, and
backward extension stops at the start of the candidate's segment. Output
changes slightly with the parse: -0.15% bytes on silesia cut into 4 KB
flushes with a 32 KB window, +0.04% on a synthetic message stream.
Compression speed of this commit plus the three preceding ones versus
unpatched dev; level 3, 32 KB window, one core, GCC 15 (silesia row
GCC 13):
workload Neoverse-V2 Sapphire Rapids
4.5-8.9 KB flushed messages,
2,048 live streams interleaved +10-11% +7-10%
one stream +11% +15%
silesia.tar, 4 KB flushes, 16 streams +22% +24%
One-shot zstd -b, same four changes, GCC 13: levels 1-2 unchanged, level
3 up to 3% faster, level 4 is 5-6% faster on Neoverse-V2 and unchanged
on Sapphire Rapids. The messages are synthetic small structured records,
not a public corpus; Neoverse-V2 is an AWS Graviton4.
Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restructures the double-fast extDict loop the way the noDict loop already is: the ip+1 long-table lookup is issued early, candidate checks use cmov instead of branching on index validity, and repcodes that cannot be valid in the block are disabled up front. This path dominates level-3 compression when a small-window stream is flushed every few KB, since nearly every block is then compressed in extDict mode.
The first three commits are preparatory refactors: sharing the cmov/branch candidate-check helpers between fast and double-fast (with an aarch64
cselso GCC keeps them branchless), and force-inlinedZSTD_countvariants so GCC does not stop inlining under distro hardening flags.Results
Measured against dev, change in throughput.
fullbench -b5(extDict), 1 MB silesia chunkszstd -b3, silesia.tar / enwik8zstd -b4zstd -b1,-b2One-shot output is byte-identical. Streaming output changes slightly with the parse once the window slides: -0.15% bytes on silesia in 4 KB flushes with a 32 KB window, +0.04% on the message stream.