Skip to content

opt: allow bitCost == scaleLog in ZSTD_rescaleFreqs (fixes #4681) - #4749

Open
tvost2 wants to merge 1 commit into
facebook:devfrom
tvost2:fix/opt-rescalefreqs-assert
Open

opt: allow bitCost == scaleLog in ZSTD_rescaleFreqs (fixes #4681)#4749
tvost2 wants to merge 1 commit into
facebook:devfrom
tvost2:fix/opt-rescalefreqs-assert

Conversation

@tvost2

@tvost2 tvost2 commented Aug 31, 2026

Copy link
Copy Markdown

Fixes #4681

ZSTD_rescaleFreqs() initializes the optimal parser's symbol statistics from a
dictionary's entropy tables. For the litLength / matchLength / offCode streams it
reads a per-symbol cost with FSE_getMaxNbBits(), which by contract returns a
"fake cost" of tableLog + 1 for zero-frequency symbols (lib/common/fse.h).
LLFSELog / MLFSELog are 9 and scaleLog here is 10, so a dictionary
whose FSE NCount leaves a symbol in 0..MaxLL / 0..MaxML at frequency 0
makes bitCost == scaleLog, tripping

assert(bitCost < scaleLog);

in DEBUGLEVEL>=1 builds. Release builds are unaffected: the next line computes
1 << (scaleLog - bitCost) = 1 << 0 = 1, which is exactly the
/* minimum to calculate cost */ fallback, so the statistics are already correct
for this case.

The literal loop immediately above already handles the analogous Huffman
case with assert(bitCost <= scaleLog). This makes the three FSE asserts
consistent with it (<<=); no behavior change beyond the assert.

Reproducer (from #4681, Aisle Research)

446-byte input to a dictionary_loader-style harness driving
ZSTD_createCDict_advanced with an optimal-parser strategy:

zstd_opt.c:195: ZSTD_rescaleFreqs: Assertion `bitCost < scaleLog' failed.
  #11 ZSTD_rescaleFreqs              lib/compress/zstd_opt.c:195
  #12 ZSTD_compressBlock_opt_generic lib/compress/zstd_opt.c:1115
  #13 ZSTD_compressBlock_opt2        lib/compress/zstd_opt.c:1451
  #14 ZSTD_buildSeqStore             lib/compress/zstd_compress.c:3444

Confirmed locally (DEBUGLEVEL=1): aborts before the change, [*] no crash
after.

Test

fuzzer.c gains test : optimal parser + dictionary with sparse FSE tables:
build a CDict with each optimal-parser strategy (btopt/btultra/btultra2)
from the 433-byte dictionary embedded in that reproducer (its FSE matchlength
NCount has zero-frequency symbols), then compress + round-trip. Fails at the
assert on dev, passes with this change. basicUnitTests green end-to-end.

Note

#4681 also suggests option (b): gate the FSE branch on
fse.matchlength_repeatMode == FSE_repeat_valid (etc.) and fall through to the
no-dict-stats path per stream when the dict NCount was not fully valid. That is
more principled but more invasive; happy to follow up with it if preferred. This
PR is the minimal correctness fix.


AI-assisted; reproduced, fixed, built and tested locally by me, and I take
responsibility for the change.

@meta-cla meta-cla Bot added the CLA Signed label Aug 31, 2026
)

When the optimal parser initializes its statistics from a dictionary's FSE
tables, it reads a per-symbol cost with FSE_getMaxNbBits(). That function
returns a "fake cost" of tableLog+1 for zero-frequency symbols. The
litLength / matchLength FSE tables can use tableLog == 9 == scaleLog-1, so
the fake cost is exactly scaleLog (10), which tripped assert(bitCost <
scaleLog) in debug builds. The value computation right below already copes
with it: 1 << (scaleLog - bitCost) == 1 << 0 == 1, i.e. identical to the
release-build result and to the 'minimum to calculate cost' fallback.

Relax the three FSE asserts to bitCost <= scaleLog, matching the literal
loop just above which already uses HUF_getNbBitsFromCTable() the same way.

Adds a fuzzer.c regression test that builds a CDict with an optimal-parser
strategy from a 433-byte dictionary whose FSE NCount has zero-frequency
matchlength symbols, then compresses and round-trips.
@tvost2
tvost2 force-pushed the fix/opt-rescalefreqs-assert branch from 4635578 to 4717a4f Compare August 31, 2026 03:16
@tvost2
tvost2 marked this pull request as ready for review August 31, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reachable assertion in ZSTD_rescaleFreqs (bitCost < scaleLog) when CDict built with optimal-parser strategy + tiny dictionary

1 participant