Repository navigation
Fix optimal-parser dictionary frequency initialization - #4978
Merged
Merged
Conversation
terrelln
force-pushed
the
2026-11-16-rescale-freqs
branch
2 times, most recently
from
November 17, 2026 19:07
9f8811a to
48184b2
Compare
Cyan4973
reviewed
Nov 18, 2026
Contributor
There was a problem hiding this comment.
This fixes dictionary compression with the optimal parser (btopt, btultra, btultra2).
The fix looks right to me, and output with zstd --train dictionaries is unchanged.
However, the trimmed test passes on dev: it only catches the uninitialized read, and only under MSan, so it no longer covers #4681. Two small changes restore that coverage. Both fail on dev and pass with this PR:
- in
FUZ_writeNCount, give symbol 1 a zero count, and list symbols 0-2 for ML/LL. That is the #4681 trigger: an absent symbol inside the listed range, at the max table log. - build the Huffman table at log 12, with skewed counts:
counts[i] = i < 16 ? 1U << (16 - i) : 1.
minor: the commit message still describes checks that were dropped with the trim (initialized costs, repeat modes, 12-bit Huffman).
ZSTD_rescaleFreqs() seeds its first-block statistics from dictionary entropy tables. At the maximum LL/ML table log, an absent FSE symbol's fake cost equals the frequency scale and trips the strict-less-than assertion. Shortened LL/ML alphabets also leave trailing compression transforms uninitialized even though the optimal parser reads every symbol. Separately, the Huffman reader accepts 12-bit tables, which can underflow the parser's 11-bit scale and cause an invalid shift in release builds. Use a shared cost-to-frequency conversion that preserves a minimum frequency of one without underflowing the shift. Build dictionary LL/ML tables over their full symbol ranges, using the zero counts supplied by FSE_readNCount(). Keep the original repeat-mode checks so absent symbols are not treated as safe to encode. Add a hand-built dictionary regression that compresses with every optimal-parser strategy and round-trips the result. The dictionary has a skewed 12-bit Huffman table, and FSE tables at the maximum table log with an absent symbol inside the listed range (the meta#4681 trigger) and, for LL/ML, omitted high symbols. It fails on dev under asserts and passes with this change.
terrelln
force-pushed
the
2026-11-16-rescale-freqs
branch
from
November 18, 2026 18:50
48184b2 to
768ef0d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ZSTD_rescaleFreqs() seeds its first-block statistics from dictionary
entropy tables. At the maximum LL/ML table log, an absent FSE symbol's
fake cost equals the frequency scale and trips the strict-less-than
assertion. Shortened LL/ML alphabets also leave trailing compression
transforms uninitialized even though the optimal parser reads every
symbol. Separately, the Huffman reader accepts 12-bit tables, which can
underflow the parser's 11-bit scale and cause an invalid shift in release
builds.
Use a shared cost-to-frequency conversion that preserves a minimum
frequency of one without underflowing the shift. Build dictionary LL/ML
tables over their full symbol ranges, using the zero counts supplied by
FSE_readNCount(). Keep the original repeat-mode checks so absent symbols
are not treated as safe to encode.
Add deterministic dictionary regressions for zero-frequency and omitted
FSE symbols, 12-bit Huffman tables, and all optimal-parser strategies.
Check the initialized costs and repeat modes, then verify public CDict
compression round trips.
Fixes #4681