Skip to content

Fix optimal-parser dictionary frequency initialization - #4978

Merged
terrelln merged 1 commit into
meta:devfrom
terrelln:2026-11-16-rescale-freqs
Nov 18, 2026
Merged

terrelln merged 1 commit into
meta:devfrom
terrelln:2026-11-16-rescale-freqs

Conversation

@terrelln

Copy link
Copy Markdown
Contributor

ZSTD_rescaleFreqs() seeds its first-block statistics from dictionary
entropy tables. At the maximum LL/ML table log, an absent FSE symbol's
fake cost equals the frequency scale and trips the strict-less-than
assertion. Shortened LL/ML alphabets also leave trailing compression
transforms uninitialized even though the optimal parser reads every
symbol. Separately, the Huffman reader accepts 12-bit tables, which can
underflow the parser's 11-bit scale and cause an invalid shift in release
builds.

Use a shared cost-to-frequency conversion that preserves a minimum
frequency of one without underflowing the shift. Build dictionary LL/ML
tables over their full symbol ranges, using the zero counts supplied by
FSE_readNCount(). Keep the original repeat-mode checks so absent symbols
are not treated as safe to encode.

Add deterministic dictionary regressions for zero-frequency and omitted
FSE symbols, 12-bit Huffman tables, and all optimal-parser strategies.
Check the initialized costs and repeat modes, then verify public CDict
compression round trips.

Fixes #4681

@meta-cla meta-cla Bot added the CLA Signed label Nov 16, 2026
@terrelln
terrelln force-pushed the 2026-11-16-rescale-freqs branch 2 times, most recently from 9f8811a to 48184b2 Compare November 17, 2026 19:07

@Cyan4973 Cyan4973 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes dictionary compression with the optimal parser (btopt, btultra, btultra2).
The fix looks right to me, and output with zstd --train dictionaries is unchanged.

However, the trimmed test passes on dev: it only catches the uninitialized read, and only under MSan, so it no longer covers #4681. Two small changes restore that coverage. Both fail on dev and pass with this PR:

  • in FUZ_writeNCount, give symbol 1 a zero count, and list symbols 0-2 for ML/LL. That is the #4681 trigger: an absent symbol inside the listed range, at the max table log.
  • build the Huffman table at log 12, with skewed counts: counts[i] = i < 16 ? 1U << (16 - i) : 1.

minor: the commit message still describes checks that were dropped with the trim (initialized costs, repeat modes, 12-bit Huffman).

ZSTD_rescaleFreqs() seeds its first-block statistics from dictionary
entropy tables. At the maximum LL/ML table log, an absent FSE symbol's
fake cost equals the frequency scale and trips the strict-less-than
assertion. Shortened LL/ML alphabets also leave trailing compression
transforms uninitialized even though the optimal parser reads every
symbol. Separately, the Huffman reader accepts 12-bit tables, which can
underflow the parser's 11-bit scale and cause an invalid shift in release
builds.

Use a shared cost-to-frequency conversion that preserves a minimum
frequency of one without underflowing the shift. Build dictionary LL/ML
tables over their full symbol ranges, using the zero counts supplied by
FSE_readNCount(). Keep the original repeat-mode checks so absent symbols
are not treated as safe to encode.

Add a hand-built dictionary regression that compresses with every
optimal-parser strategy and round-trips the result. The dictionary has a
skewed 12-bit Huffman table, and FSE tables at the maximum table log with
an absent symbol inside the listed range (the meta#4681 trigger) and, for
LL/ML, omitted high symbols. It fails on dev under asserts and passes
with this change.
@terrelln
terrelln force-pushed the 2026-11-16-rescale-freqs branch from 48184b2 to 768ef0d Compare November 18, 2026 18:50

@Cyan4973 Cyan4973 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test now covers #4681 and the 12-bit Huffman case, and it correctly fails on dev and passes here.

@terrelln
terrelln merged commit 35f14d3 into meta:dev Nov 18, 2026
110 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reachable assertion in ZSTD_rescaleFreqs (bitCost < scaleLog) when CDict built with optimal-parser strategy + tiny dictionary

2 participants