Skip to content

Fix SVE2 histogram accumulator overflow - #4711

Closed
AGSaidi wants to merge 1 commit into
meta:devfrom
AGSaidi:fix-sve2-histogram-overflow
Closed

AGSaidi wants to merge 1 commit into
meta:devfrom
AGSaidi:fix-sve2-histogram-overflow

Conversation

@AGSaidi

@AGSaidi AGSaidi commented Jul 17, 2026

Copy link
Copy Markdown

HIST_count_sve2 reduces symbol counts into 16-bit accumulators, which wrap past 65535. On blocks up to ZSTD_BLOCKSIZE_MAX (128 KB) a symbol can exceed that, wrapping a frequent symbol to a small count and producing a bad (but valid) Huffman table, inflating the block by up to ~60%. Use a saturating add (svqadd) so the count caps at 65535 and the frequency ordering is preserved.

HIST_count_sve2 reduces symbol counts into 16-bit accumulators, which
wrap past 65535. On blocks up to ZSTD_BLOCKSIZE_MAX (128 KB) a symbol
can exceed that, wrapping a frequent symbol to a small count and
producing a bad (but valid) Huffman table, inflating the block by up
to ~60%. Use a saturating add (svqadd) so the count caps at 65535 and
the frequency ordering is preserved.
@terrelln

Copy link
Copy Markdown
Contributor

Thanks for flagging the issue @AGSaidi & for putting up the PR!

This doesn't fix the entire issue, as it isn't safe to saturate a U16 counter, as we need exact counts in some cases, so I'm closing and fixing in PR #4990.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants