Skip to content

Fix SVE2 histogram counter overflow - #4990

Open
terrelln wants to merge 1 commit into
meta:devfrom
terrelln:2026-11-18-hist-sve2
Open

terrelln wants to merge 1 commit into
meta:devfrom
terrelln:2026-11-18-hist-sve2

Conversation

@terrelln

@terrelln terrelln commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

#4711 flags an issue in SVE2 histogramming that this PR fixes.

The SVE2 histogram accumulates counts in U16 lanes, so a symbol that appears 65,536 or more times in one call wraps, possibly to zero. That yields wrong probabilities, missed RLE detection, and corrupt Huffman output.

Split large inputs into chunks of at most 65,520 bytes, a multiple of 16 that also leaves room for the zero padding of the final vector load. Merge each chunk's counts into U32 totals, tracking the largest count and max symbol across the whole input.

Add a fuzzer test that checks exact counts, largest count and max symbol for every byte value and for sizes around the chunk and 16-bit boundaries, through both HIST_count_wksp and HIST_countFast_wksp. It also checks that entries past maxSymbolValue are not written and that an out-of-range symbol in a later chunk returns an error.

The SVE2 histogram accumulates counts in U16 lanes, so a symbol that
appears 65,536 or more times in one call wraps, possibly to zero. That
yields wrong probabilities, missed RLE detection, and corrupt Huffman
output.

Split large inputs into chunks of at most 65,520 bytes, a multiple of 16
that also leaves room for the zero padding of the final vector load.
Merge each chunk's counts into U32 totals, tracking the largest count and
max symbol across the whole input.

Add a fuzzer test that checks exact counts, largest count and max symbol
for every byte value and for sizes around the chunk and 16-bit
boundaries, through both HIST_count_wksp and HIST_countFast_wksp. It also
checks that entries past maxSymbolValue are not written and that an
out-of-range symbol in a later chunk returns an error.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant