Repository navigation
AArch64: Add SVE2 implementation of histogram computation - #4411
Merged
Merged
Conversation
The existing scalar implementation uses a 4-way pipelined histogram calculation which is very efficient on out-of-order CPUs. However, this can be further accelerated using the SVE2 HISTSEG instructions - which compute a histogram for 16 byte chunks in a vector register. On a system with 128-bit vectors (VL128) we need 16 HISTSEG executions to compute the histogram for the whole symbol space (0..255) of 16 bytes input. However we can only accumulate 15 of such 16 byte strips before possible overflow. So we need to extend and save the 8-bit histogram accumulators to 16-bit after every 240 byte chunks of input. To store all in registers we would need 32 128-bit registers. Longer SVE2 vectors could help here, if such machines become available. The maximum input block size in Zstd is 128 KiB, so 16-bit accumulators would not be enough. However an LZ pass will prepend the histogram calculation, so it is impossible (my assumption) to overflow the 16-bit accumulators. The symbol distribution is also not uniform, the lower values are more common, so we used a 3 pass algorithm to prevent stack spilling. In the first pass we only compute histograms for 64 symbols (4-way SIMD) while also computing the maximum symbol value. If we have symbol values larger than 64 we start the second pass to compute the next 96 elements of the histogram. The final pass calculates the remaining part of the histogram (256 symbols in total) if needed. This split of histogram generation gave the best overall results for performance. This implementation is the best performing of a number of different cache blocking schemes tested. Compression uplifts on a Neoverse V2 system, using Zstd-1.5.8 (e26dde3) as a baseline, compiled with "-O3 -march=armv8.2-a+sve2": Clang-20 GCC-14 1#silesia.tar: +6.173% +5.987% 2#silesia.tar: +5.200% +5.011% 3#silesia.tar: +4.332% +5.031% 4#silesia.tar: +2.789% +3.064% 5#silesia.tar: +2.028% +1.838% 6#silesia.tar: +1.562% +1.340% 7#silesia.tar: +1.160% +0.959%
The following tests are included: - Empty input scenario test. - Workspace size and alignment tests. - Symbol out-of-range tests. - Cover multiple input sizes, vary permitted maximum symbol values, and include diverse symbol distributions. These tests verifies count table correctness, maxSymbolValuePtr updates, and error-handling paths. It enables automated regression of core histogram logic as well.
Cyan4973
approved these changes
Jun 16, 2025
Cyan4973
left a comment
Contributor
There was a problem hiding this comment.
It's a well located change, easy to follow, complete with published benchmark and a test set.
49 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The existing scalar implementation uses a 4-way pipelined histogram calculation which is very efficient on current CPUs. However, this can be further accelerated using the SVE2 HISTSEG instruction. Details are in the commit messages.
We also added some unit tests to the
fuzzer.cto cover the existing and new SVE2 histogram functions, along with new SVE2-enabled QEMU runs in the CI (had to increase the timeout of thetest-cli-testsfrom 200s to 800s).Compression uplifts on a Neoverse V2 system, using Zstd-1.5.8 (e26dde3) as a baseline, compiled with
-O3 -march=armv8.2-a+sve2:The uplifts are much smaller at higher compression levels!