Repository navigation
AArch64: Improve ZSTD_decodeSequence performance - #4418
Conversation
LLVM's alias-analysis sometimes fails to see that a static-array member
of a struct cannot alias other members. This patch:
- Reduces array accesses via struct indirection to aid load/store alias
analysis under Clang.
- Converts dynamic array indexing into conditional-move arithmetic,
eliminating branches and extra loads/stores on out-of-order CPUs.
- Reloads the bitstream only when match-length bits are consumed
(assuming each reload only needs to happen once per match-length
read), improving branch-prediction rates.
- Removes the UNLIKELY() hint, which recent compilers already handle
well without cost.
Decompression uplifts on a Neoverse V2 system, using Zstd-1.5.8
compiled with "-O3 -march=armv8.2-a+sve2":
Clang-19 Clang-20 Clang-* GCC-14 GCC-15
1#silesia.tar: +11.556% +16.203% +0.240% +2.216% +7.891%
2#silesia.tar: +15.493% +21.140% -0.041% +2.850% +9.926%
3#silesia.tar: +16.887% +22.570% -0.183% +3.056% +10.660%
4#silesia.tar: +17.785% +23.315% -0.262% +3.343% +11.187%
5#silesia.tar: +18.125% +24.175% -0.466% +3.350% +11.228%
6#silesia.tar: +17.607% +23.339% -0.591% +3.175% +10.851%
7#silesia.tar: +17.463% +22.837% -0.486% +3.292% +10.868%
* Requires Clang-21 support from LLVM commit hash
`a53003fe23cb6c871e72d70ff2d3a075a7490da2`
(Clang-21 hasn’t been released as of this writing)
Co-authored by:
David Sherwood, David.Sherwood@arm.com
Ola Liljedahl, Ola.Liljedahl@arm.com
|
These are impressive speed gains, it's also interesting to note that the measured benefits seem only significant for latest versions of |
|
As a complementary datapoint, |
|
We mostly tested on recent compilers: GCC-13..15, Clang-19..21. Here are some results relative to Clang-19 as baseline for the first 7 compression levels on Neoverse V2 before:
Neoverse V2 after:
Cortex-A720 before:
Cortex-A720 after:
Vanilla GCC-14 is the fastest, the in-development Clang-21 closed the gap mostly. With the patch applied GCC-14 is still the fastest but Clang-20 became the closest competitor. |
Cyan4973
left a comment
There was a problem hiding this comment.
There might be ways to reduce code duplication in order to improve code maintenance,
but I'm concerned such an effort would be challenging to synchronize on github.
So let's take this optimization as is, we'll figure out a way to reduce its maintenance complexity in a future effort.
| if (mlBits > 0) { | ||
| seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/); | ||
|
|
||
| if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32)) |
There was a problem hiding this comment.
Is this one the equivalent code? @Cyan4973
In x86 version I see there is no scope for if (mlBits > 0)
if (mlBits > 0)
seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);
if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
BIT_reloadDStream(&seqState->DStream);
if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
BIT_reloadDStream(&seqState->DStream);
There was a problem hiding this comment.
Indeed, it's not "strictly equivalent".
For the first if (MEM_32bits(... branch, I believe it works, accidentally.
One can see that, if mlBits == 0,
then we will not reach mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32, because llBits is capped <= 16.
This works because STREAM_ACCUMULATOR_MIN_32 == 25 and LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5.
Unfortunately, for the second if (MEM_64bits(... branch, it doesn't work so well.
mlBits==0, but we could still max out at llBits==16 + ofBits==30, for a total of 46.
It's of course unlikely, but not impossible,
However, the limit is supposed to be (STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)).
with STREAM_ACCUMULATOR_MIN_64==57 and (LLFSELog+MLFSELog+OffFSELog)==9+9+8==26,
hence a safe maximum of 41.
Now, in practice, there is some additional margin, because it implies that all FSE symbols are maxed out too, which is less likely when the nb of symbols is low, which it probably is if llBits==16. But all this is just "probably". Not guaranteed. A specific input might be able to trigger this issue.
It also shows that it's difficult to reach these conditions, because it implies a huge amount of data, in order to reach very large offset and length values, in order to produce many bits in a single sequence. Most automated tests focus on smaller dataset, for a question of iteration speed and also memory limits.
Anyway, I think it would deserve a fix.
Even the first if() branch, which seems safe, would deserve a STATIC_ASSERT() test if it stays within the if (mlBits > 0) { ... } code block, to ensure that the constants have the required values to guarantee success, in case someone changes them in some future.
There was a problem hiding this comment.
Thanks for a such fast response
If I set in the x86 version
if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
+{
+ if (mlBits == 0) abort();
BIT_reloadDStream(&seqState->DStream);
+}Some tests from fuzzer.c tests fail with abort
There was a problem hiding this comment.
So that's good news, it seems we have a test that would have detected the issue.
It just (probably) wasn't run on this code path.
There was a problem hiding this comment.
Or it was not enough to detect the corruption, i.e. worked for other reasons
Anyway, thanks for a blazingly fast response. Credit to 1 internal team at Google which caught it during the qual release and brought it to our attention
There was a problem hiding this comment.
Thanks for pointing out that this could lead to errors in extreme cases!
If we restore the bitstream reload to its original position with an UNLIKELY hint (useful for most compilers here):
if (mlBits > 0)
seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);
if (UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
BIT_reloadDStream(&seqState->DStream);the performance gains mostly disappear on GCC (this patch was specifically created for LLVM), while Clang remains largely unaffected, Clang-19 can even improve!
Measurements on Neoverse-V2 show the following results:
Clang-19 Clang-20 Clang-21 GCC-13 GCC-14 GCC-15
1#silesia.tar: +2.950% +0.000% -0.229% -0.896% -2.331% -6.305%
2#silesia.tar: +3.752% +0.041% -0.452% -1.052% -2.615% -7.742%
3#silesia.tar: +3.872% +0.000% -0.457% -1.102% -2.768% -8.321%
4#silesia.tar: +3.734% -0.049% -0.453% -1.336% -2.964% -8.686%
5#silesia.tar: +4.158% -0.049% -0.497% -1.292% -2.945% -8.668%
6#silesia.tar: +3.965% -0.053% -0.454% -1.251% -2.930% -8.451%
7#silesia.tar: +3.647% -0.138% -0.414% -1.300% -2.989% -8.452%
Shall I create a follow-up patch?
There was a problem hiding this comment.
Yeah, we need to fix that.
But we also probably need to expand our CI test suite a bit, adding a test that would have been able to catch the issue.
There was a problem hiding this comment.
An earlier message from @danilak-G suggested that it would be relatively easy to reproduce this bug during testing using the fuzzer, making it straightforward to detect.
However, the situation is more complicated than it appears.
The assert() only demonstrates that the "dangerous" branch can be executed even when mlBits == 0. The test itself merely checks that decoding will always be successful. If the test fails, it indicates that there might be a risk—something could potentially go wrong. In reality, several conditions must occur simultaneously to actually trigger a decoding error, so successful decoding remains quite likely.
Hence, so far, I have not been able to produce a decoding issue using the fuzzer.
Since g:b219cbeda72d23b7ad6ff12cd159784b7ef00667
The following
void f(const int *restrict in,
int *restrict out,
int n, int threshold)
{
for (int i = 0; i < n; ++i) {
int v = in[i];
if (v > threshold) {
int t = v * 3;
t += 7;
t ^= 0x55;
t *= 0x55;
t -= 0x5;
t &= 0xFE;
t ^= 0x55;
out[i] = t;
} else {
out[i] = v;
}
}
}
compiled at -O2
results in aggressive if-conversion which increases the number of dynamic
instructions and the latency of the loop as it has to wait for t to be
calculated now in all cases.
This has led to big performance losses in packages like zstd [1] which in turns
affects packaging and LTO speed.
The default cost model for if-conversion is overly permissive and allows if
conversions assuming that branches are very expensive.
This patch implements an if-conversion cost model for AArch64. AArch64 has a
number of conditional instructions that need to be accounted for, however this
initial version keeps things simple and is only really concerned about csel.
The issue specifically with csel is that it may have to wait for two argument
to be evaluated before it can be executed. This means it has a direct
correlation to increases in dynamic instructions.
To fix this I add a new tuning parameter that indicates a rough estimation of
the branch misprediction cost of a branch. We then accept if-conversion while
the cost of this multiplied by the cost of branches is cheaper.
There is a basic detection of CINC and CSET because these usually are ok. We
also accept all if-conversion when not inside a loop. Because CE is not an RTL
SSA pass we can't do more extensive checks like checking if the csel is a loop
carried dependency. As such this is a best effort thing and intends to catch the
most egregious cases like the above.
This recovers the ~25% performance loss in zstd decoding and gives better
results than GCC 14 which was before the regression happened.
Additionally I've benchmarked on a number of cores all the attached examples
and checked various cases. On average the patch gives an improvement between
20-40%.
[1] meta/zstd#4418 (comment)
gcc/ChangeLog:
PR target/123017
* config/aarch64/aarch64-json-schema.h: Add br_mispredict_factor.
* config/aarch64/aarch64-json-tunings-parser-generated.inc
(parse_branch_costs): Add br_mispredict_factor.
* config/aarch64/aarch64-json-tunings-printer-generated.inc
(serialize_branch_costs): Add br_mispredict_factor.
* config/aarch64/aarch64-protos.h (struct cpu_branch_cost): Add
br_mispredict_factor.
* config/aarch64/aarch64.cc (aarch64_max_noce_ifcvt_seq_cost,
aarch64_noce_conversion_profitable_p,
TARGET_MAX_NOCE_IFCVT_SEQ_COST,
TARGET_NOCE_CONVERSION_PROFITABLE_P): New.
* config/aarch64/tuning_models/generic.h (generic_branch_cost): Add
br_mispredict_factor.
* config/aarch64/tuning_models/generic_armv8_a.h: Remove
generic_armv8_a_branch_cost and use generic_branch_cost.
gcc/testsuite/ChangeLog:
PR target/123017
* gcc.target/aarch64/pr123017_1.c: New test.
* gcc.target/aarch64/pr123017_2.c: New test.
* gcc.target/aarch64/pr123017_3.c: New test.
* gcc.target/aarch64/pr123017_4.c: New test.
* gcc.target/aarch64/pr123017_5.c: New test.
* gcc.target/aarch64/pr123017_6.c: New test.
* gcc.target/aarch64/pr123017_7.c: New test.
This patch implements an if-conversion cost model for AArch64. AArch64 has a number of conditional instructions that need to be accounted for, however this initial version keeps things simple and is only really concerned about csel. The issue specifically with csel is that it may have to wait for two argument to be evaluated before it can be executed. This means it has a direct correlation to increases in dynamic instructions. To fix this I add a new tuning parameter that indicates a rough estimation of the branch misprediction cost of a branch. We then accept if-conversion while the cost of this multiplied by the cost of branches is cheaper. There is a basic detection of CINC and CSET because these usually are ok. We also accept all if-conversion when not inside a loop. Because CE is not an RTL SSA pass we can't do more extensive checks like checking if the csel is a loop carried dependency. As such this is a best effort thing and intends to catch the most egregious cases like the above. This recovers the ~25% performance loss in zstd decoding and gives better results than GCC 14 which was before the regression happened. Additionally I've benchmarked on a number of cores all the attached examples and checked various cases. On average the patch gives an improvement between 20-40%. [1] meta/zstd#4418 (comment) gcc/ChangeLog: PR target/123017 * config/aarch64/aarch64-protos.h (struct cpu_branch_cost): Add br_mispredict_factor. * config/aarch64/aarch64.cc (aarch64_max_noce_ifcvt_seq_cost, aarch64_noce_conversion_profitable_p, TARGET_MAX_NOCE_IFCVT_SEQ_COST, TARGET_NOCE_CONVERSION_PROFITABLE_P): New. * config/aarch64/tuning_models/generic.h (generic_branch_cost): Add br_mispredict_factor. * config/aarch64/tuning_models/generic_armv8_a.h: Remove generic_armv8_a_branch_cost and use generic_branch_cost. gcc/testsuite/ChangeLog: PR target/123017 * gcc.target/aarch64/pr123017_1.c: New test. * gcc.target/aarch64/pr123017_2.c: New test. * gcc.target/aarch64/pr123017_3.c: New test. * gcc.target/aarch64/pr123017_4.c: New test. * gcc.target/aarch64/pr123017_5.c: New test. * gcc.target/aarch64/pr123017_6.c: New test. * gcc.target/aarch64/pr123017_7.c: New test.
LLVM's alias-analysis sometimes fails to see that a static-array member of a struct cannot alias other members. This patch:
UNLIKELY()hint, which recent compilers already handle well without cost.Decompression uplifts on a Neoverse V2 system, using Zstd-1.5.8 compiled with
-O3 -march=armv8.2-a+sve2:The same benchmarks on a Cortex-A720 system:
This change could be advantageous or detrimental on other platforms with some compilers (±7%), so we only enabled it for AArch64 only.
Co-authored by:
David Sherwood, David.Sherwood@arm.com
Ola Liljedahl, Ola.Liljedahl@arm.com