Skip to content

AArch64: Improve ZSTD_decodeSequence performance - #4418

Merged
Cyan4973 merged 1 commit into
meta:devfrom
arpadpanyik-arm:decode_seq_opt
Jul 19, 2025
Merged

Cyan4973 merged 1 commit into
meta:devfrom
arpadpanyik-arm:decode_seq_opt

Conversation

@arpadpanyik-arm

@arpadpanyik-arm arpadpanyik-arm commented Jun 24, 2025 •

Copy link
Copy Markdown
Contributor

LLVM's alias-analysis sometimes fails to see that a static-array member of a struct cannot alias other members. This patch:

  • Reduces array accesses via struct indirection to aid load/store alias analysis under Clang.
  • Converts dynamic array indexing into conditional-move arithmetic, eliminating branches and extra loads/stores for out-of-order CPUs.
  • Reloads the bitstream only when match-length bits are consumed (assuming each reload only needs to happen once per match-length read), improving branch-prediction rates.
  • Removes the UNLIKELY() hint, which recent compilers already handle well without cost.

Decompression uplifts on a Neoverse V2 system, using Zstd-1.5.8 compiled with -O3 -march=armv8.2-a+sve2:

                 Clang-19  Clang-20   Clang-*    GCC-13    GCC-14    GCC-15
 1#silesia.tar:  +11.556%  +16.203%   +0.240%   +0.609%   +2.216%   +7.891%
 2#silesia.tar:  +15.493%  +21.140%   -0.041%   +0.673%   +2.850%   +9.926%
 3#silesia.tar:  +16.887%  +22.570%   -0.183%   +1.029%   +3.056%  +10.660%
 4#silesia.tar:  +17.785%  +23.315%   -0.262%   +1.377%   +3.343%  +11.187%
 5#silesia.tar:  +18.125%  +24.175%   -0.466%   +1.213%   +3.350%  +11.228%
 6#silesia.tar:  +17.607%  +23.339%   -0.591%   +1.194%   +3.175%  +10.851%
 7#silesia.tar:  +17.463%  +22.837%   -0.486%   +1.372%   +3.292%  +10.868%

The same benchmarks on a Cortex-A720 system:

                 Clang-19  Clang-20   Clang-*    GCC-13    GCC-14    GCC-15
 1#silesia.tar:  +11.753%  +15.278%   +3.860%   +0.461%   +3.593%   +7.145%
 2#silesia.tar:  +14.838%  +17.210%   +4.785%   +0.809%   +4.489%   +8.910%
 3#silesia.tar:  +15.713%  +16.355%   +4.517%   +1.200%   +4.716%   +9.361%
 4#silesia.tar:  +15.733%  +14.869%   +4.065%   +1.628%   +4.638%   +9.381%
 5#silesia.tar:  +16.406%  +16.097%   +4.547%   +1.422%   +4.853%   +9.693%
 6#silesia.tar:  +16.195%  +15.881%   +4.460%   +1.279%   +4.531%   +9.260%
 7#silesia.tar:  +15.597%  +14.336%   +3.907%   +1.564%   +4.259%   +9.057%

* Requires Clang-21 support from LLVM commit hash
  `a53003fe23cb6c871e72d70ff2d3a075a7490da2`
  (Clang-21 hasn’t been released as of this writing)

This change could be advantageous or detrimental on other platforms with some compilers (±7%), so we only enabled it for AArch64 only.

Co-authored by:
David Sherwood, David.Sherwood@arm.com
Ola Liljedahl, Ola.Liljedahl@arm.com

LLVM's alias-analysis sometimes fails to see that a static-array member
of a struct cannot alias other members. This patch:

- Reduces array accesses via struct indirection to aid load/store alias
  analysis under Clang.
- Converts dynamic array indexing into conditional-move arithmetic,
  eliminating branches and extra loads/stores on out-of-order CPUs.
- Reloads the bitstream only when match-length bits are consumed
  (assuming each reload only needs to happen once per match-length
  read), improving branch-prediction rates.
- Removes the UNLIKELY() hint, which recent compilers already handle
  well without cost.

Decompression uplifts on a Neoverse V2 system, using Zstd-1.5.8
compiled with "-O3 -march=armv8.2-a+sve2":

                 Clang-19  Clang-20   Clang-*    GCC-14    GCC-15
 1#silesia.tar:  +11.556%  +16.203%   +0.240%   +2.216%   +7.891%
 2#silesia.tar:  +15.493%  +21.140%   -0.041%   +2.850%   +9.926%
 3#silesia.tar:  +16.887%  +22.570%   -0.183%   +3.056%  +10.660%
 4#silesia.tar:  +17.785%  +23.315%   -0.262%   +3.343%  +11.187%
 5#silesia.tar:  +18.125%  +24.175%   -0.466%   +3.350%  +11.228%
 6#silesia.tar:  +17.607%  +23.339%   -0.591%   +3.175%  +10.851%
 7#silesia.tar:  +17.463%  +22.837%   -0.486%   +3.292%  +10.868%

* Requires Clang-21 support from LLVM commit hash
  `a53003fe23cb6c871e72d70ff2d3a075a7490da2`
   (Clang-21 hasn’t been released as of this writing)

Co-authored by:
 David Sherwood, David.Sherwood@arm.com
 Ola Liljedahl, Ola.Liljedahl@arm.com
@Cyan4973

Cyan4973 commented Jun 25, 2025 •

Copy link
Copy Markdown
Contributor

These are impressive speed gains,

it's also interesting to note that the measured benefits seem only significant for latest versions of clang and gcc.
Since we only see the relative gains, not absolute numbers, it's unclear if this PR fixes a performance degradation that appeared in these versions, or provide much better performance when combined with recent compilers.

@Cyan4973

Copy link
Copy Markdown
Contributor

As a complementary datapoint,
using Apple clang version 17.0.0 (clang-1700.0.13.5) compiler,
I observe ~+8/9% decompression speed improvement on a M1 Pro,
which is still pretty significant.

@arpadpanyik-arm

arpadpanyik-arm commented Jun 25, 2025 •

Copy link
Copy Markdown
Contributor Author

We mostly tested on recent compilers: GCC-13..15, Clang-19..21. Here are some results relative to Clang-19 as baseline for the first 7 compression levels on silesia.tar:

Neoverse V2 before:

Clang-19 Clang-20 Clang-21 GCC-13 GCC-14 GCC-15
100.000% 103.098% 119.322% 101.879% 121.210% 106.283%
100.000% 101.894% 123.367% 99.660% 124.836% 104.561%
100.000% 101.394% 124.384% 98.772% 126.234% 104.129%
100.000% 101.188% 124.886% 98.466% 126.766% 103.661%
100.000% 100.995% 125.810% 98.107% 127.285% 103.580%
100.000% 101.242% 125.456% 98.448% 127.166% 104.153%
100.000% 101.238% 124.760% 98.598% 126.777% 104.099%

Neoverse V2 after:

Clang-19 Clang-20 Clang-21 GCC-13 GCC-14 GCC-15
111.556% 119.804% 119.610% 102.500% 123.866% 114.670%
115.493% 123.434% 123.316% 100.331% 128.429% 114.940%
116.887% 124.278% 124.156% 99.789% 130.083% 115.230%
117.785% 124.780% 124.558% 99.822% 131.022% 115.258%
118.125% 125.410% 125.224% 99.298% 131.541% 115.211%
117.607% 124.872% 124.714% 99.624% 131.178% 115.455%
117.463% 124.358% 124.153% 99.950% 130.925% 115.413%

Cortex-A720 before:

Clang-19 Clang-20 Clang-21 GCC-13 GCC-14 GCC-15
100.000% 102.657% 114.968% 105.732% 119.934% 107.245%
100.000% 102.134% 115.215% 104.371% 119.997% 104.531%
100.000% 102.300% 114.657% 104.751% 119.145% 104.198%
100.000% 102.895% 114.136% 105.519% 117.830% 103.828%
100.000% 102.496% 114.464% 104.900% 118.929% 103.712%
100.000% 102.602% 114.481% 105.119% 119.230% 104.292%
100.000% 103.061% 113.965% 105.895% 118.203% 104.326%

Cortex-A720 after:

Clang-19 Clang-20 Clang-21 GCC-13 GCC-14 GCC-15
111.753% 118.341% 119.406% 106.220% 124.243% 114.908%
114.838% 119.712% 120.728% 105.216% 125.385% 113.845%
115.713% 119.032% 119.836% 106.008% 124.764% 113.953%
115.733% 118.195% 118.777% 107.237% 123.295% 113.568%
116.406% 118.996% 119.669% 106.393% 124.702% 113.764%
116.195% 118.897% 119.588% 106.463% 124.633% 113.950%
115.597% 117.836% 118.418% 107.552% 123.238% 113.776%

Vanilla GCC-14 is the fastest, the in-development Clang-21 closed the gap mostly. With the patch applied GCC-14 is still the fastest but Clang-20 became the closest competitor.

@Cyan4973 Cyan4973 self-assigned this Jun 30, 2025

@Cyan4973 Cyan4973 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might be ways to reduce code duplication in order to improve code maintenance,
but I'm concerned such an effort would be challenging to synchronize on github.
So let's take this optimization as is, we'll figure out a way to reduce its maintenance complexity in a future effort.

@Cyan4973
Cyan4973 merged commit 8c65186 into meta:dev Jul 19, 2025
if (mlBits > 0) {
seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);

if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))

@danilak-G danilak-G Sep 29, 2025 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this one the equivalent code? @Cyan4973

In x86 version I see there is no scope for if (mlBits > 0)

        if (mlBits > 0)
            seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);

        if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
            BIT_reloadDStream(&seqState->DStream);
        if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
            BIT_reloadDStream(&seqState->DStream);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, it's not "strictly equivalent".

For the first if (MEM_32bits(... branch, I believe it works, accidentally.
One can see that, if mlBits == 0,
then we will not reach mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32, because llBits is capped <= 16.
This works because STREAM_ACCUMULATOR_MIN_32 == 25 and LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5.

Unfortunately, for the second if (MEM_64bits(... branch, it doesn't work so well.
mlBits==0, but we could still max out at llBits==16 + ofBits==30, for a total of 46.
It's of course unlikely, but not impossible,
However, the limit is supposed to be (STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)).
with STREAM_ACCUMULATOR_MIN_64==57 and (LLFSELog+MLFSELog+OffFSELog)==9+9+8==26,
hence a safe maximum of 41.

Now, in practice, there is some additional margin, because it implies that all FSE symbols are maxed out too, which is less likely when the nb of symbols is low, which it probably is if llBits==16. But all this is just "probably". Not guaranteed. A specific input might be able to trigger this issue.

It also shows that it's difficult to reach these conditions, because it implies a huge amount of data, in order to reach very large offset and length values, in order to produce many bits in a single sequence. Most automated tests focus on smaller dataset, for a question of iteration speed and also memory limits.

Anyway, I think it would deserve a fix.

Even the first if() branch, which seems safe, would deserve a STATIC_ASSERT() test if it stays within the if (mlBits > 0) { ... } code block, to ensure that the constants have the required values to guarantee success, in case someone changes them in some future.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for a such fast response

If I set in the x86 version

if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
+{
+           if (mlBits == 0) abort();
            BIT_reloadDStream(&seqState->DStream);
+}

Some tests from fuzzer.c tests fail with abort

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that's good news, it seems we have a test that would have detected the issue.
It just (probably) wasn't run on this code path.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or it was not enough to detect the corruption, i.e. worked for other reasons

Anyway, thanks for a blazingly fast response. Credit to 1 internal team at Google which caught it during the qual release and brought it to our attention

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing out that this could lead to errors in extreme cases!
If we restore the bitstream reload to its original position with an UNLIKELY hint (useful for most compilers here):

        if (mlBits > 0)
            seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);

        if (UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
            BIT_reloadDStream(&seqState->DStream);

the performance gains mostly disappear on GCC (this patch was specifically created for LLVM), while Clang remains largely unaffected, Clang-19 can even improve!

Measurements on Neoverse-V2 show the following results:

                 Clang-19  Clang-20   Clang-21   GCC-13    GCC-14    GCC-15
 1#silesia.tar:   +2.950%   +0.000%   -0.229%   -0.896%   -2.331%   -6.305%
 2#silesia.tar:   +3.752%   +0.041%   -0.452%   -1.052%   -2.615%   -7.742%
 3#silesia.tar:   +3.872%   +0.000%   -0.457%   -1.102%   -2.768%   -8.321%
 4#silesia.tar:   +3.734%   -0.049%   -0.453%   -1.336%   -2.964%   -8.686%
 5#silesia.tar:   +4.158%   -0.049%   -0.497%   -1.292%   -2.945%   -8.668%
 6#silesia.tar:   +3.965%   -0.053%   -0.454%   -1.251%   -2.930%   -8.451%
 7#silesia.tar:   +3.647%   -0.138%   -0.414%   -1.300%   -2.989%   -8.452%

Shall I create a follow-up patch?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we need to fix that.
But we also probably need to expand our CI test suite a bit, adding a test that would have been able to catch the issue.

@Cyan4973 Cyan4973 Oct 2, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An earlier message from @danilak-G suggested that it would be relatively easy to reproduce this bug during testing using the fuzzer, making it straightforward to detect.

However, the situation is more complicated than it appears.

The assert() only demonstrates that the "dangerous" branch can be executed even when mlBits == 0. The test itself merely checks that decoding will always be successful. If the test fails, it indicates that there might be a risk—something could potentially go wrong. In reality, several conditions must occur simultaneously to actually trigger a decoding error, so successful decoding remains quite likely.

Hence, so far, I have not been able to produce a decoding issue using the fuzzer.

hubot pushed a commit to gcc-mirror/gcc that referenced this pull request Jan 5, 2026
Since g:b219cbeda72d23b7ad6ff12cd159784b7ef00667

The following

void f(const int *restrict in,
   int *restrict out,
   int n, int threshold)
{
  for (int i = 0; i < n; ++i) {
    int v = in[i];
    if (v > threshold) {
      int t = v * 3;
      t += 7;
      t ^= 0x55;
      t *= 0x55;
      t -= 0x5;
      t &= 0xFE;
      t ^= 0x55;
      out[i] = t;
    } else {
      out[i] = v;
    }
  }
}

compiled at -O2

results in aggressive if-conversion which increases the number of dynamic
instructions and the latency of the loop as it has to wait for t to be
calculated now in all cases.

This has led to big performance losses in packages like zstd [1] which in turns
affects packaging and LTO speed.

The default cost model for if-conversion is overly permissive and allows if
conversions assuming that branches are very expensive.

This patch implements an if-conversion cost model for AArch64.   AArch64 has a
number of conditional instructions that need to be accounted for, however this
initial version keeps things simple and is only really concerned about csel.

The issue specifically with csel is that it may have to wait for two argument
to be evaluated before it can be executed.  This means it has a direct
correlation to increases in dynamic instructions.

To fix this I add a new tuning parameter that indicates a rough estimation of
the branch misprediction cost of a branch.   We then accept if-conversion while
the cost of this multiplied by the cost of branches is cheaper.

There is a basic detection of CINC and CSET because these usually are ok.  We
also accept all if-conversion when not inside a loop.  Because CE is not an RTL
SSA pass we can't do more extensive checks like checking if the csel is a loop
carried dependency.  As such this is a best effort thing and intends to catch the
most egregious cases like the above.

This recovers the ~25% performance loss in zstd decoding and gives better
results than GCC 14 which was before the regression happened.

Additionally I've benchmarked on a number of cores all the attached examples
and checked various cases.  On average the patch gives an improvement between
20-40%.

[1] meta/zstd#4418 (comment)

gcc/ChangeLog:

	PR target/123017
	* config/aarch64/aarch64-json-schema.h: Add br_mispredict_factor.
	* config/aarch64/aarch64-json-tunings-parser-generated.inc
	(parse_branch_costs): Add br_mispredict_factor.
	* config/aarch64/aarch64-json-tunings-printer-generated.inc
	(serialize_branch_costs): Add br_mispredict_factor.
	* config/aarch64/aarch64-protos.h (struct cpu_branch_cost): Add
	br_mispredict_factor.
	* config/aarch64/aarch64.cc (aarch64_max_noce_ifcvt_seq_cost,
	aarch64_noce_conversion_profitable_p,
	TARGET_MAX_NOCE_IFCVT_SEQ_COST,
	TARGET_NOCE_CONVERSION_PROFITABLE_P): New.
	* config/aarch64/tuning_models/generic.h (generic_branch_cost): Add
	br_mispredict_factor.
	* config/aarch64/tuning_models/generic_armv8_a.h: Remove
	generic_armv8_a_branch_cost and use generic_branch_cost.

gcc/testsuite/ChangeLog:

	PR target/123017
	* gcc.target/aarch64/pr123017_1.c: New test.
	* gcc.target/aarch64/pr123017_2.c: New test.
	* gcc.target/aarch64/pr123017_3.c: New test.
	* gcc.target/aarch64/pr123017_4.c: New test.
	* gcc.target/aarch64/pr123017_5.c: New test.
	* gcc.target/aarch64/pr123017_6.c: New test.
	* gcc.target/aarch64/pr123017_7.c: New test.
codonell pushed a commit to gnutools/gcc that referenced this pull request Mar 30, 2026
This patch implements an if-conversion cost model for AArch64.   AArch64 has a
number of conditional instructions that need to be accounted for, however this
initial version keeps things simple and is only really concerned about csel.

The issue specifically with csel is that it may have to wait for two argument
to be evaluated before it can be executed.  This means it has a direct
correlation to increases in dynamic instructions.

To fix this I add a new tuning parameter that indicates a rough estimation of
the branch misprediction cost of a branch.   We then accept if-conversion while
the cost of this multiplied by the cost of branches is cheaper.

There is a basic detection of CINC and CSET because these usually are ok.  We
also accept all if-conversion when not inside a loop.  Because CE is not an RTL
SSA pass we can't do more extensive checks like checking if the csel is a loop
carried dependency.  As such this is a best effort thing and intends to catch the
most egregious cases like the above.

This recovers the ~25% performance loss in zstd decoding and gives better
results than GCC 14 which was before the regression happened.

Additionally I've benchmarked on a number of cores all the attached examples
and checked various cases.  On average the patch gives an improvement between
20-40%.

[1] meta/zstd#4418 (comment)

gcc/ChangeLog:

	PR target/123017
	* config/aarch64/aarch64-protos.h (struct cpu_branch_cost): Add
	br_mispredict_factor.
	* config/aarch64/aarch64.cc (aarch64_max_noce_ifcvt_seq_cost,
	aarch64_noce_conversion_profitable_p,
	TARGET_MAX_NOCE_IFCVT_SEQ_COST,
	TARGET_NOCE_CONVERSION_PROFITABLE_P): New.
	* config/aarch64/tuning_models/generic.h (generic_branch_cost): Add
	br_mispredict_factor.
	* config/aarch64/tuning_models/generic_armv8_a.h: Remove
	generic_armv8_a_branch_cost and use generic_branch_cost.

gcc/testsuite/ChangeLog:

	PR target/123017
	* gcc.target/aarch64/pr123017_1.c: New test.
	* gcc.target/aarch64/pr123017_2.c: New test.
	* gcc.target/aarch64/pr123017_3.c: New test.
	* gcc.target/aarch64/pr123017_4.c: New test.
	* gcc.target/aarch64/pr123017_5.c: New test.
	* gcc.target/aarch64/pr123017_6.c: New test.
	* gcc.target/aarch64/pr123017_7.c: New test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants