Skip to content

ci: run the pinned jobs on ubuntu-26.04 - #4949

Merged
Cyan4973 merged 8 commits into
devfrom
ci-ubuntu-26.04
Nov 13, 2026
Merged

Cyan4973 merged 8 commits into
devfrom
ci-ubuntu-26.04

Conversation

@Cyan4973

@Cyan4973 Cyan4973 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #4944. This fixes the causes of the ubuntu-26.04 failures and removes the 19 pins to ubuntu-24.04. One commit per cause; each commit removes its own pins:

  1. 32-bit links and CET: libzstd.mk adds -z cet-report=error after a test in 64-bit mode. zstd32, fullbench32, fuzzer32 and zstreamtest32 add -m32 afterwards, and the 32-bit system objects have no CET properties. With gcc on Ubuntu 24.04 and 26.04, make -C programs zstd32 fails with "missing IBT and SHSTK properties". gold ignored this option, which hid the problem in the sanitizer jobs. These targets now link with LDFLAGS_32, without the option; 64-bit links keep it.
  2. gcc-8 sanitizer jobs: they set vm.mmap_rnd_bits=28. The ASan runtime of gcc-8 loops on AddressSanitizer:DEADLYSIGNAL with the 32-bit mmap ASLR of recent kernels: the ubuntu-24.04 image sets 28, ubuntu-26.04 does not.
  3. gold linker: the Makefile sanitizer targets no longer force -fuse-ld=gold. It was added in 2017 for Travis CI, and gold is not installed on ubuntu-26.04. 14 jobs, plus clang-asan-ubsan-cli-tests (added by ci: run the cli-tests under ASan and UBSan #4930, pinned to ubuntu-24.04 by b09d9aa).
  4. qemu: arminstall and ppcinstall install qemu-user instead of qemu-user-static, which is only a virtual package on ubuntu-26.04. qemu-user exists from Ubuntu 20.04 to 26.04 and on Debian 12, and does not remove an installed qemu-user-static. The targets call qemu-<arch> when qemu-arm is installed, and qemu-<arch>-static otherwise, as before. armbuild, armfuzz.
  5. powerpc: gcc-multilib-powerpc-linux-gnu replaces lib64gcc-13-dev-powerpc-cross, which only matched the GCC 13 cross compiler. short-tests-1.
  6. versions test: old versions are built with -march=x86-64 on x86-64 hosts. With the x86-64-v3 default, v1.3.1 does not compile: its threading.h defines pthread_mutex_t as a macro, which clashes with the glibc type pulled in by <immintrin.h>. versions-compatibility.
  7. libzstd size:
    • the size is measured at -march=x86-64, because GCC on ubuntu-26.04 runners targets x86-64-v3 by default, which makes libzstd.so 21% larger;
    • each limit is the size measured with GCC 15.2, plus about 30%, rounded up;
    • check_size.py now prints every size;
    • the workflow documents the rule.
  8. dynamic-bmi2: each case now starts from -march=x86-64. With the x86-64-v3 default, the cases without -mbmi2 still had BMI2, so the generic code path and the runtime dispatch were no longer tested.

Checked locally:

  • 1: the sanitizer targets link without gold (clang asan+ubsan, msan, tsan; gcc asan+ubsan).
  • 2 to 5: checked in Ubuntu 24.04 and 26.04 containers.
    • 2: package install, make armbuild, and the ARM fuzzer under qemu-arm.
    • 3: make ppc64build and make ppcbuild.
    • 4: the full versions test with GCC 15 and x86-64-v3 as default target (46 versions).
    • 0: the 4 32-bit targets with gcc, make uasan-zstd32, and the 64-bit link still has -z cet-report=error.
    • 5: the four sizes are at 75-76% of their limits.
  • 6: with a default target of x86-64-v3, -march=x86-64 brings back DYNAMIC_BMI2=1 and BMI2 off.

Compatibility, checked in containers on Ubuntu 20.04, 22.04, 24.04 and Debian 12, comparing dev and this PR:

  • the sanitizer targets build and run, with and without gold;
  • make -C programs zstd32 with gcc fails on dev on 22.04 and 24.04 (CET), and passes with this PR everywhere;
  • with only qemu-user-static installed, make armfuzz still uses qemu-arm-static; after make arminstall, it uses qemu-arm, and qemu-user-static stays installed.

Not in this PR: qemu-consistency has been pinned to ubuntu-24.04 since 2025 and installs qemu-user-static and gcc-14-riscv64-linux-gnu. It needs the same treatment before ubuntu-24.04 is retired.

@chrisbbreuer

Copy link
Copy Markdown

While independently validating the Gold removal, the original GCC 32-bit sanitizer workload exposed an additional make target-context defect: the parse-time CET probe succeeds for the default 64-bit target and then enforces that result on a 32-bit link. I recorded the actual failing startup-object diagnostics, source fix, real native/compat32 compiler/link/runtime regression, and fully green own-fork CI in #4951. The additional source commits are linked there. Your PR already owns the Gold-removal topic; this is follow-up evidence for the target-specific hardening probe. Codex assisted this investigation and patch.

libzstd.mk adds -z cet-report=error when the compiler accepts it, but it
tests this in 64-bit mode. zstd32, fullbench32, fuzzer32 and zstreamtest32
add -m32 afterwards, and 32-bit system objects usually have no CET
properties: with gcc on Ubuntu 24.04 and 26.04, `make -C programs zstd32`
fails with "missing IBT and SHSTK properties". The sanitizer targets hid
it, because they forced the gold linker, which ignores this option.

These targets now link with LDFLAGS_32, which leaves the option out.
64-bit links keep it.
The ASan runtime of gcc-8 loops on AddressSanitizer:DEADLYSIGNAL when the
kernel uses 32 bits of mmap randomization. The ubuntu-24.04 image sets
vm.mmap_rnd_bits=28; ubuntu-26.04 does not, and gcc-8-asan-ubsan-fuzz
printed 102 million such lines there. Both gcc-8 jobs now set it
themselves. Newer ASan runtimes (gcc 15, clang) are not affected.
LDFLAGS=-fuse-ld=gold was added in 2017 for Travis CI (f58ac79). The
sanitizers link with the default linker, and gold is deprecated in
binutils: it is not installed on ubuntu-26.04 runners, so these targets
failed there. The 15 sanitizer jobs, including clang-asan-ubsan-cli-tests,
return to ubuntu-latest.
qemu-user-static is only a virtual package on ubuntu-26.04, so the ARM and
PowerPC install targets failed there. arminstall and ppcinstall now install
qemu-user, which exists from Ubuntu 20.04 to 26.04 and on Debian 12, and
does not remove an installed qemu-user-static.

qemu-user names its binaries qemu-<arch>, without the -static suffix. The
targets call qemu-<arch> when qemu-arm is installed, and qemu-<arch>-static
otherwise, as before: a machine with only qemu-user-static keeps working.
The tests call every cross-compiled binary through QEMU_SYS, so no binfmt
registration is needed. armbuild and armfuzz return to ubuntu-latest.
short-tests-1 installed lib64gcc-13-dev-powerpc-cross, which only matches
the GCC 13 cross compiler of ubuntu-24.04: with GCC 15 on ubuntu-26.04,
ppc64build could not find -lgcc. gcc-multilib-powerpc-linux-gnu installs
the 64-bit libraries of the default cross compiler, whatever its version.
short-tests-1 returns to ubuntu-latest.
GCC on ubuntu-26.04 runners targets x86-64-v3 by default. With BMI2
enabled, v1.3.1 does not compile: its threading.h defines pthread_mutex_t
as a macro, which then clashes with the glibc type pulled in by
<immintrin.h>. The versions test now builds with -march=x86-64 on x86-64
hosts. Verified with GCC 15 on Ubuntu 26.04 and x86-64-v3 as default
target: the 46 versions build and the test passes. versions-compatibility
returns to ubuntu-latest.
The size limits had margins between 1% and 100%. On ubuntu-26.04 runners,
GCC 15 targets x86-64-v3 by default, which makes libzstd.so 21% larger
(1105472 bytes instead of 909760), above the 1100000 limit.

The test now builds with -march=x86-64, so the measure does not depend on
the default target of the compiler. Each limit is the size measured with
GCC 15.2, plus about 30%, rounded up: this still catches a jump such as
#3066. check_size.py prints every size, to make the next measure easy. The
workflow documents the rule. libzstd-size returns to ubuntu-latest.
GCC on ubuntu-26.04 runners targets x86-64-v3, which enables BMI2 by
default. The dynamic-bmi2 cases without -mbmi2 then still had BMI2, so the
generic code path and the runtime dispatch were no longer tested. Each
case now starts from -march=x86-64.
@Cyan4973

Copy link
Copy Markdown
Contributor Author

Thanks for the investigation @chrisbbreuer and the detailed report in #4951.

This PR includes a narrower fix for the same defect: build: link the 32-bit targets without -z cet-report=error. The 32-bit targets (zstd32, fullbench32, fuzzer32, zstreamtest32) link with LDFLAGS_32, which leaves the option out, and 64-bit links keep it. It unblocks the ubuntu-26.04 migration without changing the probe itself.

Your approach has a wider goal. We'd rather review that as its own change, once this PR lands.

If you open a PR on top of dev at that point, please keep it focused, and describe the cost of probing at each expansion of FLAGS.

@Cyan4973
Cyan4973 merged commit 9948edf into dev Nov 13, 2026
114 checks passed
@Cyan4973
Cyan4973 deleted the ci-ubuntu-26.04 branch November 13, 2026 21:49
opensource-fork-sync Bot pushed a commit to cla7aye15I4nd/opensource-meta__zstd that referenced this pull request Nov 13, 2026
The four jobs that test gcc-7 and gcc-8 install them from the focal
archive. They also added ppa:ubuntu-toolchain-r/test through make
gcc7install and gcc8install, but no package came from it: it only cost
about 10 s per job, and a Launchpad SSL error failed one job on meta#4949.
The jobs now install the same packages with apt-get directly. The make
targets are unchanged, for the machines that still need the PPA.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants