Skip to content

ci: run playTests.sh against the MSVC build in visual-runtime-tests - #4968

Merged
Cyan4973 merged 2 commits into
devfrom
ci-visual-runtime-tests
Nov 17, 2026
Merged

Cyan4973 merged 2 commits into
devfrom
ci-visual-runtime-tests

Conversation

@Cyan4973

Copy link
Copy Markdown
Contributor

visual-runtime-tests is meant to run playTests.sh against the MSVC build. It doesn't, and it never reports a failure.

  • Wrong binary. Since 2ff87ae (2025-03), playTests.sh calls the zstd under test through an alias. bash only expands aliases in a script in POSIX mode, i.e. when started as sh.
    • The alias isn't expanded when pwsh runs sh -e playTests.sh: the trace shows + zstd ... instead of + ./zstd.exe .... So every zstd call runs the zstd preinstalled on the runner.
    • The x64 and Win32 jobs both print *** Zstandard CLI (64-bit) v1.5.7, while fuzzer.exe reports 1.6.0.
    • The same happens with bash playTests.sh on any machine where zstd is installed.
  • Hidden failures. The step ran under pwsh, which doesn't stop when a program fails, and only reported the exit code of its last command, fuzzer.exe. Today, playTests.sh stops at the junction test of programs: don't follow symbolic links and junctions in -r on Windows #4905, because v1.5.7 follows the junction. The job is still green.

Changes:

  • playTests.sh enables expand_aliases when it runs in bash. The alias stays: POSIX doesn't require VAR=value function to export VAR to the commands the function runs, and tests like ZSTD_NBTHREADS=4 zstd ... rely on it.
  • The tests run in their own step, under bash, which stops at the first failing command.

Checked locally:

  • On Linux with zstd v1.5.5 installed: before, bash playTests.sh ran /usr/bin/zstd v1.5.5 and failed. After, it runs ../programs/zstd and passes. sh playTests.sh passes before and after.
  • shellcheck v0.7.1, with the CI options: no warning.

This is the first run of playTests.sh against the MSVC build since March 2025, so it may expose real failures.

playTests.sh calls the zstd under test through an alias. bash only
expands aliases in a script in POSIX mode, i.e. when started as sh.
Started as bash, `zstd` silently runs the zstd found in PATH instead.

This happens in visual-runtime-tests: since the alias replaced a
function (2ff87ae, 2025-03), playTests.sh there tests the runner's
zstd v1.5.7, not the MSVC build. It also happens with
`bash playTests.sh` on any machine where zstd is installed.

Enable expand_aliases when the script runs in bash.
The step ran under pwsh, which doesn't stop when a program fails, and
only reported the exit code of its last command, fuzzer.exe. Since the
job was added in 2021, failures of playTests.sh were never reported.

The tests now run in their own step, under bash, which stops at the
first failing command.
@meta-cla meta-cla Bot added the CLA Signed label Nov 15, 2026
@Cyan4973 Cyan4973 self-assigned this Nov 16, 2026
@Cyan4973 Cyan4973 added the CI label Nov 16, 2026
@Cyan4973
Cyan4973 merged commit 8b7a76a into dev Nov 17, 2026
114 checks passed
@Cyan4973
Cyan4973 deleted the ci-visual-runtime-tests branch November 17, 2026 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants