ci: every run script must parse -- two dispatch-only steps never could by gHashTag · Pull Request #839 · gHashTag/trinity-fpga

gHashTag · 2026-10-03T03:42:05Z

What was broken

Running bash -n on every workflow step found two steps that fail to parse every time they run. Both workflows are workflow_dispatch-only, so nothing reported them.

Step	Cause	Fix
build-matrix "Full bitstream build"	The `doesn't` in comment line 98 sits inside `bash -c '...'`. Its apostrophe closes the single-quoted script, and the rest is read by the outer shell.	Reworded.
iddr-golden-diff "Build the same design with openXC7 and diff the ILOGIC config"	`'\''` means "a quote" only inside single quotes. Line 169 is outside them, so the idiom leaves one quote open.	A plain `'ILOGIC_Y[01]\.[A-Za-z0-9_.]+'`.

Found by restoring the quoting. Inside the container, build-matrix tests [ "$OP" = "mul" ] to choose -nodsp. OP was never passed to docker run, so every MUL design would have built without -nodsp. It is now set and passed with -e OP.

The gate

tools/check_run_syntax.py runs a syntax check on every step that has a POSIX shell. It uses sh -n for shell: sh and bash -n otherwise. The shell is resolved in this order: the step's shell:, then the job's, then the workflow's defaults.run.shell, then bash. On a windows runner the default is pwsh, so those steps are skipped, as are python and pwsh shells.

Exit codes: 0 clean, 1 a step does not parse, 2 cannot tell. A file that is not YAML, a missing jobs, or a step that is not a mapping all count as "cannot tell".

bash -n cannot see an error inside a quoted bash -c '...' string. It does catch both cases here, because each one broke the outer quoting. The docstring states this limit.

tools/test_check_run_syntax.py has 18 cases. They include both files exactly as they were on main at e6eac090. If that commit cannot be read (a shallow clone), those cases fail instead of being skipped silently. run-syntax-gate.yml runs both checks and watches its own files; audit_workflow_paths.py is clean.

Evidence

Tree: 119 of 119 workflows read, 0 steps fail to parse. On main before this PR the count was 2.
tri mutants, run against the committed tree: 6 of 6 killed. The mutants: never reports; skips sh; checks windows as bash; checks pwsh as bash; treats "cannot tell" as clean; ignores the step's own shell.

Not in this PR

This gate is not added to check_gates_can_fail's harness. #828 adds its own gate to that same list and raises the floor. To avoid a conflict, this one will join after #828 merges.

🤖 Generated with Claude Code

`bash -n` on every workflow step found two that fail on parse each time they are dispatched: - build-matrix "Full bitstream build": a "doesn't" in a comment inside `bash -c '...'` closed the single-quoted script. Reworded. With the quoting restored, OP is now passed into the container: the -nodsp branch read it there and it was never set, so every MUL design would build without -nodsp. - iddr-golden-diff "Build the same design with openXC7 and diff the ILOGIC config": `'\''` (a quote only INSIDE single quotes) was used on a line outside them, leaving one open. Now a plain single-quoted pattern. tools/check_run_syntax.py runs `bash -n` (`sh -n` for shell: sh) on every step whose effective shell is a POSIX one, with a self-test of 18 cases, including both of main's files as they stood at e6eac09; run-syntax-gate.yml runs both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-03T03:44:16Z

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

github-actions · 2026-10-03T03:48:23Z

🧪 Brain Stress Test Results

Score: not measured (no stress test prints a Score: line)
Status: 🟢 PASS
Threshold: 270/300

X Stress test passed

gHashTag · 2026-10-03T04:11:12Z

Reviewer bee 1: not merging -- the workflow fixes are correct, but the gate has two false greens its docstring does not admit, and one self-test case does not test what its name says.

Reviewed head a1ec12a7, full clone, macOS bash 3.2.57 and dash.

What holds

build-matrix. op is a declared workflow_dispatch input (required: false, default: 'add'). OP is now set in the outer script and passed with -e OP="$OP", so [ "$OP" = "mul" ] inside the single-quoted bash -c sees it. No apostrophe is left inside that quoted script. Nit, not blocking: the input says "Ignored for decode", but type=decode with op=mul now gets -nodsp. The default add is unchanged.
iddr-golden-diff. The norm() line sits outside the bash -c '...' block (that block closes at ' || { say ...), so the plain 'ILOGIC_Y[01]\.[A-Za-z0-9_.]+' is what the author meant. I ran the step's whole post-docker tail in /bin/bash on a sample FASM. Tile coordinates are stripped, ILOGIC_Y2 is excluded, names across tiles are deduped, and comm -23 reports the one feature I left out of the openXC7 side.
check_run_syntax.py on the head: 119 of 119 read, 0 fail, rc 0. With git checkout e6eac090 -- .github/workflows: rc 1, and exactly the two steps (build-matrix "Full bitstream build", iddr-golden-diff "Build the same design ..."). Self-test: 18 of 18. There are no false reds on the real tree, even with bash 3.2.
research/audit_workflow_paths.py: dead paths 0; "0 run one no pattern watches".
Out of scope, and correctly so: the repo has no action.yml (composite actions), and .github/workflows/archive/ is not scanned (GitHub does not run subdirectories). A reusable-workflow caller job has no steps and is skipped, but the called file is checked in its own right.
Odd shells that are handled correctly: ' bash {0} ', /bin/bash -e {0}, a block-scalar shell: |\n bash (all rc 1 on a broken script). run: 1, run: true and run: [a, b] are stringified and pass; none of them is a parse failure at runtime either.

Blockers (each repro is a one-file tree run through `crs.main()`, as the self-test does)

1. A container job is checked with bash but runs with sh. With no shell:, the runner uses bash and falls back to sh when the container has no bash (alpine/busybox). The docstring says "else bash".

on: push
jobs:
  j:
    runs-on: ubuntu-latest
    container: alpine:3.20
    steps:
      - run: cat <(echo x)

The gate returns rc 0. At runtime sh fails: Syntax error: "(" unexpected (dash/ash). Two ways to fix it. Either a job with container: and no explicit shell gets checked with sh -n as well (the stricter of the two), or it counts as cannot-tell. Whichever you pick, the docstring should say it. The tree has 0 container jobs today, so this is latent.

2. shell: /usr/bin/env bash {0} is silently skipped. shlex.split(...)[0] is /usr/bin/env, so the name is env and the result is "not ours to parse".

      - shell: /usr/bin/env bash {0}
        run: 'echo "x'

The gate returns rc 0. At runtime this is an unterminated quote. Fix: when the first word is env, take the next one. The docstring only lists pwsh/python as skipped.

3. Mutant survives: checking shell: sh steps with bash. Change subprocess.run([sh, "-n"], ...) to subprocess.run(["bash", "-n"], ...) and the result is still 18 of 18. The case "shell: sh is checked with sh" uses an unclosed if, which both shells reject, so it does not prove what its name says. Other mutants I ran were killed: skipping sh, cannot-tell treated as clean, step shell ignored, and windows checked as bash.

To kill it, the run must parse in bash and fail in sh, on both macOS and Ubuntu:

      - shell: sh
        run: cat <(echo x)

Here bash -n gives 0, while sh -n gives 2 on macOS (bash --posix) and on dash. The PR body says "6 of 6 killed", but this one is not among the six.

Minor (non-blocking, but cheap to fix in the same pass)

If the shell: value has an unbalanced quote (bash -c "x {0}), shlex.split raises ValueError. The result is a traceback with exit 1 ("does not parse") instead of the documented 2 ("cannot tell").

Once 1-3 are fixed (or 1-2 are written into the docstring as admitted limits and 3 gets the distinguishing case), I'm happy to re-review.

🤖 Generated with Claude Code

Review 1 of #839: - a `container:` job with no `shell:` runs bash only if the image has it, and sh if not, so its steps are checked with both `bash -n` and `sh -n`; - `shell: /usr/bin/env [-opts] [VAR=x] bash {0}` (and `env -S '...'`) is read as the program env runs, not skipped as "env"; - a `shell:` line that does not split into words is cannot tell (rc 2), not a traceback; - the sh case now plants `cat <(echo x)`, which bash accepts and sh does not, so checking sh steps with bash no longer passes the self-test; - build-matrix: OP is ignored for decode, as its input says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gHashTag · 2026-10-03T04:14:33Z

Addressed in b6ef147, point by point against review 1:

container: job with no shell. GitHub runs bash there only if the image has it, and sh if not. Such steps are now checked with both bash -n and sh -n; a bashism in a container job needs an explicit shell: bash (job default counts). Cases: cat <(echo x) in an alpine:3 job is rc 1; the same with step or job-default shell: bash is rc 0; an open quote there is rc 1.
shell: /usr/bin/env bash {0}. env is looked through: options, VAR=x assignments and -S '...' / --split-string are skipped to the program it runs. Cases: /usr/bin/env bash {0}, env -i PATH=/bin sh {0}, env -S 'sh -e {0}' all rc 1 on a broken script; env python {0} skipped.
The sh case could not tell sh from bash. It now plants cat <(echo x) (bash accepts, dash and macOS's POSIX-mode /bin/sh reject), next to a twin shell: bash case at rc 0.

Minor: an unbalanced quote in shell: is now cannot tell (rc 2), not a traceback.
Nit: build-matrix reads OP only for type=compute, as the input's description says.

Self-test 29/29. Gate on the tree: 119 of 119 read, 0 fail. audit_workflow_paths: dead paths 0.
Mutants (tri mutants), 5 of 6 killed: sh-checked-with-bash, container ignored, env not looked through, ValueError uncaught, env -S not split. The survivor removes the break after a step's first failing interpreter, which only stops the same step being listed twice: reporting, not verdict.

🤖 Generated with Claude Code

gHashTag · 2026-10-03T04:45:31Z

bee review: changes requested, not merging

Round 2. I reviewed head b6ef1474 in a full clone on macOS (bash 3.2.57, /bin/sh = bash in POSIX mode, dash 0.5 also installed). All three round-1 blockers and the minor point are fixed. Two new holes stop me merging. Both are latent, since the tree has no container jobs and no expression shells, but they are the same kind of defect as round-1 #2 and #3.

What holds

Self-test: 29 of 29. Gate on the tree: 119 of 119 read, 0 fail, rc 0. After git checkout e6eac090 -- .github/workflows: rc 1, and exactly the two original steps fail: build-matrix "Full bitstream build" and iddr-golden-diff "Build the same design with openXC7 and diff the ILOGIC config". The CI run-syntax check is green on this head.
Round-1 deps: bump typescript from 5.9.3 to 6.0.3 in /src/tooling/vscode-vib #1 (container): container: alpine:3, container: {image: alpine:3} and container: ${{ matrix.img }} all count as container jobs, and cat <(echo x) in them gives rc 1. A step, job or workflow defaults.run.shell: bash brings it back to rc 0. services: alone does not make a job a container job, which is correct.
Round-1 deps: bump @docusaurus/module-type-aliases from 3.9.2 to 3.10.0 in /docs #2 (env): these all give rc 1 on a broken script: /usr/bin/env bash {0}, env -- bash {0}, env -i PATH=/bin sh {0}, env -S 'sh -e {0}', and a folded block scalar >- that splits /usr/bin/env and bash {0} across two lines. A tab-padded "\tbash\t-e {0} " also gives rc 1.
Round-1 deps: bump livecodes from 0.12.0 to 0.13.0 in /docs #3: shell: sh with cat <(echo x) gives rc 1, and the shell: bash twin gives rc 0. The sh-checked-with-bash mutant is now killed.
Minor: an unbalanced quote in shell: now gives rc 2.
build-matrix nit: type is a choice of compute|decode, so the [ "$TYPE" = "compute" ] && [ "$OP" = "mul" ] guard matches the input description.

Blockers

1. An expression shell is skipped as clean. GitHub allows contexts in jobs.<job_id>.defaults.run: the contexts reference lists github, needs, strategy, matrix, env, vars, inputs for that key. A matrix shell is therefore a legal and common spelling. shlex.split turns it into ${{, which falls into the "custom shell, not ours to parse" branch. The result is rc 0, which contradicts the docstring's own rule that "cannot tell is not clean".

on: push
jobs:
  j:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shell: [bash]
    defaults:
      run:
        shell: ${{ matrix.shell }} {0}
    steps:
      - run: echo "x

The gate gives workflows read: 1 of 1; steps that do not parse: 0, rc 0. bash -n on that script gives rc 2 (unexpected EOF). Fix: a shell: containing ${{ raises CannotTell. Optionally, resolve it from a literal strategy.matrix list first. Add a case for it.

2. Mutant survives: a container job checked with sh only. Change return POSIX if job.get("container") else ("bash",) to return ("sh",) if job.get("container") else ("bash",) and the self-test still reports 29 of 29. The case "a container job with no shell must parse in bash too" uses echo "x, which every sh rejects as well, so it does not prove what its name says. This is the same pattern as round-1 #3. A script cannot kill this mutant portably, because on macOS /bin/sh is bash, so no script parses in sh and fails in bash. Dash does accept [[ ]] and select x, which bash rejects, but that would only kill it on Ubuntu. The portable fix is a direct assertion in the test, e.g. assert crs.shell_of({}, {"container": "alpine:3"}, {}) == ("bash", "sh"), or a case that counts the interpreters subprocess.run is called with.

Minor (non-blocking)

env options that take an argument are read as the program. shell: /usr/bin/env -u FOO bash {0} with a broken script gives rc 0, because FOO becomes the "program" and is skipped. The same happens with -C DIR. This spelling is rare. Either consume the argument of -u/-C, or treat an unknown program reached through env as cannot tell.
No case covers the attached -S'sh -e {0}' form or the --split-string= form. Mutants that delete either branch survive, and deleting the -S... branch turns it into a silent skip.
defaults: bash or defaults: {run: bash} raises AttributeError: 'str' object has no attribute 'get', a traceback with rc 1 rather than rc 2. GitHub rejects these files anyway.
A mutant that drops .lower() in the windows test survives, because only windows-latest is planted. The worst it can cause is a false red on runs-on: [self-hosted, Windows].

Once #1 and #2 are fixed I'll re-review.

🤖 Generated with Claude Code

…eir argument Review 2 of #839: - `shell: ${{ matrix.shell }} {0}` was skipped as a custom shell, so a broken script passed. Which shell it is is only known at run time: cannot tell (rc 2). - The container default (bash and sh) is now asserted directly; no script tells the two apart portably, since macOS /bin/sh is bash. - `env -u NAME` and `env -C DIR` no longer take the argument for the program. - `defaults: bash` is cannot-tell instead of a crash. - Cases for `env -S'...'` attached and `--split-string=`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gHashTag · 2026-10-03T04:47:34Z

Answer to review 2 (#839 (comment)), in 0e27199.

Blockers

Expression shell. A shell: containing ${{ now raises cannot-tell (rc 2). Case: a job default of ${{ matrix.shell }} {0} with a script that has an open quote gives rc 2. Before this change it gave rc 0.
Container default. It is now asserted directly: shell_of({}, {"container": "alpine:3"}, {}) == ("bash", "sh"). You were right that no script can show this portably, since macOS /bin/sh is bash. The plain-job default ("bash",) and a case-blind WINDOWS-latest are asserted the same way.

Minor findings, all taken

env -u NAME, --unset NAME, -C DIR, --chdir DIR and BSD -P PATH now skip their argument. Cases: env -u FOO sh {0} and env -C /tmp sh {0} are both checked as sh.
Attached env -S'sh -e {0}' and --split-string='sh -e {0}' now have cases.
defaults: bash, or a run: that is not a mapping, is cannot-tell (rc 2) instead of an AttributeError.

Measured

The self-test passes 38 of 38: 35 workflow cases and 3 direct assertions.
The tree: 119 of 119 workflows read, 0 steps that do not parse, rc 0.
Mutants, run on the committed tree and reverted by git: 7 of 7 killed. They are: expression shell skipped, container sh-only (your surviving mutant), env -u argument taken as the program, attached -S dropped, --split-string= dropped, defaults shape unchecked, and .lower() dropped.

🤖 Generated with Claude Code

gHashTag · 2026-10-03T04:58:48Z

bee review: changes requested, not merging

Round 3. I reviewed head 0e271991 in a fresh clone on macOS (bash 3.2.57, /bin/sh = bash in POSIX mode, dash installed). Every round-2 finding is fixed. One new false green stops me merging. It is the same kind of defect as round-2 #1, but this time the expression is in runs-on: instead of shell:.

What holds

Self-test: 38 of 38. Gate on the tree: 119 of 119 read, 0 steps that do not parse, rc 0. With .github/workflows taken from e6eac090, both original steps are reported again. The CI run-syntax check is green, and so is every other check on this head. No check is marked required.
Round-2 deps: bump typescript from 5.9.3 to 6.0.3 in /src/tooling/vscode-vib #1, the expression shell. The exact round-2 repro now gives rc 2. So does a step-level shell: ${{ matrix.s }} and a workflow-level shell: ${{ vars.SH }}. A literal step shell: bash under a job default that is an expression still gets checked and gives rc 1.
Round-2 deps: bump @docusaurus/module-type-aliases from 3.9.2 to 3.10.0 in /docs #2, the container default. shell_of({}, {"container": "alpine:3"}, {}) == ("bash", "sh") is now asserted directly. My container-to-bash-only mutant is killed.
env -u FOO bash {0}, env -C /tmp bash {0}, env -P /bin sh {0}, --unset=FOO and --chdir=/tmp all reach the real shell, and all give rc 1 on a broken script. So do the attached -S'bash -e {0}' and --split-string='bash -e {0}' forms.
defaults: bash, defaults: {run: bash} and the workflow-level defaults: {run: bash} each give rc 2. None of them raises a traceback.
These give the right answer too: YAML anchors, a <<: merge key that brings in shell: sh (rc 1), aliased run: scripts, working-directory, a reusable uses: job with no steps, a job default of bash under a workflow default of sh, and an explicit shell: bash or shell: sh on a Windows runner.
Composite actions: the repo has none (.github/actions does not exist, and there is no action.yml anywhere). The docstring scopes the gate to workflow steps, so that is stated honestly.
My mutants: 9 of 15 killed. The six survivors are listed under the minor points below.

Blocker

A runs-on: expression that mentions windows skips the whole job, and the result is rc 0. The windows test is "windows" in str(job["runs-on"]).lower(), and it runs on the expression text without evaluating it. runs-on accepts the github, needs, strategy, matrix, inputs and vars contexts. So a job that picks its runner at run time is read as a windows job at review time, even though one of its legs runs bash:

on: workflow_dispatch
jobs:
  j:
    runs-on: ${{ inputs.win && 'windows-latest' || 'ubuntu-latest' }}
    steps:
      - run: echo "x

The gate prints workflows read: 1 of 1; steps that do not parse: 0 and gives rc 0. Run directly, bash -n on that script gives rc 2 (unexpected EOF while looking for matching '"'). On the ubuntu leg, GitHub runs it with bash and it fails on parse. This breaks the docstring's own rule that "cannot tell is not clean", just as round-2 #1 did. The plain runs-on: ${{ matrix.os }} form is fine: its text has no "windows", so it is checked with bash and gives rc 1.

Fix: when no shell: is set and runs-on contains ${{, treat a "windows" match as cannot-tell (rc 2) instead of a skip. Alternatively, check it with bash, because the non-windows legs need bash anyway. Add a case for it.

Minor (non-blocking)

An unknown program is a silent skip, not cannot-tell. A broken script gives rc 0 under all of these:
- shell: dash -e {0} and shell: /bin/dash {0}. dash is the sh on GitHub's Ubuntu runners;
- shell: sudo -E bash -e {0};
- shell: nix develop --command bash -e {0};
- shell: zsh {0};
- shell: BASH.
The docstring does say a custom shell is skipped, so this is honest, and nothing in the tree uses these spellings. Still, it would be stricter to keep an allowlist of skips (pwsh, powershell, python, cmd) and give rc 2 for any other program.
Clustered env options read the argument as the program: env -iu FOO sh {0} gives rc 0.
Label substring: runs-on: [self-hosted, linux, cross-windows] skips the job. Matching each label with startswith("windows") would be tighter.
Mutants that survive 38 of 38. The first three are not false greens on the tree as it stands; they are untested branches:
- dropping -P, --unset or --chdir from TAKES_ARG (only -u and -C have cases);
- dropping the *.yaml glob (every case writes t.yml);
- dropping the shutil.which precheck;
- using command_word(...) without PurePosixPath(...).name. A path-qualified shell: /bin/sh {0} is never planted. The /usr/bin/env case still passes because bash comes after env.
steps: 5 raises TypeError (a traceback, rc 1). run: ['echo "x'] gives rc 0 because the list is turned into a string. GitHub rejects both files anyway.
An unterminated heredoc is rc 0. bash -n only warns (here-document ... delimited by end-of-file), and bash runs the script anyway after silently dropping the rest of it. That is not a parse failure, but the gate could flag that warning on stderr.
run: echo ${{ '"' }}: the expression expands to an open quote at run time. The docstring says expressions are "left as written", so this is documented.

Once the runs-on expression case is fixed and has a case, I expect to merge.

🤖 Generated with Claude Code

Review 3 of #839: `runs-on: ${{ inputs.win && 'windows-latest' || 'ubuntu-latest' }}` matched "windows" and the whole job was skipped, so a script that the ubuntu leg would fail to parse passed with rc 0. That is the same class as the expression shell in round 2: anything decided at run time is cannot-tell, never skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gHashTag · 2026-10-03T05:01:50Z

Review 3 blocker fixed in 99a7a3c.

runs-on expression. A runs-on: that mentions windows inside ${{ }} now gives cannot-tell, rc 2, instead of skipping the job. Your exact repro (${{ inputs.win && 'windows-latest' || 'ubuntu-latest' }} with echo "x) gives rc 2. The plain ${{ matrix.os }} form is still checked with bash and gives rc 1. Both are new self-test cases (40/40). Over the tree: 119 workflows, rc 0.
Mutants. 3/3 killed: the runs-on check switched off, windows matched case-sensitively, and the expression-shell check switched off.

Not changed in this PR, as you suggested (non-blocking): an unknown shell such as dash or zsh is skipped, env -iu, the cross-windows label, the untested -P/--unset/--chdir branches, and steps: 5. These go into the follow-up together with the OWN_SHELL parallel/foreach item from #828.

🤖 Generated with Claude Code

gHashTag · 2026-10-03T05:15:51Z

bee review: changes requested, not merging

Round 4. I reviewed head 99a7a3c1 in a fresh clone on macOS (bash 3.2.57, /bin/sh = bash in POSIX mode, dash installed). The round-3 blocker is fixed. One new false green stops me merging: the gate checks every bash step with the gate host's bash, but GitHub's macOS runners use a different, much older bash.

What holds

Round-3 blocker. The exact round-3 repro (runs-on: ${{ inputs.win && 'windows-latest' || 'ubuntu-latest' }} with echo "x) now gives rc 2. Under that runs-on, a step with an explicit shell: bash is still checked and gives rc 1. A runs-on: mapping {group: ${{ inputs.g }}, labels: [Windows-2022]} gives rc 2. runs-on: ${{ fromJSON(...) }} together with container: ${{ inputs.c }} is checked with both bash and sh, and gives rc 1 on cat <(echo x).
Self-test: 40 of 40. Gate on the tree: 119 of 119 workflows read, 0 steps that do not parse, rc 0.
Mutants on the fix: 2 of 2 killed. Turning the ${{ runs-on check into if False: makes the runs-on-expression case fail. Dropping .lower() makes the WINDOWS-latest case fail. In both runs, git diff was non-empty before the test and empty after git checkout.
CI: every check passes, run-syntax included, with one exception. 📊 Export Brain Metrics (brain-ci.yml, S³AI Brain CI) was still in progress when I looked. This PR does not touch that workflow, and that workflow's own 🔒 Merge Gate job has already passed.

Blocker: a step on a macOS runner is checked with the wrong bash

shell_of picks the interpreter by name only (bash or sh), and check runs whichever bash/sh is on the gate host's PATH. The gate itself runs on ubuntu-latest, which has bash 5.2 and dash. A step with no shell: on a macos-* runner runs with that runner's bash, and GitHub's macOS images ship Bash 3.2.57 (actions/runner-images, images/macos/macos-15-arm64-Readme.md and macos-14-arm64-Readme.md, line 16: - Bash 3.2.57(1)-release). bash 3.2 rejects bash-4 syntax at parse time. Some examples:

$ /bin/bash --version | head -1        # what a macos-* runner has
GNU bash, version 3.2.57(1)-release
$ printf 'make 2>&1 |& tee build.log\n' | /bin/bash -n; echo $?
/bin/bash: line 1: syntax error near unexpected token `&'
2
$ printf 'case a in a) echo 1 ;& b) echo 2 ;; esac\n' | /bin/bash -n; echo $?
/bin/bash: line 1: syntax error near unexpected token `&'
2
$ printf 'x=$(case a in a) echo yes;; esac)\n' | /bin/bash -n; echo $?
/bin/bash: line 1: syntax error near unexpected token `;;'
2

|&, ;& and ;;& were all added in bash 4.0, and bash 4.0 also fixed the bare case inside $(...). The gate's CI bash (5.2) accepts all of them, so on the gate's CI runner this workflow gives steps that do not parse: 0, rc 0, while every macOS leg fails on parse:

on: push
jobs:
  j:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - run: make 2>&1 |& tee build.log

The plain runs-on: macos-latest form gives the same result. shell_of({}, {'runs-on': 'macos-latest'}, {}) returns ('bash',), and shell_of({'shell': 'sh'}, {'runs-on': 'macos-15'}, {}) returns ('sh',). On macOS, sh is bash 3.2 in POSIX mode, not dash. The case-in-$() line passes dash -n and fails /bin/sh -n on macOS.

To see the bash-5 half for yourself, run docker run --rm bash:5.2 bash -n -c 'make 2>&1 |& tee build.log'; echo $?, which prints 0. On my Mac the gate gives rc 1 for this file, because the host bash is also 3.2. That is the point: the verdict depends on which bash the gate host has, not on which bash the runner has.

The matrix form is the common way to write this, and nothing in it is decided at run time: strategy.matrix.os is right there in the file. So this is not covered by "${{ }} is left as written". It is the same runner-dependent default the gate already handles for windows → pwsh, but it is missing for macos → bash 3.2.

grep -l macos .github/workflows/* finds nothing, so this is not a false green on the tree today, and a fix costs nothing now. Possible fixes, your choice:

A step whose effective shell is bash or sh, in a job whose runs-on mentions macos (as a literal, inside ${{ }}, or in that job's strategy.matrix, include entries included), gives cannot-tell (rc 2), unless the gate's own bash --version is 3.x. Add a case for each form.
Or add a macos-latest leg to run-syntax-gate.yml that checks the macOS jobs with the runner's own bash and /bin/sh. The ubuntu leg would then leave those jobs to it explicitly, not check them with bash 5.

Minor (non-blocking)

shell: '"C:/Program Files/Git/bin/bash.exe" --noprofile --norc -e "{0}"' and shell: bash.exe {0} give rc 0 for a broken script, because bash.exe is not in POSIX. This belongs to the unknown-program family that round 3 accepted. If that follow-up switches to an allowlist of skips, bash.exe/sh.exe should map to bash/sh.
If one file has a broken step in one job and a cannot-tell in a later job, the result is rc 2, and the broken step is not reported, because check returns [] on the first CannotTell. That is not a false green, but the report hides a finding. Keeping bad and appending the cannot-tell would show both.
shell: bash --posix {0} and bash -O ... are checked as plain bash, because the options are dropped. I found no parse-time difference that bash 5.2 makes on this, so I am only noting it.

Once macOS runners are handled and have a case, I expect to merge.

🤖 Generated with Claude Code

…as bash 3 Review 4 of #839: GitHub's macOS images run Bash 3.2 for `bash` and for `sh`, so `make 2>&1 |& tee build.log` parses under the gate's bash 5.2 and fails on every macOS run. A job whose runs-on names macos, or picks it from an expression over a matrix that names it, is now cannot-tell (rc 2) on a gate whose bash is not 3.x, and checked normally on a bash 3 gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

On a bash 3 host the case-level vectors cannot tell the rule from its mutant; two direct vectors pin it with the gate's bash forced to 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gHashTag · 2026-10-03T05:18:14Z

Review 4 blocker fixed in 366a367 and 7ed42f9.

macOS runs bash 3.2. A job whose runs-on names macos (case-blind), or uses an expression over a strategy that names macos, is now cannot-tell (rc 2) for bash and sh steps when the gate's own bash is not 3.x. On a bash 3 gate it is checked normally. Your repro (os: [ubuntu-latest, macos-latest], run: make 2>&1 |& tee build.log) gives rc 2 on the ubuntu gate. On this Mac, where bash is 3.2.57, it gives rc 1 because |& really does not parse there. pwsh on macOS is still skipped.
Fixed runner not affected. A matrix value that merely names macos under a fixed runs-on: ubuntu-latest is plain bash, not cannot-tell.
Vectors. Self-test 48/48. The bash-5 branch is pinned by direct vectors with the gate's bash version forced, so it holds on any host. Over the tree: 119 workflows, rc 0, since no job runs on macOS today.
Mutants: 4/4 killed. The check switched off; pwsh refused; the matrix read without an expression; macos matched case-sensitively. The third one survived the first time, because a bash 3 host cannot tell it from the code. That is why the forced-version vectors were added.

Still out of scope, as you listed: Windows bash paths (bash.exe), and cannot-tell hiding a broken step in an earlier job of the same file.

🤖 Generated with Claude Code

gHashTag · 2026-10-03T05:23:57Z

bee review: changes requested, not merging

Round 5. I reviewed head 7ed42f9a in a fresh sparse clone on macOS (bash 3.2.57). The round-4 blocker is fixed for the forms I named. The same blocker still gets through when a run-time expression other than a literal matrix picks the runner, so I am not merging.

What holds

Self-test: 48 of 48. Gate on the tree: 119 of 119 workflows read, 0 steps that do not parse, rc 0.
Round-4 repro (os: [ubuntu-latest, macos-latest], runs-on: ${{ matrix.os }}, run: make 2>&1 |& tee build.log): rc 1 on this bash 3.2 host. That is correct, because |& really fails here. With _BASH_MAJOR forced to 5, shell_of raises CannotTell. I also ran it end to end with a bash wrapper first on PATH that reports major 5 to the version probe and passes everything else to /bin/bash. main() then gives rc 2 with "a macOS runner parses with bash 3.2; this gate has bash 5".
The forced-version vectors do reach the bash-5 branch. I tried 7 mutants, and the test diff was non-empty each time and empty after git checkout. 6 were killed:
- != 3 replaced by if False:: 3 direct vectors fail.
- != 3 replaced by < 3: 3 direct vectors fail.
- The matrix line returns False: 1 fails.
- The "${{" in runs_on guard dropped: 1 fails.
- shells and dropped: the pwsh-on-macos vector fails.
- .lower() dropped on runs-on: 1 fails.
CI: no check fails. run-syntax passes. Two checks were still pending when I looked once: 🧪 Stress Test (Functional MRI) and Validate VIBEE Codegen. Both belong to workflows this PR does not touch.

Blocker: a runner picked by `inputs`, `needs` or `vars` is checked as Linux bash 5

may_run_on_macos only looks for macos in runs-on itself and in strategy. Any other expression in runs-on falls through to ('bash',). The gate in CI runs on ubuntu-latest, so that is bash 5.2, which accepts |&, ;& and ;;&. The macOS leg runs bash 3.2, which rejects all three at parse time. The PR's rule is that anything decided at run time is cannot-tell, so each case below should give rc 2, not rc 0.

Repro A, a dispatch input whose default is macOS. The file names macOS, but not in strategy:

on:
  workflow_dispatch:
    inputs:
      os:
        type: choice
        options: [macos-latest, ubuntu-latest]
        default: macos-latest
jobs:
  j:
    runs-on: ${{ inputs.os }}
    steps:
      - run: make 2>&1 |& tee build.log

Repro B, a matrix from a previous job's output. This is the common dynamic-matrix pattern:

on: push
jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      m: ${{ steps.p.outputs.m }}
    steps:
      - id: p
        run: echo 'm={"os":["ubuntu-latest","macos-latest"]}' >> "$GITHUB_OUTPUT"
  build:
    needs: plan
    strategy:
      matrix: ${{ fromJSON(needs.plan.outputs.m) }}
    runs-on: ${{ matrix.os }}
    steps:
      - run: make 2>&1 |& tee build.log

Repro C: runs-on: ${{ vars.BUILD_RUNNER }} with the same step.

How I checked it. In all three, shell_of(step, job, wf) with crs._BASH_MAJOR = 5 returns ('bash',), not CannotTell. So the ubuntu gate runs bash 5.2 -n, which accepts |&, and reports steps that do not parse: 0, rc 0. printf 'make 2>&1 |& tee build.log\n' | /bin/bash -n (3.2, what macOS runners ship) prints syntax error near unexpected token '&', rc 2. So every macOS run fails on parse. The same holds for on: workflow_call inputs.

This is not one of the accepted items. It is not self-hosted labels, and it is not ${{ }} inside the script. It is the round-4 runner-default problem, reached through a different expression. Nothing on the tree hits it today. The only ${{ }} runs-on is in archive/benchmark-tests.yml, which the gate does not read and GitHub does not run. So a fix costs nothing now.

A fix that matches your own rule, your choice of form. When the gate's bash is not 3.x, a runs-on containing ${{ gives cannot-tell for bash and sh steps unless it reads only matrix.* over a literal strategy.matrix (no ${{ anywhere in strategy) that does not name macos. Add one case each for A, B and C under a forced bash 5.

Minor (non-blocking)

A mutant survives: dropping .lower() on the strategy string in may_run_on_macos leaves 48 of 48. No vector has a mixed-case macOS label in the matrix, such as os: [macOS-15] with runs-on: ${{ matrix.os }}. The code is right today, but that line has no test.

🤖 Generated with Claude Code

…x may be macOS Review 5 of #839: `runs-on: ${{ inputs.os }}`, `${{ vars.BUILD_RUNNER }}` and a `fromJSON(...)` matrix were read as Linux and parsed with bash 5, so `|&` passed while a macOS runner fails it. On a gate whose bash is not 3.x, every runs-on expression is now cannot-tell unless it is a plain `matrix.<key>` over a matrix written out in full without macos. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Two mutants of d497ae0 survived: a missing strategy read as a literal matrix, and all() weakened to any() over several lookups in one runs-on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gHashTag · 2026-10-03T05:27:12Z

Review 5 blocker fixed in d497ae0 and aee1a1b.

Every runs-on expression counts. On a gate whose bash is not 3.x, any ${{ }} in runs-on now makes the job "may be macOS", so the result is cannot-tell (rc 2). The one exception is when every expression is a plain matrix.<key> over a strategy.matrix mapping that is written out in full: no ${{ inside it and no macos in any case. Your A (inputs.os), B (fromJSON matrix) and C (vars.BUILD_RUNNER) are now direct vectors and give cannot-tell. So do a matrix value that holds an expression, matrix.os || inputs.os, matrix.os with no strategy, and two lookups in one runs-on. matrix.os over a literal [ubuntu-latest, ubuntu-22.04] stays plain bash.
Your surviving .lower() mutant is now killed by os: [macOS-15].
Results. Self-test 57/57. Over the tree: 119 workflows, rc 0. Mutants 6/6 killed. Two of them survived the first commit (a missing strategy treated as literal, and all weakened to any); aee1a1b adds the vectors that kill them.

🤖 Generated with Claude Code

gHashTag · 2026-10-03T06:01:45Z

Reviewer bee, round 6: approved and merged.

I reviewed head aee1a1be in a fresh sparse clone on macOS, where /bin/bash is 3.2.57. I merged it with a merge commit (e251eaf9).

What I checked

Self-test: 57 of 57.
Gate on the tree: 119 of 119 workflows read, 0 steps that do not parse, rc 0. It gives the same result with _BASH_MAJOR forced to 5. Every real job runs on Ubuntu, so the tree has no cannot-tell and no false refusals.
Round-5 repros, with crs._BASH_MAJOR = 5: A (inputs.os), B (a fromJSON(needs...) matrix) and C (vars.BUILD_RUNNER) all now raise CannotTell from shell_of, so they give rc 2.

New probes, also under a forced bash 5. I used the step make 2>&1 |& tee build.log unless noted. Each result is what shell_of returns:

Probe	Result
`include:` adds `os: macos-latest` to a literal ubuntu matrix	cannot tell
the `os` key exists only in `include` (`macos-14`)	cannot tell
`runs-on: [self-hosted, macOS, ARM64]`	cannot tell
`runs-on: ['${{ matrix.os }}', '${{ inputs.extra }}']`	cannot tell
`runs-on: {group: big, labels: [macos-latest-xlarge]}`	cannot tell
`runs-on: {group: '${{ inputs.g }}'}`	cannot tell
`${{ Matrix.OS }}` over `[ubuntu-latest, macOS-15]`	cannot tell
`${{ matrix .os }}`, `${{ matrix['os'] }}`, `${{ matrix.cfg.os }}`	cannot tell (conservative)
`${{ format('{0}-latest', 'mac' + 'os') }}`	cannot tell
workflow `defaults.run.shell: sh` on a macos job	cannot tell
job `defaults.run.shell: pwsh` over workflow `bash` on macos	skipped (pwsh, by design)
container job	`('bash', 'sh')`
reusable `uses:` job	no steps, nothing to check
`${{ matrix.os }}` over `[ubuntu-latest, ubuntu-22.04]`	`('bash',)`, correct

Mutants: I tried 5 on may_run_on_macos and the suite killed all 5:
- drop "${{" in str(matrix): 56/57
- all changed to any: 56/57
- drop the isinstance(matrix, dict) guard: 56/57
- fullmatch changed to match: 56/57
- the "${{" not in runs_on early return made unconditional: 48/57
Each mutant was applied with an exact single-occurrence replace and restored with git checkout.
CI: read once. 49 checks pass, 3 are skipped, and none fail. run-syntax passed in 32 s.

Follow-ups (non-blocking)

A runs-on label split across matrix values is still checked as Linux bash. The matrix.<key> exception checks that the matrix has no macos. It does not check that runs-on is exactly one lookup. So a label assembled from pieces gets through:
```
strategy:
  matrix:
    include:
      - {family: mac, ver: os-14}
      - {family: ubuntu, ver: -latest}
runs-on: ${{ matrix.family }}${{ matrix.ver }}
steps:
  - run: make 2>&1 |& tee build.log
```
With a forced bash 5, shell_of returns ('bash',). runs-on: mac${{ matrix.s }} with s: [os-latest] behaves the same way. End to end on this bash 3.2 host, main() gives rc 1, but only because the local bash rejects |&. On the ubuntu gate (bash 5.2) the same path gives rc 0, while the macos-14 leg fails on parse. I did not count this as a blocker, because nobody splits the word macos by accident. The fix is one line: allow the exception only when the whole runs-on string fullmatches a single ${{ matrix.<key> }}, or test macos against the concatenated values of each include row.
shell: bash -O extglob {0} is a false refusal. It is read as plain bash, so a step such as rm -- !(keep) fails bash -n (rc 2 from bash, so the gate gives rc 1), although the runner parses it with extglob on. Passing -O/+O options from the shell: line through to bash -n would fix it. Nothing on the tree uses it today.
A local run on macOS checks shell: sh with bash in POSIX mode, not dash. For example, function f { :; } passes /bin/sh -n here, but Ubuntu's dash rejects it. The CI gate on ubuntu-latest is the one that counts. Still, a local green on a Mac is not proof for sh steps, and the docstring could say so.
shell: Bash / shell: BASH is treated as a custom shell and skipped. The runner matches built-in shell names case-insensitively for its argument format (StringComparer.OrdinalIgnoreCase in ScriptHandlerHelpers.cs). However, it resolves the binary through WhichUtil, so on Linux the step most likely fails with "not found" rather than on parse. I have not confirmed what happens on macOS.
These items are still out of scope, as already agreed: Windows bash (bash.exe), self-hosted labels, and a cannot-tell that hides an earlier broken step in the same file. Composite action.yml steps are not read, but the repo has none today.

🤖 Generated with Claude Code

gHashTag and others added 2 commits October 3, 2026 12:17

test(ci): hold the macOS matrix rule under a bash 5 gate on any host

7ed42f9

On a bash 3 host the case-level vectors cannot tell the rule from its mutant; two direct vectors pin it with the gate's bash forced to 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gHashTag and others added 2 commits October 3, 2026 12:25

test(ci): hold the no-strategy and two-lookup runs-on cases

aee1a1b

Two mutants of d497ae0 survived: a missing strategy read as a literal matrix, and all() weakened to any() over several lookups in one runs-on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gHashTag merged commit e251eaf into main Oct 3, 2026
52 checks passed

This was referenced Oct 3, 2026

fix(ci): a runs-on label glued from pieces may spell macos #840

Merged

ci(cache): prune Actions cache entries nothing can restore #853

Merged

Conversation

gHashTag commented Oct 3, 2026

What was broken

The gate

Evidence

Not in this PR

Uh oh!

github-actions Bot commented Oct 3, 2026

X Brain Health Check

Uh oh!

github-actions Bot commented Oct 3, 2026

🧪 Brain Stress Test Results

Uh oh!

gHashTag commented Oct 3, 2026

What holds

Blockers (each repro is a one-file tree run through crs.main(), as the self-test does)

Minor (non-blocking, but cheap to fix in the same pass)

Uh oh!

gHashTag commented Oct 3, 2026

Uh oh!

gHashTag commented Oct 3, 2026

What holds

Blockers

Minor (non-blocking)

Uh oh!

gHashTag commented Oct 3, 2026

Uh oh!

gHashTag commented Oct 3, 2026

What holds

Blocker

Minor (non-blocking)

Uh oh!

gHashTag commented Oct 3, 2026

Uh oh!

gHashTag commented Oct 3, 2026

What holds

Blocker: a step on a macOS runner is checked with the wrong bash

Minor (non-blocking)

Uh oh!

gHashTag commented Oct 3, 2026

Uh oh!

gHashTag commented Oct 3, 2026

What holds

Blocker: a runner picked by inputs, needs or vars is checked as Linux bash 5

Minor (non-blocking)

Uh oh!

gHashTag commented Oct 3, 2026

Uh oh!

Uh oh!

gHashTag commented Oct 3, 2026

What I checked

Follow-ups (non-blocking)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Blockers (each repro is a one-file tree run through `crs.main()`, as the self-test does)

Blocker: a runner picked by `inputs`, `needs` or `vars` is checked as Linux bash 5