Skip to content

Release evidence gate: reuse exact-commit main CI evidence at release, fail closed (#37) - #52

Merged
kikashy merged 3 commits into
mainfrom
release-evidence-37
Oct 26, 2026
Merged

kikashy merged 3 commits into
mainfrom
release-evidence-37

Conversation

@kikashy

@kikashy kikashy commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Issue #37: a shared, fail-closed, tested gate that lets a release take the exact-commit CI evidence main already produced instead of running CI again; adopted by the kit's own release. Its "Required evidence gate" list is the acceptance contract.

#38 shipped a first gate as lib/ci_evidence.py (used by the v0.8.0 and v0.9.0 releases) and closed the issue. This PR builds on it: the gate moves to lib/release_evidence.py, is vendored as scripts/dev/release_evidence.py, and closes the gaps against the contract (below). One review round; release admission is a trust boundary.

Public surface changes

  • Command: lib/ci_evidence.py becomes lib/release_evidence.py, vendored as scripts/dev/release_evidence.py (line 2 names the kit release; tests/run.sh and the release workflow hold it to the library). Both workflows now run the vendored file. verify gains --output and always writes a decision record.
  • Configuration schema 1 becomes schema 2 (.github/ci-evidence.json), with the adopters' template templates/ci-evidence.json. The evidence artifact payload is schema 2.
  • No other repository vendored the schema 1 helper (checked: agent, gateway, runner, runtime, spec, exchange).

What changed against the contract

Contract item Now
Inventory; OS/toolchain, pin, integration, tag-context differences profiles per job: release-runner (the release job resolves image and tools itself and requires equality) or pins (exact tool output, platform, optional ImageOS; for macOS/Windows legs); pin files in inputs; fresh_at_release names checks the release runs itself (tag-sensitive) and carries them into the record. The kit's inventory is in docs/ci-evidence.md
Exact commit, repository, trusted event/ref, workflow identity, job/leg coverage run, head repository (no fork), head_sha, workflow id + path + name:, branch = the repository's default branch, push or configured dispatch, no PR association; jobs bound to the run, attempt, commit and workflow name, on the configured labels on a GitHub-hosted runner; artifacts bound to the run and repository ids; capture refuses unless GITHUB_WORKFLOW_REF/GITHUB_WORKFLOW_SHA are the configured file at the commit. No commit status or check run is ever read
Deterministic selection, never past a newer failed/cancelled/running attempt every applicable run must be completed (run-unfinished), then the run whose latest attempt started last is selected whatever its result (#38 selected by creation time, so a failed re-run of an older run could be passed over); re-read and re-selected after collection and after all workflows
Jobs, coverage, explicit permitted skips; record in release logs; pagination and reruns skipped_jobs must be skipped, required jobs may not be (job-skipped), check SKIP only for optional_skips; jobs of the selected attempt only; every page read with stable totals
Fresh checks for mutable state unchanged in release.yml: tag grammar/annotation/commit/main, notes, environment reviewer, version and copies, package run, checksums, attestation, draft and published identity
Tests and safeguard removal tests/release_evidence_test.py: 75 tests (below); 26 mutants (below)
Measured before/after; docs below; docs/ci-evidence.md, docs/releasing.md, docs/adopting.md, docs/policy.md, README, AGENTS.md

The decision record (stdout and --output; the release keeps it as the release-evidence-<attempt> artifact): decision, refusal code and detail, commit, tag object, input object ids, per workflow the id/path/name, run counts, newest candidates, selected run/attempt/event/result/start/age, referenced workflows, and per job its id, result, labels, times, artifact id/name/digest, runtime and check results.

Refusal classes tested (handwritten API fixtures)

Accepted exact-commit evidence (push, and dispatch when configured); wrong SHA (run, evidence, job, artifact); tree-equivalent other commit; wrong repository (run, fork head, API answer, artifact, evidence) and a non-default trusted branch; wrong event (PR head, synthetic merge from a fork's main, merge queue, another branch, a main run tied to a PR, evidence of another event/ref); wrong workflow (path, name, state, a run of another workflow with the same job names, another workflow ran and this one did not, evidence of another workflow); a mutable status name (green statuses and check runs named like the jobs, and the gate never reads commits/); missing job or matrix leg, extra or repeated job, missing/repeated/other-attempt artifact, missing check; unexpected skip (required leg, PR-only job ran, required check SKIP/FAIL/NOT RUN, optional skip that failed); newer failed/cancelled/timed-out/action-required run, newer or older running/queued run, newer failed attempt of the selected run, older run re-run later and failed, attempt race, partial rerun, new attempt / newer run / changed answer during collection, newer run after all workflows; changed inputs (workflow, manifest and pin file identities, missing input, pinned toolchain on both profiles, release-runner image, macOS image and architecture, tool set, unexpected members); unavailable evidence (API error at four points, truncated listing, 1,000 cap, expired/oversized/out-of-interval/digest/other-run artifact, archive with another entry, malformed answers, stale and future runs, other runners); tag movement (other commit, a tree, lightweight tag, tag moved during collection, grammar). Plus pagination, every configuration member and invalid value, command and runtime bounds, the public command in a fixture repository with a stand-in gh (reuse and refusal records, empty/missing/oversized answers, context refusals, capture refusals, capture-to-verify round trip), and the kit's own workflows against its configuration.

Mutants (one safeguard removed per scratch copy; the intended tests fail at their assertion; the unmutated library passes the same tests)

# Safeguard removed Failing test, assertion
M01 exact-commit check on the run wrong SHA + tree-equivalent listing: 'job-identity' != 'run-sha'
M02 repository check on the run 'artifact-run' != 'run-repository'
M03 trusted branch/event filter and trigger check PR head, synthetic merge, other branch: 'run-trigger' != 'run-missing'
M04 workflow id/path check on the run 'reuse' != 'run-workflow'
M05 resolved workflow name check 'reuse' != 'workflow-identity'
M06 refusal when no run exists (status name) 'unavailable-or-malformed' != 'run-missing' (still refused by the catch-all)
M07 job coverage comparison 'unavailable-or-malformed' != 'job-coverage' (extra/repeated job: reuse)
M08 one artifact per leg 'unavailable-or-malformed' != 'artifact-coverage'
M09 skipped required leg 'reuse' != 'job-skipped'
M10 PR-only job must be skipped 'reuse' != 'job-not-skipped'
M11 only optional checks may skip 'reuse' != 'check-result'
M12 latest-green search instead of newest 'reuse' != 'run-result'
M13 unfinished runs refused 'run-result' != 'run-unfinished'
M14 selection by creation, not latest attempt 'reuse' != 'run-result'
M15 attempt recheck after collection 'reuse' != 'attempt-changed'
M16 input identity comparison 'evidence-identity' != 'input-identity' (the full-equality backstop still refuses)
M17 toolchain pins 'reuse' != 'runtime-identity'
M18 release-runner image/tool equality 'reuse' != 'runtime-identity'
M19 truncated listing 'reuse' != 'pagination-partial'
M20 failed command read as an answer 'accepted' != 'api-unavailable'
M21 expired artifact 'reuse' != 'artifact-expired'
M22 a refusal recorded as reuse API error, malformed answers: refusal is not None
M23 age limit 'reuse' != 'run-age'
M24 tag held to the commit 'reuse' != 'tag-commit'
M25 tag recheck after collection 'reuse' != 'tag-changed'
M26 annotated tag 'reuse' != 'tag-annotated'

The kit's release: before and after

Checks executed at the release commit:

Release calling ci.yml (v0.6.0, v0.7.0) Release with the gate (v0.8.0, v0.9.0 via #38; this PR)
lint checks (diff-check, local-paths, agent-docs, bash-n, json, shellcheck; duplication from v0.8.0) run again in CI at the tag / lint not run again: taken from main's lint evidence
tests (tests/run.sh) run again in CI at the tag / tests not run again: taken from main's tests evidence
toolchain not compared image, OS release, bash/git/python3/go/shellcheck/rsync compared with the release runner
tag, notes, environment reviewer, version and copies, package run, checksums, attestation, draft and published identity run run, unchanged; plus the vendored release_evidence.py copy check

Observed (API job start to completion): v0.6.0 run 39844235747 lint 9 s, tests 58 s (admit end to package start 63 s); v0.7.0 run 39962449641 lint 10 s, tests 57 s (62 s); v0.8.0 run 40274516108 verify 14 s (17 s); v0.9.0 run 40351296561 verify 15 s, the gate step itself about 9 s (20 s). The last two releases already ran the #38 gate. This gate reads the same API objects plus one repository read and resolves the same tools, so its cost is expected to be of that order; it is first measured at the next release. No other saving is claimed; a refusal adds the fresh route's full CI run.

The fresh route

After a refusal the operator chooses, once: re-run all jobs of the main CI run at the commit (works after main moved on; the route for run-age and a changed hosted image), or dispatch ci.yml on main while main names the commit; then re-run the release run's failed jobs. A second refusal: fix on main and release a new version. Nothing dispatches or re-runs automatically. ci.yml keeps its workflow_call trigger, but the release does not call it: since #38 it has not, and a release-side fallback would need a second trigger on the release workflow, while the re-run and the dispatch keep fresh evidence under the trusted-branch identity the gate verifies.

Not verifiable before merge

  • capture runs only on a push or dispatch on main, so this PR's CI does not exercise it live; the first main push after merge does (GITHUB_WORKFLOW_REF/GITHUB_WORKFLOW_SHA included). A capture refusal there fails main's lint/tests visibly.
  • verify is first live at the next release. A read-only probe ran the new gate's API walk (repository, tag, workflow resolution, selection, run identity, attempt jobs with hosted group and labels, artifact metadata, interval, digest and archive) over the real v0.9.0 main run 40345389159 and passed; only the payload (schema 1) differs from what this gate expects.

Checks

Candidate: head f8fa3d6 (no uncommitted diff). jpdev check --changed against origin/main (merge base 19e7281): unmapped paths, whole list.

== check summary (dev.json) ==
PASS     diff-check               0s
PASS     local-paths              0s
PASS     agent-docs               0s
PASS     bash-n                   0s
PASS     duplication              0s
    duplication: no duplicate block (at least 100 tokens over at least 10 lines, 21 files of python, shell)
PASS     json                     0s
    parsed 12 JSON documents
PASS     shellcheck               5s
PASS     tests                    99s
    441 passed, 0 failed
== 8 passed, 0 failed, 0 skipped, 0 not run ==

tests/release_evidence_test.py also passes with RELEASE_EVIDENCE_HELPER set to the vendored copy. CI on this head and the review remain.

Fix round

For the review record at f8fa3d6 (0 HIGH, 2 MEDIUM, 4 LOW). One commit: 6b306d3.

Finding Fix Test Safeguard removed in scratch
MEDIUM 1: newer failed or running schedule/workflow_run/repository_dispatch runs did not block reuse every run of the workflow at the commit, whatever its event or branch, takes part in the scan: unfinished refuses run-unfinished; newer than the selected run and not successful refuses run-newer-failed. Only a trusted-trigger run is evidence; a newer green run of another trigger neither blocks nor is selected the six fixtures (3 triggers × failed/running) each refuse; a newer failed pull request run and another branch's run refuse; older failures and newer green runs of other triggers do not block M27, scan narrowed back to trusted runs: 'reuse' != 'run-newer-failed'
MEDIUM 2: helper-source untested the helper's bytes changed under git update-index --skip-worktree (the checkout looks clean): helper-source M28, check removed: exit 0 (reuse) where 1 was expected
MEDIUM 2: job head_branch untested a job on another branch: job-identity M29, check removed: 'reuse' != 'job-identity'
LOW 3: unsupported compression, deep nesting crash NotImplementedError, RecursionError, EOFError, zlib.error are unavailable-or-malformed refusals with a decision record compression method 99 set by hand in both zip headers; JSON nested 100,000 deep
LOW 4: four codes missing from the doc job-result, run-identity, timestamp, git-unavailable added, with run-newer-failed; every code the code raises is now in the table
LOW 5: fresh-route wording "This also works after main moved on"
LOW 6: record upload on refusal untested the release's Keep the decision record step has if: ${{ !cancelled() }}, the verify --output path and if-no-files-found: error

The selection rule in docs/ci-evidence.md and the template's workflow comment now state the wider scan. Mutants: M01–M29 all caught (M12 and M14 rewritten for the new selection code; M12 now also meets the scan, 'run-newer-failed' != 'run-result'); the unmutated library passes the same tests. tests/release_evidence_test.py: 80 tests, OK on the library and on the vendored copy.

Candidate 6b306d3, jpdev check --changed (whole list):

PASS     diff-check               0s
PASS     local-paths              0s
PASS     agent-docs               0s
PASS     bash-n                   0s
PASS     duplication              0s
PASS     json                     0s
PASS     shellcheck               7s
PASS     tests                    110s
    441 passed, 0 failed
== 8 passed, 0 failed, 0 skipped, 0 not run ==

Second commit: bb2eb1b

  • CI failure on 6b306d3 (run 40382530467, tests): the 100,000-deep JSON fixture raised RecursionError on the local Python but parsed on the runner's, where the non-object check then refused as artifact-entries. Nesting is now bounded at 16 levels by a scan before parsing, and malformed or unreadable evidence is unavailable-or-malformed on every interpreter: before the entries check, an archive that does not open, a compression other than stored or deflated, or members declared over 256 KiB; after it, content over 256 KiB when read, too-deep nesting, JSON that does not parse, or a document that is not an object. There is one test per kind, each checking its code and detail. Passes under python3 -X dev and with the recursion limit set to 10000 and to 200.
  • MEDIUM A (reviewer's confirmation): the newer-attempt scan now takes only this repository's runs on the trusted branch, leaving out pull_request* events and merge_group. A fork pull request at the release commit (newer and failed, or awaiting approval), a same-commit run on another branch, a pull_request_target or merge-queue run, and another head repository's run each no longer block. The six schedule/workflow_run/repository_dispatch cases still refuse. M30 (scan every run again): 'run-newer-failed' != 'reuse'.
  • MEDIUM B: a lower-id run re-run later that failed refuses run-newer-failed; a higher-id failed run that started before the selected attempt does not block. M31 (compare run ids): both tests fail.
  • Mutants M01–M34 all caught, including M32 (no nesting bound), M33 (no compression check) and M34 (no declared-size bound); the unmutated library passes the same tests. 85 tests pass on the library and on the vendored copy. jpdev check --changed on bb2eb1b: whole list, 8 passed, 0 failed, 0 skipped, 0 not run (tests: 441 passed, 0 failed).

🤖 Generated with Claude Code

…, fail closed (#37)

The gate #38 introduced as lib/ci_evidence.py moves to lib/release_evidence.py,
vendored as scripts/dev/release_evidence.py (the library with the kit release on
line 2, held to it by tests/run.sh and the release workflow), and closes the
gaps against the issue's evidence contract:

- configuration schema 2 (.github/ci-evidence.json, template
  templates/ci-evidence.json): strict members; the workflow by path and name;
  the trusted branch, which must be the default branch; per-job runner labels;
  matrix legs as one entry each; jobs that must be skipped; toolchain profiles
  compared with the release runner or held to exact pins (legs on another OS);
  pin files among the inputs; checks the release runs itself recorded as
  fresh_at_release
- selection: every applicable run must be complete, and the run whose latest
  attempt started last is selected whatever its result, so a newer failed,
  cancelled or running attempt, a re-run of an older run included, is never
  searched past
- verification: the repository by name and id, hosted runners, the jobs of
  the selected attempt only, artifacts bound to the run and repository, input
  identities as git object ids (manifest, configuration, the gate, workflow
  files, pin files), capture bound to the running workflow file
- a decision record, reuse or refusal with its code, written to the log and
  to a file the release keeps as an artifact; a refusal exits 1
- tests over handwritten API fixtures for every refusal class of the issue,
  the public command and the kit's own workflows; a mutant per safeguard

The release's verify job runs the gate (ci.yml is not called again) and keeps
the record; the package job checks the vendored copy. The tag, package,
checksum, attestation, approval, draft and published-asset gates are unchanged.
The fresh route after a refusal is the operator's: a full re-run of the main
CI run at the commit, or a dispatch on main, then one re-run of the release.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Review record: devkit PR #52, release evidence gate (head f8fa3d6, base 19e7281 = v0.9.0)

No predicate of the #38 gate is dropped or loosened, and no wrong-commit, wrong-event, wrong-workflow, partial or unfinished evidence is accepted. Every refusal fails the job closed. Findings: 0 HIGH, 2 MEDIUM, 4 LOW.
Clean-room review against issue #37's "Required evidence gate" list. Same-vendor review, not a cross-vendor claim.

Versus #38

Predicate #38 (lib/ci_evidence.py) #52 (lib/release_evidence.py) Verdict
Release checkout SHA grammar, HEAD = SHA, git diff --quiet HEAD, config bytes = commit same, plus status --porcelain, checkout root, and the running helper equal to its committed bytes kept, stronger
Configuration schema 1, age = 48, any event schema 2 with strict members, age 1 to 48, events limited to push and dispatch stronger
Repository config = context, run and head repository name and id same, plus the API repository identity and the artifact repository and head-repository ids stronger
Exact commit run, job, artifact and payload source same kept
Branch or ref main hard-coded (run, job, artifact, capture ref) configured branch, which must be the repository default (kit: main) kept, plus default-branch check
Event or trigger configured events, not pull_requests push or dispatch only, pull_requests == [], and capture checks GITHUB_WORKFLOW_REF/GITHUB_WORKFLOW_SHA stronger
Workflow identity resolved by file (path, id, active); run workflow_id + path same, plus name: on the resolve, every job and the payload stronger
Attempt selection newest by created_at, any status every applicable run must be completed; newest by the start of its latest attempt changed; matches the rule and closes a #38 pass-over (probe 2a)
Attempt binding the attempt's job list, artifact name with attempt, job interval same, plus job run_attempt = the selected attempt stronger
Jobs and skips names = jobs + skipped; success; labels ubuntu-latest; skipped really skipped same; labels from config (kit: ubuntu-latest); hosted group 0 "GitHub Actions"; identity checks on skipped jobs stronger
Check coverage PASS, or SKIP only when optional same kept
Inputs sha256 of config inputs, config and workflow files; workflow blob id git object ids of the same set plus dev.json and the helper (the workflow blob id is now in inputs) equivalent identity, wider set
Toolchain equal to the release runner; Linux/X64 fixed release-runner (the kit's mode) keeps that equality; pins added for other-OS legs (image version not compared, as documented) kept for the kit
Payload full equality per-field checks, then full equality kept
Age 0 to 48 h from run_started_at 0 to the configured age (at most 48 h) kept
Post-collection run re-read compared whole; re-select; final re-select; tag read again re-read compares id, attempt, status, conclusion, start and updated_at; the re-select re-runs run identity on the listing; final re-select; tag read again narrower check, same effect: identity fields are immutable and checked again by the re-select, and updated_at catches any update
Tag grammar, annotated, names the commit, same object at the end same kept
Refusal exit 1 with a code exit 1 plus a decision record (log, file and release-evidence-<attempt> artifact) stronger

Disposition

# Finding Severity Evidence What to do
1 Selection counts only runs whose event is configured. A newer failed or running run of the same workflow, on the trusted branch at the release commit, triggered by schedule, workflow_run or repository_dispatch, does not block reuse. Unreachable in the kit (its ci.yml has none of these triggers). Reachable for an adopter whose CI also runs on a schedule; the template and docs do not mention it MEDIUM fixture: a newer failed or running schedule, workflow_run or repository_dispatch run, 6 of 6 cases reuse Make every trusted-branch run of the workflow at the commit block reuse when it is newer and not green (evidence stays push/dispatch only). Or refuse such triggers at config time and document it
2 Two safeguards have no test that fails when they are removed: helper-source (R07; covered by checkout-changed/input-missing in a clean checkout, but not under skip-worktree) and the job head_branch check (R13; covered by the run-level branch check). R18 (job status) is an equivalent mutant: no action MEDIUM R07 and R13 survive all 75 tests Add a test where the helper bytes differ under git update-index --skip-worktree (expect helper-source), and one with a job on another head_branch (expect job-identity)
3 An artifact with an unsupported zip compression method raises NotImplementedError, which MALFORMED does not catch (deeply nested JSON would raise RecursionError the same way). Verify exits 1 with a traceback and writes no decision record; the upload step then fails on the missing file. The job still fails closed LOW fixture: zip with compression method 99 makes decide() raise Treat these as unavailable-or-malformed
4 The refusal-class table in docs/ci-evidence.md leaves out job-result, run-identity, timestamp and git-unavailable LOW the code's refusal codes compared with the doc Add them
5 Fresh route wording: "This works while the commit is no longer main's tip" reads as "only while" LOW docs/ci-evidence.md, fresh route step 1 Change to "also after main moved on"
6 No test checks that the decision record is uploaded on a refusal (if: ${{ !cancelled() }}); KitAdoption checks only the verify command string LOW reading the tests Assert the upload step and its condition

Probes (fake API plus a git fixture; read-only live API; all scratch deleted)

  • 2, selection. (a) An older green run re-run after a newer failed run: the re-run (run 100, attempt 2) is the newest attempt and is reused; if the failed run started later, run-result. (b) A newer run in_progress, waiting, requested, pending or action_required: run-unfinished. (c) Latest attempt failed: run-result; a re-run still running: run-unfinished. (d) 101 jobs, required leg on page 2: reuse, page 2 read; page-2 failure job-result, page missing api-unavailable, empty pagination-partial, total moved pagination-changed, repeated id pagination-duplicate, short total job-coverage. (e) Partial re-run: job-identity; if the API relabels the kept job, artifact-coverage; with the attempt-1 artifact renamed, evidence-identity.
  • 3, inputs (public command, git fixture). Evidence captured at A, verified at B (ci.yml), C (dev.json), D (pin file), E (tool pin in config), T (tree-equivalent of A): honest API run-missing; a listing returning A's run run-sha; API claiming the run at X with A's evidence evidence-identity; evidence relabelled to X input-identity, naming ci.yml, dev.json, the pin file and the config. Relabelled T reuses only when the API reports run, job and artifact all at T, which means genuine CI at T. A changed tool output or image on the release runner: runtime-identity.
  • 3, workflow identity. Same name at another path: workflow-identity (resolve), run-workflow (run), evidence-identity (payload). Same path with another name: workflow-identity, job-identity, evidence-identity. An extra label, group "Default", group id 1 or a null group: job-runner.
  • 4, release.yml. verify needs admit; package needs verify; attest-and-draft and publish follow package. A refusal exits 1 and fails the job. The record is uploaded on both outcomes (!cancelled(), if-no-files-found: error). Kept checks are unchanged and in order; the only additions are the record upload, the vendored-copy check and the tarball path list. actionlint is clean on both workflows.
  • 5, fresh route. Manual, run once, with a stop rule; consistent with "unavailable evidence never becomes success". A re-run of main's run keeps GITHUB_SHA at the release commit X, and capture refuses unless HEAD = GITHUB_SHA = GITHUB_WORKFLOW_SHA = X, so the inputs are X's blobs, which main moving cannot change; the toolchain is compared at release time. A dispatch after main moved runs at the new tip, is not applicable at X, and the gate keeps refusing. No manual route yields accepted evidence with different inputs.
  • 6, live (read-only). v0.9.0 main run 40345389159 (push, attempt 1): every API check passes (repository and default branch, annotated tag to 19e7281, workflow id/path/name/state, selection with 1 run and 1 applicable, run identity, lint and tests on ubuntu-latest, artifacts). Runner group 0 "GitHub Actions" confirmed live. It then refuses with evidence-identity: lint: schema is 1, the Batch validation policy, generated docs and trusted release CI evidence #38 payload, as expected. With a schema-2 payload built from the real coverage (archive checks on the real archive, tool resolution stubbed): reuse, all post-collection re-reads live. PR head f8fa3d6 (pull_request run 40378908297): tag-commit; with the tag stubbed, run-missing (1 run at the commit, 0 applicable). The documented durations match the API.
  • 7, mutants. Builder mutants M01, M03, M05, M09, M12, M13, M14, M16, M18, M20, M22 and M25: all caught. Mine caught: R01 (default branch), R02 (labels), R03 (hosted group), and R04, R05, R06, R08–R12, R14–R17, R19, R20. Survived: R07 and R13 (finding 2); R18 is equivalent.
  • 8, copies, text and docs. The vendored copy equals the library apart from line 2, which names v0.9.0 (= JPDEV_VERSION); 75 of 75 tests pass on both copies. No stale references to the old file names. No local paths, personal names or organization names in the diff. The docs match the code apart from findings 4 and 5.

merge: yes (no HIGH). Disposition per the kit policy: both MEDIUMs and the cheap LOWs are fixed in this PR in one round by the builder and confirmed by this reviewer.

kikashy and others added 2 commits October 26, 2026 14:58
- MEDIUM 1: every run of the workflow at the release commit takes part in
  the newer-attempt scan, whatever its event or branch (schedule,
  workflow_run, repository_dispatch, a pull request, another branch): an
  unfinished one refuses (run-unfinished), and one whose latest attempt
  started after the selected run's and did not succeed refuses
  (run-newer-failed). Only a trusted-trigger run is evidence; a newer
  successful run of another trigger neither blocks nor is selected.
- MEDIUM 2: tests for the helper-source check (the helper's bytes changed
  under skip-worktree, so the checkout looks clean) and for the job
  head_branch check.
- An unsupported zip compression method, nested JSON past the recursion
  limit, a truncated stream or zlib error is an unavailable-or-malformed
  refusal with a decision record, not a traceback.
- docs/ci-evidence.md: the selection rule, run-newer-failed and the
  job-result, run-identity, timestamp and git-unavailable codes in the
  refusal table, the fresh-route wording; the template's workflow comment.
- A test that the release keeps the decision record whatever the decision.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…scope and order

- CI failure on 6b306d3: the over-deep JSON fixture raised RecursionError
  on one Python and parsed on the runner's, where the non-object check then
  refused as artifact-entries. Nesting is now bounded (16 levels) by a scan
  before parsing, and a malformed or unreadable artifact is
  unavailable-or-malformed whatever the interpreter: an archive that does
  not open, a compression other than stored or deflated or members declared
  over 256 KiB before the entries check; content over 256 KiB when read,
  over-deep nesting, JSON that does not parse or is not an object after it.
  One test per kind, each naming its code and detail.
- Review MEDIUM A: the newer-attempt scan takes only this repository's runs
  on the trusted branch, without pull request events (pull_request*) and
  merge_group: a fork pull request at the release commit, newer and failed
  or awaiting approval, or another branch's run no longer blocks a release.
  Schedule, workflow_run and repository_dispatch runs on the branch still
  refuse.
- Review MEDIUM B: fixtures for a lower-id run re-run later that failed
  (refuses) and a higher-id failed run that started before the selected
  attempt (does not block): newer is by attempt start, not run id.

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikashy

kikashy commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Confirmation: 6b306d3

The fix round (one commit on f8fa3d6, 5 files) passes: 80 of 80 tests on both copies, and the vendored copy equals the library apart from line 2. Public text is clean. Same-vendor review, not a cross-vendor claim.

# Finding Status Evidence
1 A newer failed or running run by another trigger is passed over fixed, but goes further than asked (see A) All six schedule, workflow_run and repository_dispatch cases refuse (run-newer-failed or run-unfinished). A newer successful run by schedule, workflow_run, repository_dispatch, a pull request, a fork's main or another branch neither blocks nor is selected: reuse of run 100, and its jobs and artifacts are never read
2 helper-source and job head_branch have no failing test fixed R07 and R13 are now caught by the new tests
3 A malformed archive crashes verify with no decision record fixed NotImplementedError and RecursionError are recorded as unavailable-or-malformed; a mutant dropping them from MALFORMED is caught
4 Refusal codes missing from the doc table fixed job-result, run-identity, timestamp, git-unavailable and the new run-newer-failed are listed
5 Fresh-route wording fixed now reads "also works after main moved on"
6 No test that the record is uploaded on a refusal fixed a test now checks the upload step, its !cancelled() condition and if-no-files-found: error
A NEW. The scan covers every run at the commit, whatever its branch or repository, so an outside account can make a release refuse. It opens a pull request from a fork whose head is the release commit, against any branch that lacks it, which is possible in a public repository. If that run is newer and ends action_required or failure, the gate refuses with run-newer-failed. If it is older but still waiting for approval, run-unfinished. If it has no start time, timestamp. The gate still fails closed and accepts nothing wrong. Re-running main's CI makes main's run the newest again, but reopening the pull request makes a newer blocking run, so the block can be repeated. The builder's new test expects a pull request run, and a run on another branch, to block MEDIUM fixture: a fork pull request run at the commit refuses in all 4 cases
B NEW. The run-newer-failed ordering is not held by a test: comparing run ids instead of attempt start times survives all 80 tests MEDIUM my mutant N3 survives

Mutant checks: R07, R13 and three new-check mutants (newer-failed removed, scan narrowed back to applicable, the new MALFORMED classes removed) are caught; N3 survives (finding B). No new HIGH.

merge: yes (no HIGH; A and B go to the pre-release batch issue, which must land before the next release, the first live run of this gate)

Confirmation: bb2eb1b

The second fix round (one commit on 6b306d3, 5 files) passes: all tests on the library, the vendored copy and under -X dev, and the vendored copy equals the library apart from line 2. The PR's CI on bb2eb1b (lint, tests) is green. Same-vendor review, not a cross-vendor claim.

# Finding Status Evidence
A A fork pull request run at the commit could hold a release fixed The scan now covers only this repository's trusted-branch runs, excluding pull_request* and merge_group. Every fork pull request case reuses: newer and action_required, newer and failure, older and still waiting, a null start time, a fork's main. The six schedule, workflow_run and repository_dispatch cases on the trusted branch still refuse (run-newer-failed or run-unfinished). Mutants that widen the scan or drop any one condition (branch, head repository, pull-request prefix, merge_group), or narrow it back to applicable runs: all caught
B The run-newer-failed ordering was not held by a test fixed N3 (compare by run id) is caught by two new tests: a lower-id run re-run later that failed, and a higher-id run that started earlier
CI 100,000-deep JSON gave a different refusal on the runner than locally fixed The nesting is now counted before parsing. 100,000 deep and 17 deep both refuse with unavailable-or-malformed (nesting deeper than 16), whether the recursion limit is 200 or 10000. Mutants removing the compression, declared-size, nesting, string-skip or non-object checks: all caught
L1 NEW. The read-size check cannot be reached: zipfile stops at the declared size and fails the CRC (BadZipFile), and the declared size is already capped. Removing the check survives as an equivalent mutant. The builder's "one test per kind" overstates this one LOW fixture: an entry declared at 100 bytes holding 300 KB raises BadZipFile
L2 NEW. The nesting scan counts bytes, so a UTF-16 evidence.json (which json.loads accepts) can hide its brackets from it. The result is still unavailable-or-malformed (RecursionError at both limits, or a non-object if it parses); only the detail differs LOW fixture: 60,000 deep in UTF-16

No new HIGH or MEDIUM. The first-round findings 1–6 stay fixed.

merge: yes

@kikashy
kikashy merged commit 6de13cf into main Oct 26, 2026
3 checks passed
@kikashy
kikashy deleted the release-evidence-37 branch October 26, 2026 19:12
@github-actions

Copy link
Copy Markdown

🎉 Thank you for your contribution to Judgment Pack!

If you’d like another task, choose a path that interests you:

@kikashy kikashy mentioned this pull request Oct 26, 2026
kikashy added a commit that referenced this pull request Oct 26, 2026
Version 0.10.0 in lib/common.sh and on line 2 of the vendored scripts and
the template; the release notes for v0.10.0 (the hardened release
evidence gate, #52).

Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant