Repository navigation
Release evidence gate: reuse exact-commit main CI evidence at release, fail closed (#37) - #52
Conversation
…, fail closed (#37) The gate #38 introduced as lib/ci_evidence.py moves to lib/release_evidence.py, vendored as scripts/dev/release_evidence.py (the library with the kit release on line 2, held to it by tests/run.sh and the release workflow), and closes the gaps against the issue's evidence contract: - configuration schema 2 (.github/ci-evidence.json, template templates/ci-evidence.json): strict members; the workflow by path and name; the trusted branch, which must be the default branch; per-job runner labels; matrix legs as one entry each; jobs that must be skipped; toolchain profiles compared with the release runner or held to exact pins (legs on another OS); pin files among the inputs; checks the release runs itself recorded as fresh_at_release - selection: every applicable run must be complete, and the run whose latest attempt started last is selected whatever its result, so a newer failed, cancelled or running attempt, a re-run of an older run included, is never searched past - verification: the repository by name and id, hosted runners, the jobs of the selected attempt only, artifacts bound to the run and repository, input identities as git object ids (manifest, configuration, the gate, workflow files, pin files), capture bound to the running workflow file - a decision record, reuse or refusal with its code, written to the log and to a file the release keeps as an artifact; a refusal exits 1 - tests over handwritten API fixtures for every refusal class of the issue, the public command and the kit's own workflows; a mutant per safeguard The release's verify job runs the gate (ci.yml is not called again) and keeps the record; the package job checks the vendored copy. The tag, package, checksum, attestation, approval, draft and published-asset gates are unchanged. The fresh route after a refusal is the operator's: a full re-run of the main CI run at the commit, or a dispatch on main, then one re-run of the release. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review record: devkit PR #52, release evidence gate (head f8fa3d6, base 19e7281 = v0.9.0)No predicate of the #38 gate is dropped or loosened, and no wrong-commit, wrong-event, wrong-workflow, partial or unfinished evidence is accepted. Every refusal fails the job closed. Findings: 0 HIGH, 2 MEDIUM, 4 LOW. Versus #38
Disposition
Probes (fake API plus a git fixture; read-only live API; all scratch deleted)
merge: yes (no HIGH). Disposition per the kit policy: both MEDIUMs and the cheap LOWs are fixed in this PR in one round by the builder and confirmed by this reviewer. |
- MEDIUM 1: every run of the workflow at the release commit takes part in the newer-attempt scan, whatever its event or branch (schedule, workflow_run, repository_dispatch, a pull request, another branch): an unfinished one refuses (run-unfinished), and one whose latest attempt started after the selected run's and did not succeed refuses (run-newer-failed). Only a trusted-trigger run is evidence; a newer successful run of another trigger neither blocks nor is selected. - MEDIUM 2: tests for the helper-source check (the helper's bytes changed under skip-worktree, so the checkout looks clean) and for the job head_branch check. - An unsupported zip compression method, nested JSON past the recursion limit, a truncated stream or zlib error is an unavailable-or-malformed refusal with a decision record, not a traceback. - docs/ci-evidence.md: the selection rule, run-newer-failed and the job-result, run-identity, timestamp and git-unavailable codes in the refusal table, the fresh-route wording; the template's workflow comment. - A test that the release keeps the decision record whatever the decision. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…scope and order - CI failure on 6b306d3: the over-deep JSON fixture raised RecursionError on one Python and parsed on the runner's, where the non-object check then refused as artifact-entries. Nesting is now bounded (16 levels) by a scan before parsing, and a malformed or unreadable artifact is unavailable-or-malformed whatever the interpreter: an archive that does not open, a compression other than stored or deflated or members declared over 256 KiB before the entries check; content over 256 KiB when read, over-deep nesting, JSON that does not parse or is not an object after it. One test per kind, each naming its code and detail. - Review MEDIUM A: the newer-attempt scan takes only this repository's runs on the trusted branch, without pull request events (pull_request*) and merge_group: a fork pull request at the release commit, newer and failed or awaiting approval, or another branch's run no longer blocks a release. Schedule, workflow_run and repository_dispatch runs on the branch still refuse. - Review MEDIUM B: fixtures for a lower-id run re-run later that failed (refuses) and a higher-id failed run that started before the selected attempt (does not block): newer is by attempt start, not run id. Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirmation: 6b306d3The fix round (one commit on f8fa3d6, 5 files) passes: 80 of 80 tests on both copies, and the vendored copy equals the library apart from line 2. Public text is clean. Same-vendor review, not a cross-vendor claim.
Mutant checks: R07, R13 and three new-check mutants (newer-failed removed, scan narrowed back to applicable, the new merge: yes (no HIGH; A and B go to the pre-release batch issue, which must land before the next release, the first live run of this gate) Confirmation: bb2eb1bThe second fix round (one commit on 6b306d3, 5 files) passes: all tests on the library, the vendored copy and under
No new HIGH or MEDIUM. The first-round findings 1–6 stay fixed. merge: yes |
|
🎉 Thank you for your contribution to Judgment Pack! If you’d like another task, choose a path that interests you: |
Version 0.10.0 in lib/common.sh and on line 2 of the vendored scripts and the template; the release notes for v0.10.0 (the hardened release evidence gate, #52). Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Issue #37: a shared, fail-closed, tested gate that lets a release take the exact-commit CI evidence main already produced instead of running CI again; adopted by the kit's own release. Its "Required evidence gate" list is the acceptance contract.
#38 shipped a first gate as
lib/ci_evidence.py(used by the v0.8.0 and v0.9.0 releases) and closed the issue. This PR builds on it: the gate moves tolib/release_evidence.py, is vendored asscripts/dev/release_evidence.py, and closes the gaps against the contract (below). One review round; release admission is a trust boundary.Public surface changes
lib/ci_evidence.pybecomeslib/release_evidence.py, vendored asscripts/dev/release_evidence.py(line 2 names the kit release;tests/run.shand the release workflow hold it to the library). Both workflows now run the vendored file.verifygains--outputand always writes a decision record..github/ci-evidence.json), with the adopters' templatetemplates/ci-evidence.json. The evidence artifact payload is schema 2.What changed against the contract
release-runner(the release job resolves image and tools itself and requires equality) orpins(exact tool output, platform, optionalImageOS; for macOS/Windows legs); pin files ininputs;fresh_at_releasenames checks the release runs itself (tag-sensitive) and carries them into the record. The kit's inventory is indocs/ci-evidence.mdhead_sha, workflow id + path +name:, branch = the repository's default branch, push or configured dispatch, no PR association; jobs bound to the run, attempt, commit and workflow name, on the configured labels on a GitHub-hosted runner; artifacts bound to the run and repository ids; capture refuses unlessGITHUB_WORKFLOW_REF/GITHUB_WORKFLOW_SHAare the configured file at the commit. No commit status or check run is ever readcompleted(run-unfinished), then the run whose latest attempt started last is selected whatever its result (#38 selected by creation time, so a failed re-run of an older run could be passed over); re-read and re-selected after collection and after all workflowsskipped_jobsmust be skipped, required jobs may not be (job-skipped), checkSKIPonly foroptional_skips; jobs of the selected attempt only; every page read with stable totalsrelease.yml: tag grammar/annotation/commit/main, notes, environment reviewer, version and copies, package run, checksums, attestation, draft and published identitytests/release_evidence_test.py: 75 tests (below); 26 mutants (below)docs/ci-evidence.md,docs/releasing.md,docs/adopting.md,docs/policy.md, README, AGENTS.mdThe decision record (stdout and
--output; the release keeps it as therelease-evidence-<attempt>artifact): decision, refusal code and detail, commit, tag object, input object ids, per workflow the id/path/name, run counts, newest candidates, selected run/attempt/event/result/start/age, referenced workflows, and per job its id, result, labels, times, artifact id/name/digest, runtime and check results.Refusal classes tested (handwritten API fixtures)
Accepted exact-commit evidence (push, and dispatch when configured); wrong SHA (run, evidence, job, artifact); tree-equivalent other commit; wrong repository (run, fork head, API answer, artifact, evidence) and a non-default trusted branch; wrong event (PR head, synthetic merge from a fork's main, merge queue, another branch, a main run tied to a PR, evidence of another event/ref); wrong workflow (path, name, state, a run of another workflow with the same job names, another workflow ran and this one did not, evidence of another workflow); a mutable status name (green statuses and check runs named like the jobs, and the gate never reads
commits/); missing job or matrix leg, extra or repeated job, missing/repeated/other-attempt artifact, missing check; unexpected skip (required leg, PR-only job ran, required check SKIP/FAIL/NOT RUN, optional skip that failed); newer failed/cancelled/timed-out/action-required run, newer or older running/queued run, newer failed attempt of the selected run, older run re-run later and failed, attempt race, partial rerun, new attempt / newer run / changed answer during collection, newer run after all workflows; changed inputs (workflow, manifest and pin file identities, missing input, pinned toolchain on both profiles, release-runner image, macOS image and architecture, tool set, unexpected members); unavailable evidence (API error at four points, truncated listing, 1,000 cap, expired/oversized/out-of-interval/digest/other-run artifact, archive with another entry, malformed answers, stale and future runs, other runners); tag movement (other commit, a tree, lightweight tag, tag moved during collection, grammar). Plus pagination, every configuration member and invalid value, command and runtime bounds, the public command in a fixture repository with a stand-ingh(reuse and refusal records, empty/missing/oversized answers, context refusals, capture refusals, capture-to-verify round trip), and the kit's own workflows against its configuration.Mutants (one safeguard removed per scratch copy; the intended tests fail at their assertion; the unmutated library passes the same tests)
'job-identity' != 'run-sha''artifact-run' != 'run-repository''run-trigger' != 'run-missing''reuse' != 'run-workflow''reuse' != 'workflow-identity''unavailable-or-malformed' != 'run-missing'(still refused by the catch-all)'unavailable-or-malformed' != 'job-coverage'(extra/repeated job: reuse)'unavailable-or-malformed' != 'artifact-coverage''reuse' != 'job-skipped''reuse' != 'job-not-skipped''reuse' != 'check-result''reuse' != 'run-result''run-result' != 'run-unfinished''reuse' != 'run-result''reuse' != 'attempt-changed''evidence-identity' != 'input-identity'(the full-equality backstop still refuses)'reuse' != 'runtime-identity''reuse' != 'runtime-identity''reuse' != 'pagination-partial''accepted' != 'api-unavailable''reuse' != 'artifact-expired''reuse' != 'run-age''reuse' != 'tag-commit''reuse' != 'tag-changed''reuse' != 'tag-annotated'The kit's release: before and after
Checks executed at the release commit:
lintchecks (diff-check, local-paths, agent-docs, bash-n, json, shellcheck; duplication from v0.8.0)CI at the tag / lintlintevidencetests(tests/run.sh)CI at the tag / teststestsevidencerelease_evidence.pycopy checkObserved (API job start to completion): v0.6.0 run 39844235747 lint 9 s, tests 58 s (admit end to package start 63 s); v0.7.0 run 39962449641 lint 10 s, tests 57 s (62 s); v0.8.0 run 40274516108 verify 14 s (17 s); v0.9.0 run 40351296561 verify 15 s, the gate step itself about 9 s (20 s). The last two releases already ran the #38 gate. This gate reads the same API objects plus one repository read and resolves the same tools, so its cost is expected to be of that order; it is first measured at the next release. No other saving is claimed; a refusal adds the fresh route's full CI run.
The fresh route
After a refusal the operator chooses, once: re-run all jobs of the main CI run at the commit (works after main moved on; the route for
run-ageand a changed hosted image), or dispatchci.ymlon main while main names the commit; then re-run the release run's failed jobs. A second refusal: fix on main and release a new version. Nothing dispatches or re-runs automatically.ci.ymlkeeps itsworkflow_calltrigger, but the release does not call it: since #38 it has not, and a release-side fallback would need a second trigger on the release workflow, while the re-run and the dispatch keep fresh evidence under the trusted-branch identity the gate verifies.Not verifiable before merge
captureruns only on a push or dispatch on main, so this PR's CI does not exercise it live; the first main push after merge does (GITHUB_WORKFLOW_REF/GITHUB_WORKFLOW_SHAincluded). A capture refusal there fails main'slint/testsvisibly.verifyis first live at the next release. A read-only probe ran the new gate's API walk (repository, tag, workflow resolution, selection, run identity, attempt jobs with hosted group and labels, artifact metadata, interval, digest and archive) over the real v0.9.0 main run 40345389159 and passed; only the payload (schema 1) differs from what this gate expects.Checks
Candidate: head
f8fa3d6(no uncommitted diff).jpdev check --changedagainstorigin/main(merge base 19e7281): unmapped paths, whole list.tests/release_evidence_test.pyalso passes withRELEASE_EVIDENCE_HELPERset to the vendored copy. CI on this head and the review remain.Fix round
For the review record at
f8fa3d6(0 HIGH, 2 MEDIUM, 4 LOW). One commit:6b306d3.schedule/workflow_run/repository_dispatchruns did not block reuserun-unfinished; newer than the selected run and not successful refusesrun-newer-failed. Only a trusted-trigger run is evidence; a newer green run of another trigger neither blocks nor is selected'reuse' != 'run-newer-failed'helper-sourceuntestedgit update-index --skip-worktree(the checkout looks clean):helper-sourcehead_branchuntestedjob-identity'reuse' != 'job-identity'NotImplementedError,RecursionError,EOFError,zlib.errorareunavailable-or-malformedrefusals with a decision recordjob-result,run-identity,timestamp,git-unavailableadded, withrun-newer-failed; every code the code raises is now in the tableKeep the decision recordstep hasif: ${{ !cancelled() }}, the verify--outputpath andif-no-files-found: errorThe selection rule in
docs/ci-evidence.mdand the template's workflow comment now state the wider scan. Mutants: M01–M29 all caught (M12 and M14 rewritten for the new selection code; M12 now also meets the scan,'run-newer-failed' != 'run-result'); the unmutated library passes the same tests.tests/release_evidence_test.py: 80 tests, OK on the library and on the vendored copy.Candidate
6b306d3,jpdev check --changed(whole list):Second commit:
bb2eb1b6b306d3(run 40382530467,tests): the 100,000-deep JSON fixture raisedRecursionErroron the local Python but parsed on the runner's, where the non-object check then refused asartifact-entries. Nesting is now bounded at 16 levels by a scan before parsing, and malformed or unreadable evidence isunavailable-or-malformedon every interpreter: before the entries check, an archive that does not open, a compression other than stored or deflated, or members declared over 256 KiB; after it, content over 256 KiB when read, too-deep nesting, JSON that does not parse, or a document that is not an object. There is one test per kind, each checking its code and detail. Passes underpython3 -X devand with the recursion limit set to 10000 and to 200.pull_request*events andmerge_group. A fork pull request at the release commit (newer and failed, or awaiting approval), a same-commit run on another branch, apull_request_targetor merge-queue run, and another head repository's run each no longer block. The six schedule/workflow_run/repository_dispatchcases still refuse. M30 (scan every run again):'run-newer-failed' != 'reuse'.run-newer-failed; a higher-id failed run that started before the selected attempt does not block. M31 (compare run ids): both tests fail.jpdev check --changedonbb2eb1b: whole list, 8 passed, 0 failed, 0 skipped, 0 not run (tests: 441 passed, 0 failed).🤖 Generated with Claude Code