Skip to content

fix(ci): reliably publish Claude review results - #6907

Open
Sing303 wants to merge 4 commits into
thomhurst:mainfrom
Sing303:codex/claude-review-publication
Open

Sing303 wants to merge 4 commits into
thomhurst:mainfrom
Sing303:codex/claude-review-publication

Conversation

@Sing303

@Sing303 Sing303 commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A successful Claude review run can finish without posting a comment (run 36419506127, no comment on #6906). The workflow told the model to post through pr-review-comment.sh, while the code-review plugin stops without commenting unless it gets --comment. Publication depended on how the model resolved that conflict.

Change

  • Claude returns the review as its final response and no longer has access to the posting helper.
  • A new Post review step reads the final result from the action's execution_file and posts it through the existing pr-review-comment.sh helper (which still refuses empty bodies and always targets the triggering PR).
  • The model-side helper cap (CLAUDE_CODE_SCRIPT_CAPS) is removed because the model can no longer call the helper.

Summary by CodeRabbit

  • Improvements
    • Automated pull request reviews are now returned as Markdown and published by the review workflow after analysis.
    • Review comments are submitted through the trusted CI process, rather than being posted directly by the review assistant.

@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: b12e54a1-eed6-41fe-9c9d-b4632cfdc4a3

📥 Commits

Reviewing files that changed from the base of the PR and between edc9a67 and 3a3aaf3.

📒 Files selected for processing (1)
  • .github/workflows/claude-code-review.yml

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

The workflow asks Claude to return its review as Markdown instead of posting it. A workflow step extracts the last result from the action execution file and passes it, with pull request context, to the comment script.

Changes

Claude review publication

Layer / File(s) Summary
Claude result handoff to CI publisher
.github/workflows/claude-code-review.yml, .github/scripts/pr-review-comment.sh
The workflow removes the script call cap and Claude’s access to the comment script. It asks Claude to return Markdown, then extracts the last result and passes it with pull request context to the script. The script comment identifies the trusted CI publisher as the caller.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Claude as Claude review action
  participant Workflow as GitHub Actions workflow
  participant CommentScript as pr-review-comment.sh
  Claude->>Workflow: Return review result
  Workflow->>Workflow: Extract last result entry
  Workflow->>CommentScript: Pass result and pull request context
Loading

Suggested reviewers: thomhurst

Merge Risk: ⚪ Minimal · up to 3a3aa

The workflow now hands Claude’s final review to the CI publisher. The verified fork-PR trigger retains comment-write permission, and no concrete publication blocker remains in the reviewed change.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 3a3aa

The change reduces direct access to the publishing helper during review and keeps the comment destination tied to the selected pull request. The new publisher does not, however, verify that the text came from exactly one successful final result before posting it.

Retained concerns

  • Low · security · inferred: The new automatic publisher accepts the last execution-file result rather than enforcing a unique, successful final result before making an authenticated PR comment. An ambiguous nonempty result could therefore be published as the review; whether the external action can produce such a result on a successful run remains unverified.
Security review details

Security Blast Radius

  • inferred — For a PR-triggered run, malicious PR content can influence review text that may become a public comment, but the workflow-derived PR number and repository prevent that text from directly selecting another destination. Manual dispatch deliberately accepts a PR-number input within the same repository.

Security Findings and Attack Paths

  • inferred — If a successful action run supplies multiple result entries or a non-success result with nonempty text, the publisher can turn the last entry into an authenticated comment. Repository-local evidence does not establish whether the action can produce that state; no cross-PR retargeting path is shown.

Trust Boundaries and Controls

  • observed — The explicit helper permission is removed from review tools. The publisher runs after the action, quotes the generated body, and obtains the target from workflow context rather than that body. These controls narrow the model-to-write path but do not validate result provenance.

Resilience and Maintainability Implications

  • inferred — Cancellation, retry, or another authorized manual run can leave more than one published comment because publication has no stable comment identity or update path. The existing helper was already append-only; the new workflow does not resolve that recovery invariant.

Hardening Proposals

  • proposed — Before publication, require exactly one explicitly successful final result from the documented action output; consider a stable comment identity for retry and cancellation recovery.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 3 files. (1 skipped: 1 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: making CI publication of Claude review results reliable.
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 3 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit reads the review’s refrain,
Claude returns its words in Markdown rain.
The workflow gathers the final line,
Then sends it on with context fine.
The comment script completes the chain.

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Critical risk] Changes how untrusted code review output reaches pull requests.

The PR does not appear safe to merge until publication rejects unsuccessful SDK results.

Findings

  1. P1 Error results can be posted ▶
  2. P2 Permission denials go unnoticed ▶
  3. P2 Failed posts lose review text ▶
  4. P2 Action output contract untested ▶

Summary

The PR replaces the validated Claude review publisher with an inline jq extraction and direct call to the existing comment helper.

  • The replacement no longer validates the SDK result before posting.
  • Permission-denial warnings and the diagnostic job summary are no longer produced.
  • The publisher, its tests, and its PR validation workflow were removed.

Reviews (3) · Last reviewed commit: "Simplify Claude review publication to a ..."

Comment thread .github/scripts/publish-claude-review.mjs Outdated
Comment on lines +11 to +13
const success = result => ({
type: 'result', subtype: 'success', is_error: false, result, permission_denials: [],
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Action output contract untested

The success fixture defines the same result fields that the publisher expects. These tests can therefore pass even if the action's actual execution_file has a different shape, leaving live reviews rejected. A captured action-produced fixture would test that boundary.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that this synthetic fixture tests the publisher's validation logic, not the live action boundary.

The expected format was also checked against the producer: the action serializes an array of SDK messages and stops reading after the first result. This supports the validation rules, but does not replace a captured execution file.

The PR's rollout notes explicitly leave live verification outstanding. That controlled run should provide a sanitized action-produced fixture, preserving its structure and recording the action/SDK versions. We do not currently have that fixture, so this remains a known coverage gap. Unexpected result shapes already produce an error annotation, a failed publication step and a diagnostic summary.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The synthetic fixture appropriately tests the publisher’s validation logic, and the producer-source checks support the expected array/result contract. Since the PR documents live action verification and a sanitized execution-file fixture as rollout follow-up, I’m withdrawing this as a required change. The remaining coverage gap is clearly recorded rather than hidden.

@github-actions

Copy link
Copy Markdown
Contributor

Review: PR #6907 — CI-only change (Claude review publication pipeline)

This PR touches no .NET/TUnit engine code — it's entirely .github/scripts and .github/workflows, adding a trusted CI publisher (publish-claude-review.mjs) that moves posting the Claude review comment out of the model's own tool access and into a separate, unprivileged-input-validated step. The core design (validate execution result → validate target repo/PR → post via existing pr-review-comment.sh) is a sound way to keep untrusted PR-head content from being able to direct where/what gets posted, and the new script is well covered by 35 tests (shell-injection safety, URL/host/repo/PR confirmation, secret redaction, most failure paths).

Findings

  1. inspectExecution() fails closed on an unverified assumption (.github/scripts/publish-claude-review.mjs:21)
    The function throws (blocking publication) unless the execution file contains exactly one result-type SDK message with subtype: "success" and is_error === false. This shape is validated only against hand-written synthetic fixtures, not against a real claude-code-action execution file. If a live run ever emits zero, two, or a differently-shaped result message (e.g. a resumed/multi-turn session, or a future SDK version), every review would silently stop publishing with "Publication not confirmed." Worth flagging in the rollout notes as the one path that still needs to be exercised end-to-end against a live action run before fully trusting it, and/or adding a warning log that surfaces why publication was skipped so this doesn't fail silently in practice.

  2. New show_full_output / display_report action inputs unverified against claude-code-action@v1 schema (.github/workflows/claude-code-review.yml:85)
    These are added specifically to keep raw execution output and the plugin's PR report private. GitHub Actions silently ignores unrecognized with: keys — if either input name or accepted value doesn't match what the action actually defines, the privacy goal silently fails (defaults to exposing output) with no error surfaced. Worth double-checking these against the pinned action version's action.yml (or a first real run's logs) rather than relying on inference alone.

  3. Stale comment about CLAUDE_CODE_SCRIPT_CAPS (.github/workflows/claude-code-review.yml:43)
    The comment says the cap is retained as "defense in depth" because "only the CI publisher invokes it," but that cap is enforced inside claude-code-action's own tool-call gating and has no effect on the new publish step, which invokes pr-review-comment.sh via plain child_process.spawn outside Claude's tool-use path entirely. As written the env var is effectively dead configuration for this path, and the comment could mislead a future reader into thinking it still rate-limits the publisher.

  4. No test for validateTarget()'s malformed-input rejection path (.github/scripts/publish-claude-review.mjs:40, tests in publish-claude-review.test.mjs)
    The PR description mentions coverage of malformed results, wrong URLs, and single-call targeting, but no test exercises validateTarget() directly with a malformed GH_REPO/PR_NUMBER. A future refactor of the regex or the call-order around line 118 (e.g. swapping repo/PR argument order) could ship without a test catching it.

Notes on scope/confidence

  • No .NET/engine code is touched, so the AGENTS.md constraints on source-generator/engine parity, AOT, and TUnit.PublicAPI don't apply here.
  • I could not independently verify the execution_file action output name or the show_full_output/display_report input names against upstream claude-code-action from this environment (only gh pr view/diff was reachable, not general web/API access) — the PR body's cited commit-pinned permalinks were treated as credible for the execution_file shape, but the two newer input flags (finding Retry attribute  #2) remain unverified.
  • No prior Claude review exists on this PR to reconcile against — only bot reviews from CodeRabbit and Greptile are present so far.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Review: PR #6907 — CI-only change (Claude review publication pipeline)

CI/workflow-only change (.github/scripts, .github/workflows); no .NET/TUnit engine code is touched, so the AGENTS.md source-generator/engine-parity, AOT, and TUnit.PublicAPI constraints do not apply.

Design: moving publication out of the model own tool access and into a trusted CI step (publish-claude-review.mjs) that validates the action outcome, the SDK result shape, and the target repo/PR before shelling out to the existing pr-review-comment.sh is the right fix for the underlying problem — publication no longer depends on the model correctly resolving the conflict between "you must post" and a plugin that stays silent without --comment. Passing the body through spawn(..., { shell: false }) as a literal argv element, never shell source or a file path, keeps untrusted PR-derived Markdown from doing anything beyond being posted as a comment body. The 35-test suite exercises injection safety, URL/host/repo/PR confirmation, secret redaction, and the failure paths thoroughly.

Comparing against the earlier Claude review on this PR

The previous automated review here raised four points; checking them against the current diff:

  • Stale CLAUDE_CODE_SCRIPT_CAPS comment — addressed. The comment now correctly says the cap "does not limit the separate CI publisher, which makes one helper call and never retries automatically," rather than implying the cap still gates the new publish step.
  • No test for validateTarget() malformed-input path — addressed. publish-claude-review.test.mjs now has a dedicated table of malformed GH_REPO/PR_NUMBER cases (missing, extra path segment, whitespace, embedded newline, zero/negative/non-integer PR number) asserting rejection before the helper is invoked.
  • inspectExecution() one-result-message assumption is unverified against a real claude-code-action execution file — still an open, acknowledged risk. It is inherent to not having a live run to test against, and the PR description already calls this out explicitly as the follow-up step after merge (run Claude on a fork PR and verify). Not fixable from this sandbox either, since there is no access to a live action run, so this is carried forward as a rollout item rather than a blocking defect.
  • show_full_output/display_report input names unverified against the action schema — still open for the same reason: no network/web access is available in this review environment to check claude-code-action@v1 action.yml. Worth a first-run check of the job summary/logs after merge to confirm these inputs are recognized, since GitHub Actions silently ignores unknown with: keys, so a typo would silently fail to suppress the raw output rather than erroring.

Other notes (not blocking)

  • GH_TOKEN: ${{ github.token }} is scoped correctly to the job pull-requests: write permission and is only used by the trusted publish step, not exposed to the untrusted analysis step — consistent with the existing minimal-permissions design from fix(ci): run code review on pull requests from forks #6722.
  • The new claude-review-validation.yml pins the actionlint download to a specific version and verifies its SHA-256 before extracting, and extracts only the named actionlint binary — good practice for a workflow that downloads and runs a third-party binary in CI, even under contents: read.
  • The if: always() && !cancelled() && steps.claude-review.outcome != 'skipped' guard on the publish step correctly avoids attempting publication when the analysis step never ran, while still running after an analysis failure so a summary gets written either way.

Overall this is a solid, well-tested fix for a real reliability gap. The two residual risks are inherent to not being able to exercise a live claude-code-action run from a review sandbox, and both are already flagged for post-merge verification in the PR description itself.

Drop the Node publisher, its tests and the validation workflow. Claude
returns its review as its final response and a plain workflow step posts
it through the existing helper.
GH_REPO: ${{ github.repository }}
GH_TOKEN: ${{ github.token }}
run: |
body=$(jq -r '[.[] | select(.type == "result")] | last | .result // empty' "$EXECUTION_FILE")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Error results can be posted

If the Claude action succeeds but its result has is_error: true and nonempty .result text, this step posts that text as a completed review. The previous publisher rejected such results; this replacement checks neither the error flag nor the result subtype or count. An incomplete analysis could therefore appear on the PR as a finished review. Validate the result before calling the helper.

GH_REPO: ${{ github.repository }}
GH_TOKEN: ${{ github.token }}
run: |
body=$(jq -r '[.[] | select(.type == "result")] | last | .result // empty' "$EXECUTION_FILE")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Permission denials go unnoticed

A successful result can still contain tool permission denials, but this step posts only its Markdown. The previous publisher warned readers that the review might be incomplete and recorded denial counts by tool. Without either warning, readers cannot tell when access restrictions may have limited the review.

Comment on lines +120 to +121
body=$(jq -r '[.[] | select(.type == "result")] | last | .result // empty' "$EXECUTION_FILE")
.github/scripts/pr-review-comment.sh "$body"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Failed posts lose review text

If the helper fails to post, this step leaves no job-summary copy of the final Markdown or publication outcome. The previous publisher kept both in the summary, so maintainers could recover a review that never became a comment. Preserve that diagnostic record when publication fails.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants