[smoke-detector] 🔍 Smoke Test Investigation - Smoke OpenCode Run #18722224746: Agent Does Not Use Safe-Outputs MCP Tools

[github-actions]

🔍 Smoke Test Investigation - Run #18722224746

Summary

The Smoke OpenCode workflow failed because the create_issue job could not find the expected agent_output.json file. The OpenCode agent completed successfully but did not use the safe-outputs MCP tools despite being instructed to create an issue, leaving no output artifact for downstream jobs.

Failure Details

• Run: #18722224746
• Commit: 5d9b9b9
• Trigger: workflow_dispatch
• Duration: 2.0 minutes
• Failed Job: create_issue (6s duration)
• Workflow: Smoke OpenCode

Root Cause Analysis

Primary Issue

The OpenCode agent received the prompt: "Review the last 5 merged pull requests in this repository and post summary in an issue."

Despite clear instructions to create an issue, the agent:

1. ✅ Completed successfully (51s runtime)
2. ❌ Did NOT use the create_issue tool from the safe-outputs MCP
3. ❌ Did NOT create /tmp/gh-aw/safe-outputs/outputs.jsonl
4. ❌ Left no output artifact for downstream jobs

The agent log shows:

Output file does not exist: /tmp/gh-aw/safe-outputs/outputs.jsonl

And the agent stdio log shows:

Custom steps execution completed
Shell cwd was reset to /home/runner/work/gh-aw/gh-aw

This indicates the agent ran and completed but chose not to use the safe-outputs MCP tools.

Workflow Configuration

• Staged Mode: true (GH_AW_SAFE_OUTPUTS_STAGED=true)
• Expected Outputs: create_issue (min: 1, max: 1)
• Safe-Outputs Path: /tmp/gh-aw/safe-outputs/agent_output.json

Failed Jobs and Errors

Job Sequence

1. ✅ activation - succeeded (3s)
2. ✅ agent - succeeded (51s)
3. ✅ detection - succeeded (22s)
4. ❌ create_issue - failed (6s)

Error Details

Error reading agent output file: ENOENT: no such file or directory, 
open '/tmp/gh-aw/safe-outputs/agent_output.json'

The create_issue job expects to find agent_output.json artifact, but it was never created because the agent didn't use safe-outputs MCP tools.

Investigation Findings

Why Did This Happen?

Possible Reasons:

1. Agent Interpretation: OpenCode may have interpreted the task differently and completed it without recognizing the need to use the create_issue tool
2. MCP Tool Availability: The safe-outputs MCP tools may not be properly registered or accessible to OpenCode
3. Staged Mode Behavior: Staged mode might affect how the agent decides to use tools
4. Prompt Ambiguity: The prompt may not be explicit enough that the safe-outputs tool MUST be used

Historical Context

This is a recurring pattern. Similar failure occurred in:

• Run #18715612738 (reported in issue [smoke-outpost] 🔍 Smoke Test Investigation - Smoke OpenCode: Missing agent_output.json File #2121, now closed)

The pattern database shows this is the second occurrence of OPENCODE_NO_SAFE_OUTPUTS pattern.

Comparison with Other Engines

Other smoke test workflows (GenAIScript, Claude) have different failure patterns:

• GenAIScript: Fails due to missing OPENAI_API_KEY (configuration issue)
• Claude: Generally more reliable with safe-outputs usage
• OpenCode: Completes successfully but doesn't use safe-outputs MCP

Recommended Actions

High Priority

• Investigate OpenCode safe-outputs MCP integration: Verify that OpenCode agent can see and use the safe-outputs MCP tools
• Review MCP tool registration: Check if safe-outputs MCP is properly configured in the OpenCode workflow
• Test with explicit tool instructions: Modify prompt to be more explicit: "You MUST use the create_issue tool from the safe-outputs MCP to post the summary"

Medium Priority

• Make create_issue job conditional: Change workflow to only run create_issue when agent_output.json artifact exists

  if: hashFiles('agent_output.json') != ''

• Add graceful handling: Modify create_issue job to log info message and skip (not fail) when output file is missing
• Add validation step: Insert intermediate job between agent and create_issue to validate safe-outputs were generated

Low Priority

• Add debug logging: Log available MCP tools in the agent job to confirm safe-outputs is loaded
• Enhance error messages: Provide clearer feedback when agent completes without using expected tools
• Create empty fallback: Consider creating empty agent_output.json if no safe-outputs were generated

Prevention Strategies

1. Conditional Job Execution: Only run downstream jobs when artifacts exist

   needs: agent
   if: needs.agent.outputs.output_types != ''

2. Validation Layer: Add intermediate job to check safe-outputs before proceeding:

   validation:
     needs: agent
     runs-on: ubuntu-latest
     outputs:
       has_outputs: ${{ steps.check.outputs.has_outputs }}
     steps:
       - name: Check for safe-outputs
         id: check
         run: |
           if [ -f "/tmp/gh-aw/safe-outputs/agent_output.json" ]; then
             echo "has_outputs=true" >> $GITHUB_OUTPUT
           else
             echo "has_outputs=false" >> $GITHUB_OUTPUT
           fi

3. Enhanced Prompting: Be more explicit in agent prompts:

   Review the last 5 merged pull requests and create an issue with the summary.
   
   IMPORTANT: You MUST use the 'create_issue' tool from the safe-outputs MCP 
   to create the issue. Do not use any other method.
   

4. Pre-flight MCP Check: Add step to verify MCP tools are available before running agent

Pattern Information

• Pattern ID: OPENCODE_NO_SAFE_OUTPUTS
• Category: Safe-Outputs Mechanism
• Severity: Medium
• Flakiness: Not flaky - consistent behavior
• Recurring: Yes (2 occurrences)

Related Issues

• [smoke-outpost] 🔍 Smoke Test Investigation - Smoke OpenCode: Missing agent_output.json File #2121 - Previous occurrence of this pattern (closed)

---

Investigation Metadata:

• Investigator: Smoke Detector (Outpost Agent)
• Investigation Run: #18722273071
• Pattern Database: /tmp/gh-aw/cache-memory/patterns/opencode_no_safe_outputs.json
• Investigation Record: /tmp/gh-aw/cache-memory/investigations/2025-10-22-18722224746.json

AI generated by Smoke Detector - Smoke Test Failure Investigator

[pelikhan]

/q review opencode on how to configure the mcp servers and update the steps

[github-actions]

Agentic Q triggered by this issue comment.

[github-actions]

🔧 Q Investigation Complete

I've investigated the OpenCode MCP configuration issue mentioned in your request and created a fix.

Problem Identified

The smoke-opencode workflow was failing because OpenCode had no access to MCP servers. Specifically:

• OpenCode requires MCP servers to be configured in ~/.config/opencode/opencode.json
• The current shared/opencode.md configuration didn't create this file
• Agent completed successfully but couldn't use safe-outputs MCP tools (e.g., create_issue)
• 4 consecutive runs failed with the same pattern (runs #18722224746, #18715612738, #18725510532, #18706992084)

Solution Implemented

Created PR with the following changes to .github/workflows/shared/opencode.md:

1. Added jq installation for JSON transformation
2. Added MCP configuration step that:
   • Creates ~/.config/opencode/opencode.json
   • Transforms gh-aw's Copilot-style MCP config to OpenCode format
   • Supports both local (stdio) and remote (HTTP) MCP servers
   • Logs configuration for debugging
3. Updated documentation with MCP integration details

Expected Results

After this fix:

• ✅ OpenCode will have access to safe-outputs MCP tools
• ✅ Agent can create issues using create_issue tool
• ✅ smoke-opencode workflow should succeed
• ✅ Any MCP server configured in workflow will be available to OpenCode

Testing

The fix has been:

• ✅ Compiled successfully (smoke-opencode.lock.yml generated)
• ✅ Validated against OpenCode documentation
• ⏳ Ready for workflow_dispatch testing

Pull request created with full implementation details and evidence from live logs.

AI generated by Q for #2143