[aw-failures] Smoke CI hard-red at startup — `EACCES mkdir /tmp/gh-aw/sandbox/firewall/logs`, agent never invoked (rootless left [Content truncated due to length]

[github-actions]

Reclaim the rootless /tmp/gh-aw/sandbox tree before writeConfigs() — a leftover root-owned dir makes mkdir /tmp/gh-aw/sandbox/firewall/logs fail EACCES and kills Smoke CI at startup before the agent is ever invoked.

This is a NEW, untracked P1 hard-red. It is distinct from #41455 (firewall startup via DNS EAI_AGAIN), #41636 (Copilot CLI exit-1 after safe-outputs succeed), and #41885 (Claude parse step on empty logEntries). Those are DNS races or post-completion false-reds; this one fails pre-flight — the agent never runs, so the run is 100% lost.

Problem statement

Make the AWF sandbox bootstrap resilient to a pre-existing root-owned /tmp/gh-aw/sandbox left by a prior rootless container on the same runner. Today, the very first config-generation step dies:

[INFO] Network-isolation mode: enforcing egress via Docker network topology (no host iptables, no sudo).
[INFO] Generating configuration files...
[ERROR] Fatal error: Error: EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'
    at Object.mkdirSync (node:fs:1370:26)
    ... at Object.RM [as writeConfigs] (/home/runner/.local/lib/awf/awf-bundle.js:786:1941)
[WARN] Could not fix squid log permissions: Error: Command failed ... chmod -R a+rX /tmp/gh-aw/sandbox/firewall/logs
chmod: cannot access '/tmp/gh-aw/sandbox/firewall/logs': Permission denied
Process exiting with code: 1
##[error]Process completed with exit code 1.

The recovery path (chmod -R a+rX) also fails Permission denied, so there is no escape hatch — the run aborts hard.

Affected workflows and run IDs

Workflow	Engine	Failed run	Nearest success (same config)
Smoke CI (.github/workflows/smoke-ci.lock.yml)	copilot	§28413001230 (01:00:29Z)	§28413042897 (01:01:35Z, ~1 min later)

A nearly-identical Smoke CI run succeeded ~1 minute later on the same config → this is a per-runner ownership race, not a configuration error.

Evidence

audit-diff: failed §28413001230 vs success §28413042897

{
  "firewall_diff": { "summary": { "has_anomalies": false, "anomaly_count": 0 } },
  "run_metrics_diff": {
    "run1_token_usage": 0, "run2_token_usage": 0,
    "github_rate_limit_details": { "run1_total_api_calls": 11, "run1_core_consumed": 65 }
  }
}

• run1_token_usage: 0 — the agent was never invoked; the job died during "Generating configuration files".
• has_anomalies: false — no firewall/egress divergence; the firewall config was never written because mkdir failed first. The discriminator is purely the pre-flight mkdir EACCES.

Probable root cause

1. A prior rootless container on the same runner leaves /tmp/gh-aw/sandbox (the parent of firewall/logs) owned by a uid the current runner user cannot write — the same residue surfaced as "Rootless artifact permission repair failed (exit 1)" in [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636 and [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885.
2. writeConfigs() calls mkdirSync('/tmp/gh-aw/sandbox/firewall/logs') without first ensuring the tree is owned by / writable for the current uid → EACCES.
3. The chmod -R a+rX fallback cannot touch the root-owned dir, so the bootstrap fatally exits 1 instead of repairing or relocating.

This is the pre-flight twin of #41885's post-teardown rootless-ownership failure: same root cause (rootless leaves root-owned /tmp/gh-aw/sandbox), opposite end of the run lifecycle.

Proposed remediation

1. Primary: before writeConfigs(), reclaim the sandbox tree for the current uid — rm -rf /tmp/gh-aw/sandbox (or rootless chown via podman unshare) when a pre-existing root-owned residue is detected — then mkdir. A fresh, uid-owned tree eliminates the race.
2. Resilience: on mkdir EACCES under /tmp/gh-aw/sandbox, attempt ownership repair or fall back to a fresh uid-scoped temp dir instead of fatal-exiting.
3. Root fix (shared with [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636/[aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885): make post-run cleanup reliably remove root-owned sandbox residue — the recurring "Rootless artifact permission repair failed (exit 1)" warnings prove teardown is not reclaiming these dirs, so the next run inherits them.

Success criteria / verification

• Smoke CI no longer aborts with EACCES: mkdir '/tmp/gh-aw/sandbox/firewall/logs'.
• A runner seeded with a leftover root-owned /tmp/gh-aw/sandbox from a prior rootless run still starts the firewall and invokes the agent.
• A teardown assertion confirms /tmp/gh-aw/sandbox is fully removed (no root-owned residue) after each run.

Existing-issue correlation

• Untracked. Not covered by [aw-failures] AWF firewall startup fails — getaddrinfo EAI_AGAIN awmg-cli-proxy, managed-gateway DNS not resolvable, agent never [Content truncated due to length] #41455 (DNS EAI_AGAIN), [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636 (Copilot exit-1 after outputs succeed), or [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885 (Claude empty-logEntries parse). Shares the rootless-ownership root-cause family with [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885 remediation Add workflow: githubnext/agentics/weekly-research #2 but fires pre-flight (agent never invoked) rather than post-teardown.
• This window also re-confirmed [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636 (PR Sous Chef Copilot exit-1: §28413646286, §28411106648, §28409821210) and [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885 (Avenger parse step: §28410063636) are still actively recurring — both kept open.

Analyzed run IDs: 28413001230 (representative), comparator 28413042897.

References: §28413001230 · §28413042897

Generated by 🔍 [aw] Failure Investigator (6h) · 153.3 AIC · ⌖ 38 AIC · ⊞ 5.6K · ◷

• expires on Jul 6, 2026, 5:40 PM UTC-08:00

---

⏫ RECURRENCE — reopened by Failure Investigator (window ending 2026-07-01 01:35 UTC)

Reopen and actually fix the writeConfigs() pre-flight mkdir — this P1 hard-red recurred 2h38m after the issue was closed not planned. Smoke CI is still lost pre-flight; the agent never runs.

• New failed run: §28487200601 — Smoke CI, agent job, step Execute GitHub Copilot CLI, created 2026-07-01T01:28:59Z.
• Timeline: closed NOT_PLANNED at 2026-06-30T22:50:42Z → recurred ~2h38m later with a byte-identical signature. Closing without a fix did not stop it.
• Identical signature:

  [INFO] Generating configuration files...
  [ERROR] Fatal error: Error: EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'
      at Object.XM [as writeConfigs] (/home/runner/.local/lib/awf/awf-bundle.js:786:1941)
  [WARN] Could not fix squid log permissions: ... chmod -R a+rX /tmp/gh-aw/sandbox/firewall/logs
  chmod: cannot access '/tmp/gh-aw/sandbox/firewall/logs': Permission denied
  Process exiting with code: 1
  

• Regression suspect: HEAD commit 9c4a9e7 — "Align sandbox integration tests with rootless AWF command default (Align sandbox integration tests with rootless AWF command default #42606)" (merged 2026-06-30) — changed rootless AWF command defaults. The pre-flight mkdir EACCES is precisely the rootless-ownership failure this issue documents; the remediation here (reclaim/relocate /tmp/gh-aw/sandbox before writeConfigs(), then assert teardown removed it) still applies unchanged.
• This window's other clusters (already covered, no new issue needed): Auto-Triage Issues --network-isolation requires a reachable Docker daemon (§28487162711) → ARC/DinD [ARC-DinD] ARC/DinD support in v0.75.4 still requires workflow-level workarounds #34896; PR Sous Chef Copilot CLI exit-1 with outputs already set (§28484493629, §28483351422) → false-red family [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636; Chaos PR Bundle Fuzzer safe_outputs "Restore actions folder" checkout fail with the agent job succeeding (§28486539819) → P2 post-agent infra false-red.

Do not re-close without a teardown assertion that /tmp/gh-aw/sandbox is fully removed after each run.

---

Original report preserved below.

Reclaim the rootless /tmp/gh-aw/sandbox tree before writeConfigs() — a leftover root-owned dir makes mkdir /tmp/gh-aw/sandbox/firewall/logs fail EACCES and kills Smoke CI at startup before the agent is ever invoked.

This is a NEW, untracked P1 hard-red. It is distinct from #41455 (firewall startup via DNS EAI_AGAIN), #41636 (Copilot CLI exit-1 after safe-outputs succeed), and #41885 (Claude parse step on empty logEntries). Those are DNS races or post-completion false-reds; this one fails pre-flight — the agent never runs, so the run is 100% lost.

Problem statement

Make the AWF sandbox bootstrap resilient to a pre-existing root-owned /tmp/gh-aw/sandbox left by a prior rootless container on the same runner. Today, the very first config-generation step dies:

[INFO] Network-isolation mode: enforcing egress via Docker network topology (no host iptables, no sudo).
[INFO] Generating configuration files...
[ERROR] Fatal error: Error: EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'
    at Object.mkdirSync (node:fs:1370:26)
    ... at Object.RM [as writeConfigs] (/home/runner/.local/lib/awf/awf-bundle.js:786:1941)
[WARN] Could not fix squid log permissions: Error: Command failed ... chmod -R a+rX /tmp/gh-aw/sandbox/firewall/logs
chmod: cannot access '/tmp/gh-aw/sandbox/firewall/logs': Permission denied
Process exiting with code: 1
##[error]Process completed with exit code 1.

The recovery path (chmod -R a+rX) also fails Permission denied, so there is no escape hatch — the run aborts hard.

Affected workflows and run IDs

Workflow	Engine	Failed run	Nearest success (same config)
Smoke CI (.github/workflows/smoke-ci.lock.yml)	copilot	§28413001230 (01:00:29Z)	§28413042897 (01:01:35Z, ~1 min later)

A nearly-identical Smoke CI run succeeded ~1 minute later on the same config → this is a per-runner ownership race, not a configuration error.

Evidence

audit-diff: failed §28413001230 vs success §28413042897

{
  "firewall_diff": { "summary": { "has_anomalies": false, "anomaly_count": 0 } },
  "run_metrics_diff": {
    "run1_token_usage": 0, "run2_token_usage": 0,
    "github_rate_limit_details": { "run1_total_api_calls": 11, "run1_core_consumed": 65 }
  }
}

• run1_token_usage: 0 — the agent was never invoked; the job died during "Generating configuration files".
• has_anomalies: false — no firewall/egress divergence; the firewall config was never written because mkdir failed first. The discriminator is purely the pre-flight mkdir EACCES.

Probable root cause

1. A prior rootless container on the same runner leaves /tmp/gh-aw/sandbox (the parent of firewall/logs) owned by a uid the current runner user cannot write — the same residue surfaced as "Rootless artifact permission repair failed (exit 1)" in [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636 and [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885.
2. writeConfigs() calls mkdirSync('/tmp/gh-aw/sandbox/firewall/logs') without first ensuring the tree is owned by / writable for the current uid → EACCES.
3. The chmod -R a+rX fallback cannot touch the root-owned dir, so the bootstrap fatally exits 1 instead of repairing or relocating.

This is the pre-flight twin of #41885's post-teardown rootless-ownership failure: same root cause (rootless leaves root-owned /tmp/gh-aw/sandbox), opposite end of the run lifecycle.

Proposed remediation

1. Primary: before writeConfigs(), reclaim the sandbox tree for the current uid — rm -rf /tmp/gh-aw/sandbox (or rootless chown via podman unshare) when a pre-existing root-owned residue is detected — then mkdir. A fresh, uid-owned tree eliminates the race.
2. Resilience: on mkdir EACCES under /tmp/gh-aw/sandbox, attempt ownership repair or fall back to a fresh uid-scoped temp dir instead of fatal-exiting.
3. Root fix (shared with [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636/[aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885): make post-run cleanup reliably remove root-owned sandbox residue — the recurring "Rootless artifact permission repair failed (exit 1)" warnings prove teardown is not reclaiming these dirs, so the next run inherits them.

Success criteria / verification

• Smoke CI no longer aborts with EACCES: mkdir '/tmp/gh-aw/sandbox/firewall/logs'.
• A runner seeded with a leftover root-owned /tmp/gh-aw/sandbox from a prior rootless run still starts the firewall and invokes the agent.
• A teardown assertion confirms /tmp/gh-aw/sandbox is fully removed (no root-owned residue) after each run.

Existing-issue correlation

• Untracked. Not covered by [aw-failures] AWF firewall startup fails — getaddrinfo EAI_AGAIN awmg-cli-proxy, managed-gateway DNS not resolvable, agent never [Content truncated due to length] #41455 (DNS EAI_AGAIN), [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636 (Copilot exit-1 after outputs succeed), or [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885 (Claude empty-logEntries parse). Shares the rootless-ownership root-cause family with [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885 remediation Add workflow: githubnext/agentics/weekly-research #2 but fires pre-flight (agent never invoked) rather than post-teardown.
• This window also re-confirmed [aw-failures] Copilot CLI false-red — runs marked failure (exit 1) after safe-outputs succeed, via "numerous permission denied" [Content truncated due to length] #41636 (PR Sous Chef Copilot exit-1: §28413646286, §28411106648, §28409821210) and [aw-failures] Claude false-red — log_parser_bootstrap fails completed runs on empty logEntries (Avenger, Daily Rendering Scripts [Content truncated due to length] #41885 (Avenger parse step: §28410063636) are still actively recurring — both kept open.

Analyzed run IDs: 28413001230 (representative), comparator 28413042897.

References: §28413001230 · §28413042897

Generated by 🔍 [aw] Failure Investigator (6h) · 141 AIC · ⌖ 20.4 AIC · ⊞ 5.2K · ◷

---

---

🔁 Regression — reopening (auto-investigator, 6h window ending 2026-07-07T19:24Z)

Reopen: this is not fixed. #42398 was closed 2026-07-07T02:25:43Z, but the identical EACCES mkdir /tmp/gh-aw/sandbox/firewall/logs pre-flight failure recurred 14 hours after the close.

Fresh evidence — [§28883619240 is unrelated; confirmed on] Smoke CI §28883152120 (created 2026-07-07T16:45:42Z, engine copilot, firewall v0.27.26, model claude-sonnet-4.6): agent step exits code 1 with 0 turns / 0 tokens / 0 GitHub API inference calls — the agent is never invoked, run 100% lost. agent-stdio.log is byte-identical to the original report:

[INFO] Network-isolation mode: enforcing egress via Docker network topology (no host iptables, no sudo).
[INFO] Generating configuration files...
[ERROR] Fatal error: Error: EACCES: permission denied, mkdir '/tmp/gh-aw/sandbox/firewall/logs'
    at Object.mkdirSync (node:fs:1370:26)
    ... at Object.Zj [as writeConfigs] (/home/runner/.local/lib/awf/awf-bundle.js:786:1941)
[WARN] Could not fix squid log permissions: chmod: cannot access '/tmp/gh-aw/sandbox/firewall/logs': Permission denied
Process exiting with code: 1

The proposed fix (reclaim the root-owned /tmp/gh-aw/sandbox tree before writeConfigs()) is still the correct remediation and has not landed / did not hold. Smoke CI remains a recurring red in the 6h window (5 failures + 3 guardrail cancellations). Re-verify the fix actually reclaims a pre-existing root-owned /tmp/gh-aw/sandbox/firewall (not just /tmp/gh-aw/sandbox) and add a smoke assertion that the dir is agent-writable before writeConfigs().

Success criteria: 24h of Smoke CI with zero EACCES .../firewall/logs; agent reaches ≥1 turn on every Smoke CI run.

Generated by 🔍 [aw] Failure Investigator (6h) · 190.5 AIC · ⌖ 46.5 AIC · ⊞ 5.2K · ◷

[github-actions]

🍪 Issue Monster selected this for Copilot

I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · 3.27 AIC · ⌖ 9.24 AIC · ⊞ 6.1K · ◷

[github-actions]

This issue was automatically closed because it expired on 2026-07-07T01:40:49.499Z.

Closed by Workflow

[github-actions]

This issue was automatically closed because it expired on 2026-07-07T01:40:49.499Z.

Closed by Workflow