Skip to content

Audit test contracts, stabilize fixtures, and fix uncovered defects - #487

Merged
Tryanks merged 8 commits into
mainfrom
codex/test-contract-audit
Sep 20, 2026
Merged

Tryanks merged 8 commits into
mainfrom
codex/test-contract-audit

Conversation

@Tryanks

@Tryanks Tryanks commented Sep 20, 2026 •

Copy link
Copy Markdown
Owner

CI tests mixed meaningful product contracts with upstream checks, overlapping one-off cases and fixtures that guessed when background work had finished. This audits the repository-owned suite, removes redundant coverage, merges related scenarios, and makes asynchronous assertions wait for observable state.

Rust source tests go from 1,275 to 916 (359 fewer, 28.2%); Browser-auth tests go from 3 to 2. Counts include platform-gated and explicitly ignored tests, so they are not per-run execution totals. Each surviving source test was reviewed against its implementation and recorded with a concrete retention reason; removed/merged names have coverage mappings.

What remains covered

  • Provider protocol translation uses literal wire/legacy-storage fixtures, authorization matrices and real actor boundaries. Native adapters no longer require an environment cat or sh for their test fixtures.
  • Runtime matrices cover delivery/acknowledgement, active and parked sessions, stale startup/event completions, native subagent mirrors, orchestration scope, persistence barriers and historical cursors. Small getter and repeated launch/shutdown checks moved into these behavioral cases.
  • UI scenarios cover actual controls, focus/navigation, draft reuse, host baselines and entity identity. Locale changes are serialized by the existing test guard; settings and project-choice tests wait for the corresponding snapshot rather than an arbitrary number of executor pumps.
  • Terminal replication covers late attachment, multiple writers, resize, modes, retained history and burst publication. Explicit child-output handshakes and controlled projection timestamps replace sleeps and upstream parser-fragmentation assertions.
  • Remote tests retain authenticated identity-before-token, rejection/recovery, paired proxy routing, persistence and lifecycle contracts. Short silence checks and a timeout assertion only 250 ms above the production deadline were removed.
  • Upstream image/Markdown/terminal primitives, constant/getter checks and same-contract happy-path fragments were deleted or absorbed. Dependency/process-boundary and literal protocol checks remain.

A Windows nextest leak warning also exposed a shared Markdown fixture that returned before its detached Git status probe finished. A two-second slow-Git fixture made the old cleanup contract fail after 0.58 seconds; waiting for the actual Git status callback made it pass after 2.60 seconds. The helper now waits for that completion and shuts its host down on GPUI teardown. The temporary probe assertion was removed; all 28 chat tests and 200 targeted repetitions passed. The final Windows run passed all 878 tests with no LEAK warnings; Linux and macOS also reported none.

Production defects exposed by broader regressions

Defect Correction and evidence
Unix PTY reported raw waitpid bits, e.g. exit 37 as 9472 Normalize using ExitStatus; real PTY input/output/exit regression failed before the fix and passed afterward. Signal termination remains None.
Huge /later duration panicked inside chrono Checked duration construction plus existing checked date addition; valid schedules and invalid/overflow boundaries share one test.
Claude ReadOnly authorized tool-name substrings Restrict automatic grants to exact native Read/Glob/Grep/WebSearch; actual can_use_tool matrix rejects MCP/lookalike names. Display heuristics no longer authorize execution.
Orchestrate report exemption matched any name containing tcode_report Require Claude's exact qualified mcp__tcode_report__report_result name and approve only that request. Regression matrices fail on lookalike names, other providers and session-wide approval. Claude wire tests show single approval does not forward permission suggestions.
ACP command symlink could resolve outside the installed archive and be chmodded Reject rooted/prefixed paths and require canonical containment plus a file before chmod. Internal symlinks still work; an external target is rejected with its permissions unchanged.

These fixes stay in the existing owners; no new runtime abstraction or dependency was introduced. ACP containment is command-path validation, not a sandbox for downloaded agent code or a defense against concurrent local filesystem replacement. Unknown Claude tools now require normal approval. Providers without Claude's authoritative tool namespace keep their normal approval path.

Validation

Final commit: 123e122b (includes main’s CI simplification from #488). The obsolete Python planner and its tests were removed upstream; that deletion is not counted as this PR’s test reduction.

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo build --workspace --locked — passed on the final commit.
  • cargo nextest run --workspace --locked — 900 passed, 0 failed, 6 explicitly skipped (five libtest entries plus the native-overlay harness), no LEAK warnings.
  • cargo test -p gpui-android --lib --locked — 4 pure input tests passed; this excluded workspace package was run explicitly.
  • cargo-machete 0.9.2 — passed.
  • node --test crates/web/static/auth.test.mjs — 2 passed.
  • 450/450 targeted repetitions passed: Markdown host cleanup (2 × 100), actor cleanup (100), project choice (20), touch copy (50), terminal replication (4 × 20).
  • Opt-in foreground PTY cwd probe — passed after requiring a real / → /tmp transition; native overlay/live registry/credential-dependent usage/manual history benchmark probes remain unrun.
  • Final macOS, Windows, Linux, dependency, iOS, Android and Web CI — all seven jobs passed on the exact final commit. Linux: 886 passed / 6 skipped; macOS: 900 passed / 6 skipped; Windows: 878 passed / 4 skipped. No LEAK warnings on any desktop platform.

Independent reviews: Standards: no actionable findings, including the merge with #488 and final fixture cleanup. The connection-stamp test verifies final field preservation but does not prove a specific scheduler interleaving. Spec: no missing requirements, coverage regressions or scope violations found in the reviewed diff.

Remaining ignored desktop/live-service/manual benchmark cases were reviewed but are not claimed as executed. The ignored child-host entry is a subprocess fixture exercised by its parent liveness test; the foreground-cwd probe was run separately. No provider credentials or live accounts are required by the normal suite. Native GUI permission/input/WebView behavior still needs its opt-in platform probes. The local macOS debug link emits the existing large-unwind-table linker warning; strict Clippy passes.

@Tryanks Tryanks changed the title Audit test contracts and remove timing-dependent failures Audit test contracts, stabilize fixtures, and fix uncovered defects Sep 20, 2026
@Tryanks
Tryanks marked this pull request as ready for review September 20, 2026 17:11
@Tryanks
Tryanks merged commit cc04ebe into main Sep 20, 2026
7 checks passed
@Tryanks
Tryanks deleted the codex/test-contract-audit branch September 20, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant