Snapshot sweep passes warehouse-source, replay-vision, self-driving and audit Live Data without running them, because the e2e key lacks their scopes

[gewenyu99]

Problem

The real-TUI snapshot sweep runs each program with one PostHog personal API key. That key does not carry the scopes several programs need. The PostHog MCP hides or rejects the gated tools. The agent then records a blocker, skips the task, or marks a check as a suggestion. The run still exits 0 with runPhase: completed, so the sweep reports a pass.

The result is that these paths have no end-to-end coverage today:

• warehouse-source never creates a source. It needs external_data_source:read and external_data_source:write.
• replay-vision never enables replay or creates a scanner. It needs product_enablement:write, replay_scanner:read and replay_scanner:write.
• self-driving never enables Support or creates a scanner. It needs the same three scopes.
• audit never reads live health findings. It needs health_issue:read, which is already in the documented base set.

Nothing in the harness notices this. In CI mode the wizard sets missingScopes to an empty list, because a personal API key has no requested scope set to diff against. The scope-aware degrade and the error copy that name a missing permission never fire in e2e.

The docs do not help either. The README lists the base set plus the integration additions. It does not list what replay-vision or self-driving add. The harness docs name the key variables but no scopes.

This issue is about coverage and the key. Two related problems are tracked apart: the outros that still say success when a scope blocked the work (a comment on #951), and the harness never failing a run on a missing report or a host exit code (the snapshot-harness issue). Fix 3 below is the scope-specific part of the second.

Evidence

Sweep on the refactor stack head d938a43 (#1307), with the key read through POSTHOG_KEY_FILE against the sweep's US test project. The sweep of the Release A top 200961f (#1303) shows the same results, and its notes say main behaves the same.

• warehouse-source, app revenue/stripe/stripe-saas-demo. Frame 12 says "MCP source creation is unavailable under the active connection's scopes." The task list ends with "Record blocker" completed. The outro frame still says "Data warehouse source connected!" The Release A run's report says it created 0 of 1 detected sources because external_data_source:read and external_data_source:write were missing. Exit 0.
• replay-vision, app replay-vision/react-vite/court-booking. The report says products-enable was rejected for lack of product_enablement:write. It also says the connection has no replay_scanner:read or replay_scanner:write, so the scanner list, estimate, quota and create tools were unreachable. All three scanner tasks are skipped. The outro reads "4/4 steps completed (3 skipped as not required)." Exit 0.
• self-driving, app self-driving/react-router/expense-splitter. All nine tasks read completed. The report says Support could not be enabled without product_enablement:write, and both Replay Vision monitors were skipped without replay_scanner:read and replay_scanner:write. Exit 0.
• audit, app audit/posthog-demo-3000. The live-data-findings check in .posthog-audit-checks.json has status suggestion and says: "Skipped: health-issues-list requires the health_issue:read scope." Exit 0.

The code is the same on main (d8486dc):

• src/shared/utils/setup-utils.ts:448 takes the CI api-key path. Lines 500-502 return missingScopes: [].
• src/shared/constants.ts:283-289 defines WIZARD_OAUTH_SCOPES. Line 285 is health_issue:read.
• src/lib/oauth/program-scopes.ts:180-196 (SELF_DRIVING_SCOPE_ADDITIONS), :206-209 (WAREHOUSE_SOURCE_SCOPE_ADDITIONS) and :246-251 (REPLAY_VISION_SCOPE_ADDITIONS) list what each program needs. :264 maps them per program, and :296 getOAuthScopesForProgram resolves the union for one program.
• README.md:263-278 ("Required API Key Scopes") lists the base set and only the integration additions.
• scripts/tui-host.no-jest.ts:194-198 reads the key, and :227 builds the session with ci: true. e2e-harness/ARCHITECTURE.md:46-48 and scripts/README.md name the key variables but no scopes.
• src/lib/programs/replay-vision/test/e2e.json describes the happy path as enabling replay and creating three scanners, and warehouse-source/test/e2e.json as creating in-cli sources over the PostHog MCP. The sweep reaches neither.

The key is not in any repo. The same kind of key is stored as the GH_APP_POSTHOG_WIZARD_CI_BOT_POSTHOG_PERSONAL_KEY secret, used by .github/workflows/smoke-test.yml here and by .github/workflows/wizard-ci.yml in PostHog/wizard-workbench. This sweep did not check whether that secret has the same scopes.

Suggested fix

1. Document the scopes the e2e key needs for each program. Generate the list from getOAuthScopesForProgram for every program id, so it cannot drift from PROGRAM_SCOPE_ADDITIONS. Put the union in the README "Required API Key Scopes" section and link it from e2e-harness/ARCHITECTURE.md and scripts/README.md.
2. Issue a new CI bot key with that union. Rotate GH_APP_POSTHOG_WIZARD_CI_BOT_POSTHOG_PERSONAL_KEY in both repos and replace the local key file the sweep uses.
3. Make the harness flag steps blocked by a scope. Before the run, tui-host should compare what the key can do with getOAuthScopesForProgram(programId). Read the key's scopes if PostHog exposes them. If not, check that the MCP tool list served to the key includes the tools the program needs. Put the gap in the E2E_RESULT_JSON payload as missingScopes, and make the sweep report the program as scope-blocked, not passed. The same diff can fill credentials.missingScopes on the CI path, so CI runs get the missing-permission error copy that OAuth runs already get.