Skip to content

feat(reporters): add secrets option to redact sensitive values - #43334

Merged
Dmitry Gozman (dgozman) merged 2 commits into
microsoft:mainfrom
dgozman:reporter-secrets
Oct 14, 2026
Merged

Dmitry Gozman (dgozman) merged 2 commits into
microsoft:mainfrom
dgozman:reporter-secrets

Conversation

@dgozman

Copy link
Copy Markdown
Collaborator

Summary

  • Any reporter accepts a secrets option, a map of name to value. Values are replaced with <secret>NAME</secret> in everything the reporter receives: titles, steps, errors, stdout/stderr, attachment bodies and config.
  • Redaction uses lazy views over the live reporter objects, so the object graph stays intact and reporters with the same secrets share one set of views.
  • Secrets are removed from config.reporter so they do not leak through config dumps.
  • The convenience line/dot reporter is redacted when all configured reporters have secrets.

References #38673

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

Comment thread docs/src/test-reporters-js.md Resolved
- secrets is a list of { name?, value }
- extract wrapReportersWithSecrets()
- drop config validation, use Map instead of WeakMap
@github-actions

Copy link
Copy Markdown
Contributor

Test results for "tests 1"

2 flaky ⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:257 › third party 'Partitioned;' cookies `@firefox-ubuntu-22.04-node22`
⚠️ [firefox-library] › library/browsercontext-cookies-third-party.spec.ts:470 › top level 'Partitioned;' cookie and same origin iframe `@firefox-ubuntu-22.04-node22`

21778 passed, 523 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Test results for "MCP"

1 failed
❌ [firefox] › mcp/idle-timeout.spec.ts:19 › closes the browser after the idle timeout and relaunches it on the next call @mcp-windows-latest-firefox

9203 passed, 1495 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Hi, I'm the Playwright bot and I took a first look at the CI failure.

🟢 The failure is a confirmed pre-existing flake

The PR is clear. The same Windows/Firefox idle-timeout test and error have failed repeatedly on main.

Details

The only real failure is in MCP browser lifecycle handling. This PR changes test reporter redaction and does not modify the MCP idle timer, browser backend, or Firefox lifecycle code.

Pre-existing flake / infra

  • [firefox] › mcp/idle-timeout.spec.ts:19 › closes the browser after the idle timeout and relaunches it on the next call — after the idle close, the next tool call fails with Target page, context or browser has been closed. The exact Windows/Firefox test failed in 13 of 94 runs in the aggregated results and passed in 81; 12 failures are on other PRs or SHAs. Seven are main runs, including this run at sha 65f80bbc950380c8dfbfea3d074f123f0ba53780, with the same error.

Triaged by the Playwright bot - agent run

@dgozman
Dmitry Gozman (dgozman) merged commit adf0514 into microsoft:main Oct 14, 2026
aragossa added a commit to pii-shield/trace-scrub that referenced this pull request Oct 23, 2026
…the stand

README: the maintainers did ship masking (microsoft/playwright#43334,
#43423, #43220); say what the native option covers, recommend it, and
what remains for this tool (values the server hands back, HAR,
storageState, older versions, recorded artifacts). Measured with
1.65.0-alpha-2026-10-23: trace.secrets with password+email 35 -> 16,
HAR and storageState unchanged; this tool on top 1 / 1 / 0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bruno Almeida (brunoccalmeida) added a commit to brunoccalmeida/playwright-qa-starter-kit that referenced this pull request Nov 17, 2026
Closes #43

## What

A failed `APIRequestContext` call writes every request and response
header (including `Cookie: token=…`) to its call log. Playwright records
that log in the step **before** a test can catch the error, so the auth
cookie reached the CI log, `report.jsonl`, the `.md` attachments of the
blob and the public HTML report.

- `src/reporting/redacting-reporter.ts`, listed **first** in
`playwright.config.ts`: masks every header outside a short allowlist
(`HARMLESS_HEADERS`) in the `→`/`←` blocks of the call log, in errors
(`message`, `stack`, `value`, `cause`), steps, `onError` and text
attachments.
- **Fail closed:** if an attachment cannot be read or rewritten, the
**file** is removed from disk. The blob reporter records the path on
`onStepEnd` and zips whatever file exists at `onEnd`, so dropping the
list entry alone is not enough (reviewer finding, fixed in the second
commit).
- Sentinel `scripts/check-report-redaction.ts` (last step of `npm run
check`): runs failing requests against a local server through the
**real** config's reporters in CI mode and searches the terminal output,
the whole blob and the output dir for a random token.

## Why a reporter

The origin (Playwright's fetch call log) has no option to hide headers,
and the reporter `secrets` option was reverted upstream
(microsoft/playwright#43334, #43870; tracked in #21). Mutating reporter
objects is undocumented, which is why the sentinel exists: a Playwright
upgrade that breaks it turns `check` red instead of leaking in silence.

Sources: [OWASP Logging Cheat
Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html),
[OpenTelemetry redaction
processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/redactionprocessor)
(allowlist, fail closed), [Playwright Reporter
API](https://playwright.dev/docs/api/class-reporter).

## RED

1. Probe: closed port with a fake cookie → `Cookie:
token=FAKE-SENTINEL-VALUE` in output and JSON.
2. Test before the reporter existed: `ERR_MODULE_NOT_FOUND`.
3. Reporter out of the config → "The token reached the terminal output",
`report.jsonl` and the blob's `.md` attachments.
4. 16 of 16 mutations rejected from a green baseline.
5. Real merged HTML report: 0 token hits, 37 `[REDACTED]`.
6. Reviewer finding: a read-only file holding the token, against the old
fail-closed code → "An attachment that could not be cleaned stayed on
disk, where the blob reporter zips it." Mutation without the `catch`
around `rmSync` → `ERR_FS_EISDIR`.

## Review

Independent reviewer run on the diff against #43: 4 findings, all valid.
Blob fail-closed: fixed. Session record wording: fixed. Test
stdout/stderr (no `onStdOut`/`onStdErr`; no `console.` in the kit today)
and attachments with `body` or a non-text type: recorded as scope limits
in the session summary.

## Out of scope

Values in URLs and response bodies; trace stays off in CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants