Skip to content

Commit e57ce01

Browse files
fix(reporter): remove reporter-only-failures CLI flag (#43805)
Co-authored-by: microsoft-playwright-automation[bot] <203992400+microsoft-playwright-automation[bot]@users.noreply.github.com>
1 parent 8d30912 commit e57ce01

14 files changed

Lines changed: 41 additions & 60 deletions

File tree

‎docs/src/test-cli-js.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,6 @@ npx playwright test --ui
103103
| `--quiet` | Suppress stdio. |
104104
| `--repeat-each <N>` | Run each test `N` times (default: 1). |
105105
| `--reporter <reporter>` | Reporter to use, comma-separated, can be "dot", "line", "list", "failures", or others (default: "list" locally and "dot" on CI). You can also pass a path to a custom reporter file. |
106-
| `--reporter-only-failures` | Include only failed, flaky, and interrupted tests in JSON, JUnit, and HTML reports. |
107106
| `--retries <retries>` | Maximum retry count for flaky tests, zero for no retries (default: no retries). |
108107
| `--run-agents <mode>` | Run agents to generate the code for `page.perform`. Possible values are "missing", "all" and "none" (default: "none"). See [test-agents](./test-agents.md). |
109108
| `--shard <shard>` | Shard tests and execute only the selected shard, specified in the form "current/all", 1-based, e.g., "3/5". |
@@ -325,7 +324,6 @@ npx playwright merge-reports ./reports
325324
| :--- | :--- |
326325
| `-c, --config <file>` | Configuration file. Can be used to specify additional configuration for the output report |
327326
| `--reporter <reporter>` | Reporter to use, comma-separated, can be "list", "line", "dot", "failures", "json", "junit", "null", "github", "html", "blob", "perfetto" (default: "list" locally and "dot" on CI) |
328-
| `--reporter-only-failures` | Include only failed, flaky, and interrupted tests in JSON, JUnit, and HTML reports. |
329327

330328
### Clear Cache
331329

‎docs/src/test-reporter-api/class-reporter.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,6 @@ export default defineConfig({
7474
});
7575
```
7676

77-
When `--reporter-only-failures` is set, custom reporters receive `onlyFailures: true` in their constructor options.
78-
7977
Here is a typical order of reporter calls:
8078
* [`method: Reporter.onBegin`] is called once with a root suite that contains all other suites and tests. Learn more about [suites hierarchy][Suite].
8179
* [`method: Reporter.onTestBegin`] is called for each test run. It is given a [TestCase] that is executed, and a [TestResult] that is almost empty. Test result will be populated while the test runs (for example, with steps and stdio) and will get final `status` once the test finishes.

‎docs/src/test-reporters-js.md‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ export default defineConfig({
5252

5353
### Report only failures
5454

55-
Use `onlyFailures` with the JSON, JUnit, or HTML reporter to include only failed, flaky, and interrupted tests. Every attempt of an included test is retained, including successful retries:
55+
Use `onlyFailures` with the JSON, JUnit, or HTML reporter to include only tests with unexpected outcomes (including unexpected passes), flaky tests, and interrupted tests. Flaky tests are included even when they pass on retry. Every attempt of an included test is retained, including successful retries:
5656

5757
```js title="playwright.config.ts"
5858
import { defineConfig } from '@playwright/test';
@@ -62,12 +62,7 @@ export default defineConfig({
6262
});
6363
```
6464

65-
The command line flag enables this option for all configured reporters that support it:
66-
67-
```bash
68-
npx playwright test --reporter=failures,html --reporter-only-failures
69-
npx playwright merge-reports --reporter=html --reporter-only-failures ./blob-report
70-
```
65+
Configure `onlyFailures` independently for each reporter. To use it when merging blob reports, pass the configuration file to `npx playwright merge-reports --config=playwright.config.ts ./blob-report`.
7166

7267
Use the `failures` reporter for live terminal diagnostics without progress output. For example, retain a complete HTML report alongside concise terminal output:
7368

@@ -355,6 +350,7 @@ HTML report supports the following configuration options and environment variabl
355350
| Environment Variable Name | Reporter Config Option| Description | Default
356351
|---|---|---|---|
357352
| `PLAYWRIGHT_HTML_TITLE` | `title` | A title to display in the generated report. | No title is displayed by default
353+
| `PLAYWRIGHT_HTML_ONLY_FAILURES` | `onlyFailures` | Include only tests with unexpected outcomes, flaky tests (including successful retries), and interrupted tests. See [Report only failures](#report-only-failures). | `false`
358354
| `PLAYWRIGHT_HTML_OUTPUT_DIR` | `outputFolder` | Directory to save the report to. | `playwright-report`
359355
| `PLAYWRIGHT_HTML_OPEN` | `open` | When to open the html report in the browser, one of `'always'`, `'never'` or `'on-failure'` | `'on-failure'`
360356
| `PLAYWRIGHT_HTML_HOST` | `host` | When report opens in the browser, it will be served bound to this hostname. | `localhost`
@@ -460,6 +456,7 @@ JSON report supports following configuration options and environment variables:
460456
| Environment Variable Name | Reporter Config Option| Description | Default
461457
|---|---|---|---|
462458
| `PLAYWRIGHT_JSON_OUTPUT_DIR` | | Directory to save the output file. Ignored if output file is specified. | `cwd` or config directory.
459+
| `PLAYWRIGHT_JSON_ONLY_FAILURES` | `onlyFailures` | Include only tests with unexpected outcomes, flaky tests (including successful retries), and interrupted tests. See [Report only failures](#report-only-failures). | `false`
463460
| `PLAYWRIGHT_JSON_OUTPUT_NAME` | | Base file name for the output, relative to the output dir. | JSON report is printed to the stdout.
464461
| `PLAYWRIGHT_JSON_OUTPUT_FILE` | `outputFile` | Full path to the output file. If defined, `PLAYWRIGHT_JSON_OUTPUT_DIR` and `PLAYWRIGHT_JSON_OUTPUT_NAME` will be ignored. | JSON report is printed to the stdout.
465462

@@ -498,6 +495,7 @@ JUnit report supports following configuration options and environment variables:
498495
| Environment Variable Name | Reporter Config Option| Description | Default
499496
|---|---|---|---|
500497
| `PLAYWRIGHT_JUNIT_OUTPUT_DIR` | | Directory to save the output file. Ignored if output file is specified. | `cwd` or config directory.
498+
| `PLAYWRIGHT_JUNIT_ONLY_FAILURES` | `onlyFailures` | Include only tests with unexpected outcomes, flaky tests (including successful retries), and interrupted tests. Retains every attempt regardless of `includeRetries`. See [Report only failures](#report-only-failures). | `false`
501499
| `PLAYWRIGHT_JUNIT_OUTPUT_NAME` | | Base file name for the output, relative to the output dir. | JUnit report is printed to the stdout.
502500
| `PLAYWRIGHT_JUNIT_OUTPUT_FILE` | `outputFile` | Full path to the output file. If defined, `PLAYWRIGHT_JUNIT_OUTPUT_DIR` and `PLAYWRIGHT_JUNIT_OUTPUT_NAME` will be ignored. | JUnit report is printed to the stdout.
503501
| `PLAYWRIGHT_JUNIT_STRIP_ANSI` | `stripANSIControlSequences` | Whether to remove ANSI control sequences from the text before writing it in the report. | By default output text is added as is.

‎packages/playwright/src/cli/reportActions.ts‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,7 @@ export async function showReport(report: string | undefined, host: string, port:
2929

3030
export async function mergeReports(reportDir: string | undefined, opts: { [key: string]: any }) {
3131
const configFile = opts.config;
32-
const overrides = { reporterOnlyFailures: opts.reporterOnlyFailures ? true : undefined };
33-
const config = configFile ? await configLoader.loadConfigFromFile(configFile, overrides) : await configLoader.loadEmptyConfigForMergeReports(overrides);
32+
const config = configFile ? await configLoader.loadConfigFromFile(configFile) : await configLoader.loadEmptyConfigForMergeReports();
3433

3534
const dir = path.resolve(process.cwd(), reportDir || '');
3635
const dirStat = await fs.promises.stat(dir).catch(e => null);

‎packages/playwright/src/cli/testActions.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,6 @@ function overridesFromOptions(options: { [key: string]: any }): ipc.ConfigCLIOve
120120
repeatEach: options.repeatEach ? parseInt(options.repeatEach, 10) : undefined,
121121
retries: options.retries ? parseInt(options.retries, 10) : undefined,
122122
reporter: resolveReporterOption(options.reporter),
123-
reporterOnlyFailures: options.reporterOnlyFailures ? true : undefined,
124123
additionalReporters: resolveReporterOption(options.addReporter),
125124
shard: resolveShardOption(options.shard),
126125
shuffle: resolveShuffleOption(options.shuffle),

‎packages/playwright/src/common/ipc.ts‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,6 @@ export type ConfigCLIOverrides = {
3535
repeatEach?: number;
3636
retries?: number;
3737
reporter?: ReporterDescription[];
38-
reporterOnlyFailures?: boolean;
3938
additionalReporters?: ReporterDescription[];
4039
shard?: { current: number, total: number };
4140
shuffle?: string;

‎packages/playwright/src/program.ts‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,6 @@ function addMergeReportsCommand(program: Command) {
129129
});
130130
command.option('-c, --config <file>', `Configuration file. Can be used to specify additional configuration for the output report.`);
131131
command.option('--reporter <reporter>', `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${config.defaultReporter}")`);
132-
command.option('--reporter-only-failures', 'Only include tests with failures in JSON, JUnit, and HTML reports');
133132
command.addHelpText('afterAll', `
134133
Arguments [dir]:
135134
Directory containing blob reports.
@@ -231,7 +230,6 @@ const testOptions: [string, { description: string, choices?: string[], preset?:
231230
['--quiet', { description: `Suppress stdio` }],
232231
['--repeat-each <N>', { description: `Run each test N times (default: 1)` }],
233232
['--reporter <reporter>', { description: `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${config.defaultReporter}")` }],
234-
['--reporter-only-failures', { description: `Only include tests with failures in JSON, JUnit, and HTML reports` }],
235233
['--retries <retries>', { description: `Maximum retry count for flaky tests, zero for no retries (default: no retries)` }],
236234
['--run-agents <mode>', { description: `Run agents to generate the code for page.perform`, choices: ['missing', 'all', 'none'], preset: 'none' }],
237235
['--shard <shard>', { description: `Shard tests and execute only the selected shard, specify in the form "current/all", 1-based, for example "3/5"` }],

‎packages/playwright/src/reporters/base.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ import realColors from 'colors/safe';
2121
import { noColors } from '@isomorphic/colors';
2222
import { msToString } from '@isomorphic/formatUtils';
2323
import { parseErrorStack } from '@utils/stackTrace';
24-
import { getPackageManagerExecCommand } from '@utils/env';
24+
import { getAsBooleanFromENV, getPackageManagerExecCommand } from '@utils/env';
2525
import { fitToWidth } from '@utils/stringWidth';
2626

2727
import { resolveReporterOutputPath, stripAnsiEscapes } from '../util';
@@ -397,8 +397,9 @@ export function isFailure(test: TestCase): boolean {
397397

398398
export type TestFilter = (test: TestCase) => boolean;
399399

400-
export function createTestFilter(options: CommonReporterOptions): TestFilter | undefined {
401-
return options.onlyFailures ? isFailure : undefined;
400+
export function createTestFilter(options: CommonReporterOptions, reporterName: 'HTML' | 'JSON' | 'JUNIT'): TestFilter | undefined {
401+
const onlyFailures = options._mode !== 'list' && getAsBooleanFromENV(`PLAYWRIGHT_${reporterName}_ONLY_FAILURES`, !!options.onlyFailures);
402+
return onlyFailures ? isFailure : undefined;
402403
}
403404

404405
export function* visitTests(suite: Suite, filter?: TestFilter): Generator<TestCase> {

‎packages/playwright/src/reporters/html.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@ class HtmlReporter implements ReporterV2 {
165165
noSnippets,
166166
noCopyPrompt,
167167
mergeFiles,
168-
}, createTestFilter(this._options));
168+
}, createTestFilter(this._options, 'HTML'));
169169
this._buildResult = await builder.build(this.config.metadata, projectSuites, result, this._topLevelErrors, this._machines);
170170
}
171171

‎packages/playwright/src/reporters/json.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ class JSONReporter implements ReporterV2 {
3737

3838
constructor(options: JsonReporterOptions & CommonReporterOptions) {
3939
this._resolvedOutputFile = resolveOutputFile('JSON', options)?.outputFile;
40-
this._testFilter = createTestFilter(options);
40+
this._testFilter = createTestFilter(options, 'JSON');
4141
}
4242

4343
version(): 'v2' {

0 commit comments

Comments
 (0)