Stabilize json output

[epage]

This is to track our effort for stabilizing our json putput

See also rust-lang/testing-devex-team#1

Open questions

• Currently, success is "status": null, should it be "status": "success"?
• #[bench] macro #14
• Whether json output should be included rendered output, see Stabilize json output #42 (comment) (see 1ef1f95)
• expect assetions #69
• Output capturing #12
• Fixture support #18
• Parametrized tests support #19
• Test markers #70
• Json output evolution #71
• Reporting test span #72
• Ensure cargo callers can still differentiate "suites" #73
• Document the proposed json format and why choices were made #74
• Ability to differentiate from compiler messages #78
• Replace' RunMode with markers #84
• Failable discovery
• Do we need nested discovery?
• Run tests in parallel to discovery? #t-testing-devex > Handling slow test discovery @ 💬
• testsuite/test properties
• potential annotations
• Non-unique test names yield incorrect test output #166

[epage]

@pietroalbini privately raised a concern that rendered output should be included for pretty / terse. This is to make it easier when callers need to capture json messages while still rendering output, which they did for bootstrap

On the flip side

• I'm trying to shift knowledge of pretty/terse to cargo and change their meaning
• To reuse the rendered output is messy as it gets into whether its multi-threaded or not, whether multiple items are supposed to be printed per-line, the as-you-go details vs failure report, etc. This feels like a stabilization nightmare to get these scenarios to work out
• We are trying to open the door more for custom test harnesses. Do we put the complexity on the author of harnesses or on the test reporters?

[emilyalbini]

I'm trying to shift knowledge of pretty/terse to cargo and change their meaning

Would this mean there'd be no renderer in libtest anymore? That'd force anyone not using cargo to reimplement it for themselves. In addition to other build systems, compiletest uses libtest directly without cargo.

To reuse the rendered output is messy as it gets into whether its multi-threaded or not, whether multiple items are supposed to be printed per-line, the as-you-go details vs failure report, etc. This feels like a stabilization nightmare to get these scenarios to work out

A possible solution would be to have a new output event, that is emitted separately from the "normal" JSON events every time a write is called by the pretty/terse renderer.

We are trying to open the door more for custom test harnesses. Do we put the complexity on the author of harnesses or on the test reporters?

I imagine we'll eventually standardize to a small-ish number of test harnesses, while I can easily see lots of project hacking together scripts using the JSON events in CI.

[epage]

Would this mean there'd be no renderer in libtest anymore? That'd force anyone not using cargo to reimplement it for themselves. In addition to other build systems, compiletest uses libtest directly without cargo.

Most likely, the renderer would be frozen as-is and custom test harnesses could choose to render as they wish for default and --quiet

A possible solution would be to have a new output event, that is emitted separately from the "normal" JSON events every time a write is called by the pretty/terse renderer.

Which renderer? Both at once? If not, how does the user select?

And is this "output" supposed to literally be the equivalent of all of the write calls? Thats the only way I can see that working. We'd then likely also need to include an Event for flushing.

[epage]

On the upstream issue, I saw https://github.com/testing-cabal/subunit / https://github.com/mtreinish/subunit-rust mentioned and will need to take a look, either for inspiration or if we can use it outright.

[epage]

pytest's test / testsuite property might be reasonable to support. I've already been thinking of the issue between benchmark metrics and general test metrics. The general test metrics could just be properties.

[epage]

And someone created a json format for pytest: https://github.com/numirias/pytest-json-report

An ndjson one which has the potential to be merged is https://github.com/pytest-dev/pytest-reportlog

[epage]

Pulling in some related projects for input on benches

• Input on a json scema for bench reporting bheisler/criterion.rs#693
• Input on a json scema for bench reporting bencherdev/bencher#140
• Input on a json schema for bench reporting osiewicz/calliper#7

[emilyalbini]

Which renderer? Both at once? If not, how does the user select?

I would include only one renderer at the time (to avoid hardcoding "terse" vs "verbose" in the output), not sure how to encode that. Maybe json+terse, json+verbose as the format name?

And is this "output" supposed to literally be the equivalent of all of the write calls? Thats the only way I can see that working. We'd then likely also need to include an Event for flushing.

Yeah, I'd also include an event for flushing. That way third party runners shouldn't have many problems supporting this.

[epage]

We should probably dig into these ideas for exploring the ability to support these in the future if we need it

[epage]

Similarly, we should look to TAP for ideas.

[epage]

Found an issue specifically about including source information in the json output: rust-lang/rust#116614

[epage]

On the package information being included: rust-lang/rust#115971