generate tests from code example docstrings (to fight bitrot)

[olsonjeffery]

So I'm going through my contributions and adding code examples for most of the API stuff out there that isn't immediately obvious.

I only have one example of an existing code sample that is live in the libstd docs right now, and that's for net::tcp::listen. Suffice to say: it is very out-of-date with the current state of the art for rust. This is a sad discovery, on my part.

Since more literate code examples in the docs is probably a good thing, this is where I'm putting my energies for documentation of my work (as opposed to blog posts, emails to the list, etc). I'm trying to make the examples complete, illustrative and durable (the latter of which I have no control over).

Cutting to the chase:

I'd be happy with something like what (I'm told) we have in the tutorial markdown: have the build automation (rustdoc, in this case?) do a run-pass on all code blocks (by some known convention) that appear in docstrings, for the greater good.

Thoughts?

[catamorphism]

Yes, this seems like an obviously good idea to me. I think it is an RFC-level change, though, since it would result in rejecting more programs (which is a good thing!), so I'm adding the RFC label.

[brson]

The easiest way is probably to use rustdoc to convert the documentation to markdown, then run the python script over it to generate tests.

Here's my preference:

• Move the --test flag out of rustc completely, to a new tool called rusttest (or just leave all this logic inside rustc, but I'm wary of continuing to stuff non-compiler stuff into rustc)
• Teach rusttest to parse test cases out of doc attributes the way that the existing python script does
• Inject new tests into the AST based on the docs
• Compile the whole thing as a test runner

Later, once we've created a unified rust tool we would end up just writing rust test foo.rc

[pnkfelix]

So, this issue is a request for infrastructure for generating execution unit tests from the embedded documentation for a symbol. That sounds like a match for Maturity Milestone 4: well covered.

[emberian]

Absorbing under rustdoc2

[emberian]

Tests can be extracted from the doc comments present in the JSON rustdoc_ng emits. Tooling needs to be made to do so, and to build+run the tests.

[pnkfelix]

part of #8125

[emberian]

Unassigning self.

[alexcrichton]

Nominating, this is the only way we're going to have code examples be reliable