Add mutation testing

[TheBlueMatt]

We should start requiring full mutation coverage for functions as we build out our unit tests.
https://github.com/llogiq/mutagen looks like a good candidate, unless @apoelstra thinks we should use https://github.com/apoelstra/halfsleep.
Tagging 0.2 as our goal for 0.2 should be to move more towards better test coverage.

[apoelstra]

Nice!

halfsleep is unmaintained, and I'd probably start contributing to mutagen before I'd start maintaining it again.

[TheBlueMatt]

There’s a new option on the block though this one is language-agnostic and uses LLVM. No idea what the integration work would look like and it may end up just generating a ton of false positives for things in stdlib instead of our own code. https://github.com/mull-project/mull

…

On Sep 19, 2018, at 06:43, Andrew Poelstra ***@***.***> wrote:  Nice! halfsleep is unmaintained, and I'd probably start contributing to mutagen before I'd start maintaining it again. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[dunxen]

So there's an even newer option now called cargo-mutants that doesn't require modifying any part of the source tree permanently. It's just run as a cargo subcommand which is convenient. I'm running it on main now to see if we get anything useful. I'm already getting a few missed mutants which could indicate some gaps already found. :)

[apoelstra]

cc @tcharding if you're bored and want to play with this one, you might beat me to it:)

[tcharding]

Sweet, cheers. Do note before you put an policy in place I've found mutation testing enjoyable because one really has to understand all the code tested but it is slow to implement in my experience.

[dunxen]

Mutating all functions over everything took about 12 hours (without --jobs for spawning multiple cargo processes) for me.
2421 mutants tested in 777:57: 505 missed, 1122 caught, 778 unviable, 16 timeouts
So I'm not sure what would be the best way to schedule something like this or maybe set it to run more granularly for PRs. One can filter by functions introduced in a PR. That might be more palatable for CI. Next would be figuring out the best way to use the results.

[TheBlueMatt]

Awesome! That's cool. I'm glad there's more mutation testing projects in Rust. As for what to do with the results, I think we should hard fail if new functions can be replaced with dummies with no test failure (Iiuc this is all the cargo mutants project does, no if inversion or so). It may well be very loud, but we can revert it if we get annoyed, and we don't need to require it for merges just yet, first just show a red X.

[tcharding]

I haven't played with cargo mutate yet but mutagen gives false positives is some cases, I don't think you'll be able to hard fail. An example I have been hitting often is < vs <=

fn abs(x: i32) -> u32 {
    if x < 0 { -x } else { x };
}