DEV Community

Cover image for Mutation Testing in Rust
Nicolas Fränkel
Nicolas Fränkel

Posted on • Originally published at blog.frankel.ch

Mutation Testing in Rust

#mutationtesting #rust #opensource

I've been a big fan of Mutation Testing since I discovered PIT. As I dive deeper into Rust, I wanted to check the state of mutation testing in Rust.

Starting with cargo-mutants

I found two crates for mutation testing in Rust:

mutagen hasn't been maintained for three years, while cargo-mutants is still under active development.

I've ported the sample code from my previous Java code to Rust:

struct LowPassPredicate { threshold: i32, } impl LowPassPredicate { pub fn new(threshold: i32) -> Self { LowPassPredicate { threshold } } pub fn test(&self, value: i32) -> bool { value < self.threshold } } #[cfg(test)] mod tests { use super::*; #[test] fn should_return_true_when_under_limit() { let low_pass_predicate = LowPassPredicate::new(5); assert_eq!(low_pass_predicate.test(4), true); } #[test] fn should_return_false_when_above_limit() { let low_pass_predicate = LowPassPredicate::new(5); assert_eq!(low_pass_predicate.test(6), false); } }
Enter fullscreen mode Exit fullscreen mode

Using cargo-mutants is a two-step process:

  1. Install it, cargo install --locked cargo-mutants
  2. Use it, cargo mutants 
Found 4 mutants to test ok Unmutated baseline in 0.1s build + 0.3s test INFO Auto-set test timeout to 20s 4 mutants tested in 1s: 4 caught
Enter fullscreen mode Exit fullscreen mode

I expected a mutant to survive, as I didn't test the boundary when the test value equals the limit. Strangely enough, cargo-mutants didn't detect it.

Finding and fixing the issue

I investigated the source code and found the place where it mutates operators:

// We try replacing logical ops with == and !=, which are effectively // XNOR and XOR when applied to booleans. However, they're often unviable // because they require parenthesis for disambiguation in many expressions. BinOp::Eq(_) => vec![quote! { != }], BinOp::Ne(_) => vec![quote! { == }], BinOp::And(_) => vec![quote! { || }], BinOp::Or(_) => vec![quote! { && }], BinOp::Lt(_) => vec![quote! { == }, quote! {>}], BinOp::Gt(_) => vec![quote! { == }, quote! {<}], BinOp::Le(_) => vec![quote! {>}], BinOp::Ge(_) => vec![quote! {<}], BinOp::Add(_) => vec![quote! {-}, quote! {*}],
Enter fullscreen mode Exit fullscreen mode

Indeed, < is changed to == and >, but not to <=. I forked the repo and updated the code accordingly:

BinOp::Lt(_) => vec![quote! { == }, quote! {>}, quote!{ <= }], BinOp::Gt(_) => vec![quote! { == }, quote! {<}, quote!{ => }],
Enter fullscreen mode Exit fullscreen mode

I installed the new forked version:

cargo install --git https://github.com/nfrankel/cargo-mutants.git --locked
Enter fullscreen mode Exit fullscreen mode

I reran the command:

cargo mutants
Enter fullscreen mode Exit fullscreen mode

The output is the following:

Found 5 mutants to test ok Unmutated baseline in 0.1s build + 0.3s test INFO Auto-set test timeout to 20s MISSED src/lib.rs:11:15: replace < with <= in LowPassPredicate::test in 0.2s build + 0.2s test 5 mutants tested in 2s: 1 missed, 4 caught
Enter fullscreen mode Exit fullscreen mode

You can find the same information in the missed.txt file. I thought I fixed it and was ready to make a Pull Request to the cargo-mutants repo. I just needed to add the test at the boundary:

#[test] fn should_return_false_when_equals_limit() { let low_pass_predicate = LowPassPredicate::new(5); assert_eq!(low_pass_predicate.test(5), false); }
Enter fullscreen mode Exit fullscreen mode 
cargo test
Enter fullscreen mode Exit fullscreen mode 
running 3 tests test tests::should_return_false_when_above_limit ... ok test tests::should_return_false_when_equals_limit ... ok test tests::should_return_true_when_under_limit ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Enter fullscreen mode Exit fullscreen mode 
cargo mutants
Enter fullscreen mode Exit fullscreen mode

And all mutants are killed!

Found 5 mutants to test ok Unmutated baseline in 0.1s build + 0.2s test INFO Auto-set test timeout to 20s 5 mutants tested in 2s: 5 caught
Enter fullscreen mode Exit fullscreen mode

Conclusion

Not many blog posts end with a Pull Request, but this one does.

Also mutate `>` to `>=` and `<` to `<=` #501

nfrankel avatar
nfrankel posted on

Most tests miss testing the boundary. Mutation testing can help a lot with this. The current code misses the change from > to >= and < and <=.

View on GitHub

Unfortunately, I couldn't manage to make the tests pass; fortunately, the repository maintainer helped me–a lot. The Pull Request is merged: enjoy this slight improvement.

I learned more about cargo-mutants and could improve the code in the process.

To go further:

Originally published at A Java Geek on March 30th, 2025

Top comments (0)