Mutation Testing with C# and .NET Core

Unit Testing is widely known method of validating results produced by a code in an automated manner. Good test coverage helps with maintaining a code as it is much easier and quicker to spot potential bugs after code changes. How to check if a test suite is effective though? One of the answers is to use Mutation Testing.

The idea is to create mutants which are modified versions of your code and run existing unit tests against these mutants. If a mutation caused test run failure it means a mutant is killed (which is good). If the tests passed then a mutant survived (which is not good). The measurement of the unit tests quality is percentage of the killed mutants where the higher figure is the better. In this approach we expect the test run to fail on mutated code otherwise either not all paths are covered by the unit tests or your tests miss some conditions.

mutated code + failed tests = killed mutant

mutated code + passed tests = survived mutant

What the mutation actually is?

Mutation is code change which might be anything limited by either your imagination or a tool you use. The most basic are (with examples):

• Arithmetic operators

• Equality operators

• Logical operators

• Negation operator

There might be many more type of mutations. Above are presented to explain the general idea of mutating process.

How to run the mutations?

In theory you could mutate manually but of course it would be extremely inefficient. That’s why in this article I’d like to present a tool which automates the process, is very easy to use and presents results in a very readable form. The tool is called Stryker.NET.

Getting started with Stryker.NET

For the purpose of this demo two projects have been created: class library called Math and MSTest project called Math.Tests. Both with .NET Core 3.1.

Installing Stryker.NET

First of all we need to create tool manifest for Math solution by running below command in the root folder (in my case it is: C:\Users\…\Source\Repos\Math).

It should result with new folder being created called .config containing dotnet-tools.json file. When it is done, the actual Stryker.NET installation can be run with below command.

Successful installation should result with appropriate message in command line and also new entry in the manifest file.

Now, we’re good to go with below command which runs mutation tests. It should be executed in the test project path (in my case: C:\Users\…\Source\Repos\Math\Math.Tests).

Stryker.NET automatically identifies projects to mutate based on references added to test project. As the Math has no content yet, there is nothing to mutate, so the Stryker ends with warning message “It’s a mutant-free world, nothing to test”.

Example

Below we’ll create some logic and unit tests to present how Stryker.NET actually works. The example is trivial intentionally as it is the easiest to understand the actual sense of mutations.

Let’s create an Utilities static class containing a Sum function which sums up two integers and returns the result of the equation. It is created in Math project which is the class library containing our business logic.

As the next step let’s create a test for above component. We’ll do this by adding UtilitiesTest class in Math.Tests project. The logic is very simple and validates whether 1 + 0 results with 1.

Of course above test will pass but let’s see what will be the result of Stryker.NET run. As it was mentioned above we can run it with dotnet stryker command executed inside Math.Tests project path.

Looking at the above output we can notice few useful information. One of them is that project Math.csproj has been correctly identified to be mutated. We can also see the number of mutants with the summary of killed and survived ones. We’ll analyse it on the report though, which has been generated in HTML format (mutation-report.html).

After opening the report we can see a summary of all mutated files with details like mutation score (the higher is better), number of killed and survived mutants, timeouts, no coverage and ignored code and also runtime and compile errors.

The mutation score can be between 0 and 100, so our 0.00 result cannot be worse. Based on above table we can see that one mutant was generated and it survived. More details about it can be found after clicking the Utilities.cs file.

The expanded view displays source code of particular file with all mutations. The survived mutants are marked with red box and details of it can be expanded by clicking on it.

The survived mutation was a change of plus sign (a + b) into minus sign (a – b). It wasn’t killed because our unit test still passes even though the logic of Sum function changed.

For the given test criteria (a = 1, b = 0) both original and mutated functions return the same result which is also the value expected by unit test. It clearly tells the unit test coverage is not good enough. Let’s improve it by creating another test case with different input values and re-run Stryker.

This time again 1 mutant was created but it was successfully killed and the mutation score is 100.

What’s the benefit of mutation testing?

Performing mutation testing helps with validating whether the quality of your unit tests is good enough. In some cases high test coverage rate may be misleading and gives false impression of security. Small changes in your business logic (mutations) can verify whether test suite is robust and is able to identify unexpected logic changes. It’s especially important when your code is being refactored as the refactoring can accidentally produce the mutation.

Not only test suite might be beneficiary of mutation testing. It can also highlight some business logic issues within the component which is being tested. A mutation might be an expected behavior which was coded different way by mistake (e.g. “>” operator used instead of “>=”).

Summary

Above article is very general description of mutation testing but hopefully it sheds some light on this topic and is good starting point. I highly recommend to visit Stryker.NET page on GitHub where you can find much more information about available functionalities and also possible options of configuration.