Presentation available at https://jtourvieille.github.io/MutationTesting/
When deploying a new version of a software...
Probably because ...
A good way to reassure yourself is to cover your code with tests
Code review? Hm ok...
But there is only one universal answer:
From Wikipedia:
“In computer science, test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage.”
From StackOverflow:
“Code coverage is collected by using a specialized tool to instrument the binaries to add tracing calls and run a full set of automated tests against the instrumented product. A good tool will give you not only the percentage of the code that is executed, but also will allow you to drill into the data and see exactly which lines of code were executed during a particular test.”
Write a program that prints the numbers from 1 to x. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”
public class FizzBuzzService
{
public string Play(int upperBound)
{
string finalOutput = string.Empty;
for (int i = 1; i <= upperBound; i++)
{
string localOutput = i % 3 == 0 ?
"Fizz" :
string.Empty;
localOutput += i % 5 == 0 ?
"Buzz" :
string.Empty;
localOutput += localOutput == string.Empty ?
i.ToString() :
string.Empty;
finalOutput += localOutput;
}
return finalOutput;
}
}
[TestMethod]
public void Should_Say1_When_1IsGivenAsInput()
{
// Arrange
var fizzBuzz = new FizzBuzzService();
// Act
var output = fizzBuzz.Play(1);
// Assert
Assert.AreEqual("1", output);
}
public string Play(int upperBound)
{
string finalOutput = string.Empty;
for (int i = 1; i <= upperBound; i++)
{
string localOutput = i % 3 == 0 ?
"Fizz" :
string.Empty;
localOutput += i % 5 == 0 ?
"Buzz" :
string.Empty;
localOutput += localOutput == string.Empty ?
i.ToString() :
string.Empty;
finalOutput += localOutput;
}
return finalOutput;
}
public string Play(int upperBound)
{
string finalOutput = string.Empty;
for (int i = 1; i <= upperBound; i++)
{
string localOutput = i % 3 == 0 ?
"Fizz" :
string.Empty;
localOutput += i % 5 == 0 ?
"Buzz" :
string.Empty;
localOutput += localOutput == string.Empty ?
i.ToString() :
string.Empty;
finalOutput += localOutput;
}
return finalOutput;
}
public string Play(int upperBound)
{
string finalOutput = string.Empty;
for (int i = 1; i <= upperBound; i++)
{
string localOutput = i % 3 == 0 ?
"Fizz" :
string.Empty;
localOutput += i % 5 == 0 ?
"Buzz" :
string.Empty;
localOutput += localOutput == string.Empty ?
i.ToString() :
string.Empty;
finalOutput += localOutput;
}
return finalOutput;
}
public string Play(int upperBound)
{
string finalOutput = string.Empty;
for (int i = 1; i <= upperBound; i++)
{
string localOutput = i % 3 == 0 ?
"Fizz" :
string.Empty;
localOutput += i % 5 == 0 ?
"Buzz" :
string.Empty;
localOutput += localOutput == string.Empty ?
i.ToString() :
string.Empty;
finalOutput += localOutput;
}
return finalOutput;
}
public string Play(int upperBound)
{
string finalOutput = string.Empty;
for (int i = 1; i <= upperBound; i++)
{
string localOutput = i % 3 == 0 ?
"Fizz" :
string.Empty;
localOutput += i % 5 == 0 ?
"Buzz" :
string.Empty;
localOutput += localOutput == string.Empty ?
i.ToString() :
string.Empty;
finalOutput += localOutput;
}
return finalOutput;
}
Code coverage is 83%
In my opinion, code coverage is an important but limited metric.
In our example, it points out that 17% of the code is not covered. But it does not indicates that 83% are well covered.
A 100% code coverage does not mean your code is 100% safe.
Automated end to end tests could make a good complement.
But also...
From Wikipedia:
“Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Mutants are based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as dividing each expression by zero).”
For instance, one mutant of the previous example could replace '<=' by '>'
public string Play(int upperBound)
{
string finalOutput = string.Empty;
for (int i = 1; i <= upperBound; i++)
{
string localOutput = i % 3 == 0 ?
"Fizz" :
string.Empty;
localOutput += i % 5 == 0 ?
"Buzz" :
string.Empty;
localOutput += localOutput == string.Empty ?
i.ToString() :
string.Empty;
finalOutput += localOutput;
}
return finalOutput;
}
A new code base is generated and every related test are run.
If the test fail, then the mutant is killed, everything is fine.
If the test pass, then the mutant survives, meaning that a code change does not impact tests. You have to worry in this case.
When mutants survive, you can consider that you can't trust your code coverage.
dotnet-tool install -g dotnet-stryker
https://docs.microsoft.com/fr-fr/dotnet/core/tools/dotnet-tool-install
{
"stryker-config": {
"reporters": [
"Progress",
"ClearText",
"html"
],
"log-level": "info",
"timeout-ms": 30000,
"max-concurrent-test-runners": 1,
"files-to-exclude": [ "./Commands/Mock" ],
"excluded-mutations": []
}
}
Run script from UT folder
dotnet-stryker -p PATH_TO_CS_PROJ_TO_MUTATE -s PATH_TO_SLN
Use absolute paths, relative paths are not supported yet
DO specify -p argument for non .NET core projects
npm install i -D @stryker-mutator/core @stryker-mutator/html-reporter @stryker-mutator/typescript @stryker-mutator/jest-runnner
const jestConfig = require('./jest.config.js');
module.exports = function(config) {
config.set({
mutate: [
'app/common/**/*.ts',
'app/common/**/*.js',
'!app/common/**/*.spec.ts',
'!app/common/**/*.spec.js',
'!app/common/**/*.stories.js',
'!**/index*',
],
mutator: 'typescript',
tsconfigFile: 'tsconfig.json',
testRunner: 'jest',
coverageAnalysis: 'off',
reporters: ['dots', 'html'],
htmlReporter: {
baseDir: 'stryker/html-reports/common',
},
jest: {
config: {
...jestConfig,
coverageReporters: [],
testResultsProcessor: null,
testMatch: ['**/app/common/**/*.spec.(t|j)s'],
},
enableFindRelatedTests: true,
},
files: ['index-specrunner.js', 'html-loader-preprocessor.js', 'app/**/*.*', 'babel.config.js'],
maxConcurrentTestRunners: 2,
timeoutFactor: 10,
timeoutMs: 60000,
});
};
{
"stryker:store": "stryker run stryker.store.conf.js",
"stryker:common": "stryker run stryker.common.conf.js",
"stryker:components": "stryker run stryker.components.conf.js",
"stryker:pages": "stryker run stryker.pages.conf.js",
}
npm run stryker:common
Inline PowerShell Task to install tool if required
if (@(dotnet tool list -g | where { $_ -match "dotnet-stryker" }).Length -eq 0) {
Invoke-Expression -Command "dotnet tool install -g dotnet-stryker"
}
Command Line Task to run analysis
dotnet-stryker -p "$(Build.SourcesDirectory)\NOA\NOA.CarInsurance\NOA.CarInsurance.csproj" -s "$(Build.SourcesDirectory)\NOA\NOA.sln"
npm task to install tool
npm ci
npm task to run analysis
npm run stryker:common
Backend |
---|
C# |
300 Tests |
4 vcpu 16 Go + SSD |
Frontend |
---|
Js |
4500 Tests |
8 vcpu 32 Go + SSD |
Pros |
---|
HTML JS |
Interactive report |
JSON |
Cons |
---|
Analysis time |
No aggregation |
No temporal analysis |
No integration with existing reports |
1 more weekly team meeting |