Mutation testing

Introduction to mutation testing

Presentation available at https://jtourvieille.github.io/MutationTesting/

Who are we?


Jounad Tourvieille

Guillaume Delahaye

What will be covered

  • What is code coverage?
  • How is it calculated?
  • Code coverage weakness
  • Mutation testing
  • Full sample
  • LOB feedback

When deploying a new version of a software...

Why this fear?

Probably because ...

  • New features were not tested enough
  • An existing part of the application may not work anymore

How to solve this?

A good way to reassure yourself is to cover your code with tests

Tests will avoid this...

Unit tests advantages

  • Unit testing can increase confidence in changing and maintaining code in the development process.
  • Unit testing always has the ability to find problems in early stages in the development cycle.
  • Codes are more reusable, reliable and clean.
  • Development becomes faster.
  • Easy to automate.
  • ...

How to know if your application is safely covered by unit tests?

Code review? Hm ok...

But there is only one universal answer:


Code coverage

What is code coverage?

From Wikipedia:

“In computer science, test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage.”

How code coverage is measured?

From StackOverflow:

“Code coverage is collected by using a specialized tool to instrument the binaries to add tracing calls and run a full set of automated tests against the instrumented product. A good tool will give you not only the percentage of the code that is executed, but also will allow you to drill into the data and see exactly which lines of code were executed during a particular test.”

Deep dive with a sample

Fizz Buzz

Write a program that prints the numbers from 1 to x. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”

Demo

							
public class FizzBuzzService
{
 public string Play(int upperBound)
 {
  string finalOutput = string.Empty;
  for (int i = 1; i <= upperBound; i++)
  {
   string localOutput = i % 3 == 0 ?
	"Fizz" :
	string.Empty;
   localOutput += i % 5 == 0 ?
	"Buzz" :
	string.Empty;
   localOutput += localOutput == string.Empty ?
	i.ToString() :
	string.Empty;
   finalOutput += localOutput;
  }
  return finalOutput;
 }
}
							
						
							
[TestMethod]
public void Should_Say1_When_1IsGivenAsInput()
{
	// Arrange
	var fizzBuzz = new FizzBuzzService();

	// Act
	var output = fizzBuzz.Play(1);

	// Assert
	Assert.AreEqual("1", output);
}
							
						
							
public string Play(int upperBound)
{
  string finalOutput = string.Empty;
  for (int i = 1; i <= upperBound; i++)
  {
   string localOutput = i % 3 == 0 ?
	"Fizz" :
	string.Empty;
   localOutput += i % 5 == 0 ?
	"Buzz" :
	string.Empty;
   localOutput += localOutput == string.Empty ?
	i.ToString() :
	string.Empty;
   finalOutput += localOutput;
  }
  return finalOutput;
}
							
						
							
public string Play(int upperBound)
{
  string finalOutput = string.Empty;
  for (int i = 1; i <= upperBound; i++)
  {
   string localOutput = i % 3 == 0 ?
	"Fizz" :
	string.Empty;
   localOutput += i % 5 == 0 ?
	"Buzz" :
	string.Empty;
   localOutput += localOutput == string.Empty ?
	i.ToString() :
	string.Empty;
   finalOutput += localOutput;
  }
  return finalOutput;
}
							
						
							
public string Play(int upperBound)
{
  string finalOutput = string.Empty;
  for (int i = 1; i <= upperBound; i++)
  {
   string localOutput = i % 3 == 0 ?
	"Fizz" :
	string.Empty;
   localOutput += i % 5 == 0 ?
	"Buzz" :
	string.Empty;
   localOutput += localOutput == string.Empty ?
	i.ToString() :
	string.Empty;
   finalOutput += localOutput;
  }
  return finalOutput;
}
							
						
							
public string Play(int upperBound)
{
  string finalOutput = string.Empty;
  for (int i = 1; i <= upperBound; i++)
  {
   string localOutput = i % 3 == 0 ?
	"Fizz" :
	string.Empty;
   localOutput += i % 5 == 0 ?
	"Buzz" :
	string.Empty;
   localOutput += localOutput == string.Empty ?
	i.ToString() :
	string.Empty;
   finalOutput += localOutput;
  }
  return finalOutput;
}
							
						
							
public string Play(int upperBound)
{
  string finalOutput = string.Empty;
  for (int i = 1; i <= upperBound; i++)
  {
   string localOutput = i % 3 == 0 ?
	"Fizz" :
	string.Empty;
   localOutput += i % 5 == 0 ?
	"Buzz" :
	string.Empty;
   localOutput += localOutput == string.Empty ?
	i.ToString() :
	string.Empty;
   finalOutput += localOutput;
  }
  return finalOutput;
}
							
						

Code coverage is 83%

Is code coverage a good metric?

In my opinion, code coverage is an important but limited metric.


In our example, it points out that 17% of the code is not covered. But it does not indicates that 83% are well covered.


A 100% code coverage does not mean your code is 100% safe.

Automated end to end tests could make a good complement.

But also...

Mutants!

What is code mutation?

From Wikipedia:

“Mutation testing (or mutation analysis or program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Mutants are based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as dividing each expression by zero).”

What is code mutation?

For instance, one mutant of the previous example could replace '<=' by '>'

							
public string Play(int upperBound)
{
  string finalOutput = string.Empty;
  for (int i = 1; i <= upperBound; i++)
  {
   string localOutput = i % 3 == 0 ?
	"Fizz" :
	string.Empty;
   localOutput += i % 5 == 0 ?
	"Buzz" :
	string.Empty;
   localOutput += localOutput == string.Empty ?
	i.ToString() :
	string.Empty;
   finalOutput += localOutput;
  }
  return finalOutput;
}
							
						

A new code base is generated and every related test are run.

What is code mutation?

If the test fail, then the mutant is killed, everything is fine.
If the test pass, then the mutant survives, meaning that a code change does not impact tests. You have to worry in this case.

When mutants survive, you can consider that you can't trust your code coverage.

Tools


Pitest

Stryker

Demo 2


Demo 3

Experience Feedback

  • 3 months
  • 1 major application
  • 7 junior/senior developers
  • 4.7.2 Backend (10 000 lines of C#)
    xUnit unit tests (300) 
  • Frontend (25 000 lines of )
    Jest unit tests (4500) 

Share concept with team

  • POC
  • Chapter dev
  • DOJO

Backend tool setup

									dotnet-tool install -g dotnet-stryker
								

https://docs.microsoft.com/fr-fr/dotnet/core/tools/dotnet-tool-install

Backend configuration file

									
{
	"stryker-config": {
	"reporters": [
		"Progress",
		"ClearText",
		"html"
	],
	"log-level": "info",
	"timeout-ms": 30000,
	"max-concurrent-test-runners": 1,
	"files-to-exclude": [ "./Commands/Mock" ],
	"excluded-mutations": []
	}
}
			
									
								

Backend script

Run script from UT folder

									
dotnet-stryker -p PATH_TO_CS_PROJ_TO_MUTATE -s PATH_TO_SLN
									
								

Use absolute paths, relative paths are not supported yet

DO specify -p argument for non .NET core projects

Frontend tool setup

									npm install i -D @stryker-mutator/core @stryker-mutator/html-reporter @stryker-mutator/typescript @stryker-mutator/jest-runnner
								

Frontend configuration file

							
const jestConfig = require('./jest.config.js');
module.exports = function(config) {
  config.set({
    mutate: [
      'app/common/**/*.ts',
      'app/common/**/*.js',
      '!app/common/**/*.spec.ts',
      '!app/common/**/*.spec.js',
      '!app/common/**/*.stories.js',
      '!**/index*',
    ],
    mutator: 'typescript',
    tsconfigFile: 'tsconfig.json',
    testRunner: 'jest',
    coverageAnalysis: 'off',
    reporters: ['dots', 'html'],
    htmlReporter: {
      baseDir: 'stryker/html-reports/common',
    },
    jest: {
      config: {
        ...jestConfig,
        coverageReporters: [],
        testResultsProcessor: null,
        testMatch: ['**/app/common/**/*.spec.(t|j)s'],
      },
      enableFindRelatedTests: true,
    },
    files: ['index-specrunner.js', 'html-loader-preprocessor.js', 'app/**/*.*', 'babel.config.js'],
    maxConcurrentTestRunners: 2,
    timeoutFactor: 10,
    timeoutMs: 60000,
  });
};				
							
						

Frontend npm scripts

								
		{
		  "stryker:store": "stryker run stryker.store.conf.js",
		  "stryker:common": "stryker run stryker.common.conf.js",
		  "stryker:components": "stryker run stryker.components.conf.js",
		  "stryker:pages": "stryker run stryker.pages.conf.js",
		}
								
							
							npm run stryker:common
						

Automation

Backend Build pipeline

Backend Build pipeline

Inline PowerShell Task to install tool if required

								
if (@(dotnet tool list -g | where { $_ -match "dotnet-stryker" }).Length -eq 0) {
Invoke-Expression -Command "dotnet tool install -g dotnet-stryker"
}
								
							

Command Line Task to run analysis

									
dotnet-stryker -p "$(Build.SourcesDirectory)\NOA\NOA.CarInsurance\NOA.CarInsurance.csproj" -s "$(Build.SourcesDirectory)\NOA\NOA.sln"
									
								

Frontend Build pipeline

Frontend build pipeline

npm task to install tool

								npm ci
							

npm task to run analysis

								npm run stryker:common
							

Execution time

Backend
C#
300 Tests
4 vcpu 16 Go + SSD

1h

Frontend
Js
4500 Tests
8 vcpu 32 Go + SSD

11h30

Analysis

Pros
HTML JS
Interactive report
JSON
Cons
Analysis time
No aggregation
No temporal analysis
No integration with existing reports
1 more weekly team meeting

Tips

  • Configure mutations
  • Focus on critical code
  • Split analysis on large code base

What's next ?

  • JSON reports aggregation
  • SonarQube integration through plugin
  • incremental analysis
  • git diff analysis
  • Azure DevOps pipeline tasks

Should we go?

  • Target your business/critical code
  • Warn on global tooling