Testing & TDD

Mutation Testing

28 min Lesson 25 of 35

Mutation Testing

Mutation testing evaluates the quality of your test suite by introducing deliberate bugs (mutations) into your code and checking if your tests catch them. In this lesson, we'll explore mutation testing concepts, tools like Infection PHP and Stryker JS, and strategies for improving test effectiveness.

What is Mutation Testing?

Mutation testing answers a critical question: "Are my tests actually testing anything meaningful?"

Traditional code coverage tells you which lines are executed during tests, but it doesn't tell you if those tests would catch bugs. You could have 100% code coverage with assertions that never actually verify correct behavior.

Mutation testing works by:

Creating Mutants: Automatically modifying your source code in small ways (e.g., changing + to -, > to >=, removing method calls)
Running Tests: Executing your test suite against each mutant
Scoring: A mutant is "killed" if tests fail, "survived" if tests pass (indicating weak tests)
Measuring Quality: Mutation Score = (Killed Mutants / Total Mutants) × 100%

Key Insight: A high mutation score indicates that your tests would likely catch real bugs. A low mutation score means you have weak tests that give false confidence.

Types of Mutations

Common mutation operators introduce various types of changes:

Arithmetic Operators: + → -, * → /, % → *
Relational Operators: > → >=, == → !=, < → <=
Logical Operators: && → ||, ! removed
Return Values: return true → return false, return $value → return null
Method Calls: Remove method calls, change arguments
Conditionals: Remove if conditions, invert conditions
Increments: ++ → --, $i++ → $i

Mutation Testing with Infection PHP

Infection is the leading mutation testing framework for PHP:

# Install Infection
composer require --dev infection/infection

# Initialize configuration
vendor/bin/infection --configure

# infection.json5 (generated configuration)
{
    "$schema": "vendor/infection/infection/resources/schema.json",
    "source": {
        "directories": ["src"]
    },
    "timeout": 10,
    "logs": {
        "text": "infection.log",
        "html": "infection-report.html"
    },
    "mutators": {
        "@default": true
    },
    "minMsi": 80,        // Minimum Mutation Score Indicator
    "minCoveredMsi": 90  // Minimum covered code MSI
}

# Run Infection
vendor/bin/infection

# Run with specific mutators
vendor/bin/infection --mutators=@arithmetic,@conditional

# Run on specific files
vendor/bin/infection --filter=src/Services/Calculator.php

# Run with multiple threads (faster)
vendor/bin/infection --threads=4

Example: Weak Tests Revealed by Mutation Testing

Consider this code and its test:

<?php

// src/Services/PriceCalculator.php
class PriceCalculator
{
    public function calculateDiscount(float $price, float $discountPercent): float
    {
        if ($discountPercent > 100) {
            throw new InvalidArgumentException('Discount cannot exceed 100%');
        }

        $discount = $price * ($discountPercent / 100);
        return $price - $discount;
    }

    public function applyTax(float $amount, float $taxRate): float
    {
        return $amount + ($amount * $taxRate);
    }
}

// tests/Services/PriceCalculatorTest.php
class PriceCalculatorTest extends TestCase
{
    // WEAK TEST - Only checks happy path
    public function test_calculate_discount()
    {
        $calculator = new PriceCalculator();
        $result = $calculator->calculateDiscount(100, 10);

        $this->assertEquals(90, $result);
    }

    // WEAK TEST - Doesn't verify calculation
    public function test_apply_tax()
    {
        $calculator = new PriceCalculator();
        $result = $calculator->applyTax(100, 0.1);

        $this->assertIsFloat($result); // BAD - Only checks type!
    }
}

Run Infection on this code:

# Infection output:
Mutation Score Indicator (MSI): 30%
Covered Code MSI: 40%

Escaped mutants:
1) src/Services/PriceCalculator.php:9
   - Changed > to >=
   - Mutant survived (test didn't catch it)

2) src/Services/PriceCalculator.php:13
   - Changed - to +
   - Mutant survived (test didn't catch it)

3) src/Services/PriceCalculator.php:18
   - Changed + to -
   - Mutant survived (test didn't catch it)

4) src/Services/PriceCalculator.php:18
   - Changed * to /
   - Mutant survived (test didn't catch it)

Improving Tests Based on Mutation Results

Fix the weak tests to kill the mutants:

<?php

class PriceCalculatorTest extends TestCase
{
    // IMPROVED - Tests boundary conditions
    public function test_calculate_discount()
    {
        $calculator = new PriceCalculator();

        // Test normal discount
        $this->assertEquals(90, $calculator->calculateDiscount(100, 10));

        // Test zero discount
        $this->assertEquals(100, $calculator->calculateDiscount(100, 0));

        // Test maximum discount
        $this->assertEquals(0, $calculator->calculateDiscount(100, 100));

        // Test boundary: 100% should work
        $this->assertEquals(0, $calculator->calculateDiscount(100, 100));
    }

    // IMPROVED - Tests invalid input
    public function test_calculate_discount_throws_on_invalid_percent()
    {
        $calculator = new PriceCalculator();

        $this->expectException(InvalidArgumentException::class);
        $calculator->calculateDiscount(100, 101);
    }

    // IMPROVED - Verifies actual calculation
    public function test_apply_tax()
    {
        $calculator = new PriceCalculator();

        // Test 10% tax
        $this->assertEquals(110, $calculator->applyTax(100, 0.1));

        // Test 0% tax
        $this->assertEquals(100, $calculator->applyTax(100, 0));

        // Test 25% tax
        $this->assertEquals(125, $calculator->applyTax(100, 0.25));

        // Test with decimal amounts
        $this->assertEquals(110.55, $calculator->applyTax(99.59, 0.11), '', 0.01);
    }
}

# Run Infection again:
Mutation Score Indicator (MSI): 95%
Covered Code MSI: 98%

All mutants killed! ✓

Best Practice: Aim for a mutation score of 80-90%. Don't obsess over 100%—some mutants are equivalent (produce same behavior) or represent unrealistic scenarios.

Mutation Testing with Stryker JS

Stryker is the mutation testing framework for JavaScript/TypeScript:

# Install Stryker
npm install --save-dev @stryker-mutator/core
npm install --save-dev @stryker-mutator/jest-runner

# Initialize configuration
npx stryker init

# stryker.conf.json (generated)
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "packageManager": "npm",
  "reporters": ["html", "clear-text", "progress", "dashboard"],
  "testRunner": "jest",
  "coverageAnalysis": "perTest",
  "mutate": [
    "src/**/*.js",
    "!src/**/*.test.js"
  ],
  "thresholds": {
    "high": 80,
    "low": 60,
    "break": 50
  }
}

# Run Stryker
npx stryker run

# Example JavaScript code and tests
// src/calculator.js
export function add(a, b) {
    return a + b;
}

export function divide(a, b) {
    if (b === 0) {
        throw new Error('Division by zero');
    }
    return a / b;
}

// src/calculator.test.js (WEAK TEST)
import { add, divide } from './calculator';

test('adds numbers', () => {
    expect(add(2, 3)).toBe(5);
});

test('divides numbers', () => {
    const result = divide(10, 2);
    expect(result).toBeDefined(); // BAD - Weak assertion
});

# Stryker output:
Mutant survived: Changed + to - in add()
Mutant survived: Changed / to * in divide()
Mutant survived: Changed === to !== in divide()

Mutation score: 33%

Improving JavaScript Tests

Kill the mutants with better assertions:

// src/calculator.test.js (IMPROVED)
import { add, divide } from './calculator';

describe('add', () => {
    test('adds positive numbers', () => {
        expect(add(2, 3)).toBe(5);
    });

    test('adds negative numbers', () => {
        expect(add(-2, -3)).toBe(-5);
    });

    test('adds zero', () => {
        expect(add(5, 0)).toBe(5);
        expect(add(0, 5)).toBe(5);
    });
});

describe('divide', () => {
    test('divides positive numbers', () => {
        expect(divide(10, 2)).toBe(5);
    });

    test('divides with decimals', () => {
        expect(divide(10, 3)).toBeCloseTo(3.33, 2);
    });

    test('throws on division by zero', () => {
        expect(() => divide(10, 0)).toThrow('Division by zero');
    });

    test('divides negative numbers', () => {
        expect(divide(-10, 2)).toBe(-5);
        expect(divide(10, -2)).toBe(-5);
    });
});

# Stryker output:
Mutation score: 92%
All critical mutants killed! ✓

Common Mutation Testing Patterns

Learn to recognize and kill common mutation patterns:

// Pattern 1: Boundary conditions
function isAdult(age) {
    return age >= 18; // Mutant: >= → >
}

// Kill with boundary test
test('boundary at 18', () => {
    expect(isAdult(17)).toBe(false);
    expect(isAdult(18)).toBe(true); // Critical!
    expect(isAdult(19)).toBe(true);
});

// Pattern 2: Boolean returns
function isValid(value) {
    if (value) {
        return true; // Mutant: true → false
    }
    return false; // Mutant: false → true
}

// Kill with explicit checks
test('validates correctly', () => {
    expect(isValid('test')).toBe(true);
    expect(isValid('')).toBe(false);
    expect(isValid(null)).toBe(false);
});

// Pattern 3: Arithmetic operations
function calculateTotal(price, quantity) {
    return price * quantity; // Mutant: * → +, -, /
}

// Kill with specific values
test('calculates total', () => {
    expect(calculateTotal(10, 3)).toBe(30); // Not 13, 7, or 3.33
    expect(calculateTotal(5, 0)).toBe(0);
    expect(calculateTotal(0, 10)).toBe(0);
});

// Pattern 4: Method call removal
function processUser(user) {
    validateUser(user); // Mutant: Remove this call
    saveUser(user);     // Mutant: Remove this call
    return user;
}

// Kill with side-effect verification
test('processes user completely', () => {
    const validateSpy = jest.spyOn(module, 'validateUser');
    const saveSpy = jest.spyOn(module, 'saveUser');

    processUser({ name: 'John' });

    expect(validateSpy).toHaveBeenCalled();
    expect(saveSpy).toHaveBeenCalled();
});

// Pattern 5: Array/Collection operations
function filterActive(users) {
    return users.filter(u => u.active); // Mutant: active → !active
}

// Kill with both positive and negative cases
test('filters active users', () => {
    const users = [
        { id: 1, active: true },
        { id: 2, active: false },
        { id: 3, active: true },
    ];

    const result = filterActive(users);

    expect(result).toHaveLength(2);
    expect(result[0].id).toBe(1);
    expect(result[1].id).toBe(3);
    expect(result.find(u => u.id === 2)).toBeUndefined();
});

Interpreting Mutation Testing Reports

Understand mutation testing metrics:

# Infection/Stryker Report Metrics

1. Mutation Score Indicator (MSI)
   - Formula: (Killed + Timeout) / (Total - Ignored - NotCovered) × 100
   - Target: 80-90%
   - Measures overall test effectiveness

2. Covered Code MSI
   - Formula: (Killed + Timeout) / (Total - Ignored) × 100
   - Target: 90-95%
   - Only considers code covered by tests

3. Mutant States:
   - Killed: Test failed (good - mutant detected)
   - Survived: Test passed (bad - weak test)
   - Timeout: Mutant caused infinite loop (good)
   - Not Covered: No test executed this code
   - Ignored: Excluded by configuration
   - Error: Mutant caused syntax/runtime error

4. HTML Report Structure:
   - Green: Mutant killed ✓
   - Red: Mutant survived (needs better test) ✗
   - Grey: Not covered by tests
   - Yellow: Timeout (mutant too slow)

# Prioritize fixing:
1. Survived mutants in critical code paths
2. Survived mutants in complex logic
3. Not covered code (increase code coverage first)
4. Survived mutants in simple code (low priority)

Warning: Mutation testing is slow. A test suite that runs in 30 seconds might take 30 minutes with mutation testing. Run it periodically (nightly builds, before releases), not on every commit.

CI/CD Integration

Integrate mutation testing into your pipeline:

# .github/workflows/mutation-tests.yml
name: Mutation Testing

on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * 0' # Weekly on Sunday at 2 AM

jobs:
  mutation-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 60

    steps:
      - uses: actions/checkout@v3

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '8.2'
          coverage: pcov

      - name: Install dependencies
        run: composer install

      - name: Run PHPUnit (for baseline coverage)
        run: vendor/bin/phpunit --coverage-clover=coverage.xml

      - name: Run Infection
        run: |
          vendor/bin/infection \
            --threads=4 \
            --min-msi=80 \
            --min-covered-msi=90 \
            --logger-html=infection-report.html \
            --skip-initial-tests

      - name: Upload mutation report
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: mutation-report
          path: infection-report.html

      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const log = fs.readFileSync('infection.log', 'utf8');
            const msiMatch = log.match(/Mutation Score Indicator \(MSI\): (\d+)%/);
            const msi = msiMatch ? msiMatch[1] : 'N/A';

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## Mutation Testing Results\n\nMutation Score: ${msi}%`
            });

# For JavaScript with Stryker
      - name: Run Stryker
        run: npx stryker run

      - name: Upload Stryker report
        uses: stryker-mutator/dashboard-reporter@v1
        with:
          api-key: ${{ secrets.STRYKER_DASHBOARD_API_KEY }}

Equivalent Mutants

Some mutants are "equivalent"—they don't change behavior:

// Example 1: Equivalent mutant
function absolute(n) {
    return n < 0 ? -n : n;
    // Mutant: < 0 → <= 0
    // For n=0: Original returns 0, mutant returns 0 (equivalent)
}

// Example 2: Equivalent mutant
function count(items) {
    let total = 0;
    for (let i = 0; i < items.length; i++) {
        total++; // Mutant: total++ → ++total
    }
    return total; // Both produce same result
}

// Example 3: Equivalent mutant
const MAX_RETRIES = 3;
function retry(fn) {
    for (let i = 0; i < MAX_RETRIES; i++) {
        // Mutant: i < MAX_RETRIES → i <= MAX_RETRIES
        // If MAX_RETRIES is constant, behavior unchanged
    }
}

// How to handle:
# infection.json5
{
    "mutators": {
        "@default": true,
        "IncrementInteger": {
            "ignore": [
                "App\Services\EquivalentClass::methodName"
            ]
        }
    }
}

Exercise 1: Install Infection PHP and run it on your project. Review the mutation report and identify the top 5 survived mutants in critical business logic. Write additional tests to kill those mutants and improve your mutation score by at least 20%.

Exercise 2: Set up Stryker for a JavaScript project. Focus on a single complex module (e.g., validation logic, calculation engine). Achieve a mutation score of 90%+ by adding comprehensive tests that cover boundary conditions, edge cases, and error scenarios.

Exercise 3: Create a CI/CD workflow that runs mutation tests weekly and on pull requests that modify critical files. Configure it to: (1) fail if MSI drops below 80%, (2) generate HTML reports, (3) post mutation score to PR comments, (4) track mutation score trends over time.

Summary

In this lesson, we've covered mutation testing comprehensively:

Understanding mutation testing as a measure of test quality, not just coverage
Learning common mutation operators (arithmetic, logical, conditionals, returns)
Using Infection PHP for mutation testing in PHP projects
Using Stryker for mutation testing in JavaScript/TypeScript projects
Identifying and fixing weak tests based on survived mutants
Recognizing common mutation patterns and how to kill them
Interpreting mutation testing metrics (MSI, Covered Code MSI)
Handling equivalent mutants appropriately
Integrating mutation testing into CI/CD pipelines

Mutation testing is the ultimate test of your test suite. While code coverage tells you what code is executed, mutation testing tells you if your tests would actually catch bugs. Use it to continuously improve your testing practices and deliver more reliable software.