Testing & TDD

Visual Regression Testing

26 min Lesson 24 of 35

Visual Regression Testing

Visual regression testing detects unintended visual changes in your application by comparing screenshots over time. In this lesson, we'll explore screenshot comparison techniques, tools like Percy and Chromatic, visual diff algorithms, and strategies for maintaining visual consistency across releases.

Why Visual Regression Testing Matters

Visual regression testing helps you:

Catch unintended CSS changes that break layouts
Detect cross-browser rendering differences
Verify responsive design across different screen sizes
Prevent visual bugs from reaching production
Maintain consistent branding and design systems
Automate visual QA that traditionally requires manual inspection
Document visual changes for design review

Key Concept: Visual regression testing complements functional testing. While unit and integration tests verify behavior, visual tests verify appearance. Both are essential for comprehensive quality assurance.

How Visual Regression Testing Works

The basic workflow involves four steps:

Baseline Creation: Capture screenshots of your application in a known-good state
Test Execution: After code changes, capture new screenshots of the same views
Visual Comparison: Compare new screenshots pixel-by-pixel with baselines
Review & Approve: Review differences and approve intentional changes or reject bugs

Basic Screenshot Testing with Puppeteer

Start with simple screenshot capture using Puppeteer (Node.js):

// visual-test.js
const puppeteer = require('puppeteer');
const fs = require('fs');
const pixelmatch = require('pixelmatch');
const { PNG } = require('pngjs');

async function captureScreenshot(url, outputPath) {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    // Set consistent viewport
    await page.setViewport({ width: 1280, height: 720 });

    await page.goto(url, { waitUntil: 'networkidle0' });

    // Wait for fonts and images to load
    await page.evaluateHandle('document.fonts.ready');

    await page.screenshot({
        path: outputPath,
        fullPage: true,
    });

    await browser.close();
}

async function compareScreenshots(baselinePath, currentPath, diffPath) {
    const baseline = PNG.sync.read(fs.readFileSync(baselinePath));
    const current = PNG.sync.read(fs.readFileSync(currentPath));

    const { width, height } = baseline;
    const diff = new PNG({ width, height });

    const numDiffPixels = pixelmatch(
        baseline.data,
        current.data,
        diff.data,
        width,
        height,
        { threshold: 0.1 }
    );

    fs.writeFileSync(diffPath, PNG.sync.write(diff));

    return numDiffPixels;
}

// Usage
(async () => {
    const url = 'http://localhost:3000/';

    // Capture current screenshot
    await captureScreenshot(url, './screenshots/current.png');

    // Compare with baseline
    if (fs.existsSync('./screenshots/baseline.png')) {
        const diffPixels = await compareScreenshots(
            './screenshots/baseline.png',
            './screenshots/current.png',
            './screenshots/diff.png'
        );

        console.log(`Diff pixels: ${diffPixels}`);

        if (diffPixels > 100) { // Threshold
            console.error('Visual regression detected!');
            process.exit(1);
        }
    } else {
        // First run - create baseline
        fs.copyFileSync(
            './screenshots/current.png',
            './screenshots/baseline.png'
        );
        console.log('Baseline created');
    }
})();

Laravel Dusk for Visual Testing

Use Laravel Dusk for browser-based visual testing in PHP applications:

<?php

namespace Tests\Browser;

use Laravel\Dusk\Browser;
use Tests\DuskTestCase;

class VisualRegressionTest extends DuskTestCase
{
    protected $screenshotPath = 'tests/Browser/screenshots';
    protected $baselinePath = 'tests/Browser/baseline';

    public function test_homepage_visual_regression()
    {
        $this->browse(function (Browser $browser) {
            $browser->visit('/')
                    ->waitFor('#app')
                    ->resize(1280, 720);

            // Wait for dynamic content to load
            $browser->pause(1000);

            $screenshotName = 'homepage';
            $browser->screenshot($screenshotName);

            $this->compareScreenshots($screenshotName);
        });
    }

    public function test_product_page_visual_regression()
    {
        $this->browse(function (Browser $browser) {
            $product = Product::factory()->create();

            $browser->visit("/products/{$product->id}")
                    ->waitFor('.product-details')
                    ->resize(1280, 720)
                    ->screenshot('product-page');

            $this->compareScreenshots('product-page');
        });
    }

    public function test_responsive_layouts()
    {
        $this->browse(function (Browser $browser) {
            $viewports = [
                ['mobile', 375, 667],
                ['tablet', 768, 1024],
                ['desktop', 1920, 1080],
            ];

            foreach ($viewports as [$name, $width, $height]) {
                $browser->visit('/')
                        ->resize($width, $height)
                        ->pause(500)
                        ->screenshot("homepage-{$name}");

                $this->compareScreenshots("homepage-{$name}");
            }
        });
    }

    protected function compareScreenshots($name)
    {
        $currentPath = "{$this->screenshotPath}/{$name}.png";
        $baselinePath = "{$this->baselinePath}/{$name}.png";
        $diffPath = "{$this->screenshotPath}/{$name}-diff.png";

        if (!file_exists($baselinePath)) {
            // Create baseline on first run
            copy($currentPath, $baselinePath);
            $this->markTestSkipped('Baseline created for ' . $name);
            return;
        }

        // Use ImageMagick to compare
        exec(
            "compare -metric AE {$baselinePath} {$currentPath} {$diffPath} 2>&1",
            $output,
            $returnCode
        );

        $diffPixels = (int) $output[0];

        // Allow small differences (anti-aliasing, rendering variations)
        $threshold = 100;

        $this->assertLessThan(
            $threshold,
            $diffPixels,
            "Visual regression detected: {$diffPixels} pixels differ"
        );
    }
}

Percy for Automated Visual Testing

Percy is a commercial visual testing platform with excellent CI/CD integration:

# Install Percy CLI
npm install --save-dev @percy/cli @percy/puppeteer

# percy-test.js
const puppeteer = require('puppeteer');
const percySnapshot = require('@percy/puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    // Homepage
    await page.goto('http://localhost:3000/');
    await percySnapshot(page, 'Homepage');

    // Products page
    await page.goto('http://localhost:3000/products');
    await percySnapshot(page, 'Products Page');

    // Product detail
    await page.goto('http://localhost:3000/products/1');
    await percySnapshot(page, 'Product Detail');

    // Responsive snapshots
    await page.setViewport({ width: 375, height: 667 });
    await page.goto('http://localhost:3000/');
    await percySnapshot(page, 'Homepage Mobile');

    await page.setViewport({ width: 768, height: 1024 });
    await percySnapshot(page, 'Homepage Tablet');

    await browser.close();
})();

# Run with Percy
# Set PERCY_TOKEN environment variable
export PERCY_TOKEN=your_token_here
npx percy exec -- node percy-test.js

# Percy compares screenshots in their cloud
# View results at: https://percy.io/your-org/your-project

Best Practice: Run visual tests in a consistent environment (same OS, browser version, fonts) to avoid false positives from rendering differences. Use Docker containers or CI/CD services that provide consistent environments.

Chromatic for Storybook

Chromatic integrates with Storybook for component-level visual testing:

# Install Chromatic
npm install --save-dev chromatic

# Ensure you have Storybook stories
// Button.stories.js
import Button from './Button';

export default {
    title: 'Components/Button',
    component: Button,
};

export const Primary = () => <Button variant="primary">Click Me</Button>;
export const Secondary = () => <Button variant="secondary">Cancel</Button>;
export const Disabled = () => <Button disabled>Disabled</Button>;
export const Loading = () => <Button loading>Loading...</Button>;

# Run Chromatic
npx chromatic --project-token=your_token_here

# Chromatic automatically captures screenshots of all stories
# and compares them with baselines

# CI/CD Integration (.github/workflows/chromatic.yml)
name: Visual Tests

on: push

jobs:
  chromatic:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Install dependencies
        run: npm ci

      - name: Run Chromatic
        uses: chromaui/action@v1
        with:
          projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
          exitZeroOnChanges: true

BackstopJS for Comprehensive Visual Testing

BackstopJS is a powerful open-source visual regression tool:

# Install BackstopJS
npm install -g backstopjs

# Initialize configuration
backstop init

# backstop.json
{
  "id": "my_app_visual_tests",
  "viewports": [
    {
      "label": "phone",
      "width": 375,
      "height": 667
    },
    {
      "label": "tablet",
      "width": 768,
      "height": 1024
    },
    {
      "label": "desktop",
      "width": 1920,
      "height": 1080
    }
  ],
  "scenarios": [
    {
      "label": "Homepage",
      "url": "http://localhost:3000/",
      "delay": 1000,
      "misMatchThreshold": 0.1
    },
    {
      "label": "Products Page",
      "url": "http://localhost:3000/products",
      "delay": 1000,
      "selectors": [".product-grid"],
      "misMatchThreshold": 0.1
    },
    {
      "label": "Login Form",
      "url": "http://localhost:3000/login",
      "delay": 500,
      "selectors": ["form"],
      "hideSelectors": ["#dynamic-ad"],
      "misMatchThreshold": 0.1
    },
    {
      "label": "Dashboard - Authenticated",
      "url": "http://localhost:3000/dashboard",
      "delay": 2000,
      "cookiePath": "backstop_data/cookies.json",
      "misMatchThreshold": 0.1
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "html_report": "backstop_data/html_report"
  },
  "engine": "puppeteer",
  "report": ["browser", "CI"],
  "debug": false
}

# Create baseline (reference screenshots)
backstop reference

# Run tests (compare current with baseline)
backstop test

# Approve changes (update baseline)
backstop approve

# Generate interactive HTML report
# Opens automatically after test run

Handling Dynamic Content

Dynamic content (timestamps, animations, ads) can cause false positives. Here's how to handle them:

// Hide dynamic elements before screenshot
await page.evaluate(() => {
    // Hide elements with dynamic content
    document.querySelectorAll('.timestamp, .live-chat').forEach(el => {
        el.style.visibility = 'hidden';
    });

    // Disable animations
    document.querySelectorAll('*').forEach(el => {
        el.style.animation = 'none';
        el.style.transition = 'none';
    });
});

// Replace dynamic text with placeholder
await page.evaluate(() => {
    document.querySelectorAll('.timestamp').forEach(el => {
        el.textContent = '2024-01-01 00:00:00';
    });

    document.querySelectorAll('.random-id').forEach(el => {
        el.textContent = 'PLACEHOLDER_ID';
    });
});

// Freeze time for consistent timestamps
await page.evaluateOnNewDocument(() => {
    const constantDate = new Date('2024-01-01T00:00:00Z');
    Date = class extends Date {
        constructor(...args) {
            if (args.length === 0) {
                super(constantDate);
            } else {
                super(...args);
            }
        }
    };
    Date.now = () => constantDate.getTime();
});

// Wait for loading states to complete
await page.waitForSelector('.skeleton-loader', { hidden: true });
await page.waitForFunction(() => {
    return !document.querySelector('.loading');
});

Visual Testing Best Practices

Follow these practices for reliable visual testing:

// 1. Use consistent viewport sizes
const viewports = {
    mobile: { width: 375, height: 667 },
    tablet: { width: 768, height: 1024 },
    desktop: { width: 1920, height: 1080 },
};

// 2. Wait for fonts to load
await page.evaluateHandle('document.fonts.ready');

// 3. Wait for images to load
await page.evaluate(() => {
    return Promise.all(
        Array.from(document.images)
            .filter(img => !img.complete)
            .map(img => new Promise(resolve => {
                img.onload = img.onerror = resolve;
            }))
    );
});

// 4. Disable CSS animations
await page.addStyleTag({
    content: `
        *, *::before, *::after {
            animation-duration: 0s !important;
            transition-duration: 0s !important;
        }
    `
});

// 5. Set acceptable threshold
const threshold = 0.1; // 0.1% pixel difference allowed

// 6. Capture specific elements, not full page
await page.screenshot({
    path: 'screenshot.png',
    clip: {
        x: 0,
        y: 0,
        width: 1280,
        height: 720,
    }
});

// 7. Use data attributes to identify dynamic content
// In HTML: <div data-visual-test-ignore>Dynamic content</div>
await page.evaluate(() => {
    document.querySelectorAll('[data-visual-test-ignore]').forEach(el => {
        el.style.visibility = 'hidden';
    });
});

Warning: Visual tests are slower and more brittle than unit tests. Use them strategically for critical UI components and user flows, not every single page. Combine with functional tests for comprehensive coverage.

CI/CD Integration

Integrate visual tests into your continuous integration pipeline:

# GitHub Actions example
# .github/workflows/visual-tests.yml
name: Visual Regression Tests

on:
  pull_request:
    branches: [main]

jobs:
  visual-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Build application
        run: npm run build

      - name: Start application server
        run: npm start &
        env:
          NODE_ENV: test

      - name: Wait for server
        run: npx wait-on http://localhost:3000

      - name: Run BackstopJS tests
        run: |
          npm install -g backstopjs
          backstop test || backstop test --docker

      - name: Upload diff images
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: visual-test-diffs
          path: backstop_data/bitmaps_test/**/

      - name: Comment PR with results
        if: failure()
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '⚠️ Visual regression detected! Check artifacts for diff images.'
            })

# Percy CI Integration
      - name: Run Percy tests
        run: npx percy exec -- node visual-tests.js
        env:
          PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}

Component-Level Visual Testing

Test individual components in isolation:

// component-visual-test.js
const puppeteer = require('puppeteer');
const percySnapshot = require('@percy/puppeteer');

async function testComponent(componentName, states) {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    for (const state of states) {
        // Navigate to component showcase page
        await page.goto(`http://localhost:6006/iframe.html?id=${componentName}--${state}`);

        // Wait for component to render
        await page.waitForSelector('#root > *');

        // Capture screenshot
        await percySnapshot(page, `${componentName} - ${state}`);
    }

    await browser.close();
}

// Test button component in all states
testComponent('button', [
    'default',
    'primary',
    'secondary',
    'disabled',
    'loading',
    'with-icon',
]);

// Test form components
testComponent('input-field', [
    'empty',
    'filled',
    'error',
    'disabled',
    'focused',
]);

Exercise 1: Set up BackstopJS for your project with scenarios covering: (1) homepage at 3 viewport sizes, (2) navigation menu open/closed states, (3) form validation states (empty, filled, error), (4) modal dialogs, and (5) data tables with pagination. Configure appropriate thresholds and hiding of dynamic elements.

Exercise 2: Create a custom visual regression testing framework using Puppeteer and pixelmatch. Implement: (1) baseline management, (2) automatic diff generation, (3) HTML report with side-by-side comparisons, (4) configurable thresholds per test, and (5) ability to approve/reject changes.

Exercise 3: Integrate Percy or Chromatic into your CI/CD pipeline. Configure it to: (1) run on all pull requests, (2) block merges if visual regressions are detected, (3) post diff images to PR comments, (4) automatically approve changes on the main branch after manual review.

Summary

In this lesson, we've covered comprehensive visual regression testing strategies:

Understanding how visual regression testing complements functional testing
Implementing basic screenshot comparison with Puppeteer and pixelmatch
Using Laravel Dusk for PHP-based visual testing
Leveraging commercial platforms like Percy and Chromatic
Setting up BackstopJS for open-source visual testing
Handling dynamic content and animations in visual tests
Best practices for reliable and maintainable visual tests
Integrating visual tests into CI/CD pipelines
Component-level visual testing strategies

Visual regression testing catches UI bugs that automated functional tests miss. By integrating visual testing into your development workflow, you ensure consistent, polished user experiences across all releases.