Automatic Performance Regression Detection in CI/CD

Performance regressions are often introduced incrementally — a slightly larger image here, a new third-party script there, a less-optimized component that gets added with good intentions. Without automated detection, these regressions accumulate unseen until performance is visibly poor. By then, identifying the culprit requires digging through months of git history.

Automatic performance regression detection catches these issues at the pull request stage, when the fix is a single-line change rather than a multi-week investigation.

What a Performance Regression Looks Like

A performance regression is any change that meaningfully worsens a performance metric. "Meaningfully" is key — some variation is expected, and alerting on every small fluctuation creates noise.

Regression detection thresholds (common approach):

• LCP increased by > 20%
• Bundle size increased by > 10KB
• Lighthouse score decreased by > 5 points
• TBT increased by > 50ms

These thresholds balance sensitivity (catching real regressions) with specificity (avoiding false alarms from normal variation).

Building a CI/CD Performance Testing Pipeline

Step 1: Establish a Performance Baseline

Before you can detect regressions, you need a stable baseline. Run Lighthouse on your main/production branch multiple times and record median values:

# Run Lighthouse 5 times, report median
for i in {1..5}; do
  lighthouse https://yoursite.com --output json --output-path ./runs/run-$i.json
done

# Use Lighthouse CI to aggregate
lhci open

Step 2: Add Lighthouse CI to Your Pipeline

# .github/workflows/performance.yml
name: Performance CI

on:
  pull_request:
    branches: [main]

jobs:
  performance-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Build application
        run: npm run build
      
      - name: Start application
        run: npm start &
      
      - name: Wait for application
        run: npx wait-on http://localhost:3000 --timeout 60000
      
      - name: Run Lighthouse CI
        run: npx lhci autorun
        env:
          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}

// lighthouserc.json
{
  "ci": {
    "collect": {
      "url": ["http://localhost:3000", "http://localhost:3000/products"],
      "numberOfRuns": 3
    },
    "assert": {
      "assertions": {
        "largest-contentful-paint": ["warn", { "maxNumericValue": 2500 }],
        "cumulative-layout-shift": ["error", { "maxNumericValue": 0.1 }],
        "total-blocking-time": ["warn", { "maxNumericValue": 300 }]
      }
    },
    "upload": {
      "target": "lhci",
      "serverBaseUrl": "https://your-lhci-server.com"
    }
  }
}

Step 3: Add Bundle Size Checks

Use size-limit to enforce JavaScript bundle size budgets:

// package.json
{
  "size-limit": [
    {
      "path": "dist/js/main.*.js",
      "limit": "300 kB"
    },
    {
      "path": "dist/js/vendors.*.js",
      "limit": "400 kB"
    }
  ],
  "scripts": {
    "size": "size-limit",
    "analyze": "size-limit --why"
  }
}

# Add to CI pipeline
- name: Check bundle size
  run: npm run size

Step 4: Compare Against Main Branch

The most useful regression detection compares PR performance against the main branch, not against fixed thresholds:

// Custom script: Compare PR Lighthouse scores against main
const { execSync } = require('child_process');
const fs = require('fs');

// Assumes you've already run Lighthouse for both PR and main
const prResults = JSON.parse(fs.readFileSync('pr-lighthouse.json'));
const mainResults = JSON.parse(fs.readFileSync('main-lighthouse.json'));

const REGRESSION_THRESHOLD = 0.05;  // 5% regression triggers warning

const metrics = [
  { key: 'largest-contentful-paint', name: 'LCP' },
  { key: 'cumulative-layout-shift', name: 'CLS' },
  { key: 'total-blocking-time', name: 'TBT' }
];

let hasRegression = false;

for (const metric of metrics) {
  const prValue = prResults.audits[metric.key].numericValue;
  const mainValue = mainResults.audits[metric.key].numericValue;
  const change = (prValue - mainValue) / mainValue;
  
  if (change > REGRESSION_THRESHOLD) {
    console.log(`⚠️ ${metric.name} REGRESSION: ${mainValue.toFixed(0)}ms → ${prValue.toFixed(0)}ms (+${(change * 100).toFixed(1)}%)`);
    hasRegression = true;
  } else {
    console.log(`✓ ${metric.name}: ${mainValue.toFixed(0)}ms → ${prValue.toFixed(0)}ms (${(change * 100).toFixed(1)}%)`);
  }
}

if (hasRegression) {
  process.exit(1);  // Fail the CI check
}

Post-Deployment Production Monitoring

CI/CD testing catches regressions from code changes. But performance can also regress in production due to:

• Content changes (larger images added via CMS)
• Third-party script updates
• Traffic pattern changes
• Infrastructure changes (CDN configuration)

Continuous production monitoring with AzMonitor catches these post-deployment regressions:

# AzMonitor: Daily performance baseline checks
performance_monitor:
  pages:
    - url: https://yoursite.com
      thresholds:
        lcp: 2500
        cls: 0.1
      alert_on_regression: 20%  # Alert if 20% worse than 7-day average
  schedule: every_6_hours
  alert_channel: slack-performance

PR Comments with Performance Data

The most effective regression prevention shows performance data directly in the PR:

# Using Lighthouse CI with GitHub app
- name: Run LHCI
  run: lhci autorun
  env:
    LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}

This generates a comment on the PR showing performance comparison:

Performance Budget Check

| Metric | Main | PR | Change |
|--------|------|-----|--------|
| LCP | 1.8s | 2.4s | +33% ⚠️ |
| CLS | 0.05 | 0.04 | -20% ✓ |
| TBT | 120ms | 115ms | -4% ✓ |
| Bundle Size | 298KB | 342KB | +15% ❌ |

Reviewers can see the performance impact of a PR before merging, creating a culture where performance is discussed as part of code review.

Setting Up the Full Pipeline

1. Lighthouse CI: Performance scoring in PR checks
2. size-limit or bundlesize: JavaScript bundle enforcement
3. Image size linting: Prevent oversized images from being committed
4. AzMonitor production monitoring: Continuous post-deployment surveillance

Together, these layers catch performance regressions at every stage: code review, deployment, and ongoing production monitoring.

Add production performance monitoring with AzMonitor to complete your regression detection pipeline. CI/CD catches code changes; AzMonitor catches everything else.