Build a Flaky Test Detection Dashboard With Playwright and Grafana

Introduction: The Hidden Cost of Flaky Tests

Flaky tests are the silent productivity killer in every engineering organization. A test that passes 95% of the time sounds acceptable until you realize that in a suite of 500 tests, you will see approximately 25 false failures on every single run. Engineers learn to ignore failures, retry pipelines without investigating, and eventually stop trusting the test suite entirely. The result is that real bugs slip through because nobody believes the red signal anymore.

🎭 Want to master this with real projects? Join the Playwright Automation Mastery course at The Testing Academy.

The solution is not to eliminate flaky tests one at a time through heroic debugging sessions. The solution is to build a system that continuously monitors test reliability, identifies flaky tests before they erode confidence, and automatically quarantines the worst offenders. This article walks you through building a complete flaky test detection dashboard using Playwright custom reporters, InfluxDB for time-series storage, and Grafana for visualization and alerting.

Understanding Flakiness: What the Numbers Tell Us

A test is considered flaky when it produces different results on the same code without any code changes. The industry benchmark for acceptable test reliability is 99% pass rate per test. Any test that falls below this threshold is considered flaky and should be investigated. Tests below 95% should be quarantined immediately because they provide negative value, they waste more engineering time than they save.

The flakiness score formula quantifies the reliability of each test over a rolling window. The formula is: Flakiness Score = (Total Runs – Consistent Runs) / Total Runs * 100. A test that passed 97 out of 100 runs has a flakiness score of 3%. A test that passed 100 out of 100 runs has a flakiness score of 0%. The rolling window should be 14 days to account for intermittent issues while avoiding noise from one-off infrastructure problems.

Root causes of flakiness fall into predictable categories. Timing dependencies account for approximately 45% of flaky tests: tests that rely on setTimeout, animation completion, or network response ordering. Shared state accounts for 25%: tests that depend on data created by other tests or global state that is not properly isolated. Environment dependencies account for 20%: tests that behave differently on different OS versions, screen resolutions, or container resource limits. The remaining 10% are genuine race conditions in the application code that the test is correctly detecting.

Architecture Overview: The Monitoring Pipeline

The monitoring pipeline has four components. First, a custom Playwright reporter that captures test results with rich metadata after every test run. Second, an InfluxDB time-series database that stores these results efficiently and enables fast aggregation queries. Third, a Grafana dashboard that visualizes trends, identifies degrading tests, and provides drill-down capability. Fourth, alerting rules that notify the team when a test crosses reliability thresholds.

• Playwright Custom Reporter: Captures test name, status, duration, retry count, error message, and suite metadata
• InfluxDB: Time-series storage optimized for high write throughput and fast aggregation queries
• Grafana: Real-time dashboards with pass rate trends, flakiness scores, and worst offender rankings
• Alert Rules: Slack/PagerDuty notifications when any test drops below 99% reliability over 14 days

Building the Custom Playwright Reporter

Playwright supports custom reporters through a simple class-based interface. The reporter receives callbacks for test begin, test end, and suite end events. Our reporter captures comprehensive metadata for each test result and batches writes to InfluxDB for efficiency.

// reporters/influxdb-reporter.ts
import type {
  FullConfig, FullResult, Reporter, Suite, TestCase, TestResult
} from '@playwright/test/reporter';

interface TestMetric {
  testName: string;
  suiteName: string;
  filePath: string;
  status: string;
  duration: number;
  retryCount: number;
  errorMessage: string;
  browser: string;
  timestamp: number;
}

class InfluxDBReporter implements Reporter {
  private metrics: TestMetric[] = [];
  private influxUrl: string;
  private influxToken: string;
  private influxOrg: string;
  private influxBucket: string;

  constructor(options: {
    url?: string;
    token?: string;
    org?: string;
    bucket?: string;
  } = {}) {
    this.influxUrl = options.url || process.env.INFLUX_URL || 'http://localhost:8086';
    this.influxToken = options.token || process.env.INFLUX_TOKEN || '';
    this.influxOrg = options.org || process.env.INFLUX_ORG || 'testing';
    this.influxBucket = options.bucket || process.env.INFLUX_BUCKET || 'playwright';
  }

  onTestEnd(test: TestCase, result: TestResult): void {
    const metric: TestMetric = {
      testName: test.title,
      suiteName: test.parent.title,
      filePath: test.location.file,
      status: result.status,
      duration: result.duration,
      retryCount: result.retry,
      errorMessage: result.error?.message?.substring(0, 500) || '',
      browser: test.parent.project()?.name || 'unknown',
      timestamp: Date.now()
    };
    this.metrics.push(metric);
  }

  async onEnd(result: FullResult): Promise<void> {
    if (this.metrics.length === 0) return;

    const lines = this.metrics.map(m => {
      const tags = [
        `test_name=${this.escapeTag(m.testName)}`,
        `suite=${this.escapeTag(m.suiteName)}`,
        `status=${m.status}`,
        `browser=${m.browser}`
      ].join(',');

      const fields = [
        `duration=${m.duration}i`,
        `retry_count=${m.retryCount}i`,
        `passed=${m.status === 'passed' ? 1 : 0}i`,
        `failed=${m.status === 'failed' ? 1 : 0}i`,
        `flaky=${m.retryCount > 0 && m.status === 'passed' ? 1 : 0}i`,
        `error_message="${this.escapeField(m.errorMessage)}"`
      ].join(',');

      return `test_results,${tags} ${fields} ${m.timestamp * 1000000}`;
    });

    await this.writeToInflux(lines);
    console.log(`[InfluxDB Reporter] Wrote ${lines.length} test metrics`);
  }

  private async writeToInflux(lines: string[]): Promise<void> {
    const body = lines.join('\n');
    const url = `${this.influxUrl}/api/v2/write?org=${this.influxOrg}&bucket=${this.influxBucket}&precision=ns`;

    const response = await fetch(url, {
      method: 'POST',
      headers: {
        'Authorization': `Token ${this.influxToken}`,
        'Content-Type': 'text/plain'
      },
      body
    });

    if (!response.ok) {
      console.error(`[InfluxDB Reporter] Write failed: ${response.status}`);
    }
  }

  private escapeTag(value: string): string {
    return value.replace(/[, =]/g, '\\$&');
  }

  private escapeField(value: string): string {
    return value.replace(/["\\]/g, '\\$&').replace(/\n/g, ' ');
  }
}

export default InfluxDBReporter;

Configuring Playwright to Use the Reporter

Register the custom reporter in your Playwright configuration file. You can use it alongside the built-in HTML reporter so you maintain local debugging capability while also feeding metrics to InfluxDB.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html', { open: 'never' }],
    ['./reporters/influxdb-reporter.ts', {
      url: process.env.INFLUX_URL || 'http://localhost:8086',
      token: process.env.INFLUX_TOKEN,
      org: 'testing',
      bucket: 'playwright'
    }]
  ],
  retries: 2,  // Enable retries to detect flaky tests
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure'
  }
});

Setting Up InfluxDB for Test Metrics

InfluxDB is a time-series database optimized for high write throughput and fast aggregation queries. It is the ideal storage backend for test metrics because you need to write thousands of data points per CI run and query aggregations across millions of data points for trend analysis.

# docker-compose.yml for InfluxDB + Grafana
version: '3.8'
services:
  influxdb:
    image: influxdb:2.7
    ports:
      - "8086:8086"
    environment:
      DOCKER_INFLUXDB_INIT_MODE: setup
      DOCKER_INFLUXDB_INIT_USERNAME: admin
      DOCKER_INFLUXDB_INIT_PASSWORD: admin12345
      DOCKER_INFLUXDB_INIT_ORG: testing
      DOCKER_INFLUXDB_INIT_BUCKET: playwright
      DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: my-super-secret-token
    volumes:
      - influxdb-data:/var/lib/influxdb2

  grafana:
    image: grafana/grafana:10.2.0
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    depends_on:
      - influxdb

volumes:
  influxdb-data:
  grafana-data:

Grafana Dashboard Configuration

The Grafana dashboard provides four critical panels: an overall pass rate trend line, a flakiness score leaderboard showing the worst offenders, a test duration heatmap to spot performance regressions, and a real-time failure feed showing the latest failures with error messages.

{
  "dashboard": {
    "title": "Playwright Flaky Test Dashboard",
    "panels": [
      {
        "title": "Overall Pass Rate (14-day rolling)",
        "type": "timeseries",
        "targets": [{
          "query": "from(bucket: \"playwright\") |> range(start: -14d) |> filter(fn: (r) => r._measurement == \"test_results\") |> filter(fn: (r) => r._field == \"passed\") |> aggregateWindow(every: 1d, fn: mean) |> yield(name: \"pass_rate\")"
        }],
        "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
        "fieldConfig": {
          "defaults": {
            "unit": "percentunit",
            "thresholds": {
              "steps": [
                { "value": 0, "color": "red" },
                { "value": 0.95, "color": "yellow" },
                { "value": 0.99, "color": "green" }
              ]
            }
          }
        }
      },
      {
        "title": "Top 10 Flakiest Tests",
        "type": "table",
        "targets": [{
          "query": "from(bucket: \"playwright\") |> range(start: -14d) |> filter(fn: (r) => r._measurement == \"test_results\") |> filter(fn: (r) => r._field == \"flaky\") |> group(columns: [\"test_name\"]) |> mean() |> sort(columns: [\"_value\"], desc: true) |> limit(n: 10)"
        }],
        "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 }
      },
      {
        "title": "Test Duration Trends",
        "type": "heatmap",
        "targets": [{
          "query": "from(bucket: \"playwright\") |> range(start: -7d) |> filter(fn: (r) => r._measurement == \"test_results\") |> filter(fn: (r) => r._field == \"duration\") |> aggregateWindow(every: 1h, fn: mean)"
        }],
        "gridPos": { "x": 0, "y": 8, "w": 24, "h": 8 }
      },
      {
        "title": "Recent Failures",
        "type": "logs",
        "targets": [{
          "query": "from(bucket: \"playwright\") |> range(start: -24h) |> filter(fn: (r) => r._measurement == \"test_results\") |> filter(fn: (r) => r.status == \"failed\") |> filter(fn: (r) => r._field == \"error_message\")"
        }],
        "gridPos": { "x": 0, "y": 16, "w": 24, "h": 8 }
      }
    ]
  }
}

Alert Rules for Proactive Detection

Alerting transforms the dashboard from a passive monitoring tool into an active quality gate. Configure two types of alerts: a per-test alert that fires when any individual test drops below 99% pass rate over 14 days, and a suite-level alert that fires when the overall pass rate drops below 98%.

# Grafana alert rule configuration (YAML provisioning)
# grafana/provisioning/alerting/rules.yml
apiVersion: 1
groups:
  - orgId: 1
    name: flaky-test-alerts
    folder: Playwright
    interval: 1h
    rules:
      - uid: flaky-test-individual
        title: "Individual Test Below 99% Pass Rate"
        condition: C
        data:
          - refId: A
            queryType: ""
            relativeTimeRange:
              from: 1209600  # 14 days
              to: 0
            model:
              query: |
                from(bucket: "playwright")
                  |> range(start: -14d)
                  |> filter(fn: (r) => r._measurement == "test_results")
                  |> filter(fn: (r) => r._field == "passed")
                  |> group(columns: ["test_name"])
                  |> mean()
                  |> filter(fn: (r) => r._value < 0.99)
          - refId: C
            type: threshold
            conditions:
              - evaluator: { type: gt, params: [0] }
        annotations:
          summary: "Flaky test detected: {{ $labels.test_name }}"
        labels:
          severity: warning
        noDataState: OK
        execErrState: Error

      - uid: suite-pass-rate
        title: "Suite Pass Rate Below 98%"
        condition: C
        data:
          - refId: A
            model:
              query: |
                from(bucket: "playwright")
                  |> range(start: -24h)
                  |> filter(fn: (r) => r._measurement == "test_results")
                  |> filter(fn: (r) => r._field == "passed")
                  |> mean()
          - refId: C
            type: threshold
            conditions:
              - evaluator: { type: lt, params: [0.98] }
        annotations:
          summary: "Overall suite pass rate dropped below 98%"
        labels:
          severity: critical

Quarantine Automation: Auto-Isolating Flaky Tests

The most powerful feature of a flaky test monitoring system is automatic quarantine. When a test consistently fails to meet the reliability threshold, it should be automatically tagged and excluded from the critical CI gate. This prevents flaky tests from blocking deployments while ensuring they are still tracked and eventually fixed.

// scripts/quarantine-manager.ts
import { InfluxDB } from '@influxdata/influxdb-client';

interface FlakyTest {
  testName: string;
  passRate: number;
  totalRuns: number;
  lastFailure: string;
}

async function identifyFlakyTests(): Promise<FlakyTest[]> {
  const influx = new InfluxDB({
    url: process.env.INFLUX_URL || 'http://localhost:8086',
    token: process.env.INFLUX_TOKEN || ''
  });

  const queryApi = influx.getQueryApi('testing');
  const query = `
    from(bucket: "playwright")
      |> range(start: -14d)
      |> filter(fn: (r) => r._measurement == "test_results")
      |> filter(fn: (r) => r._field == "passed")
      |> group(columns: ["test_name"])
      |> mean()
      |> filter(fn: (r) => r._value < 0.99)
      |> sort(columns: ["_value"], desc: false)
  `;

  const results: FlakyTest[] = [];
  await queryApi.collectRows(query).then(rows => {
    rows.forEach((row: any) => {
      results.push({
        testName: row.test_name,
        passRate: row._value,
        totalRuns: row._total || 0,
        lastFailure: row._time
      });
    });
  });

  return results;
}

async function updateQuarantineFile(flakyTests: FlakyTest[]): Promise<void> {
  const fs = await import('fs/promises');

  const quarantine = {
    updatedAt: new Date().toISOString(),
    tests: flakyTests.map(t => ({
      name: t.testName,
      passRate: (t.passRate * 100).toFixed(1) + '%',
      quarantinedAt: new Date().toISOString(),
      reason: `Pass rate ${(t.passRate * 100).toFixed(1)}% over 14 days`
    }))
  };

  await fs.writeFile(
    'test-quarantine.json',
    JSON.stringify(quarantine, null, 2)
  );

  console.log(`Quarantined ${flakyTests.length} flaky tests`);
}

// Run quarantine check
identifyFlakyTests().then(updateQuarantineFile);

Using the Quarantine File in Playwright Config

Integrate the quarantine file with your Playwright configuration to automatically skip quarantined tests in the main CI pipeline while still running them in a separate quarantine pipeline for monitoring.

// playwright.config.ts - quarantine integration
import { defineConfig } from '@playwright/test';
import * as fs from 'fs';

// Load quarantine list
let quarantinedTests: string[] = [];
try {
  const quarantine = JSON.parse(
    fs.readFileSync('test-quarantine.json', 'utf-8')
  );
  quarantinedTests = quarantine.tests.map((t: any) => t.name);
} catch {
  // No quarantine file, run all tests
}

const isQuarantineRun = process.env.RUN_QUARANTINE === 'true';

export default defineConfig({
  grep: isQuarantineRun
    ? new RegExp(quarantinedTests.map(t =>
        t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
      ).join('|'))
    : undefined,
  grepInvert: !isQuarantineRun && quarantinedTests.length > 0
    ? new RegExp(quarantinedTests.map(t =>
        t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
      ).join('|'))
    : undefined,
  retries: isQuarantineRun ? 3 : 1,
  reporter: [
    ['./reporters/influxdb-reporter.ts'],
    ['html', { open: 'never' }]
  ]
});

Tracking Flakiness Score Over Time

The most valuable metric on the dashboard is the flakiness trend over time. A decreasing flakiness score means your team is successfully stabilizing the test suite. An increasing score means new flaky tests are being introduced faster than old ones are being fixed. The goal is to maintain an overall suite flakiness score below 1%.

Create a Grafana panel that shows the weekly average flakiness score as a time series. Overlay it with the total test count to see whether increasing flakiness correlates with suite growth. Teams that add tests without monitoring reliability often see their flakiness score climb steadily, eventually reaching a tipping point where the entire CI pipeline becomes unreliable.

CI/CD Integration: GitHub Actions Example

Integrate the flaky test pipeline into your CI/CD workflow. The following GitHub Actions configuration runs the main test suite, exports metrics to InfluxDB, and runs the quarantine suite separately.

# .github/workflows/playwright.yml
name: Playwright Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps

      - name: Run Main Test Suite
        run: npx playwright test
        env:
          INFLUX_URL: ${{ secrets.INFLUX_URL }}
          INFLUX_TOKEN: ${{ secrets.INFLUX_TOKEN }}

      - name: Run Quarantine Suite
        if: always()
        run: npx playwright test
        env:
          RUN_QUARANTINE: 'true'
          INFLUX_URL: ${{ secrets.INFLUX_URL }}
          INFLUX_TOKEN: ${{ secrets.INFLUX_TOKEN }}

      - name: Update Quarantine List
        if: github.ref == 'refs/heads/main'
        run: npx ts-node scripts/quarantine-manager.ts
        env:
          INFLUX_URL: ${{ secrets.INFLUX_URL }}
          INFLUX_TOKEN: ${{ secrets.INFLUX_TOKEN }}

Best Practices for Flaky Test Management

Set a team-wide target for maximum flakiness score, typically 1% or lower. Review the flakiest tests weekly in a short stand-up meeting. Assign ownership of flaky tests to the team that owns the feature, not to a centralized QA team. Track time-to-fix for quarantined tests and set a maximum quarantine duration, after which the test is either fixed or deleted. Never let quarantined tests accumulate indefinitely.

Document the root cause of every flaky test that gets fixed. Over time, this creates a knowledge base that helps the team avoid introducing similar flakiness in new tests. Common patterns emerge quickly: most teams discover that 80% of their flakiness comes from three or four root causes that can be addressed with helper utilities or linting rules.

Measuring Dashboard ROI

To justify the investment in flaky test monitoring, track three metrics before and after implementation. First, CI pipeline retry rate: how often do developers retry a failed pipeline without investigating? This typically drops 60-80% after implementing flaky test quarantine. Second, mean time to merge: the average time from PR creation to merge. This drops because developers no longer wait for flaky re-runs. Third, developer confidence: survey developers on their trust in the test suite. Teams with flaky test dashboards consistently report higher confidence in their CI signals.

Conclusion: Build Visibility, Build Trust

A test suite is only as valuable as the trust the team places in it. Flaky tests erode that trust slowly and silently until the entire suite becomes a formality that everyone ignores. Building a flaky test detection dashboard transforms flakiness from an invisible problem into a visible, measurable, and actionable metric. The combination of Playwright custom reporters, InfluxDB time-series storage, and Grafana visualization gives you the tools to maintain a reliable test suite at any scale. Start by deploying the custom reporter shown in this article. Within a week, you will have enough data to identify your worst offenders and begin the systematic cleanup that restores trust in your CI pipeline.

Frequently Asked Questions