|

Playwright Retries and Flaky Tests: Day 42

ByPromode
Playwright retries tutorial for flaky test quarantine in TypeScript

Playwright retries are useful, but they become dangerous when teams use them to hide broken test design. Day 42 of the Playwright + TypeScript series is about a better pattern: retry once, collect evidence, classify the failure, and quarantine only the tests that deserve quarantine.

I see many QA teams add retries: 2 in playwright.config.ts and call the problem solved. The build becomes green for a week, then the same product area starts failing at 2 AM during release night. Retries should buy debugging signal, not silence.

Table of Contents

Contents

Why Playwright retries exist

The official Playwright retries documentation describes retries as automatic re-runs for tests that fail intermittently. That wording matters. A retry is for intermittent failure investigation, not for a selector that is permanently wrong or a test that asserts the wrong business rule.

Playwright classifies outcomes in a way that helps managers read the build. A test that passes on the first run is passed. A test that fails on the first run and passes on retry is flaky. A test that fails on the original run and all retries is failed. That distinction gives you a clean triage queue.

Retries are not a quality strategy

A quality strategy answers questions like these:

  • Which user journeys must block a release?
  • Which failures need immediate engineering ownership?
  • Which tests are noisy because of environment, data, or timing?
  • Which tests should move to API, component, or unit level?

Playwright retries answer only one smaller question: if the test gets one more clean run, does it still fail? That is useful evidence, but it is not a fix.

The signal I want from a retry

When a test passes after retry, I want to know three things quickly:

  1. Did the app eventually behave correctly, or did the assertion become weaker?
  2. Did Playwright capture a trace, screenshot, video, or console log from the failed attempt?
  3. Can I group this failure with other failures by feature, owner, or root cause?

This is why retries work best with a trace-first setup. If a retry gives me a green build but no evidence from the failing attempt, I only moved the pain to tomorrow.

A practical retry policy for TypeScript projects

My default policy is boring: no retries locally, one retry in CI, and trace capture on the first retry. Local runs should expose problems quickly. CI runs should survive a one-off infrastructure wobble while still preserving evidence.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  timeout: 30_000,
  expect: {
    timeout: 7_000,
  },
  retries: process.env.CI ? 1 : 0,
  workers: process.env.CI ? 4 : undefined,
  reporter: [
    ['list'],
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['json', { outputFile: 'test-results/results.json' }],
  ],
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

The Playwright Trace Viewer guide explains why traces are valuable for CI debugging. A trace is not only a screenshot. It shows actions, DOM snapshots, network calls, console messages, and timing. That is exactly what I need when the first attempt fails and the retry passes.

Use project-level retry overrides when needed

Some teams test against multiple projects: Chromium, Firefox, WebKit, mobile viewport, staging, and production smoke. I do not give every project the same retry budget. A production smoke suite should be stricter. A full staging regression suite may allow one retry while the team is stabilizing infrastructure.

// playwright.config.ts
export default defineConfig({
  retries: 0,
  projects: [
    {
      name: 'prod-smoke',
      testMatch: /.*\.smoke\.spec\.ts/,
      retries: 0,
    },
    {
      name: 'staging-regression',
      testMatch: /.*\.regression\.spec\.ts/,
      retries: process.env.CI ? 1 : 0,
    },
  ],
});

This small split prevents a common release problem. The team does not accidentally treat a production smoke failure and a staging-only timing issue as the same class of risk.

Keep the retry number small

I rarely allow more than one retry for product UI tests. Two or three retries can make a broken test look stable if the system eventually lands in the expected state. If a test needs three attempts, the test is not healthy enough to protect a release.

For India-based teams working in service delivery setups, this matters. A client report that says “green after 3 retries” is not the same as a stable suite. In product companies, the same pattern becomes expensive because every flaky test burns developer attention during sprint hardening.

Trace-first debugging for retried failures

Playwright retries become powerful when every flaky result produces a trace. The trace tells me whether the failure belongs to the test, the environment, the data setup, or the application.

Open the trace from a failed retry

After CI uploads the report, I want the engineer to open the trace and answer a short checklist. The screenshot description I include in team docs is simple: “Open the HTML report, click the flaky test, expand the first failed attempt, then open the trace attachment. The left panel shows actions, the middle panel shows DOM snapshots, and the right panel shows network and console details.”

npx playwright show-report playwright-report
# or open a specific trace zip
npx playwright show-trace test-results/path-to-trace.zip

If the failure happened in CI, I prefer downloading the report artifact instead of rerunning immediately. A rerun can destroy the original evidence. GitHub documents artifact storage for workflows in its artifact upload and download guide, and the same idea applies in GitLab, Jenkins, Azure DevOps, and CircleCI.

Classify the root cause before editing the test

Before I touch the test code, I assign one root cause label:

  • Product bug: the app shows the wrong behavior.
  • Selector issue: the test uses brittle CSS, text, or positional selectors.
  • Test data issue: the account, feature flag, seed data, or cleanup is unstable.
  • Environment issue: API, queue, CDN, browser dependency, or test server is unstable.
  • Timing issue: the test waits for the wrong signal.

This classification prevents random fixes. Without it, people add waitForTimeout(5000), update an assertion, and call the test stable. That is how suites rot.

Make the test wait for business state

Most flaky UI tests do not need more time. They need a better signal. Instead of waiting for a spinner to disappear, wait for the invoice row, order status, toast, or API response that proves the business action completed.

import { test, expect } from '@playwright/test';

test('customer can approve an invoice', async ({ page }) => {
  await page.goto('/invoices');

  await page.getByRole('row', { name: /INV-1001/ })
    .getByRole('button', { name: 'Approve' })
    .click();

  await expect(page.getByRole('status'))
    .toHaveText(/invoice approved/i);

  await expect(page.getByRole('row', { name: /INV-1001 approved/i }))
    .toBeVisible();
});

If this test flakes, the trace will show whether the row never changed, the toast did not appear, or the selector pointed at the wrong invoice. That is far better than a blind sleep.

Build a flaky test quarantine without lying

A quarantine is not a place to bury failures. It is a controlled list of tests that do not block the main release gate while an owner fixes them. The Playwright annotations documentation covers tags and annotations in reports. I use those features to make quarantine visible.

Tag quarantined tests explicitly

I like a plain tag such as @quarantine. It is boring and searchable. It also works with grep filters in CI.

import { test, expect } from '@playwright/test';

test('checkout applies saved coupon @quarantine', async ({ page }, testInfo) => {
  testInfo.annotations.push({
    type: 'issue',
    description: 'QA-1842: fails when coupon cache is cold in staging',
  });

  await page.goto('/checkout');
  await page.getByLabel('Coupon code').fill('SAVE10');
  await page.getByRole('button', { name: 'Apply' }).click();
  await expect(page.getByText('10% discount applied')).toBeVisible();
});

The test still exists. The owner and issue link are visible. The team can run quarantined tests on a separate schedule and review them daily.

Exclude quarantine from the release gate

The main pipeline should exclude quarantined tests. A second pipeline should run only quarantined tests and publish a report. That second report is the debt list.

# Main release gate
npx playwright test --grep-invert @quarantine

# Flaky debt job
npx playwright test --grep @quarantine --retries=1

This is honest. The release gate tells you whether trusted tests pass. The quarantine job tells you whether known debt is improving or growing.

Set expiry rules

Every quarantined test needs an expiry date or review date. My simple rule is seven calendar days for critical journeys and fourteen days for lower-risk flows. If the owner cannot fix it within that window, the team must either delete the test, rewrite it at the right layer, or escalate the product instability.

For a manager, this gives a clean weekly review. Count quarantined tests by owner, feature, and age. A growing quarantine list is a delivery risk, not a QA housekeeping issue.

Wire retries into CI without hiding risk

The CI workflow should make flaky results visible. I want the build summary to show passed, failed, flaky, skipped, and quarantined counts. I also want the HTML report uploaded every time, even on failure.

name: playwright-tests

on:
  pull_request:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run trusted tests
        run: npx playwright test --grep-invert @quarantine
        env:
          CI: true
      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report
          retention-days: 7

The official Playwright CI guide shows CI setup patterns, including browser installation. I keep the workflow small first, then add sharding, caching, and separate smoke jobs only when the baseline is stable. If you are already splitting suites, read my internal companion on Playwright CI sharding with TypeScript.

Fail the build when flakiness crosses a threshold

Some teams allow flaky tests forever because the final status is green. I prefer a simple threshold. If more than three tests are flaky in a pull request, fail the build and force triage. Adjust the number to your suite size, but make the rule visible.

// scripts/check-flaky-budget.ts
import fs from 'node:fs';

type TestResult = {
  status: string;
  retry: number;
};

type Spec = {
  title: string;
  tests: Array<{ results: TestResult[] }>;
};

const report = JSON.parse(fs.readFileSync('test-results/results.json', 'utf-8'));
const flakySpecs: string[] = [];

function visitSuite(suite: any) {
  for (const spec of suite.specs ?? []) {
    const becameGreenAfterRetry = spec.tests.some((test: any) => {
      const failedFirst = test.results?.[0]?.status === 'failed';
      const passedLater = test.results?.some((r: TestResult) => r.retry > 0 && r.status === 'passed');
      return failedFirst && passedLater;
    });

    if (becameGreenAfterRetry) flakySpecs.push(spec.title);
  }

  for (const child of suite.suites ?? []) visitSuite(child);
}

for (const suite of report.suites ?? []) visitSuite(suite);

console.log(`Flaky specs: ${flakySpecs.length}`);
for (const title of flakySpecs) console.log(`- ${title}`);

const budget = Number(process.env.FLAKY_BUDGET ?? '3');
if (flakySpecs.length > budget) {
  process.exitCode = 1;
}

Run this after Playwright and before closing the job. The point is not to punish teams. The point is to stop silent decay.

Report flaky tests like an engineering signal

A flaky test report should be readable by QA, developers, and managers. It should not be a 900-line console dump. My preferred report has four sections:

  1. New flaky tests introduced in this run.
  2. Existing quarantined tests that still fail.
  3. Quarantined tests that passed three consecutive runs and can return to the gate.
  4. Top root causes by count.

Turn JSON into a small Markdown summary

This script reads Playwright JSON and writes a pull request comment body. You can post it with a GitHub Action later, but the local file is useful even before automation.

// scripts/flaky-summary.ts
import fs from 'node:fs';

const report = JSON.parse(fs.readFileSync('test-results/results.json', 'utf-8'));
const rows: string[] = [];

function collect(suite: any, file = '') {
  const currentFile = suite.file ?? file;

  for (const spec of suite.specs ?? []) {
    for (const test of spec.tests ?? []) {
      const first = test.results?.[0];
      const laterPass = test.results?.find((r: any) => r.retry > 0 && r.status === 'passed');
      if (first?.status === 'failed' && laterPass) {
        rows.push(`| ${currentFile} | ${spec.title} | ${test.projectName ?? 'default'} |`);
      }
    }
  }

  for (const child of suite.suites ?? []) collect(child, currentFile);
}

for (const suite of report.suites ?? []) collect(suite);

const markdown = [
  '## Playwright flaky test summary',
  '',
  '| File | Test | Project |',
  '|---|---|---|',
  ...rows,
  '',
  rows.length === 0 ? 'No flaky tests detected.' : `Detected ${rows.length} flaky test(s).`,
].join('\n');

fs.writeFileSync('test-results/flaky-summary.md', markdown);

This style also helps SDETs in interviews. When I interview automation engineers, I listen for ownership language. “I added retries” is weak. “I used retries to capture traces, grouped failures by root cause, and reduced quarantine age” is stronger.

Connect reports to framework design

If your framework already has page objects, fixtures, API helpers, and test data builders, flaky triage becomes faster. If every test creates users through the UI and shares state across files, retries only expose deeper framework design problems. For framework structure, see the earlier ScrollTest guide on Playwright debugging with TypeScript and the trace-focused guide, Playwright Trace Viewer Masterclass.

Common pitfalls I avoid

Retries are easy to configure, so teams underestimate the discipline needed around them. These are the mistakes I watch for during framework reviews.

Pitfall 1: retrying assertions with weak selectors

If a selector is wrong, retrying the test does not make it correct. Prefer role, label, placeholder, and test id locators over brittle CSS chains. If the product has no accessible names, fix the product or add stable test ids for critical flows.

Pitfall 2: using fixed waits after enabling retries

A fixed wait plus a retry creates slow and noisy automation. It also hides which signal the test actually needs. Replace sleeps with web-first assertions, response waits, or app-level events.

// Avoid this pattern
await page.waitForTimeout(5000);
await expect(page.locator('.success')).toBeVisible();

// Prefer a user-visible business signal
await expect(page.getByRole('status')).toHaveText(/saved successfully/i);

Pitfall 3: quarantining without ownership

A quarantine tag without an owner becomes a graveyard. Add an issue ID, owner, root cause, and review date. If your team uses Jira, put the issue key in the annotation. If your team uses GitHub issues, put the issue URL in the description.

Pitfall 4: treating all flakiness as QA debt

Some flaky tests expose real product instability. Slow APIs, race conditions, missing loading states, and poor feature flag cleanup are engineering problems. A strong SDET does not absorb all of that debt into the framework. They bring evidence from traces and ask the right team to fix the right layer.

Key takeaways

Playwright retries should make flaky behavior visible, not invisible. Use them as a short feedback loop for evidence, classification, and ownership.

  • Use zero retries locally and one retry in CI as a sane default.
  • Set trace: 'on-first-retry' so every flaky result gives debugging evidence.
  • Tag quarantined tests with @quarantine, issue IDs, owners, and review dates.
  • Exclude quarantine from the release gate, but run it separately as a debt report.
  • Fail the build when flaky tests cross a visible budget.

If you want to continue this series, start with the previous lesson on CI sharding, then connect it with retries and trace artifacts. That combination gives you fast feedback and honest failure evidence.

FAQ

Should I enable Playwright retries for every project?

No. I usually enable one retry only in CI and keep local retries disabled. Production smoke projects can stay at zero retries because they should represent stricter release confidence.

How many retries are safe?

For UI tests, one retry is enough in most teams. If a test needs two or three retries to pass regularly, treat it as flaky debt and classify the root cause.

Should quarantined tests block the release?

Known quarantined tests should not block the main release gate, but they must run in a separate job. If the quarantine list grows or critical journeys stay quarantined too long, that should block release planning discussions.

What is the best artifact for debugging flaky tests?

Start with Playwright trace files. A trace shows actions, DOM snapshots, network calls, console messages, and timing. Screenshots are useful, but traces explain more of the failure path.

Can retries hide product bugs?

Yes. That is the main risk. If the first run exposes a race condition or intermittent backend issue and the retry passes, the product bug still exists. Keep the failed attempt evidence and classify it before closing the issue.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.