Playwright Flaky Test Audit Template for 2026

A Playwright flaky test audit is the fastest way to find the tests that pass locally, fail in CI, and waste the team’s trust. Playwright 1.62.1 shipped as a bug-fix release on 30 July 2026, and that is exactly the type of moment where I want teams to inspect traces, selectors, retries, and network evidence instead of blindly bumping a package.

This guide gives you a working audit template, TypeScript examples, and a CI checklist you can copy into a real QA repo. I am not treating flakiness as bad luck. I am treating it as a signal that the test is missing an explicit contract.

Table of Contents

• Why a Playwright flaky test audit matters now
• What Playwright 1.62.1 tells QA teams
• The audit template I use
• Selector and assertion checks
• Network, data, and environment checks
• Trace evidence and retry policy
• CI release gate implementation
• India SDET context
• Key takeaways
• FAQ

Why a Playwright flaky test audit matters now

The dirty secret with modern browser automation is simple: a green Playwright report can still hide a weak test. I see teams celebrate a pass rate without asking whether the test actually observed the business risk. That is how retries become a painkiller instead of a diagnosis.

The adoption numbers make this more important. The npm downloads API reports more than 208 million downloads for @playwright/test in the last-month window ending 6 August 2026, and the Microsoft Playwright GitHub repository shows more than 94,000 stars. When a tool reaches that level of usage, flakiness is no longer a small framework concern. It becomes an engineering-management concern.

A Playwright flaky test audit is not a rewrite project. It is a focused inspection of evidence. You pick the tests that failed or retried, open the trace, inspect what the test actually waited for, and classify the cause. The goal is to make the next run more trustworthy, not to add another dashboard nobody reads.

False confidence costs more than red builds

A red build is annoying, but at least it asks for attention. A flaky green build is worse because it trains the team to ignore risk. If the checkout test passes because the button happened to be visible, but never verifies the order API response, the test is not protecting revenue.

This is why I prefer a small audit spreadsheet over vague labels like “unstable test”. Each row must name the failure mode, owner, evidence link, and fix. If there is no evidence, the first action is not a fix. The first action is better instrumentation.

Where ScrollTest readers should start

If your suite already uses Playwright in CI, start with the 20 tests with the highest retry count. If you are still designing the framework, read this beside our Playwright feature flag testing guide and the Jenkins integration for Playwright tests. Those two patterns expose the same problem from different angles: environment control and release confidence.

• Audit the tests that retried, not the tests people complain about most.
• Keep the evidence link beside the defect or pull request.
• Separate product bugs, test bugs, data bugs, and infrastructure bugs.
• Do not raise retry count until classification is complete.

What Playwright 1.62.1 tells QA teams

The Playwright 1.62.1 release notes list bug fixes around TypeScript project resolution, accessibility snapshots, branded primitive type checking in page.evaluate(), and image-type actionable elements in snapshots. I read that as a reminder that framework upgrades can change what your tests compile, see, and assert.

None of those fixes says “your suite is flaky now”. But a serious SDET reads a release note as a test input. If a suite depends on accessibility snapshots, TypeScript project references, or actionability checks, the upgrade deserves a targeted audit instead of a casual version bump.

Release notes are test data

For many teams, release notes are read only by the person who updates package.json. That is weak ownership. I want release notes to generate test tasks. If the release mentions snapshots, I check snapshot assertions. If it mentions TypeScript resolution, I check monorepo projects. If it mentions actionability, I check locators that depend on images, icons, or nested accessible text.

This habit takes 30 minutes and prevents days of noisy CI. It is also how senior SDETs become visible. They connect framework changes to release risk before the release manager asks why the pipeline is unstable.

Use official docs for behavior, not opinions

For mechanics, I stick to official sources. The Playwright docs explain Trace Viewer, test retries, locators, and network mocking and inspection. Blog posts can add judgment, but the official docs define behavior.

That balance matters. A Playwright flaky test audit needs both: docs for exact semantics and QA judgment for risk classification.

The Playwright flaky test audit template I use

Here is the template I use when a team says, “CI is random.” The columns are intentionally boring. Boring templates get filled. Clever templates get ignored.

1. Test name and file path
2. Failure window: last 7 days, last 30 days, or release branch only
3. Retry count and first failure message
4. Trace, video, screenshot, and console evidence links
5. Locator strategy used at the failing step
6. Wait strategy used at the failing step
7. Network call or API contract involved
8. Test data ownership and cleanup path
9. Classification: product bug, test design bug, data issue, environment issue, or framework upgrade impact
10. Fix owner, due date, and verification run

I do not add severity until the classification is clear. A flaky login test in a non-prod demo environment can be low severity. A flaky payment confirmation assertion on a release branch is different. The point is not to shame the test author. The point is to make risk visible.

The five buckets that stop arguments

Most flakiness debates waste time because people argue from memory. I use five buckets so the team can agree on the next action.

• Product bug: the app behaves inconsistently and the test exposed it.
• Test design bug: the test waits for the wrong signal or asserts the wrong thing.
• Data issue: shared state, stale records, or poor cleanup breaks isolation.
• Environment issue: CI capacity, browser dependencies, grid problems, or network instability affects the run.
• Upgrade impact: a tool, browser, dependency, or TypeScript change changed behavior.

The classification is allowed to change after deeper evidence. What is not allowed is a Jira ticket called “flaky test” with no trace and no hypothesis.

A minimal audit record

type FlakyAuditRecord = {
  testId: string;
  file: string;
  lastFailureUtc: string;
  retryCount30d: number;
  failureMessage: string;
  traceUrl: string;
  locatorAtFailure: string;
  networkDependency: string;
  bucket: 'product-bug' | 'test-design' | 'data' | 'environment' | 'upgrade-impact';
  owner: string;
  fixPr?: string;
};

Keep this record beside the pipeline artifact or export it from your test reporting system. The exact storage does not matter. The discipline matters.

Selector and assertion checks in a Playwright flaky test audit

Selectors are where many Playwright suites slowly rot. The first version of a test often uses a locator that works on the author’s machine. Six sprints later, the UI has a new icon, a hidden tooltip, a split button, or a feature flag. The locator still finds something, but not always the thing the user cares about.

Prefer user-facing locators

Playwright encourages locators that match how users perceive the page. That means roles, labels, placeholder text, and stable test IDs where domain language is not visible. A Playwright flaky test audit should flag CSS chains that know too much about layout.

// Weak: layout-dependent and easy to break
await page.locator('.checkout-panel > div:nth-child(2) button').click();

// Better: user-facing intent
await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();

The better version still needs business evidence. A heading can lie. For payment, inventory, or permissions flows, pair UI assertions with a network assertion or an API check.

Assert the contract, not the animation

Many flaky tests wait for the wrong success signal. They wait for a spinner to disappear, then assert a button is visible. That only proves the UI reached a state. It does not prove the system accepted the action.

const responsePromise = page.waitForResponse(resp =>
  resp.url().includes('/api/orders') && resp.request().method() === 'POST'
);

await page.getByRole('button', { name: 'Place order' }).click();
const response = await responsePromise;
expect(response.status()).toBe(201);

await expect(page.getByText(/order confirmed/i)).toBeVisible();

This is one of the easiest upgrades in an audit. If the test changes money, permissions, inventory, or customer-visible state, a UI-only assertion is usually not enough.

Network, data, and environment checks

A Playwright flaky test audit must look beyond the browser. The browser is only where the symptom appears. The cause may sit in a shared test account, a slow API, a queue worker, a feature flag, or a CI machine starved for CPU.

Capture network evidence

The Playwright network docs show how to inspect, wait for, and mock requests. I use that capability sparingly. Too much mocking creates fantasy tests. Too little network evidence creates detective work after every failure.

• For critical create/update flows, capture the status code and response shape.
• For third-party dependencies, decide whether the contract is mocked, stubbed, or tested live.
• For feature flags, record which flag state the test expects.
• For timeout failures, compare app response time against the test timeout before changing code.

page.on('response', async response => {
  if (response.url().includes('/api/checkout')) {
    console.log('checkout response', response.status(), response.url());
  }
});

Test data is a flakiness source

A surprising number of “Playwright issues” are data issues. Shared users, reused emails, stale carts, timezone assumptions, and leftover orders can all create random behavior. The audit should ask who owns setup and cleanup. If the answer is “the previous test”, you have already found a smell.

I prefer API setup for data, UI execution for the user journey, and API cleanup where safe. That gives the test a known starting state without turning every scenario into a 40-step UI marathon.

test.beforeEach(async ({ request }) => {
  await request.post('/test-data/users', {
    data: { role: 'buyer', state: 'clean-cart' }
  });
});

test.afterEach(async ({ request }) => {
  await request.delete('/test-data/users/buyer-clean-cart');
});

If your organization cannot expose test-data APIs, document that as a platform gap. Manual cleanup by QA engineers is not a sustainable strategy for a serious CI gate.

Trace evidence and retry policy

Trace Viewer is the strongest argument for Playwright in flaky-test work. The official Trace Viewer docs show how to inspect actions, snapshots, console logs, network calls, and timing. In an audit, I want every flaky failure to have a trace link before anybody guesses.

Turn on trace on retry

The most practical configuration is trace on first retry. It keeps routine runs lighter and captures rich evidence when a test starts behaving badly.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure'
  },
  reporter: [['html'], ['json', { outputFile: 'test-results/results.json' }]]
});

Notice the retry count. I do not start with three retries. One retry is enough to capture evidence. More retries can hide pain and waste CI minutes unless the team has a clear policy.

Retry misuse is a process smell

The Playwright retries docs explain how tests are categorized as passed, flaky, or failed. That label is useful, but it is not the end of the conversation. A flaky label should open an audit record. It should not close the incident.

My rule is simple: if the same test is flaky twice in 30 days, it needs an owner. If a test is flaky across two release branches, it needs either a fix or removal from the gate until it is trustworthy. Keeping a known liar in the release gate teaches everyone to ignore the gate.

CI release gate implementation

The audit becomes powerful when it runs inside CI. I do not mean blocking every build on every historical flaky test. I mean failing the release gate when new flakiness appears in critical paths or when known flaky tests exceed the agreed budget.

Generate a flake report from JSON

Playwright can produce JSON reports. You can parse them and fail the pipeline when retry behavior crosses the threshold. The exact schema can vary by reporter version, so treat this script as a starting point and adapt it to your report shape.

// scripts/check-flakes.ts
import fs from 'node:fs';

type TestResult = { status?: string; retry?: number; error?: { message?: string } };
type TestCase = { title?: string; outcome?: string; results?: TestResult[] };

const report = JSON.parse(fs.readFileSync('test-results/results.json', 'utf8'));
const flaky: string[] = [];

function walk(node: any, path: string[] = []) {
  for (const suite of node.suites ?? []) walk(suite, [...path, suite.title].filter(Boolean));
  for (const spec of node.specs ?? []) {
    for (const test of spec.tests ?? []) {
      const retries = (test.results ?? []).filter((r: TestResult) => (r.retry ?? 0) > 0).length;
      if (test.outcome === 'flaky' || retries > 0) {
        flaky.push([...path, spec.title].join(' › '));
      }
    }
  }
}

walk(report);

if (flaky.length > 0) {
  console.error('Flaky tests need audit records:');
  for (const name of flaky) console.error(`- ${name}`);
  process.exit(1);
}

console.log('No flaky tests detected in this run.');

Use a written release-gate policy

A CI script without a policy turns into politics. Write the policy down. For example:

• Critical path tests cannot be flaky on release branches.
• Known flaky tests must have an audit record and owner.
• A test with two flaky runs in 30 days leaves the release gate until fixed.
• Framework upgrades require targeted trace review on impacted suites.
• Retries are evidence capture, not a quality metric.

This policy is strict enough to protect releases and practical enough for Indian service teams and product companies. In a TCS or Infosys-style enterprise program, the audit record helps coordinate many teams. In a Bengaluru product company, the same record protects a fast-moving release train.

If your pipeline needs stronger quality gates for AI-assisted tests too, pair this with our PromptFoo DeepEval CI gate template. Browser flakiness and LLM flakiness are different problems, but the governance habit is similar: define evidence, define threshold, define owner.

India SDET context: why this skill pays

In India, Playwright is now a serious SDET skill, not a side curiosity. The engineers who stand out are not the ones who can record a test with codegen. They are the ones who can explain why a release gate can be trusted.

For a mid-level automation engineer aiming for product-company roles, this is practical career proof. A resume bullet that says “built Playwright framework” is common. A bullet that says “reduced release-branch flaky tests by classifying retries, trace evidence, and test-data ownership” is stronger because it shows engineering judgment.

Talk in manager language

Managers care about release confidence, cycle time, and escalation noise. When you present the audit, avoid framework worship. Say what changed in the release process.

• Before: 38 retrying tests across smoke and regression.
• After: 11 product bugs, 14 test-design fixes, 8 data fixes, 5 environment fixes.
• Decision: remove 6 known-liar tests from the release gate until fixed.
• Result: fewer false escalations and clearer ownership.

Do not invent numbers if you do not have them. Start tracking today, and within four weeks you will have a story worth telling in interviews.

Interview proof

If I interview an SDET for a ₹25-40 LPA role, I listen for this level of thinking. Can the candidate connect retries to evidence? Can they separate product bugs from test bugs? Can they write a small CI script instead of only explaining theory? This is the difference between test automation execution and test engineering ownership.

Key takeaways for your Playwright flaky test audit

A Playwright flaky test audit should end with fewer guesses and better release decisions. Do not wait for the suite to become embarrassing. Start with the next retrying test and create the first record.

• Use Playwright 1.62.1 release notes as an audit trigger, especially for TypeScript, accessibility snapshot, and actionability-related suites.
• Treat retries as evidence capture. One retry with trace is usually more useful than three silent retries.
• Prefer user-facing locators and pair critical UI assertions with API or network evidence.
• Classify every flaky test into product bug, test design, data, environment, or upgrade impact.
• Put the audit in CI so the release gate protects users instead of reporting random noise.

The focus keyword is not the point. The habit is the point. Once the team sees flakiness as evidence, a Playwright flaky test audit becomes part of release engineering, not a cleanup task for whoever is free on Friday evening.

FAQ

How often should I run a Playwright flaky test audit?

Run a lightweight audit every week and a targeted audit after framework upgrades, browser upgrades, or release-branch instability. If the same test retries twice in 30 days, assign an owner.

Should I disable retries completely?

Not in most CI setups. I prefer one retry with trace on first retry. That captures evidence without hiding chronic problems. Zero retries can be useful locally or in a special diagnostic job.

Are data-test IDs better than role locators?

Role locators are better when they express real user intent. Stable test IDs are useful for controls with unclear accessible names or highly dynamic UI. The audit should flag layout-dependent CSS chains first.

What is the biggest mistake teams make with flaky tests?

They increase retries before classification. That makes the report greener and the release gate weaker. Classify first, fix second, adjust retry policy last.

Can this template work with Selenium or Cypress?

Yes. The evidence tools differ, but the buckets stay the same: product bug, test design, data, environment, and upgrade impact. Playwright makes the workflow easier because trace evidence is built into the ecosystem.