A/B Test Variant Verification with Playwright: Pin, Stub & Verify

You ship a new “Buy Now” button color behind a feature flag, the experimentation platform splits traffic 50/50, and now your end-to-end suite is flaky because half the runs see the old UI and half see the new one. A/B tests are a nightmare for deterministic automation precisely because they are designed to be non-deterministic. In this guide on Playwright A/B test variant verification, you will learn how to pin a specific variant, verify the correct experience renders, assert exposure analytics fire, and keep CI green no matter which bucket the server assigns.

🎭 Want to master this with real projects? Join the Playwright Automation Mastery course at The Testing Academy.

Why A/B Tests Break Test Automation

An A/B test (or multivariate test) randomly assigns each visitor to a variant — usually called control and treatment. The assignment is typically driven by one of three mechanisms: a cookie set by the experimentation SDK (Optimizely, LaunchDarkly, GrowthBook, VWO), a server-rendered decision baked into the HTML response, or a client-side script that mutates the DOM after load. Each mechanism leaks randomness into your tests in a different place, and your verification strategy has to match the mechanism.

The core problem is bucketing. If your test does not control which bucket it lands in, then expect(page.getByRole('button', { name: 'Buy Now' })) passes 50% of the time and fails the rest. The fix is to force a deterministic variant, then write one focused test per variant. Playwright gives you several real, first-class hooks to do exactly that: cookies and localStorage seeding, request interception with page.route, and HAR replay with page.routeFromHAR.

Bucketing mechanism	Where randomness lives	Playwright control point
Cookie-based SDK	Browser cookie / localStorage	context.addCookies / addInitScript
Server-rendered flag	HTTP response HTML or JSON	page.route fulfill / modify
Client-side decision API	XHR to /experiments endpoint	page.route to stub the API
Edge / CDN assignment	Request header or query param	extraHTTPHeaders / setExtraHTTPHeaders

Strategy 1: Force a Variant with Cookies and localStorage

The most common assignment vector is a cookie like ab_variant=treatment or a key in localStorage that the experimentation SDK reads on boot. Set it before the page loads and the SDK will honor it. Use context.addCookies() for cookie-driven tests and page.addInitScript() to seed localStorage before any application JavaScript runs.

import { test, expect } from '@playwright/test';

// Run the SAME assertions across both buckets by parametrizing.
const variants = ['control', 'treatment'] as const;

for (const variant of variants) {
  test(`checkout CTA renders correctly for ${variant}`, async ({ context, page }) => {
    // 1. Seed the cookie the SDK reads to assign a bucket.
    await context.addCookies([
      { name: 'ab_variant', value: variant, domain: 'localhost', path: '/' },
    ]);

    // 2. Some SDKs cache the decision in localStorage instead of a cookie.
    await page.addInitScript((v) => {
      window.localStorage.setItem('exp.checkout_cta', v);
    }, variant);

    await page.goto('http://localhost:3000/cart');

    const cta = page.getByRole('button', { name: /buy now|complete purchase/i });
    await expect(cta).toBeVisible();

    // 3. Assert the variant-specific copy.
    if (variant === 'treatment') {
      await expect(cta).toHaveText('Complete Purchase');
    } else {
      await expect(cta).toHaveText('Buy Now');
    }
  });
}

The key insight is that addInitScript runs before the page’s own scripts on every navigation, so the experimentation SDK sees your forced value the instant it initializes. This avoids a race where the SDK assigns a random bucket before your test can override it.

Strategy 2: Stub the Experiment API with page.route

Modern SDKs (LaunchDarkly, GrowthBook, Optimizely Full Stack) fetch flag decisions from a network endpoint at runtime. Instead of fighting the SDK’s internal hashing, intercept that request with page.route() and return a deterministic payload. This is the most robust approach because it works regardless of how the SDK computes buckets internally.

import { test, expect } from '@playwright/test';

type FlagPayload = { features: Record<string, { value: string }> };

function flagResponse(ctaVariant: string): FlagPayload {
  return {
    features: {
      checkout_cta: { value: ctaVariant },
      hero_layout: { value: 'control' },
    },
  };
}

test('treatment variant via stubbed experiment endpoint', async ({ page }) => {
  // Intercept the SDK's decision call and force "treatment".
  await page.route('**/api/experiments/**', async (route) => {
    await route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify(flagResponse('treatment')),
    });
  });

  await page.goto('/cart');

  await expect(page.getByTestId('cta-button')).toHaveText('Complete Purchase');
  // The hero stays on control because we pinned it that way.
  await expect(page.getByTestId('hero')).toHaveClass(/hero--default/);
});

If you only want to flip one field while letting the real backend answer the rest, fetch the original response first and patch it. This keeps the remaining experiment config realistic instead of hand-mocking every flag.

test('flip a single flag, keep the rest real', async ({ page }) => {
  await page.route('**/api/experiments/**', async (route) => {
    const response = await route.fetch();        // hit the real server
    const json = await response.json();
    json.features.checkout_cta = { value: 'treatment' }; // override one flag
    await route.fulfill({ response, json });     // re-serve patched body
  });

  await page.goto('/cart');
  await expect(page.getByTestId('cta-button')).toHaveText('Complete Purchase');
});

Strategy 3: Lock Down Variants with HAR Replay

When an experiment depends on many coordinated network calls, hand-writing stubs gets brittle. Record the network traffic for a specific variant once with routeFromHAR in record mode, then replay it deterministically. The recorded HAR file freezes that exact variant, so every subsequent run sees identical responses.

import { test, expect } from '@playwright/test';

// First pass: set update: true to RECORD. Then commit treatment.har and
// switch update off so CI replays the frozen treatment variant.
test('replay the recorded treatment variant from HAR', async ({ page }) => {
  await page.routeFromHAR('./hars/treatment.har', {
    url: '**/api/experiments/**',
    update: false,          // replay only; true would re-record
    notFound: 'fallback',   // let non-matching requests hit the network
  });

  await page.goto('/cart');
  await expect(page.getByTestId('cta-button')).toHaveText('Complete Purchase');
});

Record once per variant by toggling update: true, then commit one HAR per bucket (control.har, treatment.har). Because HAR replay is hermetic, these tests run identically on a developer laptop and in CI, even with the experimentation backend offline.

Verifying Analytics and Exposure Events

Rendering the right variant is only half the job. A working A/B test must also fire an exposure (or “activation”) event so the analytics pipeline knows which users saw which variant. If exposure tracking is broken, the experiment produces no usable data even though the UI looks correct. Intercept the beacon request and assert on its payload.

import { test, expect } from '@playwright/test';

test('treatment fires the correct exposure beacon', async ({ context, page }) => {
  await context.addCookies([
    { name: 'ab_variant', value: 'treatment', domain: 'localhost', path: '/' },
  ]);

  // Capture the analytics beacon before it leaves the browser.
  const exposurePromise = page.waitForRequest((req) =>
    req.url().includes('/collect') && req.method() === 'POST'
  );

  await page.goto('http://localhost:3000/cart');
  await expect(page.getByTestId('cta-button')).toBeVisible();

  const exposure = await exposurePromise;
  const payload = JSON.parse(exposure.postData() ?? '{}');

  // Soft assertions collect every mismatch in one run instead of failing fast.
  expect.soft(payload.event).toBe('experiment_viewed');
  expect.soft(payload.experimentId).toBe('checkout_cta');
  expect.soft(payload.variant).toBe('treatment');
});

Using expect.soft here is deliberate: if the beacon fires but with the wrong variant field, you want to see all the payload problems in a single test report rather than fixing them one assertion at a time. For a fully offline check, you can also stub the /collect endpoint with page.route and push every captured payload into an array, then assert the array length and contents after the interaction completes.

Putting It Together: A Variant Fixture

Repeating cookie and route setup in every spec gets noisy. Wrap variant pinning in a custom Playwright fixture so each test simply declares the bucket it wants. This is the pattern most mature teams converge on for Playwright A/B test variant verification at scale.

import { test as base, expect } from '@playwright/test';

type VariantFixtures = {
  pinVariant: (flags: Record<string, string>) => Promise<void>;
};

export const test = base.extend<VariantFixtures>({
  pinVariant: async ({ page }, use) => {
    const pin = async (flags: Record<string, string>) => {
      await page.route('**/api/experiments/**', async (route) => {
        const features = Object.fromEntries(
          Object.entries(flags).map(([k, v]) => [k, { value: v }])
        );
        await route.fulfill({
          status: 200,
          contentType: 'application/json',
          body: JSON.stringify({ features }),
        });
      });
    };
    await use(pin);
  },
});

// Usage stays readable and intent-revealing:
test('hero treatment renders the new layout', async ({ page, pinVariant }) => {
  await pinVariant({ hero_layout: 'treatment' });
  await page.goto('/');
  await expect(page.getByTestId('hero')).toHaveClass(/hero--experimental/);
});

Best Practices for Stable Variant Tests

• One test per variant. Never let a single test pass or fail based on random bucketing — pin the bucket and assert the exact expected experience.
• Stub the decision, not the DOM. Forcing the SDK’s input (cookie or API response) is more durable than asserting against DOM mutations that change every release.
• Always verify exposure events. A variant that renders but does not log exposure silently corrupts your experiment data.
• Keep HARs versioned. Commit one HAR per variant and re-record on a schedule so replays do not drift from production.
• Tag variant tests. Use a tag like @experiment so you can run or skip the experiment suite independently when flags are cleaned up.

With these four strategies — cookie seeding, API stubbing, HAR replay, and beacon verification — your Playwright A/B test variant verification suite becomes fully deterministic. You decide the bucket, you assert the exact experience, and you confirm the analytics that make the experiment measurable. No more 50% flakiness, no more “it failed on retry,” just clean signal in every CI run.

FAQ

How do I force a specific A/B test variant in Playwright?

Set the bucketing input before the page loads. If the SDK reads a cookie, use context.addCookies(); if it reads localStorage, use page.addInitScript(); if it fetches a decision API, intercept that call with page.route() and return a fixed variant payload. Then write one test per variant so each run is deterministic.

Should I test both control and treatment variants?

Yes. Both variants ship to real users, so both need coverage. The cleanest pattern is to parametrize with a for loop over ['control', 'treatment'], pin each bucket, and assert the variant-specific behavior. This catches regressions in whichever experience the experiment eventually rolls out fully.

How do I verify that an A/B exposure event fired correctly?

Use page.waitForRequest() to capture the analytics beacon, then parse request.postData() and assert the experiment ID and variant fields with expect.soft so you see every payload mismatch at once. For offline runs, stub the analytics endpoint with page.route() and collect captured payloads into an array to assert after the interaction.