| |

AI Testing Evidence: Stop Trusting Green Checks

ByPromode
AI testing evidence featured image for Day 30 showing green check versus proof

Day 30 of 100 Days of AI in QA & SDET. AI testing evidence is the line I draw between a useful AI browser run and a green checkmark that should not be trusted. If an agent says a test passed, I want proof: the steps it took, the page state it saw, the assertion it made, the screenshot it captured, and the uncertainty it could not remove.

Most QA teams do not have an AI testing problem yet. They have an evidence problem. They are adding browser agents, prompt tests, and LLM evaluations faster than they are adding review rules, trace retention, and failure triage habits.

Table of Contents

Contents

Why Green Checks Are Dangerous in AI Testing

A normal automated test is predictable when it is written well. It opens a known route, performs known actions, checks known expectations, and fails when the product breaks that contract. An AI browser agent is different. It plans, observes, retries, summarizes, and sometimes takes a path the test author did not expect.

That flexibility is useful, but it changes the review model. A green result from an AI run does not prove the correct thing happened. It only proves the runner decided the task was complete. Without evidence, the reviewer cannot tell if the agent found the right screen, skipped a step, accepted a weak match, or hallucinated the final answer.

Agents can complete the wrong task

I see this with login flows, checkout flows, settings pages, and admin panels. The agent reaches a page with similar text and reports success. A human looking at the trace spots the issue in ten seconds, but the CI dashboard only shows a pass. That is how false confidence enters the release process.

Browserbase Stagehand’s recent browse@0.9.0 release notes are a good signal for where the ecosystem is moving. The release changed the screenshot command so bare invocations write a file by default, while base64 output moved behind a flag. That sounds small, but it matches the real need: save evidence as a file, keep it reviewable, and avoid throwing proof away in console output.

Good QA work is evidence review

Manual testers already understand this. When a tester raises a bug, they attach steps, screenshots, environment details, expected results, and actual results. AI testing needs the same discipline. A generated test case without proof is just a story. An agent run with trace, screenshot, console logs, network calls, and reviewer notes becomes something the team can trust.

This is why I like treating AI runs as junior testers. They can explore fast. They can run boring paths repeatedly. They can document what they saw. But a senior SDET still owns the release signal.

What AI Testing Evidence Actually Means

AI testing evidence is not a folder full of random screenshots. It is a structured proof package that explains what the system did and why the result should be trusted. The package should be readable by a QA engineer, a developer, and a release manager without asking the agent to explain itself again.

Evidence has three jobs

First, it proves coverage. The reviewer can see which route, user role, browser, environment, data set, and acceptance criteria were exercised. Second, it proves behavior. The reviewer can inspect the UI state, API responses, logs, and assertions. Third, it supports triage. When a failure happens, the team can reproduce the problem or reject the run as an agent mistake.

Playwright already gives teams strong raw material. The official Playwright Trace Viewer documentation describes trace viewer as a GUI for exploring recorded traces after a script runs, especially for debugging CI failures. The screenshots documentation covers page and element screenshots. Those two capabilities are enough to upgrade many AI experiments from demo mode to review mode.

Evidence should survive the pipeline

If the proof disappears when the CI job finishes, it is not release evidence. Store artifacts for the right duration, attach them to the run, and link them from the test report. For high-risk flows such as payment, authentication, permissions, and data deletion, I prefer keeping the proof longer than the default artifact retention period.

I wrote a practical companion on AI Testing Evidence Pack: Trace, Screenshot, Logs. This article builds on that idea and turns it into a release gate pattern for AI-driven QA.

The Five-Part Evidence Pack I Want From Every Agent Run

Here is the minimum evidence pack I expect from a browser agent before I let its result influence a release decision. You can add more, but do not remove these five parts unless the test is low value.

  1. Intent: the task, acceptance criteria, risk area, and user role.
  2. Trace: every meaningful action, page transition, locator, and wait.
  3. Screenshot: final state and critical intermediate checkpoints.
  4. Assertions: deterministic checks that confirm business behavior.
  5. Review note: a short human-readable verdict with uncertainty called out.

Intent stops vague automation

A weak task says, “test checkout.” A useful task says, “as a logged-in retail customer, add one in-stock item to cart, apply the first eligible coupon, complete card payment in sandbox, and verify order confirmation contains order ID and total amount.” The second task gives the agent boundaries and gives the reviewer a clear standard.

Intent is also where India-based QA teams can align with product managers and developers. In service companies, QA often receives loose tickets late in the sprint. In product companies, SDETs are expected to convert risk into executable checks. AI does not remove that responsibility. It makes the gap more visible.

Trace and screenshots catch agent shortcuts

A trace shows how the browser reached the final state. A screenshot shows what the final state looked like. I want both because either one alone can lie by omission. A screenshot may show a success banner, but the trace may reveal that the agent skipped coupon validation. A trace may show a click, but the screenshot may show an error toast hiding under a modal.

Assertions keep the agent honest

Assertions are not optional. AI can choose actions, read pages, and summarize results, but business checks should stay deterministic wherever possible. Check URLs, text, visible components, API status codes, database side effects when appropriate, and event logs for critical flows.

How to Build AI Testing Evidence With Playwright

Playwright is a practical base for this pattern because it already supports traces, screenshots, videos, reporters, and CI artifacts. The AI layer can plan or generate steps, but Playwright should record and verify the run.

Start with an evidence contract

Do not let every engineer invent a different artifact format. Create one contract and make every agent run follow it. Here is a simple TypeScript shape I use as a starting point:

type EvidenceVerdict = 'pass' | 'fail' | 'needs_review';

type AiTestEvidence = {
  runId: string;
  day: number;
  feature: string;
  riskArea: 'auth' | 'checkout' | 'settings' | 'search' | 'permissions';
  intent: string;
  environment: string;
  model?: string;
  browser: string;
  startedAt: string;
  finishedAt: string;
  verdict: EvidenceVerdict;
  assertions: Array<{ name: string; passed: boolean; details?: string }>;
  artifacts: {
    traceZip?: string;
    finalScreenshot?: string;
    consoleLog?: string;
    networkLog?: string;
  };
  reviewerNote: string;
  uncertainty: string[];
};

The point is not the exact field name. The point is consistency. When every run writes the same JSON, your CI report, Slack notification, release dashboard, and human review all read the same truth.

Record trace, screenshot, and console logs

Here is a stripped-down Playwright example. It does not depend on a specific AI framework. Replace the comments with your agent call, then keep the evidence capture deterministic.

import { test, expect } from '@playwright/test';
import fs from 'node:fs/promises';

const runId = `checkout-${Date.now()}`;

test('AI-assisted checkout evidence pack', async ({ page, context }) => {
  const consoleLines: string[] = [];
  page.on('console', msg => consoleLines.push(`${msg.type()}: ${msg.text()}`));

  await context.tracing.start({
    screenshots: true,
    snapshots: true,
    sources: true
  });

  await page.goto(process.env.BASE_URL ?? 'https://example.com');

  // AI agent can choose the next actions here.
  // Keep final verification deterministic.
  await page.getByRole('link', { name: /cart/i }).click();
  await expect(page.getByRole('heading', { name: /cart/i })).toBeVisible();

  const screenshotPath = `artifacts/${runId}-final.png`;
  const tracePath = `artifacts/${runId}-trace.zip`;
  const logPath = `artifacts/${runId}-console.log`;

  await page.screenshot({ path: screenshotPath, fullPage: true });
  await context.tracing.stop({ path: tracePath });
  await fs.writeFile(logPath, consoleLines.join('
'));

  await fs.writeFile(`artifacts/${runId}-evidence.json`, JSON.stringify({
    runId,
    feature: 'checkout',
    intent: 'Verify cart page is reachable after AI-assisted navigation',
    verdict: 'pass',
    assertions: [{ name: 'cart heading visible', passed: true }],
    artifacts: { traceZip: tracePath, finalScreenshot: screenshotPath, consoleLog: logPath },
    reviewerNote: 'Agent reached cart and deterministic assertion passed.',
    uncertainty: ['Payment path not executed in this sample']
  }, null, 2));
});

This is boring code. That is the compliment. Evidence code should be boring, repeatable, and easy to audit.

Use the same report vocabulary every time

I like three verdicts: pass, fail, and needs_review. Pass means deterministic checks succeeded and evidence matches the intent. Fail means product behavior broke an expected rule. Needs_review means the agent completed something, but proof is incomplete or ambiguous. That third state prevents the worst habit in AI testing: forcing uncertain results into a binary dashboard.

If you are new to this space, read AI Browser Agent Testing: 3 Checks Before Trust and AI Agent Testing: Why One Pass Means Nothing. Both posts explain why a single successful run should not become a release decision.

Where PromptFoo and DeepEval Fit

Browser evidence proves what happened in the UI. LLM evaluation proves whether the agent’s reasoning, prompt output, or generated test content meets quality rules. You need both when AI is making decisions inside the QA workflow.

PromptFoo for regression-style prompt checks

The npm registry currently lists promptfoo as an LLM eval and testing toolkit. I like it for prompt and provider regression because QA teams already understand test cases, expected outputs, and diffs. You can compare prompts across models, check for unsafe answers, and detect output drift before a bad prompt reaches your automation pipeline.

prompts:
  - file://prompts/bug-summary.txt
providers:
  - openai:gpt-4.1-mini
  - anthropic:claude-3-5-sonnet-latest
tests:
  - vars:
      defect: "Payment succeeds but order total excludes tax"
    assert:
      - type: contains
        value: "tax"
      - type: llm-rubric
        value: "Mentions user impact, reproduction clue, and severity rationale"

DeepEval for Python-first quality metrics

PyPI describes DeepEval as an LLM evaluation framework. I prefer it when the QA stack is already Python-heavy or when the team wants metrics such as answer relevancy, faithfulness, and custom criteria inside pytest-style workflows.

The important rule is simple: do not mix browser evidence and LLM evals into one vague “AI score.” Keep them separate. A browser agent can navigate correctly and still produce a weak summary. A prompt can pass an eval and still fail to interact with the product. Separate signals make triage cleaner.

For a QA-specific walkthrough, see DeepEval for QA Engineers: LLM Evals That Work.

CI Release Gates: Turning Evidence Into Decisions

AI testing evidence becomes useful when it changes release behavior. If the evidence pack is generated but nobody reads it, it is decoration. A proper CI gate checks whether required proof exists and whether critical assertions passed.

Gate by risk, not by excitement

Do not put every AI experiment into the critical release path. Start with three risk bands:

  • Low risk: exploratory runs, content checks, documentation checks.
  • Medium risk: common user journeys with deterministic assertions.
  • High risk: payment, permissions, data deletion, compliance, and admin flows.

Low-risk runs can inform the team without blocking the build. Medium-risk runs can create tickets or warnings. High-risk runs should block only when the deterministic assertions and evidence contract are mature enough.

A simple evidence gate script

This Python example checks whether required artifacts exist and whether the verdict is acceptable. It is intentionally small enough for a team to adapt in one sprint.

import json
from pathlib import Path
import sys

required = ['traceZip', 'finalScreenshot', 'consoleLog']
failed = []

for evidence_file in Path('artifacts').glob('*-evidence.json'):
    data = json.loads(evidence_file.read_text())
    if data.get('verdict') not in {'pass', 'needs_review'}:
        failed.append(f"{evidence_file}: product failure")
    for name in required:
        artifact = data.get('artifacts', {}).get(name)
        if not artifact or not Path(artifact).exists():
            failed.append(f"{evidence_file}: missing {name}")
    if data.get('verdict') == 'needs_review':
        failed.append(f"{evidence_file}: human review required")

if failed:
    print('
'.join(failed))
    sys.exit(1)

print('AI evidence gate passed')

This script does not ask whether the agent sounded confident. It asks whether the proof exists. That is the mindset shift.

Keep the human checkpoint explicit

For the first 30 to 60 days of any AI testing rollout, I want a human checkpoint on risky flows. Not a meeting. Not a committee. A clear reviewer field in the report is enough. The reviewer checks the trace, final screenshot, assertion list, and uncertainty note. Then they approve, reject, or send the run back for better evidence.

India SDET Context: This Is a Career Skill

For Indian QA engineers, this topic matters because the SDET role is moving from script writer to quality signal owner. TCS, Infosys, Wipro, Cognizant, and similar service environments still need people who can execute test cases. Product companies and global teams increasingly pay for people who can design reliable automation systems and explain release risk.

The salary signal is ownership

I do not see ₹25-40 LPA SDET roles going to people who only say, “I used an AI tool.” The stronger signal is, “I built an evidence workflow for AI-assisted testing, added Playwright traces and screenshots, created CI gates, and reduced noisy passes.” That sentence tells a hiring manager you understand testing, automation, and AI risk.

What to practice this week

Pick one flow from your current project or a demo app. Add trace capture, final screenshot, console logs, and an evidence JSON. Then write a one-page review note. Do not start with ten tools. Start with one flow and make the proof clean.

  • One login flow with positive and negative paths.
  • One search flow with filters and empty state.
  • One checkout or form submission flow in a sandbox.
  • One prompt eval that checks a bug summary or test case generator.

Once you can explain the evidence clearly, adding more agents becomes safer. Without that foundation, more agents only create more noise.

Common Mistakes I See Teams Make

The biggest mistake is treating AI output as proof. A summary is not proof. A confident sentence is not proof. A pass badge is not proof. Proof is inspectable and tied to the product state.

Mistake 1: No deterministic assertions

Teams let the agent decide whether the task passed. That is risky for anything business-critical. Keep AI for planning and interaction where it helps. Keep assertions deterministic wherever the product has clear rules.

Mistake 2: Screenshots without context

A final screenshot is helpful, but it needs run ID, environment, user role, browser, timestamp, and intent. Otherwise it becomes a random image in a build folder. Name artifacts predictably and link them from the report.

Mistake 3: Hiding uncertainty

Good QA reports uncertainty. Bad AI reports hide it. If the agent did not verify tax calculation, say that. If the run used test data with known limitations, say that. If the trace shows a retry that might indicate flakiness, say that.

Mistake 4: No retention policy

Teams generate traces, then the CI system deletes them after a short time. For routine checks, that may be fine. For high-risk production releases, incidents, or customer-impacting defects, evidence retention should match the risk.

Key Takeaways

AI testing evidence is not extra paperwork. It is how QA teams prevent AI-assisted automation from becoming false confidence.

  • A green check from an AI agent is not enough for release trust.
  • Every meaningful agent run should produce intent, trace, screenshot, assertions, and review notes.
  • Playwright traces and screenshots give SDETs a strong base for inspectable proof.
  • PromptFoo and DeepEval help test LLM output, but they do not replace browser evidence.
  • The best SDETs in 2026 will own the quality signal, not just the test script.

My practical recommendation: choose one high-value flow this week and add an evidence pack. Do not wait for a perfect AI testing platform. Build the habit now. The tooling will keep changing, but proof will remain the currency of QA.

FAQ

What is AI testing evidence?

AI testing evidence is the set of artifacts that proves what an AI-assisted test did. It usually includes the task intent, browser trace, screenshots, logs, deterministic assertions, and a review note that calls out uncertainty.

Is a screenshot enough evidence for an AI browser test?

No. A screenshot shows one moment. It does not show the path taken, skipped steps, console errors, network failures, or weak assertions. Use screenshots with traces, logs, and deterministic checks.

Should AI test runs block CI releases?

Only when the evidence contract is mature. Start with informational runs, then warnings, then blocking gates for high-risk flows after the team trusts the artifacts and triage process.

Where do Playwright traces fit in AI testing?

Playwright traces are one of the most useful artifacts because they show actions, snapshots, screenshots, and timing in a reviewable format. They help humans audit what the agent actually did.

How can a manual tester start learning this?

Start with one flow. Run it with Playwright, capture a screenshot and trace, write a small evidence JSON, and explain the result in plain English. That habit builds the bridge from manual testing to AI-aware SDET work.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.