| |

AI Test Failure Classification: 4 Buckets for QA

ByPromode
AI test failure classification graphic showing four buckets: prompt drift, retrieval issue, product bug, and dataset gap.

AI test failure classification is the skill QA teams need before they add one more retry to an LLM test. When an AI test fails, I do not call it “flaky” first. I classify it as prompt drift, retrieval issue, product bug, or eval dataset gap, then I choose the fix.

Table of Contents

Contents

Why “Flaky” Is Lazy Debugging

I hear this sentence too often: “The AI is flaky.” It sounds harmless, but it blocks the real investigation. A failed LLM answer can come from at least 4 different layers: prompt, retrieval, product logic, or the eval dataset itself.

Traditional UI automation gave us a useful habit. When a Playwright test fails, we look at the selector, trace, network call, console log, fixture, and assertion. We do not close the bug with “browser flaky.” AI testing needs the same discipline.

The cost of one vague label

One vague label creates three bad outcomes. First, developers stop trusting the eval suite. Second, QA engineers add random sleeps, loose assertions, or bigger thresholds. Third, release managers cannot tell if a failure is a model risk or a product regression.

The tooling is mature enough to do better. The PromptFoo GitHub repository shows 23,214 stars and describes the project as testing prompts, agents, and RAG systems with CLI and CI/CD integration. The npm downloads API reported 1,585,607 downloads for the promptfoo package between 2026-06-12 and 2026-07-11. DeepEval is also a serious option; the DeepEval GitHub repository shows 16,809 stars, and PyPI lists deepeval 4.1.0 as “The LLM Evaluation Framework.”

What a useful failure label looks like

A useful failure label gives the next owner a clear action. “Prompt drift after system prompt edit” points to a prompt owner. “Retrieval issue: missing refund policy chunk” points to the RAG pipeline. “Product bug: cancellation API returns stale status” points to engineering. “Dataset gap: expected answer no longer matches July policy” points to the eval maintainer.

That is the entire point of AI test failure classification. The label is not a report decoration. It is a routing decision.

The Four-Bucket AI Test Failure Classification Model

My default model has 4 buckets. It is simple enough for a 30-second reel and still strong enough for a release gate.

The 4 labels I use

  • Prompt drift: The prompt changed the behavior even though the product and context are fine.
  • Retrieval issue: The answer is weak because the model received missing, stale, noisy, or wrong context.
  • Product bug: The AI answer exposed a real defect in the application, API, business rule, or workflow.
  • Eval dataset gap: The test data, expected output, rubric, or threshold is outdated or incomplete.

The 5-minute triage order

I use this order because it catches the cheapest evidence first:

  1. Check the exact prompt diff and model settings.
  2. Check retrieved chunks, source URLs, and timestamps.
  3. Replay the product workflow or API call outside the LLM.
  4. Check the eval case, expected answer, rubric, and threshold.
  5. Assign one primary bucket and one optional secondary bucket.

This order matters in CI. If the prompt changed 2 minutes before the failure, do not spend 30 minutes debugging vector search first. If retrieval returns zero documents, do not blame the prompt. If the API returns 500, do not rewrite the eval. Good classification saves time because it forces sequence.

A simple decision table

Signal Likely bucket First fix
Prompt diff exists, same context, changed answer style Prompt drift Review prompt contract and examples
Retrieved chunks are missing or stale Retrieval issue Fix indexing, filters, reranking, or source sync
API/UI workflow is wrong without the LLM Product bug File normal product defect with evidence
Expected answer conflicts with current policy Eval dataset gap Update dataset and add changelog

If your team already has AI test evidence patterns, connect this model with them. ScrollTest has a related guide on AI testing evidence and why green checks are not enough. I like pairing that evidence mindset with failure classification.

Bucket 1: Prompt Drift

Prompt drift happens when a prompt edit changes behavior in a way the team did not intend. It can be a system prompt update, a new few-shot example, a stricter style rule, a tool instruction, or a provider/model setting change.

How prompt drift appears in tests

The answer may still look polished. That is the trap. A model can produce a confident answer that misses a required caveat, changes the ordering of steps, refuses a safe request, or stops using the required JSON schema.

In PromptFoo, I usually start with a declarative config because the official PromptFoo configuration guide is built around providers, prompts, tests, and assertions. That structure makes prompt drift visible in code review.

description: classify support assistant failures
prompts:
  - file://prompts/support-system.txt
providers:
  - openai:gpt-4.1-mini
  - anthropic:messages:claude-3-5-sonnet-latest

tests:
  - vars:
      question: "Can I cancel after payment?"
    assert:
      - type: contains
        value: "cancellation policy"
      - type: llm-rubric
        value: "Answer must mention refund window, cancellation status, and support escalation."

The test above is not perfect, but it has 2 useful checks: a concrete phrase and a rubric. When the prompt changes and both providers shift behavior, I inspect the prompt diff first.

Prompt drift evidence to attach

  • Prompt file diff with line numbers.
  • Model name, temperature, max tokens, tools, and provider version.
  • Old answer versus new answer for the same input.
  • Assertion output from PromptFoo or DeepEval.
  • Decision: revert prompt, update rubric, or accept intended behavior change.

I avoid “fixing” prompt drift by weakening assertions first. If the behavior change is wrong, the prompt needs a tighter contract. If the behavior change is right, the dataset or rubric needs an update with a note.

Bucket 2: Retrieval Issue

A retrieval issue happens when the model is not given the right evidence. In RAG systems, this is the bucket I check before blaming the model. The LLM can only answer from what it sees, and bad context creates good-looking wrong answers.

Common retrieval failure patterns

  • Missing context: The relevant policy, FAQ, or document never appears in the top-k results.
  • Stale context: The retrieved document is old, while the production policy changed.
  • Noisy context: The answer mixes 2 similar products, plans, or countries.
  • Filter bug: Tenant, locale, product, or role filters exclude the right document.
  • Chunking bug: The answer needs 2 adjacent chunks, but retrieval returns only one.

DeepEval is useful here because its docs include metrics for RAG-style checks. The DeepEval metrics documentation lists evaluation metrics as a first-class concept, and the framework supports test cases that compare actual output, expected output, and retrieval context.

TypeScript retrieval evidence example

Even if your eval runner is Python, the product evidence is often easier to collect from a TypeScript API test. I keep a small Playwright request test for the retrieval endpoint.

import { test, expect, request } from '@playwright/test';

test('retrieval returns current cancellation policy', async () => {
  const api = await request.newContext({
    baseURL: process.env.RAG_API_URL,
    extraHTTPHeaders: { Authorization: `Bearer ${process.env.RAG_TOKEN}` }
  });

  const response = await api.post('/retrieve', {
    data: {
      query: 'Can I cancel after payment?',
      tenant: 'india',
      product: 'pro-plan',
      topK: 5
    }
  });

  expect(response.ok()).toBeTruthy();
  const body = await response.json();
  const sources = body.chunks.map((c: { source: string }) => c.source);

  expect(sources).toContain('policies/cancellation-2026.md');
  expect(body.chunks[0].text).toContain('refund window');
});

This does not replace semantic evals. It gives me hard evidence. If this test fails, the AI answer failure is probably retrieval, not prompt drift.

What to fix first

For retrieval issues, I fix the data path before I tune the model. Check ingestion logs, document versions, embeddings, filters, reranker settings, and chunk boundaries. If the RAG endpoint cannot return the right context, a better prompt only hides the bug.

Bucket 3: Product Bug

Sometimes the AI test is correct. The product is wrong. This is the easiest bucket to miss because teams treat LLM tests as “soft” checks, especially when the assertion is semantic.

How an AI eval exposes product defects

Example: a support assistant says “Your subscription is active,” but the billing API actually marks the user as cancelled. That can be a prompt issue, but it can also be a real product bug. Replay the workflow without the model.

import { test, expect, request } from '@playwright/test';

test('billing API returns cancellation status after cancellation', async () => {
  const api = await request.newContext({ baseURL: process.env.APP_API_URL });

  const cancel = await api.post('/subscriptions/sub_123/cancel');
  expect(cancel.status()).toBe(200);

  const status = await api.get('/subscriptions/sub_123');
  const body = await status.json();

  expect(body.state).toBe('cancelled');
  expect(body.cancelledAt).toBeTruthy();
});

If this product test fails, do not create a prompt ticket. Create a product bug with API response, request payload, trace ID, environment, and release build. The LLM only made the defect visible.

Evidence that separates product bug from AI bug

  • Direct API response or UI trace without LLM involvement.
  • Database or event log showing incorrect product state.
  • Release version and feature flag state.
  • One deterministic reproduction path.
  • AI output attached as secondary evidence, not primary proof.

This is where QA teams can build trust. When an AI eval catches a real product defect, document it like any other regression. The team will respect the eval suite faster.

Bucket 4: Eval Dataset Gap

An eval dataset gap means the test is asking the wrong question, expecting an outdated answer, or scoring the response with a weak rubric. This bucket is uncomfortable because QA owns it.

Dataset gaps I see in teams

The most common gap is old business policy. A refund rule changes from 7 days to 14 days, but the eval still expects 7. Another gap is missing negative cases. The assistant passes happy paths but fails edge cases like “cancel after invoice generated,” “same request from admin,” or “India GST invoice needed.”

I also see thresholds copied from demos. A 0.7 semantic score may be fine for content suggestions and too loose for payment support. The threshold has to match user risk.

A dataset entry with owner metadata

{
  "id": "billing-cancel-003",
  "question": "Can I cancel after payment and get a GST invoice?",
  "expectedFacts": [
    "mention cancellation status",
    "mention refund window",
    "mention GST invoice availability for India users"
  ],
  "risk": "medium",
  "owner": "qa-billing",
  "lastReviewed": "2026-07-14",
  "source": "policies/cancellation-2026.md"
}

This metadata looks boring. It saves the release. When the policy changes, the owner and source are visible. When a reviewer asks why the case exists, the risk field explains it.

When to update the eval instead of the product

Update the eval when the current product behavior is correct and the test is stale. Do not hide this as a silent change. Add a changelog entry because historical pass rate will shift. In PromptFoo, commit the YAML change. In DeepEval, commit the test case change. In both cases, link the policy or ticket that justifies the update.

DeepEval and PromptFoo Workflow

I use PromptFoo when I want fast prompt/provider comparison, red-team style cases, and CI-friendly config. I use DeepEval when I want Python-native unit tests, metric objects, and deeper RAG or agent checks. Many QA teams can use both without making it political.

PromptFoo for matrix testing

PromptFoo’s docs position it around prompt, model, and RAG testing with CLI and CI/CD integration. The PromptFoo CI/CD documentation is useful when you want evals to run as part of a pull request. That matters because prompt drift is usually introduced by a small text diff.

npx promptfoo@latest eval
npx promptfoo@latest view
npx promptfoo@latest eval --filter-errors-only

My first PromptFoo suite usually has 20 to 50 cases. I do not start with 500. I want high-signal failures that a QA engineer can inspect in one review session.

DeepEval for Python test suites

DeepEval fits teams that already run pytest. The official DeepEval getting started docs show installation and test execution as normal developer workflow. That is useful for SDETs because the eval can live near integration tests.

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric


def test_cancel_answer_is_relevant():
    test_case = LLMTestCase(
        input="Can I cancel after payment?",
        actual_output="You can cancel from Billing. Refund depends on your plan window.",
        expected_output="Explain cancellation status, refund window, and escalation path."
    )
    metric = AnswerRelevancyMetric(threshold=0.8)
    assert_test(test_case, [metric])

The exact metric depends on the product risk. For a low-risk FAQ, answer relevancy may be enough. For a regulated workflow, I want factual consistency, context relevance, and a human-reviewed golden set.

Classification output format

Whether you use PromptFoo or DeepEval, make the failure output boring and structured:

{
  "caseId": "billing-cancel-003",
  "bucket": "retrieval_issue",
  "secondaryBucket": "eval_dataset_gap",
  "evidence": [
    "Top 5 retrieved chunks do not include policies/cancellation-2026.md",
    "Expected answer mentions GST invoice but dataset source is missing"
  ],
  "owner": "qa-billing",
  "nextAction": "Fix retrieval filter for tenant=india, then review dataset source"
}

This format makes the report usable in Jira, Linear, GitHub Issues, or a Slack release channel.

CI Playbook for QA Teams

AI test failure classification becomes real when CI enforces it. A green pipeline means little if failures are manually waved away. A red pipeline is also noisy if it does not explain the bucket.

My minimum CI gate

  • Run 20 to 50 smoke evals on every prompt or retrieval change.
  • Run the full eval pack nightly or before release freeze.
  • Store raw prompt, retrieved chunks, model settings, and output.
  • Require one failure bucket for every blocked merge.
  • Send product bugs to engineering and dataset gaps to QA owners.

ScrollTest has a related article on AI test evidence in CI/CD release gates. Pair that release-gate mindset with the 4-bucket classification model and the eval suite becomes much easier to defend.

GitHub Actions example

name: ai-evals
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'evals/**'
      - 'rag/**'

jobs:
  promptfoo-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx promptfoo@latest eval --config promptfooconfig.yaml
      - run: node scripts/classify-ai-eval-failures.js

The final script is where you map raw failures into the 4 buckets. Start with rules. Add human review for ambiguous cases. Do not pretend every failure can be classified automatically on day 1.

A practical severity model

I use 3 severity levels:

  • P0: Product bug or unsafe answer in a high-risk workflow. Block release.
  • P1: Retrieval issue or prompt drift affecting a core customer path. Block the prompt/RAG change.
  • P2: Dataset gap, wording issue, or low-risk answer style problem. Fix before weekly eval review.

This severity model is not universal. It gives managers a starting point. A fintech assistant and a course recommendation bot should not share the same threshold.

India Career Context for SDETs

For SDETs in India, this skill is career-relevant. Many teams are moving from “write Selenium scripts” to “own quality signals across UI, API, data, and AI.” The engineer who can classify an AI failure has more value than the engineer who only reruns the job.

Where this fits in interviews

If I interview an SDET for an AI QA role, I ask one practical question: “An LLM regression test failed after a retrieval index update. How do you debug it?” A weak answer says, “I will increase timeout or rerun.” A strong answer asks for prompt diff, retrieved chunks, model settings, product API evidence, and dataset owner.

That answer signals release ownership. In Indian product companies, a strong SDET who can own Playwright, API testing, CI/CD, and LLM evals can target a better role band than someone stuck in click automation. For mid-career QA engineers, the move from tool user to quality engineer is often the difference between routine maintenance work and ₹25-40 LPA product-company roles.

A 7-day learning plan

  1. Day 1: Run one PromptFoo eval with 10 cases.
  2. Day 2: Add one DeepEval pytest test with a relevancy metric.
  3. Day 3: Capture retrieved chunks for every failed RAG answer.
  4. Day 4: Build the 4-bucket failure labels in a JSON report.
  5. Day 5: Add a GitHub Actions job for smoke evals.
  6. Day 6: Review 20 failures and assign owners.
  7. Day 7: Present 3 real defects or risks to your team.

If you need a broader learning path, read AI Quality Engineer Roadmap: PromptFoo + DeepEval and PromptFoo vs DeepEval: QA Guide for LLM Tests. Those posts give the tool comparison; this article gives the failure triage model.

Key Takeaways

AI test failure classification turns a vague failed eval into an actionable quality signal. I do not want a report that says “LLM failed.” I want the report to say what broke, who owns it, and what evidence supports that decision.

  • Stop using “AI is flaky” as the first label. It hides the real owner.
  • Classify every failure as prompt drift, retrieval issue, product bug, or eval dataset gap.
  • Use PromptFoo for fast prompt/provider matrices and CI-friendly eval config.
  • Use DeepEval for Python-native tests, metrics, and deeper RAG checks.
  • Attach evidence: prompt diff, retrieved chunks, API replay, and dataset source.

The practical win is simple: fewer random reruns, cleaner release decisions, and better trust between QA, developers, and product teams. That is what modern test automation should do.

FAQ

What is AI test failure classification?

AI test failure classification is the process of labeling a failed LLM or agent eval by root-cause bucket. The 4 buckets I use are prompt drift, retrieval issue, product bug, and eval dataset gap.

Should I use PromptFoo or DeepEval for this?

Use PromptFoo when you want CLI-based prompt matrices, provider comparison, and CI config. Use DeepEval when you want Python tests, pytest-style workflow, and metric objects. Many QA teams use both.

Can all AI eval failures be auto-classified?

No. Start with rule-based hints and require human review for ambiguous failures. For example, missing retrieved chunks strongly suggest a retrieval issue, but a wrong expected answer may still need QA review.

How many eval cases should a team start with?

Start with 20 to 50 high-risk cases. Cover the workflows that hurt customers or block releases. A small, reviewed eval set is better than 500 noisy cases nobody trusts.

Does this replace normal UI and API automation?

No. AI evals add a quality signal for LLM, RAG, and agent behavior. Keep Playwright, API, contract, and database checks. Use them to prove whether a failed AI answer points to the product or to the AI layer.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.