|

DeepEval Flaky Tests: QA Guide for LLM Evals

ByPromode
DeepEval flaky tests QA release gate featured image

DeepEval flaky tests became a practical QA topic after DeepEval 4.1.5 shipped a flaky flag for metrics and test cases. I like the feature, but I do not treat it as permission to ignore red evals. This guide shows how I would use it inside an SDET-owned LLM evaluation pipeline without hiding real product risk.

LLM apps fail differently from normal web apps. The same prompt can pass nine times and fail once because the model sampled a weak answer, the retriever returned a slightly different chunk, or an external tool responded slowly. If you run those evals in CI with a classic binary mindset, your pipeline turns into noise. If you skip too much, your release gate becomes theatre.

Table of Contents

Contents

What changed in DeepEval 4.1.5?

DeepEval 4.1.5 was published on 29 July 2026 with the release title “Flaky tests? Skip the failures!”. The changelog says the release added a flaky flag to metrics and test cases, plus bug fixes for system instruction parsing and recursive schema handling. The package metadata on PyPI lists DeepEval as “The LLM Evaluation Framework,” and its public GitHub repo had more than 17,000 stars when I checked for this article.

The official DeepEval quickstart positions the framework around local evals, test cases, metrics, and the deepeval test run command. That matters for QA because it feels familiar. You write a test case, choose a metric, run it locally, then wire it into CI. The hard part is not the command. The hard part is deciding what a failure means when the system under test is probabilistic.

Why a flaky flag is useful

There are valid cases where an eval is temporarily noisy but still valuable. A semantic similarity metric may bounce around a threshold. A judge model may produce slightly different reasoning. A RAG test may depend on documents being re-indexed. Marking that eval as flaky can keep the build usable while the team investigates.

Why a flaky flag is risky

The risk is obvious to anyone who has maintained Selenium or Playwright suites for years. Once a team learns that “flaky” means “not blocking,” the label spreads. Yesterday it was one unstable test. Next month it is a folder of release-critical checks. At that point the pipeline is green, but the customer still gets the bug.

I have seen this pattern in UI automation, API testing, and now LLM evaluation. Tooling gives us a safety valve. Process decides whether that safety valve becomes a leak.

Why DeepEval flaky tests happen in LLM systems

DeepEval flaky tests are usually a symptom, not the root cause. The best teams do not start by asking, “How do we skip this?” They ask, “What kind of uncertainty did we just measure?” That question changes the debugging path.

1. Model randomness

LLMs are not deterministic unless you design the system to behave that way. Temperature, top-p, model version, tool calls, context window pressure, and provider-side updates can all move the output. Even when you set temperature to zero, some hosted models do not guarantee identical responses forever.

For QA, this means the old assertion style breaks quickly:

  • Exact string match becomes too brittle.
  • Single-run pass/fail hides variance.
  • One threshold cannot represent every user risk.
  • A green build may only prove that this run got lucky.

2. Retrieval drift

RAG systems add another source of movement. The prompt may be stable, but the retrieved context can change after a content update, embedding refresh, index rebuild, permission filter change, or chunking tweak. A test that passed last week can fail today because the app used the wrong source, not because the model suddenly became worse.

This is why I like linking LLM eval failures to evidence. If the eval fails, capture the prompt, output, retrieved chunks, model name, metric score, and trace ID. Without those artifacts, the triage meeting becomes opinion versus opinion.

3. Judge model variance

Many LLM evals use another model as a judge. That is useful for semantic checks like helpfulness, faithfulness, hallucination, and answer relevancy. But a judge is still a model. It can be inconsistent, too strict, too generous, or badly aligned with your domain.

When a judge-driven metric fails by one or two points around a threshold, I do not immediately call the product broken. I rerun, inspect the reasoning, and compare against a small human-reviewed sample. If the same category keeps failing, I treat it as product risk.

4. Test data quality

Many flaky LLM evals are actually weak datasets wearing a tool problem mask. Ambiguous expected outputs, duplicate scenarios, stale business rules, and missing negative cases all create unstable scores. The eval framework gets blamed, but the dataset is the real issue.

Before adding a flaky label, ask three questions:

  1. Can a human reviewer answer this test case consistently?
  2. Does the expected output describe the behavior we really want?
  3. Is the failure tied to a known product risk or just wording preference?

How to use DeepEval flaky tests without hiding bugs

DeepEval flaky tests can be useful if the team treats them as quarantined evidence, not deleted evidence. My rule is simple: a flaky eval can stop blocking the release only after it starts creating a separate work item, a dashboard signal, and a time-bound owner.

Use “flaky” as a quarantine state

Quarantine is different from ignore. Ignore means the team stops seeing the problem. Quarantine means the eval still runs, the result is still recorded, and the failure still has an owner. The only thing that changes is whether that single test blocks the release immediately.

A healthy quarantine policy includes:

  • Owner: one SDET or squad owns the flaky eval.
  • Expiry: every flaky label expires after 7 or 14 days.
  • Evidence: every run stores prompt, output, metric score, and trace.
  • Trend: repeated failures become blocking again.
  • Review: the team reviews flaky count in release readiness.

Separate noisy checks from critical checks

Not all LLM evals deserve the same policy. A style preference test for tone can be non-blocking. A hallucination test for a financial workflow should block. A tool-call permission test should block. A vague “answer quality” metric may need a warning threshold first.

I use three levels:

  1. Blocker: safety, privacy, permission, compliance, destructive action, severe hallucination.
  2. Warning: answer quality, helpfulness, minor retrieval mismatch, tone drift.
  3. Research: experimental metric, new judge prompt, dataset under construction.

Track flaky rate, not just pass rate

A pass rate alone is easy to manipulate. Mark enough tests as flaky and the dashboard looks clean. Flaky rate is harder to hide. If 18 out of 200 evals are quarantined, the release conversation should include that number. If the number increases for three runs, you have a quality trend, not a random inconvenience.

This is the same lesson QA teams learned with UI automation. “99% pass rate” means little if the 1% keeps hitting checkout, login, and payment. For LLM systems, the risky areas are policy compliance, retrieval grounding, tool use, and refusal behavior.

A 4-bucket model for DeepEval flaky test failures

I recommend classifying every flaky LLM eval into four buckets. This turns noisy red builds into actionable engineering work.

Bucket 1: Product behavior bug

The model or agent gave an answer the product should never give. It ignored instructions, hallucinated a policy, exposed sensitive data, used the wrong tool, or failed a user-critical task. Do not hide this behind a flaky label. If it can hurt the customer, it remains blocking until the product behavior is fixed or the product owner explicitly accepts the risk.

Bucket 2: Prompt or retrieval issue

The app failed because the prompt was unclear, the context was missing, the retriever selected weak chunks, or the system message conflicted with user instructions. This is common in RAG products. It belongs to the feature squad, not only the QA team.

Useful evidence includes:

  • Top retrieved chunks with scores
  • Prompt template version
  • System instruction version
  • Model and provider
  • Metric score and judge explanation

Bucket 3: Evaluation design issue

The app behaved acceptably, but the test was badly designed. Maybe the expected output was too narrow. Maybe the threshold was unrealistic. Maybe the judge prompt rewarded verbosity when the product wants concise answers. This is where SDETs can create serious impact.

Do not delete the test immediately. Fix the dataset, improve the rubric, add examples, and run an A/B comparison. If the new eval catches real failures with fewer false alarms, replace the old one.

Bucket 4: Infrastructure or environment issue

The eval failed because of rate limits, timeouts, stale test data, CI secrets, provider incidents, or tool API instability. These failures should not be mixed with product failures. Mark them separately and fix the environment.

If infrastructure noise is high, your LLM evaluation program will lose trust. Engineers will stop reading failures. Managers will stop treating the gate seriously. The fix is boring but important: stable test data, retry policy, provider status checks, and clean logs.

If you want a deeper model for evidence-based AI testing, I already wrote about it in AI Testing Evidence: Stop Trusting Green Checks and AI Test Failure Classification: 4 Buckets for QA.

CI/CD policy for DeepEval flaky tests

A flaky flag only works when the CI policy is explicit. Otherwise every squad invents its own interpretation. Here is the policy I would start with for an AI QA team.

Policy 1: Run every eval, even quarantined ones

Do not remove flaky evals from the test command. Run them in a separate job if needed, but keep collecting scores. Your trend data is the product. If the eval disappears, you lose the only evidence that could prove the fix worked.

Policy 2: Rerun only the right failures

Reruns are useful for transient model or infrastructure noise. They are dangerous for safety and policy failures. If a prompt injection test fails once and passes on retry, I still want the team to inspect it. The customer does not get three attempts in production before the bad answer disappears.

Policy 3: Escalate repeated flakes

A test that fails once in ten runs may be noisy. A test that fails in three consecutive releases is a product signal. Build this into the pipeline. For example:

  • One flaky failure: mark warning and create evidence record.
  • Two failures in 24 hours: assign owner and link issue.
  • Three failures in a release branch: block until triaged.
  • Flaky label older than 14 days: block until reviewed.

Policy 4: Keep a release note trail

Every release should state how many LLM evals ran, how many blocked, how many were flaky, and which risks were accepted. This is not bureaucracy. It is the audit trail that protects QA when a production issue appears later.

For more release-gate thinking, see AI Test Evidence in CI/CD Release Gates and PromptFoo vs DeepEval: QA Guide for LLM Tests.

A runnable pytest and DeepEval pattern

The exact syntax for flaky flags can change across DeepEval versions, so verify against the version installed in your project. The pattern below is the part I care about: keep the eval visible, record evidence, and make the CI decision outside the metric itself.

Install and pin the tool

python -m venv .venv
source .venv/bin/activate
pip install -U deepeval pytest
python -c "import deepeval; print(deepeval.__version__)"

Create an eval with explicit risk metadata

# tests/test_support_bot_eval.py
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

CRITICALITY = "warning"  # blocker | warning | research
OWNER = "qa-ai-platform"

def call_support_bot(question: str) -> dict:
    # Replace with your app call.
    return {
        "answer": "You can reset the password from Settings > Security.",
        "context": ["Password reset is available under Settings > Security."],
        "trace_id": "local-demo-001",
    }

def test_password_reset_answer_is_grounded():
    result = call_support_bot("How do I reset my password?")
    test_case = LLMTestCase(
        input="How do I reset my password?",
        actual_output=result["answer"],
        retrieval_context=result["context"],
        expected_output="Tell the user to reset the password from Settings > Security.",
    )

    metrics = [
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.8),
    ]

    # If your DeepEval version supports a flaky flag, attach it according to
    # the official docs. Still record owner, criticality, and trace ID.
    print({"owner": OWNER, "criticality": CRITICALITY, "trace_id": result["trace_id"]})
    assert_test(test_case, metrics)

Gate the build with a policy wrapper

# ci/eval_policy.py
import json
from pathlib import Path

MAX_FLAKY_WARNINGS = 5
BLOCKING_BUCKETS = {"safety", "privacy", "permission", "severe_hallucination"}

def decide(report_path: str) -> int:
    report = json.loads(Path(report_path).read_text())
    flaky = [x for x in report["results"] if x.get("status") == "flaky"]
    blockers = [x for x in report["results"] if x.get("risk_bucket") in BLOCKING_BUCKETS]

    if blockers:
        print(f"Blocking release: {len(blockers)} critical eval failures")
        return 1

    if len(flaky) > MAX_FLAKY_WARNINGS:
        print(f"Blocking release: flaky eval count {len(flaky)} exceeds budget")
        return 1

    print(f"Release allowed with {len(flaky)} quarantined evals")
    return 0

if __name__ == "__main__":
    raise SystemExit(decide("artifacts/deepeval-report.json"))

This wrapper looks simple, but it protects the team from a common mistake: putting all release judgment inside the test framework. The framework runs evals. The QA policy decides what risk the organization accepts.

India SDET context: why this skill matters now

For SDETs in India, this is a career opening. Service companies still need strong automation engineers, but product companies increasingly want people who can own AI quality gates, eval datasets, model monitoring, and CI evidence. The difference between “I can run DeepEval” and “I can design a flaky eval policy” is big.

In interviews, I would expect stronger candidates to explain:

  • Why LLM evals fail differently from Selenium or API tests
  • How to separate product bugs from evaluation design bugs
  • When reruns are acceptable and when they hide safety risk
  • How to store prompt, output, context, metric, and trace evidence
  • How flaky eval trends should influence release decisions

If you are targeting senior SDET, staff QA, or AI quality engineer roles, build a small demo. Use DeepEval for metric-based evaluation. Use PromptFoo for prompt regression if your team prefers YAML-first workflows. Put both in GitHub Actions. Add a dashboard screenshot to your portfolio. That is stronger than another generic “AI testing enthusiast” line on LinkedIn.

A practical portfolio exercise

Pick one public FAQ page. Build a tiny RAG bot over it. Create 30 eval cases: 10 happy path, 10 edge cases, 5 hallucination traps, and 5 permission or refusal cases. Run the eval suite five times. Mark flaky cases, classify them using the 4-bucket model, and write a one-page release note.

This exercise proves that you understand AI testing as engineering, not as prompt decoration.

Migration checklist for QA teams

If your team already uses DeepEval, do not turn on flaky behavior casually. Treat it like a release-process change.

Step-by-step rollout

  1. Pin the version. Record the DeepEval version, model names, and metric thresholds.
  2. Inventory current failures. Separate deterministic product failures from noisy evals.
  3. Create risk tiers. Define blocker, warning, and research evals.
  4. Add evidence capture. Store input, actual output, retrieval context, score, and trace ID.
  5. Define flaky expiry. No flaky label lives forever.
  6. Track flaky budget. Set a maximum count or percentage per release.
  7. Review weekly. Flaky evals need grooming like bugs.
  8. Report in release notes. Make accepted AI quality risk visible.

Metrics I would track

  • Total evals executed per build
  • Blocking failures by risk bucket
  • Flaky eval count and trend
  • Mean metric score for critical journeys
  • Dataset freshness date
  • Top recurring failure reason
  • Median triage time for flaky evals

Notice what is missing: vanity pass rate. Pass rate helps, but it is not enough for AI systems. I want to know whether the app is becoming safer, more grounded, and more consistent across releases.

Key takeaways for DeepEval flaky tests

DeepEval flaky tests should make your LLM eval pipeline more honest, not more relaxed. The feature is useful because LLM systems are noisy. The danger is that teams use noise as an excuse to lower the bar.

  • DeepEval 4.1.5 added flaky support for metrics and test cases.
  • A flaky label should mean quarantine, not deletion.
  • Critical safety, privacy, permission, and severe hallucination tests should remain blocking.
  • Every flaky eval needs owner, expiry, evidence, and trend tracking.
  • SDETs who can design this policy will stand out in AI QA interviews.

My recommendation: start strict. Allow flaky handling only for warning-level evals. Keep the evidence visible. Review the count weekly. If a flaky eval keeps failing, promote it back to blocking and fix the product, prompt, retrieval, dataset, or environment.

FAQ

Are DeepEval flaky tests bad practice?

No. They are bad practice only when used to hide failures. They are useful when used as a quarantine state with owner, expiry, evidence, and trend review.

Should flaky LLM evals block CI?

It depends on risk. Safety, privacy, permission, and severe hallucination failures should block. Style, tone, or experimental metrics can warn first, provided the team reviews them.

How many reruns should an LLM eval get?

I usually start with one rerun for warning-level evals and zero reruns for critical risk categories. If a critical test fails once, I want the evidence reviewed.

Is DeepEval better than PromptFoo for flaky tests?

They solve overlapping but different problems. DeepEval is strong for Python-based metric evaluation and LLM test cases. PromptFoo is strong for prompt regression workflows and matrix-style comparisons. Many QA teams can use both.

What should SDETs learn next?

Learn eval design, dataset curation, RAG failure analysis, prompt regression, and CI/CD release gates. Tools matter, but the policy around the tools is where senior QA value shows up.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.