LLM Hallucination Testing: How to Catch AI Making Things Up

Most QA teams still test AI apps the same way they test a REST endpoint: feed an input, check the status code, move on. That is exactly how an LLM hallucination ships to production. A model returns a fluent, confident, completely invented answer, and your happy-path assertion passes because the response was 200 and well-formed. LLM hallucination testing is the discipline that catches this. It is not about whether the model replied. It is about whether the reply is true.

Table of Contents

• What Is an LLM Hallucination, Exactly?
• Why LLM Hallucination Testing Is Harder Than Unit Testing
• The Two Detection Families: Reference-Based vs. Self-Consistency
• Specialized Detectors Beat a General LLM Judge
• Build a Hallucination Test Suite (With Real Code)
• Turn It Into a CI Gate That Fails the Build
• The Tool Cheat Sheet for Hallucination Testing
• India Context: What This Skill Is Worth in 2026
• Mistakes QA Teams Make When Testing Hallucinations
• Key Takeaways
• FAQ: LLM Hallucination Testing

What Is an LLM Hallucination, Exactly?

A hallucination is when a model generates text that is fluent and plausible but factually wrong, unsupported by the source material, or internally contradictory. The dangerous part is the fluency. A hallucination reads like a correct answer, so nobody notices until a user reports it or a release breaks.

There are two kinds you need to separate, because they need different tests:

• Intrinsic hallucination: the output contradicts the context or source it was given. You gave the model a document that says the launch was on 23 August 2023, and it answered 2021. This is the easier one to test, because you have a reference to compare against.
• Extrinsic hallucination: the output invents facts that are not in the context and cannot be verified from it. The model names a tool that does not exist, or cites a paper nobody wrote. No reference catches this directly, which is why it is harder.

This is different from a RAG retrieval failure. If your retriever pulls the wrong document, the model may faithfully summarize a wrong source. That is a retrieval problem, not a hallucination. I covered retrieval and faithfulness separately in my RAG testing guide. Here I am focusing on the model inventing things on its own.

Why LLM Hallucination Testing Is Harder Than Unit Testing

Unit tests have one right answer. add(2, 2) is 4, always, on every run, forever. An LLM has a distribution of answers, not a single one. The same prompt can return “August 2023”, “23 August 2023”, or “somewhere in 2023” and all three are correct. A string match fails on all three.

Three properties make hallucination testing genuinely hard:

1. No single ground truth. Correct answers are paraphrases of each other. Your assertion has to judge meaning, not bytes.
2. Plausible wrong answers. A hallucinated answer is fluent and confident. “Chandrayaan-3 landed in 2021” looks just as valid as the true 2023 to a naive checker.
3. Non-determinism. Run the same prompt five times, get five outputs. A test that passes today can fail tomorrow with no code change, which is how flaky test suites get into AI projects.

That last point is the one I see teams ignore. They add a few golden prompts, run them once in CI, see green, and call it done. Then the model gets a minor provider update, the temperature shifts, and the suite goes red for reasons nobody understands. If you have read my piece on token and latency gates, you already know LLM tests need thresholds and tolerance, not binary asserts.

The flakiness problem, concretely

I have watched a team ship a support bot where the eval suite was green for three straight weeks, then a single provider update flipped two prompts red overnight. Nothing in their code changed. The bot still answered correctly; the wording had shifted just enough to trip a string-match assertion. That is the real cost of LLM hallucination testing done badly. The team stops trusting the suite, starts ignoring the failures, and then a real hallucination ships because the alert nobody believed finally fired on a real bug.

This is why every assertion in this article uses a score and a threshold, not an equality check. A probabilistic system needs a tolerance band, or your test suite becomes the thing everyone mutes.

The Two Detection Families: Reference-Based vs. Self-Consistency

Every hallucination detection method I have seen falls into one of two buckets. Knowing which bucket you are in tells you what you can and cannot catch.

Reference-based detection (faithfulness)

You give the detector the model’s answer plus a source, and it asks: does the answer follow from the source? This is the family that powers Vectara’s HHEM model (167,000+ downloads on Hugging Face), Ragas’s faithfulness metric, and DeepEval’s HallucinationMetric. It is essentially a natural-language inference task: the answer either entails the source, contradicts it, or is neutral.

Vectara popularized this approach in 2023 with HHEM, a small model trained specifically to score whether a summary contradicts its source documents. That is why a reference-based detector is fast and cheap: it does not need a frontier model to judge every answer.

Self-consistency (sampling-based)

If you have no source, you sample the model multiple times on the same prompt and compare the answers. The idea comes from SelfCheckGPT: a model that hallucinates gives inconsistent answers across samples, while a grounded answer stays stable. If you ask five times and get five different launch years, one of them is probably invented.

Reference-based methods catch contradictions with a known source. Self-consistency catches the stuff a model makes up when there is no source. Production systems usually need both, because user prompts rarely come with a clean reference document attached.

My rule of thumb: if the answer is supposed to come from a document, a knowledge base, or a tool result, use a reference-based detector. It is faster, cheaper, and deterministic enough for CI. If the model is answering from its own parameters, self-consistency is your only option, and you should budget for the extra latency.

Specialized Detectors Beat a General LLM Judge

Here is the part most teams get wrong. They set up an “LLM-as-a-judge” prompt that asks GPT-4o to check for hallucinations, and they assume a bigger model is a better judge. The benchmark data says otherwise.

Patronus AI trained Lynx, an 8-billion-parameter model built on Llama 3, specifically for hallucination detection. They evaluated it on HaluBench, a benchmark of thousands of samples across question answering, summarization, and RAG tasks. Patronus reports that Lynx, at 8B parameters, outperformed much larger general-purpose models at spotting hallucinations on the domains it was trained for.

That matters for your budget and your latency. You do not need to burn a frontier model call for every answer you validate. A dedicated detector like HHEM or Lynx is cheaper, faster, and often more accurate at this one job. Keep the general LLM judge for the open-ended cases where you genuinely need reasoning, and use the specialized model for the high-volume faithfulness checks.

Build a Hallucination Test Suite (With Real Code)

Stop testing one-off prompts by hand. Build a golden set: a list of (question, context, expected answer) triples where you know the truth. For every triple, include at least one adversarial case where you have injected a known hallucination into the context so you can prove your detector actually fires.

Here is a DeepEval example. The metric scores the output against the context; a low score means the model invented facts that are not in the context.

from deepeval import evaluate
from deepeval.metrics import HallucinationMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What year did Chandrayaan-3 land on the Moon?",
    actual_output="Chandrayaan-3 landed near the lunar south pole in 2021.",
    context=["Chandrayaan-3 soft-landed on the Moon on 23 August 2023."],
)

metric = HallucinationMetric(threshold=0.5)
metric.measure(test_case)

print(metric.score)   # 0.0 here: output contradicts context
print(metric.reason)  # explains the contradiction it found
assert metric.score >= metric.threshold  # fails the run

The same idea in Ragas, which measures faithfulness as a score from 0 to 1 where higher is better:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness

data = Dataset.from_dict({
    "question": ["What year did Chandrayaan-3 land?"],
    "answer": ["Chandrayaan-3 landed in 2021."],
    "contexts": [["Chandrayaan-3 soft-landed on 23 August 2023."]],
})

result = evaluate(data, metrics=[faithfulness])
print(result["faithfulness"])  # low = answer not supported by context

When I design a golden set, I follow five rules so the suite actually catches bugs instead of just looking impressive:

1. Every fact gets a source. If a case has no context, it is testing the model’s memory, not your app. Attach the document the answer should come from.
2. Inject false cases on purpose. For every five clean cases, add one where the expected answer is a known hallucination. The suite is only real if it fails that case.
3. Cover paraphrases. Include the same correct fact worded three ways, so your detector learns that “August 2023” and “23 August 2023” are both right.
4. Pin the temperature. Run your eval at the same temperature you use in production, and record it, so a flaky week is easy to diagnose.
5. Version the set. Golden sets change as your product changes. Store them in the repo next to the code, not in a spreadsheet.

The trick that makes this a real suite, not a demo: keep the injected-hallucination cases in the same run as the clean cases. A detector that scores everything high, including the cases you know are false, is a detector that is not working.

Turn It Into a CI Gate That Fails the Build

A hallucination check that runs in a notebook is a toy. Wire it into the pipeline so a model or prompt change cannot ship past a failing score. PromptFoo makes this easy with a YAML config and assertions:

# promptfooconfig.yaml
prompts: [prompt.json]
providers: [openai:gpt-4o]
tests:
  - vars:
      question: What year did Chandrayaan-3 land on the Moon?
      context: Chandrayaan-3 soft-landed on 23 August 2023.
    assert:
      - type: contains
        value: "2023"
      - type: not-contains
        value: "2021"
      - type: llm-rubric
        value: "The answer must be supported by the context. Flag any invented date or fact."

Run promptfoo eval in CI and the build fails when the rubric flags a contradiction. This is the same pattern I use in my DeepEval vs Promptfoo vs Ragas comparison, where I map each tool to the CI stage it belongs in.

Set the threshold as a team decision, not a default. Too strict and every paraphrase fails. Too loose and hallucinations slip through. Start at a faithfulness score around 0.7 to 0.8, run it for a week, and tighten from the failures you actually care about.

Here is the five-step path from “we should test this” to “the build is gated on it”:

1. Put the golden set in the repo. A evals/ folder with the questions, contexts, and expected answers, versioned with the code.
2. Wire the detector in. DeepEval or Ragas in a pytest run, or PromptFoo as a separate promptfoo eval step in your pipeline.
3. Run it on every pull request. Any change to a prompt, a model, or a context chunk triggers the eval, not just deploys.
4. Fail on threshold, not on text. The build goes red when the faithfulness score drops below your number, not when wording changes.
5. Log the reasons. Store the detector’s explanation for every failure so the triage is “why did it think this was false,” not “what happened.”

The Tool Cheat Sheet for Hallucination Testing

• Vectara HHEM: a purpose-built detector for RAG summary hallucinations. Fast and cheap. Good first tool when you have a source document.
• Patronus Lynx + HaluBench: an 8B detector and its benchmark. Use it to validate your own detector on domain-specific data.
• DeepEval HallucinationMetric: a verdict-based metric with a threshold. Fits pytest workflows, which QA engineers already know.
• Ragas faithfulness: built for RAG pipelines. Pair it with answer relevancy and context precision for a full picture.
• PromptFoo: 24,000+ GitHub stars, config-driven assertions that slot straight into CI. Best for release gates.
• SelfCheckGPT: the sampling approach when you have no reference. Slower, but catches extrinsic hallucinations.

Which one fits your stack? If your team already writes pytest, DeepEval is the smallest leap. If you run CI with config files and want a release gate, PromptFoo is the fastest to stand up. If you already run a RAG pipeline, Ragas gives you faithfulness plus the retrieval metrics you need anyway. Do not adopt all six. Pick one, stand up a gate this week, and add a second only when the first is failing builds in a useful way.

If you need to watch these signals over time in production, that is the observability layer I covered in my AI observability guide. Detection finds the bug. Observability tells you whether it is getting worse.

India Context: What This Skill Is Worth in 2026

Every AI-enabled QA role I see posted in Bengaluru now lists “LLM evaluation” or “hallucination testing” in the requirements. It is not a nice-to-have anymore. A senior SDET who can build an eval gate that catches hallucinations is interviewing at a different level than one who can only automate a web form.

Here is the practical salary picture. Manual and basic automation roles in India sit around ₹6 to ₹15 LPA. Testers who add AI evaluation, faithfulness scoring, and CI eval gates to their resume are the ones moving into the ₹25 to 40 LPA senior SDET and AI quality engineer band. The gap is not a framework. It is the ability to define what “correct” means for a probabilistic system and enforce it automatically.

If you want a portfolio project that demonstrates this, build a 20-case golden set for a small RAG app, wire DeepEval or Ragas into GitHub Actions, and show the build going red on an injected hallucination. That one repo says more than ten years of Selenium on a resume. My DeepEval for QA guide gives you the exact starting point.

Mistakes QA Teams Make When Testing Hallucinations

1. Asserting on string equality. A paraphrase is not a bug. Judge meaning, not exact text.
2. Testing only happy-path prompts. If your golden set has no injected hallucinations, you are measuring nothing.
3. Trusting a general LLM judge blindly. Use a task-specific detector for volume, and audit the judge itself periodically.
4. Ignoring the temperature. Higher temperature means more variance and more flaky failures. Pin it in your test runs.
5. No threshold, no tolerance. Binary asserts on probabilistic output create a flaky suite nobody trusts.

Key Takeaways

• A hallucination is fluent, confident, and wrong. LLM hallucination testing exists to catch it before users do.
• Separate intrinsic (contradicts context) from extrinsic (invents facts) hallucinations, and test both.
• Reference-based detectors like HHEM, Ragas, and DeepEval catch contradictions with a source. SelfCheckGPT-style sampling catches inventions without one.
• A small, task-specific detector like Patronus Lynx can beat a frontier model at this job.
• Ship it as a CI gate with a real threshold, a golden set, and injected false cases, not a notebook script.

FAQ: LLM Hallucination Testing

What is the difference between a hallucination and a RAG retrieval failure?

A retrieval failure gives the model the wrong source, so it faithfully summarizes wrong information. A hallucination is the model inventing facts that are not in the source it was given. Faithfulness metrics catch the second; retrieval metrics catch the first.

Which tool should a QA team start with?

Start with DeepEval if your team writes Python and pytest, or PromptFoo if you prefer config-driven CI gates. Both wrap the same reference-based idea and have the shallowest learning curve.

Can I detect hallucinations without a reference document?

Yes, but it is harder. Self-consistency approaches like SelfCheckGPT sample the model repeatedly and flag inconsistent answers. They cost more and are slower, so use them only where no source is available.

What threshold should I set for a faithfulness score?

There is no universal number. Start around 0.7 to 0.8, run it in CI for a week, and tighten based on the failures you actually care about. The threshold is a product decision, not a default.

How do I know my detector itself is working?

Keep injected hallucination cases in your golden set. A detector that passes the false cases is broken. Re-audit it with a benchmark like HaluBench on your own domain data.