How to Evaluate LLM Outputs with DeepEval (2026 Guide)

Stop saying your AI test passed. You did not test anything. You read a chatbot’s answer, nodded, and moved on. That is not QA, that is proofreading. If you want to evaluate LLM outputs the way you evaluate a login page or an API contract, you need metrics, test cases, and thresholds. DeepEval gives you all three. In this guide I will show you exactly how to evaluate LLM outputs with DeepEval, from a first failing test to a regression suite you can run in CI.

Table of Contents

• Why “It Passed” Is Not a Test Result
• How to Evaluate LLM Outputs Without Eyeballing Them
• What DeepEval Brings to LLM Evaluation
• Setting Up DeepEval in Under Five Minutes
• Your First Real Evaluation Test
• Evaluate LLM Outputs: The Metrics QA Teams Actually Use
• Turning Metrics into a Regression Suite
• The Real Cost of Eyeballing AI Outputs
• Gotchas I Hit When I Started Evaluating LLM Outputs
• What This Means for QA Careers in India
• Key Takeaways
• FAQ: Evaluating LLM Outputs with DeepEval

Why “It Passed” Is Not a Test Result

I have watched this happen across a dozen teams in the last year. An engineer builds a RAG pipeline or a support bot. They type three prompts into the UI. The answers look okay, so they write “AI feature tested and working” in the ticket and ship it. Three weeks later, the bot tells a customer they can get a full refund on a product they bought 18 months ago. Nobody measured anything, so nobody saw it coming.

A manual eyeball gives you exactly one data point: whether one human, at one moment, was mildly satisfied. It does not tell you whether the answer was faithful to your knowledge base, whether it answered the question that was actually asked, or whether the same prompt gives a different answer tomorrow. LLMs are probabilistic, so the same input does not guarantee the same output. Testing a probabilistic system with a one-off glance is like testing a login page by clicking the button once and assuming it always works.

What you need is a repeatable measurement. You need to define what “correct” means, score the output against that definition, and fail the build when the score drops below a bar you chose. That bar is called a threshold, and it is the single most important concept in LLM evaluation.

How to Evaluate LLM Outputs Without Eyeballing Them

Evaluating an LLM output means converting a fuzzy judgment into a number between 0 and 1, then comparing that number to a threshold. The moment you have a number, you can do everything you already do in test automation: assert, log, trend, and fail CI.

The mechanics work in three layers.

1. Test cases. A test case captures the input you sent, the output the model produced, and optionally what you expected plus any retrieval context. In DeepEval this is the LLMTestCase object.
2. Metrics. A metric is the rule that turns a test case into a score. DeepEval ships with a set of research-backed metrics like AnswerRelevancyMetric, FaithfulnessMetric, and GEval for custom criteria.
3. Thresholds. Every metric takes a threshold between 0 and 1. Score below it, the test fails. That single line is what separates “it looked fine” from “it passed.”

The judging itself is done by an LLM. This is called LLM-as-a-judge, and it is the pragmatic middle ground between a human reviewing thousands of outputs and fragile string matching. You hand the judge a rubric, it returns a score and a reason. You still own the rubric, the threshold, and the pass or fail call.

What DeepEval Brings to LLM Evaluation

DeepEval is an open-source framework built by Confident AI. Its own tagline is “The LLM Evaluation Framework,” and the numbers back that up. The repository sits at roughly 17,580 GitHub stars with around 1,798 forks as of August 2026, and the latest release on PyPI is 4.1.8, published on August 12, 2026. I pulled those figures directly from the GitHub repository and the PyPI page.

Why it matters for QA is not the star count, it is the design. DeepEval is pytest-native. You write your evaluations as ordinary Python tests and run them with a single command:

deepeval test run test_chatbot.py

That means your LLM evaluation lives in the same repo, the same CI pipeline, and the same reporting surface as your Playwright and API tests. No separate dashboard you forget to check. It also means you can parametrize test cases exactly the way you already parametrize pytest tests.

The metric roster is the second reason to pick it. Out of the box you get Answer Relevancy, Faithfulness, Hallucination, Contextual Relevancy, Contextual Precision, Contextual Recall, Bias, Toxicity, Summarization, Knowledge Retention, and G-Eval for anything custom, plus a RAGAS integration if your team already speaks that dialect. The full list lives in the DeepEval metrics documentation, and the project README walks through each one with a code sample. That breadth is why it has become the default answer when a QA team asks “how do we test our LLM.”

For teams already comparing frameworks, I wrote a separate breakdown of DeepEval vs Ragas that covers when each one earns its place.

Setting Up DeepEval in Under Five Minutes

Installation is one pip command. DeepEval requires Python 3.9 or newer (and stays under 4.0), which fits nearly every QA stack.

pip install deepeval

DeepEval evaluates using an LLM judge, so by default it reads your OPENAI_API_KEY. If you are not on OpenAI, you can point it at a custom model, including local ones through Ollama. For this guide I keep it simple with the default.

export OPENAI_API_KEY="your-key-here"

If you want to push results to the Confident AI dashboard for team visibility, you run deepeval login. I skip that for local runs. The dashboard is optional, the metrics are not.

One more thing worth knowing before you start: DeepEval gives you two entry points. deepeval test run is the pytest path for CI and regression suites. deepeval evaluate is the standalone call for notebooks and quick experiments. Both take the same LLMTestCase objects, so you do not have to learn two APIs. The official getting-started guide shows both side by side, and I recommend reading it once before you write your first metric.

Your First Real Evaluation Test

Here is the canonical starting point, adapted from the official DeepEval quickstart so the API surface is exact. It evaluates correctness against an expected output using GEval, which lets you define your own criteria in plain English.

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

def test_refund_policy_correctness():
    correctness = GEval(
        name="Correctness",
        criteria="Determine if the actual output is correct based on the expected output.",
        evaluation_params=[
            SingleTurnParams.ACTUAL_OUTPUT,
            SingleTurnParams.EXPECTED_OUTPUT,
        ],
        threshold=0.5,
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=["All customers get a 30-day full refund at no extra cost."],
    )
    assert_test(test_case, [correctness])

Run it with deepeval test run test_chatbot.py and you get a pass or a fail, not a shrug. Let me break down what is happening, because this is where most testers get lost.

The Test Case Anatomy

The LLMTestCase has four fields that matter. input is what the user asked. actual_output is what your application actually returned, not what you wish it returned. expected_output is your ideal answer. retrieval_context is the source chunks your RAG pipeline pulled before answering, which is what lets faithfulness checks work at all.

The key shift for QA engineers is this: actual_output is a placeholder for your real system’s response. In a real suite you call your chatbot or pipeline inside the test and pass whatever it returns into the test case. The metric then scores the real behavior, not a hardcoded string.

Thresholds Are the Point

Every metric score lands between 0 and 1, and the threshold is what turns that score into a pass or fail. Set it too low and junk ships. Set it too high and the judge’s own noise fails your build every other run. For correctness I start at 0.5 and tune up. For answer relevancy, which matters more to end users, I start at 0.7. There is no magic number, only a number you chose deliberately and can defend in review.

Evaluate LLM Outputs: The Metrics QA Teams Actually Use

You do not need all twenty-odd metrics. You need the four or five that map to the failure modes you actually ship. Here are the ones I reach for first, with the exact class names from DeepEval 4.x.

Answer Relevancy

AnswerRelevancyMetric checks whether the answer actually addresses the question asked, and whether it wastes tokens on irrelevant filler. It is the metric that catches a support bot rambling about shipping policy when the customer asked about a refund. Set a threshold of 0.7 and watch how many of your “fine” answers fail on the first run.

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="We offer a 30-day full refund at no extra cost.",
    retrieval_context=["All customers get a 30-day full refund at no extra cost."],
)
evaluate([test_case], [metric])

Faithfulness and Hallucination

FaithfulnessMetric checks whether every claim in the answer is supported by the retrieval context you provided. It is the metric that catches hallucination. If your bot asserts a 30-day refund but your knowledge base says 14 days, faithfulness drops and you know the answer was invented rather than retrieved. For RAG systems, this is the metric that matters most, because a confident hallucination is the failure users actually notice.

Contextual Precision and Recall

ContextualPrecisionMetric and ContextualRecallMetric evaluate the retrieval step itself. Precision asks whether the chunks you pulled were relevant and ranked well. Recall asks whether you pulled everything you should have. If your embeddings are ranking a blog post about returns above the actual policy document, precision catches it before the answer stage even runs.

GEval for Anything Custom

When no off-the-shelf metric fits, GEval lets you write your own rubric in a sentence. Tone, policy compliance, formatting, brand voice, “does this sound human” are all criteria you can describe and score. This is the escape hatch that keeps you from being boxed into the shipped metrics.

Summarization, Knowledge Retention, and the Rest

Beyond the core four, a few more earn their keep. SummarizationMetric scores whether a summary captures the source without dropping the load-bearing details, which matters for any feature that compresses meeting notes or support threads. KnowledgeRetentionMetric checks how much of the important information survives the answer. BiasMetric and ToxicityMetric catch the brand-safety failures that a correctness check will happily ignore. You do not wire all of these up on day one. You add them when a specific failure mode shows up in production, which is the right way to grow a suite anyway.

Turning Metrics into a Regression Suite

One test case is a demo. A hundred is a regression suite. Because DeepEval is pytest-native, you parametrize exactly the way you already do.

import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

CASES = [
    ("What if these shoes don't fit?", "30-day refund policy"),
    ("Do you ship to India?", "international shipping policy"),
    ("What is your return window?", "return window policy"),
]

@pytest.mark.parametrize("question,source", CASES)
def test_support_bot(question, source):
    # Call your real application here.
    answer = your_support_bot(question)
    context = your_retriever(question)
    test_case = LLMTestCase(
        input=question,
        actual_output=answer,
        retrieval_context=context,
    )
    assert_test(test_case, [
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.7),
    ])

Now every prompt change, every model swap, every chunking tweak runs against your full case set in CI. When a case goes red, you know exactly which question, which metric, and why, because each metric also returns a reason alongside the score.

The Real Cost of Eyeballing AI Outputs

The cost is not theoretical. A hallucinated answer in a support bot becomes a customer refund you never intended. A RAG pipeline that retrieves the wrong source becomes a compliance answer that is flatly wrong. Manual review catches the obvious cases and misses the subtle ones, which are exactly the ones that scale to thousands of conversations a day.

There is also a talent cost. Teams that ship AI features without any evaluation spend their weekends triaging vague bug reports. I wrote about how to triage AI test failures separately, and the first lesson there is the same: you cannot triage what you never measured. Teams that build the evaluation suite up front replace “the bot said something weird” with a failing test case and a score, which is a fixable artifact instead of a mystery.

Gotchas I Hit When I Started Evaluating LLM Outputs

None of this is free, and I want to be honest about the rough edges so they do not surprise you.

• The judge is an LLM, so it has opinions. If you use a weak or small model as the judge, its scores drift. A cheap judge can pass a bad answer because it did not understand the rubric. Spend a little on a capable judge for the metrics that gate your release.
• Scores are non-deterministic. Run the same case twice and you can get 0.71 then 0.68. That is why thresholds need margin. A 0.7 threshold on a metric that jitters by 0.05 will flake, exactly like a timing-dependent UI test.
• Cost scales with cases. Every metric call is an LLM call. A hundred cases across three metrics is three hundred judge calls per run. That is cheap today, but it is not zero, so do not put the full suite on every commit. Run a smoke subset on PRs and the full suite nightly.
• Your test cases rot. The moment the product changes its policy, your expected outputs are stale. Treat the case set like test data, not a one-time write. Review it with the same cadence you review your Playwright locators.
• Garbage context makes faithfulness meaningless. If your retriever returns nothing useful, faithfulness will flag every answer. Fix retrieval before you tune the judge, or you will spend a week blaming the metric for a pipeline problem.

What This Means for QA Careers in India

Here is the part I care about most, because I live it. The Indian QA market is splitting into two tracks. Manual testers who can only click buttons are competing for roles that pay 4 to 8 LPA and are shrinking. QA engineers who can write Playwright, and now who can evaluate LLM outputs, are getting offers in the 20 to 40 LPA band at product companies and AI-first startups in Bengaluru, Hyderabad, and Pune.

LLM evaluation is the newest entry in that higher band, and almost nobody has it. Every team shipping an AI feature needs someone who can build a regression suite with metrics and thresholds, and they cannot find those people. If you can walk into an interview and show a DeepEval suite with five metrics and a CI hook, you have separated yourself from ninety percent of the candidates. I have covered the AI testing roadmap in depth before, and the MCP and tool-calling test plans and the prompt injection testing guide are the natural next reads once this one clicks.

Key Takeaways

• Eyeballing an LLM answer is not a test. To evaluate LLM outputs you need metrics, test cases, and a threshold that fails the build.
• DeepEval 4.1.8 is the current release, backed by roughly 17,580 GitHub stars and a pytest-native workflow.
• Start with GEval for correctness, AnswerRelevancyMetric for relevance, and FaithfulnessMetric for hallucination, each with a threshold you chose on purpose.
• A regression suite means parametrizing your cases and running deepeval test run in CI, so every prompt and model change is measured.
• The judge is probabilistic and costs money. Give thresholds margin, use a capable judge, and run the full suite on a schedule, not on every commit.

FAQ: Evaluating LLM Outputs with DeepEval

What is the current version of DeepEval?

DeepEval 4.1.8, released August 12, 2026, according to the PyPI project page. It is actively maintained, with the GitHub repository last pushed in August 2026.

Do I need OpenAI to use DeepEval?

No. DeepEval defaults to an OpenAI-compatible judge, but it supports custom models, including local models through Ollama, so you can evaluate with a model you host yourself.

What is a good threshold for LLM evaluation metrics?

There is no universal answer, but a sensible starting point is 0.5 for correctness and 0.7 for answer relevancy and faithfulness. Tune from there based on how much judge noise you observe across repeated runs.

Can DeepEval run in a CI pipeline?

Yes. Because it is pytest-native, deepeval test run runs inside any CI job that can install your repo. Wire it into the same pipeline as your Playwright and API tests and it reports alongside them.

How is DeepEval different from manually reading outputs?

Manual reading gives you one subjective opinion. DeepEval gives you a repeatable score between 0 and 1, a written reason, and a pass or fail against a threshold you set. That is what makes it a test instead of a gut feeling.