| |

RAG Testing: Retrieval, Faithfulness, and Answer Quality

ByPromode
RAG testing: four metrics for retrieval and generation quality

RAG testing is where most AI quality programs fall apart. Teams unit-test the model, eyeball a few chat answers, and ship a retrieval system that silently returns the wrong context half the time. If your product answers questions from a knowledge base or document store, the retrieval step is the bug you cannot see until a customer screenshots it. This guide shows you exactly how to measure retrieval quality, faithfulness, and answer relevance, and how to gate your RAG releases in CI before they reach production.

Table of Contents

Contents

Why RAG Testing Is a Different Kind of QA Problem

RAG, short for retrieval-augmented generation, is the pattern behind most “chat with your data” products. The model does not answer from its own weights. It fetches relevant chunks from a vector database, stuffs them into the prompt, and generates an answer grounded in those chunks. When it works, it is the cheapest way to give an LLM fresh, private knowledge. When it breaks, it breaks quietly.

The scale of the shift is real. LangChain has passed 144,000 GitHub stars, LlamaIndex sits near 51,700, and vector stores like Chroma are at 29,000 stars. The @langchain/core npm package alone moves more than 20 million downloads a month. RAG is no longer an experiment. It is the default architecture for customer-support bots, documentation search, and internal knowledge assistants.

But here is the gap I keep seeing in QA teams: RAG has two components, and most testers only know how to test one of them.

You Are Probably Testing Only Half the System

Manual testers type a question, read the answer, and judge whether it “looks right.” That checks the generation half. It does almost nothing to check whether the system actually retrieved the correct document chunks. A confident, fluent, completely wrong answer gets the same pass as a correct one, because the failure is upstream of the text you are reading.

RAG Failures Are Silent and Slow

  • A missing chunk does not throw an exception. The model just answers from whatever it did retrieve.
  • Hallucination in a RAG app usually means “the model made up an answer because retrieval returned nothing useful.”
  • Each failure is data-dependent, so it reproduces only for specific queries, specific chunk sizes, or a specific document.

This is why RAG testing needs its own metrics and its own harness. You cannot borrow your API test suite and call it done.

The Three Failure Surfaces: Retrieval, Generation, and the Glue

I split every RAG system into three layers, because each one fails in a different way and each one needs a different test.

1. Retrieval: Did We Fetch the Right Chunks?

The retriever turns a user question into a vector search over your document store. It fails when it returns irrelevant chunks, misses the chunk that has the answer, or returns the answer buried so deep in the list that the model never sees it. Retrieval bugs are invisible in the final text but they are the root cause of most RAG hallucinations.

2. Generation: Is the Answer Faithful to the Context?

Even with perfect retrieval, the model can contradict the chunks it was given, add facts from its training data, or refuse to answer. Generation quality is measured against the retrieved context, not against your personal opinion of the answer.

3. The Glue: Chunking, Embeddings, and Re-ranking

Between retrieval and generation sits the plumbing most teams never test: how documents are split into chunks, which embedding model converts text to vectors, and whether a re-ranker reorders results. A chunk size change from 512 to 1,024 tokens can silently break retrieval for an entire document category. This layer is the one I see ignored most often, and it is the subject of a dedicated section below.

Here is the mental model I teach: retrieval tests prove you found the needle, generation tests prove you described the needle correctly, and glue tests prove the haystack did not change shape overnight.

Retrieval Metrics That Matter: Context Precision, Context Recall, Hit Rate

Retrieval is a search problem, so it borrows from the same math that search teams have used for decades. The framework most QA teams start with is Ragas, which packages these metrics cleanly. Here are the four I gate on.

Context Precision

Context precision asks: of the chunks the retriever returned, how many were actually relevant to the question? It is precision applied to retrieval. A low score means you are stuffing the prompt with noise, which wastes tokens and confuses the model.

Context Recall

Context recall asks the reverse: of all the chunks that should have been returned, how many did the retriever actually find? A low score here means the answer is sitting in your vector database and the retriever walked right past it. In my experience, context recall failures are the single biggest source of “the bot says it does not know” complaints.

Hit Rate and MRR

Hit rate is the simplest metric and a good smoke test: did the correct chunk appear anywhere in the top-k results? Mean Reciprocal Rank (MRR) goes further and rewards the system for putting the right chunk near the top, where the model is more likely to use it. If your pipeline keeps the top 5 chunks, a correct chunk at position 5 is far weaker than one at position 1.

Why Precision and Recall Together

  • Precision-only pipelines return one perfect chunk and miss three others. They look good on paper and fail in production.
  • Recall-only pipelines dump 30 chunks into the prompt, blow the token budget, and dilute the signal.
  • You need both, reported together, on the same golden set.

One more thing worth watching: context entities recall, a Ragas metric that checks whether the retrieved context preserved the key entities (names, dates, product codes) from the ground truth. It catches the subtle case where chunks are topically relevant but have dropped the specific fact the user asked for.

Generation Metrics: Faithfulness and Response Relevancy

Once retrieval is solid, you test what the model does with it. The two metrics that matter most are faithfulness and response relevancy, and together with context relevance they form what the TruLens team popularized as the RAG triad: context relevance, groundedness, and answer relevance.

Faithfulness (Groundedness)

Faithfulness measures whether every claim in the answer is supported by the retrieved context. It is hallucination detection, formalized. Ragas scores this by breaking the answer into individual claims and checking each claim against the context with an LLM judge. A 0.6 faithfulness score means roughly four in ten claims came from nowhere.

Response Relevancy (Answer Relevance)

Response relevancy asks whether the answer actually addresses the question. A model can be perfectly faithful and still useless if it answered a different question than the one asked. Faithfulness and relevancy are independent, and you need both: one catches made-up facts, the other catches correct-but-irrelevant facts.

How These Map to Bugs You Already Know

  • Low faithfulness, high relevancy: the bot is confidently hallucinating. This is the scary one.
  • High faithfulness, low relevancy: the bot answers a related but wrong question. Retrieval is usually the culprit.
  • Both low: the whole pipeline is broken, start with retrieval.

These metrics are the reason I keep telling teams not to trust a demo click-through. The demo only exercises generation. It tells you nothing about whether the answer is grounded in your actual documents. For a deeper look at how these frameworks compare, see my breakdown of DeepEval vs Promptfoo vs Ragas.

Building a Golden RAG Test Set (Where Most Teams Fail)

Every metric above is only as good as the test set you run it against. This is where RAG testing programs actually die. A golden set of ten easy questions makes any pipeline look great and catches nothing.

A Golden Set Needs Three Things

  1. The question: a realistic query written the way a real user types, including typos and domain shorthand.
  2. The ground-truth chunks: the exact document passages that contain the answer.
  3. The reference answer: a correct answer a human wrote, used to score retrieval and relevance.

Your Test Set Must Include Hard Queries

A RAG system that passes ten “what is X” questions has proven nothing. Real users ask hard things. Your golden set should include:

  • Multi-hop questions that need facts from two different documents.
  • Negation questions (“which plans do not include SSO?”) that trip up keyword search.
  • Temporal questions (“what changed in the refund policy last quarter?”) where stale chunks are a real risk.
  • Ambiguous questions that need a clarifying response instead of a guess.

Synthetic Test Set Generation

Building a few hundred of these by hand is expensive, which is why Ragas ships a testset generator that reads your source documents and synthesizes query-context-answer triples automatically. It produces single-hop and multi-hop queries plus their ground-truth context, so you can scale a golden set from 30 hand-written questions to hundreds in an afternoon. I still recommend a human reviews the output, but the generator turns a week of manual work into an hour. The same principle applies to the data you feed the rest of your AI tests; I covered the broader synthetic data question in my guide on AI test data generation.

Pin your golden set in version control. The moment you silently “improve” the questions every sprint, your metrics stop meaning anything across releases.

A Working RAG Evaluation Harness with Ragas

Enough theory. Here is a minimal Ragas harness you can run today against your own pipeline.

# pip install ragas datasets
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    response_relevancy,
    context_precision,
    context_recall,
)

# Golden set: question, the generated answer, the retrieved contexts,
# and a human-written ground-truth answer.
golden = Dataset.from_dict({
    "question": [
        "What is the refund policy for annual plans?",
        "Does the Pro plan include SSO before payment?",
    ],
    "answer": [
        "Annual plans can be refunded within 14 days of purchase.",
        "No, SSO is only on the Enterprise tier as a paid add-on.",
    ],
    "contexts": [
        ["Annual plans can be refunded within 14 days of purchase."],
        ["SSO is available on the Enterprise tier as a paid add-on."],
    ],
    "ground_truth": [
        "Annual plans are refundable within 14 days.",
        "SSO requires the Enterprise tier after payment.",
    ],
})

result = evaluate(
    golden,
    metrics=[faithfulness, response_relevancy, context_precision, context_recall],
)
print(result)

Run this and you get a table with four numbers per question: faithfulness, response relevancy, context precision, and context recall. The first two tell you if the answer is honest and on-topic. The last two tell you if retrieval did its job. Four numbers, one command, and you finally know which half of your RAG app is broken.

What the Output Should Make You Do

  • Faithfulness below 0.8: the model is hallucinating. Tighten the prompt, add a re-ranker, or force the model to cite its chunks.
  • Context recall below 0.7: your retriever is missing chunks. Look at embedding model choice and chunk size before touching the prompt.
  • Response relevancy below 0.8: the model is answering the wrong question. This is often a retrieval problem in disguise.

These thresholds are my starting points, not laws. Tune them against your own golden set before you wire anything into CI.

Chunking and Embedding Bugs: The QA Surface Nobody Owns

The least glamorous part of RAG is the part that breaks most often. Chunking and embeddings live between the documents and the retriever, and almost nobody has a test for them.

Chunk Size Changes Break Retrieval Silently

Split your documents into 512-token chunks and a product spec with a table in the middle gets sliced across three chunks. Split into 2,048-token chunks and the retriever returns one giant chunk that buries the answer in noise. There is no universal correct chunk size, which means every change is a regression risk. I treat chunk size like a database index: you do not change it without a benchmark run.

Embedding Model Drift

Your vectors were built with one embedding model. Upgrade the model and every stored vector is now in a different vector space, which silently degrades similarity search until you re-embed the whole store. A retrieval-only test, run before generation, catches this the day it happens instead of three weeks later.

How to Test the Glue

  1. Keep a frozen retrieval benchmark: a set of questions with known correct chunk IDs.
  2. Run the retriever alone and score hit rate and MRR. Do not involve the LLM at all.
  3. On any change to chunking, embedding model, or re-ranking, re-run this benchmark and diff the scores.
  4. Re-embed the vector store as a deployment step, and gate the release on the retrieval benchmark passing.

This is cheap, deterministic, and catches the failures your chat demos never will. It also plugs directly into the observability story I wrote about in my post on AI observability for QA.

Wiring RAG Eval into CI: Release Gates and Budgets

RAG evaluation only changes outcomes when it blocks bad releases. A Jupyter notebook you run once a month is a nice-to-have. A CI gate is the whole point.

A Minimum Viable RAG Gate

  1. On every pull request, run your golden set through the pipeline and score faithfulness, response relevancy, context precision, and context recall.
  2. Compare against the last green release. Fail on a regression above a set margin, for example a 0.05 drop in faithfulness.
  3. Store the scores as build artifacts so you can chart drift over time.
  4. Keep the gate fast enough to not annoy engineers, and cache LLM-judge calls aggressively.

Add a Cost Budget to the Same Gate

Retrieval quality has a direct price tag. Returning 20 chunks instead of 5 quadruples input tokens on every single request. Your eval gate should assert context precision and total input tokens together, so an engineer cannot fix recall by dumping the whole document into the prompt. I went deep on the money side in my post on LLM performance testing, but the short version is this: cost regression is a test failure, not a finance conversation.

Watch the LLM-as-Judge Flakiness

The metrics themselves use an LLM to judge answers, and LLMs are non-deterministic. A faithfulness score of 0.79 one run and 0.81 the next is not a regression, it is noise. Set your failure thresholds wider than that noise band, pin the judge model and temperature, and run the judge at least twice on flaky cases before you page anyone. This is the same discipline you already use for flaky UI tests, just with a different tool in the loop.

India Context: RAG QA Is the Next ₹25-40 LPA Skill

Every product company I talk to in Bengaluru right now is shipping some form of “chat with your data,” and almost none of them have anyone who can test it properly. That gap is the opportunity.

Manual testers and Selenium-only automation folks are competing in a crowded market. The SDETs who can stand up a RAG evaluation harness, define a golden set, and wire faithfulness and context recall into a CI gate are still rare. That skill maps directly to the ₹25-40 LPA senior SDET band in India, and RAG eval work tends to sit at the top of it because it spans three disciplines at once: search relevance, LLM behavior, and release engineering.

If you want a weekend project that gets noticed, do this: take a public FAQ or documentation site, build a tiny RAG pipeline with LlamaIndex or LangChain and Chroma, generate a 50-question golden set, and publish the evaluation report with the four Ragas metrics. That single repo does more for your next interview than another generic Selenium framework. QA hiring managers in India are actively looking for exactly this signal, because they are being asked to ship RAG products and nobody on the team knows how to prove the retrieval works.

Key Takeaways

  • RAG testing means testing retrieval and generation, not just reading chat answers. Retrieval is where the silent failures live.
  • Gate on four metrics: context precision, context recall, faithfulness, and response relevancy. Run them together on a pinned golden set.
  • Your golden set must include multi-hop, negation, temporal, and ambiguous queries, or it proves nothing.
  • Test chunking and embeddings with a frozen retrieval-only benchmark before you ever call the LLM.
  • Wire the eval into CI with regression thresholds and a token-cost budget, and treat LLM-as-judge noise like the flakiness it is.

FAQ

What is the difference between RAG testing and regular LLM testing?

Regular LLM testing evaluates the model’s output in isolation. RAG testing also evaluates the retrieval step that supplies the model’s context, because a RAG app can fail upstream of the answer. You test both what was retrieved and whether the answer is faithful to it.

Which metrics should a RAG QA team start with?

Start with four: context precision, context recall, faithfulness, and response relevancy. They are available in Ragas and give you one number each for the retrieval half and the generation half.

How many questions should a RAG golden set have?

At least 50 hand-reviewed questions for a small product, scaling into the hundreds with synthetic generation. More important than the count is the mix: hard multi-hop and negation queries catch what ten easy questions miss.

Can I test a RAG app without a vector database?

Yes. You can evaluate retrieval over any list of retrieved chunks, regardless of where they came from, by scoring context precision and recall directly against ground-truth chunks. The vector database is an implementation detail of the retriever.

Why does my RAG eval score change between runs?

Because the metrics use an LLM as judge, and LLMs are non-deterministic. Pin the judge model and temperature, widen your failure thresholds to absorb the noise band, and re-run before treating a small drop as a real regression.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.