|

RAG Evaluation Bugs QA Engineers Must Catch

ByPromode
RAG evaluation bugs QA checks for retrieval grounding citations and CI gates

RAG evaluation bugs are not model problems first. They are usually QA problems hiding inside retrieval, context ranking, citation rules, and test data. I see teams ship a chatbot because ten demo questions passed, then discover that the same system fails when the user asks with a synonym, a stale policy, or one extra constraint.

This guide is a practical checklist for QA engineers who need to test Retrieval Augmented Generation systems without pretending that a single “answer quality” score is enough. I use Ragas, DeepEval, PromptFoo, and plain TypeScript checks because each tool catches a different class of bug.

Table of Contents

Contents

Why RAG Evaluation Bugs Are Different

Classic automation gives you a clean oracle

A login test has a clear assertion. The button appears, the token is set, and the dashboard loads. A RAG test is messier because the output can be worded differently and still be correct. That is why QA engineers need to split the system into observable parts instead of judging only the final answer.

A RAG pipeline usually has at least four moving parts: the retriever, the reranker, the prompt, and the answer generator. Each layer can pass locally and still fail together. The retriever may find a relevant policy, the prompt may hide it below noisy text, and the model may answer from prior knowledge.

One green answer can hide three red signals

The biggest trap is accepting a nice-looking response as proof. A user may ask, “Can I refund an annual plan after 35 days?” The bot may answer “No” and look correct. But if the source says “30 days for self-serve, 45 days for enterprise contracts,” the answer is incomplete.

That failure is not a spelling issue. It is a coverage issue. QA must ask whether the answer used the right source, respected the right business rule, and exposed uncertainty when the context was incomplete.

RAG evaluation is becoming a real QA skill

The tooling is not niche anymore. The Ragas GitHub repository shows more than 15,000 stars, and its PyPI page describes Ragas as an “Evaluation framework for RAG and LLM applications.” DeepEval has more than 17,000 GitHub stars, and PromptFoo has more than 23,000. These are not toy projects.

For QA engineers in India, this matters. Manual testers trying to move into automation are still fighting for ₹6-12 LPA roles. SDETs who can test AI features, build eval datasets, and wire checks into CI have a better shot at ₹25-40 LPA product-company roles. I do not say this as hype. I say it because product teams now need people who can prove AI features are safe to release.

Data Points QA Should Know Before Testing RAG

The ecosystem is moving fast

Before writing test cases, know the tools your developers may already use. Ragas 0.4.3 is listed on PyPI as the current version at the time of this run. DeepEval 4.1.4 is listed on PyPI as “The LLM Evaluation Framework.” LlamaIndex 0.14.23 is listed as an interface between LLMs and your data.

On the JavaScript side, the npm downloads API reports 1,875,637 last-month downloads for promptfoo for the period 2026-06-29 to 2026-07-28. That number tells me something simple: evals are moving into everyday pipelines, not staying inside research notebooks.

Primary docs agree on one point: evaluate components

The Ragas metrics documentation lists metrics around context, answer correctness, and faithfulness. DeepEval contextual relevancy focuses on whether retrieved context is relevant to the input. PromptFoo’s RAG guide shows how to evaluate retrieval and generation behavior in test suites.

The pattern is clear. Do not reduce RAG evaluation to one pass or fail label. QA should track retrieval quality, grounding, citation health, answer completeness, refusal behavior, latency, and regression drift.

Use ScrollTest’s existing AI testing path

If your team is new to this area, start with a foundation article like AI Quality Engineer Roadmap: PromptFoo + DeepEval. For tool selection, pair this article with PromptFoo vs DeepEval: QA Guide for LLM Tests. For evidence discipline, read AI Testing Evidence: Stop Trusting Green Checks.

Those internal guides cover the broader eval stack. This article stays focused on RAG evaluation bugs that appear after the demo works.

Bug 1: Retrieval Misses the Right Document

The symptom

The generated answer is vague, outdated, or overconfident because the retriever never fetched the right document. This is the most basic RAG failure, but it is also the one teams under-test. They check the final answer and never inspect the retrieved chunks.

Here is a real pattern I see in QA support bots. The user asks about “PTO carry forward.” The retriever returns documents about “leave policy” but misses the exact “annual leave balance transfer” page. The answer sounds helpful, but it does not answer the user’s real question.

What QA should assert

Create retrieval-level tests before answer-level tests. For every golden question, store the expected document ID, section title, or chunk tag. Your assertion should fail if the right source is missing from the top K results.

type RetrievedChunk = {
  id: string;
  title: string;
  score: number;
  text: string;
};

function expectSourceInTopK(
  chunks: RetrievedChunk[],
  expectedId: string,
  k = 5
) {
  const topIds = chunks.slice(0, k).map((chunk) => chunk.id);
  if (!topIds.includes(expectedId)) {
    throw new Error(
      `Expected ${expectedId} in top ${k}, got ${topIds.join(", ")}`
    );
  }
}

Test data that exposes it

Do not test only exact document titles. Use synonym questions, abbreviation questions, noisy questions, and user-language questions. For an HR bot, “PTO,” “earned leave,” “annual leave,” and “leave carry forward” may all point to the same rule. For a banking bot, “chargeback,” “dispute,” and “wrong debit” may point to the same workflow.

  • Ask with a synonym that does not appear in the source title.
  • Ask with one typo in a critical term.
  • Ask with a regional phrase used by Indian users.
  • Ask with a time condition such as “after 30 days.”
  • Ask with a role condition such as “for contractors.”

If top K retrieval fails on these cases, do not blame the LLM. Fix indexing, metadata, chunking, query rewriting, or reranking first.

Bug 2: Context Is Present but Unused

The symptom

The right context is retrieved, but the final answer ignores it. This is painful because engineers often say, “Retrieval worked, so the RAG pipeline is fine.” It is not fine. The answer generator can still prefer pretrained memory, earlier prompt examples, or a more common public rule.

Ragas calls this family of checks faithfulness and answer correctness in its metric set. The exact metric name matters less than the test idea: the answer must be supported by retrieved context.

What QA should assert

Split the test into two checks. First, assert that the required context appears in retrieved chunks. Second, assert that the answer includes the critical facts from that context. A semantic judge can help, but a deterministic assertion for business-critical numbers is safer.

type RagResponse = {
  answer: string;
  sources: { id: string; text: string }[];
};

function expectGroundedRefundAnswer(response: RagResponse) {
  const combinedSources = response.sources.map((s) => s.text).join("\n");

  if (!combinedSources.includes("45 days")) {
    throw new Error("Missing enterprise refund rule in retrieved context");
  }

  if (!/45\s+days/i.test(response.answer)) {
    throw new Error("Answer ignored the 45-day enterprise rule");
  }
}

Good answer, bad grounding

A generated answer can be factually correct but still fail a release gate if it used the wrong source. For regulated domains, “right answer from memory” is not acceptable. Healthcare, finance, insurance, and HR systems need source-grounded answers because policies change.

QA should mark these as grounding bugs. The defect title should name the source issue, not just “wrong answer.” A better title is: “RAG answer ignores retrieved enterprise refund policy when user asks after 35 days.”

Bug 3: The Answer Is Right for the Wrong Source

The symptom

The answer cites a source, but the source does not support the claim. This is worse than no citation because it gives users false confidence. I treat citation mismatch as a P1 bug for customer-facing RAG systems.

Example: the bot says, “Enterprise users can cancel within 45 days,” then cites a generic pricing FAQ that never mentions enterprise contracts. A human reviewer may trust the link and miss the gap unless QA checks citations directly.

Citation assertions

Every cited URL or document ID should be traceable to the exact claim. For critical answers, ask the system to return citation spans or source chunk IDs. If your product does not expose that data, request it as a testability requirement.

  1. Extract each factual claim from the answer.
  2. Map each claim to one or more cited chunks.
  3. Check that the cited chunks contain the required fact.
  4. Fail if a claim has no support.
  5. Fail if the citation points to a stale document version.

Versioned documents matter

RAG systems often keep old PDFs, old Confluence exports, or duplicate policy pages. If the answer cites the 2024 policy while the 2026 policy is live, the model may look grounded and still be wrong. Add document version checks to your golden dataset.

I prefer metadata assertions for this. Store fields such as doc_version, effective_date, owner, and region. Then test that retrieved chunks match the expected metadata for the user’s question.

Bug 4: Eval Datasets Are Too Clean

The symptom

The eval suite passes because every test question is written like a product manager wrote it. Real users do not ask that way. They paste screenshots, mix languages, abbreviate product names, and ask two questions in one line.

This bug hurts Indian QA teams because many internal tools serve users who mix English with local terms. Even if production responses stay in English, user inputs may include shorthand like “PF,” “salary slip,” “Form 16,” “UAN,” or “reimbursement.”

Dataset design for QA

A useful RAG eval dataset needs more than golden answers. It needs intent categories, source IDs, expected constraints, negative cases, and noisy variants. For every happy-path question, I create at least 3 variants.

  • Clean variant: “What is the enterprise refund period?”
  • Synonym variant: “How long can a business account reverse annual billing?”
  • Noisy variant: “paid annual plan 35 days back, can we cancel?”
  • Negative variant: “Can I get a refund after 90 days?”

A starter eval file

PromptFoo is useful when QA wants a versioned, reviewable test file. The exact provider and app endpoint will vary, but the structure below shows the idea.

description: RAG evaluation bugs for refund policy

prompts:
  - "{{question}}"

providers:
  - id: http
    config:
      url: https://your-rag-app.example.com/chat
      method: POST
      body:
        question: "{{question}}"

tests:
  - vars:
      question: "paid annual plan 35 days back, can we cancel?"
    metadata:
      expected_source: "refund_policy_enterprise_2026"
      bug_class: "retrieval_or_grounding"
    assert:
      - type: contains
        value: "45 days"
      - type: javascript
        value: |
          const sources = output.sources || [];
          return sources.some(s => s.id === "refund_policy_enterprise_2026");

Keep the dataset in git. Review it like code. A RAG eval file that nobody reviews becomes another stale test suite.

Bug 5: Thresholds Create False Confidence

The symptom

The dashboard says “average score 0.82,” so the team ships. The problem is hidden in the average. Ten easy questions can bury two serious failures. For QA, a single critical policy miss matters more than a pretty average.

LLM-as-judge scores are useful, but they are not magic. They depend on prompts, rubrics, model choice, and dataset quality. Treat them like automation signals, not truth.

Use gates by risk level

I prefer risk-based gates instead of one global threshold. A marketing FAQ can tolerate a softer answer. A refund policy, security control, medical instruction, or tax rule needs a stricter gate.

  • P0 gate: no unsupported claims, no wrong policy number, no stale source.
  • P1 gate: required source appears in top 3 retrieved chunks.
  • P2 gate: answer covers the expected intent and cites a valid source.
  • P3 gate: tone, formatting, and helpfulness checks pass.

Track failures by bucket

Do not report only “RAG score dropped.” Report the bug bucket. Retrieval miss, context ignored, citation mismatch, stale source, over-refusal, under-refusal, incomplete answer, and latency regression need different owners.

This is where QA leadership matters. If every failed eval becomes “AI is flaky,” nothing improves. If each failed eval maps to a subsystem, engineering can fix the right layer.

Bug 6: RAG Regressions Hide in Release Pipelines

The symptom

Your RAG bot works on Monday and fails on Thursday without a code change in the UI. Why? Someone re-indexed documents, changed chunk size, swapped an embedding model, updated a reranker, edited prompts, or added a new policy that competes with an older one.

That is why RAG evaluation belongs in CI. It should not be a manual demo before a launch call.

A CI pattern that works

Run a small smoke eval on every pull request and a deeper eval on every index refresh. Keep the PR gate fast. Keep the nightly gate broader. If you use Playwright for UI checks, connect eval failures to the same release report so QA sees one view of product risk.

name: rag-eval-smoke
on:
  pull_request:
  workflow_dispatch:

jobs:
  rag-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx promptfoo eval -c evals/rag-refund-policy.yaml
      - run: npx promptfoo export --output reports/rag-eval.json

Pair evals with release notes

When a dependency changes, read the release notes and update tests around the changed behavior. ScrollTest already covers this release-risk habit in AI Eval Release Watch for QA Teams. The same idea applies to RAG stacks. A version bump in Ragas, DeepEval, PromptFoo, LlamaIndex, LangChain, or your vector database should trigger focused regression checks.

QA Playbook for RAG Evaluation Bugs

Start with a 25-question seed suite

If you are building your first RAG evaluation suite, do not start with 500 questions. Start with 25 strong cases. Pick the top 5 user intents, then create 5 cases per intent: clean, synonym, noisy, negative, and stale-document conflict.

For each case, capture these fields:

  • question: the exact user input
  • expected_source_id: the source that must be retrieved
  • required_facts: numbers, dates, policy names, or constraints
  • forbidden_claims: claims the model must not make
  • risk_level: P0, P1, P2, or P3
  • owner: retrieval, prompt, content, product, or legal

Use three layers of checks

A strong QA setup uses deterministic checks, semantic checks, and human review. Deterministic checks catch exact numbers and source IDs. Semantic checks catch meaning and completeness. Human review catches policy nuance that a judge model may miss.

  1. Smoke gate: 25 cases, under 5 minutes, runs on PR.
  2. Regression gate: 100-300 cases, runs nightly or on index refresh.
  3. Human audit: 10 high-risk failures reviewed weekly.

Write defects that engineers can fix

A weak bug says, “RAG gave wrong answer.” A useful bug says, “Retriever returns 2024 refund policy above 2026 enterprise policy for annual-plan cancellation query.” Add the question, retrieved chunk IDs, answer, expected source, actual source, and risk level.

This makes the defect actionable. The search engineer can inspect embeddings. The content owner can remove duplicates. The backend engineer can change reranking. The product manager can clarify the expected rule.

Conclusion: Treat RAG Evaluation Bugs Like Product Risk

RAG evaluation bugs are release risks, not academic edge cases. If a chatbot gives wrong refund, HR, compliance, or onboarding guidance, users do not care that your average eval score was high. They care that the product misled them.

My recommendation is simple. Test retrieval first. Test grounding second. Test citations third. Then wire the smallest useful eval suite into CI and grow it from production failures.

  • Do not judge RAG quality only by the final answer.
  • Store expected source IDs and required facts in every golden case.
  • Use Ragas, DeepEval, or PromptFoo where they fit, but keep deterministic checks for critical facts.
  • Track failures by owner: retrieval, prompt, content, model, or product rule.
  • For SDETs, RAG evaluation is now a practical AI testing skill worth building.

FAQ

What is the most common RAG evaluation bug?

The most common bug is retrieval mismatch. The answer fails because the system never fetched the right source. QA should inspect top K chunks before judging the generated answer.

Should QA use Ragas, DeepEval, or PromptFoo?

Use the tool that matches your workflow. Ragas is strong for RAG metrics in Python workflows. DeepEval is useful for LLM evaluation metrics and test-style checks. PromptFoo is practical when QA wants YAML-based evals in CI, especially for prompt and RAG regression gates.

Can LLM-as-judge replace manual QA?

No. It can reduce review effort, but it cannot replace risk-based QA. Human review is still needed for legal policy, safety, compliance, and domain-specific nuance.

How many RAG eval cases should a team start with?

Start with 25 cases. Cover 5 user intents with clean, synonym, noisy, negative, and stale-source variants. Expand the dataset using production failures and support tickets.

What should SDETs learn first for RAG testing?

Learn retrieval inspection, source metadata checks, prompt regression testing, and CI gating. Then add tool-specific skills with Ragas, DeepEval, PromptFoo, LangSmith, or your team’s internal eval platform.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.