Testing RAG Applications: QA Engineer’s Guide to Retrieval-Augmented Generation

RAG combines search with LLMs. Testing requires validating both retrieval AND generation. Traditional test approaches fail here.

🤖 Learning AI-powered testing? Go hands-on with LLM, RAG, and AI-agent testing in the AI-Powered Testing Mastery course at The Testing Academy.

RAG Architecture for Testers

User Query → Embedding → Vector DB → Context Assembly → LLM → Response

4-Layer Test Strategy

Layer	What	Metric
Retrieval	Right docs found?	Precision@k, Recall@k
Context	Context relevant?	Context relevance score
Generation	Answer grounded?	Faithfulness score
End-to-End	User gets useful answer?	Human eval + metrics

Retrieval Testing

class RetrievalTester:
    def test_precision(self, query, expected_docs, k=5):
        retrieved = self.db.search(query, top_k=k)
        relevant = [d for d in retrieved if d.id in expected_docs]
        return len(relevant) / k

    def test_no_leak(self, query, forbidden_docs):
        retrieved = self.db.search(query, top_k=10)
        leaked = [d for d in retrieved if d.id in forbidden_docs]
        return len(leaked) == 0

Faithfulness Testing

def test_faithfulness(answer, context):
    # Does answer ONLY contain info from context?
    prompt = f"Identify claims in answer not supported by context.\nContext: {context}\nAnswer: {answer}"
    result = call_llm(prompt)
    return result["faithfulness_score"] > 0.95

Common RAG Failures

• Wrong doc retrieved: “Python testing” returns docs about snakes
• Stale context: Vector DB has outdated data
• Context overflow: Too many chunks, LLM ignores critical ones
• Hallucinated citations: Cites docs that do not exist
• Cross-contamination: Leaks one user’s data into another’s response

End-to-End RAG Test

@pytest.mark.parametrize("tc", load_rag_tests())
def test_rag_e2e(tc, pipeline):
    result = pipeline.query(tc.question)
    assert result.retrieval_precision >= 0.6
    assert result.faithfulness >= 0.9
    assert result.answer_relevance >= 0.8
    assert len(result.hallucinations) == 0
    assert result.latency_ms < 5000