LangGraph Agent Testing Strategy: 7 QA Checks

LangGraph agent testing strategy matters because QA teams are now asked to test something more slippery than a button click: an AI agent that plans, calls tools, remembers state, streams tokens, retries work, and may choose a different path on the second run. If you test only the final answer, you miss the real failure. I see teams ship green demos and then spend the next sprint explaining why the agent booked the wrong action, skipped a guardrail, or used stale memory.

LangGraph is not a toy project. The GitHub API for langchain-ai/langgraph showed 38,901 stars, 6,551 forks, and 673 open issues during this run. PyPI lists langgraph 1.2.10, and the GitHub release page shows version 1.2.10 published on 2026-07-28. On the TypeScript side, npm reported 11,976,331 last-month downloads for @langchain/langgraph. Adoption is no longer the question. The question is whether QA can build a test strategy that matches how agents fail.

Table of Contents

• Why normal tests break for LangGraph agents
• The 7-layer LangGraph agent testing model
• State and graph transition tests
• Tool contract and side-effect tests
• Memory, checkpoint, and replay tests
• Streaming and human-interrupt tests
• Evaluation tests with LangSmith-style datasets
• CI gate design for QA teams
• India hiring context for SDETs
• FAQ

Why normal tests break for LangGraph agents

A Selenium or Playwright test usually has a crisp contract. Click login. Enter OTP. Check dashboard. An agent graph has more moving parts. The node chooses a next step, the tool returns data, the model interprets that data, memory can change the next prompt, and the final answer may be valid even when the path was unsafe.

Final-answer testing hides path bugs

I do not trust a test that checks only response.includes("done"). An agent can return a polished answer after using the wrong tool, skipping a policy check, or writing into the wrong customer account in a test environment. The pass condition must include path evidence: which nodes ran, which tools were called, what state changed, and whether a guardrail blocked risky behavior.

Non-determinism is not an excuse

Teams often say, “LLMs are probabilistic, so we cannot test them like normal software.” That is half true and half dangerous. You cannot assert every token. You can assert contracts around graph state, tool schemas, allowed transitions, retries, latency budgets, and evaluator scores. That is where a practical LangGraph agent testing strategy starts.

The source docs point to the right primitives

The official LangGraph overview positions LangGraph for stateful, multi-actor agent applications. The persistence documentation covers checkpointers and durable execution. The streaming documentation covers stream modes that expose live graph progress. These are not only developer features. For QA, they are test hooks.

The 7-layer LangGraph agent testing model

I use a 7-layer model because a single “agent test” becomes too vague. Each layer catches a different class of failure. If you own QA for an agent workflow, ask for these layers in the test plan before the first production pilot.

The seven checks

1. Graph compile check: the graph builds with the expected nodes, edges, conditional branches, and finish points.
2. State transition check: each node reads and writes only the expected state keys.
3. Tool contract check: tools validate input, output, errors, and side effects.
4. Memory and checkpoint check: the same thread can pause, resume, replay, and recover.
5. Streaming check: users see progress without exposing private data or raw chain-of-thought style internals.
6. Evaluation check: datasets score answer quality, faithfulness, safety, and task completion.
7. CI release gate: failures block risky prompt, model, tool, and dependency changes.

What to automate first

Do not start with 200 golden answers. Start with 20 high-risk scenarios: refund request, account deletion, wrong document retrieval, empty tool response, timeout, duplicate tool call, and policy conflict. A small suite with sharp failure signals beats a large suite nobody trusts.

How this differs from normal API testing

API testing checks request and response contracts. Agent testing checks a decision loop. That means you need evidence for intermediate decisions, not only the HTTP 200. If your report cannot show node trace, state diff, tool payload, and evaluator reason, the bug will turn into a debate.

State and graph transition tests

LangGraph agents are built around state. For QA, state is your black box recorder. A good state test proves that every node changes only what it should change and that conditional routing does not send the user into a silent wrong branch.

State schema assertions

Make the state schema explicit. In TypeScript, I usually define a minimal state object for tests and assert state diffs after each important node. This is not about mocking the whole LLM. It is about making the graph contract visible.

type AgentState = {
  userInput: string;
  intent?: "search" | "refund" | "handoff";
  toolCalls: Array<{ name: string; args: unknown }>;
  answer?: string;
  riskFlags: string[];
};

function expectStateDiff(before: AgentState, after: AgentState) {
  expect(after.userInput).toBe(before.userInput);
  expect(after.toolCalls.length).toBeGreaterThanOrEqual(before.toolCalls.length);
  expect(after.riskFlags).toEqual(expect.any(Array));
}

Conditional edge tests

Conditional routing is where many agent bugs hide. Test the edge function like a normal pure function. Feed it 10 states and assert the next node. Use boring names: routes_refund_to_policy_check, routes_low_confidence_to_handoff, routes_empty_context_to_retrieval. Boring tests save production incidents.

Negative states matter

Most demo tests use happy state: valid input, valid tool, valid answer. QA should add malformed state: missing intent, repeated tool call, previous answer still present, invalid risk flag, and stale checkpoint. If the graph silently continues, file a bug. Silent continuation is usually worse than a visible failure.

Tool contract and side-effect tests

Agents become dangerous when tools are attached. A chat answer can be wrong. A tool call can change data. That is why tool testing deserves its own layer in a LangGraph agent testing strategy.

Test the tool before the agent

Every tool should have contract tests without the model. If the refund tool expects orderId, reason, and approvedByPolicy, assert all three. If a tool can write data, run it against a sandbox with deterministic fixtures.

import { z } from "zod";

const RefundToolInput = z.object({
  orderId: z.string().regex(/^ORD-[0-9]{6}$/),
  reason: z.string().min(10),
  approvedByPolicy: z.literal(true)
});

test("refund tool rejects unapproved requests", async () => {
  const badPayload = { orderId: "ORD-123456", reason: "changed mind", approvedByPolicy: false };
  expect(() => RefundToolInput.parse(badPayload)).toThrow();
});

Assert side effects, not logs

A log line saying “refund created” is not enough. Check the database row, the audit event, the idempotency key, and the external mock call. If the agent retries after a timeout, the tool should not create two refunds. Idempotency is a QA requirement, not only a backend detail.

Use tool-call snapshots carefully

Snapshots are useful for payload shape, but they become noisy when you snapshot full model text. Snapshot the tool name, required arguments, and safety flags. Avoid snapshotting generated prose unless the text is part of a regulated message.

Memory, checkpoint, and replay tests

The official LangGraph persistence docs make one thing clear: agent state can survive across steps. That is powerful, and it is also where subtle bugs appear. QA must test memory as a first-class surface.

Checkpoint recovery

Run a test that pauses after node 2, kills the worker, resumes with the same thread ID, and asserts that the graph continues from the correct checkpoint. If the agent repeats a charged action or loses a user approval, the checkpoint design is broken.

Replay for bug reports

When a user reports “the agent did the wrong thing,” QA needs replay evidence. Store input, model version, prompt version, tool version, state diff, and evaluator result. You do not need to store private data in raw form. You need enough structured evidence to reproduce the decision path.

Memory isolation tests

Test tenant isolation and user isolation directly. User A’s preference must not appear in User B’s answer. A Bengaluru customer asking for invoice help should not receive a previous Mumbai customer address because the memory key was too broad. I have seen similar bugs in non-agent systems; agents make them easier to miss because the final prose looks confident.

Streaming and human-interrupt tests

Streaming feels like a UI feature, but it is also a quality signal. LangGraph streaming can expose progress, updates, and messages while the graph runs. QA should test what the user sees during slow tool calls and what the operator sees during human review.

Progress without leaking internals

Users need useful progress: “Checking policy,” “Searching documents,” “Preparing answer.” They do not need hidden prompt text, raw retrieved chunks with secrets, or internal model reasoning. Add tests that inspect streamed events for banned fields and private values.

Human-in-the-loop interrupts

High-risk flows should stop for approval. Example: payment, deletion, legal answer, medical answer, or data export. The test should force the risky branch, assert that the graph pauses, submit a human decision, and then assert the resumed path. If the agent can bypass the approval node, treat it as a severity-1 bug.

Timeout and retry behavior

A slow retrieval tool should not freeze the full workflow forever. Test a 30-second timeout in a controlled fixture. Assert the fallback message, retry count, and final state. For Indian product teams running lean QA squads, this one test prevents many late-night production support calls.

Evaluation tests with LangSmith-style datasets

Unit tests catch contracts. Evaluations catch quality drift. The LangSmith evaluation concepts describe datasets, target functions, evaluators, and experiments. QA engineers should understand this vocabulary because it maps well to regression testing.

Build a small dataset first

Start with 50 examples, not 5,000. Include 20 happy paths, 15 edge cases, 10 adversarial cases, and 5 policy conflicts. For each example, store input, expected behavior, minimum score, and the reason. A dataset without reasons becomes a spreadsheet of opinions.

Use multiple evaluators

One evaluator is not enough. Use task completion, groundedness, refusal correctness, tool accuracy, and tone safety where relevant. For retrieval-heavy agents, pair this with the ideas in AI Test Failure Classification: 4 Buckets for QA. Classify every failure as prompt issue, retrieval issue, product bug, or dataset gap.

Connect to existing LLM regression practice

If your team already uses PromptFoo or DeepEval, do not throw that work away. Use the graph trace from LangGraph and score the final answer through your eval framework. ScrollTest has practical starting points in PromptFoo vs DeepEval: QA Guide for LLM Tests and LLM Regression Testing for QA: Day 32 Lab.

CI gate design for QA teams

A test strategy is not real until it blocks a bad release. For LangGraph agents, CI should run fast contract checks on every pull request and slower evaluation checks on prompt, tool, model, or dependency changes.

Pull request gate

Run graph compile checks, state transition tests, tool schema tests, and 10 critical evals on every pull request. Keep this under 10 minutes. If it takes 47 minutes, developers will skip it or move it out of the merge path.

name: agent-quality-gate
on: [pull_request]
jobs:
  langgraph-agent-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm test -- --runInBand tests/agent/contracts
      - run: npm test -- --runInBand tests/agent/critical-evals

Nightly gate

Run the 50-example dataset nightly. Track pass rate, evaluator scores, latency, tool-call count, and cost. Fail the build when task completion drops below your threshold or when a sensitive refusal test starts passing incorrectly.

Release-note risk checks

LangGraph 1.2.10 was published on 2026-07-28, and checkpoint packages also had late-July releases. Before upgrading, read the release notes and run the checkpoint, streaming, and tool retry suites. This connects with the habit I recommend in AI Eval Release Watch for QA Teams: convert release notes into risk tickets before production sees them.

India hiring context for SDETs

In India, AI testing skills are moving from “nice to have” to interview filter for senior SDET roles. I see the strongest signal around people who can test tools, agents, RAG, and CI gates, not people who only prompt a chatbot. For a mid-level QA engineer, this is the difference between being seen as a script maintainer and being seen as an automation architect.

What hiring managers ask

Expect questions like these:

• How do you test an agent that can take multiple valid paths?
• How do you prove a tool call was safe?
• How do you stop prompt drift from reaching production?
• How do you test memory isolation between tenants?
• How do you design a CI gate when LLM output is non-deterministic?

Salary signal

I will not claim a magic salary number for every company. Service companies, GCCs, and product companies price this skill differently. But for senior SDETs targeting ₹25-40 LPA product roles, a portfolio that shows LangGraph state tests, tool-contract tests, and eval gates is stronger than another generic Selenium framework clone.

Portfolio project idea

Build a support-ticket triage agent with 3 tools: search knowledge base, classify severity, and draft response. Add 50 eval cases and a CI gate. Then write a README with screenshots of graph traces and failures. This pairs well with the advice in The SDET Take-Home Assignment.

Test data and observability rules

LangGraph agent testing also needs disciplined test data. If every run uses a fresh random account, you cannot compare traces across builds. If every run uses the same shared account, one flaky run poisons the next one. I prefer named fixtures: customer_refund_approved_001, customer_policy_conflict_002, and customer_empty_knowledge_base_003. The name tells the engineer what risk the fixture exists to cover.

Keep prompts, tools, and data versioned

Version the prompt, graph, tool schema, and dataset together. A failing eval is useless if you cannot tell whether the prompt changed, the retriever changed, or the tool schema changed. Add four fields to every result: promptVersion, graphVersion, toolVersion, and datasetVersion. This is boring metadata, but it turns a flaky AI failure into a traceable software failure.

Red-team the obvious abuse cases

For a QA-owned agent, I want at least 10 abuse cases in the first suite. Ask the agent to ignore policy, reveal hidden notes, call a write tool without approval, mix two customers, retry a payment twice, and answer from memory when retrieval returns no documents. These cases do not need fancy prompts. They need clear expected behavior and a binary release decision.

Make failures reviewable by non-LLM experts

Your manager, product owner, and support lead should be able to read the failure report. Include the user task, expected action, actual node path, failed assertion, and the one screenshot or trace link that proves the issue. If the report requires an LLM researcher to decode it, the QA strategy is too academic for production.

Implementation checklist

If I had to set this up for a QA team this week, I would keep the plan simple and visible. Do not build a research lab. Build a release gate.

Week 1 plan

1. List the top 10 agent journeys by business risk.
2. Define the graph state schema and expected state diffs.
3. Write contract tests for every tool.
4. Create 20 evaluation examples with expected behavior and reasons.
5. Add trace capture to every CI run.
6. Block the pull request on compile, state, and tool failures.
7. Run the full eval set nightly and review failures every morning.

Failure report template

Every failed agent test should include: scenario ID, graph version, model name, prompt version, node path, state diff, tool calls, evaluator score, expected behavior, actual behavior, and owner. This is the difference between a bug report and a Slack argument.

Key takeaways

• A LangGraph agent testing strategy must test the path, not only the answer.
• State diffs, tool contracts, checkpoints, streaming events, and eval scores are QA artifacts.
• Start with 20 high-risk scenarios and grow to 50 eval examples before scaling further.
• CI should block unsafe graph, tool, prompt, and dependency changes.
• For SDETs in India, agent testing is a portfolio skill with real hiring signal.

FAQ

What is the best first test for a LangGraph agent?

Start with a graph compile test and a state transition test for the highest-risk journey. If the graph cannot prove which nodes ran and which state keys changed, final-answer tests will not give you enough confidence.

Should QA mock the LLM?

Mock the LLM for graph routing and tool-contract tests. Use the real model for a smaller evaluation suite. This split keeps PR checks fast while still catching model and prompt drift.

How many eval examples do I need?

For a first production gate, 50 examples is a practical target: 20 happy paths, 15 edge cases, 10 adversarial cases, and 5 policy conflicts. Add examples every time production or UAT finds a new failure mode.

Can Playwright help with LangGraph agent testing?

Yes, but use it at the UI boundary. Playwright can verify the user-facing agent experience, streaming messages, file uploads, and approval screens. Keep graph state, tool contracts, and eval scoring in lower-level tests.

What should block a release?

Block on unsafe tool calls, missing approval interrupts, tenant memory leaks, checkpoint replay failures, and critical eval regressions. Do not block on harmless wording variation unless the exact wording is a product or compliance requirement.