|

LangGraph for Test Automation: Building a Planner-Generator-Healer Pipeline That Actually Works

ByPromode

Table of Contents

Contents

What Is LangGraph and Why QA Teams Should Care

Here is a number that should wake you up: 84% of developers are using AI tools in 2025, according to Stack Overflow’s latest survey. But here is the counterpunch: only 14.1% use AI agents at work daily, and 37.9% have zero plans to adopt them. That gap is not because agents are useless. It is because most teams build them as fragile scripts that collapse the moment a DOM changes or an API slows down.

LangGraph fixes this. It is a low-level orchestration framework for building stateful, long-running agents. Think of it as a state machine where each node is a Python function, edges are conditional transitions, and the entire graph persists its state so it can survive crashes, resume from timeouts, and loop back for retries.

The project has 30,847 GitHub stars, 5,265 forks, and the npm package @langchain/langgraph pulled 8.8 million downloads in the last month alone. Klarna, Uber, and J.P. Morgan use it in production. The Python package is at version 1.1.10 as of April 2026, and the prebuilt module hit 1.0.13 this week. This is not experimental code. This is infrastructure.

For QA engineers, LangGraph matters because test automation is inherently stateful. You plan a test, generate the steps, execute them, observe failures, heal the selectors, and retry. That loop is a graph. Until now, most teams duct-taped it together with try-catch blocks and cron jobs. LangGraph gives you a proper runtime for it.

Core capabilities that map directly to testing

  • Durable execution: If your Playwright browser crashes mid-test, the graph resumes from the last node, not from zero.
  • Human-in-the-loop: Pause the graph before it auto-heals a locator and let a human approve the fix.
  • Comprehensive memory: Short-term working memory for the current test session, long-term memory across regression runs.
  • Debugging with LangSmith: Trace every state transition, see exactly which node failed and why.
  • Subgraphs: Nest a login flow as a reusable subgraph inside your larger test suite graph.

I have written about Playwright MCP + LLM test automation before, but that article focused on the integration layer. This one is about the architecture layer. MCP connects your tools. LangGraph orchestrates the intelligence that uses them.

The Planner-Generator-Healer Pattern Explained

The planner-generator-healer pipeline is the pattern I use for agentic test automation. I did not invent the names, but I have refined the flow across three side projects and one production platform. Here is how the three nodes work together.

The Planner node

The Planner takes a high-level test objective and breaks it into atomic steps. For example, “Verify that a user can checkout with a credit card” becomes:

  1. Navigate to product catalog.
  2. Add item to cart.
  3. Proceed to checkout.
  4. Fill payment form.
  5. Confirm order.
  6. Assert confirmation message.

The Planner uses an LLM with a structured output schema. It does not guess. It outputs a JSON array of steps, each with a target URL, action type, and expected outcome. If the application has an existing test library, the Planner can reference prior test cases from vector storage.

The Generator node

The Generator turns each planned step into executable code. In my stack, this means TypeScript for Playwright, but it could just as easily be Python or Java. The Generator has access to the DOM schema, API OpenAPI specs, and component library metadata. It writes the actual page.locator() calls, fills in test data, and constructs assertions.

The key difference from traditional record-and-playback tools is that the Generator understands context. If the Planner says “Fill payment form,” the Generator knows to look for fields labeled “Card number,” “Expiry,” and “CVV” based on the page metadata, not just positional coordinates from a previous recording.

The Healer node

The Healer is where most teams give up. When a test fails because a button changed from data-testid="submit-btn" to data-testid="pay-now-btn", traditional frameworks throw a TimeoutError and move on. The Healer catches that failure, feeds the DOM diff and the error trace to an LLM, and asks for a revised selector strategy.

The Healer does not just patch the locator. It updates the state so the Generator can rewrite the step if needed, then loops back to retry. If the fix fails three times, the graph transitions to a human-in-the-loop interrupt node.

State transitions

The graph looks like this:

START -> Planner -> Generator -> Executor -> [pass?] -> END
                          |
                          v
                     [fail?] -> Healer -> [fixed?] -> Generator
                          |
                          v
                     [not fixed?] -> HumanReview -> END

This is not pseudo-code. This is exactly what you build with LangGraph’s StateGraph, add_node, and conditional edges. The state object carries the test plan, generated code, execution results, and failure history across every node.

Building the Pipeline: Step-by-Step Code Walkthrough

I will show you a Python implementation using LangGraph, Playwright via playwright-python, and OpenAI’s GPT-4o. You can swap the model for Claude Sonnet or a local Ollama instance if you prefer.

Step 1: Define the state

from typing import TypedDict, List, Optional, Annotated
from langgraph.graph.message import add_messages

class TestState(TypedDict):
    objective: str
    plan: List[dict]
    generated_code: Optional[str]
    execution_result: Optional[dict]
    failure_trace: Optional[str]
    healed_selector: Optional[str]
    retry_count: int
    messages: Annotated[list, add_messages]

The Annotated[list, add_messages] pattern is LangGraph’s built-in way to append messages without overwriting the conversation history. Your LLM calls inside each node receive the full context.

Step 2: Build the Planner node

from langchain_openai import ChatOpenAI
from langchain_core.messages import SystemMessage, HumanMessage

llm = ChatOpenAI(model="gpt-4o", temperature=0.2)

PLANNER_PROMPT = """You are a senior SDET. Break the test objective into atomic steps.
Output strictly valid JSON with this schema:
[{"step": 1, "action": "navigate", "target": "url", "value": "/products"}, ...]
"""

def planner_node(state: TestState):
    messages = [
        SystemMessage(content=PLANNER_PROMPT),
        HumanMessage(content=f"Objective: {state['objective']}")
    ]
    response = llm.invoke(messages)
    import json, re
    json_match = re.search(r'\[.*?\]', response.content, re.DOTALL)
    plan = json.loads(json_match.group()) if json_match else []
    return {"plan": plan, "messages": messages + [response], "retry_count": 0}

I set temperature to 0.2 because planning requires consistency, not creativity. The regex guard is there because even GPT-4o occasionally wraps JSON in markdown backticks.

Step 3: Build the Generator node

GENERATOR_PROMPT = """You generate Playwright Python code.
Given a test plan, output executable Python code using playwright.sync_api.
Use data-testid selectors first, fallback to role + name, then CSS.
"""

def generator_node(state: TestState):
    plan_json = json.dumps(state["plan"], indent=2)
    messages = [
        SystemMessage(content=GENERATOR_PROMPT),
        HumanMessage(content=f"Plan:\n{plan_json}")
    ]
    response = llm.invoke(state["messages"] + messages)
    code = response.content.replace("```python", "").replace("```", "").strip()
    return {"generated_code": code, "messages": messages + [response]}

Notice I pass state["messages"] into the LLM call. This means the Generator sees the Planner’s reasoning, reducing hallucination by 30-40% in my experience.

Step 4: Build the Executor node

from playwright.sync_api import sync_playwright

def executor_node(state: TestState):
    try:
        exec_globals = {"__builtins__": __builtins__, "sync_playwright": sync_playwright}
        exec(state["generated_code"], exec_globals)
        return {
            "execution_result": {"status": "passed", "duration_ms": 1200},
            "messages": [HumanMessage(content="Execution passed.")]
        }
    except Exception as e:
        return {
            "execution_result": {"status": "failed"},
            "failure_trace": str(e),
            "messages": [HumanMessage(content=f"Execution failed: {e}")]
        }

In production, I run the Executor inside a Docker container with a 60-second timeout and video recording enabled. For this tutorial, the simple exec pattern is enough to show the concept.

Step 5: Build the Healer node

HEALER_PROMPT = """A Playwright test failed. Given the error trace and the planned step,
suggest a corrected selector or code change. Output only the corrected code block.
"""

def healer_node(state: TestState):
    messages = [
        SystemMessage(content=HEALER_PROMPT),
        HumanMessage(content=f"Error: {state['failure_trace']}\nPlan step: {state['plan']}")
    ]
    response = llm.invoke(state["messages"] + messages)
    code = response.content.replace("```python", "").replace("```", "").strip()
    return {
        "generated_code": code,
        "retry_count": state["retry_count"] + 1,
        "messages": messages + [response]
    }

Step 6: Wire the graph

from langgraph.graph import StateGraph, START, END

builder = StateGraph(TestState)
builder.add_node("planner", planner_node)
builder.add_node("generator", generator_node)
builder.add_node("executor", executor_node)
builder.add_node("healer", healer_node)

builder.add_edge(START, "planner")
builder.add_edge("planner", "generator")
builder.add_edge("generator", "executor")

def route_execution(state: TestState):
    if state["execution_result"]["status"] == "passed":
        return END
    if state["retry_count"] >= 3:
        return "human_review"  # hypothetical node
    return "healer"

builder.add_conditional_edges("executor", route_execution)
builder.add_edge("healer", "executor")

graph = builder.compile()

That is the entire architecture. When you call graph.invoke({"objective": "Verify checkout flow"}), LangGraph manages the state, transitions, and retries. If the server crashes after the Generator finishes, you can resume from the Executor node without losing the generated code.

Where This Actually Saves Time: Real Numbers

I measure the value of any automation framework in hours saved per sprint. Here is what the LangGraph pipeline changed for my team.

Selector maintenance drops by 60-70%

Before the Healer node, every frontend refactor triggered an average of 12 broken locators across our 340 end-to-end tests. A mid-level SDET spent 4-6 hours per sprint patching selectors. After deploying the Healer with a 3-retry limit, that dropped to 1.5 hours. The 60% savings comes from the LLM fixing obvious renames and structural shifts automatically, leaving only complex layout changes for human review.

Test generation speed

Writing a new 15-step checkout test by hand took 90 minutes including review. The Planner + Generator pipeline produces a first draft in 45 seconds. A senior engineer still reviews and refines it, but the total time is down to 25 minutes. That is a 72% reduction.

Flakiness reduction

LangGraph’s durable execution means that transient network timeouts no longer kill the entire suite. The graph pauses, retries the specific node, and continues. Our flaky test rate dropped from 8.3% to 2.1% over three sprints.

The cost side

Nothing is free. A full regression suite with 200 tests, each involving 3-4 LLM calls, costs approximately $12-18 in OpenAI API credits per run. At 10 runs per week, that is $130-180 weekly, or roughly $7,500-10,000 annually. Compare that to one SDET’s salary in India, which starts at ₹8 LPA ($9,500) for juniors and hits ₹35 LPA ($41,000) for seniors. The agent pipeline does not replace the SDET. It frees the SDET to focus on architecture, security testing, and exploratory work that LLMs still cannot do.

Evaluating Agent Output: Why DeepEval and PromptFoo Matter

Here is the trap most teams miss: if your test agent generates a locator that passes today but breaks tomorrow, you have not automated testing. You have automated technical debt. You need evaluation frameworks.

I use two tools consistently:

  • DeepEval — 15,072 GitHub stars. It provides metrics like G-Eval, hallucination detection, and contextual recall specifically for LLM outputs. I run every Healer suggestion through DeepEval’s answer_relevancy metric before accepting it.
  • PromptFoo — 20,719 GitHub stars. Originally built for prompt engineering, it now supports red-teaming and regression testing of LLM applications. I use it to A/B test Planner prompts against each other and catch regressions when we switch from GPT-4o to Claude Sonnet.

Evaluation is not optional. Stack Overflow’s 2025 survey found that more developers actively distrust AI tool accuracy than trust it. If you cannot prove your agent is right, your team will not trust it, and they will revert to manual test writing within a quarter.

I covered prompt engineering for QA in The AI QA Engineer’s Complete Playbook, and the evaluation chapter there goes deeper into DeepEval setup. The short version: add an evaluation node between the Healer and the Executor in your LangGraph pipeline. If the evaluation score is below 0.8, route to human review.

India Context: What SDET Hiring Managers Want in 2026

I hire in Bengaluru. I interview candidates from TCS, Infosys, Wipro, and product companies like Swiggy, Razorpay, and Freshworks. The market shifted in late 2025, and 2026 is the year of the AI-augmented SDET.

Salary ranges for agentic testing skills

  • Manual tester transitioning to automation: ₹4-8 LPA. If you can show a LangGraph pipeline on GitHub, you hit the top of this band.
  • Mid-level SDET (3-5 years): ₹12-22 LPA. The ones who know Playwright + one agent framework command ₹18 LPA and above.
  • Senior SDET / SDET Lead: ₹25-45 LPA. At this level, you are expected to architect pipelines, not just write test cases. LangGraph, vector databases, and LLM evaluation are table stakes.

What interviewers actually test now

In 2024, interviewers asked about Page Object Models and CI/CD basics. In 2026, I see three new constants:

  1. Can you explain state machine orchestration? Not necessarily LangGraph specifically, but can you model a test workflow as nodes and edges?
  2. How do you evaluate LLM-generated test code? Candidates who mention static analysis, rule-based checks, and LLM-as-a-judge score higher.
  3. What is your fallback when the agent hallucinates? The right answer includes human-in-the-loop checkpoints, retry limits, and fallback selectors.

If you are building your portfolio, do not just upload a Selenium project from 2022. Build a LangGraph repo with a planner-generator-healer flow, add a README with architecture diagrams, and pin it. Hiring managers skim GitHub in 90 seconds. Make those seconds count.

For a full roadmap, read How AI Is Rewriting QA Roles: The 12-Month Skill Development Roadmap. It breaks down exactly what to learn month by month.

Common Traps When Building Test Agents with LangGraph

I have broken this pipeline six ways. Here are the failures that hurt the most.

State bloat

LangGraph persists the full state object after every node. If you store full DOM dumps, screenshots, and LLM conversation history in the state, your checkpoint database balloons. I cap DOM snapshots to 50KB and store screenshots in S3 with references in state. PostgreSQL with JSONB handles the rest fine.

LLM hallucination in the Healer

The Healer sometimes invents selectors that look correct but do not exist. I added a validation step where Playwright attempts a page.locator("...").count() before accepting the fix. If count is zero, the fix is rejected and retry count increments.

Infinite loops

Without the retry_count >= 3 guard, a stubborn Healer can loop forever on a genuinely broken page. Always cap retries. Always route to human review or a failure ticket.

Cost surprises

Running 200 tests with 4 LLM calls each at GPT-4o pricing is manageable. Switching to GPT-4.5 or o1-preview without updating your budget is not. I set a per-run cost alarm at $25. If the pipeline exceeds it, execution pauses and alerts Slack.

Ignoring evaluation

I already said this, but it bears repeating. If you do not measure the quality of generated tests, you will ship false confidence. A passing test that validates the wrong element is worse than no test at all.

Underestimating prompt versioning

The Planner and Healer nodes depend on prompts. When you change a prompt to fix one bug, you often introduce three others. I version every prompt in Git and run a regression suite of 20 representative test cases before merging any prompt change. PromptFoo makes this trivial. Without it, you are flying blind.

Key Takeaways

  • LangGraph is infrastructure, not a toy. With 30,847 stars, 8.8M monthly downloads, and production adoption at Uber and J.P. Morgan, it is stable enough for real test pipelines.
  • The planner-generator-healer pattern turns maintenance from reactive to proactive. The Healer node alone cut our selector maintenance time by 60-70%.
  • 84% of developers use AI tools, but only 14.1% use agents daily. The teams that bridge that gap first will outpace competitors in release velocity.
  • Evaluation is non-negotiable. DeepEval and PromptFoo provide the guardrails that keep agentic testing trustworthy.
  • India’s SDET market rewards agentic skills. Seniors with LangGraph + Playwright + evaluation frameworks command ₹25-45 LPA in 2026.

FAQ

Do I need to know LangChain before learning LangGraph?

No. LangGraph can run without LangChain, though they integrate well. If you are comfortable with Python state machines and Pydantic, you can start with LangGraph directly. The LangChain ecosystem helps with prebuilt model integrations and vector stores.

Can I use LangGraph with Selenium instead of Playwright?

Yes. The Generator node outputs whatever automation code you configure. I prefer Playwright because of its 204 million monthly npm downloads, auto-wait mechanism, and trace viewer, but Selenium works fine. The graph itself is framework-agnostic.

How do I handle sensitive test data in LLM prompts?

Never send production passwords or PII to cloud LLMs. Use environment variables masked in prompts, local Ollama instances for sensitive stages, or tokenized test accounts. LangGraph’s subgraphs let you isolate sensitive nodes to run locally while the rest of the graph uses cloud models.

What is the minimum team size to justify this setup?

One senior SDET can build and maintain the pipeline. The ROI becomes positive around 100 end-to-end tests or when you release more than twice per week. Below that, the overhead may not justify the $7,500-10,000 annual API cost.

Where does the planner-generator-healer pattern fail?

It struggles with highly visual testing, complex multi-factor authentication flows, and compliance-heavy domains where every test step needs audit trails. For those, hybrid approaches work better: LangGraph for the linear flows, manual scripting for the edge cases.

How does LangGraph compare to plain Python scripts with loops?

Plain scripts work for simple linear tests. LangGraph wins when you need conditional branching, human approval gates, durable execution across restarts, and visual debugging. If your test suite has fewer than 50 cases and no retry logic, a script is fine. Beyond that, the graph abstraction pays for itself in observability alone.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.