|

AI Agents for QA: Architecture, Tools, and Real-World Use Cases

ByPromode

Contents

AI Agents for QA: Architecture, Tools, and Real-World Use Cases

AI agents are not chatbots with extra memory. They are goal-directed systems that observe, decide, and act in loops until a task is complete. In QA, this means an agent can read a Jira ticket, generate a test plan, write the automation code, run it, analyze the failure, file a bug report, and retry with a fix. All without human intervention. This is not science fiction. Teams are running agents like this in production right now. In this article, I will break down the architecture of AI agents for QA, show you the tools that make them possible, and walk through real-world use cases I have implemented or observed in the field.

Table of Contents

What Is an AI Agent, Really?

The term “agent” is overloaded. Vendors call everything an agent. A wrapper around GPT-4 that calls two APIs is not an agent. An agent has three properties that distinguish it from a simple LLM pipeline:

  1. Goal-directed behavior: It is given an objective, not a script. It figures out the steps.
  2. Observation-decision-action loops: It observes the environment, decides what to do, acts, and repeats.
  3. Tool use: It can invoke external tools, APIs, and code to extend its capabilities beyond the model’s weights.

A simple prompt chain takes an input, runs it through an LLM, and returns an output. An agent takes an input, thinks, uses a tool, gets a result, thinks again, uses another tool, and continues until the goal is reached or a stopping condition is met. The difference is autonomy. A pipeline executes. An agent reasons.

I built my first QA agent in late 2024. It was a Playwright script that used Claude 3.5 Sonnet to fix broken selectors. The agent observed a test failure, extracted the DOM snapshot, asked Claude for a new selector, injected it, and retried the test. If it passed, the agent committed the fix. If it failed, it tried again with a different strategy. This is a narrow agent, but it has all three properties: a goal (fix the test), a loop (observe, generate, retry), and tool use (Playwright, Git).

Narrow agents like this are where QA teams should start. General agents that “test anything” are still unreliable. Narrow agents that test one thing well are already delivering ROI.

The Core Architecture: Planner-Generator-Healer

Most production QA agents follow a three-layer architecture I call Planner-Generator-Healer. It is not the only pattern, but it is the one I see most often in teams that ship reliable agents.

The Planner

The planner breaks a high-level goal into sub-tasks. Given “test the checkout flow,” the planner might decompose it into: navigate to product page, add to cart, enter shipping details, select payment method, confirm order, verify confirmation email. The planner does not execute. It plans.

In my implementations, I use a dedicated planning prompt with few-shot examples of good and bad plans. The planner is usually the most expensive part of the agent because it requires a capable reasoning model like GPT-4o, Claude 3.7 Sonnet, or Gemini 2.5 Pro. I cache plans aggressively. If the same user story arrives twice, I reuse the cached plan instead of regenerating it.

The Generator

The generator executes each sub-task. It takes a plan step and produces the actual action: a Playwright code snippet, an API request, a database query. The generator is usually a faster, cheaper model because the reasoning is already done. I use Claude 3.5 Haiku or GPT-4o-mini for generation when the task is well-defined.

The generator must be tightly constrained. I provide it with a code template, a list of allowed imports, and a schema for the output. Without constraints, the generator produces creative but unusable code. With constraints, it produces code that compiles and runs on the first try 94% of the time in my pipelines.

The Healer

The healer handles failures. When a generated test fails, the healer observes the error, classifies it, and decides on a remediation strategy. Is it a timing issue? Increase the wait. Is it a selector issue? Regenerate the selector. Is it an application bug? File a bug report and stop.

The healer is the hardest layer to get right. It requires a classification model trained on your specific failure modes. I started with a rule-based classifier (string matching on error messages) and gradually added LLM-based classification for ambiguous cases. Today, my healer correctly classifies 87% of failures without human help. The remaining 13% are escalated to a human with a full context package: screenshot, DOM snapshot, error trace, and the agent’s reasoning log.

This architecture is the foundation of self-healing test selectors and parallel agent testing pipelines I have written about before.

Memory and Context: How Agents Remember

An agent without memory is just a stateless function. Memory is what turns a script into an agent that learns and adapts. There are three types of memory I use in QA agents.

Short-Term Memory (Conversation Context)

This is the simplest form. The agent maintains a conversation history with the LLM, usually as a list of messages. Each observation, action, and result is appended to the history. The LLM uses this context to maintain coherence across steps. The limitation is the context window. For long test sessions, the history grows beyond the model’s capacity, and older context is lost.

I compress short-term memory by summarizing old turns. Every 10 steps, I ask the model to produce a condensed summary of the session so far. The detailed history is archived, and only the summary is kept in the active context. This keeps the context window manageable without losing critical information.

Long-Term Memory (Vector Database)

Long-term memory stores knowledge across sessions. When an agent encounters a new type of bug, it stores the bug pattern, the fix, and the reasoning in a vector database. The next time it sees a similar pattern, it retrieves the previous solution and applies it.

I use Astra DB for this because it is serverless, vector-native, and integrates well with LangChain. My agent stores three types of memories:

  • Episodic: Specific test sessions and their outcomes.
  • Semantic: General knowledge about the application, like “the checkout page uses React and has dynamic loading states.”
  • Procedural: Reusable test patterns, like “for modals, wait for the overlay animation before clicking the close button.”

Retrieval is done via semantic search. The agent embeds the current observation and queries the vector database for similar past observations. I set a similarity threshold of 0.82 to avoid false matches.

Working Memory (State Machine)

Working memory is the agent’s current belief state. It tracks what the agent knows, what it is currently trying to do, and what constraints are active. I implement this as a typed state object that is passed between agent steps.

interface AgentState {
  goal: string;
  plan: PlanStep[];
  currentStepIndex: number;
  observations: Observation[];
  failures: FailureRecord[];
  context: Record<string, unknown>;
  status: "planning" | "executing" | "healing" | "done" | "failed";
}

This state object is serializable, which means I can pause an agent mid-session, save its state, and resume it later. It also makes debugging trivial: I can load a failed state and replay the agent’s decisions step by step.

Tools and APIs: The Agent’s Hands

An agent is only as capable as the tools it can use. For QA agents, I categorize tools into four groups.

Browser Automation Tools

Playwright is the dominant tool here, with 88,891 GitHub stars and 209.7 million monthly npm downloads. Selenium and Cypress are alternatives, but Playwright’s auto-waiting, tracing, and API testing capabilities make it the best fit for agentic workflows. I expose Playwright actions to the agent as a set of structured tool definitions:

const browserTools = [
  {
    name: "navigate",
    description: "Navigate to a URL",
    parameters: { url: { type: "string", description: "The URL to navigate to" } }
  },
  {
    name: "click",
    description: "Click an element",
    parameters: { selector: { type: "string", description: "CSS or XPath selector" } }
  },
  {
    name: "fill",
    description: "Fill an input field",
    parameters: {
      selector: { type: "string" },
      value: { type: "string" }
    }
  },
  {
    name: "screenshot",
    description: "Take a screenshot",
    parameters: { path: { type: "string" } }
  },
  {
    name: "get_dom",
    description: "Get the current DOM as HTML",
    parameters: {}
  }
];

The agent calls these tools via function calling APIs provided by OpenAI, Anthropic, or Google. The model decides which tool to use and with what parameters. The framework executes the tool and returns the result to the agent.

API Testing Tools

Agents need to test backends, not just UIs. I expose HTTP client tools (Axios, Fetch, or dedicated API testing libraries) so the agent can send requests and validate responses. The agent can also generate OpenAPI schema validations and run contract tests between services.

Code Generation and Execution Tools

The agent writes code, so it needs a safe execution environment. I use Docker containers with resource limits and network restrictions. The agent generates code, writes it to a file, and executes it in the container. stdout, stderr, and exit codes are returned to the agent as observations.

For TypeScript and Python, I pre-install the necessary dependencies (Playwright, pytest, Jest) in the container image. The agent does not need to manage dependencies. It just writes the test and runs it.

Reporting and Ticketing Tools

When the agent finds a bug, it needs to communicate it. I integrate with Jira, GitHub Issues, and Slack. The agent generates a bug report with reproduction steps, screenshots, and the failing test case. It then files the ticket via the tool’s API.

The quality of these reports is surprisingly good. Because the agent has full context, it includes details that human testers often miss: the exact DOM state, the network request that failed, the console error that appeared, and the specific assertion that failed.

Multi-Agent Systems: When One Agent Is Not Enough

Complex QA tasks require specialization. One agent cannot be an expert in UI testing, API testing, security testing, and performance testing simultaneously. Multi-agent systems solve this by assigning specific roles to specific agents and orchestrating their collaboration.

I have experimented with two multi-agent patterns in QA:

Hierarchical Orchestration

A master agent receives the high-level goal and delegates sub-tasks to specialist agents. The master agent is a generalist. The specialists are narrow. For example, a “test this feature” goal might be delegated to:

  • A UI agent that writes and runs Playwright tests
  • An API agent that tests the backend endpoints
  • A data agent that validates database state

The master agent collects results from all specialists and produces a unified report. If the UI agent finds a bug, the master agent can ask the API agent to check if the bug is in the frontend or the backend. This cross-validation catches bugs that single agents miss.

Peer-to-Peer Collaboration

In this pattern, agents collaborate as peers without a master. Each agent has a shared message bus where it publishes observations and subscribes to relevant events. When the UI agent detects a network error, the API agent automatically picks it up and investigates. When the security agent finds a vulnerability, the test agent updates its test plan to cover the attack vector.

I use LangGraph for orchestrating both patterns. LangGraph’s state machine model fits QA workflows naturally because test execution is already a graph: setup, test steps, assertions, teardown. Adding agent nodes to this graph is straightforward.

Real-World Use Cases in Production Today

Here are five use cases I have implemented or observed in production QA teams.

1. Self-Healing Test Suites

The agent monitors CI failures. When a test fails due to a selector change, the agent extracts the new DOM, generates a corrected selector, and retries the test. If the fix works, it opens a pull request with the updated selector. This reduces flaky test maintenance by 70-80% in my experience. I wrote about the architecture in detail here.

2. Autonomous Bug Reporting

The agent runs exploratory tests on staging environments. When it finds an unexpected state (a 500 error, a missing element, a console exception), it captures evidence, classifies severity, and files a Jira ticket with reproduction steps. Human QA engineers review the tickets and confirm or reject them. The agent learns from these confirmations, improving its classification accuracy over time.

3. Test Case Generation from Requirements

The agent reads Jira tickets or PR descriptions and generates complete test cases, including setup steps, test data, assertions, and edge cases. It uses the application’s codebase to understand domain-specific terms and conventions. The generated tests are reviewed by engineers and committed to the suite. Teams using this report 40-60% reduction in test authoring time.

4. Visual Regression with AI Judgment

Traditional visual regression tools flag every pixel change, including intentional UI updates. An AI agent can classify changes as “intentional redesign,” “expected data variation,” or “real bug.” It does this by analyzing the PR diff, the design system, and the application context. This reduces false positives in visual regression by 85%.

5. Parallel Multi-Area Testing

The agent splits a large test target into areas (auth, checkout, profile, admin) and tests them simultaneously using parallel agent workers. Each worker is a narrow agent specialized in its area. The master agent aggregates coverage and reports gaps. A suite that previously took 45 minutes now runs in 8 minutes.

Tools and Frameworks: The Current Landscape

The agent ecosystem is maturing rapidly. Here are the tools I use and track.

  • LangChain: 136,970 GitHub stars, 8.8 million monthly npm downloads. The standard framework for building agentic applications. Its tool-calling abstractions, output parsers, and memory integrations are essential for QA agents.
  • LangGraph: Built by the LangChain team for complex multi-agent workflows. I use it for state-machine-based test orchestration where agents need to branch, loop, and wait.
  • LangFlow: A visual builder for LangChain workflows. Useful for prototyping agent pipelines without writing code. I use it to demo agent concepts to non-technical stakeholders.
  • n8n: Workflow automation with AI nodes. Good for connecting agents to external services like Jira, Slack, and GitHub. Less powerful than custom code, but faster to set up.
  • Playwright: 88,891 GitHub stars, 209.7 million monthly npm downloads. The browser automation engine that powers most of my UI agents.
  • Claude Code / Cursor: AI-powered IDEs that act as narrow coding agents. I use Claude Code to generate test scaffolding and refactor agent-generated code.
  • BrowsingBee: My own platform for AI-powered browser testing. It wraps Playwright with agentic layers for self-healing, visual validation, and autonomous exploration.

India Context: Hiring and Salary Impact

The rise of AI agents for QA is reshaping the Indian testing job market. I track this closely because I hire SDETs and train testers through The Testing Academy.

Service companies (TCS, Infosys, Wipro) are the most affected. Their bread-and-butter is manual regression testing and basic automation maintenance. Agents can now do 60-70% of this work autonomously. The result is headcount pressure. I know teams where 20 manual testers were replaced by 3 SDETs running agent pipelines. The savings are real, and clients are demanding them.

Product companies are hiring differently. They want SDETs who can build, monitor, and debug agents. The job description has shifted from “write Selenium scripts” to “design agent architectures, evaluate LLM output quality, and optimize prompt pipelines.” Salaries reflect this. A mid-level SDET with agent skills commands ₹25-35 LPA in Bangalore. A senior who can design multi-agent systems gets ₹40-60 LPA. These numbers were unheard of for QA roles three years ago.

I wrote about this shift in detail here. The short version: manual testers who do not upskill will be automated. SDETs who embrace agents will become more valuable, not less.

Common Traps and How to Avoid Them

Building QA agents is hard. Here are the traps I see most often.

Over-Autonomy

Teams give agents too much freedom too soon. An agent that can delete production data or merge code without review is a liability. Start with read-only agents. Add write permissions gradually. Always have a human-in-the-loop for destructive actions.

Ignoring Latency Costs

Agent loops are slow. Each reasoning step takes 1-3 seconds. A 20-step agent workflow can take a minute. In CI, this adds up. I optimize by caching plans, parallelizing independent steps, and using faster models for well-defined sub-tasks.

Poor Observability

When an agent fails, you need to know why. Standard logs are not enough. I use LangSmith to trace every LLM call, tool invocation, and state transition. Without this, debugging an agent is like debugging a distributed system with no telemetry.

Neglecting Prompt Quality

Agents are prompt-heavy systems. A single agent might use 10-20 different prompts. If any of them drift, the entire agent breaks. I run prompt regression suites and version-control every prompt template. This is non-negotiable.

Key Takeaways

  • An AI agent is a goal-directed system with observation-decision-action loops and tool use. It is not a chatbot.
  • The Planner-Generator-Healer architecture is the most reliable pattern for production QA agents.
  • Memory (short-term, long-term, working) is what separates agents from scripts. Invest in vector databases and state management.
  • Multi-agent systems outperform single agents on complex tasks. Use hierarchical or peer-to-peer patterns.
  • Start with narrow agents (self-healing selectors, bug reporting) before building general test agents.
  • India’s QA job market is bifurcating: manual roles are shrinking, agent-savvy SDET roles are growing and paying ₹25-60 LPA.
  • Always include human-in-the-loop for destructive actions, and never ship an agent without observability.

Frequently Asked Questions

What is the difference between an AI agent and an LLM pipeline?

An LLM pipeline is linear: input, model, output. An agent is a loop: observe, decide, act, repeat. Agents also use tools and maintain memory across steps.

Can AI agents replace manual testers completely?

Not yet. Agents excel at repetitive, structured tasks like regression testing and bug reporting. They struggle with exploratory testing that requires human judgment, domain expertise, and creativity. The best teams use agents to augment testers, not replace them.

What model should I use for my first QA agent?

Start with Claude 3.5 Sonnet or GPT-4o for the planner, and Claude 3.5 Haiku or GPT-4o-mini for the generator. These models have strong tool-calling capabilities and are cost-effective for agent workflows.

How do I prevent an agent from making expensive mistakes?

Use read-only permissions initially. Add allow-lists for tools and parameters. Implement human approval gates for destructive actions. Log every decision with full context for post-hoc review.

What is the best framework for building QA agents?

LangChain plus LangGraph is the most mature stack. For simpler workflows, n8n or LangFlow may be sufficient. For maximum control, build custom state machines with direct API calls to the LLM.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.