API Testing with AI Agents: Contracts and Chaos

Day 2 of 100 Days of AI in QA & SDET: API testing with AI agents.

API testing with AI agents is not about asking ChatGPT to write random Postman tests. The useful version is smaller and sharper: give an agent your OpenAPI spec, Pact contracts, logs, and a controlled chaos sandbox, then let it find contract drift and failure modes before customers do.

I see many QA teams add AI at the wrong layer. They ask an LLM to generate 100 happy-path tests, then celebrate volume. The better question is: can the agent prove that the consumer and provider still agree when latency spikes, fields disappear, tokens expire, or downstream services return 503?

Table of Contents

• What API Testing with AI Agents Means
• Why Contract Validation Needs an Agent
• Reference Architecture for Agentic API Testing
• Contract Validation Workflow
• Chaos Testing Workflow
• TypeScript Example with Playwright, Ajv, and Pact Ideas
• CI/CD and Governance Rules
• India Context for QA and SDET Teams
• Common Mistakes
• Key Takeaways
• FAQ

What API Testing with AI Agents Means

API testing with AI agents means using an LLM-driven workflow to plan, execute, inspect, and improve API checks with tool access. The agent does not replace your test runner. It sits above the runner and decides what evidence to collect, which risks to test, and how to explain failures.

A basic API test calls an endpoint and checks status code 200. A better API test validates schema, headers, authentication, idempotency, error formats, retries, data cleanup, and observability signals. An agent helps because it can connect these pieces into a repeatable investigation.

Agent, not magic button

I define an API testing agent as four parts:

• Planner: reads the user story, OpenAPI spec, Pact files, and recent incidents.
• Executor: runs Playwright API tests, contract checks, curl commands, and chaos experiments.
• Critic: compares responses against schemas, contracts, logs, and business rules.
• Reporter: writes a defect report with reproduction steps and confidence level.

This is close to the multi-agent pattern I covered in LangGraph for QA Engineers, but here the tools are API-native. The agent needs permissions to call test environments, not production. It should have rate limits, audit logs, and a kill switch.

What the agent should never own

The agent should not decide release approval alone. It should not mutate shared data without cleanup. It should not invent acceptance criteria. Human-written contracts, schemas, and service-level expectations remain the source of truth.

That boundary matters. If your agent creates the expected result and validates against the same expected result, you have a beautiful hallucination loop. Real API testing with AI agents starts from external truth: consumer contracts, OpenAPI specs, database constraints, and production-like failure patterns.

Why Contract Validation Needs an Agent

Contract testing has been around for years. Pact describes consumer-driven contract testing as a way for consumers and providers to agree on request and response expectations before integration breaks. Pact JS is still active, and the GitHub API showed 1,776 stars for pact-foundation/pact-js during my research for this post. The npm API reported 2,883,021 last-month downloads for @pact-foundation/pact.

So why add an agent? Because real contract failures are rarely clean. A breaking change may show up as a renamed field, a stricter enum, a nullable value becoming required, a header missing under one gateway path, or a provider returning a correct schema with the wrong business meaning.

The contract drift problem

Contract drift happens when teams change services faster than their tests and documentation. Microservices make this worse. One mobile app, one web app, one partner integration, and one analytics pipeline may all consume the same endpoint differently.

An agent can inspect drift from multiple angles:

1. Compare the OpenAPI spec against the actual response.
2. Compare Pact contracts against provider behavior.
3. Compare old production samples against the current test environment.
4. Generate negative tests for missing, null, oversized, and invalid values.
5. Summarize the blast radius by consumer.

This is where a normal script becomes difficult to maintain. You can code every comparison by hand, but the investigation logic becomes messy. An agent can decide which comparator to run next based on the failure it just saw.

Contracts are executable documentation

I like contract tests because they reduce arguments. Instead of saying, “The frontend team expected this field,” you point to an executable contract. The test either passes or fails.

For QA teams, this changes the conversation from UI-only validation to service ownership. It also connects well with the microservices strategy I wrote about in Microservices Test Automation Strategy. UI tests still matter, but they are the wrong place to catch every API mismatch.

Reference Architecture for Agentic API Testing

A practical architecture is not complicated. You need a repository of truth, a safe execution layer, and a reporting layer that engineers trust.

Inputs

The agent should read:

• OpenAPI or Swagger specs
• Pact contract files
• Playwright API test files
• Postman collections, if your team still uses them
• Recent production samples with sensitive data removed
• Incident notes and flaky test history
• Service dependency maps

Do not feed raw secrets, customer data, or full production logs to a hosted model. Redact tokens, emails, phone numbers, addresses, and IDs. If your company has strict data rules, run the model through an approved gateway or use a local model for planning and a normal deterministic runner for execution.

Tools

The agent needs tools with narrow permissions:

• run_api_test to execute a specific Playwright project
• validate_schema to run Ajv against JSON Schema
• verify_contract to run Pact provider checks
• inject_latency to add a controlled network delay in a sandbox
• read_logs to inspect service logs for a request ID
• create_bug_report to write a draft issue, not auto-file a blocker

Notice the tool names. They are boring. That is good. A boring tool is easier to audit than a broad “execute anything” shell.

Outputs

The final output should include:

• Scenario tested
• Contract or schema used
• Request and response sample with secrets masked
• Observed failure
• Expected behavior
• Possible consumer impact
• Confidence score and reason
• Links to logs, traces, and CI artifacts

If the report does not include evidence, engineers will ignore it. That is true for AI and non-AI automation.

Contract Validation Workflow for API Testing with AI Agents

Here is the workflow I recommend for API testing with AI agents when contract validation is the goal.

Step 1: Build the contract inventory

Start by listing each service, endpoint, consumer, and contract source. Many teams skip this and go straight to test generation. That creates noise.

A simple inventory table is enough:

Service	Endpoint	Consumer	Contract Source	Risk
Orders	POST /orders	Web checkout	Pact	High
Payments	POST /payments/authorize	Orders service	OpenAPI + Pact	High
Catalog	GET /products/{id}	Mobile app	OpenAPI	Medium

The agent can maintain this inventory by reading repository metadata and pull requests. But the first version should be reviewed by humans.

Step 2: Ask the agent to generate risk-based checks

The prompt should be strict. Do not ask “write all tests.” Ask for risk-based checks from the contract.

You are an API test planner.
Input: OpenAPI spec, Pact contract, and recent incident notes.
Task: propose the 10 highest-risk API checks for POST /orders.
Rules:
- Use only fields present in the provided spec or contract.
- Mark each check as contract, schema, auth, idempotency, or chaos.
- Do not invent business rules.
- Return Playwright test names and the evidence needed.

This prompt forces the agent to stay grounded. It can still be wrong, but the output is reviewable.

Step 3: Run deterministic validators

The agent should not be the validator. Ajv validates JSON Schema. Pact validates provider contracts. Playwright runs HTTP calls. The LLM chooses and explains; deterministic tools decide pass or fail.

This separation is critical for auditability. If a payment contract fails, I want the CI artifact, not a paragraph saying the model “thinks” it failed.

Step 4: Create a drift report

When a failure occurs, the agent should write a drift report. Example:

Contract drift detected: Orders API
Endpoint: POST /orders
Consumer: web-checkout
Expected: response.items[].price.currency is required
Actual: currency missing in 3 of 5 sampled responses
Impact: checkout UI cannot format price for multi-currency carts
Evidence: CI run #4821, trace ID req-91f2, Pact interaction create-order-success
Suggested owner: Orders API team
Confidence: High, because schema and Pact both fail on the same field

That report is useful. It gives the developer the exact field, consumer, artifact, and likely impact.

Chaos Testing Workflow for API Testing with AI Agents

Contract validation answers, “Do we still agree?” Chaos testing asks, “What happens when the world is ugly?”

Chaos Mesh describes itself as a cloud-native chaos engineering platform for Kubernetes. During research, the GitHub API showed 7,740 stars for chaos-mesh/chaos-mesh. LitmusChaos is another common option, and the GitHub API showed 5,429 stars for litmuschaos/litmus. These tools are not QA toys. They are serious infrastructure tools, so use them carefully.

Good API chaos scenarios

Start with small, reversible experiments:

• Add 500 ms latency between API gateway and payment service.
• Return HTTP 503 from an inventory dependency for 2 minutes.
• Drop 5% of packets between two services in a staging namespace.
• Expire a token mid-flow and check the error contract.
• Throttle a downstream dependency and verify retry limits.
• Make a read replica stale and verify the API response message.

The agent can select scenarios based on service dependency maps and past incidents. For example, if the last three incidents involved payment timeouts, the agent should prioritize payment latency and retry behavior.

Bad API chaos scenarios

Do not start with random failure injection. Do not run experiments in production unless your organization already has mature chaos engineering controls. Do not let an agent create cluster-level chaos without approval.

For most QA teams, the first version should run in a dedicated namespace with fake data and a fixed experiment duration. The agent should propose the experiment, but CI should enforce limits.

Chaos report format

A useful chaos report has four lines:

• Experiment: 500 ms latency on payment authorization.
• Expected: API returns pending status within 2 seconds with retry-safe response.
• Actual: API times out after 30 seconds and creates duplicate authorization attempts.
• Risk: customer may see failed checkout while payment is still captured.

This is where API testing with AI agents becomes valuable. The agent connects latency, duplicate calls, logs, and contract expectations into one readable story.

TypeScript Example with Playwright, Ajv, and Pact Ideas

Here is a small TypeScript example. It is not a full framework, but it shows the pattern: deterministic checks first, agent-readable evidence second.

import { test, expect, request } from '@playwright/test';
import Ajv from 'ajv';

const ajv = new Ajv({ allErrors: true });

const orderResponseSchema = {
  type: 'object',
  required: ['id', 'status', 'items', 'total'],
  properties: {
    id: { type: 'string' },
    status: { enum: ['CREATED', 'PENDING', 'FAILED'] },
    items: {
      type: 'array',
      items: {
        type: 'object',
        required: ['sku', 'quantity', 'price'],
        properties: {
          sku: { type: 'string' },
          quantity: { type: 'integer', minimum: 1 },
          price: {
            type: 'object',
            required: ['amount', 'currency'],
            properties: {
              amount: { type: 'number' },
              currency: { type: 'string', minLength: 3, maxLength: 3 }
            }
          }
        }
      }
    },
    total: { type: 'number' }
  }
};

test('POST /orders respects consumer contract for priced items', async () => {
  const api = await request.newContext({
    baseURL: process.env.API_BASE_URL,
    extraHTTPHeaders: {
      Authorization: `Bearer ${process.env.TEST_TOKEN}`,
      'x-test-run-id': process.env.CI_RUN_ID ?? 'local'
    }
  });

  const response = await api.post('/orders', {
    data: {
      customerId: 'test-customer-001',
      items: [{ sku: 'SKU-123', quantity: 1 }]
    }
  });

  expect(response.status()).toBe(201);
  const body = await response.json();

  const validate = ajv.compile(orderResponseSchema);
  const valid = validate(body);

  expect(valid, JSON.stringify(validate.errors, null, 2)).toBeTruthy();
  expect(body.items[0].price.currency).toBe('INR');
});

Now add an agent step around it. The agent reads the schema errors and writes the defect summary. It does not decide the assertion.

{
  "tool": "create_api_drift_report",
  "input": {
    "endpoint": "POST /orders",
    "test": "respects consumer contract for priced items",
    "schemaErrors": "currency is required at items[0].price",
    "sampleResponsePath": "artifacts/order-response.json",
    "traceId": "req-91f2"
  }
}

If you already use Playwright for API checks, this fits naturally. I also recommend reading Playwright TypeScript Setup if your team is standardizing a fresh test stack.

CI/CD and Governance Rules

Agentic API testing fails when teams skip governance. The agent must be useful, but it also must be boring enough for CI.

Recommended pipeline

1. Run lint and unit tests.
2. Run schema validation for changed endpoints.
3. Run Pact provider verification for impacted consumers.
4. Ask the agent to propose 5 extra risk checks from the diff.
5. Run only approved generated checks in a sandbox.
6. Run one or two chaos experiments for high-risk services.
7. Publish an evidence report in the pull request.

Keep the agent out of the main path at first. Make it advisory. Once the reports are trusted, promote specific checks to blocking status.

Security rules

Use these rules from day one:

• No production secrets in prompts.
• No customer PII in model context.
• No broad shell access for the agent.
• No write access to production systems.
• Every agent action must have a run ID.
• Every generated test must be committed or stored as an artifact.

These rules are not optional in enterprise teams. If you work in a services company like TCS, Infosys, Wipro, or Accenture, client security teams will ask these questions before they approve AI workflows. Product companies will ask them too, usually through platform engineering or security.

When to block a build

Block the build when deterministic tools fail: schema validation, contract verification, status code expectations, auth checks, or duplicate transaction checks. Do not block purely because the LLM says the response “looks risky.” Use that as a warning until a human converts it into a deterministic rule.

India Context for QA and SDET Teams

For Indian QA engineers, API testing with AI agents is a strong career move because it sits at the intersection of automation, backend understanding, and AI. That combination is more valuable than only knowing prompt writing.

In interviews, I expect more teams to ask questions like:

• How do you validate API contracts between services?
• How would you test retries and timeouts?
• How do you prevent AI-generated tests from becoming flaky?
• How do you secure test data when using LLMs?
• Can you explain the difference between schema testing and contract testing?

If you are targeting ₹25-40 LPA SDET roles in product companies, this is the kind of depth that separates you from someone who only records UI scripts. You do not need to become a platform engineer overnight. But you should understand APIs, CI, Docker basics, observability, and at least one AI workflow pattern.

For service-company testers, this is also practical. Many client projects have API layers, unstable environments, and integration defects. A small agent that reads contracts, runs checks, and writes clean evidence can save hours during regression cycles.

Common Mistakes

Mistake 1: Generating too many tests

More tests are not automatically better. If the agent creates 300 shallow checks, your CI time goes up and trust goes down. Start with the top 10 risks per critical endpoint.

Mistake 2: Letting the LLM validate correctness

Use code for validation. Use the LLM for planning, summarizing, and connecting evidence. This one rule prevents many bad AI testing workflows.

Mistake 3: Ignoring negative and chaos scenarios

Happy-path API testing misses the failures customers remember. Timeouts, retries, stale reads, duplicate requests, and partial failures are where API quality is tested.

Mistake 4: No cleanup strategy

API tests create data. Chaos tests create weird states. Your framework needs cleanup hooks, idempotent test data, and isolated environments.

Mistake 5: Hiding uncertainty

Agent reports should include confidence and reason. If the model is guessing, say so. A low-confidence warning is still useful when it points engineers to the right artifact.

Key Takeaways

API testing with AI agents works when you keep the agent grounded in contracts, schemas, logs, and controlled experiments. It fails when you treat the model as an oracle.

• Use agents for planning, evidence gathering, drift analysis, and reporting.
• Use deterministic tools like Pact, Ajv, and Playwright for pass or fail decisions.
• Start with contract validation before advanced chaos experiments.
• Run chaos tests only in controlled environments with strict limits.
• For SDETs, this skill is more valuable than generic prompt engineering.

My recommendation is simple: pick one high-risk API this week. Add schema validation, map one consumer contract, run one latency experiment in staging, and ask an agent to write the evidence report. That is a real starting point for API testing with AI agents.

FAQ

Is API testing with AI agents ready for production teams?

Yes, if the agent is bounded. Use it for planning, triage, and reports. Keep validation deterministic and keep production write access out of scope.

Do AI agents replace Pact or schema testing?

No. They make those tools easier to apply across changes. Pact, JSON Schema, Ajv, and Playwright still provide the hard pass or fail signal.

Should QA teams run chaos testing in production?

Most teams should not start there. Begin in staging or a dedicated Kubernetes namespace. Move to production only when engineering, SRE, security, and business owners agree on blast-radius controls.

What skills should an SDET learn for this?

Learn HTTP deeply, OpenAPI, contract testing, Playwright API testing, Docker basics, CI/CD, logs and traces, and one agent framework such as LangGraph. That stack gives you practical power.

What is a good first project?

Build an agent that reads an OpenAPI spec, generates five risk-based Playwright API checks, runs Ajv validation, and writes a drift report. Keep it small. Make it reliable. Then add chaos scenarios.

Sources checked: Pact documentation, npm download API for @pact-foundation/pact, GitHub API for Pact JS, Chaos Mesh, and LitmusChaos, Chaos Mesh documentation, and existing ScrollTest posts on microservices testing, LangGraph for QA, and Playwright TypeScript setup.