LLM Performance Testing: Token and Latency Gates for QA
Most QA teams still treat LLM features as “test the output and ship it.” Nobody owns the bill or the latency budget, and that is exactly why AI products quietly burn thousands of dollars a month while users complain the app “feels slow.” LLM performance testing is the discipline that fixes both: it puts token budgets and latency gates in your CI pipeline, right next to your functional tests.
Table of Contents
- Why LLM Performance Testing Is Now a QA Problem
- The Metrics That Actually Matter
- Building a Golden Set for Performance
- Token Accounting: Input, Output, and Cached
- Latency Gates: p95, TTFT, and the 2-Second Rule
- Cost Regression Testing in CI
- A Working LLM Performance Testing Harness
- Traps I Hit the First Time
- A Performance Gate Checklist
- India Context: Why This Skill Pays
- Key Takeaways
- FAQ
Contents
Why LLM Performance Testing Is Now a QA Problem
I used to think performance testing meant load-testing an API with JMeter until it fell over. That model breaks the moment your product starts calling an LLM. Now your “API” bills you per token and thinks for a variable number of seconds on every request.
The scale makes this unavoidable. LangChain alone crossed 271 million downloads a month as of August 2026, and langchain-core is at 167.7 million. The LangChain repo sits at over 144,000 GitHub stars. DeepEval, a library that exists purely for testing AI, does 5.6 million downloads a month on PyPI. When that many apps are calling models, token cost and latency stop being a data-science problem and become a QA problem.
Here is the shift in one sentence: a functional test answers “is the answer right?”, while LLM performance testing answers “does the right answer arrive fast enough, and at a price the business can afford?” Both are release blockers. You only catch the second one if you measure it.
The team that ignores this discovers it in the worst possible way: a finance report three months after launch, a surprise cloud bill, and a support queue full of “the bot is slow” tickets that nobody can reproduce. The team that owns it treats every token and millisecond as a testable artifact from day one.
The Metrics That Actually Matter
You do not need a metrics dashboard with forty graphs. You need five numbers, tracked per request and rolled up per release:
Tokens in, tokens out
- Input tokens: everything you send, including the system prompt, retrieved context, and conversation history.
- Output tokens: everything the model generates, including reasoning traces on reasoning models.
- Cached input tokens: the part of your input the provider already saw and billed at a discount.
Latency
- TTFT (time to first token): how long until the first character appears. This is what a user feels first.
- Total latency: full response time from request to last token.
- p95 and p99: the tail. Average latency hides the slow requests that generate support tickets.
Cost
- Cost per request and cost per conversation. These are your regression numbers.
- Cost per 1,000 successful interactions. This is the number your manager actually asks about.
Every one of these is measurable from the API response itself. You do not need a vendor observability tool to get started, and that is the point. You can build the whole harness in an afternoon and it will already beat “we’ll look at the dashboard later.”
The reason these five numbers beat a forty-graph dashboard is focus. Latency and cost are the two things that get a product team to actually change code. Every other metric is a proxy for one of them. If a number cannot be traced back to a token, a millisecond, or a dollar, it is probably a vanity metric, and QA does not ship vanity metrics.
Building a Golden Set for Performance
You cannot benchmark an LLM feature against a random list of prompts. You need a golden set that looks like your real traffic, and it has to be stable across releases or your numbers will bounce for no reason.
Three rules for building it:
- Mirror your traffic mix. If 60 percent of your queries are short product questions and 40 percent are long multi-turn conversations, your golden set needs that same split. Ten short queries and zero long ones will flatter your latency numbers.
- Pin it. Store the set in version control and never edit it between baselines. A golden set that changes every sprint is not a baseline, it is a moving target.
- Include the ugly cases. Add the longest conversation, the biggest context, and the worst retrieval result. Happy-path-only benchmarks are how teams ship a feature that collapses on the one query that matters.
- Output costs 6x input. A chatty model that over-explains is a cost bug, not a personality trait.
- Cached input costs 10x less than fresh input. Prompt structure directly determines your bill.
- Model choice swings cost 25x. gpt-5.6-luna is $0.20/M input and $1.20/M output. Routing a simple query to the flagship model is pure waste.
- Input per day: 50 million tokens
- Output per day: 15 million tokens
- Uncached cost: (50M × $5/M) + (15M × $30/M) = $250 + $450 = $700 a day
- That is roughly $21,000 a month before you add retrieval, embeddings, or a second model.
- Vector search: usually 50 to 200 ms
- Reranking: another 50 to 150 ms
- Generation TTFT: the model’s own time to first token
- Pin a golden prompt set. Ten to thirty real queries that represent your traffic mix, not just happy paths.
- Run each query through the new code. Capture tokens, latency, and cost per request.
- Compare against the last release’s baseline. Fail if p95 latency or cost per 1,000 interactions rises by more than a set threshold, usually 10 to 15 percent.
- Fail on budget overrun. If a single query now costs 3x what it did, stop the merge, not after launch.
- Cold cache vs warm cache. The first request after a prompt change is slow and expensive; the hundredth is fast and cheap. Always warm the cache before you benchmark, or your numbers measure the wrong thing.
- Non-determinism. The same prompt returns different token counts and latencies run to run. That is why you measure a distribution over many runs, not one number.
- Reasoning models hide their cost. Models that “think” emit reasoning tokens before the visible answer. You pay for all of them. Count total output tokens, not the visible answer length.
- Parallel vs serial requests. Measuring one request at a time hides contention. If you care about real users, drive a realistic concurrency level.
- Token baseline recorded. You know input, output, and cached token counts for the golden set.
- Cost per 1,000 interactions computed. And it is below the product’s agreed budget.
- p95 latency under the 2-second budget. Measured at realistic concurrency, not one request at a time.
- TTFT under 500 ms. Because that is the number the user perceives first.
- No cache-busting prompt bug. Timestamps and request IDs are not sitting above the cacheable prefix.
- Regression threshold wired into CI. The build fails when cost or p95 latency jumps more than 10 to 15 percent.
- LLM performance testing means measuring tokens, latency, and cost per request, not just load-testing a server.
- Output tokens cost 6x input on flagship models, and cached input costs 10x less than fresh input.
- Gate on p95 latency, not average, and budget TTFT separately from total latency.
- A 10,000-conversation-a-day chatbot can cost $21,000 a month before caching.
- Cost regression is a CI gate: compare every PR against the last release’s baseline.
Generating realistic synthetic queries is its own skill, and I covered the full playbook in the AI test data generation guide. The short version: ask the model for failure rows on purpose, then validate every one through a schema check before it enters your set.
Token Accounting: Input, Output, and Cached
Most teams assume cost scales with “number of requests.” It does not. Cost scales with tokens, and tokens behave in three different ways that all get billed differently.
Look at OpenAI’s current flagship pricing for gpt-5.6-sol, straight from the OpenAI pricing page: $5.00 per million input tokens, $30.00 per million output tokens, and $0.50 per million cached input tokens. Three observations that change how you test:
Let me make this concrete. A support chatbot handling 10,000 conversations a day, five turns each, averaging 1,000 input tokens and 300 output tokens per turn:
Now cache the input and watch the input line drop from $250 to $25 a day. Same product, same answers, one prompt-structuring decision. That is a real budget line, and QA is the team that can prove it.
The caching detail matters more than people think. Cached input is billed at a discount only when the prefix of your prompt is byte-identical to a previous request. If you stuff a timestamp or a random request ID near the top of the prompt, you break the cache on every call and silently pay the 10x fresh-input rate. A simple token-accounting check catches that bug in one afternoon.
There are three places token waste hides, and each one is testable. First, a bloated system prompt: a 4,000-token instruction block you pay for on every single request. Second, RAG context stuffing: retrieval that dumps twenty chunks into the prompt when the top three would do. Third, verbosity drift: the model slowly returning longer answers because a prompt tweak removed a length constraint. Each of these shows up as a rising input or output token count on your golden set, which means your harness catches them before they become a $5,000 surprise.
Latency Gates: p95, TTFT, and the 2-Second Rule
Latency is a UX bug that does not show up in a screenshot. Users notice it, product managers get complaints about it, and nobody has a test for it.
The rule of thumb I use: interactive chat degrades after roughly two seconds of total latency, and the first token should land in under 500 milliseconds for a chat UI to feel responsive. These are not laws of physics, but they are the budgets most product teams implicitly design around.
The number that matters most is p95, not average. If 95 out of 100 requests return in 1.8 seconds but five take 6 seconds, your average looks fine at about 2 seconds and your users still think the app is broken. Tail latency is where the complaints live.
Streaming vs non-streaming
Streaming changes perception, not physics. A streamed response with a 400 ms TTFT feels responsive even if the full answer takes three seconds to complete, because the user is reading while the model writes. A non-streamed response that shows a spinner for three seconds feels broken, even though the total time is identical. That is why TTFT and total latency are separate gates, and why you should never collapse them into one number.
For RAG apps, budget the pieces separately:
If retrieval plus reranking eats 400 ms of a 2-second budget, you now know exactly which layer to optimize when the gate goes red. That is the difference between a QA team that reports “it’s slow” and one that reports “reranking added 180 ms to p95, here is the commit.”
Cost Regression Testing in CI
Cost is a regression, and regressions belong in CI. The pattern is identical to a functional gate, except the assertion is on tokens and dollars instead of status codes.
Here is the workflow I run on every PR that touches prompts, retrieval, or model routing:
This sits naturally alongside your existing evaluation work. If you already run DeepEval or PromptFoo gates in CI, the performance harness is one more script in the same pipeline. If you are still wiring up AI evals, start with the framework comparison and add performance gates on top.
A Working LLM Performance Testing Harness
Here is a compact Python harness that captures tokens, latency, and cost from an OpenAI-compatible API. It is the seed of a real performance gate.
import time
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY from env
# gpt-5.6-sol, standard tier, USD per 1M tokens (Aug 2026)
PRICE_PER_1M = {"input": 5.00, "cached": 0.50, "output": 30.00}
def measure(prompt: str, model: str = "gpt-5.6-sol") -> dict:
t0 = time.perf_counter()
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
)
latency_s = time.perf_counter() - t0
u = resp.usage
in_tok = u.prompt_tokens
out_tok = u.completion_tokens
det = getattr(u, "prompt_tokens_details", None)
cached_tok = det.cached_tokens if det else 0
fresh_tok = in_tok - cached_tok
cost = (
fresh_tok * PRICE_PER_1M["input"] / 1_000_000
+ cached_tok * PRICE_PER_1M["cached"] / 1_000_000
+ out_tok * PRICE_PER_1M["output"] / 1_000_000
)
return {
"latency_s": round(latency_s, 3),
"input_tokens": in_tok,
"cached_tokens": cached_tok,
"output_tokens": out_tok,
"cost_usd": round(cost, 6),
}
For real TTFT you need streaming, because the non-streaming call only returns when the whole response is done:
def measure_ttft(prompt: str, model: str = "gpt-5.6-sol") -> float:
t0 = time.perf_counter()
stream = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
stream=True,
)
for _ in stream:
break # first chunk arrived
return round(time.perf_counter() - t0, 3)
Run every prompt in your golden set a few times, collect the numbers, and compute percentiles in plain Python:
import statistics
latencies = [run["latency_s"] for run in results]
costs = [run["cost_usd"] for run in results]
def pct(data, p):
return sorted(data)[int(len(data) * p)]
print("p50 latency:", round(statistics.median(latencies), 3))
print("p95 latency:", round(pct(latencies, 0.95), 3))
print("total cost:", round(sum(costs), 4))
Write those three numbers to a JSON artifact. Your CI gate compares the artifact to the baseline and fails the build on a regression, no observability vendor required.
Traps I Hit the First Time
Four traps will make your numbers meaningless if you are not careful.
This is the same discipline as load testing, applied to a non-deterministic, metered dependency. The instinct to “just average it” is the trap.
A Performance Gate Checklist
If you want to ship this today, here is the exact checklist I use to decide whether an LLM feature is release-ready:
Work through that list once and you will have something most AI teams do not: a non-functional gate for a non-deterministic dependency. It is not glamorous, and it is exactly the kind of thing that separates a QA team that keeps up with AI from one that gets blamed for the bill.
India Context: Why This Skill Pays
Here is the career angle, because I get asked about this constantly. Indian QA teams at product companies are being asked two new interview questions: “how do you test an LLM feature’s cost?” and “how do you set a latency budget for an AI feature?”
Most candidates can talk about prompt injection and hallucination now. Almost nobody can walk through token accounting and a p95 latency gate. That gap is your advantage.
The senior SDET band in India sits around ₹25 to 40 LPA, and the roles that specifically ask for AI quality engineering pay at the top of that range. Cost and latency testing is the differentiator, because it is the one non-functional skill that maps directly to business outcomes. When you can tell a founder “your chatbot is spending $700 a day and 40 percent of it is uncached input,” you are no longer a tester. You are the person who protects the runway.
If you are on the AI-testing track, pair this with a solid observability foundation so you can trace exactly where each token and millisecond goes, and keep your security testing sharp while you are at it.
For a portfolio piece, build the harness from this article against a free or cheap model, wrap it in a GitHub Actions gate that fails on a 15 percent cost regression, and publish the repo. It is a weekend-sized project that demonstrates the exact skill hiring managers are starting to ask for, and it gives you a number to cite in interviews: “I cut a benchmark suite’s cost by 40 percent by caching input.”
Key Takeaways
FAQ
What is LLM performance testing?
It is the practice of measuring token usage, latency, and cost for LLM-backed features and gating releases on those numbers, the same way you gate on functional test results. It covers TTFT, p95 latency, tokens in and out, and cost per request or conversation.
Why is output more expensive than input?
Providers price generation higher because it is the expensive part to serve. On OpenAI’s flagship tier, output tokens cost $30 per million while input costs $5 per million. That 6x ratio is why verbosity and reasoning traces matter so much to your bill.
How do I measure TTFT?
You stream the response and time from request start to the first chunk. A non-streaming call only returns when the entire response is complete, so it cannot tell you time-to-first-token. The harness above shows both patterns.
How do I make LLM calls cheaper?
Three levers, in order of impact: cache your input (10x cheaper), route simple queries to a smaller model, and trim output tokens by tightening prompts and capping response length. Measure before and after so you can prove the savings.
Where should cost gates live in CI?
Alongside your eval gates, on every PR that changes prompts, retrieval, or model routing. Compare cost per 1,000 interactions and p95 latency against the last release, and fail the merge on a regression of more than 10 to 15 percent.
