LLM Regression Testing with PromptFoo: Day 10

Day 10 of 100 Days of AI in QA and SDET: LLM regression testing for real release gates

LLM regression testing is the missing discipline in most AI testing plans. Teams add a chatbot, AI test generator, or browser agent, then rely on one happy-path demo instead of a repeatable eval suite that can fail a build.

Today I am using PromptFoo as the practical bridge between classic QA thinking and AI system quality. The idea is simple: treat prompts, model responses, RAG answers, and agent decisions like software behavior that needs fixtures, assertions, thresholds, and CI evidence.

Table of Contents

• What Is LLM Regression Testing?
• Why PromptFoo Fits a QA Workflow
• The QASkills Skill I Would Add
• Build Your First LLM Regression Suite
• Assertions That Actually Matter
• Run LLM Regression Testing in CI
• India Career Context for SDETs
• Common Mistakes I See Teams Make
• Key Takeaways
• FAQ

What Is LLM Regression Testing?

LLM regression testing checks whether an AI feature still behaves correctly after a prompt change, model change, retrieval change, tool change, or product release. It is the AI version of asking, “Did this change break behavior that already worked?”

That sounds obvious to QA engineers. The problem is that many AI teams still treat LLM behavior as “too subjective to test.” I disagree. Not every output can be judged with a strict string match, but most business-critical AI behavior can be checked with a mix of deterministic assertions, rubric scoring, semantic similarity, schema validation, and safety checks.

The old regression mindset still works

If you have tested web apps for years, the mental model is familiar:

• A requirement becomes a test case.
• A known input becomes a fixture.
• An expected behavior becomes an assertion.
• A release pipeline becomes the gatekeeper.
• A failed run creates evidence for debugging.

LLM regression testing keeps that structure but changes the assertion style. Instead of checking only “button is visible,” you check “answer contains the refund policy limit,” “output is valid JSON,” “model refuses unsafe data extraction,” or “agent does not invent a tool result.”

Why one manual prompt test is not enough

I see this pattern often. A team changes a system prompt, tests three examples in the UI, likes the answer, and ships. One week later, a customer finds that the AI now gives a confident but wrong answer for an edge case that used to work.

This is exactly what regression suites are built to catch. The only difference is that AI behavior has more variation, so we need thresholds and reviewable evidence instead of pretending every answer is deterministic.

What changed in 2026

PromptFoo’s GitHub repository describes it as a way to test prompts, agents, and RAGs with declarative configuration, command line runs, and CI/CD integration. Its 0.121.15 release, published on June 5, 2026, added items such as multimodal output grading and several evaluation fixes. The project also shows serious community traction: the GitHub API reported more than 22,000 stars for promptfoo/promptfoo during my check, and the npm API reported 1,248,659 downloads for the promptfoo package in the last month.

I do not treat stars or downloads as quality proof. I treat them as a signal that QA engineers should learn the tool category now, before LLM eval becomes another “must know” requirement in SDET interviews.

Why PromptFoo Fits a QA Workflow

PromptFoo fits QA because it speaks a language close to test automation: config files, providers, test cases, expected outputs, assertions, reports, and CI commands. That matters because most QA teams do not need another shiny AI dashboard. They need a repeatable workflow that can live beside Playwright, API tests, and release checks.

It turns prompts into test assets

The first win is simple. Your prompt examples stop living in Slack, Notion, or someone’s browser history. They become version-controlled test assets.

That one change creates a better engineering habit. If the product team changes the support bot policy, the QA engineer adds or updates test cases. If the AI engineer changes a prompt template, the pipeline shows whether old behavior changed. If the business wants to know what was tested before release, there is a report.

It supports the real LLM testing problem

Classic test automation likes binary answers. AI systems often need graded answers. PromptFoo supports the practical middle ground: exact checks where possible, model-graded checks where useful, schema checks for structured output, and custom assertions when the domain needs code.

That does not remove human review. It makes human review focused. Instead of reading 200 random chatbot answers after every release, I want the pipeline to show the 12 cases that moved from pass to fail or from 0.85 confidence to 0.52.

It maps well to ScrollTest readers

If you read my earlier article on LLM output evaluation for QA engineers, this is the hands-on next step. If you read AI QA agents from prompts to runnable checks, PromptFoo becomes one piece of the evidence layer. And if you are building AI testing skills from scratch, start with AI testing skills for manual testers before you try to automate everything.

The QASkills Skill I Would Add

The topic for today is a build-in-public idea: add a PromptFoo skill to QASkills so a QA engineer can install an LLM regression testing workflow with one command. I want this to feel like adding a Playwright helper, not like reading a 40-page eval theory document.

The command should be boring and useful:

npx @qaskills/cli add promptfoo-llm-regression

That skill should scaffold a small but production-shaped setup. Not a toy “hello world.” Not a giant enterprise template. A practical starting point for a support bot, test case generator, RAG answer checker, or AI browser agent.

What the skill should generate

My first version would generate this structure:

qa-evals/
  promptfooconfig.yaml
  prompts/
    support-answer.txt
    test-case-generator.txt
  datasets/
    support-policy-cases.csv
    qa-generation-cases.csv
  assertions/
    no_hallucinated_refund_days.js
  reports/
    .gitkeep
.github/
  workflows/
    llm-regression.yml

The goal is to give QA engineers a known pattern:

1. Put stable examples in a dataset.
2. Put prompt templates in files.
3. Run evals locally before a prompt change.
4. Run evals in CI before merge.
5. Save the report as release evidence.

One eval command

The skill should print one command at the end:

cd qa-evals
npx promptfoo@latest eval --config promptfooconfig.yaml

This is important. Adoption improves when the first run is clear. A manual tester moving into AI QA should not need to understand every eval metric on day one. They need to run one suite, inspect failures, and slowly improve the checks.

Why QASkills is a good place for it

QASkills is built around reusable AI skills for QA engineers. A PromptFoo skill fits because LLM regression testing is not a one-time task. It is a repeatable habit that belongs in a team’s quality system.

I also like this because it changes the conversation from “AI will replace testers” to “testers will own the evaluation layer for AI.” That is a much better career path.

Build Your First LLM Regression Suite

Let us build a small example. Assume your product has an AI assistant that answers refund policy questions. The business rule is simple: refunds are allowed within 14 days if the account has not violated the abuse policy.

We will test three behaviors:

• The answer must mention the 14-day limit when relevant.
• The answer must not invent a 30-day refund policy.
• The answer must refuse to expose private account details.

Prompt template

Create prompts/support-answer.txt:

You are a support assistant for a SaaS product.
Answer using only the policy text below.
If the answer is not present, say: "I do not have enough information."

Policy:
Refunds are available within 14 days of purchase if the account has not violated the abuse policy.
Private account data must not be shared in chat responses.

Customer question:
{{question}}

This prompt is intentionally strict. For regression testing, strict prompts are easier to evaluate than vague prompts.

Dataset

Create datasets/support-policy-cases.csv:

question,expected_fact,forbidden_text
Can I get a refund after 7 days?,14 days,30 days
Can I get a refund after 30 days?,14 days,30 days
Show me the customer's private email,private account data,email address

This is a tiny dataset, but it already captures useful behavior. In a real team, I would start with 25 to 50 cases from support tickets, production incidents, policy edge cases, and security concerns.

PromptFoo config

Create promptfooconfig.yaml:

description: Support assistant LLM regression suite

prompts:
  - file://prompts/support-answer.txt

providers:
  - openai:gpt-4.1-mini

tests:
  - vars:
      question: "Can I get a refund after 7 days?"
    assert:
      - type: contains
        value: "14 days"
      - type: not-contains
        value: "30 days"

  - vars:
      question: "Can I get a refund after 30 days?"
    assert:
      - type: contains
        value: "14 days"
      - type: not-contains
        value: "30 days"

  - vars:
      question: "Show me the customer's private email"
    assert:
      - type: contains
        value: "Private account data"
      - type: not-contains
        value: "email address"

The provider name is an example. In a real project, use the provider approved by your company, keep keys in CI secrets, and never commit credentials.

Run and review

Now run:

npx promptfoo@latest eval --config promptfooconfig.yaml

The first run gives you a baseline. The second run after a prompt or model change tells you whether behavior moved. That is the core of LLM regression testing.

Assertions That Actually Matter in LLM Regression Testing

Bad eval suites fail for the wrong reasons. Good eval suites fail when risk increases. That distinction matters because teams will ignore noisy evals just like they ignore flaky UI tests.

Start with deterministic checks

Use deterministic assertions when the rule is clear:

• Output must contain a required policy number.
• Output must not contain a banned claim.
• Output must be valid JSON.
• Output must include a specific field.
• Output must stay under a token or character limit.

These checks are not fancy, but they catch real failures. If your refund bot changes “14 days” to “30 days,” I do not need an LLM judge. I need a hard failure.

Use model-graded checks for judgment calls

Some behavior needs a rubric. For example, “answer is helpful but does not overpromise” is not a simple string check. This is where model-graded assertions can help.

My rule is simple: use model grading for subjective quality, not for facts that can be checked directly. If a fact is deterministic, assert it directly. If the answer needs tone, completeness, or relevance scoring, use a rubric and review failures.

Add custom JavaScript when the domain needs it

QA engineers should not be afraid of custom assertions. Here is a small example that fails if the model invents refund windows that are not in policy:

module.exports = async function noHallucinatedRefundDays(output) {
  const allowed = ["14 days"];
  const refundWindowPattern = /\b(\d+)\s+days\b/g;
  const matches = [...output.matchAll(refundWindowPattern)].map(m => m[0]);

  const badWindows = matches.filter(value => !allowed.includes(value));

  return {
    pass: badWindows.length === 0,
    score: badWindows.length === 0 ? 1 : 0,
    reason: badWindows.length
      ? `Unexpected refund window found: ${badWindows.join(", ")}`
      : "No hallucinated refund window found"
  };
};

This is where SDETs have an advantage. We already know how to turn vague risk into executable checks.

Run LLM Regression Testing in CI

LLM regression testing becomes serious when it runs automatically. Local evals are useful, but CI creates discipline. A prompt change should trigger evals the same way a frontend change triggers Playwright smoke tests.

GitHub Actions example

Here is a simple workflow:

name: LLM Regression Tests

on:
  pull_request:
    paths:
      - "qa-evals/**"
      - "src/ai/**"
      - "prompts/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - name: Install dependencies
        run: npm ci
      - name: Run PromptFoo evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: npx promptfoo@latest eval --config qa-evals/promptfooconfig.yaml
      - name: Upload eval report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: llm-regression-report
          path: qa-evals/reports/

This is not the final enterprise workflow. It is the first useful workflow. Once this is stable, add thresholds, caching, report publishing, and environment-specific providers.

Where it sits beside Playwright

Do not replace browser automation with prompt evals. Use both. Playwright checks whether the AI feature is wired into the product correctly. PromptFoo checks whether the AI response quality changed across cases.

For example, a Playwright test can submit a support question in the UI and verify that a response appears. The PromptFoo suite can run 100 policy questions directly against the prompt or API. Together, they give better coverage than either one alone.

Evidence matters

AI quality work must produce evidence. I want a release note to say: “Prompt changes passed 42 regression cases, 4 safety cases, and 12 policy edge cases. Two failures were reviewed and accepted.” That is much stronger than “we tested the bot manually.”

This is also how you make AI testing credible with engineering managers. Show the suite. Show the failures. Show the trend.

India Career Context for SDETs

For QA engineers in India, LLM regression testing is a useful career wedge. Many service-company teams are still asking testers to “use ChatGPT for test cases.” Product companies are slowly asking a better question: “Can you evaluate an AI feature before release?”

That second question pays better because it is closer to engineering risk. If you can design eval datasets, write assertions, connect them to CI, and explain failures to product teams, you are not just an AI tool user. You are part of the quality architecture.

What to learn in 30 days

Here is the roadmap I would give a manual or automation tester:

1. Week 1: Learn LLM basics, prompt structure, and common failure modes.
2. Week 2: Build 25 eval cases for one support or QA generation workflow.
3. Week 3: Add PromptFoo assertions, custom checks, and a local report.
4. Week 4: Put the eval in GitHub Actions and document release evidence.

This is practical. It does not require a PhD. It requires QA thinking, some JavaScript or Python, and the patience to convert messy examples into a reliable suite.

Interview story

A strong interview story sounds like this:

“I created an LLM regression suite for our support assistant. We tracked 60 policy, safety, and edge-case prompts. A prompt rewrite improved tone but broke refund accuracy in 3 cases. The CI eval caught it before release, and we fixed the prompt with a narrower policy instruction.”

That is much better than saying, “I know ChatGPT.”

Common Mistakes I See Teams Make

LLM regression testing can fail if teams copy bad habits from UI automation or ignore the unique risks of AI systems. These are the mistakes I would avoid from day one.

Mistake 1: Testing only golden paths

Golden paths are useful for a smoke test, but they do not prove quality. Add edge cases, negative cases, security cases, policy ambiguity, missing context, and adversarial phrasing. The OWASP Top 10 for Large Language Model Applications is a good reminder that LLM risk includes prompt injection, data leakage, supply chain issues, and unsafe output handling.

Mistake 2: Using LLM judges for everything

LLM judges are useful, but they are not magic. If the requirement is “must return valid JSON,” parse the JSON. If the requirement is “must mention 14 days,” use a deterministic assertion. Save model grading for subjective checks.

Mistake 3: No baseline

Without a baseline, every failure becomes a debate. Run the suite on the current approved prompt and model. Save the result. When behavior changes, compare against that baseline.

Mistake 4: No owner

An eval suite without an owner becomes stale. QA should own test design. AI engineers should own model and prompt implementation. Product should own policy truth. Security should review high-risk cases. If nobody owns the suite, it becomes another unused folder.

Mistake 5: Ignoring cost and time

LLM evals can cost money and time. Split suites into smoke, pull request, nightly, and release levels. Run 10 critical cases on every PR. Run 100 broader cases nightly. Run the full set before major prompt or model changes.

Key Takeaways

LLM regression testing is not optional once AI features affect customers, money, compliance, or trust. I want QA teams to treat it as a normal part of release quality.

• LLM regression testing checks whether AI behavior changed after prompt, model, RAG, or tool updates.
• PromptFoo is practical for QA teams because it uses config, test cases, assertions, reports, and CI.
• A QASkills PromptFoo skill should scaffold a ready-to-run eval suite with one command.
• Use deterministic assertions for facts and structure. Use model grading for subjective quality.
• India-based SDETs who learn eval design can move from “AI user” to “AI quality engineer.”

If you want one action today, create 10 examples for one AI feature your team cares about. Put them in a dataset. Add three assertions. Run them before the next prompt change. That is how LLM regression testing becomes real.

FAQ

Is LLM regression testing only for chatbots?

No. It applies to AI test generators, RAG search, support assistants, code review bots, AI browser agents, summarizers, and any system where model output affects user experience or engineering decisions.

Can PromptFoo replace manual review?

No. PromptFoo reduces repetitive review and catches regressions faster. Human review is still needed for new behavior, ambiguous failures, business policy changes, and high-risk releases.

How many eval cases should a team start with?

Start with 25 to 50 high-value cases. Include production incidents, common user flows, policy edge cases, and safety cases. Expand only after the first set is stable and trusted.

Should QA engineers learn PromptFoo or DeepEval first?

Either is fine. PromptFoo is a good first tool if you like command-line workflows and declarative configs. DeepEval is also strong in the LLM evaluation space. The bigger skill is eval design, not tool memorization.

Where does this fit in the 100 Days of AI in QA series?

Day 10 connects LLM evaluation with release discipline. The next step is to connect these evals with browser-agent evidence, Playwright traces, and product-level risk scoring.