|

PromptFoo Upgrade Checklist for QA Teams

ByPromode
PromptFoo upgrade checklist featured image for AI QA release gates

Day 43 of 100 Days of AI in QA & SDET. A PromptFoo upgrade checklist is no longer a nice document sitting in Confluence. When your LLM eval tool changes, your release gate changes too, because provider adapters, red-team strategies, token accounting, assertion behavior, and CI output can all affect the signal your team trusts.

I picked PromptFoo 0.121.19 for today because the release is exactly the kind of update QA teams often treat as “just a dev dependency.” It is not. The official GitHub release was published on 14 July 2026, the npm registry shows 0.121.19 as the latest package version, and npm reported 1,626,279 downloads in the last month for the package during my check. That is enough adoption to justify a disciplined upgrade flow.

Table of Contents

Contents

Why a PromptFoo Upgrade Checklist Matters

Most teams upgrade test dependencies with a small pull request, a lockfile diff, and a quick CI run. That works for a formatter. It is risky for an eval framework that decides whether an AI feature is safe enough to ship.

PromptFoo sits close to the decision layer. It compares model outputs, checks assertions, runs red-team probes, stores results, and gives teams a pass/fail signal. If that signal changes after an upgrade, the team needs to know whether the product got better, the model changed, the dataset drifted, or the tool changed its own behavior.

The failure mode is not always a crash

The scary failure is not “PromptFoo fails to install.” That is easy to catch. The real problem is a silent change in evaluation behavior that turns a weak response into a pass or marks a valid response as a fail.

For AI QA, a false pass is expensive. It can let prompt injection, unsafe tool use, hallucinated policies, or broken retrieval answers reach users. A false fail is also expensive because it blocks releases and burns engineering time.

PromptFoo 0.121.19 is a useful case study

The 0.121.19 release includes provider additions, red-team updates, token-usage work, and assertion fixes. I do not treat that as a random dependency bump. I treat it as a release that can touch:

  • Which model providers your eval suite can call
  • How red-team generation cost is tracked
  • How assertions handle edge values like numeric zero
  • How tool-call failures are represented
  • How CI evidence should be saved for release review

If your team already uses PromptFoo, this is enough to run a controlled upgrade. If your team does not use PromptFoo yet, read this as a model for any LLM evaluation framework upgrade. The same thinking applies to DeepEval, homegrown eval harnesses, RAG scorecards, and agent test runners.

Read Release Notes Like a QA Owner

The first step in this PromptFoo upgrade checklist is not running npm update. The first step is reading the release notes with a QA owner’s eye.

The PromptFoo 0.121.19 changelog lists provider support for Bedrock GPT-5.6 frontier, Meta Model API, Open Interpreter, xAI Grok 4.5, and GPT-5.6 GA. It also includes red-team token accounting, a new goblin strategy, public schema fields for token usage, and fixes around assertions and tool calls.

Classify every release-note item

I use a simple classification before I let a dependency upgrade enter CI:

  1. Provider impact: Does this change how models are called or configured?
  2. Assertion impact: Does this change pass/fail logic?
  3. Dataset impact: Does this change how prompts, vars, or scenarios are interpreted?
  4. Red-team impact: Does this add, remove, or modify attack generation?
  5. Reporting impact: Does this change output files, traces, cost fields, or dashboards?
  6. CI impact: Does this change exit codes, cache behavior, or command output?

For 0.121.19, I would mark provider, assertion, red-team, reporting, and CI evidence as “review required.” That does not mean the upgrade is unsafe. It means the upgrade deserves a release card, not a blind merge.

Write the risk note before the code change

A good upgrade pull request starts with a small risk note. Keep it short and concrete:

Risk note: PromptFoo 0.121.19 touches provider adapters,
red-team token accounting, assertion behavior, and tool-call handling.
Upgrade will be accepted only if golden prompts, red-team cases,
CI JSON output, and cost metrics are compared against the frozen baseline.

This makes reviewers think like release owners. It also helps managers see that AI eval upgrades are not random tooling chores.

Freeze the Eval Baseline Before Upgrade

Before the upgrade, freeze the current behavior. This is the step most teams skip, then they argue for two hours when scores move.

A baseline gives you a reference point. Without it, you only know that “something changed.” With it, you can say exactly what changed, where it changed, and whether the change is acceptable.

Pin the old version and capture evidence

Start with the version currently running in your project. Then capture a baseline run with JSON output, HTML output, and raw logs. Do this before changing the package lockfile.

# 1. Record current package version
npm ls promptfoo --depth=0 || true
npx promptfoo@0.121.18 --version

# 2. Run the current eval suite
mkdir -p artifacts/promptfoo/baseline
npx promptfoo@0.121.18 eval \
  --config promptfooconfig.yaml \
  --output artifacts/promptfoo/baseline/results.json

# 3. Save human-readable evidence
npx promptfoo@0.121.18 view \
  --yes \
  --output artifacts/promptfoo/baseline/report.html

If your current version is not 0.121.18, use your actual installed version. The point is not the number. The point is freezing behavior before the upgrade.

Freeze model and provider inputs too

An eval baseline is weak if the model changes at the same time. Pin these inputs before comparing old and new tool versions:

  • PromptFoo version
  • Model name and provider endpoint
  • Temperature and sampling settings
  • Dataset file
  • Assertion thresholds
  • Environment variables that affect provider selection
  • Secrets names, not secret values

One common mistake is upgrading PromptFoo and changing the model in the same PR. That creates a noisy diff. Keep the tool upgrade separate from model migration unless you intentionally want a combined release review.

Upgrade PromptFoo in a Safe Branch

Now create a safe branch and upgrade the package. I prefer a branch name that makes release intent obvious.

git checkout -b qa/promptfoo-0-121-19-upgrade
npm install --save-dev promptfoo@0.121.19
npm ls promptfoo --depth=0
npx promptfoo@0.121.19 --version

Do not combine this with unrelated prompt rewrites, dataset cleanup, or CI refactoring. A clean dependency upgrade PR is easier to review and easier to roll back.

Review the lockfile like test evidence

The lockfile diff tells you whether the upgrade pulled new transitive dependencies. If the diff is large, call it out. If a security scanner flags packages, resolve that before running the eval comparison.

For AI QA, I also check whether the update changes command behavior. Small CLI changes can break CI scripts that parse output, collect JSON, or attach artifacts to a release card.

Keep a rollback command in the PR

A rollback plan sounds boring until a release gate blocks a production hotfix. Add the rollback command directly in the PR description:

# Rollback plan
npm install --save-dev promptfoo@0.121.18
git checkout package-lock.json
npx promptfoo@0.121.18 eval --config promptfooconfig.yaml

This is basic release hygiene. It also signals maturity. Good SDETs do not just find bugs. They reduce release drama.

Rerun Golden Prompts and Red-Team Cases

The second run uses the upgraded version. Keep everything else frozen.

mkdir -p artifacts/promptfoo/upgrade
npx promptfoo@0.121.19 eval \
  --config promptfooconfig.yaml \
  --output artifacts/promptfoo/upgrade/results.json

npx promptfoo@0.121.19 view \
  --yes \
  --output artifacts/promptfoo/upgrade/report.html

Now compare. Do not only compare the final percentage score. Compare individual failures, assertion messages, token usage, provider responses, and red-team cases.

Use a small diff script

I like a small Node.js comparison script because it turns a fuzzy review into a concrete artifact.

import fs from 'node:fs';

const before = JSON.parse(fs.readFileSync('artifacts/promptfoo/baseline/results.json', 'utf8'));
const after = JSON.parse(fs.readFileSync('artifacts/promptfoo/upgrade/results.json', 'utf8'));

function flatten(results) {
  const rows = results.results?.results || results.results || [];
  return rows.map((r, index) => ({
    index,
    prompt: r.prompt?.raw || r.vars?.question || `case-${index}`,
    pass: Boolean(r.success),
    score: r.score ?? null,
    reason: r.gradingResult?.reason || r.failureReason || '',
  }));
}

const oldRows = flatten(before);
const newRows = flatten(after);
const changes = [];

for (let i = 0; i < Math.max(oldRows.length, newRows.length); i++) {
  const oldCase = oldRows[i];
  const newCase = newRows[i];
  if (!oldCase || !newCase || oldCase.pass !== newCase.pass || oldCase.score !== newCase.score) {
    changes.push({ oldCase, newCase });
  }
}

console.log(JSON.stringify({ changedCases: changes.length, changes }, null, 2));
if (changes.length > 0) process.exitCode = 1;

Save this diff as a CI artifact. It gives reviewers something better than a screenshot. It also helps future you explain why the team accepted or rejected an upgrade.

Review red-team changes manually

The 0.121.19 release mentions red-team token usage and a new goblin strategy. I do not accept red-team changes with only a green CI badge. I sample the generated cases and read the failures.

Ask these questions:

  • Did the upgraded version generate new attack patterns?
  • Did any previously failing prompt injection case now pass?
  • Did any refusal policy case become weaker?
  • Did cost increase because generation now tracks more token usage?
  • Did the report expose enough evidence for a release decision?

This is where human review still matters. AI eval tools speed up testing, but they do not remove accountability.

Inspect Token Usage and Cost Signals

Token usage is not just a finance metric. It is also a reliability signal. If an upgrade changes token accounting, you need to know whether the suite is genuinely more expensive or simply more accurate in how it reports cost.

The 0.121.19 changelog includes token-usage helpers and optional token-usage fields on public types and schemas. That means your reporting pipeline may see new fields. If your CI dashboard expects a fixed JSON shape, test that parser before merging.

Compare cost fields without guessing

Create a tiny script that extracts token fields from both runs. The exact shape depends on your PromptFoo output, so keep the script defensive.

import json
from pathlib import Path

paths = {
    "baseline": Path("artifacts/promptfoo/baseline/results.json"),
    "upgrade": Path("artifacts/promptfoo/upgrade/results.json"),
}

for name, path in paths.items():
    data = json.loads(path.read_text())
    text = json.dumps(data)
    print(name)
    print("  promptTokens mentions:", text.count("promptTokens"))
    print("  completionTokens mentions:", text.count("completionTokens"))
    print("  totalTokens mentions:", text.count("totalTokens"))
    print("  cost mentions:", text.lower().count("cost"))

This is intentionally simple. The goal is to catch output-shape surprises early, not build a perfect analytics system in the upgrade PR.

Set a cost guardrail

For production eval gates, I like a soft threshold and a hard threshold:

  • Soft threshold: more than 15% increase in token usage requires review.
  • Hard threshold: more than 30% increase blocks merge unless approved.
  • Exception: red-team expansion can exceed the threshold if it catches meaningful risk.

Use your own numbers. The important part is making the rule explicit before the upgrade changes the bill.

CI/CD Release Gate for PromptFoo

A PromptFoo upgrade checklist becomes useful when it runs inside CI, not only on one SDET’s laptop. The release gate should answer four questions automatically:

  • Did the eval suite run with the expected PromptFoo version?
  • Did golden prompts keep the same pass/fail behavior?
  • Did red-team failures stay within the accepted risk threshold?
  • Did the pipeline save enough evidence for a reviewer?

GitHub Actions example

Here is a minimal workflow you can adapt:

name: promptfoo-upgrade-gate

on:
  pull_request:
    paths:
      - 'package.json'
      - 'package-lock.json'
      - 'promptfooconfig.yaml'
      - 'evals/**'

jobs:
  eval-upgrade:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Verify PromptFoo version
        run: npx promptfoo --version
      - name: Run PromptFoo eval
        run: |
          mkdir -p artifacts/promptfoo
          npx promptfoo eval \
            --config promptfooconfig.yaml \
            --output artifacts/promptfoo/results.json
      - name: Upload eval evidence
        uses: actions/upload-artifact@v4
        with:
          name: promptfoo-evidence
          path: artifacts/promptfoo

For stronger governance, add a required reviewer when package changes touch PromptFoo, DeepEval, model provider SDKs, or prompt datasets. This is not bureaucracy. This is quality ownership.

Connect the gate to release notes

I wrote about this pattern in Release Notes to Test Plan: QA Workflow That Works. The idea is simple: every meaningful tool release should produce a test plan delta. PromptFoo 0.121.19 is a perfect example because the changelog gives you test areas directly.

If you maintain an eval CI gate, also read AI QA Portfolio Project: Build an Eval CI Gate. That article turns the same release-gate thinking into a portfolio project QA engineers can show in interviews.

India SDET Leadership Angle

In India, many QA teams still split work into “automation testing” and “AI experimentation.” That split is already outdated. Product companies now expect senior SDETs to own quality signals for AI features, not just Selenium or Playwright suites.

If you are targeting ₹25-40 LPA SDET roles, this is the kind of work that separates you from someone who only writes UI scripts. You can explain release risk, build eval gates, review model behavior, and protect the team from silent quality drift.

What managers want to hear

In interviews, do not say, “I know PromptFoo.” Say this:

I built a controlled upgrade gate for our LLM eval framework.
Before upgrading, I froze golden prompts, pinned model settings,
ran before/after comparisons, reviewed red-team changes, and attached
JSON plus HTML evidence to the release PR.

That answer sounds like ownership. It shows you understand testing as a release discipline, not a tool demo.

TCS/Infosys teams can still use this

You do not need a fancy AI-native startup to practice this. Service-company teams can build a small internal sample:

  • Pick a support chatbot or policy-answering use case.
  • Create 30 golden prompts and 10 red-team prompts.
  • Run PromptFoo locally and save JSON output.
  • Upgrade the tool version in a branch.
  • Compare results and write a release note.

That artifact is portfolio-ready. It proves you can test AI systems with discipline.

PromptFoo Upgrade Checklist Template

Use this template for your next PromptFoo upgrade. Copy it into a PR description, Jira ticket, or release card.

Pre-upgrade

  • Confirm current PromptFoo version with npm ls promptfoo --depth=0.
  • Read the official release notes and classify risk areas.
  • Freeze model/provider settings.
  • Freeze golden prompt dataset.
  • Run baseline eval and save JSON plus HTML evidence.
  • Record current pass rate, failing cases, token usage, and known exceptions.

Upgrade branch

  • Create a dedicated branch for the version bump.
  • Upgrade only PromptFoo and required lockfile changes.
  • Verify CLI version in local and CI environments.
  • Run the same eval suite with the upgraded version.
  • Compare case-level pass/fail changes.
  • Review red-team output manually.
  • Check CI artifact upload still works.

Merge decision

  • Accept if pass/fail changes are explained and approved.
  • Block if safety failures improve only because assertions changed unexpectedly.
  • Block if CI output is missing evidence.
  • Escalate if token usage jumps beyond the agreed threshold.
  • Document rollback steps in the PR.

For broader AI eval design, link this checklist with PromptFoo vs DeepEval: QA Guide for LLM Evals and AI Eval Dependency Monitoring for QA Teams. Together, these give you a practical system: choose the eval tool, monitor the dependency, and upgrade it without drama.

FAQ

Should every PromptFoo upgrade block a release?

No. A patch upgrade does not always need a full release freeze. But it should trigger a controlled eval comparison when the changelog touches providers, assertions, red-team behavior, output schemas, or CI commands.

Can I use this checklist for DeepEval?

Yes. The tool commands change, but the method stays the same: freeze the baseline, pin model settings, rerun golden prompts, compare failures, inspect cost, and save evidence. DeepEval’s PyPI page describes it as an LLM evaluation framework, so the same release-gate thinking applies.

What is the minimum dataset size for an upgrade check?

Start with 30 to 50 golden prompts and 10 to 20 negative or red-team cases. Small teams can begin smaller, but the dataset must include real failure examples, not only happy paths.

Should QA own PromptFoo upgrades or developers?

Both should be involved. Developers usually make the package change. QA or SDET owners should define the acceptance criteria, review the eval diff, and sign off on risk. That partnership is the point.

What should I save as release evidence?

Save the version number, release-note link, baseline JSON, upgraded JSON, HTML report, failing-case diff, token usage summary, and rollback command. Evidence beats memory when a production issue appears two weeks later.

Key Takeaways

A PromptFoo upgrade checklist helps QA teams treat LLM eval tooling like release infrastructure. That mindset matters because eval frameworks now influence whether AI features ship, roll back, or require human review.

  • PromptFoo 0.121.19 was published on 14 July 2026 and includes provider, red-team, token-usage, and assertion-related changes.
  • Freeze the baseline before upgrading, including model settings and datasets.
  • Compare case-level results, not only the final pass percentage.
  • Review red-team output manually when generation strategies change.
  • Save JSON, HTML, token, and rollback evidence in CI before merging.

Day 43 lesson: AI QA is release engineering now. The team that controls eval drift controls release confidence.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.