| |

LLM QA Release Cards for PromptFoo and DeepEval

ByPromode
LLM QA release cards workflow for PromptFoo and DeepEval upgrades

LLM QA release cards are how I stop AI testing tool upgrades from becoming random Slack messages, vague Jira tickets, or risky Friday merges. This product update shows the ScrollTest workflow I use to convert PromptFoo and DeepEval releases into owner-assigned checks with datasets, scorer versions, evidence links, and a clear release decision.

PromptFoo and DeepEval are moving fast. The current npm registry entry lists PromptFoo 0.121.19, npm reports 1,743,398 PromptFoo downloads in the last month, and PyPI lists DeepEval 4.1.3. That pace is good for innovation, but it is painful when a QA team has no release evidence system.

Table of Contents

Contents

Why LLM QA release cards exist

I see a pattern in teams adopting AI testing tools. The first month feels exciting because every prompt, scorer, and dataset looks like progress. By the third month, nobody can answer a basic question: which tool version produced the evidence we trusted for the last release?

That is the gap ScrollTest eval-release cards are built to close. A card is not a blog note. It is a small release-control object that links one tool update to one QA owner, one dataset slice, one scorer version, one CI run, and one decision.

The problem is not tool quality

PromptFoo and DeepEval are serious tools. PromptFoo describes itself as an LLM eval and testing toolkit on the npm package page. DeepEval describes itself as an LLM evaluation framework on PyPI. Both projects are useful when a QA engineer treats them like test infrastructure instead of demo software.

The weak link is usually the release process around the tool. A team upgrades a package, reruns a few evals, sees green output, and assumes the application is safe. That is too thin for customer-facing AI features.

Green checks need evidence

A green evaluation run can hide scorer drift, dataset leakage, flaky model output, provider configuration changes, or threshold changes. I wrote about this mindset in AI Testing Evidence: Stop Trusting Green Checks. The short version is simple: a pass without traceable evidence is only a screenshot with confidence problems.

LLM QA release cards make the evidence visible before the team accepts a tool upgrade. They force the conversation from “did the command pass?” to “what changed, who reviewed it, and which evidence supports the decision?”

The release-card mindset

I use release cards for the same reason I use test plans for major Selenium or Playwright upgrades. The tool version is part of the product risk. If the scorer changes, the meaning of “pass” may change. If the dataset changes, the coverage story changes. If the provider changes, the result distribution can change.

A card makes those variables explicit. It is small enough for daily work and structured enough for audit conversations with engineering managers, product owners, and SRE teams.

What shipped in ScrollTest

The new ScrollTest workflow converts PromptFoo and DeepEval release activity into LLM QA release cards. Each card has an owner, dataset scope, scorer version, evidence links, and release decision fields. It is designed for QA teams that already run evals but still manage release decisions in chat.

This is a product update, not a generic checklist. The workflow is built around the way SDETs actually work: release notes come in, the CI job runs, evidence gets attached, and somebody has to say “approved,” “blocked,” or “monitor only.”

Card fields added

The first version of the workflow standardizes these fields:

  • Tool: PromptFoo, DeepEval, or another eval framework.
  • Version: exact package version, for example PromptFoo 0.121.19 or DeepEval 4.1.3.
  • Owner: one QA engineer or SDET accountable for review.
  • Dataset: dataset name, commit SHA, or storage path.
  • Scorer: scorer name, scorer version, and threshold.
  • Evidence: CI run, HTML report, JSON artifact, dashboard, or pull request.
  • Decision: approve, block, monitor, or rollback.

Why owner assignment matters

Release-note tracking fails when it is owned by “the QA team.” A named owner changes the behavior. One person reads the release note, checks the dataset, confirms scorer stability, and writes the decision. The card can be reviewed by others, but accountability is not vague.

For a team of 8 SDETs, I would rotate ownership weekly. For a smaller team of 2 QA engineers, I would assign ownership by application area: support bot evals, search RAG evals, checkout assistant evals, and internal coding-agent evals.

Evidence links are first-class

Every card requires evidence links. A useful evidence link is not “CI passed.” It points to the exact CI run, the exact config file, the exact dataset commit, and the exact eval output. If the evidence is spread across 4 systems, the card links all 4 systems.

This is the same thinking behind AI Test Evidence in CI/CD Release Gates. If a release gate cannot explain itself, it will fail when production pressure arrives.

PromptFoo and DeepEval release context

PromptFoo and DeepEval are both active, widely watched projects. The PromptFoo GitHub repository shows 23,573 stars and 2,119 forks at the time of this run. The DeepEval GitHub repository shows 17,101 stars and 1,712 forks. Those numbers do not prove quality by themselves, but they prove that changes can reach a large user base quickly.

That is why release cards are useful. A small version bump in an eval tool can influence many teams’ release decisions if they run that tool in CI.

PromptFoo version and usage signals

The npm registry currently reports PromptFoo version 0.121.19. The npm downloads API reports 1,743,398 downloads between 2026-06-24 and 2026-07-23. I use those two numbers as release-card inputs: version and adoption signal.

I do not treat downloads as quality. I treat downloads as blast radius. If a package with more than 1.7 million monthly downloads changes behavior, I want my QA process to preserve the exact evidence from the upgrade.

DeepEval version and project context

PyPI currently reports DeepEval version 4.1.3 with the summary “The LLM Evaluation Framework.” The DeepEval repository is under confident-ai/deepeval. For QA teams, the important point is not the marketing line. The important point is that scorer behavior, metrics, and framework defaults can become part of your release gate.

When a framework becomes part of a release gate, it deserves release control. I want the same discipline for LLM eval tools that I expect for Playwright, Selenium Grid, Docker images, and database migration tooling.

How this relates to existing ScrollTest labs

If your team is still comparing tools, start with PromptFoo vs DeepEval: QA Guide for LLM Tests. If you already run LLM regression suites, read LLM Regression Testing for QA: Day 32 Lab. The release-card workflow sits after those two steps. It assumes you already know what to test and now need a repeatable release review system.

The release card schema

A good LLM QA release card is boring on purpose. It avoids long descriptions and captures fields that influence release safety. If a field does not change a decision, I do not put it on the card.

Minimum card schema

Here is the minimum schema I recommend for a QA team:

  1. Tool version: package name, version, source, and release date when available.
  2. Change summary: one paragraph written by the owner, not copied blindly from release notes.
  3. Risk area: prompt drift, scorer drift, dataset drift, provider behavior, reporting, or CLI behavior.
  4. Dataset scope: which dataset slice was used and why it is enough for this change.
  5. Scorer scope: scorer name, scorer version, metric threshold, and any custom rubric.
  6. Evidence links: CI run, report, PR, artifact, and failure triage notes.
  7. Decision: approved, blocked, monitor only, or rollback.

Risk labels that work

I prefer 6 risk labels because they are easy to remember during review:

  • Prompt drift: prompts changed or provider output shifted.
  • Scorer drift: metric calculation, rubric, or threshold changed.
  • Dataset gap: release affects cases not covered by the current dataset.
  • Provider risk: OpenAI, Anthropic, Gemini, or local model behavior changes matter.
  • CLI/reporting risk: the tool runs differently or reports results differently.
  • CI/CD risk: cache, retry, parallelism, secrets, or artifact retention changed.

What not to track

Do not turn the card into a wiki page. Do not paste the full changelog. Do not add 14 optional fields because one audit meeting asked for them. The card should help a QA owner make a decision in 10 minutes.

If the team needs a longer postmortem, link it as evidence. Keep the card compact.

The workflow from release note to QA decision

The ScrollTest workflow is intentionally simple. It turns a release note into a card, runs a defined evidence check, and asks for a decision. The workflow has 7 steps.

Step-by-step release flow

  1. Detect: identify a new PromptFoo, DeepEval, or eval-tool release.
  2. Create: generate a release card with package name, version, and source link.
  3. Assign: set one QA owner and one backup reviewer.
  4. Classify: choose the risk label: scorer, dataset, prompt, provider, CLI, or CI/CD.
  5. Run: execute the agreed eval suite with pinned dependencies.
  6. Attach: link CI run, JSON output, report, and pull request.
  7. Decide: approve, block, monitor, or rollback.

Decision rules

I use these decision rules:

  • Approve: no threshold change, no unexplained regression, evidence links complete.
  • Block: scorer output changed and the owner cannot explain why.
  • Monitor: low-risk reporting change, but evidence is still worth storing.
  • Rollback: CI behavior changes or false passes appear after upgrade.

The key is consistency. A release card should not depend on the mood of the reviewer.

Where ScrollTest helps

ScrollTest keeps the card close to the release decision. The card is not just a task. It is a QA evidence object. The owner can attach artifacts, link a source release, record scorer versions, and add the final decision without hunting across Slack, GitHub Actions, and Google Docs.

TypeScript example: generating a card

Here is a small TypeScript example that turns package metadata into a release card object. It is not tied to one CI provider. You can run the same idea in GitHub Actions, GitLab CI, Jenkins, or a local release-check command.

type RiskLabel =
  | "prompt_drift"
  | "scorer_drift"
  | "dataset_gap"
  | "provider_risk"
  | "cli_reporting"
  | "cicd_risk";

type ReleaseDecision = "approve" | "block" | "monitor" | "rollback";

type EvalReleaseCard = {
  tool: "promptfoo" | "deepeval";
  version: string;
  sourceUrl: string;
  owner: string;
  dataset: {
    name: string;
    commitSha: string;
    caseCount: number;
  };
  scorer: {
    name: string;
    version: string;
    threshold: number;
  };
  risk: RiskLabel;
  evidence: {
    ciRunUrl: string;
    reportUrl: string;
    pullRequestUrl: string;
  };
  decision: ReleaseDecision;
  notes: string;
};

export function buildPromptFooCard(): EvalReleaseCard {
  return {
    tool: "promptfoo",
    version: "0.121.19",
    sourceUrl: "https://registry.npmjs.org/promptfoo/latest",
    owner: "qa-llm-owner",
    dataset: {
      name: "support-bot-regression-v3",
      commitSha: process.env.DATASET_SHA ?? "unknown",
      caseCount: 120
    },
    scorer: {
      name: "answer-relevance",
      version: "2026.07",
      threshold: 0.82
    },
    risk: "scorer_drift",
    evidence: {
      ciRunUrl: process.env.CI_RUN_URL ?? "missing",
      reportUrl: process.env.EVAL_REPORT_URL ?? "missing",
      pullRequestUrl: process.env.PR_URL ?? "missing"
    },
    decision: "monitor",
    notes: "No unexplained regressions. Keep 7-day watch on failed edge cases."
  };
}

Why I store case count

The caseCount field matters. If yesterday’s run used 450 cases and today’s release card used 120 cases, the release decision needs that context. A smaller dataset can be valid for a targeted change, but the card should say so.

Why I store scorer version

Scorer version is the field many teams miss. If the evaluator model, metric code, rubric, or threshold changes, your pass rate can move even when the product did not change. That is why the scorer field is non-negotiable.

CI/CD release gates for eval tooling

LLM eval tools should not float silently in CI. If your release gate uses PromptFoo or DeepEval, pin the version, store the report, and publish the card as part of the pipeline output.

GitHub Actions example

This simplified GitHub Actions pattern pins the package version and uploads evidence. Adjust the commands for your repo, but keep the evidence discipline.

name: llm-eval-release-card

on:
  pull_request:
    paths:
      - "evals/**"
      - "package.json"
      - "requirements.txt"

jobs:
  eval-release-card:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - name: Install pinned PromptFoo
        run: npm install promptfoo@0.121.19
      - name: Run evals
        run: npx promptfoo eval --config evals/promptfoo.yaml --output reports/promptfoo.json
      - name: Upload evidence
        uses: actions/upload-artifact@v4
        with:
          name: llm-eval-evidence
          path: reports/promptfoo.json

Do not ignore artifacts

Many teams run evals but lose the artifact after 7 or 14 days. That is a process bug. If the release decision may be questioned later, store the JSON output and report link long enough for your release audit window.

Use release gates carefully

I do not recommend blocking every deploy on one LLM score. LLM outputs are probabilistic and evaluators can be imperfect. I prefer layered gates: deterministic unit tests, API contract tests, Playwright flows, eval score thresholds, and human review for high-risk changes.

India QA team context

For India-based QA teams, this workflow is also a career signal. SDETs who can explain eval evidence, scorer drift, dataset gaps, and release gates stand out in product companies. I see AI testing roles asking for more than Selenium scripts now.

A mid-level automation engineer in India may already know Playwright, Java, Python, CI/CD, and SQL. The next jump is not only “learn AI.” The next jump is owning AI quality decisions with evidence. In strong product teams, that skill can move a candidate toward senior SDET, QA architect, or AI quality engineer tracks where compensation commonly moves far above entry-level manual testing ranges.

TCS/Infosys versus product-company expectations

Service-company projects often reward execution discipline: run the regression pack, update the status, follow the release checklist. Product companies add ownership pressure: explain risk, defend evidence, and improve the release gate. LLM QA release cards help bridge that gap because they force both execution and ownership.

What to show in interviews

If I were interviewing for an AI QA role, I would show one release card with these 5 artifacts:

  • A PromptFoo or DeepEval config file.
  • A dataset sample with 20 to 50 representative cases.
  • A CI run link with raw output.
  • A scorer definition with threshold reasoning.
  • A release decision note explaining approve, block, or monitor.

That portfolio beats a generic “I know AI testing” statement. It shows judgment.

Implementation checklist for QA leads

The fastest way to adopt this workflow is to start with one application and one eval tool. Do not roll it out to every AI feature on Monday morning. Pick the chatbot, RAG search, test-case generator, or coding-agent workflow where a wrong answer has visible customer or engineering cost.

I recommend a 2-week pilot. In week 1, create cards manually for every PromptFoo or DeepEval version change that touches the pilot area. In week 2, automate card creation from package metadata and CI output. By the end of 10 working days, the team should know whether the workflow improves release confidence or adds noise.

Week 1 manual setup

Start with a shared template and a strict definition of done. The owner should not close the card until the version, dataset, scorer, CI run, and decision fields are complete. A missing field is not a small formatting issue. It means the release evidence is incomplete.

  • Create 1 card template in ScrollTest.
  • Select 1 dataset slice with 20 to 50 high-signal examples.
  • Pin 1 scorer configuration and write down the threshold.
  • Run the eval suite from CI, not only from a laptop.
  • Attach the report and write a 3-line decision note.

Week 2 automation setup

Automation should remove typing, not judgment. The script can fetch the package version, source URL, CI run URL, artifact URL, and dataset commit. The QA owner still classifies risk and writes the final decision. That separation keeps the workflow fast without pretending that eval governance is fully automatic.

A simple automation target is enough: when package-lock.json, requirements.txt, or eval config changes, create or update a release card. If no eval tool version changed, the card can stay in monitor mode or be skipped based on team policy.

Definition of done

For a release-card pilot, I use this definition of done:

  1. The card links to the source package page or repository.
  2. The CI evidence is reproducible from a commit SHA.
  3. The dataset and scorer versions are visible without asking the owner.
  4. The decision is written in plain English.
  5. The card can be understood by a QA lead who did not attend the standup.

If the card passes those 5 checks, it is useful. If it fails them, the workflow is still theatre.

Key takeaways

LLM QA release cards give QA teams a practical way to control PromptFoo and DeepEval upgrades without slowing every release meeting.

  • PromptFoo 0.121.19 and DeepEval 4.1.3 are concrete examples of why eval tooling needs version-aware evidence.
  • A card should record owner, dataset, scorer version, evidence links, and decision.
  • Downloads and GitHub stars are adoption signals, not proof that an upgrade is safe.
  • CI/CD gates need stored artifacts, not only green checks.
  • For SDETs in India, release-card ownership is a strong AI quality engineering portfolio artifact.

If your team already runs LLM evals, the next improvement is not another dashboard. It is a small release card that makes the decision traceable.

FAQ

What is an LLM QA release card?

An LLM QA release card is a structured record for an eval-tool release. It captures the tool version, owner, dataset, scorer version, evidence links, and release decision. It helps QA teams explain why a PromptFoo, DeepEval, or similar upgrade was approved or blocked.

Do I need release cards for every PromptFoo or DeepEval update?

No. Use them for updates that affect CI, scorer behavior, reporting, datasets, provider integrations, or release gates. For low-risk documentation-only changes, a monitor-only card may be enough.

Can this work with tools beyond PromptFoo and DeepEval?

Yes. The same schema works for OpenAI Evals, custom pytest eval suites, RAG evaluation scripts, or agent-testing harnesses. The core fields stay the same: owner, version, dataset, scorer, evidence, and decision.

Should an eval failure automatically block production?

Not always. A severe regression in a high-risk flow should block. A small score shift in a low-risk workflow may move to monitor. The card makes that judgment explicit instead of hiding it behind a green or red CI icon.

How many cases should the dataset include?

There is no universal number. For a smoke release card, 20 to 50 high-signal cases can work. For a major scorer or provider change, I prefer a larger regression slice with production-like examples and clear expected behavior.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.