|

AI QA Portfolio Project: Build an Eval CI Gate

ByPromode
AI QA portfolio project eval CI gate featured image with PromptFoo DeepEval and GitHub Actions pipeline

An AI QA portfolio project should prove one thing clearly: you can protect an LLM feature from bad answers before it reaches users. Day 40 of the 100 Days of AI in QA & SDET series turns a normal sample app into a CI-gated project with PromptFoo, DeepEval, and evidence a reviewer can inspect in five minutes.

I see many QA engineers add “AI testing” to the resume after using ChatGPT for test cases. That is not enough anymore. The stronger signal is a repository where every prompt change, model change, and retrieval change must pass measurable checks.

Table of Contents

Contents

Why this AI QA portfolio project matters

The job market is separating AI tool users from AI quality engineers. Tool users ask a model to write test cases. AI quality engineers define failure modes, build datasets, run evaluations, and block unsafe changes in CI. This project helps you show the second skill set.

What a weak AI portfolio looks like

  • A screenshot of ChatGPT generating Selenium code.
  • A README that says “used AI for automation” with no reproducible commands.
  • A demo video where every prompt is manually checked by the author.
  • No dataset, no expected behavior, no CI result, no rollback story.

What a strong AI portfolio looks like

A strong project gives the reviewer a command and a pass/fail result. It shows the dataset, the scoring rule, the CI badge, and a short report with failures explained in plain English. It does not pretend LLM quality is binary. It makes the risk visible.

  1. Pick a narrow AI feature with realistic user intent.
  2. Write 20 to 40 evaluation cases that represent real failures.
  3. Run cheap deterministic checks first: JSON shape, policy phrases, forbidden claims.
  4. Add semantic scoring for answer quality, correctness, and faithfulness.
  5. Block merges when the quality score drops below a threshold.

That is the portfolio signal. It tells a hiring manager that you understand automation, product risk, and AI behavior at the same time.

Project scope: a small assistant with real risk

Do not start with a giant chatbot. Start with a small assistant where the expected behavior is clear. For this AI QA portfolio project, I use a course recommendation assistant for QA learners. The assistant receives a learner profile and recommends a course path.

Why this scope works

Course recommendation looks simple, but it has enough risk to test properly. The assistant can overpromise jobs, recommend the wrong level, ignore prerequisites, invent discounts, or give unsafe career advice. These are exactly the kinds of failures QA should catch.

  • Fresher profile should get manual testing, API basics, and projects before advanced agents.
  • Manual tester profile should get Python or Java basics before framework design.
  • Automation beginner should get CI/CD, Playwright, API testing, and debugging practice.
  • SDET profile can get evaluation frameworks, AI agents, MCP, and observability.
  • Every answer must avoid fake job guarantees and unverifiable salary promises.

The user story

Here is the user story I would put in the README: “As a QA learner, I want the assistant to recommend a realistic learning path based on my current level, so that I do not waste time on tools before prerequisites.” This is enough to drive test cases, prompts, and evaluation metrics.

The acceptance criteria

  • Answer must classify the learner stage correctly.
  • Answer must recommend 3 to 5 next steps, not a 20-item dump.
  • Answer must include at least one project-based practice task.
  • Answer must avoid false guarantees about jobs, salary, or placement.
  • Answer must be concise enough for a learner to act today.

These criteria are boring in the best way. Boring criteria are testable. Testable criteria become portfolio evidence.

Repo structure and test assets

A reviewer should understand your repository without a 30-minute call. Keep the structure small and explicit. I prefer a Python service with a tiny prompt wrapper, plus PromptFoo and DeepEval evaluation folders.

ai-qa-portfolio-eval-gate/
  app/
    assistant.py
    prompt.txt
  evals/
    promptfoo.yaml
    cases.csv
    deepeval_tests.py
    golden_answers.json
  reports/
    latest-summary.md
  .github/
    workflows/
      eval-gate.yml
  README.md
  requirements.txt
  package.json

Minimal assistant code

The assistant code can be intentionally small. The goal is not to build a production LLM platform. The goal is to show testing discipline around an AI behavior.

from pathlib import Path

POLICY = """
You are a QA learning-path assistant.
Give realistic advice. Do not promise jobs, salary, interviews, or placement.
Keep the answer practical and project-focused.
"""

def build_prompt(profile: str) -> str:
    template = Path("app/prompt.txt").read_text(encoding="utf-8")
    return template.replace("{{profile}}", profile).replace("{{policy}}", POLICY.strip())

# In the portfolio repo, call your provider here.
def recommend_path(profile: str) -> str:
    prompt = build_prompt(profile)
    return call_model(prompt)

Evaluation dataset

Your dataset is the core artifact. A good evaluator with bad cases is still a weak project. Start with 20 cases and grow to 40 after you see real failures.

id,profile,expected_stage,forbidden
F01,"I am a fresher and know basic computers only",fresher,"guaranteed job"
M01,"I test manually for 4 years and want automation",manual_to_automation,"skip programming"
A01,"I know Selenium basics but fail in framework interviews",automation_beginner,"learn AI agents first"
S01,"I am an SDET and need to test LLM features",sdet,"only use ChatGPT manually"

Notice the forbidden column. I like this because it converts vague career advice risk into a check. You can add more columns later for expected topics, tone, JSON schema, or retrieval source.

PromptFoo gate for prompt regressions

PromptFoo is useful when you want prompt regression tests that run from the command line. The npm registry describes promptfoo as an “LLM eval & testing toolkit,” and the latest metadata I checked reports version 0.121.19 with an MIT license. That makes it easy to add to a portfolio repo without a heavy setup story.

Install and run

npm init -y
npm install --save-dev promptfoo
npx promptfoo --version

PromptFoo configuration

The first gate should catch obvious regressions: missing learning path, fake guarantee, wrong learner level, and answers that are too generic. Keep the assertions readable. A hiring manager may not know every eval tool, but they can understand a good test case.

description: QA learner recommendation eval
prompts:
  - file://app/prompt.txt
providers:
  - id: openai:gpt-4o-mini
    config:
      temperature: 0

tests:
  - vars:
      profile: "I am a fresher and know basic computers only"
    assert:
      - type: contains-any
        value: ["manual testing", "testing fundamentals", "test cases"]
      - type: not-contains
        value: "guaranteed job"
      - type: llm-rubric
        value: "The answer recommends beginner-friendly QA steps and does not jump directly to advanced AI agents."

  - vars:
      profile: "I test manually for 4 years and want automation"
    assert:
      - type: contains-any
        value: ["programming", "Python", "Java", "API testing"]
      - type: not-contains
        value: "skip programming"
      - type: llm-rubric
        value: "The answer gives a practical manual-to-automation transition path."

What the gate catches

Prompt regressions usually look small in code review. Someone changes two lines in a system prompt to make the answer shorter. Suddenly the assistant stops mentioning projects, prerequisites, or safety policy. A prompt eval catches that before the bad version becomes the default.

  • Missing required concept.
  • Forbidden phrase or claim.
  • Wrong learner stage.
  • Answer too vague to act on.
  • Policy violation after a prompt cleanup.

DeepEval gate for semantic quality

DeepEval is useful when the project needs Python-native tests and richer scoring. PyPI currently summarizes deepeval as “The LLM Evaluation Framework,” with latest package metadata showing version 4.1.1 at the time of this run. For a QA portfolio, that matters because the test code looks familiar to automation engineers who already know pytest.

Install dependencies

python -m venv .venv
source .venv/bin/activate
pip install deepeval pytest

A DeepEval test file

This sample test uses a simple rubric style. In a real repo, I would also include actual output snapshots in reports/latest-summary.md so the reviewer can see exactly what failed.

import pytest
from deepeval.test_case import LLMTestCase
from deepeval.metrics import GEval
from deepeval import assert_test
from app.assistant import recommend_path

quality_metric = GEval(
    name="qa_learning_path_quality",
    criteria=(
        "Score whether the answer identifies the learner stage, "
        "recommends realistic next steps, includes project practice, "
        "and avoids job or salary guarantees."
    ),
    threshold=0.75,
)

@pytest.mark.parametrize("profile", [
    "I am a fresher and know basic computers only",
    "I test manually for 4 years and want automation",
    "I know Selenium basics but fail in framework interviews",
])
def test_learning_path_quality(profile):
    actual = recommend_path(profile)
    test_case = LLMTestCase(input=profile, actual_output=actual)
    assert_test(test_case, [quality_metric])

Why use both PromptFoo and DeepEval?

For a portfolio, using both tools is not about tool collecting. It shows you understand test layers. PromptFoo gives a fast declarative gate for prompt changes. DeepEval gives Python-native tests for deeper semantic quality and custom metrics. Together they resemble how modern automation suites combine API checks, UI checks, and contract tests.

GitHub Actions CI gate

A portfolio project becomes credible when the tests run outside your laptop. GitHub’s Actions documentation includes build-and-test examples for Python projects at docs.github.com, and the pattern maps well to AI evaluation gates.

The CI workflow

name: ai-eval-gate

on:
  pull_request:
  push:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          npm ci
          pip install -r requirements.txt

      - name: Run PromptFoo regression gate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: npx promptfoo eval --config evals/promptfoo.yaml --output reports/promptfoo.json

      - name: Run DeepEval semantic gate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest evals/deepeval_tests.py -q

      - name: Upload evaluation evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ai-eval-report
          path: reports/

Threshold policy

Do not set thresholds randomly. Put the policy in the README so the reviewer sees your thinking. For example: PRs fail when any safety assertion fails, or when semantic quality drops below 0.75, or when more than two learner-stage classifications are wrong.

  • Safety checks are hard gates.
  • Quality scores can start at 0.75 and tighten after the dataset matures.
  • New prompts must add at least two regression cases for every fixed defect.
  • CI artifacts must include failed input, actual answer, expected behavior, and score.

Cost control

LLM evals cost money, so keep the default CI suite small. Run 20 smoke evals on every PR and the full 100-case suite nightly. This shows maturity. You are not just adding tests; you are designing a sustainable quality loop.

Portfolio evidence hiring managers trust

A GitHub repo alone is not enough. Hiring managers are busy. They may open your project for three minutes between interviews. Your job is to make the evidence obvious.

README proof section

Add a section near the top named “Evidence.” Do not hide the strongest artifacts under folders. Link to the CI workflow, latest report, sample failing case, and a short design note.

## Evidence

- CI gate: `.github/workflows/eval-gate.yml`
- Prompt regression suite: `evals/promptfoo.yaml`
- Semantic quality tests: `evals/deepeval_tests.py`
- Latest report: `reports/latest-summary.md`
- Sample failure: `reports/sample-failure.md`

### Current quality policy
- Safety assertions: 100% pass required
- Semantic quality threshold: 0.75
- Smoke suite: 20 cases on every PR
- Full suite: 100 cases nightly

Sample failure report

The failure report is where QA thinking becomes visible. Do not only write “test failed.” Explain risk, impact, likely cause, and recommended fix. This sounds obvious, but very few portfolios do it.

# Sample Failure: Manual tester profile skipped programming basics

Input:
I test manually for 4 years and want automation.

Actual output problem:
The assistant recommended AI agents and LangGraph before programming basics.

Risk:
Learner may skip the foundation needed for real automation interviews.

Likely cause:
Prompt over-prioritized new AI topics and ignored learner stage.

Fix:
Update stage policy and add regression case M02 for manual-to-automation path.

What to show in your resume

Use one crisp bullet. Do not write “worked on AI testing.” Write the outcome and the mechanics.

  • Built an AI QA evaluation gate for an LLM course assistant using PromptFoo, DeepEval, pytest, and GitHub Actions.
  • Created 40 learner-profile eval cases covering stage classification, safety claims, and practical recommendation quality.
  • Blocked prompt regressions in CI and published evaluation artifacts for review.

This bullet is stronger than a generic AI certificate line because it proves implementation.

India career context for SDETs

In India, many QA resumes now say Selenium, Postman, API automation, and a few AI tools. The difference between service-company screening and product-company SDET interviews is evidence. TCS or Infosys project experience can open the first door, but product companies usually ask what you built, how it failed, and how you debugged it.

I am careful with salary claims because public numbers change fast and depend on company, location, and interview strength. The practical point is stable: an SDET who can combine automation, CI/CD, and LLM evaluation has a better story than someone who only says “I use ChatGPT for testing.”

What interviewers can ask from this project

  • How did you choose the evaluation threshold?
  • What failures did PromptFoo catch that unit tests missed?
  • Where did DeepEval produce noisy scores, and how did you reduce noise?
  • How do you control LLM eval cost in CI?
  • What would you monitor after release?

How to answer like an SDET

Answer with trade-offs. For example: “I keep safety checks deterministic and strict. I allow semantic quality scores to be probabilistic, so I trend them over time and review failures manually before tightening the gate.” That answer sounds like ownership, not tool excitement.

If you are learning this path, keep a companion note with commands, screenshots, failed cases, and fixes. That note becomes interview preparation. It also becomes content for LinkedIn without exposing private company work.

Common mistakes to avoid

This AI QA portfolio project can become messy if you try to impress everyone. Keep it narrow. Hiring managers prefer a clean project with strong evidence over a giant repo with five half-working agents.

Mistake 1: testing only happy paths

Happy paths make demos look good. Edge cases make QA credible. Add cases where the learner asks for a guaranteed job, asks to skip basics, asks for a shortcut, or provides conflicting experience details.

Mistake 2: using only LLM-as-judge

LLM-as-judge is useful, but it should not be your only check. Add deterministic assertions for forbidden claims, schema, required terms, and answer length. Cheap checks catch many serious failures.

Mistake 3: hiding the prompt

If this is a portfolio repo, show the prompt. A secret prompt makes the project harder to review. If you worry about API keys or private data, remove secrets and keep the prompt readable.

Mistake 4: no failure examples

A perfect green dashboard is less believable than a repo that shows one real failure and the fix. QA is about finding risk. Show the bug, the regression case, the prompt change, and the next CI pass.

Mistake 5: no product framing

Do not describe the project as “I tried PromptFoo.” Describe it as “I protected a learner recommendation assistant from unsafe and low-quality answers.” Tools support the story. Product risk is the story.

For related ScrollTest reading, see the PromptFoo vs DeepEval QA guide, the updated LLM eval comparison, and Day 32 on LLM regression testing.

Key takeaways for your AI QA portfolio project

The conclusion is simple: an AI QA portfolio project should show a working quality system, not just AI enthusiasm. If the reviewer can run your evals, inspect your dataset, and understand your gate, you are ahead of most generic AI testing resumes.

  • Pick a small AI feature with clear risk and acceptance criteria.
  • Use PromptFoo for fast prompt regression checks.
  • Use DeepEval for Python-native semantic quality tests.
  • Run the gate in GitHub Actions and upload evidence artifacts.
  • Write failure reports like a QA owner, not like a tool collector.

For Day 40, your task is practical: create the repo structure, add 10 initial eval cases, wire one PromptFoo gate, wire one DeepEval test, and push a CI run. Tomorrow, improve the dataset from real failures.

FAQ

Is PromptFoo or DeepEval better for this portfolio project?

Use both if you can explain the difference. PromptFoo is great for declarative prompt regression checks. DeepEval is strong when you want Python tests and custom evaluation logic. If you must choose one, pick the tool that matches your strongest programming stack.

How many eval cases should I include?

Start with 20. That is enough to show structure without making the project slow. Add more cases from failures. A 40-case suite with clear categories is better than 200 random prompts.

Can I build this without paid LLM APIs?

Yes, but keep the README honest. You can design the repo around local models or mock outputs for some checks. For semantic evaluation, you may still need a judge model. If cost is a concern, run the full suite manually or nightly instead of on every commit.

Should manual testers build this project?

Yes, if you already understand testing fundamentals and can write basic Python or JavaScript. If not, learn API testing and programming basics first. AI QA still needs core QA thinking.

What should I post on LinkedIn after building it?

Post the problem, the repo structure, one failed eval, and the CI badge. Avoid vague claims. A screenshot of a blocked unsafe prompt regression is stronger than a motivational paragraph about AI changing testing.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.