Who Tests AI-Generated Code? Building an Automated Security Review Pipeline

40% of code written last year was generated by AI. 48% of that code contains security vulnerabilities. Somebody has to verify all of it before it reaches production. Here is how to build an automated security review pipeline specifically for AI-generated code.

🤖 Learning AI-powered testing? Go hands-on with LLM, RAG, and AI-agent testing in the AI-Powered Testing Mastery course at The Testing Academy.

Why AI Code Needs Different Testing

• Hallucinated APIs — AI imports packages that do not exist
• Insecure defaults — AI skips input validation, uses weak crypto
• Logic plausibility — code looks correct but implements wrong business rules
• Dependency confusion — AI suggests deprecated or vulnerable packages

The 5-Stage Security Pipeline

1. Dependency audit — verify all imports exist and are not vulnerable (npm audit, pip audit)
2. SAST scan — static analysis for injection, XSS, insecure crypto (Semgrep, CodeQL)
3. Property-based testing — random input generation to find edge cases (Hypothesis, fast-check)
4. DAST scan — runtime security testing against deployed endpoints (OWASP ZAP)
5. Human review — business logic validation that no tool can automate

GitHub Actions Security Pipeline

name: AI Code Security Review
on: pull_request

jobs:
  dependency-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm audit --audit-level=high
      - run: npx better-npm-audit audit

  sast-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: returntocorp/semgrep-action@v1
        with:
          config: p/security-audit p/owasp-top-ten

  property-testing:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx vitest run tests/property/