Building an Agentic Testing Pipeline: AI That Plans, Executes, and Triages Tests

Agentic testing = AI agents that autonomously plan, execute, and analyze tests. Not scripts that run in order — agents that think about what to test next based on results.

🤖 Learning AI-powered testing? Go hands-on with LLM, RAG, and AI-agent testing in the AI-Powered Testing Mastery course at The Testing Academy.

Agentic vs Traditional Test Execution

Traditional	Agentic
Run predefined test list	Agent decides what to test next
Fixed test order	Dynamic prioritization based on risk
Pass/fail report	Root cause analysis + fix suggestion
Human triages failures	Agent triages and files bugs
Manual maintenance	Self-healing locators

Building an Agentic Pipeline

class AgenticTestPipeline:
    def __init__(self, test_suite, llm, tools):
        self.suite = test_suite
        self.llm = llm
        self.tools = tools

    async def run(self, code_diff: str):
        # Step 1: Analyze code changes
        risk_areas = await self.llm.analyze(
            f"What test areas are impacted by this diff?\n{code_diff}"
        )

        # Step 2: Prioritize tests
        priority_tests = await self.llm.prioritize(
            f"Given risk areas {risk_areas}, which tests to run first?"
        )

        # Step 3: Execute prioritized tests
        results = []
        for test in priority_tests:
            result = await self.tools.run_test(test)
            results.append(result)

            # Step 4: If failure, analyze immediately
            if result.failed:
                analysis = await self.llm.analyze_failure(
                    test_code=result.test_code,
                    error=result.error,
                    trace=result.trace
                )
                # Step 5: Auto-file bug if real issue
                if analysis.is_real_bug:
                    await self.tools.create_bug_report(analysis)

        return AgenticReport(results, risk_areas)

Self-Healing Locators

class SelfHealingLocator:
    def __init__(self, primary, fallbacks, llm):
        self.primary = primary
        self.fallbacks = fallbacks
        self.llm = llm

    async def find(self, page):
        # Try primary locator
        try:
            el = page.locator(self.primary)
            if await el.count() > 0:
                return el
        except: pass

        # Try fallbacks
        for fb in self.fallbacks:
            try:
                el = page.locator(fb)
                if await el.count() > 0:
                    # Log healed locator for review
                    self._log_heal(self.primary, fb)
                    return el
            except: pass

        # Ask LLM to find element
        dom = await page.content()
        suggestion = await self.llm.find_element(dom, self.primary)
        return page.locator(suggestion)