|

MCP Server Smoke Tests After SDK Upgrades

ByPromode
MCP server smoke tests checklist after SDK upgrades

MCP server smoke tests are now a release gate, not a nice-to-have checklist. The MCP Python SDK v2.0.0 release moved the ecosystem to the 2026-07-28 protocol revision, kept older 2025 clients working from the same server, and changed enough client/server behavior that a casual package bump can break an AI QA stack in strange ways.

I see this pattern with AI tooling every month: the demo still works, but the agent silently loses one tool, a streaming endpoint changes shape, or a stdio server starts printing noisy logs into the protocol channel. This guide gives you a practical smoke-test suite you can run after every MCP SDK upgrade before your QA agents touch Jira, test data, CI logs, databases, or production-like environments.

Table of Contents

Contents

Why MCP Server Smoke Tests Matter After SDK Upgrades

The Model Context Protocol is becoming the connector layer between AI agents and the systems QA teams already use: test management, CI, feature flags, logs, databases, browser automation, and internal developer portals. When that connector breaks, the agent does not always crash loudly. Sometimes it makes worse decisions with less context.

That is why MCP server smoke tests need to sit next to your normal package upgrade checks. Treat your MCP server like an API gateway for agents. A broken endpoint can waste a single request. A broken MCP server can make an agent run the wrong test suite, use stale test data, or skip a failure pattern it should have found.

Smoke tests catch integration breakage before full regression

A smoke test is not a full contract suite. It is the fast gate that answers one question: can this server still perform its core job after the SDK changed?

For MCP servers, that core job usually has six parts:

  • Start the server without protocol noise or import errors.
  • Connect through the transports your clients use.
  • Discover the expected tools, resources, and prompts.
  • Execute one safe happy-path call per critical tool.
  • Reject unsafe or unauthenticated requests correctly.
  • Emit logs, traces, and errors that QA can debug in CI.

MCP has multiple moving parts

MCP is not just a single REST endpoint. A real server can expose tools, resources, prompts, subscriptions, auth, streamable HTTP, stdio, and version negotiation. It may talk to Jira, GitHub, S3, a test database, or a Playwright runner behind the scenes.

The official MCP architecture documentation describes the host, client, and server split. That split is powerful, but it also means bugs can hide in the boundaries. The client may negotiate a protocol version you did not expect. The server may expose a tool schema with a subtle breaking change. A subprocess may print to stdout and corrupt stdio traffic.

If you already test APIs, the mindset is familiar. I would connect this guide with ScrollTest’s MCP Server Testing Checklist for SDETs and MCP for QA Engineers. Those cover the bigger testing strategy. This article focuses on the upgrade gate: the tests you run within minutes of bumping the SDK.

What Changed in MCP Python SDK v2.0.0

The trigger for this checklist is the MCP Python SDK v2.0.0 release. The official GitHub release notes say v2.0.0 was published on 28 July 2026, supports the 2026-07-28 protocol revision, and still serves earlier protocol revisions from the same server.

The same release notes also call out several changes that matter to QA:

  • pip install mcp now installs the 2.x line.
  • v1.x is in maintenance mode and receives only security fixes.
  • FastMCP is now MCPServer, with a first-class Client.
  • The SDK supports stateless requests with no handshake for the 2026-07-28 revision.
  • server/discover, subscriptions/listen, and multi-round-trip requests are part of the new protocol surface.
  • OpenTelemetry tracing ships on by default.
  • Streamable HTTP servers reject request bodies over 4 MiB with HTTP 413.

Those are not cosmetic changes. They change how you connect, how you discover capabilities, how you debug, and how you test compatibility.

Community signals are strong, but popularity is not safety

The MCP Python SDK repository had 23,805 GitHub stars and 3,720 forks when I checked the GitHub API for this article. PyPI lists mcp version 2.0.0 as the current Python package. Those are healthy adoption signals, but they do not remove the need for testing.

Protocol revision changes create hidden compatibility risk

The v2 release note says one server can serve both the 2026-07-28 revision and earlier 2025-era clients. That is great for migration, but it creates a test obligation. You need at least one smoke test with the newest client and one smoke test with the oldest client version you still support.

Here is the practical QA translation:

  1. Pin the current production client and server versions.
  2. Upgrade the server SDK in a branch.
  3. Run smoke tests with the new client.
  4. Run compatibility smoke tests with the old client.
  5. Only then run deeper contract and agent workflow tests.

The official migration guide is the right source for breaking changes. Your smoke suite should not duplicate that guide line by line. It should convert the important migration risks into executable checks.

What to Smoke Test Before Agents Use the Server

The best MCP server smoke tests are boring, fast, and ruthless. They do not try to test every edge case. They prove that the server is safe enough for a deeper pipeline.

I split the smoke suite into five buckets:

  • Startup: imports, configuration, environment variables, dependency injection.
  • Discovery: tools, resources, prompts, schemas, names, and descriptions.
  • Execution: one read-only or sandboxed happy path per critical capability.
  • Protection: auth, size limits, unsafe input, missing permission, and tool allowlists.
  • Observability: logs, traces, correlation IDs, error messages, and CI artifacts.

Define critical capabilities first

Do not start by writing tests. Start by writing a short capability inventory. For each MCP server, list what the agent can do through it.

Example for a QA MCP server:

  • list_failed_tests reads the latest CI failure summary.
  • get_test_artifact fetches a trace, video, screenshot, or log.
  • create_bug_draft creates a draft bug report, not a final production ticket.
  • run_playwright_spec runs one approved spec in a sandbox.
  • query_test_data reads masked test data from a safe dataset.

Now mark each capability as read-only, write, or destructive. Your smoke tests should prefer read-only tools. For write tools, use a sandbox project and assert cleanup.

Keep the first gate under five minutes

A smoke gate that takes 40 minutes will be skipped by busy teams. My target is simple: local smoke tests under 60 seconds, CI smoke tests under five minutes.

Use a separate nightly job for full agent workflow regression. That longer suite can ask the LLM to triage a failed Playwright trace, create a bug draft, and compare logs. The upgrade smoke gate should answer the smaller question: did the SDK bump break basic connectivity and safety?

Protocol and Transport Checks

Transport bugs are painful because they look like application bugs from the outside. After an SDK upgrade, smoke-test each transport your stack supports. If your production host uses Streamable HTTP, do not pass the build with stdio-only tests.

Check startup on a clean environment

Run the server from a clean environment with the same package manager you use in CI. If production uses uv, test with uv. If your Docker image installs from a lock file, test that image.

python -m venv .venv
. .venv/bin/activate
pip install -U "mcp[cli]"
python -m pytest tests/smoke/test_mcp_startup.py -q

Your startup smoke test should fail on:

  • missing environment variables,
  • renamed imports,
  • invalid server configuration,
  • dependency conflicts,
  • unexpected stdout writes in stdio mode,
  • and malformed tool registration.

Check protocol negotiation

The v2 release note says the new Client negotiates the version automatically. That is useful, but automatic negotiation still deserves tests.

Create a small matrix:

Client Server Expected result
latest 2.x client upgraded server 2026-07-28 path works
old supported client upgraded server 2025-era compatibility path works
unsupported client upgraded server clear rejection or clear error

The third row matters. Silent partial support is worse than a clear error. I want unsupported clients to fail in a way that a developer understands in one CI log.

Check Streamable HTTP and request size behavior

If you expose Streamable HTTP, test basic connectivity, status codes, content types, and the request-size limit. The v2 release notes state that Streamable HTTP servers reject bodies over 4 MiB with HTTP 413. That is a perfect smoke test because it checks both transport handling and safety behavior.

import requests

BASE_URL = "http://localhost:8000/mcp"

def test_streamable_http_rejects_large_body():
    payload = b"x" * (4 * 1024 * 1024 + 1)
    response = requests.post(
        BASE_URL,
        data=payload,
        headers={"content-type": "application/json"},
        timeout=10,
    )
    assert response.status_code == 413

Tool, Resource, and Prompt Checks

After connectivity, test the capability layer. This is where most agent failures start: a tool is missing, a schema changed, a resource returns different fields, or a prompt template no longer exists.

Assert discovery results by name and schema

Tool names are part of your agent contract. If list_failed_tests becomes get_failed_tests, your prompt and router may still reference the old name.

EXPECTED_TOOLS = {
    "list_failed_tests": ["pipeline_id", "branch"],
    "get_test_artifact": ["run_id", "artifact_type"],
    "create_bug_draft": ["title", "steps", "evidence"],
}

def test_tool_discovery_contract(mcp_client):
    tools = mcp_client.list_tools()
    by_name = {tool.name: tool for tool in tools}

    assert set(EXPECTED_TOOLS).issubset(by_name)

    for name, required_fields in EXPECTED_TOOLS.items():
        schema = by_name[name].input_schema
        properties = schema.get("properties", {})
        for field in required_fields:
            assert field in properties, f"{name} missing {field}"

Keep the expected list short. Smoke tests should cover critical tools only. Full schema compatibility belongs in your contract suite.

Run one safe tool call per critical path

Discovery alone is not enough. A registered tool can still fail at runtime because a resolver, token, database connection, or import changed.

def test_list_failed_tests_happy_path(mcp_client):
    result = mcp_client.call_tool(
        "list_failed_tests",
        {"pipeline_id": "smoke-pipeline-001", "branch": "main"},
    )

    assert result.is_error is False
    assert result.content
    assert "failed_tests" in result.structured_content
    assert isinstance(result.structured_content["failed_tests"], list)

Test resources and prompts as first-class contracts

Many teams over-test tools and ignore resources and prompts. That is risky. If a QA agent relies on a resource like qa://release/latest-risk-summary, the smoke suite should assert that the resource exists and returns the expected high-level shape.

def test_release_risk_resource(mcp_client):
    resource = mcp_client.read_resource("qa://release/latest-risk-summary")
    text = resource.text

    assert "risk_score" in text
    assert "changed_services" in text
    assert "recommended_suite" in text

If your server provides reusable prompts, test prompt names and required arguments. A prompt rename can break an agent workflow just as badly as a tool rename.

Security, Auth, and Observability Checks

MCP servers often sit close to sensitive systems. A QA server may read logs, retrieve test data, run specs, or create bug drafts. After an SDK upgrade, you need proof that the safety rails still hold.

Auth smoke tests should cover allowed and denied paths

The v2 release notes mention OAuth hardening, including RFC 9207 issuer validation and related identity flows. Even if you do not use OAuth yet, your smoke suite should test auth behavior at the server boundary.

Minimum auth checks:

  • Valid token can call one read-only tool.
  • Missing token is rejected.
  • Expired token is rejected.
  • Token with wrong scope cannot call write tools.
  • Cross-tenant or wrong-project input is rejected.
def test_write_tool_requires_scope(raw_mcp_client):
    client = raw_mcp_client(token="read_only_smoke_token")

    result = client.call_tool(
        "create_bug_draft",
        {"title": "Smoke", "steps": ["open page"], "evidence": []},
    )

    assert result.is_error is True
    assert "permission" in result.error.message.lower()

Observability is part of the smoke gate

The Python SDK v2 release notes say OpenTelemetry tracing ships on by default. That gives QA teams a better debugging path, but only if traces are actually wired to your collector or CI artifact store.

Add one smoke assertion for observability:

  • A request produces a trace or structured log event.
  • The event includes server version, tool name, request ID, and outcome.
  • Errors include safe diagnostic detail without leaking secrets.

Protect stdio from accidental output

The v2 notes call out hardened stdio behavior: handler subprocesses and stray prints are kept off the wire, and stdout is diverted to stderr while serving. That is good, but your server code can still introduce noisy output through dependencies or debug statements.

Smoke-test stdio mode with a subprocess and fail if stdout contains non-protocol junk before the expected message. This is especially important for internal QA utilities where someone may add a quick print() while debugging a flaky test runner.

CI Implementation With Pytest

Here is the implementation pattern I use. Keep the smoke suite in its own folder, run it on every SDK upgrade branch, and run it before the broader agent regression pack.

Suggested repository layout

mcp-qa-server/
  pyproject.toml
  src/
    qa_mcp_server/
      server.py
      tools.py
      auth.py
  tests/
    smoke/
      test_startup.py
      test_discovery.py
      test_tools.py
      test_transport.py
      test_auth.py
      test_observability.py
    contract/
    agent_workflows/
  .github/
    workflows/
      mcp-smoke.yml

Pytest fixture for a local server

Your actual client code will depend on the SDK version and transport. The fixture below shows the shape. The important bit is not the exact import. The important bit is that the test owns the server lifecycle and does not depend on a manually running process.

import os
import subprocess
import time
import pytest

@pytest.fixture(scope="session")
def mcp_server_process():
    env = os.environ.copy()
    env["QA_MCP_ENV"] = "smoke"
    env["QA_MCP_PROJECT"] = "sandbox"

    proc = subprocess.Popen(
        ["python", "-m", "qa_mcp_server.server"],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True,
        env=env,
    )

    time.sleep(2)
    if proc.poll() is not None:
        stderr = proc.stderr.read()
        raise RuntimeError(f"MCP server failed to start: {stderr}")

    yield proc

    proc.terminate()
    proc.wait(timeout=10)

If the server fails to start, fail fast. Do not let ten downstream tests produce noisy connection errors.

GitHub Actions smoke workflow

name: MCP Server Smoke Tests

on:
  pull_request:
    paths:
      - "pyproject.toml"
      - "uv.lock"
      - "src/**"
      - "tests/smoke/**"
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - name: Install dependencies
        run: uv sync --all-extras
      - name: Run MCP smoke tests
        run: uv run pytest tests/smoke -q --maxfail=1

Version pinning policy

The v2 release notes explicitly tell teams not ready to migrate to keep an upper bound like mcp>=1.28,<2. That is not being conservative. That is basic release hygiene.

Use this policy:

  1. Production services pin major versions.
  2. Renovate or Dependabot opens SDK upgrade pull requests.
  3. Smoke tests run first.
  4. Contract tests run next.
  5. Agent workflow regression runs last.
  6. The release note and migration guide are linked in the PR.

This gives you a clean audit trail when someone asks why the agent server was upgraded.

India Team Rollout: From TCS-Style Process to Product-Speed Gates

India QA teams sit in two very different operating models. Service-company projects often have formal change approvals, release calendars, and tool ownership split across teams. Product companies move faster and expect SDETs to own CI gates directly. MCP server smoke tests fit both models, but the rollout differs.

For service-company QA teams

If you work in a TCS, Infosys, Wipro, Cognizant, or client-services setup, start with a checklist artifact and attach it to the change request. Keep it simple:

  • SDK version before and after.
  • Protocol revision supported.
  • Transports tested.
  • Critical tools tested.
  • Auth negative checks passed.
  • Rollback command and package pin.

This makes the MCP upgrade understandable to project managers who do not care about protocol details but do care about release risk.

For product-company SDETs

If you work in a product company, put the gate in CI and make the report visible in the pull request. A senior SDET in the ₹25-40 LPA band is expected to do more than write browser scripts. They are expected to protect release systems, own test infrastructure, and debug messy integration failures.

MCP is a good place to show that ownership. You are not just testing screens. You are testing the layer that lets AI agents touch engineering systems.

Map smoke tests to owner roles

Do not make one person own everything. Use a simple ownership map:

Area Owner Review signal
Tool schemas SDET Discovery contract diff
Auth and scopes Platform engineer Denied-path tests
CI workflow DevOps/SDET 5-minute gate
Agent workflows AI QA engineer Deterministic regression cases
Rollback Tech lead Version pin and release note

Key Takeaways

MCP server smoke tests turn SDK upgrades from a blind package bump into a controlled release gate. The v2.0.0 Python SDK release is a good reminder: protocol compatibility, transport behavior, discovery contracts, auth, and observability can all shift during a major version change.

  • Run smoke tests after every MCP SDK upgrade, before agent workflow regression.
  • Test startup, discovery, one safe tool call, auth denial, and observability.
  • Cover the real transport your clients use: Streamable HTTP, stdio, or both.
  • Use the official release notes and migration guide as source material for new checks.
  • Keep the gate under five minutes so developers do not bypass it.

If I had to start with only three tests, I would pick tool discovery, one read-only happy path, and one denied write call. Those three catch a surprising amount of upgrade risk without slowing the team.

FAQ

What is the difference between MCP server smoke tests and contract tests?

Smoke tests are the first fast gate. They prove that the upgraded MCP server starts, connects, discovers critical capabilities, executes one or two safe operations, and rejects unsafe calls. Contract tests go deeper into every schema field, error shape, resource format, and backward compatibility rule.

Should I test MCP servers through the LLM agent or directly through the MCP client?

Do both, but not in the same layer. For smoke tests, call the MCP server directly through a client with deterministic inputs. For agent regression, test the LLM workflow on top. This separation tells you whether the bug is in the protocol server or in the agent’s planning behavior.

How often should MCP server smoke tests run?

Run them on every pull request that changes SDK versions, server code, auth code, tool registration, or transport configuration. Also run them nightly if the server depends on external systems such as CI, Jira, GitHub, or a test database.

Do I need separate smoke tests for old MCP clients?

Yes, if you still support old clients. The MCP Python SDK v2 release says the upgraded server can serve earlier protocol revisions. That is exactly the kind of compatibility claim you should verify with a small matrix before rolling the upgrade to teams.

What should fail the build immediately?

Fail the build if the server cannot start, critical tools disappear, required schema fields change, a safe read-only tool cannot execute, an unauthorized write succeeds, or traces/logs disappear. Those failures mean the agent stack is not safe enough for deeper testing.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.