How to Instrument MCP Servers with OpenTelemetry: Step by Step Guide 2026

Model Context Protocol (MCP) servers enable LLMs to call external tools and services, but without observability, a slow database query or failed API call inside an MCP tool can silently degrade agent performance for minutes before anyone notices. According to the OpenTelemetry 2024 End User Survey, 68% of organizations now use distributed tracing in production, and MCP servers are distributed systems that benefit from the same instrumentation patterns.

This guide covers how to instrument MCP servers with OpenTelemetry to track tool call latency, errors, and request flow from client to tool execution. You will learn how to add OpenTelemetry tracing to Python and Node.js MCP servers, configure exporters for production backends, and troubleshoot common instrumentation issues.

Prerequisites

Before instrumenting your MCP server, ensure you have:

• Working MCP server implementation (Python FastMCP, Node.js SDK, or custom implementation)
• Python 3.10+ or Node.js 18+ depending on your server stack
• Access to an OpenTelemetry backend (Jaeger, Zipkin, CubeAPM, or OTLP endpoint)
• Basic understanding of distributed tracing concepts (spans, traces, context propagation)
• Admin access to install dependencies in your deployment environment

For cloud deployments: ensure your firewall rules allow outbound HTTPS to your observability backend, and check that IAM roles permit trace export if deploying to AWS, GCP, or Azure.

Step 1: Add OpenTelemetry Dependencies to Your MCP Server

The first step is installing the OpenTelemetry SDK and exporter packages for your language runtime.

Python (FastMCP or custom MCP server)

FastMCP includes built in OpenTelemetry instrumentation starting from version 3.2.0, but you still need to install the SDK and exporter manually.

Add these dependencies to your project:

pip install opentelemetry-api==1.40.0 \
  opentelemetry-sdk==1.40.0 \
  opentelemetry-exporter-otlp-proto-grpc==1.40.0

If you are using a custom MCP server implementation without FastMCP, also install the instrumentation library:

pip install opentelemetry-instrumentation==0.41b0

Node.js MCP servers

For Node.js based MCP servers, install the OpenTelemetry JavaScript SDK:

npm install @opentelemetry/[email protected] \
  @opentelemetry/[email protected] \
  @opentelemetry/[email protected] \
  @opentelemetry/[email protected] \
  @opentelemetry/[email protected]

These packages provide the core tracing APIs, SDK runtime, OTLP exporter for sending traces, and semantic conventions for standard attribute naming.

Step 2: Configure OpenTelemetry Service Name and Resource Attributes

OpenTelemetry uses resource attributes to identify the service generating telemetry. The service name is the most critical attribute as it groups all spans from your MCP server under one logical service in your observability backend.

Python setup

Create a file named otel_setup.py in your MCP server directory:

from opentelemetry import trace
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
def setup_opentelemetry(service_name: str, endpoint: str = "http://localhost:4317") -> None:
    resource = Resource.create(
        attributes={
            SERVICE_NAME: service_name,
            "deployment.environment": "production",
            "service.version": "1.0.0"
        }
    )
    
    tracer_provider = TracerProvider(resource=resource)
    
    otlp_exporter = OTLPSpanExporter(
        endpoint=endpoint,
        insecure=True
    )
    
    tracer_provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
    trace.set_tracer_provider(tracer_provider)

Call this function at the top of your MCP server entry point before any MCP code runs:

from otel_setup import setup_opentelemetry
setup_opentelemetry("mcp-tool-server", "http://your-otel-collector:4317")

Node.js setup

Create otel-setup.js:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SEMRESATTRS_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');
const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: 'mcp-tool-server',
    'deployment.environment': 'production',
    'service.version': '1.0.0'
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://your-otel-collector:4317'
  })
});
sdk.start();
process.on('SIGTERM', () => {
  sdk.shutdown().then(() => console.log('Tracing terminated'));
});

Import this at the top of your MCP server entry file before any other code:

require('./otel-setup');
const { McpServer } = require('@modelcontextprotocol/sdk/server/mcp.js');

The resource attributes appear as tags on every span, enabling filtering and grouping in your observability backend. The service name is mandatory, environment and version are recommended for multi environment deployments.

Step 3: Instrument MCP Tool Calls with Custom Spans

MCP tool calls are the critical path to instrument. Each tool invocation should generate a span capturing tool name, input parameters, execution time, and result or error status.

Python with FastMCP

FastMCP automatically creates spans for tool calls if OpenTelemetry is configured. To add custom attributes to those spans, use the tracer directly:

from opentelemetry import trace
from fastmcp import FastMCP
tracer = trace.get_tracer(__name__)
mcp = FastMCP("instrumented-server")
@mcp.tool()
def calculate_sum(a: int, b: int) -> int:
    with tracer.start_as_current_span("calculate_sum") as span:
        span.set_attribute("mcp.tool.name", "calculate_sum")
        span.set_attribute("mcp.tool.input.a", a)
        span.set_attribute("mcp.tool.input.b", b)
        
        result = a + b
        
        span.set_attribute("mcp.tool.result", result)
        return result

Python custom MCP server

For custom MCP implementations without FastMCP, wrap tool execution manually:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)
async def execute_tool(tool_name: str, args: dict):
    with tracer.start_as_current_span(f"mcp.tool.{tool_name}") as span:
        span.set_attribute("mcp.tool.name", tool_name)
        span.set_attribute("mcp.tool.args", str(args))
        
        try:
            result = await tool_registry[tool_name](**args)
            span.set_attribute("mcp.tool.status", "success")
            return result
        except Exception as e:
            span.set_attribute("mcp.tool.status", "error")
            span.set_attribute("mcp.tool.error", str(e))
            span.record_exception(e)
            raise

Node.js MCP server

For Node.js, use the OpenTelemetry API to create spans around tool handlers:

const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('mcp-server');
async function handleToolCall(toolName, args) {
  return tracer.startActiveSpan(`mcp.tool.${toolName}`, async (span) => {
    span.setAttribute('mcp.tool.name', toolName);
    span.setAttribute('mcp.tool.args', JSON.stringify(args));
    
    try {
      const result = await toolHandlers[toolName](args);
      span.setAttribute('mcp.tool.status', 'success');
      span.end();
      return result;
    } catch (error) {
      span.setAttribute('mcp.tool.status', 'error');
      span.setAttribute('mcp.tool.error', error.message);
      span.recordException(error);
      span.end();
      throw error;
    }
  });
}

Custom attributes like mcp.tool.name and mcp.tool.args make traces searchable in your observability backend. You can filter for slow tool calls, failed executions, or specific input patterns without parsing log messages.

Step 4: Propagate Trace Context from MCP Client to Server

Distributed tracing requires propagating trace context across process boundaries. When an LLM client calls your MCP server, the client should inject trace context into the request so server spans become children of the client span.

Python client trace propagation

If your MCP client is instrumented with OpenTelemetry, context propagates automatically via HTTP headers or gRPC metadata. To propagate manually in a custom client:

from opentelemetry import trace
from opentelemetry.propagate import inject
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("mcp.client.call") as span:
    headers = {}
    inject(headers)
    
    response = await mcp_client.call_tool(
        "calculate_sum",
        {"a": 5, "b": 3},
        headers=headers
    )

Python server context extraction

On the server side, extract the propagated context from incoming request headers:

from opentelemetry.propagate import extract
def handle_mcp_request(request_headers):
    context = extract(request_headers)
    
    with tracer.start_as_current_span("mcp.server.handle", context=context) as span:
        result = execute_tool(request.tool_name, request.args)
        return result

Node.js context propagation

For Node.js, use the propagation API:

const { propagation, context } = require('@opentelemetry/api');
// Client side
const carrier = {};
propagation.inject(context.active(), carrier);
await mcpClient.callTool('calculate_sum', { a: 5, b: 3 }, carrier);
// Server side
const extractedContext = propagation.extract(context.active(), requestHeaders);
context.with(extractedContext, () => {
  handleToolCall(toolName, args);
});

Without context propagation, client and server spans appear as separate traces instead of one connected request flow. Propagation is what enables end to end visibility from LLM prompt to tool execution and back.

Step 5: Export Traces to Your Observability Backend

OpenTelemetry supports multiple trace export protocols. OTLP (OpenTelemetry Protocol) over gRPC is the standard, but you can also export directly to Jaeger, Zipkin, or vendor specific backends.

Exporting to an OTLP collector

Most production deployments send traces to an OpenTelemetry Collector, which then forwards to one or more backends. Update your exporter endpoint to point to the collector:

otlp_exporter = OTLPSpanExporter(
    endpoint="http://otel-collector.internal:4317",
    insecure=False,
    credentials=grpc.ssl_channel_credentials()
)

For Node.js:

traceExporter: new OTLPTraceExporter({
  url: 'https://otel-collector.internal:4317',
  credentials: grpc.credentials.createSsl()
})

Exporting to CubeAPM

CubeAPM runs inside your cloud VPC and accepts OTLP traces directly. Point your exporter to the CubeAPM OTLP endpoint:

otlp_exporter = OTLPSpanExporter(
    endpoint="http://cubeapm.internal:4317",
    insecure=True
)

CubeAPM indexes all spans automatically with no sampling configuration required. You can query MCP tool calls by tool name, input parameters, or error status without writing custom queries. Unlike SaaS backends, traces stay inside your infrastructure and there are no data egress costs for sending telemetry.

Exporting to Jaeger

To send traces directly to Jaeger without an OTLP collector:

pip install opentelemetry-exporter-jaeger==1.21.0

from opentelemetry.exporter.jaeger.thrift import JaegerExporter
jaeger_exporter = JaegerExporter(
    agent_host_name="jaeger-agent.internal",
    agent_port=6831
)
tracer_provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))

Jaeger is lightweight and works well for development, but requires manual setup for production scale retention and search.

Exporting to cloud vendor backends

For AWS X-Ray, GCP Cloud Trace, or Azure Monitor, install the vendor specific exporter:

# AWS X-Ray
pip install opentelemetry-exporter-aws-xray==1.0.0
# GCP Cloud Trace
pip install opentelemetry-exporter-gcp-trace==1.6.0
# Azure Monitor
pip install azure-monitor-opentelemetry-exporter==1.0.0b26

Each vendor exporter handles authentication and endpoint configuration automatically when running inside their cloud environment. Be aware that sending traces to cloud vendor backends incurs data egress charges if your MCP server runs outside that cloud, typically around $0.09 per GB transferred.

Step 6: Add Span Attributes for MCP Semantic Conventions

OpenTelemetry semantic conventions define standard attribute names for common operations. Following these conventions makes traces consistent across different MCP servers and compatible with observability tools that recognize MCP patterns.

The MCP semantic conventions are published in the OpenTelemetry GenAI repository. Key attributes to include on MCP spans:

span.set_attribute("mcp.method.name", "tools/call")
span.set_attribute("mcp.protocol.version", "2025-06-18")
span.set_attribute("mcp.session.id", session_id)
span.set_attribute("mcp.resource.uri", resource_uri)
span.set_attribute("gen_ai.operation.name", "tool_call")

For tool call spans specifically:

span.set_attribute("mcp.tool.name", tool_name)
span.set_attribute("mcp.tool.input", json.dumps(tool_args))
span.set_attribute("mcp.tool.output", json.dumps(tool_result))
span.set_attribute("mcp.tool.error", error_message if error else None)

Following semantic conventions enables observability backends to automatically group MCP spans, calculate tool call success rates, and surface common failure modes without custom dashboards.

Step 7: Test Your Instrumentation Locally

Before deploying instrumented MCP servers to production, verify traces are being generated and exported correctly.

Run a local OTLP collector

Start an OpenTelemetry Collector locally to receive traces:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  logging:
    loglevel: debug
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging]

docker run -p 4317:4317 \
  -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
  otel/opentelemetry-collector:latest \
  --config=/etc/otel-collector-config.yaml

Run your instrumented MCP server

Start your MCP server with OpenTelemetry configured to send to localhost:4317:

python server.py

Trigger a tool call

Call an MCP tool from your client or test script:

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def test_instrumented_call():
    server_params = StdioServerParameters(
        command="python",
        args=["server.py"]
    )
    
    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            
            result = await session.call_tool(
                "calculate_sum",
                {"a": 10, "b": 20}
            )
            
            print(f"Result: {result}")
asyncio.run(test_instrumented_call())

Check the collector logs for exported spans. You should see span records with your service name and tool attributes. If spans do not appear, verify the exporter endpoint is reachable and no firewall rules block port 4317.

Troubleshooting Common Issues

Spans not appearing in observability backend

Symptom: MCP server runs without errors but no traces appear in Jaeger, CubeAPM, or your chosen backend.

Common causes:

• Exporter endpoint unreachable: verify with telnet otel-collector.internal 4317
• Firewall blocking outbound gRPC: check security groups or iptables rules
• Incorrect exporter configuration: ensure endpoint URL matches your backend exactly
• BatchSpanProcessor not flushing: add explicit shutdown in your server shutdown handler

Fix: Add logging to confirm spans are being created:

import logging
logging.basicConfig(level=logging.DEBUG)

Spans should log to console when created. If they do not, OpenTelemetry is not initialized correctly.

Trace context not propagating between client and server

Symptom: Client and server spans appear as separate traces instead of one connected flow.

Common causes:

• Client not injecting context into request headers
• Server not extracting context from request headers
• Different propagation formats on client and server (W3C vs B3)

Fix: Explicitly set the propagator on both client and server:

from opentelemetry.propagators.w3c import W3CTraceContextPropagator
from opentelemetry.propagate import set_global_textmap
set_global_textmap(W3CTraceContextPropagator())

Verify propagation by inspecting request headers. The traceparent header should contain trace ID and span ID.

High trace volume causing performance impact

Symptom: MCP server latency increases after enabling OpenTelemetry instrumentation.

Common causes:

• Synchronous exporter blocking tool execution
• Excessive span attributes or large payloads
• No sampling configured for high throughput tools

Fix: Switch to BatchSpanProcessor (default in most SDKs) and configure sampling:

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
tracer_provider = TracerProvider(
    resource=resource,
    sampler=TraceIdRatioBased(0.1)  # Sample 10% of traces
)

For production MCP servers handling thousands of tool calls per minute, sampling prevents trace volume from overwhelming your observability backend. CubeAPM handles high cardinality spans efficiently, but sampling still reduces ingestion cost.

Missing span attributes in observability UI

Symptom: Spans appear in traces but custom attributes like mcp.tool.name are missing.

Common causes:

• Attributes set after span ends
• Attribute values exceed backend size limits (typically 1KB per attribute)
• Backend filtering or dropping high cardinality attributes

Fix: Set attributes before calling span.end() and keep values under 1KB:

span.set_attribute("mcp.tool.input", json.dumps(args)[:1000])

If using a managed backend, check whether it samples or drops high cardinality attributes. Some SaaS platforms limit attributes per span to control storage costs.

Monitoring MCP Servers with CubeAPM

CubeAPM runs inside your cloud VPC and ingests OpenTelemetry traces at $0.15 per GB with unlimited retention. Unlike SaaS backends, there are no per host fees or user seat charges, and traces never leave your infrastructure.

To send MCP server traces to CubeAPM:

1. Deploy CubeAPM in your VPC or data center following the self hosted deployment guide
2. Configure your MCP server OTLP exporter to point to the CubeAPM OTLP endpoint (default port 4317)
3. Traces appear in the CubeAPM UI within seconds with automatic indexing on all span attributes

CubeAPM automatically correlates MCP tool call traces with logs and infrastructure metrics. If a tool call fails, you can drill from the error span directly into application logs and host metrics at that timestamp. This unified visibility is critical when debugging distributed systems where an MCP tool depends on external APIs, databases, or cloud services.

CubeAPM supports all major observability signals including APM, infrastructure monitoring, logs, and real user monitoring. For teams already using CubeAPM to monitor their main application stack, adding MCP server traces requires no additional backend setup, and traces are automatically linked to existing service maps and dependency graphs.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions