How to Monitor MCP Tool Call Latency and Error Rates: Step-by-Step Guide 2026

Model Context Protocol servers fail silently in production. A tool that times out or returns malformed JSON appears healthy in logs but corrupts agent decisions downstream. Without visibility into tool invocation latency, error rates, and selection accuracy, you cannot diagnose whether an agent failure stems from model drift, tool timeout, or context contamination between calls.

According to the CNCF Annual Survey 2024, 87% of organizations now use distributed tracing for production debugging. MCP tool monitoring requires the same distributed tracing discipline: every tool invocation must propagate correlation IDs, capture timing data, and link back to the parent agent decision that triggered it.

This guide walks through setting up end to end monitoring for MCP tool calls using OpenTelemetry instrumentation, time series metrics, and distributed tracing. Each step includes working code examples for Python and TypeScript MCP servers, configuration for popular observability backends, and troubleshooting patterns for the most common failure modes.

Prerequisites

Before starting, ensure you have:

• An MCP server running in Python (using mcp package) or TypeScript (using @modelcontextprotocol/sdk)
• Access to an observability platform that supports OpenTelemetry (CubeAPM, Datadog, Grafana Cloud, or self hosted OpenTelemetry Collector)
• Basic familiarity with distributed tracing concepts (spans, traces, correlation IDs)
• Admin access to configure environment variables and deploy instrumentation code
• A test environment to validate instrumentation before production rollout

Recommended but optional: a metrics backend like Prometheus or a platform with native metrics support to track request rates and latency distributions over time.

Step 1: Instrument MCP Tool Calls with OpenTelemetry

The first step is wrapping every MCP tool invocation in an OpenTelemetry span. This creates a structured record of each call with start time, end time, tool name, parameters, and outcome.

For Python MCP servers, use the OpenTelemetry Python SDK to auto instrument HTTP calls and manually trace tool invocations:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from mcp.server import Server
from mcp.types import Tool
# Initialize OpenTelemetry
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
# Configure OTLP exporter (replace endpoint with your observability platform)
otlp_exporter = OTLPSpanExporter(endpoint="https://your-collector:4317")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
# Wrap tool calls in spans
async def call_tool_with_tracing(tool_name: str, params: dict):
    with tracer.start_as_current_span(
        f"mcp.tool.call",
        attributes={
            "tool.name": tool_name,
            "tool.params": str(params),
            "mcp.server": "credit-scoring-v2"
        }
    ) as span:
        try:
            result = await execute_tool(tool_name, params)
            span.set_attribute("tool.result.size_bytes", len(str(result)))
            span.set_status(trace.Status(trace.StatusCode.OK))
            return result
        except Exception as e:
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            span.record_exception(e)
            raise

For TypeScript MCP servers using @modelcontextprotocol/sdk:

import { trace } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
const provider = new NodeTracerProvider();
const exporter = new OTLPTraceExporter({
  url: 'https://your-collector:4317'
});
provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();
const tracer = trace.getTracer('mcp-tool-server');
async function callToolWithTracing(toolName: string, params: any) {
  const span = tracer.startSpan('mcp.tool.call', {
    attributes: {
      'tool.name': toolName,
      'tool.params': JSON.stringify(params),
      'mcp.server': 'document-parser-v3'
    }
  });
  try {
    const result = await executeTool(toolName, params);
    span.setAttribute('tool.result.size_bytes', JSON.stringify(result).length);
    span.setStatus({ code: 1 }); // OK
    return result;
  } catch (error) {
    span.setStatus({ code: 2, message: error.message }); // ERROR
    span.recordException(error);
    throw error;
  } finally {
    span.end();
  }
}

This instrumentation captures execution time automatically by recording span start and end timestamps. Tool name, parameters, and result size are stored as span attributes, making them queryable later for debugging slow or failing tools.

Step 2: Export Telemetry to Your Observability Platform

Once instrumentation is in place, configure the OpenTelemetry exporter to send trace data to your chosen backend. The exporter endpoint varies by platform:

CubeAPM (self hosted): Set OTEL_EXPORTER_OTLP_ENDPOINT=http://your-cubeapm-host:4317 in your environment. CubeAPM runs inside your VPC and accepts OTLP over gRPC with no external data egress. All MCP telemetry stays within your infrastructure.

Datadog: Use the Datadog Agent as an OTLP collector. Set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 and configure the agent with your API key. Note that Datadog charges $31 per host per month for APM, which can scale quickly if MCP servers run across multiple instances.

Grafana Cloud: Export directly to Grafana Cloud Tempo. Set OTEL_EXPORTER_OTLP_ENDPOINT=https://tempo-prod-us-central-0.grafana.net:443 and add your Grafana Cloud API token as a header. Free tier includes 50 GB of trace data per month.

Self hosted OpenTelemetry Collector: Deploy the OpenTelemetry Collector as a sidecar or gateway. Point your MCP server at http://otel-collector:4317 and configure the collector to forward spans to Jaeger, Zipkin, or Prometheus for storage.

Environment variable based configuration works across all platforms:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-backend:4317"
export OTEL_SERVICE_NAME="mcp-tool-server"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=2.1.0"

Restart your MCP server after setting these variables. Verify that traces appear in your observability platform by triggering a test tool call and searching for spans tagged with tool.name.

Step 3: Configure Metrics for Tool Latency and Error Rates

Traces show individual tool invocations. Metrics aggregate those invocations into time series data for latency percentiles and error rates. Use OpenTelemetry metrics or a metrics library to track:

• mcp.tool.call.duration (histogram): Tool execution time in milliseconds, tagged by tool name
• mcp.tool.call.errors (counter): Failed tool calls, tagged by tool name and error type
• mcp.tool.call.count (counter): Total tool invocations, tagged by tool name

Python example using OpenTelemetry metrics:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
# Initialize metrics
metric_exporter = OTLPMetricExporter(endpoint="https://your-collector:4317")
metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=10000)
metrics.set_meter_provider(MeterProvider(metric_readers=[metric_reader]))
meter = metrics.get_meter(__name__)
# Create metric instruments
tool_duration = meter.create_histogram(
    "mcp.tool.call.duration",
    unit="ms",
    description="Tool execution time"
)
tool_errors = meter.create_counter(
    "mcp.tool.call.errors",
    description="Failed tool calls"
)
# Record metrics in tool wrapper
async def call_tool_with_metrics(tool_name: str, params: dict):
    start = time.time()
    try:
        result = await execute_tool(tool_name, params)
        duration_ms = (time.time() - start) * 1000
        tool_duration.record(duration_ms, {"tool.name": tool_name})
        return result
    except Exception as e:
        tool_errors.add(1, {"tool.name": tool_name, "error.type": type(e).__name__})
        raise

For TypeScript, use the OpenTelemetry metrics API:

import { metrics } from '@opentelemetry/api';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
const metricExporter = new OTLPMetricExporter({ url: 'https://your-collector:4317' });
const metricReader = new PeriodicExportingMetricReader({
  exporter: metricExporter,
  exportIntervalMillis: 10000
});
const meterProvider = new MeterProvider({ readers: [metricReader] });
metrics.setGlobalMeterProvider(meterProvider);
const meter = metrics.getMeter('mcp-tool-server');
const toolDuration = meter.createHistogram('mcp.tool.call.duration', { unit: 'ms' });
const toolErrors = meter.createCounter('mcp.tool.call.errors');
async function callToolWithMetrics(toolName: string, params: any) {
  const start = Date.now();
  try {
    const result = await executeTool(toolName, params);
    toolDuration.record(Date.now() - start, { 'tool.name': toolName });
    return result;
  } catch (error) {
    toolErrors.add(1, { 'tool.name': toolName, 'error.type': error.name });
    throw error;
  }
}

These metrics feed into dashboards showing p95 latency, error rate trends, and per-tool performance over time. Platforms like infrastructure monitoring tools that support OTLP metrics will visualize these automatically.

Step 4: Set Up Distributed Tracing Across Agent and Tool Calls

MCP tool calls exist within a larger agent workflow. To debug failures, you need to trace the full sequence: agent receives user query, selects tool, invokes tool via MCP server, processes tool result, generates final response. This requires propagating trace context from the agent to the MCP server.

In Python, propagate trace context using OpenTelemetry context propagation headers:

from opentelemetry.propagate import inject
import httpx
# In agent code calling MCP server
async def invoke_mcp_tool(tool_name: str, params: dict):
    headers = {}
    inject(headers)  # Injects traceparent header
    
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://mcp-server:8080/tool/call",
            json={"tool": tool_name, "params": params},
            headers=headers
        )
    return response.json()

On the MCP server side, extract the trace context:

from opentelemetry.propagate import extract
from starlette.requests import Request
@app.post("/tool/call")
async def handle_tool_call(request: Request):
    # Extract trace context from headers
    context = extract(request.headers)
    
    with tracer.start_as_current_span("mcp.tool.execute", context=context):
        tool_name = request.json()["tool"]
        params = request.json()["params"]
        result = await execute_tool(tool_name, params)
        return {"result": result}

This creates a single distributed trace spanning agent reasoning, tool selection, MCP server invocation, and result processing. When a tool call fails, you can see the full context: which prompt triggered tool selection, what parameters the agent passed, and where in the tool execution the failure occurred.

Step 5: Create Alerts for Latency Spikes and Error Rate Thresholds

Monitoring without alerting is passive observation. Set up proactive alerts that notify you when MCP tool performance degrades before users report issues.

Use p95 latency thresholds, not averages. A single slow tool call does not indicate a systemic problem. Alert when 95% of calls exceed your target latency for a sustained period.

Example alert rules in Prometheus query language (PromQL):

# Alert when p95 latency exceeds 2 seconds for any tool
- alert: MCPToolLatencyHigh
  expr: histogram_quantile(0.95, rate(mcp_tool_call_duration_bucket[5m])) > 2000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "MCP tool {{ $labels.tool_name }} p95 latency above 2s"
# Alert when error rate exceeds 5% for any tool
- alert: MCPToolErrorRateHigh
  expr: |
    (
      sum(rate(mcp_tool_call_errors[5m])) by (tool_name)
      /
      sum(rate(mcp_tool_call_count[5m])) by (tool_name)
    ) > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "MCP tool {{ $labels.tool_name }} error rate above 5%"

Configure alert destinations (Slack, PagerDuty, email) in your observability platform. For CubeAPM users, alerts route through the native alerting system with full trace context attached to each notification. For Datadog, configure monitors in the Datadog UI and route to your on call rotation.

Reduce alert noise by grouping MCP alerts by workflow or namespace instead of firing individual alerts per tool invocation. A production team reduced alert fatigue by 60% after switching from per-pod alerts to grouped Kubernetes alerts in Kubernetes monitoring platforms.

Step 6: Build Dashboards for Real Time Tool Performance Visibility

Dashboards turn raw telemetry into actionable insight. Create a dedicated MCP tool performance dashboard showing:

• Tool invocation rate over time (requests per minute)
• Latency distribution by tool (p50, p95, p99)
• Error rate percentage by tool
• Top 5 slowest tools by average latency
• Recent failed tool calls with trace links

Grafana dashboard example using Prometheus data source:

panels:
  - title: "MCP Tool Call Rate"
    type: graph
    targets:
      - expr: sum(rate(mcp_tool_call_count[5m])) by (tool_name)
  
  - title: "Tool Latency (p95)"
    type: graph
    targets:
      - expr: histogram_quantile(0.95, rate(mcp_tool_call_duration_bucket[5m]))
  
  - title: "Error Rate by Tool"
    type: table
    targets:
      - expr: |
          (sum(rate(mcp_tool_call_errors[5m])) by (tool_name)
           / sum(rate(mcp_tool_call_count[5m])) by (tool_name)) * 100

CubeAPM provides pre built dashboards for OpenTelemetry instrumented services. After Step 2, MCP tool metrics appear automatically in the service overview with drill down to individual traces. Synthetic monitoring tools can proactively test critical MCP tool paths every 5 minutes to detect failures before production traffic hits them.

Link dashboard panels to trace search. When you see latency spike in the graph, click through to see which specific tool calls contributed to that spike and view their full trace timeline.

Troubleshooting Common Issues

Traces not appearing in observability platform

Verify the OTLP exporter endpoint is reachable from your MCP server. Test with curl:

curl -v https://your-collector:4317

Check that firewall rules allow outbound gRPC on port 4317. If using a self hosted collector, confirm the collector is running and accepting spans:

docker logs otel-collector | grep "Trace span processor"

Ensure service name is set correctly. Without OTEL_SERVICE_NAME, spans may be grouped under “unknown_service” and hard to find.

High cardinality causing metric explosion

Avoid adding unbounded values like user IDs or request IDs as metric labels. Only use low cardinality attributes: tool name, error type, deployment environment. If you need to track per-user metrics, use exemplars that link metrics to traces instead of creating separate metric series per user.

Tool calls not linking to parent agent traces

Verify trace context propagation headers are being sent. Log the traceparent header value in both agent and MCP server to confirm it matches. If using HTTP libraries that do not auto inject headers, manually call inject() before making requests.

For message queue based MCP servers (Kafka, RabbitMQ), propagate context in message headers:

from opentelemetry.propagate import inject
headers = {}
inject(headers)
producer.send("mcp-tool-queue", value=payload, headers=headers)

Alert fatigue from noisy error alerts

MCP tool errors fall into transient (network timeout, temporary unavailability) and persistent (authentication failure, malformed schema). Alert only on persistent errors or error rates sustained for 5 minutes or longer. Use alert grouping to collapse multiple errors from the same tool into a single notification.

One team reduced MCP alert noise by 70% after filtering out single occurrence errors and only alerting when error rate exceeded 3% over a 10 minute window.

Monitoring MCP Tool Calls with CubeAPM

CubeAPM provides native support for MCP tool observability with OpenTelemetry based distributed tracing, metrics, and logs in a single platform. Unlike SaaS tools that charge per host or per user, CubeAPM runs inside your VPC with predictable $0.15/GB pricing and unlimited retention.

MCP tool spans automatically link to parent agent traces when OpenTelemetry context propagation is configured. You can drill down from an agent decision failure directly to the slow database query inside the MCP tool that caused it, all within the same trace view.

CubeAPM’s self hosted deployment means no MCP telemetry data leaves your infrastructure. For teams handling sensitive customer data or operating under GDPR, HIPAA, or SOC 2 requirements, this eliminates data residency concerns that plague cloud-only monitoring platforms.

To instrument MCP servers with CubeAPM, set the OTLP endpoint to your CubeAPM instance and enable trace collection. Pre built dashboards for OpenTelemetry services surface tool latency, error rates, and invocation counts without custom configuration. Alerts integrate natively with Slack, PagerDuty, and email, attaching full trace context to every notification.

Pricing based on publicly available information as of April 2026. Feature availability may vary by plan tier. Verify current feature sets on CubeAPM’s official documentation.

MCP tool monitoring is one component of a broader infrastructure monitoring strategy. Production MCP deployments also require monitoring the underlying Kubernetes cluster health, database query performance, and message queue throughput. Tools like Kafka monitoring platforms complement MCP tool monitoring when your agent workflow uses Kafka for asynchronous tool invocation.

Without comprehensive observability into tool latency, error patterns, and selection accuracy, MCP agents operate as black boxes that fail unpredictably under load. The instrumentation patterns covered in this guide bring the same distributed tracing discipline to MCP servers that production engineering teams already apply to microservices, giving you the visibility needed to debug failures, optimize costs, and maintain SLAs.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions