How to Monitor gRPC Service Latency and Error Rates: Step by Step Guide (2026)

gRPC services fail in production in two ways that matter most to users: they become slow or they return errors. A gRPC method that takes 600 milliseconds instead of 300 milliseconds doubles user perceived latency. A method that returns UNAVAILABLE or DEADLINE_EXCEEDED status codes breaks client flows entirely. Without instrumentation, both problems are invisible until users complain.

This guide covers how to instrument gRPC services to track request latency and error rates using OpenTelemetry, Prometheus, and observability platforms. Each step includes working code examples for Go and Java, the two most common gRPC server languages. By the end, you will have per-method latency histograms, error rate counters, and alerts configured to catch problems before they cascade.

Prerequisites

Before starting, ensure you have:

• A working gRPC service in Go or Java already handling production traffic
• Access to modify server code and redeploy (required for instrumentation)
• OpenTelemetry SDK or Prometheus client library available in your language
• An observability backend to send metrics to: Prometheus, CubeAPM, Datadog, or similar
• Basic understanding of gRPC status codes and protobuf service definitions

If you are using Kubernetes, you will also need permission to deploy sidecars or configure service mesh instrumentation.

Step 1: Choose Your Instrumentation Approach

gRPC services can be instrumented three ways: using OpenTelemetry native instrumentation, using Prometheus client libraries directly, or using service mesh sidecars like Istio or Linkerd. Each approach has different trade-offs.

OpenTelemetry instrumentation is the most portable option. It auto-instruments gRPC servers and clients with minimal code changes, exports metrics in standard formats, and works with any OpenTelemetry-compatible backend. This is the recommended approach for new projects.

Prometheus client libraries give you direct control over metric definitions and labels but require more manual instrumentation code. This approach is common in Go services that already use Prometheus for infrastructure metrics.

Service mesh sidecars auto-inject instrumentation at the proxy layer without touching application code. This is useful for large microservice fleets but adds operational complexity and latency overhead from the sidecar proxy itself.

For this guide, we will focus on OpenTelemetry instrumentation because it provides the best balance of automation, portability, and ecosystem support. If you are already committed to Prometheus, the concepts translate directly you just replace OpenTelemetry SDK calls with Prometheus client library calls.

Step 2: Install OpenTelemetry gRPC Instrumentation

OpenTelemetry provides auto-instrumentation packages for gRPC in most languages. These packages register interceptors that capture method calls, measure duration, and record status codes without requiring you to modify every RPC handler.

For Go services, install the OpenTelemetry gRPC package:

go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

Then register the interceptor when creating your gRPC server:

import (
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
    "google.golang.org/grpc"
)

server := grpc.NewServer(
    grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
    grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor()),
)

For Java services, add the OpenTelemetry gRPC dependency to your Maven or Gradle configuration:

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-grpc-1.6</artifactId>
    <version>2.12.0-alpha</version>
</dependency>

Then wrap your server with OpenTelemetry interceptors:

import io.opentelemetry.instrumentation.grpc.v1_6.GrpcTelemetry;

GrpcTelemetry telemetry = GrpcTelemetry.create(openTelemetry);

Server server = ServerBuilder.forPort(50051)
    .addService(ServerInterceptors.intercept(
        new YourServiceImpl(),
        telemetry.newServerInterceptor()
    ))
    .build();

After deploying this change, your gRPC server will automatically emit OpenTelemetry metrics for every RPC call. The metrics include grpc.server.call.duration (latency histogram) and grpc.server.call.started (request counter with status labels).

Step 3: Configure Metric Export to Your Observability Backend

OpenTelemetry collects metrics but does not store them. You need to configure an exporter to send metrics to your observability platform.

For Prometheus scraping, configure the Prometheus exporter in your OpenTelemetry SDK initialization:

import (
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/sdk/metric"
)

exporter, err := prometheus.New()
if err != nil {
    log.Fatal(err)
}

provider := metric.NewMeterProvider(metric.WithReader(exporter))

This exposes metrics on an HTTP endpoint (typically /metrics on port 9090) that Prometheus can scrape every 15 seconds.

For push-based backends like CubeAPM, Datadog, or New Relic, configure the OTLP exporter to send metrics directly:

import (
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
    "go.opentelemetry.io/otel/sdk/metric"
)

exporter, err := otlpmetricgrpc.New(
    context.Background(),
    otlpmetricgrpc.WithEndpoint("cubeapm.example.com:4317"),
    otlpmetricgrpc.WithInsecure(),
)

provider := metric.NewMeterProvider(
    metric.WithReader(metric.NewPeriodicReader(exporter)),
)

Replace cubeapm.example.com:4317 with your actual OTLP collector endpoint. CubeAPM accepts OTLP metrics on port 4317 by default and automatically indexes them for fast querying without requiring separate Prometheus setup.

Verify the export is working by checking your observability platform for grpc.server.call.duration metrics tagged with grpc.method and grpc.status labels.

Step 4: Query gRPC Latency Metrics

Once metrics are flowing, you can query gRPC latency per method to identify slow endpoints.

In Prometheus or Grafana, calculate the 95th percentile latency for each gRPC method:

histogram_quantile(0.95, 
  rate(grpc_server_call_duration_bucket{job="your-service"}[5m])
)

Group by method to see which endpoints are slowest:

histogram_quantile(0.95, 
  sum by (grpc_method) (
    rate(grpc_server_call_duration_bucket[5m])
  )
)

In CubeAPM or other OpenTelemetry-native platforms, latency metrics are queryable directly from the metrics explorer. Filter by grpc.method to drill down to specific endpoints, or compare latency distributions across services using the trace-to-metric correlation built into infrastructure monitoring platforms that support OpenTelemetry natively.

A production gRPC service handling 10,000 requests per minute across 20 methods should show clear latency separation between fast methods (under 50ms) and slower methods that make database calls or call downstream services (200ms to 500ms). If every method shows similar latency, the bottleneck is likely shared infrastructure (database connection pool, network, or CPU saturation) rather than method-specific logic.

Step 5: Query gRPC Error Rates by Status Code

gRPC uses 16 standard status codes. Only OK (code 0) indicates success. Every other status code represents a failure, but not all failures are equal.

The status codes that matter most for alerting:

• UNAVAILABLE (14) — service is down or unreachable
• DEADLINE_EXCEEDED (4) — request timed out
• RESOURCE_EXHAUSTED (8) — rate limit hit or out of capacity
• INTERNAL (13) — unexpected server error
• UNAUTHENTICATED (16) — authentication failed

In Prometheus, calculate error rate as a percentage of non-OK responses:

sum(rate(grpc_server_call_started_total{grpc_status!="OK"}[5m]))
/
sum(rate(grpc_server_call_started_total[5m]))
* 100

Break down by status code to see which error types dominate:

sum by (grpc_status) (
  rate(grpc_server_call_started_total{grpc_status!="OK"}[5m])
)

In observability platforms with built-in gRPC support, error rates are usually pre-calculated and displayed as service-level RED metrics (Rate, Errors, Duration). Platforms like CubeAPM show per-method error rates with automatic grouping by status code and correlation to distributed traces, so you can jump from a spike in DEADLINE_EXCEEDED errors directly to the trace showing which downstream service caused the timeout.

A healthy gRPC service typically maintains error rates below 0.1 percent during normal operation. Spikes above 1 percent indicate a real problem. Spikes above 5 percent mean the service is degraded to the point where most users are experiencing failures.

Step 6: Configure Latency and Error Rate Alerts

Metrics are only useful if they trigger action. Configure alerts for two scenarios: latency degradation and error rate spikes.

Latency alert example in Prometheus Alertmanager:

groups:
  - name: grpc_latency
    interval: 30s
    rules:
      - alert: GrpcHighLatency
        expr: |
          histogram_quantile(0.95, 
            sum by (grpc_method) (
              rate(grpc_server_call_duration_bucket[5m])
            )
          ) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "gRPC method {{ $labels.grpc_method }} latency above 500ms"

This fires if any gRPC method’s 95th percentile latency stays above 500 milliseconds for 5 consecutive minutes.

Error rate alert example:

- alert: GrpcHighErrorRate
  expr: |
    sum(rate(grpc_server_call_started_total{grpc_status!="OK"}[5m]))
    /
    sum(rate(grpc_server_call_started_total[5m]))
    > 0.01
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "gRPC error rate above 1% for 2 minutes"

This fires if the overall error rate exceeds 1 percent for 2 minutes.

In CubeAPM or other unified observability platforms, alerts can be configured directly in the UI with automatic anomaly detection. Instead of setting static thresholds, you can enable dynamic alerts that fire when latency or error rates deviate significantly from baseline behavior observed over the past 7 days. This reduces false positives during expected traffic spikes while still catching real regressions.

For alerting to be actionable, route gRPC-specific alerts to the team that owns the service. A spike in RESOURCE_EXHAUSTED errors should go to the on-call engineer with context showing which method hit the rate limit and what the traffic pattern looked like before the spike.

Step 7: Add Custom Labels for Business Context

Default OpenTelemetry instrumentation captures method name and status code, but not business-specific dimensions like customer tier, region, or feature flag state. Adding custom labels lets you answer questions like “are premium customers seeing higher error rates than free users?” or “is latency worse in EU than US regions?”

In Go, add custom attributes to the gRPC context before the method executes:

import (
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/trace"
)

func (s *server) YourMethod(ctx context.Context, req *pb.Request) (*pb.Response, error) {
    span := trace.SpanFromContext(ctx)
    span.SetAttributes(
        attribute.String("customer.tier", req.CustomerTier),
        attribute.String("region", req.Region),
    )
    
    // method logic here
}

In Java, use the OpenTelemetry API to set attributes on the current span:

import io.opentelemetry.api.trace.Span;

public void yourMethod(Request request, StreamObserver<Response> responseObserver) {
    Span span = Span.current();
    span.setAttribute("customer.tier", request.getCustomerTier());
    span.setAttribute("region", request.getRegion());
    
    // method logic here
}

After deploying this change, your metrics will include customer.tier and region labels. You can then query error rates segmented by customer tier to detect if a specific cohort is disproportionately affected by failures.

Custom labels increase metric cardinality. A service with 20 methods, 3 customer tiers, and 5 regions generates 300 unique metric time series just for latency histograms. High-cardinality metrics can overwhelm Prometheus but are handled natively by systems like CubeAPM, which use columnar storage and smart indexing to query millions of unique label combinations without performance degradation.

Troubleshooting Common Issues

Metrics are not appearing in Prometheus scrape endpoint

Check that the OpenTelemetry Prometheus exporter is configured and that the /metrics HTTP handler is registered. Curl the endpoint directly from localhost to verify it returns data:

curl http://localhost:9090/metrics | grep grpc_server

If this returns nothing, the exporter is not initialized or the metric names do not match. Verify you are using the official OpenTelemetry gRPC instrumentation package and not a custom wrapper that might use different metric names.

Latency metrics show zeros or unrealistic values

OpenTelemetry records latency in seconds by default. If your queries assume milliseconds, you will see values like 0.3 instead of 300. Convert to milliseconds by multiplying by 1000 in your PromQL query:

histogram_quantile(0.95, rate(grpc_server_call_duration_bucket[5m])) * 1000

Error rate is always zero even when clients report failures

Verify that the gRPC server is actually returning non-OK status codes. If your server catches all errors and returns OK with an error payload in the response body, OpenTelemetry will not count these as errors. gRPC status codes must be set using the official gRPC status package for instrumentation to recognize them.

In Go, return errors using status.Error:

import "google.golang.org/grpc/status"
import "google.golang.org/grpc/codes"

return nil, status.Error(codes.Unavailable, "database connection failed")

In Java, throw a StatusRuntimeException:

throw new StatusRuntimeException(Status.UNAVAILABLE.withDescription("database connection failed"));

Metric cardinality is too high and Prometheus is rejecting samples

If you added custom labels with high cardinality values (user IDs, request IDs, timestamps), Prometheus will hit its series limit and start dropping metrics. Remove high-cardinality labels from metrics and move them to traces instead. Metrics should only include low-cardinality dimensions like method name, status code, region, and customer tier. Individual request details belong in distributed traces, not metrics.

Platforms designed for high-cardinality observability like Cassandra monitoring tools or CubeAPM handle this differently by indexing all labels without cardinality limits, but if you are using Prometheus, keep metric cardinality under 10,000 unique series per service.

Alerts are firing during normal traffic spikes

Static threshold alerts fire false positives during expected traffic growth. Switch to anomaly-based alerts that compare current error rates or latency to historical baselines. Most observability platforms support this natively. If you are using Prometheus, you can implement basic anomaly detection using the predict_linear function to compare current values against trend lines.

Monitoring gRPC latency and error rates is not optional for production services. Without instrumentation, you are blind to performance degradation until it affects enough users to generate support tickets. With proper instrumentation using OpenTelemetry, you get per-method latency histograms, error rates broken down by status code, and alerts that fire before problems cascade.

The steps in this guide work for any gRPC service in Go or Java. The same concepts apply to other languages OpenTelemetry provides instrumentation libraries for Python, Node.js, Rust, and C++. The key is to instrument at the framework level using interceptors rather than manually logging metrics in every RPC handler.

Once instrumentation is in place, the next step is correlating gRPC metrics with distributed traces to understand why a method became slow or started returning errors. Traces show the full request path across services, database queries, and external API calls. Combined with metrics, traces turn “the service is slow” into “the service is slow because this specific database query is taking 800ms instead of the usual 50ms.”

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions