How to Set Up Java gRPC Monitoring: Traces, Latency, and Error Rate

A gRPC service that returns UNAVAILABLE or DEADLINE_EXCEEDED at 2 AM gives you nothing to work with unless you already have traces, latency histograms, and error rate metrics flowing to a backend. Unlike HTTP APIs where status codes appear in access logs by default, gRPC status codes are invisible without explicit instrumentation — a failed RPC looks identical to a successful one at the transport layer if you are only watching TCP connections.

This guide walks through the full setup for Java gRPC monitoring using OpenTelemetry and Micrometer: adding interceptors, capturing grpc.server.call.duration histograms, computing error rates from grpc.status != OK, and sending telemetry to a backend where you can build alerts and dashboards. By the end you will have traces that show every hop in a service call, latency percentiles per method, and a query that gives you error rate for any RPC in production.

According to the CNCF 2024 Microservices Observability Survey, gRPC is used in production by over 50% of respondents running microservices workloads, yet distributed tracing coverage for gRPC remains lower than for REST which makes proper instrumentation here a meaningful reliability gap to close.

Prerequisites

• Java 11 or higher (Java 17 recommended)
• Maven or Gradle build tool
• A running gRPC server and client (generated from .proto definitions)
• Basic familiarity with OpenTelemetry concepts: spans, traces, and exporters
• An OTel-compatible backend to receive telemetry (Jaeger, Prometheus, or a platform like CubeAPM)
• Docker (optional, for running a local collector)

—

Step 1: Add OpenTelemetry and gRPC Instrumentation Dependencies

The opentelemetry-grpc-1.6 instrumentation library provides server and client interceptors that auto-instrument every RPC without touching your handler code. Add the following to your pom.xml:

<!-- pom.xml -->
<properties>
  <otel.version>1.38.0</otel.version>
  <otel.instrumentation.version>2.4.0</otel.instrumentation.version>
</properties>
<dependencies>
  <!-- OpenTelemetry API and SDK -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>${otel.version}</version>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
    <version>${otel.version}</version>
  </dependency>
  <!-- OTLP gRPC exporter -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>${otel.version}</version>
  </dependency>
  <!-- gRPC instrumentation interceptors -->
  <dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-grpc-1.6</artifactId>
    <version>${otel.instrumentation.version}</version>
  </dependency>
  <!-- Semantic conventions -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-semconv</artifactId>
    <version>${otel.version}-alpha</version>
  </dependency>
</dependencies>

For Gradle:

// build.gradle
def otelVersion = "1.38.0"
def otelInstrVersion = "2.4.0"
dependencies {
    implementation "io.opentelemetry:opentelemetry-api:${otelVersion}"
    implementation "io.opentelemetry:opentelemetry-sdk:${otelVersion}"
    implementation "io.opentelemetry:opentelemetry-exporter-otlp:${otelVersion}"
    implementation "io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:${otelInstrVersion}"
    implementation "io.opentelemetry:opentelemetry-semconv:${otelVersion}-alpha"
}

If you are also using Micrometer for Prometheus metrics alongside OpenTelemetry traces, add the micrometer-registry-prometheus dependency:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
  <version>1.13.0</version>
</dependency>

—

Step 2: Configure the OpenTelemetry SDK and OTLP Exporter

Before attaching interceptors, you need a configured OpenTelemetry instance. This sets the service name, connects an OTLP exporter to your collector or backend, and registers the global tracer provider.

// src/main/java/com/example/grpc/telemetry/OtelConfig.java
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
public class OtelConfig {
    public static OpenTelemetry init() {
        // Define your service resource attributes
        Resource resource = Resource.getDefault()
            .merge(Resource.create(Attributes.of(
                AttributeKey.stringKey("service.name"), "order-service",
                AttributeKey.stringKey("service.version"), "2.1.0",
                AttributeKey.stringKey("deployment.environment"), "production"
            )));
        // Configure the OTLP exporter — point to your OTel Collector or backend
        OtlpGrpcSpanExporter exporter = OtlpGrpcSpanExporter.builder()
            .setEndpoint("http://otel-collector:4317")
            .build();
        // Build the tracer provider with a batch processor
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
            .setResource(resource)
            .build();
        // Build and register globally
        OpenTelemetrySdk sdk = OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .buildAndRegisterGlobal();
        // Register a shutdown hook
        Runtime.getRuntime().addShutdownHook(
            new Thread(tracerProvider::close)
        );
        return sdk;
    }
}

The setEndpoint value should point to your OTel Collector’s gRPC receiver port (4317 by default). If you are sending directly to a backend like CubeAPM or Jaeger, replace this with their OTLP ingestion endpoint.

—

Step 3: Attach gRPC Interceptors to Server and Client

GrpcTelemetry from the opentelemetry-grpc-1.6 library creates interceptors that wrap every incoming and outgoing RPC. The server interceptor creates a span for each received call; the client interceptor creates a span for each outgoing call and injects trace context into gRPC metadata so the receiver can continue the same trace.

// src/main/java/com/example/grpc/GrpcServerSetup.java
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import io.grpc.Server;
import io.grpc.ServerBuilder;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.grpc.v1_6.GrpcTelemetry;
import com.example.grpc.telemetry.OtelConfig;
import com.example.grpc.service.OrderServiceImpl;
public class GrpcServerSetup {
    public static void main(String[] args) throws Exception {
        // 1. Initialize OpenTelemetry
        OpenTelemetry openTelemetry = OtelConfig.init();
        // 2. Create GrpcTelemetry from the OpenTelemetry instance
        GrpcTelemetry grpcTelemetry = GrpcTelemetry.create(openTelemetry);
        // 3. Build the server and attach the server interceptor
        Server server = ServerBuilder.forPort(50051)
            .intercept(grpcTelemetry.newServerInterceptor())
            .addService(new OrderServiceImpl())
            .build()
            .start();
        System.out.println("gRPC server started on port 50051");
        server.awaitTermination();
    }
}

For the client side:

// src/main/java/com/example/grpc/GrpcClientSetup.java
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.grpc.v1_6.GrpcTelemetry;
import com.example.proto.OrderServiceGrpc;
public class GrpcClientSetup {
    private final OrderServiceGrpc.OrderServiceBlockingStub stub;
    public GrpcClientSetup(OpenTelemetry openTelemetry) {
        GrpcTelemetry grpcTelemetry = GrpcTelemetry.create(openTelemetry);
        ManagedChannel channel = ManagedChannelBuilder
            .forAddress("order-service", 50051)
            .usePlaintext()
            // Attach the client interceptor here
            .intercept(grpcTelemetry.newClientInterceptor())
            .build();
        this.stub = OrderServiceGrpc.newBlockingStub(channel);
    }
}

After this step, every RPC generates two spans: a CLIENT span on the calling side and a SERVER span on the receiving side, linked by the same trace ID propagated via gRPC metadata headers.

—

Step 4: Add Metrics — Latency Histograms and Error Rate

Traces give you individual request detail. Metrics give you aggregated latency percentiles and error rates across all requests. The OpenTelemetry gRPC instrumentation emits grpc.server.call.duration and grpc.client.attempt.duration as histogram metrics automatically when you configure a MeterProvider.

First, add the Prometheus metric exporter and configure it alongside your tracer provider:

// src/main/java/com/example/grpc/telemetry/OtelConfig.java (extended)
import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;
import io.opentelemetry.exporter.prometheus.PrometheusHttpServer;
public class OtelConfig {
    public static OpenTelemetry init() {
        Resource resource = Resource.getDefault()
            .merge(Resource.create(Attributes.of(
                AttributeKey.stringKey("service.name"), "order-service",
                AttributeKey.stringKey("deployment.environment"), "production"
            )));
        // Trace exporter (OTLP)
        OtlpGrpcSpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
            .setEndpoint("http://otel-collector:4317")
            .build();
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(BatchSpanProcessor.builder(spanExporter).build())
            .setResource(resource)
            .build();
        // Prometheus metrics server on port 9464
        // Prometheus scrapes this endpoint directly
        PrometheusHttpServer prometheusServer = PrometheusHttpServer.builder()
            .setPort(9464)
            .build();
        SdkMeterProvider meterProvider = SdkMeterProvider.builder()
            .registerMetricReader(prometheusServer)
            .setResource(resource)
            .build();
        return OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setMeterProvider(meterProvider)
            .buildAndRegisterGlobal();
    }
}

Add the Prometheus exporter dependency to pom.xml:

<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-prometheus</artifactId>
  <version>${otel.version}</version>
</dependency>

Once running, Prometheus scrapes http://your-service:9464/metrics and you get metrics like:

# TYPE grpc_server_call_duration_seconds histogram
grpc_server_call_duration_seconds_bucket{grpc_method="CreateOrder",grpc_status="OK",le="0.005"} 142
grpc_server_call_duration_seconds_bucket{grpc_method="CreateOrder",grpc_status="OK",le="0.025"} 898
grpc_server_call_duration_seconds_bucket{grpc_method="CreateOrder",grpc_status="OK",le="0.1"} 1203
grpc_server_call_duration_seconds_sum{grpc_method="CreateOrder",grpc_status="OK"} 45.7
grpc_server_call_duration_seconds_count{grpc_method="CreateOrder",grpc_status="OK"} 1248

—

Step 5: Query Latency Percentiles and Error Rate

With the histogram emitting to Prometheus, you can now compute P50, P95, and P99 latency per RPC method, plus error rate.

P99 latency per gRPC method:

histogram_quantile(
  0.99,
  sum by (grpc_method, le) (
    rate(grpc_server_call_duration_seconds_bucket[5m])
  )
)

P95 latency:

histogram_quantile(
  0.95,
  sum by (grpc_method, le) (
    rate(grpc_server_call_duration_seconds_bucket[5m])
  )
)

Error rate per method (percentage of non-OK status calls):

sum by (grpc_method) (
  rate(grpc_server_call_duration_seconds_count{grpc_status!="OK"}[5m])
)
/
sum by (grpc_method) (
  rate(grpc_server_call_duration_seconds_count[5m])
)
* 100

This query divides non-OK status calls by total calls for each method. gRPC status codes that count as errors include UNAVAILABLE, INTERNAL, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, and UNKNOWN. The OK status is the only successful outcome in the gRPC status model — unlike HTTP where 2xx has a range.

Total request throughput (RPS) per method:

sum by (grpc_method) (
  rate(grpc_server_call_duration_seconds_count[5m])
)

These three queries together give you the RED metrics (Rate, Errors, Duration) for every gRPC method in your service.

—

Step 6: Add Custom Spans for Business Logic Inside Handlers

The interceptor creates a span for the full RPC duration. But if your handler calls a database, an internal cache, or a downstream service, those child operations are invisible unless you create spans for them manually. This is where production debugging value compounds — a slow CreateOrder RPC might be fast at the gRPC layer but slow inside a PostgreSQL query.

// src/main/java/com/example/grpc/service/OrderServiceImpl.java
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.grpc.stub.StreamObserver;
import com.example.proto.CreateOrderRequest;
import com.example.proto.CreateOrderResponse;
import com.example.proto.OrderServiceGrpc;
public class OrderServiceImpl extends OrderServiceGrpc.OrderServiceImplBase {
    private final Tracer tracer = GlobalOpenTelemetry.getTracer(
        "com.example.order-service", "2.1.0"
    );
    @Override
    public void createOrder(
            CreateOrderRequest request,
            StreamObserver<CreateOrderResponse> responseObserver) {
        // Child span for the database insert
        Span dbSpan = tracer.spanBuilder("db.order.insert")
            .setAttribute("db.system", "postgresql")
            .setAttribute("db.name", "orders")
            .setAttribute("order.customer_id", request.getCustomerId())
            .startSpan();
        try (Scope scope = dbSpan.makeCurrent()) {
            // Simulate database operation
            String orderId = insertOrder(request);
            CreateOrderResponse response = CreateOrderResponse.newBuilder()
                .setOrderId(orderId)
                .build();
            responseObserver.onNext(response);
            responseObserver.onCompleted();
        } catch (Exception e) {
            // Record the exception on the span — this surfaces in trace UIs
            dbSpan.recordException(e);
            dbSpan.setStatus(StatusCode.ERROR, e.getMessage());
            responseObserver.onError(e);
        } finally {
            dbSpan.end();
        }
    }
    private String insertOrder(CreateOrderRequest request) {
        // Database logic here
        return "ORD-" + System.currentTimeMillis();
    }
}

The key pattern: always call dbSpan.end() in a finally block. If you close the span only in the success path, failed requests produce open spans that never flush, and your trace backend accumulates orphaned span records.

—

Step 7: Deploy an OpenTelemetry Collector and Connect a Backend

Running a local OTel Collector between your service and the backend lets you pipeline, filter, and route telemetry without changing application code. Here is a minimal collector configuration that receives OTLP from your Java service and exports to both Prometheus and an OTLP-compatible backend:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
exporters:
  # Prometheus metrics endpoint (scraped by Prometheus)
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: grpc_service
  # OTLP to your trace backend (CubeAPM, Jaeger, etc.)
  otlp/backend:
    endpoint: "http://your-backend:4317"
    tls:
      insecure: true
  # Debug logging (remove in production)
  debug:
    verbosity: basic
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend, debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus, otlp/backend]

Run the collector with Docker:

docker run --rm \
  -v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 8889:8889 \
  otel/opentelemetry-collector-contrib:latest

If you are using CubeAPM as your backend, set the OTLP endpoint in the exporter to CubeAPM’s ingestion URL. CubeAPM is OpenTelemetry native, so it ingests traces and metrics without any custom agent or proprietary SDK — your gRPC spans land in the same trace view as your HTTP service spans, Kafka consumer spans, and database query spans.

—

Step 8: Set Alerts on gRPC Error Rate and Latency

Metrics are only useful if something fires when they breach thresholds. Here are Prometheus alerting rules for gRPC services:

# grpc-alerts.yaml
groups:
  - name: grpc_service_alerts
    interval: 30s
    rules:
      # Alert when error rate exceeds 5% for any gRPC method
      - alert: GrpcHighErrorRate
        expr: |
          (
            sum by (grpc_method, job) (
              rate(grpc_server_call_duration_seconds_count{grpc_status!="OK"}[5m])
            )
            /
            sum by (grpc_method, job) (
              rate(grpc_server_call_duration_seconds_count[5m])
            )
          ) * 100 > 5
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "gRPC error rate high on {{ $labels.grpc_method }}"
          description: "Error rate is {{ $value | printf \"%.1f\" }}% for {{ $labels.grpc_method }}"
      # Alert when P99 latency exceeds 500ms
      - alert: GrpcHighP99Latency
        expr: |
          histogram_quantile(
            0.99,
            sum by (grpc_method, le) (
              rate(grpc_server_call_duration_seconds_bucket[5m])
            )
          ) > 0.5
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 500ms for {{ $labels.grpc_method }}"
          description: "P99 is {{ $value | printf \"%.3f\" }}s for {{ $labels.grpc_method }}"
      # Alert when a gRPC method receives zero traffic (possible service down)
      - alert: GrpcNoTraffic
        expr: |
          sum by (grpc_method, job) (
            rate(grpc_server_call_duration_seconds_count[10m])
          ) == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "No traffic on {{ $labels.grpc_method }} — possible service outage"

The 2-minute for duration on GrpcHighErrorRate prevents alert storms from a brief retry burst. For production services, adjust the threshold from 5% to match your SLO baseline — a payment service might alert at 1%, a background sync service at 10%.

If you are sending metrics to CubeAPM rather than a standalone Prometheus, you can build the same alert conditions through CubeAPM’s alerting UI without writing PromQL rules manually. The underlying infrastructure monitoring context from host and pod metrics is correlated automatically alongside the gRPC service metrics.

—

Troubleshooting Common Issues

Traces are not connected across services — each service shows a separate root span

The client interceptor injects trace context into gRPC metadata, and the server interceptor extracts it. If either interceptor is missing, the chain breaks. Check that both grpcTelemetry.newClientInterceptor() and grpcTelemetry.newServerInterceptor() are attached. Also confirm that the W3CTraceContextPropagator is registered:

import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.extension.trace.propagation.W3CTraceContextPropagator;
OpenTelemetrySdk.builder()
    .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
    // ... rest of config
    .buildAndRegisterGlobal();

Prometheus is showing no gRPC metrics

The PrometheusHttpServer needs to be registered on the SdkMeterProvider before the gRPC interceptors attach. If OTel is initialized after the gRPC server starts, the meter provider may not be in place when the first RPC arrives. Always call OtelConfig.init() as the first line in main().

`grpc_status` label is missing from metrics

The grpc.status attribute is only populated on completed calls. Streaming RPCs may not report status until the stream closes. For server streaming methods, latency and status are only reported per-stream, not per-message.

`DEADLINE_EXCEEDED` errors appearing under load but not in tests

This is almost always a thread pool saturation issue in the gRPC executor, not an application bug. The default gRPC server executor is a cached thread pool with no backpressure. Under sustained load it can queue RPCs until callers time out. Add a span attribute for thread.pool.queue_depth or check JVM thread metrics alongside the gRPC latency histogram. Understanding this connection between JVM runtime metrics and gRPC latency is exactly the kind of signal that proper application performance monitoring surfaces automatically when traces and host metrics are correlated in one view.

Spans are created but never exported — collector connection refused

The BatchSpanProcessor buffers spans and exports them in the background. If the collector is unreachable at startup, the exporter will fail silently and drop spans. Enable the SimpleSpanProcessor temporarily during local development so failures are synchronous and visible. Switch back to BatchSpanProcessor before deploying to production.

—

Conclusion

Instrumenting a Java gRPC service end to end takes four concrete pieces: the opentelemetry-grpc-1.6 interceptors attached to both server and client, a MeterProvider with a Prometheus or OTLP metrics reader, PromQL queries for P95/P99 latency and error rate, and alerting rules that fire before users report problems. The custom span work in Step 6 is where the real debugging value comes from in production — the RPC duration alone rarely tells you which internal operation caused a latency spike.

The setup described here exports all telemetry in standard OpenTelemetry format, which means you are not tied to any specific backend. Whether you route to Jaeger, Grafana Tempo, or a platform like CubeAPM that correlates traces with logs and infrastructure metrics in one view, the instrumentation code stays identical.

—

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

—

Frequently Asked Questions