Go applications have a reputation for being straightforward to operate. The binary is self-contained, startup is fast, memory usage is predictable, and the runtime is lean.
But when things go wrong in production, Go’s simplicity can work against you. There are no stack traces on panics unless you explicitly recover them. Goroutine leaks accumulate silently until the process runs out of memory. A slow database query blocks the goroutine handling that request but leaves all other goroutines unaffected, so error rates stay flat while p99 latency climbs. And because Go does not have a JVM or a managed runtime with built-in profiling hooks, instrumenting the application is entirely the developer’s responsibility.
Observability for Go means having structured logs that write JSON and carry trace context, metrics that track goroutine count, memory allocations, and custom business signals, and distributed traces that show which function call, database query, or downstream HTTP call caused a specific slow request. The OpenTelemetry Go SDK is the standard instrumentation framework for all three. The current stable release is v1.42.0 (March 2026), requiring Go 1.25 or higher.
This guide covers structured logging with Go’s slog package, setting up the OTel Go SDK, manual tracing, custom metrics, auto-instrumentation libraries for common frameworks, and sending all three signals to CubeAPM.
Key Takeaways
- The OTel Go SDK v1.42.0 (March 2026) requires Go 1.25 or higher; Go 1.24 support was dropped in v1.42.0, with v1.41.0 (March 2, 2026) being the last release supporting Go 1.24.
- Traces, metrics, and logs APIs and SDKs are all stable in v1.42.0; use semconv/v1.40.0 which contains breaking changes to RPC semantic conventions from v1.39.0.
- The Zipkin exporter (go.opentelemetry.io/otel/exporters/zipkin) is deprecated; migrate to OTLP.
- Always call defer span.End() immediately after tracer.Start(); a span that is never ended will not be exported.
- There is no zero-code agent comparable to the Java or Python agent for Go; the eBPF-based opentelemetry-go-instrumentation requires a privileged process and supports a limited set of frameworks.
- Track goroutine count via Int64ObservableGauge reading runtime.NumGoroutine(); a monotonically growing goroutine count is the primary signal of a goroutine leak.
- Always pass context.Context through function calls and use slog.InfoContext(ctx, …) to enable trace context injection into log records.
- grpc.NewClient is the current gRPC connection function; grpc.Dial is deprecated.
Why Go Observability Has Specific Challenges
- Goroutine leaks are invisible without metrics: A goroutine blocked on a channel receive that will never fire, or a goroutine waiting on a context that was never cancelled, accumulates silently. Go’s runtime exposes the current goroutine count via runtime.NumGoroutine(). Tracking this as a metric over time is the only reliable way to detect goroutine leaks before they exhaust process memory.
- Panics do not produce traces without explicit recovery: An unrecovered panic in a goroutine crashes the entire process with a stack trace written to stderr. A recovered panic inside an HTTP handler silently returns a 500 without any span recording the error unless span.RecordError() is called inside the recovery function. Standard Go HTTP middleware often swallows these, making them invisible to both logs and traces unless explicitly instrumented.
- There is no zero-code agent comparable to the Java or Python agent: The OpenTelemetry Go instrumentation project (opentelemetry-go-instrumentation) provides eBPF-based auto-instrumentation without code changes, but it requires running as a privileged process alongside the Go binary and supports a more limited set of frameworks than the Java or Python agents. For most production Go applications, instrumentation is still code-based. The contrib repository (opentelemetry-go-contrib) provides instrumentation packages for popular frameworks and libraries that reduce the amount of manual code needed.
Step 1: Structured Logging with slog
Go 1.21 introduced log/slog as the standard library’s structured logging package. It is the recommended logging approach for new Go applications and does not require any third-party dependency.
Set up a JSON handler at application startup:
package main
import (
"log/slog"
"os"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
}))
slog.SetDefault(logger)
}Use structured logging with key-value pairs throughout the application:
slog.Info("processing order",
slog.String("order_id", orderID),
slog.String("user_id", userID),
slog.String("stage", "validate"),
)This produces JSON output with every field indexable:
{
"time": "2026-06-09T10:22:31.412Z",
"level": "INFO",
"msg": "processing order",
"order_id": "ord_123",
"user_id": "usr_456",
"stage": "validate"
}Connecting logs to traces: Inject the active trace ID and span ID into log records so you can jump directly from a log event to the trace that produced it. Use a custom slog.Handler that reads the active span from the context:
import (
"context"
"log/slog"
"go.opentelemetry.io/otel/trace"
)
type traceHandler struct {
slog.Handler
}
func (h *traceHandler) Handle(ctx context.Context, r slog.Record) error {
if span := trace.SpanFromContext(ctx); span.SpanContext().IsValid() {
sc := span.SpanContext()
r.AddAttrs(
slog.String("trace_id", sc.TraceID().String()),
slog.String("span_id", sc.SpanID().String()),
)
}
return h.Handler.Handle(ctx, r)
}
// Wrap the JSON handler
logger := slog.New(&traceHandler{slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
})})
slog.SetDefault(logger)Always pass context.Context through function calls in Go and use slog.InfoContext(ctx, …) rather than slog.Info(…) to enable trace context injection:
slog.InfoContext(ctx, "processing order",
slog.String("order_id", orderID),
slog.String("user_id", userID),
)Step 2: Install the OpenTelemetry Go SDK
The OTel Go SDK v1.42.0 requires Go 1.25 or higher. Go 1.24 support was dropped in v1.42.0 (March 6, 2026); v1.41.0 (March 2, 2026) was the last release supporting Go 1.24.
Add the required modules:
go get go.opentelemetry.io/[email protected]
go get go.opentelemetry.io/otel/[email protected]
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/[email protected]
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/[email protected]
go get go.opentelemetry.io/otel/exporters/otlp/otlplog/[email protected]
go get go.opentelemetry.io/otel/sdk/[email protected]
go get go.opentelemetry.io/otel/sdk/[email protected]Signal stability in Go SDK v1.42.0:
- Traces API and SDK: stable
- Metrics API and SDK: stable
- Logs API and SDK: stable
Note on Zipkin exporter: go.opentelemetry.io/otel/exporters/zipkin is deprecated. Migrate to OTLP.
Note on semconv breaking changes in v1.39.0: The semconv/v1.39.0 package introduced breaking changes to RPC semantic conventions: rpc.system is renamed to rpc.system.name, rpc.method and rpc.service are merged into a single fully-qualified rpc.method attribute, and rpc.client|server.duration is renamed to rpc.client|server.call.duration with the unit changed to seconds. These changes are reflected in the contrib instrumentation packages. Review the migration docs if upgrading from semconv v1.37.0 or earlier.
Step 3: Initialize the OTel SDK
Create a dedicated telemetry.go file that sets up all three signal providers at application startup:
package telemetry
import (
"context"
"fmt"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
"go.opentelemetry.io/otel/propagation"
sdklog "go.opentelemetry.io/otel/sdk/log"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
sdkresource "go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
otellog "go.opentelemetry.io/otel/log/global"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func Setup(ctx context.Context, serviceName, serviceVersion, otlpEndpoint string) (shutdown func(context.Context) error, err error) {
conn, err := grpc.NewClient(otlpEndpoint,
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
return nil, fmt.Errorf("creating gRPC connection: %w", err)
}
res, err := sdkresource.New(ctx,
sdkresource.WithAttributes(
semconv.ServiceName(serviceName),
semconv.ServiceVersion(serviceVersion),
),
sdkresource.WithProcess(),
sdkresource.WithOS(),
)
if err != nil {
return nil, fmt.Errorf("creating resource: %w", err)
}
// Traces
traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("creating trace exporter: %w", err)
}
tracerProvider := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(traceExporter),
sdktrace.WithResource(res),
sdktrace.WithSampler(sdktrace.AlwaysSample()),
)
otel.SetTracerProvider(tracerProvider)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
))
// Metrics
metricExporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("creating metric exporter: %w", err)
}
meterProvider := sdkmetric.NewMeterProvider(
sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter)),
sdkmetric.WithResource(res),
)
otel.SetMeterProvider(meterProvider)
// Logs
logExporter, err := otlploggrpc.New(ctx, otlploggrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("creating log exporter: %w", err)
}
loggerProvider := sdklog.NewLoggerProvider(
sdklog.WithProcessor(sdklog.NewBatchProcessor(logExporter)),
sdklog.WithResource(res),
)
otellog.SetLoggerProvider(loggerProvider)
return func(ctx context.Context) error {
var errs []error
if err := tracerProvider.Shutdown(ctx); err != nil {
errs = append(errs, err)
}
if err := meterProvider.Shutdown(ctx); err != nil {
errs = append(errs, err)
}
if err := loggerProvider.Shutdown(ctx); err != nil {
errs = append(errs, err)
}
if len(errs) > 0 {
return fmt.Errorf("shutdown errors: %v", errs)
}
return nil
}, nil
}Call this at the start of main():
func main() {
ctx := context.Background()
shutdown, err := telemetry.Setup(ctx,
"my-go-service",
"1.0.0",
os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
)
if err != nil {
log.Fatalf("setting up telemetry: %v", err)
}
defer func() {
if err := shutdown(ctx); err != nil {
slog.Error("shutting down telemetry", slog.Any("error", err))
}
}()
}Note on insecure.NewCredentials(): Use only within a private network. For production deployments sending telemetry over the public internet, configure TLS credentials instead.
Step 4: Manual Tracing
Get a tracer and create spans for important operations:
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
)
var tracer = otel.Tracer("my-go-service")
func processOrder(ctx context.Context, orderID string, items []Item) (*PricingResult, error) {
ctx, span := tracer.Start(ctx, "process_order",
trace.WithAttributes(
attribute.String("order.id", orderID),
attribute.Int("order.item_count", len(items)),
),
)
defer span.End()
result, err := calculatePricing(ctx, items)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return nil, fmt.Errorf("calculating pricing: %w", err)
}
span.SetAttributes(attribute.Float64("pricing.total", result.Total))
return result, nil
}Important: Always call defer span.End() immediately after tracer.Start(). A span that is never ended will not be exported. Passing ctx into child function calls enables automatic parent-child span relationships.
Recovering panics within a span:
func handleRequest(ctx context.Context, req *Request) (resp *Response, err error) {
ctx, span := tracer.Start(ctx, "handle_request")
defer span.End()
defer func() {
if r := recover(); r != nil {
span.RecordError(fmt.Errorf("panic: %v", r))
span.SetStatus(codes.Error, fmt.Sprintf("panic: %v", r))
err = fmt.Errorf("internal error")
}
}()
return processRequest(ctx, req)
}Step 5: Auto-Instrumentation Libraries
The opentelemetry-go-contrib repository provides instrumentation packages for common Go frameworks and libraries. The contrib packages track the same Go version requirement as the core SDK: Go 1.25+ for the latest releases.
# HTTP server and client
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
# gRPC
go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
# Gin
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
# Echo
go get go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho
# database/sql
go get go.opentelemetry.io/contrib/instrumentation/database/sql/otelsqlnet/http server instrumentation:
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
mux := http.NewServeMux()
mux.HandleFunc("/orders", handleOrders)
handler := otelhttp.NewHandler(mux, "http-server")
http.ListenAndServe(":8080", handler)net/http client instrumentation:
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
client := &http.Client{
Transport: otelhttp.NewTransport(http.DefaultTransport),
}
req, _ := http.NewRequestWithContext(ctx, "GET", "https://api.example.com/data", nil)
resp, err := client.Do(req)What contrib instrumentation covers:
| Package | What is instrumented |
| otelhttp | net/http server request spans and client call spans |
| otelgrpc | gRPC server and client call spans |
| otelgin | Gin framework route-level spans |
| otelecho | Echo framework route-level spans |
| otelsql | database/sql query spans with sanitized SQL |
| otelmongo | MongoDB operation spans |
| otelaws | AWS SDK v2 service call spans |
Step 6: Custom Metrics
Get a meter and create instruments for business-level metrics:
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
)
var meter = otel.Meter("my-go-service")
ordersProcessed, _ := meter.Int64Counter("orders.processed",
metric.WithDescription("Total number of orders processed"),
metric.WithUnit("1"),
)
processingDuration, _ := meter.Int64Histogram("orders.processing.duration",
metric.WithDescription("Time to process an order"),
metric.WithUnit("ms"),
)
activeJobs, _ := meter.Int64UpDownCounter("jobs.active",
metric.WithDescription("Number of jobs currently being processed"),
metric.WithUnit("1"),
)Record values with attributes:
func processOrder(ctx context.Context, order *Order) error {
start := time.Now()
defer func() {
processingDuration.Record(ctx,
time.Since(start).Milliseconds(),
metric.WithAttributes(attribute.String("order.type", order.Type)),
)
}()
err := doProcessOrder(ctx, order)
if err != nil {
return err
}
ordersProcessed.Add(ctx, 1,
metric.WithAttributes(
attribute.String("order.type", order.Type),
attribute.String("region", order.Region),
),
)
return nil
}Observable gauge for runtime metrics like goroutine count:
_, _ = meter.Int64ObservableGauge("runtime.goroutines",
metric.WithDescription("Current number of goroutines"),
metric.WithUnit("1"),
metric.WithInt64Callback(func(_ context.Context, o metric.Int64Observer) error {
o.Observe(int64(runtime.NumGoroutine()))
return nil
}),
)Step 7: Monitor Go Applications with CubeAPM
CubeAPM receives all three signal types from Go applications via OpenTelemetry: traces, metrics, and logs. Setting OTEL_EXPORTER_OTLP_ENDPOINT to your CubeAPM instance is the only change required from the standard OTel setup above.
What CubeAPM monitors for Go applications:
- HTTP request rate, error rate, and p99 latency per route (via otelhttp)
- gRPC call spans with service name, method, and status code
- Database query spans with sanitized SQL (via otelsql)
- AWS SDK service call spans (via otelaws)
- Goroutine count trend via observable gauge
- Custom business metrics (counters, histograms, up-down counters)
- Structured JSON logs from slog correlated to traces via injected trace ID and span ID
- Distributed traces across Go microservices with end-to-end flame graphs
Key alerts to configure for Go applications in CubeAPM:
| Alert | Condition | Severity |
| High error rate | HTTP error rate > 1% for 5 min | Warning |
| High p99 latency | p99 request duration > 2,000 ms | Warning |
| Goroutine leak | runtime.goroutines growing monotonically over 30 min | Warning |
| High goroutine count | runtime.goroutines > 10,000 | Warning |
| Slow database queries | Any otelsql span > 500 ms | Warning |
| Custom metric threshold | orders.processed rate drops to 0 | Critical |
| Panic recovered | Span with error attribute from panic recovery present | Critical |
Read the docs to configure OTLP ingestion and Go application monitoring.
Summary
Go observability requires explicit instrumentation. The OTel Go SDK v1.42.0 provides stable APIs for all three signal types. Use slog with a custom trace context handler for structured logs, manual spans plus contrib packages for traces, and observable and synchronous instruments for metrics.
| Signal | Collection method | Key data |
| Structured logs | log/slog + traceHandler wrapping slog.NewJSONHandler | JSON log lines with trace_id and span_id injected from context |
| Distributed traces | OTel Go SDK manual tracing + contrib packages (otelhttp, otelgrpc, otelsql) | HTTP, gRPC, SQL spans; panic recovery spans |
| Metrics | OTel Metrics API (Int64Counter, Int64Histogram, Int64ObservableGauge) | Request rate, latency, goroutine count, custom business metrics |
| Auto-instrumentation | opentelemetry-go-contrib per-framework packages | net/http, gRPC, Gin, Echo, database/sql, MongoDB, AWS SDK |
Disclaimer: All OTel Go package paths and API calls sourced from the official OTel Go documentation at opentelemetry.io/docs/languages/go/ and Go Packages (pkg.go.dev), verified June 2026. Current stable release: go.opentelemetry.io/otel v1.42.0 (March 6, 2026). Requires Go 1.25 or higher; Go 1.24 support was dropped in v1.42.0 (source: github.com/open-telemetry/opentelemetry-go/releases). v1.41.0 (March 2, 2026) was the last release supporting Go 1.24. semconv/v1.40.0 is the latest semantic conventions package in v1.42.0. Breaking changes in semconv v1.39.0 affect RPC attribute naming; see migration docs at github.com/open-telemetry/opentelemetry-go/tree/main/semconv/v1.40.0/MIGRATION.md. Zipkin exporter (go.opentelemetry.io/otel/exporters/zipkin) is deprecated; migrate to OTLP. grpc.NewClient is the current gRPC dial function (replacing deprecated grpc.Dial). insecure.NewCredentials() disables TLS and should only be used within private networks. log/slog is available from Go 1.21+. OTel Go auto-instrumentation via eBPF (opentelemetry-go-instrumentation) requires a privileged process and has more limited framework coverage than code-based instrumentation. CubeAPM: $0.15/GB, no per-service or per-host fees.
Also read:
Observability for Java Applications: A Complete Guide
Observability for Node.js Applications: Logs, Metrics, and Traces
Observability for Python Applications: Logs, Metrics, and Traces