Prometheus Cardinality Problems: How to Find and Fix Them

Cardinality is the number of unique time series in your Prometheus database. Each unique combination of metric name and label values creates a separate time series. When a single metric with five labels has 10 possible values per label, that creates 100,000 time series from one metric alone. According to CNCF’s 2023 observability survey, 59% of teams using Prometheus report storage and query performance issues at scale, with cardinality explosion being the most common root cause.

High cardinality crashes Prometheus by exhausting memory during ingestion, slowing queries to timeouts, and ballooning storage costs. The problem is silent until it breaks your monitoring, often during an incident when you need it most.

This guide walks through finding high cardinality metrics using Prometheus’s built-in TSDB status API and PromQL queries, fixing the most common label problems at the source, and implementing relabeling rules to prevent cardinality from exploding again. Every technique includes working examples tested on production Prometheus setups handling billions of samples per day.

Prerequisites

Before starting, you need:

• Prometheus 2.x or later running in production with active scraping
• Access to the Prometheus web UI at http://prometheus:9090
• kubectl access if running Prometheus in Kubernetes
• curl and jq installed for API queries
• Write access to Prometheus configuration files or ServiceMonitor CRDs

Step 1: Check Your Current Cardinality

Start by measuring total time series and identifying growth patterns. Prometheus exposes these metrics about itself.

Query total active time series in the Prometheus UI query box:

prometheus_tsdb_head_series

This returns the current count of active time series in memory. If this number exceeds 10 million on a single Prometheus instance with default resource limits, you are in high cardinality territory and query performance will degrade noticeably.

Check how fast new series are being created:

rate(prometheus_tsdb_head_series_created_total[5m])

If this rate is consistently above 1,000 new series per second, cardinality is growing faster than most Prometheus setups can handle sustainably.

Track series churn to see how many are being removed:

rate(prometheus_tsdb_head_series_removed_total[5m])

When creation rate far exceeds removal rate, storage and memory pressure builds continuously until Prometheus crashes or restarts.

Step 2: Find High Cardinality Metrics

The fastest way to find problem metrics is using Prometheus’s TSDB status API. This endpoint ranks metrics by series count without writing any PromQL.

Port forward to your Prometheus instance if running in Kubernetes:

kubectl port-forward -n monitoring prometheus-kube-prometheus-stack-prometheus-0 9090:9090

Query the TSDB status endpoint:

curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data'

The response includes four critical rankings:

• seriesCountByMetricName: Top metrics by total series count
• labelValueCountByLabelName: Labels with the most unique values
• memoryInBytesByLabelName: Labels consuming the most memory
• seriesCountByLabelValuePair: Most explosive label value combinations

Focus on seriesCountByMetricName first. The output looks like this:

{
  "seriesCountByMetricName": [
    {"name": "http_requests_total", "value": 2847293},
    {"name": "api_request_duration_seconds_bucket", "value": 1923847},
    {"name": "container_cpu_usage_seconds_total", "value": 892341}
  ]
}

Any metric with over 100,000 series is a cardinality problem. Any metric over 1 million series is a severe problem that will crash Prometheus during high traffic.

You can also rank metrics using PromQL:

topk(20, count by (__name__)({__name__=~".+"}))

This query counts series per metric and returns the top 20 offenders. Run this every week and track whether the top metrics are growing or stable.

Step 3: Identify Label Problems

High cardinality always comes from labels. Once you know which metrics are exploding, inspect their label combinations to find the root cause.

Check cardinality of a specific metric by label:

count by (endpoint) (http_requests_total)

If endpoint has 10,000 unique values, every additional label multiplies that number. Adding user_id with 50,000 values would create 500 million series.

Find labels with the most unique values across all metrics:

curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.labelValueCountByLabelName'

The most dangerous labels typically are:

• user_id, session_id, request_id — unbounded identifiers that grow with traffic
• path, endpoint, url — when not normalized, every unique URL becomes a series
• trace_id, span_id — distributed tracing IDs that should never be labels
• pod, container_id — in Kubernetes, when pods scale or churn, these multiply series

Check how many unique values a suspicious label has:

count(count by (user_id) (http_requests_total))

If this returns more than 10,000, user_id is causing cardinality explosion and must be removed or aggregated.

Step 4: Fix Label Problems at the Source

The best fix is preventing high cardinality labels from reaching Prometheus. This requires changing how your application or exporter emits metrics.

Problem: Unbounded user IDs in labels

Bad example that creates millions of series:

http_requests = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'user_id']  # user_id is unbounded
)

http_requests.labels(method='GET', endpoint='/api/orders', user_id='user_12345').inc()

Fix by removing user_id or bucketing it into user types:

http_requests = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'user_type']
)

def get_user_type(user_id):
    # Aggregate users into buckets
    if is_premium(user_id):
        return 'premium'
    elif is_trial(user_id):
        return 'trial'
    else:
        return 'free'

http_requests.labels(
    method='GET',
    endpoint='/api/orders',
    user_type=get_user_type(user_id)
).inc()

This reduces cardinality from millions of unique user IDs to three user types.

Problem: Path parameters in URL labels

Bad example where every unique URL creates a series:

httpRequests := prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    },
    []string{"method", "path", "status"},
)

// This creates a new series for every user ID and order ID
httpRequests.WithLabelValues("GET", "/users/123/orders/456", "200").Inc()
httpRequests.WithLabelValues("GET", "/users/789/orders/012", "200").Inc()

Fix by normalizing paths to route patterns:

func normalizePath(path string) string {
    // Replace numeric IDs with placeholders
    re := regexp.MustCompile(`/\d+`)
    return re.ReplaceAllString(path, "/{id}")
}

normalizedPath := normalizePath("/users/123/orders/456")
// Returns: "/users/{id}/orders/{id}"

httpRequests.WithLabelValues("GET", normalizedPath, "200").Inc()

This collapses millions of unique paths into dozens of route patterns.

Problem: Timestamp or UUID labels

Never use timestamps, UUIDs, or trace IDs as Prometheus labels:

# Bad: every request creates a new series
- metric: request_duration_seconds
  labels:
    request_id: "550e8400-e29b-41d4-a716-446655440000"
    timestamp: "2026-01-15T10:30:00Z"

Fix by removing these labels entirely. Use histogram buckets to measure duration distribution instead:

request_duration = Histogram(
    'request_duration_seconds',
    'Request duration in seconds',
    ['method', 'endpoint'],
    buckets=[0.1, 0.5, 1.0, 2.5, 5.0, 10.0]
)

request_duration.labels(method='GET', endpoint='/api/users').observe(0.34)

Histograms give you percentile queries without creating a series per request.

Step 5: Drop High Cardinality Labels Using Relabeling

If you cannot fix the source immediately, use Prometheus relabeling to drop problematic labels before ingestion.

Drop specific labels from all metrics

Add this to your Prometheus scrape config:

scrape_configs:
  - job_name: 'api-service'
    static_configs:
      - targets: ['api:9090']
    metric_relabel_configs:
      # Drop high cardinality labels
      - regex: '(user_id|session_id|request_id|trace_id)'
        action: labeldrop

This removes those four labels from every metric scraped from the api-service job.

Replace dynamic values with placeholders

Normalize path labels inside Prometheus:

metric_relabel_configs:
  # Replace numeric IDs in path label
  - source_labels: [path]
    regex: '(.*/)\d+(.*)'
    replacement: '${1}{id}${2}'
    target_label: path

This rule transforms /users/123/orders/456 into /users/{id}/orders/{id} at scrape time.

Add aggregated labels for grouping

Create a status_class label for better aggregation:

metric_relabel_configs:
  # Add status class for 2xx responses
  - source_labels: [status_code]
    regex: '2..'
    replacement: '2xx'
    target_label: status_class
  
  # Add status class for 4xx responses
  - source_labels: [status_code]
    regex: '4..'
    replacement: '4xx'
    target_label: status_class
  
  # Add status class for 5xx responses
  - source_labels: [status_code]
    regex: '5..'
    replacement: '5xx'
    target_label: status_class

Now you can query by status_class instead of hundreds of individual status codes.

Step 6: Drop Entire Metrics

Some metrics are not worth keeping. Drop them before ingestion to save memory and storage.

Drop metrics by name pattern:

metric_relabel_configs:
  # Drop debug metrics in production
  - source_labels: [__name__]
    regex: 'debug_.*'
    action: drop
  
  # Drop unused node exporter metrics
  - source_labels: [__name__]
    regex: 'node_(nf_conntrack_stat|netstat_.*6|timex_pps).*'
    action: drop

Drop metrics with specific label values:

metric_relabel_configs:
  # Drop metrics from non-production environments
  - source_labels: [__name__, environment]
    regex: '.*;(development|staging)'
    action: drop

This prevents dev and staging metrics from consuming production Prometheus resources.

Step 7: Set Cardinality Limits

Prometheus allows setting hard limits to prevent cardinality from crashing the instance.

Add these limits to your Prometheus config:

global:
  scrape_interval: 15s
  # Limit samples per scrape to 10,000
  sample_limit: 10000
  # Limit labels per sample to 30
  label_limit: 30
  # Limit label name length to 128 characters
  label_name_length_limit: 128
  # Limit label value length to 512 characters
  label_value_length_limit: 512

scrape_configs:
  - job_name: 'high-cardinality-app'
    # Override global limit for this job
    sample_limit: 5000
    static_configs:
      - targets: ['app:9090']

When a scrape exceeds sample_limit, Prometheus rejects the entire scrape and logs an error. This protects the instance but creates a blind spot. Monitor the prometheus_target_scrapes_exceeded_sample_limit_total metric to detect when limits are hit.

Step 8: Create Recording Rules to Pre-Aggregate

Recording rules compute expensive queries in advance and store the results as new time series with lower cardinality.

Create a recording rule to aggregate per-instance metrics to per-service:

groups:
  - name: cardinality_reduction
    interval: 1m
    rules:
      # Aggregate HTTP requests per service instead of per pod
      - record: service:http_requests:rate5m
        expr: |
          sum by (service, method, status_class) (
            rate(http_requests_total[5m])
          )
      
      # Aggregate error rate per service
      - record: service:http_errors:rate5m
        expr: |
          sum by (service) (
            rate(http_requests_total{status_class="5xx"}[5m])
          )

Dashboards and alerts should query service:http_requests:rate5m instead of the raw http_requests_total metric. This reduces cardinality in queries by 100x when you have hundreds of pods per service.

Step 9: Monitor Cardinality Continuously

Set up alerts to catch cardinality explosions before they crash Prometheus.

Alert when total series crosses a threshold:

groups:
  - name: cardinality_alerts
    rules:
      - alert: HighTotalCardinality
        expr: prometheus_tsdb_head_series > 8000000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus has {{ $value }} active series"
          description: "Total series count is above 8 million. Investigate high cardinality metrics."
      
      - alert: FastCardinalityGrowth
        expr: rate(prometheus_tsdb_head_series_created_total[5m]) > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus creating {{ $value }} series per second"
          description: "Series creation rate is unsustainable. Check for label explosions."

Alert when a specific metric crosses a cardinality threshold:

- alert: HighMetricCardinality
  expr: count by (__name__) ({__name__=~".+"}) > 100000
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Metric {{ $labels.__name__ }} has {{ $value }} series"
    description: "Metric cardinality is above 100k. Review labels."

These alerts catch cardinality problems hours before Prometheus runs out of memory.

Step 10: Monitoring CubeAPM Metrics with Prometheus

CubeAPM exposes Prometheus-compatible metrics for monitoring its own performance. If you run CubeAPM as part of your observability stack, scraping these metrics helps track trace ingestion, query performance, and storage health alongside your application metrics.

Add CubeAPM to your Prometheus scrape targets:

scrape_configs:
  - job_name: 'cubeapm'
    static_configs:
      - targets: ['cubeapm:9090']
    metric_relabel_configs:
      # Drop high cardinality trace IDs
      - regex: 'trace_id'
        action: labeldrop

CubeAPM uses OpenTelemetry natively and stores traces in ClickHouse, which keeps cardinality under control by design. Unlike Prometheus, ClickHouse handles high cardinality queries efficiently, so trace IDs and span IDs do not cause the same memory pressure. However, when exposing metrics about CubeAPM’s performance to Prometheus, dropping trace IDs from metric labels prevents Prometheus cardinality issues.

Troubleshooting Common Issues

Prometheus crashes with “out of memory” during high traffic

Symptom: Prometheus pod or process crashes with OOMKilled or out of memory errors during traffic spikes.

Cause: High cardinality metrics consume too much memory during ingestion. Each unique label combination requires memory for indexing.

Fix: Identify the top 5 metrics by series count using the TSDB status API. Drop or relabel the highest cardinality offenders. Increase memory limits only after reducing cardinality — adding memory without fixing labels just delays the crash.

Queries timeout or take over 30 seconds

Symptom: PromQL queries hang or return query processing would load too many samples errors.

Cause: Queries touching millions of series exhaust query engine limits.

Fix: Use recording rules to pre-aggregate high cardinality metrics. Query the recording rule output instead of raw metrics. Avoid queries like sum(rate(http_requests_total[5m])) without filtering by service or endpoint first.

Prometheus scrape times exceed scrape interval

Symptom: Prometheus logs show scrapes taking longer than the configured scrape interval, causing missed scrapes.

Cause: Exporter exposes too many metrics or Prometheus is overloaded.

Fix: Set sample_limit per job to reject oversized scrapes. Disable unused collectors at the exporter level. For node exporter, use --no-collector.arp, --no-collector.ipvs, and --no-collector.sockstat flags.

Relabeling rules are not applied

Symptom: High cardinality labels still appear in metrics after adding labeldrop rules.

Cause: Rules are placed in relabel_configs instead of metric_relabel_configs, or the regex pattern does not match.

Fix: Use metric_relabel_configs to modify metrics after scraping. Use relabel_configs to modify target labels before scraping. Test regex patterns with a tool like regex101.com before deploying.

Series churn is high but total series stays stable

Symptom: rate(prometheus_tsdb_head_series_created_total[5m]) and rate(prometheus_tsdb_head_series_removed_total[5m]) are both high, but prometheus_tsdb_head_series is stable.

Cause: Labels with dynamic values change frequently, creating and removing series continuously. Common with pod names in Kubernetes or short lived containers.

Fix: Drop or aggregate dynamic labels. Use recording rules to pre-aggregate at a stable level like service or namespace instead of pod.

High cardinality is a design problem, not a configuration problem. The only permanent fix is changing how metrics are instrumented at the source. Relabeling and limits buy time but do not solve the root cause.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions