Prometheus Memory Usage High: Causes and Fixes

Prometheus memory consumption often catches teams off guard. A monitoring server that starts at 8 GB can balloon to 128 GB within months as infrastructure scales. Engineers on Reddit have documented Prometheus pods hitting 960 GB memory limits with 1,200 ServiceMonitors before crashing with OOM errors. The standard advice “just add more RAM” only delays the problem and inflates cloud bills.

This guide covers why Prometheus uses so much memory, the specific factors that drive consumption higher, and the concrete fixes that reduce memory usage without losing monitoring coverage. If your Prometheus instance is already struggling, the fixes in this article can cut memory usage by 30 to 60 percent without sampling away the traces that matter during incidents.

What Is Prometheus Memory Usage and Why It Matters

Prometheus memory usage refers to the amount of RAM the Prometheus server process consumes while collecting, storing, and querying time series metrics. Unlike traditional databases that write data to disk immediately, Prometheus keeps recent metrics in memory for fast query access. This in-memory approach delivers the low latency queries that make Prometheus popular, but it also means memory consumption scales directly with the number of active time series being monitored.

Memory usage matters for three reasons. First, running out of memory triggers OOM kills, restarting Prometheus and losing recent metric data. Second, higher memory usage means larger cloud instances, which increases monthly infrastructure costs. Third, excessive memory pressure slows query performance, making dashboards and alerts unreliable during incidents when teams need them most.

Prometheus stores metrics in a time series database (TSDB) organized into blocks. Each unique combination of metric name and label set creates a separate time series. More time series means more memory. A cluster with 50 nodes and 10 labels per metric can generate 500,000 active time series. At roughly 3 KB per series, that is 1.5 GB of memory before any queries run.

The memory footprint grows further when queries execute. Complex PromQL queries that aggregate across thousands of series or perform rate calculations over long time windows require additional memory to process results. Query-induced spikes can temporarily double or triple memory usage, triggering OOM errors even when baseline memory consumption looks normal.

Prometheus memory usage typically breaks into three categories: head block memory for recent samples, query processing memory for active PromQL evaluations, and WAL (write-ahead log) memory for buffering incoming scrapes before they flush to disk. The head block consumes the largest share, usually 60 to 80 percent of total memory.

How Prometheus Memory Consumption Works

Prometheus memory consumption follows a predictable pattern tied to its TSDB architecture. Every metric Prometheus scrapes becomes a data point in a time series. Each time series is uniquely identified by its metric name and label set. Prometheus stores these series in memory until enough samples accumulate to form a block, which then gets written to disk and compressed.

The head block stores the most recent two hours of data by default. This block lives entirely in memory to enable fast queries. Every active time series in the head block consumes approximately 3 KB of RAM. That figure includes overhead for storing timestamps, sample values, and internal indexing structures. As scrape intervals decrease or the number of monitored targets increases, more time series stay active, pushing memory usage higher.

Label cardinality drives memory consumption more than any other factor. Cardinality refers to the number of unique combinations of label values. A metric with labels for pod name, namespace, and node across 1,000 pods creates 1,000 time series for that metric alone. Adding a request_id label that changes with every request creates a new time series for every single request, causing cardinality to explode. This is why Prometheus documentation warns against high cardinality labels.

Scrape frequency also impacts memory. Scraping targets every 15 seconds instead of every 60 seconds generates four times as many samples in the same time window. More samples mean more data points stored in the head block, increasing memory consumption. Teams that reduce scrape intervals to capture finer granularity often see memory usage jump 3x or 4x without realizing the connection.

Query processing adds temporary memory spikes on top of baseline usage. When a PromQL query requests data spanning thousands of time series, Prometheus loads those series into memory, performs aggregations, and holds intermediate results until the query completes. Queries that calculate rates over long windows or aggregate across high cardinality dimensions can temporarily consume several gigabytes of RAM. If multiple queries run concurrently, those spikes stack, sometimes exceeding available memory and triggering OOM kills.

The WAL buffers incoming scrape data before writing to disk. Under normal load, WAL memory usage stays low. But during traffic spikes or when disk I/O slows, the WAL grows as Prometheus queues more samples waiting to flush. WAL growth rarely causes memory problems on its own, but it compounds issues when head block and query memory are already high.

Retention settings control how long Prometheus keeps data on disk, but they do not directly limit memory usage. The head block always holds the most recent two hours regardless of retention. Increasing retention from 15 days to 90 days increases disk usage but does not change memory consumption. This surprises teams who assume longer retention causes higher memory usage.

Main Causes of High Prometheus Memory Usage

High cardinality metrics are the number one cause of excessive Prometheus memory usage. Cardinality refers to the number of unique time series created by a metric. Each unique combination of labels produces a separate series. A metric with labels for user_id, request_id, and session_id can generate millions of series if each request creates new label values. Prometheus stores every active series in memory, so high cardinality directly translates to high memory consumption.

Real-world example: A Kubernetes cluster monitoring 1,200 ServiceMonitors with 1:1 target-to-endpoint mapping can create 50 million time series. Each series consumes roughly 3 KB of RAM. That alone requires 150 GB of memory before accounting for queries or WAL overhead. Optimizing ServiceMonitors from 1:1 to 1:many reduced active series and cut memory usage by over 50 percent.

Frequent scraping intervals multiply memory usage. Scraping every 10 seconds instead of every 60 seconds produces six times as many data points in the same retention window. More samples mean more data in the head block. Teams that set aggressive scrape intervals to capture granular metrics often see memory usage climb 4x to 6x compared to a standard 30 second interval.

Large numbers of monitored targets scale memory linearly. Each target Prometheus scrapes generates metrics, and each metric with unique labels creates new time series. Monitoring 500 pods with 20 metrics per pod at 10 label combinations per metric creates 100,000 active series just from pod-level metrics. Add node metrics, service mesh metrics, and application metrics, and total series count can exceed 500,000 in a mid-size cluster.

Complex PromQL queries cause temporary memory spikes. Queries that aggregate across thousands of series or calculate rates over long time ranges load all relevant series into memory. A query like rate(http_requests_total[5m]) over 10,000 series pulls every series into RAM, performs calculations, and holds results until the query completes. If dashboards run multiple such queries concurrently, memory spikes stack, sometimes exceeding limits and triggering OOM kills.

Poorly designed ServiceMonitors and PodMonitors in Kubernetes environments generate unnecessary time series. A ServiceMonitor that scrapes every endpoint individually instead of grouping by service creates redundant series. Labels like pod_ip, container_id, or request_id that change with every scrape produce unbounded cardinality. These labels should be dropped at scrape time using relabel_configs, but many default configurations leave them enabled.

Insufficient relabeling and metric filtering let low-value metrics consume memory. Prometheus scrapes everything exposed by a target unless explicitly filtered. Applications that expose debugging metrics, per-request metrics, or metrics with high cardinality labels flood Prometheus with series that rarely get queried. Without relabel rules to drop or aggregate these metrics, they accumulate in memory and disk.

Recording rules that create new high cardinality series compound the problem. Recording rules pre-compute expensive queries and store the results as new metrics. If the rule output includes high cardinality labels, it generates additional series on top of the originals. A recording rule that computes per-pod request rates while keeping pod-level labels intact can double the number of active series.

Kubernetes environments with dynamic workloads scale memory unpredictably. Auto-scaling clusters that spin up 100 pods during peak traffic create 100 new scrape targets instantly. Each pod exposes metrics, and each metric generates time series. Memory usage spikes in sync with scaling events. Teams that provision Prometheus memory based on baseline load hit OOM errors when traffic peaks, even though the peak is expected behavior.

Retention configuration does not limit memory, only disk usage. Increasing retention from 15 days to 90 days increases disk consumption but does not reduce memory usage. The head block always holds the most recent data regardless of retention settings. Teams that assume reducing retention will free memory are mistaken. Memory usage is driven by active series count and scrape frequency, not retention duration.

Specific Fixes to Reduce Prometheus Memory Usage

Reduce metric cardinality by removing or aggregating high cardinality labels. The fastest way to cut memory usage is to identify metrics with unbounded cardinality and fix them at the source. Labels like request_id, user_id, session_id, and trace_id create a new time series for every unique value. Drop these labels using Prometheus relabel_configs at scrape time. If the labels are needed for queries, aggregate them in recording rules instead of storing every combination.

Example relabel config to drop high cardinality labels:

scrape_configs:
  - job_name: 'kubernetes-pods'
    relabel_configs:
      - source_labels: [__name__]
        regex: 'http_requests_total'
        action: keep
      - regex: 'request_id|session_id|trace_id'
        action: labeldrop

Increase scrape intervals to reduce sample volume. Changing the scrape interval from 15 seconds to 60 seconds cuts the number of samples by 75 percent. For most metrics, 60 second granularity is sufficient. Reserve 15 second intervals for critical services where sub-minute resolution matters. Use per-job scrape intervals in Prometheus config to apply different intervals to different targets.

Example config with longer scrape interval:

scrape_configs:
  - job_name: 'kubernetes-pods'
    scrape_interval: 60s
  - job_name: 'critical-services'
    scrape_interval: 15s

Optimize Kubernetes ServiceMonitors to reduce endpoint count. A ServiceMonitor that creates one scrape target per pod can be refactored to scrape at the service level instead. This reduces the number of active targets and the associated overhead. Kubernetes service discovery generates separate targets for every endpoint unless configured otherwise. Grouping by service reduces memory consumption by consolidating scrapes.

Drop unused metrics at scrape time using metric_relabel_configs. Most applications expose metrics that are never queried. Identify these using Prometheus queries to find metrics with zero queries over 30 days, then drop them. Dropping unused metrics reduces series count without losing visibility.

Example to drop specific metrics:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_gc_.*|process_.*'
    action: drop

Tune TSDB settings to reduce memory overhead. The storage.tsdb.min-block-duration and storage.tsdb.max-block-duration flags control how often Prometheus writes head block data to disk. Reducing the head block retention window from 2 hours to 1 hour cuts memory usage by roughly 50 percent. However, this increases disk write frequency, so test before applying in production.

Example command line flag:

--storage.tsdb.min-block-duration=1h
--storage.tsdb.max-block-duration=1h

Use recording rules to pre-aggregate high cardinality queries. Recording rules compute expensive queries at regular intervals and store the results as new metrics. This shifts memory cost from query time to scrape time. For dashboards that query the same high cardinality aggregation repeatedly, recording rules cut query-induced memory spikes.

Example recording rule:

groups:
  - name: example
    interval: 60s
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

Enable Prometheus remote write to offload older data. Remote write sends metrics to external storage systems like AWS Lambda Monitoring or Thanos. This lets Prometheus retain only recent data locally while querying historical data from remote storage. Memory usage drops because only the most recent metrics stay in the head block.

Federation reduces memory by distributing scrape load across multiple Prometheus instances. A single Prometheus instance scraping 5,000 targets can be split into five instances scraping 1,000 targets each. A central Prometheus instance federates aggregated metrics from the edge instances. This distributes memory load and prevents a single instance from hitting resource limits.

Limit query complexity and concurrency. Queries that aggregate across 100,000 series or calculate rates over 24 hour windows consume large amounts of memory. Restrict these queries using query timeouts and result size limits. The --query.max-samples flag limits the number of samples a query can process, preventing memory exhaustion from runaway queries.

Example flag to limit query samples:

--query.max-samples=50000000

Horizontal pod autoscaling in Kubernetes can create memory spikes. When clusters auto-scale from 20 to 200 pods, Prometheus memory usage spikes as it scrapes 180 new targets. Pre-provision Prometheus memory based on peak cluster size, not average size. Alternatively, use federation so edge Prometheus instances scale with the cluster while the central instance stays stable.

Monitoring Prometheus Memory Usage

Track Prometheus memory usage by monitoring specific metrics exposed by the Prometheus /metrics endpoint. The most important metric is process_resident_memory_bytes, which shows total memory consumed by the Prometheus process. This metric includes all memory used by TSDB, queries, and internal operations. Plotting this metric over time reveals trends and spikes that indicate when memory usage is growing unsustainably.

The prometheus_tsdb_head_series metric shows the number of active time series stored in the head block. This is the single best indicator of baseline memory consumption. Each series consumes roughly 3 KB, so if this metric shows 500,000 series, expect at least 1.5 GB of memory usage just from the head block. Watch for sudden increases that indicate cardinality problems.

Query topk(10, count by (__name__)({__name__=~".+"})) to identify the top 10 metrics by series count. This query reveals which metrics contribute most to cardinality. If a single metric accounts for 40 percent of all series, it is a prime candidate for cardinality reduction.

Monitor prometheus_engine_query_duration_seconds to track query performance. Long query durations often correlate with high memory usage. Queries that take more than 10 seconds are likely loading large volumes of data into memory. Identify these queries and optimize them using recording rules or by reducing the time range.

The prometheus_tsdb_head_samples_appended_total metric shows how many samples Prometheus ingests per second. Multiply this by the scrape interval to estimate memory growth rate. If this metric increases suddenly, it indicates new targets or higher metric volumes, both of which drive memory usage higher.

Set up alerts for memory usage thresholds. An alert that fires when process_resident_memory_bytes exceeds 80 percent of available memory gives time to investigate before OOM kills occur. Combine this with alerts on prometheus_tsdb_head_series to catch cardinality explosions early.

Example alert rule:

groups:
  - name: prometheus_memory
    rules:
      - alert: PrometheusHighMemory
        expr: process_resident_memory_bytes / node_memory_MemTotal_bytes > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus memory usage above 80%"

Use Grafana dashboards to visualize memory metrics over time. The Prometheus 2.0 Stats dashboard provides pre-built panels for memory usage, active series, and query performance. Monitoring these metrics in real time helps teams understand how configuration changes impact memory consumption.

Tools and Alternatives for High-Scale Prometheus Environments

Thanos extends Prometheus by adding long-term storage and global query capabilities. Thanos Sidecar runs alongside Prometheus and uploads blocks to object storage like S3. This offloads historical data, reducing local memory and disk usage. Thanos Query provides a unified query interface across multiple Prometheus instances, enabling queries that span clusters without federation overhead. Thanos is open source and widely adopted in large Kubernetes environments.

Cortex provides horizontally scalable Prometheus-compatible storage. Unlike Prometheus, which scales vertically, Cortex distributes metric storage and queries across multiple nodes. This removes the single-instance memory limit. Cortex is particularly useful for multi-tenant environments where different teams need isolated Prometheus endpoints with shared backend storage. However, Cortex requires significant operational expertise to deploy and maintain.

VictoriaMetrics is a high-performance time series database that claims 10x better resource efficiency than Prometheus. VictoriaMetrics compresses data more aggressively and handles high cardinality better than Prometheus TSDB. It supports Prometheus query API, so existing dashboards and alerts work without modification. VictoriaMetrics can run as a single node or as a cluster for horizontal scaling. Teams report memory reductions of 50 to 70 percent after switching from Prometheus to VictoriaMetrics.

Grafana Mimir is a horizontally scalable, multi-tenant time series database built by Grafana Labs. Mimir uses the same data format as Cortex but with improved performance and easier operations. Mimir distributes query load and storage across nodes, removing per-instance memory limits. Grafana Cloud offers Mimir as a managed service, eliminating operational overhead. Self-hosted Mimir requires Kubernetes and object storage.

CubeAPM provides full-stack observability with native support for Prometheus metrics alongside traces and logs. Unlike SaaS tools that charge per host or per user, CubeAPM uses a flat $0.15/GB ingestion model. CubeAPM runs on-premises or in your VPC, keeping metric data inside your infrastructure. This eliminates egress costs and data residency concerns. Teams running Prometheus at scale use CubeAPM to unify metrics, traces, and logs in one platform without losing Prometheus compatibility. CubeAPM supports OpenTelemetry-native ingestion and works with existing Prometheus exporters.

Datadog offers managed observability with Prometheus metric collection through the Datadog Agent. Datadog’s per-host pricing starts at $15/host/month for infrastructure monitoring and $31/host/month for APM. For a 50-host cluster, that is $2,300/month before adding logs, synthetics, or custom metrics. Datadog handles scaling and storage, but costs compound as infrastructure grows. High cardinality metrics still drive costs higher through increased ingestion volume.

New Relic ingests Prometheus metrics through OpenTelemetry or the New Relic Prometheus integration. New Relic uses a compute capacity unit (CCU) pricing model that varies based on query frequency and data volume. This makes cost forecasting difficult. A team processing 10 TB of metrics per month can see bills range from $5,000 to $15,000 depending on query patterns. New Relic is cloud-only, so teams with data residency requirements cannot use it.

Grafana Cloud provides managed Prometheus storage with usage-based pricing. Grafana Cloud charges $0.40/GB for metrics ingestion and $8/GB for long-term storage. A 10 TB/month workload costs roughly $4,000 for ingestion plus storage fees. Grafana Cloud includes dashboards, alerting, and integration with Loki and Tempo for logs and traces. Pricing scales predictably, but costs rise quickly at high ingestion volumes.

How to Choose Between Self-Hosted Prometheus, SaaS, and On-Prem Alternatives

Teams running Prometheus in production face three deployment models: self-hosted Prometheus with extensions like Thanos, SaaS platforms like Datadog or Grafana Cloud, or on-prem alternatives like CubeAPM or VictoriaMetrics. The right choice depends on team size, operational expertise, data residency requirements, and cost tolerance.

Self-hosted Prometheus with Thanos or Cortex works best for large engineering teams with Kubernetes expertise and strict data control requirements. This model keeps metric data entirely on-premises or in your VPC. Operational burden is high. Teams must manage Prometheus instances, configure storage backends, tune queries, and handle upgrades. Memory issues remain your responsibility. Self-hosting makes sense when egress costs, data residency, or vendor independence are top priorities. It does not make sense for small teams without dedicated platform engineers.

SaaS platforms like Datadog, New Relic, or Grafana Cloud remove operational overhead at the cost of higher long-term expense and reduced data control. These platforms handle scaling, storage, and query optimization. Setup is fast. Costs are predictable in early stages but compound as infrastructure grows. SaaS works well for teams under 50 hosts or teams that prioritize speed over cost. It stops making sense when monthly bills exceed $10,000 and your team has the skills to run infrastructure.

On-prem managed platforms like CubeAPM combine the data control of self-hosting with the managed experience of SaaS. CubeAPM runs inside your VPC but the vendor handles upgrades, tuning, and scaling. This removes Day 2 operational burden while keeping telemetry data on-premises. Pricing is ingestion-based ($0.15/GB), so costs scale predictably with data volume rather than host count. On-prem managed platforms work best for teams that need data residency, want to avoid SaaS lock-in, but lack the engineering bandwidth to run Prometheus at scale themselves.

VictoriaMetrics or Mimir self-hosted suit teams with strong Kubernetes and observability experience who want better resource efficiency than Prometheus. These tools require upfront investment to deploy and tune but deliver 50 to 70 percent memory savings compared to Prometheus. They work best when your team already runs Prometheus successfully but needs to scale beyond what a single instance can handle.

Decision framework: Use self-hosted Prometheus if you have platform engineers, strict data control requirements, and are comfortable managing upgrades. Use SaaS if you have fewer than 50 hosts, limited operational capacity, or need to move fast. Use on-prem managed platforms like CubeAPM if you need data residency, want predictable costs, and do not want to manage infrastructure. Use VictoriaMetrics or Mimir if you have Kubernetes expertise and need to scale beyond single-instance Prometheus limits.

Prometheus memory usage grows with cardinality and scrape frequency. Fixing it requires reducing high cardinality labels, increasing scrape intervals, optimizing ServiceMonitors, and using recording rules to pre-aggregate queries. For teams that cannot reduce cardinality further, CubeAPM as a Datadog alternative offers unified metrics, traces, and logs with flat ingestion pricing and on-prem deployment. Teams hitting memory limits should monitor prometheus_tsdb_head_series and process_resident_memory_bytes to track trends and set alerts before OOM kills occur.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions