Prometheus Storage: Local vs Remote, Retention and Compaction

Prometheus runs a high-performance time series database (TSDB) that keeps every scraped metric sample on disk. For most teams, this starts simple: Prometheus scrapes targets, writes samples to local storage, and you query recent data through Grafana. But as scrape intervals tighten, label cardinality climbs, and retention windows extend beyond the default 15 days, the storage layer stops being invisible. Disk usage grows faster than expected, compaction cycles slow queries, and restarts take longer than they should.

At that point, you are managing Prometheus TSDB directly deciding how long to retain data locally, whether to push samples to remote storage, and how compaction affects both query performance and disk overhead. According to the 2024 CNCF Survey, Prometheus is used by 61% of organizations for monitoring, making storage architecture a common production concern as teams scale.

This guide explains how Prometheus local storage works, when remote storage makes sense, how retention and compaction interact, and what trade-offs matter when choosing between architectures.

Quick Comparison: Local vs Remote Prometheus Storage

Feature	Local Storage	Remote Storage
Deployment	Built into Prometheus	External system (VictoriaMetrics, Thanos, Cortex, Mimir, cloud providers)
Scalability	Single node limit	Horizontally scalable
Durability	Risk of data loss on disk failure	Replicated and durable
Query Performance	Fast for recent data	Depends on backend, often slower for large ranges
Retention	Limited by disk size	Unlimited with object storage
Cost	Local disk only	Remote storage cost + egress fees
Complexity	Zero additional setup	Requires external system configuration
HA / Multi-DC	Not supported	Supported
Best For	Small to mid-scale, short retention	Long-term retention, multi-cluster aggregation, compliance

This comparison models standard deployment patterns. Actual performance and cost depend on ingestion rate, label cardinality, retention requirements, and infrastructure setup.

Prometheus Local Storage: How It Works

Prometheus writes all scraped samples to local storage by default. This is not an external database, Prometheus ships its own TSDB optimized for append-heavy writes, numeric samples, and label-based queries.

On-Disk Layout and Block Structure

Prometheus organizes data into time-based blocks, typically covering two hours each. Every block lives in its own directory under the data/ path and contains:

• Chunks: Compressed time series samples using delta encoding
• Index: Maps metric names and label pairs to series IDs
• Metadata: Block time range and basic statistics

New samples land first in an in-memory head block. Prometheus appends them to the write-ahead log (WAL) on disk simultaneously. If Prometheus crashes, it replays the WAL to rebuild recent state. Once enough data accumulates or the two-hour window passes, Prometheus cuts a new persistent block on disk.

A typical Prometheus data directory looks like this:

./data
├── 01BKGV7JBM69T2G1BGBGM6KB12
│   ├── chunks
│   │   └── 000001
│   ├── index
│   ├── meta.json
│   └── tombstones
├── 01BKGTZQ1SYQJTR4PB43C8PD98
│   ├── chunks
│   │   └── 000001
│   ├── index
│   ├── meta.json
│   └── tombstones
├── chunks_head
│   └── 000001
└── wal
    ├── 000000002
    └── checkpoint.00000001
        └── 00000000

Write-Ahead Log (WAL) and Durability

The WAL ensures durability for recent data that has not yet been flushed to a persistent block. Prometheus keeps at least three WAL files by default, and high-traffic servers may retain more to cover at least two hours of raw data. WAL files are significantly larger than compressed blocks because they contain uncompacted samples.

If Prometheus restarts, it replays the WAL to restore the in-memory head block. This makes recovery fast for most deployments, but WAL size and replay time grow with ingestion rate.

Compaction: Merging Blocks Over Time

Prometheus runs background compaction to merge smaller two-hour blocks into larger ones. Compaction creates blocks spanning up to 10% of the retention time or 31 days, whichever is smaller. This reduces the number of blocks Prometheus must query, keeps disk usage predictable, and removes tombstones left by series deletions.

Compaction happens asynchronously and requires temporary disk space while new blocks are written. If disk space runs low during compaction, Prometheus may fail to complete the process, leading to query slowdowns or ingestion pauses.

Compaction reduces block count and query overhead, but it requires headroom on disk. Teams should provision storage with at least 20% buffer beyond expected data volume.

Local Storage Retention Policies

Prometheus provides two retention flags:

• --storage.tsdb.retention.time: How long to keep data (default: 15 days)
• --storage.tsdb.retention.size: Maximum storage size in bytes (default: disabled)

If both are set, Prometheus deletes data when either limit is reached. Retention enforcement runs every two hours and removes entire blocks that fall outside the configured window.

A common mistake: setting retention too low for alerting or recording rule evaluation windows. If an alert checks a 7-day trend but retention is set to 5 days, the alert will fail.

Disk Space Calculation for Local Storage

Prometheus stores roughly 1–2 bytes per sample after compression. To estimate disk requirements:

disk_space = retention_seconds × samples_per_second × bytes_per_sample

For example:

• Retention: 30 days (2,592,000 seconds)
• Ingestion rate: 100,000 samples/second
• Bytes per sample: 1.5 (average after compression)

disk_space = 2,592,000 × 100,000 × 1.5 = 388.8 GB

Add 20–30% buffer for WAL overhead, compaction headroom, and operational safety.

When Local Storage Works Well

Local storage is the right choice when:

• Retention requirements are under 30 days
• A single Prometheus server can handle ingestion load
• Queries focus on recent data (last few hours to days)
• Disk failure is acceptable (with backups via snapshots)
• You want zero operational overhead for storage backends

Many production teams run Prometheus with 7–15 day retention locally and push samples to remote storage for long-term retention and cross-cluster aggregation.

Prometheus Remote Storage: When and Why to Use It

Remote storage extends Prometheus beyond the limits of local disk. Instead of storing all samples on the Prometheus server, you configure Prometheus to replicate samples to an external system over HTTP using the remote write protocol.

How Remote Write Works

Prometheus sends scraped samples to one or more remote endpoints in near real time. The remote system stores the data and serves it back via remote read when Prometheus queries data outside its local retention window.

A minimal prometheus.yml configuration:

remote_write:
  - url: "https://remote-storage.example.com/api/v1/write"
    queue_config:
      max_samples_per_send: 1000
      batch_send_deadline: 5s
remote_read:
  - url: "https://remote-storage.example.com/api/v1/read"
    read_recent: true

Prometheus continues to scrape targets and write to local storage as usual. Remote write happens asynchronously, so network latency or remote storage outages do not block scraping.

Common Remote Storage Backends

Backend	Type	Best For
VictoriaMetrics	Open source + commercial	High ingestion rate, low cost, long retention
Thanos	Open source	Multi-cluster federation, object storage (S3, GCS)
Cortex / Mimir	Open source	Multi-tenant, horizontally scalable, Kubernetes-native
AWS Managed Prometheus	Managed service	AWS-native, zero ops, pay per ingestion
Google Cloud Managed Prometheus	Managed service	GKE integration, pay per ingestion
Grafana Cloud	Managed service	Hosted Prometheus with Grafana dashboards

This estimate models standard setups. Actual costs depend on ingestion rate, retention, query volume, and cloud provider pricing.

Remote Storage Retention Strategy

With remote storage, local retention can be minimal — often 2 hours to 2 days. This keeps local disk usage low while remote storage handles long-term retention, compliance, and historical queries.

Example setup:

• Local retention: --storage.tsdb.retention.time=2h
• Remote storage: unlimited retention in object storage (S3, GCS)
• Queries for recent data hit local TSDB (fast)
• Queries for older data hit remote storage (slower but available)

Remote Storage Costs: What Most Teams Miss

Remote storage is not free. Costs include:

1. Remote storage backend: Ingestion fees, storage fees, query fees
2. Network egress: Sending samples from Prometheus to remote storage (~$0.10/GB on AWS/GCP)
3. Object storage: S3/GCS bucket storage costs
4. Query overhead: Remote read queries are slower and more expensive than local queries

A mid-scale deployment (500K samples/second, 30-day retention) can generate 1–2 TB of outbound traffic monthly just for remote write. On AWS, that is $100–$200 in egress fees alone, before the remote storage platform’s own costs.

When evaluating remote storage, factor in both the platform cost and public cloud egress fees. Some teams save money by running remote storage in the same VPC as Prometheus to eliminate egress charges.

When Remote Storage Makes Sense

Use remote storage when:

• Retention requirements exceed local disk capacity (90+ days)
• You need to aggregate metrics across multiple Prometheus instances
• Compliance requires durable, replicated storage
• You want to separate monitoring data from Prometheus server lifecycle
• You are running Kubernetes clusters with ephemeral storage volumes

Remote storage does not replace local storage — it extends it. Prometheus still scrapes and stores data locally, and remote write runs in parallel.

Retention and Compaction: How They Interact

Retention and compaction are separate but related processes. Retention deletes old data once a time or size limit is reached, removing entire blocks that fall outside your configured window. Compaction merges smaller blocks into larger ones to reduce query overhead and disk fragmentation.

Retention Enforcement Timing

Prometheus checks retention every two hours. When it runs, it:

1. Identifies blocks older than --storage.tsdb.retention.time or exceeding --storage.tsdb.retention.size
2. Deletes entire blocks that fall outside the retention window
3. Leaves partial blocks untouched until they fully expire

This means that actual disk usage can temporarily exceed your retention size limit, especially during compaction.

Compaction and Disk Space Requirements

Compaction creates new blocks while keeping old ones until the process completes. This requires temporary disk space equal to the size of the blocks being merged.

If disk space runs low during compaction:

• Compaction may fail
• Query performance degrades because Prometheus must scan more small blocks
• WAL size grows because blocks are not being cut and flushed

Provisioning disk with 20–30% headroom prevents compaction failures.

Compaction Settings

Prometheus provides --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration flags, but these should rarely be changed. The defaults (2 hours minimum, 31 days or 10% of retention maximum) are optimized for most workloads.

Disabling compaction entirely is not recommended. Without compaction, Prometheus accumulates thousands of small blocks, slowing queries and increasing memory usage.

WAL Cleanup and Checkpointing

WAL files are not removed until their data has been persisted to a block and that block has been successfully written to disk. Prometheus also creates checkpoints to speed up WAL replay after restarts. Checkpoints consolidate older WAL segments into a single file.

You can safely delete WAL files only during an outage or after stopping Prometheus. Deleting WAL files while Prometheus is running causes data loss.

Choosing Between Local and Remote Storage

The decision comes down to retention requirements, scale, durability needs, and operational complexity.

Use Local Storage Alone When:

• Retention is under 30 days
• A single Prometheus instance handles your scrape targets
• Queries focus on recent data (last few hours to days)
• Disk failure is acceptable with snapshot backups
• You want zero external dependencies

Use Remote Storage When:

• Retention exceeds 30–90 days
• You need to aggregate metrics across multiple Prometheus instances
• Compliance requires durable, replicated storage
• You are running ephemeral Kubernetes clusters
• You want to decouple monitoring data from Prometheus server lifecycle

Hybrid Approach: Local + Remote

Most production teams run both:

• Short local retention (2 hours to 7 days) for fast recent queries
• Remote storage for long-term retention and cross-cluster aggregation
• Local storage handles alerting and short-term dashboards
• Remote storage handles compliance, capacity planning, and historical analysis

This setup keeps local disk usage predictable and query performance high while ensuring data durability and long-term availability.

Monitoring Prometheus Storage with CubeAPM

CubeAPM provides unified observability for Prometheus and the infrastructure it runs on. Unlike building a custom Grafana dashboard stack, CubeAPM connects to Prometheus via OpenTelemetry or native Prometheus remote write and surfaces storage-specific signals alongside APM traces and logs.

What CubeAPM Monitors for Prometheus Storage

• Disk usage and growth rate: Track TSDB size, WAL size, and compaction overhead
• Ingestion rate and sample lag: Detect scrape delays and WAL replay bottlenecks
• Block count and compaction cycles: Identify compaction failures or slow merges
• Query performance: Correlate slow queries with block count and retention settings
• Remote write lag and errors: Monitor remote storage connectivity and backpressure

CubeAPM runs on-prem or in your VPC, so Prometheus telemetry stays inside your infrastructure. There are no public cloud egress fees for sending metrics to CubeAPM, unlike SaaS observability platforms.

Setting Up Prometheus Monitoring in CubeAPM

CubeAPM supports native Prometheus remote write. Add this to your prometheus.yml:

remote_write:
  - url: "https://cubeapm.example.com/api/v1/write"
    queue_config:
      max_samples_per_send: 1000
      batch_send_deadline: 5s

CubeAPM automatically correlates Prometheus metrics with infrastructure telemetry from the same nodes, giving you full context when disk usage spikes or compaction slows down.

For teams running AWS Lambda monitoring, CubeAPM also tracks Lambda invocations and correlates them with Prometheus metrics if Lambda functions scrape or expose metrics.

CubeAPM pricing is $0.15/GB for all telemetry — metrics, traces, and logs — with unlimited retention. There are no per-host fees, user seat charges, or surprise overages.

Common Prometheus Storage Issues and How to Fix Them

High Disk Usage Despite Short Retention

Cause: High cardinality (too many unique time series) or WAL bloat from high ingestion rate.

Fix:

• Reduce scrape frequency or filter high-cardinality labels at scrape time
• Lower retention time or size if not needed
• Check for unused or duplicate scrape targets

Slow Queries After Uptime Increases

Cause: Too many small blocks due to compaction failures or disabled compaction.

Fix:

• Ensure disk has 20–30% free space for compaction
• Check Prometheus logs for compaction errors
• Restart Prometheus to trigger compaction cycle

WAL Replay Takes Minutes After Restart

Cause: High ingestion rate generates large WAL files.

Fix:

• Reduce --storage.tsdb.wal-segment-size (default: 128MB)
• Use remote storage to keep local retention minimal
• Run Prometheus on faster disks (SSD/NVMe)

Remote Write Lag or Errors

Cause: Network issues, remote storage backpressure, or misconfigured queue settings.

Fix:

• Increase queue_config.max_samples_per_send and queue_config.capacity
• Monitor remote storage backend for capacity limits
• Check network latency and firewall rules

Out of Disk Space During Compaction

Cause: Insufficient headroom for temporary block creation.

Fix:

• Provision disk with at least 20% free space beyond expected data volume
• Lower retention time to reduce active block count
• Delete old WAL checkpoints manually if safe to do so

Frequently Asked Questions