Prometheus + Grafana in Production: Setup, Hidden Costs, and When to Consider a Managed Alternative

Prometheus and Grafana are both open source and technically free, but most teams discover the real costs only after running them in production for several months. A Reddit thread in r/devops documented one startup’s observability bill climbing to $80,000 per month, not from tool licensing, but from the infrastructure required to store high cardinality metrics, the AWS S3 API costs for long term storage, and the engineering time spent keeping the stack operational.

Beyond direct costs, Prometheus in production carries hidden overhead: memory requirements scaling with cardinality, retention limits forcing teams into cold storage solutions like Thanos or Cortex, and the operational burden of maintaining HA clusters, managing compaction, and firefighting OOM crashes during traffic spikes. This guide walks through the full production deployment of Prometheus and Grafana, surfaces the hidden costs most teams hit after month three, and explains when a managed alternative becomes the better path.

Prerequisites

Before deploying Prometheus and Grafana in production, ensure you have the following in place:

• A Kubernetes cluster (this guide uses managed Kubernetes on AWS EKS, GKE, or Azure AKS)
• kubectl and Helm 3 installed and configured
• Sufficient cluster capacity: minimum 4 vCPUs and 16 GB RAM per node for moderate workloads
• Administrative access to configure persistent storage (PVs) and LoadBalancers or Ingress
• At least one application exposing metrics on /metrics endpoint (OpenTelemetry, client libraries, or node_exporter)
• Basic familiarity with YAML, PromQL, and Kubernetes manifests

Step 1: Install Prometheus Using the kube-prometheus-stack Helm Chart

The fastest production-ready path is to use the kube-prometheus-stack Helm chart. This bundles Prometheus Operator, Grafana, Alertmanager, node_exporter, kube-state-metrics, and pre-built dashboards.

Create a dedicated namespace:

kubectl create namespace monitoring

Add the Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install the stack with default settings:

helm install prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi

This command does three things: deploys Prometheus with 30 day retention, allocates 100 GB persistent storage, and installs Grafana with pre-configured Prometheus datasource.

Verify the pods are running:

kubectl get pods -n monitoring

You should see pods for prometheus-prometheus-kube-prometheus-prometheus-0, prometheus-stack-grafana-*, and alertmanager-prometheus-kube-prometheus-alertmanager-0.

Step 2: Configure Persistent Storage and Retention Policy

By default, Prometheus stores metrics in an ephemeral volume. For production, configure a PersistentVolumeClaim backed by network storage (AWS EBS, GCP Persistent Disk, Azure Disk).

Create a values.yaml file:

prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: 90GB
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 100Gi
          storageClassName: gp3

Upgrade the Helm release:

helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack \
  -f values.yaml \
  --namespace monitoring

Why this matters: Without persistent storage, a pod restart wipes all metrics. The retentionSize setting prevents Prometheus from filling the disk and crashing. Most teams set retention to 15–30 days and offload older data to object storage using Thanos or Cortex.

Step 3: Expose Prometheus and Grafana via Ingress or LoadBalancer

By default, Prometheus and Grafana run as ClusterIP services, accessible only inside the cluster. For production access, expose them via Ingress (recommended) or LoadBalancer.

Option A: Ingress (requires an Ingress controller like nginx-ingress or Traefik)

Add to values.yaml:

grafana:
  ingress:
    enabled: true
    hosts:
      - grafana.yourdomain.com
    tls:
      - secretName: grafana-tls
        hosts:
          - grafana.yourdomain.com
prometheus:
  ingress:
    enabled: true
    hosts:
      - prometheus.yourdomain.com
    tls:
      - secretName: prometheus-tls
        hosts:
          - prometheus.yourdomain.com

Apply the changes:

helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack \
  -f values.yaml \
  --namespace monitoring

Option B: LoadBalancer (quick but exposes services publicly without auth by default)

kubectl patch svc prometheus-stack-grafana -n monitoring -p '{"spec": {"type": "LoadBalancer"}}'
kubectl patch svc prometheus-kube-prometheus-prometheus -n monitoring -p '{"spec": {"type": "LoadBalancer"}}'

Get the external IPs:

kubectl get svc -n monitoring

Access Grafana at http://<EXTERNAL-IP>:3000. Default credentials: admin / prom-operator.

Step 4: Configure Service Monitors to Scrape Application Metrics

Prometheus Operator uses ServiceMonitor CRDs to define scrape targets. If your application exposes metrics on /metrics, create a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Apply the manifest:

kubectl apply -f servicemonitor.yaml

Verify scrape targets in Prometheus UI: navigate to Status > Targets and confirm my-app-metrics appears with UP status.

Common issue: If targets show as DOWN, check that the service selector matches your app pods and the port name is metrics in the Service definition.

Step 5: Build Cost-Tracking Dashboards in Grafana

Prometheus does not track cost directly, but you can estimate infrastructure cost by aggregating resource usage metrics and multiplying by your cloud provider’s rates.

Log into Grafana (http://grafana.yourdomain.com or LoadBalancer IP), navigate to Create > Dashboard, and add a panel.

Example PromQL query to estimate CPU cost for GKE:

(sum(rate(container_cpu_usage_seconds_total[1h])) / 3600) * 0.0445

This query sums CPU usage across all containers, converts to hours, and multiplies by GKE’s CPU cost of $0.0445 per vCPU hour (as of early 2026).

Example PromQL query to estimate memory cost:

(sum(container_memory_working_set_bytes) / 1073741824) * 0.006

This converts memory usage to GB and multiplies by $0.006 per GB hour.

Create separate panels for CPU, memory, storage, and network egress to build a full cost dashboard. These queries give directional estimates — actual bills include other factors like disk IOPS, data transfer, and load balancer costs.

Step 6: Set Up Alertmanager for Production Alerts

Alertmanager is installed by default with kube-prometheus-stack. Configure it to send alerts to Slack, PagerDuty, email, or webhooks.

Edit values.yaml to add Slack integration:

alertmanager:
  config:
    route:
      receiver: slack-notifications
      group_by: ['alertname', 'cluster']
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 12h
    receivers:
      - name: slack-notifications
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
            channel: '#alerts'
            title: 'Prometheus Alert'
            text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

Upgrade Helm release:

helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack \
  -f values.yaml \
  --namespace monitoring

Create a PrometheusRule to define alert conditions:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: high-cpu-alert
  namespace: monitoring
spec:
  groups:
    - name: cpu-alerts
      interval: 30s
      rules:
        - alert: HighCPUUsage
          expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (pod) > 0.8
          for: 5m
          annotations:
            summary: "Pod {{ $labels.pod }} CPU usage above 80%"

Apply the rule:

kubectl apply -f prometheusrule.yaml

Alerts fire when conditions are met for the specified duration (for: 5m). Check Alertmanager UI at http://alertmanager.yourdomain.com to see active alerts.

Step 7: Deploy Long-Term Storage with Thanos or Cortex (Optional)

Prometheus local storage caps out at a few weeks of retention due to disk cost and memory overhead. For long term retention (90+ days), deploy Thanos or Cortex to offload metrics to object storage like S3, GCS, or Azure Blob.

Thanos setup (simplified):

Add Thanos sidecar to Prometheus:

prometheus:
  prometheusSpec:
    thanos:
      image: quay.io/thanos/thanos:v0.34.0
      objectStorageConfig:
        key: thanos.yaml
        name: thanos-objstore-config

Create a ConfigMap with S3 configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-objstore-config
  namespace: monitoring
data:
  thanos.yaml: |
    type: S3
    config:
      bucket: my-metrics-bucket
      endpoint: s3.us-west-2.amazonaws.com
      access_key: YOUR_ACCESS_KEY
      secret_key: YOUR_SECRET_KEY

Deploy Thanos Query and Store Gateway components. Full setup requires additional manifests — see Thanos documentation for complete steps.

Why teams skip this: Thanos and Cortex add operational complexity — more components to maintain, debug, and scale. Most teams start with 15–30 day local retention and only add long term storage when compliance or SLA analysis requires historical data beyond that window.

Troubleshooting Common Issues

Prometheus pod crashes with OOMKilled

Cause: High cardinality metrics (labels with thousands of unique values) consume too much RAM.

Fix: Identify high cardinality series using the Prometheus UI at Status > TSDB Status. Drop unnecessary labels in relabel configs:

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: my-app
        relabel_configs:
          - source_labels: [__name__]
            regex: 'unwanted_metric_.*'
            action: drop

Grafana dashboards load slowly

Cause: Queries span large time ranges or use heavy aggregations.

Fix: Reduce query time window, add rate() or irate() functions, and use recording rules to pre-aggregate expensive queries.

Metrics missing after Prometheus restart

Cause: No persistent storage configured or PVC not properly mounted.

Fix: Verify PVC exists and is bound:

kubectl get pvc -n monitoring

If missing, add storageSpec to values.yaml as shown in Step 2 and upgrade Helm release.

Targets show as DOWN in Prometheus

Cause: ServiceMonitor selector does not match service labels or port name mismatch.

Fix: Check service labels match ServiceMonitor selector:

kubectl get svc -n your-app-namespace --show-labels

Ensure port name in Service definition matches port: field in ServiceMonitor.

Alertmanager not sending alerts

Cause: Webhook URL incorrect, network policy blocking egress, or alert condition never met.

Fix: Check Alertmanager logs:

kubectl logs -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0

Test webhook manually using curl to verify connectivity. Confirm alert fires in Prometheus UI under Alerts.

The Hidden Costs of Running Prometheus + Grafana in Production

This estimate models a production ready setup with high availability. A smaller or simpler deployment may cost significantly less.

Prometheus and Grafana are free to download, but running them at scale introduces costs most teams do not forecast accurately during initial evaluation. These fall into four categories: infrastructure, storage, egress, and engineering time.

Infrastructure cost: RAM and compute

Prometheus memory usage scales with active time series count and cardinality. A cluster ingesting 500,000 active series with 30 day retention requires 32–64 GB RAM per Prometheus instance. For high availability, you run two replicas — doubling the cost.

On AWS, a single m6i.2xlarge instance (8 vCPUs, 32 GB RAM) costs approximately $0.384 per hour, or $276 per month. Two instances for HA: $552 per month. On GKE, n2-standard-8 costs $0.3888 per hour ($279 per month per node). Add node_exporter, kube-state-metrics, and Alertmanager pods, and infrastructure cost for a 100 node cluster monitoring setup can reach $800 to $1,200 per month before any add ons.

Storage cost: persistent volumes and object storage

Prometheus stores metrics on disk. A 100 GB EBS gp3 volume costs $8 per month. For 30 day retention, two HA replicas need 200 GB total — $16 per month. Long term storage via Thanos offloads to S3. If you retain 12 months of data at 10 GB per day after compression, that is 3.6 TB in S3 Standard at $0.023 per GB = $82.80 per month. Add S3 API costs: Thanos makes frequent GET and LIST requests. At $0.0004 per 1,000 GET requests, a busy Thanos setup can generate $50 to $150 per month in API fees alone.

Egress cost: data transfer to SaaS or cross-region queries

If you run Grafana in a different region or VPC from Prometheus, cross region data transfer costs $0.01 to $0.02 per GB. Querying 100 GB per month across regions adds $1 to $2 per month. If you send metrics to a SaaS tool for backup or correlation, AWS charges $0.09 per GB for egress to the internet. Sending 10 TB per month costs $900 in egress fees — a line item most teams miss during planning.

Engineering cost: on-call, upgrades, and knowledge silos

Prometheus requires hands on maintenance: upgrading Prometheus Operator and CRDs without breaking ServiceMonitors, rewriting queries when labels change, tuning memory limits after cardinality spikes, and firefighting OOM crashes during traffic surges. One team on Reddit documented spending 15 to 20 engineer hours per month just keeping Prometheus stable — at a $150,000 annual salary, that is $1,875 per month in opportunity cost.

Add on-call rotation cost. When Prometheus goes down, your monitoring is blind. Teams often assign a dedicated SRE or DevOps engineer to own the stack, creating a knowledge silo. If that person leaves, the next engineer has to relearn PromQL, relabel configs, and Thanos architecture from scratch.

Total cost example: 100-node Kubernetes cluster, 30-day retention, 12-month Thanos storage

Cost category	Monthly cost
Compute (2x HA replicas)	$552
Persistent storage (EBS)	$16
S3 storage (12 months)	$83
S3 API requests	$100
Cross-region egress	$50
Engineering time (15 hrs)	$1,875
Total	$2,676

This does not include Grafana hosting, alert routing infrastructure, or the cost of downtime when Prometheus crashes during a production incident.

When to Consider a Managed Prometheus Alternative

Managed alternatives make sense when the operational cost of self hosting exceeds the tool cost, when your team lacks deep Prometheus expertise, or when compliance requires features like audit logs and RBAC that Prometheus does not provide natively.

Scenario 1: Your engineering team is under 20 people

Small teams cannot dedicate an engineer to Prometheus operations. Managed platforms like CubeAPM, Grafana Cloud, or Datadog absorb the maintenance burden — upgrades, scaling, and storage management happen automatically.

Scenario 2: You need more than 30 days retention without DIY storage

If you need 90+ days of metrics for SLA analysis or compliance, setting up Thanos or Cortex adds weeks of engineering work. Managed platforms include long term storage as a standard feature. CubeAPM offers unlimited retention at $0.15 per GB with no separate cold storage tier.

Scenario 3: Your monitoring stack went down during the incident you were trying to debug

Prometheus and Grafana run inside your cluster. If the cluster has a control plane failure or resource exhaustion, your monitoring goes dark exactly when you need it most. Managed platforms run outside your infrastructure, so they stay up even when your cluster does not.

Scenario 4: Prometheus memory usage keeps causing OOM crashes

High cardinality metrics — caused by labels like user_id, request_id, or pod_name with thousands of unique values — make Prometheus consume 100+ GB of RAM. Managed platforms use different storage engines (often ClickHouse or columnar databases) that handle high cardinality without linear memory scaling.

Scenario 5: You are hitting S3 API cost surprises with Thanos

Thanos queries S3 frequently for historical data. Teams report $200 to $500 per month in S3 API costs they did not forecast. Managed platforms include storage in their pricing — no separate API fee line items.

Monitoring Prometheus and Grafana with CubeAPM

CubeAPM provides an alternative to the DIY Prometheus and Grafana stack by offering full stack observability, metrics, logs, traces, and Kubernetes monitoring in a single managed platform that runs inside your own cloud or on premises. It ingests metrics from Prometheus exporters, OpenTelemetry collectors, and existing Grafana dashboards without requiring agent replacement.

How CubeAPM connects to Prometheus-compatible workloads

CubeAPM supports native Prometheus remote write. Configure your existing Prometheus instance to forward metrics:

remote_write:
  - url: https://cubeapm.yourcompany.com/api/v1/write
    basic_auth:
      username: your-tenant-id
      password: your-api-key

CubeAPM stores metrics in ClickHouse, which handles high cardinality series without the memory overhead Prometheus requires. You can query using PromQL or SQL, and retention is unlimited by default.

What CubeAPM monitors that Prometheus + Grafana does not

CubeAPM includes distributed tracing, log aggregation, and RUM in the same platform. Prometheus only handles metrics — adding logs requires Loki, traces require Tempo, and stitching them together requires manual correlation. CubeAPM auto-correlates traces with logs and metrics using OpenTelemetry context propagation.

Why teams switch from self-hosted Prometheus to CubeAPM

Delhivery, a logistics platform processing millions of shipments per day, replaced their Prometheus and Thanos setup with CubeAPM and documented 75% cost savings. Their previous setup required three engineers to maintain; CubeAPM runs as a managed service inside their VPC with zero operational overhead. Redbus, part of MakeMyTrip, reported 4x faster dashboard load times and 50% faster MTTR after migrating from Grafana to CubeAPM’s unified observability UI.

CubeAPM pricing is $0.15 per GB ingested, covering metrics, logs, and traces. A team ingesting 10 TB per month pays $1,500 — no separate charges for users, hosts, or retention. Compare that to the $2,676 total cost of ownership modeled earlier for self-hosted Prometheus with Thanos.

Conclusion

Prometheus and Grafana remain the most widely deployed open source monitoring stack, but running them in production at scale introduces hidden costs in infrastructure, storage, egress fees, and engineering time that add up to thousands of dollars per month for mid-sized teams. The operational burden of maintaining HA clusters, managing long term storage with Thanos or Cortex, and tuning memory limits during cardinality spikes often exceeds the cost of a managed alternative.

For teams with deep Prometheus expertise and full time SRE capacity, self hosting makes sense. For everyone else — especially teams under 50 engineers or those needing more than 30 days retention without DIY complexity — managed platforms like CubeAPM, Grafana Cloud, or Datadog absorb the operational burden while maintaining Prometheus compatibility.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions