How to set up Prometheus on GKE using kube-prometheus-stack?

Introduction

Prometheus on GKE is one of the most reliable ways to monitor Kubernetes clusters with open-source metrics, Grafana dashboards, and alerting. It helps teams track CPU usage, memory pressure, pod health, node performance, and service-level issues before incidents become guesswork.

The kube-prometheus-stack Helm chart packages everything into a single install: Prometheus, Grafana, Alertmanager, kube-state-metrics, the Prometheus Operator, and node-exporter, all maintained by the Prometheus community. The latest release as of May 2026 is version 84.5.0.

This guide walks you through every step: adding the Helm repo, configuring values for GKE Standard vs. Autopilot, installing the stack, accessing Grafana, and verifying metrics are flowing. It also covers common GKE-specific pitfalls and when Google Managed Prometheus might be the better fit.

Prerequisites

Before you begin, make sure you have the following in place.

• A GKE cluster (Standard or Autopilot) running Kubernetes 1.25 or later.
• kubectl configured to talk to that cluster (gcloud container clusters get-credentials CLUSTER_NAME –region REGION –project PROJECT_ID).
• Helm 3.x installed locally.
• Cluster-admin permissions. For GKE specifically, you may need to bind your Google account: kubectl create clusterrolebinding owner-cluster-admin-binding –clusterrole cluster-admin –user $(gcloud config get-value account)
• At least 2 vCPU and 4 GiB of free cluster capacity. The full stack uses roughly 1 CPU and 2 GiB at rest.

Step 1: Add the Helm Repository

Add the Prometheus community chart repository and refresh the local index.

helm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelm repo update

helm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelm repo update

Step 2: Create a Namespace

Install all monitoring components into a dedicated namespace. This keeps resources cleanly separated from your application workloads and makes RBAC policies easier to reason about.

kubectl create namespace monitoring

kubectl create namespace monitoring

Step 3: Create a values.yaml File

The kube-prometheus-stack chart has thousands of configuration options. You control all of them through a values.yaml override file. The defaults work fine on a basic cluster, but on GKE you need to adjust several settings.

For GKE Standard Clusters

GKE Standard gives you full access to nodes, so the complete stack runs without modification. The main thing to configure is storage for Prometheus data persistence.

# values-gke-standard.yamlprometheus:  prometheusSpec:    retention: 15d    storageSpec:      volumeClaimTemplate:        spec:          accessModes: ["ReadWriteOnce"]          resources:            requests:              storage: 50Gi
grafana:  adminPassword: "change-me-now"  persistence:    enabled: true    size: 10Gi
alertmanager:  alertmanagerSpec:    storage:      volumeClaimTemplate:        spec:          accessModes: ["ReadWriteOnce"]          resources:            requests:              storage: 10Gi

# values-gke-standard.yamlprometheus:  prometheusSpec:    retention: 15d    storageSpec:      volumeClaimTemplate:        spec:          accessModes: ["ReadWriteOnce"]          resources:            requests:              storage: 50Gi
grafana:  adminPassword: "change-me-now"  persistence:    enabled: true    size: 10Gi
alertmanager:  alertmanagerSpec:    storage:      volumeClaimTemplate:        spec:          accessModes: ["ReadWriteOnce"]          resources:            requests:              storage: 10Gi

For GKE Autopilot Clusters

GKE Autopilot manages nodes on your behalf and enforces strict security policies. The default kube-prometheus-stack chart tries to deploy node-exporter as a DaemonSet with host-level access (hostPID, hostNetwork, hostPort, and restricted hostPath volumes). Autopilot blocks all of these by default.

You will see errors like: admission webhook denied the request... hostPath volume uses path /proc which is not allowed, and container node-exporter specifies a host port; disallowed in Autopilot. 

The fix is to disable the components that require privileged node access. You still get workload metrics, pod-level data, and custom application metrics. You lose low-level node metrics (CPU temperature, disk I/O from the OS), but GKE Autopilot already exposes those through Google Cloud Monitoring.

# values-gke-autopilot.yaml# Disable components that violate Autopilot security policiesnodeExporter:  enabled: false
kubeProxy:  enabled: false
kubeScheduler:  enabled: false
kubeControllerManager:  enabled: false
kubeEtcd:  enabled: false
coreDns:  enabled: false
# kube-state-metrics works fine on AutopilotkubeStateMetrics:  enabled: true
prometheus:  prometheusSpec:    retention: 15d    # Scrape cAdvisor metrics from nodes via the Kubernetes API    additionalScrapeConfigs:      - job_name: 'kubernetes-pods-cadvisor'        scheme: https        metrics_path: /metrics/cadvisor        kubernetes_sd_configs:          - role: node        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token        tls_config:          insecure_skip_verify: true        relabel_configs:          - action: labelmap            regex: __meta_kubernetes_node_label_(.+)
grafana:  adminPassword: "change-me-now"

# values-gke-autopilot.yaml# Disable components that violate Autopilot security policiesnodeExporter:  enabled: false
kubeProxy:  enabled: false
kubeScheduler:  enabled: false
kubeControllerManager:  enabled: false
kubeEtcd:  enabled: false
coreDns:  enabled: false
# kube-state-metrics works fine on AutopilotkubeStateMetrics:  enabled: true
prometheus:  prometheusSpec:    retention: 15d    # Scrape cAdvisor metrics from nodes via the Kubernetes API    additionalScrapeConfigs:      - job_name: 'kubernetes-pods-cadvisor'        scheme: https        metrics_path: /metrics/cadvisor        kubernetes_sd_configs:          - role: node        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token        tls_config:          insecure_skip_verify: true        relabel_configs:          - action: labelmap            regex: __meta_kubernetes_node_label_(.+)
grafana:  adminPassword: "change-me-now"

Step 4: Install the Chart

Run the install command, pointing to the appropriate values file for your cluster type.

GKE Standard

helm install kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-standard.yaml \  --create-namespace

helm install kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-standard.yaml \  --create-namespace

GKE Autopilot

helm install kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-autopilot.yaml \  --create-namespace

helm install kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-autopilot.yaml \  --create-namespace

The install takes 2 to 3 minutes. GKE Autopilot may take longer on the first run because it needs to provision new nodes to accommodate the monitoring pods.

Step 5: Verify the Installation

Check that all pods are running in the monitoring namespace.

kubectl get pods -n monitoring

kubectl get pods -n monitoring

You should see pods for: prometheus-kube-prometheus-stack-prometheus-0, alertmanager-kube-prometheus-stack-alertmanager-0, kube-prometheus-stack-grafana, kube-prometheus-stack-kube-state-metrics, and (on Standard only) kube-prometheus-stack-prometheus-node-exporter.

All pods should reach Running status within 3 to 5 minutes. If any pod stays in Pending, check events with kubectl describe pod POD_NAME -n monitoring.

Step 6: Access Grafana

The chart deploys Grafana with a ClusterIP service by default, which is not reachable from outside the cluster. Use port-forward to access it locally.

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

Open http://localhost:3000 in your browser. Log in with:

• Username: admin
• Password: prom-operator (the chart default, unless you set adminPassword in your values.yaml)

Accessing Prometheus

Similarly, use port-forward to reach the Prometheus UI.

kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring

kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring

Open http://localhost:9090 in your browser to run PromQL queries and check scrape targets.

Step 7: Explore Pre-built Grafana Dashboards

One of the biggest advantages of kube-prometheus-stack is the 30+ Grafana dashboards it ships with. After logging in, go to Dashboards > Browse to find them.

The most useful dashboards for GKE workloads are:

• Kubernetes / Compute Resources / Cluster: overall CPU and memory across the entire cluster.
• Kubernetes / Compute Resources / Namespace (Pods): per-pod resource breakdown inside a namespace.
• Kubernetes / Compute Resources / Node (Pods): node-level resource usage (Standard clusters only).
• Kubernetes / Networking / Cluster: network ingress and egress across the cluster.
• Kubernetes / Persistent Volumes: PVC usage and capacity.
• Node Exporter / Full: detailed system-level metrics for each node (Standard clusters only).

Step 8: Upgrading the Chart

When new versions of kube-prometheus-stack are released, upgrade with Helm. Always diff your values before applying.

helm repo update
# Preview what will changehelm diff upgrade kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-standard.yaml
# Apply the upgradehelm upgrade kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-standard.yaml

helm repo update
# Preview what will changehelm diff upgrade kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-standard.yaml
# Apply the upgradehelm upgrade kube-prometheus-stack \  prometheus-community/kube-prometheus-stack \  --namespace monitoring \  --values values-gke-standard.yaml

Note: Some major chart versions include CRD updates that Helm does not apply automatically. Check the changelog at https://github.com/prometheus-community/helm-charts/releases before each upgrade.

Optional: Remote Write for Long-Term Storage

Prometheus stores data locally by default. The default retention is 10 days. For longer retention across multiple clusters, configure remote write to an external backend such as Thanos, Cortex, Mimir, or Grafana Cloud.

# Add to your values.yaml under prometheus.prometheusSpecprometheus:  prometheusSpec:    remoteWrite:      - url: "https://your-remote-write-endpoint/api/v1/push"        basicAuth:          username:            name: prometheus-remote-write-auth            key: username          password:            name: prometheus-remote-write-auth            key: password    externalLabels:      cluster: "my-gke-cluster"      env: "production"      region: "us-central1"

# Add to your values.yaml under prometheus.prometheusSpecprometheus:  prometheusSpec:    remoteWrite:      - url: "https://your-remote-write-endpoint/api/v1/push"        basicAuth:          username:            name: prometheus-remote-write-auth            key: username          password:            name: prometheus-remote-write-auth            key: password    externalLabels:      cluster: "my-gke-cluster"      env: "production"      region: "us-central1"

Create the Kubernetes Secret for credentials before applying the values:

kubectl create secret generic prometheus-remote-write-auth \  --from-literal=username=YOUR_USERNAME \  --from-literal=password=YOUR_PASSWORD \  --namespace monitoring

kubectl create secret generic prometheus-remote-write-auth \  --from-literal=username=YOUR_USERNAME \  --from-literal=password=YOUR_PASSWORD \  --namespace monitoring

Source reference for remote write configuration pattern: https://github.com/prometheus-community/helm-charts (Infrastructure Observability Docs Hub guide)

Alternative: Google Managed Prometheus (GMP)

If you do not want to manage Prometheus yourself, Google offers a fully managed alternative called Google Managed Service for Prometheus (GMP). It is built on Monarch, Google’s globally scalable time-series database. GMP is enabled by default on all new GKE Autopilot clusters.

With GMP, you do not run your own Prometheus binary. Instead, GKE deploys lightweight collectors in the gmp-system namespace that scrape your workloads and forward metrics to Google Cloud Monitoring. You then query those metrics using PromQL in Metrics Explorer or point Grafana at Cloud Monitoring as a data source.

The trade-off: GMP uses different CRDs. PrometheusRule becomes ClusterRules. ServiceMonitor becomes PodMonitoring. If you migrate from kube-prometheus-stack to GMP, every monitoring rule and scrape config needs to be rewritten. 

GMP makes sense when you want zero operational overhead and are comfortable with vendor lock-in. kube-prometheus-stack makes sense when you need maximum control, portability across clouds, and the full Prometheus ecosystem.

Enabling GMP in Terraform

resource "google_container_cluster" "cluster" {  # ...  monitoring_config {    managed_prometheus {      enabled = true    }    enable_components = [      "SYSTEM_COMPONENTS",      "CADVISOR",      "KUBELET",      "POD",      "DEPLOYMENT",    ]  }}

resource "google_container_cluster" "cluster" {  # ...  monitoring_config {    managed_prometheus {      enabled = true    }    enable_components = [      "SYSTEM_COMPONENTS",      "CADVISOR",      "KUBELET",      "POD",      "DEPLOYMENT",    ]  }}

Common Issues and Fixes

Conclusion

The kube-prometheus-stack Helm chart gives you a production-grade observability stack on GKE with a single install command. On GKE Standard, the defaults work well with minor storage configuration. On GKE Autopilot, you need to disable the components that conflict with Autopilot’s security model: node-exporter, kubeProxy, kubeScheduler, kubeControllerManager, and kubeEtcd.

Once installed, Grafana ships with 30+ pre-built Kubernetes dashboards that give you immediate visibility into cluster health, workload resource usage, networking, and storage. For long-term metric retention, configure remote write to Thanos, Mimir, or Grafana Cloud.

If you prefer a fully managed path without running Prometheus yourself, Google Managed Prometheus is a viable alternative, particularly for GKE Autopilot users who want native integration with Google Cloud Monitoring.

Disclaimer: This article contains pricing estimates based on publicly available AWS CloudWatch Logs rates as of May 2026. Actual costs may vary by AWS region, account type, and usage patterns. Always verify current pricing before making infrastructure decisions.

FAQs