How to Monitor Karpenter Node Provisioning on AWS EKS: Step-by-Step Guide 2026

Karpenter provisions nodes on AWS EKS clusters in seconds by reading pod scheduling requirements directly from the Kubernetes API and calling EC2 Fleet APIs without autoscaling groups or node groups. According to the CNCF Annual Survey 2023, 71% of organizations use Kubernetes in production, and as workloads scale, autoscaling becomes a bottleneck. Karpenter solves this by provisioning exactly the node type each pod needs, but without proper monitoring, scaling failures, Spot interruptions, and throttling issues go undetected until they impact production workloads.

This guide walks through setting up Karpenter monitoring on AWS EKS. You will learn how to track node provisioning events, debug failed scaling attempts, and configure alerts for provisioning delays or errors using Prometheus, CloudWatch, and OpenTelemetry compatible observability platforms. By the end, you will have a production ready monitoring stack that surfaces every Karpenter decision in real time.

Prerequisites

Before starting, ensure you have:

• An AWS EKS cluster running Kubernetes 1.29 or later
• Karpenter v1.0 or later installed via Helm chart
• kubectl CLI configured to access the cluster
• AWS CLI installed and configured with IAM permissions for CloudWatch and EKS
• Prometheus or an OpenTelemetry compatible observability platform installed (or follow setup steps below)
• Basic familiarity with Kubernetes metrics and pod scheduling concepts

Step 1: Enable Karpenter Metrics Endpoint

Karpenter exposes metrics in Prometheus format at the /metrics endpoint on port 8000 by default. These metrics track every provisioning decision, node launch event, scheduling constraint, and disruption action Karpenter takes.

Verify that the metrics endpoint is enabled by checking the Karpenter controller deployment:

kubectl get deployment karpenter -n kube-system -o yaml | grep -A5 metrics

If the --metrics-port=8000 argument is not present, add it to the Karpenter Helm values file:

controller:
  env:
    - name: METRICS_PORT
      value: "8000"

Apply the update:

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --namespace kube-system \
  --values karpenter-values.yaml

Confirm the metrics endpoint is reachable by port-forwarding to the Karpenter controller pod:

kubectl port-forward -n kube-system deployment/karpenter 8000:8000
curl http://localhost:8000/metrics | grep karpenter

You should see metrics like karpenter_nodes_created, karpenter_pods_startup_duration_seconds, and karpenter_interruption_actions_performed_total. If the endpoint returns an empty response or connection refused, check that the Karpenter controller pod is running and the metrics port is correctly configured.

Step 2: Deploy Prometheus to Scrape Karpenter Metrics

Prometheus is the standard way to collect Karpenter metrics. If you already have Prometheus running in your cluster, skip to adding the Karpenter scrape config. If not, deploy Prometheus using the community Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus \
  --namespace monitoring \
  --create-namespace

Add a ServiceMonitor or scrape config to collect Karpenter metrics. Create a file named karpenter-servicemonitor.yaml:

apiVersion: v1
kind: Service
metadata:
  name: karpenter-metrics
  namespace: kube-system
  labels:
    app: karpenter
spec:
  ports:
  - name: metrics
    port: 8000
    targetPort: 8000
  selector:
    app.kubernetes.io/name: karpenter
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: karpenter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: karpenter
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: metrics
    interval: 30s

Apply the configuration:

kubectl apply -f karpenter-servicemonitor.yaml

Verify Prometheus is scraping Karpenter by querying the Prometheus UI:

kubectl port-forward -n monitoring svc/prometheus-server 9090:80

Open http://localhost:9090 and run a query like karpenter_nodes_created. If no data appears, check the Prometheus targets page (http://localhost:9090/targets) to confirm the Karpenter endpoint is listed and healthy.

Step 3: Track Node Provisioning Events with Key Metrics

Karpenter exposes several critical metrics that surface every node provisioning decision. Understanding these metrics is essential for debugging scaling issues.

Core Karpenter metrics to monitor:

• karpenter_nodes_created — Total nodes launched by Karpenter. A sudden drop indicates provisioning failures or API throttling.
• karpenter_pods_startup_duration_seconds — Time from pod unschedulable to node ready and pod scheduled. Spikes here mean slow EC2 launches or AMI pull delays.
• karpenter_provisioner_scheduling_duration_seconds — Time Karpenter spends evaluating scheduling constraints before launching a node. High values indicate complex NodePool selectors or large pending pod queues.
• karpenter_interruption_actions_performed_total — Count of Spot interruptions, scheduled maintenance events, or manual node deletions handled by Karpenter. A spike means increased Spot reclaims or instance retirements.
• karpenter_nodes_terminated — Nodes removed by Karpenter due to consolidation, drift, or expiration policies.
• karpenter_cloudprovider_errors_total — EC2 API errors encountered during node provisioning. Non-zero values indicate throttling, capacity issues, or IAM permission problems.

Query Prometheus to see how many nodes Karpenter has launched in the past hour:

increase(karpenter_nodes_created[1h])

To see average pod startup time over the last 5 minutes:

rate(karpenter_pods_startup_duration_seconds_sum[5m]) / rate(karpenter_pods_startup_duration_seconds_count[5m])

If you see pod startup times above 3 minutes consistently, investigate EC2 instance launch delays or AMI caching issues. Karpenter should provision nodes in under 2 minutes for most instance types.

Step 4: Configure CloudWatch Logs for Karpenter Controller Events

Karpenter logs contain structured event data that Prometheus metrics do not capture, including why specific instance types were chosen, which NodePools matched a pod, and detailed error messages when provisioning fails.

Enable CloudWatch logging for the Karpenter controller by updating the Helm values:

controller:
  logging:
    level: info
  podAnnotations:
    fluentbit.io/parser: json

Deploy Fluent Bit to ship logs to CloudWatch:

helm repo add eks https://aws.github.io/eks-charts
helm install aws-for-fluent-bit eks/aws-for-fluent-bit \
  --namespace kube-system \
  --set cloudWatch.enabled=true \
  --set cloudWatch.region=us-west-2 \
  --set cloudWatch.logGroupName=/aws/eks/karpenter/logs

Verify logs are flowing to CloudWatch:

aws logs tail /aws/eks/karpenter/logs --follow

Search for specific events using CloudWatch Logs Insights. To find all failed node launches in the past hour:

fields @timestamp, @message
| filter @message like /failed to launch/
| sort @timestamp desc
| limit 100

Common log patterns to watch for:

• "insufficient capacity" — EC2 does not have the requested instance type available in the chosen availability zone. Karpenter will retry with a different instance type or zone.
• "unauthorized operation" — IAM role lacks required permissions (e.g., ec2:RunInstances, ec2:CreateFleet).
• "throttling exception" — Karpenter is hitting EC2 API rate limits. This happens when launching hundreds of nodes simultaneously.

If you see repeated capacity errors for a specific instance type, adjust your NodePool instanceTypes requirements to include more fallback options.

Step 5: Set Up Alerts for Provisioning Failures and Delays

Alerting ensures you detect Karpenter issues before they cascade into pod scheduling delays or application downtime. Create alerts for three critical scenarios: provisioning failures, slow scaling, and Spot interruptions.

Alert 1: Karpenter CloudProvider Errors

This fires when Karpenter encounters EC2 API errors like throttling, capacity exhaustion, or IAM permission issues.

groups:
- name: karpenter
  interval: 30s
  rules:
  - alert: KarpenterProvisioningErrors
    expr: rate(karpenter_cloudprovider_errors_total[5m]) > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Karpenter is encountering EC2 API errors"
      description: "Karpenter has logged {{ $value }} EC2 errors per second over the past 5 minutes. Check CloudWatch logs for throttling or capacity issues."

Alert 2: Slow Node Provisioning

This fires when pod startup time exceeds 5 minutes, indicating EC2 launch delays or AMI pull bottlenecks.

  - alert: KarpenterSlowProvisioning
    expr: |
      rate(karpenter_pods_startup_duration_seconds_sum[5m]) 
      / rate(karpenter_pods_startup_duration_seconds_count[5m]) > 300
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Karpenter node provisioning is taking longer than expected"
      description: "Average pod startup time is {{ $value }} seconds. Expected under 180 seconds."

Alert 3: High Spot Interruption Rate

This fires when Spot interruptions spike above baseline, signaling increased reclaim events.

  - alert: KarpenterHighSpotInterruptions
    expr: rate(karpenter_interruption_actions_performed_total[10m]) > 0.1
    for: 5m
    labels:
      severity: info
    annotations:
      summary: "Spot interruptions are higher than normal"
      description: "Karpenter is handling {{ $value }} interruptions per second. Workloads may experience more rescheduling."

Save these rules to a file named karpenter-alerts.yaml and load them into Prometheus:

kubectl apply -f karpenter-alerts.yaml

Configure Alertmanager to route alerts to Slack, PagerDuty, or email. Example Alertmanager config for Slack:

route:
  receiver: slack-karpenter
receivers:
- name: slack-karpenter
  slack_configs:
  - api_url: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
    channel: '#karpenter-alerts'
    text: '{{ range .Alerts }}{{ .Annotations.summary }}: {{ .Annotations.description }}{{ end }}'

Step 6: Monitor Karpenter Node Lifecycle with Kubernetes Events

Kubernetes events provide real time visibility into Karpenter’s node lifecycle decisions. Events are emitted when Karpenter launches a node, terminates a node for consolidation, or fails to provision due to constraints.

View recent Karpenter events:

kubectl get events -n kube-system --field-selector involvedObject.kind=Node --sort-by='.lastTimestamp'

Filter for Karpenter specific events:

kubectl get events -A | grep karpenter

Common event types:

• NodeClaimCreated — Karpenter created a new node claim to satisfy pending pods.
• NodeLaunched — EC2 instance successfully launched and joined the cluster.
• NodeTerminated — Node removed due to consolidation, drift, or expiration.
• ProvisioningFailed — Karpenter could not provision a node due to capacity, IAM, or subnet issues.

To retain events beyond the default 1 hour TTL, forward them to a centralized logging system using event exporters like kubernetes-event-exporter or ship them to CloudWatch using Fluent Bit with the kube-events input plugin.

Step 7: Integrate Karpenter Metrics with Observability Platforms

For teams using full stack observability platforms, integrating Karpenter metrics with APM, logs, and infrastructure monitoring provides a unified view of cluster health. Most OpenTelemetry compatible platforms can scrape Prometheus metrics directly or ingest them via remote write.

Option 1: Using CubeAPM for Karpenter Monitoring

CubeAPM runs inside your VPC and integrates natively with Prometheus, OpenTelemetry, and Kubernetes events. To monitor Karpenter with CubeAPM:

1. Deploy CubeAPM in your EKS cluster following the infrastructure monitoring setup guide.
2. Configure CubeAPM to scrape the Karpenter metrics endpoint by adding the Karpenter service to the Prometheus scrape config managed by CubeAPM.
3. Create dashboards in CubeAPM to visualize karpenter_nodes_created, karpenter_pods_startup_duration_seconds, and karpenter_interruption_actions_performed_total alongside pod logs and APM traces.
4. Set up alerts in CubeAPM for provisioning failures and slow scaling using the same PromQL expressions from Step 5.

CubeAPM correlates Karpenter metrics with application traces and logs, allowing you to see exactly which pods triggered node provisioning and how long it took for those pods to start serving traffic. Pricing starts at $0.15/GB with unlimited retention and no per-host fees.

Option 2: Using Prometheus Remote Write

If you already use Datadog, Grafana Cloud, or another SaaS observability platform, configure Prometheus remote write to forward Karpenter metrics:

prometheus:
  server:
    remoteWrite:
    - url: https://your-platform.com/api/v1/write
      basicAuth:
        username: your-username
        password: your-password

This approach works but adds egress costs when sending metrics to a SaaS platform outside your AWS region. At 10 GB of metrics per month, AWS charges approximately $0.90 in data transfer fees to send metrics from us-west-2 to a SaaS provider in us-east-1.

Troubleshooting Common Issues

Karpenter metrics endpoint not reachable

Check that the Karpenter controller pod is running and the metrics port is exposed:

kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter
kubectl describe pod -n kube-system <karpenter-pod-name> | grep metrics

If the pod is running but the endpoint times out, verify the service is correctly configured:

kubectl get svc -n kube-system karpenter-metrics

Prometheus not scraping Karpenter

Check the Prometheus targets page to see if the Karpenter endpoint is listed. If it appears as down, verify the ServiceMonitor namespace selector matches the Karpenter service namespace (kube-system). Prometheus ServiceMonitors require exact label matches.

Node provisioning failing with “insufficient capacity”

This means EC2 does not have the requested instance type available in the chosen availability zone. Karpenter automatically retries with different instance types if your NodePool includes fallback options. To reduce capacity errors, expand your NodePool instanceTypes to include multiple families (e.g., m5, m6i, c5, c6i).

Pod startup time exceeding 5 minutes

Slow node provisioning is usually caused by AMI pull delays or EC2 throttling during large scale-up events. Check the Karpenter logs for throttling exception messages. If throttling is occurring, spread node launches across multiple availability zones or request an EC2 API rate limit increase from AWS Support.

Spot interruptions causing pod rescheduling storms

Karpenter handles Spot interruptions by draining nodes gracefully before termination, but if interruption rates spike, pods may reschedule faster than new nodes can launch. Monitor the karpenter_interruption_actions_performed_total metric. If interruptions exceed 10 per minute, consider mixing On Demand capacity into your NodePools using karpenter.sh/capacity-type requirements.

Successfully monitoring Karpenter on AWS EKS requires visibility into metrics, logs, and Kubernetes events. This guide covered deploying Prometheus to scrape Karpenter metrics, configuring CloudWatch logging, setting up alerts for provisioning failures and delays, and integrating with observability platforms. By tracking key metrics like karpenter_nodes_created, karpenter_pods_startup_duration_seconds, and karpenter_cloudprovider_errors_total, you can detect scaling issues before they impact production workloads.

Teams running Karpenter at scale should prioritize three areas: multi-zone capacity planning to reduce “insufficient capacity” errors, aggressive alerting on EC2 API throttling, and correlation of Karpenter metrics with application performance data to understand the full impact of node provisioning delays.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions