How to Monitor AWS EKS Control Plane Metrics (API Server Latency, etcd, Scheduler): Step-by-Step Guide

AWS manages the EKS control plane in a separate account, which means direct metric scraping is impossible. API server latency spikes, etcd write delays, or scheduler queue backlogs often surface only after they have degraded cluster performance. The upstream Kubernetes SLO defines acceptable API server latency at the 99th percentile as under one second for resource-returning requests. Most teams discover they have exceeded this threshold hours after user impact has already occurred.

This guide shows how to surface AWS EKS control plane metrics using CloudWatch Container Insights, Prometheus federated scraping, and CloudWatch Logs Insights queries. Each step includes configuration examples, metric definitions, and alert thresholds sourced from upstream Kubernetes SLOs and AWS observability documentation.

Prerequisites

Before monitoring EKS control plane metrics, ensure the following requirements are met:

• An active AWS EKS cluster running version 1.24 or later
• IAM permissions to enable CloudWatch Container Insights and access CloudWatch Logs
• kubectl configured to access the target EKS cluster
• Helm 3.x installed if using Prometheus for metric collection
• CloudWatch Container Insights enabled on the cluster (covered in Step 1)
• Basic familiarity with Prometheus query syntax and CloudWatch Logs Insights query language

Step 1: Enable CloudWatch Container Insights for EKS Control Plane Metrics

CloudWatch Container Insights surfaces control plane metrics that AWS exposes from the managed EKS control plane. These include API server request counts, latency percentiles, etcd object counts, and admission controller performance.

Navigate to the EKS console, select your cluster, and go to the Observability tab. Click Enable Container Insights if not already active. Alternatively, enable it via the AWS CLI:

aws eks update-cluster-config \
  --region us-east-1 \
  --name production-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

This command enables all five control plane log types. Each log stream feeds CloudWatch Logs and powers downstream metric extraction. Verify logging is active:

aws eks describe-cluster \
  --region us-east-1 \
  --name production-cluster \
  --query 'cluster.logging'

Once enabled, control plane logs appear in CloudWatch Logs under /aws/eks/production-cluster/cluster. Metrics derived from these logs surface in the CloudWatch Metrics console under the AWS/EKS namespace within 5 to 10 minutes.

Step 2: Query API Server Latency Metrics in CloudWatch

API server latency is the most direct indicator of control plane health. High latency suggests the API server is overloaded, throttling requests, or waiting on slow etcd writes. The upstream Kubernetes SLO defines acceptable latency as under one second at the 99th percentile for resource-returning requests.

Open the CloudWatch console and navigate to All Metrics. Select the AWS/EKS namespace. Choose the metric apiserver_request_duration_seconds. This metric aggregates API server request durations across all request verbs (GET, LIST, POST, PATCH, DELETE) and resource types.

Filter by cluster name and request verb to isolate LIST operations, which are the most latency-sensitive:

• Dimension: cluster_name = production-cluster
• Dimension: verb = LIST
• Statistic: p99 (99th percentile)

Set the period to 1 minute for near real time visibility. If the p99 latency exceeds one second consistently, the API server is likely saturated or waiting on slow etcd responses.

Create a CloudWatch alarm on this metric:

aws cloudwatch put-metric-alarm \
  --alarm-name eks-api-server-latency-high \
  --alarm-description "API server p99 latency exceeds 1 second" \
  --metric-name apiserver_request_duration_seconds \
  --namespace AWS/EKS \
  --statistic p99 \
  --period 60 \
  --threshold 1.0 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --dimensions Name=cluster_name,Value=production-cluster Name=verb,Value=LIST

This alarm fires if p99 latency exceeds one second for two consecutive minutes. Adjust the threshold based on your SLO requirements.

Step 3: Track etcd Performance with CloudWatch Metrics

etcd is the control plane’s backing store. Slow etcd writes directly cause API server latency spikes. AWS exposes etcd metrics through CloudWatch Container Insights including etcd database size, backend commit duration, and leader changes.

In the CloudWatch Metrics console under AWS/EKS, locate the following etcd metrics:

• etcd_db_total_size_in_bytes — total etcd database size. If this grows unbounded, it signals retention issues or excessive object churn.
• etcd_disk_backend_commit_duration_seconds — duration of etcd disk commits. The upstream Kubernetes SLO recommends p99 commit duration under 25 milliseconds for healthy etcd performance.
• etcd_server_leader_changes_seen_total — cumulative count of etcd leader elections. Frequent leader changes indicate network instability or resource pressure.

Query etcd commit duration at the 99th percentile:

• Metric: etcd_disk_backend_commit_duration_seconds
• Dimension: cluster_name = production-cluster
• Statistic: p99
• Period: 1 minute

If p99 commit duration exceeds 25 milliseconds, etcd is struggling with disk I/O. This often correlates with API server latency spikes. Check AWS EBS volume IOPS and burst credits if etcd commit duration is consistently high.

Create an alert for slow etcd commits:

aws cloudwatch put-metric-alarm \
  --alarm-name eks-etcd-commit-duration-high \
  --alarm-description "etcd p99 commit duration exceeds 25ms" \
  --metric-name etcd_disk_backend_commit_duration_seconds \
  --namespace AWS/EKS \
  --statistic p99 \
  --period 60 \
  --threshold 0.025 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --dimensions Name=cluster_name,Value=production-cluster

Step 4: Monitor Scheduler Queue Depth with CloudWatch Logs Queries

The Kubernetes scheduler assigns pods to nodes. If the scheduler queue depth grows, it means pods are waiting longer to be scheduled. This surfaces as slow application deployment or autoscaling lag.

AWS does not expose scheduler queue depth as a direct CloudWatch metric. Instead, extract it from scheduler logs using CloudWatch Logs Insights. Navigate to CloudWatch Logs Insights and select the log group /aws/eks/production-cluster/cluster. Run this query:

fields @timestamp, @message
| filter @logStream like /scheduler/
| filter @message like /unable to schedule pod/
| stats count() as unscheduled_pods by bin(5m)

This query counts pods that failed to schedule in five minute intervals. A rising count indicates node capacity pressure, resource constraints, or pod affinity rules blocking scheduling.

For clusters running Kubernetes 1.27 or later, the scheduler emits schedule_attempts_total and pending_pods metrics to its /metrics endpoint. Since AWS does not expose this endpoint directly, you can scrape it by deploying a Prometheus instance with federated scraping (covered in Step 6).

Step 5: Set Up Alerts for API Server Request Rate and Error Rates

High request rates or elevated error rates signal API server stress or misconfigured controllers making excessive API calls. CloudWatch surfaces apiserver_request_total as a counter of all API requests and apiserver_request_error_rate as the fraction of requests returning errors.

Query total API requests per minute:

• Metric: apiserver_request_total
• Dimension: cluster_name = production-cluster
• Statistic: Sum
• Period: 1 minute

If the request rate spikes without corresponding workload growth, investigate controllers or operators making excessive LIST or WATCH calls. Use CloudWatch Logs Insights to surface the userAgent making the requests:

fields @timestamp, userAgent, requestURI
| filter @logStream like /audit/
| filter verb = "list"
| stats count() as request_count by userAgent
| sort request_count desc

This query identifies which controller or client is making the most LIST requests. Common offenders include custom operators, monitoring agents, or CI/CD tools polling cluster state excessively.

Create an alert for API error rate:

aws cloudwatch put-metric-alarm \
  --alarm-name eks-api-server-error-rate-high \
  --alarm-description "API server error rate exceeds 1%" \
  --metric-name apiserver_request_error_rate \
  --namespace AWS/EKS \
  --statistic Average \
  --period 60 \
  --threshold 0.01 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --dimensions Name=cluster_name,Value=production-cluster

Step 6: Deploy Prometheus to Scrape Additional Control Plane Metrics

CloudWatch Container Insights exposes a subset of control plane metrics. For deeper visibility, deploy Prometheus in the cluster and configure it to scrape the API server /metrics endpoint via the Kubernetes service account token.

Install Prometheus using Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus \
  --namespace monitoring \
  --create-namespace \
  --set server.persistentVolume.size=50Gi

Configure Prometheus to scrape the API server metrics endpoint. Add this job to the Prometheus configuration:

scrape_configs:
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

This configuration scrapes the API server /metrics endpoint using the pod’s service account token. Metrics include apiserver_request_duration_seconds, apiserver_current_inflight_requests, and etcd_request_duration_seconds.

Query API server request duration in Prometheus:

histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket[5m])) by (le, verb))

This query calculates the 99th percentile request duration by verb over the last five minutes. If the result exceeds one second for LIST or GET operations, the API server is under stress.

Step 7: Correlate Control Plane Metrics with Data Plane Performance

Control plane issues often manifest as data plane symptoms: slow pod startup, failed deployments, or delayed autoscaling. Infrastructure monitoring platforms help correlate control plane metrics with node health, pod resource usage, and application latency.

When troubleshooting, check these correlations:

• API server latency spike + etcd commit duration spike → etcd disk I/O bottleneck
• Scheduler queue depth increase + high node CPU → insufficient cluster capacity
• API request rate spike + specific userAgent → misconfigured controller

CubeAPM monitors Kubernetes control plane and data plane metrics in a unified view. It ingests Prometheus metrics, CloudWatch logs, and OpenTelemetry traces from application workloads. This allows teams to correlate API server latency with specific pod restarts, deployment failures, or application error rates. CubeAPM runs on premises or in your VPC, keeping all telemetry data inside your infrastructure. Pricing is $0.15/GB of ingested telemetry with unlimited retention and no per-seat fees.

For teams already using CloudWatch, AWS monitoring tools like CloudWatch Synthetics and X-Ray can complement control plane metrics by tracking end user latency and distributed trace data.

Troubleshooting Common Issues

API server latency is high but etcd commit duration is normal

This suggests the API server itself is overloaded, not waiting on etcd. Check API Priority and Fairness (APF) queue saturation using the apiserver_flowcontrol_rejected_requests_total metric. If this counter is rising, certain priority levels are saturated and requests are being dropped. Review APF configuration and consider increasing concurrency shares for high-priority queues.

etcd database size is growing unbounded

Excessive object churn or missing retention policies cause etcd to grow. Query the number of objects stored in etcd using CloudWatch Logs Insights:

fields @timestamp, @message
| filter @logStream like /audit/
| filter verb = "create"
| stats count() as object_creates by bin(1h)

If object creation rate is spiking, identify the controller creating objects excessively. Common causes include Helm releases with large ConfigMaps or operators creating thousands of CRDs.

Scheduler queue depth is rising but nodes have available capacity

This indicates pod affinity rules, taints, or tolerations are blocking scheduling. Query scheduler logs for the specific failure reason:

fields @timestamp, @message
| filter @logStream like /scheduler/
| filter @message like /FailedScheduling/
| display @message

The failure message identifies whether the issue is pod affinity mismatch, insufficient resources, or node selector constraints.

CloudWatch metrics are not appearing

Verify that Container Insights is enabled and that the CloudWatch agent is running in the cluster:

kubectl get pods -n amazon-cloudwatch

If no pods are running, install the CloudWatch agent using the AWS-provided manifest:

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml

Conclusion

Monitoring EKS control plane metrics requires combining CloudWatch Container Insights, Prometheus, and CloudWatch Logs Insights queries. API server latency, etcd commit duration, and scheduler queue depth are the three most critical signals for control plane health. Setting alerts based on upstream Kubernetes SLOs ensures problems surface before they degrade application performance.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions