Book a Demo
Book a Demo
CubeAPM
CubeAPM CubeAPM

Kubernetes Service 503 Error Explained: Readiness Probes, Endpoints Failures, and Observability Best Practices

Kubernetes Service 503 Error Explained: Readiness Probes, Endpoints Failures, and Observability Best Practices

Table of Contents

A Kubernetes “Service 503 (Service Unavailable)” error occurs when traffic can’t reach any healthy Pod or endpoint, often during rollouts, probe failures, or misconfigurations. The impact is immediate—requests fail, frontends break, and downtime follows. In fact, over half of major outages cost more than $100,000. Even a short 503 storm can trigger financial loss and erode user trust.

CubeAPM helps reduce the blast radius of 503 errors by correlating failing service endpoints with Pod readiness, rollout events, and error logs in real time. Instead of chasing symptoms across dashboards, teams see exactly which Deployment or probe failure triggered the outage. This makes root cause detection faster and recovery far less disruptive.

In this guide, we’ll break down the root causes of the Kubernetes Service 503 error, explain step-by-step fixes, and show how to monitor and prevent it using CubeAPM.

What is Kubernetes ‘Service 503’ (Service Unavailable) Error

Kubernetes ‘Service 503’ (Service Unavailable) Error

This error is particularly common during deployment rollouts, health check failures, or misconfigured services. For example, if all Pods behind a Service fail readiness probes, Kubernetes temporarily removes them from the endpoints list. Similarly, if an Ingress routes to an empty backend, requests are dropped with a 503. These failures are disruptive because they often happen during live traffic shifts, amplifying user impact.

Typical causes include:

     

      • No endpoints are available — All Pods are failing readiness checks or not yet started.

      • Service discovery breaks — Endpoints are not registered properly in kube-proxy or DNS.

      • Ingress/LoadBalancer misroutes traffic — An Ingress controller or external LB points to an empty backend.

      • Rollouts temporarily drain Pods — During updates, Pods are taken down faster than new ones come online.

    In short, a Service 503 error is Kubernetes signaling that it knows the Service exists, but has no healthy destination to forward requests to.

    Why Kubernetes ‘Service 503’ (Service Unavailable) Error Happens

    1. No Ready Endpoints

    If all Pods behind a Service fail readiness probes, Kubernetes may return a 503 Service Error until at least one endpoint becomes healthy. The Service object continues to exist, but since there are no active targets, every request is met with a 503 response. This is one of the most common root causes during workload initialization.

    Quick check:

    Bash

    kubectl get endpoints <service-name>

    If the endpoints list is empty, this is the root cause.

    2. Misconfigured Service Selector

    Services depend on correct label matching to forward traffic. If the Service spec points to labels that don’t match any Pods, endpoints never get registered. This mismatch can happen after updates or refactors where labels were changed but not updated in the Service definition.

    Quick check:

    Bash

    kubectl describe service <service-name>

    Compare Selector labels with kubectl get pods –show-labels.

    3. Ingress or Load Balancer Misrouting

    When an Ingress controller or external LoadBalancer points to a backend Service with no active Pods, requests drop with a 503. It’s often caused by configuration errors or when backends are drained prematurely during upgrades. This issue tends to be more visible in multi-cluster or hybrid networking setups.

    Quick check:

    Bash

    kubectl describe ingress <ingress-name>

    Verify backend services and confirm they have healthy Pods.

    4. Rollout Draining Pods Too Quickly

    In rolling updates, Pods may be terminated before new ones are marked Ready. If there’s no overlap, requests hit an empty pool, leading to 503s. Misconfigured deployment strategies with high maxUnavailable or low maxSurge values make this problem worse.

    Quick check:

    Bash

    kubectl rollout status deployment <deployment-name>

    Check if Pods were scaled down faster than replacements came up.

    5. DNS or kube-proxy Sync Issues

    Kubernetes relies on kube-proxy and CoreDNS to keep Service discovery in sync. If kube-proxy rules are stale or DNS caching points to terminated Pods, clients may see intermittent 503s. This is especially common during node restarts or control plane instability.

    Quick check:

    Bash

    kubectl get pods -n kube-system | grep -E "coredns|kube-proxy"

    Look for frequent restarts or failed Pods.

    6. Pod Resource Starvation

    Even if Pods are technically “running,” CPU or memory starvation may cause them to fail readiness probes intermittently. Kubernetes then marks them Unready, effectively leaving the Service with no healthy endpoints. These transient failures often create short-lived 503 spikes during traffic bursts.

    Quick check:

    Bash

    kubectl top pod <pod-name>

    Check if resource usage is exceeding limits.

    7. Network Policy or Istio/Envoy Misconfiguration

    When NetworkPolicies or service mesh sidecars (Istio, Envoy) are misconfigured, Pods may be blocked from accepting traffic even if they appear healthy. In these cases, the Service technically routes requests, but connections are denied or dropped, surfacing as 503 errors at the client.

    Quick check:

    Bash

    kubectl describe networkpolicy -n <namespace>

    Review rules for allowed ingress/egress traffic.

    How to Fix Kubernetes ‘Service 503’ (Service Unavailable) Error

    1. Restore Ready Endpoints (Readiness/Startup Probes)

    If all Pods are Unready, the Service has zero endpoints. First confirm probe failures, then relax thresholds or fix the app start path.
    Check Events and recent Pod logs to see probe failures and adjust probe timing if startup is slow.

    Check current endpoints and probe status:

    Bash

    kubectl get endpoints <service-name> -o wide

    Bash

    kubectl describe pod <pod-name>

    Quick probe tune (example patch to increase startup grace):

    Bash

    kubectl patch deployment <deploy> --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/startupProbe","value":{"httpGet":{"path":"/healthz","port":8080},"failureThreshold":30,"periodSeconds":5}}]'

    Redeploy to pick changes:

    Bash

    kubectl rollout restart deployment <deploy>

    2. Fix Service–Pod Label Selector Mismatch

    If selectors don’t match Pod labels, Kubernetes registers no endpoints. Align the Service selector with the Deployment’s labels or vice-versa.

    Compare labels:

    Bash

    kubectl get pods -l app=<label> --show-labels

    Bash

    kubectl describe service <service-name>

    Patch Service selector to match Deployment labels:

    Bash

    kubectl patch service <service-name> -p='{"spec":{"selector":{"app":"<correct-label>"}}}'

    3. Repair Ingress or Load Balancer Backends

    Ingress routing to an empty or wrong Service results in 503s. Verify the Ingress backend Service and its endpoints; fix the Service name/port or the Ingress rule.

    Show Ingress backends and health:

    Bash

    kubectl describe ingress <ingress-name>

    Bash

    kubectl get endpoints <backend-service> -o wide

    Patch Ingress backend service/port (example):

    Bash

    kubectl patch ingress <ingress-name> --type='json' -p='[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service/name","value":"<service-name>"},{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service/port/number","value":8080}]'

    Bounce the controller if it’s stuck:

    Bash

    kubectl rollout restart deployment -n ingress-nginx ingress-nginx-controller

    4. Fix Rollout Strategy That Drains Pods Too Fast

    Aggressive maxUnavailable can leave zero Ready Pods mid-rollout. Ensure overlap between old and new Pods by using a safer strategy and readiness gates.

    Inspect strategy and rollout:

    Bash

    kubectl get deploy <deploy> -o jsonpath='{.spec.strategy.rollingUpdate}'

    Bash

    kubectl rollout status deployment <deploy>

    Patch strategy to maintain capacity (example):

    ‘

    Bash

    kubectl patch deployment <deploy> -p='{"spec":{"strategy":{"rollingUpdate":{"maxUnavailable":"0","maxSurge":"25%"}}}}

    Optionally increase replicas temporarily:

    Bash

    kubectl scale deployment <deploy> --replicas=<higher-count>

    5. Refresh DNS/kube-proxy Service Discovery

    Stale kube-proxy rules or DNS cache can misroute to nowhere. Check CoreDNS and kube-proxy health; restart if crash-looping and clear bad caches.

    Check system components quickly:

    Bash

    kubectl get pods -n kube-system -o wide | grep -E "coredns|kube-proxy"

    Restart unhealthy CoreDNS Pods (safe, stateless):

    Bash

    kubectl rollout restart deployment -n kube-system coredns

    Force proxy refresh by restarting a problematic node’s proxy (DaemonSet example name may vary):

    Bash

    kubectl delete pod -n kube-system -l k8s-app=kube-proxy
``]
 
---
 
### 6. Remove Pod Resource Starvation Causing Unready Flaps
Starved Pods fail probes intermittently, emptying endpoints and yielding 503 bursts.
*Explain first:* Right-size requests/limits and confirm HPA isn’t lagging behind traffic.
 
**Check live usage vs limits:**
```bash
kubectl top pod -n <ns>

    Patch container resources (example bump):

    Bash

    kubectl patch deployment <deploy> --type='json' -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources","value":{"requests":{"cpu":"250m","memory":"512Mi"},"limits":{"cpu":"1","memory":"1Gi"}}}]'

    Scale temporarily during spikes:

    Bash

    kubectl scale deployment <deploy> --replicas=<n>

    6. Open Traffic Paths Blocked by NetworkPolicy or Mesh

    NetworkPolicies or sidecar/mTLS policies can block traffic despite “healthy” Pods. Validate that Service ports are allowed and mesh route/authorization rules permit traffic.

    List NetworkPolicies and spot denials:

    Bash

    kubectl get networkpolicy -n <ns> -o wide

    Allow Service port (example minimal ingress rule):

    Bash

    kubectl apply -n <ns> -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-svc-port-8080
spec:
  podSelector: {matchLabels: {app: <label>}}
  policyTypes: ["Ingress"]
  ingress:
  - ports:
    - port: 8080
EOF

    If using Istio, confirm VirtualService routes and DestinationRule subsets exist:

    Bash

    kubectl describe virtualservice -n <ns> <vs-name>

    Bash

    kubectl describe destinationrule -n <ns> <dr-name>

    Quick mesh fix (example host/port correction in VirtualService):

    Bash

    kubectl patch virtualservice -n <ns> <vs-name> --type='json' -p='[{"op":"replace","path":"/spec/http/0/route/0/destination/host","value":"<service-name>.<ns>.svc.cluster.local"},{"op":"replace","path":"/spec/http/0/route/0/destination/port/number","value":8080}]'

    Monitoring Kubernetes ‘Service 503’ (Service Unavailable) Error with CubeAPM

    Opening explainer paragraph

    Fastest path to root cause: correlate four signal streams in one timeline—Events, Metrics, Logs, and Rollouts—so you can see the exact moment endpoints went empty, which probe failed, and which Deployment/Ingress change caused it. CubeAPM stitches these signals automatically, so a 503 spike is traced back to the specific readiness failure, selector mismatch, or misrouted backend within minutes (see docs: docs.cubeapm.com).

    Step 1 — Install CubeAPM (Helm)

    Install or upgrade the lightweight CubeAPM agent to receive OTLP from your collectors (set your tenant and token; values.yaml can hold secrets and scrape rules).

    Install (fresh):

    Bash

    helm repo add cubeapm https://charts.cubeapm.com && helm upgrade --install cubeapm-agent cubeapm/agent --namespace cubeapm --create-namespace --set otlp.endpoint=https://ingest.cubeapm.com:4317 --set auth.token=<CUBEAPM_TOKEN> --set cluster.name=<CLUSTER_NAME>

    Upgrade (existing):

    Bash

    helm upgrade cubeapm-agent cubeapm/agent --namespace cubeapm --set otlp.endpoint=https://ingest.cubeapm.com:4317 --set auth.token=<CUBEAPM_TOKEN> --set cluster.name=<CLUSTER_NAME>

    Use a values.yaml to keep tokens and custom labels out of the command line, e.g., –values values.yaml.

    Step 2 — Deploy the OpenTelemetry Collector (DaemonSet + Deployment)

    Use DaemonSet for node/pod–level metrics & logs (kubelet, container logs, Events) and Deployment for cluster/ingress scraping and central pipelines (Prometheus scraping, k8scluster, processors, batching).

    DaemonSet (node collectors):

    Bash

    helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts && helm upgrade --install otel-ds open-telemetry/opentelemetry-collector --namespace observability --create-namespace --values otel-ds-values.yaml

    Deployment (central pipeline):

    Bash

    helm upgrade --install otel-ctrl open-telemetry/opentelemetry-collector --namespace observability --values otel-ctrl-values.yaml

    Step 3 — Collector Configs Focused on Service 503

    3.1 DaemonSet config (minimal, 503-focused)

    YAML

    receivers:
  k8s_events: {}
  filelog:
    include: [/var/log/containers/*ingress*.log, /var/log/containers/*gateway*.log, /var/log/containers/*app*.log]
    start_at: beginning
    operators:
      - type: regex_parser
        regex: 'HTTP/(?P<http_version>\d\.\d)" (?P<status>\d{3})'
        parse_to: attributes
      - type: filter
        expr: 'attributes["status"] == "503"'
  kubeletstats:
    collection_interval: 30s
    auth_type: serviceAccount
    endpoint: "${KUBE_NODE_NAME}:10250"
    insecure_skip_verify: true
 
processors:
  k8sattributes: {}
  batch: {}
 
exporters:
  otlp:
    endpoint: ingest.cubeapm.com:4317
    tls: { insecure: false }
    headers: { "Authorization": "Bearer ${CUBEAPM_TOKEN}" }
 
service:
  pipelines:
    logs/ingress_503:
      receivers: [filelog]
      processors: [k8sattributes, batch]
      exporters: [otlp]
    metrics/node:
      receivers: [kubeletstats]
      processors: [k8sattributes, batch]
      exporters: [otlp]
    logs/events:
      receivers: [k8s_events]
      processors: [k8sattributes, batch]
      exporters: [otlp]

       

        • filelog tails container logs and filters only 503s from ingress/app containers for high-signal troubleshooting.

        • k8s_events captures readiness probe failures, scale events, rollout warnings aligned to the same timeline.

        • kubeletstats surfaces node pressure (CPU/mem) that can cause transient Unready Pods → empty endpoints → 503s.

        • k8sattributes enriches with Pod/Namespace/Deployment; batch optimizes export; otlp ships to CubeAPM.

      3.2 Deployment config (cluster & ingress scraping)

      YAML

      receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-apiservers'
          kubernetes_sd_configs: [{ role: endpoints }]
          relabel_configs: [{ action: keep, source_labels: [__meta_kubernetes_service_name], regex: kubernetes }]
        - job_name: 'kube-state-metrics'
          static_configs: [{ targets: ['kube-state-metrics.kube-system.svc:8080'] }]
        - job_name: 'ingress-nginx'
          kubernetes_sd_configs: [{ role: pod }]
          relabel_configs:
            - action: keep
              source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
              regex: ingress-nginx
processors:
  k8sattributes: {}
  transform:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["is_503"], true) where metric.name == "nginx_ingress_controller_requests" and attributes["status"] == "503"
  batch: {}
 
exporters:
  otlp:
    endpoint: ingest.cubeapm.com:4317
    tls: { insecure: false }
    headers: { "Authorization": "Bearer ${CUBEAPM_TOKEN}" }
 
service:
  pipelines:
    metrics/ingress:
      receivers: [prometheus]
      processors: [k8sattributes, transform, batch]
      exporters: [otlp]

         

          • prometheus scrapes ingress-nginx metrics and kube-state-metrics (to correlate empty endpoints, rollout state).

          • transform tags 503 datapoints for fast filtering and dashboarding.

          • k8sattributes adds workload labels for Service/Deployment correlation; batch and otlp forward to CubeAPM.

        Step 4 — Supporting Components (optional but recommended)

        Some correlations require additional sources like kube-state-metrics.

        kube-state-metrics install:

        YAML

        helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && helm upgrade --install kube-state-metrics prometheus-community/kube-state-metrics --namespace kube-system

        Step 5 — Verification (What You Should See in CubeAPM)

           

            • Events: You should see readiness probe failures, rollout warnings, and scaling events time-aligned with 503 spikes.

            • Metrics: You should see ingress requests labeled with status=503, plus endpoint counts dropping to zero during incidents.

            • Logs: You should see ingress/app logs filtered to 503 with Pod/Service/Namespace attributes for instant pivoting.

            • Restarts: You should see Pod restarts or OOMKilled patterns preceding endpoint depletion on the same timeline.

            • Rollout context: You should see Deployment/Ingress changes (image updates, strategy changes) correlated with the first 503 burst.

          Example Alert Rules for Kubernetes “Service 503” Error

          1) Ingress 503 Rate Spike

          A sudden surge of HTTP 503s at the edge usually means your backends have no Ready endpoints or your Ingress is routing to an empty Service. Alert on the per-ingress 503 rate so you can pivot to the exact backend quickly.

          YAML

          - alert: Ingress503RateHigh
  expr: sum by (namespace,ingress) (rate(nginx_ingress_controller_requests{status="503"}[5m])) > 5
  for: 5m
  labels: {severity: page, team: platform}
  annotations: {summary: "High 503 rate on Ingress {{ $labels.ingress }}", description: "503s >5 rps for 5m. Check endpoints and readiness for backend Services."}

          2) Service With Zero Endpoints

          If a Service has zero available endpoints, every request routed to it will return 503. This catches label-selector mistakes, drained rollouts, or mass readiness failures.

          YAML

          - alert: ServiceZeroEndpoints
  expr: sum by (namespace,service) (kube_endpoint_address_available) == 0
  for: 2m
  labels: {severity: critical}
  annotations: {summary: "Service {{ $labels.service }} has 0 endpoints", description: "No Ready endpoints for 2m. Verify selectors, readiness probes, and rollout status."}

          3) Deployment Availability Gap During Rollout

          Aggressive strategies (maxUnavailable) can momentarily drop Available replicas to 0, creating a 503 window. Alert when a deployment’s available/desired ratio dips.

          YAML

          - alert: DeploymentAvailabilityDrop
  expr: (kube_deployment_status_replicas_available / kube_deployment_spec_replicas) < 0.2
  for: 3m
  labels: {severity: warning}
  annotations: {summary: "Available replicas low for {{ $labels.deployment }}", description: "Available/Desired <20% for 3m; rollout may be draining faster than Pods become Ready."}

          4) Readiness Failures Draining Endpoints

          Waves of failing readiness probes will empty the Service endpoints list, surfacing as 503s. Watch for a spike in not-ready containers.

          YAML

          - alert: ReadinessFailuresSpike
  expr: sum by (namespace, pod) (1 - max by (namespace,pod,container) (kube_pod_container_status_ready{condition="true"})) > 0
  for: 5m
  labels: {severity: warning}
  annotations: {summary: "Readiness failures detected", description: "Containers reporting NotReady for 5m; endpoints may be dropping to zero."}

          5) Istio/Envoy 503s (Service Mesh Backends)

          In service-mesh environments, Envoy will emit 503s even when Pods look healthy if routes/authN/Z are wrong. Alert on 503s at the mesh layer to catch policy/routing issues.

          YAML

          - alert: Istio503RateHigh
  expr: sum by (destination_workload, destination_workload_namespace) (rate(istio_requests_total{response_code="503"}[5m])) > 5
  for: 5m
  labels: {severity: page, mesh: istio}
  annotations: {summary: "Mesh 503s to {{ $labels.destination_workload }}", description: "Sustained 503s via Envoy. Check VirtualService/DestinationRule and authorization policies."}

          6) CoreDNS / kube-proxy Instability (Discovery at Risk)

          If CoreDNS or kube-proxy is flapping, Services can’t resolve/endpoints can go stale—leading to intermittent 503s. Page when control-plane networking components restart.

          YAML

          - alert: DiscoveryPlaneUnstable
  expr: increase(kube_pod_container_status_restarts_total{namespace="kube-system",container=~"coredns|kube-proxy"}[10m]) > 0
  for: 10m
  labels: {severity: critical}
  annotations: {summary: "CoreDNS/kube-proxy instability", description: "Discovery plane restarts in last 10m; Service routing may be inconsistent."}

          7) Backend Saturation Causing Unready Flaps

          Resource starvation makes Pods intermittently Unready, shrinking endpoints and producing bursty 503s. Alert on sustained CPU throttling or OOM symptoms on the backend.

          YAML

          - alert: BackendSaturationRisk
  expr: sum by (namespace,pod) (rate(container_cpu_cfs_throttled_seconds_total{container!=""}[5m])) > 1 or increase(container_oom_events_total[10m]) > 0
  for: 5m
  labels: {severity: warning}
  annotations: {summary: "Backend saturation/unreliability", description: "CPU throttling or OOMs detected; readiness may flap and trigger 503s."}

          Conclusion

          Kubernetes ‘Service 503’ errors usually mean your Service exists but has no healthy endpoints to send traffic to—often due to rollout gaps, probe failures, or routing misconfigurations. Left unchecked, even brief 503 spikes can ripple into customer-visible downtime.

          CubeAPM shortens time-to-diagnosis by correlating Events, Metrics, Logs, and Rollouts on a single timeline, so you can see exactly when endpoints dropped to zero, which probe failed, and which Deployment or Ingress change triggered it.

          Set up the collectors and alerts above, then use CubeAPM’s correlated view to validate fixes and prevent recurrences. Ready to harden your clusters against 503s? Instrument now and ship with confidence.

          FAQs

          1. What does a Kubernetes Service 503 error mean?

          It indicates that the Service exists, but Kubernetes has no healthy Pod endpoints to forward requests to. Platforms like CubeAPM help validate this quickly by correlating endpoint availability with Service events.

          A 503 means traffic reached the cluster and the Service object was found, but endpoint resolution failed inside Kubernetes. With CubeAPM, you can confirm this distinction by aligning service-level logs with cluster networking metrics.

          Yes. Using safer rollout strategies such as keeping at least one Pod available during updates helps prevent outages. CubeAPM makes this easier by overlaying rollout events and 503 spikes, so you can fine-tune strategies with confidence.

          You can view the Service’s endpoints in Kubernetes to confirm if any Pods are available. CubeAPM surfaces the same information in its dashboards, alongside readiness failures and Pod logs.

          Start by verifying endpoints, selectors, and rollout status, then review ingress logs and readiness probes. With CubeAPM, you can do this in one place by viewing Events, Logs, Metrics, and Rollouts aligned on a single timeline.

          Book a Demo

          Related Posts

          ×