How to Monitor Crossplane Resource Provisioning Failures: Step by Step Guide

Crossplane reconciliation failures happen silently until a critical cloud resource fails to provision. A managed SQS queue marked pending for hours, an SNS topic stuck in creating state, or a database instance that never reaches ready state can block an entire deployment without triggering any visible alert in your default Kubernetes monitoring setup.

According to the CNCF 2024 Annual Survey, 56% of organizations now use Kubernetes to manage cloud infrastructure, with Crossplane emerging as the leading infrastructure orchestration layer. Yet most teams only discover provisioning failures through manual kubectl checks or customer reported outages, not through proactive monitoring.

This guide walks through setting up complete Crossplane provisioning failure monitoring using kubectl commands, Prometheus metrics, event based alerts, and observability platforms that integrate with Crossplane natively. By the end, you will have a working monitoring stack that catches provisioning failures in real time and surfaces the exact root cause for faster resolution.

Prerequisites

Before setting up Crossplane monitoring, ensure you have:

• A working Crossplane installation (v1.14 or later recommended) with at least one provider configured
• kubectl access to the cluster running Crossplane with permissions to view resources, events, and metrics
• Basic familiarity with Crossplane architecture: managed resources, composite resources (XRs), and provider controllers
• Prometheus or an OpenTelemetry compatible observability platform deployed in cluster (optional but required for automated alerting)
• Access to your cloud provider console (AWS, Azure, GCP) to verify actual resource state during troubleshooting

If you are new to infrastructure monitoring, set up basic Kubernetes cluster monitoring first to establish baseline visibility before adding Crossplane specific signals.

Step 1: Enable Crossplane Metrics Export

Crossplane emits Prometheus style metrics on port 8080 by default, but these are not scraped automatically unless you configure your monitoring stack to discover them.

First, verify that Crossplane metrics are exposed:

kubectl port-forward -n crossplane-system deployment/crossplane 8080:8080
curl http://localhost:8080/metrics | grep crossplane_managed_resource

You should see output including metrics like crossplane_managed_resource_exists, crossplane_managed_resource_ready, and crossplane_managed_resource_synced. If you see no output, your Crossplane version may predate metrics support. Upgrade to v1.14 or later.

Next, configure Prometheus to scrape Crossplane metrics. If using the Prometheus Operator, create a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: crossplane-metrics
  namespace: crossplane-system
spec:
  selector:
    matchLabels:
      app: crossplane
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Apply this manifest:

kubectl apply -f crossplane-servicemonitor.yaml

Wait 60 seconds and verify metrics appear in Prometheus by querying crossplane_managed_resource_exists. If using a different observability platform, configure it to scrape the Crossplane pod on port 8080 using your platform’s Kubernetes service discovery mechanism.

Step 2: Monitor Managed Resource Conditions with kubectl

Every managed resource in Crossplane reports its state through Kubernetes conditions. The two critical conditions for provisioning monitoring are Ready and Synced.

List all managed resources across your cluster and filter by condition:

kubectl get managed -A

This command shows all managed resources with their Ready status. To drill into a specific resource that shows False, use kubectl describe:

kubectl describe cloudsqlinstance my-db -n production

Look for the Conditions section in the output:

Conditions:
  Last Transition Time: 2026-01-15T14:32:18Z
  Reason: Creating
  Status: False
  Type: Ready

Common failure reasons include:

• Creating — resource is still provisioning, not yet ready
• ReconcileError — provider failed to communicate with cloud API
• CannotInitializeManagedResource — Crossplane paused reconciliation due to transient failure (see Step 4)

For resources stuck in Creating for more than 10 minutes, check the cloud provider console directly. The resource may exist but failed to report back to Crossplane, or it may have failed silently in the cloud provider without surfacing an error.

To monitor conditions programmatically across all resources:

kubectl get managed -A -o json | jq '.items[] | select(.status.conditions[] | select(.type=="Ready" and .status=="False")) | {name: .metadata.name, namespace: .metadata.namespace, reason: .status.conditions[].reason}'

This outputs every resource currently not ready with its failure reason. Run this in a cron job or monitoring script to track resources that remain in failed state beyond your SLA threshold.

Step 3: Track Kubernetes Events for Provisioning Errors

Crossplane emits Kubernetes events when provisioning fails or reconciliation encounters errors. Events provide human readable context that metrics and conditions alone do not surface.

View recent events for a specific managed resource:

kubectl describe cloudsqlinstance my-db -n production | grep -A 10 Events

To search for all CannotInitializeManagedResource events across the cluster:

kubectl get events -A --field-selector type=Warning,reason=CannotInitializeManagedResource --sort-by=.lastTimestamp

This command surfaces resources where Crossplane explicitly paused reconciliation and requires manual intervention. These events include the message “cannot determine creation result – remove the crossplane.io/external-create-pending annotation if it is safe to proceed.”

For broader provisioning failure detection, filter events by type Warning:

kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | grep -E 'CannotConnectToProvider|ReconcileError|CannotCreateExternalResource'

Common event reasons to alert on:

• CannotConnectToProvider — provider credentials invalid or provider controller not running
• CannotCreateExternalResource — cloud API rejected the resource creation request
• ReconcileError — generic error during reconciliation, check provider logs for specifics

Events are namespaced in Kubernetes, but Crossplane emits events for cluster scoped resources (like XRs) to the default namespace. Always check both the resource namespace and the default namespace when investigating failures.

Step 4: Set Up Alerts on Circuit Breaker Metrics

Crossplane v1.16 introduced circuit breaker metrics to detect reconciliation thrashing. When a composite resource receives excessive watch events (over 100 per second by default), the circuit breaker opens and blocks most reconciliation requests to prevent cluster instability.

Monitor circuit breaker state using these Prometheus queries:

# Rate of circuit breaker opens over 5 minutes
rate(circuit_breaker_opens_total[5m])

# Percentage of events being dropped due to circuit breaker
sum by (controller) (rate(circuit_breaker_events_total{result="Dropped"}[5m])) 
/ 
sum by (controller) (rate(circuit_breaker_events_total[5m])) * 100

A circuit breaker opening indicates a deeper problem: either a composition is creating a reconciliation loop, or an external system is repeatedly reverting changes Crossplane makes. Both require immediate investigation.

Create a Prometheus alert for circuit breaker opens:

- alert: CrossplaneCircuitBreakerOpen
  expr: |
    sum by (controller) (
      rate(circuit_breaker_opens_total[5m])
    ) * 3600 > 6
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Crossplane circuit breaker opening frequently for {{ $labels.controller }}"
    description: "Circuit breaker for {{ $labels.controller }} is opening more than 6 times per hour, indicating reconciliation thrashing."

When this alert fires, use kubectl to find which resource is causing excessive watch events. The alert includes the controller label formatted as composite/xpostgresqlinstances.example.com. Query that XRD to find active XRs:

kubectl get xpostgresqlinstances.example.com -A

Then describe the XR to see its Responsive condition:

kubectl describe xpostgresqlinstances my-instance -n production

If the condition shows WatchCircuitOpen, the circuit breaker message identifies which composed resource is causing the loop. Address that resource first, then remove the crossplane.io/paused annotation to resume reconciliation.

Step 5: Monitor Provider Controller Health

Provider controllers run as separate pods in the crossplane-system namespace. If a provider pod crashes or restarts frequently, all managed resources for that provider will fail to reconcile.

List all provider pods and check their restart count:

kubectl get pods -n crossplane-system -l pkg.crossplane.io/provider

Look for pods with restart counts above 5 in the past hour. High restart counts indicate the provider is crash looping, often due to:

• Invalid cloud credentials in the ProviderConfig
• Memory limits set too low for the provider’s workload
• API rate limiting from the cloud provider

View provider logs to identify the root cause:

kubectl logs -n crossplane-system <provider-pod-name> --tail=100

Common log errors include cannot get credentials (ProviderConfig misconfigured) and rate limit exceeded (cloud API throttling).

Enable debug logging on a provider by adding a DeploymentRuntimeConfig:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: debug-config
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        spec:
          containers:
          - name: package-runtime
            args:
            - --debug
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-aws-s3:v2.0.0
  runtimeConfigRef:
    apiVersion: pkg.crossplane.io/v1beta1
    kind: DeploymentRuntimeConfig
    name: debug-config

Apply this config, wait for the provider to restart, and check logs again for detailed error context.

Step 6: Integrate Crossplane Metrics with an Observability Platform

Manual kubectl checks do not scale beyond a handful of resources. For production environments, integrate Crossplane metrics into an observability platform that supports Prometheus or OpenTelemetry.

CubeAPM offers native Crossplane monitoring by ingesting Prometheus metrics and correlating them with Kubernetes events and pod logs. Deploy CubeAPM in your cluster and configure it to scrape Crossplane metrics:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cubeapm-scrape-config
  namespace: cubeapm-system
data:
  prometheus.yml: |
    scrape_configs:
    - job_name: 'crossplane'
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
          - crossplane-system
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: crossplane
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

CubeAPM automatically correlates Crossplane managed resource metrics with Kubernetes events, surfacing provisioning failures in a unified dashboard. Set up alerts in CubeAPM for:

• crossplane_managed_resource_ready == 0 — resource not ready
• rate(crossplane_managed_resource_deletion_seconds[5m]) > 300 — resource taking longer than 5 minutes to delete
• circuit_breaker_opens_total — circuit breaker opening

CubeAPM runs inside your VPC, keeping Crossplane telemetry data local with no egress costs. It supports OpenTelemetry ingestion, so you can send Crossplane metrics alongside APM traces and application logs for full infrastructure to application correlation.

For teams already using Datadog or New Relic, configure those platforms to scrape Crossplane metrics using their Kubernetes integrations. Datadog requires a custom check configuration, while New Relic supports Prometheus endpoints natively through its Kubernetes integration.

Step 7: Automate Remediation with Alerting Workflows

Once monitoring surfaces provisioning failures, automate triage and remediation to reduce mean time to resolution.

Create a Prometheus alert that triggers when a managed resource remains not ready for more than 10 minutes:

- alert: CrossplaneManagedResourceNotReady
  expr: |
    crossplane_managed_resource_ready == 0
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Crossplane resource {{ $labels.name }} not ready for 10 minutes"
    description: "Managed resource {{ $labels.name }} in namespace {{ $labels.namespace }} has been not ready for over 10 minutes. Check kubectl describe and provider logs."

Route this alert to your incident management system (PagerDuty, Opsgenie, Slack). Include a runbook link in the alert annotation that walks the on call engineer through:

1. Running kubectl describe on the affected resource
2. Checking kubectl get events for recent errors
3. Viewing provider pod logs if the error is ReconcileError
4. Verifying the resource exists in the cloud provider console

For resources stuck with the external-create-pending annotation, automate the remediation check:

#!/bin/bash
RESOURCE=$1
NAMESPACE=$2

# Check if resource exists in cloud provider (AWS example for RDS)
INSTANCE_ID=$(kubectl get $RESOURCE -n $NAMESPACE -o json | jq -r '.status.atProvider.dbInstanceIdentifier')
aws rds describe-db-instances --db-instance-identifier $INSTANCE_ID > /dev/null 2>&1

if [ $? -eq 0 ]; then
  echo "Resource exists in AWS, safe to remove annotation"
  kubectl annotate $RESOURCE -n $NAMESPACE crossplane.io/external-create-pending-
else
  echo "Resource does not exist in AWS, investigate further before removing annotation"
fi

Run this script as part of your alert response workflow. Only remove the annotation if the cloud resource actually exists and matches the desired state.

Troubleshooting Common Issues

Managed resource stuck in Creating state for hours

First, verify the resource exists in the cloud provider console. If it does not exist:

• Check provider pod logs for API errors: kubectl logs -n crossplane-system <provider-pod>
• Verify ProviderConfig credentials: kubectl describe providerconfig default
• Confirm cloud API rate limits are not exceeded in provider logs

If the resource exists in the cloud but Crossplane does not see it:

• The resource may have been created manually outside Crossplane and is conflicting
• The provider may have lost connection during creation (check for pod restarts during that window)
• Remove the external-create-pending annotation only after confirming the resource is healthy in the cloud console

Circuit breaker opens repeatedly for the same XR

Identify the composed resource causing excessive watch events:

kubectl describe xr <xr-name> -n <namespace> | grep -A 5 Responsive

The condition message shows which resource is looping. Common causes:

• A composition patch references a field that triggers another reconciliation when updated
• An external controller (like a CRD webhook) is reverting changes Crossplane makes
• The composed resource has a status field that updates on every reconciliation

Fix the composition to avoid creating the loop, then reset the circuit breaker by restarting the Crossplane pod:

kubectl rollout restart deployment/crossplane -n crossplane-system

Provider pod crash looping with OOMKilled

Increase memory limits in the provider’s DeploymentRuntimeConfig:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: memory-config
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          containers:
          - name: package-runtime
            resources:
              limits:
                memory: 2Gi
              requests:
                memory: 1Gi

Reference this config from the Provider spec and wait for the deployment to update. Monitor memory usage with kubectl top pod to verify the limit is sufficient.

Crossplane provisioning failures are solvable with the right monitoring setup. By combining kubectl condition checks, Prometheus metrics, event based alerts, and an observability platform like CubeAPM, teams can detect failures in real time and resolve them before they block deployments or impact production workloads.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions