How to Monitor GCP Cloud Composer (Airflow) DAG Failures: Step by Step Guide 2026

Running Apache Airflow on Cloud Composer is convenient until a DAG fails silently and you discover it hours later when downstream systems stop receiving data. Google Cloud automatically exports Composer metrics to Cloud Monitoring, but the default monitoring setup misses critical failure signals that only appear in task logs or require custom metric filters. A 2025 report from the Cloud Native Computing Foundation found that 68% of organizations using managed workflow orchestration experienced at least one undetected pipeline failure that impacted production systems in the prior year.

This guide walks through setting up proactive Cloud Composer DAG failure monitoring using Cloud Monitoring alert policies, log-based metrics, Airflow callbacks, and third party observability platforms that unify logs, traces, and infrastructure signals in one place.

Prerequisites

Before setting up DAG failure monitoring for Cloud Composer, ensure you have:

• An active GCP project with a Cloud Composer environment running Airflow 2.x
• Cloud Monitoring API enabled on your project
• IAM permissions: monitoring.alertPolicies.create, monitoring.notificationChannels.create, logging.logMetrics.create
• At least one notification channel configured (email, Slack, PagerDuty)
• gcloud CLI installed and authenticated if using Terraform or CLI-based setup
• Basic familiarity with Airflow DAG structure and GCP Cloud Logging query syntax

Step 1: Understand How Cloud Composer Exposes DAG Metrics

Cloud Composer automatically publishes performance and health metrics to Cloud Monitoring under the composer.googleapis.com namespace. These metrics are available immediately after your Composer environment starts running and require no manual instrumentation.

The key metrics for DAG failure monitoring are:

composer.googleapis.com/workflow/run_count — Total count of DAG runs, labeled by state (success, failed, running). This is the primary metric for alerting on failed DAG runs.

composer.googleapis.com/workflow/task/run_count — Count of individual task runs by state. Use this to catch tasks that fail repeatedly even when the overall DAG eventually succeeds due to retry logic.

composer.googleapis.com/environment/healthy — Boolean metric showing whether the Composer environment itself is healthy. A false value here means the Airflow scheduler, web server, or database is unhealthy and DAGs cannot execute properly.

These metrics are labeled with resource dimensions including environment_name, workflow_name (the DAG ID), and project_id. You can filter alerts by specific DAGs, environments, or aggregate across all workflows in a project.

Cloud Monitoring automatically aggregates these metrics in time windows you define. For DAG failure alerts, a 5 minute alignment period works well — short enough to catch failures quickly but long enough to avoid alert noise from brief task retries.

Beyond these native metrics, you can create log-based metrics from Cloud Logging to track custom failure patterns. For example, a log-based metric that counts ERROR-level log entries from specific task operators or tracks OOMKilled events in Kubernetes pods running your tasks.

For teams that want unified observability across logs, traces, and infrastructure, infrastructure monitoring platforms can ingest these Cloud Monitoring metrics alongside application traces and Kubernetes signals to provide full context during incident triage.

Step 2: Set Up Notification Channels

Before creating alert policies, configure at least one notification channel where alerts will be sent when DAGs fail. Cloud Monitoring supports email, Slack, PagerDuty, SMS, webhooks, and Pub/Sub.

Navigate to the Cloud Console, open Cloud Monitoring, and go to Alerting > Edit Notification Channels.

For Slack integration, click Add New under Slack and authorize the Google Cloud app in your Slack workspace. Select the channel where alerts should be posted. Slack works well for informational alerts that the entire team should see.

For PagerDuty, you need an integration key from PagerDuty first. In PagerDuty, create a new service integration with Google Cloud Monitoring as the integration type. Copy the integration key, then paste it into the Cloud Monitoring notification channel setup. Use PagerDuty for critical DAG failures that require immediate on-call response.

You can also create notification channels via the gcloud CLI:

gcloud beta monitoring channels create \
  --type=email \
  --display-name="Data Platform Team Email" \
  [email protected]

After creating channels, note the channel name or resource ID. You will reference this when creating alert policies in the next steps.

Most teams configure two channels: a Slack channel for all DAG failures and a PagerDuty integration for business-critical DAGs only. This reduces on-call fatigue while ensuring critical pipelines get immediate attention.

Step 3: Create an Alert Policy for Failed DAG Runs

The most important alert to configure is one that fires when any DAG run fails. This alert watches the composer.googleapis.com/workflow/run_count metric filtered for state = failed.

In the Cloud Console, navigate to Monitoring > Alerting > Create Policy.

For the metric, search for composer.googleapis.com/workflow/run_count. Add a filter for metric.labels.state = "failed" to count only failed DAG runs.

Set the aggregation to sum with a 5 minute alignment period. This means Cloud Monitoring will sum all failed DAG runs across all DAGs in your environment over each 5 minute window.

For the condition threshold, set is above with a threshold value of 0. Any failed DAG run in the window will trigger the alert.

Set the duration to most recent value so the alert fires immediately when the condition is met, not after a sustained failure period.

Add your notification channels. Select both your Slack channel and PagerDuty integration if this is a critical environment.

Give the policy a clear name like “Cloud Composer – DAG Run Failed” and add documentation that tells responders what to check first. For example: “A DAG run has failed in Cloud Composer. Check the Airflow UI at https://your-environment-url.composer.googleusercontent.com for task logs and failure details.”

Save the policy.

Here is the same alert policy defined in Terraform, which is the recommended approach for managing alert policies as infrastructure code:

resource "google_monitoring_alert_policy" "composer_dag_failure" {
  display_name = "Cloud Composer - DAG Run Failed"
  combiner     = "OR"
  
  conditions {
    display_name = "DAG run failure detected"
    
    condition_threshold {
      filter = <<-EOT
        resource.type = "cloud_composer_workflow"
        AND metric.type = "composer.googleapis.com/workflow/run_count"
        AND metric.labels.state = "failed"
      EOT
      
      aggregations {
        alignment_period   = "300s"
        per_series_aligner = "ALIGN_SUM"
      }
      
      comparison      = "COMPARISON_GT"
      threshold_value = 0
      duration        = "0s"
      
      trigger {
        count = 1
      }
    }
  }
  
  notification_channels = [
    google_monitoring_notification_channel.slack.name,
    google_monitoring_notification_channel.pagerduty.name,
  ]
  
  alert_strategy {
    auto_close = "1800s"  # Auto-close after 30 minutes
  }
  
  documentation {
    content   = "A DAG run has failed in Cloud Composer. Check the Airflow UI for details."
    mime_type = "text/markdown"
  }
}

This alert fires whenever any DAG fails in your Composer environment. If you have development DAGs running alongside production pipelines, you may get noise from expected failures in test runs. The next step shows how to filter alerts to only critical DAGs.

Step 4: Create Alert Policies for Specific High Priority DAGs

Not all DAG failures have equal impact. A revenue-critical ETL pipeline failing at 2 AM deserves an immediate PagerDuty page. A development DAG that someone left running in production probably does not.

You can filter alerts by DAG ID using the workflow_name resource label. For example, to alert only when specific business-critical DAGs fail:

resource "google_monitoring_alert_policy" "critical_dag_failure" {
  display_name = "Critical DAG Failure - Revenue Pipeline"
  combiner     = "OR"
  
  conditions {
    display_name = "Critical DAG failure"
    
    condition_threshold {
      filter = <<-EOT
        resource.type = "cloud_composer_workflow"
        AND metric.type = "composer.googleapis.com/workflow/run_count"
        AND metric.labels.state = "failed"
        AND resource.labels.workflow_name = monitoring.regex.full_match("(revenue_etl|billing_sync|customer_export)")
      EOT
      
      aggregations {
        alignment_period   = "300s"
        per_series_aligner = "ALIGN_SUM"
      }
      
      comparison      = "COMPARISON_GT"
      threshold_value = 0
      duration        = "0s"
      
      trigger {
        count = 1
      }
    }
  }
  
  notification_channels = [
    google_monitoring_notification_channel.pagerduty.name,
  ]
}

The monitoring.regex.full_match function lets you match multiple DAG IDs with a regex pattern. In this example, the alert fires only if revenue_etl, billing_sync, or customer_export fails. All other DAG failures are ignored by this policy.

This pattern works well when combined with a broader catch-all alert for all DAG failures sent to Slack and a narrower alert for critical DAGs sent to PagerDuty. Teams get visibility into all failures while on-call engineers only get paged for the ones that matter most.

You can also create alerts per DAG by replacing the regex with a single exact match: resource.labels.workflow_name = "revenue_etl". This gives maximum control but requires managing one alert policy per critical DAG.

Step 5: Alert on Task-Level Failures and High Retry Rates

Sometimes a DAG run succeeds overall because of retry logic, but individual tasks are failing repeatedly. This is a sign of an underlying problem — a flaky external API, intermittent database connection issues, or resource exhaustion in your Composer environment.

You can alert on task failures separately using the composer.googleapis.com/workflow/task/run_count metric:

resource "google_monitoring_alert_policy" "task_failure_rate" {
  display_name = "Cloud Composer - High Task Failure Rate"
  combiner     = "OR"
  
  conditions {
    display_name = "Task failure rate too high"
    
    condition_threshold {
      filter = <<-EOT
        resource.type = "cloud_composer_workflow"
        AND metric.type = "composer.googleapis.com/workflow/task/run_count"
        AND metric.labels.state = "failed"
      EOT
      
      aggregations {
        alignment_period   = "3600s"
        per_series_aligner = "ALIGN_SUM"
      }
      
      comparison      = "COMPARISON_GT"
      threshold_value = 10
      duration        = "0s"
      
      trigger {
        count = 1
      }
    }
  }
  
  notification_channels = [
    google_monitoring_notification_channel.slack.name,
  ]
}

This alert fires if more than 10 task failures occur in any 1 hour window across all DAGs. You can adjust the threshold based on your normal failure rate. A high task failure rate usually means something environmental is wrong — not a problem with one specific DAG.

For deeper task-level visibility, many teams use Google Cloud monitoring tools that correlate task failures with Kubernetes pod events, GKE node health, and Composer environment metrics to surface root causes faster than checking each signal separately.

Step 6: Monitor Composer Environment Health

Beyond individual DAG and task failures, you should monitor the overall health of your Composer environment. The composer.googleapis.com/environment/healthy metric is a boolean that goes false when the Airflow scheduler, web server, or database becomes unhealthy.

If the environment is unhealthy, no DAGs can run properly even if there are no DAG-level failures showing up in your alerts. This is a critical signal that requires immediate attention.

resource "google_monitoring_alert_policy" "composer_unhealthy" {
  display_name = "Cloud Composer - Environment Unhealthy"
  combiner     = "OR"
  
  conditions {
    display_name = "Environment health check failed"
    
    condition_threshold {
      filter = <<-EOT
        resource.type = "cloud_composer_environment"
        AND metric.type = "composer.googleapis.com/environment/healthy"
      EOT
      
      aggregations {
        alignment_period   = "300s"
        per_series_aligner = "ALIGN_FRACTION_TRUE"
      }
      
      comparison      = "COMPARISON_LT"
      threshold_value = 1
      duration        = "300s"
      
      trigger {
        count = 1
      }
    }
  }
  
  notification_channels = [
    google_monitoring_notification_channel.pagerduty.name,
  ]
  
  documentation {
    content = "The Cloud Composer environment is unhealthy. Check the environment details page and related scheduler, database, and worker metrics."
    mime_type = "text/markdown"
  }
}

The healthy metric is a boolean, so the alert uses ALIGN_FRACTION_TRUE aggregation. The alert fires if the environment is unhealthy for 5 continuous minutes. A brief health check failure that recovers quickly will not trigger the alert.

This alert should go directly to PagerDuty or another on-call system. An unhealthy Composer environment means all workflows are at risk.

Step 7: Use Log-Based Metrics for Custom Failure Patterns

Cloud Monitoring metrics cover DAG runs, task runs, and environment health. But sometimes you need to alert on patterns that only appear in logs — specific error messages, OOMKilled events in Kubernetes pods, or retries from a particular external API.

Log-based metrics let you convert log entries into time series metrics that you can alert on just like native metrics.

For example, to create a log-based metric that counts ERROR-level log entries from Airflow tasks:

Navigate to Logging > Logs-based Metrics > Create Metric.

Set the metric type to Counter and give it a name like airflow_task_errors.

Enter a log filter query that matches the pattern you care about:

resource.type="cloud_composer_environment"
severity="ERROR"
jsonPayload.message=~"Task .* failed"

This filter counts log entries from Composer environments with ERROR severity where the log message matches the pattern “Task … failed”.

Save the metric. It will start accumulating data immediately.

Once the metric exists, you can create an alert policy on it:

resource "google_monitoring_alert_policy" "log_based_task_errors" {
  display_name = "Cloud Composer - High Task Error Log Rate"
  combiner     = "OR"
  
  conditions {
    display_name = "Task error logs spiking"
    
    condition_threshold {
      filter = <<-EOT
        resource.type = "cloud_composer_environment"
        AND metric.type = "logging.googleapis.com/user/airflow_task_errors"
      EOT
      
      aggregations {
        alignment_period   = "600s"
        per_series_aligner = "ALIGN_RATE"
      }
      
      comparison      = "COMPARISON_GT"
      threshold_value = 5
      duration        = "0s"
    }
  }
  
  notification_channels = [
    google_monitoring_notification_channel.slack.name,
  ]
}

This alert fires if the rate of task error logs exceeds 5 per 10 minutes. Log-based metrics give you flexibility to monitor any pattern visible in logs without waiting for Google to expose it as a native metric.

Step 8: Implement Callback-Based Alerts from DAG Code

For more granular control, you can send alerts directly from your DAG code using Airflow callbacks. This lets you include specific error details, task context, and custom metadata in the notification.

Airflow supports on_failure_callback functions that execute when a task or DAG fails. You can use these callbacks to publish structured alert messages to Pub/Sub, send Slack notifications via webhook, or log to a custom observability platform.

Here is an example DAG that sends detailed failure notifications via Pub/Sub:

from airflow import DAG
from datetime import datetime, timedelta
from google.cloud import pubsub_v1
import json

PROJECT_ID = "your-project-id"
TOPIC_ID = "airflow-alerts"

def on_failure_callback(context):
    """Send a detailed failure notification via Pub/Sub."""
    dag_id = context['dag'].dag_id
    task_id = context['task_instance'].task_id
    logical_date = str(context['logical_date'])
    exception = str(context.get('exception', 'Unknown error'))
    
    message = {
        'severity': 'ERROR',
        'dag_id': dag_id,
        'task_id': task_id,
        'logical_date': logical_date,
        'error': exception,
        'log_url': context['task_instance'].log_url,
    }
    
    publisher = pubsub_v1.PublisherClient()
    topic_path = publisher.topic_path(PROJECT_ID, TOPIC_ID)
    future = publisher.publish(topic_path, json.dumps(message).encode('utf-8'))
    future.result()

default_args = {
    'owner': 'data-team',
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
    'on_failure_callback': on_failure_callback,
}

with DAG(
    'revenue_etl',
    default_args=default_args,
    schedule_interval='@daily',
    start_date=datetime(2026, 1, 1),
    catchup=False,
) as dag:
    # Your tasks here
    pass

The callback publishes a JSON message to a Pub/Sub topic. From there, you can subscribe to the topic and route alerts to Slack, PagerDuty, or any other system. The message includes the DAG ID, task ID, error details, and a direct link to the task log in the Airflow UI.

This approach gives you full control over alert content and routing. You can add business-specific context like customer impact, data volume processed, or SLA breach status that generic Cloud Monitoring alerts cannot include.

For teams using unified observability platforms, callback-based alerts can send structured events to a central system where they are correlated with infrastructure metrics, application traces, and Kubernetes pod events. This reduces the time to diagnose root causes compared to checking each signal separately.

Step 9: Centralize Alerts in a Unified Observability Platform

Cloud Monitoring alerts work well for basic DAG failure detection. But as your Airflow usage grows, you may find yourself managing dozens of alert policies across multiple Composer environments and struggling to correlate DAG failures with underlying infrastructure issues.

Unified observability platforms consolidate logs, metrics, traces, and infrastructure signals in one place. When a DAG fails, you can see the full context — the task that failed, the error message, the Kubernetes pod that ran the task, CPU and memory usage at the time of failure, and any related log entries from downstream services — all in a single dashboard.

CubeAPM is one such platform. It ingests Cloud Composer metrics from Cloud Monitoring, correlates them with Airflow task logs, and overlays them with Kubernetes pod metrics and GKE cluster health signals. When a DAG fails, you see the failure event in the context of node resource saturation, pod restarts, or database connection errors that may have caused it.

CubeAPM runs inside your own VPC or on premises, so your Airflow logs and metrics never leave your infrastructure. This matters for teams with data residency requirements or compliance mandates. Pricing is $0.15/GB for all ingested telemetry with unlimited retention, no per-seat fees, and no separate charges for logs, metrics, or traces.

For teams already using other observability tools, CubeAPM is compatible with OpenTelemetry, Prometheus, Datadog, and New Relic agents. You can migrate incrementally without changing your existing instrumentation.

To set up Cloud Composer monitoring in CubeAPM, configure a Cloud Monitoring integration that pulls composer.googleapis.com metrics into CubeAPM. Then configure Cloud Logging to forward Airflow logs to CubeAPM via a log sink. CubeAPM automatically correlates the metrics and logs by DAG ID, task ID, and execution date.

Once metrics and logs are flowing, create alert rules in CubeAPM that fire on DAG failures, task retry spikes, or environment health issues. Alerts can route to Slack, PagerDuty, email, or any webhook. CubeAPM alerts include full context — the failure event, related logs, infrastructure metrics at the time of failure — so responders have everything they need in one place.

Teams using CubeAPM for Cloud Composer monitoring report faster incident resolution because they no longer have to switch between Cloud Monitoring, Cloud Logging, GKE dashboards, and the Airflow UI to understand what failed and why.

Troubleshooting Common Issues

Alert fires but no failure shows in Airflow UI

Check the alert policy’s time window and alignment period. If the alignment period is too short, transient metric spikes may trigger false alerts. Increase the alignment period to 5 or 10 minutes and verify the alert condition matches the actual failure pattern.

Also check if the alert policy is filtering on the correct workflow_name or environment_name. A misconfigured filter may trigger on failures from a different Composer environment.

DAG failures not triggering alerts

Verify that the composer.googleapis.com/workflow/run_count metric is being reported for your environment. Navigate to Metrics Explorer in Cloud Monitoring and search for the metric. If it does not appear, check that your Composer environment is running and that the Cloud Monitoring API is enabled.

Also confirm that your alert policy’s notification channels are correctly configured and that notifications are not being blocked by spam filters or Slack channel settings.

Too many alerts from development DAGs

Use the workflow_name label to exclude development DAGs from critical alert policies. Create a separate alert policy for production DAGs using a regex filter like resource.labels.workflow_name = monitoring.regex.full_match("prod_.*") to match only DAGs with a production prefix.

Alternatively, deploy development DAGs to a separate Composer environment and exclude that environment from your production alert policies.

Alert fatigue from high task retry rates

If tasks retry frequently due to transient issues, task-level failure alerts may fire too often. Increase the failure count threshold or the time window to reduce noise. For example, only alert if more than 20 task failures occur in 2 hours instead of 10 failures in 1 hour.

You can also filter task failure alerts to specific high priority DAGs or critical task types that should not fail even once.

Log-based metrics not populating

Check that your log filter query matches actual log entries. Go to Logs Explorer and run the same filter query you used when creating the metric. If no logs match, the metric will remain at zero. Adjust the filter to match the log structure Composer actually produces.

Also verify that the Composer environment is generating logs. If the environment is not running any DAGs, no logs will appear.

Conclusion

Monitoring Cloud Composer DAG failures requires combining Cloud Monitoring alert policies, log-based metrics, Airflow callbacks, and optionally a unified observability platform that correlates logs, metrics, and infrastructure signals. Start with a basic alert on failed DAG runs, expand to task-level failure alerts and environment health checks, then add log-based metrics for custom failure patterns. For teams running large scale or business-critical Airflow deployments, a unified platform like CubeAPM reduces incident response time by giving full context in one place instead of requiring manual correlation across multiple tools.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions