Google Cloud Dataflow processes billions of records daily across streaming and batch workloads, but without proper monitoring, a single straggler worker or data freshness spike can silently degrade your entire pipeline for hours. Research from the DORA 2021 State of DevOps Report found that elite performers with mature observability practices are 4.1 times more likely to meet or exceed their reliability targets compared to those relying on default dashboards alone.
This guide covers how to monitor Dataflow pipelines using Cloud Monitoring, how to implement custom metrics, how to set up alerts that catch real problems, and what third-party tools can fill gaps that native monitoring leaves open.
What Is Google Cloud Dataflow Monitoring?
Google Cloud Dataflow monitoring is the practice of tracking pipeline health, performance, and resource usage across streaming and batch jobs using metrics, logs, and diagnostic tools. Dataflow integrates natively with Cloud Monitoring (formerly Stackdriver) to expose job-level metrics like throughput, system lag, worker count, and element counts without requiring custom instrumentation.
Dataflow monitoring matters because pipelines operate as distributed systems where failures can cascade silently. A blocked stage, a slow external API call, or a memory leak in a DoFn can throttle an entire job without throwing visible errors. Monitoring surfaces these issues before they violate SLAs or accumulate data backlog that takes hours to clear.
Dataflow runs on Google Cloud infrastructure and scales workers dynamically based on workload. This autoscaling behavior makes cost monitoring critical: a pipeline consuming 50 vCPUs one minute can scale to 200 vCPUs the next if a traffic spike hits, and without visibility into resource usage, your bill can triple before you notice.
How Google Cloud Dataflow Monitoring Works
Dataflow monitoring operates through three layers: job metrics, worker metrics, and application logs. Each layer provides a different view into pipeline behavior.
Job-level metrics: Dataflow publishes job-level metrics to Cloud Monitoring every 30 seconds. These cover job status (running, failed, succeeded), elapsed time, system lag for streaming jobs, element counts per PCollection, and resource usage like vCPU count and persistent disk consumption. System lag is the most critical streaming metric: it measures the maximum amount of time an item of data has been processing or awaiting processing. A system lag spike from 5 seconds to 120 seconds signals that your pipeline cannot keep up with input rate, usually caused by slow external calls, stragglers, or insufficient worker capacity.
Worker-level metrics: Worker metrics track individual VM performance including CPU utilization, memory usage, disk I/O, and network throughput. These metrics are not enabled by default. To collect worker VM metrics, launch your pipeline with the –experiments=enable_stackdriver_agent_metrics flag, which installs the Monitoring agent on each worker. Worker metrics help diagnose resource bottlenecks that job-level metrics miss. If CPU is pegged at 100% across all workers, you need more parallelism or faster machines. If memory is spiking, you may have a leak in a stateful DoFn.
Application logs: Dataflow writes operational logs to Cloud Logging under the service name dataflow.googleapis.com. These logs capture worker startup events, autoscaling decisions, DoFn exceptions, and pipeline errors. Logs are stored in the _Default log bucket with a default retention of 30 days. Metrics tell you that a job failed; logs tell you why. A NullPointerException in a custom DoFn, a timeout calling an external API, or a permissions error reading from BigQuery will all appear in logs before surfacing as metric anomalies.
Core Dataflow Metrics Every Pipeline Should Track
Dataflow exposes dozens of metrics, but five categories matter most for operational monitoring: job status, throughput, latency, resource consumption, and errors.
Job status and elapsed time
Job status reports whether a pipeline is running, succeeded, or failed. This metric updates every 30 seconds and on job completion. The Failed metric explicitly sets to 1 when a job exits with failure, making it straightforward to create alerts.
Elapsed time tracks how long a job has been running in seconds. For batch jobs, this helps detect stuck pipelines. A job that normally completes in 10 minutes but has been running for 45 minutes likely hit a straggler or data skew issue.
System lag and data freshness
System lag (job/system_lag) measures the maximum amount of time an item of data has been processing or awaiting processing, in seconds. This is a streaming-only metric. A sustained spike indicates the pipeline cannot keep up with the input rate.
Data freshness (job/data_freshness) measures the difference between the current processing time and the event timestamp of the oldest unprocessed element. High data freshness means the pipeline is falling behind on event-time data, which delays windowing operations and can cause late data to be dropped if it exceeds your allowed lateness setting. Both metrics are critical for streaming pipelines where freshness SLAs matter. A payment processing pipeline with a 10-second system lag SLA should alert when lag exceeds 15 seconds for more than 1 minute.
Element count and estimated byte count
Element count (job/element_count) tracks the number of elements added to each PCollection. This is a per-PCollection metric, not a job-level aggregate, so you need to filter by pcollection label when querying it.
Estimated byte count measures the volume of data processed in bytes. Together, these metrics help identify throughput bottlenecks. If elements are piling up in one PCollection but draining slowly from the next, the connecting transform is your bottleneck.
Resource usage metrics
Current vCPU count shows the number of virtual CPUs currently allocated to the job, updated when workers scale up or down.
Total vCPU usage tracks cumulative vCPU-seconds consumed by the job, useful for cost tracking.
Total Persistent Disk usage reports cumulative disk usage in GB-seconds, reported separately for SSD and HDD using a metric label. These metrics help monitor autoscaling behavior and cost. A streaming job that normally runs on 20 vCPUs but suddenly spikes to 80 vCPUs may be reacting to a traffic surge or an inefficient transform.
Custom metrics from Apache Beam
Any metric you define in your Apache Beam pipeline using the Metrics API is reported by Dataflow to Cloud Monitoring as a custom metric. Beam supports three metric types: Counter, Distribution, and Gauge. Dataflow reports Counter and Distribution to Cloud Monitoring. Distribution is reported as four submetrics suffixed _MAX, _MIN, _MEAN, and _COUNT.
Custom metrics appear in Cloud Monitoring as dataflow.googleapis.com/job/user_counter with metric_name and ptransform labels. For backward compatibility, they are also published as custom.googleapis.com/dataflow/metric-name.
Each project has a limit of 100 Dataflow custom metrics. Exceeding this limit causes new metrics to be dropped. Custom metrics incur Cloud Monitoring charges based on time series ingested.
Dataflow Monitoring Dashboard and Metrics Explorer
Cloud Monitoring provides two interfaces for viewing Dataflow metrics: the project-level dashboard and Metrics Explorer.
Project monitoring dashboard
The Dataflow project dashboard shows job-level metrics aggregated across all pipelines in a project. Access it by navigating to Dataflow, then Monitoring in the Google Cloud console. The dashboard includes time-series charts for running jobs count, workers per job (top 25), total vCPU count across all jobs, quota exceeded errors, average system latency for streaming jobs, system lag (top 25 streaming jobs), data freshness per stage (top 25), Streaming Engine Compute Units usage (top 25), user processing latencies (top 25), and max backlog bytes (top 25).
The dashboard helps detect quota errors, autoscaling anomalies, and slow streaming jobs at a glance. To filter which jobs appear, use regular expressions in the dashboard filter.
Metrics Explorer
Metrics Explorer lets you query and chart any Dataflow metric with full label filtering. Navigate to Monitoring, then Metrics Explorer in the Cloud console. Enter “Dataflow Job” in the metric selector, choose a metric category (Job, System, or User Counter), select a specific metric, and add filters by job ID, PCollection, or other labels.
Metrics Explorer supports PromQL queries for advanced use cases. For example, to calculate the delta of a custom counter over a time window:
sum by (ptransform, metric_name) (
delta({
"__name__"="dataflow.googleapis.com/job/user_counter",
"monitored_resource"="dataflow_job",
"job_id"="[JobID]"
}[${__interval}])
)Setting Up Alerts for Dataflow Pipelines
Cloud Monitoring alerting policies notify you when metrics cross thresholds. Alerts are essential for catching system lag spikes, job failures, and quota errors before they impact users.
Note: Starting no sooner than September 1, 2026, Cloud Monitoring will begin charging for alerting at $0.35 per month per metric reference in an alerting policy. Budget accordingly when creating alert rules at scale.
Alert on job failures
Navigate to Monitoring, then Alerting in the Cloud console. Click Create Policy, select the “Failed” metric under the Dataflow Job category, set the condition to fire when the metric equals 1, configure notification channels (email, Slack, PagerDuty), and save. This alert fires whenever any Dataflow job in the project exits with a failure. To scope alerts to specific jobs, add a filter on job_name or job_id.
Alert on streaming system lag
Create a new alerting policy, select the “System lag” metric, set the threshold condition to metric value > 15 seconds, set the duration to “violates for 1 minute,” add notification channels, and save. This configuration filters out transient spikes during traffic bursts, firing only when lag is sustained.
Alert on quota exceeded errors
Create a new alerting policy, select “Quota exceeded errors,” set the condition to any time series > 0, add notification channels, and save. When this alert fires, check your Compute Engine quotas under IAM & Admin, then Quotas in the Cloud console.
Common Dataflow Monitoring Issues and How to Fix Them
Stragglers causing high system lag
A straggler is a work item that takes significantly longer to process than others, holding back the watermark and adding latency to the job. Common causes include data skew, hot keys, or slow external calls.
To diagnose stragglers, check the Dataflow job graph in the Cloud console for stages with uneven element distribution, review worker metrics to find workers with high CPU or memory, and examine logs for repeated exceptions or timeouts. Fix stragglers by using a better grouping key to distribute load more evenly, or by increasing worker machine type with –workerMachineType.
High data freshness
High data freshness indicates the pipeline is falling behind on event-time data, which delays windowing operations and can cause late data to be dropped if it exceeds the pipeline’s allowed lateness setting.
Common causes include insufficient worker capacity (increase –maxNumWorkers), slow external API calls (add caching or use async I/O), and large windows with high cardinality (switch to session windows or reduce key space).
Missing worker VM metrics
If worker-level CPU and memory metrics are not appearing in Cloud Monitoring, the Monitoring agent is likely not enabled. Update your pipeline with –experiments=enable_stackdriver_agent_metrics. The worker service account must also have the roles/monitoring.metricWriter IAM role; without it, the agent cannot write metrics to Cloud Monitoring.
Tools and Implementation
Several tools extend Dataflow monitoring beyond Cloud Monitoring’s native capabilities, including third-party observability platforms that provide unified views across Dataflow, application services, and other infrastructure.
Native Cloud Monitoring
Cloud Monitoring is included with Google Cloud at no additional cost for default platform metrics. Custom metrics and log storage beyond the free tier incur charges based on time series ingested and log volume stored. Starting no sooner than September 1, 2026, alerting will carry a charge of $0.35/month per metric reference.
Cloud Monitoring works best for teams fully invested in Google Cloud who want native integration with Dataflow, BigQuery, Pub/Sub, and other GCP services. Its main limitations are the absence of cross-cloud visibility for workloads running on AWS or Azure alongside GCP, and no native distributed tracing correlation for application-layer context.
Dynatrace for Dataflow
Dynatrace connects to Google Cloud through its GCP integration, which pulls service metrics from the Google Cloud APIs, including Dataflow job health metrics. It surfaces these alongside application traces, infrastructure metrics, and user sessions in a single interface, with Davis AI providing automated root cause analysis when pipeline anomalies correlate with upstream or downstream service behavior.
Dynatrace operates under the Dynatrace Platform Subscription (DPS) model. Log Analytics ingestion is $0.20/GiB. Full-Stack Monitoring is $58/month per 8 GiB host (billed at $0.01/GiB-hour), and Infrastructure Monitoring is $29/month per host. All pricing sourced from dynatrace.com/pricing. Dynatrace suits enterprises running complex multi-cloud environments where correlating Dataflow pipeline health with downstream services is a priority. For log-heavy Dataflow environments, the $0.20/GiB log ingest rate compounds quickly at scale.
CubeAPM for unified observability
CubeAPM is a self-hosted, OpenTelemetry-native observability platform covering APM, logs, infrastructure, and Kubernetes monitoring with predictable $0.15/GB pricing and unlimited retention. CubeAPM supports OpenTelemetry-based instrumentation for custom metrics and can ingest Dataflow logs via Fluentd or the Cloud Logging export pipeline, making it a practical option for teams that want to unify Dataflow telemetry with application and infrastructure signals in a single platform.
Teams using CubeAPM benefit from keeping all telemetry data inside their own cloud or on-premises environment, eliminating data egress costs and satisfying data residency requirements. CubeAPM runs as a vendor-managed self-hosted platform, meaning your team does not handle upgrades or patches. For Dataflow-specific monitoring, export custom metrics from your Apache Beam pipeline using the Metrics API and forward them to CubeAPM via OpenTelemetry. Worker VM metrics can be collected using the Monitoring agent and shipped to CubeAPM’s infrastructure monitoring module.
For teams running Dataflow alongside application services and Kubernetes workloads, CubeAPM provides a single unified view that Cloud Monitoring cannot offer across non-GCP infrastructure.
Dataflow Monitoring Best Practices
Enable worker VM metrics from day one: Do not wait until a production incident to enable worker-level metrics. Launch every pipeline with –experiments=enable_stackdriver_agent_metrics so you have CPU, memory, and disk data when you need it. The worker service account must have roles/monitoring.metricWriter for metrics to reach Cloud Monitoring.
Use custom metrics for business logic: Job-level metrics track Dataflow internals but do not tell you if your business logic is healthy. Instrument your DoFns with custom counters using the Beam Metrics API:
Counter recordsProcessed = Metrics.counter("MyDoFn", "records_processed");
recordsProcessed.inc();These metrics appear in Cloud Monitoring as dataflow.googleapis.com/job/user_counter and help correlate pipeline performance with business outcomes.
- Set alerts with context, not just thresholds: A 1-second system lag spike during a traffic burst is normal. A 30-second spike sustained for 5 minutes is not. Use alerting policy duration conditions to filter transient noise, and scope alerts to specific job names where possible to avoid alert fatigue across large projects.
- Monitor Dataflow snapshots for recovery: Dataflow snapshots capture pipeline state for recovery after failures. Monitor snapshot creation success and storage usage to ensure snapshots are available when needed. Snapshot metrics are not exposed in Cloud Monitoring by default; track them using custom logs or the Dataflow REST API.
- Track cost metrics alongside performance: Use “Total vCPU usage” and “Total Persistent Disk usage” to monitor pipeline cost in near real time. Correlate cost spikes with throughput changes to identify inefficient transforms that consume resources without improving output.
- Correlate Dataflow metrics with source and sink health: Dataflow does not operate in isolation. If your pipeline reads from Pub/Sub and writes to BigQuery, monitor Pub/Sub subscription backlog and BigQuery streaming insert errors alongside Dataflow metrics. A BigQuery quota error will cause Dataflow system lag to spike even if the pipeline itself is healthy.
Streaming vs Batch Pipeline Monitoring
Streaming pipeline monitoring
Streaming jobs run indefinitely and process unbounded data. The critical metrics are system lag, data freshness, backlog size, and throughput stability. A healthy streaming pipeline maintains consistent system lag under varying input rates and scales workers dynamically without manual intervention.
Monitor autoscaling behavior by tracking “Current vCPU count” over time. If workers scale up but system lag does not decrease, you have a bottleneck that more parallelism cannot fix. This typically indicates a slow external call or a hot key that needs to be addressed at the application level.
Batch pipeline monitoring
Batch jobs process bounded datasets and terminate after completion. The critical metrics are elapsed time, job status, and element count per stage. A batch job that normally completes in 15 minutes but takes 60 minutes likely hit a straggler or data skew problem.
Monitor batch job SLAs by setting alerts on elapsed time thresholds. If a job runs longer than expected, check the job graph for stages with uneven element distribution and review logs for repeated retries or exceptions.
Conclusion
Monitoring Google Cloud Dataflow pipelines requires combining Cloud Monitoring’s native metrics with worker-level diagnostics, custom application metrics, and context-aware alerts. System lag, data freshness, and resource usage form the foundation of streaming pipeline health, while elapsed time and element distribution matter most for batch jobs.
Enable worker VM metrics at pipeline launch, instrument custom counters for business logic, and correlate Dataflow metrics with your source and sink health to resolve incidents faster. Ready to unify Dataflow observability with your broader application and infrastructure stack? Explore CubeAPM’s documentation to get started.
Disclaimer: Pricing data was sourced from official vendor websites and documentation as of June 2026. Vendor pricing changes frequently; verify all figures directly with each vendor before making purchasing decisions. CubeAPM is the platform behind this blog.
FAQs
What is the difference between system lag and data freshness in Dataflow?
System lag measures the maximum time an item of data has been processing or awaiting processing. Data freshness measures the difference between the current processing time and the event timestamp of the oldest unprocessed element. System lag reflects end-to-end pipeline latency; data freshness indicates how far behind the pipeline is on event-time data.
How do I enable worker VM metrics for a Dataflow pipeline?
Add –experiments=enable_stackdriver_agent_metrics when launching your pipeline, and ensure the worker service account has the roles/monitoring.metricWriter IAM role. Worker metrics then appear in Cloud Monitoring under the agent metrics namespace.
What are Dataflow custom metrics and how are they priced?
Custom metrics are defined in your Apache Beam pipeline using the Metrics API and appear in Cloud Monitoring as dataflow.googleapis.com/job/user_counter. Each project is limited to 100 Dataflow custom metrics. They incur Cloud Monitoring charges based on time series ingested beyond the free tier.
How do I alert on Dataflow job failures?
Create a Cloud Monitoring alerting policy on the “Failed” metric under the Dataflow Job category. Set the condition to fire when the metric equals 1 and configure your notification channels. Add a filter on job_name to scope the alert to specific pipelines.
What causes high system lag in streaming Dataflow pipelines?
High system lag typically means the pipeline cannot keep up with the input rate. Common causes are stragglers from data skew or hot keys, slow external API calls, insufficient worker capacity, or bottlenecks in stateful transforms like GroupByKey.
Can I monitor Dataflow pipelines with third-party tools?
Yes. Platforms like Dynatrace and CubeAPM can ingest Dataflow metrics via the Cloud Monitoring API or custom OpenTelemetry exporters. Custom Beam metrics can be forwarded to any OTel-compatible backend.
How long does Dataflow retain job metrics and logs?
Cloud Monitoring retains job information for 30 days after a job completes or is cancelled. Operational logs in Cloud Logging default to 30-day retention in the _Default log bucket. Configure custom log buckets with extended retention periods for compliance or long-term trend analysis.