Databricks Monitoring: Cluster Costs, DBU Consumption, and Job Performance

Databricks bills by Databricks Units (DBUs), not by time or data processed alone. A DBU is a normalized unit of compute capacity that varies by cluster type, runtime, and workload. A single job running on an All Purpose cluster with Photon enabled can consume DBUs at a rate 30% to 50% higher than the same job on a Jobs cluster without Photon, even if it completes faster. This means your Databricks bill is shaped by choices you make at cluster creation time, not just how much data you process or how long jobs run. Without visibility into DBU consumption patterns, cluster utilization, and job efficiency, Databricks costs can scale unpredictably as workloads grow.

This guide covers how to monitor Databricks cluster costs, track DBU consumption across workloads, and measure job performance using Databricks system tables, AI/BI dashboards, and third party observability platforms. It includes real queries, cost attribution methods, and tools that give you the visibility needed to control spend and optimize performance.

What Is Databricks Monitoring

Databricks monitoring is the practice of tracking the health, performance, and cost of Databricks workloads running on clusters. It covers three interconnected layers: cluster resource utilization (CPU, memory, disk), DBU consumption and billing patterns, and job execution metrics (runtime, failures, retries).

Unlike traditional infrastructure monitoring where you track VM uptime and CPU percent, Databricks monitoring requires tracking DBU consumption per cluster type, DBU rates per SKU, and how workload patterns affect both cost and job completion time. A cluster can be healthy in terms of CPU and memory but still drive excessive cost if it is configured with the wrong cluster type or runs idle longer than necessary.

Databricks provides system tables in the Unity Catalog that contain detailed billing, usage, and job execution data. These tables are queryable via SQL, making it possible to build custom dashboards, set alerts, and track cost drivers without relying on external log aggregation or APM tools for billing visibility. System tables include system.billing.usage for DBU consumption, system.lakeflow.jobs for job metadata, and system.compute.clusters for cluster lifecycle events.

Monitoring Databricks effectively means answering three questions continuously: Which clusters or jobs are consuming the most DBUs? Are jobs completing within expected time and cost thresholds? Where is usage growing faster than expected, and why?

How Databricks Billing Works: Understanding DBU Consumption

Databricks charges based on DBU consumption, not raw compute hours. A DBU represents a normalized unit of processing capability, and the number of DBUs consumed by a workload depends on the cluster type, instance size, runtime configuration, and whether Photon is enabled.

Each Databricks SKU has a DBU rate per instance hour. For example, an All Purpose cluster running on a Standard_DS3_v2 instance in Azure might consume 0.75 DBUs per hour, while the same instance running as a Jobs cluster consumes 0.07 DBUs per hour. The SKU rate is multiplied by the total runtime hours to calculate total DBU consumption. If a cluster runs for 10 hours, and its DBU rate is 0.75 DBU/hour, the total consumption is 7.5 DBUs for that period.

Photon increases the DBU consumption rate but can reduce job runtime significantly. A Photon enabled cluster consumes 30% to 50% more DBUs per hour than the same cluster without Photon, but if it completes the job 50% faster, the total DBU consumption for the job can be lower. This means Photon is cost effective for I/O heavy workloads like large table scans, aggregations, and joins, but not always cost effective for compute bound tasks where runtime improvement is minimal.

Serverless SQL warehouses and serverless compute for notebooks and jobs consume DBUs differently than classic clusters. Serverless workloads scale compute automatically and charge per query or per second of compute time. This eliminates idle cluster costs but can create unpredictable billing if queries run longer than expected or if multiple users trigger concurrent queries during peak periods.

Cost control in Databricks starts with understanding which cluster types you are using, what DBU rates apply, and whether your workload patterns justify the cluster type selected. For teams running scheduled ETL jobs, Jobs clusters are nearly always more cost effective than All Purpose clusters. For interactive workloads where multiple users share the same cluster, All Purpose clusters make sense, but they should be terminated when not in use to avoid idle DBU consumption.

Tracking Databricks Cluster Costs with System Tables

Databricks system tables provide detailed billing and usage data that can be queried directly in SQL. The primary table for cost monitoring is system.billing.usage, which records every billable event, including cluster runtime, job execution, SQL warehouse queries, and notebook compute.

Each row in system.billing.usage includes columns for usage_quantity (the number of DBUs consumed), sku_name (the specific SKU or cluster type), usage_start_time and usage_end_time, workspace_id, and usage_metadata (a struct containing details about the cluster, job, or pipeline that triggered the usage). This makes it possible to attribute costs to specific teams, projects, or workloads by filtering on workspace, cluster tags, or job IDs.

To see total DBU consumption by product for the current month, use this query:

SELECT 
  billing_origin_product, 
  usage_date, 
  SUM(usage_quantity) AS total_dbus
FROM system.billing.usage
WHERE 
  MONTH(usage_date) = MONTH(NOW()) 
  AND YEAR(usage_date) = YEAR(NOW())
GROUP BY billing_origin_product, usage_date
ORDER BY usage_date, total_dbus DESC;

This shows which Databricks products (Jobs, All Purpose Compute, SQL Warehouses, Serverless) are consuming the most DBUs, and how consumption trends over time within the month.

To identify which specific jobs consumed the most DBUs, filter by usage_metadata.job_id:

SELECT 
  usage_metadata.job_id AS job_id, 
  SUM(usage_quantity) AS total_dbus
FROM system.billing.usage
WHERE usage_metadata.job_id IS NOT NULL
GROUP BY job_id
ORDER BY total_dbus DESC;

This surfaces high cost jobs that may need optimization or should be moved to a more cost effective cluster type.

To attribute costs to specific teams or projects, use custom tags. Tags applied to clusters appear in the custom_tags column. For example, to see usage by a specific team:

SELECT 
  sku_name, 
  usage_unit, 
  SUM(usage_quantity) AS total_usage
FROM system.billing.usage
WHERE custom_tags['team'] = 'data_engineering'
GROUP BY sku_name, usage_unit;

This makes cost allocation straightforward for teams that apply consistent tagging to clusters and jobs. If no tags are applied, cost attribution requires joining system.billing.usage with system.compute.clusters or system.lakeflow.jobs to map usage back to cluster owners or job creators.

To see DBU consumption trends over time for a specific SKU (for example, All Purpose Compute with Photon):

SELECT 
  usage_date, 
  SUM(usage_quantity) AS dbus_consumed
FROM system.billing.usage
WHERE sku_name = 'ENTERPRISE_ALL_PURPOSE_COMPUTE_(PHOTON)'
  AND usage_date > '2026-01-01'
GROUP BY usage_date
ORDER BY usage_date ASC;

This helps detect unexpected growth in specific cluster types or product features.

Monitoring Databricks Job Performance with System Tables

Job performance monitoring in Databricks means tracking job runtime, success rates, failure reasons, retry counts, and task level execution breakdowns. The primary table for job monitoring is system.lakeflow.jobs, which contains metadata for every job including job name, owner, schedule, and last run status. Task level details are in system.lakeflow.job_task_run_timeline, which records the start and end time for each task within a job run.

To see which jobs are taking the longest to complete:

SELECT 
  job_id, 
  job_name, 
  AVG(run_duration) AS avg_runtime_seconds
FROM system.lakeflow.jobs
WHERE run_status = 'SUCCESS'
GROUP BY job_id, job_name
ORDER BY avg_runtime_seconds DESC
LIMIT 20;

This identifies jobs that may benefit from cluster resizing, query optimization, or splitting into smaller tasks.

To see which jobs have the highest failure rates:

SELECT 
  job_id, 
  job_name, 
  COUNT(*) AS total_runs,
  SUM(CASE WHEN run_status = 'FAILED' THEN 1 ELSE 0 END) AS failed_runs,
  (SUM(CASE WHEN run_status = 'FAILED' THEN 1 ELSE 0 END) / COUNT(*)) * 100 AS failure_rate_pct
FROM system.lakeflow.jobs
WHERE run_start_time > CURRENT_DATE - INTERVAL 30 DAYS
GROUP BY job_id, job_name
HAVING failed_runs > 0
ORDER BY failure_rate_pct DESC;

This surfaces jobs with reliability problems that need attention.

To see jobs with the highest retry counts:

SELECT 
  job_id, 
  job_name, 
  SUM(retry_count) AS total_retries
FROM system.lakeflow.jobs
WHERE run_start_time > CURRENT_DATE - INTERVAL 30 DAYS
GROUP BY job_id, job_name
ORDER BY total_retries DESC
LIMIT 20;

High retry counts indicate transient failures, resource contention, or cluster startup delays that are extending job completion time unnecessarily.

To track job cost and performance together, join system.billing.usage with system.lakeflow.jobs:

WITH job_costs AS (
  SELECT 
    usage_metadata.job_id AS job_id,
    SUM(usage_quantity) AS total_dbus
  FROM system.billing.usage
  WHERE usage_metadata.job_id IS NOT NULL
  GROUP BY job_id
)
SELECT 
  j.job_id, 
  j.job_name, 
  AVG(j.run_duration) AS avg_runtime_seconds, 
  c.total_dbus
FROM system.lakeflow.jobs j
JOIN job_costs c ON j.job_id = c.job_id
WHERE j.run_status = 'SUCCESS'
GROUP BY j.job_id, j.job_name, c.total_dbus
ORDER BY c.total_dbus DESC;

This shows which jobs consume the most DBUs and how long they take to complete, making it easier to prioritize optimization work.

Using AI/BI Dashboards for Databricks Cost and Usage Monitoring

Databricks recommends using AI/BI dashboards to operationalize billing and usage data from system tables. AI/BI dashboards are interactive, SQL powered dashboards that can be shared across teams and updated in real time as new data lands in system tables.

Account admins can import a pre built cost monitoring dashboard provided by Databricks. This dashboard includes tiles for total DBU consumption by product, daily DBU trends, top jobs by cost, and cluster utilization breakdowns. It provides a starting point for cost visibility without requiring custom query development.

To create a custom dashboard, start by defining the questions you need to answer. For example: What is the daily DBU consumption trend for the last 90 days? Which workspaces or teams are driving the most cost? Are there clusters running idle for more than 2 hours? Which jobs had the largest increase in cost over the last 7 days compared to the previous 7 days?

Each question becomes a query against system.billing.usage, system.lakeflow.jobs, or system.compute.clusters. The results are visualized as line charts for trends, bar charts for rankings, or tables for detailed breakdowns.

For example, to track jobs with increasing costs week over week:

WITH last_week AS (
  SELECT 
    usage_metadata.job_id AS job_id,
    SUM(usage_quantity) AS dbus_last_week
  FROM system.billing.usage
  WHERE usage_date BETWEEN CURRENT_DATE - INTERVAL 14 DAYS AND CURRENT_DATE - INTERVAL 7 DAYS
  GROUP BY job_id
),
this_week AS (
  SELECT 
    usage_metadata.job_id AS job_id,
    SUM(usage_quantity) AS dbus_this_week
  FROM system.billing.usage
  WHERE usage_date >= CURRENT_DATE - INTERVAL 7 DAYS
  GROUP BY job_id
)
SELECT 
  t.job_id, 
  l.dbus_last_week, 
  t.dbus_this_week,
  ((t.dbus_this_week - l.dbus_last_week) / l.dbus_last_week) * 100 AS pct_change
FROM this_week t
JOIN last_week l ON t.job_id = l.job_id
WHERE l.dbus_last_week > 0
ORDER BY pct_change DESC
LIMIT 20;

This identifies jobs where cost is growing unexpectedly, signaling potential issues with query efficiency, data volume growth, or cluster configuration changes.

Alerts can be added to queries to notify teams when usage exceeds thresholds. For example, an alert can fire when total DBU consumption for a workspace exceeds 10,000 DBUs in a single day, or when a specific job consumes more than 500 DBUs in a single run.

Monitoring Databricks Clusters with CubeAPM

CubeAPM provides unified observability for Databricks workloads, covering cluster health, DBU consumption, job performance, and application traces in a single platform. It runs on premises or inside your own VPC, so telemetry data from Databricks clusters never leaves your infrastructure. This makes it suitable for teams with data residency requirements or compliance constraints that rule out cloud only SaaS monitoring tools.

CubeAPM integrates with Databricks via OpenTelemetry. Databricks supports exporting logs, metrics, and traces to OpenTelemetry compatible backends. Once configured, CubeAPM receives cluster metrics (CPU, memory, disk I/O), Spark application metrics (task duration, shuffle read/write, executor memory), and job execution traces that show the full path of a Databricks job from notebook execution through Spark driver, executors, and data source interactions.

CubeAPM surfaces DBU consumption patterns by correlating billing data from system.billing.usage with cluster and job metadata. This gives you a unified view of which clusters are consuming the most DBUs, which jobs are driving cost, and whether resource utilization is aligned with DBU spend. For example, a cluster consuming 1,000 DBUs over 8 hours but averaging 30% CPU utilization indicates overprovisioning or idle time that could be eliminated by right sizing the cluster or reducing auto termination thresholds.

CubeAPM tracks job performance by ingesting Spark event logs and Databricks job run metadata. It visualizes job runtime, task breakdowns, shuffle volume, and failure reasons in real time. When a job fails or exceeds its expected runtime, CubeAPM surfaces the exact task that took longest, the executor that processed it, and the resource bottleneck (CPU, memory, or I/O) that caused the delay. This eliminates the need to manually parse Spark UI or cluster logs to diagnose job performance issues.

CubeAPM pricing is $0.15/GB of telemetry ingested, with unlimited data retention and no per user or per host fees. For a team ingesting 5TB of Databricks logs, metrics, and traces per month, the monthly cost is $750. This is predictable and does not scale with cluster count or user count, making it easier to forecast monitoring costs as Databricks usage grows.

CubeAPM runs as a self hosted deployment managed by the CubeAPM team. This means your team does not need to provision, upgrade, or maintain the monitoring infrastructure, but the platform runs inside your environment with no external telemetry export. This hybrid model gives you the operational simplicity of SaaS with the data control of self hosting.

Best Practices for Databricks Cost and Performance Monitoring

Use Jobs clusters for scheduled workloads. Jobs clusters consume fewer DBUs per hour than All Purpose clusters and terminate automatically after the job completes. For teams running nightly ETL pipelines or scheduled data processing, switching from All Purpose to Jobs clusters can reduce DBU consumption by 80% or more for the same workload.

Set auto termination for All Purpose clusters. If a cluster sits idle for more than 30 minutes, it should terminate automatically. Idle clusters still consume DBUs, and long running idle clusters are one of the most common sources of unnecessary Databricks spend. Configure auto termination at 15 to 30 minutes for interactive clusters to balance cost and user experience.

Tag every cluster and job. Apply consistent tags for team, project, environment, and cost center to every cluster and job. This makes cost attribution straightforward when querying system.billing.usage. Without tags, cost allocation requires manual mapping of cluster IDs to teams, which breaks down as teams scale.

Monitor DBU consumption daily. Set up a dashboard or scheduled query that tracks total DBU consumption by day, by product, and by team. Review it weekly to catch unexpected growth early. A 20% increase in DBU consumption over one week can indicate a misconfigured job, a data volume spike, or a new workload that was not planned for.

Review high cost jobs monthly. Query system.billing.usage to identify the top 20 jobs by DBU consumption each month. For each high cost job, ask: Is this job running on the right cluster type? Can query logic be optimized to reduce runtime? Can data be partitioned or pre filtered to reduce shuffle volume?

Use Photon selectively. Photon improves performance for I/O heavy workloads but increases DBU consumption rates. Test Photon on representative workloads to measure whether the runtime reduction justifies the higher DBU rate. For compute bound workloads with minimal I/O, Photon often increases cost without meaningful performance gains.

Set budget alerts. Use Databricks budget alerts or query based alerts in your monitoring platform to notify teams when DBU consumption exceeds thresholds. For example, alert when total monthly DBU consumption exceeds 80% of budget with 7 days remaining in the month. This gives teams time to investigate and adjust before costs exceed budget.

Conclusion

Databricks monitoring requires tracking three interconnected layers: cluster resource utilization, DBU consumption patterns, and job execution performance. Without visibility into how clusters consume DBUs and which jobs drive cost, Databricks bills scale unpredictably as workloads grow. System tables in Unity Catalog provide the raw data needed to build cost dashboards, attribute spend to teams, and identify optimization opportunities. AI/BI dashboards operationalize this data for teams that need real time visibility without building custom tooling. For teams that want unified observability across Databricks and the rest of their infrastructure, platforms like infrastructure monitoring tools integrate Databricks telemetry with logs, traces, and metrics from other systems.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions