How to Monitor CircleCI Build Performance and Failures: Step by Step Guide 2026

CircleCI is the CI/CD backbone for thousands of engineering teams, but a pipeline that degrades from 8 minutes to 22 minutes over three months can quietly erode developer velocity before anyone notices. According to the 2025 DORA State of DevOps report, elite performing teams deploy 973 times more frequently than low performers and fast, reliable CI is a foundational requirement. Without visibility into build duration trends, queue times, and failure patterns, teams spend more time debugging pipelines than shipping features.

This guide covers how to monitor CircleCI build performance and failures using native CircleCI Insights, external observability platforms, and custom instrumentation. You will learn how to track the metrics that matter, set up alerts for regressions, and identify bottlenecks that slow deployments.

Prerequisites

Before setting up CircleCI monitoring, ensure you have:

• Active CircleCI account with at least one configured project
• Admin or project-level permissions to access CircleCI Insights dashboard and API
• API token for CircleCI v2 API (generate from User Settings > Personal API Tokens)
• Access to your team’s observability platform (Datadog, CubeAPM, Grafana, or similar) if using external monitoring
• Basic familiarity with YAML configuration for CircleCI pipelines
• Understanding of your pipeline’s expected baseline performance (median build time, typical queue duration)

Step 1: Enable and Use CircleCI Insights Dashboard

CircleCI provides a built-in Insights dashboard that surfaces performance trends, success rates, and duration breakdowns across all projects.

Navigate to your CircleCI dashboard and select Insights from the left sidebar. Choose the project you want to monitor. The Insights view displays:

• Workflow duration trends over the last 90 days
• Success rate percentages for workflows and jobs
• Queue time (time from workflow trigger to first job start)
• Median and p95 build duration for each workflow

Use the workflow selector dropdown to compare performance across different branches. For example, filtering by your main branch surfaces production-impacting trends, while filtering by feature branches reveals developer-specific bottlenecks.

Insights data updates every 10 minutes. If a workflow that normally completes in 6 minutes suddenly takes 14 minutes, the duration chart surfaces the spike immediately.

What to look for: Sudden increases in median duration, declining success rates below 95%, or queue times exceeding 2 minutes during normal traffic periods. These indicate resource contention, flaky tests, or configuration issues.

Step 2: Set Up Datadog CircleCI Integration (SaaS Option)

Teams already using Datadog can ingest CircleCI webhook events directly into their observability stack.

In CircleCI, navigate to Project Settings > Webhooks and click Add Webhook. Set the webhook URL to your Datadog intake endpoint (provided in Datadog’s CircleCI integration documentation). Select the events to forward:

• workflow-completed
• job-completed

In Datadog, enable the CircleCI integration from the Integrations page and configure your API key. Within 5 minutes, CircleCI events appear in Datadog as pipeline metrics.

Create a dashboard in Datadog with the following widgets:

• Average workflow duration by project (metric: circleci.workflow.duration)
• Job failure rate as a percentage (metric: circleci.job.failed)
• Queue time trends (metric: circleci.workflow.queued_duration)

Set up monitors in Datadog to alert when workflow duration exceeds baseline by 50% for three consecutive runs, or when job failure rate for a project crosses 10%.

Cost consideration: Datadog’s APM pricing starts at $31/host/month. For a team monitoring 50 CircleCI executors across projects, this can exceed $1,500/month before adding logs or custom metrics.

Step 3: Monitor CircleCI with CubeAPM (Self Hosted Option)

CubeAPM runs inside your VPC and ingests CircleCI telemetry via OpenTelemetry or direct API polling. This keeps all CI/CD performance data local and avoids SaaS egress fees.

Install the CubeAPM OpenTelemetry collector in your infrastructure. Configure it to poll the CircleCI v2 API every 60 seconds:

receivers:
  circleciapi:
    endpoint: https://circleci.com/api/v2
    auth:
      api_key: ${CIRCLECI_API_TOKEN}
    poll_interval: 60s
    projects:
      - org/repo-name
exporters:
  cubeapm:
    endpoint: http://cubeapm-collector:4317
service:
  pipelines:
    metrics:
      receivers: [circleciapi]
      exporters: [cubeapm]

In CubeAPM, create a dashboard with:

• Workflow duration by project and branch
• Job-level failure breakdown showing which jobs fail most frequently
• Queue time heatmap by hour of day

Set alert rules in CubeAPM:

• Workflow duration for main branch exceeds 15 minutes for 3 consecutive runs
• Job failure rate for any job in production pipelines exceeds 8%
• Queue time exceeds 5 minutes during business hours

Alerts route to Slack or PagerDuty with full context including the failing workflow URL and recent commit SHAs.

Cost advantage: CubeAPM charges $0.15/GB ingested. For a team ingesting 2 TB/month of CI telemetry, total cost is $300/month with unlimited retention. No per-host fees, no seat licenses.

Step 4: Track Build Performance with CircleCI API

For teams building custom monitoring dashboards or integrating CI metrics into existing infrastructure monitoring platforms, the CircleCI v2 API provides programmatic access to all pipeline and job data.

Fetch recent workflow runs for a project:

curl -X GET \
  "https://circleci.com/api/v2/insights/gh/your-org/your-repo/workflows/workflow-name?branch=main" \
  -H "Circle-Token: ${CIRCLECI_API_TOKEN}"

The response includes:

• duration (total workflow runtime in seconds)
• status (success, failed, canceled)
• created_at and stopped_at timestamps
• credits_used (CircleCI compute credit consumption)

Parse this data and push it to your observability backend. For example, send workflow duration as a metric to Prometheus:

import requests
from prometheus_client import Gauge, push_to_gateway
workflow_duration = Gauge('circleci_workflow_duration_seconds', 
                          'Workflow duration', 
                          ['project', 'workflow', 'branch'])
response = requests.get(
    "https://circleci.com/api/v2/insights/gh/org/repo/workflows/build/jobs",
    headers={"Circle-Token": api_token}
)
for item in response.json()['items']:
    workflow_duration.labels(
        project='repo',
        workflow='build',
        branch='main'
    ).set(item['duration'] / 1000)
push_to_gateway('prometheus:9091', job='circleci_exporter', 
                registry=workflow_duration)

Run this script every 5 minutes via cron or a Kubernetes CronJob.

What to track: Workflow duration, job duration, success rate, queue time, and credit consumption. These five metrics surface 90% of CI performance issues.

Step 5: Instrument CircleCI Pipelines with Custom Metrics

Beyond external monitoring, instrumenting your CircleCI configuration itself surfaces job-level bottlenecks that aggregate metrics miss.

Add timing instrumentation to long-running jobs:

version: 2.1
jobs:
  integration-tests:
    docker:
      - image: cimg/node:18.0
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: |
            start_time=$(date +%s)
            npm ci
            end_time=$(date +%s)
            echo "npm_install_duration=$((end_time - start_time))s"
      - run:
          name: Run integration tests
          command: |
            start_time=$(date +%s)
            npm run test:integration
            end_time=$(date +%s)
            duration=$((end_time - start_time))
            echo "test_duration=${duration}s"
            if [ $duration -gt 600 ]; then
              echo "WARNING: Integration tests exceeded 10 minutes"
            fi

For teams using frontend performance monitoring tools, emit custom metrics to your observability platform directly from the pipeline:

curl -X POST https://your-observability-platform/api/metrics \
  -H "Authorization: Bearer ${APM_TOKEN}" \
  -d '{
    "metric": "circleci.job.duration",
    "value": '$duration',
    "tags": ["job:integration-tests", "branch:'$CIRCLE_BRANCH'"]
  }'

This approach captures timing data that CircleCI Insights does not expose by default such as time spent in specific test suites or dependency installation steps.

Step 6: Set Up Alerts for Build Failures and Performance Regressions

Monitoring without alerting is passive. Alerts ensure your team responds to issues before they compound.

Define alert thresholds based on your team’s baseline:

• Workflow duration regression: Alert when main branch workflows exceed the 90-day median by 40% for 3 consecutive runs
• Job failure rate: Alert when any job in production workflows fails more than 10% of the time in a 6-hour window
• Queue time spike: Alert when queue time exceeds 8 minutes during business hours

In CircleCI, enable email notifications for failed workflows under Project Settings > Advanced > Email Notifications. This is baseline protection but lacks context.

For richer alerts, configure your observability platform:

In Datadog: Create a monitor with the condition avg(last_5m):avg:circleci.workflow.duration{workflow:deploy-production} by {project} > 900 and route to Slack channel #eng-ci-alerts.

In CubeAPM: Use the alert builder to trigger when circleci_workflow_duration_seconds{branch="main"} exceeds 15 minutes for 3 runs and send to PagerDuty with workflow URL attached.

In Grafana: Set up an alert rule on your CircleCI dashboard panel tracking job failure rate. Use Grafana’s Alerting feature to route to a webhook or email when failure rate exceeds 12%.

Alerts should include:

• Workflow or job name
• Branch (main vs. feature branch)
• Recent commit SHA and author
• Link to CircleCI workflow details page
• Comparison to baseline (e.g., “Duration: 18m vs. median 9m”)

Step 7: Analyze Build Performance Trends and Bottlenecks

Once monitoring and alerts are running, the next step is analysis. Aggregate data over time to identify patterns.

Use CircleCI Insights Workflow Trends view to:

• Compare duration across branches to see if feature branch builds are slower than main
• Identify which jobs contribute most to total workflow time (use the workflow breakdown chart)
• Spot flaky tests by filtering jobs with inconsistent success rates

For deeper analysis, export CircleCI metrics to a data warehouse or observability platform and query:

• Slowest jobs by week: Identify jobs where p95 duration increased more than 25% week over week
• Failure patterns by time of day: Spot if failures cluster during deploy windows or off-hours
• Credit consumption trends: Track which workflows consume the most CircleCI compute credits

Example query in SQL (assumes metrics exported to a warehouse):

SELECT 
  workflow_name,
  job_name,
  AVG(duration_seconds) AS avg_duration,
  PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_seconds) AS p95_duration,
  SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) * 100.0 / COUNT(*) AS failure_rate
FROM circleci_jobs
WHERE created_at >= NOW() - INTERVAL '30 days'
GROUP BY workflow_name, job_name
ORDER BY p95_duration DESC
LIMIT 20;

This surfaces the 20 slowest jobs by p95 duration and their failure rates over the last 30 days.

For teams monitoring Cassandra clusters or other databases in their CI pipelines, correlate job failures with database availability metrics to catch infrastructure-related CI failures.

Troubleshooting Common Issues

CircleCI Insights Data Missing or Delayed

Insights data refreshes every 10 minutes. If workflows completed in the last 5 minutes are not visible, wait for the next refresh cycle. For workflows older than 90 days, Insights data is not available — export metrics via API if longer retention is required.

Webhook Events Not Appearing in Observability Platform

Verify the webhook URL is correct in Project Settings > Webhooks. Test the webhook by clicking Test Ping. Check your observability platform’s ingestion logs for errors. Ensure the API key or token used has write permissions.

High Queue Times Not Reflected in Alerts

Queue time is measured from workflow trigger to first job start. If you are using resource classes with limited concurrency, queue times spike when all executors are busy. Check Plan > Resource Classes in CircleCI to see current concurrency limits. Upgrade to a higher concurrency plan or optimize job parallelism to reduce queue pressure.

API Rate Limits When Polling CircleCI API

CircleCI v2 API rate limit is 1,000 requests per hour per token. If polling every 60 seconds across 20 projects, you will exceed this. Increase polling interval to 5 minutes or use webhooks instead of polling.

False Positive Alerts for Flaky Tests

If alerts fire frequently due to intermittent test failures, adjust alert thresholds to require failures in 3 out of 5 runs instead of a single failure. Alternatively, track test-level failure rates separately and alert only when a specific test fails more than 20% of the time over 24 hours.

Monitoring CircleCI build performance and failures is not a one-time setup. As your codebase grows, test suites expand, and deployment frequency increases, CI pipelines degrade unless actively managed. The setup in this guide gives you the baseline visibility to catch regressions early, prioritize optimization work, and keep deployments fast and reliable.

Start with CircleCI Insights to understand your current baseline. Add external monitoring via Datadog or CubeAPM if you need long term retention, cross-platform correlation, or self hosted data control. Instrument pipelines with custom metrics to surface job level bottlenecks that aggregate metrics miss. Set alerts that route to the right team with enough context to act immediately.

For teams managing observability across multiple platforms, understanding what is synthetic monitoring and what is real user monitoring helps round out full stack visibility beyond CI/CD pipelines.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions