How to Monitor Celery Task Queue Backlogs and Worker Health: Step by Step Guide

Without proper monitoring, a Celery task queue can silently fill with thousands of backlogged tasks while workers crash, hang, or stop consuming entirely. By the time users report slow background jobs or failed emails, the backlog may have been building for hours. Celery monitoring gives you real time visibility into queue depth, worker availability, task failure rates, and processing latency so you can detect and fix bottlenecks before they cascade into outages.

This guide walks through how to monitor Celery task queues and worker health in production using native Celery tools, Prometheus metrics, and dedicated observability platforms. You will learn how to track queue backlogs, detect unresponsive workers, expose metrics for alerting, and correlate task failures with application traces.

Prerequisites

Before starting, ensure you have:

• A working Celery deployment with Redis or RabbitMQ as the broker
• Python 3.8 or later installed
• Access to your Celery worker logs and broker instance
• Basic familiarity with Celery task routing and worker configuration
• Optional: Prometheus or an APM platform for centralized metrics collection

Step 1: Understand What You Need to Monitor in Celery

Celery monitoring breaks into three layers: broker queue depth, worker process health, and task execution outcomes. Each layer answers a different operational question.

Queue depth tells you how many tasks are waiting to be processed. A growing queue depth over time signals that workers cannot keep up with the rate of task creation. This can happen during traffic spikes, after a worker crash, or when tasks take longer than expected to complete.

Worker health tracks whether worker processes are alive, connected to the broker, and actively consuming tasks. Workers can appear online but silently stop processing tasks due to memory exhaustion, hung tasks blocking slots, or network issues with the broker.

Task execution metrics show success rates, failure patterns, retry behavior, and execution duration. These metrics surface application level issues like database timeouts, third party API failures, or logic errors in task code.

To get complete visibility, you need to monitor all three layers together. A backlog might be caused by slow task execution, insufficient worker capacity, or a combination of both. Without full context, you cannot distinguish between a traffic surge that will resolve itself and a production incident that needs immediate intervention.

Step 2: Monitor Queue Depth Using Broker Inspection Tools

The simplest way to check Celery queue depth is to query the broker directly. The method depends on whether you are using Redis or RabbitMQ.

For Redis based deployments, use the redis-cli command to inspect queue length:

redis-cli -h localhost -p 6379 llen celery

This returns the number of tasks waiting in the default celery queue. If you use multiple queues for task routing, check each queue individually:

redis-cli llen high_priority
redis-cli llen low_priority
redis-cli llen email_queue

For RabbitMQ based deployments, use rabbitmqctl to list queue depths:

rabbitmqctl list_queues name messages_ready messages_unacknowledged

The messages_ready column shows tasks waiting to be picked up by workers. The messages_unacknowledged column shows tasks currently being processed but not yet acknowledged. A high unacknowledged count combined with no task completion suggests workers are hung or stuck.

Queue depth checks should run continuously, not just during incidents. A scheduled script running every 30 seconds gives you trending data to detect gradual backlog growth before it becomes critical.

#!/bin/bash
while true; do
  echo "$(date) - Queue depth: $(redis-cli llen celery)"
  sleep 30
done

This basic watchdog approach works for small deployments but does not scale. As your Celery infrastructure grows, you need structured metrics and alerting instead of manual log review. Platforms designed for infrastructure monitoring provide automated collection, trending, and alerting for broker metrics without requiring custom scripts.

Step 3: Use Celery Inspect Commands to Check Worker Status

Celery includes built in inspect commands that query worker state directly through the control plane. These commands work regardless of your broker type and provide detailed worker information without needing broker credentials.

To check which workers are online and responding:

celery -A myproject inspect ping

This sends a ping to all registered workers and waits for responses. Workers that do not respond within a few seconds are either offline, unresponsive, or disconnected from the broker.

To see which tasks each worker is currently executing:

celery -A myproject inspect active

This returns a JSON structure showing active tasks per worker, including task name, arguments, start time, and worker hostname. If a task has been running for an unusually long time, this command surfaces it immediately.

To view tasks that workers have reserved but not yet started:

celery -A myproject inspect reserved

Reserved tasks sit in the worker’s prefetch buffer. A large reserved count with no active tasks suggests the worker has stopped processing entirely.

To check worker configuration and registered task names:

celery -A myproject inspect registered
celery -A myproject inspect stats

The stats command returns detailed metrics including total tasks processed, prefetch count, pool type, and current load. Use this during incident response to verify worker configuration matches your expectations.

Inspect commands provide point in time visibility but do not give you historical trends or automated alerting. For production monitoring, these commands should be wrapped in scripts that log output to a centralized system or expose them as Prometheus metrics.

Step 4: Expose Celery Metrics Using Prometheus Client Library

To monitor Celery in production, export task and worker metrics in a format that integrates with observability platforms. The Prometheus client library for Python makes this straightforward.

Install the Prometheus client:

pip install prometheus-client

Add metric instrumentation to your Celery task definitions:

from celery import Celery
from prometheus_client import Counter, Histogram, Gauge, start_http_server

app = Celery('tasks', broker='redis://localhost:6379/0')

task_counter = Counter('celery_tasks_total', 'Total tasks executed', ['task_name', 'status'])
task_duration = Histogram('celery_task_duration_seconds', 'Task execution time', ['task_name'])
tasks_in_progress = Gauge('celery_tasks_in_progress', 'Tasks currently executing', ['task_name'])

@app.task(bind=True)
def process_data(self, data):
    tasks_in_progress.labels(task_name='process_data').inc()
    try:
        with task_duration.labels(task_name='process_data').time():
            result = expensive_operation(data)
        task_counter.labels(task_name='process_data', status='success').inc()
        return result
    except Exception as e:
        task_counter.labels(task_name='process_data', status='failure').inc()
        raise
    finally:
        tasks_in_progress.labels(task_name='process_data').dec()

if __name__ == '__main__':
    start_http_server(8000)
    app.start()

This exposes metrics on http://localhost:8000/metrics in Prometheus format. Configure Prometheus to scrape this endpoint every 15 seconds by adding it to your prometheus.yml:

scrape_configs:
  - job_name: 'celery_workers'
    static_configs:
      - targets: ['worker1:8000', 'worker2:8000', 'worker3:8000']

The metrics exposed include task execution counts, success and failure rates, task duration histograms, and in-flight task counts. These form the foundation for alerting rules and Grafana dashboards.

For queue depth metrics, add a separate exporter that queries your broker and exposes queue lengths as Prometheus metrics. For Redis:

from prometheus_client import Gauge, start_http_server
import redis
import time

queue_depth = Gauge('celery_queue_depth', 'Tasks waiting in queue', ['queue_name'])

r = redis.Redis(host='localhost', port=6379, db=0)

def collect_metrics():
    while True:
        queue_depth.labels(queue_name='celery').set(r.llen('celery'))
        queue_depth.labels(queue_name='high_priority').set(r.llen('high_priority'))
        time.sleep(15)

if __name__ == '__main__':
    start_http_server(8001)
    collect_metrics()

This pattern scales to hundreds of queues and thousands of workers without requiring manual log parsing or custom dashboards.

Step 5: Set Up Alerts for Queue Backlogs and Worker Failures

Metrics are only useful if they trigger alerts when thresholds are breached. Define alert rules in Prometheus or your observability platform to notify your team when Celery health degrades.

A basic queue backlog alert fires when queue depth exceeds a threshold for a sustained period:

groups:
  - name: celery_alerts
    interval: 30s
    rules:
      - alert: CeleryQueueBacklog
        expr: celery_queue_depth{queue_name="celery"} > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Celery queue backlog detected"
          description: "Queue {{ $labels.queue_name }} has {{ $value }} tasks waiting for over 5 minutes"

This alert fires if the queue grows above 1000 tasks and stays there for 5 minutes. Adjust the threshold based on your normal traffic patterns and worker capacity.

A worker availability alert fires when fewer workers are online than expected:

- alert: CeleryWorkerDown
  expr: up{job="celery_workers"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Celery worker {{ $labels.instance }} is down"
    description: "Worker has not responded to health checks for 2 minutes"

A task failure rate alert fires when the ratio of failed tasks to total tasks exceeds a threshold:

- alert: CeleryHighFailureRate
  expr: rate(celery_tasks_total{status="failure"}[5m]) / rate(celery_tasks_total[5m]) > 0.1
  for: 3m
  labels:
    severity: warning
  annotations:
    summary: "High task failure rate detected"
    description: "Task failure rate is {{ $value | humanizePercentage }} over the last 5 minutes"

This fires if more than 10 percent of tasks are failing over a 5 minute window. Route these alerts to Slack, PagerDuty, or email based on severity.

For teams using a full stack observability platform instead of a DIY Prometheus stack, alert configuration is simpler. CubeAPM provides built in alerting for Celery metrics with anomaly detection that reduces false positives. Define alert rules directly in the UI based on queue depth, worker count, or task failure rate without writing PromQL queries. Alerts include full context, linking directly to the traces and logs for failed tasks so your team can diagnose root cause without switching tools.

Step 6: Visualize Celery Metrics in Grafana or Your Observability Platform

Once metrics are collected and alerts are configured, build dashboards to visualize Celery health in real time. Grafana is the most common choice for Prometheus based monitoring.

Create a Grafana dashboard with panels for:

• Queue depth over time for each queue
• Active worker count
• Task execution rate (tasks per second)
• Task success and failure rate
• Task duration percentiles (p50, p95, p99)
• Worker CPU and memory usage

A sample Grafana panel query for queue depth:

celery_queue_depth{queue_name="celery"}

A sample query for task failure rate:

rate(celery_tasks_total{status="failure"}[5m])

Grafana provides pre-built Celery dashboards through the Grafana community. Import dashboard ID 13054 or similar from grafana.com/grafana/dashboards to get started quickly.

For teams that prefer a unified monitoring platform over assembling Grafana, Prometheus, and exporters manually, CubeAPM provides native Celery monitoring with pre-built dashboards and deep trace correlation. CubeAPM automatically detects Celery workers via OpenTelemetry instrumentation, tracks queue depth by polling your broker, and correlates task failures with application traces and logs. This eliminates the need to maintain custom exporters or stitch together multiple tools. View queue backlogs, worker health, and task-level performance in a single dashboard with drill-down to individual task traces and error stack traces.

Step 7: Monitor Worker Health with Celery Events and Flower

Celery emits real time events whenever tasks are received, started, succeeded, failed, or retried. These events can be consumed by monitoring tools to provide live visibility into task execution without adding instrumentation to every task.

Start the Celery events listener:

celery -A myproject events

This opens a curses based terminal UI showing live task activity. While useful for debugging, it does not scale for production monitoring.

Flower is a web based monitoring tool that consumes Celery events and provides a dashboard showing active tasks, worker status, task history, and runtime statistics. Install Flower:

pip install flower

Start Flower connected to your Celery broker:

celery -A myproject flower --port=5555

Access the Flower UI at http://localhost:5555. The dashboard shows:

• All registered workers and their status
• Active, scheduled, and reserved tasks per worker
• Task execution history with success and failure counts
• Real time task rate and latency charts

Flower also exposes a Prometheus metrics endpoint at /metrics if you enable the integration in flowerconfig.py:

prometheus = True

This allows you to scrape Flower metrics alongside your custom task instrumentation for a complete view.

Flower is suitable for small to midsize Celery deployments. For larger production environments with hundreds of workers or high task throughput, Flower’s SQLite backend and event stream processing can become a bottleneck. In these cases, export metrics to a dedicated observability platform instead.

Step 8: Correlate Celery Task Failures with Application Traces

When a Celery task fails, you need to know why. Task failure logs often contain exception messages but lack the full request context that caused the failure. Distributed tracing solves this by linking Celery tasks to the HTTP requests, database queries, and external API calls that triggered them.

Instrument your Celery tasks with OpenTelemetry to automatically generate trace spans:

from opentelemetry import trace
from opentelemetry.instrumentation.celery import CeleryInstrumentor

CeleryInstrumentor().instrument()

tracer = trace.get_tracer(__name__)

@app.task
def send_email(user_id, message):
    with tracer.start_as_current_span("send_email_task"):
        user = fetch_user_from_db(user_id)
        result = email_service.send(user.email, message)
        return result

OpenTelemetry automatically captures the task name, arguments, execution time, and outcome. If the task fails, the trace includes the exception and stack trace.

Configure OpenTelemetry to export traces to your APM platform or observability backend. For example, export to CubeAPM via OTLP:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="http://cubeapm-collector:4317")
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))

With distributed tracing enabled, every Celery task failure appears in your APM dashboard with full context: the originating HTTP request, upstream service calls, downstream task dependencies, and the exact database query or API call that caused the failure. This reduces mean time to resolution by eliminating the need to correlate logs manually across services.

CubeAPM provides native support for OpenTelemetry instrumented Celery tasks. Tasks appear automatically in the trace timeline, linked to the request that enqueued them. Click a failed task span to view the exception, stack trace, task arguments, and all related logs in one unified view.

Troubleshooting Common Issues

Workers appear online but tasks are not being processed: Check the worker prefetch multiplier setting. If set too high, tasks may be reserved by workers but not executed if the worker pool is full. Lower worker_prefetch_multiplier to 1 or 2 to reduce task hoarding.

Queue depth grows steadily despite workers running: Measure task execution time and compare it to the task creation rate. If tasks take longer to execute than the rate at which they are created, you need to add more workers or optimize task code. Use the task duration histogram metric to identify slow tasks.

Tasks succeed in development but fail in production: Check for environment specific configuration issues like missing environment variables, network access restrictions, or resource limits. Enable debug logging temporarily to capture the full error context.

Redis broker shows high memory usage: Celery stores task metadata in Redis until tasks are acknowledged. If workers crash or hang without acknowledging tasks, Redis memory usage can grow unbounded. Set task_acks_late = True and task_reject_on_worker_lost = True in Celery configuration to ensure tasks are acknowledged only after successful completion.

Flower dashboard shows no workers despite workers running: Verify that workers and Flower are connected to the same broker URL. Check for firewall rules blocking the Celery event stream port (usually 5555).

Prometheus scrape targets show as down: Confirm that the Prometheus metrics endpoint is accessible from the Prometheus server. Use curl http://worker-host:8000/metrics to verify the endpoint responds. Check for network policies or security groups blocking the port.

Conclusion

Monitoring Celery task queues and worker health requires tracking three layers: broker queue depth, worker process availability, and task execution outcomes. Use Celery inspect commands for point in time visibility, expose metrics via Prometheus for trending and alerting, and integrate distributed tracing to correlate task failures with application behavior. Tools like Flower provide real time dashboards for smaller deployments, while full stack observability platforms like CubeAPM centralize Celery metrics, traces, and logs in one place to reduce context switching and speed up troubleshooting.

Start by exposing queue depth and worker status metrics, set up alerts for backlog growth and worker failures, and instrument tasks with OpenTelemetry to capture full execution context. This gives your team the visibility needed to maintain reliable background task processing at scale.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions