Weaviate exposes over 50 Prometheus compatible metrics covering query latency, batch throughput, vector index health, and memory usage. Without monitoring these signals, a memory leak or tombstone buildup can quietly degrade search quality for hours before anyone notices. With the right metrics in place, the same issue triggers an alert, surfaces the exact shard or collection causing the bottleneck, and gives operators the context to fix it before users complain.
According to the CNCF Annual Survey 2024, 87% of organizations now use observability tools to monitor production workloads, but most struggle to connect application layer performance with infrastructure and database metrics. For Weaviate, that gap means teams often see API latency spikes without understanding whether the root cause is indexing overhead, tombstone cleanup cycles, or cluster resource saturation.
This guide covers what Weaviate performance metrics exist, how to expose and scrape them, which metrics matter most for production stability, and how to build monitoring dashboards that connect Weaviate’s internal health to your application’s user experience.
What Are Weaviate Performance Metrics
Weaviate performance metrics are Prometheus compatible gauges, counters, histograms, and summaries that Weaviate exposes at the /metrics endpoint when monitoring is enabled. These metrics measure every layer of Weaviate’s runtime behavior: object storage operations, vector index maintenance, query execution, batch ingestion, and resource consumption.
The metrics system in Weaviate is designed to answer specific operational questions in production: Is query latency increasing? Are batch writes slowing down? Is the vector index healthy or accumulating tombstones? Is memory growing faster than expected? Each metric provides a specific signal that maps to one of these questions.
Weaviate’s metrics cover five broad categories:
Object operations track how long it takes to read, write, update, or delete objects in Weaviate’s storage layer. These metrics include labels for class name and shard name, making it possible to isolate slow operations to specific collections or partitions.
Query operations measure how long vector searches, filtered queries, and hybrid searches take to execute. Query metrics include labels for query type and class name, allowing you to track whether latency is caused by a specific type of query or a specific collection.
Vector index operations expose the health and maintenance state of HNSW indexes. These metrics track index size, tombstone counts, cleanup cycles, and vector index maintenance durations. High tombstone counts or long cleanup cycles indicate that delete operations are causing index fragmentation.
Batch operations measure throughput and latency for batch imports. Batch metrics break down into sub operations: object storage, inverted index updates, and vector index updates. This granularity helps identify which part of the batch pipeline is slow.
Infrastructure metrics expose memory usage, goroutine counts, and runtime statistics. These are standard Go runtime metrics but labeled by Weaviate class and shard, connecting resource consumption directly to workload patterns.
Weaviate’s metric system does not follow semantic versioning. Metrics can change between minor releases. The Weaviate documentation warns that metrics have shorter feature lifecycles than core APIs, and breaking changes are possible in minor releases. This means monitoring setups should be reviewed after each Weaviate upgrade.
How Weaviate Exposes Performance Metrics
Weaviate metrics are disabled by default. To enable them, set the PROMETHEUS_MONITORING_ENABLED environment variable to true in your Weaviate deployment configuration. Once enabled, Weaviate exposes metrics at http://<weaviate-host>:2112/metrics by default.
The port can be changed using the PROMETHEUS_MONITORING_PORT environment variable. For example, setting PROMETHEUS_MONITORING_PORT=3456 moves the metrics endpoint to port 3456.
Metrics are scraped by a Prometheus server or compatible agent. The scrape interval determines how often metrics are collected. A 15 second scrape interval is common for production Weaviate deployments. Shorter intervals increase cardinality and storage cost but provide finer resolution for debugging latency spikes.
For multi tenant Weaviate deployments, set PROMETHEUS_MONITORING_GROUP=true to aggregate metrics across all tenants. Without this setting, metrics are exposed per tenant, which can create extremely high cardinality and overwhelm Prometheus storage if you have thousands of tenants.
The metrics endpoint is unauthenticated by default. If Weaviate is exposed to the public internet, restrict access to the metrics port using firewall rules or a reverse proxy with authentication. Exposing Weaviate metrics publicly can leak information about collection names, shard counts, and query patterns.
Core Weaviate Performance Metrics to Monitor
Not all metrics matter equally. The list below covers the metrics that most directly impact production stability and user experience.
Query latency and throughput
queries_durations_ms is a histogram tracking query execution time in milliseconds. It includes labels for class_name and query_type, making it possible to isolate slow queries to specific collections or query types like vector search, filtered vector search, or hybrid search.
Rising query latency often indicates vector index fragmentation, cluster resource saturation, or inefficient query patterns. Query latency above 200ms for simple vector searches typically signals a problem.
queries_total is a counter tracking the total number of queries executed. Use this metric to calculate queries per second and identify traffic spikes that may correlate with latency increases.
Batch import performance
batch_durations_ms measures how long batch operations take to complete. The operation label breaks this metric into sub operations: objects for object storage writes, inverted for inverted index updates, and vector for vector index updates.
Slow batch imports usually show up first in the vector operation. HNSW index updates are computationally expensive and scale with index size. If batch latency increases over time without changes to batch size, the vector index may be fragmenting or memory pressure may be slowing down index writes.
batch_objects_processed_total counts the total number of objects processed in batches. Divide this by batch_durations_ms to calculate batch throughput in objects per second.
Vector index health
vector_index_tombstones tracks the number of active tombstones in the vector index. Tombstones are created when objects are deleted but the underlying vector entries are not immediately removed from the HNSW graph. High tombstone counts degrade search quality and increase query latency.
Weaviate runs periodic tombstone cleanup cycles to remove deleted vectors from the index. The vector_index_tombstone_cleanup_threads gauge shows how many cleanup threads are currently active. If tombstone counts remain high despite active cleanup threads, the cleanup process may not be keeping up with delete rate.
vector_index_tombstone_cleaned counts the total number of tombstones removed during cleanup cycles. Compare this to vector_index_tombstones to understand cleanup effectiveness.
vector_index_size measures the total capacity of the vector index. This number is typically larger than the number of vectors imported because HNSW indexes pre allocate capacity to avoid resizing during growth.
Memory usage patterns
go_memstats_heap_inuse_bytes tracks heap memory in use by the Go runtime. For Weaviate, memory usage should follow a sawtooth pattern: memory increases as data is loaded, then drops sharply during garbage collection cycles. If memory usage stays close to GOMEMLIMIT without dropping, Weaviate is memory constrained and may start throttling operations or experiencing out of memory kills.
Rising memory usage without corresponding increases in object count or query load usually indicates a memory leak or inefficient query patterns holding references to large result sets.
Object count and storage
object_count is a gauge showing the total number of objects stored in each shard. Use this metric to track collection growth over time and detect unexpected changes in object count that may indicate data loss or batch import failures.
Concurrent operations
concurrent_queries_count tracks the number of queries running simultaneously. High concurrency can saturate CPU or memory and cause query latency to spike. Compare concurrent query count to query latency to identify whether concurrency is the bottleneck.
concurrent_goroutines shows the number of goroutines running in the Weaviate process. High goroutine counts indicate many parallel operations or potential goroutine leaks.
Setting Up Weaviate Monitoring with Prometheus and Grafana
The fastest way to get Weaviate metrics into a queryable system is to scrape them into Prometheus and visualize them in Grafana. The Weaviate examples repository includes a Docker Compose setup that runs Weaviate, Prometheus, and Grafana together with pre built dashboards.
To use this setup, clone the repository and run docker-compose up in the monitoring example directory. This starts:
- Weaviate with metrics enabled
- Prometheus configured to scrape Weaviate every 15 seconds
- Grafana with Prometheus as a datasource and sample dashboards loaded
The included dashboards cover query latency, batch throughput, vector index health, and memory usage. These dashboards provide a starting point but should be customized based on your specific workload patterns.
For production deployments, Prometheus should run outside the Weaviate cluster to ensure monitoring remains available during Weaviate failures. Configure Prometheus to scrape all Weaviate nodes in a cluster to track per node metrics and detect asymmetric load distribution.
Grafana dashboards should include alert thresholds for critical metrics like query latency above 200ms, batch throughput below expected baseline, tombstone count growing without cleanup, and memory usage above 80% of limit. Alerts should route to Slack, PagerDuty, or email depending on severity.
Monitoring Weaviate Performance with CubeAPM
CubeAPM provides unified monitoring for Weaviate alongside your application layer traces, logs, and infrastructure metrics. CubeAPM runs on premises inside your VPC or data center, keeping Weaviate telemetry data within your infrastructure and avoiding public cloud egress fees.
CubeAPM connects to Weaviate via Prometheus remote write or by scraping the /metrics endpoint directly. Once connected, CubeAPM automatically indexes all Weaviate metrics and surfaces them in pre built dashboards covering query latency, batch performance, vector index health, and resource consumption.
Unlike Grafana dashboards that require manual PromQL queries, CubeAPM dashboards use auto indexing to make all Weaviate metrics searchable by class name, shard name, operation type, and time range. This makes it faster to isolate slow queries to specific collections or trace batch latency spikes to specific shards without writing custom queries.
CubeAPM correlates Weaviate metrics with application layer traces, making it possible to see which API endpoints are triggering slow Weaviate queries and how Weaviate latency impacts end user experience. For example, if an API endpoint latency spikes, CubeAPM shows whether the spike is caused by slow Weaviate vector searches, slow database queries, or external API calls in a single unified view.
CubeAPM pricing is $0.15/GB for all ingested telemetry with unlimited retention and no per seat fees. For a Weaviate deployment generating 500 GB of metrics and logs per month, CubeAPM costs $75/month with full access for all team members. Compare this to Datadog’s infrastructure monitoring at $18/host/month and APM at $42/host/month, which would cost $3,000/month for the same 50 host Weaviate cluster before adding logs or custom metrics.
Common Weaviate Performance Bottlenecks and How to Detect Them
Tombstone buildup causing slow queries
High vector_index_tombstones counts indicate that deleted objects are accumulating in the vector index. Tombstones degrade search quality and increase query latency because the HNSW graph contains stale entries that must be filtered out during search.
If tombstone counts grow faster than cleanup can remove them, increase the tombstone cleanup thread count or reduce delete rate. The Weaviate documentation recommends keeping tombstone counts below 10% of total vector index size.
Memory pressure throttling batch imports
If go_memstats_heap_inuse_bytes stays close to GOMEMLIMIT without dropping during garbage collection cycles, Weaviate is memory constrained. Memory pressure causes Go to throttle allocations, slowing down batch imports and query execution.
The fix is to increase GOMEMLIMIT or add more memory to Weaviate nodes. For batch heavy workloads, allocate at least 2x the working set size to allow room for garbage collection cycles.
HNSW index fragmentation slowing vector search
If query latency increases over time without changes to query patterns or object count, the HNSW index may be fragmenting. Fragmentation happens when the index grows through many small updates rather than bulk imports. Each small update requires the HNSW graph to rebalance, which degrades index quality.
The fix is to rebuild the index from scratch using bulk import. Export all objects, delete the collection, and reimport using batch operations. This rebuilds the HNSW index in optimal form and resets query latency to baseline.
Unbalanced shard load in multi node clusters
In multi node Weaviate clusters, check object_count and queries_durations_ms per shard to detect unbalanced load. If one shard handles significantly more objects or queries than others, cluster performance is limited by the slowest shard.
Rebalance shards by adjusting shard key distribution or by splitting hot shards into multiple smaller shards. The Weaviate documentation covers shard management in detail.
Best Practices for Weaviate Performance Monitoring
Monitor query latency at the 95th and 99th percentiles, not just averages. Average query latency hides tail latency spikes that degrade user experience. Use Prometheus histograms to track p95 and p99 latency and alert when either exceeds acceptable thresholds.
Track batch throughput as objects per second, not just batch duration. Batch duration alone does not account for batch size. Calculate throughput by dividing batch_objects_processed_total by batch_durations_ms to get a rate that is comparable across different batch sizes.
Set up alerts for tombstone count growing without cleanup. If vector_index_tombstones increases for more than one hour without vector_index_tombstone_cleaned increasing proportionally, tombstone cleanup is not keeping up. This condition should trigger an alert to investigate cleanup thread health.
Correlate Weaviate metrics with application traces to understand user impact. High query latency or batch latency does not matter if it is not affecting users. Use distributed tracing tools like those compatible with OpenTelemetry to connect Weaviate performance metrics to API latency and user session impact.
Test metric cardinality before deploying to production. High cardinality metrics like per tenant metrics in multi tenant deployments can overwhelm Prometheus storage. Use the PROMETHEUS_MONITORING_GROUP=true setting to aggregate metrics and reduce cardinality, or use a high cardinality capable storage backend like Thanos or Cortex.
Review metric definitions after every Weaviate upgrade. Metrics do not follow semantic versioning and can change between minor releases. Check the Weaviate changelog and update dashboards and alerts after upgrading to avoid broken queries or missed alerts.
—
Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.
Frequently Asked Questions
What are Weaviate performance metrics?
Weaviate performance metrics are Prometheus compatible measurements of query latency, batch throughput, vector index health, memory usage, and object operations exposed at the `/metrics` endpoint when monitoring is enabled.
How do I enable Weaviate metrics?
Set the `PROMETHEUS_MONITORING_ENABLED=true` environment variable in your Weaviate deployment configuration. Metrics are exposed at port 2112 by default and can be scraped by Prometheus or compatible monitoring tools.
Which Weaviate metrics matter most for production?
Monitor `queries_durations_ms` for query latency, `batch_durations_ms` for batch performance, `vector_index_tombstones` for index health, and `go_memstats_heap_inuse_bytes` for memory usage. These metrics cover the most common production failure modes.
How do I monitor Weaviate in a multi tenant deployment?
Set `PROMETHEUS_MONITORING_GROUP=true` to aggregate metrics across tenants and reduce cardinality. Without this setting, per tenant metrics can overwhelm Prometheus storage if you have thousands of tenants.
What causes high tombstone counts in Weaviate?
Tombstones accumulate when objects are deleted faster than the cleanup process can remove them from the vector index. High delete rates or insufficient cleanup threads cause tombstone buildup.
How do I reduce Weaviate query latency?
Check for high tombstone counts, memory pressure, or HNSW index fragmentation. Rebuild the index using bulk import, increase memory allocation, or increase tombstone cleanup thread count depending on the root cause.
Can CubeAPM monitor Weaviate alongside application traces?
Yes, CubeAPM scrapes Weaviate metrics via Prometheus and correlates them with distributed traces, logs, and infrastructure metrics in a unified view. This makes it faster to trace user facing latency back to Weaviate performance issues.