How to Monitor OpenAI API Latency and Cost: Step-by-Step Guide

Without instrumentation, OpenAI API costs are a black box until your monthly bill arrives. A single retry loop bug can 10x your daily spend before you notice. A verbose system prompt added to one feature can double input token consumption overnight. By the time a cost spike shows up in the OpenAI dashboard, you have already paid for it.

This guide walks through setting up latency and cost monitoring for OpenAI API calls using OpenTelemetry. You will instrument a Python or Node.js application, compute cost at the point of the API call using a pricing table you control, attribute spend by feature and user, and set up alerts that catch anomalies before they hit your billing account. Every code block is copy-paste ready.

Prerequisites

Before you start, ensure you have:

• An OpenAI API key with active usage
• Python 3.8+ or Node.js 16+ installed
• Access to an OpenTelemetry-compatible observability backend (this guide uses OpenObserve, but Prometheus, Grafana Cloud, or any OTLP endpoint works)
• Basic familiarity with your application’s request flow
• A pricing table for OpenAI models you use (GPT-4o, GPT-4o-mini, etc.)

If you are starting from zero, create a test OpenAI account and grab an API key from the OpenAI platform dashboard. For the observability backend, OpenObserve offers a free tier that works for this tutorial.

Step 1: Install OpenTelemetry Instrumentation for OpenAI

OpenTelemetry has official instrumentation libraries for the OpenAI SDK in both Python and Node.js. These libraries wrap every API call automatically and emit spans with token counts, model names, and request metadata.

For Python

Install the three required packages:

pip install opentelemetry-distro
pip install opentelemetry-exporter-otlp
pip install opentelemetry-instrumentation-openai-v2

Then run the bootstrap command to install auto-instrumentation for any other libraries in your app:

opentelemetry-bootstrap --action=install

This installs instrumentation for Flask, FastAPI, requests, and other common Python libraries if they are present in your environment.

For Node.js

Install the OpenTelemetry Node SDK and OpenAI instrumentation:

npm install @opentelemetry/api \
  @opentelemetry/sdk-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/instrumentation-openai

The Node.js instrumentation wraps the OpenAI SDK at require time, similar to the Python approach.

Step 2: Configure the OTLP Exporter

OpenTelemetry sends trace data to your observability backend via OTLP (OpenTelemetry Protocol). You need to point the exporter at your backend’s OTLP endpoint.

Python configuration

Set these environment variables before running your application:

export OTEL_SERVICE_NAME=openai-cost-tracker
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.openobserve.ai/api/<your-org>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your-auth-token>"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

Replace <your-org> and <your-auth-token> with your actual OpenObserve credentials from the Data Sources section. If you are self-hosting or using a different backend, adjust the endpoint URL accordingly.

Run your application with:

opentelemetry-instrument python app.py

No code changes required. The wrapper imports the instrumentation before your app code runs.

Node.js configuration

Create a tracing.js file:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { OpenAIInstrumentation } = require('@opentelemetry/instrumentation-openai');
const { Resource } = require('@opentelemetry/resources');

const sdk = new NodeSDK({
  resource: new Resource({
    'service.name': 'openai-cost-tracker',
    'deployment.environment': process.env.NODE_ENV || 'development',
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces`,
    headers: {
      Authorization: process.env.OTEL_EXPORTER_OTLP_HEADERS,
    },
  }),
  instrumentations: [new OpenAIInstrumentation()],
});

sdk.start();

Set the same environment variables as Python (except the logging one), then preload the tracing file when running your app:

node --require ./tracing.js app.js

Every OpenAI call now emits a span with token counts and model metadata automatically.

Step 3: Add Feature and User Attributes to Every Span

Token counts alone are not enough. You need to know which feature or user is driving spend. OpenTelemetry lets you add custom attributes to spans.

Python example

from openai import OpenAI
from opentelemetry import trace

client = OpenAI()
tracer = trace.get_tracer(__name__)

def summarize_for_user(user_id, feature_name, text):
    with tracer.start_as_current_span("openai_summarize") as span:
        span.set_attribute("user.id", user_id)
        span.set_attribute("feature.name", feature_name)
        
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"Summarize: {text}"}],
        )
        
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
        
        return response.choices[0].message.content

This tags every OpenAI call with the user ID and feature name, so you can slice cost by both dimensions in your observability backend.

Node.js example

const openai = require('openai');
const { trace } = require('@opentelemetry/api');

const client = new openai.OpenAI();
const tracer = trace.getTracer('openai-cost-tracker');

async function summarizeForUser(userId, featureName, text) {
  return tracer.startActiveSpan('openai_summarize', async (span) => {
    span.setAttribute('user.id', userId);
    span.setAttribute('feature.name', featureName);
    
    const response = await client.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: `Summarize: ${text}` }],
    });
    
    span.setAttribute('gen_ai.usage.input_tokens', response.usage.prompt_tokens);
    span.setAttribute('gen_ai.usage.output_tokens', response.usage.completion_tokens);
    span.end();
    
    return response.choices[0].message.content;
  });
}

The pattern is the same across both languages: wrap your OpenAI call in a span, add business context as attributes, and the instrumentation captures the token counts automatically.

Step 4: Compute Cost from Token Counts

OpenAI charges per token, but the rate varies by model and token type. Input tokens and output tokens have different prices. You need a pricing table in your code.

Pricing table (as of April 2026)

This data comes from OpenAI’s official pricing page:

Model	Input token price	Output token price
gpt-4o	$0.0025 per 1K	$0.010 per 1K
gpt-4o-mini	$0.00015 per 1K	$0.0006 per 1K
gpt-3.5-turbo	$0.0005 per 1K	$0.0015 per 1K

Pricing reflects publicly available information as of April 2026. Verify current rates at [OpenAI’s pricing page](https://openai.com/api/pricing/) before deploying to production.

Python cost calculation

PRICING = {
    "gpt-4o": {"input": 0.0025 / 1000, "output": 0.010 / 1000},
    "gpt-4o-mini": {"input": 0.00015 / 1000, "output": 0.0006 / 1000},
    "gpt-3.5-turbo": {"input": 0.0005 / 1000, "output": 0.0015 / 1000},
}

def calculate_cost(model, input_tokens, output_tokens):
    rates = PRICING.get(model, {"input": 0, "output": 0})
    cost = (input_tokens * rates["input"]) + (output_tokens * rates["output"])
    return round(cost, 6)

# Inside your span:
cost_usd = calculate_cost(
    "gpt-4o-mini",
    response.usage.prompt_tokens,
    response.usage.completion_tokens
)
span.set_attribute("gen_ai.usage.cost_usd", cost_usd)

This computes cost at the point of the API call and attaches it to the span. Your observability backend can now sum gen_ai.usage.cost_usd to get total spend per feature, user, or time window.

Node.js cost calculation

const PRICING = {
  'gpt-4o': { input: 0.0025 / 1000, output: 0.010 / 1000 },
  'gpt-4o-mini': { input: 0.00015 / 1000, output: 0.0006 / 1000 },
  'gpt-3.5-turbo': { input: 0.0005 / 1000, output: 0.0015 / 1000 },
};

function calculateCost(model, inputTokens, outputTokens) {
  const rates = PRICING[model] || { input: 0, output: 0 };
  const cost = (inputTokens * rates.input) + (outputTokens * rates.output);
  return parseFloat(cost.toFixed(6));
}

// Inside your span:
const costUsd = calculateCost(
  'gpt-4o-mini',
  response.usage.prompt_tokens,
  response.usage.completion_tokens
);
span.setAttribute('gen_ai.usage.cost_usd', costUsd);

The pricing table lives in your code, not your observability backend. This gives you control: when OpenAI changes prices, you update one table and redeploy.

Step 5: Emit Cost as a Histogram Metric

Spans give you per-request drill down. Metrics give you aggregated dashboards and alerting. Emit gen_ai.usage.cost_usd as both a span attribute and a histogram metric.

Python metric emission

from opentelemetry import metrics

meter = metrics.get_meter(__name__)
cost_histogram = meter.create_histogram(
    name="openai.request.cost",
    description="Cost per OpenAI API request in USD",
    unit="USD",
)

# After computing cost:
cost_histogram.record(cost_usd, {
    "model": "gpt-4o-mini",
    "feature": feature_name,
    "user_id": user_id,
})

This histogram tracks the distribution of request costs. You can query the 95th percentile, sum total spend per feature, or alert when the rate of spend exceeds a threshold.

Node.js metric emission

const { metrics } = require('@opentelemetry/api');

const meter = metrics.getMeter('openai-cost-tracker');
const costHistogram = meter.createHistogram('openai.request.cost', {
  description: 'Cost per OpenAI API request in USD',
  unit: 'USD',
});

// After computing cost:
costHistogram.record(costUsd, {
  model: 'gpt-4o-mini',
  feature: featureName,
  user_id: userId,
});

Same pattern in both languages: emit the cost with dimensions that let you filter by model, feature, or user in your dashboards.

Step 6: Monitor Latency and Time to First Token

Cost is half the story. Latency is the other half. OpenAI streaming endpoints return the first token before the full response completes. Measure both total latency and time to first token.

The OpenTelemetry instrumentation captures total latency automatically in the span duration. For streaming, you need to manually instrument.

Python streaming latency

import time

def stream_completion(user_id, feature_name, prompt):
    with tracer.start_as_current_span("openai_stream") as span:
        span.set_attribute("user.id", user_id)
        span.set_attribute("feature.name", feature_name)
        
        start_time = time.time()
        first_token_time = None
        
        stream = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            stream=True,
        )
        
        for chunk in stream:
            if first_token_time is None:
                first_token_time = time.time() - start_time
                span.set_attribute("latency.first_token_ms", int(first_token_time * 1000))
            
            yield chunk.choices[0].delta.content or ""
        
        total_time = time.time() - start_time
        span.set_attribute("latency.total_ms", int(total_time * 1000))

This captures both time to first token and total response time as span attributes.

Node.js streaming latency

async function* streamCompletion(userId, featureName, prompt) {
  return tracer.startActiveSpan('openai_stream', async (span) => {
    span.setAttribute('user.id', userId);
    span.setAttribute('feature.name', featureName);
    
    const startTime = Date.now();
    let firstTokenTime = null;
    
    const stream = await client.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: prompt }],
      stream: true,
    });
    
    for await (const chunk of stream) {
      if (firstTokenTime === null) {
        firstTokenTime = Date.now() - startTime;
        span.setAttribute('latency.first_token_ms', firstTokenTime);
      }
      
      yield chunk.choices[0]?.delta?.content || '';
    }
    
    const totalTime = Date.now() - startTime;
    span.setAttribute('latency.total_ms', totalTime);
    span.end();
  });
}

With both latency dimensions instrumented, you can track whether slow responses are due to OpenAI queuing time (high time to first token) or large output size (high total latency).

Step 7: Set Up Cost Anomaly Alerts

Static budget alerts fire too late. A $1,000 daily budget alert does not tell you that today’s spend is 5x yesterday’s rate at the same time of day. Anomaly detection catches spikes early.

Most observability platforms support alerting on metric deltas. Here is the logic to implement:

1. Query the rolling 7-day average cost per hour for your service
2. Alert when the current hour exceeds 3x the rolling average
3. Route the alert to Slack or PagerDuty with the feature and user dimensions attached

Example alert query (PromQL syntax)

rate(openai_request_cost_sum[1h]) 
  > 3 * avg_over_time(rate(openai_request_cost_sum[1h])[7d:1h])

This fires when the current hour’s spend rate is more than 3x the average hourly spend over the past week. Adjust the multiplier based on your traffic patterns.

For CubeAPM users, navigate to Alerts → Create Alert → Select Metric: openai.request.cost → Condition: rate exceeds 3x baseline → Route to: Slack channel.

Troubleshooting Common Issues

Spans are not appearing in my observability backend

Check that the OTLP endpoint is correct and reachable. Test with curl:

curl -X POST $OTEL_EXPORTER_OTLP_ENDPOINT/v1/traces \
  -H "Authorization: Basic $OTEL_EXPORTER_OTLP_HEADERS" \
  -H "Content-Type: application/x-protobuf" \
  --data-binary @- < /dev/null

If this fails with 401 or 404, your endpoint or credentials are wrong.

Token counts are zero or missing

The OpenTelemetry instrumentation only captures token counts from the response object. If you are using an older OpenAI SDK version that does not include usage in the response, upgrade to the latest SDK version.

Cost calculations are off by 10x

Check your pricing table. The prices are per 1,000 tokens, so you must divide by 1,000 in your calculation. If you multiply instead of divide, costs will be 1 million times higher than reality.

Streaming responses are not instrumented

The auto-instrumentation only wraps non-streaming calls. For streaming, you must manually create spans as shown in Step 6. This is a limitation of how OpenTelemetry wraps async generators.

Alerts are firing constantly

Your anomaly threshold is too tight. Start with 5x baseline and tune down once you understand your normal variance. Check the feature.name dimension to see if one feature is driving the anomaly.

Monitoring OpenAI API Costs with CubeAPM

CubeAPM is an on-premises observability platform that ingests OpenTelemetry traces, metrics, and logs with unlimited retention at $0.15 per GB. It runs inside your VPC, so telemetry data never leaves your infrastructure. For teams monitoring OpenAI API costs, this means no data egress fees and full control over sensitive prompt and response content.

CubeAPM natively understands OpenTelemetry’s GenAI semantic conventions, so all the gen_ai.* attributes emitted by the instrumentation libraries in this guide are automatically indexed and queryable. You can filter traces by model, feature, user, or cost threshold without writing custom queries.

To set up cost monitoring in CubeAPM:

1. Deploy CubeAPM in your VPC following the self-hosted deployment guide
2. Point your OTLP exporter at http://<cubeapm-host>:4318
3. Navigate to Traces → Filter by gen_ai.usage.cost_usd to see per-request cost distribution
4. Create a dashboard with the openai.request.cost histogram metric to track spend over time
5. Set up an alert on rate(openai_request_cost_sum[1h]) > threshold to catch cost spikes

CubeAPM’s flat $0.15 per GB pricing means monitoring cost is predictable regardless of trace volume. A team ingesting 10 TB of traces and metrics per month pays $1,500 total with no per-seat, per-host, or per-feature add-ons.

For more on infrastructure monitoring or tools for API monitoring, see the linked guides.

This guide demonstrates a production-ready setup for teams with moderate trace volume. If you are ingesting more than 50 TB per month, consider sampling strategies or centralized metric aggregation before the OTLP export step.

Monitoring OpenAI API latency and cost is not optional once your application depends on LLM calls in production. Without instrumentation, you are flying blind until the bill arrives. With OpenTelemetry and a pricing table you control, you can track cost per request, attribute spend by feature and user, and alert on anomalies before they compound into budget overruns. The code in this guide is production-ready. Deploy it, adjust the pricing table when OpenAI updates rates, and you will never be surprised by an LLM bill again.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions