How to Monitor LLM Token Usage and Cost Per Request: Step-by-Step Guide

LLM API costs grow unpredictably because pricing depends on four variables that only resolve at runtime: input token count, output token count, the model used, and whether cached tokens were reused. A prompt containing a long document context can cost thirty times more than a short question, even if both produce identical responses. Without per-request visibility, teams discover cost problems only when the invoice arrives.

According to the CNCF’s 2024 observability survey, 68% of organizations now run AI or machine learning workloads in production, but most still rely on monthly invoices rather than real time token attribution. This guide covers how to instrument LLM calls, calculate cost per request, build cost dashboards, and enforce spend limits before bills scale beyond control.

Prerequisites

Before implementing LLM cost tracking, ensure you have:

• Active LLM API access: OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, or Google AI with valid API keys
• Application instrumentation rights: Ability to modify application code or deploy a proxy layer in front of LLM calls
• Observability backend: Prometheus, Grafana, OpenTelemetry Collector, or a managed observability platform
• Basic understanding of token counting: Familiarity with how LLM providers charge per input and output token
• Programming environment: Python, Node.js, or another language with LLM SDK support

Step 1: Capture Token Usage from the LLM Response

Every major LLM provider returns token counts in the API response. This is the most accurate source of cost data and should be logged on every call.

OpenAI, Anthropic, AWS Bedrock, and Azure OpenAI all include a usage object in their responses. Here is an example from OpenAI’s API:

import openai

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain Kubernetes pods"}]
)

usage = response['usage']
input_tokens = usage['prompt_tokens']
output_tokens = usage['completion_tokens']
total_tokens = usage['total_tokens']

print(f"Input: {input_tokens}, Output: {output_tokens}, Total: {total_tokens}")

For Anthropic’s Claude API, the structure is similar:

import anthropic

client = anthropic.Anthropic(api_key="your-api-key")
message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain Kubernetes pods"}]
)

input_tokens = message.usage.input_tokens
output_tokens = message.usage.output_tokens
print(f"Input: {input_tokens}, Output: {output_tokens}")

Why this matters: If you are not capturing the usage object from every LLM response, you are discarding the most accurate cost data available. Invoice-level summaries tell you the total bill but provide no attribution to user, feature, or workflow.

Store these values immediately in your observability backend as structured logs or metrics. Tag each entry with metadata like user_id, model, feature, and environment so cost can be attributed later.

Step 2: Attach Business Context to Every LLM Call

Raw token counts mean nothing without knowing which user, feature, or workflow generated them. Attribution metadata must be attached at request time, not reconstructed from logs afterward.

Create a wrapper function that logs every LLM call with contextual tags:

import openai
import logging
from datetime import datetime

def track_llm_call(model, messages, user_id, feature, environment):
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages
    )
    
    usage = response['usage']
    
    log_entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "model": model,
        "user_id": user_id,
        "feature": feature,
        "environment": environment,
        "input_tokens": usage['prompt_tokens'],
        "output_tokens": usage['completion_tokens'],
        "total_tokens": usage['total_tokens']
    }
    
    logging.info(log_entry)
    return response

# Usage example
track_llm_call(
    model="gpt-4",
    messages=[{"role": "user", "content": "Summarize this document"}],
    user_id="user_12345",
    feature="document_summary",
    environment="production"
)

Critical rule: Every service, microservice, and background job that calls an LLM must pass metadata through the call stack. If only the main application tags calls but internal services do not, attribution data becomes unreliable and governance impossible.

For teams using OpenTelemetry, trace context propagation handles much of this automatically. A trace started at the HTTP request boundary carries identifiers like user_id and session_id that are automatically linked to any LLM spans created within that trace.

Step 3: Calculate Cost Per Request Using Provider Pricing

Token counts become costs when multiplied by the provider’s pricing model. Most LLM providers charge separately for input tokens and output tokens, with prices varying by model.

Common pricing models as of early 2026 (verify current rates on each provider’s pricing page):

• OpenAI GPT-4o: $2.50 per 1M input tokens, $10.00 per 1M output tokens
• Anthropic Claude 3.5 Sonnet: $3.00 per 1M input tokens, $15.00 per 1M output tokens
• AWS Bedrock (Claude): $3.00 per 1M input tokens, $15.00 per 1M output tokens
• Azure OpenAI GPT-4: Pricing varies by region — verify at Azure OpenAI pricing

The cost calculation formula:

cost = (input_tokens / 1,000,000 × input_price) + (output_tokens / 1,000,000 × output_price)

Example calculation for a GPT-4o request with 500 input tokens and 1,000 output tokens:

cost = (500 / 1,000,000 × $2.50) + (1,000 / 1,000,000 × $10.00)
cost = $0.00125 + $0.01000
cost = $0.01125

Implement this in your tracking function:

PRICING = {
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gpt-4": {"input": 30.00, "output": 60.00},
    "claude-3-5-sonnet-20241022": {"input": 3.00, "output": 15.00}
}

def calculate_cost(model, input_tokens, output_tokens):
    if model not in PRICING:
        return None
    
    prices = PRICING[model]
    input_cost = (input_tokens / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    
    return input_cost + output_cost

# Example
cost = calculate_cost("gpt-4o", 500, 1000)
print(f"Request cost: ${cost:.5f}")

Hidden cost layer: If you are calling LLM APIs from AWS, GCP, or Azure and sending responses to a SaaS observability tool, you pay cloud egress fees of roughly $0.10 per GB on top of the LLM API cost. For high-volume LLM workloads, this can add 15–20% to the total bill.

Step 4: Route LLM Traffic Through a Proxy for Centralized Tracking

An LLM proxy sits between your application and the LLM provider, routing all model traffic through a single checkpoint. This creates a unified place for metadata tagging, token logging, budget enforcement, and model routing, regardless of which service made the call.

Popular open source LLM gateways include Portkey, LiteLLM, and AI Gateway. These proxies capture token usage automatically and expose it via Prometheus metrics or OpenTelemetry traces.

Example setup with LiteLLM proxy:

pip install litellm[proxy]

litellm --model gpt-4 --port 8000

Configure your application to call the proxy instead of OpenAI directly:

import openai

openai.api_base = "http://localhost:8000"
openai.api_key = "your-openai-key"

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain caching"}]
)

The proxy logs every request with token counts, model used, and response time. You can then query aggregated metrics from the proxy’s Prometheus endpoint:

curl http://localhost:8000/metrics

Tradeoff: Adding a proxy introduces an additional network hop (typically 20–50ms latency) and creates a dependency on the proxy’s availability. For most production workloads, the visibility gain outweighs the cost, but teams with strict latency SLAs should benchmark before deploying.

Step 5: Export Metrics to an Observability Platform

Token and cost data should land in a centralized observability backend where it can be queried, visualized, and alerted on alongside application traces and infrastructure metrics.

Option 1: Prometheus + Grafana

If you are using an LLM proxy that exposes a /metrics endpoint, configure Prometheus to scrape it:

scrape_configs:
  - job_name: 'llm-proxy'
    static_configs:
      - targets: ['localhost:8000']

Create a Grafana dashboard that visualizes cost by model, user, and feature:

# Total LLM cost per hour
sum(rate(llm_token_usage_sum{token_type="input"}[1h]) / 1000000 * 2.50) +
sum(rate(llm_token_usage_sum{token_type="output"}[1h]) / 1000000 * 10.00)

# Cost by user
sum by (user_id) (
  (rate(llm_token_usage_sum{token_type="input"}[24h]) / 1000000 * 2.50) +
  (rate(llm_token_usage_sum{token_type="output"}[24h]) / 1000000 * 10.00)
)

Option 2: OpenTelemetry

For teams using OpenTelemetry, LLM spans automatically include token counts as span attributes following the OpenTelemetry semantic conventions for GenAI:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("llm_call") as span:
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Explain pods"}]
    )
    
    span.set_attribute("gen_ai.usage.input_tokens", response['usage']['prompt_tokens'])
    span.set_attribute("gen_ai.usage.output_tokens", response['usage']['completion_tokens'])
    span.set_attribute("gen_ai.request.model", "gpt-4")
    span.set_attribute("user_id", "user_12345")

These spans can then be queried in any OpenTelemetry-compatible backend like Grafana Tempo, Jaeger, or CubeAPM.

Option 3: CubeAPM

CubeAPM supports OpenTelemetry ingestion and correlates LLM token spans with application traces, logs, and infrastructure monitoring in a single view. Because CubeAPM runs on your infrastructure, there are no cloud egress fees when sending telemetry data, and all LLM usage data remains within your VPC.

CubeAPM pricing is $0.15/GB of ingested telemetry data, with unlimited retention and no per-seat fees. For a team processing 10TB of telemetry monthly (including LLM spans, application traces, and logs), the total cost is $1,500/month.

Step 6: Set Up Cost Alerts and Budget Enforcement

Once cost data is flowing into your observability platform, create alerts that trigger when spending exceeds thresholds.

Example Prometheus alert rule for daily LLM spend over $100:

groups:
  - name: llm_cost_alerts
    rules:
      - alert: HighDailyLLMCost
        expr: |
          (
            (sum(rate(llm_token_usage_sum{token_type="input"}[24h]) * 86400) / 1000000 * 2.50) +
            (sum(rate(llm_token_usage_sum{token_type="output"}[24h]) * 86400) / 1000000 * 10.00)
          ) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Daily LLM costs exceed $100"
          description: "Estimated daily cost is {{ $value | humanize }}. Review usage patterns."

For per-user budget enforcement, combine cost tracking with rate limiting. Many LLM proxies support token-based rate limits keyed by user_id:

rate_limits:
  - key: user_id
    type: token
    limit: 100000
    window: 24h

This configuration allows each user 100,000 tokens per day. At GPT-4o pricing, that is roughly $0.25 to $1.00 per user per day depending on input/output ratio.

Real example: A SaaS platform running a coding assistant feature set a per-user daily limit of 50,000 tokens after discovering that 3% of users were generating 60% of total LLM costs through repeated prompt retries. The limit reduced monthly spend by 40% without impacting the majority of users.

Step 7: Track Cost Per Feature and Workflow

Aggregate dashboards show total LLM spend, but they do not identify which feature, workflow, or user segment drives the cost. Breakdown by feature requires tagging every LLM call with a feature or workflow identifier.

Example PromQL query to calculate cost by feature:

sum by (feature) (
  (rate(llm_token_usage_sum{token_type="input"}[24h]) / 1000000 * 2.50) +
  (rate(llm_token_usage_sum{token_type="output"}[24h]) / 1000000 * 10.00)
)

This query returns daily cost grouped by feature, making it clear whether the document summarization feature costs more than the chatbot or email assistant.

For multi-step AI workflows like agent loops, tool calls, and retrieval-augmented generation, trace-level cost attribution becomes critical. A single user action can trigger five LLM calls: one for intent classification, one for retrieval query generation, one for the main response, and two for follow up tool calls. Without span-level tracking, you see only the total cost, not which step drove it.

Tools built on OpenTelemetry propagate application trace context into LLM spans automatically, so you can see not just that a workflow cost $0.05 but exactly which LLM call, tool invocation, and retrieval step contributed to that cost.

Troubleshooting Common Issues

Issue: Token counts in logs do not match invoice totals

Cause: Cached tokens, reasoning tokens (o-series models), or batch API usage are not captured in standard usage objects.

Solution: OpenAI’s o-series models return completion_tokens_details with reasoning token counts. Anthropic’s prompt caching reduces input token charges but requires checking cache_creation_input_tokens and cache_read_input_tokens fields. Always log the full usage object, not just top-level fields.

Issue: Cost attribution is incomplete

Cause: Some services or background jobs are not passing user_id or feature tags.

Solution: Enforce tagging at the infrastructure level through the LLM proxy. Configure the proxy to reject requests that lack required metadata fields, forcing every caller to include attribution tags.

Issue: Prometheus queries return no data

Cause: Metric names do not match the conventions used by your LLM proxy or instrumentation library.

Solution: Check the /metrics endpoint of your proxy or OpenTelemetry collector to verify the exact metric names and labels. OpenTelemetry uses gen_ai_client_token_usage by default; custom instrumentation may use different names.

Issue: High egress costs when using SaaS observability tools

Cause: Sending LLM telemetry data from AWS, GCP, or Azure to a SaaS platform incurs cloud egress fees of roughly $0.10/GB.

Solution: Use a self-hosted observability platform like CubeAPM or Grafana that runs inside your VPC. This eliminates egress charges and keeps all telemetry data within your infrastructure.

LLM costs grow when token usage is invisible until the invoice arrives. Effective monitoring requires capturing token counts at request time, attaching business context, calculating cost per call, and surfacing attribution data in dashboards. Teams that implement this early avoid the cost surprises that appear once LLM features scale to production traffic.

For teams building LLM-powered applications, pairing token cost tracking with real user monitoring helps correlate LLM spend with actual user experience. If a feature costs $500/day in tokens but drives negligible engagement, that is a signal to rethink the implementation or remove the feature entirely.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions