As agentic systems mature, much of the work developers have them do is delegated, one subagent at a time. Many teams are also exploring and using a shared, multi-tenant Agentic Infrastructure, where cost isn't tied to a single owner. That's where Observability becomes key to monitoring infrastructure costs.

In this guide, you'll learn how observability works, then enable Claude Code's built-in telemetry, run a backend to collect it, and read the metrics, logs, and traces it emits. This will help you start tracking your team's costs more effectively, and it'll only improve as emitted telemetry matures and correlates more cleanly with your sessions.

Note: In its current state, the emitted telemetry from Claude Code provides no attributes that allow a reliable map to named sessions. Usage can be tracked using session_id, but it's still clumsy in a longer session mixing multiple prompts/skills.

This guide is scoped to Claude Code's telemetry for metrics, logs, and tracing. Note that it applies to Linux and macOS only.

Table of Contents

• Observability with OpenTelemetry

  • Telemetry Data

• Instrumenting Claude Code

  • Pull vs Push: How Telemetry Leaves an App

  • When to run a Collector

  • Prerequisites

  • Setup

• Exploring Telemetry

  • Metrics

  • Logs

  • Tracing

• Conclusion

• References

Observability with OpenTelemetry

Observability is the ability to answer questions about a system's runtime behavior from the data it emits. You do this without looking into its internals, attaching a debugger, reading source code, or manually trying to reproduce the behavior.

Here, a system's runtime behavior means what's externally visible. You can ask questions like:

• How much time 95% of all requests take.

• What the failure rate is across all requests received.

• What the cache hit ratio is for the in-memory cache the service uses.

• The difference between the configured and deployed replica counts for a service.

For Claude Code, the inaccessible inner workings are: how it manages context, how work is divided across multiple LLM calls, and how subagents are orchestrated. But you can read the emitted telemetry from Claude code to answer questions like:

• How much a dev or a team spent over a day, week, or month.

• How that usage is distributed across the supported models and effort levels.

• How many tokens are spent per dollar, and how much that varies by type (input, output, cacheRead, cacheCreation).

• When a compaction event kicked in, and by how much it reduced the context's token usage.

Only an instrumented system can answer these questions. Instrumentation is a piece of code added by the developer or built into the tool that records a program's runtime behavior and emits it as telemetry. For example, a measurement like this request spent 100 tokens.

The telemetry data helps avoid silent failures by providing a well-structured data trail of the system's behavior over time. For example, here's a chart from GitHub's August 17, 2026 outage postmortem explaining a rise in GitHub Actions runs over time from ~30M to ~110M:

Telemetry Data

The emitted telemetry consists of three categories of data:

• Metrics: Numeric measurements aggregated over a time window, like queries per second (QPS).

• Logs: A detailed record of an individual event, with a timestamp. For example, a compaction event in Claude Code.

• Traces: The path of one request through the system, split by time into nested requests. For example, an order-placement request on an ecommerce website, showing which internal services it calls to complete the request.

OpenTelemetry (https://opentelemetry.io/) is an observability framework that helps you generate, collect, and export these signals to a backend, which handles storage, querying, and visualization. Keeping that split makes it tool/vendor agnostic where the backend can be open-source or proprietary. It provides instrumentation SDKs for multiple programming languages.

Instrumenting Claude Code

Instrumentation code usually runs alongside the application, at the points best for measuring: a middleware with a request arriving or a response going out, or a token being counted.

It is plugged into the application in two ways:

1. A Shared Instrumentation Library: Applications using an open source framework can add an instrumentation library as a dependency and link it with the application's lifecycle methods. OpenTelemetry publishes instrumentation libraries for many frameworks (Example: Spring Framework).

2. Customized Implementation by Application Developers: Telemetry data emitters are added directly in the codebase using the OpenTelemetry SDK. For a closed-source product, the code is private, but it can still emit telemetry data compatible with OpenTelemetry standards.

Example: HTTP Instrumentation

Middleware in HTTP handling is a common codepath for all requests, which is why it's chosen for application-wide settings like authentication. The same reason makes it a good place for instrumentation code: wrap the handler once, measure every request.

The diagram shows where instrumentation sits relative to the request path.

• A client request passes through an HTTP middleware before reaching the application's request handler and downstream calls. The middleware uses SDK constructs to time the handler and record duration, status, and a count.

• The SDK then buffers those measurements and pushes OTLP to the Collector on a background thread, off the request path.

Claude Code is the second case, where both the core app and its instrumentation module are provided by Anthropic. The code that measures token usage, cost, and tool calls is built in and emits OpenTelemetry over OTLP. As a Claude Code user, you only need to enable the telemetry and prepare a backend to receive and analyze it.

Note: OTLP is a telemetry data delivery protocol designed in the scope of the OpenTelemetry project. This guide assumes end-to-end compatibility with OTLP. Wiring up incompatible telemetry or using incompatible backend components can give unpredictable results and is out of scope.

To collect, store, and read the telemetry, you'll need the following components:

• OpenTelemetry Collector: A vendor-agnostic implementation of how to receive, process, and export telemetry data. This is optional but great to have for personal setup. Must have for a production use case.

• Jaeger: Distributed tracing backend, released as an open source tool by Uber.

• Prometheus: Collects and stores metrics as Timeseries Data.

• Loki: Scalable Log Aggregation system by Grafana.

• Grafana: for UI visualization of logs, metrics, and traces.

Pull vs Push: How Telemetry Leaves an App

Telemetry leaves an application in one of two ways:

Pull (Scrape): The app exposes its current metrics on an HTTP endpoint, and a scraper (Prometheus) reads that endpoint periodically. Every running instance needs its own port, and the scraper must know all of those addresses ahead of time. The app is passive: the scraper drives the data movement. This suits long-lived processes with stable addresses.

In OpenTelemetry, pull-based scraping is configured with OTEL_METRICS_EXPORTER=prometheus (see the SDK environment variables for the accepted exporter values).

By convention, the app makes its metrics available at http://localhost:9464/metrics. This exporter handles metrics only.

Push (OTLP): The app sends its telemetry to a receiving endpoint on a regular interval. This is more flexible: any number of processes can push to the same endpoint with no registration ahead of time, so apps can start and stop freely even as their addresses change.

In OpenTelemetry, push is configured with OTEL_METRICS_EXPORTER=otlp, which ships over the OTLP exporter.

The push model can carry metrics, logs, and traces.

When to Run a Collector

A Collector is deployed when the existing stack isn't enough to handle the system's growing scale and complexity. It helps in the following ways:

• One export config: every producer points at the Collector instead of each carrying its own per-backend exporter setup.

• Outbound-only connections: enterprise networks often block the inbound connections a pull-based scraper needs. With a Collector, the app pushes out to it and it pushes onward, so nothing has to accept inbound traffic.

• Fan-out and translation: the Collector can convert telemetry into a vendor's storage format and send the same signal to more than one backend.

• Buffering: if a backend goes down, the Collector holds the data and retries, absorbing transient failures.

• Processing: it can apply processors before data leaves for storage, such as redacting attributes.

Note: This guide runs a push-based configuration with a Collector even though the setup is single-user. The stack has three backends that store and query data differently, and letting the Collector receive Claude Code's OTLP once and route each signal to the right place is simpler than wiring the app to all three. That's why the tool list calls it optional for personal use but a must-have in production: its value grows with the number of producers and backends.

Prerequisites

Each section in this guide links relevant docs, but you'll make faster progress if the tools and query languages specified below are already familiar:

You'll need

• Latest version of Claude Code

• Docker Engine (29.4.1+) with Docker Compose.

  • Capacity to run 5 Containers (4-core CPU, 8GB RAM, 15GB Disk)

• A Claude Plan that includes Enhanced Telemetry beta (Pro+ / Max)

• Git

• Ensure these ports are free at localhost:

  • 3000 (Grafana)

  • 3100 (Loki)

  • 4317/4318 (OTel Collector OTLP gRPC/HTTP)

  • 9090 (Prometheus)

  • 16686 (Jaeger UI)

Knowledge that would help

• Docker Compose: bringing up containers defined in compose file, reading container status and logs by docker compose ... commands.

• Bash and Config files: Setting environment variables, editing JSON files.

• PromQL (Prometheus): How to use counters/gauges, range selectors, and sum / increase / rate / by (label) grouping.

• Grafana: Explore a datasource, build dashboards with stat and timeseries panels. Panel transformations and Global variables.

• LogQL (Loki): stream selectors, logfmt, and label_format.

• Jaeger and tracing: the trace/span model (parent-child spans, span count, duration) and the Jaeger UI's tag search.

• Claude Code's execution model: sessions, subagents, skills, tools, and context compaction.

• Claude billing basics: tokens and the prompt-caching tiers.

Setup

The test observability stack is deployed using Docker Compose. For telemetry export to work, Claude Code must be able to reach Collector's OTLP endpoint which is localhost:4317 (gRPC) or localhost:4318(HTTP) when running on same machine. All backend services run in a container with their own Docker volume for persistence.

The diagram shows how the telemetry components we're using are linked.

• Apart from Claude Code, every component runs in a container managed by Docker Compose.

• Multiple instances of Claude Code running on any machine (host or cloud VM) should be able to export telemetry to the Collector as long as those machines have connectivity to it (ports 4317/4318 of the host running the Collector container are reachable).

Moving from top to bottom:

• Claude Code exports all three signals over OTLP to the Collector.

• The Collector then splits them by type, pushing traces to Jaeger and logs to Loki, while exposing metrics on port 8889 for Prometheus to scrape.

• Jaeger, Prometheus, and Loki each persist to their own Docker volume.

• Grafana queries all three as the single dashboard layer.

The goal here is a live stream of telemetry from Claude Code that gets stored in a backend and can be queried on demand, during a session or long after. Two things must be in place:

• Enable telemetry in Claude Code. The instrumentation is built in but emits nothing until telemetry is enabled and its OTLP exporter points at the Collector.

• Run the observability backend. The Collector processes each signal, forwarding it to Prometheus, Loki, and Jaeger for storing and serving queries.

You'll start the backend first, so the telemetry has somewhere to go.

Start the Observability Backend

Before enabling telemetry in Claude Code, ensure the stack is up to collect, process, and read the data. The code for the test observability backend lives in this Github repo.

The repo has the following structure:

.
├── README.md
└── compose
    ├── docker-compose.yml # Docker config for 5 containers in the observability stack. Applies pinned image versions, port mappings and named volumes for each service.
    ├── grafana
    │   └── provisioning
    │       ├── alerting
    │       ├── dashboards
    │       ├── datasources # datasources(Prometheus, Loki, Jaeger) and dashboards. Empty initially.
    │       └── plugins
    ├── jaeger-config.yaml # Jaeger v2, badger (local-file) storage for traces. Ties to the user: root TIP below.
    ├── loki-config.yaml # single-binary Loki, filesystem storage. Near default settings.
    ├── otel-collector-config.yaml # receive/process/export pipeline: OTLP in on 4317/4318, traces out to Jaeger, logs to Loki, metrics exposed on :8889 for Prometheus.
    └── prometheus.yml # a single scrape job against the Collector's :8889, 30s interval.

You'll only need docker compose command to start the containers. It reads docker-compose.yml and starts the containers, linking them to their respective config files.

Connectivity between containers:

All containers start within the same Docker network, which allows them to communicate using container names directly. For example, collector's exporter config uses container names:

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

Note that it doesn't contain Prometheus config, since Prometheus ends up scraping it from the collector as configured in prometheus.yml:

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: otel-collector
    static_configs:
      - targets: ["otel-collector:8889"]

Start the containers using Docker compose:

git clone https://github.com/ps-mir/otel-dev-stack.git
cd otel-dev-stack/compose
docker compose up -d

# Output
 ✔ Volume compose_loki_data                          Created                                                                                                                                          0.0s
 ✔ Volume compose_grafana_data                       Created                                                                                                                                          0.0s
 ✔ Volume compose_prometheus_data                    Created                                                                                                                                          0.0s
 ✔ Volume compose_jaeger_data                        Created                                                                                                                                          0.0s
 ✔ Network compose_default                           Created                                                                                                                                          0.1s
 ✔ Container compose-prometheus-1                    Started                                                                                                                                          4.1s
 ✔ Container compose-loki-1                          Started                                                                                                                                          4.2s
 ✔ Container compose-jaeger-1                        Started                                                                                                                                          4.3s
 ✔ Container compose-otel-collector-1                Started                                                                                                                                          3.3s
 ✔ Container compose-grafana-1                       Started                                                                                                                                          2.7s

Check container status:

# all five services should show "Up"
docker compose ps

# Output
NAME                       IMAGE                                              COMMAND                  SERVICE          CREATED         STATUS         PORTS
compose-grafana-1          grafana/grafana:13.2.0                            "/run.sh"                grafana          3 minutes ago   Up 3 minutes   0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp
compose-jaeger-1           cr.jaegertracing.io/jaegertracing/jaeger:2.20.0   "/go/bin/jaeger --co…"   jaeger           3 minutes ago   Up 3 minutes   0.0.0.0:16686->16686/tcp, [::]:16686->16686/tcp
compose-loki-1             grafana/loki:3.7.6                                "/usr/bin/loki -conf…"   loki             3 minutes ago   Up 3 minutes   0.0.0.0:3100->3100/tcp, [::]:3100->3100/tcp
compose-otel-collector-1   otel/opentelemetry-collector-contrib:0.159.0      "/otelcol-contrib --…"   otel-collector   3 minutes ago   Up 3 minutes   0.0.0.0:4317-4318->4317-4318/tcp, [::]:4317-4318->4317-4318/tcp, 55679/tcp
compose-prometheus-1       prom/prometheus:v3.11.2                           "/bin/prometheus --c…"   prometheus       3 minutes ago   Up 3 minutes   0.0.0.0:9090->9090/tcp, [::]:9090->9090/tcp

TIP: Jaeger runs as user: root (Compose file) to create the badger dir. Not doing so causes a failure: mkdir /badger/key: permission denied. Jaeger itself doesn't need root permission, but Docker volumes are owned by root:root on first mount.

Enable Telemetry

OpenTelemetry instrumentation, once added to an application, stays disabled until specific configuration enables it.

There are two ways to enable telemetry in Claude Code:

1. Environment Variables

Setting specific environment variables enables telemetry generation. Beyond the standard OTEL_* variables, Claude Code defines its own CLAUDE_CODE_* variables.

This guide uses the following settings:

# master switch: when unset or 0, Claude Code produces no telemetry at all
export CLAUDE_CODE_ENABLE_TELEMETRY=1

# opt into the beta enhanced-telemetry attributes and events (extra session and tool detail)
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1

# per-signal exporter selection; "otlp" ships the signal over OTLP.
# other accepted values are "console" (print locally), "prometheus" (metrics only), and "none" (drop the signal)
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp

# OTLP transport: "grpc" talks to the collector's 4317 port; "http/protobuf" would use 4318
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# one endpoint for all three signals: the collector's OTLP listener on the local machine
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# how often metrics are flushed, in milliseconds; the default is 60000 (60s),
# shortened here so a manual check sees fresh data without a long wait
export OTEL_METRIC_EXPORT_INTERVAL=5000

# emit cumulative counters instead of delta (see "Aggregation Temporality" below)
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative

Environment variables, however, are process-wide and can affect more than Claude Code. For example,

• Accidentally enabling instrumentation in other applications.

• Interfering with OpenTelemetry code/tests if you're developing your own instrumentation or working on any OpenTelemetry SDK.

2. Claude Code settings.json

OpenTelemetry defines Declarative Config, a YAML based configuration to enable telemetry, but Claude Code doesn't support it. But it lets you set the same env variables in ~/.claude/settings.json. That isn't declarative config, but it's better than shell environment variables because it applies only to Claude Code. An example:

{
  "effortLevel": "medium",
  "tui": "fullscreen",
  "env": {
     "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
     "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
     "OTEL_METRICS_EXPORTER": "otlp",
     "OTEL_LOGS_EXPORTER": "otlp",
     "OTEL_TRACES_EXPORTER": "otlp",
     "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
     "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4317",
     "OTEL_METRIC_EXPORT_INTERVAL": "5000",
     "OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE": "cumulative"
  }
}

Only the env block matters for telemetry. effortLevel and tui are unrelated settings you may already have. The variables match the annotated list above.

Aggregation Temporality

Prometheus counter type metrics only increase over time. Their raw value isn't useful, so you read them through per-second growth (rate()) or total growth over a time window (increase()).

Aggregation temporality decides what number a counter reports on each telemetry export: the change since the previous export(Delta), or the running total since the process started(Cumulative).

A short example. Say Claude Code spends tokens over four 5-second export intervals:

By default, Claude Code emits metrics with AggregationTemporality: Delta. This can be inspected and confirmed from the Collector's container logs using the command:

# Command only works from directory containing docker-compose.yml
docker compose logs otel-collector

Note: To enable detailed logs in the Collector, you need to add the debug exporter to the Collector config.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, debug]

Then restart the container:

# Command only works from directory containing docker-compose.yml
docker compose up -d --force-recreate otel-collector

Log output with AggregationTemporality: Delta:

otel-collector-1  | Descriptor:
otel-collector-1  |      -> Name: claude_code.active_time.total
otel-collector-1  |      -> Description: Total active time in seconds
otel-collector-1  |      -> Unit: s
otel-collector-1  |      -> DataType: Sum
otel-collector-1  |      -> IsMonotonic: true
otel-collector-1  |      -> AggregationTemporality: Delta <---
otel-collector-1  | NumberDataPoints #0

Delta doesn't work well with Prometheus functions like rate()/increase(), since they expect cumulative values.

Setting OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative switches the exported metrics from delta to cumulative temporality. It has been added to both the env and JSON config used in this guide.

As before, you need to restart the Collector container after any config change for it to take effect.

Exploring Telemetry

With telemetry flowing, you can start querying it. Metrics, logs, and traces each answer a different kind of question about Claude Code usage, so the three sections below are largely independent.

The data behind them comes from two places.

• The Metrics and Logs sections query whatever Claude Code usage has accumulated in the backend, so your panels will show your own sessions and the numbers won't match the screenshots. Give it a few real sessions before expecting much to show.

• The Tracing section instead walks a single deliberate run, a custom skill summarizing a batch of meetings, described in enough detail to follow along. You don't need to reproduce it.

Metrics

Metrics are the aggregate, time-windowed view of Claude Code usage. Example: total cost, token volume, and how each trends and breaks down by attributes like model, effort, and token type. Use them to watch spend and spot shifts in consumption.

Each metric is a numeric measurement recorded over time, a time series of timestamped values you can plot or aggregate. Claude Code's metrics are running totals (counters), so a query reports the change over a chosen window rather than the raw value. See Aggregation Temporality above for how that works.

Prometheus is the metrics backend we're using here. It scrapes the Collector, stores the series, and answers queries written in PromQL. Grafana reads the same data for dashboards at localhost:3000. The full list of metrics and their attributes is in the Claude Code monitoring docs.

Open Prometheus in your browser (localhost:9090), type claude in the query field, and you should see the supported metrics:

For each metric below, you'll explore it first using PromQL, and then use the same query to add it to the Grafana dashboard as a panel.

Total USD Spent

claude_code_cost_usage_USD_total represents cumulative usage cost, in USD, tracked per session. It's useful for controlling budgets and spotting sudden spikes in usage.

This is a client-side estimate based on token counts priced at Anthropic's per-model, per-type rates and accumulated. It's completely normal for it to exceed your Claude Code plan's subscription cost.

Note: This metric is more critical if you're paying per raw API call. A subscription gives you a usage allowance with increased but bounded rate limits.

Test the following query in Prometheus first (localhost:9090/query):

sum(increase(claude_code_cost_usage_USD_total[10m]))

increase(...[10m]) gives the counter's growth over the last 10 minutes. sum(...) with no by clause collapses the per-attribute series (model, effort, and others) into one number.

For a Grafana panel, swap the fixed [10m] window for the $__range built-in variable so the value follows the dashboard's time picker:

sum(increase(claude_code_cost_usage_USD_total[$__range]))

To add it as a panel, open Explore, select Prometheus as the data source, and run the query. The result depends on how much you've used Claude Code in the window.

On adding to the dashboard you should get more Panel Options. Select the Stat panel:

This panel needs to be added to the Grafana dashboard.

Total Token Usage

claude_code_token_usage_tokens_total is the cumulative token count, with the same counter shape as the USD cost metric. Read a raw series in Prometheus first to see which labels you can aggregate by. A single series looks like:

claude_code_token_usage_tokens_total{effort="high", exported_job="claude-code", instance="otel-collector:8889", job="otel-collector", model="claude-sonnet-5", otel_scope_name="com.anthropic.claude_code", otel_scope_version="2.1.252", query_source="auxiliary", session_id="e0b9795b-4da3-4171-8fa6-a2866bf44d86", terminal_type="ssh-session", type="cacheCreation"}    213730

The trailing number is the counter value. Key attributes you'll be working with are type, model, and effort. The Token Usage by Type chart below groups on type.

The window total is the same query shape as Total USD Spent, with the token counter:

sum(increase(claude_code_token_usage_tokens_total[$__range]))

Tokens Per USD

Unlike the previous two metrics, this is a derived figure, calculated over a time window as Total Tokens / Total Cost.

sum(increase(claude_code_token_usage_tokens_total[$__range])) / sum(increase(claude_code_cost_usage_USD_total[$__range]))

This one number collapses every attribute combination into a single value. Each distinct combination, for example model A at medium effort versus model B at high effort, is its own time series, and the query sums across all of them.

To analyze a specific combination, run the same ratio split by an attribute and compare.

sum by (model) (increase(claude_code_token_usage_tokens_total[$__range]))
  / sum by (model) (increase(claude_code_cost_usage_USD_total[$__range]))

Swap model for effort or type; the raw series above lists the rest of the labels.

Overall Result:

Stats Panel(6hr window): Total USD Spent ($4.28), Total Tokens Spent (3.02M), and Tokens Per USD (707k).

Token Usage by Type

You'll see how claude_code_token_usage_tokens_total changes over time, broken down by type. A single stat hides the shape, so use a time-series panel.

The type attribute has four values, which differ a lot in cost:

• cacheRead: tokens served from an existing cache entry. They dominate token spend in a long session, and are cheaper than the baseline rate.

• cacheCreation: tokens written into the prompt cache on the first prefix load. Costly.

• input: new, uncached prompt tokens.

• output: model-generated tokens.

To also see the total, use two queries: the per-type breakdown and the un-split total for reference.

# per-type breakdown
sum by (type) (increase(claude_code_token_usage_tokens_total[$__rate_interval]))
# total
sum(increase(claude_code_token_usage_tokens_total[$__rate_interval]))

__rate_interval is Grafana's per-step window for time-series panels, the counterpart to the __range used for the stat panels above.

The two queries running in Explore, before saving them as a panel.

After adding to the dashboard:

Each spike is a burst of Claude Code activity, the flat stretches are idle time. Hovering a point splits the total into the four types: here the total is about 1.0M tokens, of which cacheRead is about 925k (roughly 92%), the rest cacheCreation, output, and input.

TIP: Token consumption is dominated by cacheRead, which is also the cheapest type.

Token Usage by Model and Effort

This is the concrete version of the breakdown suggested under Tokens Per USD: which model and effort pairs are actually consuming tokens.

sum by (model, effort) (
  increase(claude_code_token_usage_tokens_total[$__rate_interval])
)

Here the token counter is grouped by its model and effort attributes. In this window every series is claude-sonnet-5 at either medium or high effort, and one burst of medium effort near 18:13 reaches about 2.6M tokens. The number of unique groupings depends on the cardinality of the chosen attributes.

Logs

A log record is a timestamped event with its full field set attached. You query logs when you want that specific event and the context around it: what happened, when, and with which values.

Metrics are the pre-aggregated form of the same activity. Anything that means counting, summing, or taking percentiles across many records belongs in a metric. If you're aggregating log output downstream, that data should have been a metric from the start.

Logs are the right tool for:

1. Per-event context: the full detail of one occurrence, not a rolled-up number.

2. Discrete or irregular events: a compaction firing, a session start, or an API error.

3. Post-incident forensics: reading raw records back while debugging after the fact.

4. Trace correlation: a log line carrying a trace and span ID drops you into the request it came from.

The log backend we're using here is Loki, queried with LogQL. Running logs through a backend like this buys you:

1. Structured fields: filter and compute on named keys instead of regex over text.

2. Field indexing: label lookups return without scanning every line.

3. Trace and span correlation: pivot from a log to its trace, or pull every log for one trace.

4. Time-bounded queries: each query is scoped to a window, keeping the scan cheap.

Grafana reads Loki for dashboards, the same as it does for Prometheus.

Compaction Event

Compaction is Claude Code trimming its own context when it grows too large. Each compaction emits a log event (event_name="compaction") carrying the token counts before and after (pre_tokens, post_tokens) and the span_id it happened under, so a query over those events shows how often it fires and how much it reclaims each time.

In Grafana Explore, select Loki as the data source and paste the LogQL below. It selects the compaction events, parses their fields with logfmt, and derives a per-event reduction percentage with label_format. The fields only exist once compaction has actually happened, so trigger a few first.

{service_name="claude-code"} | event_name="compaction"
  | logfmt
  | label_format reduction_pct=`{{ printf "%.1f" (mulf (divf (subf .pre_tokens .post_tokens) .pre_tokens) 100) }}`

Deriving reduction_pct for each record is fine here because it stays per-event. A running average across compactions would belong in a metric.

The label_format line adds a reduction_pct label. To show it as a table, switch the panel to Table view and add three Grafana transformations:

1. Extract fields from the labels object.

2. Filter fields by name to keep Time, pre_tokens, post_tokens, reduction_pct, and span_id.

3. Convert field type to turn pre_tokens, post_tokens, and reduction_pct into numbers.

Tracing

Tracing provides a detailed picture of the full path a request takes through an application, from start to completion. Some fundamental concepts behind tracing:

• Span: a timed operation representing a unit of work. Building block for traces. All trace data is recorded as a sequence of spans, and each has a type, its operation name:

  • claude_code.interaction: one prompt and everything Claude Code does to answer it. Normally the root span, so one interaction is effectively one trace.

  • claude_code.llm_request: a single model call inside an interaction.

  • claude_code.tool: a single tool call inside an interaction (Bash, Write, Agent, and so on).

• Trace: a tree of spans representing a request path from start to completion.

• Session: one Claude Code run, identified by session.id. It can result in many interactions and traces.

• Subagent: a nested Claude Code instance started by the Agent tool, running its own interactions.

Jaeger is the tracing backend used here. The Collector forwards spans to it over OTLP. Jaeger stores them and lets you search traces by service and span tags and inspect each one as a span tree. Everything below uses its UI at localhost:16686.

Generating Traces

To generate trace data, this guide will use a test prompt to spawn agents and prepare some text. This prompt was tested with Sonnet 5 at medium effort.

You can paste the prompt directly into Claude Code:

Spawn 4 subagents in parallel, one per topic below. Each subagent researches its topic from your own knowledge and returns a ~150-word summary with 3 key points. Do not have them read files or run commands.
Topics:
1. How TCP congestion control works
2. The CAP theorem
3. How DNS resolution works
4. What a Bloom filter is

Once all 4 return, combine the summaries into one markdown document and write it to summary.md

Note: Ask Claude Code for the session_id in the same session after the prompt finishes. This will be used to find related traces in Jaeger.

Trace By Session ID

The search filters by service = claude-code and the tag session.id=<id>. It returns 6 traces, all rooted at claude_code.interaction, with span counts from 1 to 20 and durations from about 1 second to 33 seconds.

The list alone doesn't say which trace did what. Going through them by hand, or scripting it against the trace API for a real session, gives the following:

A few observations:

• Half the traces are noise. Traces 1, 3, and 4 are single-span interactions with no model call or tool, an idle session being pinged. Trace 2 is a brief exchange. Only Traces 5 and 6 are the run.

• The parallel dispatch is a single interaction. Trace 5 fires all four Agent calls inside one claude_code.interaction. Their nested model calls (7 to 10 seconds each) overlap, so the interaction finishes in about 16 seconds despite roughly 35 seconds of combined subagent LLM time.

• Each subagent's model call is nested under its Agent span and carries an agent_id, so you can tell the four apart.

• agent_id is opaque. There's no agent.name or skill.name. The trace tells you four subagents ran and how long each took, not which topic each was given.

• Spans carry token counts but no USD cost. Each claude_code.llm_request has input_tokens, output_tokens, cache_read_tokens, and cache_creation_tokens, but no USD figure.

• The write is a separate, later interaction. Trace 6 has no Agent spans: one claude_code.llm_request of about 23 seconds produces the combined markdown, then a short Write. The two ScheduleWakeup spans are background coordination.

TIP: From the trace you get the four subagent calls, each with an agent_id, token counts, and timing, but no span says which topic a subagent was handed. In contrast, Metrics can provide attribution by model, effort, and skill.

CAUTION: user_prompt is redacted by default on interaction spans. OTEL_LOG_USER_PROMPTS=1 disables this and logs raw prompt text. Avoid enabling it in multi-user/tenant environments since it exposes prompt content to anyone with access to the telemetry backend.

Conclusion

This guide was an end-to-end walkthrough of observability in Claude Code, enabling its telemetry and collecting each of the three signals in a local backend for analysis.

Metrics give you the ability to dissect cumulative cost and usage readings by attribute over a chosen time period. That matters most in a shared or multi-tenant setup, where cost isn't tied to a single owner and someone still has to account for it.

Logs are records of individual events, useful for digging into exactly what changed during one, like compaction.

Traces show how one prompt expands into subagents and model calls, with timing and token counts on each. That's the starting point for debugging or tightening a complex or multi-agent prompt, though the spans don't yet record which prompt or skill drove a given call.

Some of this telemetry is behind the Enhanced Telemetry beta, so span names and attributes can still change, and gaps like per-call attribution may close as it matures. It's worth re-checking the monitoring docs as the surface settles.

References

• Claude Pricing: Prompt Caching

• Claude Code: Monitoring Usage

• Prometheus: Counter Metric Type

• Jaeger: Finding Traces

• Loki: LogQL Log Queries

• Grafana: Stat Panel

• OpenTelemetry Collector: Configuration

You might even know the basic definition, but knowing what OpenTelemetry is vs how it actually works are two different things.

By the end of this guide, you'll understand how OpenTelemetry works end-to-end, from the moment a request enters your application to the moment you can see it in your observability backend. You'll learn how traces, spans, context propagation, and exporters all fit together into one pipeline.

If you're completely new to OpenTelemetry, don't worry: the next section will get you up to speed before we go any further.

Table of Contents

• What is OpenTelemetry?

• How OpenTelemetry Works

  • Step 1: Instrument Your Application

  • Step 2: OpenTelemetry Creates Telemetry Signals

  • Step 3: Traces Follow Requests Through Your Application

  • Step 4: Context Propagation Connects Work Across Services

  • Step 5: The OpenTelemetry SDK Processes the Telemetry

  • Step 6: Exporters Send the Telemetry

  • Step 7: The OpenTelemetry Collector Receives and Processes the Data

  • Step 8: An Observability Backend Stores and Analyzes the Telemetry

• Putting the OpenTelemetry Flow Together

• Do You Need Every OpenTelemetry Component?

• Why Use OpenTelemetry?

• Conclusion

What is OpenTelemetry?

OpenTelemetry is an open-source, vendor-neutral observability framework. It gives you a standard way to instrument your application, generate telemetry data, and export that data to any observability backend of your choice.

Before OpenTelemetry, every monitoring tool had its own way of collecting data. If you used Datadog, you’ll have to instrument your app the Datadog way. If you switched to Jaeger, you started over. OpenTelemetry changed that by giving you one standard way to instrument your application, regardless of which backend you use

[!NOTE] OpenTelemetry is not a monitoring platform, dashboard, or data store. It provides the tools and standards for collecting and exporting telemetry from your application to an observability backend, where the data can be stored, queried, and analyzed.

The data OpenTelemetry collects is called telemetry. It's the information your application produces about itself as it runs, and it comes in three forms:

• Traces tell you how a request traveled through your system.

• Metrics give you numbers, like how many requests per second your app is handling, or how much memory it's using.

• Logs are timestamped records of specific events that happened inside your application.

How OpenTelemetry Works

When a request hits your application, a lot happens behind the scenes. OpenTelemetry's job is to capture all of that activity (the traces, metrics, and logs) and send them to the right place.

Here’s what it looks like:

Each stage has a specific job. Let's walk through them one by one.

Step 1: Instrument Your Application

Before OpenTelemetry can capture anything, your application needs to be instrumented. Instrumentation is simply the process of adding code that tells OpenTelemetry what to watch and what to record.

There are two ways to instrument your application: automatically or manually.

1. Automatic instrumentation

This is the easiest one to start with. You add a library to your project, and it instruments your application for you with no changes to your existing code.

For example, if you're running a Node.js Express app, you can add the OpenTelemetry auto-instrumentation package, and it will automatically start capturing incoming HTTP requests, outgoing calls, database queries, and more.

Here's what that setup looks like:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Once this runs before your app starts, OpenTelemetry begins capturing telemetry automatically. For a complete setup guide, see Getting started with OpenTelemetry in Node.js.

2. Manual instrumentation

Automatic instrumentation covers a lot, but it can't capture everything that happens inside your own code. If you want to track what happens inside a specific function, like how long it takes to process a payment or validate a user, you need to add that yourself.

Here's a simple example. Let’s say you have a function that processes an order:

function processOrder(orderId) {
  // processing logic
}

With manual instrumentation, you wrap it like this:

const { trace } = require('@opentelemetry/api');

const tracer = trace.getTracer('order-service');

function processOrder(orderId) {
  return tracer.startActiveSpan('processOrder', (span) => {
    // processing logic

    span.end();
  });
}

What happens is that you created a span. That span now records when processOrder started, when it ended, and how long it took. You'll learn more about spans in Step 3.

For the full manual instrumentation reference, see OpenTelemetry JavaScript instrumentation.

Step 2: OpenTelemetry Creates Telemetry Signals

Once your application is instrumented, OpenTelemetry starts producing telemetry data about what your application is doing. That data comes in three forms, called signals, which we've already briefly talked about: traces, metrics, and logs. Each signal answers a different kind of observability question.

Traces show you how a request moved through your system, which services it touched, and how long each step took. Metrics give you numbers over time, things like request rate, error rate, and memory usage. Logs are timestamped records of specific events that happened inside your application.

Here's a quick comparison:

You don't have to choose between them. In practice, you'll use all three together. A metric tells you something is wrong, a trace shows you where, and a log tells you exactly what happened.

Step 3: Traces Follow Requests Through Your Application

When a user sends a request to your application, that request usually touches multiple services before a response comes back. A trace is the complete record of that journey, from the moment the request enters your system to the moment it finishes.

A trace is actually made up of smaller units called spans. Each span represents one operation, like an API call, a database query, or a function execution, and together they give you the full picture of what happened.

Every trace gets a unique trace ID, and every span gets its own span ID. The trace ID is what links all the spans together. No matter how many services a request passes through, they all share the same trace ID, so you can follow the request from start to finish in your observability backend.

Here's a simple example. A user places an order, and the request flows through four services:

Each span has a start time and an end time, so you can see how long each operation took. If something slowed down or failed, you can pinpoint exactly where it happened just by looking at the spans.

Step 4: Context Propagation Connects Work Across Services

In Step 3, you saw how a single trace is made up of spans from multiple services. But here's a question you need to ask: how does OpenTelemetry know that a span in your payment service belongs to the same trace as a span in your order service?

Without something connecting them, each service would record its own spans independently. Your API gateway would see one operation, your order service would see another, and your payment service would see a third. They'd look like completely separate requests with no relationship to each other, which makes debugging across services nearly impossible.

That's where context propagation comes in. As a request moves from one service to another, OpenTelemetry attaches the trace context to it, typically as HTTP headers. That context carries the trace ID and the parent span ID, so every service that handles the request knows which trace it belongs to and where it sits in the chain.

Here's what that looks like in practice:

All three services share the same trace ID. That's what lets your observability backend connect the spans together into one complete trace.

OpenTelemetry doesn't invent its own rules for this. It follows the W3C Trace Context standard, a widely adopted specification that defines how trace context should be formatted and passed between services, so it works consistently across different languages, frameworks, and vendors.

The good news is that if you're using automatic instrumentation, context propagation happens automatically. OpenTelemetry handles the headers for you, so you don't have to think about it unless you're working with a custom transport or a non-standard setup.

Step 5: The OpenTelemetry SDK Processes the Telemetry

At this point, OpenTelemetry is capturing telemetry and keeping traces connected across services. But between the moment a span is created and the moment it leaves your application, something has to process it. That's the SDK's job.

When your instrumented code creates a span, it does that through the OpenTelemetry API. The API is what you interact with as a developer, things like trace.getTracer() and tracer.startActiveSpan(). But the API alone doesn't process or send anything. It needs the SDK behind it to actually do the work.

Once the SDK receives the telemetry, it runs it through a processor. The processor is responsible for things like batching spans together before sending them, adding extra attributes, or filtering out data you don't need. The most common one you'll see is the BatchSpanProcessor, which groups spans and exports them in batches rather than one at a time, making it more efficient in production.

Before the processor even runs, the SDK also handles sampling. Sampling lets you control how much telemetry you actually collect. In high-traffic applications, recording every single span would generate an enormous amount of data. With sampling, you can tell the SDK to only capture a percentage of traces, which keeps your costs and data volume manageable without losing visibility.

Once the processor is done, it hands the data to the exporter, which is what actually sends it to its destination. You'll see how that works in the next step.

Step 6: Exporters Send the Telemetry

The exporter's job is simple: take the telemetry the SDK prepared and send it to whatever destination you've configured.

OpenTelemetry uses OTLP, the OpenTelemetry Protocol, to transport telemetry data. It's a standard wire protocol designed specifically for transmitting traces, metrics, and logs, and it runs over either HTTP or gRPC.

[!NOTE] OTLP and OpenTelemetry are not the same thing. OpenTelemetry is the full framework, covering instrumentation, the SDK, the Collector, and more. OTLP is just the protocol it uses to transport data.

Although OTLP is the default, not every exporter uses it. Some exporters send data directly to specific backends in their own format, like Jaeger or Prometheus. So depending on your setup, you might use an OTLP exporter to send data to a Collector or backend, or a vendor-specific exporter to send it directly."

Step 7: The OpenTelemetry Collector Receives and Processes the Data

The OpenTelemetry Collector is a standalone service that sits between your application and your observability backend. It receives telemetry data, processes it, and forwards it to one or more destinations.

Using the Collector is common but not mandatory. You can configure your exporter to send data directly to your backend and skip the Collector entirely. But in most production setups, teams add a Collector because it gives them a central place to manage telemetry, without touching application code.

The Collector has three stages:

• Receivers accept incoming telemetry from your applications, typically over OTLP.

• Processors transform the data at the Collector level, things like batching spans, filtering out noise, or adding attributes before forwarding.

• Exporters send the processed data to your backend, or multiple backends if needed.

Here's a minimal Collector configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  otlphttp:
    endpoint: https://your-backend.com

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

This config accepts traces over gRPC, batches them for efficiency, and forwards them to an observability backend over HTTP.

The Collector can also receive data from multiple applications simultaneously and route it to one or more destinations. So instead of each application shipping telemetry directly to your backend, they all send it to the Collector, and the Collector handles the rest.

For the full configuration, see the OpenTelemetry Collector configuration docs.

Step 8: An Observability Backend Stores and Analyzes the Telemetry

This is where OpenTelemetry's job ends. Once your telemetry leaves the Collector, it arrives at your observability backend.

The backend is what stores your data, lets you query it, and gives you the dashboards and alerts you actually interact with day to day. OpenTelemetry doesn't provide any of that. It gets the data there, and the backend does the rest.

A few popular backends that support OpenTelemetry natively:

• Open-source: Jaeger, Prometheus, Grafana Tempo

• Commercial: Datadog, New Relic, Honeycomb, Dynatrace, Elastic, Lightstep, Grafana Cloud, Middleware.

Once your data is in the backend, you can search through traces to debug a slow request, build dashboards to monitor your application's health, and set up alerts when something goes wrong.

Putting the OpenTelemetry Flow Together

Let’s assume a user initiates a bank transfer on a mobile banking app. The request hits your API gateway, and because your application is instrumented, OpenTelemetry immediately starts capturing what’s happening. It creates a span for the incoming request and assigns it a trace ID.

As the request moves to the authentication service, context propagation carries that trace ID along in the request headers. The authentication service creates its own span and attaches it to the same trace. The same thing happens when the authentication service calls the transaction service, and when the transaction service hits the database to process the transfer. Four services and four spans, with one trace ID connecting them all.

Meanwhile, the SDK processes the telemetry in the background, runs the spans through the batch processor, and hands them to the exporter. The exporter packages everything into OTLP and sends it to the Collector, which applies your processing rules and forwards it to your observability backend.

Here’s a summary of every component involved in that flow:

From that single transfer request, you now have a complete trace in your backend. When you open your dashboard and search the trace ID, you'll see every service, every span, and every millisecond of that transaction laid out in front of you.

Do You Need Every OpenTelemetry Component?

To be honest, you don’t need every component to get started with OpenTelemetry. The pipeline you’ve seen throughout this article is the full setup, but not every team uses all of that.

1. Without the Collector: Your exporter sends telemetry directly to your backend with nothing in between. Works well for smaller projects or when you're just getting started.

2. With the Collector: The more common production setup. Teams add the Collector when they need more control, like routing telemetry to multiple backends, filtering sensitive data, or managing telemetry from dozens of services in one place.

The Collector is powerful, but it’s not mandatory. Start without it if your setup is simple, and add it when you actually need it.

Why Use OpenTelemetry?

You've now seen how every piece of OpenTelemetry fits together. Here's why it's worth using:

• Vendor-neutral instrumentation: Before OpenTelemetry, switching observability tools meant you had to re-instrument your entire application from scratch. With OpenTelemetry, you instrument once and switch backends freely.

• Consistent telemetry across services and languages: Your backend might be in Go, your microservices in Python, and your data pipeline in Java. OpenTelemetry has SDKs for all of them, and they all produce telemetry in the same format, so your whole stack speaks the same language.

• Correlated observability signals: Traces, metrics, and logs all flow through the same pipeline. So when something goes wrong, a spike in your metrics leads you to a trace, and that trace points you to the exact log entry where things broke down.

• It's becoming the industry standard: Most observability backends already support OpenTelemetry natively. Instead of learning a vendor-specific instrumentation approach every time you adopt a new tool, you learn OpenTelemetry once and it works everywhere.

Conclusion

OpenTelemetry gives you a standard way to instrument your application, collect telemetry, and ship it to any backend you choose, without locking you into a specific vendor or tool.

If you remember nothing else from this article, remember this: you instrument your application, generate telemetry, process it, export it, and analyze it. Every step has a component behind it, and now you know what each one does.

If you're just getting started, you don't need to set everything up at once. Start with instrumentation, get your telemetry flowing to a backend, and add the Collector when you need it. You can always add more as your needs grow.

If you found this helpful, I'd love to connect. You can find me on LinkedIn or X. Feel free to reach out if you have questions or just want to talk about observability and developer tooling.

When you run five agents in a swarm, each spawning its own tool calls and producing its own output, "read the log" stops being a strategy.

I built Claude Forge as an adversarial multi-agent coding framework on top of Claude Code. A typical run spawns a planner, an implementer, a reviewer, and a fixer. They evaluate each other's work and loop back when quality checks fail.

But when something went wrong, I had timestamps and text dumps but no way to see which agent was responsible, how long it actually took, or where the tokens went.

Jaeger fixed that. This article covers setting up Jaeger v2 with Docker, wiring it into a multi-agent system through OpenTelemetry, and what I learned along the way.

Table of Contents

• What Is Distributed Tracing?

• Why Jaeger v2?

• Prerequisites

• Installing Docker on Debian

• Setting Up Jaeger v2

• Setting Up Claude Forge Tracing

• Understanding the Span Model

• Instrumenting a Multi-Agent Swarm

• Viewing Traces in the Jaeger UI

• Lessons from the Trenches

• Environment Variable Reference

• Wrapping Up

What Is Distributed Tracing?

Distributed tracing tracks a single operation as it moves through multiple services. A span is one unit of work with a start time, end time, and key-value attributes. Spans nest into parent-child trees. One tree per operation is one trace.

Microservices people already know this pattern: follow an HTTP request from the gateway through auth, the database, and the cache. Same idea works for multi-agent AI. Follow one swarm invocation from the orchestrator through each subagent and its tool calls.

OpenTelemetry (OTel) is the standard. It gives you SDKs for creating spans and shipping them over OTLP. Jaeger receives that data and renders it as a searchable timeline.

Why Jaeger v2?

Jaeger started at Uber and graduated as a CNCF project in 2019. v1 hit end of life in December 2025. v2 is the current release, built on the OpenTelemetry Collector framework. Single binary: collector, query service, and UI. It speaks OTLP natively on port 4317 (gRPC) and 4318 (HTTP). There's no separate collector needed for local work.

One important difference from v1: configuration moved from CLI flags and environment variables to a YAML file. The old -e SPAN_STORAGE_TYPE=badger env vars are silently ignored in v2. The container starts fine but falls back to in-memory storage. I lost two days of traces before noticing. More on the correct setup below.

Prerequisites

• Docker installed and running.

• Claude Code installed.

• Python 3.8+ for the tracing hook.

• Claude Forge or another multi-agent system to instrument.

Installing Docker on Debian

Skip this if you already have Docker. macOS and Windows users can use Docker Desktop. On Debian:

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/debian \
  \((. /etc/os-release && echo "\)VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Ubuntu users: replace both linux/debian URLs with linux/ubuntu.

Setting Up Jaeger v2

Basic Run

For quick testing with no persistence:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/jaeger:2.17.0

Port 16686 is the UI. Port 4317 is OTLP/gRPC ingestion. Port 4318 is OTLP/HTTP. Remove the container and your traces are gone.

Persistent Storage with Badger

v2 reads configuration from a YAML file, not environment variables. Save this as ~/.local/share/jaeger/config.yaml:

service:
  extensions: [jaeger_storage, jaeger_query, healthcheckv2]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]
extensions:
  healthcheckv2:
    use_v2: true
    http: { endpoint: 0.0.0.0:13133 }
  jaeger_query:
    storage: { traces: main_store }
  jaeger_storage:
    backends:
      main_store:
        badger:
          directories: { keys: /badger/key, values: /badger/data }
          ephemeral: false
          ttl: { spans: 720h }
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch:
exporters:
  jaeger_storage_exporter:
    trace_storage: main_store

The Jaeger container runs as UID 10001. Docker named volumes default to root ownership. Without fixing permissions first, the container crash-loops with mkdir /badger/key: permission denied.

Pre-create the volume and fix ownership:

docker volume create jaeger-data

docker run --rm \
  -v jaeger-data:/badger \
  alpine sh -c "mkdir -p /badger/data /badger/key && chown -R 10001:10001 /badger"

Then run Jaeger with the config mounted in:

docker run -d --name jaeger \
  --restart unless-stopped \
  -v ~/.local/share/jaeger/config.yaml:/etc/jaeger/config.yaml:ro \
  -v jaeger-data:/badger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/jaeger:2.17.0 \
  --config /etc/jaeger/config.yaml

Verify persistence by running docker restart jaeger and confirming a previously recorded trace is still there. Hit http://localhost:16686 and you should see the UI.

Setting Up Claude Forge Tracing

Installing Claude Forge

Install it through the Claude Code plugin marketplace:

/plugin marketplace add hatmanstack/claude-forge
/plugin install forge@claude-forge
/reload-plugins

The install opens a TUI to confirm scope and settings. After reload, commands use the forge: prefix (for example, /forge:pipeline).

You can also clone the repo from GitHub.

Installing the Tracing Hook

From your target project directory, run the install script. For plugin installs:

cd your-project
forge-trace                # if you set up the alias from the README
# or, without the alias:
bash "$(find ~/.claude -path '*/forge*' -name install-tracing.sh 2>/dev/null | head -1)"

For clone installs:

cd your-project
bash /path/to/claude-forge/bin/install-tracing.sh

The script builds a dedicated venv at ~/.local/share/claude-forge/venv (prefers uv, falls back to python3 -m venv), installs the OpenTelemetry packages, copies the hook into place, merges hook entries into .claude/settings.local.json, and self-tests against the OTLP endpoint.

Pass --no-settings to skip the settings merge, or --uninstall to tear everything down.

Opting In

Add to your shell init and restart your terminal:

export CLAUDE_FORGE_TRACING=1

Restart Claude Code, run /pipeline, then check http://localhost:16686 for the claude-forge service.

Understanding the Span Model

Here's what the hierarchy looks like for a typical swarm run:

session: "implement login form with OAuth"        <- root span
├── subagent:planner
│   ├── tool:Write  (Phase-0.md)                  <- mutation spans (on by default)
│   ├── tool:Write  (Phase-1.md)
│   └── subagent_result:planner                   <- duration, token counts, output
├── subagent:implementer
│   ├── tool:Edit   (src/auth.ts)
│   ├── tool:Bash   (npm test)
│   ├── tool:Write  (src/oauth.ts)
│   └── subagent_result:implementer
├── subagent:reviewer
│   └── subagent_result:reviewer
└── session_complete                              <- session totals

The root span's name comes from the first line of your prompt. Find traces by what you asked for, not by a UUID.

Subagents get an anchor span on start and a result span on completion. The result carries duration, token counts, prompt, and output.

Three Tiers of Detail

Not all inner tool calls are equally interesting. Write, Edit, MultiEdit, and Bash are mutational: small in number, high signal. They tell you what actually changed. Read, Glob, Grep, and WebFetch are navigation: lots of them, mostly noise.

Tracing captures mutations by default. That middle ground turned out to be the right one. Before this change, you either saw nothing inside subagents or you saw 200+ spans per run.

Span Attributes

On session_complete: session.tokens.input, session.tokens.output, session.tokens.total, session.tokens.turns, session.duration_ms, user.prompt (first 2KB).

On subagent_result: agent.description, agent.prompt, agent.output, agent.duration_ms, agent.is_error, agent.tokens.input, agent.tokens.output.

On tool:*: tool.name, tool.input, tool.output, tool.duration_ms, tool.is_error.

Instrumenting a Multi-Agent Swarm

Hook Architecture

Claude Code has lifecycle hooks that fire scripts on specific events. Four matter here:

1. UserPromptSubmit (create the root span),

2. PreToolUse (start a span),

3. PostToolUse (end it with results), and

4. Stop (finalize the trace). Each hook gets a JSON payload on stdin and runs as a subprocess.

Sending Spans with OpenTelemetry

Here's some minimal Python to get a span into Jaeger:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "my-agent-system"})
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("agent-tracer")

with tracer.start_as_current_span("my-agent-task") as span:
    span.set_attribute("agent.name", "planner")
    span.set_attribute("agent.tokens.input", 1500)
    span.set_attribute("agent.tokens.output", 800)

Refresh localhost:16686, pick your service, click "Find Traces."

Correlating Pre and Post Events

You need to match each PreToolUse to its PostToolUse. Agent-type tool calls didn't include a tool_use_id in the payload, so I hashed the tool name and input instead. Pre and Post carry identical tool_input, so the hashes line up.

import hashlib, json

def correlation_key(tool_name: str, tool_input: dict) -> str:
    content = json.dumps({"tool": tool_name, "input": tool_input}, sort_keys=True)
    return hashlib.sha1(content.encode()).hexdigest()[:16]

State Across Invocations

Every hook call is a separate process. No shared memory. So I wrote span context to JSON files on Pre and read them back on Post:

/tmp/claude-forge-tracing/<session_id>/
├── _root.json              # trace ID, root span context
├── _session_start_ns.json  # timestamp for duration calculation
├── subagent_<hash>.json    # per-subagent span context
└── tool_<hash>.json        # per-tool span context

File names get sanitized against path traversal. _safe_name() strips everything outside [A-Za-z0-9._-] and falls back to a SHA1 slug.

Flushing Without Blocking

try:
    provider.force_flush(timeout_millis=1000)
except Exception:
    pass  # Never block the swarm

I tried 2000ms first and the swarm felt slow. 100ms lost spans on cold TLS connections. 1000ms worked. If Jaeger is down, the swarm keeps running regardless.

Viewing Traces in the Jaeger UI

Open http://localhost:16686. Pick claude-forge from the service dropdown. Click "Find Traces."

The trace search filters by operation name, tags, and time range. Since session spans take their name from your prompt, searching "login form" pulls up the runs where you asked for one.

The timeline view is where I spend most of my time. Every span is a horizontal bar, nested by parent-child relationships. I can see the planner took 12 seconds, the implementer 45, the reviewer 8. Click any bar to see token counts, prompts, outputs, error status.

Trace comparison puts two runs side by side. This is good for figuring out why one run succeeded and another did not.

Lessons from the Trenches

One trace per swarm, not per subagent: My first version wiped the root span's state file on every Stop event, so each subagent started a new trace. I changed Stop to mark a timestamp while preserving the root.

Use descriptions, not type names: Subagents all report their type as general-purpose. The description field is where the actual role lives.

Token attribution needs per-agent transcripts: Claude Code writes subagent transcripts to ~/.claude/projects/<project>/<session>/subagents/agent-*.jsonl. Match them via agent-*.meta.json.

Parse boolean env vars explicitly: bool("0") in Python is True. Use an allowlist: {"1", "true", "yes", "on"}.

Environment Variable Reference

Wrapping Up

Without visibility into the process, you're being inefficient with tokens and your time. Multi-agent swarms cost real money on every run. When an agent fails and retries, or when a reviewer rejects work that was close, you're paying for that blind.

Tracing gives you the map. You find out where the failure modes are. You find out which agents burn tokens going nowhere. A 45-second implementer run might have been 10 seconds with a better planner prompt. But you would never know that without seeing the breakdown.

Get observability in early. Jaeger and OpenTelemetry make it cheap to set up. Once you can see where things go wrong you can actually fix them.

Claude Forge tracing is on the main branch.

Instead of relying on vendor-specific agents or opaque SDKs, we will manually design traces, spans, and semantic attributes that capture the full lifecycle of an LLM-powered request.

Table of Contents

• Introduction

• Prerequisites and Technical Context

• Why LLM Observability Is Fundamentally Different

• Reference Architecture: A Traceable RAG Request

• Reference Architecture Explained

• Why This Design Is Better Than Simpler Alternatives

• LLM Models That Work Best for This Architecture

• OpenTelemetry Primer (LLM-Relevant Concepts Only)

• Designing LLM-Aware Spans

• FastAPI Example: End-to-End LLM Spans (Complete and Explained)

• Semantic Attributes: Best Practices for LLM Observability

• Evaluation Hooks Inside Traces

• Exporting and Visualizing Traces (Where This Fits with Vendor Tooling)

• Operational Patterns and Anti-Patterns

• Extending the System

• Conclusion

Introduction

Large Language Models (LLMs) are rapidly becoming a core component of modern software systems. Applications that once relied on deterministic APIs are now incorporating LLM-powered features such as conversational assistants, document summarization, intelligent search, and retrieval-augmented generation (RAG).

While these capabilities unlock new user experiences, they also introduce operational complexity that traditional monitoring approaches were never designed to handle.

Unlike conventional software services, LLM systems are probabilistic by nature. The same request may produce slightly different responses depending on factors such as prompt structure, model configuration, retrieval context, and sampling parameters such as temperature or top-p.

In addition, LLM workloads introduce entirely new operational dimensions such as token consumption, prompt construction latency, inference cost, context window limits, and response quality.

These factors mean that a request can appear technically successful from an infrastructure perspective while still producing an incorrect, hallucinated, or low-quality result.

Traditional observability tools typically focus on infrastructure-level signals such as latency, error rate, and throughput. While these metrics remain important, they are insufficient for understanding how an LLM application behaves in production.

Engineers must also understand what prompt was constructed, which documents were retrieved, how many tokens were consumed, which model configuration was used, and how the final response was evaluated. Without this visibility, debugging LLM behavior becomes extremely difficult and operational costs can quickly spiral out of control.

This is where LLM observability becomes essential. Observability for LLM systems extends beyond infrastructure monitoring. It captures the full lifecycle of an AI-driven request — from user input and context retrieval to prompt construction, model inference, post-processing, and quality evaluation.

When implemented correctly, observability allows teams to answer why the model generated a particular response, which retrieval results influenced the output, how much a request cost in terms of tokens, where latency occurred within the request pipeline, and whether the response passed basic quality or safety checks.

This article demonstrates how to implement end-to-end LLM observability in a FastAPI application using OpenTelemetry. Instead of relying on proprietary monitoring agents or opaque vendor SDKs, we take a code-first approach to instrumentation. By explicitly designing traces, spans, and semantic attributes, we gain precise control over how LLM interactions are observed and analyzed.

Throughout the guide, we will walk through a practical architecture for tracing a retrieval-augmented generation (RAG) workflow, where each stage of the request lifecycle is represented as a trace span. We will explore how to design meaningful span boundaries, capture prompt and model metadata safely, record token usage and cost signals, and attach evaluation results directly to traces.

The article also explains how this instrumentation can be exported to any OpenTelemetry-compatible backend such as Jaeger, Grafana Tempo, or LLM-specific platforms like Phoenix.

By the end of this guide, you will understand how to:

• Structure traces so that each user request maps to a single end-to-end LLM interaction

• Design span hierarchies that reflect the logical stages of an LLM pipeline

• Capture prompt metadata, model configuration, and token usage safely

• Attach evaluation and quality signals to traces for deeper analysis

• Export observability data to different backends without changing instrumentation

Most importantly, the goal of this article is not simply to demonstrate how to add telemetry to an application. Instead, it aims to show how to think about observability when building LLM-powered systems.

When LLM operations are treated as first-class components within a distributed system, traces become a powerful tool for debugging, optimization, cost management, and continuous improvement of model behavior.

Prerequisites and Technical Context

Before following this guide, you should be familiar with the Python programming language, basic web API concepts, and general microservice architecture. Below are some key tools and concepts used in this article.

FastAPI (Web Framework)

FastAPI is used as the primary web framework for the application. It is a modern Python framework designed for building high-performance APIs using standard Python type hints. FastAPI simplifies request validation, serialization, and API documentation while remaining lightweight and fast.

Large Language Models (LLMs)

Large Language Models (LLMs) are the computational core of the example system. An LLM is a model trained on vast amounts of text data to generate or transform language in ways that resemble human communication. In production environments, LLMs are commonly used for tasks such as conversational interfaces, summarization, and question answering.

Observability (Concept)

Observability is the overarching concept that connects all the technical pieces in this article. At a high level, observability refers to the ability to understand a system's internal behavior by examining the data it produces during execution. Rather than asking whether a system is simply "up" or "down," observability helps answer deeper questions about why a request behaved a certain way, where latency was introduced, or how different components interacted.

OpenTelemetry (Instrumentation Standard)

OpenTelemetry is the mechanism used to implement observability within the application. It is an open, vendor-neutral standard for generating telemetry data such as traces, metrics, and logs. By instrumenting key parts of the LLM workflow, we can observe how requests flow through the system, how long each step takes, and what contextual data influenced the final outcome. OpenTelemetry serves as the foundation for collecting this information in a consistent and portable way, independent of any specific monitoring backend.

Why LLM Observability Is Fundamentally Different

Traditional observability assumes deterministic behavior: the same input produces the same output. LLM systems violate this assumption. The same request can vary due to prompt template changes, retrieval differences, sampling parameters (temperature, top-p), model version upgrades, and context window truncation.​

As a result, teams need visibility into what the model saw, how it was configured, what it retrieved, how long it took, and how much it cost, all correlated to a single user request. Logs alone are insufficient, and metrics lack dimensionality. Distributed traces are the backbone of LLM observability.

Reference Architecture: A Traceable RAG Request

A typical FastAPI-based RAG service follows this flow:

Each step is observable, but only if we deliberately instrument it. The goal is one trace per user request, with child spans representing each logical LLM step.

Reference Architecture Explained

Client Sends a Request to /chat

The architecture begins when a client sends a request to the /chat endpoint. This request typically contains the user's query along with any session or conversation context required by the application.

Keeping the client interface minimal and well-defined is intentional: it ensures the backend receives a predictable input shape and prevents application-specific logic from leaking into downstream LLM processing.

From an observability perspective, this request marks the start of a single end-to-end trace, allowing every subsequent operation to be correlated back to the original user action.

FastAPI Validates Input and Authenticates the User

Once the request reaches the service, FastAPI performs schema validation and authentication. Validation guarantees that only well-formed inputs proceed through the pipeline, while authentication ensures that expensive LLM operations are only executed for authorized users.

Placing this step early reduces unnecessary computation and protects the system from abuse. It also improves trace quality by ensuring that all observed requests represent legitimate execution paths rather than malformed or rejected traffic.

Retriever Queries the Vector Database​

After validation, the system queries a vector database to retrieve documents relevant to the user's request. This retrieval step is the foundation of retrieval-augmented generation (RAG). By grounding the LLM in external knowledge, the system improves factual accuracy and reduces hallucinations.

Separating retrieval from generation allows teams to tune similarity thresholds, embedding models, and top-k values independently, and it makes it easier to diagnose whether poor responses are caused by bad retrieval or model behavior.

Prompt Is Assembled Using Retrieved Documents

With relevant documents in hand, the system constructs the final prompt that will be sent to the LLM. This step combines the user query, retrieved context, system instructions, and formatting rules into a single structured prompt.

Making prompt assembly an explicit stage enables prompt versioning, experimentation, and observability. It also provides a natural place to detect issues such as context window overflows or excessive prompt size before invoking the model.

LLM API Is Invoked

The LLM API call is the most expensive and non-deterministic operation in the pipeline, which is why it occurs only after all preparatory work is complete. At this stage, the model receives a fully constructed prompt and produces a response based on its configuration parameters.

This step is the primary focus of latency, cost, and reliability controls such as retries, timeouts, and circuit breakers. From an observability standpoint, this span becomes the anchor for token usage, cost attribution, and prompt-level debugging.

Response Is Post-Processed and Returned

After the LLM returns a response, the system performs post-processing before sending the result back to the client. This may include formatting, filtering, validation, or enrichment of the output. Post-processing acts as a final safeguard against malformed or low-quality responses and ensures consistency with application requirements. It also provides a clean boundary for attaching evaluation signals, such as response length, relevance scores, or truncation indicators, before the request completes.

Why This Design Is Better Than Simpler Alternatives

This architecture intentionally avoids coupling responsibilities together. Validation, retrieval, prompt construction, model execution, and response handling are all distinct steps. This separation makes the system easier to test, easier to observe, and easier to evolve. When something fails, engineers can identify where and why rather than treating the LLM as a black box.​

Compared to a monolithic "send user input directly to the LLM" approach, this design offers better correctness, lower cost, and higher resilience. It also aligns naturally with distributed tracing, since each block maps cleanly to a trace span with a clear semantic purpose. As the system grows, additional features such as caching, fallback models, or policy enforcement can be added without destabilizing the entire flow.​

Most importantly, this architecture treats the LLM as one component in a larger system, not the system itself. That mindset is essential for building reliable production applications.

LLM Models That Work Best for This Architecture

This architecture is model-agnostic, but certain model characteristics work particularly well with retrieval-augmented workflows.

Models with strong instruction-following and reasoning capabilities tend to perform best, especially when prompts include structured context from retrieved documents. General-purpose models such as GPT-4-class systems perform well when accuracy and reasoning depth are critical.

For lower-latency or cost-sensitive use cases, smaller instruction-tuned models can be effective when paired with high-quality retrieval. Open-source models such as LLaMA-derived or Mistral-based systems also fit well into this architecture, particularly when deployed behind a private inference endpoint.​

The key requirement is not the model itself, but how it is used. Models that can reliably ground their responses in provided context, respect system instructions, and produce stable outputs under varying prompts integrate most cleanly into this design. Because retrieval and prompt construction are explicit stages, models can be swapped or compared without changing the overall system structure.

OpenTelemetry Primer (LLM-Relevant Concepts Only)

OpenTelemetry defines three core types of telemetry data: traces, metrics, and logs. For LLM systems, traces are the most important. To make them useful, you need to understand a few building blocks:

• a trace represents a single end-to-end request

• a span is a timed operation within that trace

• attributes are key–value metadata attached to spans

• events are time-stamped annotations

• context propagation ensures child spans attach to the correct parent.

FastAPI’s async nature makes correct context propagation essential, but OpenTelemetry’s Python SDK handles this as long as spans are created correctly.

With those concepts in place, the next step is to wire OpenTelemetry into the app. Start by configuring the OpenTelemetry SDK in FastAPI: define a TracerProvider, attach a Resource (service name and environment), configure an exporter (Jaeger, Tempo, Phoenix, and so on), and enable FastAPI auto-instrumentation.

Designing LLM-Aware Spans

Span Taxonomy

A clean span hierarchy is critical. In this guide, a single http.request span (usually auto-generated) acts as the root, and it contains child spans such as rag.retrieval, rag.prompt.build, llm.call, llm.postprocess, and, optionally, llm.eval. Each of these spans represents a logical unit of work rather than an implementation detail.

Span Boundaries

Getting span boundaries right is just as important as picking the right span names. Avoid extremes like wrapping the entire LLM workflow in one giant span, creating a separate span for every token, or dumping all data into logs.

Instead, aim for a few coarse-grained spans that each represent a meaningful step in the request, enrich them with well-chosen attributes, and use events to mark important milestones within a span rather than splitting everything into smaller spans.

Instrumenting the LLM Call

When instrumenting the LLM call, treat it as the most critical span in the trace. Whether you are calling OpenAI, Anthropic, or another provider, start the span immediately before the API request and end it only after the full response (or stream) is complete.

Within that span, capture retries, timeouts, and errors so it becomes the central place for latency analysis, cost attribution, and prompt debugging.

For streaming responses, you can emit events for each chunk to track progress, but avoid creating separate child spans unless you truly need fine-grained timing.

FastAPI Example: End-to-End LLM Spans (Complete and Explained)

from fastapi import FastAPI, Request
from opentelemetry import trace
from opentelemetry.trace import Tracer
from typing import List
import asyncio
import hashlib

# Obtain a tracer instance from OpenTelemetry.
# All spans created with this tracer will be part of the same distributed
# tracing system and exported to the configured backend.
tracer: Tracer = trace.get_tracer(__name__)

# Initialize the FastAPI application.
app = FastAPI()

# Helper functions used by the observable endpoint
async def retrieve_documents(query: str) -> List[str]:
    """
    Simulate document retrieval (e.g., vector search or knowledge base lookup).
    This function represents the retrieval stage in a RAG pipeline.
    In a real system, this might query a vector database or search index.
    """
    await asyncio.sleep(0.05)  # Simulate I/O latency
    return [
        "FastAPI enables high-performance async APIs.",
        "OpenTelemetry provides vendor-neutral observability.",
        "LLM observability requires tracing prompts and tokens.",
    ]


def build_prompt(query: str, documents: List[str]) -> str:
    """
    Construct the final prompt from retrieved documents and the user query.
    Prompt construction is kept separate so it can be observed or modified
    independently if needed (for example, to measure prompt assembly latency).
    """
    context = "\n".join(documents)
    return f"""
Context:
{context}

Question:
{query}
"""


class LLMResponse:
    """
    Minimal abstraction for an LLM response.
    This keeps the example self-contained while still allowing us to attach
    token usage and other metadata for observability.
    """

    def __init__(self, text: str, prompt_tokens: int, completion_tokens: int):
        self.text = text
        self.prompt_tokens = prompt_tokens
        self.completion_tokens = completion_token
    
    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

async def call_llm(prompt: str) -> LLMResponse:
    """
    Simulate an LLM API call.
    In a real implementation, this would call OpenAI, Anthropic, or another
    provider. The artificial delay represents model latency.
    """
    await asyncio.sleep(0.2)  # Simulate inference time
    response_text = "FastAPI and OpenTelemetry enable end-to-end LLM observability."
    # Token count is approximated here for demonstration purposes.
    prompt_tokens = len(prompt.split())
    completion_tokens = len(response_text.split())
    return LLMResponse(response_text, prompt_tokens, completion_tokens)


def summarize_response(response: LLMResponse) -> str:
    """
    Example post-processing step.
    Post-processing is separated into its own phase so any additional latency
    or errors are not incorrectly attributed to the LLM itself.
    """
    return response.text


# Observable FastAPI endpoint
@app.post("/query")
async def rag_query(request: Request, query: str):
    """
    Handle a single RAG-style request with explicit OpenTelemetry spans.
    This endpoint demonstrates how to create one trace per request, with child
    spans for retrieval, LLM invocation, and post-processing.
    """

    # Create a top-level span for the HTTP request.
    # Even if FastAPI auto-instrumentation is enabled, defining this explicitly
    # allows us to attach domain-specific metadata.
    with tracer.start_as_current_span("http.request") as http_span:
        http_span.set_attribute("http.method", "POST")
        http_span.set_attribute("http.route", "/query")

        # Retrieval phase
        # This span isolates the retrieval step so that relevance issues can be
        # debugged independently of LLM behavior.
        with tracer.start_as_current_span("rag.retrieval") as retrieval_span:
            retrieval_span.set_attribute("rag.top_k", 5)
            retrieval_span.set_attribute("rag.similarity_threshold", 0.8)
            documents = await retrieve_documents(query)

            # Record how many documents were returned.
            # This is a key signal when diagnosing hallucinations
            # or missing context in the final response.
            retrieval_span.set_attribute(
                "rag.documents_returned",
                len(documents),
            )

        # LLM invocation phase
        # This span wraps the actual LLM call and is the primary anchor for
        # latency, cost, and prompt-related analysis.
        with tracer.start_as_current_span("llm.call") as llm_span:
            llm_span.set_attribute("llm.provider", "example")
            llm_span.set_attribute("llm.model", "example-llm")
            llm_span.set_attribute("llm.temperature", 0.7)
            llm_span.set_attribute("llm.prompt_template_id", "rag_v1")

            # Build the final prompt using retrieved context.
            # The raw prompt is intentionally not stored as a span attribute.
            prompt = build_prompt(query, documents)
            
            # Prompt metadata
            prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
            llm_span.set_attribute("llm.prompt_hash", prompt_hash)
            llm_span.set_attribute("llm.prompt_length", len(prompt))

            response = await call_llm(prompt)

            # Hash the response instead of storing raw text.
            # This allows correlation across traces without exposing content.
            response_hash = hashlib.sha256(
                response.text.encode()
            ).hexdigest()
            llm_span.set_attribute("llm.response_hash", response_hash)

            # Record token usage to enable cost attribution
            # and capacity planning.
            llm_span.set_attribute("llm.usage.prompt_tokens", response.prompt_tokens)
            llm_span.set_attribute("llm.usage.completion_tokens", response.completion_tokens)
            llm_span.set_attribute("llm.usage.total_tokens", response.total_tokens)
            
            # example price per token
            estimated_cost = response.total_tokens * 0.000002
            llm_span.set_attribute("llm.cost_estimated_usd", estimated_cost)

        # Post-processing phase
        # Any transformation after the LLM response is captured here,
        # ensuring inference latency is not overstated.
        with tracer.start_as_current_span("llm.postprocess") as post_span:
            summary = summarize_response(response)
            post_span.set_attribute(
                "llm.summary_length",
                len(summary),
            )

    # Return the final response to the client.
    # All spans above belong to the same distributed trace.
    return {"summary": summary}

Before examining the full code example, it helps to understand how the instrumentation relates to the observability principles described earlier in this article.

The goal of the example is not simply to show how to create spans, but to demonstrate how a single user request can be represented as a structured trace containing meaningful metadata about each stage of the LLM pipeline.

At a high level, the code follows three key design ideas:

1. One trace per user request

2. One span per logical LLM workflow stage

3. Semantic attributes attached to spans for debugging, cost tracking, and analysis

Each of these concepts directly corresponds to the observability practices discussed earlier.

Top-Level Request Span

The FastAPI endpoint begins by creating a top-level span called http.request. This span represents the entire lifecycle of the incoming request and serves as the root span for the trace.

with tracer.start_as_current_span("http.request") as http_span:

Although FastAPI can generate HTTP spans automatically through OpenTelemetry auto-instrumentation, explicitly creating this span allows the application to attach domain-specific metadata such as route names or user identifiers.

Attributes such as the HTTP method and route are attached here:

http_span.set_attribute("http.method", "POST")
http_span.set_attribute("http.route", "/query")

This ensures that every trace can be easily filtered by endpoint when analyzing production traffic.

Retrieval Span

The next span captures the retrieval phase of the RAG pipeline:

with tracer.start_as_current_span("rag.retrieval") as retrieval_span:

This span isolates the vector search or knowledge retrieval step from the rest of the pipeline. If users report irrelevant answers, engineers can inspect this span to determine whether the issue originates from poor retrieval results rather than model behavior.

Several semantic attributes are attached here:

• rag.top_k – number of documents requested

• rag.similarity_threshold – similarity cutoff used for filtering results

• rag.documents_returned – number of documents actually retrieved

These attributes align with the RAG observability signals discussed in the earlier section of the article.

LLM Invocation Span

The most important span in the trace is the llm.call span, which wraps the actual model invocation.

with tracer.start_as_current_span("llm.call") as llm_span:

This span captures the latency, configuration, and token usage associated with the LLM request. In production systems, it becomes the primary location for analyzing model behavior and cost.

Key attributes recorded in this span include:

• llm.provider – the model provider (OpenAI, Anthropic, etc.)

• llm.model – the specific model version

• llm.temperature – sampling parameter controlling response randomness

• llm.prompt_template_id – identifier for the prompt template used

These attributes make it possible to correlate changes in model configuration with downstream quality or cost changes.

Prompt Handling and Privacy

Instead of storing the full prompt or response text directly in the trace, the example demonstrates a safer practice: hashing sensitive data.

response_hash = hashlib.sha256(response.text.encode()).hexdigest()

The resulting hash is stored as a span attribute:

llm_span.set_attribute("llm.response_hash", response_hash)

This approach allows engineers to correlate repeated responses across traces without exposing potentially sensitive content in observability systems.

Token Usage Tracking

The llm.call span also records token usage:

llm_span.set_attribute(
    "llm.usage.total_tokens",
    response.total_tokens
)

Capturing token usage at the span level is critical for monitoring cost and efficiency, since token consumption directly determines billing for most LLM providers.

Post-Processing Span

Finally, the example includes a llm.postprocess span:

with tracer.start_as_current_span("llm.postprocess") as post_span:

This span represents any transformation applied after the model generates its response. Separating post-processing from the LLM call ensures that additional latency — such as formatting, filtering, or validation — is not incorrectly attributed to the model itself.

An attribute such as response length is recorded here:

post_span.set_attribute("llm.summary_length", len(summary))

This can be useful when diagnosing issues such as unexpectedly short or truncated outputs.

How the Spans Form a Complete Trace

When the request finishes, all spans belong to the same distributed trace:

http.request
 ├── rag.retrieval
 ├── llm.call
 └── llm.postprocess

This hierarchy reflects the logical workflow of a retrieval-augmented LLM system. Because each span contains structured metadata, engineers can quickly answer questions such as:

• Was the latency caused by retrieval or model inference?

• How many documents influenced the prompt?

• Which model configuration produced the response?

• How many tokens were consumed?

• Was the response post-processed or truncated?

This structured trace design is what transforms observability from simple monitoring into a practical debugging and optimization tool for LLM systems.

Semantic Attributes: Best Practices for LLM Observability

The goal is not to capture every possible detail, but to record the minimal set of stable, high-signal attributes that enable effective debugging, cost control, and quality analysis in production. Poor attribute design leads to noisy traces, privacy risks, and dashboards that are impossible to reason about.

Prompt, Response, and Model Metadata​

Storing raw prompts is often unsafe and expensive, so it is better to record minimal, structured metadata instead. In practice, this means attaching a stable template identifier with llm.prompt_template_id, a hashed version of the final prompt using llm.prompt_hash (to avoid storing raw text), and a size indicator such as llm.prompt_length, which captures the number of tokens or characters.

You should also always record key inference parameters: llm.provider (for example, "openai" or "anthropic"), llm.model (for example, "gpt-4.1"), llm.temperature and llm.top_p (sampling parameters), llm.max_tokens (the maximum tokens allowed), and llm.stream to indicate whether streaming was enabled, while staying within your organization’s privacy and compliance requirements.

with tracer.start_as_current_span("llm.call") as llm_span:
            llm_span.set_attribute("llm.provider", "example")
            llm_span.set_attribute("llm.model", "example-llm")
            llm_span.set_attribute("llm.temperature", 0.7)
            llm_span.set_attribute("llm.top_p", 0.9)
            llm_span.set_attribute("llm.max_tokens", 512)
            llm_span.set_attribute("llm.stream", False)
            llm_span.set_attribute("llm.prompt_template_id", "rag_v1")

            # Build the final prompt using retrieved context.
            # The raw prompt is intentionally not stored as a span attribute.
            prompt = build_prompt(query, documents)
            
            # Prompt metadata
            prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
            llm_span.set_attribute("llm.prompt_hash", prompt_hash)
            llm_span.set_attribute("llm.prompt_length", len(prompt))

Token Usage and Cost (Why This Matters in Practice)

Token usage is one of the most common blind spots in LLM systems. Many teams monitor latency and error rates but discover runaway costs only after invoices spike. Because token consumption varies significantly by prompt structure, retrieved context, and model configuration, it must be captured explicitly at the span level.​

The most important practice is to record token usage at the end of the LLM span, once the model has completed inference. This ensures that the values reflect the full request rather than partial or streamed output.

At minimum, capture the attributes:​llm.usage.prompt_tokens ,llm.usage.completion_tokens and llm.usage.total_tokens​.

def __init__(self, text: str, prompt_tokens: int, completion_tokens: int):
        self.text = text
        self.prompt_tokens = prompt_tokens
        self.completion_tokens = completion_token
    
    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

async def call_llm(prompt: str) -> LLMResponse:
    """
    Simulate an LLM API call.
    In a real implementation, this would call OpenAI, Anthropic, or another
    provider. The artificial delay represents model latency.
    """
    await asyncio.sleep(0.2)  # Simulate inference time
    response_text = "FastAPI and OpenTelemetry enable end-to-end LLM observability."
    # Token count is approximated here for demonstration purposes.
    prompt_tokens = len(prompt.split())
    completion_tokens = len(response_text.split())
    return LLMResponse(response_text, prompt_tokens, completion_tokens)

These values allow you to distinguish between requests that are expensive because of large prompts (often caused by excessive retrieval or poor prompt construction) versus those that are expensive because of long model-generated outputs.

*Where possible, also attach an estimated cost:*​ llm.cost_estimated_usd​

    # example price per token
    estimated_cost = response.total_tokens * 0.000002
    llm_span.set_attribute("llm.cost_estimated_usd", estimated_cost)

This value is typically derived by multiplying token counts by the model's published pricing. Even if the estimate is approximate, it enables powerful analysis. For example, you can identify which endpoints, prompt templates, or user flows are responsible for the highest cumulative cost, rather than relying on coarse, account-level billing dashboards.

Once spans carry the right attributes, the next step is to connect them to output quality, not just system health.

Evaluation Hooks Inside Traces

This section describes an additional pattern you can layer on top of the core instrumentation in this guide. It is optional and not implemented in the sample code, but it shows how to attach quality signals directly to your traces.

Observability is not just about whether the system stayed up, it is also about whether the model produced a useful answer. Evaluation hooks inside traces let you attach lightweight quality signals directly to the same spans you use for latency and cost.

Inline evaluations are the simplest approach. You can run quick checks synchronously and record the results as span attributes, such as llm.eval.passed for a simple boolean check, llm.eval.relevance_score for an optional numerical score, or flags like llm.eval.hallucination_detected and llm.eval.refusal_detected. These attributes travel with the trace, so you can filter and aggregate on them in your observability backend just like any other field.

For higher accuracy, you can introduce model-based evaluation as a separate step. In this pattern, an evaluator LLM runs asynchronously on the original prompt and response, and its work is captured in a child span (for example, llm.eval) that shares the same trace ID as the main llm.call span. You then attach scores such as relevance, faithfulness, or toxicity to that evaluation span.

Because the evaluation span shares the same trace ID, you can correlate quality regressions with changes in prompts or retrieval.

Exporting and Visualizing Traces (Where This Fits with Vendor Tooling)

This code-first observability design is vendor-agnostic. Once traces are emitted using OpenTelemetry, they can be exported to different backends without changing instrumentation.

General-purpose tracing systems like Jaeger and Grafana Tempo help engineers debug latency, errors, and request flow across retrieval, prompting, and model calls, answering how the system behaved. LLM-focused platforms such as Arize Phoenix use the same data but add model-specific insights like prompt clustering, token analysis, and quality correlation.

Because instrumentation stays OpenTelemetry-native, you maintain full control over attributes and trace structure while still using vendor dashboards, and you can switch backends as your needs evolve without touching the application code.

Operational Patterns and Anti-Patterns

Effective LLM observability requires disciplined practices. High-volume systems should sample traces to limit overhead, and prompts or responses should be hashed by default to reduce storage and privacy risk. Traces must be treated as production data, with proper access control and retention policies.

Common pitfalls include relying only on vendor SDK traces, logging prompts without trace correlation, or ignoring evaluation signals. These issues fragment visibility and hide quality regressions, especially when observability focuses only on agents instead of full application context.

Extending the System

Once traces are reliable, they support advanced capabilities. Metrics like p95 latency can be derived from spans, logs can be linked using trace IDs, and historical traces can power offline evaluation or prompt testing.​

By following OpenTelemetry conventions, the observability stack also stays aligned with emerging LLM semantic standards, keeping the system flexible and future-proof.

Conclusion

End-to-end LLM observability is not achieved by installing another agent. It is achieved through intentional span design, meaningful semantic attributes, and, where needed, lightweight evaluation hooks.​

By treating LLM calls as first-class operations within distributed traces, you gain faster debugging, controlled costs, safer deployments, and measurable quality improvements. The backend — Jaeger, Tempo, Phoenix — is interchangeable. The instrumentation strategy is not.​

A well-designed trace is the most valuable artifact in a production LLM system.

This is the reality of debugging modern applications. Traditional monitoring wasn't built for containers that live for seconds, services that shift across nodes, or network paths that change constantly.

eBPF changes this. It lets you see inside the kernel itself, watching every system call, every network packet, and every process execution – without modifying a single line of code.

In this tutorial, you will trace a real Kubernetes application using eBPF-powered tools. You’ll learn fundamentals that apply across the entire modern observability ecosystem, with gadgets from the Inspektor Gadget ecosystem.

By the end, you’ll be able to:

• Trace requests as they move through your Kubernetes pods

• Observe behavior at the kernel and syscall level

• Debug failures that logs and metrics simply can’t explain

Prerequisites

Knowledge requirements:

• Basic Kubernetes concepts: pods, deployments, services, namespaces

• Familiarity with kubectl: get, describe, logs, exec

• Container basics

• Basic Linux concepts: processes, system calls

Technical requirements:

• Kubernetes cluster (local or cloud-based)

• kubectl installed and configured

• Cluster admin permissions

• Linux kernel 5.10+ (most managed services have this)

Table of Contents

• Understanding eBPF Observability

• How eBPF Tracing Works (Without Getting Lost in the Kernel)

• How to Set Up Your Environment

• How to Trace Your First Request: Hands-On Tutorial

• How to Interpret Traces

• Real-World Debugging Scenarios

• Advanced Tracing Insights

• Best Practices and Production Considerations

• Next Steps and Resources

Understanding eBPF Observability

eBPF (extended Berkeley Packet Filter) is a technology that allows you to run custom programs inside the Linux kernel without changing kernel code or loading kernel modules.

The Linux kernel is the control center of your operating system. Historically, if you wanted to observe low-level activity (like network packets, system calls, or file operations), you had to rely on kernel changes or kernel modules. Both approaches were fragile, difficult to maintain, and carried real stability and security risks.

eBPF shifts how we approach observability. It provides a safe, sandboxed environment where you can run observability programs directly in the kernel with built-in safety checks that prevent crashes or security vulnerabilities.

Why does this matter for observability?

In traditional observability, you instrument your application code. You add logging statements, metrics libraries, and tracing SDKs. This works, but has significant limitations:

• Code changes are required: You must modify and redeploy applications

• It’s language-specific: Different languages need different libraries

• There will likely be blind spots: You can only see what you explicitly instrument

• The overhead: Heavy instrumentation slows down applications

• Container challenges: By the time you add instrumentation and redeploy, the problem may have disappeared

eBPF takes a different approach. Instead of instrumenting applications, you instrument the kernel. Since every application ultimately makes system calls to the kernel for network I/O, file operations, and process management, you can observe everything from one vantage point.

The eBPF advantage for Kubernetes

Kubernetes adds another layer of complexity. Your application might be spread across multiple containers, pods, and nodes. Traditional APM (Application Performance Monitoring) tools struggle here because containers come and go rapidly, network topology changes constantly, service meshes add routing complexity, and you often don't control application code (think third-party services or legacy applications you can't modify.)

eBPF doesn't care about any of this. It sees all activity at the kernel level, regardless of what language your app is written in, whether it's containerized, how many times the pod has been rescheduled, or whether you have access to modify the source code. This universal visibility is why the Cloud Native Computing Foundation (CNCF) and major cloud providers are betting heavily on eBPF for the future of observability.

How eBPF Tracing Works (Without Getting Lost in the Kernel)

When your application runs on Kubernetes, there's a clear separation between user space and kernel space. Your code runs in user space, where it's isolated, safe, and has limited access to system resources. To do anything useful – make network calls, read files, allocate memory – your application must ask the kernel for help. The kernel handles these requests via system calls, commonly called syscalls.

eBPF lets us hook into these syscalls without slowing the system down. It’s like having a CCTV camera at every doorway between user space and kernel space, watching who passes through, when, and what they’re carrying.

A Simple Example: HTTP Request Tracing

Your application initiates an HTTP GET request, which needs to go through the network stack. To establish a connection, your application first makes a socket() system call to create a network socket. Then it calls connect() to establish a connection to the remote server. Once connected, it uses send() to transmit the HTTP request. Network packets are sent across the wire, and eventually your application calls recv() to receive the response.

With eBPF tools like Inspektor Gadget's Traceloop, you can automatically hook into these syscalls. The eBPF program captures request metadata including source and destination IPs, ports, timing information, and payload sizes. You get a complete trace of the request without touching your application code.

The eBPF Execution Flow

Here's what happens under the hood when you run a trace. When you deploy Inspektor Gadget and run a gadget, several things happen behind the scenes. Once deployed, the eBPF program springs into action whenever a traced event occurs.

When your application makes a syscall, the eBPF hook triggers and quickly collects relevant data: timestamps, process IDs, container IDs, pod names, request details, and latency information. This data is sent to user space through eBPF maps, which are efficient data structures for kernel-to-userspace communication.

Inspektor Gadget adds Kubernetes context to raw kernel data. Instead of seeing only process IDs, you can see pod names, namespaces, labels, and other metadata. For example, you can tell that a request originated from the frontend pod in the production namespace and targeted the backend service.

The gadget then presents this information in a format that's immediately useful, whether you're using the CLI or integrating with other observability tools.

eBPF is fast because:

• JIT compilation: Programs are turned into native machine code for maximum performance

• Event-driven: Only execute when relevant events occur, not continuously polling

• Kernel-resident: No expensive context switching between kernel and user space

• Highly optimized: Typically adds less than 5% overhead even under heavy load

The Tool: Inspektor Gadget & Traceloop

For this tutorial, we're using Traceloop, an eBPF-based tool that traces request flows through applications by observing syscalls, network calls, and I/O operations at the kernel level.

Why are we using Traceloop for this tutorial?

• It’s quick to install and run (one command)

• The output maps directly to the application’s behavior

• It automatically adds Kubernetes context (pod names, namespaces)

• You don’t need to make any application code changes

What you'll learn applies beyond Traceloop. All eBPF tracing tools (Pixie, Cilium Hubble, Tetragon) work the same way under the hood. They attach to kernel hooks and collect event data. Once you understand the concepts here, you can use any eBPF observability tool effectively.

How to Set Up Your Environment

To get your environment ready for hands-on tracing, we'll verify that your cluster meets the requirements, install Inspektor Gadget, and deploy a sample application to trace.

Verify that Your Cluster Meets the Requirements

Before installing anything, confirm that your Kubernetes cluster is ready for eBPF.

Check your Kubernetes version:

kubectl version --short

You need Kubernetes 1.19 or later. Most modern clusters exceed this requirement, but it's worth verifying.

Verify kernel version on your nodes:

kubectl get nodes -o wide

Then check the kernel version on one of your nodes:

# If using a local cluster like minikube or kind
uname -r

# For cloud clusters, you might need to check node details
kubectl debug node/<node-name> -it --image=ubuntu -- bash -c "uname -r"

You need Linux kernel 5.10 or later for the best eBPF support. Kernel 4.18+ works but with some limitations. If you're using a managed Kubernetes service (GKE, EKS, AKS), you almost certainly have a compatible kernel.

Confirm that you have cluster admin permissions:

kubectl auth can-i create deployments --all-namespaces

This should return "yes". Inspektor Gadget needs elevated permissions to load eBPF programs into the kernel.

Install Inspektor Gadget

You can install Inspektor Gadget in several ways. We'll use the kubectl plugin method as it's the most straightforward for learning.

Install the kubectl gadget plugin:

# Download and install kubectl-gadget
kubectl krew install gadget

# Verify installation
kubectl gadget version

If you don't have krew (the kubectl plugin manager), you can install it first:

# Install krew
(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

# Add krew to your PATH
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

Deploy Inspektor Gadget to your cluster:

kubectl gadget deploy

This creates a gadget namespace and deploys the Inspektor Gadget daemon as a DaemonSet, ensuring each node in your cluster can run eBPF programs.

Verify the deployment:

kubectl get pods -n gadget

You should see one gadget-* pod per node, all in the Running state. If a pod is stuck in Pending or CrashLoopBackOff, check that your kernel meets the version requirements.

Deploying a sample application

To learn tracing effectively, we need an application that does something interesting. We'll deploy a simple microservices application with multiple components so you can see traces flowing across service boundaries.

Start by creating a namespace for our demo app:

kubectl create namespace demo-app

Then deploy a simple web application with a backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: gcr.io/google-samples/microservices-demo/frontend:v0.8.0
        ports:
        - containerPort: 8080
        env:
        - name: PORT
          value: "8080"
        - name: PRODUCT_CATALOG_SERVICE_ADDR
          value: "productcatalog:3550"
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: demo-app
spec:
  type: LoadBalancer
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productcatalog
  namespace: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productcatalog
  template:
    metadata:
      labels:
        app: productcatalog
    spec:
      containers:
      - name: server
        image: gcr.io/google-samples/microservices-demo/productcatalogservice:v0.8.0
        ports:
        - containerPort: 3550
        env:
        - name: PORT
          value: "3550"
---
apiVersion: v1
kind: Service
metadata:
  name: productcatalog
  namespace: demo-app
spec:
  selector:
    app: productcatalog
  ports:
  - port: 3550
    targetPort: 3550

Apply the configuration:

kubectl apply -f demo-app.yaml

And wait for pods to be ready:

kubectl wait --for=condition=ready pod -l app=frontend -n demo-app --timeout=300s
kubectl wait --for=condition=ready pod -l app=productcatalog -n demo-app --timeout=300s

Then just verify that everything is running:

kubectl get pods -n demo-app

You should see both frontend and productcatalog pods in the Running state.

Now you’ll need to get the frontend URL:

# For local clusters (minikube, kind, Docker Desktop)
kubectl port-forward -n demo-app service/frontend 8080:80

# Then access http://localhost:8080 in your browser

# For cloud clusters
kubectl get service frontend -n demo-app
# Look for the EXTERNAL-IP

Visit the application in your browser to confirm it's working. You should see a simple e-commerce storefront. This application makes HTTP requests from the frontend to the product catalog service, which is perfect for tracing.

How to Trace Your First Request: Hands-On Tutorial

Now that everything is set up, let's capture our first trace and see eBPF observability in action.

Generate the Traffic to Trace

First, we need some application activity to observe. We will generate a few requests for our demo application.

In one terminal, start the Traceloop gadget:

kubectl gadget traceloop -n demo-app

This command starts tracing HTTP request handling in the demo-app namespace. Inspektor Gadget monitors the kernel to capture the function calls and system events that occur while processing each request.

In another terminal, generate some traffic:

# If using port-forward
curl http://localhost:8080

# If you have an external IP
curl http://<EXTERNAL-IP>

# Generate multiple requests
for i in {1..10}; do curl http://localhost:8080; sleep 1; done
```

### Viewing Your First Trace

Switch back to the terminal running the trace loop gadget. You should see output appearing as requests flow through your application. The output will look something like this:
```
NODE         NAMESPACE   POD              CONTAINER    PID    TYPE       COUNT  
minikube     demo-app    frontend-abc123  frontend     1234   loop       1      
minikube     demo-app    frontend-abc123  frontend     1234   loop       2

Each line shows a traced execution flow, with the count increasing as the same pattern is observed again.

We can make the output more interesting by filtering:

# Stop the previous trace with Ctrl+C, then run:
kubectl gadget traceloop -n demo-app --podname frontend

This narrows our observation to just the frontend pod, reducing noise and making patterns clearer.

Understanding what you're seeing:

Each column shows different information about your application:

• NODE: Which Kubernetes node the traced event occurred on. In multi-node clusters, this helps you understand workload distribution and identify node-specific issues.

• NAMESPACE: The Kubernetes namespace. We filtered to demo-app, so you'll only see that namespace. In production, filtering by namespace is crucial for focusing on specific applications.

• POD: The specific pod where the event occurred. Each pod gets a unique name (like frontend-abc123), allowing you to distinguish between replicas of the same application.

• CONTAINER: Which container within the pod. Pods can have multiple containers (main application, sidecars, init containers), so this helps you pinpoint exactly where activity is happening.

• PID: The process ID inside the container. This is the actual Linux process that made the syscalls eBPF observed. Multiple PIDs might appear if your application uses multiple processes or threads.

• TYPE: The type of event traced. For Traceloop, this identifies kernel-level patterns detected during request processing.

• COUNT: How many times this pattern has been observed. A rapidly incrementing count indicates high request volume.

What this tells you about your application:

Even from this simple output, you can derive insights. If you see events appearing for the frontend pod but not the productcatalog pod, it might indicate that requests aren't making it to the backend. This is a potential configuration issue. If the COUNT increases rapidly for one pod but not others, you know which replica is receiving traffic, useful for debugging load balancing issues.

The real power becomes clear when you correlate these kernel-level observations with what you know about your application. When you made 10 curl requests, you should see corresponding activity in the trace output. This direct relationship between application behavior and kernel observations is the foundation of eBPF observability.

How to Interpret Traces

Understanding raw trace output is valuable, but interpreting what it means for your application's health and performance is where the real skill lies.

Trace Anatomy: Spans, Timing, and Request Flow

A trace represents a single request's journey through your system. When you curl the frontend, that generates one trace. A span represents a single operation within that trace like "frontend handles request," "frontend calls product catalog," "product catalog queries data," and "frontend returns response." Each span has timing information: when it started, when it ended, and therefore how long it took.

In traditional distributed tracing with OpenTelemetry or Jaeger, you'd explicitly create these spans in your application code. With eBPF, the tool infers spans from syscall patterns. When eBPF sees your frontend process call connect() to the product catalog's IP, followed by send() and recv(), it understands that's a span representing an HTTP request to the backend service.

The request flow is the sequence of spans showing how your request moved through services. In our demo app,

1. The user request arrives at the frontend,

2. the frontend connects to the product catalog,

3. the product catalog processes the request,

4. the product catalog returns the data, the frontend renders the page,

5. and finally, the response is sent to user.

How to Follow Requests Across Services

Let's trace a request across service boundaries to see this flow in action.

First, we’ll start a more detailed trace:

kubectl gadget trace_tcp -n demo-app

The trace_tcp gadget shows network connections, giving us visibility into service-to-service communication.

Next, generate a request:

curl http://localhost:8080

In the trace output, look for connection patterns:

You should see the frontend pod establishing a TCP connection to the product catalog service. The trace will show the source (frontend) and destination (product catalog) IPs and ports, along with timing information.

This is how eBPF lets you follow requests: by observing the network syscalls that implement service communication. You don't need a service mesh or instrumentation libraries, the kernel sees all network activity and eBPF captures it.

Understanding the flow:

1. Your curl command triggers a TCP connection to the frontend pod's IP on port 8080

2. The frontend processes the request and opens a TCP connection to the product catalog's IP on port 3550

3. Data flows back and forth (you'll see send/receive events)

4. Connections close when requests complete

Each step is visible to eBPF because each step requires syscalls that the kernel handles.

How to Identify Bottlenecks and Errors

We can also use tracing to identify performance issues.

First, let’s start by simulating a slow backend:

# Create a deliberately slow endpoint by modifying our deployment
kubectl scale deployment productcatalog -n demo-app --replicas=0

# Wait a moment, then scale back up
kubectl scale deployment productcatalog -n demo-app --replicas=1

While the product catalog is down, generate some requests:

for i in {1..5}; do curl http://localhost:8080; done

You should see connection attempts from the frontend to the product catalog, but if the service is unavailable, you'll see different patterns, possibly connection timeouts or connection refused errors, depending on the exact timing.

What bottlenecks look like in traces:

• Long spans: A span that takes significantly longer than others indicates a bottleneck. In trace loop output, you might see gaps between events or notice certain operations taking longer.

• Retries: Repeated connection attempts to the same destination suggest a failing or slow service.

• Error patterns: Connection failures, timeouts, or unusual syscall sequences indicate problems.

The best skill to have is pattern recognition. A typical, healthy request flow has a rhythm, and events occur in predictable sequences with consistent timing. When something breaks, the rhythm changes. Requests take longer, errors appear, or expected events don't occur at all.

Real-World Debugging Scenarios

Now let's go through three realistic scenarios where eBPF helps:

Scenario 1: Finding a Slow Endpoint

The problem: Users report that the product catalog page sometimes loads very slowly, but metrics show normal average latency.

Let’s use Traceloop to investigate:

# Start tracing with timing information
kubectl gadget traceloop -n demo-app --podname frontend

We’ll generate some mixed traffic:

# Some requests to the homepage (fast)
curl http://localhost:8080

# Some requests to the product catalog (potentially slow)
curl http://localhost:8080/products

In the trace output, compare the COUNT increments for different request patterns. If certain patterns show significantly more loop iterations or longer gaps between events, that indicates those requests are doing more work, possibly hitting a slow endpoint.

The diagnosis:

You might notice that requests to /products cause the frontend to make multiple calls to the product catalog service (visible with kubectl gadget trace_tcp), while homepage requests don't. This explains why the product page is slow: it's making synchronous calls to a backend service, and if that service is slow or the network is congested, users feel the delay.

The fix:

You might implement caching, make the backend calls asynchronous, or optimize the product catalog service itself. The key is that eBPF helped you identify which specific code path was slow without adding instrumentation to your application.

Scenario 2: Tracking Down Failed Requests

The problem: Your monitoring shows a 5% error rate, but application logs don't show any errors. Where are the failures happening?

Now let’s use eBPF to investigate:

# Trace network connections to see connection failures
kubectl gadget trace_tcp -n demo-app

We’ll simulate intermittent failures:

# Create a failing scenario by temporarily breaking service connectivity
kubectl delete service productcatalog -n demo-app

# Generate requests
for i in {1..10}; do curl http://localhost:8080; sleep 1; done

# Restore the service
kubectl apply -f demo-app.yaml

In the TCP trace, you'll see connection attempts from the frontend to the product catalog that fail or time out. The trace will show the source, destination, and what happened (connection refused, timeout, and so on).

The diagnosis:

The failures are happening at the network level, the frontend can't reach the product catalog. This might be due to network policy issues, service mesh misconfiguration, or DNS problems. Traditional application logs might not capture this because the application never receives a response to log, and the connection fails before the application layer even gets involved.

Why eBPF finds this when logs don't:

Your application logs what it experiences. If a connection fails at the TCP level, your application might just see "connection refused" and retry without detailed logging.

eBPF sees the actual syscalls and network events, giving you visibility into what's happening beneath your application layer.

Scenario 3: Understanding Service Dependencies

The problem: You're not sure which services depend on each other, and you want to understand the actual runtime dependencies before making changes.

We’ll use eBPF to map dependencies:

# Trace all TCP connections to see who talks to whom
kubectl gadget trace_tcp -n demo-app

And then generate normal traffic:

# Make various requests to exercise different code paths
curl http://localhost:8080
curl http://localhost:8080/products
curl http://localhost:8080/cart

The trace output shows source and destination for every connection. Build a mental (or actual) map of which pods connect to which services.

The discovery:

You'll see that the frontend pod connects to the product catalog service, but you might also discover unexpected dependencies. Perhaps the frontend also makes calls to a Redis cache, an authentication service, or external APIs. These runtime dependencies might not be documented or might differ from what architectural diagrams show.

Why this matters:

Before deploying a change to the product catalog service, you now know exactly which services will be affected. Before implementing a network policy, you know which connections to allow. Before decomposing a monolith, you understand the actual communication patterns.

This is observability-driven architecture understanding: letting the system show you how it actually works, not how you think it works.

Advanced Tracing Insights

Once you're comfortable with basic request tracing, Inspektor Gadget offers deeper observability capabilities that reveal even more about your system's behavior.

Syscall-Level Observation

The traceloop and trace_tcp gadgets give you application-level insights, but sometimes you need to go deeper. The trace_exec gadget shows you every process execution in your containers.

First, let’s monitor process execution:

kubectl gadget trace_exec -n demo-app

And generate activity:

# Exec into a pod and run commands
kubectl exec -it -n demo-app deployment/frontend -- /bin/sh
ls -la
ps aux
exit

Every command you run inside the container appears in the trace: /bin/sh, ls, ps, and anything else. This helps you understand what's running in your containers, detect suspicious activity, or debug initialization issues.

In production scenarios, this helps you answer questions like: Is my application spawning unexpected subprocesses? Are there security issues like someone running curl to download malicious scripts? Is my init script actually running the commands I think it is?

Network Tracing Insights

Beyond TCP connections, you can trace DNS queries, which often reveal surprising things about your application's behavior.

Run trace_dns:

kubectl gadget trace_dns -n demo-app

Generate requests:

curl http://localhost:8080

You'll see every DNS query your application makes: resolving service names, checking for external APIs, perhaps even unexpected queries that indicate misconfiguration or dependencies you didn't know about.

Common insights from DNS tracing include discovering that your application is using external dependencies you didn't document, finding DNS resolution failures that cause intermittent errors, or identifying excessive DNS queries that could be cached.

Combining eBPF Data with Logs and Metrics

eBPF observability delivers the best results when combined with traditional observability signals. To combine them effectively:

• Use metrics for high-level health monitoring, alerting on anomalies, tracking trends over time, and dashboard visualization.

• Use logs for application-specific context, business logic details, error messages with stack traces, and debugging application code.

• Use eBPF traces for understanding request flows, identifying where time is spent, discovering runtime dependencies, and debugging issues that don't appear in logs.

A practical workflow:

Your metrics alert you that latency increased. You check logs but don't see errors, requests are succeeding, just slowly. You use eBPF tracing to identify that requests are spending extra time in network I/O to a particular backend service. Now you check that service's metrics and logs, and discover it's under heavy load. The eBPF trace gave you the clue that logs and metrics alone couldn't provide.

This approach to observability, using the right tool for each question, is how experienced engineers debug complex systems efficiently.

What eBPF Can and Can't See

eBPF excels at:

• Network traffic (requests, responses, latency)

• System calls (file I/O, process creation, memory allocation)

• Kernel functions (scheduling, locking, resource usage)

• Function calls in binaries (with uprobes)

But keep in mind that eBPF has limitations:

• Cannot decrypt encrypted payloads (unless hooking SSL libraries before encryption)

• Doesn't automatically understand application logic

• Captures low-level events but may need context for high-level semantics

That's why eBPF complements traditional observability rather than replacing it entirely. It gives you infrastructure-level visibility with no code changes and universal coverage. Traditional APM provides application-level context, business metrics, and custom instrumentation. Together, they give you complete observability across your entire stack.

Best Practices and Production Considerations

Before using eBPF tracing in production, there are important considerations around performance, security, and operational practices.

Performance Impact

eBPF's reputation for low overhead is well-deserved, but "low" isn't "zero."

Most eBPF tracing tools add 2-5% CPU overhead and negligible memory overhead. The exact number depends on event frequency, tracing a service that handles 10,000 requests per second will have more overhead than one handling 10 per second.

Measuring the impact:

# Before enabling tracing, check baseline resource usage
kubectl top pods -n demo-app

# Enable tracing
kubectl gadget traceloop -n demo-app

# Check resource usage again
kubectl top pods -n demo-app

You should see a small increase in CPU usage in the pods where tracing is active. This is the cost of the eBPF programs running in the kernel and processing events.

Production best practices:

Use targeted tracing rather than tracing everything everywhere. Trace specific namespaces, pods, or individual containers when investigating issues. For high-volume services, reduce overhead by applying filters, aggregation, or sampling where supported by the tracing tool.

Stop tracing when you’re done investigating. Unlike metrics collection, which typically runs continuously, eBPF-based tracing is best used as an on-demand diagnostic tool to capture detailed insights during active debugging.

When overhead matters:

If you're running latency-sensitive applications (like high-frequency trading systems or real-time communications), even 2-5% overhead might be unacceptable. In these cases, use eBPF tracing in pre-production environments to identify issues, or enable it temporarily in production only when actively debugging.

Security Considerations

eBPF is powerful, which means it requires elevated privileges. Understanding the security implications is crucial.

What eBPF can access:

eBPF programs can observe all syscalls, network traffic, and process execution in the kernel. This includes potentially sensitive data like connection details, file paths, and process arguments. While eBPF programs run in a sandbox and can't modify data or crash the kernel, they can read information that might be sensitive.

Privilege requirements:

Loading eBPF programs requires CAP_SYS_ADMIN or CAP_BPF capabilities (on newer kernels). This is a privileged operation, only trusted users should have this access. The Inspektor Gadget DaemonSet runs with these privileges, so protect access to it accordingly.

Best practices:

Implement RBAC (Role-Based Access Control) to restrict who can run gadgets. Not every developer needs the ability to trace production systems.

Also, be mindful of what data you're collecting, if your traces might contain sensitive information (like authentication tokens in HTTP headers), restrict access to trace data.

Lastly, consider using admission controllers to prevent unauthorized eBPF program loading. Audit eBPF usage in production environments to track who ran which gadgets when.

Network policies:

Inspektor Gadget's DaemonSet needs to communicate with the API server and between its components. Ensure your network policies allow this communication while still maintaining appropriate segmentation.

When to Use eBPF Tracing vs. Traditional APM

eBPF tracing and traditional APM tools like New Relic, Datadog, or Dynatrace serve different purposes. Understanding when to use each helps you build an effective observability strategy.

Use eBPF tracing when:

• You can't modify application code (third-party applications, legacy systems, compiled binaries)

• You need infrastructure-level visibility (network, syscalls, kernel behavior)

• You're debugging issues that span service boundaries but don't show up in application logs

• You want zero instrumentation overhead during normal operation (run tracing only when needed)

• You need to understand what's actually happening versus what the application reports

Use traditional APM when:

• You need business-context metrics (user IDs, transaction types, business-specific data)

• You want automatic instrumentation with minimal setup for supported frameworks

• You need long-term storage and analysis of all traces (eBPF tracing is often used for real-time investigation)

• You want pre-built dashboards and alerting for common application patterns

• You need application code-level visibility (stack traces, variable values, function calls)

The Ideal Approach: Use Both

Many teams run traditional APM for continuous monitoring and use eBPF tracing for targeted investigation when APM data isn't sufficient. For example, your APM shows that a service is slow but doesn't explain why. You enable eBPF tracing on that service to understand what's happening at the kernel level, network delays, excessive syscalls, unexpected dependencies, and find the root cause.

This complementary approach gives you both the continuous visibility of APM and the deep diagnostic power of eBPF without the overhead of running both at maximum depth all the time.

Next Steps and Resources

If you got this far, thanks for reading! Now that you have learned the fundamentals of eBPF observability, and hands-on tracing with Inspektor Gadget, you can continue your journey by:

Exploring Other eBPF Tools

Now that you understand eBPF concepts through traceloop, exploring other tools will be much easier.

Try other Inspektor Gadget gadgets:

# See all available gadgets
kubectl gadget --help

# Some useful ones to explore:
kubectl gadget trace_open -n demo-app     # File I/O tracing
kubectl gadget trace_bind -n demo-app     # Port binding events
kubectl gadget profile cpu -n demo-app    # CPU profiling
kubectl gadget snapshot process -n demo-app  # Process listing

Each gadget teaches you something different about system behavior and gives you another diagnostic tool in your toolkit.

Experiment with other eBPF platforms:

If you're interested in broader observability platforms, try Pixie for its auto-instrumentation and rich UI. Install Cilium with Hubble if you're focused on network observability and want to understand service mesh behavior. Explore Tetragon if security observability interests you, seeing what processes are executing and what files they're accessing.

The concepts transfer directly: all these tools attach eBPF programs to kernel hooks, collect event data, and present it in different ways. Your understanding of syscalls, traces, and kernel-level observation applies universally.

Connect to the CNCF Observability Ecosystem

eBPF observability tools don't exist in isolation. They're part of the broader Cloud Native Computing Foundation ecosystem.

OpenTelemetry integration:

Many eBPF tools can export data in OpenTelemetry format, allowing you to combine kernel-level traces with application-level traces in a unified observability backend. This gives you the complete picture: eBPF shows you infrastructure behavior while OpenTelemetry shows you application context.

Prometheus and Grafana:

eBPF-derived metrics can be exposed as Prometheus metrics and visualized in Grafana alongside your application metrics. This unified dashboard approach helps you correlate infrastructure and application behavior.

Service mesh integration:

If you're using Istio, Linkerd, or other service meshes, eBPF tools like Cilium Hubble can provide deeper visibility into service-to-service communication than the mesh alone provides. The mesh handles traffic management while eBPF gives you kernel-level visibility.

Jaeger and Zipkin:

For organizations using distributed tracing backends, eBPF traces can be exported to these systems, enriching your trace data with infrastructure-level spans that application instrumentation misses.

Community Resources and Learning Paths

The eBPF community is vibrant and welcoming. You can continue learning from the resources below.

Official documentation and blog:

• eBPF.io: The central hub for eBPF documentation, tutorials, and project listings

• Inspektor Gadget docs: Comprehensive guides for all gadgets and use cases

• Cilium documentation: Deep dives into eBPF networking

• CNCF Blog — “What is Observability 2.0?: A quick overview of how modern observability moves beyond traditional tools by unifying metrics, logs, and traces for real-time insight in cloud-native systems.

Learning resources:

• Learning eBPF by Liz Rice: Comprehensive book covering eBPF fundamentals

• eBPF Summit: Annual conference with talks from eBPF creators and users

• CNCF webinars: Regular sessions on observability topics

• Kubernetes observability SIGs: Community discussions and projects

To make this tutorial easy to follow and experiment with, I have included all Kubernetes manifests, demo applications, and eBPF tracing commands in this repository. You can also connect with me on LinkedIn if you’d like to stay in touch.

I had the same questions when I started learning about cloud-native technologies. In this article, I'll share my understanding of core observability concepts, introduce essential observability tools, and share insights from a related project I’m working on.

Table of Contents

• My Introduction to Cloud Native Technologies

• What is Observability?

• Types of Observability Data

  • 1. Metrics

  • 2. Logs

  • 3. Traces

• Observability Tools

  • Prometheus

  • OpenTelemetry

• What are OTLP Resource Attributes?

• Importance of OTLP Resource Attributes

• How My Project Work Fits into All This

• Conclusion

• Additional Resources

My Introduction to Cloud Native Technologies

I recently got selected as a mentee for the Linux Foundation Mentorship to work on the CNCF - Prometheus project. The project is UX-focused, and for the next few months, I'll be working with my mentors to understand how users expect to use OpenTelemetry Line Protocol (OTLP) Resource Attributes in Prometheus.

That's quite a mouthful, I know. I was overwhelmed at first, and honestly, I’m still figuring it out. This is my third week, and although I still have a lot to learn—given that I had no knowledge of cloud native technologies when I applied for this internship—I've already learned quite a bit.

As I often do, I intend to document what I learn through articles to help reinforce concepts in my memory and serve as a resource for other newcomers who may find themselves grappling with these technical terms in the future. You know what they say: you can't say you've understood something until you're able to explain it to someone else who's also new to the topic.

What is Observability?

First, I had to learn what the unfamiliar terms meant—and there were a lot of them flying around. OpenTelemetry. Prometheus. Resource attributes. I’ve come to understand that these terms fall under one umbrella: Observability. Let's start there.

Let’s use a food delivery app to illustrate. When someone orders food, a lot happens behind the scenes:

• The app connects to different services (restaurants, payments, delivery)

• Data flows between different systems to process the order, assign a driver, and track delivery

Engineers need to monitor all the processes to ensure everything works smoothly. Are orders taking too long to process? Is the payment system failing? Does the app suddenly crash under load? Which part of the system is causing delays?

To answer these questions, engineers instrument their code. This means that they configure it to send back real-time data about the state, performance, and behavior of the application. This practice of understanding what's happening inside a complex system based on the data it generates is known as Observability.

You can see the process illustrated in the image below:

In the above flowchart diagram, you can see how data might move through a food delivery application system. The flow starts with a User who orders food from a Food App. The Food App connects to three services (Restaurant, Payment, and Delivery). All these components send data to OpenTelemetry, which collects three types of data: Metrics, Logs, and Traces. OpenTelemetry then forwards only the Metrics data to Prometheus, which stores metrics.

Types of Observability Data

There are three key types of data that systems generate for observability:

1. Metrics

Metrics are numerical measurements collected over time that represent the state or performance of your system. Examples in a food delivery app would be the number of orders processed per minute, average order processing time in milliseconds, number of active users or delivery drivers, and so on.

2. Logs

Logs are text-based records of discrete events that occur within your application. Logs for our food delivery app would look something like this:

ERROR: Payment failed for order #12345 - Credit card declined
INFO: Driver #789 assigned to order #12345

3. Traces

Traces track the entire lifecycle of a request as it moves through different services in a system. They help engineers see how different components interact and identify bottlenecks in complex, distributed systems.

For example, in our food delivery app, a single order request might go through the following steps:
User places an order → Request sent to restaurant system → Payment processor verifies payment → Delivery system assigns a driver → User receives confirmation.

Each step in this journey is recorded as part of a trace. This helps engineers pinpoint where delays occur and optimize the system for better performance.

Observability relies on metrics, logs, and traces working together to provide full system visibility. Metrics tell you something is wrong (“Error rate increased by 5%”). Logs tell you why it happened (“Payment failed due to invalid card details”). Traces show exactly where it happened (“Delay in restaurant service response”).

Observability Tools

Observability tools give you visibility into what’s going on within your application. There are a lot of them, but for the purpose of this article, we’ll talk about two: Prometheus and OpenTelemetry.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit. It does two things:

• Collects data from applications, specifically metrics (remember the data types we talked about earlier)

• and stores them in a time-series database.

A time-series database is a database specifically designed to handle measurements or events that occur over time.

Prometheus uses what's called a pull-based model to collect metrics from applications. Pull-based means Prometheus actively requests (pulls) data from services at regular intervals. Think of it like refreshing a webpage to get the latest content.

OpenTelemetry

OpenTelemetry (OTel) collects, processes, and exports observability data. Unlike Prometheus, which mainly focuses on metrics, OpenTelemetry provides a standardized way to instrument applications for all three types of observability data: logs, metrics, and traces.

OpenTelemetry is designed to be vendor-agnostic. This means you can instrument your applications once with OpenTelemetry and then send that telemetry data to any supported observability backend, which could be an open-source solution like Jaeger or Prometheus, or commercial platforms like Datadog, New Relic, Dynatrace, or Honeycomb.

So, for example, you can use OpenTelemetry to instrument your application – and then Prometheus can pull metrics from OpenTelemetry while other tools handle logs and traces.

What are OTLP Resource Attributes?

When OpenTelemetry collects data from applications, it does more than just gather raw telemetry data. It also provides context about that data. This context comes in the form of resource attributes, which describe where the data came from and what it relates to.

The 'resource' is the component (or entity) producing the data, while the 'attributes' are specific details about that resource.

Resource attributes are structured as pairs of information:

• The "key" is the name or identifier of the attribute (like service.name or host.id)

• The "value" is the specific information for that attribute (like payment-service or server-123)

Together, these key-value pairs identify and describe the specific component that's generating the observability data.

For example, if a payment processing service is sending metrics about transaction times, the resource attributes might include:

• service.name: "payment-service"

• service.version: "1.2.3"

• deployment.environment: "production"

These attributes tell you exactly which service, which version, and in which environment the data is coming from, providing context for interpreting the metrics, logs, or traces.

Resource attributes are not arbitrary. OpenTelemetry provides a standardized set of attribute names and formats that everyone should follow, similar to having an agreed-upon language for describing services and their properties.

For example, OpenTelemetry specifies that you should use service.name (not app_name or service_id) to identify your service. They've created these standardized naming conventions (called semantic conventions) so that:

1. All tools in the ecosystem can understand the same attributes

2. Engineers across different companies use consistent terminology

3. Observability data can be easily shared between different systems

You can still create your own custom attributes when you need something specific (like payment.provider for a payment service), but using the standard attributes whenever possible means your telemetry data will work better with existing tools and be more easily understood by other engineers.

Importance of OTLP Resource Attributes

Let’s say engineers want to monitor how long food deliveries take and whether there are delays in specific locations. Without resource attributes, OpenTelemetry might simply collect and report this metric like this:

delivery_time_seconds: 1800

This tells us that a delivery took 1,800 seconds, or 30 minutes, but nothing else. That’s useful, but it lacks context. Where did this happen? Which service handled it? If there was a delay in delivery and engineers wanted to investigate the cause, this alone would not help.

With OpenTelemetry’s resource attributes, the metric becomes more meaningful:

delivery_time_seconds: 1800
resource:
  service.name: "delivery-service"
  service.instance.id: "instance-456"
  cloud.region: "ng-west-2"
  deployment.environment: "production"
  customer.city: "Lagos"
  restaurant.id: "rest-789"

This tells us:

• The data came from the delivery service.

• The instance handling the request is "instance-456".

• It’s running in the ng-west-2 cloud region.

• The environment is Production (not testing or staging), and so on.

Now, engineers can answer more specific questions:

• Are deliveries slower in certain cities? (Filter by customer.city)

• Are certain restaurants taking longer to prepare food? (Filter by restaurant.id)

• Are delays only happening in a specific cloud region? (Filter by cloud.region)

• Are issues only happening in production or also in staging? (Filter by deployment.environment)

When issues arise, resource attributes allow engineers to quickly narrow down the source of problems. Rather than investigating every service, they can filter by specific attributes to focus their efforts.

How My Project Work Fits into All This

Many engineers use OpenTelemetry for data collection and then send metrics to Prometheus for storage, querying, and analysis.

But Prometheus does not natively support resource attributes in the same way as OpenTelemetry. Instead, it relies on labels to organize metrics. Since Prometheus traditionally has its own labeling system for metrics, integrating OpenTelemetry's resource attributes creates interesting UX challenges.

One key challenge is the cardinality explosion. Cardinality refers to the number of unique combinations of label values (or dimensions) that a metric can have. A "cardinality explosion" occurs when you add labels with many possible values. OpenTelemetry often includes many detailed attributes that, if directly converted to Prometheus labels, would create an overwhelming number of time series. This can slow down Prometheus dramatically or even cause it to crash.

The existing solution involves stuffing all resource attributes into a single JSON-encoded Prometheus label. While this prevents the cardinality explosion, it makes querying extremely cumbersome. Users have to use complex join operations and specialized query syntax to filter or aggregate based on these attributes.

This approach is technically functional but creates a poor user experience. My research aims to understand how users mentally model the transition from OpenTelemetry's rich attribute system to Prometheus's more constrained label system.

The research goals are to:

1. Understand how engineers currently use OpenTelemetry resource attributes with Prometheus

2. Identify pain points in the current integration between these systems

3. Discover user expectations for how resource attributes should be represented in Prometheus

This work is particularly important as more organizations adopt OpenTelemetry as their instrumentation standard while continuing to use Prometheus for metrics monitoring. Creating a seamless experience between these two popular open-source projects will help improve the overall observability ecosystem.

Conclusion

Observability in cloud native applications is clearly an interesting subject and important for building reliable, performant systems. The tools and concepts we've explored – metrics, logs, traces, Prometheus, and OpenTelemetry – form the foundation of modern observability practices.

As I continue my mentorship program, I'll share more insights about how these technologies work together and try to break them down from the perspective of a first-time learner.

Additional Resources

Learn more about:

1. OpenTelemetry

2. Prometheus

3. My UX research project