As agentic systems mature, much of the work developers have them do is delegated, one subagent at a time. Many teams are also exploring and using a shared, multi-tenant Agentic Infrastructure, where cost isn't tied to a single owner. That's where Observability becomes key to monitoring infrastructure costs.

In this guide, you'll learn how observability works, then enable Claude Code's built-in telemetry, run a backend to collect it, and read the metrics, logs, and traces it emits. This will help you start tracking your team's costs more effectively, and it'll only improve as emitted telemetry matures and correlates more cleanly with your sessions.

Note: In its current state, the emitted telemetry from Claude Code provides no attributes that allow a reliable map to named sessions. Usage can be tracked using session_id, but it's still clumsy in a longer session mixing multiple prompts/skills.

This guide is scoped to Claude Code's telemetry for metrics, logs, and tracing. Note that it applies to Linux and macOS only.

Table of Contents

• Observability with OpenTelemetry

  • Telemetry Data

• Instrumenting Claude Code

  • Pull vs Push: How Telemetry Leaves an App

  • When to run a Collector

  • Prerequisites

  • Setup

• Exploring Telemetry

  • Metrics

  • Logs

  • Tracing

• Conclusion

• References

Observability with OpenTelemetry

Observability is the ability to answer questions about a system's runtime behavior from the data it emits. You do this without looking into its internals, attaching a debugger, reading source code, or manually trying to reproduce the behavior.

Here, a system's runtime behavior means what's externally visible. You can ask questions like:

• How much time 95% of all requests take.

• What the failure rate is across all requests received.

• What the cache hit ratio is for the in-memory cache the service uses.

• The difference between the configured and deployed replica counts for a service.

For Claude Code, the inaccessible inner workings are: how it manages context, how work is divided across multiple LLM calls, and how subagents are orchestrated. But you can read the emitted telemetry from Claude code to answer questions like:

• How much a dev or a team spent over a day, week, or month.

• How that usage is distributed across the supported models and effort levels.

• How many tokens are spent per dollar, and how much that varies by type (input, output, cacheRead, cacheCreation).

• When a compaction event kicked in, and by how much it reduced the context's token usage.

Only an instrumented system can answer these questions. Instrumentation is a piece of code added by the developer or built into the tool that records a program's runtime behavior and emits it as telemetry. For example, a measurement like this request spent 100 tokens.

The telemetry data helps avoid silent failures by providing a well-structured data trail of the system's behavior over time. For example, here's a chart from GitHub's August 17, 2026 outage postmortem explaining a rise in GitHub Actions runs over time from ~30M to ~110M:

Telemetry Data

The emitted telemetry consists of three categories of data:

• Metrics: Numeric measurements aggregated over a time window, like queries per second (QPS).

• Logs: A detailed record of an individual event, with a timestamp. For example, a compaction event in Claude Code.

• Traces: The path of one request through the system, split by time into nested requests. For example, an order-placement request on an ecommerce website, showing which internal services it calls to complete the request.

OpenTelemetry (https://opentelemetry.io/) is an observability framework that helps you generate, collect, and export these signals to a backend, which handles storage, querying, and visualization. Keeping that split makes it tool/vendor agnostic where the backend can be open-source or proprietary. It provides instrumentation SDKs for multiple programming languages.

Instrumenting Claude Code

Instrumentation code usually runs alongside the application, at the points best for measuring: a middleware with a request arriving or a response going out, or a token being counted.

It is plugged into the application in two ways:

1. A Shared Instrumentation Library: Applications using an open source framework can add an instrumentation library as a dependency and link it with the application's lifecycle methods. OpenTelemetry publishes instrumentation libraries for many frameworks (Example: Spring Framework).

2. Customized Implementation by Application Developers: Telemetry data emitters are added directly in the codebase using the OpenTelemetry SDK. For a closed-source product, the code is private, but it can still emit telemetry data compatible with OpenTelemetry standards.

Example: HTTP Instrumentation

Middleware in HTTP handling is a common codepath for all requests, which is why it's chosen for application-wide settings like authentication. The same reason makes it a good place for instrumentation code: wrap the handler once, measure every request.

The diagram shows where instrumentation sits relative to the request path.

• A client request passes through an HTTP middleware before reaching the application's request handler and downstream calls. The middleware uses SDK constructs to time the handler and record duration, status, and a count.

• The SDK then buffers those measurements and pushes OTLP to the Collector on a background thread, off the request path.

Claude Code is the second case, where both the core app and its instrumentation module are provided by Anthropic. The code that measures token usage, cost, and tool calls is built in and emits OpenTelemetry over OTLP. As a Claude Code user, you only need to enable the telemetry and prepare a backend to receive and analyze it.

Note: OTLP is a telemetry data delivery protocol designed in the scope of the OpenTelemetry project. This guide assumes end-to-end compatibility with OTLP. Wiring up incompatible telemetry or using incompatible backend components can give unpredictable results and is out of scope.

To collect, store, and read the telemetry, you'll need the following components:

• OpenTelemetry Collector: A vendor-agnostic implementation of how to receive, process, and export telemetry data. This is optional but great to have for personal setup. Must have for a production use case.

• Jaeger: Distributed tracing backend, released as an open source tool by Uber.

• Prometheus: Collects and stores metrics as Timeseries Data.

• Loki: Scalable Log Aggregation system by Grafana.

• Grafana: for UI visualization of logs, metrics, and traces.

Pull vs Push: How Telemetry Leaves an App

Telemetry leaves an application in one of two ways:

Pull (Scrape): The app exposes its current metrics on an HTTP endpoint, and a scraper (Prometheus) reads that endpoint periodically. Every running instance needs its own port, and the scraper must know all of those addresses ahead of time. The app is passive: the scraper drives the data movement. This suits long-lived processes with stable addresses.

In OpenTelemetry, pull-based scraping is configured with OTEL_METRICS_EXPORTER=prometheus (see the SDK environment variables for the accepted exporter values).

By convention, the app makes its metrics available at http://localhost:9464/metrics. This exporter handles metrics only.

Push (OTLP): The app sends its telemetry to a receiving endpoint on a regular interval. This is more flexible: any number of processes can push to the same endpoint with no registration ahead of time, so apps can start and stop freely even as their addresses change.

In OpenTelemetry, push is configured with OTEL_METRICS_EXPORTER=otlp, which ships over the OTLP exporter.

The push model can carry metrics, logs, and traces.

When to Run a Collector

A Collector is deployed when the existing stack isn't enough to handle the system's growing scale and complexity. It helps in the following ways:

• One export config: every producer points at the Collector instead of each carrying its own per-backend exporter setup.

• Outbound-only connections: enterprise networks often block the inbound connections a pull-based scraper needs. With a Collector, the app pushes out to it and it pushes onward, so nothing has to accept inbound traffic.

• Fan-out and translation: the Collector can convert telemetry into a vendor's storage format and send the same signal to more than one backend.

• Buffering: if a backend goes down, the Collector holds the data and retries, absorbing transient failures.

• Processing: it can apply processors before data leaves for storage, such as redacting attributes.

Note: This guide runs a push-based configuration with a Collector even though the setup is single-user. The stack has three backends that store and query data differently, and letting the Collector receive Claude Code's OTLP once and route each signal to the right place is simpler than wiring the app to all three. That's why the tool list calls it optional for personal use but a must-have in production: its value grows with the number of producers and backends.

Prerequisites

Each section in this guide links relevant docs, but you'll make faster progress if the tools and query languages specified below are already familiar:

You'll need

• Latest version of Claude Code

• Docker Engine (29.4.1+) with Docker Compose.

  • Capacity to run 5 Containers (4-core CPU, 8GB RAM, 15GB Disk)

• A Claude Plan that includes Enhanced Telemetry beta (Pro+ / Max)

• Git

• Ensure these ports are free at localhost:

  • 3000 (Grafana)

  • 3100 (Loki)

  • 4317/4318 (OTel Collector OTLP gRPC/HTTP)

  • 9090 (Prometheus)

  • 16686 (Jaeger UI)

Knowledge that would help

• Docker Compose: bringing up containers defined in compose file, reading container status and logs by docker compose ... commands.

• Bash and Config files: Setting environment variables, editing JSON files.

• PromQL (Prometheus): How to use counters/gauges, range selectors, and sum / increase / rate / by (label) grouping.

• Grafana: Explore a datasource, build dashboards with stat and timeseries panels. Panel transformations and Global variables.

• LogQL (Loki): stream selectors, logfmt, and label_format.

• Jaeger and tracing: the trace/span model (parent-child spans, span count, duration) and the Jaeger UI's tag search.

• Claude Code's execution model: sessions, subagents, skills, tools, and context compaction.

• Claude billing basics: tokens and the prompt-caching tiers.

Setup

The test observability stack is deployed using Docker Compose. For telemetry export to work, Claude Code must be able to reach Collector's OTLP endpoint which is localhost:4317 (gRPC) or localhost:4318(HTTP) when running on same machine. All backend services run in a container with their own Docker volume for persistence.

The diagram shows how the telemetry components we're using are linked.

• Apart from Claude Code, every component runs in a container managed by Docker Compose.

• Multiple instances of Claude Code running on any machine (host or cloud VM) should be able to export telemetry to the Collector as long as those machines have connectivity to it (ports 4317/4318 of the host running the Collector container are reachable).

Moving from top to bottom:

• Claude Code exports all three signals over OTLP to the Collector.

• The Collector then splits them by type, pushing traces to Jaeger and logs to Loki, while exposing metrics on port 8889 for Prometheus to scrape.

• Jaeger, Prometheus, and Loki each persist to their own Docker volume.

• Grafana queries all three as the single dashboard layer.

The goal here is a live stream of telemetry from Claude Code that gets stored in a backend and can be queried on demand, during a session or long after. Two things must be in place:

• Enable telemetry in Claude Code. The instrumentation is built in but emits nothing until telemetry is enabled and its OTLP exporter points at the Collector.

• Run the observability backend. The Collector processes each signal, forwarding it to Prometheus, Loki, and Jaeger for storing and serving queries.

You'll start the backend first, so the telemetry has somewhere to go.

Start the Observability Backend

Before enabling telemetry in Claude Code, ensure the stack is up to collect, process, and read the data. The code for the test observability backend lives in this Github repo.

The repo has the following structure:

.
├── README.md
└── compose
    ├── docker-compose.yml # Docker config for 5 containers in the observability stack. Applies pinned image versions, port mappings and named volumes for each service.
    ├── grafana
    │   └── provisioning
    │       ├── alerting
    │       ├── dashboards
    │       ├── datasources # datasources(Prometheus, Loki, Jaeger) and dashboards. Empty initially.
    │       └── plugins
    ├── jaeger-config.yaml # Jaeger v2, badger (local-file) storage for traces. Ties to the user: root TIP below.
    ├── loki-config.yaml # single-binary Loki, filesystem storage. Near default settings.
    ├── otel-collector-config.yaml # receive/process/export pipeline: OTLP in on 4317/4318, traces out to Jaeger, logs to Loki, metrics exposed on :8889 for Prometheus.
    └── prometheus.yml # a single scrape job against the Collector's :8889, 30s interval.

You'll only need docker compose command to start the containers. It reads docker-compose.yml and starts the containers, linking them to their respective config files.

Connectivity between containers:

All containers start within the same Docker network, which allows them to communicate using container names directly. For example, collector's exporter config uses container names:

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

Note that it doesn't contain Prometheus config, since Prometheus ends up scraping it from the collector as configured in prometheus.yml:

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: otel-collector
    static_configs:
      - targets: ["otel-collector:8889"]

Start the containers using Docker compose:

git clone https://github.com/ps-mir/otel-dev-stack.git
cd otel-dev-stack/compose
docker compose up -d

# Output
 ✔ Volume compose_loki_data                          Created                                                                                                                                          0.0s
 ✔ Volume compose_grafana_data                       Created                                                                                                                                          0.0s
 ✔ Volume compose_prometheus_data                    Created                                                                                                                                          0.0s
 ✔ Volume compose_jaeger_data                        Created                                                                                                                                          0.0s
 ✔ Network compose_default                           Created                                                                                                                                          0.1s
 ✔ Container compose-prometheus-1                    Started                                                                                                                                          4.1s
 ✔ Container compose-loki-1                          Started                                                                                                                                          4.2s
 ✔ Container compose-jaeger-1                        Started                                                                                                                                          4.3s
 ✔ Container compose-otel-collector-1                Started                                                                                                                                          3.3s
 ✔ Container compose-grafana-1                       Started                                                                                                                                          2.7s

Check container status:

# all five services should show "Up"
docker compose ps

# Output
NAME                       IMAGE                                              COMMAND                  SERVICE          CREATED         STATUS         PORTS
compose-grafana-1          grafana/grafana:13.2.0                            "/run.sh"                grafana          3 minutes ago   Up 3 minutes   0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp
compose-jaeger-1           cr.jaegertracing.io/jaegertracing/jaeger:2.20.0   "/go/bin/jaeger --co…"   jaeger           3 minutes ago   Up 3 minutes   0.0.0.0:16686->16686/tcp, [::]:16686->16686/tcp
compose-loki-1             grafana/loki:3.7.6                                "/usr/bin/loki -conf…"   loki             3 minutes ago   Up 3 minutes   0.0.0.0:3100->3100/tcp, [::]:3100->3100/tcp
compose-otel-collector-1   otel/opentelemetry-collector-contrib:0.159.0      "/otelcol-contrib --…"   otel-collector   3 minutes ago   Up 3 minutes   0.0.0.0:4317-4318->4317-4318/tcp, [::]:4317-4318->4317-4318/tcp, 55679/tcp
compose-prometheus-1       prom/prometheus:v3.11.2                           "/bin/prometheus --c…"   prometheus       3 minutes ago   Up 3 minutes   0.0.0.0:9090->9090/tcp, [::]:9090->9090/tcp

TIP: Jaeger runs as user: root (Compose file) to create the badger dir. Not doing so causes a failure: mkdir /badger/key: permission denied. Jaeger itself doesn't need root permission, but Docker volumes are owned by root:root on first mount.

Enable Telemetry

OpenTelemetry instrumentation, once added to an application, stays disabled until specific configuration enables it.

There are two ways to enable telemetry in Claude Code:

1. Environment Variables

Setting specific environment variables enables telemetry generation. Beyond the standard OTEL_* variables, Claude Code defines its own CLAUDE_CODE_* variables.

This guide uses the following settings:

# master switch: when unset or 0, Claude Code produces no telemetry at all
export CLAUDE_CODE_ENABLE_TELEMETRY=1

# opt into the beta enhanced-telemetry attributes and events (extra session and tool detail)
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1

# per-signal exporter selection; "otlp" ships the signal over OTLP.
# other accepted values are "console" (print locally), "prometheus" (metrics only), and "none" (drop the signal)
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp

# OTLP transport: "grpc" talks to the collector's 4317 port; "http/protobuf" would use 4318
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# one endpoint for all three signals: the collector's OTLP listener on the local machine
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# how often metrics are flushed, in milliseconds; the default is 60000 (60s),
# shortened here so a manual check sees fresh data without a long wait
export OTEL_METRIC_EXPORT_INTERVAL=5000

# emit cumulative counters instead of delta (see "Aggregation Temporality" below)
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative

Environment variables, however, are process-wide and can affect more than Claude Code. For example,

• Accidentally enabling instrumentation in other applications.

• Interfering with OpenTelemetry code/tests if you're developing your own instrumentation or working on any OpenTelemetry SDK.

2. Claude Code settings.json

OpenTelemetry defines Declarative Config, a YAML based configuration to enable telemetry, but Claude Code doesn't support it. But it lets you set the same env variables in ~/.claude/settings.json. That isn't declarative config, but it's better than shell environment variables because it applies only to Claude Code. An example:

{
  "effortLevel": "medium",
  "tui": "fullscreen",
  "env": {
     "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
     "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
     "OTEL_METRICS_EXPORTER": "otlp",
     "OTEL_LOGS_EXPORTER": "otlp",
     "OTEL_TRACES_EXPORTER": "otlp",
     "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
     "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4317",
     "OTEL_METRIC_EXPORT_INTERVAL": "5000",
     "OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE": "cumulative"
  }
}

Only the env block matters for telemetry. effortLevel and tui are unrelated settings you may already have. The variables match the annotated list above.

Aggregation Temporality

Prometheus counter type metrics only increase over time. Their raw value isn't useful, so you read them through per-second growth (rate()) or total growth over a time window (increase()).

Aggregation temporality decides what number a counter reports on each telemetry export: the change since the previous export(Delta), or the running total since the process started(Cumulative).

A short example. Say Claude Code spends tokens over four 5-second export intervals:

By default, Claude Code emits metrics with AggregationTemporality: Delta. This can be inspected and confirmed from the Collector's container logs using the command:

# Command only works from directory containing docker-compose.yml
docker compose logs otel-collector

Note: To enable detailed logs in the Collector, you need to add the debug exporter to the Collector config.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, debug]

Then restart the container:

# Command only works from directory containing docker-compose.yml
docker compose up -d --force-recreate otel-collector

Log output with AggregationTemporality: Delta:

otel-collector-1  | Descriptor:
otel-collector-1  |      -> Name: claude_code.active_time.total
otel-collector-1  |      -> Description: Total active time in seconds
otel-collector-1  |      -> Unit: s
otel-collector-1  |      -> DataType: Sum
otel-collector-1  |      -> IsMonotonic: true
otel-collector-1  |      -> AggregationTemporality: Delta <---
otel-collector-1  | NumberDataPoints #0

Delta doesn't work well with Prometheus functions like rate()/increase(), since they expect cumulative values.

Setting OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative switches the exported metrics from delta to cumulative temporality. It has been added to both the env and JSON config used in this guide.

As before, you need to restart the Collector container after any config change for it to take effect.

Exploring Telemetry

With telemetry flowing, you can start querying it. Metrics, logs, and traces each answer a different kind of question about Claude Code usage, so the three sections below are largely independent.

The data behind them comes from two places.

• The Metrics and Logs sections query whatever Claude Code usage has accumulated in the backend, so your panels will show your own sessions and the numbers won't match the screenshots. Give it a few real sessions before expecting much to show.

• The Tracing section instead walks a single deliberate run, a custom skill summarizing a batch of meetings, described in enough detail to follow along. You don't need to reproduce it.

Metrics

Metrics are the aggregate, time-windowed view of Claude Code usage. Example: total cost, token volume, and how each trends and breaks down by attributes like model, effort, and token type. Use them to watch spend and spot shifts in consumption.

Each metric is a numeric measurement recorded over time, a time series of timestamped values you can plot or aggregate. Claude Code's metrics are running totals (counters), so a query reports the change over a chosen window rather than the raw value. See Aggregation Temporality above for how that works.

Prometheus is the metrics backend we're using here. It scrapes the Collector, stores the series, and answers queries written in PromQL. Grafana reads the same data for dashboards at localhost:3000. The full list of metrics and their attributes is in the Claude Code monitoring docs.

Open Prometheus in your browser (localhost:9090), type claude in the query field, and you should see the supported metrics:

For each metric below, you'll explore it first using PromQL, and then use the same query to add it to the Grafana dashboard as a panel.

Total USD Spent

claude_code_cost_usage_USD_total represents cumulative usage cost, in USD, tracked per session. It's useful for controlling budgets and spotting sudden spikes in usage.

This is a client-side estimate based on token counts priced at Anthropic's per-model, per-type rates and accumulated. It's completely normal for it to exceed your Claude Code plan's subscription cost.

Note: This metric is more critical if you're paying per raw API call. A subscription gives you a usage allowance with increased but bounded rate limits.

Test the following query in Prometheus first (localhost:9090/query):

sum(increase(claude_code_cost_usage_USD_total[10m]))

increase(...[10m]) gives the counter's growth over the last 10 minutes. sum(...) with no by clause collapses the per-attribute series (model, effort, and others) into one number.

For a Grafana panel, swap the fixed [10m] window for the $__range built-in variable so the value follows the dashboard's time picker:

sum(increase(claude_code_cost_usage_USD_total[$__range]))

To add it as a panel, open Explore, select Prometheus as the data source, and run the query. The result depends on how much you've used Claude Code in the window.

On adding to the dashboard you should get more Panel Options. Select the Stat panel:

This panel needs to be added to the Grafana dashboard.

Total Token Usage

claude_code_token_usage_tokens_total is the cumulative token count, with the same counter shape as the USD cost metric. Read a raw series in Prometheus first to see which labels you can aggregate by. A single series looks like:

claude_code_token_usage_tokens_total{effort="high", exported_job="claude-code", instance="otel-collector:8889", job="otel-collector", model="claude-sonnet-5", otel_scope_name="com.anthropic.claude_code", otel_scope_version="2.1.252", query_source="auxiliary", session_id="e0b9795b-4da3-4171-8fa6-a2866bf44d86", terminal_type="ssh-session", type="cacheCreation"}    213730

The trailing number is the counter value. Key attributes you'll be working with are type, model, and effort. The Token Usage by Type chart below groups on type.

The window total is the same query shape as Total USD Spent, with the token counter:

sum(increase(claude_code_token_usage_tokens_total[$__range]))

Tokens Per USD

Unlike the previous two metrics, this is a derived figure, calculated over a time window as Total Tokens / Total Cost.

sum(increase(claude_code_token_usage_tokens_total[$__range])) / sum(increase(claude_code_cost_usage_USD_total[$__range]))

This one number collapses every attribute combination into a single value. Each distinct combination, for example model A at medium effort versus model B at high effort, is its own time series, and the query sums across all of them.

To analyze a specific combination, run the same ratio split by an attribute and compare.

sum by (model) (increase(claude_code_token_usage_tokens_total[$__range]))
  / sum by (model) (increase(claude_code_cost_usage_USD_total[$__range]))

Swap model for effort or type; the raw series above lists the rest of the labels.

Overall Result:

Stats Panel(6hr window): Total USD Spent ($4.28), Total Tokens Spent (3.02M), and Tokens Per USD (707k).

Token Usage by Type

You'll see how claude_code_token_usage_tokens_total changes over time, broken down by type. A single stat hides the shape, so use a time-series panel.

The type attribute has four values, which differ a lot in cost:

• cacheRead: tokens served from an existing cache entry. They dominate token spend in a long session, and are cheaper than the baseline rate.

• cacheCreation: tokens written into the prompt cache on the first prefix load. Costly.

• input: new, uncached prompt tokens.

• output: model-generated tokens.

To also see the total, use two queries: the per-type breakdown and the un-split total for reference.

# per-type breakdown
sum by (type) (increase(claude_code_token_usage_tokens_total[$__rate_interval]))
# total
sum(increase(claude_code_token_usage_tokens_total[$__rate_interval]))

__rate_interval is Grafana's per-step window for time-series panels, the counterpart to the __range used for the stat panels above.

The two queries running in Explore, before saving them as a panel.

After adding to the dashboard:

Each spike is a burst of Claude Code activity, the flat stretches are idle time. Hovering a point splits the total into the four types: here the total is about 1.0M tokens, of which cacheRead is about 925k (roughly 92%), the rest cacheCreation, output, and input.

TIP: Token consumption is dominated by cacheRead, which is also the cheapest type.

Token Usage by Model and Effort

This is the concrete version of the breakdown suggested under Tokens Per USD: which model and effort pairs are actually consuming tokens.

sum by (model, effort) (
  increase(claude_code_token_usage_tokens_total[$__rate_interval])
)

Here the token counter is grouped by its model and effort attributes. In this window every series is claude-sonnet-5 at either medium or high effort, and one burst of medium effort near 18:13 reaches about 2.6M tokens. The number of unique groupings depends on the cardinality of the chosen attributes.

Logs

A log record is a timestamped event with its full field set attached. You query logs when you want that specific event and the context around it: what happened, when, and with which values.

Metrics are the pre-aggregated form of the same activity. Anything that means counting, summing, or taking percentiles across many records belongs in a metric. If you're aggregating log output downstream, that data should have been a metric from the start.

Logs are the right tool for:

1. Per-event context: the full detail of one occurrence, not a rolled-up number.

2. Discrete or irregular events: a compaction firing, a session start, or an API error.

3. Post-incident forensics: reading raw records back while debugging after the fact.

4. Trace correlation: a log line carrying a trace and span ID drops you into the request it came from.

The log backend we're using here is Loki, queried with LogQL. Running logs through a backend like this buys you:

1. Structured fields: filter and compute on named keys instead of regex over text.

2. Field indexing: label lookups return without scanning every line.

3. Trace and span correlation: pivot from a log to its trace, or pull every log for one trace.

4. Time-bounded queries: each query is scoped to a window, keeping the scan cheap.

Grafana reads Loki for dashboards, the same as it does for Prometheus.

Compaction Event

Compaction is Claude Code trimming its own context when it grows too large. Each compaction emits a log event (event_name="compaction") carrying the token counts before and after (pre_tokens, post_tokens) and the span_id it happened under, so a query over those events shows how often it fires and how much it reclaims each time.

In Grafana Explore, select Loki as the data source and paste the LogQL below. It selects the compaction events, parses their fields with logfmt, and derives a per-event reduction percentage with label_format. The fields only exist once compaction has actually happened, so trigger a few first.

{service_name="claude-code"} | event_name="compaction"
  | logfmt
  | label_format reduction_pct=`{{ printf "%.1f" (mulf (divf (subf .pre_tokens .post_tokens) .pre_tokens) 100) }}`

Deriving reduction_pct for each record is fine here because it stays per-event. A running average across compactions would belong in a metric.

The label_format line adds a reduction_pct label. To show it as a table, switch the panel to Table view and add three Grafana transformations:

1. Extract fields from the labels object.

2. Filter fields by name to keep Time, pre_tokens, post_tokens, reduction_pct, and span_id.

3. Convert field type to turn pre_tokens, post_tokens, and reduction_pct into numbers.

Tracing

Tracing provides a detailed picture of the full path a request takes through an application, from start to completion. Some fundamental concepts behind tracing:

• Span: a timed operation representing a unit of work. Building block for traces. All trace data is recorded as a sequence of spans, and each has a type, its operation name:

  • claude_code.interaction: one prompt and everything Claude Code does to answer it. Normally the root span, so one interaction is effectively one trace.

  • claude_code.llm_request: a single model call inside an interaction.

  • claude_code.tool: a single tool call inside an interaction (Bash, Write, Agent, and so on).

• Trace: a tree of spans representing a request path from start to completion.

• Session: one Claude Code run, identified by session.id. It can result in many interactions and traces.

• Subagent: a nested Claude Code instance started by the Agent tool, running its own interactions.

Jaeger is the tracing backend used here. The Collector forwards spans to it over OTLP. Jaeger stores them and lets you search traces by service and span tags and inspect each one as a span tree. Everything below uses its UI at localhost:16686.

Generating Traces

To generate trace data, this guide will use a test prompt to spawn agents and prepare some text. This prompt was tested with Sonnet 5 at medium effort.

You can paste the prompt directly into Claude Code:

Spawn 4 subagents in parallel, one per topic below. Each subagent researches its topic from your own knowledge and returns a ~150-word summary with 3 key points. Do not have them read files or run commands.
Topics:
1. How TCP congestion control works
2. The CAP theorem
3. How DNS resolution works
4. What a Bloom filter is

Once all 4 return, combine the summaries into one markdown document and write it to summary.md

Note: Ask Claude Code for the session_id in the same session after the prompt finishes. This will be used to find related traces in Jaeger.

Trace By Session ID

The search filters by service = claude-code and the tag session.id=<id>. It returns 6 traces, all rooted at claude_code.interaction, with span counts from 1 to 20 and durations from about 1 second to 33 seconds.

The list alone doesn't say which trace did what. Going through them by hand, or scripting it against the trace API for a real session, gives the following:

A few observations:

• Half the traces are noise. Traces 1, 3, and 4 are single-span interactions with no model call or tool, an idle session being pinged. Trace 2 is a brief exchange. Only Traces 5 and 6 are the run.

• The parallel dispatch is a single interaction. Trace 5 fires all four Agent calls inside one claude_code.interaction. Their nested model calls (7 to 10 seconds each) overlap, so the interaction finishes in about 16 seconds despite roughly 35 seconds of combined subagent LLM time.

• Each subagent's model call is nested under its Agent span and carries an agent_id, so you can tell the four apart.

• agent_id is opaque. There's no agent.name or skill.name. The trace tells you four subagents ran and how long each took, not which topic each was given.

• Spans carry token counts but no USD cost. Each claude_code.llm_request has input_tokens, output_tokens, cache_read_tokens, and cache_creation_tokens, but no USD figure.

• The write is a separate, later interaction. Trace 6 has no Agent spans: one claude_code.llm_request of about 23 seconds produces the combined markdown, then a short Write. The two ScheduleWakeup spans are background coordination.

TIP: From the trace you get the four subagent calls, each with an agent_id, token counts, and timing, but no span says which topic a subagent was handed. In contrast, Metrics can provide attribution by model, effort, and skill.

CAUTION: user_prompt is redacted by default on interaction spans. OTEL_LOG_USER_PROMPTS=1 disables this and logs raw prompt text. Avoid enabling it in multi-user/tenant environments since it exposes prompt content to anyone with access to the telemetry backend.

Conclusion

This guide was an end-to-end walkthrough of observability in Claude Code, enabling its telemetry and collecting each of the three signals in a local backend for analysis.

Metrics give you the ability to dissect cumulative cost and usage readings by attribute over a chosen time period. That matters most in a shared or multi-tenant setup, where cost isn't tied to a single owner and someone still has to account for it.

Logs are records of individual events, useful for digging into exactly what changed during one, like compaction.

Traces show how one prompt expands into subagents and model calls, with timing and token counts on each. That's the starting point for debugging or tightening a complex or multi-agent prompt, though the spans don't yet record which prompt or skill drove a given call.

Some of this telemetry is behind the Enhanced Telemetry beta, so span names and attributes can still change, and gaps like per-call attribution may close as it matures. It's worth re-checking the monitoring docs as the surface settles.

References

• Claude Pricing: Prompt Caching

• Claude Code: Monitoring Usage

• Prometheus: Counter Metric Type

• Jaeger: Finding Traces

• Loki: LogQL Log Queries

• Grafana: Stat Panel

• OpenTelemetry Collector: Configuration

Among these metrics, histograms are particularly useful. They show how often values fall into different ranges, like HTTP request durations, database query times, or queue processing latencies.

Unlike simple averages, histograms show the full picture: you can see how many requests are fast, how many are slow, and where the occasional outliers occur that might be silently degrading the user experience.

In payment systems, for instance, a sudden spike in transactions can expose hidden bottlenecks. Most of the requests might complete quickly, but a small percentage of slow transactions can ripple through the system, impacting retries, failures, and overall throughput. Histograms help identify these issues early by showing how values are distributed and highlighting outliers that averages obscure.

But not all backends understand Prometheus metrics natively. Many modern observability platforms prefer OTLP (OpenTelemetry Protocol). Forwarding Prometheus metrics without converting them can lead to incomplete or misinterpreted data. That’s why we need a pipline to scrape, transform, and export histograms into OTLP so that your observability pipeline remains consistent and actionable.

In this article, we'll use the OpenTelemetry Collector to scrape Prometheus histograms from application /metrics endpoints, map them to the OpenTelemetry Histogram data model, and export them to our observability backend using OTLP. The Collector acts as a bridge that preserves data fidelity while ensuring compatibility with your monitoring platform.

To make this concrete, we'll use a small FastAPI application that simulates payment transactions. It exposes two Prometheus metrics: payment_transaction_duration_seconds (a histogram tracking how long each transaction takes) and payment_transactions_total (a counter of completed transactions). You can follow along with your own instrumented application, as anything exposing Prometheus metrics at /metrics will work the same way. This is the metric we'll follow from the application all the way to the observability backend.

What We'll Cover:

• Prerequisites

• 1. How to Scrape Metrics with Prometheus Receiver

• 2. Transforming Prometheus Histograms

• 3. Exporting Metrics via OTLP

• 4. Putting the Pipeline Together

• 5. Running the OpenTelemetry Collector

  • 5.1 Set Up SigNoz Cloud

  • 5.2 Start the FastAPI Application

  • 5.3 Start the Collector

  • 5.4 Generate Test Transactions

  • 5.5 Confirm Backend Receipt

  • 5.6 Troubleshooting

• Conclusion

Prerequisites

Before you begin, make sure you have:

• Docker installed

• An application exposing Prometheus metrics through a /metrics endpoint

• An OTLP-compatible observability backend

• Basic knowledge of Prometheus metrics

• Basic knowledge of YAML

• Basic familiarity with Docker and OpenTelemetry

You don't need advanced OpenTelemetry knowledge to follow this tutorial. I'll walk through the Prometheus histogram and show what happens to it as it moves through the OpenTelemetry Collector.

1. How to Scrape Metrics with Prometheus Receiver

First, we'll collect metrics from the application. The demo application exposes its Prometheus metrics at the /metrics endpoint, and the Prometheus receiver periodically scrapes this endpoint and ingests the metrics into the observability pipeline. You can also inspect /metrics directly to see the data before the Collector reads it.

Configuration setup example:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: payment-demo
          scrape_interval: 15s
          static_configs:
            - targets: ["payment-api:8080"]

What's happening behind the scenes here:

scrape_interval controls how often the Collector scrapes the target. Here, we're using 15 seconds; adjust it based on how frequently you need metric updates and the load your application can handle.

In high-throughput systems like payment platforms or real-time processing services, the scrape interval becomes particularly important. Setting it too long may cause you to miss short-lived performance issues or transient errors, while setting it too short risks overwhelming the service with scraping requests or generating excessive network traffic.

targets lists the specific endpoints that expose Prometheus metrics. You can add multiple targets when you need to scrape more than one service.

Finally, job_name is an identifier that helps group metrics logically, making them easier to manage and analyze once they reach your backend.

2. Transforming Prometheus Histograms

Once the metrics are scraped, the next step is transformation. Prometheus exposes histograms as multiple time series:

• _bucket shows how many requests fall below a certain duration.

• _sum is the total of all observed durations.

• _count is the number of observations.

The payment_transaction_duration_seconds histogram records how long each payment transaction takes. Prometheus exposes it as _bucket, _count, and _sum series. When the Prometheus receiver collects these metrics, it converts them into the OpenTelemetry Histogram data model, which can then be exported through OTLP while preserving the information needed to analyze transaction latency and calculate percentiles.

Without histograms, averages obscure latency distributions. If most requests complete in 50ms but 5% take 2+ seconds, the average of 150ms masks that serious performance issue. Histograms capture the complete picture by recording how many observations fall into each latency bucket.

Keeping the distribution allows you to identify changes in latency and investigate issues such as slow database queries, overloaded services, or delays from downstream dependencies.

Actual Prometheus Histogram

This is how your actual /metrics output would look, for example:

And here's the actual OpenTelemetry Histogram representation:

The output above shows the same transaction-duration data represented as an OpenTelemetry Histogram. Instead of three separate Prometheus series, the Collector now has one histogram containing the count, sum, explicit bucket boundaries, and bucket counts.

3. Exporting Metrics via OTLP

Once the metrics have been scraped and processed, the Collector sends them to your observability backend using OTLP. The OTLP exporter sends the processed metrics, including histograms, counters, and gauges, to the backend.

exporters:
  otlp:
    endpoint: "otlp.backend.example.com:4317"
    tls:
      insecure: false

endpoint specifies the backend address for receiving OTLP metrics. This typically points to a central observability platform that aggregates metrics from multiple services across your infrastructure.

tls ensures secure data transmission between the collector and your backend. Set insecure: true only when intentionally connecting to an endpoint that does not use TLS, such as some local development setups

4. Putting the Pipeline Together

We've looked at each part of the pipeline individually. Now let's connect them and follow a metric from the application all the way to the backend.

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp]

This is the complete path our payment_transaction_duration_seconds metric follows, from the FastAPI application to the observability backend. The diagram above shows this flow. The next section walks through actually running it and connecting to SigNoz

Pipeline flow:

The above architecture diagram shows a FastAPI application exposing metrics through /metrics, the OpenTelemetry Collector scraping them with the Prometheus receiver, processing the metrics, and exporting them through OTLP to an observability backend.

5. Running the OpenTelemetry Collector

We'll run the complete pipeline and verify that the transaction metrics make it from the application to SigNoz. Follow these steps (which I'll walk you through in detail below):

1. Set up SigNoz Cloud: Configure the endpoint and ingestion key that the Collector will use.

2. Start the FastAPI application: The application exposes the Prometheus metrics at /metrics.

3. Start the OpenTelemetry Collector: The Collector begins scraping the application's /metrics endpoint using the Prometheus receiver.

4. Generate test transactions: Send requests to the application to create transaction-duration measurements.

5. Verify the Collector and backend: Check the Collector logs to confirm that the pipeline is running, then open SigNoz and verify that the payment_transaction_duration_seconds metric has arrived.

6. Troubleshooting

5.1 Set Up SigNoz Cloud

SigNoz provides the observability backend that will receive the metrics exported by the OpenTelemetry Collector. For this demo, we'll use SigNoz Cloud as the observability backend, so there's nothing to install locally.

First, sign up for a free account at signoz.io. In the SigNoz Cloud dashboard, go to Settings → Ingestion Keys. The page shows your Ingestion URL, Region, and Ingestion Key.

Add these to a .env file in your project directory:

SIGNOZ_INGESTION_KEY=your-real-key
SIGNOZ_OTLP_ENDPOINT=ingest.<your-region>.signoz.cloud:443

Treat the ingestion key like a password and never commit it or share it publicly.

Reference those variables in your Collector configuration:

exporters:
  otlp:
    endpoint: "${SIGNOZ_OTLP_ENDPOINT}"
    tls:
      insecure: false
    headers:
      signoz-ingestion-key: "${SIGNOZ_INGESTION_KEY}"

Then add .env to your .gitignore so the key never gets committed:

echo ".env" >> .gitignore

5.2 Start the FastAPI Application.

Start the application on port 8080:

uvicorn app.main:app --reload --port 8080

Verify that the application is running:

curl http://localhost:8080/

Now inspect the Prometheus metrics:

curl -Ls http://localhost:8080/metrics

The /metrics endpoint exposes the application's Prometheus metrics, including the payment_transaction_duration_seconds histogram that the Collector will scrape.

5.3 Start the Collector

With the application running and the Collector configuration in place, start the Collector:

docker compose up --build

Check the Collector logs to confirm that the pipeline is running and that metrics are being processed.

This builds the payment-api image, starts both containers on the shared telemetry network, and the Collector immediately begins scraping /metrics from the application using the Prometheus receiver, reading your SigNoz ingestion key and endpoint from .env automatically.

5.4 Generate Test Transactions

Generate transactions with different processing times:

curl -X POST "http://localhost:8080/transactions?delay_ms=50"

curl -X POST "http://localhost:8080/transactions?delay_ms=250"

curl -X POST "http://localhost:8080/transactions?delay_ms=2500"

Generate additional requests if you want more observations in the histogram.

These requests create transaction-duration observations that are recorded by the payment_transaction_duration_seconds histogram. The Collector picks up the updated metric during its next scrape.

5.5 Confirm Backend Receipt

Open SigNoz Cloud and search for:

payment_transaction_duration_seconds

The metric should be represented as a histogram, with its bucket distribution _bucket, _sum, and _count available to the backend rather than appearing as unrelated Prometheus series.

5.6 Troubleshooting

If metrics aren't flowing correctly, start by checking the log:

docker logs <collector-container>

Look for connection errors, authentication failures, failed scrape attempts, or configuration errors.

Also verify that:

• The FastAPI application is running.

• /metrics is accessible.

• The Collector can reach the application.

• The SigNoz endpoint and ingestion key are correct.

• The .env variables are available to the Collector.

• The receiver and exporter names match the pipeline configuration.

Use the --dry-run flag if available to validate before deployment.

Conclusion

Prometheus histograms provide a useful view of transaction latency by showing how observations are distributed across different buckets rather than reducing them to a single average.

In this tutorial, we followed payment_transaction_duration_seconds from a FastAPI application's /metrics endpoint through the OpenTelemetry Collector and into SigNoz.

The Prometheus receiver mapped the _bucket, _count, and _sum series into the OpenTelemetry Histogram data model, preserving the distribution of transaction durations for analysis in the backend.

This allows the same metric to move from a Prometheus-instrumented application into an OTLP-based observability platform without manually reconstructing the histogram.

Observability tools, such as logs, metrics, and traces, provide the visibility you need to pinpoint issues quickly. In this handbook, we’ll explore free and open-source tools you can use to make your CI/CD pipelines more reliable. We’ll use practical steps to troubleshoot like a pro – no enterprise licenses required.

Table of Contents

1. Prerequisites

2. Why Observability is Important

3. How to Install and Configure Grafana Loki on Budget Infrastructure

4. How to Implement an ELK Stack Alternative for Pipeline Observability

5. How to Create a Unified Logging Strategy Across Pipeline Components

6. How to Query and Analyze Logs for Effective Troubleshooting

7. How to Set Up Prometheus Metrics Alongside Your Logs

8. How to Create Grafana Dashboards That Combine Metrics and Logs

9. How to Use Exemplars to Jump from Metrics to Relevant Logs

10. How to Diagnose and Fix Common CI/CD Problems

11. How to Implement Advanced Debugging Techniques

12. How to Conduct Effective Postmortems Using Logs

13. How to Optimize Log Storage and Management

14. Conclusion

Prerequisites

There are some things you should know and have to get the most out of this handbook:

Technical Knowledge:

• Basic understanding of CI/CD pipelines (for example, build, test, deploy stages).

• Familiarity with Linux/Unix commands (for example, mkdir, grep, curl).

• Comfortable with Docker basics (for example, docker run, docker-compose up).

• Optional: Awareness of observability concepts (logs, metrics, traces) or YAML configuration.

Software and Tools:

• Docker and Docker Compose: Installed and running (verify with docker --version and docker-compose --version).

• CI/CD Platform: Access to GitHub Actions, Jenkins, or GitLab CI with a sample pipeline that generates logs.

• Text Editor: For editing YAML files (for example, VS Code, Nano).

• Web Browser: To access tool UIs (for example, Grafana on port 3000, Kibana on 5601).

• Optional: curl for testing log forwarding, Git for version control.

Hardware and Infrastructure:

• Machine with:

  • OS: Linux, Windows (with WSL2), or macOS.

  • 4GB RAM (8GB recommended), 20GB free disk space.

  • Stable internet and ability to open ports (for example, 3100 for Loki, 9200 for Elasticsearch).

• Optional: Cloud provider access (for example, AWS, GCP) for scalable setups.

Access and Permissions:

• Admin access to install Docker and configure CI/CD tools.

• Permissions to modify pipeline configs (for example, .github/workflows, .gitlab-ci.yml).

• Optional: Container registry access (for example, Docker Hub) for custom images.

Why Observability is Important

Modern CI/CD pipelines are no longer linear scripts – they are now complex, distributed systems involving multiple tools, environments, and infrastructure layers. One job runs on GitHub Actions, another deploys via Jenkins, and a third builds Docker images in a Kubernetes cluster.

So when something breaks, you’re left chasing logs across tools, guessing where the issue originated, and wasting hours trying to reproduce it.

And worse still, traditional debugging tools often stop at the surface, only showing failed jobs without the context of why they failed or where in the system the fault actually lies.

Observability flips the script. Instead of hunting through disconnected logs or rerunning failed builds blindly, observability gives you insight, not just data. By combining structured logs, metrics, and traces, you can:

• Reconstruct exactly what happened in a pipeline failure

• Trace a failure across CI agents, deployment steps, and containers

• Visualize patterns and anomalies before they become outages

More importantly, observability helps you move from reactive debugging to proactive prevention.

Here’s what you’ll learn about and accomplish in this guide:

• Set up cost-effective observability using Grafana Loki, lightweight ELK, and OpenTelemetry

• Create a unified logging strategy to connect your pipeline

• Write precise queries to quickly pinpoint root causes, correlate logs, metrics, and traces for comprehensive debugging

• Troubleshoot CI/CD issues like build failures, flaky tests, and container crashes

• Build custom dashboards and automated diagnostic tools

• Promote observability through documentation and post-mortems

Whether you're a solo developer or part of a DevOps team, this guide will transform your chaotic CI/CD pipelines into clear, reliable, and observable systems.

How to Choose the Right Observability Tool for CI/CD

Here’s a quick comparison of Grafana Loki, Lightweight ELK, and Vector for CI/CD observability:

How to choose:

• Loki: Ideal for startups or solo devs with limited resources. Integrates well with Prometheus/Grafana.

• ELK: Best for teams needing Kibana’s advanced visualizations or handling large log volumes.

• Vector: Great for lightweight log forwarding in distributed CI/CD setups.

Grafana Loki is a log aggregation system like ELK, but it's more lightweight, and it’s ideal for CI/CD pipelines with limited infrastructure.

How to Install and Configure Grafana Loki on Budget Infrastructure

🛠 Option A: Quick Docker Setup (Recommended for Budget Infra)

1. Create a directory for configuration:

    mkdir -p ~/loki-setup && cd ~/loki-setup
   

2. Create a docker-compose.yml:

    # Defines a Docker Compose setup for Grafana Loki and Promtail to aggregate and scrape logs efficiently.
    version: "3"
   
    services:
      loki:
        image: grafana/loki:2.9.4  # Uses Loki version 2.9.4 for lightweight log aggregation.
        ports:
          - "3100:3100"  # Exposes Loki’s HTTP API port for log ingestion and queries.
        command: -config.file=/etc/loki/loki-config.yaml  # Specifies the configuration file for Loki.
        volumes:
          - ./loki-config.yaml:/etc/loki/loki-config.yaml  # Mounts the local config file into the container.
   
      promtail:
        image: grafana/promtail:2.9.4  # Uses Promtail version 2.9.4 to scrape and forward logs to Loki.
        volumes:
          - /var/log:/var/log  # Mounts the host’s log directory for Promtail to scrape.
          - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml  # Mounts the Promtail config file.
        command: -config.file=/etc/promtail/promtail-config.yaml  # Specifies the configuration file for Promtail.
   

3. Create a basic loki-config.yaml:

    # Configures Grafana Loki for lightweight log storage and querying in a CI/CD environment.
    auth_enabled: false  # Disables authentication for simplicity (not recommended for production).
   
    server:
      http_listen_port: 3100  # Sets the port for Loki’s HTTP API.
   
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: inmemory  # Uses in-memory storage for the ring, suitable for small setups.
          replication_factor: 1  # Sets single replica for minimal resource use.
      chunk_idle_period: 3m  # Flushes chunks to storage after 3 minutes of inactivity.
      max_chunk_age: 1h  # Retires chunks after 1 hour to balance storage and query performance.
   
    schema_config:
      configs:
        - from: 2023-01-01  # Defines the schema start date.
          store: boltdb-shipper  # Uses BoltDB for indexing logs.
          object_store: filesystem  # Stores logs on the local filesystem.
          schema: v11  # Specifies schema version for log storage.
          index:
            prefix: index_  # Prefix for index files.
            period: 24h  # Rotates indexes daily.
   
    storage_config:
      boltdb_shipper:
        active_index_directory: /tmp/loki/index  # Directory for active index files.
        cache_location: /tmp/loki/boltdb-cache  # Cache location for BoltDB.
      filesystem:
        directory: /tmp/loki/chunks  # Directory for storing log chunks.
   
    limits_config:
      enforce_metric_name: false  # Disables strict metric name enforcement for flexibility.
   

4. Create a basic promtail-config.yaml:

    # Configures Promtail to scrape system logs and forward them to Loki.
    server:
      http_listen_port: 9080  # Sets Promtail’s HTTP port for metrics and health checks.
      grpc_listen_port: 0  # Disables gRPC to reduce resource usage.
   
    positions:
      filename: /tmp/positions.yaml  # Stores the position of scraped logs to resume after restarts.
   
    clients:
      - url: http://loki:3100/loki/api/v1/push  # Specifies the Loki endpoint for log ingestion.
   
    scrape_configs:
      - job_name: system  # Defines a scraping job for system logs.
        static_configs:
          - targets:
              - localhost  # Targets the local host for log collection.
            labels:
              job: varlogs  # Labels logs for easy querying in Loki.
              __path__: /var/log/*.log  # Scrapes all log files in /var/log directory.
   

5. Run it:

    # Starts the Loki and Promtail containers in detached mode for background operation.
    docker-compose up -d
   

✨ This brings up Loki and Promtail with minimal resources, no authentication, and logs scraping from /var/log.

Troubleshooting Loki Setup Issues

If Loki or Promtail fails to start, one of the following may be the issue:

1. Container crashes: Check logs with docker logs loki or docker logs promtail. Look for errors like “out of memory” or “port already in use.”

   • Fix: Increase memory (for example, docker-compose.yml resource limits) or change ports (e.g., 3101:3100).

2. Logs not ingested: Verify Promtail is scraping the correct path (/var/log/ci/*.log) using docker exec promtail cat /etc/promtail/promtail-config.yaml

   • Fix: Update __path__ in promtail-config.yaml to match your CI/CD log directory.

3. Resource Constraints: Monitor resource usage with docker stats or top on the host.

   • Fix: Ensure your machine has at least 4GB RAM and 20GB disk space, as specified in the prerequisites.

Configuration for CI/CD Logging

To adapt for CI/CD logs, you should:

1. Configure your CI/CD tools to write logs to disk:

For example, GitHub Actions with a custom runner can write logs to /var/log/gha/*.log.

Update Promtail:

# Configures Promtail to scrape logs from GitHub Actions runners for CI/CD observability.
scrape_configs:
  - job_name: github_actions  # Defines a scraping job for GitHub Actions logs.
    static_configs:
      - targets: ['localhost']  # Targets the local host where the runner writes logs.
        labels:
          job: gha  # Labels logs for identification in Loki queries.
          __path__: /var/log/gha/*.log  # Scrapes logs from the specified directory.

2. Use structured logging (JSON):

Make sure your CI/CD tools or scripts output logs in structured format:

Example:

# Example of a structured JSON log for CI/CD pipelines, enabling easy parsing and querying.
{
  "timestamp": "2025-05-10T13:00:00Z",  # UTC timestamp for log entry.
  "level": "error",  # Log level to indicate severity.
  "job": "deploy",  # Identifies the CI/CD job (e.g., deploy stage).
  "message": "Image pull failed"  # Descriptive message for the error.
}

This helps when querying with LogQL.

How to Connect CI Agents to Loki

This section explains three different ways to get your CI pipeline logs into Loki for monitoring and analysis:

Option 1 – Local setup:

Your CI agents write log files to disk, and Promtail (running on the same machine) reads those files and sends them to Loki.

Option 2 – Using Docker logging driver (Docker containers):

If your CI agents run in Docker containers, you install a special Loki plugin that automatically captures all container output and sends it directly to Loki without needing separate log files.

# Installs the Loki Docker logging driver to send container logs directly to Loki.
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Then run your agent container:

# Runs a CI agent container with the Loki logging driver to forward logs.
docker run --log-driver=loki \
  --log-opt loki-url="http://<your-loki-host>:3100/loki/api/v1/push" \
  my-ci-agent-image

Option 3 – Remote setup:

If you can't install Promtail locally, you can use a log forwarding tool like Fluent Bit or Vector to collect logs and push them to Loki over the network.

The goal: Regardless of which option you choose, you’ll end up with all your CI pipeline logs centralized in Loki, where you can search through them, create dashboards in Grafana, and set up alerts when things go wrong.

It essentially gives you flexibility to integrate log collection based on your infrastructure setup – whether you prefer local agents, Docker plugins, or remote forwarding.

How to Implement an ELK Stack Alternative for Pipeline Observability

When full ELK (Elasticsearch, Logstash, Kibana) is too heavy for your infrastructure, you can go with lightweight setups that achieve similar observability at a lower cost and resource usage.

How to Install Lightweight Versions of Elasticsearch, Logstash, and Kibana

Goal: Stand up a minimal yet functional ELK stack for debugging CI/CD pipelines.

1. Use Docker to spin up lightweight containers

Create a docker-compose.yml:

# Defines a Docker Compose setup for a lightweight ELK stack to aggregate and visualize CI/CD logs.
version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0  # Uses Elasticsearch 7.17.0.
    container_name: elasticsearch
    environment:
      - discovery.type=single-node  # Runs Elasticsearch in single-node mode for simplicity.
      - xpack.security.enabled=false  # Disables security features for lightweight setup.
    ports:
      - "9200:9200"  # Exposes Elasticsearch’s HTTP API port.
    volumes:
      - esdata:/usr/share/elasticsearch/data  # Persists Elasticsearch data.

  logstash:
    image: docker.elastic.co/logstash/logstash:7.17.0  # Uses Logstash 7.17.0.
    container_name: logstash
    ports:
      - "5044:5044"  # Port for receiving logs from Beats.
      - "9600:9600"  # Port for Logstash monitoring.
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf  # Mounts Logstash config file.

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.0  # Uses Kibana 7.17.0 for visualization.
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200  # Links Kibana to Elasticsearch.
    ports:
      - "5601:5601"  # Exposes Kibana’s web UI port.

volumes:
  esdata:  # Defines a volume for persisting Elasticsearch data.

2. Minimal Logstash pipeline configuration (logstash.conf)

// Configures Logstash to process and forward CI/CD logs to Elasticsearch.
input {
  beats {
    port => 5044  // Listens for logs from Filebeat on port 5044.
  }
}

filter {
  json {
    source => "message"  // Parses JSON-formatted log messages for structured data.
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]  // Sends processed logs to Elasticsearch.
    index => "ci-logs-%{+YYYY.MM.dd}"  // Stores logs in daily indexes (e.g., ci-logs-2025.05.14).
  }
}

Troubleshooting ELK Setup Issues

If Elasticsearch, Logstash, or Kibana fails to start, one of the following might be the issue:

1. Container crashes: Check logs with docker logs elasticsearch, docker logs logstash, or docker logs kibana. Look for errors like “insufficient disk space” or “port conflict” (for example, 9200, 5601).

   • Fix: Free up disk space (ensure at least 20GB available) or change ports in docker-compose.yml (for example, 9201:9200).

2. Logs not ingested: Verify Logstash is receiving data from Filebeat or Vector using docker logs logstash. Check the logstash.conf input port (for example, 5044).

   • Fix: Ensure Filebeat or Vector is configured to send to the correct Logstash endpoint (e.g., localhost:5044) and update if needed.

3. Resource constraints: Monitor resource usage with Docker stats or top on the host.

   • Fix: Allocate at least 8GB RAM and 30GB disk space, as Elasticsearch requires more resources than Loki. Adjust memory limits in docker-compose.yml if necessary.

How to Configure Log Shippers for Different CI/CD Components

Goal: Get logs from your pipeline into Logstash or Elasticsearch.

Option 1: Use Filebeat (lightweight log shipper)

Install Filebeat on your CI/CD hosts (GitHub runner, Jenkins node, GitLab runner, and so on).

Filebeat config snippet (filebeat.yml):

# Configures Filebeat to collect CI/CD logs and forward them to Logstash.
filebeat.inputs:
  - type: log  # Specifies log file input.
    enabled: true  # Enables the input.
    paths:
      - /var/log/ci/*.log  # Scrapes logs from the specified CI log directory.

output.logstash:
  hosts: ["localhost:5044"]  # Forwards logs to Logstash on port 5044.

Then run:

# Runs Filebeat with the specified configuration file for log collection.
filebeat -e -c filebeat.yml

Option 2: Use Vector.dev as a more resource-efficient alternative to Filebeat

Vector configuration (vector.toml):

# Configures Vector to collect, parse, and forward CI/CD logs to Elasticsearch efficiently.
[sources.ci_logs]
  type = "file"  # Specifies file-based log collection.
  include = ["/var/log/ci/*.log"]  # Targets CI log files.

[transforms.json_parser]
  type = "remap"  # Uses remap transform to parse logs.
  inputs = ["ci_logs"]  # Processes logs from the ci_logs source.
  source = '''
  . = parse_json!(.message)  # Parses JSON log messages into structured data.
  '''

[sinks.to_elasticsearch]
  type = "elasticsearch"  # Sends logs to Elasticsearch.
  inputs = ["json_parser"]  # Uses parsed logs from the json_parser transform.
  endpoint = "http://localhost:9200"  # Specifies the Elasticsearch endpoint.
  index = "ci-logs"  # Stores logs in the ci-logs index.

Run:

# Runs Vector with the specified configuration file for log processing.
vector -c vector.toml

How to Set Up Index Patterns and Basic Visualizations

Goal: Make CI/CD logs queryable and visual in Kibana.

1. Open Kibana (http://localhost:5601)

• Go to Stack Management → Index Patterns

• Create a new pattern: ci-logs-*

• Choose a time field like @timestamp

2. Visualizations for Common CI/CD Use Cases

• Bar charts: Number of failed vs passed builds per day

• Pie chart: Top error types or most frequent failing test names

• Line chart: Duration of builds over time (if duration is logged)

3. Saved Searches & Dashboards

You can save a search like this:

message: "error" AND job_name: "build"

You can also combine visualizations into a CI/CD Health Dashboard.

How to Create a Unified Logging Strategy Across Pipeline Components

Creating a unified logging strategy across your CI/CD pipeline components ensures that logs are consistent, traceable, and easy to correlate. This helps you quickly debug issues, monitor system health, and trace requests across different tools and services. Let’s discuss some key practices for achieving a unified logging strategy:

Implementing Consistent Log Formats Across Different Tools

Consistent log formats are important for various reasons. First of all, a standardized log format enables easier querying, searching, and visualization. It also helps with correlation of logs from different services. And consistency also ensures that all logs provide necessary details like timestamp, log level, and request context.

There are also some best practices you should follow when formatting logs:

JSON Format is highly recommended as it’s structured, machine-readable, and compatible with many observability tools (for example, Loki, Elasticsearch, Grafana).

There are also some key fields you should include:

• timestamp: The time the log entry was created (preferably in UTC).

• log_level: Indicate whether the log is an INFO, ERROR, DEBUG, and so on.

• service: The service or component generating the log.

• message: A concise description of the event or error.

• correlation_id: A unique identifier for requests to trace logs across systems.

Here’s an example of a consistent log in JSON format:

{
  "timestamp": "2025-05-10T12:34:56Z",
  "log_level": "ERROR",
  "service": "ci_cd_pipeline",
  "message": "Build failed due to missing dependency",
  "correlation_id": "1234567890abcdef"
}

How to Set Up Log Forwarding from GitHub Actions, Jenkins, or GitLab

Log forwarding refers to shipping logs from your CI/CD pipelines to a central spot for easy tracking. It’s helpful because it lets you spot issues fast and debug without digging through scattered files.

For GitHub Actions, you can configure workflows to write logs to a file or send them directly to a log aggregation tool like Loki. In Jenkins, you can use pipeline scripts to forward logs to a log server or file system. Similarly, for GitHub CI, you can add scripts in .gitlab-ci.yml to forward logs to a centralized endpoint.

Using Actions for Outputting Logs:
You can store logs in files and then forward them to a logging system (like Loki or Elasticsearch).
Here’s an example in a GitHub Action workflow:

# Defines a GitHub Actions workflow to run tests and forward logs for observability.
jobs:
  build:
    runs-on: ubuntu-latest  # Uses an Ubuntu runner.
    steps:
      - name: Checkout repository  # Checks out the repository code.
        uses: actions/checkout@v2
      - name: Run tests and log output  # Runs tests and saves output to a log file.
        run: |
          echo "Starting tests..."
          npm test | tee test.log  # Captures test output to test.log.
          # Forwards the log file to a Loki endpoint via HTTP POST.
          curl -X POST -F 'file=@test.log' http://your-loki-endpoint

Log Forwarding with Promtail:
If you are using Grafana Loki for log aggregation, set up Promtail to scrape the logs from the GitHub Actions runner.

Jenkins:

Jenkins logs can be forwarded to external systems (like Elasticsearch or Loki) by using log shippers or plugins.

You can use the Logstash Plugin to forward Jenkins logs to an ELK stack or other systems:

• Install the Logstash plugin on Jenkins.

• Configure the plugin to forward logs to an Elasticsearch server or a logging system of choice.

• In Jenkins, add log forwarding configurations:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        script {
          // Example of forwarding logs to a log server
          sh 'echo "Build successful" | curl -X POST -d @- http://your-log-server'
        }
      }
    }
  }
}

Forward to Loki:
Jenkins supports the loki logging driver for containers if running Jenkins in Docker. You can send logs directly to Loki using this driver:

# Runs a Jenkins container with the Loki logging driver to send logs directly to Loki.
docker run --log-driver=loki --log-opt loki-url=http://loki:3100 jenkins/jenkins:lts

GitLab:

GitLab CI allows logs to be forwarded to external systems for centralized collection and analysis.

Use GitLab CI/CD to Output Logs:
Example in .gitlab-ci.yml:

# Defines a GitLab CI/CD pipeline to run a build and forward logs to Loki.
stages:
  - build
build:
  script:
    - echo "Starting the build" | tee build.log  # Saves build output to build.log.
    - curl -X POST -d @build.log http://your-loki-endpoint  # Forwards the log to Loki.

GitLab Runners:
Configure GitLab runners to forward logs to an external service like Loki or Elasticsearch using log-driver settings or the fluentd log shipper.

How to Add Correlation IDs to Trace Requests Through the System

Why Correlation IDs Are Important:

Correlation IDs allow you to trace a single request as it travels through different services and tools, enabling end-to-end visibility and troubleshooting.

They are critical for debugging distributed systems, especially when different services (for example, CI tool, deployment tool, API service) are involved.

How to Add Correlation IDs:

You can use a UUID (Universally Unique Identifier) or a GUID (Globally Unique Identifier) to generate a unique ID for each request.

If you are using microservices or multiple services in the pipeline, just make sure that the same ID is propagated across each service.

Many logging libraries (for example, winston for Node.js, log4j for Java) support automatic correlation ID generation and logging.

Here’s an example in Node.js (using winston):

// Sets up Winston for structured logging with correlation IDs in a CI/CD pipeline.
const { createLogger, transports, format } = require('winston');
const { printf } = format;

// Creates a logger with a custom format including correlation IDs.
const logger = createLogger({
  format: printf(({ level, message, timestamp }) => {
    return `${timestamp} [${level}] ${message} correlation_id=${generateCorrelationId()}`;
  }),
  transports: [
    new transports.Console(),  // Outputs logs to the console.
  ],
});

// Generates a random correlation ID for tracing requests.
function generateCorrelationId() {
  return Math.random().toString(36).substring(2, 15);
}

// Logs a sample message.
logger.info('Pipeline execution started');

How to Propagate Correlation IDs Between Services:

In CI/CD tools, you can configure your pipeline to inject the correlation ID into logs. For example, in GitHub Actions, you can generate a correlation ID in the env section and propagate it in each job:

# Defines a GitHub Actions workflow that includes a correlation ID for log tracing.
jobs:
  build:
    runs-on: ubuntu-latest  # Uses an Ubuntu runner.
    env:
      CORRELATION_ID: ${{ github.run_id }}  # Uses the GitHub run ID as a correlation ID.
    steps:
      - name: Checkout repository  # Checks out the repository code.
        uses: actions/checkout@v2
      - name: Log build start with correlation ID  # Logs the build start with the correlation ID.
        run: echo "Build started with Correlation ID: $CORRELATION_ID"

Include Correlation IDs in All Logs:

You’ll want to make sure that logs from all components in the pipeline (GitHub Actions, Jenkins, GitLab, deployment tools, and so on) include the correlation ID as part of the log message. This allows you to trace the logs of a single request or pipeline run across different services.

Visualize Your Log Flow

You can create a diagram showing how logs move from your CI/CD tool (for example, GitHub Actions) to Promtail/Vector, then to Loki/Elasticsearch, and finally to Grafana/Kibana for visualization. Use tools like Draw.io to map your pipeline’s observability flow

How to Query and Analyze Logs for Effective Troubleshooting

In this section, you’ll learn how to use LogQL (Loki's query language) to cut through the noise and find the specific logs that matter. Whether you're hunting down a mysterious build failure or tracking deployment issues across multiple services, these query patterns always help.

This bar chart illustrates the CI/CD build performance from May 20 to May 26, 2025. It compares the number of successful builds (in blue) to failed builds (in pink) each day. Successful builds consistently range between 40 and 50, while failed builds peak at 10 on May 23, with other days showing 2 to 8 failures. This indicates a generally stable pipeline with occasional issues.

How to Write Advanced LogQL Queries to Pinpoint CI/CD Issues

LogQL is Grafana Loki's query language, designed for querying logs with a syntax similar to Prometheus’s PromQL. It enables efficient log searches and is particularly useful in troubleshooting CI/CD issues.

Basic LogQL Syntax:

1. Log Streams:

{job="ci_cd", level="error"}

This query retrieves logs where the job label is ci_cd and the level label is error.

2. Log Filters:

{job="ci_cd"} |= "build failed"

The |= operator filters logs to include only those that contain the specified string, for example "build failed".

3. Regular Expressions:

{job="ci_cd"} |~ "error.*timeout"

This uses the |~ operator to filter logs using a regular expression. In this case, it finds logs that contain an "error" followed by "timeout".

Advanced LogQL Queries for CI/CD Issues:

1. Filter Logs for Specific Build Failures:

If your pipeline uses a specific label for build names:

{job="ci_cd", build="build123"} |= "failure"

This finds logs related to the build123 job that contain the word "failure".

2. Using Time Range and Grouping:

To find error logs in the last 15 minutes:

{job="ci_cd", level="error"} | "build failed" | range(start="15m")

To group logs by job and error type:

sum by (job) (count_over_time({job="ci_cd", level="error"}[5m]))

This will return the count of error logs per job, grouped by job name, over the last 5 minutes.

How to Create Pipeline-Specific Queries for Common Failure Patterns

Common Failure Patterns in CI/CD Pipelines:

1. Build Failures:

If your CI system logs contain build errors, you can identify them with:

{job="ci_cd", level="error"} |= "build failed"

You can extend this to filter by specific steps or stages, for example, “test failed”, or “compilation error”.

2. Test Failures:

Logs from your test runner (for example, Jest, Mocha, JUnit) can contain specific failure messages:

{job="ci_cd", stage="test"} |= "test failed"

3. Dependency Issues:

If your pipeline is failing due to missing or conflicting dependencies, look for npm, maven, or docker related errors:

{job="ci_cd", image="node"} |= "npm ERR!"

Or for Maven-related issues:

{job="ci_cd", image="maven"} |= "[ERROR]"

4. Resource Constraints (for example, Out of Memory):

If you experience resource constraints, you might see logs like "OutOfMemoryError":

{job="ci_cd", level="error"} |= "OutOfMemoryError"

Example of combining filters:

{job="ci_cd", level="error"} |= "build failed" |~ "timeout|dependency" | range(start="1h")

This combines log filters for "build failed", matching any logs with the terms "timeout" or "dependency", from the last hour.

How to Set Up Alert Rules Based on Log Patterns

Alerts help detect recurring issues proactively. They notify you when a specific pattern appears in your logs, allowing you to take quick action.

Steps for Setting Up Alerts:

1. Create a Query for the Alert:

First, define the log pattern you want to monitor. For example, an alert for build failures:

{job="ci_cd", level="error"} |= "build failed"

2. Create an Alert in Grafana:

Follow these steps to set up Grafana alerts:

• Go to your Grafana dashboard.

• Choose the panel you want to set the alert on (or create a new panel for this purpose).

• In the panel, click the Alert tab.

• Set the Query field to your LogQL query, such as the one above.

• Under Conditions, define when the alert should trigger, e.g., if the error occurs more than 3 times within 5 minutes.

3. Alert Settings:

Now you’ll want to set up the alert evaluation interval and conditions for triggering the alert (e.g., if the query returns results above a certain threshold).

Here’s an example: Trigger an alert if the number of errors exceeds 5 within 5 minutes:

count_over_time({job="ci_cd", level="error"} |= "build failed"[5m]) > 5

4. Set Alert Notifications:

You can choose where you want the alert to be sent (like to Slack, email, or PagerDuty). And Grafana can be integrated with these systems to send real-time alerts to the right team members.

Example alert query for test failures:

count_over_time({job="ci_cd", stage="test"} |= "test failed"[5m]) > 3

This query triggers an alert if more than 3 test failures are logged within the last 5 minutes.

Kibana Query Language Deep Dive for CI/CD Contexts

Kibana Query Language (KQL) is a powerful tool for searching and filtering logs within Elasticsearch, and it becomes especially useful for debugging CI/CD pipelines.

Basic Query Syntax:

• Field:

    textCopyEditfieldname:value
  

  Example: status: "failure"

• Wildcard: Use * to match any number of characters:

    textCopyEditmessage: "test*"
  

• Range Queries: To search for logs within a specific time frame:

    textCopyEdittimestamp:[2023-05-01 TO 2023-05-15]
  

• Boolean Queries: Combine queries using AND, OR, and NOT:

    textCopyEditstatus: "failure" AND build_id: "12345"
  

Time-Based Queries:

Since CI/CD logs are often tied to time-sensitive operations (builds, deployments), KQL allows you to filter logs by time:

textCopyEdit@timestamp:[now-1d TO now]

Nested Queries (For Complex Pipelines):

CI/CD logs can have nested or multi-level structures (for example, logs within containers). You can query these nested fields:

textCopyEditpipeline.logs.message: "build failed"

Aggregations and Grouping:

You can aggregate logs based on certain fields to identify trends or recurring issues:

textCopyEditterms aggregation on "status" field

This helps identify the most common failure statuses in your pipeline.

Field-Specific Filtering:

When debugging specific components like a build tool or deployment step, you can filter by those component-specific fields:

textCopyEditbuild_tool: "Jenkins" AND status: "failure"

Creating Saved Searches for Recurring Issues

Once you’ve built queries that help you identify common issues in your CI/CD pipeline, you can save them in Kibana for future use.

1. Create a Saved Search:

Run your desired query in the Kibana Discover tab. Click on the “Save” button and give it a meaningful name, such as "Failed Builds - Last Week". You can add filters and customize the time range to match your typical issue patterns.

2. Use Filters to Pinpoint Recurring Problems:

Create saved searches that focus on specific recurring issues like:

• Build failures based on a specific tool or version.

• Test failures within a particular module or set of tests.

Example search for “flaky tests”:

textCopyEdittest_status: "failed" AND error_message: "*timeout*"

3. Saving Multiple Variations:

You can save multiple variations of queries based on different error types or CI/CD tools:

• Failed Jobs: status: "failure"

• Test Failures in Build: log_type: "test" AND status: "failure"

• Resource Constraints: error_message: "*memory*"

These saved searches will allow you to quickly troubleshoot specific issues that occur frequently.

Building Visualizations to Spot Patterns Over Time

Once you have saved searches, Kibana allows you to create visualizations from your data, making it easier to spot trends, anomalies, or patterns over time.

1. Create a Visualization:

Go to the Visualize tab in Kibana. Select the appropriate visualization type. Common visualizations for debugging CI/CD pipelines include:

• Line Chart: Track build failure rates over time.

• Bar Chart: Show the number of failures per CI tool or service.

• Pie Chart: Breakdown of failure reasons (for example, compilation errors, test failures, resource constraints).

2. Track Failure Trends Over Time:

Create a line chart to track build failures over a given period:

• X-Axis: Time (for example, daily or weekly).

• Y-Axis: Count of build failures.

• Aggregation: Date histogram with @timestamp field.

This will help you visualize how build failures are trending, making it easier to identify recurring issues or spikes in failures.

3. Monitor Failure Types by CI Tool:

Create a bar chart that shows the number of failures broken down by CI tool:

• X-Axis: CI tool (Jenkins, GitHub Actions, GitLab, and so on).

• Y-Axis: Count of failures.

• Aggregation: Terms aggregation on the ci_tool field.

This visualization helps identify which CI tool is experiencing the most failures and focus troubleshooting efforts there.

4. Visualize Error Messages by Frequency:

You can visualize which error messages appear most frequently, helping you understand what might be causing recurring issues:

• X-Axis: Error message type.

• Y-Axis: Count of occurrences.

• Aggregation: Terms aggregation on the error_message field.

5. Dashboard for Holistic Monitoring:

Create a dashboard that brings together multiple visualizations. You can have one graph for failure trends, another for failure types (bar chart), and a pie chart showing the percentage of failures caused by different issues. This dashboard gives you a holistic view of your pipeline's health.

Advanced Visualization Techniques:

There are various advanced techniques you can use to dig further into your data.

• Heatmaps: Use heatmaps to spot time-based anomalies in build durations or test failures.

• Anomaly Detection: Kibana has built-in anomaly detection that can be applied to log data to automatically detect patterns that deviate from the norm. This is especially useful for catching rare or unexpected errors in your CI/CD pipeline.

  Example for anomaly detection:

    textCopyEditfield: duration
    aggregation: average
    anomaly detection model: "baseline"
  

How to Set Up Prometheus Metrics Alongside Your Logs

To fully understand your CI/CD pipeline's health and performance, combining metrics and logs is essential. Prometheus is an excellent tool for capturing time-series metrics, and it works seamlessly with Grafana and Loki (or any log aggregation system).

How to Set Up Prometheus for CI/CD Metrics Collection:

1. Install Prometheus:

You can install Prometheus using Docker or Kubernetes for easy deployment.

For Docker-based installation:

docker run -d -p 9090:9090 --name prometheus prom/prometheus

2. Configure Prometheus to Scrape Metrics:

Prometheus needs to be configured to scrape metrics from your CI/CD services.

Edit the prometheus.yml file:

scrape_configs:
  - job_name: 'ci_cd_metrics'
    static_configs:
      - targets: ['localhost:8080', 'localhost:9091']

3. Instrument Your CI/CD Services:

To expose metrics, you need to integrate Prometheus client libraries into your CI/CD services.

For example, to expose build metrics from a Jenkins job, use the Prometheus plugin for Jenkins. In GitHub Actions, you can use Prometheus to expose job metrics.

4. Expose Metrics Endpoint:

You’ll want to make sure your services expose a /metrics endpoint that Prometheus can scrape. For example, use Prometheus client libraries in your application to expose this endpoint.

Troubleshooting Prometheus Setup Issues

If Prometheus fails to start or scrape metrics, here are some things that might be going wrong:

1. Container Crashes: Check logs with docker logs prometheus. Look for errors like “port already in use” (for example, 9090) or configuration parsing issues.

   • Fix: Change the port in docker run (for example, -p 9091:9090) or correct the prometheus.yml file syntax.

2. Metrics Not Scraped: Verify targets are reachable using docker logs prometheus or test with curl http://localhost:9090/targets. Check prometheus.yml for correct endpoints.

   • Fix: Update targets in scrape_configs (for example, localhost:8080) to match your CI/CD service’s metrics endpoint.

3. Resource Constraints: Monitor usage with docker stats or top on the host.

   • Fix: Ensure at least 4GB RAM and 10GB disk space. Increase storage retention or reduce scrape frequency in prometheus.yml if needed.

How to Create Grafana Dashboards That Combine Metrics and Logs

Once Prometheus is collecting metrics, the next step is to visualize and correlate them in Grafana.

How to Integrate Prometheus with Grafana:

First, you’ll need to install Grafana. You can use Docker or Kubernetes for quick deployment:

docker run -d -p 3000:3000 --name grafana grafana/grafana

Next, configure Grafana to use Prometheus as a data source. To do this, log in to Grafana (localhost:3000 by default). Go to Configuration > Data Sources > Add Data Source > Choose Prometheus. Enter your Prometheus server URL (for example, http://localhost:9090) and click Save & Test.

Now it’s time to build a unified dashboard. To do this, create a new dashboard in Grafana that combines both logs (Loki) and metrics (Prometheus).

Add a panel with Prometheus data queries to visualize pipeline metrics like build success rate, deployment duration, and failure count. Use the Graph visualization type for time-series data and Stat for quick summary metrics.

Finally, in the same Grafana dashboard, add panels for logs (from Loki or any other logging system). Use the Logs panel to visualize log data and link them with the relevant Prometheus metrics by using time-based correlations.

Example: If a spike in CPU usage is detected (Prometheus metric), the logs panel could show related logs, like errors or failed build jobs.

How to Use Exemplars to Jump from Metrics to Relevant Logs

Exemplars are an advanced feature in Prometheus that allow you to connect metric data with logs and traces. Grafana supports this feature, and it can be incredibly helpful when investigating issues.

How to Set Up Exemplars in Prometheus:

1. Enable Exemplars in Your Application:

Exemplars are essentially traces embedded into your metrics. To use them, you’ll need to make sure your application is instrumented to send exemplar data alongside your metrics.

Many libraries support adding exemplars to Prometheus metrics, such as prom-client (Node.js) and prometheus-net (C#).

Here’s an example in Node.js:

// Demonstrates adding an exemplar to a Prometheus metric for linking to logs or traces.
const promClient = require('prom-client');

// Creates a counter metric to track failed CI/CD builds.
const counter = new promClient.Counter({
  name: 'ci_cd_failed_builds_total',  // Metric name for failed builds.
  help: 'Total number of failed builds',  // Description of the metric.
});

// Increments the counter with an exemplar for tracing.
counter.inc({ exemplar: 'build_failed' });

2. Enable Exemplars in Prometheus Config:

Make sure your Prometheus server is configured to store and expose exemplars. Exemplars are typically included with histogram or summary metrics, so make sure you’ve configured them correctly.

3. Visualizing Exemplars in Grafana:

In Grafana, when you query Prometheus for metrics with exemplars, Grafana will show the linked logs or traces when you hover over a metric.

Use the Exemplar option in Grafana panels to quickly access logs from specific metrics.

For example, if you have a build_failure_total metric and you detect a failure in your pipeline, you can click on the failure metric in Grafana and instantly view the relevant logs for that specific failure using the exemplars.

How to Diagnose and Fix Common CI/CD Problems

CI/CD pipelines often encounter issues like build failures, dependency problems, and flaky tests that can disrupt development workflows. This section provides practical strategies to diagnose and resolve these common problems using log analysis and systematic debugging techniques, helping you restore pipeline stability quickly.

Strategy 1: Systematically Debug Build Failures

Build failures are a frequent CI/CD challenge, often stemming from errors in code, tests, or configurations. Systematically debugging these issues involves analyzing logs to pinpoint root causes, using the following approaches.

Identifying Patterns in Compiler and Test Output

When debugging build failures, you need to first examine the logs from the compiler and test outputs. Let’s go over some key strategies.

1. Check for Specific Error Messages:

There are a few common types of error messages you might get. They are:

• Syntax errors: Look for lines indicating that there's a mismatch in syntax, such as missing semicolons, undeclared variables, or incorrect function calls.

• Linker errors: These often occur when the required libraries or dependencies are not found. You'll typically see errors like undefined reference or symbol not found.

• Build tool errors: If you are using build systems like Maven, Gradle, or MSBuild, their logs will give specific error codes or missing configurations.

2. Look for Common Error Patterns:

Often, failed builds repeat the same error or pattern across multiple runs. Check logs for recurring terms or errors that point to specific modules or functions. And remember that grouping similar issues can help you identify the root cause faster.

3. Use Regular Expressions for Log Filtering:

You can use regular expressions to search for keywords in the logs that match common failure patterns (for example, "error", "failed", "exception", "out of memory"). This will help you filter out unrelated messages and focus on the failures.

As an example:

• If the build fails with an "Out of Memory" error, search for any memory allocation issues or settings that can be increased.

• If test failures are related to specific modules, inspect those modules for recent changes or dependency issues.

Strategy 2: Troubleshooting Dependency Issues with Log Analysis

Dependency issues are common in build failures, especially in complex CI/CD pipelines with multiple modules or services. To resolve these issues, consider the following:

1. Check for Missing or Outdated Dependencies:

Start by reviewing the build tool’s output to check for messages related to missing dependencies (for example, dependency not found, version conflict).

Many build tools (like Maven, npm, or .NET) will include specific error messages when a dependency is missing or incompatible.

2. Inspect Dependency Resolution Logs:

Some build tools provide detailed logs showing how dependencies were resolved (for example, the version of a library that was used). These logs can show you if there’s a version mismatch.

Make sure that your package.json (for JavaScript projects), pom.xml (for Java), or csproj (for C#) files are correctly defined with compatible versions.

3. Verify Network Connectivity:

CI/CD tools sometimes fail to fetch dependencies due to network issues (for example, proxy settings, repository access). Look for any errors indicating that a repository couldn’t be reached.

4. Log Example:

If a Java project fails with Could not find artifact, it's likely a dependency missing or inaccessible. Check the repository URL or if the artifact exists in your Maven repo.

5. Resolve Version Conflicts:

Version conflicts occur when different dependencies require incompatible versions of the same library. This is especially true in Java (with Maven/Gradle) and .NET projects. Consider using tools to resolve version conflicts automatically or define compatible versions manually.

Fixing Flaky Tests Based on Historical Log Data

Note: Issues like container crashes, logs not ingested, or resource constraints here may resemble those in other sections. These are common across CI/CD services and processes, but each section offers unique context to avoid redundancy.

Flaky tests – that is, those that pass sometimes and fail at other times – are common in CI/CD pipelines, and they can be frustrating. Let’s discuss some strategies for how you can tackle them:

1. Analyze Test Logs Over Time:

Review historical logs to identify patterns in when the test fails. Look for timing issues, resource limits, or external dependencies that could affect test reliability.

For example, if a test intermittently fails after a certain amount of time or only during specific pipeline stages, it could indicate resource exhaustion or race conditions.

2. Check Test Dependencies:

Often, flaky tests are dependent on external services or resources (for example, databases, APIs, file systems). Check if these services are consistently available and properly mocked during test execution.

Logs that mention failed connections to external services or unstable environments can give you insights into potential issues with dependencies.

3. Run Tests with Increased Logging:

Increase the verbosity of test logs to capture more information about the failures. This can help you detect why tests fail in certain conditions.

For example, adding debug logs inside tests can provide more context on the state of the application when the failure occurs.

4. Time of Day Issues:

Some flaky tests may fail during peak usage times, especially if they rely on shared resources. Look for patterns that correlate with resource contention (for example, database locks, API rate limits).

Logs showing high CPU or memory usage can indicate that resource constraints are affecting the stability of your tests.

5. Implement Retry Logic for Flaky Tests:

To mitigate the effects of flaky tests, implement automatic retries for tests that fail intermittently. This can help reduce the noise in your CI/CD pipeline while you investigate the root causes.

For example, if a database connection test fails intermittently, you may want to inspect database logs for signs of timeouts or connection pool exhaustion.

How to Resolve Deployment Pipeline Failures

Deployment pipeline failures can stem from several sources, and diagnosing them requires a systematic approach using logs and available observability tools. Below, we will outline the common patterns in logs that indicate resource constraints, permission/authentication issues, and configuration drift between environments.

Log Patterns That Indicate Resource Constraints

Resource constraints are a common cause of pipeline failures. These can include CPU limits, memory usage, or disk space running out. Here's how to recognize these patterns:

Key Indicators in Logs:

• Memory Issues: Look for messages like "out of memory", "memory limit exceeded", or "OOM killed" in your logs. Here’s an example in Kubernetes logs:

pod has been OOMKilled

• CPU Limits: Watch for logs showing that a process exceeded CPU limits or was throttled. Here’s an example:

process 'foo' hit CPU limit, throttling at 100%

• Disk Space: Logs may show file write errors or messages about a disk being full. Here’s an example:

Unable to write to file, disk space is full.

You can resolve the memory issues by increasing the allocated memory for your containers, VM, or cloud instances.

You can resolve the CPU issues by adjusting CPU limits or scaling your infrastructure to add more resources.

And finally, you can resolve disk space issues by cleaning up unused files or increasing disk capacity on the server/container.

Identify Permission and Authentication Issues

Permission and authentication issues often result in pipeline failures due to a lack of access to necessary resources or services. These issues might occur when you’re trying to access databases, deploy to cloud services, or authenticate third-party APIs.

There are some key indicators in the logs that you can look out for:

1. Authentication Failures:

Look for messages related to failed logins, incorrect credentials, or invalid tokens.

Here’s an example:

Authentication failed for user 'admin'

Invalid API token provided.

2. Permission Denied:

Logs may indicate that the CI/CD pipeline lacks the permissions to perform a certain action.

Here’s an example:

Access denied for /path/to/deployment/target

Unauthorized request to cloud service.

How to resolve these errors:

• Credentials: Ensure the credentials (API keys, access tokens, SSH keys) used in the pipeline are up-to-date and correctly configured.

• Permissions: Review and update the role-based access control (RBAC) settings for the service account running the pipeline to ensure it has the necessary permissions.

• Secrets Management: Use tools like Vault, AWS Secrets Manager, or Azure Key Vault to securely manage secrets and credentials.

Troubleshooting Configuration Drift Between Environments

Configuration drift occurs when different environments (like development, staging, production) are not synchronized. This can lead to inconsistent behavior during deployments, and often results in failures in one environment but not in others.

Look out for these key indicators in the logs:

1. Mismatch in Environment Variables:

If you’re using environment variables, check for discrepancies across different stages. For example:

Environment variable DATABASE_URL not found in production

2. Dependency Versions:

Mismatched versions of dependencies between environments can cause unexpected issues.

Here’s an example:

Error: Dependency 'libxyz' version mismatch between environments

3. Service Configuration:

Look for configuration-related errors that might not be present in a development environment but occur in production.

Here’s an example:

Error: Invalid config in 'production-config.yaml'

How to resolve these errors:

• Use Infrastructure as Code (IaC): Tools like Terraform, Ansible, or CloudFormation can help ensure that environments are provisioned consistently.

• Automated Configuration Management: Use CI/CD pipeline steps to automate environment setup to avoid manual changes that can cause drift.

• Environment Consistency Checks: Implement checks to compare configurations and dependencies across environments before deployment.

  • Example: You can add a pre-deployment stage to run a script that compares environment variables, configurations, and dependency versions between staging and production.

• Configuration Management Tools: Use configuration management tools like Chef, Puppet, or SaltStack to maintain consistent configurations across environments.

How to Debug Container-Based Deployment Issues

Debugging container-based deployment issues requires specialized tools and techniques to trace errors in containerized environments. Below are strategies to efficiently collect logs, diagnose failures, and use ephemeral containers for investigation.

Collecting and Analyzing Container Logs Effectively

Container logs are essential for troubleshooting issues, and effective collection and analysis can significantly speed up the debugging process.

Here’s how you can collect container logs:

1. Docker Logs:

You can use Docker’s logs command to view logs of a specific container:

docker logs <container_name_or_id>

If your container uses a logging driver (like json-file or fluentd), ensure that logs are being written to an accessible location.

2. Kubernetes Logs:

For Kubernetes-managed containers, use kubectl to access pod logs:

kubectl logs <pod_name>

To view logs for all containers in a pod:

kubectl logs <pod_name> --all-containers=true

3. Log Aggregation:

You can integrate with centralized logging systems (like, Grafana Loki, Elastic Stack). You can also use Fluentd or Logstash as log shippers for forwarding logs from containers to a logging backend.

Analyzing Logs:

1. Filter and Search Logs:

Use grep to filter logs for specific error messages or patterns:

docker logs <container_name> | grep "ERROR"

In Kubernetes, you can combine kubectl with grep or other tools for advanced filtering.

2. Log Contextualization:

Include metadata in your logs (for example, container ID, environment, timestamps) for easier debugging. Ensure logs are structured in formats like JSON to allow for better querying and filtering.

How to Diagnose Image Pull and Networking Failures

Container deployment failures often stem from issues related to image pulling or network connectivity. Here’s how to troubleshoot these problems:

Image Pull Failures:

There are some common issues you might see, such as:

• Authentication failures: If the container registry requires authentication, ensure your credentials (username/password or tokens) are correct.

• Network connectivity: Check if the container can access the registry endpoint. Often, firewalls or DNS issues block the image pull.

• Image not found: Verify the image name and tag are correct. Use docker pull to manually pull the image to see if the issue is specific to the deployment process.

There are various ways to diagnose them:

For Docker, use:

docker pull <image_name>

This will output the specific error message if the image pull fails.

For Kubernetes, check the event logs for the pod:

kubectl describe pod <pod_name>

Look for the Failed status under "Events" for information about why the image pull failed (for example, wrong credentials or tag). If the issue is with the registry authentication, configure the Kubernetes imagePullSecrets or Docker's credentials to ensure the correct access.

Networking Failures:

Some common issues you may encounter are:

• DNS resolution problems: Containers may fail to resolve hostnames if DNS configurations are incorrect.

• Network policies and firewall rules: Network policies or firewalls may block necessary ports.

• Inter-container communication: If containers need to talk to each other, ensure they’re on the same network or subnet.

Again, there are various ways to diagnose these issues:

For Docker networking:

You can do this to view all Docker networks:

docker network ls

You can also inspect the network of your container like this:

docker network inspect <network_name>

Check if the container is correctly attached to the network and if necessary ports are exposed.

For Kubernetes Networking:

You can use kubectl to check network policies:

kubectl get networkpolicies

You can also check the pod’s network settings like this:

kubectl describe pod <pod_name> | grep -i "Network"

Testing Connectivity Inside Containers:

For Docker, exec into the container and test:

docker exec -it <container_id> /bin/bash
ping <hostname_or_ip>
curl http://<service_address>:<port>

In Kubernetes, use kubectl exec to access the pod and test connectivity:

kubectl exec -it <pod_name> -- /bin/bash

How to Use Ephemeral Debug Containers for Investigation

Ephemeral debug containers are short-lived containers that help investigate issues in a running environment without altering the main application container.

What are Ephemeral Debug Containers?

Ephemeral debug containers allow you to run diagnostic commands (like shell access, ping, or curl) in the same network environment as the failing application container, without modifying the application itself.

How to Set Up Ephemeral Containers in Docker:

1. Use the docker run Command:

You can create a new container for debugging by running a container with the same network settings as the failing container:

docker run -it --network container:<container_name_or_id> --entrypoint /bin/bash <debug_image>

This command runs an interactive shell inside the debug container using the same network as the target container.

Ephemeral Containers in Kubernetes:

Kubernetes allows you to inject an ephemeral debug container into a running pod. You can add a temporary debug container to your pod using the following command:

kubectl debug <pod_name> -it --image=<debug_image> --target=<container_name>

This command will run a new container in the same pod as the target container, allowing you to run diagnostic commands.

Example use cases are investigating file systems, running network diagnostics, checking configuration files, and so on.

These debug containers are meant to be temporary and can be discarded after the issue is resolved.

How to Implement Advanced Debugging Techniques

This section covers advanced methods to diagnose complex CI/CD pipeline issues that standard log analysis might miss. We’ll explore distributed tracing to track requests across multiple services and combine traces with logs and metrics for deeper insights.

These techniques are designed to work within budget constraints, ensuring effective debugging for your CI/CD workflows.

Choosing a Tracing Backend for CI/CD

Distributed tracing enables you to monitor a request’s path through various services in your CI/CD pipeline, such as from a build step to a deployment, identifying delays or failures. Choosing a tracing backend involves selecting a tool to store and analyze these trace data. Below, we compare Jaeger, Tempo, and hosted solutions for distributed tracing.

When to choose each one:

• Jaeger: Ideal for quick, local tracing setups with minimal overhead.

• Tempo: Best for teams already using Grafana Loki/Prometheus for unified observability.

• Hosted Solutions: Suited for large-scale pipelines needing managed scalability.

How to Set Up Distributed Tracing on a Budget

Distributed tracing is crucial for debugging and observing complex, multi-step operations across services. It allows you to follow requests as they propagate through different services and components of your pipeline. Implementing this on a budget can still provide valuable insights.

How to Use OpenTelemetry with Free Backends

OpenTelemetry is an open-source framework that enables you to collect, process, and export telemetry data like traces and metrics. It supports multiple backends, and we’ll focus on using free, budget-friendly backends for trace storage and analysis.

1. Install OpenTelemetry Collector:

OpenTelemetry provides an agent (collector) that collects traces and metrics from your application and sends them to a backend.

To install the OpenTelemetry Collector, download the binary for your OS or use Docker to deploy it:

docker pull otel/opentelemetry-collector:latest

Then run the OpenTelemetry Collector in Docker with a configuration file:

docker run -d --name opentelemetry-collector -p 55680:55680 -p 14250:14250 otel/opentelemetry-collector

2. Configure OpenTelemetry to Export to Free Backends:

There are a few popular free backends you can use for distributed tracing, like Jaeger and Prometheus + Tempo. Let’s see how to use both here.

We’ll start with Jaeger, an open-source tracing backend. It’s highly scalable and works well with OpenTelemetry.

You can use the Docker version for easy deployment:

docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775 -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14250:14250 -p 14268:14268 -p 14250:14250 -p 9431:9431 jaegertracing/all-in-one:1.30

Alternatively, you can use hosted services like Lightstep, AWS X-Ray, or Honeycomb for cloud-native environments.

Now let’s see how to use Prometheus + Tempo for logs and metrics correlation.

Tempo is a distributed tracing backend built by Grafana that integrates well with other Grafana tools (Loki and Prometheus).

You can install Tempo using Docker:

docker run -d --name tempo -p 14268:14268 grafana/tempo:latest

3. Instrument Your Code with OpenTelemetry SDK:

For Python/Node.js/Java/Go applications, you can install the appropriate OpenTelemetry SDK and start tracing.

Here’s a Python example:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation

And a Node.js example:

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/instrumentation

And one in Java:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.0.0</version>
</dependency>

After installation, you can use the OpenTelemetry SDK to instrument the application and start collecting traces for HTTP requests, database queries, and other pipeline interactions.

4. Send Data to the Collector:

You can configure the SDK to send trace data to your OpenTelemetry Collector, which will then forward it to your backend (Jaeger, Tempo, and so on). Here’s an example for Python:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchExportSpanProcessor

trace.set_tracer_provider(TracerProvider())
exporter = OTLPSpanExporter(endpoint="http://localhost:55680")
processor = BatchExportSpanProcessor(exporter)
trace.get_tracer_provider().add_span_processor(processor)

If traces aren’t appearing, several issues might be occurring:

1. Collector fails to start: Check logs with docker logs otel-collector. Look for errors like “port conflict” or “invalid config.”

   • Fix: Change ports (for example, 55681:55680) or verify the config file.

2. No traces in Jaeger: Ensure the collector is sending data to Jaeger (http://localhost:14250). Test with curl http://localhost:55680.

   • Fix: Update the exporter endpoint in your SDK configuration.

3. Resource constraints: Monitor usage with docker stats.

   • Fix: Allocate at least 2GB RAM and 10GB disk space for the collector and backend.

Correlating Traces with Logs and Metrics

Combining traces with logs and metrics provides a holistic view of your pipeline’s operations, allowing you to pinpoint the root cause of issues more effectively.

OpenTelemetry and Grafana allow you to link traces, logs, and metrics into a unified view.

Let’s see how you can do this now.

1. Link Logs and Traces Using Correlation IDs:

When generating logs, include trace and span IDs in the log entries. This allows you to correlate logs with specific trace requests.

Here’s an example:

{
  "timestamp": "2025-05-10T12:00:00Z",
  "level": "error",
  "message": "Build failure",
  "trace_id": "1234567890abcdef",
  "span_id": "0987654321abcdef"
}

2. Integrating Logs (Loki) with Traces (Jaeger/Tempo) in Grafana:

Grafana can integrate traces from Jaeger or Tempo and correlate them with logs from Loki.

To do this:

1. Set up Loki and Tempo in Grafana.

2. In Grafana’s Explore view, you can search logs and traces side-by-side.

3. Create dashboards that show metrics, logs, and traces for a complete view of a request flow.

3. Using Prometheus Metrics with Traces:

Prometheus provides metrics that can be correlated with traces. For example, you can use exemplars in Prometheus to link specific metric data to trace data.

Example: If you have a high error rate in your build step, you can correlate this with trace data to identify which requests failed.

Creating Trace Visualizations for Complex Pipeline Operations

You can visualize traces with Jaeger or Tempo.

To do this in Jaeger:

Once your traces are in Jaeger, you can access the Jaeger UI (http://localhost:16686 by default) and use the search functionality to explore traces based on service name, trace ID, or specific operations.

Jaeger allows you to create custom dashboards to visualize the latency, throughput, and errors of requests across services.

To do this in Tempo (Grafana Integration):

Tempo integrates with Grafana, where you can create dashboards that visualize trace data from your pipeline.

Create a Grafana dashboard:

1. Add Tempo as a data source in Grafana.

2. Use the "Trace" panel to query and visualize traces.

3. Combine trace visualizations with metrics (from Prometheus) and logs (from Loki) to get a unified view of your pipeline.

A typical trace visualization dashboard could show the duration of each step in your pipeline (build, test, deploy) and highlight where delays or errors occur, such as slow database queries or flaky tests.

Troubleshooting Tempo Setup Issues

If Tempo fails to collect or display traces:

1. Container fails to start: Check logs with docker logs tempo. Look for errors like “port already in use” (for example, 14268) or “storage backend unavailable.”

   • Fix: Change ports in the Docker command (for example, -p 14269:14268) or ensure the storage directory (for example, /tmp/tempo) exists and is writable.

2. No traces in Tempo: Verify the OpenTelemetry Collector is sending traces to Tempo’s endpoint (http://localhost:14268). Test connectivity with curl http://localhost:14268.

   • Fix: Update the collector’s exporter configuration to point to the correct Tempo endpoint, and ensure no firewalls are blocking the connection.

3. Resource constraints: Monitor usage with docker stats or top on the host.

   • Fix: Allocate at least 2GB RAM and 10GB disk space for Tempo, as tracing data can grow quickly with high-volume pipelines.

This bar chart displays the average latency (in milliseconds) for key stages of a CI/CD pipeline in May 2025. The Build stage averages around 1,200 ms (blue), the Test stage around 800 ms (yellow), and the Deploy stage around 1,500 ms (pink), highlighting that deployment is the most time-intensive step.

How to Build Comprehensive Debugging Dashboards

This section explains how to create Grafana dashboards to troubleshoot CI/CD pipeline issues effectively. We’ll focus on setting up visualizations for key metrics, logs, and system resources to identify problems like build failures or resource bottlenecks, using budget-friendly tools to keep your observability stack lean and actionable.

Designing Grafana Dashboards Specifically for Troubleshooting

Step 1: Understand the Key Metrics and Logs to Monitor

When designing a Grafana dashboard for debugging, you should focus on metrics and logs that help identify issues in the pipeline. These could include:

• Build failures: Errors during build processes (compilation, test failures).

• Deployment failures: Issues in deployment, such as failed jobs, resource limitations, or misconfigurations.

• Container logs: Information about container status and logs (if using containers in your pipeline).

• System resource usage: CPU, memory, and disk usage that may lead to performance bottlenecks.

• CI/CD-specific metrics: Number of successful vs. failed pipeline runs, job duration, job queue times.

Step 2: Set Up Data Sources

To start building the dashboard, you’ll need to set up your data sources in Grafana. First, connect your Prometheus instance for collecting metrics. To do this, go to Configuration > Data Sources in Grafana. Then just add Prometheus as a data source and enter the URL (for example, http://localhost:9090).

Next, you need to connect your Loki instance for logs. So go ahead and add Loki as a data source by specifying the URL (for example, http://localhost:3100).

Note that if you're using other sources like InfluxDB or Elasticsearch, you’ll need to make sure that they’re properly connected as data sources.

Step 3: Create Panels and Visualizations

Now that your data sources are connected, you can start building your dashboard with the following panels:

• Build Status Panel:

  • Create a stat panel or gauge panel to show the success/failure ratio of pipeline runs.

  • Query Prometheus or Loki for data like build status (success or failure), number of errors, and job durations.

• Error Breakdown Panel:

  • Use a pie chart to visualize the types of errors (for example, build, deployment, or system resource failures).

  • Query the logs in Loki to break down error types based on the CI tool (for example, Jenkins, GitHub Actions).

• Resource Utilization Panel:

  • Use time series graphs to monitor CPU, memory, and disk usage over time, especially for resource-heavy builds or deployments.

• Job Duration Panel:

  • Use bar charts or line graphs to track the average duration of jobs over time. Set thresholds for warning signs if a job takes longer than expected.

Troubleshooting Grafana Dashboard Issues

If Grafana dashboards fail to display data or show errors, you might be having one of these issues:

1. Missing data sources: If metrics, logs, or traces aren’t appearing, verify data source connections in Grafana (for example, Prometheus, Loki, Tempo). Check under Configuration > Data Sources.

   • Fix: Ensure the data source URLs are correct (for example, http://localhost:9090 for Prometheus) and test the connection. Re-add the data source if needed.

2. Incorrect Trace IDs: If trace visualizations (for example, Tempo panels) show no data, confirm that trace IDs in logs match those in Tempo. Use a query like {job="ci_cd"} | json | trace_id="1234567890abcdef" in Loki to cross-check.

   • Fix: Ensure your application logs include trace and span IDs, and verify the OpenTelemetry SDK is correctly instrumented to send traces to Tempo.

3. Resource Constraints: Monitor Grafana’s resource usage with docker stats if running in a container, or top on the host.

   • Fix: Allocate at least 4GB RAM and 10GB disk space for Grafana, especially when rendering complex dashboards with multiple data sources.

How to Set Up Drill-Down Paths from High-Level to Detailed Views

Step 1: Create High-Level Overview Panel

At the top of the dashboard, include a high-level overview panel that summarizes the overall status of the pipeline. This could be:

• Success/Failure Count: A simple stat panel showing the count of successful vs. failed runs.

• Pipeline Health Status: Display an overall health check of your pipeline using color-coded indicators (green for healthy, red for issues).

Step 2: Set Up Drill-Down Links

To allow users to drill down from high-level information to detailed views:

1. Link to detailed build information:

You can create a time series graph that shows build job durations. Add a link to a detailed log view when clicking on a failed job.

For example, when clicking a failed build, you can link to a detailed panel or a separate dashboard that shows the logs and error messages related to that specific run.

2. Link to Logs in Loki:

You can use Loki's LogQL queries to set up a drill-down path. When users click on an error type or a specific job name, it should automatically filter logs for that job or error type.

You can set up drill-down interactions using Dashboard Links in Grafana. In the panel settings, under Links, specify the link to another dashboard that shows detailed logs filtered by the job name or failure type.

Step 3: Implement Time Range Filters

To enhance drill-down functionality, you can add a time range filter to allow users to adjust the time window for both logs and metrics. This enables them to zoom in on a specific time frame where failures occurred.

How to Create Shared Dashboards for Team Troubleshooting

Step 1: Share Your Dashboard

Once your dashboard is designed, you can share it with your team for collaborative troubleshooting:

First, you’ll want to make sure that the correct permissions are set up for your team. You can define specific roles in Grafana with access to the dashboard. Go to Dashboard Settings > Permissions, and grant view or edit access to users or teams.

Next, you can directly share a link to the dashboard with your team members. Use the Share option in the top-right corner of the dashboard, which provides a direct URL and also options to embed the dashboard into other tools (for example, Slack, email).

You can also use template variables to allow users to filter and adjust the dashboard for different pipeline runs or environments. For example, add a variable for build_id, job_name, or branch_name that allows users to select specific builds or branches for more granular troubleshooting.

Step 2: Set Up Alerting

To ensure your team is notified of any pipeline failures, you can set up alerting rules. There are a few important ones you’ll want to set up.

First, create alerts for critical issues, like when a pipeline fails or exceeds expected resource usage. This could be for things like build time exceeding a threshold or failure of a deployment stage.

Grafana can send alerts via various channels such as Slack, email, or webhook.

You can also integrate your dashboards with tools like Slack or Teams for real-time notifications and collaboration. Set up automated messages for your team when the dashboard indicates an issue.

How to Create Automated Diagnostic Tools

Building Scripts that Collect Relevant Logs During Failures

To automate log collection during failures, you need scripts that can capture logs from different CI/CD stages and services as soon as a failure is detected. Here are the steps you can follow to do this:

1. Write Failure Detection Script:

You can leverage the exit status codes of your CI/CD tools to detect failures. For example, in GitLab CI/CD or GitHub Actions, you can check if the last command failed by inspecting $? in Unix-based systems.

# Example for GitLab CI/CD
if [ $? -ne 0 ]; then
    echo "Failure detected, collecting logs..."
    # Custom log collection script call
    ./collect_logs.sh
fi

2. Log Collection Script (collect_logs.sh):

The script should collect relevant logs, system metrics, and trace information. For instance:

#!/bin/bash
LOG_DIR="/path/to/logs"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="${LOG_DIR}/backup/${TIMESTAMP}"
mkdir -p $BACKUP_DIR

# Collect logs from CI/CD agents, containers, or system logs
cp /var/log/ci_cd/*.log $BACKUP_DIR/
cp /path/to/docker_logs/*.log $BACKUP_DIR/
# Collect metrics or traces from monitoring systems if needed

3. Use CI/CD Artifacts:

For platforms like GitLab, GitHub Actions, or Jenkins, you can upload logs as artifacts for further investigation. Configure these platforms to save logs in case of a failure.

Here’s an example for GitHub Actions:

steps:
  - name: Run Tests
    run: |
      npm run test
  - name: Upload logs if test fails
    if: failure()
    uses: actions/upload-artifact@v2
    with:
      name: test-logs
      path: /path/to/test/logs

4. Centralized Logging:

Instead of manually collecting logs, you can centralize log storage using logging systems like Grafana Loki, ELK stack, or even cloud-based solutions. This will ensure that logs are accessible even if they are overwritten or lost on individual systems.

How to Implement Automatic Analysis of Common Error Patterns

Once logs are collected, you can automate the analysis process by defining common error patterns and automatically searching for them in your logs.

Step 1: Define Error Patterns:

Establish error signatures or patterns that are common in your CI/CD process, such as failed builds due to missing dependencies, permission issues, or network timeouts.

You can use regex or regular expressions to capture these patterns. Here’s an example – define a regex for failed test patterns:

TEST_FAILURE_REGEX=".*FAILURE.*"

Step 2: Create Log Analysis Script:

Next, you can write a script that scans logs for these common patterns. The script could then categorize or flag errors.

Here’s an example using grep to detect failure patterns:

#!/bin/bash
LOG_DIR="/path/to/logs"
ERROR_LOG="${LOG_DIR}/error_patterns.log"
touch $ERROR_LOG

# Define error patterns to search for
ERROR_PATTERNS=("FAILURE" "ERROR" "TIMEOUT")

for PATTERN in "${ERROR_PATTERNS[@]}"; do
    grep -i $PATTERN $LOG_DIR/*.log >> $ERROR_LOG
done

if [ -s $ERROR_LOG ]; then
    echo "Error patterns found, review the log file."
fi

Step 3: Automate Alerting:

Once an error pattern is detected, you can integrate the log analysis script with your alerting system (for example, sending an email or Slack notification).

Here’s an example of sending a Slack notification:

if [ -s $ERROR_LOG ]; then
    curl -X POST -H 'Content-type: application/json' \
         --data '{"text":"Error detected in CI pipeline. Check error log."}' \
         https://hooks.slack.com/services/YOUR_SLACK_WEBHOOK_URL
fi

Step 4: Use Observability Tools for Pattern Recognition:

Leverage observability tools (Grafana Loki, Prometheus) that support log querying and visualization. You can create dashboards that automatically detect anomalies like high failure rates or recurring errors.

Example: Set up a Grafana dashboard with alert rules based on log frequency.

How to Create Self-Healing Pipelines Based on Known Issues

Self-healing pipelines can automatically address issues when they are detected by executing pre-defined corrective actions. Let’s walk through how you can set one up.

Step 1: Define Common Failures and Solutions:

Identify recurring issues (for example, dependency issues, build timeouts, flaky tests) that occur in your pipeline. Then, define self-healing actions to mitigate these issues.

Here’s an example of automatically retrying a failed step if it is a known flaky test:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Run Tests
        run: |
          npm run test
      - name: Retry Tests if Failed
        if: failure() && (steps.tests.outcome == 'failure')
        run: |
          echo "Retrying tests..."
          npm run test

Step 2: Automatic Rollbacks:

Set up a rollback process for failed deployments. For instance, if a deployment to production fails, the pipeline can automatically revert to the last successful build.

Example in GitLab CI/CD:

deploy_production:
  script:
    - ./deploy.sh
  when: on_failure
  retry: 3

Step 3: Build Self-Healing Logic Using Retry Mechanisms:

Implement retry logic for transient issues (like network glitches) that often cause failures.

Example of retrying a step in GitHub Actions:

steps:
  - name: Retry Deployment
    run: |
      attempts=0
      max_attempts=3
      until [ $attempts -ge $max_attempts ]
      do
        deploy_script && break
        attempts=$((attempts+1))
        echo "Attempt $attempts failed. Retrying..."
        sleep 5
      done

Step 4: Automate Corrective Actions for Dependency Issues:

Set up automatic fixes for dependency-related failures, like clearing caches or re-installing dependencies:

if [[ $(cat error.log) =~ "dependency not found" ]]; then
    echo "Dependency issue detected, reinstalling dependencies..."
    npm install
fi

Step 5: Integrate with Self-Healing Services:

For more complex self-healing, you can integrate tools like Ansible, Puppet, or even create custom scripts that auto-patch common configuration issues.

How to Conduct Effective Postmortems Using Logs

Logs are often the single most valuable resource when reconstructing what went wrong in a CI/CD pipeline. Conducting effective postmortems with log data allows teams to extract clear timelines, pinpoint root causes, and define steps to prevent recurrence – all based on concrete evidence.

Extract Timeline and Key Events from the Logs

To accurately understand what happened and when from the info contained in your logs, there’s a straightforward process you can follow.

Step 1: Centralize and Structure Logs:

First, make sure that the logs from all pipeline stages (build, test, deploy) are aggregated in a central place like Grafana Loki, ELK, or OpenSearch.

And you’ll want to use a consistent log format (like structured JSON) that includes timestamps, log levels, pipeline stage identifiers, and correlation/request IDs.

Step 2: Build a Chronological View:

You can use timestamp filters in your log UI (for example, Kibana, Grafana Explore) to isolate logs from the incident timeframe.

Look for key lifecycle events, like:

• Start and completion of pipeline steps

• Status changes (for example, "test failed", "deployment started", "build queued")

• Error messages and warnings

• Retry events or unexpected restarts

Step 3: Extract Logs Programmatically (optional):

Use queries (LogQL, Elasticsearch DSL) to export relevant logs for analysis or inclusion in a post-mortem document.

How to Identify Root Causes Through Log Analysis

To go beyond symptoms and find the real issue, there are various steps you can take.

Start by looking for the first failure. You can filter logs by level=error or use log pattern matching to identify the earliest sign of failure. Then trace backward from the failure using correlation IDs or pipeline step identifiers.

Second, make sure you correlate logs across systems. Match logs across CI/CD tools (like GitHub Actions → Docker logs → Kubernetes logs). You can use shared correlation IDs or job IDs to group logs from related events.

Next, pay attention to intermittent signals. Warnings, retries, or degraded performance preceding the failure may reveal environmental or configuration-related issues.

And finally, check for external dependencies. Look for timeout or connection errors involving third-party services, cloud APIs, or internal infrastructure components.

How to Create Actionable Follow-Ups to Prevent Recurrence

There are various things you can do to turn your findings into meaningful process improvements.

1. Document the Findings Clearly:

Create a structured post-mortem doc that includes:

• Timeline of events with log excerpts

• Immediate trigger and root cause (based on logs)

• Impact summary and affected components

• Screenshots or saved log queries for reference

2. Define Preventive Actions:

Examples include:

• Adding missing alerts or log-based monitors

• Improving log verbosity or adding missing metadata

• Fixing brittle test cases or deployment scripts

• Updating infrastructure limits or retry strategies

3. Assign Ownership and Deadlines:

Each action item should have a responsible owner and a due date. If applicable, create automated tests or guardrails to catch similar issues in the future.

4. Update Runbooks and Incident Playbooks:

Add log patterns, example queries, and resolutions to shared documentation. This ensures the next person facing a similar issue can act faster.

Pro Tip: Automate part of your post-mortem process by tagging logs from failed CI runs, exporting them to a shared location, and pre-generating dashboards or incident reports. This reduces manual effort and increases consistency.

How to Optimize Log Storage and Management

As your CI/CD system grows, logs can become massive, consuming storage and impacting performance. Optimizing log storage helps you make sure that you're retaining what's valuable while staying efficient.

How to Implement Log Rotation and Retention Policies

Without rotation and retention, logs will pile up endlessly, leading to disk space exhaustion and poor performance. You can help prevent this with log rotation.

Log rotation involves creating new log files after a size or time threshold and archiving or deleting old ones.

Linux logrotate tool – Configure /etc/logrotate.d/<your-app>:

/var/log/ci_cd/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0640 root adm
}

This example:

• Rotates daily

• Keeps 7 days of logs

• Compresses old logs to save space

Docker logs rotation – in daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

Retention policies ensure that old logs are automatically deleted based on age or storage usage.

You can set one up in Loki like this:

table_manager:
  retention_deletes_enabled: true
  retention_period: 168h  # 7 days

Or in Elasticsearch, use Index Lifecycle Management (ILM):

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "3d", "max_size": "1gb" }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": { "delete": {} }
      }
    }
  }
}

How to Set Up Log Compaction for Long-Term Storage

Compaction reduces redundancy and keeps only critical log info, which is ideal for long-term audits or analytics.

Compaction Techniques:

There are various different compaction techniques you can try. Here are a couple:

1. Loki (boltdb-shipper mode):

• Uses compaction to merge log chunks and reduce storage.

• Configure in loki-config.yaml:

    schema_config:
      configs:
        - from: 2023-01-01
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
  

• Use a low-retention, high-compaction strategy for archived logs.

2. Elasticsearch:

• Use rollup jobs to reduce resolution of old data.

• Stores summarized logs, for example, hourly counts of similar events.

3. Archive to cheaper storage:

• Move infrequent-access logs to S3 or Azure Blob Storage using lifecycle rules.

How to Balance Observability with Resource Constraints

More logs = more observability, but also more cost and overhead. This means that you need a balance. There are various strategies that can help you achieve this balance:

1. Log at appropriate levels:

   • Avoid excessive debug or trace logs in production.

   • Use info and warn levels judiciously.

   • Only use error or critical for actionable failures.

2. Sample logs:

   • If high-volume pipelines generate repetitive logs, enable log sampling to reduce duplicates.

   • Tools like Vector or Fluent Bit support sampling.

3. Filter out noise:

   • Use log filters to exclude non-critical logs before they reach the central system.

4. Separate hot vs. cold logs:

   • Hot logs: recent, real-time data for active debugging.

   • Cold logs: archived for compliance, stored with lower performance/storage priority.

5. Compress everything:

   • Use gzip/zstd compression for both stored and transmitted logs.

   • Loki, Elasticsearch, and Vector support compression out of the box.

Conclusion

In this handbook, you have built a full-stack observability layer specifically optimized for CI/CD pipelines without breaking your infrastructure budget. You now have the tools and know-how to:

• Deploy Grafana Loki or a lightweight ELK alternative to capture structured logs from all parts of your pipeline.

• Unify and enrich logs across CI/CD tools (for example, GitHub Actions, Jenkins, GitLab) using consistent formats and correlation IDs.

• Use powerful log queries (LogQL, Kibana Query Language) to diagnose build failures, flaky tests, and deployment issues with precision.

• Correlate logs with metrics and traces to gain deep, contextual visibility into pipeline behavior.

• Design reusable debugging dashboards and automation that turn raw logs into insights and action.

• Build a culture of shared troubleshooting knowledge through post-mortems, runbooks, and log-driven retrospectives.

To see the full-stack observability layer in action, check out the complete code and configurations in my GitHub repository: github.com/Emidowojo/CICDObservability. This repo includes all the setups for Grafana Loki, OpenTelemetry, Prometheus, and more, so you can deploy and explore the entire pipeline observability stack.

Next Steps for Advanced Observability Implementation

Here’s how you can take your setup even further:

1. Fully integrate distributed tracing: Deploy OpenTelemetry agents across your build and deployment stages. This will help you visualize how code, builds, and deployments flow across systems in real-time.

2. Automate diagnostic scripts and alerts: Build scripts to auto-collect logs and metrics on failure, and trigger alerts when known patterns reoccur. This enables faster detection and even self-healing pipelines.

3. Scale and harden your log infrastructure: As usage grows, implement log retention, compaction, and storage policies. Explore scalable backends like ClickHouse or object storage (e.g., S3) for long-term archiving.

4. Train your team on observability best practices: Share dashboards, create onboarding docs, and schedule log-analysis sessions to build team familiarity with your tools and practices.

📚 Resources for Continued Learning

Official Docs and Tools:

• Grafana Loki Documentation

• Promtail Configuration Guide

• OpenTelemetry

• LogQL Syntax

• Kibana Query Language

• Vector (log forwarding)

Communities:

• r/devops on Reddit

• CNCF Slack – #observability channel

• Log Management Best Practices on Stack Overflow

By investing in observability early and thoughtfully, you not only reduce the time to detect and resolve issues, you also build a more resilient, predictable, and transparent delivery process for your entire engineering team.

I hope this comes in handy for you someday. If you made it to the end of this handbook, thanks for reading! You can connect with me on LinkedIn or on X @Emidowojo if you’d like to stay in touch.

I had the same questions when I started learning about cloud-native technologies. In this article, I'll share my understanding of core observability concepts, introduce essential observability tools, and share insights from a related project I’m working on.

Table of Contents

• My Introduction to Cloud Native Technologies

• What is Observability?

• Types of Observability Data

  • 1. Metrics

  • 2. Logs

  • 3. Traces

• Observability Tools

  • Prometheus

  • OpenTelemetry

• What are OTLP Resource Attributes?

• Importance of OTLP Resource Attributes

• How My Project Work Fits into All This

• Conclusion

• Additional Resources

My Introduction to Cloud Native Technologies

I recently got selected as a mentee for the Linux Foundation Mentorship to work on the CNCF - Prometheus project. The project is UX-focused, and for the next few months, I'll be working with my mentors to understand how users expect to use OpenTelemetry Line Protocol (OTLP) Resource Attributes in Prometheus.

That's quite a mouthful, I know. I was overwhelmed at first, and honestly, I’m still figuring it out. This is my third week, and although I still have a lot to learn—given that I had no knowledge of cloud native technologies when I applied for this internship—I've already learned quite a bit.

As I often do, I intend to document what I learn through articles to help reinforce concepts in my memory and serve as a resource for other newcomers who may find themselves grappling with these technical terms in the future. You know what they say: you can't say you've understood something until you're able to explain it to someone else who's also new to the topic.

What is Observability?

First, I had to learn what the unfamiliar terms meant—and there were a lot of them flying around. OpenTelemetry. Prometheus. Resource attributes. I’ve come to understand that these terms fall under one umbrella: Observability. Let's start there.

Let’s use a food delivery app to illustrate. When someone orders food, a lot happens behind the scenes:

• The app connects to different services (restaurants, payments, delivery)

• Data flows between different systems to process the order, assign a driver, and track delivery

Engineers need to monitor all the processes to ensure everything works smoothly. Are orders taking too long to process? Is the payment system failing? Does the app suddenly crash under load? Which part of the system is causing delays?

To answer these questions, engineers instrument their code. This means that they configure it to send back real-time data about the state, performance, and behavior of the application. This practice of understanding what's happening inside a complex system based on the data it generates is known as Observability.

You can see the process illustrated in the image below:

In the above flowchart diagram, you can see how data might move through a food delivery application system. The flow starts with a User who orders food from a Food App. The Food App connects to three services (Restaurant, Payment, and Delivery). All these components send data to OpenTelemetry, which collects three types of data: Metrics, Logs, and Traces. OpenTelemetry then forwards only the Metrics data to Prometheus, which stores metrics.

Types of Observability Data

There are three key types of data that systems generate for observability:

1. Metrics

Metrics are numerical measurements collected over time that represent the state or performance of your system. Examples in a food delivery app would be the number of orders processed per minute, average order processing time in milliseconds, number of active users or delivery drivers, and so on.

2. Logs

Logs are text-based records of discrete events that occur within your application. Logs for our food delivery app would look something like this:

ERROR: Payment failed for order #12345 - Credit card declined
INFO: Driver #789 assigned to order #12345

3. Traces

Traces track the entire lifecycle of a request as it moves through different services in a system. They help engineers see how different components interact and identify bottlenecks in complex, distributed systems.

For example, in our food delivery app, a single order request might go through the following steps:
User places an order → Request sent to restaurant system → Payment processor verifies payment → Delivery system assigns a driver → User receives confirmation.

Each step in this journey is recorded as part of a trace. This helps engineers pinpoint where delays occur and optimize the system for better performance.

Observability relies on metrics, logs, and traces working together to provide full system visibility. Metrics tell you something is wrong (“Error rate increased by 5%”). Logs tell you why it happened (“Payment failed due to invalid card details”). Traces show exactly where it happened (“Delay in restaurant service response”).

Observability Tools

Observability tools give you visibility into what’s going on within your application. There are a lot of them, but for the purpose of this article, we’ll talk about two: Prometheus and OpenTelemetry.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit. It does two things:

• Collects data from applications, specifically metrics (remember the data types we talked about earlier)

• and stores them in a time-series database.

A time-series database is a database specifically designed to handle measurements or events that occur over time.

Prometheus uses what's called a pull-based model to collect metrics from applications. Pull-based means Prometheus actively requests (pulls) data from services at regular intervals. Think of it like refreshing a webpage to get the latest content.

OpenTelemetry

OpenTelemetry (OTel) collects, processes, and exports observability data. Unlike Prometheus, which mainly focuses on metrics, OpenTelemetry provides a standardized way to instrument applications for all three types of observability data: logs, metrics, and traces.

OpenTelemetry is designed to be vendor-agnostic. This means you can instrument your applications once with OpenTelemetry and then send that telemetry data to any supported observability backend, which could be an open-source solution like Jaeger or Prometheus, or commercial platforms like Datadog, New Relic, Dynatrace, or Honeycomb.

So, for example, you can use OpenTelemetry to instrument your application – and then Prometheus can pull metrics from OpenTelemetry while other tools handle logs and traces.

What are OTLP Resource Attributes?

When OpenTelemetry collects data from applications, it does more than just gather raw telemetry data. It also provides context about that data. This context comes in the form of resource attributes, which describe where the data came from and what it relates to.

The 'resource' is the component (or entity) producing the data, while the 'attributes' are specific details about that resource.

Resource attributes are structured as pairs of information:

• The "key" is the name or identifier of the attribute (like service.name or host.id)

• The "value" is the specific information for that attribute (like payment-service or server-123)

Together, these key-value pairs identify and describe the specific component that's generating the observability data.

For example, if a payment processing service is sending metrics about transaction times, the resource attributes might include:

• service.name: "payment-service"

• service.version: "1.2.3"

• deployment.environment: "production"

These attributes tell you exactly which service, which version, and in which environment the data is coming from, providing context for interpreting the metrics, logs, or traces.

Resource attributes are not arbitrary. OpenTelemetry provides a standardized set of attribute names and formats that everyone should follow, similar to having an agreed-upon language for describing services and their properties.

For example, OpenTelemetry specifies that you should use service.name (not app_name or service_id) to identify your service. They've created these standardized naming conventions (called semantic conventions) so that:

1. All tools in the ecosystem can understand the same attributes

2. Engineers across different companies use consistent terminology

3. Observability data can be easily shared between different systems

You can still create your own custom attributes when you need something specific (like payment.provider for a payment service), but using the standard attributes whenever possible means your telemetry data will work better with existing tools and be more easily understood by other engineers.

Importance of OTLP Resource Attributes

Let’s say engineers want to monitor how long food deliveries take and whether there are delays in specific locations. Without resource attributes, OpenTelemetry might simply collect and report this metric like this:

delivery_time_seconds: 1800

This tells us that a delivery took 1,800 seconds, or 30 minutes, but nothing else. That’s useful, but it lacks context. Where did this happen? Which service handled it? If there was a delay in delivery and engineers wanted to investigate the cause, this alone would not help.

With OpenTelemetry’s resource attributes, the metric becomes more meaningful:

delivery_time_seconds: 1800
resource:
  service.name: "delivery-service"
  service.instance.id: "instance-456"
  cloud.region: "ng-west-2"
  deployment.environment: "production"
  customer.city: "Lagos"
  restaurant.id: "rest-789"

This tells us:

• The data came from the delivery service.

• The instance handling the request is "instance-456".

• It’s running in the ng-west-2 cloud region.

• The environment is Production (not testing or staging), and so on.

Now, engineers can answer more specific questions:

• Are deliveries slower in certain cities? (Filter by customer.city)

• Are certain restaurants taking longer to prepare food? (Filter by restaurant.id)

• Are delays only happening in a specific cloud region? (Filter by cloud.region)

• Are issues only happening in production or also in staging? (Filter by deployment.environment)

When issues arise, resource attributes allow engineers to quickly narrow down the source of problems. Rather than investigating every service, they can filter by specific attributes to focus their efforts.

How My Project Work Fits into All This

Many engineers use OpenTelemetry for data collection and then send metrics to Prometheus for storage, querying, and analysis.

But Prometheus does not natively support resource attributes in the same way as OpenTelemetry. Instead, it relies on labels to organize metrics. Since Prometheus traditionally has its own labeling system for metrics, integrating OpenTelemetry's resource attributes creates interesting UX challenges.

One key challenge is the cardinality explosion. Cardinality refers to the number of unique combinations of label values (or dimensions) that a metric can have. A "cardinality explosion" occurs when you add labels with many possible values. OpenTelemetry often includes many detailed attributes that, if directly converted to Prometheus labels, would create an overwhelming number of time series. This can slow down Prometheus dramatically or even cause it to crash.

The existing solution involves stuffing all resource attributes into a single JSON-encoded Prometheus label. While this prevents the cardinality explosion, it makes querying extremely cumbersome. Users have to use complex join operations and specialized query syntax to filter or aggregate based on these attributes.

This approach is technically functional but creates a poor user experience. My research aims to understand how users mentally model the transition from OpenTelemetry's rich attribute system to Prometheus's more constrained label system.

The research goals are to:

1. Understand how engineers currently use OpenTelemetry resource attributes with Prometheus

2. Identify pain points in the current integration between these systems

3. Discover user expectations for how resource attributes should be represented in Prometheus

This work is particularly important as more organizations adopt OpenTelemetry as their instrumentation standard while continuing to use Prometheus for metrics monitoring. Creating a seamless experience between these two popular open-source projects will help improve the overall observability ecosystem.

Conclusion

Observability in cloud native applications is clearly an interesting subject and important for building reliable, performant systems. The tools and concepts we've explored – metrics, logs, traces, Prometheus, and OpenTelemetry – form the foundation of modern observability practices.

As I continue my mentorship program, I'll share more insights about how these technologies work together and try to break them down from the perspective of a first-time learner.

Additional Resources

Learn more about:

1. OpenTelemetry

2. Prometheus

3. My UX research project

One of the coolest things I have learned in the last year is how to constantly deliver value into production without causing too much chaos.

In this post, I’ll explain the metrics-driven development approach and how it helped me to achieve that. By the end of the post, you’ll be able to answer the following questions:

• What are metrics and why should I use them
• What are the different types of metrics
• What tools could I use to store and display metrics
• What is a real-world example of metrics-driven development

What are metrics and why should I use them?

Metrics give you the ability to collect information on an actively running system without changing its code.

It allows you to gain valuable data on the behavior of your application while it runs so you can make data-driven decisions based on real customer feedback and usage in production.

What are the types of metrics available to me?

These are the most common metrics used today:

• Counter — Represents a monotonically increasing value.

Counters are really useful for measuring rates!

In this example, a counter metric is used to calculate the rate of events over time, by counting events per second

• Gauge — Represents a single value that can go up or down.

Gauges are really useful for measuring CPU usage!

In this example, a gauge metric is used to monitor the user CPU in percentages

• Histogram — A counting of observations (like request durations or sizes) in configurable buckets.

Histograms are really useful for measuring request duration!

In this example, a histogram metric is used to calculate the 75th and 90th percentiles of an HTTP request duration.

The bits and bytes of the types: counter, histogram, and gauge can be quite confusing. Try reading about it further here.

What tools can I use to store and display metrics?

Most monitoring systems consist of a few parts:

1. Time-series database — A database software that optimizes storing and serving time-series data. Two examples of this kind of database are Whisper and Prometheus.
2. Querying engine (with a querying language) — Two examples of common query engines are: Graphite and PromQL
3. Alerting system — The mechanism that allows you to configure alerts based on graphs created by the querying language. The system can send these alerts to Mail, Slack, PagerDuty. Two examples of common alerting systems are: Grafana and Prometheus.
4. UI — Allows you to view the graphs generated by the incoming data and configure queries and alerts. Two examples of common UI systems are: Graphite and Grafana

The setup we are using today in BigPanda Engineering is

• Telegraf — used as a StatsD server.
• Prometheus — used as our scrapping engine, Time-series database and querying engine.
• Grafana — used for Alerting, and UI

And the constraints we had in mind while choosing this stack were:

• We want scalable and elastic metrics scraping
• We want a performant query engine
• We want the ability to query our metrics using custom tags(such as service names, hosts, etc.)

A real-world example of Metrics-driven development of a Sentiment Analysis service

Let’s develop a new pipeline service that calculates sentiments based on textual inputs and does it in a Metrics Driven Development way!

Let’s say I need to develop this pipeline service:

Sentiment analysis pipeline architecture

And this is my usual development process:

Usual development process - Test, code and deploy. Oh my!

So I write the following implementation:

let senService: SentimentAnalysisService = new SentimentAnalysisService();
while (true) {
    let tweetInformation = kafkaConsumer.consume()
    let deserializedTweet: { msg: string } = deSerialize(tweetInformation)
    let sentimentResult = senService.calculateSentiment(deserializedTweet.msg)
    let serializedSentimentResult = serialize(sentimentResult)
    sentimentStore.store(sentimentResult);
    kafkaProducer.produce(serializedSentimentResult, 'sentiment_topic', 0);
}

The full gist can be found here.

And this method works perfectly fine**.

But what happens when it doesn’t?

The reality is that while working (in an agile development process) we make mistakes. That’s a fact of life.

I believe that the real challenge with making mistakes is not to avoid them, but rather to optimize how fast we detect and repair them. So, we need to gain the ability to quickly discover our mistakes.

It's time for the MDD-way.

The Metrics Driven Development (MDD) way

Behold! The Three Commandments of Production!

The MDD approach is heavily inspired by the Three Commandments of Production (which I had learned about the hard way).

The Three Commandments of Production are:

1. There are mistakes and bugs in the code you write and deploy.
2. The data flowing in production is unpredictable and unique!
3. Perfect your code from real customer feedback and usage in production.

And since we now know the Commandments, it's time to go over the 4 step plan of the Metrics-Driven development process.

The 4-step plan for a successful MDD

Metrics-driven development ?Oh wow!

Develop code 

I write the code, and whenever possible, wrap it with a feature flag that allows me to gradually open it for users.

Metrics

This consists of two parts:

Add metrics on relevant parts

In this part, I ask myself what are the success or failure metrics I can define to make sure my feature works? In this case, does my new pipeline application perform its logic correctly?

Add alerts on top of them so that I’ll be alerted when a bug occurs

In this part, I ask myself What metric could alert me if I forgot something or did not implement it correctly?

Deployment

I deploy the code and immediately monitor it to verify that it’s behaving as I have anticipated.

Iterate this process to perfection

And that's it! Now that we have learned the process, let's tackle an important task inside it.

Metrics to Report — what should we monitor?

One of the toughest questions for me, when I’m doing MDD, is: “what should I monitor”?

That’s a lovely gif. but un-realistic in most cases.

In order to answer the question, lets try to zoom out and look at the big picture.
All the possible information available to monitor can be divided into two parts:

1. Applicative information — Information that has an applicative context and meaning. An example of this will be — “How many tweets did we classify as positive in the last hour”?
2. Operational information — Information that is related to the infrastructure that surrounds our application — Cloud data, CPU and disk utilization, network usage, etc.

Now, since we cannot monitor everything, we need to choose what applicative and operational information we want to monitor.

• The operational part really depends on your ops stack and has built-in solutions for (almost) all your monitoring needs.
• The applicative part is more unique to your needs, and I'll try to explain how I think about it later in this post.

After we do that, we can ask ourselves the question: what alerts do we want to set up on top of the metrics we just defined?

The diagram (of information, metrics, alerts) can be drawn like this:

The world of information, metrics, and alerts.

Applicative metrics

I usually add applicative metrics out of two needs:

To answer questions

A question is something like, “When my service misbehaves, what information would be helpful to know about?”

Some answers to that question can be — latencies of all IO calls, processing rate, throughput, etc…

Most of these questions will be helpful while you are searching for the answer. But once you found it, chances are you will not look at it again (since you already know the answer).

These questions are usually driven by RND and are (usually) used to gather information internally.

To add Alerts

This may sound backward, but I usually add applicative metrics in order to define alerts on top of them. Meaning, we define the list of alerts and then deduce from them what are the applicative metrics to report.

These alerts are derived from the SLA of the product and are usually treated with mission-critical importance.

Common types of alerts

Alerts can be broken down into three parts:

Alerts types to Metrics list

SLA Alerts

SLA alerts in reality

SLA alerts surround the places in our system where an SLA is specified to meet explicit customer or internal requirements (i.e availability, throughput, latency, etc.). SLA breaches involve paging RND and waking people up, so try to keep the alerts in this list to a minimum.

Also, we can define Degradation Alerts in addition to SLA Alerts.
Degradation alerts are defined with lower thresholds then SLA alerts, and are therefore useful in reducing the amount of SLA breaches — by giving you a proper heads-up before they happen.

An example of an SLA alert would be, “All sentiment requests must finish in under 500ms.”

An example of a Degradation Alert will be: “All sentiment requests must finish in under 400ms”.

These are the alerts I defined:

1. Latency — I expect the 90th percentile of a single request duration not to exceed 300ms.
2. Success/Failure ratio of requests — I expect the number of failures per second, success per second, to remain under 0.01.
3. Throughput — I expect that the number of operations per second (ops) that the application handles will be > 200
4. Data Size — I expect the amount of data that we store in a single day should not exceed 2GB.

200 ops 60 bytes(Size of Sentiment Result) 86400 sec in a day = 1GB < 2GB

Baseline Breaching Alerts

These alerts usually involve measuring and defining a baseline and making sure it doesn’t (dramatically) change over time with alerts.

For example, the 99th processing latency for an event must stay relatively the same across time unless we have made dramatic changes to the logic.

These are the alerts I defined:

1. Amount of Positive or Neutral or Negative Sentiment tweets — If for whatever reason, the sum of Positive tweets has increased or decreased dramatically, I might have a bug somewhere in my application.
2. All latency \ Success ratio of requests \ Throughput \ Data size must not increase\decrease dramatically over time.

Runtime Properties Alerts

I’ve given a talk about Property-Based Tests and their insane strength. As it turns out, collecting metrics allows us to run property-based tests on our system in production!

Some properties of our system:

1. Since we consume messages from a Kafka topic, the handled offset must monotonically increase over time.
2. 1 ≥ sentiment score ≥ 0
3. A tweet should classify as either Negative \ Positive \ Neutral.
4. A tweet classification must be unique.

These alerts helped me validate that:

1. We are reading with the same group-id. Changing consumer group ids by mistake in deployment is a common mistake when using Kafka. It causes a lot of mayhem in production.
2. The sentiment score is consistently between 0 and 1.
3. Tweet category length should always be 1.

In order to define these alerts, you need to submit metrics from your application. Go here for the complete metrics list.

Using these metrics, I can create alerts that will “page” me whenever one of these properties do not hold anymore in production.

Processing latency breached configured SLA! Oh my! ?

Let’s take a look at a possible implementation of all these metrics

import SDC = require("statsd-client");
let sdc = new SDC({ host: 'localhost' });
let senService: SentimentAnalysisService; //...
while (true) {
    let tweetInformation = kafkaConsumer.consume()
    sdc.increment('incoming_requests_count')
    let deserializedTweet: { msg: string } = deSerialize(tweetInformation)
    sdc.histogram('request_size_chars', deserializedTweet.msg.length);
    let sentimentResult = senService.calculateSentiment(deserializedTweet.msg)
    if (sentimentResult !== undefined) {
        let serializedSentimentResult = serialize(sentimentResult)
        sdc.histogram('outgoing_event_size_chars', serializedSentimentResult.length);
        sentimentStore.store(sentimentResult)
        kafkaProducer.produce(serializedSentimentResult, 'sentiment_topic', 0);
    }

}

The full code can be found here

A few thoughts on the code example above:

1. There has been a staggering amount of metrics added to this codebase.
2. Metrics add complexity to the codebase, so, like all good things, add them responsibly and in moderation.
3. Choosing correct metric names is hard. Take your time selecting proper names. Here’s an excellent post about this.
4. You still need to collect these metrics and display them in a monitoring system (like Grafana), plus add alerts on top of them, but that’s a topic for a different post.

Did we reach the initial goal of identifying issues and resolving them faster?

YESSSS, it was!

We can now make sure the application latency and throughput do not degrade over time. Also, adding alerts on these metrics allows for a much faster issue discovery and resolution.

Conclusion

Metrics-driven development goes hand in hand with CI\CD, DevOps, and agile development process. If you are using any of the above keywords, then you are in the right place.

When done right, metrics make you feel more confident in your deployment in the same way that seeing passing unit-tests in your build makes you feel confident in the code you write.

Adding metrics allows you to deploy code and feel confident that your production environment is stable and that your application is behaving as expected over time. So I encourage you to try it out!

Some references

1. Here is a link to the code shown in this post, and here is the full metrics list described.
2. If you are eager to try writing some metrics and to connect them to a monitoring system, check out Prometheus, Grafana and possibly this post
3. This guy wrote a delightful post about metrics-driven development. GO read it.