When you run five agents in a swarm, each spawning its own tool calls and producing its own output, "read the log" stops being a strategy.

I built Claude Forge as an adversarial multi-agent coding framework on top of Claude Code. A typical run spawns a planner, an implementer, a reviewer, and a fixer. They evaluate each other's work and loop back when quality checks fail.

But when something went wrong, I had timestamps and text dumps but no way to see which agent was responsible, how long it actually took, or where the tokens went.

Jaeger fixed that. This article covers setting up Jaeger v2 with Docker, wiring it into a multi-agent system through OpenTelemetry, and what I learned along the way.

Table of Contents

• What Is Distributed Tracing?

• Why Jaeger v2?

• Prerequisites

• Installing Docker on Debian

• Setting Up Jaeger v2

• Setting Up Claude Forge Tracing

• Understanding the Span Model

• Instrumenting a Multi-Agent Swarm

• Viewing Traces in the Jaeger UI

• Lessons from the Trenches

• Environment Variable Reference

• Wrapping Up

What Is Distributed Tracing?

Distributed tracing tracks a single operation as it moves through multiple services. A span is one unit of work with a start time, end time, and key-value attributes. Spans nest into parent-child trees. One tree per operation is one trace.

Microservices people already know this pattern: follow an HTTP request from the gateway through auth, the database, and the cache. Same idea works for multi-agent AI. Follow one swarm invocation from the orchestrator through each subagent and its tool calls.

OpenTelemetry (OTel) is the standard. It gives you SDKs for creating spans and shipping them over OTLP. Jaeger receives that data and renders it as a searchable timeline.

Why Jaeger v2?

Jaeger started at Uber and graduated as a CNCF project in 2019. v1 hit end of life in December 2025. v2 is the current release, built on the OpenTelemetry Collector framework. Single binary: collector, query service, and UI. It speaks OTLP natively on port 4317 (gRPC) and 4318 (HTTP). There's no separate collector needed for local work.

One important difference from v1: configuration moved from CLI flags and environment variables to a YAML file. The old -e SPAN_STORAGE_TYPE=badger env vars are silently ignored in v2. The container starts fine but falls back to in-memory storage. I lost two days of traces before noticing. More on the correct setup below.

Prerequisites

• Docker installed and running.

• Claude Code installed.

• Python 3.8+ for the tracing hook.

• Claude Forge or another multi-agent system to instrument.

Installing Docker on Debian

Skip this if you already have Docker. macOS and Windows users can use Docker Desktop. On Debian:

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/debian \
  \((. /etc/os-release && echo "\)VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Ubuntu users: replace both linux/debian URLs with linux/ubuntu.

Setting Up Jaeger v2

Basic Run

For quick testing with no persistence:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/jaeger:2.17.0

Port 16686 is the UI. Port 4317 is OTLP/gRPC ingestion. Port 4318 is OTLP/HTTP. Remove the container and your traces are gone.

Persistent Storage with Badger

v2 reads configuration from a YAML file, not environment variables. Save this as ~/.local/share/jaeger/config.yaml:

service:
  extensions: [jaeger_storage, jaeger_query, healthcheckv2]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]
extensions:
  healthcheckv2:
    use_v2: true
    http: { endpoint: 0.0.0.0:13133 }
  jaeger_query:
    storage: { traces: main_store }
  jaeger_storage:
    backends:
      main_store:
        badger:
          directories: { keys: /badger/key, values: /badger/data }
          ephemeral: false
          ttl: { spans: 720h }
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch:
exporters:
  jaeger_storage_exporter:
    trace_storage: main_store

The Jaeger container runs as UID 10001. Docker named volumes default to root ownership. Without fixing permissions first, the container crash-loops with mkdir /badger/key: permission denied.

Pre-create the volume and fix ownership:

docker volume create jaeger-data

docker run --rm \
  -v jaeger-data:/badger \
  alpine sh -c "mkdir -p /badger/data /badger/key && chown -R 10001:10001 /badger"

Then run Jaeger with the config mounted in:

docker run -d --name jaeger \
  --restart unless-stopped \
  -v ~/.local/share/jaeger/config.yaml:/etc/jaeger/config.yaml:ro \
  -v jaeger-data:/badger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/jaeger:2.17.0 \
  --config /etc/jaeger/config.yaml

Verify persistence by running docker restart jaeger and confirming a previously recorded trace is still there. Hit http://localhost:16686 and you should see the UI.

Setting Up Claude Forge Tracing

Installing Claude Forge

Install it through the Claude Code plugin marketplace:

/plugin marketplace add hatmanstack/claude-forge
/plugin install forge@claude-forge
/reload-plugins

The install opens a TUI to confirm scope and settings. After reload, commands use the forge: prefix (for example, /forge:pipeline).

You can also clone the repo from GitHub.

Installing the Tracing Hook

From your target project directory, run the install script. For plugin installs:

cd your-project
forge-trace                # if you set up the alias from the README
# or, without the alias:
bash "$(find ~/.claude -path '*/forge*' -name install-tracing.sh 2>/dev/null | head -1)"

For clone installs:

cd your-project
bash /path/to/claude-forge/bin/install-tracing.sh

The script builds a dedicated venv at ~/.local/share/claude-forge/venv (prefers uv, falls back to python3 -m venv), installs the OpenTelemetry packages, copies the hook into place, merges hook entries into .claude/settings.local.json, and self-tests against the OTLP endpoint.

Pass --no-settings to skip the settings merge, or --uninstall to tear everything down.

Opting In

Add to your shell init and restart your terminal:

export CLAUDE_FORGE_TRACING=1

Restart Claude Code, run /pipeline, then check http://localhost:16686 for the claude-forge service.

Understanding the Span Model

Here's what the hierarchy looks like for a typical swarm run:

session: "implement login form with OAuth"        <- root span
├── subagent:planner
│   ├── tool:Write  (Phase-0.md)                  <- mutation spans (on by default)
│   ├── tool:Write  (Phase-1.md)
│   └── subagent_result:planner                   <- duration, token counts, output
├── subagent:implementer
│   ├── tool:Edit   (src/auth.ts)
│   ├── tool:Bash   (npm test)
│   ├── tool:Write  (src/oauth.ts)
│   └── subagent_result:implementer
├── subagent:reviewer
│   └── subagent_result:reviewer
└── session_complete                              <- session totals

The root span's name comes from the first line of your prompt. Find traces by what you asked for, not by a UUID.

Subagents get an anchor span on start and a result span on completion. The result carries duration, token counts, prompt, and output.

Three Tiers of Detail

Not all inner tool calls are equally interesting. Write, Edit, MultiEdit, and Bash are mutational: small in number, high signal. They tell you what actually changed. Read, Glob, Grep, and WebFetch are navigation: lots of them, mostly noise.

Tracing captures mutations by default. That middle ground turned out to be the right one. Before this change, you either saw nothing inside subagents or you saw 200+ spans per run.

Span Attributes

On session_complete: session.tokens.input, session.tokens.output, session.tokens.total, session.tokens.turns, session.duration_ms, user.prompt (first 2KB).

On subagent_result: agent.description, agent.prompt, agent.output, agent.duration_ms, agent.is_error, agent.tokens.input, agent.tokens.output.

On tool:*: tool.name, tool.input, tool.output, tool.duration_ms, tool.is_error.

Instrumenting a Multi-Agent Swarm

Hook Architecture

Claude Code has lifecycle hooks that fire scripts on specific events. Four matter here:

1. UserPromptSubmit (create the root span),

2. PreToolUse (start a span),

3. PostToolUse (end it with results), and

4. Stop (finalize the trace). Each hook gets a JSON payload on stdin and runs as a subprocess.

Sending Spans with OpenTelemetry

Here's some minimal Python to get a span into Jaeger:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "my-agent-system"})
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("agent-tracer")

with tracer.start_as_current_span("my-agent-task") as span:
    span.set_attribute("agent.name", "planner")
    span.set_attribute("agent.tokens.input", 1500)
    span.set_attribute("agent.tokens.output", 800)

Refresh localhost:16686, pick your service, click "Find Traces."

Correlating Pre and Post Events

You need to match each PreToolUse to its PostToolUse. Agent-type tool calls didn't include a tool_use_id in the payload, so I hashed the tool name and input instead. Pre and Post carry identical tool_input, so the hashes line up.

import hashlib, json

def correlation_key(tool_name: str, tool_input: dict) -> str:
    content = json.dumps({"tool": tool_name, "input": tool_input}, sort_keys=True)
    return hashlib.sha1(content.encode()).hexdigest()[:16]

State Across Invocations

Every hook call is a separate process. No shared memory. So I wrote span context to JSON files on Pre and read them back on Post:

/tmp/claude-forge-tracing/<session_id>/
├── _root.json              # trace ID, root span context
├── _session_start_ns.json  # timestamp for duration calculation
├── subagent_<hash>.json    # per-subagent span context
└── tool_<hash>.json        # per-tool span context

File names get sanitized against path traversal. _safe_name() strips everything outside [A-Za-z0-9._-] and falls back to a SHA1 slug.

Flushing Without Blocking

try:
    provider.force_flush(timeout_millis=1000)
except Exception:
    pass  # Never block the swarm

I tried 2000ms first and the swarm felt slow. 100ms lost spans on cold TLS connections. 1000ms worked. If Jaeger is down, the swarm keeps running regardless.

Viewing Traces in the Jaeger UI

Open http://localhost:16686. Pick claude-forge from the service dropdown. Click "Find Traces."

The trace search filters by operation name, tags, and time range. Since session spans take their name from your prompt, searching "login form" pulls up the runs where you asked for one.

The timeline view is where I spend most of my time. Every span is a horizontal bar, nested by parent-child relationships. I can see the planner took 12 seconds, the implementer 45, the reviewer 8. Click any bar to see token counts, prompts, outputs, error status.

Trace comparison puts two runs side by side. This is good for figuring out why one run succeeded and another did not.

Lessons from the Trenches

One trace per swarm, not per subagent: My first version wiped the root span's state file on every Stop event, so each subagent started a new trace. I changed Stop to mark a timestamp while preserving the root.

Use descriptions, not type names: Subagents all report their type as general-purpose. The description field is where the actual role lives.

Token attribution needs per-agent transcripts: Claude Code writes subagent transcripts to ~/.claude/projects/<project>/<session>/subagents/agent-*.jsonl. Match them via agent-*.meta.json.

Parse boolean env vars explicitly: bool("0") in Python is True. Use an allowlist: {"1", "true", "yes", "on"}.

Environment Variable Reference

Wrapping Up

Without visibility into the process, you're being inefficient with tokens and your time. Multi-agent swarms cost real money on every run. When an agent fails and retries, or when a reviewer rejects work that was close, you're paying for that blind.

Tracing gives you the map. You find out where the failure modes are. You find out which agents burn tokens going nowhere. A 45-second implementer run might have been 10 seconds with a better planner prompt. But you would never know that without seeing the breakdown.

Get observability in early. Jaeger and OpenTelemetry make it cheap to set up. Once you can see where things go wrong you can actually fix them.

Claude Forge tracing is on the main branch.

Instead of relying on vendor-specific agents or opaque SDKs, we will manually design traces, spans, and semantic attributes that capture the full lifecycle of an LLM-powered request.

Table of Contents

• Introduction

• Prerequisites and Technical Context

• Why LLM Observability Is Fundamentally Different

• Reference Architecture: A Traceable RAG Request

• Reference Architecture Explained

• Why This Design Is Better Than Simpler Alternatives

• LLM Models That Work Best for This Architecture

• OpenTelemetry Primer (LLM-Relevant Concepts Only)

• Designing LLM-Aware Spans

• FastAPI Example: End-to-End LLM Spans (Complete and Explained)

• Semantic Attributes: Best Practices for LLM Observability

• Evaluation Hooks Inside Traces

• Exporting and Visualizing Traces (Where This Fits with Vendor Tooling)

• Operational Patterns and Anti-Patterns

• Extending the System

• Conclusion

Introduction

Large Language Models (LLMs) are rapidly becoming a core component of modern software systems. Applications that once relied on deterministic APIs are now incorporating LLM-powered features such as conversational assistants, document summarization, intelligent search, and retrieval-augmented generation (RAG).

While these capabilities unlock new user experiences, they also introduce operational complexity that traditional monitoring approaches were never designed to handle.

Unlike conventional software services, LLM systems are probabilistic by nature. The same request may produce slightly different responses depending on factors such as prompt structure, model configuration, retrieval context, and sampling parameters such as temperature or top-p.

In addition, LLM workloads introduce entirely new operational dimensions such as token consumption, prompt construction latency, inference cost, context window limits, and response quality.

These factors mean that a request can appear technically successful from an infrastructure perspective while still producing an incorrect, hallucinated, or low-quality result.

Traditional observability tools typically focus on infrastructure-level signals such as latency, error rate, and throughput. While these metrics remain important, they are insufficient for understanding how an LLM application behaves in production.

Engineers must also understand what prompt was constructed, which documents were retrieved, how many tokens were consumed, which model configuration was used, and how the final response was evaluated. Without this visibility, debugging LLM behavior becomes extremely difficult and operational costs can quickly spiral out of control.

This is where LLM observability becomes essential. Observability for LLM systems extends beyond infrastructure monitoring. It captures the full lifecycle of an AI-driven request — from user input and context retrieval to prompt construction, model inference, post-processing, and quality evaluation.

When implemented correctly, observability allows teams to answer why the model generated a particular response, which retrieval results influenced the output, how much a request cost in terms of tokens, where latency occurred within the request pipeline, and whether the response passed basic quality or safety checks.

This article demonstrates how to implement end-to-end LLM observability in a FastAPI application using OpenTelemetry. Instead of relying on proprietary monitoring agents or opaque vendor SDKs, we take a code-first approach to instrumentation. By explicitly designing traces, spans, and semantic attributes, we gain precise control over how LLM interactions are observed and analyzed.

Throughout the guide, we will walk through a practical architecture for tracing a retrieval-augmented generation (RAG) workflow, where each stage of the request lifecycle is represented as a trace span. We will explore how to design meaningful span boundaries, capture prompt and model metadata safely, record token usage and cost signals, and attach evaluation results directly to traces.

The article also explains how this instrumentation can be exported to any OpenTelemetry-compatible backend such as Jaeger, Grafana Tempo, or LLM-specific platforms like Phoenix.

By the end of this guide, you will understand how to:

• Structure traces so that each user request maps to a single end-to-end LLM interaction

• Design span hierarchies that reflect the logical stages of an LLM pipeline

• Capture prompt metadata, model configuration, and token usage safely

• Attach evaluation and quality signals to traces for deeper analysis

• Export observability data to different backends without changing instrumentation

Most importantly, the goal of this article is not simply to demonstrate how to add telemetry to an application. Instead, it aims to show how to think about observability when building LLM-powered systems.

When LLM operations are treated as first-class components within a distributed system, traces become a powerful tool for debugging, optimization, cost management, and continuous improvement of model behavior.

Prerequisites and Technical Context

Before following this guide, you should be familiar with the Python programming language, basic web API concepts, and general microservice architecture. Below are some key tools and concepts used in this article.

FastAPI (Web Framework)

FastAPI is used as the primary web framework for the application. It is a modern Python framework designed for building high-performance APIs using standard Python type hints. FastAPI simplifies request validation, serialization, and API documentation while remaining lightweight and fast.

Large Language Models (LLMs)

Large Language Models (LLMs) are the computational core of the example system. An LLM is a model trained on vast amounts of text data to generate or transform language in ways that resemble human communication. In production environments, LLMs are commonly used for tasks such as conversational interfaces, summarization, and question answering.

Observability (Concept)

Observability is the overarching concept that connects all the technical pieces in this article. At a high level, observability refers to the ability to understand a system's internal behavior by examining the data it produces during execution. Rather than asking whether a system is simply "up" or "down," observability helps answer deeper questions about why a request behaved a certain way, where latency was introduced, or how different components interacted.

OpenTelemetry (Instrumentation Standard)

OpenTelemetry is the mechanism used to implement observability within the application. It is an open, vendor-neutral standard for generating telemetry data such as traces, metrics, and logs. By instrumenting key parts of the LLM workflow, we can observe how requests flow through the system, how long each step takes, and what contextual data influenced the final outcome. OpenTelemetry serves as the foundation for collecting this information in a consistent and portable way, independent of any specific monitoring backend.

Why LLM Observability Is Fundamentally Different

Traditional observability assumes deterministic behavior: the same input produces the same output. LLM systems violate this assumption. The same request can vary due to prompt template changes, retrieval differences, sampling parameters (temperature, top-p), model version upgrades, and context window truncation.​

As a result, teams need visibility into what the model saw, how it was configured, what it retrieved, how long it took, and how much it cost, all correlated to a single user request. Logs alone are insufficient, and metrics lack dimensionality. Distributed traces are the backbone of LLM observability.

Reference Architecture: A Traceable RAG Request

A typical FastAPI-based RAG service follows this flow:

Each step is observable, but only if we deliberately instrument it. The goal is one trace per user request, with child spans representing each logical LLM step.

Reference Architecture Explained

Client Sends a Request to /chat

The architecture begins when a client sends a request to the /chat endpoint. This request typically contains the user's query along with any session or conversation context required by the application.

Keeping the client interface minimal and well-defined is intentional: it ensures the backend receives a predictable input shape and prevents application-specific logic from leaking into downstream LLM processing.

From an observability perspective, this request marks the start of a single end-to-end trace, allowing every subsequent operation to be correlated back to the original user action.

FastAPI Validates Input and Authenticates the User

Once the request reaches the service, FastAPI performs schema validation and authentication. Validation guarantees that only well-formed inputs proceed through the pipeline, while authentication ensures that expensive LLM operations are only executed for authorized users.

Placing this step early reduces unnecessary computation and protects the system from abuse. It also improves trace quality by ensuring that all observed requests represent legitimate execution paths rather than malformed or rejected traffic.

Retriever Queries the Vector Database​

After validation, the system queries a vector database to retrieve documents relevant to the user's request. This retrieval step is the foundation of retrieval-augmented generation (RAG). By grounding the LLM in external knowledge, the system improves factual accuracy and reduces hallucinations.

Separating retrieval from generation allows teams to tune similarity thresholds, embedding models, and top-k values independently, and it makes it easier to diagnose whether poor responses are caused by bad retrieval or model behavior.

Prompt Is Assembled Using Retrieved Documents

With relevant documents in hand, the system constructs the final prompt that will be sent to the LLM. This step combines the user query, retrieved context, system instructions, and formatting rules into a single structured prompt.

Making prompt assembly an explicit stage enables prompt versioning, experimentation, and observability. It also provides a natural place to detect issues such as context window overflows or excessive prompt size before invoking the model.

LLM API Is Invoked

The LLM API call is the most expensive and non-deterministic operation in the pipeline, which is why it occurs only after all preparatory work is complete. At this stage, the model receives a fully constructed prompt and produces a response based on its configuration parameters.

This step is the primary focus of latency, cost, and reliability controls such as retries, timeouts, and circuit breakers. From an observability standpoint, this span becomes the anchor for token usage, cost attribution, and prompt-level debugging.

Response Is Post-Processed and Returned

After the LLM returns a response, the system performs post-processing before sending the result back to the client. This may include formatting, filtering, validation, or enrichment of the output. Post-processing acts as a final safeguard against malformed or low-quality responses and ensures consistency with application requirements. It also provides a clean boundary for attaching evaluation signals, such as response length, relevance scores, or truncation indicators, before the request completes.

Why This Design Is Better Than Simpler Alternatives

This architecture intentionally avoids coupling responsibilities together. Validation, retrieval, prompt construction, model execution, and response handling are all distinct steps. This separation makes the system easier to test, easier to observe, and easier to evolve. When something fails, engineers can identify where and why rather than treating the LLM as a black box.​

Compared to a monolithic "send user input directly to the LLM" approach, this design offers better correctness, lower cost, and higher resilience. It also aligns naturally with distributed tracing, since each block maps cleanly to a trace span with a clear semantic purpose. As the system grows, additional features such as caching, fallback models, or policy enforcement can be added without destabilizing the entire flow.​

Most importantly, this architecture treats the LLM as one component in a larger system, not the system itself. That mindset is essential for building reliable production applications.

LLM Models That Work Best for This Architecture

This architecture is model-agnostic, but certain model characteristics work particularly well with retrieval-augmented workflows.

Models with strong instruction-following and reasoning capabilities tend to perform best, especially when prompts include structured context from retrieved documents. General-purpose models such as GPT-4-class systems perform well when accuracy and reasoning depth are critical.

For lower-latency or cost-sensitive use cases, smaller instruction-tuned models can be effective when paired with high-quality retrieval. Open-source models such as LLaMA-derived or Mistral-based systems also fit well into this architecture, particularly when deployed behind a private inference endpoint.​

The key requirement is not the model itself, but how it is used. Models that can reliably ground their responses in provided context, respect system instructions, and produce stable outputs under varying prompts integrate most cleanly into this design. Because retrieval and prompt construction are explicit stages, models can be swapped or compared without changing the overall system structure.

OpenTelemetry Primer (LLM-Relevant Concepts Only)

OpenTelemetry defines three core types of telemetry data: traces, metrics, and logs. For LLM systems, traces are the most important. To make them useful, you need to understand a few building blocks:

• a trace represents a single end-to-end request

• a span is a timed operation within that trace

• attributes are key–value metadata attached to spans

• events are time-stamped annotations

• context propagation ensures child spans attach to the correct parent.

FastAPI’s async nature makes correct context propagation essential, but OpenTelemetry’s Python SDK handles this as long as spans are created correctly.

With those concepts in place, the next step is to wire OpenTelemetry into the app. Start by configuring the OpenTelemetry SDK in FastAPI: define a TracerProvider, attach a Resource (service name and environment), configure an exporter (Jaeger, Tempo, Phoenix, and so on), and enable FastAPI auto-instrumentation.

Designing LLM-Aware Spans

Span Taxonomy

A clean span hierarchy is critical. In this guide, a single http.request span (usually auto-generated) acts as the root, and it contains child spans such as rag.retrieval, rag.prompt.build, llm.call, llm.postprocess, and, optionally, llm.eval. Each of these spans represents a logical unit of work rather than an implementation detail.

Span Boundaries

Getting span boundaries right is just as important as picking the right span names. Avoid extremes like wrapping the entire LLM workflow in one giant span, creating a separate span for every token, or dumping all data into logs.

Instead, aim for a few coarse-grained spans that each represent a meaningful step in the request, enrich them with well-chosen attributes, and use events to mark important milestones within a span rather than splitting everything into smaller spans.

Instrumenting the LLM Call

When instrumenting the LLM call, treat it as the most critical span in the trace. Whether you are calling OpenAI, Anthropic, or another provider, start the span immediately before the API request and end it only after the full response (or stream) is complete.

Within that span, capture retries, timeouts, and errors so it becomes the central place for latency analysis, cost attribution, and prompt debugging.

For streaming responses, you can emit events for each chunk to track progress, but avoid creating separate child spans unless you truly need fine-grained timing.

FastAPI Example: End-to-End LLM Spans (Complete and Explained)

from fastapi import FastAPI, Request
from opentelemetry import trace
from opentelemetry.trace import Tracer
from typing import List
import asyncio
import hashlib

# Obtain a tracer instance from OpenTelemetry.
# All spans created with this tracer will be part of the same distributed
# tracing system and exported to the configured backend.
tracer: Tracer = trace.get_tracer(__name__)

# Initialize the FastAPI application.
app = FastAPI()

# Helper functions used by the observable endpoint
async def retrieve_documents(query: str) -> List[str]:
    """
    Simulate document retrieval (e.g., vector search or knowledge base lookup).
    This function represents the retrieval stage in a RAG pipeline.
    In a real system, this might query a vector database or search index.
    """
    await asyncio.sleep(0.05)  # Simulate I/O latency
    return [
        "FastAPI enables high-performance async APIs.",
        "OpenTelemetry provides vendor-neutral observability.",
        "LLM observability requires tracing prompts and tokens.",
    ]


def build_prompt(query: str, documents: List[str]) -> str:
    """
    Construct the final prompt from retrieved documents and the user query.
    Prompt construction is kept separate so it can be observed or modified
    independently if needed (for example, to measure prompt assembly latency).
    """
    context = "\n".join(documents)
    return f"""
Context:
{context}

Question:
{query}
"""


class LLMResponse:
    """
    Minimal abstraction for an LLM response.
    This keeps the example self-contained while still allowing us to attach
    token usage and other metadata for observability.
    """

    def __init__(self, text: str, prompt_tokens: int, completion_tokens: int):
        self.text = text
        self.prompt_tokens = prompt_tokens
        self.completion_tokens = completion_token
    
    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

async def call_llm(prompt: str) -> LLMResponse:
    """
    Simulate an LLM API call.
    In a real implementation, this would call OpenAI, Anthropic, or another
    provider. The artificial delay represents model latency.
    """
    await asyncio.sleep(0.2)  # Simulate inference time
    response_text = "FastAPI and OpenTelemetry enable end-to-end LLM observability."
    # Token count is approximated here for demonstration purposes.
    prompt_tokens = len(prompt.split())
    completion_tokens = len(response_text.split())
    return LLMResponse(response_text, prompt_tokens, completion_tokens)


def summarize_response(response: LLMResponse) -> str:
    """
    Example post-processing step.
    Post-processing is separated into its own phase so any additional latency
    or errors are not incorrectly attributed to the LLM itself.
    """
    return response.text


# Observable FastAPI endpoint
@app.post("/query")
async def rag_query(request: Request, query: str):
    """
    Handle a single RAG-style request with explicit OpenTelemetry spans.
    This endpoint demonstrates how to create one trace per request, with child
    spans for retrieval, LLM invocation, and post-processing.
    """

    # Create a top-level span for the HTTP request.
    # Even if FastAPI auto-instrumentation is enabled, defining this explicitly
    # allows us to attach domain-specific metadata.
    with tracer.start_as_current_span("http.request") as http_span:
        http_span.set_attribute("http.method", "POST")
        http_span.set_attribute("http.route", "/query")

        # Retrieval phase
        # This span isolates the retrieval step so that relevance issues can be
        # debugged independently of LLM behavior.
        with tracer.start_as_current_span("rag.retrieval") as retrieval_span:
            retrieval_span.set_attribute("rag.top_k", 5)
            retrieval_span.set_attribute("rag.similarity_threshold", 0.8)
            documents = await retrieve_documents(query)

            # Record how many documents were returned.
            # This is a key signal when diagnosing hallucinations
            # or missing context in the final response.
            retrieval_span.set_attribute(
                "rag.documents_returned",
                len(documents),
            )

        # LLM invocation phase
        # This span wraps the actual LLM call and is the primary anchor for
        # latency, cost, and prompt-related analysis.
        with tracer.start_as_current_span("llm.call") as llm_span:
            llm_span.set_attribute("llm.provider", "example")
            llm_span.set_attribute("llm.model", "example-llm")
            llm_span.set_attribute("llm.temperature", 0.7)
            llm_span.set_attribute("llm.prompt_template_id", "rag_v1")

            # Build the final prompt using retrieved context.
            # The raw prompt is intentionally not stored as a span attribute.
            prompt = build_prompt(query, documents)
            
            # Prompt metadata
            prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
            llm_span.set_attribute("llm.prompt_hash", prompt_hash)
            llm_span.set_attribute("llm.prompt_length", len(prompt))

            response = await call_llm(prompt)

            # Hash the response instead of storing raw text.
            # This allows correlation across traces without exposing content.
            response_hash = hashlib.sha256(
                response.text.encode()
            ).hexdigest()
            llm_span.set_attribute("llm.response_hash", response_hash)

            # Record token usage to enable cost attribution
            # and capacity planning.
            llm_span.set_attribute("llm.usage.prompt_tokens", response.prompt_tokens)
            llm_span.set_attribute("llm.usage.completion_tokens", response.completion_tokens)
            llm_span.set_attribute("llm.usage.total_tokens", response.total_tokens)
            
            # example price per token
            estimated_cost = response.total_tokens * 0.000002
            llm_span.set_attribute("llm.cost_estimated_usd", estimated_cost)

        # Post-processing phase
        # Any transformation after the LLM response is captured here,
        # ensuring inference latency is not overstated.
        with tracer.start_as_current_span("llm.postprocess") as post_span:
            summary = summarize_response(response)
            post_span.set_attribute(
                "llm.summary_length",
                len(summary),
            )

    # Return the final response to the client.
    # All spans above belong to the same distributed trace.
    return {"summary": summary}

Before examining the full code example, it helps to understand how the instrumentation relates to the observability principles described earlier in this article.

The goal of the example is not simply to show how to create spans, but to demonstrate how a single user request can be represented as a structured trace containing meaningful metadata about each stage of the LLM pipeline.

At a high level, the code follows three key design ideas:

1. One trace per user request

2. One span per logical LLM workflow stage

3. Semantic attributes attached to spans for debugging, cost tracking, and analysis

Each of these concepts directly corresponds to the observability practices discussed earlier.

Top-Level Request Span

The FastAPI endpoint begins by creating a top-level span called http.request. This span represents the entire lifecycle of the incoming request and serves as the root span for the trace.

with tracer.start_as_current_span("http.request") as http_span:

Although FastAPI can generate HTTP spans automatically through OpenTelemetry auto-instrumentation, explicitly creating this span allows the application to attach domain-specific metadata such as route names or user identifiers.

Attributes such as the HTTP method and route are attached here:

http_span.set_attribute("http.method", "POST")
http_span.set_attribute("http.route", "/query")

This ensures that every trace can be easily filtered by endpoint when analyzing production traffic.

Retrieval Span

The next span captures the retrieval phase of the RAG pipeline:

with tracer.start_as_current_span("rag.retrieval") as retrieval_span:

This span isolates the vector search or knowledge retrieval step from the rest of the pipeline. If users report irrelevant answers, engineers can inspect this span to determine whether the issue originates from poor retrieval results rather than model behavior.

Several semantic attributes are attached here:

• rag.top_k – number of documents requested

• rag.similarity_threshold – similarity cutoff used for filtering results

• rag.documents_returned – number of documents actually retrieved

These attributes align with the RAG observability signals discussed in the earlier section of the article.

LLM Invocation Span

The most important span in the trace is the llm.call span, which wraps the actual model invocation.

with tracer.start_as_current_span("llm.call") as llm_span:

This span captures the latency, configuration, and token usage associated with the LLM request. In production systems, it becomes the primary location for analyzing model behavior and cost.

Key attributes recorded in this span include:

• llm.provider – the model provider (OpenAI, Anthropic, etc.)

• llm.model – the specific model version

• llm.temperature – sampling parameter controlling response randomness

• llm.prompt_template_id – identifier for the prompt template used

These attributes make it possible to correlate changes in model configuration with downstream quality or cost changes.

Prompt Handling and Privacy

Instead of storing the full prompt or response text directly in the trace, the example demonstrates a safer practice: hashing sensitive data.

response_hash = hashlib.sha256(response.text.encode()).hexdigest()

The resulting hash is stored as a span attribute:

llm_span.set_attribute("llm.response_hash", response_hash)

This approach allows engineers to correlate repeated responses across traces without exposing potentially sensitive content in observability systems.

Token Usage Tracking

The llm.call span also records token usage:

llm_span.set_attribute(
    "llm.usage.total_tokens",
    response.total_tokens
)

Capturing token usage at the span level is critical for monitoring cost and efficiency, since token consumption directly determines billing for most LLM providers.

Post-Processing Span

Finally, the example includes a llm.postprocess span:

with tracer.start_as_current_span("llm.postprocess") as post_span:

This span represents any transformation applied after the model generates its response. Separating post-processing from the LLM call ensures that additional latency — such as formatting, filtering, or validation — is not incorrectly attributed to the model itself.

An attribute such as response length is recorded here:

post_span.set_attribute("llm.summary_length", len(summary))

This can be useful when diagnosing issues such as unexpectedly short or truncated outputs.

How the Spans Form a Complete Trace

When the request finishes, all spans belong to the same distributed trace:

http.request
 ├── rag.retrieval
 ├── llm.call
 └── llm.postprocess

This hierarchy reflects the logical workflow of a retrieval-augmented LLM system. Because each span contains structured metadata, engineers can quickly answer questions such as:

• Was the latency caused by retrieval or model inference?

• How many documents influenced the prompt?

• Which model configuration produced the response?

• How many tokens were consumed?

• Was the response post-processed or truncated?

This structured trace design is what transforms observability from simple monitoring into a practical debugging and optimization tool for LLM systems.

Semantic Attributes: Best Practices for LLM Observability

The goal is not to capture every possible detail, but to record the minimal set of stable, high-signal attributes that enable effective debugging, cost control, and quality analysis in production. Poor attribute design leads to noisy traces, privacy risks, and dashboards that are impossible to reason about.

Prompt, Response, and Model Metadata​

Storing raw prompts is often unsafe and expensive, so it is better to record minimal, structured metadata instead. In practice, this means attaching a stable template identifier with llm.prompt_template_id, a hashed version of the final prompt using llm.prompt_hash (to avoid storing raw text), and a size indicator such as llm.prompt_length, which captures the number of tokens or characters.

You should also always record key inference parameters: llm.provider (for example, "openai" or "anthropic"), llm.model (for example, "gpt-4.1"), llm.temperature and llm.top_p (sampling parameters), llm.max_tokens (the maximum tokens allowed), and llm.stream to indicate whether streaming was enabled, while staying within your organization’s privacy and compliance requirements.

with tracer.start_as_current_span("llm.call") as llm_span:
            llm_span.set_attribute("llm.provider", "example")
            llm_span.set_attribute("llm.model", "example-llm")
            llm_span.set_attribute("llm.temperature", 0.7)
            llm_span.set_attribute("llm.top_p", 0.9)
            llm_span.set_attribute("llm.max_tokens", 512)
            llm_span.set_attribute("llm.stream", False)
            llm_span.set_attribute("llm.prompt_template_id", "rag_v1")

            # Build the final prompt using retrieved context.
            # The raw prompt is intentionally not stored as a span attribute.
            prompt = build_prompt(query, documents)
            
            # Prompt metadata
            prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
            llm_span.set_attribute("llm.prompt_hash", prompt_hash)
            llm_span.set_attribute("llm.prompt_length", len(prompt))

Token Usage and Cost (Why This Matters in Practice)

Token usage is one of the most common blind spots in LLM systems. Many teams monitor latency and error rates but discover runaway costs only after invoices spike. Because token consumption varies significantly by prompt structure, retrieved context, and model configuration, it must be captured explicitly at the span level.​

The most important practice is to record token usage at the end of the LLM span, once the model has completed inference. This ensures that the values reflect the full request rather than partial or streamed output.

At minimum, capture the attributes:​llm.usage.prompt_tokens ,llm.usage.completion_tokens and llm.usage.total_tokens​.

def __init__(self, text: str, prompt_tokens: int, completion_tokens: int):
        self.text = text
        self.prompt_tokens = prompt_tokens
        self.completion_tokens = completion_token
    
    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

async def call_llm(prompt: str) -> LLMResponse:
    """
    Simulate an LLM API call.
    In a real implementation, this would call OpenAI, Anthropic, or another
    provider. The artificial delay represents model latency.
    """
    await asyncio.sleep(0.2)  # Simulate inference time
    response_text = "FastAPI and OpenTelemetry enable end-to-end LLM observability."
    # Token count is approximated here for demonstration purposes.
    prompt_tokens = len(prompt.split())
    completion_tokens = len(response_text.split())
    return LLMResponse(response_text, prompt_tokens, completion_tokens)

These values allow you to distinguish between requests that are expensive because of large prompts (often caused by excessive retrieval or poor prompt construction) versus those that are expensive because of long model-generated outputs.

*Where possible, also attach an estimated cost:*​ llm.cost_estimated_usd​

    # example price per token
    estimated_cost = response.total_tokens * 0.000002
    llm_span.set_attribute("llm.cost_estimated_usd", estimated_cost)

This value is typically derived by multiplying token counts by the model's published pricing. Even if the estimate is approximate, it enables powerful analysis. For example, you can identify which endpoints, prompt templates, or user flows are responsible for the highest cumulative cost, rather than relying on coarse, account-level billing dashboards.

Once spans carry the right attributes, the next step is to connect them to output quality, not just system health.

Evaluation Hooks Inside Traces

This section describes an additional pattern you can layer on top of the core instrumentation in this guide. It is optional and not implemented in the sample code, but it shows how to attach quality signals directly to your traces.

Observability is not just about whether the system stayed up, it is also about whether the model produced a useful answer. Evaluation hooks inside traces let you attach lightweight quality signals directly to the same spans you use for latency and cost.

Inline evaluations are the simplest approach. You can run quick checks synchronously and record the results as span attributes, such as llm.eval.passed for a simple boolean check, llm.eval.relevance_score for an optional numerical score, or flags like llm.eval.hallucination_detected and llm.eval.refusal_detected. These attributes travel with the trace, so you can filter and aggregate on them in your observability backend just like any other field.

For higher accuracy, you can introduce model-based evaluation as a separate step. In this pattern, an evaluator LLM runs asynchronously on the original prompt and response, and its work is captured in a child span (for example, llm.eval) that shares the same trace ID as the main llm.call span. You then attach scores such as relevance, faithfulness, or toxicity to that evaluation span.

Because the evaluation span shares the same trace ID, you can correlate quality regressions with changes in prompts or retrieval.

Exporting and Visualizing Traces (Where This Fits with Vendor Tooling)

This code-first observability design is vendor-agnostic. Once traces are emitted using OpenTelemetry, they can be exported to different backends without changing instrumentation.

General-purpose tracing systems like Jaeger and Grafana Tempo help engineers debug latency, errors, and request flow across retrieval, prompting, and model calls, answering how the system behaved. LLM-focused platforms such as Arize Phoenix use the same data but add model-specific insights like prompt clustering, token analysis, and quality correlation.

Because instrumentation stays OpenTelemetry-native, you maintain full control over attributes and trace structure while still using vendor dashboards, and you can switch backends as your needs evolve without touching the application code.

Operational Patterns and Anti-Patterns

Effective LLM observability requires disciplined practices. High-volume systems should sample traces to limit overhead, and prompts or responses should be hashed by default to reduce storage and privacy risk. Traces must be treated as production data, with proper access control and retention policies.

Common pitfalls include relying only on vendor SDK traces, logging prompts without trace correlation, or ignoring evaluation signals. These issues fragment visibility and hide quality regressions, especially when observability focuses only on agents instead of full application context.

Extending the System

Once traces are reliable, they support advanced capabilities. Metrics like p95 latency can be derived from spans, logs can be linked using trace IDs, and historical traces can power offline evaluation or prompt testing.​

By following OpenTelemetry conventions, the observability stack also stays aligned with emerging LLM semantic standards, keeping the system flexible and future-proof.

Conclusion

End-to-end LLM observability is not achieved by installing another agent. It is achieved through intentional span design, meaningful semantic attributes, and, where needed, lightweight evaluation hooks.​

By treating LLM calls as first-class operations within distributed traces, you gain faster debugging, controlled costs, safer deployments, and measurable quality improvements. The backend — Jaeger, Tempo, Phoenix — is interchangeable. The instrumentation strategy is not.​

A well-designed trace is the most valuable artifact in a production LLM system.

Personalization is one of the most requested features in AI-powered applications. Users expect an agent to remember their preferences, adapt to their style, and improve over time.

In practice, personalization is unfortunately also one of the fastest ways to break an otherwise working AI agent.

Many agents start with a simple idea: keep adding more conversation history to the prompt. This approach works for demos, but it quickly fails in real applications. Context windows grow too large. Irrelevant information leaks into decisions. Costs increase. Debugging becomes nearly impossible.

If you want a personalized agent that survives production, you need more than a large language model. You need a way to connect the agent to tools, manage multi-step workflows, and store user preferences safely over time – without turning your system into a tangled mess of prompts and callbacks.

In this tutorial, you’ll learn how to design a personalized AI agent using three core building blocks:

• Agent Development Kit (ADK) to orchestrate agent reasoning and execution

• Model Context Protocol (MCP) to connect tools with clear boundaries

• Long-term memory to store preferences without polluting context

Rather than focusing on setup commands or vendor-specific walkthroughs, we'll focus on the architectural patterns that make personalized agents reliable, debuggable, and maintainable.

Figure 1 — Personalization influences agent responses

Table of Contents

• Prerequisites

• What “Personalized” Means in a Real AI Agent

• How the Agent Architecture Fits Together

• How to Design the Agent Core with ADK

• How to Connect Tools Safely with MCP

• How to Add Long-Term Memory Without Polluting Context

  • Privacy, Consent, and Lifecycle Controls (Production Checklist)

• How the End-to-End Agent Flow Works

• Common Pitfalls You’ll Hit (and How to Avoid Them)

• What You Learned and Where to Go Next

Prerequisites

To follow along with this tutorial, you should have:

• Basic familiarity with Python

• A general understanding of how large language models work

• Optional: a Google Cloud account if you want to run an end-to-end demo. Otherwise, you can follow the architecture and code patterns locally with stubs. We’ll avoid deep infrastructure setup and focus on design patterns rather than deployment mechanics.

You don’t need prior experience with ADK or MCP. I’ll introduce each concept as it appears.

What “Personalized” Means in a Real AI Agent

Figure 2 — Keep preferences out of the prompt: agent ↔ tools across a protocol boundary

Before writing any code, it’s important to define what personalization means in an AI agent.

Personalization is not the same as “remembering everything.” In practice, agent state usually falls into three categories:

1. Short-term context: Information needed to complete the current task. This belongs in the prompt.

2. Session state: Temporary decisions or selections made during a workflow. This should be structured and scoped to a session.

3. Long-term memory: Durable user preferences or facts that should persist across sessions.

Figure 3 — Three kinds of agent state: context (now), session (today), memory (always)

Most problems happen when these categories are mixed together.

If you store long-term preferences directly in the prompt, the agent’s behavior becomes unpredictable. If you store everything permanently, memory grows without bounds. If you don’t scope memory at all, unrelated sessions start influencing each other.

A well-designed, personalized agent treats memory as a first-class system component, not as extra text added to a prompt.

In the next section, we'll look at how to structure the agent so these concerns stay separated.

By the end of this tutorial, you’ll understand how to design a personalized AI agent that uses long-term memory safely, connects to tools through clear boundaries, and remains debuggable as it grows.

How the Agent Architecture Fits Together

Figure 4 — Reference architecture: agent core + tools + memory service + orchestration runtime

The above diagram shows a high-level, personalized AI agent architecture. In it, an agent core handles reasoning and planning while interacting with a tool interface layer, a long-term memory service, and an orchestration runtime.

Let’s now understand the moving parts of a personalized agent and how they interact.

At a high level, the system has four responsibilities:

1. Reasoning – deciding what to do next

2. Execution – calling tools and services

3. Memory – storing and retrieving long-term preferences

4. Boundaries – controlling what the agent is allowed to do

A common mistake you’ll see is to blur these responsibilities together. For example, letting the model decide when to write memory, or allowing tools to execute actions without clear constraints.

Instead, you'll design the system so each responsibility has a clear owner. The core components look like this:

• Agent core: Handles reasoning and planning

• Tools: Perform external actions (read or write)

• MCP layer: Defines how tools are exposed and invoked

• Memory services: Store long-term user data safely

ADK sits at the center, orchestrating how requests flow between these components. The model never directly talks to databases or services. It reasons about actions, and ADK coordinates execution.

This separation makes the system easier to reason about, debug, and extend.

How to Design the Agent Core with ADK

Before we dive in, a quick note on what ADK is.

Agent Development Kit (ADK) is an agent orchestration framework – the glue code between a large language model and your application. Instead of treating the model as a black box that directly “does things”, ADK helps you structure the agent as a system:

• The model focuses on reasoning (turning user intent, context, and memory into a structured plan)

• Your runtime stays in control of execution (deciding which tools can run, how they run, and what gets logged or persisted)

In other words, ADK is what lets you take tool calling and multi-step workflows out of a giant prompt and turn them into a maintainable and testable architecture. In this tutorial, we’ll use ADK to refer to that orchestration layer. The same patterns apply if you use a different agent framework.

Note: The following code snippets are simplified reference examples intended to illustrate architectural patterns. They’re not production-ready drop-ins.

Once you understand the architecture, you can start designing the agent core. The agent core is responsible for reasoning, not execution.

A helpful mental model is to think of the agent as a planner, not a doer. Its role is to interpret the user’s goal, consider available context and memory, and produce a structured plan that can later be executed in a controlled way.

To make this concrete, the following example shows how an agent can translate user input and memory into an explicit plan. In practice, ADK orchestrates this using a large language model, but the important idea is that the output is structured intent, not side effects.

# Reference example for illustration.

from dataclasses import dataclass
from typing import List, Dict, Any

@dataclass
class Step:
    tool: str
    args: Dict[str, Any]

@dataclass
class Plan:
    goal: str
    steps: List[Step]

def build_plan(user_text: str, memory: Dict[str, Any]) -> Plan:
    # In practice, the LLM produces this structure via ADK orchestration.
    goal = f"Help user: {user_text}"
    steps = []
    if memory.get("prefers_short_answers"):
        steps.append(Step(tool="set_style", args={"verbosity": "low"}))
    steps.append(Step(tool="search_docs", args={"query": user_text}))
    steps.append(Step(tool="summarize", args={"max_bullets": 5}))
    return Plan(goal=goal, steps=steps)

This example illustrates an important constraint: the agent produces a plan, but it doesn’t execute anything directly.

The agent decides what should happen and in what order, while ADK controls when and how each step runs. This separation lets you inspect, test, and reason about decisions before they result in real-world actions.

When personalization is involved, this distinction becomes critical. Preferences may influence planning, but execution should remain tightly controlled by the runtime.

Again, we can consider the agent to be a planner, not a doer.

It should not:

• Perform side effects directly

• Write to databases

• Call external APIs without supervision

In ADK, this separation is natural. The agent produces intents and tool calls, while the runtime controls how and when those calls are executed.

This design has two major benefits:

1. Safety – you can restrict which tools the agent can access

2. Debuggability – you can inspect decisions before execution

When personalization is involved, this becomes even more important. Preferences influence reasoning, but execution should remain tightly controlled.

How to Connect Tools Safely with MCP

Figure 5 — Tool calls with guardrails: request → validate → execute → respond

Tools are how agents interact with the real world. They fetch data, generate artifacts, and sometimes perform actions with side effects.

Without clear boundaries, tool usage quickly becomes a source of fragility. Hardcoded API calls leak into prompts, tools evolve independently, and agents gain more authority than intended.

To avoid these problems, tools should be explicitly registered and invoked through a narrow interface. The following example shows a simple tool registry pattern that mirrors how MCP exposes tools to an agent without tightly coupling it to implementations.

# Reference example (pseudocode for illustration)

from typing import Callable, Dict, Any

ToolFn = Callable[[Dict[str, Any]], Dict[str, Any]]

TOOLS: Dict[str, ToolFn] = {}

def register_tool(name: str):
    def decorator(fn: ToolFn):
        TOOLS[name] = fn
        return fn
    return decorator

@register_tool("search_docs")
def search_docs(args: Dict[str, Any]) -> Dict[str, Any]:
    query = args["query"]
    # Replace with your MCP client call (or local tool implementation).
    return {"results": [f"doc://example?q={query}"]}

def invoke_tool(name: str, args: Dict[str, Any]) -> Dict[str, Any]:
    if name not in TOOLS:
        raise ValueError(f"Tool not allowed: {name}")
    return TOOLS[name](args)

The Model Context Protocol (MCP) provides a clean way to formalize this pattern. You can think of MCP the same way operating systems treat system calls.

An application does not directly manipulate hardware. Instead, it requests operations through well-defined system calls. The kernel decides whether the operation is allowed and how it executes.

In the same way, the agent knows what capabilities exist, MCP defines how those capabilities are invoked, and the runtime controls when and whether they execute.

This separation prevents several common problems, including hardcoded API details in prompts, unexpected breakage when tools change, and agents performing unrestricted side effects.

When designing tools, it helps to classify them by risk: read tools for safe queries, generate tools for planning or synthesis, and commit tools for irreversible actions. In a personalized agent, commit tools should be rare and tightly guarded.

Figure 6 — Observability around tool calls: logs, traces, timing, decision points

How to Add Long-Term Memory Without Polluting Context

Figure 7 — Memory admission pipeline: extract → filter/validate → persist asynchronously

Memory is where personalization either succeeds or fails.

You can start by storing everything the user says and feed it back into the prompt. This works briefly, then collapses under its own weight as context grows, costs rise, and behavior becomes unpredictable.

A better approach is to treat memory as structured, curated data so you can control what the agent remembers and why with clear admission rules. Before persisting anything, the system should explicitly decide whether the information is worth remembering. The following function demonstrates a simple memory admission policy.

# Simplified Reference Only
from typing import Optional, Dict, Any

def memory_candidate(user_text: str) -> Optional[Dict[str, Any]]:
    text = user_text.lower()

    # Durable
    if "for this session" in text or "ignore after" in text:
        return None

    # Reusable
    if "my preferred language is" in text:
        return {"type": "preference", "key": "language", "value": user_text.split()[-1]}

    # Safe (basic example; add PII checks for your use case)
    if "password" in text or "ssn" in text:
        return None

    return None  # default: don’t store

This policy encodes three questions every memory candidate must answer:

• Is it durable? Will it still matter in the future?

• Is it reusable? Will it influence future decisions meaningfully?

• Is it safe to persist? Does it avoid sensitive or session-specific data?

Only information that passes all three checks should become long-term memory. In practice, this usually includes stable preferences and long-lived constraints, not temporary instructions or intermediate reasoning.

Privacy, Consent, and Lifecycle Controls (Production Checklist)

Even if your admission rules are solid, long-term memory introduces governance requirements:

• User control: allow users to view, export, and delete stored preferences at any time.

• Sensitive data handling: never store secrets/PII. Run PII detection on every memory candidate (and consider redaction).

• Retention + consent: use explicit consent for persistent memory and apply retention windows (TTL) so memory expires unless it’s still useful.

• Security + auditability: encrypt at rest, restrict access by service identity, and keep an audit log of memory writes/updates.

Memory writes should also be asynchronous. The agent should never block while persisting memory, which keeps interactions responsive and avoids coupling reasoning to storage latency.

How the End-to-End Agent Flow Works

Figure 8 — End-to-end request lifecycle: user input → plan → tools → memory updates

At this point, you can trace exactly how memory and tools interact during a single request. With the individual components in place, it’s helpful to see how they work together during a single request. The following example walks through the full lifecycle of a personalized interaction, from user input to response.

# Reference example (pseudocode for illustration)

def handle_request(user_id: str, user_text: str) -> str:
    memory = memory_store.get(user_id)  # e.g., {"prefers_short_answers": True}
    plan = build_plan(user_text, memory)

    tool_outputs = []
    for step in plan.steps:
        out = invoke_tool(step.tool, step.args)
        tool_outputs.append({step.tool: out})

    response = render_response(goal=plan.goal, tool_outputs=tool_outputs, memory=memory)

    cand = memory_candidate(user_text)
    if cand:
        # Never block the user on storage.
        memory_store.write_async(user_id, cand)
    return response

At a high level, the flow looks like this:

1. The user sends a message.

2. Relevant long-term memory is retrieved.

3. The agent reasons about the request and produces a plan.

4. ADK invokes tools through MCP as needed.

5. Results flow back to the agent.

6. The agent decides whether new information should be persisted.

7. Memory is written asynchronously.

8. The final response is returned to the user.

Notice what does not happen: the model does not directly write memory, tools do not execute without coordination, and context does not grow without bounds. This structure keeps personalization controlled and predictable.

Common Pitfalls You’ll Hit (and How to Avoid Them)

Even with a solid architecture, there are a few failure modes that show up repeatedly in real systems. Many of them stem from allowing agents to perform irreversible actions without explicit checks.

The following example shows a simple guardrail for commit-style tools that require approval before execution.

# Reference example (pseudocode for illustration)

def invoke_commit_tool(name: str, args: Dict[str, Any], approved: bool) -> Dict[str, Any]:
    if not approved:
        # Require explicit confirmation or policy approval before side effects.
        return {"status": "blocked", "reason": "commit tools require approval"}

    # For example: create_ticket, send_email, submit_order, update_record
    return invoke_tool(name, args)

This pattern forces a clear decision point before side effects occur. It also creates an audit trail that explains why an action was allowed or blocked.

Other common pitfalls include over-personalization, leaky memory that persists session-specific data, uncontrolled tool growth, and debugging blind spots caused by unclear boundaries. If you see these symptoms, it usually means responsibilities are not clearly separated.

What You Learned and Where to Go Next

Personalized AI agents are powerful, but they require discipline. The key insight is that personalization is a systems problem, not a prompt problem.

By separating reasoning from execution, structuring memory carefully, and using protocols like MCP to enforce boundaries, you can build agents that scale beyond demos and remain maintainable in production.

As you extend this system, resist the urge to add “just one more prompt tweak.” Instead, ask whether the change belongs in memory, tools, or orchestration.

That mindset will save you time as your agent grows in complexity.

If you’d like to continue the conversation, you can find me on LinkedIn.

*All diagrams in this article were created by the author for educational purposes.

This is the reality of debugging modern applications. Traditional monitoring wasn't built for containers that live for seconds, services that shift across nodes, or network paths that change constantly.

eBPF changes this. It lets you see inside the kernel itself, watching every system call, every network packet, and every process execution – without modifying a single line of code.

In this tutorial, you will trace a real Kubernetes application using eBPF-powered tools. You’ll learn fundamentals that apply across the entire modern observability ecosystem, with gadgets from the Inspektor Gadget ecosystem.

By the end, you’ll be able to:

• Trace requests as they move through your Kubernetes pods

• Observe behavior at the kernel and syscall level

• Debug failures that logs and metrics simply can’t explain

Prerequisites

Knowledge requirements:

• Basic Kubernetes concepts: pods, deployments, services, namespaces

• Familiarity with kubectl: get, describe, logs, exec

• Container basics

• Basic Linux concepts: processes, system calls

Technical requirements:

• Kubernetes cluster (local or cloud-based)

• kubectl installed and configured

• Cluster admin permissions

• Linux kernel 5.10+ (most managed services have this)

Table of Contents

• Understanding eBPF Observability

• How eBPF Tracing Works (Without Getting Lost in the Kernel)

• How to Set Up Your Environment

• How to Trace Your First Request: Hands-On Tutorial

• How to Interpret Traces

• Real-World Debugging Scenarios

• Advanced Tracing Insights

• Best Practices and Production Considerations

• Next Steps and Resources

Understanding eBPF Observability

eBPF (extended Berkeley Packet Filter) is a technology that allows you to run custom programs inside the Linux kernel without changing kernel code or loading kernel modules.

The Linux kernel is the control center of your operating system. Historically, if you wanted to observe low-level activity (like network packets, system calls, or file operations), you had to rely on kernel changes or kernel modules. Both approaches were fragile, difficult to maintain, and carried real stability and security risks.

eBPF shifts how we approach observability. It provides a safe, sandboxed environment where you can run observability programs directly in the kernel with built-in safety checks that prevent crashes or security vulnerabilities.

Why does this matter for observability?

In traditional observability, you instrument your application code. You add logging statements, metrics libraries, and tracing SDKs. This works, but has significant limitations:

• Code changes are required: You must modify and redeploy applications

• It’s language-specific: Different languages need different libraries

• There will likely be blind spots: You can only see what you explicitly instrument

• The overhead: Heavy instrumentation slows down applications

• Container challenges: By the time you add instrumentation and redeploy, the problem may have disappeared

eBPF takes a different approach. Instead of instrumenting applications, you instrument the kernel. Since every application ultimately makes system calls to the kernel for network I/O, file operations, and process management, you can observe everything from one vantage point.

The eBPF advantage for Kubernetes

Kubernetes adds another layer of complexity. Your application might be spread across multiple containers, pods, and nodes. Traditional APM (Application Performance Monitoring) tools struggle here because containers come and go rapidly, network topology changes constantly, service meshes add routing complexity, and you often don't control application code (think third-party services or legacy applications you can't modify.)

eBPF doesn't care about any of this. It sees all activity at the kernel level, regardless of what language your app is written in, whether it's containerized, how many times the pod has been rescheduled, or whether you have access to modify the source code. This universal visibility is why the Cloud Native Computing Foundation (CNCF) and major cloud providers are betting heavily on eBPF for the future of observability.

How eBPF Tracing Works (Without Getting Lost in the Kernel)

When your application runs on Kubernetes, there's a clear separation between user space and kernel space. Your code runs in user space, where it's isolated, safe, and has limited access to system resources. To do anything useful – make network calls, read files, allocate memory – your application must ask the kernel for help. The kernel handles these requests via system calls, commonly called syscalls.

eBPF lets us hook into these syscalls without slowing the system down. It’s like having a CCTV camera at every doorway between user space and kernel space, watching who passes through, when, and what they’re carrying.

A Simple Example: HTTP Request Tracing

Your application initiates an HTTP GET request, which needs to go through the network stack. To establish a connection, your application first makes a socket() system call to create a network socket. Then it calls connect() to establish a connection to the remote server. Once connected, it uses send() to transmit the HTTP request. Network packets are sent across the wire, and eventually your application calls recv() to receive the response.

With eBPF tools like Inspektor Gadget's Traceloop, you can automatically hook into these syscalls. The eBPF program captures request metadata including source and destination IPs, ports, timing information, and payload sizes. You get a complete trace of the request without touching your application code.

The eBPF Execution Flow

Here's what happens under the hood when you run a trace. When you deploy Inspektor Gadget and run a gadget, several things happen behind the scenes. Once deployed, the eBPF program springs into action whenever a traced event occurs.

When your application makes a syscall, the eBPF hook triggers and quickly collects relevant data: timestamps, process IDs, container IDs, pod names, request details, and latency information. This data is sent to user space through eBPF maps, which are efficient data structures for kernel-to-userspace communication.

Inspektor Gadget adds Kubernetes context to raw kernel data. Instead of seeing only process IDs, you can see pod names, namespaces, labels, and other metadata. For example, you can tell that a request originated from the frontend pod in the production namespace and targeted the backend service.

The gadget then presents this information in a format that's immediately useful, whether you're using the CLI or integrating with other observability tools.

eBPF is fast because:

• JIT compilation: Programs are turned into native machine code for maximum performance

• Event-driven: Only execute when relevant events occur, not continuously polling

• Kernel-resident: No expensive context switching between kernel and user space

• Highly optimized: Typically adds less than 5% overhead even under heavy load

The Tool: Inspektor Gadget & Traceloop

For this tutorial, we're using Traceloop, an eBPF-based tool that traces request flows through applications by observing syscalls, network calls, and I/O operations at the kernel level.

Why are we using Traceloop for this tutorial?

• It’s quick to install and run (one command)

• The output maps directly to the application’s behavior

• It automatically adds Kubernetes context (pod names, namespaces)

• You don’t need to make any application code changes

What you'll learn applies beyond Traceloop. All eBPF tracing tools (Pixie, Cilium Hubble, Tetragon) work the same way under the hood. They attach to kernel hooks and collect event data. Once you understand the concepts here, you can use any eBPF observability tool effectively.

How to Set Up Your Environment

To get your environment ready for hands-on tracing, we'll verify that your cluster meets the requirements, install Inspektor Gadget, and deploy a sample application to trace.

Verify that Your Cluster Meets the Requirements

Before installing anything, confirm that your Kubernetes cluster is ready for eBPF.

Check your Kubernetes version:

kubectl version --short

You need Kubernetes 1.19 or later. Most modern clusters exceed this requirement, but it's worth verifying.

Verify kernel version on your nodes:

kubectl get nodes -o wide

Then check the kernel version on one of your nodes:

# If using a local cluster like minikube or kind
uname -r

# For cloud clusters, you might need to check node details
kubectl debug node/<node-name> -it --image=ubuntu -- bash -c "uname -r"

You need Linux kernel 5.10 or later for the best eBPF support. Kernel 4.18+ works but with some limitations. If you're using a managed Kubernetes service (GKE, EKS, AKS), you almost certainly have a compatible kernel.

Confirm that you have cluster admin permissions:

kubectl auth can-i create deployments --all-namespaces

This should return "yes". Inspektor Gadget needs elevated permissions to load eBPF programs into the kernel.

Install Inspektor Gadget

You can install Inspektor Gadget in several ways. We'll use the kubectl plugin method as it's the most straightforward for learning.

Install the kubectl gadget plugin:

# Download and install kubectl-gadget
kubectl krew install gadget

# Verify installation
kubectl gadget version

If you don't have krew (the kubectl plugin manager), you can install it first:

# Install krew
(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

# Add krew to your PATH
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

Deploy Inspektor Gadget to your cluster:

kubectl gadget deploy

This creates a gadget namespace and deploys the Inspektor Gadget daemon as a DaemonSet, ensuring each node in your cluster can run eBPF programs.

Verify the deployment:

kubectl get pods -n gadget

You should see one gadget-* pod per node, all in the Running state. If a pod is stuck in Pending or CrashLoopBackOff, check that your kernel meets the version requirements.

Deploying a sample application

To learn tracing effectively, we need an application that does something interesting. We'll deploy a simple microservices application with multiple components so you can see traces flowing across service boundaries.

Start by creating a namespace for our demo app:

kubectl create namespace demo-app

Then deploy a simple web application with a backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: gcr.io/google-samples/microservices-demo/frontend:v0.8.0
        ports:
        - containerPort: 8080
        env:
        - name: PORT
          value: "8080"
        - name: PRODUCT_CATALOG_SERVICE_ADDR
          value: "productcatalog:3550"
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: demo-app
spec:
  type: LoadBalancer
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productcatalog
  namespace: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productcatalog
  template:
    metadata:
      labels:
        app: productcatalog
    spec:
      containers:
      - name: server
        image: gcr.io/google-samples/microservices-demo/productcatalogservice:v0.8.0
        ports:
        - containerPort: 3550
        env:
        - name: PORT
          value: "3550"
---
apiVersion: v1
kind: Service
metadata:
  name: productcatalog
  namespace: demo-app
spec:
  selector:
    app: productcatalog
  ports:
  - port: 3550
    targetPort: 3550

Apply the configuration:

kubectl apply -f demo-app.yaml

And wait for pods to be ready:

kubectl wait --for=condition=ready pod -l app=frontend -n demo-app --timeout=300s
kubectl wait --for=condition=ready pod -l app=productcatalog -n demo-app --timeout=300s

Then just verify that everything is running:

kubectl get pods -n demo-app

You should see both frontend and productcatalog pods in the Running state.

Now you’ll need to get the frontend URL:

# For local clusters (minikube, kind, Docker Desktop)
kubectl port-forward -n demo-app service/frontend 8080:80

# Then access http://localhost:8080 in your browser

# For cloud clusters
kubectl get service frontend -n demo-app
# Look for the EXTERNAL-IP

Visit the application in your browser to confirm it's working. You should see a simple e-commerce storefront. This application makes HTTP requests from the frontend to the product catalog service, which is perfect for tracing.

How to Trace Your First Request: Hands-On Tutorial

Now that everything is set up, let's capture our first trace and see eBPF observability in action.

Generate the Traffic to Trace

First, we need some application activity to observe. We will generate a few requests for our demo application.

In one terminal, start the Traceloop gadget:

kubectl gadget traceloop -n demo-app

This command starts tracing HTTP request handling in the demo-app namespace. Inspektor Gadget monitors the kernel to capture the function calls and system events that occur while processing each request.

In another terminal, generate some traffic:

# If using port-forward
curl http://localhost:8080

# If you have an external IP
curl http://<EXTERNAL-IP>

# Generate multiple requests
for i in {1..10}; do curl http://localhost:8080; sleep 1; done
```

### Viewing Your First Trace

Switch back to the terminal running the trace loop gadget. You should see output appearing as requests flow through your application. The output will look something like this:
```
NODE         NAMESPACE   POD              CONTAINER    PID    TYPE       COUNT  
minikube     demo-app    frontend-abc123  frontend     1234   loop       1      
minikube     demo-app    frontend-abc123  frontend     1234   loop       2

Each line shows a traced execution flow, with the count increasing as the same pattern is observed again.

We can make the output more interesting by filtering:

# Stop the previous trace with Ctrl+C, then run:
kubectl gadget traceloop -n demo-app --podname frontend

This narrows our observation to just the frontend pod, reducing noise and making patterns clearer.

Understanding what you're seeing:

Each column shows different information about your application:

• NODE: Which Kubernetes node the traced event occurred on. In multi-node clusters, this helps you understand workload distribution and identify node-specific issues.

• NAMESPACE: The Kubernetes namespace. We filtered to demo-app, so you'll only see that namespace. In production, filtering by namespace is crucial for focusing on specific applications.

• POD: The specific pod where the event occurred. Each pod gets a unique name (like frontend-abc123), allowing you to distinguish between replicas of the same application.

• CONTAINER: Which container within the pod. Pods can have multiple containers (main application, sidecars, init containers), so this helps you pinpoint exactly where activity is happening.

• PID: The process ID inside the container. This is the actual Linux process that made the syscalls eBPF observed. Multiple PIDs might appear if your application uses multiple processes or threads.

• TYPE: The type of event traced. For Traceloop, this identifies kernel-level patterns detected during request processing.

• COUNT: How many times this pattern has been observed. A rapidly incrementing count indicates high request volume.

What this tells you about your application:

Even from this simple output, you can derive insights. If you see events appearing for the frontend pod but not the productcatalog pod, it might indicate that requests aren't making it to the backend. This is a potential configuration issue. If the COUNT increases rapidly for one pod but not others, you know which replica is receiving traffic, useful for debugging load balancing issues.

The real power becomes clear when you correlate these kernel-level observations with what you know about your application. When you made 10 curl requests, you should see corresponding activity in the trace output. This direct relationship between application behavior and kernel observations is the foundation of eBPF observability.

How to Interpret Traces

Understanding raw trace output is valuable, but interpreting what it means for your application's health and performance is where the real skill lies.

Trace Anatomy: Spans, Timing, and Request Flow

A trace represents a single request's journey through your system. When you curl the frontend, that generates one trace. A span represents a single operation within that trace like "frontend handles request," "frontend calls product catalog," "product catalog queries data," and "frontend returns response." Each span has timing information: when it started, when it ended, and therefore how long it took.

In traditional distributed tracing with OpenTelemetry or Jaeger, you'd explicitly create these spans in your application code. With eBPF, the tool infers spans from syscall patterns. When eBPF sees your frontend process call connect() to the product catalog's IP, followed by send() and recv(), it understands that's a span representing an HTTP request to the backend service.

The request flow is the sequence of spans showing how your request moved through services. In our demo app,

1. The user request arrives at the frontend,

2. the frontend connects to the product catalog,

3. the product catalog processes the request,

4. the product catalog returns the data, the frontend renders the page,

5. and finally, the response is sent to user.

How to Follow Requests Across Services

Let's trace a request across service boundaries to see this flow in action.

First, we’ll start a more detailed trace:

kubectl gadget trace_tcp -n demo-app

The trace_tcp gadget shows network connections, giving us visibility into service-to-service communication.

Next, generate a request:

curl http://localhost:8080

In the trace output, look for connection patterns:

You should see the frontend pod establishing a TCP connection to the product catalog service. The trace will show the source (frontend) and destination (product catalog) IPs and ports, along with timing information.

This is how eBPF lets you follow requests: by observing the network syscalls that implement service communication. You don't need a service mesh or instrumentation libraries, the kernel sees all network activity and eBPF captures it.

Understanding the flow:

1. Your curl command triggers a TCP connection to the frontend pod's IP on port 8080

2. The frontend processes the request and opens a TCP connection to the product catalog's IP on port 3550

3. Data flows back and forth (you'll see send/receive events)

4. Connections close when requests complete

Each step is visible to eBPF because each step requires syscalls that the kernel handles.

How to Identify Bottlenecks and Errors

We can also use tracing to identify performance issues.

First, let’s start by simulating a slow backend:

# Create a deliberately slow endpoint by modifying our deployment
kubectl scale deployment productcatalog -n demo-app --replicas=0

# Wait a moment, then scale back up
kubectl scale deployment productcatalog -n demo-app --replicas=1

While the product catalog is down, generate some requests:

for i in {1..5}; do curl http://localhost:8080; done

You should see connection attempts from the frontend to the product catalog, but if the service is unavailable, you'll see different patterns, possibly connection timeouts or connection refused errors, depending on the exact timing.

What bottlenecks look like in traces:

• Long spans: A span that takes significantly longer than others indicates a bottleneck. In trace loop output, you might see gaps between events or notice certain operations taking longer.

• Retries: Repeated connection attempts to the same destination suggest a failing or slow service.

• Error patterns: Connection failures, timeouts, or unusual syscall sequences indicate problems.

The best skill to have is pattern recognition. A typical, healthy request flow has a rhythm, and events occur in predictable sequences with consistent timing. When something breaks, the rhythm changes. Requests take longer, errors appear, or expected events don't occur at all.

Real-World Debugging Scenarios

Now let's go through three realistic scenarios where eBPF helps:

Scenario 1: Finding a Slow Endpoint

The problem: Users report that the product catalog page sometimes loads very slowly, but metrics show normal average latency.

Let’s use Traceloop to investigate:

# Start tracing with timing information
kubectl gadget traceloop -n demo-app --podname frontend

We’ll generate some mixed traffic:

# Some requests to the homepage (fast)
curl http://localhost:8080

# Some requests to the product catalog (potentially slow)
curl http://localhost:8080/products

In the trace output, compare the COUNT increments for different request patterns. If certain patterns show significantly more loop iterations or longer gaps between events, that indicates those requests are doing more work, possibly hitting a slow endpoint.

The diagnosis:

You might notice that requests to /products cause the frontend to make multiple calls to the product catalog service (visible with kubectl gadget trace_tcp), while homepage requests don't. This explains why the product page is slow: it's making synchronous calls to a backend service, and if that service is slow or the network is congested, users feel the delay.

The fix:

You might implement caching, make the backend calls asynchronous, or optimize the product catalog service itself. The key is that eBPF helped you identify which specific code path was slow without adding instrumentation to your application.

Scenario 2: Tracking Down Failed Requests

The problem: Your monitoring shows a 5% error rate, but application logs don't show any errors. Where are the failures happening?

Now let’s use eBPF to investigate:

# Trace network connections to see connection failures
kubectl gadget trace_tcp -n demo-app

We’ll simulate intermittent failures:

# Create a failing scenario by temporarily breaking service connectivity
kubectl delete service productcatalog -n demo-app

# Generate requests
for i in {1..10}; do curl http://localhost:8080; sleep 1; done

# Restore the service
kubectl apply -f demo-app.yaml

In the TCP trace, you'll see connection attempts from the frontend to the product catalog that fail or time out. The trace will show the source, destination, and what happened (connection refused, timeout, and so on).

The diagnosis:

The failures are happening at the network level, the frontend can't reach the product catalog. This might be due to network policy issues, service mesh misconfiguration, or DNS problems. Traditional application logs might not capture this because the application never receives a response to log, and the connection fails before the application layer even gets involved.

Why eBPF finds this when logs don't:

Your application logs what it experiences. If a connection fails at the TCP level, your application might just see "connection refused" and retry without detailed logging.

eBPF sees the actual syscalls and network events, giving you visibility into what's happening beneath your application layer.

Scenario 3: Understanding Service Dependencies

The problem: You're not sure which services depend on each other, and you want to understand the actual runtime dependencies before making changes.

We’ll use eBPF to map dependencies:

# Trace all TCP connections to see who talks to whom
kubectl gadget trace_tcp -n demo-app

And then generate normal traffic:

# Make various requests to exercise different code paths
curl http://localhost:8080
curl http://localhost:8080/products
curl http://localhost:8080/cart

The trace output shows source and destination for every connection. Build a mental (or actual) map of which pods connect to which services.

The discovery:

You'll see that the frontend pod connects to the product catalog service, but you might also discover unexpected dependencies. Perhaps the frontend also makes calls to a Redis cache, an authentication service, or external APIs. These runtime dependencies might not be documented or might differ from what architectural diagrams show.

Why this matters:

Before deploying a change to the product catalog service, you now know exactly which services will be affected. Before implementing a network policy, you know which connections to allow. Before decomposing a monolith, you understand the actual communication patterns.

This is observability-driven architecture understanding: letting the system show you how it actually works, not how you think it works.

Advanced Tracing Insights

Once you're comfortable with basic request tracing, Inspektor Gadget offers deeper observability capabilities that reveal even more about your system's behavior.

Syscall-Level Observation

The traceloop and trace_tcp gadgets give you application-level insights, but sometimes you need to go deeper. The trace_exec gadget shows you every process execution in your containers.

First, let’s monitor process execution:

kubectl gadget trace_exec -n demo-app

And generate activity:

# Exec into a pod and run commands
kubectl exec -it -n demo-app deployment/frontend -- /bin/sh
ls -la
ps aux
exit

Every command you run inside the container appears in the trace: /bin/sh, ls, ps, and anything else. This helps you understand what's running in your containers, detect suspicious activity, or debug initialization issues.

In production scenarios, this helps you answer questions like: Is my application spawning unexpected subprocesses? Are there security issues like someone running curl to download malicious scripts? Is my init script actually running the commands I think it is?

Network Tracing Insights

Beyond TCP connections, you can trace DNS queries, which often reveal surprising things about your application's behavior.

Run trace_dns:

kubectl gadget trace_dns -n demo-app

Generate requests:

curl http://localhost:8080

You'll see every DNS query your application makes: resolving service names, checking for external APIs, perhaps even unexpected queries that indicate misconfiguration or dependencies you didn't know about.

Common insights from DNS tracing include discovering that your application is using external dependencies you didn't document, finding DNS resolution failures that cause intermittent errors, or identifying excessive DNS queries that could be cached.

Combining eBPF Data with Logs and Metrics

eBPF observability delivers the best results when combined with traditional observability signals. To combine them effectively:

• Use metrics for high-level health monitoring, alerting on anomalies, tracking trends over time, and dashboard visualization.

• Use logs for application-specific context, business logic details, error messages with stack traces, and debugging application code.

• Use eBPF traces for understanding request flows, identifying where time is spent, discovering runtime dependencies, and debugging issues that don't appear in logs.

A practical workflow:

Your metrics alert you that latency increased. You check logs but don't see errors, requests are succeeding, just slowly. You use eBPF tracing to identify that requests are spending extra time in network I/O to a particular backend service. Now you check that service's metrics and logs, and discover it's under heavy load. The eBPF trace gave you the clue that logs and metrics alone couldn't provide.

This approach to observability, using the right tool for each question, is how experienced engineers debug complex systems efficiently.

What eBPF Can and Can't See

eBPF excels at:

• Network traffic (requests, responses, latency)

• System calls (file I/O, process creation, memory allocation)

• Kernel functions (scheduling, locking, resource usage)

• Function calls in binaries (with uprobes)

But keep in mind that eBPF has limitations:

• Cannot decrypt encrypted payloads (unless hooking SSL libraries before encryption)

• Doesn't automatically understand application logic

• Captures low-level events but may need context for high-level semantics

That's why eBPF complements traditional observability rather than replacing it entirely. It gives you infrastructure-level visibility with no code changes and universal coverage. Traditional APM provides application-level context, business metrics, and custom instrumentation. Together, they give you complete observability across your entire stack.

Best Practices and Production Considerations

Before using eBPF tracing in production, there are important considerations around performance, security, and operational practices.

Performance Impact

eBPF's reputation for low overhead is well-deserved, but "low" isn't "zero."

Most eBPF tracing tools add 2-5% CPU overhead and negligible memory overhead. The exact number depends on event frequency, tracing a service that handles 10,000 requests per second will have more overhead than one handling 10 per second.

Measuring the impact:

# Before enabling tracing, check baseline resource usage
kubectl top pods -n demo-app

# Enable tracing
kubectl gadget traceloop -n demo-app

# Check resource usage again
kubectl top pods -n demo-app

You should see a small increase in CPU usage in the pods where tracing is active. This is the cost of the eBPF programs running in the kernel and processing events.

Production best practices:

Use targeted tracing rather than tracing everything everywhere. Trace specific namespaces, pods, or individual containers when investigating issues. For high-volume services, reduce overhead by applying filters, aggregation, or sampling where supported by the tracing tool.

Stop tracing when you’re done investigating. Unlike metrics collection, which typically runs continuously, eBPF-based tracing is best used as an on-demand diagnostic tool to capture detailed insights during active debugging.

When overhead matters:

If you're running latency-sensitive applications (like high-frequency trading systems or real-time communications), even 2-5% overhead might be unacceptable. In these cases, use eBPF tracing in pre-production environments to identify issues, or enable it temporarily in production only when actively debugging.

Security Considerations

eBPF is powerful, which means it requires elevated privileges. Understanding the security implications is crucial.

What eBPF can access:

eBPF programs can observe all syscalls, network traffic, and process execution in the kernel. This includes potentially sensitive data like connection details, file paths, and process arguments. While eBPF programs run in a sandbox and can't modify data or crash the kernel, they can read information that might be sensitive.

Privilege requirements:

Loading eBPF programs requires CAP_SYS_ADMIN or CAP_BPF capabilities (on newer kernels). This is a privileged operation, only trusted users should have this access. The Inspektor Gadget DaemonSet runs with these privileges, so protect access to it accordingly.

Best practices:

Implement RBAC (Role-Based Access Control) to restrict who can run gadgets. Not every developer needs the ability to trace production systems.

Also, be mindful of what data you're collecting, if your traces might contain sensitive information (like authentication tokens in HTTP headers), restrict access to trace data.

Lastly, consider using admission controllers to prevent unauthorized eBPF program loading. Audit eBPF usage in production environments to track who ran which gadgets when.

Network policies:

Inspektor Gadget's DaemonSet needs to communicate with the API server and between its components. Ensure your network policies allow this communication while still maintaining appropriate segmentation.

When to Use eBPF Tracing vs. Traditional APM

eBPF tracing and traditional APM tools like New Relic, Datadog, or Dynatrace serve different purposes. Understanding when to use each helps you build an effective observability strategy.

Use eBPF tracing when:

• You can't modify application code (third-party applications, legacy systems, compiled binaries)

• You need infrastructure-level visibility (network, syscalls, kernel behavior)

• You're debugging issues that span service boundaries but don't show up in application logs

• You want zero instrumentation overhead during normal operation (run tracing only when needed)

• You need to understand what's actually happening versus what the application reports

Use traditional APM when:

• You need business-context metrics (user IDs, transaction types, business-specific data)

• You want automatic instrumentation with minimal setup for supported frameworks

• You need long-term storage and analysis of all traces (eBPF tracing is often used for real-time investigation)

• You want pre-built dashboards and alerting for common application patterns

• You need application code-level visibility (stack traces, variable values, function calls)

The Ideal Approach: Use Both

Many teams run traditional APM for continuous monitoring and use eBPF tracing for targeted investigation when APM data isn't sufficient. For example, your APM shows that a service is slow but doesn't explain why. You enable eBPF tracing on that service to understand what's happening at the kernel level, network delays, excessive syscalls, unexpected dependencies, and find the root cause.

This complementary approach gives you both the continuous visibility of APM and the deep diagnostic power of eBPF without the overhead of running both at maximum depth all the time.

Next Steps and Resources

If you got this far, thanks for reading! Now that you have learned the fundamentals of eBPF observability, and hands-on tracing with Inspektor Gadget, you can continue your journey by:

Exploring Other eBPF Tools

Now that you understand eBPF concepts through traceloop, exploring other tools will be much easier.

Try other Inspektor Gadget gadgets:

# See all available gadgets
kubectl gadget --help

# Some useful ones to explore:
kubectl gadget trace_open -n demo-app     # File I/O tracing
kubectl gadget trace_bind -n demo-app     # Port binding events
kubectl gadget profile cpu -n demo-app    # CPU profiling
kubectl gadget snapshot process -n demo-app  # Process listing

Each gadget teaches you something different about system behavior and gives you another diagnostic tool in your toolkit.

Experiment with other eBPF platforms:

If you're interested in broader observability platforms, try Pixie for its auto-instrumentation and rich UI. Install Cilium with Hubble if you're focused on network observability and want to understand service mesh behavior. Explore Tetragon if security observability interests you, seeing what processes are executing and what files they're accessing.

The concepts transfer directly: all these tools attach eBPF programs to kernel hooks, collect event data, and present it in different ways. Your understanding of syscalls, traces, and kernel-level observation applies universally.

Connect to the CNCF Observability Ecosystem

eBPF observability tools don't exist in isolation. They're part of the broader Cloud Native Computing Foundation ecosystem.

OpenTelemetry integration:

Many eBPF tools can export data in OpenTelemetry format, allowing you to combine kernel-level traces with application-level traces in a unified observability backend. This gives you the complete picture: eBPF shows you infrastructure behavior while OpenTelemetry shows you application context.

Prometheus and Grafana:

eBPF-derived metrics can be exposed as Prometheus metrics and visualized in Grafana alongside your application metrics. This unified dashboard approach helps you correlate infrastructure and application behavior.

Service mesh integration:

If you're using Istio, Linkerd, or other service meshes, eBPF tools like Cilium Hubble can provide deeper visibility into service-to-service communication than the mesh alone provides. The mesh handles traffic management while eBPF gives you kernel-level visibility.

Jaeger and Zipkin:

For organizations using distributed tracing backends, eBPF traces can be exported to these systems, enriching your trace data with infrastructure-level spans that application instrumentation misses.

Community Resources and Learning Paths

The eBPF community is vibrant and welcoming. You can continue learning from the resources below.

Official documentation and blog:

• eBPF.io: The central hub for eBPF documentation, tutorials, and project listings

• Inspektor Gadget docs: Comprehensive guides for all gadgets and use cases

• Cilium documentation: Deep dives into eBPF networking

• CNCF Blog — “What is Observability 2.0?: A quick overview of how modern observability moves beyond traditional tools by unifying metrics, logs, and traces for real-time insight in cloud-native systems.

Learning resources:

• Learning eBPF by Liz Rice: Comprehensive book covering eBPF fundamentals

• eBPF Summit: Annual conference with talks from eBPF creators and users

• CNCF webinars: Regular sessions on observability topics

• Kubernetes observability SIGs: Community discussions and projects

To make this tutorial easy to follow and experiment with, I have included all Kubernetes manifests, demo applications, and eBPF tracing commands in this repository. You can also connect with me on LinkedIn if you’d like to stay in touch.

What if you had a flight recorder for your applications, something that captures every system call in real-time, so you can "rewind" and see the exact sequence of events that led to a crash? That's what Traceloop does. It continuously traces system calls in your pods, giving you a detailed replay of what happened before, during, and after issues occur.

In this guide, you’ll learn how to use Traceloop's system call tracing to debug pod issues that would otherwise be nearly impossible to diagnose.

Prerequisites

Before we begin, here are some prerequisites – things you’ll need to know and have:

• Basic Kubernetes concepts: Understanding of pods, deployments, services, and namespaces

• kubectl fundamentals: Comfortable with commands like kubectl get, kubectl describe, kubectl logs, and kubectl exec

• Container basics: Understanding how containerized applications work

• Basic Linux concepts: Understanding of processes and system calls (helpful, but we'll explain as we go)

Technical Requirements

• Kubernetes cluster access: Local (minikube, kind, Docker Desktop) or cloud-based cluster

• kubectl installed and configured to connect to your cluster

• Sufficient permissions (cluster admin or equivalent RBAC) to:

  • Install and run eBPF-based tools (Traceloop uses eBPF)

  • Create/modify pods and deployments

  • Access pod logs and system-level data

• Linux-based Kubernetes nodes: Most clusters already run on Linux.

System Requirements

• Extended Berkeley Packet Filter (eBPF) support: Used for tracing and monitoring at the kernel level. Kernel version 5.10+ recommended.

• Sufficient cluster resources: Traceloop runs alongside your applications

Table of Contents

1. What is Traceloop?

2. How Traceloop Works

3. How to Set Up Traceloop

4. Your First Trace: Hands-On Tutorial

5. Step-by-Step Debugging Walkthrough

6. Real-World Debugging Scenarios

7. Best Practices

8. Conclusion

What is Traceloop?

Traceloop is a system call tracing and observability tool that works across containerized environments, from Docker containers running locally to pods in production Kubernetes clusters. But before we discuss what that means, let's talk about why system calls matter for debugging.

Every time your application does anything (like opening a file, making a network request, allocating memory, or crashing), it has to interact with the operating system through system calls. These are the fundamental building blocks of how any program interacts with the world around it.

Here's where traditional debugging falls short: when your container crashes, the logs might tell you "segmentation fault" or "out of memory," but they don't tell you the sequence of events that led there. Did the application try to access a file that didn't exist? Was it making network calls that failed? Did it run out of file descriptors?

Traceloop captures this missing piece. It sits at the kernel level using eBPF technology, recording every system call your application makes in real-time. Think of it as installing a dashcam in your application. It's always recording with minimal resources, and when something goes wrong, you have the footage.

Strace is another popular debugging tool – but it requires you to know that there's a problem first. With Traceloop, we can conveniently run it continuously in the background with minimal overhead. If your container crashes at 3am, you can immediately "rewind the tape" and see exactly what system calls happened leading up to the crash.

This helps debug intermittent issues that happen randomly in production but never when you are watching. Because Traceloop is always recording, you finally have visibility into what your application was doing when these mysterious failures occur.

How Traceloop Works

Now that you understand what Traceloop does, let's look under the hood at how it captures and processes system calls in your containerized environments.

The Technical Foundation

Traceloop is built on eBPF, a technology that allows programs to run safely in the Linux kernel without changing kernel code. Think of eBPF as a way to install "hooks" directly into the kernel that can observe everything happening on your system with minimal performance impact.

Unlike traditional monitoring tools that work from userspace, eBPF programs run in kernel space, giving them access to system calls as they happen, without relying on the application logging appropriate error messages. This is why Traceloop can capture events that never make it to application logs, like failed system calls or crashes that happen before the application can write anything.

The Flight Recorder Architecture

Traceloop uses eBPF maps as an overwriteable ring buffer. Imagine a tape recorder that continuously records over itself. It's always capturing system calls, but it only keeps the most recent data in memory. When something goes wrong, the recording automatically preserves what happened leading up to the incident, just like an airplane's flight recorder after a crash.

This approach solves the production debugging problem: you don't need to predict when issues will happen or attach debuggers after the fact. The recording is always running, waiting for you to need it.

System Call Capture Flow

Here's how Traceloop captures and processes system calls across your Kubernetes environment:

1. Application pods generate system calls through normal operation – opening files, making network connections, allocating memory.

2. eBPF probes (also called hooks) intercept these system calls at the kernel level before they're processed.

3. Traceloop recorder captures the events, buffers them, and adds container context using Inspektor Gadget enrichment (pod name, namespace, container ID).

4. Output stream formats the data and makes it available for analysis in real-time or after an incident.

5. Traceloop user views and analyzes the captured trace to diagnose the root cause of issues.

Below is a visual representation of the flow. The key advantage is that Traceloop sees everything your application does, even actions that fail silently or happen too quickly for traditional logging to catch. This gives you complete visibility into your application's interaction with the operating system.

Container Isolation and Context

One of Traceloop's strengths is understanding containerized environments. It doesn't just capture raw system calls – it adds context about which pod, container, and namespace generated each call. This means you can trace specific applications without getting overwhelmed by system calls from other containers running on the same node.

This container awareness makes Traceloop particularly powerful in Kubernetes environments where you might have dozens of pods running on a single node, but you only care about debugging one specific application.

How to Set Up Traceloop

Before we can start tracing system calls, we need to set up Traceloop in your Kubernetes environment. Traceloop is part of the Inspektor Gadget ecosystem, which provides flexibility in how you use it.

Installation Overview

This setup:

• Deploys Inspektor Gadget components to all worker nodes

• Eliminates the download and initialization overhead on each use, as components are pre-loaded and ready

• Eliminates the need to reinstall or reconfigure for each debugging session – just run your traces immediately

• Requires cluster admin permissions

• Works best for teams doing regular debugging

Installation Requirements

First, ensure your cluster meets the requirements:

• Kubernetes cluster with Linux nodes

• eBPF support

• kubectl installed and configured

• Cluster admin permissions

Install kubectl gadget

The recommended way is using krew (kubectl plugin manager):

# Install krew if you don't have it
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz"
tar zxvf krew-linux_amd64.tar.gz
./krew-linux_amd64 install krew
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

# Install kubectl gadget
kubectl krew install gadget

Alternatively, you can install directly:

# For Linux/macOS
curl -sL https://github.com/inspektor-gadget/inspektor-gadget/releases/latest/download/kubectl-gadget-linux-amd64.tar.gz | sudo tar -C /usr/local/bin -xzf - kubectl-gadget

# Verify installation
kubectl gadget version

Deploy Inspektor Gadget to Your Cluster

Deploy the Inspektor Gadget components to your cluster:

kubectl gadget deploy

This installs the necessary DaemonSets and RBAC configurations that allow gadgets like Traceloop to run on your cluster nodes.

Alternatively, you can also deploy using Helm.

Verify Installation

Check that the gadget pods are running:

kubectl get pods -n gadget

You should see gadget pods running on each node in your cluster.

Your First Trace: Hands-On Tutorial

Now let's capture our first system call trace. We'll create a simple scenario and watch what happens at the system level.

Setting Up the Test Environment

First, create a dedicated namespace for our tracing experiments:

kubectl create ns test-traceloop-ns

Expected output:

namespace/test-traceloop-ns created

Next, create a simple pod that we can interact with:

kubectl run -n test-traceloop-ns --image busybox test-traceloop-pod --command -- sleep inf

Expected output:

pod/test-traceloop-pod created

This creates a BusyBox container that sleeps indefinitely, giving us a stable target for tracing.

Starting Your First Trace

Next, start tracing system calls for our test pod:

kubectl gadget run traceloop:latest --namespace test-traceloop-ns

This command starts the flight recorder. You'll see column headers showing what information Traceloop captures:

K8S.NODE    K8S.NAMESPACE    K8S.PODNAME    K8S.CONTAINERNAME    CPU    PID    COMM    SYSCALL    PARAMETERS    RET

The trace is now running in the background, continuously recording system calls from our pod.

Generating System Calls

With the trace running, let's generate some activity. In a new terminal window, run a command inside your test pod:

kubectl exec -ti -n test-traceloop-ns test-traceloop-pod -- /bin/sh

Once inside the container, run some basic commands:

ls /
echo "Hello World" > /tmp/test.txt
cat /tmp/test.txt

Collecting the Trace

Back in your original terminal where Traceloop is running, press Ctrl+C to stop the recording and see the captured system calls.

You'll see output similar to this:

K8S.NODE            K8S.NAMESPACE        K8S.PODNAME          K8S.CONTAINERNAME    CPU  PID    COMM  SYSCALL      PARAMETERS                   RET
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    openat       dfd=-100, filename="/lib"    3
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    getdents64   fd=3, dirent=0x...          201
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    write        fd=1, buf="bin dev etc..."   201
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    exit_group   error_code=0                 0

Understanding Your First Trace

Let's break down what we're seeing:

• K8S.PODNAME: Which pod generated these system calls

• PID: Process ID of the command that ran

• COMM: The command name (ls, echo, cat)

• SYSCALL: The actual system call made (openat, write, exit_group)

• PARAMETERS: Arguments passed to the system call

• RET: Return value (0 usually means success)

This trace shows the ls command opening the /lib directory, reading directory entries, writing the output to stdout, and exiting successfully.

Clean Up

Remove the test resources:

kubectl delete pod test-traceloop-pod -n test-traceloop-ns
kubectl delete ns test-traceloop-ns

You can now see exactly what your applications are doing at the kernel level, something that traditional logs and kubectl commands can't show you.

Let's try this with an application that crashes.

Step-by-Step Debugging Walkthrough

Now that you know how to capture traces, let's take a look at a real debugging scenario. We'll create an application that crashes and use Traceloop to uncover the root cause. Something that would be nearly impossible with traditional kubectl debugging.

The Scenario: A Mysterious Crash

Let's create a Python application that has a subtle bug. It tries to write to a file it doesn't have permission to access, then crashes. This mimics real-world scenarios where applications fail due to permission issues, missing files, or resource constraints.

Setting Up the Problematic Application

First, we’ll create a new namespace for our debugging exercise:

kubectl create ns debug-traceloop-ns

Now, let's create a pod with an application that will crash:

kubectl run -n debug-traceloop-ns crash-app --image=python:3.9-slim --restart=Never -- python3 -c "
import time
import os
print('App starting...')
time.sleep(5)
print('Trying to write to restricted file...')
try:
    with open('/etc/passwd', 'w') as f:
        f.write('malicious content')
except Exception as e:
    print(f'Error: {e}')
    exit(1)
"

This creates a pod that will:

1. Start successfully

2. Try to write to /etc/passwd (a restricted system file)

3. Fail and crash with exit code 1

Starting the Trace Before the Crash

Here's the key difference from traditional debugging. We start tracing before we know there's a problem. In a real scenario, you'd have Traceloop running continuously.

kubectl gadget run traceloop:latest --namespace debug-traceloop-ns

The trace starts recording immediately. You'll see the column headers, and the flight recorder is now capturing every system call.

Observing the Application Behavior

In another terminal, check the pod status:

kubectl get pods -n debug-traceloop-ns -w

You'll see the pod go through these states:

• Pending → Running → Error → CrashLoopBackOff

Traditional debugging would show you:

kubectl logs -n debug-traceloop-ns crash-app

Output:

App starting...
Trying to write to restricted file...
Error: [Errno 13] Permission denied: '/etc/passwd'

But this doesn't tell you exactly what the application tried to do at the system level.

Collecting and Analyzing the Trace

Back in your Traceloop terminal, press Ctrl+C to stop the recording. You'll see system calls like this:

K8S.NODE        K8S.NAMESPACE      K8S.PODNAME  COMM    SYSCALL    PARAMETERS                           RET
minikube-docker debug-traceloop-ns crash-app    python3 openat     dfd=-100, filename="/etc/passwd"    -13
minikube-docker debug-traceloop-ns crash-app    python3 write      fd=3, buf="App starting..."         16
minikube-docker debug-traceloop-ns crash-app    python3 openat     dfd=-100, filename="/etc/passwd"    -13
minikube-docker debug-traceloop-ns crash-app    python3 exit_group error_code=1                        0

Reading the System Call Story

The trace reveals the exact sequence of events:

1. openat filename="/etc/passwd" RET=-13: The application tried to open /etc/passwd for writing

   • Return code -13 = EACCES (Permission denied)

2. write buf="App starting...": Normal logging output (successful)

3. openat filename="/etc/passwd" RET=-13: Second attempt to open the restricted file (still denied)

4. exit_group error_code=1: Application exits with error code 1

What Traceloop Revealed

Traditional debugging told us "Permission denied" but Traceloop shows us:

• Exactly which file the application tried to access

• When the permission denial happened in the execution flow

• How many times it tried (twice in this case)

• The exact system call that failed (openat)

Real-World Applications

This same approach works for debugging:

• File not found errors: See exactly which files your app is looking for

• Network connection failures: Observe failed connect() system calls with specific addresses

• Memory issues: Watch mmap() and brk() calls that fail

• Container startup problems: See which system calls fail during initialization

Clean Up

Remove the test resources:

kubectl delete pod crash-app -n debug-traceloop-ns
kubectl delete ns debug-traceloop-ns

Key Takeaway

Traditional Kubernetes debugging shows you what went wrong after it happened. Traceloop's continuous recording shows you exactly how it went wrong at the system level. This level of detail is invaluable for debugging complex production issues where the logs don't tell the full story.

Real-World Debugging Scenarios

Now that you understand the fundamentals, let's explore common production issues and how Traceloop helps diagnose them. These scenarios mirror real problems you'll encounter in Kubernetes environments.

Scenario 1: Container Startup Failures

The problem: Your pod gets stuck in CrashLoopBackOff with unhelpful logs.

Traditional kubectl commands show limited information:

kubectl describe pod failing-app
# Events: Back-off restarting failed container

kubectl logs failing-app
# (Empty or minimal output)

System calls show the application tried to:

1. Access configuration files that don't exist

2. Connect to services that aren't available

3. Write to directories without proper permissions

Key system calls to watch:

1. openat with -2 return (file not found)

2. connect with -111 return (connection refused)

3. access with -13 return (permission denied)

Scenario 2: Memory and Resource Issues

The problem: Application performance degrades or gets OOMKilled.

What Traceloop shows:

1. mmap calls failing (memory allocation issues)

2. brk system calls indicating heap growth

3. File descriptor exhaustion through failed openat calls

4. Excessive write calls indicating memory pressure

Example pattern:

SYSCALL    PARAMETERS           RET
mmap       length=1048576       -12  # ENOMEM - out of memory
brk        brk=0x55555557d000   0    # Heap expansion
openat     filename="/tmp/..."   -24  # EMFILE - too many open files

Scenario 3: Network Connectivity Problems

The problem: Service-to-service communication fails intermittently.

Traditional debugging limitations:

1. Application logs show "connection timeout"

2. Network policies seem correct

3. DNS resolution appears to work

What Traceloop reveals:

1. Exact IP addresses and ports being attempted

2. DNS resolution patterns through openat on /etc/resolv.conf

3. Failed connect calls with specific error codes

4. Socket creation and binding issues

Key indicators:

SYSCALL    PARAMETERS                    RET
socket     family=AF_INET, type=SOCK     3
connect    fd=3, addr=10.96.0.1:443     -110  # ETIMEDOUT
close      fd=3                         0

Scenario 4: Configuration and Secret Issues

The problem: Application can't access mounted secrets or config maps.

What system calls reveal:

1. File access patterns for mounted volumes

2. Permission checks on secret files

3. Configuration file parsing attempts

Common patterns:

1. Multiple openat attempts on different config file paths

2. access calls checking file permissions before opening

3. Failed reads from mounted secret volumes

Scenario 5: Performance Bottlenecks

The problem: Application response times are slow without obvious cause.

Traceloop analysis:

1. Excessive fsync calls (disk I/O bottlenecks)

2. Many futex calls (lock contention)

3. Frequent recvfrom timeouts (network issues)

4. Repeated file system operations

Performance indicators:

SYSCALL     FREQUENCY    ISSUE
fsync       High         Disk I/O bottleneck
futex       Excessive    Lock contention
poll        Many         Waiting for I/O
recvfrom    Timeouts     Network delays

Best Practices

When to Use Traceloop

Traceloop is most useful when you’re dealing with the kinds of problems that are notoriously difficult to pin down. If you’ve ever struggled with debugging intermittent crashes that don’t happen on demand, or run into confusing permission and access issues, this is where it works best.

It also helps uncover performance bottlenecks at the system level and provides visibility into application behavior during tricky startup failures. Another common use case is diagnosing network connectivity problems between pods, where other tools usually can't help

Of course, not every problem requires system call tracing. For application-level issues, logs and APM tools are more effective. Cluster-level concerns are often better handled with kubectl describe or by looking at events, and if you’re primarily monitoring resources, standard metrics and dashboards show you what's happening.

Performance Considerations

Like any tracing tool, Traceloop adds some overhead, but it keeps the overhead low. You can keep it efficient by narrowing the scope of your traces. For example, filtering by namespace with --namespace specific-ns, or targeting specific pods using --podname target-pod. In high-traffic environments, it’s best to run traces for shorter periods, and node-specific tracing can further isolate debugging when you don’t want to instrument the entire cluster.

In most cases, Traceloop uses very little CPU and memory, thanks to its eBPF-based approach. This makes it lighter than traditional tools like strace. The actual cost depends on the volume of system calls being recorded, so it’s a good practice to monitor resource usage in your own environment to confirm it’s operating within acceptable limits.

Integration with Your Workflow

Traceloop works well in dev and production workflows. In development, it’s a powerful way to understand how your application interacts with the system. You can use it to confirm that your app handles edge cases correctly, or to validate permission and resource configurations before promoting workloads into production.

In production environments, you can deploy it in different ways. Depending on how much overhead you're okay with, some teams run it continuously on a small subset of nodes, while others use it only when traditional debugging methods don’t provide enough insight. Pairing Traceloop with your existing monitoring and logging stack can give you a much more complete picture of system behavior.

It also helps with teamwork. Sharing trace outputs makes it easier for teams to reason about complex issues together. The data it provides can guide improvements in error handling and logging, and documenting common system call patterns can help onboard new developers more quickly.

Security Considerations

Because Traceloop records low-level system activity, you need to be mindful of what it captures.

What Traceloop Can See:

• System call parameters (such as filenames and network addresses)

• Process information and command arguments

• File access patterns and permissions

Privacy Measures:

• Limit trace duration to minimize data collection

• Use namespace isolation to avoid capturing unrelated workloads

• Apply data retention policies for trace outputs

• Watch for sensitive information in file paths or system call parameters

Conclusion

Traceloop doesn’t just tell you something went wrong – it shows you how. By recording every system call in real time, it turns mysterious Kubernetes failures into solvable problems. Whether the issue happened seconds ago or in the middle of the night, the tool gives you the ability to rewind, inspect, and respond with confidence.

When to Use It

Keep in mind that Traceloop complements your existing debugging toolkit rather than replacing it. Reach for it when logs don’t tell the whole story, when intermittent problems are hiding in the shadows, when kubectl commands leave you guessing, or when you need to see how your application is really interacting with the system.

Once you’re comfortable with Traceloop, you can add more tools. Inspektor Gadget offers other tools for network, security, and performance debugging that pair well with Traceloop. Integrating it into your incident response workflow, sharing insights across your team, and even considering continuous tracing for critical workloads are good things to try next.

The next time you run into a stubborn Kubernetes pod failure, you won’t be stuck speculating. With Traceloop, you can “rewind the tape” and see exactly what happened. System call tracing may sound complex at first, but in practice, it’s one of the most powerful ways to truly understand how applications behave in containerized environments.

PS: Have any questions about Traceloop or want to share your debugging challenges? The Inspektor Gadget team and community hang out in the #inspektor-gadget channel on Kubernetes Slack. It's a great place to get help from the engineers who built these tools, share experiences, and maybe even contribute to making the ecosystem even better.

You can also connect with me on LinkedIn if you’d like to stay in touch. If you made it to the end of this tutorial, thanks for reading!

Observability tools, such as logs, metrics, and traces, provide the visibility you need to pinpoint issues quickly. In this handbook, we’ll explore free and open-source tools you can use to make your CI/CD pipelines more reliable. We’ll use practical steps to troubleshoot like a pro – no enterprise licenses required.

Table of Contents

1. Prerequisites

2. Why Observability is Important

3. How to Install and Configure Grafana Loki on Budget Infrastructure

4. How to Implement an ELK Stack Alternative for Pipeline Observability

5. How to Create a Unified Logging Strategy Across Pipeline Components

6. How to Query and Analyze Logs for Effective Troubleshooting

7. How to Set Up Prometheus Metrics Alongside Your Logs

8. How to Create Grafana Dashboards That Combine Metrics and Logs

9. How to Use Exemplars to Jump from Metrics to Relevant Logs

10. How to Diagnose and Fix Common CI/CD Problems

11. How to Implement Advanced Debugging Techniques

12. How to Conduct Effective Postmortems Using Logs

13. How to Optimize Log Storage and Management

14. Conclusion

Prerequisites

There are some things you should know and have to get the most out of this handbook:

Technical Knowledge:

• Basic understanding of CI/CD pipelines (for example, build, test, deploy stages).

• Familiarity with Linux/Unix commands (for example, mkdir, grep, curl).

• Comfortable with Docker basics (for example, docker run, docker-compose up).

• Optional: Awareness of observability concepts (logs, metrics, traces) or YAML configuration.

Software and Tools:

• Docker and Docker Compose: Installed and running (verify with docker --version and docker-compose --version).

• CI/CD Platform: Access to GitHub Actions, Jenkins, or GitLab CI with a sample pipeline that generates logs.

• Text Editor: For editing YAML files (for example, VS Code, Nano).

• Web Browser: To access tool UIs (for example, Grafana on port 3000, Kibana on 5601).

• Optional: curl for testing log forwarding, Git for version control.

Hardware and Infrastructure:

• Machine with:

  • OS: Linux, Windows (with WSL2), or macOS.

  • 4GB RAM (8GB recommended), 20GB free disk space.

  • Stable internet and ability to open ports (for example, 3100 for Loki, 9200 for Elasticsearch).

• Optional: Cloud provider access (for example, AWS, GCP) for scalable setups.

Access and Permissions:

• Admin access to install Docker and configure CI/CD tools.

• Permissions to modify pipeline configs (for example, .github/workflows, .gitlab-ci.yml).

• Optional: Container registry access (for example, Docker Hub) for custom images.

Why Observability is Important

Modern CI/CD pipelines are no longer linear scripts – they are now complex, distributed systems involving multiple tools, environments, and infrastructure layers. One job runs on GitHub Actions, another deploys via Jenkins, and a third builds Docker images in a Kubernetes cluster.

So when something breaks, you’re left chasing logs across tools, guessing where the issue originated, and wasting hours trying to reproduce it.

And worse still, traditional debugging tools often stop at the surface, only showing failed jobs without the context of why they failed or where in the system the fault actually lies.

Observability flips the script. Instead of hunting through disconnected logs or rerunning failed builds blindly, observability gives you insight, not just data. By combining structured logs, metrics, and traces, you can:

• Reconstruct exactly what happened in a pipeline failure

• Trace a failure across CI agents, deployment steps, and containers

• Visualize patterns and anomalies before they become outages

More importantly, observability helps you move from reactive debugging to proactive prevention.

Here’s what you’ll learn about and accomplish in this guide:

• Set up cost-effective observability using Grafana Loki, lightweight ELK, and OpenTelemetry

• Create a unified logging strategy to connect your pipeline

• Write precise queries to quickly pinpoint root causes, correlate logs, metrics, and traces for comprehensive debugging

• Troubleshoot CI/CD issues like build failures, flaky tests, and container crashes

• Build custom dashboards and automated diagnostic tools

• Promote observability through documentation and post-mortems

Whether you're a solo developer or part of a DevOps team, this guide will transform your chaotic CI/CD pipelines into clear, reliable, and observable systems.

How to Choose the Right Observability Tool for CI/CD

Here’s a quick comparison of Grafana Loki, Lightweight ELK, and Vector for CI/CD observability:

How to choose:

• Loki: Ideal for startups or solo devs with limited resources. Integrates well with Prometheus/Grafana.

• ELK: Best for teams needing Kibana’s advanced visualizations or handling large log volumes.

• Vector: Great for lightweight log forwarding in distributed CI/CD setups.

Grafana Loki is a log aggregation system like ELK, but it's more lightweight, and it’s ideal for CI/CD pipelines with limited infrastructure.

How to Install and Configure Grafana Loki on Budget Infrastructure

🛠 Option A: Quick Docker Setup (Recommended for Budget Infra)

1. Create a directory for configuration:

    mkdir -p ~/loki-setup && cd ~/loki-setup
   

2. Create a docker-compose.yml:

    # Defines a Docker Compose setup for Grafana Loki and Promtail to aggregate and scrape logs efficiently.
    version: "3"
   
    services:
      loki:
        image: grafana/loki:2.9.4  # Uses Loki version 2.9.4 for lightweight log aggregation.
        ports:
          - "3100:3100"  # Exposes Loki’s HTTP API port for log ingestion and queries.
        command: -config.file=/etc/loki/loki-config.yaml  # Specifies the configuration file for Loki.
        volumes:
          - ./loki-config.yaml:/etc/loki/loki-config.yaml  # Mounts the local config file into the container.
   
      promtail:
        image: grafana/promtail:2.9.4  # Uses Promtail version 2.9.4 to scrape and forward logs to Loki.
        volumes:
          - /var/log:/var/log  # Mounts the host’s log directory for Promtail to scrape.
          - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml  # Mounts the Promtail config file.
        command: -config.file=/etc/promtail/promtail-config.yaml  # Specifies the configuration file for Promtail.
   

3. Create a basic loki-config.yaml:

    # Configures Grafana Loki for lightweight log storage and querying in a CI/CD environment.
    auth_enabled: false  # Disables authentication for simplicity (not recommended for production).
   
    server:
      http_listen_port: 3100  # Sets the port for Loki’s HTTP API.
   
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: inmemory  # Uses in-memory storage for the ring, suitable for small setups.
          replication_factor: 1  # Sets single replica for minimal resource use.
      chunk_idle_period: 3m  # Flushes chunks to storage after 3 minutes of inactivity.
      max_chunk_age: 1h  # Retires chunks after 1 hour to balance storage and query performance.
   
    schema_config:
      configs:
        - from: 2023-01-01  # Defines the schema start date.
          store: boltdb-shipper  # Uses BoltDB for indexing logs.
          object_store: filesystem  # Stores logs on the local filesystem.
          schema: v11  # Specifies schema version for log storage.
          index:
            prefix: index_  # Prefix for index files.
            period: 24h  # Rotates indexes daily.
   
    storage_config:
      boltdb_shipper:
        active_index_directory: /tmp/loki/index  # Directory for active index files.
        cache_location: /tmp/loki/boltdb-cache  # Cache location for BoltDB.
      filesystem:
        directory: /tmp/loki/chunks  # Directory for storing log chunks.
   
    limits_config:
      enforce_metric_name: false  # Disables strict metric name enforcement for flexibility.
   

4. Create a basic promtail-config.yaml:

    # Configures Promtail to scrape system logs and forward them to Loki.
    server:
      http_listen_port: 9080  # Sets Promtail’s HTTP port for metrics and health checks.
      grpc_listen_port: 0  # Disables gRPC to reduce resource usage.
   
    positions:
      filename: /tmp/positions.yaml  # Stores the position of scraped logs to resume after restarts.
   
    clients:
      - url: http://loki:3100/loki/api/v1/push  # Specifies the Loki endpoint for log ingestion.
   
    scrape_configs:
      - job_name: system  # Defines a scraping job for system logs.
        static_configs:
          - targets:
              - localhost  # Targets the local host for log collection.
            labels:
              job: varlogs  # Labels logs for easy querying in Loki.
              __path__: /var/log/*.log  # Scrapes all log files in /var/log directory.
   

5. Run it:

    # Starts the Loki and Promtail containers in detached mode for background operation.
    docker-compose up -d
   

✨ This brings up Loki and Promtail with minimal resources, no authentication, and logs scraping from /var/log.

Troubleshooting Loki Setup Issues

If Loki or Promtail fails to start, one of the following may be the issue:

1. Container crashes: Check logs with docker logs loki or docker logs promtail. Look for errors like “out of memory” or “port already in use.”

   • Fix: Increase memory (for example, docker-compose.yml resource limits) or change ports (e.g., 3101:3100).

2. Logs not ingested: Verify Promtail is scraping the correct path (/var/log/ci/*.log) using docker exec promtail cat /etc/promtail/promtail-config.yaml

   • Fix: Update __path__ in promtail-config.yaml to match your CI/CD log directory.

3. Resource Constraints: Monitor resource usage with docker stats or top on the host.

   • Fix: Ensure your machine has at least 4GB RAM and 20GB disk space, as specified in the prerequisites.

Configuration for CI/CD Logging

To adapt for CI/CD logs, you should:

1. Configure your CI/CD tools to write logs to disk:

For example, GitHub Actions with a custom runner can write logs to /var/log/gha/*.log.

Update Promtail:

# Configures Promtail to scrape logs from GitHub Actions runners for CI/CD observability.
scrape_configs:
  - job_name: github_actions  # Defines a scraping job for GitHub Actions logs.
    static_configs:
      - targets: ['localhost']  # Targets the local host where the runner writes logs.
        labels:
          job: gha  # Labels logs for identification in Loki queries.
          __path__: /var/log/gha/*.log  # Scrapes logs from the specified directory.

2. Use structured logging (JSON):

Make sure your CI/CD tools or scripts output logs in structured format:

Example:

# Example of a structured JSON log for CI/CD pipelines, enabling easy parsing and querying.
{
  "timestamp": "2025-05-10T13:00:00Z",  # UTC timestamp for log entry.
  "level": "error",  # Log level to indicate severity.
  "job": "deploy",  # Identifies the CI/CD job (e.g., deploy stage).
  "message": "Image pull failed"  # Descriptive message for the error.
}

This helps when querying with LogQL.

How to Connect CI Agents to Loki

This section explains three different ways to get your CI pipeline logs into Loki for monitoring and analysis:

Option 1 – Local setup:

Your CI agents write log files to disk, and Promtail (running on the same machine) reads those files and sends them to Loki.

Option 2 – Using Docker logging driver (Docker containers):

If your CI agents run in Docker containers, you install a special Loki plugin that automatically captures all container output and sends it directly to Loki without needing separate log files.

# Installs the Loki Docker logging driver to send container logs directly to Loki.
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Then run your agent container:

# Runs a CI agent container with the Loki logging driver to forward logs.
docker run --log-driver=loki \
  --log-opt loki-url="http://<your-loki-host>:3100/loki/api/v1/push" \
  my-ci-agent-image

Option 3 – Remote setup:

If you can't install Promtail locally, you can use a log forwarding tool like Fluent Bit or Vector to collect logs and push them to Loki over the network.

The goal: Regardless of which option you choose, you’ll end up with all your CI pipeline logs centralized in Loki, where you can search through them, create dashboards in Grafana, and set up alerts when things go wrong.

It essentially gives you flexibility to integrate log collection based on your infrastructure setup – whether you prefer local agents, Docker plugins, or remote forwarding.

How to Implement an ELK Stack Alternative for Pipeline Observability

When full ELK (Elasticsearch, Logstash, Kibana) is too heavy for your infrastructure, you can go with lightweight setups that achieve similar observability at a lower cost and resource usage.

How to Install Lightweight Versions of Elasticsearch, Logstash, and Kibana

Goal: Stand up a minimal yet functional ELK stack for debugging CI/CD pipelines.

1. Use Docker to spin up lightweight containers

Create a docker-compose.yml:

# Defines a Docker Compose setup for a lightweight ELK stack to aggregate and visualize CI/CD logs.
version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0  # Uses Elasticsearch 7.17.0.
    container_name: elasticsearch
    environment:
      - discovery.type=single-node  # Runs Elasticsearch in single-node mode for simplicity.
      - xpack.security.enabled=false  # Disables security features for lightweight setup.
    ports:
      - "9200:9200"  # Exposes Elasticsearch’s HTTP API port.
    volumes:
      - esdata:/usr/share/elasticsearch/data  # Persists Elasticsearch data.

  logstash:
    image: docker.elastic.co/logstash/logstash:7.17.0  # Uses Logstash 7.17.0.
    container_name: logstash
    ports:
      - "5044:5044"  # Port for receiving logs from Beats.
      - "9600:9600"  # Port for Logstash monitoring.
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf  # Mounts Logstash config file.

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.0  # Uses Kibana 7.17.0 for visualization.
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200  # Links Kibana to Elasticsearch.
    ports:
      - "5601:5601"  # Exposes Kibana’s web UI port.

volumes:
  esdata:  # Defines a volume for persisting Elasticsearch data.

2. Minimal Logstash pipeline configuration (logstash.conf)

// Configures Logstash to process and forward CI/CD logs to Elasticsearch.
input {
  beats {
    port => 5044  // Listens for logs from Filebeat on port 5044.
  }
}

filter {
  json {
    source => "message"  // Parses JSON-formatted log messages for structured data.
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]  // Sends processed logs to Elasticsearch.
    index => "ci-logs-%{+YYYY.MM.dd}"  // Stores logs in daily indexes (e.g., ci-logs-2025.05.14).
  }
}

Troubleshooting ELK Setup Issues

If Elasticsearch, Logstash, or Kibana fails to start, one of the following might be the issue:

1. Container crashes: Check logs with docker logs elasticsearch, docker logs logstash, or docker logs kibana. Look for errors like “insufficient disk space” or “port conflict” (for example, 9200, 5601).

   • Fix: Free up disk space (ensure at least 20GB available) or change ports in docker-compose.yml (for example, 9201:9200).

2. Logs not ingested: Verify Logstash is receiving data from Filebeat or Vector using docker logs logstash. Check the logstash.conf input port (for example, 5044).

   • Fix: Ensure Filebeat or Vector is configured to send to the correct Logstash endpoint (e.g., localhost:5044) and update if needed.

3. Resource constraints: Monitor resource usage with Docker stats or top on the host.

   • Fix: Allocate at least 8GB RAM and 30GB disk space, as Elasticsearch requires more resources than Loki. Adjust memory limits in docker-compose.yml if necessary.

How to Configure Log Shippers for Different CI/CD Components

Goal: Get logs from your pipeline into Logstash or Elasticsearch.

Option 1: Use Filebeat (lightweight log shipper)

Install Filebeat on your CI/CD hosts (GitHub runner, Jenkins node, GitLab runner, and so on).

Filebeat config snippet (filebeat.yml):

# Configures Filebeat to collect CI/CD logs and forward them to Logstash.
filebeat.inputs:
  - type: log  # Specifies log file input.
    enabled: true  # Enables the input.
    paths:
      - /var/log/ci/*.log  # Scrapes logs from the specified CI log directory.

output.logstash:
  hosts: ["localhost:5044"]  # Forwards logs to Logstash on port 5044.

Then run:

# Runs Filebeat with the specified configuration file for log collection.
filebeat -e -c filebeat.yml

Option 2: Use Vector.dev as a more resource-efficient alternative to Filebeat

Vector configuration (vector.toml):

# Configures Vector to collect, parse, and forward CI/CD logs to Elasticsearch efficiently.
[sources.ci_logs]
  type = "file"  # Specifies file-based log collection.
  include = ["/var/log/ci/*.log"]  # Targets CI log files.

[transforms.json_parser]
  type = "remap"  # Uses remap transform to parse logs.
  inputs = ["ci_logs"]  # Processes logs from the ci_logs source.
  source = '''
  . = parse_json!(.message)  # Parses JSON log messages into structured data.
  '''

[sinks.to_elasticsearch]
  type = "elasticsearch"  # Sends logs to Elasticsearch.
  inputs = ["json_parser"]  # Uses parsed logs from the json_parser transform.
  endpoint = "http://localhost:9200"  # Specifies the Elasticsearch endpoint.
  index = "ci-logs"  # Stores logs in the ci-logs index.

Run:

# Runs Vector with the specified configuration file for log processing.
vector -c vector.toml

How to Set Up Index Patterns and Basic Visualizations

Goal: Make CI/CD logs queryable and visual in Kibana.

1. Open Kibana (http://localhost:5601)

• Go to Stack Management → Index Patterns

• Create a new pattern: ci-logs-*

• Choose a time field like @timestamp

2. Visualizations for Common CI/CD Use Cases

• Bar charts: Number of failed vs passed builds per day

• Pie chart: Top error types or most frequent failing test names

• Line chart: Duration of builds over time (if duration is logged)

3. Saved Searches & Dashboards

You can save a search like this:

message: "error" AND job_name: "build"

You can also combine visualizations into a CI/CD Health Dashboard.

How to Create a Unified Logging Strategy Across Pipeline Components

Creating a unified logging strategy across your CI/CD pipeline components ensures that logs are consistent, traceable, and easy to correlate. This helps you quickly debug issues, monitor system health, and trace requests across different tools and services. Let’s discuss some key practices for achieving a unified logging strategy:

Implementing Consistent Log Formats Across Different Tools

Consistent log formats are important for various reasons. First of all, a standardized log format enables easier querying, searching, and visualization. It also helps with correlation of logs from different services. And consistency also ensures that all logs provide necessary details like timestamp, log level, and request context.

There are also some best practices you should follow when formatting logs:

JSON Format is highly recommended as it’s structured, machine-readable, and compatible with many observability tools (for example, Loki, Elasticsearch, Grafana).

There are also some key fields you should include:

• timestamp: The time the log entry was created (preferably in UTC).

• log_level: Indicate whether the log is an INFO, ERROR, DEBUG, and so on.

• service: The service or component generating the log.

• message: A concise description of the event or error.

• correlation_id: A unique identifier for requests to trace logs across systems.

Here’s an example of a consistent log in JSON format:

{
  "timestamp": "2025-05-10T12:34:56Z",
  "log_level": "ERROR",
  "service": "ci_cd_pipeline",
  "message": "Build failed due to missing dependency",
  "correlation_id": "1234567890abcdef"
}

How to Set Up Log Forwarding from GitHub Actions, Jenkins, or GitLab

Log forwarding refers to shipping logs from your CI/CD pipelines to a central spot for easy tracking. It’s helpful because it lets you spot issues fast and debug without digging through scattered files.

For GitHub Actions, you can configure workflows to write logs to a file or send them directly to a log aggregation tool like Loki. In Jenkins, you can use pipeline scripts to forward logs to a log server or file system. Similarly, for GitHub CI, you can add scripts in .gitlab-ci.yml to forward logs to a centralized endpoint.

Using Actions for Outputting Logs:
You can store logs in files and then forward them to a logging system (like Loki or Elasticsearch).
Here’s an example in a GitHub Action workflow:

# Defines a GitHub Actions workflow to run tests and forward logs for observability.
jobs:
  build:
    runs-on: ubuntu-latest  # Uses an Ubuntu runner.
    steps:
      - name: Checkout repository  # Checks out the repository code.
        uses: actions/checkout@v2
      - name: Run tests and log output  # Runs tests and saves output to a log file.
        run: |
          echo "Starting tests..."
          npm test | tee test.log  # Captures test output to test.log.
          # Forwards the log file to a Loki endpoint via HTTP POST.
          curl -X POST -F 'file=@test.log' http://your-loki-endpoint

Log Forwarding with Promtail:
If you are using Grafana Loki for log aggregation, set up Promtail to scrape the logs from the GitHub Actions runner.

Jenkins:

Jenkins logs can be forwarded to external systems (like Elasticsearch or Loki) by using log shippers or plugins.

You can use the Logstash Plugin to forward Jenkins logs to an ELK stack or other systems:

• Install the Logstash plugin on Jenkins.

• Configure the plugin to forward logs to an Elasticsearch server or a logging system of choice.

• In Jenkins, add log forwarding configurations:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        script {
          // Example of forwarding logs to a log server
          sh 'echo "Build successful" | curl -X POST -d @- http://your-log-server'
        }
      }
    }
  }
}

Forward to Loki:
Jenkins supports the loki logging driver for containers if running Jenkins in Docker. You can send logs directly to Loki using this driver:

# Runs a Jenkins container with the Loki logging driver to send logs directly to Loki.
docker run --log-driver=loki --log-opt loki-url=http://loki:3100 jenkins/jenkins:lts

GitLab:

GitLab CI allows logs to be forwarded to external systems for centralized collection and analysis.

Use GitLab CI/CD to Output Logs:
Example in .gitlab-ci.yml:

# Defines a GitLab CI/CD pipeline to run a build and forward logs to Loki.
stages:
  - build
build:
  script:
    - echo "Starting the build" | tee build.log  # Saves build output to build.log.
    - curl -X POST -d @build.log http://your-loki-endpoint  # Forwards the log to Loki.

GitLab Runners:
Configure GitLab runners to forward logs to an external service like Loki or Elasticsearch using log-driver settings or the fluentd log shipper.

How to Add Correlation IDs to Trace Requests Through the System

Why Correlation IDs Are Important:

Correlation IDs allow you to trace a single request as it travels through different services and tools, enabling end-to-end visibility and troubleshooting.

They are critical for debugging distributed systems, especially when different services (for example, CI tool, deployment tool, API service) are involved.

How to Add Correlation IDs:

You can use a UUID (Universally Unique Identifier) or a GUID (Globally Unique Identifier) to generate a unique ID for each request.

If you are using microservices or multiple services in the pipeline, just make sure that the same ID is propagated across each service.

Many logging libraries (for example, winston for Node.js, log4j for Java) support automatic correlation ID generation and logging.

Here’s an example in Node.js (using winston):

// Sets up Winston for structured logging with correlation IDs in a CI/CD pipeline.
const { createLogger, transports, format } = require('winston');
const { printf } = format;

// Creates a logger with a custom format including correlation IDs.
const logger = createLogger({
  format: printf(({ level, message, timestamp }) => {
    return `${timestamp} [${level}] ${message} correlation_id=${generateCorrelationId()}`;
  }),
  transports: [
    new transports.Console(),  // Outputs logs to the console.
  ],
});

// Generates a random correlation ID for tracing requests.
function generateCorrelationId() {
  return Math.random().toString(36).substring(2, 15);
}

// Logs a sample message.
logger.info('Pipeline execution started');

How to Propagate Correlation IDs Between Services:

In CI/CD tools, you can configure your pipeline to inject the correlation ID into logs. For example, in GitHub Actions, you can generate a correlation ID in the env section and propagate it in each job:

# Defines a GitHub Actions workflow that includes a correlation ID for log tracing.
jobs:
  build:
    runs-on: ubuntu-latest  # Uses an Ubuntu runner.
    env:
      CORRELATION_ID: ${{ github.run_id }}  # Uses the GitHub run ID as a correlation ID.
    steps:
      - name: Checkout repository  # Checks out the repository code.
        uses: actions/checkout@v2
      - name: Log build start with correlation ID  # Logs the build start with the correlation ID.
        run: echo "Build started with Correlation ID: $CORRELATION_ID"

Include Correlation IDs in All Logs:

You’ll want to make sure that logs from all components in the pipeline (GitHub Actions, Jenkins, GitLab, deployment tools, and so on) include the correlation ID as part of the log message. This allows you to trace the logs of a single request or pipeline run across different services.

Visualize Your Log Flow

You can create a diagram showing how logs move from your CI/CD tool (for example, GitHub Actions) to Promtail/Vector, then to Loki/Elasticsearch, and finally to Grafana/Kibana for visualization. Use tools like Draw.io to map your pipeline’s observability flow

How to Query and Analyze Logs for Effective Troubleshooting

In this section, you’ll learn how to use LogQL (Loki's query language) to cut through the noise and find the specific logs that matter. Whether you're hunting down a mysterious build failure or tracking deployment issues across multiple services, these query patterns always help.

This bar chart illustrates the CI/CD build performance from May 20 to May 26, 2025. It compares the number of successful builds (in blue) to failed builds (in pink) each day. Successful builds consistently range between 40 and 50, while failed builds peak at 10 on May 23, with other days showing 2 to 8 failures. This indicates a generally stable pipeline with occasional issues.

How to Write Advanced LogQL Queries to Pinpoint CI/CD Issues

LogQL is Grafana Loki's query language, designed for querying logs with a syntax similar to Prometheus’s PromQL. It enables efficient log searches and is particularly useful in troubleshooting CI/CD issues.

Basic LogQL Syntax:

1. Log Streams:

{job="ci_cd", level="error"}

This query retrieves logs where the job label is ci_cd and the level label is error.

2. Log Filters:

{job="ci_cd"} |= "build failed"

The |= operator filters logs to include only those that contain the specified string, for example "build failed".

3. Regular Expressions:

{job="ci_cd"} |~ "error.*timeout"

This uses the |~ operator to filter logs using a regular expression. In this case, it finds logs that contain an "error" followed by "timeout".

Advanced LogQL Queries for CI/CD Issues:

1. Filter Logs for Specific Build Failures:

If your pipeline uses a specific label for build names:

{job="ci_cd", build="build123"} |= "failure"

This finds logs related to the build123 job that contain the word "failure".

2. Using Time Range and Grouping:

To find error logs in the last 15 minutes:

{job="ci_cd", level="error"} | "build failed" | range(start="15m")

To group logs by job and error type:

sum by (job) (count_over_time({job="ci_cd", level="error"}[5m]))

This will return the count of error logs per job, grouped by job name, over the last 5 minutes.

How to Create Pipeline-Specific Queries for Common Failure Patterns

Common Failure Patterns in CI/CD Pipelines:

1. Build Failures:

If your CI system logs contain build errors, you can identify them with:

{job="ci_cd", level="error"} |= "build failed"

You can extend this to filter by specific steps or stages, for example, “test failed”, or “compilation error”.

2. Test Failures:

Logs from your test runner (for example, Jest, Mocha, JUnit) can contain specific failure messages:

{job="ci_cd", stage="test"} |= "test failed"

3. Dependency Issues:

If your pipeline is failing due to missing or conflicting dependencies, look for npm, maven, or docker related errors:

{job="ci_cd", image="node"} |= "npm ERR!"

Or for Maven-related issues:

{job="ci_cd", image="maven"} |= "[ERROR]"

4. Resource Constraints (for example, Out of Memory):

If you experience resource constraints, you might see logs like "OutOfMemoryError":

{job="ci_cd", level="error"} |= "OutOfMemoryError"

Example of combining filters:

{job="ci_cd", level="error"} |= "build failed" |~ "timeout|dependency" | range(start="1h")

This combines log filters for "build failed", matching any logs with the terms "timeout" or "dependency", from the last hour.

How to Set Up Alert Rules Based on Log Patterns

Alerts help detect recurring issues proactively. They notify you when a specific pattern appears in your logs, allowing you to take quick action.

Steps for Setting Up Alerts:

1. Create a Query for the Alert:

First, define the log pattern you want to monitor. For example, an alert for build failures:

{job="ci_cd", level="error"} |= "build failed"

2. Create an Alert in Grafana:

Follow these steps to set up Grafana alerts:

• Go to your Grafana dashboard.

• Choose the panel you want to set the alert on (or create a new panel for this purpose).

• In the panel, click the Alert tab.

• Set the Query field to your LogQL query, such as the one above.

• Under Conditions, define when the alert should trigger, e.g., if the error occurs more than 3 times within 5 minutes.

3. Alert Settings:

Now you’ll want to set up the alert evaluation interval and conditions for triggering the alert (e.g., if the query returns results above a certain threshold).

Here’s an example: Trigger an alert if the number of errors exceeds 5 within 5 minutes:

count_over_time({job="ci_cd", level="error"} |= "build failed"[5m]) > 5

4. Set Alert Notifications:

You can choose where you want the alert to be sent (like to Slack, email, or PagerDuty). And Grafana can be integrated with these systems to send real-time alerts to the right team members.

Example alert query for test failures:

count_over_time({job="ci_cd", stage="test"} |= "test failed"[5m]) > 3

This query triggers an alert if more than 3 test failures are logged within the last 5 minutes.

Kibana Query Language Deep Dive for CI/CD Contexts

Kibana Query Language (KQL) is a powerful tool for searching and filtering logs within Elasticsearch, and it becomes especially useful for debugging CI/CD pipelines.

Basic Query Syntax:

• Field:

    textCopyEditfieldname:value
  

  Example: status: "failure"

• Wildcard: Use * to match any number of characters:

    textCopyEditmessage: "test*"
  

• Range Queries: To search for logs within a specific time frame:

    textCopyEdittimestamp:[2023-05-01 TO 2023-05-15]
  

• Boolean Queries: Combine queries using AND, OR, and NOT:

    textCopyEditstatus: "failure" AND build_id: "12345"
  

Time-Based Queries:

Since CI/CD logs are often tied to time-sensitive operations (builds, deployments), KQL allows you to filter logs by time:

textCopyEdit@timestamp:[now-1d TO now]

Nested Queries (For Complex Pipelines):

CI/CD logs can have nested or multi-level structures (for example, logs within containers). You can query these nested fields:

textCopyEditpipeline.logs.message: "build failed"

Aggregations and Grouping:

You can aggregate logs based on certain fields to identify trends or recurring issues:

textCopyEditterms aggregation on "status" field

This helps identify the most common failure statuses in your pipeline.

Field-Specific Filtering:

When debugging specific components like a build tool or deployment step, you can filter by those component-specific fields:

textCopyEditbuild_tool: "Jenkins" AND status: "failure"

Creating Saved Searches for Recurring Issues

Once you’ve built queries that help you identify common issues in your CI/CD pipeline, you can save them in Kibana for future use.

1. Create a Saved Search:

Run your desired query in the Kibana Discover tab. Click on the “Save” button and give it a meaningful name, such as "Failed Builds - Last Week". You can add filters and customize the time range to match your typical issue patterns.

2. Use Filters to Pinpoint Recurring Problems:

Create saved searches that focus on specific recurring issues like:

• Build failures based on a specific tool or version.

• Test failures within a particular module or set of tests.

Example search for “flaky tests”:

textCopyEdittest_status: "failed" AND error_message: "*timeout*"

3. Saving Multiple Variations:

You can save multiple variations of queries based on different error types or CI/CD tools:

• Failed Jobs: status: "failure"

• Test Failures in Build: log_type: "test" AND status: "failure"

• Resource Constraints: error_message: "*memory*"

These saved searches will allow you to quickly troubleshoot specific issues that occur frequently.

Building Visualizations to Spot Patterns Over Time

Once you have saved searches, Kibana allows you to create visualizations from your data, making it easier to spot trends, anomalies, or patterns over time.

1. Create a Visualization:

Go to the Visualize tab in Kibana. Select the appropriate visualization type. Common visualizations for debugging CI/CD pipelines include:

• Line Chart: Track build failure rates over time.

• Bar Chart: Show the number of failures per CI tool or service.

• Pie Chart: Breakdown of failure reasons (for example, compilation errors, test failures, resource constraints).

2. Track Failure Trends Over Time:

Create a line chart to track build failures over a given period:

• X-Axis: Time (for example, daily or weekly).

• Y-Axis: Count of build failures.

• Aggregation: Date histogram with @timestamp field.

This will help you visualize how build failures are trending, making it easier to identify recurring issues or spikes in failures.

3. Monitor Failure Types by CI Tool:

Create a bar chart that shows the number of failures broken down by CI tool:

• X-Axis: CI tool (Jenkins, GitHub Actions, GitLab, and so on).

• Y-Axis: Count of failures.

• Aggregation: Terms aggregation on the ci_tool field.

This visualization helps identify which CI tool is experiencing the most failures and focus troubleshooting efforts there.

4. Visualize Error Messages by Frequency:

You can visualize which error messages appear most frequently, helping you understand what might be causing recurring issues:

• X-Axis: Error message type.

• Y-Axis: Count of occurrences.

• Aggregation: Terms aggregation on the error_message field.

5. Dashboard for Holistic Monitoring:

Create a dashboard that brings together multiple visualizations. You can have one graph for failure trends, another for failure types (bar chart), and a pie chart showing the percentage of failures caused by different issues. This dashboard gives you a holistic view of your pipeline's health.

Advanced Visualization Techniques:

There are various advanced techniques you can use to dig further into your data.

• Heatmaps: Use heatmaps to spot time-based anomalies in build durations or test failures.

• Anomaly Detection: Kibana has built-in anomaly detection that can be applied to log data to automatically detect patterns that deviate from the norm. This is especially useful for catching rare or unexpected errors in your CI/CD pipeline.

  Example for anomaly detection:

    textCopyEditfield: duration
    aggregation: average
    anomaly detection model: "baseline"
  

How to Set Up Prometheus Metrics Alongside Your Logs

To fully understand your CI/CD pipeline's health and performance, combining metrics and logs is essential. Prometheus is an excellent tool for capturing time-series metrics, and it works seamlessly with Grafana and Loki (or any log aggregation system).

How to Set Up Prometheus for CI/CD Metrics Collection:

1. Install Prometheus:

You can install Prometheus using Docker or Kubernetes for easy deployment.

For Docker-based installation:

docker run -d -p 9090:9090 --name prometheus prom/prometheus

2. Configure Prometheus to Scrape Metrics:

Prometheus needs to be configured to scrape metrics from your CI/CD services.

Edit the prometheus.yml file:

scrape_configs:
  - job_name: 'ci_cd_metrics'
    static_configs:
      - targets: ['localhost:8080', 'localhost:9091']

3. Instrument Your CI/CD Services:

To expose metrics, you need to integrate Prometheus client libraries into your CI/CD services.

For example, to expose build metrics from a Jenkins job, use the Prometheus plugin for Jenkins. In GitHub Actions, you can use Prometheus to expose job metrics.

4. Expose Metrics Endpoint:

You’ll want to make sure your services expose a /metrics endpoint that Prometheus can scrape. For example, use Prometheus client libraries in your application to expose this endpoint.

Troubleshooting Prometheus Setup Issues

If Prometheus fails to start or scrape metrics, here are some things that might be going wrong:

1. Container Crashes: Check logs with docker logs prometheus. Look for errors like “port already in use” (for example, 9090) or configuration parsing issues.

   • Fix: Change the port in docker run (for example, -p 9091:9090) or correct the prometheus.yml file syntax.

2. Metrics Not Scraped: Verify targets are reachable using docker logs prometheus or test with curl http://localhost:9090/targets. Check prometheus.yml for correct endpoints.

   • Fix: Update targets in scrape_configs (for example, localhost:8080) to match your CI/CD service’s metrics endpoint.

3. Resource Constraints: Monitor usage with docker stats or top on the host.

   • Fix: Ensure at least 4GB RAM and 10GB disk space. Increase storage retention or reduce scrape frequency in prometheus.yml if needed.

How to Create Grafana Dashboards That Combine Metrics and Logs

Once Prometheus is collecting metrics, the next step is to visualize and correlate them in Grafana.

How to Integrate Prometheus with Grafana:

First, you’ll need to install Grafana. You can use Docker or Kubernetes for quick deployment:

docker run -d -p 3000:3000 --name grafana grafana/grafana

Next, configure Grafana to use Prometheus as a data source. To do this, log in to Grafana (localhost:3000 by default). Go to Configuration > Data Sources > Add Data Source > Choose Prometheus. Enter your Prometheus server URL (for example, http://localhost:9090) and click Save & Test.

Now it’s time to build a unified dashboard. To do this, create a new dashboard in Grafana that combines both logs (Loki) and metrics (Prometheus).

Add a panel with Prometheus data queries to visualize pipeline metrics like build success rate, deployment duration, and failure count. Use the Graph visualization type for time-series data and Stat for quick summary metrics.

Finally, in the same Grafana dashboard, add panels for logs (from Loki or any other logging system). Use the Logs panel to visualize log data and link them with the relevant Prometheus metrics by using time-based correlations.

Example: If a spike in CPU usage is detected (Prometheus metric), the logs panel could show related logs, like errors or failed build jobs.

How to Use Exemplars to Jump from Metrics to Relevant Logs

Exemplars are an advanced feature in Prometheus that allow you to connect metric data with logs and traces. Grafana supports this feature, and it can be incredibly helpful when investigating issues.

How to Set Up Exemplars in Prometheus:

1. Enable Exemplars in Your Application:

Exemplars are essentially traces embedded into your metrics. To use them, you’ll need to make sure your application is instrumented to send exemplar data alongside your metrics.

Many libraries support adding exemplars to Prometheus metrics, such as prom-client (Node.js) and prometheus-net (C#).

Here’s an example in Node.js:

// Demonstrates adding an exemplar to a Prometheus metric for linking to logs or traces.
const promClient = require('prom-client');

// Creates a counter metric to track failed CI/CD builds.
const counter = new promClient.Counter({
  name: 'ci_cd_failed_builds_total',  // Metric name for failed builds.
  help: 'Total number of failed builds',  // Description of the metric.
});

// Increments the counter with an exemplar for tracing.
counter.inc({ exemplar: 'build_failed' });

2. Enable Exemplars in Prometheus Config:

Make sure your Prometheus server is configured to store and expose exemplars. Exemplars are typically included with histogram or summary metrics, so make sure you’ve configured them correctly.

3. Visualizing Exemplars in Grafana:

In Grafana, when you query Prometheus for metrics with exemplars, Grafana will show the linked logs or traces when you hover over a metric.

Use the Exemplar option in Grafana panels to quickly access logs from specific metrics.

For example, if you have a build_failure_total metric and you detect a failure in your pipeline, you can click on the failure metric in Grafana and instantly view the relevant logs for that specific failure using the exemplars.

How to Diagnose and Fix Common CI/CD Problems

CI/CD pipelines often encounter issues like build failures, dependency problems, and flaky tests that can disrupt development workflows. This section provides practical strategies to diagnose and resolve these common problems using log analysis and systematic debugging techniques, helping you restore pipeline stability quickly.

Strategy 1: Systematically Debug Build Failures

Build failures are a frequent CI/CD challenge, often stemming from errors in code, tests, or configurations. Systematically debugging these issues involves analyzing logs to pinpoint root causes, using the following approaches.

Identifying Patterns in Compiler and Test Output

When debugging build failures, you need to first examine the logs from the compiler and test outputs. Let’s go over some key strategies.

1. Check for Specific Error Messages:

There are a few common types of error messages you might get. They are:

• Syntax errors: Look for lines indicating that there's a mismatch in syntax, such as missing semicolons, undeclared variables, or incorrect function calls.

• Linker errors: These often occur when the required libraries or dependencies are not found. You'll typically see errors like undefined reference or symbol not found.

• Build tool errors: If you are using build systems like Maven, Gradle, or MSBuild, their logs will give specific error codes or missing configurations.

2. Look for Common Error Patterns:

Often, failed builds repeat the same error or pattern across multiple runs. Check logs for recurring terms or errors that point to specific modules or functions. And remember that grouping similar issues can help you identify the root cause faster.

3. Use Regular Expressions for Log Filtering:

You can use regular expressions to search for keywords in the logs that match common failure patterns (for example, "error", "failed", "exception", "out of memory"). This will help you filter out unrelated messages and focus on the failures.

As an example:

• If the build fails with an "Out of Memory" error, search for any memory allocation issues or settings that can be increased.

• If test failures are related to specific modules, inspect those modules for recent changes or dependency issues.

Strategy 2: Troubleshooting Dependency Issues with Log Analysis

Dependency issues are common in build failures, especially in complex CI/CD pipelines with multiple modules or services. To resolve these issues, consider the following:

1. Check for Missing or Outdated Dependencies:

Start by reviewing the build tool’s output to check for messages related to missing dependencies (for example, dependency not found, version conflict).

Many build tools (like Maven, npm, or .NET) will include specific error messages when a dependency is missing or incompatible.

2. Inspect Dependency Resolution Logs:

Some build tools provide detailed logs showing how dependencies were resolved (for example, the version of a library that was used). These logs can show you if there’s a version mismatch.

Make sure that your package.json (for JavaScript projects), pom.xml (for Java), or csproj (for C#) files are correctly defined with compatible versions.

3. Verify Network Connectivity:

CI/CD tools sometimes fail to fetch dependencies due to network issues (for example, proxy settings, repository access). Look for any errors indicating that a repository couldn’t be reached.

4. Log Example:

If a Java project fails with Could not find artifact, it's likely a dependency missing or inaccessible. Check the repository URL or if the artifact exists in your Maven repo.

5. Resolve Version Conflicts:

Version conflicts occur when different dependencies require incompatible versions of the same library. This is especially true in Java (with Maven/Gradle) and .NET projects. Consider using tools to resolve version conflicts automatically or define compatible versions manually.

Fixing Flaky Tests Based on Historical Log Data

Note: Issues like container crashes, logs not ingested, or resource constraints here may resemble those in other sections. These are common across CI/CD services and processes, but each section offers unique context to avoid redundancy.

Flaky tests – that is, those that pass sometimes and fail at other times – are common in CI/CD pipelines, and they can be frustrating. Let’s discuss some strategies for how you can tackle them:

1. Analyze Test Logs Over Time:

Review historical logs to identify patterns in when the test fails. Look for timing issues, resource limits, or external dependencies that could affect test reliability.

For example, if a test intermittently fails after a certain amount of time or only during specific pipeline stages, it could indicate resource exhaustion or race conditions.

2. Check Test Dependencies:

Often, flaky tests are dependent on external services or resources (for example, databases, APIs, file systems). Check if these services are consistently available and properly mocked during test execution.

Logs that mention failed connections to external services or unstable environments can give you insights into potential issues with dependencies.

3. Run Tests with Increased Logging:

Increase the verbosity of test logs to capture more information about the failures. This can help you detect why tests fail in certain conditions.

For example, adding debug logs inside tests can provide more context on the state of the application when the failure occurs.

4. Time of Day Issues:

Some flaky tests may fail during peak usage times, especially if they rely on shared resources. Look for patterns that correlate with resource contention (for example, database locks, API rate limits).

Logs showing high CPU or memory usage can indicate that resource constraints are affecting the stability of your tests.

5. Implement Retry Logic for Flaky Tests:

To mitigate the effects of flaky tests, implement automatic retries for tests that fail intermittently. This can help reduce the noise in your CI/CD pipeline while you investigate the root causes.

For example, if a database connection test fails intermittently, you may want to inspect database logs for signs of timeouts or connection pool exhaustion.

How to Resolve Deployment Pipeline Failures

Deployment pipeline failures can stem from several sources, and diagnosing them requires a systematic approach using logs and available observability tools. Below, we will outline the common patterns in logs that indicate resource constraints, permission/authentication issues, and configuration drift between environments.

Log Patterns That Indicate Resource Constraints

Resource constraints are a common cause of pipeline failures. These can include CPU limits, memory usage, or disk space running out. Here's how to recognize these patterns:

Key Indicators in Logs:

• Memory Issues: Look for messages like "out of memory", "memory limit exceeded", or "OOM killed" in your logs. Here’s an example in Kubernetes logs:

pod has been OOMKilled

• CPU Limits: Watch for logs showing that a process exceeded CPU limits or was throttled. Here’s an example:

process 'foo' hit CPU limit, throttling at 100%

• Disk Space: Logs may show file write errors or messages about a disk being full. Here’s an example:

Unable to write to file, disk space is full.

You can resolve the memory issues by increasing the allocated memory for your containers, VM, or cloud instances.

You can resolve the CPU issues by adjusting CPU limits or scaling your infrastructure to add more resources.

And finally, you can resolve disk space issues by cleaning up unused files or increasing disk capacity on the server/container.

Identify Permission and Authentication Issues

Permission and authentication issues often result in pipeline failures due to a lack of access to necessary resources or services. These issues might occur when you’re trying to access databases, deploy to cloud services, or authenticate third-party APIs.

There are some key indicators in the logs that you can look out for:

1. Authentication Failures:

Look for messages related to failed logins, incorrect credentials, or invalid tokens.

Here’s an example:

Authentication failed for user 'admin'

Invalid API token provided.

2. Permission Denied:

Logs may indicate that the CI/CD pipeline lacks the permissions to perform a certain action.

Here’s an example:

Access denied for /path/to/deployment/target

Unauthorized request to cloud service.

How to resolve these errors:

• Credentials: Ensure the credentials (API keys, access tokens, SSH keys) used in the pipeline are up-to-date and correctly configured.

• Permissions: Review and update the role-based access control (RBAC) settings for the service account running the pipeline to ensure it has the necessary permissions.

• Secrets Management: Use tools like Vault, AWS Secrets Manager, or Azure Key Vault to securely manage secrets and credentials.

Troubleshooting Configuration Drift Between Environments

Configuration drift occurs when different environments (like development, staging, production) are not synchronized. This can lead to inconsistent behavior during deployments, and often results in failures in one environment but not in others.

Look out for these key indicators in the logs:

1. Mismatch in Environment Variables:

If you’re using environment variables, check for discrepancies across different stages. For example:

Environment variable DATABASE_URL not found in production

2. Dependency Versions:

Mismatched versions of dependencies between environments can cause unexpected issues.

Here’s an example:

Error: Dependency 'libxyz' version mismatch between environments

3. Service Configuration:

Look for configuration-related errors that might not be present in a development environment but occur in production.

Here’s an example:

Error: Invalid config in 'production-config.yaml'

How to resolve these errors:

• Use Infrastructure as Code (IaC): Tools like Terraform, Ansible, or CloudFormation can help ensure that environments are provisioned consistently.

• Automated Configuration Management: Use CI/CD pipeline steps to automate environment setup to avoid manual changes that can cause drift.

• Environment Consistency Checks: Implement checks to compare configurations and dependencies across environments before deployment.

  • Example: You can add a pre-deployment stage to run a script that compares environment variables, configurations, and dependency versions between staging and production.

• Configuration Management Tools: Use configuration management tools like Chef, Puppet, or SaltStack to maintain consistent configurations across environments.

How to Debug Container-Based Deployment Issues

Debugging container-based deployment issues requires specialized tools and techniques to trace errors in containerized environments. Below are strategies to efficiently collect logs, diagnose failures, and use ephemeral containers for investigation.

Collecting and Analyzing Container Logs Effectively

Container logs are essential for troubleshooting issues, and effective collection and analysis can significantly speed up the debugging process.

Here’s how you can collect container logs:

1. Docker Logs:

You can use Docker’s logs command to view logs of a specific container:

docker logs <container_name_or_id>

If your container uses a logging driver (like json-file or fluentd), ensure that logs are being written to an accessible location.

2. Kubernetes Logs:

For Kubernetes-managed containers, use kubectl to access pod logs:

kubectl logs <pod_name>

To view logs for all containers in a pod:

kubectl logs <pod_name> --all-containers=true

3. Log Aggregation:

You can integrate with centralized logging systems (like, Grafana Loki, Elastic Stack). You can also use Fluentd or Logstash as log shippers for forwarding logs from containers to a logging backend.

Analyzing Logs:

1. Filter and Search Logs:

Use grep to filter logs for specific error messages or patterns:

docker logs <container_name> | grep "ERROR"

In Kubernetes, you can combine kubectl with grep or other tools for advanced filtering.

2. Log Contextualization:

Include metadata in your logs (for example, container ID, environment, timestamps) for easier debugging. Ensure logs are structured in formats like JSON to allow for better querying and filtering.

How to Diagnose Image Pull and Networking Failures

Container deployment failures often stem from issues related to image pulling or network connectivity. Here’s how to troubleshoot these problems:

Image Pull Failures:

There are some common issues you might see, such as:

• Authentication failures: If the container registry requires authentication, ensure your credentials (username/password or tokens) are correct.

• Network connectivity: Check if the container can access the registry endpoint. Often, firewalls or DNS issues block the image pull.

• Image not found: Verify the image name and tag are correct. Use docker pull to manually pull the image to see if the issue is specific to the deployment process.

There are various ways to diagnose them:

For Docker, use:

docker pull <image_name>

This will output the specific error message if the image pull fails.

For Kubernetes, check the event logs for the pod:

kubectl describe pod <pod_name>

Look for the Failed status under "Events" for information about why the image pull failed (for example, wrong credentials or tag). If the issue is with the registry authentication, configure the Kubernetes imagePullSecrets or Docker's credentials to ensure the correct access.

Networking Failures:

Some common issues you may encounter are:

• DNS resolution problems: Containers may fail to resolve hostnames if DNS configurations are incorrect.

• Network policies and firewall rules: Network policies or firewalls may block necessary ports.

• Inter-container communication: If containers need to talk to each other, ensure they’re on the same network or subnet.

Again, there are various ways to diagnose these issues:

For Docker networking:

You can do this to view all Docker networks:

docker network ls

You can also inspect the network of your container like this:

docker network inspect <network_name>

Check if the container is correctly attached to the network and if necessary ports are exposed.

For Kubernetes Networking:

You can use kubectl to check network policies:

kubectl get networkpolicies

You can also check the pod’s network settings like this:

kubectl describe pod <pod_name> | grep -i "Network"

Testing Connectivity Inside Containers:

For Docker, exec into the container and test:

docker exec -it <container_id> /bin/bash
ping <hostname_or_ip>
curl http://<service_address>:<port>

In Kubernetes, use kubectl exec to access the pod and test connectivity:

kubectl exec -it <pod_name> -- /bin/bash

How to Use Ephemeral Debug Containers for Investigation

Ephemeral debug containers are short-lived containers that help investigate issues in a running environment without altering the main application container.

What are Ephemeral Debug Containers?

Ephemeral debug containers allow you to run diagnostic commands (like shell access, ping, or curl) in the same network environment as the failing application container, without modifying the application itself.

How to Set Up Ephemeral Containers in Docker:

1. Use the docker run Command:

You can create a new container for debugging by running a container with the same network settings as the failing container:

docker run -it --network container:<container_name_or_id> --entrypoint /bin/bash <debug_image>

This command runs an interactive shell inside the debug container using the same network as the target container.

Ephemeral Containers in Kubernetes:

Kubernetes allows you to inject an ephemeral debug container into a running pod. You can add a temporary debug container to your pod using the following command:

kubectl debug <pod_name> -it --image=<debug_image> --target=<container_name>

This command will run a new container in the same pod as the target container, allowing you to run diagnostic commands.

Example use cases are investigating file systems, running network diagnostics, checking configuration files, and so on.

These debug containers are meant to be temporary and can be discarded after the issue is resolved.

How to Implement Advanced Debugging Techniques

This section covers advanced methods to diagnose complex CI/CD pipeline issues that standard log analysis might miss. We’ll explore distributed tracing to track requests across multiple services and combine traces with logs and metrics for deeper insights.

These techniques are designed to work within budget constraints, ensuring effective debugging for your CI/CD workflows.

Choosing a Tracing Backend for CI/CD

Distributed tracing enables you to monitor a request’s path through various services in your CI/CD pipeline, such as from a build step to a deployment, identifying delays or failures. Choosing a tracing backend involves selecting a tool to store and analyze these trace data. Below, we compare Jaeger, Tempo, and hosted solutions for distributed tracing.

When to choose each one:

• Jaeger: Ideal for quick, local tracing setups with minimal overhead.

• Tempo: Best for teams already using Grafana Loki/Prometheus for unified observability.

• Hosted Solutions: Suited for large-scale pipelines needing managed scalability.

How to Set Up Distributed Tracing on a Budget

Distributed tracing is crucial for debugging and observing complex, multi-step operations across services. It allows you to follow requests as they propagate through different services and components of your pipeline. Implementing this on a budget can still provide valuable insights.

How to Use OpenTelemetry with Free Backends

OpenTelemetry is an open-source framework that enables you to collect, process, and export telemetry data like traces and metrics. It supports multiple backends, and we’ll focus on using free, budget-friendly backends for trace storage and analysis.

1. Install OpenTelemetry Collector:

OpenTelemetry provides an agent (collector) that collects traces and metrics from your application and sends them to a backend.

To install the OpenTelemetry Collector, download the binary for your OS or use Docker to deploy it:

docker pull otel/opentelemetry-collector:latest

Then run the OpenTelemetry Collector in Docker with a configuration file:

docker run -d --name opentelemetry-collector -p 55680:55680 -p 14250:14250 otel/opentelemetry-collector

2. Configure OpenTelemetry to Export to Free Backends:

There are a few popular free backends you can use for distributed tracing, like Jaeger and Prometheus + Tempo. Let’s see how to use both here.

We’ll start with Jaeger, an open-source tracing backend. It’s highly scalable and works well with OpenTelemetry.

You can use the Docker version for easy deployment:

docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775 -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14250:14250 -p 14268:14268 -p 14250:14250 -p 9431:9431 jaegertracing/all-in-one:1.30

Alternatively, you can use hosted services like Lightstep, AWS X-Ray, or Honeycomb for cloud-native environments.

Now let’s see how to use Prometheus + Tempo for logs and metrics correlation.

Tempo is a distributed tracing backend built by Grafana that integrates well with other Grafana tools (Loki and Prometheus).

You can install Tempo using Docker:

docker run -d --name tempo -p 14268:14268 grafana/tempo:latest

3. Instrument Your Code with OpenTelemetry SDK:

For Python/Node.js/Java/Go applications, you can install the appropriate OpenTelemetry SDK and start tracing.

Here’s a Python example:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation

And a Node.js example:

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/instrumentation

And one in Java:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.0.0</version>
</dependency>

After installation, you can use the OpenTelemetry SDK to instrument the application and start collecting traces for HTTP requests, database queries, and other pipeline interactions.

4. Send Data to the Collector:

You can configure the SDK to send trace data to your OpenTelemetry Collector, which will then forward it to your backend (Jaeger, Tempo, and so on). Here’s an example for Python:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchExportSpanProcessor

trace.set_tracer_provider(TracerProvider())
exporter = OTLPSpanExporter(endpoint="http://localhost:55680")
processor = BatchExportSpanProcessor(exporter)
trace.get_tracer_provider().add_span_processor(processor)

If traces aren’t appearing, several issues might be occurring:

1. Collector fails to start: Check logs with docker logs otel-collector. Look for errors like “port conflict” or “invalid config.”

   • Fix: Change ports (for example, 55681:55680) or verify the config file.

2. No traces in Jaeger: Ensure the collector is sending data to Jaeger (http://localhost:14250). Test with curl http://localhost:55680.

   • Fix: Update the exporter endpoint in your SDK configuration.

3. Resource constraints: Monitor usage with docker stats.

   • Fix: Allocate at least 2GB RAM and 10GB disk space for the collector and backend.

Correlating Traces with Logs and Metrics

Combining traces with logs and metrics provides a holistic view of your pipeline’s operations, allowing you to pinpoint the root cause of issues more effectively.

OpenTelemetry and Grafana allow you to link traces, logs, and metrics into a unified view.

Let’s see how you can do this now.

1. Link Logs and Traces Using Correlation IDs:

When generating logs, include trace and span IDs in the log entries. This allows you to correlate logs with specific trace requests.

Here’s an example:

{
  "timestamp": "2025-05-10T12:00:00Z",
  "level": "error",
  "message": "Build failure",
  "trace_id": "1234567890abcdef",
  "span_id": "0987654321abcdef"
}

2. Integrating Logs (Loki) with Traces (Jaeger/Tempo) in Grafana:

Grafana can integrate traces from Jaeger or Tempo and correlate them with logs from Loki.

To do this:

1. Set up Loki and Tempo in Grafana.

2. In Grafana’s Explore view, you can search logs and traces side-by-side.

3. Create dashboards that show metrics, logs, and traces for a complete view of a request flow.

3. Using Prometheus Metrics with Traces:

Prometheus provides metrics that can be correlated with traces. For example, you can use exemplars in Prometheus to link specific metric data to trace data.

Example: If you have a high error rate in your build step, you can correlate this with trace data to identify which requests failed.

Creating Trace Visualizations for Complex Pipeline Operations

You can visualize traces with Jaeger or Tempo.

To do this in Jaeger:

Once your traces are in Jaeger, you can access the Jaeger UI (http://localhost:16686 by default) and use the search functionality to explore traces based on service name, trace ID, or specific operations.

Jaeger allows you to create custom dashboards to visualize the latency, throughput, and errors of requests across services.

To do this in Tempo (Grafana Integration):

Tempo integrates with Grafana, where you can create dashboards that visualize trace data from your pipeline.

Create a Grafana dashboard:

1. Add Tempo as a data source in Grafana.

2. Use the "Trace" panel to query and visualize traces.

3. Combine trace visualizations with metrics (from Prometheus) and logs (from Loki) to get a unified view of your pipeline.

A typical trace visualization dashboard could show the duration of each step in your pipeline (build, test, deploy) and highlight where delays or errors occur, such as slow database queries or flaky tests.

Troubleshooting Tempo Setup Issues

If Tempo fails to collect or display traces:

1. Container fails to start: Check logs with docker logs tempo. Look for errors like “port already in use” (for example, 14268) or “storage backend unavailable.”

   • Fix: Change ports in the Docker command (for example, -p 14269:14268) or ensure the storage directory (for example, /tmp/tempo) exists and is writable.

2. No traces in Tempo: Verify the OpenTelemetry Collector is sending traces to Tempo’s endpoint (http://localhost:14268). Test connectivity with curl http://localhost:14268.

   • Fix: Update the collector’s exporter configuration to point to the correct Tempo endpoint, and ensure no firewalls are blocking the connection.

3. Resource constraints: Monitor usage with docker stats or top on the host.

   • Fix: Allocate at least 2GB RAM and 10GB disk space for Tempo, as tracing data can grow quickly with high-volume pipelines.

This bar chart displays the average latency (in milliseconds) for key stages of a CI/CD pipeline in May 2025. The Build stage averages around 1,200 ms (blue), the Test stage around 800 ms (yellow), and the Deploy stage around 1,500 ms (pink), highlighting that deployment is the most time-intensive step.

How to Build Comprehensive Debugging Dashboards

This section explains how to create Grafana dashboards to troubleshoot CI/CD pipeline issues effectively. We’ll focus on setting up visualizations for key metrics, logs, and system resources to identify problems like build failures or resource bottlenecks, using budget-friendly tools to keep your observability stack lean and actionable.

Designing Grafana Dashboards Specifically for Troubleshooting

Step 1: Understand the Key Metrics and Logs to Monitor

When designing a Grafana dashboard for debugging, you should focus on metrics and logs that help identify issues in the pipeline. These could include:

• Build failures: Errors during build processes (compilation, test failures).

• Deployment failures: Issues in deployment, such as failed jobs, resource limitations, or misconfigurations.

• Container logs: Information about container status and logs (if using containers in your pipeline).

• System resource usage: CPU, memory, and disk usage that may lead to performance bottlenecks.

• CI/CD-specific metrics: Number of successful vs. failed pipeline runs, job duration, job queue times.

Step 2: Set Up Data Sources

To start building the dashboard, you’ll need to set up your data sources in Grafana. First, connect your Prometheus instance for collecting metrics. To do this, go to Configuration > Data Sources in Grafana. Then just add Prometheus as a data source and enter the URL (for example, http://localhost:9090).

Next, you need to connect your Loki instance for logs. So go ahead and add Loki as a data source by specifying the URL (for example, http://localhost:3100).

Note that if you're using other sources like InfluxDB or Elasticsearch, you’ll need to make sure that they’re properly connected as data sources.

Step 3: Create Panels and Visualizations

Now that your data sources are connected, you can start building your dashboard with the following panels:

• Build Status Panel:

  • Create a stat panel or gauge panel to show the success/failure ratio of pipeline runs.

  • Query Prometheus or Loki for data like build status (success or failure), number of errors, and job durations.

• Error Breakdown Panel:

  • Use a pie chart to visualize the types of errors (for example, build, deployment, or system resource failures).

  • Query the logs in Loki to break down error types based on the CI tool (for example, Jenkins, GitHub Actions).

• Resource Utilization Panel:

  • Use time series graphs to monitor CPU, memory, and disk usage over time, especially for resource-heavy builds or deployments.

• Job Duration Panel:

  • Use bar charts or line graphs to track the average duration of jobs over time. Set thresholds for warning signs if a job takes longer than expected.

Troubleshooting Grafana Dashboard Issues

If Grafana dashboards fail to display data or show errors, you might be having one of these issues:

1. Missing data sources: If metrics, logs, or traces aren’t appearing, verify data source connections in Grafana (for example, Prometheus, Loki, Tempo). Check under Configuration > Data Sources.

   • Fix: Ensure the data source URLs are correct (for example, http://localhost:9090 for Prometheus) and test the connection. Re-add the data source if needed.

2. Incorrect Trace IDs: If trace visualizations (for example, Tempo panels) show no data, confirm that trace IDs in logs match those in Tempo. Use a query like {job="ci_cd"} | json | trace_id="1234567890abcdef" in Loki to cross-check.

   • Fix: Ensure your application logs include trace and span IDs, and verify the OpenTelemetry SDK is correctly instrumented to send traces to Tempo.

3. Resource Constraints: Monitor Grafana’s resource usage with docker stats if running in a container, or top on the host.

   • Fix: Allocate at least 4GB RAM and 10GB disk space for Grafana, especially when rendering complex dashboards with multiple data sources.

How to Set Up Drill-Down Paths from High-Level to Detailed Views

Step 1: Create High-Level Overview Panel

At the top of the dashboard, include a high-level overview panel that summarizes the overall status of the pipeline. This could be:

• Success/Failure Count: A simple stat panel showing the count of successful vs. failed runs.

• Pipeline Health Status: Display an overall health check of your pipeline using color-coded indicators (green for healthy, red for issues).

Step 2: Set Up Drill-Down Links

To allow users to drill down from high-level information to detailed views:

1. Link to detailed build information:

You can create a time series graph that shows build job durations. Add a link to a detailed log view when clicking on a failed job.

For example, when clicking a failed build, you can link to a detailed panel or a separate dashboard that shows the logs and error messages related to that specific run.

2. Link to Logs in Loki:

You can use Loki's LogQL queries to set up a drill-down path. When users click on an error type or a specific job name, it should automatically filter logs for that job or error type.

You can set up drill-down interactions using Dashboard Links in Grafana. In the panel settings, under Links, specify the link to another dashboard that shows detailed logs filtered by the job name or failure type.

Step 3: Implement Time Range Filters

To enhance drill-down functionality, you can add a time range filter to allow users to adjust the time window for both logs and metrics. This enables them to zoom in on a specific time frame where failures occurred.

How to Create Shared Dashboards for Team Troubleshooting

Step 1: Share Your Dashboard

Once your dashboard is designed, you can share it with your team for collaborative troubleshooting:

First, you’ll want to make sure that the correct permissions are set up for your team. You can define specific roles in Grafana with access to the dashboard. Go to Dashboard Settings > Permissions, and grant view or edit access to users or teams.

Next, you can directly share a link to the dashboard with your team members. Use the Share option in the top-right corner of the dashboard, which provides a direct URL and also options to embed the dashboard into other tools (for example, Slack, email).

You can also use template variables to allow users to filter and adjust the dashboard for different pipeline runs or environments. For example, add a variable for build_id, job_name, or branch_name that allows users to select specific builds or branches for more granular troubleshooting.

Step 2: Set Up Alerting

To ensure your team is notified of any pipeline failures, you can set up alerting rules. There are a few important ones you’ll want to set up.

First, create alerts for critical issues, like when a pipeline fails or exceeds expected resource usage. This could be for things like build time exceeding a threshold or failure of a deployment stage.

Grafana can send alerts via various channels such as Slack, email, or webhook.

You can also integrate your dashboards with tools like Slack or Teams for real-time notifications and collaboration. Set up automated messages for your team when the dashboard indicates an issue.

How to Create Automated Diagnostic Tools

Building Scripts that Collect Relevant Logs During Failures

To automate log collection during failures, you need scripts that can capture logs from different CI/CD stages and services as soon as a failure is detected. Here are the steps you can follow to do this:

1. Write Failure Detection Script:

You can leverage the exit status codes of your CI/CD tools to detect failures. For example, in GitLab CI/CD or GitHub Actions, you can check if the last command failed by inspecting $? in Unix-based systems.

# Example for GitLab CI/CD
if [ $? -ne 0 ]; then
    echo "Failure detected, collecting logs..."
    # Custom log collection script call
    ./collect_logs.sh
fi

2. Log Collection Script (collect_logs.sh):

The script should collect relevant logs, system metrics, and trace information. For instance:

#!/bin/bash
LOG_DIR="/path/to/logs"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="${LOG_DIR}/backup/${TIMESTAMP}"
mkdir -p $BACKUP_DIR

# Collect logs from CI/CD agents, containers, or system logs
cp /var/log/ci_cd/*.log $BACKUP_DIR/
cp /path/to/docker_logs/*.log $BACKUP_DIR/
# Collect metrics or traces from monitoring systems if needed

3. Use CI/CD Artifacts:

For platforms like GitLab, GitHub Actions, or Jenkins, you can upload logs as artifacts for further investigation. Configure these platforms to save logs in case of a failure.

Here’s an example for GitHub Actions:

steps:
  - name: Run Tests
    run: |
      npm run test
  - name: Upload logs if test fails
    if: failure()
    uses: actions/upload-artifact@v2
    with:
      name: test-logs
      path: /path/to/test/logs

4. Centralized Logging:

Instead of manually collecting logs, you can centralize log storage using logging systems like Grafana Loki, ELK stack, or even cloud-based solutions. This will ensure that logs are accessible even if they are overwritten or lost on individual systems.

How to Implement Automatic Analysis of Common Error Patterns

Once logs are collected, you can automate the analysis process by defining common error patterns and automatically searching for them in your logs.

Step 1: Define Error Patterns:

Establish error signatures or patterns that are common in your CI/CD process, such as failed builds due to missing dependencies, permission issues, or network timeouts.

You can use regex or regular expressions to capture these patterns. Here’s an example – define a regex for failed test patterns:

TEST_FAILURE_REGEX=".*FAILURE.*"

Step 2: Create Log Analysis Script:

Next, you can write a script that scans logs for these common patterns. The script could then categorize or flag errors.

Here’s an example using grep to detect failure patterns:

#!/bin/bash
LOG_DIR="/path/to/logs"
ERROR_LOG="${LOG_DIR}/error_patterns.log"
touch $ERROR_LOG

# Define error patterns to search for
ERROR_PATTERNS=("FAILURE" "ERROR" "TIMEOUT")

for PATTERN in "${ERROR_PATTERNS[@]}"; do
    grep -i $PATTERN $LOG_DIR/*.log >> $ERROR_LOG
done

if [ -s $ERROR_LOG ]; then
    echo "Error patterns found, review the log file."
fi

Step 3: Automate Alerting:

Once an error pattern is detected, you can integrate the log analysis script with your alerting system (for example, sending an email or Slack notification).

Here’s an example of sending a Slack notification:

if [ -s $ERROR_LOG ]; then
    curl -X POST -H 'Content-type: application/json' \
         --data '{"text":"Error detected in CI pipeline. Check error log."}' \
         https://hooks.slack.com/services/YOUR_SLACK_WEBHOOK_URL
fi

Step 4: Use Observability Tools for Pattern Recognition:

Leverage observability tools (Grafana Loki, Prometheus) that support log querying and visualization. You can create dashboards that automatically detect anomalies like high failure rates or recurring errors.

Example: Set up a Grafana dashboard with alert rules based on log frequency.

How to Create Self-Healing Pipelines Based on Known Issues

Self-healing pipelines can automatically address issues when they are detected by executing pre-defined corrective actions. Let’s walk through how you can set one up.

Step 1: Define Common Failures and Solutions:

Identify recurring issues (for example, dependency issues, build timeouts, flaky tests) that occur in your pipeline. Then, define self-healing actions to mitigate these issues.

Here’s an example of automatically retrying a failed step if it is a known flaky test:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Run Tests
        run: |
          npm run test
      - name: Retry Tests if Failed
        if: failure() && (steps.tests.outcome == 'failure')
        run: |
          echo "Retrying tests..."
          npm run test

Step 2: Automatic Rollbacks:

Set up a rollback process for failed deployments. For instance, if a deployment to production fails, the pipeline can automatically revert to the last successful build.

Example in GitLab CI/CD:

deploy_production:
  script:
    - ./deploy.sh
  when: on_failure
  retry: 3

Step 3: Build Self-Healing Logic Using Retry Mechanisms:

Implement retry logic for transient issues (like network glitches) that often cause failures.

Example of retrying a step in GitHub Actions:

steps:
  - name: Retry Deployment
    run: |
      attempts=0
      max_attempts=3
      until [ $attempts -ge $max_attempts ]
      do
        deploy_script && break
        attempts=$((attempts+1))
        echo "Attempt $attempts failed. Retrying..."
        sleep 5
      done

Step 4: Automate Corrective Actions for Dependency Issues:

Set up automatic fixes for dependency-related failures, like clearing caches or re-installing dependencies:

if [[ $(cat error.log) =~ "dependency not found" ]]; then
    echo "Dependency issue detected, reinstalling dependencies..."
    npm install
fi

Step 5: Integrate with Self-Healing Services:

For more complex self-healing, you can integrate tools like Ansible, Puppet, or even create custom scripts that auto-patch common configuration issues.

How to Conduct Effective Postmortems Using Logs

Logs are often the single most valuable resource when reconstructing what went wrong in a CI/CD pipeline. Conducting effective postmortems with log data allows teams to extract clear timelines, pinpoint root causes, and define steps to prevent recurrence – all based on concrete evidence.

Extract Timeline and Key Events from the Logs

To accurately understand what happened and when from the info contained in your logs, there’s a straightforward process you can follow.

Step 1: Centralize and Structure Logs:

First, make sure that the logs from all pipeline stages (build, test, deploy) are aggregated in a central place like Grafana Loki, ELK, or OpenSearch.

And you’ll want to use a consistent log format (like structured JSON) that includes timestamps, log levels, pipeline stage identifiers, and correlation/request IDs.

Step 2: Build a Chronological View:

You can use timestamp filters in your log UI (for example, Kibana, Grafana Explore) to isolate logs from the incident timeframe.

Look for key lifecycle events, like:

• Start and completion of pipeline steps

• Status changes (for example, "test failed", "deployment started", "build queued")

• Error messages and warnings

• Retry events or unexpected restarts

Step 3: Extract Logs Programmatically (optional):

Use queries (LogQL, Elasticsearch DSL) to export relevant logs for analysis or inclusion in a post-mortem document.

How to Identify Root Causes Through Log Analysis

To go beyond symptoms and find the real issue, there are various steps you can take.

Start by looking for the first failure. You can filter logs by level=error or use log pattern matching to identify the earliest sign of failure. Then trace backward from the failure using correlation IDs or pipeline step identifiers.

Second, make sure you correlate logs across systems. Match logs across CI/CD tools (like GitHub Actions → Docker logs → Kubernetes logs). You can use shared correlation IDs or job IDs to group logs from related events.

Next, pay attention to intermittent signals. Warnings, retries, or degraded performance preceding the failure may reveal environmental or configuration-related issues.

And finally, check for external dependencies. Look for timeout or connection errors involving third-party services, cloud APIs, or internal infrastructure components.

How to Create Actionable Follow-Ups to Prevent Recurrence

There are various things you can do to turn your findings into meaningful process improvements.

1. Document the Findings Clearly:

Create a structured post-mortem doc that includes:

• Timeline of events with log excerpts

• Immediate trigger and root cause (based on logs)

• Impact summary and affected components

• Screenshots or saved log queries for reference

2. Define Preventive Actions:

Examples include:

• Adding missing alerts or log-based monitors

• Improving log verbosity or adding missing metadata

• Fixing brittle test cases or deployment scripts

• Updating infrastructure limits or retry strategies

3. Assign Ownership and Deadlines:

Each action item should have a responsible owner and a due date. If applicable, create automated tests or guardrails to catch similar issues in the future.

4. Update Runbooks and Incident Playbooks:

Add log patterns, example queries, and resolutions to shared documentation. This ensures the next person facing a similar issue can act faster.

Pro Tip: Automate part of your post-mortem process by tagging logs from failed CI runs, exporting them to a shared location, and pre-generating dashboards or incident reports. This reduces manual effort and increases consistency.

How to Optimize Log Storage and Management

As your CI/CD system grows, logs can become massive, consuming storage and impacting performance. Optimizing log storage helps you make sure that you're retaining what's valuable while staying efficient.

How to Implement Log Rotation and Retention Policies

Without rotation and retention, logs will pile up endlessly, leading to disk space exhaustion and poor performance. You can help prevent this with log rotation.

Log rotation involves creating new log files after a size or time threshold and archiving or deleting old ones.

Linux logrotate tool – Configure /etc/logrotate.d/<your-app>:

/var/log/ci_cd/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0640 root adm
}

This example:

• Rotates daily

• Keeps 7 days of logs

• Compresses old logs to save space

Docker logs rotation – in daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

Retention policies ensure that old logs are automatically deleted based on age or storage usage.

You can set one up in Loki like this:

table_manager:
  retention_deletes_enabled: true
  retention_period: 168h  # 7 days

Or in Elasticsearch, use Index Lifecycle Management (ILM):

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "3d", "max_size": "1gb" }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": { "delete": {} }
      }
    }
  }
}

How to Set Up Log Compaction for Long-Term Storage

Compaction reduces redundancy and keeps only critical log info, which is ideal for long-term audits or analytics.

Compaction Techniques:

There are various different compaction techniques you can try. Here are a couple:

1. Loki (boltdb-shipper mode):

• Uses compaction to merge log chunks and reduce storage.

• Configure in loki-config.yaml:

    schema_config:
      configs:
        - from: 2023-01-01
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
  

• Use a low-retention, high-compaction strategy for archived logs.

2. Elasticsearch:

• Use rollup jobs to reduce resolution of old data.

• Stores summarized logs, for example, hourly counts of similar events.

3. Archive to cheaper storage:

• Move infrequent-access logs to S3 or Azure Blob Storage using lifecycle rules.

How to Balance Observability with Resource Constraints

More logs = more observability, but also more cost and overhead. This means that you need a balance. There are various strategies that can help you achieve this balance:

1. Log at appropriate levels:

   • Avoid excessive debug or trace logs in production.

   • Use info and warn levels judiciously.

   • Only use error or critical for actionable failures.

2. Sample logs:

   • If high-volume pipelines generate repetitive logs, enable log sampling to reduce duplicates.

   • Tools like Vector or Fluent Bit support sampling.

3. Filter out noise:

   • Use log filters to exclude non-critical logs before they reach the central system.

4. Separate hot vs. cold logs:

   • Hot logs: recent, real-time data for active debugging.

   • Cold logs: archived for compliance, stored with lower performance/storage priority.

5. Compress everything:

   • Use gzip/zstd compression for both stored and transmitted logs.

   • Loki, Elasticsearch, and Vector support compression out of the box.

Conclusion

In this handbook, you have built a full-stack observability layer specifically optimized for CI/CD pipelines without breaking your infrastructure budget. You now have the tools and know-how to:

• Deploy Grafana Loki or a lightweight ELK alternative to capture structured logs from all parts of your pipeline.

• Unify and enrich logs across CI/CD tools (for example, GitHub Actions, Jenkins, GitLab) using consistent formats and correlation IDs.

• Use powerful log queries (LogQL, Kibana Query Language) to diagnose build failures, flaky tests, and deployment issues with precision.

• Correlate logs with metrics and traces to gain deep, contextual visibility into pipeline behavior.

• Design reusable debugging dashboards and automation that turn raw logs into insights and action.

• Build a culture of shared troubleshooting knowledge through post-mortems, runbooks, and log-driven retrospectives.

To see the full-stack observability layer in action, check out the complete code and configurations in my GitHub repository: github.com/Emidowojo/CICDObservability. This repo includes all the setups for Grafana Loki, OpenTelemetry, Prometheus, and more, so you can deploy and explore the entire pipeline observability stack.

Next Steps for Advanced Observability Implementation

Here’s how you can take your setup even further:

1. Fully integrate distributed tracing: Deploy OpenTelemetry agents across your build and deployment stages. This will help you visualize how code, builds, and deployments flow across systems in real-time.

2. Automate diagnostic scripts and alerts: Build scripts to auto-collect logs and metrics on failure, and trigger alerts when known patterns reoccur. This enables faster detection and even self-healing pipelines.

3. Scale and harden your log infrastructure: As usage grows, implement log retention, compaction, and storage policies. Explore scalable backends like ClickHouse or object storage (e.g., S3) for long-term archiving.

4. Train your team on observability best practices: Share dashboards, create onboarding docs, and schedule log-analysis sessions to build team familiarity with your tools and practices.

📚 Resources for Continued Learning

Official Docs and Tools:

• Grafana Loki Documentation

• Promtail Configuration Guide

• OpenTelemetry

• LogQL Syntax

• Kibana Query Language

• Vector (log forwarding)

Communities:

• r/devops on Reddit

• CNCF Slack – #observability channel

• Log Management Best Practices on Stack Overflow

By investing in observability early and thoughtfully, you not only reduce the time to detect and resolve issues, you also build a more resilient, predictable, and transparent delivery process for your entire engineering team.

I hope this comes in handy for you someday. If you made it to the end of this handbook, thanks for reading! You can connect with me on LinkedIn or on X @Emidowojo if you’d like to stay in touch.

I had the same questions when I started learning about cloud-native technologies. In this article, I'll share my understanding of core observability concepts, introduce essential observability tools, and share insights from a related project I’m working on.

Table of Contents

• My Introduction to Cloud Native Technologies

• What is Observability?

• Types of Observability Data

  • 1. Metrics

  • 2. Logs

  • 3. Traces

• Observability Tools

  • Prometheus

  • OpenTelemetry

• What are OTLP Resource Attributes?

• Importance of OTLP Resource Attributes

• How My Project Work Fits into All This

• Conclusion

• Additional Resources

My Introduction to Cloud Native Technologies

I recently got selected as a mentee for the Linux Foundation Mentorship to work on the CNCF - Prometheus project. The project is UX-focused, and for the next few months, I'll be working with my mentors to understand how users expect to use OpenTelemetry Line Protocol (OTLP) Resource Attributes in Prometheus.

That's quite a mouthful, I know. I was overwhelmed at first, and honestly, I’m still figuring it out. This is my third week, and although I still have a lot to learn—given that I had no knowledge of cloud native technologies when I applied for this internship—I've already learned quite a bit.

As I often do, I intend to document what I learn through articles to help reinforce concepts in my memory and serve as a resource for other newcomers who may find themselves grappling with these technical terms in the future. You know what they say: you can't say you've understood something until you're able to explain it to someone else who's also new to the topic.

What is Observability?

First, I had to learn what the unfamiliar terms meant—and there were a lot of them flying around. OpenTelemetry. Prometheus. Resource attributes. I’ve come to understand that these terms fall under one umbrella: Observability. Let's start there.

Let’s use a food delivery app to illustrate. When someone orders food, a lot happens behind the scenes:

• The app connects to different services (restaurants, payments, delivery)

• Data flows between different systems to process the order, assign a driver, and track delivery

Engineers need to monitor all the processes to ensure everything works smoothly. Are orders taking too long to process? Is the payment system failing? Does the app suddenly crash under load? Which part of the system is causing delays?

To answer these questions, engineers instrument their code. This means that they configure it to send back real-time data about the state, performance, and behavior of the application. This practice of understanding what's happening inside a complex system based on the data it generates is known as Observability.

You can see the process illustrated in the image below:

In the above flowchart diagram, you can see how data might move through a food delivery application system. The flow starts with a User who orders food from a Food App. The Food App connects to three services (Restaurant, Payment, and Delivery). All these components send data to OpenTelemetry, which collects three types of data: Metrics, Logs, and Traces. OpenTelemetry then forwards only the Metrics data to Prometheus, which stores metrics.

Types of Observability Data

There are three key types of data that systems generate for observability:

1. Metrics

Metrics are numerical measurements collected over time that represent the state or performance of your system. Examples in a food delivery app would be the number of orders processed per minute, average order processing time in milliseconds, number of active users or delivery drivers, and so on.

2. Logs

Logs are text-based records of discrete events that occur within your application. Logs for our food delivery app would look something like this:

ERROR: Payment failed for order #12345 - Credit card declined
INFO: Driver #789 assigned to order #12345

3. Traces

Traces track the entire lifecycle of a request as it moves through different services in a system. They help engineers see how different components interact and identify bottlenecks in complex, distributed systems.

For example, in our food delivery app, a single order request might go through the following steps:
User places an order → Request sent to restaurant system → Payment processor verifies payment → Delivery system assigns a driver → User receives confirmation.

Each step in this journey is recorded as part of a trace. This helps engineers pinpoint where delays occur and optimize the system for better performance.

Observability relies on metrics, logs, and traces working together to provide full system visibility. Metrics tell you something is wrong (“Error rate increased by 5%”). Logs tell you why it happened (“Payment failed due to invalid card details”). Traces show exactly where it happened (“Delay in restaurant service response”).

Observability Tools

Observability tools give you visibility into what’s going on within your application. There are a lot of them, but for the purpose of this article, we’ll talk about two: Prometheus and OpenTelemetry.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit. It does two things:

• Collects data from applications, specifically metrics (remember the data types we talked about earlier)

• and stores them in a time-series database.

A time-series database is a database specifically designed to handle measurements or events that occur over time.

Prometheus uses what's called a pull-based model to collect metrics from applications. Pull-based means Prometheus actively requests (pulls) data from services at regular intervals. Think of it like refreshing a webpage to get the latest content.

OpenTelemetry

OpenTelemetry (OTel) collects, processes, and exports observability data. Unlike Prometheus, which mainly focuses on metrics, OpenTelemetry provides a standardized way to instrument applications for all three types of observability data: logs, metrics, and traces.

OpenTelemetry is designed to be vendor-agnostic. This means you can instrument your applications once with OpenTelemetry and then send that telemetry data to any supported observability backend, which could be an open-source solution like Jaeger or Prometheus, or commercial platforms like Datadog, New Relic, Dynatrace, or Honeycomb.

So, for example, you can use OpenTelemetry to instrument your application – and then Prometheus can pull metrics from OpenTelemetry while other tools handle logs and traces.

What are OTLP Resource Attributes?

When OpenTelemetry collects data from applications, it does more than just gather raw telemetry data. It also provides context about that data. This context comes in the form of resource attributes, which describe where the data came from and what it relates to.

The 'resource' is the component (or entity) producing the data, while the 'attributes' are specific details about that resource.

Resource attributes are structured as pairs of information:

• The "key" is the name or identifier of the attribute (like service.name or host.id)

• The "value" is the specific information for that attribute (like payment-service or server-123)

Together, these key-value pairs identify and describe the specific component that's generating the observability data.

For example, if a payment processing service is sending metrics about transaction times, the resource attributes might include:

• service.name: "payment-service"

• service.version: "1.2.3"

• deployment.environment: "production"

These attributes tell you exactly which service, which version, and in which environment the data is coming from, providing context for interpreting the metrics, logs, or traces.

Resource attributes are not arbitrary. OpenTelemetry provides a standardized set of attribute names and formats that everyone should follow, similar to having an agreed-upon language for describing services and their properties.

For example, OpenTelemetry specifies that you should use service.name (not app_name or service_id) to identify your service. They've created these standardized naming conventions (called semantic conventions) so that:

1. All tools in the ecosystem can understand the same attributes

2. Engineers across different companies use consistent terminology

3. Observability data can be easily shared between different systems

You can still create your own custom attributes when you need something specific (like payment.provider for a payment service), but using the standard attributes whenever possible means your telemetry data will work better with existing tools and be more easily understood by other engineers.

Importance of OTLP Resource Attributes

Let’s say engineers want to monitor how long food deliveries take and whether there are delays in specific locations. Without resource attributes, OpenTelemetry might simply collect and report this metric like this:

delivery_time_seconds: 1800

This tells us that a delivery took 1,800 seconds, or 30 minutes, but nothing else. That’s useful, but it lacks context. Where did this happen? Which service handled it? If there was a delay in delivery and engineers wanted to investigate the cause, this alone would not help.

With OpenTelemetry’s resource attributes, the metric becomes more meaningful:

delivery_time_seconds: 1800
resource:
  service.name: "delivery-service"
  service.instance.id: "instance-456"
  cloud.region: "ng-west-2"
  deployment.environment: "production"
  customer.city: "Lagos"
  restaurant.id: "rest-789"

This tells us:

• The data came from the delivery service.

• The instance handling the request is "instance-456".

• It’s running in the ng-west-2 cloud region.

• The environment is Production (not testing or staging), and so on.

Now, engineers can answer more specific questions:

• Are deliveries slower in certain cities? (Filter by customer.city)

• Are certain restaurants taking longer to prepare food? (Filter by restaurant.id)

• Are delays only happening in a specific cloud region? (Filter by cloud.region)

• Are issues only happening in production or also in staging? (Filter by deployment.environment)

When issues arise, resource attributes allow engineers to quickly narrow down the source of problems. Rather than investigating every service, they can filter by specific attributes to focus their efforts.

How My Project Work Fits into All This

Many engineers use OpenTelemetry for data collection and then send metrics to Prometheus for storage, querying, and analysis.

But Prometheus does not natively support resource attributes in the same way as OpenTelemetry. Instead, it relies on labels to organize metrics. Since Prometheus traditionally has its own labeling system for metrics, integrating OpenTelemetry's resource attributes creates interesting UX challenges.

One key challenge is the cardinality explosion. Cardinality refers to the number of unique combinations of label values (or dimensions) that a metric can have. A "cardinality explosion" occurs when you add labels with many possible values. OpenTelemetry often includes many detailed attributes that, if directly converted to Prometheus labels, would create an overwhelming number of time series. This can slow down Prometheus dramatically or even cause it to crash.

The existing solution involves stuffing all resource attributes into a single JSON-encoded Prometheus label. While this prevents the cardinality explosion, it makes querying extremely cumbersome. Users have to use complex join operations and specialized query syntax to filter or aggregate based on these attributes.

This approach is technically functional but creates a poor user experience. My research aims to understand how users mentally model the transition from OpenTelemetry's rich attribute system to Prometheus's more constrained label system.

The research goals are to:

1. Understand how engineers currently use OpenTelemetry resource attributes with Prometheus

2. Identify pain points in the current integration between these systems

3. Discover user expectations for how resource attributes should be represented in Prometheus

This work is particularly important as more organizations adopt OpenTelemetry as their instrumentation standard while continuing to use Prometheus for metrics monitoring. Creating a seamless experience between these two popular open-source projects will help improve the overall observability ecosystem.

Conclusion

Observability in cloud native applications is clearly an interesting subject and important for building reliable, performant systems. The tools and concepts we've explored – metrics, logs, traces, Prometheus, and OpenTelemetry – form the foundation of modern observability practices.

As I continue my mentorship program, I'll share more insights about how these technologies work together and try to break them down from the perspective of a first-time learner.

Additional Resources

Learn more about:

1. OpenTelemetry

2. Prometheus

3. My UX research project

Automating alert provisioning allows you to enforce consistency, secure sensitive credentials, and integrate monitoring into your deployment pipelines.

This guide dives deep into how you can use the SigNoz Terraform Provider to define and manage alert configurations as code, making your observability setup resilient and adaptable.

Here’s what we’ll cover:

1. Why Automate Alert Provisioning?

2. What are SigNoz and Terraform?

3. Overview of the Setup

4. Prerequisites

5. Steps to Set Up the Project

6. Best Practices and Security Considerations

7. Integrating with CI/CD Pipelines

8. Advanced Customizations and Troubleshooting

9. Conclusion

Why Automate Alert Provisioning?

It’s a good idea to automate your alert provisioning for various reasons.

First of all, configuring things manually often leads to discrepancies between environments (development, staging, production). Automating alerts ensures that all environments adhere to the same monitoring standards, reducing the likelihood of configuration drift and improving consistency and uniformity.

Also, when alerts are defined as code, every change is tracked in your version control system. This audit trail makes it easier to trace and review changes, collaborate with team members, and roll back configurations if issues arise.

Something else to consider is that as your infrastructure grows, manually managing alerts becomes unsustainable. Automation allows you to quickly and efficiently update your alerting rules across multiple services without the need for repetitive manual intervention.

Automation also helps improve security. Storing sensitive information like API tokens as environment variables or in secret management systems helps maintain security. Automating the process also minimizes human exposure to critical credentials.

And finally, defining alerts as code enables you to integrate monitoring configurations into your CI/CD pipelines. This leads to continuous testing, validation, and deployment of alert rules alongside application updates.

So as you can see, there are many compelling reasons to go the automation route. Now let’s see how you can do this in practice.

What Are SigNoz and Terraform?

SigNoz is an open-source observability platform designed to collect, analyze, and visualize metrics, logs, and traces from your applications. Its most helpful features include:

• It has comprehensive monitoring abilities: Provides detailed insights into system performance, error rates, and user behaviors.

• It comes equipped with real-time analytics: Enables proactive issue detection and performance optimization.

• It’s community-driven: As an open-source solution, it benefits from community contributions, transparency, and customization.

• It’s cost-effective: Offers powerful observability capabilities without the hefty licensing fees of proprietary solutions.

Terraform is an Infrastructure as Code (IaC) tool developed by HashiCorp. It allows you to define and provision infrastructure using declarative configuration files. Terraform’s core advantages include:

• Its declarative syntax: You specify the desired state of your infrastructure, and Terraform handles the implementation.

• Its version Control: Configuration files can be managed in Git repositories, enabling traceability and rollback of changes.

• Powerful automation: Facilitates automated provisioning and updates, reducing manual effort and errors.

• Multi-cloud support: Manages resources across different cloud providers with a consistent workflow.

So you might be wondering: why should you use Terraform with SigNoz?

First of all, Terraform ensures that your infrastructure is managed consistently across different environments, reducing the risk of configuration drift. It also simplifies managing multiple alerts and resources, making it easier to scale your observability setup.

Beyond this, automating the provisioning process reduces manual setup efforts and minimizes the potential for human error.

And finally, Terraform configurations can be version-controlled, allowing teams to track changes over time and collaborate more effectively.

Overview of the Setup

This setup utilizes the SigNoz Terraform Provider to manage alerts and notification channels within SigNoz Cloud. The configuration includes:

• Provider configuration: Establishes the connection to SigNoz using the API endpoint and a securely provided API token.

• Notification channels: Defines where alerts are sent (for example, via email) to ensure the right teams are notified.

• Alert rules: Specifies the conditions under which alerts are triggered, including thresholds and evaluation windows.

• External variables: Enhances flexibility by allowing critical values (like CPU thresholds and email addresses) to be managed externally.

Prerequisites

Before diving into the setup, make sure you have the following:

1. SigNoz Cloud account: If you don't have one, sign up for SigNoz Cloud to host your observability data and configure alerts.

2. Terraform installed: Install Terraform on your machine. Terraform is the tool you'll use to manage your infrastructure as code.

3. SigNoz API token:

   • Log in to your SigNoz Cloud dashboard.

   • Navigate to Settings > API Tokens.

   • Click Generate API Token.

   • Copy the token, as you'll need it to authenticate Terraform with SigNoz.

4. Basic knowledge of Terraform: Familiarity with Terraform's syntax and concepts, including writing configuration files and running Terraform commands, is essential.

5. Text editor: Use any code editor like Visual Studio Code or Sublime Text to write your Terraform configuration files.

Steps to Set Up the Project

1. Understand the signoz_alert Resource

The signoz_alert resource allows you to create and manage alert rules in SigNoz via Terraform. It supports various alert types, conditions, and configurations. Understanding this resource is crucial as it forms the basis of your alert configuration.

2. Set Up Your Terraform Configuration

Create a new directory for your Terraform configuration:

mkdir signoz-terraform
cd signoz-terraform

Create a main.tf file with the following content:

terraform {
  required_providers {
    signoz = {
      source  = "SigNoz/signoz"
      version = "0.1.3" # Use the latest version from the Terraform Registry
    }
  }
}

provider "signoz" {
  endpoint  = "https://api.us.signoz.cloud" # Replace with your SigNoz Cloud API endpoint
  api_token = var.signoz_api_token
}

variable "signoz_api_token" {}

The provider block configures the SigNoz provider, where endpoint specifies the API endpoint and api_token is passed through a variable for security.

3. Define a Notification Channel (Optional)

If you plan to send alerts to specific channels, define them using signoz_notification_channel. For example, create a channels.tf file:

resource "signoz_notification_channel" "email_channel" {
  name = "Email Channel"
  type = "email"

  receivers {
    email_config {
      to = ["alerts@example.com"]
    }
  }
}

Defining a notification channel ensures that alerts are sent to the correct recipients, enhancing the utility of your alerting system.

4. Create an Alert Using the signoz_alert Resource

Create an alerts.tf file to define your alert:

resource "signoz_alert" "cpu_high_usage" {
  alert            = "High CPU Usage Alert"
  alert_type       = "METRIC_BASED_ALERT"
  severity         = "critical"
  description      = "Alert when CPU usage exceeds 80% over 5 minutes"
  rule_type        = "threshold_rule"
  broadcast_to_all = false
  disabled         = false
  eval_window      = "5m0s"
  frequency        = "1m0s"
  version          = "v4"

  condition = jsonencode({
    compositeQuery = {
      builderQueries = {
        A = {
          aggregateOperator = "avg"
          dataSource        = "metrics"
          metricName        = "cpu_usage_user"
          reduceTo          = "avg"
          filters           = {
            items = []
            op    = "AND"
          }
          groupBy = []
        }
      }
      queryType = "builder"
      panelType = "graph"
      unit      = "%"
    }
    op                = ">"
    target            = 80
    matchType         = "EQUALS"
    selectedQueryName = "A"
    targetUnit        = "%"
  })

  preferred_channels = [signoz_notification_channel.email_channel.name]

  labels = {
    severity = "critical"
    team     = "DevOps"
  }
}

This configuration creates a high CPU usage alert with specific conditions and notifications. The condition parameter is crucial as it defines the alert triggering logic.

5. Provide the API Token

Set the signoz_api_token as an environment variable:

export TF_VAR_signoz_api_token="YOUR_SIGNOZ_API_TOKEN"

This ensures that your API token is securely used by Terraform without hardcoding it in your configuration files.

6. Initialize Terraform

Run:

terraform init

This command initializes your Terraform working directory, downloading necessary plugins, and preparing the environment.

7. Review the Execution Plan

Generate the execution plan:

terraform plan

This step previews the changes Terraform will make, allowing you to verify the configuration before applying it.

8. Apply the Configuration

Apply the changes:

terraform apply

Type yes when prompted. This command applies the configuration, creating or updating resources as specified.

9. Verify the Alert in SigNoz Cloud

To do this, follow these steps:

• Log in to your SigNoz Cloud dashboard.

• Navigate to Alerts.

• Confirm that the "High CPU Usage Alert" is listed.

• Click on the alert to view its details and ensure it matches your configuration.

10. Modify the Alert (Optional)

To change the CPU usage threshold to 75%, follow these steps:

• Update the target in alerts.tf:

    target = 75
  

• Apply the changes:

    terraform apply
  

11. Destroy the Resources (Optional)

To remove the alert and notification channel:

terraform destroy

Type yes to confirm. This command will delete the resources created by Terraform.

Best Practices and Security Considerations

In modern infrastructure automation, robust best practices and security measures are paramount.

Use version pinning

To ensure your alert provisioning remains reliable and maintainable, start with strict version control. Avoid using the latest tag and instead specify an exact version number. This ensures your infrastructure configuration remains consistent and predictable.

By pinning your provider version (for example, use version = "0.1.3" instead of version = ">= 0.1.3".), you eliminate unexpected behavior that can arise from upstream changes. This practice is critical for long-term stability, especially when your infrastructure scales across multiple environments.

Externalize Credentials

Security is non-negotiable. Instead of embedding sensitive details like API tokens in your codebase, leverage environment variables or dedicated secret management tools such as HashiCorp Vault or AWS Secrets Manager.

For instance, storing your SigNoz API token as an environment variable (TF_VAR_signoz_api_token) not only mitigates the risk of credential exposure but also simplifies the process of credential rotation. Also, enforce access control policies around your configuration repositories and CI/CD pipelines to further secure these secrets.

Use Version Control

A mature setup also demands rigorous infrastructure version control. Hosting your Terraform configuration in a Git repository with branch protection and code review policies allows you to track changes meticulously, roll back problematic updates, and maintain an audit trail. This traceability is essential when troubleshooting issues or validating compliance during audits.

You should also document your configuration decisions extensively—explain why a particular CPU threshold was chosen or why specific labels (like severity and team) are used. Such documentation becomes invaluable for onboarding new team members or when revisiting configurations months later.

Integrating with CI/CD Pipelines

Integrating Terraform with your CI/CD pipeline is a cornerstone of a modern, automated deployment strategy. A well-architected pipeline not only validates your infrastructure changes but also ensures that your alerting rules remain in sync with your evolving application environment.

Continuous Integration (CI) involves automatically merging code changes into a shared repository and running automated tests on each commit. In practice, embedding Terraform plan into your pull request workflow provides early feedback, catching misconfigurations before they reach production. For instance, a GitHub Actions workflow can automatically check your changes:

name: Terraform CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
      - name: Terraform Init
        run: terraform init
      - name: Terraform Plan
        run: terraform plan -no-color
        env:
          TF_VAR_signoz_api_token: ${{ secrets.SIGNOZ_API_TOKEN }}

This workflow uses GitHub secrets to securely manage your API tokens while validating the configuration changes. Continuous Delivery (CD) takes this further by automating deployments. Once your plan is approved, an automated Terraform apply step (often scheduled during off-peak hours or coordinated with application deployments) ensures smooth, coordinated rollouts.

Advanced pipelines can also include automated rollback mechanisms. For example, if a deployment triggers an anomaly, scripts can automatically revert to a previous version using your version control history—minimizing downtime and reinforcing the feedback loop between application performance and infrastructure configuration.

Advanced Customizations and Troubleshooting

As your observability requirements evolve, you may need to implement advanced customizations. One powerful approach is using multi-metric composite alerts. Instead of triggering an alert on a single threshold, you can design rules that combine multiple conditions—for example, firing only when both CPU usage and memory consumption exceed critical levels. This nuanced alerting minimizes false positives and ensures alerts are issued only during genuine performance issues.

Terraform’s modular design is especially useful here. By creating reusable modules that encapsulate your alert configurations, you can parameterize key variables—such as thresholds, evaluation windows, and notification channels—across a microservices architecture. This modularity enforces consistency while simplifying management and scaling.

Troubleshooting advanced configurations starts with reviewing your terraform plan output to ensure every change aligns with expectations. If an alert isn’t triggering as expected, inspect the JSON structure generated by the jsonencode function. Even minor syntax errors can cause significant issues.

When integrating with incident management tools like PagerDuty or Opsgenie, run comprehensive end-to-end tests in a staging environment. For example, deploy a test alert to a dedicated channel to verify that the complete alerting pipeline—from condition detection to incident escalation—is functioning correctly.

In one real-world scenario, a misconfigured composite query in an alert’s JSON payload led to intermittent failures. By enabling detailed provider logs and iteratively validating the JSON output, the issue was rapidly isolated and resolved. Such experiences underscore the importance of rigorous logging, validation, and testing in production-grade setups.

Conclusion

Automating alert provisioning is a transformative approach to managing observability in modern infrastructures.

By defining alerts and notification channels as code, you make your systems more consistent, scalable, secure, and easily integratabtle with CI/CD. You can set up uniform alert rules across all environments, quickly update and deploy monitoring configs, easily handle secure credentials, and automate CI/CD workflows that stay in sync with application changes. They also become easier to integrate with CI/CD workflows.

I hope you’ve enjoyed this tutorial and have learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy coding!