Observability tools, such as logs, metrics, and traces, provide the visibility you need to pinpoint issues quickly. In this handbook, we’ll explore free and open-source tools you can use to make your CI/CD pipelines more reliable. We’ll use practical steps to troubleshoot like a pro – no enterprise licenses required.

Table of Contents

1. Prerequisites

2. Why Observability is Important

3. How to Install and Configure Grafana Loki on Budget Infrastructure

4. How to Implement an ELK Stack Alternative for Pipeline Observability

5. How to Create a Unified Logging Strategy Across Pipeline Components

6. How to Query and Analyze Logs for Effective Troubleshooting

7. How to Set Up Prometheus Metrics Alongside Your Logs

8. How to Create Grafana Dashboards That Combine Metrics and Logs

9. How to Use Exemplars to Jump from Metrics to Relevant Logs

10. How to Diagnose and Fix Common CI/CD Problems

11. How to Implement Advanced Debugging Techniques

12. How to Conduct Effective Postmortems Using Logs

13. How to Optimize Log Storage and Management

14. Conclusion

Prerequisites

There are some things you should know and have to get the most out of this handbook:

Technical Knowledge:

• Basic understanding of CI/CD pipelines (for example, build, test, deploy stages).

• Familiarity with Linux/Unix commands (for example, mkdir, grep, curl).

• Comfortable with Docker basics (for example, docker run, docker-compose up).

• Optional: Awareness of observability concepts (logs, metrics, traces) or YAML configuration.

Software and Tools:

• Docker and Docker Compose: Installed and running (verify with docker --version and docker-compose --version).

• CI/CD Platform: Access to GitHub Actions, Jenkins, or GitLab CI with a sample pipeline that generates logs.

• Text Editor: For editing YAML files (for example, VS Code, Nano).

• Web Browser: To access tool UIs (for example, Grafana on port 3000, Kibana on 5601).

• Optional: curl for testing log forwarding, Git for version control.

Hardware and Infrastructure:

• Machine with:

  • OS: Linux, Windows (with WSL2), or macOS.

  • 4GB RAM (8GB recommended), 20GB free disk space.

  • Stable internet and ability to open ports (for example, 3100 for Loki, 9200 for Elasticsearch).

• Optional: Cloud provider access (for example, AWS, GCP) for scalable setups.

Access and Permissions:

• Admin access to install Docker and configure CI/CD tools.

• Permissions to modify pipeline configs (for example, .github/workflows, .gitlab-ci.yml).

• Optional: Container registry access (for example, Docker Hub) for custom images.

Why Observability is Important

Modern CI/CD pipelines are no longer linear scripts – they are now complex, distributed systems involving multiple tools, environments, and infrastructure layers. One job runs on GitHub Actions, another deploys via Jenkins, and a third builds Docker images in a Kubernetes cluster.

So when something breaks, you’re left chasing logs across tools, guessing where the issue originated, and wasting hours trying to reproduce it.

And worse still, traditional debugging tools often stop at the surface, only showing failed jobs without the context of why they failed or where in the system the fault actually lies.

Observability flips the script. Instead of hunting through disconnected logs or rerunning failed builds blindly, observability gives you insight, not just data. By combining structured logs, metrics, and traces, you can:

• Reconstruct exactly what happened in a pipeline failure

• Trace a failure across CI agents, deployment steps, and containers

• Visualize patterns and anomalies before they become outages

More importantly, observability helps you move from reactive debugging to proactive prevention.

Here’s what you’ll learn about and accomplish in this guide:

• Set up cost-effective observability using Grafana Loki, lightweight ELK, and OpenTelemetry

• Create a unified logging strategy to connect your pipeline

• Write precise queries to quickly pinpoint root causes, correlate logs, metrics, and traces for comprehensive debugging

• Troubleshoot CI/CD issues like build failures, flaky tests, and container crashes

• Build custom dashboards and automated diagnostic tools

• Promote observability through documentation and post-mortems

Whether you're a solo developer or part of a DevOps team, this guide will transform your chaotic CI/CD pipelines into clear, reliable, and observable systems.

How to Choose the Right Observability Tool for CI/CD

Here’s a quick comparison of Grafana Loki, Lightweight ELK, and Vector for CI/CD observability:

How to choose:

• Loki: Ideal for startups or solo devs with limited resources. Integrates well with Prometheus/Grafana.

• ELK: Best for teams needing Kibana’s advanced visualizations or handling large log volumes.

• Vector: Great for lightweight log forwarding in distributed CI/CD setups.

Grafana Loki is a log aggregation system like ELK, but it's more lightweight, and it’s ideal for CI/CD pipelines with limited infrastructure.

How to Install and Configure Grafana Loki on Budget Infrastructure

🛠 Option A: Quick Docker Setup (Recommended for Budget Infra)

1. Create a directory for configuration:

    mkdir -p ~/loki-setup && cd ~/loki-setup
   

2. Create a docker-compose.yml:

    # Defines a Docker Compose setup for Grafana Loki and Promtail to aggregate and scrape logs efficiently.
    version: "3"
   
    services:
      loki:
        image: grafana/loki:2.9.4  # Uses Loki version 2.9.4 for lightweight log aggregation.
        ports:
          - "3100:3100"  # Exposes Loki’s HTTP API port for log ingestion and queries.
        command: -config.file=/etc/loki/loki-config.yaml  # Specifies the configuration file for Loki.
        volumes:
          - ./loki-config.yaml:/etc/loki/loki-config.yaml  # Mounts the local config file into the container.
   
      promtail:
        image: grafana/promtail:2.9.4  # Uses Promtail version 2.9.4 to scrape and forward logs to Loki.
        volumes:
          - /var/log:/var/log  # Mounts the host’s log directory for Promtail to scrape.
          - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml  # Mounts the Promtail config file.
        command: -config.file=/etc/promtail/promtail-config.yaml  # Specifies the configuration file for Promtail.
   

3. Create a basic loki-config.yaml:

    # Configures Grafana Loki for lightweight log storage and querying in a CI/CD environment.
    auth_enabled: false  # Disables authentication for simplicity (not recommended for production).
   
    server:
      http_listen_port: 3100  # Sets the port for Loki’s HTTP API.
   
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: inmemory  # Uses in-memory storage for the ring, suitable for small setups.
          replication_factor: 1  # Sets single replica for minimal resource use.
      chunk_idle_period: 3m  # Flushes chunks to storage after 3 minutes of inactivity.
      max_chunk_age: 1h  # Retires chunks after 1 hour to balance storage and query performance.
   
    schema_config:
      configs:
        - from: 2023-01-01  # Defines the schema start date.
          store: boltdb-shipper  # Uses BoltDB for indexing logs.
          object_store: filesystem  # Stores logs on the local filesystem.
          schema: v11  # Specifies schema version for log storage.
          index:
            prefix: index_  # Prefix for index files.
            period: 24h  # Rotates indexes daily.
   
    storage_config:
      boltdb_shipper:
        active_index_directory: /tmp/loki/index  # Directory for active index files.
        cache_location: /tmp/loki/boltdb-cache  # Cache location for BoltDB.
      filesystem:
        directory: /tmp/loki/chunks  # Directory for storing log chunks.
   
    limits_config:
      enforce_metric_name: false  # Disables strict metric name enforcement for flexibility.
   

4. Create a basic promtail-config.yaml:

    # Configures Promtail to scrape system logs and forward them to Loki.
    server:
      http_listen_port: 9080  # Sets Promtail’s HTTP port for metrics and health checks.
      grpc_listen_port: 0  # Disables gRPC to reduce resource usage.
   
    positions:
      filename: /tmp/positions.yaml  # Stores the position of scraped logs to resume after restarts.
   
    clients:
      - url: http://loki:3100/loki/api/v1/push  # Specifies the Loki endpoint for log ingestion.
   
    scrape_configs:
      - job_name: system  # Defines a scraping job for system logs.
        static_configs:
          - targets:
              - localhost  # Targets the local host for log collection.
            labels:
              job: varlogs  # Labels logs for easy querying in Loki.
              __path__: /var/log/*.log  # Scrapes all log files in /var/log directory.
   

5. Run it:

    # Starts the Loki and Promtail containers in detached mode for background operation.
    docker-compose up -d
   

✨ This brings up Loki and Promtail with minimal resources, no authentication, and logs scraping from /var/log.

Troubleshooting Loki Setup Issues

If Loki or Promtail fails to start, one of the following may be the issue:

1. Container crashes: Check logs with docker logs loki or docker logs promtail. Look for errors like “out of memory” or “port already in use.”

   • Fix: Increase memory (for example, docker-compose.yml resource limits) or change ports (e.g., 3101:3100).

2. Logs not ingested: Verify Promtail is scraping the correct path (/var/log/ci/*.log) using docker exec promtail cat /etc/promtail/promtail-config.yaml

   • Fix: Update __path__ in promtail-config.yaml to match your CI/CD log directory.

3. Resource Constraints: Monitor resource usage with docker stats or top on the host.

   • Fix: Ensure your machine has at least 4GB RAM and 20GB disk space, as specified in the prerequisites.

Configuration for CI/CD Logging

To adapt for CI/CD logs, you should:

1. Configure your CI/CD tools to write logs to disk:

For example, GitHub Actions with a custom runner can write logs to /var/log/gha/*.log.

Update Promtail:

# Configures Promtail to scrape logs from GitHub Actions runners for CI/CD observability.
scrape_configs:
  - job_name: github_actions  # Defines a scraping job for GitHub Actions logs.
    static_configs:
      - targets: ['localhost']  # Targets the local host where the runner writes logs.
        labels:
          job: gha  # Labels logs for identification in Loki queries.
          __path__: /var/log/gha/*.log  # Scrapes logs from the specified directory.

2. Use structured logging (JSON):

Make sure your CI/CD tools or scripts output logs in structured format:

Example:

# Example of a structured JSON log for CI/CD pipelines, enabling easy parsing and querying.
{
  "timestamp": "2025-05-10T13:00:00Z",  # UTC timestamp for log entry.
  "level": "error",  # Log level to indicate severity.
  "job": "deploy",  # Identifies the CI/CD job (e.g., deploy stage).
  "message": "Image pull failed"  # Descriptive message for the error.
}

This helps when querying with LogQL.

How to Connect CI Agents to Loki

This section explains three different ways to get your CI pipeline logs into Loki for monitoring and analysis:

Option 1 – Local setup:

Your CI agents write log files to disk, and Promtail (running on the same machine) reads those files and sends them to Loki.

Option 2 – Using Docker logging driver (Docker containers):

If your CI agents run in Docker containers, you install a special Loki plugin that automatically captures all container output and sends it directly to Loki without needing separate log files.

# Installs the Loki Docker logging driver to send container logs directly to Loki.
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Then run your agent container:

# Runs a CI agent container with the Loki logging driver to forward logs.
docker run --log-driver=loki \
  --log-opt loki-url="http://<your-loki-host>:3100/loki/api/v1/push" \
  my-ci-agent-image

Option 3 – Remote setup:

If you can't install Promtail locally, you can use a log forwarding tool like Fluent Bit or Vector to collect logs and push them to Loki over the network.

The goal: Regardless of which option you choose, you’ll end up with all your CI pipeline logs centralized in Loki, where you can search through them, create dashboards in Grafana, and set up alerts when things go wrong.

It essentially gives you flexibility to integrate log collection based on your infrastructure setup – whether you prefer local agents, Docker plugins, or remote forwarding.

How to Implement an ELK Stack Alternative for Pipeline Observability

When full ELK (Elasticsearch, Logstash, Kibana) is too heavy for your infrastructure, you can go with lightweight setups that achieve similar observability at a lower cost and resource usage.

How to Install Lightweight Versions of Elasticsearch, Logstash, and Kibana

Goal: Stand up a minimal yet functional ELK stack for debugging CI/CD pipelines.

1. Use Docker to spin up lightweight containers

Create a docker-compose.yml:

# Defines a Docker Compose setup for a lightweight ELK stack to aggregate and visualize CI/CD logs.
version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0  # Uses Elasticsearch 7.17.0.
    container_name: elasticsearch
    environment:
      - discovery.type=single-node  # Runs Elasticsearch in single-node mode for simplicity.
      - xpack.security.enabled=false  # Disables security features for lightweight setup.
    ports:
      - "9200:9200"  # Exposes Elasticsearch’s HTTP API port.
    volumes:
      - esdata:/usr/share/elasticsearch/data  # Persists Elasticsearch data.

  logstash:
    image: docker.elastic.co/logstash/logstash:7.17.0  # Uses Logstash 7.17.0.
    container_name: logstash
    ports:
      - "5044:5044"  # Port for receiving logs from Beats.
      - "9600:9600"  # Port for Logstash monitoring.
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf  # Mounts Logstash config file.

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.0  # Uses Kibana 7.17.0 for visualization.
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200  # Links Kibana to Elasticsearch.
    ports:
      - "5601:5601"  # Exposes Kibana’s web UI port.

volumes:
  esdata:  # Defines a volume for persisting Elasticsearch data.

2. Minimal Logstash pipeline configuration (logstash.conf)

// Configures Logstash to process and forward CI/CD logs to Elasticsearch.
input {
  beats {
    port => 5044  // Listens for logs from Filebeat on port 5044.
  }
}

filter {
  json {
    source => "message"  // Parses JSON-formatted log messages for structured data.
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]  // Sends processed logs to Elasticsearch.
    index => "ci-logs-%{+YYYY.MM.dd}"  // Stores logs in daily indexes (e.g., ci-logs-2025.05.14).
  }
}

Troubleshooting ELK Setup Issues

If Elasticsearch, Logstash, or Kibana fails to start, one of the following might be the issue:

1. Container crashes: Check logs with docker logs elasticsearch, docker logs logstash, or docker logs kibana. Look for errors like “insufficient disk space” or “port conflict” (for example, 9200, 5601).

   • Fix: Free up disk space (ensure at least 20GB available) or change ports in docker-compose.yml (for example, 9201:9200).

2. Logs not ingested: Verify Logstash is receiving data from Filebeat or Vector using docker logs logstash. Check the logstash.conf input port (for example, 5044).

   • Fix: Ensure Filebeat or Vector is configured to send to the correct Logstash endpoint (e.g., localhost:5044) and update if needed.

3. Resource constraints: Monitor resource usage with Docker stats or top on the host.

   • Fix: Allocate at least 8GB RAM and 30GB disk space, as Elasticsearch requires more resources than Loki. Adjust memory limits in docker-compose.yml if necessary.

How to Configure Log Shippers for Different CI/CD Components

Goal: Get logs from your pipeline into Logstash or Elasticsearch.

Option 1: Use Filebeat (lightweight log shipper)

Install Filebeat on your CI/CD hosts (GitHub runner, Jenkins node, GitLab runner, and so on).

Filebeat config snippet (filebeat.yml):

# Configures Filebeat to collect CI/CD logs and forward them to Logstash.
filebeat.inputs:
  - type: log  # Specifies log file input.
    enabled: true  # Enables the input.
    paths:
      - /var/log/ci/*.log  # Scrapes logs from the specified CI log directory.

output.logstash:
  hosts: ["localhost:5044"]  # Forwards logs to Logstash on port 5044.

Then run:

# Runs Filebeat with the specified configuration file for log collection.
filebeat -e -c filebeat.yml

Option 2: Use Vector.dev as a more resource-efficient alternative to Filebeat

Vector configuration (vector.toml):

# Configures Vector to collect, parse, and forward CI/CD logs to Elasticsearch efficiently.
[sources.ci_logs]
  type = "file"  # Specifies file-based log collection.
  include = ["/var/log/ci/*.log"]  # Targets CI log files.

[transforms.json_parser]
  type = "remap"  # Uses remap transform to parse logs.
  inputs = ["ci_logs"]  # Processes logs from the ci_logs source.
  source = '''
  . = parse_json!(.message)  # Parses JSON log messages into structured data.
  '''

[sinks.to_elasticsearch]
  type = "elasticsearch"  # Sends logs to Elasticsearch.
  inputs = ["json_parser"]  # Uses parsed logs from the json_parser transform.
  endpoint = "http://localhost:9200"  # Specifies the Elasticsearch endpoint.
  index = "ci-logs"  # Stores logs in the ci-logs index.

Run:

# Runs Vector with the specified configuration file for log processing.
vector -c vector.toml

How to Set Up Index Patterns and Basic Visualizations

Goal: Make CI/CD logs queryable and visual in Kibana.

1. Open Kibana (http://localhost:5601)

• Go to Stack Management → Index Patterns

• Create a new pattern: ci-logs-*

• Choose a time field like @timestamp

2. Visualizations for Common CI/CD Use Cases

• Bar charts: Number of failed vs passed builds per day

• Pie chart: Top error types or most frequent failing test names

• Line chart: Duration of builds over time (if duration is logged)

3. Saved Searches & Dashboards

You can save a search like this:

message: "error" AND job_name: "build"

You can also combine visualizations into a CI/CD Health Dashboard.

How to Create a Unified Logging Strategy Across Pipeline Components

Creating a unified logging strategy across your CI/CD pipeline components ensures that logs are consistent, traceable, and easy to correlate. This helps you quickly debug issues, monitor system health, and trace requests across different tools and services. Let’s discuss some key practices for achieving a unified logging strategy:

Implementing Consistent Log Formats Across Different Tools

Consistent log formats are important for various reasons. First of all, a standardized log format enables easier querying, searching, and visualization. It also helps with correlation of logs from different services. And consistency also ensures that all logs provide necessary details like timestamp, log level, and request context.

There are also some best practices you should follow when formatting logs:

JSON Format is highly recommended as it’s structured, machine-readable, and compatible with many observability tools (for example, Loki, Elasticsearch, Grafana).

There are also some key fields you should include:

• timestamp: The time the log entry was created (preferably in UTC).

• log_level: Indicate whether the log is an INFO, ERROR, DEBUG, and so on.

• service: The service or component generating the log.

• message: A concise description of the event or error.

• correlation_id: A unique identifier for requests to trace logs across systems.

Here’s an example of a consistent log in JSON format:

{
  "timestamp": "2025-05-10T12:34:56Z",
  "log_level": "ERROR",
  "service": "ci_cd_pipeline",
  "message": "Build failed due to missing dependency",
  "correlation_id": "1234567890abcdef"
}

How to Set Up Log Forwarding from GitHub Actions, Jenkins, or GitLab

Log forwarding refers to shipping logs from your CI/CD pipelines to a central spot for easy tracking. It’s helpful because it lets you spot issues fast and debug without digging through scattered files.

For GitHub Actions, you can configure workflows to write logs to a file or send them directly to a log aggregation tool like Loki. In Jenkins, you can use pipeline scripts to forward logs to a log server or file system. Similarly, for GitHub CI, you can add scripts in .gitlab-ci.yml to forward logs to a centralized endpoint.

Using Actions for Outputting Logs:
You can store logs in files and then forward them to a logging system (like Loki or Elasticsearch).
Here’s an example in a GitHub Action workflow:

# Defines a GitHub Actions workflow to run tests and forward logs for observability.
jobs:
  build:
    runs-on: ubuntu-latest  # Uses an Ubuntu runner.
    steps:
      - name: Checkout repository  # Checks out the repository code.
        uses: actions/checkout@v2
      - name: Run tests and log output  # Runs tests and saves output to a log file.
        run: |
          echo "Starting tests..."
          npm test | tee test.log  # Captures test output to test.log.
          # Forwards the log file to a Loki endpoint via HTTP POST.
          curl -X POST -F 'file=@test.log' http://your-loki-endpoint

Log Forwarding with Promtail:
If you are using Grafana Loki for log aggregation, set up Promtail to scrape the logs from the GitHub Actions runner.

Jenkins:

Jenkins logs can be forwarded to external systems (like Elasticsearch or Loki) by using log shippers or plugins.

You can use the Logstash Plugin to forward Jenkins logs to an ELK stack or other systems:

• Install the Logstash plugin on Jenkins.

• Configure the plugin to forward logs to an Elasticsearch server or a logging system of choice.

• In Jenkins, add log forwarding configurations:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        script {
          // Example of forwarding logs to a log server
          sh 'echo "Build successful" | curl -X POST -d @- http://your-log-server'
        }
      }
    }
  }
}

Forward to Loki:
Jenkins supports the loki logging driver for containers if running Jenkins in Docker. You can send logs directly to Loki using this driver:

# Runs a Jenkins container with the Loki logging driver to send logs directly to Loki.
docker run --log-driver=loki --log-opt loki-url=http://loki:3100 jenkins/jenkins:lts

GitLab:

GitLab CI allows logs to be forwarded to external systems for centralized collection and analysis.

Use GitLab CI/CD to Output Logs:
Example in .gitlab-ci.yml:

# Defines a GitLab CI/CD pipeline to run a build and forward logs to Loki.
stages:
  - build
build:
  script:
    - echo "Starting the build" | tee build.log  # Saves build output to build.log.
    - curl -X POST -d @build.log http://your-loki-endpoint  # Forwards the log to Loki.

GitLab Runners:
Configure GitLab runners to forward logs to an external service like Loki or Elasticsearch using log-driver settings or the fluentd log shipper.

How to Add Correlation IDs to Trace Requests Through the System

Why Correlation IDs Are Important:

Correlation IDs allow you to trace a single request as it travels through different services and tools, enabling end-to-end visibility and troubleshooting.

They are critical for debugging distributed systems, especially when different services (for example, CI tool, deployment tool, API service) are involved.

How to Add Correlation IDs:

You can use a UUID (Universally Unique Identifier) or a GUID (Globally Unique Identifier) to generate a unique ID for each request.

If you are using microservices or multiple services in the pipeline, just make sure that the same ID is propagated across each service.

Many logging libraries (for example, winston for Node.js, log4j for Java) support automatic correlation ID generation and logging.

Here’s an example in Node.js (using winston):

// Sets up Winston for structured logging with correlation IDs in a CI/CD pipeline.
const { createLogger, transports, format } = require('winston');
const { printf } = format;

// Creates a logger with a custom format including correlation IDs.
const logger = createLogger({
  format: printf(({ level, message, timestamp }) => {
    return `${timestamp} [${level}] ${message} correlation_id=${generateCorrelationId()}`;
  }),
  transports: [
    new transports.Console(),  // Outputs logs to the console.
  ],
});

// Generates a random correlation ID for tracing requests.
function generateCorrelationId() {
  return Math.random().toString(36).substring(2, 15);
}

// Logs a sample message.
logger.info('Pipeline execution started');

How to Propagate Correlation IDs Between Services:

In CI/CD tools, you can configure your pipeline to inject the correlation ID into logs. For example, in GitHub Actions, you can generate a correlation ID in the env section and propagate it in each job:

# Defines a GitHub Actions workflow that includes a correlation ID for log tracing.
jobs:
  build:
    runs-on: ubuntu-latest  # Uses an Ubuntu runner.
    env:
      CORRELATION_ID: ${{ github.run_id }}  # Uses the GitHub run ID as a correlation ID.
    steps:
      - name: Checkout repository  # Checks out the repository code.
        uses: actions/checkout@v2
      - name: Log build start with correlation ID  # Logs the build start with the correlation ID.
        run: echo "Build started with Correlation ID: $CORRELATION_ID"

Include Correlation IDs in All Logs:

You’ll want to make sure that logs from all components in the pipeline (GitHub Actions, Jenkins, GitLab, deployment tools, and so on) include the correlation ID as part of the log message. This allows you to trace the logs of a single request or pipeline run across different services.

Visualize Your Log Flow

You can create a diagram showing how logs move from your CI/CD tool (for example, GitHub Actions) to Promtail/Vector, then to Loki/Elasticsearch, and finally to Grafana/Kibana for visualization. Use tools like Draw.io to map your pipeline’s observability flow

How to Query and Analyze Logs for Effective Troubleshooting

In this section, you’ll learn how to use LogQL (Loki's query language) to cut through the noise and find the specific logs that matter. Whether you're hunting down a mysterious build failure or tracking deployment issues across multiple services, these query patterns always help.

This bar chart illustrates the CI/CD build performance from May 20 to May 26, 2025. It compares the number of successful builds (in blue) to failed builds (in pink) each day. Successful builds consistently range between 40 and 50, while failed builds peak at 10 on May 23, with other days showing 2 to 8 failures. This indicates a generally stable pipeline with occasional issues.

How to Write Advanced LogQL Queries to Pinpoint CI/CD Issues

LogQL is Grafana Loki's query language, designed for querying logs with a syntax similar to Prometheus’s PromQL. It enables efficient log searches and is particularly useful in troubleshooting CI/CD issues.

Basic LogQL Syntax:

1. Log Streams:

{job="ci_cd", level="error"}

This query retrieves logs where the job label is ci_cd and the level label is error.

2. Log Filters:

{job="ci_cd"} |= "build failed"

The |= operator filters logs to include only those that contain the specified string, for example "build failed".

3. Regular Expressions:

{job="ci_cd"} |~ "error.*timeout"

This uses the |~ operator to filter logs using a regular expression. In this case, it finds logs that contain an "error" followed by "timeout".

Advanced LogQL Queries for CI/CD Issues:

1. Filter Logs for Specific Build Failures:

If your pipeline uses a specific label for build names:

{job="ci_cd", build="build123"} |= "failure"

This finds logs related to the build123 job that contain the word "failure".

2. Using Time Range and Grouping:

To find error logs in the last 15 minutes:

{job="ci_cd", level="error"} | "build failed" | range(start="15m")

To group logs by job and error type:

sum by (job) (count_over_time({job="ci_cd", level="error"}[5m]))

This will return the count of error logs per job, grouped by job name, over the last 5 minutes.

How to Create Pipeline-Specific Queries for Common Failure Patterns

Common Failure Patterns in CI/CD Pipelines:

1. Build Failures:

If your CI system logs contain build errors, you can identify them with:

{job="ci_cd", level="error"} |= "build failed"

You can extend this to filter by specific steps or stages, for example, “test failed”, or “compilation error”.

2. Test Failures:

Logs from your test runner (for example, Jest, Mocha, JUnit) can contain specific failure messages:

{job="ci_cd", stage="test"} |= "test failed"

3. Dependency Issues:

If your pipeline is failing due to missing or conflicting dependencies, look for npm, maven, or docker related errors:

{job="ci_cd", image="node"} |= "npm ERR!"

Or for Maven-related issues:

{job="ci_cd", image="maven"} |= "[ERROR]"

4. Resource Constraints (for example, Out of Memory):

If you experience resource constraints, you might see logs like "OutOfMemoryError":

{job="ci_cd", level="error"} |= "OutOfMemoryError"

Example of combining filters:

{job="ci_cd", level="error"} |= "build failed" |~ "timeout|dependency" | range(start="1h")

This combines log filters for "build failed", matching any logs with the terms "timeout" or "dependency", from the last hour.

How to Set Up Alert Rules Based on Log Patterns

Alerts help detect recurring issues proactively. They notify you when a specific pattern appears in your logs, allowing you to take quick action.

Steps for Setting Up Alerts:

1. Create a Query for the Alert:

First, define the log pattern you want to monitor. For example, an alert for build failures:

{job="ci_cd", level="error"} |= "build failed"

2. Create an Alert in Grafana:

Follow these steps to set up Grafana alerts:

• Go to your Grafana dashboard.

• Choose the panel you want to set the alert on (or create a new panel for this purpose).

• In the panel, click the Alert tab.

• Set the Query field to your LogQL query, such as the one above.

• Under Conditions, define when the alert should trigger, e.g., if the error occurs more than 3 times within 5 minutes.

3. Alert Settings:

Now you’ll want to set up the alert evaluation interval and conditions for triggering the alert (e.g., if the query returns results above a certain threshold).

Here’s an example: Trigger an alert if the number of errors exceeds 5 within 5 minutes:

count_over_time({job="ci_cd", level="error"} |= "build failed"[5m]) > 5

4. Set Alert Notifications:

You can choose where you want the alert to be sent (like to Slack, email, or PagerDuty). And Grafana can be integrated with these systems to send real-time alerts to the right team members.

Example alert query for test failures:

count_over_time({job="ci_cd", stage="test"} |= "test failed"[5m]) > 3

This query triggers an alert if more than 3 test failures are logged within the last 5 minutes.

Kibana Query Language Deep Dive for CI/CD Contexts

Kibana Query Language (KQL) is a powerful tool for searching and filtering logs within Elasticsearch, and it becomes especially useful for debugging CI/CD pipelines.

Basic Query Syntax:

• Field:

    textCopyEditfieldname:value
  

  Example: status: "failure"

• Wildcard: Use * to match any number of characters:

    textCopyEditmessage: "test*"
  

• Range Queries: To search for logs within a specific time frame:

    textCopyEdittimestamp:[2023-05-01 TO 2023-05-15]
  

• Boolean Queries: Combine queries using AND, OR, and NOT:

    textCopyEditstatus: "failure" AND build_id: "12345"
  

Time-Based Queries:

Since CI/CD logs are often tied to time-sensitive operations (builds, deployments), KQL allows you to filter logs by time:

textCopyEdit@timestamp:[now-1d TO now]

Nested Queries (For Complex Pipelines):

CI/CD logs can have nested or multi-level structures (for example, logs within containers). You can query these nested fields:

textCopyEditpipeline.logs.message: "build failed"

Aggregations and Grouping:

You can aggregate logs based on certain fields to identify trends or recurring issues:

textCopyEditterms aggregation on "status" field

This helps identify the most common failure statuses in your pipeline.

Field-Specific Filtering:

When debugging specific components like a build tool or deployment step, you can filter by those component-specific fields:

textCopyEditbuild_tool: "Jenkins" AND status: "failure"

Creating Saved Searches for Recurring Issues

Once you’ve built queries that help you identify common issues in your CI/CD pipeline, you can save them in Kibana for future use.

1. Create a Saved Search:

Run your desired query in the Kibana Discover tab. Click on the “Save” button and give it a meaningful name, such as "Failed Builds - Last Week". You can add filters and customize the time range to match your typical issue patterns.

2. Use Filters to Pinpoint Recurring Problems:

Create saved searches that focus on specific recurring issues like:

• Build failures based on a specific tool or version.

• Test failures within a particular module or set of tests.

Example search for “flaky tests”:

textCopyEdittest_status: "failed" AND error_message: "*timeout*"

3. Saving Multiple Variations:

You can save multiple variations of queries based on different error types or CI/CD tools:

• Failed Jobs: status: "failure"

• Test Failures in Build: log_type: "test" AND status: "failure"

• Resource Constraints: error_message: "*memory*"

These saved searches will allow you to quickly troubleshoot specific issues that occur frequently.

Building Visualizations to Spot Patterns Over Time

Once you have saved searches, Kibana allows you to create visualizations from your data, making it easier to spot trends, anomalies, or patterns over time.

1. Create a Visualization:

Go to the Visualize tab in Kibana. Select the appropriate visualization type. Common visualizations for debugging CI/CD pipelines include:

• Line Chart: Track build failure rates over time.

• Bar Chart: Show the number of failures per CI tool or service.

• Pie Chart: Breakdown of failure reasons (for example, compilation errors, test failures, resource constraints).

2. Track Failure Trends Over Time:

Create a line chart to track build failures over a given period:

• X-Axis: Time (for example, daily or weekly).

• Y-Axis: Count of build failures.

• Aggregation: Date histogram with @timestamp field.

This will help you visualize how build failures are trending, making it easier to identify recurring issues or spikes in failures.

3. Monitor Failure Types by CI Tool:

Create a bar chart that shows the number of failures broken down by CI tool:

• X-Axis: CI tool (Jenkins, GitHub Actions, GitLab, and so on).

• Y-Axis: Count of failures.

• Aggregation: Terms aggregation on the ci_tool field.

This visualization helps identify which CI tool is experiencing the most failures and focus troubleshooting efforts there.

4. Visualize Error Messages by Frequency:

You can visualize which error messages appear most frequently, helping you understand what might be causing recurring issues:

• X-Axis: Error message type.

• Y-Axis: Count of occurrences.

• Aggregation: Terms aggregation on the error_message field.

5. Dashboard for Holistic Monitoring:

Create a dashboard that brings together multiple visualizations. You can have one graph for failure trends, another for failure types (bar chart), and a pie chart showing the percentage of failures caused by different issues. This dashboard gives you a holistic view of your pipeline's health.

Advanced Visualization Techniques:

There are various advanced techniques you can use to dig further into your data.

• Heatmaps: Use heatmaps to spot time-based anomalies in build durations or test failures.

• Anomaly Detection: Kibana has built-in anomaly detection that can be applied to log data to automatically detect patterns that deviate from the norm. This is especially useful for catching rare or unexpected errors in your CI/CD pipeline.

  Example for anomaly detection:

    textCopyEditfield: duration
    aggregation: average
    anomaly detection model: "baseline"
  

How to Set Up Prometheus Metrics Alongside Your Logs

To fully understand your CI/CD pipeline's health and performance, combining metrics and logs is essential. Prometheus is an excellent tool for capturing time-series metrics, and it works seamlessly with Grafana and Loki (or any log aggregation system).

How to Set Up Prometheus for CI/CD Metrics Collection:

1. Install Prometheus:

You can install Prometheus using Docker or Kubernetes for easy deployment.

For Docker-based installation:

docker run -d -p 9090:9090 --name prometheus prom/prometheus

2. Configure Prometheus to Scrape Metrics:

Prometheus needs to be configured to scrape metrics from your CI/CD services.

Edit the prometheus.yml file:

scrape_configs:
  - job_name: 'ci_cd_metrics'
    static_configs:
      - targets: ['localhost:8080', 'localhost:9091']

3. Instrument Your CI/CD Services:

To expose metrics, you need to integrate Prometheus client libraries into your CI/CD services.

For example, to expose build metrics from a Jenkins job, use the Prometheus plugin for Jenkins. In GitHub Actions, you can use Prometheus to expose job metrics.

4. Expose Metrics Endpoint:

You’ll want to make sure your services expose a /metrics endpoint that Prometheus can scrape. For example, use Prometheus client libraries in your application to expose this endpoint.

Troubleshooting Prometheus Setup Issues

If Prometheus fails to start or scrape metrics, here are some things that might be going wrong:

1. Container Crashes: Check logs with docker logs prometheus. Look for errors like “port already in use” (for example, 9090) or configuration parsing issues.

   • Fix: Change the port in docker run (for example, -p 9091:9090) or correct the prometheus.yml file syntax.

2. Metrics Not Scraped: Verify targets are reachable using docker logs prometheus or test with curl http://localhost:9090/targets. Check prometheus.yml for correct endpoints.

   • Fix: Update targets in scrape_configs (for example, localhost:8080) to match your CI/CD service’s metrics endpoint.

3. Resource Constraints: Monitor usage with docker stats or top on the host.

   • Fix: Ensure at least 4GB RAM and 10GB disk space. Increase storage retention or reduce scrape frequency in prometheus.yml if needed.

How to Create Grafana Dashboards That Combine Metrics and Logs

Once Prometheus is collecting metrics, the next step is to visualize and correlate them in Grafana.

How to Integrate Prometheus with Grafana:

First, you’ll need to install Grafana. You can use Docker or Kubernetes for quick deployment:

docker run -d -p 3000:3000 --name grafana grafana/grafana

Next, configure Grafana to use Prometheus as a data source. To do this, log in to Grafana (localhost:3000 by default). Go to Configuration > Data Sources > Add Data Source > Choose Prometheus. Enter your Prometheus server URL (for example, http://localhost:9090) and click Save & Test.

Now it’s time to build a unified dashboard. To do this, create a new dashboard in Grafana that combines both logs (Loki) and metrics (Prometheus).

Add a panel with Prometheus data queries to visualize pipeline metrics like build success rate, deployment duration, and failure count. Use the Graph visualization type for time-series data and Stat for quick summary metrics.

Finally, in the same Grafana dashboard, add panels for logs (from Loki or any other logging system). Use the Logs panel to visualize log data and link them with the relevant Prometheus metrics by using time-based correlations.

Example: If a spike in CPU usage is detected (Prometheus metric), the logs panel could show related logs, like errors or failed build jobs.

How to Use Exemplars to Jump from Metrics to Relevant Logs

Exemplars are an advanced feature in Prometheus that allow you to connect metric data with logs and traces. Grafana supports this feature, and it can be incredibly helpful when investigating issues.

How to Set Up Exemplars in Prometheus:

1. Enable Exemplars in Your Application:

Exemplars are essentially traces embedded into your metrics. To use them, you’ll need to make sure your application is instrumented to send exemplar data alongside your metrics.

Many libraries support adding exemplars to Prometheus metrics, such as prom-client (Node.js) and prometheus-net (C#).

Here’s an example in Node.js:

// Demonstrates adding an exemplar to a Prometheus metric for linking to logs or traces.
const promClient = require('prom-client');

// Creates a counter metric to track failed CI/CD builds.
const counter = new promClient.Counter({
  name: 'ci_cd_failed_builds_total',  // Metric name for failed builds.
  help: 'Total number of failed builds',  // Description of the metric.
});

// Increments the counter with an exemplar for tracing.
counter.inc({ exemplar: 'build_failed' });

2. Enable Exemplars in Prometheus Config:

Make sure your Prometheus server is configured to store and expose exemplars. Exemplars are typically included with histogram or summary metrics, so make sure you’ve configured them correctly.

3. Visualizing Exemplars in Grafana:

In Grafana, when you query Prometheus for metrics with exemplars, Grafana will show the linked logs or traces when you hover over a metric.

Use the Exemplar option in Grafana panels to quickly access logs from specific metrics.

For example, if you have a build_failure_total metric and you detect a failure in your pipeline, you can click on the failure metric in Grafana and instantly view the relevant logs for that specific failure using the exemplars.

How to Diagnose and Fix Common CI/CD Problems

CI/CD pipelines often encounter issues like build failures, dependency problems, and flaky tests that can disrupt development workflows. This section provides practical strategies to diagnose and resolve these common problems using log analysis and systematic debugging techniques, helping you restore pipeline stability quickly.

Strategy 1: Systematically Debug Build Failures

Build failures are a frequent CI/CD challenge, often stemming from errors in code, tests, or configurations. Systematically debugging these issues involves analyzing logs to pinpoint root causes, using the following approaches.

Identifying Patterns in Compiler and Test Output

When debugging build failures, you need to first examine the logs from the compiler and test outputs. Let’s go over some key strategies.

1. Check for Specific Error Messages:

There are a few common types of error messages you might get. They are:

• Syntax errors: Look for lines indicating that there's a mismatch in syntax, such as missing semicolons, undeclared variables, or incorrect function calls.

• Linker errors: These often occur when the required libraries or dependencies are not found. You'll typically see errors like undefined reference or symbol not found.

• Build tool errors: If you are using build systems like Maven, Gradle, or MSBuild, their logs will give specific error codes or missing configurations.

2. Look for Common Error Patterns:

Often, failed builds repeat the same error or pattern across multiple runs. Check logs for recurring terms or errors that point to specific modules or functions. And remember that grouping similar issues can help you identify the root cause faster.

3. Use Regular Expressions for Log Filtering:

You can use regular expressions to search for keywords in the logs that match common failure patterns (for example, "error", "failed", "exception", "out of memory"). This will help you filter out unrelated messages and focus on the failures.

As an example:

• If the build fails with an "Out of Memory" error, search for any memory allocation issues or settings that can be increased.

• If test failures are related to specific modules, inspect those modules for recent changes or dependency issues.

Strategy 2: Troubleshooting Dependency Issues with Log Analysis

Dependency issues are common in build failures, especially in complex CI/CD pipelines with multiple modules or services. To resolve these issues, consider the following:

1. Check for Missing or Outdated Dependencies:

Start by reviewing the build tool’s output to check for messages related to missing dependencies (for example, dependency not found, version conflict).

Many build tools (like Maven, npm, or .NET) will include specific error messages when a dependency is missing or incompatible.

2. Inspect Dependency Resolution Logs:

Some build tools provide detailed logs showing how dependencies were resolved (for example, the version of a library that was used). These logs can show you if there’s a version mismatch.

Make sure that your package.json (for JavaScript projects), pom.xml (for Java), or csproj (for C#) files are correctly defined with compatible versions.

3. Verify Network Connectivity:

CI/CD tools sometimes fail to fetch dependencies due to network issues (for example, proxy settings, repository access). Look for any errors indicating that a repository couldn’t be reached.

4. Log Example:

If a Java project fails with Could not find artifact, it's likely a dependency missing or inaccessible. Check the repository URL or if the artifact exists in your Maven repo.

5. Resolve Version Conflicts:

Version conflicts occur when different dependencies require incompatible versions of the same library. This is especially true in Java (with Maven/Gradle) and .NET projects. Consider using tools to resolve version conflicts automatically or define compatible versions manually.

Fixing Flaky Tests Based on Historical Log Data

Note: Issues like container crashes, logs not ingested, or resource constraints here may resemble those in other sections. These are common across CI/CD services and processes, but each section offers unique context to avoid redundancy.

Flaky tests – that is, those that pass sometimes and fail at other times – are common in CI/CD pipelines, and they can be frustrating. Let’s discuss some strategies for how you can tackle them:

1. Analyze Test Logs Over Time:

Review historical logs to identify patterns in when the test fails. Look for timing issues, resource limits, or external dependencies that could affect test reliability.

For example, if a test intermittently fails after a certain amount of time or only during specific pipeline stages, it could indicate resource exhaustion or race conditions.

2. Check Test Dependencies:

Often, flaky tests are dependent on external services or resources (for example, databases, APIs, file systems). Check if these services are consistently available and properly mocked during test execution.

Logs that mention failed connections to external services or unstable environments can give you insights into potential issues with dependencies.

3. Run Tests with Increased Logging:

Increase the verbosity of test logs to capture more information about the failures. This can help you detect why tests fail in certain conditions.

For example, adding debug logs inside tests can provide more context on the state of the application when the failure occurs.

4. Time of Day Issues:

Some flaky tests may fail during peak usage times, especially if they rely on shared resources. Look for patterns that correlate with resource contention (for example, database locks, API rate limits).

Logs showing high CPU or memory usage can indicate that resource constraints are affecting the stability of your tests.

5. Implement Retry Logic for Flaky Tests:

To mitigate the effects of flaky tests, implement automatic retries for tests that fail intermittently. This can help reduce the noise in your CI/CD pipeline while you investigate the root causes.

For example, if a database connection test fails intermittently, you may want to inspect database logs for signs of timeouts or connection pool exhaustion.

How to Resolve Deployment Pipeline Failures

Deployment pipeline failures can stem from several sources, and diagnosing them requires a systematic approach using logs and available observability tools. Below, we will outline the common patterns in logs that indicate resource constraints, permission/authentication issues, and configuration drift between environments.

Log Patterns That Indicate Resource Constraints

Resource constraints are a common cause of pipeline failures. These can include CPU limits, memory usage, or disk space running out. Here's how to recognize these patterns:

Key Indicators in Logs:

• Memory Issues: Look for messages like "out of memory", "memory limit exceeded", or "OOM killed" in your logs. Here’s an example in Kubernetes logs:

pod has been OOMKilled

• CPU Limits: Watch for logs showing that a process exceeded CPU limits or was throttled. Here’s an example:

process 'foo' hit CPU limit, throttling at 100%

• Disk Space: Logs may show file write errors or messages about a disk being full. Here’s an example:

Unable to write to file, disk space is full.

You can resolve the memory issues by increasing the allocated memory for your containers, VM, or cloud instances.

You can resolve the CPU issues by adjusting CPU limits or scaling your infrastructure to add more resources.

And finally, you can resolve disk space issues by cleaning up unused files or increasing disk capacity on the server/container.

Identify Permission and Authentication Issues

Permission and authentication issues often result in pipeline failures due to a lack of access to necessary resources or services. These issues might occur when you’re trying to access databases, deploy to cloud services, or authenticate third-party APIs.

There are some key indicators in the logs that you can look out for:

1. Authentication Failures:

Look for messages related to failed logins, incorrect credentials, or invalid tokens.

Here’s an example:

Authentication failed for user 'admin'

Invalid API token provided.

2. Permission Denied:

Logs may indicate that the CI/CD pipeline lacks the permissions to perform a certain action.

Here’s an example:

Access denied for /path/to/deployment/target

Unauthorized request to cloud service.

How to resolve these errors:

• Credentials: Ensure the credentials (API keys, access tokens, SSH keys) used in the pipeline are up-to-date and correctly configured.

• Permissions: Review and update the role-based access control (RBAC) settings for the service account running the pipeline to ensure it has the necessary permissions.

• Secrets Management: Use tools like Vault, AWS Secrets Manager, or Azure Key Vault to securely manage secrets and credentials.

Troubleshooting Configuration Drift Between Environments

Configuration drift occurs when different environments (like development, staging, production) are not synchronized. This can lead to inconsistent behavior during deployments, and often results in failures in one environment but not in others.

Look out for these key indicators in the logs:

1. Mismatch in Environment Variables:

If you’re using environment variables, check for discrepancies across different stages. For example:

Environment variable DATABASE_URL not found in production

2. Dependency Versions:

Mismatched versions of dependencies between environments can cause unexpected issues.

Here’s an example:

Error: Dependency 'libxyz' version mismatch between environments

3. Service Configuration:

Look for configuration-related errors that might not be present in a development environment but occur in production.

Here’s an example:

Error: Invalid config in 'production-config.yaml'

How to resolve these errors:

• Use Infrastructure as Code (IaC): Tools like Terraform, Ansible, or CloudFormation can help ensure that environments are provisioned consistently.

• Automated Configuration Management: Use CI/CD pipeline steps to automate environment setup to avoid manual changes that can cause drift.

• Environment Consistency Checks: Implement checks to compare configurations and dependencies across environments before deployment.

  • Example: You can add a pre-deployment stage to run a script that compares environment variables, configurations, and dependency versions between staging and production.

• Configuration Management Tools: Use configuration management tools like Chef, Puppet, or SaltStack to maintain consistent configurations across environments.

How to Debug Container-Based Deployment Issues

Debugging container-based deployment issues requires specialized tools and techniques to trace errors in containerized environments. Below are strategies to efficiently collect logs, diagnose failures, and use ephemeral containers for investigation.

Collecting and Analyzing Container Logs Effectively

Container logs are essential for troubleshooting issues, and effective collection and analysis can significantly speed up the debugging process.

Here’s how you can collect container logs:

1. Docker Logs:

You can use Docker’s logs command to view logs of a specific container:

docker logs <container_name_or_id>

If your container uses a logging driver (like json-file or fluentd), ensure that logs are being written to an accessible location.

2. Kubernetes Logs:

For Kubernetes-managed containers, use kubectl to access pod logs:

kubectl logs <pod_name>

To view logs for all containers in a pod:

kubectl logs <pod_name> --all-containers=true

3. Log Aggregation:

You can integrate with centralized logging systems (like, Grafana Loki, Elastic Stack). You can also use Fluentd or Logstash as log shippers for forwarding logs from containers to a logging backend.

Analyzing Logs:

1. Filter and Search Logs:

Use grep to filter logs for specific error messages or patterns:

docker logs <container_name> | grep "ERROR"

In Kubernetes, you can combine kubectl with grep or other tools for advanced filtering.

2. Log Contextualization:

Include metadata in your logs (for example, container ID, environment, timestamps) for easier debugging. Ensure logs are structured in formats like JSON to allow for better querying and filtering.

How to Diagnose Image Pull and Networking Failures

Container deployment failures often stem from issues related to image pulling or network connectivity. Here’s how to troubleshoot these problems:

Image Pull Failures:

There are some common issues you might see, such as:

• Authentication failures: If the container registry requires authentication, ensure your credentials (username/password or tokens) are correct.

• Network connectivity: Check if the container can access the registry endpoint. Often, firewalls or DNS issues block the image pull.

• Image not found: Verify the image name and tag are correct. Use docker pull to manually pull the image to see if the issue is specific to the deployment process.

There are various ways to diagnose them:

For Docker, use:

docker pull <image_name>

This will output the specific error message if the image pull fails.

For Kubernetes, check the event logs for the pod:

kubectl describe pod <pod_name>

Look for the Failed status under "Events" for information about why the image pull failed (for example, wrong credentials or tag). If the issue is with the registry authentication, configure the Kubernetes imagePullSecrets or Docker's credentials to ensure the correct access.

Networking Failures:

Some common issues you may encounter are:

• DNS resolution problems: Containers may fail to resolve hostnames if DNS configurations are incorrect.

• Network policies and firewall rules: Network policies or firewalls may block necessary ports.

• Inter-container communication: If containers need to talk to each other, ensure they’re on the same network or subnet.

Again, there are various ways to diagnose these issues:

For Docker networking:

You can do this to view all Docker networks:

docker network ls

You can also inspect the network of your container like this:

docker network inspect <network_name>

Check if the container is correctly attached to the network and if necessary ports are exposed.

For Kubernetes Networking:

You can use kubectl to check network policies:

kubectl get networkpolicies

You can also check the pod’s network settings like this:

kubectl describe pod <pod_name> | grep -i "Network"

Testing Connectivity Inside Containers:

For Docker, exec into the container and test:

docker exec -it <container_id> /bin/bash
ping <hostname_or_ip>
curl http://<service_address>:<port>

In Kubernetes, use kubectl exec to access the pod and test connectivity:

kubectl exec -it <pod_name> -- /bin/bash

How to Use Ephemeral Debug Containers for Investigation

Ephemeral debug containers are short-lived containers that help investigate issues in a running environment without altering the main application container.

What are Ephemeral Debug Containers?

Ephemeral debug containers allow you to run diagnostic commands (like shell access, ping, or curl) in the same network environment as the failing application container, without modifying the application itself.

How to Set Up Ephemeral Containers in Docker:

1. Use the docker run Command:

You can create a new container for debugging by running a container with the same network settings as the failing container:

docker run -it --network container:<container_name_or_id> --entrypoint /bin/bash <debug_image>

This command runs an interactive shell inside the debug container using the same network as the target container.

Ephemeral Containers in Kubernetes:

Kubernetes allows you to inject an ephemeral debug container into a running pod. You can add a temporary debug container to your pod using the following command:

kubectl debug <pod_name> -it --image=<debug_image> --target=<container_name>

This command will run a new container in the same pod as the target container, allowing you to run diagnostic commands.

Example use cases are investigating file systems, running network diagnostics, checking configuration files, and so on.

These debug containers are meant to be temporary and can be discarded after the issue is resolved.

How to Implement Advanced Debugging Techniques

This section covers advanced methods to diagnose complex CI/CD pipeline issues that standard log analysis might miss. We’ll explore distributed tracing to track requests across multiple services and combine traces with logs and metrics for deeper insights.

These techniques are designed to work within budget constraints, ensuring effective debugging for your CI/CD workflows.

Choosing a Tracing Backend for CI/CD

Distributed tracing enables you to monitor a request’s path through various services in your CI/CD pipeline, such as from a build step to a deployment, identifying delays or failures. Choosing a tracing backend involves selecting a tool to store and analyze these trace data. Below, we compare Jaeger, Tempo, and hosted solutions for distributed tracing.

When to choose each one:

• Jaeger: Ideal for quick, local tracing setups with minimal overhead.

• Tempo: Best for teams already using Grafana Loki/Prometheus for unified observability.

• Hosted Solutions: Suited for large-scale pipelines needing managed scalability.

How to Set Up Distributed Tracing on a Budget

Distributed tracing is crucial for debugging and observing complex, multi-step operations across services. It allows you to follow requests as they propagate through different services and components of your pipeline. Implementing this on a budget can still provide valuable insights.

How to Use OpenTelemetry with Free Backends

OpenTelemetry is an open-source framework that enables you to collect, process, and export telemetry data like traces and metrics. It supports multiple backends, and we’ll focus on using free, budget-friendly backends for trace storage and analysis.

1. Install OpenTelemetry Collector:

OpenTelemetry provides an agent (collector) that collects traces and metrics from your application and sends them to a backend.

To install the OpenTelemetry Collector, download the binary for your OS or use Docker to deploy it:

docker pull otel/opentelemetry-collector:latest

Then run the OpenTelemetry Collector in Docker with a configuration file:

docker run -d --name opentelemetry-collector -p 55680:55680 -p 14250:14250 otel/opentelemetry-collector

2. Configure OpenTelemetry to Export to Free Backends:

There are a few popular free backends you can use for distributed tracing, like Jaeger and Prometheus + Tempo. Let’s see how to use both here.

We’ll start with Jaeger, an open-source tracing backend. It’s highly scalable and works well with OpenTelemetry.

You can use the Docker version for easy deployment:

docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 5775:5775 -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14250:14250 -p 14268:14268 -p 14250:14250 -p 9431:9431 jaegertracing/all-in-one:1.30

Alternatively, you can use hosted services like Lightstep, AWS X-Ray, or Honeycomb for cloud-native environments.

Now let’s see how to use Prometheus + Tempo for logs and metrics correlation.

Tempo is a distributed tracing backend built by Grafana that integrates well with other Grafana tools (Loki and Prometheus).

You can install Tempo using Docker:

docker run -d --name tempo -p 14268:14268 grafana/tempo:latest

3. Instrument Your Code with OpenTelemetry SDK:

For Python/Node.js/Java/Go applications, you can install the appropriate OpenTelemetry SDK and start tracing.

Here’s a Python example:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation

And a Node.js example:

npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/instrumentation

And one in Java:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.0.0</version>
</dependency>

After installation, you can use the OpenTelemetry SDK to instrument the application and start collecting traces for HTTP requests, database queries, and other pipeline interactions.

4. Send Data to the Collector:

You can configure the SDK to send trace data to your OpenTelemetry Collector, which will then forward it to your backend (Jaeger, Tempo, and so on). Here’s an example for Python:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchExportSpanProcessor

trace.set_tracer_provider(TracerProvider())
exporter = OTLPSpanExporter(endpoint="http://localhost:55680")
processor = BatchExportSpanProcessor(exporter)
trace.get_tracer_provider().add_span_processor(processor)

If traces aren’t appearing, several issues might be occurring:

1. Collector fails to start: Check logs with docker logs otel-collector. Look for errors like “port conflict” or “invalid config.”

   • Fix: Change ports (for example, 55681:55680) or verify the config file.

2. No traces in Jaeger: Ensure the collector is sending data to Jaeger (http://localhost:14250). Test with curl http://localhost:55680.

   • Fix: Update the exporter endpoint in your SDK configuration.

3. Resource constraints: Monitor usage with docker stats.

   • Fix: Allocate at least 2GB RAM and 10GB disk space for the collector and backend.

Correlating Traces with Logs and Metrics

Combining traces with logs and metrics provides a holistic view of your pipeline’s operations, allowing you to pinpoint the root cause of issues more effectively.

OpenTelemetry and Grafana allow you to link traces, logs, and metrics into a unified view.

Let’s see how you can do this now.

1. Link Logs and Traces Using Correlation IDs:

When generating logs, include trace and span IDs in the log entries. This allows you to correlate logs with specific trace requests.

Here’s an example:

{
  "timestamp": "2025-05-10T12:00:00Z",
  "level": "error",
  "message": "Build failure",
  "trace_id": "1234567890abcdef",
  "span_id": "0987654321abcdef"
}

2. Integrating Logs (Loki) with Traces (Jaeger/Tempo) in Grafana:

Grafana can integrate traces from Jaeger or Tempo and correlate them with logs from Loki.

To do this:

1. Set up Loki and Tempo in Grafana.

2. In Grafana’s Explore view, you can search logs and traces side-by-side.

3. Create dashboards that show metrics, logs, and traces for a complete view of a request flow.

3. Using Prometheus Metrics with Traces:

Prometheus provides metrics that can be correlated with traces. For example, you can use exemplars in Prometheus to link specific metric data to trace data.

Example: If you have a high error rate in your build step, you can correlate this with trace data to identify which requests failed.

Creating Trace Visualizations for Complex Pipeline Operations

You can visualize traces with Jaeger or Tempo.

To do this in Jaeger:

Once your traces are in Jaeger, you can access the Jaeger UI (http://localhost:16686 by default) and use the search functionality to explore traces based on service name, trace ID, or specific operations.

Jaeger allows you to create custom dashboards to visualize the latency, throughput, and errors of requests across services.

To do this in Tempo (Grafana Integration):

Tempo integrates with Grafana, where you can create dashboards that visualize trace data from your pipeline.

Create a Grafana dashboard:

1. Add Tempo as a data source in Grafana.

2. Use the "Trace" panel to query and visualize traces.

3. Combine trace visualizations with metrics (from Prometheus) and logs (from Loki) to get a unified view of your pipeline.

A typical trace visualization dashboard could show the duration of each step in your pipeline (build, test, deploy) and highlight where delays or errors occur, such as slow database queries or flaky tests.

Troubleshooting Tempo Setup Issues

If Tempo fails to collect or display traces:

1. Container fails to start: Check logs with docker logs tempo. Look for errors like “port already in use” (for example, 14268) or “storage backend unavailable.”

   • Fix: Change ports in the Docker command (for example, -p 14269:14268) or ensure the storage directory (for example, /tmp/tempo) exists and is writable.

2. No traces in Tempo: Verify the OpenTelemetry Collector is sending traces to Tempo’s endpoint (http://localhost:14268). Test connectivity with curl http://localhost:14268.

   • Fix: Update the collector’s exporter configuration to point to the correct Tempo endpoint, and ensure no firewalls are blocking the connection.

3. Resource constraints: Monitor usage with docker stats or top on the host.

   • Fix: Allocate at least 2GB RAM and 10GB disk space for Tempo, as tracing data can grow quickly with high-volume pipelines.

This bar chart displays the average latency (in milliseconds) for key stages of a CI/CD pipeline in May 2025. The Build stage averages around 1,200 ms (blue), the Test stage around 800 ms (yellow), and the Deploy stage around 1,500 ms (pink), highlighting that deployment is the most time-intensive step.

How to Build Comprehensive Debugging Dashboards

This section explains how to create Grafana dashboards to troubleshoot CI/CD pipeline issues effectively. We’ll focus on setting up visualizations for key metrics, logs, and system resources to identify problems like build failures or resource bottlenecks, using budget-friendly tools to keep your observability stack lean and actionable.

Designing Grafana Dashboards Specifically for Troubleshooting

Step 1: Understand the Key Metrics and Logs to Monitor

When designing a Grafana dashboard for debugging, you should focus on metrics and logs that help identify issues in the pipeline. These could include:

• Build failures: Errors during build processes (compilation, test failures).

• Deployment failures: Issues in deployment, such as failed jobs, resource limitations, or misconfigurations.

• Container logs: Information about container status and logs (if using containers in your pipeline).

• System resource usage: CPU, memory, and disk usage that may lead to performance bottlenecks.

• CI/CD-specific metrics: Number of successful vs. failed pipeline runs, job duration, job queue times.

Step 2: Set Up Data Sources

To start building the dashboard, you’ll need to set up your data sources in Grafana. First, connect your Prometheus instance for collecting metrics. To do this, go to Configuration > Data Sources in Grafana. Then just add Prometheus as a data source and enter the URL (for example, http://localhost:9090).

Next, you need to connect your Loki instance for logs. So go ahead and add Loki as a data source by specifying the URL (for example, http://localhost:3100).

Note that if you're using other sources like InfluxDB or Elasticsearch, you’ll need to make sure that they’re properly connected as data sources.

Step 3: Create Panels and Visualizations

Now that your data sources are connected, you can start building your dashboard with the following panels:

• Build Status Panel:

  • Create a stat panel or gauge panel to show the success/failure ratio of pipeline runs.

  • Query Prometheus or Loki for data like build status (success or failure), number of errors, and job durations.

• Error Breakdown Panel:

  • Use a pie chart to visualize the types of errors (for example, build, deployment, or system resource failures).

  • Query the logs in Loki to break down error types based on the CI tool (for example, Jenkins, GitHub Actions).

• Resource Utilization Panel:

  • Use time series graphs to monitor CPU, memory, and disk usage over time, especially for resource-heavy builds or deployments.

• Job Duration Panel:

  • Use bar charts or line graphs to track the average duration of jobs over time. Set thresholds for warning signs if a job takes longer than expected.

Troubleshooting Grafana Dashboard Issues

If Grafana dashboards fail to display data or show errors, you might be having one of these issues:

1. Missing data sources: If metrics, logs, or traces aren’t appearing, verify data source connections in Grafana (for example, Prometheus, Loki, Tempo). Check under Configuration > Data Sources.

   • Fix: Ensure the data source URLs are correct (for example, http://localhost:9090 for Prometheus) and test the connection. Re-add the data source if needed.

2. Incorrect Trace IDs: If trace visualizations (for example, Tempo panels) show no data, confirm that trace IDs in logs match those in Tempo. Use a query like {job="ci_cd"} | json | trace_id="1234567890abcdef" in Loki to cross-check.

   • Fix: Ensure your application logs include trace and span IDs, and verify the OpenTelemetry SDK is correctly instrumented to send traces to Tempo.

3. Resource Constraints: Monitor Grafana’s resource usage with docker stats if running in a container, or top on the host.

   • Fix: Allocate at least 4GB RAM and 10GB disk space for Grafana, especially when rendering complex dashboards with multiple data sources.

How to Set Up Drill-Down Paths from High-Level to Detailed Views

Step 1: Create High-Level Overview Panel

At the top of the dashboard, include a high-level overview panel that summarizes the overall status of the pipeline. This could be:

• Success/Failure Count: A simple stat panel showing the count of successful vs. failed runs.

• Pipeline Health Status: Display an overall health check of your pipeline using color-coded indicators (green for healthy, red for issues).

Step 2: Set Up Drill-Down Links

To allow users to drill down from high-level information to detailed views:

1. Link to detailed build information:

You can create a time series graph that shows build job durations. Add a link to a detailed log view when clicking on a failed job.

For example, when clicking a failed build, you can link to a detailed panel or a separate dashboard that shows the logs and error messages related to that specific run.

2. Link to Logs in Loki:

You can use Loki's LogQL queries to set up a drill-down path. When users click on an error type or a specific job name, it should automatically filter logs for that job or error type.

You can set up drill-down interactions using Dashboard Links in Grafana. In the panel settings, under Links, specify the link to another dashboard that shows detailed logs filtered by the job name or failure type.

Step 3: Implement Time Range Filters

To enhance drill-down functionality, you can add a time range filter to allow users to adjust the time window for both logs and metrics. This enables them to zoom in on a specific time frame where failures occurred.

How to Create Shared Dashboards for Team Troubleshooting

Step 1: Share Your Dashboard

Once your dashboard is designed, you can share it with your team for collaborative troubleshooting:

First, you’ll want to make sure that the correct permissions are set up for your team. You can define specific roles in Grafana with access to the dashboard. Go to Dashboard Settings > Permissions, and grant view or edit access to users or teams.

Next, you can directly share a link to the dashboard with your team members. Use the Share option in the top-right corner of the dashboard, which provides a direct URL and also options to embed the dashboard into other tools (for example, Slack, email).

You can also use template variables to allow users to filter and adjust the dashboard for different pipeline runs or environments. For example, add a variable for build_id, job_name, or branch_name that allows users to select specific builds or branches for more granular troubleshooting.

Step 2: Set Up Alerting

To ensure your team is notified of any pipeline failures, you can set up alerting rules. There are a few important ones you’ll want to set up.

First, create alerts for critical issues, like when a pipeline fails or exceeds expected resource usage. This could be for things like build time exceeding a threshold or failure of a deployment stage.

Grafana can send alerts via various channels such as Slack, email, or webhook.

You can also integrate your dashboards with tools like Slack or Teams for real-time notifications and collaboration. Set up automated messages for your team when the dashboard indicates an issue.

How to Create Automated Diagnostic Tools

Building Scripts that Collect Relevant Logs During Failures

To automate log collection during failures, you need scripts that can capture logs from different CI/CD stages and services as soon as a failure is detected. Here are the steps you can follow to do this:

1. Write Failure Detection Script:

You can leverage the exit status codes of your CI/CD tools to detect failures. For example, in GitLab CI/CD or GitHub Actions, you can check if the last command failed by inspecting $? in Unix-based systems.

# Example for GitLab CI/CD
if [ $? -ne 0 ]; then
    echo "Failure detected, collecting logs..."
    # Custom log collection script call
    ./collect_logs.sh
fi

2. Log Collection Script (collect_logs.sh):

The script should collect relevant logs, system metrics, and trace information. For instance:

#!/bin/bash
LOG_DIR="/path/to/logs"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="${LOG_DIR}/backup/${TIMESTAMP}"
mkdir -p $BACKUP_DIR

# Collect logs from CI/CD agents, containers, or system logs
cp /var/log/ci_cd/*.log $BACKUP_DIR/
cp /path/to/docker_logs/*.log $BACKUP_DIR/
# Collect metrics or traces from monitoring systems if needed

3. Use CI/CD Artifacts:

For platforms like GitLab, GitHub Actions, or Jenkins, you can upload logs as artifacts for further investigation. Configure these platforms to save logs in case of a failure.

Here’s an example for GitHub Actions:

steps:
  - name: Run Tests
    run: |
      npm run test
  - name: Upload logs if test fails
    if: failure()
    uses: actions/upload-artifact@v2
    with:
      name: test-logs
      path: /path/to/test/logs

4. Centralized Logging:

Instead of manually collecting logs, you can centralize log storage using logging systems like Grafana Loki, ELK stack, or even cloud-based solutions. This will ensure that logs are accessible even if they are overwritten or lost on individual systems.

How to Implement Automatic Analysis of Common Error Patterns

Once logs are collected, you can automate the analysis process by defining common error patterns and automatically searching for them in your logs.

Step 1: Define Error Patterns:

Establish error signatures or patterns that are common in your CI/CD process, such as failed builds due to missing dependencies, permission issues, or network timeouts.

You can use regex or regular expressions to capture these patterns. Here’s an example – define a regex for failed test patterns:

TEST_FAILURE_REGEX=".*FAILURE.*"

Step 2: Create Log Analysis Script:

Next, you can write a script that scans logs for these common patterns. The script could then categorize or flag errors.

Here’s an example using grep to detect failure patterns:

#!/bin/bash
LOG_DIR="/path/to/logs"
ERROR_LOG="${LOG_DIR}/error_patterns.log"
touch $ERROR_LOG

# Define error patterns to search for
ERROR_PATTERNS=("FAILURE" "ERROR" "TIMEOUT")

for PATTERN in "${ERROR_PATTERNS[@]}"; do
    grep -i $PATTERN $LOG_DIR/*.log >> $ERROR_LOG
done

if [ -s $ERROR_LOG ]; then
    echo "Error patterns found, review the log file."
fi

Step 3: Automate Alerting:

Once an error pattern is detected, you can integrate the log analysis script with your alerting system (for example, sending an email or Slack notification).

Here’s an example of sending a Slack notification:

if [ -s $ERROR_LOG ]; then
    curl -X POST -H 'Content-type: application/json' \
         --data '{"text":"Error detected in CI pipeline. Check error log."}' \
         https://hooks.slack.com/services/YOUR_SLACK_WEBHOOK_URL
fi

Step 4: Use Observability Tools for Pattern Recognition:

Leverage observability tools (Grafana Loki, Prometheus) that support log querying and visualization. You can create dashboards that automatically detect anomalies like high failure rates or recurring errors.

Example: Set up a Grafana dashboard with alert rules based on log frequency.

How to Create Self-Healing Pipelines Based on Known Issues

Self-healing pipelines can automatically address issues when they are detected by executing pre-defined corrective actions. Let’s walk through how you can set one up.

Step 1: Define Common Failures and Solutions:

Identify recurring issues (for example, dependency issues, build timeouts, flaky tests) that occur in your pipeline. Then, define self-healing actions to mitigate these issues.

Here’s an example of automatically retrying a failed step if it is a known flaky test:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Run Tests
        run: |
          npm run test
      - name: Retry Tests if Failed
        if: failure() && (steps.tests.outcome == 'failure')
        run: |
          echo "Retrying tests..."
          npm run test

Step 2: Automatic Rollbacks:

Set up a rollback process for failed deployments. For instance, if a deployment to production fails, the pipeline can automatically revert to the last successful build.

Example in GitLab CI/CD:

deploy_production:
  script:
    - ./deploy.sh
  when: on_failure
  retry: 3

Step 3: Build Self-Healing Logic Using Retry Mechanisms:

Implement retry logic for transient issues (like network glitches) that often cause failures.

Example of retrying a step in GitHub Actions:

steps:
  - name: Retry Deployment
    run: |
      attempts=0
      max_attempts=3
      until [ $attempts -ge $max_attempts ]
      do
        deploy_script && break
        attempts=$((attempts+1))
        echo "Attempt $attempts failed. Retrying..."
        sleep 5
      done

Step 4: Automate Corrective Actions for Dependency Issues:

Set up automatic fixes for dependency-related failures, like clearing caches or re-installing dependencies:

if [[ $(cat error.log) =~ "dependency not found" ]]; then
    echo "Dependency issue detected, reinstalling dependencies..."
    npm install
fi

Step 5: Integrate with Self-Healing Services:

For more complex self-healing, you can integrate tools like Ansible, Puppet, or even create custom scripts that auto-patch common configuration issues.

How to Conduct Effective Postmortems Using Logs

Logs are often the single most valuable resource when reconstructing what went wrong in a CI/CD pipeline. Conducting effective postmortems with log data allows teams to extract clear timelines, pinpoint root causes, and define steps to prevent recurrence – all based on concrete evidence.

Extract Timeline and Key Events from the Logs

To accurately understand what happened and when from the info contained in your logs, there’s a straightforward process you can follow.

Step 1: Centralize and Structure Logs:

First, make sure that the logs from all pipeline stages (build, test, deploy) are aggregated in a central place like Grafana Loki, ELK, or OpenSearch.

And you’ll want to use a consistent log format (like structured JSON) that includes timestamps, log levels, pipeline stage identifiers, and correlation/request IDs.

Step 2: Build a Chronological View:

You can use timestamp filters in your log UI (for example, Kibana, Grafana Explore) to isolate logs from the incident timeframe.

Look for key lifecycle events, like:

• Start and completion of pipeline steps

• Status changes (for example, "test failed", "deployment started", "build queued")

• Error messages and warnings

• Retry events or unexpected restarts

Step 3: Extract Logs Programmatically (optional):

Use queries (LogQL, Elasticsearch DSL) to export relevant logs for analysis or inclusion in a post-mortem document.

How to Identify Root Causes Through Log Analysis

To go beyond symptoms and find the real issue, there are various steps you can take.

Start by looking for the first failure. You can filter logs by level=error or use log pattern matching to identify the earliest sign of failure. Then trace backward from the failure using correlation IDs or pipeline step identifiers.

Second, make sure you correlate logs across systems. Match logs across CI/CD tools (like GitHub Actions → Docker logs → Kubernetes logs). You can use shared correlation IDs or job IDs to group logs from related events.

Next, pay attention to intermittent signals. Warnings, retries, or degraded performance preceding the failure may reveal environmental or configuration-related issues.

And finally, check for external dependencies. Look for timeout or connection errors involving third-party services, cloud APIs, or internal infrastructure components.

How to Create Actionable Follow-Ups to Prevent Recurrence

There are various things you can do to turn your findings into meaningful process improvements.

1. Document the Findings Clearly:

Create a structured post-mortem doc that includes:

• Timeline of events with log excerpts

• Immediate trigger and root cause (based on logs)

• Impact summary and affected components

• Screenshots or saved log queries for reference

2. Define Preventive Actions:

Examples include:

• Adding missing alerts or log-based monitors

• Improving log verbosity or adding missing metadata

• Fixing brittle test cases or deployment scripts

• Updating infrastructure limits or retry strategies

3. Assign Ownership and Deadlines:

Each action item should have a responsible owner and a due date. If applicable, create automated tests or guardrails to catch similar issues in the future.

4. Update Runbooks and Incident Playbooks:

Add log patterns, example queries, and resolutions to shared documentation. This ensures the next person facing a similar issue can act faster.

Pro Tip: Automate part of your post-mortem process by tagging logs from failed CI runs, exporting them to a shared location, and pre-generating dashboards or incident reports. This reduces manual effort and increases consistency.

How to Optimize Log Storage and Management

As your CI/CD system grows, logs can become massive, consuming storage and impacting performance. Optimizing log storage helps you make sure that you're retaining what's valuable while staying efficient.

How to Implement Log Rotation and Retention Policies

Without rotation and retention, logs will pile up endlessly, leading to disk space exhaustion and poor performance. You can help prevent this with log rotation.

Log rotation involves creating new log files after a size or time threshold and archiving or deleting old ones.

Linux logrotate tool – Configure /etc/logrotate.d/<your-app>:

/var/log/ci_cd/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0640 root adm
}

This example:

• Rotates daily

• Keeps 7 days of logs

• Compresses old logs to save space

Docker logs rotation – in daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

Retention policies ensure that old logs are automatically deleted based on age or storage usage.

You can set one up in Loki like this:

table_manager:
  retention_deletes_enabled: true
  retention_period: 168h  # 7 days

Or in Elasticsearch, use Index Lifecycle Management (ILM):

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "3d", "max_size": "1gb" }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": { "delete": {} }
      }
    }
  }
}

How to Set Up Log Compaction for Long-Term Storage

Compaction reduces redundancy and keeps only critical log info, which is ideal for long-term audits or analytics.

Compaction Techniques:

There are various different compaction techniques you can try. Here are a couple:

1. Loki (boltdb-shipper mode):

• Uses compaction to merge log chunks and reduce storage.

• Configure in loki-config.yaml:

    schema_config:
      configs:
        - from: 2023-01-01
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
  

• Use a low-retention, high-compaction strategy for archived logs.

2. Elasticsearch:

• Use rollup jobs to reduce resolution of old data.

• Stores summarized logs, for example, hourly counts of similar events.

3. Archive to cheaper storage:

• Move infrequent-access logs to S3 or Azure Blob Storage using lifecycle rules.

How to Balance Observability with Resource Constraints

More logs = more observability, but also more cost and overhead. This means that you need a balance. There are various strategies that can help you achieve this balance:

1. Log at appropriate levels:

   • Avoid excessive debug or trace logs in production.

   • Use info and warn levels judiciously.

   • Only use error or critical for actionable failures.

2. Sample logs:

   • If high-volume pipelines generate repetitive logs, enable log sampling to reduce duplicates.

   • Tools like Vector or Fluent Bit support sampling.

3. Filter out noise:

   • Use log filters to exclude non-critical logs before they reach the central system.

4. Separate hot vs. cold logs:

   • Hot logs: recent, real-time data for active debugging.

   • Cold logs: archived for compliance, stored with lower performance/storage priority.

5. Compress everything:

   • Use gzip/zstd compression for both stored and transmitted logs.

   • Loki, Elasticsearch, and Vector support compression out of the box.

Conclusion

In this handbook, you have built a full-stack observability layer specifically optimized for CI/CD pipelines without breaking your infrastructure budget. You now have the tools and know-how to:

• Deploy Grafana Loki or a lightweight ELK alternative to capture structured logs from all parts of your pipeline.

• Unify and enrich logs across CI/CD tools (for example, GitHub Actions, Jenkins, GitLab) using consistent formats and correlation IDs.

• Use powerful log queries (LogQL, Kibana Query Language) to diagnose build failures, flaky tests, and deployment issues with precision.

• Correlate logs with metrics and traces to gain deep, contextual visibility into pipeline behavior.

• Design reusable debugging dashboards and automation that turn raw logs into insights and action.

• Build a culture of shared troubleshooting knowledge through post-mortems, runbooks, and log-driven retrospectives.

To see the full-stack observability layer in action, check out the complete code and configurations in my GitHub repository: github.com/Emidowojo/CICDObservability. This repo includes all the setups for Grafana Loki, OpenTelemetry, Prometheus, and more, so you can deploy and explore the entire pipeline observability stack.

Next Steps for Advanced Observability Implementation

Here’s how you can take your setup even further:

1. Fully integrate distributed tracing: Deploy OpenTelemetry agents across your build and deployment stages. This will help you visualize how code, builds, and deployments flow across systems in real-time.

2. Automate diagnostic scripts and alerts: Build scripts to auto-collect logs and metrics on failure, and trigger alerts when known patterns reoccur. This enables faster detection and even self-healing pipelines.

3. Scale and harden your log infrastructure: As usage grows, implement log retention, compaction, and storage policies. Explore scalable backends like ClickHouse or object storage (e.g., S3) for long-term archiving.

4. Train your team on observability best practices: Share dashboards, create onboarding docs, and schedule log-analysis sessions to build team familiarity with your tools and practices.

📚 Resources for Continued Learning

Official Docs and Tools:

• Grafana Loki Documentation

• Promtail Configuration Guide

• OpenTelemetry

• LogQL Syntax

• Kibana Query Language

• Vector (log forwarding)

Communities:

• r/devops on Reddit

• CNCF Slack – #observability channel

• Log Management Best Practices on Stack Overflow

By investing in observability early and thoughtfully, you not only reduce the time to detect and resolve issues, you also build a more resilient, predictable, and transparent delivery process for your entire engineering team.

I hope this comes in handy for you someday. If you made it to the end of this handbook, thanks for reading! You can connect with me on LinkedIn or on X @Emidowojo if you’d like to stay in touch.

The thing is that most solar inverter brands have a form of remote data monitoring platform, from Victron’s VRM to Growatt’s ShineServer to Deye’s Cloud, among others. But I’m a fan of self-hosting and local control of data. This is one of the best ways to visualize and showcase all that beautiful data you have publicly to fellow tinkerers, solar inverter users, and the general public without relying on the company’s cloud data logger solution.

In this article, we will be using data available in our Home Assistant setup, sending it to InfluxDB and making a Grafana dashboard out of it. There are a good number of ways to connect your inverter to Home Assistant, depending on the manufacturer. I use a Growatt SPF ES 6000 inverter, and I shared a guide on how to make a local data logger for it that works with Home Assistant here.

Table of Contents

• Prerequisites

• How to Install and Configure InfluxDB

• How to Install and Configure Grafana

• How to Create the Grafana Solar Dashboard

• How to Create a New Admin User and Delete the Default Admin User

• How to Enable Remote Access to the Solar Dashboard

• Conclusion

Prerequisites

• Home Assistant OS

• A domain name

• An inverter connected to your Home Assistant instance

How to Install and Configure InfluxDB

We will be starting by setting up InfluxDB. InfluxDB is an open-source time series database, which differs from the database that Home Assistant uses by default, SQLite. We will be using InfluxDB v1, as it’s much easier to set up.

Go to your Home Assistant dashboard and go to Settings > Add-ons and click on the Add-On Store.

Inside the Add-on Store, search for “InfluxDB“ and click on the Add-on. You should see the screen below, then install.

Toggle the “Watchdog” on, as this allows the add-on to restart if it crashes. Also, toggle the “show in sidebar” on, which allows you to see the add-on on Home Assistant’s sidebar.

Start the add-on and look at the logs to be sure it is working. The “Starting NGINX” is an indicator it’s working.

Next, go to your Home Assistant sidebar and click on InfluxDB. You need to create a new database to hold your data and also create a new user that has admin privileges to read and write data. Go to the InfluxDB Admin tab.

Click on Create Database – and you can name the database anything you want. I will be naming mine homeassistant.

By default, the retention policy for a created database is infinity (which is forever), but you can configure this to be any time frame you want. Retention policy refers to the time frame of data the database can hold. I prefer to stick with infinity as I want to keep as much data as possible and I have enough storage in my Home Assistant hardware for that.

Once the database is created, go to the Users tab so you can create the new admin user. Input a username and password for that user and click on Grant Admin, so the permission level can be set to all. I created a new user called root.

At this point, what is left on the InfluxDB side is to tell Home Assistant to start sending sensor data to InfluxDB. You can do this by going to your Home Assistant configuration.yaml file and adding this config below to it. Your host is the IP of your Home Assistant, the port is the default port for the InfluxDB add-on, and the remaining values are based on the values you used during setup.

influxdb:
  host: 192.168.8.12
  port: 8086
  database: homeassistant
  username: root
  password: password
  max_retries: 3
  default_measurement: state

Restart your Home Assistant and go to InfluxDB. Click on the Explore tab, and check to see if you have a database.autogen file there. Click on it, and if you see some values under Measurements & Tags, you are good to go.

How to Install and Configure Grafana

Next on our agenda is to install and configure Grafana. The goal is to have Grafana query InfluxDB and make dashboards based on the queried data.

Go to the Add-on store, search for Grafana, and install it. Remember to toggle on those important settings, then start the add-on.

Once it has started, click on Grafana on the sidebar. You will arrive at Grafana’s homepage which is where you can create those dashboards.

But before you do that, you need to connect InfluxDB to Grafana. Navigate to Grafana’s tab >> Connections. You should see an “Add new connection” page. Search for InfluxDB and choose it. Then click on the add new datasource button.

Under HTTP, edit the URL and use http://ha_ip_address:8086 – don’t omit the http:// or try to use localhost with it. Scroll down to the InfluxDB Details and fill in the data you used while setting up InfluxDB. Then click on Save & Test. If the config is correct, you should see a green tick and text saying “datasource is working…measurements found.”

How to Create the Grafana Solar Dashboard

With that, you should have InfluxDB running and connected to Grafana. Let’s get to building beautiful dashboards out of all the data being generated. This part is subjective, so you can feel free to edit and modify the design to your taste. We will be using this dashboard here as the inspiration for our design.

So now go to your Grafana in Home Assistant, click on the + icon and create a new dashboard.

You should know that a dashboard in Grafana refers to the full space and each thing placed on the dashboard is a panel. Each visualization on the dashboard is a panel.

Let’s create a new panel. Pick InfluxDB as the data source, and at the FROM row, pick W which is the unit we want to create a visualization from. WHERE is entity_id::tag, as that is the way to sort the values by Home Assistant sensor entity name. Then pick the entity id of your panel – mine is growatt_pv1_charge_power. You can change the panel title, change the visualization to stat, and add the watt as the unit and the base colour to yellow.

The raw query looks like this:

SELECT mean("value") FROM "W" WHERE ("entity_id"::tag = 'growatt_pv1_charge_power') AND $timeFilter GROUP BY time($__interval) fill(null)

The Grafana edit page looks like this:

At this point, you should be able to recreate the remaining parts of the dashboard. But I manually did all that, so you don’t have to go through it all yourself if you don’t want to.

Here is the link to a GitHub repo that has the JSON file of this dashboard. When you go to create a dashboard, you will see an option to import from a JSON file. You can choose to copy and paste or upload the file, whichever works for you.

After importing, you only need to edit each panel through the GUI to use your own entity ID tag in Home Assistant and also the UID of your InfluxDB database.

Here is a picture of my finished result:

How to Create a New Admin User and Delete the Default Admin User

By default, the Grafana Add-on in Home Assistant uses an auth proxy and creates a default user (admin) with a password (hassio) that's synced with your HA login session. This prevents password or user changes through the UI.

For context, an auth proxy, or authentication proxy, acts as an intermediary between a client and a target resource, handling authentication and authorization on behalf of the client

As a security step, we need to create a new user for the Grafana Add-on and edit their permission to have admin privileges, then delete the default admin user. This is because you can’t change the default admin user password on the Add-on.

Go to Grafana’s menu > Administration> Users and access > Users. Then create a new user.

Next, give it admin privileges. Edit Grafana Admin to be yes and make sure the organization role is set to admin, then save.

Go back to the Add-on Configuration tab. Scroll to the Network setting and add a port to expose the Add-on. I will be using port 3000. Save and restart the Add-on. If you have SSL turned on and it isn’t configured, the add-on won’t start. You can disable it as we will have Cloudflare handle that.

To confirm that the port has been exposed properly, go to http://ha_ip:3000/ and confirm you see this Grafana login screen. Make sure it is http and not https.

Log in as the new user you created. Then go to your list of users and delete the default admin user.

After that, go back to the Grafana Add-on Configuration tab. Click on the 3 dots on the Options row and choose Edit in YAML. Then add this line below to your configuration file and save.

grafana_ingress_user: usernameofnewuser

How to Enable Remote Access to the Solar Dashboard

At this point, we have the solar dashboard all ready and we can access it in Home Assistant while inside our home network. But we don’t want it only that way. We want anyone to be able to visit the link without having access to our home network.

I will be implementing this part with the aid of a Home Assistant Cloudflared Add-on that leverages Cloudflare Tunnel. Here is the Github repository – the installation is simple and stress-free.

After going through the setup and having remote access to your Home Assistant network (remember to have 2FA turned on), go to the Cloudflared Add-on configuration tab and edit the Additional Hosts part.

- hostname: subdomain_you_want.your_domain.xyz
  service: http://ha_ip:3000
  disableChunkedEncoding: true

Save and restart the Add-on and check the logs. You should see it creating a DNS entry for the hostname you added.

As another security step, go to your Grafana Add-on Configuration tab. Add these values to the environment variables.

- name: GF_AUTH_ANONYMOUS_ENABLED
  value: "true"
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
  value: "Viewer"
- name: GF_AUTH_DISABLE_LOGIN_FORM
  value: "true"

• GF_AUTH_ANONYMOUS_ENABLED: Anyone who visits Grafana without logging in will still be allowed in.

• GF_AUTH_ANONYMOUS_ORG_ROLE: This sets the default permission for anonymous users. In this case, anonymous users will have the viewer role.

• GF_AUTH_DISABLE_LOGIN_FORM: Disables the login form on the Grafana login page. Make sure you are already logged in on the remote hostname. But you can always edit this on the Add-on Configuration tab if you get locked out.

Wrapping Up

Finally, go to the Remote hostname for your Grafana and you should see the Grafana home page. Then go to your dashboards and click on the solar dashboard created. Share it and choose publicly. Now you can share that link (the URL on that page and not the actual copied URL from the share button) with anyone and they can get to see your beautiful dashboard.

This method serves as an all-in-one way of having everything done, through your Home Assistant machine. I hope you had fun tinkering, see you next time.

In Kubernetes environments, the complexity of managing distributed microservices can be challenging. For instance, an application usually spans multiple pods, nodes, and clusters. Because of Kubernetes’s dynamic nature, where pods are frequently created and terminated, proper monitoring and observability are ideal for capturing its fleeting behavior.

Imagine building a microservices application with several connected services handling critical components such as authentication, payments, and databases without proper monitoring. A sudden traffic spike could affect a single service, cascading to other services, causing the system to crash and resulting in downtime.

Without proper visibility, you may struggle to find the root cause of the issue. You may spend hours manually going through logs – and meanwhile, users are frustrated, and businesses are losing revenue and customer trust.

Before we begin the project, you’ll learn key monitoring and observability concepts, as well as why tools like Prometheus and Grafana are crucial for setting up a robust monitoring stack on your Kubernetes infrastructure.

Here’s what we’ll cover:

• Understanding Monitoring and Observability

• Tools for Monitoring and Observability

• How to Deploy Prometheus and Grafana on AWS EKS using Helm

• Conclusion

Understanding Monitoring and Observability

Implementing a proper monitoring and observability approach is important in fast-paced production Kubernetes environments. This helps in situations where downtime can lead to serious business loss and damage to customer trust. It’ll hopefully help you avoid the dreaded 2 am calls that are usually triggered by alert noise so you can focus on adding more innovative features to your software (rather than spending so much energy firefighting).

Monitoring and observability are often referred to as the same thing. But they serve two different purposes, especially for development and engineering teams.

Monitoring

In the software development lifecycle, monitoring is the practice of analyzing data in real time or reviewing data trends to ensure the health and performance of systems, infrastructure, and applications. Monitoring acts as the eyes and ears of your IT operations, collecting insightful data and presenting it in a way that is actionable.

If you have visited the IT department of a well-established organization, you have most likely seen large screens displaying colorful dashboards with charts and real-time statistics. This offers a centralized view of the key metrics, such as the server uptime, application response times, and resource usage.

Observability

Observability helps to address issues that haven’t been anticipated. These are usually called “unknown unknowns." Unlike monitoring, which deals with predefined parameters and data, observability goes deeper into the application to give a broader view.

This not only helps to answer what is happening within your system and why it is happening. It also uses patterns within the system and application operations to detect and resolve issues efficiently.

Observability revolves around the three pillars of data: metrics, logs, and traces.

1. Metrics

Metrics consist of time-series measurements such as CPU usage and memory consumption. These data points help teams to manage, optimize, and predict system performance and deviations from expected behavior.

2. Logs

Logs serve as a history of what happened within the system. It is a trail for engineers, especially during troubleshooting. Logs are important in diagnosing root causes and discovering malicious activities.

3. Traces

Traces provide insights into application workflows by tracking requests as they move through various components. They are good for highlighting latency issues and potential points of failure.

Tools for Monitoring and Observability

Now that you understand the theory behind monitoring and observability, you may be wondering what platforms and tools are available to developers to collect data and get insights about their services.

In the world of cloud-native infrastructure and Kubernetes, many users gravitate towards the popular stack of Prometheus and Grafana.

Prometheus

Prometheus is an open-source tool that specializes in collecting metrics as time-series data. The information is stored with the timestamp when it was recorded.
The Prometheus ecosystem includes the main Prometheus server, which scrapes and stores time-series data, an alert manager for managing alerts, a push gateway for handling metrics from short-lived jobs, and exporters for collecting metrics from various services connected to the cluster.

It fits both in machine-centric and application-centric monitoring, especially for microservices in a Kubernetes cluster. It’s designed to be the system you go to if there is a system outage and you need to quickly diagnose problems.

The Prometheus ecosystem includes the main Prometheus server, which scrapes and stores time-series data, an alert manager for managing alerts, a push gateway for handling metrics from short-lived jobs, and exporters for collecting metrics from various services connected to the cluster.

Prometheus fits both in machine-centric and application-centric monitoring, especially for microservices in a Kubernetes cluster. It’s designed to be the system you go to if there is a system outage and you need to quickly diagnose problems.

Grafana

Grafana is a visualization tool that transforms, queries, visualizes, and sets alerts on raw metrics stored in Prometheus. With Grafana, you can explore metrics and logs wherever they are stored and display the data on live dashboards. This allows teams to monitor system performance, identify trends, and act quickly on anomalies.

Prometheus and Grafana are compatible with containerized applications, especially in Kubernetes environments. It can also manage workloads outside Kubernetes for flexibility. They are both open-source tools that give developers control over the implementation. There is no licensing cost, which helps teams that cannot afford expensive, powerful solutions.

By combining Prometheus and Grafana, your team gets helpful insights into the system to optimize performance, track errors, and aid troubleshooting processes.

How to Deploy Prometheus and Grafana on AWS EKS using Helm

Prerequisites

For this project, we will use an EC2 instance with the Ubuntu 22.04 operating system. If you are using Windows or a Mac, log into AWS to create your virtual machine.

Here’s what else you’ll need:

1. AWS account setup with access keys and secret keys

• AWS Sign Up

• AWS Access ID and Secret Keys

2. Knowledge of Kubernetes

• Kubernetes Official Documentation

3. AWS CLI installation for the virtual server

• AWS CLI Installation Guide

Getting Started

Let’s start by setting up an EKS cluster on a virtual server and installing the required tools on the server. Then, we’ll deploy our monitoring tools, Prometheus and Grafana, using Helm charts. In the end, we’ll deploy an NGINX web application on Kubernetes and use Grafana to visualize the pod performance and cluster resource usage on the cluster.

Step 1: Install AWS CLI, eksctl, kubectl, and Helm

AWS CLI is a tool that allows users to interact with AWS services using the command-line interface. It makes the management of cloud resources simpler and enables admins to configure AWS services.

Here, we will install AWS CLI on our server to be able to create Kubernetes resources.

On your server, run the following commands:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

sudo apt install unzip

unzip awscliv2.zip

sudo ./aws/install

Verify the installation by running this:

aws --version

After installation, configure the AWS CLI with your credentials using the following command:

aws configure

You will be prompted to enter your AWS Access Key ID, Secret Access Key, Default region name, and default output format.

Next, we need to install eksctl. eksctl is a command-line tool that simplifies the creation and management of Kubernetes clusters on AWS. It helps you configure, set, and maintain clusters and allows you to manage clusters more effectively.

This tool removes the complexities of setting up a production-grade cluster, helping you and your admins focus only on application development and deployment.

To set up eksctl on your machine, download the latest release using the following command:

# for ARM systems, set ARCH to: arm64, armv6 or armv7
ARCH=amd64

PLATFORM=$(uname -s)_$ARCH

curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"

# (Optional) Verify checksum

curl -sL "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_checksums.txt" | grep $PLATFORM | sha256sum --check

tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz

sudo mv /tmp/eksctl /usr/local/bin

Run eksctl version to confirm its successful installation and the version downloaded.

eksctl version # 0.198.0

Next, we’ll run Kubectl which is a command line interface for managing and interacting with Kubernetes clusters. It enables users to deploy and manage applications within a Kubernetes environment.

With Kubectl, you can perform various crucial operations such as scaling, deployments, inspecting cluster status, and managing networking.

To install kubectl, run the following commands:

curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin

Run kubectl on your command line to confirm it has been installed successfully:

kubectl version 
# client version: 0.198.0
# Kustomize Versionv: 5.4.2
# Server Version: v1.30.7-eks-56e63d8

Finally, we’ll install Helm which is a Kubernetes Package manager that simplifies the deployments and management of applications in Kubernetes. It uses charts to define Kubernetes resources into a collection of files, handles templating and versioning, and makes application deployment easier.

Here, we will install the Helm package manager on our virtual machine for our cluster deployments. This downloads the installation script and saves it in the get_helm.sh file.

Next, the file is set to executable, which allows only the user to run it. Finally, the script is executed using the ./get_helm.sh command.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

chmod 700 get_helm.sh

./get_helm.sh

Step 2: Create a Kubernetes Cluster

Next, we need to create our Kubernetes cluster in AWS with the eksctl command line. We can do this with the following command:

eksctl create cluster --name my-prac-cluster-1 --version 1.30 --region us-east-1 --nodegroup-name worker-nodes --node-type t2.medium --nodes 2 --nodes-min 2 --nodes-max 3

Let’s break down the command:

• –name my-prac-cluster-1: This specifies the name of the EKS cluster that will be created. In this case, the cluster will be named my-prac-cluster-1.

• –version 1.30: This sets the Kubernetes version for the cluster. Here, the version will be version 1.30.

• --region us-east-1: This specifies the AWS region where the cluster will be provisioned on AWS. Here, it is set to us-east-1.

• --nodegroup-name worker-nodes: This defines the name of the node groups that will be created. In this case, it’s named worker-nodes.

• --node-type t2.large: This sets the instance type for the worker nodes in the node-group.

• --nodes 2: This sets the desired number of worker nodes in the node group.

• --nodes-min 2: This sets the minimum number of worker nodes that should be maintained in the node group to 2.

• --nodes-max 3: This defines the maximum number of worker nodes allowed in the node group and sets it to 3.

Once the cluster comes up, run the command kubectl get nodes to ensure that the cluster is set up properly.

Step 3: Install the Metrics Server

The metrics server is a component that collects resource data from the Kubelets on each node in the cluster. This includes metrics such as CPU, memory, and network usage, which Prometheus can access. The server provides a single source of truth for resource data and is easy to deploy and use.

Run the following script to install the metrics server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

To verify the installation, run the following command:

kubectl get deployment metrics-server -n kube-system

Step 4: Install the IAM OIDC Identity Provider and Amazon EBS CSI Driver

The IAM OpenID connect provider allows Kubernetes access to AWS resources within the cluster. Here, we need EBS volumes to create persistent storage for Prometheus pods.

Run the following commands to create the IAM OIDC provider:

eksctl utils associate-iam-oidc-provider --cluster my-prac-cluster-1 --approve

Next, we will create the Amazon EBS CSI Driver that will provide permissions for the cluster to access the EBS volumes. Replace the placeholder “my-cluster” with your cluster name.

eksctl create iamserviceaccount \

--name ebs-csi-controller-sa \

--namespace kube-system \

--cluster my-prac-cluster-1 \

--role-name AmazonEKS_EBS_CSI_DriverRole \

--role-only \

--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \

--approve

Now, we need to add the AWS EBS Driver Addon to the cluster using the following commands:

eksctl create addon --name aws-ebs-csi-driver --cluster <cluster_name> --service-account-role-arn arn:aws:iam::<AWS_ACCOUNT_ID>:role/AmazonEKS_EBS_CSI_DriverRole --force

Adding the AWS EBS CSI Driver to your Kubernetes cluster enables the cluster to dynamically create and manage EBS volumes for persistent storage within the cluster. Since our Prometheus installation needs persistent volumes, this add-on will enable the cluster to create EBS volumes to persist data.

Now, our future Prometheus installation will create EBS volumes for persistent storage.

Step 5: Install Prometheus and Grafana.

To install Prometheus and Grafana, we need to add the Helm Stable Charts for the local client.

Run the command below:

helm repo add stable https://charts.helm.sh/stable

Next, we will add the Prometheus Helm repo:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

We’ll use the Prometheus community version because it is well-maintained by the Prometheus community. It offers faster updates and continuous improvements for different Kubernetes environments.

Next, create the Prometheus namespace:

kubectl create namespace prometheus

Install Prometheus and Grafana through the kube-prometheus-stack Helm Chart:

helm install stable prometheus-community/kube-prometheus-stack -n prometheus

When that’s done, verify that the Prometheus deployment and service are installed by using the command below:

kubectl get all -n prometheus

At this stage, you should change the service type from a ClusterIP to a LoadBalancer in the manifest file. We can update the file by running the command below:

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

After the update, a LoadBalancer URL will be generated for you to access your Prometheus Dashboard.

Next, we’ll move over to Grafana. Change the SVC file of Grafana to create a LoadBalancer and expose it to the public using the command below:

kubectl edit svc stable-grafana -n prometheus

Next, we will update the Grafana SVC file by changing the service type from ClusterIP to LoadBalancer to expose it to the public using the command below:

kubectl edit svc stable-grafana -n prometheus

Once the settings are saved, you can use the LoadBalancer link to access your Grafana Dashboard from the browser. The username is admin. To get the login password printed in the terminal, run the following command:

kubectl get secret --namespace prometheus stable-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

After successfully logging into the Grafana dashboard, the first step is to create a datasource that will provide the metrics for the Grafana visualization.

Go to Add your first data source and choose Prometheus as the Data Source.

Insert the Prometheus URL, and click on “Save and Test”. It should show success if Grafana queries the Prometheus URL successfully.

The next step is to create a Dashboard that our Grafana visualization will use to view the metrics of our pods. To do so, click on “Dashboards” and then on “Add Visualization.”

You’d be taken to an environment where you’d be required to import a dashboard. Select the data source as “Prometheus-1” and use the code “15760” to import the Node Exporter dashboard to view our pods.

Click on Load after importing the dashboard, and you will see your newly created dashboard.

Here, we can see the entire data of the cluster, the CPU and RAM use, and data regarding pods in a specified namespace.

Step 6: Deploying an Application on Kubernetes to Monitor on Grafana.

Finally, we will deploy an NGINX container in our EKS Cluster to monitor using Grafana. We need to create a Yaml deployment and service file.

apiVersion: apps/v1

kind: Deployment

metadata:

name: nginx-app

spec:

replicas: 2

selector:

matchLabels:

app: nginx-app

template:

metadata:

labels:

app: nginx-app

spec:

containers:

- name: nginx-app

image: nginx:latest

ports:

- containerPort: 80

---

apiVersion: v1

kind: Service

metadata:

name: nginx-app

spec:

type: LoadBalancer

ports:

- port: 80

targetPort: 80

selector:

app: nginx-app

To deploy the Node.js application on the Kubernetes cluster, use the following kubectl command. Verify the deployment by running the following kubectl command:

kubectl apply -f deployment.yml

kubectl get deployment

kubectl get pods

Click the load balancer URL to see your application on your browser:

Let’s refresh our Grafana dashboard to see our NGINX web application in Grafana.

Step 7: Deleting the Cluster

Now that everything is set up, we can delete our Kubernetes Cluster to avoid extra costs. Run the following commands to do so:

eksctl delete cluster my-prac-cluster-1 –region us-east-1

Conclusion

This article teaches the theory behind monitoring and observability and highlights the roles of Prometheus and Grafana in these processes.

We went through a hands-on deployment of Prometheus and Grafana on an EKS cluster and a web application to illustrate how they can be effectively monitored using Grafana.

By leveraging these tools, administrators can enjoy real-time visibility into their Kubernetes infrastructure, easily spot performance bottlenecks, and confidently make decisions that enhance application performance and reliability.

Grafana is an open-source analytics and monitoring platform. It has gained widespread recognition among developers and enterprises looking to extract more insights from the data produced by their systems.

Grafana has many powerful visualization features, and when combined with Amazon EC2's scalability and flexibility, it creates a stable environment for efficient monitoring.

This article will walk you through setting up Grafana on Amazon EC2 and creating informative dashboards out of raw data.

For Whom is this Intended?

This tutorial is intended for both novices to the cloud and experts in DevOps. The goal of this post is to make the installation process easier so you can use Grafana on AWS to its fullest. Now let's get going.

How to Configure Your EC2 Instance

You need to configure the inbound rule for your EC2 instance to access port 3000, as Grafana operates on this port. But first, you need to establish an EC2 instance. You can follow this guide on how to set up your AWS EC2 instance. It takes less than 5 minutes.

Once you have created your EC2 instance, you'll need to configure the network inbound rules. So head to your instance page and click on it. On the button widget, click on the security tab and click on the security group link (it should look like this: “sg-547**”).

Once you open the page in the inbound rules section, click on ‘Edit inbound rules’. Click on Add a new rule and add 3000 to the port range field, and on the source field, select 0.0.0.0/0. Then save.

Inbound rules

How to Create an IAM Role

Now you need to construct an IAM (Identity Access Management) role. You're developing an identity role so that you can generate credentials that you'll subsequently use to log in to your Grafana service.

IAM Dashboard

So, in the search field, type "IAM service" and click it. Click 'Create role', and select the AWS service as the trusted entity type.

IAM Trusted Entity

On the use case section, select EC2, then click next.

IAM role use case

On the Add Permissions page, click on the AdministratorAccess policy, then click next. Enter a role name – in this case, I used Grafana-Server-Role.

Role creation

How to Download Grafana

Now that you've configured your EC2 inbound rule and also configured the IAM role, let's set up Grafana on your EC2 instance.

So head over to Grafana's download page. Since we'll be downloading the version for Amazon Linux in this tutorial, you need to type in the following command on your Linux command line. Note: You need to connect to your VM instance through SSH (Secure Shell). In this case, I am using the EC2 Instance Connect.

sudo yum install -y https://dl.grafana.com/enterprise/release/grafana-enterprise-11.1.0-1.x86_64.rpm

Now you'll enable the Grafana service on your terminal by typing the following command:

systemctl enable grafana-server.service

Then start the service:

systemctl start grafana-server.service

Check the status of the Grafana service on the EC2 instance by running this command:

systemctl status grafana-server.service

Grafana Service Status

Now that you've confirmed that the service is currently active, you'll also need to check if the Grafana service is active on port 3000, as you've already created an inbound rule to cater for this.

You can do this by typing the following command:

netstat -tunpl | grep grafana

Port 3000 confirmation

Now that you've confirmed that the service runs on port 3000, you can go ahead and set up your Grafana dashboard.

You can access the Grafana dashboard by typing the Public IP of your EC2 instance and adding port 3000 on your web browser, something like this: 34.239.101.172:3000.

Grafana Login

The default username and password for Grafana are admin, but you'll be given the option to change your password after you sign in with the default credentials. You can also skip the password change process if you like.

Change password on Grafana

After this step, go to the home page. The next thing to do is to start connecting your Grafana dashboard to a data source. In this case, you're going to connect it to the AWS cloud watch service.

Grafana

How to Connect Data Sources to the Grafana Dashboard

Click on the connections tab on the side menu and click on data sources. Search for the CloudWatch service.

Cloudwatch configuration

Now you'll be prompted to input your access key ID and secret access key. You will need to create this on your AWS IAM service.

So go back to your IAM management dashboard and go to the user's tab. If you haven’t created an IAM user, you can do so by checking out this IAM user creation tutorial.

In the user IAM dashboard, scroll down to the access keys section and click on Create access key.

Access key

Select the Command Line Interface use case.

Access key use case

Set the description tag. This step is optional. Then click on the Create access key.

Access keys

Now copy the Access Key ID and Secret access key and paste them into the CloudWatch Datasource configuration page on Grafana. Set your default cloud region – in this case, mine is us-east-1

Additional settings

When you’re done, click on the save and test buttons. Grafana will query the Cloudwatch logs, and if it works fine it will save the configuration.

How to Create a Dashboard on Grafana

Now that you have successfully configured your grafana service, let’s start creating dashboards.

Click on the dashboard tab on the side menu click on New and select new dashboard. You should see the screen below:

Create a new dashboard

Then select Import dashboard.

Import a dashboard

For this case, you'll be importing an already-made dashboard from Grafana. Grafana has a lot of dashboards for a lot of use cases and services. But in this case, you'll be importing an EC2 dashboard (Grafana EC2 dashboard).

If you want to import it, you can easily copy the ID of the dashboard that you want to import. It is always accompanied by the dashboard.

So now copy the ID – in this case, it's 11265. Then paste it into the import field on the import dashboard, and click on the load button.

Grafana Dashboard

Now you have successfully created a dashboard in Grafana. This dashboard lets you monitor the performance of your EC2 instance. You can monitor metrics such as CPU Utilization, CPU Credit, Disk Ops, Disk Bytes, Network, Network Packets, Status check, and so on.

Wrapping Up

Thank you for reading! I hope this step by step guide has helped you learn how to create and set up efficient dashboards using Grafana.

One of the coolest things I have learned in the last year is how to constantly deliver value into production without causing too much chaos.

In this post, I’ll explain the metrics-driven development approach and how it helped me to achieve that. By the end of the post, you’ll be able to answer the following questions:

• What are metrics and why should I use them
• What are the different types of metrics
• What tools could I use to store and display metrics
• What is a real-world example of metrics-driven development

What are metrics and why should I use them?

Metrics give you the ability to collect information on an actively running system without changing its code.

It allows you to gain valuable data on the behavior of your application while it runs so you can make data-driven decisions based on real customer feedback and usage in production.

What are the types of metrics available to me?

These are the most common metrics used today:

• Counter — Represents a monotonically increasing value.

Counters are really useful for measuring rates!

In this example, a counter metric is used to calculate the rate of events over time, by counting events per second

• Gauge — Represents a single value that can go up or down.

Gauges are really useful for measuring CPU usage!

In this example, a gauge metric is used to monitor the user CPU in percentages

• Histogram — A counting of observations (like request durations or sizes) in configurable buckets.

Histograms are really useful for measuring request duration!

In this example, a histogram metric is used to calculate the 75th and 90th percentiles of an HTTP request duration.

The bits and bytes of the types: counter, histogram, and gauge can be quite confusing. Try reading about it further here.

What tools can I use to store and display metrics?

Most monitoring systems consist of a few parts:

1. Time-series database — A database software that optimizes storing and serving time-series data. Two examples of this kind of database are Whisper and Prometheus.
2. Querying engine (with a querying language) — Two examples of common query engines are: Graphite and PromQL
3. Alerting system — The mechanism that allows you to configure alerts based on graphs created by the querying language. The system can send these alerts to Mail, Slack, PagerDuty. Two examples of common alerting systems are: Grafana and Prometheus.
4. UI — Allows you to view the graphs generated by the incoming data and configure queries and alerts. Two examples of common UI systems are: Graphite and Grafana

The setup we are using today in BigPanda Engineering is

• Telegraf — used as a StatsD server.
• Prometheus — used as our scrapping engine, Time-series database and querying engine.
• Grafana — used for Alerting, and UI

And the constraints we had in mind while choosing this stack were:

• We want scalable and elastic metrics scraping
• We want a performant query engine
• We want the ability to query our metrics using custom tags(such as service names, hosts, etc.)

A real-world example of Metrics-driven development of a Sentiment Analysis service

Let’s develop a new pipeline service that calculates sentiments based on textual inputs and does it in a Metrics Driven Development way!

Let’s say I need to develop this pipeline service:

Sentiment analysis pipeline architecture

And this is my usual development process:

Usual development process - Test, code and deploy. Oh my!

So I write the following implementation:

let senService: SentimentAnalysisService = new SentimentAnalysisService();
while (true) {
    let tweetInformation = kafkaConsumer.consume()
    let deserializedTweet: { msg: string } = deSerialize(tweetInformation)
    let sentimentResult = senService.calculateSentiment(deserializedTweet.msg)
    let serializedSentimentResult = serialize(sentimentResult)
    sentimentStore.store(sentimentResult);
    kafkaProducer.produce(serializedSentimentResult, 'sentiment_topic', 0);
}

The full gist can be found here.

And this method works perfectly fine**.

But what happens when it doesn’t?

The reality is that while working (in an agile development process) we make mistakes. That’s a fact of life.

I believe that the real challenge with making mistakes is not to avoid them, but rather to optimize how fast we detect and repair them. So, we need to gain the ability to quickly discover our mistakes.

It's time for the MDD-way.

The Metrics Driven Development (MDD) way

Behold! The Three Commandments of Production!

The MDD approach is heavily inspired by the Three Commandments of Production (which I had learned about the hard way).

The Three Commandments of Production are:

1. There are mistakes and bugs in the code you write and deploy.
2. The data flowing in production is unpredictable and unique!
3. Perfect your code from real customer feedback and usage in production.

And since we now know the Commandments, it's time to go over the 4 step plan of the Metrics-Driven development process.

The 4-step plan for a successful MDD

Metrics-driven development ?Oh wow!

Develop code 

I write the code, and whenever possible, wrap it with a feature flag that allows me to gradually open it for users.

Metrics

This consists of two parts:

Add metrics on relevant parts

In this part, I ask myself what are the success or failure metrics I can define to make sure my feature works? In this case, does my new pipeline application perform its logic correctly?

Add alerts on top of them so that I’ll be alerted when a bug occurs

In this part, I ask myself What metric could alert me if I forgot something or did not implement it correctly?

Deployment

I deploy the code and immediately monitor it to verify that it’s behaving as I have anticipated.

Iterate this process to perfection

And that's it! Now that we have learned the process, let's tackle an important task inside it.

Metrics to Report — what should we monitor?

One of the toughest questions for me, when I’m doing MDD, is: “what should I monitor”?

That’s a lovely gif. but un-realistic in most cases.

In order to answer the question, lets try to zoom out and look at the big picture.
All the possible information available to monitor can be divided into two parts:

1. Applicative information — Information that has an applicative context and meaning. An example of this will be — “How many tweets did we classify as positive in the last hour”?
2. Operational information — Information that is related to the infrastructure that surrounds our application — Cloud data, CPU and disk utilization, network usage, etc.

Now, since we cannot monitor everything, we need to choose what applicative and operational information we want to monitor.

• The operational part really depends on your ops stack and has built-in solutions for (almost) all your monitoring needs.
• The applicative part is more unique to your needs, and I'll try to explain how I think about it later in this post.

After we do that, we can ask ourselves the question: what alerts do we want to set up on top of the metrics we just defined?

The diagram (of information, metrics, alerts) can be drawn like this:

The world of information, metrics, and alerts.

Applicative metrics

I usually add applicative metrics out of two needs:

To answer questions

A question is something like, “When my service misbehaves, what information would be helpful to know about?”

Some answers to that question can be — latencies of all IO calls, processing rate, throughput, etc…

Most of these questions will be helpful while you are searching for the answer. But once you found it, chances are you will not look at it again (since you already know the answer).

These questions are usually driven by RND and are (usually) used to gather information internally.

To add Alerts

This may sound backward, but I usually add applicative metrics in order to define alerts on top of them. Meaning, we define the list of alerts and then deduce from them what are the applicative metrics to report.

These alerts are derived from the SLA of the product and are usually treated with mission-critical importance.

Common types of alerts

Alerts can be broken down into three parts:

Alerts types to Metrics list

SLA Alerts

SLA alerts in reality

SLA alerts surround the places in our system where an SLA is specified to meet explicit customer or internal requirements (i.e availability, throughput, latency, etc.). SLA breaches involve paging RND and waking people up, so try to keep the alerts in this list to a minimum.

Also, we can define Degradation Alerts in addition to SLA Alerts.
Degradation alerts are defined with lower thresholds then SLA alerts, and are therefore useful in reducing the amount of SLA breaches — by giving you a proper heads-up before they happen.

An example of an SLA alert would be, “All sentiment requests must finish in under 500ms.”

An example of a Degradation Alert will be: “All sentiment requests must finish in under 400ms”.

These are the alerts I defined:

1. Latency — I expect the 90th percentile of a single request duration not to exceed 300ms.
2. Success/Failure ratio of requests — I expect the number of failures per second, success per second, to remain under 0.01.
3. Throughput — I expect that the number of operations per second (ops) that the application handles will be > 200
4. Data Size — I expect the amount of data that we store in a single day should not exceed 2GB.

200 ops 60 bytes(Size of Sentiment Result) 86400 sec in a day = 1GB < 2GB

Baseline Breaching Alerts

These alerts usually involve measuring and defining a baseline and making sure it doesn’t (dramatically) change over time with alerts.

For example, the 99th processing latency for an event must stay relatively the same across time unless we have made dramatic changes to the logic.

These are the alerts I defined:

1. Amount of Positive or Neutral or Negative Sentiment tweets — If for whatever reason, the sum of Positive tweets has increased or decreased dramatically, I might have a bug somewhere in my application.
2. All latency \ Success ratio of requests \ Throughput \ Data size must not increase\decrease dramatically over time.

Runtime Properties Alerts

I’ve given a talk about Property-Based Tests and their insane strength. As it turns out, collecting metrics allows us to run property-based tests on our system in production!

Some properties of our system:

1. Since we consume messages from a Kafka topic, the handled offset must monotonically increase over time.
2. 1 ≥ sentiment score ≥ 0
3. A tweet should classify as either Negative \ Positive \ Neutral.
4. A tweet classification must be unique.

These alerts helped me validate that:

1. We are reading with the same group-id. Changing consumer group ids by mistake in deployment is a common mistake when using Kafka. It causes a lot of mayhem in production.
2. The sentiment score is consistently between 0 and 1.
3. Tweet category length should always be 1.

In order to define these alerts, you need to submit metrics from your application. Go here for the complete metrics list.

Using these metrics, I can create alerts that will “page” me whenever one of these properties do not hold anymore in production.

Processing latency breached configured SLA! Oh my! ?

Let’s take a look at a possible implementation of all these metrics

import SDC = require("statsd-client");
let sdc = new SDC({ host: 'localhost' });
let senService: SentimentAnalysisService; //...
while (true) {
    let tweetInformation = kafkaConsumer.consume()
    sdc.increment('incoming_requests_count')
    let deserializedTweet: { msg: string } = deSerialize(tweetInformation)
    sdc.histogram('request_size_chars', deserializedTweet.msg.length);
    let sentimentResult = senService.calculateSentiment(deserializedTweet.msg)
    if (sentimentResult !== undefined) {
        let serializedSentimentResult = serialize(sentimentResult)
        sdc.histogram('outgoing_event_size_chars', serializedSentimentResult.length);
        sentimentStore.store(sentimentResult)
        kafkaProducer.produce(serializedSentimentResult, 'sentiment_topic', 0);
    }

}

The full code can be found here

A few thoughts on the code example above:

1. There has been a staggering amount of metrics added to this codebase.
2. Metrics add complexity to the codebase, so, like all good things, add them responsibly and in moderation.
3. Choosing correct metric names is hard. Take your time selecting proper names. Here’s an excellent post about this.
4. You still need to collect these metrics and display them in a monitoring system (like Grafana), plus add alerts on top of them, but that’s a topic for a different post.

Did we reach the initial goal of identifying issues and resolving them faster?

YESSSS, it was!

We can now make sure the application latency and throughput do not degrade over time. Also, adding alerts on these metrics allows for a much faster issue discovery and resolution.

Conclusion

Metrics-driven development goes hand in hand with CI\CD, DevOps, and agile development process. If you are using any of the above keywords, then you are in the right place.

When done right, metrics make you feel more confident in your deployment in the same way that seeing passing unit-tests in your build makes you feel confident in the code you write.

Adding metrics allows you to deploy code and feel confident that your production environment is stable and that your application is behaving as expected over time. So I encourage you to try it out!

Some references

1. Here is a link to the code shown in this post, and here is the full metrics list described.
2. If you are eager to try writing some metrics and to connect them to a monitoring system, check out Prometheus, Grafana and possibly this post
3. This guy wrote a delightful post about metrics-driven development. GO read it.

DeviceHive is an IoT platform which has plenty of different components. The Grafana plugin is one of them. This plugin can gather data from a DeviceHive server and display it with different dashboards using the very popular tool — Grafana. This article explains how to create a Grafana dashboard with DeviceHive. As an example, this uses the ESP8266 chip analog pin to visualise the voltage on it.

Data

To display anything on a dashboard we need data. In terms of a DeviceHive server, data can be provided via ‘commands’ and ‘notifications’. Commands are typically used to deliver any messages to a device which the device should execute while ‘notifications’ are the opposite, devices notify their subscribers about certain events. ‘Commands’ and ‘notifications’ are basically simple JSON messages.

Both of these two entities can be used to plot graphs, display static text, a gauge, table or any other Grafana components. For this article we will generate notifications using special DeviceHive firmware for the ESP8266 chip. This firmware allows the chip to connect directly to a DeviceHive server using its protocol and has plenty of documented commands which can be issued from the server side.

Generating notifications with ESP8266 firmware

The binaries for the DeviceHive firmware are available here. Download the latest version and flash this firmware to your chip. The release archive contains documentation on how to do that, but if you have a ‘nodemcu’-like board you just need to connect the board via a microUSB cable to your computer and run the ‘esp-flasher’ util from the release archive for your operating system and wait until it flashes the board. Having flashed the board, there is a need to configure the chip which Wi-Fi network, DeviceHive server, and credential it should use. There are two ways to do that: using a posix-like terminal with the ‘esp-terminal’ util or wirelessly as described here.

There is a free playground service, which can be used for absolutely free to try a DeviceHive server. After your chip is connected to your server or playground, go to the server admin panel, find your ESP8266 device in the device list and issue the ‘adc/int’ command with the parameters ‘{“0”: 500}’.

This command causes the esp8266 to report every 500ms the voltage on ADC input #0(the only ESP8266 has). After switching to ‘notifications’ there should be screen like:

That is the voltage on the chip’s input pin. And this kind of data is suitable for us to display with Grafana: notifications contain data (parameters in our case), notifications come continuously, and all DeviceHive’s notifications always have a timestamp. Having an analogue sensor connected to this pin it is possible to display this data with Grafana.

Installing the DeviceHive Grafana plugin to Grafana

Grafana can be used as a local service or as a hosted service. To install Grafana locally, please, refer to the “Official documentation. Grafana installation”.

You can find how to install plugins in the “Official documentation. Plugin installation”.

To install DeviceHive datasource via grafana-cli you can use the following command:

$ grafana-cli plugins install devicehive-devicehive-datasource

If you want to install the plugin manually, you should perform the following steps:

Prerequisites, these packages should be installed:

• Grafana >= 4.6
• NodeJs >= 8 (optional)
• NPM >= 5 (optional)
• Grunt (npm install -g grunt) (optional)

Also you should have permissions to copy data to the Plugins folder (you could set it in grafana.ini in Paths->plugins).

1. Clone this repo to the Plugins folder — git clone [https://github.com/devicehive/devicehive-grafana-datasource.git](https://github.com/devicehive/devicehive-grafana-datasource.git;);
2. Next steps are optional (in case if you want to rebuild datasource sources code):
   2.1 Go into folder — cd devicehive-grafana-datasource;
   2.2 Install all packages — npm install;
   2.3 Build plugin — npm run build;
3. Restart Grafana server
4. Open Grafana in browser;
5. Open the side menu by clicking the Grafana icon in the top header;
6. In the side menu click Data Sources;
7. Click the + Add data source in the top header;
8. Select DeviceHive from the Type dropdown;
9. Configure the datasource.

After installation you will be able to see the DeviceHive datasource plugin in the installed plugins list (look at the picture below).

Adding Grafana datasource

To add DeviceHive datasource, you should perform the following steps:

1. Open the side menu by clicking the Grafana icon in the top header;
2. In the side menu click Data Sources;
3. Click the + Add data source in the top header;
4. Select DeviceHive from the Type dropdown;

Look at the picture below:

To configure the DeviceHive datasource you should fill on the following fields:

Server URL (is the path to the DeviceHive WebSocket server. For the playground this is ws://playground.devicehive.com/api/websocket)
Device ID (unique identifier of the DeviceHive device)
Login/Password or AccessToken — credentials to pass authentication

Also, you are able to specify the RefreshToken for auto refreshing the AccessToken

On the picture below you can observe the configuration workflow:

After adding and configuring a DeviceHive datasource, it should exist in the datasource list as in the picture below:

Create new dashboard

To create a new dashboard you should just click on the “New” button in the sidebar panel as shown in the picture below:

In this article we will show examples on the Graph panel, so, click on the Graph button:

After that you will be able to see line chart on your dashboard:

Displaying notifications/commands with Grafana’s graph

Notification and commands are DeviceHive entities:
Command: represents a message dispatched by clients for devices
Notification: represents a message dispatched by devices for clients

By default, a Notification or Command message provides the field named “parameters” in which a user can pass their own data.

At the start of this article we configured the ESP8266 device to send notifications with data that represents the state on analogue pin#0 of the chip. In the picture below you are able to observe how to configure the Grafana graph panel to make it show the data on the line chart:

Displaying annotations on Grafana’s graph

Annotations provide a way to mark points on the graph with rich events. When you hover over an annotation you can get an event description and event tags. The text field can include links to other systems with more detail.
More information about annotations you can find by following this link.

The picture below shows how to configure annotations powered by a DeviceHive datasource.

Advanced graph tuning

After clicking on the “Add converter” button you will be able to select a converter.
A converter is the simple function that transforms a value in some way.

For now, DeviceHive datasources support the following types of converters:

• Scale — multiplies by a given value
• Offset — adds a given value
• Unit converter — converts value between different units of below mentioned measurement types:
• Temperature (‘c’ — Celsius, ‘f’ — Fahrenheit, ‘k’ — Kelvin)
• Length (‘m’ — Meter, ‘mi’ — Mile, ‘yd’ — Yard, ‘ft’ — Feet, ‘in’ — Inch)
• Weight (‘kg’ — Kilogram, ‘lb’ — Pound, ‘oz’ — Ounces)
• Volume (‘l’ — Liter, ‘gal’ — Gallon, ‘pt’ — Pint)

An example of this functionality is shown in the picture below.

Conclusion

Grafana is a perfect tool to visualise data. It is very flexible and provides many different features to make visualisation in a way thay you like. Grafana can use many data sources from a wide range of software solutions and DeviceHive is one of them. The sample which we described in this article is very simple. Using these principles it is possible to create more advanced graphs and we hope it will be helpful for you. Using Grafana and DeviceHive you can build your own IoT visualisation solutions and moreover you can modify both projects as you wish since Grafana and DeviceHive are open source software.

_Written in collaboration with Igor Trambovetskiy, Senior Developer at DeviceHive._