EDA enables different parts of an application to interact in a decoupled way, allowing them to communicate through events instead of direct calls. This setup lets components work independently, respond to events asynchronously, and adjust to changing business needs without major system reconfiguration, promoting agility.

New and modern applications now heavily rely on real-time data processing and responsiveness. The EDA’s importance cannot be overstated because it provides the framework that supports those requirements. By using asynchronous communication and event-driven interactions, systems can efficiently handle high volumes of transactions and maintain performance under unstable loads. These features are particularly appreciated in environments where changes are very spontaneous, such as e-commerce platforms or IoT applications.

Some key components of EDA include:

• Event Sources: These are the producers that generate events when significant actions occur within the system. Examples include user interactions or data changes.

• Listeners: These are entities that subscribe to specific events and respond when those events occur. Listeners enable the system to react dynamically to changes.

• Handlers: These are responsible for processing the events once they are detected by listeners, executing the necessary business logic or workflows triggered by the event.

In this article, you will learn how to implement event-driven data processing using Traefik, Kafka, and Docker.

Here is a simple application hosted on GitHub that you can quickly run to get an overview of what you will be building today.

Table of Contents

Here is what we'll cover:

• Table of Contents

• Prerequisites

• Understanding the Technologies

• How to Set Up the Environment

• How to Build the Event-Driven System

• How to Integrate Traefik with Kafka

• Testing the Setup

• Conclusion

Let's get started!

Prerequisites

Before you begin:

• Deploy an Ubuntu 24.04 instance with at least 4 GB of RAM and a minimum of 20 GB of free disk space to accommodate Docker images, containers, and Kafka data.

• Access the instance with a non-root user with sudo privileges.

• Update the package index.

sudo apt update

Understanding the Technologies

Apache Kafka

Apache Kafka is a distributed event streaming platform built for high-throughput data pipelines and real-time streaming applications. It acts as the backbone for implementing EDA by efficiently managing large volumes of events. Kafka uses a publish-subscribe model where producers send events to topics, and consumers subscribe to these topics to receive the events.

Some of the key features of Kafka include:

• High Throughput: Kafka is capable of handling millions of events per second with low latency, making it suitable for high-volume applications.

• Fault Tolerance: Kafka's distributed architecture ensures data durability and availability even in the face of server failures. It replicates data across multiple brokers within a cluster.

• Scalability: Kafka can easily scale horizontally by adding more brokers to the cluster or partitions to topics, accommodating growing data needs without significant reconfiguration.

Traefik

Traefik is a modern HTTP reverse proxy and load balancer designed specifically for microservices architectures. It automatically discovers services running in your infrastructure and routes traffic accordingly. Traefik simplifies the management of microservices by providing dynamic routing capabilities based on service metadata.

Some of the key features of Traefik include:

• Dynamic Configuration: Traefik automatically updates its routing configuration as services are added or removed, eliminating manual intervention.

• Load Balancing: It efficiently distributes incoming requests across multiple service instances, improving performance and reliability.

• Integrated Dashboard: Traefik provides a user-friendly dashboard for monitoring traffic and service health in real-time.

By using Kafka and Traefik in an event-driven architecture, you can build responsive systems that efficiently handle real-time data processing while maintaining high availability and scalability.

How to Set Up the Environment

How to Install Docker on Ubuntu 24.04

1. Install the required packages.

sudo apt install ca-certificates curl gnupg lsb-release

2. Add Docker’s official GPG Key.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

3. Add the Docker repository to your APT sources.

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

4. Update the package index again and install Docker Engine with the Docker Compose plugin.

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

5. Check to verify the installation.

sudo docker run hello-world

Expected Output:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:305243c734571da2d100c8c8b3c3167a098cab6049c9a5b066b6021a60fcb966
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

How to Configure Docker Compose

Docker Compose simplifies the management of multi-container applications, allowing you to define and run services in a single file.

1. Create a project directory

mkdir ~/kafka-traefik-setup && cd ~/kafka-traefik-setup

2. Create a docker-compose.yml file.

nano docker-compose.yml

3. Add the following configuration to the file to define your services.

version: '3.8'

services:
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"

  traefik:
    image: traefik:v2.9
    ports:
      - "80:80"       # HTTP traffic
      - "8080:8080"   # Traefik dashboard (insecure)
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

Save your changes with ctrl + o, then exit with ctrl + x.

4. Start your services.

docker compose up -d

Expected Output:

[+] Running 4/4
 ✔ Network kafka-traefik-setup_default        Created                  0.2s
 ✔ Container kafka-traefik-setup-zookeeper-1  Started                  1.9s
 ✔ Container kafka-traefik-setup-traefik-1    Started                  1.9s
 ✔ Container kafka-traefik-setup-kafka-1      Started                  1.9s

How to Build the Event-Driven System

How to Create Event Producers

To produce events in Kafka, you will need to implement a Kafka producer. Below is an example using Java.

1. Create a file kafka-producer.java.

nano kafka-producer.java

2. Add the following configuration for a Kafka Producer.

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

import java.util.Properties;

public class SimpleProducer {
    public static void main(String[] args) {
        // Set up the producer properties
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        // Create the producer
        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        try {
            // Send a message to the topic "my-topic"
            ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "Hello, Kafka!");
            RecordMetadata metadata = producer.send(record).get(); // Synchronous send
            System.out.printf("Sent message with key %s to partition %d with offset %d%n", 
                              record.key(), metadata.partition(), metadata.offset());
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            // Close the producer
            producer.close();
        }
    }
}

Save your changes with ctrl + o, then exit with ctrl + x.

In the above configuration, the producer sends a message with the key "key1" and the value "Hello, Kafka!" to the topic "my-topic".

How to Set Up Kafka Topics

Before producing or consuming messages, you need to create topics in Kafka.

1. Use the kafka-topics.sh script included with your Kafka installation to create a topic.

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic <TopicName> --partitions <NumberOfPartitions> --replication-factor <ReplicationFactor>

For example, if you want to create a topic named my-topic with 3 partitions and a replication factor of 1, run:

docker exec <Kafka Container ID> /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 3 --replication-factor 1

Expected Output:

Created topic my-topic.

2. Check to confirm if the Topic was created successfully.

docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Expected Output:

my-topic

How to Create Event Consumers

After you have created your producers and topics, you can create consumers to read messages from those topics.

1. Create a file kafka-consumer.java.

nano kafka-consumer.java

2. Add the following configuration for a Kafka consumer.

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class SimpleConsumer {
    public static void main(String[] args) {
        // Set up the consumer properties
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
        props.put(ConsumerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");

        // Create the consumer
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

        // Subscribe to the topic
        consumer.subscribe(Collections.singletonList("my-topic"));

        try {
            while (true) {
                // Poll for new records
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("Consumed message with key %s and value %s from partition %d at offset %d%n",
                                      record.key(), record.value(), record.partition(), record.offset());
                }
            }
        } finally {
            // Close the consumer
            consumer.close();
        }
    }
}

Save your changes with ctrl + o, then exit with ctrl + x.

In the above configuration, the consumer subscribes to my-topic and continuously polls for new messages. When messages are received, it prints out their keys and values along with partition and offset information.

How to Integrate Traefik with Kafka

Configure Traefik as a Reverse Proxy.

Integrating Traefik as a reverse proxy for Kafka allows you to manage incoming traffic efficiently while providing features such as dynamic routing and SSL termination.

1. Update the docker-compose.yml file.

version: '3.8'

services:
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.kafka.rule=Host(`kafka.example.com`)"
      - "traefik.http.services.kafka.loadbalancer.server.port=9092"

  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"

  traefik:
    image: traefik:v2.9
    ports:
      - "80:80"        # HTTP traffic
      - "8080:8080"    # Traefik dashboard (insecure)
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

In this configuration, replace kafka.example.com with your actual domain name. The labels define the routing rules that Traefik will use to direct traffic to the Kafka service.

2. Restart your services.

docker compose up -d

3. Access your Traefik dashboard by accessing http://localhost:8080 on your web browser.

   

   Load Balancing with Traefik

   Traefik provides built-in load balancing capabilities that can help distribute requests across multiple instances of your Kafka producers and consumers.

   Strategies for Load Balancing Event-Driven Microservices

   1. Round Robin:

By default, Traefik uses a round-robin strategy to distribute incoming requests evenly across all available instances of a service. This is effective for balancing load when multiple instances of Kafka producers or consumers are running.

2. Sticky Sessions:

If you require that requests from a specific client always go to the same instance (for example, maintaining session state), you can configure sticky sessions in Traefik using cookies or headers.

3. Health Checks:

Configure health checks in Traefik to ensure that traffic is only routed to healthy instances of your Kafka services. You can do this by adding health check parameters in the service definitions within your docker-compose.yml file:

    labels:
      - "traefik.http.services.kafka.loadbalancer.healthcheck.path=/health"
      - "traefik.http.services.kafka.loadbalancer.healthcheck.interval=10s"
      - "traefik.http.services.kafka.loadbalancer.healthcheck.timeout=3s"

Testing the Setup

Verifying Event Production and Consumption

1. Kafka provides built-in command-line tools for testing. Start a Console producer.

    docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

After running this command, you can type messages into the terminal, which will be sent to the specified Kafka topic.

2. Start another terminal session and start a console consumer.

    docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

This command will display all messages in my-topic, including those produced before the consumer started.

3. To see how well your consumers are keeping up with producers, you can run the following command to check the lag for a specific consumer group.

    docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <your-consumer-group>

Monitoring and Logging

1. Kafka Metrics:

Kafka exposes numerous metrics that can be monitored using JMX (Java Management Extensions). You can configure JMX to export these metrics to monitoring systems like Prometheus or Grafana. Key metrics to monitor include:

• Message Throughput: The rate of messages produced and consumed.

• Consumer Lag: The difference between the last produced message offset and the last consumed message offset.

• Broker Health: Metrics related to broker performance, such as request rates and error rates.

2. Prometheus and Grafana Integration:

To visualize Kafka metrics, you can set up Prometheus to scrape metrics from your Kafka brokers. Follow these steps:

• Enable JMX Exporter on your Kafka brokers by adding it as a Java agent in your broker configuration.

• Configure Prometheus by adding a scrape job in its configuration file (prometheus.yml) that points to your JMX Exporter endpoint.

• Use Grafana to create dashboards that visualize these metrics in real-time.

How to Implement Monitoring for Traefik

1. Traefik Metrics Endpoint.

Traefik provides built-in support for exporting metrics via Prometheus. To enable this feature, add the following configuration in your Traefik service definition within docker-compose.yml:

    command:
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.addservice=true"

2. Visualizing Traefik Metrics with Grafana.

Once Prometheus is scraping Traefik metrics, you can visualize them using Grafana:

• Create a new dashboard in Grafana and add panels that display key Traefik metrics such as:

• traefik_entrypoint_requests_total: Total number of requests received.

• traefik_backend_request_duration_seconds: Response times of backend services.

• traefik_service_requests_total: Total requests forwarded to backend services.

3. Setting Up Alerts.

Configure alerts in Prometheus or Grafana based on specific thresholds (e.g., high consumer lag or increased error rates).

Conclusion

In this guide, you successfully implemented Event Driven Architecture (EDA) using Kafka and Traefik within the Ubuntu 24.04 environment.

Additional Resources

To learn more you can visit:

• The Apache Kafka Official Documentation

• The Traefik Official Documentation

• The Docker Official Documentation

• Vultr guide for for setting up Traefik Proxy on Ubuntu 24.04

Workflows are set steps by which individual jobs are executed. You can use Argo Workflows for various purposes, such as data processing, machine learning, CI/CD, and infrastructure automation.

In this article, you will set up Argo Workflows on a Kubernetes cluster and use available templates to create a Workflow to manage in a Kubernetes cluster.

Argo Workflows Key Concepts

As I mentioned above, Argo Workflows is implemented as a Kubernetes custom resource definition (CRD) by its own controller. Argo Workflows is made up of two main concepts: workflow and Template.

Workflow

A workflow is a central resource in Argo Workflows that defines the workflow to be executed and stores the workflow’s state.

A workflow consists of a specification that contains an entrypoint and a list of templates.

A workflow can model complex logic using directed acyclic graphs (DAGs) or steps to capture the dependencies or sequences between the templates.

Template

A template is a core concept in Argo Workflows that defines the instructions to execute in a workflow step. A template can be one of the following types:

• Container: Allows you to be able to define the container. Here is an example:

  - name: hello-world
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo hello world"]

• Script: A template that runs a script in a container image. This is similar to the container type, but allows you to write the script inline instead of using a separate file. Here is an example:

  - name: factorial
    inputs:
      parameters:
      - name: num
    script:
      image: python:alpine 3.6
      command: [python]
      source: |
        def factorial(n):
          if n == 0:
            return 1
          else:
            return n * factorial(n-1)
        print(factorial(int({{inputs.parameters.num}})))

• Resource: This template allows direct manipulation of cluster resources. It can be used for operations such as retrieval, creation, modification, or deletion, including GET, CREATE, APPLY, PATCH, REPLACE, or DELETE requests. Here is an example:

  - name: create-configmap
    resource:
      action: create
      manifest: |
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: my-config
        data:
          foo: bar
          hello: world

• DAG: This template defines a directed acyclic graph of other templates. In this example, the DAG has three tasks: A, B, and C. Task A runs first, then tasks B and C run in parallel. Here is an example:

  - name: my-dag
    dag:
      tasks:
      - name: A
        template: echo
        arguments:
          parameters: [{name: message, value: A}]
      - name: B
        dependencies: [A]
        template: echo
        arguments:
          parameters: [{name: message, value: B}]
      - name: C
        dependencies: [A]
        template: echo
        arguments:
          parameters: [{name: message, value: C}]

Now, let's get started.

Prerequisites

To follow this guide, make sure you have the following:

• A Kubernetes cluster. To create a new Kubernetes cluster, follow this guide

• Basic knowledge of what Kubernetes is. You can learn more about Kubernetes from their official docs

• The Kubectl CLI

Step 1 - Install Argo Workflows

1. Using Kubectl, create a namespace for Argo Workflows to segregate your kubernetes cluster resources.

$ kubectl create namespace argo

2. Install the latest Argo Workflows release file from Argo Workflows github page.

$ kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v<VERSION>/install.yaml

The version of Argo Workflows used in this guide is v3.5.0.

3. Check if all the resources have been installed correctly.

$ kubectl get all -n argo

Output:

NAME                                      READY   STATUS    RESTARTS   AGE
pod/workflow-controller-7f8c9f8f5-9qj2l   1/1     Running   0          2m
pod/argo-server-6f8f9c9f8f-6kx4d          1/1     Running   0          2m

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/argo-server                   ClusterIP   10.3.240.123    <none>        2746/TCP   2m
service/workflow-controller-metrics   ClusterIP   10.3.240.124    <none>        9090/TCP   2m

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/workflow-controller   1/1     1            1           3m05s
deployment.apps/argo-server           1/1     1            1           3m07s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/workflow-controller-7f8c9f8f5   1         1         1       3m33s
replicaset.apps/argo-server-6f8f9c9f8f          1         1         1       2m33s

Step 2 - Start the Argo UI for Monitoring

Argo Server has a graphical user interface that you can use to manage and monitor your Kubernetes cluster Workflows.

In this guide, you will bypass the authentication process of requesting a token to access the Argo web interface because it cannot be accessed publicly. This is called the Server Authentication mode, although you can use the other mode, Client Authentication, which requires that you request a token to be able to access the web interface.

1. Change the authentication mode to Server Authentication.

$ kubectl patch deployment \
  argo-server \
  --namespace argo \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": [
  "server",
  "--auth-mode=server"
]}]'

Output:

deployment.apps/argo-server patched

Note that this mode is not recommended for production environments, as it creates a significant security risks. A more secure option is to use the client authentication mode, which require clients to provide their Kubernetes bearer token.

2. Configure Kubernetes Role-Based Access Control (RBAC) to grant Argo Admin-level permissions for managing resources within the argo namespace.

$ kubectl create rolebinding argo-default-admin --clusterrole=admin --serviceaccount=argo:default -n argo

3. Forward the Argo server web interface port with kubectl port-forward.

$ kubectl -n argo port-forward deployment/argo-server 2746:2746

Using a browser like Chrome, visit the IP Address http://localhost:2746.

Create a New Workflow

You can use a YAML manifest to define Agro Workflows and apply to your Kubernetes cluster.

1. Create a new Workflow file.

Nano argo-workflow.yaml

2. Add the following to the file:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
    name: demo-workflow
spec:
    entrypoint: main
    templates:
    - name: main
    container:
        image: busybox
        command: ["/bin/sh"]
        args: ["-c", "echo 'The first step of the Workflow'"]

Here is a quick breakdown of the components of the file:

• entrypoint specifies the entry point for the workflow, which is defined as main.

• templates contains a list of templates, which define the steps or tasks within the workflow.

• name is the name of the template, which is set as main.

• container specifies a container-based step within the workflow.

• image specifies the Docker image to use for the container, set here as busybox.

• command specifies the command to be executed inside the container, set as ["/bin/sh"].

• args specifies the arguments to be passed to the command inside the container, set as ["-c", "echo 'The first step of the Workflow'"]. This command will run echo to print "The first step of the Workflow".

3. Apply the Workflow to your cluster:

$ kubectl -n argo create -f argo-workflow.yaml

Here's the output:

workflow.argoproj.io/hello-world-nb42c created

How to Manage Argo Workflows

1. To list all workflows within the argo namespace, do the following:

$ kubectl -n argo get wf

Here's the output:

NAME                      STATUS        AGE     MESSAGE
demo-workflow             Succeeded     4m23s

2. To see the pod logs for your Workflow, do the following:

$ kubectl -n argo logs demo-workflow

Here's the output:

This template is the first step of the Workflow
time="2024-02-13T19:56:54.629Z" level=info msg="sub-process exited" argo=true error="<nil>"

3. To delete a workflow, do this:

$ kubectl -n argo delete wf workflow-name

4. To suspend or resume a workflow, do this:

$ kubectl -n argo suspend wf workflow-name
$ kubectl -n argo resume wf workflow-name

5. To submit a workflow using the Argo CLI, do this:

$ argo submit -n argo workflow.yaml

You can learn more about Argo Workflows on their official documentation.

Conclusion

You have now explored Argo Workflows and successfully set it up. This powerful tool enables you to create logic using DAGs, or individual steps, helping you execute various tasks through different templates. You can also interact with your workflows and keep track of their progress by utilizing tools like Argo CLI, Argo UI, and Argo Events.

By using Argo Workflows, you can take advantage of Kubernetes scalability and flexibility to ensure reliable execution of tasks.

Observability involves collecting and analyzing data from sources within a system to monitor its performance and address problems effectively.

Why is Observability Useful?

1. Detecting and Troubleshooting Problems: Observability plays a role in identifying and diagnosing issues within a system. When something goes wrong, having access to data helps pinpoint the cause and resolve problems more quickly.

2. Optimizing Performance: Through monitoring metrics and performance indicators, observability helps in optimizing the performance of your system. This includes identifying bottlenecks, improving resource utilization, and ensuring operation.

3. Planning for Future Capacity: Understanding how your system behaves over time is vital for planning capacity requirements. Observability data can reveal trends, peak usage periods, and resource needs, helping your decisions regarding scaling.

4. Enhancing User Experience: By observing user interactions with your system through logs and metrics, you can improve the user experience. It assists in recognizing patterns, preferences, and potential areas that can be enhanced for user satisfaction.

Why Should I Use OpenTelementary?

Observability is essential for ensuring the reliability and availability of your Node.js applications. But manually instrumenting your code to collect and export telemetry data, such as traces, metrics, and logs, can become very stressful.

Manual instrumentation is very tedious, error-prone, and inconsistent. It can also introduce additional overhead and complexity to your application logic.

In this guide, you will learn how to use OpenTelemetry’s auto-instrumentation to help you achieve effortless Node.js monitoring.

Prerequisites

Before you go through this guide, make sure you have the following:

• A Node.js application

• A Datadog account and an API key. If you don't have one, you can sign up here to get one.

• A Backend service. You can use a backend service like Zepkin or Jaeger to store and analyze trace data. For this guide, we'll be using Jaeger.

• Some basic knowledge of Linux commands. You should be familiar with using the command line and editing configuration files.

Prepare Your Application

In this guide, you will be using a Node.js application that has two services that transfer data between themselves. You will use OpenTelemetry’s Node.js client library to send trace data to an OpenTelementay collector.

Firstly, clone the Repo Locally:

$ git clone https://github.com/<github-account>/nodejs-example.git

Then run the application:

npm install

Go to the directory of the first service using this command:

$ cd <ServiceA>

And start the first service.

$ node index.js

Then go to the directory of the second service

$ cd <ServiceB>

And start the second service.

$ node index.js

Open Service A, in this case port 5555, and input some information. Then repeat the same for Service B.

How to Set Up OpenTelementary

After starting the services, it's time to install the OpenTelementary modules you'll need for auto-instrumentation.

Here are what we need to install:

$ npm install --save @opentelemetry/api

$ npm install --save @opentelemetry/instrumentation

$ npm install --save @opentelemetry/tracing

$ npm install --save @opentelemetry/exporter-trace-otlp-http

$ npm install --save @opentelemetry/resources

$ npm install --save @opentelemetry/semantic-conventions

$ npm install --save @opentelemetry/auto-instrumentations-node

$ npm install --save @opentelemetry/sdk-node

$ npm install --save @opentelemetry/exporter-jaeger

Here's break down of what each module does:

• @opentelemetry/api: This module provides the OpenTelemetry API for Node.js.

• @opentelemetry/instrumentation: The instrumentation libraries provide automatic instrumentation for your Node.js application. They automatically capture telemetry data without requiring manual code modifications.

• @opentelemetry/tracing: This module contains the core tracing functionality for OpenTelemetry in your Node.js application. It includes the Tracer and Span interfaces, which are important for capturing and representing distributed traces within your applications.

• @opentelemetry/exporter-trace-otlp-http: This exporter module enables sending trace data to an OpenTelemetry Protocol (OTLP) compatible backend over HTTP.

• @opentelemetry/resources: This module provides a way to define and manage resources associated with traces.

• @opentelemetry/semantic-conventions: This module defines a set of semantic conventions for tracing. It establishes a common set of attribute keys and value formats to ensure consistency in how telemetry data is represented and interpreted.

• @opentelemetry/auto-instrumentations-node: This module simplifies the process of instrumenting your application by automatically applying instrumentation to supported libraries.

• @opentelemetry/sdk-node: The Software Development Kit (SDK) for Node.js provides the implementation of the OpenTelemetry API.

• @opentelemetry/exporter-jaeger: This exporter module allows exporting trace data to Jaeger. Jaeger provides a user-friendly interface for monitoring and analyzing trace data.

Configure the Node.js Application

Next, add a Node.js SDk tracer to handle the instantiation and shutdown of the tracing.

To add the tracer, create a file tracer.js:

$ nano tracer.js

Then add the following code to the file:

"use strict";

const {
    BasicTracerProvider,
    SimpleSpanProcessor,
} = require("@opentelemetry/tracing");
// Import the JaegerExporter
const { JaegerExporter } = require("@opentelemetry/exporter-jaeger");
const { Resource } = require("@opentelemetry/resources");
const {
    SemanticResourceAttributes,
} = require("@opentelemetry/semantic-conventions");

const opentelemetry = require("@opentelemetry/sdk-node");
const {
    getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");

// Create a new instance of JaegerExporter with the options
const exporter = new JaegerExporter({
    serviceName: "YOUR-SERVICE-NAME",
    host: "localhost", // optional, can be set by OTEL_EXPORTER_JAEGER_AGENT_HOST
    port: 16686 // optional
});

const provider = new BasicTracerProvider({
    resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]:
            "YOUR-SERVICE-NAME",
    }),
});
// Add the JaegerExporter to the span processor
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

provider.register();
const sdk = new opentelemetry.NodeSDK({
    traceExporter: exporter,
    instrumentations: [getNodeAutoInstrumentations()],
});

sdk
    .start()
    .then(() => {
        console.log("Tracing initialized");
    })
    .catch((error) => console.log("Error initializing tracing", error));

process.on("SIGTERM", () => {
    sdk
        .shutdown()
        .then(() => console.log("Tracing terminated"))
        .catch((error) => console.log("Error terminating tracing", error))
        .finally(() => process.exit(0));

Here is a simple breakdown of the code:

• The code starts by importing the modules BasicTracerProvider and SimpleSpanProcessor for setting up tracing from the OpenTelemetry library

• It then imports the JaegerExporter module for exporting trace data to Jaeger.

• The code creates a new instance of the JaegerExporter, specifying the service name, host, and port.

• It then creates a BasicTracerProvider and adds the JaegerExporter to the span processor using SimpleSpanProcessor.

• The provider is registered, setting it as the default provider for the application.

• An OpenTelemetry SDK instance is created, configuring it with the JaegerExporter and enabling auto-instrumentations for Node.js.

• The OpenTelemetry SDK is started, initializing tracing.

• A handler for the SIGTERM signal is set up to shut down tracing when the application is terminated.

• The code then configures the trace provider with a trace exporter. To verify the instrumentation, ConsoleSpanExporter is used to print some of the tracer output to the console.

How to Set Up OpenTelemetry to Export the Traces

Next, you'll need to write the configurations to collect and export data in the OpenTelemetry Collector.

Create a file config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  datadog:
    api: # Replace with your Datadog API key
      key: "<YOUR_DATADOG_API_KEY>"
    # Optional:
    #   - endpoint: https://app.datadoghq.eu  # For EU region

processors:
  batch:

extensions:
  pprof:
    endpoint: :1777
  zpages:
    endpoint: :55679
  health_check:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

The configuration sets up OpenTelemetry with the OTLP (OpenTelemetry Protocol) receiver and the Datadog exporter. Here’s a break down of the code:

• receivers: Specifies the components that receive the telemetry data. In this case, it includes the OTLP receiver, which supports both gRPC and HTTP protocols.

• exporters: Defines the components responsible for exporting telemetry data. Here, it configures the Datadog exporter, providing the Datadog API key. Additionally, an optional endpoint is provided for using Datadog's EU region.

• processors: Specifies the data processing components. In this case, the batch processor is used to batch and send data in larger chunks for efficiency.

• extensions: Defines additional components that extend the functionality. Here, it includes extensions for pprof (profiling data), zpages (debugging pages), and a health check extension.

• service: Configures the overall service behavior, including the extensions and pipelines. The extensions section lists the extensions to be used, and the pipelines section configures the telemetry data pipeline. Here, the traces pipeline includes the OTLP receiver, the batch processor, and the Datadog exporter.

This code is configured by the collector with the Datadog exporter to send the traces to Datadog Distributed Tracing services. However, there are other distributed tracing services that you can use like New Relic, Logzio, and Zipkin.

How to Start the Application

After correctly setting up auto-instrumentation, start the application again to test and verify the tracing configuration.

Begin by starting the OpenTelemetry Collector:

./otelcontribcol_darwin_amd64 --config ./config.yaml

The collector will start on port 4317.

Next, go to the directory of the first service:

$ cd <ServiceA>

Then start the first service with the “--require './tracer.js'” parameter to enable the application instrumentation.

$ node --require './tracer.js' index.js

Repeat this to start the second service.

Using a browser like Chrome, go to the endpoints of your two applications' services, add some data, and send some requests to test the tracing configuration.

Once the requests are made, these traces are picked up by the collector, which then dispatches them to the distributed tracing backend specified by the exporter configuration in the collector's configuration file.

It's worth noting that our tracer not only facilitates the transmission of traces to the designated backend, but also exports them to the console at the same time.

This dual functionality allows for real-time visibility into the traces being generated and sent, helping us in the monitoring and debugging processes.

Now, let’s use Jaeger UI to monitor the traces:

Start Jaeger with the following command:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.32

Using a browser, start Jaeger UI at the http://localhost:16686/ endpoint.

There you have it! The initiation of the trace starting from the inception point of one service, navigating through a sequence of operations.

This path is created as the service starts its operations, resulting in the set up of the other service to fulfill the original request you initiated earlier.

The trace provides a visual narrative of what happens between these services, offering insights into each step of the process.

How Can You Use Observability Data?

1. Monitoring Metrics: Keep an eye on key metrics such as response times, error rates, and resource usage. Sudden spikes or anomalies can indicate issues that require attention.

2. Logging: Log data provides detailed information about events and actions within a system. Analyzing logs helps in understanding the sequence of activities and tracing the steps leading to an issue.

3. Tracing: Tracing involves tracking the flow of requests or transactions across different components of a system. This helps in understanding the journey of a request and identifying any bottlenecks or delays.

4. Alerting: Set up alerts based on specific conditions or thresholds. When certain metrics exceed predefined limits, alerts can notify you in real-time, allowing for immediate action.

5. Visualization: Use graphical representations and dashboards to visualize complex data. This makes it easier to identify patterns, trends, and correlations in the observability data.

Observability, when implemented effectively, empowers teams to proactively manage and improve the performance, reliability, and user experience of their systems. It's a crucial aspect of modern software development and operations.

Conclusion

In this guide you learned how to auto-instrument Node.js applications with little code by:

• Installing and configuring the OpenTelemetry Node.js SDK and the auto-instrumentation package

• Enabling automatic tracing and metrics collection for your Node.js applications and their dependencies

• Exporting to visualize your telemetry data on a backend, Jaeger.

Using OpenTelemetry’s auto-instrumentation can help you gain valuable insights into the performance and behavior of your Node.js applications without having to manually instrument each library or framework.

CUDA is also a programming model and an API that enables programmers to write code that can run on both the CPU and GPU while also handling data transfer between them.

By using the CUDA Toolkit, you can improve performance, scalability, and efficiency across a range of applications. These include computing, deep learning, computer vision, gaming, and more.

The toolkit supports programming languages like C, C++, Fortran, Python, and Java. It seamlessly integrates with frameworks and libraries such as TensorFlow, PyTorch OpenCV, and cuDNN.

Also, the use of the CUDA Toolkit extends across different domains, such as healthcare finance, robotics, the automotive industry, and entertainment. If you're looking to speed up image processing or natural language processing, enhance cryptography, or advance ray tracing techniques, the CUDA Toolkit empowers you to solve problems faster and more efficiently.

In terms of compatibility, the CUDA Toolkit offers support for Linux distributions, including Ubuntu, Debian, Fedora, CentOS, and OpenSUSE.

In this article, I will guide you through the process of installing the CUDA Toolkit on Ubuntu 22.04, which happens to be the LTS (Long Term Support) version of Ubuntu.

Prerequisites

To install the CUDA Toolkit on Ubuntu 22.04, you need the following:

• A supported NVIDIA GPU with a minimum compute capability of 3.0

• NVIDIA driver compatible with the CUDA Toolkit version

In this guide I will be using a Paperspace GPU instance with Ubuntu 22.04 LTS operating system.

Please note that you can use any other Cloud Service provider, like Google Cloud and Vultr, or even your own computer, as long as it meets the requirements listed above.

To get started, you'll need to create a new user, like seconduser and then switch to the new user.

Install CUDA Toolkit

You can install CUDA using the release file or alternatively, by using Conda. In this guide, we will be installing CUDA with the release file from the official Toolkit Archive.

Step 1: Download the CUDA release file.

 $ wget https://developer.download.nvidia.com/compute/cuda/12.0.1/local_installers/cuda_12.0.1_525.85.12_linux.run

Step 2: Execute the release file.

 $ sudo sh cuda_12.0.1_525.85.12_linux.run

You will be prompted to accept the End User License Agreement, then press Enter to configure your installation.

Once the installation is complete, you should see an output similar to this:

Configuration and Verification

Step 1: Configure the server

Configure the server to work with the CUDA toolkit. Move the CUDA path to the system’s PATH, then add the CUDA Toolkit library path to the LD_LIBRARY_PATH so that the CUDA toolkit link loader will be updated with the location of shared libraries.

  $ echo "export PATH=/usr/local/cuda-12.0/bin${PATH:+:${PATH}}" >> /home/seconduser/.bashrc

  $ echo "export LD_LIBRARY_PATH=/usr/local/cuda-12.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> /home/seconduser/.bashrc

Step 2: Activate Environment

After configuring the server to work with the CUDA toolkit, activate the environment variable changes so that the system can find and use CUDA.

$ source /home/seconduser/.bashrc

Step 3: Verify Installation

 $ nvidia-smi

Output:

Step 4: Check Package Installation.

Verify that the package from the CUDA Toolkit is successfully installed on your server.

 $ nvcc --version

Output:

Testing

To test your newly installed CUDA programs, you will use some already-made test scripts by CUDA that will allow you to comprehensively verify the compatibility and functionality of your CUDA-enabled environment.

Step 1: Clone the test scripts repository

 $ git clone https://github.com/NVIDIA/cuda-samples.git

Step 2: Go to the directory containing the deviceQuery sample script.

 $ cd cuda-samples/Samples/1_Utilities/deviceQuery

Step 3: Compile the script.

 $ make

Step 4: Run the script.

 $ ./deviceQuery

Your output should look similar to the one below if your CUDA program ran the script successfully:

Conclusion

In this article, you learned how to install the CUDA Toolkit on Ubuntu 22.04.

Some of the best practices for using CUDA on Ubuntu are:

• Keep your system and NVIDIA drivers up to date to ensure the compatibility and stability of the CUDA Toolkit.

• Use the CUDA APT PPA to install and update the CUDA Toolkit easily and quickly.

• Use the nvcc compiler options and flags to optimize and debug your CUDA code.

• Use the CUDA libraries and tools to enhance and simplify your CUDA development process.

• Follow the CUDA coding standards and best practices to write efficient and maintainable CUDA code.

Here are some resources to learn more about CUDA:

• CUDA Official Docs

• CUDA Refresher Tutorial

• Read The Docs: Say Hello to CUDA

But unfortunately, they often become targets for hackers and malicious individuals seeking to exploit any vulnerabilities to compromise data and disrupt functionality. As a result, you'll need to prioritize the security of your web server by updating it and implementing safeguards against threats.

To enhance the security of your web server, one effective approach is to use Continous Integration (CI). CI is a DevOps technique that allows the automated merging of code modifications from software engineers into a single repository. This practice enhances code quality, minimizes bugs and speeds up code delivery.

By using CI, you can automate the testing, building, and deployment processes for your web servers’ code and configuration. You can also ensure that your web server consistently maintains a stable state.

In this tutorial, I'll guide you through the process of strengthening the security of your web server by using two popular and powerful tools: NGINX and CircleCI.

NGINX, which is an open source web server, provides a range of features and modules that can greatly enhance the security of your web server. These include SSL/TLS encryption, security headers, and support for HTTP/2.

On the other hand, CircleCI offers both cloud-based and self-hosted options for Continous Integration (CI) and Continuous Delivery (CD), enabling seamless deployment processes.

By following this guide, you will learn how to:

• Configure NGINX to use SSL/TLS encryption and security headers

• Create a GitHub repository and push your NGINX configuration files to it

• Create a CircleCI project and link it to your GitHub repository

• Create a CircleCI configuration file and define your CI pipeline

• Test and deploy your web server with CircleCI

Here is what we'll cover:

• Prerequisites

• Step 1: Configure NGINX to Use SSL/TLS Encryption

• Step 2: Configure NGINX to Include Security Headers

• Step 3: Create a GitHub Repository and Push Your NGINX Configuration

• Step 4: Create a CircleCI Project and Link it to Your GitHub Repository

• Step 5: Create a CircleCI Configuration File and Define Your CI Pipeline

• Step 6: Test and Deploy Your Web Server with CircleCI

Let's get started!

Prerequisites

Before you start this guide, you need to ensure you have the following:

• A web server running NGINX. If you don't have one, you can follow this guide to install NGINX on Ubuntu 20.04. You can also use any other operating system or cloud provider that supports NGINX.

• A GitHub account. If you don't have one, you can sign up for free here.

• A CircleCI account. If you don't have one, you can sign up for free here. You will also need to link your GitHub account to your CircleCI account.

• Some basic knowledge of web development and Linux commands. You should be familiar with the concepts of web servers, SSL/TLS encryption, security headers, and CI. You should also be comfortable with using the command line and editing configuration files.

Once you have these, you are ready to proceed with the next steps.

Step 1: Configure NGINX to Use SSL/TLS Encryption

SSL/TLS encryption ensures the transmission of data between your web server and clients. It safeguards against interception or manipulation of information. It also plays a role in verifying the identity and reliability of your web server.

You need an SSL/TLS certificate to use SSL/TLS for your web server. An SSL/TLS certificate contains information about your web server, such as its domain name, owner, and public key. The validity of the certificate is verified through the unique digital signature from a Certificate Authority (CA).

You can either purchase an SSL/TLS certificate from a commercial CA, such as DigiCert, Symantec, or GlobalSign, or you can just get one for free from a non-profit CA, such as Let's Encrypt. You can also create your own self-signed certificate, but this is not recommended for production use, as it will not be trusted by most browsers and clients.

In this guide, you will use Let's Encrypt to get a free SSL/TLS certificate for your web server. To use Let's Encrypt, you need to install a software client on your web server that can communicate with the CA and perform the necessary tasks.

One of the most common and recommended clients for Let's Encrypt is Certbot. Certbot is a command-line tool that can automatically request, install, and renew certificates for your web server. It can also configure your web server to use the certificates and enable HTTPS.

To install Certbot on your web server, run the following commands:

sudo apt update
sudo apt install certbot python3-certbot-nginx

After installing Certbot, use it to request and install a certificate for your web server. You need to provide your domain name and your email address for the certificate.

To request and install a certificate for your web server, run the following command:

sudo certbot --nginx -d yourdomain.com

Replace yourdomain.com with your actual domain name.

Follow the prompts and answer the questions. Certbot will automatically verify your domain ownership, obtain a certificate, and install it on your web server. It will also ask you whether you want to redirect all HTTP traffic to HTTPS. Choose option 2 to enable the redirection.

After the process is completed, you will see a message like this:

You have now successfully configured NGINX to use SSL/TLS encryption with a certificate from Let's Encrypt. You can now access your web server using HTTPS and see the lock icon in your browser.

You can also test your web server security using online tools, such as SSL Labs. You should see a grade of A or higher.

Note: Let's Encrypt certificates are valid for 90 days. Certbot can automatically renew them for you before they expire.

To enable the automatic renewal, you need to create a CRON job or a systemd timer that runs the following command at least once per day:

sudo certbot renew

You can also test the renewal process manually by running the following command:

sudo certbot renew --dry-run

This will perform a trial run without making any changes.

If you encounter any errors or issues, you can check the Certbot documentation or the Let's Encrypt community forum for help.

Step 2: Configure NGINX to Include Security Headers

Security headers help instruct the browser to apply certain security policies or restrictions when handling your web content. They can prevent or mitigate common web attacks, such as cross-site scripting (XSS), clickjacking, content injection, and more.

In this step, you will be adding security headers to your NGINX configuration. These headers include X Frame Options, X Content Type Options, X XSS Protection, and Content Security Policy.

X-Frame-Options

The X-Frame-Options header tells the browser whether or not to allow your web page to be displayed in a frame, iframe, embed, or object element. This can help you prevent clickjacking attacks, where an attacker overlays a hidden frame on top of your web page and tricks the user into clicking on it.

There are three possible values for this header:

• DENY: This value prevents your web page from being displayed in any frame.

• SAMEORIGIN: This value allows your web page to be displayed in a frame only if the frame is from the same origin as your web page.

• ALLOW-FROM URI: This value allows your web page to be displayed in a frame only if the frame is from the specified URI.

To enable the X-Frame-Options header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header X-Frame-Options "SAMEORIGIN";

This will allow your web page to be displayed in a frame only if the frame is from the same origin as your web page. You can change the value to DENY or ALLOW-FROM uri according to your needs.

Save the file and restart NGINX to apply the changes.

X-Content-Type-Options

The X Content Type Options header instructs the browser to perform MIME-type sniffing. This feature attempts to determine the content type of a resource by analyzing its content or file extension.

By using this header you can safeguard against content injection attacks. These attacks involve uploading a file with a content type to exploit the browser’s interpretation of it.

There is only one possible value for this header:

• nosniff: This value prevents the browser from performing MIME type sniffing.

To enable the X-Content-Type-Options header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header X-Content-Type-Options "nosniff";

This will prevent the browser from performing MIME type sniffing on your web resources.

Save the file and restart NGINX to apply the changes.

X-XSS-Protection

The X-XSS-Protection header tells the browser to enable or disable its built-in XSS filter, which can detect and block some types of XSS attacks. This can help you prevent XSS attacks, where an attacker injects malicious code into your web page that executes in the browser.

There are three possible values for this header:

• 0: This value disables the XSS filter.

• 1: This value enables the XSS filter and sanitizes the page if an XSS attack is detected.

• 1; mode=block: This value enables the XSS filter and blocks the page if an XSS attack is detected.

To enable the X-XSS-Protection header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header X-XSS-Protection "1; mode=block";

This will enable the XSS filter and block the page if an XSS attack is detected. You can change the value to 0 or 1 according to your needs.

Save the file and restart NGINX to apply the changes.

Content-Security-Policy

The Content-Security-Policy header tells the browser to enforce a set of rules that restrict what sources and types of content can be loaded and executed on your web page. This can help you prevent XSS, content injection, and other types of attacks that rely on loading malicious or untrusted content.

The value of this header is a complex policy that consists of multiple directives and values. Each directive specifies a type of content and a list of sources that are allowed or disallowed for that content.

For example, the following policy allows only scripts styles from the same origin, and images from the same origin or yourdomain.com:

Content-Security-Policy: default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' yourdomain.com;

The Content-Security-Policy header is very powerful and flexible, but also very complicated and error-prone. You need to carefully design and test your policy to ensure that it does not break your web functionality or introduce new vulnerabilities. You can use tools like CSP Evaluator or CSP Scanner to check and improve your policy.

To enable the Content-Security-Policy header in NGINX, add the following line to your server block in your NGINX configuration file (/etc/nginx/sites-enabled/example.conf):

add_header Content-Security-Policy "default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self' yourdomain.com;";

This will enforce the policy that you described above. You can change the policy according to your needs.

Save the file and restart NGINX to apply the changes.

Step 3: Create a GitHub Repository and Push Your NGINX Configuration

To create a GitHub repository and push your NGINX configuration files to it, follow these steps:

Create a GitHub repository

First, log in to your GitHub account and go to the GitHub homepage.

Click on the plus icon in the top right corner of the page, and select "New repository" from the dropdown menu.

On the next page, enter a name for your repository in the "Repository name" field. This should be a short and descriptive name that accurately reflects the contents of the repository. For example, you can name it "nginx-config".

In the "Description" field, you can enter a longer description of the repository if you want. This is optional, but it can be helpful to provide more information about the purpose of the repository.

For example, you can write "A repository for storing and managing my NGINX configuration files".

You can set the visibility to whatever you prefer. If you want others to be able to see your work, set it to "Public". Otherwise, set it to "Private".

Leave the "Initialize this repository with a README" option unchecked, as you want to create an empty repository.

Click on the "Create repository" button to create the repository.

Your new empty repository will be created and you will be taken to the repository page.

Push your NGINX configuration files to the GitHub repository

On your web server, navigate to the directory where your NGINX configuration files are located. By default, this is /etc/nginx on most Linux distributions.

Initialize a new Git repository in this directory by running the following command:

git init

This will create a new .git directory in the current directory, which will be used to store all the version control information for your project.

Add all the configuration files that you want to include in the repository by running the following command:

git add .

This will add all the files in the current directory and its subdirectories to the repository. You can also specify individual files to be added by replacing the (.) with the file names, separated by spaces.

Then commit the files to the repository by running the following command:

git commit -m "Initial commit"

This will create the first commit in the repository, which will include all the files that were added in the previous step. The -m flag is used to specify a commit message, which should briefly describe the changes that were made in this commit.

Go back to your GitHub repository page and copy the URL of your repository. You can find it under the "Code" section. It should look something like this:

On your web server, add the URL of your GitHub repository as a remote for your Git repository by running the following command:

git remote add origin https://github.com/username/nginx-config.git

Replace username with your GitHub username and nginx-config with your repository name. The origin is the name of the remote, which you can change to anything you want.

Push your local Git repository to the GitHub repository by running the following command:

git push -u origin master

This will push your master branch, which is the default branch in Git, to the origin remote, which is the GitHub repository that you created. The -u flag is used to set the upstream for your branch, which means that you can use git push or git pull without specifying the remote or the branch in the future.

Enter your GitHub username and password when prompted. If you have enabled two-factor authentication, you will need to use a personal access token instead of your password. You can generate one from your GitHub settings page.

You have successfully created a GitHub repository and pushed your NGINX configuration files to it. You can now view and manage your configuration files on GitHub.

Step 4: Create a CircleCI Project and Link it to Your GitHub Repository

CircleCI is a platform that offers cloud-based and self-hosted options for continuous integration and delivery. It allows you to create and run pipelines that automate and streamline your web server deployment and update process.

To use CircleCI, you need to create a CircleCI project and link it to your GitHub repository. This will enable CircleCI to access your code and configuration files, and trigger builds whenever you push to GitHub.

To create a CircleCI project and link it to your GitHub repository, follow these steps:

Sign up for CircleCI and connect your GitHub account

Start by logging in to your CircleCI account or sign up for free here.

On the CircleCI dashboard, click on the "Create Project" button in the top right corner of the page.

On the next page, select "GitHub" as your version control provider and click on the "Connect with GitHub" button.

On the next page, authorize CircleCI to access your GitHub account by clicking on the "Authorize circleci" button.

On the next page, Enter a “Project Name” and follow the remaining instructions to successfully create a CircleCI project.

Create a CircleCI project and link it to your GitHub repository

Next, create a new SSH key pair in your terminal:

ssh-keygen -t ed25519 -f ~/.ssh/project_key -C email@example.com

Then copy the private key generated:

pbcopy < ~/.ssh/project_key

Next, enter it in the private key field:

You will see a list of your GitHub repositories. Find the repository that you created in the previous step and click on the "Create Project" button next to it.

Now you will see a list of templates for different languages and frameworks. Since you are using Python and Flask, select the "Python" template and click on the "Use this config" button.

On the next page, you will see the generated CircleCI configuration file (config.yml) that defines your pipeline. You can review and edit the file if you want, or leave it as it is for now. Click on the "Start building" button to create the project and link it to your GitHub repository.

Your new CircleCI project will be created and linked to your GitHub repository.

You have now successfully created a CircleCI project and linked it to your GitHub repository. You can now configure and run your pipeline on CircleCI.

Step 5: Create a CircleCI Configuration File and Define Your CI Pipeline

A CircleCI configuration file is a YAML file that defines your CI pipeline. A CI pipeline is a sequence of jobs that run whenever you push changes to your GitHub repository. Each job consists of steps that perform specific tasks, such as running commands, installing dependencies, or deploying your web server.

In this step, you will create a CircleCI configuration file and define your CI pipeline. You will use the Python template that you selected in the previous step as a starting point, and modify it to suit our needs. You will also explain what each step in the pipeline does and how it helps to automate and secure your web server deployment.

Create a CircleCI configuration file

On your web server, navigate to the directory where your NGINX configuration files are located. By default, this is /etc/nginx on most Linux distributions.

Create a new directory called .circleci in this directory by running the following command:

mkdir .circleci

This is where you will store our CircleCI configuration file.

Then create a new file called config.yml in the .circleci directory by running the following command:

touch .circleci/config.yml

This is your CircleCI configuration file.

Open the config.yml file with your preferred text editor and paste the following code:

version: 2.1

jobs:
  build:
    docker:
      - image: cimg/python:3.9
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: |
            pip install -r requirements.txt
      - run:
          name: Run tests
          command: |
            pytest
  deploy:
    machine:
      image: ubuntu-2004:202101-01
    steps:
      - checkout
      - add_ssh_keys:
          fingerprints:
            - "YOUR_FINGERPRINT"
      - run:
          name: Deploy Nginx configuration
          command: |
            scp -r nginx root@YOUR_IP:/etc
            ssh root@YOUR_IP "systemctl restart nginx"

workflows:
  version: 2
  build-and-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:
            branches:
              only: main

This is your CircleCI configuration file that defines your CI pipeline. You will explain each part of the file in the next section.

Finally, fave and close the file.

Define your CI pipeline

Let's go through each part of the config.yml file and see what it does.

• Line 1: This indicates the version of the CircleCI platform you are using. 2.1 is the most recent version.

• Line 3: The jobs level contains a collection of children, representing your jobs. You specify the names for these jobs, for example, build, test, deploy.

• Line 6: This is the Docker image. The example shows cimg/python:3.9, which is a CircleCI-provided image that contains Python 3.9 and other common tools.

• Line 9: The run directive executes a shell command or script. You can specify a name and a command for each run directive.

• Line 11: The command attribute is a list of shell commands that you want to execute. In this case, you are installing the dependencies for your web application using pip.

• Line 12: This is another run directive that runs the tests for your web application using pytest.

• Line 13: deploy is the second child in the jobs collection. This job is responsible for deploying your NGINX configuration to your web server.

• Line 14: This specifies that you are using a machine executor for this job. A machine executor provides a full virtual machine with root access and various tools installed.

• Line 15: This is the machine image. The example shows ubuntu-2004:202101-01, which is a CircleCI-provided image that contains Ubuntu 20.04 and other common tools.

• Line 16: The steps collection is a list of run directives and other commands that you want to execute in this job.

• Line 18: The add_ssh_keys command adds your SSH keys to the machine. You need to provide the fingerprints of the keys that you want to use. You can generate and add SSH keys from your CircleCI settings page.

• Line 21: The command attribute is a list of shell commands that you want to execute. In this case, you are using SCP to copy your NGINX configuration files from the machine to your web server, and SSH to restart the NGINX service on your web server. You need to replace YOUR_FINGERPRINT with the fingerprint of your SSH key, and YOUR_IP with the IP address of your web server.

• Line 24: This indicates the version of the workflow syntax you are using. 2 is the most recent version.

• Line 29: The required attribute specifies the dependencies of this job. In this case, you are saying that the deploy job requires the build job to finish successfully before running.

• Line 30: The filters attribute specifies the conditions for running this job. In this case, you are saying that the deploy job should only run on the main branch of our GitHub repository.

Push your CircleCI configuration file to your GitHub repository

On your web server, add, commit, and push your CircleCI configuration file to your GitHub repository by running the following commands:

git add .circleci/config.yml
git commit -m "Add CircleCI config file"
git push origin main

This will trigger a new build on CircleCI and run your CI pipeline.

Go to your CircleCI dashboard and click on the build-and-deploy workflow.

You can click on each job to see the details and logs of the steps.

Wait for the workflow to finish.

You have successfully created a CircleCI configuration file and defined your CI pipeline. You can now automate and secure your web server deployment with CircleCI. You can also modify and improve your configuration file according to your needs.

Step 6: Test and Deploy Your Web Server with CircleCI

Now that you have created a CircleCI project and a configuration file for your CI pipeline, you can test and deploy your web server with CircleCI. You can trigger and monitor your CI pipeline from the CircleCI web app or the command line. You can also verify that your web server is deployed and secured correctly by using online tools or by accessing your web server using HTTPS.

Trigger and monitor your CI pipeline from the CircleCI web app

To trigger and monitor your CI pipeline from the CircleCI web app, follow these steps:

• Go to the CircleCI dashboard.

• On the dashboard, you will see a list of your projects and pipelines. Find the project that you created in the previous step and click on it.

• On the project page, you will see a list of your branches and workflows. Find the branch that you pushed your CircleCI configuration file to in the previous step and click on the build-and-deploy workflow.

• On the workflow page, you will see a graphical representation of your pipeline, showing the status and duration of each job and step. You can click on each job or step to see the details and logs of the commands that were executed.

• Wait for the workflow to finish. If everything goes well, you will see a green check mark next to each job and step, indicating that they were successful.

You have successfully triggered and monitored your CI pipeline from the CircleCI web app. You can also trigger and monitor your CI pipeline from the command line.

Trigger and monitor your CI pipeline from the command line

To trigger and monitor your CI pipeline from the command line, follow these steps:

• On your web server, navigate to the directory where your NGINX configuration files are located. By default, this is /etc/nginx on most Linux distributions.

• Make some changes to your configuration files, such as adding or removing security headers, and save them.

• Add, commit, and push your changes to your GitHub repository by running the following commands:

git add .
git commit -m "Update Nginx configuration"
git push origin main

This will trigger a new build on CircleCI and run your CI pipeline.

To monitor your CI pipeline from the command line, you can use the CircleCI CLI, which is a tool that allows you to interact with CircleCI from your terminal. You can install the CircleCI CLI by following the instructions on the official website.

After installing the CircleCI CLI, you can use the circleci command to perform various actions, such as listing your projects, pipelines, workflows, jobs, and artifacts. You can also use the --help flag to see the available options and arguments for each command.

To monitor your CI pipeline from the command line, you can use the circleci pipeline command to list and describe your pipelines.

For example, you can run the following command to list the pipelines for your project:

circleci pipeline list --org-slug <VCS>/<your-vcs-org-or-username> --project-slug <VCS>/<your-repo-name>

Replace <VCS> with either gh or bb depending on your version control system. Replace <your-vcs-org-or-username> with your GitHub or Bitbucket organization or username. Replace <your-repo-name> with your repository name. You will see something like this:

You can use the pipeline ID or number to describe a specific pipeline and see its details, such as the status, workflows, jobs, and steps. For example, you can run the following command to describe the latest pipeline for your project:

circleci pipeline describe --org-slug <VCS>/<your-vcs-org-or-username> --project-slug <VCS>/<your-repo-name> --pipeline-number <number>

Replace <number> with the pipeline number that you want to describe.

Wait for the pipeline to finish. If everything goes well, you will see a success message for each job and step, indicating that they were successful.

You have successfully triggered and monitored your CI pipeline from the command line. You can also verify that your web server is deployed and secured correctly.

Verify that your web server is deployed and secured correctly

To verify that your web server is deployed and secured correctly, you can use online tools or access your web server using HTTPS. Here are some examples:

To verify that your web server is using the latest Nginx configuration that you pushed to GitHub, you can use a tool like curl or wget to make a request to your web server and inspect the response headers.

For example, you can run the following command to see the security headers that your web server is sending:

curl -I https://www.yourdomain.com

Replace yourdomain.com with your actual domain name.

You can compare the headers with the ones that you configured in your NGINX configuration file and see if they match.

To verify that your web server is using the SSL/TLS certificate that you installed with Certbot, you can use a tool like SSL Labs or HTBridge to scan your web server and check its SSL/TLS configuration and rating. You can check the details of your certificate, such as the issuer, validity, and chain. You can also check the grade of your SSL/TLS configuration, which should be A or higher.

To verify that your web server is accessible and functional using HTTPS, you can simply open your web browser and visit your web server using HTTPS. For example, you can go to https://www.yourdomain.com and see your web page.

You can check the lock icon in your browser, which indicates that your connection is secure. You can also click on the icon and see the details of your certificate and connection.

Conclusion

In this article, you have learned how to secure your web server using NGINX and CicleCI. NGINX and CircleCI, when used together, can provide a powerful solution for ensuring the continuous security of your web applications.

Stay ahead of the curve by integrating these technologies into your workflow, and empower your team to deliver secure and reliable web services.

Please don't forget to share this guide with your colleagues, friends, and online communities if you find it insightful.

There is no global way of storing and sharing secrets in containers. Traditional methods of storing them, such as hardcoding them in source code and tucking them away in text files, leave them vulnerable to exposure and exploitation.

Docker Secrets was introduced to address this issue. Docker secrets offer a secure way to store your sensitive data, safeguarding your applications from the pitfalls of careless storage.

In this article, you will learn what Docker secrets are and their default structures. I will also walk you through the steps on how to manage secrets in Docker to safeguard your sensitive data and unlock the full potential of your Docker applications.

What are Docker Secrets?

To effectively manage secrets, you must first understand what they are. Docker secrets include a wide array of sensitive data, including:

• Credentials such as usernames and passwords for databases, servers, third-party services, and more.

• API keys, which are unique keys that grant access to external APIs and services.

• Digital certificates, which are used for secure communication and authentication, such as SSL/TLS certificates.

• Encryption keys, which are keys used to encrypt sensitive data or files.

• Tokens, such as Access tokens for authentication and authorization purposes.

• Software licenses that may contain sensitive information.

• Other sensitive data that could pose a security risk if exposed.

How Secrets Are Stored

Docker offers several pathways to create and reference secrets within your containerized environment:

• Files: You can store secrets in plain text files, but this method is less secure and not recommended for production environments.

• Environment Variables: You can set secrets as environment variables within containers, but this may still expose them in logs and process listings.

• Docker Secrets: You can utilize Docker's built-in secrets management feature for basic encryption and access control.

• Docker Compose: You can define secrets within your Docker Compose files for convenient management during development and testing.

• Docker Swarm Secrets: You can leverage advanced secret management capabilities for clustered Docker environments, providing secure storage and granular access control.

Solutions for Managing Key Secrets

Docker offers two main ways to manage your sensitive data: built-in solutions within the platform itself and external solutions for more advanced needs.

Built-in solutions

For those starting out or seeking basic secret management, Docker's built-in solutions offer a convenient entry point (this guide is based on the built-in solutions):

• Docker Secrets: This lightweight method allows creating and injecting basic secrets into containers via CLI or Compose files. While easy to use, it lacks advanced features like centralized storage and granular access control. It is best suited for simple deployments with minimal secrets.

• Docker Swarm Secrets: For clustered environments, Swarm secrets offer a step up in security. Secrets reside securely on Swarm managers and are distributed to nodes on demand. You gain granular access control and audit trails, making it ideal for production deployments with multiple secrets.

External solutions

For robust security and advanced features, external secrets management platforms stand out. Here are some popular options:

• Vault: A feature-rich, open-source platform offering encryption, access control, logging, and audit trails. It integrates seamlessly with various tools and platforms, making it a versatile choice for complex deployments.

• Keywhiz: Another open-source option, Keywhiz, focuses on ease of use and cloud-native deployments. Its lightweight design makes it ideal for managing secrets in Kubernetes environments.

• AWS Secrets Manager: If you're an AWS user, this native service provides secure storage, rotation, and retrieval of secrets, integrating seamlessly with your existing infrastructure.

• Cloud Native Options: Most major cloud providers offer dedicated secrets management services like Azure Key Vault and GCP Secret Manager. These leverage the security and scalability of their respective platforms for secure and streamlined management.

How to choose the right solution

The optimal approach depends on your personal or team’s specific needs and environment. Consider these factors:

• Application complexity: Complex applications with numerous secrets likely require the advanced features of an external solution.

• Deployment environment: Production environments demand robust security and access control, favoring external options.

• Team expertise: Evaluate your team's familiarity with managing and integrating external tools.

• Scalability needs: If you envision scaling your containerized infrastructure, choose solutions that cater to multi-node deployments.

Remember, there's no one-size-fits-all approach. Prioritize secure practices for creating, storing, rotating, and deleting secrets, regardless of your chosen method.

How to Get Started with Managing Secrets

Now, let's get practical, as I will show you how to create and manage Docker Secrets.

How to create a Docker secret

Start by creating a file with a secret, like a file containing your password:

$ echo yourpassword > password.txt

Next, create the Docker secret object using the docker secret command:

$ docker secret create your_secret ./path/to/password.txt

In the process of creating and managing Docker Secrets, you start by generating a file containing a secret, such as a password, using the echo command and saving it to a file (such as password.txt).

After that, you use the docker secret create command to establish a Docker secret object named your_secret from the contents of the specified file (./path/to/password.txt). This sequence of commands enables the creation and storage of confidential information, like passwords, into your Docker environment securely.

How to initiate a service using the secret

Now to create a service that uses the secret, use this template:

$ docker service create --name <service_name>  --secret <secret_name>   <image_name>:<tag>

The docker service create command starts the creation of a new service within a Docker Swarm cluster. To customize and configure the service, several options are available:

1. --name <service_name>: Assigns a specific name to the service, aiding in easy identification. For example, --name my-nginx-service designates the service as my-nginx-service.

2. --secret <secret_name>: Uses an existing Docker Swarm secret into the service. This allows the secret to be accessed within the containers associated with the service. For instance, --secret your_secret associates the your_secret secret with the service.

3. <image_name>:<tag>: Specifies the Docker image to be utilized for the service, including its tag. For example, <image_name>:<tag> can be replaced with nginx:latest to use the latest version of the Nginx image for the service.

How to Manage Docker Secrets

How to list Docker secrets

You can check out the list of Docker Secrets available by using the Docker ls command:

$ docker secret ls

This is what the output should look like:

This command does not provide comprehensive information about your Docker secrets, as only the metadata is displayed. To inspect a specific secret's metadata, use the Docker Inspect command:

$ docker secret inspect <secret_name> #use 'your_secret' for this example

When executed, it provides a comprehensive output that includes metadata about the specified secret, such as its ID, version, and labels. Additionally, it displays the secret's creation and update timestamps. This inspection command is valuable for administrators and developers seeking to understand the properties and attributes of a Docker secret, aiding in effective management and utilization within the Docker Swarm cluster.

However, Docker secrets prioritize security, and unveiling their contents through CLI commands would compromise their integrity. To access the secret's content, it must be attached to a service within your Docker Swarm. Subsequently, the secret becomes accessible as a file within the containers associated with that service.

How to remove Docker secrets

To remove and delete unused Docker Secrets use the docker secret rm command:

$ docker secret rm your_secret

Remember, this action is irreversible, so make sure to double-check your choice!

How to Use Docker Secrets with Docker Compose

If you are not a big fan of using the command-line interface, then don’t worry, as you can still use Docker compose files to manage Docker secrets.

version: '3.1'

services:
  web:
    image: nginxdemos/hello
    secrets:
      - your_secret
      - your_external_secret
  nginx:
    image: nginx:latest
    ports:
      - "80:80"
    volumes:
      - ./conf/nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - web

secrets:
  your_file_secret:
    file:  ./path/to/password.txt
  your_external_secret:
    external: true

This YAML configuration is a Docker Compose file, specifying the setup for two services –web and nginx– along with some associated configurations and secrets.

Let's me breakdown what I did.

Services configuration

The web service uses the image nginxdemos/hello and includes two secrets: your_secret and your_external_secret.

The nginx service uses the nginx:latest image and maps port 80 on the host to port 80 on the container.

Also, it mounts the local file ./conf/nginx.conf into the container at /etc/nginx/nginx.conf and depends on the web' service, which means that web needs to be running before nginx starts.

Secrets configuration

There are two secrets defined.

The first one is your_file_secret, which reads the content of ./path/to/password.txt and creates a secret from it.

The second one is your_external_secret, which specifies that this secret is external. This implies that the actual secret content is managed externally, and the configuration here is a reference to that external secret.

In summary, this Docker Compose file example sets up two services, web and nginx, with the nginx service depending on web. Secrets, both from files (your_file_secret) and external sources (your_external_secret), are utilized for the secure handling of sensitive information within the services. The nginx service also has additional configurations related to port mapping and volume mounting.

Best Practices

Below are some tips and best practices to follow when working with Docker Secrets.

How to Choose a Method

Selecting the right method depends on several factors:

• Application Needs: Consider the complexity of your application and its security requirements. Simple applications might thrive with built-in Docker secrets, while complex deployments might necessitate an external solution.

• Infrastructure: Analyze your existing infrastructure and tools. If you already utilize specific cloud platforms or orchestration engines, their native secrets management solutions might offer seamless integration and familiarity.

• Desired Control: Assess your need for advanced features like granular access control, automated rotation, and centralized management. External solutions typically offer greater control compared to built-in options.

How to Secure Secret Creation and Storage

The foundation of secure secrets management lies in protecting the data itself:

• Encryption: Implement encryption for secrets both at rest (stored on disk) and in transit (transmitted between systems). Utilize strong encryption algorithms and key management best practices.

• Access Control: Enforce the principle of least privilege. Grant access to secrets only on a need-to-know basis. Consider employing role-based access control (RBAC) mechanisms for granular control.

Secret Rotation and Lifecycle Management

Preventing the compromise of even a single secret can significantly mitigate risk. Employ these measures:

• Rotation: Regularly rotate your secrets, even if they haven't been exposed. Define automated rotation schedules based on best practices and your specific security needs.

• Lifecycle Management: Implement secure deletion processes for outdated or unused secrets. Avoid simply deleting files, as data recovery tools might still access them. Secure deletion methods offer a safer approach.

How to Integrate Secrets with CI/CD Pipelines

Incorporate secrets management seamlessly into your development workflow:

• Injection Techniques: Utilize environment variables, dynamic secret injection tools, or file mounts to provide secrets to containers only when needed. This avoids their storage within container images.

• Automated Credential Management: Integrate your chosen secrets management solution with your CI/CD pipeline to automate credential retrieval, rotation, and injection as part of your deployment process.

• Minimize Exposure: Avoid logging secrets in plain text during the build or deployment stages. Implement masking techniques or utilize tools that handle secrets securely within your CI/CD pipeline.

Using these best practices and customizing them to your specific needs, you can build a robust and secure foundation for managing your Docker secrets.

Remember, the solution you choose is just one piece of the puzzle. Constant vigilance, adherence to best practices, and regular security audits are crucial for maintaining a resilient defense against potential threats.

Additional Resources

Below are some additional resources to use if you want to learn more about Docker Secrets and other security practices:

• Docker documentation Official guides for Docker secrets and secure practices.

• Secret management platforms guides: Each solution provides extensive documentation and tutorials.

  • Official Docker Docs guide on managing Docker secrets (read for more in-depth knowledge)

  • A guide to using an external solution: Vault

  • A guide to AWS Secrets manager

• Community Forums and Blogs: Actively engage with the security and Docker communities for further learning and support.

  • Stackoverflow

  • The official blog page of the Cloud Native Computing Foundation (CNCF)

Conclusion

Now, in this guide, you've learned what Docker secrets are, how they are stored, the different methods of storing them, and how to manage Docker secrets in your personal projects.

Remember, secure secrets are the cornerstones of secure applications. Build your fortress wisely, and your data will remain safe and sound.

Please feel free to ask if you have any specific questions on any of these aspects!

Also, if you liked this guide and found it insightful, please make sure to share it with your colleagues and online communities.