But traffic between your pods? Plaintext by default. Ingress traffic from the internet to your services? Only encrypted if you explicitly configure TLS. And certificates for internal services? You have to provision those yourself.

This is not a Kubernetes oversight. It's a deliberate design choice — Kubernetes provides the primitives and leaves the implementation to you. The problem is that certificate management is notoriously painful. Certificates expire. Provisioning them manually doesn't scale. Forgetting to rotate them causes outages.

cert-manager solves this. It runs as a controller inside your cluster, watches for Certificate resources, requests certificates from configured issuers, stores them in Kubernetes Secrets, and rotates them automatically before they expire. You declare what you want, cert-manager makes it happen and keeps it that way.

In this article you'll work through how cert-manager's core model works, automate public Ingress TLS using Let's Encrypt, set up an internal Certificate Authority for service-to-service encryption, and understand how certificate rotation works so outages caused by expired certificates become a thing of the past.

Prerequisites

• A kind cluster with the nginx Ingress controller installed

• Helm 3 installed

• A domain name with DNS you control — needed for the Let's Encrypt demo

• Basic understanding of TLS: you know what a certificate, a private key, and a CA are

All demo files are in the DevOps-Cloud-Projects GitHub repository.

Table of Contents

• What Is and Isn't Encrypted in Kubernetes

• How cert-manager Works

  • The Four Core Resources

  • Issuers and ClusterIssuers

  • The Certificate Lifecycle

  • ACME Challenges: HTTP-01 vs DNS-01

• Demo 1 — Install cert-manager and Issue a Let's Encrypt Certificate

• How to Get a Wildcard Certificate with DNS-01

• Demo 2 — Set Up an Internal CA for Service-to-Service TLS

• How Certificate Rotation Works

• Cleanup

• Conclusion

What Is and Isn't Encrypted in Kubernetes?

Before installing anything, it's worth being precise about what the cluster already protects and what it leaves open.

The two gaps that matter most in practice are pod-to-pod traffic and Ingress TLS. This article covers both Ingress TLS with Let's Encrypt and internal service-to-service encryption using a private CA.

How cert-manager Works

cert-manager is a Kubernetes operator. It extends the Kubernetes API with custom resources that represent certificate requests and their configuration. When you create a Certificate resource, cert-manager's controller picks it up, requests a certificate from the configured issuer, and stores the resulting certificate and private key in a Kubernetes Secret. When the certificate approaches its expiry, cert-manager renews it automatically.

This model means your application doesn't know or care about certificate management. It reads a Secret. cert-manager keeps that Secret fresh.

The Four Core Resources

cert-manager introduces four custom resources that you'll use regularly:

In practice you'll mostly deal with ClusterIssuer and Certificate. The ClusterIssuer defines where certificates come from. The Certificate defines what certificate you want and where to store it.

Issuers and ClusterIssuers

An Issuer can only issue certificates within its own namespace. A ClusterIssuer can issue certificates in any namespace. For shared infrastructure like Let's Encrypt, you almost always want a ClusterIssuer. For application-specific internal CAs, an Issuer scoped to that application's namespace is the safer choice.

cert-manager supports several issuer types. The three you'll encounter most often are:

ACME — for public certificates from Let's Encrypt or any ACME-compatible CA. Ownership of the domain is proven via an HTTP-01 or DNS-01 challenge.

CA — for internal certificates signed by a CA whose private key is stored in a Kubernetes Secret. Used for service-to-service TLS within the cluster.

Self-signed — generates self-signed certificates. Rarely useful on its own, but essential as the bootstrap step when creating an internal CA.

The Certificate Lifecycle

When you create a Certificate resource, cert-manager follows this sequence:

1. Creates a CertificateRequest with a CSR (Certificate Signing Request)

2. Passes the CSR to the configured issuer

3. For ACME issuers: creates a Challenge resource and fulfils it (more on this below)

4. Receives the signed certificate from the issuer

5. Stores the certificate and private key in the Kubernetes Secret named in spec.secretName

6. Monitors the certificate's expiry — by default, renews when 2/3 of the validity period has elapsed

Your application mounts the Secret. cert-manager updates it silently. Most applications that watch for file changes will pick up the new certificate without a restart.

ACME Challenges: HTTP-01 vs DNS-01

Let's Encrypt needs proof that you control the domain before it issues a certificate. ACME defines two challenge types for this.

HTTP-01 works by having cert-manager create a temporary HTTP endpoint at http://<your-domain>/.well-known/acme-challenge/<token>. Let's Encrypt sends a request to that URL. If the response matches the expected token, the challenge passes. This requires your cluster to be reachable from the internet on port 80.

DNS-01 works by having cert-manager create a temporary DNS TXT record at _acme-challenge.<your-domain>. Let's Encrypt checks for that record. This doesn't require inbound HTTP access, which makes it the right choice for private clusters, and it's the only way to get wildcard certificates (*.example.com).

The trade-off: HTTP-01 is simpler to set up but only works for single domains and requires internet-accessible infrastructure. DNS-01 requires API access to your DNS provider but works for internal clusters and wildcards.

Demo 1 — Install cert-manager and Issue a Certificate Using Pebble and Let's Encrypt

Pebble is Let's Encrypt's local ACME test server. It runs inside your cluster, issues certificates using the same ACME protocol as Let's Encrypt, and requires no public domain or internet access. Using Pebble lets you test the full cert-manager flow — challenge, issuance, renewal — on a plain kind cluster.

Once you understand the flow locally, switching to real Let's Encrypt is a one-line change: replace the ClusterIssuer server URL and point a DNS record at a publicly reachable cluster. The rest of the configuration is identical.

You'll install cert-manager, create a ClusterIssuer for Let's Encrypt, deploy a sample application with an Ingress, and watch a real certificate be issued and stored automatically.

Step 1: Install cert-manager

cert-manager is now distributed via OCI Helm charts from quay.io/jetstack. The --set crds.enabled=true flag installs the Custom Resource Definitions as part of the chart:

helm upgrade cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --install \
  --create-namespace \
  --namespace cert-manager \
  --set crds.enabled=true \
  --version v1.17.0 \
  --wait

You also need the nginx Ingress controller — cert-manager routes HTTP-01 challenges through it. The controller.service.type=ClusterIP override is for kind specifically: the default LoadBalancer Service never gets an EXTERNAL-IP on kind (there's no cloud LB), which makes --wait hang forever. On a real cluster, drop the override and keep LoadBalancer.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=ClusterIP \
  --wait

Confirm all four components are running:

kubectl get pods -n cert-manager
kubectl get pods -n ingress-nginx

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-76f84784c8-r4fx4              1/1     Running   0          6m45s
cert-manager-cainjector-66fbf49587-gv25n   1/1     Running   0          6m45s
cert-manager-webhook-577fddf86-l5wj4       1/1     Running   0          6m45s

NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6c7cd85885-h7zgx   1/1     Running   0          3m34s

kind-specific gotcha — remove the nginx admission webhook now.** On kind, the nginx admission webhook serves with a self-signed certificate that the Kubernetes API server cannot verify. The first time you try to create any Ingress resource you'll see failed calling webhook "validate.nginx.ingress.kubernetes.io": ... x509: certificate signed by unknown authority. Delete the webhook up front so the rest of the demo doesn't trip over it:

kubectl delete validatingwebhookconfiguration ingress-nginx-admission

Step 2: Install Pebble

Pebble is the local ACME test server, distributed by the JupyterHub project. It ships with a companion CoreDNS deployment (pebble-coredns) that Pebble uses to resolve names during ACME validation.

helm install pebble pebble \
  --repo https://jupyterhub.github.io/helm-chart/ \
  --namespace pebble \
  --create-namespace \
  --wait

Confirm both pods are running:

kubectl get pods -n pebble

NAME                              READY   STATUS    RESTARTS   AGE
pebble-8d8d49d64-lz8ck            1/1     Running   0          36s
pebble-coredns-7fb5c7cbf4-4jw9h   1/1     Running   0          36s

Step 3: Wire up DNS for the fake hostname

We're going to issue a cert for echo.pebble.local. That hostname is fake — it doesn't exist in any real DNS — so we have to teach two independent resolvers about it before issuance will work:

If you skip either layer, the Order will go to invalid state with a DNS lookup failure.

First grab the two IPs you'll need:

NGINX_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.spec.clusterIP}')
PEBBLE_DNS_IP=$(kubectl get svc pebble-coredns -n pebble \
  -o jsonpath='{.spec.clusterIP}')
echo "NGINX_IP=\(NGINX_IP  PEBBLE_DNS_IP=\)PEBBLE_DNS_IP"

Patch pebble-coredns to answer for *.pebble.local with the ingress controller's IP. The CoreDNS template plugin parses unreliably when the whole block is collapsed onto one line, so apply a real multi-line ConfigMap:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: pebble-coredns
  namespace: pebble
data:
  Corefile: |
    .:8053 {
      errors
      health
      ready
      template ANY ANY pebble.local {
        answer "{{ .Name }} 60 IN A ${NGINX_IP}"
      }
      forward . /etc/resolv.conf
      cache 2
      reload
    }
EOF

kubectl rollout restart deploy/pebble-coredns -n pebble
kubectl rollout status deploy/pebble-coredns -n pebble

Verify it answers correctly:

kubectl run dnstest --rm -it --restart=Never --image=busybox -- \
  nslookup echo.pebble.local ${PEBBLE_DNS_IP}

You should see Address: <NGINX_IP> in the response. If you get SERVFAIL, check kubectl logs -n pebble deploy/pebble-coredns — a parser error like not a TTL: "}" means the template block collapsed onto one line again.

Patch the cluster CoreDNS so cert-manager's self-check can resolve the same name. Add a stub zone that forwards pebble.local to pebble-coredns:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
    pebble.local:53 {
        forward . ${PEBBLE_DNS_IP}
    }
EOF

kubectl rollout restart deploy/coredns -n kube-system
kubectl rollout status deploy/coredns -n kube-system

Verify the cluster resolver now answers for echo.pebble.local (without specifying a server — it'll use the default kube-dns):

kubectl run dnstest --rm -it --restart=Never --image=busybox -- \
  nslookup echo.pebble.local

Both Server: 10.96.0.10 and Address: <NGINX_IP> should appear.

Step 4: Fetch the Pebble CA and create the ClusterIssuer

Pebble signs its certificates with a self-signed root that lives in the pebble ConfigMap under root-cert.pem. cert-manager needs to trust this CA to talk to Pebble's ACME directory, so we pass it as a base64-encoded caBundle in the ClusterIssuer:

kubectl get configmap pebble -n pebble \
  -o jsonpath='{.data.root-cert\.pem}' > pebble-ca.crt

head -1 pebble-ca.crt   # should print -----BEGIN CERTIFICATE-----

CA_BUNDLE=$(base64 -i pebble-ca.crt | tr -d '\n')
echo "CA_BUNDLE length: ${#CA_BUNDLE}"   # ~1600 chars, one continuous line

Create the ClusterIssuer using the heredoc — the ${CA_BUNDLE} shell variable gets substituted into the YAML before kubectl reads it:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: pebble
spec:
  acme:
    server: https://pebble.pebble.svc.cluster.local/dir
    email: test@example.com
    privateKeySecretRef:
      name: pebble-account-key
    caBundle: ${CA_BUNDLE}
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
EOF

Check the issuer is ready:

kubectl get clusterissuer pebble

NAME     READY   AGE
pebble   True    5s

If READY stays False, the two most common causes are a malformed caBundle (verify it's a single unbroken base64 line with no newlines) or Pebble being unreachable from the cert-manager namespace. To check reachability:

kubectl run test-curl --rm -it --restart=Never \
  --image=curlimages/curl:latest \
  --namespace cert-manager -- \
  curl -k https://pebble.pebble.svc.cluster.local/dir

If that returns JSON, Pebble is reachable.

Step 5: Deploy a sample application

# echo-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: ealen/echo-server:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: default
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80

kubectl apply -f echo-app.yaml

Verify the resources came up:

kubectl get deploy,pod,svc -n default

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/echo   1/1     1            1           32s

NAME                        READY   STATUS    RESTARTS   AGE
pod/echo-5665fbcfdd-mbgxj   1/1     Running   0          36s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/echo         ClusterIP   10.96.103.114   <none>        80/TCP    40s
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   32m

Step 6: Create an Ingress with TLS

The cert-manager.io/cluster-issuer: pebble annotation tells cert-manager to automatically create a Certificate resource for this Ingress, using the issuer we just created. The hostname echo.pebble.local doesn't need to resolve externally — we taught both DNS resolvers about it in Step 3.

# echo-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: pebble
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - echo.pebble.local
      secretName: echo-tls     # cert-manager will create this Secret
  rules:
    - host: echo.pebble.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: echo
                port:
                  number: 80

kubectl apply -f echo-ingress.yaml

Step 7: Watch the certificate being issued

# Watch the Certificate resource (Ctrl-C once Ready=True)
kubectl get certificate echo-tls -n default -w

NAME       READY   SECRET     AGE
echo-tls   False   echo-tls   5s
echo-tls   True    echo-tls   28s

When READY becomes True, the certificate has been issued and stored in the echo-tls Secret. The full chain — CertificateRequest → Order → Challenge → solver pod → Secret — happens in well under a minute on a healthy cluster:

kubectl get certificate,certificaterequest,order,challenge -n default

NAME                                   READY   SECRET     AGE
certificate.cert-manager.io/echo-tls   True    echo-tls   81s

NAME                                            APPROVED   DENIED   READY   ISSUER   AGE
certificaterequest.cert-manager.io/echo-tls-1   True                True    pebble   81s

NAME                                               STATE   AGE
order.acme.cert-manager.io/echo-tls-1-1824732543   valid   81s

(Challenges are deleted automatically once an Order completes, so kubectl get challenge -n default typically shows nothing at this point — that's success, not failure.)

If READY stays False for more than a minute, see the troubleshooting tips at the end of this section.

Inspect the issued certificate to confirm Pebble signed it:

kubectl get secret echo-tls -n default -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -issuer -subject -dates

issuer=CN=Pebble Intermediate CA 05478c
subject=
notBefore=May 17 19:09:22 2026 GMT
notAfter=Aug 15 19:09:21 2026 GMT

Issuer is Pebble's intermediate CA — proof the full ACME flow worked end-to-end. The cert is valid for 90 days, and cert-manager will renew it automatically at day 60.

Hit the ingress over HTTPS from inside the cluster to confirm everything is wired together:

kubectl run curltest --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sk https://echo.pebble.local/

The echo server should return a JSON blob — note the "x-forwarded-proto":"https" field, which proves the request came through nginx over TLS.

Troubleshooting if the cert never goes Ready:

• kubectl describe order -n default — look for "DNS problem" or "Connection refused" in the events.

• kubectl logs -n pebble deploy/pebble --tail=50 — Pebble logs the exact URL it tried to fetch during validation and any errors.

• If the Order is stuck pending with no events: cert-manager hasn't reconciled yet. Wait 30s.

• If the Order is invalid: one of the two DNS layers (Step 3) is misconfigured. Re-run both nslookup checks.

• If the Ingress apply itself failed with an x509 webhook error: you skipped the kubectl delete validatingwebhookconfiguration ingress-nginx-admission step in Step 1.

Step 8: Switch to Let's Encrypt staging (real public domain)

Pebble proved the flow works locally. Now move to a publicly-reachable domain pointed at a publicly-reachable cluster. The DNS gymnastics from Step 3 go away — the domain is real, so both resolvers find it without intervention.

Use Let's Encrypt staging first. It speaks the same ACME protocol as production but with generous rate limits, so failed attempts during testing won't lock you out:

# clusterissuer-staging.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

kubectl apply -f clusterissuer-staging.yaml

# Point the Ingress at staging and the real hostname, then force re-issuance
kubectl annotate ingress echo \
  cert-manager.io/cluster-issuer=letsencrypt-staging --overwrite -n default
kubectl delete secret echo-tls -n default

The new cert's issuer will look something like (STAGING) Let's Encrypt.

Step 9: Switch to Let's Encrypt production

Once staging works, repeat with the production ClusterIssuer. The only difference is the server URL:

# clusterissuer-prod.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

kubectl apply -f clusterissuer-prod.yaml
kubectl annotate ingress echo \
  cert-manager.io/cluster-issuer=letsencrypt-prod --overwrite -n default
kubectl delete secret echo-tls -n default

cert-manager detects the missing Secret and immediately requests a browser-trusted certificate from production Let's Encrypt.

cert-manager detects the missing Secret and immediately triggers a new certificate request using the production issuer.

How to Get a Wildcard Certificate with DNS-01

HTTP-01 challenges work well for single domains with public ingress. But there are two situations where you need DNS-01 instead: when your cluster is not publicly accessible (internal clusters, air-gapped environments, staging namespaces behind a VPN), and when you want a wildcard certificate that covers all subdomains of your domain.

DNS-01 requires cert-manager to be able to create and delete TXT records in your DNS provider. cert-manager has built-in support for Route53, Cloud DNS, Cloudflare, Azure DNS, and many others.

Here is a ClusterIssuer for DNS-01 using AWS Route53:

# clusterissuer-dns01.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-dns01-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            # Use IRSA (IAM Roles for Service Accounts) in production
            # rather than static credentials
            hostedZoneID: YOUR_HOSTED_ZONE_ID

A wildcard Certificate using that issuer:

# wildcard-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: default
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer
  commonName: "*.example.com"
  dnsNames:
    - "*.example.com"
    - "example.com"        # Also cover the apex domain
  duration: 2160h           # 90 days
  renewBefore: 720h         # Renew 30 days before expiry

The resulting Secret wildcard-example-com-tls can be referenced by any Ingress in the default namespace. All subdomains — api.example.com, dashboard.example.com, staging.example.com — are covered by a single certificate that rotates automatically.

For Cloudflare instead of Route53, the solver section looks like this:

    solvers:
      - dns01:
          cloudflare:
            email: your-email@example.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Demo 2 — Set Up an Internal CA for Service-to-Service TLS

Let's Encrypt certificates are great for public-facing services. But for internal services — a gRPC microservice calling another, a web application talking to its database — you don't need public trust. You need a CA that the cluster trusts, and you need it to issue certificates for service names that don't exist as public DNS records.

cert-manager's CA issuer handles this. You create a root CA, tell cert-manager about it, and then issue certificates for internal services using that CA. Every service that trusts the root CA trusts every certificate it issues.

Step 1: Create a self-signed ClusterIssuer

A self-signed issuer generates certificates that are signed by the certificate itself — it is its own CA. You use this as a bootstrap step to create the root CA certificate:

# selfsigned-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

kubectl apply -f selfsigned-issuer.yaml

Step 2: Create the root CA certificate

Use the self-signed issuer to create a CA certificate. The isCA: true field tells cert-manager this certificate can sign other certificates:

# internal-ca.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager    # Store in cert-manager namespace
spec:
  isCA: true
  commonName: internal-ca
  secretName: internal-ca-secret
  duration: 87600h           # 10 years — this is a root CA
  renewBefore: 720h
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

kubectl apply -f internal-ca.yaml
kubectl get certificate internal-ca -n cert-manager

NAME          READY   SECRET               AGE
internal-ca   True    internal-ca-secret   8s

Step 3: Create a CA ClusterIssuer backed by the root CA

Now create a ClusterIssuer that uses the root CA Secret you just created. This is the issuer that will sign certificates for your internal services:

# internal-ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-secret   # References the Secret in cert-manager namespace

kubectl apply -f internal-ca-issuer.yaml
kubectl get clusterissuer internal-ca

NAME          READY   AGE
internal-ca   True    5s

Step 4: Issue a certificate for an internal service

Now issue a certificate for an internal gRPC service. The dnsNames use Kubernetes internal DNS names — <service>.<namespace>.svc.cluster.local:

# payments-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: payments-tls
  namespace: production
spec:
  secretName: payments-tls-secret
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  commonName: payments.production.svc.cluster.local
  dnsNames:
    - payments.production.svc.cluster.local
    - payments.production.svc
    - payments
  duration: 2160h     # 90 days
  renewBefore: 360h   # Renew 15 days before expiry

kubectl create namespace production
kubectl apply -f payments-cert.yaml
kubectl get certificate payments-tls -n production

NAME           READY   SECRET                AGE
payments-tls   True    payments-tls-secret   6s

The Secret payments-tls-secret now contains tls.crt, tls.key, and ca.crt. Mount this into your application pod:

# In your Deployment spec
volumes:
  - name: tls
    secret:
      secretName: payments-tls-secret
containers:
  - name: payments
    volumeMounts:
      - name: tls
        mountPath: /etc/tls
        readOnly: true

Your application reads /etc/tls/tls.crt and /etc/tls/tls.key to configure TLS. Other services that need to trust it read /etc/tls/ca.crt.

Step 5: Distribute the CA bundle with trust-manager

The problem with a custom CA is that every service needs to know about it. cert-manager's companion tool, trust-manager, handles this by distributing the CA bundle as a ConfigMap to every namespace:

helm upgrade trust-manager oci://quay.io/jetstack/charts/trust-manager \
  --install \
  --namespace cert-manager \
  --wait

Create a Bundle resource that takes the CA certificate from the internal-ca-secret and distributes it cluster-wide:

# ca-bundle.yaml
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: internal-ca-bundle
spec:
  sources:
    - secret:
        name: internal-ca-secret
        key: ca.crt
  target:
    configMap:
      key: ca-bundle.crt
    namespaceSelector:
      matchLabels:
        # Distribute to all namespaces with this label
        kubernetes.io/metadata.name: production

kubectl apply -f ca-bundle.yaml

After a few seconds, every matching namespace has a ConfigMap named internal-ca-bundle containing the CA certificate. Applications mount this ConfigMap to trust internally-issued certificates without any per-service configuration.

Step 6: Verify the certificate chain

# Extract the CA cert and service cert
kubectl get secret payments-tls-secret -n production \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

kubectl get secret payments-tls-secret -n production \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > payments.crt

# Verify the cert was signed by the CA
openssl verify -CAfile ca.crt payments.crt

payments.crt: OK

How Certificate Rotation Works

Certificate rotation is the part of certificate management that breaks production clusters most often. cert-manager handles it automatically, but understanding the mechanism helps you tune it and debug it when things go wrong.

cert-manager watches every Certificate resource it manages and checks the expiry of the underlying certificate in the Secret. When the remaining validity drops below the renewBefore threshold, cert-manager triggers a renewal. The default renewBefore is 1/3 of the certificate's total validity period — so a 90-day certificate starts renewing at day 60.

The renewal creates a new CertificateRequest, goes through the full issuance flow, and updates the Secret in place. The new certificate replaces the old one atomically. Applications that use file mounts and watch for changes (most modern web servers and gRPC frameworks do) will pick up the new certificate without restarting.

# See the current rotation status
kubectl describe certificate echo-tls -n default

Look for these fields in the output:

Status:
  Not After:   2024-06-18T10:00:00Z
  Not Before:  2024-03-20T10:00:00Z
  Renewal Time: 2024-05-18T10:00:00Z   # When cert-manager will start renewing
  Conditions:
    Type:    Ready
    Status:  True
    Message: Certificate is up to date and has not expired

If a renewal fails — for example, because the HTTP-01 challenge can't be completed — cert-manager retries with exponential backoff. The existing certificate continues to serve until it actually expires, giving you a window to debug the issue.

To see renewal events in real time:

kubectl get events -n default --field-selector reason=Issued
kubectl get events -n default --field-selector reason=Failed

Setting renewBefore correctly: For public-facing services, 30 days before a 90-day certificate is a sensible buffer. For internal short-lived certificates (24-hour validity), set renewBefore to 8 hours so rotation happens well before expiry even if the first attempt fails. Never set renewBefore to more than half the certificate's validity — cert-manager will immediately try to renew a certificate it just issued.

Cleanup

# Remove demo resources
kubectl delete ingress echo -n default
kubectl delete service echo -n default
kubectl delete deployment echo -n default
kubectl delete secret echo-tls -n default
kubectl delete certificate payments-tls -n production
kubectl delete namespace production

# Uninstall cert-manager and trust-manager
helm uninstall trust-manager -n cert-manager
helm uninstall cert-manager -n cert-manager
kubectl delete namespace cert-manager

# Remove ClusterIssuers
kubectl delete clusterissuer letsencrypt-staging letsencrypt-prod \
  internal-ca selfsigned 2>/dev/null

Conclusion

Kubernetes leaves TLS configuration entirely to you. In this article you worked through both the public and internal sides of that responsibility.

On the public side, you installed cert-manager using the current OCI Helm chart, created a ClusterIssuer backed by Let's Encrypt, and watched cert-manager go through the full ACME HTTP-01 challenge flow — from creating a temporary solver pod to storing a valid certificate in a Kubernetes Secret. You saw how switching from staging to production is a one-line annotation change, and how cert-manager renews certificates automatically before they expire.

On the internal side, you bootstrapped a private CA using cert-manager's self-signed issuer, created a ClusterIssuer backed by that CA, and issued certificates for internal service names that only exist inside the cluster. You used trust-manager to distribute the CA bundle cluster-wide so services can trust each other's certificates without per-service configuration. And you saw how to verify the certificate chain with openssl so you can confirm it's working before deploying to production.

Understanding certificate rotation is what separates teams that manage TLS confidently from teams that get woken up at 3am by an expired certificate. cert-manager automates the renewal, but the renewBefore field is your safety margin — set it correctly and know how to read the renewal status.

All YAML manifests and Helm values from this article are available in the DevOps-Cloud-Projects GitHub repository.

An attacker had found it, deployed pods inside Tesla's cluster, and was using them to mine cryptocurrency – all on Tesla's AWS bill. The cluster had no authentication on the dashboard, no network restrictions on egress, and nothing monitoring for intrusion. Any one of those controls would have stopped the attack. None of them were in place.

This wasn't a sophisticated zero-day exploit. It was a misconfigured default.

Kubernetes ships with powerful security primitives. The problem is that almost none of them are enabled by default. A fresh cluster is deliberately permissive so it's easy to get started. That permissiveness is a feature in development. In production, it's a liability.

In this handbook, we'll work through the three most impactful security layers in Kubernetes. We'll start with Role-Based Access Control, which governs who can do what to which resources in the API. From there we'll move to pod runtime security, which locks down what containers can actually do once they're running on a node. Finally we'll deploy Falco, a syscall-level detection engine that watches for attacks in progress and alerts in real time.

By the end, you'll have a hardened cluster with working RBAC policies, enforced pod security standards, and live detection rules that fire when something suspicious happens.

Prerequisites

• kubectl installed and configured

• Docker Desktop or a Linux machine (to run kind)

• Basic Kubernetes familiarity – you know what a Pod, Deployment, and Namespace are

• No prior security experience needed

All demos run on a local kind cluster. Full YAML and setup scripts are in the companion GitHub repository.

Table of Contents

• The Kubernetes Threat Landscape

• What You'll Build

• Demo 1 — Run a Cluster Security Baseline with kube-bench

• How to Configure RBAC

  • The Four RBAC Objects

  • How to Discover Resources, Verbs, and API Groups

  • Roles and ClusterRoles

  • RoleBindings and ClusterRoleBindings

  • How to Use Service Accounts Safely

  • How to Audit Your RBAC Configuration

• Demo 2 — Build a Least-Privilege RBAC Policy for a CI Pipeline

• Demo 3 — Audit RBAC with rakkess and rbac-lookup

• How to Harden Pod Runtime Security

  • Pod Security Admission

  • How to Configure securityContext

  • OPA/Gatekeeper vs Kyverno

  • How to Detect Runtime Threats with Falco

• Demo 4 — Harden a Pod with securityContext

• Demo 5 — Deploy Falco and Write a Custom Detection Rule

• Cleanup

• Conclusion

The Kubernetes Threat Landscape

To understand what you're defending against, you need to understand where Kubernetes exposes attack surface. There are six main areas, and most production incidents trace back to at least one of them.

The API server is the front door to your cluster. Every kubectl command, every CI deploy, and every controller reconciliation loop sends requests here. Unauthenticated or over-privileged access to the API server is effectively game over: an attacker who can talk to it can create pods, read secrets, and modify workloads freely.

etcd is the key-value store where all cluster state lives, including your Secrets. Kubernetes Secrets are base64-encoded by default, not encrypted. Anyone with direct access to etcd can read every password, token, and certificate in the cluster without going through the API server at all.

The kubelet runs on each node and manages the pods assigned to it. If its API is reachable without authentication – which is the default on older clusters – an attacker can exec into any pod on that node and read its memory without ever touching the API server.

The container runtime is the layer that actually runs your containers. A container that escapes its isolation boundary lands directly in the host OS. A privileged container with hostPID: true can read the memory of every other process on the node, including other containers.

Your supply chain (base images, third-party dependencies, Helm charts, operators) is a potential entry point at every step. The XZ Utils backdoor discovered in 2024 showed how close a well-positioned supply chain attack can come to widespread infrastructure compromise.

Finally, the network: by default, every pod in a Kubernetes cluster can reach every other pod on any port. There are no internal firewalls between workloads unless you explicitly create them with NetworkPolicy.

Real-World Breaches

These three incidents are worth understanding before you write a single line of YAML. They're not theoretical – they're documented post-mortems from real production clusters.

The pattern across all three: not zero-day exploits, but misconfigured defaults and missing controls that should have been standard practice.

This article addresses the RBAC and pod security gaps directly.

What You'll Build

Before the first command, here is the security posture you'll have by the end of this article:

You'll start by running kube-bench to get a CIS Benchmark baseline – a concrete score showing where a default cluster stands before any hardening. From there you'll build a least-privilege RBAC policy for a CI pipeline service account and verify its permission boundaries, then audit the full cluster to confirm no over-privileged accounts exist.

On the pod security side, you'll enforce the restricted Pod Security Admission profile on your workload namespace and apply a hardened securityContext to a deployment: non-root user, read-only root filesystem, dropped capabilities, and seccomp profile. To close out, you'll deploy Falco in eBPF mode with a custom detection rule that fires when suspicious tools are run inside a container.

Start to finish, with a kind cluster already running, the demos take about 45–60 minutes.

Demo 1: Run a Cluster Security Baseline with kube-bench

Before hardening anything, it's a good idea to measure where you are. kube-bench runs the CIS Kubernetes Benchmark against your cluster and reports which checks pass and which fail. A baseline run gives you a concrete picture of your cluster's default security posture – and a reference point you can re-run after applying any hardening changes.

Step 1: Create a kind cluster

Save the following as kind-config.yaml:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --name k8s-security --config kind-config.yaml

Expected output:

Creating cluster "k8s-security" ...
 ✓ Ensuring node image (kindest/node:v1.29.0) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-k8s-security"

Step 2: Run kube-bench

kube-bench runs as a Job inside the cluster, mounting the host filesystem to inspect Kubernetes configuration files and processes:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl wait --for=condition=complete job/kube-bench --timeout=120s
kubectl logs job/kube-bench

The output is long. Scroll to the summary at the bottom:

== Summary master ==
0 checks PASS
11 checks FAIL
 9 checks WARN
 0 checks INFO

== Summary node ==
17 checks PASS
 2 checks FAIL
40 checks WARN
 0 checks INFO

A fresh kind cluster typically fails around 14 checks. Three of the most important failures explain why defaults are a problem:

Note: Some kube-bench findings are expected on kind because kind is a development tool, not a production-hardened environment. The important thing is to understand what each finding means and whether it applies to your target production setup.

Delete the Job when you're done:

kubectl delete job kube-bench

Now that you have a baseline, you know what you're starting from. The next step is to work through the most impactful control on that list: access control. RBAC governs every interaction with the Kubernetes API, and getting it right is the foundation everything else builds on.

How to Configure RBAC

Role-Based Access Control is the authorisation layer in Kubernetes. Every request that reaches the API server – from kubectl, from a pod, from a controller – is checked against RBAC rules after authentication succeeds. If there is no rule that explicitly allows the action, Kubernetes denies it.

The key word is "explicitly". RBAC in Kubernetes is additive only. There is no deny rule. You grant access by creating rules, and you remove access by deleting them. This makes the mental model clean: if a subject can do something, you gave it permission to do that thing.

A Brief Case Study: The Shopify Kubernetes Misconfiguration

In 2021, security researcher Silas Cutler discovered that a Shopify internal service exposed Kubernetes metadata through an SSRF vulnerability. The metadata included pod environment variables that contained secrets. The root cause was partly RBAC: the service's service account had broader cluster access than it needed, and there was no least-privilege review process.

Shopify paid a $25,000 bug bounty and fixed the issue. The lesson is straightforward: a service account should only have the permissions it needs to do its specific job. Nothing more.

This is the principle you'll apply in Demo 2.

The Four RBAC Objects

RBAC in Kubernetes is built from four API objects. Two define permissions, two bind those permissions to subjects:

A subject is a user, a group, or a service account. Users and groups come from your authentication layer – client certificates, OIDC tokens, or cloud provider identity. Service accounts are Kubernetes-native identities created for pods.

How to Discover Resources, Verbs, and API Groups

Before you can write a Role, you need to know three things: the resource name, the API group it belongs to, and the verbs it supports. You shouldn't have to guess any of them – kubectl can tell you everything.

List all available resources and their API groups

kubectl api-resources

Partial output:

NAME                    SHORTNAMES  APIVERSION                     NAMESPACED  KIND
bindings                            v1                             true        Binding
configmaps              cm          v1                             true        ConfigMap
endpoints               ep          v1                             true        Endpoints
events                  ev          v1                             true        Event
namespaces              ns          v1                             false       Namespace
nodes                   no          v1                             false       Node
pods                    po          v1                             true        Pod
secrets                             v1                             true        Secret
serviceaccounts         sa          v1                             true        ServiceAccount
services                svc         v1                             true        Service
deployments             deploy      apps/v1                        true        Deployment
replicasets             rs          apps/v1                        true        ReplicaSet
statefulsets            sts         apps/v1                        true        StatefulSet
cronjobs                cj          batch/v1                       true        CronJob
jobs                                batch/v1                       true        Job
ingresses               ing         networking.k8s.io/v1           true        Ingress
networkpolicies         netpol      networking.k8s.io/v1           true        NetworkPolicy
clusterroles                        rbac.authorization.k8s.io/v1   false       ClusterRole
roles                               rbac.authorization.k8s.io/v1   true        Role

The APIVERSION column is what you put in apiGroups. Strip the version suffix and use only the group part:

The NAMESPACED column tells you whether to use a Role (namespaced resources) or a ClusterRole (non-namespaced resources like nodes).

Filter by API group

If you want to see only resources in a specific group, for example, everything in apps:

kubectl api-resources --api-group=apps

NAME                  SHORTNAMES  APIVERSION  NAMESPACED  KIND
controllerrevisions               apps/v1     true        ControllerRevision
daemonsets            ds          apps/v1     true        DaemonSet
deployments           deploy      apps/v1     true        Deployment
replicasets           rs          apps/v1     true        ReplicaSet
statefulsets          sts         apps/v1     true        StatefulSet

List all verbs for a specific resource

Each resource supports a different set of verbs. To see exactly which verbs a resource supports, use kubectl api-resources with -o wide and look at the VERBS column:

kubectl api-resources -o wide | grep -E "^NAME|^pods "

NAME  SHORTNAMES  APIVERSION  NAMESPACED  KIND  VERBS
pods  po          v1          true        Pod   create,delete,deletecollection,get,list,patch,update,watch

Or explain the resource directly:

kubectl explain pod --api-version=v1 | head -10

The full set of verbs Kubernetes supports in RBAC rules is:

Important: get and list are separate verbs. Granting list on secrets lets a subject enumerate every secret name and value in a namespace, even if you didn't also grant get. Always think about both when working with sensitive resources like secrets, serviceaccounts, and configmaps.

Look up a resource's group with kubectl explain

If you already know the resource name but aren't sure of its group, kubectl explain tells you:

kubectl explain deployment

GROUP:      apps
KIND:       Deployment
VERSION:    v1
...

kubectl explain ingress

GROUP:      networking.k8s.io
KIND:       Ingress
VERSION:    v1
...

This is the fastest way to look up the apiGroups value for any resource when writing a Role.

A complete lookup workflow

Here is the practical workflow when writing a new Role from scratch:

# 1. Find the resource name and API group
kubectl api-resources | grep deployment

# Output:
# deployments   deploy   apps/v1   true   Deployment

# 2. Find the verbs it supports
kubectl api-resources -o wide | grep deployment

# Output:
# deployments   deploy   apps/v1   true   Deployment   create,delete,...,get,list,patch,update,watch

# 3. Write the Role using the group (strip the version) and the verbs you need

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployment-reader
  namespace: staging
rules:
  - apiGroups: ["apps"]       # from: apps/v1 → strip /v1
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

With this workflow, you never have to guess an API group or verb. You look it up, then write the minimal rule you need.

Roles and ClusterRoles

A Role defines which verbs are allowed on which resources. Here is a Role that grants read-only access to Pods and ConfigMaps inside the staging namespace:

# role-ci-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-reader
  namespace: staging
rules:
  - apiGroups: [""]          # "" = the core API group (Pods, Services, Secrets, ConfigMaps)
    resources: ["pods", "configmaps"]
    verbs: ["get", "list", "watch"]

The apiGroups field tells Kubernetes which API group owns the resource. The core group uses an empty string "". Apps-level resources like Deployments use "apps". Custom resources use their own group, such as "networking.k8s.io".

A ClusterRole is structurally identical but omits the namespace and can reference cluster-scoped resources like Nodes and PersistentVolumes:

# clusterrole-node-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader    # no namespace field
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]

When to use which:

Use a Role when the permission is specific to one namespace. A compromised service account can only affect that namespace: the blast radius is contained. Use a ClusterRole when you need access to cluster-scoped resources, or when you want a reusable permission template that multiple namespaces can share.

A common mistake is reaching for a ClusterRole "just to be safe" because it's easier to configure. Namespace-scoped Roles are almost always the right default.

RoleBindings and ClusterRoleBindings

A Role by itself does nothing. You need a binding to attach it to a subject. Here is a RoleBinding that grants the ci-reader Role to the ci-pipeline service account:

# rolebinding-ci.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-reader-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline       # the service account name
    namespace: staging      # the namespace the SA lives in
roleRef:
  kind: Role
  name: ci-reader           # must match the Role name exactly
  apiGroup: rbac.authorization.k8s.io

There is a useful pattern worth knowing: you can bind a ClusterRole using a RoleBinding. This creates namespace-scoped access using a reusable permission template. The ClusterRole defines the rules, while the RoleBinding constrains those rules to a single namespace.

# RoleBinding referencing a ClusterRole — scoped to one namespace only
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: view-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline
    namespace: staging
roleRef:
  kind: ClusterRole          # ClusterRole, but bound to one namespace via RoleBinding
  name: view                 # Kubernetes built-in ClusterRole: read-only access to most resources
  apiGroup: rbac.authorization.k8s.io

Kubernetes ships with several useful built-in ClusterRoles: view (read-only access to most resources), edit (read/write to most resources), admin (full namespace admin), and cluster-admin (full cluster admin). Use them rather than reinventing them.

How to Use Service Accounts Safely

Every pod in Kubernetes runs as a service account. If you don't specify one, Kubernetes uses the default service account in that namespace.

The default service account starts with no permissions – but it still has a token automatically mounted into every pod at /var/run/secrets/kubernetes.io/serviceaccount/token. This means every container in your cluster can authenticate to the API server by default, even if it has nothing useful to do there.

The single most impactful change you can make is to disable this automatic token mounting on service accounts that don't need API access:

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
automountServiceAccountToken: false   # no token mounted into pods by default

You can also control it at the pod level:

spec:
  automountServiceAccountToken: false   # override at pod level
  serviceAccountName: my-app
  containers:
    - name: app
      image: my-app:1.0

The cluster-admin anti-pattern:

Never bind cluster-admin to a service account that runs in a pod. cluster-admin grants full read/write access to every resource in the cluster. An attacker who compromises a pod running as cluster-admin owns your cluster completely.

You will see this in Helm charts and tutorials because it "makes things work". It works because it disables the entire authorisation layer. That is not a solution – it's a ticking clock.

The Capital One breach is a direct example of this pattern at the cloud layer: an EC2 instance role had permissions far beyond what the application needed. The SSRF vulnerability was the initial foothold. The over-privileged role was what turned a minor bug into a $80 million fine.

How to Audit Your RBAC Configuration

The kubectl auth can-i command lets you check permissions for any subject. Use --as to impersonate a service account:

SA="system:serviceaccount:staging:ci-pipeline"

# These should return 'yes'
kubectl auth can-i list pods        --namespace staging --as $SA
kubectl auth can-i get  configmaps  --namespace staging --as $SA

# These should return 'no'
kubectl auth can-i delete pods      --namespace staging --as $SA
kubectl auth can-i get  secrets     --namespace staging --as $SA
kubectl auth can-i list pods        --namespace production --as $SA

To list every permission a subject has in a namespace:

kubectl auth can-i --list \
  --namespace staging \
  --as system:serviceaccount:staging:ci-pipeline

For a visual matrix across the whole cluster, install rakkess (part of krew):

kubectl krew install access-matrix

# Permission matrix for all service accounts in staging
kubectl access-matrix --namespace staging

Example output:

NAME          GET  LIST  WATCH  CREATE  UPDATE  PATCH  DELETE
ci-pipeline    ✓    ✓     ✓      ✗       ✗       ✗      ✗
default        ✗    ✗     ✗      ✗       ✗       ✗      ✗
monitoring     ✓    ✓     ✓      ✗       ✗       ✗      ✗

If you see ✓ in the CREATE, UPDATE, PATCH, or DELETE columns for a service account that should only read, that's a finding that needs remediation.

⚠️ The wildcard danger: The most dangerous RBAC configuration is a wildcard on all three dimensions:

apiGroups: [""] 
resources: [""] 
verbs: ["*"]

This is functionally identical to cluster-admin. You will find it in Helm charts for controllers installed with "convenience" permissions. Always audit third-party RBAC before installing operators into a production cluster.

Demo 2 – Build a Least-Privilege RBAC Policy for a CI Pipeline

In this demo, you'll create a service account for a CI pipeline that can list pods and read configmaps in the staging namespace – and nothing else.

Step 1: Create the namespace and service account

kubectl create namespace staging

# ci-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-pipeline
  namespace: staging
automountServiceAccountToken: false

kubectl apply -f ci-serviceaccount.yaml

Step 2: Create the Role

# ci-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-reader
  namespace: staging
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]

kubectl apply -f ci-role.yaml

Step 3: Bind the Role to the service account

# ci-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-reader-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline
    namespace: staging
roleRef:
  kind: Role
  name: ci-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f ci-rolebinding.yaml

Step 4: Test allowed operations

SA="system:serviceaccount:staging:ci-pipeline"

kubectl auth can-i list pods       --namespace staging     --as $SA   # yes
kubectl auth can-i get  pods       --namespace staging     --as $SA   # yes
kubectl auth can-i list configmaps --namespace staging     --as $SA   # yes

Step 5: Test denied operations

kubectl auth can-i delete pods       --namespace staging     --as $SA   # no
kubectl auth can-i get  secrets      --namespace staging     --as $SA   # no
kubectl auth can-i list pods         --namespace production  --as $SA   # no
kubectl auth can-i create deployments --namespace staging    --as $SA   # no

All four should return no. Notice the third test: even if there were a matching Role in the staging namespace, the service account cannot access production. A RoleBinding cannot cross namespace boundaries, this is by design.

Writing a least-privilege policy for a service account you control is the easy part. The harder part is auditing what already exists in a cluster. That's what Demo 3 covers.

Demo 3 – Audit RBAC with rakkess and rbac-lookup

Now you'll scan the full cluster to surface any accounts with more permissions than they need.

Step 1: Install the tools

kubectl krew install access-matrix
kubectl krew install rbac-lookup

Step 2: Run rakkess across the cluster

# All service accounts in kube-system
kubectl access-matrix --namespace kube-system

# All ServiceAccounts cluster-wide
kubectl access-matrix

Step 3: Find all cluster-admin bindings

There are two ways subjects get cluster-admin access: via a ClusterRoleBinding (cluster-wide), or via a RoleBinding that references the cluster-admin ClusterRole (namespace-scoped, still dangerous). Check both:

# Find ClusterRoleBindings that grant cluster-admin
kubectl rbac-lookup cluster-admin --kind ClusterRole --output wide

On a fresh kind cluster this returns:

No RBAC Bindings found

That is the correct and expected result. A default kind cluster doesn't create any ClusterRoleBindings to cluster-admin. The role exists, but nothing is bound to it at the cluster level by default. If you see entries here in your production cluster, each one is a finding worth investigating.

To find who has cluster-level admin access through other means, query the bindings directly:

# Find all ClusterRoleBindings and the subjects they grant
kubectl get clusterrolebindings -o wide

NAME                                                   ROLE                                                                       AGE   USERS                         GROUPS                         SERVICEACCOUNTS
cluster-admin                                          ClusterRole/cluster-admin                                                  10d   system:masters
system:kube-controller-manager                         ClusterRole/system:kube-controller-manager                                 10d
system:kube-scheduler                                  ClusterRole/system:kube-scheduler                                          10d
system:node                                            ClusterRole/system:node                                                    10d
...

The cluster-admin ClusterRoleBinding grants access to the system:masters group – the group your kubeconfig certificate belongs to. This is expected. Every other binding in this list is worth reviewing to understand what it grants and why.

What to look for: Any binding where the SERVICEACCOUNTS column is populated with an application service account (not a system: prefixed one) is a potential over-privilege finding. Application pods should never need cluster-admin.

Step 4: Verify the ci-pipeline service account

kubectl rbac-lookup ci-pipeline --kind ServiceAccount --output wide

Expected output:

SUBJECT                               SCOPE     ROLE             SOURCE
ServiceAccount/staging:ci-pipeline    staging   Role/ci-reader   RoleBinding/ci-reader-binding

The format is /<role-name> <binding-kind>/<binding-name>. This tells you:

• The service account is bound to the ci-reader Role

• The binding is a RoleBinding named ci-reader-binding

• There is no namespace prefix on the role name because it is a namespaced Role, not a ClusterRole

If the output showed ClusterRole/something here, that would be a finding. It would mean the service account has cluster-wide permissions, not namespace-scoped ones.

rbac-lookup vs kubectl get: rbac-lookup gives you a subject-centric view: "what does this account have access to?" kubectl get rolebindings,clusterrolebindings -A gives you a binding-centric view: "what bindings exist in the cluster?" Use both. rbac-lookup is faster for auditing a specific service account, while the kubectl get approach is better for a full cluster inventory.

With RBAC locked down, the API server is protected. But RBAC says nothing about what a container can do once it's running. That's a separate layer entirely.

How to Harden Pod Runtime Security

RBAC controls who can talk to the Kubernetes API. Pod security controls what containers can do once they're running on a node. These are different threat vectors: RBAC protects the control plane, pod security protects the data plane.

A container that runs as root with no capability restrictions can, if compromised, write backdoors to the host filesystem, load kernel modules, read the memory of other processes if hostPID: true is set, and in some configurations escape the container entirely. Pod security closes these doors before an attacker can open them.

A Case Study: The Hildegard Malware Campaign

In early 2021, Palo Alto's Unit 42 research team documented a cryptomining malware campaign called Hildegard that specifically targeted Kubernetes clusters. The attack chain was:

1. Find a cluster with the kubelet API exposed without authentication

2. Deploy a privileged pod with hostPID: true

3. Use the privileged pod to read credentials from other containers' memory

4. Establish persistence by writing to the host filesystem

Steps 3 and 4 would have been impossible if the pods in the cluster had been running with readOnlyRootFilesystem: true, dropped capabilities, and no hostPID. The attacker had the initial foothold. Pod security would have contained the blast radius.

Pod Security Admission

Pod Security Admission (PSA) is the built-in admission controller that enforces pod security standards at the namespace level. It replaced PodSecurityPolicy in Kubernetes 1.25.

Migrating from PSP? If you're on Kubernetes < 1.25, you may still be using PodSecurityPolicy, which was removed in 1.25. The migration path is: enable PSA in audit mode first to identify violations, fix them workload by workload, then switch to enforce. For policies PSA cannot express, add Kyverno alongside it.

PSA defines three profiles:

And three enforcement modes:

The migration path: start with audit and warn to identify violations, fix them, then switch to enforce. The two modes can run simultaneously.

Apply them as namespace labels:

# namespace-staging.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: staging
  labels:
    # Start here: audit and warn simultaneously
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Once violations are resolved, add enforce:

kubectl label namespace staging \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  --overwrite

Note: don't use --overwrite here. Without it, if enforce is already set to a different value the command will error – which is exactly what you want. You should see:

namespace/staging labeled

If you see namespace/staging not labeled, it means enforce=restricted and enforce-version=latest were already set to those exact values. Confirm enforcement is active:

kubectl get namespace staging --show-labels

Look for pod-security.kubernetes.io/enforce=restricted in the output. If it's there, enforcement is active.

How to Configure securityContext

A securityContext defines the privilege and access control settings for a pod or container. These are the seven fields you should configure on every production workload:

The annotated YAML below shows all seven in context:

# secure-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: staging
spec:
  replicas: 2
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      securityContext:
        runAsNonRoot: true         # container must run as a non-root user
        runAsUser: 10001           # explicit UID — don't rely on the image's default
        runAsGroup: 10001          # explicit GID
        fsGroup: 10001             # volumes are owned by this group
        seccompProfile:
          type: RuntimeDefault     # use the container runtime's default seccomp profile
      automountServiceAccountToken: false
      containers:
        - name: app
          image: nginx:1.25-alpine
          securityContext:
            allowPrivilegeEscalation: false   # block setuid and sudo inside the container
            readOnlyRootFilesystem: true      # the single highest-impact setting
            capabilities:
              drop:
                - ALL                         # drop every Linux capability
              add: []                         # add back only what is explicitly needed
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-run
              mountPath: /var/run
      volumes:
        # nginx needs writable directories — provide them as emptyDir volumes
        - name: tmp
          emptyDir: {}
        - name: nginx-cache
          emptyDir: {}
        - name: nginx-run
          emptyDir: {}

Why readOnlyRootFilesystem: true is the most important setting:

Most post-exploitation techniques require writing to the filesystem. Dropping a backdoor, modifying a binary, writing a cron job, or installing a keylogger all require a writable filesystem. Set readOnlyRootFilesystem: true and every one of these techniques is blocked.

The downside is that many applications write to directories like /tmp or /var/cache. The fix is to mount emptyDir volumes at those specific paths, as shown above. The rest of the filesystem stays read-only.

What each field prevents:

OPA/Gatekeeper vs Kyverno

PSA covers the fundamentals. But you'll eventually need policies that PSA cannot express: all images must come from your private registry, all pods must have resource limits, no container may use the latest tag. For these, you need a policy engine.

Two mature options exist:

If you're starting fresh, Kyverno gets you to working policies faster. Here is a Kyverno policy that blocks images from outside your trusted registry:

# kyverno-registry-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-registries
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "Images must come from registry.corp.internal/"
        pattern:
          spec:
            containers:
              - image: "registry.corp.internal/*"

How to Detect Runtime Threats with Falco

PSA and securityContext are preventive controls: they block known-bad configurations before pods start. Falco is a detective control. It watches what containers do while they're running and alerts when something looks wrong.

Falco operates at the syscall level using eBPF. It attaches to the Linux kernel and intercepts every system call made by every container on the node – file opens, network connections, process spawns, privilege escalations. It does this without modifying containers, without injecting sidecars, and with minimal overhead.

What Falco detects out of the box:

Falco's default ruleset covers the most common attack patterns. It fires when a shell is opened inside a running container, whether that's a kubectl exec session or a reverse shell from an exploit.

It watches for reads on sensitive files like /etc/shadow, /etc/kubernetes/admin.conf, and /root/.ssh/. It catches the dropper pattern: a binary written to disk and immediately executed. It detects outbound connections to known malicious IPs, writes to /proc or /sys that suggest kernel manipulation, and package managers like apt, yum, or pip being run inside containers that have no business installing software.

Each of these is a rule in Falco's default ruleset. You can extend it with custom rules for your specific workloads – which is exactly what you'll do in Demo 5. But first let's harden the Pod.

Demo 4 – Harden a Pod with securityContext

In this demo, you'll start with a default nginx deployment, observe the PSA violations it triggers, harden it step by step, and confirm it passes under the restricted profile.

Step 1: Apply PSA labels in audit mode

kubectl label namespace staging \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

Step 2: Deploy insecure nginx and observe the warnings

# insecure-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-insecure
  namespace: staging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-insecure
  template:
    metadata:
      labels:
        app: nginx-insecure
    spec:
      containers:
        - name: nginx
          image: nginx:1.25-alpine

kubectl apply -f insecure-nginx.yaml

Expected output (PSA warns but still creates the deployment in warn mode):

Warning: would violate PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false (container "nginx" must set
    securityContext.allowPrivilegeEscalation=false)
  unrestricted capabilities (container "nginx" must set
    securityContext.capabilities.drop=["ALL"])
  runAsNonRoot != true (pod or container "nginx" must set
    securityContext.runAsNonRoot=true)
  seccompProfile not set (pod or container "nginx" must set
    securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx-insecure created

Four violations. Every one of them is a real security gap. But the pod was still created "deployment.apps/nginx-insecure created"

Step 3: Deploy the hardened version

kubectl apply -f secure-deployment.yaml   # the YAML from the securityContext section above

No warnings this time.

Step 4: Switch the namespace to enforce

kubectl label namespace staging \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest

Expected output:

namespace/staging labeled

This is the moment enforcement becomes active. Any new pod that violates the restricted profile will be rejected from this point on.

Step 5: Confirm insecure deployments are now rejected

kubectl delete deployment nginx-insecure -n staging
kubectl apply -f insecure-nginx.yaml

Expected output:

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false ...
deployment.apps/nginx-insecure created

The Deployment object is created. PSA enforces at the pod level, not the Deployment level. The Deployment and its ReplicaSet exist, but every attempt to create a pod is rejected. Check the ReplicaSet:

kubectl get replicaset -n staging -l app=nginx-insecure

NAME                       DESIRED   CURRENT   READY   AGE
nginx-insecure-b668d867b   1         0         0       30s

DESIRED=1 but CURRENT=0. The ReplicaSet cannot create any pods because they're rejected at admission. Describe the ReplicaSet to see the rejection events:

kubectl describe replicaset -n staging -l app=nginx-insecure

Warning  FailedCreate  ReplicaSet "nginx-insecure-b668d867b" create Pod
  "nginx-insecure-xxx" failed: pods is forbidden: violates PodSecurity
  "restricted:latest": allowPrivilegeEscalation != false, unrestricted
  capabilities, runAsNonRoot != true, seccompProfile not set

The hardened deployment continues running with its pods intact. The insecure one has zero pods and never will. This is exactly how PSA is supposed to work.

Step 6: Score the hardened pod with kube-score

kube-score is a static analysis tool that scores Kubernetes manifests against security and reliability best practices:

# macOS
brew install kube-score
# Linux: https://github.com/zegl/kube-score/releases

kube-score score secure-deployment.yaml -v

Expected output (abridged):

apps/v1/Deployment secure-app in staging 
  path=secure-deployment.yaml
    [OK] Stable version
    [OK] Label values
    [CRITICAL] Container Resources
        · app -> CPU limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.cpu
        · app -> Memory limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.memory
        · app -> CPU request is not set
            Resource requests are recommended to make sure that the application can start and run without crashing. Set resources.requests.cpu
        · app -> Memory request is not set
            Resource requests are recommended to make sure that the application can start and run without crashing. Set resources.requests.memory
    [CRITICAL] Container Image Pull Policy
        · app -> ImagePullPolicy is not set to Always
            It's recommended to always set the ImagePullPolicy to Always, to make sure that the imagePullSecrets are always correct, and to always get the image you want.
    [OK] Pod Probes Identical
    [CRITICAL] Container Ephemeral Storage Request and Limit
        · app -> Ephemeral Storage limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.ephemeral-storage
        · app -> Ephemeral Storage request is not set
            Resource requests are recommended to make sure the application can start and run without crashing. Set resource.requests.ephemeral-storage
    [OK] Environment Variable Key Duplication
    [OK] Container Security Context Privileged
    [OK] Pod Topology Spread Constraints
        · Pod Topology Spread Constraints
            No Pod Topology Spread Constraints set, kube-scheduler defaults assumed
    [OK] Container Image Tag
    [CRITICAL] Pod NetworkPolicy
        · The pod does not have a matching NetworkPolicy
            Create a NetworkPolicy that targets this pod to control who/what can communicate with this pod. Note, this feature needs to be supported by the CNI implementation used in the Kubernetes cluster to have an effect.
    [OK] Container Security Context User Group ID
    [OK] Container Security Context ReadOnlyRootFilesystem
    [CRITICAL] Deployment has PodDisruptionBudget
        · No matching PodDisruptionBudget was found
            It's recommended to define a PodDisruptionBudget to avoid unexpected downtime during Kubernetes maintenance operations, such as when draining a node.
    [WARNING] Deployment has host PodAntiAffinity
        · Deployment does not have a host podAntiAffinity set
            It's recommended to set a podAntiAffinity that stops multiple pods from a deployment from being scheduled on the same node. This increases availability in case the node becomes unavailable.
    [OK] Deployment Pod Selector labels match template metadata labels

Notice there are no security context violations: securityContext, readOnlyRootFilesystem, seccompProfile, and runAsNonRoot all pass. The remaining findings are about resource management (CPU/memory limits, ephemeral storage), availability (PodDisruptionBudget, anti-affinity), and network policy – not security context hardening. Those are important for production readiness, but they're a separate concern from the pod security hardening we did here.

You now have a pod that PSA accepts and kube-score validates. The next step is to add a detection layer – something that watches what the pod does at runtime, not just how it was configured at admission.

Demo 5 – Deploy Falco and Write a Custom Detection Rule

Now, you'll deploy Falco in eBPF mode, trigger a default alert, then extend Falco with a custom rule that catches curl and wget being run inside containers.

Step 1: Install Falco via Helm

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set tty=true \
  --wait

Confirm Falco is running on every node:

kubectl get pods -n falco

NAME           READY   STATUS    RESTARTS   AGE
falco-x8k2p    1/1     Running   0          45s
falco-m9nqr    1/1     Running   0          45s
falco-j4tpw    1/1     Running   0          45s

One pod per node. Falco runs as a DaemonSet because it needs to monitor syscalls on every node independently.

Step 2: Trigger a default alert

Open a second terminal and stream the Falco logs:

# Terminal 2 — watch for alerts
kubectl logs -n falco -l app.kubernetes.io/name=falco -f --max-log-requests 3

In your first terminal, exec into the secure-app pod:

# Terminal 1 — trigger the shell detection
POD=$(kubectl get pod -n staging -l app=secure-app \
  -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD -n staging -- sh

Within a second, Terminal 2 shows:

2024-03-15T14:23:41.456Z: Notice A shell was spawned in a container with an attached terminal
  (user=root user_loginuid=-1 k8s.ns=staging k8s.pod=secure-app-7d9f8b-xxx
   container=app shell=sh parent=runc cmdline=sh terminal=34816)
  rule=Terminal shell in container  priority=NOTICE
  tags=[container, shell, mitre_execution]

This is Falco's built-in Terminal shell in container rule firing. It detected the kubectl exec session the moment you ran it.

Step 3: Write a custom rule

The built-in rules are comprehensive, but every production environment has workloads with unique behaviour. Here is a custom rule that alerts when curl or wget is executed inside any container:

# custom-rules.yaml
customRules:
  custom-rules.yaml: |-
    - rule: Suspicious network tool in container
      desc: >
        Detects execution of curl or wget inside a running container.
        These tools are commonly used for data exfiltration, downloading
        attacker payloads, or reaching command-and-control servers.
        Production containers should not be making ad-hoc HTTP requests.
      condition: >
        spawned_process
        and container
        and proc.name in (curl, wget)
      output: >
        Network tool executed in container
        (user=%user.name tool=%proc.name cmd=%proc.cmdline
         pod=%k8s.pod.name ns=%k8s.ns.name image=%container.image)
      priority: WARNING
      tags: [network, exfiltration, custom]

Apply it by upgrading the Helm release:

 helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set driver.kind=modern_ebpf \
  --set tty=true \
  -f custom-rules.yaml

Good, it deployed. Now wait for pods to be ready and test your custom rule:

Step 4: Test the custom rule

# Terminal 1 — run curl inside the container
kubectl exec -it $POD -n staging -- sh -c 'curl https://example.com'

Terminal 2 immediately shows:

2024-03-15T14:31:07.812Z: Warning Network tool executed in container
  (user=root tool=curl cmd=curl https://example.com
   pod=secure-app-7d9f8b-xxx ns=staging image=nginx:1.25-alpine)
  rule=Suspicious network tool in container  priority=WARNING
  tags=[network, exfiltration, custom]

Step 5: Route alerts to Slack with Falcosidekick

Streaming logs is useful during development. In production, you need alerts routed to your alerting pipeline. Falcosidekick handles this with support for Slack, PagerDuty, Datadog, Elasticsearch, and over 50 other outputs:

# falcosidekick-values.yaml
config:
  slack:
    webhookurl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
    minimumpriority: "warning"
    messageformat: >
      [{{.Priority}}] {{.Rule}} |
      pod: {{.OutputFields.k8s.pod.name}} |
      ns: {{.OutputFields.k8s.ns.name}} |
      image: {{.OutputFields.container.image}}

helm install falcosidekick falcosecurity/falcosidekick \
  --namespace falco \
  -f falcosidekick-values.yaml

Tuning Falco for production: A fresh Falco deployment will generate false positives, especially in the first week. Your job is to tune rules to match your workloads' normal behaviour, not to respond to every alert.

Here's the workflow: deploy in staging → identify false positives → add except conditions to rules → validate the false positive rate is low → enable in production with alerting.

Cleanup

To remove everything created in this article:

# Delete the staging namespace and everything in it
kubectl delete namespace staging
 
# Delete Falco and Falcosidekick
helm uninstall falco -n falco
helm uninstall falcosidekick -n falco
kubectl delete namespace falco
 
# Delete the kind cluster entirely
kind delete cluster --name k8s-security

Conclusion

In this handbook, you secured a Kubernetes cluster across three layers: RBAC, pod runtime security, and runtime threat detection.

You built a least-privilege service account, enforced the restricted Pod Security Admission profile, hardened pods with securityContext, deployed Falco for syscall-level detection, and wrote a custom rule to catch suspicious tools inside containers.

Each layer maps to a real-world breach – Tesla, Capital One, Hildegard – showing how these controls would have contained the damage. Run kube-bench again to measure the improvement.

All YAML manifests, Helm values, and setup scripts from this article are available in the companion GitHub repository.

We’ve all been there: dozens of experiments scattered across random notebooks, and model files saved as model_v2_final_FINAL.pkl because no one is quite sure which version actually worked.

Once you move from a solo project to a team, or try to push something to production, that "organized chaos" quickly becomes a serious bottleneck.

Solving this mess requires more than just better naming conventions: it requires a way to standardize how we track and hand off our work. This is the specific gap MLflow was built to fill.

Originally released by the team at Databricks in 2018, it has become a standard open-source platform for managing the entire machine learning lifecycle. It acts as a central hub where your experiments, code, and models live together, rather than being tucked away in forgotten folders.

In this tutorial, we'll cover the core philosophy behind MLflow and how its modular architecture solves the 'dependency hell' of machine learning. We'll break down the four primary pillars of Tracking, Projects, Models, and the Model Registry, and walk through a practical implementation of each so you can move your projects from local notebooks to a production-ready lifecycle.

Table of Contents:

• Prerequisites:

• MLflow Architecture: The Big Picture

• Understanding MLflow Tracking

  • A Tracking Example

  • Where Does the Data Actually Go?

  • Why Bother with This Setup?

• Understanding MLflow Projects

  • The MLproject File

  • Why this Actually Matters

• Understanding the MLflow Model Registry

• Moving a Model through the Pipeline

  • Why Does This Matter?

• How the Components Fit Together

• Wrapping Up

Prerequisites:

To get the most out of this tutorial, you should have:

• Basic Python proficiency: Comfort with context managers (with statements) and decorators.

• Machine Learning fundamentals: A general understanding of training/testing splits and model evaluation metrics (like accuracy or loss).

• Local Environment: Python 3.8+ installed. Familiarity with pip or conda for installing packages is helpful.

MLflow Architecture: The Big Picture

To understand why MLflow is so effective, you have to look at how it's actually put together. MLflow isn't one giant or rigid tool. It’s a modular system designed around four loosely coupled components that are its core pillars.

This is a big deal because it means you don’t have to commit to the entire ecosystem at once. If you only need to track experiments and don't care about the other features, you can just use that part and ignore the rest.

To make this a bit more concrete, here is how those pieces map to things you probably already use:

• MLflow Tracking: Logs experiments, metrics, and parameters. (Think: Git commits for ML runs)

• MLflow Projects: Packages code for reproducibility. (Think: A Docker image for ML code)

• MLflow Models: A standard format for multiple frameworks. (Think: A universal adapter)

• Model Registry: Handles versioning and governing models. (Think: A CI/CD pipeline for models)

Architecturally, you can think of MLflow in two layers: the Client and the Server.

The Client is where you spend most of your time. It’s your training script or your Jupyter notebook where you log metrics or register a model.

The Server is the brain in the background that handles the storage. It consists of a Tracking Server, a Backend Store (usually a database like PostgreSQL), and an Artifact Store. That’s the place where big files like model weights live, such as S3 or GCS.

This separation is why MLflow is so flexible. You can start with everything running locally on your laptop using just your file system. When you're ready to scale up to a larger team, you can swap that out for a centralized server and cloud storage with almost no changes to your actual code. It grows with your project instead of forcing you to start over once things get serious.

Now, let's look at each of these four pillars of MLflow so you understand how they work.

Understanding MLflow Tracking

For most teams, the Tracking component is the front door to MLflow. Its job is simple: it acts as a digital lab notebook that records everything happening during a training run.

Instead of you frantically trying to remember what your learning rate was or where you saved that accuracy plot, MLflow just sits in the background and logs it for you.

The core unit here is the run. Think of a run as a single execution of your training code. During that run, the architecture captures four specific types of information:

• Parameters: Your inputs, like batch size or the number of trees in a forest.

• Metrics: Your outputs, like accuracy or loss, which can be tracked over time.

• Artifacts: The "heavy" stuff, such as model weights, confusion matrices, or images.

• Tags and Metadata: Context like which developer ran the code and which Git commit was used.

A Tracking Example

Seeing this in practice is the best way to understand how the architecture actually works. You don't need to rebuild your entire pipeline – you just wrap your training logic in a context manager.

Here is what a basic integration looks like in Python:

import mlflow 
import mlflow.sklearn 
from sklearn.ensemble import RandomForestClassifier 
from sklearn.metrics import accuracy_score 

# This block opens the run and keeps things organized
with mlflow.start_run():    
    # Log parameters    
    mlflow.log_param("n_estimators", 100)    
    mlflow.log_param("max_depth", 5)    
    
    # Train the model    
    model = RandomForestClassifier(n_estimators=100, max_depth=5)    
    model.fit(X_train, y_train)    
    
    # Log metrics    
    accuracy = accuracy_score(y_test, model.predict(X_test))    
    mlflow.log_metric("accuracy", accuracy)    
    
    # Log the model as an artifact    
    mlflow.sklearn.log_model(model, "random_forest_model")

The mlflow.start_run() context manager creates a new run and automatically closes it when the block exits. Everything logged inside that block is associated with that run and stored in the Backend Store.

Where Does the Data Actually Go?

When you’re just starting out on your laptop, MLflow keeps things simple by creating a local ./mlruns directory. The real power shows up when you move to a team environment and point everyone to a centralized Tracking Server.

The system splits the data based on how "heavy" it is. Your structured data (parameters and metrics) is small and needs to be searchable, so it goes into a SQL database like PostgreSQL. Your unstructured data (the actual model files or large plots) is too bulky for a database. The architecture ships that off to an Artifact Store like Amazon S3 or Google Cloud Storage.

Why Bother with This Setup?

Relying on "vibes" and messy naming conventions is a recipe for disaster once your project grows. It might work for a day or two, but it falls apart the moment you need to compare twenty different versions of a model.

By separating the tracking into its own architectural pillar, MLflow gives you a queryable history. Instead of digging through old notebooks, you can just hop into the UI, filter for the best results, and see exactly which configuration got you there. It takes the guesswork out of the "science" part of data science.

Understanding MLflow Projects

You can train the most accurate model in the world, but if your colleague can’t reproduce your results on their machine, that model isn't worth much.

This is where MLflow Projects come in. They solve the reproducibility headache by providing a standard way to package your code, your dependencies, and your entry points into one neat bundle.

Think of an MLflow Project as a directory (or a Git repo) with a special "instruction manual" at its root called an MLproject file. This file tells anyone (or any server) exactly what environment is needed and how to kick off the execution.

The MLproject File

Instead of sending someone a long README with installation steps, you just give them this file. Here is what a typical MLproject setup looks like for a training pipeline:

name: my_ml_project
conda_env: conda.yaml

entry_points:
  train:
    parameters:
      learning_rate: {type: float, default: 0.01}
      epochs: {type: int, default: 50}
      data_path: {type: str}
    command: "python train.py --lr {learning_rate} --epochs {epochs} --data {data_path}"
  
  evaluate:
    parameters:
      model_path: {type: str}
    command: "python evaluate.py --model {model_path}"

The conda_env line points to a conda.yaml file that lists the exact Python packages and versions your code needs. If you want even more isolation, MLflow supports Docker environments too.

The beauty of this setup is the simplicity. Anyone with MLflow installed can run your entire project with a single command:

mlflow run . -P learning_rate=0.001 -P epochs=100 -P data_path=./data/train.csv

Why this Actually Matters

MLflow Projects really shine in two specific scenarios. The first is onboarding. A new team member can clone your repo and be up and running in minutes, rather than spending their entire first day debugging library version conflicts.

The second is CI/CD. Because these projects are triggered programmatically, they fit perfectly into automated retraining pipelines. When reproducibility is non-negotiable, having a "single source of truth" for how to run your code makes life a lot easier for everyone involved.

Understanding the MLflow Model Registry

Tracking experiments tells you which model is the "winner," but the Model Registry is where you actually manage that winner’s journey from your notebook to a live production environment.

Think of it as the governance layer. It handles versioning, stage management, and creates a clear audit trail so you never have to guess which model is currently running in the wild.

The Registry uses a few simple concepts to keep things organized:

• Registered Model: This is the overall name for your project, like CustomerChurnPredictor.

• Model Version: Every time you push a new iteration, MLflow auto-increments the version (v1, v2, and so on).

• Stage: These are labels like Staging, Production, or Archived. They tell your team exactly where a model stands in its lifecycle.

• Annotations: These are just notes and tags. They’re great for documenting why a specific version was promoted or what its quirks are.

Moving a Model through the Pipeline

In a real-world workflow, you don't just "deploy" a file. You transition it through stages. Here's how that looks using the MLflow Client:

Python
import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient()

# First, we register the model from a run that went well
result = mlflow.register_model(
    model_uri=f"runs:/{run_id}/random_forest_model",
    name="CustomerChurnPredictor"
)

# Then, we move Version 1 to Staging so the QA team can look at it
client.transition_model_version_stage(
    name="CustomerChurnPredictor",
    version=1,
    stage="Staging"
)

# Once everything checks out, we promote it to Production
client.transition_model_version_stage(
    name="CustomerChurnPredictor",
    version=1,
    stage="Production"
)

Why Does This Matter?

The Model Registry solves a problem that usually gets messy the moment a team grows: knowing exactly which version is live, who approved it, and what it was compared against. Without this, that information usually ends up buried in Slack threads or outdated spreadsheets.

It also makes rollbacks incredibly painless. If Version 3 starts acting up in production, you don't need to redeploy your entire stack. You can just transition Version 2 back to the "Production" stage in the registry. Since your serving infrastructure is built to always pull the "Production" tag, it will automatically swap back to the stable version.

How the Components Fit Together

To see how all of this actually works in the real world, it helps to walk through a typical workflow from start to finish. It's essentially a relay race where each component hands off the baton to the next one.

It starts with a data scientist running a handful of experiments. Every time they hit run, MLflow Tracking is in the background taking notes. It logs metrics and saves model artifacts into the Backend Store automatically. At this stage, everything is about exploration and finding that one winner.

Once that best run is identified, the model gets officially registered in the Model Registry. This is where the team takes over. They can hop into the UI to check the annotations, review the evaluation results, and move the model into Staging. After it passes a few more validation tests, it gets the green light and is promoted to Production.

When it is time to actually serve the model, the deployment system simply asks the Registry for the current Production version. This happens whether you are using Kubernetes, a cloud endpoint, or MLflow’s built-in server.

Because the MLproject file handled the dependencies and the MLflow Models format handled the framework details, the serving infrastructure does not have to care if the model was built with Scikit-learn or PyTorch. The hand-off is smooth because all the necessary info is already there.

This flow is what turns MLflow from a collection of useful utilities into a full MLOps platform. It connects the messy experimental phase of data science to the rigid world of production software.

Wrapping Up

At the end of the day, MLflow architecture is built to stay out of your way. It doesn't force you to change how you write your code or which libraries you use. Instead, it just provides the structure needed to make your machine learning projects reproducible and easier to manage as a team.

Whether you're just trying to get away from naming files model_final_v2.pkl or you are building a complex CI/CD pipeline for your models, understanding these four pillars is the best place to start. The best way to learn is to just fire up a local tracking server and start logging. You will probably find that once you have that "source of truth" for your experiments, you will never want to go back to the old way of doing things.

I get it. I held that same opinion for years. Compose was the thing I used to spin up a Postgres database on my laptop, not something I'd trust with a staging environment, let alone a workload that needed GPU access.

Then 2024 and 2025 happened. Docker shipped a set of features that quietly transformed Compose from a developer convenience tool into something that can handle complex deployment scenarios. Profiles let you manage multiple environments from a single file. Watch mode killed the painful rebuild cycle that made container-based development feel sluggish. GPU support opened the door to ML inference workloads. And a bunch of smaller improvements (better health checks, Bake integration, structured logging) filled in the gaps that used to make Compose feel like a toy.

Here's what I'll cover: using Docker Compose profiles to manage multiple environments from one file, setting up watch mode for instant code syncing during development, configuring GPU passthrough for machine learning workloads, implementing proper health checks and startup ordering so your services stop crashing on cold starts, and using Bake to bridge the gap between your local Compose workflow and production image builds. I'll also tell you where Compose still falls short and where you should reach for something else.

Prerequisites

You should be comfortable with Docker basics and have written a compose.yaml file before. You'll need Docker Compose v2 installed. The minimum version depends on which features you want: service_healthy dependency conditions require v2.20.0+, watch mode requires v2.22.0+, and the gpus: shorthand requires v2.30.0+. Run docker compose version to check what you have.

Table of Contents

• Prerequisites

• The Modern Compose File: What's Changed

• How to Use Profiles to Manage Multiple Environments

  • Real-World Profile Patterns I've Used

• How to Use Watch Mode to End the Rebuild Cycle

  • Watch Mode vs. Bind Mounts

• How to Set Up GPU Support for Machine Learning Workloads

  • How to Combine Multi-GPU Workloads with Profiles

• How to Configure Health Checks, Dependencies, and Startup Ordering

• How to Use Bake for Production Image Builds

• What Compose Is Not (An Honest Assessment)

• A Practical Adoption Path

• Wrapping Up

The Modern Compose File: What's Changed

If you haven't looked at a Compose file recently, the first thing you'll notice is that the version field is gone. Docker Compose v2 ignores it entirely, and including it actually triggers a deprecation warning. A modern compose.yaml starts cleanly with your services, no preamble needed.

But the structural changes go deeper than that. Here's what a modern, production-aware Compose file looks like for a typical web application stack:

services:
  api:
    image: ghcr.io/myorg/api:${TAG:-latest}
    env_file: [configs/common.env]
    environment:
      - NODE_ENV=${NODE_ENV:-production}
    ports:
      - "8080:8080"
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "1.0"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  db:
    image: postgres:16-alpine
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 5

volumes:
  db-data:

Look at what's in there: resource limits, health checks with dependency conditions, proper volume management. These aren't nice-to-haves. They're the features that make Compose viable beyond your laptop.

Health checks in particular solve one of Compose's oldest and most annoying pain points: the race condition where your web server starts before the database is actually ready to accept connections. If you've ever added sleep 10 to a startup script and crossed your fingers, you know what I'm talking about.

How to Use Profiles to Manage Multiple Environments

This is the feature that changed my relationship with Compose. Before profiles, managing different environments meant choosing between two painful approaches. Either you maintained multiple Compose files (docker-compose.yml, docker-compose.dev.yml, docker-compose.test.yml, docker-compose.prod.yml) and dealt with the inevitable drift between them. Or you used one big bloated file where you commented out services depending on the context. Both approaches were fragile, and both led to those fun "works on my machine" conversations.

Profiles give you a much cleaner path. You assign services to named groups. Services without a profile always start. Services with a profile only start when you explicitly activate that profile. You can also activate profiles with the COMPOSE_PROFILES environment variable instead of the CLI flag, which is handy for CI (see the official profiles docs for the full syntax).

Here's what that looks like:

services:
  api:
    image: myapp:latest
    # No profiles = always starts

  db:
    image: postgres:16
    # No profiles = always starts

  debug-tools:
    image: busybox
    profiles: [debug]
    # Only starts with --profile debug

  prometheus:
    image: prom/prometheus
    profiles: [monitoring]
    # Only starts with --profile monitoring

  grafana:
    image: grafana/grafana
    profiles: [monitoring]
    depends_on: [prometheus]

Now your team operates with simple, memorable commands:

# Development: just the core stack
docker compose up -d

# Development with observability
docker compose --profile monitoring up -d

# CI: core stack only (no monitoring overhead)
docker compose up -d

# Full stack with debugging
docker compose --profile debug --profile monitoring up

One Compose file. No drift. No guesswork about which override file to pass.

Real-World Profile Patterns I've Used

Four patterns I keep coming back to:

The "infra-only" pattern. This is for developers who run application code natively on their host machine but need infrastructure services like databases, message queues, and caches in containers. You leave infrastructure services without a profile and put application services behind one. Your backend developer runs docker compose up to get Postgres and Redis, then starts the API directly on their host with their favorite debugger attached.

The "mock vs. real" pattern. You put a payments-mock service in the dev profile and a real payments gateway service in the prod profile. Same Compose file, totally different behavior depending on context. This one saved my team from accidentally hitting a live payment API during development more than once.

The "CI optimization" pattern. Heavy services like Selenium browsers and monitoring stacks go behind profiles so your CI pipeline skips them. Your test suite runs faster without that overhead, and you only pull those services in when you actually need end-to-end integration tests.

The "AI/ML workloads" pattern. GPU-dependent services (inference servers, model training containers) go into a gpu profile. Developers without GPUs can still work on the rest of the stack without anything breaking.

One practical tip that's saved me a lot of headaches: document your profiles in the project's README. It sounds obvious, but when a new team member runs docker compose up and wonders why the monitoring dashboard isn't starting, they need a single place to find the answer. A quick table listing each profile and what it includes will save you from answering the same Slack question every onboarding cycle.

How to Use Watch Mode to End the Rebuild Cycle

If profiles solved the environment management problem, watch mode solved the developer experience problem.

You probably know the old workflow for container-based development. It went like this: edit code, run docker compose build, run docker compose up, test your change, find a bug, edit again, rebuild, restart, test. Each iteration costs you thirty seconds to a minute of waiting. Over a full day of active development, you're losing an hour or more just sitting there watching build logs scroll by.

Watch mode (introduced in Compose v2.22.0 and significantly improved in later releases) monitors your local files and automatically takes action when something changes. It supports three synchronization strategies, and picking the right one for each situation is the key to making it work well. The official watch mode docs cover the full spec if you want to dig deeper.

sync copies changed files directly into the running container. This works best for interpreted languages like Python, JavaScript, and Ruby, and for frameworks with hot module reloading like React, Vue, or Next.js. The file lands in the container, the framework picks up the change, and your browser updates. No rebuild, no restart. If you're working with a compiled language like Go, Rust, or Java, sync won't help you since the code needs to be recompiled. Use rebuild for those instead.

rebuild triggers a full image rebuild and container replacement. You want this for dependency changes, like when you update package.json or requirements.txt, or when you modify the Dockerfile itself. In those cases, syncing files isn't enough. You need a fresh image.

sync+restart syncs files into the container, then restarts the main process. This is ideal for configuration file changes like nginx.conf or database configs, where the application needs to reload to pick up the new settings but the image itself is fine.

Here's what a real-world watch configuration looks like for a Node.js application:

services:
  api:
    build: .
    ports: ["3000:3000"]
    command: npx nodemon server.js
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src
          ignore:
            - node_modules/
        - action: rebuild
          path: package.json
        - action: sync+restart
          path: ./config
          target: /app/config

You start it with docker compose up --watch, or you can run docker compose watch as a standalone command if you'd rather keep the file sync events separate from your application logs.

A few things to know before you set this up. Watch mode only works with services that have a local build: context. If you're pulling a prebuilt image from a registry, there's nothing for Compose to sync or rebuild, so watch will ignore that service. Your container also needs basic file utilities (stat, mkdir) installed, and the container USER must have write access to the target path. If you're using a minimal base image like scratch or distroless, the sync action won't work. And if you're on an older Compose version, check which actions are supported: sync+restart and sync+exec were added in later minor releases after the initial v2.22.0 launch.

It's a massive improvement. Edit a source file, save it, and the change is live in under a second for frameworks with hot reload. No context switching to run build commands. No waiting. Just code.

Watch Mode vs. Bind Mounts

A fair question you might be asking: bind mounts have provided a form of live-reload for years. Why does watch mode need to exist?

Bind mounts work, but they come with platform-specific issues that have plagued Docker Desktop for a long time. On macOS and Windows, bind mounts go through a filesystem sharing layer between the host OS and the Linux VM running Docker. This introduces permission quirks, performance problems on large directories (ever watched a node_modules folder choke a bind mount on macOS?), and inconsistent file notification behavior that makes hot reload unreliable.

Watch mode sidesteps these issues by explicitly syncing files at the application level. It's more predictable, works consistently across platforms, and gives you more control over what happens when a file changes.

That said, bind mounts still work well for many use cases, especially if you're on native Linux where the performance overhead doesn't exist. Watch mode is the better choice for teams that have run into cross-platform issues, or for anyone who wants the automatic rebuild and restart triggers that bind mounts can't provide.

How to Set Up GPU Support for Machine Learning Workloads

This is the feature that made me rethink what Compose can do.

Docker has supported GPU passthrough for individual containers for years through the NVIDIA Container Toolkit and the --gpus flag. But configuring GPU access in Compose files used to require clunky runtime declarations that were poorly documented and changed between Compose versions. It was the kind of thing where you'd find a Stack Overflow answer from 2021, try it, and discover it didn't work anymore.

The modern Compose spec handles it cleanly through the deploy.resources.reservations.devices block:

services:
  inference:
    image: myorg/model-server:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

If you're on Compose v2.30.0 or later, there's also a shorter syntax using the gpus: field:

services:
  inference:
    image: myorg/model-server:latest
    gpus:
      - driver: nvidia
        count: 1

Both approaches do the same thing. The deploy.resources syntax works on older Compose versions and gives you more control (like setting device_ids to pin specific GPUs). The gpus: shorthand is cleaner when you just need basic access.

One thing that will trip you up if you skip it: your host machine needs the right GPU drivers and nvidia-container-toolkit installed before any of this works. Run nvidia-smi on the host first. If that command doesn't show your GPUs, Compose won't see them either. For CUDA workloads, use official GPU base images like nvidia/cuda or the PyTorch/TensorFlow GPU images. The Compose GPU access docs walk through the full setup.

That's the whole thing. When you run docker compose up, the inference service gets access to one NVIDIA GPU. You can set count to "all" if you want every available GPU, or use device_ids to assign specific GPUs to specific services.

How to Combine Multi-GPU Workloads with Profiles

Here's where profiles and GPU support work really well together. Consider an ML workload where you need an LLM for text generation, an embedding model for vector search, and a vector database:

services:
  vectordb:
    image: milvus/milvus:latest
    # Runs on CPU, no profile needed

  llm-server:
    image: ollama/ollama:latest
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]
    volumes:
      - model-cache:/root/.ollama

  embedding-server:
    image: myorg/embeddings:latest
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

Developers without GPUs work on the application logic with just docker compose up. The vector database starts, they can write code against its API, and everything runs fine. When it's time to test the full ML pipeline, someone with a multi-GPU workstation runs docker compose --profile gpu up and gets the complete stack with specific GPU assignments.

This pattern has become central to our AIOps platform development. The team building alerting logic doesn't need GPUs. The team training anomaly detection models does. One Compose file serves both teams.

How to Configure Health Checks, Dependencies, and Startup Ordering

One of Compose's most underappreciated improvements is how it handles service dependencies. The depends_on directive now supports conditions that actually mean something (this requires Compose v2.20.0+, see the startup ordering docs for the full picture):

depends_on:
  db:
    condition: service_healthy
  redis:
    condition: service_started

When you combine this with proper health checks, you eliminate the "sleep 10 and hope" pattern that plagues so many Compose setups. Your API service actually waits until PostgreSQL is accepting connections before it tries to start. Not just until the container is running, but until the database process inside it has passed its health check.

One detail that catches people: tune your start_period. Databases like PostgreSQL need time to initialize on first boot, especially if they're running migrations. Without a start_period, the health check starts counting retries immediately and can declare the service unhealthy before it even had a chance to finish starting up. A config like this works well for most database services:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres"]
  interval: 5s
  timeout: 2s
  retries: 10
  start_period: 30s

The start_period gives the container 30 seconds of grace time where failed health checks don't count against the retry limit.

This might seem like a small detail, but if you've ever worked on a stack with eight or ten interconnected services, you know how much time you can waste debugging cascading failures during cold starts. Proper startup ordering prevents all of that and makes your local environment behave much more like production.

How to Use Bake for Production Image Builds

I mentioned Bake integration earlier, and it's worth its own section because it solves a problem you'll hit as soon as you start using Compose for anything beyond local dev: your development Compose file and your production build process have different needs.

During development, you want fast builds, local caches, and single-platform images. For production, you want tagged images pushed to a registry, multi-platform builds, and build attestations. Trying to cram both into your compose.yaml gets messy fast.

Docker Bake (docker buildx bake) can read your compose.yaml and generate build targets from it, but you can override and extend those targets with a separate docker-bake.hcl file. This keeps your development workflow clean while giving CI the knobs it needs. The Bake documentation covers the full HCL syntax and Compose integration.

Here's a minimal docker-bake.hcl:

group "default" {
  targets = ["api", "worker"]
}

target "api" {
  context    = "api"
  dockerfile = "Dockerfile"
  tags       = ["registry.example.com/team/api:release"]
  platforms  = ["linux/amd64"]
}

target "worker" {
  context    = "worker"
  dockerfile = "Dockerfile"
  tags       = ["registry.example.com/team/worker:release"]
}

Then your CI pipeline runs docker buildx bake to produce release images, while developers keep using docker compose up --build locally. The two workflows share the same Dockerfiles but have separate build configurations where they need them.

The pattern I've landed on: use Compose for local development and CI test environments, use Bake in CI to produce the release images, and push those images into whatever deployment target your team uses (staging server, Kubernetes cluster, edge node). Compose gets you from code to running containers fast. Bake gets you from code to production-ready images with proper tags and attestations.

What Compose Is Not (An Honest Assessment)

I've spent this entire article making the case that Compose has grown up. But I should also tell you where it falls short. I'd rather you hear it from me now than discover it the hard way in production.

Compose is not a container orchestrator. It doesn't schedule work across multiple hosts. It doesn't do automatic failover. It won't give you rolling updates with zero downtime, and it has no concept of service mesh networking. If you need any of those things, you need Kubernetes, Nomad, or Docker Swarm (if you're still using it).

Compose doesn't replace Helm or Kustomize. If you're deploying to Kubernetes, Compose files don't translate directly. Docker offers Compose Bridge to convert Compose files into Kubernetes manifests, but it's still experimental and won't handle complex Kubernetes-specific configurations like custom resource definitions or ingress rules.

Compose doesn't handle secrets well in production. The secrets support exists, but it's limited compared to HashiCorp Vault, AWS Secrets Manager, or Kubernetes secrets. For anything beyond a staging environment, you'll want an external secrets management solution.

The sweet spot for modern Compose is clear: local development, CI/CD testing environments, single-node staging environments, and workloads where a single powerful machine (particularly for GPU work) is the right deployment target. Within that scope, Compose is excellent. Outside of it, you'll hit walls fast.

If you do run Compose in a staging or single-node production setup, a few more things are worth adding that I haven't covered here: restart: unless-stopped on every service so containers come back after a host reboot, a logging driver config so your logs go somewhere searchable instead of disappearing into docker logs, and a backup strategy for your named volumes. These aren't Compose-specific problems, but Compose won't solve them for you either.

A Practical Adoption Path

If you're currently working with a basic Compose setup and want to start using these features, here's the order I'd recommend. Each step is incremental, backward-compatible, and valuable on its own. You don't have to do all of this at once.

Week 1: Add health checks and proper depends_on conditions. This alone will eliminate the most common frustration: services crashing on startup because their dependencies aren't ready yet. Start with your database and your main application service. Once those two are wired up with condition: service_healthy, you'll notice the difference immediately.

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres"]
  interval: 5s
  timeout: 2s
  retries: 10
  start_period: 30s

Week 2: Introduce profiles. Start by putting your monitoring stack behind a monitoring profile and your debug tools behind a debug profile. Then delete whatever extra Compose files you've been maintaining. Having one source of truth instead of four files that are almost-but-not-quite the same makes everything simpler.

Week 3: Set up watch mode for your most-edited service. Pick the service where your developers spend the most time iterating. Get watch mode working there first. Once the team sees the difference (saving a file and seeing the change reflected in under a second) they'll ask for it on everything else.

Week 4: Add resource limits. Define memory and CPU limits for every service. This prevents one runaway container from starving the rest and gives you a realistic preview of how your services behave under production constraints. It's also useful for catching memory leaks early.

deploy:
  resources:
    limits:
      memory: 512M
      cpus: "1.0"

Wrapping Up

Docker Compose in 2026 is not the same tool it was a few years ago. Profiles, watch mode, GPU support, proper dependency management, and Bake integration have turned it into something that can handle real, complex workloads, as long as those workloads fit on a single node.

It's not Kubernetes, and it shouldn't try to be. But for local development, CI pipelines, staging environments, and single-machine GPU workloads, it's become hard to argue against. If you've been dismissing Compose because of what it used to be, the current version deserves a second look.

If you found this useful, you can find me writing about DevOps, containers, and AIOps best practices on my blog.

Understanding different container runtimes gives you more options. You can choose the right tool for your specific needs, whether that's better security, lower resource usage, or easier integration with Kubernetes.

In this tutorial, you'll learn about three major container runtimes and how to use them on your system. We’ll dive into practical examples with complete code you can run right now. By the end, you’ll understand when to use each runtime and how to move containers between them.

Table of Contents

1. What Are Container Runtimes?

2. How to Understand High-Level vs Low-Level Runtimes

3. How to Use Docker as Your Baseline

4. How to Use Podman – The Daemonless Alternative

5. How to Work with Containerd

6. How to Move Containers Between Runtimes

7. Real-World Use Cases

8. Quick Reference Guide

9. Conclusion

What Are Container Runtimes?

A container runtime is the software that actually runs your containers. When you type docker run nginx, for example, several things happen behind the scenes. The Docker CLI talks to the Docker daemon, which then uses a container runtime (usually containerd) to actually create and run the container.

Think of it like this: if containers are apps on your phone, the container runtime is the operating system that makes those apps work. Just like you can install the same app on different phones (iPhone vs Android), you can run the same container on different runtimes.

Why Does This Matter?

You might wonder why you should care about what's running your containers. Docker works fine, right? Here are a few reasons:

1. Security: Some runtimes like Podman can run containers without root privileges. This means if someone breaks out of your container, they don't have full system access.

2. Resource usage: Different runtimes use different amounts of memory and CPU. On a resource-constrained server or edge device, this matters a lot.

3. Integration: If you're deploying to Kubernetes, understanding containerd or CRI-O helps you troubleshoot production issues.

4. Licensing: Docker Desktop has licensing requirements for large companies. Alternatives like Podman are completely free.

Here’s a chart that summarizes these key points:

How to Understand High-Level vs Low-Level Runtimes

Container runtimes are split into two categories, and understanding this distinction helps you see how everything fits together.

Low-Level Runtimes

Low-level runtimes like runc and crun do the actual work of creating containers. They interact directly with the Linux kernel to create isolated environments using features like namespaces and cgroups.

Namespaces isolate what a process can see. For example, a process namespace means the container can't see other processes running on your system. A network namespace means it has its own network stack.

Cgroups (control groups) limit what a process can use. You can limit a container to 512MB of RAM or 50% of one CPU core. This prevents one container from hogging all your resources.

These low-level runtimes implement the OCI (Open Container Initiative) Runtime Specification. This is a standard that defines exactly how to run a container. Because of this standard, you can swap out runtimes and your containers still work.

High-Level Runtimes

High-level runtimes like Docker, Podman, and containerd manage images, networking, volumes, and provide user-friendly interfaces. They handle pulling images from registries, setting up networks between containers, and managing container lifecycles.

These high-level runtimes use low-level runtimes under the hood. When you run docker run, Docker ultimately calls runc to create the container. This layering means you get a nice interface while still benefiting from the standard, battle-tested low-level runtime.

Why This Layering Matters:

This separation of concerns is powerful. High-level runtimes can focus on user experience and features while low-level runtimes focus on reliably creating containers. You can swap low-level runtimes without changing your workflow. Some people use crun instead of runc because it's written in C and starts faster.

How to Use Docker as Your Baseline

Let's start with Docker since you're probably already familiar with it. This will give us a baseline to compare other runtimes against. We'll build a simple web application and then run the same application in different runtimes to see how they compare.

How to Install Docker

You can find installation guides for your operating system:

• Docker Desktop for Mac

• Docker Desktop for Windows

• Docker Engine for Linux

How to Run a Test Container

Let's verify that Docker works by running a simple container:

docker run hello-world

You should see a message that says:

Hello from Docker!
This message shows that your installation appears to be working correctly.

What Just Happened?

When you ran that command, Docker checked if the hello-world image exists locally. It didn't find it, so it pulled the image from Docker Hub (a public registry). Then it created a container from that image, started the container, and the container printed its message and exited.

All of this happened in a few seconds. Now let's build something more useful.

How to Create a Web Server

Create a new directory for your project:

mkdir ~/container-demo
cd ~/container-demo

The ~ symbol means your home directory. On macOS, this is /Users/yourname. On Linux, it's /home/yourname.

Create a simple HTML file:

cat > index.html << 'EOF'
<!DOCTYPE html>
<html>
<head><title>Container Demo</title></head>
<body>
  <h1>Hello from Docker!</h1>
  <p>This is running in a container.</p>
</body>
</html>
EOF

This creates a basic HTML file. The cat > command writes to a file, and << 'EOF' means "read until you see EOF" (End Of File). This is a handy way to create files from the command line.

How to Create a Dockerfile

You can create a dockerfile like this:

cat > Dockerfile << 'EOF'
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/
EOF

Understanding the Dockerfile:

The Dockerfile has two instructions:

1. FROM nginx:alpine: This starts with the official Nginx image. The :alpine tag means we're using the Alpine Linux version, which is much smaller (about 20MB instead of 130MB). Alpine is a minimal Linux distribution popular in containers because of its small size.

2. COPY index.html /usr/share/nginx/html/: This copies your HTML file into the location where Nginx serves files. Inside the container, Nginx is configured to serve files from /usr/share/nginx/html/.

How to Build a Docker Image

docker build -t my-web-app .

The -t flag means "tag" – we're naming the image my-web-app. The . at the end means "use the current directory as the build context". Docker will look for a Dockerfile in the current directory and send all files here to the Docker daemon for building.

You'll see output like:

[+] Building 2.3s (7/7) FINISHED
=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 98B
=> [internal] load .dockerignore
...
=> => naming to docker.io/library/my-web-app

This shows Docker building your image layer by layer. Each instruction in the Dockerfile creates a new layer. These layers are cached, so if you rebuild without changes, it's instant.

How to Run a Docker Container

docker run -d -p 8080:80 my-web-app

Understanding the Flags:

• -d means "detached mode" – run in the background. Without this, the container runs in the foreground and you'll see Nginx's log output. With -d, it returns immediately and runs in the background.

• -p 8080:80 maps port 8080 on your host machine to port 80 inside the container. Nginx listens on port 80 inside the container. To access it from your browser, you need to map it to a port on your machine. We chose 8080, but you could use any available port.

Open your browser and visit http://localhost:8080. You should see your HTML page!

How to Check Running Containers:

docker ps

This shows all running containers. You'll see something like:

CONTAINER ID   IMAGE        COMMAND                  PORTS                  NAMES
a1b2c3d4e5f6   my-web-app   "/docker-entrypoint.…"   0.0.0.0:8080->80/tcp   peaceful_curie

Docker automatically generated a random name (peaceful_curie in this example). You can specify a name with --name if you prefer.

How to View Container Logs:

docker logs <container-id>

Replace <container-id> with the ID from docker ps (just the first few characters work). This shows what's happening inside the container. For Nginx, you'll see access logs showing requests to your web server.

How to Stop the Container:

docker stop <container-id>

This gracefully stops the container. Nginx receives a signal to shut down cleanly.

Now that you understand how to use Docker, let’s check out how Podman works next.

How to Use Podman – The Daemonless Alternative

Now let's try Podman. It's designed to be a drop-in replacement for Docker, but with some key differences that make it interesting for specific use cases.

Why Podman Exists

Docker runs as a daemon (a background service) that requires root privileges. This daemon always runs, listening for commands. This architecture has some downsides:

1. Security: The Docker daemon runs as root. If someone compromises the daemon, they have root access to your entire system.

2. Resource Usage: The daemon consumes resources even when you're not running containers.

3. Single Point of Failure: If the daemon crashes, all your containers stop.

Podman solves these problems by not using a daemon at all. Each podman command runs independently. This is called a "daemonless" architecture.

Key Podman Features

To summarize, here are some key helpful features of Podman that might make it a good fit for your projects:

1. No daemon required: Each command runs independently. No background service needed.

2. Rootless by default: Containers run as your regular user, not as root. This dramatically improves security.

3. Drop-in Docker replacement: Most Docker commands work exactly the same. You can even alias docker=podman and many applications won't notice the difference.

4. Pod support: Podman has a concept of "pods" like Kubernetes. This is unique among container tools.

Now that you understand the benefits of Podman, let’s see how you can use it.

How to Install Podman

Podman installation varies by operating system. Here are the official guides:

• Podman for macOS

• Podman for Windows

• Podman for Linux

For macOS users (what we'll use in this tutorial), you can install Podman using Homebrew:

brew install podman

How to Initialize and Start Podman Machine

On macOS, Podman needs a Linux VM to run containers (since containers use Linux kernel features). Podman Machine handles this for you:

podman machine init

This creates a small Linux VM. You’ll only need to do this once. The VM is about 1GB and uses minimal resources when running.

Start the machine:

podman machine start

Verify it's working:

podman --version

You should see something like:

podman version 4.5.0

How to Run Containers with Podman

Here's where it gets interesting. You can use nearly identical commands to Docker. Let's build and run the same web server you created earlier:

# Build the image (same command as Docker)
podman build -t my-web-app .

# Run the container
podman run -d -p 8081:80 my-web-app

# See running container
podman ps

Notice that we used port 8081 this time so it doesn't conflict with the Docker container if it's still running. Visit http://localhost:8081 and you'll see the same page, but this time it's running in Podman!

If you experience issue when running the podman build command, you can delete the docker image using docker image rm my-web-app:latest.

What's Different Under the Hood?

Even though the commands look the same, what's happening is different: first no daemon was involved. The podman command directly created and started the container. And the container is running as your user, not as root.

You can verify this by checking what user owns the process:

podman top <container-id> user

You'll see your username, not root.

Podman Pods – A Unique Feature

Podman has a unique feature that Docker doesn't have: pods. A pod is a group of containers that share networking and storage. This is the same concept Kubernetes uses, which makes Podman excellent for local Kubernetes development.

Why Pods Matter:

In real applications, you often have multiple containers that need to work together. For example, a web application typically needs a database to store data, a cache layer for temporary storage of frequently accessed data and a logging container for request, response, and non-sensitive critical application metadata.

These four containers (web, database, cache, logger) need to communicate with each other. In Docker, you'd create a custom network and connect each container to it. In Podman, you can create a pod that automatically handles this networking.

How to Create a Podman Pod

podman pod create --name my-app-pod -p 8082:80

This creates a pod named my-app-pod and exposes port 8082 on your host to port 80 inside the pod. Notice that you don't expose ports on individual containers – you expose them on the pod.

Add a web server to the pod:

podman run -d --pod my-app-pod --name web nginx:alpine

The --pod flag tells Podman to run this container inside the pod. The container doesn't need its own port mapping because the pod handles that.

Add Redis (an in-memory database) to the pod:

podman run -d --pod my-app-pod --name cache redis:alpine

Now you have two containers running in the same pod. Here's the powerful part: they share the same network namespace.

To check your pod:

# List all pods
podman pod ps -a

# Show details for one pod
podman pod inspect <pod-name-or-id>

# Check processes running in the pod
podman top pod <pod-name-or-id>

# See logs from containers in that pod
podman logs <container-name-or-id>

Understanding Shared Networking:

Both containers can reach each other using localhost. The web container can connect to Redis using localhost:6379 (Redis's default port). It's as if they're running on the same machine.

This is exactly how Kubernetes pods work. If you learn Podman pods, you're learning Kubernetes networking too.

How to Generate Kubernetes YAML from Pods

Here's where Podman really shines. You can generate Kubernetes-compatible YAML from your pod:

podman generate kube my-app-pod > my-app-pod.yaml

Open my-app-pod.yaml and you'll see proper Kubernetes configuration:

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-5.7.1
apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.SandboxID/cache: 5e56bd9eab1a02a88654e3614312302d0f3f8d3652480498e6d1eef7d4824019
    io.kubernetes.cri-o.SandboxID/web: 5e56bd9eab1a02a88654e3614312302d0f3f8d3652480498e6d1eef7d4824019
  creationTimestamp: "2026-02-12T13:44:55Z"
  labels:
    app: my-app-pod
  name: my-app-pod
spec:
  containers:
  - args:
    - nginx
    - -g
    - daemon off;
    image: docker.io/library/nginx:alpine
    name: web
    ports:
    - containerPort: 80
      hostPort: 8082
  - args:
    - redis-server
    image: docker.io/library/redis:alpine
    name: cache

This file can be deployed directly to any Kubernetes cluster:

# using minikube cluster
kubectl apply -f my-app-pod.yaml

This is incredibly useful for local development. You can prototype your application using Podman pods, generate the YAML, and deploy to Kubernetes without rewriting anything.

How to Manage Podman Machines

When working with Podman on macOS or Windows, you're using a Linux VM. Here's how to manage it.

List all Podman machines:

podman machine list

This shows all your Podman VMs, their status (running or stopped), and their names. The default machine is usually called podman-machine-default.

Check machine status and info:

podman machine info

This displays detailed information about your current machine including CPU, memory, and disk usage.

Stop the Podman machine:

podman machine stop

If you have multiple machines, specify the name:

podman machine stop podman-machine-default

This stops the VM but preserves it. All your images and containers remain intact. When you stop the machine, all running containers inside it are stopped.

Start a stopped machine:

podman machine start

Or with a specific name:

podman machine start podman-machine-default

This restarts the VM. Your images are still there, but containers remain stopped unless you started them with a restart policy.

Delete a Podman machine:

podman machine rm podman-machine-default

This completely destroys the VM and all its contents (images, containers, volumes). Use this when you want to start fresh or free up disk space.

With this basic understanding of how Podman works, we can move on and learn about how to use Containerd.

How to Work with Containerd

Containerd is the runtime that Docker itself uses under the hood. It's also the default runtime for most Kubernetes installations. When you run Docker, you're actually using containerd without knowing it.

Why Use containerd Directly?

You might wonder why you'd use containerd directly if Docker already uses it. Here are a few reasons:

1. Kubernetes: Most Kubernetes clusters use containerd as their container runtime. Understanding it helps you troubleshoot production issues.

2. Minimal footprint: containerd has no UI and minimal features. It uses less memory than Docker Desktop (about 50MB vs 2GB).

3. Building tools: If you're building container orchestration tools, working directly with containerd gives you fine-grained control.

Understanding the Architecture

The containerd architecture looks like this:

Your Command → nerdctl → containerd → runc → Container

In this chain, nerdctl provides a Docker-like CLI, containerd manages images and container lifecycle, and runc actually creates the container using kernel features.

How to Install containerd with nerdctl

containerd is designed for systems (like Kubernetes) rather than direct developer use. The installation approach differs by operating system:

• Lima for macOS (includes nerdctl)

• containerd for Linux (native installation)

• nerdctl releases (for all platforms)

For macOS users (what we'll use in this tutorial), we’ll use Lima, which provides a Linux VM with containerd and nerdctl already installed.

brew install lima

Lima comes with nerdctl built-in, so you don't need to install it separately.

For Linux users, you can install containerd directly from your package manager and download nerdctl from the GitHub releases page. Containerd runs natively on Linux without needing a VM.

How to Start a Lima Instance

limactl start

This creates a default Linux VM running containerd with nerdctl available. The VM is configured with reasonable defaults (2GB RAM, 100GB disk). You can customize these settings if needed.

Lima mounts your home directory inside the VM, so you can access your files. This makes working with Lima feel transparent – you don't need to copy files into the VM.

Verify it's working:

lima nerdctl run hello-world

How to Run Your App with nerdctl

The commands are nearly identical to Docker. This is intentional – nerdctl aims for Docker compatibility. Since we're running through Lima, we’ll prefix commands with lima.

Navigate to your project directory:

cd ~/container-demo

Build the image:

lima nerdctl build -t my-web-app .

Run the container:

lima nerdctl run -d -p 8083:80 my-web-app

Visit http://localhost:8083 to see your app running on containerd!

What's Different from Docker?

Under the hood, a lot is different. Containerd is managing your image and container. There's no daemon in the traditional sense (containerd runs differently than dockerd). Images are stored differently (though they're OCI-compliant so they're compatible).

But from your perspective as a developer, the commands feel the same. This is the power of standards like OCI.

How to Check Running Containers:

lima nerdctl ps

This shows all running containers.

How to Manage Lima VMs

When working with containerd through Lima, you're using a Linux VM. Here's how to manage it.

List all Lima VMs:

limactl list

This shows all your Lima VMs, their status (running or stopped), and their names. The default VM is usually called default.

Check VM status and info:

limactl info default

This displays detailed information about the specified VM including its configuration and resource usage.

Stop the Lima VM:

limactl stop default

This stops the VM but preserves it. All your images and containers remain intact. When you stop the VM, all running containers inside it are stopped. The next time you start it, your images will still be there but containers remain stopped.

Start a stopped VM:

limactl start default

This restarts the VM. Your images persist across restarts, so you don't need to rebuild them.

Delete a Lima VM:

limactl delete default

This completely destroys the VM and all its contents (images, containers, volumes). Use this when you want to start fresh or free up disk space. You'll need to run limactl start again to create a new VM.

Create a new VM with custom settings:

limactl start --name my-custom-vm --cpus 4 --memory 8

This creates a new VM with 4 CPUs and 8GB of memory. You can have multiple Lima VMs for different projects.

How to Move Containers Between Runtimes

Thanks to the OCI (Open Container Initiative) standard, you can move container images between different runtimes. This is incredibly powerful – you can build with one tool and deploy with another.

Why Standards Matter

Before OCI, each container runtime used its own image format. Moving images between runtimes was difficult or impossible.

OCI created standards for the Runtime Specification (how to run a container), the Image Specification (how to package a container image), and the Distribution Specification (how to transfer images between systems).

Now all major runtimes follow these standards, making images portable.

Method 1 – Using Container Registries

The easiest way to share images is through a container registry like Docker Hub, GitHub Container Registry, or your own private registry. Any runtime can push and pull from registries.

First, build with Docker:

docker build -t my-username/my-app:v1 .

The image name has three parts: my-username (your registry username), my-app (the application name), and v1 (a version tag).

Push to Docker Hub:

docker login
docker push my-username/my-app:v1

You'll need to create a free Docker Hub account if you don't have one. The docker login command prompts for your credentials.

Now pull with Podman:

podman pull my-username/my-app:v1

Podman downloads the image from Docker Hub. Even though it was built with Docker, Podman can use it because both follow OCI standards.

Or pull with nerdctl:

lima nerdctl pull my-username/my-app:v1

Same image, three different runtimes. This is the power of standards.

Method 2 – Export and Import

If you don't want to use a public registry (maybe your image contains proprietary code), you can export images as tar files. This is perfect for air-gapped environments or simply moving images between machines.

Export from Docker:

docker save my-web-app -o my-web-app.tar

This creates a file called my-web-app.tar containing the image and all its layers. The file might be large (tens or hundreds of megabytes) depending on your image.

Import to Podman:

podman load -i my-web-app.tar

Import to nerdctl:

lima nerdctl load -i my-web-app.tar

Now you have the same image available in all three runtimes! You can verify:

docker images
podman images  
lima nerdctl images

All three commands will show my-web-app in their image lists.

Understanding Image Layers:

When you export an image, you're exporting all its layers. Each line in your Dockerfile creates a layer. These layers are shared between images, which saves disk space.

For example, if you have 10 images all based on nginx:alpine, they all share the nginx layers. Only the layers unique to each image take up additional space.

Real-World Use Cases

Let's look at some real scenarios where choosing the right runtime matters. These examples show how technical decisions have practical impacts.

Use Case 1 – Security-First Development

If you're working on security-sensitive applications (financial services, healthcare, government), Podman's rootless containers are a huge advantage.

The Security Problem:

Traditional Docker requires root privileges. If someone exploits a vulnerability in your container and escapes to the host system, they have root access. This is called a "container escape" vulnerability.

Podman's rootless mode solves this:

# All Podman commands run as your user by default
podman run --rm -it alpine whoami

This outputs your username, not root. The command uses --rm to remove the container when it exits (cleanup), -it to make it interactive with a terminal, alpine as a minimal Linux distribution, and whoami as a command that prints your username.

Even if someone breaks out of the container, they only have your user's permissions. They can't install system-wide malware, access other users' data, modify system configuration, or install kernel modules.

This dramatically reduces the impact of a container escape.

Example Security Scenario:

Imagine you're running a web application that processes user uploads. A vulnerability lets an attacker execute code in your container. With Docker running as root, they could escape the container, install a rootkit, steal all data from your server, and persist even after you patch the vulnerability.

With Podman rootless, they might escape the container but can only access files your user can access. They can't persist beyond the container and can't affect other users or system files.

The difference is dramatic.

Use Case 2 – Testing Kubernetes Locally

Podman can generate Kubernetes YAML from running containers. This is perfect for prototyping before you commit to a Kubernetes configuration.

The Development Workflow:

1. Run your application locally with Podman

2. Test and iterate quickly

3. Generate Kubernetes YAML when it works

4. Deploy to a real cluster

Here's a practical example. Let's say you're building a web application with a database:

Run your containers:

# Create a pod (like a Kubernetes pod)
podman pod create --name myapp -p 8080:80

# Add web server
podman run -d --pod myapp --name web nginx:alpine

# Add PostgreSQL
podman run -d --pod myapp --name db \
  -e POSTGRES_PASSWORD=secret \
  postgres:alpine

Test your application at http://localhost:8080. When it works, generate Kubernetes YAML:

podman generate kube myapp > myapp.yaml

Now you can deploy myapp.yaml to any Kubernetes cluster:

kubectl apply -f myapp.yaml

This is much faster than writing Kubernetes YAML by hand and debugging in a cluster. You iterate locally, then deploy when ready.

Why This Matters:

Kubernetes has a steep learning curve. The YAML configuration is verbose and error-prone. By starting with simple Podman commands and generating YAML, you can focus on your application first, learn Kubernetes gradually, catch configuration errors early, and iterate quickly without cloud costs.

Use Case 3 – Resource-Constrained Environments

containerd has the smallest footprint. If you're running containers on edge devices, Raspberry Pi, or resource-constrained servers, this matters a lot.

Comparing Memory Usage:

Here are typical memory footprints for each runtime:

• Docker Desktop uses approximately 2GB RAM (includes the VM, daemon, UI, and Kubernetes).

• Podman uses approximately 500MB RAM (includes the VM on macOS).

• Containerd uses approximately 50MB RAM (just the runtime, no extras).

On a developer laptop with 16GB RAM, this difference doesn't matter much. But consider these scenarios:

1. Edge Computing:

You're running containers on edge devices with 1GB RAM total. Docker Desktop won't fit. containerd leaves room for your application.

2. IoT Devices:

A Raspberry Pi with 2GB RAM running Docker Desktop leaves little room for your application. containerd uses minimal resources.

3. High-Density Servers:

Running 100 containers per server. Every MB counts. Using containerd instead of full Docker saves 2GB per server × 100 servers = 200GB.

Example Setup for Edge Device:

# On a Raspberry Pi or similar device
sudo apt-get install containerd
sudo apt-get install nerdctl

# Now you can run containers with minimal overhead
nerdctl run -d my-lightweight-app

Your application gets to use most of the available RAM instead of competing with a heavy runtime.

Quick Reference Guide

Here's a handy comparison of common commands across runtimes:

Conclusion

In this guide, we’ve explored three major container runtimes and learned how to use Docker, Podman, and containerd. The container ecosystem is much bigger than just Docker, and knowing alternatives gives you more options for security, performance, and specialized use cases.

Use Docker when you're learning or need the best documentation. Use Podman when you need rootless security or are building CI/CD pipelines. Use containerd when you need minimal resource usage or are deploying to Kubernetes clusters.

Thanks to OCI standards, your containers are portable. Build with Docker, test with Podman, deploy with containerd – it all works together! You're not locked into one vendor or tool.

As always, I hope you enjoyed this guide and learned something. If you want to stay connected or see more hands-on DevOps content, you can follow me on LinkedIn and DevOps Cloud Projects

Happy containerizing!

Each environment has its own quirks, missing libraries, or slightly different configurations. This is where many “works on my machine” problems begin.

Docker was created to solve this exact issue, and it has become a core skill for anyone building and deploying software today.

In this article, you’ll learn how to Dockerize a LogAnalyzer Agent project and prepare it for deployment.

We’ll first understand what Docker is and why it matters. Then we’ll walk through converting this FastAPI-based project into a Dockerized application. Finally, we’ll cover how to build and upload the Docker image so it can be deployed to a cloud platform like Sevalla.

You only need a basic understanding of Python for this project. If you want to learn Docker in detail, go through this detailed tutorial.

What We’ll Cover

• What is Docker?

• Why Docker Matters

• Understanding the Project

• Writing the Dockerfile

• Handling Environment Variables in Docker

• Building the Docker Image

• Testing the Container Locally

• Preparing the Image for Deployment

• Adding the Docker Image to Sevalla

• Final Thoughts

What is Docker?

Docker is a tool that packages your application together with everything it needs to run. This includes the operating system libraries, system dependencies, Python version, and Python packages. The result is called a Docker image. When this image runs, it becomes a container.

A container behaves the same way everywhere. If it runs on your laptop, it will run the same way on a cloud server. This consistency is the main reason Docker is so widely used.

For the LogAnalyzer Agent, this means that FastAPI, LangChain, and all Python dependencies will always be available, regardless of where the app is deployed.

Why Docker Matters

Without Docker, deployment usually involves manually installing dependencies on a server. This process is slow and error prone. A missing system package or a wrong Python version can break the app.

Docker removes this uncertainty. You define the environment once, using a Dockerfile, and reuse it everywhere. This makes onboarding new developers easier, simplifies CI pipelines, and reduces production bugs.

For AI-powered services like the LogAnalyzer Agent, Docker is even more important. These services often rely on specific library versions and environment variables, such as API keys. Docker ensures that these details are controlled and repeatable.

Understanding the Project

Before containerizing the application, it’s important to understand its structure. The LogAnalyzer Agent consists of a FastAPI backend that serves an HTML frontend and exposes an API endpoint for log analysis.

The backend depends on Python packages like FastAPI, LangChain, and the OpenAI client. It also relies on an environment variable for the OpenAI API key.

From Docker’s point of view, this is a typical Python web service. That makes it an ideal candidate for containerization.

At this stage, you should clone the project repository to your local machine. You can run the app using the command python app.py

Writing the Dockerfile

The Dockerfile is the recipe that tells Docker how to build your image. It starts with a base image, installs dependencies, copies your code, and defines how the application should start.

For this project, a lightweight Python image is a good choice. The Dockerfile might look like this:

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Each line has a purpose: the base image provides Python and the working directory keeps files organized.

Dependencies are installed before copying the full code to improve build caching. The expose instruction documents the port used by the app. The command starts the FastAPI server.

This file alone turns your project into something Docker understands.

Handling Environment Variables in Docker

The LogAnalyzer Agent relies on an OpenAI API key. This key should never be hardcoded into the image. Instead, Docker allows environment variables to be passed at runtime.

During local testing, you can still use a .env file. When running the container, you can pass the variable using Docker’s environment flags or your deployment platform’s settings.

This separation keeps secrets secure and allows the same image to be used in multiple environments.

Building the Docker Image

Once the Dockerfile is ready, building the image is straightforward. From the root of the project, you run a Docker build command:

docker build -t loganalyzer:latest .

Docker reads the Dockerfile, executes each step, and produces an image.

This image contains your FastAPI app, the HTML UI, and all dependencies. At this point, you can run it locally to verify that everything works exactly as before.

Running the container locally is an important validation step. If the app works inside Docker on your machine, it’s very likely to work in production as well.

Testing the Container Locally

After building the image, you can start a container and map its port to your local machine. When the container starts, Uvicorn runs inside it, just like it did outside Docker.

docker run -d -p 8000:8000 -e OPENAI_API_KEY=your_api_key_here loganalyzer:latest

You should be able to open a browser, upload a log file, and receive analysis results. If something fails, the container logs will usually point you to missing files or incorrect paths.

This feedback loop is fast and helps you fix issues before deployment.

Preparing the Image for Deployment

At this stage, the Docker image is ready to be uploaded to a container registry. A registry is a place where Docker images are stored and shared. Your deployment platform will later pull the image from this registry.

We’ll use DockerHub to push our image. Create an account and run docker login command to authenticate it with your terminal.

Now let’s tag and push your image to the repository:

docker tag loganalyzer:latest your-dockerhub-username/loganalyzer:latest
docker push your-dockerhub-username/loganalyzer:latest

Adding the Docker Image to Sevalla

The final step is to upload the Docker image for deployment.

You can choose any cloud provider, like AWS, DigitalOcean, or others, to run your application. I’ll be using Sevalla for this example.

Sevalla is a developer-friendly PaaS provider. It offers application hosting, database, object storage, and static site hosting for your projects.

Every platform will charge you for creating a cloud resource. Sevalla comes with a $20 credit for us to use, so we won’t incur any costs for this example.

Log in to Sevalla and click on Applications -> Create new application:

You can see the option to link your container repository. Use the default settings. Click “Create application”.

Now we have to add our OpenAI API key to the environment variables. Click on the “Environment variables” section once the application is created, and save the OPENAI_API_KEY value as an environment variable.

We’re now ready to deploy our application. Click on “Deployments” and click “Deploy now”. It will take 2–3 minutes for the deployment to complete.

Once done, click on “Visit app”. You will see the application served via a URL ending with sevalla.app.

Congrats! Your log analyser service is now Dockerized and live.

From this point on, deployment becomes simple. A new version of the app is just a new Docker image. You can push an image to the repository and Sevalla will pull it automatically.

Final Thoughts

Docker turns your application into a portable, predictable unit. For the LogAnalyzer Agent, this means the AI logic, the FastAPI server, and the frontend all move together as one artifact.

By cloning the project, adding a Dockerfile, and building an image, you convert a local prototype into a deployable service. Uploading that image to Sevalla completes the journey from code to production.

Once you’re comfortable with this workflow, you’ll find that Docker isn’t just a deployment tool. It becomes a core part of how you design, test, and ship applications with confidence.

Hope you enjoyed this article. Learn more about me by visiting my website.

But when you need to run it on the cloud, things get complicated. You need to think about servers, environments, dependencies, and deployment pipelines. That’s where containerization comes in.

Containers make your application portable and predictable. You can run the same code with the same setup anywhere, from your laptop to the cloud.

In this guide, we will walk through how to containerize a simple Node.js API and deploy it to the cloud. By the end, you will know how to set up Docker for your app, push it to a registry, and see your application running on the cloud.

Table of Contents

• Prerequisites

• What is Containerization?

• Setting Up a Node.js App

• Writing the Dockerfile

• Building and Testing the Container

• Preparing for Deployment

• Deploying to the Cloud

• Scaling Your App

• Updating Your App

• Benefits of sing Containers

• Conclusion

Prerequisites

Before we dive into containerizing and deploying your Node.js application, make sure you have the following set up on your system. These basics will help you follow along without running into errors.

Node.js and npm
You should have Node.js (v18 or higher) and npm installed on your local machine. This ensures you can run your app locally before containerizing it.
To check your versions, run:

node -v
npm -v

Docker installed and running
Docker is the core tool we’ll use to containerize the app. Install Docker Desktop or Docker Engine depending on your system. Once installed, confirm that it’s running and working by typing:

docker --version

Docker Hub account (or any container registry)
You’ll need a Docker Hub account to push your container image to the cloud. This allows your deployment platform to pull and run the image. You can create one for free at hub.docker.com.

Once you have these prerequisites ready, you’ll be set to build your first containerized Node.js app and deploy it to the cloud.

What is Containerization?

Containerization is a way to package an application along with everything it needs to run. That includes the code, libraries, system tools, and settings. The package is called a container image.

When you run that image, you get a container that behaves exactly the same on any system that supports Docker.

Without containers, deployment can be messy. Your app might work on your machine but fail in production due to missing libraries or version mismatches.

Containers solve this by locking in the environment. Think of them as lightweight virtual machines that only contain what your app needs.

Setting Up a Node.js App

Let’s start by building a simple Node.js API. We will keep it minimal so we can focus on the containerization and deployment steps.

Create a new folder and add a file called server.js:

const express = require('express');
const app = express();
const PORT = process.env.PORT || 3000;

app.get('/', (req, res) => {
  res.json({ message: 'Hello from Container!' });
});
app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

Next, create a package.json file with the following content:

{
  "name": "container-node-app",
  "version": "1.0.0",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^5.1.0"
  }
}

Run npm install to install the Express dependency. You now have a simple Node.js API that runs locally. You can test it with npm start and open http://localhost:3000 in your browser.

Writing the Dockerfile

To run this app in a container, we need to write a Dockerfile. This file defines how to build the container image. Create a new file called Dockerfile and add this:

FROM node:24

WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["npm", "start"]

Let’s break this down. We start with the official Node.js 24 image. We set a working directory inside the container. We copy the package files and install dependencies.

Then we copy the rest of the code. We expose port 3000 so that the app can accept traffic. Finally, we run npm start as the default command.

Building and Testing the Container

Now that we have the Dockerfile, we can build the image. Run the following command:

docker build -t container-node-app .

This builds an image named container-node-app. To test it locally, run:

docker run -p 3000:3000 container-node-app

Open http://localhost:3000 in your browser, and you should see the JSON message {"message":"Hello from Container!"}. At this point, we know our app works in a container.

Preparing for Deployment

To deploy on any cloud platform, you need to push your image to a container registry. A registry is a place where container images are stored and shared. Your cloud provider can pull images from Docker Hub or other registries.

Tag your image with a registry path. For Docker Hub, it looks like this:

docker tag container-node-app your-dockerhub-username/container-node-app:latest

Then log in and push it:

docker login
docker push your-dockerhub-username/container-node-app:latest

Your image should now be available in the cloud registry and ready for deployment.

Here’s mine:

Deploying to the Cloud

In this tutorial, I’ll be using Sevalla since it offers a free tier, so there are no costs involved to deploy this container to the cloud. You can use other providers like AWS or Heroku, but just note that you will incur costs for creating resources.

Sevalla is a modern, usage-based Platform-as-a-service provider. It offers application hosting, database, object storage, and static site hosting for your projects.

Once you have your account set up, you can create a new application and tell it which container image to use. Sevalla will pull the image from the registry, create a container, and handle the networking, scaling, and updates for you.

To get started, login to Sevalla. In the dashboard, choose to create a new application. Give it a name like node-api. Provide the registry path of your image.

Choose a location and use the “Hobby” plan. Sevalla comes with a $50 free credit, so you wont be charged for deploying this image.

Click “Create and Deploy”. Sevalla will handle the rest. You can watch it configure the application and run the deployment.

Once the deployment is complete, click on “Visit app” to get your app’s live URL. You can see the response from the API.

Scaling Your App

One of the main benefits of Sevalla is easy scaling. If you start getting more traffic, you can increase the number of containers running your app with just a few clicks. Sevalla will load balance traffic between them. This means your app can handle more requests without downtime.

Scaling with containers is efficient because each container runs the exact same code. There is no need to configure extra servers manually. Sevalla takes care of orchestration, so your focus stays on writing code instead of managing infrastructure.

Updating Your App

When you make changes to your Node.js app, updating is straightforward. You rebuild the Docker image, push it to the registry, and tell Sevalla to redeploy.

Since containers are immutable, every new build creates a fresh environment. This ensures your updates are clean, consistent, and free of old dependencies.

For example, if you change the message in server.js and want to deploy it, you would run:

docker build -t your-dockerhub-username/container-node-app:latest .
docker push your-dockerhub-username/container-node-app:latest

Then trigger a redeploy in the Sevalla dashboard. Within minutes, your users will see the updated response.

Benefits of sing Containers

Containers bring many advantages when deploying Node.js applications. They make your app portable because the container holds both the code and its dependencies, ensuring it runs the same way everywhere.

They improve consistency, since every build creates an isolated environment without leftover files or mismatched versions. Scaling becomes simple because you can spin up more containers as traffic grows, and each one behaves identically. Updates are cleaner too, as you replace old containers with fresh ones built from the latest code.

For developers, this means fewer surprises and less time fixing environment issues. Containers provide a reliable foundation, so you can focus on building features rather than troubleshooting deployments.

Conclusion

Containerization is one of the most important shifts in modern software development. By learning how to put your Node.js app into a Docker container, you unlock the ability to run it anywhere.

In this guide, we built a small Node.js API, created a Dockerfile, tested the container locally, pushed it to a registry, and deployed it to the cloud. The steps you followed here apply to much larger and more complex applications as well. Once you get the basics, you can scale up your workflows to production-level projects.

Hope you enjoyed this article. Connect with me on Linkedin or visit my website.

What if you had a flight recorder for your applications, something that captures every system call in real-time, so you can "rewind" and see the exact sequence of events that led to a crash? That's what Traceloop does. It continuously traces system calls in your pods, giving you a detailed replay of what happened before, during, and after issues occur.

In this guide, you’ll learn how to use Traceloop's system call tracing to debug pod issues that would otherwise be nearly impossible to diagnose.

Prerequisites

Before we begin, here are some prerequisites – things you’ll need to know and have:

• Basic Kubernetes concepts: Understanding of pods, deployments, services, and namespaces

• kubectl fundamentals: Comfortable with commands like kubectl get, kubectl describe, kubectl logs, and kubectl exec

• Container basics: Understanding how containerized applications work

• Basic Linux concepts: Understanding of processes and system calls (helpful, but we'll explain as we go)

Technical Requirements

• Kubernetes cluster access: Local (minikube, kind, Docker Desktop) or cloud-based cluster

• kubectl installed and configured to connect to your cluster

• Sufficient permissions (cluster admin or equivalent RBAC) to:

  • Install and run eBPF-based tools (Traceloop uses eBPF)

  • Create/modify pods and deployments

  • Access pod logs and system-level data

• Linux-based Kubernetes nodes: Most clusters already run on Linux.

System Requirements

• Extended Berkeley Packet Filter (eBPF) support: Used for tracing and monitoring at the kernel level. Kernel version 5.10+ recommended.

• Sufficient cluster resources: Traceloop runs alongside your applications

Table of Contents

1. What is Traceloop?

2. How Traceloop Works

3. How to Set Up Traceloop

4. Your First Trace: Hands-On Tutorial

5. Step-by-Step Debugging Walkthrough

6. Real-World Debugging Scenarios

7. Best Practices

8. Conclusion

What is Traceloop?

Traceloop is a system call tracing and observability tool that works across containerized environments, from Docker containers running locally to pods in production Kubernetes clusters. But before we discuss what that means, let's talk about why system calls matter for debugging.

Every time your application does anything (like opening a file, making a network request, allocating memory, or crashing), it has to interact with the operating system through system calls. These are the fundamental building blocks of how any program interacts with the world around it.

Here's where traditional debugging falls short: when your container crashes, the logs might tell you "segmentation fault" or "out of memory," but they don't tell you the sequence of events that led there. Did the application try to access a file that didn't exist? Was it making network calls that failed? Did it run out of file descriptors?

Traceloop captures this missing piece. It sits at the kernel level using eBPF technology, recording every system call your application makes in real-time. Think of it as installing a dashcam in your application. It's always recording with minimal resources, and when something goes wrong, you have the footage.

Strace is another popular debugging tool – but it requires you to know that there's a problem first. With Traceloop, we can conveniently run it continuously in the background with minimal overhead. If your container crashes at 3am, you can immediately "rewind the tape" and see exactly what system calls happened leading up to the crash.

This helps debug intermittent issues that happen randomly in production but never when you are watching. Because Traceloop is always recording, you finally have visibility into what your application was doing when these mysterious failures occur.

How Traceloop Works

Now that you understand what Traceloop does, let's look under the hood at how it captures and processes system calls in your containerized environments.

The Technical Foundation

Traceloop is built on eBPF, a technology that allows programs to run safely in the Linux kernel without changing kernel code. Think of eBPF as a way to install "hooks" directly into the kernel that can observe everything happening on your system with minimal performance impact.

Unlike traditional monitoring tools that work from userspace, eBPF programs run in kernel space, giving them access to system calls as they happen, without relying on the application logging appropriate error messages. This is why Traceloop can capture events that never make it to application logs, like failed system calls or crashes that happen before the application can write anything.

The Flight Recorder Architecture

Traceloop uses eBPF maps as an overwriteable ring buffer. Imagine a tape recorder that continuously records over itself. It's always capturing system calls, but it only keeps the most recent data in memory. When something goes wrong, the recording automatically preserves what happened leading up to the incident, just like an airplane's flight recorder after a crash.

This approach solves the production debugging problem: you don't need to predict when issues will happen or attach debuggers after the fact. The recording is always running, waiting for you to need it.

System Call Capture Flow

Here's how Traceloop captures and processes system calls across your Kubernetes environment:

1. Application pods generate system calls through normal operation – opening files, making network connections, allocating memory.

2. eBPF probes (also called hooks) intercept these system calls at the kernel level before they're processed.

3. Traceloop recorder captures the events, buffers them, and adds container context using Inspektor Gadget enrichment (pod name, namespace, container ID).

4. Output stream formats the data and makes it available for analysis in real-time or after an incident.

5. Traceloop user views and analyzes the captured trace to diagnose the root cause of issues.

Below is a visual representation of the flow. The key advantage is that Traceloop sees everything your application does, even actions that fail silently or happen too quickly for traditional logging to catch. This gives you complete visibility into your application's interaction with the operating system.

Container Isolation and Context

One of Traceloop's strengths is understanding containerized environments. It doesn't just capture raw system calls – it adds context about which pod, container, and namespace generated each call. This means you can trace specific applications without getting overwhelmed by system calls from other containers running on the same node.

This container awareness makes Traceloop particularly powerful in Kubernetes environments where you might have dozens of pods running on a single node, but you only care about debugging one specific application.

How to Set Up Traceloop

Before we can start tracing system calls, we need to set up Traceloop in your Kubernetes environment. Traceloop is part of the Inspektor Gadget ecosystem, which provides flexibility in how you use it.

Installation Overview

This setup:

• Deploys Inspektor Gadget components to all worker nodes

• Eliminates the download and initialization overhead on each use, as components are pre-loaded and ready

• Eliminates the need to reinstall or reconfigure for each debugging session – just run your traces immediately

• Requires cluster admin permissions

• Works best for teams doing regular debugging

Installation Requirements

First, ensure your cluster meets the requirements:

• Kubernetes cluster with Linux nodes

• eBPF support

• kubectl installed and configured

• Cluster admin permissions

Install kubectl gadget

The recommended way is using krew (kubectl plugin manager):

# Install krew if you don't have it
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz"
tar zxvf krew-linux_amd64.tar.gz
./krew-linux_amd64 install krew
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

# Install kubectl gadget
kubectl krew install gadget

Alternatively, you can install directly:

# For Linux/macOS
curl -sL https://github.com/inspektor-gadget/inspektor-gadget/releases/latest/download/kubectl-gadget-linux-amd64.tar.gz | sudo tar -C /usr/local/bin -xzf - kubectl-gadget

# Verify installation
kubectl gadget version

Deploy Inspektor Gadget to Your Cluster

Deploy the Inspektor Gadget components to your cluster:

kubectl gadget deploy

This installs the necessary DaemonSets and RBAC configurations that allow gadgets like Traceloop to run on your cluster nodes.

Alternatively, you can also deploy using Helm.

Verify Installation

Check that the gadget pods are running:

kubectl get pods -n gadget

You should see gadget pods running on each node in your cluster.

Your First Trace: Hands-On Tutorial

Now let's capture our first system call trace. We'll create a simple scenario and watch what happens at the system level.

Setting Up the Test Environment

First, create a dedicated namespace for our tracing experiments:

kubectl create ns test-traceloop-ns

Expected output:

namespace/test-traceloop-ns created

Next, create a simple pod that we can interact with:

kubectl run -n test-traceloop-ns --image busybox test-traceloop-pod --command -- sleep inf

Expected output:

pod/test-traceloop-pod created

This creates a BusyBox container that sleeps indefinitely, giving us a stable target for tracing.

Starting Your First Trace

Next, start tracing system calls for our test pod:

kubectl gadget run traceloop:latest --namespace test-traceloop-ns

This command starts the flight recorder. You'll see column headers showing what information Traceloop captures:

K8S.NODE    K8S.NAMESPACE    K8S.PODNAME    K8S.CONTAINERNAME    CPU    PID    COMM    SYSCALL    PARAMETERS    RET

The trace is now running in the background, continuously recording system calls from our pod.

Generating System Calls

With the trace running, let's generate some activity. In a new terminal window, run a command inside your test pod:

kubectl exec -ti -n test-traceloop-ns test-traceloop-pod -- /bin/sh

Once inside the container, run some basic commands:

ls /
echo "Hello World" > /tmp/test.txt
cat /tmp/test.txt

Collecting the Trace

Back in your original terminal where Traceloop is running, press Ctrl+C to stop the recording and see the captured system calls.

You'll see output similar to this:

K8S.NODE            K8S.NAMESPACE        K8S.PODNAME          K8S.CONTAINERNAME    CPU  PID    COMM  SYSCALL      PARAMETERS                   RET
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    openat       dfd=-100, filename="/lib"    3
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    getdents64   fd=3, dirent=0x...          201
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    write        fd=1, buf="bin dev etc..."   201
minikube-docker     test-traceloop-ns    test-traceloop-pod   test-traceloop-pod   2    95419  ls    exit_group   error_code=0                 0

Understanding Your First Trace

Let's break down what we're seeing:

• K8S.PODNAME: Which pod generated these system calls

• PID: Process ID of the command that ran

• COMM: The command name (ls, echo, cat)

• SYSCALL: The actual system call made (openat, write, exit_group)

• PARAMETERS: Arguments passed to the system call

• RET: Return value (0 usually means success)

This trace shows the ls command opening the /lib directory, reading directory entries, writing the output to stdout, and exiting successfully.

Clean Up

Remove the test resources:

kubectl delete pod test-traceloop-pod -n test-traceloop-ns
kubectl delete ns test-traceloop-ns

You can now see exactly what your applications are doing at the kernel level, something that traditional logs and kubectl commands can't show you.

Let's try this with an application that crashes.

Step-by-Step Debugging Walkthrough

Now that you know how to capture traces, let's take a look at a real debugging scenario. We'll create an application that crashes and use Traceloop to uncover the root cause. Something that would be nearly impossible with traditional kubectl debugging.

The Scenario: A Mysterious Crash

Let's create a Python application that has a subtle bug. It tries to write to a file it doesn't have permission to access, then crashes. This mimics real-world scenarios where applications fail due to permission issues, missing files, or resource constraints.

Setting Up the Problematic Application

First, we’ll create a new namespace for our debugging exercise:

kubectl create ns debug-traceloop-ns

Now, let's create a pod with an application that will crash:

kubectl run -n debug-traceloop-ns crash-app --image=python:3.9-slim --restart=Never -- python3 -c "
import time
import os
print('App starting...')
time.sleep(5)
print('Trying to write to restricted file...')
try:
    with open('/etc/passwd', 'w') as f:
        f.write('malicious content')
except Exception as e:
    print(f'Error: {e}')
    exit(1)
"

This creates a pod that will:

1. Start successfully

2. Try to write to /etc/passwd (a restricted system file)

3. Fail and crash with exit code 1

Starting the Trace Before the Crash

Here's the key difference from traditional debugging. We start tracing before we know there's a problem. In a real scenario, you'd have Traceloop running continuously.

kubectl gadget run traceloop:latest --namespace debug-traceloop-ns

The trace starts recording immediately. You'll see the column headers, and the flight recorder is now capturing every system call.

Observing the Application Behavior

In another terminal, check the pod status:

kubectl get pods -n debug-traceloop-ns -w

You'll see the pod go through these states:

• Pending → Running → Error → CrashLoopBackOff

Traditional debugging would show you:

kubectl logs -n debug-traceloop-ns crash-app

Output:

App starting...
Trying to write to restricted file...
Error: [Errno 13] Permission denied: '/etc/passwd'

But this doesn't tell you exactly what the application tried to do at the system level.

Collecting and Analyzing the Trace

Back in your Traceloop terminal, press Ctrl+C to stop the recording. You'll see system calls like this:

K8S.NODE        K8S.NAMESPACE      K8S.PODNAME  COMM    SYSCALL    PARAMETERS                           RET
minikube-docker debug-traceloop-ns crash-app    python3 openat     dfd=-100, filename="/etc/passwd"    -13
minikube-docker debug-traceloop-ns crash-app    python3 write      fd=3, buf="App starting..."         16
minikube-docker debug-traceloop-ns crash-app    python3 openat     dfd=-100, filename="/etc/passwd"    -13
minikube-docker debug-traceloop-ns crash-app    python3 exit_group error_code=1                        0

Reading the System Call Story

The trace reveals the exact sequence of events:

1. openat filename="/etc/passwd" RET=-13: The application tried to open /etc/passwd for writing

   • Return code -13 = EACCES (Permission denied)

2. write buf="App starting...": Normal logging output (successful)

3. openat filename="/etc/passwd" RET=-13: Second attempt to open the restricted file (still denied)

4. exit_group error_code=1: Application exits with error code 1

What Traceloop Revealed

Traditional debugging told us "Permission denied" but Traceloop shows us:

• Exactly which file the application tried to access

• When the permission denial happened in the execution flow

• How many times it tried (twice in this case)

• The exact system call that failed (openat)

Real-World Applications

This same approach works for debugging:

• File not found errors: See exactly which files your app is looking for

• Network connection failures: Observe failed connect() system calls with specific addresses

• Memory issues: Watch mmap() and brk() calls that fail

• Container startup problems: See which system calls fail during initialization

Clean Up

Remove the test resources:

kubectl delete pod crash-app -n debug-traceloop-ns
kubectl delete ns debug-traceloop-ns

Key Takeaway

Traditional Kubernetes debugging shows you what went wrong after it happened. Traceloop's continuous recording shows you exactly how it went wrong at the system level. This level of detail is invaluable for debugging complex production issues where the logs don't tell the full story.

Real-World Debugging Scenarios

Now that you understand the fundamentals, let's explore common production issues and how Traceloop helps diagnose them. These scenarios mirror real problems you'll encounter in Kubernetes environments.

Scenario 1: Container Startup Failures

The problem: Your pod gets stuck in CrashLoopBackOff with unhelpful logs.

Traditional kubectl commands show limited information:

kubectl describe pod failing-app
# Events: Back-off restarting failed container

kubectl logs failing-app
# (Empty or minimal output)

System calls show the application tried to:

1. Access configuration files that don't exist

2. Connect to services that aren't available

3. Write to directories without proper permissions

Key system calls to watch:

1. openat with -2 return (file not found)

2. connect with -111 return (connection refused)

3. access with -13 return (permission denied)

Scenario 2: Memory and Resource Issues

The problem: Application performance degrades or gets OOMKilled.

What Traceloop shows:

1. mmap calls failing (memory allocation issues)

2. brk system calls indicating heap growth

3. File descriptor exhaustion through failed openat calls

4. Excessive write calls indicating memory pressure

Example pattern:

SYSCALL    PARAMETERS           RET
mmap       length=1048576       -12  # ENOMEM - out of memory
brk        brk=0x55555557d000   0    # Heap expansion
openat     filename="/tmp/..."   -24  # EMFILE - too many open files

Scenario 3: Network Connectivity Problems

The problem: Service-to-service communication fails intermittently.

Traditional debugging limitations:

1. Application logs show "connection timeout"

2. Network policies seem correct

3. DNS resolution appears to work

What Traceloop reveals:

1. Exact IP addresses and ports being attempted

2. DNS resolution patterns through openat on /etc/resolv.conf

3. Failed connect calls with specific error codes

4. Socket creation and binding issues

Key indicators:

SYSCALL    PARAMETERS                    RET
socket     family=AF_INET, type=SOCK     3
connect    fd=3, addr=10.96.0.1:443     -110  # ETIMEDOUT
close      fd=3                         0

Scenario 4: Configuration and Secret Issues

The problem: Application can't access mounted secrets or config maps.

What system calls reveal:

1. File access patterns for mounted volumes

2. Permission checks on secret files

3. Configuration file parsing attempts

Common patterns:

1. Multiple openat attempts on different config file paths

2. access calls checking file permissions before opening

3. Failed reads from mounted secret volumes

Scenario 5: Performance Bottlenecks

The problem: Application response times are slow without obvious cause.

Traceloop analysis:

1. Excessive fsync calls (disk I/O bottlenecks)

2. Many futex calls (lock contention)

3. Frequent recvfrom timeouts (network issues)

4. Repeated file system operations

Performance indicators:

SYSCALL     FREQUENCY    ISSUE
fsync       High         Disk I/O bottleneck
futex       Excessive    Lock contention
poll        Many         Waiting for I/O
recvfrom    Timeouts     Network delays

Best Practices

When to Use Traceloop

Traceloop is most useful when you’re dealing with the kinds of problems that are notoriously difficult to pin down. If you’ve ever struggled with debugging intermittent crashes that don’t happen on demand, or run into confusing permission and access issues, this is where it works best.

It also helps uncover performance bottlenecks at the system level and provides visibility into application behavior during tricky startup failures. Another common use case is diagnosing network connectivity problems between pods, where other tools usually can't help

Of course, not every problem requires system call tracing. For application-level issues, logs and APM tools are more effective. Cluster-level concerns are often better handled with kubectl describe or by looking at events, and if you’re primarily monitoring resources, standard metrics and dashboards show you what's happening.

Performance Considerations

Like any tracing tool, Traceloop adds some overhead, but it keeps the overhead low. You can keep it efficient by narrowing the scope of your traces. For example, filtering by namespace with --namespace specific-ns, or targeting specific pods using --podname target-pod. In high-traffic environments, it’s best to run traces for shorter periods, and node-specific tracing can further isolate debugging when you don’t want to instrument the entire cluster.

In most cases, Traceloop uses very little CPU and memory, thanks to its eBPF-based approach. This makes it lighter than traditional tools like strace. The actual cost depends on the volume of system calls being recorded, so it’s a good practice to monitor resource usage in your own environment to confirm it’s operating within acceptable limits.

Integration with Your Workflow

Traceloop works well in dev and production workflows. In development, it’s a powerful way to understand how your application interacts with the system. You can use it to confirm that your app handles edge cases correctly, or to validate permission and resource configurations before promoting workloads into production.

In production environments, you can deploy it in different ways. Depending on how much overhead you're okay with, some teams run it continuously on a small subset of nodes, while others use it only when traditional debugging methods don’t provide enough insight. Pairing Traceloop with your existing monitoring and logging stack can give you a much more complete picture of system behavior.

It also helps with teamwork. Sharing trace outputs makes it easier for teams to reason about complex issues together. The data it provides can guide improvements in error handling and logging, and documenting common system call patterns can help onboard new developers more quickly.

Security Considerations

Because Traceloop records low-level system activity, you need to be mindful of what it captures.

What Traceloop Can See:

• System call parameters (such as filenames and network addresses)

• Process information and command arguments

• File access patterns and permissions

Privacy Measures:

• Limit trace duration to minimize data collection

• Use namespace isolation to avoid capturing unrelated workloads

• Apply data retention policies for trace outputs

• Watch for sensitive information in file paths or system call parameters

Conclusion

Traceloop doesn’t just tell you something went wrong – it shows you how. By recording every system call in real time, it turns mysterious Kubernetes failures into solvable problems. Whether the issue happened seconds ago or in the middle of the night, the tool gives you the ability to rewind, inspect, and respond with confidence.

When to Use It

Keep in mind that Traceloop complements your existing debugging toolkit rather than replacing it. Reach for it when logs don’t tell the whole story, when intermittent problems are hiding in the shadows, when kubectl commands leave you guessing, or when you need to see how your application is really interacting with the system.

Once you’re comfortable with Traceloop, you can add more tools. Inspektor Gadget offers other tools for network, security, and performance debugging that pair well with Traceloop. Integrating it into your incident response workflow, sharing insights across your team, and even considering continuous tracing for critical workloads are good things to try next.

The next time you run into a stubborn Kubernetes pod failure, you won’t be stuck speculating. With Traceloop, you can “rewind the tape” and see exactly what happened. System call tracing may sound complex at first, but in practice, it’s one of the most powerful ways to truly understand how applications behave in containerized environments.

PS: Have any questions about Traceloop or want to share your debugging challenges? The Inspektor Gadget team and community hang out in the #inspektor-gadget channel on Kubernetes Slack. It's a great place to get help from the engineers who built these tools, share experiences, and maybe even contribute to making the ecosystem even better.

You can also connect with me on LinkedIn if you’d like to stay in touch. If you made it to the end of this tutorial, thanks for reading!

So what is Kubernetes, really? Why is it everywhere? And should you care?

In this handbook, we’ll unpack Kubernetes in a way that actually makes sense. No buzzwords. No overwhelming tech-speak. Just straight talk. You’ll learn what Kubernetes is, how it came about, and why it became such a big deal – especially for teams building and running huge apps with millions of users.

We’ll rewind a bit to see how things were done before Kubernetes showed up (spoiler: it wasn’t pretty), and walk through the real problems it was designed to solve.

By the end, you’ll not only understand the purpose of Kubernetes, but you’ll also know how to deploy a simple app on a Kubernetes cluster – even if you’re just getting started.

Yep, by the time we’re done, you’ll go from “I keep hearing about Kubernetes” to “Hey, I kinda get it now!” 😄

📚 Table of Contents

1. What is Kubernetes?

2. How Applications Were Deployed Before Kubernetes

3. The Problem Kubernetes Solves 🧠

4. How Kubernetes Works – Components of a Kubernetes Environment 🧑‍🔧

5. Kubernetes Workloads 🛠️ – Pods, Deployments, Services, & More

6. How to Create a Kubernetes Cluster in a Demo Environment with play-with-k8s

   • Sign in to Play with Kubernetes

   • Create Your Kubernetes Cluster

7. How to Deploy an Application on Your Kubernetes Cluster

8. ✅ Advantages of Using Kubernetes in Business

9. 😬 Disadvantages of Using Kubernetes

10. Use Cases: When (and When Not) to Use Kubernetes

11. Conclusion

12. Study Further 📚

13. About the Author 👨‍💻

What is Kubernetes?

Imagine you're building a huge software platform, like a banking app. This app needs many features, like user onboarding, depositing money, withdrawals, payments, and so on. These features are so big and complex that it’s easier to split them into separate applications. These individual applications are called microservices.

So what are Microservices? Think of them like little building blocks that work together to create a bigger platform. So, you might have:

• One microservice for user onboarding

• Another for processing deposits

• Another for handling payments

• And many, many more!

To the user, it still looks like they’re using one smooth, unified banking app. But behind the scenes, it’s like a bunch of little apps working together to make everything run.

But here’s where things get tricky...

When you have dozens (or even hundreds) of these microservices, managing them becomes a nightmare. You might need to:

• Deploy each one separately

• Monitor them individually (to ensure they don’t crash/become slow due to too much load)

• Scale them (make them bigger to handle more users) as traffic surges, one by one

So, if your banking app suddenly gets millions of users, you'd have to manually tweak and update each microservice to keep it running smoothly. 😖 It’s a lot of work, and if something goes wrong, you’re in deep trouble.

This is where Kubernetes comes to the rescue! 🚀

Kubernetes is like a super-efficient manager for all these microservices. It’s a platform that helps you:

• Automate the deployment (getting the apps up and running)

• Scale the microservices (making them bigger or smaller as needed based on the inflow of traffic – your customers)

• Monitor them (keeping an eye on their health)

• Ensure reliability (so if one microservice breaks/fails, k8s replaces it immediately)

In simple terms, Kubernetes takes all your little microservices and organizes them, ensuring they run smoothly together, no matter how much traffic your app gets. It handles everything behind the scenes, like a conductor leading an orchestra, so your microservices work together without chaos.

How Applications Were Deployed Before Kubernetes

Before Kubernetes came into the picture, software teams had quite the juggling act when it came to deploying applications – especially when they were made up of lots of microservices.

One popular method was using a distributed system setup. Here’s what that looked like:

Imagine each microservice (like your user onboarding, payments, deposits, and so on) being installed on separate servers (physical computers or virtual machines). Each of these servers had to be carefully prepared:

• The microservice itself needed to be installed.

• The software dependencies it needed (like programming languages, libraries, tools) also had to be installed.

• Everything had to be configured manually ON EACH server.

And all of these servers had to talk to each other – sometimes over the public internet, or via private networks like VPNs.

Sounds like a lot of work, right? 😮 It was! Managing updates, fixing bugs, scaling up during traffic spikes, and keeping things from crashing could turn into a full-time headache for developers and system admins. 😖

Then Came Containers 🚢

A more modern solution that eased the pain (a little) was using containers.

So, what are containers?

Think of a container like a lunchbox for your microservice. Instead of installing the microservice and its supporting tools directly on a server, you pack everything it needs – code, settings, software libraries – into this single, neat container. Wherever the container goes, the microservice runs exactly the same way. No surprises!

Tools like Docker made this super easy. Once your microservice was packed into a container, you could deploy it on:

• A single server

• Multiple servers

• Or cloud platforms like AWS Elastic Beanstalk, Azure App Service, or Google Cloud Run.

The Problem Kubernetes Solves 🧠

At first, when containers arrived on the scene, it felt like developers had struck gold.

You could package a microservice into a neat little container and run it anywhere – no more installing the same software on every server again and again. Tools like Docker and Docker Compose made this smooth for small projects.

But the real world? That’s where it got messy.

The Growing Headache of Managing Containers 💡

When you have just a few microservices, you can manually deploy and manage their containers without much stress. But when your app grows – and you suddenly have dozens or even hundreds of microservices – managing them becomes an uphill battle:

• You had to deploy each container manually.

• You had to restart them if one crashed.

• You had to scale them one by one when more users started flooding in.

Docker and Docker Compose were great for a small playground or startups, but not for an enterprise application with high traffic inflow.

Cloud-Managed Services Helped... But Only Up To a Point 🧑‍💻

Cloud services like AWS Elastic Beanstalk, Azure App Service, and Google Code Engine offered a shortcut. They let you deploy containers without worrying about setting up servers.

You could:

• Deploy each container on its own managed cloud instance.

• Scale them automatically based on traffic.

BUT there were still some big headaches:

📦 Grouping microservices was awkward and expensive

Sure, you could organize containers by environment (like “testing” or “production”) or even by team (like “Finance” or “HR”). But each new microservice usually needed its own cloud instance – for example, a separate Azure App Service or Elastic Beanstalk environment FOR EVERY SINGLE CONTAINER.

Imagine this:

• Each App Service instance costs ~$50 per month.

• You’ve got 10 microservices.

• That’s $500/month... even if they’re barely used. 💸 Yikes!

Kubernetes: Smarter, Leaner, and More Flexible 💪

With Kubernetes, you don’t need to spin up a separate server for each microservice. You can start with just one or two servers (VMs) – and Kubernetes will automatically decide which container goes where based on available space and resources.

No stress, no waste! 💡

🧑‍🍳 Kubernetes Lets You Customize Everything

1. You can assign resources to each microservice container.
   👉 Example: If you have a "Payment" microservice that’s lightweight, you might give it 0.5 vCPUs and 512MB of memory. If you have a "Data Analytics" microservice that’s resource-hungry, you could give it 2 vCPUs and 4GB of memory.

2. You can set a minimum number of instances for each microservice.
   👉 Example: If you want at least 2 copies of your "Login" service always running (so your app doesn’t break if one fails), Kubernetes makes sure you always have 2 live copies at all times.

3. You can group your containers however you like:
   👉 By teams (Finance, HR, DevOps) or by environments (Testing, Staging, Production). Kubernetes makes this grouping super clean and logical.

4. You can automatically scale individual containers.
   👉 When more users flood your app, Kubernetes can create extra copies (called “replicas”) of only the containers that are under pressure. No more wasting resources on containers that don’t need it.

5. You can even scale your servers!
   👉 Kubernetes can automatically increase the number of servers (VMs) in your environment – called a Cluster – when traffic grows. So you could start with 2 VMs at $30 each ($60/month) and let Kubernetes add more servers only when necessary, rather than locking yourself into high fixed costs like $500/month for cloud-managed services.

Also, Kubernetes works the same way everywhere. Whether you deploy your containers on AWS, Google Cloud, Azure, or even your own laptop – Kubernetes doesn’t care. Your setup stays the same.

Compare that to managed services like Elastic Beanstalk or Azure App Service – which tie you to their platform, making it super hard to switch later.

✅ In short: Kubernetes saves you money, time, and a whole lot of headaches. It lets you run, scale, and organize your microservices without being chained to a single cloud provider — and without drowning in manual work.

How Kubernetes Works — Components of a Kubernetes Environment 🧑‍🔧

So by now you’ve seen the problem: running dozens (or hundreds!) of microservices manually is like juggling too many balls – you’re bound to drop some.

That’s why Kubernetes was created. But... how does it actually do all this magic? Let’s first break it down with the technical definition (simple but sharp – perfect for interviews) and then the layperson’s analogy (so it sticks in your head!).

1️⃣ Cluster 🏰

A Kubernetes Cluster is the entire setup of machines (physical or cloud-based) where Kubernetes runs. It’s made of one or more Master Nodes and Worker Nodes, working together to deploy and manage containerized applications.

Think of a Kubernetes Cluster as your entire playground. This is the environment where all your microservices live, grow, and play together.

A cluster is made up of two types of computers (called nodes):

• Master Node (nowadays often called the Control Plane)

• Worker Nodes

2️⃣ Master Node (Control Plane) 👑

The Master Node is like the brain of Kubernetes. It manages and coordinates the whole cluster – deciding which applications run where, monitoring health, and scaling things up or down as needed.

It’s like the boss of the entire cluster. It doesn’t run your applications directly. Instead, it:

• Watches over the worker nodes

• Decides which microservice (container) goes where

• Makes sure everything runs smoothly and fairly

Think of it like a factory manager who tells machines what to do, when to start, when to stop, and where to send the next package.

Inside the Master Node are a few clever mini-components that handle the real work.

3️⃣ API Server 💌

The API Server is the front door to Kubernetes. It handles communication between users and the system, taking commands and feeding them into the cluster.

This is where you (or your team) give Kubernetes instructions. Whether you're deploying a new app or scaling an existing one, you "talk" to the API Server first. It's like submitting a request at the front desk – the API server passes it on to the right people (or machines).

4️⃣ Scheduler 📅

The Scheduler assigns Pods (applications) to Worker Nodes based on available resources and needs.

Imagine you’ve asked Kubernetes to launch a new microservice. The Scheduler checks:

• Which worker node has enough space?

• Which node has enough memory and CPU?

• Where would this service run best?

It makes the decision and assigns the microservice to the perfect spot. Smart, huh?

5️⃣ Controller Manager 🎛️

The Controller Manager runs controllers that watch over the cluster and ensures that the system’s actual state matches the desired state.

This component watches over the system like a hawk. Let’s say you told Kubernetes:
"Hey, I want 3 copies of my payment microservice running at all times."

If one of them crashes, the Controller Manager sees that and spins up a new one to replace it automatically. It makes sure the reality always matches the plan.

6️⃣ etcd 📚

etcd is Kubernetes' memory – a distributed key-value store where cluster data is saved: config files, state, and metadata.

Imagine a notebook where all rules, records, and plans are written down. Without etcd, Kubernetes would forget everything.

7️⃣ Worker Nodes 💪

Worker Nodes are the servers that run the actual application containers, doing the heavy lifting in the cluster.

These are the machines where your microservices actually live and run. The Master Node gives orders, but the Worker Nodes do the heavy lifting – they run your containers!

Each worker node has a few helpers to manage its microservices:

• The Kubelet

• The Kube Proxy

8️⃣ Kubelet 📢

The Kubelet is the agent which lives on each Worker Node that makes sure containers are healthy and running as expected.

It listens to the Master Node’s instructions. If the Master Node says:"Hey, run this container!", the Kubelet makes it happen and keeps it running. If something goes wrong, the Kubelet reports back to the Master Node

9️⃣ Kube Proxy 🚦

Kube Proxy handles network traffic, ensuring that Pods can talk to each other and to the outside world.

Imagine your banking app’s login service needs to talk to the payments service. The Kube Proxy handles the routing so the request reaches the right place. It also handles load balancing, so no single microservice gets overwhelmed.

So, to summarize:

• The Master Node is the boss – it plans, watches, and assigns tasks.

• The Worker Nodes do the actual work – running your microservices.

• Components like etcd, Kubelet, Scheduler, Controller Manager, and Kube Proxy all work together like parts of a well-oiled machine.

Kubernetes is designed to handle your microservices automatically – keeping them alive, scaling them up, moving them around, and restarting them if they crash – so you don’t have to babysit them yourself.

Kubernetes Workloads 🛠️ — Pods, Deployments, Services, & More

Kubernetes workloads are the objects you use to manage and run your applications. Think of them as blueprints 📐 that tell Kubernetes what to run and how to run it – whether it’s a single app container, a group of containers, a database, or a batch job. Here are some of the workloads in Kubernetes:

1️⃣ Pods

A Pod is the smallest and simplest unit in the Kubernetes object model. It represents a single instance of a running process in your cluster and can contain one or more containers that share storage and network resources. ​

Think of a Pod as a wrapper around one or more containers that need to work together. They share the same network IP and storage, allowing them to communicate easily and share data. Pods are ephemeral (live for a short time, they can be replaced very easily). If a Pod dies, Kubernetes can create a new one to replace it almost instantly.​

Say you have an application which is split into 2 distributed monoliths – a frontend and a backend. The frontend will run in a container in Pod A, while the backend app will run in a container in another Pod B.

2️⃣ Deployments

A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate.

Deployments manage the lifecycle of your application Pods. They ensure that the specified number of Pods are running and can handle updates, rollbacks, and scaling. If a Pod fails, the Deployment automatically replaces it to maintain the desired state.​

Imagine you're managing a store. A Deployment is like the store manager – you tell it how many workers (Pods) you want, and it makes sure they’re always present. If one doesn't show up for work, the manager finds a replacement automatically. You can also tell it to hire more workers or fire some when needed.

3️⃣ Services

A Service in Kubernetes defines a way to access/communicate with Pods. Services enable communication between different Pods (for example, your frontend Pod A can communicate with your backend Pod B via a service) and can expose your application to external traffic (for example the public internet). ​

Services act as a stable endpoint to access a set of Pods. Even if the underlying Pods change, the Service's IP and DNS name remain constant, ensuring communication between the Pods within the cluster or with the internet.

A Service is like the front door to your app. No matter which worker (Pod) is behind it, people always use the same entrance to access it. It hides the messy stuff happening behind the scenes and gives users a simple way to connect to your app.

4️⃣ ReplicaSets

A ReplicaSet ensures that a specified number of identical Pods are running at any given time. It is often used to guarantee the availability of a specified number of Pods (horizontal scaling). ​

ReplicaSets maintain a stable set of running Pods. If a Pod crashes or is deleted, the ReplicaSet automatically creates a new one to replace it, ensuring your application remains available.​

Think of a ReplicaSet like a robot that counts how many copies of your app are running. If one goes missing, it automatically makes a new one. It keeps the number steady, just like you told it to.

5️⃣ DaemonSets

A DaemonSet ensures that all (or some) Nodes run an instance (a copy) of a specific Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are also removed. ​

DaemonSets are used to deploy a Pod on every node in the cluster. This is useful for running background tasks like log collection or monitoring agents on all nodes (for example to get the CPU, memory, and disk usage of each node).​

A DaemonSet is like saying, “I want this helper app to run on every single computer we have.” As mentioned earlier, it’s great for things like log collectors or security checkers – small helpers that every machine should have.

6️⃣ StatefulSets

A StatefulSet is the workload API object used to manage stateful applications (applications that store data, for example in their filesystem – databases). It manages the deployment and scaling of a set of Pods and provides guarantees about the ordering and uniqueness of these Pods.

StatefulSets are designed for applications that require persistent storage and stable network identities, like databases.

Let’s say you’re running a database or anything that needs to save info. A StatefulSet is like giving each app a name tag and a personal drawer to store their stuff. Even if you restart them, they come back with the same name and same drawer.

7️⃣ Jobs

A Job creates one or more Pods and ensures that a specified number of them successfully terminate. As Pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the Job is complete. ​

A Job is like a one-time task. Imagine sending out a batch of emails or processing a report. You want the task to run, finish, and then stop. That’s exactly what a Job does.

8️⃣ CronJobs

A CronJob creates Jobs on a time-based schedule. It runs a Job periodically on a given schedule, written in Cron format.

A CronJob is like setting a reminder or alarm. It tells your app (in this case the Job) to do something every night at 2 AM, every Monday morning, or once a month – whatever schedule you give it.

🛠️ How to Create a Kubernetes Cluster in a Demo Environment with play-with-k8s

As we've discussed earlier, a Kubernetes cluster is a set of machines (called nodes) that run containerized applications.

Setting up a Kubernetes cluster locally or in the cloud can be complex and expensive. To simplify the learning process, Docker provides a free, browser-based platform called Play with Kubernetes. This environment allows you to create and interact with a Kubernetes cluster without installing anything on your local machine. It's an excellent tool for beginners to get hands-on experience with Kubernetes.​

🔐 Sign in to Play with Kubernetes

1. Visit the platform at https://labs.play-with-k8s.com/.​

2. Authenticate:

   • Click on the "Login" button.

   • You can sign in using your Docker Hub or GitHub account.

   • If you don't have an account, you can create one for free on Docker Hub or GitHub.​

🚀 Create Your Kubernetes Cluster

Once signed in, follow these steps to set up your cluster:

Step 1: Start a New Session:

Click on the "Start" button to initiate a new session.​ This will create a new session giving you about 4 hours of play time, after which the cluster and it’s resources will be automatically terminated.

Step 2: Add Instances:

Then click on "+ Add New Instance" to create a new node (Virtual Machine).

This will open a terminal window where you can run commands.​

Step 3: Initialize the Master Node:

In the terminal, run the following command to initialize the master node:​

kubeadm init --apiserver-advertise-address $(hostname -i) --pod-network-cidr <SPECIFIED_IP_ADDRESS>

You can find the command in the terminal. In my case, the IP address is 10.5.0.0/16. Replace the <SPECIFIED_IP_ADDRESS> placeholder with the IP address specified in your terminal.

This process will set up the control plane of your Kubernetes cluster.​

Step 4: Add Worker Nodes:

If you want to add worker nodes, in the master node terminal, you'll find a kubeadm join... command after running the kubeadm init --apiserver-advertise-address $(hostname -i) --pod-network-cidr <SPECIFIED_IP_ADDRESS> command.

Click on "+ Add New Instance" to create another node just as you did earlier.

Run this command in the new node's terminal to join it to the cluster:

Step 5: Configure the Cluster’s networking:

Navigate to the master node, and run the command below to configure the cluster’s networking.

kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml

Step 6: Verify the Cluster:

In the master node terminal (the first node with the highlighted user profile), run:​

kubectl get nodes

You should see a list of nodes in your cluster, including the master and any worker nodes you've added.​

Congratulations! You just created your very own Kubernetes cluster with 2 VMs: the master node (where the control plane resides), and the worker nodes (where the Kubernetes workloads, for example Pods, will be deployed).

🚀 How to Deploy an Application on Your Kubernetes Cluster

Now that we've set up our Kubernetes cluster using Play with Kubernetes, it's time to deploy the application and make it accessible over the internet.

🧠 Understanding Imperative vs. Declarative Approaches in Kubernetes

Before we proceed, it's essential to grasp the two primary methods for managing resources in Kubernetes: Imperative and Declarative.

🖋️ Imperative Approach

In the imperative approach, you directly issue commands to the Kubernetes API to create or modify resources. Each command specifies the desired action, and Kubernetes executes it immediately.​

Imagine telling someone, "Turn on the light." You're giving a direct command, and the action happens right away. Similarly, with imperative commands, you instruct Kubernetes step-by-step on what to do.

Example:
To create a pod running an NGINX container, run the below command in the terminal of the master node:​

kubectl run nginx-pod --image=nginx

Now wait a few seconds and run the command below to check the status of the pod:

kubectl get pods

You should get a response similar to this

Now let’s expose our Pod to the internet by creating a Service. Run the command below to expose the Pod:

kubectl expose pod nginx-pod --type=NodePort --port=80

To get the IP address of the Cluster so we can access our Pod, run the command below:

kubectl get svc

The command displays the IP address from which we can access our service. You should get an output similar to this:

Now, copy the IP address for the nginx-pod service and run the command below to make a request to your Pod:

curl <YOUR-SERVICE-IP-ADDRESS>

Replace the <YOUR-SERVICE-IP-ADDRESS> placeholder with the IP address of your nginx-pod service. In my case, it’s 10.98.108.173.

You should get a response from your nginx-pod Pod:

We couldn’t access the Pod from the internet, that is our browser, because our Cluster isn’t connected to a cloud service like AWS or Google Cloud which can provide us with an external load balancer.

Now let’s try doing the same thing but using the Declarative method.

🚀 Declarative Approach

So far, we used the imperative approach, where we typed commands like kubectl run or kubectl expose directly into the terminal to make Kubernetes do something immediately.

But Kubernetes has another (and often better) way to do things: the declarative approach.

🧾 What Is the Declarative Approach?

Instead of giving Kubernetes instructions step-by-step like a chef in a kitchen, you give it a full recipe – a file that describes exactly what you want (for example, what app to run, how many copies of it, how to expose it, and so on).

This recipe is written in a file called a manifest.

📘 What’s a Manifest?

A manifest is a file (usually written in YAML format) that describes a Kubernetes object – like a Pod, a Deployment, or a Service.

It’s like writing down what you want, handing it over to Kubernetes, and saying: “Hey, please make sure this exists exactly how I described it.”

We’ll use two manifests:

1. One to deploy our application

2. Another to expose it to the internet

Let’s walk through it!

📁 Step 1: Clone the GitHub Repo

We already have a GitHub repo that contains the two manifest files we need. Let’s clone it into our Kubernetes environment.

Run this in the terminal (on your master node):

git clone https://github.com/onukwilip/simple-kubernetes-app

Now, let’s go into the folder:

cd simple-kubernetes-app

You should see two files:

• deployment.yaml

• service.yaml

📦 Step 2: Understanding the Deployment Manifest (deployment.yaml)

This manifest will tell Kubernetes to deploy our app and ensure it’s always running.

Here’s what’s inside:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

Now, let’s break this down:

• apiVersion: apps/v1: This tells Kubernetes which version of the API we’re using to define this object.

• kind: Deployment: This means we’re creating a Deployment (a controller that manages Pods).

• metadata.name: We’re giving our Deployment a name: nginx-deployment.

• spec.replicas: 3: We’re telling Kubernetes: “Please run 3 copies (replicas) of this app.”

• selector.matchLabels: Kubernetes will use this label to find which Pods this Deployment is managing.

• template.metadata.labels & spec.containers: This section describes the Pods that the Deployment should create – each Pod will run a container using the official nginx image.

✅ In plain terms: We're asking Kubernetes to create and maintain 3 copies of an app that runs NGINX, and automatically restart them if any fails.

🌐 Step 3: Understanding the Service Manifest (service.yaml)

This file tells Kubernetes to expose our NGINX app to the outside world using a Service.

Here’s the file – let’s break this down, too:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

• apiVersion: v1: We’re using version 1 of the Kubernetes API.

• kind: Service: We’re creating a Service object.

• metadata.name: nginx-service: Giving it a name.

• spec.type: NodePort: We’re exposing it through a port on the node (so we can access it via the node's IP address).

• selector.app: nginx: This tells Kubernetes to connect this Service to Pods with the label app: nginx.

• ports.port and targetPort: The Service will listen on port 80 and forward traffic to port 80 on the Pod.

✅ In plain terms: This file says, “Expose our NGINX app through the cluster’s network so we can access it from the outside world.”

🧹 Step 4: Clean Up Previous Resources

If you’re still running the Pod and Service we created using the imperative approach, let’s delete them to avoid conflicts:

kubectl delete pod nginx-pod
kubectl delete service nginx-pod

📥 Step 5: Apply the Manifests

Now let’s deploy the NGINX app and expose it – this time using the declarative way.

From inside the simple-kubernetes-app folder, run:

kubectl apply -f deployment.yaml

Then:

kubectl apply -f service.yaml

This will create the Deployment and the Service described in the files. 🎉

🔍 Step 6: Check That It’s Running

Let’s see if the Pods were created:

kubectl get pods

You should see 3 Pods running!

And let’s check the service:

kubectl get svc

Look for the nginx-service. You’ll see something like:

Note the NodePort (for example, 30001) as we’ll use it to access the app.

🌍 Step 7: Access the App

You can now send a request to your app like this:

curl http://<YOUR-NODE-IP>:<NODE-PORT>

Replace <YOUR-NODE-IP> with the IP of your master node (you’ll usually find this in Play With Kubernetes at the top of your terminal), and <NODE-PORT> with the NodePort shown in the kubectl get svc command.

You should see the HTML content of the NGINX welcome page printed out.

Now terminate the cluster environment by clicking the CLOSE SESSION button:

🆚 Why Declarative Is Better (In Most Cases)

• 🔁 Reusable: You can use the same files again and again.

• 📦 Version-controlled: You can push these files to GitHub and track changes over time.

• 🛠️ Fixes mistakes easily: Want to change 3 replicas to 5? Just update the file and re-apply!

• 🧠 Easier to maintain: Especially when you have many resources to manage.

💼 Advantages of Using Kubernetes in Business

Kubernetes isn’t just a developer tool—it’s a business enabler as well. It helps companies deliver products faster, more reliably, and with reduced operational overhead.

Let’s break down how Kubernetes translates to real-world business benefits:

1️⃣ Better Use of Cloud Resources = Cost Savings

Before Kubernetes, deploying many microservices for a single application often meant creating separate cloud resources (like one Azure App Service per microservice), which could rack up huge costs quickly. Imagine $50/month per service × 10 services = $500/month 😬.

With Kubernetes:
You can run multiple microservices on fewer virtual machines (VMs) while Kubernetes automatically decides the most efficient way to use the available servers. That means you pay for fewer servers and get more out of them 💸.

2️⃣ High Availability and Uptime = Happy Customers

Kubernetes watches your apps like a hawk 👀. If one of them crashes or fails, Kubernetes restarts or replaces it immediately – automatically.

For your business:
This means less downtime, fewer support tickets, and happier customers who don’t even notice when things go wrong in the background.

3️⃣ Easy Scaling During High Demand

Manually scaling apps during high traffic (like Black Friday) can be a nightmare 😰. And if you don't act fast, customers experience slowness or crashes.

With Kubernetes:
You can configure each microservice to automatically scale — meaning it adds more instances of that service only when needed (too many users on your site trying to purchase different products) and scales back down when traffic drops. This ensures your app is always responsive and you only pay for what you use.

4️⃣ Faster Deployment = Faster Time to Market

Kubernetes supports automation and repeatability. Teams can deploy new features or microservices faster without worrying about infrastructure setup every time.

For business:
This means faster product updates, quicker response to market demands, and competitive advantage 🚀.

5️⃣ Consistent Environments = Fewer Bugs

Each microservice in Kubernetes is containerized, meaning it runs with all its dependencies in a self-contained package. You can run the exact same app setup in:

• Development

• Testing

• Production

This reduces bugs caused by "it works on my machine" issues 🤦‍♂️ and helps teams build with confidence.

6️⃣ Vendor Independence (Bye-bye to Vendor lock-in)

When you use cloud-managed services (like AWS Elastic Beanstalk or Azure App Service), it’s often hard to move to another provider because everything is tailored to that specific platform.

With Kubernetes:
It works the same way on AWS, Azure, GCP, or even your own data center. This means you can switch cloud providers easily and avoid being locked into one vendor – aka cloud freedom! ☁️🕊️

7️⃣ Organizational Clarity

Kubernetes lets you organize your apps clearly. You can group workloads by:

• Team (for example, Finance, HR)

• Environment (for example, testing, staging, production)

This structure helps large teams collaborate better, stay organized, and manage resources efficiently.

😬 Disadvantages of Using Kubernetes

Like everything in tech, Kubernetes isn’t all rainbows and rockets 🚀. Just like any other tool, it has its pros and its cons. And it's super important for startup founders, product managers, or even CEOs to know when Kubernetes is the right fit – and when it’s just overkill.

Let’s break down the main disadvantages in a simple, honest way:

👨‍🔧 1. You’ll Likely Need a DevOps Engineer or Team

Kubernetes is powerful, yes. But that power comes with great responsibility 😅.

In simple terms:

• You don't just "click a button" and your app is magically running.

• Kubernetes needs someone who understands how to set it up, keep it running, and fix issues when they pop up. This person (or team) is usually called a DevOps Engineer, SIte Relability Engineer or Cloud Engineer.

Here’s what they’ll typically handle:

• Creating the cluster (the environment where your apps will run)

• Defining how your app containers should behave (how many should run, how much memory they need, when they should restart, and so on)

• Monitoring the apps and making sure they’re healthy

• Ensuring security rules are followed

• Handling automated scaling, deployment rollouts, backups, and so on.

💡 In short: You’ll need someone skilled to manage this tool. If you’re a solo founder or a small team with no DevOps experience, Kubernetes might be too much upfront.

💰 2. Kubernetes Can Be Expensive (If Used Prematurely)

Kubernetes saves money at scale – but can cost more if you adopt it too early or for the wrong use case.

Here's why:

• Kubernetes is meant for managing multiple applications or microservices. If your business only has one small app, you’re using a rocket to deliver a pizza 🍕 – it’s just not necessary.

• Kubernetes is also best when you have high or unpredictable traffic. It can automatically scale up your services when traffic spikes...but if your traffic is steady and small, you won’t benefit much from that power.

Let’s say:

• You have one app with moderate traffic.

• You deploy it on Kubernetes (which requires at least 1–2 VMs + setup).

• You hire a DevOps engineer to manage it.

• You pay for cloud compute + storage + monitoring.

You could end up spending $300–$800/month or more... for something that could’ve been hosted on a simple service like Render, Heroku, or a basic VM for a fraction of the cost.

So when should you consider Kubernetes?

• When your platform is made up of multiple services (For example, separate services for user auth, payments, analytics, notifications, and so on)

• When you’re expecting traffic spikes (for example, launching in new countries, going viral, seasonal demand like black Friday)

• When you want flexibility in managing your infrastructure across cloud providers (AWS, GCP, Azure) or even on-premises

🧭 Use Cases: When (and When Not) to Use Kubernetes

Kubernetes is an incredibly powerful tool – but it’s not always the right solution from day one.

Let’s break down when it makes sense to use Kubernetes and when it might be overkill 👇

✅ When You Should Use Kubernetes

Kubernetes becomes essential in these scenarios:

1. Your Application Is Made of Many Microservices

If your app is broken down into multiple microservices – like user authentication, payments, orders, notifications, and more – it’s a good sign that Kubernetes might eventually help.

Kubernetes can:

• Help manage each microservice independently

• Automatically scale each one based on demand

• Restart failed services automatically

• Make it easier to roll out updates to specific parts of the application

2. You’re Getting Steady and High Traffic

It’s not just about complexity – it’s about demand.

If your app receives a consistent, high volume of users (like hundreds or thousands every day), and you start seeing signs that your servers are getting overloaded, Kubernetes shines here. It can:

• Automatically increase resources when traffic surges

• Balance the load across multiple servers

• Prevent downtime due to traffic spikes

3. You Want Portability and Cloud Independence

If your business doesn’t want to be locked into just one cloud provider (for example, only AWS), Kubernetes gives you flexibility. You can move your application between AWS, GCP, Azure – or even to your own data center – with fewer changes.

4. Your DevOps Team Is Growing

When you have multiple developers or teams working on different parts of the app, Kubernetes helps:

• Organize and isolate workloads per team

• Improve collaboration and consistency

• Provide easy access control and monitoring

❌ When You Should Not Use Kubernetes

Let’s be honest: Kubernetes is not for everyone, especially not at the beginning.

1. You Just Launched Your App

In the early days of your product, when you’ve just launched and traffic is still low, Kubernetes is overkill. You don’t need its complexity (yet).

👉 Instead, deploy your app or each microservice on a simple virtual machine (VM). It’s cheaper and faster to get started.

2. You Don’t Need Auto-scaling (Yet)

If traffic to your app is still small and manageable, a single server (or a few of them) can easily handle the load. In that case, it’s better to:

• Deploy your microservices manually or with Docker Compose

• Monitor and scale manually when needed

• Keep things simple until the need for automation becomes obvious

3. You Don’t Have a DevOps Team

Kubernetes is powerful – but it needs expertise to set up and maintain. If you don’t have a DevOps engineer or someone who understands Kubernetes, it may cause more problems than it solves.

Hiring a DevOps team can be expensive, and setting up Kubernetes incorrectly can lead to outages, security risks, or wasted resources 💸

📈 When to Move to Kubernetes

So, what’s the best path forward?

Here’s a simple roadmap:

1. Start small: Deploy your app (or microservices) on one or a few VMs

2. Watch traffic: As user demand grows, increase VM size or replicate the app manually

3. Track pain points: If scaling becomes too manual, or if services crash under load...

4. Then adopt Kubernetes 🧠

It’s not about how complex your app is – it’s about when the traffic and growth demand an upgrade in how you manage things.

🎯 TL;DR for Founders and DevOps Teams

• Don’t jump to Kubernetes just because it’s trendy

• Use it only when traffic grows steadily and auto-scaling becomes necessary

• Kubernetes is most valuable when you want to scale reliably and efficiently

• Before that point, stick to simple deployments – it’ll save you time, money, and stress

🎉 Conclusion

Wow! What a journey we’ve been on 😄

We started by answering the big question — What is Kubernetes? We discovered that it’s not some mythical beast, but a powerful orchestration tool that helps us manage, deploy, scale, and maintain containerized applications in a smarter way.

Then, we took a step back in time to see how applications were deployed before Kubernetes — the headaches of manually installing software on servers, spinning up separate cloud instances for every microservice, and racking up huge cloud bills just to stay afloat. We also saw how containers simplified things, but even they had their own limitations when managed at scale.

That’s where Kubernetes came to the rescue

We explored:

• The problems Kubernetes solves – like auto-scaling, efficient resource management, cost savings, and seamless container grouping.

• Kubernetes architecture and components – breaking down complex terms like the cluster, master node, worker nodes, Pods, Services, Kubelet, and more, into simple, easy-to-digest ideas.

• Kubernetes workloads like Deployments, Pods, Services, DaemonSets, and StatefulSets, and what they do behind the scenes to keep our apps running reliably.

From theory to practice, we even got our hands dirty:

• We created a free Kubernetes cluster using Play with Kubernetes 🧪

• Deployed a real application using both imperative (direct command) and declarative (manifest file) approaches

• Understood why the declarative method makes our infrastructure easier to manage, especially when our systems grow.

Then we took a business lens 🔍 and looked at:

• The advantages of Kubernetes – from auto-scaling during traffic surges, to cost efficiency, and cloud-agnostic deployment.

• And also the disadvantages – like needing experienced DevOps engineers and not being ideal for every stage of a product's lifecycle.

Finally, we wrapped up with real-life use cases, highlighting when Kubernetes is a must-have, and when it’s better to wait – especially for early-stage startups still trying to find their audience.

So, whether you're a DevOps newbie, a startup founder, or just someone curious about how modern tech keeps your favorite apps online – you now have a strong foundational understanding of Kubernetes 🙌

Kubernetes is powerful, but it doesn't have to be overwhelming. With a solid grasp of the basics (which you now have 💪), you're well on your way to managing scalable applications like a pro.

Start simple. Grow smart. And when the time is right – Kubernetes will be your best friend.

Study Further 📚

If you would like to learn more about Kubernetes, you can check out the courses below:

• Docker & Kubernetes: The Practical Guide (Academind - Udemy)

• Certified Kubernetes Application Developer (CKAD) Specialization (Coursera)

About the Author 👨‍💻

Hi, I’m Prince! I’m a DevOps engineer and Cloud architect passionate about building, deploying, and managing scalable applications and sharing knowledge with the tech community.

If you enjoyed this article, you can learn more about me by exploring more of my blogs and projects on my LinkedIn profile. You can find my LinkedIn articles here. You can also visit my website to read more of my articles as well. Let’s connect and grow together! 😊

Kubernetes networking helps you manage how pods, services, and other external entities interact in this environment to ensure proper connectivity, isolation, and load distribution where necessary. It offers a flexible yet sophisticated networking system that implements fine-grained security controls through Network Policies.

One unique feature of Kubernetes is that it lets you deploy and manage multiple applications at scale within a single cluster. This helps you manage resources efficiently and optimizes costs as your applications run. But this also introduces challenges related to resource isolation and security. This is where proper Kubernetes networking becomes essential.

In this article, we will discuss the fundamentals of Kubernetes networking and how it facilitates secure connections within a cluster. We will also explore Network Policies as a mechanism for defining rules that regulate pod-to-pod and pod-to-external communication, ensuring fine-grained control over traffic flow within the cluster.

Here’s what we’ll cover:

1. Breakdown of Network Connectivity Types

2. What Are Kubernetes Network Policies?

   • How Kubernetes Network Policies Work

   • How to Implement Networking Policies

3. How to Set Up a Simple Kubernetes Network Policy on EKS

4. When and Why to Use Kubernetes Network Policies

5. Best practices for Implementing Kubernetes Network Policies on your cluster

Breakdown of Network Connectivity Types

Kubernetes networking is designed to achieve four important goals within the Kubernetes environment to ensure the seamless operation of a Kubernetes cluster. These goals are set to ensure that there is proper communication between the containers, pods, and external entities, enabling them to work together effectively within the Kubernetes infrastructure.

Container-to-Container Communication

One of the goals of implementing proper Kubernetes networking is to allow containers within the same pod to communicate directly with each other. Sharing the same networking namespace allows these containers to interact with each other using localhost, resulting in low-latency communication that helps multi-container applications function properly.

Container-to-container communication is useful when working with workloads that have tightly coupled processes and need to communicate quickly without latency within a single pod.

Pod-to-Pod connectivity

Within a Kubernetes environment, pods are assigned unique IP addresses, making pod-to-pod communication simple and straightforward. Understanding traditional networking between servers, Kubernetes removes the complexity of Network Address Translation (NAT), enabling pods to communicate with ease.

Pod-to-pod communication is the backbone of microservices architecture, allowing each pod to operate independently while remaining connected to others.

Pod-to-Service Interaction

Services in Kubernetes are often described as stable endpoints that help pods access each other. They ensure that traffic is routed to the right pod, regardless of the complexity of the pod setup. Service-to-pod communication is typically reliable, especially in environments where traffic and pod configurations are constantly evolving.

External-to-Internal Access

One of the goals of Kubernetes networking is also to manage traffic that comes from outside the cluster. There are several tools, like Ingress Controllers and LoadBalancers, that help handle external-to-internal communication. These tools help ensure that the right application is exposed to end-users, ensuring the proper delivery of services.

While Kubernetes networking meets the requirements of these goals, communication is usually open-ended by default. This means that pods within a cluster can freely communicate with each other without any restriction. This is not ideal, especially in a production environment where isolation and security are important. This is where Kubernetes Network Policies come into play.

What Are Kubernetes Network Policies?

Kubernetes Network Policies give you a way to enforce fine-grained control over the flow of traffic within your pods. These policies allow you to define which pods can communicate with each other or with other devices – so they act as a security layer with rules that restrict or allow specific types of traffic.

For example, if certain pods handle sensitive data or information, Network Policies can ensure that only authorized pods or external systems can gain access to it.

Implementing Network Policies also helps your Kubernetes clusters maintain security and compliance by restricting unnecessary communication and reducing traffic flow that could cause a security breach.

How Kubernetes Network Policies Work

Kubernetes Network Policies provide fine-grained access control within the Kubernetes cluster to manage network traffic at the pod level. Here, you can define separate rules for ingress and egress and restrict traffic to a particular port range.

In Kubernetes Network Policies, multiple policies can target the same pod. In this case, you can create "allow" rules to determine which traffic is permitted. Any traffic that doesn’t match the "allow" rule will be blocked.

Network Policies use IP addresses and port numbers to regulate traffic. This provides control over network flows to adhere to specific security requirements.

On the other hand, Network Policies aren’t a complete solution within an environment due to certain limitations. They cannot log blocked traffic events, meaning you cannot observe or debug why and when the Kubernetes Network Policy is blocking specific traffic. To achieve this, you need to use external tools supported by your CNI plugin.

CNI stands for Container Network Interface, a standard interface used by Kubernetes to manage network resources in containers. The CNI plugin is essential for providing container networking capabilities such as IP address allocation, routing, and enforcement of network policies. The plugin also enables the cluster to handle pod networking, including assigning network policies to pods and managing traffic flow between them.

Some popular network plugins include Calico, Cilium, Flannel, and Weave Net, each offering unique features and support for Network Policy integration.

How to Implement Networking Policies

Properly implementing Network Policies relies on the CNI plugin you’re using in the Kubernetes Cluster. For Network Policies to take effect, the CNI plugin configured on your cluster must support them.

Network policies are usually enabled by default in managed Kubernetes services provided by cloud platforms such as Amazon EKS, Microsoft Azure AKS, or Google Cloud GKE. But if you manage your own cluster, you need to ensure that your CNI plugin is compatible. For example, the popular CNI plugin Flannel doesn’t support network policies, whereas Calico does.

How to Set Up a Simple Kubernetes Network Policy on EKS

Prerequisites

Ensure you have the following installed on your Ubuntu Server:

• AWS CLI: For authentication and interactions with AWS resources

• kubectl: Kubernetes CLI

• eksctl: This is a CLI for managing EKS clusters

Steps

First, create your AWS EKS cluster using the following CLI commands:

eksctl create cluster \

  --name my-eks-cluster \

  --region us-east-1 \

  --nodegroup-name ng-eks \

  --node-type t3.medium \

  --nodes 3 \

  --nodes-min 2 \

  --nodes-max 4 \

  --with-oidc \

  --version 1.31

Next, enable the Amazon VPC CNI plugin for your Kubernetes cluster. To do this, within your EKS Cluster, make sure that the Amazon VPC CNI plugin is installed to manage pod networking.

Check the status like this:

kubectl get pods -n kube-system | grep aws-node

If it’s not running, deploy or update it to run on the cluster

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/v1.12/aws-k8s-cni.yaml

Amazon VPC CNI does not support and enforce network policies. So we have to install Calico, which is a CNI that works with the VPC CNI for Network Policies.

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

Confirm that Calico is installed and running like this:

kubectl get pods -n kube-system | grep calico

Now that we have set up Calico on our AWS EKS Cluster, let's examine various Kubernetes Network Policies that we can apply to it.

Allow all traffic to a specific pod in the cluster:

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: pod-network-policy

spec:

  podSelector:

    matchLabels:

      app: application-demo

  policyTypes:

    - Ingress

    - Egress

  ingress:

    - {}

  egress:

    - {}

This configuration defines a NetworkPolicy named pod-network-policy that applies to all pods with the label app: application-demo. The podSelector ensures that only the pods with this label are targeted.

• The policyTypes field indicates that this policy controls both Ingress (incoming traffic) and Egress (outgoing traffic).

• The ingress and egress rules are defined with empty braces {}, meaning no restrictions are applied—all traffic is allowed, both inbound and outbound.

Deny all traffic to a pod in the cluster:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: pod-network-policy
spec:
  podSelector:
    matchLabels:
      app: application-demo
  policyTypes:
    - Ingress
    - Egress

This configuration also selects pods labeled app: application-demo and applies the policy to both Ingress and Egress traffic.

Since no specific rules are defined, Kubernetes denies all traffic by default. This is also known as a "deny by default" policy, used to enforce strict isolation, preventing pods from communicating with others unless explicitly allowed by additional policies.

Deny all ingress traffic to the pods in the cluster

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: pod-network-policy
spec:
  podSelector: {}
  policyTypes:
    - Ingress

This configuration applies a NetworkPolicy to all pods in the namespace.

• The empty podSelector is empty ({}), meaning it applies to all pods in the namespace, regardless of their labels.

• The policyTypes field specifies that the policy only applies to Ingress traffic.

• Since no explicit Ingress rules are defined, Kubernetes blocks all incoming traffic by default.

Deny all egress traffic to the pods in the cluster

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: pod-network-policy

spec:

  podSelector: {}

  policyTypes:

    - Egress

In the configuration above:

• The podSelector is empty ({}), meaning the policy applies to all pods in the namespace.

• The policyTypes field specifies that this policy only applies to Egress traffic.

• Since no explicit Egress rules are defined, Kubernetes blocks all outgoing traffic for the target pods by default.

When and Why to Use Kubernetes Network Policies

There are various use cases for implementing Kubernetes Network Policies to improve cluster security.

For example, perhaps you want to restrict who/what can access the database. If you have a database deployed within the cluster, Kubernetes Network Policies ensure that only authorized pods can communicate with it, blocking access from unauthorized applications within the cluster.

Or perhaps you want to isolate sensitive pods. Properly implementating Network Policies helps isolate sensitive pods that do not need to accept inbound traffic from other pods, strengthening security within the infrastructure.

Best Practices for Implementing Kubernetes Network Policies on Your Cluster

To maximize the effectiveness and security benefits of your Kubernetes Network Policies, keep these best practices in mind:

• Ensure all pods are covered by a Network Policy: In an ideal production environment, all pods within the cluster should be covered by a network policy that limits their network to only the Ingress and Egress targets set in the configuration. Without this policy in place, all the pods can communicate freely, posing a huge security risk.

• Complement Network Policies with other security measures: While Network Policies are essential for Network isolation, they should be part of a wider security and networking strategy for your cluster. Additional safeguards should include Role-Based Access Control, which restricts unauthorized users from accessing or modifying the pod configurations, and advanced security contexts, which limit container capabilities.

• Always test Network Policies before deploying to production. Kubernetes Network Policies can be a bit of a hassle to validate, especially in a production environment, because they may hinder many running processes within the cluster. Always test new policies to ensure that they are working as intended within the cluster. For example, if you implement a new testing policy, use tools like curl or ping to verify blocked connectivity within the cluster.

• Always review your Network Policies as the cluster grows. As your cluster grows with the increasing user base and engineering needs, your Network Policies must always reflect new workloads, such as Pods and Namespaces. It is always best to review and update your Network Policies to stay relevant and ensure your environment is secure.

• Use precise target selectors for the configurations: Be specific when defining pod selectors, namespaces, and ipBlock ranges within your Network policies. For example, if you are working with namespace selectors, ensure that all the pods within that namespace conform to its security goals. Avoid using namespace selectors if you need to deploy pods that should communicate with other pods in the namespace. This is ideal because implementing namespace or pod selectors vaguely will impact the server, leading to unintended access.

Conclusion

In this article, you learned about Kubernetes Network Policies as a way to manage and restrict communication between pods. Since pods don’t have network isolation by default, setting up the right policies is important for security.

While Network Policies play an important role, it is also important to protect your Cloud environment by ensuring your infrastructure is hardened – so make sure you also implement RBAC and regular vulnerability scans. You should also allocate only needed pod resources, build minimal base images for the pods, and follow Kubernetes security best practices in general.

By doing this, you can achieve end-to-end protection for your workloads.

Fortunately, Docker comes to the rescue. But you might be wondering – what is Docker, and how does it help? You’ll learn all this and more in this tutorial.

But before we start, here are some prerequisites:

• Knowledge of Linux commands

• Knowledge of terminal usage

• Knowledge of Node.js and Express.js

Table Of Contents

1. What is Docker?

2. How to Install Docker

3. Demo Project: How to Containerize a Node.js Application

4. Wrapping Up

What is Docker?

Docker is an open-source tool that makes it easy to run software in a consistent way, no matter where you are. It does this by putting your application and everything it needs (like libraries and settings) into a container (which I’ll discuss more in a moment).

Think of a container like a box: it holds your app and all its parts, so it works exactly the same on your laptop, a server, or in the cloud. Docker helps developers avoid the "it works on my machine" problem by ensuring everything is packaged together in a reliable and portable way.

Docker was created by Solomon Hykes in 2013. Over the years, it has evolved to cover a wide range of tools. It’s become a go-to tool for improving the application deployment and networking processes.

Before we proceed, here are some key terms you will come across as we go through this tutorial:

Docker Engine

The Docker engine, as its name implies, is the powerhouse for Docker applications. It has a client and a server component. The Docker client, in our case, is the command-line interface tool or Docker terminal we’ll be using to send relevant commands for project execution. The Docker server, popularly known as the daemon, is the server that handles running the various Docker images and containers.

Docker Image

Docker images are premade templates of executable software and systems. Docker offers a wide range of images ranging from operating system templates to server templates, software templates, and so on. You can find all these on the Docker hub registry where these images are stored.

You can also build a specific image and host it either publicly on the Docker hub or in a private registry.

Docker Containers

Docker containers are executable compact instances built on the template generated which is the Docker image. They’re lightweight, portable packages that include everything needed to run a piece of software—code, runtime, libraries, and system tools. A container ensures the application runs consistently regardless of the environment.

Benefits of Using Docker

Here are some of the benefits of using Docker as a backend developer:

• Docker is a great tool for creating a solid DevOps culture for application development, as it clarifies the functions of the development and operations teams.

• It’s also quite flexible, allowing for easy deployment of microservices and distributed monolithic backend applications.

• It also minimizes errors from dependency misconfigurations during installations as it ports the app with its necessary dependencies all at once.

Moving on, we will be diving into how to Dockerize a Node.JS Express application. But before that, you’ll need to install Docker on your computer. You can skip this if you already have it installed.

How to Install Docker

Docker is a cross-platform tool which can be installed across all popular operating systems (Windows, Mac OS, and Linux distros). For this tutorial, I’ll only be highlighting how to set up Docker on Windows.

If you’re currently using any OS other than Windows, you can easily set Docker up by following the steps in the Docker documentation here.

For windows users, it is essential that your PC meets the minimum specifications – otherwise the installation won't be successful. The minimum requirements are the following:

• A Windows OS version not less than Windows 10 home

• A PC with WSL-2 installed or Hypervisor enabled.

With that, let's move on to downloading the Docker installer executable. You can download the latest Docker installer from here. After you do that, run the software and accept the terms and conditions. On successful completion, launch the application. This is what you should see:

To confirm that you’ve successfully installed the application, navigate to the command prompt terminal and run Docker –-version. You should see the exact version of the Docker engine tool you’ve installed if it was successful.

We’ll now move on to the project proper.

Demo Project: How to Containerize a Node.js Application

In this section, we will be containerizing a simple Node.js-based backend service with minimal dependencies. This will show you how to containerize and port an application using a Docker application containerization technique known as the Dockerfile. Keep in mind that if you have a more complex application, it may be better to use the Docker compose YAML tool.

To begin with, we will set up the sample Node.js application. I’ll provide the entire code setup in this article, below. But first, let’s understand what a dockerfile is.

What is a Dockerfile?

Basically, a Dockerfile is a template system which allows the user to input commands which, when executed, can produce a functional image of the application. This image can then be converted into a container.

Here are some commands included in the basic structure of a Dockerfile:

• CMD: sets the default command to run if no command is specified when the container starts. It can be overridden by providing a command when running the container (docker run ...).

• ENTRYPOINT: Specifies the main command that always runs when the container starts. It’s not easily overridden, but arguments can be appended.
  Note that CMD and ENTRYPOINT both specify what command or process the container should run when it starts. But they’re used differently and have distinct purposes. Use CMD for default behavior that can be overridden. Use ENTRYPOINT for a fixed command that defines the container's primary purpose.

• FROM: This is usually the opening statement in a Dockerfile. This command fetches a base image which forms the foundation for building the image of the application in question. For instance, in our application, the base image for a Node.js application is to have the baseline Node.js engine installed.

• WORKDIR: This syntax defines the active working directory where the application files will live within the defined container. An automatic folder will be created if it’s not already available.

• COPY: This syntax is used to ensure that the files necessary for creating the Docker image from the code base project file are copied into the newly created Docker container. The directories of these files are carefully highlighted.

• RUN: This syntax specifies the script that you want to be run before completing the application’s containerization.

• ENV: This syntax is used to highlight environmental variables and secrets which will be invoked during the process of running the application.

• EXPOSE: This syntax maps out the browsing port where the application is used to communicate with the external internet. For example EXPOSE: 3000 maps out the application web interface to localhost:3000.

Diving deeper into Docker, let’s quickly go over some key Docker commands we’ll be using throughout this tutorial:

• Docker ps: This command lists all the running containers on your Docker terminal.

• Docker run: This command executes a Docker image to trigger an instance of a container.

• Docker build: This command works based on the Docker file to generate an image of a service or application.

• Docker rm: this command can be used to delete an image using the image identification details.

How to Containerize the App

Now we can start containerizing our simple Node/Express application. To follow along with the tutorial, you can get the base code from here.

On testing it locally, it returns a CRUD API where you can create, fetch, update, and delete products when executed. We’ll package the application for easy deployment on the cloud using our Docker engine. We’ll be able to do this using the Dockerfile tool we discussed above.

Step 1: Create the dockerfile

In your project folder, create a file named Dockerfile. Make sure the name is exactly "Dockerfile" (no extension, and case-sensitive in some systems – so make sure it’s capitalized).

If you're using a code editor, simply create a new file named Dockerfile. If you're using a basic text editor, save the file with the name Dockerfile and ensure it doesn’t accidentally save with an extension like .txt.

Then enter the first line:

FROM Node:18-alpine

This command fetches the base image we’ll use to power our Express application which is the Node engine itself.

You might be wondering what the alpine is for. Alpine is a lightweight, much more compressed version of a Docker image. It excludes incorporating additional packages not directly essential to the base operating system. It's advocated as a standard good code practice to use lightweight distros for faster execution and easy use.

Step 2: Set the working directory

WORKDIR /app

This sets the working directory of the image to the /app folder of the container. It makes sure that all file actions occur here and all files are copied into this directory.

Step 3: Copy the necessary files

COPY package.json

This command copies the package.json files which has a list of dependences and packages to be installed to power our application.

Step 4: Execute a setup script

RUN npm install

This command ensures that all the necessary dependencies to power our Node.js applications are installed on the container.

Step 5: Copy the code files

COPY . .

This command ensures that all the files within the local directory get copied into the container file system within the established working directory.

Step 6: Expose the server port

EXPOSE 3000

This command exposes the server port that we intend to use to access the container. In this case it's port 3000.

Step 7: Include the command to bring the container to life

CMD ["npm", "run", "dev"]4

This command is executed a the end in order to power on the Node.js application. It simply runs the npm run dev command which is what you’d use for a development environment. To run it in a production environment, you’d use the npm start command instead.

Having completed this process, here is how the final Dockerfile structure should look:

FROM Node:18-alpine
WORKDIR /app

COPY package.json

RUN npm install

COPY . .

CMD ["npm", "run", "dev"]

Testing the Docker container

To round it up, we will be creating a Docker image of our Node.js application. To do this, execute the command docker build -t nodeapp . . The docker build command builds the image, while the -t allows for specifying the image tag’s details.

In our case, we’re assigning the name nodeapp to the image we will be creating and the image will be created within the working directory.

Congratulations! You have successfully built your first Docker image. To see all the images on your local repo, execute the command docker images.

In order to create a working instance of your image for testing, execute the command docker run nodeapp.

We’re using Mongo DB as our database for this tutorial, so we’ll need to pass the MongoDB URL as an environment variable to the Docker container. Environment variables help you safeguard certain key variables which shouldn’t be exposed to the public. Other variables which can be passed as environment variables include API keys and encryption codes.

To pass the MongoDB URL to the Docker container, we use the -e tag to ensure that Docker recognizes the corresponding value inputted as an environment variable.

docker run -e JWT_SECRETS={enter the value of your choice} -e MONGO_URL={The mongo url of your choice} nodeapp.

To also use the container in the background, just attach the -d tag which represents the detach option. This option allows the container to run in the background despite exiting the command line terminal.

In the event of no errors, navigating to localhost:5000 should also produce something similar to the image below.

Wrapping Up

In this article, you learned about what Docker is and how it works, along with its common commands and how to use it to containerize a backend application. Moving on from the basics, you can also explore other uses of Docker in continuous integration and development. To learn more about Docker, you can check out its documentation here.

I would also recommend using your new knowledge to deploy projects with real-life use cases, as well as exploring networking in Docker applications. To make your app live, you can easily deploy the Docker image you created to any of the popular cloud service providers like AWS, GCP, Azure, and so on.

Feel free to ask me any questions! You can also check out my other articles here. Till next time, keep on coding!

In this article, I want to show another alternative in case you use GitHub Actions as your CI platform (the most popular CI/CD solution at the moment). This alternative is called Service Containers, and I’ve realized that not many developers seem to know about it.

In this hands-on tutorial, I’ll demonstrate how to create a GitHub Actions workflow for integration tests with external dependencies (MongoDB and Redis) using the demo Go application we created in that previous tutorial. We’ll also review the pros and cons of GitHub Service Containers.

Prerequisites

• A basic understanding of GitHub Actions workflows.

• Familiarity with Docker containers.

• Basic knowledge of Go toolchain.

Table of Contents

• What are Service Containers?

• Why not Docker Compose?

• Job Runtime

• Readiness Healthcheck

• Private Container Registries

• Sharing Data Between Services

• Golang Integration Tests

• Personal Experience & Limitations

• Conclusion

• Resources

What are Service Containers?

Service Containers are Docker containers that offer a simple and portable way to host dependencies like databases (MongoDB in our example), web services, or caching systems (Redis in our example) that your application needs within a workflow.

This article focuses on integration tests, but there are many other possible applications for service containers. For example, you can also use them to run supporting tools required by your workflow, such as code analysis tools, linters, or security scanners.

Why Not Docker Compose?

Sounds similar to services in Docker Compose, right? Well, that’s because it is.

But while you could technically use Docker Compose within a GitHub Actions workflow by installing Docker Compose and running docker-compose up, service containers provide a more integrated and streamlined approach that’s specifically designed for the GitHub Actions environment.

Also, while they are similar, they solve different problems and have different general purposes:

• Docker Compose is good when you need to manage a multi-container application on your local machine or a single server. It’s best suited for long-living environments.

• Service Containers are ephemeral and exist only for the duration of a workflow run, and they’re defined directly within your GitHub Actions workflow file.

Just keep in mind that the feature set of service containers (at least as of now) is more limited compared to Docker Compose, so be ready to discover some potential bottlenecks. We will cover some of them at the end of this article.

Job Runtime

You can run GitHub jobs directly on a runner machine or in a Docker container (by specifying the container property). The second option simplifies the access to your services by using labels you define in the services section.

To run directly on a runner machine:

.github/workflows/test.yaml

jobs:
  integration-tests:
    runs-on: ubuntu-24.04

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017

    steps:
      - run: |
          echo "addr 127.0.0.1:27017"

Or you can run it in a container (Chainguard Go Image in our case):

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017
    steps:
      - run: |
          echo "addr mongo:27017"

You can also omit the host port, so the container port will be randomly assigned to a free port on the host. You can then access the port using the variable.

Benefits of omitting the host port:

• Avoids port conflicts – for example when you run many services on the same host.

• Enhances Portability – your configurations become less dependent on the specific host environment.

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:1.23

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/tcp
    steps:
      - run: |
          echo "addr mongo:${{ job.services.mongo.ports['27017'] }}"

Of course, there are pros and cons to each approach.

Running in a container:

• Pros: Simplified network access (use labels as hostnames), and automatic port exposure within the container network. You also get better isolation/security as the job runs in an isolated environment.

• Cons: Implied overhead of containerization.

Running on the runner machine:

• Pros: Potentially less overhead than running the job inside a container.

• Cons: Requires manual port mapping for service container access (using localhost:). There’s also less isolation/security, as the job runs directly on the runner machine. This potentially affects other jobs or the runner itself if something goes wrong.

Readiness Healthcheck

Prior to running the integration tests that connect to your provisioned containers, you’ll often need to make sure that the services are ready. You can do this by specifying docker create options such as health-cmd.

This is very important – otherwise the services may not be ready when you start accessing them.

In the case of MongoDB and Redis, these will be the following:

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017/27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

In the Action logs, you can see the readiness status:

Private Container Registries

In our example, we’re using public images from Dockerhub, but it’s possible to use private images from you private registries as well, such as Amazon Elastic Container Registry (ECR), Google Artifact Registry, and so on.

Make sure to store the credentials in Secrets and then reference them in the credentials section.

services:
  private_service:
    image: ghcr.io/org/service_repo
    credentials:
      username: ${{ secrets.registry_username }}
      password: ${{ secrets.registry_token }}

Sharing Data Between Services

You can use volumes to share data between services or other steps in a job. You can specify named Docker volumes, anonymous Docker volumes, or bind mounts on the host. But it’s not directly possible to mount the source code as a container volume. You can refer to this open discussion for more context.

To specify a volume, you specify the source and destination path: <source>:<destinationPath>

The <source> is a volume name or an absolute path on the host machine, and <destinationPath> is an absolute path in the container.

volumes:
  - /src/dir:/dst/dir

Volumes in Docker (and GitHub Actions using Docker) provide persistent data storage and sharing between containers or job steps, decoupling data from container images.

Project Setup

Before diving into the full source code, let's set up our project for running integration tests with GitHub Service Containers.

1. Create a new GitHub repository.

2. Initialize a Go module using go mod init

3. Create a simple Go application.

4. Add integration tests in integration_test.go

5. Create a .github/workflows directory.

6. Create a file named integration-tests.yaml inside the .github/workflows directory.

Golang Integration Tests

Now as we can provision our external dependencies, let’s have a look at how to run our integration tests in Go. We will do it in the steps section of our workflow file.

We will run our tests in a container which uses Chainguard Go image. This means we don’t have to install/setup Go. If you want to run your tests directly on a runner machine, you need to use the setup-go Action.

You can find the full source code with tests and this workflow here.

.github/workflows/integration-tests.yaml

name: "integration-tests"

on:
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  integration-tests:
    runs-on: ubuntu-24.04
    container: cgr.dev/chainguard/go:latest

    env:
      MONGO_URI: mongodb://mongo:27017
      REDIS_URI: redis://redis:6379

    services:
      mongo:
        image: mongodb/mongodb-community-server:7.0-ubi8
        ports:
          - 27017:27017
        options: >-
          --health-cmd "echo 'db.runCommand("ping").ok' | mongosh mongodb://localhost:27017/test --quiet"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

      redis:
        image: redis:7
        ports:
          - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s
          --health-timeout 10s
          --health-retries 10

    steps:
      - name: Check out repository code
        uses: actions/checkout@v4

      - name: Download dependencies
        run: go mod download

      - name: Run Integration Tests
        run: go test -tags=integration -timeout=120s -v ./...

To summarize what’s going on here:

1. We run our job in a container with Go (container)

2. We spin up two services: MongoDB and Redis (services)

3. We configure healthchecks to make sure our services are “Healthy” when we run the tests (options)

4. We perform a standard code checkout

5. Then we run the Go tests

Once the Action is completed (it took ~1 min for this example), all the services will be stopped and orphaned so we don’t need to worry about that.

Personal Experience & Limitations

We’ve been using service containers for running backend integration tests at BINARLY for some time, and they work great. But the initial workflow creation took some time and we encountered the following bottlenecks:

• It’s not possible to override or run custom commands in an action service container (as you would do in Docker Compose using the command property). Open pull request

  • Workaround: we had to find a solution that doesn’t require that. In our case, we were lucky and could do the same with environment variables.

• It’s not directly possible to mount the source code as a container volume. Open discussion

  • While this is indeed a big limitation, you can copy the code from your repository into your mounted directory after the service container has started.

Conclusion

GitHub service containers are a great option to scaffold an ephemeral testing environment by configuring it directly in your GitHub workflow. With configuration being somewhat similar to Docker Compose, it’s easy to run any containerised application and communication with it in your pipeline. This ensures that GitHub runners take care of shutting everything down upon completion.

If you use Github Actions, this approach works extremely well as it is specifically designed for the GitHub Actions environment.

Resources

• Source Code

• GitHub Documentation

• Discover more articles on packagemain.tech

Containers and Docker are technologies that have revolutionized how software is built, tested, and deployed.

Whether you're new to the world of tech or just looking to understand the basics of Docker, this article will guide you through the essentials.

Table of Content

• What Are Containers?

• What is Docker?

• Why Docker?

• Docker Architecture

• Docker’s Container Runtime: containerd

• How to Create a Simple Container Using Docker

• Wrapping Up

What Are Containers?

Before diving into Docker, let’s first understand containers. Imagine that you’re working on a project, and your application works perfectly on your laptop. But when you try to run the same application on a different machine, it fails. This is often due to differences in environments: different operating systems, installed software versions, or configurations.

Containers solve this problem by packaging an application and all its dependencies like libraries, frameworks, and configuration files into a single, standardized unit. This ensures that the application runs the same no matter where it's deployed, whether on your laptop, a server, or in the cloud.

Key features of containers:

• Lightweight: Containers share the host system's kernel, unlike virtual machines (VMs) that require separate OS instances, making them faster and more efficient.

• Portable: Once built, a container can run consistently across various environments.

• Isolated: Containers run in isolated processes, meaning that they don’t interfere with other applications running on the same system.

What is Docker?

Now that we understand containers, let’s talk about Docker, the platform that has made containers mainstream.

Docker is an open-source tool designed to simplify the process of creating, managing, and deploying containers. Launched in 2013, Docker has rapidly become the go-to solution for containerization due to its ease of use, community support, and powerful ecosystem of tools.

Key Concepts in Docker

1. Docker Images: Think of a Docker image as a blueprint for your container. It contains everything needed to run the application, including code, libraries, and system dependencies. Images are built from a set of instructions written in a Dockerfile.

2. Docker Containers: A container is a running instance of a Docker image. When you create and start a container, Docker launches the image into an isolated environment where your application can run.

3. Dockerfile: This is a text file that contains the steps needed to create a Docker image. It’s where you define what your container will look like, including the base image, application code, and any additional dependencies.

4. Docker Hub: Docker Hub is a public registry where developers can share and access pre-built images. If you're working on a common application or technology stack, chances are that there’s already an image available on Docker Hub, saving you time.

5. Docker Compose: For applications that require multiple containers (for example, a web server and a database), Docker Compose allows you to define and manage multi-container environments using a simple YAML file.

Why Docker?

Docker's popularity stems from its ability to solve a variety of challenges developers face today:

• Consistency Across Environments: Developers can "build once, run anywhere," ensuring the same application works the same way in different environments, from local development to production.

• Speed: Docker containers are fast to start and stop, making them ideal for testing and deployment pipelines.

• Efficient Use of Resources: Since containers share the host system's resources more effectively than virtual machines, they reduce overhead and allow for greater density in deployments.

• Version Control for Your Applications: Docker allows you to version control not only your code but also the environment in which your code runs. This is particularly useful for rolling back to previous versions or debugging issues in production.

Docker Architecture

When you first start using Docker, you may treat it as a box that "just works." While that’s fine for getting started, a deeper understanding of Docker’s architecture will help you troubleshoot issues, optimize performance, and make informed decisions about your containerization strategy.

Docker's architecture is designed to ensure efficiency, flexibility, and scalability. It’s composed of several components that work together to create, manage, and run containers. Let’s take a closer look at each of these components.

Docker Architecture: Key Components

Docker’s architecture is built around a client-server model that includes the following components

• Docker Client

• Docker Daemon (dockerd)

• Docker Engine

• Docker Images

• Docker Containers

• Docker Registries

1. Docker Client

The Docker Client is the primary way users interact with Docker. It’s a command-line tool that sends instructions to the Docker Daemon (which we’ll cover next) using REST APIs. Commands like docker build, docker pull, and docker run are executed from the Docker Client.

When you type a command like docker run nginx, the Docker Client translates that into a request that the Docker Daemon can understand and act upon. Essentially, the Docker Client acts as a front-end for interacting with Docker’s more complex backend components.

2. Docker Daemon (dockerd)

The Docker Daemon, also known as dockerd, is the brain of the entire Docker operation. It’s a background process that listens for requests from the Docker Client and manages Docker objects like containers, images, networks, and volumes.

Here’s what the Docker Daemon is responsible for

• Building and running containers: When the client sends a command to run a container, the daemon pulls the image, creates the container, and starts it.

• Managing Docker resources: The daemon handles tasks like network configurations and volume management.

• The Docker Daemon runs on the host machine and communicates with the Docker Client using a REST API, Unix sockets, or a network interface. It’s also responsible for interacting with container runtimes, which handle the actual execution of containers.

3. Docker Engine

The Docker Engine is the core part of Docker. It’s what makes the entire platform work, combining the client, daemon, and container runtime. Docker Engine can run on various operating systems, including Linux, Windows, and macOS.

There are two versions of the Docker Engine

• Docker CE (Community Edition): This is the free, open-source version of Docker that’s widely used for personal and smaller-scale projects.

• Docker EE (Enterprise Edition): The paid, enterprise-level version of Docker comes with additional features like enhanced security, support, and certification.

The Docker Engine simplifies the complexities of container orchestration by integrating the various components required to build, run, and manage containers.

4. Docker Images

A Docker Image is a read-only template that contains everything your application needs to run—code, libraries, dependencies, and configurations. Images are the building blocks of containers. When you run a container, you are essentially creating a writable layer on top of a Docker image.

Docker Images are typically built from Dockerfiles, which are text files that contain instructions on how to build the image. For example, a basic Dockerfile might start with a base image like nginx or ubuntu and include commands to copy files, install dependencies, or set environment variables.

Here’s a simple example of a Dockerfile

dockerfileCopy codeFROM nginx:latest
COPY ./html /usr/share/nginx/html
EXPOSE 80

In this example, we’re using the official Nginx image as the base and copying our local HTML files into the container’s web directory.

Once the image is built, it can be stored in a Docker Registry and shared with others.

5. Docker Containers

A Docker Container is a running instance of a Docker Image. It’s lightweight and isolated from other containers, yet it shares the kernel of the host operating system. Each container has its own file system, memory, CPU allocation, and network settings, which makes it portable and reproducible.

Containers can be created, started, stopped, and destroyed, and they can even be persisted between reboots. Because containers are based on images, they ensure that applications will behave the same way no matter where they’re run.

A few key characteristics of Docker containers:

• Isolation: Containers are isolated from each other and the host, but they still share the same OS kernel.

• Portability: Containers can run anywhere, whether on your local machine, a virtual machine, or a cloud provider.

6. Docker Registries

A Docker Registry is a centralized place where Docker Images are stored and distributed. The most popular registry is Docker Hub, which hosts millions of publicly available images. Organizations can also set up private registries to store and distribute their own images securely.

Docker Registries provide several key features:

• Image Versioning: Images are versioned using tags, making it easy to manage different versions of an application.

• Access Control: Registries can be public or private, with role-based access control to manage who can pull or push images.

• Distribution: Images can be pulled from a registry and deployed anywhere, making it easy to share and reuse containerized applications.

Docker’s Container Runtime: containerd

One important recent development in Docker’s architecture is the use of containerd. Docker used to have its own container runtime, but now it uses containerd, a container runtime that follows industry standards and is also used by other platforms like Kubernetes.

1. containerd is responsible for

   • Starting and stopping containers

   • Managing storage and networking for containers

   • Pulling container images from registries

By separating the container runtime from Docker’s higher-level functionality, Docker has become more modular, allowing other tools to use containerd while Docker focuses on user-facing features.

How to Create a Simple Container Using Docker

Pull the Linux Image

Start by pulling the alpine image from Docker Hub. The alpine image is a minimal Linux distribution, designed to be lightweight and fast.

Run the following command:

docker pull alpine

This will download the alpine image to your local system.

Run the Container

Create and start a container using the alpine image. We’ll also launch a terminal session inside the container.

docker run -it alpine /bin/sh

Here’s what each option means:

• docker run: Creates and starts a new container.

• -it: Allows you to interact with the container (interactive mode + terminal).

• alpine: Specifies the image to use.

• /bin/sh: Specifies the command to run inside the container (a shell session in this case).

Explore the Container

Once the container is running, you’ll see a shell prompt that looks something like this

/ #

This indicates you are inside the Alpine Linux container. You can now run Linux commands. For example:

Check the current directory:

pwd

List files in the directory:

ls

Output: A minimal directory structure, as Alpine is a lightweight image.

You can also install a package (Alpine uses apk as its package manager):

apk add curl

Exit the Container

When you're done exploring, type exit to close the session and stop the container

bashCopy codeexit

Access the Container After It’s Stopped

If you want to access the container again after stopping it, you can use this command to list all containers (including stopped ones):

docker ps -a

You’ll see a list of containers with their IDs and statuses, then you can start the stopped container:

docker start <container-id>

You can attach to the container's shell using this command:

docker exec -it <container-id> /bin/sh

If you no longer need the container, you can remove it

1. Stop the container (if it’s still running):

    docker stop <container-id>
   

2. Remove the container:

    docker rm <container-id>
   

Key Docker Commands Recap

Wrapping Up

Now that you've got a foundational understanding, it's time to put your knowledge to use. Start experimenting with Docker, build your first container, and explore its vast ecosystem.

You'll soon see why Docker has become a cornerstone of modern DevOps and software engineering.

You can follow me on

• Twitter

• LinkedIn

EDA enables different parts of an application to interact in a decoupled way, allowing them to communicate through events instead of direct calls. This setup lets components work independently, respond to events asynchronously, and adjust to changing business needs without major system reconfiguration, promoting agility.

New and modern applications now heavily rely on real-time data processing and responsiveness. The EDA’s importance cannot be overstated because it provides the framework that supports those requirements. By using asynchronous communication and event-driven interactions, systems can efficiently handle high volumes of transactions and maintain performance under unstable loads. These features are particularly appreciated in environments where changes are very spontaneous, such as e-commerce platforms or IoT applications.

Some key components of EDA include:

• Event Sources: These are the producers that generate events when significant actions occur within the system. Examples include user interactions or data changes.

• Listeners: These are entities that subscribe to specific events and respond when those events occur. Listeners enable the system to react dynamically to changes.

• Handlers: These are responsible for processing the events once they are detected by listeners, executing the necessary business logic or workflows triggered by the event.

In this article, you will learn how to implement event-driven data processing using Traefik, Kafka, and Docker.

Here is a simple application hosted on GitHub that you can quickly run to get an overview of what you will be building today.

Table of Contents

Here is what we'll cover:

• Table of Contents

• Prerequisites

• Understanding the Technologies

• How to Set Up the Environment

• How to Build the Event-Driven System

• How to Integrate Traefik with Kafka

• Testing the Setup

• Conclusion

Let's get started!

Prerequisites

Before you begin:

• Deploy an Ubuntu 24.04 instance with at least 4 GB of RAM and a minimum of 20 GB of free disk space to accommodate Docker images, containers, and Kafka data.

• Access the instance with a non-root user with sudo privileges.

• Update the package index.

sudo apt update

Understanding the Technologies

Apache Kafka

Apache Kafka is a distributed event streaming platform built for high-throughput data pipelines and real-time streaming applications. It acts as the backbone for implementing EDA by efficiently managing large volumes of events. Kafka uses a publish-subscribe model where producers send events to topics, and consumers subscribe to these topics to receive the events.

Some of the key features of Kafka include:

• High Throughput: Kafka is capable of handling millions of events per second with low latency, making it suitable for high-volume applications.

• Fault Tolerance: Kafka's distributed architecture ensures data durability and availability even in the face of server failures. It replicates data across multiple brokers within a cluster.

• Scalability: Kafka can easily scale horizontally by adding more brokers to the cluster or partitions to topics, accommodating growing data needs without significant reconfiguration.

Traefik

Traefik is a modern HTTP reverse proxy and load balancer designed specifically for microservices architectures. It automatically discovers services running in your infrastructure and routes traffic accordingly. Traefik simplifies the management of microservices by providing dynamic routing capabilities based on service metadata.

Some of the key features of Traefik include:

• Dynamic Configuration: Traefik automatically updates its routing configuration as services are added or removed, eliminating manual intervention.

• Load Balancing: It efficiently distributes incoming requests across multiple service instances, improving performance and reliability.

• Integrated Dashboard: Traefik provides a user-friendly dashboard for monitoring traffic and service health in real-time.

By using Kafka and Traefik in an event-driven architecture, you can build responsive systems that efficiently handle real-time data processing while maintaining high availability and scalability.

How to Set Up the Environment

How to Install Docker on Ubuntu 24.04

1. Install the required packages.

sudo apt install ca-certificates curl gnupg lsb-release

2. Add Docker’s official GPG Key.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

3. Add the Docker repository to your APT sources.

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

4. Update the package index again and install Docker Engine with the Docker Compose plugin.

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

5. Check to verify the installation.

sudo docker run hello-world

Expected Output:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:305243c734571da2d100c8c8b3c3167a098cab6049c9a5b066b6021a60fcb966
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

How to Configure Docker Compose

Docker Compose simplifies the management of multi-container applications, allowing you to define and run services in a single file.

1. Create a project directory

mkdir ~/kafka-traefik-setup && cd ~/kafka-traefik-setup

2. Create a docker-compose.yml file.

nano docker-compose.yml

3. Add the following configuration to the file to define your services.

version: '3.8'

services:
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"

  traefik:
    image: traefik:v2.9
    ports:
      - "80:80"       # HTTP traffic
      - "8080:8080"   # Traefik dashboard (insecure)
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

Save your changes with ctrl + o, then exit with ctrl + x.

4. Start your services.

docker compose up -d

Expected Output:

[+] Running 4/4
 ✔ Network kafka-traefik-setup_default        Created                  0.2s
 ✔ Container kafka-traefik-setup-zookeeper-1  Started                  1.9s
 ✔ Container kafka-traefik-setup-traefik-1    Started                  1.9s
 ✔ Container kafka-traefik-setup-kafka-1      Started                  1.9s

How to Build the Event-Driven System

How to Create Event Producers

To produce events in Kafka, you will need to implement a Kafka producer. Below is an example using Java.

1. Create a file kafka-producer.java.

nano kafka-producer.java

2. Add the following configuration for a Kafka Producer.

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

import java.util.Properties;

public class SimpleProducer {
    public static void main(String[] args) {
        // Set up the producer properties
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        // Create the producer
        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        try {
            // Send a message to the topic "my-topic"
            ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "Hello, Kafka!");
            RecordMetadata metadata = producer.send(record).get(); // Synchronous send
            System.out.printf("Sent message with key %s to partition %d with offset %d%n", 
                              record.key(), metadata.partition(), metadata.offset());
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            // Close the producer
            producer.close();
        }
    }
}

Save your changes with ctrl + o, then exit with ctrl + x.

In the above configuration, the producer sends a message with the key "key1" and the value "Hello, Kafka!" to the topic "my-topic".

How to Set Up Kafka Topics

Before producing or consuming messages, you need to create topics in Kafka.

1. Use the kafka-topics.sh script included with your Kafka installation to create a topic.

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic <TopicName> --partitions <NumberOfPartitions> --replication-factor <ReplicationFactor>

For example, if you want to create a topic named my-topic with 3 partitions and a replication factor of 1, run:

docker exec <Kafka Container ID> /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-topic --partitions 3 --replication-factor 1

Expected Output:

Created topic my-topic.

2. Check to confirm if the Topic was created successfully.

docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Expected Output:

my-topic

How to Create Event Consumers

After you have created your producers and topics, you can create consumers to read messages from those topics.

1. Create a file kafka-consumer.java.

nano kafka-consumer.java

2. Add the following configuration for a Kafka consumer.

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class SimpleConsumer {
    public static void main(String[] args) {
        // Set up the consumer properties
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
        props.put(ConsumerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");

        // Create the consumer
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

        // Subscribe to the topic
        consumer.subscribe(Collections.singletonList("my-topic"));

        try {
            while (true) {
                // Poll for new records
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("Consumed message with key %s and value %s from partition %d at offset %d%n",
                                      record.key(), record.value(), record.partition(), record.offset());
                }
            }
        } finally {
            // Close the consumer
            consumer.close();
        }
    }
}

Save your changes with ctrl + o, then exit with ctrl + x.

In the above configuration, the consumer subscribes to my-topic and continuously polls for new messages. When messages are received, it prints out their keys and values along with partition and offset information.

How to Integrate Traefik with Kafka

Configure Traefik as a Reverse Proxy.

Integrating Traefik as a reverse proxy for Kafka allows you to manage incoming traffic efficiently while providing features such as dynamic routing and SSL termination.

1. Update the docker-compose.yml file.

version: '3.8'

services:
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.kafka.rule=Host(`kafka.example.com`)"
      - "traefik.http.services.kafka.loadbalancer.server.port=9092"

  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"

  traefik:
    image: traefik:v2.9
    ports:
      - "80:80"        # HTTP traffic
      - "8080:8080"    # Traefik dashboard (insecure)
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

In this configuration, replace kafka.example.com with your actual domain name. The labels define the routing rules that Traefik will use to direct traffic to the Kafka service.

2. Restart your services.

docker compose up -d

3. Access your Traefik dashboard by accessing http://localhost:8080 on your web browser.

   

   Load Balancing with Traefik

   Traefik provides built-in load balancing capabilities that can help distribute requests across multiple instances of your Kafka producers and consumers.

   Strategies for Load Balancing Event-Driven Microservices

   1. Round Robin:

By default, Traefik uses a round-robin strategy to distribute incoming requests evenly across all available instances of a service. This is effective for balancing load when multiple instances of Kafka producers or consumers are running.

2. Sticky Sessions:

If you require that requests from a specific client always go to the same instance (for example, maintaining session state), you can configure sticky sessions in Traefik using cookies or headers.

3. Health Checks:

Configure health checks in Traefik to ensure that traffic is only routed to healthy instances of your Kafka services. You can do this by adding health check parameters in the service definitions within your docker-compose.yml file:

    labels:
      - "traefik.http.services.kafka.loadbalancer.healthcheck.path=/health"
      - "traefik.http.services.kafka.loadbalancer.healthcheck.interval=10s"
      - "traefik.http.services.kafka.loadbalancer.healthcheck.timeout=3s"

Testing the Setup

Verifying Event Production and Consumption

1. Kafka provides built-in command-line tools for testing. Start a Console producer.

    docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

After running this command, you can type messages into the terminal, which will be sent to the specified Kafka topic.

2. Start another terminal session and start a console consumer.

    docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

This command will display all messages in my-topic, including those produced before the consumer started.

3. To see how well your consumers are keeping up with producers, you can run the following command to check the lag for a specific consumer group.

    docker exec -it kafka-traefik-setup-kafka-1 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <your-consumer-group>

Monitoring and Logging

1. Kafka Metrics:

Kafka exposes numerous metrics that can be monitored using JMX (Java Management Extensions). You can configure JMX to export these metrics to monitoring systems like Prometheus or Grafana. Key metrics to monitor include:

• Message Throughput: The rate of messages produced and consumed.

• Consumer Lag: The difference between the last produced message offset and the last consumed message offset.

• Broker Health: Metrics related to broker performance, such as request rates and error rates.

2. Prometheus and Grafana Integration:

To visualize Kafka metrics, you can set up Prometheus to scrape metrics from your Kafka brokers. Follow these steps:

• Enable JMX Exporter on your Kafka brokers by adding it as a Java agent in your broker configuration.

• Configure Prometheus by adding a scrape job in its configuration file (prometheus.yml) that points to your JMX Exporter endpoint.

• Use Grafana to create dashboards that visualize these metrics in real-time.

How to Implement Monitoring for Traefik

1. Traefik Metrics Endpoint.

Traefik provides built-in support for exporting metrics via Prometheus. To enable this feature, add the following configuration in your Traefik service definition within docker-compose.yml:

    command:
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.addservice=true"

2. Visualizing Traefik Metrics with Grafana.

Once Prometheus is scraping Traefik metrics, you can visualize them using Grafana:

• Create a new dashboard in Grafana and add panels that display key Traefik metrics such as:

• traefik_entrypoint_requests_total: Total number of requests received.

• traefik_backend_request_duration_seconds: Response times of backend services.

• traefik_service_requests_total: Total requests forwarded to backend services.

3. Setting Up Alerts.

Configure alerts in Prometheus or Grafana based on specific thresholds (e.g., high consumer lag or increased error rates).

Conclusion

In this guide, you successfully implemented Event Driven Architecture (EDA) using Kafka and Traefik within the Ubuntu 24.04 environment.

Additional Resources

To learn more you can visit:

• The Apache Kafka Official Documentation

• The Traefik Official Documentation

• The Docker Official Documentation

• Vultr guide for for setting up Traefik Proxy on Ubuntu 24.04

There are many public and private registries available to developers such as Docker Hub, Amazon ECR, and Google Cloud Artifact Registry. But sometimes, instead of relying on an external vendor, you might want to host your images yourself. This gives you more control over how the registry is configured and where the container images are hosted.

This article is a hands-on tutorial that’ll teach you how to self-host a Container Registry.

Table of Contents

• What is a Container Image?

• What is a Container Registry?

• Why you might want to self-host a Container Registry

• How to self-host a Container Registry

• Step 1: Install Docker and Docker Compose on the server

• Step 2: Configure and run the registry container

• Step 3: Run NGINX for handling TLS

• Ready to go!

• Other options

• Conclusion

You will get the most out of this article if you’re already familiar with the tools like Docker and NGINX, and have a general understanding of what a container is.

What is a Container Image?

Before we talk about container registries, let's first understand what a container image is. In a nutshell, a container image is a package that includes all of the files, libraries, and configurations to run a container. They are composed of layers where each layer represents a set of file system changes that add, remove, or modify files.

The most common way to create a container image is to use a Dockerfile.

# build an image
docker build -t pliutau/hello-world:v0 .

# check the images locally
docker images
# REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
# hello-world   latest    9facd12bbcdd   22 seconds ago   11MB

This creates a container image that is stored on your local machine. But what if you want to share this image with others or use it on a different machine? This is where container registries come in.

What is a Container Registry?

A container registry is a storage catalog where you can push and pull container images from. The images are grouped into repositories, which are collections of related images with the same name. For example, on Docker Hub registry, nginx is the name of the repository that contains different versions of the NGINX images.

Some registries are public, meaning that the images hosted on them are accessible to anyone on the Internet. Public registries such as Docker Hub are a good option to host open-source projects.

On the other hand, private registries provide a way to incorporate security and privacy into enterprise container image storage, either hosted in cloud or on-premises. These private registries often come with advanced security features and technical support.

There is a growing list of private registries available such as Amazon ECR, GCP Artifact Registry, GitHub Container Registry, and Docker Hub also offers a private repository feature.

As a developer, you interact with a container registry when using the docker push and docker pull commands.

docker push docker.io/pliutau/hello-world:v0

# In case of Docker Hub we could also skip the registry part
docker push pliutau/hello-world:v0

Let's look at the anatomy of a container image URL:

docker pull docker.io/pliutau/hello-world:v0@sha256:dc11b2...
                |            |            |          |
                ↓            ↓            ↓          ↓
             registry    repository      tag       digest

Why You Might Want to Self-host a Container Registry

Sometimes, instead of relying on a provider like AWS or GCP, you might want to host your images yourself. This keeps your infrastructure internal and makes you less reliant on external vendors. In some heavily regulated industries, this is even a requirement.

A self-hosted registry runs on your own servers, giving you more control over how the registry is configured and where the container images are hosted. At the same time it comes with a cost of maintaining and securing the registry.

How to Self-host a Container Registry

There are several open-source container registry solutions available. The most popular one is officially supported by Docker, called registry, with its implementation for storing and distributing of container images and artifacts. This means that you can run your own registry inside a container.

Here are the main steps to run a registry on a server:

• Install Docker and Docker Compose on the server.

• Configure and run the registry container.

• Run NGINX for handling TLS and forwarding requests to the registry container.

• Setup SSL certificates and configure a domain.

Step 1: Install Docker and Docker Compose on the server

You can use any server that supports Docker. For example, you can use a DigitalOcean Droplet with Ubuntu. For this demo I used Google Cloud Compute to create a VM with Ubuntu.

neofetch

# OS: Ubuntu 20.04.6 LTS x86_64
# CPU: Intel Xeon (2) @ 2.200GHz
# Memory: 3908MiB

Once we're inside our VM, we should install Docker and Docker Compose. Docker Compose is optional, but it makes it easier to manage multi-container applications.

# install docker engine and docker-compose
sudo snap install docker

# verify the installation
docker --version
docker-compose --version

Step 2: Configure and run the registry container

Next we need to configure our registry container. The following compose.yaml file will create a registry container with a volume for storing the images and a volume for storing the password file.

services:
  registry:
    image: registry:latest
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
    volumes:
      # Mount the password file
      - ./registry/registry.password:/auth/registry.password
      # Mount the data directory
      - ./registry/data:/data
    ports:
      - 5000

The password file defined in REGISTRY_AUTH_HTPASSWD_PATH is used to authenticate users when they push or pull images from the registry. We should create a password file using the htpasswd command. We should also create a folder for storing the images.

mkdir -p ./registry/data

# install htpasswd
sudo apt install apache2-utils

# create a password file. username: busy, password: bee
htpasswd -Bbn busy bee > ./registry/registry.password

Now we can start the registry container. If you see this message, than everything is working as it should:

docker-compose up

# successfull run should output something like this:
# registry | level=info msg="listening on [::]:5000"

Step 3: Run NGINX for handling TLS

As mentioned earlier, we can use NGINX to handle TLS and forward requests to the registry container.

The Docker Registry requires a valid trusted SSL certificate to work. You can use something like Let's Encrypt or obtain it manually. Make sure you have a domain name pointing to your server (registry.pliutau.com in my case). For this demo I already obtained the certificates using certbot and put it in the ./nginx/certs directory.

Since we're running our Docker Registry in a container, we can run NGINX in a container as well by adding the following service to the compose.yaml file:

services:
  registry:
    # ...
  nginx:
    image: nginx:latest
    depends_on:
      - registry
    volumes:
      # mount the nginx configuration
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      # mount the certificates obtained from Let's Encrypt
      - ./nginx/certs:/etc/nginx/certs
    ports:
      - "443:443"

Our nginx.conf file could look like this:

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    upstream registry {
        server registry:5000;
    }

    server {
        server_name registry.pliutau.com;
        listen 443 ssl;

        ssl_certificate /etc/nginx/certs/fullchain.pem;
        ssl_certificate_key /etc/nginx/certs/privkey.pem;

        location / {
            # important setting for large images
            client_max_body_size                1000m;

            proxy_pass                          http://registry;
            proxy_set_header  Host              $http_host;
            proxy_set_header  X-Real-IP         $remote_addr;
            proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header  X-Forwarded-Proto $scheme;
            proxy_read_timeout                  900;
        }
    }
}

Ready to go!

After these steps we can run our registry and Nginx containers.

docker-compose up

Now, on the client side, you can push and pull the images from your registry. But first we need to login to the registry.

docker login registry.pliutau.com

# Username: busy
# Password: bee
# Login Succeeded

Time to build and push our image to our self-hosted registry:

docker build -t registry.pliutau.com/pliutau/hello-world:v0 .

docker push registry.pliutau.com/pliutau/hello-world:v0
# v0: digest: sha256:a56ea4... size: 738

On your server you can check the uploaded images in the data folder:

ls -la ./registry/data/docker/registry/v2/repositories/

Other options

Following the example above, you can also run the registry on Kubernetes. Or you could use a managed registry service like Harbor, which is an open-source registry that provides advanced security features and is compatible with Docker and Kubernetes.

Also, if you want to have a UI for your self-hosted registry, you could use a project like joxit/docker-registry-ui and run it in a separate container.

Conclusion

Self-hosted Container Registries allow you to have complete control over your registry and the way it's deployed. At the same time it comes with a cost of maintaining and securing the registry.

Whatever your reasons for running a self-hosted registry, you now know how it's done. From here you can compare the different options and choose the one that best fits your needs.

You can find the full source code for this demo on GitHub. Also, you can watch it as a video on our YouTube channel.