I did. I spun up an EC2 instance, followed along for an hour, and then forgot to shut it down. A week later I had a $34 bill for a machine running nothing.

That was the last time I practiced on someone else's infrastructure.

Everything in this guide runs on your laptop. No cloud account, no credit card, no bill at the end of the month. By the end, you'll be able to spin up a multi-server environment from scratch, configure it automatically with Ansible, serve a site you wrote yourself, and diagnose what breaks when you intentionally destroy it.

That last part is where the actual learning happens.

Prerequisites

Before you start, make sure you have:

• A laptop with at least 8GB of RAM (16GB is better)

• At least 20GB of free disk space

• Windows, macOS, or Linux operating system

• Administrator access to your computer

• Virtualization enabled in your BIOS/UEFI settings

• A stable internet connection for the initial downloads

Knowledge and comfort level:

• You should be comfortable using a terminal (running commands, changing directories, and editing small text files with whatever editor you like).

• Basic familiarity with concepts like “a server,” “SSH,” and “a port” helps, but you don't need prior experience with Docker, Kubernetes, Vagrant, or Ansible. This guide introduces them as you go.

If you can follow step-by-step instructions and read error output without panicking, you're ready.

Table of Contents

1. What is DevOps?

2. Why Build a Local Lab?

3. How to Set Up Docker

4. How to Set Up Kubernetes

5. How to Install kubectl

6. How to Set Up Vagrant

7. How to Install Ansible

8. How to Build Your First DevOps Project

9. How to Break Your Lab on Purpose

10. What You Can Now Do

What is DevOps?

DevOps is the practice of breaking down the wall between software development and IT operations teams.

Traditionally, developers write code and hand it off to operations teams to deploy and maintain. That handoff causes delays, misunderstandings, and outages. DevOps is what happens when both teams work together from the start.

The tools you'll install in this guide each solve a specific part of that process:

• Docker packages your application and everything it needs into a portable container that runs the same way on any machine.

• Kubernetes manages multiple containers at scale, handling restarts, networking, and load balancing automatically.

• Vagrant creates and manages virtual machine environments so your whole team always works on identical setups.

• Ansible automates repetitive configuration tasks across many servers without writing a script for each one.

Why Build a Local Lab?

A local lab gives you a safe place to break things, fix them, and learn from that process without any cost or risk.

Here's what you get with a local setup:

• Zero cost. No cloud bills, no surprise charges, and no credit card required.

• Works offline. Practice anywhere, even without internet after the initial setup.

• Full control. You manage every layer from the OS up to the application.

• Safe experimentation. Break things freely. Nothing here affects production.

• Fast feedback. No waiting for cloud resources to spin up. Everything runs on your machine.

The tradeoff is resource limits. Your laptop's CPU and RAM are the ceiling. You can't simulate large-scale deployments, and some cloud-native services like AWS Lambda or S3 have no direct local equivalent. But for learning core DevOps workflows, none of that matters.

How to Set Up Docker

Docker is the foundation of this lab. Every other tool in this guide either runs inside Docker containers or works alongside them.

How to Install Docker on Windows

First, enable virtualization in your BIOS:

1. Restart your computer and enter BIOS/UEFI setup. The key is usually F2, F10, Del, or Esc during boot.

2. Find the virtualization setting. It's usually listed as Intel VT-x, AMD-V, SVM, or Virtualization Technology.

3. Enable it, save your changes, and exit.

Then install Docker Desktop:

1. Download Docker Desktop from Docker's official website.

2. Run the installer and follow the prompts.

3. Enable WSL 2 (Windows Subsystem for Linux) when asked.

4. Restart your computer.

5. Open Docker Desktop from the Start menu and wait for the whale icon in the taskbar to stop animating.

Troubleshooting: If Docker fails to start, run this in PowerShell as Administrator to verify virtualization is active:

systeminfo | findstr "Hyper-V Requirements"

All items should show "Yes". If they don't, revisit your BIOS settings.

How to Install Docker on Mac

1. Download Docker Desktop for Mac from Docker's website.

2. Open the downloaded .dmg file and drag Docker to your Applications folder.

3. Open Docker from Applications.

4. Enter your password when prompted.

5. Wait for the whale icon in the menu bar to stop animating.

How to Install Docker on Linux

Run these commands in order:

# Update your package lists
sudo apt-get update

# Install prerequisites
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Add the Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Update and install Docker
sudo apt-get update
sudo apt-get install docker-ce

# Start and enable Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add your user to the docker group
sudo usermod -aG docker $USER

Log out and back in for the group change to take effect.

How to Test Docker

Run this command:

docker run hello-world

If you see "Hello from Docker!" then Docker is working correctly.

Docker is set up. Next, you'll install Kubernetes to manage containers at scale.

How to Set Up Kubernetes

Kubernetes manages containers at scale. For a local lab, you have four options. Here's how to choose:

If you're just starting out, use Minikube. It has the simplest setup and a visual dashboard that helps you understand what's happening inside the cluster.

If your laptop has 8GB RAM or less, use k3s. It runs lean and behaves closer to a real cluster than Minikube does.

Use kubeadm only if you want to understand how Kubernetes nodes join a cluster — it requires more manual steps and isn't beginner-friendly.

How to Install Minikube (Recommended for Beginners)

Minikube creates a single-node Kubernetes cluster on your laptop.

On Windows:

1. Download the Minikube installer from Minikube's GitHub releases page.

2. Run the .exe installer.

3. Open Command Prompt as Administrator and start Minikube:

minikube start --driver=docker

On Mac:

brew install minikube
minikube start --driver=docker

On Linux:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube-linux-amd64
sudo mv minikube-linux-amd64 /usr/local/bin/minikube
minikube start --driver=docker

Test your cluster:

minikube status
minikube dashboard

How to Install k3s (Recommended for Low-RAM Machines)

k3s is a lightweight version of Kubernetes that installs in under a minute. It runs lean and behaves like a real cluster — not a simplified demo version.

On Linux (and Mac via Multipass):

curl -sfL https://get.k3s.io | sh -

That single command installs k3s and runs it automatically in the background. Check that it is running:

sudo k3s kubectl get nodes

You should see one node with status Ready.

On Mac directly — k3s doesn't run natively on macOS. Use Multipass to spin up a lightweight Ubuntu VM first, then run the install command inside it.

On Windows — use WSL2 (Ubuntu), then run the install command inside your WSL2 terminal.

How to Install Kind (Kubernetes IN Docker)

Kind runs a full Kubernetes cluster inside Docker containers. It starts faster than Minikube and is useful if you want to run multiple clusters simultaneously.

# Mac or Linux
brew install kind

# Windows
choco install kind

Create a cluster:

kind create cluster --name my-local-lab

How to Install kubeadm (For Understanding Cluster Bootstrap)

kubeadm is the tool Kubernetes uses to initialize and join nodes in a real cluster. Use this when you want to understand what happens under the hood — not as your daily driver.

It requires at least two machines (or VMs). The setup is more involved than the options above. Follow the official kubeadm installation guide for your OS, then initialize your cluster:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

After init, join worker nodes using the command kubeadm prints at the end of the output.

How to Install kubectl

kubectl is the command-line tool you use to interact with any Kubernetes cluster.

On Windows:

Download kubectl.exe from Kubernetes' website and place it in a directory that is in your PATH. Or install with Chocolatey:

choco install kubernetes-cli

On Mac:

brew install kubectl

On Linux:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl

Test it:

kubectl get pods --all-namespaces

On a fresh cluster, you'll see system pods running in the kube-system namespace — things like coredns and storage-provisioner. That's the expected output. It means your cluster is up and kubectl can talk to it.

Kubernetes is running. Next is Vagrant. But before that, there's one important distinction worth making.

Docker vs Vagrant — they aren't the same thing

Docker creates containers: lightweight processes that share your operating system's kernel. Vagrant creates full virtual machines: isolated computers with their own OS running inside your laptop.

Containers are fast and small. VMs are heavier but behave exactly like real servers. You'll use both in this lab for different reasons.

How to Set Up Vagrant

Vagrant lets you create and manage reproducible virtual machine environments. It is ideal for simulating multi-server setups on a single laptop.

How to Install Vagrant on Windows

1. Download and install VirtualBox with default options.

2. Download and install Vagrant.

3. Restart your computer if prompted.

Note: VirtualBox and Hyper-V can't run at the same time on Windows. Check if Hyper-V is active:

systeminfo | findstr "Hyper-V"

If it's enabled, you have two options: switch to the Hyper-V Vagrant provider, or disable Hyper-V with:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All

Restart after disabling.

How to Install Vagrant on Mac and Linux

On Mac:

1. Download and install VirtualBox.

2. After installation, open System Preferences > Security & Privacy > General. You will see a message saying system software from Oracle was blocked. Click Allow and restart your Mac. Without this step, VirtualBox will not run.

3. Download and install Vagrant.

Note for Apple Silicon (M1/M2/M3) Macs: VirtualBox support on Apple Silicon is still limited. If you're on an M-series Mac, use UTM as your VM provider instead, or use Multipass which works natively on Apple Silicon.

On Linux:

1. Download and install VirtualBox.

2. Download and install Vagrant.

Verify both are installed:

vboxmanage --version
vagrant --version

How to Create Your First Vagrant Environment

Create a new directory for your project. Inside it, create a file named Vagrantfile with this content:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  # Create a private network between VMs
  config.vm.network "private_network", type: "dhcp"

  # Forward port 8080 on your laptop to port 80 on the VM
  config.vm.network "forwarded_port", guest: 80, host: 8080

  # Install Nginx when the VM starts
  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y nginx
    echo "Hello from Vagrant!" > /var/www/html/index.html
  SHELL
end

Start the VM:

vagrant up

Visit http://localhost:8080 in your browser. You should see "Hello from Vagrant!"

Troubleshooting SSH on Windows

If vagrant ssh fails, try:

vagrant ssh -- -v

Or connect manually:

ssh -i .vagrant/machines/default/virtualbox/private_key vagrant@127.0.0.1 -p 2222

How to Create a Local Vagrant Box Without Internet

Note: Most readers can skip this. Only do this if you want to work fully offline after the initial setup.

1. Download Ubuntu 20.04 LTS and save the .iso file locally.

2. Open VirtualBox and create a new VM: Name it ubuntu-devops, Type: Linux, Version: Ubuntu (64-bit).

3. Assign 2048MB RAM and a 20GB VDI disk.

4. Attach the .iso under Storage > Optical Drive.

5. Start the VM and complete the Ubuntu installation.

6. Once installed, shut down the VM and run:

VBoxManage list vms
vagrant package --base "ubuntu-devops" --output ubuntu2004.box
vagrant box add ubuntu2004 ubuntu2004.box

You now have a reusable local box that works without internet.

You can spin up virtual machines. Next is Ansible, which automates what goes inside them.

How to Install Ansible

Ansible automates configuration and software installation across multiple servers. Instead of SSH-ing into ten machines and running the same commands manually, you write a playbook once and Ansible handles the rest.

How to Install Ansible on Windows

Ansible doesn't run natively on Windows. You need to use it through WSL (Windows Subsystem for Linux).

1. Open PowerShell as Administrator and enable WSL:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

1. Restart your computer.

2. Install Ubuntu from the Microsoft Store.

3. Open Ubuntu and install Ansible:

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

How to Install Ansible on Mac

brew install ansible

How to Install Ansible on Linux

# Ubuntu/Debian
sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

# Red Hat/CentOS
sudo yum install ansible

How to Test Ansible

Create a file called hosts in your current directory:

[local]
localhost ansible_connection=local

Create a file called playbook.yml in the same directory:

---
- name: Test playbook
  hosts: local
  tasks:
    - name: Print a message
      debug:
        msg: "Ansible is working!"

Run the playbook, passing the local hosts file with -i:

ansible-playbook -i hosts playbook.yml

You should see the message "Ansible is working!" in the output.

Alright, all your tools are installed. Now you'll use them together to build something real.

How to Build Your First DevOps Project

You can find the entire code for this lab in this repo: https://github.com/Osomudeya/homelab-demo-article

Now you'll put these tools together in one project. Each tool will perform its actual job, and nothing is forced.

Before you start, create a fresh directory for this project. Don't run it inside the directory you used to test Vagrant earlier, as the Vagrantfile here is different and will conflict.

You'll be building a two-VM environment: one machine serves a web page you write yourself inside a Docker container, and the other runs a MariaDB database. Vagrant creates the machines and Ansible configures them. The page you see at the end is yours.

Step 1: Create the Project Directory

mkdir devops-lab-project && cd devops-lab-project

Step 2: Write Your Site Content

Create a file called index.html in the project directory. Write whatever you want on this page — it's what you'll see in your browser at the end:

<!DOCTYPE html>
<html>
  <head><title>My DevOps Lab</title></head>
  <body>
    <h1>My DevOps Lab</h1>
    <p>Provisioned by Vagrant. Configured by Ansible. Served by Docker.</p>
    <p>Built on a laptop. No cloud account needed.</p>
  </body>
</html>

Change the text to whatever you like. This is your page.

Step 3: Write the Vagrantfile

Create a file called Vagrantfile in the same directory:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  config.vm.define "web" do |web|
    web.vm.network "private_network", ip: "192.168.33.10"
    web.vm.network "forwarded_port", guest: 80, host: 8080
  end

  config.vm.define "db" do |db|
    db.vm.network "private_network", ip: "192.168.33.11"
  end
end

Step 4: Start the Virtual Machines

vagrant up

The first run downloads the ubuntu/focal64 box, which is around 500MB.

Expect this to take 10–30 minutes depending on your connection. Subsequent runs will be much faster since the box is cached locally.

Step 5: Create the Ansible Inventory

Create a file called inventory in the same directory:

[webservers]
192.168.33.10 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

[dbservers]
192.168.33.11 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/db/virtualbox/private_key

Ansible uses the Vagrant-generated private keys so it can SSH in as the vagrant user. Host key checking for this lab is turned off in ansible.cfg (next step), not in the inventory.

Step 6: Create the Ansible Config File

Before running the playbook, create a file called ansible.cfg in the same directory:

[defaults]
inventory = inventory
host_key_checking = False

The inventory line tells Ansible to use the inventory file in this folder by default. host_key_checking = False tells Ansible not to verify SSH host keys when connecting to your Vagrant VMs. Without it, Ansible will fail with a Host key verification failed error on first connection because the VM's key is not yet in your known_hosts file.

These settings are for a local lab only. Do not use host_key_checking = False for production systems.

Step 7: Create the Ansible Playbook

Create a file called playbook.yml:

---
- name: Configure web server
  hosts: webservers
  become: yes
  tasks:

    - name: Install Docker
      apt:
        name: docker.io
        state: present
        update_cache: yes

    - name: Start Docker service
      service:
        name: docker
        state: started
        enabled: yes

    # Create the directory that will hold your site content
    - name: Create web content directory
      file:
        path: /var/www/html
        state: directory
        mode: '0755'

    # This copies your index.html from your laptop into the VM
    - name: Copy site content to web server
      copy:
        src: index.html
        dest: /var/www/html/index.html

    # This mounts that file into the Nginx container so it serves your page
    # The -v flag connects /var/www/html on the VM to /usr/share/nginx/html inside the container
    - name: Run Nginx serving your content
      shell: |
        docker rm -f webapp 2>/dev/null || true
        docker run -d --name webapp --restart always -p 80:80 \
          -v /var/www/html:/usr/share/nginx/html:ro nginx

- name: Configure database server
  hosts: dbservers
  become: yes
  tasks:

    # Hash sum mismatch on .deb downloads is often stale lists, a flaky mirror, or apt pipelining
    # behind NAT; fresh indices + Pipeline-Depth 0 usually fixes it on lab VMs.
    - name: Disable apt HTTP pipelining (mirror/proxy hash mismatch workaround)
      copy:
        dest: /etc/apt/apt.conf.d/99disable-pipelining
        content: 'Acquire::http::Pipeline-Depth "0";'
        mode: "0644"

    - name: Clear apt package index cache
      shell: apt-get clean && rm -rf /var/lib/apt/lists/* /var/lib/apt/lists/auxfiles/*
      changed_when: true

    - name: Update apt cache after reset
      apt:
        update_cache: yes

    - name: Install MariaDB
      apt:
        name: mariadb-server
        state: present
        update_cache: no

    - name: Start MariaDB service
      service:
        name: mariadb
        state: started
        enabled: yes

Two lines worth paying attention to:

• src: index.html — Ansible looks for this file in the same directory as the playbook. That is the file you wrote in Step 2.

• -v /var/www/html:/usr/share/nginx/html:ro — this mounts the directory from the VM into the Nginx container. The :ro means read-only. Nginx serves whatever is in that folder.

Step 8: Run the Playbook

ansible-playbook -i inventory playbook.yml

You'll see task-by-task output as Ansible connects to each VM over SSH and configures it. A green ok or yellow changed next to each task means it worked. Red fatal means something failed.

Step 9: Verify the Setup

Open http://localhost:8080 in your browser. You should see the page you wrote in Step 2 served from inside a Docker container, running on a Vagrant VM, configured automatically by Ansible.

If you see the page, every tool in this lab is working together.

Step 9: Clean Up (Optional)

When you're done:

vagrant destroy -f

This shuts down and deletes both VMs. Your Vagrantfile, inventory, playbook.yml, and index.html stay on disk — run vagrant up followed by ansible-playbook -i inventory playbook.yml any time to bring it all back.

Now that you have a working lab, let's use it properly.

How to Break Your Lab on Purpose

Following these steps has gotten you a running lab. Breaking things teaches you how everything actually works.

Here are five things to break and what to look for when you do.

Break 1: Crash the Main Process Inside the Container (and Watch It Come Back)

Doing this just proves that something inside the container can die (like a real bug or OOM), Docker can restart the container because of --restart always, and your site can come back without re-running Ansible.

After vagrant ssh web, every docker command below runs on the web VM. So keep your browser on your laptop at http://localhost:8080 (Vagrant forwards your host port to the VM’s port 80).

Troubleshooting: If Your Lab Isn't Ready

From your project folder on the host (your laptop) – unless the step says to run it on the VM:

• You ran vagrant destroy -f. Run vagrant up, then ansible-playbook -i inventory playbook.yml.

• docker ps shows webapp but status is Exited. On the web VM, run sudo docker start webapp, then sudo docker ps again.

• There's no webapp row in docker ps -a. Re-run ansible-playbook -i inventory playbook.yml on the host.

If the playbook is already applied and webapp is Up, skip this section and start at step 1 under Steps (happy path) below. (Don't skip SSH or docker ps. You need the VM shell and a quick check before you run docker exec.)

Steps (happy path)

1. SSH into the web VM:

vagrant ssh web

1. Confirm webapp is Up:

   sudo docker ps
   

2. Break it on purpose: kill the container’s main process from inside (PID 1). That ends the container the same way a crashing app would, not the same as docker stop on the host:

sudo docker exec webapp sh -c 'sleep 5 && kill 1'

The sleep 5 gives you a moment to switch to the browser. Right after you run the command, open or refresh http://localhost:8080. You may catch a brief error or blank page while nothing is listening on port 80.

1. Watch Docker restart the container:

watch sudo docker ps -a

Within a few seconds you should see Exited (137) become Up again. (Press Ctrl+C to exit watch.)

5. Refresh the browser. You should see the same HTML as before, because the files live on the VM under /var/www/html and are bind-mounted into the container; restarting only replaced the Nginx process, not those files.

Why not docker stop or docker kill on the host for this demo?

Those commands go through Docker’s API. On many setups (including recent Docker), Docker treats them as you choosing to stop the container (hasBeenManuallyStopped), and --restart always may not bring the container back until you docker start it or similar.

Killing PID 1 from inside the container is treated more like an internal crash, so the restart policy you set in the playbook is the one you actually get to observe here.

Kubernetes analogy: A pod whose containers exit can be restarted by the kubelet; a pod you delete does not come back by itself.

What to observe (three separate checks):

1. Exit code: After kill 1, docker ps -a should show the container exited with code 137, meaning the main process was killed by a signal. That confirms the container really died, not that you ran docker stop on the host.

2. Restart delay vs browser: Watch how many seconds pass between Exited and Up in docker ps -a; that interval is Docker applying --restart always. That's separate from what you see in the browser: the browser only shows whether something is accepting connections on port 80 on the VM, so it may show an error or blank page during the gap even while Docker is about to restart the container.

3. Content after recovery: After status is Up again, refresh the page. You should see the same HTML as before. That shows your content lives on the VM disk (mounted into the container with -v), not inside a file that vanishes when the container process restarts. The process was replaced, not your index.html on the host path.

Break 2: Cause a Container Name Conflict

On a single Docker daemon (here, on your web VM), a container name is a unique label. Two running (or stopped) containers can't share the same name. Scripts and playbooks that always use docker run --name webapp without cleaning up first hit this error constantly and recognizing it saves time in real work.

Before you start: Ansible already created one container named webapp.
Stay on the web VM (for example still inside vagrant ssh web) so the commands below run where that container lives.

So now, try to start a second container and also call it webapp. The image is plain nginx here on purpose – the point is the name clash, not matching your site’s ports or volume mounts.

sudo docker run -d --name webapp nginx

What actually happens here is that Docker doesn't create a second container. It returns an error immediately. Your original webapp is unchanged.

This is because the name webapp is already registered to the existing container (the error shows that container’s ID). Docker refuses to reuse the name until the old container is removed or renamed.

Example error (your ID will differ):

docker: Error response from daemon: Conflict. The container name "/webapp" is already in use by container "2e48b81a311c4b71cdc1e25e0df75a22296845c7eb53aab82f9ae739fb6410ec". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

To fix it, free the name, then create webapp again the same way the playbook does (publish port 80, mount your HTML, restart policy):

sudo docker rm -f webapp
sudo docker run -d --name webapp --restart always -p 80:80 \
  -v /var/www/html:/usr/share/nginx/html:ro nginx

After that, your site should behave as before (refresh http://localhost:8080 from your laptop).

What to observe:

Read Docker’s Conflict message end to end. You should see that the name /webapp is already in use and a container ID pointing at the existing box. In production, that pattern means “something already claimed this name. Just remove it, rename it, or pick a different name before you run docker run again.”

Break 3: Make Ansible Fail to Reach a VM

Ansible separates “could not connect” from “connected, but a task broke.” The first is UNREACHABLE, the second is FAILED. Knowing which one you have tells you whether to fix network / SSH or playbook / packages / permissions.

On your laptop, in the project folder, edit inventory and change the web server address from 192.168.33.10 to an IP no VM uses, for example 192.168.33.99. Save the file.

[webservers]
192.168.33.99 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

What you run (from the same project folder on the host):

ansible-playbook -i inventory playbook.yml

After this, Ansible tries to SSH to 192.168.33.99. Nothing on your lab network answers as that host (or SSH never succeeds), so Ansible never runs tasks on the web server. It stops that host with UNREACHABLE:

fatal: [192.168.33.99]: UNREACHABLE! => {"msg": "Failed to connect to the host via ssh"}

This is realistic because the same message shape appears when the IP is wrong, the VM isn't running, a firewall blocks port 22, or the network is misconfigured. The common thread is no working SSH session.

Now it's time to put it back: restore 192.168.33.10 in inventory and run ansible-playbook -i inventory playbook.yml again. The web play should reach the VM and complete (assuming your lab is up).

UNREACHABLE vs FAILED – what to observe:

• If Ansible prints UNREACHABLE, you should assume it never opened SSH on that host and never ran tasks there. Go ahead and fix the connection (IP, VM up, firewall, key path) before you debug playbook logic.

• If Ansible prints FAILED, you should assume SSH worked and a task returned an error. Read the task output for the real cause (package name, permissions, syntax), not the network first.

When you debug later, you should look at the keyword Ansible prints: UNREACHABLE points to reachability while FAILED points to task output and the first failed task under that host.

Break 4: Fill the VM's Disk

Databases and other services need free disk for logs, temp files, and data. When the filesystem is full or nearly full, a service may fail to start or fail at runtime. This break walks through the same diagnosis habit you would use on a real server: check space, then read systemd and journal output for the service.

All commands below run on the db VM after vagrant ssh db. MariaDB was installed there by your playbook.

What you do:

1. Open a shell on the db VM:

   vagrant ssh db
   

2. Allocate a large file full of zeros (here 1GB) to simulate something eating disk space:

   sudo dd if=/dev/zero of=/tmp/bigfile bs=1M count=1024
   
   df -h
   

   Use df -h to see how full the root filesystem (or relevant mount) is. Your Vagrant disk may be large enough that 1GB only raises usage. If MariaDB still starts, you still practiced the checks. To see a stronger effect, you can repeat with a larger count= only in a lab (never fill production disks on purpose without a plan).

3. Ask systemd to restart MariaDB and show status:

   sudo systemctl restart mariadb
   sudo systemctl status mariadb
   

   If the disk is critically full, restart may fail or the service may show failed or not running.

4. If something looks wrong, read recent logs for the MariaDB unit:

   sudo journalctl -u mariadb --no-pager | tail -20
   

   Errors often mention disk, space, read-only filesystem, or InnoDB being unable to write.

5. Clean up so your VM stays usable:

   sudo rm /tmp/bigfile
   

   Optionally run sudo systemctl restart mariadb again and confirm it is active (running).

What to observe:

• You should use df -h first to confirm whether the filesystem is actually tight. That avoids blaming the database when disk space is fine.

• You should read systemctl status mariadb to see whether systemd thinks the service is active, failed, or flapping.

• You should read journalctl -u mariadb when status is bad, so you can tie the failure to concrete errors from MariaDB or the kernel (often mentioning disk, space, or read-only filesystem). Space + status + logs is the same order you would use on a production server.

Break 5: Run Minikube Out of Resources

Kubernetes schedules pods onto nodes that have enough CPU and memory. If you ask for more than the cluster can place, some pods stay Pending and Events explain why (for example Insufficient cpu). That is not the same as a pod that starts and then crashes.

To do this, you'll need a local cluster (we're using Minikube in this guide) and kubectl on your laptop. This break doesn't use the Vagrant VMs. If you haven't installed Minikube yet, complete the "How to Set Up Kubernetes" section first, or skip this break until you do.

You'll run this on your Mac, Linux, or Windows terminal (host), not inside vagrant ssh. If you're still inside a VM, type exit until your prompt is back on the host.

What you do:

1. Check Minikube:

   minikube status
   

   If it's stopped, start it (Docker driver matches earlier sections):

   minikube start --driver=docker
   

2. Create a deployment with many replicas so your single Minikube node can't run them all at once:

   kubectl create deployment stress --image=nginx --replicas=20
   
   #watch pods start
   kubectl get pods -w
   

   Press Ctrl+C when you're done watching. Some pods may stay Pending while others are Running.

3. Pick one Pending pod name from kubectl get pods and inspect it:

   kubectl describe pod <pod-name>
   

   Under Events, look for FailedScheduling and a line similar to:

   Warning  FailedScheduling  0/1 nodes are available: 1 Insufficient cpu.
   

   You might see Insufficient memory instead, depending on your machine.

4. Fix the lab by scaling back so the cluster can catch up:

   kubectl scale deployment stress --replicas=2
   

   You can delete the deployment entirely when finished: kubectl delete deployment stress.

What to observe:

• You should see Pending pods stay unscheduled until capacity frees up. That means the scheduler hasn't placed them on any node yet, usually because the node is out of CPU or memory for that workload.

• You should read kubectl describe pod <pod-name> and scroll to Events. Messages like Insufficient cpu or Insufficient memory mean the cluster ran out of schedulable capacity, not that the container image image is corrupt.

• You should contrast that with a pod that reaches Running and then CrashLoopBackOff, which usually means the process inside the container keeps exiting. that is an application or config problem, not a “nowhere to run” problem.

What You Can Now Do

You didn't just install tools in this tutorial. You also used them.

You can now spin up two servers from a single file. You can write a playbook that installs software and deploys a container without touching either machine manually.

You can serve a page you wrote from inside a Docker container running on a Vagrant VM, and bring the whole thing back from scratch in one command.

You also broke it. You saw what a container conflict looks like, what Ansible prints when it can't reach a machine, what disk pressure does to a running service, and what a Kubernetes scheduler says when it runs out of resources. Those error messages aren't unfamiliar anymore.

That's the difference between someone who has read about DevOps and someone who has run it.

Here are four free projects you can run in this same lab to go further:

• DevOps Home-Lab 2026 — Build a multi-service app (frontend, API, PostgreSQL, Redis) end-to-end with Docker Compose, Kubernetes, Prometheus/Grafana monitoring, GitOps with ArgoCD, and Cloudflare for global exposure.

• KubeLab — Trigger real Kubernetes failure scenarios, pod crashes, OOMKills, node drains, cascading failures, and watch how the cluster responds using live metrics.

• K8s Secrets Lab — Build a full secret management pipeline from AWS Secrets Manager into your cluster, including rotation behavior and IRSA.

• DevOps Troubleshooting Toolkit — Structured debugging guides across Linux, containers, Kubernetes, cloud, databases, and observability with copy-paste commands for real incidents.

All free and open source: github.com/Osomudeya/List-Of-DevOps-Projects.

If you want to go deeper, you can find six full chapters covering Terraform, Ansible, monitoring, CI/CD, and a simulated three-VM production environment at Build Your Own DevOps Lab.

Google Cloud Axion is Google's own custom ARM-based chip, built to handle the demands of modern cloud workloads. The performance and cost numbers are striking: Google claims Axion delivers up to 60% better energy efficiency and up to 65% better price-performance compared to comparable x86 machines.

AWS has Graviton. Azure has Cobalt. ARM is no longer niche. It's the direction the entire cloud industry is moving.

But there's a problem that catches almost every team off guard when they start this transition: container architecture mismatch.

If you build a Docker image on your M-series Mac and push it to an x86 server, it crashes on startup with a cryptic exec format error.

The server isn't broken. It just can't read the compiled instructions inside your image. An ARM binary and an x86 binary are written in fundamentally different languages at the machine level. The CPU literally can't execute instructions it wasn't designed for.

We're going to solve this problem completely in this tutorial. You'll build a single Docker image tag that automatically serves the correct binary on both ARM and x86 machines — no separate pipelines, no separate tags. Then you'll provision Google Cloud ARM nodes in GKE and configure your Kubernetes deployment to route workloads precisely to those cost-efficient nodes.

Here's what you'll build, step by step:

• A Go HTTP server that reports the CPU architecture it's running on at runtime

• A multi-stage Dockerfile that cross-compiles for both linux/amd64 and linux/arm64 without slow QEMU emulation

• A multi-arch image in Google Artifact Registry that acts as a single entry point for any architecture

• A GKE cluster with two node pools: a standard x86 pool and an ARM Axion pool

• A Kubernetes Deployment that pins your workload exclusively to the ARM nodes

By the end, you'll hit a live endpoint and see the word arm64 staring back at you from a Google Cloud ARM node. Let's get into it.

Table of Contents

• Prerequisites

• Step 1: Set Up Your Google Cloud Project

• Step 2: Create the GKE Cluster

• Step 3: Write the Application

• Step 4: Enable Multi-Arch Builds with Docker Buildx

• Step 5: Write the Dockerfile

• Step 6: Build and Push the Multi-Arch Image

• Step 7: Add the Axion ARM Node Pool

• Step 8: Deploy the App to the ARM Node Pool

• Step 9: Verify the Deployment

• Step 10: Cost Savings and Tradeoffs

• Cleanup

• Conclusion

• Project File Structure

Prerequisites

Before you start, make sure you have the following ready:

• A Google Cloud project with billing enabled. If you don't have one, create it at console.cloud.google.com. The total cost to follow this tutorial is around $5–10.

• gcloud CLI installed and authenticated. Run gcloud auth login to sign in and gcloud config set project YOUR_PROJECT_ID to point it at your project.

• Docker Desktop version 19.03 or later. Docker Buildx (the tool we'll use for multi-arch builds) ships bundled with it.

• kubectl installed. This is the CLI for interacting with Kubernetes clusters.

• Basic familiarity with Docker (images, layers, Dockerfile) and Kubernetes (pods, deployments, services). You don't need to be an expert, but you should know what these things are.

Step 1: Set Up Your Google Cloud Project

Before writing a single line of application code, let's get the cloud infrastructure side ready. This is the foundation everything else will build on.

Enable the Required APIs

Google Cloud services are off by default in any new project. Run this command to turn on the three APIs we'll need:

gcloud services enable \
  artifactregistry.googleapis.com \
  container.googleapis.com \
  containeranalysis.googleapis.com

Here's what each one does:

• artifactregistry.googleapis.com — enables Artifact Registry, where we'll store our Docker images

• container.googleapis.com — enables Google Kubernetes Engine (GKE), where our cluster will run

• containeranalysis.googleapis.com — enables vulnerability scanning for images stored in Artifact Registry

Create a Docker Repository in Artifact Registry

Artifact Registry is Google Cloud's managed container image store — the place where our built images will live before being deployed to the cluster. Create a dedicated repository for this tutorial:

gcloud artifacts repositories create multi-arch-repo \
  --repository-format=docker \
  --location=us-central1 \
  --description="Multi-arch tutorial images"

Breaking down the flags:

• --repository-format=docker — tells Artifact Registry this repository stores Docker images (as opposed to npm packages, Maven artifacts, and so on)

• --location=us-central1 — the Google Cloud region where your images will be stored. Use a region that's close to where your cluster will run to minimize image pull latency. Run gcloud artifacts locations list to see all options.

• --description — a human-readable label for the repository, shown in the console.

Authenticate Docker to Push to Artifact Registry

Docker needs credentials before it can push images to Google Cloud. Run this command to wire up authentication automatically:

gcloud auth configure-docker us-central1-docker.pkg.dev

This adds a credential helper entry to your ~/.docker/config.json file. What that means in practice: any time Docker tries to push or pull from a URL under us-central1-docker.pkg.dev, it will automatically call gcloud to get a valid auth token. You won't need to run docker login manually.

Step 2: Create the GKE Cluster

With Artifact Registry ready to receive images, let's create the Kubernetes cluster. We'll start with a standard cluster using x86 nodes and add an ARM node pool later once we have an image to deploy.

gcloud container clusters create axion-tutorial-cluster \
  --zone=us-central1-a \
  --num-nodes=2 \
  --machine-type=e2-standard-2 \
  --workload-pool=PROJECT_ID.svc.id.goog

Replace PROJECT_ID with your actual Google Cloud project ID.

What each flag does:

• --zone=us-central1-a — creates a zonal cluster in a single availability zone. A regional cluster (using --region) would spread nodes across three zones for higher resilience, but for this tutorial a single zone keeps things simple and avoids capacity issues that can affect specific zones. If us-central1-a is unavailable, try us-central1-b.

• --num-nodes=2 — two x86 nodes in this zone. We need at least 2 to have enough capacity alongside our ARM node pool later.

• --machine-type=e2-standard-2 — the machine type for this default node pool. e2-standard-2 is a cost-effective x86 machine with 2 vCPUs and 8 GB of memory, good for general workloads.

• --workload-pool=PROJECT_ID.svc.id.goog — enables Workload Identity, which is Google's recommended way for pods to authenticate with Google Cloud APIs. It avoids the need to download and store service account key files inside your cluster.

This command takes a few minutes. While it runs, you can move on to writing the application. We'll come back to the cluster in Step 6.

Step 3: Write the Application

We need an application to containerize. We'll use Go for three specific reasons:

1. Go compiles into a single, statically-linked binary. There's no runtime to install, no interpreter — just the binary. This makes for extremely lean container images.

2. Go has first-class, built-in cross-compilation support. We can compile an ARM64 binary from an x86 Mac, or vice versa, by setting two environment variables. This will matter a lot when we get to the Dockerfile.

3. Go exposes the architecture the binary was compiled for via runtime.GOARCH. Our server will report this at runtime, giving us hard proof that the correct binary is running on the correct hardware.

Start by creating the project directories:

mkdir -p hello-axion/app hello-axion/k8s
cd hello-axion/app

Initialize the Go module from inside app/. This creates go.mod in the current directory:

go mod init hello-axion

go mod init is Go's built-in command for starting a new module. It writes a go.mod file that declares the module name (hello-axion) and the minimum Go version required. Every modern Go project needs this file — without it, the compiler doesn't know how to resolve packages.

Now create the application at app/main.go:

package main

import (
    "fmt"
    "net/http"
    "os"
    "runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
    hostname, _ := os.Hostname()
    fmt.Fprintf(w, "Hello from freeCodeCamp!\n")
    fmt.Fprintf(w, "Architecture : %s\n", runtime.GOARCH)
    fmt.Fprintf(w, "OS           : %s\n", runtime.GOOS)
    fmt.Fprintf(w, "Pod hostname : %s\n", hostname)
}

func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintln(w, "ok")
}

func main() {
    http.HandleFunc("/", handler)
    http.HandleFunc("/healthz", healthz)
    fmt.Println("Server starting on port 8080...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Fprintf(os.Stderr, "server error: %v\n", err)
        os.Exit(1)
    }
}

Verify both files were created:

ls -la

You should see go.mod and main.go listed.

Let's walk through what this code does:

• import "runtime" — imports Go's built-in runtime package, which exposes information about the Go runtime environment, including the CPU architecture.

• runtime.GOARCH — returns a string like "arm64" or "amd64" representing the architecture this binary was compiled for. When we deploy to an ARM node, this value will be arm64. This is the core of our proof.

• os.Hostname() — returns the pod's hostname, which Kubernetes sets to the pod name. This lets us see which specific pod responded when we test the app later.

• handler — the main HTTP handler, registered on the root path /. It writes the architecture, OS, and hostname to the response.

• healthz — a separate handler registered on /healthz. It returns HTTP 200 with the text ok. Kubernetes will use this endpoint to check whether the container is alive and ready to serve traffic — we'll wire this up in the deployment manifest later.

• http.ListenAndServe(":8080", nil) — starts the server on port 8080. If it fails to start (for example, if the port is already in use), it prints the error and exits with a non-zero code so Kubernetes knows something went wrong.

Step 4: Enable Multi-Arch Builds with Docker Buildx

Before we write the Dockerfile, we need to understand a fundamental constraint, because it directly shapes how the Dockerfile must be written.

Why Your Docker Images Are Architecture-Specific By Default

A CPU only understands instructions written for its specific Instruction Set Architecture (ISA). ARM64 and x86_64 are different ISAs — different vocabularies of machine-level operations. When you compile a Go program, the compiler translates your source code into binary instructions for exactly one ISA. That binary can't run on a different ISA.

When you build a Docker image the normal way (docker build), the binary inside that image is compiled for your local machine's ISA. If you're on an Apple Silicon Mac, you get an ARM64 binary. Push that image to an x86 server, and when Docker tries to execute the binary, the kernel rejects it:

standard_init_linux.go:228: exec user process caused: exec format error

That's the operating system saying: "This binary was written for a different processor. I have no idea what to do with it."

The Solution: A Single Image Tag That Serves Any Architecture

Docker solves this with a structure called a Manifest List (also called a multi-arch image index). Instead of one image, a Manifest List is a pointer table. It holds multiple image references — one per architecture — all under the same tag.

When a server pulls hello-axion:v1, here's what actually happens:

1. Docker contacts the registry and requests the manifest for hello-axion:v1

2. The registry returns the Manifest List, which looks like this internally:

{
  "manifests": [
    { "digest": "sha256:a1b2...", "platform": { "architecture": "amd64", "os": "linux" } },
    { "digest": "sha256:c3d4...", "platform": { "architecture": "arm64", "os": "linux" } }
  ]
}

1. Docker checks the current machine's architecture, finds the matching entry, and pulls only that specific image layer. The x86 image never downloads onto your ARM server, and vice versa.

One tag, two actual images. Completely transparent to your deployment manifests.

Set Up Docker Buildx

Docker Buildx is the CLI tool that builds these Manifest Lists. It's powered by the BuildKit engine and ships bundled with Docker Desktop. Run the following to create and activate a new builder instance:

docker buildx create --name multiarch-builder --use

• --name multiarch-builder — gives this builder a memorable name. You can have multiple builders. This command creates a new one named multiarch-builder.

• --use — immediately sets this new builder as the active one, so all future docker buildx build commands use it.

Now boot the builder and confirm it supports the platforms we need:

docker buildx inspect --bootstrap

• --bootstrap — starts the builder container if it isn't already running, and prints its full configuration.

You should see output like this:

Name:          multiarch-builder
Driver:        docker-container
Platforms:     linux/amd64, linux/arm64, linux/arm/v7, linux/386, ...

The Platforms line lists every architecture this builder can produce images for. As long as you see linux/amd64 and linux/arm64 in that list, you're ready to build for both x86 and ARM.

Step 5: Write the Dockerfile

Now we can write the Dockerfile. We'll use two techniques together: a multi-stage build to keep the final image tiny, and a cross-compilation trick to avoid slow CPU emulation.

Create app/Dockerfile with the following content:

# -----------------------------------------------------------
# Stage 1: Build
# -----------------------------------------------------------
# $BUILDPLATFORM = the machine running this build (your laptop)
# \(TARGETOS / \)TARGETARCH = the platform we are building FOR
# -----------------------------------------------------------
FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

ARG TARGETOS
ARG TARGETARCH

WORKDIR /app

COPY go.mod .
RUN go mod download

COPY main.go .

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

# -----------------------------------------------------------
# Stage 2: Runtime
# -----------------------------------------------------------

FROM alpine:latest

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

WORKDIR /app
COPY --from=builder /app/server .

EXPOSE 8080
CMD ["./server"]

There's a lot happening here. Let's go through it carefully.

Stage 1: The Builder

FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

This is the most important line in the file. \(BUILDPLATFORM is a special build argument that Docker Buildx automatically injects — it equals the platform of the machine running the build (your laptop). By pinning the builder stage to \)BUILDPLATFORM, the Go compiler always runs natively on your machine, not inside a CPU emulator. This is what makes multi-arch builds fast.

Without --platform=$BUILDPLATFORM, Buildx would have to use QEMU — a full CPU emulator — to run an ARM64 build environment on your x86 machine (or vice versa). QEMU works, but it's typically 5–10 times slower than native execution. For a project with many dependencies, that's the difference between a 2-minute build and a 20-minute build.

ARG TARGETOS and ARG TARGETARCH

These two lines declare that our Dockerfile expects build arguments named TARGETOS and TARGETARCH. Buildx injects these automatically based on the --platform flag you pass at build time. For a linux/arm64 target, TARGETOS will be linux and TARGETARCH will be arm64.

COPY go.mod . and RUN go mod download

We copy go.mod first, before copying the rest of the source code. Docker builds images layer by layer and caches each layer. By copying only the module file first, we create a cached layer for go mod download.

On future builds, as long as go.mod hasn't changed, Docker skips the download step entirely — even if the source code changed. This speeds up iterative development significantly.

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

This is the cross-compilation step. GOOS and GOARCH are Go's built-in cross-compilation environment variables. Setting them tells the Go compiler to produce a binary for a different target than the machine it's running on. We set them from the \(TARGETOS and \)TARGETARCH build args injected by Buildx.

The -ldflags="-w -s" flag strips the debug symbol table and the DWARF debugging information from the binary. This has no effect on runtime behavior but reduces the binary size by roughly 30%.

Stage 2: The Runtime Image

FROM alpine:latest

This starts a brand-new image from Alpine Linux — a minimal Linux distribution that weighs about 5 MB. Critically, alpine:latest is itself a multi-arch image, so Docker automatically selects the arm64 or amd64 Alpine variant depending on which platform this stage is built for.

Everything from Stage 1 — the Go toolchain, the source files, the intermediate object files — is discarded. The final image contains only Alpine Linux plus our binary. Compared to a naive single-stage Go image (~300 MB), this approach produces an image under 15 MB.

RUN addgroup -S appgroup && adduser -S appuser -G appgroup and USER appuser

These two lines create a non-root user and set it as the active user for the container. Running containers as root is a security risk — if an attacker exploits a vulnerability in your application, they gain root access inside the container. Running as a non-root user limits the blast radius.

COPY --from=builder /app/server .

This is how multi-stage builds work: the --from=builder flag tells Docker to copy files from the builder stage (Stage 1), not from your local disk. Only the compiled binary (server) makes it into the final image.

Step 6: Build and Push the Multi-Arch Image

With the application and Dockerfile in place, we can now build images for both architectures and push them to Artifact Registry — all in a single command.

From inside the app/ directory, run:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 \
  --push \
  .

Replace PROJECT_ID with your actual GCP project ID.

Here's what each part of this command does:

• docker buildx build — uses the Buildx CLI instead of the standard docker build. Buildx is required for multi-platform builds.

• --platform linux/amd64,linux/arm64 — instructs Buildx to build the image twice: once targeting x86 Intel/AMD machines, and once targeting ARM64. Both builds run in parallel. Because our Dockerfile uses the $BUILDPLATFORM cross-compilation trick, both builds run natively on your machine without QEMU emulation.

• -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 — the full image path in Artifact Registry. The format is always REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME:TAG.

• --push — multi-arch images can't be loaded into your local Docker daemon (which only understands single-architecture images). This flag tells Buildx to skip local storage and push the completed Manifest List — with both architecture variants — directly to the registry.

• . — the build context, the directory Docker scans for the Dockerfile and any files the build needs.

Watch the output as the build runs. You'll see BuildKit working on both platforms simultaneously:

 => [linux/amd64 builder 1/5] FROM golang:1.23-alpine
 => [linux/arm64 builder 1/5] FROM golang:1.23-alpine
 ...
 => pushing manifest for us-central1-docker.pkg.dev/.../hello-axion:v1

Verify the Multi-Arch Image in Artifact Registry

Once the push completes, navigate to GCP Console → Artifact Registry → Repositories → multi-arch-repo and click on hello-axion.

You won't see a single image — you'll see something labelled "Image Index". That's the Manifest List we created. Click into it, and you'll find two child images with separate digests, one for linux/amd64 and one for linux/arm64.

You can also inspect this from the command line:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

The output lists every manifest inside the image index. You'll see entries for linux/amd64 and linux/arm64 — those are our two real images. You'll also see two entries with Platform: unknown/unknown labelled as attestation-manifest. These are build provenance records that Docker Buildx automatically attaches to prove how and where the image was built (a supply chain security feature called SLSA attestation).

The two entries you care about are linux/amd64 and linux/arm64. Note the digest for the arm64 entry — we'll use it in the verification step to confirm the cluster pulled the right variant.

Step 7: Add the Axion ARM Node Pool

We have a universal image. Now we need somewhere to run it.

Recall the cluster we created in Step 2 — it's running e2-standard-2 x86 machines. We're going to add a second node pool running ARM machines. This is the key architectural move: a mixed-architecture cluster where different workloads can be routed to different hardware.

Choosing Your ARM Machine Type

Google Cloud currently offers two ARM-based machine series in GKE:

This tutorial uses t2a-standard-2 because it's widely available. The commands are identical for c4a-standard-2 — just swap the --machine-type value. If t2a-standard-2 isn't available in your zone, GKE will tell you immediately when you run the node pool creation command below, and you can try a neighbouring zone.

Create the ARM Node Pool

Add the ARM node pool to your existing cluster:

gcloud container node-pools create axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a \
  --machine-type=t2a-standard-2 \
  --num-nodes=2 \
  --node-labels=workload-type=arm-optimized

What each flag does:

• --cluster=axion-tutorial-cluster — the name of the cluster we created in Step 2. Node pools are always added to an existing cluster.

• --zone=us-central1-a — must match the zone you used when creating the cluster.

• --machine-type=t2a-standard-2 — GKE detects this is an ARM machine type and automatically provisions the nodes with an ARM-compatible version of Container-Optimized OS (COS). You don't need to configure anything special for ARM at the OS level.

• --num-nodes=2 — two ARM nodes in the zone, enough to schedule our 3-replica deployment alongside other cluster overhead.

• --node-labels=workload-type=arm-optimized — attaches a custom label to every node in this pool. We'll use this label in our deployment manifest to target these specific nodes. Using a descriptive custom label (rather than just relying on the automatic kubernetes.io/arch=arm64 label) is good practice in real clusters — it communicates the intent of the pool, not just its hardware.

This command takes a few minutes. Once it completes, let's confirm our cluster now has both node pools:

gcloud container clusters get-credentials axion-tutorial-cluster --zone=us-central1-a

kubectl get nodes --label-columns=kubernetes.io/arch

The get-credentials command configures kubectl to authenticate with your new cluster. The get nodes command then lists all nodes and adds a column showing the kubernetes.io/arch label.

You should see something like:

NAME                                    STATUS   ARCH    AGE
gke-...default-pool-abc...              Ready    amd64   15m
gke-...default-pool-def...              Ready    amd64   15m
gke-...axion-pool-jkl...                Ready    arm64   3m
gke-...axion-pool-mno...                Ready    arm64   3m

amd64 for the default x86 pool, arm64 for our new Axion pool. This kubernetes.io/arch label is applied automatically by GKE — you don't set it, it's derived from the hardware.

Step 8: Deploy the App to the ARM Node Pool

We have a multi-arch image and a mixed-architecture cluster. Here's something important to understand before writing the deployment manifest: Kubernetes doesn't know or care about image architecture by default.

If you applied a standard Deployment right now, the scheduler would look for any available node with enough CPU and memory and place pods there — potentially landing on x86 nodes instead of your ARM Axion nodes. The multi-arch Manifest List would handle this gracefully (the right binary would run regardless), but you'd lose the cost efficiency you provisioned Axion nodes for in the first place.

To guarantee that pods land on ARM nodes and only ARM nodes, we use a nodeSelector.

How nodeSelector Works

A nodeSelector is a set of key-value pairs in your pod spec. Before the Kubernetes scheduler places a pod, it checks every available node's labels. If a node doesn't have all the labels in the nodeSelector, the scheduler skips it — the pod will remain in Pending state rather than land on the wrong node.

This is a hard constraint, which is exactly what we want for cost optimization. Contrast this with Node Affinity's soft preference mode (preferredDuringSchedulingIgnoredDuringExecution), which says "try to use ARM, but fall back to x86 if needed." Soft preferences are useful for resilience, but they undermine the whole point of dedicated ARM pools. We want the hard constraint.

Write the Deployment Manifest

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-axion
  labels:
    app: hello-axion
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-axion
  template:
    metadata:
      labels:
        app: hello-axion
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64

      containers:
      - name: hello-axion
        image: us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
        resources:
          requests:
            cpu: "250m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

Replace PROJECT_ID with your project ID. Here's what the key sections do:

replicas: 3 — tells Kubernetes to keep three instances of this pod running at all times. If one crashes or a node goes down, the scheduler spins up a replacement. Three replicas also means one pod per ARM node in us-central1, which distributes load across availability zones.

selector.matchLabels and template.metadata.labels — these two blocks must match. The selector tells the Deployment which pods it "owns," and the template.metadata.labels is what those pods will be tagged with. If they don't match, Kubernetes won't be able to manage the pods.

nodeSelector: kubernetes.io/arch: arm64 — this is the pin. The Kubernetes scheduler filters out every node that doesn't carry this label before considering resource availability. Since GKE automatically applies kubernetes.io/arch=arm64 to all ARM nodes, our pods will schedule only onto the axion-pool nodes.

livenessProbe — periodically calls GET /healthz. If this check fails a certain number of times in a row (indicating the container has deadlocked or is otherwise unresponsive), Kubernetes restarts the container. initialDelaySeconds: 5 gives the server 5 seconds to start up before the first check.

readinessProbe — similar to the liveness probe, but with a different purpose. While the readiness probe is failing, Kubernetes removes the pod from the service's load balancer, so no traffic is sent to it. This is important during startup — the pod won't receive traffic until it signals it's ready.

resources.requests — reserves 250m (25% of a CPU core) and 64Mi of memory on the node for this pod. The scheduler uses these numbers to decide whether a node has enough room for the pod. Setting requests is required for sensible bin-packing. Without them, nodes can be silently overcommitted.

resources.limits — caps the container at 500m CPU and 128Mi memory. If the container exceeds these limits, Kubernetes throttles the CPU or kills the container (for memory). This prevents a single misbehaving pod from starving other workloads on the same node.

A Note on Taints and Tolerations

Once you're comfortable with nodeSelector, the next step in production clusters is adding a taint to your ARM node pool. A taint is a repellent — any pod without an explicit toleration for that taint is blocked from landing on the tainted node.

This means other workloads in your cluster can't accidentally consume your ARM capacity. You'd add the taint when creating the pool:

# Add --node-taints to the pool creation command:
--node-taints=workload-type=arm-optimized:NoSchedule

And a matching toleration in the pod spec:

tolerations:
- key: "workload-type"
  operator: "Equal"
  value: "arm-optimized"
  effect: "NoSchedule"

We're not doing this in the tutorial to keep things simple, but it's the pattern production multi-tenant clusters use to enforce hard separation between workload types.

Write the Service Manifest

We also need a Kubernetes Service to expose the pods over the network. Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-axion-svc
spec:
  selector:
    app: hello-axion
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

• selector: app: hello-axion — the Service discovers pods using labels. Any pod with app: hello-axion on it will be added to this Service's load balancer pool.

• port: 80 — the port the Service is reachable on from outside the cluster.

• targetPort: 8080 — the port on the pod that traffic gets forwarded to. Our Go server listens on port 8080, so this must match.

• type: LoadBalancer — tells GKE to provision an external Google Cloud load balancer and assign it a public IP. This is what makes the Service reachable from the internet.

Apply Both Manifests

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl apply reads each manifest file and creates or updates the resources described in it. If the resources don't exist yet, they're created. If they already exist, Kubernetes only applies the diff — it won't restart pods unnecessarily.

Watch the pods come up in real time:

kubectl get pods -w

The -w flag watches for changes and prints updates as they happen. You should see pods transition from Pending → ContainerCreating → Running. Once all three show Running, press Ctrl+C to stop watching.

Step 9: Verify the Deployment

Everything is running. Now we need evidence — not just that pods are up, but that they're on the right nodes and serving the right binary.

Confirm Pod Placement

kubectl get pods -o wide

The -o wide flag adds extra columns to the output, including the name of the node each pod was scheduled on. Look at the NODE column:

NAME                          READY   STATUS    NODE
hello-axion-7b8d9f-abc12      1/1     Running   gke-axion-tutorial-axion-pool-a-...
hello-axion-7b8d9f-def34      1/1     Running   gke-axion-tutorial-axion-pool-b-...
hello-axion-7b8d9f-ghi56      1/1     Running   gke-axion-tutorial-axion-pool-c-...

All three pods should show node names containing axion-pool. None should show default-pool.

Confirm the Nodes Are ARM

Take one of those node names and verify its architecture label:

kubectl get node NODE_NAME --show-labels | grep kubernetes.io/arch

Replace NODE_NAME with one of the node names from the previous command. You should see:

kubernetes.io/arch=arm64

That's the automatic label GKE applied when it provisioned the ARM hardware. Our nodeSelector matched on this label to pin the pods here.

Ask the Application Itself

This is the most satisfying verification step. Our Go server reports the architecture of the binary that's running. Let's ask it directly.

Use kubectl port-forward to create a secure tunnel from port 8080 on your local machine to port 8080 on the Deployment:

kubectl port-forward deployment/hello-axion 8080:8080

This command stays running in the foreground — open a second terminal window and run:

curl http://localhost:8080

You should see:

Hello from freeCodeCamp!
Architecture : arm64
OS           : linux
Pod hostname : hello-axion-7b8d9f-abc12

Architecture : arm64. That's our Go binary confirming that it was compiled for ARM64 and is executing on an ARM64 CPU. The single image tag we built does the right thing automatically.

The Bonus: See the Manifest List in Action

Want to see the multi-arch image indexing at work? Stop the port-forward, then run:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Replace PROJECT_ID with your actual Google Cloud project ID.

You'll see four entries in the manifest list. Two are real images — Platform: linux/amd64 and Platform: linux/arm64. The other two show Platform: unknown/unknown with an attestation-manifest annotation. These are build provenance records that Docker Buildx automatically attaches to every image — a supply chain security feature (SLSA attestation) that proves how and where the image was built.

You may notice that if you check the image digest recorded in a running pod:

kubectl get pod POD_NAME \
  -o jsonpath='{.status.containerStatuses[0].imageID}'

Replace POD_NAME with one of the pod names from earlier.

The digest returned matches the top-level manifest list digest, not the arm64-specific one. This is expected behaviour. Modern Kubernetes (using containerd) records the manifest list digest, not the resolved platform digest. The platform resolution already happened when the node pulled the correct image variant.

The definitive proof that the right binary is running is what you already have: the node labeled kubernetes.io/arch=arm64 and the application reporting Architecture: arm64.

Step 10: Cost Savings and Tradeoffs

The hands-on work is done. Let's talk about why any of this is worth the effort.

The Cost Math

At the time of writing, here's how ARM compares to equivalent x86 machines on Google Cloud (prices are approximate and change over time — check the official pricing page before making decisions):

That's a raw 25–30% reduction in compute cost per node. Factor in Google's published claim of up to 65% better price-performance for Axion on relevant workloads — meaning you may need fewer nodes to handle the same traffic — and the savings compound further.

Here's how that looks at scale, for a service running 20 nodes continuously for a year:

• 20 × n2-standard-4 × \(0.19 × 8,760 hours = \)33,288/year

• 20 × t2a-standard-4 × \(0.14 × 8,760 hours = \)24,528/year

That's roughly $8,760 saved annually on compute, before committed use discounts (which further widen the gap).

When ARM Is the Right Choice

ARM works best for:

• Stateless API servers and web applications — like the app we built. ARM excels at high-throughput, low-latency network workloads.

• Background workers and queue processors — long-running services that don't depend on x86-specific binaries.

• Microservices written in Go, Rust, or Python — these languages have excellent ARM64 support and are built cross-platform by default.

When to Proceed Carefully

• Native library dependencies — some older C libraries, proprietary SDKs, or compiled ML model-serving runtimes don't have ARM64 builds. Always audit your dependency tree before migrating.

• CI pipelines need ARM too — your automated tests should run on ARM, not just x86. An image that silently fails only on ARM is harder to debug than one that never claimed ARM support.

• Profile before optimizing — the cost savings are real, but measure your actual workload behavior on ARM before committing. Not every workload benefits equally.

Cleanup

When you're done, clean up to avoid ongoing charges:

# Remove the Kubernetes resources from the cluster
kubectl delete -f k8s/

# Delete the ARM node pool
gcloud container node-pools delete axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the cluster itself
gcloud container clusters delete axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the images from Artifact Registry (optional — storage costs are minimal)
gcloud artifacts docker images delete \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Conclusion

Let's recap what you built and why each part matters.

You started with a Go application, a Dockerfile, and a docker buildx build command that produced two images — one for x86, one for ARM64 — wrapped in a single Manifest List tag. Any server that pulls that tag gets the right binary automatically, without you maintaining separate pipelines or separate tags.

You provisioned a GKE cluster with two node pools running different CPU architectures, then used nodeSelector to make sure your ARM-optimized workload lands only on the ARM Axion nodes — not on x86 by accident. The result is a deployment that's both architecture-correct and cost-efficient.

The patterns you practiced here don't stop at this demo. The same Dockerfile technique works for any language with cross-compilation support. The same nodeSelector approach works for any workload you want to pin to ARM. As more teams migrate services to ARM over the coming years, having these skills will be a real asset.

Where to go from here:

• Add a GitHub Actions workflow that runs docker buildx build --platform linux/amd64,linux/arm64 on every push, automating this entire process in CI.

• Audit one of your existing stateless services for ARM compatibility and try migrating it.

• Explore Node Affinity as a softer alternative to nodeSelector for workloads that can run on either architecture but prefer ARM.

• Look into GKE Autopilot, which now supports ARM nodes and handles node pool management automatically.

Happy building.

Project File Structure

hello-axion/
├── app/
│   ├── main.go          — Go HTTP server
│   ├── go.mod           — Go module definition
│   └── Dockerfile       — Multi-stage Dockerfile
└── k8s/
    ├── deployment.yaml  — Deployment with nodeSelector and probes
    └── service.yaml     — LoadBalancer Service

All source files for this tutorial are available in the companion GitHub repository: https://github.com/Amiynarh/multi-arch-docker-gke-arm

It has no user database, no built-in login system, no password file. When you run kubectl get pods, Kubernetes receives an HTTP request and asks one question: who signed this, and do I trust that signature? Everything else — what you're allowed to do, which namespaces you can access, whether your request goes through at all — comes after that question is answered.

This surprises most engineers who are new to Kubernetes. They expect something like a database of users with passwords. Instead, they find a pluggable chain of authenticators, each one able to vouch for a request in a different way:

• Client certificates

• OIDC tokens from an external identity provider

• Cloud provider IAM tokens

• Service account tokens projected into pods.

Any of these can be active at the same time.

Understanding this model is what separates engineers who can debug authentication failures from engineers who copy kubeconfig files and hope for the best.

In this article, you'll work through how the Kubernetes authentication chain works from first principles. You'll see how x509 client certificates are used — and why they're a poor choice for human users in production. You'll configure OIDC authentication with Dex, giving your cluster a real browser-based login flow. And you'll see how AWS, GCP, and Azure each plug into the same underlying model.

Prerequisites

• A running kind cluster — a fresh one works fine, or reuse an existing one

• kubectl and helm installed

• openssl available on your machine (comes pre-installed on macOS and most Linux distros)

• Basic familiarity with what a JWT is (a signed JSON object with claims) — you don't need to be able to write one, just recognise one

All demo files are in the companion GitHub repository.

Table of Contents

• How Kubernetes Authentication Works

  • The Authenticator Chain

  • Users vs Service Accounts

  • What Happens After Authentication

• How to Use x509 Client Certificates

  • How the Certificate Maps to an Identity

  • The Cluster CA

  • The Limits of Certificate-Based Auth

• Demo 1 — Create and Use an x509 Client Certificate

• How to Set Up OIDC Authentication

  • How the OIDC Flow Works in Kubernetes

  • The API Server Configuration

  • JWT Claims Kubernetes Uses

  • How kubelogin Works

• Demo 2 — Configure OIDC Login with Dex and kubelogin

• Cloud Provider Authentication

  • AWS EKS

  • Google GKE

  • Azure AKS

• Webhook Token Authentication

• Cleanup

• Conclusion

How Kubernetes Authentication Works

Every request that reaches the Kubernetes API server — whether from kubectl, a pod, a controller, or a CI pipeline — carries a credential of some kind.

The API server passes that credential through a chain of authenticators in sequence. The first authenticator that can verify the credential wins. If none can, the request is treated as anonymous.

The Authenticator Chain

Kubernetes supports several authentication strategies simultaneously. You can have client certificate authentication and OIDC authentication active on the same cluster at the same time, which is common in production: cluster administrators use certificates, regular developers use OIDC. The strategies active on a cluster are determined by flags passed to the kube-apiserver process.

The strategies available are x509 client certificates, bearer tokens (static token files — rarely used in production), bootstrap tokens (used during node join operations), service account tokens, OIDC tokens, authenticating proxies, and webhook token authentication. A cluster doesn't have to use all of them, and most don't. But knowing they all exist helps when you're diagnosing an auth failure.

Users vs Service Accounts

There is an important distinction in how Kubernetes thinks about identity. Service accounts are Kubernetes objects — they live in a namespace, get created with kubectl create serviceaccount, and have tokens managed by the cluster itself. Every pod runs as a service account. These are machine identities for workloads.

Users, on the other hand, don't exist as Kubernetes objects at all. There is no kubectl create user command. Kubernetes doesn't manage user accounts. Instead, it trusts external systems to assert user identity — a certificate authority, an OIDC provider, or a cloud provider's IAM system. Kubernetes just verifies the assertion and extracts the username and group memberships from it.

What Happens After Authentication

Authentication only answers one question: who is this? Once the API server has a verified identity — a username and zero or more group memberships — it passes the request to the authorisation layer. By default that is RBAC, which checks the identity against Role and ClusterRole bindings to determine what the request is allowed to do.

This is why authentication and authorisation are separate concerns in Kubernetes. A valid certificate gets you past the front door. What you can do inside is RBAC's job. An authenticated user with no RBAC bindings can authenticate successfully but will be denied every API call.

If you want a deep dive into how RBAC rules, roles, and bindings work, check out this handbook on How to Secure a Kubernetes Cluster: RBAC, Pod Hardening, and Runtime Protection.

How to Use x509 Client Certificates

x509 client certificate authentication is the oldest and simplest authentication method in Kubernetes. It's how kubectl works out of the box when you create a cluster — the kubeconfig file that kind or kubeadm generates contains an embedded client certificate signed by the cluster's Certificate Authority.

How the Certificate Maps to an Identity

When the API server receives a request with a client certificate, it validates the certificate against its trusted CA, then reads two fields (The Common Name and Organization) from the certificate to construct an identity.

The Common Name (CN) field becomes the username. The Organization (O) field, which can contain multiple values, becomes the list of groups the user belongs to.

So a certificate with CN=jane and O=engineering authenticates as username jane in group engineering. If you want to give jane permissions, you create a RoleBinding that references either the username jane or the group engineering as a subject.

This is the same mechanism behind system:masters. When kind creates a cluster and writes a kubeconfig for you, it generates a certificate with O=system:masters. Kubernetes has a built-in ClusterRoleBinding that grants cluster-admin to anyone in the system:masters group. That's why your default kubeconfig has full admin access — it's not magic, it's a certificate with the right group.

The Cluster CA

Every Kubernetes cluster has a root Certificate Authority — a private key and a self-signed certificate that the API server trusts. Any client certificate signed by this CA is trusted by the cluster.

The CA certificate and key are typically stored in /etc/kubernetes/pki/ on the control plane node, or in the kube-system namespace as a secret, depending on how the cluster was created.

On kind clusters, you can copy the CA cert and key directly from the control plane container:

docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.crt ./ca.crt
docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.key ./ca.key

Whoever holds the CA key can issue certificates for any username and any group, including system:masters. This makes the CA key the most sensitive secret in a Kubernetes cluster. Guard it accordingly.

The Limits of Certificate-Based Auth

Client certificates work, but they have two fundamental problems that make them a poor choice for human users in production.

The first is that Kubernetes doesn't check certificate revocation lists (CRLs). If a developer's kubeconfig is stolen, the embedded certificate remains valid until it expires — which is typically one year in most Kubernetes setups. There's no way to immediately invalidate it. You can't "log out" a certificate. The only mitigation is to rotate the entire cluster CA, which invalidates every certificate including those belonging to other legitimate users.

The second is operational overhead. Certificates must be generated, distributed to users, and rotated before expiry. There's no self-service. In a team of ten engineers, managing certificates is annoying. In a team of a hundred, it's a full-time job.

For human access in production, OIDC is the right answer: short-lived tokens issued by a trusted identity provider, with a central revocation mechanism, and a standard browser-based login flow. Certificates are fine for service accounts and automation, where token management can be automated and rotation is handled programmatically.

That said, understanding certificates isn't optional. Your kubeconfig uses one. Your CI system probably does too. And cert-based auth is what you fall back to when everything else breaks.

Demo 1 — Create and Use an x509 Client Certificate

In this section, you'll generate a user certificate signed by the cluster CA, bind it to an RBAC role, and use it to authenticate to the cluster as a different user.

This guide is for local development and learning only. Manually signing certificates with the cluster CA and storing keys on disk is done here for simplicity.

In production, you should use the Kubernetes CertificateSigningRequest API or cert-manager for certificate issuance, enforce short-lived certificates with automatic rotation, and store private keys in a secrets manager (HashiCorp Vault, AWS Secrets Manager) or hardware security module (HSM) — never distribute the cluster CA key.

Step 1: Copy the CA cert and key from the kind control plane

docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.crt ./ca.crt
docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.key ./ca.key

This will create two files in your current directory called ca.crt and ca.key

Step 2: Generate a private key and CSR for a new user

You're creating a certificate for a user named jane in the engineering group:

# Generate the private key
openssl genrsa -out jane.key 2048

# Generate a Certificate Signing Request
# CN = username, O = group
openssl req -new \
  -key jane.key \
  -out jane.csr \
  -subj "/CN=jane/O=engineering"

Step 3: Sign the CSR with the cluster CA

openssl x509 -req \
  -in jane.csr \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -out jane.crt \
  -days 365

Expected output:

Certificate request self-signature ok
subject=CN=jane, O=engineering

Step 4: Inspect the certificate

Before using it, confirm the identity it carries:

openssl x509 -in jane.crt -noout -subject -dates

subject=CN=jane, O=engineering
notBefore=Mar 20 10:00:00 2024 GMT
notAfter=Mar 20 10:00:00 2025 GMT

One year from now, this certificate becomes invalid and must be replaced. There's no way to extend it — you have to issue a new one.

Step 5: Build a kubeconfig entry for jane

# Get the cluster API server address from the current context
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

# Create a kubeconfig for jane
kubectl config set-cluster k8s-security \
  --server=$APISERVER \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --kubeconfig=jane.kubeconfig

kubectl config set-credentials jane \
  --client-certificate=jane.crt \
  --client-key=jane.key \
  --embed-certs=true \
  --kubeconfig=jane.kubeconfig

kubectl config set-context jane@k8s-security \
  --cluster=k8s-security \
  --user=jane \
  --kubeconfig=jane.kubeconfig

kubectl config use-context jane@k8s-security \
  --kubeconfig=jane.kubeconfig

Step 6: Test authentication — before RBAC

Try to list pods using jane's kubeconfig:

kubectl get pods -n staging --kubeconfig=jane.kubeconfig

Error from server (Forbidden): pods is forbidden: User "jane" cannot list
resource "pods" in API group "" in the namespace "staging"

This is correct. Jane authenticated successfully — Kubernetes knows who she is. But she has no RBAC bindings, so every API call is denied. Authentication passed, but authorisation failed.

Step 7: Grant jane access with RBAC

RBAC bindings use the username exactly as it appears in the certificate's CN field. If you need a refresher on how Roles, ClusterRoles, and RoleBindings work, this handbook How to Secure a Kubernetes Cluster: RBAC, Pod Hardening, and Runtime Protection covers the full RBAC model. For now, a simple RoleBinding using the built-in view ClusterRole is enough:

# jane-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jane-reader
  namespace: staging
subjects:
  - kind: User
    name: jane          # matches the CN in the certificate
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f jane-rolebinding.yaml
kubectl get pods -n staging --kubeconfig=jane.kubeconfig

No resources found in staging namespace.

No error — jane can now list pods in staging. She can't delete them, create them, or access other namespaces. The certificate got her in. RBAC determines what she can do.

How to Set Up OIDC Authentication

OpenID Connect is an identity layer on top of OAuth 2.0. It's how Kubernetes integrates with enterprise identity providers — Active Directory, Okta, Google Workspace, Keycloak, and any other provider that speaks OIDC. Understanding how Kubernetes uses it requires following the token from the user's browser to the API server's decision.

How the OIDC Flow Works in Kubernetes

When a developer runs kubectl get pods with OIDC configured, the following happens:

1. kubectl checks whether the current credential in the kubeconfig is a valid, unexpired OIDC token

2. If not, it launches kubelogin, a kubectl plugin that opens a browser window

3. The browser redirects to the OIDC provider (Dex, Okta, your corporate IdP)

4. The user logs in with their corporate credentials

5. The OIDC provider issues a signed JWT and returns it to kubelogin

6. kubelogin caches the token locally (under ~/.kube/cache/oidc-login/) and returns it to kubectl

7. kubectl sends the token to the API server as a Bearer header

8. The API server fetches the provider's public keys from its JWKS endpoint and verifies the token signature

9. If valid, the API server extracts the username and group claims from the token

10. RBAC takes over from there

The Kubernetes API server never contacts the OIDC provider for each request. It only fetches the provider's public keys periodically to verify signatures locally. This makes OIDC authentication stateless and scalable.

The API Server Configuration

For OIDC to work, the API server needs to know where to find the identity provider and how to interpret the tokens it issues.

In Kubernetes v1.30+, this is configured through an AuthenticationConfiguration file passed via the --authentication-config flag. (In older versions, individual --oidc-* flags were used instead, but these were removed in v1.35.)

The AuthenticationConfiguration defines OIDC providers under the jwt key:

On a kind cluster, the --authentication-config flag is set in the cluster configuration before creation, not after. You'll see this in the next demo.

JWT Claims Kubernetes Uses

A JWT is a signed JSON object with three sections: a header, a payload, and a signature. The payload is a set of claims – key-value pairs that assert facts about the token. Kubernetes reads specific claims from the payload to build an identity.

The required claims are iss (the issuer URL, must match issuer.url in the AuthenticationConfiguration), sub (the subject, a unique identifier for the user), and aud (the audience, must match the issuer.audiences list). The exp claim (expiry time) is also required as the API server rejects expired tokens.

The most useful optional claim is groups (or whatever you configure via claimMappings.groups.claim). When this claim is present, Kubernetes can map OIDC group memberships directly to RBAC group bindings. A user in the platform-engineers group in your identity provider automatically gets the RBAC permissions you've bound to that group in Kubernetes — no manual user management required.

How kubelogin Works

kubelogin (also distributed as kubectl oidc-login) is a kubectl credential plugin. Instead of embedding a static certificate or token in your kubeconfig, you configure a credential plugin that runs a helper binary when kubectl needs a token.

When kubelogin is invoked, it checks its local token cache. If the cached token is still valid, it returns it immediately. If the token has expired, it initiates the OIDC authorization code flow — opens a browser, redirects to the identity provider, receives the token after login, caches it locally, and returns it to kubectl. The whole flow takes about five seconds when it triggers.

This means tokens are short-lived (typically an hour) and rotate automatically. If a developer's machine is compromised, the token expires on its own. There is no long-lived credential sitting in a file somewhere.

Demo 2 — Configure OIDC Login with Dex and kubelogin

In this section, you'll deploy Dex as a self-hosted OIDC provider, configure a kind cluster to trust it, and log in with a browser. Dex is a good demo vehicle because it runs inside the cluster and doesn't require a cloud account or an external service.

This guide is for local development and learning only. Self-signed certificates, static passwords, and certs stored on disk are used here for simplicity.

In production, use a managed identity provider (Azure Entra ID, Google Workspace, Okta), automate certificate lifecycle with cert-manager, and store secrets in a secrets manager (HashiCorp Vault, AWS Secrets Manager) or inject them via CSI driver — never commit or store certs as local files.

Step 1: Create a kind cluster with OIDC authentication

OIDC authentication for the API server must be configured at cluster creation time on Kind because the API server needs to know which identity provider to trust before it starts accepting requests.

Note: Kubernetes v1.30+ deprecated the --oidc-* API server flags in favor of the structured AuthenticationConfiguration API (via --authentication-config). In v1.35+ the old flags are removed entirely. This guide uses the new approach.

nip.io is a wildcard DNS service — dex.127.0.0.1.nip.io resolves to 127.0.0.1. This lets us use a real hostname for TLS without editing /etc/hosts.

First, generate a self-signed CA and TLS certificate for Dex:

# Generate a CA for Dex
openssl req -x509 -newkey rsa:4096 -keyout dex-ca.key \
  -out dex-ca.crt -days 365 -nodes \
  -subj "/CN=dex-ca"

# Generate a certificate for Dex signed by that CA
openssl req -newkey rsa:2048 -keyout dex.key \
  -out dex.csr -nodes \
  -subj "/CN=dex.127.0.0.1.nip.io"

openssl x509 -req -in dex.csr \
  -CA dex-ca.crt -CAkey dex-ca.key \
  -CAcreateserial -out dex.crt -days 365 \
  -extfile <(printf "subjectAltName=DNS:dex.127.0.0.1.nip.io")

Next, generate the AuthenticationConfiguration file. This tells the API server how to validate JWTs — which issuer to trust (url), which audience to expect (audiences), and which JWT claims map to Kubernetes usernames and groups (claimMappings). The CA cert is inlined so the API server can verify Dex's TLS certificate when fetching signing keys:

cat > auth-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: https://dex.127.0.0.1.nip.io:32000
      audiences:
        - kubernetes
      certificateAuthority: |
$(sed 's/^/        /' dex-ca.crt)
    claimMappings:
      username:
        claim: email
        prefix: ""
      groups:
        claim: groups
        prefix: ""
EOF

The kind-oidc.yaml config uses extraPortMappings to expose Dex's port to your browser, extraMounts to copy files into the Kind node, and a kubeadmConfigPatch to pass --authentication-config to the API server:

# kind-oidc.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      # Forward port 32000 from the Docker container to localhost,
      # so your browser can reach Dex's login page
      - containerPort: 32000
        hostPort: 32000
        protocol: TCP
    extraMounts:
      # Copy files from your machine into the Kind node's filesystem
      - hostPath: ./dex-ca.crt
        containerPath: /etc/ca-certificates/dex-ca.crt
        readOnly: true
      - hostPath: ./auth-config.yaml
        containerPath: /etc/kubernetes/auth-config.yaml
        readOnly: true
    kubeadmConfigPatches:
      # Patch the API server to enable OIDC authentication
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            # Tell the API server to load our AuthenticationConfiguration
            authentication-config: /etc/kubernetes/auth-config.yaml
          extraVolumes:
            # Mount files into the API server pod (it runs as a static pod,
            # so it needs explicit volume mounts even though files are on the node)
            - name: dex-ca
              hostPath: /etc/ca-certificates/dex-ca.crt
              mountPath: /etc/ca-certificates/dex-ca.crt
              readOnly: true
              pathType: File
            - name: auth-config
              hostPath: /etc/kubernetes/auth-config.yaml
              mountPath: /etc/kubernetes/auth-config.yaml
              readOnly: true
              pathType: File

Create the cluster:

kind create cluster --name k8s-auth --config kind-oidc.yaml

Step 2: Deploy Dex

Dex is an OIDC-compliant identity provider that acts as a bridge between Kubernetes and upstream identity sources (LDAP, SAML, GitHub, and so on). In this demo it runs inside the cluster with a static password database — two hardcoded users you can log in as.

The API server doesn't talk to Dex directly on every request. It only needs Dex's CA certificate (which you inlined in the AuthenticationConfiguration) to verify the JWT signatures on tokens that Dex issues.

The deployment has four parts: a ConfigMap with Dex's configuration, a Deployment to run Dex, a NodePort Service to expose it on port 32000 (matching the issuer URL), and RBAC resources so Dex can store state using Kubernetes CRDs.

First, create the namespace and load the TLS certificate as a Kubernetes Secret. Dex needs this to serve HTTPS. Without it, your browser and the API server would refuse to connect:

kubectl create namespace dex

kubectl create secret tls dex-tls \
  --cert=dex.crt \
  --key=dex.key \
  -n dex

Save the following as dex-config.yaml. This configures Dex with a static password connector — two hardcoded users for the demo:

# dex-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dex-config
  namespace: dex
data:
  config.yaml: |
    # issuer must exactly match the URL in your AuthenticationConfiguration
    issuer: https://dex.127.0.0.1.nip.io:32000

    # Dex stores refresh tokens and auth codes — here it uses Kubernetes CRDs
    storage:
      type: kubernetes
      config:
        inCluster: true

    # Dex's HTTPS listener — serves the login page and token endpoints
    web:
      https: 0.0.0.0:5556
      tlsCert: /etc/dex/tls/tls.crt
      tlsKey: /etc/dex/tls/tls.key

    # staticClients defines which applications can request tokens.
    # "kubernetes" is the client ID that kubelogin uses when authenticating
    staticClients:
      - id: kubernetes
        redirectURIs:
          - http://localhost:8000     # kubelogin listens here to receive the callback
        name: Kubernetes
        secret: kubernetes-secret     # shared secret between kubelogin and Dex

    # Two demo users with the password "password" (bcrypt-hashed).
    # In production, you'd connect Dex to LDAP, SAML, or a social login instead
    enablePasswordDB: true
    staticPasswords:
      - email: "jane@example.com"
        # bcrypt hash of "password" — generate your own with: htpasswd -bnBC 10 "" password
        hash: "\(2a\)10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
        username: "jane"
        userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"
      - email: "admin@example.com"
        hash: "\(2a\)10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
        username: "admin"
        userID: "a8b53e13-7e8c-4f7b-9a33-6c2f4d8c6a1b"
        groups:
          - platform-engineers

Save the following as dex-deployment.yaml. This creates the Deployment, Service, ServiceAccount, and RBAC that Dex needs to run:

# dex-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dex
  namespace: dex
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dex
  template:
    metadata:
      labels:
        app: dex
    spec:
      serviceAccountName: dex
      containers:
        - name: dex
          # v2.45.0+ required — earlier versions don't include groups from staticPasswords in tokens
          image: ghcr.io/dexidp/dex:v2.45.0
          command: ["dex", "serve", "/etc/dex/cfg/config.yaml"]
          ports:
            - name: https
              containerPort: 5556
          volumeMounts:
            - name: config
              mountPath: /etc/dex/cfg
            - name: tls
              mountPath: /etc/dex/tls
      volumes:
        - name: config
          configMap:
            name: dex-config
        - name: tls
          secret:
            secretName: dex-tls
---
# NodePort Service — exposes Dex on port 32000 on the Kind node.
# Combined with extraPortMappings, this makes Dex reachable from your browser
apiVersion: v1
kind: Service
metadata:
  name: dex
  namespace: dex
spec:
  type: NodePort
  ports:
    - name: https
      port: 5556
      targetPort: 5556
      nodePort: 32000
  selector:
    app: dex
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dex
  namespace: dex
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: dex
rules:
  - apiGroups: ["dex.coreos.com"]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dex
subjects:
  - kind: ServiceAccount
    name: dex
    namespace: dex
roleRef:
  kind: ClusterRole
  name: dex
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f dex-config.yaml
kubectl apply -f dex-deployment.yaml
kubectl rollout status deployment/dex -n dex

Step 3: Install kubelogin

# macOS
brew install int128/kubelogin/kubelogin

# Linux
curl -LO https://github.com/int128/kubelogin/releases/latest/download/kubelogin_linux_amd64.zip
unzip -j kubelogin_linux_amd64.zip kubelogin -d /tmp
sudo mv /tmp/kubelogin /usr/local/bin/kubectl-oidc_login
rm kubelogin_linux_amd64.zip

Confirm it's installed:

kubectl oidc-login --version

Step 4: Configure a kubeconfig entry for OIDC

This creates a new user and context in your kubeconfig. Instead of using a client certificate (like the default Kind admin), it tells kubectl to use kubelogin to get a token from Dex.

The --oidc-extra-scope flags are important: without email and groups, Dex won't include those claims in the JWT, and the API server won't know who you are or what groups you belong to.

kubectl config set-credentials oidc-user \
  --exec-api-version=client.authentication.k8s.io/v1beta1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg=--oidc-issuer-url=https://dex.127.0.0.1.nip.io:32000 \
  --exec-arg=--oidc-client-id=kubernetes \
  --exec-arg=--oidc-client-secret=kubernetes-secret \
  --exec-arg=--oidc-extra-scope=email \
  --exec-arg=--oidc-extra-scope=groups \
  --exec-arg=--certificate-authority=$(pwd)/dex-ca.crt

kubectl config set-context oidc@k8s-auth \
  --cluster=kind-k8s-auth \
  --user=oidc-user

kubectl config use-context oidc@k8s-auth

Step 5: Trigger the login flow

Jane has no RBAC permissions yet, so first grant her read access from the admin context:

kubectl --context kind-k8s-auth create clusterrolebinding jane-view \
  --clusterrole=view --user=jane@example.com

Now switch to the OIDC context and trigger a login:

kubectl get pods -n default

Your browser opens and redirects to the Dex login page. Log in as jane@example.com with password password.

After login, the terminal completes:

No resources found in default namespace.

The browser-based authentication worked. kubectl received the token from Dex, sent it to the API server, the API server validated the JWT signature using the CA certificate from the AuthenticationConfiguration, extracted jane@example.com from the email claim, matched it against the RBAC binding, and authorized the request.

Without the clusterrolebinding, you would see Error from server (Forbidden) — authentication succeeds (the API server knows who you are) but authorization fails (jane has no permissions). This is the distinction between 401 Unauthorized and 403 Forbidden.

Step 6: Inspect the JWT

A JWT (JSON Web Token) is a signed JSON payload that contains claims about the user. kubelogin caches the token locally under ~/.kube/cache/oidc-login/ so you don't have to log in on every kubectl command.

List the directory to find the cached file:

ls ~/.kube/cache/oidc-login/

Decode the JWT payload directly from the cache:

cat ~/.kube/cache/oidc-login/$(ls ~/.kube/cache/oidc-login/ | grep -v lock | head -1) | \
  python3 -c "
import json, sys, base64
token = json.load(sys.stdin)['id_token'].split('.')[1]
token += '=' * (4 - len(token) % 4)
print(json.dumps(json.loads(base64.urlsafe_b64decode(token)), indent=2))
"

You'll see something like:

{
  "iss": "https://dex.127.0.0.1.nip.io:32000",
  "sub": "CiQwOGE4Njg0Yi1kYjg4LTRiNzMtOTBhOS0zY2QxNjYxZjU0NjYSBWxvY2Fs",
  "aud": "kubernetes",
  "exp": 1775307910,
  "iat": 1775221510,
  "email": "jane@example.com",
  "email_verified": true
}

The email claim becomes jane's Kubernetes username because the AuthenticationConfiguration maps username.claim: email. The aud matches the configured audiences. The iss matches the issuer url. This is how the API server validates the token without contacting Dex on every request — it only needs the CA certificate to verify the JWT signature.

Step 7: Map OIDC groups to RBAC

The admin@example.com user has a groups claim in the Dex config containing platform-engineers. Instead of creating individual RBAC bindings per user, you can bind permissions to a group — anyone whose JWT contains that group gets the permissions automatically:

# platform-engineers-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: platform-engineers-admin
subjects:
  - kind: Group
    name: platform-engineers     # matches the groups claim in the JWT
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

You're currently logged in as jane@example.com via the OIDC context, but jane only has view permissions — she can't create cluster-wide RBAC bindings. Switch back to the admin context to apply this:

kubectl config use-context kind-k8s-auth
kubectl apply -f platform-engineers-binding.yaml
kubectl config use-context oidc@k8s-auth

Now clear the cached token to log out of jane's session, then trigger a new login as admin@example.com:

# Clear the cached token — this is how you "log out" with kubelogin
rm -rf ~/.kube/cache/oidc-login/

# This will open the browser again for a fresh login
kubectl get pods -n default

Log in as admin@example.com with password password. This time the JWT will contain "groups": ["platform-engineers"], which matches the ClusterRoleBinding you just created. The admin user gets full cluster access — without ever being added to a kubeconfig by name.

You can verify by decoding the new token (Step 6) — the groups claim will be present:

{
  "email": "admin@example.com",
  "groups": ["platform-engineers"]
}

This is the real power of OIDC group claims: you manage group membership in your identity provider, and Kubernetes permissions follow automatically. Add someone to the platform-engineers group in Dex (or any upstream IdP), and they get cluster-admin access on their next login — no kubeconfig or RBAC changes needed.

Cloud Provider Authentication

AWS, GCP, and Azure each give Kubernetes clusters a native authentication mechanism that ties into their IAM systems.

The implementations differ in API surface, but they all use the same underlying mechanism: OIDC token projection. Once you understand how Dex works above, these are all variations on the same theme.

AWS EKS

EKS uses the aws-iam-authenticator to translate AWS IAM identities into Kubernetes identities. When you run kubectl against an EKS cluster, the AWS CLI generates a short-lived token signed with your IAM credentials. The API server passes this token to the aws-iam-authenticator webhook, which verifies it against AWS STS and returns the corresponding username and groups.

User access is controlled via the aws-auth ConfigMap in kube-system, which maps IAM role ARNs and IAM user ARNs to Kubernetes usernames and groups. A typical entry looks like this:

# In kube-system/aws-auth ConfigMap
mapRoles:
  - rolearn: arn:aws:iam::123456789:role/platform-engineers
    username: platform-engineer:{{SessionName}}
    groups:
      - platform-engineers

AWS is migrating from the aws-auth ConfigMap to a newer Access Entries API, which manages the same mapping through the EKS API rather than a ConfigMap. The underlying authentication mechanism is the same.

Google GKE

GKE integrates with Google Cloud IAM using two different mechanisms, depending on whether you're authenticating as a human user or as a workload.

For human users, GKE accepts standard Google OAuth2 tokens. Running gcloud container clusters get-credentials writes a kubeconfig that uses the gcloud CLI as a credential plugin, generating short-lived tokens from your Google account automatically.

For pod-level identity — letting a pod assume a Google Cloud IAM role — GKE uses Workload Identity. You annotate a Kubernetes service account to bind it to a Google Service Account, and pods running as that service account can call Google Cloud APIs using the GSA's permissions:

# Bind a Kubernetes SA to a Google Service Account
kubectl annotate serviceaccount my-app \
  --namespace production \
  iam.gke.io/gcp-service-account=my-app@my-project.iam.gserviceaccount.com

Azure AKS

AKS integrates with Azure Active Directory. When Azure AD integration is enabled, kubectl requests an Azure AD token on behalf of the user via the Azure CLI, and the AKS API server validates it against Azure AD.

For pod-level identity, AKS uses Azure Workload Identity, which follows the same OIDC federation pattern as GKE Workload Identity. A Kubernetes service account is annotated with an Azure Managed Identity client ID, and pods can request Azure AD tokens without storing any credentials:

# Annotate a service account with the Azure Managed Identity client ID
kubectl annotate serviceaccount my-app \
  --namespace production \
  azure.workload.identity/client-id=<MANAGED_IDENTITY_CLIENT_ID>

The underlying pattern across all three providers is the same: a trusted OIDC token is issued by the cloud provider, verified by the Kubernetes API server, and mapped to an identity through a binding (the aws-auth ConfigMap, a GKE Workload Identity binding, or an AKS federated identity credential). The OIDC section in this article is the conceptual foundation for all of them.

Webhook Token Authentication

Webhook token authentication is worth knowing about because it appears in several common Kubernetes setups, even if you never configure it yourself.

When a request arrives with a bearer token that no other authenticator recognises, Kubernetes can send that token to an external HTTP endpoint for validation. The endpoint returns a response indicating who the token belongs to.

This is how EKS authentication worked before the aws-iam-authenticator was built into the API server. It's also how bootstrap tokens work during node join operations: a token is generated, embedded in the kubeadm join command, and validated by the bootstrap webhook when the new node contacts the API server for the first time.

For most clusters, you'll encounter webhook auth as something already running rather than something you configure. The main thing to know is that it exists and what it looks like when it appears in logs or configuration.

Cleanup

To remove everything created in this article:

# Delete the OIDC demo cluster
kind delete cluster --name k8s-auth

# Remove generated certificate files
rm -f ca.crt ca.key jane.key jane.csr jane.crt jane.kubeconfig
rm -f dex-ca.crt dex-ca.key dex.crt dex.key dex.csr dex-ca.srl auth-config.yaml

# Remove the kubelogin token cache
rm -rf ~/.kube/cache/oidc-login/

Conclusion

Kubernetes authentication is not a single mechanism — it's a chain of pluggable strategies, each one suited to different use cases. In this article you worked through the most important ones.

x509 client certificates are how Kubernetes works out of the box. The CN field becomes the username, the O field becomes the group, and the cluster CA is the trust anchor. You created a certificate for a new user, bound it to RBAC, and saw exactly how authentication and authorisation interact — authentication gets you in, RBAC determines what you can do.

You also saw the fundamental limitation: Kubernetes doesn't check certificate revocation lists, so a compromised certificate remains valid until it expires. This makes certificates a poor fit for human users in production environments.

OIDC is the production-grade answer. Tokens are short-lived, issued by a trusted identity provider, and map directly to Kubernetes groups through JWT claims. You deployed Dex as a self-hosted OIDC provider, configured the API server to trust it, and set up kubelogin for browser-based authentication.

You then decoded a JWT to see exactly what the API server reads from it, and mapped an OIDC group claim to a Kubernetes ClusterRoleBinding.

Cloud provider authentication — EKS, GKE, AKS — uses the same OIDC foundation with provider-specific wrappers. Understanding how Dex works makes each of those systems immediately readable.

All YAML, certificates, and configuration files from this article are in the companion GitHub repository.

But as your requirements grow, especially around compliance, security, multi-tenancy, or conflicting dependencies, your team will likely move beyond namespaces and start creating separate clusters.

What starts as a clean separation quickly becomes cluster sprawl, bringing higher costs, complex networking, and constant operational overhead.

In this article, we'll explore how kcp can help fix this problem by allowing you to run multiple “logical clusters” inside a single control plane.

Table of Contents

• Prerequisites

• The Challenge of Namespaces and Multiple Kubernetes Clusters

• Introducing kcp

• Getting Started with kcp

• Deploying and Managing Applications

• Beyond the Primitives: What We Didn't Cover

Prerequisites

• kubectl installed.

• A terminal to run commands

• Curl installed

The Challenge of Namespaces and Multiple Kubernetes Clusters

While namespaces provide some level of isolation, many teams often default to creating entirely new Kubernetes clusters to achieve stronger multi-tenancy, environment separation, or geographic distribution.

At first, this approach works well. But as systems grow, managing a fleet of clusters introduces challenges that often outweigh the benefits.

Every new cluster comes with its own control plane, which you'll need to continuously patch, upgrade, and monitor. Over time, this operational overhead will add up, consuming cycles that platform teams could otherwise spend on higher-value work.

Also, clusters don't naturally share service discovery or identity. This forces you to introduce extra layers like service meshes or VPN-based networking, which increases your system's complexity and expands the overall attack surface.

There’s also the cost factor. Clusters incur baseline infrastructure costs regardless of how much workload they run. Creating dedicated clusters for small teams can lead to underutilized resources or, worse, delay the creation of necessary environments because the cost feels too high.

As a result, platform teams often find themselves acting as “cluster plumbers”, spending more time maintaining infrastructure than enabling developer productivity.

Illustrating the Namespace Problem

As I mentioned earlier, when managing multiple clusters gets too complex, a natural alternative is to use namespaces for isolation within a single cluster.

At first glance, this seems like the perfect solution.

But to understand where this approach falls short, let’s walk through a real-world example using a common requirement in shared Kubernetes environments: running databases.

We'll start by creating different namespaces for each team:

➜ ~ kubectl create namespace team-a 
➜ ~ kubectl create namespace team-b

Let's say Team A needs a MongoDB database for one of its services. The team must first install the required MongoDB Custom Resource Definitions (CRDs) into the cluster, so Kubernetes knows how to understand the different MongoDB resources:

➜ ~ kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/crds.yaml

customresourcedefinition.apiextensions.k8s.io/clustermongodbroles.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbmulticluster.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbsearch.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbusers.mongodb.com created customresourcedefinition.apiextensions.k8s.io/opsmanagers.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbcommunity.mongodbcommunity.mongodb.com created

Secondly, Team A installs the actual Operator application (the controller that continuously runs the database logic) into their designated namespace:

➜ ~ kubectl apply -n team-a -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/mongodb-kubernetes.yaml

But the installation isn't completed due to the error below:

the namespace from the provided object "mongodb" does not match the namespace "team-a". You must pass '--namespace=mongodb' to perform this operation.

Why did this fail? This is because most Kubernetes Operators are designed assuming they own the entire cluster and not just a single namespace.

To force the operator to run in team-a, we can modify the manifest on the fly:

curl -s https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/mongodb-kubernetes.yaml \
  | sed 's/namespace: mongodb/namespace: team-a/g' \
  | kubectl apply -f 

We can then confirm that the operator is installed and running:

➜ ~ k get po -n team-a 
NAME                                          READY STATUS  RESTARTS AGE 
mongodb-kubernetes-operator-6f5f8bb7fd-8h5hj  1/1   Running 0        59s

But even after tricking the Operator into running inside team-a's namespace, we still haven't solved the real problem.

At first glance, team-a's operator is neatly confined to their namespace. But remember Step 1? The CRDs aren't namespaced – they're strictly cluster-scoped. So, even though team-a orchestrated this deployment purely for their own use, those CRDs are now globally registered across the entire cluster.

If Team B checks the API, they'll see all the MongoDB-related CRDs installed by Team A.

➜ ~ kubectl get crds | grep mongodb

clustermongodbroles.mongodb.com               2026-03-24T10:49:35Z
mongodb.mongodb.com                           2026-03-24T10:49:36Z
mongodbcommunity.mongodbcommunity.mongodb.com 2026-03-24T10:49:38Z
mongodbmulticluster.mongodb.com               2026-03-24T10:49:36Z
mongodbsearch.mongodb.com                     2026-03-24T10:49:37Z 
mongodbusers.mongodb.com                      2026-03-24T10:49:37Z 
opsmanagers.mongodb.com                       2026-03-24T10:49:37Z

Now consider what happens if Team B needs to install a different version of MongoDB for its own services. Because the CRDs are shared across the cluster, both teams are now coupled to the same definitions. This means one team’s changes can easily impact the other, turning what should be isolated environments into a source of conflict.

Introducing kcp

kcp is an open-source project that lets you run multiple logical Kubernetes clusters on a single control plane.

These logical clusters are called workspaces, and each one behaves like an independent Kubernetes cluster. Every workspace has its own API endpoint, authentication, authorization, and policies, giving teams the experience of working in fully isolated environments.

This decoupling of the control plane from the worker nodes is what makes kcp different.

In traditional Kubernetes, spinning up a new cluster means provisioning a new API server, a new etcd instance, and all the associated controllers. With kcp, you spin up a workspace, and you have a strong, confined environment for your workload.

It's worth noting that kcp itself doesn't run workloads. It's strictly a control plane. Your actual applications still run on physical Kubernetes clusters. kcp only manages the workspaces and the synchronization of resources to those underlying clusters.

Getting Started with kcp

Now that we've covered what kcp is and why it matters, let's get our hands dirty. We'll set up a local kcp environment and explore the core concepts in action.

To make this realistic, we'll follow a common kcp workflow: a platform team that provides custom APIs, and tenant teams that consume them.

In our case, the platform team will export a MongoDB API, and our two tenant teams will subscribe to those APIs using APIBindings. Once bound, they can deploy MongoDB instances into their workspaces and sync them to physical clusters.

This pattern is at the heart of how kcp enables scalable multi-tenancy. The platform team controls the API definitions and versioning. Tenant teams get self-service access without needing to understand the underlying infrastructure. Let's see how it works!

Installing kcp

Running kcp locally is incredibly lightweight since there are no heavy worker nodes to spin up. You will need two things: the kcp server itself, kubectl-kcp , and the kubectl-ws plugin to manage workspaces.

To install the binaries, let's head over to the kcp-dev releases page.

The commands below are for macOS Apple Silicon. If you're using an Intel Mac or Linux, simply replace darwin_arm64 with your respective architecture.

1. Download the kcp server and workspace plugins:

➜ ~ curl -LO https://github.com/kcp-dev/kcp/releases/download/v0.30.1/kcp_0.30.1_darwin_arm64.tar.gz 

➜ ~ curl -LO https://github.com/kcp-dev/kcp/releases/download/v0.30.1/kubectl-kcp-plugin_0.30.1_darwin_arm64.tar.gz

➜ ~ curl -LO https://github.com/kcp-dev/kcp/releases/download/v0.30.1/kubectl-ws-plugin_0.30.1_darwin_arm64.tar.gz

1. Extract the archives:

➜ ~ tar -xzf kcp_0.30.1_darwin_arm64.tar.gz 
➜ ~ tar -xzf kubectl-kcp-plugin_0.30.1_darwin_arm64.tar.gz
➜ ~ tar -xzf kubectl-ws-plugin_0.30.1_darwin_arm64.tar.gz

1. Move the required binaries into your PATH:

➜ ~ sudo mv bin/kcp /usr/local/bin/
➜ ~ sudo mv bin/kubectl-kcp /usr/local/bin/
➜ ~ sudo mv bin/kubectl-ws /usr/local/bin/

You can confirm the installation by checking the version.

➜ ~ kcp --version
kcp version v1.33.3+kcp-v0.0.0-627385a6

Starting the Server

With the binaries installed, let's boot up your local control plane and bind it to localhost. But first, let's create a "work-folder".

➜ ~ mkdir kcp-test
➜ ~ cd kcp-test

We can then start the kcp server in this directory.

➜ ~ kcp start --bind-address=127.0.0.1

You'll see a flurry of logs as kcp boots up its internal database and exposes the API server. Leave this terminal running in the background.

Connecting to the Root Workspace

Open a new terminal window and navigate back into the kcp-test folder we just created.

At first, if you run a standard ls command, the folder will look empty. But during startup, kcp silently generated a hidden .kcp directory that contains our local certificates and our administrative kubeconfig file. Let's verify that:

➜ ~ cd kcp-test 
➜ kcp-test ls
➜ kcp-test ls -a . .. .kcp 
➜ kcp-test ls .kcp admin.kubeconfig apiserver.crt apiserver.key etcd-server sa.key

Now that we know exactly where the configuration file lives, let's export it so our kubectl commands are routed to kcp instead of your default cluster:

export KUBECONFIG=$PWD/.kcp/admin.kubeconfig

Finally, let's use the workspace plugin we installed earlier to verify that we're connected accurately:

 ➜ kubectl ws .

You should see the message below printed to the console:

Current workspace is 'root'.

This shows that you're now officially inside the kcp root workspace. This is the highest-level administrative boundary where we'll begin creating our tenant logical clusters.

Creating and Managing Workspaces

As we discussed above, in a standard Kubernetes cluster, separating teams means using kubectl create namespace. In kcp, we solve the problem by creating entirely isolated logical clusters – workspaces.

If you recall our architecture diagram from earlier, we want to create three distinct environments for our company: one for the platform engineers to manage shared APIs, and two for our isolated tenant development teams.

Since we're currently inside the administrative root workspace, we can create our new tenant workspaces as children of the root:

➜ kubectl ws create platform-team
Workspace "platform-team" (type root:organization) created.
Waiting for it to be ready... 
Workspace "platform-team" (type root:organization) is ready to use.

➜ kubectl ws create team-a 
Workspace "team-a" (type root:organization) created.
Waiting for it to be ready... 
Workspace "team-a" (type root:organization) is ready to use.

➜ kubectl ws create team-b
Workspace "team-b" (type root:organization) created.
Waiting for it to be ready... 
Workspace "team-b" (type root:organization) is ready to use.

Now, here is where kcp truly shines. Unlike a standard cluster, where objects are just a massive flat list, kcp manages its API as a hierarchy. We can visually prove the structure of our new logical clusters using the tree command:

➜ kubectl ws tree
.
└── root
      ├── platform-team
      ├── team-a
      └── team-b

Jumping between these logical clusters is as fast as changing directories in a terminal. Let's switch our context over into Team A's workspace:

➜ kubectl ws team-a 
Current workspace is 'root:team-a' (type root:organization).

Proving the Isolation

To truly understand the power of what we just did, let's try running a standard Kubernetes command while inside team-a:

➜ kubectl get namespaces

NAME STATUS AGE 
default Active 15m

Let's also ask the cluster what APIs are actually available to us out of the box:

➜ kubectl api-resources

Your output should be similar to what is in the image below:

When you take a closer look at that list. You'll notice that there are no Pods, Deployments, or even ReplicaSets. You don't see all the available APIs that you see in a standard Kubernetes Cluster.

This output proves exactly what we discussed in the architecture section. kcp is incredibly lightweight because every new workspace is born completely stripped of compute. Out of the box, it only contains the absolute bare-minimum control plane APIs needed for routing, RBAC, namespaces, and authentication.

From Team A's perspective, they own this pristine, empty universe. If they install a massive, noisy operator right now, like the MongoDB CRD, it will only exist right here in this specific API bucket.

But this raises the ultimate question: If there are no Deployments or Pods APIs in this workspace... how do we actually deploy our applications?

Deploying and Managing Applications

Now that we have set up our isolated environments, we must address the glaring issue from our last terminal output: How do developers actually deploy applications if there are no Deployment or Pod APIs?

In standard Kubernetes, the API is monolithic. You get everything whether you need it or not, and adding a new schema (like an Operator) forces it globally onto everyone.

kcp takes the exact opposite approach. Every workspace starts completely empty. You then selectively "subscribe" your workspace to only the APIs you actually need using two incredibly powerful new concepts: APIExports and APIBindings.

Let's see exactly how this solves our MongoDB multi-tenancy problem, step by step.

1. The Platform Team "Exports" the API

Instead of treating Custom Resource Definitions as global hazards, the platform engineers manage them centrally. First, lets switch into the platform-team workspace:

➜ kubectl ws :root:platform-team

Current workspace is 'root:platform-team' (type root:organization).

Here, we'll install the MongoDB Operator CRDs in the platform-team's workspace:

➜ kubectl apply -f kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/crds.yaml

To confirm that this is indeed isolated, let's first check what CRDs were installed,

➜ kubectl get crd

NAME                                          CREATED AT
clustermongodbroles.mongodb.com               2026-03-24T20:45:50Z
mongodb.mongodb.com                           2026-03-24T20:45:50Z
mongodbcommunity.mongodbcommunity.mongodb.com 2026-03-24T20:45:51Z
mongodbmulticluster.mongodb.com               2026-03-24T20:45:50Z
mongodbsearch.mongodb.com                     2026-03-24T20:45:51Z
mongodbusers.mongodb.com                      2026-03-24T20:45:51Z
opsmanagers.mongodb.com                       2026-03-24T20:45:51Z

We can switch to team-a's workspace (any of the team's workspaces can be used, we're just trying to establish that the installed CRD is only visible in the platform-team's workspace).

➜ kubectl ws :root:team-a

Current workspace is 'root:team-a' (type root:organization).

➜ kubectl get crd 
No resources found

What we get as output is that there are no custom resources found or registered. This is the power of kcp.

If you don't want to continually type out paths to switch between your logical clusters, the kcp plugin includes a powerful interactive UI right in your terminal.

By running kubectl ws -i, you can use your arrow keys to navigate through your hierarchy and press Enter to instantly switch your context. Even better, this interactive mode provides a holistic view of your environment at any given time. With a single glance, you can see exactly how many APIExports are hosted inside a specific workspace, or which APIs are currently bound by other workspaces.

Let's switch back to the platform-team's workspace to continue with our setup.

Now, we need to do something kcp-specific. If you check your resources right now, those CRDs are strictly local to this workspace. To safely share them with our tenant teams, we need to convert them into an internal kcp tracking object called an APIResourceSchema. This is how kcp structurally version-controls APIs so they can be securely exported.

To do this, we use our kcp plugin to take a "snapshot" of the local MongoDB CRD:

kubectl get crd mongodbcommunity.mongodbcommunity.mongodb.com -o yaml | kubectl kcp crd snapshot -f - --prefix v1 | kubectl apply -f -

You should see an output that says:

apiresourceschema.apis.kcp.io/v1.mongodbcommunity.mongodbcommunity.mongodb.com created

This tells kcp: "Get the CRD we just installed, take a snapshot with the prefix 'v1', and apply the resulting APIResourceSchema back to the cluster."

Now, let's look for the schema kcp just generated for us:

➜ kubectl get apiresourceschemas

NAME                                             AGE
v1.mongodbcommunity.mongodbcommunity.mongodb.com 11s

To safely share this API with our teams, we wrap that generated schema into an APIExport. This acts like "APIs as a Service," publishing the schema so that other workspaces can optionally choose to consume it.

Let's create the Export using the exact schema name we just found:

➜ cat <<EOF | kubectl apply -f -
apiVersion: apis.kcp.io/v1alpha1
kind: APIExport
metadata:
  name: mongodb-v1
spec:
  latestResourceSchemas:
    - v1.mongodbcommunity.mongodbcommunity.mongodb.com
EOF

We can confirm this was successfully created by checking the APIExport resource we have

➜ kubectl get apiexports

NAME       AGE
mongodb-v1 2m46s

2. Tenant Teams "Bind" to the API

Now let's switch our terminal context back over to Team A. Remember our previous output? Their workspace currently has no idea what a MongoDB cluster is. Let's prove it:

➜ kubectl ws :root:team-a
Current workspace is "root:team-a" (type root:organization).

➜ kubectl api-resources | grep mongodb
# (No output. The API does not exist here!)

To securely subscribe to the platform team's newly created API service, Team A needs to create an APIBinding.

While we can write standard Kubernetes YAML to do this, the kcp plugin provides a bind command. Team A simply points the bind command directly at the workspace and the specific API export they want to consume:

➜ kubectl kcp bind apiexport root:platform-team:mongodb-v1
apibinding mongodb-v1 created. Waiting to successfully bind ...
mongodb-v1 created and bound.

➜ kcp-test kubectl get apibindings
NAME                  AGE   READY
mongodb-v1            73s   True
tenancy.kcp.io-bqt7a  7h10m True
topology.kcp.io-9dlvq 7h10m True

The moment Team A executes that bind command, their workspace is magically updated with the new capabilities. Let's check our api-resources one more time:

➜ kubectl api-resources | grep mongodb
mongodbcommunity mdbc mongodbcommunity.mongodb.com/v1 true MongoDBCommunity

Beyond the Primitives: What We Didn't Cover

At this point, you should have a firm, hands-on grasp of the core user primitives of kcp, that is Workspaces, APIExports, and APIBindings. But we've only just scratched the surface of what this architecture makes possible.

To keep this guide digestible, there are a few massive topics that I deliberately didn't cover in this article:

1. Shards and High Availability: Since kcp is designed to host thousands of logical clusters, a single database isn't enough. kcp introduces the Shard primitive, allowing platform administrators to horizontally partition workspace state across multiple underlying etcd instances. This gives kcp infinite scalability and massive High Availability (HA) without complicating the developer experience.

2. Front-Proxy: When kcp scales to host millions of logical clusters, it needs a way to seamlessly direct traffic. The kcp Front-Proxy sits at the absolute edge of the architecture, dynamically routing incoming kubectl API requests go straight to the correct underlying workspace and shard. It ensures the developer experience feels perfectly unified, no matter how massive the background infrastructure actually becomes.

3. Virtual Workspaces: While the workspaces we built today act as simple isolated buckets of state, kcp also supports Virtual Workspaces. These act as dynamic, read-only projections of data. For example, kcp uses virtual workspaces to project a unified view of a specific API across multiple tenant workspaces so that controllers can easily watch them all at once.

4. APIExportEndpointSlices: Just like standard Kubernetes uses endpoints to route traffic to pods, kcp uses EndpointSlices to efficiently route and scale the delivery of massive APIExports across thousands of consuming workspaces.

5. Wiring up the Sync Agent (api-syncagent): We discussed this conceptually in our architecture diagram, but we didn't actually attach a physical cluster. In a production scenario, you deploy the Sync Agent onto a fleet of downstream execution clusters (like EKS, GKE, or On-Premises environments) to automatically pull workloads safely out of kcp and execute them seamlessly on physical hardware.

6. External Integrations Like Crossplane: Because kcp acts purely as a multi-tenant API control plane, it pairs incredibly well with Crossplane. By publishing Crossplane as an APIExport, you can empower developer teams to provision actual cloud infrastructure (like AWS databases or Cloud Spanners) using standard YAML directly from their completely isolated kcp workspaces.

We will cover those advanced integrations in a future deep-dive. But armed with just the base primitives we built today, we can already solve the incredibly complex infrastructure problems we outlined at the beginning of the article.

Storing them is straightforward. But handling rotation, stale env vars, and the gap between what your pod reads and what AWS actually holds is where many engineers go quiet.

In this guide, you'll build a complete secrets pipeline from AWS Secrets Manager into Kubernetes pods. You'll provision the infrastructure with Terraform, sync secrets using the External Secrets Operator, and run a sample application that reads the same credentials in two different ways: via environment variables and via a volume mount.

By the end, you'll be able to:

• Explain the full architecture from vault to pod

• Run the lab locally in about 15 minutes

• Prove why environment variables go stale after rotation, while mounted secret files stay fresh

• Deploy the same pattern on Amazon Elastic Kubernetes Service with OpenID Connect-based CI/CD

• Troubleshoot the most common failures

Below is an architecture diagram showing secrets flowing from AWS Secrets Manager through the External Secrets Operator into a Kubernetes Secret, then splitting into environment variables set at pod start and a volume mount that updates within 60 seconds.

Table of Contents

• Prerequisites

• How to Understand the Secret Flow

• How to Run the Local Lab

• How to Inspect the ExternalSecret and the Application

• How to Test Secret Rotation

• How to Choose Between External Secrets Operator and the CSI Driver

• How to Deploy the Pattern on Amazon Elastic Kubernetes Service

• How to Configure GitHub Actions Without Stored AWS Credentials

• How to Troubleshoot the Most Common Failures

• Conclusion

Prerequisites

Before you begin, make sure you have the following tools installed and configured.

For the local lab:

• An AWS account with access to AWS Secrets Manager

• The AWS CLI installed and configured. Run aws configure and provide your access key, secret key, default region, and output format. The credentials need permission to read and write secrets in AWS Secrets Manager.

• kubectl installed. For Microk8s, run microk8s kubectl config view --raw > ~/.kube/config after installation to connect kubectl to your local cluster.

• Terraform installed

• Helm installed

• Docker installed

• A local Kubernetes cluster: the lab supports Microk8s and kind. If you do not have either installed, follow the Microk8s install guide before continuing.

For the Amazon Elastic Kubernetes Service sections:

• An Amazon Elastic Kubernetes Service cluster you can create or manage

• A GitHub repository you can configure for workflows and secrets

The lab repository includes two deployment paths: a local path for fast learning and an Amazon Elastic Kubernetes Service path for a production-like setup. All the exact commands for each path live in the repo's docs/DEPLOY-LOCAL.md and docs/DEPLOY-EKS.md.

How to Understand the Secret Flow

Before you run any command, you need to understand how the pieces connect.

The flow has four stages:

1. A developer or automated system updates a secret in AWS Secrets Manager.

2. The External Secrets Operator polls AWS Secrets Manager on a schedule and creates or updates a Kubernetes Secret.

3. Your pod reads that Kubernetes Secret.

4. During rotation, the Kubernetes Secret updates, but your two consumption modes behave differently.

How the External Secrets Operator Sync Works

The External Secrets Operator reads a custom Kubernetes resource called ExternalSecret. That resource tells the operator three things:

• Which secret store to connect to

• Which Kubernetes Secret name to create or update

• How often to refresh

In this lab, the ExternalSecret creates a Kubernetes Secret named myapp-database-creds. The operator also adds a template annotation that can trigger a pod restart when the secret rotates.

How the App Consumes Secrets

The sample application exposes three endpoints so you can validate behavior at any time.

• /secrets/env shows what environment variables the pod sees

• /secrets/volume shows what files in the mounted secret directory look like

• /secrets/compare compares both and reports whether rotation has been detected

The app checks four keys: DB_USERNAME, DB_PASSWORD, DB_HOST, and DB_PORT.

How to Run the Local Lab

The local lab gives you a fast learning loop. You can see the full pipeline working and test rotation without waiting for a cloud deployment.

Step 1: Clone the Repo

git clone https://github.com/Osomudeya/k8s-secret-lab
cd k8s-secret-lab

Step 2: Run the Spin-Up Script

bash spinup.sh

The script will ask you to choose a local cluster type. Pick Microk8s or kind, depending on what you have installed. The script installs the External Secrets Operator via Helm, applies the Terraform configuration, and deploys the sample application.

If the script fails at any point, check docs/TROUBLESHOOTING.md before retrying. The most common causes are missing AWS credentials, a misconfigured kubeconfig, or a Microk8s storage add-on that is not enabled.

Important: Run the Lab UI

The lab ships with a separate guided tutorial interface that runs on your laptop. This is not the in-cluster application, it's a React-based checklist at lab-ui/ that walks you through each concept and checkpoint as you work through the lab.

To start it, open a second terminal and run:

cd lab-ui && npm install && npm run dev

Then open http://localhost:5173. You'll see a module-by-module guide covering the full flow from external secrets to rotation to CI/CD.

Keep this terminal running alongside your lab. The Lab UI and the in-cluster app (localhost:3000) are two separate things, the UI guides you through the steps, the app shows you the live secrets.

Step 3: Access the Application

Once the lab finishes, port-forward the service.

kubectl port-forward svc/myapp 3000:80 -n default

Open http://localhost:3000. You should see a table showing each secret key and whether the environment variable value matches the volume mount value.

Step 4: Validate That Secrets Match

Run the compare endpoint directly from the terminal.

curl -s http://localhost:3000/secrets/compare | python3 -m json.tool

When everything is working, the response will include "all_match": true.

How to Inspect the ExternalSecret and the Application

At this point the lab is running. Now you'll want to inspect the manifests so you understand what each part does.

Step 1: Read the ExternalSecret Manifest

Open k8s/aws/external-secret.yaml. Focus on these four fields:

• refreshInterval: how often the operator polls AWS Secrets Manager

• secretStoreRef: which store the operator authenticates against

• target: the name of the Kubernetes Secret to create

• data: the mapping from AWS Secrets Manager JSON keys to Kubernetes Secret keys

Here is what that mapping looks like in this lab:

spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myapp-database-creds
    creationPolicy: Owner
  data:
    - secretKey: DB_USERNAME
      remoteRef:
        key: prod/myapp/database
        property: username

The property field tells the operator which JSON key inside the AWS secret to extract. If your secret in AWS Secrets Manager is a JSON object, each field gets its own entry here.

Two fields here are worth understanding before you move on. creationPolicy: Owner means the operator owns the Kubernetes Secret it creates. If you delete the ExternalSecret, the Secret is deleted too. ClusterSecretStore is a cluster-scoped store, meaning any namespace in the cluster can use it. A plain SecretStore is namespace-scoped. For this lab, cluster-scoped is the right choice because it keeps the setup simple.

Step 2: Read the Deployment Manifest

Open k8s/aws/deployment.yaml. You are looking for two sections: envFrom and volumeMounts.

envFrom:
  - secretRef:
      name: myapp-database-creds

volumeMounts:
  - name: db-secret-vol
    mountPath: /etc/secrets
    readOnly: true

Both paths read from the same Kubernetes Secret, myapp-database-creds. The envFrom block injects all keys as environment variables at pod start.
The volumeMounts block mounts the same secret as files under /etc/secrets.

This is the core of the rotation lesson. Both paths read the same source. But they behave differently after that source changes.

Step 3: Read the App Comparison Logic

Open app/server.js. The comparison logic reads environment variables from process.env and reads mounted secret files from /etc/secrets/<key>. Then it computes a per-key match and a global all_match value.

The /secrets/compare endpoint sets rotation_detected: true when any key differs between env and volume.

How to Test Secret Rotation

Secret rotation is where real teams feel pain. This lab makes that pain visible so you can explain it clearly and fix it confidently.

How the Rotation Gap Works

When a pod starts, Kubernetes gives it two ways to read a secret.

The first way is environment variables. Think of these like sticky notes written on the wall of the container the moment it boots up. The value gets written once, at startup, and never changes. Even if the secret in AWS gets updated ten minutes later, the sticky note still says the old value. The container cannot see the update because nobody rewrote the note.

The second way is a volume mount. Think of this like a shared folder that someone else can update remotely. Kubernetes creates a small folder inside the container and puts the secret value in a file there. When the secret changes in AWS and ESO syncs it into Kubernetes, the kubelet quietly updates that file within about 60 seconds. The container reads the file fresh every time it needs the value, so it sees the new password automatically.

Same secret, two paths. One goes stale while one stays fresh.

The problem happens when your app reads the database password from the environment variable, the sticky note, and someone rotates the password in AWS. ESO updates Kubernetes. The file gets the new password. But your app is still reading the sticky note, which has the old one. Connection fails.

That difference isn't a bug. It's how the Linux process model and the kubelet work. Understanding it is the difference between knowing Kubernetes secrets and actually operating them.

Here is what you're about to observe in the lab:

• The rotation script updates the secret in AWS

• ESO syncs the new value into Kubernetes within seconds

• The volume file updates automatically

• The environment variable stays stale until the pod restarts

• The /secrets/compare endpoint shows both values side by side so you can see the gap live

Step 1: Confirm the Lab Is Ready

Make sure your pod and the External Secrets Operator are both running before you start.

kubectl get pods -n external-secrets
kubectl get pods -n default

Both should show Running.

Step 2: Run the Rotation Test Script

bash rotation/test-rotation.sh

The script performs these actions in order:

1. Reads the current DB_PASSWORD from the volume mount at /etc/secrets/DB_PASSWORD

2. Reads the current DB_PASSWORD from the environment variable

3. Updates AWS Secrets Manager with a new password using put-secret-value

4. Forces an immediate ESO sync by annotating the ExternalSecret with force-sync

5. Reads the volume value again

6. Reads the environment variable again

After the script runs, the volume and the env var will show different values.

Step 3: Validate With the Compare Endpoint

Hit the compare endpoint and look at the output.

curl -s http://localhost:3000/secrets/compare | python3 -m json.tool

You'll see something like this:

{
  "comparison": {
    "DB_PASSWORD": {
      "env": "old-password-value",
      "volume": "new-password-value",
      "match": false
    }
  },
  "all_match": false,
  "rotation_detected": true,
  "message": "Volume has new value; env still has old value."
}

Step 4: Restart the Deployment to Sync Env Vars

Env vars don't update in place. You need a pod restart so new containers start with the updated Kubernetes Secret.

kubectl rollout restart deployment/myapp -n default
kubectl rollout status deployment/myapp -n default

Then hit /secrets/compare again. All rows should now show "all_match": true.

How to Automate Restarts With Reloader

If you don't want to restart deployments manually after every rotation, you can install Stakater Reloader. It watches an annotation on the Deployment and triggers a rolling restart automatically when the referenced Kubernetes Secret changes. New pods start with fresh env vars, while old pods drain cleanly. The repo's local deployment guide includes the install steps.

How to Choose Between External Secrets Operator and the CSI Driver

Two patterns dominate when it comes to pulling external secrets into Kubernetes: the External Secrets Operator and the Secrets Store CSI Driver.

Both get cloud secrets into pods, but they do it differently. Here's a plain comparison:

This lab uses the External Secrets Operator for two reasons. First, it produces a native Kubernetes Secret, which means your application and deployment patterns match standard Kubernetes workflows. Second, having both envFrom and a volume mount point to the same Secret makes the rotation behavior easy to observe side by side.

Use the CSI Driver when your security team prohibits storing secrets in etcd through a Kubernetes Secret. The driver mounts secret data directly into the pod file system without creating a Kubernetes Secret. The tradeoff is that you lose the native envFrom model.

How to Deploy the Pattern on Amazon Elastic Kubernetes Service

The local lab is ideal for learning. The Amazon Elastic Kubernetes Service path adds the production-like pieces: IAM role-based permissions for the operator, a load balancer for the app, and a full CI/CD workflow.

Step 1: Prepare Terraform and OpenID Connect Access

The repository includes a one-time setup guide for OpenID Connect-based access from GitHub Actions to AWS. Run these commands in the terraform/github-oidc folder.

cd terraform/github-oidc
terraform init
terraform plan -var="github_repo=YOUR_ORG/YOUR_REPO"
terraform apply -var="github_repo=YOUR_ORG/YOUR_REPO"
terraform output role_arn

Copy the role ARN from the output. You'll need it in the next step.

Step 2: Set the Required Environment Variable

The Amazon Elastic Kubernetes Service spin-up path needs your GitHub Actions role ARN so Terraform can grant the CI/CD runner access to the cluster.

To find your AWS account ID, run:

aws sts get-caller-identity --query Account --output text

Then set the variable, replacing ACCOUNT with the number that command returns.

export GITHUB_ACTIONS_ROLE_ARN=arn:aws:iam::ACCOUNT:role/your-github-oidc-role

Step 3: Run the Spin-Up Script for Amazon Elastic Kubernetes Service

bash spinup.sh --cluster eks

When the script finishes, it prints the application URL. Open that URL in a browser and confirm that you see the same secrets table you saw locally, with all keys showing Match ✓.

Step 4: Test Rotation on the Deployed App

After you confirm normal operation, run the rotation test the same way you did locally.

bash rotation/test-rotation.sh

Then use /secrets/compare on the Amazon Elastic Kubernetes Service load balancer URL to validate behavior in the cloud environment.

⚠️ Cost warning: Amazon Elastic Kubernetes Service runs at approximately $0.16 per hour. When you're done with the lab, run bash teardown.sh from the repo root to destroy all AWS resources and stop charges.

How to Configure GitHub Actions Without Stored AWS Credentials

The typical CI/CD setup stores AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub repository secrets. Those keys never rotate. Anyone with repo access can read them. When someone leaves the team, you have to revoke keys and update every workflow.

OpenID Connect eliminates that problem entirely.

How OpenID Connect Works for GitHub Actions

GitHub can issue a short-lived token for each workflow run. That token identifies the run: the repository, branch, and workflow name. You create an IAM role in AWS whose trust policy says: only accept requests that come from this specific GitHub repository and branch. The GitHub Actions runner exchanges that token for temporary AWS credentials via AssumeRoleWithWebIdentity. No long-lived keys are ever stored anywhere.

Step 1: Create the IAM Role With Terraform

The terraform/github-oidc folder creates the OpenID Connect provider and the IAM role for you. You already ran this in the Amazon Elastic Kubernetes Service setup above. The role ARN is the only value you need to store.

Step 2: Add the Role ARN to GitHub Repository Secrets

In your GitHub repository:

1. Go to Settings → Secrets and variables → Actions

2. Click New repository secret

3. Name it AWS_ROLE_ARN

4. Paste the role ARN from the Terraform output

That is the only secret you store. The role ARN isn't sensitive. It's an identifier, not a credential.

Step 3: Configure Terraform State

For CI/CD to work consistently across runs, Terraform needs a shared state backend. The lab stores Terraform state in an Amazon S3 bucket and uses an Amazon DynamoDB table for state locking. The Amazon Elastic Kubernetes Service deployment guide in the repo covers the backend setup in full.

Step 4: Push to Main and Let Workflows Run

After your first spin-up, every push to the main branch drives the CI/CD pipeline. The repo includes separate workflow files for Terraform infrastructure changes and application deployment changes. Once your application is reachable, use /secrets/compare to validate rotation behavior on the live environment.

How to Troubleshoot the Most Common Failures

Here's a shortlist of the most common symptoms and their fixes.

How to Inspect the Operator and the ExternalSecret

When something isn't syncing, start with these two commands.

# Check the ExternalSecret status
kubectl describe externalsecret app-db-secret -n default

# Check the ESO operator logs
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

The status conditions on the ExternalSecret resource will usually tell you exactly what failed.

How to Validate Rotation From the App Side

When you are debugging rotation, don't rely only on Kubernetes resource state. Use the /secrets/compare endpoint to see what the running application actually observes. The endpoint tells you whether env and volume match and whether rotation has been detected. That is the ground truth for your application's behavior.

Conclusion

You now have a complete secrets pipeline from AWS Secrets Manager into Kubernetes pods using Terraform and the External Secrets Operator. You ran the local lab, inspected the ExternalSecret and Deployment manifests, and validated that the application sees the right credentials.

You also tested secret rotation and observed the key behavior firsthand: mounted secret files update within the kubelet sync period, while environment variables stay stale until the pod restarts. That single observation explains a large class of production incidents.

Finally, you saw how the same design extends to Amazon Elastic Kubernetes Service with OpenID Connect-based CI/CD, and you have a troubleshooting checklist for the failures most teams hit.

The lab repository is at github.com/Osomudeya/k8s-secret-lab. If you ran the local lab, the natural next step is phases 4 and 5 from the repo's staged learning path: try the CSI driver path on Microk8s, then follow the EKS setup to see the same pipeline with a real CI/CD workflow and no credentials stored in GitHub. Both are documented in the repo and take less than 30 minutes each.

If this helped you, star the repo and share it with someone who is learning Kubernetes.

I send weekly breakdowns of real production incidents and how engineers actually fix them, not tutorials but real failures
→ Join the newsletter

An attacker had found it, deployed pods inside Tesla's cluster, and was using them to mine cryptocurrency – all on Tesla's AWS bill. The cluster had no authentication on the dashboard, no network restrictions on egress, and nothing monitoring for intrusion. Any one of those controls would have stopped the attack. None of them were in place.

This wasn't a sophisticated zero-day exploit. It was a misconfigured default.

Kubernetes ships with powerful security primitives. The problem is that almost none of them are enabled by default. A fresh cluster is deliberately permissive so it's easy to get started. That permissiveness is a feature in development. In production, it's a liability.

In this handbook, we'll work through the three most impactful security layers in Kubernetes. We'll start with Role-Based Access Control, which governs who can do what to which resources in the API. From there we'll move to pod runtime security, which locks down what containers can actually do once they're running on a node. Finally we'll deploy Falco, a syscall-level detection engine that watches for attacks in progress and alerts in real time.

By the end, you'll have a hardened cluster with working RBAC policies, enforced pod security standards, and live detection rules that fire when something suspicious happens.

Prerequisites

• kubectl installed and configured

• Docker Desktop or a Linux machine (to run kind)

• Basic Kubernetes familiarity – you know what a Pod, Deployment, and Namespace are

• No prior security experience needed

All demos run on a local kind cluster. Full YAML and setup scripts are in the companion GitHub repository.

Table of Contents

• The Kubernetes Threat Landscape

• What You'll Build

• Demo 1 — Run a Cluster Security Baseline with kube-bench

• How to Configure RBAC

  • The Four RBAC Objects

  • How to Discover Resources, Verbs, and API Groups

  • Roles and ClusterRoles

  • RoleBindings and ClusterRoleBindings

  • How to Use Service Accounts Safely

  • How to Audit Your RBAC Configuration

• Demo 2 — Build a Least-Privilege RBAC Policy for a CI Pipeline

• Demo 3 — Audit RBAC with rakkess and rbac-lookup

• How to Harden Pod Runtime Security

  • Pod Security Admission

  • How to Configure securityContext

  • OPA/Gatekeeper vs Kyverno

  • How to Detect Runtime Threats with Falco

• Demo 4 — Harden a Pod with securityContext

• Demo 5 — Deploy Falco and Write a Custom Detection Rule

• Cleanup

• Conclusion

The Kubernetes Threat Landscape

To understand what you're defending against, you need to understand where Kubernetes exposes attack surface. There are six main areas, and most production incidents trace back to at least one of them.

The API server is the front door to your cluster. Every kubectl command, every CI deploy, and every controller reconciliation loop sends requests here. Unauthenticated or over-privileged access to the API server is effectively game over: an attacker who can talk to it can create pods, read secrets, and modify workloads freely.

etcd is the key-value store where all cluster state lives, including your Secrets. Kubernetes Secrets are base64-encoded by default, not encrypted. Anyone with direct access to etcd can read every password, token, and certificate in the cluster without going through the API server at all.

The kubelet runs on each node and manages the pods assigned to it. If its API is reachable without authentication – which is the default on older clusters – an attacker can exec into any pod on that node and read its memory without ever touching the API server.

The container runtime is the layer that actually runs your containers. A container that escapes its isolation boundary lands directly in the host OS. A privileged container with hostPID: true can read the memory of every other process on the node, including other containers.

Your supply chain (base images, third-party dependencies, Helm charts, operators) is a potential entry point at every step. The XZ Utils backdoor discovered in 2024 showed how close a well-positioned supply chain attack can come to widespread infrastructure compromise.

Finally, the network: by default, every pod in a Kubernetes cluster can reach every other pod on any port. There are no internal firewalls between workloads unless you explicitly create them with NetworkPolicy.

Real-World Breaches

These three incidents are worth understanding before you write a single line of YAML. They're not theoretical – they're documented post-mortems from real production clusters.

The pattern across all three: not zero-day exploits, but misconfigured defaults and missing controls that should have been standard practice.

This article addresses the RBAC and pod security gaps directly.

What You'll Build

Before the first command, here is the security posture you'll have by the end of this article:

You'll start by running kube-bench to get a CIS Benchmark baseline – a concrete score showing where a default cluster stands before any hardening. From there you'll build a least-privilege RBAC policy for a CI pipeline service account and verify its permission boundaries, then audit the full cluster to confirm no over-privileged accounts exist.

On the pod security side, you'll enforce the restricted Pod Security Admission profile on your workload namespace and apply a hardened securityContext to a deployment: non-root user, read-only root filesystem, dropped capabilities, and seccomp profile. To close out, you'll deploy Falco in eBPF mode with a custom detection rule that fires when suspicious tools are run inside a container.

Start to finish, with a kind cluster already running, the demos take about 45–60 minutes.

Demo 1: Run a Cluster Security Baseline with kube-bench

Before hardening anything, it's a good idea to measure where you are. kube-bench runs the CIS Kubernetes Benchmark against your cluster and reports which checks pass and which fail. A baseline run gives you a concrete picture of your cluster's default security posture – and a reference point you can re-run after applying any hardening changes.

Step 1: Create a kind cluster

Save the following as kind-config.yaml:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --name k8s-security --config kind-config.yaml

Expected output:

Creating cluster "k8s-security" ...
 ✓ Ensuring node image (kindest/node:v1.29.0) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-k8s-security"

Step 2: Run kube-bench

kube-bench runs as a Job inside the cluster, mounting the host filesystem to inspect Kubernetes configuration files and processes:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl wait --for=condition=complete job/kube-bench --timeout=120s
kubectl logs job/kube-bench

The output is long. Scroll to the summary at the bottom:

== Summary master ==
0 checks PASS
11 checks FAIL
 9 checks WARN
 0 checks INFO

== Summary node ==
17 checks PASS
 2 checks FAIL
40 checks WARN
 0 checks INFO

A fresh kind cluster typically fails around 14 checks. Three of the most important failures explain why defaults are a problem:

Note: Some kube-bench findings are expected on kind because kind is a development tool, not a production-hardened environment. The important thing is to understand what each finding means and whether it applies to your target production setup.

Delete the Job when you're done:

kubectl delete job kube-bench

Now that you have a baseline, you know what you're starting from. The next step is to work through the most impactful control on that list: access control. RBAC governs every interaction with the Kubernetes API, and getting it right is the foundation everything else builds on.

How to Configure RBAC

Role-Based Access Control is the authorisation layer in Kubernetes. Every request that reaches the API server – from kubectl, from a pod, from a controller – is checked against RBAC rules after authentication succeeds. If there is no rule that explicitly allows the action, Kubernetes denies it.

The key word is "explicitly". RBAC in Kubernetes is additive only. There is no deny rule. You grant access by creating rules, and you remove access by deleting them. This makes the mental model clean: if a subject can do something, you gave it permission to do that thing.

A Brief Case Study: The Shopify Kubernetes Misconfiguration

In 2021, security researcher Silas Cutler discovered that a Shopify internal service exposed Kubernetes metadata through an SSRF vulnerability. The metadata included pod environment variables that contained secrets. The root cause was partly RBAC: the service's service account had broader cluster access than it needed, and there was no least-privilege review process.

Shopify paid a $25,000 bug bounty and fixed the issue. The lesson is straightforward: a service account should only have the permissions it needs to do its specific job. Nothing more.

This is the principle you'll apply in Demo 2.

The Four RBAC Objects

RBAC in Kubernetes is built from four API objects. Two define permissions, two bind those permissions to subjects:

A subject is a user, a group, or a service account. Users and groups come from your authentication layer – client certificates, OIDC tokens, or cloud provider identity. Service accounts are Kubernetes-native identities created for pods.

How to Discover Resources, Verbs, and API Groups

Before you can write a Role, you need to know three things: the resource name, the API group it belongs to, and the verbs it supports. You shouldn't have to guess any of them – kubectl can tell you everything.

List all available resources and their API groups

kubectl api-resources

Partial output:

NAME                    SHORTNAMES  APIVERSION                     NAMESPACED  KIND
bindings                            v1                             true        Binding
configmaps              cm          v1                             true        ConfigMap
endpoints               ep          v1                             true        Endpoints
events                  ev          v1                             true        Event
namespaces              ns          v1                             false       Namespace
nodes                   no          v1                             false       Node
pods                    po          v1                             true        Pod
secrets                             v1                             true        Secret
serviceaccounts         sa          v1                             true        ServiceAccount
services                svc         v1                             true        Service
deployments             deploy      apps/v1                        true        Deployment
replicasets             rs          apps/v1                        true        ReplicaSet
statefulsets            sts         apps/v1                        true        StatefulSet
cronjobs                cj          batch/v1                       true        CronJob
jobs                                batch/v1                       true        Job
ingresses               ing         networking.k8s.io/v1           true        Ingress
networkpolicies         netpol      networking.k8s.io/v1           true        NetworkPolicy
clusterroles                        rbac.authorization.k8s.io/v1   false       ClusterRole
roles                               rbac.authorization.k8s.io/v1   true        Role

The APIVERSION column is what you put in apiGroups. Strip the version suffix and use only the group part:

The NAMESPACED column tells you whether to use a Role (namespaced resources) or a ClusterRole (non-namespaced resources like nodes).

Filter by API group

If you want to see only resources in a specific group, for example, everything in apps:

kubectl api-resources --api-group=apps

NAME                  SHORTNAMES  APIVERSION  NAMESPACED  KIND
controllerrevisions               apps/v1     true        ControllerRevision
daemonsets            ds          apps/v1     true        DaemonSet
deployments           deploy      apps/v1     true        Deployment
replicasets           rs          apps/v1     true        ReplicaSet
statefulsets          sts         apps/v1     true        StatefulSet

List all verbs for a specific resource

Each resource supports a different set of verbs. To see exactly which verbs a resource supports, use kubectl api-resources with -o wide and look at the VERBS column:

kubectl api-resources -o wide | grep -E "^NAME|^pods "

NAME  SHORTNAMES  APIVERSION  NAMESPACED  KIND  VERBS
pods  po          v1          true        Pod   create,delete,deletecollection,get,list,patch,update,watch

Or explain the resource directly:

kubectl explain pod --api-version=v1 | head -10

The full set of verbs Kubernetes supports in RBAC rules is:

Important: get and list are separate verbs. Granting list on secrets lets a subject enumerate every secret name and value in a namespace, even if you didn't also grant get. Always think about both when working with sensitive resources like secrets, serviceaccounts, and configmaps.

Look up a resource's group with kubectl explain

If you already know the resource name but aren't sure of its group, kubectl explain tells you:

kubectl explain deployment

GROUP:      apps
KIND:       Deployment
VERSION:    v1
...

kubectl explain ingress

GROUP:      networking.k8s.io
KIND:       Ingress
VERSION:    v1
...

This is the fastest way to look up the apiGroups value for any resource when writing a Role.

A complete lookup workflow

Here is the practical workflow when writing a new Role from scratch:

# 1. Find the resource name and API group
kubectl api-resources | grep deployment

# Output:
# deployments   deploy   apps/v1   true   Deployment

# 2. Find the verbs it supports
kubectl api-resources -o wide | grep deployment

# Output:
# deployments   deploy   apps/v1   true   Deployment   create,delete,...,get,list,patch,update,watch

# 3. Write the Role using the group (strip the version) and the verbs you need

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployment-reader
  namespace: staging
rules:
  - apiGroups: ["apps"]       # from: apps/v1 → strip /v1
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

With this workflow, you never have to guess an API group or verb. You look it up, then write the minimal rule you need.

Roles and ClusterRoles

A Role defines which verbs are allowed on which resources. Here is a Role that grants read-only access to Pods and ConfigMaps inside the staging namespace:

# role-ci-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-reader
  namespace: staging
rules:
  - apiGroups: [""]          # "" = the core API group (Pods, Services, Secrets, ConfigMaps)
    resources: ["pods", "configmaps"]
    verbs: ["get", "list", "watch"]

The apiGroups field tells Kubernetes which API group owns the resource. The core group uses an empty string "". Apps-level resources like Deployments use "apps". Custom resources use their own group, such as "networking.k8s.io".

A ClusterRole is structurally identical but omits the namespace and can reference cluster-scoped resources like Nodes and PersistentVolumes:

# clusterrole-node-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader    # no namespace field
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]

When to use which:

Use a Role when the permission is specific to one namespace. A compromised service account can only affect that namespace: the blast radius is contained. Use a ClusterRole when you need access to cluster-scoped resources, or when you want a reusable permission template that multiple namespaces can share.

A common mistake is reaching for a ClusterRole "just to be safe" because it's easier to configure. Namespace-scoped Roles are almost always the right default.

RoleBindings and ClusterRoleBindings

A Role by itself does nothing. You need a binding to attach it to a subject. Here is a RoleBinding that grants the ci-reader Role to the ci-pipeline service account:

# rolebinding-ci.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-reader-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline       # the service account name
    namespace: staging      # the namespace the SA lives in
roleRef:
  kind: Role
  name: ci-reader           # must match the Role name exactly
  apiGroup: rbac.authorization.k8s.io

There is a useful pattern worth knowing: you can bind a ClusterRole using a RoleBinding. This creates namespace-scoped access using a reusable permission template. The ClusterRole defines the rules, while the RoleBinding constrains those rules to a single namespace.

# RoleBinding referencing a ClusterRole — scoped to one namespace only
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: view-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline
    namespace: staging
roleRef:
  kind: ClusterRole          # ClusterRole, but bound to one namespace via RoleBinding
  name: view                 # Kubernetes built-in ClusterRole: read-only access to most resources
  apiGroup: rbac.authorization.k8s.io

Kubernetes ships with several useful built-in ClusterRoles: view (read-only access to most resources), edit (read/write to most resources), admin (full namespace admin), and cluster-admin (full cluster admin). Use them rather than reinventing them.

How to Use Service Accounts Safely

Every pod in Kubernetes runs as a service account. If you don't specify one, Kubernetes uses the default service account in that namespace.

The default service account starts with no permissions – but it still has a token automatically mounted into every pod at /var/run/secrets/kubernetes.io/serviceaccount/token. This means every container in your cluster can authenticate to the API server by default, even if it has nothing useful to do there.

The single most impactful change you can make is to disable this automatic token mounting on service accounts that don't need API access:

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
automountServiceAccountToken: false   # no token mounted into pods by default

You can also control it at the pod level:

spec:
  automountServiceAccountToken: false   # override at pod level
  serviceAccountName: my-app
  containers:
    - name: app
      image: my-app:1.0

The cluster-admin anti-pattern:

Never bind cluster-admin to a service account that runs in a pod. cluster-admin grants full read/write access to every resource in the cluster. An attacker who compromises a pod running as cluster-admin owns your cluster completely.

You will see this in Helm charts and tutorials because it "makes things work". It works because it disables the entire authorisation layer. That is not a solution – it's a ticking clock.

The Capital One breach is a direct example of this pattern at the cloud layer: an EC2 instance role had permissions far beyond what the application needed. The SSRF vulnerability was the initial foothold. The over-privileged role was what turned a minor bug into a $80 million fine.

How to Audit Your RBAC Configuration

The kubectl auth can-i command lets you check permissions for any subject. Use --as to impersonate a service account:

SA="system:serviceaccount:staging:ci-pipeline"

# These should return 'yes'
kubectl auth can-i list pods        --namespace staging --as $SA
kubectl auth can-i get  configmaps  --namespace staging --as $SA

# These should return 'no'
kubectl auth can-i delete pods      --namespace staging --as $SA
kubectl auth can-i get  secrets     --namespace staging --as $SA
kubectl auth can-i list pods        --namespace production --as $SA

To list every permission a subject has in a namespace:

kubectl auth can-i --list \
  --namespace staging \
  --as system:serviceaccount:staging:ci-pipeline

For a visual matrix across the whole cluster, install rakkess (part of krew):

kubectl krew install access-matrix

# Permission matrix for all service accounts in staging
kubectl access-matrix --namespace staging

Example output:

NAME          GET  LIST  WATCH  CREATE  UPDATE  PATCH  DELETE
ci-pipeline    ✓    ✓     ✓      ✗       ✗       ✗      ✗
default        ✗    ✗     ✗      ✗       ✗       ✗      ✗
monitoring     ✓    ✓     ✓      ✗       ✗       ✗      ✗

If you see ✓ in the CREATE, UPDATE, PATCH, or DELETE columns for a service account that should only read, that's a finding that needs remediation.

⚠️ The wildcard danger: The most dangerous RBAC configuration is a wildcard on all three dimensions:

apiGroups: [""] 
resources: [""] 
verbs: ["*"]

This is functionally identical to cluster-admin. You will find it in Helm charts for controllers installed with "convenience" permissions. Always audit third-party RBAC before installing operators into a production cluster.

Demo 2 – Build a Least-Privilege RBAC Policy for a CI Pipeline

In this demo, you'll create a service account for a CI pipeline that can list pods and read configmaps in the staging namespace – and nothing else.

Step 1: Create the namespace and service account

kubectl create namespace staging

# ci-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-pipeline
  namespace: staging
automountServiceAccountToken: false

kubectl apply -f ci-serviceaccount.yaml

Step 2: Create the Role

# ci-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-reader
  namespace: staging
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]

kubectl apply -f ci-role.yaml

Step 3: Bind the Role to the service account

# ci-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-reader-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline
    namespace: staging
roleRef:
  kind: Role
  name: ci-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f ci-rolebinding.yaml

Step 4: Test allowed operations

SA="system:serviceaccount:staging:ci-pipeline"

kubectl auth can-i list pods       --namespace staging     --as $SA   # yes
kubectl auth can-i get  pods       --namespace staging     --as $SA   # yes
kubectl auth can-i list configmaps --namespace staging     --as $SA   # yes

Step 5: Test denied operations

kubectl auth can-i delete pods       --namespace staging     --as $SA   # no
kubectl auth can-i get  secrets      --namespace staging     --as $SA   # no
kubectl auth can-i list pods         --namespace production  --as $SA   # no
kubectl auth can-i create deployments --namespace staging    --as $SA   # no

All four should return no. Notice the third test: even if there were a matching Role in the staging namespace, the service account cannot access production. A RoleBinding cannot cross namespace boundaries, this is by design.

Writing a least-privilege policy for a service account you control is the easy part. The harder part is auditing what already exists in a cluster. That's what Demo 3 covers.

Demo 3 – Audit RBAC with rakkess and rbac-lookup

Now you'll scan the full cluster to surface any accounts with more permissions than they need.

Step 1: Install the tools

kubectl krew install access-matrix
kubectl krew install rbac-lookup

Step 2: Run rakkess across the cluster

# All service accounts in kube-system
kubectl access-matrix --namespace kube-system

# All ServiceAccounts cluster-wide
kubectl access-matrix

Step 3: Find all cluster-admin bindings

There are two ways subjects get cluster-admin access: via a ClusterRoleBinding (cluster-wide), or via a RoleBinding that references the cluster-admin ClusterRole (namespace-scoped, still dangerous). Check both:

# Find ClusterRoleBindings that grant cluster-admin
kubectl rbac-lookup cluster-admin --kind ClusterRole --output wide

On a fresh kind cluster this returns:

No RBAC Bindings found

That is the correct and expected result. A default kind cluster doesn't create any ClusterRoleBindings to cluster-admin. The role exists, but nothing is bound to it at the cluster level by default. If you see entries here in your production cluster, each one is a finding worth investigating.

To find who has cluster-level admin access through other means, query the bindings directly:

# Find all ClusterRoleBindings and the subjects they grant
kubectl get clusterrolebindings -o wide

NAME                                                   ROLE                                                                       AGE   USERS                         GROUPS                         SERVICEACCOUNTS
cluster-admin                                          ClusterRole/cluster-admin                                                  10d   system:masters
system:kube-controller-manager                         ClusterRole/system:kube-controller-manager                                 10d
system:kube-scheduler                                  ClusterRole/system:kube-scheduler                                          10d
system:node                                            ClusterRole/system:node                                                    10d
...

The cluster-admin ClusterRoleBinding grants access to the system:masters group – the group your kubeconfig certificate belongs to. This is expected. Every other binding in this list is worth reviewing to understand what it grants and why.

What to look for: Any binding where the SERVICEACCOUNTS column is populated with an application service account (not a system: prefixed one) is a potential over-privilege finding. Application pods should never need cluster-admin.

Step 4: Verify the ci-pipeline service account

kubectl rbac-lookup ci-pipeline --kind ServiceAccount --output wide

Expected output:

SUBJECT                               SCOPE     ROLE             SOURCE
ServiceAccount/staging:ci-pipeline    staging   Role/ci-reader   RoleBinding/ci-reader-binding

The format is /<role-name> <binding-kind>/<binding-name>. This tells you:

• The service account is bound to the ci-reader Role

• The binding is a RoleBinding named ci-reader-binding

• There is no namespace prefix on the role name because it is a namespaced Role, not a ClusterRole

If the output showed ClusterRole/something here, that would be a finding. It would mean the service account has cluster-wide permissions, not namespace-scoped ones.

rbac-lookup vs kubectl get: rbac-lookup gives you a subject-centric view: "what does this account have access to?" kubectl get rolebindings,clusterrolebindings -A gives you a binding-centric view: "what bindings exist in the cluster?" Use both. rbac-lookup is faster for auditing a specific service account, while the kubectl get approach is better for a full cluster inventory.

With RBAC locked down, the API server is protected. But RBAC says nothing about what a container can do once it's running. That's a separate layer entirely.

How to Harden Pod Runtime Security

RBAC controls who can talk to the Kubernetes API. Pod security controls what containers can do once they're running on a node. These are different threat vectors: RBAC protects the control plane, pod security protects the data plane.

A container that runs as root with no capability restrictions can, if compromised, write backdoors to the host filesystem, load kernel modules, read the memory of other processes if hostPID: true is set, and in some configurations escape the container entirely. Pod security closes these doors before an attacker can open them.

A Case Study: The Hildegard Malware Campaign

In early 2021, Palo Alto's Unit 42 research team documented a cryptomining malware campaign called Hildegard that specifically targeted Kubernetes clusters. The attack chain was:

1. Find a cluster with the kubelet API exposed without authentication

2. Deploy a privileged pod with hostPID: true

3. Use the privileged pod to read credentials from other containers' memory

4. Establish persistence by writing to the host filesystem

Steps 3 and 4 would have been impossible if the pods in the cluster had been running with readOnlyRootFilesystem: true, dropped capabilities, and no hostPID. The attacker had the initial foothold. Pod security would have contained the blast radius.

Pod Security Admission

Pod Security Admission (PSA) is the built-in admission controller that enforces pod security standards at the namespace level. It replaced PodSecurityPolicy in Kubernetes 1.25.

Migrating from PSP? If you're on Kubernetes < 1.25, you may still be using PodSecurityPolicy, which was removed in 1.25. The migration path is: enable PSA in audit mode first to identify violations, fix them workload by workload, then switch to enforce. For policies PSA cannot express, add Kyverno alongside it.

PSA defines three profiles:

And three enforcement modes:

The migration path: start with audit and warn to identify violations, fix them, then switch to enforce. The two modes can run simultaneously.

Apply them as namespace labels:

# namespace-staging.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: staging
  labels:
    # Start here: audit and warn simultaneously
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Once violations are resolved, add enforce:

kubectl label namespace staging \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  --overwrite

Note: don't use --overwrite here. Without it, if enforce is already set to a different value the command will error – which is exactly what you want. You should see:

namespace/staging labeled

If you see namespace/staging not labeled, it means enforce=restricted and enforce-version=latest were already set to those exact values. Confirm enforcement is active:

kubectl get namespace staging --show-labels

Look for pod-security.kubernetes.io/enforce=restricted in the output. If it's there, enforcement is active.

How to Configure securityContext

A securityContext defines the privilege and access control settings for a pod or container. These are the seven fields you should configure on every production workload:

The annotated YAML below shows all seven in context:

# secure-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: staging
spec:
  replicas: 2
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      securityContext:
        runAsNonRoot: true         # container must run as a non-root user
        runAsUser: 10001           # explicit UID — don't rely on the image's default
        runAsGroup: 10001          # explicit GID
        fsGroup: 10001             # volumes are owned by this group
        seccompProfile:
          type: RuntimeDefault     # use the container runtime's default seccomp profile
      automountServiceAccountToken: false
      containers:
        - name: app
          image: nginx:1.25-alpine
          securityContext:
            allowPrivilegeEscalation: false   # block setuid and sudo inside the container
            readOnlyRootFilesystem: true      # the single highest-impact setting
            capabilities:
              drop:
                - ALL                         # drop every Linux capability
              add: []                         # add back only what is explicitly needed
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-run
              mountPath: /var/run
      volumes:
        # nginx needs writable directories — provide them as emptyDir volumes
        - name: tmp
          emptyDir: {}
        - name: nginx-cache
          emptyDir: {}
        - name: nginx-run
          emptyDir: {}

Why readOnlyRootFilesystem: true is the most important setting:

Most post-exploitation techniques require writing to the filesystem. Dropping a backdoor, modifying a binary, writing a cron job, or installing a keylogger all require a writable filesystem. Set readOnlyRootFilesystem: true and every one of these techniques is blocked.

The downside is that many applications write to directories like /tmp or /var/cache. The fix is to mount emptyDir volumes at those specific paths, as shown above. The rest of the filesystem stays read-only.

What each field prevents:

OPA/Gatekeeper vs Kyverno

PSA covers the fundamentals. But you'll eventually need policies that PSA cannot express: all images must come from your private registry, all pods must have resource limits, no container may use the latest tag. For these, you need a policy engine.

Two mature options exist:

If you're starting fresh, Kyverno gets you to working policies faster. Here is a Kyverno policy that blocks images from outside your trusted registry:

# kyverno-registry-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-registries
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "Images must come from registry.corp.internal/"
        pattern:
          spec:
            containers:
              - image: "registry.corp.internal/*"

How to Detect Runtime Threats with Falco

PSA and securityContext are preventive controls: they block known-bad configurations before pods start. Falco is a detective control. It watches what containers do while they're running and alerts when something looks wrong.

Falco operates at the syscall level using eBPF. It attaches to the Linux kernel and intercepts every system call made by every container on the node – file opens, network connections, process spawns, privilege escalations. It does this without modifying containers, without injecting sidecars, and with minimal overhead.

What Falco detects out of the box:

Falco's default ruleset covers the most common attack patterns. It fires when a shell is opened inside a running container, whether that's a kubectl exec session or a reverse shell from an exploit.

It watches for reads on sensitive files like /etc/shadow, /etc/kubernetes/admin.conf, and /root/.ssh/. It catches the dropper pattern: a binary written to disk and immediately executed. It detects outbound connections to known malicious IPs, writes to /proc or /sys that suggest kernel manipulation, and package managers like apt, yum, or pip being run inside containers that have no business installing software.

Each of these is a rule in Falco's default ruleset. You can extend it with custom rules for your specific workloads – which is exactly what you'll do in Demo 5. But first let's harden the Pod.

Demo 4 – Harden a Pod with securityContext

In this demo, you'll start with a default nginx deployment, observe the PSA violations it triggers, harden it step by step, and confirm it passes under the restricted profile.

Step 1: Apply PSA labels in audit mode

kubectl label namespace staging \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

Step 2: Deploy insecure nginx and observe the warnings

# insecure-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-insecure
  namespace: staging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-insecure
  template:
    metadata:
      labels:
        app: nginx-insecure
    spec:
      containers:
        - name: nginx
          image: nginx:1.25-alpine

kubectl apply -f insecure-nginx.yaml

Expected output (PSA warns but still creates the deployment in warn mode):

Warning: would violate PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false (container "nginx" must set
    securityContext.allowPrivilegeEscalation=false)
  unrestricted capabilities (container "nginx" must set
    securityContext.capabilities.drop=["ALL"])
  runAsNonRoot != true (pod or container "nginx" must set
    securityContext.runAsNonRoot=true)
  seccompProfile not set (pod or container "nginx" must set
    securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx-insecure created

Four violations. Every one of them is a real security gap. But the pod was still created "deployment.apps/nginx-insecure created"

Step 3: Deploy the hardened version

kubectl apply -f secure-deployment.yaml   # the YAML from the securityContext section above

No warnings this time.

Step 4: Switch the namespace to enforce

kubectl label namespace staging \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest

Expected output:

namespace/staging labeled

This is the moment enforcement becomes active. Any new pod that violates the restricted profile will be rejected from this point on.

Step 5: Confirm insecure deployments are now rejected

kubectl delete deployment nginx-insecure -n staging
kubectl apply -f insecure-nginx.yaml

Expected output:

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false ...
deployment.apps/nginx-insecure created

The Deployment object is created. PSA enforces at the pod level, not the Deployment level. The Deployment and its ReplicaSet exist, but every attempt to create a pod is rejected. Check the ReplicaSet:

kubectl get replicaset -n staging -l app=nginx-insecure

NAME                       DESIRED   CURRENT   READY   AGE
nginx-insecure-b668d867b   1         0         0       30s

DESIRED=1 but CURRENT=0. The ReplicaSet cannot create any pods because they're rejected at admission. Describe the ReplicaSet to see the rejection events:

kubectl describe replicaset -n staging -l app=nginx-insecure

Warning  FailedCreate  ReplicaSet "nginx-insecure-b668d867b" create Pod
  "nginx-insecure-xxx" failed: pods is forbidden: violates PodSecurity
  "restricted:latest": allowPrivilegeEscalation != false, unrestricted
  capabilities, runAsNonRoot != true, seccompProfile not set

The hardened deployment continues running with its pods intact. The insecure one has zero pods and never will. This is exactly how PSA is supposed to work.

Step 6: Score the hardened pod with kube-score

kube-score is a static analysis tool that scores Kubernetes manifests against security and reliability best practices:

# macOS
brew install kube-score
# Linux: https://github.com/zegl/kube-score/releases

kube-score score secure-deployment.yaml -v

Expected output (abridged):

apps/v1/Deployment secure-app in staging 
  path=secure-deployment.yaml
    [OK] Stable version
    [OK] Label values
    [CRITICAL] Container Resources
        · app -> CPU limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.cpu
        · app -> Memory limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.memory
        · app -> CPU request is not set
            Resource requests are recommended to make sure that the application can start and run without crashing. Set resources.requests.cpu
        · app -> Memory request is not set
            Resource requests are recommended to make sure that the application can start and run without crashing. Set resources.requests.memory
    [CRITICAL] Container Image Pull Policy
        · app -> ImagePullPolicy is not set to Always
            It's recommended to always set the ImagePullPolicy to Always, to make sure that the imagePullSecrets are always correct, and to always get the image you want.
    [OK] Pod Probes Identical
    [CRITICAL] Container Ephemeral Storage Request and Limit
        · app -> Ephemeral Storage limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.ephemeral-storage
        · app -> Ephemeral Storage request is not set
            Resource requests are recommended to make sure the application can start and run without crashing. Set resource.requests.ephemeral-storage
    [OK] Environment Variable Key Duplication
    [OK] Container Security Context Privileged
    [OK] Pod Topology Spread Constraints
        · Pod Topology Spread Constraints
            No Pod Topology Spread Constraints set, kube-scheduler defaults assumed
    [OK] Container Image Tag
    [CRITICAL] Pod NetworkPolicy
        · The pod does not have a matching NetworkPolicy
            Create a NetworkPolicy that targets this pod to control who/what can communicate with this pod. Note, this feature needs to be supported by the CNI implementation used in the Kubernetes cluster to have an effect.
    [OK] Container Security Context User Group ID
    [OK] Container Security Context ReadOnlyRootFilesystem
    [CRITICAL] Deployment has PodDisruptionBudget
        · No matching PodDisruptionBudget was found
            It's recommended to define a PodDisruptionBudget to avoid unexpected downtime during Kubernetes maintenance operations, such as when draining a node.
    [WARNING] Deployment has host PodAntiAffinity
        · Deployment does not have a host podAntiAffinity set
            It's recommended to set a podAntiAffinity that stops multiple pods from a deployment from being scheduled on the same node. This increases availability in case the node becomes unavailable.
    [OK] Deployment Pod Selector labels match template metadata labels

Notice there are no security context violations: securityContext, readOnlyRootFilesystem, seccompProfile, and runAsNonRoot all pass. The remaining findings are about resource management (CPU/memory limits, ephemeral storage), availability (PodDisruptionBudget, anti-affinity), and network policy – not security context hardening. Those are important for production readiness, but they're a separate concern from the pod security hardening we did here.

You now have a pod that PSA accepts and kube-score validates. The next step is to add a detection layer – something that watches what the pod does at runtime, not just how it was configured at admission.

Demo 5 – Deploy Falco and Write a Custom Detection Rule

Now, you'll deploy Falco in eBPF mode, trigger a default alert, then extend Falco with a custom rule that catches curl and wget being run inside containers.

Step 1: Install Falco via Helm

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set tty=true \
  --wait

Confirm Falco is running on every node:

kubectl get pods -n falco

NAME           READY   STATUS    RESTARTS   AGE
falco-x8k2p    1/1     Running   0          45s
falco-m9nqr    1/1     Running   0          45s
falco-j4tpw    1/1     Running   0          45s

One pod per node. Falco runs as a DaemonSet because it needs to monitor syscalls on every node independently.

Step 2: Trigger a default alert

Open a second terminal and stream the Falco logs:

# Terminal 2 — watch for alerts
kubectl logs -n falco -l app.kubernetes.io/name=falco -f --max-log-requests 3

In your first terminal, exec into the secure-app pod:

# Terminal 1 — trigger the shell detection
POD=$(kubectl get pod -n staging -l app=secure-app \
  -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD -n staging -- sh

Within a second, Terminal 2 shows:

2024-03-15T14:23:41.456Z: Notice A shell was spawned in a container with an attached terminal
  (user=root user_loginuid=-1 k8s.ns=staging k8s.pod=secure-app-7d9f8b-xxx
   container=app shell=sh parent=runc cmdline=sh terminal=34816)
  rule=Terminal shell in container  priority=NOTICE
  tags=[container, shell, mitre_execution]

This is Falco's built-in Terminal shell in container rule firing. It detected the kubectl exec session the moment you ran it.

Step 3: Write a custom rule

The built-in rules are comprehensive, but every production environment has workloads with unique behaviour. Here is a custom rule that alerts when curl or wget is executed inside any container:

# custom-rules.yaml
customRules:
  custom-rules.yaml: |-
    - rule: Suspicious network tool in container
      desc: >
        Detects execution of curl or wget inside a running container.
        These tools are commonly used for data exfiltration, downloading
        attacker payloads, or reaching command-and-control servers.
        Production containers should not be making ad-hoc HTTP requests.
      condition: >
        spawned_process
        and container
        and proc.name in (curl, wget)
      output: >
        Network tool executed in container
        (user=%user.name tool=%proc.name cmd=%proc.cmdline
         pod=%k8s.pod.name ns=%k8s.ns.name image=%container.image)
      priority: WARNING
      tags: [network, exfiltration, custom]

Apply it by upgrading the Helm release:

 helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set driver.kind=modern_ebpf \
  --set tty=true \
  -f custom-rules.yaml

Good, it deployed. Now wait for pods to be ready and test your custom rule:

Step 4: Test the custom rule

# Terminal 1 — run curl inside the container
kubectl exec -it $POD -n staging -- sh -c 'curl https://example.com'

Terminal 2 immediately shows:

2024-03-15T14:31:07.812Z: Warning Network tool executed in container
  (user=root tool=curl cmd=curl https://example.com
   pod=secure-app-7d9f8b-xxx ns=staging image=nginx:1.25-alpine)
  rule=Suspicious network tool in container  priority=WARNING
  tags=[network, exfiltration, custom]

Step 5: Route alerts to Slack with Falcosidekick

Streaming logs is useful during development. In production, you need alerts routed to your alerting pipeline. Falcosidekick handles this with support for Slack, PagerDuty, Datadog, Elasticsearch, and over 50 other outputs:

# falcosidekick-values.yaml
config:
  slack:
    webhookurl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
    minimumpriority: "warning"
    messageformat: >
      [{{.Priority}}] {{.Rule}} |
      pod: {{.OutputFields.k8s.pod.name}} |
      ns: {{.OutputFields.k8s.ns.name}} |
      image: {{.OutputFields.container.image}}

helm install falcosidekick falcosecurity/falcosidekick \
  --namespace falco \
  -f falcosidekick-values.yaml

Tuning Falco for production: A fresh Falco deployment will generate false positives, especially in the first week. Your job is to tune rules to match your workloads' normal behaviour, not to respond to every alert.

Here's the workflow: deploy in staging → identify false positives → add except conditions to rules → validate the false positive rate is low → enable in production with alerting.

Cleanup

To remove everything created in this article:

# Delete the staging namespace and everything in it
kubectl delete namespace staging
 
# Delete Falco and Falcosidekick
helm uninstall falco -n falco
helm uninstall falcosidekick -n falco
kubectl delete namespace falco
 
# Delete the kind cluster entirely
kind delete cluster --name k8s-security

Conclusion

In this handbook, you secured a Kubernetes cluster across three layers: RBAC, pod runtime security, and runtime threat detection.

You built a least-privilege service account, enforced the restricted Pod Security Admission profile, hardened pods with securityContext, deployed Falco for syscall-level detection, and wrote a custom rule to catch suspicious tools inside containers.

Each layer maps to a real-world breach – Tesla, Capital One, Hildegard – showing how these controls would have contained the damage. Run kube-bench again to measure the improvement.

All YAML manifests, Helm values, and setup scripts from this article are available in the companion GitHub repository.

I’ve spent more nights than I care to admit staring at a terminal, trying to figure out why a staging environment suddenly "broke" even though no one supposedly touched it.

We’ve all been there, right? a manual edit here, a quick hotfix there, and suddenly your Git repository is no longer a Source of Truth, it’s a historical document of what used to be running.

Without a reliable strategy, Kubernetes deployments quickly descend into a mess of drift, painful rollbacks, and non-existent audit trails. I learned the hard way that simply storing manifests in Git isn't enough. If your cluster isn't actively listening to your code, you're still working with a gap.

GitOps closes that gap. It turns your cluster into a mirror of your repository. If it isn't in Git, it doesn't exist.

In this tutorial, you aren't just going to read about the theory. You’re going to implement a "Zero-Touch" deployment loop from scratch. We’ll use Argo CD, GitHub Actions, and the Argo CD Image Updater to build a system that builds, tags, and deploys your code the second you hit git push.

Table of Contents

1. Prerequisites

2. What GitOps Really Means

3. What is Argo CD and How Does it Implement GitOps?

4. Preparing the Application Source Code and Repo Structure

5. Automating Image Builds with GitHub Actions

6. How to Install and Access Argo CD

7. Understanding the Argo CD Application

8. Deploying the Application Manifest

9. Automating Updates with Argo CD Image Updater

10. Conclusion

Prerequisites

Before you begin, make sure you have the following ready in your environment:

• A GitHub repository: You'll need a repository (for example, my-gitops-demo) to serve as your Single Source of Truth. If you're following this tutorial from scratch, start with an empty repo.

• A DockerHub account: This will act as your Container Registry. You’ll need this to build, push, and store the Docker images that GitHub Actions creates.

• A running Kubernetes cluster: You can use a local solution like Minikube or Kind, or a cloud-managed service like Amazon EKS or GKE.

• Kubernetes tooling: Ensure kubectl is installed and configured to communicate with your cluster.

• Fundamental K8s knowledge: You should be comfortable with basic Kubernetes concepts like Pods, Deployments, and Services.

Note for Readers with Existing Projects

If you already have a project and want to migrate it to this GitOps workflow, you don't need to start over. You can adapt your existing repository by following these three steps:

1. Standardize your manifests: Move all your existing Kubernetes YAML files into a dedicated Kubernetes-manifest/ directory at the root of your project.

2. Containerize your services: Ensure every service you intend to deploy has a Dockerfile in its respective subdirectory (for example, /main-api/Dockerfile).

3. Prepare for automation: Be ready to replace any manual kubectl apply steps in your current CI pipeline with the automated tagging strategy we’ll implement in the next sections.

What GitOps Really Means

At its core, GitOps is an operational framework that uses Git as the single source of truth for your infrastructure and applications. In a traditional setup, you might run kubectl apply -f deployment.yaml from your laptop. This makes it impossible to track who changed what, leading to "snowflake" clusters that no one can reproduce.

GitOps enforces four key principles:

1. Declarative: You describe the desired state (for example, "3 replicas of Nginx"), not the commands to get there.

2. Versioned and immutable: Your entire state is in Git. If a deployment fails, you git revert to a previous known-good state.

3. Pulled automatically: A software agent (Argo CD) pulls the state from Git.

4. Continuously reconciled: The system constantly fixes "drift." If a developer manually changes a service in the cluster, Argo CD will overwrite it to match Git.

What is Argo CD and How Does it Implement GitOps

Before we dive into the setup, let’s define the tool we'll be working with.

Argo CD is a declarative, GitOps' continuous delivery engine built specifically for Kubernetes. As a graduated project of the Cloud Native Computing Foundation (CNCF), it has become the industry standard for managing modern infrastructure.

Think of Argo CD as a persistent watchdog that lives inside your cluster. To understand why it's so powerful, we have to look at how it differs from traditional CI/CD tools like Jenkins or GitHub Actions.

The "Push" vs. "Pull" Model

Traditional tools like the one I mentioned above use a "Push" model. In this setup, an external pipeline sends commands (like kubectl apply) into your cluster. This is risky because you must store sensitive cluster administrative keys inside your external CI tool. If your CI tool is compromised, your cluster is, too.

Argo CD flips this script using a "Pull" model:

• The bridge: It sits between your Git repo (the "Desired State") and your cluster (the "Live State").

• Continuous monitoring: It watches your Git repo 24/7. The moment it detects a new commit, it "pulls" that change and applies it from inside the cluster.

• Self-healing: If someone manually changes a setting in the cluster (known as "drift"), Argo CD detects the discrepancy and automatically overwrites it to match what is written in Git.

This approach is not only more secure, since no cluster credentials ever leave the environment, but it also ensures that your infrastructure is a perfect, predictable mirror of your code.

Preparing the Application Source Code

Before we automate the build, we need actual code in our repository. We'll create two simple microservices: a Main API and an Auxiliary Service.

Repo Structure

Ensure your repository follows this structure exactly. Consistency in naming is vital for the automation to find your files.

GITOPS-ARGOCD-DEMO/
├── .github/workflows/main.yml
├── auxiliary-service/
│   └── Dockerfile
├── main-api/
│   └── Dockerfile
├── Kubernetes-manifest/
│   ├── aux-api.yaml
│   ├── kustomization.yaml
│   └── main-api.yaml
├── application.yaml
└── image-updater.yaml

Create the Dockerfiles

In each service folder, create a simple Dockerfile so our pipeline has something to build.

main-api/Dockerfile

FROM nginx:alpine
RUN echo "<h1>Main API - Version 1.0</h1>" > /usr/share/nginx/html/index.html
EXPOSE 80

auxiliary-service/Dockerfile

FROM nginx:alpine
RUN echo "<h1>Auxiliary Service - Version 1.0</h1>" > /usr/share/nginx/html/index.html
EXPOSE 80

Automating Image Builds with GitHub Actions

In a professional GitOps workflow, your Kubernetes manifests and your application source code often live in the same repository (or linked ones). While Argo CD handles the deployment, you still need a way to turn your code into Docker images. This is where Continuous Integration (CI) comes in.

I have included a GitHub Actions workflow in this demo to automate this. Every time you push code to the main branch, this pipeline builds your images and pushes them to DockerHub.

The CI Pipeline Workflow

Create a file at .github/workflows/main.yml and add the following:

name: Build and Push Image to DockerHub

on:
  push:
    branches:
      - main
    # Skip builds for image updater commits
    paths-ignore:
      - 'Kubernetes-manifest/**'

jobs:
  docker_build:
    name: Build & Push ${{ matrix.service }}
    environment: argocd-demo
    runs-on: ubuntu-latest
    permissions:
      contents: write

    strategy:
      matrix:
        include:
          - service: aux-service
            dockerfile: auxiliary-service/Dockerfile
          - service: main-service
            dockerfile: main-api/Dockerfile

    env:
      DOCKER_USER: ${{ secrets.DOCKERHUB_USERNAME }}
      RUN_TAG: ${{ github.run_number }}

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ env.DOCKER_USER }}
          password: ${{ secrets.DOCKERHUB_PASSWORD }}

      - name: Build and Push ${{ matrix.service }}
        uses: docker/build-push-action@v6
        with:
          context: .
          file: ${{ matrix.dockerfile }}
          push: true
          tags: \({{ env.DOCKER_USER }}/\){{ matrix.service }}:${{ env.RUN_TAG }}
          cache-from: type=gha,scope=${{ matrix.service }}
          cache-to: type=gha,mode=max,scope=${{ matrix.service }}

Pro tip: The paths-ignore section is critical. Later, the Argo CD Image Updater will write changes back to the Kubernetes-manifest/ folder. Without this ignore rule, your pipeline would trigger itself forever in an infinite loop.

Note: You must add DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD to your GitHub Repo Settings > Secrets.

How to Install and Access Argo CD

Now that your cluster is running, you can install Argo CD. You'll perform the installation using a standard Kubernetes manifest provided by the Argo project.

Step 1: Create the Namespace and Apply the Manifests

In Kubernetes, it is a best practice to keep your administrative tools separate from your applications. You will create a dedicated namespace named argocd and then apply the official installation script from the Argo project. This script includes all the necessary ServiceAccounts, Roles, and Deployments.

Run the following commands in your terminal:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

You'll see a long list of resources being created. Wait a minute or two for the pods to initialize. you can verify that all the core components of Argo CD are running.:

kubectl get all -n argocd

Ensure all pods show a status of Running before proceeding.

Step 2: Access the Argo CD User Interface

To access the dashboard, we use a technique called port forwarding. Since the Argo CD server is running inside the cluster's private network, your browser can't see it yet. Port forwarding creates a secure 'tunnel' between a port on your local machine (8080) and a port on the cluster service (443). This allows you to interact with internal services without exposing them to the public internet.

Run the following command:

kubectl port-forward svc/argocd-server -n argocd 8080:443

You can now open your browser and navigate to https://localhost:8080. Your browser may warn you that the connection is not private because of a self-signed certificate. You can safely click "Advanced" and proceed to the site.

Step 3: How to Log In

The default username for Argo CD is admin. The password is autogenerated during the installation process and is stored securely as a Kubernetes secret.

To retrieve this password, open a new terminal tab (so the port-forwarding keeps running) and run:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Copy the output and use it as the password to log into the dashboard.

Understanding the Argo CD Application

An Argo CD Application is a Custom Resource (CRD) that acts as a "contract" between your Git repo and your cluster. It defines:

• repoURL & path: This tells Argo CD exactly which Git repository to watch and which folder inside that repo contains your YAML manifests.

• destination: This defines where the app should live. We use https://kubernetes.default.svc to point to the local cluster where Argo CD is installed.

• syncPolicy: This is the heart of GitOps. By setting automated with selfHeal: true, we tell Argo CD to automatically fix the cluster if someone manually changes something (drift). The prune: true setting ensures that if you delete a file in Git, it also gets deleted in the cluster.

The Application Manifest

Create application.yaml in your project root:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gitops-argocd-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_REPO_NAME>.git
    targetRevision: HEAD
    path: Kubernetes-manifest
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd-demo-ns
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

Deploying the Application Manifest

Now we'll define our Kubernetes resources in the Kubernetes-manifest/ folder.

main-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: main-deployment
  namespace: argocd-demo-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: main-api
  template:
    metadata:
      labels:
        app: main-api
    spec:
      containers:
      - name: main-service
        image: <YOUR_DOCKERHUB_USERNAME>/main-service:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: main-service-lb
  namespace: argocd-demo-ns
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: main-api

aux-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aux-deployment
  namespace: argocd-demo-ns
spec:
  replicas: 2
  selector:
    matchLabels:
      app: aux-service
  template:
    metadata:
      labels:
        app: aux-service
    spec:
      containers:
      - name: aux-service
        image: <YOUR_DOCKERHUB_USERNAME>/aux-service:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: aux-service
  namespace: argocd-demo-ns
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: aux-service

Push and Sync

Step 1: Apply the Application Manifest

Use kubectl to deploy this manifest into the argocd namespace:

kubectl apply -f application.yaml -n argocd

Step 2: Push to Your Repository

To trigger the initial deployment and ensure Argo CD stays in sync with your source of truth, add, commit, and push your latest changes to the GitHub repository you configured in the manifest:

git add .
git commit -m "initial argo application deployment"
git push origin main

Step 3: Verify the Result in Argo CD

Once you push your changes, head over to your Argo CD dashboard. You'll see the gitops-argocd-demo application appear. After the initial sync, the dashboard will display a healthy, green status indicating that your live cluster state perfectly matches your Git repository.

Note: As you can see in the screenshot above, Argo CD provides a visual representation of how your Kubernetes objects – Services, Deployments, and Pods – are related and confirms they are "Synced" with your Git repo.

Automating Updates with Argo CD Image Updater

Now that we have automated the deployment, let’s solve the final manual hurdle: automatically updating image tags in our manifests whenever a new build is pushed to DockerHub.

Step 1: Install ArgoCD Image Updater

Install the Image Updater into the argocd namespace:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/config/install.yaml

Verify the pod is running:

kubectl get pods -n argocd | grep image-updater

Note: Version 1.1+ uses a CRD-based approach (ImageUpdater custom resources) instead of the annotation-based approach used in older versions. This guide covers the CRD method.

Step 2: Create a GitHub Personal Access Token

The Image Updater needs Git credentials to push write-back commits to your repository.

1. Go to GitHub → Settings → Developer Settings → Personal Access Tokens → Tokens (classic)

2. Click Generate new token

3. Select the repo scope (full control of private repositories)

4. Copy the generated token

Step 3: Create the Git Credentials Secret

Store the GitHub credentials as a Kubernetes secret in the argocd namespace:

kubectl -n argocd create secret generic git-creds \
  --from-literal=username=<YOUR_GITHUB_USERNAME> \
  --from-literal=password=<YOUR_GITHUB_PAT>

Replace <YOUR_GITHUB_USERNAME> and <YOUR_GITHUB_PAT> with your actual values.

Step 4: Add a Kustomization File to Your Manifests

The Image Updater uses Kustomize's images field to write updated tags. If your Kubernetes-manifest/ directory contains plain YAML files, you'll need to wrap them with a kustomization.yaml file.

Create a kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - main-api.yaml
  - aux-api.yaml

How it works: When the Image Updater detects a new tag, it appends an images section to this file:

images:
  - name: <YOUR_GITHUB_USERNAME>/main-service
    newTag: "12"
  - name: <YOUR_GITHUB_USERNAME>/aux-service
    newTag: "12"

Kustomize then overrides the image tags at deploy time, without modifying your original deployment YAML files.

We use Kustomize here because it allows the Image Updater to manage image tags in a separate, clean way. Instead of the Updater 'messing' with your original main-api.yaml file, it simply updates the kustomization.yaml file. Argo CD then uses Kustomize to merge those changes during deployment.

Step 5: Create the ImageUpdater Custom Resource

Create image-updater.yaml in your project root:

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: gitops-argocd-demo-updater
  namespace: argocd
spec:
  commonUpdateSettings:
    updateStrategy: newest-build
    allowTags: "regexp:^[0-9]+$"
  applicationRefs:
    - namePattern: "gitops-argocd-demo"
      writeBackConfig:
        method: "git:secret:argocd/git-creds"
        gitConfig:
          branch: main
          writeBackTarget: "kustomization:."
      images:
        - alias: main-service
          imageName: <YOUR_DOCKERHUB_USERNAME>/main-service
        - alias: aux-service
          imageName: <YOUR_DOCKERHUB_USERNAME>/aux-service

This ImageUpdater resource is the "brain" of our automated tagging system. Here is what the specific fields are doing:

updateStrategy:

• newest-build: It tells the updater to always look for the most recent image version in DockerHub based on creation time.

writeBackConfig: This is where the magic happens. It uses the git-creds secret we created to authorize the updater to 'write' back to your repository.

writeBackTarget:

• kustomization: We are telling the updater specifically to modify the kustomization.yaml file in the manifests folder rather than touching the deployment files directly.

images: We provide aliases (main-service and aux-service) so the updater knows exactly which images in DockerHub correspond to which containers in our Kubernetes manifests.

Apply the ImageUpdater CR to the cluster:

kubectl apply -f image-updater.yaml -n argocd

Push the kustomization.yaml to your Git repository (the Image Updater clones the repo, so it must exist remotely):

git add Kubernetes-manifest/kustomization.yaml
git commit -m "Add kustomization.yaml for image updater write-back"
git push origin main

Step 6: Verify the Image Updater

Check the Image Updater logs to confirm it's working:

kubectl logs -n argocd deployment/argocd-image-updater-controller --tail=20

Successful output looks like:

msg="Starting image update cycle, considering 1 application(s) for update"
msg="Setting new image to YOUR_DOCKERHUB_USERNAME/main-service:11"
msg="Successfully updated image 'YOUR_DOCKERHUB_USERNAME/main-service:7' to 'YOUR_DOCKERHUB_USERNAME/main-service:11'"
msg="Setting new image to YOUR_DOCKERHUB_USERNAME/aux-service:11"
msg="Committing 2 parameter update(s) for application gitops-argocd-demo"
msg="git push origin main"
msg="Successfully updated the live application spec"
msg="Processing results: applications=1 images_considered=2 images_skipped=0 images_updated=2 errors=0"

Conclusion

You have successfully implemented a professional-grade GitOps loop from scratch. By integrating GitHub Actions, Argo CD, and the Argo CD Image Updater, you’ve bridged the gap between your source code and your live environment.

Think about the workflow you just built:

1. You push code to GitHub.

2. GitHub Actions builds and tags a fresh Docker image.

3. Argo CD Image Updater detects that new tag and automatically commits it back to your Git manifests.

4. Argo CD pulls those changes and reconciles your cluster to the new desired state.

No more manual kubectl apply, no more configuration drift, and no more 2:00 AM mysteries. Your Git repository is now truly the Single Source of Truth. If it isn't in Git, it doesn't exist in your cluster, and that is the ultimate DevOps superpower.

You will clone one repo, spin up a real 3-node cluster, break it seven different ways, and watch it fix itself each time. No simulated output or fake clusters. Real Kubernetes, real failures, and real recovery. By the end, you will recognize these failure patterns when they show up in your production environment.

Table of Contents

• What KubeLab Is?

• Prerequisites

• How to Get the Lab Running

• Simulation 1 — Kill Random Pod

• Simulation 2 — Drain a Worker Node

• Simulation 3 — CPU Stress and Throttling

• Simulation 4 — Memory Stress and OOMKill

• Simulation 5 — Database Failure

• Simulation 6 — Cascading Pod Failure

• Simulation 7 — Readiness Probe Failure

• How to Read the Signals in Grafana

• How to Use This for Production Debugging

What is KubeLab?

KubeLab is an open-source Kubernetes failure simulation lab. It runs a real Node.js backend, a PostgreSQL database, Prometheus and Grafana, all inside a real cluster. When you click "Kill Pod", the backend calls the Kubernetes API and deletes an actual running pod. Nothing is fake.

Plan about 90 minutes for the full path. Or jump directly to any simulation if you have a specific production problem you want to reproduce.

Prerequisites

You need basic familiarity with Docker and comfort with the command line, but no prior Kubernetes experience is required.

Hardware: 8GB RAM minimum, 16GB recommended. The lab can run on Mac, Linux, or Windows with WSL2. You'll need to install three tools. Multipass spins up Ubuntu VMs for the cluster. kubectl is the Kubernetes CLI you will use for every simulation. Git clones the repo. If you cannot run three VMs, the repo includes a Docker Compose preview at setup/docker-compose-preview.md full UI with mock data, no real cluster needed.

How to Get the Lab Running

Full cluster setup lives at setup/k8s-cluster-setup.md in the repo. It walks through creating three VMs with Multipass, installing MicroK8s, joining the worker nodes, and deploying KubeLab. Follow it until all eleven pods show Running:

kubectl get pods -n kubelab
# All 11 pods should show STATUS: Running

Then open two port-forwards in separate terminal tabs and keep them running for the entire tutorial:

# Tab 1 — KubeLab UI at http://localhost:8080
kubectl port-forward -n kubelab svc/frontend 8080:80

# Tab 2 — Grafana at http://localhost:3000
kubectl port-forward -n kubelab svc/grafana 3000:3000

Grafana login: admin / kubelab-grafana-2026.

Position the KubeLab UI and Grafana side by side. Left half of the screen is the app. Right half is Grafana. You will watch both simultaneously from Simulation 3 onward.

Simulation 1: Kill Random Pod

This simulation deletes a running backend pod via the Kubernetes API. Without Kubernetes, you would SSH to the server, find the crashed process, and restart it manually, usually discovered by a user alert at 3am.

Before you click: Run kubectl get pods -n kubelab -w. Watch for a pod to go Terminating then a new one to appear.

kubectl get pods -n kubelab -w
# backend-abc123  1/1   Terminating   0   2m
# backend-xyz789  1/1   Running       0   0s   ← ReplicaSet created a replacement

What happened: The ReplicaSet controller noticed actual(1) did not match desired(2) and created a replacement in parallel with the shutdown. The Endpoints controller removed the dying pod from the Service before SIGTERM fired, so zero traffic hit a dying pod.

The production trap: A missing readiness probe means the new pod receives traffic before it has opened a DB connection. You get 500s on every deployment for 2–3 seconds.

The fix: Set replicas: 2, add a readiness probe, and set terminationGracePeriodSeconds to match your longest request timeout.

Simulation 2: Drain a Worker Node

This simulation cordons a worker node, then evicts all its pods to the remaining node.

To "cordon" a worker node means to mark it as unschedulable. When you run kubectl cordon <node-name>, the Kubernetes control plane adds the node.kubernetes.io/unschedulable:NoSchedule taint to the node. (A taint is a marker that tells the scheduler to avoid placing pods on that node unless they have a matching "toleration.") This tells the scheduler to stop placing any new pods onto that node. It does not affect the pods that are already running there.

Cordoning is the first, safe step in preparing a node for maintenance. It ensures that while you are draining the node, the scheduler isn't simultaneously trying to schedule new workloads onto it, which would defeat the purpose of the drain.

Without Kubernetes you would drain the server manually, guess when in-flight requests finish, patch it, and bring it back, the window of downtime is unpredictable.

Before you click: Run kubectl get pods -n kubelab -o wide -w. Watch which node each pod runs on.

kubectl get pods -n kubelab -o wide -w

NAME                     NODE               STATUS
backend-abc123-xk2qp    kubelab-worker-1   Terminating   ← evicted
backend-abc123-n7mw3    kubelab-worker-2   Running       ← rescheduled

In kubectl get nodes the node shows Ready,SchedulingDisabled until you run kubectl uncordon.

What happened: The node spec got spec.unschedulable=true. The Eviction API ran per pod. That path goes through PodDisruptionBudget policy checks before proceeding, unlike a raw delete. A raw kubectl delete pod bypasses this check entirely — which is why draining with kubectl drain is always safer than deleting pods manually during maintenance.

The production trap: Two replicas with no pod anti-affinity often land on the same node. Drain that node and both pods evict at once. Complete downtime despite replicas: 2.

The fix: Use pod anti-affinity with topology key: kubernetes.io/hostname and a PodDisruptionBudget with minAvailable: 1.

Simulation 3: CPU Stress and Throttling

This simulation burns CPU inside a backend pod for 60 seconds, hitting the 200m limit. Without Kubernetes, one runaway process can consume all CPU on the host and starve every other service.

Before you click: Run watch -n 2 kubectl top pods -n kubelab and open the Grafana CPU Usage panel.

kubectl top pods -n kubelab
# backend-abc123   200m   ← pegged at limit for 60s; the other pod stays ~15m

What happened: The Linux CFS scheduler enforces the cgroup limit by granting 20ms of CPU per 100ms period then freezing all processes in the cgroup for 80ms. The pod is not slow because it is broken. It is slow because it is frozen 80% of the time.

The production trap: kubectl top shows the pod using 95-150m, which looks normal. The metric shows usage at the ceiling, not the throttle rate. Teams spend hours checking application code for a latency bug that is actually a CPU limit set too low.

The fix: For latency-sensitive workloads, set CPU requests but remove CPU limits. Requests tell the scheduler where to place the pod without throttling at runtime. Confirm throttling with rate(container_cpu_cfs_throttled_seconds_total{namespace="kubelab"}[5m]).

Simulation 4: Memory Stress and OOMKill

This simulation allocates memory in 50MB chunks inside a backend pod until the kernel kills it. Without Kubernetes the process dies, the server goes down, and someone gets paged.

Before you click: Run kubectl get pods -n kubelab -l app=backend -w and open the Grafana Memory Usage panel.

kubectl get pods -n kubelab -l app=backend -w
# backend-abc123   0/1   OOMKilled   3   5m   ← no Terminating phase; SIGKILL bypasses graceful shutdown

What happened: The cgroup memory limit crossed 256Mi. The Linux kernel OOM killer scored processes in the container's cgroup and sent SIGKILL (exit code 137) to the top consumer. Not Kubernetes, the kernel. SIGKILL cannot be caught or handled, so no preStop hook runs and in-memory data or open transactions can be lost. Kubernetes only observed the exit, labeled it OOMKilled, and started a fresh container.

The production trap: The pod runs fine for 8 hours, OOMKills, and restarts. Memory resets to zero and everything looks healthy again. This repeats every 8 hours. The restart count climbs to 7, then 15, then 30, but no alert fires because the metrics look normal between crashes. You find out when a user emails saying the app has been "a bit glitchy lately."

The fix: Alert on rate(kube_pod_container_status_restarts_total{namespace="kubelab"}[1h]) > 3 before users notice.
The Prometheus expression means: look at how many times containers in the kubelab namespace have restarted over the last hour, calculate how fast that number is increasing per second, and fire an alert if that rate exceeds the equivalent of 3 restarts per hour. A healthy pod rarely restarts. Several restarts in an hour usually means the container is hitting its memory limit, dying, and coming back in a loop, so this alert catches the silent OOMKill pattern before users do.

Confirm it happened:

kubectl describe pod -n kubelab <pod-name> | grep -A 5 "Last State:"
# Reason: OOMKilled
# Exit Code: 137

To see the last output before the kernel killed the process, run kubectl logs -n kubelab <pod-name> --previous. The log stream stops abruptly with no shutdown message, SIGKILL leaves no time for cleanup or final logs.

Simulation 5: Database Failure

This simulation scales the PostgreSQL StatefulSet to 0 replicas. The pod terminates completely. Without Kubernetes, the database server crashes and data recovery depends on whether backups exist and when they ran.

Before you click: Run kubectl get pods,pvc -n kubelab. Note that the PVC exists before you start.

kubectl get pods,pvc -n kubelab
# postgres-0   (gone)
# postgres-data-postgres-0   Bound   ← PVC stays; data lives on the volume

A PVC, or PersistentVolumeClaim, is a request for storage by a user. Think of it as a pod's way of saying, "I need a certain amount of durable, persistent storage." In the context of a stateful application like PostgreSQL, the PVC is critical. When the database pod is deleted, the PVC (and the underlying PersistentVolume it is bound to) remains. This is where the actual database files are stored. When a new postgres-0 pod is created, the StatefulSet knows to re-attach the same PVC, ensuring the new pod has access to all the old data, preventing data loss.

What happened: The StatefulSet controller deleted the pod but left the PersistentVolumeClaim untouched. StatefulSets guarantee stable names and stable PVC binding. postgres-0 always mounts postgres-data-postgres-0. When you restore, the same pod name comes back and reattaches the same volume. PostgreSQL replays WAL to reach a consistent state.

The production trap: Apps without connection retry logic return 500s and stay broken even after PostgreSQL restores. Connection pools that do not validate on acquire hold dead connections forever.

The fix: Add connection retry with exponential backoff in your app. Use network-attached storage (EBS, GCE PD) in production so the pod can reschedule to any node.

Simulation 6: Cascading Pod Failure

This simulation deletes both backend replicas at the same time. If everything is down, without Kubernetes, you'd have to restart every service manually, and hope they come up in the right order.

Before you click: Run kubectl get endpoints -n kubelab backend-service -w. Watch the IP list.

kubectl get endpoints -n kubelab backend-service -w
# ENDPOINTS   <none>   ← every request in this window gets Connection refused

What happened: Both pods were deleted. The Service had zero endpoints. The ReplicaSet created two replacements in parallel, but traffic stayed broken until both passed their readiness probes. The endpoint list went empty and came back. You can see the exact downtime window in Grafana's HTTP Request Rate panel.

The production trap: replicas: 2 protects you from one pod dying at a time, nothing more.
If both replicas land on the same node and that node goes down, you have zero replicas and full downtime.
Check right now with kubectl get pods -n kubelab -o wide | grep backend, and if both pods show the same NODE, you are one node failure away from an outage.

The fix: Use pod anti-affinity to force replicas onto different nodes and a PodDisruptionBudget with minAvailable: 1 to block any voluntary action that would leave zero replicas.

Simulation 7: Readiness Probe Failure

This simulation makes one backend pod fail its readiness probe for 120 seconds without restarting it. Without Kubernetes, you'd have no way to take a pod out of traffic rotation without killing it. This is what happens in production when your app connects to a database on startup but the DB is slow. The pod is alive, but it's not ready. Kubernetes holds it out of rotation until it is.

Before you click: Run kubectl get pods -n kubelab -w in one tab and kubectl get endpoints -n kubelab backend-service -w in another.

# Pods tab: STATUS Running, RESTARTS 0 — almost nothing changes
# Endpoints tab: one IP disappears — the pod is alive but not receiving traffic

What happened: /ready returned 503. The kubelet marked the pod Ready=False. The Endpoints controller removed its IP from the Service. The liveness probe /health) still returned 200, so no restart. After 120 seconds /ready recovered and the pod rejoined. Run kubectl logs -n kubelab <failing-pod> -f to see the app log 503s for the readiness endpoint while the pod stays Running and receives no traffic.

The production trap: Readiness probes that check external dependencies (database, cache, downstream API) will remove all pods from rotation when that dependency goes down. Instead of degrading gracefully, your entire app goes offline.

The fix: Readiness probes should test only what the pod itself controls. Use a separate deep health endpoint for dependency checks and never tie readiness to external service availability.

4. How to Read the Signals in Grafana

kubectl shows current state. Grafana shows what happened over time. That history is essential when you are debugging something that started 4 hours ago.

The Four Panels that Matter

Pod Restarts: A flat line is good. A step up every few hours is a silent OOMKill loop — the most common invisible production failure.

CPU Usage: A healthy pod's CPU fluctuates. A throttled pod's CPU is unnaturally flat at its limit. That flat ceiling is the signal, not the number.

Memory Usage: Watch for a line that climbs steadily then disappears. That disappearance is an OOMKill. The line reappearing from zero is the restart.

HTTP Request Rate: During Cascading Failure you see a spike of 5xx for 5–15 seconds, the exact downtime window, timestamped.

5. How to Read the Terminal Signals

What you see in the terminal during and after each simulation tells you things Grafana cannot. Five commands matter.

The -w flag on kubectl get pods -n kubelab -w streams changes in real time. The columns that matter are READY, STATUS, and RESTARTS. READY shows containers ready vs total — 1/2 means one container is alive but not passing its readiness probe. STATUS shows the pod lifecycle phase: Running, Pending, Terminating, OOMKilled. RESTARTS is the most important column in production. A number climbing silently over days is a memory leak or a crash loop nobody has noticed yet.

kubectl get events -n kubelab --sort-by=.lastTimestamp is the control plane's diary. Every action the cluster took is here: Killing, SuccessfulCreate, Scheduled, Pulled, Started, OOMKilling, BackOff. When something breaks and you do not know why, read the events. The timestamp gap between a Killing event and the next Started event is your actual downtime window — not an estimate, the exact number.

kubectl describe pod -n kubelab <pod-name> is the deepest single-pod view. Three sections matter: Conditions (Ready: True/False tells you if the pod is in the Service endpoints), Last State (shows the previous container's exit reason — OOMKilled, exit code 137, or a crash), and Events at the bottom (the scheduler's reasoning for every placement decision). This is the first command to run when a pod is misbehaving.

kubectl get endpoints -n kubelab backend-service shows which pod IPs are actually receiving traffic right now. A pod can show Running in kubectl get pods and be completely absent from this list. That is a readiness probe failure. If this list is empty, no request to that Service will succeed regardless of how many pods show Running. Check this whenever users report errors but pods look healthy.

kubectl logs -n kubelab <pod-name> shows the container's stdout and stderr. Use -f to follow the stream. After a pod restarts, use --previous to see the logs from the container that just exited, essential when you need to know what the app was doing right before an OOMKill or crash. Logs are per container and are gone once the pod is replaced, so grab them before the ReplicaSet creates a new pod with a new name.

A full event sequence during Kill Pod recovery looks like this:

kubectl get events -n kubelab --sort-by=.lastTimestamp | tail -10

REASON            MESSAGE
Killing           Stopping container backend          ← SIGTERM sent
SuccessfulCreate  Created pod backend-xyz789          ← ReplicaSet fired
Scheduled         Successfully assigned to worker-2   ← Scheduler placed it
Pulled            Container image already present     ← no pull delay
Started           Started container backend           ← running

The line between Killing and Started is your actual recovery time. In a healthy cluster with a cached image it is 3–8 seconds. If it takes longer, check the Scheduled line, the scheduler may have struggled to find a node.

Two Prometheus Queries Worth Memorizing

First query: silent restart loop. rate(kube_pod_container_status_restarts_total{namespace="kubelab"}[1h]) counts how many times containers in that namespace have restarted over the last hour and expresses it as a rate (restarts per second). A healthy workload rarely restarts. If this rate is high (for example more than 3 restarts per hour), something is killing the container repeatedly, often an OOMKill or a crash. Alert when it exceeds a threshold so you see the pattern before users report errors.

Second query: invisible CPU throttling. rate(container_cpu_cfs_throttled_seconds_total{namespace="kubelab"}[5m]) measures how much time, per second, the Linux scheduler spent throttling containers in that namespace over the last 5 minutes. A result of 0.25 means the container was frozen 25% of the time. High latency with no restarts and "normal" CPU usage in kubectl top often means the CPU limit is too low and the kernel is throttling the process. Alert when this rate exceeds about 0.25 (25% throttled).

# Silent restart loop — alert when this exceeds 3 per hour
rate(kube_pod_container_status_restarts_total{namespace="kubelab"}[1h])

# Invisible throttling — alert when this exceeds 25%
rate(container_cpu_cfs_throttled_seconds_total{namespace="kubelab"}[5m])

Run these against your own cluster. Not just KubeLab. These are production queries.

6. How to Use This for Production Debugging

The repo includes docs/diagnose.md, a symptom-to-simulation map. Find the simulation that reproduces your issue, run it in KubeLab, and understand the mechanics before you touch production.

Exit code 137, pods restarting. Run the Memory Stress simulation. Confirm with kubectl describe pod | grep -A 5 "Last State:" and look for Reason: OOMKilled. Raise limits or find the leak. The simulation shows both.

High latency, pods look healthy, zero restarts. Run the CPU Stress simulation. Check container_cpu_cfs_throttled_seconds_total in Prometheus. If it climbs, your CPU limit is too low and the pod is frozen by CFS.

503 on some requests, pods show Running. Run the Readiness Probe Failure simulation. Check kubectl get endpoints — one pod IP is missing despite Running. The pod gets zero traffic.

Pods stuck Pending after a node went down. Run the Drain Node simulation. Run kubectl describe pod <pending-pod> and read Events. The scheduler will state why it cannot place the pod, often insufficient capacity or a PVC on the failed node.

Conclusion

You just broke a real Kubernetes cluster seven ways and watched it fix itself each time. You have seen the ReplicaSet controller fire, read an OOMKill from kubectl describe, watched endpoints go empty during a cascading failure, and understood why a pod can be Running and receiving zero traffic at the same time.

What you practiced here applies to other clusters, staging or production you can read but not safely break. That muscle memory (events, endpoints, restart counter) is what you reach for at 3 am when something is wrong. KubeLab is the safe place to build that reflex.

The repo holds more than this article covered. Explore mode lets you run simulations without the guided flow. The full interview prep doc at docs/interview-prep.md has answers to the 13 most common Kubernetes interview questions. The observability guide at docs/observability.md covers Prometheus and Grafana setup in detail.

If this helped you, star the repo at https://github.com/Osomudeya/kube-lab and share it with someone who is learning Kubernetes the hard way.

We just posted comprehensive Kubernetes course on the freeCodeCamp.org YouTube channel. This hands-on course is designed to bridge the gap between theoretical container orchestration and real-world deployment. Saiyam Pathak developed this course.

You will learn to deploy a cloud-native microservices stack from the ground up. You’ll explore the inner workings of the Kubernetes architecture, including the Control Plane, Worker Nodes, and essential interfaces like CRI, CNI, and CSI. You'll learn to ship a functional application complete with a frontend, auth service, and game engine.

This course is a deep dive into the modern Kubernetes ecosystem. You will implement advanced industry standards such as Gateway API for traffic management, CloudNativePG for managing PostgreSQL databases, and cert-manager for automated HTTPS security. By the time you reach the final demo, you’ll have integrated full-stack observability using Prometheus and Grafana, giving you the confidence to manage production-grade environments.

Watch the full course on the freeCodeCamp.org YouTube channel (6-hour watch).

Understanding different container runtimes gives you more options. You can choose the right tool for your specific needs, whether that's better security, lower resource usage, or easier integration with Kubernetes.

In this tutorial, you'll learn about three major container runtimes and how to use them on your system. We’ll dive into practical examples with complete code you can run right now. By the end, you’ll understand when to use each runtime and how to move containers between them.

Table of Contents

1. What Are Container Runtimes?

2. How to Understand High-Level vs Low-Level Runtimes

3. How to Use Docker as Your Baseline

4. How to Use Podman – The Daemonless Alternative

5. How to Work with Containerd

6. How to Move Containers Between Runtimes

7. Real-World Use Cases

8. Quick Reference Guide

9. Conclusion

What Are Container Runtimes?

A container runtime is the software that actually runs your containers. When you type docker run nginx, for example, several things happen behind the scenes. The Docker CLI talks to the Docker daemon, which then uses a container runtime (usually containerd) to actually create and run the container.

Think of it like this: if containers are apps on your phone, the container runtime is the operating system that makes those apps work. Just like you can install the same app on different phones (iPhone vs Android), you can run the same container on different runtimes.

Why Does This Matter?

You might wonder why you should care about what's running your containers. Docker works fine, right? Here are a few reasons:

1. Security: Some runtimes like Podman can run containers without root privileges. This means if someone breaks out of your container, they don't have full system access.

2. Resource usage: Different runtimes use different amounts of memory and CPU. On a resource-constrained server or edge device, this matters a lot.

3. Integration: If you're deploying to Kubernetes, understanding containerd or CRI-O helps you troubleshoot production issues.

4. Licensing: Docker Desktop has licensing requirements for large companies. Alternatives like Podman are completely free.

Here’s a chart that summarizes these key points:

How to Understand High-Level vs Low-Level Runtimes

Container runtimes are split into two categories, and understanding this distinction helps you see how everything fits together.

Low-Level Runtimes

Low-level runtimes like runc and crun do the actual work of creating containers. They interact directly with the Linux kernel to create isolated environments using features like namespaces and cgroups.

Namespaces isolate what a process can see. For example, a process namespace means the container can't see other processes running on your system. A network namespace means it has its own network stack.

Cgroups (control groups) limit what a process can use. You can limit a container to 512MB of RAM or 50% of one CPU core. This prevents one container from hogging all your resources.

These low-level runtimes implement the OCI (Open Container Initiative) Runtime Specification. This is a standard that defines exactly how to run a container. Because of this standard, you can swap out runtimes and your containers still work.

High-Level Runtimes

High-level runtimes like Docker, Podman, and containerd manage images, networking, volumes, and provide user-friendly interfaces. They handle pulling images from registries, setting up networks between containers, and managing container lifecycles.

These high-level runtimes use low-level runtimes under the hood. When you run docker run, Docker ultimately calls runc to create the container. This layering means you get a nice interface while still benefiting from the standard, battle-tested low-level runtime.

Why This Layering Matters:

This separation of concerns is powerful. High-level runtimes can focus on user experience and features while low-level runtimes focus on reliably creating containers. You can swap low-level runtimes without changing your workflow. Some people use crun instead of runc because it's written in C and starts faster.

How to Use Docker as Your Baseline

Let's start with Docker since you're probably already familiar with it. This will give us a baseline to compare other runtimes against. We'll build a simple web application and then run the same application in different runtimes to see how they compare.

How to Install Docker

You can find installation guides for your operating system:

• Docker Desktop for Mac

• Docker Desktop for Windows

• Docker Engine for Linux

How to Run a Test Container

Let's verify that Docker works by running a simple container:

docker run hello-world

You should see a message that says:

Hello from Docker!
This message shows that your installation appears to be working correctly.

What Just Happened?

When you ran that command, Docker checked if the hello-world image exists locally. It didn't find it, so it pulled the image from Docker Hub (a public registry). Then it created a container from that image, started the container, and the container printed its message and exited.

All of this happened in a few seconds. Now let's build something more useful.

How to Create a Web Server

Create a new directory for your project:

mkdir ~/container-demo
cd ~/container-demo

The ~ symbol means your home directory. On macOS, this is /Users/yourname. On Linux, it's /home/yourname.

Create a simple HTML file:

cat > index.html << 'EOF'
<!DOCTYPE html>
<html>
<head><title>Container Demo</title></head>
<body>
  <h1>Hello from Docker!</h1>
  <p>This is running in a container.</p>
</body>
</html>
EOF

This creates a basic HTML file. The cat > command writes to a file, and << 'EOF' means "read until you see EOF" (End Of File). This is a handy way to create files from the command line.

How to Create a Dockerfile

You can create a dockerfile like this:

cat > Dockerfile << 'EOF'
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/
EOF

Understanding the Dockerfile:

The Dockerfile has two instructions:

1. FROM nginx:alpine: This starts with the official Nginx image. The :alpine tag means we're using the Alpine Linux version, which is much smaller (about 20MB instead of 130MB). Alpine is a minimal Linux distribution popular in containers because of its small size.

2. COPY index.html /usr/share/nginx/html/: This copies your HTML file into the location where Nginx serves files. Inside the container, Nginx is configured to serve files from /usr/share/nginx/html/.

How to Build a Docker Image

docker build -t my-web-app .

The -t flag means "tag" – we're naming the image my-web-app. The . at the end means "use the current directory as the build context". Docker will look for a Dockerfile in the current directory and send all files here to the Docker daemon for building.

You'll see output like:

[+] Building 2.3s (7/7) FINISHED
=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 98B
=> [internal] load .dockerignore
...
=> => naming to docker.io/library/my-web-app

This shows Docker building your image layer by layer. Each instruction in the Dockerfile creates a new layer. These layers are cached, so if you rebuild without changes, it's instant.

How to Run a Docker Container

docker run -d -p 8080:80 my-web-app

Understanding the Flags:

• -d means "detached mode" – run in the background. Without this, the container runs in the foreground and you'll see Nginx's log output. With -d, it returns immediately and runs in the background.

• -p 8080:80 maps port 8080 on your host machine to port 80 inside the container. Nginx listens on port 80 inside the container. To access it from your browser, you need to map it to a port on your machine. We chose 8080, but you could use any available port.

Open your browser and visit http://localhost:8080. You should see your HTML page!

How to Check Running Containers:

docker ps

This shows all running containers. You'll see something like:

CONTAINER ID   IMAGE        COMMAND                  PORTS                  NAMES
a1b2c3d4e5f6   my-web-app   "/docker-entrypoint.…"   0.0.0.0:8080->80/tcp   peaceful_curie

Docker automatically generated a random name (peaceful_curie in this example). You can specify a name with --name if you prefer.

How to View Container Logs:

docker logs <container-id>

Replace <container-id> with the ID from docker ps (just the first few characters work). This shows what's happening inside the container. For Nginx, you'll see access logs showing requests to your web server.

How to Stop the Container:

docker stop <container-id>

This gracefully stops the container. Nginx receives a signal to shut down cleanly.

Now that you understand how to use Docker, let’s check out how Podman works next.

How to Use Podman – The Daemonless Alternative

Now let's try Podman. It's designed to be a drop-in replacement for Docker, but with some key differences that make it interesting for specific use cases.

Why Podman Exists

Docker runs as a daemon (a background service) that requires root privileges. This daemon always runs, listening for commands. This architecture has some downsides:

1. Security: The Docker daemon runs as root. If someone compromises the daemon, they have root access to your entire system.

2. Resource Usage: The daemon consumes resources even when you're not running containers.

3. Single Point of Failure: If the daemon crashes, all your containers stop.

Podman solves these problems by not using a daemon at all. Each podman command runs independently. This is called a "daemonless" architecture.

Key Podman Features

To summarize, here are some key helpful features of Podman that might make it a good fit for your projects:

1. No daemon required: Each command runs independently. No background service needed.

2. Rootless by default: Containers run as your regular user, not as root. This dramatically improves security.

3. Drop-in Docker replacement: Most Docker commands work exactly the same. You can even alias docker=podman and many applications won't notice the difference.

4. Pod support: Podman has a concept of "pods" like Kubernetes. This is unique among container tools.

Now that you understand the benefits of Podman, let’s see how you can use it.

How to Install Podman

Podman installation varies by operating system. Here are the official guides:

• Podman for macOS

• Podman for Windows

• Podman for Linux

For macOS users (what we'll use in this tutorial), you can install Podman using Homebrew:

brew install podman

How to Initialize and Start Podman Machine

On macOS, Podman needs a Linux VM to run containers (since containers use Linux kernel features). Podman Machine handles this for you:

podman machine init

This creates a small Linux VM. You’ll only need to do this once. The VM is about 1GB and uses minimal resources when running.

Start the machine:

podman machine start

Verify it's working:

podman --version

You should see something like:

podman version 4.5.0

How to Run Containers with Podman

Here's where it gets interesting. You can use nearly identical commands to Docker. Let's build and run the same web server you created earlier:

# Build the image (same command as Docker)
podman build -t my-web-app .

# Run the container
podman run -d -p 8081:80 my-web-app

# See running container
podman ps

Notice that we used port 8081 this time so it doesn't conflict with the Docker container if it's still running. Visit http://localhost:8081 and you'll see the same page, but this time it's running in Podman!

If you experience issue when running the podman build command, you can delete the docker image using docker image rm my-web-app:latest.

What's Different Under the Hood?

Even though the commands look the same, what's happening is different: first no daemon was involved. The podman command directly created and started the container. And the container is running as your user, not as root.

You can verify this by checking what user owns the process:

podman top <container-id> user

You'll see your username, not root.

Podman Pods – A Unique Feature

Podman has a unique feature that Docker doesn't have: pods. A pod is a group of containers that share networking and storage. This is the same concept Kubernetes uses, which makes Podman excellent for local Kubernetes development.

Why Pods Matter:

In real applications, you often have multiple containers that need to work together. For example, a web application typically needs a database to store data, a cache layer for temporary storage of frequently accessed data and a logging container for request, response, and non-sensitive critical application metadata.

These four containers (web, database, cache, logger) need to communicate with each other. In Docker, you'd create a custom network and connect each container to it. In Podman, you can create a pod that automatically handles this networking.

How to Create a Podman Pod

podman pod create --name my-app-pod -p 8082:80

This creates a pod named my-app-pod and exposes port 8082 on your host to port 80 inside the pod. Notice that you don't expose ports on individual containers – you expose them on the pod.

Add a web server to the pod:

podman run -d --pod my-app-pod --name web nginx:alpine

The --pod flag tells Podman to run this container inside the pod. The container doesn't need its own port mapping because the pod handles that.

Add Redis (an in-memory database) to the pod:

podman run -d --pod my-app-pod --name cache redis:alpine

Now you have two containers running in the same pod. Here's the powerful part: they share the same network namespace.

To check your pod:

# List all pods
podman pod ps -a

# Show details for one pod
podman pod inspect <pod-name-or-id>

# Check processes running in the pod
podman top pod <pod-name-or-id>

# See logs from containers in that pod
podman logs <container-name-or-id>

Understanding Shared Networking:

Both containers can reach each other using localhost. The web container can connect to Redis using localhost:6379 (Redis's default port). It's as if they're running on the same machine.

This is exactly how Kubernetes pods work. If you learn Podman pods, you're learning Kubernetes networking too.

How to Generate Kubernetes YAML from Pods

Here's where Podman really shines. You can generate Kubernetes-compatible YAML from your pod:

podman generate kube my-app-pod > my-app-pod.yaml

Open my-app-pod.yaml and you'll see proper Kubernetes configuration:

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-5.7.1
apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.SandboxID/cache: 5e56bd9eab1a02a88654e3614312302d0f3f8d3652480498e6d1eef7d4824019
    io.kubernetes.cri-o.SandboxID/web: 5e56bd9eab1a02a88654e3614312302d0f3f8d3652480498e6d1eef7d4824019
  creationTimestamp: "2026-02-12T13:44:55Z"
  labels:
    app: my-app-pod
  name: my-app-pod
spec:
  containers:
  - args:
    - nginx
    - -g
    - daemon off;
    image: docker.io/library/nginx:alpine
    name: web
    ports:
    - containerPort: 80
      hostPort: 8082
  - args:
    - redis-server
    image: docker.io/library/redis:alpine
    name: cache

This file can be deployed directly to any Kubernetes cluster:

# using minikube cluster
kubectl apply -f my-app-pod.yaml

This is incredibly useful for local development. You can prototype your application using Podman pods, generate the YAML, and deploy to Kubernetes without rewriting anything.

How to Manage Podman Machines

When working with Podman on macOS or Windows, you're using a Linux VM. Here's how to manage it.

List all Podman machines:

podman machine list

This shows all your Podman VMs, their status (running or stopped), and their names. The default machine is usually called podman-machine-default.

Check machine status and info:

podman machine info

This displays detailed information about your current machine including CPU, memory, and disk usage.

Stop the Podman machine:

podman machine stop

If you have multiple machines, specify the name:

podman machine stop podman-machine-default

This stops the VM but preserves it. All your images and containers remain intact. When you stop the machine, all running containers inside it are stopped.

Start a stopped machine:

podman machine start

Or with a specific name:

podman machine start podman-machine-default

This restarts the VM. Your images are still there, but containers remain stopped unless you started them with a restart policy.

Delete a Podman machine:

podman machine rm podman-machine-default

This completely destroys the VM and all its contents (images, containers, volumes). Use this when you want to start fresh or free up disk space.

With this basic understanding of how Podman works, we can move on and learn about how to use Containerd.

How to Work with Containerd

Containerd is the runtime that Docker itself uses under the hood. It's also the default runtime for most Kubernetes installations. When you run Docker, you're actually using containerd without knowing it.

Why Use containerd Directly?

You might wonder why you'd use containerd directly if Docker already uses it. Here are a few reasons:

1. Kubernetes: Most Kubernetes clusters use containerd as their container runtime. Understanding it helps you troubleshoot production issues.

2. Minimal footprint: containerd has no UI and minimal features. It uses less memory than Docker Desktop (about 50MB vs 2GB).

3. Building tools: If you're building container orchestration tools, working directly with containerd gives you fine-grained control.

Understanding the Architecture

The containerd architecture looks like this:

Your Command → nerdctl → containerd → runc → Container

In this chain, nerdctl provides a Docker-like CLI, containerd manages images and container lifecycle, and runc actually creates the container using kernel features.

How to Install containerd with nerdctl

containerd is designed for systems (like Kubernetes) rather than direct developer use. The installation approach differs by operating system:

• Lima for macOS (includes nerdctl)

• containerd for Linux (native installation)

• nerdctl releases (for all platforms)

For macOS users (what we'll use in this tutorial), we’ll use Lima, which provides a Linux VM with containerd and nerdctl already installed.

brew install lima

Lima comes with nerdctl built-in, so you don't need to install it separately.

For Linux users, you can install containerd directly from your package manager and download nerdctl from the GitHub releases page. Containerd runs natively on Linux without needing a VM.

How to Start a Lima Instance

limactl start

This creates a default Linux VM running containerd with nerdctl available. The VM is configured with reasonable defaults (2GB RAM, 100GB disk). You can customize these settings if needed.

Lima mounts your home directory inside the VM, so you can access your files. This makes working with Lima feel transparent – you don't need to copy files into the VM.

Verify it's working:

lima nerdctl run hello-world

How to Run Your App with nerdctl

The commands are nearly identical to Docker. This is intentional – nerdctl aims for Docker compatibility. Since we're running through Lima, we’ll prefix commands with lima.

Navigate to your project directory:

cd ~/container-demo

Build the image:

lima nerdctl build -t my-web-app .

Run the container:

lima nerdctl run -d -p 8083:80 my-web-app

Visit http://localhost:8083 to see your app running on containerd!

What's Different from Docker?

Under the hood, a lot is different. Containerd is managing your image and container. There's no daemon in the traditional sense (containerd runs differently than dockerd). Images are stored differently (though they're OCI-compliant so they're compatible).

But from your perspective as a developer, the commands feel the same. This is the power of standards like OCI.

How to Check Running Containers:

lima nerdctl ps

This shows all running containers.

How to Manage Lima VMs

When working with containerd through Lima, you're using a Linux VM. Here's how to manage it.

List all Lima VMs:

limactl list

This shows all your Lima VMs, their status (running or stopped), and their names. The default VM is usually called default.

Check VM status and info:

limactl info default

This displays detailed information about the specified VM including its configuration and resource usage.

Stop the Lima VM:

limactl stop default

This stops the VM but preserves it. All your images and containers remain intact. When you stop the VM, all running containers inside it are stopped. The next time you start it, your images will still be there but containers remain stopped.

Start a stopped VM:

limactl start default

This restarts the VM. Your images persist across restarts, so you don't need to rebuild them.

Delete a Lima VM:

limactl delete default

This completely destroys the VM and all its contents (images, containers, volumes). Use this when you want to start fresh or free up disk space. You'll need to run limactl start again to create a new VM.

Create a new VM with custom settings:

limactl start --name my-custom-vm --cpus 4 --memory 8

This creates a new VM with 4 CPUs and 8GB of memory. You can have multiple Lima VMs for different projects.

How to Move Containers Between Runtimes

Thanks to the OCI (Open Container Initiative) standard, you can move container images between different runtimes. This is incredibly powerful – you can build with one tool and deploy with another.

Why Standards Matter

Before OCI, each container runtime used its own image format. Moving images between runtimes was difficult or impossible.

OCI created standards for the Runtime Specification (how to run a container), the Image Specification (how to package a container image), and the Distribution Specification (how to transfer images between systems).

Now all major runtimes follow these standards, making images portable.

Method 1 – Using Container Registries

The easiest way to share images is through a container registry like Docker Hub, GitHub Container Registry, or your own private registry. Any runtime can push and pull from registries.

First, build with Docker:

docker build -t my-username/my-app:v1 .

The image name has three parts: my-username (your registry username), my-app (the application name), and v1 (a version tag).

Push to Docker Hub:

docker login
docker push my-username/my-app:v1

You'll need to create a free Docker Hub account if you don't have one. The docker login command prompts for your credentials.

Now pull with Podman:

podman pull my-username/my-app:v1

Podman downloads the image from Docker Hub. Even though it was built with Docker, Podman can use it because both follow OCI standards.

Or pull with nerdctl:

lima nerdctl pull my-username/my-app:v1

Same image, three different runtimes. This is the power of standards.

Method 2 – Export and Import

If you don't want to use a public registry (maybe your image contains proprietary code), you can export images as tar files. This is perfect for air-gapped environments or simply moving images between machines.

Export from Docker:

docker save my-web-app -o my-web-app.tar

This creates a file called my-web-app.tar containing the image and all its layers. The file might be large (tens or hundreds of megabytes) depending on your image.

Import to Podman:

podman load -i my-web-app.tar

Import to nerdctl:

lima nerdctl load -i my-web-app.tar

Now you have the same image available in all three runtimes! You can verify:

docker images
podman images  
lima nerdctl images

All three commands will show my-web-app in their image lists.

Understanding Image Layers:

When you export an image, you're exporting all its layers. Each line in your Dockerfile creates a layer. These layers are shared between images, which saves disk space.

For example, if you have 10 images all based on nginx:alpine, they all share the nginx layers. Only the layers unique to each image take up additional space.

Real-World Use Cases

Let's look at some real scenarios where choosing the right runtime matters. These examples show how technical decisions have practical impacts.

Use Case 1 – Security-First Development

If you're working on security-sensitive applications (financial services, healthcare, government), Podman's rootless containers are a huge advantage.

The Security Problem:

Traditional Docker requires root privileges. If someone exploits a vulnerability in your container and escapes to the host system, they have root access. This is called a "container escape" vulnerability.

Podman's rootless mode solves this:

# All Podman commands run as your user by default
podman run --rm -it alpine whoami

This outputs your username, not root. The command uses --rm to remove the container when it exits (cleanup), -it to make it interactive with a terminal, alpine as a minimal Linux distribution, and whoami as a command that prints your username.

Even if someone breaks out of the container, they only have your user's permissions. They can't install system-wide malware, access other users' data, modify system configuration, or install kernel modules.

This dramatically reduces the impact of a container escape.

Example Security Scenario:

Imagine you're running a web application that processes user uploads. A vulnerability lets an attacker execute code in your container. With Docker running as root, they could escape the container, install a rootkit, steal all data from your server, and persist even after you patch the vulnerability.

With Podman rootless, they might escape the container but can only access files your user can access. They can't persist beyond the container and can't affect other users or system files.

The difference is dramatic.

Use Case 2 – Testing Kubernetes Locally

Podman can generate Kubernetes YAML from running containers. This is perfect for prototyping before you commit to a Kubernetes configuration.

The Development Workflow:

1. Run your application locally with Podman

2. Test and iterate quickly

3. Generate Kubernetes YAML when it works

4. Deploy to a real cluster

Here's a practical example. Let's say you're building a web application with a database:

Run your containers:

# Create a pod (like a Kubernetes pod)
podman pod create --name myapp -p 8080:80

# Add web server
podman run -d --pod myapp --name web nginx:alpine

# Add PostgreSQL
podman run -d --pod myapp --name db \
  -e POSTGRES_PASSWORD=secret \
  postgres:alpine

Test your application at http://localhost:8080. When it works, generate Kubernetes YAML:

podman generate kube myapp > myapp.yaml

Now you can deploy myapp.yaml to any Kubernetes cluster:

kubectl apply -f myapp.yaml

This is much faster than writing Kubernetes YAML by hand and debugging in a cluster. You iterate locally, then deploy when ready.

Why This Matters:

Kubernetes has a steep learning curve. The YAML configuration is verbose and error-prone. By starting with simple Podman commands and generating YAML, you can focus on your application first, learn Kubernetes gradually, catch configuration errors early, and iterate quickly without cloud costs.

Use Case 3 – Resource-Constrained Environments

containerd has the smallest footprint. If you're running containers on edge devices, Raspberry Pi, or resource-constrained servers, this matters a lot.

Comparing Memory Usage:

Here are typical memory footprints for each runtime:

• Docker Desktop uses approximately 2GB RAM (includes the VM, daemon, UI, and Kubernetes).

• Podman uses approximately 500MB RAM (includes the VM on macOS).

• Containerd uses approximately 50MB RAM (just the runtime, no extras).

On a developer laptop with 16GB RAM, this difference doesn't matter much. But consider these scenarios:

1. Edge Computing:

You're running containers on edge devices with 1GB RAM total. Docker Desktop won't fit. containerd leaves room for your application.

2. IoT Devices:

A Raspberry Pi with 2GB RAM running Docker Desktop leaves little room for your application. containerd uses minimal resources.

3. High-Density Servers:

Running 100 containers per server. Every MB counts. Using containerd instead of full Docker saves 2GB per server × 100 servers = 200GB.

Example Setup for Edge Device:

# On a Raspberry Pi or similar device
sudo apt-get install containerd
sudo apt-get install nerdctl

# Now you can run containers with minimal overhead
nerdctl run -d my-lightweight-app

Your application gets to use most of the available RAM instead of competing with a heavy runtime.

Quick Reference Guide

Here's a handy comparison of common commands across runtimes:

Conclusion

In this guide, we’ve explored three major container runtimes and learned how to use Docker, Podman, and containerd. The container ecosystem is much bigger than just Docker, and knowing alternatives gives you more options for security, performance, and specialized use cases.

Use Docker when you're learning or need the best documentation. Use Podman when you need rootless security or are building CI/CD pipelines. Use containerd when you need minimal resource usage or are deploying to Kubernetes clusters.

Thanks to OCI standards, your containers are portable. Build with Docker, test with Podman, deploy with containerd – it all works together! You're not locked into one vendor or tool.

As always, I hope you enjoyed this guide and learned something. If you want to stay connected or see more hands-on DevOps content, you can follow me on LinkedIn and DevOps Cloud Projects

Happy containerizing!

You can watch the course on the freeCodeCamp.org YouTube channel (2-hour watch).

This course was made possible by a grant from Linux Foundation. Use code FREECODECAMP to get 30% off training, certifications, and bundles from Linux Foundation.

There are many demos in the course using Kubernetes. Below you can find all the commands used in the course so it is easier for you to follow along on your local machine.

CKA Hands-On Companion: Commands and Demos

Part 1: Kubernetes Fundamentals and Lab Setup

This section covers the setup of a single-node cluster using kubeadm to create an environment that mirrors the CKA exam.

Section 1.3: Setting Up Your CKA Practice Environment

Step 1: Install a Container Runtime (on all nodes)

1. Load required kernel modules:

    cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
    overlay
    br_netfilter
    EOF
   
    sudo modprobe overlay
    sudo modprobe br_netfilter
   

2. Configure sysctl for networking:

    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-iptables  = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward               = 1
    EOF
   
    sudo sysctl --system
   

3. Install containerd:

    sudo apt-get update
    sudo apt-get install -y containerd
   

4. Configure containerd for systemd cgroup driver:

    sudo mkdir -p /etc/containerd
    sudo containerd config default | sudo tee /etc/containerd/config.toml
    sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
   

5. Restart and enable containerd:

    sudo systemctl restart containerd
    sudo systemctl enable containerd
   

Step 2: Install Kubernetes Binaries (on all nodes)

1. Disable swap memory:

    sudo swapoff -a
    # Comment out swap in fstab to make it persistent:
    sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
   

2. Add the Kubernetes apt repository:

    sudo apt-get update
    sudo apt-get install -y apt-transport-https ca-certificates curl gpg
    sudo mkdir -p -m 755 /etc/apt/keyrings
    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
   

3. Install and hold binaries (adjust version as needed):

    sudo apt-get update
    sudo apt-get install -y kubelet kubeadm kubectl
    sudo apt-mark hold kubelet kubeadm kubectl
   

Step 3: Configure a Single-Node Cluster (on the control plane)

1. Initialize the control-plane node:

    sudo kubeadm init --pod-network-cidr=10.244.0.0/16
   

2. Configure kubectl for the administrative user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
   

3. Remove the control-plane taint:

    kubectl taint nodes --all node-role.kubernetes.io/control-plane-
   

4. Install the Flannel CNI plugin:

    kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
   

5. Verify the cluster:

    kubectl get nodes
    kubectl get pods -n kube-system
   

Part 2: Cluster Architecture, Installation & Configuration (25%)

Section 2.1: Bootstrapping a Multi-Node Cluster with kubeadm

Initializing the Control Plane (Run on Control Plane node)

1. Run kubeadm init (Replace <control-plane-private-ip>):

    sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=<control-plane-private-ip>
   

   • Note: Save the kubeadm join command from the output.

2. Install Calico CNI Plugin:

    kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml
   

3. Verify Cluster and CNI installation:

    kubectl get pods -n kube-system
    kubectl get nodes
   

Joining Worker Nodes (Run on each Worker node)

1. Run the join command saved from kubeadm init:

    # EXAMPLE - Use the exact command from your kubeadm init output
    sudo kubeadm join <control-plane-private-ip>:6443 --token <token> \
        --discovery-token-ca-cert-hash sha256:<hash>
   

2. Verify the full cluster (from Control Plane node):

    kubectl get nodes -o wide
   

Section 2.2: Managing the Cluster Lifecycle

Upgrading Clusters with kubeadm (Example: Upgrade to 1.29.1)

1. Upgrade Control Plane: Upgrade kubeadm binary:

    sudo apt-mark unhold kubeadm
    sudo apt-get update && sudo apt-get install -y kubeadm='1.29.1-1.1'
    sudo apt-mark hold kubeadm
   

2. Plan and apply the upgrade (on Control Plane node):

    sudo kubeadm upgrade plan
    sudo kubeadm upgrade apply v1.29.1
   

3. Upgrade kubelet and kubectl (on Control Plane node):

    sudo apt-mark unhold kubelet kubectl
    sudo apt-get update && sudo apt-get install -y kubelet='1.29.1-1.1' kubectl='1.29.1-1.1'
    sudo apt-mark hold kubelet kubectl
    sudo systemctl daemon-reload
    sudo systemctl restart kubelet
   

4. Upgrade Worker Node: Drain the node (from Control Plane node):

    kubectl drain <node-to-upgrade> --ignore-daemonsets
   

5. Upgrade binaries (on Worker Node):

    # On the worker node
    sudo apt-mark unhold kubeadm kubelet
    sudo apt-get update
    sudo apt-get install -y kubeadm='1.29.1-1.1' kubelet='1.29.1-1.1'
    sudo apt-mark hold kubeadm kubelet
   

6. Upgrade node configuration and restart kubelet (on Worker Node):

    # On the worker node
    sudo kubeadm upgrade node
    sudo systemctl daemon-reload
    sudo systemctl restart kubelet
   

7. Uncordon the Node (from Control Plane node):

    kubectl uncordon <node-to-upgrade>
   

Backing Up and Restoring etcd (Run on Control Plane node)

1. Perform a Backup (using host etcdctl):

    # Create the backup directory first
    sudo mkdir -p /var/lib/etcd-backup
   
    sudo ETCDCTL_API=3 etcdctl snapshot save /var/lib/etcd-backup/snapshot.db \
        --endpoints=https://127.0.0.1:2379 \
        --cacert=/etc/kubernetes/pki/etcd/ca.crt \
        --cert=/etc/kubernetes/pki/etcd/server.crt \
        --key=/etc/kubernetes/pki/etcd/server.key
   

2. Perform a Restore (on the control plane node):

    # Stop kubelet to stop static pods
    sudo systemctl stop kubelet
   
    # Restore the snapshot to a new data directory
    sudo ETCDCTL_API=3 etcdctl snapshot restore /var/lib/etcd-backup/snapshot.db \
        --data-dir /var/lib/etcd-restored
   
    # !! IMPORTANT: Manually edit /etc/kubernetes/manifests/etcd.yaml to point to the new data-dir /var/lib/etcd-restored !!
   
    # Restart kubelet to pick up the manifest change
    sudo systemctl start kubelet
   

Section 2.3: Implementing a Highly-Available (HA) Control Plane

1. Initialize the First Control-Plane Node (Replace <load-balancer-address:port>):

    sudo kubeadm init --control-plane-endpoint "load-balancer.example.com:6443" --upload-certs
   

   • Note: Save the HA-specific join command and the --certificate-key.

2. Join Additional Control-Plane Nodes (Run on the second and third Control Plane nodes):

    # EXAMPLE - Use the exact command from your `kubeadm init` output
    sudo kubeadm join load-balancer.example.com:6443 --token <token> \
        --discovery-token-ca-cert-hash sha256:<hash> \
        --control-plane --certificate-key <key>
   

Section 2.4: Managing Role-Based Access Control (RBAC)

Demo: Granting Read-Only Access

1. Create a Namespace and ServiceAccount:

    kubectl create namespace rbac-test
    kubectl create serviceaccount dev-user -n rbac-test
   

2. Create the Role manifest (role.yaml):

    # role.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      namespace: rbac-test
      name: pod-reader
    rules:
    - apiGroups: [""] # "" indicates the core API group
      resources: ["pods"]
      verbs: ["get", "list", "watch"]
   

   Apply: kubectl apply -f role.yaml

3. Create the RoleBinding manifest (rolebinding.yaml):

    # rolebinding.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: read-pods
      namespace: rbac-test
    subjects:
    - kind: ServiceAccount
      name: dev-user
      namespace: rbac-test
    roleRef:
      kind: Role
      name: pod-reader
      apiGroup: rbac.authorization.k8s.io
   

   Apply: kubectl apply -f rolebinding.yaml

4. Verify Permissions:

    # Check if the ServiceAccount can list pods (Should be YES)
    kubectl auth can-i list pods --as=system:serviceaccount:rbac-test:dev-user -n rbac-test
   
    # Check if the ServiceAccount can delete pods (Should be NO)
    kubectl auth can-i delete pods --as=system:serviceaccount:rbac-test:dev-user -n rbac-test
   

Section 2.5: Application Management with Helm and Kustomize

Demo: Installing an Application with Helm

1. Add a Chart Repository:

    helm repo add bitnami https://charts.bitnami.com/bitnami
    helm repo update
   

2. Install a Chart with a value override:

    helm install my-nginx bitnami/nginx --set service.type=NodePort
   

3. Manage the application:

    helm upgrade my-nginx bitnami/nginx --set service.type=ClusterIP
    helm rollback my-nginx 1
    helm uninstall my-nginx
   

Demo: Customizing a Deployment with Kustomize

1. Create base manifest (my-app/base/deployment.yaml):

    mkdir -p my-app/base
    cat <<EOF > my-app/base/deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: my-app
      template:
        metadata:
          labels:
            app: my-app
        spec:
          containers:
          - name: nginx
            image: nginx:1.25.0
    EOF
   

2. Create base Kustomization file (my-app/base/kustomization.yaml):

    cat <<EOF > my-app/base/kustomization.yaml
    resources:
    - deployment.yaml
    EOF
   

3. Create production overlay and patch:

    mkdir -p my-app/overlays/production
    cat <<EOF > my-app/overlays/production/patch.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      replicas: 3
    EOF
    cat <<EOF > my-app/overlays/production/kustomization.yaml
    bases:
    -../../base
    patches:
    - path: patch.yaml
    EOF
   

4. Apply the overlay (note the -k flag for kustomize):

    kubectl apply -k my-app/overlays/production
   

5. Verify the change:

    kubectl get deployment my-app
   

Part 3: Workloads & Scheduling (15%)

Section 3.1: Mastering Deployments

Demo: Performing a Rolling Update

1. Create a base Deployment manifest (deployment.yaml):

    # deployment.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.24.0
            ports:
            - containerPort: 80
   

   Apply: kubectl apply -f deployment.yaml

2. Update the Container Image to trigger the rolling update:

    kubectl set image deployment/nginx-deployment nginx=nginx:1.25.0
   

3. Observe the rollout:

    kubectl rollout status deployment/nginx-deployment
    kubectl get pods -l app=nginx -w
   

Executing and Verifying Rollbacks

1. View Revision History:

    kubectl rollout history deployment/nginx-deployment
   

2. Roll back to the previous version:

    kubectl rollout undo deployment/nginx-deployment
   

3. Roll back to a specific revision (e.g., revision 1):

    kubectl rollout undo deployment/nginx-deployment --to-revision=1
   

Section 3.2: Configuring Applications with ConfigMaps and Secrets

Creation Methods

1. ConfigMap: Imperative Creation:

    # From literal values
    kubectl create configmap app-config --from-literal=app.color=blue --from-literal=app.mode=production
   
    # From a file
    echo "retries = 3" > config.properties
    kubectl create configmap app-config-file --from-file=config.properties
   

2. Secret: Imperative Creation:

    # Kubernetes will automatically base64 encode
    kubectl create secret generic db-credentials --from-literal=username=admin --from-literal=password='s3cr3t'
   

Demo: Consuming ConfigMaps and Secrets in Pods

1. Manifest: Environment Variables (pod-config.yaml):

    # pod-config.yaml (Assumes app-config-declarative ConfigMap and db-credentials Secret exist)
    apiVersion: v1
    kind: Pod
    metadata:
      name: config-demo-pod
    spec:
      containers:
      - name: demo-container
        image: busybox
        command: ["/bin/sh", "-c", "env && sleep 3600"]
        env:
          - name: THEME
            valueFrom:
              configMapKeyRef:
                name: app-config-declarative
                key: ui.theme
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: db-credentials
                key: password
      restartPolicy: Never
   

   Apply: kubectl apply -f pod-config.yaml Verify: kubectl logs config-demo-pod

2. Manifest: Mounted Volumes (pod-volume.yaml):

    # pod-volume.yaml (Assumes app-config-file ConfigMap exists)
    apiVersion: v1
    kind: Pod
    metadata:
      name: volume-demo-pod
    spec:
      containers:
      - name: demo-container
        image: busybox
        command: ["/bin/sh", "-c", "cat /etc/config/config.properties && sleep 3600"]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
      - name: config-volume
        configMap:
          name: app-config-file
      restartPolicy: Never
   

   Apply: kubectl apply -f pod-volume.yaml Verify: kubectl logs volume-demo-pod

Section 3.3: Implementing Workload Autoscaling

Demo: Installing and Verifying the Metrics Server

1. Install the Metrics Server:

    kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
   

2. Verify Installation:

    kubectl top nodes
    kubectl top pods -A
   

Demo: Autoscaling a Deployment

1. Create a Deployment with Resource Requests (requires hpa-demo-deployment.yaml manifest not provided, use a simple one):

    kubectl create deployment php-apache --image=k8s.gcr.io/hpa-example --requests="cpu=200m"
    kubectl expose deployment php-apache --port=80
   

2. Create an HPA (target 50% CPU, scale 1-10 replicas):

    kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
   

3. Generate Load (will run in the background):

    kubectl run -it --rm load-generator --image=busybox -- /bin/sh -c "while true; do wget -q -O- http://php-apache; done"
   

4. Observe Scaling:

    kubectl get hpa -w
   

   (Stop the load generator to observe scale down)

Section 3.5: Advanced Scheduling

Demo: Using Node Affinity

1. Label a Node:

    kubectl label node <your-worker-node-name> disktype=ssd
   

2. Create a Pod with Node Affinity (requires affinity-pod.yaml manifest not provided, create a dummy pod for the node label):

    # Create the pod using the affinity rules
    kubectl apply -f affinity-pod.yaml # Or equivalent manifest with node affinity
   

Demo: Using Taints and Tolerations

1. Taint a Node (Effect: NoSchedule):

    kubectl taint node <another-worker-node-name> app=gpu:NoSchedule
   

2. Create a Pod with a Toleration (requires toleration-pod.yaml manifest not provided, create a dummy pod for the taint):

    # Create the pod using the toleration rules
    kubectl apply -f toleration-pod.yaml # Or equivalent manifest with toleration
   

3. Verify Pod scheduling on the tainted node:

    kubectl get pod gpu-pod -o wide
   

Part 4: Services & Networking (20%)

Section 4.2: Kubernetes Services

Demo: Creating a ClusterIP Service

1. Create a Deployment:

    kubectl create deployment my-app --image=nginx --replicas=2
   

2. Expose the Deployment with a ClusterIP Service (requires clusterip-service.yaml manifest not provided, use an imperative command):

    kubectl expose deployment my-app --port=80 --target-port=80 --name=my-app-service --type=ClusterIP
   

3. Verify Access (inside a temporary Pod):

    kubectl run tmp-shell --rm -it --image=busybox -- /bin/sh
    # Inside the shell:
    # wget -O- my-app-service
   

Demo: Creating a NodePort Service

1. Create a NodePort Service (requires nodeport-service.yaml manifest not provided, use an imperative command):

    kubectl expose deployment my-app --port=80 --target-port=80 --name=my-app-nodeport --type=NodePort
   

2. Verify Access information:

    kubectl get service my-app-nodeport
    kubectl get nodes -o wide
    # Access from outside via <NodeIP>:<NodePort>
   

Section 4.3: Ingress and the Gateway API

Demo: Path-Based Routing with NGINX Ingress

1. Install the NGINX Ingress Controller:

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/cloud/deploy.yaml
   

2. Deploy Two Sample Applications and Services:

    kubectl create deployment app-one --image=k8s.gcr.io/echoserver:1.4
    kubectl expose deployment app-one --port=8080
   
    kubectl create deployment app-two --image=k8s.gcr.io/echoserver:1.4
    kubectl expose deployment app-two --port=8080
   

3. Create an Ingress Resource (requires ingress.yaml manifest not provided, use the provided structure to create the file):

    # Apply ingress.yaml
    kubectl apply -f ingress.yaml
   

4. Test the Ingress:

    INGRESS_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress.ip}')
    curl http://$INGRESS_IP/app1
    curl http://$INGRESS_IP/app2
   

Section 4.4: Network Policies

Demo: Securing an Application with Network Policies

1. Create a Default Deny-All Ingress Policy (deny-all.yaml):

    # deny-all.yaml
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: default-deny-ingress
    spec:
      podSelector: {} # Matches all pods in the namespace
      policyTypes:
      - Ingress
   

   Apply: kubectl apply -f deny-all.yaml

2. Deploy a Web Server and a Service:

    kubectl create deployment web-server --image=nginx
    kubectl expose deployment web-server --port=80
   

3. Attempt connection (will fail):

    kubectl run tmp-shell --rm -it --image=busybox -- /bin/sh -c "wget -O- --timeout=2 web-server"
   

4. Create an "Allow" Policy (allow-web-access.yaml):

    # allow-web-access.yaml
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: allow-web-access
    spec:
      podSelector:
        matchLabels:
          app: web-server
      policyTypes:
      - Ingress
      ingress:
      - from:
        - podSelector:
            matchLabels:
              access: "true"
        ports:
        - protocol: TCP
          port: 80
   

   Apply: kubectl apply -f allow-web-access.yaml

5. Test the "Allow" Policy (connection will succeed):

    kubectl run tmp-shell --rm -it --labels=access=true --image=busybox -- /bin/sh -c "wget -O- web-server"
   

Section 4.5: CoreDNS

Demo: Customizing CoreDNS for an External Domain

1. Edit the CoreDNS ConfigMap:

    kubectl edit configmap coredns -n kube-system
   

2. Add a new server block inside the Corefile data structure (e.g., for my-corp.com):

    # ... inside the data.Corefile string...
        my-corp.com:53 {
            errors
            cache 30
            forward . 10.10.0.53 # Forward to your internal DNS server
        }
    # ...
   

Part 5: Storage (10%)

Section 5.2: Volume Configuration

Static Provisioning Demo

1. Create a PersistentVolume (pv.yaml):

    # pv.yaml (Using hostPath for local testing)
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: task-pv-volume
    spec:
      capacity:
        storage: 10Gi
      accessModes:
        - ReadWriteOnce
      persistentVolumeReclaimPolicy: Retain
      storageClassName: manual
      hostPath:
        path: "/mnt/data"
   

   Apply: kubectl apply -f pv.yaml

2. Create a PersistentVolumeClaim (pvc.yaml):

    # pvc.yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: task-pv-claim
    spec:
      storageClassName: manual
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 3Gi
   

   Apply: kubectl apply -f pvc.yaml

3. Verify Binding:

    kubectl get pv,pvc
   

4. Create a Pod that Uses the PVC (pod-storage.yaml):

    # pod-storage.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: storage-pod
    spec:
      containers:
        - name: nginx
          image: nginx
          volumeMounts:
          - mountPath: "/usr/share/nginx/html"
            name: my-storage
      volumes:
        - name: my-storage
          persistentVolumeClaim:
            claimName: task-pv-claim
   

   Apply: kubectl apply -f pod-storage.yaml

Section 5.3: StorageClasses and Dynamic Provisioning

Demo: Using a Default StorageClass

1. Inspect the Available StorageClasses:

    kubectl get storageclass
   

2. Create a PVC without a PV (relies on a default StorageClass):

    # dynamic-pvc.yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: my-dynamic-claim
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
   

   Apply: kubectl apply -f dynamic-pvc.yaml

3. Observe Dynamic Provisioning:

    kubectl get pv
   

Part 6: Troubleshooting (30%)

Section 6.2: Troubleshooting Applications and Pods

Debugging Tools for Crashes and Failures

1. Get detailed information on a resource (the most critical debugging command):

    kubectl describe pod <pod-name>
   

2. Check application logs (for current container):

    kubectl logs <pod-name>
   

3. Check application logs (for previous crashed container instance):

    kubectl logs <pod-name> --previous
   

4. Get a shell inside a running container for live debugging:

    kubectl exec -it <pod-name> -- /bin/sh
   

Section 6.3: Troubleshooting Cluster and Nodes

1. Check node status:

    kubectl get nodes
   

2. Get detailed node information:

    kubectl describe node <node-name>
   

3. View node resource capacity (for scheduling issues):

    kubectl describe node <node-name> | grep Allocatable
   

4. Check the kubelet service status (on the affected node via SSH):

    sudo systemctl status kubelet
    sudo journalctl -u kubelet -f
   

5. Re-enable scheduling on a cordoned node:

    kubectl uncordon <node-name>
   

Section 6.5: Troubleshooting Services and Networking

1. Check Service and Endpoints (for connectivity issues):

    kubectl describe service <service-name>
   

2. Check DNS resolution from a client Pod (from inside the client Pod's shell):

    kubectl exec -it client-pod -- nslookup <service-name>
   

3. Check Network Policies (to see if traffic is being blocked):

    kubectl get networkpolicy
   

Section 6.6: Monitoring Cluster and Application Resource Usage

1. Get node resource usage (requires Metrics Server):

    kubectl top nodes
   

2. Get Pod resource usage (requires Metrics Server):

    kubectl top pods -n <namespace>
   

You will learn how to build a real-world operator that manages AWS EC2 instances directly from Kubernetes, covering everything from the internal architecture of Informers and Caches to advanced concepts like Finalizers and Idempotency.

The course is broken up into six parts:

• Part 1: The Theory of Controllers

• Part 2: Kubernetes Extensibility

• Part 3: Setting Up the Environment

• Part 4: Building the API & Logic

• Part 5: Hands-on Development

• Part 6: Advanced Internals & Deployment

Watch the full course on the freeCodeCamp.org YouTube channel (6-hour watch).

This is the reality of debugging modern applications. Traditional monitoring wasn't built for containers that live for seconds, services that shift across nodes, or network paths that change constantly.

eBPF changes this. It lets you see inside the kernel itself, watching every system call, every network packet, and every process execution – without modifying a single line of code.

In this tutorial, you will trace a real Kubernetes application using eBPF-powered tools. You’ll learn fundamentals that apply across the entire modern observability ecosystem, with gadgets from the Inspektor Gadget ecosystem.

By the end, you’ll be able to:

• Trace requests as they move through your Kubernetes pods

• Observe behavior at the kernel and syscall level

• Debug failures that logs and metrics simply can’t explain

Prerequisites

Knowledge requirements:

• Basic Kubernetes concepts: pods, deployments, services, namespaces

• Familiarity with kubectl: get, describe, logs, exec

• Container basics

• Basic Linux concepts: processes, system calls

Technical requirements:

• Kubernetes cluster (local or cloud-based)

• kubectl installed and configured

• Cluster admin permissions

• Linux kernel 5.10+ (most managed services have this)

Table of Contents

• Understanding eBPF Observability

• How eBPF Tracing Works (Without Getting Lost in the Kernel)

• How to Set Up Your Environment

• How to Trace Your First Request: Hands-On Tutorial

• How to Interpret Traces

• Real-World Debugging Scenarios

• Advanced Tracing Insights

• Best Practices and Production Considerations

• Next Steps and Resources

Understanding eBPF Observability

eBPF (extended Berkeley Packet Filter) is a technology that allows you to run custom programs inside the Linux kernel without changing kernel code or loading kernel modules.

The Linux kernel is the control center of your operating system. Historically, if you wanted to observe low-level activity (like network packets, system calls, or file operations), you had to rely on kernel changes or kernel modules. Both approaches were fragile, difficult to maintain, and carried real stability and security risks.

eBPF shifts how we approach observability. It provides a safe, sandboxed environment where you can run observability programs directly in the kernel with built-in safety checks that prevent crashes or security vulnerabilities.

Why does this matter for observability?

In traditional observability, you instrument your application code. You add logging statements, metrics libraries, and tracing SDKs. This works, but has significant limitations:

• Code changes are required: You must modify and redeploy applications

• It’s language-specific: Different languages need different libraries

• There will likely be blind spots: You can only see what you explicitly instrument

• The overhead: Heavy instrumentation slows down applications

• Container challenges: By the time you add instrumentation and redeploy, the problem may have disappeared

eBPF takes a different approach. Instead of instrumenting applications, you instrument the kernel. Since every application ultimately makes system calls to the kernel for network I/O, file operations, and process management, you can observe everything from one vantage point.

The eBPF advantage for Kubernetes

Kubernetes adds another layer of complexity. Your application might be spread across multiple containers, pods, and nodes. Traditional APM (Application Performance Monitoring) tools struggle here because containers come and go rapidly, network topology changes constantly, service meshes add routing complexity, and you often don't control application code (think third-party services or legacy applications you can't modify.)

eBPF doesn't care about any of this. It sees all activity at the kernel level, regardless of what language your app is written in, whether it's containerized, how many times the pod has been rescheduled, or whether you have access to modify the source code. This universal visibility is why the Cloud Native Computing Foundation (CNCF) and major cloud providers are betting heavily on eBPF for the future of observability.

How eBPF Tracing Works (Without Getting Lost in the Kernel)

When your application runs on Kubernetes, there's a clear separation between user space and kernel space. Your code runs in user space, where it's isolated, safe, and has limited access to system resources. To do anything useful – make network calls, read files, allocate memory – your application must ask the kernel for help. The kernel handles these requests via system calls, commonly called syscalls.

eBPF lets us hook into these syscalls without slowing the system down. It’s like having a CCTV camera at every doorway between user space and kernel space, watching who passes through, when, and what they’re carrying.

A Simple Example: HTTP Request Tracing

Your application initiates an HTTP GET request, which needs to go through the network stack. To establish a connection, your application first makes a socket() system call to create a network socket. Then it calls connect() to establish a connection to the remote server. Once connected, it uses send() to transmit the HTTP request. Network packets are sent across the wire, and eventually your application calls recv() to receive the response.

With eBPF tools like Inspektor Gadget's Traceloop, you can automatically hook into these syscalls. The eBPF program captures request metadata including source and destination IPs, ports, timing information, and payload sizes. You get a complete trace of the request without touching your application code.

The eBPF Execution Flow

Here's what happens under the hood when you run a trace. When you deploy Inspektor Gadget and run a gadget, several things happen behind the scenes. Once deployed, the eBPF program springs into action whenever a traced event occurs.

When your application makes a syscall, the eBPF hook triggers and quickly collects relevant data: timestamps, process IDs, container IDs, pod names, request details, and latency information. This data is sent to user space through eBPF maps, which are efficient data structures for kernel-to-userspace communication.

Inspektor Gadget adds Kubernetes context to raw kernel data. Instead of seeing only process IDs, you can see pod names, namespaces, labels, and other metadata. For example, you can tell that a request originated from the frontend pod in the production namespace and targeted the backend service.

The gadget then presents this information in a format that's immediately useful, whether you're using the CLI or integrating with other observability tools.

eBPF is fast because:

• JIT compilation: Programs are turned into native machine code for maximum performance

• Event-driven: Only execute when relevant events occur, not continuously polling

• Kernel-resident: No expensive context switching between kernel and user space

• Highly optimized: Typically adds less than 5% overhead even under heavy load

The Tool: Inspektor Gadget & Traceloop

For this tutorial, we're using Traceloop, an eBPF-based tool that traces request flows through applications by observing syscalls, network calls, and I/O operations at the kernel level.

Why are we using Traceloop for this tutorial?

• It’s quick to install and run (one command)

• The output maps directly to the application’s behavior

• It automatically adds Kubernetes context (pod names, namespaces)

• You don’t need to make any application code changes

What you'll learn applies beyond Traceloop. All eBPF tracing tools (Pixie, Cilium Hubble, Tetragon) work the same way under the hood. They attach to kernel hooks and collect event data. Once you understand the concepts here, you can use any eBPF observability tool effectively.

How to Set Up Your Environment

To get your environment ready for hands-on tracing, we'll verify that your cluster meets the requirements, install Inspektor Gadget, and deploy a sample application to trace.

Verify that Your Cluster Meets the Requirements

Before installing anything, confirm that your Kubernetes cluster is ready for eBPF.

Check your Kubernetes version:

kubectl version --short

You need Kubernetes 1.19 or later. Most modern clusters exceed this requirement, but it's worth verifying.

Verify kernel version on your nodes:

kubectl get nodes -o wide

Then check the kernel version on one of your nodes:

# If using a local cluster like minikube or kind
uname -r

# For cloud clusters, you might need to check node details
kubectl debug node/<node-name> -it --image=ubuntu -- bash -c "uname -r"

You need Linux kernel 5.10 or later for the best eBPF support. Kernel 4.18+ works but with some limitations. If you're using a managed Kubernetes service (GKE, EKS, AKS), you almost certainly have a compatible kernel.

Confirm that you have cluster admin permissions:

kubectl auth can-i create deployments --all-namespaces

This should return "yes". Inspektor Gadget needs elevated permissions to load eBPF programs into the kernel.

Install Inspektor Gadget

You can install Inspektor Gadget in several ways. We'll use the kubectl plugin method as it's the most straightforward for learning.

Install the kubectl gadget plugin:

# Download and install kubectl-gadget
kubectl krew install gadget

# Verify installation
kubectl gadget version

If you don't have krew (the kubectl plugin manager), you can install it first:

# Install krew
(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

# Add krew to your PATH
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

Deploy Inspektor Gadget to your cluster:

kubectl gadget deploy

This creates a gadget namespace and deploys the Inspektor Gadget daemon as a DaemonSet, ensuring each node in your cluster can run eBPF programs.

Verify the deployment:

kubectl get pods -n gadget

You should see one gadget-* pod per node, all in the Running state. If a pod is stuck in Pending or CrashLoopBackOff, check that your kernel meets the version requirements.

Deploying a sample application

To learn tracing effectively, we need an application that does something interesting. We'll deploy a simple microservices application with multiple components so you can see traces flowing across service boundaries.

Start by creating a namespace for our demo app:

kubectl create namespace demo-app

Then deploy a simple web application with a backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: gcr.io/google-samples/microservices-demo/frontend:v0.8.0
        ports:
        - containerPort: 8080
        env:
        - name: PORT
          value: "8080"
        - name: PRODUCT_CATALOG_SERVICE_ADDR
          value: "productcatalog:3550"
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: demo-app
spec:
  type: LoadBalancer
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productcatalog
  namespace: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productcatalog
  template:
    metadata:
      labels:
        app: productcatalog
    spec:
      containers:
      - name: server
        image: gcr.io/google-samples/microservices-demo/productcatalogservice:v0.8.0
        ports:
        - containerPort: 3550
        env:
        - name: PORT
          value: "3550"
---
apiVersion: v1
kind: Service
metadata:
  name: productcatalog
  namespace: demo-app
spec:
  selector:
    app: productcatalog
  ports:
  - port: 3550
    targetPort: 3550

Apply the configuration:

kubectl apply -f demo-app.yaml

And wait for pods to be ready:

kubectl wait --for=condition=ready pod -l app=frontend -n demo-app --timeout=300s
kubectl wait --for=condition=ready pod -l app=productcatalog -n demo-app --timeout=300s

Then just verify that everything is running:

kubectl get pods -n demo-app

You should see both frontend and productcatalog pods in the Running state.

Now you’ll need to get the frontend URL:

# For local clusters (minikube, kind, Docker Desktop)
kubectl port-forward -n demo-app service/frontend 8080:80

# Then access http://localhost:8080 in your browser

# For cloud clusters
kubectl get service frontend -n demo-app
# Look for the EXTERNAL-IP

Visit the application in your browser to confirm it's working. You should see a simple e-commerce storefront. This application makes HTTP requests from the frontend to the product catalog service, which is perfect for tracing.

How to Trace Your First Request: Hands-On Tutorial

Now that everything is set up, let's capture our first trace and see eBPF observability in action.

Generate the Traffic to Trace

First, we need some application activity to observe. We will generate a few requests for our demo application.

In one terminal, start the Traceloop gadget:

kubectl gadget traceloop -n demo-app

This command starts tracing HTTP request handling in the demo-app namespace. Inspektor Gadget monitors the kernel to capture the function calls and system events that occur while processing each request.

In another terminal, generate some traffic:

# If using port-forward
curl http://localhost:8080

# If you have an external IP
curl http://<EXTERNAL-IP>

# Generate multiple requests
for i in {1..10}; do curl http://localhost:8080; sleep 1; done
```

### Viewing Your First Trace

Switch back to the terminal running the trace loop gadget. You should see output appearing as requests flow through your application. The output will look something like this:
```
NODE         NAMESPACE   POD              CONTAINER    PID    TYPE       COUNT  
minikube     demo-app    frontend-abc123  frontend     1234   loop       1      
minikube     demo-app    frontend-abc123  frontend     1234   loop       2

Each line shows a traced execution flow, with the count increasing as the same pattern is observed again.

We can make the output more interesting by filtering:

# Stop the previous trace with Ctrl+C, then run:
kubectl gadget traceloop -n demo-app --podname frontend

This narrows our observation to just the frontend pod, reducing noise and making patterns clearer.

Understanding what you're seeing:

Each column shows different information about your application:

• NODE: Which Kubernetes node the traced event occurred on. In multi-node clusters, this helps you understand workload distribution and identify node-specific issues.

• NAMESPACE: The Kubernetes namespace. We filtered to demo-app, so you'll only see that namespace. In production, filtering by namespace is crucial for focusing on specific applications.

• POD: The specific pod where the event occurred. Each pod gets a unique name (like frontend-abc123), allowing you to distinguish between replicas of the same application.

• CONTAINER: Which container within the pod. Pods can have multiple containers (main application, sidecars, init containers), so this helps you pinpoint exactly where activity is happening.

• PID: The process ID inside the container. This is the actual Linux process that made the syscalls eBPF observed. Multiple PIDs might appear if your application uses multiple processes or threads.

• TYPE: The type of event traced. For Traceloop, this identifies kernel-level patterns detected during request processing.

• COUNT: How many times this pattern has been observed. A rapidly incrementing count indicates high request volume.

What this tells you about your application:

Even from this simple output, you can derive insights. If you see events appearing for the frontend pod but not the productcatalog pod, it might indicate that requests aren't making it to the backend. This is a potential configuration issue. If the COUNT increases rapidly for one pod but not others, you know which replica is receiving traffic, useful for debugging load balancing issues.

The real power becomes clear when you correlate these kernel-level observations with what you know about your application. When you made 10 curl requests, you should see corresponding activity in the trace output. This direct relationship between application behavior and kernel observations is the foundation of eBPF observability.

How to Interpret Traces

Understanding raw trace output is valuable, but interpreting what it means for your application's health and performance is where the real skill lies.

Trace Anatomy: Spans, Timing, and Request Flow

A trace represents a single request's journey through your system. When you curl the frontend, that generates one trace. A span represents a single operation within that trace like "frontend handles request," "frontend calls product catalog," "product catalog queries data," and "frontend returns response." Each span has timing information: when it started, when it ended, and therefore how long it took.

In traditional distributed tracing with OpenTelemetry or Jaeger, you'd explicitly create these spans in your application code. With eBPF, the tool infers spans from syscall patterns. When eBPF sees your frontend process call connect() to the product catalog's IP, followed by send() and recv(), it understands that's a span representing an HTTP request to the backend service.

The request flow is the sequence of spans showing how your request moved through services. In our demo app,

1. The user request arrives at the frontend,

2. the frontend connects to the product catalog,

3. the product catalog processes the request,

4. the product catalog returns the data, the frontend renders the page,

5. and finally, the response is sent to user.

How to Follow Requests Across Services

Let's trace a request across service boundaries to see this flow in action.

First, we’ll start a more detailed trace:

kubectl gadget trace_tcp -n demo-app

The trace_tcp gadget shows network connections, giving us visibility into service-to-service communication.

Next, generate a request:

curl http://localhost:8080

In the trace output, look for connection patterns:

You should see the frontend pod establishing a TCP connection to the product catalog service. The trace will show the source (frontend) and destination (product catalog) IPs and ports, along with timing information.

This is how eBPF lets you follow requests: by observing the network syscalls that implement service communication. You don't need a service mesh or instrumentation libraries, the kernel sees all network activity and eBPF captures it.

Understanding the flow:

1. Your curl command triggers a TCP connection to the frontend pod's IP on port 8080

2. The frontend processes the request and opens a TCP connection to the product catalog's IP on port 3550

3. Data flows back and forth (you'll see send/receive events)

4. Connections close when requests complete

Each step is visible to eBPF because each step requires syscalls that the kernel handles.

How to Identify Bottlenecks and Errors

We can also use tracing to identify performance issues.

First, let’s start by simulating a slow backend:

# Create a deliberately slow endpoint by modifying our deployment
kubectl scale deployment productcatalog -n demo-app --replicas=0

# Wait a moment, then scale back up
kubectl scale deployment productcatalog -n demo-app --replicas=1

While the product catalog is down, generate some requests:

for i in {1..5}; do curl http://localhost:8080; done

You should see connection attempts from the frontend to the product catalog, but if the service is unavailable, you'll see different patterns, possibly connection timeouts or connection refused errors, depending on the exact timing.

What bottlenecks look like in traces:

• Long spans: A span that takes significantly longer than others indicates a bottleneck. In trace loop output, you might see gaps between events or notice certain operations taking longer.

• Retries: Repeated connection attempts to the same destination suggest a failing or slow service.

• Error patterns: Connection failures, timeouts, or unusual syscall sequences indicate problems.

The best skill to have is pattern recognition. A typical, healthy request flow has a rhythm, and events occur in predictable sequences with consistent timing. When something breaks, the rhythm changes. Requests take longer, errors appear, or expected events don't occur at all.

Real-World Debugging Scenarios

Now let's go through three realistic scenarios where eBPF helps:

Scenario 1: Finding a Slow Endpoint

The problem: Users report that the product catalog page sometimes loads very slowly, but metrics show normal average latency.

Let’s use Traceloop to investigate:

# Start tracing with timing information
kubectl gadget traceloop -n demo-app --podname frontend

We’ll generate some mixed traffic:

# Some requests to the homepage (fast)
curl http://localhost:8080

# Some requests to the product catalog (potentially slow)
curl http://localhost:8080/products

In the trace output, compare the COUNT increments for different request patterns. If certain patterns show significantly more loop iterations or longer gaps between events, that indicates those requests are doing more work, possibly hitting a slow endpoint.

The diagnosis:

You might notice that requests to /products cause the frontend to make multiple calls to the product catalog service (visible with kubectl gadget trace_tcp), while homepage requests don't. This explains why the product page is slow: it's making synchronous calls to a backend service, and if that service is slow or the network is congested, users feel the delay.

The fix:

You might implement caching, make the backend calls asynchronous, or optimize the product catalog service itself. The key is that eBPF helped you identify which specific code path was slow without adding instrumentation to your application.

Scenario 2: Tracking Down Failed Requests

The problem: Your monitoring shows a 5% error rate, but application logs don't show any errors. Where are the failures happening?

Now let’s use eBPF to investigate:

# Trace network connections to see connection failures
kubectl gadget trace_tcp -n demo-app

We’ll simulate intermittent failures:

# Create a failing scenario by temporarily breaking service connectivity
kubectl delete service productcatalog -n demo-app

# Generate requests
for i in {1..10}; do curl http://localhost:8080; sleep 1; done

# Restore the service
kubectl apply -f demo-app.yaml

In the TCP trace, you'll see connection attempts from the frontend to the product catalog that fail or time out. The trace will show the source, destination, and what happened (connection refused, timeout, and so on).

The diagnosis:

The failures are happening at the network level, the frontend can't reach the product catalog. This might be due to network policy issues, service mesh misconfiguration, or DNS problems. Traditional application logs might not capture this because the application never receives a response to log, and the connection fails before the application layer even gets involved.

Why eBPF finds this when logs don't:

Your application logs what it experiences. If a connection fails at the TCP level, your application might just see "connection refused" and retry without detailed logging.

eBPF sees the actual syscalls and network events, giving you visibility into what's happening beneath your application layer.

Scenario 3: Understanding Service Dependencies

The problem: You're not sure which services depend on each other, and you want to understand the actual runtime dependencies before making changes.

We’ll use eBPF to map dependencies:

# Trace all TCP connections to see who talks to whom
kubectl gadget trace_tcp -n demo-app

And then generate normal traffic:

# Make various requests to exercise different code paths
curl http://localhost:8080
curl http://localhost:8080/products
curl http://localhost:8080/cart

The trace output shows source and destination for every connection. Build a mental (or actual) map of which pods connect to which services.

The discovery:

You'll see that the frontend pod connects to the product catalog service, but you might also discover unexpected dependencies. Perhaps the frontend also makes calls to a Redis cache, an authentication service, or external APIs. These runtime dependencies might not be documented or might differ from what architectural diagrams show.

Why this matters:

Before deploying a change to the product catalog service, you now know exactly which services will be affected. Before implementing a network policy, you know which connections to allow. Before decomposing a monolith, you understand the actual communication patterns.

This is observability-driven architecture understanding: letting the system show you how it actually works, not how you think it works.

Advanced Tracing Insights

Once you're comfortable with basic request tracing, Inspektor Gadget offers deeper observability capabilities that reveal even more about your system's behavior.

Syscall-Level Observation

The traceloop and trace_tcp gadgets give you application-level insights, but sometimes you need to go deeper. The trace_exec gadget shows you every process execution in your containers.

First, let’s monitor process execution:

kubectl gadget trace_exec -n demo-app

And generate activity:

# Exec into a pod and run commands
kubectl exec -it -n demo-app deployment/frontend -- /bin/sh
ls -la
ps aux
exit

Every command you run inside the container appears in the trace: /bin/sh, ls, ps, and anything else. This helps you understand what's running in your containers, detect suspicious activity, or debug initialization issues.

In production scenarios, this helps you answer questions like: Is my application spawning unexpected subprocesses? Are there security issues like someone running curl to download malicious scripts? Is my init script actually running the commands I think it is?

Network Tracing Insights

Beyond TCP connections, you can trace DNS queries, which often reveal surprising things about your application's behavior.

Run trace_dns:

kubectl gadget trace_dns -n demo-app

Generate requests:

curl http://localhost:8080

You'll see every DNS query your application makes: resolving service names, checking for external APIs, perhaps even unexpected queries that indicate misconfiguration or dependencies you didn't know about.

Common insights from DNS tracing include discovering that your application is using external dependencies you didn't document, finding DNS resolution failures that cause intermittent errors, or identifying excessive DNS queries that could be cached.

Combining eBPF Data with Logs and Metrics

eBPF observability delivers the best results when combined with traditional observability signals. To combine them effectively:

• Use metrics for high-level health monitoring, alerting on anomalies, tracking trends over time, and dashboard visualization.

• Use logs for application-specific context, business logic details, error messages with stack traces, and debugging application code.

• Use eBPF traces for understanding request flows, identifying where time is spent, discovering runtime dependencies, and debugging issues that don't appear in logs.

A practical workflow:

Your metrics alert you that latency increased. You check logs but don't see errors, requests are succeeding, just slowly. You use eBPF tracing to identify that requests are spending extra time in network I/O to a particular backend service. Now you check that service's metrics and logs, and discover it's under heavy load. The eBPF trace gave you the clue that logs and metrics alone couldn't provide.

This approach to observability, using the right tool for each question, is how experienced engineers debug complex systems efficiently.

What eBPF Can and Can't See

eBPF excels at:

• Network traffic (requests, responses, latency)

• System calls (file I/O, process creation, memory allocation)

• Kernel functions (scheduling, locking, resource usage)

• Function calls in binaries (with uprobes)

But keep in mind that eBPF has limitations:

• Cannot decrypt encrypted payloads (unless hooking SSL libraries before encryption)

• Doesn't automatically understand application logic

• Captures low-level events but may need context for high-level semantics

That's why eBPF complements traditional observability rather than replacing it entirely. It gives you infrastructure-level visibility with no code changes and universal coverage. Traditional APM provides application-level context, business metrics, and custom instrumentation. Together, they give you complete observability across your entire stack.

Best Practices and Production Considerations

Before using eBPF tracing in production, there are important considerations around performance, security, and operational practices.

Performance Impact

eBPF's reputation for low overhead is well-deserved, but "low" isn't "zero."

Most eBPF tracing tools add 2-5% CPU overhead and negligible memory overhead. The exact number depends on event frequency, tracing a service that handles 10,000 requests per second will have more overhead than one handling 10 per second.

Measuring the impact:

# Before enabling tracing, check baseline resource usage
kubectl top pods -n demo-app

# Enable tracing
kubectl gadget traceloop -n demo-app

# Check resource usage again
kubectl top pods -n demo-app

You should see a small increase in CPU usage in the pods where tracing is active. This is the cost of the eBPF programs running in the kernel and processing events.

Production best practices:

Use targeted tracing rather than tracing everything everywhere. Trace specific namespaces, pods, or individual containers when investigating issues. For high-volume services, reduce overhead by applying filters, aggregation, or sampling where supported by the tracing tool.

Stop tracing when you’re done investigating. Unlike metrics collection, which typically runs continuously, eBPF-based tracing is best used as an on-demand diagnostic tool to capture detailed insights during active debugging.

When overhead matters:

If you're running latency-sensitive applications (like high-frequency trading systems or real-time communications), even 2-5% overhead might be unacceptable. In these cases, use eBPF tracing in pre-production environments to identify issues, or enable it temporarily in production only when actively debugging.

Security Considerations

eBPF is powerful, which means it requires elevated privileges. Understanding the security implications is crucial.

What eBPF can access:

eBPF programs can observe all syscalls, network traffic, and process execution in the kernel. This includes potentially sensitive data like connection details, file paths, and process arguments. While eBPF programs run in a sandbox and can't modify data or crash the kernel, they can read information that might be sensitive.

Privilege requirements:

Loading eBPF programs requires CAP_SYS_ADMIN or CAP_BPF capabilities (on newer kernels). This is a privileged operation, only trusted users should have this access. The Inspektor Gadget DaemonSet runs with these privileges, so protect access to it accordingly.

Best practices:

Implement RBAC (Role-Based Access Control) to restrict who can run gadgets. Not every developer needs the ability to trace production systems.

Also, be mindful of what data you're collecting, if your traces might contain sensitive information (like authentication tokens in HTTP headers), restrict access to trace data.

Lastly, consider using admission controllers to prevent unauthorized eBPF program loading. Audit eBPF usage in production environments to track who ran which gadgets when.

Network policies:

Inspektor Gadget's DaemonSet needs to communicate with the API server and between its components. Ensure your network policies allow this communication while still maintaining appropriate segmentation.

When to Use eBPF Tracing vs. Traditional APM

eBPF tracing and traditional APM tools like New Relic, Datadog, or Dynatrace serve different purposes. Understanding when to use each helps you build an effective observability strategy.

Use eBPF tracing when:

• You can't modify application code (third-party applications, legacy systems, compiled binaries)

• You need infrastructure-level visibility (network, syscalls, kernel behavior)

• You're debugging issues that span service boundaries but don't show up in application logs

• You want zero instrumentation overhead during normal operation (run tracing only when needed)

• You need to understand what's actually happening versus what the application reports

Use traditional APM when:

• You need business-context metrics (user IDs, transaction types, business-specific data)

• You want automatic instrumentation with minimal setup for supported frameworks

• You need long-term storage and analysis of all traces (eBPF tracing is often used for real-time investigation)

• You want pre-built dashboards and alerting for common application patterns

• You need application code-level visibility (stack traces, variable values, function calls)

The Ideal Approach: Use Both

Many teams run traditional APM for continuous monitoring and use eBPF tracing for targeted investigation when APM data isn't sufficient. For example, your APM shows that a service is slow but doesn't explain why. You enable eBPF tracing on that service to understand what's happening at the kernel level, network delays, excessive syscalls, unexpected dependencies, and find the root cause.

This complementary approach gives you both the continuous visibility of APM and the deep diagnostic power of eBPF without the overhead of running both at maximum depth all the time.

Next Steps and Resources

If you got this far, thanks for reading! Now that you have learned the fundamentals of eBPF observability, and hands-on tracing with Inspektor Gadget, you can continue your journey by:

Exploring Other eBPF Tools

Now that you understand eBPF concepts through traceloop, exploring other tools will be much easier.

Try other Inspektor Gadget gadgets:

# See all available gadgets
kubectl gadget --help

# Some useful ones to explore:
kubectl gadget trace_open -n demo-app     # File I/O tracing
kubectl gadget trace_bind -n demo-app     # Port binding events
kubectl gadget profile cpu -n demo-app    # CPU profiling
kubectl gadget snapshot process -n demo-app  # Process listing

Each gadget teaches you something different about system behavior and gives you another diagnostic tool in your toolkit.

Experiment with other eBPF platforms:

If you're interested in broader observability platforms, try Pixie for its auto-instrumentation and rich UI. Install Cilium with Hubble if you're focused on network observability and want to understand service mesh behavior. Explore Tetragon if security observability interests you, seeing what processes are executing and what files they're accessing.

The concepts transfer directly: all these tools attach eBPF programs to kernel hooks, collect event data, and present it in different ways. Your understanding of syscalls, traces, and kernel-level observation applies universally.

Connect to the CNCF Observability Ecosystem

eBPF observability tools don't exist in isolation. They're part of the broader Cloud Native Computing Foundation ecosystem.

OpenTelemetry integration:

Many eBPF tools can export data in OpenTelemetry format, allowing you to combine kernel-level traces with application-level traces in a unified observability backend. This gives you the complete picture: eBPF shows you infrastructure behavior while OpenTelemetry shows you application context.

Prometheus and Grafana:

eBPF-derived metrics can be exposed as Prometheus metrics and visualized in Grafana alongside your application metrics. This unified dashboard approach helps you correlate infrastructure and application behavior.

Service mesh integration:

If you're using Istio, Linkerd, or other service meshes, eBPF tools like Cilium Hubble can provide deeper visibility into service-to-service communication than the mesh alone provides. The mesh handles traffic management while eBPF gives you kernel-level visibility.

Jaeger and Zipkin:

For organizations using distributed tracing backends, eBPF traces can be exported to these systems, enriching your trace data with infrastructure-level spans that application instrumentation misses.

Community Resources and Learning Paths

The eBPF community is vibrant and welcoming. You can continue learning from the resources below.

Official documentation and blog:

• eBPF.io: The central hub for eBPF documentation, tutorials, and project listings

• Inspektor Gadget docs: Comprehensive guides for all gadgets and use cases

• Cilium documentation: Deep dives into eBPF networking

• CNCF Blog — “What is Observability 2.0?: A quick overview of how modern observability moves beyond traditional tools by unifying metrics, logs, and traces for real-time insight in cloud-native systems.

Learning resources:

• Learning eBPF by Liz Rice: Comprehensive book covering eBPF fundamentals

• eBPF Summit: Annual conference with talks from eBPF creators and users

• CNCF webinars: Regular sessions on observability topics

• Kubernetes observability SIGs: Community discussions and projects

To make this tutorial easy to follow and experiment with, I have included all Kubernetes manifests, demo applications, and eBPF tracing commands in this repository. You can also connect with me on LinkedIn if you’d like to stay in touch.

Like, what in the world – out of every other possible name, they decided to name a database after a literal cockroach? 🤣

I mean, I get it: cockroaches are known for being resilient, and the devs were probably trying to say “our database never dies”… but still…a cockroach?

The name aside, out of all the databases out there, you might be wondering why would you choose CockroachDB? And if you did choose it, where would you even start when trying to host and deploy it? Would you go for a managed cloud service? Or could you actually self-manage it?

If you ever thought of doing it yourself – maybe in a dev environment, or even introducing it to your company – how would you go about it?

Well, just calm your nerves 😄

In this book, we’ll explore everything you need to know about deploying and managing CockroachDB on Kubernetes. We’ll dive deep into:

• Understanding how CockroachDB’s masterless (multi-primary) architecture actually works

• Setting up and deploying CockroachDB on a Kubernetes cluster

• Automating backups to Google Cloud Storage using just a few queries in the CockroachDB cluster

• Managing service accounts and authentication securely

• Tuning CockroachDB’s memory settings for stable performance

• Scaling the cluster horizontally and vertically without downtime

• Monitoring and maintaining the database like a pro

By the end, you’ll not only understand how CockroachDB works, you’ll be confident enough to deploy and manage your own resilient, production-ready instance. 🚀

Table of Contents

1. What Even Is CockroachDB? 🤔

   • Simple Definition

   • Who Made CockroachDB? When Was it Released?

   • What Problems Does CockroachDB Try to Solve?

   • Key Terms You Should Know (in plain language):

   • Why the name “CockroachDB”? 😅

2. Why Choose CockroachDB Over PostgreSQL or MongoDB 🤷🏾‍♂️?

   • How Fault Tolerance is Handled in PostgreSQL and MongoDB

   • How CockroachDB Handles It Differently

3. How CockroachDB Works Behind the Scenes ⚙️

   • Ranges: The Small Pieces of Data

   • Replication: Many Copies for Safety

   • Raft Consensus: How All Copies Agree

   • MultiRaft: Keeping Raft Efficient When Things Scale

   • Rebalancing: Movement for Balance

   • Distributed Transactions: Doing Work Across Multiple Ranges

   • How It All Fits Together: Read + Write Flow (What Happens When You Use It)

   • Why This All Matters (Putting It in Plain English)

4. Where (and How) Should You Host CockroachDB? ☁️

   • Option 1: CockroachDB Cloud (fully managed by Cockroach Labs)

   • Option 2: Bring Your Own Cloud (BYOC)

   • Option 3: Use Cloud Marketplaces (AWS, GCP, Azure)

   • Option 4 (My Favorite 😁): Self-Hosting — Especially Using Kubernetes

5. Setting Up Your Local Environment 🧑‍💻

   • Why these tools?

   • Step 1: Install Minikube

   • Step 2: Install kubectl

   • Step 3: Install Helm

6. Deploying CockroachDB on Minikube (The Fun Part Begins 😁!)

   • Step 1: Visit ArtifactHub

   • Step 2: Explore the Helm Chart

   • Step 3: Copy the Default Values

   • Step 4: Create a Folder for Our Project

   • Step 5: Understanding the Key Configurations

   • Step 6: Create a Simplified Values Config for the CockroachDB Helm Chart

   • Overview of the YAML values

   • 🚀 Step 7: Install the CockroachDB Cluster Using Helm

7. Accessing the CockroachDB Console & Viewing Metrics

   • Step 1: Locate the CockroachDB Public Service

   • Step 2: Learn More About the Service

   • Step 3: Access the CockroachDB Dashboard

   • Step 4: Visit the Dashboard

   • Step 5: Exploring the Metrics Dashboard

   • Step 6: Creating a Little Load on the CockroachDB Cluster

   • Step 7: Viewing the Metrics from the Load

   • Step 8: View the List of Created Items in the Database

8. Backing Up CockroachDB to Google Cloud Storage ☁️

   • Why Backups Are Absolutely Critical

   • Connecting to Our DB – Installing Beekeeper Studio

   • How to Install Beekeeper Studio

   • Connecting Beekeeper Studio to CockroachDB

   • Exposing the Cluster for Local Access

   • 🐝 Connecting via Beekeeper Studio

   • Verify the Connection

   • Creating a Google Cloud Account

   • Creating a Google Cloud Storage Bucket

   • Giving CockroachDB Access to the Bucket

   • Attaching the Key to Our CockroachDB Cluster

   • Testing Our Backup — Disaster Recovery Time

9. Managing Resources & Optimizing Memory Usage

   • How CockroachDB Uses Memory

   • The Memory Usage Formula You Must Follow

   • Where You Find These Settings

   • Concrete Example (Step-by-Step)

   • ⚠️ On Requests vs Limits in Kubernetes

   • Overriding the Default Fractions

10. Scaling CockroachDB the Right Way

    • Key Metrics to Understand

    • When (and What) to Scale Based on Your Metrics

    • Disk-Bound Situations — What to Do When Your Disk Is the Limiting Factor

    • Memory Pressure — What to Do When Your Database Hits the Limit

    • When Queries Are Slow but Everything Else (CPU, Memory & Disk) Looks “Fine”

    • Understanding Disk Speed (IOPS & Throughput) Across Cloud Providers

    • Downsizing the Cluster (Reducing Replicas)

    • ⚠️ The Wrong Way to Downscale

    • Decommissioning a Node Before Scaling Down the Cluster

11. What to Consider When Deploying CockroachDB on Google Kubernetes Engine (GKE) ☁️

    • Creating Your GKE Cluster

    • Connecting to your GKE cluster

    • Deploying CockroachDB in Production (on GKE)

    • Understanding the Configuration

    • Installing the CockroachDB Cluster on GKE

    • Connecting to Our CockroachDB Cluster (Now That TLS + mTLS Are Enabled)

    • Connecting via Mutual TLS (mTLS) — Why We Need a Certificate for Our root User

    • Let’s Explore Our Cluster’s Certificate

    • Understanding the Certificate Sections (Explained Super Simply)

    • Creating a Client Certificate (So We Can Finally Connect to CockroachDB)

    • Connecting to Our CockroachDB Cluster Securely (Using mTLS)

    • Restoring Our Previous Database into the New GKE CockroachDB Cluster (without SA keys)

    • Restoring Our Previous Database from Google Cloud Storage

    • Now, Let’s Restore the Data 🎉

    • Connecting to the Database with a New User

    • Connecting with Passwordless Authentication (Mutual TLS)

    • Connecting via Mutual TLS (mTLS) from Our Apps on Kubernetes

12. How to Get a CockroachDB Enterprise License for FREEE!

    • Three Types of Licenses

    • How to Apply for the Free Enterprise License

    • Adding Your License to the CockroachDB Cluster

13. Conclusion & Next Steps ✨

    • About the Author 👨🏾‍💻

What Even Is CockroachDB? 🤔

Hey! before we jump into setting up our Kubernetes cluster and deploying our CockroachDB cluster, let’s get grounded in what CockroachDB really is. (Because if you don’t understand the why and how, the implementation and practical session will just feel like magic 😅.)

Simple Definition

CockroachDB is a distributed SQL database. This means it gives you the features of a relational database (tables, SQL queries, JOINS, transactions) but copies data across multiple replicas (servers, nodes, instances). No need for sharding manually. 😃

It’s built to survive failures, scale easily (compared to other SQL databases), and keep your data consistent no matter what (across all the instances).

Who Made CockroachDB? When Was it Released?

CockroachDB was created by Cockroach Labs, founded by Spencer Kimball, Peter Mattis, and Ben Darnell. The idea first started taking shape around 2014, and by 2015 Cockroach Labs was formally founded.

Its 1.0 “production-ready” version was announced in 2017, marking its transition from beta to being suitable for real-world use.

What Problems Does CockroachDB Try to Solve?

Traditional relational databases are great, but they run into real challenges when your app grows. CockroachDB was built to solve those. Here are the key pain points and how CockroachDB addresses them:

So instead of fighting your database as it grows, CockroachDB handles much of the hard work for you.

Key Terms You Should Know (in plain language):

• Node: Duplicates or copies of your database. These are also known as replicas. They can be read-only (databases from which data can only be read, for example using SELECT statements), OR read-write (databases from which data can be read, created, updated, and deleted).

• Replication: making copies of data on multiple nodes. If one node fails, others still have the data.

• Raft (consensus algorithm): a system that ensures copies (replicas) agree on changes in a safe, reliable way. For example, when you want to write data, Raft ensures that most copies agree before it’s accepted.

• Sharding / Ranges: Instead of putting all your data in one big blob, CockroachDB splits it into smaller chunks called ranges. Each range is replicated and can move between nodes.

• Distributed transaction: a transaction (series of operations) that might touch data stored in different nodes. CockroachDB manages this, so you still get ACID (atomic, consistent, isolated, durable) properties.

Why the name “CockroachDB”? 😅

You might wonder: Why name a database after a cockroach? It sounds weird at first, but there's a reason:

Cockroaches are known for surviving harsh conditions: radiation, natural disasters, and so on. The founders wanted a database that feels almost “impossible to kill,” that can survive node failures, outages, and network splits. The name is a tongue-in-cheek nod to resilience.

Why Choose CockroachDB Over PostgreSQL or MongoDB 🤷🏾‍♂️?

Let’s compare the classic setup (Postgres / MongoDB) to CockroachDB, especially why you might want to go with CockroachDB, and how it helps ease scaling. I’ll also explain some terms to make sure you’re following.

In many setups, when you use Postgres or MongoDB, you’ll often have one “primary” node that handles all writes (that is, inserts, updates, deletes).

Then you have multiple “read replicas” that copy the primary’s data and serve read requests (selects). That works okay – reads can be spread out – but all write traffic goes to that one primary node.

Usually, the primary eventually gets stressed when the write volume grows (for example, more customers create accounts and products on your platform).

You can add more read replicas (horizontal scaling for reads, for example customers trying to view their accounts, or previously created products on your site), but scaling the primary is much harder.

To scale the primary, you often resort to upgrading its resources (CPU, RAM, disk) – that’s vertical scaling – which often needs downtime (shut down the primary database, increase its CPU and RAM, then spin it back up).

Or you’d have to manually shard (split) your data across multiple primaries, route traffic carefully, and manage complexity.

How Fault Tolerance is Handled in PostgreSQL and MongoDB

When you try to make Postgres (or MongoDB) highly available and fault tolerant in a self-managed setup, you often need two+ read replicas and one primary.

The tricky part is handling what happens when the primary fails (or is taken down temporarily for an upgrade). You need something that can promote a replica to a primary automatically.

In Postgres land, that’s often handled by Patroni or repmgr (tools that handle cluster management, failover, leader election, and so on).

In MongoDB, such logic is part of the replica set behavior: it does automatic elections among replicas.

Here are some of the core challenges with that classic model:

• Every write must go to a single primary. If that primary fails or is overloaded, your whole system suffers.

• Scaling reads is easy (add more replicas), but scaling writes is hard.

• Vertical scaling (give more resources to one server) has its cons. If the primary node needs more resources, you might experience some downtime when it’s being scaled up.

• Manual sharding is messy: you decide which piece of data goes to which shard, handle cross-shard queries, and build routing logic. That’s a lot of maintenance and can lead to unexpected issues if not handled properly.

• One service (or load balancer/proxy) points to the primary (for ALL write queries).

• Another service or routing logic handles read queries and can share reads across replicas.

• You might use HAProxy, pgpool-II, or pgBouncer for Postgres to route traffic, do read/write splitting, or manage connection pooling. These are external (not part of the database core) tools you have to configure.

So when the primary fails, Patroni (or repmgr, and so on) will detect it and promote one of the read replicas to be the new primary.

But that promotion, reconfiguration, and traffic rerouting often cause a brief window of downtime (when your primary database node becomes unavailable).

How CockroachDB Handles It Differently

CockroachDB changes the rules:

• All replicas are equal for reads and writes. You don’t have a special “primary” that handles writes. Every node in the cluster can accept write requests.

• CockroachDB breaks your data into small chunks (ranges) and replicates them across nodes. If you add a new node, data moves around automatically to balance the load.

• Every write is automatically copied to other replicas, and consistency is managed by a protocol (Raft), so you don’t have to build this yourself.

• No manual sharding needed. Because the database handles how data is split and moved, you don’t need to decide how to shard by hand.

• You don’t need a special service to route writes vs reads queries. Any node can accept both reads and writes.

• During scaling, you don’t have to worry about which node is the primary – because there is no primary.

• You can scale your nodes one at a time (rollout style). When one node is being upgraded, the others continue to serve traffic. You won’t hit a downtime window just because you're scaling the “primary.”

• Because there's no replica promotion logic to fight with, there's no moment where a replica needs to be “elevated” to primary – it’s all just nodes continuing to serve.

How CockroachDB Works Behind the Scenes ⚙️

In CockroachDB, there are many moving parts behind the scenes. But they work together, so you don’t have to babysit them. The core ideas, which we’ve mostly already touched on, are:

• Splitting data into pieces (ranges)

• Keeping multiple copies of each piece (replicas/replication)

• Making sure all copies agree via Raft consensus

• Moving pieces around to balance the load (automatic rebalancing/distribution)

• Coordinating transactions that might touch many pieces

Let’s go through each of those, one by one.

Ranges: The Small Pieces of Data

Imagine you have a giant book of recipes. If you try to carry the whole thing, it’s heavy. So you split the book into smaller booklets, each covering recipes for a certain range of meals: breakfasts, lunches, dinners, desserts.

In CockroachDB, data is split into ranges, which are like those smaller booklets:

• Each range covers a certain block of data (like “all users whose ID is 1-1000”)

• When a range gets too big (like having too many recipes in one booklet) it’s cut/split into two smaller ones. That makes each piece easier to manage.

• If two neighboring ranges have become very small (few recipes), they might be merged (joined) back together so you’re not keeping too many tiny booklets.

• These splits and merges happen automatically, behind the scenes, so the database stays smooth as things grow or shrink.

This chopping helps the system in many ways: moving pieces, copying them, balancing load, recovering from node failures becomes easier.

Replication: Many Copies for Safety

Nobody likes losing their work, so you keep backup copies. CockroachDB does this for data as well.

For each range, there are usually 3 copies (replicas) stored on different machines (nodes). If one machine dies, you still have others. (cockroachlabs.com). And these copies are always kept in sync: when you write something (for example, insert or update), the change is propagated to the other copies.

The database also tolerates failures. If one node goes down, the system detects it and eventually makes a new copy elsewhere to replace it. So the target number of copies is maintained. This gives you fault tolerance: your data stays safe even when parts of your system fail.

Raft Consensus: How All Copies Agree

Having copies is useful, but you also need them to agree with each other – like all your recipe booklets have the same content in each copy. The Raft protocol is a way to make sure that happens reliably.

Here’s how Raft works in simple terms:

• Each range has a group of replicas. One of these replicas acts as the leader. Others are followers.

• All write requests for that range go through the leader. The leader gets the request, then tells followers to record the same change.

• Once most of the copies (a majority) say “yep, we got it,” the change is considered final (committed). Then the leader tells the client, “Done.”

• If the leader stops working (the machine dies or the network fails), the followers notice it (they stop getting regular “I’m alive” messages), then they hold an election to pick a new leader, and the show goes on.

• This way, the system ensures everyone has the same final data and no conflicting changes happen.

So Raft is the agreement protocol that keeps all copies in sync and safe.

MultiRaft: Keeping Raft Efficient When Things Scale

When you have many ranges (many pieces of the booklets), each range has its own Raft group. That can mean a lot of “are you alive?” messages between nodes, and a lot of overhead. MultiRaft is the trick CockroachDB uses to make this efficient.

MultiRaft groups together Raft work for many ranges that share nodes, so overhead is reduced. Instead of sending separate heartbeat (are you alive?) messages for each range, some of the messages are bundled.

This reduces network chatter and resource waste and helps the database scale smoothly when you have tons of data and many pieces.

Rebalancing: Movement for Balance

When your ranges are not evenly spread across nodes (machines), some machines are doing way too much work, and some hardly any. That’s not good. So CockroachDB automatically moves pieces around to balance things.

• The system watches how busy each node is (how many ranges it holds, how much data, how much read/write traffic).

• If one node is overloaded, it will move some ranges to other nodes.

• If a node dies, the system notices and makes sure that ranges that were on that node get copied somewhere else so safety (replica count) is maintained.

• If you add a new node, the system starts moving ranges to the new node so its resources are used.

This happens without you having to manually decide “move this here, move that there.”

Distributed Transactions: Doing Work Across Multiple Ranges

Often, an operation touches multiple ranges. For example, “transfer money from account A (in range 1) to account B (in range 2)”. That must be handled carefully so that either both parts succeed, or neither do.

CockroachDB supports distributed transactions, meaning a single transaction can work across many ranges. It uses “intent” writes (temporary placeholders) and once everything is ready, it commits the transaction so it becomes permanent. If something fails, it aborts (cancels) the whole thing. The system ensures atomic behavior: all or nothing.

How It All Fits Together: Read + Write Flow (What Happens When You Use It)

Let’s picture a write, step by step:

1. Your app sends a write (for example, “add new user”) to any node in the CockroachDB cluster.

2. That node figures out which range(s) are involved (which pieces hold the data you want to write).

3. For each range, the write goes to that range’s leader.

4. The leader writes the change to their own copy, then tells followers to do the same.

5. Once most copies confirm they have the change, the leader declares it “committed” and tells your app, “yes, write done.”

6. If a node is busy or down, others still handle traffic.

Read flow:

• Your app sends a read (for example “get user by ID”) to any node.

• That node checks its copies. If it has a fresh copy, it answers. If not, it asks the node that does.

Everything works so data is correct, up to date, and reliably available even if machines fail or network lags.

Why This All Matters (Putting It in Plain English)

All these tweaks are important for several key reasons. First of all, because data is chopped into ranges and replicated, no single node is a bottleneck. Also, Raft ensures consensus, so you can trust that data is consistent across all working replicas.

Beyond this, rebalancing is automatic, you don’t have to micromanage shards or worry about nodes drowning in load. And because transactions that touch multiple ranges are coordinated, you can trust ACID properties even in a distributed setup.

Where (and How) Should You Host CockroachDB? ☁️

There isn’t just one “right” way to host CockroachDB. There are a few paths you can pick, each with pros and cons. What you pick depends on cost, control, ease of use, and your risk tolerance.

In this section, we’ll explore:

• Cockroach Labs’ own managed cloud (CockroachDB Cloud)

• “Bring Your Own Cloud” (BYOC) – letting Cockroach Labs manage it inside your cloud account

• Hosting via cloud marketplaces (AWS, GCP, Azure)

• Self-hosting / Kubernetes / your own infrastructure

• And notes on DigitalOcean support

Let’s dive in.

Option 1: CockroachDB Cloud (fully managed by Cockroach Labs)

This is the easiest option if you want to offload operations. You don’t manage nodes (computers, Virtual machines, and so on), upgrades, or backups, as Cockroach Labs handles all that.

What it offers:

• You sign up and click “create cluster.”

• Automatic scaling, zero-downtime upgrades, and managed backups.

• It supports multiple cloud providers behind the scenes (you pick region(s)).

• You get tools, APIs, and Terraform integration to automate it.

• They often give free credits to get started.

Tradeoffs:

• You have less control over underlying infrastructure, for example Virtual Machines, networking, disks, and so on (you trade control for convenience).

• You pay for the managed service premium.

• You rely on Cockroach Labs’ SLAs, uptime, and support.

If you want, you can check it out here: CockroachDB Cloud (managed by Cockroach Labs).

Option 2: Bring Your Own Cloud (BYOC)

This is a middle ground: you keep your cloud environment, but let Cockroach Labs manage the database. It gives you control over infrastructure, billing, network, and so on, while still offloading operational complexity.

How it works:

• You run CockroachDB Cloud inside your cloud account (AWS, GCP, and so on).

• Cockroach Labs still handles provisioning, upgrades, backups, and observability. You manage roles, networking, and logs.

• Useful for complying with regulations, keeping data within your cloud folder/account, and using your cloud discounts.

Tradeoffs:

• You still need to set up cloud aspects (VPCs, IAM, roles) correctly.

• There’s more complexity than pure managed, but more control as well.

• Cockroach Labs needs access to certain parts of your account (permissions).

If you want to explore BYOC, you can read more here: CockroachDB Bring Your Own Cloud.

Option 3: Use Cloud Marketplaces (AWS, GCP, Azure)

If you already use a cloud provider, sometimes the easiest way is to deploy via their marketplace offerings. It gives you familiarity, billing simplicity, and so on.

• GCP Marketplace – CockroachDB is available on the Google Cloud Marketplace, making it easier to deploy within your GCP environment. You can learn more here: GCP Marketplace.

• AWS Marketplace – CockroachDB is listed there: AWS Marketplace.

• Azure Marketplace – Also supported for Azure deployments (SaaS/managed listings): Azure Marketplace.

• DigitalOcean – There is support for CockroachDB deployment on DigitalOcean using their infrastructure: Deploy CockroachDB on DigitalOcean.

These options let you stay in your cloud console, use your existing cloud accounts, and integrate with other resources you already have.

But you're still responsible for certain operational tasks (networking, security, monitoring, backups) depending on how the marketplace offering is configured.

Option 4 (My Favorite 😁): Self-Hosting — Especially Using Kubernetes

If you self-host CockroachDB, you get full control. You’re the boss of everything: the machines, storage, networking, backups, upgrades, monitoring – all of it.

What’s even better is that using Kubernetes means your setup isn’t tied to one cloud provider. You can run it on AWS, GCP, Azure, or even on-premises later, with very little change. Kubernetes gives you a “portable infra” layer.

Managed CockroachDB services charge you extra for “maintenance, upgrades, backup, etc.” – those are baked into the price. But when you self-host, you accept the burden, but also avoid paying that extra margin. You pay for compute, disks, network, and your time/ops work.

You can also self-host in the cloud (using cloud VMs) but still manage every layer: disks, network, security, and so on. Using Kubernetes, there is a sweet middle ground: you get cloud reliability for VMs, but you fully control everything above that.

Why Kubernetes Beats Tools Like Docker Swarm or Hashicorp Nomad for Databases

Because CockroachDB is a stateful system (it holds data), you need strong support for “data that stays even when a pod restarts or moves.” Kubernetes is designed with good primitives for that. Other tools don’t always shine there.

Here’s the comparison in simple terms:

• Docker Swarm / Docker Compose: Great for stateless apps (web servers, APIs), but when it comes to databases, it struggles. Swarm doesn’t natively support persistent volume claims at a cluster level, so if a container (database replica) moves to a different node (VM), it might lose access to its storage. Devs often pin containers to specific nodes manually to avoid this.

• Nomad: More flexible and simpler in some ways, but it’s not as rich in features around connectivity, storage management, and built-in tooling for containers. It works well in mixed workloads, but handling complex databases usually means you need to build extra layers.

• Kubernetes: It has built-in support for stateful workloads:

  • StatefulSets (Properly managing data for each database): This ensures that each CockroachDB replica (pod) keeps its identity and storage intact even if the pod restarts. So the database replica doesn’t lose its “name” or data when things change.

  • Persistent volumes and persistent volume claims (external disks): These are like dedicated hard drives or disks attached to pods (database replicas). Even if a pod moves, crashes, or restarts, the disk (data) stays. Kubernetes makes sure the data stays safe.

  • StorageClasses (choose your disk): You can customize the disks in which your data will be stored, that is:

    • HDD (most affordable, but slower),

    • Balanced Disk (SSD enabled, a balance between costs and speed),

    • Fast SSD (Very fast, recommended by the CockroachDB team, but a bit more expensive than a Balanced Disk).

    • Rolling updates, anti-affinity, (No Downtime, High Availability, Fault tolerance).
      Anti-affinity means you can tell Kubernetes, “don’t put more than one CockroachDB replica on the same VM or physical machine.” This protects you if one VM goes bad, other replicas are safe.

    • Rolling updates let you update one replica at a time (configuration, version, resources) without bringing down the whole cluster. While one replica updates, others serve traffic. That helps avoid downtime.

    • Kubernetes also has ordered start/stop for replicas (via StatefulSets) so things are predictable and safe

  • Vertical vs horizontal scaling (earlier talk – reminder)
    You remember we talked about scaling in prior sections:

    • Horizontal scaling means adding more replicas (more pods, more nodes) so load spreads out.

    • Vertical scaling means increasing the resources (CPU, RAM, disk) of existing nodes/replicas.

In tools like Nomad or Docker Swarm, vertical scaling tends to be harder, often involves stopping services, shutting things down, and restarting VMs, which causes downtime.

Kubernetes makes vertical and horizontal scaling easier at the pod level (you can resize one pod CPU + RAM) and manage rolling upgrades so you don’t take everything down at once.

You can also add more database replicas to the cluster easily (to balance load and make the database process queries faster), and the data is automatically copied to the new database replica (replication), especially when you use the official CockroachDB Helm Chart.

Why Other Tools (Swarm / Nomad / Docker Compose) Don’t Match Up Here

Docker Swarm and Docker Compose are simpler to use and are good when you don’t have much complexity. But they lack robust features for stable storage, default support for replication, vertical scaling, horizontal scaling of stateful services, and so on. For example, Swarm doesn’t have built-in StatefulSets or dynamic volume provisioning like Kubernetes.

Nomad is more flexible than Swarm in some ways, but many users say storage plugins (CSI) are weaker than what Kubernetes has. Also, less built-in for ordering things, rolling updates for stateful apps.

So while these work fine for simpler apps (stateless services, small apps), when you have a distributed stateful SQL database like CockroachDB, Kubernetes gives you more safety, more control, less chance of data loss or misconfiguration.

Because of all this, running CockroachDB on Kubernetes gives you the tools you need baked in, reducing how much custom plumbing you must write yourself.

Trade-offs (things to watch out for)

• You have to manage everything: backups, monitoring the ENTIRE CockroachDB cluster, withstanding failures (fault tolerance), and upgrades. That’s work 🥲.

• You need to know your way around infra (VMs, disks, networking, and inter-node connections) and operations (or have teammates who do – DevOps Engineers, Cloud Architects, Site Reliability Engineers).

• Using managed Kubernetes (like GKE, EKS, AKS) helps as you offload the control plane. You still manage the nodes, storage, and higher layers.

• But even with that, you avoid paying for “database management as a service” markup – you're only paying for infrastructure plus your time.

Setting Up Your Local Environment 🧑‍💻

Alright, we’ve learned quite a bit so far: what CockroachDB is, how it works behind the scenes, and where you can host it. Now, it’s time to roll up our sleeves and get our hands dirty with some practical setup.

Before we deploy CockroachDB, we need a safe “playground” where we can test and experiment without touching the cloud or spending a dime.

Why these tools?

Before we jump into running commands, here’s a quick lookup of what tools we’ll use and why:

• Minikube: A tool that runs a small Kubernetes cluster on your computer. It gives you a local “mini cloud” where you can deploy and experiment.

• Kubectl: The command line tool you’ll use to talk to your Kubernetes cluster to deploy apps, check status, and manage resources.

• Helm: A package manager for Kubernetes. It helps you install complex applications (like CockroachDB) with fewer manual steps.

Step 1: Install Minikube

What is Minikube?
Minikube is a lightweight tool that helps you run a small Kubernetes cluster on your personal computer.

Think of it as your own mini-cloud environment where you can test, deploy, and learn Kubernetes (and in our case, CockroachDB) locally. It’s perfect for learning and experimenting before deploying on the cloud.

Here’s how to get it on different operating systems:

🪟 Windows

1. Make sure you have a hypervisor (VirtualBox, Hyper-V) or Docker installed.

2. Open PowerShell as Administrator.

3. Run:

    choco install minikube
   

   or use:

    winget install minikube
   

4. After installation, check the version:

    minikube version
   

   If it returns a version number, you’re good 👍🏾

If you don’t have the choco or winget package manager, you can install Minikube via PowerShell by following the steps in the docs.

🍎 macOS

1. Ensure you have Homebrew installed.

2. In Terminal, run:

    brew install minikube
   

3. Start the cluster:

    minikube start
   

4. Verify:

    minikube version
   

🐧 Linux

1. Ensure you’re on a supported distribution (Ubuntu, Fedora, and so on) and virtualization (Docker, KVM, and so on) is enabled.

2. Run:

    curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
    sudo install minikube-linux-amd64 /usr/local/bin/minikube
    rm minikube-linux-amd64
   

3. Start the cluster:

    minikube start
   

4. Verify:

    minikube status
   

✅ At this point you should have a local Kubernetes cluster up and running on your machine! Next, we’ll install Kubectl so you can talk to the cluster from your command line.

Step 2: Install kubectl

What kubectl does:
kubectl is the command-line tool that lets you talk to your Kubernetes cluster. Using it, you can deploy applications, check your cluster’s health, and manage resources inside your cluster.

You’ll use it a lot when working with Kubernetes on Minikube and later when you deploy CockroachDB.

Here’s how to install it on Windows, macOS, and Linux:

🪟 Windows

1. Open PowerShell as Administrator.

2. Run:

    choco install kubernetes-cli
   

   or if you prefer:

    choco install kubectl
   

3. Then check the version:

    kubectl version --client
   

   If it prints a version number, you’re good.

🍎 macOS

1. Open Terminal.

2. If you have Homebrew installed, run:

    brew install kubectl
   

3. Check the version:

    kubectl version --client
   

   That should show something like “Client Version: v1.x.x”.

🐧 Linux

1. Open your terminal.

2. Download the latest kubectl binary:

    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   

3. Make it executable and move it into your PATH:

    chmod +x ./kubectl
    sudo mv ./kubectl /usr/local/bin/kubectl
   

4. Verify:

    kubectl version --client