But what happens when the task actually takes time? Something like "go through the last three days of my emails, draft replies for anything urgent, and create Linear tickets for the engineering-related ones." That's not a quick job. It might take minutes, hours, or even longer. And this is just one example.

That's a full workflow, and the moment your server crashes or your process restarts, you lose everything. No retry, and no resume. You're starting from scratch.

That's the problem this article is about.

In this article, you'll build a durable background agent runtime that holds up under real conditions. Dispatch a task, walk away, and it gets done.

Under the hood, it runs on Kubernetes with KEDA autoscaling so workers scale to zero when idle and spin back up the moment work arrives. For crash recovery and durable execution we'll use Temporal, and for agentic capabilities and tool usage we'll use Composio.

What's Covered?

In this tutorial, you'll build a durable background agent runtime that runs on Kubernetes and scales based on actual workload. Here's what you'll learn along the way:

• What an agent loop is and how to build one with Claude and Composio

• How to make long-running agent tasks handle crashes using Temporal

• How to build a gateway that decouples task dispatch from execution

• How to containerize the worker and gateway with Docker

• How to deploy the full system to a local Kubernetes cluster

• How to autoscale workers to zero with KEDA based on queue depth

This gets into some advanced concepts, but follow along and you'll learn a lot along the way.

Table of Contents

• What's the Plan (the Architecture)

  • Dispatching a Task:

  • Running the Task

  • Scaling

• How to Set Up the Project

• Core Components in the Application

  • The Agent Loop

  • Making it Durable with Temporal

  • The Agent Gateway

  • Containerizing the Application

  • Deploying to Kubernetes

  • Autoscaling with KEDA

• Agent in Action

• Conclusion

What's the Plan (the Architecture)

Before diving into the code, it helps to understand how everything fits together.

The system is split into two distinct planes: a control plane that handles user-facing interactions (Next.js frontend), and an execution plane where the actual agent work happens. They never directly call each other, and that separation is intentional.

Here's the flow from start to finish:

Dispatching a Task

When a user submits a goal, the gateway first runs a pre-flight check to verify the required Composio tool connections are active for that user. If they are, it hands the task off to Temporal and returns immediately. The user doesn't wait around.

NOTE: You don't wait for the response to be back from the agent. It just happens all in the background. This isn't your regular chat app. You just launch the task and you forget.

Running the Task

Temporal puts the task on a queue and a worker pod picks it up. The worker runs the agent loop, LLM reasons over the goal, Composio executes the tools, and the result gets written back to Temporal. The frontend automatically polls the gateway for status updates so the user can see progress without doing anything.

Scaling

KEDA watches the Temporal queue depth and scales worker pods based on how much work is pending. When the queue is empty, workers scale down to zero. When tasks come in, they load back up. That's the beauty!

The reason the gateway never touches agent code is straightforward: agent tasks can take minutes, or even hours based on the work, and you don't want that in your API layer. Keeping them separate helps the control plane stay fast regardless of what's happening in the background.

Also, the application supports Linux CronJob-style task scheduling with no human involved. So, having a pre-flight check helps there, because failing fast at dispatch is much better than having a task silently fail because a tool connection was missing.

That's pretty much the high level architecture of our application. To put it simply:

• Kubernetes (k8s): Orchestration Layer

• KEDA: Auto-scaling Layer

• Temporal: Durability Layer

• Composio: Tool Layer

• Any LLM of your choice (in our case, Anthropic) = Reasoning layer

How to Set Up the Project

Before you start, make sure you have the following installed:

• Docker

• k3d (for running a local Kubernetes cluster)

• kubectl

• Helm

• Node.js and Python 3.11+

You'll also need API keys for Anthropic and Composio.

Start by cloning the repository:

git clone https://github.com/shricodev/kron-k8s-agent.git
cd kron-k8s-agent

Next, create the cluster, build the images, and load them in:

# Create the local cluster
k3d cluster create agent --wait

# Build both images and import them into the cluster
bash scripts/build-images.sh
bash scripts/load-images.sh

# Deploy Temporal (creates the temporal namespace, Postgres, and server)
kubectl apply -f infra/k8s/temporal/temporal-dev.yaml

Next, create the namespace and your secret. The secret has to exist before the app gets deployed, since the pods read their keys from it:

# Create the agent namespace
kubectl apply -f infra/k8s/00-namespace.yaml

# Create the secret with your keys (you're supposed to remove the placeholders with the actual values...)
kubectl create secret generic agent-secrets -n agent \
 --from-literal=ANTHROPIC_API_KEY=sk-ant-... \
 --from-literal=COMPOSIO_API_KEY=ak_... \
 --from-literal=JWT_SECRET=$(openssl rand -hex 32)

With that in place, deploy the app and set up autoscaling:

# Install KEDA, then apply the scalers
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda -n keda --create-namespace --wait

kubectl apply -f infra/k8s/40-keda-worker-scaledobject.yaml -f infra/k8s/41-gateway-hpa.yaml

Finally, port-forward the gateway so you can reach it from your machine:

# Port-forward the gateway to localhost:8000
kubectl -n agent port-forward svc/gateway 8000:8000

Point the frontend at http://localhost:8000 and you're ready to start tasks.

Note: You don't need to touch the .env files in apps/worker/ or apps/gateway/ for this. Those are only for running the apps directly on your machine.

In the cluster, the pods get their config from the ConfigMap and the secret you just created gets injected as environment variables at runtime.

Core Components in the Application

The project is huge. Walking through every single line from scratch would turn this into an hours-long read, so instead I'll focus on the core components that actually make the system work.

The Agent Loop

The agent loop is the brain of the entire system. Every time a task gets dispatched, this is what runs.

The idea is simple even if the implementation isn't. Give the LLM a goal, let it reason, let it call tools, feed the results back, and repeat until it's done.

async def run_agent(
user_id: str,
goal: str,
toolkit_hint: str | None = None,
) -> dict:

It takes three things: the user's ID (so Composio knows which connected accounts to use), the goal itself, and an optional toolkit hint. The hint lets you scope which tools get loaded. If the task is clearly a Gmail job, passing "gmail" avoids loading every tool the user has connected.

Before the loop starts, it creates a Composio session and fetches the tools for that user:

session = await create_session(user_id, toolkit_hint);
tools = await get_tools(session);

Then the actual loop runs:

for turn in range(1, settings.max_iterations + 1):
    response = await client.messages.create(
        model=settings.model,
        max_tokens=settings.max_tokens,
        system=SYSTEM_PROMPT,
        tools=tools,
        messages=messages,
    )

if response.stop_reason == "end_turn":
return finish("completed", _extract_text(response.content))

if response.stop_reason == "tool_use":
      # execute the tools, append results, continue
      # ...

Each turn, Claude looks at the goal and the conversation history, then decides what to do next. The stop_reason tells what happened:

• "end_turn": Claude is done. It completed the task and is returning a final answer.

• "tool_use": it wants to call one or more tools. The loop executes those through Composio, appends the results back into the message history, and goes around again.

If a tool call fails, the error gets fed back into the conversation rather than crashing the run:

except ComposioError as exc:
tool_result_blocks = [
    {
        "type": "tool_result",
        "tool_use_id": block.id,
        "content": f"Tool execution failed: {exc}",
        "is_error": True,
    }
        for block in tool_use_blocks
]

The loop runs for at most max_iterations turns which is 20 by default, defined in apps/worker/agent/config.py. If it hits that ceiling without finishing, it returns a max_iterations_reached status instead of hanging indefinitely.

Every run_agent call returns the same dict shape: a status, a summary, and a list of every step taken. That consistent shape is what makes it straightforward for Temporal to store and inspect the result, which we'll get to next.

Making it Durable with Temporal

The agent loop by itself has a real problem. If the worker process crashes partway through a 15-step task, everything is gone. You have no way to know how far it got, and you have to start from scratch.

Workflows and Activities

Temporal splits your code into two distinct pieces: workflows and activities.

A workflow describes what should happen and in what order, but it never does the actual work itself. No network calls, nothing. That constraint is exactly what lets Temporal safely replay it to reconstruct state after a crash.

An activity is where the real work happens. Network calls, LLM requests, tool executions – all of that goes inside an activity. Activities can fail and be retried independently without affecting the workflow state.

In this project, AgentWorkflow in apps/worker/workflows.py is the workflow, and run_agent_activity in apps/worker/activities.py is the activity that wraps the agent loop.

The Workflow

@workflow.defn(name="AgentWorkflow")
class AgentWorkflow:
    def init(self) -> None:
        self._status: str = "running"
        self._result: dict | None = None

When a task gets dispatched, Temporal starts this workflow. It sets up a retry policy and hands all the real work off to the activity:

retry = RetryPolicy(
  (initial_interval = timedelta((seconds = 2))),
  (backoff_coefficient = 2.0),
  (maximum_interval = timedelta((minutes = 2))),
  (maximum_attempts = 5),
  (non_retryable_error_types = ["ValueError", "AuthenticationError"]),
);

result = await workflow.execute_activity(
  run_agent_activity,
  (args = [user_id, goal, toolkit_hint]),
  (start_to_close_timeout = timedelta((minutes = 30))),
  (retry_policy = retry),
);

The start_to_close_timeout is set to 30 minutes and caps at 5 attempts, because agent tasks can genuinely take that long. You can increase or decrease the timer based on your work requirement.

Querying the Workflow

One thing that makes Temporal convenient here is query handlers. The workflow exposes its current status and result without needing a separate database to track it:

@workflow.query
def status(self) -> str:
return self._status

@workflow.query
def result(self) -> dict | None:
return self._result

The gateway can ask Temporal "what is the status of workflow X?" at any point and get a live answer back. That's how the frontend polling works.

The Activity

The activity is straightforward. It wraps run_agent and logs what happens:

@activity.defn(name="run_agent_activity")
async def run_agent_activity(user_id: str, goal: str, toolkit_hint: str | None) -> dict:
    result = await run_agent(user_id=user_id, goal=goal,             toolkit_hint=toolkit_hint)
    return result

Anything that touches the network lives here, not in the workflow. That separation is what lets Temporal do its job.

The Worker

The worker process is what registers everything and starts polling the queue:

worker = Worker(
            client,
            task_queue=temporal_settings.temporal_task_queue,
            workflows=[AgentWorkflow],
            activities=[run_agent_activity, notify_activity],
            max_concurrent_activities=5,
)

It connects to Temporal, registers the workflow and activities, and listens on the task queue. When a task arrives, it picks it up and runs it. This is the process running inside the Kubernetes pod, and it's exactly what KEDA will scale based on queue depth later.

The Agent Gateway

The gateway is a FastAPI app that sits between the user and Temporal. It handles task dispatch, status polling, and cancellation. Crucially, it never runs agent code itself. Its only job is to talk to Temporal and return quickly.

Dispatching a Task

The dispatch endpoint in apps/gateway/routes/tasks.py is where everything begins:

  @router.post("/dispatch", response_model=DispatchResponse)
  async def dispatch(
      body: DispatchRequest,
      user_id: str = Depends(current_user_id),
  ) -> DispatchResponse:
      if body.toolkit:
          access = await check_toolkit_access(user_id, body.toolkit)
          if not access["allowed"]:
              raise HTTPException(
                  status_code=status.HTTP_409_CONFLICT,
                  detail={
                      "error": "toolkit_not_connected",
                      "connect_url": access["connect_url"],
                  },
              )

      workflow_id = f"agent-{user_id}-{uuid.uuid4().hex[:8]}"
      await client.start_workflow(
          WORKFLOW_NAME,
          args=[user_id, body.goal, body.toolkit],
          id=workflow_id,
          task_queue=settings.temporal_task_queue,
          cron_schedule=body.schedule or "",
      )
      return DispatchResponse(workflow_id=workflow_id, status="dispatched")

The request carries three fields: the goal, an optional toolkit name (to not spend time figuring out toolkit names), and an optional cron schedule. The endpoint runs a preflight check, hands the task off to Temporal, and returns the workflow ID immediately. The user doesn't wait for the agent to finish.

Notice the cron_schedule field. Passing a standard cron expression here turns the task into a recurring job. Temporal handles the scheduling itself, no extra infra needed.

The Preflight Check

The preflight check lives in apps/gateway/routes/preflight.py. Before a task gets dispatched, it verifies that the user actually has the required toolkit connected in Composio:

  async def check_toolkit_access(user_id: str, toolkit_hint: str | None) -> dict:
      if not toolkit_hint:
          return {"allowed": True}

      connected = await asyncio.to_thread(
          _has_active_account, composio, user_id, toolkit_hint
      )
      if connected:
          return {"allowed": True}

      connect_url = await asyncio.to_thread(
          _connect_link, composio, user_id, toolkit_hint
      )
      return {"allowed": False, "toolkit": toolkit_hint, "connect_url": connect_url}

If the connection is missing, the gateway returns a connect_url so the user can authorize the app right away. This matters especially for scheduled tasks.

Checking Status

Once a task is running, the frontend polls this endpoint:

  @router.get("/{workflow_id}", response_model=TaskStatusResponse)
  async def get_task(workflow_id: str, ...) -> TaskStatusResponse:
      if not _owns(workflow_id, user_id):
          raise HTTPException(status_code=404, detail="task not found")

      handle = client.get_workflow_handle(workflow_id, run_id=run_id)
      agent_status = await handle.query("status")

      if desc.status == WorkflowExecutionStatus.COMPLETED:
          result = await handle.query("result")

      return TaskStatusResponse(...)

The status and result come straight from Temporal's query handlers that you saw in the workflow. There's no separate status table, and no database write after each step. Temporal is the source of truth.

Containerizing the Application

The gateway and the worker are packaged as two separate images. They share nothing at runtime, which is exactly what you want since they scale independently and have different responsibilities.

Both Dockerfiles live in the /docker directory, and use a multi-stage build.

Why Multi-Stage? 🤔

The builder stage installs compilers and build tools to compile Python packages. The runtime stage gets only the finished dependencies and the application code. There's no point in putting the build tools into the final image.

The Gateway Image

FROM python:3.14-slim-bookworm AS builder

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install -r requirements.txt

FROM python:3.14-slim-bookworm AS runtime

ENV PYTHONUNBUFFERED=1 PATH="/opt/venv/bin:$PATH"

RUN useradd --create-home --uid 10001 app
WORKDIR /app

COPY --from=builder /opt/venv /opt/venv
COPY . .

USER app
EXPOSE 8000
CMD ["python", "main.py"]

The runtime stage copies the venv from the builder, drops into a non-root user, and starts the FastAPI app. And as always, we run it as a non-root user (be a good dev and follow proper security practices 😺).

The Worker Image

The worker Dockerfile is nearly identical with one small difference:

# procps gives us pgrep for the liveness probe
RUN apt-get update \
 && apt-get install -y --no-install-recommends procps \
 && rm -rf /var/lib/apt/lists/*

It installs procps so the Kubernetes liveness probe can run pgrep to check that the process is still alive.

Building and Loading the Images

The build script in scripts/build-images.sh builds both images, passing each app directory as the build context:

docker build \
 -f "$ROOT/docker/Dockerfile.gateway" \
    -t "agent-gateway:$TAG" \
 "$ROOT/apps/gateway"

docker build \
 -f "$ROOT/docker/Dockerfile.worker" \
    -t "agent-worker:$TAG" \
 "$ROOT/apps/worker"

The Dockerfiles live under docker/ but each one is built against its own app directory. That's what COPY . . actually copies.

After building, there's one more step before the images can run in the cluster. A local k3d cluster has no access to your Docker daemon, so images built locally aren't accessible to it. You have to import them explicitly:

k3d image import "agent-gateway:dev" "agent-worker:dev" -c agent

scripts/load-images.sh does this for you. Once the import completes, the cluster can pull the images like it usually does and your pods will start. 🎊

Deploying to Kubernetes

With the images built and loaded into the cluster, the next step is applying the manifests. The setup is organized into two tiers. Tier 1 is the core application: the namespace, config, and deployments. Tier 2 is autoscaling, covered in the next section.

Config and Secrets

Non-sensitive config lives in a ConfigMap at infra/k8s/01-configmap.yaml:

data:
  MODEL: "claude-opus-4-8"
  MAX_TOKENS: "4096"
  MAX_ITERATIONS: "20"
  TEMPORAL_HOST: "temporal-frontend.temporal.svc.cluster.local:7233"
  TEMPORAL_TASK_QUEUE: "agent-tasks"
  GATEWAY_HOST: "0.0.0.0"
  GATEWAY_PORT: "8000"

This is where the Temporal host address comes from. Notice that it uses the full in-cluster DNS name pointing at the Temporal frontend service in the temporal namespace. That address only resolves from inside the cluster, which is fine since both the gateway and the worker run there.

API keys go in a Kubernetes Secret that you create manually and never commit in Git. Both the ConfigMap and the Secret are mounted as environment variables using envFrom in each deployment.

The Gateway Deployment

spec:
  replicas: 2
  containers:
    - name: gateway
      image: agent-gateway:dev
      imagePullPolicy: IfNotPresent
      command:
        [
          "python",
          "-m",
          "uvicorn",
          "main:/app",
          "--host",
          "0.0.0.0",
          "--port",
          "8000",
        ]
      readinessProbe:
        httpGet:
          path: /health
          port: 8000
      resources:
        requests:
          cpu: 100m
          memory: 256Mi
        limits:
          cpu: 500m
          memory: 512Mi

A few things worth noting. imagePullPolicy: IfNotPresent tells Kubernetes to use the locally loaded image instead of trying to pull from a registry. The startup command bypasses the reload=True flag that main.py uses when run directly locally. The readiness probe hits /health before Kubernetes sends any traffic to the pod, so the gateway only receives requests once it's actually up.

The gateway also gets a ClusterIP Service so other pods and the port-forward can reach it:

apiVersion: v1
kind: Service
metadata:
  name: gateway
spec:
  type: ClusterIP
  ports:
    - port: 8000
      targetPort: 8000

The Worker Deployment

# just polls Temporal.
spec:
  replicas: 1
  containers:
    - name: worker
      image: agent-worker:dev
      livenessProbe:
        exec:
          command: ["pgrep", "-f", "worker.py"]
        initialDelaySeconds: 15
        periodSeconds: 20
      resources:
        requests:
          cpu: 250m
          memory: 512Mi
        limits:
          cpu: "1"
          memory: 1Gi

The worker has no Service. It never accepts incoming connections. It connects outward to Temporal and polls for work, so nothing needs to reach it. That's why procps was installed in the Dockerfile.

The worker also gets more resources than the gateway. It's the one running LLM calls and executing tools, so it needs more resources. You can cap it depending on your requirements.

Applying Everything

The deploy script at scripts/deploy.sh applies Tier 1 in the correct order:

kubectl apply -f "$K8S/00-namespace.yaml"
kubectl apply -f "$K8S/01-configmap.yaml"
kubectl apply -f "$K8S/10-gateway-deployment.yaml"
kubectl apply -f "$K8S/20-worker-deployment.yaml"

Order matters here. The namespace has to exist before anything else can be created inside it, and the ConfigMap has to exist before the pods that read from it start up.

Autoscaling with KEDA

Kubernetes scales pods based on CPU or memory. That works fine for the gateway, which handles HTTP requests and actually uses CPU proportional to traffic. But it's the not the right signal for workers.

The worker sits completely idle when no tasks are queued. It doesn't burn CPU waiting. When tasks arrive, it gets busy fast. What you actually want to scale on is queue depth: how many tasks are waiting to be picked up.

That's what KEDA does. It reads external metrics like queue lengths, message counts, or in this case Temporal task queue depth, and scales your deployments accordingly.

Scaling the Worker

The ScaledObject in infra/k8s/40-keda-worker-scaledobject.yaml is what KEDA watches:

spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 10
  cooldownPeriod: 120
  triggers:
    - type: temporal
      metadata:
        endpoint: temporal-frontend.temporal.svc.cluster.local:7233
        namespace: default
        taskQueue: agent-tasks
        queueTypes: "workflow,activity"
        targetQueueSize: "5"
        activationTargetQueueSize: "0"

Let's walk through the important fields:

• minReplicaCount: 0 is the big one. KEDA can scale to zero, which a standard HPA can't do. When the queue is empty, every worker pod shuts down. You pay for nothing while the system is idle.

• activationTargetQueueSize: "0" means KEDA wakes the deployment the moment a single task enters the queue. Zero tasks, zero pods. One task, pods start spinning up.

• targetQueueSize: "5" tells KEDA to target roughly one worker pod per 5 pending tasks. Ten tasks in the queue means two pods.

• cooldownPeriod: 120 adds a 120-second buffer before KEDA scales back down after the queue clears.

• queueTypes: "workflow,activity" watches both queues. Without this, KEDA would only see part of the pending work.

Note: The Temporal scaler requires KEDA v2.17 or later. Make sure your Helm install is on that version or above.

Scaling the Gateway

The gateway gets a plain CPU-based HPA at infra/k8s/41-gateway-hpa.yaml:

spec:
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

CPU is the right signal here because the gateway does real work proportional to incoming HTTP requests. It stays at a minimum of 2 replicas so there's no cold start delay on the API side.

Installing KEDA

KEDA is installed via Helm before applying the ScaledObject:

helm install keda kedacore/keda -n keda --create-namespace --wait
kubectl apply -f infra/k8s/40-keda-worker-scaledobject.yaml -f infra/k8s/41-gateway-hpa.yaml

Once those are applied, the system is fully operational. Submit a task and watch a worker pod appear. Let the queue empty and watch it disappear. That's the whole point.

And just like that, you have a fully durable, autoscaling AI Agent that you can schedule to run anytime. How cool is that? 😎

Agent in Action

Here's a quick demo of the agent in action (running inside a Kubernetes Cluster):

Conclusion

Running AI agents in production is a completely different problem than building them. I tried to focus on that gap here, and hopefully it gave you a solid reference for how to think about durability and scaling. And I hope it also helped you build or understand something different from a regular AI chat application.

The Temporal and KEDA combination is really something you should learn and know more about if you're into building AI agents or doing DevOps in general. Temporal helps with the biggest issue with AI agents (the durability), and KEDA makes sure that you aren't paying for idle workers at 2am (if used in prod) if nothing is running. You aren't just scaling on CPU, but based on events and that is important.

There's a lot of room to extend this from here. You could swap the dev JWT for proper OIDC, or expand the toolkit coverage through Composio to support more of your workflows.

The foundation is there. The rest is just building on top of it.

You can find the complete source code here: shricodev/kron-k8s-agent

Kubernetes can seem intimidating at first, but Amazon EKS (Elastic Kubernetes Service) makes it much more approachable, especially when you have a step-by-step guide to follow.

In this tutorial, we'll walk through exactly how to get a Spring Boot application with a MySQL database up and running on Amazon EKS. I'll take you from from containerizing your app to connecting it to a managed database, all the way to accessing it live in the cloud. Let’s get started.

Table of Contents

• Prerequisites

• Application Overview

• What is Amazon EKS?

• How to Deploy a Spring Boot App with MySQL on Amazon EKS

  • Step 1: Create the VPC

  • Step 2: Set Up the MySQL Database in a Private Subnet

  • Step 3: Deploy EC2 Instance in a Public Subnet

  • Step 4: Create SSH Tunneling for the Database

  • Step 5: Set Up a Simple SpringBoot Application Development

  • Step 6: Configure SpringBoot App for Database

  • Step 7: Dockerize the Spring Boot Application

  • Step 8: Push the Image to Elastic Container Registry (ECR)

  • Step 9: Implement AWS App Load Balancer

  • Step 10: Create a Cluster in EKS

  • Step 11: Install AWS Load Balancing

  • Step 12: Create and Deploy Kubernetes

  • Step 13: Delete Cluster

• Conclusion

Prerequisites

Before you begin, ensure you have the following:

• Basic knowledge of AWS (AWS Console access).

• Basic knowledge of containerization.

• Working knowledge of Kubernetes.

• Basic knowledge of databases.

• Helm installed

• Kubectl installed

• Eksctl installed

• An IDE

Application Overview

The application runs inside an AWS VPC spread across two availability zones for high availability. When a user makes a request, it flows through an Internet Gateway into an AWS Application Load Balancer sitting in the public subnet, which handles incoming traffic via an Ingress rule.

The Load Balancer routes requests to the App Service, which distributes them across multiple App Pods running inside AWS EKS (Elastic Kubernetes Service) in the private subnets.

The Docker images for these pods are pulled from AWS ECR (Elastic Container Registry). For data persistence, the app pods connect to Amazon RDS MySQL databases through a MySQL External Service, with an RDS instance in each availability zone to ensure redundancy.

A NAT Gateway in the public subnet allows the private resources to make outbound internet calls without being directly exposed to the internet.

What is Amazon EKS?

If you've ever tried to manage containers manually, you already know it can get messy pretty quickly, tracking which containers are running, restarting ones that crash, scaling up when traffic spikes... It's a lot.

That's exactly the problem Kubernetes was built to solve. It automates the deployment, scaling, and management of containerized applications. But setting up and maintaining your own Kubernetes cluster from scratch? That's a whole other challenge.

That's where Amazon EKS comes in. EKS is a fully managed Kubernetes service provided by AWS, which means AWS handles the heavy lifting of setting up, securing, and maintaining the Kubernetes control plane for you. You just focus on deploying your application.

How to Deploy a Spring Boot App with MySQL on Amazon EKS

In this section, we’ll cover the steps to follow in deploying your SpringBoot application with MySQL on Amazon EKS.

Step 1: Create the VPC

To create a VPC, log in to the AWS IAM Console and search for “VPC,” then click create VPC.

Select the "VPC and more option:, and give your VPC a name for your project, for example, spring-demo. Set the IPv4 CIDR block to 10.4.0.0/16. For the NAT gateway configuration, select Zonal, then In 1 AZ.

Select None for VPC endpoints configuration. Next, click Create VPC, then click View VPC. This takes you to the VPC resource map.

Step 2: Set Up the MySQL Database in a Private Subnet

First, you need to create the security group for the MySQL and EC2 instance deployment. To do that, navigate to EC2 > Security Groups. For the inbound rule, select Type: All traffic and Source: Anywhere-IPv4. Then click Create security group.

Next, we’ll create the subnet group for the database. To do that, navigate to Aurora and RDS > Subnet groups and click Create DB subnet group. Next, configure the DB subnet to include:

• Name: private-subnet-db

• Description: private-subnet-db

• VPC: Select VPC

• Add subnets: Choose us-east-1a and us-east-1b as the availability zones, then select the private and public subnets

Click Create**.**

Now, navigate to Databases, click Create database, and select Full configuration. Select MySQL as the engine type.

Select the Free tier when choosing a sample template. Next, give your DB a username and a strong password. Choose db.t3.micro as the instance type.

Select your VPC and associated private subnet. Now, uncheck the "Enable auto minor version upgrade" option in the Additional configuration section and click Create database.

While our database initializes, let's create a key pair for the EC2 instance, which will be launched in a public subnet. To do that, navigate to EC2 > Network & Security > Key Pairs and click Create key pair.

Give your key pair a name, for example, ece-db-key-pair. Leave everything else as-is and click Create key pair. This automatically downloads the key-pair into your local machine.

Step 3: Deploy EC2 Instance in a Public Subnet

Now it’s time to create an EC2 instance. To do this, navigate to EC2 > Instances and click Launch instances. Select the key pair you just created in the Key pair section.

Next, in the Network section, select the VPC created earlier for the project. For Auto-assign public IP, choose Enable. Next, choose the Select existing security group option and select the all-access-sg security group created earlier. Next, click Launch instance.

Step 4: Create SSH Tunneling for the Database

For this step, go into your terminal and navigate to the folder where your key pair is downloaded. Run the ls command, and you should see your key pair there.

Next, you need to change the permission of the key pair file. Use the command below:

chmod 0400 ece-db-key-pair.pem 

Now, run the SSH tunneling command below:

ssh -i <YOUR-KEY-PAIR>.pem -f -N -L <LOCAL-PORT>:<YOUR-RDS-ENDPOINT>:<RDS-PORT> <EC2-USERNAME>@<YOUR-EC2-PUBLIC-DNS> -v

• <YOUR-KEY-PAIR>.pem: the name of your downloaded key pair file

• <LOCAL-PORT>:  the port on your laptop (3306 for MySQL, 5432 for PostgreSQL)

• <YOUR-RDS-ENDPOINT>: found in AWS Console > RDS > Your database > Connectivity & Security > Endpoint

• <RDS-PORT>: same as local port (3306 for MySQL, 5432 for PostgreSQL)

• <EC2-USERNAME>: usually ec2-user for Amazon Linux, ubuntu for Ubuntu

• <YOUR-EC2-PUBLIC-DNS>: found in AWS Console > EC2 > Your instance > Public IPv4 DNS

This command lets your laptop or local machine talk directly to your remote database, as if the database were sitting on your own computer.

After running this command, you can open a database tool (like MySQL Workbench, DBeaver, or TablePlus) on your laptop and connect to:

• Host: localhost

• Port: 3306

For this tutorial, I’ll be using the community version of DBeaver. You can use other similar tools, but if you prefer to use the same tool for the purpose of this guide, you can install the community version from the official DBeaver download page.

After download and installation, open the DBeaver client and click the Connect to a database icon in the top-left corner of the app.

Select MySQL and click Next. On the next window, enter your database username and password, and set Server Host to 127.0.0.1.

Click Test Connection.

You should see a window appear on your screen, indicating that the connection is successful.

Click OK and Finish.

Now, on the left panel, you should see your connection. Expand it to see the database structure.

Now, you have successfully created SSH tunneling for your database.

Troubleshooting

While attempting to test the database connection, I initially ran into a “Plugin 'mysql_native_password' is not loaded” error. If you encounter this error, follow the steps below to fix it.

1. On the Connection Settings window, navigate to the Driver properties tab.

2. Look for allowPublicKeyRetrieval and set it to FALSE.

3. Navigate back to the Main tab and click Test Connection.

Everything should work fine now.

Step 5: Set Up a Simple SpringBoot Application Development

To get started, head over to the Spring Initializr website. Rename Artifact to “springboot-mysql-eks”. Then click ADD DEPENDENCIES… to add dependencies for the REST APIs. Search for the following dependencies:

• Spring Web: Build web apps, including RESTful applications using Spring MVC. Uses Apache Tomcat as the default embedded container.

• Spring Data JPA: Persist data in SQL stores with the Java Persistence API using Spring Data and Hibernate.

• IBM DB2 Driver: A JDBC driver that provides access to IBM DB2.

• Lombok: A Java annotation library that helps to reduce boilerplate code.

Next, click GENERATE at the bottom center of the page. This action downloads a zip file to your local machine. Open this file in an IDE, such as VSCode or IntelliJ IDEA. For this tutorial, I use VSCode. In the build.gradle file, you can see all the added dependencies:

dependencies {
   implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
   implementation 'org.springframework.boot:spring-boot-starter-webmvc'
   compileOnly 'org.projectlombok:lombok'
   runtimeOnly 'com.ibm.db2:jcc'
   annotationProcessor 'org.projectlombok:lombok'
   testImplementation 'org.springframework.boot:spring-boot-starter-data-jpa-test'
   testImplementation 'org.springframework.boot:spring-boot-starter-webmvc-test'
   testCompileOnly 'org.projectlombok:lombok'
   testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
   testAnnotationProcessor 'org.projectlombok:lombok'
}

What we're building

The Spring Boot app is a currency exchange rate and conversion app:

We'll be inserting the exchange data into the database table.

To continue with this tutorial, you can clone the project repo here to save time.

In main > java > com.. > model > ExchangeRate, you’ll see the code below:

package com.example.springbootmysqleks.model;

import jakarta.persistence.*;
import lombok.Getter;
import lombok.Setter;

import java.sql.Date;

@Getter
@Setter
@Entity
@Table(name = "exchange-rate")
public class ExchangeRate {
   @Id
   @GeneratedValue(strategy=GenerationType.AUTO)
   private Integer transactionId;
   private String sourceCurrency;
   private String targetCurrency;
   private double amount;
   private Date lastUpdated;
}

This class is essentially a blueprint for storing currency exchange rate data in our database. It uses the libraries and dependencies added earlier. Lombok handles all the repetitive getter/setter boilerplate so you don't have to write it yourself, while JPA annotations like @Entity and @Table tell Spring, "hey, this class maps to a database table called exchange-rate."

Inside the class, there are five fields that become database columns:

• A self-incrementing transactionId as the primary key.

• sourceCurrency and targetCurrency to track which currencies are being converted,

• The amount holding the actual exchange rate

• lastUpdated date, so you always know how fresh your data is.

To store the data, create a repository file in main > java > com.. > repository > ExchangeRateRepository:

package com.example.springbootmysqleks.repository;

import com.example.springbootmysqleks.model.ExchangeRate;
import org.springframework.data.jpa.repository.JpaRepository;

public interface ExchangeRateRepository extends JpaRepository<ExchangeRate, Integer> {
   ExchangeRate findBySourceCurrencyAndTargetCurrency(String sourceCurrency, String targetCurrency);
}

This file acts as the middleman between your code and the database. By simply extending JpaRepository, you instantly get a whole suite of built-in database operations (like save, delete, findAll, and so on) completely for free, without writing a single SQL query.

The interface is typed to work with the ExchangeRate model we just looked at, using Integer as the primary key type.

The one custom method, findBySourceCurrencyAndTargetCurrency, is where Spring's magic really shines. Just by following a naming convention, Spring automatically figures out the SQL query it needs to run, so you can look up an exchange rate by simply passing in two currency codes like "USD" and "EUR" without writing any query logic yourself.

To use the findBySourceCurrencyAndTargetCurrency method, create a service file in main > java > com.. > service > ExchangeRateService with the code below:

package com.example.springbootmysqleks.service;

import com.example.springbootmysqleks.model.ExchangeRate;
import com.example.springbootmysqleks.repository.ExchangeRateRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ExchangeRateService {

   @Autowired
   private ExchangeRateRepository exchangeRateRepository;

   public ExchangeRate addExchangeRate(ExchangeRate exchangeRate) {
       return exchangeRateRepository.save(exchangeRate);
   }

   public double getAmount(String sourceCurrency, String targetCurrency) {
       ExchangeRate exchangeRate =  exchangeRateRepository.findBySourceCurrencyAndTargetCurrency(sourceCurrency, targetCurrency);
       return exchangeRate == null ? 0 : exchangeRate.getAmount();
   }
}

Here, we created a @Service class that interacts with the repository.

The class has two methods, the addExchangeRate, which simply takes an ExchangeRate object and saves it to the database, and getAmount, which takes a source and target currency, uses our custom repository method to look up the matching record, and then either returns the exchange rate amount or a safe default of 0 if no record is found.

That little ternary check (exchangeRate == null ? 0 : exchangeRate.getAmount()) ensures the app doesn't crash if you query a currency pair that doesn't exist in the database yet.

In main > java > com.. > controller > ExchangeRateService, we have the following code:

package com.example.springbootmysqleks.controller;

import com.example.springbootmysqleks.model.ExchangeRate;
import com.example.springbootmysqleks.service.ExchangeRateService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@RestController
public class ExchangeRateController {

   @Autowired
   ExchangeRateService exchangeRateService;

   @GetMapping("/getAmount")
   public double getAmount(@RequestParam String sourceCurrency, @RequestParam String targetCurrency) {
       return exchangeRateService.getAmount(sourceCurrency, targetCurrency);
   }

   @PostMapping("/addExchangeRate")
   public ExchangeRate addExchangeRate(@RequestBody ExchangeRate exchangeRate) {
       return exchangeRateService.addExchangeRate(exchangeRate);
   }

   @GetMapping("/")
   public String getHealth() {
       return "up";
   }

}

The @RestController annotation tells Spring this class will be serving up REST API endpoints, and again @Autowired wires in the service layer automatically.

There are three endpoints:

1. a GET request to /getAmount that accepts sourceCurrency and targetCurrency as query parameters and returns the exchange rate amount

2. a POST request to /addExchangeRate that accepts a full ExchangeRate object as a JSON body and saves it to the database

3. and finally a simple health check endpoint at / that just returns "up",  which is a common pattern in cloud deployments to let load balancers and orchestration tools know the app is alive and running.

Step 6: Configure SpringBoot App for Database

Now, it’s time to configure the application for the database. Navigate to src > main > resources > application.properties, and you should see this:

spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.url=jdbc:mysql://\({MYSQL_HOSTNAME}:\){MYSQL_PORT}/${MYSQL_DATABASE}?createDatabaseIfNotExist=true
spring.datasource.username=${MYSQL_USERNAME}
spring.datasource.password=${MYSQL_PASSWORD}

spring.jpa.hibernate.ddl-auto=update

spring.jpa.show-sql: true

These are the configurations that allow your app to connect with the database.

• spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver: The driver class for the MySQL database.

• spring.datasource.url=jdbc:mysql://\({MYSQL_HOSTNAME}:\){MYSQL_PORT}/${MYSQL_DATABASE}?createDatabaseIfNotExist=true: This is the data source URL in which we are using the MySQL hostname (127.0.0.1), port name, and database name.

• spring.datasource.username=${MYSQL_USERNAME}: your database user name.

• spring.datasource.password=${MYSQL_PASSWORD}: your database password.

One thing to note: the process of configuring environment variables with your actual credentials varies depending on the IDE you're using. If you're using IntelliJ IDEA, this process is pretty straightforward. If you're using VS Code, the process is different.

To configure your actual credentials for the env variables, create a .vscode/launch.json file in your project root folder and paste in the following configuration:

{
 "version": "0.2.0",
 "configurations": [
   {
     "type": "java",
     "name": "Spring Boot App",
     "request": "launch",
     "mainClass": "com.example.springbootmysqleks.SpringbootMysqlEksApplication",
     "projectName": "springboot-mysql-eks",
     "env": {
       "MYSQL_HOSTNAME": "localhost",
       "MYSQL_PORT": "3306",
       "MYSQL_DATABASE": "exchangedb",
       "MYSQL_USERNAME": "root",
       "MYSQL_PASSWORD": "CHANGE_ME"
     }
   }
 ]
}

Configure the credentials to use your actual credentials.

Now, when you run the app, you should be able to see the created exchangedb table in DBeaver:

Use an API testing tool like Postman to send a POST request to the database:

Next, run the select * from exchange_rate er script in the exchangedb SQL script editor:

At the bottom of the editor, you should see the created table from the Postman request.

Now, run a GET request to the endpoint below:

http://localhost:8080/getAmount?sourceCurrency=USD&targetCurrency=EUR&transactionId=1

You should get a 200 OK response with the currency exchange value, for example, 0.93.

Step 7: Dockerize the SpringBoot Application

To Dockerize your application, create a file named Dockerfile and paste in the configuration below:

FROM eclipse-temurin:17-jre-jammy
WORKDIR /app
COPY build/libs/springboot-mysql-eks.jar /app
EXPOSE 8080
CMD ["java", "-jar", "springboot-mysql-eks.jar"]

Our Dockerfile starts by pulling the lightweight eclipse-temurin:17-jre-jammy base image to keep things lean, then sets /app as the working directory inside the container. It copies our compiled Spring Boot JAR file from the local build/libs/ folder into that directory, exposes port 8080 for incoming traffic, and finally runs the app with java -jar when the container starts up.

Next, build the app to create the .jar file. To do that, run the command below:

./gradlew clean assemble 

You should get a successful build output as shown below:

Navigate to build > the libs folder. You’ll see the springboot-mysql-eks file created.

If you run into an “operation couldn’t be completed.” error, try running the export commands to fix this issue. If you’re using a Mac, then run the command below:

brew install openjdk@21

Next, run the export commands:

export JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home

export PATH=\(JAVA_HOME/bin:\)PATH

Then run the ./gradlew clean assemble command again.

Step 8: Push the Image to Elastic Container Registry (ECR)

In this next step, we’ll create an Amazon ECR and push our image to the registry.

To get started, head back into your AWS Console and search for “ECR”. On the ECR page, click Create**.** Then, enter a repository name, for example, “springboot-mysql-eks.” Next, click Create.

Next, select the repo and click View push commands at the top of the page. This presents a window with a bunch of commands you can use to push your image to the registry. Open your terminal and run these commands. You'll need to ensure Docker is running on your local machine before running the commands.

After running the commands, you should see that your image has been successfully pushed to the registry.

Step 9: Implement AWS App Load Balancer

Before getting started with this step, make sure you check out the installation steps and link to additional AWS documentation in the project README. This will help you follow along.

Now, to get started, create a new folder in your root directory named cluster . This is where you'll download the AWS IAM policy for the load balancer. To download the policy, go into your terminal and cd into cluster, then run the command below:

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.14.1/docs/install/iam_policy.json

This command is gotten from the AWS documentation. Now, when you go to the folder, you’ll see an iam_policy.json file automatically generated.

Next, apply the IAM policy using the command below:

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

You should get an output like this in your terminal:

This shows that the IAM policy has been successfully created. To confirm this, head over to the IAM section in your console, navigate to Policies**,** and search for “AWSLoad…”. You should see the policy created there.

The next step is creating the Kubernetes service account. But before that, you need to tag your public and private subnets as described in this documentation.

Now, head over to the VPC dashboard, navigate to Subnets, click into a subnet, and navigate to Tags. Then, click Manage tags.

Click Add new tag, then enter the key/pair value in the documentation.

Step 10: Create a Cluster in EKS

To create a Kubernetes cluster on EKS, you need the eksctl CLI. Follow the instructions in the AWS eksctl documentation to install the CLI. Next, you need a config file schema to create the cluster. To use this schema, create a new file called cluster.yaml in the cluster folder.

Next, paste in the following configurations:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: spring-test-cluster
  region: us-east-1
  version: "1.30"

vpc:
  id: "<your-vpc-id>"
  subnets:
    private:
      us-east-1a:
        id: "<your-private-subnet-1a-id>" # spring-demo-subnet-private1-us-east-1a
      us-east-1b:
        id: "<your-private-subnet-1b-id>" # spring-demo-subnet-private2-us-east-1b
    public:
      us-east-1a:
        id: "<your-public-subnet-1a-id>" # spring-demo-subnet-public1-us-east-1a
      us-east-1b:
        id: "<your-public-subnet-1b-id>" # spring-demo-subnet-public2-us-east-1b

nodeGroups:
  - name: ng-1
    labels: { role: backend }
    instanceType: t2.micro
    desiredCapacity: 3
    minSize: 3
    maxSize: 5
    privateNetworking: true
    ssh:
      allow: true
      publicKeyName: <your-ec2-key-name>
    iam:
      withAddonPolicies:
        imageBuilder: true
        awsLoadBalancerController: true
        autoScaler: true
iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      attachPolicyARNs:
        - arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy

Th ClusterConfig file is used by eksctl to create our EKS cluster called spring-test-cluster in the us-east-1 region, running Kubernetes version 1.30. It plugs into our existing VPC, placing the worker nodes across private subnets in two availability zones us-east-1a and us-east-1b) for high availability, while keeping public subnets available for the load balancer.

The node group spins up t2.micro EC2 instances with a desired count of 3 (scaling up to 5 if needed), all with private networking enabled for security. It also sets up the necessary IAM permissions for the AWS Load Balancer Controller, Auto Scaler, and ECR image access so our cluster has everything it needs to manage traffic and pull our Docker images automatically.

Now, after updating your configuration with your credentials, run the command below:

eksctl create cluster -f cluster.yaml

This creates the cluster. You should see an output like this on your terminal:

Now, in your AWS console, navigate to CloudFormation, and you’ll see your cluster creation process in progress.

Now, when you go into the EC2 instance page, you should see the three nodes created.

Step 11: Install AWS Load Balancing

The next step is installing a load balancer for our application. To get started, run the command below:

 kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"

This installs custom resource definitions (CRDs) for our controller. Next, run the command below to add the Helm chart repo.

helm repo add eks https://aws.github.io/eks-charts

Update your local repo to ensure you have the most recent charts:

helm repo update eks

Next, install the Helm chart:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --version 1.14.0

Next, verify that the controller is installed:

kubectl get deployment -n kube-system aws-load-balancer-controller

You should see this on your terminal:

This indicates that your controller is ready.

Step 12: Create and Deploy Kubernetes

To get started, you'll first need to create a Kubernetes manifest file. For that, we’ll use Helm Chart.

helm create ytchart

The command above creates a folder named ytchart with the templates for the components. In this folder, you need to make some configurations for your use case. First, navigate to ytchart > templates and delete the serviceaccount.yaml file, since we already created the service account earlier.

Next, go to values.yaml and make the following changes:

• For repository, navigate to the ECR service page on the AWS Console and copy the image URI.

• Tag is latest.

• Set database name

mysql:
 databaseName: exchangedb

• Change service account creation to false.

• Scroll down a bit more and change the service type to NodePort and port to 8080.

You also need to store the database username and password using secrets. Navigate to the templates folder and go into the file named secrets.yaml. Here, set your database username and password, then comment out the liveness and readiness probe in deployment.yaml.

Next, we’ll create a service to connect to the database. To do that, navigate to the mysql.yaml file, then for externalName. Navigate to the RDS service page on the AWS console and copy the database endpoint.

Now, in the deployment.yaml file, paste in the following configuration:

          env:
            - name: SPRING_DATASOURCE_URL
              value: jdbc:mysql://spring-mysql:3306/{{ .Values.mysql.databaseName }}?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useUnicode=true&useSSL=false&allowPublicKeyRetrieval=true
            - name: SPRING_DATASOURCE_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mysql-username
                  key: username
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-root-password
                  key: password

You have successfully created environment variables to secure your database credentials.

In the ingress.yaml file, paste in the following configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "spring-microservice-ingress"
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/load-balancer-name: spring-alb-test
  labels:
    app: spring-microservice
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "ytchart.fullname" . }}
                port:
                  number: 8080

This is your configuration for the ingress service.

Run the command below to see all your configuration values:

helm template ytchart/

Next, run the command below to deploy the chart:

helm install mychart ytchart

You should see an output like this on your terminal:

Now, when you run kubectl get all, you should see this:

Now, navigate to EC2 > Load balancers, copy the DNS name, and enter it into a browser. You should see the “up” text. This indicates that your application is working properly.

Now, when you call the API using the DNS URL as such:

http://spring-alb-test-260424558.us-east-1.elb.amazonaws.com/addExchangeRate

You should get a 200 OK response. Congratulations, you have successfully deployed a SpringBoot app in Kubernetes!

Step 13: Delete Cluster

If you’re familiar with AWS and the cloud, you should already be aware of how costly it can be to leave resources running for extended periods, especially when you’re not using them actively.

Now that we've come to the end of this tutorial, it’s time to delete the resources.

These are the resources to delete:

• RDS database.

• Cluster using the command eksctl delete cluster -f cluster.yaml.

• Navigate to VPC and delete the NAT Gateway

Conclusion

Deploying a Spring Boot application with MySQL on Amazon EKS involves a lot of moving parts, but each step builds logically on the last.

In this tutorial, you've gone from setting up a VPC and provisioning a managed database to containerizing your app, pushing it to ECR, and finally orchestrating everything with Kubernetes and an Application Load Balancer.

What you get is a production-grade setup with high availability, private networking, secure credential management, and auto-scaling built in. This is the kind of infrastructure that would take significant manual effort to replicate without managed services like EKS and RDS.

As a next step, consider adding HTTPS support via AWS Certificate Manager, setting up horizontal pod autoscaling, or integrating a CI/CD pipeline to automate future deployments. And remember to clean up your AWS resources when you're done experimenting. Your wallet will thank you.

Six months later, you have fourteen microservices, each holding their own API keys, writing their own retry logic, and failing in their own unique ways.

Nobody knows how much you're spending on tokens or which service is hammering the rate limit. And when OpenAI goes down, everything goes down with it.

That scenario plays out across engineering teams every single day, and the root cause is almost always the same: moving fast with LLMs while skipping the infrastructure thinking that holds everything together at scale.

Fortunately, a well-established architectural pattern solves exactly these problems. If you already run Kubernetes, you're more than halfway to implementing it. That pattern is called the LLM Gateway Pattern, and this article walks you through what it is, why it matters, and how to put it into practice.

Table of Contents

• What Is the LLM Gateway Pattern?

  • How It Works

• The Problem Without a Gateway

• Deploying an LLM Gateway on Kubernetes

  • Storing API Keys Securely

  • Defining Routing Rules in a ConfigMap

  • Scaling the Gateway

  • Wiring Up Observability

• Features of an LLM Gateway

  • Multi-Provider Routing

  • Semantic Caching

  • Rate Limiting Per Consumer

  • Fallback and Failover

  • Token Usage Tracking

• Wrapping Up

What Is the LLM Gateway Pattern?

The LLM Gateway Pattern is an architectural approach where all LLM API traffic from your applications flows through a single, centralized proxy service before reaching any external provider. Think of it as the AI equivalent of an API gateway, except it's purpose-built for the unique challenges that come with language models: token budgets, streaming responses, model routing, semantic caching, and multi-provider fallback.

Instead of every service in your cluster talking directly to OpenAI or Anthropic, they all talk to one internal gateway. That gateway handles authentication, routing, rate limiting, logging, and failover. Your application services stay clean and focused on business logic, while the gateway takes on all the messy operational concerns of working with LLMs at scale.

The pattern itself is not new in concept. Engineers have used API gateways for years to manage REST traffic. What makes LLM gateways distinct is that they understand the specific shape of LLM requests, including token counts, model parameters, prompt structure, and streaming semantics.

How It Works

The core components of an LLM Gateway on Kubernetes are straightforward. Here is the high-level flow:

App Pods send requests to the gateway using a standard OpenAI-compatible API format. Because of this, most existing LLM client libraries work without modification — you just change the base URL to point at your internal gateway service.

The Gateway Service receives each incoming request, authenticates the caller, applies any configured rate limits, checks the cache, selects the appropriate upstream provider based on routing rules, and forwards the request. On the way back, it logs token usage and latency before returning the response to the caller.

ConfigMap holds the routing rules. Which model should handle requests tagged as fast? Which provider should the system fall back to if the primary one is unavailable? All of this lives in configuration, not code, so you can update routing behaviour without redeploying anything.

Secrets hold the actual API keys for each provider. The gateway is the only service in the cluster that needs access to them. Application pods never touch provider credentials directly.

Provider endpoints are the actual LLM APIs: OpenAI, Anthropic, a self-hosted vLLM instance running in your cluster, or any other provider that exposes an OpenAI-compatible interface.

The Problem Without a Gateway

To appreciate why this pattern matters, it helps to look at what happens when you skip it.

1. Scattered Secrets and No Central Control

Every service that calls an LLM needs an API key. In Kubernetes, this usually means creating a Secret per namespace or per deployment.

When that key rotates or gets compromised, you're hunting through dozens of manifests to update it. There's no single place to revoke access or audit who is calling what.

2. No Visibility into Cost or Usage

LLM APIs charge per token. Without a centralized layer collecting usage data, you have no reliable way to know which service is responsible for that spike in your monthly bill.

3. Provider Lock-in at the Application Level

When you hardcode https://api.openai.com into your service, switching to a different provider or routing certain requests to a cheaper model becomes a code change. You need to redeploy your application just to change which model handles a request type.

4. No Caching

Many LLM applications send semantically similar or identical prompts repeatedly. Without a shared caching layer, each one incurs full token costs and full latency. The savings from even basic caching can be significant.

All of these problems compound as your team grows and more services start calling LLMs. The gateway pattern cuts through all of them in one architectural decision.

Deploying an LLM Gateway on Kubernetes

There are several tools that can serve as an LLM gateway in a Kubernetes environment, including LiteLLM Proxy, Portkey, OpenRouter, and Envoy with custom filters.

For the rest of this walkthrough, we'll use LiteLLM Proxy. It ships with a Helm chart, supports over a hundred models across all major providers, and comes with a management UI that makes initial configuration straightforward.

Storing API Keys Securely

Start by creating a Kubernetes Secret that holds your provider API keys. Your gateway pods will consume these credentials as environment variables, which means no provider key ever needs to live inside your application containers:

apiVersion: v1
kind: Secret
metadata:
  name: llm-gateway-secrets
  namespace: ai-platform
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-..."
  ANTHROPIC_API_KEY: "sk-ant-..."

Defining Routing Rules in a ConfigMap

The routing configuration tells the gateway which models are available and how to reach each one. Keeping this in a ConfigMap means you can update your routing rules without touching a single line of application code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-gateway-config
  namespace: ai-platform
data:
  config.yaml: |
    model_list:
      - model_name: gpt-4o
        litellm_params:
          model: openai/gpt-4o
          api_key: os.environ/OPENAI_API_KEY
      - model_name: claude-sonnet
        litellm_params:
          model: anthropic/claude-sonnet-4-20250514
          api_key: os.environ/ANTHROPIC_API_KEY
      - model_name: fast
        litellm_params:
          model: openai/gpt-4o-mini
          api_key: os.environ/OPENAI_API_KEY

With this configuration in place, any application in your cluster can reach the gateway at http://llm-gateway.ai-platform.svc.cluster.local using the standard OpenAI client format, regardless of which actual provider sits behind it.

Scaling the Gateway

Because the gateway is stateless, horizontal scaling is straightforward. You can attach a HorizontalPodAutoscaler to scale based on CPU utilization or request rate:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-gateway-hpa
  namespace: ai-platform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-gateway
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Wiring Up Observability

A gateway you can't observe is a gateway you can't trust, so wiring up monitoring before you go to production is worth the extra hour it takes.

LiteLLM exposes a /metrics endpoint in Prometheus format. You can scrape it with a standard ServiceMonitor if you run the Prometheus Operator, or configure Prometheus directly to target the gateway service.

The metrics that matter most in day-to-day operations are token throughput per model, request latency percentiles, error rates per provider, and cache hit ratio.

Once Prometheus is collecting that data, you can build Grafana dashboards that show token spend broken down by caller, model, and time period. This gives engineering managers and finance teams the cost visibility they've been asking for, and it takes surprisingly little effort to set up once the metrics pipeline is in place.

If you run an OpenTelemetry collector in your cluster, you can also configure the gateway to emit trace spans for every LLM request. This lets you see the full latency breakdown from the moment a user action triggers a call in your application all the way through to the provider response. So when something is slow, you can tell immediately whether the bottleneck sits in your service, the gateway, or upstream with the provider.

Features of an LLM Gateway

Not all gateway implementations are equal, so as your needs grow, these are the core capabilities worth evaluating.

Multi-Provider Routing

A well-built gateway routes requests to different providers based on declarative, configurable rules that live entirely outside your application code. This means that changing a model never requires a redeployment.

Semantic Caching

Rather than only caching byte-for-byte identical prompts, a semantic cache uses embedding similarity to recognise when two different prompts are asking essentially the same thing. This can cut redundant API calls dramatically.

Rate Limiting Per Consumer

The gateway should let you set token budgets and request limits per team, per namespace, or per application, so no single runaway service can starve the rest of your cluster or drive up costs unchecked.

Fallback and Failover

When a primary provider fails or exceeds acceptable latency thresholds, the gateway should automatically retry against a configured fallback. This centralizes logic that is notoriously hard to get right inside individual services.

Token Usage Tracking

Every request should produce a detailed usage record capturing input tokens, output tokens, model, caller identity, and latency. This gives engineering managers the clear, actionable picture of AI spending they need.

Wrapping Up

The LLM Gateway Pattern solves a set of operational problems that every team building on language models at scale will eventually run into. Scattered secrets, invisible costs, inconsistent failure handling, and provider lock-in are all symptoms of the same underlying issue: infrastructure concerns leaking into services that shouldn't have to deal with them.

A centralized gateway on Kubernetes gives your application teams a stable, provider-agnostic interface while giving your platform team the visibility and controls they need to manage cost and reliability effectively. When a provider goes down in the middle of the night, your configured fallback kicks in automatically instead of someone waking up to a page.

Start with LiteLLM Proxy, wire up the Prometheus metrics, build a simple Grafana dashboard, and watch how quickly the pattern pays for itself. Once you have seen what centralized LLM traffic management looks like in practice, it becomes very hard to go back to doing it any other way.

Who this is for:

• Platform engineers, SREs, and security-focused cloud architects who operate mixed on‑prem and cloud Kubernetes estates.

• Teams that need scalable, auditable access from on‑prem workloads to GCP resources (especially GPU instances) while minimizing operational overhead and blast radius.

What you’ll get from this guide:

• The motivation and economics behind a hybrid approach (why GPUs often push workloads to the cloud).

• Common pitfalls with service account keys and how “accidental air gaps” occur in real environments.

• A practical, end‑to‑end pattern that uses Workload Identity Federation to give on‑prem pods short‑lived, auditable access to GCP without embedding keys.

What’s included:

• Conceptual explanations, security tradeoffs, and operational best practices.

• Concrete examples and Kubernetes/Terraform artifacts (linked in the GitHub repo at the end of this article) so you can reproduce the setup in your environment.

Read on for the theory, then follow the hands‑on sections to provision GCP resources, configure federation, enforce policies with CEL and Kyverno, and validate secure, scalable GPU access from your on‑prem Kubernetes clusters.

Note: Kubernetes and Terraform artifacts are linked in the GitHub repo at the end of this article.

Table of Contents

• Prerequisites

• Why Hybrid Cloud Matters

• The Economics of Hybrid: GPUs Changed Everything

• Why Service Account Keys Fail at Scale

• How the Accidental Air Gap Happens

• How Workload Identity Federation Bridges the Gap

• How Kubernetes Identity Works

• How to prepare Google Cloud Platform resources

• How to Use CEL for Fine-Grained Access Control

• How to Inject Credentials Automatically with Kyverno

• How to Grant IAM Permissions to Federated Identities

• How to Verify the Setup

• How to Connect On-Prem Apps to Cloud GPUs

• How to Scale GPU Access with CEL Conditions

• The Security Properties Compared

• The Complete Infrastructure as Code Layout

• How to Run a Proof of Concept with vCluster

• Common Issues and How to Solve Them

• Conclusion

Prerequisites

Before following along, you'll need:

• A Kubernetes cluster that is not GKE (on-premises, bare-metal, or a virtual cluster)

• A Google Cloud project with the following APIs enabled: IAM, Security Token Service (STS), and Workload Identity

• Terraform installed and configured

• Kyverno installed in your cluster

• Python 3 with google-cloud-secret-manager and google-cloud-aiplatform libraries (for the verification steps. Code available in the github repository.)

• kubectl access to your cluster

Why Hybrid Cloud Matters

If everything goes right, a hybrid cloud platform lets your on-premises and cloud workloads talk to each other as if they were part of the same network.

There are many practical reasons to run a hybrid cloud setup:

• Offloading analytics to BigQuery: You keep your analytics apps on-prem for data sovereignty, but pipe large datasets into BigQuery for world-class processing power — without buying extra servers.

• Creating a unified network with Cloud Interconnect: Using Cloud Interconnect or Cloud VPN, your on-premises datacenter becomes an extension of the Google Cloud Platform (GCP) Virtual Private Cloud (VPC). Your on-prem invoice apps can talk to cloud-based user services with low latency and no public internet exposure.

• Cost-effective scalability via Cloud Storage: You can use cloud storage as a backend for local apps, storing logs, backups, and historical data while paying only for what you use.

• Event-driven syncing with Pub/Sub: When something happens on-prem, a message through Cloud Pub/Sub lets cloud services react instantly — no manual polling required.

The Economics of Hybrid: GPUs Changed Everything

Before diving into the technical problem, it's worth understanding why hybrid clouds matter more than ever.

Your organization, like most enterprises, has made significant investments in on-premises datacenters. Servers are bought. Racks are filled. Network infrastructure is paid for. The marginal cost of running one more workload is essentially zero.

Then came the AI wave.

Suddenly every team needs Graphics Processing Units (GPUs). Not one or two — dozens of A100s for training, fleets of inference endpoints, vector databases that need to sit close to the models. GPUs are scarce. Lead times for on-prem GPU hardware stretch into months. Cloud providers have them available in minutes.

The architecture that actually makes economic sense looks like this:

• The on-prem datacenter handles the bulk of compute — web servers, business logic, databases, batch processing. This is commodity compute you've already paid for.

• The cloud handles what's scarce — GPU-accelerated inference, model training, AI/ML endpoints. You pay per request, scale on demand, and don't wait six months for hardware.

The cloud isn't a full migration destination — it's an extension for capabilities you can't easily build on-prem.

But those on-prem workloads need to authenticate to cloud services. Every API call from the datacenter to a Vertex AI endpoint, every request to a GPU-powered inference service, every write to Cloud Storage for model artifacts — all of it needs credentials. That's the problem this article solves.

Why Service Account Keys Fail at Scale

Here's a scenario that plays out in thousands of enterprises daily.

A development team needs their on-prem application to write to Google Cloud Storage. The "obvious" solution? Generate a GCP service account key, base64 encode it, store it in a Kubernetes Secret, and mount it in the pod:

apiVersion: v1
kind: Secret
metadata:
  name: gcp-credentials
type: Opaque
data:
  key.json: eyJ0eXBlIjoic2VydmljZV9hY2NvdW50IiwicHJvamVjdF9pZCI6…

This works. It also introduces serious problems:

• Never expires. That key is valid until someone remembers to rotate it (they won't) or it gets compromised (it will).

• Can be exfiltrated trivially. Anyone with read access to that namespace can run kubectl get secret -o yaml and walk away with permanent GCP access.

• Has no audit trail for the actual workload. GCP sees "service-account-xyz accessed this bucket" — not "pod frontend-abc-123 in namespace production."

• Scales terribly. 50 teams × 3 environments × 4 GCP projects = 600 keys to track, rotate, and hope haven't been committed to git.

Security teams know this. That's why many organizations have done the only sensible thing: they have disabled service account key generation entirely.

How the Accidental Air Gap Happens

When you disable key generation, you haven't solved the hybrid cloud platform problem — you've just made it someone else's problem. That someone is usually a platform team staring at a Jira ticket that says "cannot access GCP from on-prem, P1, blocking release."

The result? Your "hybrid cloud platform" isn't hybrid at all. It's two disconnected systems.

Teams resort to building intermediary services, API gateways that proxy requests, or finding creative ways to get keys anyway. None of this is a platform. It's duct tape.

How Workload Identity Federation Bridges the Gap

Every Kubernetes cluster already issues cryptographically signed identity tokens to every pod. And Google Cloud has a service specifically designed to trust those tokens.

This is Workload Identity Federation — and combined with OpenID Connect (OIDC), it's the missing piece that makes hybrid platforms actually work.

The service is quite well named because of the word Federation. it means GCP doesn't store your identity — it agrees to trust identities issued by another system, as long as they can be cryptographically verified. This all works with a very well orchestrated set of steps in the following order:

1. Pod presents its Kubernetes-issued JWT to GCP's STS endpoint.

2. STS verifies the signature against your cluster's public JWKS.

3. STS checks the JWT's claims against the Workload Identity Pool's rules (audience, issuer, CEL conditions).

4. STS returns a short-lived Google access token (typically 1 hour) that the pod uses for API calls.

It is also worth mentioning that Workload Identity Federation is not Kubernetes specific. It works with AWS IAM, Azure AD, GitHub Actions OIDC, and any OIDC-compliant identity provider.

How Kubernetes Identity Works

Every pod with a ServiceAccount gets a JSON Web Token (JWT) automatically mounted at /run/secrets/kubernetes.io/serviceaccount/token. This isn't just an opaque blob — it's a signed assertion of identity:

{
  "iss": "https://kubernetes.default.svc.cluster.local",
  "sub": "system:serviceaccount:production:backend-api",
  "aud": ["https://iam.googleapis.com/..."],
  "kubernetes.io": {
    "namespace": "production",
    "serviceaccount": {
      "name": "backend-api"
    }
  },
  "exp": 1735689600
}

In a JWT, claims are just the key-value pairs inside the token's payload — each one is a claim the issuer is making about the subject. Think of them as facts the token is asserting, signed cryptographically so the verifier can trust them.

The critical insight: this token is created by a set of JSON Web Key Set (JWKS) and is verifiable by anyone who has your cluster's public keys, exposed via the JSON Web Key Set (JWKS) endpoint:

kubectl get --raw /openid/v1/jwks

Google Cloud's Security Token Service (STS) can validate these tokens. No keys are exchanged. No secrets are stored. Just cryptographic proof of identity.

How to Prepare Google Cloud Platform resources

The Workload Identity Pool is a trust boundary — a declaration that says "I accept identities from external sources." The OIDC Provider configures how to validate those identities.

resource "google_iam_workload_identity_pool" "pool" {
  workload_identity_pool_id = "hybrid-platform-pool"
  project                   = "my-project"
}

resource "google_iam_workload_identity_pool_provider" "k8s_provider" {
  project                            = "my-project"
  workload_identity_pool_id          = google_iam_workload_identity_pool.pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "on-prem-cluster"

  attribute_mapping = {
    "google.subject"      = "assertion.sub"
    "attribute.namespace" = "assertion['kubernetes.io']['namespace']"
  }

  attribute_condition = "attribute.namespace in [\"production\", \"staging\"]"

  oidc {
    issuer_uri = "https://kubernetes.default.svc.cluster.local"
    jwks_json  = file("jwks.json")  # Your cluster's public keys
  }
}

Two things to note here:

1. attribute_mapping extracts claims from the Kubernetes JWT and makes them available as GCP attributes. By using `assertion['kubernetes.io']['namespace']`, the namespace is pulled out so you can use it for access control.

2. attribute_condition is where security policy lives. More on this in the next section.

How to Use CEL for Fine-Grained Access Control

The attribute_condition field uses Common Expression Language (CEL). This single line of policy can replace dozens of Identity and Access Management (IAM) bindings:

attribute.namespace in ["production", "staging"]

With this condition, a pod in the kube-system namespace cannot authenticate to GCP at all — the token exchange is rejected before IAM is even consulted.

You can get more sophisticated:

// Only production namespace, and only specific service accounts
attribute.namespace == "production" &&
  attribute.service_account in ["payment-processor", "order-service"]

// Allow staging, but only during business hours
attribute.namespace == "staging" &&
  request.time.getHours("America/New_York") >= 9 &&
  request.time.getHours("America/New_York") < 17

This is defense in depth. Even if someone creates a rogue ServiceAccount or has kubectl access, they cannot authenticate to GCP unless the CEL condition passes. The security boundary is enforced by Google's infrastructure, not by hoping developers follow policy.

How to Inject Credentials Automatically with Kyverno

Having a working identity federation is only half the battle. Your customers and developers shouldn't need to understand OIDC, STS, or credential configuration files. They should deploy their app and have it work.

Before we get to the automation, it's worth pausing on what a credential configuration file actually is — because the name is a little misleading.

A credential configuration file (sometimes called an "external account config" or "ADC config") is a small JSON document that tells Google's client libraries how to obtain a credential at runtime. It is not itself a credential. You'll see the actual file later in this article — it contains no secrets. Just metadata: the Workload Identity Pool audience, the STS token-exchange endpoint, the source token type, and the path on the pod's filesystem where the real (short-lived) Kubernetes ServiceAccount token lives.

Compare that to a traditional service account key:

Both files end up referenced by GOOGLE_APPLICATION_CREDENTIALS and look interchangeable from the application's point of view — but only one of them is dangerous to lose. The credential config file is safe to ship in a ConfigMap precisely because there's nothing to steal.

Having this file in the ConfigMap is half the solution. It actually needs to end up in the workload pods that need access to GCP services. This is where Kyverno comes in. A single ClusterPolicy automatically injects everything a pod needs:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: workload-identity-federation
spec:
  rules:
    - name: inject-gcp-credentials
      match:
        any:
          - resources:
              kinds:
                - Deployment
              selector:
                matchLabels:
                  workload-identity-federation: "enabled"
      mutate:
        patchStrategicMerge:
          spec:
            template:
              spec:
                volumes:
                  - name: workload-identity-credential-configuration
                    configMap:
                      name: workload-identity-federation-config
                containers:
                  - (name): "*"
                    volumeMounts:
                      - name: workload-identity-credential-configuration
                        mountPath: /etc/workload-identity
                        readOnly: true
                    env:
                      - name: GOOGLE_APPLICATION_CREDENTIALS
                        value: "/etc/workload-identity/credential-configuration.json"

The above cluster policy does three things:

1. Mounts the configmap inside the containers in the deployment at /etc/workload-identity.

2. Injects an environment variable called GOOGLE_APPLICATION_CREDENTIALS that points to the absolute path of the credential config file.

From a developer's perspective, this is their entire integration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    workload-identity-federation: "enabled" # That's it.
spec:
  # ... normal deployment spec

The credential configuration file (created by Terraform as a ConfigMap) tells Google's client libraries how to exchange tokens:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": {
    "file": "/run/secrets/kubernetes.io/serviceaccount/token"
  }
}

This JSON file is a credential configuration for Google's Workload Identity Federation. It instructs Google Cloud client libraries to obtain cloud access tokens by exchanging a Kubernetes ServiceAccount token (located at /run/secrets/kubernetes.io/serviceaccount/token) for a Google Cloud access token, using an external identity provider configured via a Workload Identity Pool. This allows workloads running outside of GCP, such as on-premises Kubernetes clusters, to authenticate to Google Cloud services without needing to manage long-lived service account keys.

Every Google Cloud SDK and client library understands this format. Python, Go, Java, and Node.js all just work.

How to Grant IAM Permissions to Federated Identities

The service account token that has been trusted by the STS service, also known as a federated identity, need permissions to access resources. You bind IAM roles to the identity pool attributes:

resource "google_project_iam_member" "secret_access" {
  for_each = toset(["production", "staging"])
  project  = "my-project"
  role     = "roles/secretmanager.secretAccessor"
  member   = "principalSet://iam.googleapis.com/projects/\({PROJECT_NUMBER}/locations/global/workloadIdentityPools/\){POOL_ID}/attribute.namespace/${each.value}"
}

This grants Secret Manager access to all pods authenticated from the production or staging namespaces. The principalSet syntax allows matching on attributes. You can also restrict to specific service accounts:

member = "principal://iam.googleapis.com/.../subject/system:serviceaccount:production:payment-processor"

How to Verify the Setup

You can verify the setup with a simple Python script that lists secrets from Secret Manager. This runs inside a pod on your on-premises cluster:

# list_secrets.py - running on-prem, accessing GCP Secret Manager
from google.cloud import secretmanager

def list_secrets(project_id: str):
    """
    List all secrets in a GCP project.

    No credentials are passed explicitly. The google-cloud-secret-manager
    library automatically:
    1. Reads GOOGLE_APPLICATION_CREDENTIALS env var (set by Kyverno)
    2. Loads the credential configuration JSON
    3. Reads the K8s ServiceAccount token from /run/secrets/...
    4. Exchanges it for a GCP access token via STS
    5. Uses that token to call the Secret Manager API
    """
    client = secretmanager.SecretManagerServiceClient()
    parent = f"projects/{project_id}"

    print(f"Secrets in {project_id}:")
    print("-" * 40)

    for secret in client.list_secrets(request={"parent": parent}):
        secret_name = secret.name.split("/")[-1]
        print(f"  - {secret_name}")

    print("-" * 40)
    print("Authentication: Workload Identity Federation")
    print("Credentials: None stored, token exchanged at runtime")

if __name__ == "__main__":
    list_secrets("my-project-id")

Run this inside your labeled pod:

$ kubectl exec -it my-app-xyz -- python list_secrets.py

Secrets in my-project-id:
----------------------------------------
  - database-password
  - api-key-stripe
  - oauth-client-secret
  - ml-model-api-key
----------------------------------------
Authentication: Workload Identity Federation
Credentials: None stored, token exchanged at runtime

No service account key. No secret mounted. Just a Kubernetes ServiceAccount token exchanged for GCP credentials at runtime.

This same pattern works for any GCP service — Secret Manager, Cloud Storage, BigQuery, Pub/Sub, and Vertex AI.

How to Connect On-Prem Apps to Cloud GPUs

Consider a typical flow: an on-prem order processing service needs to call a Vertex AI endpoint for fraud detection. The model runs on GPUs in Google Cloud (you can spin up A100s in minutes, not months). The application logic stays on-prem (you've already paid for that compute).

With the IAM bindings in place, any pod in the allowed namespaces can call Vertex AI:

# fraud_detector.py - running on-prem, calling cloud GPUs
from google.cloud import aiplatform

def check_fraud(transaction: dict) -> float:
    """
    Call a Vertex AI endpoint for fraud detection.

    The model runs on A100 GPUs in Google Cloud.
    This code runs on-prem in the datacenter.

    Authentication is automatic:
    1. Kyverno injected GOOGLE_APPLICATION_CREDENTIALS
    2. The aiplatform SDK reads the credential config
    3. K8s SA token is exchanged for GCP token via STS
    4. Request is authenticated to Vertex AI
    """
    endpoint = aiplatform.Endpoint(
        endpoint_name="projects/my-project/locations/us-central1/endpoints/fraud-model"
    )
    prediction = endpoint.predict(instances=[transaction])
    return prediction.predictions[0]["fraud_score"]


def generate_embeddings(texts: list[str]) -> list[list[float]]:
    """
    Generate text embeddings using a cloud-hosted model.

    Embedding models are GPU-intensive. Running them on-prem
    would require dedicated hardware. In the cloud, you pay per request.
    """
    from vertexai.language_models import TextEmbeddingModel

    model = TextEmbeddingModel.from_pretrained("text-embedding-004")
    embeddings = model.get_embeddings(texts)
    return [e.values for e in embeddings]

The developer doesn't think about authentication at all. They add the label to their deployment, and their on-prem pod can call:

• Vertex AI endpoints for ML inference on cloud GPUs

• Cloud Storage for model artifacts and training data

• BigQuery for feature stores and analytics

• Pub/Sub for event streaming between environments

• Secret Manager for API keys and configuration

This is the hybrid platform working as intended.

How to Scale GPU Access with CEL Conditions

CEL conditions become especially powerful when you want to restrict GPU access to specific namespaces. For example, to allow only ML-related namespaces to access Vertex AI:

attribute.namespace in ["ml-inference", "ml-training", "data-science"] &&
  attribute.service_account.startsWith("ml-")

You can also grant different access levels per namespace:

# ML inference namespace gets prediction access
resource "google_project_iam_member" "ml_inference" {
  project = "my-project"
  role    = "roles/aiplatform.user"
  member  = "principalSet://iam.googleapis.com/.../attribute.namespace/ml-inference"
}

# Data science namespace gets full Vertex AI access (for experimentation)
resource "google_project_iam_member" "data_science" {
  project = "my-project"
  role    = "roles/aiplatform.admin"
  member  = "principalSet://iam.googleapis.com/.../attribute.namespace/data-science"
}

The on-prem application teams don't need to know or care about GCP IAM. They deploy to the right namespace, add a label, and the platform handles the rest.

The Security Properties Compared

Here's a side-by-side comparison of the two authentication approaches:

The short-lived nature of tokens deserves emphasis. Even in a worst-case scenario where a token is somehow exfiltrated, it expires. Kubernetes ServiceAccount tokens have a configurable lifetime, and the GCP access tokens issued by STS are valid for one hour. A service account key, by contrast, remains valid until someone explicitly rotates it — often years.

The Complete Infrastructure as Code Layout

The entire solution is codified in Terraform, managing both GCP and Kubernetes resources:

workload-identity-federation/
├── providers.tf      # Google + Kubernetes providers
├── locals.tf         # Configuration (namespaces, project ID, etc.)
├── gcp.tf            # Identity pool, provider, IAM bindings
└── kubernetes.tf     # ConfigMap with credential configuration

A single terraform apply:

1. Creates the Workload Identity Pool in GCP

2. Configures the OIDC provider with your cluster's JWKS

3. Sets up IAM bindings for allowed namespaces

4. Creates ConfigMaps in each namespace with the credential configuration

Combined with the Kyverno policy, you get a fully automated pipeline:

New namespace added to allowed list
        │
        ▼
Terraform creates ConfigMap in that namespace
        │
        ▼
Developer deploys with label
        │
        ▼
Kyverno injects credentials automatically
        │
        ▼
Pod authenticates to GCP via OIDC
        │
        ▼
Application accesses GCP services

No tickets. No key requests. No secrets to manage.

How to Run a Proof of Concept with vCluster

To validate this works outside GKE, you can set up a demonstration using vCluster — a virtual Kubernetes cluster that runs inside another Kubernetes cluster. This proves the solution works for any cluster. You can setup vCluster in Docker using vind

# vcluster.yaml
experimental:
  docker:
    nodes:
      - name: worker-1
      - name: worker-2
deploy:
  cni:
    flannel:
      enabled: true
controlPlane:
  distro:
    k8s:
      version: "v1.35.0"

[root@localhost #] vcluster create hybrid --driver docker -f vcluster.yaml
[root@localhost #] kubectl get nodes
hybrid-control-plane   Ready    control-plane   14d   v1.34.0   192.168.107.2   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3
hybrid-worker          Ready    <none>          14d   v1.34.0   192.168.107.3   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3
hybrid-worker2         Ready    <none>          14d   v1.34.0   192.168.107.4   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3

Inside the vCluster, deploy a simple test deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gcp-test
  labels:
    workload-identity-federation: "enabled"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gcp-test
  template:
    metadata:
      labels:
        app: gcp-test
    spec:
      containers:
        - name: test
          image: google/cloud-sdk:slim
          command: ["sleep", "infinity"]

Exec into the pod and verify:

$ kubectl exec -it gcp-test-xxx -- bash

# Inside the pod:
\( gcloud auth login --cred-file=\)GOOGLE_APPLICATION_CREDENTIALS
Authenticated with external account credentials for: [principal://iam.googleapis.com/...]

$ gcloud secrets list --project=my-project
NAME                 CREATED
database-password    2024-01-15T10:30:00Z
api-key              2024-01-14T09:15:00Z

No keys. No secrets mounted. Just identity federation working as designed.

Common Issues and How to Solve Them

How to Handle JWKS Retrieval for Air-Gapped Clusters

If your cluster's OIDC discovery endpoint isn't publicly reachable (most on-prem clusters aren't), you need to manually export the JWKS and upload it to GCP:

kubectl get --raw /openid/v1/jwks > jwks.json

This file must be updated if the cluster's signing keys rotate. Set up a periodic job that checks for key changes and updates the Terraform configuration.

How to Fix Issuer URL Mismatches

The iss claim in the Kubernetes token must exactly match the issuer URL configured in the OIDC provider. For clusters using internal DNS:

issuer_uri = "https://kubernetes.default.svc.cluster.local"

This URL doesn't need to be reachable from GCP — the JWKS file provides the validation keys. But it must match what's in the token exactly.

How to Debug Token Exchange Failures

When authentication fails, the error messages can be cryptic. Common causes and fixes:

Conclusion

After implementing this setup, on-premises workloads authenticate to Google Cloud exactly like GKE workloads do — without a single long-lived credential. The security team is happy (no keys to audit), developers are happy (just add a label), and the platform team is happy (no more credential management tickets).

Here's what you accomplished in this tutorial:

1. /Understood why service account keys fail at scale and the security risks they introduce

2. Created a Workload Identity Pool and OIDC provider in GCP to trust your cluster's token issuer

3. Used CEL conditions to enforce fine-grained, namespace-level access policies

4. Automated credential injection into pods using a Kyverno ClusterPolicy

5. Bound IAM roles to federated identity attributes — no long-lived keys anywhere

6. Verified the setup by calling GCP APIs (Secret Manager, Vertex AI) from an on-prem pod

7. Proved the solution works on any Kubernetes cluster using vCluster

The technologies used here aren't new. OIDC has been in Kubernetes since version 1.20. Workload Identity Federation has been in GCP for years. Kyverno and Terraform are mature tools. What this tutorial puts together is an end-to-end solution that developers can adopt with minimal effort.

If your organization has disabled service account keys (or should), this is the path forward. Your on-prem and cloud clusters can finally be what they were always meant to be: secure extensions of each other.

The complete implementation is available as a Terraform module with Kyverno policies: github.com/shkatara/hybrid-platform-gcp-workload-identity-federation

If this helps, you can follow me on https://www.linkedin.com/in/shubhamkatara/, https://www.youtube.com/@kubesimplify, https://www.linkedin.com/company/kubesimplify/ and

Then the real work starts. Infrastructure must be provisioned. CI/CD pipelines need to be set up. Secrets require management. Monitoring needs wiring. Databases need deployment. Logging needs configuration. Security policies need implementation. Networking rules need review.

Weeks disappear before users see anything useful. Many organizations treat this as normal. They call it engineering rigour. They assume this operational setup phase is simply part of software development.

It is not.

For teams already running production systems, rebuilding infrastructure foundations for every new project is organizational waste. It is repetitive operational labour disguised as an engineering discipline.

The uncomfortable question is not, “How can we do this setup faster?” The real question is: why are we still doing it ourselves at all?

This is where Platform as a Service changes the conversation. A good PaaS shifts the starting point from “rebuild the foundations” to “start shipping. Because new projects should begin closer to customer value, not closer to infrastructure assembly.

In this article, we'll look at why many production teams waste time rebuilding the same infrastructure for every new project, how PaaS helps remove that work, and why engineering teams should question if managing complex infrastructure still makes sense for most projects.

What We'll Cover:

• Most Teams Were Not Hired to Build Infrastructure

• AWS Primitives Are Not a Competitive Advantage

• Most Teams Should Not Be Managing Kubernetes

• PaaS Changes the Starting Point

• Repetition Creates Hidden Organizational Waste

• Standardization Is Usually Faster Than Flexibility

• Platform Teams Become Multipliers

• Easier Starts Create More Innovation

• When Specialized Control Actually Matters

• Starting From Zero Is a Process Failure

Most Teams Were Not Hired to Build Infrastructure

Software teams exist to solve business problems. Customers do not care whether Kubernetes manifests were structured elegantly. They do not admire carefully designed Terraform modules. They do not celebrate handcrafted networking policies.

Customers care about outcomes. They care about faster onboarding. Better recommendations. Smoother payments. Fewer bugs. Simpler workflows.

Yet many engineering organizations spend huge portions of time doing work customers never see.

Teams repeatedly create deployment pipelines. Configure environments. Manage certificates. Set up observability stacks. Tune infrastructure rules. Assemble cloud primitives.

Infrastructure matters. Reliability matters. Security matters.

The problem is duplication. If every project independently recreates the same operational systems, organizations keep rebuilding internal platforms over and over again without admitting it.

This behaviour has become so normalized that teams barely notice it anymore. But rebuilding the same foundation repeatedly is not operational maturity. It is inefficiency scaled across the organization.

AWS Primitives Are Not a Competitive Advantage

Many teams confuse cloud ownership with strategic advantage. Owning Kubernetes clusters does not create differentiation. Managing IAM rules does not create customer value. Writing infrastructure glue code does not strengthen market position.

These are implementation details. Yet many organizations spend extraordinary energy managing them as if they are core business assets.

Some teams effectively become part-time infrastructure companies without realizing it. Their engineers slowly accumulate operational responsibilities until maintaining systems consumes more effort than delivering products.

The outcome becomes predictable. Infrastructure expands. Operational complexity grows. Delivery speed declines. Nobody notices because the pain arrives gradually.

A team starts with one Kubernetes cluster. Then another environment appears. More deployment pipelines emerge. Additional tooling gets layered on top. Logging systems become fragmented. Monitoring evolves differently across products.

Eventually, teams spend increasing amounts of time maintaining systems they never intended to own.

Infrastructure ownership is often not a strategy. It is inertia.

Most Teams Should Not Be Managing Kubernetes

Kubernetes has become an engineering culture. It appears in architecture diagrams, conference talks, hiring requirements, and internal roadmaps. Its adoption often feels inevitable.

But normalization and necessity are not the same thing. Many organizations adopted Kubernetes because industry momentum made it seem like the default path. Not because they had workloads that required its complexity. But the result is predictable.

Small and medium teams end up managing orchestration systems designed for massive operational environments.

They maintain YAML configurations, networking layers, ingress systems, deployment strategies, and operational tooling stacks before delivering meaningful product value. This has become strangely accepted.

A ten-person engineering team maintaining infrastructure patterns designed for internet-scale organizations should raise serious questions. A small team pretending to be a platform team is an operational dysfunction.

Many companies adopt infrastructure complexity built for organizations operating at a vastly different scale. They inherit the burden without inheriting the benefits.

PaaS Changes the Starting Point

Traditional infrastructure approaches force teams to think from the bottom upward. Servers come first. Then operating systems. Then networking. Then deployment systems. Then monitoring. Eventually, applications arrive.

PaaS reverses this sequence. Developers begin with applications and business goals. The platform absorbs operational complexity.

Teams stop asking, “How do we provision resources?” They start asking, “What problem are we solving?” That sounds like a small shift. In practice, it changes everything.

A mature PaaS environment often provides deployment pipelines, integrated observability, databases, scaling behaviour, security controls, and operational standards before a team writes meaningful application logic.

Projects begin with product development rather than infrastructure construction. That dramatically changes time-to-value.

Repetition Creates Hidden Organizational Waste

Organizations often underestimate operational waste because repetitive work feels familiar. Setting up a deployment pipeline may consume only a few days. Configuring logging may feel routine. Creating security rules may seem manageable.

No individual task appears expensive. The cost appears when repetition scales.

If ten projects independently spend two weeks rebuilding nearly identical operational systems, months of engineering capacity disappear. Those engineers could have shipped customer capabilities. They could have reduced friction. They could have tested new ideas. Instead, they rebuilt plumbing.

Engineering teams understand leverage in nearly every other area. Nobody rewrites sorting algorithms for every application. Nobody recreates database engines from scratch. Nobody builds networking stacks repeatedly.

Reuse is accepted as basic engineering wisdom. Infrastructure should not receive special treatment. Build once. Reuse many times.

PaaS simply applies software engineering principles to operational systems.

Standardization Is Usually Faster Than Flexibility

Engineering teams often resist standardization because they fear losing control. Every project feels unique. Every system appears different. The desire for flexibility sounds reasonable.

But complete flexibility often creates operational chaos. Different teams deploy applications differently. Logging behaves inconsistently. Monitoring varies across systems. Security implementations drift.

Documentation fragments. Onboarding slows. Incident response becomes harder. Complexity quietly accumulates.

PaaS introduces constraints, and many engineers instinctively resist constraints. They should not. Useful constraints often increase speed.

Predictable deployment patterns reduce confusion. Shared monitoring standards simplify troubleshooting. Consistent environments reduce cognitive overhead.

Developers spend less energy understanding infrastructure differences and more time delivering product functionality.

Consistency compounds.

Platform Teams Become Multipliers

Many organizations interpret PaaS as buying a vendor product. That misses the bigger idea.

PaaS is fundamentally about creating reusable capabilities. Some organizations buy platforms. Others build internal platforms.

The principle remains the same.

A platform team creates systems once and allows everyone else to benefit. Instead of dozens of product teams independently solving operational problems, a dedicated group centralizes expertise and builds reusable solutions.

The effect becomes substantial. One deployment improvement accelerates every future release. One observability improvement strengthens every application. One security enhancement protects every team.

Platform teams create organizational leverage. Without this model, expertise stays fragmented. With it, expertise compounds.

Easier Starts Create More Innovation

Operational friction changes behaviour. When launching projects becomes expensive, organizations become cautious. Teams avoid experiments. Small ideas feel risky. Prototypes become difficult to justify.

Over time, innovation slows. Not because organizations lack ideas, but because starting became too expensive.

Teams running mature platforms understand this relationship. Reducing startup friction increases experimentation. Smaller projects become practical. Learning cycles become shorter.

New ideas appear more often because the cost of testing them falls dramatically. The easier it becomes to launch something, the more opportunities organizations create.

PaaS reduces startup friction. That reduction changes culture.

When Specialized Control Actually Matters

There are exceptions. Massive data platforms, highly specialized machine learning systems, and extremely customized environments may require lower-level infrastructure ownership.

Some workloads genuinely need deeper operational control. But these scenarios are exceptions, not defaults. Too many teams inherit infrastructure complexity designed for edge cases and treat it as standard practice.

Most production applications do not need custom orchestration layers. Most teams do not need to own Kubernetes. Most engineering groups do not need to spend weeks assembling infrastructure before shipping software.

The default assumption should be the opposite.

Starting From Zero Is a Process Failure

Many organizations normalize unnecessary operational drag. Long setup cycles become accepted. Infrastructure duplication becomes routine. Cloud complexity becomes expected.

Eventually, teams stop questioning it. They assume this is simply how engineering works. It is not.

If launching a new application requires weeks of foundational setup before customer value appears, that is not an engineering discipline.

The goal was never to become an infrastructure company. It was to ship software.

But traffic between your pods? Plaintext by default. Ingress traffic from the internet to your services? Only encrypted if you explicitly configure TLS. And certificates for internal services? You have to provision those yourself.

This is not a Kubernetes oversight. It's a deliberate design choice — Kubernetes provides the primitives and leaves the implementation to you. The problem is that certificate management is notoriously painful. Certificates expire. Provisioning them manually doesn't scale. Forgetting to rotate them causes outages.

cert-manager solves this. It runs as a controller inside your cluster, watches for Certificate resources, requests certificates from configured issuers, stores them in Kubernetes Secrets, and rotates them automatically before they expire. You declare what you want, cert-manager makes it happen and keeps it that way.

In this article you'll work through how cert-manager's core model works, automate public Ingress TLS using Let's Encrypt, set up an internal Certificate Authority for service-to-service encryption, and understand how certificate rotation works so outages caused by expired certificates become a thing of the past.

Prerequisites

• A kind cluster with the nginx Ingress controller installed

• Helm 3 installed

• A domain name with DNS you control — needed for the Let's Encrypt demo

• Basic understanding of TLS: you know what a certificate, a private key, and a CA are

All demo files are in the DevOps-Cloud-Projects GitHub repository.

Table of Contents

• What Is and Isn't Encrypted in Kubernetes

• How cert-manager Works

  • The Four Core Resources

  • Issuers and ClusterIssuers

  • The Certificate Lifecycle

  • ACME Challenges: HTTP-01 vs DNS-01

• Demo 1 — Install cert-manager and Issue a Let's Encrypt Certificate

• How to Get a Wildcard Certificate with DNS-01

• Demo 2 — Set Up an Internal CA for Service-to-Service TLS

• How Certificate Rotation Works

• Cleanup

• Conclusion

What Is and Isn't Encrypted in Kubernetes?

Before installing anything, it's worth being precise about what the cluster already protects and what it leaves open.

The two gaps that matter most in practice are pod-to-pod traffic and Ingress TLS. This article covers both Ingress TLS with Let's Encrypt and internal service-to-service encryption using a private CA.

How cert-manager Works

cert-manager is a Kubernetes operator. It extends the Kubernetes API with custom resources that represent certificate requests and their configuration. When you create a Certificate resource, cert-manager's controller picks it up, requests a certificate from the configured issuer, and stores the resulting certificate and private key in a Kubernetes Secret. When the certificate approaches its expiry, cert-manager renews it automatically.

This model means your application doesn't know or care about certificate management. It reads a Secret. cert-manager keeps that Secret fresh.

The Four Core Resources

cert-manager introduces four custom resources that you'll use regularly:

In practice you'll mostly deal with ClusterIssuer and Certificate. The ClusterIssuer defines where certificates come from. The Certificate defines what certificate you want and where to store it.

Issuers and ClusterIssuers

An Issuer can only issue certificates within its own namespace. A ClusterIssuer can issue certificates in any namespace. For shared infrastructure like Let's Encrypt, you almost always want a ClusterIssuer. For application-specific internal CAs, an Issuer scoped to that application's namespace is the safer choice.

cert-manager supports several issuer types. The three you'll encounter most often are:

ACME — for public certificates from Let's Encrypt or any ACME-compatible CA. Ownership of the domain is proven via an HTTP-01 or DNS-01 challenge.

CA — for internal certificates signed by a CA whose private key is stored in a Kubernetes Secret. Used for service-to-service TLS within the cluster.

Self-signed — generates self-signed certificates. Rarely useful on its own, but essential as the bootstrap step when creating an internal CA.

The Certificate Lifecycle

When you create a Certificate resource, cert-manager follows this sequence:

1. Creates a CertificateRequest with a CSR (Certificate Signing Request)

2. Passes the CSR to the configured issuer

3. For ACME issuers: creates a Challenge resource and fulfils it (more on this below)

4. Receives the signed certificate from the issuer

5. Stores the certificate and private key in the Kubernetes Secret named in spec.secretName

6. Monitors the certificate's expiry — by default, renews when 2/3 of the validity period has elapsed

Your application mounts the Secret. cert-manager updates it silently. Most applications that watch for file changes will pick up the new certificate without a restart.

ACME Challenges: HTTP-01 vs DNS-01

Let's Encrypt needs proof that you control the domain before it issues a certificate. ACME defines two challenge types for this.

HTTP-01 works by having cert-manager create a temporary HTTP endpoint at http://<your-domain>/.well-known/acme-challenge/<token>. Let's Encrypt sends a request to that URL. If the response matches the expected token, the challenge passes. This requires your cluster to be reachable from the internet on port 80.

DNS-01 works by having cert-manager create a temporary DNS TXT record at _acme-challenge.<your-domain>. Let's Encrypt checks for that record. This doesn't require inbound HTTP access, which makes it the right choice for private clusters, and it's the only way to get wildcard certificates (*.example.com).

The trade-off: HTTP-01 is simpler to set up but only works for single domains and requires internet-accessible infrastructure. DNS-01 requires API access to your DNS provider but works for internal clusters and wildcards.

Demo 1 — Install cert-manager and Issue a Certificate Using Pebble and Let's Encrypt

Pebble is Let's Encrypt's local ACME test server. It runs inside your cluster, issues certificates using the same ACME protocol as Let's Encrypt, and requires no public domain or internet access. Using Pebble lets you test the full cert-manager flow — challenge, issuance, renewal — on a plain kind cluster.

Once you understand the flow locally, switching to real Let's Encrypt is a one-line change: replace the ClusterIssuer server URL and point a DNS record at a publicly reachable cluster. The rest of the configuration is identical.

You'll install cert-manager, create a ClusterIssuer for Let's Encrypt, deploy a sample application with an Ingress, and watch a real certificate be issued and stored automatically.

Step 1: Install cert-manager

cert-manager is now distributed via OCI Helm charts from quay.io/jetstack. The --set crds.enabled=true flag installs the Custom Resource Definitions as part of the chart:

helm upgrade cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --install \
  --create-namespace \
  --namespace cert-manager \
  --set crds.enabled=true \
  --version v1.17.0 \
  --wait

You also need the nginx Ingress controller — cert-manager routes HTTP-01 challenges through it. The controller.service.type=ClusterIP override is for kind specifically: the default LoadBalancer Service never gets an EXTERNAL-IP on kind (there's no cloud LB), which makes --wait hang forever. On a real cluster, drop the override and keep LoadBalancer.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=ClusterIP \
  --wait

Confirm all four components are running:

kubectl get pods -n cert-manager
kubectl get pods -n ingress-nginx

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-76f84784c8-r4fx4              1/1     Running   0          6m45s
cert-manager-cainjector-66fbf49587-gv25n   1/1     Running   0          6m45s
cert-manager-webhook-577fddf86-l5wj4       1/1     Running   0          6m45s

NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6c7cd85885-h7zgx   1/1     Running   0          3m34s

kind-specific gotcha — remove the nginx admission webhook now.** On kind, the nginx admission webhook serves with a self-signed certificate that the Kubernetes API server cannot verify. The first time you try to create any Ingress resource you'll see failed calling webhook "validate.nginx.ingress.kubernetes.io": ... x509: certificate signed by unknown authority. Delete the webhook up front so the rest of the demo doesn't trip over it:

kubectl delete validatingwebhookconfiguration ingress-nginx-admission

Step 2: Install Pebble

Pebble is the local ACME test server, distributed by the JupyterHub project. It ships with a companion CoreDNS deployment (pebble-coredns) that Pebble uses to resolve names during ACME validation.

helm install pebble pebble \
  --repo https://jupyterhub.github.io/helm-chart/ \
  --namespace pebble \
  --create-namespace \
  --wait

Confirm both pods are running:

kubectl get pods -n pebble

NAME                              READY   STATUS    RESTARTS   AGE
pebble-8d8d49d64-lz8ck            1/1     Running   0          36s
pebble-coredns-7fb5c7cbf4-4jw9h   1/1     Running   0          36s

Step 3: Wire up DNS for the fake hostname

We're going to issue a cert for echo.pebble.local. That hostname is fake — it doesn't exist in any real DNS — so we have to teach two independent resolvers about it before issuance will work:

If you skip either layer, the Order will go to invalid state with a DNS lookup failure.

First grab the two IPs you'll need:

NGINX_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.spec.clusterIP}')
PEBBLE_DNS_IP=$(kubectl get svc pebble-coredns -n pebble \
  -o jsonpath='{.spec.clusterIP}')
echo "NGINX_IP=\(NGINX_IP  PEBBLE_DNS_IP=\)PEBBLE_DNS_IP"

Patch pebble-coredns to answer for *.pebble.local with the ingress controller's IP. The CoreDNS template plugin parses unreliably when the whole block is collapsed onto one line, so apply a real multi-line ConfigMap:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: pebble-coredns
  namespace: pebble
data:
  Corefile: |
    .:8053 {
      errors
      health
      ready
      template ANY ANY pebble.local {
        answer "{{ .Name }} 60 IN A ${NGINX_IP}"
      }
      forward . /etc/resolv.conf
      cache 2
      reload
    }
EOF

kubectl rollout restart deploy/pebble-coredns -n pebble
kubectl rollout status deploy/pebble-coredns -n pebble

Verify it answers correctly:

kubectl run dnstest --rm -it --restart=Never --image=busybox -- \
  nslookup echo.pebble.local ${PEBBLE_DNS_IP}

You should see Address: <NGINX_IP> in the response. If you get SERVFAIL, check kubectl logs -n pebble deploy/pebble-coredns — a parser error like not a TTL: "}" means the template block collapsed onto one line again.

Patch the cluster CoreDNS so cert-manager's self-check can resolve the same name. Add a stub zone that forwards pebble.local to pebble-coredns:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
    pebble.local:53 {
        forward . ${PEBBLE_DNS_IP}
    }
EOF

kubectl rollout restart deploy/coredns -n kube-system
kubectl rollout status deploy/coredns -n kube-system

Verify the cluster resolver now answers for echo.pebble.local (without specifying a server — it'll use the default kube-dns):

kubectl run dnstest --rm -it --restart=Never --image=busybox -- \
  nslookup echo.pebble.local

Both Server: 10.96.0.10 and Address: <NGINX_IP> should appear.

Step 4: Fetch the Pebble CA and create the ClusterIssuer

Pebble signs its certificates with a self-signed root that lives in the pebble ConfigMap under root-cert.pem. cert-manager needs to trust this CA to talk to Pebble's ACME directory, so we pass it as a base64-encoded caBundle in the ClusterIssuer:

kubectl get configmap pebble -n pebble \
  -o jsonpath='{.data.root-cert\.pem}' > pebble-ca.crt

head -1 pebble-ca.crt   # should print -----BEGIN CERTIFICATE-----

CA_BUNDLE=$(base64 -i pebble-ca.crt | tr -d '\n')
echo "CA_BUNDLE length: ${#CA_BUNDLE}"   # ~1600 chars, one continuous line

Create the ClusterIssuer using the heredoc — the ${CA_BUNDLE} shell variable gets substituted into the YAML before kubectl reads it:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: pebble
spec:
  acme:
    server: https://pebble.pebble.svc.cluster.local/dir
    email: test@example.com
    privateKeySecretRef:
      name: pebble-account-key
    caBundle: ${CA_BUNDLE}
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
EOF

Check the issuer is ready:

kubectl get clusterissuer pebble

NAME     READY   AGE
pebble   True    5s

If READY stays False, the two most common causes are a malformed caBundle (verify it's a single unbroken base64 line with no newlines) or Pebble being unreachable from the cert-manager namespace. To check reachability:

kubectl run test-curl --rm -it --restart=Never \
  --image=curlimages/curl:latest \
  --namespace cert-manager -- \
  curl -k https://pebble.pebble.svc.cluster.local/dir

If that returns JSON, Pebble is reachable.

Step 5: Deploy a sample application

# echo-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: ealen/echo-server:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: default
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80

kubectl apply -f echo-app.yaml

Verify the resources came up:

kubectl get deploy,pod,svc -n default

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/echo   1/1     1            1           32s

NAME                        READY   STATUS    RESTARTS   AGE
pod/echo-5665fbcfdd-mbgxj   1/1     Running   0          36s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/echo         ClusterIP   10.96.103.114   <none>        80/TCP    40s
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   32m

Step 6: Create an Ingress with TLS

The cert-manager.io/cluster-issuer: pebble annotation tells cert-manager to automatically create a Certificate resource for this Ingress, using the issuer we just created. The hostname echo.pebble.local doesn't need to resolve externally — we taught both DNS resolvers about it in Step 3.

# echo-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: pebble
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - echo.pebble.local
      secretName: echo-tls     # cert-manager will create this Secret
  rules:
    - host: echo.pebble.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: echo
                port:
                  number: 80

kubectl apply -f echo-ingress.yaml

Step 7: Watch the certificate being issued

# Watch the Certificate resource (Ctrl-C once Ready=True)
kubectl get certificate echo-tls -n default -w

NAME       READY   SECRET     AGE
echo-tls   False   echo-tls   5s
echo-tls   True    echo-tls   28s

When READY becomes True, the certificate has been issued and stored in the echo-tls Secret. The full chain — CertificateRequest → Order → Challenge → solver pod → Secret — happens in well under a minute on a healthy cluster:

kubectl get certificate,certificaterequest,order,challenge -n default

NAME                                   READY   SECRET     AGE
certificate.cert-manager.io/echo-tls   True    echo-tls   81s

NAME                                            APPROVED   DENIED   READY   ISSUER   AGE
certificaterequest.cert-manager.io/echo-tls-1   True                True    pebble   81s

NAME                                               STATE   AGE
order.acme.cert-manager.io/echo-tls-1-1824732543   valid   81s

(Challenges are deleted automatically once an Order completes, so kubectl get challenge -n default typically shows nothing at this point — that's success, not failure.)

If READY stays False for more than a minute, see the troubleshooting tips at the end of this section.

Inspect the issued certificate to confirm Pebble signed it:

kubectl get secret echo-tls -n default -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -issuer -subject -dates

issuer=CN=Pebble Intermediate CA 05478c
subject=
notBefore=May 17 19:09:22 2026 GMT
notAfter=Aug 15 19:09:21 2026 GMT

Issuer is Pebble's intermediate CA — proof the full ACME flow worked end-to-end. The cert is valid for 90 days, and cert-manager will renew it automatically at day 60.

Hit the ingress over HTTPS from inside the cluster to confirm everything is wired together:

kubectl run curltest --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sk https://echo.pebble.local/

The echo server should return a JSON blob — note the "x-forwarded-proto":"https" field, which proves the request came through nginx over TLS.

Troubleshooting if the cert never goes Ready:

• kubectl describe order -n default — look for "DNS problem" or "Connection refused" in the events.

• kubectl logs -n pebble deploy/pebble --tail=50 — Pebble logs the exact URL it tried to fetch during validation and any errors.

• If the Order is stuck pending with no events: cert-manager hasn't reconciled yet. Wait 30s.

• If the Order is invalid: one of the two DNS layers (Step 3) is misconfigured. Re-run both nslookup checks.

• If the Ingress apply itself failed with an x509 webhook error: you skipped the kubectl delete validatingwebhookconfiguration ingress-nginx-admission step in Step 1.

Step 8: Switch to Let's Encrypt staging (real public domain)

Pebble proved the flow works locally. Now move to a publicly-reachable domain pointed at a publicly-reachable cluster. The DNS gymnastics from Step 3 go away — the domain is real, so both resolvers find it without intervention.

Use Let's Encrypt staging first. It speaks the same ACME protocol as production but with generous rate limits, so failed attempts during testing won't lock you out:

# clusterissuer-staging.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

kubectl apply -f clusterissuer-staging.yaml

# Point the Ingress at staging and the real hostname, then force re-issuance
kubectl annotate ingress echo \
  cert-manager.io/cluster-issuer=letsencrypt-staging --overwrite -n default
kubectl delete secret echo-tls -n default

The new cert's issuer will look something like (STAGING) Let's Encrypt.

Step 9: Switch to Let's Encrypt production

Once staging works, repeat with the production ClusterIssuer. The only difference is the server URL:

# clusterissuer-prod.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

kubectl apply -f clusterissuer-prod.yaml
kubectl annotate ingress echo \
  cert-manager.io/cluster-issuer=letsencrypt-prod --overwrite -n default
kubectl delete secret echo-tls -n default

cert-manager detects the missing Secret and immediately requests a browser-trusted certificate from production Let's Encrypt.

cert-manager detects the missing Secret and immediately triggers a new certificate request using the production issuer.

How to Get a Wildcard Certificate with DNS-01

HTTP-01 challenges work well for single domains with public ingress. But there are two situations where you need DNS-01 instead: when your cluster is not publicly accessible (internal clusters, air-gapped environments, staging namespaces behind a VPN), and when you want a wildcard certificate that covers all subdomains of your domain.

DNS-01 requires cert-manager to be able to create and delete TXT records in your DNS provider. cert-manager has built-in support for Route53, Cloud DNS, Cloudflare, Azure DNS, and many others.

Here is a ClusterIssuer for DNS-01 using AWS Route53:

# clusterissuer-dns01.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-dns01-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            # Use IRSA (IAM Roles for Service Accounts) in production
            # rather than static credentials
            hostedZoneID: YOUR_HOSTED_ZONE_ID

A wildcard Certificate using that issuer:

# wildcard-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: default
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer
  commonName: "*.example.com"
  dnsNames:
    - "*.example.com"
    - "example.com"        # Also cover the apex domain
  duration: 2160h           # 90 days
  renewBefore: 720h         # Renew 30 days before expiry

The resulting Secret wildcard-example-com-tls can be referenced by any Ingress in the default namespace. All subdomains — api.example.com, dashboard.example.com, staging.example.com — are covered by a single certificate that rotates automatically.

For Cloudflare instead of Route53, the solver section looks like this:

    solvers:
      - dns01:
          cloudflare:
            email: your-email@example.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Demo 2 — Set Up an Internal CA for Service-to-Service TLS

Let's Encrypt certificates are great for public-facing services. But for internal services — a gRPC microservice calling another, a web application talking to its database — you don't need public trust. You need a CA that the cluster trusts, and you need it to issue certificates for service names that don't exist as public DNS records.

cert-manager's CA issuer handles this. You create a root CA, tell cert-manager about it, and then issue certificates for internal services using that CA. Every service that trusts the root CA trusts every certificate it issues.

Step 1: Create a self-signed ClusterIssuer

A self-signed issuer generates certificates that are signed by the certificate itself — it is its own CA. You use this as a bootstrap step to create the root CA certificate:

# selfsigned-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

kubectl apply -f selfsigned-issuer.yaml

Step 2: Create the root CA certificate

Use the self-signed issuer to create a CA certificate. The isCA: true field tells cert-manager this certificate can sign other certificates:

# internal-ca.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager    # Store in cert-manager namespace
spec:
  isCA: true
  commonName: internal-ca
  secretName: internal-ca-secret
  duration: 87600h           # 10 years — this is a root CA
  renewBefore: 720h
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

kubectl apply -f internal-ca.yaml
kubectl get certificate internal-ca -n cert-manager

NAME          READY   SECRET               AGE
internal-ca   True    internal-ca-secret   8s

Step 3: Create a CA ClusterIssuer backed by the root CA

Now create a ClusterIssuer that uses the root CA Secret you just created. This is the issuer that will sign certificates for your internal services:

# internal-ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-secret   # References the Secret in cert-manager namespace

kubectl apply -f internal-ca-issuer.yaml
kubectl get clusterissuer internal-ca

NAME          READY   AGE
internal-ca   True    5s

Step 4: Issue a certificate for an internal service

Now issue a certificate for an internal gRPC service. The dnsNames use Kubernetes internal DNS names — <service>.<namespace>.svc.cluster.local:

# payments-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: payments-tls
  namespace: production
spec:
  secretName: payments-tls-secret
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  commonName: payments.production.svc.cluster.local
  dnsNames:
    - payments.production.svc.cluster.local
    - payments.production.svc
    - payments
  duration: 2160h     # 90 days
  renewBefore: 360h   # Renew 15 days before expiry

kubectl create namespace production
kubectl apply -f payments-cert.yaml
kubectl get certificate payments-tls -n production

NAME           READY   SECRET                AGE
payments-tls   True    payments-tls-secret   6s

The Secret payments-tls-secret now contains tls.crt, tls.key, and ca.crt. Mount this into your application pod:

# In your Deployment spec
volumes:
  - name: tls
    secret:
      secretName: payments-tls-secret
containers:
  - name: payments
    volumeMounts:
      - name: tls
        mountPath: /etc/tls
        readOnly: true

Your application reads /etc/tls/tls.crt and /etc/tls/tls.key to configure TLS. Other services that need to trust it read /etc/tls/ca.crt.

Step 5: Distribute the CA bundle with trust-manager

The problem with a custom CA is that every service needs to know about it. cert-manager's companion tool, trust-manager, handles this by distributing the CA bundle as a ConfigMap to every namespace:

helm upgrade trust-manager oci://quay.io/jetstack/charts/trust-manager \
  --install \
  --namespace cert-manager \
  --wait

Create a Bundle resource that takes the CA certificate from the internal-ca-secret and distributes it cluster-wide:

# ca-bundle.yaml
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: internal-ca-bundle
spec:
  sources:
    - secret:
        name: internal-ca-secret
        key: ca.crt
  target:
    configMap:
      key: ca-bundle.crt
    namespaceSelector:
      matchLabels:
        # Distribute to all namespaces with this label
        kubernetes.io/metadata.name: production

kubectl apply -f ca-bundle.yaml

After a few seconds, every matching namespace has a ConfigMap named internal-ca-bundle containing the CA certificate. Applications mount this ConfigMap to trust internally-issued certificates without any per-service configuration.

Step 6: Verify the certificate chain

# Extract the CA cert and service cert
kubectl get secret payments-tls-secret -n production \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

kubectl get secret payments-tls-secret -n production \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > payments.crt

# Verify the cert was signed by the CA
openssl verify -CAfile ca.crt payments.crt

payments.crt: OK

How Certificate Rotation Works

Certificate rotation is the part of certificate management that breaks production clusters most often. cert-manager handles it automatically, but understanding the mechanism helps you tune it and debug it when things go wrong.

cert-manager watches every Certificate resource it manages and checks the expiry of the underlying certificate in the Secret. When the remaining validity drops below the renewBefore threshold, cert-manager triggers a renewal. The default renewBefore is 1/3 of the certificate's total validity period — so a 90-day certificate starts renewing at day 60.

The renewal creates a new CertificateRequest, goes through the full issuance flow, and updates the Secret in place. The new certificate replaces the old one atomically. Applications that use file mounts and watch for changes (most modern web servers and gRPC frameworks do) will pick up the new certificate without restarting.

# See the current rotation status
kubectl describe certificate echo-tls -n default

Look for these fields in the output:

Status:
  Not After:   2024-06-18T10:00:00Z
  Not Before:  2024-03-20T10:00:00Z
  Renewal Time: 2024-05-18T10:00:00Z   # When cert-manager will start renewing
  Conditions:
    Type:    Ready
    Status:  True
    Message: Certificate is up to date and has not expired

If a renewal fails — for example, because the HTTP-01 challenge can't be completed — cert-manager retries with exponential backoff. The existing certificate continues to serve until it actually expires, giving you a window to debug the issue.

To see renewal events in real time:

kubectl get events -n default --field-selector reason=Issued
kubectl get events -n default --field-selector reason=Failed

Setting renewBefore correctly: For public-facing services, 30 days before a 90-day certificate is a sensible buffer. For internal short-lived certificates (24-hour validity), set renewBefore to 8 hours so rotation happens well before expiry even if the first attempt fails. Never set renewBefore to more than half the certificate's validity — cert-manager will immediately try to renew a certificate it just issued.

Cleanup

# Remove demo resources
kubectl delete ingress echo -n default
kubectl delete service echo -n default
kubectl delete deployment echo -n default
kubectl delete secret echo-tls -n default
kubectl delete certificate payments-tls -n production
kubectl delete namespace production

# Uninstall cert-manager and trust-manager
helm uninstall trust-manager -n cert-manager
helm uninstall cert-manager -n cert-manager
kubectl delete namespace cert-manager

# Remove ClusterIssuers
kubectl delete clusterissuer letsencrypt-staging letsencrypt-prod \
  internal-ca selfsigned 2>/dev/null

Conclusion

Kubernetes leaves TLS configuration entirely to you. In this article you worked through both the public and internal sides of that responsibility.

On the public side, you installed cert-manager using the current OCI Helm chart, created a ClusterIssuer backed by Let's Encrypt, and watched cert-manager go through the full ACME HTTP-01 challenge flow — from creating a temporary solver pod to storing a valid certificate in a Kubernetes Secret. You saw how switching from staging to production is a one-line annotation change, and how cert-manager renews certificates automatically before they expire.

On the internal side, you bootstrapped a private CA using cert-manager's self-signed issuer, created a ClusterIssuer backed by that CA, and issued certificates for internal service names that only exist inside the cluster. You used trust-manager to distribute the CA bundle cluster-wide so services can trust each other's certificates without per-service configuration. And you saw how to verify the certificate chain with openssl so you can confirm it's working before deploying to production.

Understanding certificate rotation is what separates teams that manage TLS confidently from teams that get woken up at 3am by an expired certificate. cert-manager automates the renewal, but the renewBefore field is your safety margin — set it correctly and know how to read the renewal status.

All YAML manifests and Helm values from this article are available in the DevOps-Cloud-Projects GitHub repository.

I did. I spun up an EC2 instance, followed along for an hour, and then forgot to shut it down. A week later I had a $34 bill for a machine running nothing.

That was the last time I practiced on someone else's infrastructure.

Everything in this guide runs on your laptop. No cloud account, no credit card, no bill at the end of the month. By the end, you'll be able to spin up a multi-server environment from scratch, configure it automatically with Ansible, serve a site you wrote yourself, and diagnose what breaks when you intentionally destroy it.

That last part is where the actual learning happens.

Prerequisites

Before you start, make sure you have:

• A laptop with at least 8GB of RAM (16GB is better)

• At least 20GB of free disk space

• Windows, macOS, or Linux operating system

• Administrator access to your computer

• Virtualization enabled in your BIOS/UEFI settings

• A stable internet connection for the initial downloads

Knowledge and comfort level:

• You should be comfortable using a terminal (running commands, changing directories, and editing small text files with whatever editor you like).

• Basic familiarity with concepts like “a server,” “SSH,” and “a port” helps, but you don't need prior experience with Docker, Kubernetes, Vagrant, or Ansible. This guide introduces them as you go.

If you can follow step-by-step instructions and read error output without panicking, you're ready.

Table of Contents

1. What is DevOps?

2. Why Build a Local Lab?

3. How to Set Up Docker

4. How to Set Up Kubernetes

5. How to Install kubectl

6. How to Set Up Vagrant

7. How to Install Ansible

8. How to Build Your First DevOps Project

9. How to Break Your Lab on Purpose

10. What You Can Now Do

What is DevOps?

DevOps is the practice of breaking down the wall between software development and IT operations teams.

Traditionally, developers write code and hand it off to operations teams to deploy and maintain. That handoff causes delays, misunderstandings, and outages. DevOps is what happens when both teams work together from the start.

The tools you'll install in this guide each solve a specific part of that process:

• Docker packages your application and everything it needs into a portable container that runs the same way on any machine.

• Kubernetes manages multiple containers at scale, handling restarts, networking, and load balancing automatically.

• Vagrant creates and manages virtual machine environments so your whole team always works on identical setups.

• Ansible automates repetitive configuration tasks across many servers without writing a script for each one.

Why Build a Local Lab?

A local lab gives you a safe place to break things, fix them, and learn from that process without any cost or risk.

Here's what you get with a local setup:

• Zero cost. No cloud bills, no surprise charges, and no credit card required.

• Works offline. Practice anywhere, even without internet after the initial setup.

• Full control. You manage every layer from the OS up to the application.

• Safe experimentation. Break things freely. Nothing here affects production.

• Fast feedback. No waiting for cloud resources to spin up. Everything runs on your machine.

The tradeoff is resource limits. Your laptop's CPU and RAM are the ceiling. You can't simulate large-scale deployments, and some cloud-native services like AWS Lambda or S3 have no direct local equivalent. But for learning core DevOps workflows, none of that matters.

How to Set Up Docker

Docker is the foundation of this lab. Every other tool in this guide either runs inside Docker containers or works alongside them.

How to Install Docker on Windows

First, enable virtualization in your BIOS:

1. Restart your computer and enter BIOS/UEFI setup. The key is usually F2, F10, Del, or Esc during boot.

2. Find the virtualization setting. It's usually listed as Intel VT-x, AMD-V, SVM, or Virtualization Technology.

3. Enable it, save your changes, and exit.

Then install Docker Desktop:

1. Download Docker Desktop from Docker's official website.

2. Run the installer and follow the prompts.

3. Enable WSL 2 (Windows Subsystem for Linux) when asked.

4. Restart your computer.

5. Open Docker Desktop from the Start menu and wait for the whale icon in the taskbar to stop animating.

Troubleshooting: If Docker fails to start, run this in PowerShell as Administrator to verify virtualization is active:

systeminfo | findstr "Hyper-V Requirements"

All items should show "Yes". If they don't, revisit your BIOS settings.

How to Install Docker on Mac

1. Download Docker Desktop for Mac from Docker's website.

2. Open the downloaded .dmg file and drag Docker to your Applications folder.

3. Open Docker from Applications.

4. Enter your password when prompted.

5. Wait for the whale icon in the menu bar to stop animating.

How to Install Docker on Linux

Run these commands in order:

# Update your package lists
sudo apt-get update

# Install prerequisites
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Add the Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Update and install Docker
sudo apt-get update
sudo apt-get install docker-ce

# Start and enable Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add your user to the docker group
sudo usermod -aG docker $USER

Log out and back in for the group change to take effect.

How to Test Docker

Run this command:

docker run hello-world

If you see "Hello from Docker!" then Docker is working correctly.

Docker is set up. Next, you'll install Kubernetes to manage containers at scale.

How to Set Up Kubernetes

Kubernetes manages containers at scale. For a local lab, you have four options. Here's how to choose:

If you're just starting out, use Minikube. It has the simplest setup and a visual dashboard that helps you understand what's happening inside the cluster.

If your laptop has 8GB RAM or less, use k3s. It runs lean and behaves closer to a real cluster than Minikube does.

Use kubeadm only if you want to understand how Kubernetes nodes join a cluster — it requires more manual steps and isn't beginner-friendly.

How to Install Minikube (Recommended for Beginners)

Minikube creates a single-node Kubernetes cluster on your laptop.

On Windows:

1. Download the Minikube installer from Minikube's GitHub releases page.

2. Run the .exe installer.

3. Open Command Prompt as Administrator and start Minikube:

minikube start --driver=docker

On Mac:

brew install minikube
minikube start --driver=docker

On Linux:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube-linux-amd64
sudo mv minikube-linux-amd64 /usr/local/bin/minikube
minikube start --driver=docker

Test your cluster:

minikube status
minikube dashboard

How to Install k3s (Recommended for Low-RAM Machines)

k3s is a lightweight version of Kubernetes that installs in under a minute. It runs lean and behaves like a real cluster — not a simplified demo version.

On Linux (and Mac via Multipass):

curl -sfL https://get.k3s.io | sh -

That single command installs k3s and runs it automatically in the background. Check that it is running:

sudo k3s kubectl get nodes

You should see one node with status Ready.

On Mac directly — k3s doesn't run natively on macOS. Use Multipass to spin up a lightweight Ubuntu VM first, then run the install command inside it.

On Windows — use WSL2 (Ubuntu), then run the install command inside your WSL2 terminal.

How to Install Kind (Kubernetes IN Docker)

Kind runs a full Kubernetes cluster inside Docker containers. It starts faster than Minikube and is useful if you want to run multiple clusters simultaneously.

# Mac or Linux
brew install kind

# Windows
choco install kind

Create a cluster:

kind create cluster --name my-local-lab

How to Install kubeadm (For Understanding Cluster Bootstrap)

kubeadm is the tool Kubernetes uses to initialize and join nodes in a real cluster. Use this when you want to understand what happens under the hood — not as your daily driver.

It requires at least two machines (or VMs). The setup is more involved than the options above. Follow the official kubeadm installation guide for your OS, then initialize your cluster:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

After init, join worker nodes using the command kubeadm prints at the end of the output.

How to Install kubectl

kubectl is the command-line tool you use to interact with any Kubernetes cluster.

On Windows:

Download kubectl.exe from Kubernetes' website and place it in a directory that is in your PATH. Or install with Chocolatey:

choco install kubernetes-cli

On Mac:

brew install kubectl

On Linux:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl

Test it:

kubectl get pods --all-namespaces

On a fresh cluster, you'll see system pods running in the kube-system namespace — things like coredns and storage-provisioner. That's the expected output. It means your cluster is up and kubectl can talk to it.

Kubernetes is running. Next is Vagrant. But before that, there's one important distinction worth making.

Docker vs Vagrant — they aren't the same thing

Docker creates containers: lightweight processes that share your operating system's kernel. Vagrant creates full virtual machines: isolated computers with their own OS running inside your laptop.

Containers are fast and small. VMs are heavier but behave exactly like real servers. You'll use both in this lab for different reasons.

How to Set Up Vagrant

Vagrant lets you create and manage reproducible virtual machine environments. It is ideal for simulating multi-server setups on a single laptop.

How to Install Vagrant on Windows

1. Download and install VirtualBox with default options.

2. Download and install Vagrant.

3. Restart your computer if prompted.

Note: VirtualBox and Hyper-V can't run at the same time on Windows. Check if Hyper-V is active:

systeminfo | findstr "Hyper-V"

If it's enabled, you have two options: switch to the Hyper-V Vagrant provider, or disable Hyper-V with:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All

Restart after disabling.

How to Install Vagrant on Mac and Linux

On Mac:

1. Download and install VirtualBox.

2. After installation, open System Preferences > Security & Privacy > General. You will see a message saying system software from Oracle was blocked. Click Allow and restart your Mac. Without this step, VirtualBox will not run.

3. Download and install Vagrant.

Note for Apple Silicon (M1/M2/M3) Macs: VirtualBox support on Apple Silicon is still limited. If you're on an M-series Mac, use UTM as your VM provider instead, or use Multipass which works natively on Apple Silicon.

On Linux:

1. Download and install VirtualBox.

2. Download and install Vagrant.

Verify both are installed:

vboxmanage --version
vagrant --version

How to Create Your First Vagrant Environment

Create a new directory for your project. Inside it, create a file named Vagrantfile with this content:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  # Create a private network between VMs
  config.vm.network "private_network", type: "dhcp"

  # Forward port 8080 on your laptop to port 80 on the VM
  config.vm.network "forwarded_port", guest: 80, host: 8080

  # Install Nginx when the VM starts
  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y nginx
    echo "Hello from Vagrant!" > /var/www/html/index.html
  SHELL
end

Start the VM:

vagrant up

Visit http://localhost:8080 in your browser. You should see "Hello from Vagrant!"

Troubleshooting SSH on Windows

If vagrant ssh fails, try:

vagrant ssh -- -v

Or connect manually:

ssh -i .vagrant/machines/default/virtualbox/private_key vagrant@127.0.0.1 -p 2222

How to Create a Local Vagrant Box Without Internet

Note: Most readers can skip this. Only do this if you want to work fully offline after the initial setup.

1. Download Ubuntu 20.04 LTS and save the .iso file locally.

2. Open VirtualBox and create a new VM: Name it ubuntu-devops, Type: Linux, Version: Ubuntu (64-bit).

3. Assign 2048MB RAM and a 20GB VDI disk.

4. Attach the .iso under Storage > Optical Drive.

5. Start the VM and complete the Ubuntu installation.

6. Once installed, shut down the VM and run:

VBoxManage list vms
vagrant package --base "ubuntu-devops" --output ubuntu2004.box
vagrant box add ubuntu2004 ubuntu2004.box

You now have a reusable local box that works without internet.

You can spin up virtual machines. Next is Ansible, which automates what goes inside them.

How to Install Ansible

Ansible automates configuration and software installation across multiple servers. Instead of SSH-ing into ten machines and running the same commands manually, you write a playbook once and Ansible handles the rest.

How to Install Ansible on Windows

Ansible doesn't run natively on Windows. You need to use it through WSL (Windows Subsystem for Linux).

1. Open PowerShell as Administrator and enable WSL:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

1. Restart your computer.

2. Install Ubuntu from the Microsoft Store.

3. Open Ubuntu and install Ansible:

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

How to Install Ansible on Mac

brew install ansible

How to Install Ansible on Linux

# Ubuntu/Debian
sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

# Red Hat/CentOS
sudo yum install ansible

How to Test Ansible

Create a file called hosts in your current directory:

[local]
localhost ansible_connection=local

Create a file called playbook.yml in the same directory:

---
- name: Test playbook
  hosts: local
  tasks:
    - name: Print a message
      debug:
        msg: "Ansible is working!"

Run the playbook, passing the local hosts file with -i:

ansible-playbook -i hosts playbook.yml

You should see the message "Ansible is working!" in the output.

Alright, all your tools are installed. Now you'll use them together to build something real.

How to Build Your First DevOps Project

You can find the entire code for this lab in this repo: https://github.com/Osomudeya/homelab-demo-article

Now you'll put these tools together in one project. Each tool will perform its actual job, and nothing is forced.

Before you start, create a fresh directory for this project. Don't run it inside the directory you used to test Vagrant earlier, as the Vagrantfile here is different and will conflict.

You'll be building a two-VM environment: one machine serves a web page you write yourself inside a Docker container, and the other runs a MariaDB database. Vagrant creates the machines and Ansible configures them. The page you see at the end is yours.

Step 1: Create the Project Directory

mkdir devops-lab-project && cd devops-lab-project

Step 2: Write Your Site Content

Create a file called index.html in the project directory. Write whatever you want on this page — it's what you'll see in your browser at the end:

<!DOCTYPE html>
<html>
  <head><title>My DevOps Lab</title></head>
  <body>
    <h1>My DevOps Lab</h1>
    <p>Provisioned by Vagrant. Configured by Ansible. Served by Docker.</p>
    <p>Built on a laptop. No cloud account needed.</p>
  </body>
</html>

Change the text to whatever you like. This is your page.

Step 3: Write the Vagrantfile

Create a file called Vagrantfile in the same directory:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  config.vm.define "web" do |web|
    web.vm.network "private_network", ip: "192.168.33.10"
    web.vm.network "forwarded_port", guest: 80, host: 8080
  end

  config.vm.define "db" do |db|
    db.vm.network "private_network", ip: "192.168.33.11"
  end
end

Step 4: Start the Virtual Machines

vagrant up

The first run downloads the ubuntu/focal64 box, which is around 500MB.

Expect this to take 10–30 minutes depending on your connection. Subsequent runs will be much faster since the box is cached locally.

Step 5: Create the Ansible Inventory

Create a file called inventory in the same directory:

[webservers]
192.168.33.10 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

[dbservers]
192.168.33.11 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/db/virtualbox/private_key

Ansible uses the Vagrant-generated private keys so it can SSH in as the vagrant user. Host key checking for this lab is turned off in ansible.cfg (next step), not in the inventory.

Step 6: Create the Ansible Config File

Before running the playbook, create a file called ansible.cfg in the same directory:

[defaults]
inventory = inventory
host_key_checking = False

The inventory line tells Ansible to use the inventory file in this folder by default. host_key_checking = False tells Ansible not to verify SSH host keys when connecting to your Vagrant VMs. Without it, Ansible will fail with a Host key verification failed error on first connection because the VM's key is not yet in your known_hosts file.

These settings are for a local lab only. Do not use host_key_checking = False for production systems.

Step 7: Create the Ansible Playbook

Create a file called playbook.yml:

---
- name: Configure web server
  hosts: webservers
  become: yes
  tasks:

    - name: Install Docker
      apt:
        name: docker.io
        state: present
        update_cache: yes

    - name: Start Docker service
      service:
        name: docker
        state: started
        enabled: yes

    # Create the directory that will hold your site content
    - name: Create web content directory
      file:
        path: /var/www/html
        state: directory
        mode: '0755'

    # This copies your index.html from your laptop into the VM
    - name: Copy site content to web server
      copy:
        src: index.html
        dest: /var/www/html/index.html

    # This mounts that file into the Nginx container so it serves your page
    # The -v flag connects /var/www/html on the VM to /usr/share/nginx/html inside the container
    - name: Run Nginx serving your content
      shell: |
        docker rm -f webapp 2>/dev/null || true
        docker run -d --name webapp --restart always -p 80:80 \
          -v /var/www/html:/usr/share/nginx/html:ro nginx

- name: Configure database server
  hosts: dbservers
  become: yes
  tasks:

    # Hash sum mismatch on .deb downloads is often stale lists, a flaky mirror, or apt pipelining
    # behind NAT; fresh indices + Pipeline-Depth 0 usually fixes it on lab VMs.
    - name: Disable apt HTTP pipelining (mirror/proxy hash mismatch workaround)
      copy:
        dest: /etc/apt/apt.conf.d/99disable-pipelining
        content: 'Acquire::http::Pipeline-Depth "0";'
        mode: "0644"

    - name: Clear apt package index cache
      shell: apt-get clean && rm -rf /var/lib/apt/lists/* /var/lib/apt/lists/auxfiles/*
      changed_when: true

    - name: Update apt cache after reset
      apt:
        update_cache: yes

    - name: Install MariaDB
      apt:
        name: mariadb-server
        state: present
        update_cache: no

    - name: Start MariaDB service
      service:
        name: mariadb
        state: started
        enabled: yes

Two lines worth paying attention to:

• src: index.html — Ansible looks for this file in the same directory as the playbook. That is the file you wrote in Step 2.

• -v /var/www/html:/usr/share/nginx/html:ro — this mounts the directory from the VM into the Nginx container. The :ro means read-only. Nginx serves whatever is in that folder.

Step 8: Run the Playbook

ansible-playbook -i inventory playbook.yml

You'll see task-by-task output as Ansible connects to each VM over SSH and configures it. A green ok or yellow changed next to each task means it worked. Red fatal means something failed.

Step 9: Verify the Setup

Open http://localhost:8080 in your browser. You should see the page you wrote in Step 2 served from inside a Docker container, running on a Vagrant VM, configured automatically by Ansible.

If you see the page, every tool in this lab is working together.

Step 9: Clean Up (Optional)

When you're done:

vagrant destroy -f

This shuts down and deletes both VMs. Your Vagrantfile, inventory, playbook.yml, and index.html stay on disk — run vagrant up followed by ansible-playbook -i inventory playbook.yml any time to bring it all back.

Now that you have a working lab, let's use it properly.

How to Break Your Lab on Purpose

Following these steps has gotten you a running lab. Breaking things teaches you how everything actually works.

Here are five things to break and what to look for when you do.

Break 1: Crash the Main Process Inside the Container (and Watch It Come Back)

Doing this just proves that something inside the container can die (like a real bug or OOM), Docker can restart the container because of --restart always, and your site can come back without re-running Ansible.

After vagrant ssh web, every docker command below runs on the web VM. So keep your browser on your laptop at http://localhost:8080 (Vagrant forwards your host port to the VM’s port 80).

Troubleshooting: If Your Lab Isn't Ready

From your project folder on the host (your laptop) – unless the step says to run it on the VM:

• You ran vagrant destroy -f. Run vagrant up, then ansible-playbook -i inventory playbook.yml.

• docker ps shows webapp but status is Exited. On the web VM, run sudo docker start webapp, then sudo docker ps again.

• There's no webapp row in docker ps -a. Re-run ansible-playbook -i inventory playbook.yml on the host.

If the playbook is already applied and webapp is Up, skip this section and start at step 1 under Steps (happy path) below. (Don't skip SSH or docker ps. You need the VM shell and a quick check before you run docker exec.)

Steps (happy path)

1. SSH into the web VM:

vagrant ssh web

1. Confirm webapp is Up:

   sudo docker ps
   

2. Break it on purpose: kill the container’s main process from inside (PID 1). That ends the container the same way a crashing app would, not the same as docker stop on the host:

sudo docker exec webapp sh -c 'sleep 5 && kill 1'

The sleep 5 gives you a moment to switch to the browser. Right after you run the command, open or refresh http://localhost:8080. You may catch a brief error or blank page while nothing is listening on port 80.

1. Watch Docker restart the container:

watch sudo docker ps -a

Within a few seconds you should see Exited (137) become Up again. (Press Ctrl+C to exit watch.)

5. Refresh the browser. You should see the same HTML as before, because the files live on the VM under /var/www/html and are bind-mounted into the container; restarting only replaced the Nginx process, not those files.

Why not docker stop or docker kill on the host for this demo?

Those commands go through Docker’s API. On many setups (including recent Docker), Docker treats them as you choosing to stop the container (hasBeenManuallyStopped), and --restart always may not bring the container back until you docker start it or similar.

Killing PID 1 from inside the container is treated more like an internal crash, so the restart policy you set in the playbook is the one you actually get to observe here.

Kubernetes analogy: A pod whose containers exit can be restarted by the kubelet; a pod you delete does not come back by itself.

What to observe (three separate checks):

1. Exit code: After kill 1, docker ps -a should show the container exited with code 137, meaning the main process was killed by a signal. That confirms the container really died, not that you ran docker stop on the host.

2. Restart delay vs browser: Watch how many seconds pass between Exited and Up in docker ps -a; that interval is Docker applying --restart always. That's separate from what you see in the browser: the browser only shows whether something is accepting connections on port 80 on the VM, so it may show an error or blank page during the gap even while Docker is about to restart the container.

3. Content after recovery: After status is Up again, refresh the page. You should see the same HTML as before. That shows your content lives on the VM disk (mounted into the container with -v), not inside a file that vanishes when the container process restarts. The process was replaced, not your index.html on the host path.

Break 2: Cause a Container Name Conflict

On a single Docker daemon (here, on your web VM), a container name is a unique label. Two running (or stopped) containers can't share the same name. Scripts and playbooks that always use docker run --name webapp without cleaning up first hit this error constantly and recognizing it saves time in real work.

Before you start: Ansible already created one container named webapp.
Stay on the web VM (for example still inside vagrant ssh web) so the commands below run where that container lives.

So now, try to start a second container and also call it webapp. The image is plain nginx here on purpose – the point is the name clash, not matching your site’s ports or volume mounts.

sudo docker run -d --name webapp nginx

What actually happens here is that Docker doesn't create a second container. It returns an error immediately. Your original webapp is unchanged.

This is because the name webapp is already registered to the existing container (the error shows that container’s ID). Docker refuses to reuse the name until the old container is removed or renamed.

Example error (your ID will differ):

docker: Error response from daemon: Conflict. The container name "/webapp" is already in use by container "2e48b81a311c4b71cdc1e25e0df75a22296845c7eb53aab82f9ae739fb6410ec". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

To fix it, free the name, then create webapp again the same way the playbook does (publish port 80, mount your HTML, restart policy):

sudo docker rm -f webapp
sudo docker run -d --name webapp --restart always -p 80:80 \
  -v /var/www/html:/usr/share/nginx/html:ro nginx

After that, your site should behave as before (refresh http://localhost:8080 from your laptop).

What to observe:

Read Docker’s Conflict message end to end. You should see that the name /webapp is already in use and a container ID pointing at the existing box. In production, that pattern means “something already claimed this name. Just remove it, rename it, or pick a different name before you run docker run again.”

Break 3: Make Ansible Fail to Reach a VM

Ansible separates “could not connect” from “connected, but a task broke.” The first is UNREACHABLE, the second is FAILED. Knowing which one you have tells you whether to fix network / SSH or playbook / packages / permissions.

On your laptop, in the project folder, edit inventory and change the web server address from 192.168.33.10 to an IP no VM uses, for example 192.168.33.99. Save the file.

[webservers]
192.168.33.99 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

What you run (from the same project folder on the host):

ansible-playbook -i inventory playbook.yml

After this, Ansible tries to SSH to 192.168.33.99. Nothing on your lab network answers as that host (or SSH never succeeds), so Ansible never runs tasks on the web server. It stops that host with UNREACHABLE:

fatal: [192.168.33.99]: UNREACHABLE! => {"msg": "Failed to connect to the host via ssh"}

This is realistic because the same message shape appears when the IP is wrong, the VM isn't running, a firewall blocks port 22, or the network is misconfigured. The common thread is no working SSH session.

Now it's time to put it back: restore 192.168.33.10 in inventory and run ansible-playbook -i inventory playbook.yml again. The web play should reach the VM and complete (assuming your lab is up).

UNREACHABLE vs FAILED – what to observe:

• If Ansible prints UNREACHABLE, you should assume it never opened SSH on that host and never ran tasks there. Go ahead and fix the connection (IP, VM up, firewall, key path) before you debug playbook logic.

• If Ansible prints FAILED, you should assume SSH worked and a task returned an error. Read the task output for the real cause (package name, permissions, syntax), not the network first.

When you debug later, you should look at the keyword Ansible prints: UNREACHABLE points to reachability while FAILED points to task output and the first failed task under that host.

Break 4: Fill the VM's Disk

Databases and other services need free disk for logs, temp files, and data. When the filesystem is full or nearly full, a service may fail to start or fail at runtime. This break walks through the same diagnosis habit you would use on a real server: check space, then read systemd and journal output for the service.

All commands below run on the db VM after vagrant ssh db. MariaDB was installed there by your playbook.

What you do:

1. Open a shell on the db VM:

   vagrant ssh db
   

2. Allocate a large file full of zeros (here 1GB) to simulate something eating disk space:

   sudo dd if=/dev/zero of=/tmp/bigfile bs=1M count=1024
   
   df -h
   

   Use df -h to see how full the root filesystem (or relevant mount) is. Your Vagrant disk may be large enough that 1GB only raises usage. If MariaDB still starts, you still practiced the checks. To see a stronger effect, you can repeat with a larger count= only in a lab (never fill production disks on purpose without a plan).

3. Ask systemd to restart MariaDB and show status:

   sudo systemctl restart mariadb
   sudo systemctl status mariadb
   

   If the disk is critically full, restart may fail or the service may show failed or not running.

4. If something looks wrong, read recent logs for the MariaDB unit:

   sudo journalctl -u mariadb --no-pager | tail -20
   

   Errors often mention disk, space, read-only filesystem, or InnoDB being unable to write.

5. Clean up so your VM stays usable:

   sudo rm /tmp/bigfile
   

   Optionally run sudo systemctl restart mariadb again and confirm it is active (running).

What to observe:

• You should use df -h first to confirm whether the filesystem is actually tight. That avoids blaming the database when disk space is fine.

• You should read systemctl status mariadb to see whether systemd thinks the service is active, failed, or flapping.

• You should read journalctl -u mariadb when status is bad, so you can tie the failure to concrete errors from MariaDB or the kernel (often mentioning disk, space, or read-only filesystem). Space + status + logs is the same order you would use on a production server.

Break 5: Run Minikube Out of Resources

Kubernetes schedules pods onto nodes that have enough CPU and memory. If you ask for more than the cluster can place, some pods stay Pending and Events explain why (for example Insufficient cpu). That is not the same as a pod that starts and then crashes.

To do this, you'll need a local cluster (we're using Minikube in this guide) and kubectl on your laptop. This break doesn't use the Vagrant VMs. If you haven't installed Minikube yet, complete the "How to Set Up Kubernetes" section first, or skip this break until you do.

You'll run this on your Mac, Linux, or Windows terminal (host), not inside vagrant ssh. If you're still inside a VM, type exit until your prompt is back on the host.

What you do:

1. Check Minikube:

   minikube status
   

   If it's stopped, start it (Docker driver matches earlier sections):

   minikube start --driver=docker
   

2. Create a deployment with many replicas so your single Minikube node can't run them all at once:

   kubectl create deployment stress --image=nginx --replicas=20
   
   #watch pods start
   kubectl get pods -w
   

   Press Ctrl+C when you're done watching. Some pods may stay Pending while others are Running.

3. Pick one Pending pod name from kubectl get pods and inspect it:

   kubectl describe pod <pod-name>
   

   Under Events, look for FailedScheduling and a line similar to:

   Warning  FailedScheduling  0/1 nodes are available: 1 Insufficient cpu.
   

   You might see Insufficient memory instead, depending on your machine.

4. Fix the lab by scaling back so the cluster can catch up:

   kubectl scale deployment stress --replicas=2
   

   You can delete the deployment entirely when finished: kubectl delete deployment stress.

What to observe:

• You should see Pending pods stay unscheduled until capacity frees up. That means the scheduler hasn't placed them on any node yet, usually because the node is out of CPU or memory for that workload.

• You should read kubectl describe pod <pod-name> and scroll to Events. Messages like Insufficient cpu or Insufficient memory mean the cluster ran out of schedulable capacity, not that the container image image is corrupt.

• You should contrast that with a pod that reaches Running and then CrashLoopBackOff, which usually means the process inside the container keeps exiting. that is an application or config problem, not a “nowhere to run” problem.

What You Can Now Do

You didn't just install tools in this tutorial. You also used them.

You can now spin up two servers from a single file. You can write a playbook that installs software and deploys a container without touching either machine manually.

You can serve a page you wrote from inside a Docker container running on a Vagrant VM, and bring the whole thing back from scratch in one command.

You also broke it. You saw what a container conflict looks like, what Ansible prints when it can't reach a machine, what disk pressure does to a running service, and what a Kubernetes scheduler says when it runs out of resources. Those error messages aren't unfamiliar anymore.

That's the difference between someone who has read about DevOps and someone who has run it.

Here are four free projects you can run in this same lab to go further:

• DevOps Home-Lab 2026 — Build a multi-service app (frontend, API, PostgreSQL, Redis) end-to-end with Docker Compose, Kubernetes, Prometheus/Grafana monitoring, GitOps with ArgoCD, and Cloudflare for global exposure.

• KubeLab — Trigger real Kubernetes failure scenarios, pod crashes, OOMKills, node drains, cascading failures, and watch how the cluster responds using live metrics.

• K8s Secrets Lab — Build a full secret management pipeline from AWS Secrets Manager into your cluster, including rotation behavior and IRSA.

• DevOps Troubleshooting Toolkit — Structured debugging guides across Linux, containers, Kubernetes, cloud, databases, and observability with copy-paste commands for real incidents.

All free and open source: github.com/Osomudeya/List-Of-DevOps-Projects.

If you want to go deeper, you can find six full chapters covering Terraform, Ansible, monitoring, CI/CD, and a simulated three-VM production environment at Build Your Own DevOps Lab.

Google Cloud Axion is Google's own custom ARM-based chip, built to handle the demands of modern cloud workloads. The performance and cost numbers are striking: Google claims Axion delivers up to 60% better energy efficiency and up to 65% better price-performance compared to comparable x86 machines.

AWS has Graviton. Azure has Cobalt. ARM is no longer niche. It's the direction the entire cloud industry is moving.

But there's a problem that catches almost every team off guard when they start this transition: container architecture mismatch.

If you build a Docker image on your M-series Mac and push it to an x86 server, it crashes on startup with a cryptic exec format error.

The server isn't broken. It just can't read the compiled instructions inside your image. An ARM binary and an x86 binary are written in fundamentally different languages at the machine level. The CPU literally can't execute instructions it wasn't designed for.

We're going to solve this problem completely in this tutorial. You'll build a single Docker image tag that automatically serves the correct binary on both ARM and x86 machines — no separate pipelines, no separate tags. Then you'll provision Google Cloud ARM nodes in GKE and configure your Kubernetes deployment to route workloads precisely to those cost-efficient nodes.

Here's what you'll build, step by step:

• A Go HTTP server that reports the CPU architecture it's running on at runtime

• A multi-stage Dockerfile that cross-compiles for both linux/amd64 and linux/arm64 without slow QEMU emulation

• A multi-arch image in Google Artifact Registry that acts as a single entry point for any architecture

• A GKE cluster with two node pools: a standard x86 pool and an ARM Axion pool

• A Kubernetes Deployment that pins your workload exclusively to the ARM nodes

By the end, you'll hit a live endpoint and see the word arm64 staring back at you from a Google Cloud ARM node. Let's get into it.

Table of Contents

• Prerequisites

• Step 1: Set Up Your Google Cloud Project

• Step 2: Create the GKE Cluster

• Step 3: Write the Application

• Step 4: Enable Multi-Arch Builds with Docker Buildx

• Step 5: Write the Dockerfile

• Step 6: Build and Push the Multi-Arch Image

• Step 7: Add the Axion ARM Node Pool

• Step 8: Deploy the App to the ARM Node Pool

• Step 9: Verify the Deployment

• Step 10: Cost Savings and Tradeoffs

• Cleanup

• Conclusion

• Project File Structure

Prerequisites

Before you start, make sure you have the following ready:

• A Google Cloud project with billing enabled. If you don't have one, create it at console.cloud.google.com. The total cost to follow this tutorial is around $5–10.

• gcloud CLI installed and authenticated. Run gcloud auth login to sign in and gcloud config set project YOUR_PROJECT_ID to point it at your project.

• Docker Desktop version 19.03 or later. Docker Buildx (the tool we'll use for multi-arch builds) ships bundled with it.

• kubectl installed. This is the CLI for interacting with Kubernetes clusters.

• Basic familiarity with Docker (images, layers, Dockerfile) and Kubernetes (pods, deployments, services). You don't need to be an expert, but you should know what these things are.

Step 1: Set Up Your Google Cloud Project

Before writing a single line of application code, let's get the cloud infrastructure side ready. This is the foundation everything else will build on.

Enable the Required APIs

Google Cloud services are off by default in any new project. Run this command to turn on the three APIs we'll need:

gcloud services enable \
  artifactregistry.googleapis.com \
  container.googleapis.com \
  containeranalysis.googleapis.com

Here's what each one does:

• artifactregistry.googleapis.com — enables Artifact Registry, where we'll store our Docker images

• container.googleapis.com — enables Google Kubernetes Engine (GKE), where our cluster will run

• containeranalysis.googleapis.com — enables vulnerability scanning for images stored in Artifact Registry

Create a Docker Repository in Artifact Registry

Artifact Registry is Google Cloud's managed container image store — the place where our built images will live before being deployed to the cluster. Create a dedicated repository for this tutorial:

gcloud artifacts repositories create multi-arch-repo \
  --repository-format=docker \
  --location=us-central1 \
  --description="Multi-arch tutorial images"

Breaking down the flags:

• --repository-format=docker — tells Artifact Registry this repository stores Docker images (as opposed to npm packages, Maven artifacts, and so on)

• --location=us-central1 — the Google Cloud region where your images will be stored. Use a region that's close to where your cluster will run to minimize image pull latency. Run gcloud artifacts locations list to see all options.

• --description — a human-readable label for the repository, shown in the console.

Authenticate Docker to Push to Artifact Registry

Docker needs credentials before it can push images to Google Cloud. Run this command to wire up authentication automatically:

gcloud auth configure-docker us-central1-docker.pkg.dev

This adds a credential helper entry to your ~/.docker/config.json file. What that means in practice: any time Docker tries to push or pull from a URL under us-central1-docker.pkg.dev, it will automatically call gcloud to get a valid auth token. You won't need to run docker login manually.

Step 2: Create the GKE Cluster

With Artifact Registry ready to receive images, let's create the Kubernetes cluster. We'll start with a standard cluster using x86 nodes and add an ARM node pool later once we have an image to deploy.

gcloud container clusters create axion-tutorial-cluster \
  --zone=us-central1-a \
  --num-nodes=2 \
  --machine-type=e2-standard-2 \
  --workload-pool=PROJECT_ID.svc.id.goog

Replace PROJECT_ID with your actual Google Cloud project ID.

What each flag does:

• --zone=us-central1-a — creates a zonal cluster in a single availability zone. A regional cluster (using --region) would spread nodes across three zones for higher resilience, but for this tutorial a single zone keeps things simple and avoids capacity issues that can affect specific zones. If us-central1-a is unavailable, try us-central1-b.

• --num-nodes=2 — two x86 nodes in this zone. We need at least 2 to have enough capacity alongside our ARM node pool later.

• --machine-type=e2-standard-2 — the machine type for this default node pool. e2-standard-2 is a cost-effective x86 machine with 2 vCPUs and 8 GB of memory, good for general workloads.

• --workload-pool=PROJECT_ID.svc.id.goog — enables Workload Identity, which is Google's recommended way for pods to authenticate with Google Cloud APIs. It avoids the need to download and store service account key files inside your cluster.

This command takes a few minutes. While it runs, you can move on to writing the application. We'll come back to the cluster in Step 6.

Step 3: Write the Application

We need an application to containerize. We'll use Go for three specific reasons:

1. Go compiles into a single, statically-linked binary. There's no runtime to install, no interpreter — just the binary. This makes for extremely lean container images.

2. Go has first-class, built-in cross-compilation support. We can compile an ARM64 binary from an x86 Mac, or vice versa, by setting two environment variables. This will matter a lot when we get to the Dockerfile.

3. Go exposes the architecture the binary was compiled for via runtime.GOARCH. Our server will report this at runtime, giving us hard proof that the correct binary is running on the correct hardware.

Start by creating the project directories:

mkdir -p hello-axion/app hello-axion/k8s
cd hello-axion/app

Initialize the Go module from inside app/. This creates go.mod in the current directory:

go mod init hello-axion

go mod init is Go's built-in command for starting a new module. It writes a go.mod file that declares the module name (hello-axion) and the minimum Go version required. Every modern Go project needs this file — without it, the compiler doesn't know how to resolve packages.

Now create the application at app/main.go:

package main

import (
    "fmt"
    "net/http"
    "os"
    "runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
    hostname, _ := os.Hostname()
    fmt.Fprintf(w, "Hello from freeCodeCamp!\n")
    fmt.Fprintf(w, "Architecture : %s\n", runtime.GOARCH)
    fmt.Fprintf(w, "OS           : %s\n", runtime.GOOS)
    fmt.Fprintf(w, "Pod hostname : %s\n", hostname)
}

func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintln(w, "ok")
}

func main() {
    http.HandleFunc("/", handler)
    http.HandleFunc("/healthz", healthz)
    fmt.Println("Server starting on port 8080...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Fprintf(os.Stderr, "server error: %v\n", err)
        os.Exit(1)
    }
}

Verify both files were created:

ls -la

You should see go.mod and main.go listed.

Let's walk through what this code does:

• import "runtime" — imports Go's built-in runtime package, which exposes information about the Go runtime environment, including the CPU architecture.

• runtime.GOARCH — returns a string like "arm64" or "amd64" representing the architecture this binary was compiled for. When we deploy to an ARM node, this value will be arm64. This is the core of our proof.

• os.Hostname() — returns the pod's hostname, which Kubernetes sets to the pod name. This lets us see which specific pod responded when we test the app later.

• handler — the main HTTP handler, registered on the root path /. It writes the architecture, OS, and hostname to the response.

• healthz — a separate handler registered on /healthz. It returns HTTP 200 with the text ok. Kubernetes will use this endpoint to check whether the container is alive and ready to serve traffic — we'll wire this up in the deployment manifest later.

• http.ListenAndServe(":8080", nil) — starts the server on port 8080. If it fails to start (for example, if the port is already in use), it prints the error and exits with a non-zero code so Kubernetes knows something went wrong.

Step 4: Enable Multi-Arch Builds with Docker Buildx

Before we write the Dockerfile, we need to understand a fundamental constraint, because it directly shapes how the Dockerfile must be written.

Why Your Docker Images Are Architecture-Specific By Default

A CPU only understands instructions written for its specific Instruction Set Architecture (ISA). ARM64 and x86_64 are different ISAs — different vocabularies of machine-level operations. When you compile a Go program, the compiler translates your source code into binary instructions for exactly one ISA. That binary can't run on a different ISA.

When you build a Docker image the normal way (docker build), the binary inside that image is compiled for your local machine's ISA. If you're on an Apple Silicon Mac, you get an ARM64 binary. Push that image to an x86 server, and when Docker tries to execute the binary, the kernel rejects it:

standard_init_linux.go:228: exec user process caused: exec format error

That's the operating system saying: "This binary was written for a different processor. I have no idea what to do with it."

The Solution: A Single Image Tag That Serves Any Architecture

Docker solves this with a structure called a Manifest List (also called a multi-arch image index). Instead of one image, a Manifest List is a pointer table. It holds multiple image references — one per architecture — all under the same tag.

When a server pulls hello-axion:v1, here's what actually happens:

1. Docker contacts the registry and requests the manifest for hello-axion:v1

2. The registry returns the Manifest List, which looks like this internally:

{
  "manifests": [
    { "digest": "sha256:a1b2...", "platform": { "architecture": "amd64", "os": "linux" } },
    { "digest": "sha256:c3d4...", "platform": { "architecture": "arm64", "os": "linux" } }
  ]
}

1. Docker checks the current machine's architecture, finds the matching entry, and pulls only that specific image layer. The x86 image never downloads onto your ARM server, and vice versa.

One tag, two actual images. Completely transparent to your deployment manifests.

Set Up Docker Buildx

Docker Buildx is the CLI tool that builds these Manifest Lists. It's powered by the BuildKit engine and ships bundled with Docker Desktop. Run the following to create and activate a new builder instance:

docker buildx create --name multiarch-builder --use

• --name multiarch-builder — gives this builder a memorable name. You can have multiple builders. This command creates a new one named multiarch-builder.

• --use — immediately sets this new builder as the active one, so all future docker buildx build commands use it.

Now boot the builder and confirm it supports the platforms we need:

docker buildx inspect --bootstrap

• --bootstrap — starts the builder container if it isn't already running, and prints its full configuration.

You should see output like this:

Name:          multiarch-builder
Driver:        docker-container
Platforms:     linux/amd64, linux/arm64, linux/arm/v7, linux/386, ...

The Platforms line lists every architecture this builder can produce images for. As long as you see linux/amd64 and linux/arm64 in that list, you're ready to build for both x86 and ARM.

Step 5: Write the Dockerfile

Now we can write the Dockerfile. We'll use two techniques together: a multi-stage build to keep the final image tiny, and a cross-compilation trick to avoid slow CPU emulation.

Create app/Dockerfile with the following content:

# -----------------------------------------------------------
# Stage 1: Build
# -----------------------------------------------------------
# $BUILDPLATFORM = the machine running this build (your laptop)
# \(TARGETOS / \)TARGETARCH = the platform we are building FOR
# -----------------------------------------------------------
FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

ARG TARGETOS
ARG TARGETARCH

WORKDIR /app

COPY go.mod .
RUN go mod download

COPY main.go .

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

# -----------------------------------------------------------
# Stage 2: Runtime
# -----------------------------------------------------------

FROM alpine:latest

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

WORKDIR /app
COPY --from=builder /app/server .

EXPOSE 8080
CMD ["./server"]

There's a lot happening here. Let's go through it carefully.

Stage 1: The Builder

FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

This is the most important line in the file. \(BUILDPLATFORM is a special build argument that Docker Buildx automatically injects — it equals the platform of the machine running the build (your laptop). By pinning the builder stage to \)BUILDPLATFORM, the Go compiler always runs natively on your machine, not inside a CPU emulator. This is what makes multi-arch builds fast.

Without --platform=$BUILDPLATFORM, Buildx would have to use QEMU — a full CPU emulator — to run an ARM64 build environment on your x86 machine (or vice versa). QEMU works, but it's typically 5–10 times slower than native execution. For a project with many dependencies, that's the difference between a 2-minute build and a 20-minute build.

ARG TARGETOS and ARG TARGETARCH

These two lines declare that our Dockerfile expects build arguments named TARGETOS and TARGETARCH. Buildx injects these automatically based on the --platform flag you pass at build time. For a linux/arm64 target, TARGETOS will be linux and TARGETARCH will be arm64.

COPY go.mod . and RUN go mod download

We copy go.mod first, before copying the rest of the source code. Docker builds images layer by layer and caches each layer. By copying only the module file first, we create a cached layer for go mod download.

On future builds, as long as go.mod hasn't changed, Docker skips the download step entirely — even if the source code changed. This speeds up iterative development significantly.

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

This is the cross-compilation step. GOOS and GOARCH are Go's built-in cross-compilation environment variables. Setting them tells the Go compiler to produce a binary for a different target than the machine it's running on. We set them from the \(TARGETOS and \)TARGETARCH build args injected by Buildx.

The -ldflags="-w -s" flag strips the debug symbol table and the DWARF debugging information from the binary. This has no effect on runtime behavior but reduces the binary size by roughly 30%.

Stage 2: The Runtime Image

FROM alpine:latest

This starts a brand-new image from Alpine Linux — a minimal Linux distribution that weighs about 5 MB. Critically, alpine:latest is itself a multi-arch image, so Docker automatically selects the arm64 or amd64 Alpine variant depending on which platform this stage is built for.

Everything from Stage 1 — the Go toolchain, the source files, the intermediate object files — is discarded. The final image contains only Alpine Linux plus our binary. Compared to a naive single-stage Go image (~300 MB), this approach produces an image under 15 MB.

RUN addgroup -S appgroup && adduser -S appuser -G appgroup and USER appuser

These two lines create a non-root user and set it as the active user for the container. Running containers as root is a security risk — if an attacker exploits a vulnerability in your application, they gain root access inside the container. Running as a non-root user limits the blast radius.

COPY --from=builder /app/server .

This is how multi-stage builds work: the --from=builder flag tells Docker to copy files from the builder stage (Stage 1), not from your local disk. Only the compiled binary (server) makes it into the final image.

Step 6: Build and Push the Multi-Arch Image

With the application and Dockerfile in place, we can now build images for both architectures and push them to Artifact Registry — all in a single command.

From inside the app/ directory, run:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 \
  --push \
  .

Replace PROJECT_ID with your actual GCP project ID.

Here's what each part of this command does:

• docker buildx build — uses the Buildx CLI instead of the standard docker build. Buildx is required for multi-platform builds.

• --platform linux/amd64,linux/arm64 — instructs Buildx to build the image twice: once targeting x86 Intel/AMD machines, and once targeting ARM64. Both builds run in parallel. Because our Dockerfile uses the $BUILDPLATFORM cross-compilation trick, both builds run natively on your machine without QEMU emulation.

• -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 — the full image path in Artifact Registry. The format is always REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME:TAG.

• --push — multi-arch images can't be loaded into your local Docker daemon (which only understands single-architecture images). This flag tells Buildx to skip local storage and push the completed Manifest List — with both architecture variants — directly to the registry.

• . — the build context, the directory Docker scans for the Dockerfile and any files the build needs.

Watch the output as the build runs. You'll see BuildKit working on both platforms simultaneously:

 => [linux/amd64 builder 1/5] FROM golang:1.23-alpine
 => [linux/arm64 builder 1/5] FROM golang:1.23-alpine
 ...
 => pushing manifest for us-central1-docker.pkg.dev/.../hello-axion:v1

Verify the Multi-Arch Image in Artifact Registry

Once the push completes, navigate to GCP Console → Artifact Registry → Repositories → multi-arch-repo and click on hello-axion.

You won't see a single image — you'll see something labelled "Image Index". That's the Manifest List we created. Click into it, and you'll find two child images with separate digests, one for linux/amd64 and one for linux/arm64.

You can also inspect this from the command line:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

The output lists every manifest inside the image index. You'll see entries for linux/amd64 and linux/arm64 — those are our two real images. You'll also see two entries with Platform: unknown/unknown labelled as attestation-manifest. These are build provenance records that Docker Buildx automatically attaches to prove how and where the image was built (a supply chain security feature called SLSA attestation).

The two entries you care about are linux/amd64 and linux/arm64. Note the digest for the arm64 entry — we'll use it in the verification step to confirm the cluster pulled the right variant.

Step 7: Add the Axion ARM Node Pool

We have a universal image. Now we need somewhere to run it.

Recall the cluster we created in Step 2 — it's running e2-standard-2 x86 machines. We're going to add a second node pool running ARM machines. This is the key architectural move: a mixed-architecture cluster where different workloads can be routed to different hardware.

Choosing Your ARM Machine Type

Google Cloud currently offers two ARM-based machine series in GKE:

This tutorial uses t2a-standard-2 because it's widely available. The commands are identical for c4a-standard-2 — just swap the --machine-type value. If t2a-standard-2 isn't available in your zone, GKE will tell you immediately when you run the node pool creation command below, and you can try a neighbouring zone.

Create the ARM Node Pool

Add the ARM node pool to your existing cluster:

gcloud container node-pools create axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a \
  --machine-type=t2a-standard-2 \
  --num-nodes=2 \
  --node-labels=workload-type=arm-optimized

What each flag does:

• --cluster=axion-tutorial-cluster — the name of the cluster we created in Step 2. Node pools are always added to an existing cluster.

• --zone=us-central1-a — must match the zone you used when creating the cluster.

• --machine-type=t2a-standard-2 — GKE detects this is an ARM machine type and automatically provisions the nodes with an ARM-compatible version of Container-Optimized OS (COS). You don't need to configure anything special for ARM at the OS level.

• --num-nodes=2 — two ARM nodes in the zone, enough to schedule our 3-replica deployment alongside other cluster overhead.

• --node-labels=workload-type=arm-optimized — attaches a custom label to every node in this pool. We'll use this label in our deployment manifest to target these specific nodes. Using a descriptive custom label (rather than just relying on the automatic kubernetes.io/arch=arm64 label) is good practice in real clusters — it communicates the intent of the pool, not just its hardware.

This command takes a few minutes. Once it completes, let's confirm our cluster now has both node pools:

gcloud container clusters get-credentials axion-tutorial-cluster --zone=us-central1-a

kubectl get nodes --label-columns=kubernetes.io/arch

The get-credentials command configures kubectl to authenticate with your new cluster. The get nodes command then lists all nodes and adds a column showing the kubernetes.io/arch label.

You should see something like:

NAME                                    STATUS   ARCH    AGE
gke-...default-pool-abc...              Ready    amd64   15m
gke-...default-pool-def...              Ready    amd64   15m
gke-...axion-pool-jkl...                Ready    arm64   3m
gke-...axion-pool-mno...                Ready    arm64   3m

amd64 for the default x86 pool, arm64 for our new Axion pool. This kubernetes.io/arch label is applied automatically by GKE — you don't set it, it's derived from the hardware.

Step 8: Deploy the App to the ARM Node Pool

We have a multi-arch image and a mixed-architecture cluster. Here's something important to understand before writing the deployment manifest: Kubernetes doesn't know or care about image architecture by default.

If you applied a standard Deployment right now, the scheduler would look for any available node with enough CPU and memory and place pods there — potentially landing on x86 nodes instead of your ARM Axion nodes. The multi-arch Manifest List would handle this gracefully (the right binary would run regardless), but you'd lose the cost efficiency you provisioned Axion nodes for in the first place.

To guarantee that pods land on ARM nodes and only ARM nodes, we use a nodeSelector.

How nodeSelector Works

A nodeSelector is a set of key-value pairs in your pod spec. Before the Kubernetes scheduler places a pod, it checks every available node's labels. If a node doesn't have all the labels in the nodeSelector, the scheduler skips it — the pod will remain in Pending state rather than land on the wrong node.

This is a hard constraint, which is exactly what we want for cost optimization. Contrast this with Node Affinity's soft preference mode (preferredDuringSchedulingIgnoredDuringExecution), which says "try to use ARM, but fall back to x86 if needed." Soft preferences are useful for resilience, but they undermine the whole point of dedicated ARM pools. We want the hard constraint.

Write the Deployment Manifest

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-axion
  labels:
    app: hello-axion
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-axion
  template:
    metadata:
      labels:
        app: hello-axion
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64

      containers:
      - name: hello-axion
        image: us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
        resources:
          requests:
            cpu: "250m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

Replace PROJECT_ID with your project ID. Here's what the key sections do:

replicas: 3 — tells Kubernetes to keep three instances of this pod running at all times. If one crashes or a node goes down, the scheduler spins up a replacement. Three replicas also means one pod per ARM node in us-central1, which distributes load across availability zones.

selector.matchLabels and template.metadata.labels — these two blocks must match. The selector tells the Deployment which pods it "owns," and the template.metadata.labels is what those pods will be tagged with. If they don't match, Kubernetes won't be able to manage the pods.

nodeSelector: kubernetes.io/arch: arm64 — this is the pin. The Kubernetes scheduler filters out every node that doesn't carry this label before considering resource availability. Since GKE automatically applies kubernetes.io/arch=arm64 to all ARM nodes, our pods will schedule only onto the axion-pool nodes.

livenessProbe — periodically calls GET /healthz. If this check fails a certain number of times in a row (indicating the container has deadlocked or is otherwise unresponsive), Kubernetes restarts the container. initialDelaySeconds: 5 gives the server 5 seconds to start up before the first check.

readinessProbe — similar to the liveness probe, but with a different purpose. While the readiness probe is failing, Kubernetes removes the pod from the service's load balancer, so no traffic is sent to it. This is important during startup — the pod won't receive traffic until it signals it's ready.

resources.requests — reserves 250m (25% of a CPU core) and 64Mi of memory on the node for this pod. The scheduler uses these numbers to decide whether a node has enough room for the pod. Setting requests is required for sensible bin-packing. Without them, nodes can be silently overcommitted.

resources.limits — caps the container at 500m CPU and 128Mi memory. If the container exceeds these limits, Kubernetes throttles the CPU or kills the container (for memory). This prevents a single misbehaving pod from starving other workloads on the same node.

A Note on Taints and Tolerations

Once you're comfortable with nodeSelector, the next step in production clusters is adding a taint to your ARM node pool. A taint is a repellent — any pod without an explicit toleration for that taint is blocked from landing on the tainted node.

This means other workloads in your cluster can't accidentally consume your ARM capacity. You'd add the taint when creating the pool:

# Add --node-taints to the pool creation command:
--node-taints=workload-type=arm-optimized:NoSchedule

And a matching toleration in the pod spec:

tolerations:
- key: "workload-type"
  operator: "Equal"
  value: "arm-optimized"
  effect: "NoSchedule"

We're not doing this in the tutorial to keep things simple, but it's the pattern production multi-tenant clusters use to enforce hard separation between workload types.

Write the Service Manifest

We also need a Kubernetes Service to expose the pods over the network. Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-axion-svc
spec:
  selector:
    app: hello-axion
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

• selector: app: hello-axion — the Service discovers pods using labels. Any pod with app: hello-axion on it will be added to this Service's load balancer pool.

• port: 80 — the port the Service is reachable on from outside the cluster.

• targetPort: 8080 — the port on the pod that traffic gets forwarded to. Our Go server listens on port 8080, so this must match.

• type: LoadBalancer — tells GKE to provision an external Google Cloud load balancer and assign it a public IP. This is what makes the Service reachable from the internet.

Apply Both Manifests

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl apply reads each manifest file and creates or updates the resources described in it. If the resources don't exist yet, they're created. If they already exist, Kubernetes only applies the diff — it won't restart pods unnecessarily.

Watch the pods come up in real time:

kubectl get pods -w

The -w flag watches for changes and prints updates as they happen. You should see pods transition from Pending → ContainerCreating → Running. Once all three show Running, press Ctrl+C to stop watching.

Step 9: Verify the Deployment

Everything is running. Now we need evidence — not just that pods are up, but that they're on the right nodes and serving the right binary.

Confirm Pod Placement

kubectl get pods -o wide

The -o wide flag adds extra columns to the output, including the name of the node each pod was scheduled on. Look at the NODE column:

NAME                          READY   STATUS    NODE
hello-axion-7b8d9f-abc12      1/1     Running   gke-axion-tutorial-axion-pool-a-...
hello-axion-7b8d9f-def34      1/1     Running   gke-axion-tutorial-axion-pool-b-...
hello-axion-7b8d9f-ghi56      1/1     Running   gke-axion-tutorial-axion-pool-c-...

All three pods should show node names containing axion-pool. None should show default-pool.

Confirm the Nodes Are ARM

Take one of those node names and verify its architecture label:

kubectl get node NODE_NAME --show-labels | grep kubernetes.io/arch

Replace NODE_NAME with one of the node names from the previous command. You should see:

kubernetes.io/arch=arm64

That's the automatic label GKE applied when it provisioned the ARM hardware. Our nodeSelector matched on this label to pin the pods here.

Ask the Application Itself

This is the most satisfying verification step. Our Go server reports the architecture of the binary that's running. Let's ask it directly.

Use kubectl port-forward to create a secure tunnel from port 8080 on your local machine to port 8080 on the Deployment:

kubectl port-forward deployment/hello-axion 8080:8080

This command stays running in the foreground — open a second terminal window and run:

curl http://localhost:8080

You should see:

Hello from freeCodeCamp!
Architecture : arm64
OS           : linux
Pod hostname : hello-axion-7b8d9f-abc12

Architecture : arm64. That's our Go binary confirming that it was compiled for ARM64 and is executing on an ARM64 CPU. The single image tag we built does the right thing automatically.

The Bonus: See the Manifest List in Action

Want to see the multi-arch image indexing at work? Stop the port-forward, then run:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Replace PROJECT_ID with your actual Google Cloud project ID.

You'll see four entries in the manifest list. Two are real images — Platform: linux/amd64 and Platform: linux/arm64. The other two show Platform: unknown/unknown with an attestation-manifest annotation. These are build provenance records that Docker Buildx automatically attaches to every image — a supply chain security feature (SLSA attestation) that proves how and where the image was built.

You may notice that if you check the image digest recorded in a running pod:

kubectl get pod POD_NAME \
  -o jsonpath='{.status.containerStatuses[0].imageID}'

Replace POD_NAME with one of the pod names from earlier.

The digest returned matches the top-level manifest list digest, not the arm64-specific one. This is expected behaviour. Modern Kubernetes (using containerd) records the manifest list digest, not the resolved platform digest. The platform resolution already happened when the node pulled the correct image variant.

The definitive proof that the right binary is running is what you already have: the node labeled kubernetes.io/arch=arm64 and the application reporting Architecture: arm64.

Step 10: Cost Savings and Tradeoffs

The hands-on work is done. Let's talk about why any of this is worth the effort.

The Cost Math

At the time of writing, here's how ARM compares to equivalent x86 machines on Google Cloud (prices are approximate and change over time — check the official pricing page before making decisions):

That's a raw 25–30% reduction in compute cost per node. Factor in Google's published claim of up to 65% better price-performance for Axion on relevant workloads — meaning you may need fewer nodes to handle the same traffic — and the savings compound further.

Here's how that looks at scale, for a service running 20 nodes continuously for a year:

• 20 × n2-standard-4 × \(0.19 × 8,760 hours = \)33,288/year

• 20 × t2a-standard-4 × \(0.14 × 8,760 hours = \)24,528/year

That's roughly $8,760 saved annually on compute, before committed use discounts (which further widen the gap).

When ARM Is the Right Choice

ARM works best for:

• Stateless API servers and web applications — like the app we built. ARM excels at high-throughput, low-latency network workloads.

• Background workers and queue processors — long-running services that don't depend on x86-specific binaries.

• Microservices written in Go, Rust, or Python — these languages have excellent ARM64 support and are built cross-platform by default.

When to Proceed Carefully

• Native library dependencies — some older C libraries, proprietary SDKs, or compiled ML model-serving runtimes don't have ARM64 builds. Always audit your dependency tree before migrating.

• CI pipelines need ARM too — your automated tests should run on ARM, not just x86. An image that silently fails only on ARM is harder to debug than one that never claimed ARM support.

• Profile before optimizing — the cost savings are real, but measure your actual workload behavior on ARM before committing. Not every workload benefits equally.

Cleanup

When you're done, clean up to avoid ongoing charges:

# Remove the Kubernetes resources from the cluster
kubectl delete -f k8s/

# Delete the ARM node pool
gcloud container node-pools delete axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the cluster itself
gcloud container clusters delete axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the images from Artifact Registry (optional — storage costs are minimal)
gcloud artifacts docker images delete \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Conclusion

Let's recap what you built and why each part matters.

You started with a Go application, a Dockerfile, and a docker buildx build command that produced two images — one for x86, one for ARM64 — wrapped in a single Manifest List tag. Any server that pulls that tag gets the right binary automatically, without you maintaining separate pipelines or separate tags.

You provisioned a GKE cluster with two node pools running different CPU architectures, then used nodeSelector to make sure your ARM-optimized workload lands only on the ARM Axion nodes — not on x86 by accident. The result is a deployment that's both architecture-correct and cost-efficient.

The patterns you practiced here don't stop at this demo. The same Dockerfile technique works for any language with cross-compilation support. The same nodeSelector approach works for any workload you want to pin to ARM. As more teams migrate services to ARM over the coming years, having these skills will be a real asset.

Where to go from here:

• Add a GitHub Actions workflow that runs docker buildx build --platform linux/amd64,linux/arm64 on every push, automating this entire process in CI.

• Audit one of your existing stateless services for ARM compatibility and try migrating it.

• Explore Node Affinity as a softer alternative to nodeSelector for workloads that can run on either architecture but prefer ARM.

• Look into GKE Autopilot, which now supports ARM nodes and handles node pool management automatically.

Happy building.

Project File Structure

hello-axion/
├── app/
│   ├── main.go          — Go HTTP server
│   ├── go.mod           — Go module definition
│   └── Dockerfile       — Multi-stage Dockerfile
└── k8s/
    ├── deployment.yaml  — Deployment with nodeSelector and probes
    └── service.yaml     — LoadBalancer Service

All source files for this tutorial are available in the companion GitHub repository: https://github.com/Amiynarh/multi-arch-docker-gke-arm

It has no user database, no built-in login system, no password file. When you run kubectl get pods, Kubernetes receives an HTTP request and asks one question: who signed this, and do I trust that signature? Everything else — what you're allowed to do, which namespaces you can access, whether your request goes through at all — comes after that question is answered.

This surprises most engineers who are new to Kubernetes. They expect something like a database of users with passwords. Instead, they find a pluggable chain of authenticators, each one able to vouch for a request in a different way:

• Client certificates

• OIDC tokens from an external identity provider

• Cloud provider IAM tokens

• Service account tokens projected into pods.

Any of these can be active at the same time.

Understanding this model is what separates engineers who can debug authentication failures from engineers who copy kubeconfig files and hope for the best.

In this article, you'll work through how the Kubernetes authentication chain works from first principles. You'll see how x509 client certificates are used — and why they're a poor choice for human users in production. You'll configure OIDC authentication with Dex, giving your cluster a real browser-based login flow. And you'll see how AWS, GCP, and Azure each plug into the same underlying model.

Prerequisites

• A running kind cluster — a fresh one works fine, or reuse an existing one

• kubectl and helm installed

• openssl available on your machine (comes pre-installed on macOS and most Linux distros)

• Basic familiarity with what a JWT is (a signed JSON object with claims) — you don't need to be able to write one, just recognise one

All demo files are in the companion GitHub repository.

Table of Contents

• How Kubernetes Authentication Works

  • The Authenticator Chain

  • Users vs Service Accounts

  • What Happens After Authentication

• How to Use x509 Client Certificates

  • How the Certificate Maps to an Identity

  • The Cluster CA

  • The Limits of Certificate-Based Auth

• Demo 1 — Create and Use an x509 Client Certificate

• How to Set Up OIDC Authentication

  • How the OIDC Flow Works in Kubernetes

  • The API Server Configuration

  • JWT Claims Kubernetes Uses

  • How kubelogin Works

• Demo 2 — Configure OIDC Login with Dex and kubelogin

• Cloud Provider Authentication

  • AWS EKS

  • Google GKE

  • Azure AKS

• Webhook Token Authentication

• Cleanup

• Conclusion

How Kubernetes Authentication Works

Every request that reaches the Kubernetes API server — whether from kubectl, a pod, a controller, or a CI pipeline — carries a credential of some kind.

The API server passes that credential through a chain of authenticators in sequence. The first authenticator that can verify the credential wins. If none can, the request is treated as anonymous.

The Authenticator Chain

Kubernetes supports several authentication strategies simultaneously. You can have client certificate authentication and OIDC authentication active on the same cluster at the same time, which is common in production: cluster administrators use certificates, regular developers use OIDC. The strategies active on a cluster are determined by flags passed to the kube-apiserver process.

The strategies available are x509 client certificates, bearer tokens (static token files — rarely used in production), bootstrap tokens (used during node join operations), service account tokens, OIDC tokens, authenticating proxies, and webhook token authentication. A cluster doesn't have to use all of them, and most don't. But knowing they all exist helps when you're diagnosing an auth failure.

Users vs Service Accounts

There is an important distinction in how Kubernetes thinks about identity. Service accounts are Kubernetes objects — they live in a namespace, get created with kubectl create serviceaccount, and have tokens managed by the cluster itself. Every pod runs as a service account. These are machine identities for workloads.

Users, on the other hand, don't exist as Kubernetes objects at all. There is no kubectl create user command. Kubernetes doesn't manage user accounts. Instead, it trusts external systems to assert user identity — a certificate authority, an OIDC provider, or a cloud provider's IAM system. Kubernetes just verifies the assertion and extracts the username and group memberships from it.

What Happens After Authentication

Authentication only answers one question: who is this? Once the API server has a verified identity — a username and zero or more group memberships — it passes the request to the authorisation layer. By default that is RBAC, which checks the identity against Role and ClusterRole bindings to determine what the request is allowed to do.

This is why authentication and authorisation are separate concerns in Kubernetes. A valid certificate gets you past the front door. What you can do inside is RBAC's job. An authenticated user with no RBAC bindings can authenticate successfully but will be denied every API call.

If you want a deep dive into how RBAC rules, roles, and bindings work, check out this handbook on How to Secure a Kubernetes Cluster: RBAC, Pod Hardening, and Runtime Protection.

How to Use x509 Client Certificates

x509 client certificate authentication is the oldest and simplest authentication method in Kubernetes. It's how kubectl works out of the box when you create a cluster — the kubeconfig file that kind or kubeadm generates contains an embedded client certificate signed by the cluster's Certificate Authority.

How the Certificate Maps to an Identity

When the API server receives a request with a client certificate, it validates the certificate against its trusted CA, then reads two fields (The Common Name and Organization) from the certificate to construct an identity.

The Common Name (CN) field becomes the username. The Organization (O) field, which can contain multiple values, becomes the list of groups the user belongs to.

So a certificate with CN=jane and O=engineering authenticates as username jane in group engineering. If you want to give jane permissions, you create a RoleBinding that references either the username jane or the group engineering as a subject.

This is the same mechanism behind system:masters. When kind creates a cluster and writes a kubeconfig for you, it generates a certificate with O=system:masters. Kubernetes has a built-in ClusterRoleBinding that grants cluster-admin to anyone in the system:masters group. That's why your default kubeconfig has full admin access — it's not magic, it's a certificate with the right group.

The Cluster CA

Every Kubernetes cluster has a root Certificate Authority — a private key and a self-signed certificate that the API server trusts. Any client certificate signed by this CA is trusted by the cluster.

The CA certificate and key are typically stored in /etc/kubernetes/pki/ on the control plane node, or in the kube-system namespace as a secret, depending on how the cluster was created.

On kind clusters, you can copy the CA cert and key directly from the control plane container:

docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.crt ./ca.crt
docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.key ./ca.key

Whoever holds the CA key can issue certificates for any username and any group, including system:masters. This makes the CA key the most sensitive secret in a Kubernetes cluster. Guard it accordingly.

The Limits of Certificate-Based Auth

Client certificates work, but they have two fundamental problems that make them a poor choice for human users in production.

The first is that Kubernetes doesn't check certificate revocation lists (CRLs). If a developer's kubeconfig is stolen, the embedded certificate remains valid until it expires — which is typically one year in most Kubernetes setups. There's no way to immediately invalidate it. You can't "log out" a certificate. The only mitigation is to rotate the entire cluster CA, which invalidates every certificate including those belonging to other legitimate users.

The second is operational overhead. Certificates must be generated, distributed to users, and rotated before expiry. There's no self-service. In a team of ten engineers, managing certificates is annoying. In a team of a hundred, it's a full-time job.

For human access in production, OIDC is the right answer: short-lived tokens issued by a trusted identity provider, with a central revocation mechanism, and a standard browser-based login flow. Certificates are fine for service accounts and automation, where token management can be automated and rotation is handled programmatically.

That said, understanding certificates isn't optional. Your kubeconfig uses one. Your CI system probably does too. And cert-based auth is what you fall back to when everything else breaks.

Demo 1 — Create and Use an x509 Client Certificate

In this section, you'll generate a user certificate signed by the cluster CA, bind it to an RBAC role, and use it to authenticate to the cluster as a different user.

This guide is for local development and learning only. Manually signing certificates with the cluster CA and storing keys on disk is done here for simplicity.

In production, you should use the Kubernetes CertificateSigningRequest API or cert-manager for certificate issuance, enforce short-lived certificates with automatic rotation, and store private keys in a secrets manager (HashiCorp Vault, AWS Secrets Manager) or hardware security module (HSM) — never distribute the cluster CA key.

Step 1: Copy the CA cert and key from the kind control plane

docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.crt ./ca.crt
docker cp k8s-security-control-plane:/etc/kubernetes/pki/ca.key ./ca.key

This will create two files in your current directory called ca.crt and ca.key

Step 2: Generate a private key and CSR for a new user

You're creating a certificate for a user named jane in the engineering group:

# Generate the private key
openssl genrsa -out jane.key 2048

# Generate a Certificate Signing Request
# CN = username, O = group
openssl req -new \
  -key jane.key \
  -out jane.csr \
  -subj "/CN=jane/O=engineering"

Step 3: Sign the CSR with the cluster CA

openssl x509 -req \
  -in jane.csr \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -out jane.crt \
  -days 365

Expected output:

Certificate request self-signature ok
subject=CN=jane, O=engineering

Step 4: Inspect the certificate

Before using it, confirm the identity it carries:

openssl x509 -in jane.crt -noout -subject -dates

subject=CN=jane, O=engineering
notBefore=Mar 20 10:00:00 2024 GMT
notAfter=Mar 20 10:00:00 2025 GMT

One year from now, this certificate becomes invalid and must be replaced. There's no way to extend it — you have to issue a new one.

Step 5: Build a kubeconfig entry for jane

# Get the cluster API server address from the current context
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

# Create a kubeconfig for jane
kubectl config set-cluster k8s-security \
  --server=$APISERVER \
  --certificate-authority=ca.crt \
  --embed-certs=true \
  --kubeconfig=jane.kubeconfig

kubectl config set-credentials jane \
  --client-certificate=jane.crt \
  --client-key=jane.key \
  --embed-certs=true \
  --kubeconfig=jane.kubeconfig

kubectl config set-context jane@k8s-security \
  --cluster=k8s-security \
  --user=jane \
  --kubeconfig=jane.kubeconfig

kubectl config use-context jane@k8s-security \
  --kubeconfig=jane.kubeconfig

Step 6: Test authentication — before RBAC

Try to list pods using jane's kubeconfig:

kubectl get pods -n staging --kubeconfig=jane.kubeconfig

Error from server (Forbidden): pods is forbidden: User "jane" cannot list
resource "pods" in API group "" in the namespace "staging"

This is correct. Jane authenticated successfully — Kubernetes knows who she is. But she has no RBAC bindings, so every API call is denied. Authentication passed, but authorisation failed.

Step 7: Grant jane access with RBAC

RBAC bindings use the username exactly as it appears in the certificate's CN field. If you need a refresher on how Roles, ClusterRoles, and RoleBindings work, this handbook How to Secure a Kubernetes Cluster: RBAC, Pod Hardening, and Runtime Protection covers the full RBAC model. For now, a simple RoleBinding using the built-in view ClusterRole is enough:

# jane-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jane-reader
  namespace: staging
subjects:
  - kind: User
    name: jane          # matches the CN in the certificate
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f jane-rolebinding.yaml
kubectl get pods -n staging --kubeconfig=jane.kubeconfig

No resources found in staging namespace.

No error — jane can now list pods in staging. She can't delete them, create them, or access other namespaces. The certificate got her in. RBAC determines what she can do.

How to Set Up OIDC Authentication

OpenID Connect is an identity layer on top of OAuth 2.0. It's how Kubernetes integrates with enterprise identity providers — Active Directory, Okta, Google Workspace, Keycloak, and any other provider that speaks OIDC. Understanding how Kubernetes uses it requires following the token from the user's browser to the API server's decision.

How the OIDC Flow Works in Kubernetes

When a developer runs kubectl get pods with OIDC configured, the following happens:

1. kubectl checks whether the current credential in the kubeconfig is a valid, unexpired OIDC token

2. If not, it launches kubelogin, a kubectl plugin that opens a browser window

3. The browser redirects to the OIDC provider (Dex, Okta, your corporate IdP)

4. The user logs in with their corporate credentials

5. The OIDC provider issues a signed JWT and returns it to kubelogin

6. kubelogin caches the token locally (under ~/.kube/cache/oidc-login/) and returns it to kubectl

7. kubectl sends the token to the API server as a Bearer header

8. The API server fetches the provider's public keys from its JWKS endpoint and verifies the token signature

9. If valid, the API server extracts the username and group claims from the token

10. RBAC takes over from there

The Kubernetes API server never contacts the OIDC provider for each request. It only fetches the provider's public keys periodically to verify signatures locally. This makes OIDC authentication stateless and scalable.

The API Server Configuration

For OIDC to work, the API server needs to know where to find the identity provider and how to interpret the tokens it issues.

In Kubernetes v1.30+, this is configured through an AuthenticationConfiguration file passed via the --authentication-config flag. (In older versions, individual --oidc-* flags were used instead, but these were removed in v1.35.)

The AuthenticationConfiguration defines OIDC providers under the jwt key:

On a kind cluster, the --authentication-config flag is set in the cluster configuration before creation, not after. You'll see this in the next demo.

JWT Claims Kubernetes Uses

A JWT is a signed JSON object with three sections: a header, a payload, and a signature. The payload is a set of claims – key-value pairs that assert facts about the token. Kubernetes reads specific claims from the payload to build an identity.

The required claims are iss (the issuer URL, must match issuer.url in the AuthenticationConfiguration), sub (the subject, a unique identifier for the user), and aud (the audience, must match the issuer.audiences list). The exp claim (expiry time) is also required as the API server rejects expired tokens.

The most useful optional claim is groups (or whatever you configure via claimMappings.groups.claim). When this claim is present, Kubernetes can map OIDC group memberships directly to RBAC group bindings. A user in the platform-engineers group in your identity provider automatically gets the RBAC permissions you've bound to that group in Kubernetes — no manual user management required.

How kubelogin Works

kubelogin (also distributed as kubectl oidc-login) is a kubectl credential plugin. Instead of embedding a static certificate or token in your kubeconfig, you configure a credential plugin that runs a helper binary when kubectl needs a token.

When kubelogin is invoked, it checks its local token cache. If the cached token is still valid, it returns it immediately. If the token has expired, it initiates the OIDC authorization code flow — opens a browser, redirects to the identity provider, receives the token after login, caches it locally, and returns it to kubectl. The whole flow takes about five seconds when it triggers.

This means tokens are short-lived (typically an hour) and rotate automatically. If a developer's machine is compromised, the token expires on its own. There is no long-lived credential sitting in a file somewhere.

Demo 2 — Configure OIDC Login with Dex and kubelogin

In this section, you'll deploy Dex as a self-hosted OIDC provider, configure a kind cluster to trust it, and log in with a browser. Dex is a good demo vehicle because it runs inside the cluster and doesn't require a cloud account or an external service.

This guide is for local development and learning only. Self-signed certificates, static passwords, and certs stored on disk are used here for simplicity.

In production, use a managed identity provider (Azure Entra ID, Google Workspace, Okta), automate certificate lifecycle with cert-manager, and store secrets in a secrets manager (HashiCorp Vault, AWS Secrets Manager) or inject them via CSI driver — never commit or store certs as local files.

Step 1: Create a kind cluster with OIDC authentication

OIDC authentication for the API server must be configured at cluster creation time on Kind because the API server needs to know which identity provider to trust before it starts accepting requests.

Note: Kubernetes v1.30+ deprecated the --oidc-* API server flags in favor of the structured AuthenticationConfiguration API (via --authentication-config). In v1.35+ the old flags are removed entirely. This guide uses the new approach.

nip.io is a wildcard DNS service — dex.127.0.0.1.nip.io resolves to 127.0.0.1. This lets us use a real hostname for TLS without editing /etc/hosts.

First, generate a self-signed CA and TLS certificate for Dex:

# Generate a CA for Dex
openssl req -x509 -newkey rsa:4096 -keyout dex-ca.key \
  -out dex-ca.crt -days 365 -nodes \
  -subj "/CN=dex-ca"

# Generate a certificate for Dex signed by that CA
openssl req -newkey rsa:2048 -keyout dex.key \
  -out dex.csr -nodes \
  -subj "/CN=dex.127.0.0.1.nip.io"

openssl x509 -req -in dex.csr \
  -CA dex-ca.crt -CAkey dex-ca.key \
  -CAcreateserial -out dex.crt -days 365 \
  -extfile <(printf "subjectAltName=DNS:dex.127.0.0.1.nip.io")

Next, generate the AuthenticationConfiguration file. This tells the API server how to validate JWTs — which issuer to trust (url), which audience to expect (audiences), and which JWT claims map to Kubernetes usernames and groups (claimMappings). The CA cert is inlined so the API server can verify Dex's TLS certificate when fetching signing keys:

cat > auth-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: https://dex.127.0.0.1.nip.io:32000
      audiences:
        - kubernetes
      certificateAuthority: |
$(sed 's/^/        /' dex-ca.crt)
    claimMappings:
      username:
        claim: email
        prefix: ""
      groups:
        claim: groups
        prefix: ""
EOF

The kind-oidc.yaml config uses extraPortMappings to expose Dex's port to your browser, extraMounts to copy files into the Kind node, and a kubeadmConfigPatch to pass --authentication-config to the API server:

# kind-oidc.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      # Forward port 32000 from the Docker container to localhost,
      # so your browser can reach Dex's login page
      - containerPort: 32000
        hostPort: 32000
        protocol: TCP
    extraMounts:
      # Copy files from your machine into the Kind node's filesystem
      - hostPath: ./dex-ca.crt
        containerPath: /etc/ca-certificates/dex-ca.crt
        readOnly: true
      - hostPath: ./auth-config.yaml
        containerPath: /etc/kubernetes/auth-config.yaml
        readOnly: true
    kubeadmConfigPatches:
      # Patch the API server to enable OIDC authentication
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            # Tell the API server to load our AuthenticationConfiguration
            authentication-config: /etc/kubernetes/auth-config.yaml
          extraVolumes:
            # Mount files into the API server pod (it runs as a static pod,
            # so it needs explicit volume mounts even though files are on the node)
            - name: dex-ca
              hostPath: /etc/ca-certificates/dex-ca.crt
              mountPath: /etc/ca-certificates/dex-ca.crt
              readOnly: true
              pathType: File
            - name: auth-config
              hostPath: /etc/kubernetes/auth-config.yaml
              mountPath: /etc/kubernetes/auth-config.yaml
              readOnly: true
              pathType: File

Create the cluster:

kind create cluster --name k8s-auth --config kind-oidc.yaml

Step 2: Deploy Dex

Dex is an OIDC-compliant identity provider that acts as a bridge between Kubernetes and upstream identity sources (LDAP, SAML, GitHub, and so on). In this demo it runs inside the cluster with a static password database — two hardcoded users you can log in as.

The API server doesn't talk to Dex directly on every request. It only needs Dex's CA certificate (which you inlined in the AuthenticationConfiguration) to verify the JWT signatures on tokens that Dex issues.

The deployment has four parts: a ConfigMap with Dex's configuration, a Deployment to run Dex, a NodePort Service to expose it on port 32000 (matching the issuer URL), and RBAC resources so Dex can store state using Kubernetes CRDs.

First, create the namespace and load the TLS certificate as a Kubernetes Secret. Dex needs this to serve HTTPS. Without it, your browser and the API server would refuse to connect:

kubectl create namespace dex

kubectl create secret tls dex-tls \
  --cert=dex.crt \
  --key=dex.key \
  -n dex

Save the following as dex-config.yaml. This configures Dex with a static password connector — two hardcoded users for the demo:

# dex-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dex-config
  namespace: dex
data:
  config.yaml: |
    # issuer must exactly match the URL in your AuthenticationConfiguration
    issuer: https://dex.127.0.0.1.nip.io:32000

    # Dex stores refresh tokens and auth codes — here it uses Kubernetes CRDs
    storage:
      type: kubernetes
      config:
        inCluster: true

    # Dex's HTTPS listener — serves the login page and token endpoints
    web:
      https: 0.0.0.0:5556
      tlsCert: /etc/dex/tls/tls.crt
      tlsKey: /etc/dex/tls/tls.key

    # staticClients defines which applications can request tokens.
    # "kubernetes" is the client ID that kubelogin uses when authenticating
    staticClients:
      - id: kubernetes
        redirectURIs:
          - http://localhost:8000     # kubelogin listens here to receive the callback
        name: Kubernetes
        secret: kubernetes-secret     # shared secret between kubelogin and Dex

    # Two demo users with the password "password" (bcrypt-hashed).
    # In production, you'd connect Dex to LDAP, SAML, or a social login instead
    enablePasswordDB: true
    staticPasswords:
      - email: "jane@example.com"
        # bcrypt hash of "password" — generate your own with: htpasswd -bnBC 10 "" password
        hash: "\(2a\)10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
        username: "jane"
        userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"
      - email: "admin@example.com"
        hash: "\(2a\)10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
        username: "admin"
        userID: "a8b53e13-7e8c-4f7b-9a33-6c2f4d8c6a1b"
        groups:
          - platform-engineers

Save the following as dex-deployment.yaml. This creates the Deployment, Service, ServiceAccount, and RBAC that Dex needs to run:

# dex-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dex
  namespace: dex
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dex
  template:
    metadata:
      labels:
        app: dex
    spec:
      serviceAccountName: dex
      containers:
        - name: dex
          # v2.45.0+ required — earlier versions don't include groups from staticPasswords in tokens
          image: ghcr.io/dexidp/dex:v2.45.0
          command: ["dex", "serve", "/etc/dex/cfg/config.yaml"]
          ports:
            - name: https
              containerPort: 5556
          volumeMounts:
            - name: config
              mountPath: /etc/dex/cfg
            - name: tls
              mountPath: /etc/dex/tls
      volumes:
        - name: config
          configMap:
            name: dex-config
        - name: tls
          secret:
            secretName: dex-tls
---
# NodePort Service — exposes Dex on port 32000 on the Kind node.
# Combined with extraPortMappings, this makes Dex reachable from your browser
apiVersion: v1
kind: Service
metadata:
  name: dex
  namespace: dex
spec:
  type: NodePort
  ports:
    - name: https
      port: 5556
      targetPort: 5556
      nodePort: 32000
  selector:
    app: dex
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dex
  namespace: dex
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: dex
rules:
  - apiGroups: ["dex.coreos.com"]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dex
subjects:
  - kind: ServiceAccount
    name: dex
    namespace: dex
roleRef:
  kind: ClusterRole
  name: dex
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f dex-config.yaml
kubectl apply -f dex-deployment.yaml
kubectl rollout status deployment/dex -n dex

Step 3: Install kubelogin

# macOS
brew install int128/kubelogin/kubelogin

# Linux
curl -LO https://github.com/int128/kubelogin/releases/latest/download/kubelogin_linux_amd64.zip
unzip -j kubelogin_linux_amd64.zip kubelogin -d /tmp
sudo mv /tmp/kubelogin /usr/local/bin/kubectl-oidc_login
rm kubelogin_linux_amd64.zip

Confirm it's installed:

kubectl oidc-login --version

Step 4: Configure a kubeconfig entry for OIDC

This creates a new user and context in your kubeconfig. Instead of using a client certificate (like the default Kind admin), it tells kubectl to use kubelogin to get a token from Dex.

The --oidc-extra-scope flags are important: without email and groups, Dex won't include those claims in the JWT, and the API server won't know who you are or what groups you belong to.

kubectl config set-credentials oidc-user \
  --exec-api-version=client.authentication.k8s.io/v1beta1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg=--oidc-issuer-url=https://dex.127.0.0.1.nip.io:32000 \
  --exec-arg=--oidc-client-id=kubernetes \
  --exec-arg=--oidc-client-secret=kubernetes-secret \
  --exec-arg=--oidc-extra-scope=email \
  --exec-arg=--oidc-extra-scope=groups \
  --exec-arg=--certificate-authority=$(pwd)/dex-ca.crt

kubectl config set-context oidc@k8s-auth \
  --cluster=kind-k8s-auth \
  --user=oidc-user

kubectl config use-context oidc@k8s-auth

Step 5: Trigger the login flow

Jane has no RBAC permissions yet, so first grant her read access from the admin context:

kubectl --context kind-k8s-auth create clusterrolebinding jane-view \
  --clusterrole=view --user=jane@example.com

Now switch to the OIDC context and trigger a login:

kubectl get pods -n default

Your browser opens and redirects to the Dex login page. Log in as jane@example.com with password password.

After login, the terminal completes:

No resources found in default namespace.

The browser-based authentication worked. kubectl received the token from Dex, sent it to the API server, the API server validated the JWT signature using the CA certificate from the AuthenticationConfiguration, extracted jane@example.com from the email claim, matched it against the RBAC binding, and authorized the request.

Without the clusterrolebinding, you would see Error from server (Forbidden) — authentication succeeds (the API server knows who you are) but authorization fails (jane has no permissions). This is the distinction between 401 Unauthorized and 403 Forbidden.

Step 6: Inspect the JWT

A JWT (JSON Web Token) is a signed JSON payload that contains claims about the user. kubelogin caches the token locally under ~/.kube/cache/oidc-login/ so you don't have to log in on every kubectl command.

List the directory to find the cached file:

ls ~/.kube/cache/oidc-login/

Decode the JWT payload directly from the cache:

cat ~/.kube/cache/oidc-login/$(ls ~/.kube/cache/oidc-login/ | grep -v lock | head -1) | \
  python3 -c "
import json, sys, base64
token = json.load(sys.stdin)['id_token'].split('.')[1]
token += '=' * (4 - len(token) % 4)
print(json.dumps(json.loads(base64.urlsafe_b64decode(token)), indent=2))
"

You'll see something like:

{
  "iss": "https://dex.127.0.0.1.nip.io:32000",
  "sub": "CiQwOGE4Njg0Yi1kYjg4LTRiNzMtOTBhOS0zY2QxNjYxZjU0NjYSBWxvY2Fs",
  "aud": "kubernetes",
  "exp": 1775307910,
  "iat": 1775221510,
  "email": "jane@example.com",
  "email_verified": true
}

The email claim becomes jane's Kubernetes username because the AuthenticationConfiguration maps username.claim: email. The aud matches the configured audiences. The iss matches the issuer url. This is how the API server validates the token without contacting Dex on every request — it only needs the CA certificate to verify the JWT signature.

Step 7: Map OIDC groups to RBAC

The admin@example.com user has a groups claim in the Dex config containing platform-engineers. Instead of creating individual RBAC bindings per user, you can bind permissions to a group — anyone whose JWT contains that group gets the permissions automatically:

# platform-engineers-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: platform-engineers-admin
subjects:
  - kind: Group
    name: platform-engineers     # matches the groups claim in the JWT
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

You're currently logged in as jane@example.com via the OIDC context, but jane only has view permissions — she can't create cluster-wide RBAC bindings. Switch back to the admin context to apply this:

kubectl config use-context kind-k8s-auth
kubectl apply -f platform-engineers-binding.yaml
kubectl config use-context oidc@k8s-auth

Now clear the cached token to log out of jane's session, then trigger a new login as admin@example.com:

# Clear the cached token — this is how you "log out" with kubelogin
rm -rf ~/.kube/cache/oidc-login/

# This will open the browser again for a fresh login
kubectl get pods -n default

Log in as admin@example.com with password password. This time the JWT will contain "groups": ["platform-engineers"], which matches the ClusterRoleBinding you just created. The admin user gets full cluster access — without ever being added to a kubeconfig by name.

You can verify by decoding the new token (Step 6) — the groups claim will be present:

{
  "email": "admin@example.com",
  "groups": ["platform-engineers"]
}

This is the real power of OIDC group claims: you manage group membership in your identity provider, and Kubernetes permissions follow automatically. Add someone to the platform-engineers group in Dex (or any upstream IdP), and they get cluster-admin access on their next login — no kubeconfig or RBAC changes needed.

Cloud Provider Authentication

AWS, GCP, and Azure each give Kubernetes clusters a native authentication mechanism that ties into their IAM systems.

The implementations differ in API surface, but they all use the same underlying mechanism: OIDC token projection. Once you understand how Dex works above, these are all variations on the same theme.

AWS EKS

EKS uses the aws-iam-authenticator to translate AWS IAM identities into Kubernetes identities. When you run kubectl against an EKS cluster, the AWS CLI generates a short-lived token signed with your IAM credentials. The API server passes this token to the aws-iam-authenticator webhook, which verifies it against AWS STS and returns the corresponding username and groups.

User access is controlled via the aws-auth ConfigMap in kube-system, which maps IAM role ARNs and IAM user ARNs to Kubernetes usernames and groups. A typical entry looks like this:

# In kube-system/aws-auth ConfigMap
mapRoles:
  - rolearn: arn:aws:iam::123456789:role/platform-engineers
    username: platform-engineer:{{SessionName}}
    groups:
      - platform-engineers

AWS is migrating from the aws-auth ConfigMap to a newer Access Entries API, which manages the same mapping through the EKS API rather than a ConfigMap. The underlying authentication mechanism is the same.

Google GKE

GKE integrates with Google Cloud IAM using two different mechanisms, depending on whether you're authenticating as a human user or as a workload.

For human users, GKE accepts standard Google OAuth2 tokens. Running gcloud container clusters get-credentials writes a kubeconfig that uses the gcloud CLI as a credential plugin, generating short-lived tokens from your Google account automatically.

For pod-level identity — letting a pod assume a Google Cloud IAM role — GKE uses Workload Identity. You annotate a Kubernetes service account to bind it to a Google Service Account, and pods running as that service account can call Google Cloud APIs using the GSA's permissions:

# Bind a Kubernetes SA to a Google Service Account
kubectl annotate serviceaccount my-app \
  --namespace production \
  iam.gke.io/gcp-service-account=my-app@my-project.iam.gserviceaccount.com

Azure AKS

AKS integrates with Azure Active Directory. When Azure AD integration is enabled, kubectl requests an Azure AD token on behalf of the user via the Azure CLI, and the AKS API server validates it against Azure AD.

For pod-level identity, AKS uses Azure Workload Identity, which follows the same OIDC federation pattern as GKE Workload Identity. A Kubernetes service account is annotated with an Azure Managed Identity client ID, and pods can request Azure AD tokens without storing any credentials:

# Annotate a service account with the Azure Managed Identity client ID
kubectl annotate serviceaccount my-app \
  --namespace production \
  azure.workload.identity/client-id=<MANAGED_IDENTITY_CLIENT_ID>

The underlying pattern across all three providers is the same: a trusted OIDC token is issued by the cloud provider, verified by the Kubernetes API server, and mapped to an identity through a binding (the aws-auth ConfigMap, a GKE Workload Identity binding, or an AKS federated identity credential). The OIDC section in this article is the conceptual foundation for all of them.

Webhook Token Authentication

Webhook token authentication is worth knowing about because it appears in several common Kubernetes setups, even if you never configure it yourself.

When a request arrives with a bearer token that no other authenticator recognises, Kubernetes can send that token to an external HTTP endpoint for validation. The endpoint returns a response indicating who the token belongs to.

This is how EKS authentication worked before the aws-iam-authenticator was built into the API server. It's also how bootstrap tokens work during node join operations: a token is generated, embedded in the kubeadm join command, and validated by the bootstrap webhook when the new node contacts the API server for the first time.

For most clusters, you'll encounter webhook auth as something already running rather than something you configure. The main thing to know is that it exists and what it looks like when it appears in logs or configuration.

Cleanup

To remove everything created in this article:

# Delete the OIDC demo cluster
kind delete cluster --name k8s-auth

# Remove generated certificate files
rm -f ca.crt ca.key jane.key jane.csr jane.crt jane.kubeconfig
rm -f dex-ca.crt dex-ca.key dex.crt dex.key dex.csr dex-ca.srl auth-config.yaml

# Remove the kubelogin token cache
rm -rf ~/.kube/cache/oidc-login/

Conclusion

Kubernetes authentication is not a single mechanism — it's a chain of pluggable strategies, each one suited to different use cases. In this article you worked through the most important ones.

x509 client certificates are how Kubernetes works out of the box. The CN field becomes the username, the O field becomes the group, and the cluster CA is the trust anchor. You created a certificate for a new user, bound it to RBAC, and saw exactly how authentication and authorisation interact — authentication gets you in, RBAC determines what you can do.

You also saw the fundamental limitation: Kubernetes doesn't check certificate revocation lists, so a compromised certificate remains valid until it expires. This makes certificates a poor fit for human users in production environments.

OIDC is the production-grade answer. Tokens are short-lived, issued by a trusted identity provider, and map directly to Kubernetes groups through JWT claims. You deployed Dex as a self-hosted OIDC provider, configured the API server to trust it, and set up kubelogin for browser-based authentication.

You then decoded a JWT to see exactly what the API server reads from it, and mapped an OIDC group claim to a Kubernetes ClusterRoleBinding.

Cloud provider authentication — EKS, GKE, AKS — uses the same OIDC foundation with provider-specific wrappers. Understanding how Dex works makes each of those systems immediately readable.

All YAML, certificates, and configuration files from this article are in the companion GitHub repository.

But as your requirements grow, especially around compliance, security, multi-tenancy, or conflicting dependencies, your team will likely move beyond namespaces and start creating separate clusters.

What starts as a clean separation quickly becomes cluster sprawl, bringing higher costs, complex networking, and constant operational overhead.

In this article, we'll explore how kcp can help fix this problem by allowing you to run multiple “logical clusters” inside a single control plane.

Table of Contents

• Prerequisites

• The Challenge of Namespaces and Multiple Kubernetes Clusters

• Introducing kcp

• Getting Started with kcp

• Deploying and Managing Applications

• Beyond the Primitives: What We Didn't Cover

Prerequisites

• kubectl installed.

• A terminal to run commands

• Curl installed

The Challenge of Namespaces and Multiple Kubernetes Clusters

While namespaces provide some level of isolation, many teams often default to creating entirely new Kubernetes clusters to achieve stronger multi-tenancy, environment separation, or geographic distribution.

At first, this approach works well. But as systems grow, managing a fleet of clusters introduces challenges that often outweigh the benefits.

Every new cluster comes with its own control plane, which you'll need to continuously patch, upgrade, and monitor. Over time, this operational overhead will add up, consuming cycles that platform teams could otherwise spend on higher-value work.

Also, clusters don't naturally share service discovery or identity. This forces you to introduce extra layers like service meshes or VPN-based networking, which increases your system's complexity and expands the overall attack surface.

There’s also the cost factor. Clusters incur baseline infrastructure costs regardless of how much workload they run. Creating dedicated clusters for small teams can lead to underutilized resources or, worse, delay the creation of necessary environments because the cost feels too high.

As a result, platform teams often find themselves acting as “cluster plumbers”, spending more time maintaining infrastructure than enabling developer productivity.

Illustrating the Namespace Problem

As I mentioned earlier, when managing multiple clusters gets too complex, a natural alternative is to use namespaces for isolation within a single cluster.

At first glance, this seems like the perfect solution.

But to understand where this approach falls short, let’s walk through a real-world example using a common requirement in shared Kubernetes environments: running databases.

We'll start by creating different namespaces for each team:

➜ ~ kubectl create namespace team-a 
➜ ~ kubectl create namespace team-b

Let's say Team A needs a MongoDB database for one of its services. The team must first install the required MongoDB Custom Resource Definitions (CRDs) into the cluster, so Kubernetes knows how to understand the different MongoDB resources:

➜ ~ kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/crds.yaml

customresourcedefinition.apiextensions.k8s.io/clustermongodbroles.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbmulticluster.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbsearch.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbusers.mongodb.com created customresourcedefinition.apiextensions.k8s.io/opsmanagers.mongodb.com created customresourcedefinition.apiextensions.k8s.io/mongodbcommunity.mongodbcommunity.mongodb.com created

Secondly, Team A installs the actual Operator application (the controller that continuously runs the database logic) into their designated namespace:

➜ ~ kubectl apply -n team-a -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/mongodb-kubernetes.yaml

But the installation isn't completed due to the error below:

the namespace from the provided object "mongodb" does not match the namespace "team-a". You must pass '--namespace=mongodb' to perform this operation.

Why did this fail? This is because most Kubernetes Operators are designed assuming they own the entire cluster and not just a single namespace.

To force the operator to run in team-a, we can modify the manifest on the fly:

curl -s https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/mongodb-kubernetes.yaml \
  | sed 's/namespace: mongodb/namespace: team-a/g' \
  | kubectl apply -f 

We can then confirm that the operator is installed and running:

➜ ~ k get po -n team-a 
NAME                                          READY STATUS  RESTARTS AGE 
mongodb-kubernetes-operator-6f5f8bb7fd-8h5hj  1/1   Running 0        59s

But even after tricking the Operator into running inside team-a's namespace, we still haven't solved the real problem.

At first glance, team-a's operator is neatly confined to their namespace. But remember Step 1? The CRDs aren't namespaced – they're strictly cluster-scoped. So, even though team-a orchestrated this deployment purely for their own use, those CRDs are now globally registered across the entire cluster.

If Team B checks the API, they'll see all the MongoDB-related CRDs installed by Team A.

➜ ~ kubectl get crds | grep mongodb

clustermongodbroles.mongodb.com               2026-03-24T10:49:35Z
mongodb.mongodb.com                           2026-03-24T10:49:36Z
mongodbcommunity.mongodbcommunity.mongodb.com 2026-03-24T10:49:38Z
mongodbmulticluster.mongodb.com               2026-03-24T10:49:36Z
mongodbsearch.mongodb.com                     2026-03-24T10:49:37Z 
mongodbusers.mongodb.com                      2026-03-24T10:49:37Z 
opsmanagers.mongodb.com                       2026-03-24T10:49:37Z

Now consider what happens if Team B needs to install a different version of MongoDB for its own services. Because the CRDs are shared across the cluster, both teams are now coupled to the same definitions. This means one team’s changes can easily impact the other, turning what should be isolated environments into a source of conflict.

Introducing kcp

kcp is an open-source project that lets you run multiple logical Kubernetes clusters on a single control plane.

These logical clusters are called workspaces, and each one behaves like an independent Kubernetes cluster. Every workspace has its own API endpoint, authentication, authorization, and policies, giving teams the experience of working in fully isolated environments.

This decoupling of the control plane from the worker nodes is what makes kcp different.

In traditional Kubernetes, spinning up a new cluster means provisioning a new API server, a new etcd instance, and all the associated controllers. With kcp, you spin up a workspace, and you have a strong, confined environment for your workload.

It's worth noting that kcp itself doesn't run workloads. It's strictly a control plane. Your actual applications still run on physical Kubernetes clusters. kcp only manages the workspaces and the synchronization of resources to those underlying clusters.

Getting Started with kcp

Now that we've covered what kcp is and why it matters, let's get our hands dirty. We'll set up a local kcp environment and explore the core concepts in action.

To make this realistic, we'll follow a common kcp workflow: a platform team that provides custom APIs, and tenant teams that consume them.

In our case, the platform team will export a MongoDB API, and our two tenant teams will subscribe to those APIs using APIBindings. Once bound, they can deploy MongoDB instances into their workspaces and sync them to physical clusters.

This pattern is at the heart of how kcp enables scalable multi-tenancy. The platform team controls the API definitions and versioning. Tenant teams get self-service access without needing to understand the underlying infrastructure. Let's see how it works!

Installing kcp

Running kcp locally is incredibly lightweight since there are no heavy worker nodes to spin up. You will need two things: the kcp server itself, kubectl-kcp , and the kubectl-ws plugin to manage workspaces.

To install the binaries, let's head over to the kcp-dev releases page.

The commands below are for macOS Apple Silicon. If you're using an Intel Mac or Linux, simply replace darwin_arm64 with your respective architecture.

1. Download the kcp server and workspace plugins:

➜ ~ curl -LO https://github.com/kcp-dev/kcp/releases/download/v0.30.1/kcp_0.30.1_darwin_arm64.tar.gz 

➜ ~ curl -LO https://github.com/kcp-dev/kcp/releases/download/v0.30.1/kubectl-kcp-plugin_0.30.1_darwin_arm64.tar.gz

➜ ~ curl -LO https://github.com/kcp-dev/kcp/releases/download/v0.30.1/kubectl-ws-plugin_0.30.1_darwin_arm64.tar.gz

1. Extract the archives:

➜ ~ tar -xzf kcp_0.30.1_darwin_arm64.tar.gz 
➜ ~ tar -xzf kubectl-kcp-plugin_0.30.1_darwin_arm64.tar.gz
➜ ~ tar -xzf kubectl-ws-plugin_0.30.1_darwin_arm64.tar.gz

1. Move the required binaries into your PATH:

➜ ~ sudo mv bin/kcp /usr/local/bin/
➜ ~ sudo mv bin/kubectl-kcp /usr/local/bin/
➜ ~ sudo mv bin/kubectl-ws /usr/local/bin/

You can confirm the installation by checking the version.

➜ ~ kcp --version
kcp version v1.33.3+kcp-v0.0.0-627385a6

Starting the Server

With the binaries installed, let's boot up your local control plane and bind it to localhost. But first, let's create a "work-folder".

➜ ~ mkdir kcp-test
➜ ~ cd kcp-test

We can then start the kcp server in this directory.

➜ ~ kcp start --bind-address=127.0.0.1

You'll see a flurry of logs as kcp boots up its internal database and exposes the API server. Leave this terminal running in the background.

Connecting to the Root Workspace

Open a new terminal window and navigate back into the kcp-test folder we just created.

At first, if you run a standard ls command, the folder will look empty. But during startup, kcp silently generated a hidden .kcp directory that contains our local certificates and our administrative kubeconfig file. Let's verify that:

➜ ~ cd kcp-test 
➜ kcp-test ls
➜ kcp-test ls -a . .. .kcp 
➜ kcp-test ls .kcp admin.kubeconfig apiserver.crt apiserver.key etcd-server sa.key

Now that we know exactly where the configuration file lives, let's export it so our kubectl commands are routed to kcp instead of your default cluster:

export KUBECONFIG=$PWD/.kcp/admin.kubeconfig

Finally, let's use the workspace plugin we installed earlier to verify that we're connected accurately:

 ➜ kubectl ws .

You should see the message below printed to the console:

Current workspace is 'root'.

This shows that you're now officially inside the kcp root workspace. This is the highest-level administrative boundary where we'll begin creating our tenant logical clusters.

Creating and Managing Workspaces

As we discussed above, in a standard Kubernetes cluster, separating teams means using kubectl create namespace. In kcp, we solve the problem by creating entirely isolated logical clusters – workspaces.

If you recall our architecture diagram from earlier, we want to create three distinct environments for our company: one for the platform engineers to manage shared APIs, and two for our isolated tenant development teams.

Since we're currently inside the administrative root workspace, we can create our new tenant workspaces as children of the root:

➜ kubectl ws create platform-team
Workspace "platform-team" (type root:organization) created.
Waiting for it to be ready... 
Workspace "platform-team" (type root:organization) is ready to use.

➜ kubectl ws create team-a 
Workspace "team-a" (type root:organization) created.
Waiting for it to be ready... 
Workspace "team-a" (type root:organization) is ready to use.

➜ kubectl ws create team-b
Workspace "team-b" (type root:organization) created.
Waiting for it to be ready... 
Workspace "team-b" (type root:organization) is ready to use.

Now, here is where kcp truly shines. Unlike a standard cluster, where objects are just a massive flat list, kcp manages its API as a hierarchy. We can visually prove the structure of our new logical clusters using the tree command:

➜ kubectl ws tree
.
└── root
      ├── platform-team
      ├── team-a
      └── team-b

Jumping between these logical clusters is as fast as changing directories in a terminal. Let's switch our context over into Team A's workspace:

➜ kubectl ws team-a 
Current workspace is 'root:team-a' (type root:organization).

Proving the Isolation

To truly understand the power of what we just did, let's try running a standard Kubernetes command while inside team-a:

➜ kubectl get namespaces

NAME STATUS AGE 
default Active 15m

Let's also ask the cluster what APIs are actually available to us out of the box:

➜ kubectl api-resources

Your output should be similar to what is in the image below:

When you take a closer look at that list. You'll notice that there are no Pods, Deployments, or even ReplicaSets. You don't see all the available APIs that you see in a standard Kubernetes Cluster.

This output proves exactly what we discussed in the architecture section. kcp is incredibly lightweight because every new workspace is born completely stripped of compute. Out of the box, it only contains the absolute bare-minimum control plane APIs needed for routing, RBAC, namespaces, and authentication.

From Team A's perspective, they own this pristine, empty universe. If they install a massive, noisy operator right now, like the MongoDB CRD, it will only exist right here in this specific API bucket.

But this raises the ultimate question: If there are no Deployments or Pods APIs in this workspace... how do we actually deploy our applications?

Deploying and Managing Applications

Now that we have set up our isolated environments, we must address the glaring issue from our last terminal output: How do developers actually deploy applications if there are no Deployment or Pod APIs?

In standard Kubernetes, the API is monolithic. You get everything whether you need it or not, and adding a new schema (like an Operator) forces it globally onto everyone.

kcp takes the exact opposite approach. Every workspace starts completely empty. You then selectively "subscribe" your workspace to only the APIs you actually need using two incredibly powerful new concepts: APIExports and APIBindings.

Let's see exactly how this solves our MongoDB multi-tenancy problem, step by step.

1. The Platform Team "Exports" the API

Instead of treating Custom Resource Definitions as global hazards, the platform engineers manage them centrally. First, lets switch into the platform-team workspace:

➜ kubectl ws :root:platform-team

Current workspace is 'root:platform-team' (type root:organization).

Here, we'll install the MongoDB Operator CRDs in the platform-team's workspace:

➜ kubectl apply -f kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes/1.7.0/public/crds.yaml

To confirm that this is indeed isolated, let's first check what CRDs were installed,

➜ kubectl get crd

NAME                                          CREATED AT
clustermongodbroles.mongodb.com               2026-03-24T20:45:50Z
mongodb.mongodb.com                           2026-03-24T20:45:50Z
mongodbcommunity.mongodbcommunity.mongodb.com 2026-03-24T20:45:51Z
mongodbmulticluster.mongodb.com               2026-03-24T20:45:50Z
mongodbsearch.mongodb.com                     2026-03-24T20:45:51Z
mongodbusers.mongodb.com                      2026-03-24T20:45:51Z
opsmanagers.mongodb.com                       2026-03-24T20:45:51Z

We can switch to team-a's workspace (any of the team's workspaces can be used, we're just trying to establish that the installed CRD is only visible in the platform-team's workspace).

➜ kubectl ws :root:team-a

Current workspace is 'root:team-a' (type root:organization).

➜ kubectl get crd 
No resources found

What we get as output is that there are no custom resources found or registered. This is the power of kcp.

If you don't want to continually type out paths to switch between your logical clusters, the kcp plugin includes a powerful interactive UI right in your terminal.

By running kubectl ws -i, you can use your arrow keys to navigate through your hierarchy and press Enter to instantly switch your context. Even better, this interactive mode provides a holistic view of your environment at any given time. With a single glance, you can see exactly how many APIExports are hosted inside a specific workspace, or which APIs are currently bound by other workspaces.

Let's switch back to the platform-team's workspace to continue with our setup.

Now, we need to do something kcp-specific. If you check your resources right now, those CRDs are strictly local to this workspace. To safely share them with our tenant teams, we need to convert them into an internal kcp tracking object called an APIResourceSchema. This is how kcp structurally version-controls APIs so they can be securely exported.

To do this, we use our kcp plugin to take a "snapshot" of the local MongoDB CRD:

kubectl get crd mongodbcommunity.mongodbcommunity.mongodb.com -o yaml | kubectl kcp crd snapshot -f - --prefix v1 | kubectl apply -f -

You should see an output that says:

apiresourceschema.apis.kcp.io/v1.mongodbcommunity.mongodbcommunity.mongodb.com created

This tells kcp: "Get the CRD we just installed, take a snapshot with the prefix 'v1', and apply the resulting APIResourceSchema back to the cluster."

Now, let's look for the schema kcp just generated for us:

➜ kubectl get apiresourceschemas

NAME                                             AGE
v1.mongodbcommunity.mongodbcommunity.mongodb.com 11s

To safely share this API with our teams, we wrap that generated schema into an APIExport. This acts like "APIs as a Service," publishing the schema so that other workspaces can optionally choose to consume it.

Let's create the Export using the exact schema name we just found:

➜ cat <<EOF | kubectl apply -f -
apiVersion: apis.kcp.io/v1alpha1
kind: APIExport
metadata:
  name: mongodb-v1
spec:
  latestResourceSchemas:
    - v1.mongodbcommunity.mongodbcommunity.mongodb.com
EOF

We can confirm this was successfully created by checking the APIExport resource we have

➜ kubectl get apiexports

NAME       AGE
mongodb-v1 2m46s

2. Tenant Teams "Bind" to the API

Now let's switch our terminal context back over to Team A. Remember our previous output? Their workspace currently has no idea what a MongoDB cluster is. Let's prove it:

➜ kubectl ws :root:team-a
Current workspace is "root:team-a" (type root:organization).

➜ kubectl api-resources | grep mongodb
# (No output. The API does not exist here!)

To securely subscribe to the platform team's newly created API service, Team A needs to create an APIBinding.

While we can write standard Kubernetes YAML to do this, the kcp plugin provides a bind command. Team A simply points the bind command directly at the workspace and the specific API export they want to consume:

➜ kubectl kcp bind apiexport root:platform-team:mongodb-v1
apibinding mongodb-v1 created. Waiting to successfully bind ...
mongodb-v1 created and bound.

➜ kcp-test kubectl get apibindings
NAME                  AGE   READY
mongodb-v1            73s   True
tenancy.kcp.io-bqt7a  7h10m True
topology.kcp.io-9dlvq 7h10m True

The moment Team A executes that bind command, their workspace is magically updated with the new capabilities. Let's check our api-resources one more time:

➜ kubectl api-resources | grep mongodb
mongodbcommunity mdbc mongodbcommunity.mongodb.com/v1 true MongoDBCommunity

Beyond the Primitives: What We Didn't Cover

At this point, you should have a firm, hands-on grasp of the core user primitives of kcp, that is Workspaces, APIExports, and APIBindings. But we've only just scratched the surface of what this architecture makes possible.

To keep this guide digestible, there are a few massive topics that I deliberately didn't cover in this article:

1. Shards and High Availability: Since kcp is designed to host thousands of logical clusters, a single database isn't enough. kcp introduces the Shard primitive, allowing platform administrators to horizontally partition workspace state across multiple underlying etcd instances. This gives kcp infinite scalability and massive High Availability (HA) without complicating the developer experience.

2. Front-Proxy: When kcp scales to host millions of logical clusters, it needs a way to seamlessly direct traffic. The kcp Front-Proxy sits at the absolute edge of the architecture, dynamically routing incoming kubectl API requests go straight to the correct underlying workspace and shard. It ensures the developer experience feels perfectly unified, no matter how massive the background infrastructure actually becomes.

3. Virtual Workspaces: While the workspaces we built today act as simple isolated buckets of state, kcp also supports Virtual Workspaces. These act as dynamic, read-only projections of data. For example, kcp uses virtual workspaces to project a unified view of a specific API across multiple tenant workspaces so that controllers can easily watch them all at once.

4. APIExportEndpointSlices: Just like standard Kubernetes uses endpoints to route traffic to pods, kcp uses EndpointSlices to efficiently route and scale the delivery of massive APIExports across thousands of consuming workspaces.

5. Wiring up the Sync Agent (api-syncagent): We discussed this conceptually in our architecture diagram, but we didn't actually attach a physical cluster. In a production scenario, you deploy the Sync Agent onto a fleet of downstream execution clusters (like EKS, GKE, or On-Premises environments) to automatically pull workloads safely out of kcp and execute them seamlessly on physical hardware.

6. External Integrations Like Crossplane: Because kcp acts purely as a multi-tenant API control plane, it pairs incredibly well with Crossplane. By publishing Crossplane as an APIExport, you can empower developer teams to provision actual cloud infrastructure (like AWS databases or Cloud Spanners) using standard YAML directly from their completely isolated kcp workspaces.

We will cover those advanced integrations in a future deep-dive. But armed with just the base primitives we built today, we can already solve the incredibly complex infrastructure problems we outlined at the beginning of the article.

Storing them is straightforward. But handling rotation, stale env vars, and the gap between what your pod reads and what AWS actually holds is where many engineers go quiet.

In this guide, you'll build a complete secrets pipeline from AWS Secrets Manager into Kubernetes pods. You'll provision the infrastructure with Terraform, sync secrets using the External Secrets Operator, and run a sample application that reads the same credentials in two different ways: via environment variables and via a volume mount.

By the end, you'll be able to:

• Explain the full architecture from vault to pod

• Run the lab locally in about 15 minutes

• Prove why environment variables go stale after rotation, while mounted secret files stay fresh

• Deploy the same pattern on Amazon Elastic Kubernetes Service with OpenID Connect-based CI/CD

• Troubleshoot the most common failures

Below is an architecture diagram showing secrets flowing from AWS Secrets Manager through the External Secrets Operator into a Kubernetes Secret, then splitting into environment variables set at pod start and a volume mount that updates within 60 seconds.

Table of Contents

• Prerequisites

• How to Understand the Secret Flow

• How to Run the Local Lab

• How to Inspect the ExternalSecret and the Application

• How to Test Secret Rotation

• How to Choose Between External Secrets Operator and the CSI Driver

• How to Deploy the Pattern on Amazon Elastic Kubernetes Service

• How to Configure GitHub Actions Without Stored AWS Credentials

• How to Troubleshoot the Most Common Failures

• Conclusion

Prerequisites

Before you begin, make sure you have the following tools installed and configured.

For the local lab:

• An AWS account with access to AWS Secrets Manager

• The AWS CLI installed and configured. Run aws configure and provide your access key, secret key, default region, and output format. The credentials need permission to read and write secrets in AWS Secrets Manager.

• kubectl installed. For Microk8s, run microk8s kubectl config view --raw > ~/.kube/config after installation to connect kubectl to your local cluster.

• Terraform installed

• Helm installed

• Docker installed

• A local Kubernetes cluster: the lab supports Microk8s and kind. If you do not have either installed, follow the Microk8s install guide before continuing.

For the Amazon Elastic Kubernetes Service sections:

• An Amazon Elastic Kubernetes Service cluster you can create or manage

• A GitHub repository you can configure for workflows and secrets

The lab repository includes two deployment paths: a local path for fast learning and an Amazon Elastic Kubernetes Service path for a production-like setup. All the exact commands for each path live in the repo's docs/DEPLOY-LOCAL.md and docs/DEPLOY-EKS.md.

How to Understand the Secret Flow

Before you run any command, you need to understand how the pieces connect.

The flow has four stages:

1. A developer or automated system updates a secret in AWS Secrets Manager.

2. The External Secrets Operator polls AWS Secrets Manager on a schedule and creates or updates a Kubernetes Secret.

3. Your pod reads that Kubernetes Secret.

4. During rotation, the Kubernetes Secret updates, but your two consumption modes behave differently.

How the External Secrets Operator Sync Works

The External Secrets Operator reads a custom Kubernetes resource called ExternalSecret. That resource tells the operator three things:

• Which secret store to connect to

• Which Kubernetes Secret name to create or update

• How often to refresh

In this lab, the ExternalSecret creates a Kubernetes Secret named myapp-database-creds. The operator also adds a template annotation that can trigger a pod restart when the secret rotates.

How the App Consumes Secrets

The sample application exposes three endpoints so you can validate behavior at any time.

• /secrets/env shows what environment variables the pod sees

• /secrets/volume shows what files in the mounted secret directory look like

• /secrets/compare compares both and reports whether rotation has been detected

The app checks four keys: DB_USERNAME, DB_PASSWORD, DB_HOST, and DB_PORT.

How to Run the Local Lab

The local lab gives you a fast learning loop. You can see the full pipeline working and test rotation without waiting for a cloud deployment.

Step 1: Clone the Repo

git clone https://github.com/Osomudeya/k8s-secret-lab
cd k8s-secret-lab

Step 2: Run the Spin-Up Script

bash spinup.sh

The script will ask you to choose a local cluster type. Pick Microk8s or kind, depending on what you have installed. The script installs the External Secrets Operator via Helm, applies the Terraform configuration, and deploys the sample application.

If the script fails at any point, check docs/TROUBLESHOOTING.md before retrying. The most common causes are missing AWS credentials, a misconfigured kubeconfig, or a Microk8s storage add-on that is not enabled.

Important: Run the Lab UI

The lab ships with a separate guided tutorial interface that runs on your laptop. This is not the in-cluster application, it's a React-based checklist at lab-ui/ that walks you through each concept and checkpoint as you work through the lab.

To start it, open a second terminal and run:

cd lab-ui && npm install && npm run dev

Then open http://localhost:5173. You'll see a module-by-module guide covering the full flow from external secrets to rotation to CI/CD.

Keep this terminal running alongside your lab. The Lab UI and the in-cluster app (localhost:3000) are two separate things, the UI guides you through the steps, the app shows you the live secrets.

Step 3: Access the Application

Once the lab finishes, port-forward the service.

kubectl port-forward svc/myapp 3000:80 -n default

Open http://localhost:3000. You should see a table showing each secret key and whether the environment variable value matches the volume mount value.

Step 4: Validate That Secrets Match

Run the compare endpoint directly from the terminal.

curl -s http://localhost:3000/secrets/compare | python3 -m json.tool

When everything is working, the response will include "all_match": true.

How to Inspect the ExternalSecret and the Application

At this point the lab is running. Now you'll want to inspect the manifests so you understand what each part does.

Step 1: Read the ExternalSecret Manifest

Open k8s/aws/external-secret.yaml. Focus on these four fields:

• refreshInterval: how often the operator polls AWS Secrets Manager

• secretStoreRef: which store the operator authenticates against

• target: the name of the Kubernetes Secret to create

• data: the mapping from AWS Secrets Manager JSON keys to Kubernetes Secret keys

Here is what that mapping looks like in this lab:

spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myapp-database-creds
    creationPolicy: Owner
  data:
    - secretKey: DB_USERNAME
      remoteRef:
        key: prod/myapp/database
        property: username

The property field tells the operator which JSON key inside the AWS secret to extract. If your secret in AWS Secrets Manager is a JSON object, each field gets its own entry here.

Two fields here are worth understanding before you move on. creationPolicy: Owner means the operator owns the Kubernetes Secret it creates. If you delete the ExternalSecret, the Secret is deleted too. ClusterSecretStore is a cluster-scoped store, meaning any namespace in the cluster can use it. A plain SecretStore is namespace-scoped. For this lab, cluster-scoped is the right choice because it keeps the setup simple.

Step 2: Read the Deployment Manifest

Open k8s/aws/deployment.yaml. You are looking for two sections: envFrom and volumeMounts.

envFrom:
  - secretRef:
      name: myapp-database-creds

volumeMounts:
  - name: db-secret-vol
    mountPath: /etc/secrets
    readOnly: true

Both paths read from the same Kubernetes Secret, myapp-database-creds. The envFrom block injects all keys as environment variables at pod start.
The volumeMounts block mounts the same secret as files under /etc/secrets.

This is the core of the rotation lesson. Both paths read the same source. But they behave differently after that source changes.

Step 3: Read the App Comparison Logic

Open app/server.js. The comparison logic reads environment variables from process.env and reads mounted secret files from /etc/secrets/<key>. Then it computes a per-key match and a global all_match value.

The /secrets/compare endpoint sets rotation_detected: true when any key differs between env and volume.

How to Test Secret Rotation

Secret rotation is where real teams feel pain. This lab makes that pain visible so you can explain it clearly and fix it confidently.

How the Rotation Gap Works

When a pod starts, Kubernetes gives it two ways to read a secret.

The first way is environment variables. Think of these like sticky notes written on the wall of the container the moment it boots up. The value gets written once, at startup, and never changes. Even if the secret in AWS gets updated ten minutes later, the sticky note still says the old value. The container cannot see the update because nobody rewrote the note.

The second way is a volume mount. Think of this like a shared folder that someone else can update remotely. Kubernetes creates a small folder inside the container and puts the secret value in a file there. When the secret changes in AWS and ESO syncs it into Kubernetes, the kubelet quietly updates that file within about 60 seconds. The container reads the file fresh every time it needs the value, so it sees the new password automatically.

Same secret, two paths. One goes stale while one stays fresh.

The problem happens when your app reads the database password from the environment variable, the sticky note, and someone rotates the password in AWS. ESO updates Kubernetes. The file gets the new password. But your app is still reading the sticky note, which has the old one. Connection fails.

That difference isn't a bug. It's how the Linux process model and the kubelet work. Understanding it is the difference between knowing Kubernetes secrets and actually operating them.

Here is what you're about to observe in the lab:

• The rotation script updates the secret in AWS

• ESO syncs the new value into Kubernetes within seconds

• The volume file updates automatically

• The environment variable stays stale until the pod restarts

• The /secrets/compare endpoint shows both values side by side so you can see the gap live

Step 1: Confirm the Lab Is Ready

Make sure your pod and the External Secrets Operator are both running before you start.

kubectl get pods -n external-secrets
kubectl get pods -n default

Both should show Running.

Step 2: Run the Rotation Test Script

bash rotation/test-rotation.sh

The script performs these actions in order:

1. Reads the current DB_PASSWORD from the volume mount at /etc/secrets/DB_PASSWORD

2. Reads the current DB_PASSWORD from the environment variable

3. Updates AWS Secrets Manager with a new password using put-secret-value

4. Forces an immediate ESO sync by annotating the ExternalSecret with force-sync

5. Reads the volume value again

6. Reads the environment variable again

After the script runs, the volume and the env var will show different values.

Step 3: Validate With the Compare Endpoint

Hit the compare endpoint and look at the output.

curl -s http://localhost:3000/secrets/compare | python3 -m json.tool

You'll see something like this:

{
  "comparison": {
    "DB_PASSWORD": {
      "env": "old-password-value",
      "volume": "new-password-value",
      "match": false
    }
  },
  "all_match": false,
  "rotation_detected": true,
  "message": "Volume has new value; env still has old value."
}

Step 4: Restart the Deployment to Sync Env Vars

Env vars don't update in place. You need a pod restart so new containers start with the updated Kubernetes Secret.

kubectl rollout restart deployment/myapp -n default
kubectl rollout status deployment/myapp -n default

Then hit /secrets/compare again. All rows should now show "all_match": true.

How to Automate Restarts With Reloader

If you don't want to restart deployments manually after every rotation, you can install Stakater Reloader. It watches an annotation on the Deployment and triggers a rolling restart automatically when the referenced Kubernetes Secret changes. New pods start with fresh env vars, while old pods drain cleanly. The repo's local deployment guide includes the install steps.

How to Choose Between External Secrets Operator and the CSI Driver

Two patterns dominate when it comes to pulling external secrets into Kubernetes: the External Secrets Operator and the Secrets Store CSI Driver.

Both get cloud secrets into pods, but they do it differently. Here's a plain comparison:

This lab uses the External Secrets Operator for two reasons. First, it produces a native Kubernetes Secret, which means your application and deployment patterns match standard Kubernetes workflows. Second, having both envFrom and a volume mount point to the same Secret makes the rotation behavior easy to observe side by side.

Use the CSI Driver when your security team prohibits storing secrets in etcd through a Kubernetes Secret. The driver mounts secret data directly into the pod file system without creating a Kubernetes Secret. The tradeoff is that you lose the native envFrom model.

How to Deploy the Pattern on Amazon Elastic Kubernetes Service

The local lab is ideal for learning. The Amazon Elastic Kubernetes Service path adds the production-like pieces: IAM role-based permissions for the operator, a load balancer for the app, and a full CI/CD workflow.

Step 1: Prepare Terraform and OpenID Connect Access

The repository includes a one-time setup guide for OpenID Connect-based access from GitHub Actions to AWS. Run these commands in the terraform/github-oidc folder.

cd terraform/github-oidc
terraform init
terraform plan -var="github_repo=YOUR_ORG/YOUR_REPO"
terraform apply -var="github_repo=YOUR_ORG/YOUR_REPO"
terraform output role_arn

Copy the role ARN from the output. You'll need it in the next step.

Step 2: Set the Required Environment Variable

The Amazon Elastic Kubernetes Service spin-up path needs your GitHub Actions role ARN so Terraform can grant the CI/CD runner access to the cluster.

To find your AWS account ID, run:

aws sts get-caller-identity --query Account --output text

Then set the variable, replacing ACCOUNT with the number that command returns.

export GITHUB_ACTIONS_ROLE_ARN=arn:aws:iam::ACCOUNT:role/your-github-oidc-role

Step 3: Run the Spin-Up Script for Amazon Elastic Kubernetes Service

bash spinup.sh --cluster eks

When the script finishes, it prints the application URL. Open that URL in a browser and confirm that you see the same secrets table you saw locally, with all keys showing Match ✓.

Step 4: Test Rotation on the Deployed App

After you confirm normal operation, run the rotation test the same way you did locally.

bash rotation/test-rotation.sh

Then use /secrets/compare on the Amazon Elastic Kubernetes Service load balancer URL to validate behavior in the cloud environment.

⚠️ Cost warning: Amazon Elastic Kubernetes Service runs at approximately $0.16 per hour. When you're done with the lab, run bash teardown.sh from the repo root to destroy all AWS resources and stop charges.

How to Configure GitHub Actions Without Stored AWS Credentials

The typical CI/CD setup stores AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub repository secrets. Those keys never rotate. Anyone with repo access can read them. When someone leaves the team, you have to revoke keys and update every workflow.

OpenID Connect eliminates that problem entirely.

How OpenID Connect Works for GitHub Actions

GitHub can issue a short-lived token for each workflow run. That token identifies the run: the repository, branch, and workflow name. You create an IAM role in AWS whose trust policy says: only accept requests that come from this specific GitHub repository and branch. The GitHub Actions runner exchanges that token for temporary AWS credentials via AssumeRoleWithWebIdentity. No long-lived keys are ever stored anywhere.

Step 1: Create the IAM Role With Terraform

The terraform/github-oidc folder creates the OpenID Connect provider and the IAM role for you. You already ran this in the Amazon Elastic Kubernetes Service setup above. The role ARN is the only value you need to store.

Step 2: Add the Role ARN to GitHub Repository Secrets

In your GitHub repository:

1. Go to Settings → Secrets and variables → Actions

2. Click New repository secret

3. Name it AWS_ROLE_ARN

4. Paste the role ARN from the Terraform output

That is the only secret you store. The role ARN isn't sensitive. It's an identifier, not a credential.

Step 3: Configure Terraform State

For CI/CD to work consistently across runs, Terraform needs a shared state backend. The lab stores Terraform state in an Amazon S3 bucket and uses an Amazon DynamoDB table for state locking. The Amazon Elastic Kubernetes Service deployment guide in the repo covers the backend setup in full.

Step 4: Push to Main and Let Workflows Run

After your first spin-up, every push to the main branch drives the CI/CD pipeline. The repo includes separate workflow files for Terraform infrastructure changes and application deployment changes. Once your application is reachable, use /secrets/compare to validate rotation behavior on the live environment.

How to Troubleshoot the Most Common Failures

Here's a shortlist of the most common symptoms and their fixes.

How to Inspect the Operator and the ExternalSecret

When something isn't syncing, start with these two commands.

# Check the ExternalSecret status
kubectl describe externalsecret app-db-secret -n default

# Check the ESO operator logs
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

The status conditions on the ExternalSecret resource will usually tell you exactly what failed.

How to Validate Rotation From the App Side

When you are debugging rotation, don't rely only on Kubernetes resource state. Use the /secrets/compare endpoint to see what the running application actually observes. The endpoint tells you whether env and volume match and whether rotation has been detected. That is the ground truth for your application's behavior.

Conclusion

You now have a complete secrets pipeline from AWS Secrets Manager into Kubernetes pods using Terraform and the External Secrets Operator. You ran the local lab, inspected the ExternalSecret and Deployment manifests, and validated that the application sees the right credentials.

You also tested secret rotation and observed the key behavior firsthand: mounted secret files update within the kubelet sync period, while environment variables stay stale until the pod restarts. That single observation explains a large class of production incidents.

Finally, you saw how the same design extends to Amazon Elastic Kubernetes Service with OpenID Connect-based CI/CD, and you have a troubleshooting checklist for the failures most teams hit.

The lab repository is at github.com/Osomudeya/k8s-secret-lab. If you ran the local lab, the natural next step is phases 4 and 5 from the repo's staged learning path: try the CSI driver path on Microk8s, then follow the EKS setup to see the same pipeline with a real CI/CD workflow and no credentials stored in GitHub. Both are documented in the repo and take less than 30 minutes each.

If this helped you, star the repo and share it with someone who is learning Kubernetes.

I send weekly breakdowns of real production incidents and how engineers actually fix them, not tutorials but real failures
→ Join the newsletter

An attacker had found it, deployed pods inside Tesla's cluster, and was using them to mine cryptocurrency – all on Tesla's AWS bill. The cluster had no authentication on the dashboard, no network restrictions on egress, and nothing monitoring for intrusion. Any one of those controls would have stopped the attack. None of them were in place.

This wasn't a sophisticated zero-day exploit. It was a misconfigured default.

Kubernetes ships with powerful security primitives. The problem is that almost none of them are enabled by default. A fresh cluster is deliberately permissive so it's easy to get started. That permissiveness is a feature in development. In production, it's a liability.

In this handbook, we'll work through the three most impactful security layers in Kubernetes. We'll start with Role-Based Access Control, which governs who can do what to which resources in the API. From there we'll move to pod runtime security, which locks down what containers can actually do once they're running on a node. Finally we'll deploy Falco, a syscall-level detection engine that watches for attacks in progress and alerts in real time.

By the end, you'll have a hardened cluster with working RBAC policies, enforced pod security standards, and live detection rules that fire when something suspicious happens.

Prerequisites

• kubectl installed and configured

• Docker Desktop or a Linux machine (to run kind)

• Basic Kubernetes familiarity – you know what a Pod, Deployment, and Namespace are

• No prior security experience needed

All demos run on a local kind cluster. Full YAML and setup scripts are in the companion GitHub repository.

Table of Contents

• The Kubernetes Threat Landscape

• What You'll Build

• Demo 1 — Run a Cluster Security Baseline with kube-bench

• How to Configure RBAC

  • The Four RBAC Objects

  • How to Discover Resources, Verbs, and API Groups

  • Roles and ClusterRoles

  • RoleBindings and ClusterRoleBindings

  • How to Use Service Accounts Safely

  • How to Audit Your RBAC Configuration

• Demo 2 — Build a Least-Privilege RBAC Policy for a CI Pipeline

• Demo 3 — Audit RBAC with rakkess and rbac-lookup

• How to Harden Pod Runtime Security

  • Pod Security Admission

  • How to Configure securityContext

  • OPA/Gatekeeper vs Kyverno

  • How to Detect Runtime Threats with Falco

• Demo 4 — Harden a Pod with securityContext

• Demo 5 — Deploy Falco and Write a Custom Detection Rule

• Cleanup

• Conclusion

The Kubernetes Threat Landscape

To understand what you're defending against, you need to understand where Kubernetes exposes attack surface. There are six main areas, and most production incidents trace back to at least one of them.

The API server is the front door to your cluster. Every kubectl command, every CI deploy, and every controller reconciliation loop sends requests here. Unauthenticated or over-privileged access to the API server is effectively game over: an attacker who can talk to it can create pods, read secrets, and modify workloads freely.

etcd is the key-value store where all cluster state lives, including your Secrets. Kubernetes Secrets are base64-encoded by default, not encrypted. Anyone with direct access to etcd can read every password, token, and certificate in the cluster without going through the API server at all.

The kubelet runs on each node and manages the pods assigned to it. If its API is reachable without authentication – which is the default on older clusters – an attacker can exec into any pod on that node and read its memory without ever touching the API server.

The container runtime is the layer that actually runs your containers. A container that escapes its isolation boundary lands directly in the host OS. A privileged container with hostPID: true can read the memory of every other process on the node, including other containers.

Your supply chain (base images, third-party dependencies, Helm charts, operators) is a potential entry point at every step. The XZ Utils backdoor discovered in 2024 showed how close a well-positioned supply chain attack can come to widespread infrastructure compromise.

Finally, the network: by default, every pod in a Kubernetes cluster can reach every other pod on any port. There are no internal firewalls between workloads unless you explicitly create them with NetworkPolicy.

Real-World Breaches

These three incidents are worth understanding before you write a single line of YAML. They're not theoretical – they're documented post-mortems from real production clusters.

The pattern across all three: not zero-day exploits, but misconfigured defaults and missing controls that should have been standard practice.

This article addresses the RBAC and pod security gaps directly.

What You'll Build

Before the first command, here is the security posture you'll have by the end of this article:

You'll start by running kube-bench to get a CIS Benchmark baseline – a concrete score showing where a default cluster stands before any hardening. From there you'll build a least-privilege RBAC policy for a CI pipeline service account and verify its permission boundaries, then audit the full cluster to confirm no over-privileged accounts exist.

On the pod security side, you'll enforce the restricted Pod Security Admission profile on your workload namespace and apply a hardened securityContext to a deployment: non-root user, read-only root filesystem, dropped capabilities, and seccomp profile. To close out, you'll deploy Falco in eBPF mode with a custom detection rule that fires when suspicious tools are run inside a container.

Start to finish, with a kind cluster already running, the demos take about 45–60 minutes.

Demo 1: Run a Cluster Security Baseline with kube-bench

Before hardening anything, it's a good idea to measure where you are. kube-bench runs the CIS Kubernetes Benchmark against your cluster and reports which checks pass and which fail. A baseline run gives you a concrete picture of your cluster's default security posture – and a reference point you can re-run after applying any hardening changes.

Step 1: Create a kind cluster

Save the following as kind-config.yaml:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --name k8s-security --config kind-config.yaml

Expected output:

Creating cluster "k8s-security" ...
 ✓ Ensuring node image (kindest/node:v1.29.0) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-k8s-security"

Step 2: Run kube-bench

kube-bench runs as a Job inside the cluster, mounting the host filesystem to inspect Kubernetes configuration files and processes:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl wait --for=condition=complete job/kube-bench --timeout=120s
kubectl logs job/kube-bench

The output is long. Scroll to the summary at the bottom:

== Summary master ==
0 checks PASS
11 checks FAIL
 9 checks WARN
 0 checks INFO

== Summary node ==
17 checks PASS
 2 checks FAIL
40 checks WARN
 0 checks INFO

A fresh kind cluster typically fails around 14 checks. Three of the most important failures explain why defaults are a problem:

Note: Some kube-bench findings are expected on kind because kind is a development tool, not a production-hardened environment. The important thing is to understand what each finding means and whether it applies to your target production setup.

Delete the Job when you're done:

kubectl delete job kube-bench

Now that you have a baseline, you know what you're starting from. The next step is to work through the most impactful control on that list: access control. RBAC governs every interaction with the Kubernetes API, and getting it right is the foundation everything else builds on.

How to Configure RBAC

Role-Based Access Control is the authorisation layer in Kubernetes. Every request that reaches the API server – from kubectl, from a pod, from a controller – is checked against RBAC rules after authentication succeeds. If there is no rule that explicitly allows the action, Kubernetes denies it.

The key word is "explicitly". RBAC in Kubernetes is additive only. There is no deny rule. You grant access by creating rules, and you remove access by deleting them. This makes the mental model clean: if a subject can do something, you gave it permission to do that thing.

A Brief Case Study: The Shopify Kubernetes Misconfiguration

In 2021, security researcher Silas Cutler discovered that a Shopify internal service exposed Kubernetes metadata through an SSRF vulnerability. The metadata included pod environment variables that contained secrets. The root cause was partly RBAC: the service's service account had broader cluster access than it needed, and there was no least-privilege review process.

Shopify paid a $25,000 bug bounty and fixed the issue. The lesson is straightforward: a service account should only have the permissions it needs to do its specific job. Nothing more.

This is the principle you'll apply in Demo 2.

The Four RBAC Objects

RBAC in Kubernetes is built from four API objects. Two define permissions, two bind those permissions to subjects:

A subject is a user, a group, or a service account. Users and groups come from your authentication layer – client certificates, OIDC tokens, or cloud provider identity. Service accounts are Kubernetes-native identities created for pods.

How to Discover Resources, Verbs, and API Groups

Before you can write a Role, you need to know three things: the resource name, the API group it belongs to, and the verbs it supports. You shouldn't have to guess any of them – kubectl can tell you everything.

List all available resources and their API groups

kubectl api-resources

Partial output:

NAME                    SHORTNAMES  APIVERSION                     NAMESPACED  KIND
bindings                            v1                             true        Binding
configmaps              cm          v1                             true        ConfigMap
endpoints               ep          v1                             true        Endpoints
events                  ev          v1                             true        Event
namespaces              ns          v1                             false       Namespace
nodes                   no          v1                             false       Node
pods                    po          v1                             true        Pod
secrets                             v1                             true        Secret
serviceaccounts         sa          v1                             true        ServiceAccount
services                svc         v1                             true        Service
deployments             deploy      apps/v1                        true        Deployment
replicasets             rs          apps/v1                        true        ReplicaSet
statefulsets            sts         apps/v1                        true        StatefulSet
cronjobs                cj          batch/v1                       true        CronJob
jobs                                batch/v1                       true        Job
ingresses               ing         networking.k8s.io/v1           true        Ingress
networkpolicies         netpol      networking.k8s.io/v1           true        NetworkPolicy
clusterroles                        rbac.authorization.k8s.io/v1   false       ClusterRole
roles                               rbac.authorization.k8s.io/v1   true        Role

The APIVERSION column is what you put in apiGroups. Strip the version suffix and use only the group part:

The NAMESPACED column tells you whether to use a Role (namespaced resources) or a ClusterRole (non-namespaced resources like nodes).

Filter by API group

If you want to see only resources in a specific group, for example, everything in apps:

kubectl api-resources --api-group=apps

NAME                  SHORTNAMES  APIVERSION  NAMESPACED  KIND
controllerrevisions               apps/v1     true        ControllerRevision
daemonsets            ds          apps/v1     true        DaemonSet
deployments           deploy      apps/v1     true        Deployment
replicasets           rs          apps/v1     true        ReplicaSet
statefulsets          sts         apps/v1     true        StatefulSet

List all verbs for a specific resource

Each resource supports a different set of verbs. To see exactly which verbs a resource supports, use kubectl api-resources with -o wide and look at the VERBS column:

kubectl api-resources -o wide | grep -E "^NAME|^pods "

NAME  SHORTNAMES  APIVERSION  NAMESPACED  KIND  VERBS
pods  po          v1          true        Pod   create,delete,deletecollection,get,list,patch,update,watch

Or explain the resource directly:

kubectl explain pod --api-version=v1 | head -10

The full set of verbs Kubernetes supports in RBAC rules is:

Important: get and list are separate verbs. Granting list on secrets lets a subject enumerate every secret name and value in a namespace, even if you didn't also grant get. Always think about both when working with sensitive resources like secrets, serviceaccounts, and configmaps.

Look up a resource's group with kubectl explain

If you already know the resource name but aren't sure of its group, kubectl explain tells you:

kubectl explain deployment

GROUP:      apps
KIND:       Deployment
VERSION:    v1
...

kubectl explain ingress

GROUP:      networking.k8s.io
KIND:       Ingress
VERSION:    v1
...

This is the fastest way to look up the apiGroups value for any resource when writing a Role.

A complete lookup workflow

Here is the practical workflow when writing a new Role from scratch:

# 1. Find the resource name and API group
kubectl api-resources | grep deployment

# Output:
# deployments   deploy   apps/v1   true   Deployment

# 2. Find the verbs it supports
kubectl api-resources -o wide | grep deployment

# Output:
# deployments   deploy   apps/v1   true   Deployment   create,delete,...,get,list,patch,update,watch

# 3. Write the Role using the group (strip the version) and the verbs you need

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployment-reader
  namespace: staging
rules:
  - apiGroups: ["apps"]       # from: apps/v1 → strip /v1
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]

With this workflow, you never have to guess an API group or verb. You look it up, then write the minimal rule you need.

Roles and ClusterRoles

A Role defines which verbs are allowed on which resources. Here is a Role that grants read-only access to Pods and ConfigMaps inside the staging namespace:

# role-ci-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-reader
  namespace: staging
rules:
  - apiGroups: [""]          # "" = the core API group (Pods, Services, Secrets, ConfigMaps)
    resources: ["pods", "configmaps"]
    verbs: ["get", "list", "watch"]

The apiGroups field tells Kubernetes which API group owns the resource. The core group uses an empty string "". Apps-level resources like Deployments use "apps". Custom resources use their own group, such as "networking.k8s.io".

A ClusterRole is structurally identical but omits the namespace and can reference cluster-scoped resources like Nodes and PersistentVolumes:

# clusterrole-node-reader.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader    # no namespace field
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]

When to use which:

Use a Role when the permission is specific to one namespace. A compromised service account can only affect that namespace: the blast radius is contained. Use a ClusterRole when you need access to cluster-scoped resources, or when you want a reusable permission template that multiple namespaces can share.

A common mistake is reaching for a ClusterRole "just to be safe" because it's easier to configure. Namespace-scoped Roles are almost always the right default.

RoleBindings and ClusterRoleBindings

A Role by itself does nothing. You need a binding to attach it to a subject. Here is a RoleBinding that grants the ci-reader Role to the ci-pipeline service account:

# rolebinding-ci.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-reader-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline       # the service account name
    namespace: staging      # the namespace the SA lives in
roleRef:
  kind: Role
  name: ci-reader           # must match the Role name exactly
  apiGroup: rbac.authorization.k8s.io

There is a useful pattern worth knowing: you can bind a ClusterRole using a RoleBinding. This creates namespace-scoped access using a reusable permission template. The ClusterRole defines the rules, while the RoleBinding constrains those rules to a single namespace.

# RoleBinding referencing a ClusterRole — scoped to one namespace only
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: view-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline
    namespace: staging
roleRef:
  kind: ClusterRole          # ClusterRole, but bound to one namespace via RoleBinding
  name: view                 # Kubernetes built-in ClusterRole: read-only access to most resources
  apiGroup: rbac.authorization.k8s.io

Kubernetes ships with several useful built-in ClusterRoles: view (read-only access to most resources), edit (read/write to most resources), admin (full namespace admin), and cluster-admin (full cluster admin). Use them rather than reinventing them.

How to Use Service Accounts Safely

Every pod in Kubernetes runs as a service account. If you don't specify one, Kubernetes uses the default service account in that namespace.

The default service account starts with no permissions – but it still has a token automatically mounted into every pod at /var/run/secrets/kubernetes.io/serviceaccount/token. This means every container in your cluster can authenticate to the API server by default, even if it has nothing useful to do there.

The single most impactful change you can make is to disable this automatic token mounting on service accounts that don't need API access:

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
automountServiceAccountToken: false   # no token mounted into pods by default

You can also control it at the pod level:

spec:
  automountServiceAccountToken: false   # override at pod level
  serviceAccountName: my-app
  containers:
    - name: app
      image: my-app:1.0

The cluster-admin anti-pattern:

Never bind cluster-admin to a service account that runs in a pod. cluster-admin grants full read/write access to every resource in the cluster. An attacker who compromises a pod running as cluster-admin owns your cluster completely.

You will see this in Helm charts and tutorials because it "makes things work". It works because it disables the entire authorisation layer. That is not a solution – it's a ticking clock.

The Capital One breach is a direct example of this pattern at the cloud layer: an EC2 instance role had permissions far beyond what the application needed. The SSRF vulnerability was the initial foothold. The over-privileged role was what turned a minor bug into a $80 million fine.

How to Audit Your RBAC Configuration

The kubectl auth can-i command lets you check permissions for any subject. Use --as to impersonate a service account:

SA="system:serviceaccount:staging:ci-pipeline"

# These should return 'yes'
kubectl auth can-i list pods        --namespace staging --as $SA
kubectl auth can-i get  configmaps  --namespace staging --as $SA

# These should return 'no'
kubectl auth can-i delete pods      --namespace staging --as $SA
kubectl auth can-i get  secrets     --namespace staging --as $SA
kubectl auth can-i list pods        --namespace production --as $SA

To list every permission a subject has in a namespace:

kubectl auth can-i --list \
  --namespace staging \
  --as system:serviceaccount:staging:ci-pipeline

For a visual matrix across the whole cluster, install rakkess (part of krew):

kubectl krew install access-matrix

# Permission matrix for all service accounts in staging
kubectl access-matrix --namespace staging

Example output:

NAME          GET  LIST  WATCH  CREATE  UPDATE  PATCH  DELETE
ci-pipeline    ✓    ✓     ✓      ✗       ✗       ✗      ✗
default        ✗    ✗     ✗      ✗       ✗       ✗      ✗
monitoring     ✓    ✓     ✓      ✗       ✗       ✗      ✗

If you see ✓ in the CREATE, UPDATE, PATCH, or DELETE columns for a service account that should only read, that's a finding that needs remediation.

⚠️ The wildcard danger: The most dangerous RBAC configuration is a wildcard on all three dimensions:

apiGroups: [""] 
resources: [""] 
verbs: ["*"]

This is functionally identical to cluster-admin. You will find it in Helm charts for controllers installed with "convenience" permissions. Always audit third-party RBAC before installing operators into a production cluster.

Demo 2 – Build a Least-Privilege RBAC Policy for a CI Pipeline

In this demo, you'll create a service account for a CI pipeline that can list pods and read configmaps in the staging namespace – and nothing else.

Step 1: Create the namespace and service account

kubectl create namespace staging

# ci-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-pipeline
  namespace: staging
automountServiceAccountToken: false

kubectl apply -f ci-serviceaccount.yaml

Step 2: Create the Role

# ci-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-reader
  namespace: staging
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]

kubectl apply -f ci-role.yaml

Step 3: Bind the Role to the service account

# ci-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-reader-binding
  namespace: staging
subjects:
  - kind: ServiceAccount
    name: ci-pipeline
    namespace: staging
roleRef:
  kind: Role
  name: ci-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f ci-rolebinding.yaml

Step 4: Test allowed operations

SA="system:serviceaccount:staging:ci-pipeline"

kubectl auth can-i list pods       --namespace staging     --as $SA   # yes
kubectl auth can-i get  pods       --namespace staging     --as $SA   # yes
kubectl auth can-i list configmaps --namespace staging     --as $SA   # yes

Step 5: Test denied operations

kubectl auth can-i delete pods       --namespace staging     --as $SA   # no
kubectl auth can-i get  secrets      --namespace staging     --as $SA   # no
kubectl auth can-i list pods         --namespace production  --as $SA   # no
kubectl auth can-i create deployments --namespace staging    --as $SA   # no

All four should return no. Notice the third test: even if there were a matching Role in the staging namespace, the service account cannot access production. A RoleBinding cannot cross namespace boundaries, this is by design.

Writing a least-privilege policy for a service account you control is the easy part. The harder part is auditing what already exists in a cluster. That's what Demo 3 covers.

Demo 3 – Audit RBAC with rakkess and rbac-lookup

Now you'll scan the full cluster to surface any accounts with more permissions than they need.

Step 1: Install the tools

kubectl krew install access-matrix
kubectl krew install rbac-lookup

Step 2: Run rakkess across the cluster

# All service accounts in kube-system
kubectl access-matrix --namespace kube-system

# All ServiceAccounts cluster-wide
kubectl access-matrix

Step 3: Find all cluster-admin bindings

There are two ways subjects get cluster-admin access: via a ClusterRoleBinding (cluster-wide), or via a RoleBinding that references the cluster-admin ClusterRole (namespace-scoped, still dangerous). Check both:

# Find ClusterRoleBindings that grant cluster-admin
kubectl rbac-lookup cluster-admin --kind ClusterRole --output wide

On a fresh kind cluster this returns:

No RBAC Bindings found

That is the correct and expected result. A default kind cluster doesn't create any ClusterRoleBindings to cluster-admin. The role exists, but nothing is bound to it at the cluster level by default. If you see entries here in your production cluster, each one is a finding worth investigating.

To find who has cluster-level admin access through other means, query the bindings directly:

# Find all ClusterRoleBindings and the subjects they grant
kubectl get clusterrolebindings -o wide

NAME                                                   ROLE                                                                       AGE   USERS                         GROUPS                         SERVICEACCOUNTS
cluster-admin                                          ClusterRole/cluster-admin                                                  10d   system:masters
system:kube-controller-manager                         ClusterRole/system:kube-controller-manager                                 10d
system:kube-scheduler                                  ClusterRole/system:kube-scheduler                                          10d
system:node                                            ClusterRole/system:node                                                    10d
...

The cluster-admin ClusterRoleBinding grants access to the system:masters group – the group your kubeconfig certificate belongs to. This is expected. Every other binding in this list is worth reviewing to understand what it grants and why.

What to look for: Any binding where the SERVICEACCOUNTS column is populated with an application service account (not a system: prefixed one) is a potential over-privilege finding. Application pods should never need cluster-admin.

Step 4: Verify the ci-pipeline service account

kubectl rbac-lookup ci-pipeline --kind ServiceAccount --output wide

Expected output:

SUBJECT                               SCOPE     ROLE             SOURCE
ServiceAccount/staging:ci-pipeline    staging   Role/ci-reader   RoleBinding/ci-reader-binding

The format is /<role-name> <binding-kind>/<binding-name>. This tells you:

• The service account is bound to the ci-reader Role

• The binding is a RoleBinding named ci-reader-binding

• There is no namespace prefix on the role name because it is a namespaced Role, not a ClusterRole

If the output showed ClusterRole/something here, that would be a finding. It would mean the service account has cluster-wide permissions, not namespace-scoped ones.

rbac-lookup vs kubectl get: rbac-lookup gives you a subject-centric view: "what does this account have access to?" kubectl get rolebindings,clusterrolebindings -A gives you a binding-centric view: "what bindings exist in the cluster?" Use both. rbac-lookup is faster for auditing a specific service account, while the kubectl get approach is better for a full cluster inventory.

With RBAC locked down, the API server is protected. But RBAC says nothing about what a container can do once it's running. That's a separate layer entirely.

How to Harden Pod Runtime Security

RBAC controls who can talk to the Kubernetes API. Pod security controls what containers can do once they're running on a node. These are different threat vectors: RBAC protects the control plane, pod security protects the data plane.

A container that runs as root with no capability restrictions can, if compromised, write backdoors to the host filesystem, load kernel modules, read the memory of other processes if hostPID: true is set, and in some configurations escape the container entirely. Pod security closes these doors before an attacker can open them.

A Case Study: The Hildegard Malware Campaign

In early 2021, Palo Alto's Unit 42 research team documented a cryptomining malware campaign called Hildegard that specifically targeted Kubernetes clusters. The attack chain was:

1. Find a cluster with the kubelet API exposed without authentication

2. Deploy a privileged pod with hostPID: true

3. Use the privileged pod to read credentials from other containers' memory

4. Establish persistence by writing to the host filesystem

Steps 3 and 4 would have been impossible if the pods in the cluster had been running with readOnlyRootFilesystem: true, dropped capabilities, and no hostPID. The attacker had the initial foothold. Pod security would have contained the blast radius.

Pod Security Admission

Pod Security Admission (PSA) is the built-in admission controller that enforces pod security standards at the namespace level. It replaced PodSecurityPolicy in Kubernetes 1.25.

Migrating from PSP? If you're on Kubernetes < 1.25, you may still be using PodSecurityPolicy, which was removed in 1.25. The migration path is: enable PSA in audit mode first to identify violations, fix them workload by workload, then switch to enforce. For policies PSA cannot express, add Kyverno alongside it.

PSA defines three profiles:

And three enforcement modes:

The migration path: start with audit and warn to identify violations, fix them, then switch to enforce. The two modes can run simultaneously.

Apply them as namespace labels:

# namespace-staging.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: staging
  labels:
    # Start here: audit and warn simultaneously
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Once violations are resolved, add enforce:

kubectl label namespace staging \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  --overwrite

Note: don't use --overwrite here. Without it, if enforce is already set to a different value the command will error – which is exactly what you want. You should see:

namespace/staging labeled

If you see namespace/staging not labeled, it means enforce=restricted and enforce-version=latest were already set to those exact values. Confirm enforcement is active:

kubectl get namespace staging --show-labels

Look for pod-security.kubernetes.io/enforce=restricted in the output. If it's there, enforcement is active.

How to Configure securityContext

A securityContext defines the privilege and access control settings for a pod or container. These are the seven fields you should configure on every production workload:

The annotated YAML below shows all seven in context:

# secure-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: staging
spec:
  replicas: 2
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      securityContext:
        runAsNonRoot: true         # container must run as a non-root user
        runAsUser: 10001           # explicit UID — don't rely on the image's default
        runAsGroup: 10001          # explicit GID
        fsGroup: 10001             # volumes are owned by this group
        seccompProfile:
          type: RuntimeDefault     # use the container runtime's default seccomp profile
      automountServiceAccountToken: false
      containers:
        - name: app
          image: nginx:1.25-alpine
          securityContext:
            allowPrivilegeEscalation: false   # block setuid and sudo inside the container
            readOnlyRootFilesystem: true      # the single highest-impact setting
            capabilities:
              drop:
                - ALL                         # drop every Linux capability
              add: []                         # add back only what is explicitly needed
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-run
              mountPath: /var/run
      volumes:
        # nginx needs writable directories — provide them as emptyDir volumes
        - name: tmp
          emptyDir: {}
        - name: nginx-cache
          emptyDir: {}
        - name: nginx-run
          emptyDir: {}

Why readOnlyRootFilesystem: true is the most important setting:

Most post-exploitation techniques require writing to the filesystem. Dropping a backdoor, modifying a binary, writing a cron job, or installing a keylogger all require a writable filesystem. Set readOnlyRootFilesystem: true and every one of these techniques is blocked.

The downside is that many applications write to directories like /tmp or /var/cache. The fix is to mount emptyDir volumes at those specific paths, as shown above. The rest of the filesystem stays read-only.

What each field prevents:

OPA/Gatekeeper vs Kyverno

PSA covers the fundamentals. But you'll eventually need policies that PSA cannot express: all images must come from your private registry, all pods must have resource limits, no container may use the latest tag. For these, you need a policy engine.

Two mature options exist:

If you're starting fresh, Kyverno gets you to working policies faster. Here is a Kyverno policy that blocks images from outside your trusted registry:

# kyverno-registry-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-registries
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "Images must come from registry.corp.internal/"
        pattern:
          spec:
            containers:
              - image: "registry.corp.internal/*"

How to Detect Runtime Threats with Falco

PSA and securityContext are preventive controls: they block known-bad configurations before pods start. Falco is a detective control. It watches what containers do while they're running and alerts when something looks wrong.

Falco operates at the syscall level using eBPF. It attaches to the Linux kernel and intercepts every system call made by every container on the node – file opens, network connections, process spawns, privilege escalations. It does this without modifying containers, without injecting sidecars, and with minimal overhead.

What Falco detects out of the box:

Falco's default ruleset covers the most common attack patterns. It fires when a shell is opened inside a running container, whether that's a kubectl exec session or a reverse shell from an exploit.

It watches for reads on sensitive files like /etc/shadow, /etc/kubernetes/admin.conf, and /root/.ssh/. It catches the dropper pattern: a binary written to disk and immediately executed. It detects outbound connections to known malicious IPs, writes to /proc or /sys that suggest kernel manipulation, and package managers like apt, yum, or pip being run inside containers that have no business installing software.

Each of these is a rule in Falco's default ruleset. You can extend it with custom rules for your specific workloads – which is exactly what you'll do in Demo 5. But first let's harden the Pod.

Demo 4 – Harden a Pod with securityContext

In this demo, you'll start with a default nginx deployment, observe the PSA violations it triggers, harden it step by step, and confirm it passes under the restricted profile.

Step 1: Apply PSA labels in audit mode

kubectl label namespace staging \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

Step 2: Deploy insecure nginx and observe the warnings

# insecure-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-insecure
  namespace: staging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-insecure
  template:
    metadata:
      labels:
        app: nginx-insecure
    spec:
      containers:
        - name: nginx
          image: nginx:1.25-alpine

kubectl apply -f insecure-nginx.yaml

Expected output (PSA warns but still creates the deployment in warn mode):

Warning: would violate PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false (container "nginx" must set
    securityContext.allowPrivilegeEscalation=false)
  unrestricted capabilities (container "nginx" must set
    securityContext.capabilities.drop=["ALL"])
  runAsNonRoot != true (pod or container "nginx" must set
    securityContext.runAsNonRoot=true)
  seccompProfile not set (pod or container "nginx" must set
    securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx-insecure created

Four violations. Every one of them is a real security gap. But the pod was still created "deployment.apps/nginx-insecure created"

Step 3: Deploy the hardened version

kubectl apply -f secure-deployment.yaml   # the YAML from the securityContext section above

No warnings this time.

Step 4: Switch the namespace to enforce

kubectl label namespace staging \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest

Expected output:

namespace/staging labeled

This is the moment enforcement becomes active. Any new pod that violates the restricted profile will be rejected from this point on.

Step 5: Confirm insecure deployments are now rejected

kubectl delete deployment nginx-insecure -n staging
kubectl apply -f insecure-nginx.yaml

Expected output:

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false ...
deployment.apps/nginx-insecure created

The Deployment object is created. PSA enforces at the pod level, not the Deployment level. The Deployment and its ReplicaSet exist, but every attempt to create a pod is rejected. Check the ReplicaSet:

kubectl get replicaset -n staging -l app=nginx-insecure

NAME                       DESIRED   CURRENT   READY   AGE
nginx-insecure-b668d867b   1         0         0       30s

DESIRED=1 but CURRENT=0. The ReplicaSet cannot create any pods because they're rejected at admission. Describe the ReplicaSet to see the rejection events:

kubectl describe replicaset -n staging -l app=nginx-insecure

Warning  FailedCreate  ReplicaSet "nginx-insecure-b668d867b" create Pod
  "nginx-insecure-xxx" failed: pods is forbidden: violates PodSecurity
  "restricted:latest": allowPrivilegeEscalation != false, unrestricted
  capabilities, runAsNonRoot != true, seccompProfile not set

The hardened deployment continues running with its pods intact. The insecure one has zero pods and never will. This is exactly how PSA is supposed to work.

Step 6: Score the hardened pod with kube-score

kube-score is a static analysis tool that scores Kubernetes manifests against security and reliability best practices:

# macOS
brew install kube-score
# Linux: https://github.com/zegl/kube-score/releases

kube-score score secure-deployment.yaml -v

Expected output (abridged):

apps/v1/Deployment secure-app in staging 
  path=secure-deployment.yaml
    [OK] Stable version
    [OK] Label values
    [CRITICAL] Container Resources
        · app -> CPU limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.cpu
        · app -> Memory limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.memory
        · app -> CPU request is not set
            Resource requests are recommended to make sure that the application can start and run without crashing. Set resources.requests.cpu
        · app -> Memory request is not set
            Resource requests are recommended to make sure that the application can start and run without crashing. Set resources.requests.memory
    [CRITICAL] Container Image Pull Policy
        · app -> ImagePullPolicy is not set to Always
            It's recommended to always set the ImagePullPolicy to Always, to make sure that the imagePullSecrets are always correct, and to always get the image you want.
    [OK] Pod Probes Identical
    [CRITICAL] Container Ephemeral Storage Request and Limit
        · app -> Ephemeral Storage limit is not set
            Resource limits are recommended to avoid resource DDOS. Set resources.limits.ephemeral-storage
        · app -> Ephemeral Storage request is not set
            Resource requests are recommended to make sure the application can start and run without crashing. Set resource.requests.ephemeral-storage
    [OK] Environment Variable Key Duplication
    [OK] Container Security Context Privileged
    [OK] Pod Topology Spread Constraints
        · Pod Topology Spread Constraints
            No Pod Topology Spread Constraints set, kube-scheduler defaults assumed
    [OK] Container Image Tag
    [CRITICAL] Pod NetworkPolicy
        · The pod does not have a matching NetworkPolicy
            Create a NetworkPolicy that targets this pod to control who/what can communicate with this pod. Note, this feature needs to be supported by the CNI implementation used in the Kubernetes cluster to have an effect.
    [OK] Container Security Context User Group ID
    [OK] Container Security Context ReadOnlyRootFilesystem
    [CRITICAL] Deployment has PodDisruptionBudget
        · No matching PodDisruptionBudget was found
            It's recommended to define a PodDisruptionBudget to avoid unexpected downtime during Kubernetes maintenance operations, such as when draining a node.
    [WARNING] Deployment has host PodAntiAffinity
        · Deployment does not have a host podAntiAffinity set
            It's recommended to set a podAntiAffinity that stops multiple pods from a deployment from being scheduled on the same node. This increases availability in case the node becomes unavailable.
    [OK] Deployment Pod Selector labels match template metadata labels

Notice there are no security context violations: securityContext, readOnlyRootFilesystem, seccompProfile, and runAsNonRoot all pass. The remaining findings are about resource management (CPU/memory limits, ephemeral storage), availability (PodDisruptionBudget, anti-affinity), and network policy – not security context hardening. Those are important for production readiness, but they're a separate concern from the pod security hardening we did here.

You now have a pod that PSA accepts and kube-score validates. The next step is to add a detection layer – something that watches what the pod does at runtime, not just how it was configured at admission.

Demo 5 – Deploy Falco and Write a Custom Detection Rule

Now, you'll deploy Falco in eBPF mode, trigger a default alert, then extend Falco with a custom rule that catches curl and wget being run inside containers.

Step 1: Install Falco via Helm

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set tty=true \
  --wait

Confirm Falco is running on every node:

kubectl get pods -n falco

NAME           READY   STATUS    RESTARTS   AGE
falco-x8k2p    1/1     Running   0          45s
falco-m9nqr    1/1     Running   0          45s
falco-j4tpw    1/1     Running   0          45s

One pod per node. Falco runs as a DaemonSet because it needs to monitor syscalls on every node independently.

Step 2: Trigger a default alert

Open a second terminal and stream the Falco logs:

# Terminal 2 — watch for alerts
kubectl logs -n falco -l app.kubernetes.io/name=falco -f --max-log-requests 3

In your first terminal, exec into the secure-app pod:

# Terminal 1 — trigger the shell detection
POD=$(kubectl get pod -n staging -l app=secure-app \
  -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD -n staging -- sh

Within a second, Terminal 2 shows:

2024-03-15T14:23:41.456Z: Notice A shell was spawned in a container with an attached terminal
  (user=root user_loginuid=-1 k8s.ns=staging k8s.pod=secure-app-7d9f8b-xxx
   container=app shell=sh parent=runc cmdline=sh terminal=34816)
  rule=Terminal shell in container  priority=NOTICE
  tags=[container, shell, mitre_execution]

This is Falco's built-in Terminal shell in container rule firing. It detected the kubectl exec session the moment you ran it.

Step 3: Write a custom rule

The built-in rules are comprehensive, but every production environment has workloads with unique behaviour. Here is a custom rule that alerts when curl or wget is executed inside any container:

# custom-rules.yaml
customRules:
  custom-rules.yaml: |-
    - rule: Suspicious network tool in container
      desc: >
        Detects execution of curl or wget inside a running container.
        These tools are commonly used for data exfiltration, downloading
        attacker payloads, or reaching command-and-control servers.
        Production containers should not be making ad-hoc HTTP requests.
      condition: >
        spawned_process
        and container
        and proc.name in (curl, wget)
      output: >
        Network tool executed in container
        (user=%user.name tool=%proc.name cmd=%proc.cmdline
         pod=%k8s.pod.name ns=%k8s.ns.name image=%container.image)
      priority: WARNING
      tags: [network, exfiltration, custom]

Apply it by upgrading the Helm release:

 helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set driver.kind=modern_ebpf \
  --set tty=true \
  -f custom-rules.yaml

Good, it deployed. Now wait for pods to be ready and test your custom rule:

Step 4: Test the custom rule

# Terminal 1 — run curl inside the container
kubectl exec -it $POD -n staging -- sh -c 'curl https://example.com'

Terminal 2 immediately shows:

2024-03-15T14:31:07.812Z: Warning Network tool executed in container
  (user=root tool=curl cmd=curl https://example.com
   pod=secure-app-7d9f8b-xxx ns=staging image=nginx:1.25-alpine)
  rule=Suspicious network tool in container  priority=WARNING
  tags=[network, exfiltration, custom]

Step 5: Route alerts to Slack with Falcosidekick

Streaming logs is useful during development. In production, you need alerts routed to your alerting pipeline. Falcosidekick handles this with support for Slack, PagerDuty, Datadog, Elasticsearch, and over 50 other outputs:

# falcosidekick-values.yaml
config:
  slack:
    webhookurl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
    minimumpriority: "warning"
    messageformat: >
      [{{.Priority}}] {{.Rule}} |
      pod: {{.OutputFields.k8s.pod.name}} |
      ns: {{.OutputFields.k8s.ns.name}} |
      image: {{.OutputFields.container.image}}

helm install falcosidekick falcosecurity/falcosidekick \
  --namespace falco \
  -f falcosidekick-values.yaml

Tuning Falco for production: A fresh Falco deployment will generate false positives, especially in the first week. Your job is to tune rules to match your workloads' normal behaviour, not to respond to every alert.

Here's the workflow: deploy in staging → identify false positives → add except conditions to rules → validate the false positive rate is low → enable in production with alerting.

Cleanup

To remove everything created in this article:

# Delete the staging namespace and everything in it
kubectl delete namespace staging
 
# Delete Falco and Falcosidekick
helm uninstall falco -n falco
helm uninstall falcosidekick -n falco
kubectl delete namespace falco
 
# Delete the kind cluster entirely
kind delete cluster --name k8s-security

Conclusion

In this handbook, you secured a Kubernetes cluster across three layers: RBAC, pod runtime security, and runtime threat detection.

You built a least-privilege service account, enforced the restricted Pod Security Admission profile, hardened pods with securityContext, deployed Falco for syscall-level detection, and wrote a custom rule to catch suspicious tools inside containers.

Each layer maps to a real-world breach – Tesla, Capital One, Hildegard – showing how these controls would have contained the damage. Run kube-bench again to measure the improvement.

All YAML manifests, Helm values, and setup scripts from this article are available in the companion GitHub repository.

I’ve spent more nights than I care to admit staring at a terminal, trying to figure out why a staging environment suddenly "broke" even though no one supposedly touched it.

We’ve all been there, right? a manual edit here, a quick hotfix there, and suddenly your Git repository is no longer a Source of Truth, it’s a historical document of what used to be running.

Without a reliable strategy, Kubernetes deployments quickly descend into a mess of drift, painful rollbacks, and non-existent audit trails. I learned the hard way that simply storing manifests in Git isn't enough. If your cluster isn't actively listening to your code, you're still working with a gap.

GitOps closes that gap. It turns your cluster into a mirror of your repository. If it isn't in Git, it doesn't exist.

In this tutorial, you aren't just going to read about the theory. You’re going to implement a "Zero-Touch" deployment loop from scratch. We’ll use Argo CD, GitHub Actions, and the Argo CD Image Updater to build a system that builds, tags, and deploys your code the second you hit git push.

Table of Contents

1. Prerequisites

2. What GitOps Really Means

3. What is Argo CD and How Does it Implement GitOps?

4. Preparing the Application Source Code and Repo Structure

5. Automating Image Builds with GitHub Actions

6. How to Install and Access Argo CD

7. Understanding the Argo CD Application

8. Deploying the Application Manifest

9. Automating Updates with Argo CD Image Updater

10. Conclusion

Prerequisites

Before you begin, make sure you have the following ready in your environment:

• A GitHub repository: You'll need a repository (for example, my-gitops-demo) to serve as your Single Source of Truth. If you're following this tutorial from scratch, start with an empty repo.

• A DockerHub account: This will act as your Container Registry. You’ll need this to build, push, and store the Docker images that GitHub Actions creates.

• A running Kubernetes cluster: You can use a local solution like Minikube or Kind, or a cloud-managed service like Amazon EKS or GKE.

• Kubernetes tooling: Ensure kubectl is installed and configured to communicate with your cluster.

• Fundamental K8s knowledge: You should be comfortable with basic Kubernetes concepts like Pods, Deployments, and Services.

Note for Readers with Existing Projects

If you already have a project and want to migrate it to this GitOps workflow, you don't need to start over. You can adapt your existing repository by following these three steps:

1. Standardize your manifests: Move all your existing Kubernetes YAML files into a dedicated Kubernetes-manifest/ directory at the root of your project.

2. Containerize your services: Ensure every service you intend to deploy has a Dockerfile in its respective subdirectory (for example, /main-api/Dockerfile).

3. Prepare for automation: Be ready to replace any manual kubectl apply steps in your current CI pipeline with the automated tagging strategy we’ll implement in the next sections.

What GitOps Really Means

At its core, GitOps is an operational framework that uses Git as the single source of truth for your infrastructure and applications. In a traditional setup, you might run kubectl apply -f deployment.yaml from your laptop. This makes it impossible to track who changed what, leading to "snowflake" clusters that no one can reproduce.

GitOps enforces four key principles:

1. Declarative: You describe the desired state (for example, "3 replicas of Nginx"), not the commands to get there.

2. Versioned and immutable: Your entire state is in Git. If a deployment fails, you git revert to a previous known-good state.

3. Pulled automatically: A software agent (Argo CD) pulls the state from Git.

4. Continuously reconciled: The system constantly fixes "drift." If a developer manually changes a service in the cluster, Argo CD will overwrite it to match Git.

What is Argo CD and How Does it Implement GitOps

Before we dive into the setup, let’s define the tool we'll be working with.

Argo CD is a declarative, GitOps' continuous delivery engine built specifically for Kubernetes. As a graduated project of the Cloud Native Computing Foundation (CNCF), it has become the industry standard for managing modern infrastructure.

Think of Argo CD as a persistent watchdog that lives inside your cluster. To understand why it's so powerful, we have to look at how it differs from traditional CI/CD tools like Jenkins or GitHub Actions.

The "Push" vs. "Pull" Model

Traditional tools like the one I mentioned above use a "Push" model. In this setup, an external pipeline sends commands (like kubectl apply) into your cluster. This is risky because you must store sensitive cluster administrative keys inside your external CI tool. If your CI tool is compromised, your cluster is, too.

Argo CD flips this script using a "Pull" model:

• The bridge: It sits between your Git repo (the "Desired State") and your cluster (the "Live State").

• Continuous monitoring: It watches your Git repo 24/7. The moment it detects a new commit, it "pulls" that change and applies it from inside the cluster.

• Self-healing: If someone manually changes a setting in the cluster (known as "drift"), Argo CD detects the discrepancy and automatically overwrites it to match what is written in Git.

This approach is not only more secure, since no cluster credentials ever leave the environment, but it also ensures that your infrastructure is a perfect, predictable mirror of your code.

Preparing the Application Source Code

Before we automate the build, we need actual code in our repository. We'll create two simple microservices: a Main API and an Auxiliary Service.

Repo Structure

Ensure your repository follows this structure exactly. Consistency in naming is vital for the automation to find your files.

GITOPS-ARGOCD-DEMO/
├── .github/workflows/main.yml
├── auxiliary-service/
│   └── Dockerfile
├── main-api/
│   └── Dockerfile
├── Kubernetes-manifest/
│   ├── aux-api.yaml
│   ├── kustomization.yaml
│   └── main-api.yaml
├── application.yaml
└── image-updater.yaml

Create the Dockerfiles

In each service folder, create a simple Dockerfile so our pipeline has something to build.

main-api/Dockerfile

FROM nginx:alpine
RUN echo "<h1>Main API - Version 1.0</h1>" > /usr/share/nginx/html/index.html
EXPOSE 80

auxiliary-service/Dockerfile

FROM nginx:alpine
RUN echo "<h1>Auxiliary Service - Version 1.0</h1>" > /usr/share/nginx/html/index.html
EXPOSE 80

Automating Image Builds with GitHub Actions

In a professional GitOps workflow, your Kubernetes manifests and your application source code often live in the same repository (or linked ones). While Argo CD handles the deployment, you still need a way to turn your code into Docker images. This is where Continuous Integration (CI) comes in.

I have included a GitHub Actions workflow in this demo to automate this. Every time you push code to the main branch, this pipeline builds your images and pushes them to DockerHub.

The CI Pipeline Workflow

Create a file at .github/workflows/main.yml and add the following:

name: Build and Push Image to DockerHub

on:
  push:
    branches:
      - main
    # Skip builds for image updater commits
    paths-ignore:
      - 'Kubernetes-manifest/**'

jobs:
  docker_build:
    name: Build & Push ${{ matrix.service }}
    environment: argocd-demo
    runs-on: ubuntu-latest
    permissions:
      contents: write

    strategy:
      matrix:
        include:
          - service: aux-service
            dockerfile: auxiliary-service/Dockerfile
          - service: main-service
            dockerfile: main-api/Dockerfile

    env:
      DOCKER_USER: ${{ secrets.DOCKERHUB_USERNAME }}
      RUN_TAG: ${{ github.run_number }}

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ env.DOCKER_USER }}
          password: ${{ secrets.DOCKERHUB_PASSWORD }}

      - name: Build and Push ${{ matrix.service }}
        uses: docker/build-push-action@v6
        with:
          context: .
          file: ${{ matrix.dockerfile }}
          push: true
          tags: \({{ env.DOCKER_USER }}/\){{ matrix.service }}:${{ env.RUN_TAG }}
          cache-from: type=gha,scope=${{ matrix.service }}
          cache-to: type=gha,mode=max,scope=${{ matrix.service }}

Pro tip: The paths-ignore section is critical. Later, the Argo CD Image Updater will write changes back to the Kubernetes-manifest/ folder. Without this ignore rule, your pipeline would trigger itself forever in an infinite loop.

Note: You must add DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD to your GitHub Repo Settings > Secrets.

How to Install and Access Argo CD

Now that your cluster is running, you can install Argo CD. You'll perform the installation using a standard Kubernetes manifest provided by the Argo project.

Step 1: Create the Namespace and Apply the Manifests

In Kubernetes, it is a best practice to keep your administrative tools separate from your applications. You will create a dedicated namespace named argocd and then apply the official installation script from the Argo project. This script includes all the necessary ServiceAccounts, Roles, and Deployments.

Run the following commands in your terminal:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

You'll see a long list of resources being created. Wait a minute or two for the pods to initialize. you can verify that all the core components of Argo CD are running.:

kubectl get all -n argocd

Ensure all pods show a status of Running before proceeding.

Step 2: Access the Argo CD User Interface

To access the dashboard, we use a technique called port forwarding. Since the Argo CD server is running inside the cluster's private network, your browser can't see it yet. Port forwarding creates a secure 'tunnel' between a port on your local machine (8080) and a port on the cluster service (443). This allows you to interact with internal services without exposing them to the public internet.

Run the following command:

kubectl port-forward svc/argocd-server -n argocd 8080:443

You can now open your browser and navigate to https://localhost:8080. Your browser may warn you that the connection is not private because of a self-signed certificate. You can safely click "Advanced" and proceed to the site.

Step 3: How to Log In

The default username for Argo CD is admin. The password is autogenerated during the installation process and is stored securely as a Kubernetes secret.

To retrieve this password, open a new terminal tab (so the port-forwarding keeps running) and run:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Copy the output and use it as the password to log into the dashboard.

Understanding the Argo CD Application

An Argo CD Application is a Custom Resource (CRD) that acts as a "contract" between your Git repo and your cluster. It defines:

• repoURL & path: This tells Argo CD exactly which Git repository to watch and which folder inside that repo contains your YAML manifests.

• destination: This defines where the app should live. We use https://kubernetes.default.svc to point to the local cluster where Argo CD is installed.

• syncPolicy: This is the heart of GitOps. By setting automated with selfHeal: true, we tell Argo CD to automatically fix the cluster if someone manually changes something (drift). The prune: true setting ensures that if you delete a file in Git, it also gets deleted in the cluster.

The Application Manifest

Create application.yaml in your project root:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gitops-argocd-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_REPO_NAME>.git
    targetRevision: HEAD
    path: Kubernetes-manifest
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd-demo-ns
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

Deploying the Application Manifest

Now we'll define our Kubernetes resources in the Kubernetes-manifest/ folder.

main-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: main-deployment
  namespace: argocd-demo-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: main-api
  template:
    metadata:
      labels:
        app: main-api
    spec:
      containers:
      - name: main-service
        image: <YOUR_DOCKERHUB_USERNAME>/main-service:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: main-service-lb
  namespace: argocd-demo-ns
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: main-api

aux-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aux-deployment
  namespace: argocd-demo-ns
spec:
  replicas: 2
  selector:
    matchLabels:
      app: aux-service
  template:
    metadata:
      labels:
        app: aux-service
    spec:
      containers:
      - name: aux-service
        image: <YOUR_DOCKERHUB_USERNAME>/aux-service:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: aux-service
  namespace: argocd-demo-ns
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: aux-service

Push and Sync

Step 1: Apply the Application Manifest

Use kubectl to deploy this manifest into the argocd namespace:

kubectl apply -f application.yaml -n argocd

Step 2: Push to Your Repository

To trigger the initial deployment and ensure Argo CD stays in sync with your source of truth, add, commit, and push your latest changes to the GitHub repository you configured in the manifest:

git add .
git commit -m "initial argo application deployment"
git push origin main

Step 3: Verify the Result in Argo CD

Once you push your changes, head over to your Argo CD dashboard. You'll see the gitops-argocd-demo application appear. After the initial sync, the dashboard will display a healthy, green status indicating that your live cluster state perfectly matches your Git repository.

Note: As you can see in the screenshot above, Argo CD provides a visual representation of how your Kubernetes objects – Services, Deployments, and Pods – are related and confirms they are "Synced" with your Git repo.

Automating Updates with Argo CD Image Updater

Now that we have automated the deployment, let’s solve the final manual hurdle: automatically updating image tags in our manifests whenever a new build is pushed to DockerHub.

Step 1: Install ArgoCD Image Updater

Install the Image Updater into the argocd namespace:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/config/install.yaml

Verify the pod is running:

kubectl get pods -n argocd | grep image-updater

Note: Version 1.1+ uses a CRD-based approach (ImageUpdater custom resources) instead of the annotation-based approach used in older versions. This guide covers the CRD method.

Step 2: Create a GitHub Personal Access Token

The Image Updater needs Git credentials to push write-back commits to your repository.

1. Go to GitHub → Settings → Developer Settings → Personal Access Tokens → Tokens (classic)

2. Click Generate new token

3. Select the repo scope (full control of private repositories)

4. Copy the generated token

Step 3: Create the Git Credentials Secret

Store the GitHub credentials as a Kubernetes secret in the argocd namespace:

kubectl -n argocd create secret generic git-creds \
  --from-literal=username=<YOUR_GITHUB_USERNAME> \
  --from-literal=password=<YOUR_GITHUB_PAT>

Replace <YOUR_GITHUB_USERNAME> and <YOUR_GITHUB_PAT> with your actual values.

Step 4: Add a Kustomization File to Your Manifests

The Image Updater uses Kustomize's images field to write updated tags. If your Kubernetes-manifest/ directory contains plain YAML files, you'll need to wrap them with a kustomization.yaml file.

Create a kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - main-api.yaml
  - aux-api.yaml

How it works: When the Image Updater detects a new tag, it appends an images section to this file:

images:
  - name: <YOUR_GITHUB_USERNAME>/main-service
    newTag: "12"
  - name: <YOUR_GITHUB_USERNAME>/aux-service
    newTag: "12"

Kustomize then overrides the image tags at deploy time, without modifying your original deployment YAML files.

We use Kustomize here because it allows the Image Updater to manage image tags in a separate, clean way. Instead of the Updater 'messing' with your original main-api.yaml file, it simply updates the kustomization.yaml file. Argo CD then uses Kustomize to merge those changes during deployment.

Step 5: Create the ImageUpdater Custom Resource

Create image-updater.yaml in your project root:

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: gitops-argocd-demo-updater
  namespace: argocd
spec:
  commonUpdateSettings:
    updateStrategy: newest-build
    allowTags: "regexp:^[0-9]+$"
  applicationRefs:
    - namePattern: "gitops-argocd-demo"
      writeBackConfig:
        method: "git:secret:argocd/git-creds"
        gitConfig:
          branch: main
          writeBackTarget: "kustomization:."
      images:
        - alias: main-service
          imageName: <YOUR_DOCKERHUB_USERNAME>/main-service
        - alias: aux-service
          imageName: <YOUR_DOCKERHUB_USERNAME>/aux-service

This ImageUpdater resource is the "brain" of our automated tagging system. Here is what the specific fields are doing:

updateStrategy:

• newest-build: It tells the updater to always look for the most recent image version in DockerHub based on creation time.

writeBackConfig: This is where the magic happens. It uses the git-creds secret we created to authorize the updater to 'write' back to your repository.

writeBackTarget:

• kustomization: We are telling the updater specifically to modify the kustomization.yaml file in the manifests folder rather than touching the deployment files directly.

images: We provide aliases (main-service and aux-service) so the updater knows exactly which images in DockerHub correspond to which containers in our Kubernetes manifests.

Apply the ImageUpdater CR to the cluster:

kubectl apply -f image-updater.yaml -n argocd

Push the kustomization.yaml to your Git repository (the Image Updater clones the repo, so it must exist remotely):

git add Kubernetes-manifest/kustomization.yaml
git commit -m "Add kustomization.yaml for image updater write-back"
git push origin main

Step 6: Verify the Image Updater

Check the Image Updater logs to confirm it's working:

kubectl logs -n argocd deployment/argocd-image-updater-controller --tail=20

Successful output looks like:

msg="Starting image update cycle, considering 1 application(s) for update"
msg="Setting new image to YOUR_DOCKERHUB_USERNAME/main-service:11"
msg="Successfully updated image 'YOUR_DOCKERHUB_USERNAME/main-service:7' to 'YOUR_DOCKERHUB_USERNAME/main-service:11'"
msg="Setting new image to YOUR_DOCKERHUB_USERNAME/aux-service:11"
msg="Committing 2 parameter update(s) for application gitops-argocd-demo"
msg="git push origin main"
msg="Successfully updated the live application spec"
msg="Processing results: applications=1 images_considered=2 images_skipped=0 images_updated=2 errors=0"

Conclusion

You have successfully implemented a professional-grade GitOps loop from scratch. By integrating GitHub Actions, Argo CD, and the Argo CD Image Updater, you’ve bridged the gap between your source code and your live environment.

Think about the workflow you just built:

1. You push code to GitHub.

2. GitHub Actions builds and tags a fresh Docker image.

3. Argo CD Image Updater detects that new tag and automatically commits it back to your Git manifests.

4. Argo CD pulls those changes and reconciles your cluster to the new desired state.

No more manual kubectl apply, no more configuration drift, and no more 2:00 AM mysteries. Your Git repository is now truly the Single Source of Truth. If it isn't in Git, it doesn't exist in your cluster, and that is the ultimate DevOps superpower.

You will clone one repo, spin up a real 3-node cluster, break it seven different ways, and watch it fix itself each time. No simulated output or fake clusters. Real Kubernetes, real failures, and real recovery. By the end, you will recognize these failure patterns when they show up in your production environment.

Table of Contents

• What KubeLab Is?

• Prerequisites

• How to Get the Lab Running

• Simulation 1 — Kill Random Pod

• Simulation 2 — Drain a Worker Node

• Simulation 3 — CPU Stress and Throttling

• Simulation 4 — Memory Stress and OOMKill

• Simulation 5 — Database Failure

• Simulation 6 — Cascading Pod Failure

• Simulation 7 — Readiness Probe Failure

• How to Read the Signals in Grafana

• How to Use This for Production Debugging

What is KubeLab?

KubeLab is an open-source Kubernetes failure simulation lab. It runs a real Node.js backend, a PostgreSQL database, Prometheus and Grafana, all inside a real cluster. When you click "Kill Pod", the backend calls the Kubernetes API and deletes an actual running pod. Nothing is fake.

Plan about 90 minutes for the full path. Or jump directly to any simulation if you have a specific production problem you want to reproduce.

Prerequisites

You need basic familiarity with Docker and comfort with the command line, but no prior Kubernetes experience is required.

Hardware: 8GB RAM minimum, 16GB recommended. The lab can run on Mac, Linux, or Windows with WSL2. You'll need to install three tools. Multipass spins up Ubuntu VMs for the cluster. kubectl is the Kubernetes CLI you will use for every simulation. Git clones the repo. If you cannot run three VMs, the repo includes a Docker Compose preview at setup/docker-compose-preview.md full UI with mock data, no real cluster needed.

How to Get the Lab Running

Full cluster setup lives at setup/k8s-cluster-setup.md in the repo. It walks through creating three VMs with Multipass, installing MicroK8s, joining the worker nodes, and deploying KubeLab. Follow it until all eleven pods show Running:

kubectl get pods -n kubelab
# All 11 pods should show STATUS: Running

Then open two port-forwards in separate terminal tabs and keep them running for the entire tutorial:

# Tab 1 — KubeLab UI at http://localhost:8080
kubectl port-forward -n kubelab svc/frontend 8080:80

# Tab 2 — Grafana at http://localhost:3000
kubectl port-forward -n kubelab svc/grafana 3000:3000

Grafana login: admin / kubelab-grafana-2026.

Position the KubeLab UI and Grafana side by side. Left half of the screen is the app. Right half is Grafana. You will watch both simultaneously from Simulation 3 onward.

Simulation 1: Kill Random Pod

This simulation deletes a running backend pod via the Kubernetes API. Without Kubernetes, you would SSH to the server, find the crashed process, and restart it manually, usually discovered by a user alert at 3am.

Before you click: Run kubectl get pods -n kubelab -w. Watch for a pod to go Terminating then a new one to appear.

kubectl get pods -n kubelab -w
# backend-abc123  1/1   Terminating   0   2m
# backend-xyz789  1/1   Running       0   0s   ← ReplicaSet created a replacement

What happened: The ReplicaSet controller noticed actual(1) did not match desired(2) and created a replacement in parallel with the shutdown. The Endpoints controller removed the dying pod from the Service before SIGTERM fired, so zero traffic hit a dying pod.

The production trap: A missing readiness probe means the new pod receives traffic before it has opened a DB connection. You get 500s on every deployment for 2–3 seconds.

The fix: Set replicas: 2, add a readiness probe, and set terminationGracePeriodSeconds to match your longest request timeout.

Simulation 2: Drain a Worker Node

This simulation cordons a worker node, then evicts all its pods to the remaining node.

To "cordon" a worker node means to mark it as unschedulable. When you run kubectl cordon <node-name>, the Kubernetes control plane adds the node.kubernetes.io/unschedulable:NoSchedule taint to the node. (A taint is a marker that tells the scheduler to avoid placing pods on that node unless they have a matching "toleration.") This tells the scheduler to stop placing any new pods onto that node. It does not affect the pods that are already running there.

Cordoning is the first, safe step in preparing a node for maintenance. It ensures that while you are draining the node, the scheduler isn't simultaneously trying to schedule new workloads onto it, which would defeat the purpose of the drain.

Without Kubernetes you would drain the server manually, guess when in-flight requests finish, patch it, and bring it back, the window of downtime is unpredictable.

Before you click: Run kubectl get pods -n kubelab -o wide -w. Watch which node each pod runs on.

kubectl get pods -n kubelab -o wide -w

NAME                     NODE               STATUS
backend-abc123-xk2qp    kubelab-worker-1   Terminating   ← evicted
backend-abc123-n7mw3    kubelab-worker-2   Running       ← rescheduled

In kubectl get nodes the node shows Ready,SchedulingDisabled until you run kubectl uncordon.

What happened: The node spec got spec.unschedulable=true. The Eviction API ran per pod. That path goes through PodDisruptionBudget policy checks before proceeding, unlike a raw delete. A raw kubectl delete pod bypasses this check entirely — which is why draining with kubectl drain is always safer than deleting pods manually during maintenance.

The production trap: Two replicas with no pod anti-affinity often land on the same node. Drain that node and both pods evict at once. Complete downtime despite replicas: 2.

The fix: Use pod anti-affinity with topology key: kubernetes.io/hostname and a PodDisruptionBudget with minAvailable: 1.

Simulation 3: CPU Stress and Throttling

This simulation burns CPU inside a backend pod for 60 seconds, hitting the 200m limit. Without Kubernetes, one runaway process can consume all CPU on the host and starve every other service.

Before you click: Run watch -n 2 kubectl top pods -n kubelab and open the Grafana CPU Usage panel.

kubectl top pods -n kubelab
# backend-abc123   200m   ← pegged at limit for 60s; the other pod stays ~15m

What happened: The Linux CFS scheduler enforces the cgroup limit by granting 20ms of CPU per 100ms period then freezing all processes in the cgroup for 80ms. The pod is not slow because it is broken. It is slow because it is frozen 80% of the time.

The production trap: kubectl top shows the pod using 95-150m, which looks normal. The metric shows usage at the ceiling, not the throttle rate. Teams spend hours checking application code for a latency bug that is actually a CPU limit set too low.

The fix: For latency-sensitive workloads, set CPU requests but remove CPU limits. Requests tell the scheduler where to place the pod without throttling at runtime. Confirm throttling with rate(container_cpu_cfs_throttled_seconds_total{namespace="kubelab"}[5m]).

Simulation 4: Memory Stress and OOMKill

This simulation allocates memory in 50MB chunks inside a backend pod until the kernel kills it. Without Kubernetes the process dies, the server goes down, and someone gets paged.

Before you click: Run kubectl get pods -n kubelab -l app=backend -w and open the Grafana Memory Usage panel.

kubectl get pods -n kubelab -l app=backend -w
# backend-abc123   0/1   OOMKilled   3   5m   ← no Terminating phase; SIGKILL bypasses graceful shutdown

What happened: The cgroup memory limit crossed 256Mi. The Linux kernel OOM killer scored processes in the container's cgroup and sent SIGKILL (exit code 137) to the top consumer. Not Kubernetes, the kernel. SIGKILL cannot be caught or handled, so no preStop hook runs and in-memory data or open transactions can be lost. Kubernetes only observed the exit, labeled it OOMKilled, and started a fresh container.

The production trap: The pod runs fine for 8 hours, OOMKills, and restarts. Memory resets to zero and everything looks healthy again. This repeats every 8 hours. The restart count climbs to 7, then 15, then 30, but no alert fires because the metrics look normal between crashes. You find out when a user emails saying the app has been "a bit glitchy lately."

The fix: Alert on rate(kube_pod_container_status_restarts_total{namespace="kubelab"}[1h]) > 3 before users notice.
The Prometheus expression means: look at how many times containers in the kubelab namespace have restarted over the last hour, calculate how fast that number is increasing per second, and fire an alert if that rate exceeds the equivalent of 3 restarts per hour. A healthy pod rarely restarts. Several restarts in an hour usually means the container is hitting its memory limit, dying, and coming back in a loop, so this alert catches the silent OOMKill pattern before users do.

Confirm it happened:

kubectl describe pod -n kubelab <pod-name> | grep -A 5 "Last State:"
# Reason: OOMKilled
# Exit Code: 137

To see the last output before the kernel killed the process, run kubectl logs -n kubelab <pod-name> --previous. The log stream stops abruptly with no shutdown message, SIGKILL leaves no time for cleanup or final logs.

Simulation 5: Database Failure

This simulation scales the PostgreSQL StatefulSet to 0 replicas. The pod terminates completely. Without Kubernetes, the database server crashes and data recovery depends on whether backups exist and when they ran.

Before you click: Run kubectl get pods,pvc -n kubelab. Note that the PVC exists before you start.

kubectl get pods,pvc -n kubelab
# postgres-0   (gone)
# postgres-data-postgres-0   Bound   ← PVC stays; data lives on the volume

A PVC, or PersistentVolumeClaim, is a request for storage by a user. Think of it as a pod's way of saying, "I need a certain amount of durable, persistent storage." In the context of a stateful application like PostgreSQL, the PVC is critical. When the database pod is deleted, the PVC (and the underlying PersistentVolume it is bound to) remains. This is where the actual database files are stored. When a new postgres-0 pod is created, the StatefulSet knows to re-attach the same PVC, ensuring the new pod has access to all the old data, preventing data loss.

What happened: The StatefulSet controller deleted the pod but left the PersistentVolumeClaim untouched. StatefulSets guarantee stable names and stable PVC binding. postgres-0 always mounts postgres-data-postgres-0. When you restore, the same pod name comes back and reattaches the same volume. PostgreSQL replays WAL to reach a consistent state.

The production trap: Apps without connection retry logic return 500s and stay broken even after PostgreSQL restores. Connection pools that do not validate on acquire hold dead connections forever.

The fix: Add connection retry with exponential backoff in your app. Use network-attached storage (EBS, GCE PD) in production so the pod can reschedule to any node.

Simulation 6: Cascading Pod Failure

This simulation deletes both backend replicas at the same time. If everything is down, without Kubernetes, you'd have to restart every service manually, and hope they come up in the right order.

Before you click: Run kubectl get endpoints -n kubelab backend-service -w. Watch the IP list.

kubectl get endpoints -n kubelab backend-service -w
# ENDPOINTS   <none>   ← every request in this window gets Connection refused

What happened: Both pods were deleted. The Service had zero endpoints. The ReplicaSet created two replacements in parallel, but traffic stayed broken until both passed their readiness probes. The endpoint list went empty and came back. You can see the exact downtime window in Grafana's HTTP Request Rate panel.

The production trap: replicas: 2 protects you from one pod dying at a time, nothing more.
If both replicas land on the same node and that node goes down, you have zero replicas and full downtime.
Check right now with kubectl get pods -n kubelab -o wide | grep backend, and if both pods show the same NODE, you are one node failure away from an outage.

The fix: Use pod anti-affinity to force replicas onto different nodes and a PodDisruptionBudget with minAvailable: 1 to block any voluntary action that would leave zero replicas.

Simulation 7: Readiness Probe Failure

This simulation makes one backend pod fail its readiness probe for 120 seconds without restarting it. Without Kubernetes, you'd have no way to take a pod out of traffic rotation without killing it. This is what happens in production when your app connects to a database on startup but the DB is slow. The pod is alive, but it's not ready. Kubernetes holds it out of rotation until it is.

Before you click: Run kubectl get pods -n kubelab -w in one tab and kubectl get endpoints -n kubelab backend-service -w in another.

# Pods tab: STATUS Running, RESTARTS 0 — almost nothing changes
# Endpoints tab: one IP disappears — the pod is alive but not receiving traffic

What happened: /ready returned 503. The kubelet marked the pod Ready=False. The Endpoints controller removed its IP from the Service. The liveness probe /health) still returned 200, so no restart. After 120 seconds /ready recovered and the pod rejoined. Run kubectl logs -n kubelab <failing-pod> -f to see the app log 503s for the readiness endpoint while the pod stays Running and receives no traffic.

The production trap: Readiness probes that check external dependencies (database, cache, downstream API) will remove all pods from rotation when that dependency goes down. Instead of degrading gracefully, your entire app goes offline.

The fix: Readiness probes should test only what the pod itself controls. Use a separate deep health endpoint for dependency checks and never tie readiness to external service availability.

4. How to Read the Signals in Grafana

kubectl shows current state. Grafana shows what happened over time. That history is essential when you are debugging something that started 4 hours ago.

The Four Panels that Matter

Pod Restarts: A flat line is good. A step up every few hours is a silent OOMKill loop — the most common invisible production failure.

CPU Usage: A healthy pod's CPU fluctuates. A throttled pod's CPU is unnaturally flat at its limit. That flat ceiling is the signal, not the number.

Memory Usage: Watch for a line that climbs steadily then disappears. That disappearance is an OOMKill. The line reappearing from zero is the restart.

HTTP Request Rate: During Cascading Failure you see a spike of 5xx for 5–15 seconds, the exact downtime window, timestamped.

5. How to Read the Terminal Signals

What you see in the terminal during and after each simulation tells you things Grafana cannot. Five commands matter.

The -w flag on kubectl get pods -n kubelab -w streams changes in real time. The columns that matter are READY, STATUS, and RESTARTS. READY shows containers ready vs total — 1/2 means one container is alive but not passing its readiness probe. STATUS shows the pod lifecycle phase: Running, Pending, Terminating, OOMKilled. RESTARTS is the most important column in production. A number climbing silently over days is a memory leak or a crash loop nobody has noticed yet.

kubectl get events -n kubelab --sort-by=.lastTimestamp is the control plane's diary. Every action the cluster took is here: Killing, SuccessfulCreate, Scheduled, Pulled, Started, OOMKilling, BackOff. When something breaks and you do not know why, read the events. The timestamp gap between a Killing event and the next Started event is your actual downtime window — not an estimate, the exact number.

kubectl describe pod -n kubelab <pod-name> is the deepest single-pod view. Three sections matter: Conditions (Ready: True/False tells you if the pod is in the Service endpoints), Last State (shows the previous container's exit reason — OOMKilled, exit code 137, or a crash), and Events at the bottom (the scheduler's reasoning for every placement decision). This is the first command to run when a pod is misbehaving.

kubectl get endpoints -n kubelab backend-service shows which pod IPs are actually receiving traffic right now. A pod can show Running in kubectl get pods and be completely absent from this list. That is a readiness probe failure. If this list is empty, no request to that Service will succeed regardless of how many pods show Running. Check this whenever users report errors but pods look healthy.

kubectl logs -n kubelab <pod-name> shows the container's stdout and stderr. Use -f to follow the stream. After a pod restarts, use --previous to see the logs from the container that just exited, essential when you need to know what the app was doing right before an OOMKill or crash. Logs are per container and are gone once the pod is replaced, so grab them before the ReplicaSet creates a new pod with a new name.

A full event sequence during Kill Pod recovery looks like this:

kubectl get events -n kubelab --sort-by=.lastTimestamp | tail -10

REASON            MESSAGE
Killing           Stopping container backend          ← SIGTERM sent
SuccessfulCreate  Created pod backend-xyz789          ← ReplicaSet fired
Scheduled         Successfully assigned to worker-2   ← Scheduler placed it
Pulled            Container image already present     ← no pull delay
Started           Started container backend           ← running

The line between Killing and Started is your actual recovery time. In a healthy cluster with a cached image it is 3–8 seconds. If it takes longer, check the Scheduled line, the scheduler may have struggled to find a node.

Two Prometheus Queries Worth Memorizing

First query: silent restart loop. rate(kube_pod_container_status_restarts_total{namespace="kubelab"}[1h]) counts how many times containers in that namespace have restarted over the last hour and expresses it as a rate (restarts per second). A healthy workload rarely restarts. If this rate is high (for example more than 3 restarts per hour), something is killing the container repeatedly, often an OOMKill or a crash. Alert when it exceeds a threshold so you see the pattern before users report errors.

Second query: invisible CPU throttling. rate(container_cpu_cfs_throttled_seconds_total{namespace="kubelab"}[5m]) measures how much time, per second, the Linux scheduler spent throttling containers in that namespace over the last 5 minutes. A result of 0.25 means the container was frozen 25% of the time. High latency with no restarts and "normal" CPU usage in kubectl top often means the CPU limit is too low and the kernel is throttling the process. Alert when this rate exceeds about 0.25 (25% throttled).

# Silent restart loop — alert when this exceeds 3 per hour
rate(kube_pod_container_status_restarts_total{namespace="kubelab"}[1h])

# Invisible throttling — alert when this exceeds 25%
rate(container_cpu_cfs_throttled_seconds_total{namespace="kubelab"}[5m])

Run these against your own cluster. Not just KubeLab. These are production queries.

6. How to Use This for Production Debugging

The repo includes docs/diagnose.md, a symptom-to-simulation map. Find the simulation that reproduces your issue, run it in KubeLab, and understand the mechanics before you touch production.

Exit code 137, pods restarting. Run the Memory Stress simulation. Confirm with kubectl describe pod | grep -A 5 "Last State:" and look for Reason: OOMKilled. Raise limits or find the leak. The simulation shows both.

High latency, pods look healthy, zero restarts. Run the CPU Stress simulation. Check container_cpu_cfs_throttled_seconds_total in Prometheus. If it climbs, your CPU limit is too low and the pod is frozen by CFS.

503 on some requests, pods show Running. Run the Readiness Probe Failure simulation. Check kubectl get endpoints — one pod IP is missing despite Running. The pod gets zero traffic.

Pods stuck Pending after a node went down. Run the Drain Node simulation. Run kubectl describe pod <pending-pod> and read Events. The scheduler will state why it cannot place the pod, often insufficient capacity or a PVC on the failed node.

Conclusion

You just broke a real Kubernetes cluster seven ways and watched it fix itself each time. You have seen the ReplicaSet controller fire, read an OOMKill from kubectl describe, watched endpoints go empty during a cascading failure, and understood why a pod can be Running and receiving zero traffic at the same time.

What you practiced here applies to other clusters, staging or production you can read but not safely break. That muscle memory (events, endpoints, restart counter) is what you reach for at 3 am when something is wrong. KubeLab is the safe place to build that reflex.

The repo holds more than this article covered. Explore mode lets you run simulations without the guided flow. The full interview prep doc at docs/interview-prep.md has answers to the 13 most common Kubernetes interview questions. The observability guide at docs/observability.md covers Prometheus and Grafana setup in detail.

If this helped you, star the repo at https://github.com/Osomudeya/kube-lab and share it with someone who is learning Kubernetes the hard way.

We just posted comprehensive Kubernetes course on the freeCodeCamp.org YouTube channel. This hands-on course is designed to bridge the gap between theoretical container orchestration and real-world deployment. Saiyam Pathak developed this course.

You will learn to deploy a cloud-native microservices stack from the ground up. You’ll explore the inner workings of the Kubernetes architecture, including the Control Plane, Worker Nodes, and essential interfaces like CRI, CNI, and CSI. You'll learn to ship a functional application complete with a frontend, auth service, and game engine.

This course is a deep dive into the modern Kubernetes ecosystem. You will implement advanced industry standards such as Gateway API for traffic management, CloudNativePG for managing PostgreSQL databases, and cert-manager for automated HTTPS security. By the time you reach the final demo, you’ll have integrated full-stack observability using Prometheus and Grafana, giving you the confidence to manage production-grade environments.

Watch the full course on the freeCodeCamp.org YouTube channel (6-hour watch).