The support channel has the same complaint arriving in six different tones. Is it down? Mine is just spinning. It worked this morning. Somebody has posted a screenshot of a loading indicator with no caption, which you suspect they're enjoying.

So you open the dashboard.

Nodes ready. Pods running. CPU at 20%. No restarts, nothing in CrashLoopBackOff, no alert fired. By every signal Kubernetes gives you, the system is healthy.

It is not healthy. Somebody is eleven seconds into waiting for the first word of an answer, and they're about to go back to doing it the old way, and they're not coming back.

That is the failure this piece is about. Not the outage. The quiet one, where everything is technically up and the product is still unusable.

Table of Contents

• Your First Instinct is Wrong

• You Are Not the Only One

• The Thing Nobody Told You When You Deployed It

• Where the Eleven Seconds Went

  • Stage One, Routing: Your Load Balancer is Guessing

  • Stage Two, The Queue: Nothing is Coming to Help

  • Stage Three, Prefill: The GPUs Are There, and Useless

  • Stage Four, Decode: Your Ingress is Cutting People Off

  • And Underneath All Four: Bursty Traffic and Nobody's Name on the Problem

• Meanwhile, the Platform Did Not Stand Still

• How This Got Here in the First Place

• What To Do Once the Fire is Out

  • Capacity

  • Performance

  • Placement

  • Resilience

  • Ownership

• Four Things Your Monitoring Still Isn't Telling You

• The Thing That is Still Unsettled

• Monday Morning

• Back to Thursday

Your First Instinct is Wrong

The instinct at that moment is to suspect the model. Wrong version, bad quantization, context too long, or that someone changed the system prompt.

It is almost never the model.

Walk one request end to end and you find four stages. It gets routed to a replica. It waits in a queue. The model reads the prompt. The model streams an answer. Only the last two involve the model doing anything you are paying for. The first two are queueing and routing, which is to say infrastructure.

When people say their model is slow, the model is usually fine. The wait is somewhere else.

You Are Not the Only One

It's worth knowing before you go hunting that this is the most common complaint in the field, not an unlucky configuration on your part.

A practitioner survey of 200 people running AI in production found that nearly half, 49.5%, named latency at peak load as their single hardest scaling problem. Not accuracy. Not hallucination. Not cost per token. Latency, at the exact moment people are trying to use the thing.

Two more figures from the same research are worth holding alongside it. 59.5% said running inference closer to users or decision points is critical or very important. 45.5% still serve from a single cloud region. People know proximity matters and have not managed to act on it, which tells you something about what multi-region GPU capacity costs to stand up.

The same research mentions organizations mandating sub-250ms response times alongside 99.9% availability. Read that carefully, because taken literally it is impossible. No meaningful LLM response completes in a quarter of a second. It has to mean time to first token (how long before the first word of the answer appears), and that is the right thing to hold yourself to anyway. Once tokens are flowing at a readable pace, people stop counting. The silence before the first one is what loses them.

The Thing Nobody Told You When You Deployed It

Here's the part that explains everything else.

A normal web request is short, small, and costs roughly what the last one cost. Kubernetes scheduling assumes that. Service load balancing assumes it. The Horizontal Pod Autoscaler assumes it. Your ingress controller's default timeout assumes it. Nobody wrote the assumption down, because for fifteen years it was simply true.

An LLM request breaks it in three places.

It runs in two phases with completely different profiles. Prefill: the model reads the entire prompt in one pass. Compute-bound, bursty, and this is what decides how long your user stares at nothing. Then decode: one token at a time, each needing the accumulated state of every token before it. That state is the KV cache. It lives in GPU memory next to the weights, and it grows as the answer grows.

So requests are not short, they can run for a minute. They do not cost the same, since a prompt with a document pasted into it can cost fifty times what a one-liner costs. And the genuinely scarce resource is GPU memory, which does not appear on a single dashboard you inherited.

That is why your monitoring said everything was fine. It was measuring the wrong machine.

Where the Eleven Seconds Went

Follow the request through the four stages and the failure modes fall out in order. These six are the ones practitioners name most often, and they map neatly onto the path.

Stage One, Routing: Your Load Balancer is Guessing

Round-robin stops being fair the moment request costs diverge. Three heavy prompts land on one pod while its neighbour handles one-liners. The loaded pod's queue grows, its tail latency climbs, and the fleet average still looks completely reasonable, which is why nobody noticed before today.

Long-lived connections make it worse. With HTTP/2 or keepalive, a client opens one connection and sends everything down it, and a Kubernetes Service balances per connection rather than per request. All that traffic pins to a single backend and stays there.

Then there is the cost most teams have never considered. If a replica already holds the prefix of this prompt in its KV cache, routing there skips real prefill work. Since most production prompts share a long system preamble, that is not an edge case, it is most of your traffic. Round-robin throws the benefit away at random, every time.

Stage Two, The Queue: Nothing is Coming to Help

Your autoscaler could add capacity. It will be late, and the reason is unglamorous: a new replica has to pull a multi-gigabyte serving image, fetch 20GB to 70GB of weights over the network, and load them into GPU memory. Minutes, not seconds, and that assumes a node with a free card already exists. If it does not, add provisioning time, and add whether your provider has stock in that zone today.

There is a second problem stacked on the first. The signal is usually wrong. CPU tells you nothing here, because the CPU is idle while the GPU works. GPU utilization is barely better: it reports that a kernel was executing during the sample window, and a server handling one request and a server handling sixty both read 100%. It cannot tell you whether anyone is waiting.

Queue depth can. vLLM already exposes running and waiting request counts as Prometheus metrics. Scale on those:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-server
spec:
  scaleTargetRef:
    name: llm-server
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{app="llm-server"})
        threshold: "5"

Check the metric names against your vLLM version. Several have been renamed across releases, and the failure is silent.

Two values there are deliberate. The minimum of 2, because a cold start means you cannot sit at zero and expect to serve. The ten minute cooldown, because scaling down eagerly means paying that cold start again shortly afterwards.

Which leads to the uncomfortable conclusion about autoscaling inference. Reactive scaling is permanently late by however long your cold start is, and no trigger fixes that. The fix is to keep more spare capacity than feels comfortable. Run extra replicas at all times, scale up earlier than you think you need to, and scale down more slowly. This also means scale to zero isn't a good fit for user-facing inference. Save it for batch jobs, where nobody is sitting there waiting for a response.

Stage Three, Prefill: The GPUs Are There, and Useless

Kubernetes hands out CPU in millicores and GPUs whole. A pod asks for one card and gets the entire card, whether it needs 10% of it or all of it.

The waste is the obvious problem. Fragmentation is the one that ruins a Thursday. A 70B model in 16-bit needs roughly 140GB of weights, so a replica needs several GPUs on one node. You can have six free GPUs spread two-two-one-one across four nodes, and the replica sits in Pending indefinitely. Capacity on paper, none of it usable, and no alert, because nothing is broken.

Sharing a card gives three options, each with a catch you should know before choosing. Time-slicing is easy to enable and gives no memory isolation, so one greedy pod can take down its neighbour. Fine for development, not for anything customer-facing. MIG gives real hardware isolation but fixed slice geometry, which means predicting your workload mix before you have one. Dynamic Resource Allocation is the structurally correct answer and brings accelerator requests properly into the core Kubernetes API.

None of these are on by default. Each needs a deliberate decision from somebody who understands the workload, who is usually not in the room when the cluster gets built.

Stage Four, Decode: Your Ingress is Cutting People Off

Many ingress controllers ship with a 60 second read timeout and response buffering enabled.

The timeout terminates long generations mid-sentence. Users report that as a crash, and you cannot reproduce it, because your test prompt is short. The buffering holds tokens and releases them in clumps, which destroys the feel of streaming while the model is behaving perfectly.

Two config lines, written for a workload that no longer exists.

And Underneath All four: Bursty Traffic and Nobody's Name on the Problem

Internal AI tools don't get steady traffic. They get sudden spikes: when everyone starts work at 9am, in the hour after a company all-hands meeting, or when someone shares the link in a busy Slack channel and says it's actually quite good. A fixed replica count handles exactly one of those. Most teams set it once during a quiet week and never revisit it, so they get both waste and collapse on the same day.

Then the sixth failure mode, which is not technical at all, and which is the reason the other five survive for quarters.

Platform engineering owns the cluster. The cluster is green. From where they sit, the job is done and done well. Application developers can see latency is bad and cannot see why, because every cause sits in scheduling, scaling and routing that they neither control nor observe. MLOps owns the model artifact and almost nothing that determines how it performs at quarter past two on a Thursday.

Three teams, all telling the truth, and the problem living in the gap between their on-call rotas, where no alert is configured and no dashboard points.

Meanwhile, the Platform Did Not Stand Still

The encouraging part of this story is that the gaps above are known, and the fixes have been shipping.

Dynamic Resource Allocation brings accelerator hardware requests into the core Kubernetes API, so the scheduler can reason about the device rather than counting opaque units. Kueue adds job queueing and multi-tenant GPU quota alongside CPU and memory, which is how you stop a research job starving the endpoint your users depend on. Gateway API Inference Extension and llm-d add model-aware routing: request criticality, dynamic balancing from live model metrics, and awareness of cache state. That last one is the direct answer to the guessing load balancer in stage one.

Published benchmarks give a sense of scale: Kueue reducing multi-stage workload makespan by up to 15%, dynamic accelerator slicing cutting mean job completion time by 36%, and Gateway API plus llm-d improving tail time to first token by up to 90% under heavy load.

Treat those as headroom rather than forecast. Up to 90% is measured on a configuration chosen to demonstrate the improvement, and your gain depends entirely on how poor your baseline is. If you already run least-outstanding-requests balancing with a warm cache, expect far less. If you are on stock round-robin behind a default ingress, possibly something dramatic, because that baseline is genuinely bad.

The direction is what matters. The biggest published gain is in tail time to first token, which is precisely what half the surveyed practitioners named as their worst problem. Somebody built the fix for the thing people were complaining about.

How This Got Here in the First Place

Worth a short detour, because it explains why the tooling arrived late.

When generative AI landed in enterprise planning documents, the confident position was that Kubernetes could not hold it. Container orchestration was built for small fungible units of CPU and memory, cheap restarts, interchangeable replicas. AI needed scheduled specialized silicon carrying enormous state that takes minutes to place. A purpose-built platform was coming.

It never arrived. AI went into the microservices stack and Kubernetes absorbed it.

The survey data explains why. More than half of enterprises do not train models at all, and only 7% deploy a model on any given day. Meanwhile 82% of container users run Kubernetes in production, and 66% of organizations hosting generative AI already serve it there. The industry spent years designing for the workload almost nobody has, while everybody else quietly downloaded weights and put them behind an endpoint.

What actually converged was not a place to run AI. It was a standard way to run it, with the location left negotiable. Which is exactly why your problems today are placement, routing and capacity rather than anything model-shaped.

What To Do Once the Fire is Out

The path from working to reliable comes down to five things: capacity, performance, placement, resilience and ownership.

Capacity

Requests per second is meaningless when one request is 50 tokens and the next is 8,000. Plan in tokens per second and track prompt and completion separately, because they stress different parts of the system. Prompt tokens are a compute burst during prefill. Completion tokens are a sustained drip during decode that holds GPU memory for the life of the response.

Load test with your worst realistic prompt at your worst realistic concurrency, and plan from that number. A benchmark built on short prompts gives a reassuring figure that evaporates in week one.

On supply, treat GPUs as something you reserve ahead of time rather than request when needed. Keep a committed baseline and burst on top. Find out your provider's quota and regional stock position before the evening you need it, and use Kueue so one team cannot quietly consume the pool.

Performance

Two numbers matter most. Time to first token (TTFT) is how long a user waits between sending a prompt and seeing the first word of the answer. It decides whether people trust the product. Inter-token latency is the gap between each word after that, and it decides whether the answer feels smooth to read.

Track both at p95 and p99, not the average. These are percentiles. p95 is the time that 95% of requests come in under, so it shows what your slowest 1 in 20 users experience. p99 does the same for the slowest 1 in 100. An average can look healthy while those users wait far too long, and they're the ones who give up.

Next, tune the serving layer. This is the software that loads the model and handles requests, such as vLLM. It's where you'll get the biggest improvements for the least effort and cost. Continuous batching keeps the GPU busy without making early requests wait for a batch to fill. Quantization roughly halves memory footprint where the quality tradeoff is acceptable, and freed memory becomes concurrent requests. Set maximum context length to what your product actually needs rather than what the model supports. If the model handles 128K and your longest genuine prompt is 6K, you are reserving KV cache for a scenario that never occurs. One config line, and it can substantially raise effective concurrency.

Placement

Taint GPU nodes so general workloads cannot land on them and only inference pods with matching tolerations schedule there. A logging sidecar should never be the reason a model replica cannot place.

Keep weights close. Node-local NVMe or a zone-local cache turns a 40GB network pull into a fast local read, which feeds directly into cold start, autoscaling lag, and the peak-hour latency you spent Thursday afternoon on. This is the highest-leverage unglamorous fix on the list.

Respect topology. Cards connected by NVLink behave very differently from cards that merely share a PCIe bus, and for a sharded model that interconnect sits on the critical path of every token generated.

Then be honest about geography. If your users are in London and your GPUs are in Virginia, every token crosses an ocean, and a well-tuned model behind a long network path is still a slow product.

Resilience

Readiness should pass only when the model is loaded and genuinely able to serve. Use a startup probe with a generous failure threshold, otherwise liveness kills the pod partway through loading weights and you get a restart loop that presents as a mystery and costs an hour.

Set a PodDisruptionBudget so a routine node upgrade cannot remove half your replicas at once.

Raise the termination grace period past the 30 second default and add a preStop hook. A generation can easily run longer than that, so otherwise every rollout severs in-flight streams and your users experience deployments as random failure.

Shed load deliberately. Past a queue depth threshold, return a fast busy-try-again rather than accepting a request that will hang for two minutes. This feels wrong to engineers and is right for users. People forgive a quick honest retry. Nobody forgives a frozen screen.

Have somewhere to fall back to: a second region, a smaller model covering the common cases, or an external API at the edge of disaster. Given that 45.5% of practitioners run from a single region, this is the most commonly skipped item on the list, which makes it the most likely candidate for your next incident.

Ownership

Agree on the numbers and write them down: time to first token, inter-token latency, error rate, queue time. Put them on one dashboard showing cluster and model metrics side by side, and make sure both teams open the same link rather than maintaining separate versions of reality.

Split responsibility explicitly. Platform owns the ability to hit the target: capacity, scheduling, routing, scaling, placement. Application owns how the model is configured and called: context length, batching parameters, prompt size, retry behaviour. When the number slips, both show up, and the conversation starts from the same graph instead of two that disagree.

Four Things Your Monitoring Still Isn't Telling You

Latency, traffic, errors and saturation are still necessary and no longer sufficient. Four more belong on the board.

Token consumption rate, split prompt and completion, because that is your real unit of capacity. Cost per request by tenant and query type, because we run eight GPUs is not an answer to what a feature costs per user. Output quality, tracking hallucination rate and guardrail drift, because latency work that degrades quality is not a win. Model attribution, recording which checkpoint and which prompt version produced a response, because when quality moves you need to know what changed.

That last one connects to a shift worth noticing. Model registries have become peers to container registries, and prompts and agent configurations are increasingly versioned as GitOps artifacts in the same delivery pipelines as everything else.

The prompt is code now. It changes production behaviour, it can break things, and it should go through review, versioning and rollback like anything else that does. Plenty of teams are still editing prompts in a web console and wondering why quality shifted on a Tuesday.

The Thing That is Still Unsettled

The substrate has converged, and the speed of it is the evidence. A Kubernetes AI conformance programme launched in late 2025 with 18 platforms. Within roughly four months it had grown to 31 and extended to agentic workloads, covering the major hyperscalers, the enterprise and private cloud distributions, GPU neoclouds and edge networks. Companies that agree on almost nothing agreed on this.

The layer above has not converged at all. AI gateways, evaluation platforms, agent control planes, agent observability. All advancing faster in commercial products than in any standard body, and all sitting exactly where the interesting business logic is heading.

The risk is worth naming plainly. You can be perfectly portable at the Kubernetes layer and completely locked in at the layer where you actually build. Your pods will migrate between clouds beautifully while your agent definitions, eval suites and gateway policies stay exactly where they are, because there is nowhere standard to move them to.

That is not a reason to avoid those tools. They solve real problems today. It is a reason to know which of your components has an exit and which does not, and to make that a decision you wrote down rather than one you discover during a renewal negotiation.

Monday Morning

Start with one measurement. Time a cold start end to end, from pod created to first token served, and write the number down. That figure is your autoscaling floor, and your minimum replica count falls directly out of it.

Put time to first token and queue depth on your main dashboard at p99. Replace CPU-based autoscaling with a queue-depth trigger and minimums that respect the cold start. Check your ingress read timeout and buffering against a real long streaming response rather than a test prompt. Fix the termination grace period so deployments stop cutting people off.

Then, over the next few weeks: taint your GPU nodes, move weights to node-local or zone-local storage, set maximum context length to what you actually use, and find out whether you are still on round-robin. You probably are.

And before the quarter ends, decide who owns the end-to-end latency number, and say it out loud in a room with the other team present.

Back to Thursday

The dashboard was never lying. It was answering a different question.

It was telling you whether the containers were alive, which they were. It could not tell you that requests were queueing behind a badly routed batch, that no new replica was coming for four minutes, that six GPUs were free and unusable, or that the ingress was buffering tokens it should have been streaming. Nothing in the standard toolkit is pointed at any of that.

Getting a model to answer on Kubernetes takes an afternoon. Getting it to answer well when everyone logs in at once is a different discipline, and it looks far more like ordinary infrastructure engineering than the current conversation suggests. Scheduling, routing, capacity, ownership. We have been solving those since long before any of this carried the AI label. The skills are already in the building. They need pointing at the right metric.

A green cluster is where the work starts. What counts is what the person typing the prompt sees.

I hope you’ve enjoyed this and learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving stars on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy exploring!

I've debugged Kubernetes networking issues at more than 10 companies. The same knowledge gaps appear every time. Engineers don't understand how ClusterIP works under the hood. They don't understand why Pods in different namespaces can talk to each other by default. And they don't understand what a CNI plugin actually does at the kernel level.

This tutorial is the fix. You'll learn how Kubernetes networking works from the bottom up: how Pod IPs are assigned and why they work across nodes, how kube-proxy implements ClusterIP using iptables rules, how Ingress controllers route external traffic through a single load balancer, how Network Policies enforce micro-segmentation for SOC2 compliance, and how Cilium uses eBPF to replace all of this with a faster, more observable, and more secure alternative.

By the end of this guide, you'll be able to debug "why can't my pod talk to that service?", implement default-deny Network Policies that satisfy SOC2 CC6.1, and choose the right CNI for your cluster with confidence.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: Pod IPs and the Container Network Model

• Part 2: Services — ClusterIP, NodePort, and LoadBalancer

• Part 3: Ingress — External Traffic Routing

• Part 4: Network Policies — Micro-Segmentation

• Part 5: CNI Comparison — Cilium vs Calico vs AWS VPC CNI

• Part 6: Service Mesh — Cilium vs Istio vs Linkerd

• Best Practices Summary

• Resources

What You'll Learn

• Pod IPs, the container network model, and how the CNI assigns addresses

• How kube-proxy implements ClusterIP with iptables and why eBPF is faster

• Ingress controllers: routing all external traffic through a single load balancer

• Network Policies: default-deny and per-service allow rules for zero-trust networking

• CNI comparison: Cilium vs Calico vs AWS VPC CNI and when to use each

• Service mesh: Cilium vs Istio vs Linkerd for mTLS and observability

Let's dive in.

Prerequisites

Before following along, you should have:

Knowledge:

• Basic Kubernetes familiarity: you can deploy a Pod and create a Service

• Basic Linux networking concepts: you know what an IP address and a port are

• A general understanding of what a load balancer does

Tools and access:

• A running Kubernetes cluster (EKS, GKE, or a local cluster via kind)

• kubectl configured and pointing at your cluster

• helm 3 installed (for Cilium installation in Part 4)

• For Part 4 onwards: Cilium installed on your cluster (helm install cilium cilium/cilium)

A note on CNI: Parts 1–3 apply to any Kubernetes cluster regardless of CNI. Parts 4–6 use Cilium-specific resources (CiliumNetworkPolicy, Hubble). If you're on a different CNI, the concepts are identical and only the YAML syntax differs.

Part 1: Pod IPs and the Container Network Model

1.1 Why Every Pod Gets Its Own IP

The Kubernetes networking model has one foundational rule: every Pod gets its own unique IP address, and every Pod can communicate with every other Pod using those IPs – without Network Address Translation (NAT).

This is different from how Docker works by default, where containers share the host network or use port mapping. In Kubernetes, there's no port mapping between pods. Pod A at IP 10.244.1.2 can directly reach Pod B at 10.244.2.3 across a different node, and the source IP is preserved.

Verify this for your cluster:

# List all pods across all namespaces with their IP addresses and node placement
kubectl get pods -o wide --all-namespaces

Expected output:

NAMESPACE     NAME                                READY   STATUS    IP            NODE
production    payment-api-5d6b8d8c4f-abc12        1/1     Running   10.244.1.2    node-1
production    user-api-5d6b8d8c4f-def34           1/1     Running   10.244.2.3    node-2
production    redis-master-0                      1/1     Running   10.244.1.4    node-1

Each pod has a unique IP. The payment-api on node-1 and the user-api on node-2 can reach each other directly at those IPs. Notice that the IPs come from the 10.244.0.0/16 CIDR: this is the Pod network, separate from the node network.

1.2 What the CNI Plugin Actually Does

The Container Network Interface (CNI) is the plugin responsible for making the Kubernetes networking model work. When a new Pod is scheduled on a node, the Kubernetes kubelet calls the CNI plugin, which performs four operations:

1. Creates a new network namespace for the Pod: an isolated networking environment

2. Creates a virtual Ethernet pair (veth): one end inside the Pod's namespace, one end on the node

3. Assigns an IP address from the cluster's Pod CIDR to the Pod's end of the veth pair

4. Adds routing rules so the node knows how to reach every Pod IP in the cluster

Without the CNI, pods would have no network connectivity. With it, the flat Pod network model becomes reality.

Check which CNI plugin is installed on your cluster:

# List the CNI binaries installed on a node
ls /opt/cni/bin/

Here are some common CNI plugins and when to use each:

1.3 Verifying Pod-to-Pod Communication

The most fundamental networking test: exec into one Pod and ping another by IP.

# Step 1: Get the IP of a target pod
TARGET_IP=$(kubectl get pod redis-master-0 -o jsonpath='{.status.podIP}')
echo "Target IP: $TARGET_IP"

# Step 2: Exec into another pod and ping the target
kubectl exec -it payment-api-5d6b8d8c4f-abc12 -- ping -c 3 $TARGET_IP

Expected output:

PING 10.244.1.4 (10.244.1.4): 56 data bytes
64 bytes from 10.244.1.4: icmp_seq=0 ttl=62 time=0.8ms
64 bytes from 10.244.1.4: icmp_seq=1 ttl=62 time=0.7ms
64 bytes from 10.244.1.4: icmp_seq=2 ttl=62 time=0.9ms

If this succeeds, the CNI is working correctly. If it fails, check whether a Network Policy is blocking ICMP traffic (Part 4 covers this).

The one rule to remember: every Pod gets an IP. Pods can communicate directly using those IPs. The CNI plugin makes both of these things true.

Part 2: Services — ClusterIP, NodePort, and LoadBalancer

2.1 The Problem: Pod IPs Are Not Stable

Pod IPs change every time a Pod restarts. If you deploy a new version of your payment API, the old Pods are deleted and new Pods are created with new IPs. Any service that was configured to call the old IPs now has dead references.

Here's the incorrect approach: hardcoding a Pod IP.

# Bad: Direct Pod IP in application configuration
# This IP will stop working the next time the database Pod restarts
apiVersion: v1
kind: Pod
metadata:
  name: payment-api
spec:
  containers:
  - name: api
    env:
    - name: DATABASE_HOST
      value: "10.244.1.4"  # Pod IP — will change on next restart

This is fragile in development and catastrophic in production. A routine Pod restart – from a node drain, an OOM kill, or a deployment rollout – will break any application that hardcoded the old IP.

2.2 How Services Solve the Stability Problem

A Kubernetes Service provides two things that Pod IPs can't: a stable IP address (the ClusterIP) that never changes as long as the Service exists, and a stable DNS name that other Pods can use regardless of the IP.

When you create a Service, Kubernetes assigns it a virtual ClusterIP from the service CIDR (for example, 10.100.0.0/16), creates a DNS record in CoreDNS as <service-name>.<namespace>.svc.cluster.local, and configures kube-proxy on every node to add iptables rules that load-balance traffic from the ClusterIP to the healthy Pod IPs behind it.

Here's the correct implementation: a ClusterIP Service.

# Good: ClusterIP Service provides a stable IP and DNS name
# redis.production.svc.cluster.local always resolves to 10.100.0.1
# regardless of which Redis pods are running behind it
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: production
spec:
  selector:
    app: redis
    role: master   # Only pods with these labels receive traffic
  ports:
  - port: 6379        # Port the Service listens on
    targetPort: 6379  # Port the Pod actually runs on
  type: ClusterIP     # Default: accessible only inside the cluster

How kube-proxy implements the load balancing using iptables: when a Service is created, kube-proxy adds iptables rules to every node in the cluster. These rules intercept traffic destined for the ClusterIP and redirect it to one of the healthy Pod IPs. Run this on a node to see the rules in action:

# View the iptables rules kube-proxy created for the redis Service
# Each KUBE-SEP entry represents one Pod endpoint
sudo iptables -t nat -L KUBE-SERVICES | grep redis

Expected output:

Chain KUBE-SVC-REDIS (1 references)
target          prot  source    destination
KUBE-SEP-AAA    all   anywhere  anywhere    /* production/redis */ statistic mode random probability 0.50
KUBE-SEP-BBB    all   anywhere  anywhere    /* production/redis */

Traffic to the Redis ClusterIP is distributed 50/50 between the two Pod endpoints via these iptables rules. When a Pod restarts and gets a new IP, kube-proxy updates the rules automatically.

2.3 When to Use Each Service Type

Verify a Service is routing traffic correctly:

# Describe a Service to see its endpoints (the actual Pod IPs behind it)
kubectl describe service redis -n production

Expected output:

Name:              redis
Namespace:         production
Type:              ClusterIP
IP:                10.100.0.1
Port:              6379/TCP
TargetPort:        6379/TCP
Endpoints:         10.244.1.4:6379,10.244.2.5:6379
Session Affinity:  None

If Endpoints shows <none>, the Service selector doesn't match any running Pods. This is the most common cause of "connection refused" errors in Kubernetes.

The one rule to remember is that pods should always connect to Service DNS names, never to Pod IPs. The Service handles stability, load balancing, and health checking automatically.

Part 3: Ingress — External Traffic Routing

3.1 The Problem: A LoadBalancer Service Per Microservice Is Expensive

Each LoadBalancer Service creates a dedicated cloud load balancer. On AWS, each Application Load Balancer costs approximately \(0.008/LCU-hour plus \)0.0225/hour base charge. That's roughly $16–27/month per load balancer.

At 20 microservices, that's $320–$540/month in load balancer charges alone, plus $0.008/LCU for each request processed.

Here's the incorrect approach with one LoadBalancer per microservice:

# Bad: This creates a new AWS ALB every time it is applied
# 20 microservices = 20 ALBs = $300-500/month before any traffic charges
apiVersion: v1
kind: Service
metadata:
  name: payment-api
spec:
  type: LoadBalancer   # Creates a dedicated ALB
  ports:
  - port: 80
    targetPort: 8080

3.2 How an Ingress Controller Solves This

An Ingress controller is a Pod running inside your cluster that watches for Ingress resources and programs a single external load balancer to route traffic to multiple Services based on the hostname and URL path.

The AWS Load Balancer Controller, for example, creates one ALB for all your Ingress resources and programs its listener rules to route api.company.com/payments to the payment Service and api.company.com/users to the user Service, all through the same load balancer.

Here's the correct implementation: one Ingress for all services.

# Good: One Ingress resource routes all external traffic
# One ALB is created total, regardless of how many services are listed
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shared-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
spec:
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /payments
        pathType: Prefix
        backend:
          service:
            name: payment-service
            port:
              number: 8080
      - path: /users
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 8080
  - host: dashboard.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboard-service
            port:
              number: 3000
  tls:
  - hosts:
    - api.company.com
    - dashboard.company.com
    secretName: tls-wildcard-cert

Verify the Ingress is provisioned and the ALB DNS name is assigned:

# Watch until the ADDRESS column shows the ALB DNS name (typically 2-3 minutes)
kubectl get ingress shared-ingress -n production -w

Expected output:

NAME             CLASS   HOSTS                                    ADDRESS                                              PORTS
shared-ingress   alb     api.company.com,dashboard.company.com   k8s-prod-sharedin-abc123.us-east-1.elb.amazonaws.com   80, 443

The cost difference:

The one rule to remember: one Ingress controller with path-based routing serves all your services through a single load balancer. The per-service LoadBalancer approach is for early prototyping only.

Part 4: Network Policies — Micro-Segmentation

4.1 The Default: Every Pod Can Talk to Every Other Pod

Out of the box, Kubernetes applies no network restrictions between Pods. A frontend Pod can make direct API calls to a database Pod. An analytics service can query the payment database. A compromised Pod can scan every other Pod in the cluster.

This isn't secure. For SOC2 CC6.1 (logical access controls), HIPAA, and most enterprise security frameworks, you need to be able to prove that network traffic is restricted to what's necessary.

Verify that unrestricted traffic is currently possible:

# Without Network Policies, this call from the frontend to the payment DB will succeed
# It should not be allowed in a secure cluster
kubectl exec -it frontend-pod -n production -- \
  curl http://payment-postgres.production.svc.cluster.local:5432

If this succeeds on your cluster, you have no network segmentation.

4.2 The Solution: Default-Deny with Cilium Network Policies

The correct approach is default-deny: block all traffic between Pods first, then explicitly allow only the specific communication paths that your application requires.

Step 1 — Apply the default-deny policy:

# This policy applies to all pods in the namespace (empty endpointSelector matches all)
# It blocks all ingress and egress traffic by default
# Warning: apply this and all pod-to-pod communication immediately stops
# Have your allow rules ready before applying this in production
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  description: "Block all inter-pod traffic by default — zero-trust baseline"
  endpointSelector: {}  # Matches all pods in this namespace
  ingress:
  - {}                  # Empty ingress rule = deny all inbound
  egress:
  - {}                  # Empty egress rule = deny all outbound

Applying the default-deny policy will break all pod-to-pod communication in the namespace immediately. Apply your allow rules (below) in the same kubectl apply command, or apply allow rules first.

Step 2 — Add namespace-level isolation:

# Allow pods to communicate within the same namespace
# Block cross-namespace traffic by default
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: production
spec:
  endpointSelector: {}
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: production
  egress:
  - toEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: production

Step 3 — Add per-service allow rules:

# Grant the payment service only the specific network access it needs
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-service-network-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  egress:
  # Allow: payment-service → postgres on port 5432
  - toEndpoints:
    - matchLabels:
        app: postgres-db
    toPorts:
    - ports:
      - port: "5432"
        protocol: TCP
  # Allow: payment-service → Stripe API externally
  - toFQDNs:
    - matchName: "api.stripe.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP

4.3 Using Hubble to Verify Policies and Collect SOC2 Evidence

Cilium includes Hubble, a network observability tool that shows you exactly which flows are being allowed and which are being dropped by your Network Policies. Hubble is your SOC2 evidence that network segmentation is operating correctly.

# Install the Hubble CLI
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
tar xzvf hubble-linux-amd64.tar.gz
sudo mv hubble /usr/local/bin/

# Port-forward to the Hubble relay
kubectl port-forward -n kube-system svc/hubble-relay 4245:80 &

# Show all flows in the production namespace from the last hour
hubble observe --namespace production --since 1h

# Show only dropped flows — proves policies are blocking unauthorised traffic
hubble observe --namespace production --verdict DROPPED --since 1h

Example Hubble output showing a blocked connection attempt:

Apr 19 03:17:41.234   DROPPED   TCP   10.244.1.5:52341 → 10.244.1.4:5432   policy-deny
Apr 19 03:17:41.235   ALLOWED   TCP   10.244.1.2:43211 → 10.244.1.4:5432   allow-same-namespace

The first line shows an unauthorized connection attempt blocked. The second shows a legitimate connection allowed. Export this log daily to your SOC2 evidence bucket.

The one rule to remember: default-deny is the zero-trust baseline. Then add explicit allow rules for every required communication path. Hubble gives you the evidence that it's working.

Part 5: CNI Comparison — Cilium vs Calico vs AWS VPC CNI

Choosing the right CNI is a decision that's difficult to reverse. Migrating between CNIs requires draining and replacing every node in the cluster. Make the decision once, for the right reasons.

Here's a real comparison across the capabilities that matter for production EKS clusters:

The recommendation matrix:

The one rule to remember: for SOC2 compliance and zero-trust networking on EKS, Cilium is the right choice. It provides pod-level isolation, Layer 7 policies, and Hubble flow logs that serve as audit evidence. These are capabilities no other CNI provides together.

Part 6: Service Mesh — Cilium vs Istio vs Linkerd

6.1 What a Service Mesh Provides

A service mesh adds three capabilities to your cluster's networking that Kubernetes doesn't provide natively.

mTLS (mutual TLS) encrypts communication between every pair of services and verifies both sides' identities. Without mTLS, traffic between your payment service and your database travels in plaintext inside the cluster.

Traffic observability tracks request rates, latency percentiles, and error rates for every service-to-service call, giving you a real-time performance map of your application.

Traffic management controls how traffic flows: retries on failure, timeouts, circuit breaking when a downstream service is degraded, and traffic splitting for canary deployments.

6.2 The Sidecar Problem

Traditional service meshes (like Istio and Linkerd) inject a sidecar proxy container into every Pod. This sidecar intercepts all network traffic and applies the mesh policies. The problem is resource overhead: Istio's Envoy sidecar adds approximately 128MB of memory and 5–10% latency overhead per Pod.

On a cluster with 200 Pods, Istio sidecars add 25.6GB of memory overhead and measurable latency to every service call.

Cilium solves this differently. It implements the service mesh at the kernel level using eBPF without any sidecar at all.

6.3 The Full Comparison

The recommendation matrix:

Enable Cilium's service mesh mode (no sidecars required):

# Upgrade your Cilium installation to enable service mesh features
helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set envoy.enabled=true \
  --set ingressController.enabled=true

# Verify the service mesh is active
cilium status | grep "Service Mesh"

The one rule to remember: Cilium gives you mTLS and SOC2 evidence with zero sidecar overhead. For teams that need advanced traffic management or complex canary release patterns, Istio provides more control at the cost of higher operational complexity.

Best Practices for Kubernetes Networking

✅ Do: Use Services, not Pod IPs. Pod IPs change on every restart. Service DNS names never change.

✅ Do: Use a single Ingress controller with path-based routing. One ALB serves all your services and saves $300–$400/month versus per-service LoadBalancer.

✅ Do: Implement default-deny Network Policies with Cilium. This is the technical control required by SOC2 CC6.1.

✅ Do: Use Hubble flow logs as SOC2 evidence. Export daily dropped-flow logs to your evidence bucket.

✅ Do: Enable mTLS with Cilium for encrypted service-to-service communication. No sidecar required.

✅ Do: Use topology-aware routing to keep traffic within the same Availability Zone and reduce cross-AZ data transfer costs.

❌ Don't: Create a LoadBalancer Service for every microservice. Use Ingress for external routing.

❌ Don't: Rely on Security Groups alone for pod-level isolation. Security Groups work at the node level. Any pod on a node shares the node's security group. Network Policies work at the pod level.

❌ Don't: Assume the default "allow all" pod networking is secure. Apply default-deny before your first enterprise customer asks for your network segmentation diagram.

Resources

• Cilium Documentation: Official Cilium installation guide, CiliumNetworkPolicy reference, and Hubble observability documentation

• Cilium Service Mesh Guide: How to enable mTLS and Layer 7 policies without sidecars

• Kubernetes Network Policy Documentation: The standard Kubernetes NetworkPolicy API reference

• AWS Load Balancer Controller: Official documentation for the Ingress controller that provisions AWS ALBs from Kubernetes Ingress resources

• Hubble CLI Installation: Install the Hubble CLI for observing Cilium network flows

• kube-proxy iptables mode: Kubernetes documentation explaining how kube-proxy implements Service routing using iptables

• Kubernetes CNI Plugin Specification: The CNI interface specification that all CNI plugins implement

• AWS VPC CNI Plugin GitHub: Source code and documentation for the default EKS networking plugin

• Companion Repository: CiliumNetworkPolicy manifests and Hubble evidence export scripts from this guide

Between those two moments runs a long chain of machinery: builds, artefacts, migrations, health checks, traffic shifts. Every production engineer depends on that chain, and many teams still build and operate it themselves.

Deployment infrastructure has quietly become operational overhead. It started as a technical necessity, something every team had to assemble because nothing else existed.

Today it is a second system your engineers maintain alongside the product, consuming on-call rotations, sprint capacity, and 2 a.m. attention that could go somewhere better.

In this article, we'll walk through each stage of a real production deployment: the build, the artefact it produces, database migrations, health checks, rolling updates, and rollbacks. Along the way, we'll look at why platform-as-a-service (PaaS) tools handle most of these steps for you, and what it costs a team to keep handling them itself.

Table of Contents

• The Build: Turning Code into Something That Can Run

• The Artefact: One Version, Frozen in Time

• Database Migrations: The Riskiest Step

• Health Checks: Proving the New Version Is Alive

• Rolling Updates: Replacing the Plane's Engine Mid-Flight

• Rollbacks: The Escape Hatch

• When You Don't Need a PaaS

• Should You Still Be Running This Yourself?

The Build: Turning Code into Something That Can Run

A deployment does not ship your source code as-is. It ships the result of a build. The build stage takes your code and turns it into something a server can run.

What this looks like depends on your stack. A Java or Go project gets compiled into a binary. A JavaScript front end gets bundled and minified. A Python app gets its dependencies resolved and pinned. In most modern setups, all of this gets packed into a container image, which is a frozen snapshot of your app plus everything it needs to run.

The build stage also runs your tests. Unit tests, linting, and security scans all happen here. If any of them fail, the deployment stops before it can touch production. This is the cheapest place to catch a bug. A failed build costs you a few minutes. A failed deployment can cost you customers.

Teams that run their own pipelines spend real effort here. They maintain build servers, cache dependencies, and debug flaky test runners.

None of that work ships a feature. It is pure upkeep, and it never ends. A PaaS bakes this whole stage into the platform. You push code, and the platform detects your language, builds it the same way every time, and fails fast when something is wrong. The build still happens. Your engineers just stop paying for it in hours.

The Artefact: One Version, Frozen in Time

The output of a build is called an artefact. It might be a container image, a compiled binary, or a zipped bundle. Whatever the format, the artefact has one job: to be exact. It represents one precise version of your app, frozen at one point in time.

This matters more than it sounds. The artefact that passed your tests must be the exact same one that reaches production. If you rebuild between testing and shipping, you risk shipping something slightly different. A dependency may have updated. A build flag may have changed. "It worked in staging" often means "we built it twice and got two different results."

Good pipelines build once and promote the same artefact through every stage. Artefacts get versioned and stored in a registry, so any version can be pulled and run again later. That stored history is also what makes rollbacks possible, which we will get to soon.

On a PaaS, artefact handling is standard practice by default. Every deploy produces a numbered release. The platform stores it, tracks it, and can restore it. You do not have to design a registry strategy, write promotion scripts, or assign an engineer to own them. The discipline is built in.

Database Migrations: The Riskiest Step

Before new code goes live, the database often has to change with it. Maybe the new version needs a new column or a new table. These changes are called migrations, and they are the most dangerous part of most deployments.

Why? Code is easy to replace. Data is not. If you deploy a bad code version, you can swap it out. If a migration corrupts or drops data, there may be no clean way back. Migrations also create a tricky window of time. For a few minutes, old code and new code may run against the same database at once. Both versions have to work with the schema during that window.

The safe pattern is to make migrations backwards-compatible. Add the new column first, deploy code that can handle both shapes, then clean up the old column in a later release. It takes more steps, but each step is safe on its own.

A PaaS cannot write your migrations for you. No tool can know what your data means. But a good platform gives migrations a defined place in the release process, runs them in order, and logs exactly what ran and when. That structure prevents the classic failure where someone runs a migration by hand and forgets to tell the team.

Health Checks: Proving the New Version Is Alive

Once the new version starts, the platform does not just trust it. It checks. A health check is a small endpoint in your app, often just a route that returns "OK." The platform calls it over and over. If the app answers, it is considered healthy. If it does not, the platform assumes something is wrong.

There are usually two kinds of checks. A readiness check asks, "Are you ready to receive traffic?" A liveness check asks, "Are you still working, or should I restart you?" The difference matters. An app can be alive but not ready, such as when it is still warming up a cache.

Health checks are the gatekeepers of a deployment. No traffic reaches a new version until it proves it can handle requests. Without them, you would be routing real users to an app that might still be crashing on startup.

Every serious PaaS runs health checks automatically. You define the endpoint, and the platform handles the polling, the timeouts, and the decisions. Teams that build this themselves tune all of those settings by hand, and they usually learn the right values through painful trial and error. That tuition is paid in engineering time, on a problem the industry solved years ago.

Rolling Updates: Replacing the Plane's Engine Mid-Flight

Here is the hard part. Your old version is serving live traffic right now. You need to replace it without dropping a single request. The most common answer is a rolling update.

It works like this. Say you have four copies of your app running. The platform starts one copy of the new version and waits for its health checks to pass. Then it shifts a slice of traffic to it and shuts down one old copy. It repeats this, one copy at a time, until only the new version remains. Users never notice, because at every moment there are enough healthy copies to serve everyone.

Some teams use variations of this idea. A blue-green deployment runs the full new version beside the old one, then flips all traffic at once. A canary release sends a tiny share of users to the new version first, watching for errors before going wider.

Doing this by hand means writing orchestration logic, managing load balancer rules, and handling every edge case where a step fails halfway. That is months of engineering effort to build and a permanent tax to maintain, all for behavior a PaaS ships as the default. On a platform, you get zero-downtime releases out of the box, not as a project your team has to staff.

Rollbacks: The Escape Hatch

Sometimes the new version passes every check and still breaks something real. An error rate climbs. A page loads blank. Now speed matters more than anything, and the fastest fix is rarely a new patch. It is a rollback: redeploying the previous artefact that you already know works.

This is why frozen, versioned artefacts are so important. A rollback is only fast if the old version is stored, tested, and ready to run. Teams that rebuild from an old commit under pressure are gambling at the worst possible time.

On most PaaS platforms, a rollback is one command or one click. The platform keeps your release history and can restore any previous version in seconds. That single feature has saved more on-call engineers' nights than perhaps any other.

When You Don't Need a PaaS

The case for handing deployment to a platform is strong, but it isn't universal. There are teams for whom owning the pipeline is not overhead; it is a deliberate and justified engineering decision.

When Compliance Demands It

Regulated industries like finance, healthcare, government, often operate under requirements that a standard PaaS cannot satisfy out of the box. Data residency rules may dictate exactly which physical infrastructure your builds touch. Audit requirements may demand a level of provenance and access logging that a managed platform doesn't expose.

Security controls may need to extend into the build environment itself, not just the runtime. In these contexts, the cost of owning the pipeline is real, but it is the cost of operating in that industry.

When Deployment is Your Product

If your company sells deployment infrastructure, a CI/CD platform, a release orchestration tool, an internal developer platform, then your pipeline is not overhead at all. It is the product.

The engineers maintaining it are doing product work, not distraction work. The same applies to platform engineering teams at large organizations whose explicit charter is to build and own the deployment layer for dozens of other internal teams. In both cases, the question of "why are we running this ourselves" has an obvious answer: because this is what we do.

When Your Infrastructure is Genuinely Unusual

Most PaaS platforms are optimized for stateless web services and standard container workloads. If your system falls outside that envelope, GPU clusters, real-time systems with strict latency requirements, hybrid on-premise and cloud deployments, hardware-in-the-loop testing, a general-purpose platform may simply not fit.

Shoehorning an unusual workload into a PaaS often produces more friction than building narrow, purpose-built deployment tooling around the specific constraints you actually have.

The common thread across all three cases is specificity. The teams that are right to own their pipelines can usually state clearly why a platform doesn't fit. If the answer is "we've always done it this way" or "we like having control," that's worth questioning. If the answer is "our compliance requirements mandate X" or "we sell this," that's a reason.

Should You Still Be Running This Yourself?

A PaaS does not make any of these steps disappear. The build still runs. Artefacts still get stored. Migrations still execute, health checks still poll, and traffic still shifts one copy at a time. Abstracting these mechanics does not eliminate them. It standardizes them, and pushes their maintenance onto a team whose entire product is deployment.

That is the question every product team should now ask plainly: why are we still building and operating this machinery ourselves? A decade ago, a custom pipeline was unavoidable. Today it is a choice, and for most teams it is the wrong one. Every hour spent debugging a flaky build agent, tuning a health check timeout, or patching orchestration scripts is an hour taken from the product your customers actually pay for. The pipeline does not differentiate you. It cannot. Your competitors' deploys work the same way yours do.

Know how the chain works, because on-call at 2 a.m. demands it. But knowing how it works is not a reason to own it. "We built our own deployment system" is not a badge of honor anymore. It is an admission that your team maintains a second product with no customers. Unless deployment infrastructure is your business, hand the machinery to a platform, and put your engineers back on the work only they can do.

Hope you enjoyed this article. You can connect with me on LinkedIn.

An operator extends the same pattern to resources Kubernetes doesn't know about natively, letting you manage custom, often external, systems the same declarative way you manage everything else in the cluster.

This guide is divided into four parts: what an operator actually is, the anatomy of one, building one from scratch, and preparing it for production.

Table of Contents

• Part 1: Introduction

  • What is an Operator?

  • Operator vs Controller vs CRD

  • Why Not Just a Helm Chart, a CronJob, or a Script?

• Part 2: Anatomy of an Operator

  • Custom Resource

  • Watching for Change

  • Manager

  • Reconciliation Loop

• Part 3: Building the Operator

  • Setup

  • Mock Provider

  • Defining the VirtualMachine CRD

  • Reconciler

  • Failure Handling & Retries

  • Finalizer

  • Predicate

  • Owned Resources

  • Cross-Resource Reconciliation

  • RBAC

• Part 4: Production & Deployment

  • Packaging & Deployment

  • Performance & Resilience

  • Security

  • Observability

• Next steps

Part 1: Introduction

What is an Operator?

Kubernetes works by comparing the state we describe against actual state. A controller acts to close the gap, whether that's the Deployment controller replacing a pod we killed or scaling down the ones we no longer want.

This loop of observe, compare, and act is called reconciliation. It means looking up a resource's desired and actual state, deciding what to do next, and recomputing that decision fresh on every run regardless of what changed.

That's what makes the loop resilient: it never has to trust that it saw every event, only that it gets called again.

An operator applies that exact same loop to a resource Kubernetes doesn't understand natively. We define a Custom Resource, describe our domain's desired state in it, and write a controller that knows how to reconcile that domain.

That's the whole concept. Everything else in this guide (informers, workqueues, finalizers, status conditions) exists to make that one loop reliable for a resource type Kubernetes only knows about because we defined it.

Operator vs Controller vs CRD

These three terms are often used interchangeably, but they describe three different layers of the same system.

• CRD (CustomResourceDefinition): a schema we register with the Kubernetes API server to teach it about a new resource type. On its own, a CRD does nothing. It only gives the API server a shape to store, validate, and serve.

• Controller: any piece of software running a reconciliation loop against a resource type, from the built-in Deployment controller to a custom controller reconciling a PostgresCluster.

• Operator: a controller, or a small set of controllers, that targets a custom resource and encodes enough domain-specific knowledge to manage its full lifecycle without a human: provisioning, upgrades, failure recovery, and so on.

Every operator is a controller, but not every controller is an operator. A CRD without a controller behind it is just a schema that nothing acts on.

Why Not Just a Helm Chart, a CronJob, or a Script?

A Helm chart renders a set of values into YAML and applies it once. It has no way to keep watching afterward. if a resource it created is deleted or drifts, Helm has no idea until we run helm upgrade again by hand.

A CronJob gives us a loop back, at the cost of granularity, staleness up to one interval, no state carried between runs, and no way for one CronJob to react to a status change another one made.

And a one-off script only acts when triggered, manually or by a CI pipeline, and does nothing about drift in between runs. It's also rarely written with retries and idempotency as first-class concerns.

Why an operator wins here:

An operator is event-driven and continuous. The API server notifies it the instant a custom resource is created, updated, or deleted, and it keeps reconciling for that resource's entire lifetime, not just at apply time. That matters most for state that takes time to converge, can fail partway through, and can drift after it's first created.

This comes at a cost, though. An operator is a long-running process with its own RBAC (Role-Based Access Control), failure modes, and observability surface. This is more to build and operate than a chart or a script.

If the problem really is rendering some YAML once, a Helm chart is the right tool. An operator earns its keep when the problem is keeping something continuously correct, which is what the rest of this guide builds toward.

Part 2: Anatomy of an Operator

Next, we'll look at the pieces that make that loop actually work: the Custom Resource itself, the machinery that notices when something changed, and the manager that runs it all, before going into the reconciliation loop in detail.

Custom Resource

Before a controller can reconcile anything, the API server needs to know the shape of what it's storing. Registering a CRD teaches it that shape.

Every Custom Resource carries the same identity fields every Kubernetes object already has (kind, name, namespace, labels, and so on), plus two fields that are entirely ours to define: a spec and a status. That split isn't a style choice. It maps directly to the reconciliation loop.

• Spec is desired state. Whoever creates or edits the resource writes it, and the controller only ever reads it.

• Status is observed state. It's written only by the controller, to record what it found and what it did.

A client that writes to status directly is working around the controller instead of through it, which is why status is usually served as its own subresource with separate permissions.

The last piece is registration. The API server and any client talking to it need a shared, agreed-upon way to encode and decode our type, so we register it once against a scheme before anything can use it. Without that, our type is just a definition nobody can serve. With it, the API server can store and serve it exactly the way it serves Pods or Deployments.

We'll see exactly what that registration looks like when we build one for real in Part 3.

Watching for Change

A reconciler doesn't poll the API server in a loop asking "did anything change yet?" Three pieces work together to avoid that.

An informer opens a long-lived watch against the API server and keeps a local, in-memory cache of every object of a given type, updating it as add, update, and delete events arrive.

A lister reads from that cache instead of the API server, so a reconciler checking "does this Resource already exist?" costs a local map lookup, not a network call.

A workqueue sits between the informer and the reconciler. When the informer sees a change, it doesn't call the reconciler directly. Instead, it enqueues a key, namespace, and name, not the object itself. Workers pull keys off the queue and reconcile them, and the queue deduplicates and rate-limits on our behalf, so ten rapid updates to the same object collapse into one pending item instead of ten redundant reconciles.

This is also why a reconciler receives a key and not an object. By the time a worker picks the key off the queue, the object may have changed again, so the reconciler always looks up the current state itself rather than trusting whatever triggered it.

Manager

The manager is the process that owns all of this: the shared cache the informers populate, the client the reconciler uses to read and write objects, and the health, readiness, and metrics endpoints the rest of the cluster uses to know the controller is alive. Every reconciler we register runs inside one manager.

If we run more than one replica of the same controller for availability, we don't want both replicas reconciling the same object at once and racing each other.

The manager coordinates this through leader election. Replicas compete for a lease, exactly one holds it and actively reconciles, and the rest sit idle until the leader stops renewing it.

We'll come back to this in practice in Part 4. For now it's enough to know the manager is what makes it possible.

Reconciliation Loop

This is the part that matters most. Once we understand this loop well, most of what an operator does is a variation on it.

A reconciler's entry point is called with just a namespace and a name, nothing else. No spec, status, or diff. The reconciler has to fetch the object itself, compare its spec against what it can observe of the actual state, and decide what to do. That constraint is deliberate, and it's the reason for everything below.

Idempotency

Because the reconciler only ever gets a key, and because it can be called any number of times for the same object (in a row, out of order, or after a long gap), it has to produce the same end result no matter how many times it runs. A reconciler that blindly calls create every time it runs breaks the moment it runs twice, since the second call fails against an object that already exists.

The fix is to always check current state before acting: create only if missing, update only if different, and delete only if it shouldn't exist.

Event-driven reconciliation

A reconcile is triggered by a watch event on the resource being reconciled, and by convention also on anything it owns or otherwise depends on. On top of that, most controllers set a periodic resync so the loop also runs on a schedule even with no watch event at all, which matters once state can drift for reasons a watch would never catch.

Requeues

Sometimes a single pass through reconcile can't finish the job, becuase the work it's waiting on is still in progress elsewhere. A reconciler can ask to be called again after a delay without treating this as a failure. This is how it polls something that takes time to converge, rather than blocking inside a single call.

Error handling

Returning an error does something similar: it requeues, but with exponential backoff instead of a fixed delay. So a persistently failing reconcile doesn't hammer whatever it's failing against.

It's worth distinguishing errors that are worth retrying (like a timeout or lock conflict) from ones that aren't (like a spec that will never be valid, which should be surfaced as a status condition instead of retried forever).

Drift correction

Put all of the above together and the loop is self-healing by construction. Because reconcile recomputes the full diff every time rather than reacting to what specifically changed, it doesn't matter whether the drift came from someone running kubectl edit, another controller, or the underlying system the resource represents changing state on its own. The next reconcile, whether triggered by a watch event or a resync, sees the same gap either way and closes it the same way.

Part 3: Building the Operator

Everything so far has been building toward this. We now know what an operator is, how the terms around it relate, and what pieces a reconciliation loop is made of.

Now we'll put all of it to use and build VMOperator. It's an operator that manages a VirtualMachine Custom Resource backed by a mock cloud provider, a small HTTP service we'll also write that stands in for a real one.

apiVersion: compute.example.com/v1
kind: VirtualMachine
spec:
  image: ubuntu-22.04
  cpu: 2
  memory: 4Gi
status:
  phase: Running
  id: vm-123

Say we want to represent a virtual machine in a Kubernetes-native way, kubectl apply a YAML file, and get a VM – all without touching a cloud console or a separate CLI.

That's the motivation behind VMOperator: something that lives entirely outside Kubernetes becomes just another object the cluster's own tooling (kubectl, RBAC, GitOps pipelines) already knows how to work with.

Setup

We'll need a local cluster, and kind is the easiest way to get one:

kind create cluster --name vmoperator

Beyond that, we'll be using Go in this part for the operator, kubectl pointed at the new cluster, and Python with Flask for the mock provider below.

pip install flask

Mock Provider

Before we write controller code, we need something for it to control. The mock provider is a small HTTP service with three endpoints:

• POST /vms to create one

• GET /vms/{id} to check on it

• DELETE /vms/{id} to remove it

backed by nothing more than a dict in memory.

Every VM it creates starts in Provisioning and flips to Running a few seconds later on its own, which is enough to force our reconciler to actually poll instead of assuming success.

We're writing this one in Python rather than Go. It has nothing to do with the operator's code, as this is only for mock purposes.

import random
import string
import threading
import time

from flask import Flask, jsonify, request

app = Flask(__name__)
vms = {}  # in-memory store, keyed by VM id

def provision(vm):
    time.sleep(5)  # simulate provisioning taking time
    vm["phase"] = "Running"

@app.post("/vms")
def create_vm():
    body = request.get_json()
    vm_id = "vm-" + "".join(random.choices(string.digits, k=6))
    vm = {"id": vm_id, "image": body["image"], "phase": "Provisioning"}
    vms[vm_id] = vm

    threading.Thread(target=provision, args=(vm,), daemon=True).start()  # flips to Running in the background

    return jsonify(vm)

@app.get("/vms/<vm_id>")
def get_vm(vm_id):
    vm = vms.get(vm_id)
    if vm is None:
        return "", 404
    return jsonify(vm)

@app.delete("/vms/<vm_id>")
def delete_vm(vm_id):
    vms.pop(vm_id, None)
    return "", 204

if __name__ == "__main__":
    app.run(port=8080, threaded=True)

We'll run this as its own process, alongside the cluster, listening on the port the operator will be configured to call. Nothing about it knows Kubernetes exists, which is the point: it's standing in for a real cloud API.

Defining the VirtualMachine CRD

With something to control, we can define what we're controlling. The VirtualMachine type follows exactly the spec and status split from Part 2:

type VirtualMachineSpec struct {
	Image  string `json:"image"`
	CPU    int    `json:"cpu"`
	Memory string `json:"memory"`
}

type VirtualMachineStatus struct {
	ID    string `json:"id,omitempty"`    // provider-assigned id, empty until first provisioned
	Phase string `json:"phase,omitempty"` // mirrors the provider's lifecycle phase
}

type VirtualMachine struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   VirtualMachineSpec   `json:"spec,omitempty"`
	Status VirtualMachineStatus `json:"status,omitempty"`
}

type VirtualMachineList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []VirtualMachine `json:"items"`
}

TypeMeta carries kind and apiVersion, the same two fields on every Kubernetes object, built-in or custom, that say what this thing is.

ListMeta is its counterpart for list types, resourceVersion and continue for pagination (instead of name/namespace). This is why VirtualMachineList embeds ListMeta next to its TypeMeta while VirtualMachine itself embeds ObjectMeta.

Every type we register needs to satisfy runtime.Object, which means implementing DeepCopyObject. This is normally generated for us, but since we're doing this by hand, here's what that generated code actually looks like for VirtualMachine. The rest follow the same mechanical pattern:

func (in *VirtualMachine) DeepCopyObject() runtime.Object {
	out := VirtualMachine{
		TypeMeta:   in.TypeMeta,
		ObjectMeta: *in.ObjectMeta.DeepCopy(), // ObjectMeta already knows how to copy itself
		Spec:       in.Spec,                   // no pointers or slices in Spec, a plain copy is safe
		Status:     in.Status,
	}
	return &out
}

And the CRD manifest that teaches the API server about it:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: virtualmachines.compute.example.com
spec:
  group: compute.example.com
  scope: Namespaced
  names:
    kind: VirtualMachine
    listKind: VirtualMachineList
    plural: virtualmachines
    singular: virtualmachine
    shortNames: [vm] # lets us type `kubectl get vm` instead of the full plural
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {} # splits status into its own subresource, see Part 2
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [image, cpu, memory]
              properties:
                image: { type: string }
                cpu: { type: integer }
                memory: { type: string }
            status:
              type: object
              properties:
                phase: { type: string }
                id: { type: string }

The subresources.status line matters. It's what makes status a separate subresource with its own update path. This is exactly the boundary we talked about in Part 2 between what a client can write and what only the controller can.

The names block is also what kubectl resolves against, kubectl get virtualmachines works because plural says so. shortNames is why kubectl get vm works too, the same way kubectl get po works for Pods.

Reconciler

The reconciler's job is small on paper, look at a VirtualMachine, and make sure a matching VM exists in the provider and its status reflects reality. We wrap the provider's HTTP API behind a small client so the reconciler itself stays readable:

func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var vm computev1.VirtualMachine
	if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err) // object was deleted, nothing left to do
	}

	if vm.Status.ID == "" {
		// no VM yet, this is the first time we've seen this object
		created, err := r.Provider.Create(ctx, vm.Spec.Image)
		if err != nil {
			return ctrl.Result{}, err
		}

		vm.Status.ID = created.ID
		vm.Status.Phase = created.Phase
		if err := r.Status().Update(ctx, &vm); err != nil {
			return ctrl.Result{}, err
		}

		return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // check back shortly instead of blocking here
	}

	// VM already exists, poll the provider for whatever it knows right now
	current, err := r.Provider.Get(ctx, vm.Status.ID)
	if err != nil {
		return ctrl.Result{}, err
	}

	vm.Status.Phase = current.Phase
	if err := r.Status().Update(ctx, &vm); err != nil {
		return ctrl.Result{}, err
	}

	if current.Phase != "Running" {
		return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // still provisioning, keep polling
	}

	return ctrl.Result{}, nil
}

Two things are worth calling out. First, this is only reachable at all because we've registered a watch on VirtualMachine. The API server tells us the moment one is created or edited, which is what triggers the first call.

Second, every branch ends by writing to vm.Status, mapping whatever the provider told us onto the resource. Kubernetes never talks to the provider directly. The only way anyone finds out a VM is running is because our reconciler wrote it into status.

Failure Handling & Retries

Notice the reconciler above never retries anything itself. When r.Provider.Create or r.Provider.Get fails (like because of a network blip or the mock provider not being up yet), it just returns the error. That's deliberate. Returning an error is how we ask controller-runtime to requeue with exponential backoff on our behalf. This means we don't need to hand-roll a retry loop, and a persistently unreachable provider doesn't get flooded with retries.

The one thing worth being careful about is treating every failure the same way. A timeout talking to the provider is worth retrying. A VirtualMachine whose spec.image the provider will never accept is not. Retrying that forever just produces a busy loop that never succeeds.

We'll leave surfacing that distinction through status conditions to the exercises. The reconciler above only has one failure mode to worry about, since the mock provider never rejects a request outright.

Finalizer

If we delete a VirtualMachine right now, Kubernetes removes the object and we're left with an orphaned VM the provider still thinks is running. A finalizer closes that gap: it's a string on the object that tells Kubernetes "don't actually delete this until I say so."

const vmFinalizer = "compute.example.com/vm-cleanup"

func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var vm computev1.VirtualMachine
	if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	if !vm.DeletionTimestamp.IsZero() {
		// being deleted, deprovision through the provider before letting it go
		if controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
			if vm.Status.ID != "" {
				if err := r.Provider.Delete(ctx, vm.Status.ID); err != nil {
					return ctrl.Result{}, err
				}
			}
			controllerutil.RemoveFinalizer(&vm, vmFinalizer) // safe to let the delete proceed now
			return ctrl.Result{}, r.Update(ctx, &vm)
		}
		return ctrl.Result{}, nil
	}

	if !controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
		controllerutil.AddFinalizer(&vm, vmFinalizer) // register before we ever provision anything
		if err := r.Update(ctx, &vm); err != nil {
			return ctrl.Result{}, err
		}
	}

	// ... provisioning logic from before
	return ctrl.Result{}, nil
}

A kubectl delete on a VirtualMachine with our finalizer present doesn't remove it. Rather, it sets deletionTimestamp and waits.

Our reconciler sees that on the next call, deprovisions the VM through the provider, and only then removes the finalizer. At this point Kubernetes finally deletes the object. If there's no finalizer, there's no guarantee that cleanup ever runs.

Predicate

There's a subtle bug already sitting in the reconciler above. Every time it calls r.Status().Update, that write is itself a change to the object. This triggers our own watch, which calls reconcile again.

Left alone, this doesn't spin forever, since we're recomputing the same status until it settles. But it's still wasted work reconciling in response to writes we made ourselves.

A predicate filters those events that actually enqueue a reconcile, before our code ever runs:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). // drop status only events
		Complete(r)
}

generation only increments when spec changes. Status updates don't touch it. GenerationChangedPredicate uses that to drop events where nothing but status moved, so our own writes stop retriggering us. Then we're back to reconciling only when something meaningful changed, or when we explicitly ask to be requeued.

Owned Resources

A VirtualMachine being Running somewhere isn't very useful on its own, so let's make the operator also create a Secret holding the VM's connection details in-cluster:

func (r *VirtualMachineReconciler) reconcileConnectionSecret(ctx context.Context, vm *computev1.VirtualMachine) error {
	secret := &corev1.Secret{
		ObjectMeta: metav1.ObjectMeta{
			Name:      vm.Name + "-connection",
			Namespace: vm.Namespace,
		},
		StringData: map[string]string{"id": vm.Status.ID},
	}

	if err := controllerutil.SetControllerReference(vm, secret, r.Scheme); err != nil {
		return err // ties the Secret's lifecycle to this VirtualMachine
	}

	return r.Patch(ctx, secret, client.Apply, client.ForceOwnership, client.FieldOwner("vmoperator")) // create or update, either way
}

SetControllerReference is what makes this an owned resource, it stamps an owner reference onto the Secret pointing back at the VirtualMachine.

Two things fall out of that for free. Deleting the VirtualMachine now cascades, Kubernetes garbage collects the Secret automatically, and there's no finalizer needed since it's an in-cluster object, not an external one.

And if we add Owns(&corev1.Secret{}) alongside For(&computev1.VirtualMachine{}) in SetupWithManager, an edit or deletion of the Secret itself re-triggers reconciliation of its owning VirtualMachine. So if someone deletes it by hand, we notice and recreate it.

The same pattern, SetControllerReference call, and Owns() registration creates a second owned resource: a Service fronting the VM in-cluster. That's two different resource kinds owned by one VirtualMachine, which is all multiple owned resources means in practice. There's nothing more to it than calling the same pattern twice for different types.

Cross-Resource Reconciliation

Every VirtualMachine so far talks to one hardcoded provider endpoint. Real deployments need that to be configurable, and it's rarely a one-off: fifty VirtualMachines in the same AWS account share the same endpoint and credentials, and a hundred more might live in Azure instead.

We could put an endpoint field directly on VirtualMachineSpec, but rotating a credential or fixing a typo would then mean editing every VirtualMachine that uses it, one at a time. Pulling that into its own object lets many VirtualMachines reference it by name instead, so a single edit propagates to all of them.

Now let's add a second, small CRD:

type ProviderConfigSpec struct {
	Endpoint string `json:"endpoint"`
}

And a providerRef field on VirtualMachineSpec pointing at one by name. The interesting part isn't the new type. It's what happens when a ProviderConfig changes.

A VirtualMachine doesn't watch ProviderConfig directly, and there's no owner reference between them, so a plain Owns() won't do it. Instead, we watch the type and map each event onto every VirtualMachine that references it:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
		Owns(&corev1.Secret{}).
		Owns(&corev1.Service{}).
		Watches(
			&computev1.ProviderConfig{}, // not owned, so Owns() won't catch its changes
			handler.EnqueueRequestsFromMapFunc(r.findVirtualMachinesForProviderConfig),
		).
		Complete(r)
}

func (r *VirtualMachineReconciler) findVirtualMachinesForProviderConfig(ctx context.Context, obj client.Object) []reconcile.Request {
	var vms computev1.VirtualMachineList
	if err := r.List(ctx, &vms, client.InNamespace(obj.GetNamespace())); err != nil {
		return nil
	}

	var requests []reconcile.Request
	for _, vm := range vms.Items {
		if vm.Spec.ProviderRef == obj.GetName() { // only re-enqueue VMs that actually reference this config
			requests = append(requests, reconcile.Request{NamespacedName: client.ObjectKeyFromObject(&vm)})
		}
	}
	return requests
}

This is cross-resource reconciliation: one resource's change causing a different resource type entirely to reconcile, connected only by a field value rather than ownership.

It's also where the theme of this whole project comes back around. ProviderConfig is what would hold real credentials and a real endpoint for AWS, Azure, or GCP in a production version of this operator. The mock provider is standing in for exactly that boundary.

RBAC

None of the above works without permission to act on it. The manifest just has to list what we actually touch: VirtualMachine and ProviderConfig objects, the VirtualMachine status subresource separately, and the Secret/Service objects we create:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vmoperator-manager-role
rules:
  - apiGroups: ['compute.example.com']
    resources: ['virtualmachines', 'providerconfigs']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
  - apiGroups: ['compute.example.com']
    resources: ['virtualmachines/status'] # separate rule, it's a separate subresource
    verbs: ['get', 'update', 'patch']
  - apiGroups: ['']
    resources: ['secrets', 'services']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vmoperator-manager-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vmoperator-manager-role
subjects:
  - kind: ServiceAccount
    name: vmoperator-controller-manager
    namespace: vmoperator-system

Note: VMOperator manages a resource entirely outside the cluster through a hand-rolled HTTP client, but it's not a novel pattern. Crossplane, AWS Controllers for Kubernetes, Cluster API, and cert-manager all reconcile external or non-Kubernetes state through CRDs the same way. These resources are worth reading once this pattern feels familiar.

Another note: we're keeping VMOperator's scope narrow on purpose. Resizing a running VM, stopping and restarting one, taking snapshots, and supporting more than one real provider behind ProviderConfig are all natural extensions of what's here, and a reasonable next step once the core loop feels solid.

Part 4: Production & Deployment

Now that VMOperator works, let's see how to package and deploy it and improve it for production.

Packaging & Deployment

Everything so far has run as a binary on our own machine. go run against whatever cluster kubectl happens to be pointed at.

A Deployment needs an image instead, so the operator gets a multi-stage Dockerfile: one stage to compile it, and a second, much smaller image to actually run it:

FROM golang:1.26 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /vmoperator ./cmd/manager

FROM gcr.io/distroless/static-debian12
COPY --from=build /vmoperator /vmoperator
USER 65532:65532 # nonroot, matches the security context on the Deployment below
ENTRYPOINT ["/vmoperator"]

The build stage has the full Go toolchain and every source file, none of which need to ship. The final image only has the compiled binary, which covers most of what a container security context later in this part would otherwise have to ask for. There's no shell to get a foothold in even before runAsNonRoot is set.

docker build -t registry.example.com/vmoperator:v0.1.0 .
docker push registry.example.com/vmoperator:v0.1.0

That image is what the Deployment manifest under config/manager/ actually references. With it pushed somewhere the cluster can pull from, the rest of the manifests can go on: the CRDs, RBAC, the operator's Deployment, and a Deployment and Service for the mock provider. So it's no longer something we run as a side process on our own machine either:

kubectl apply -f config/crd/
kubectl apply -f config/rbac/
kubectl apply -f config/manager/

Installing the CRDs before anything else matters. Otherwise the operator's Deployment will crash-loop if it starts and immediately (it tries to watch a resource type the API server has never heard of).

Schema changes are the part that hand-written manifests make us feel directly. Adding a field to VirtualMachineSpec is harmless, but existing objects just don't have it set. Renaming or restructuring one isn't: every stored VirtualMachine was serialized against the old shape.

The CRD's versions list is built for exactly this, as more than one version can be served at once. One is marked storage to say which shape objects are actually persisted as, and a conversion webhook translates between them when a client asks for a version that isn't the stored one.

We don't need this for VMOperator today, since v1 is the only version that's ever existed. But it's why the versions field was a list and not a single value from the very first manifest we wrote.

None of the above replaces a person running kubectl apply by hand forever. A CI pipeline that builds the operator's image, pushes it, and applies the manifests on merge to main is the natural next step. This is ordinary CI/CD, nothing operator-specific about it once the manifests themselves are in Git.

Performance & Resilience

By default, a controller only processes one reconcile at a time. That's fine while we're the only ones testing it, but with hundreds of VirtualMachine objects it means that most of them sit in the workqueue waiting their turn even though nothing about reconciling one blocks reconciling another.

MaxConcurrentReconciles raises that:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
		Owns(&corev1.Secret{}).
		Owns(&corev1.Service{}).
		Watches(&computev1.ProviderConfig{}, handler.EnqueueRequestsFromMapFunc(r.findVirtualMachinesForProviderConfig)).
		WithOptions(controller.Options{MaxConcurrentReconciles: 5}). // five VMs in flight instead of one
		Complete(r)
}

Caching only helps one side of this reconciler. Reading vm back from r.Get is already fast and local, as informers keep that in memory. But r.Provider.Get is a real HTTP round trip every single time, and there's no cache in front of it.

That asymmetry is worth sitting with, because it's the same one from Part 2: in-cluster reads are cheap because Kubernetes built the caching layer for us, and external reads are exactly as expensive as whatever's on the other end of the wire. We could add a short-lived cache in front of the provider client, but it comes with a real cost: a cached Running for a VM that just failed is a lie our status will repeat until the cache expires.

The provider not having a cache in front of it also means nothing is stopping us from hammering it. A burst of reconciles, say every VirtualMachine getting touched at once after a cluster restart, turns into a burst of HTTP calls with no coordination between them. Wrapping the client in a rate limiter caps that independently of whatever backoff the workqueue is already doing on failures:

type Client struct {
	baseURL string
	http    *http.Client
	limiter *rate.Limiter // shared across every reconcile using this client
}

func (c *Client) Create(ctx context.Context, image string) (*VM, error) {
	if err := c.limiter.Wait(ctx); err != nil {
		return nil, err
	}
	// ... existing HTTP call
}

Leader election is the other half of running more than one replica safely. We turned this down to a concept in Part 2, but in practice it's two fields on the manager:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
	LeaderElection:   true,
	LeaderElectionID: "vmoperator-leader",
})

With this set, every replica starts up, but only the one holding the lease actually reconciles. The rest sit ready to take over the moment it doesn't renew in time.

Concurrency also surfaces a race we glossed over in Part 3. Say the reconciler calls r.Provider.Create, the provider creates the VM and returns its id, and then the process crashes before r.Status().Update ever runs. vm.Status.ID is still empty, so the next reconcile sees an object with no VM yet and calls Create again. Now the provider has two VMs for one VirtualMachine.

Nothing about MaxConcurrentReconciles or leader election prevents this. It's a gap in the create step itself, and it only shows up once something can fail between the external call and the write that records it.

Closing it for real means the provider needs to accept an idempotency key, generated once and stored on the object before the first Create call, so a retried create recognizes that it already happened instead of making a second VM.

Security

The ClusterRole from Part 3 works, but it's broader than it needs to be. It grants every verb on secrets and services cluster-wide, when the operator only ever touches the ones it owns.

A tighter version scopes to a single namespace with Role/RoleBinding instead of ClusterRole/ClusterRoleBinding wherever VMOperator is only expected to run in one, and it drops verbs we never call. We never list or watch arbitrary Secrets outside our own, only the ones we create. This is also the RBAC the manifests applied in the previous section were referring to.

Credentials are the other gap. ProviderConfig currently holds a plaintext endpoint, and a real provider needs an API key alongside it, which has no business sitting in a CRD spec anyone with read access to the object can see. It belongs in a Secret, referenced by name instead of embedded:

type ProviderConfigSpec struct {
	Endpoint  string                      `json:"endpoint"`
	SecretRef corev1.LocalObjectReference `json:"secretRef"` // Secret holding the provider's API key
}

The reconciler resolves SecretRef at the point it builds the provider client, reads the key out of the Secret's data, and never logs it or writes it back to anything with wider read access, including the VirtualMachine's own status.

The last piece is the operator's own pod. A container security context that runs as a non-root user sets a read-only root filesystem, drops Linux capabilities it doesn't need, and shrinks what's possible if the binary itself is ever compromised. This is standard practice for any workload, not something specific to operators.

If we'd added an admission webhook anywhere in this guide, its certificates would belong here too. We didn't need one for VMOperator, so we'll leave that as a pointer rather than something to configure.

Observability

The manager exposes a Prometheus endpoint without us writing anything for it. Workqueue depth, reconcile duration, and reconcile error counts are already there per controller.

What isn't there automatically is anything about the provider, so we add a metric the same way any Go service would:

var providerCallDuration = prometheus.NewHistogramVec(
	prometheus.HistogramOpts{
		Name: "vmoperator_provider_call_duration_seconds",
		Help: "Duration of calls to the VM provider, by operation",
	},
	[]string{"operation"},
)

func init() {
	metrics.Registry.MustRegister(providerCallDuration) // shares the manager's existing /metrics endpoint
}

Wrapping each provider call with a timer around this turns "is the provider slow" from a question we'd have to guess at into one we can graph.

Logging benefits from the same instinct. log.FromContext(ctx) inside Reconcile already carries the VirtualMachine's name and namespace on every line if we set that up once in SetupWithManager. Adding vm.Status.ID to that logger right after it's set means every subsequent log line for that reconcile also carries the provider's own identifier for the VM. That one field is what makes it possible to grep a mock provider log and an operator log for the same request and find both sides of the same failure.

Next Steps

In this guide, you learned what a Kubernetes operator is, how to build one from scratch, and how to prepare it for production. You also learned about finalizers, predicates, owned resources, and cross-resource reconciliation along the way.

None of this is specific to managing VMs. The next operator, whatever it manages, is the same shape.

You can also review the resources below to keep learning:

• K8s custom resources

• client-go

• controller-runtime

• Docker docs

The Container Network Interface (CNI) is one of the great black boxes of Kubernetes. Most people who run clusters every day have never once looked inside it. They know the name of their CNI ("we run Calico," "we're on Cilium") the way you know the brand of the alternator in your car: as a label, not as a thing you actually understand.

It lives at the very bottom of the stack, beneath the kubelet, beneath your pods, quietly moving every single packet. And precisely because it never fails loudly on a good day, almost nobody learns what it does.

We won't run helm install cilium. We won't apply a single manifest. Instead, we'll wire up pod networking from scratch, feel exactly where it breaks the moment traffic tries to leave a physical machine, and fix it manually.

By the end, you'll understand it in your bones, not just in theory, why tools like Cilium exist and what they're really solving under the hood.

Who this is for:

• Developers, platform engineers, and SREs who use Kubernetes every day but quietly treat pod-to-pod networking as magic.

• Anyone who has ever watched a pod flip to Running and assumed the network "just works" and wants to know what's actually happening.

What you'll build with your own hands:

• Two isolated network namespaces wired together with a virtual cable (veth pair).

• A three-namespace virtual switch using a Linux bridge, the same trick legacy CNIs use on a single node.

• A deliberately broken two-node setup where a packet gets dropped on the floor, plus the manual fix that makes it work.

• A clear picture of the three jobs every CNI does, and why cloud providers force advanced CNIs like Cilium to use overlays and eBPF.

Note: Every command here needs a Linux host and root. Run this in a throwaway VM or lab environment, not on anything you care about. The whole point is to make a mess and learn from it.

Table of Contents

• Prerequisites

• The Illusion: Kubernetes Routes Zero Packets

• The Foundation: Virtual Ethernet (veth) Pairs

• How to Scale Locally with a Linux Bridge

• The Multi-Node Boundary Problem

• How to Fix It Manually with Direct Routing

• So What Is a CNI, Really?

• The Cloud Catch and Why Cilium Changes the Game

• Conclusion

Prerequisites

Before following along, you'll need:

• Two Linux VMs on the same network for the multi-node section so you can watch traffic cross a real machine boundary.

• The ip command from the iproute2 package (already installed on virtually every modern distro).

• A basic comfort with IP addresses, subnets, and the word "gateway." You don't need to be a network engineer.

• No Kubernetes. That's not a typo. We're going underneath Kubernetes on purpose.

The Illusion: Kubernetes Routes Zero Packets

Here's the uncomfortable truth most people never confront: Kubernetes can't route a single network packet.

Not one. Kubernetes is a orchestrator. It schedules pods, watches their health, and updates state in etcd. But when it comes to actually moving a packet from one container to another, it has zero built-in capability. None.

So how do your pods talk to each other? They rely completely on an external agent to wire up the virtual network plumbing on every node. That agent is the Container Network Interface (CNI). What the CNI does under the hood quietly, is what we would do ourselves to feel the pain and then the solution a CNI provides.

Here's the proof that it's load-bearing. Spin up a brand-new cluster with kubeadm and look at your nodes:

$ kubectl get nodes
NAME       STATUS     ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION                        CONTAINER-RUNTIME
no-cni     NotReady   control-plane,master   52s   v1.35.0   192.168.117.2   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6
worker-1   NotReady   <none>                 46s   v1.35.0   192.168.117.3   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6
worker-2   NotReady   <none>                 40s   v1.35.0   192.168.117.4   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6

NotReady. Every node. The control plane is healthy, etcd is up, the scheduler is alive, and the cluster still flatly refuses to be Ready. If you describe the node, it tells you precisely what's missing:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  Ready            False   Sat, 18 Jul 2026 10:25:51 +0200   Sat, 18 Jul 2026 10:25:20 +0200   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Read that again: your cluster is not Ready until you install a CNI. Not "mostly ready." Not "ready except for networking." NotReady, full stop. Until an external plugin shows up and takes responsibility for the packets Kubernetes itself refuses to touch.

A cluster without a CNI is a telephone exchange with no lines plugged in: every operator is present and ready, but not a single call is able to connect.

So what do most people do at this exact moment? They copy one line from a getting-started page:

kubectl apply -f https://.../calico.yaml

They watch the nodes flip to Ready, and they move on. That's the entire relationship most engineers have with the single component that makes their cluster work. They wing it. It works, so they never ask what "it" is.

This matters because "the network just works" is a dangerous story to tell yourself. The moment something breaks (a pod can't reach a service, cross-node traffic vanishes, a cloud migration mysteriously blackholes packets), you're standing in front of a system you never actually understood. So let's understand it. From the bottom up.

The Foundation: Virtual Ethernet (veth) Pairs

To understand container networking, you first have to understand how Linux isolates it.

When a container (or a Kubernetes pod) is created, the kernel wraps it in an isolated Network Namespace (netns). Think of a fresh network namespace as an island with no bridges to the mainland. By default it's completely blind to the outside world: no interfaces, no IP addresses, and no routing tables. It can't talk to anything, and nothing can talk to it.

So how do we get off the island? With a kernel primitive called a Virtual Ethernet (veth) pair.

A veth pair is a virtual network cable. Whatever packet enters one end immediately pops out the other end, even if the two ends live in different namespaces. Plug one end into the island and the other end into the mainland, and suddenly you have a connection.

Let's wire two isolated namespaces, red and blue, directly together.

# Step 1: Create the isolated network namespaces
sudo ip netns add red
sudo ip netns add blue

# Step 2: Create the virtual ethernet cable (veth pair)
sudo ip link add veth-red type veth peer name veth-blue

# Step 3: Move each end of the cable into its namespace
sudo ip link set veth-red netns red
sudo ip link set veth-blue netns blue

# Step 4: Assign IP addresses and bring the interfaces UP
sudo ip netns exec red ip addr add 10.0.0.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

sudo ip netns exec blue ip addr add 10.0.0.2/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

Now test the connection by pinging blue from inside red:

sudo ip netns exec red ping -c 2 10.0.0.2

In the end, it would look something like this:

• Isolation broken safely: The container transitions from an unreachable, isolated namespace (no IP, no routing) to an addressable endpoint (10.1.1.2) linked directly to the host network.

• Bi-directional traffic flow: Packets originating inside the container can reach external public IP networks, and incoming response packets from the internet can traverse back through the host's eth0 interface (192.168.1.10) directly into the container's veth pairs.

• Zero-latency in-kernel bridging: The veth pair (veth-island <--> veth-mainland) acts as a direct virtual pipe, allowing instant packet transit between distinct Linux network namespaces (netns) without requiring external physical hardware .

The verdict: the ping succeeds. You just manually wired two isolated environments together with nothing but a virtual cable.

But here's the problem with this approach: it scales horribly. It does not scale well because a veth pair is strictly point to point.

Following is the number of pairs need to be configured for the number of containers:

• 2 containers: 1 pair

• 3 containers: 3 pairs

• 4 containers: 6 cables

For ten, you'd need 45 cables to connect every pair.

• Explodes in cable count: The number of veth pairs grow quadratically as containers increase

• Complex to manage: Too many interfaces, routes and rules to configure and maintain.

This is the same reason data centers don't run a physical cable between every pair of servers. You need a switch.

How to Scale Locally with a Linux Bridge

When you need to connect more than two interfaces on a single host, you stop running cables between everything and plug everything into a central hub instead. In the Linux kernel, that hub is a Linux Bridge. It's a software Layer 2 virtual switch (you'll often see it named br0 or cni0).

A bridge does exactly what a physical switch does: it learns MAC addresses and forwards frames across every connected interface in the same broadcast domain.

The pattern changes slightly. Instead of connecting namespaces directly to each other, you attach one end of a veth pair to the namespace, and plug the other end into the host's bridge.

Let's tear down the old setup and build a three-namespace switch: red, blue, and green. They'll all share one broadcast domain.

# Clean up any previous configuration
sudo ip netns del red 2>/dev/null || true
sudo ip netns del blue 2>/dev/null || true
sudo ip netns del green 2>/dev/null || true
sudo ip link del br0 2>/dev/null || true

# Step 1: Create the host switch (bridge) and bring it up
sudo ip link add br0 type bridge
sudo ip link set br0 up

# Step 2: Wire namespace 1 (red) into the bridge
sudo ip netns add red
sudo ip link add veth-red type veth peer name veth-red-host
sudo ip link set veth-red netns red
sudo ip link set veth-red-host master br0
sudo ip link set veth-red-host up
sudo ip netns exec red ip addr add 10.0.0.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

# Step 3: Wire namespace 2 (blue) into the bridge
sudo ip netns add blue
sudo ip link add veth-blue type veth peer name veth-blue-host
sudo ip link set veth-blue netns blue
sudo ip link set veth-blue-host master br0
sudo ip link set veth-blue-host up
sudo ip netns exec blue ip addr add 10.0.0.2/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

# Step 4: Wire namespace 3 (green) into the bridge
sudo ip netns add green
sudo ip link add veth-green type veth peer name veth-green-host
sudo ip link set veth-green netns green
sudo ip link set veth-green-host master br0
sudo ip link set veth-green-host up
sudo ip netns exec green ip addr add 10.0.0.3/24 dev veth-green
sudo ip netns exec green ip link set veth-green up

Because all three namespaces are connected to the shared br0 device, they can communicate freely with each other across the virtual network switch. But how do they actually "find" each other on the network? This is where ARP comes in.

ARP stands for Address Resolution Protocol. It's a fundamental part of local networking. When one computer (or namespace, in our case) wants to talk to another using an IP address, it needs to discover the other computer's hardware address (called a MAC address) to actually send packets on the network.

ARP is the system that allows this to happen — it sends out a broadcast asking "Who has IP address X? Please tell me your MAC address," and the right system answers back.

Thanks to ARP, all the namespaces plugged into br0 can learn each other's MAC addresses automatically and send packets directly within their shared network segment. Let's prove it by pinging every pair, both ways:

# red reaches blue and green
sudo ip netns exec red ping -c 1 10.0.0.2
sudo ip netns exec red ping -c 1 10.0.0.3

# blue reaches red and green
sudo ip netns exec blue ping -c 1 10.0.0.1
sudo ip netns exec blue ping -c 1 10.0.0.3

# green reaches red and blue
sudo ip netns exec green ping -c 1 10.0.0.1
sudo ip netns exec green ping -c 1 10.0.0.2

All six pings succeed. And notice there isn't a single routing rule involved anywhere. Every namespace lives in the same 10.0.0.0/24 subnet on the same Layer 2 switch, so the kernel resolves the whole mesh with plain ARP.

This is exactly how legacy single-node CNIs (the old kubenet) operate. On one machine, it's clean and simple.

But Kubernetes is a distributed system designed to scale across thousands of physical machines. So here's the question that breaks everything: what happens when our namespaces need to leave the host?

The Multi-Node Boundary Problem

Everything so far has lived on one machine. Kubernetes doesn't. So let's do the honest thing: stand up two real VMs and watch the single-node trick fall apart. Don't take my word for it: build this and watch the packet die.

Here's the setup:

• VM 1 (host IP 10.1.44.216): home to the red namespace, pod subnet 10.0.1.0/24.

• VM 2 (host IP 10.1.44.178): home to the blue namespace, pod subnet 10.0.2.0/24.

Two things to notice before we start. First, each node gets its own pod subnet (10.0.1.0/24 on VM 1, 10.0.2.0/24 on VM 2) because if both nodes handed out 10.0.0.x addresses, you'd get IP collisions the instant two pods landed on the same number.

Second, because the subnets now differ, each namespace needs a gateway to route through, and that gateway is its own host's bridge.

Note: this is the cleanup-multinode.sh script that should only be used in case you make any errors while setting up the cross node routes and veth pairs.

#!/usr/bin/env bash
#
# cleanup-multinode.sh
# Tears down the manual multi-node CNI lab (bridge + namespaces + veth
# pairs + cross-node static routes) from "Build a Mental Model for
# Kubernetes CNI by Doing It Manually."
#
# Safe to run on BOTH VMs. Every step is idempotent: anything that was
# never created on this host is skipped instead of erroring out, so
# re-running it is harmless.
#
# Usage:  sudo ./cleanup-multinode.sh
#
set -u

if [[ $EUID -ne 0 ]]; then
  echo "This script needs root. Run:  sudo $0" >&2
  exit 1
fi

echo "==> Deleting network namespaces (this also destroys their veth pairs)..."
ip netns del red  2>/dev/null && echo "    - removed netns 'red'"  || true
ip netns del blue 2>/dev/null && echo "    - removed netns 'blue'" || true

echo "==> Removing any orphaned host-side veth interfaces..."
ip link del veth-red-host  2>/dev/null && echo "    - removed veth-red-host"  || true
ip link del veth-blue-host 2>/dev/null && echo "    - removed veth-blue-host" || true

echo "==> Deleting the bridge..."
ip link del br0 2>/dev/null && echo "    - removed bridge 'br0'" || true

echo "==> Removing cross-node static routes..."
ip route del 10.0.1.0/24 2>/dev/null && echo "    - removed route to 10.0.1.0/24" || true
ip route del 10.0.2.0/24 2>/dev/null && echo "    - removed route to 10.0.2.0/24" || true

echo "==> Disabling IP forwarding (non-persistent; resets on reboot anyway)..."
sysctl -w net.ipv4.ip_forward=0 >/dev/null

# --- Optional: undo the 'Common Gotchas' tweaks, ONLY if you applied them ---
# On a throwaway lab VM, leaving FORWARD at ACCEPT or rp_filter at 0 is
# usually harmless, so these are opt-in. Uncomment whatever you changed.
# sysctl -w net.ipv4.conf.all.rp_filter=1 >/dev/null
# iptables -P FORWARD DROP

echo
echo "==> Teardown complete. Verifying nothing is left behind:"
echo "--- namespaces (expect: no red/blue) ---"
out=$(ip netns list);                          echo "${out:-  (none)}"
echo "--- bridges (expect: no br0) ---"
out=$(ip -br link show type bridge 2>/dev/null); echo "${out:-  (none)}"
echo "--- lab routes (expect: none) ---"
out=$(ip route | grep -E '10\.0\.[12]\.0/24'); echo "${out:-  (none)}"

On VM 1 (10.1.44.216), build the bridge, wire up red, and turn the host into a router:

# Make the host a router so it can transit packets that aren't its own
sudo sysctl -w net.ipv4.ip_forward=1

# Build the bridge and give it a gateway IP for VM 1's pod subnet
sudo ip link add br0 type bridge
sudo ip addr add 10.0.1.254/24 dev br0
sudo ip link set br0 up

# Wire the red namespace into the bridge
sudo ip netns add red
sudo ip link add veth-red type veth peer name veth-red-host
sudo ip link set veth-red netns red
sudo ip link set veth-red-host master br0
sudo ip link set veth-red-host up
sudo ip netns exec red ip addr add 10.0.1.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

# Point the namespace's default route at its bridge gateway
sudo ip netns exec red ip route add default via 10.0.1.254

On VM 2 (10.1.44.178), do the mirror image for blue:

sudo sysctl -w net.ipv4.ip_forward=1

sudo ip link add br0 type bridge
sudo ip addr add 10.0.2.254/24 dev br0
sudo ip link set br0 up

sudo ip netns add blue
sudo ip link add veth-blue type veth peer name veth-blue-host
sudo ip link set veth-blue netns blue
sudo ip link set veth-blue-host master br0
sudo ip link set veth-blue-host up
sudo ip netns exec blue ip addr add 10.0.2.1/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

sudo ip netns exec blue ip route add default via 10.0.2.254

Both hosts are routers now. Both namespaces are wired up. Ping blue on VM 2 from red on VM 1:

# On VM 1
sudo ip netns exec red ping -c 3 10.0.2.1

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.

--- 10.0.2.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

100% packet loss. The packet is dropped on the floor, exactly as promised, but now you've seen it with your own eyes.

Here's the part worth proving to yourself: the packet really does leave VM 1. It just never arrives at VM 2. Run tcpdump on both boxes and ping again:

# Detect your physical NIC once (enp1s0, ens3, eth0, ...)
NIC=$(ip route get 1.1.1.1 | grep -oP 'dev \K\S+')

# On VM 1: the echo requests march out the door
sudo tcpdump -ni "$NIC" icmp
IP 10.0.1.1 > 10.0.2.1: ICMP echo request, id 5, seq 1, length 64
IP 10.0.1.1 > 10.0.2.1: ICMP echo request, id 5, seq 2, length 64

# On VM 2: dead silence. Nothing ever shows up.
sudo tcpdump -ni "$NIC" icmp
(no output)

So where does it die? Follow the life and death of that packet:

To keep it even simpler, the flow is as follows:

Red Pod (10.0.1.1)
      │
      ▼
eth0
      │
      ▼
veth-red
      │
      ▼
br0 (10.0.1.254)
      │
      ▼
VM 1 Routing Table
(No route for 10.0.2.0/24)
      │
      ▼
Default Route (0.0.0.0/0)
      │
      ▼
Physical NIC (eth0)
      │
      ▼
LAN Gateway (192.168.1.1)
      │
      ▼
❌ No route for 10.0.2.0/24
(Packet Dropped)
      │
      ▼
VM 2 Never Receives the Packet

That's the multi-node boundary problem in one sentence: your per-node scripts are completely blind to the rest of the cluster's topology. VM 1 built its island, VM 2 built its island, and neither has any idea the other exists.

How to Fix It Manually with Direct Routing

The fix is almost insultingly small. VM 1 doesn't need a smarter network. It needs a map. We just have to tell each host one fact it's missing: "the other node's pod subnet lives behind the other node's physical IP." That's a single static route per side. Nothing gets rebuilt: the bridges, namespaces, and forwarding you set up a moment ago all stay exactly as they are.

On VM 1 (10.1.44.216), teach it where VM 2's pods live:

# VM 2's pods (10.0.2.0/24) are reachable via VM 2's physical IP
sudo ip route add 10.0.2.0/24 via 10.1.44.178

On VM 2 (10.1.44.178), teach it the way back:

# VM 1's pods (10.0.1.0/24) are reachable via VM 1's physical IP
sudo ip route add 10.0.1.0/24 via 10.1.44.216

That's it. Two lines. Re-run the exact same ping from red on VM 1:

sudo ip netns exec red ping -c 3 10.0.2.1

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.
64 bytes from 10.0.2.1: icmp_seq=1 ttl=62 time=0.412 ms
64 bytes from 10.0.2.1: icmp_seq=2 ttl=62 time=0.388 ms
64 bytes from 10.0.2.1: icmp_seq=3 ttl=62 time=0.401 ms

--- 10.0.2.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

It works. (See ttl=62? Your packet started at 64 and lost one hop on each host it was forwarded through, proof it crossed two routers to get there.) The packet now completes the full journey:

The simpler flow looks like:

1. Pod (10.0.1.1)
        │
        ▼
2. veth-red
        │
        ▼
3. VM 1 br0 (10.0.1.254)
        │
        ▼
4. VM 1 Routing Table
   ✅ Static route:
   10.0.2.0/24 → 10.1.44.178
        │
        ▼
5. VM 1 Physical NIC
        │
        ▼
6. Direct Link
   VM 1 → VM 2
        │
        ▼
7. VM 2 Physical NIC
        │
        ▼
8. VM 2 Routing Table
   ✅ 10.0.2.0/24 is directly connected
        │
        ▼
9. VM 2 br0
        │
        ▼
10. Blue Pod (10.0.2.1)
        │
        ▼
✅ Reply follows the same path back

That one line changed everything. Instead of dumping the packet at your LAN's clueless gateway, VM 1 now hands it directly to VM 2, which knows exactly which local namespace owns 10.0.2.1. The reply follows the mirror route home. You just hand-built cross-node pod networking.

Now sit with how painful that was. Two nodes took a stack of careful commands and a hand-written route on each side.

Imagine a thousand nodes, pods being created and destroyed every second, each one needing a fresh IP and a route on every other node in the cluster. Doing that by hand isn't just tedious. It's impossible.

So What Is a CNI, Really?

Everything you just did by hand (the namespaces, the veth pairs, the bridges, the IP assignment, the routes) is exactly what a Container Network Interface automates dynamically, at scale, the instant a pod is scheduled.

When you apply a pod manifest, the CNI plugin intercepts the lifecycle event and performs three core jobs:

1. Namespace and interface provisioning: It creates the network namespace, generates the veth pair, and attaches it to the bridge (or its own datapath), cleanly, every time, with no fat-fingered typos.

2. IP Address Management (IPAM): It hands out unique, non-colliding subnets per node and leases an individual IP to every single container in the cluster. That "unique Pod CIDR per node" rule you set up manually? IPAM enforces it automatically.

3. Cluster-wide route distribution: It programs the routing so every node knows how to reach pods on every other node: the static routes you wrote by hand, generated and pushed everywhere, kept in sync as nodes and pods come and go.

That's the mental model. A CNI is the thing that does your dozen-command lab a thousand times a second and never makes a mistake.

The Cloud Catch and Why Cilium Changes the Game

Here's the part that surprises people. The manual direct-routing approach we just built works flawlessly in a bare-metal lab. In a modern public cloud (AWS, GCP, Azure), it breaks completely.

Why? Cloud providers don't let arbitrary IP addresses roam across their network fabric. Your 10.0.1.0/24 pod subnet means nothing to the VPC. Unless those IPs are explicitly registered through heavyweight cloud-controller API calls, the underlying network sees pod traffic as illegitimate and drops it: the exact failure from the boundary-problem section, except now the cloud itself is the thing saying "no."

This is where advanced CNIs like Cilium stop playing by the old rules. Instead of leaning on fragile Linux bridges and hand-written host routes, Cilium reaches for two much stronger mechanisms.

• Overlay networks (VXLAN / Geneve). Cilium takes your raw pod packet and encapsulates it, wrapping it inside an ordinary UDP packet addressed from Node 1's physical IP to Node 2's physical IP. To the cloud provider, it looks like completely normal node-to-node host traffic, so it sails straight through every VPC restriction. Your pod's real addresses are hidden inside the envelope.

• eBPF kernel programmability. Traditional CNIs push every packet through the full Linux bridge path and hundreds of sequential iptables rules: slow, and slower as your cluster grows. Cilium replaces that entire pipeline by loading compiled eBPF programs directly into the kernel at the network-interface level. Packets get short-circuited from the pod's veth straight toward the physical NIC, giving you near line-rate performance and deep security visibility for free.

Here's the whole progression in one table:

Look at that last column, then look at the last row. That's the entire value proposition of a CNI in two cells.

Conclusion

You didn't read about Kubernetes networking. You built it, broke it, and fixed it. Here's the mental model you now carry:

1. Kubernetes routes zero packets. It fully delegates the network to a CNI, and that CNI is doing real, physical plumbing on every node.

2. A veth pair is a virtual cable, and it's the atom of container networking: great for two endpoints, useless at scale.

3. A Linux bridge is a virtual switch that connects many namespaces on one host with nothing but Layer 2 and ARP. That's a single-node CNI in a nutshell.

4. The node boundary is where naïve networking dies. Different subnets and an unaware physical network mean cross-node packets get dropped until you teach every host how to route.

5. Static routes plus IP forwarding fix it manually, and doing that by hand for two nodes shows you instantly why nobody does it for a thousand.

6. A CNI automates three jobs: interface provisioning, IPAM, and cluster-wide route distribution.

7. The cloud breaks direct routing, which is precisely why Cilium leans on VXLAN/Geneve overlays and eBPF instead of bridges and iptables.

The next time a pod flips to Running and the network "just works," you'll know the truth: nothing just works. A CNI just did (silently, and at a scale you now truly respect) everything you just did by hand.

From here, the natural next step is to tear down these scripts, deploy Cilium into a real cluster, and watch eBPF orchestrate this entire topology automatically. Having felt the manual pain first, you'll actually appreciate the elegance.

If this helped you build a clearer picture of Kubernetes networking, come say hi:

• LinkedIn: linkedin.com/in/shubhamkatara

• YouTube: @kubesimplify

A developer needs a new staging environment, so they file a ticket. The platform team queues it.

Two weeks later, the environment exists. It's configured slightly differently from the last one, with a naming convention that doesn't match the production setup, missing the observability stack the previous environment had. The developer deploys. Something breaks. Nobody knows why.

The problem isn't the ticket queue. The problem is the absence of a platform: a paved road where developers can self-serve infrastructure, deployments, and environments that are consistent, auditable, and safe without requiring a platform engineer for every request.

An Internal Developer Platform (IDP) solves this. Not by removing platform engineers from the picture, but by shifting their work from executing individual requests to building the systems that execute those requests automatically.

This handbook builds a production-grade IDP from the three CNCF tools that form its core in 2026: Backstage as the developer portal and software catalog, ArgoCD as the GitOps continuous delivery engine, and Crossplane as the Kubernetes-native infrastructure control plane.

By the end, developers on your platform will be able to provision a cloud database, deploy an application to staging, and register a new service in the catalog — all without filing a single ticket.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: IDP Architecture — The Three-Layer Model

• Part 2: ArgoCD — The GitOps Foundation

• Part 3: Crossplane — Infrastructure as Kubernetes Resources

• Part 4: Backstage — The Developer Portal

• Part 5: Wiring It Together — The Golden Path

• Part 6: FinOps Integration — Cost Attribution on the IDP

• Part 7: The Platform Maturity Model — Measuring What You've Built

• Best Practices Summary

• Resources

What You'll Learn

• The three-layer IDP architecture and why each layer must be implemented in a specific order

• How to install and configure ArgoCD with ApplicationSets for multi-environment GitOps delivery

• How to define cloud infrastructure as Kubernetes custom resources using Crossplane Compositions

• How to deploy and configure Backstage with a software catalog and Software Templates

• How to wire Backstage, ArgoCD, and Crossplane together into a single self-service golden path

• How to implement cost attribution on your IDP so every resource provisioned through it carries team and cost center metadata

• How to measure your IDP's maturity using the CNCF Platform Engineering Maturity Model

Let's build it.

Prerequisites

Before following along, you should have:

Knowledge:

• Working familiarity with Kubernetes: you can deploy applications, write YAML manifests, and understand namespaces and RBAC

• Basic GitOps understanding: you know what "Git as source of truth" means in practice

• Comfort with Helm, Terraform HCL, and TypeScript at a reading level

• Understanding of AWS services: EKS, RDS, S3, IAM

Tools and access:

• An EKS cluster running Kubernetes 1.28 or later with at least 3 nodes (m5.xlarge or equivalent)

• kubectl configured and pointing at your cluster

• helm 3.12 or later installed

• AWS CLI v2 configured with admin-level permissions for the provisioning steps

• Node.js 18 or later and Yarn (for Backstage)

• A GitHub organisation you control (for the GitOps repositories and Backstage GitHub integration)

Companion repository:

git clone https://github.com/aayostem/platform-toolkit
cd platform-toolkit

The repository contains all manifests, Helm values files, Crossplane Compositions, and Backstage templates referenced in this guide. Each part maps to a directory in the repo.

Estimated time: The full implementation takes one to two days for an experienced platform engineer. Parts 1–3 can be completed in the morning and produce a working GitOps delivery layer.

Part 1: IDP Architecture — The Three-Layer Model

1.1 What an IDP Actually Is

An Internal Developer Platform isn't a tool. It's a product: a collection of tools, workflows, and abstractions that platform teams build and maintain so that application developers can move fast without managing infrastructure directly.

The distinction matters because it shapes every architectural decision. A tool is installed and configured. A product is designed for users, iterated based on feedback, and measured by whether those users actually adopt it. The platform teams that build the IDPs that developers love think like product managers, not system administrators.

The DORA 2025 report found that nearly 90% of enterprises now have some form of internal platform. But having a platform and having a platform that developers actually use are different things.

The survey found that developer satisfaction with internal platforms varied dramatically. And the gap between satisfied and unsatisfied teams correlated directly with whether the platform team treated the IDP as a product with a roadmap and user research, or as an infrastructure project with a ticket queue.

The three tools in this guide — Backstage, ArgoCD, and Crossplane — are the most widely adopted open-source stack for production IDPs in 2026. But the architecture that connects them matters as much as the tools themselves.

1.2 The Three-Layer Architecture

A production IDP has three distinct layers, each with a single responsibility:

Layer 1: Developer Interface (Backstage)
├── Software catalog — inventory of all services, APIs, and resources
├── Software Templates — self-service forms that trigger provisioning workflows
├── TechDocs — documentation co-located with each catalog entity
└── Plugins — integrations with ArgoCD, Kubernetes, PagerDuty, Grafana

Layer 2: Delivery Layer (ArgoCD)
├── GitOps sync — continuous reconciliation of cluster state to Git
├── ApplicationSets — multi-environment deployment from a single definition
├── Rollout management — progressive delivery with health checks
└── Audit trail — every deployment change linked to a Git commit

Layer 3: Infrastructure Layer (Crossplane)
├── Composite Resources — cloud resources defined as Kubernetes CRDs
├── Compositions — templates that expand a simple claim into full AWS infrastructure
├── ProviderConfigs — credentials and region configuration for each cloud provider
└── Usage tracking — every provisioned resource tagged with team and cost centre

The critical architectural rule: Backstage never talks directly to Kubernetes or cloud APIs. When a developer submits a Software Template in Backstage, the output is a Git commit — a YAML file representing a Crossplane claim or an ArgoCD Application manifest. ArgoCD picks up that commit and applies it to the cluster. Crossplane translates the cluster resource into actual cloud infrastructure.

This indirect path isn't complexity for complexity's sake. It means every infrastructure change is a Git commit, with an author, a timestamp, a pull request, and a review. The audit trail is automatic. The rollback mechanism is git revert.

Developer → Backstage Template → Git commit → ArgoCD → Crossplane → AWS
                                     ↑
                          Single source of truth
                          Full audit trail
                          Rollback = git revert

Here's what the incorrect alternative looks like — Backstage calling cloud APIs directly:

// Bad: Backstage template calling AWS SDK directly
// No audit trail, no rollback, no reconciliation loop
// If the call fails halfway, you have partial infrastructure with no record
import { S3Client, CreateBucketCommand } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "us-east-1" });
await client.send(new CreateBucketCommand({ Bucket: bucketName }));

And the correct approach — Backstage outputting a Crossplane claim to Git:

# Good: Backstage template output — a Crossplane claim committed to Git
# ArgoCD applies it, Crossplane reconciles it, AWS creates the bucket
# Every step is tracked, auditable, and reversible
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: S3Bucket
metadata:
  name: ${{ values.bucket_name }}
  namespace: ${{ values.team_namespace }}
  labels:
    team: ${{ values.team_name }}
    cost-centre: ${{ values.cost_centre }}
    environment: ${{ values.environment }}
spec:
  versioning: true
  encryption: AES256
  region: us-east-1

1.3 Implementation Order

Build in this order. Deviating from it creates integration problems that are difficult to debug:

Step 1: ArgoCD — the delivery foundation everything else depends on
Step 2: Crossplane — infrastructure control plane, delivered by ArgoCD
Step 3: Backstage — the portal, pointing at ArgoCD and Crossplane as backends
Step 4: Wire together — Software Templates that produce GitOps manifests
Step 5: FinOps layer — cost attribution metadata in every provisioned resource

Part 2: ArgoCD — The GitOps Foundation

ArgoCD is a declarative continuous delivery tool for Kubernetes that implements the GitOps pattern. If you haven't used a GitOps tool before, the core idea is simple: your Git repository is the single source of truth for what should be running in your cluster, and ArgoCD continuously reconciles actual cluster state to match it.

If a developer manually changes a resource in the cluster, ArgoCD detects the drift and resyncs from Git. If Git changes, ArgoCD applies the change to the cluster. Human intervention isn't required, and is actively discouraged — the goal is a cluster whose state is always fully explained by what's in Git.

ArgoCD is a CNCF Graduated project, meaning it's production-ready and widely used. It runs as a set of pods in your cluster with a web UI, a CLI, and a REST API. Everything you need to manage deployments across multiple environments lives in one place.

2.1 Installing ArgoCD

# Create the ArgoCD namespace
kubectl create namespace argocd

# Install ArgoCD using the official manifest
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all pods to be running before proceeding
kubectl wait --for=condition=Ready pods \
  --all -n argocd --timeout=300s

# Get the initial admin password
argocd_password=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

echo "ArgoCD initial password: $argocd_password"
echo "Save this somewhere secure before proceeding"

# Port-forward to access the ArgoCD UI locally
kubectl port-forward svc/argocd-server -n argocd 8080:443 &

# Login via CLI
argocd login localhost:8080 \
  --username admin \
  --password "$argocd_password" \
  --insecure

# Change the password immediately
argocd account update-password \
  --current-password "$argocd_password" \
  --new-password "your-secure-password"

2.2 Repository Structure for GitOps

The repository structure ArgoCD watches determines how you manage multiple environments. The pattern that scales best is environment-per-directory, with overlays managed by Kustomize.

Kustomize is a Kubernetes-native configuration management tool that lets you define a base configuration once and layer environment-specific overrides on top of it. This means your staging and production configurations share the same YAML structure but differ in replica counts, image tags, and resource limits.

gitops-repo/
├── apps/
│   ├── base/                    # Shared configuration across all environments
│   │   ├── payment-api/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── user-api/
│   │       ├── deployment.yaml
│   │       ├── service.yaml
│   │       └── kustomization.yaml
│   └── overlays/
│       ├── staging/             # Staging-specific overrides
│       │   ├── payment-api/
│       │   │   └── kustomization.yaml   # Override: 1 replica, staging image tag
│       │   └── kustomization.yaml
│       └── production/          # Production-specific overrides
│           ├── payment-api/
│           │   └── kustomization.yaml   # Override: 3 replicas, pinned image tag
│           └── kustomization.yaml
└── infrastructure/
    ├── crossplane/              # Crossplane installation and providers
    ├── monitoring/              # Prometheus, Grafana
    └── ingress/                 # NGINX or ALB ingress controller

2.3 ApplicationSets — Managing Multiple Environments

An ApplicationSet is an ArgoCD resource that generates multiple Application objects from a single template. Instead of creating one Application manifest per service per environment — which becomes unmanageable at scale — you define one ApplicationSet that covers all services across all environments. A matrix generator combines a list of environments with a Git directory scan to produce every combination automatically:

# applicationset-apps.yaml
# This single resource generates one ArgoCD Application
# for each combination of environment and application directory
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          # Generator 1: environments
          - list:
              elements:
                - environment: staging
                  cluster: https://staging.eks.cluster.local
                - environment: production
                  cluster: https://production.eks.cluster.local

          # Generator 2: application directories in the overlay
          - git:
              repoURL: https://github.com/your-org/gitops-repo
              revision: HEAD
              directories:
                - path: apps/overlays/{{environment}}/*

  template:
    metadata:
      name: "{{environment}}-{{path.basename}}"
      labels:
        environment: "{{environment}}"
        app: "{{path.basename}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/gitops-repo
        targetRevision: HEAD
        path: "apps/overlays/{{environment}}/{{path.basename}}"
      destination:
        server: "{{cluster}}"
        namespace: "{{path.basename}}"
      syncPolicy:
        automated:
          prune: true        # Delete resources removed from Git
          selfHeal: true     # Revert manual cluster changes
        syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=foreground

Verify the ApplicationSet is generating the expected Applications:

# List all generated Applications
kubectl get applications -n argocd

# Expected output: one Application per environment per app
# staging-payment-api    Synced    Healthy
# staging-user-api       Synced    Healthy
# production-payment-api Synced    Healthy
# production-user-api    Synced    Healthy

# Check sync status for a specific application
argocd app get staging-payment-api

2.4 ArgoCD RBAC for Platform Teams

In a multi-team IDP, different teams need different levels of access to ArgoCD. Application teams should be able to view and sync their own applications. Platform teams should have broader access. Nobody should have unrestricted cluster admin through ArgoCD.

The default policy is readonly — every authenticated user can see everything but change nothing:

# argocd-rbac-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Platform team: full access to all applications
    p, role:platform-team, applications, *, */*, allow
    p, role:platform-team, clusters, get, *, allow
    p, role:platform-team, repositories, *, *, allow

    # Application teams: sync and get their own namespace only
    p, role:app-team, applications, get, */staging-*, allow
    p, role:app-team, applications, sync, */staging-*, allow

    # Bind roles to GitHub teams
    g, your-org:platform-engineers, role:platform-team
    g, your-org:developers, role:app-team

  scopes: '[groups]'

Part 3: Crossplane — Infrastructure as Kubernetes Resources

Crossplane is a CNCF Graduated open-source framework that extends Kubernetes into a universal infrastructure control plane.

The core idea: instead of managing cloud resources with separate tools like Terraform or CloudFormation that live outside your cluster, you define cloud resources — RDS databases, S3 buckets, VPCs, IAM roles — as Kubernetes custom resource definitions.

Once you apply a Crossplane resource to the cluster, Crossplane's controllers take over and reconcile the desired state to the actual AWS state, exactly the way Kubernetes reconciles a Deployment to a set of running pods.

The key abstraction Crossplane adds on top of that is the Composite Resource. A platform team defines a high-level PostgreSQLDatabase type that abstracts over the thirty-plus configuration fields an actual RDS instance requires.

Developers interact with the simple type. Crossplane expands it into the full AWS resource configuration behind the scenes, applying the platform team's security and operational standards automatically — standards that developers can't bypass because they never see the underlying fields.

3.1 Installing Crossplane

Crossplane is delivered to your cluster by ArgoCD — the first integration between the two tools. By installing Crossplane through an ArgoCD Application rather than running helm install directly, you make Crossplane itself part of the GitOps-managed infrastructure. Any change to Crossplane's configuration goes through a Git commit and review:

# infrastructure/crossplane/application.yaml
# ArgoCD Application that installs Crossplane via Helm
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: crossplane
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.crossplane.io/stable
    chart: crossplane
    targetRevision: 1.15.0
    helm:
      values: |
        provider:
          packages:
            # AWS provider — manages all AWS resources
            - xpkg.upbound.io/upbound/provider-aws-s3:v1.2.0
            - xpkg.upbound.io/upbound/provider-aws-rds:v1.2.0
            - xpkg.upbound.io/upbound/provider-aws-iam:v1.2.0
  destination:
    server: https://kubernetes.default.svc
    namespace: crossplane-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

# Apply the ArgoCD Application — ArgoCD installs Crossplane
kubectl apply -f infrastructure/crossplane/application.yaml

# Watch Crossplane pods come up
kubectl get pods -n crossplane-system -w

# Verify providers are installed and healthy
kubectl get providers
# Expected:
# NAME                          INSTALLED   HEALTHY   PACKAGE
# upbound-provider-aws-s3       True        True      xpkg.upbound.io/...
# upbound-provider-aws-rds      True        True      xpkg.upbound.io/...

3.2 Provider Credentials

Crossplane needs AWS credentials to provision resources. The recommended approach for EKS is IAM Roles for Service Accounts (IRSA) — a mechanism that lets Kubernetes pods assume IAM roles directly without storing any credentials in the cluster.

The pod's Kubernetes service account is annotated with an IAM role ARN, and AWS automatically provides short-lived credentials when the pod makes API calls. No access keys, no secrets to rotate, and no credentials to accidentally expose:

# Create the IAM role for Crossplane with the necessary AWS permissions
aws iam create-role \
  --role-name CrossplaneProviderRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID:sub":
            "system:serviceaccount:crossplane-system:provider-aws"
        }
      }
    }]
  }'

# Attach the permissions policy (scope this to minimum required in production)
aws iam attach-role-policy \
  --role-name CrossplaneProviderRole \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

# provider-config.yaml
# Configure the AWS provider with IRSA — no static credentials
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: IRSA   # Use the IAM role attached to the provider service account

3.3 Defining a Composite Resource — PostgreSQL Database

This is where the IDP abstraction lives. The platform team defines two YAML files: the CompositeResourceDefinition (XRD), which specifies the shape of what developers can request, and the Composition, which specifies how that request expands into actual AWS resources with platform standards applied.

The XRD is the API contract with developers. Keep it simple — only fields developers genuinely need to control should appear here:

# xrd-postgresql.yaml
# Defines the PostgreSQLDatabase type that developers can request
# Developers never see the RDS-specific configuration below
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresqldatabases.platform.cloudfrugal.com
spec:
  group: platform.cloudfrugal.com
  names:
    kind: XPostgreSQLDatabase
    plural: xpostgresqldatabases
  claimNames:
    kind: PostgreSQLDatabase     # This is what developers create
    plural: postgresqldatabases
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                # Developer-facing fields only — simple and bounded
                storageGB:
                  type: integer
                  minimum: 20
                  maximum: 1000
                  description: "Storage in GB. Min 20, max 1000."
                instanceClass:
                  type: string
                  enum: ["small", "medium", "large"]
                  description: "small=db.t4g.medium, medium=db.r7g.large, large=db.r7g.2xlarge"
                environment:
                  type: string
                  enum: ["staging", "production"]

The Composition is the platform team's implementation. It maps the simple developer fields to the full RDS configuration and enforces platform standards that developers can't override:

# composition-postgresql.yaml
# Defines what a PostgreSQLDatabase claim expands into
# Platform standards (encryption, backup, deletion protection) are applied here
# Developers cannot override them — the platform enforces them
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: postgresql-aws-composition
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: platform.cloudfrugal.com/v1alpha1
    kind: XPostgreSQLDatabase

  resources:
    # The actual RDS instance — expanded from the simple developer claim
    - name: rds-instance
      base:
        apiVersion: rds.aws.upbound.io/v1beta1
        kind: Instance
        spec:
          forProvider:
            region: us-east-1
            engine: postgres
            engineVersion: "15.4"
            # Platform standards — always applied, not developer-configurable
            storageEncrypted: true           # Always encrypted
            backupRetentionPeriod: 7         # Always 7-day backup
            deletionProtection: true         # Always deletion-protected
            multiAZ: false                   # Overridden to true for production (see patches)
            dbSubnetGroupNameSelector:
              matchLabels:
                platform.cloudfrugal.com/subnet-group: private
      patches:
        # Map the developer's simple instanceClass to the actual RDS instance type
        - type: CombineFromComposite
          combine:
            variables:
              - fromFieldPath: spec.instanceClass
            strategy: string
            string:
              fmt: |
                %s
          toFieldPath: spec.forProvider.dbInstanceClass
          transforms:
            - type: map
              map:
                small:  db.t4g.medium
                medium: db.r7g.large
                large:  db.r7g.2xlarge

        # Enable Multi-AZ for production automatically
        - type: FromCompositeFieldPath
          fromFieldPath: spec.environment
          toFieldPath: spec.forProvider.multiAZ
          transforms:
            - type: map
              map:
                staging:    "false"
                production: "true"

        # Copy team labels from the claim to the RDS instance for cost attribution
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels
          toFieldPath: spec.forProvider.tags

A developer requesting a PostgreSQL database now writes this — nothing more:

# Developer creates this in their team's namespace
# No RDS knowledge required. No IAM configuration. No subnet group lookup.
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: PostgreSQLDatabase
metadata:
  name: payment-service-db
  namespace: payments-team
  labels:
    team: payments
    cost-centre: payments-engineering
    environment: staging
spec:
  storageGB: 100
  instanceClass: medium
  environment: staging

Crossplane reconciles this claim to a full RDS instance within minutes, with encryption, backup, and all platform standards applied automatically.

3.4 Verifying Crossplane Resource Provisioning

# Watch the claim status — it should transition to Ready=True
kubectl get postgresqldatabases -n payments-team -w

# Check the composite resource for detailed status
kubectl describe xpostgresqldatabases.platform.cloudfrugal.com

# Verify the actual AWS resource was created
aws rds describe-db-instances \
  --query 'DBInstances[?TagList[?Key==`team` && Value==`payments`]].[DBInstanceIdentifier,DBInstanceStatus]' \
  --output table

Part 4: Backstage — The Developer Portal

Backstage is a CNCF incubating open-source framework originally built by Spotify. It serves as the developer-facing interface of your IDP — the single place where developers discover services, request infrastructure, and find documentation, without needing to know which underlying system provides any of it.

Backstage provides three core capabilities:

1. A software catalog that inventories every service, API, library, and resource in your organisation

2. Software Templates that give developers self-service forms for provisioning infrastructure and scaffolding new services

3. TechDocs that co-locate documentation with the catalog entity it documents so that documentation is always findable from the same place as the service it covers.

Backstage is built in TypeScript with a React frontend and a Node.js backend. It's configured rather than installed: you create a Backstage app, configure it with your organisation's specifics, and deploy it to your cluster.

4.1 Creating and Configuring Backstage

# Create a new Backstage app
npx @backstage/create-app@latest

# When prompted:
# App name: platform-portal
# Choose SQLite for local development, PostgreSQL for production

cd platform-portal

Configure Backstage to connect to your ArgoCD instance and GitHub:

# app-config.production.yaml
app:
  title: Cloudfrugal Platform Portal
  baseUrl: https://platform.your-company.com

backend:
  baseUrl: https://platform.your-company.com
  database:
    client: pg
    connection:
      host: ${POSTGRES_HOST}
      port: 5432
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
      database: backstage

# GitHub integration for catalog discovery and template scaffolding
integrations:
  github:
    - host: github.com
      apps:
        - appId: ${GITHUB_APP_ID}
          webhookSecret: ${GITHUB_WEBHOOK_SECRET}
          clientId: ${GITHUB_CLIENT_ID}
          clientSecret: ${GITHUB_CLIENT_SECRET}
          privateKey: ${GITHUB_PRIVATE_KEY}

# ArgoCD plugin configuration
argocd:
  username: ${ARGOCD_USERNAME}
  password: ${ARGOCD_PASSWORD}
  appLocatorMethods:
    - type: 'config'
      instances:
        - name: main
          url: https://argocd.your-company.com

# Catalog auto-discovery — finds catalog-info.yaml files across your GitHub org
catalog:
  providers:
    github:
      your-org:
        organization: 'your-github-org'
        catalogPath: '/catalog-info.yaml'
        filters:
          branch: 'main'

4.2 The Software Catalog — Registering Services

Every service, API, library, and resource in your platform should be registered in the Backstage catalog via a catalog-info.yaml file committed to the service's repository. Backstage discovers these files automatically through the GitHub integration — no manual registration required once the file exists:

# catalog-info.yaml — committed to each service's repository root
# Backstage discovers this automatically via the GitHub integration
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-api
  title: Payment API
  description: "Core payment processing service. Handles transaction initiation, authorisation, and settlement."
  annotations:
    # Links ArgoCD to show deployment status in the Backstage UI
    argocd/app-name: production-payment-api
    # Links GitHub Actions workflow status
    github.com/project-slug: your-org/payment-api
    # Links Grafana dashboard for this service
    grafana/dashboard-selector: "title=Payment API"
    # Links PagerDuty on-call schedule
    pagerduty.com/service-id: P123456
  tags:
    - payments
    - typescript
    - critical
  links:
    - url: https://payment-api.docs.your-company.com
      title: Documentation
    - url: https://grafana.your-company.com/d/payment-api
      title: Grafana Dashboard
spec:
  type: service
  lifecycle: production
  owner: group:payments-team
  system: payment-platform
  dependsOn:
    - component:user-api
    - resource:payment-service-db
  providesApis:
    - payment-api-v2

4.3 Software Templates — Self-Service Infrastructure

A Software Template is a Backstage form that, when submitted, produces a Git commit. The commit contains whatever YAML, code, or configuration the template defines.

For infrastructure provisioning, the output is a Crossplane claim. For new service scaffolding, the output is a complete service skeleton committed to a new repository.

The key design decision: templates should create pull requests, not merge directly. The PR gives platform teams visibility, gives developers a review moment, and gives everyone an audit trail. Auto-merge policies can eliminate the review step for low-risk provisioning once you've built trust in the template's outputs:

# templates/postgresql-database/template.yaml
# This template gives developers a form to request a PostgreSQL database
# The output is a Crossplane PostgreSQLDatabase claim committed to the GitOps repo
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: postgresql-database
  title: PostgreSQL Database
  description: Provision a managed PostgreSQL database on AWS RDS. Encryption, backups, and deletion protection are configured automatically by the platform.
  tags:
    - database
    - postgresql
    - aws
spec:
  owner: group:platform-team
  type: infrastructure

  # The form developers fill out in the Backstage UI
  parameters:
    - title: Database Configuration
      required: [name, team, environment, storageGB, instanceClass]
      properties:
        name:
          title: Database Name
          type: string
          description: "Lowercase, hyphens only. E.g. payment-service-db"
          pattern: '^[a-z][a-z0-9-]*$'

        team:
          title: Owning Team
          type: string
          description: "Your team name. Used for cost attribution and ownership."
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: Group

        environment:
          title: Environment
          type: string
          enum: [staging, production]
          default: staging

        storageGB:
          title: Storage (GB)
          type: integer
          minimum: 20
          maximum: 1000
          default: 50

        instanceClass:
          title: Instance Size
          type: string
          enum: [small, medium, large]
          enumNames:
            - "Small (db.t4g.medium) — dev/staging workloads"
            - "Medium (db.r7g.large) — moderate production traffic"
            - "Large (db.r7g.2xlarge) — high-throughput production"
          default: small

  # What the template does when submitted
  steps:
    - id: generate-claim
      name: Generate Crossplane Claim
      action: fetch:template
      input:
        url: ./skeleton    # Contains the Crossplane claim YAML template
        values:
          name: ${{ parameters.name }}
          team: ${{ parameters.team | parseEntityRef | pick('name') }}
          environment: ${{ parameters.environment }}
          storageGB: ${{ parameters.storageGB }}
          instanceClass: ${{ parameters.instanceClass }}

    - id: create-pr
      name: Create Pull Request to GitOps Repo
      action: publish:github:pull-request
      input:
        repoUrl: github.com?repo=gitops-repo&owner=your-org
        title: "Platform: Provision PostgreSQL database ${{ parameters.name }} for ${{ parameters.team }}"
        branchName: "provision-db-${{ parameters.name }}-${{ '' | now }}"
        description: |
          Requesting PostgreSQL database provisioned by Crossplane.

          - **Name:** ${{ parameters.name }}
          - **Team:** ${{ parameters.team }}
          - **Environment:** ${{ parameters.environment }}
          - **Storage:** ${{ parameters.storageGB }}GB
          - **Instance:** ${{ parameters.instanceClass }}

          Approve this PR to trigger provisioning. ArgoCD will pick up the change and Crossplane will create the RDS instance within ~5 minutes of merge.
        sourcePath: ./skeleton

  output:
    links:
      - title: View Pull Request
        url: ${{ steps['create-pr'].output.remoteUrl }}
      - title: Track Provisioning in ArgoCD
        url: https://argocd.your-company.com/applications

The template skeleton directory contains the Crossplane claim with template variable placeholders:

# templates/postgresql-database/skeleton/databases/${{ values.name }}.yaml
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: PostgreSQLDatabase
metadata:
  name: ${{ values.name }}
  namespace: ${{ values.team }}-platform
  labels:
    team: ${{ values.team }}
    cost-centre: ${{ values.team }}-engineering
    environment: ${{ values.environment }}
    managed-by: backstage-scaffolder
spec:
  storageGB: ${{ values.storageGB }}
  instanceClass: ${{ values.instanceClass }}
  environment: ${{ values.environment }}

Part 5: Wiring It Together — The Golden Path

The Golden Path is the complete end-to-end workflow: a developer uses Backstage to request infrastructure, that request becomes a Git commit, ArgoCD applies the commit to the cluster, Crossplane provisions the actual AWS resource, and the result appears in both the Backstage catalog and the ArgoCD dashboard.

5.1 The Complete Flow

Developer fills form in Backstage
    ↓
Backstage Software Template renders the Crossplane claim YAML
    ↓
Backstage creates a Pull Request in the GitOps repository
    ↓
Platform engineer (or auto-merge policy) approves and merges the PR
    ↓
ArgoCD detects the new file in the GitOps repository
    ↓
ArgoCD applies the Crossplane claim to the cluster
    ↓
Crossplane reconciles the claim to an actual AWS RDS instance
    ↓
Developer receives the database endpoint via Kubernetes Secret
    ↓
Backstage catalog shows the new resource, owned by the requesting team

5.2 Surfacing Resource Status Back in Backstage

The Backstage Kubernetes plugin pulls live pod and resource status from your clusters and displays it on each catalog entity page. Developers can see whether their service is running, how many replicas are healthy, and whether the last deployment synced — without leaving Backstage or learning kubectl:

# Install the Kubernetes plugin packages
cd platform-portal
yarn --cwd packages/app add @backstage/plugin-kubernetes
yarn --cwd packages/backend add @backstage/plugin-kubernetes-backend

# app-config.production.yaml — add Kubernetes cluster configuration
kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'config'
      clusters:
        - name: production-eks
          url: ${PRODUCTION_CLUSTER_URL}
          authProvider: serviceAccount
          serviceAccountToken: ${PRODUCTION_SA_TOKEN}
          caData: ${PRODUCTION_CA_DATA}
        - name: staging-eks
          url: ${STAGING_CLUSTER_URL}
          authProvider: serviceAccount
          serviceAccountToken: ${STAGING_SA_TOKEN}
          caData: ${STAGING_CA_DATA}

Annotate each catalog entity to link it to its Kubernetes resources:

# In each service's catalog-info.yaml
annotations:
  backstage.io/kubernetes-label-selector: 'app=payment-api'
  backstage.io/kubernetes-namespace: payments-team

5.3 Installing the ArgoCD Plugin

The ArgoCD plugin shows deployment history and sync status directly in the Backstage entity page. When a developer opens the payment-api page in the catalog, they can see the last 10 deployments, the current sync state, and whether the application is healthy — all without opening the ArgoCD UI:

yarn --cwd packages/app add @roadiehq/backstage-plugin-argo-cd

// packages/app/src/components/catalog/EntityPage.tsx
import { EntityArgoCDOverviewCard } from '@roadiehq/backstage-plugin-argo-cd';

// Add to the service entity page layout
const serviceEntityPage = (
  <EntityLayout>
    <EntityLayout.Route path="/" title="Overview">
      <Grid container spacing={3}>
        <Grid item md={6}>
          <EntityAboutCard variant="gridItem" />
        </Grid>
        <Grid item md={6}>
          {/* ArgoCD deployment status — shows sync state and recent deployments */}
          <EntityArgoCDOverviewCard />
        </Grid>
      </Grid>
    </EntityLayout.Route>
  </EntityLayout>
);

Part 6: FinOps Integration — Cost Attribution on the IDP

An IDP that provisions resources without cost attribution creates a new problem: you now have automated infrastructure provisioning with no clear ownership of the bill it generates. Every resource created through the IDP must carry team and cost centre metadata from the moment it's provisioned.

6.1 Mandatory Labels on Every Crossplane Composition

The Crossplane Compositions are where cost attribution is enforced — not in the developer-facing claim, but in the platform layer that the developer can't bypass. These labels flow through to the actual AWS resource as tags, which means they appear in AWS Cost Explorer and can be used to build team-level cost reports:

# In every Composition, add mandatory cost attribution patches
patches:
  # These labels flow to the actual AWS resource as tags
  # They can't be omitted or overridden by the developer claim
  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[team]
    toFieldPath: spec.forProvider.tags[team]

  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[cost-centre]
    toFieldPath: spec.forProvider.tags[cost-centre]

  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[environment]
    toFieldPath: spec.forProvider.tags[environment]

  # Add a managed-by tag to identify all IDP-provisioned resources
  - type: FromCompositeFieldPath
    fromFieldPath: metadata.name
    toFieldPath: spec.forProvider.tags[managed-by]
    transforms:
      - type: string
        string:
          fmt: "idp-crossplane"

6.2 Cost Attribution Query

With mandatory tags on every resource, you can query actual cost by team directly from AWS Cost Explorer:

# Monthly cost breakdown by team — all IDP-provisioned resources
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \
  --granularity MONTHLY \
  --filter '{
    "Tags": {
      "Key": "managed-by",
      "Values": ["idp-crossplane"]
    }
  }' \
  --group-by Type=TAG,Key=team \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Team:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table

Every team that provisions resources through the IDP now has a line on the cost report with their name on it. This is the chargeback model that makes FinOps sustainable at platform scale — attribution is automatic, not manual.

Part 7: The Platform Maturity Model — Measuring What You've Built

The CNCF Platform Engineering Maturity Model defines five levels of platform maturity. Knowing where you sit helps you decide what to build next and communicate progress to engineering leadership.

A complete Backstage + ArgoCD + Crossplane implementation, with cost attribution and Software Templates covering your most common developer requests, puts you at Level 3. Moving to Level 4 requires adding SLO alerting on the platform's own health, running quarterly developer experience surveys, and producing a monthly cost-by-team report from the attribution tags.

The most common mistake at Level 3: building more features instead of measuring adoption. A platform that has 12 Software Templates but only 2 are regularly used hasn't reached Level 3 — it's reached Level 2 with more YAML. Measure which golden paths are used, interview developers who aren't using the portal, and fix the friction before adding capabilities.

Best Practices Summary

✅ Do: Build in order — ArgoCD first, then Crossplane, then Backstage. Each layer depends on the previous one.

✅ Do: Use Backstage as a Git commit generator, not as an infrastructure caller. All infrastructure changes must be auditable Git commits.

✅ Do: Apply cost attribution tags in the Crossplane Composition layer, not in the developer claim. Attribution that developers can bypass will be bypassed.

✅ Do: Start with two or three Software Templates and make them excellent before building more. Template adoption is your most important early metric.

✅ Do: Register every service in the Backstage catalog from day one. The catalog's value is proportional to its coverage.

✅ Do: Deliver Crossplane to your cluster via ArgoCD, not helm install. Everything the IDP manages should itself be managed by the IDP.

❌ Don't: Connect Backstage directly to cloud APIs. No audit trail, no rollback, no reconciliation.

❌ Don't: Give developers the Crossplane XRD directly. The Composition abstraction exists to hide RDS-specific configuration and enforce platform standards. Bypassing it defeats the purpose.

❌ Don't: Build the IDP in isolation and announce it as done. Platform engineering is product engineering. Schedule user interviews after the first two templates are live.

❌ Don't: Skip the ArgoCD RBAC configuration. An IDP that gives all developers cluster-admin through the delivery layer has created a security problem larger than the one it solved.

Resources

• Backstage Documentation — Official reference for plugin development, Software Templates, and catalog configuration

• Crossplane Documentation — CompositeResourceDefinition and Composition reference, provider installation guides

• ArgoCD Documentation — ApplicationSet generator reference, RBAC configuration, and sync policy options

• CNCF Platform Engineering Maturity Model — The maturity framework referenced in Part 7

• AWS Provider for Crossplane — Complete reference for all AWS resource types available through Crossplane

• Backstage Kubernetes Plugin — Setup guide for the Kubernetes resource visibility integration in Part 5

• FinOps Foundation — FinOps for Platform Engineering — Framework reference for the cost attribution model in Part 6

• Companion Repository — All manifests, Compositions, ApplicationSets, and Backstage templates from this guide

• 2025 DORA State of AI-assisted Software Development Report

Yesterday, 10.0.1.45 was your payment service. Today, after a rolling deployment, it's your logging agent. Your payment service is now at 10.0.1.89.

Kubernetes has already updated all the endpoints and service records — but your network policy has no idea. It silently allows traffic through based on an IP address that no longer belongs to the workload you intended to trust.

This is the workload identity problem. IP addresses aren't an identity, they're a location. And in a Kubernetes cluster, location changes constantly. Building security policy on top of IP addresses means your security posture silently degrades every time a pod is scheduled, rescheduled, or scaled.

The answer is cryptographic workload identity: every workload gets a certificate-backed identity that proves who it is, not where it is. Services authenticate each other using those certificates before exchanging any data. If the certificate doesn't match, the connection is refused, regardless of what IP address it came from.

This is what SPIFFE and SPIRE provide. And this is how Cilium enforces it using eBPF, without injecting a sidecar into every pod.

In this article you'll understand how the SPIFFE identity model works, deploy SPIRE to issue cryptographic identities to workloads, and use Cilium's built-in SPIRE integration to enforce mutual TLS between services without touching your application code.

Prerequisites

• Familiarity with Kubernetes RBAC and pod security — this handbook covers the foundations

• Familiarity with TLS certificates and Kubernetes Secrets — this handbook covers cert-manager and certificate concepts

• Helm 3 and the Cilium CLI installed

• A kind cluster — you'll create a fresh one with Cilium as the CNI in this article

• Patience: this is the most complex demo I've covered in this group of articles. SPIRE has more moving parts than anything else covered so far.

All demo files are in the companion GitHub repository.

Table of Contents

• Prerequisites

• The Workload Identity Problem

• How SPIFFE Works

  • SPIFFE IDs and Trust Domains

  • SVIDs: The Cryptographic Identity Document

  • The Trust Bundle

• How SPIRE Works

  • SPIRE Server and SPIRE Agent

  • Node Attestation

  • Workload Attestation

  • SVID Issuance and Rotation

• How Cilium Implements Mutual TLS with SPIFFE

• Demo 1 — Install Cilium with SPIRE Integration

  • Step 1: Install the Cilium CLI

  • Step 2: Create a kind cluster without a default CNI

  • Step 3: Install Cilium with SPIRE enabled

  • Step 4: Verify the installation

• Demo 2 — Enforce Mutual TLS with a CiliumNetworkPolicy

  • Step 1: Deploy a client and server

  • Step 2: Confirm traffic flows without authentication

  • Step 3: Apply a CiliumNetworkPolicy requiring mutual authentication

  • Step 4: Verify authenticated traffic still flows

  • Step 5: Observe the authentication with Hubble (optional)

  • Step 6: Verify that a pod without the matching label is blocked

  • Step 7: Check the workload entries in SPIRE

• Conclusion

• Cleanup (kind)

The Workload Identity Problem

The opening scenario isn't theoretical. In Kubernetes, pods are ephemeral. The scheduler can place a pod on any node, and a pod's IP address is assigned at scheduling time from the node's IP pool.

When a pod is deleted and recreated through a rolling deployment, a node drain, or an autoscaler event, it gets a new IP address. If you've written a NetworkPolicy that says, "allow traffic from this IP", that policy is now pointing at nothing, or worse, at a different workload.

Kubernetes service names help here for east-west traffic — a Service name resolves consistently regardless of which pods back it. But a NetworkPolicy based on a Service name is still a label selector match, not a cryptographic assertion. Any pod that can spoof the right labels can bypass it.

What you actually want is this: before service A sends a request to service B, service B proves its identity cryptographically. If service B can't prove it is who it claims to be, service A refuses the connection. This is mutual TLS, and the key question is: where do the identities come from?

SPIFFE answers that question.

How SPIFFE Works

SPIFFE — Secure Production Identity Framework for Everyone — is a CNCF standard that defines a model for workload identity. It doesn't implement anything by itself. It specifies the format of identities, the API for requesting them, and the trust model that makes them verifiable across services, clusters, and clouds. SPIRE is the reference implementation of that specification.

SPIFFE IDs and Trust Domains

A SPIFFE identity is a URI with a specific format:

spiffe://<trust-domain>/<workload-path>

The trust domain is a string that identifies the administrative boundary — typically your organisation, cluster, or environment. Everything within the same trust domain can verify each other's identities. Identities from different trust domains require explicit federation configuration.

Some concrete examples:

spiffe://payments.corp/ns/production/sa/checkout
spiffe://analytics.corp/ns/data/sa/pipeline-worker
spiffe://cluster.local/ns/monitoring/sa/prometheus

The path after the trust domain is arbitrary — it's defined by your SPIRE configuration and typically encodes the Kubernetes namespace and service account of the workload.

SVIDs: The Cryptographic Identity Document

An SVID — SPIFFE Verifiable Identity Document — is how a SPIFFE identity is materialised into something a service can actually use.

There are two SVID formats.

An X.509 SVID is a standard TLS certificate where the SPIFFE ID is embedded in the Subject Alternative Name (SAN) URI field. Because it's a standard X.509 certificate, any TLS library can use it without modification.

The workload presents this certificate in a TLS handshake, and the peer verifies the certificate was signed by a trusted SPIRE server. This is the format used for long-lived connections like gRPC streams.

A JWT SVID is a signed JSON Web Token containing the SPIFFE ID as a claim. It's suitable for request-based authentication over HTTP — pass it in an Authorization header, and the receiving service verifies the signature.

JWT SVIDs are shorter-lived than X.509 SVIDs and scoped to a specific audience to prevent token reuse across services.

For Cilium's mutual authentication, X.509 SVIDs are used. The rest of this article focuses on X.509.

The Trust Bundle

For service A to verify service B's certificate, service A needs to know which Certificate Authority signed it. In SPIFFE, this is called the trust bundle — the set of CA certificates that are trusted within a trust domain.

SPIRE makes the trust bundle available via the Workload API. When a workload requests its identity, it also receives the current trust bundle. When the SPIRE server rotates its CA, it distributes the new trust bundle to all agents, which push it to all workloads. Your application never has to manage trust bundles manually.

How SPIRE Works

SPIRE is the engine that issues SVIDs and manages the identity lifecycle. Understanding its architecture is what makes the Cilium integration make sense.

SPIRE Server and SPIRE Agent

SPIRE has two main components. The SPIRE Server is the central CA. It maintains a registry of workload entries (records that describe which SPIFFE IDs should be issued to which workloads). It issues SVIDs to agents on behalf of workloads, and it's the root of trust for the entire trust domain.

The SPIRE Agent runs on every node as a DaemonSet. It has two jobs. First, it proves to the SPIRE Server that it's running on a legitimate node. This is called node attestation. Second, it exposes the SPIFFE Workload API on a Unix socket on the node, which workloads use to request their SVIDs.

The agent caches SVIDs locally so that a temporary loss of connection to the SPIRE Server doesn't immediately break workload identity.

This split — central server, per-node agents — is deliberate. Workloads never contact the SPIRE Server directly. They only talk to the agent on their own node. The agent mediates all identity requests, which limits the blast radius if a node is compromised.

Node Attestation

When a SPIRE Agent starts up on a new node, it needs to prove its own identity to the SPIRE Server before it can serve identities to workloads. This is node attestation.

In Kubernetes, SPIRE uses PSAT — Projected Service Account Tokens — for node attestation. The agent presents a Kubernetes service account token that is projected specifically for the SPIRE server's audience. The SPIRE Server contacts the Kubernetes API to verify the token, confirms the agent is running in the expected namespace with the expected service account, and issues the agent its own SVID.

This is the reason SPIRE requires specific Kubernetes API flags. The kube-apiserver must be configured to support projected service account tokens with the right audience, which is why the kind cluster config in the demo below sets --api-audiences and --service-account-issuer.

Workload Attestation

Once a node has been attested, its agent can attest workloads. When a workload connects to the Workload API socket and requests an SVID, the agent collects facts about that workload (like its Kubernetes namespace, service account, pod name, and labels) by querying the Kubernetes API. It matches those facts against the workload entries registered in the SPIRE Server. If a matching entry exists, the agent issues the corresponding SVID.

A workload entry looks like this:

SPIFFE ID: spiffe://example.org/ns/production/sa/checkout
Parent ID: spiffe://example.org/spire/agent/k8s_psat/default/<node-uid>
Selectors:
  k8s:ns:production
  k8s:sa:checkout

The selectors describe the Kubernetes facts that must match. A pod running in the production namespace with service account checkout will receive the SPIFFE ID spiffe://example.org/ns/production/sa/checkout. Any other pod will not.

SVID Issuance and Rotation

SVIDs are short-lived by design. The default TTL for X.509 SVIDs in SPIRE is one hour. The SPIRE Agent automatically rotates them in the background — generating a new key pair, requesting a fresh SVID from the server, and making the new SVID available on the Workload API before the old one expires.

Workloads that use the Workload API directly or tools like the SPIFFE CSI driver get the new SVID transparently.

Short-lived credentials are the zero-trust way. If a workload's SVID is compromised, it's only valid for an hour. Compare that to a Kubernetes service account token, which was historically valid forever.

How Cilium Implements Mutual TLS with SPIFFE

Traditional approaches to service mesh mTLS (like Istio or Linkerd) inject a sidecar proxy into every pod. The proxy intercepts all traffic and handles the TLS handshake. The application has no idea TLS is happening. The sidecar adds memory overhead (roughly 50–100MB per pod for Envoy), an extra network hop on every request, and a complex certificate injection mechanism.

Cilium takes a different path. Rather than injecting a proxy, it handles authentication at the network layer using eBPF. The Cilium agent running on each node intercepts connections, performs the mutual TLS handshake using SPIFFE SVIDs, and enforces the authentication result — all in the kernel, without any user-space proxy.

The mechanism works like this. When pod A initiates a connection to pod B, the Cilium agent on pod A's node intercepts the connection. It retrieves pod A's SVID from the SPIRE Workload API. It checks whether there's a CiliumNetworkPolicy requiring mutual authentication for this connection. If there is, it performs a TLS handshake with the Cilium agent on pod B's node, presenting pod A's SVID and requesting pod B's SVID in return.

Both agents verify the SVID against the SPIRE trust bundle. If both SVIDs are valid and the policy allows the connection, it proceeds. If either SVID is invalid or missing, the connection is dropped.

The application on pod A receives data from the application on pod B. Neither application wrote any TLS code. Neither has a sidecar. The authentication happened entirely in the Cilium agents on their respective nodes.

In Cilium's model, the Cilium agent itself gets a SPIFFE identity from SPIRE. It acts as a delegate identity that can request SVIDs on behalf of workloads.

This is slightly different from the standalone SPIRE model where each workload requests its own SVID directly. The Cilium operator registers workload entries in SPIRE automatically based on the Kubernetes Identities it manages, so you don't need to manually create SPIRE entries for every pod.

Demo 1 — Install Cilium with SPIRE Integration

You'll create a kind cluster with Cilium as the CNI and enable its built-in SPIRE integration in a single Helm command.

Step 1: Install the Cilium CLI

# macOS
brew install cilium-cli

# Linux
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar -xzf cilium-linux-amd64.tar.gz -C /usr/local/bin

Step 2: Create a kind Cluster Without a Default CNI

kind's default CNI (kindnet) must be disabled so Cilium can take its place. Save this as kind-cilium.yaml:

# kind-cilium.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true   # Required: let Cilium be the CNI
  kubeProxyMode: none       # Cilium replaces kube-proxy too

kind create cluster --name k8s-mtls --config kind-cilium.yaml

The nodes will be in a NotReady state until Cilium is installed. This is expected because there's no CNI yet.

Step 3: Install Cilium with SPIRE Enabled

Because Step 2 set kubeProxyMode: none, Cilium has to play the kube-proxy role itself. That means its bootstrap pods can't reach the API server via the kubernetes Service ClusterIP, because nothing is routing it yet.

You have to pass the API server's real address up front. Grab the kind control-plane's IP from Docker:

API_SERVER_IP=$(docker inspect k8s-mtls-control-plane \
  --format='{{ .NetworkSettings.Networks.kind.IPAddress }}')
echo "API_SERVER_IP=$API_SERVER_IP"

Then install Cilium with SPIRE:

helm repo add cilium https://helm.cilium.io/
helm repo update

helm upgrade cilium cilium/cilium \
  --install \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=6443 \
  --set authentication.enabled=true \
  --set authentication.mutual.spire.enabled=true \
  --set authentication.mutual.spire.install.enabled=true \
  --set authentication.mutual.spire.install.server.dataStorage.enabled=false

A few of these flags are easy to miss but each is load-bearing:

• kubeProxyReplacement=true: Cilium installs its eBPF-based replacement for kube-proxy. Mandatory whenever the kind config sets kubeProxyMode: none.

• k8sServiceHost / k8sServicePort: direct API server address used during bootstrap, before Cilium can route the Service ClusterIP. On EKS/GKE/AKS you don't need this because kube-proxy is still present during install.

• authentication.enabled=true: required alongside authentication.mutual.spire.enabled=true. The chart's validate.yaml rejects the install with SPIRE integration requires .Values.authentication.enabled=true and .Values.authentication.mutual.spire.enabled=true if you set only the mutual flag.

• dataStorage.enabled=false: switches the SPIRE server from a PVC-backed datastore to in-memory. Fine for a lab cluster, but in production leave this enabled and ensure your cluster has PersistentVolume support.

Notice there's no --wait flag here. On a fresh cluster, --wait will appear to fail with context deadline exceeded because the install is racey by design. The SPIRE server has to schedule on a NotReady node thanks to its tolerations, then Cilium agents come up using SPIRE, then nodes flip to Ready. Let the install return immediately and watch the pods come up over the next ~2 minutes:

kubectl get pods -A -w

Step 4: Verify the Installation

cilium status --wait

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet              cilium-envoy       Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2

Three Cilium agents, one per node, including the control-plane (no taints in the kind config). Check the SPIRE components in the cilium-spire namespace:

kubectl get all -n cilium-spire

NAME                    READY   STATUS    RESTARTS   AGE
pod/spire-agent-2cpsr   1/1     Running   0          3m
pod/spire-agent-klhjx   1/1     Running   0          3m
pod/spire-agent-vhsnc   1/1     Running   0          3m
pod/spire-server-0      2/2     Running   0          3m

NAME                              TYPE        CLUSTER-IP    PORT(S)    AGE
service/spire-server              ClusterIP   10.96.x.x     8081/TCP   3m

NAME                          DESIRED   CURRENT   READY   AGE
daemonset.apps/spire-agent    3         3         3       3m

NAME                             READY   AGE
statefulset.apps/spire-server    1/1     3m

One SPIRE agent per node. The SPIRE server is a StatefulSet with two containers: the server itself plus the SPIRE controller manager, which automatically creates workload registration entries for Cilium identities.

Run a health check on the SPIRE server:

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server healthcheck

Server is healthy.

Verify the SPIRE agents have been attested:

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server agent list

Found 3 attested agents:

SPIFFE ID         : spiffe://spiffe.cilium/spire/agent/k8s_psat/default/<node-uid-1>
Attestation type  : k8s_psat
Expiration time   : 2026-05-17 21:08:47 +0000 UTC
Serial number     : 91532884191503307904684123063465502141
Can re-attest     : true

SPIFFE ID         : spiffe://spiffe.cilium/spire/agent/k8s_psat/default/<node-uid-2>
...

Three agents, one per node, all attested via Kubernetes PSAT. The SPIRE server trusts every node and will issue SVIDs to workloads running on them.

At this point the identity platform is fully in place, but nothing is using it yet. Demo 1 built the machinery that issues cryptographic identities. Demo 2, which we'll walk through next, puts that machinery to work, turning those SVIDs into an enforced mutual-TLS policy between two real services. Keep the cluster from Demo 1 running, as Demo 2 builds directly on it.

Demo 2 — Enforce Mutual TLS with a CiliumNetworkPolicy

Picking up in the same cluster from Demo 1, you'll deploy two services, enforce mutual authentication between them with a CiliumNetworkPolicy, verify that authenticated traffic flows, and confirm that unauthenticated connections are blocked.

Every request here is authenticated with the SVIDs that the SPIRE server you just verified hands out. These two demos are one continuous walkthrough, not standalone exercises.

Step 1: Deploy a Client and Server

This file contains both the server and the client — the client is a sleeping curl pod we'll use to exec into.

# echo-workloads.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
    spec:
      containers:
        - name: echo-server
          image: ealen/echo-server:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo-server
  namespace: default
spec:
  selector:
    app: echo-server
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-client
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-client
  template:
    metadata:
      labels:
        app: echo-client
    spec:
      containers:
        - name: client
          image: curlimages/curl:latest
          command: ["sleep", "infinity"]

kubectl apply -f echo-workloads.yaml
# kubectl rollout status only takes one resource at a time
kubectl rollout status deployment/echo-server -n default
kubectl rollout status deployment/echo-client -n default

Step 2: Confirm Traffic Flows Without Authentication

Before enforcing mTLS, confirm the client can reach the server:

CLIENT=$(kubectl get pod -l app=echo-client -o jsonpath='{.items[0].metadata.name}')
kubectl exec $CLIENT -- curl -s http://echo-server/

You should get a JSON response from the echo server. Traffic flows freely with no authentication.

Step 3: Apply a CiliumNetworkPolicy Requiring Mutual Authentication

Adding authentication.mode: required to a CiliumNetworkPolicy tells Cilium to enforce mutual TLS for matching traffic. Both sides of the connection must present a valid SPIFFE SVID:

# mtls-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: echo-server-mtls
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      app: echo-server
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: echo-client
      authentication:
        mode: required     # Require mutual TLS for this traffic

kubectl apply -f mtls-policy.yaml

Step 4: Verify Authenticated Traffic Still Flows

kubectl exec $CLIENT -- curl -s http://echo-server/

The connection succeeds. Cilium intercepted it, performed the SPIFFE mTLS handshake between the Cilium agents on both pods' nodes, verified both SVIDs, and allowed the traffic through. The application on the client sent a plain HTTP request and received a response — the mutual authentication happened transparently at the network layer.

Step 5: Observe the Authentication with Hubble (Optional)

Hubble is Cilium's observability layer. It needs its own CLI:

# macOS
brew install hubble

# Linux
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --remote-name-all \
  https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz
sudo tar -xzf hubble-linux-amd64.tar.gz -C /usr/local/bin

Enable Hubble in the cluster, then watch flows. cilium hubble enable deploys Hubble Relay and restarts the Cilium agents to switch on the Hubble server inside them, so wait for it to settle before port-forwarding. If you skip the wait, the port-forward connects before Relay is listening, then dies with connection reset by peer / rpc error … EOF:

cilium hubble enable
cilium status --wait          # wait for "Hubble Relay: OK" before continuing

cilium hubble port-forward &

# Watch flows for the echo-server (Ctrl-C to stop)
hubble observe --namespace default --pod echo-server --follow

Trigger another request in a second terminal:

kubectl exec $CLIENT -- curl -s http://echo-server/

In the Hubble output you'll see:

ℹ️  Hubble Relay is available at 127.0.0.1:4245
Jul  7 12:44:42.380: default/echo-client-86d446b8f-9bn5v:47500 (ID:2822) -> default/echo-server-7467b4b54d-5tvkz:80 (ID:30854) policy-verdict:none TRAFFIC_DIRECTION_UNKNOWN ALLOWED (TCP Flags: SYN)
Jul  7 12:44:42.380: default/echo-client-86d446b8f-9bn5v:47500 (ID:2822) -> default/echo-server-7467b4b54d-5tvkz:80 (ID:30854) to-endpoint FORWARDED (TCP Flags: SYN)
Jul  7 12:44:42.381: default/echo-client-86d446b8f-9bn5v:47500 (ID:2822) <- default/echo-server-7467b4b54d-5tvkz:80 (ID:30854) to-endpoint FORWARDED (TCP Flags: SYN, ACK)

The ALLOWED verdict with the policy-verdict reason confirms the CiliumNetworkPolicy matched and authentication was verified. No sidecar involved — this happened in the Cilium agents.

Prefer a graphical view? Enable the Hubble UI. Everything above is the API + terminal path (Relay on port 4245 backs the hubble CLI). Hubble also ships a web dashboard with a live service map — but it's a separate component that cilium hubble enable does not start by default:

# Add the UI (re-runs enable, keeps Relay, adds the hubble-ui deployment)
cilium hubble enable --ui

# Wait for it to be Ready before opening — same race as Relay. Skip this and
# `cilium hubble ui` fails with "connection refused" on port 8081, because the
# UI's frontend container isn't listening yet.
kubectl -n kube-system rollout status deployment/hubble-ui --timeout=90s

# Port-forwards hubble-ui and opens http://localhost:12000 in your browser
cilium hubble ui

Select the default namespace from the dropdown. That's where the demo pods and the policy live. The map is live: it renders edges from flows as they happen, so an idle namespace looks empty. Trigger a request to light it up:

kubectl exec $CLIENT -- curl -s http://echo-server/

You'll see a forwarded edge echo-client → echo-server. Click it (or open the flow table at the bottom) to read the policy-verdict: ALLOWED. Leave the UI open through Step 6. When you run the unauthorized-client test there, its connection shows up as a red dropped edge, the visual counterpart to the curl timeout.

The UI has three parts.

The service map at the top draws each workload identity as a box and each observed connection as an edge colored by verdict: echo-client → echo-server:80 is a solid green (forwarded) edge, while the box labelled default (that's the unauthorized pod, which carries only the namespace identity because it has no app label, so Hubble names it after that) reaches echo-server over a red dashed (dropped) line. The 🔒 lock on echo-server's → 80 TCP port marks that endpoint as mutually authenticated by the policy.

The flow table underneath logs one row per flow: source identity, destination identity, destination port, L7 info, Verdict, and timestamp. This lets you read both outcomes side by side, with echo-client → echo-server rows marked forwarded and default → echo-server rows marked dropped. This is the same allow/deny split as the CLI, one line per packet.

The top bar holds the namespace selector, a flow filter, the Any verdict / Visual toggle, and a live flows/s rate alongside the count of reporting nodes (3/3).

Step 6: Verify That a Pod Without the Matching Label is Blocked

Deploy a third pod without the echo-client label and try to reach the server:

# unauthorized-client.yaml
apiVersion: v1
kind: Pod
metadata:
  name: unauthorized
  namespace: default
spec:
  containers:
    - name: client
      image: curlimages/curl:latest
      command: ["sleep", "infinity"]

kubectl apply -f unauthorized-client.yaml
kubectl wait --for=condition=Ready pod/unauthorized --timeout=60s
kubectl exec unauthorized -- curl -sS --max-time 5 http://echo-server/

curl: (28) Connection timed out after 5000 milliseconds

The connection times out. The CiliumNetworkPolicy only permits ingress from pods with app: echo-client. A pod without that label gets no SVID match and no policy match. Cilium drops the traffic silently.

There are two gotchas to watch out for here. Run kubectl wait before exec. Run exec too soon after apply and you get container not found ("client") because the pod's container hasn't started yet.

And use curl -sS, not plain -s. With only -s, curl swallows the error text and you just see command terminated with exit code 28. That's the same result — 28 is curl's timeout code — but the -S restores the readable message. The fact that it times out (rather than "connection refused") is the signature of a policy drop: the packets are silently blackholed, not actively rejected. A refusal would return instantly with a different error.

Step 7: Check the Workload Entries in SPIRE

Cilium's SPIRE controller manager automatically created SPIFFE identities for the Cilium security identities in this cluster. You can see them:

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server entry show \
  -selector cilium:mutual-auth

Each entry maps a Cilium security identity to a SPIFFE ID. The Cilium operator manages this registry automatically, so you never need to register workloads manually when using Cilium's built-in integration.

Conclusion

IP addresses are location, not identity. And in Kubernetes, location changes with every deployment, so any policy built on address matching silently degrades over time.

Cryptographic workload identity fixes that at the foundation. SPIFFE defines the model (a SPIFFE ID names a workload within a trust domain, an X.509 SVID materialises it into a certificate any TLS library can verify), and SPIRE implements it: the server is the CA and registry, while per-node agents attest via Kubernetes PSAT and issue short-lived, auto-rotating SVIDs.

Cilium wires that identity layer into the network. Add authentication.mode: required to a CiliumNetworkPolicy and its eBPF agents fetch both workloads' SVIDs, run the mutual TLS handshake, and enforce the verdict. There's no sidecar, no application changes, and near-zero overhead versus a service mesh. And you deployed the whole stack in a single Helm command: the complexity lives in the infrastructure, not in your code.

Cleanup (kind)

# Delete demo workloads
kubectl delete deployment echo-server echo-client -n default
kubectl delete service echo-server -n default
kubectl delete pod unauthorized -n default
kubectl delete ciliumnetworkpolicy echo-server-mtls -n default

# Uninstall Cilium (helm doesn't delete the cilium-spire namespace it created)
helm uninstall cilium -n kube-system
kubectl delete namespace cilium-spire

# Delete the cluster (easiest reset on kind)
kind delete cluster --name k8s-mtls

But what happens when the task actually takes time? Something like "go through the last three days of my emails, draft replies for anything urgent, and create Linear tickets for the engineering-related ones." That's not a quick job. It might take minutes, hours, or even longer. And this is just one example.

That's a full workflow, and the moment your server crashes or your process restarts, you lose everything. No retry, and no resume. You're starting from scratch.

That's the problem this article is about.

In this article, you'll build a durable background agent runtime that holds up under real conditions. Dispatch a task, walk away, and it gets done.

Under the hood, it runs on Kubernetes with KEDA autoscaling so workers scale to zero when idle and spin back up the moment work arrives. For crash recovery and durable execution we'll use Temporal, and for agentic capabilities and tool usage we'll use Composio.

What's Covered?

In this tutorial, you'll build a durable background agent runtime that runs on Kubernetes and scales based on actual workload. Here's what you'll learn along the way:

• What an agent loop is and how to build one with Claude and Composio

• How to make long-running agent tasks handle crashes using Temporal

• How to build a gateway that decouples task dispatch from execution

• How to containerize the worker and gateway with Docker

• How to deploy the full system to a local Kubernetes cluster

• How to autoscale workers to zero with KEDA based on queue depth

This gets into some advanced concepts, but follow along and you'll learn a lot along the way.

Table of Contents

• What's the Plan (the Architecture)

  • Dispatching a Task:

  • Running the Task

  • Scaling

• How to Set Up the Project

• Core Components in the Application

  • The Agent Loop

  • Making it Durable with Temporal

  • The Agent Gateway

  • Containerizing the Application

  • Deploying to Kubernetes

  • Autoscaling with KEDA

• Agent in Action

• Conclusion

What's the Plan (the Architecture)

Before diving into the code, it helps to understand how everything fits together.

The system is split into two distinct planes: a control plane that handles user-facing interactions (Next.js frontend), and an execution plane where the actual agent work happens. They never directly call each other, and that separation is intentional.

Here's the flow from start to finish:

Dispatching a Task

When a user submits a goal, the gateway first runs a pre-flight check to verify the required Composio tool connections are active for that user. If they are, it hands the task off to Temporal and returns immediately. The user doesn't wait around.

NOTE: You don't wait for the response to be back from the agent. It just happens all in the background. This isn't your regular chat app. You just launch the task and you forget.

Running the Task

Temporal puts the task on a queue and a worker pod picks it up. The worker runs the agent loop, LLM reasons over the goal, Composio executes the tools, and the result gets written back to Temporal. The frontend automatically polls the gateway for status updates so the user can see progress without doing anything.

Scaling

KEDA watches the Temporal queue depth and scales worker pods based on how much work is pending. When the queue is empty, workers scale down to zero. When tasks come in, they load back up. That's the beauty!

The reason the gateway never touches agent code is straightforward: agent tasks can take minutes, or even hours based on the work, and you don't want that in your API layer. Keeping them separate helps the control plane stay fast regardless of what's happening in the background.

Also, the application supports Linux CronJob-style task scheduling with no human involved. So, having a pre-flight check helps there, because failing fast at dispatch is much better than having a task silently fail because a tool connection was missing.

That's pretty much the high level architecture of our application. To put it simply:

• Kubernetes (k8s): Orchestration Layer

• KEDA: Auto-scaling Layer

• Temporal: Durability Layer

• Composio: Tool Layer

• Any LLM of your choice (in our case, Anthropic) = Reasoning layer

How to Set Up the Project

Before you start, make sure you have the following installed:

• Docker

• k3d (for running a local Kubernetes cluster)

• kubectl

• Helm

• Node.js and Python 3.11+

You'll also need API keys for Anthropic and Composio.

Start by cloning the repository:

git clone https://github.com/shricodev/kron-k8s-agent.git
cd kron-k8s-agent

Next, create the cluster, build the images, and load them in:

# Create the local cluster
k3d cluster create agent --wait

# Build both images and import them into the cluster
bash scripts/build-images.sh
bash scripts/load-images.sh

# Deploy Temporal (creates the temporal namespace, Postgres, and server)
kubectl apply -f infra/k8s/temporal/temporal-dev.yaml

Next, create the namespace and your secret. The secret has to exist before the app gets deployed, since the pods read their keys from it:

# Create the agent namespace
kubectl apply -f infra/k8s/00-namespace.yaml

# Create the secret with your keys (you're supposed to remove the placeholders with the actual values...)
kubectl create secret generic agent-secrets -n agent \
 --from-literal=ANTHROPIC_API_KEY=sk-ant-... \
 --from-literal=COMPOSIO_API_KEY=ak_... \
 --from-literal=JWT_SECRET=$(openssl rand -hex 32)

With that in place, deploy the app and set up autoscaling:

# Install KEDA, then apply the scalers
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda -n keda --create-namespace --wait

kubectl apply -f infra/k8s/40-keda-worker-scaledobject.yaml -f infra/k8s/41-gateway-hpa.yaml

Finally, port-forward the gateway so you can reach it from your machine:

# Port-forward the gateway to localhost:8000
kubectl -n agent port-forward svc/gateway 8000:8000

Point the frontend at http://localhost:8000 and you're ready to start tasks.

Note: You don't need to touch the .env files in apps/worker/ or apps/gateway/ for this. Those are only for running the apps directly on your machine.

In the cluster, the pods get their config from the ConfigMap and the secret you just created gets injected as environment variables at runtime.

Core Components in the Application

The project is huge. Walking through every single line from scratch would turn this into an hours-long read, so instead I'll focus on the core components that actually make the system work.

The Agent Loop

The agent loop is the brain of the entire system. Every time a task gets dispatched, this is what runs.

The idea is simple even if the implementation isn't. Give the LLM a goal, let it reason, let it call tools, feed the results back, and repeat until it's done.

async def run_agent(
user_id: str,
goal: str,
toolkit_hint: str | None = None,
) -> dict:

It takes three things: the user's ID (so Composio knows which connected accounts to use), the goal itself, and an optional toolkit hint. The hint lets you scope which tools get loaded. If the task is clearly a Gmail job, passing "gmail" avoids loading every tool the user has connected.

Before the loop starts, it creates a Composio session and fetches the tools for that user:

session = await create_session(user_id, toolkit_hint);
tools = await get_tools(session);

Then the actual loop runs:

for turn in range(1, settings.max_iterations + 1):
    response = await client.messages.create(
        model=settings.model,
        max_tokens=settings.max_tokens,
        system=SYSTEM_PROMPT,
        tools=tools,
        messages=messages,
    )

if response.stop_reason == "end_turn":
return finish("completed", _extract_text(response.content))

if response.stop_reason == "tool_use":
      # execute the tools, append results, continue
      # ...

Each turn, Claude looks at the goal and the conversation history, then decides what to do next. The stop_reason tells what happened:

• "end_turn": Claude is done. It completed the task and is returning a final answer.

• "tool_use": it wants to call one or more tools. The loop executes those through Composio, appends the results back into the message history, and goes around again.

If a tool call fails, the error gets fed back into the conversation rather than crashing the run:

except ComposioError as exc:
tool_result_blocks = [
    {
        "type": "tool_result",
        "tool_use_id": block.id,
        "content": f"Tool execution failed: {exc}",
        "is_error": True,
    }
        for block in tool_use_blocks
]

The loop runs for at most max_iterations turns which is 20 by default, defined in apps/worker/agent/config.py. If it hits that ceiling without finishing, it returns a max_iterations_reached status instead of hanging indefinitely.

Every run_agent call returns the same dict shape: a status, a summary, and a list of every step taken. That consistent shape is what makes it straightforward for Temporal to store and inspect the result, which we'll get to next.

Making it Durable with Temporal

The agent loop by itself has a real problem. If the worker process crashes partway through a 15-step task, everything is gone. You have no way to know how far it got, and you have to start from scratch.

Workflows and Activities

Temporal splits your code into two distinct pieces: workflows and activities.

A workflow describes what should happen and in what order, but it never does the actual work itself. No network calls, nothing. That constraint is exactly what lets Temporal safely replay it to reconstruct state after a crash.

An activity is where the real work happens. Network calls, LLM requests, tool executions – all of that goes inside an activity. Activities can fail and be retried independently without affecting the workflow state.

In this project, AgentWorkflow in apps/worker/workflows.py is the workflow, and run_agent_activity in apps/worker/activities.py is the activity that wraps the agent loop.

The Workflow

@workflow.defn(name="AgentWorkflow")
class AgentWorkflow:
    def init(self) -> None:
        self._status: str = "running"
        self._result: dict | None = None

When a task gets dispatched, Temporal starts this workflow. It sets up a retry policy and hands all the real work off to the activity:

retry = RetryPolicy(
  (initial_interval = timedelta((seconds = 2))),
  (backoff_coefficient = 2.0),
  (maximum_interval = timedelta((minutes = 2))),
  (maximum_attempts = 5),
  (non_retryable_error_types = ["ValueError", "AuthenticationError"]),
);

result = await workflow.execute_activity(
  run_agent_activity,
  (args = [user_id, goal, toolkit_hint]),
  (start_to_close_timeout = timedelta((minutes = 30))),
  (retry_policy = retry),
);

The start_to_close_timeout is set to 30 minutes and caps at 5 attempts, because agent tasks can genuinely take that long. You can increase or decrease the timer based on your work requirement.

Querying the Workflow

One thing that makes Temporal convenient here is query handlers. The workflow exposes its current status and result without needing a separate database to track it:

@workflow.query
def status(self) -> str:
return self._status

@workflow.query
def result(self) -> dict | None:
return self._result

The gateway can ask Temporal "what is the status of workflow X?" at any point and get a live answer back. That's how the frontend polling works.

The Activity

The activity is straightforward. It wraps run_agent and logs what happens:

@activity.defn(name="run_agent_activity")
async def run_agent_activity(user_id: str, goal: str, toolkit_hint: str | None) -> dict:
    result = await run_agent(user_id=user_id, goal=goal,             toolkit_hint=toolkit_hint)
    return result

Anything that touches the network lives here, not in the workflow. That separation is what lets Temporal do its job.

The Worker

The worker process is what registers everything and starts polling the queue:

worker = Worker(
            client,
            task_queue=temporal_settings.temporal_task_queue,
            workflows=[AgentWorkflow],
            activities=[run_agent_activity, notify_activity],
            max_concurrent_activities=5,
)

It connects to Temporal, registers the workflow and activities, and listens on the task queue. When a task arrives, it picks it up and runs it. This is the process running inside the Kubernetes pod, and it's exactly what KEDA will scale based on queue depth later.

The Agent Gateway

The gateway is a FastAPI app that sits between the user and Temporal. It handles task dispatch, status polling, and cancellation. Crucially, it never runs agent code itself. Its only job is to talk to Temporal and return quickly.

Dispatching a Task

The dispatch endpoint in apps/gateway/routes/tasks.py is where everything begins:

  @router.post("/dispatch", response_model=DispatchResponse)
  async def dispatch(
      body: DispatchRequest,
      user_id: str = Depends(current_user_id),
  ) -> DispatchResponse:
      if body.toolkit:
          access = await check_toolkit_access(user_id, body.toolkit)
          if not access["allowed"]:
              raise HTTPException(
                  status_code=status.HTTP_409_CONFLICT,
                  detail={
                      "error": "toolkit_not_connected",
                      "connect_url": access["connect_url"],
                  },
              )

      workflow_id = f"agent-{user_id}-{uuid.uuid4().hex[:8]}"
      await client.start_workflow(
          WORKFLOW_NAME,
          args=[user_id, body.goal, body.toolkit],
          id=workflow_id,
          task_queue=settings.temporal_task_queue,
          cron_schedule=body.schedule or "",
      )
      return DispatchResponse(workflow_id=workflow_id, status="dispatched")

The request carries three fields: the goal, an optional toolkit name (to not spend time figuring out toolkit names), and an optional cron schedule. The endpoint runs a preflight check, hands the task off to Temporal, and returns the workflow ID immediately. The user doesn't wait for the agent to finish.

Notice the cron_schedule field. Passing a standard cron expression here turns the task into a recurring job. Temporal handles the scheduling itself, no extra infra needed.

The Preflight Check

The preflight check lives in apps/gateway/routes/preflight.py. Before a task gets dispatched, it verifies that the user actually has the required toolkit connected in Composio:

  async def check_toolkit_access(user_id: str, toolkit_hint: str | None) -> dict:
      if not toolkit_hint:
          return {"allowed": True}

      connected = await asyncio.to_thread(
          _has_active_account, composio, user_id, toolkit_hint
      )
      if connected:
          return {"allowed": True}

      connect_url = await asyncio.to_thread(
          _connect_link, composio, user_id, toolkit_hint
      )
      return {"allowed": False, "toolkit": toolkit_hint, "connect_url": connect_url}

If the connection is missing, the gateway returns a connect_url so the user can authorize the app right away. This matters especially for scheduled tasks.

Checking Status

Once a task is running, the frontend polls this endpoint:

  @router.get("/{workflow_id}", response_model=TaskStatusResponse)
  async def get_task(workflow_id: str, ...) -> TaskStatusResponse:
      if not _owns(workflow_id, user_id):
          raise HTTPException(status_code=404, detail="task not found")

      handle = client.get_workflow_handle(workflow_id, run_id=run_id)
      agent_status = await handle.query("status")

      if desc.status == WorkflowExecutionStatus.COMPLETED:
          result = await handle.query("result")

      return TaskStatusResponse(...)

The status and result come straight from Temporal's query handlers that you saw in the workflow. There's no separate status table, and no database write after each step. Temporal is the source of truth.

Containerizing the Application

The gateway and the worker are packaged as two separate images. They share nothing at runtime, which is exactly what you want since they scale independently and have different responsibilities.

Both Dockerfiles live in the /docker directory, and use a multi-stage build.

Why Multi-Stage? 🤔

The builder stage installs compilers and build tools to compile Python packages. The runtime stage gets only the finished dependencies and the application code. There's no point in putting the build tools into the final image.

The Gateway Image

FROM python:3.14-slim-bookworm AS builder

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install -r requirements.txt

FROM python:3.14-slim-bookworm AS runtime

ENV PYTHONUNBUFFERED=1 PATH="/opt/venv/bin:$PATH"

RUN useradd --create-home --uid 10001 app
WORKDIR /app

COPY --from=builder /opt/venv /opt/venv
COPY . .

USER app
EXPOSE 8000
CMD ["python", "main.py"]

The runtime stage copies the venv from the builder, drops into a non-root user, and starts the FastAPI app. And as always, we run it as a non-root user (be a good dev and follow proper security practices 😺).

The Worker Image

The worker Dockerfile is nearly identical with one small difference:

# procps gives us pgrep for the liveness probe
RUN apt-get update \
 && apt-get install -y --no-install-recommends procps \
 && rm -rf /var/lib/apt/lists/*

It installs procps so the Kubernetes liveness probe can run pgrep to check that the process is still alive.

Building and Loading the Images

The build script in scripts/build-images.sh builds both images, passing each app directory as the build context:

docker build \
 -f "$ROOT/docker/Dockerfile.gateway" \
    -t "agent-gateway:$TAG" \
 "$ROOT/apps/gateway"

docker build \
 -f "$ROOT/docker/Dockerfile.worker" \
    -t "agent-worker:$TAG" \
 "$ROOT/apps/worker"

The Dockerfiles live under docker/ but each one is built against its own app directory. That's what COPY . . actually copies.

After building, there's one more step before the images can run in the cluster. A local k3d cluster has no access to your Docker daemon, so images built locally aren't accessible to it. You have to import them explicitly:

k3d image import "agent-gateway:dev" "agent-worker:dev" -c agent

scripts/load-images.sh does this for you. Once the import completes, the cluster can pull the images like it usually does and your pods will start. 🎊

Deploying to Kubernetes

With the images built and loaded into the cluster, the next step is applying the manifests. The setup is organized into two tiers. Tier 1 is the core application: the namespace, config, and deployments. Tier 2 is autoscaling, covered in the next section.

Config and Secrets

Non-sensitive config lives in a ConfigMap at infra/k8s/01-configmap.yaml:

data:
  MODEL: "claude-opus-4-8"
  MAX_TOKENS: "4096"
  MAX_ITERATIONS: "20"
  TEMPORAL_HOST: "temporal-frontend.temporal.svc.cluster.local:7233"
  TEMPORAL_TASK_QUEUE: "agent-tasks"
  GATEWAY_HOST: "0.0.0.0"
  GATEWAY_PORT: "8000"

This is where the Temporal host address comes from. Notice that it uses the full in-cluster DNS name pointing at the Temporal frontend service in the temporal namespace. That address only resolves from inside the cluster, which is fine since both the gateway and the worker run there.

API keys go in a Kubernetes Secret that you create manually and never commit in Git. Both the ConfigMap and the Secret are mounted as environment variables using envFrom in each deployment.

The Gateway Deployment

spec:
  replicas: 2
  containers:
    - name: gateway
      image: agent-gateway:dev
      imagePullPolicy: IfNotPresent
      command:
        [
          "python",
          "-m",
          "uvicorn",
          "main:/app",
          "--host",
          "0.0.0.0",
          "--port",
          "8000",
        ]
      readinessProbe:
        httpGet:
          path: /health
          port: 8000
      resources:
        requests:
          cpu: 100m
          memory: 256Mi
        limits:
          cpu: 500m
          memory: 512Mi

A few things worth noting. imagePullPolicy: IfNotPresent tells Kubernetes to use the locally loaded image instead of trying to pull from a registry. The startup command bypasses the reload=True flag that main.py uses when run directly locally. The readiness probe hits /health before Kubernetes sends any traffic to the pod, so the gateway only receives requests once it's actually up.

The gateway also gets a ClusterIP Service so other pods and the port-forward can reach it:

apiVersion: v1
kind: Service
metadata:
  name: gateway
spec:
  type: ClusterIP
  ports:
    - port: 8000
      targetPort: 8000

The Worker Deployment

# just polls Temporal.
spec:
  replicas: 1
  containers:
    - name: worker
      image: agent-worker:dev
      livenessProbe:
        exec:
          command: ["pgrep", "-f", "worker.py"]
        initialDelaySeconds: 15
        periodSeconds: 20
      resources:
        requests:
          cpu: 250m
          memory: 512Mi
        limits:
          cpu: "1"
          memory: 1Gi

The worker has no Service. It never accepts incoming connections. It connects outward to Temporal and polls for work, so nothing needs to reach it. That's why procps was installed in the Dockerfile.

The worker also gets more resources than the gateway. It's the one running LLM calls and executing tools, so it needs more resources. You can cap it depending on your requirements.

Applying Everything

The deploy script at scripts/deploy.sh applies Tier 1 in the correct order:

kubectl apply -f "$K8S/00-namespace.yaml"
kubectl apply -f "$K8S/01-configmap.yaml"
kubectl apply -f "$K8S/10-gateway-deployment.yaml"
kubectl apply -f "$K8S/20-worker-deployment.yaml"

Order matters here. The namespace has to exist before anything else can be created inside it, and the ConfigMap has to exist before the pods that read from it start up.

Autoscaling with KEDA

Kubernetes scales pods based on CPU or memory. That works fine for the gateway, which handles HTTP requests and actually uses CPU proportional to traffic. But it's the not the right signal for workers.

The worker sits completely idle when no tasks are queued. It doesn't burn CPU waiting. When tasks arrive, it gets busy fast. What you actually want to scale on is queue depth: how many tasks are waiting to be picked up.

That's what KEDA does. It reads external metrics like queue lengths, message counts, or in this case Temporal task queue depth, and scales your deployments accordingly.

Scaling the Worker

The ScaledObject in infra/k8s/40-keda-worker-scaledobject.yaml is what KEDA watches:

spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 10
  cooldownPeriod: 120
  triggers:
    - type: temporal
      metadata:
        endpoint: temporal-frontend.temporal.svc.cluster.local:7233
        namespace: default
        taskQueue: agent-tasks
        queueTypes: "workflow,activity"
        targetQueueSize: "5"
        activationTargetQueueSize: "0"

Let's walk through the important fields:

• minReplicaCount: 0 is the big one. KEDA can scale to zero, which a standard HPA can't do. When the queue is empty, every worker pod shuts down. You pay for nothing while the system is idle.

• activationTargetQueueSize: "0" means KEDA wakes the deployment the moment a single task enters the queue. Zero tasks, zero pods. One task, pods start spinning up.

• targetQueueSize: "5" tells KEDA to target roughly one worker pod per 5 pending tasks. Ten tasks in the queue means two pods.

• cooldownPeriod: 120 adds a 120-second buffer before KEDA scales back down after the queue clears.

• queueTypes: "workflow,activity" watches both queues. Without this, KEDA would only see part of the pending work.

Note: The Temporal scaler requires KEDA v2.17 or later. Make sure your Helm install is on that version or above.

Scaling the Gateway

The gateway gets a plain CPU-based HPA at infra/k8s/41-gateway-hpa.yaml:

spec:
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

CPU is the right signal here because the gateway does real work proportional to incoming HTTP requests. It stays at a minimum of 2 replicas so there's no cold start delay on the API side.

Installing KEDA

KEDA is installed via Helm before applying the ScaledObject:

helm install keda kedacore/keda -n keda --create-namespace --wait
kubectl apply -f infra/k8s/40-keda-worker-scaledobject.yaml -f infra/k8s/41-gateway-hpa.yaml

Once those are applied, the system is fully operational. Submit a task and watch a worker pod appear. Let the queue empty and watch it disappear. That's the whole point.

And just like that, you have a fully durable, autoscaling AI Agent that you can schedule to run anytime. How cool is that? 😎

Agent in Action

Here's a quick demo of the agent in action (running inside a Kubernetes Cluster):

Conclusion

Running AI agents in production is a completely different problem than building them. I tried to focus on that gap here, and hopefully it gave you a solid reference for how to think about durability and scaling. And I hope it also helped you build or understand something different from a regular AI chat application.

The Temporal and KEDA combination is really something you should learn and know more about if you're into building AI agents or doing DevOps in general. Temporal helps with the biggest issue with AI agents (the durability), and KEDA makes sure that you aren't paying for idle workers at 2am (if used in prod) if nothing is running. You aren't just scaling on CPU, but based on events and that is important.

There's a lot of room to extend this from here. You could swap the dev JWT for proper OIDC, or expand the toolkit coverage through Composio to support more of your workflows.

The foundation is there. The rest is just building on top of it.

You can find the complete source code here: shricodev/kron-k8s-agent

Kubernetes can seem intimidating at first, but Amazon EKS (Elastic Kubernetes Service) makes it much more approachable, especially when you have a step-by-step guide to follow.

In this tutorial, we'll walk through exactly how to get a Spring Boot application with a MySQL database up and running on Amazon EKS. I'll take you from from containerizing your app to connecting it to a managed database, all the way to accessing it live in the cloud. Let’s get started.

Table of Contents

• Prerequisites

• Application Overview

• What is Amazon EKS?

• How to Deploy a Spring Boot App with MySQL on Amazon EKS

  • Step 1: Create the VPC

  • Step 2: Set Up the MySQL Database in a Private Subnet

  • Step 3: Deploy EC2 Instance in a Public Subnet

  • Step 4: Create SSH Tunneling for the Database

  • Step 5: Set Up a Simple SpringBoot Application Development

  • Step 6: Configure SpringBoot App for Database

  • Step 7: Dockerize the Spring Boot Application

  • Step 8: Push the Image to Elastic Container Registry (ECR)

  • Step 9: Implement AWS App Load Balancer

  • Step 10: Create a Cluster in EKS

  • Step 11: Install AWS Load Balancing

  • Step 12: Create and Deploy Kubernetes

  • Step 13: Delete Cluster

• Conclusion

Prerequisites

Before you begin, ensure you have the following:

• Basic knowledge of AWS (AWS Console access).

• Basic knowledge of containerization.

• Working knowledge of Kubernetes.

• Basic knowledge of databases.

• Helm installed

• Kubectl installed

• Eksctl installed

• An IDE

Application Overview

The application runs inside an AWS VPC spread across two availability zones for high availability. When a user makes a request, it flows through an Internet Gateway into an AWS Application Load Balancer sitting in the public subnet, which handles incoming traffic via an Ingress rule.

The Load Balancer routes requests to the App Service, which distributes them across multiple App Pods running inside AWS EKS (Elastic Kubernetes Service) in the private subnets.

The Docker images for these pods are pulled from AWS ECR (Elastic Container Registry). For data persistence, the app pods connect to Amazon RDS MySQL databases through a MySQL External Service, with an RDS instance in each availability zone to ensure redundancy.

A NAT Gateway in the public subnet allows the private resources to make outbound internet calls without being directly exposed to the internet.

What is Amazon EKS?

If you've ever tried to manage containers manually, you already know it can get messy pretty quickly, tracking which containers are running, restarting ones that crash, scaling up when traffic spikes... It's a lot.

That's exactly the problem Kubernetes was built to solve. It automates the deployment, scaling, and management of containerized applications. But setting up and maintaining your own Kubernetes cluster from scratch? That's a whole other challenge.

That's where Amazon EKS comes in. EKS is a fully managed Kubernetes service provided by AWS, which means AWS handles the heavy lifting of setting up, securing, and maintaining the Kubernetes control plane for you. You just focus on deploying your application.

How to Deploy a Spring Boot App with MySQL on Amazon EKS

In this section, we’ll cover the steps to follow in deploying your SpringBoot application with MySQL on Amazon EKS.

Step 1: Create the VPC

To create a VPC, log in to the AWS IAM Console and search for “VPC,” then click create VPC.

Select the "VPC and more option:, and give your VPC a name for your project, for example, spring-demo. Set the IPv4 CIDR block to 10.4.0.0/16. For the NAT gateway configuration, select Zonal, then In 1 AZ.

Select None for VPC endpoints configuration. Next, click Create VPC, then click View VPC. This takes you to the VPC resource map.

Step 2: Set Up the MySQL Database in a Private Subnet

First, you need to create the security group for the MySQL and EC2 instance deployment. To do that, navigate to EC2 > Security Groups. For the inbound rule, select Type: All traffic and Source: Anywhere-IPv4. Then click Create security group.

Next, we’ll create the subnet group for the database. To do that, navigate to Aurora and RDS > Subnet groups and click Create DB subnet group. Next, configure the DB subnet to include:

• Name: private-subnet-db

• Description: private-subnet-db

• VPC: Select VPC

• Add subnets: Choose us-east-1a and us-east-1b as the availability zones, then select the private and public subnets

Click Create**.**

Now, navigate to Databases, click Create database, and select Full configuration. Select MySQL as the engine type.

Select the Free tier when choosing a sample template. Next, give your DB a username and a strong password. Choose db.t3.micro as the instance type.

Select your VPC and associated private subnet. Now, uncheck the "Enable auto minor version upgrade" option in the Additional configuration section and click Create database.

While our database initializes, let's create a key pair for the EC2 instance, which will be launched in a public subnet. To do that, navigate to EC2 > Network & Security > Key Pairs and click Create key pair.

Give your key pair a name, for example, ece-db-key-pair. Leave everything else as-is and click Create key pair. This automatically downloads the key-pair into your local machine.

Step 3: Deploy EC2 Instance in a Public Subnet

Now it’s time to create an EC2 instance. To do this, navigate to EC2 > Instances and click Launch instances. Select the key pair you just created in the Key pair section.

Next, in the Network section, select the VPC created earlier for the project. For Auto-assign public IP, choose Enable. Next, choose the Select existing security group option and select the all-access-sg security group created earlier. Next, click Launch instance.

Step 4: Create SSH Tunneling for the Database

For this step, go into your terminal and navigate to the folder where your key pair is downloaded. Run the ls command, and you should see your key pair there.

Next, you need to change the permission of the key pair file. Use the command below:

chmod 0400 ece-db-key-pair.pem 

Now, run the SSH tunneling command below:

ssh -i <YOUR-KEY-PAIR>.pem -f -N -L <LOCAL-PORT>:<YOUR-RDS-ENDPOINT>:<RDS-PORT> <EC2-USERNAME>@<YOUR-EC2-PUBLIC-DNS> -v

• <YOUR-KEY-PAIR>.pem: the name of your downloaded key pair file

• <LOCAL-PORT>:  the port on your laptop (3306 for MySQL, 5432 for PostgreSQL)

• <YOUR-RDS-ENDPOINT>: found in AWS Console > RDS > Your database > Connectivity & Security > Endpoint

• <RDS-PORT>: same as local port (3306 for MySQL, 5432 for PostgreSQL)

• <EC2-USERNAME>: usually ec2-user for Amazon Linux, ubuntu for Ubuntu

• <YOUR-EC2-PUBLIC-DNS>: found in AWS Console > EC2 > Your instance > Public IPv4 DNS

This command lets your laptop or local machine talk directly to your remote database, as if the database were sitting on your own computer.

After running this command, you can open a database tool (like MySQL Workbench, DBeaver, or TablePlus) on your laptop and connect to:

• Host: localhost

• Port: 3306

For this tutorial, I’ll be using the community version of DBeaver. You can use other similar tools, but if you prefer to use the same tool for the purpose of this guide, you can install the community version from the official DBeaver download page.

After download and installation, open the DBeaver client and click the Connect to a database icon in the top-left corner of the app.

Select MySQL and click Next. On the next window, enter your database username and password, and set Server Host to 127.0.0.1.

Click Test Connection.

You should see a window appear on your screen, indicating that the connection is successful.

Click OK and Finish.

Now, on the left panel, you should see your connection. Expand it to see the database structure.

Now, you have successfully created SSH tunneling for your database.

Troubleshooting

While attempting to test the database connection, I initially ran into a “Plugin 'mysql_native_password' is not loaded” error. If you encounter this error, follow the steps below to fix it.

1. On the Connection Settings window, navigate to the Driver properties tab.

2. Look for allowPublicKeyRetrieval and set it to FALSE.

3. Navigate back to the Main tab and click Test Connection.

Everything should work fine now.

Step 5: Set Up a Simple SpringBoot Application Development

To get started, head over to the Spring Initializr website. Rename Artifact to “springboot-mysql-eks”. Then click ADD DEPENDENCIES… to add dependencies for the REST APIs. Search for the following dependencies:

• Spring Web: Build web apps, including RESTful applications using Spring MVC. Uses Apache Tomcat as the default embedded container.

• Spring Data JPA: Persist data in SQL stores with the Java Persistence API using Spring Data and Hibernate.

• IBM DB2 Driver: A JDBC driver that provides access to IBM DB2.

• Lombok: A Java annotation library that helps to reduce boilerplate code.

Next, click GENERATE at the bottom center of the page. This action downloads a zip file to your local machine. Open this file in an IDE, such as VSCode or IntelliJ IDEA. For this tutorial, I use VSCode. In the build.gradle file, you can see all the added dependencies:

dependencies {
   implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
   implementation 'org.springframework.boot:spring-boot-starter-webmvc'
   compileOnly 'org.projectlombok:lombok'
   runtimeOnly 'com.ibm.db2:jcc'
   annotationProcessor 'org.projectlombok:lombok'
   testImplementation 'org.springframework.boot:spring-boot-starter-data-jpa-test'
   testImplementation 'org.springframework.boot:spring-boot-starter-webmvc-test'
   testCompileOnly 'org.projectlombok:lombok'
   testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
   testAnnotationProcessor 'org.projectlombok:lombok'
}

What we're building

The Spring Boot app is a currency exchange rate and conversion app:

We'll be inserting the exchange data into the database table.

To continue with this tutorial, you can clone the project repo here to save time.

In main > java > com.. > model > ExchangeRate, you’ll see the code below:

package com.example.springbootmysqleks.model;

import jakarta.persistence.*;
import lombok.Getter;
import lombok.Setter;

import java.sql.Date;

@Getter
@Setter
@Entity
@Table(name = "exchange-rate")
public class ExchangeRate {
   @Id
   @GeneratedValue(strategy=GenerationType.AUTO)
   private Integer transactionId;
   private String sourceCurrency;
   private String targetCurrency;
   private double amount;
   private Date lastUpdated;
}

This class is essentially a blueprint for storing currency exchange rate data in our database. It uses the libraries and dependencies added earlier. Lombok handles all the repetitive getter/setter boilerplate so you don't have to write it yourself, while JPA annotations like @Entity and @Table tell Spring, "hey, this class maps to a database table called exchange-rate."

Inside the class, there are five fields that become database columns:

• A self-incrementing transactionId as the primary key.

• sourceCurrency and targetCurrency to track which currencies are being converted,

• The amount holding the actual exchange rate

• lastUpdated date, so you always know how fresh your data is.

To store the data, create a repository file in main > java > com.. > repository > ExchangeRateRepository:

package com.example.springbootmysqleks.repository;

import com.example.springbootmysqleks.model.ExchangeRate;
import org.springframework.data.jpa.repository.JpaRepository;

public interface ExchangeRateRepository extends JpaRepository<ExchangeRate, Integer> {
   ExchangeRate findBySourceCurrencyAndTargetCurrency(String sourceCurrency, String targetCurrency);
}

This file acts as the middleman between your code and the database. By simply extending JpaRepository, you instantly get a whole suite of built-in database operations (like save, delete, findAll, and so on) completely for free, without writing a single SQL query.

The interface is typed to work with the ExchangeRate model we just looked at, using Integer as the primary key type.

The one custom method, findBySourceCurrencyAndTargetCurrency, is where Spring's magic really shines. Just by following a naming convention, Spring automatically figures out the SQL query it needs to run, so you can look up an exchange rate by simply passing in two currency codes like "USD" and "EUR" without writing any query logic yourself.

To use the findBySourceCurrencyAndTargetCurrency method, create a service file in main > java > com.. > service > ExchangeRateService with the code below:

package com.example.springbootmysqleks.service;

import com.example.springbootmysqleks.model.ExchangeRate;
import com.example.springbootmysqleks.repository.ExchangeRateRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ExchangeRateService {

   @Autowired
   private ExchangeRateRepository exchangeRateRepository;

   public ExchangeRate addExchangeRate(ExchangeRate exchangeRate) {
       return exchangeRateRepository.save(exchangeRate);
   }

   public double getAmount(String sourceCurrency, String targetCurrency) {
       ExchangeRate exchangeRate =  exchangeRateRepository.findBySourceCurrencyAndTargetCurrency(sourceCurrency, targetCurrency);
       return exchangeRate == null ? 0 : exchangeRate.getAmount();
   }
}

Here, we created a @Service class that interacts with the repository.

The class has two methods, the addExchangeRate, which simply takes an ExchangeRate object and saves it to the database, and getAmount, which takes a source and target currency, uses our custom repository method to look up the matching record, and then either returns the exchange rate amount or a safe default of 0 if no record is found.

That little ternary check (exchangeRate == null ? 0 : exchangeRate.getAmount()) ensures the app doesn't crash if you query a currency pair that doesn't exist in the database yet.

In main > java > com.. > controller > ExchangeRateService, we have the following code:

package com.example.springbootmysqleks.controller;

import com.example.springbootmysqleks.model.ExchangeRate;
import com.example.springbootmysqleks.service.ExchangeRateService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@RestController
public class ExchangeRateController {

   @Autowired
   ExchangeRateService exchangeRateService;

   @GetMapping("/getAmount")
   public double getAmount(@RequestParam String sourceCurrency, @RequestParam String targetCurrency) {
       return exchangeRateService.getAmount(sourceCurrency, targetCurrency);
   }

   @PostMapping("/addExchangeRate")
   public ExchangeRate addExchangeRate(@RequestBody ExchangeRate exchangeRate) {
       return exchangeRateService.addExchangeRate(exchangeRate);
   }

   @GetMapping("/")
   public String getHealth() {
       return "up";
   }

}

The @RestController annotation tells Spring this class will be serving up REST API endpoints, and again @Autowired wires in the service layer automatically.

There are three endpoints:

1. a GET request to /getAmount that accepts sourceCurrency and targetCurrency as query parameters and returns the exchange rate amount

2. a POST request to /addExchangeRate that accepts a full ExchangeRate object as a JSON body and saves it to the database

3. and finally a simple health check endpoint at / that just returns "up",  which is a common pattern in cloud deployments to let load balancers and orchestration tools know the app is alive and running.

Step 6: Configure SpringBoot App for Database

Now, it’s time to configure the application for the database. Navigate to src > main > resources > application.properties, and you should see this:

spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.url=jdbc:mysql://\({MYSQL_HOSTNAME}:\){MYSQL_PORT}/${MYSQL_DATABASE}?createDatabaseIfNotExist=true
spring.datasource.username=${MYSQL_USERNAME}
spring.datasource.password=${MYSQL_PASSWORD}

spring.jpa.hibernate.ddl-auto=update

spring.jpa.show-sql: true

These are the configurations that allow your app to connect with the database.

• spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver: The driver class for the MySQL database.

• spring.datasource.url=jdbc:mysql://\({MYSQL_HOSTNAME}:\){MYSQL_PORT}/${MYSQL_DATABASE}?createDatabaseIfNotExist=true: This is the data source URL in which we are using the MySQL hostname (127.0.0.1), port name, and database name.

• spring.datasource.username=${MYSQL_USERNAME}: your database user name.

• spring.datasource.password=${MYSQL_PASSWORD}: your database password.

One thing to note: the process of configuring environment variables with your actual credentials varies depending on the IDE you're using. If you're using IntelliJ IDEA, this process is pretty straightforward. If you're using VS Code, the process is different.

To configure your actual credentials for the env variables, create a .vscode/launch.json file in your project root folder and paste in the following configuration:

{
 "version": "0.2.0",
 "configurations": [
   {
     "type": "java",
     "name": "Spring Boot App",
     "request": "launch",
     "mainClass": "com.example.springbootmysqleks.SpringbootMysqlEksApplication",
     "projectName": "springboot-mysql-eks",
     "env": {
       "MYSQL_HOSTNAME": "localhost",
       "MYSQL_PORT": "3306",
       "MYSQL_DATABASE": "exchangedb",
       "MYSQL_USERNAME": "root",
       "MYSQL_PASSWORD": "CHANGE_ME"
     }
   }
 ]
}

Configure the credentials to use your actual credentials.

Now, when you run the app, you should be able to see the created exchangedb table in DBeaver:

Use an API testing tool like Postman to send a POST request to the database:

Next, run the select * from exchange_rate er script in the exchangedb SQL script editor:

At the bottom of the editor, you should see the created table from the Postman request.

Now, run a GET request to the endpoint below:

http://localhost:8080/getAmount?sourceCurrency=USD&targetCurrency=EUR&transactionId=1

You should get a 200 OK response with the currency exchange value, for example, 0.93.

Step 7: Dockerize the SpringBoot Application

To Dockerize your application, create a file named Dockerfile and paste in the configuration below:

FROM eclipse-temurin:17-jre-jammy
WORKDIR /app
COPY build/libs/springboot-mysql-eks.jar /app
EXPOSE 8080
CMD ["java", "-jar", "springboot-mysql-eks.jar"]

Our Dockerfile starts by pulling the lightweight eclipse-temurin:17-jre-jammy base image to keep things lean, then sets /app as the working directory inside the container. It copies our compiled Spring Boot JAR file from the local build/libs/ folder into that directory, exposes port 8080 for incoming traffic, and finally runs the app with java -jar when the container starts up.

Next, build the app to create the .jar file. To do that, run the command below:

./gradlew clean assemble 

You should get a successful build output as shown below:

Navigate to build > the libs folder. You’ll see the springboot-mysql-eks file created.

If you run into an “operation couldn’t be completed.” error, try running the export commands to fix this issue. If you’re using a Mac, then run the command below:

brew install openjdk@21

Next, run the export commands:

export JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home

export PATH=\(JAVA_HOME/bin:\)PATH

Then run the ./gradlew clean assemble command again.

Step 8: Push the Image to Elastic Container Registry (ECR)

In this next step, we’ll create an Amazon ECR and push our image to the registry.

To get started, head back into your AWS Console and search for “ECR”. On the ECR page, click Create**.** Then, enter a repository name, for example, “springboot-mysql-eks.” Next, click Create.

Next, select the repo and click View push commands at the top of the page. This presents a window with a bunch of commands you can use to push your image to the registry. Open your terminal and run these commands. You'll need to ensure Docker is running on your local machine before running the commands.

After running the commands, you should see that your image has been successfully pushed to the registry.

Step 9: Implement AWS App Load Balancer

Before getting started with this step, make sure you check out the installation steps and link to additional AWS documentation in the project README. This will help you follow along.

Now, to get started, create a new folder in your root directory named cluster . This is where you'll download the AWS IAM policy for the load balancer. To download the policy, go into your terminal and cd into cluster, then run the command below:

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.14.1/docs/install/iam_policy.json

This command is gotten from the AWS documentation. Now, when you go to the folder, you’ll see an iam_policy.json file automatically generated.

Next, apply the IAM policy using the command below:

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

You should get an output like this in your terminal:

This shows that the IAM policy has been successfully created. To confirm this, head over to the IAM section in your console, navigate to Policies**,** and search for “AWSLoad…”. You should see the policy created there.

The next step is creating the Kubernetes service account. But before that, you need to tag your public and private subnets as described in this documentation.

Now, head over to the VPC dashboard, navigate to Subnets, click into a subnet, and navigate to Tags. Then, click Manage tags.

Click Add new tag, then enter the key/pair value in the documentation.

Step 10: Create a Cluster in EKS

To create a Kubernetes cluster on EKS, you need the eksctl CLI. Follow the instructions in the AWS eksctl documentation to install the CLI. Next, you need a config file schema to create the cluster. To use this schema, create a new file called cluster.yaml in the cluster folder.

Next, paste in the following configurations:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: spring-test-cluster
  region: us-east-1
  version: "1.30"

vpc:
  id: "<your-vpc-id>"
  subnets:
    private:
      us-east-1a:
        id: "<your-private-subnet-1a-id>" # spring-demo-subnet-private1-us-east-1a
      us-east-1b:
        id: "<your-private-subnet-1b-id>" # spring-demo-subnet-private2-us-east-1b
    public:
      us-east-1a:
        id: "<your-public-subnet-1a-id>" # spring-demo-subnet-public1-us-east-1a
      us-east-1b:
        id: "<your-public-subnet-1b-id>" # spring-demo-subnet-public2-us-east-1b

nodeGroups:
  - name: ng-1
    labels: { role: backend }
    instanceType: t2.micro
    desiredCapacity: 3
    minSize: 3
    maxSize: 5
    privateNetworking: true
    ssh:
      allow: true
      publicKeyName: <your-ec2-key-name>
    iam:
      withAddonPolicies:
        imageBuilder: true
        awsLoadBalancerController: true
        autoScaler: true
iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      attachPolicyARNs:
        - arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy

Th ClusterConfig file is used by eksctl to create our EKS cluster called spring-test-cluster in the us-east-1 region, running Kubernetes version 1.30. It plugs into our existing VPC, placing the worker nodes across private subnets in two availability zones us-east-1a and us-east-1b) for high availability, while keeping public subnets available for the load balancer.

The node group spins up t2.micro EC2 instances with a desired count of 3 (scaling up to 5 if needed), all with private networking enabled for security. It also sets up the necessary IAM permissions for the AWS Load Balancer Controller, Auto Scaler, and ECR image access so our cluster has everything it needs to manage traffic and pull our Docker images automatically.

Now, after updating your configuration with your credentials, run the command below:

eksctl create cluster -f cluster.yaml

This creates the cluster. You should see an output like this on your terminal:

Now, in your AWS console, navigate to CloudFormation, and you’ll see your cluster creation process in progress.

Now, when you go into the EC2 instance page, you should see the three nodes created.

Step 11: Install AWS Load Balancing

The next step is installing a load balancer for our application. To get started, run the command below:

 kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"

This installs custom resource definitions (CRDs) for our controller. Next, run the command below to add the Helm chart repo.

helm repo add eks https://aws.github.io/eks-charts

Update your local repo to ensure you have the most recent charts:

helm repo update eks

Next, install the Helm chart:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --version 1.14.0

Next, verify that the controller is installed:

kubectl get deployment -n kube-system aws-load-balancer-controller

You should see this on your terminal:

This indicates that your controller is ready.

Step 12: Create and Deploy Kubernetes

To get started, you'll first need to create a Kubernetes manifest file. For that, we’ll use Helm Chart.

helm create ytchart

The command above creates a folder named ytchart with the templates for the components. In this folder, you need to make some configurations for your use case. First, navigate to ytchart > templates and delete the serviceaccount.yaml file, since we already created the service account earlier.

Next, go to values.yaml and make the following changes:

• For repository, navigate to the ECR service page on the AWS Console and copy the image URI.

• Tag is latest.

• Set database name

mysql:
 databaseName: exchangedb

• Change service account creation to false.

• Scroll down a bit more and change the service type to NodePort and port to 8080.

You also need to store the database username and password using secrets. Navigate to the templates folder and go into the file named secrets.yaml. Here, set your database username and password, then comment out the liveness and readiness probe in deployment.yaml.

Next, we’ll create a service to connect to the database. To do that, navigate to the mysql.yaml file, then for externalName. Navigate to the RDS service page on the AWS console and copy the database endpoint.

Now, in the deployment.yaml file, paste in the following configuration:

          env:
            - name: SPRING_DATASOURCE_URL
              value: jdbc:mysql://spring-mysql:3306/{{ .Values.mysql.databaseName }}?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useUnicode=true&useSSL=false&allowPublicKeyRetrieval=true
            - name: SPRING_DATASOURCE_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mysql-username
                  key: username
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-root-password
                  key: password

You have successfully created environment variables to secure your database credentials.

In the ingress.yaml file, paste in the following configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "spring-microservice-ingress"
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/load-balancer-name: spring-alb-test
  labels:
    app: spring-microservice
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "ytchart.fullname" . }}
                port:
                  number: 8080

This is your configuration for the ingress service.

Run the command below to see all your configuration values:

helm template ytchart/

Next, run the command below to deploy the chart:

helm install mychart ytchart

You should see an output like this on your terminal:

Now, when you run kubectl get all, you should see this:

Now, navigate to EC2 > Load balancers, copy the DNS name, and enter it into a browser. You should see the “up” text. This indicates that your application is working properly.

Now, when you call the API using the DNS URL as such:

http://spring-alb-test-260424558.us-east-1.elb.amazonaws.com/addExchangeRate

You should get a 200 OK response. Congratulations, you have successfully deployed a SpringBoot app in Kubernetes!

Step 13: Delete Cluster

If you’re familiar with AWS and the cloud, you should already be aware of how costly it can be to leave resources running for extended periods, especially when you’re not using them actively.

Now that we've come to the end of this tutorial, it’s time to delete the resources.

These are the resources to delete:

• RDS database.

• Cluster using the command eksctl delete cluster -f cluster.yaml.

• Navigate to VPC and delete the NAT Gateway

Conclusion

Deploying a Spring Boot application with MySQL on Amazon EKS involves a lot of moving parts, but each step builds logically on the last.

In this tutorial, you've gone from setting up a VPC and provisioning a managed database to containerizing your app, pushing it to ECR, and finally orchestrating everything with Kubernetes and an Application Load Balancer.

What you get is a production-grade setup with high availability, private networking, secure credential management, and auto-scaling built in. This is the kind of infrastructure that would take significant manual effort to replicate without managed services like EKS and RDS.

As a next step, consider adding HTTPS support via AWS Certificate Manager, setting up horizontal pod autoscaling, or integrating a CI/CD pipeline to automate future deployments. And remember to clean up your AWS resources when you're done experimenting. Your wallet will thank you.

Six months later, you have fourteen microservices, each holding their own API keys, writing their own retry logic, and failing in their own unique ways.

Nobody knows how much you're spending on tokens or which service is hammering the rate limit. And when OpenAI goes down, everything goes down with it.

That scenario plays out across engineering teams every single day, and the root cause is almost always the same: moving fast with LLMs while skipping the infrastructure thinking that holds everything together at scale.

Fortunately, a well-established architectural pattern solves exactly these problems. If you already run Kubernetes, you're more than halfway to implementing it. That pattern is called the LLM Gateway Pattern, and this article walks you through what it is, why it matters, and how to put it into practice.

Table of Contents

• What Is the LLM Gateway Pattern?

  • How It Works

• The Problem Without a Gateway

• Deploying an LLM Gateway on Kubernetes

  • Storing API Keys Securely

  • Defining Routing Rules in a ConfigMap

  • Scaling the Gateway

  • Wiring Up Observability

• Features of an LLM Gateway

  • Multi-Provider Routing

  • Semantic Caching

  • Rate Limiting Per Consumer

  • Fallback and Failover

  • Token Usage Tracking

• Wrapping Up

What Is the LLM Gateway Pattern?

The LLM Gateway Pattern is an architectural approach where all LLM API traffic from your applications flows through a single, centralized proxy service before reaching any external provider. Think of it as the AI equivalent of an API gateway, except it's purpose-built for the unique challenges that come with language models: token budgets, streaming responses, model routing, semantic caching, and multi-provider fallback.

Instead of every service in your cluster talking directly to OpenAI or Anthropic, they all talk to one internal gateway. That gateway handles authentication, routing, rate limiting, logging, and failover. Your application services stay clean and focused on business logic, while the gateway takes on all the messy operational concerns of working with LLMs at scale.

The pattern itself is not new in concept. Engineers have used API gateways for years to manage REST traffic. What makes LLM gateways distinct is that they understand the specific shape of LLM requests, including token counts, model parameters, prompt structure, and streaming semantics.

How It Works

The core components of an LLM Gateway on Kubernetes are straightforward. Here is the high-level flow:

App Pods send requests to the gateway using a standard OpenAI-compatible API format. Because of this, most existing LLM client libraries work without modification — you just change the base URL to point at your internal gateway service.

The Gateway Service receives each incoming request, authenticates the caller, applies any configured rate limits, checks the cache, selects the appropriate upstream provider based on routing rules, and forwards the request. On the way back, it logs token usage and latency before returning the response to the caller.

ConfigMap holds the routing rules. Which model should handle requests tagged as fast? Which provider should the system fall back to if the primary one is unavailable? All of this lives in configuration, not code, so you can update routing behaviour without redeploying anything.

Secrets hold the actual API keys for each provider. The gateway is the only service in the cluster that needs access to them. Application pods never touch provider credentials directly.

Provider endpoints are the actual LLM APIs: OpenAI, Anthropic, a self-hosted vLLM instance running in your cluster, or any other provider that exposes an OpenAI-compatible interface.

The Problem Without a Gateway

To appreciate why this pattern matters, it helps to look at what happens when you skip it.

1. Scattered Secrets and No Central Control

Every service that calls an LLM needs an API key. In Kubernetes, this usually means creating a Secret per namespace or per deployment.

When that key rotates or gets compromised, you're hunting through dozens of manifests to update it. There's no single place to revoke access or audit who is calling what.

2. No Visibility into Cost or Usage

LLM APIs charge per token. Without a centralized layer collecting usage data, you have no reliable way to know which service is responsible for that spike in your monthly bill.

3. Provider Lock-in at the Application Level

When you hardcode https://api.openai.com into your service, switching to a different provider or routing certain requests to a cheaper model becomes a code change. You need to redeploy your application just to change which model handles a request type.

4. No Caching

Many LLM applications send semantically similar or identical prompts repeatedly. Without a shared caching layer, each one incurs full token costs and full latency. The savings from even basic caching can be significant.

All of these problems compound as your team grows and more services start calling LLMs. The gateway pattern cuts through all of them in one architectural decision.

Deploying an LLM Gateway on Kubernetes

There are several tools that can serve as an LLM gateway in a Kubernetes environment, including LiteLLM Proxy, Portkey, OpenRouter, and Envoy with custom filters.

For the rest of this walkthrough, we'll use LiteLLM Proxy. It ships with a Helm chart, supports over a hundred models across all major providers, and comes with a management UI that makes initial configuration straightforward.

Storing API Keys Securely

Start by creating a Kubernetes Secret that holds your provider API keys. Your gateway pods will consume these credentials as environment variables, which means no provider key ever needs to live inside your application containers:

apiVersion: v1
kind: Secret
metadata:
  name: llm-gateway-secrets
  namespace: ai-platform
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-..."
  ANTHROPIC_API_KEY: "sk-ant-..."

Defining Routing Rules in a ConfigMap

The routing configuration tells the gateway which models are available and how to reach each one. Keeping this in a ConfigMap means you can update your routing rules without touching a single line of application code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-gateway-config
  namespace: ai-platform
data:
  config.yaml: |
    model_list:
      - model_name: gpt-4o
        litellm_params:
          model: openai/gpt-4o
          api_key: os.environ/OPENAI_API_KEY
      - model_name: claude-sonnet
        litellm_params:
          model: anthropic/claude-sonnet-4-20250514
          api_key: os.environ/ANTHROPIC_API_KEY
      - model_name: fast
        litellm_params:
          model: openai/gpt-4o-mini
          api_key: os.environ/OPENAI_API_KEY

With this configuration in place, any application in your cluster can reach the gateway at http://llm-gateway.ai-platform.svc.cluster.local using the standard OpenAI client format, regardless of which actual provider sits behind it.

Scaling the Gateway

Because the gateway is stateless, horizontal scaling is straightforward. You can attach a HorizontalPodAutoscaler to scale based on CPU utilization or request rate:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-gateway-hpa
  namespace: ai-platform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-gateway
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Wiring Up Observability

A gateway you can't observe is a gateway you can't trust, so wiring up monitoring before you go to production is worth the extra hour it takes.

LiteLLM exposes a /metrics endpoint in Prometheus format. You can scrape it with a standard ServiceMonitor if you run the Prometheus Operator, or configure Prometheus directly to target the gateway service.

The metrics that matter most in day-to-day operations are token throughput per model, request latency percentiles, error rates per provider, and cache hit ratio.

Once Prometheus is collecting that data, you can build Grafana dashboards that show token spend broken down by caller, model, and time period. This gives engineering managers and finance teams the cost visibility they've been asking for, and it takes surprisingly little effort to set up once the metrics pipeline is in place.

If you run an OpenTelemetry collector in your cluster, you can also configure the gateway to emit trace spans for every LLM request. This lets you see the full latency breakdown from the moment a user action triggers a call in your application all the way through to the provider response. So when something is slow, you can tell immediately whether the bottleneck sits in your service, the gateway, or upstream with the provider.

Features of an LLM Gateway

Not all gateway implementations are equal, so as your needs grow, these are the core capabilities worth evaluating.

Multi-Provider Routing

A well-built gateway routes requests to different providers based on declarative, configurable rules that live entirely outside your application code. This means that changing a model never requires a redeployment.

Semantic Caching

Rather than only caching byte-for-byte identical prompts, a semantic cache uses embedding similarity to recognise when two different prompts are asking essentially the same thing. This can cut redundant API calls dramatically.

Rate Limiting Per Consumer

The gateway should let you set token budgets and request limits per team, per namespace, or per application, so no single runaway service can starve the rest of your cluster or drive up costs unchecked.

Fallback and Failover

When a primary provider fails or exceeds acceptable latency thresholds, the gateway should automatically retry against a configured fallback. This centralizes logic that is notoriously hard to get right inside individual services.

Token Usage Tracking

Every request should produce a detailed usage record capturing input tokens, output tokens, model, caller identity, and latency. This gives engineering managers the clear, actionable picture of AI spending they need.

Wrapping Up

The LLM Gateway Pattern solves a set of operational problems that every team building on language models at scale will eventually run into. Scattered secrets, invisible costs, inconsistent failure handling, and provider lock-in are all symptoms of the same underlying issue: infrastructure concerns leaking into services that shouldn't have to deal with them.

A centralized gateway on Kubernetes gives your application teams a stable, provider-agnostic interface while giving your platform team the visibility and controls they need to manage cost and reliability effectively. When a provider goes down in the middle of the night, your configured fallback kicks in automatically instead of someone waking up to a page.

Start with LiteLLM Proxy, wire up the Prometheus metrics, build a simple Grafana dashboard, and watch how quickly the pattern pays for itself. Once you have seen what centralized LLM traffic management looks like in practice, it becomes very hard to go back to doing it any other way.

Who this is for:

• Platform engineers, SREs, and security-focused cloud architects who operate mixed on‑prem and cloud Kubernetes estates.

• Teams that need scalable, auditable access from on‑prem workloads to GCP resources (especially GPU instances) while minimizing operational overhead and blast radius.

What you’ll get from this guide:

• The motivation and economics behind a hybrid approach (why GPUs often push workloads to the cloud).

• Common pitfalls with service account keys and how “accidental air gaps” occur in real environments.

• A practical, end‑to‑end pattern that uses Workload Identity Federation to give on‑prem pods short‑lived, auditable access to GCP without embedding keys.

What’s included:

• Conceptual explanations, security tradeoffs, and operational best practices.

• Concrete examples and Kubernetes/Terraform artifacts (linked in the GitHub repo at the end of this article) so you can reproduce the setup in your environment.

Read on for the theory, then follow the hands‑on sections to provision GCP resources, configure federation, enforce policies with CEL and Kyverno, and validate secure, scalable GPU access from your on‑prem Kubernetes clusters.

Note: Kubernetes and Terraform artifacts are linked in the GitHub repo at the end of this article.

Table of Contents

• Prerequisites

• Why Hybrid Cloud Matters

• The Economics of Hybrid: GPUs Changed Everything

• Why Service Account Keys Fail at Scale

• How the Accidental Air Gap Happens

• How Workload Identity Federation Bridges the Gap

• How Kubernetes Identity Works

• How to prepare Google Cloud Platform resources

• How to Use CEL for Fine-Grained Access Control

• How to Inject Credentials Automatically with Kyverno

• How to Grant IAM Permissions to Federated Identities

• How to Verify the Setup

• How to Connect On-Prem Apps to Cloud GPUs

• How to Scale GPU Access with CEL Conditions

• The Security Properties Compared

• The Complete Infrastructure as Code Layout

• How to Run a Proof of Concept with vCluster

• Common Issues and How to Solve Them

• Conclusion

Prerequisites

Before following along, you'll need:

• A Kubernetes cluster that is not GKE (on-premises, bare-metal, or a virtual cluster)

• A Google Cloud project with the following APIs enabled: IAM, Security Token Service (STS), and Workload Identity

• Terraform installed and configured

• Kyverno installed in your cluster

• Python 3 with google-cloud-secret-manager and google-cloud-aiplatform libraries (for the verification steps. Code available in the github repository.)

• kubectl access to your cluster

Why Hybrid Cloud Matters

If everything goes right, a hybrid cloud platform lets your on-premises and cloud workloads talk to each other as if they were part of the same network.

There are many practical reasons to run a hybrid cloud setup:

• Offloading analytics to BigQuery: You keep your analytics apps on-prem for data sovereignty, but pipe large datasets into BigQuery for world-class processing power — without buying extra servers.

• Creating a unified network with Cloud Interconnect: Using Cloud Interconnect or Cloud VPN, your on-premises datacenter becomes an extension of the Google Cloud Platform (GCP) Virtual Private Cloud (VPC). Your on-prem invoice apps can talk to cloud-based user services with low latency and no public internet exposure.

• Cost-effective scalability via Cloud Storage: You can use cloud storage as a backend for local apps, storing logs, backups, and historical data while paying only for what you use.

• Event-driven syncing with Pub/Sub: When something happens on-prem, a message through Cloud Pub/Sub lets cloud services react instantly — no manual polling required.

The Economics of Hybrid: GPUs Changed Everything

Before diving into the technical problem, it's worth understanding why hybrid clouds matter more than ever.

Your organization, like most enterprises, has made significant investments in on-premises datacenters. Servers are bought. Racks are filled. Network infrastructure is paid for. The marginal cost of running one more workload is essentially zero.

Then came the AI wave.

Suddenly every team needs Graphics Processing Units (GPUs). Not one or two — dozens of A100s for training, fleets of inference endpoints, vector databases that need to sit close to the models. GPUs are scarce. Lead times for on-prem GPU hardware stretch into months. Cloud providers have them available in minutes.

The architecture that actually makes economic sense looks like this:

• The on-prem datacenter handles the bulk of compute — web servers, business logic, databases, batch processing. This is commodity compute you've already paid for.

• The cloud handles what's scarce — GPU-accelerated inference, model training, AI/ML endpoints. You pay per request, scale on demand, and don't wait six months for hardware.

The cloud isn't a full migration destination — it's an extension for capabilities you can't easily build on-prem.

But those on-prem workloads need to authenticate to cloud services. Every API call from the datacenter to a Vertex AI endpoint, every request to a GPU-powered inference service, every write to Cloud Storage for model artifacts — all of it needs credentials. That's the problem this article solves.

Why Service Account Keys Fail at Scale

Here's a scenario that plays out in thousands of enterprises daily.

A development team needs their on-prem application to write to Google Cloud Storage. The "obvious" solution? Generate a GCP service account key, base64 encode it, store it in a Kubernetes Secret, and mount it in the pod:

apiVersion: v1
kind: Secret
metadata:
  name: gcp-credentials
type: Opaque
data:
  key.json: eyJ0eXBlIjoic2VydmljZV9hY2NvdW50IiwicHJvamVjdF9pZCI6…

This works. It also introduces serious problems:

• Never expires. That key is valid until someone remembers to rotate it (they won't) or it gets compromised (it will).

• Can be exfiltrated trivially. Anyone with read access to that namespace can run kubectl get secret -o yaml and walk away with permanent GCP access.

• Has no audit trail for the actual workload. GCP sees "service-account-xyz accessed this bucket" — not "pod frontend-abc-123 in namespace production."

• Scales terribly. 50 teams × 3 environments × 4 GCP projects = 600 keys to track, rotate, and hope haven't been committed to git.

Security teams know this. That's why many organizations have done the only sensible thing: they have disabled service account key generation entirely.

How the Accidental Air Gap Happens

When you disable key generation, you haven't solved the hybrid cloud platform problem — you've just made it someone else's problem. That someone is usually a platform team staring at a Jira ticket that says "cannot access GCP from on-prem, P1, blocking release."

The result? Your "hybrid cloud platform" isn't hybrid at all. It's two disconnected systems.

Teams resort to building intermediary services, API gateways that proxy requests, or finding creative ways to get keys anyway. None of this is a platform. It's duct tape.

How Workload Identity Federation Bridges the Gap

Every Kubernetes cluster already issues cryptographically signed identity tokens to every pod. And Google Cloud has a service specifically designed to trust those tokens.

This is Workload Identity Federation — and combined with OpenID Connect (OIDC), it's the missing piece that makes hybrid platforms actually work.

The service is quite well named because of the word Federation. it means GCP doesn't store your identity — it agrees to trust identities issued by another system, as long as they can be cryptographically verified. This all works with a very well orchestrated set of steps in the following order:

1. Pod presents its Kubernetes-issued JWT to GCP's STS endpoint.

2. STS verifies the signature against your cluster's public JWKS.

3. STS checks the JWT's claims against the Workload Identity Pool's rules (audience, issuer, CEL conditions).

4. STS returns a short-lived Google access token (typically 1 hour) that the pod uses for API calls.

It is also worth mentioning that Workload Identity Federation is not Kubernetes specific. It works with AWS IAM, Azure AD, GitHub Actions OIDC, and any OIDC-compliant identity provider.

How Kubernetes Identity Works

Every pod with a ServiceAccount gets a JSON Web Token (JWT) automatically mounted at /run/secrets/kubernetes.io/serviceaccount/token. This isn't just an opaque blob — it's a signed assertion of identity:

{
  "iss": "https://kubernetes.default.svc.cluster.local",
  "sub": "system:serviceaccount:production:backend-api",
  "aud": ["https://iam.googleapis.com/..."],
  "kubernetes.io": {
    "namespace": "production",
    "serviceaccount": {
      "name": "backend-api"
    }
  },
  "exp": 1735689600
}

In a JWT, claims are just the key-value pairs inside the token's payload — each one is a claim the issuer is making about the subject. Think of them as facts the token is asserting, signed cryptographically so the verifier can trust them.

The critical insight: this token is created by a set of JSON Web Key Set (JWKS) and is verifiable by anyone who has your cluster's public keys, exposed via the JSON Web Key Set (JWKS) endpoint:

kubectl get --raw /openid/v1/jwks

Google Cloud's Security Token Service (STS) can validate these tokens. No keys are exchanged. No secrets are stored. Just cryptographic proof of identity.

How to Prepare Google Cloud Platform resources

The Workload Identity Pool is a trust boundary — a declaration that says "I accept identities from external sources." The OIDC Provider configures how to validate those identities.

resource "google_iam_workload_identity_pool" "pool" {
  workload_identity_pool_id = "hybrid-platform-pool"
  project                   = "my-project"
}

resource "google_iam_workload_identity_pool_provider" "k8s_provider" {
  project                            = "my-project"
  workload_identity_pool_id          = google_iam_workload_identity_pool.pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "on-prem-cluster"

  attribute_mapping = {
    "google.subject"      = "assertion.sub"
    "attribute.namespace" = "assertion['kubernetes.io']['namespace']"
  }

  attribute_condition = "attribute.namespace in [\"production\", \"staging\"]"

  oidc {
    issuer_uri = "https://kubernetes.default.svc.cluster.local"
    jwks_json  = file("jwks.json")  # Your cluster's public keys
  }
}

Two things to note here:

1. attribute_mapping extracts claims from the Kubernetes JWT and makes them available as GCP attributes. By using `assertion['kubernetes.io']['namespace']`, the namespace is pulled out so you can use it for access control.

2. attribute_condition is where security policy lives. More on this in the next section.

How to Use CEL for Fine-Grained Access Control

The attribute_condition field uses Common Expression Language (CEL). This single line of policy can replace dozens of Identity and Access Management (IAM) bindings:

attribute.namespace in ["production", "staging"]

With this condition, a pod in the kube-system namespace cannot authenticate to GCP at all — the token exchange is rejected before IAM is even consulted.

You can get more sophisticated:

// Only production namespace, and only specific service accounts
attribute.namespace == "production" &&
  attribute.service_account in ["payment-processor", "order-service"]

// Allow staging, but only during business hours
attribute.namespace == "staging" &&
  request.time.getHours("America/New_York") >= 9 &&
  request.time.getHours("America/New_York") < 17

This is defense in depth. Even if someone creates a rogue ServiceAccount or has kubectl access, they cannot authenticate to GCP unless the CEL condition passes. The security boundary is enforced by Google's infrastructure, not by hoping developers follow policy.

How to Inject Credentials Automatically with Kyverno

Having a working identity federation is only half the battle. Your customers and developers shouldn't need to understand OIDC, STS, or credential configuration files. They should deploy their app and have it work.

Before we get to the automation, it's worth pausing on what a credential configuration file actually is — because the name is a little misleading.

A credential configuration file (sometimes called an "external account config" or "ADC config") is a small JSON document that tells Google's client libraries how to obtain a credential at runtime. It is not itself a credential. You'll see the actual file later in this article — it contains no secrets. Just metadata: the Workload Identity Pool audience, the STS token-exchange endpoint, the source token type, and the path on the pod's filesystem where the real (short-lived) Kubernetes ServiceAccount token lives.

Compare that to a traditional service account key:

Both files end up referenced by GOOGLE_APPLICATION_CREDENTIALS and look interchangeable from the application's point of view — but only one of them is dangerous to lose. The credential config file is safe to ship in a ConfigMap precisely because there's nothing to steal.

Having this file in the ConfigMap is half the solution. It actually needs to end up in the workload pods that need access to GCP services. This is where Kyverno comes in. A single ClusterPolicy automatically injects everything a pod needs:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: workload-identity-federation
spec:
  rules:
    - name: inject-gcp-credentials
      match:
        any:
          - resources:
              kinds:
                - Deployment
              selector:
                matchLabels:
                  workload-identity-federation: "enabled"
      mutate:
        patchStrategicMerge:
          spec:
            template:
              spec:
                volumes:
                  - name: workload-identity-credential-configuration
                    configMap:
                      name: workload-identity-federation-config
                containers:
                  - (name): "*"
                    volumeMounts:
                      - name: workload-identity-credential-configuration
                        mountPath: /etc/workload-identity
                        readOnly: true
                    env:
                      - name: GOOGLE_APPLICATION_CREDENTIALS
                        value: "/etc/workload-identity/credential-configuration.json"

The above cluster policy does three things:

1. Mounts the configmap inside the containers in the deployment at /etc/workload-identity.

2. Injects an environment variable called GOOGLE_APPLICATION_CREDENTIALS that points to the absolute path of the credential config file.

From a developer's perspective, this is their entire integration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    workload-identity-federation: "enabled" # That's it.
spec:
  # ... normal deployment spec

The credential configuration file (created by Terraform as a ConfigMap) tells Google's client libraries how to exchange tokens:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": {
    "file": "/run/secrets/kubernetes.io/serviceaccount/token"
  }
}

This JSON file is a credential configuration for Google's Workload Identity Federation. It instructs Google Cloud client libraries to obtain cloud access tokens by exchanging a Kubernetes ServiceAccount token (located at /run/secrets/kubernetes.io/serviceaccount/token) for a Google Cloud access token, using an external identity provider configured via a Workload Identity Pool. This allows workloads running outside of GCP, such as on-premises Kubernetes clusters, to authenticate to Google Cloud services without needing to manage long-lived service account keys.

Every Google Cloud SDK and client library understands this format. Python, Go, Java, and Node.js all just work.

How to Grant IAM Permissions to Federated Identities

The service account token that has been trusted by the STS service, also known as a federated identity, need permissions to access resources. You bind IAM roles to the identity pool attributes:

resource "google_project_iam_member" "secret_access" {
  for_each = toset(["production", "staging"])
  project  = "my-project"
  role     = "roles/secretmanager.secretAccessor"
  member   = "principalSet://iam.googleapis.com/projects/\({PROJECT_NUMBER}/locations/global/workloadIdentityPools/\){POOL_ID}/attribute.namespace/${each.value}"
}

This grants Secret Manager access to all pods authenticated from the production or staging namespaces. The principalSet syntax allows matching on attributes. You can also restrict to specific service accounts:

member = "principal://iam.googleapis.com/.../subject/system:serviceaccount:production:payment-processor"

How to Verify the Setup

You can verify the setup with a simple Python script that lists secrets from Secret Manager. This runs inside a pod on your on-premises cluster:

# list_secrets.py - running on-prem, accessing GCP Secret Manager
from google.cloud import secretmanager

def list_secrets(project_id: str):
    """
    List all secrets in a GCP project.

    No credentials are passed explicitly. The google-cloud-secret-manager
    library automatically:
    1. Reads GOOGLE_APPLICATION_CREDENTIALS env var (set by Kyverno)
    2. Loads the credential configuration JSON
    3. Reads the K8s ServiceAccount token from /run/secrets/...
    4. Exchanges it for a GCP access token via STS
    5. Uses that token to call the Secret Manager API
    """
    client = secretmanager.SecretManagerServiceClient()
    parent = f"projects/{project_id}"

    print(f"Secrets in {project_id}:")
    print("-" * 40)

    for secret in client.list_secrets(request={"parent": parent}):
        secret_name = secret.name.split("/")[-1]
        print(f"  - {secret_name}")

    print("-" * 40)
    print("Authentication: Workload Identity Federation")
    print("Credentials: None stored, token exchanged at runtime")

if __name__ == "__main__":
    list_secrets("my-project-id")

Run this inside your labeled pod:

$ kubectl exec -it my-app-xyz -- python list_secrets.py

Secrets in my-project-id:
----------------------------------------
  - database-password
  - api-key-stripe
  - oauth-client-secret
  - ml-model-api-key
----------------------------------------
Authentication: Workload Identity Federation
Credentials: None stored, token exchanged at runtime

No service account key. No secret mounted. Just a Kubernetes ServiceAccount token exchanged for GCP credentials at runtime.

This same pattern works for any GCP service — Secret Manager, Cloud Storage, BigQuery, Pub/Sub, and Vertex AI.

How to Connect On-Prem Apps to Cloud GPUs

Consider a typical flow: an on-prem order processing service needs to call a Vertex AI endpoint for fraud detection. The model runs on GPUs in Google Cloud (you can spin up A100s in minutes, not months). The application logic stays on-prem (you've already paid for that compute).

With the IAM bindings in place, any pod in the allowed namespaces can call Vertex AI:

# fraud_detector.py - running on-prem, calling cloud GPUs
from google.cloud import aiplatform

def check_fraud(transaction: dict) -> float:
    """
    Call a Vertex AI endpoint for fraud detection.

    The model runs on A100 GPUs in Google Cloud.
    This code runs on-prem in the datacenter.

    Authentication is automatic:
    1. Kyverno injected GOOGLE_APPLICATION_CREDENTIALS
    2. The aiplatform SDK reads the credential config
    3. K8s SA token is exchanged for GCP token via STS
    4. Request is authenticated to Vertex AI
    """
    endpoint = aiplatform.Endpoint(
        endpoint_name="projects/my-project/locations/us-central1/endpoints/fraud-model"
    )
    prediction = endpoint.predict(instances=[transaction])
    return prediction.predictions[0]["fraud_score"]


def generate_embeddings(texts: list[str]) -> list[list[float]]:
    """
    Generate text embeddings using a cloud-hosted model.

    Embedding models are GPU-intensive. Running them on-prem
    would require dedicated hardware. In the cloud, you pay per request.
    """
    from vertexai.language_models import TextEmbeddingModel

    model = TextEmbeddingModel.from_pretrained("text-embedding-004")
    embeddings = model.get_embeddings(texts)
    return [e.values for e in embeddings]

The developer doesn't think about authentication at all. They add the label to their deployment, and their on-prem pod can call:

• Vertex AI endpoints for ML inference on cloud GPUs

• Cloud Storage for model artifacts and training data

• BigQuery for feature stores and analytics

• Pub/Sub for event streaming between environments

• Secret Manager for API keys and configuration

This is the hybrid platform working as intended.

How to Scale GPU Access with CEL Conditions

CEL conditions become especially powerful when you want to restrict GPU access to specific namespaces. For example, to allow only ML-related namespaces to access Vertex AI:

attribute.namespace in ["ml-inference", "ml-training", "data-science"] &&
  attribute.service_account.startsWith("ml-")

You can also grant different access levels per namespace:

# ML inference namespace gets prediction access
resource "google_project_iam_member" "ml_inference" {
  project = "my-project"
  role    = "roles/aiplatform.user"
  member  = "principalSet://iam.googleapis.com/.../attribute.namespace/ml-inference"
}

# Data science namespace gets full Vertex AI access (for experimentation)
resource "google_project_iam_member" "data_science" {
  project = "my-project"
  role    = "roles/aiplatform.admin"
  member  = "principalSet://iam.googleapis.com/.../attribute.namespace/data-science"
}

The on-prem application teams don't need to know or care about GCP IAM. They deploy to the right namespace, add a label, and the platform handles the rest.

The Security Properties Compared

Here's a side-by-side comparison of the two authentication approaches:

The short-lived nature of tokens deserves emphasis. Even in a worst-case scenario where a token is somehow exfiltrated, it expires. Kubernetes ServiceAccount tokens have a configurable lifetime, and the GCP access tokens issued by STS are valid for one hour. A service account key, by contrast, remains valid until someone explicitly rotates it — often years.

The Complete Infrastructure as Code Layout

The entire solution is codified in Terraform, managing both GCP and Kubernetes resources:

workload-identity-federation/
├── providers.tf      # Google + Kubernetes providers
├── locals.tf         # Configuration (namespaces, project ID, etc.)
├── gcp.tf            # Identity pool, provider, IAM bindings
└── kubernetes.tf     # ConfigMap with credential configuration

A single terraform apply:

1. Creates the Workload Identity Pool in GCP

2. Configures the OIDC provider with your cluster's JWKS

3. Sets up IAM bindings for allowed namespaces

4. Creates ConfigMaps in each namespace with the credential configuration

Combined with the Kyverno policy, you get a fully automated pipeline:

New namespace added to allowed list
        │
        ▼
Terraform creates ConfigMap in that namespace
        │
        ▼
Developer deploys with label
        │
        ▼
Kyverno injects credentials automatically
        │
        ▼
Pod authenticates to GCP via OIDC
        │
        ▼
Application accesses GCP services

No tickets. No key requests. No secrets to manage.

How to Run a Proof of Concept with vCluster

To validate this works outside GKE, you can set up a demonstration using vCluster — a virtual Kubernetes cluster that runs inside another Kubernetes cluster. This proves the solution works for any cluster. You can setup vCluster in Docker using vind

# vcluster.yaml
experimental:
  docker:
    nodes:
      - name: worker-1
      - name: worker-2
deploy:
  cni:
    flannel:
      enabled: true
controlPlane:
  distro:
    k8s:
      version: "v1.35.0"

[root@localhost #] vcluster create hybrid --driver docker -f vcluster.yaml
[root@localhost #] kubectl get nodes
hybrid-control-plane   Ready    control-plane   14d   v1.34.0   192.168.107.2   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3
hybrid-worker          Ready    <none>          14d   v1.34.0   192.168.107.3   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3
hybrid-worker2         Ready    <none>          14d   v1.34.0   192.168.107.4   <none>        Debian GNU/Linux 12 (bookworm)   7.0.5-orbstack-00330-ge3df4e19b0a0-dirty   containerd://2.1.3

Inside the vCluster, deploy a simple test deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gcp-test
  labels:
    workload-identity-federation: "enabled"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gcp-test
  template:
    metadata:
      labels:
        app: gcp-test
    spec:
      containers:
        - name: test
          image: google/cloud-sdk:slim
          command: ["sleep", "infinity"]

Exec into the pod and verify:

$ kubectl exec -it gcp-test-xxx -- bash

# Inside the pod:
\( gcloud auth login --cred-file=\)GOOGLE_APPLICATION_CREDENTIALS
Authenticated with external account credentials for: [principal://iam.googleapis.com/...]

$ gcloud secrets list --project=my-project
NAME                 CREATED
database-password    2024-01-15T10:30:00Z
api-key              2024-01-14T09:15:00Z

No keys. No secrets mounted. Just identity federation working as designed.

Common Issues and How to Solve Them

How to Handle JWKS Retrieval for Air-Gapped Clusters

If your cluster's OIDC discovery endpoint isn't publicly reachable (most on-prem clusters aren't), you need to manually export the JWKS and upload it to GCP:

kubectl get --raw /openid/v1/jwks > jwks.json

This file must be updated if the cluster's signing keys rotate. Set up a periodic job that checks for key changes and updates the Terraform configuration.

How to Fix Issuer URL Mismatches

The iss claim in the Kubernetes token must exactly match the issuer URL configured in the OIDC provider. For clusters using internal DNS:

issuer_uri = "https://kubernetes.default.svc.cluster.local"

This URL doesn't need to be reachable from GCP — the JWKS file provides the validation keys. But it must match what's in the token exactly.

How to Debug Token Exchange Failures

When authentication fails, the error messages can be cryptic. Common causes and fixes:

Conclusion

After implementing this setup, on-premises workloads authenticate to Google Cloud exactly like GKE workloads do — without a single long-lived credential. The security team is happy (no keys to audit), developers are happy (just add a label), and the platform team is happy (no more credential management tickets).

Here's what you accomplished in this tutorial:

1. /Understood why service account keys fail at scale and the security risks they introduce

2. Created a Workload Identity Pool and OIDC provider in GCP to trust your cluster's token issuer

3. Used CEL conditions to enforce fine-grained, namespace-level access policies

4. Automated credential injection into pods using a Kyverno ClusterPolicy

5. Bound IAM roles to federated identity attributes — no long-lived keys anywhere

6. Verified the setup by calling GCP APIs (Secret Manager, Vertex AI) from an on-prem pod

7. Proved the solution works on any Kubernetes cluster using vCluster

The technologies used here aren't new. OIDC has been in Kubernetes since version 1.20. Workload Identity Federation has been in GCP for years. Kyverno and Terraform are mature tools. What this tutorial puts together is an end-to-end solution that developers can adopt with minimal effort.

If your organization has disabled service account keys (or should), this is the path forward. Your on-prem and cloud clusters can finally be what they were always meant to be: secure extensions of each other.

The complete implementation is available as a Terraform module with Kyverno policies: github.com/shkatara/hybrid-platform-gcp-workload-identity-federation

If this helps, you can follow me on https://www.linkedin.com/in/shubhamkatara/, https://www.youtube.com/@kubesimplify, https://www.linkedin.com/company/kubesimplify/ and

Then the real work starts. Infrastructure must be provisioned. CI/CD pipelines need to be set up. Secrets require management. Monitoring needs wiring. Databases need deployment. Logging needs configuration. Security policies need implementation. Networking rules need review.

Weeks disappear before users see anything useful. Many organizations treat this as normal. They call it engineering rigour. They assume this operational setup phase is simply part of software development.

It is not.

For teams already running production systems, rebuilding infrastructure foundations for every new project is organizational waste. It is repetitive operational labour disguised as an engineering discipline.

The uncomfortable question is not, “How can we do this setup faster?” The real question is: why are we still doing it ourselves at all?

This is where Platform as a Service changes the conversation. A good PaaS shifts the starting point from “rebuild the foundations” to “start shipping. Because new projects should begin closer to customer value, not closer to infrastructure assembly.

In this article, we'll look at why many production teams waste time rebuilding the same infrastructure for every new project, how PaaS helps remove that work, and why engineering teams should question if managing complex infrastructure still makes sense for most projects.

What We'll Cover:

• Most Teams Were Not Hired to Build Infrastructure

• AWS Primitives Are Not a Competitive Advantage

• Most Teams Should Not Be Managing Kubernetes

• PaaS Changes the Starting Point

• Repetition Creates Hidden Organizational Waste

• Standardization Is Usually Faster Than Flexibility

• Platform Teams Become Multipliers

• Easier Starts Create More Innovation

• When Specialized Control Actually Matters

• Starting From Zero Is a Process Failure

Most Teams Were Not Hired to Build Infrastructure

Software teams exist to solve business problems. Customers do not care whether Kubernetes manifests were structured elegantly. They do not admire carefully designed Terraform modules. They do not celebrate handcrafted networking policies.

Customers care about outcomes. They care about faster onboarding. Better recommendations. Smoother payments. Fewer bugs. Simpler workflows.

Yet many engineering organizations spend huge portions of time doing work customers never see.

Teams repeatedly create deployment pipelines. Configure environments. Manage certificates. Set up observability stacks. Tune infrastructure rules. Assemble cloud primitives.

Infrastructure matters. Reliability matters. Security matters.

The problem is duplication. If every project independently recreates the same operational systems, organizations keep rebuilding internal platforms over and over again without admitting it.

This behaviour has become so normalized that teams barely notice it anymore. But rebuilding the same foundation repeatedly is not operational maturity. It is inefficiency scaled across the organization.

AWS Primitives Are Not a Competitive Advantage

Many teams confuse cloud ownership with strategic advantage. Owning Kubernetes clusters does not create differentiation. Managing IAM rules does not create customer value. Writing infrastructure glue code does not strengthen market position.

These are implementation details. Yet many organizations spend extraordinary energy managing them as if they are core business assets.

Some teams effectively become part-time infrastructure companies without realizing it. Their engineers slowly accumulate operational responsibilities until maintaining systems consumes more effort than delivering products.

The outcome becomes predictable. Infrastructure expands. Operational complexity grows. Delivery speed declines. Nobody notices because the pain arrives gradually.

A team starts with one Kubernetes cluster. Then another environment appears. More deployment pipelines emerge. Additional tooling gets layered on top. Logging systems become fragmented. Monitoring evolves differently across products.

Eventually, teams spend increasing amounts of time maintaining systems they never intended to own.

Infrastructure ownership is often not a strategy. It is inertia.

Most Teams Should Not Be Managing Kubernetes

Kubernetes has become an engineering culture. It appears in architecture diagrams, conference talks, hiring requirements, and internal roadmaps. Its adoption often feels inevitable.

But normalization and necessity are not the same thing. Many organizations adopted Kubernetes because industry momentum made it seem like the default path. Not because they had workloads that required its complexity. But the result is predictable.

Small and medium teams end up managing orchestration systems designed for massive operational environments.

They maintain YAML configurations, networking layers, ingress systems, deployment strategies, and operational tooling stacks before delivering meaningful product value. This has become strangely accepted.

A ten-person engineering team maintaining infrastructure patterns designed for internet-scale organizations should raise serious questions. A small team pretending to be a platform team is an operational dysfunction.

Many companies adopt infrastructure complexity built for organizations operating at a vastly different scale. They inherit the burden without inheriting the benefits.

PaaS Changes the Starting Point

Traditional infrastructure approaches force teams to think from the bottom upward. Servers come first. Then operating systems. Then networking. Then deployment systems. Then monitoring. Eventually, applications arrive.

PaaS reverses this sequence. Developers begin with applications and business goals. The platform absorbs operational complexity.

Teams stop asking, “How do we provision resources?” They start asking, “What problem are we solving?” That sounds like a small shift. In practice, it changes everything.

A mature PaaS environment often provides deployment pipelines, integrated observability, databases, scaling behaviour, security controls, and operational standards before a team writes meaningful application logic.

Projects begin with product development rather than infrastructure construction. That dramatically changes time-to-value.

Repetition Creates Hidden Organizational Waste

Organizations often underestimate operational waste because repetitive work feels familiar. Setting up a deployment pipeline may consume only a few days. Configuring logging may feel routine. Creating security rules may seem manageable.

No individual task appears expensive. The cost appears when repetition scales.

If ten projects independently spend two weeks rebuilding nearly identical operational systems, months of engineering capacity disappear. Those engineers could have shipped customer capabilities. They could have reduced friction. They could have tested new ideas. Instead, they rebuilt plumbing.

Engineering teams understand leverage in nearly every other area. Nobody rewrites sorting algorithms for every application. Nobody recreates database engines from scratch. Nobody builds networking stacks repeatedly.

Reuse is accepted as basic engineering wisdom. Infrastructure should not receive special treatment. Build once. Reuse many times.

PaaS simply applies software engineering principles to operational systems.

Standardization Is Usually Faster Than Flexibility

Engineering teams often resist standardization because they fear losing control. Every project feels unique. Every system appears different. The desire for flexibility sounds reasonable.

But complete flexibility often creates operational chaos. Different teams deploy applications differently. Logging behaves inconsistently. Monitoring varies across systems. Security implementations drift.

Documentation fragments. Onboarding slows. Incident response becomes harder. Complexity quietly accumulates.

PaaS introduces constraints, and many engineers instinctively resist constraints. They should not. Useful constraints often increase speed.

Predictable deployment patterns reduce confusion. Shared monitoring standards simplify troubleshooting. Consistent environments reduce cognitive overhead.

Developers spend less energy understanding infrastructure differences and more time delivering product functionality.

Consistency compounds.

Platform Teams Become Multipliers

Many organizations interpret PaaS as buying a vendor product. That misses the bigger idea.

PaaS is fundamentally about creating reusable capabilities. Some organizations buy platforms. Others build internal platforms.

The principle remains the same.

A platform team creates systems once and allows everyone else to benefit. Instead of dozens of product teams independently solving operational problems, a dedicated group centralizes expertise and builds reusable solutions.

The effect becomes substantial. One deployment improvement accelerates every future release. One observability improvement strengthens every application. One security enhancement protects every team.

Platform teams create organizational leverage. Without this model, expertise stays fragmented. With it, expertise compounds.

Easier Starts Create More Innovation

Operational friction changes behaviour. When launching projects becomes expensive, organizations become cautious. Teams avoid experiments. Small ideas feel risky. Prototypes become difficult to justify.

Over time, innovation slows. Not because organizations lack ideas, but because starting became too expensive.

Teams running mature platforms understand this relationship. Reducing startup friction increases experimentation. Smaller projects become practical. Learning cycles become shorter.

New ideas appear more often because the cost of testing them falls dramatically. The easier it becomes to launch something, the more opportunities organizations create.

PaaS reduces startup friction. That reduction changes culture.

When Specialized Control Actually Matters

There are exceptions. Massive data platforms, highly specialized machine learning systems, and extremely customized environments may require lower-level infrastructure ownership.

Some workloads genuinely need deeper operational control. But these scenarios are exceptions, not defaults. Too many teams inherit infrastructure complexity designed for edge cases and treat it as standard practice.

Most production applications do not need custom orchestration layers. Most teams do not need to own Kubernetes. Most engineering groups do not need to spend weeks assembling infrastructure before shipping software.

The default assumption should be the opposite.

Starting From Zero Is a Process Failure

Many organizations normalize unnecessary operational drag. Long setup cycles become accepted. Infrastructure duplication becomes routine. Cloud complexity becomes expected.

Eventually, teams stop questioning it. They assume this is simply how engineering works. It is not.

If launching a new application requires weeks of foundational setup before customer value appears, that is not an engineering discipline.

The goal was never to become an infrastructure company. It was to ship software.

But traffic between your pods? Plaintext by default. Ingress traffic from the internet to your services? Only encrypted if you explicitly configure TLS. And certificates for internal services? You have to provision those yourself.

This is not a Kubernetes oversight. It's a deliberate design choice — Kubernetes provides the primitives and leaves the implementation to you. The problem is that certificate management is notoriously painful. Certificates expire. Provisioning them manually doesn't scale. Forgetting to rotate them causes outages.

cert-manager solves this. It runs as a controller inside your cluster, watches for Certificate resources, requests certificates from configured issuers, stores them in Kubernetes Secrets, and rotates them automatically before they expire. You declare what you want, cert-manager makes it happen and keeps it that way.

In this article you'll work through how cert-manager's core model works, automate public Ingress TLS using Let's Encrypt, set up an internal Certificate Authority for service-to-service encryption, and understand how certificate rotation works so outages caused by expired certificates become a thing of the past.

Prerequisites

• A kind cluster with the nginx Ingress controller installed

• Helm 3 installed

• A domain name with DNS you control — needed for the Let's Encrypt demo

• Basic understanding of TLS: you know what a certificate, a private key, and a CA are

All demo files are in the DevOps-Cloud-Projects GitHub repository.

Table of Contents

• What Is and Isn't Encrypted in Kubernetes

• How cert-manager Works

  • The Four Core Resources

  • Issuers and ClusterIssuers

  • The Certificate Lifecycle

  • ACME Challenges: HTTP-01 vs DNS-01

• Demo 1 — Install cert-manager and Issue a Let's Encrypt Certificate

• How to Get a Wildcard Certificate with DNS-01

• Demo 2 — Set Up an Internal CA for Service-to-Service TLS

• How Certificate Rotation Works

• Cleanup

• Conclusion

What Is and Isn't Encrypted in Kubernetes?

Before installing anything, it's worth being precise about what the cluster already protects and what it leaves open.

The two gaps that matter most in practice are pod-to-pod traffic and Ingress TLS. This article covers both Ingress TLS with Let's Encrypt and internal service-to-service encryption using a private CA.

How cert-manager Works

cert-manager is a Kubernetes operator. It extends the Kubernetes API with custom resources that represent certificate requests and their configuration. When you create a Certificate resource, cert-manager's controller picks it up, requests a certificate from the configured issuer, and stores the resulting certificate and private key in a Kubernetes Secret. When the certificate approaches its expiry, cert-manager renews it automatically.

This model means your application doesn't know or care about certificate management. It reads a Secret. cert-manager keeps that Secret fresh.

The Four Core Resources

cert-manager introduces four custom resources that you'll use regularly:

In practice you'll mostly deal with ClusterIssuer and Certificate. The ClusterIssuer defines where certificates come from. The Certificate defines what certificate you want and where to store it.

Issuers and ClusterIssuers

An Issuer can only issue certificates within its own namespace. A ClusterIssuer can issue certificates in any namespace. For shared infrastructure like Let's Encrypt, you almost always want a ClusterIssuer. For application-specific internal CAs, an Issuer scoped to that application's namespace is the safer choice.

cert-manager supports several issuer types. The three you'll encounter most often are:

ACME — for public certificates from Let's Encrypt or any ACME-compatible CA. Ownership of the domain is proven via an HTTP-01 or DNS-01 challenge.

CA — for internal certificates signed by a CA whose private key is stored in a Kubernetes Secret. Used for service-to-service TLS within the cluster.

Self-signed — generates self-signed certificates. Rarely useful on its own, but essential as the bootstrap step when creating an internal CA.

The Certificate Lifecycle

When you create a Certificate resource, cert-manager follows this sequence:

1. Creates a CertificateRequest with a CSR (Certificate Signing Request)

2. Passes the CSR to the configured issuer

3. For ACME issuers: creates a Challenge resource and fulfils it (more on this below)

4. Receives the signed certificate from the issuer

5. Stores the certificate and private key in the Kubernetes Secret named in spec.secretName

6. Monitors the certificate's expiry — by default, renews when 2/3 of the validity period has elapsed

Your application mounts the Secret. cert-manager updates it silently. Most applications that watch for file changes will pick up the new certificate without a restart.

ACME Challenges: HTTP-01 vs DNS-01

Let's Encrypt needs proof that you control the domain before it issues a certificate. ACME defines two challenge types for this.

HTTP-01 works by having cert-manager create a temporary HTTP endpoint at http://<your-domain>/.well-known/acme-challenge/<token>. Let's Encrypt sends a request to that URL. If the response matches the expected token, the challenge passes. This requires your cluster to be reachable from the internet on port 80.

DNS-01 works by having cert-manager create a temporary DNS TXT record at _acme-challenge.<your-domain>. Let's Encrypt checks for that record. This doesn't require inbound HTTP access, which makes it the right choice for private clusters, and it's the only way to get wildcard certificates (*.example.com).

The trade-off: HTTP-01 is simpler to set up but only works for single domains and requires internet-accessible infrastructure. DNS-01 requires API access to your DNS provider but works for internal clusters and wildcards.

Demo 1 — Install cert-manager and Issue a Certificate Using Pebble and Let's Encrypt

Pebble is Let's Encrypt's local ACME test server. It runs inside your cluster, issues certificates using the same ACME protocol as Let's Encrypt, and requires no public domain or internet access. Using Pebble lets you test the full cert-manager flow — challenge, issuance, renewal — on a plain kind cluster.

Once you understand the flow locally, switching to real Let's Encrypt is a one-line change: replace the ClusterIssuer server URL and point a DNS record at a publicly reachable cluster. The rest of the configuration is identical.

You'll install cert-manager, create a ClusterIssuer for Let's Encrypt, deploy a sample application with an Ingress, and watch a real certificate be issued and stored automatically.

Step 1: Install cert-manager

cert-manager is now distributed via OCI Helm charts from quay.io/jetstack. The --set crds.enabled=true flag installs the Custom Resource Definitions as part of the chart:

helm upgrade cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --install \
  --create-namespace \
  --namespace cert-manager \
  --set crds.enabled=true \
  --version v1.17.0 \
  --wait

You also need the nginx Ingress controller — cert-manager routes HTTP-01 challenges through it. The controller.service.type=ClusterIP override is for kind specifically: the default LoadBalancer Service never gets an EXTERNAL-IP on kind (there's no cloud LB), which makes --wait hang forever. On a real cluster, drop the override and keep LoadBalancer.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.type=ClusterIP \
  --wait

Confirm all four components are running:

kubectl get pods -n cert-manager
kubectl get pods -n ingress-nginx

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-76f84784c8-r4fx4              1/1     Running   0          6m45s
cert-manager-cainjector-66fbf49587-gv25n   1/1     Running   0          6m45s
cert-manager-webhook-577fddf86-l5wj4       1/1     Running   0          6m45s

NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6c7cd85885-h7zgx   1/1     Running   0          3m34s

kind-specific gotcha — remove the nginx admission webhook now.** On kind, the nginx admission webhook serves with a self-signed certificate that the Kubernetes API server cannot verify. The first time you try to create any Ingress resource you'll see failed calling webhook "validate.nginx.ingress.kubernetes.io": ... x509: certificate signed by unknown authority. Delete the webhook up front so the rest of the demo doesn't trip over it:

kubectl delete validatingwebhookconfiguration ingress-nginx-admission

Step 2: Install Pebble

Pebble is the local ACME test server, distributed by the JupyterHub project. It ships with a companion CoreDNS deployment (pebble-coredns) that Pebble uses to resolve names during ACME validation.

helm install pebble pebble \
  --repo https://jupyterhub.github.io/helm-chart/ \
  --namespace pebble \
  --create-namespace \
  --wait

Confirm both pods are running:

kubectl get pods -n pebble

NAME                              READY   STATUS    RESTARTS   AGE
pebble-8d8d49d64-lz8ck            1/1     Running   0          36s
pebble-coredns-7fb5c7cbf4-4jw9h   1/1     Running   0          36s

Step 3: Wire up DNS for the fake hostname

We're going to issue a cert for echo.pebble.local. That hostname is fake — it doesn't exist in any real DNS — so we have to teach two independent resolvers about it before issuance will work:

If you skip either layer, the Order will go to invalid state with a DNS lookup failure.

First grab the two IPs you'll need:

NGINX_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.spec.clusterIP}')
PEBBLE_DNS_IP=$(kubectl get svc pebble-coredns -n pebble \
  -o jsonpath='{.spec.clusterIP}')
echo "NGINX_IP=\(NGINX_IP  PEBBLE_DNS_IP=\)PEBBLE_DNS_IP"

Patch pebble-coredns to answer for *.pebble.local with the ingress controller's IP. The CoreDNS template plugin parses unreliably when the whole block is collapsed onto one line, so apply a real multi-line ConfigMap:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: pebble-coredns
  namespace: pebble
data:
  Corefile: |
    .:8053 {
      errors
      health
      ready
      template ANY ANY pebble.local {
        answer "{{ .Name }} 60 IN A ${NGINX_IP}"
      }
      forward . /etc/resolv.conf
      cache 2
      reload
    }
EOF

kubectl rollout restart deploy/pebble-coredns -n pebble
kubectl rollout status deploy/pebble-coredns -n pebble

Verify it answers correctly:

kubectl run dnstest --rm -it --restart=Never --image=busybox -- \
  nslookup echo.pebble.local ${PEBBLE_DNS_IP}

You should see Address: <NGINX_IP> in the response. If you get SERVFAIL, check kubectl logs -n pebble deploy/pebble-coredns — a parser error like not a TTL: "}" means the template block collapsed onto one line again.

Patch the cluster CoreDNS so cert-manager's self-check can resolve the same name. Add a stub zone that forwards pebble.local to pebble-coredns:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
    pebble.local:53 {
        forward . ${PEBBLE_DNS_IP}
    }
EOF

kubectl rollout restart deploy/coredns -n kube-system
kubectl rollout status deploy/coredns -n kube-system

Verify the cluster resolver now answers for echo.pebble.local (without specifying a server — it'll use the default kube-dns):

kubectl run dnstest --rm -it --restart=Never --image=busybox -- \
  nslookup echo.pebble.local

Both Server: 10.96.0.10 and Address: <NGINX_IP> should appear.

Step 4: Fetch the Pebble CA and create the ClusterIssuer

Pebble signs its certificates with a self-signed root that lives in the pebble ConfigMap under root-cert.pem. cert-manager needs to trust this CA to talk to Pebble's ACME directory, so we pass it as a base64-encoded caBundle in the ClusterIssuer:

kubectl get configmap pebble -n pebble \
  -o jsonpath='{.data.root-cert\.pem}' > pebble-ca.crt

head -1 pebble-ca.crt   # should print -----BEGIN CERTIFICATE-----

CA_BUNDLE=$(base64 -i pebble-ca.crt | tr -d '\n')
echo "CA_BUNDLE length: ${#CA_BUNDLE}"   # ~1600 chars, one continuous line

Create the ClusterIssuer using the heredoc — the ${CA_BUNDLE} shell variable gets substituted into the YAML before kubectl reads it:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: pebble
spec:
  acme:
    server: https://pebble.pebble.svc.cluster.local/dir
    email: test@example.com
    privateKeySecretRef:
      name: pebble-account-key
    caBundle: ${CA_BUNDLE}
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
EOF

Check the issuer is ready:

kubectl get clusterissuer pebble

NAME     READY   AGE
pebble   True    5s

If READY stays False, the two most common causes are a malformed caBundle (verify it's a single unbroken base64 line with no newlines) or Pebble being unreachable from the cert-manager namespace. To check reachability:

kubectl run test-curl --rm -it --restart=Never \
  --image=curlimages/curl:latest \
  --namespace cert-manager -- \
  curl -k https://pebble.pebble.svc.cluster.local/dir

If that returns JSON, Pebble is reachable.

Step 5: Deploy a sample application

# echo-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: ealen/echo-server:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: default
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80

kubectl apply -f echo-app.yaml

Verify the resources came up:

kubectl get deploy,pod,svc -n default

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/echo   1/1     1            1           32s

NAME                        READY   STATUS    RESTARTS   AGE
pod/echo-5665fbcfdd-mbgxj   1/1     Running   0          36s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/echo         ClusterIP   10.96.103.114   <none>        80/TCP    40s
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   32m

Step 6: Create an Ingress with TLS

The cert-manager.io/cluster-issuer: pebble annotation tells cert-manager to automatically create a Certificate resource for this Ingress, using the issuer we just created. The hostname echo.pebble.local doesn't need to resolve externally — we taught both DNS resolvers about it in Step 3.

# echo-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: pebble
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - echo.pebble.local
      secretName: echo-tls     # cert-manager will create this Secret
  rules:
    - host: echo.pebble.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: echo
                port:
                  number: 80

kubectl apply -f echo-ingress.yaml

Step 7: Watch the certificate being issued

# Watch the Certificate resource (Ctrl-C once Ready=True)
kubectl get certificate echo-tls -n default -w

NAME       READY   SECRET     AGE
echo-tls   False   echo-tls   5s
echo-tls   True    echo-tls   28s

When READY becomes True, the certificate has been issued and stored in the echo-tls Secret. The full chain — CertificateRequest → Order → Challenge → solver pod → Secret — happens in well under a minute on a healthy cluster:

kubectl get certificate,certificaterequest,order,challenge -n default

NAME                                   READY   SECRET     AGE
certificate.cert-manager.io/echo-tls   True    echo-tls   81s

NAME                                            APPROVED   DENIED   READY   ISSUER   AGE
certificaterequest.cert-manager.io/echo-tls-1   True                True    pebble   81s

NAME                                               STATE   AGE
order.acme.cert-manager.io/echo-tls-1-1824732543   valid   81s

(Challenges are deleted automatically once an Order completes, so kubectl get challenge -n default typically shows nothing at this point — that's success, not failure.)

If READY stays False for more than a minute, see the troubleshooting tips at the end of this section.

Inspect the issued certificate to confirm Pebble signed it:

kubectl get secret echo-tls -n default -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -issuer -subject -dates

issuer=CN=Pebble Intermediate CA 05478c
subject=
notBefore=May 17 19:09:22 2026 GMT
notAfter=Aug 15 19:09:21 2026 GMT

Issuer is Pebble's intermediate CA — proof the full ACME flow worked end-to-end. The cert is valid for 90 days, and cert-manager will renew it automatically at day 60.

Hit the ingress over HTTPS from inside the cluster to confirm everything is wired together:

kubectl run curltest --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sk https://echo.pebble.local/

The echo server should return a JSON blob — note the "x-forwarded-proto":"https" field, which proves the request came through nginx over TLS.

Troubleshooting if the cert never goes Ready:

• kubectl describe order -n default — look for "DNS problem" or "Connection refused" in the events.

• kubectl logs -n pebble deploy/pebble --tail=50 — Pebble logs the exact URL it tried to fetch during validation and any errors.

• If the Order is stuck pending with no events: cert-manager hasn't reconciled yet. Wait 30s.

• If the Order is invalid: one of the two DNS layers (Step 3) is misconfigured. Re-run both nslookup checks.

• If the Ingress apply itself failed with an x509 webhook error: you skipped the kubectl delete validatingwebhookconfiguration ingress-nginx-admission step in Step 1.

Step 8: Switch to Let's Encrypt staging (real public domain)

Pebble proved the flow works locally. Now move to a publicly-reachable domain pointed at a publicly-reachable cluster. The DNS gymnastics from Step 3 go away — the domain is real, so both resolvers find it without intervention.

Use Let's Encrypt staging first. It speaks the same ACME protocol as production but with generous rate limits, so failed attempts during testing won't lock you out:

# clusterissuer-staging.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

kubectl apply -f clusterissuer-staging.yaml

# Point the Ingress at staging and the real hostname, then force re-issuance
kubectl annotate ingress echo \
  cert-manager.io/cluster-issuer=letsencrypt-staging --overwrite -n default
kubectl delete secret echo-tls -n default

The new cert's issuer will look something like (STAGING) Let's Encrypt.

Step 9: Switch to Let's Encrypt production

Once staging works, repeat with the production ClusterIssuer. The only difference is the server URL:

# clusterissuer-prod.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

kubectl apply -f clusterissuer-prod.yaml
kubectl annotate ingress echo \
  cert-manager.io/cluster-issuer=letsencrypt-prod --overwrite -n default
kubectl delete secret echo-tls -n default

cert-manager detects the missing Secret and immediately requests a browser-trusted certificate from production Let's Encrypt.

cert-manager detects the missing Secret and immediately triggers a new certificate request using the production issuer.

How to Get a Wildcard Certificate with DNS-01

HTTP-01 challenges work well for single domains with public ingress. But there are two situations where you need DNS-01 instead: when your cluster is not publicly accessible (internal clusters, air-gapped environments, staging namespaces behind a VPN), and when you want a wildcard certificate that covers all subdomains of your domain.

DNS-01 requires cert-manager to be able to create and delete TXT records in your DNS provider. cert-manager has built-in support for Route53, Cloud DNS, Cloudflare, Azure DNS, and many others.

Here is a ClusterIssuer for DNS-01 using AWS Route53:

# clusterissuer-dns01.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-dns01-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            # Use IRSA (IAM Roles for Service Accounts) in production
            # rather than static credentials
            hostedZoneID: YOUR_HOSTED_ZONE_ID

A wildcard Certificate using that issuer:

# wildcard-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: default
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer
  commonName: "*.example.com"
  dnsNames:
    - "*.example.com"
    - "example.com"        # Also cover the apex domain
  duration: 2160h           # 90 days
  renewBefore: 720h         # Renew 30 days before expiry

The resulting Secret wildcard-example-com-tls can be referenced by any Ingress in the default namespace. All subdomains — api.example.com, dashboard.example.com, staging.example.com — are covered by a single certificate that rotates automatically.

For Cloudflare instead of Route53, the solver section looks like this:

    solvers:
      - dns01:
          cloudflare:
            email: your-email@example.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

Demo 2 — Set Up an Internal CA for Service-to-Service TLS

Let's Encrypt certificates are great for public-facing services. But for internal services — a gRPC microservice calling another, a web application talking to its database — you don't need public trust. You need a CA that the cluster trusts, and you need it to issue certificates for service names that don't exist as public DNS records.

cert-manager's CA issuer handles this. You create a root CA, tell cert-manager about it, and then issue certificates for internal services using that CA. Every service that trusts the root CA trusts every certificate it issues.

Step 1: Create a self-signed ClusterIssuer

A self-signed issuer generates certificates that are signed by the certificate itself — it is its own CA. You use this as a bootstrap step to create the root CA certificate:

# selfsigned-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

kubectl apply -f selfsigned-issuer.yaml

Step 2: Create the root CA certificate

Use the self-signed issuer to create a CA certificate. The isCA: true field tells cert-manager this certificate can sign other certificates:

# internal-ca.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager    # Store in cert-manager namespace
spec:
  isCA: true
  commonName: internal-ca
  secretName: internal-ca-secret
  duration: 87600h           # 10 years — this is a root CA
  renewBefore: 720h
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

kubectl apply -f internal-ca.yaml
kubectl get certificate internal-ca -n cert-manager

NAME          READY   SECRET               AGE
internal-ca   True    internal-ca-secret   8s

Step 3: Create a CA ClusterIssuer backed by the root CA

Now create a ClusterIssuer that uses the root CA Secret you just created. This is the issuer that will sign certificates for your internal services:

# internal-ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-secret   # References the Secret in cert-manager namespace

kubectl apply -f internal-ca-issuer.yaml
kubectl get clusterissuer internal-ca

NAME          READY   AGE
internal-ca   True    5s

Step 4: Issue a certificate for an internal service

Now issue a certificate for an internal gRPC service. The dnsNames use Kubernetes internal DNS names — <service>.<namespace>.svc.cluster.local:

# payments-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: payments-tls
  namespace: production
spec:
  secretName: payments-tls-secret
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  commonName: payments.production.svc.cluster.local
  dnsNames:
    - payments.production.svc.cluster.local
    - payments.production.svc
    - payments
  duration: 2160h     # 90 days
  renewBefore: 360h   # Renew 15 days before expiry

kubectl create namespace production
kubectl apply -f payments-cert.yaml
kubectl get certificate payments-tls -n production

NAME           READY   SECRET                AGE
payments-tls   True    payments-tls-secret   6s

The Secret payments-tls-secret now contains tls.crt, tls.key, and ca.crt. Mount this into your application pod:

# In your Deployment spec
volumes:
  - name: tls
    secret:
      secretName: payments-tls-secret
containers:
  - name: payments
    volumeMounts:
      - name: tls
        mountPath: /etc/tls
        readOnly: true

Your application reads /etc/tls/tls.crt and /etc/tls/tls.key to configure TLS. Other services that need to trust it read /etc/tls/ca.crt.

Step 5: Distribute the CA bundle with trust-manager

The problem with a custom CA is that every service needs to know about it. cert-manager's companion tool, trust-manager, handles this by distributing the CA bundle as a ConfigMap to every namespace:

helm upgrade trust-manager oci://quay.io/jetstack/charts/trust-manager \
  --install \
  --namespace cert-manager \
  --wait

Create a Bundle resource that takes the CA certificate from the internal-ca-secret and distributes it cluster-wide:

# ca-bundle.yaml
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: internal-ca-bundle
spec:
  sources:
    - secret:
        name: internal-ca-secret
        key: ca.crt
  target:
    configMap:
      key: ca-bundle.crt
    namespaceSelector:
      matchLabels:
        # Distribute to all namespaces with this label
        kubernetes.io/metadata.name: production

kubectl apply -f ca-bundle.yaml

After a few seconds, every matching namespace has a ConfigMap named internal-ca-bundle containing the CA certificate. Applications mount this ConfigMap to trust internally-issued certificates without any per-service configuration.

Step 6: Verify the certificate chain

# Extract the CA cert and service cert
kubectl get secret payments-tls-secret -n production \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

kubectl get secret payments-tls-secret -n production \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > payments.crt

# Verify the cert was signed by the CA
openssl verify -CAfile ca.crt payments.crt

payments.crt: OK

How Certificate Rotation Works

Certificate rotation is the part of certificate management that breaks production clusters most often. cert-manager handles it automatically, but understanding the mechanism helps you tune it and debug it when things go wrong.

cert-manager watches every Certificate resource it manages and checks the expiry of the underlying certificate in the Secret. When the remaining validity drops below the renewBefore threshold, cert-manager triggers a renewal. The default renewBefore is 1/3 of the certificate's total validity period — so a 90-day certificate starts renewing at day 60.

The renewal creates a new CertificateRequest, goes through the full issuance flow, and updates the Secret in place. The new certificate replaces the old one atomically. Applications that use file mounts and watch for changes (most modern web servers and gRPC frameworks do) will pick up the new certificate without restarting.

# See the current rotation status
kubectl describe certificate echo-tls -n default

Look for these fields in the output:

Status:
  Not After:   2024-06-18T10:00:00Z
  Not Before:  2024-03-20T10:00:00Z
  Renewal Time: 2024-05-18T10:00:00Z   # When cert-manager will start renewing
  Conditions:
    Type:    Ready
    Status:  True
    Message: Certificate is up to date and has not expired

If a renewal fails — for example, because the HTTP-01 challenge can't be completed — cert-manager retries with exponential backoff. The existing certificate continues to serve until it actually expires, giving you a window to debug the issue.

To see renewal events in real time:

kubectl get events -n default --field-selector reason=Issued
kubectl get events -n default --field-selector reason=Failed

Setting renewBefore correctly: For public-facing services, 30 days before a 90-day certificate is a sensible buffer. For internal short-lived certificates (24-hour validity), set renewBefore to 8 hours so rotation happens well before expiry even if the first attempt fails. Never set renewBefore to more than half the certificate's validity — cert-manager will immediately try to renew a certificate it just issued.

Cleanup

# Remove demo resources
kubectl delete ingress echo -n default
kubectl delete service echo -n default
kubectl delete deployment echo -n default
kubectl delete secret echo-tls -n default
kubectl delete certificate payments-tls -n production
kubectl delete namespace production

# Uninstall cert-manager and trust-manager
helm uninstall trust-manager -n cert-manager
helm uninstall cert-manager -n cert-manager
kubectl delete namespace cert-manager

# Remove ClusterIssuers
kubectl delete clusterissuer letsencrypt-staging letsencrypt-prod \
  internal-ca selfsigned 2>/dev/null

Conclusion

Kubernetes leaves TLS configuration entirely to you. In this article you worked through both the public and internal sides of that responsibility.

On the public side, you installed cert-manager using the current OCI Helm chart, created a ClusterIssuer backed by Let's Encrypt, and watched cert-manager go through the full ACME HTTP-01 challenge flow — from creating a temporary solver pod to storing a valid certificate in a Kubernetes Secret. You saw how switching from staging to production is a one-line annotation change, and how cert-manager renews certificates automatically before they expire.

On the internal side, you bootstrapped a private CA using cert-manager's self-signed issuer, created a ClusterIssuer backed by that CA, and issued certificates for internal service names that only exist inside the cluster. You used trust-manager to distribute the CA bundle cluster-wide so services can trust each other's certificates without per-service configuration. And you saw how to verify the certificate chain with openssl so you can confirm it's working before deploying to production.

Understanding certificate rotation is what separates teams that manage TLS confidently from teams that get woken up at 3am by an expired certificate. cert-manager automates the renewal, but the renewBefore field is your safety margin — set it correctly and know how to read the renewal status.

All YAML manifests and Helm values from this article are available in the DevOps-Cloud-Projects GitHub repository.

I did. I spun up an EC2 instance, followed along for an hour, and then forgot to shut it down. A week later I had a $34 bill for a machine running nothing.

That was the last time I practiced on someone else's infrastructure.

Everything in this guide runs on your laptop. No cloud account, no credit card, no bill at the end of the month. By the end, you'll be able to spin up a multi-server environment from scratch, configure it automatically with Ansible, serve a site you wrote yourself, and diagnose what breaks when you intentionally destroy it.

That last part is where the actual learning happens.

Prerequisites

Before you start, make sure you have:

• A laptop with at least 8GB of RAM (16GB is better)

• At least 20GB of free disk space

• Windows, macOS, or Linux operating system

• Administrator access to your computer

• Virtualization enabled in your BIOS/UEFI settings

• A stable internet connection for the initial downloads

Knowledge and comfort level:

• You should be comfortable using a terminal (running commands, changing directories, and editing small text files with whatever editor you like).

• Basic familiarity with concepts like “a server,” “SSH,” and “a port” helps, but you don't need prior experience with Docker, Kubernetes, Vagrant, or Ansible. This guide introduces them as you go.

If you can follow step-by-step instructions and read error output without panicking, you're ready.

Table of Contents

1. What is DevOps?

2. Why Build a Local Lab?

3. How to Set Up Docker

4. How to Set Up Kubernetes

5. How to Install kubectl

6. How to Set Up Vagrant

7. How to Install Ansible

8. How to Build Your First DevOps Project

9. How to Break Your Lab on Purpose

10. What You Can Now Do

What is DevOps?

DevOps is the practice of breaking down the wall between software development and IT operations teams.

Traditionally, developers write code and hand it off to operations teams to deploy and maintain. That handoff causes delays, misunderstandings, and outages. DevOps is what happens when both teams work together from the start.

The tools you'll install in this guide each solve a specific part of that process:

• Docker packages your application and everything it needs into a portable container that runs the same way on any machine.

• Kubernetes manages multiple containers at scale, handling restarts, networking, and load balancing automatically.

• Vagrant creates and manages virtual machine environments so your whole team always works on identical setups.

• Ansible automates repetitive configuration tasks across many servers without writing a script for each one.

Why Build a Local Lab?

A local lab gives you a safe place to break things, fix them, and learn from that process without any cost or risk.

Here's what you get with a local setup:

• Zero cost. No cloud bills, no surprise charges, and no credit card required.

• Works offline. Practice anywhere, even without internet after the initial setup.

• Full control. You manage every layer from the OS up to the application.

• Safe experimentation. Break things freely. Nothing here affects production.

• Fast feedback. No waiting for cloud resources to spin up. Everything runs on your machine.

The tradeoff is resource limits. Your laptop's CPU and RAM are the ceiling. You can't simulate large-scale deployments, and some cloud-native services like AWS Lambda or S3 have no direct local equivalent. But for learning core DevOps workflows, none of that matters.

How to Set Up Docker

Docker is the foundation of this lab. Every other tool in this guide either runs inside Docker containers or works alongside them.

How to Install Docker on Windows

First, enable virtualization in your BIOS:

1. Restart your computer and enter BIOS/UEFI setup. The key is usually F2, F10, Del, or Esc during boot.

2. Find the virtualization setting. It's usually listed as Intel VT-x, AMD-V, SVM, or Virtualization Technology.

3. Enable it, save your changes, and exit.

Then install Docker Desktop:

1. Download Docker Desktop from Docker's official website.

2. Run the installer and follow the prompts.

3. Enable WSL 2 (Windows Subsystem for Linux) when asked.

4. Restart your computer.

5. Open Docker Desktop from the Start menu and wait for the whale icon in the taskbar to stop animating.

Troubleshooting: If Docker fails to start, run this in PowerShell as Administrator to verify virtualization is active:

systeminfo | findstr "Hyper-V Requirements"

All items should show "Yes". If they don't, revisit your BIOS settings.

How to Install Docker on Mac

1. Download Docker Desktop for Mac from Docker's website.

2. Open the downloaded .dmg file and drag Docker to your Applications folder.

3. Open Docker from Applications.

4. Enter your password when prompted.

5. Wait for the whale icon in the menu bar to stop animating.

How to Install Docker on Linux

Run these commands in order:

# Update your package lists
sudo apt-get update

# Install prerequisites
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Add the Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Update and install Docker
sudo apt-get update
sudo apt-get install docker-ce

# Start and enable Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add your user to the docker group
sudo usermod -aG docker $USER

Log out and back in for the group change to take effect.

How to Test Docker

Run this command:

docker run hello-world

If you see "Hello from Docker!" then Docker is working correctly.

Docker is set up. Next, you'll install Kubernetes to manage containers at scale.

How to Set Up Kubernetes

Kubernetes manages containers at scale. For a local lab, you have four options. Here's how to choose:

If you're just starting out, use Minikube. It has the simplest setup and a visual dashboard that helps you understand what's happening inside the cluster.

If your laptop has 8GB RAM or less, use k3s. It runs lean and behaves closer to a real cluster than Minikube does.

Use kubeadm only if you want to understand how Kubernetes nodes join a cluster — it requires more manual steps and isn't beginner-friendly.

How to Install Minikube (Recommended for Beginners)

Minikube creates a single-node Kubernetes cluster on your laptop.

On Windows:

1. Download the Minikube installer from Minikube's GitHub releases page.

2. Run the .exe installer.

3. Open Command Prompt as Administrator and start Minikube:

minikube start --driver=docker

On Mac:

brew install minikube
minikube start --driver=docker

On Linux:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube-linux-amd64
sudo mv minikube-linux-amd64 /usr/local/bin/minikube
minikube start --driver=docker

Test your cluster:

minikube status
minikube dashboard

How to Install k3s (Recommended for Low-RAM Machines)

k3s is a lightweight version of Kubernetes that installs in under a minute. It runs lean and behaves like a real cluster — not a simplified demo version.

On Linux (and Mac via Multipass):

curl -sfL https://get.k3s.io | sh -

That single command installs k3s and runs it automatically in the background. Check that it is running:

sudo k3s kubectl get nodes

You should see one node with status Ready.

On Mac directly — k3s doesn't run natively on macOS. Use Multipass to spin up a lightweight Ubuntu VM first, then run the install command inside it.

On Windows — use WSL2 (Ubuntu), then run the install command inside your WSL2 terminal.

How to Install Kind (Kubernetes IN Docker)

Kind runs a full Kubernetes cluster inside Docker containers. It starts faster than Minikube and is useful if you want to run multiple clusters simultaneously.

# Mac or Linux
brew install kind

# Windows
choco install kind

Create a cluster:

kind create cluster --name my-local-lab

How to Install kubeadm (For Understanding Cluster Bootstrap)

kubeadm is the tool Kubernetes uses to initialize and join nodes in a real cluster. Use this when you want to understand what happens under the hood — not as your daily driver.

It requires at least two machines (or VMs). The setup is more involved than the options above. Follow the official kubeadm installation guide for your OS, then initialize your cluster:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

After init, join worker nodes using the command kubeadm prints at the end of the output.

How to Install kubectl

kubectl is the command-line tool you use to interact with any Kubernetes cluster.

On Windows:

Download kubectl.exe from Kubernetes' website and place it in a directory that is in your PATH. Or install with Chocolatey:

choco install kubernetes-cli

On Mac:

brew install kubectl

On Linux:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl

Test it:

kubectl get pods --all-namespaces

On a fresh cluster, you'll see system pods running in the kube-system namespace — things like coredns and storage-provisioner. That's the expected output. It means your cluster is up and kubectl can talk to it.

Kubernetes is running. Next is Vagrant. But before that, there's one important distinction worth making.

Docker vs Vagrant — they aren't the same thing

Docker creates containers: lightweight processes that share your operating system's kernel. Vagrant creates full virtual machines: isolated computers with their own OS running inside your laptop.

Containers are fast and small. VMs are heavier but behave exactly like real servers. You'll use both in this lab for different reasons.

How to Set Up Vagrant

Vagrant lets you create and manage reproducible virtual machine environments. It is ideal for simulating multi-server setups on a single laptop.

How to Install Vagrant on Windows

1. Download and install VirtualBox with default options.

2. Download and install Vagrant.

3. Restart your computer if prompted.

Note: VirtualBox and Hyper-V can't run at the same time on Windows. Check if Hyper-V is active:

systeminfo | findstr "Hyper-V"

If it's enabled, you have two options: switch to the Hyper-V Vagrant provider, or disable Hyper-V with:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All

Restart after disabling.

How to Install Vagrant on Mac and Linux

On Mac:

1. Download and install VirtualBox.

2. After installation, open System Preferences > Security & Privacy > General. You will see a message saying system software from Oracle was blocked. Click Allow and restart your Mac. Without this step, VirtualBox will not run.

3. Download and install Vagrant.

Note for Apple Silicon (M1/M2/M3) Macs: VirtualBox support on Apple Silicon is still limited. If you're on an M-series Mac, use UTM as your VM provider instead, or use Multipass which works natively on Apple Silicon.

On Linux:

1. Download and install VirtualBox.

2. Download and install Vagrant.

Verify both are installed:

vboxmanage --version
vagrant --version

How to Create Your First Vagrant Environment

Create a new directory for your project. Inside it, create a file named Vagrantfile with this content:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  # Create a private network between VMs
  config.vm.network "private_network", type: "dhcp"

  # Forward port 8080 on your laptop to port 80 on the VM
  config.vm.network "forwarded_port", guest: 80, host: 8080

  # Install Nginx when the VM starts
  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y nginx
    echo "Hello from Vagrant!" > /var/www/html/index.html
  SHELL
end

Start the VM:

vagrant up

Visit http://localhost:8080 in your browser. You should see "Hello from Vagrant!"

Troubleshooting SSH on Windows

If vagrant ssh fails, try:

vagrant ssh -- -v

Or connect manually:

ssh -i .vagrant/machines/default/virtualbox/private_key vagrant@127.0.0.1 -p 2222

How to Create a Local Vagrant Box Without Internet

Note: Most readers can skip this. Only do this if you want to work fully offline after the initial setup.

1. Download Ubuntu 20.04 LTS and save the .iso file locally.

2. Open VirtualBox and create a new VM: Name it ubuntu-devops, Type: Linux, Version: Ubuntu (64-bit).

3. Assign 2048MB RAM and a 20GB VDI disk.

4. Attach the .iso under Storage > Optical Drive.

5. Start the VM and complete the Ubuntu installation.

6. Once installed, shut down the VM and run:

VBoxManage list vms
vagrant package --base "ubuntu-devops" --output ubuntu2004.box
vagrant box add ubuntu2004 ubuntu2004.box

You now have a reusable local box that works without internet.

You can spin up virtual machines. Next is Ansible, which automates what goes inside them.

How to Install Ansible

Ansible automates configuration and software installation across multiple servers. Instead of SSH-ing into ten machines and running the same commands manually, you write a playbook once and Ansible handles the rest.

How to Install Ansible on Windows

Ansible doesn't run natively on Windows. You need to use it through WSL (Windows Subsystem for Linux).

1. Open PowerShell as Administrator and enable WSL:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

1. Restart your computer.

2. Install Ubuntu from the Microsoft Store.

3. Open Ubuntu and install Ansible:

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

How to Install Ansible on Mac

brew install ansible

How to Install Ansible on Linux

# Ubuntu/Debian
sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

# Red Hat/CentOS
sudo yum install ansible

How to Test Ansible

Create a file called hosts in your current directory:

[local]
localhost ansible_connection=local

Create a file called playbook.yml in the same directory:

---
- name: Test playbook
  hosts: local
  tasks:
    - name: Print a message
      debug:
        msg: "Ansible is working!"

Run the playbook, passing the local hosts file with -i:

ansible-playbook -i hosts playbook.yml

You should see the message "Ansible is working!" in the output.

Alright, all your tools are installed. Now you'll use them together to build something real.

How to Build Your First DevOps Project

You can find the entire code for this lab in this repo: https://github.com/Osomudeya/homelab-demo-article

Now you'll put these tools together in one project. Each tool will perform its actual job, and nothing is forced.

Before you start, create a fresh directory for this project. Don't run it inside the directory you used to test Vagrant earlier, as the Vagrantfile here is different and will conflict.

You'll be building a two-VM environment: one machine serves a web page you write yourself inside a Docker container, and the other runs a MariaDB database. Vagrant creates the machines and Ansible configures them. The page you see at the end is yours.

Step 1: Create the Project Directory

mkdir devops-lab-project && cd devops-lab-project

Step 2: Write Your Site Content

Create a file called index.html in the project directory. Write whatever you want on this page — it's what you'll see in your browser at the end:

<!DOCTYPE html>
<html>
  <head><title>My DevOps Lab</title></head>
  <body>
    <h1>My DevOps Lab</h1>
    <p>Provisioned by Vagrant. Configured by Ansible. Served by Docker.</p>
    <p>Built on a laptop. No cloud account needed.</p>
  </body>
</html>

Change the text to whatever you like. This is your page.

Step 3: Write the Vagrantfile

Create a file called Vagrantfile in the same directory:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  config.vm.define "web" do |web|
    web.vm.network "private_network", ip: "192.168.33.10"
    web.vm.network "forwarded_port", guest: 80, host: 8080
  end

  config.vm.define "db" do |db|
    db.vm.network "private_network", ip: "192.168.33.11"
  end
end

Step 4: Start the Virtual Machines

vagrant up

The first run downloads the ubuntu/focal64 box, which is around 500MB.

Expect this to take 10–30 minutes depending on your connection. Subsequent runs will be much faster since the box is cached locally.

Step 5: Create the Ansible Inventory

Create a file called inventory in the same directory:

[webservers]
192.168.33.10 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

[dbservers]
192.168.33.11 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/db/virtualbox/private_key

Ansible uses the Vagrant-generated private keys so it can SSH in as the vagrant user. Host key checking for this lab is turned off in ansible.cfg (next step), not in the inventory.

Step 6: Create the Ansible Config File

Before running the playbook, create a file called ansible.cfg in the same directory:

[defaults]
inventory = inventory
host_key_checking = False

The inventory line tells Ansible to use the inventory file in this folder by default. host_key_checking = False tells Ansible not to verify SSH host keys when connecting to your Vagrant VMs. Without it, Ansible will fail with a Host key verification failed error on first connection because the VM's key is not yet in your known_hosts file.

These settings are for a local lab only. Do not use host_key_checking = False for production systems.

Step 7: Create the Ansible Playbook

Create a file called playbook.yml:

---
- name: Configure web server
  hosts: webservers
  become: yes
  tasks:

    - name: Install Docker
      apt:
        name: docker.io
        state: present
        update_cache: yes

    - name: Start Docker service
      service:
        name: docker
        state: started
        enabled: yes

    # Create the directory that will hold your site content
    - name: Create web content directory
      file:
        path: /var/www/html
        state: directory
        mode: '0755'

    # This copies your index.html from your laptop into the VM
    - name: Copy site content to web server
      copy:
        src: index.html
        dest: /var/www/html/index.html

    # This mounts that file into the Nginx container so it serves your page
    # The -v flag connects /var/www/html on the VM to /usr/share/nginx/html inside the container
    - name: Run Nginx serving your content
      shell: |
        docker rm -f webapp 2>/dev/null || true
        docker run -d --name webapp --restart always -p 80:80 \
          -v /var/www/html:/usr/share/nginx/html:ro nginx

- name: Configure database server
  hosts: dbservers
  become: yes
  tasks:

    # Hash sum mismatch on .deb downloads is often stale lists, a flaky mirror, or apt pipelining
    # behind NAT; fresh indices + Pipeline-Depth 0 usually fixes it on lab VMs.
    - name: Disable apt HTTP pipelining (mirror/proxy hash mismatch workaround)
      copy:
        dest: /etc/apt/apt.conf.d/99disable-pipelining
        content: 'Acquire::http::Pipeline-Depth "0";'
        mode: "0644"

    - name: Clear apt package index cache
      shell: apt-get clean && rm -rf /var/lib/apt/lists/* /var/lib/apt/lists/auxfiles/*
      changed_when: true

    - name: Update apt cache after reset
      apt:
        update_cache: yes

    - name: Install MariaDB
      apt:
        name: mariadb-server
        state: present
        update_cache: no

    - name: Start MariaDB service
      service:
        name: mariadb
        state: started
        enabled: yes

Two lines worth paying attention to:

• src: index.html — Ansible looks for this file in the same directory as the playbook. That is the file you wrote in Step 2.

• -v /var/www/html:/usr/share/nginx/html:ro — this mounts the directory from the VM into the Nginx container. The :ro means read-only. Nginx serves whatever is in that folder.

Step 8: Run the Playbook

ansible-playbook -i inventory playbook.yml

You'll see task-by-task output as Ansible connects to each VM over SSH and configures it. A green ok or yellow changed next to each task means it worked. Red fatal means something failed.

Step 9: Verify the Setup

Open http://localhost:8080 in your browser. You should see the page you wrote in Step 2 served from inside a Docker container, running on a Vagrant VM, configured automatically by Ansible.

If you see the page, every tool in this lab is working together.

Step 9: Clean Up (Optional)

When you're done:

vagrant destroy -f

This shuts down and deletes both VMs. Your Vagrantfile, inventory, playbook.yml, and index.html stay on disk — run vagrant up followed by ansible-playbook -i inventory playbook.yml any time to bring it all back.

Now that you have a working lab, let's use it properly.

How to Break Your Lab on Purpose

Following these steps has gotten you a running lab. Breaking things teaches you how everything actually works.

Here are five things to break and what to look for when you do.

Break 1: Crash the Main Process Inside the Container (and Watch It Come Back)

Doing this just proves that something inside the container can die (like a real bug or OOM), Docker can restart the container because of --restart always, and your site can come back without re-running Ansible.

After vagrant ssh web, every docker command below runs on the web VM. So keep your browser on your laptop at http://localhost:8080 (Vagrant forwards your host port to the VM’s port 80).

Troubleshooting: If Your Lab Isn't Ready

From your project folder on the host (your laptop) – unless the step says to run it on the VM:

• You ran vagrant destroy -f. Run vagrant up, then ansible-playbook -i inventory playbook.yml.

• docker ps shows webapp but status is Exited. On the web VM, run sudo docker start webapp, then sudo docker ps again.

• There's no webapp row in docker ps -a. Re-run ansible-playbook -i inventory playbook.yml on the host.

If the playbook is already applied and webapp is Up, skip this section and start at step 1 under Steps (happy path) below. (Don't skip SSH or docker ps. You need the VM shell and a quick check before you run docker exec.)

Steps (happy path)

1. SSH into the web VM:

vagrant ssh web

1. Confirm webapp is Up:

   sudo docker ps
   

2. Break it on purpose: kill the container’s main process from inside (PID 1). That ends the container the same way a crashing app would, not the same as docker stop on the host:

sudo docker exec webapp sh -c 'sleep 5 && kill 1'

The sleep 5 gives you a moment to switch to the browser. Right after you run the command, open or refresh http://localhost:8080. You may catch a brief error or blank page while nothing is listening on port 80.

1. Watch Docker restart the container:

watch sudo docker ps -a

Within a few seconds you should see Exited (137) become Up again. (Press Ctrl+C to exit watch.)

5. Refresh the browser. You should see the same HTML as before, because the files live on the VM under /var/www/html and are bind-mounted into the container; restarting only replaced the Nginx process, not those files.

Why not docker stop or docker kill on the host for this demo?

Those commands go through Docker’s API. On many setups (including recent Docker), Docker treats them as you choosing to stop the container (hasBeenManuallyStopped), and --restart always may not bring the container back until you docker start it or similar.

Killing PID 1 from inside the container is treated more like an internal crash, so the restart policy you set in the playbook is the one you actually get to observe here.

Kubernetes analogy: A pod whose containers exit can be restarted by the kubelet; a pod you delete does not come back by itself.

What to observe (three separate checks):

1. Exit code: After kill 1, docker ps -a should show the container exited with code 137, meaning the main process was killed by a signal. That confirms the container really died, not that you ran docker stop on the host.

2. Restart delay vs browser: Watch how many seconds pass between Exited and Up in docker ps -a; that interval is Docker applying --restart always. That's separate from what you see in the browser: the browser only shows whether something is accepting connections on port 80 on the VM, so it may show an error or blank page during the gap even while Docker is about to restart the container.

3. Content after recovery: After status is Up again, refresh the page. You should see the same HTML as before. That shows your content lives on the VM disk (mounted into the container with -v), not inside a file that vanishes when the container process restarts. The process was replaced, not your index.html on the host path.

Break 2: Cause a Container Name Conflict

On a single Docker daemon (here, on your web VM), a container name is a unique label. Two running (or stopped) containers can't share the same name. Scripts and playbooks that always use docker run --name webapp without cleaning up first hit this error constantly and recognizing it saves time in real work.

Before you start: Ansible already created one container named webapp.
Stay on the web VM (for example still inside vagrant ssh web) so the commands below run where that container lives.

So now, try to start a second container and also call it webapp. The image is plain nginx here on purpose – the point is the name clash, not matching your site’s ports or volume mounts.

sudo docker run -d --name webapp nginx

What actually happens here is that Docker doesn't create a second container. It returns an error immediately. Your original webapp is unchanged.

This is because the name webapp is already registered to the existing container (the error shows that container’s ID). Docker refuses to reuse the name until the old container is removed or renamed.

Example error (your ID will differ):

docker: Error response from daemon: Conflict. The container name "/webapp" is already in use by container "2e48b81a311c4b71cdc1e25e0df75a22296845c7eb53aab82f9ae739fb6410ec". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

To fix it, free the name, then create webapp again the same way the playbook does (publish port 80, mount your HTML, restart policy):

sudo docker rm -f webapp
sudo docker run -d --name webapp --restart always -p 80:80 \
  -v /var/www/html:/usr/share/nginx/html:ro nginx

After that, your site should behave as before (refresh http://localhost:8080 from your laptop).

What to observe:

Read Docker’s Conflict message end to end. You should see that the name /webapp is already in use and a container ID pointing at the existing box. In production, that pattern means “something already claimed this name. Just remove it, rename it, or pick a different name before you run docker run again.”

Break 3: Make Ansible Fail to Reach a VM

Ansible separates “could not connect” from “connected, but a task broke.” The first is UNREACHABLE, the second is FAILED. Knowing which one you have tells you whether to fix network / SSH or playbook / packages / permissions.

On your laptop, in the project folder, edit inventory and change the web server address from 192.168.33.10 to an IP no VM uses, for example 192.168.33.99. Save the file.

[webservers]
192.168.33.99 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

What you run (from the same project folder on the host):

ansible-playbook -i inventory playbook.yml

After this, Ansible tries to SSH to 192.168.33.99. Nothing on your lab network answers as that host (or SSH never succeeds), so Ansible never runs tasks on the web server. It stops that host with UNREACHABLE:

fatal: [192.168.33.99]: UNREACHABLE! => {"msg": "Failed to connect to the host via ssh"}

This is realistic because the same message shape appears when the IP is wrong, the VM isn't running, a firewall blocks port 22, or the network is misconfigured. The common thread is no working SSH session.

Now it's time to put it back: restore 192.168.33.10 in inventory and run ansible-playbook -i inventory playbook.yml again. The web play should reach the VM and complete (assuming your lab is up).

UNREACHABLE vs FAILED – what to observe:

• If Ansible prints UNREACHABLE, you should assume it never opened SSH on that host and never ran tasks there. Go ahead and fix the connection (IP, VM up, firewall, key path) before you debug playbook logic.

• If Ansible prints FAILED, you should assume SSH worked and a task returned an error. Read the task output for the real cause (package name, permissions, syntax), not the network first.

When you debug later, you should look at the keyword Ansible prints: UNREACHABLE points to reachability while FAILED points to task output and the first failed task under that host.

Break 4: Fill the VM's Disk

Databases and other services need free disk for logs, temp files, and data. When the filesystem is full or nearly full, a service may fail to start or fail at runtime. This break walks through the same diagnosis habit you would use on a real server: check space, then read systemd and journal output for the service.

All commands below run on the db VM after vagrant ssh db. MariaDB was installed there by your playbook.

What you do:

1. Open a shell on the db VM:

   vagrant ssh db
   

2. Allocate a large file full of zeros (here 1GB) to simulate something eating disk space:

   sudo dd if=/dev/zero of=/tmp/bigfile bs=1M count=1024
   
   df -h
   

   Use df -h to see how full the root filesystem (or relevant mount) is. Your Vagrant disk may be large enough that 1GB only raises usage. If MariaDB still starts, you still practiced the checks. To see a stronger effect, you can repeat with a larger count= only in a lab (never fill production disks on purpose without a plan).

3. Ask systemd to restart MariaDB and show status:

   sudo systemctl restart mariadb
   sudo systemctl status mariadb
   

   If the disk is critically full, restart may fail or the service may show failed or not running.

4. If something looks wrong, read recent logs for the MariaDB unit:

   sudo journalctl -u mariadb --no-pager | tail -20
   

   Errors often mention disk, space, read-only filesystem, or InnoDB being unable to write.

5. Clean up so your VM stays usable:

   sudo rm /tmp/bigfile
   

   Optionally run sudo systemctl restart mariadb again and confirm it is active (running).

What to observe:

• You should use df -h first to confirm whether the filesystem is actually tight. That avoids blaming the database when disk space is fine.

• You should read systemctl status mariadb to see whether systemd thinks the service is active, failed, or flapping.

• You should read journalctl -u mariadb when status is bad, so you can tie the failure to concrete errors from MariaDB or the kernel (often mentioning disk, space, or read-only filesystem). Space + status + logs is the same order you would use on a production server.

Break 5: Run Minikube Out of Resources

Kubernetes schedules pods onto nodes that have enough CPU and memory. If you ask for more than the cluster can place, some pods stay Pending and Events explain why (for example Insufficient cpu). That is not the same as a pod that starts and then crashes.

To do this, you'll need a local cluster (we're using Minikube in this guide) and kubectl on your laptop. This break doesn't use the Vagrant VMs. If you haven't installed Minikube yet, complete the "How to Set Up Kubernetes" section first, or skip this break until you do.

You'll run this on your Mac, Linux, or Windows terminal (host), not inside vagrant ssh. If you're still inside a VM, type exit until your prompt is back on the host.

What you do:

1. Check Minikube:

   minikube status
   

   If it's stopped, start it (Docker driver matches earlier sections):

   minikube start --driver=docker
   

2. Create a deployment with many replicas so your single Minikube node can't run them all at once:

   kubectl create deployment stress --image=nginx --replicas=20
   
   #watch pods start
   kubectl get pods -w
   

   Press Ctrl+C when you're done watching. Some pods may stay Pending while others are Running.

3. Pick one Pending pod name from kubectl get pods and inspect it:

   kubectl describe pod <pod-name>
   

   Under Events, look for FailedScheduling and a line similar to:

   Warning  FailedScheduling  0/1 nodes are available: 1 Insufficient cpu.
   

   You might see Insufficient memory instead, depending on your machine.

4. Fix the lab by scaling back so the cluster can catch up:

   kubectl scale deployment stress --replicas=2
   

   You can delete the deployment entirely when finished: kubectl delete deployment stress.

What to observe:

• You should see Pending pods stay unscheduled until capacity frees up. That means the scheduler hasn't placed them on any node yet, usually because the node is out of CPU or memory for that workload.

• You should read kubectl describe pod <pod-name> and scroll to Events. Messages like Insufficient cpu or Insufficient memory mean the cluster ran out of schedulable capacity, not that the container image image is corrupt.

• You should contrast that with a pod that reaches Running and then CrashLoopBackOff, which usually means the process inside the container keeps exiting. that is an application or config problem, not a “nowhere to run” problem.

What You Can Now Do

You didn't just install tools in this tutorial. You also used them.

You can now spin up two servers from a single file. You can write a playbook that installs software and deploys a container without touching either machine manually.

You can serve a page you wrote from inside a Docker container running on a Vagrant VM, and bring the whole thing back from scratch in one command.

You also broke it. You saw what a container conflict looks like, what Ansible prints when it can't reach a machine, what disk pressure does to a running service, and what a Kubernetes scheduler says when it runs out of resources. Those error messages aren't unfamiliar anymore.

That's the difference between someone who has read about DevOps and someone who has run it.

Here are four free projects you can run in this same lab to go further:

• DevOps Home-Lab 2026 — Build a multi-service app (frontend, API, PostgreSQL, Redis) end-to-end with Docker Compose, Kubernetes, Prometheus/Grafana monitoring, GitOps with ArgoCD, and Cloudflare for global exposure.

• KubeLab — Trigger real Kubernetes failure scenarios, pod crashes, OOMKills, node drains, cascading failures, and watch how the cluster responds using live metrics.

• K8s Secrets Lab — Build a full secret management pipeline from AWS Secrets Manager into your cluster, including rotation behavior and IRSA.

• DevOps Troubleshooting Toolkit — Structured debugging guides across Linux, containers, Kubernetes, cloud, databases, and observability with copy-paste commands for real incidents.

All free and open source: github.com/Osomudeya/List-Of-DevOps-Projects.

If you want to go deeper, you can find six full chapters covering Terraform, Ansible, monitoring, CI/CD, and a simulated three-VM production environment at Build Your Own DevOps Lab.

Google Cloud Axion is Google's own custom ARM-based chip, built to handle the demands of modern cloud workloads. The performance and cost numbers are striking: Google claims Axion delivers up to 60% better energy efficiency and up to 65% better price-performance compared to comparable x86 machines.

AWS has Graviton. Azure has Cobalt. ARM is no longer niche. It's the direction the entire cloud industry is moving.

But there's a problem that catches almost every team off guard when they start this transition: container architecture mismatch.

If you build a Docker image on your M-series Mac and push it to an x86 server, it crashes on startup with a cryptic exec format error.

The server isn't broken. It just can't read the compiled instructions inside your image. An ARM binary and an x86 binary are written in fundamentally different languages at the machine level. The CPU literally can't execute instructions it wasn't designed for.

We're going to solve this problem completely in this tutorial. You'll build a single Docker image tag that automatically serves the correct binary on both ARM and x86 machines — no separate pipelines, no separate tags. Then you'll provision Google Cloud ARM nodes in GKE and configure your Kubernetes deployment to route workloads precisely to those cost-efficient nodes.

Here's what you'll build, step by step:

• A Go HTTP server that reports the CPU architecture it's running on at runtime

• A multi-stage Dockerfile that cross-compiles for both linux/amd64 and linux/arm64 without slow QEMU emulation

• A multi-arch image in Google Artifact Registry that acts as a single entry point for any architecture

• A GKE cluster with two node pools: a standard x86 pool and an ARM Axion pool

• A Kubernetes Deployment that pins your workload exclusively to the ARM nodes

By the end, you'll hit a live endpoint and see the word arm64 staring back at you from a Google Cloud ARM node. Let's get into it.

Table of Contents

• Prerequisites

• Step 1: Set Up Your Google Cloud Project

• Step 2: Create the GKE Cluster

• Step 3: Write the Application

• Step 4: Enable Multi-Arch Builds with Docker Buildx

• Step 5: Write the Dockerfile

• Step 6: Build and Push the Multi-Arch Image

• Step 7: Add the Axion ARM Node Pool

• Step 8: Deploy the App to the ARM Node Pool

• Step 9: Verify the Deployment

• Step 10: Cost Savings and Tradeoffs

• Cleanup

• Conclusion

• Project File Structure

Prerequisites

Before you start, make sure you have the following ready:

• A Google Cloud project with billing enabled. If you don't have one, create it at console.cloud.google.com. The total cost to follow this tutorial is around $5–10.

• gcloud CLI installed and authenticated. Run gcloud auth login to sign in and gcloud config set project YOUR_PROJECT_ID to point it at your project.

• Docker Desktop version 19.03 or later. Docker Buildx (the tool we'll use for multi-arch builds) ships bundled with it.

• kubectl installed. This is the CLI for interacting with Kubernetes clusters.

• Basic familiarity with Docker (images, layers, Dockerfile) and Kubernetes (pods, deployments, services). You don't need to be an expert, but you should know what these things are.

Step 1: Set Up Your Google Cloud Project

Before writing a single line of application code, let's get the cloud infrastructure side ready. This is the foundation everything else will build on.

Enable the Required APIs

Google Cloud services are off by default in any new project. Run this command to turn on the three APIs we'll need:

gcloud services enable \
  artifactregistry.googleapis.com \
  container.googleapis.com \
  containeranalysis.googleapis.com

Here's what each one does:

• artifactregistry.googleapis.com — enables Artifact Registry, where we'll store our Docker images

• container.googleapis.com — enables Google Kubernetes Engine (GKE), where our cluster will run

• containeranalysis.googleapis.com — enables vulnerability scanning for images stored in Artifact Registry

Create a Docker Repository in Artifact Registry

Artifact Registry is Google Cloud's managed container image store — the place where our built images will live before being deployed to the cluster. Create a dedicated repository for this tutorial:

gcloud artifacts repositories create multi-arch-repo \
  --repository-format=docker \
  --location=us-central1 \
  --description="Multi-arch tutorial images"

Breaking down the flags:

• --repository-format=docker — tells Artifact Registry this repository stores Docker images (as opposed to npm packages, Maven artifacts, and so on)

• --location=us-central1 — the Google Cloud region where your images will be stored. Use a region that's close to where your cluster will run to minimize image pull latency. Run gcloud artifacts locations list to see all options.

• --description — a human-readable label for the repository, shown in the console.

Authenticate Docker to Push to Artifact Registry

Docker needs credentials before it can push images to Google Cloud. Run this command to wire up authentication automatically:

gcloud auth configure-docker us-central1-docker.pkg.dev

This adds a credential helper entry to your ~/.docker/config.json file. What that means in practice: any time Docker tries to push or pull from a URL under us-central1-docker.pkg.dev, it will automatically call gcloud to get a valid auth token. You won't need to run docker login manually.

Step 2: Create the GKE Cluster

With Artifact Registry ready to receive images, let's create the Kubernetes cluster. We'll start with a standard cluster using x86 nodes and add an ARM node pool later once we have an image to deploy.

gcloud container clusters create axion-tutorial-cluster \
  --zone=us-central1-a \
  --num-nodes=2 \
  --machine-type=e2-standard-2 \
  --workload-pool=PROJECT_ID.svc.id.goog

Replace PROJECT_ID with your actual Google Cloud project ID.

What each flag does:

• --zone=us-central1-a — creates a zonal cluster in a single availability zone. A regional cluster (using --region) would spread nodes across three zones for higher resilience, but for this tutorial a single zone keeps things simple and avoids capacity issues that can affect specific zones. If us-central1-a is unavailable, try us-central1-b.

• --num-nodes=2 — two x86 nodes in this zone. We need at least 2 to have enough capacity alongside our ARM node pool later.

• --machine-type=e2-standard-2 — the machine type for this default node pool. e2-standard-2 is a cost-effective x86 machine with 2 vCPUs and 8 GB of memory, good for general workloads.

• --workload-pool=PROJECT_ID.svc.id.goog — enables Workload Identity, which is Google's recommended way for pods to authenticate with Google Cloud APIs. It avoids the need to download and store service account key files inside your cluster.

This command takes a few minutes. While it runs, you can move on to writing the application. We'll come back to the cluster in Step 6.

Step 3: Write the Application

We need an application to containerize. We'll use Go for three specific reasons:

1. Go compiles into a single, statically-linked binary. There's no runtime to install, no interpreter — just the binary. This makes for extremely lean container images.

2. Go has first-class, built-in cross-compilation support. We can compile an ARM64 binary from an x86 Mac, or vice versa, by setting two environment variables. This will matter a lot when we get to the Dockerfile.

3. Go exposes the architecture the binary was compiled for via runtime.GOARCH. Our server will report this at runtime, giving us hard proof that the correct binary is running on the correct hardware.

Start by creating the project directories:

mkdir -p hello-axion/app hello-axion/k8s
cd hello-axion/app

Initialize the Go module from inside app/. This creates go.mod in the current directory:

go mod init hello-axion

go mod init is Go's built-in command for starting a new module. It writes a go.mod file that declares the module name (hello-axion) and the minimum Go version required. Every modern Go project needs this file — without it, the compiler doesn't know how to resolve packages.

Now create the application at app/main.go:

package main

import (
    "fmt"
    "net/http"
    "os"
    "runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
    hostname, _ := os.Hostname()
    fmt.Fprintf(w, "Hello from freeCodeCamp!\n")
    fmt.Fprintf(w, "Architecture : %s\n", runtime.GOARCH)
    fmt.Fprintf(w, "OS           : %s\n", runtime.GOOS)
    fmt.Fprintf(w, "Pod hostname : %s\n", hostname)
}

func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintln(w, "ok")
}

func main() {
    http.HandleFunc("/", handler)
    http.HandleFunc("/healthz", healthz)
    fmt.Println("Server starting on port 8080...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Fprintf(os.Stderr, "server error: %v\n", err)
        os.Exit(1)
    }
}

Verify both files were created:

ls -la

You should see go.mod and main.go listed.

Let's walk through what this code does:

• import "runtime" — imports Go's built-in runtime package, which exposes information about the Go runtime environment, including the CPU architecture.

• runtime.GOARCH — returns a string like "arm64" or "amd64" representing the architecture this binary was compiled for. When we deploy to an ARM node, this value will be arm64. This is the core of our proof.

• os.Hostname() — returns the pod's hostname, which Kubernetes sets to the pod name. This lets us see which specific pod responded when we test the app later.

• handler — the main HTTP handler, registered on the root path /. It writes the architecture, OS, and hostname to the response.

• healthz — a separate handler registered on /healthz. It returns HTTP 200 with the text ok. Kubernetes will use this endpoint to check whether the container is alive and ready to serve traffic — we'll wire this up in the deployment manifest later.

• http.ListenAndServe(":8080", nil) — starts the server on port 8080. If it fails to start (for example, if the port is already in use), it prints the error and exits with a non-zero code so Kubernetes knows something went wrong.

Step 4: Enable Multi-Arch Builds with Docker Buildx

Before we write the Dockerfile, we need to understand a fundamental constraint, because it directly shapes how the Dockerfile must be written.

Why Your Docker Images Are Architecture-Specific By Default

A CPU only understands instructions written for its specific Instruction Set Architecture (ISA). ARM64 and x86_64 are different ISAs — different vocabularies of machine-level operations. When you compile a Go program, the compiler translates your source code into binary instructions for exactly one ISA. That binary can't run on a different ISA.

When you build a Docker image the normal way (docker build), the binary inside that image is compiled for your local machine's ISA. If you're on an Apple Silicon Mac, you get an ARM64 binary. Push that image to an x86 server, and when Docker tries to execute the binary, the kernel rejects it:

standard_init_linux.go:228: exec user process caused: exec format error

That's the operating system saying: "This binary was written for a different processor. I have no idea what to do with it."

The Solution: A Single Image Tag That Serves Any Architecture

Docker solves this with a structure called a Manifest List (also called a multi-arch image index). Instead of one image, a Manifest List is a pointer table. It holds multiple image references — one per architecture — all under the same tag.

When a server pulls hello-axion:v1, here's what actually happens:

1. Docker contacts the registry and requests the manifest for hello-axion:v1

2. The registry returns the Manifest List, which looks like this internally:

{
  "manifests": [
    { "digest": "sha256:a1b2...", "platform": { "architecture": "amd64", "os": "linux" } },
    { "digest": "sha256:c3d4...", "platform": { "architecture": "arm64", "os": "linux" } }
  ]
}

1. Docker checks the current machine's architecture, finds the matching entry, and pulls only that specific image layer. The x86 image never downloads onto your ARM server, and vice versa.

One tag, two actual images. Completely transparent to your deployment manifests.

Set Up Docker Buildx

Docker Buildx is the CLI tool that builds these Manifest Lists. It's powered by the BuildKit engine and ships bundled with Docker Desktop. Run the following to create and activate a new builder instance:

docker buildx create --name multiarch-builder --use

• --name multiarch-builder — gives this builder a memorable name. You can have multiple builders. This command creates a new one named multiarch-builder.

• --use — immediately sets this new builder as the active one, so all future docker buildx build commands use it.

Now boot the builder and confirm it supports the platforms we need:

docker buildx inspect --bootstrap

• --bootstrap — starts the builder container if it isn't already running, and prints its full configuration.

You should see output like this:

Name:          multiarch-builder
Driver:        docker-container
Platforms:     linux/amd64, linux/arm64, linux/arm/v7, linux/386, ...

The Platforms line lists every architecture this builder can produce images for. As long as you see linux/amd64 and linux/arm64 in that list, you're ready to build for both x86 and ARM.

Step 5: Write the Dockerfile

Now we can write the Dockerfile. We'll use two techniques together: a multi-stage build to keep the final image tiny, and a cross-compilation trick to avoid slow CPU emulation.

Create app/Dockerfile with the following content:

# -----------------------------------------------------------
# Stage 1: Build
# -----------------------------------------------------------
# $BUILDPLATFORM = the machine running this build (your laptop)
# \(TARGETOS / \)TARGETARCH = the platform we are building FOR
# -----------------------------------------------------------
FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

ARG TARGETOS
ARG TARGETARCH

WORKDIR /app

COPY go.mod .
RUN go mod download

COPY main.go .

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

# -----------------------------------------------------------
# Stage 2: Runtime
# -----------------------------------------------------------

FROM alpine:latest

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

WORKDIR /app
COPY --from=builder /app/server .

EXPOSE 8080
CMD ["./server"]

There's a lot happening here. Let's go through it carefully.

Stage 1: The Builder

FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

This is the most important line in the file. \(BUILDPLATFORM is a special build argument that Docker Buildx automatically injects — it equals the platform of the machine running the build (your laptop). By pinning the builder stage to \)BUILDPLATFORM, the Go compiler always runs natively on your machine, not inside a CPU emulator. This is what makes multi-arch builds fast.

Without --platform=$BUILDPLATFORM, Buildx would have to use QEMU — a full CPU emulator — to run an ARM64 build environment on your x86 machine (or vice versa). QEMU works, but it's typically 5–10 times slower than native execution. For a project with many dependencies, that's the difference between a 2-minute build and a 20-minute build.

ARG TARGETOS and ARG TARGETARCH

These two lines declare that our Dockerfile expects build arguments named TARGETOS and TARGETARCH. Buildx injects these automatically based on the --platform flag you pass at build time. For a linux/arm64 target, TARGETOS will be linux and TARGETARCH will be arm64.

COPY go.mod . and RUN go mod download

We copy go.mod first, before copying the rest of the source code. Docker builds images layer by layer and caches each layer. By copying only the module file first, we create a cached layer for go mod download.

On future builds, as long as go.mod hasn't changed, Docker skips the download step entirely — even if the source code changed. This speeds up iterative development significantly.

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

This is the cross-compilation step. GOOS and GOARCH are Go's built-in cross-compilation environment variables. Setting them tells the Go compiler to produce a binary for a different target than the machine it's running on. We set them from the \(TARGETOS and \)TARGETARCH build args injected by Buildx.

The -ldflags="-w -s" flag strips the debug symbol table and the DWARF debugging information from the binary. This has no effect on runtime behavior but reduces the binary size by roughly 30%.

Stage 2: The Runtime Image

FROM alpine:latest

This starts a brand-new image from Alpine Linux — a minimal Linux distribution that weighs about 5 MB. Critically, alpine:latest is itself a multi-arch image, so Docker automatically selects the arm64 or amd64 Alpine variant depending on which platform this stage is built for.

Everything from Stage 1 — the Go toolchain, the source files, the intermediate object files — is discarded. The final image contains only Alpine Linux plus our binary. Compared to a naive single-stage Go image (~300 MB), this approach produces an image under 15 MB.

RUN addgroup -S appgroup && adduser -S appuser -G appgroup and USER appuser

These two lines create a non-root user and set it as the active user for the container. Running containers as root is a security risk — if an attacker exploits a vulnerability in your application, they gain root access inside the container. Running as a non-root user limits the blast radius.

COPY --from=builder /app/server .

This is how multi-stage builds work: the --from=builder flag tells Docker to copy files from the builder stage (Stage 1), not from your local disk. Only the compiled binary (server) makes it into the final image.

Step 6: Build and Push the Multi-Arch Image

With the application and Dockerfile in place, we can now build images for both architectures and push them to Artifact Registry — all in a single command.

From inside the app/ directory, run:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 \
  --push \
  .

Replace PROJECT_ID with your actual GCP project ID.

Here's what each part of this command does:

• docker buildx build — uses the Buildx CLI instead of the standard docker build. Buildx is required for multi-platform builds.

• --platform linux/amd64,linux/arm64 — instructs Buildx to build the image twice: once targeting x86 Intel/AMD machines, and once targeting ARM64. Both builds run in parallel. Because our Dockerfile uses the $BUILDPLATFORM cross-compilation trick, both builds run natively on your machine without QEMU emulation.

• -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 — the full image path in Artifact Registry. The format is always REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME:TAG.

• --push — multi-arch images can't be loaded into your local Docker daemon (which only understands single-architecture images). This flag tells Buildx to skip local storage and push the completed Manifest List — with both architecture variants — directly to the registry.

• . — the build context, the directory Docker scans for the Dockerfile and any files the build needs.

Watch the output as the build runs. You'll see BuildKit working on both platforms simultaneously:

 => [linux/amd64 builder 1/5] FROM golang:1.23-alpine
 => [linux/arm64 builder 1/5] FROM golang:1.23-alpine
 ...
 => pushing manifest for us-central1-docker.pkg.dev/.../hello-axion:v1

Verify the Multi-Arch Image in Artifact Registry

Once the push completes, navigate to GCP Console → Artifact Registry → Repositories → multi-arch-repo and click on hello-axion.

You won't see a single image — you'll see something labelled "Image Index". That's the Manifest List we created. Click into it, and you'll find two child images with separate digests, one for linux/amd64 and one for linux/arm64.

You can also inspect this from the command line:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

The output lists every manifest inside the image index. You'll see entries for linux/amd64 and linux/arm64 — those are our two real images. You'll also see two entries with Platform: unknown/unknown labelled as attestation-manifest. These are build provenance records that Docker Buildx automatically attaches to prove how and where the image was built (a supply chain security feature called SLSA attestation).

The two entries you care about are linux/amd64 and linux/arm64. Note the digest for the arm64 entry — we'll use it in the verification step to confirm the cluster pulled the right variant.

Step 7: Add the Axion ARM Node Pool

We have a universal image. Now we need somewhere to run it.

Recall the cluster we created in Step 2 — it's running e2-standard-2 x86 machines. We're going to add a second node pool running ARM machines. This is the key architectural move: a mixed-architecture cluster where different workloads can be routed to different hardware.

Choosing Your ARM Machine Type

Google Cloud currently offers two ARM-based machine series in GKE:

This tutorial uses t2a-standard-2 because it's widely available. The commands are identical for c4a-standard-2 — just swap the --machine-type value. If t2a-standard-2 isn't available in your zone, GKE will tell you immediately when you run the node pool creation command below, and you can try a neighbouring zone.

Create the ARM Node Pool

Add the ARM node pool to your existing cluster:

gcloud container node-pools create axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a \
  --machine-type=t2a-standard-2 \
  --num-nodes=2 \
  --node-labels=workload-type=arm-optimized

What each flag does:

• --cluster=axion-tutorial-cluster — the name of the cluster we created in Step 2. Node pools are always added to an existing cluster.

• --zone=us-central1-a — must match the zone you used when creating the cluster.

• --machine-type=t2a-standard-2 — GKE detects this is an ARM machine type and automatically provisions the nodes with an ARM-compatible version of Container-Optimized OS (COS). You don't need to configure anything special for ARM at the OS level.

• --num-nodes=2 — two ARM nodes in the zone, enough to schedule our 3-replica deployment alongside other cluster overhead.

• --node-labels=workload-type=arm-optimized — attaches a custom label to every node in this pool. We'll use this label in our deployment manifest to target these specific nodes. Using a descriptive custom label (rather than just relying on the automatic kubernetes.io/arch=arm64 label) is good practice in real clusters — it communicates the intent of the pool, not just its hardware.

This command takes a few minutes. Once it completes, let's confirm our cluster now has both node pools:

gcloud container clusters get-credentials axion-tutorial-cluster --zone=us-central1-a

kubectl get nodes --label-columns=kubernetes.io/arch

The get-credentials command configures kubectl to authenticate with your new cluster. The get nodes command then lists all nodes and adds a column showing the kubernetes.io/arch label.

You should see something like:

NAME                                    STATUS   ARCH    AGE
gke-...default-pool-abc...              Ready    amd64   15m
gke-...default-pool-def...              Ready    amd64   15m
gke-...axion-pool-jkl...                Ready    arm64   3m
gke-...axion-pool-mno...                Ready    arm64   3m

amd64 for the default x86 pool, arm64 for our new Axion pool. This kubernetes.io/arch label is applied automatically by GKE — you don't set it, it's derived from the hardware.

Step 8: Deploy the App to the ARM Node Pool

We have a multi-arch image and a mixed-architecture cluster. Here's something important to understand before writing the deployment manifest: Kubernetes doesn't know or care about image architecture by default.

If you applied a standard Deployment right now, the scheduler would look for any available node with enough CPU and memory and place pods there — potentially landing on x86 nodes instead of your ARM Axion nodes. The multi-arch Manifest List would handle this gracefully (the right binary would run regardless), but you'd lose the cost efficiency you provisioned Axion nodes for in the first place.

To guarantee that pods land on ARM nodes and only ARM nodes, we use a nodeSelector.

How nodeSelector Works

A nodeSelector is a set of key-value pairs in your pod spec. Before the Kubernetes scheduler places a pod, it checks every available node's labels. If a node doesn't have all the labels in the nodeSelector, the scheduler skips it — the pod will remain in Pending state rather than land on the wrong node.

This is a hard constraint, which is exactly what we want for cost optimization. Contrast this with Node Affinity's soft preference mode (preferredDuringSchedulingIgnoredDuringExecution), which says "try to use ARM, but fall back to x86 if needed." Soft preferences are useful for resilience, but they undermine the whole point of dedicated ARM pools. We want the hard constraint.

Write the Deployment Manifest

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-axion
  labels:
    app: hello-axion
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-axion
  template:
    metadata:
      labels:
        app: hello-axion
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64

      containers:
      - name: hello-axion
        image: us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
        resources:
          requests:
            cpu: "250m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

Replace PROJECT_ID with your project ID. Here's what the key sections do:

replicas: 3 — tells Kubernetes to keep three instances of this pod running at all times. If one crashes or a node goes down, the scheduler spins up a replacement. Three replicas also means one pod per ARM node in us-central1, which distributes load across availability zones.

selector.matchLabels and template.metadata.labels — these two blocks must match. The selector tells the Deployment which pods it "owns," and the template.metadata.labels is what those pods will be tagged with. If they don't match, Kubernetes won't be able to manage the pods.

nodeSelector: kubernetes.io/arch: arm64 — this is the pin. The Kubernetes scheduler filters out every node that doesn't carry this label before considering resource availability. Since GKE automatically applies kubernetes.io/arch=arm64 to all ARM nodes, our pods will schedule only onto the axion-pool nodes.

livenessProbe — periodically calls GET /healthz. If this check fails a certain number of times in a row (indicating the container has deadlocked or is otherwise unresponsive), Kubernetes restarts the container. initialDelaySeconds: 5 gives the server 5 seconds to start up before the first check.

readinessProbe — similar to the liveness probe, but with a different purpose. While the readiness probe is failing, Kubernetes removes the pod from the service's load balancer, so no traffic is sent to it. This is important during startup — the pod won't receive traffic until it signals it's ready.

resources.requests — reserves 250m (25% of a CPU core) and 64Mi of memory on the node for this pod. The scheduler uses these numbers to decide whether a node has enough room for the pod. Setting requests is required for sensible bin-packing. Without them, nodes can be silently overcommitted.

resources.limits — caps the container at 500m CPU and 128Mi memory. If the container exceeds these limits, Kubernetes throttles the CPU or kills the container (for memory). This prevents a single misbehaving pod from starving other workloads on the same node.

A Note on Taints and Tolerations

Once you're comfortable with nodeSelector, the next step in production clusters is adding a taint to your ARM node pool. A taint is a repellent — any pod without an explicit toleration for that taint is blocked from landing on the tainted node.

This means other workloads in your cluster can't accidentally consume your ARM capacity. You'd add the taint when creating the pool:

# Add --node-taints to the pool creation command:
--node-taints=workload-type=arm-optimized:NoSchedule

And a matching toleration in the pod spec:

tolerations:
- key: "workload-type"
  operator: "Equal"
  value: "arm-optimized"
  effect: "NoSchedule"

We're not doing this in the tutorial to keep things simple, but it's the pattern production multi-tenant clusters use to enforce hard separation between workload types.

Write the Service Manifest

We also need a Kubernetes Service to expose the pods over the network. Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-axion-svc
spec:
  selector:
    app: hello-axion
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

• selector: app: hello-axion — the Service discovers pods using labels. Any pod with app: hello-axion on it will be added to this Service's load balancer pool.

• port: 80 — the port the Service is reachable on from outside the cluster.

• targetPort: 8080 — the port on the pod that traffic gets forwarded to. Our Go server listens on port 8080, so this must match.

• type: LoadBalancer — tells GKE to provision an external Google Cloud load balancer and assign it a public IP. This is what makes the Service reachable from the internet.

Apply Both Manifests

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl apply reads each manifest file and creates or updates the resources described in it. If the resources don't exist yet, they're created. If they already exist, Kubernetes only applies the diff — it won't restart pods unnecessarily.

Watch the pods come up in real time:

kubectl get pods -w

The -w flag watches for changes and prints updates as they happen. You should see pods transition from Pending → ContainerCreating → Running. Once all three show Running, press Ctrl+C to stop watching.

Step 9: Verify the Deployment

Everything is running. Now we need evidence — not just that pods are up, but that they're on the right nodes and serving the right binary.

Confirm Pod Placement

kubectl get pods -o wide

The -o wide flag adds extra columns to the output, including the name of the node each pod was scheduled on. Look at the NODE column:

NAME                          READY   STATUS    NODE
hello-axion-7b8d9f-abc12      1/1     Running   gke-axion-tutorial-axion-pool-a-...
hello-axion-7b8d9f-def34      1/1     Running   gke-axion-tutorial-axion-pool-b-...
hello-axion-7b8d9f-ghi56      1/1     Running   gke-axion-tutorial-axion-pool-c-...

All three pods should show node names containing axion-pool. None should show default-pool.

Confirm the Nodes Are ARM

Take one of those node names and verify its architecture label:

kubectl get node NODE_NAME --show-labels | grep kubernetes.io/arch

Replace NODE_NAME with one of the node names from the previous command. You should see:

kubernetes.io/arch=arm64

That's the automatic label GKE applied when it provisioned the ARM hardware. Our nodeSelector matched on this label to pin the pods here.

Ask the Application Itself

This is the most satisfying verification step. Our Go server reports the architecture of the binary that's running. Let's ask it directly.

Use kubectl port-forward to create a secure tunnel from port 8080 on your local machine to port 8080 on the Deployment:

kubectl port-forward deployment/hello-axion 8080:8080

This command stays running in the foreground — open a second terminal window and run:

curl http://localhost:8080

You should see:

Hello from freeCodeCamp!
Architecture : arm64
OS           : linux
Pod hostname : hello-axion-7b8d9f-abc12

Architecture : arm64. That's our Go binary confirming that it was compiled for ARM64 and is executing on an ARM64 CPU. The single image tag we built does the right thing automatically.

The Bonus: See the Manifest List in Action

Want to see the multi-arch image indexing at work? Stop the port-forward, then run:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Replace PROJECT_ID with your actual Google Cloud project ID.

You'll see four entries in the manifest list. Two are real images — Platform: linux/amd64 and Platform: linux/arm64. The other two show Platform: unknown/unknown with an attestation-manifest annotation. These are build provenance records that Docker Buildx automatically attaches to every image — a supply chain security feature (SLSA attestation) that proves how and where the image was built.

You may notice that if you check the image digest recorded in a running pod:

kubectl get pod POD_NAME \
  -o jsonpath='{.status.containerStatuses[0].imageID}'

Replace POD_NAME with one of the pod names from earlier.

The digest returned matches the top-level manifest list digest, not the arm64-specific one. This is expected behaviour. Modern Kubernetes (using containerd) records the manifest list digest, not the resolved platform digest. The platform resolution already happened when the node pulled the correct image variant.

The definitive proof that the right binary is running is what you already have: the node labeled kubernetes.io/arch=arm64 and the application reporting Architecture: arm64.

Step 10: Cost Savings and Tradeoffs

The hands-on work is done. Let's talk about why any of this is worth the effort.

The Cost Math

At the time of writing, here's how ARM compares to equivalent x86 machines on Google Cloud (prices are approximate and change over time — check the official pricing page before making decisions):

That's a raw 25–30% reduction in compute cost per node. Factor in Google's published claim of up to 65% better price-performance for Axion on relevant workloads — meaning you may need fewer nodes to handle the same traffic — and the savings compound further.

Here's how that looks at scale, for a service running 20 nodes continuously for a year:

• 20 × n2-standard-4 × \(0.19 × 8,760 hours = \)33,288/year

• 20 × t2a-standard-4 × \(0.14 × 8,760 hours = \)24,528/year

That's roughly $8,760 saved annually on compute, before committed use discounts (which further widen the gap).

When ARM Is the Right Choice

ARM works best for:

• Stateless API servers and web applications — like the app we built. ARM excels at high-throughput, low-latency network workloads.

• Background workers and queue processors — long-running services that don't depend on x86-specific binaries.

• Microservices written in Go, Rust, or Python — these languages have excellent ARM64 support and are built cross-platform by default.

When to Proceed Carefully

• Native library dependencies — some older C libraries, proprietary SDKs, or compiled ML model-serving runtimes don't have ARM64 builds. Always audit your dependency tree before migrating.

• CI pipelines need ARM too — your automated tests should run on ARM, not just x86. An image that silently fails only on ARM is harder to debug than one that never claimed ARM support.

• Profile before optimizing — the cost savings are real, but measure your actual workload behavior on ARM before committing. Not every workload benefits equally.

Cleanup

When you're done, clean up to avoid ongoing charges:

# Remove the Kubernetes resources from the cluster
kubectl delete -f k8s/

# Delete the ARM node pool
gcloud container node-pools delete axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the cluster itself
gcloud container clusters delete axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the images from Artifact Registry (optional — storage costs are minimal)
gcloud artifacts docker images delete \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Conclusion

Let's recap what you built and why each part matters.

You started with a Go application, a Dockerfile, and a docker buildx build command that produced two images — one for x86, one for ARM64 — wrapped in a single Manifest List tag. Any server that pulls that tag gets the right binary automatically, without you maintaining separate pipelines or separate tags.

You provisioned a GKE cluster with two node pools running different CPU architectures, then used nodeSelector to make sure your ARM-optimized workload lands only on the ARM Axion nodes — not on x86 by accident. The result is a deployment that's both architecture-correct and cost-efficient.

The patterns you practiced here don't stop at this demo. The same Dockerfile technique works for any language with cross-compilation support. The same nodeSelector approach works for any workload you want to pin to ARM. As more teams migrate services to ARM over the coming years, having these skills will be a real asset.

Where to go from here:

• Add a GitHub Actions workflow that runs docker buildx build --platform linux/amd64,linux/arm64 on every push, automating this entire process in CI.

• Audit one of your existing stateless services for ARM compatibility and try migrating it.

• Explore Node Affinity as a softer alternative to nodeSelector for workloads that can run on either architecture but prefer ARM.

• Look into GKE Autopilot, which now supports ARM nodes and handles node pool management automatically.

Happy building.

Project File Structure

hello-axion/
├── app/
│   ├── main.go          — Go HTTP server
│   ├── go.mod           — Go module definition
│   └── Dockerfile       — Multi-stage Dockerfile
└── k8s/
    ├── deployment.yaml  — Deployment with nodeSelector and probes
    └── service.yaml     — LoadBalancer Service

All source files for this tutorial are available in the companion GitHub repository: https://github.com/Amiynarh/multi-arch-docker-gke-arm