You’ll move the UI and API to Google Kubernetes Engine without changing the public URL customers already use, gradually shifting traffic from AWS to GKE a few percent at a time.

By the end, you’ll have run the app locally, stood up AWS, built GKE in eight Terraform stages, tested the new environment, shifted production traffic to GKE, completed the cutover, and torn down what you no longer need.

The image below shows the middle of the migration, with both clouds live, customers still using one URL, and the database remaining on AWS.

Around that request path, three more systems do their jobs:

Table of Contents

• Before You Start

  • What You Need Installed

  • Skills You Need

  • What This Costs

  • How This Book Works

  • The Team

• Part 1. How to Run ClearPay and Put It on AWS

  • Chapter 1. Run ClearPay on Your Laptop

  • Chapter 2. Read the Code That Matters

  • Chapter 3. Prove a Retry Never Charges Twice

  • Chapter 4. Build the AWS Server With Terraform

  • Chapter 5. Deploy ClearPay to the AWS Server

  • Chapter 6. Record the AWS Baseline

• Part 2. How to Plan the Move

  • Chapter 7. Read the Migration Assignment

  • Chapter 8. Map What ClearPay Depends On

  • Chapter 9. Understand the GCP Design

• Part 3. How to Build Google Cloud With Terraform

  • Chapter 10. Set Upgcloudand Pick Your Project

  • Chapter 11. Build GCP in Eight Terraform Stages

• Part 4. How to Get ClearPay Onto GKE Staging

  • Chapter 12. Check the API Image

  • Chapter 13. Prepare the Cluster for ClearPay

  • Chapter 14. Build the CI/CD Pipeline

  • Chapter 15. Check Staging and Run the Promotion Gate

• Part 5. How to Make GKE Production-Ready

  • Chapter 16. Run the Security Review

  • Chapter 17. Watch ClearPay With Prometheus and Grafana (Optional)

  • Chapter 18. Load-Test and Break Staging on Purpose

• Part 6. How to Move Customers to GKE

  • Chapter 19. Deploy Production and the GKE Gateway

  • Chapter 20. Walk the Pre-Cutover Checklist

  • Chapter 21. Send 1 Percent of Traffic to GKE

  • Chapter 22. Raise Traffic to 100 Percent

  • Chapter 23. When the Numbers Go Bad

• Part 7. How to Finish the Migration

  • Chapter 24. Prove the Cutover With Numbers

  • Chapter 25. Run on GKE and Set a Date for AWS

  • Chapter 26. Retire the AWS Server

  • Chapter 27. Tear Down GCP and the Database

  • Chapter 28. Write the Migration Record

• What Comes After This Project

  • What Question This Project Leaves Open

• What You Should Have at the End

• How to Discuss This Project in an Interview

Before You Start

What You Need Installed

Install each tool when its chapter needs it. You don't need everything on day one.

On Windows, use Git Bash or WSL for every command in this book. The commands are written for a Bash shell.

Skills You Need

Prior Kubernetes, Terraform, GCP, or load-balancer experience isn't required. Don't buy a domain. On AWS, you open https:// plus the Elastic IP. The browser warns about a self-signed certificate. Click through it and you get the same ClearPay login as on your laptop. On GCP you add one /etc/hosts line so https://api.clearpay.test points at the load balancer, unless you turn on Cloud DNS.

What This Costs

AWS costs about $1.25 per day. GCP costs about $1.60 to $3 per day once it's up. A weekend run of the whole book costs a few dollars.

Neither environment should run for a month. If you stop for more than a couple of days, tear both down:

./scripts/teardown-gcp.sh
cd infra/aws/terraform && terraform destroy

Two files hold the detailed numbers: infra/aws/COST.md and infra/gcp/COST.md. The book sends you to those two files and no other documents.

How This Book Works

Clone the repository and run every command from its root folder unless a step says cd somewhere else:

git clone https://github.com/Osomudeya/migrate-api-aws-gke-devops.git

cd migrate-api-aws-gke-devops

The repository already holds the app, the Terraform, the Kubernetes files, the pipeline, and the scripts.

You won't write new code. Your work is to run each step, read what the system prints, and decide whether to continue.

This book reads like a story where you join a small payments company in week one, and every chapter is a task someone on the team hands you.

Each chapter opens with a time stamp and a scene, then follows the same loop:

Most Important Rule

Don't run a command you don't understand. If the book hasn't told you why yet, keep reading, because the "why" is the point.

This book has two kinds of steps.

The first simply asks whether the step worked, such as whether the file copied or the container started. If it did, continue.

The second asks you to think about what happened. Once the command finishes, read the output and understand what it means before moving to the next chapter.

The Chapters

Read the chapters in order and skip none. Chapter 1 doesn't ask you to install Kubernetes or open GCP. Those come later.

If Something Doesn't Work

Stay in that chapter. Use its "If it fails" steps. Don't skip ahead. A problem you carry forward gets harder to find, because the next chapter adds another cloud on top of it.

The job isn't to memorize Kubernetes or Terraform. It's to learn how to work through a real migration: understand what's running, make decisions based on what you find, verify the result, and know what to do when something breaks.

How Long This Takes

This might take more than one day. If you'll be away for more than a day or two, destroy the AWS and GCP resources so you're not billed for them while it's idle. The commands are in "What this costs" above.

The Team

This lab starts as Zee, a junior DevOps engineer on the ClearPay team, week one. You play Zee.

• Maya is the engineering manager. She asks for evidence.

• Daniel is a senior engineer. He pairs with you on servers, Kubernetes, and capacity.

• Amara maintains the payments API. She cares about one thing: no customer is charged twice.

• Chris handles customer support. He hears first when transfers fail.

Your job is to move the payments API from one AWS EC2 server to GKE without a customer-visible outage, while PostgreSQL stays on AWS.

Part 1. How to Run ClearPay and Put It on AWS

A migration needs a starting point you trust. In this part, you run ClearPay on your laptop, deploy it to AWS the way the company runs it today, and write down how fast it is. Every later chapter compares GKE against those numbers.

Chapter 1. Run ClearPay on Your Laptop

Week one, Monday, 9:05 a.m.

Your laptop is new. The Wi-Fi password is on a sticky note. The first message in your inbox is from Maya, the engineering manager, and it's short.

Maya: "Get ClearPay running on your laptop first. Log in, send $20, and then we'll talk about moving the API to GKE."

The code is in the repository: api/, ui/, and db/. Nothing is running yet.

Why This Matters

Later chapters need a picture of a good login and a good $20 send. When something fails on AWS or GKE, you compare it with this.

What You're About to Run

ClearPay has three parts, and each part has one job.

• The UI is what a customer sees and taps: the login page, the balance, and the Send button. UI stands for user interface.

• The API is the part that does the work. It has no screen. The UI sends it a message, such as "send $20 to Bob". The API checks the message, does the work, and sends back a reply, such as "done". API stands for application programming interface. Think of it as a door with rules. The UI asks in a fixed way, and the API answers in a fixed way.

• The database is where ClearPay writes everything down: every account, balance, and payment. PostgreSQL is the database program ClearPay uses.

• A server is a computer that stays on all day and waits for requests. The API runs on one.

A café works the same way. The menu is the UI. The waiter is the API. The kitchen and its records are the database. A customer never walks into the kitchen. The customer tells the waiter what they want.

This chapter runs all three on your laptop. Later chapters move the UI and the API to Google Cloud. The database stays on AWS.

Steps

Start Docker Desktop. Don't install Node or PostgreSQL for this chapter. They run inside the containers.

From the folder that holds docker-compose.yml:

docker compose up --build

The first build takes a couple of minutes. This terminal keeps printing logs and doesn't give you a prompt back until you press Ctrl+C. Leave it running.

Wait until all three of these lines appear. The names on the left may start with your folder name:

Container <folder>-postgres-1 Healthy
api-1       | {"message":"api_started", ...}
ui-1        |   ➜  Local:   http://localhost:5173/

Healthy means Postgres is up. api_started means the API is up. The localhost:5173 line means the UI is up.

Open a second terminal in the same folder. Create two demo accounts:

docker compose exec api node db/seed.js

This runs the seed script directly. The API image has no npm, on purpose, because the security scan in Chapter 14 flags it.

Expected output:

seeded: alice@clearpay.dev (password: password123)
seeded: bob@clearpay.dev (password: password123)

If you see skip (already exists): alice@clearpay.dev, the accounts exist from an earlier run. That's fine.

This is what's now running:

Check

In the second terminal:

curl localhost:4000/health

Here's the expected output. Your instance value will be a container ID:

{"status":"ok","served_by":{"platform":"aws","version":"1.0.0","instance":"14b8d5d767ea","commit":"local","rollback_active":false}}

"status":"ok" is the check. served_by names the platform that answered the request. It says aws on your laptop because docker-compose.yml sets PLATFORM: aws. Later, GKE answers with gke, and that one field tells you which cloud handled each request.

Now open http://localhost:5173 and sign in. Seed already created Alice, so don't register her:

• Email: alice@clearpay.dev

• Password: password123

Send $20 to bob@clearpay.dev. The transfer appears under Recent activity as Sent with a short ID, for example Sent · a1b2c3d4.

Look at two places on this dashboard. You'll watch them for the rest of the book:

1. Each line under Recent activity ends with served by aws/1.0.0. That's the platform and version of the API process that wrote the transfer. It's stored with the transfer in the database.

2. The black bar at the top says All systems operational when /health/ready succeeds. That means the API process is up and can reach Postgres. It says nothing about logins. A wrong password shows a red error on the form, while the bar keeps saying All systems operational.

After Chapter 5, you open the same dashboard on the AWS address. Same login, same two places.

If it Fails:

• failed to connect to the docker API or docker.sock: Docker Desktop isn't running. Start it, wait until it is ready, and run docker compose up --build again.

• email already registered on the Register page: Alice already exists. Go back to Sign in.

• invalid credentials: the password is the full string password123. If that still fails, run the seed command again. A skip (already exists) line means the account is there.

• docker compose up exits right away: run docker compose logs api in the second terminal. On a slow first boot the API can start before Postgres accepts connections. Run docker compose up again.

• Port 4000 or 5173 already in use: another program on your machine holds that port. Stop it, or change the port mapping in docker-compose.yml.

Decision

If the transfer went through, the application works. Continue.

If it didn't, fix it here, on the laptop. AWS and GKE are harder to debug. On this machine, the answer is one docker compose logs command away.

Done When:

• docker compose up runs without errors

• The seed reports two accounts, created or skipped

• You sent $20 in the UI and saw it under Recent activity

Chapter 2. Read the Code That Matters

Monday, 11:30 a.m.

The $20 went through. You're still looking at the transfer line when a chair rolls up next to yours. Daniel, the senior engineer at the next desk, has his coffee and no hurry.

Daniel looks over at your screen. "You don't need to memorize every line. Know which file does which job, so you can open it fast during the migration."

Why This Matters

Every later chapter touches one of these four files. Five minutes now saves you a search under pressure later.

Steps

Open these files in this order:

1. api/src/index.js starts the API and connects every route: health checks, /api/info, login, and transactions.

2. api/src/servedBy.js is 17 lines long. It reads PLATFORM, APP_VERSION, and ROLLBACK_ACTIVE from environment variables. The dashboard uses it to print served by aws/1.0.0.

3. api/src/routes/transactions.js handles sending money. Find the idempotency_key check near the top. It runs before anything else.

4. api/src/db.js connects the API to Postgres. Find DB_POOL_SIZE || '10'.

Daniel stops you on the last file.

"A connection is an open line to the database. One API process can hold up to ten of those lines at the same time. Remember that number."

What You Now Know

index.js                 → starts the API and connects the routes
servedBy.js              → says which cloud answered (aws or gke)
routes/transactions.js   → sends money, checks the idempotency key first
db.js                    → talks to Postgres, up to 10 connections per process

servedBy.js fills the dashboard line served by aws/1.0.0. That's which cloud answered, not who logged in.

Today there's one API process, so ten connections is the whole pool. On GKE, several copies of the API run at once. Kubernetes calls each copy a pod. Each pod gets its own pool of ten, and every pod shares the same database. Chapter 22 comes back to that math.

Done When:

• You can say, without looking, which file sets the connection limit and which file names the platform

Chapter 3. Prove a Retry Never Charges Twice

Monday, 2:10 p.m.

After lunch, you get a message from someone you haven't met yet. Amara maintains the payments API. Maya told her you have ClearPay running on your laptop, so she skips the greeting.

Amara: "Send $20 twice with the same key. Alice's balance should drop by $20, not $40."

Why This Matters

Alice sends $20 to Bob. She taps Send, and the screen freezes. She doesn't know if the payment worked, so she taps Send again.

Without protection, the API sees two requests and takes $20 two times. Alice loses $40 for one payment.

The idempotency_key stops this. "Idempotent" means safe to repeat: doing it twice gives the same result as doing it once.

The key is a label the app puts on each payment, for example round-a.

1. The first request arrives with the label round-a. The API takes $20 and saves the label.

2. The second request arrives with the same label. The API sees round-a again. It knows this is the same payment, so it takes no money. It sends back the first result.

Same label means same payment.

This matters more during the migration. Sometimes the first tap goes to AWS and the retry goes to GKE. Those are two different servers, but they use one database. GKE looks in that database, finds round-a, and doesn't charge again. One shared database keeps the customer safe on both clouds.

Use curl for this chapter, not the Send money button. The UI gives every click a new key, so every click is a new transfer.

For a deeper walk-through of idempotency, see PayFlow: A Production-Grade Fintech System for Learning Real DevOps.

Steps

Stay in the second terminal, in the folder with docker-compose.yml. Run one command at a time and read the output before the next one.

Log in as Alice. This stores her login token in a shell variable called TOKEN. A token is proof that Alice signed in. Later commands send it with $TOKEN.

TOKEN=$(curl -s -X POST localhost:4000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"alice@clearpay.dev","password":"password123"}' \
  | python3 -c "import sys, json; print(json.load(sys.stdin)['token'])")

This prints nothing when it works. Run echo $TOKEN to confirm. You should see a long string.

Then read Alice's account. Write down the balance_cents number.

curl -s localhost:4000/api/accounts/me -H "Authorization: Bearer $TOKEN"

This is the expected shape. Your numbers and id will differ:

{"account":{"id":"...","balance_cents":"94000","currency":"USD"},"served_by":{"platform":"aws","version":"1.0.0","instance":"...","commit":"local","rollback_active":false}}

94000 means $940.00. If you haven't sent anything yet, you'll see 100000 ($1,000.00). If the command says missing_token or invalid_token, run step 1 again.

Now pick a label you haven't used yet:

LABEL=round-a

Send $20 to Bob. 2000 is $20.00 in cents.

curl -s -X POST localhost:4000/api/transactions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"to_email\":\"bob@clearpay.dev\",\"amount_cents\":2000,\"idempotency_key\":\"$LABEL\"}"

Look for "replayed" in the reply:

• "replayed":false means this send took $20. Continue.

• "replayed":true means this label was used before, probably in an earlier run. No money moved. Set LABEL=round-b and send again until you see "replayed":false.

Send the exact same request again. Don't change LABEL.

curl -s -X POST localhost:4000/api/transactions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"to_email\":\"bob@clearpay.dev\",\"amount_cents\":2000,\"idempotency_key\":\"$LABEL\"}"

Now you should see "replayed":true, and the transaction id matches the first send.

Read the account again:

curl -s localhost:4000/api/accounts/me -H "Authorization: Bearer $TOKEN"

Check

balance_cents is 2000 lower than the number you wrote down, not 4000 lower. If you started at 94000, you now see 92000.

If it Fails:

• echo $TOKEN is empty: run the login command again.

• The second send still shows "replayed":false: LABEL changed between the two sends. Run echo $LABEL, then send again.

• Both sends showed "replayed":true: the balance won't change. Pick a new LABEL and repeat steps 3 and 4.

Done When:

• One send returned "replayed":false and the next returned "replayed":true for the same label

• balance_cents fell by 2000, not 4000

Chapter 4. Build the AWS Server With Terraform

Tuesday, 9:15 a.m.

Maya stops at your desk on her way to a meeting. Yesterday you told her that ClearPay runs on your laptop and that a retried payment charges only once.

Maya: "Good. Now, localhost isn't a baseline. Get ClearPay onto AWS and prove a customer can complete a transaction there. Then we can talk about leaving AWS. Daniel will work with you on this one."

Daniel rolls his chair over from the next desk.

Daniel: "One rule before you start. Don't skip terraform plan."

Why This Matters

This book starts from the setup many small companies run: one Linux server with no Docker on it.

systemd, the Linux service manager, starts the Node API when the server boots. nginx sits in front of it and handles HTTPS on port 443. PostgreSQL runs on Amazon RDS, the managed database service.

There's no AWS load balancer. A load balancer spreads traffic across several servers, and this setup has one. Adding one would add an hourly bill for no gain. The numbers are in infra/aws/COST.md.

The file api/Dockerfile exists, but only your laptop uses it. The AWS server never runs Docker.

Daniel: "The EC2 server won't have Docker. systemd starts Node on boot, next to nginx. That's the server you're inheriting."

The diagram above shows how a customer request reaches ClearPay on AWS. The Elastic IP provides a fixed public address for the EC2 server. Nginx handles HTTPS traffic on port 443, serves the frontend files, and forwards API requests to the Node.js process managed by systemd on port 4000. The API reads and writes payment data in PostgreSQL on Amazon RDS, which has no public endpoint at this stage.

Two AWS terms appear in this chapter:

• A VPC is the private network your servers sit in. AWS gives every account a default one. This lab uses it instead of building its own.

• A security group is AWS's firewall. It lists which ports may reach a server. This lab allows SSH from your IP only, HTTPS from anywhere, and Postgres only from the API server.

Steps

First confirm your laptop is ready. Each of these must work:

docker --version          # still needed later
aws sts get-caller-identity
terraform version         # 1.5 or newer
ssh-keygen -t ed25519 -f ~/.ssh/clearpay-deployer -N ""

aws sts get-caller-identity prints your AWS account. If it errors, run aws configure with an access key from an IAM user that can create EC2, RDS, VPCs, and CloudWatch alarms. An admin user on a throwaway account also works.

From the repository root:

cd infra/aws/terraform
cp terraform.tfvars.example terraform.tfvars

Open terraform.tfvars and change two values:

• ssh_allowed_cidr: your public IP address followed by /32. Find your IP with curl ifconfig.me.

• ssh_public_key_path: the path to your key's .pub file.

Then:

terraform init
terraform plan
terraform apply

Read the plan before you type yes. This first run should only create things: EC2, RDS, and security groups. If the plan says it will destroy anything, stop and don't apply.

This is what Terraform builds. The server is empty until Chapter 5:

The diagram shows the AWS infrastructure Terraform creates: an EC2 Ubuntu server with an Elastic IP, security groups controlling SSH and HTTPS access, an RDS PostgreSQL database accessible only from the API server, and a CloudWatch status check. Both resources sit inside the default VPC. The EC2 server is still empty. Chapter 5 deploys ClearPay to it.

A note on cost: the Elastic IP costs about $0.12 per day even while the server runs, because AWS bills every public IPv4 address (AWS public IPv4 charge).

Check

terraform output instance_public_ip

This prints a public IPv4 address. That is how you reach the server. You do not need a domain, so Route 53 stays off (manage_dns = false).

If it Fails:

• Error: InvalidKeyPair.Duplicate: a key pair named clearpay-deployer already exists in this AWS region from an earlier attempt. Delete it in the AWS console, or rename it in infra/aws/terraform/ec2.tf.

• Error: UnauthorizedOperation: your AWS user lacks permission for one resource type. The error names the operation. Add that one permission instead of reaching for full admin.

Decision

The infrastructure exists. A server with nothing running on it isn't done yet. Continue to Chapter 5.

Done When:

• terraform apply finished with no errors

• terraform output instance_public_ip prints a real IP

Chapter 5. Deploy ClearPay to the AWS Server

Tuesday, 1:40 p.m.

Terraform apply finished before lunch. AWS now shows a server and a database, but nothing is serving customers yet.

Terraform built an empty server and a database. Nothing serves customers yet.

You open a terminal to start the deploy. Daniel pulls his chair over and sits down next to you.

Daniel: "Terraform finishing only tells us the server exists. Let's get the app on it and send a real request."

Why This Matters

An empty server isn't a migration source. Chapter 6 asks whether a real customer request works on AWS. That question has no answer until the login page and the API are up.

Steps

Start from the repository root. If you're still in infra/aws/terraform, run cd ../../.. first.

Start by installing Node, nginx, and certbot on the server:

cd infra/aws/deploy
INSTANCE_IP=$(terraform -chdir=../terraform output -raw instance_public_ip)
echo "$INSTANCE_IP"
# Write that IP down. A new terminal will not have INSTANCE_IP.

ssh -o StrictHostKeyChecking=accept-new \
  -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP 'bash -s' < bootstrap.sh

You should see == ClearPay AWS bootstrap ==, then a few minutes of apt-get output. bootstrap.sh is short. Read it.

The step worked when the output ends with these lines:

== Bootstrap complete ==
Node: v20.20.2
Self-signed TLS is in /etc/letsencrypt/live/clearpay, use curl -k.

Your Node version may differ. Lines that start with debconf: unable to initialize frontend are harmless. They appear because the SSH session has no interactive screen. The rows of + and * come from OpenSSL creating the self-signed certificate.

If you see no output at all, SSH didn't connect. Your home IP may have changed since Chapter 4, so the security group still allows the old one. Check your IP with curl -s ifconfig.me; echo, put it plus /32 in terraform.tfvars as ssh_allowed_cidr, run terraform apply in infra/aws/terraform, and retry.

Next, write the .env file the API needs and deploy the app. The database URL comes from Terraform. The JWT secret signs login tokens and is generated fresh:

DATABASE_URL=$(terraform -chdir=../terraform output -raw database_url)

cat > /tmp/clearpay.env <<EOF
DATABASE_URL=$DATABASE_URL
JWT_SECRET=$(openssl rand -hex 32)
PLATFORM=aws
APP_VERSION=1.0.0
GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo local)
ROLLBACK_ACTIVE=false
PORT=4000
EOF

./deploy.sh "$INSTANCE_IP" ~/.ssh/clearpay-deployer /tmp/clearpay.env
rm /tmp/clearpay.env

deploy.sh builds the React UI and copies it to /opt/clearpay-ui. It copies the API to /opt/clearpay-api, installs its packages, writes .env, installs the systemd service and the nginx config from infra/aws/deploy/, and restarts both.

Now load the database schema and seed Alice and Bob. These commands run on the server, not your laptop. RDS has no public endpoint yet (publicly_accessible = false in infra/aws/terraform/rds.tf), so your laptop can't reach it.

Your laptop can't reach the database directly. It uses the EC2 server as a middleman:

The diagram above shows why your laptop can't connect directly to the private RDS database. SSH access to the EC2 server is allowed, and the server is allowed to connect to RDS within the AWS network. Direct access from your laptop to the database is blocked. You use the EC2 server as the entry point for database access.

Each command below starts with ssh ... ubuntu@$INSTANCE_IP "...". That means: log in to the EC2 server and run the part in quotes there. So psql and the seed run on EC2, and EC2 talks to RDS for you. You see the result later in the browser: you log in as Alice, and EC2 reads her account from RDS.

scp -i ~/.ssh/clearpay-deployer ../../../db/schema.sql \
  ubuntu@$INSTANCE_IP:/tmp/schema.sql

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo apt-get install -y postgresql-client -qq && psql '$DATABASE_URL' -f /tmp/schema.sql"

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "cd /opt/clearpay-api && sudo -u clearpay npm run seed"

The seed is the same script from Chapter 1. This time it writes to RDS.

Check

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP 'sudo systemctl is-active clearpay-api'

Expected output: active.

Then from your laptop:

curl -k https://$INSTANCE_IP/health

Look for "status":"ok" and "platform":"aws". The -k flag tells curl to accept the self-signed certificate that bootstrap.sh created.

Open https://<your instance IP> in a browser. Click through the "Not Secure" warning. That warning comes from the self-signed certificate. You should see the same ClearPay login as on your laptop. Sign in as Alice with password123. The dashboard shows her balance and served by aws/1.0.0.

Daniel: "The health check only tells us the app is running. Before we write down a baseline, I want to see a transfer go through on this IP."

Send $20 to Bob in the browser.

If it Fails:

• systemctl is-active returns failed: SSH in and run sudo journalctl -u clearpay-api -n 50. The usual cause is a DATABASE_URL that doesn't match RDS. Compare it with terraform output database_url, character by character.

• seed_failed with SELF_SIGNED_CERT_IN_CHAIN: psql already worked, so the database is fine. Newer versions of the Node pg library treat sslmode=require as full certificate checking. Copy the current api/src/pgConfig.js, api/src/db.js, and api/db/seed.js onto the server, restart clearpay-api, and run only the seed command again. Don't run schema.sql again.

• nginx returns 502 Bad Gateway: nginx is running but the API behind it is not. Check the systemd status above first.

• The browser shows Cannot GET /: nginx still has an old API-only config. Run ./deploy.sh "$INSTANCE_IP" ~/.ssh/clearpay-deployer with no third argument. That keeps the .env already on the server. Don't seed again.

• invalid credentials for Alice: the browser probably filled in a saved password. Clear the box, type password123, and try again.

Done When:

• systemctl is-active clearpay-api prints active

• curl -k https://$INSTANCE_IP/health returns "status":"ok"

• You sent $20 in the browser on the AWS address

Chapter 6. Record the AWS Baseline

Wednesday, 10:00 a.m.

You message Maya: "health is ok, and Alice signed in on the Elastic IP." Her reply comes back in under a minute.

Maya: "Good. Now run the five checks against that IP. Send me the terminal output, not a screenshot."

A second message follows: "Then load-test it and send me the p95. I can't compare GKE against 'it works'."

Why This Matters

This chapter gives you two things. You'll use both for the rest of the book.

1. A pass from the promotion gate. The promotion gate is a short script. It tests five things every payment app must do: the app is running, it can reach the database, a user can log in, a repeated payment charges only once, and the balance drops by the right amount. If all five pass, the app works.

In Chapter 15 you run the same script on GKE. If GKE passes the same test AWS passed, GKE does the job, too.

2. A scorecard. The load test sends many requests at the same time, like many customers using the app at once. Then it writes down how fast the app answered and how many requests failed. That's how fast AWS is today. Later you run the same test on GKE and compare the two.

Think of timing a runner before they change shoes. Without the first time, the second time means nothing. This chapter records the first time.

The scripts in this chapter use Node on your laptop to read JSON. Install Node.js 20 or newer now if you haven't already.

Steps

Start from the repository root. If you're still in infra/aws/deploy, run cd ../../.. first.

Run the promotion gate against AWS:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)

./scripts/promotion-gate.sh https://$INSTANCE_IP \
  alice@clearpay.dev password123 bob@clearpay.dev

Use https://. With http://, nginx redirects and the script looks like it hung.

The gate checks five things, in order:

1. The process is up (/health).

2. It can reach the database (/health/ready).

3. Alice can log in.

4. A replayed payment key returns replayed=true.

5. Alice's balance drops once.

Expected output:

PASS: health endpoint returns ok
PASS: readiness endpoint confirms DB connectivity
PASS: login succeeds and returns a token
PASS: replaying an idempotency key returns replayed=true
PASS: balance debited exactly once (100 cents), not twice

5 passed, 0 failed

GATE PASSED: safe to promote to production.

The last line talks about production. There's no production to promote to yet. It's the same script you run on GKE later, where that line means what it says.

Now load-test AWS. The script logs in and hits /api/accounts/me and /api/transactions, which both query Postgres. Don't load-test /health. The API answers /health by itself, straight away. It never asks the database anything. A real customer request has to ask the database and wait for the answer, and that wait is most of the time:

/health                  laptop → API → reply                    (fast, no database)
/api/accounts/me         laptop → API → database → API → reply   (the real work)

Load-testing /health times only the fast half. Think of asking a shop worker "are you open?" versus "do you have this shoe in size 42?" Only the second question makes them walk to the storeroom.

./scripts/load-test.sh https://$INSTANCE_IP \
  alice@clearpay.dev password123 10 30

The last two numbers are concurrency (10 requests at once) and duration (30 seconds).

Check

The script prints a scorecard. This is the one this lab printed:

== DB load test: https://184.194.120.157  concurrency=10  duration=30s ==
Endpoints: GET /api/accounts/me  GET /api/transactions


ok=372  failed=0

Migration Scorecard, this run

p50 latency:          673 ms
p95 latency:          745 ms
p99 latency:          1326 ms
Throughput:           12.4 req/s
4xx rate:             0.0 %
5xx rate:             0.0 %
Transaction success:  100.0 %

Your scorecard will differ. Your IP address, request counts, and milliseconds depend on where you are. The load test runs on your laptop, so every request travels from your home to AWS in us-east-1 and back.

A reader close to that region sees much lower numbers than a reader on another continent. Neither is wrong. What matters is that you compare GKE against your own scorecard, measured from the same place.

The run passes when you see failed=0, 5xx rate at 0.0 %, and Transaction success at 100.0 %.

• p95 latency means 95 percent of requests finished faster than this number. It shows what slow customers feel, which an average hides.

• 4xx means the client sent a bad request, such as a missing token.

• 5xx means the server failed.

You copy the whole scorecard into a notes file as printed, and send the same block to Maya. Her answer comes back fast: "Don't round it to look tidier. I will compare GKE to this page."

Set Your Limits Now

Write these two lines under the scorecard, with numbers:

Outage:   5xx rate above 1 % for more than 2 minutes in a row
Failure:  p95 more than 3 times your AWS p95 for more than 5 minutes in a row

The failure limit uses your own AWS p95, because every reader's baseline is different. With this lab's 745 ms, the limit is about 2,235 ms. These are example limits. Pick your own, but pick them now, before GKE exists. A limit you set after you see the GKE numbers is a limit you bent to fit them.

Start the cost ledger with one line:

Cost ledger, entry 1:  AWS about $1.25/day (infra/aws/COST.md)

If it Fails:

Stay here until all five checks pass. A problem you carry into the migration comes back later with two clouds involved, and is much harder to find.

• A check fails on readiness: the API can't reach RDS. Go back to Chapter 5's "If it fails" list.

• node: command not found: install Node.js on your laptop.

Decision

The gate passed. This EC2 server is the baseline. Every GKE number in this book gets compared against this scorecard.

A minute later Maya adds one more line: "Good. Those are my numbers now."

Done When:

• The gate printed 5 passed, 0 failed against the AWS IP

• The load test printed failed=0, and the scorecard is in your notes

• Your outage and failure limits are written down with numbers

Part 2. How to Plan the Move

Tell Maya the gate passed and send her the scorecard.

She writes back: "The next job is GKE. I need the API running there, and I need customers to keep sending money the whole time you move it. Leave Postgres on AWS. The database isn't in this assignment."

No cloud gets touched in this part. You read, map, and decide.

Chapter 7. Read the Migration Assignment

Wednesday, 4:00 p.m.

Maya forwards the assignment with one line on top: "Read all of it before you open Terraform."

The Assignment: Migrate the ClearPay API from AWS EC2 to GKE.

ClearPay runs on one EC2 server. It works, but it has no autoscaling, no staged rollout, and a deploy process that lives in one person's head. You ran that process by hand in Chapter 5, so you know what it lacks.

The business wants the API on Kubernetes: a staging environment, automated deploys, and room to grow. Again, the database isn't part of this assignment.

Current Architecture

ClearPay lives in one AWS account, in the default VPC from Chapter 4: one EC2 server and one RDS database.

What Can't Change

• The database. Moving Postgres is a separate, later project.

• The API contract. The same paths return the same JSON: /health, /api/auth/login, /api/transactions. A browser or script that works on AWS must work on GKE with no changes.

• Customers see no lasting outage at any point.

Acceptance Criteria

Done means this, in order:

1. GKE serves all production traffic, and it stays that way. Errors and latency stay inside the limits you wrote in Chapter 6.

2. While AWS still exists, rollback to AWS still works.

3. Then you retire AWS, after GKE has proven itself.

What moves and what stays:

The diagram shows the migration from AWS to GCP. The ClearPay UI and API move from the EC2 server to Google Kubernetes Engine (GKE), where staging and production run. PostgreSQL remains on Amazon RDS in AWS, so both environments use the same database during the migration.

The whole book follows one path:

AWS only  →  AWS and GKE together  →  GKE only, then EC2 gone

The middle step is where the work happens. Both clouds run the same app, both talk to the same database, and traffic moves between them in small steps.

How Traffic Will Move

Today, customers type the Elastic IP and every request goes to AWS.

1. You put a Google load balancer in front, and customers switch to its address once. They keep using that address for the rest of the migration.

2. The load balancer then sends some requests to AWS and the rest to GKE. It starts with 100% AWS and 0% GKE.

3. You raise the GKE share by hand with scripts/shift-traffic.sh, a few steps at a time, checking numbers after each step.

4. If something goes wrong on GKE, you send 100% of the traffic back to AWS with shift-traffic.sh 100 0. The rollback takes seconds.

The diagram above shows how customers reach ClearPay through one public address while the UI and API run on both AWS and GKE during migration. Both environments connect to the same PostgreSQL database on Amazon RDS in AWS. Keeping the database in one place avoids migrating payment data alongside the application.

Why not use DNS to split traffic? DNS is the lookup that turns a name into an IP address. Laptops and internet providers cache the answer.

After you change a DNS record, some customers keep going to the old place for minutes or hours. A rollback that takes an hour isn't a rollback. The load balancer changes the split in seconds.

Rollback Requirement

At every stage of the shift, going back to 100 percent AWS must be fast and safe. Not a redeploy. Not a DNS change that takes an hour to spread. Chapter 20 tests exactly that, before the first shift.

What "Zero Downtime" Means Here

It doesn't mean zero failed requests, ever. No real system can promise that.

It means no customer sees a lasting outage. A few failed requests are allowed if they stay under the limits you wrote in Chapter 6.

Why the API Moves First

If the API and the database move together and something breaks, the fault could be in either one, or in how they interact. Moving only the API keeps the question small: can this API run on GKE without hurting customers?

That's an answerable question. "Can you move everything at once safely?" is much harder to answer, and much worse to debug live.

Why the Database Stays on AWS

Chapter 13 shows what connecting across clouds involves. For now: PostgreSQL doesn't move in this project.

A future, separate project handles that, most likely with Google's Database Migration Service, continuous replication, and its own cutover. This project ends with the API on GKE and the database still on AWS. That's the plan, not an unfinished loose end.

Why Both Clouds Must Run at the Same Time

The rollback plan depends on it. You send a little traffic to GKE and keep AWS ready. If GKE fails, AWS takes everything back. That only works while AWS still exists.

Set a Migration Budget Before Touching GCP

Do this now, before Chapter 9. Write four numbers in your notes. They decide how much this move may cost you.

Plan your spending the way you plan money for a trip:

1. What AWS costs today. About $1.25 a day, from infra/aws/COST.md.

2. What GCP will cost per day. You don't know yet. Write "TBD" (to be decided). Chapter 9 works it out. infra/gcp/COST.md puts it at about $1.60 to $5 a day.

3. What the move costs while both clouds run. During the move you pay AWS and GCP at the same time. That is $1.25 plus the GCP daily cost, for every day of the move.

4. Your budget. The most you will spend on the whole move. Pick it now, before you spend anything.

This lab's numbers:

Current AWS cost:   $1.25/day
Expected GCP cost:  about $3/day (Chapter 9 checks this)
Dual-run cost:      $1.25 + $3 = $4.25/day while both run
Migration budget:   5 days × $4.25 = about $21. Round up to $25.

Keep a running total as you go. Chapters 19 to 25 add the real dual-run days. Chapter 28 asks one question: did you stay under your budget? If the move took longer, for example because of a rollback, the total shows exactly what that cost.

Migration phases

Done When:

• You can explain to Maya, in two sentences, what moves, what stays, and how you undo a bad step

Chapter 8. Map What ClearPay Depends On

Thursday, 9:00 a.m.

Before anyone stands up GKE, the team needs a written map of what ClearPay depends on. This system has run by hand since Tuesday. Nothing on paper says what must keep working during the move.

Maya: "Before we stand up another cloud, write down what this system depends on."

Why This Matters

A migration without a map moves the wrong thing, or misses something that breaks the day you switch. Answer each question below from the repository, not from memory.

1. Is the app ready for containers?

Yes. api/Dockerfile has worked since Chapter 1. What changes is its role. On your laptop it was a convenience. On GKE it becomes the thing production runs.

2. Where do the secrets live?

Open infra/aws/deploy/systemd/clearpay-api.service. It loads EnvironmentFile=/opt/clearpay-api/.env. That file isn't in Git. It sits on the EC2 server, where deploy.sh copied it in Chapter 5.

Every secret the API needs is in one plain text file on one server. Note that. Don't open it to print secrets. GKE will read the same values from Google Secret Manager instead.

3. What outside systems does it call?

One: PostgreSQL. The connection stays inside the AWS default VPC, and RDS has no public endpoint. There's no third-party API. Confirm that by reading api/src/.

4. Does the server remember users between requests?

No. Open api/src/index.js and api/src/routes/transactions.js. Each request carries a JWT (a signed login token), and all account data lives in Postgres. The server holds nothing in memory.

That's why AWS and GKE can split requests later. A user can log in on AWS and send money on GKE, because both clouds check the same token secret and read the same database.

5. Can a retry charge twice?

No. Chapter 3 proved the idempotency key works. It must keep working when a first try lands on AWS and the retry lands on GKE.

The Map

Write these five answers in your notes:

Container ready:   yes, api/Dockerfile
Secrets:           one .env file on the EC2 server
Outside systems:   PostgreSQL only
Server state:      none, JWT + database
Retry safety:      idempotency key, proven in Chapter 3

What Actually Needs to Move

Only the UI and the API: off the EC2 server, onto GKE. Postgres stays on AWS.

Browsers and scripts still hit the same URLs and get the same JSON back. A retry still must not take money twice. Those rules don't change.

Done when

• All five answers are written down, each pointing at a file

Chapter 9. Understand the GCP Design

Thursday, 2:30 p.m.

The Terraform for Google Cloud is already written. Somebody made two big decisions in it before you joined. Maya wants to know whether you can defend them. She writes in the team channel.

Maya: "Open infra/gcp/COST.md and infra/gcp/terraform/03-gke. Then tell me why this repo uses GKE Standard, not Autopilot. I need a reason I can defend."

Daniel, who sits at the next desk, has infra/gcp/terraform/03-gke/main.tf open. He points at a block named primary. It's the node pool: the group of virtual machines GKE runs the app on. This one holds one to three e2-medium machines.

Daniel: "If we keep these machines, how many copies of the API can run on them at once?"

This chapter doesn't run Terraform. The files are already written. You read the decisions in them.

Standard or Autopilot

GKE is Google's managed Kubernetes. It runs in two modes:

• Standard: you choose the virtual machines, grouped in a node pool. Google bills those machines whether the app is busy or idle.

• Autopilot: Google hides the machines. You say how much CPU and memory each container needs, and Google bills that.

This repo uses Standard. The reason isn't cost. It keeps node pools, machine sizes, and autoscaling visible, and those are what this book teaches. Autopilot would hide them.

Work out both costs yourself. Standard bills the machines. One e2-medium running all day is about $1.44 in infra/gcp/COST.md, plus Cloud NAT.

Autopilot bills the CPU and memory each pod reserves. k8s/base/deployment.yaml reserves 250m CPU (a quarter of one CPU core) and 256Mi of memory per API pod. Using the Autopilot rates in infra/gcp/COST.md, three API pods cost roughly:

0.25 vCPU × 3 × $0.0445/hour × 24  ≈  $0.80 / day
0.25 GiB  × 3 × $0.0049/hour × 24  ≈  $0.09 / day

That's about $0.89 a day for three API pods, before the UI pods. Autopilot can cost less than one Standard machine. The lab still uses Standard so you can see the node pool.

Tell Maya both numbers and the reason.

How Many API Processes

AWS runs one API process. On GKE, a HorizontalPodAutoscaler (HPA) adds or removes API pods based on CPU use. Each pod is one API process with its own pool of ten database connections.

This lab's cluster is three small e2-medium machines. The production HPA in k8s/overlays/production/patch-hpa.yaml runs between 1 and 3 API pods, because three machines of that size can't fit more alongside GKE's own system pods.

Answer Daniel: "One to three API pods on GKE, plus one on AWS while both run. Each can open ten connections."

A company with bigger machines would set a higher ceiling, such as 3 to 8 pods. Chapter 22 shows why that ceiling needs checking against the database limit before traffic moves.

A Designed Network

On AWS, this lab used the default VPC that AWS created for you. Nobody designed it.

On GCP, infra/gcp/terraform/02-network builds its own VPC with a named subnet and Cloud NAT. The cluster's machines get private addresses only. They reach the internet, and later the AWS database, through Cloud NAT, which gives all of them one shared public IP.

Region

01-foundation/variables.tf sets the region to us-east4, the Google region closest to AWS's us-east-1. Every database query from GKE crosses between the two clouds, so a short distance keeps that trip short. If your AWS setup isn't in us-east-1, change this before you build anything.

The Map

The left side is what exists. The right side is what Part 3 builds.

The diagram above maps the migration from AWS to GCP. Customers use one public address, managed by the GCP load balancer, which splits traffic between the existing AWS EC2 server and GKE. GKE runs the UI and API in private nodes, with Cloud NAT providing outbound internet access. Both environments connect to the same PostgreSQL database on AWS. Traffic shifts gradually to GKE while the database stays in place.

You'll meet three public IP addresses, in this order:

1. The Elastic IP. That's the EC2 server. Customers use it today.

2. The GKE Gateway IP. The load balancer sends GKE's share of traffic to it. Customers never type it.

3. The load balancer IP from 08-edge. Customers switch to this address once, and the traffic split happens behind it.

If You Need to Pause AWS

Part 3 doesn't need AWS running every minute. AWS is needed again in Chapter 11 (to copy the database URL into Secret Manager) and Chapter 13 (to open the database to GKE).

Don't stop the EC2 server and leave RDS running. RDS is most of the daily AWS bill, a stopped RDS restarts itself after seven days, and the Elastic IP bills either way.

If you continue within a day or two, leave AWS up. If you stop for several days, destroy it:

cd infra/aws/terraform && terraform destroy

To come back, run terraform apply in that folder and repeat Chapter 5. You get a new Elastic IP and a new DATABASE_URL. Your Chapter 6 scorecard still counts as the baseline.

Done When:

• You told Maya the Standard and Autopilot daily numbers and why the lab uses Standard

• You told Daniel how many API processes can run, and how many connections each can open

Part 3. How to Build Google Cloud With Terraform

There's a working AWS app and a scorecard. Google Cloud isn't set up yet. There's no network, no cluster, and no registry until you build them.

The UI and the API will run on that cluster later. Postgres stays on AWS.
So the order is: build an empty place in GCP first, then put the app there, then teach the app how to reach RDS across the internet.

This part is only the empty place. Customers don't move yet.

Chapter 10. Set Up gcloud and Pick Your Project

Friday, 8:45 a.m.

Google Cloud is a blank page. Before you build anything, your laptop needs to know which project to build in, and who you are.

Get gcloud, kubectl, and Terraform pointed at the right Google project, with billing on.

Steps

gcloud is the Google Cloud command-line tool. Install it, then close the terminal and open a new one.

On a Mac:

brew install --cask google-cloud-sdk

On Windows, download the installer from Install the Google Cloud CLI, tick the box that adds gcloud to PATH, and reopen Git Bash. On Linux, follow the Linux tab on the same page. gcloud version should print a version.

If kubectl version --client fails after gcloud works, install kubectl and the plugin it uses to log in to GKE:

gcloud components install kubectl
gcloud components install gke-gcloud-auth-plugin

The plugin must be on your PATH. If a later kubectl get nodes says gke-gcloud-auth-plugin not found, run this and open a new terminal:

echo 'export PATH="$(gcloud info --format="value(installation.sdk_root)")/bin:$PATH"' >> ~/.zshrc

That line is for zsh, the default Mac shell. On Linux with Bash, change ~/.zshrc to ~/.bashrc. On Windows, the installer usually handles PATH. Reopen Git Bash after the install.

Now log in and pick the project:

gcloud auth login --update-adc
gcloud projects list
gcloud config set project YOUR_PROJECT_ID
gcloud billing projects describe YOUR_PROJECT_ID
terraform version
kubectl version --client

--update-adc matters. Plain gcloud auth login only lets gcloud use your account. Terraform is a separate program that looks for a file called Application Default Credentials. The flag writes that file in the same browser login. On the Google page, click Allow and don't close the tab early.

Which ID to Use

gcloud projects list prints two columns that look alike:

• NAME is a label, and many projects share the default label My First Project.

• PROJECT_ID is unique, for example my-first-project-123456.

Copy the PROJECT_ID, never the name. Paste it wherever the commands say YOUR_PROJECT_ID.

If you have several projects, pick one with billing on. Run gcloud billing projects describe with each ID until the output includes billingEnabled: true.

You can also find the ID in the Google Cloud Console: click the project name at the top of the page and read the ID column.

Check

gcloud config get-value project

This prints your Project ID. If it prints [None], run the gcloud config set project command again.

If it Fails:

• Terraform later says could not find default credentials: you logged in without the flag. Run:

gcloud auth application-default login

• That command says cloud-platform scope is required but not consented: the browser closed before you clicked Allow. Run gcloud auth login --update-adc again and click Allow.

Today is only a gcloud login. The name api.clearpay.test shows up later, when Stage 8 builds the load balancer. Skip it until then.

Done When:

• gcloud config get-value project prints your Project ID

• gcloud billing projects describe shows billingEnabled: true

• kubectl version --client prints a version

Message Maya: "gcloud is logged in. The project is set. Billing is on. Ready to start GCP."

Maya: "Build one stage at a time. Tell me when each one is verified. If you run all eight at once and the third one breaks, you won't know which one to fix."

Chapter 11. Build GCP in Eight Terraform Stages

Friday, 10:30 a.m., to Monday of week two

GCP is built in eight Terraform stages. Each stage creates one layer of the destination environment.

Run them in order. After each stage, verify the result before moving to the next.

Build the Google side of the migration: APIs, network, cluster, image registry, identities, secrets, error alerting, and the public load balancer.

Why Eight Stages?

The Terraform lives in eight folders, 01-foundation through 08-edge. Each folder has its own state file and reads only the earlier stages it needs. If 03-gke fails, the problem is in the cluster, not somewhere in one 1,500-line file.

This isn't "run Terraform eight times". The work is to build eight things and prove each one exists before the next one needs it.

State stays local on your laptop, which is fine for one person. A team would move each stage's state to a Google Cloud Storage bucket before a second person runs apply.

The diagram above shows how Terraform builds GCP infrastructure in eight stages. It enables the required APIs, creates the network and private GKE cluster with Cloud NAT, then sets up the image registry, identities, secrets, error monitoring, and public load balancer. Cloud NAT provides outbound connectivity for private nodes, including access to AWS RDS later in the migration.

How Every Stage Runs

Each stage uses the same three commands inside its folder:

terraform init
terraform plan
terraform apply

Read the plan before you apply. Every stage below lists the plan count to expect, for example Plan: 7 to add, 0 to change, 0 to destroy. If the count differs, or the plan wants to destroy something, stop and read it before typing yes.

terraform apply asks Do you want to perform these actions?. Type yes and press Enter.

Folder paths below start from the repository root. If you're already inside infra/gcp/terraform/0X-..., use cd ../0Y-... to move to the next stage.

For the "In the console" checks: pick your project at the top of the page, and set the region to us-east4 when a page has a region menu. An empty list often means the menu is still on us-central1.

Stage 1: Turn on the Google APIs (01-foundation)

Google Cloud services are off by default in a new project. This stage turns on ten of them: GKE, Artifact Registry, Secret Manager, IAM, logging, and others. If you skip it, 03-gke fails later with a permission error that points nowhere useful.

cd infra/gcp/terraform/01-foundation
cp terraform.tfvars.example terraform.tfvars

If terraform.tfvars already exists with your Project ID, skip the copy. A second copy overwrites your edit.

Open terraform.tfvars and replace clearpay-migration-demo with your Project ID. Then run:

terraform init
terraform plan
terraform apply

Expected plan: google_project_service ten times, and no cluster.

Plan: 10 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + project_id = "YOUR_PROJECT_ID"
  + region     = "us-east4"

If the plan prints clearpay-migration-demo, fix terraform.tfvars first. Secret Manager and Service Networking can sit on Still creating for about half a minute during apply.

Check:

gcloud services list --enabled --filter='config.name:container.googleapis.com'

NAME                      TITLE
container.googleapis.com  Kubernetes Engine API

If this prints nothing, the API isn't on for the project gcloud is looking at. Don't go to Stage 2 yet. Check these three causes, in order:

1. gcloud points at a different project. Run gcloud config get-value project. It must print the same Project ID you put in terraform.tfvars. If it doesn't, run gcloud config set project YOUR_PROJECT_ID and check again.

2. The apply didn't finish. Run terraform plan in 01-foundation again. A finished stage prints No changes. If the plan still wants to add resources, run terraform apply and type yes.

3. Google hasn't caught up yet. A newly enabled API can take a minute to show in the list. Wait one minute and run the check again.

Message Maya: "the first stage applied, and Kubernetes Engine API shows as enabled."

Maya: "Good. Network next. Nothing else."

Stage 2: Build the Network (02-network)

A VPC is a private network inside your Google project. Think of it as the neighborhood the cluster will live in, and the subnet as one street in it. This stage creates:

• clearpay-vpc, the private network

• clearpay-gke-subnet, the address range the cluster's machines use, with two extra ranges named pods and services reserved for Stage 3

• clearpay-router and clearpay-nat, the Cloud NAT setup

• clearpay-nat-ip, the one public IP that all cluster machines share when they call out

• two firewall rules so machines inside can talk to each other and Google's health checks can reach them

This folder has no terraform.tfvars. It reads your Project ID from Stage 1's state file.

cd infra/gcp/terraform/02-network

Run:

terraform init
terraform plan
terraform apply

Expected plan:

Plan: 7 to add, 0 to change, 0 to destroy.

Expected end of apply:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

nat_ip_address = "x.x.x.x"
network_name = "clearpay-vpc"
subnet_name = "clearpay-gke-subnet"

Check:

gcloud compute networks describe clearpay-vpc
terraform output nat_ip_address

Look for name: clearpay-vpc, a subnetworks line ending in clearpay-gke-subnet, and autoCreateSubnetworks: false. That last line means this network was designed, not generated.

Write the NAT IP down. Chapter 13 opens the AWS database to exactly this address.

In the console: VPC networks shows clearpay-vpc. Cloud NAT shows clearpay-nat on clearpay-router. IP addresses shows clearpay-nat-ip.

If the plan can't read ../01-foundation/terraform.tfstate, Stage 1 isn't done.

Message Maya: "clearpay-vpc and Cloud NAT exist. The NAT IP is in your notes."

Daniel sees the NAT IP in your notes.

Daniel: "Every call our pods make to the outside world leaves from that one address. That includes calls to the database on AWS. Later we tell the AWS database to accept connections from this address and nothing else."

That happens in Chapter 13. Keep the NAT IP in your notes until then.

Stage 3: Create the GKE Cluster (03-gke)

This stage creates a Standard cluster named clearpay in zone us-east4-a, with a node pool named primary of one to three e2-medium machines. The machines have private IPs only and use Cloud NAT to reach out. The control plane, Google's managed API server for the cluster, has a public endpoint so kubectl on your laptop can reach it.

GKE first creates a default node pool, then Terraform deletes it and creates primary, which you control.

cd infra/gcp/terraform/03-gke

Run:

terraform init
terraform plan
terraform apply

Expected plan: the cluster and the primary node pool.

Plan: 2 to add, 0 to change, 0 to destroy.

The apply sits on Still creating for seven to ten minutes. Don't press Ctrl+C.

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

cluster_name = "clearpay"
get_credentials_command = "gcloud container clusters get-credentials clearpay --zone us-east4-a --project YOUR_PROJECT_ID"

cluster_endpoint prints <sensitive>. You don't need it.

Point kubectl at the new cluster:

$(terraform output -raw get_credentials_command)

A good run prints kubeconfig entry generated for clearpay.

Check:

kubectl get nodes

NAME                                 STATUS   ROLES    AGE   VERSION
gke-clearpay-primary-........        Ready    <none>   2m    v1.xx

Ready means the machine is on and joined the cluster. ClearPay isn't on it yet.

If the apply mentions container.googleapis.com, go back to Stage 1. If it mentions the subnet or the pods or services ranges, go back to Stage 2. Don't edit 03-gke to hide those errors.

If kubectl says gke-gcloud-auth-plugin not found, the cluster is fine. Don't destroy it. Install the plugin and fix PATH for this terminal:

gcloud components install gke-gcloud-auth-plugin
export PATH="$(gcloud info --format='value(installation.sdk_root)')/bin:$PATH"
kubectl get nodes

In the console: Kubernetes clusters shows clearpay running. VM instances shows a gke-clearpay-primary- machine with an internal IP and no external IP.

Message Maya: "the cluster is up. One node is Ready. Nothing of ours runs on it yet."

The apply took nine minutes. You watched every one of them.

Stage 4: Create the Image Registry (04-registry)

A container image is a packaged copy of the app. GKE downloads images from a registry. This stage creates a private Docker registry in Artifact Registry, named clearpay, in us-east4. It stays empty until the pipeline pushes to it in Chapter 14.

cd infra/gcp/terraform/04-registry

Run:

terraform init
terraform plan
terraform apply

The two resources are the registry and a rule that lets the cluster's machines pull images from it.

Plan: 2 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

repository_url = "us-east4-docker.pkg.dev/YOUR_PROJECT_ID/clearpay"

Check:

gcloud artifacts repositories describe clearpay --location=us-east4

Look for format: DOCKER, a name ending in /repositories/clearpay, and Repository Size: 0.000MB. An empty registry is correct today. You'll also see SCANNING_DISABLED. The pipeline scans images with Trivy instead, in Chapter 14.

Message Maya: "the image registry exists. It's empty."

Stage 5: Create Identities with No Key Files (05-identity)

Two things will need Google permissions later: the ClearPay pods on GKE, and GitHub Actions when it pushes images and deploys.

The old way is to download a service account key file. A key file is a password on disk. If it leaks, anyone can use it. This lab never creates one.

Instead, this stage creates three Google service accounts (identities with permissions) and lets named callers borrow them for short periods:

• clearpay-api-staging, borrowed by pods in the staging namespace

• clearpay-api-production, borrowed by pods in the production namespace

• clearpay-ci, borrowed by GitHub Actions through a Workload Identity pool named clearpay-github-pool

Workload Identity is the Google feature that links a Kubernetes service account, or a GitHub repository, to a Google service account without a key.

Think of each service account as a badge. Nothing wears a badge today. Later, Google lets a named pod or a named repository wear one for a short time, and no key file ever exists.

Namespaces are separate areas inside one cluster. Staging and production share the cluster's machines but not their identities, so a staging pod can't read production's secrets. A real company would usually run separate clusters, often in separate projects. This lab uses one cluster to keep the bill small.

cd infra/gcp/terraform/05-identity
cp terraform.tfvars.example terraform.tfvars

The only value in the file is github_repo. If you already have your GitHub copy of this repository, put its owner/name there now, with no .git at the end. If not, leave YOUR_GITHUB_USER/YOUR_REPO. Chapter 14 tells you when to set it and apply this stage again.

Run:

terraform init
terraform plan
terraform apply

Expected plan:

Plan: 10 to add, 0 to change, 0 to destroy.

The ten are the three service accounts, the GitHub pool and its provider, and the permission links between them.

Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

ci_service_account_email = "clearpay-ci@YOUR_PROJECT_ID.iam.gserviceaccount.com"
service_account_emails = {
  "production" = "clearpay-api-production@YOUR_PROJECT_ID.iam.gserviceaccount.com"
  "staging" = "clearpay-api-staging@YOUR_PROJECT_ID.iam.gserviceaccount.com"
}

Check:

gcloud iam service-accounts list --filter='email~clearpay-'

Look for three emails: clearpay-api-staging, clearpay-api-production, and clearpay-ci.

In the console: Service accounts lists the three. Workload Identity Federation shows clearpay-github-pool with provider clearpay-github-provider.

If apply fails with Error 409: Requested entity already exists on the pool, you ran this stage before. Google keeps a deleted pool for 30 days and blocks its name. Restore it with gcloud iam workload-identity-pools undelete clearpay-github-pool --location=global,
then run terraform import google_iam_workload_identity_pool.github projects/YOUR_PROJECT_ID/locations/global/workloadIdentityPools/clearpay-github-pool and apply again.

If the same error then names the provider, restore it with gcloud iam workload-identity-pools providers undelete clearpay-github-provider --workload-identity-pool=clearpay-github-pool --location=global,
then run terraform import google_iam_workload_identity_pool_provider.github projects/YOUR_PROJECT_ID/locations/global/workloadIdentityPools/clearpay-github-pool/providers/clearpay-github-provider and apply again.

Message Maya: "three service accounts exist. Nothing uses them yet."

Maya: "No key files?"

"None."

Stage 6: Create the Secrets (06-secrets)

ClearPay on GKE needs two secrets: the database URL and the JWT signing key. Both already exist on AWS. This stage creates four secrets in Secret Manager, one pair per namespace:

• clearpay-staging-jwt-secret

• clearpay-staging-database-url

• clearpay-production-jwt-secret

• clearpay-production-database-url

The staging service account can read only the staging pair. The production service account can read only the production pair.

Think of each secret as a locked drawer. Each badge from Stage 5 opens only its own drawers.

This lab puts the same values in both pairs. Both environments use the same RDS, and a login on AWS must still work when the next request lands on GKE. A real company would give each environment its own values.

Terraform creates the secrets with a placeholder value, REPLACE_ME. Terraform is allowed to create the secrets but not to invent their values. So the order is: apply first, then copy the real values in by hand.

cd infra/gcp/terraform/06-secrets

Run:

terraform init
terraform plan
terraform apply

Terraform creates fourteen resources:

• four secrets

• four first versions

• four access rules for the API service accounts

• two rules that let the pipeline's clearpay-ci account read the staging pair

Plan: 14 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Now copy the AWS values in. AWS must be running.

Run these from the repository root, the top folder of the project, not from 06-secrets and not from infra/aws/terraform. You're still in 06-secrets, so go up four folders first:

cd ../../../..

ls should now show api, infra, k8s, and scripts.

You don't need to move into the AWS folder. terraform -chdir=infra/aws/terraform reads the AWS values from where you stand. Each command pipes a value straight into Secret Manager, so nothing prints on screen.

The database URL, once for each namespace:

terraform -chdir=infra/aws/terraform output -raw database_url \
  | gcloud secrets versions add clearpay-staging-database-url --data-file=-
terraform -chdir=infra/aws/terraform output -raw database_url \
  | gcloud secrets versions add clearpay-production-database-url --data-file=-

A good run prints Created version [2] and doesn't print the URL.

The JWT secret, read from the .env file on the AWS server:

INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo grep '^JWT_SECRET=' /opt/clearpay-api/.env | cut -d= -f2-" \
  | gcloud secrets versions add clearpay-staging-jwt-secret --data-file=-
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo grep '^JWT_SECRET=' /opt/clearpay-api/.env | cut -d= -f2-" \
  | gcloud secrets versions add clearpay-production-jwt-secret --data-file=-

A later terraform apply in this folder won't put REPLACE_ME back. The Terraform file tells it to leave the value alone after the first apply.

Check:

gcloud secrets list --filter='name~clearpay-'

Then prove no secret still holds the placeholder. This loop prints ok or STILL_PLACEHOLDER and never prints a secret:

for s in clearpay-staging-database-url clearpay-staging-jwt-secret \
         clearpay-production-database-url clearpay-production-jwt-secret; do
  val=$(gcloud secrets versions access latest --secret="$s")
  if [ "$val" = "REPLACE_ME" ]; then echo "STILL_PLACEHOLDER $s"; else echo "ok $s"; fi
done

Look for four ok lines. If one says STILL_PLACEHOLDER, run its copy command again.

In the console: Secret Manager lists the four secrets, each with version 2 as the latest. Don't click "Show secret value" on a shared screen.

Message Maya: "the four secrets hold the AWS values. Nothing was printed on screen or pasted into a chat."

Stage 7: Count API Errors (07-observability)

Every time something happens in the API, it writes a short note called a log line. A login writes one. A payment writes one. When something goes wrong, the note says "level":"error". A failed payment writes this:

{"level":"error","message":"transaction_failed","timestamp":"2026-09-29T10:15:02.000Z","error":"..."}

On GKE, Google Cloud collects every one of these notes. This stage builds two things on top of them:

1. A counter, named clearpay-api-error-count. It goes up by one each time an error note appears.

2. An alarm, named ClearPay API - elevated error rate. It switches on when the counter goes up more than five times in five minutes. One error can be bad luck. Five in five minutes means something is broken.

The alarm has no phone number or email attached yet. When it switches on, the Google Cloud console shows it, but nobody gets a message. You can add an email later in the console.

The counter stays at zero for now, because ClearPay doesn't run on GKE yet. Zero is the correct result today.

cd infra/gcp/terraform/07-observability

Run:

terraform init
terraform plan
terraform apply

Expected plan:

Plan: 2 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

log_metric_name = "clearpay-api-error-count"

If apply stops with Error 404: Cannot find metric(s) that match type, nothing is broken. Google needs a few minutes to register a brand-new counter before an alarm can watch it. The counter was created. Wait five minutes and run terraform apply again. This time the plan shows 1 to add, only the alarm.

Check:

gcloud logging metrics describe clearpay-api-error-count

Look for name: clearpay-api-error-count and a filter that mentions clearpay-api and jsonPayload.level="error".

Message Maya: "the error counter exists and reads zero. No app runs on GKE yet."

Maya: "Last one. This is the address customers will use, so take your time."

Stage 8. Build the Public Load Balancer (08-edge)

Stages 1 to 7 built the house. This stage builds the front door: the address customers keep for the rest of the migration. It's a Google global load balancer with a traffic dial behind it: a percentage to AWS, the rest to GKE. Today it sends 100 percent to AWS.

This stage creates:

• clearpay-api-ip, the public address customers will use

• a self-signed certificate so HTTPS works

• the traffic split, set to AWS 100, GKE 0

• clearpay-gke-edge-ip, a second IP for GKE's side. The load balancer uses it. Customers never do.

The diagram below shows how the GCP load balancer directs customer traffic between AWS and GKE using configurable traffic percentages. Customers connect through one public IP. AWS receives traffic through its Elastic IP, while GKE receives traffic through a separate edge IP that customers never access directly. Both environments connect to the same PostgreSQL database on AWS. The GKE cluster is empty at this stage.

cd infra/gcp/terraform/08-edge
cp terraform.tfvars.example terraform.tfvars

Skip the copy if the file exists. Put your Elastic IP in aws_backend_ip. Get it from a second terminal at the repository root:

terraform -chdir=infra/aws/terraform output -raw instance_public_ip

Keep the weights at 100 and 0:

aws_traffic_weight = 100
gke_traffic_weight = 0

Run:

terraform init
terraform plan
terraform apply

The apply takes about two minutes.

Plan: 17 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

Outputs:

current_traffic_split = "AWS: 100% / GKE: 0%"
static_ip_address = "x.x.x.x"
gke_edge_ip_address = "y.y.y.y"
hosts_file_line = "x.x.x.x api.clearpay.test"

Write both IPs down. static_ip_address is the customer address. gke_edge_ip_address is not.

Give the Address a Name

You don't own a domain, so you point the name api.clearpay.test at the load balancer in your computer's hosts file.

You're already in 08-edge after the apply, so skip the cd line below. Run it only if you opened a new terminal at the repository root:

cd infra/gcp/terraform/08-edge
LINE=$(terraform output -raw hosts_file_line)
echo "LINE is: [$LINE]"

If LINE is empty, stop. If it looks like x.x.x.x api.clearpay.test, add it to the hosts file.

On a Mac:

echo "$LINE" | sudo tee -a /etc/hosts
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

On Linux, run only the first line. On Windows, add the same IP and api.clearpay.test to C:\Windows\System32\drivers\etc\hosts in an editor opened as Administrator.

Don't run sudo cd. cd is built into the shell, so sudo can't change your folder.

Confirm the line is there:

grep api.clearpay.test /etc/hosts

The name ends in .test on purpose. .test is reserved for testing, so Chrome lets you click through a self-signed certificate warning. Don't change lab_hostname to a .dev name. Chrome forces strict HTTPS on every .dev site and hides the button to continue.

Check

curl -k https://api.clearpay.test/health

Expected output:

{"status":"ok","served_by":{"platform":"aws","version":"1.0.0","instance":"ip-172-31-64-73","commit":"504e0cd","rollback_active":false}}

Your instance and commit values will differ. "platform":"aws" is the check. The request went to the new Google address, and the load balancer passed it to AWS, because the weights are 100 and 0.

A new Google load balancer can take a few minutes to start. If you don't see the JSON yet, you may see one of these first:

• Connection reset. The load balancer is still starting. Wait a minute and run the same curl again.

• A Google page titled 404 Not Found. The load balancer answered but hasn't reached AWS yet. Confirm AWS itself is up:

curl -k https://YOUR_ELASTIC_IP/health

If AWS answers, wait a few minutes and try api.clearpay.test again. Stage 8 is done even if this is still slow. Chapter 20 checks it again.

When the JSON appears, open https://api.clearpay.test in a browser. In Chrome, click Advanced, then "Proceed to api.clearpay.test (unsafe)". You see the ClearPay login, served by AWS through the new address. If the browser says NXDOMAIN, the hosts line is missing or the cache wasn't flushed.

In the console: IP addresses shows clearpay-api-ip and clearpay-gke-edge-ip. Load balancing shows clearpay-migration-forwarding-rule with your Elastic IP as a backend.

Done When:

• Stage 1: container.googleapis.com is enabled

• Stage 2: terraform output nat_ip_address prints an IP, and you wrote it down

• Stage 3: kubectl get nodes shows at least one Ready node

• Stage 4: the clearpay registry exists

• Stage 5: three clearpay- service accounts exist

• Stage 6: the loop prints four ok lines

• Stage 7: the error metric exists

• Stage 8: /etc/hosts has api.clearpay.test, and weights are 100 / 0

Tell Maya: "all eight stages are verified. Weights are 100 and 0. Nothing runs on GKE yet."

Daniel: "The cluster is empty. Next we put ClearPay on it."

Part 4. How to Get ClearPay Onto GKE Staging

Maya stops by to look at the new address in your browser. She reads the dashboard line and smiles.

Maya: "Nice work on the front door. But look, it still says aws. Customers can reach the new address, and every request still lands on the old server. The cluster on GKE is empty, so nobody can send $20 there yet."

She points at your terminal.

Maya: "Next step is to get ClearPay onto that cluster. One thing I care about: let the pipeline do it, not a person typing commands. A pipeline does it the same way every time, and we can see exactly what it did."

https://api.clearpay.test already works in your browser, so it's easy to think ClearPay is on GKE. It's not. That address is Google's new front door, the load balancer from Stage 8. With the weights at 100 and 0, the front door sends every request to AWS. The served by aws/1.0.0 line on the dashboard proves it. The GKE cluster itself is still empty.

Browser → api.clearpay.test (Google load balancer) → 100% → AWS EC2 → RDS
                                                  →   0% → GKE (empty)

In this part the pipeline puts ClearPay on GKE staging, and the promotion gate from Chapter 6 decides whether staging is good enough. Staging is a private copy of the app inside the cluster. No customer reaches it.

Chapter 12. Check the API Image

Week two, Tuesday, 9:00 a.m.

The cluster is built and empty. Before anything goes on it, Daniel pulls the API Dockerfile up on your screen.

Daniel: "We're about to put this on a cluster. Show me the container doesn't run as root."

Prove the API image is small, holds no secrets, and does not run as root.

Why This Matters

A process that runs as root inside a container has more power than it needs if an attacker breaks in. Production images should run as a regular user.

Steps

Open api/Dockerfile. It has two stages:

1. The first stage, FROM node:22-slim AS deps, installs only production packages with npm install --omit=dev.

2. The second stage starts again from node:22-slim. It applies Debian security updates and deletes npm and yarn, because the running API only needs node src/index.js. It creates appuser with user ID 1001 and copies in node_modules, src, and db. The last lines switch to USER appuser.

Notice what the image doesn't contain: DATABASE_URL and JWT_SECRET. Those arrive at run time from Secret Manager.

Docker Desktop must be running. From the repository root:

docker build -t clearpay-api:test ./api
docker run --rm clearpay-api:test whoami

Check

The second command prints:

appuser

If it Fails:

• It prints root: the USER appuser line is missing. Fix api/Dockerfile and run both commands again.

• Docker can't connect: start Docker Desktop and retry. The cluster is fine.

Done When:

• whoami prints appuser

Chapter 13. Prepare the Cluster for ClearPay

Tuesday, 11:00 a.m.

The image check is done. You open GitHub, ready to run the pipeline. Daniel looks at your screen and shakes his head.

Daniel: "The first deploy will fail twice, for reasons that have nothing to do with the pipeline. Fix those first."

Before the first deploy, the cluster needs two things it doesn't have yet:

1. A way for pods to read secrets from Secret Manager.

2. A network path to the database on AWS.

Without the first, Kubernetes rejects the staging files. Without the second, the pods start but never become ready.

Part A. Turn on the Secret Manager Add-ons

Daniel: "The pods should read the Secret Manager values from Stage 6. If they don't, write down why they still use a file."

The staging and production files in k8s/overlays/ read the database URL and JWT secret from Secret Manager. They use two GKE add-ons:

• The Secret Manager CSI driver mounts secrets as files inside the pod.

• Secret Sync copies them into a normal Kubernetes Secret named clearpay-api-secrets. The Deployment loads that Secret into the API's environment variables.

  

The pod proves who it is through Workload Identity, using the service account from Stage 5. No key file exists anywhere. The API code doesn't change. It still reads plain environment variables, as it does on AWS.

Turn on both add-ons. gcloud needs two separate commands, and each takes a few minutes:

gcloud container clusters update clearpay --zone us-east4-a \
  --enable-secret-manager
gcloud container clusters update clearpay --zone us-east4-a \
  --enable-secret-sync

These run here, not in 03-gke, so the cluster stage stays small and you see the moment secret access is added.

Leave PROJECT_ID as it is in k8s/overlays/staging/secretproviderclass.yaml and patch-serviceaccount.yaml. The pipeline replaces it with your real Project ID at deploy time. Don't type your ID into those files.

Part B. Open the AWS Database to GKE

Don't describe this as "connecting GKE to the database". That phrasing suggests the database moves too, and it doesn't. This step lets GKE reach and use the real, unmoved AWS database, safely.

The cluster's machines have no public IP. Their outbound traffic leaves through Cloud NAT with one public address, the NAT IP from Stage 2. RDS will accept connections from exactly that address on port 5432, and the connection string already requires TLS (sslmode=require).

GKE Pod  (clearpay-api)
   │
   ▼
Private node  (no public IP)
   │
   ▼
Cloud NAT  (clearpay-nat-ip)
   │
   │  public egress, one /32
   ▼
AWS security group  (additional_db_ingress_cidrs)
   │  port 5432 + SSL
   ▼
RDS PostgreSQL 16

This is one of three ways to connect two clouds:

To compare them for your own system:

(expected monthly GB) × (price per GB on that path) + fixed monthly cost of that option

ClearPay sends small queries, so a VPN would cost more each month than the data is worth. This lab uses the allowlist. A busier system might choose differently.

Get the NAT IP:

cd infra/gcp/terraform/02-network
terraform output nat_ip_address

Put it in infra/aws/terraform/terraform.tfvars:

db_publicly_accessible      = true
additional_db_ingress_cidrs = ["<the NAT IP>/32"]

Then apply the AWS stack:

cd infra/aws/terraform
terraform plan
terraform apply

The plan adds a new security group rule for the NAT IP and makes RDS publicly reachable. It must not remove the rule that lets the EC2 server reach RDS. Those are two separate rules in Terraform: aws_security_group_rule.db_from_nat for GKE and aws_security_group_rule.db_from_api for EC2.

Check

AWS must still reach its database. From the repository root:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
curl -k https://$INSTANCE_IP/health/ready

{"status":"ready", ... "platform":"aws" ...}

If this prints not_ready, or the AWS login page says internal error, the database stopped accepting the EC2 server. Stop here and read the Terraform plan again. Fix AWS before you touch GKE, because AWS still serves every customer.

Check Before You Go On

Chapter 14 pushes code and starts the pipeline. If any step above was skipped, the pipeline fails two chapters later with an error that doesn't say which step you missed. Run this check now instead. It prints ok or MISSING for each thing the pipeline needs. Run it from the repository root:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
NAT_IP=$(terraform -chdir=infra/gcp/terraform/02-network output -raw nat_ip_address)
DB_HOST=$(terraform -chdir=infra/aws/terraform output -raw db_endpoint | cut -d: -f1)

grep -q '^db_publicly_accessible *= *true' infra/aws/terraform/terraform.tfvars \
  && echo "ok   RDS public setting" \
  || echo "MISSING db_publicly_accessible = true in infra/aws/terraform/terraform.tfvars"
grep -q "$NAT_IP/32" infra/aws/terraform/terraform.tfvars \
  && echo "ok   NAT IP in the allowlist" \
  || echo "MISSING $NAT_IP/32 in additional_db_ingress_cidrs"
kubectl get csidriver secrets-store-gke.csi.k8s.io >/dev/null 2>&1 \
  && echo "ok   Secret Manager driver" \
  || echo "MISSING Secret Manager driver: run the two gcloud commands in Part A"
kubectl get crd secretsyncs.secret-sync.gke.io >/dev/null 2>&1 \
  && echo "ok   Secret Sync" \
  || echo "MISSING Secret Sync: run the two gcloud commands in Part A"
curl -sk https://$INSTANCE_IP/health/ready | grep -q '"status":"ready"' \
  && echo "ok   AWS still reaches its database" \
  || echo "MISSING AWS cannot reach its database: read the Terraform plan again"

The last check tests the real path from GKE to the database. It starts a small throwaway pod, asks the database port if it's accepting connections, and deletes the pod. The first run takes about a minute, because Kubernetes downloads the image:

kubectl run dbcheck --rm -i --restart=Never --image=postgres:16 -- \
  pg_isready -h "$DB_HOST" -p 5432 -t 10 \
  && echo "ok   GKE reaches the database" \
  || echo "MISSING GKE cannot reach the database"

You need six ok lines and no MISSING. Each MISSING line names the fix. Apply it, then run the check again. Don't start Chapter 14 until all six print ok.

If the last line says MISSING but the first two say ok, RDS may still be applying the change. Wait two minutes and run that command again.

Done When:

• Both gcloud container clusters update commands finished

• RDS allows the NAT IP, and curl -k https://$INSTANCE_IP/health/ready still prints ready

• The check above printed six ok lines and no MISSING

Chapter 14. Build the CI/CD Pipeline

Tuesday, 3:00 p.m.

The cluster is ready for its first deploy. Maya reads the team channel and adds one rule.

Maya: "From now on, let's not run kubectl apply on staging by hand. The pipeline should build the image, scan it, push it, and deploy it, the same way every time."

The workflow file is already written: .github/workflows/deploy.yml. Your job is to understand it, connect it to Google Cloud, and watch it deploy staging.

Why a Pipeline?

When a person deploys by hand, each deploy is slightly different. Someone skips the scan because it's late. Someone deploys from a laptop with uncommitted changes. When something breaks, nobody can say what ran.

A pipeline removes those differences. Every image comes from a commit, passes the same scan, and reaches the cluster through the same steps. The promotion gate in Chapter 15 means something because it checks an image built that way.

How the Pipeline Works

The workflow has two jobs:

Open .github/workflows/deploy.yml in your repository and keep it open alongside this diagram. The diagram maps the workflow's two jobs and their sequence. The file contains the actual GitHub Actions configuration. Follow both together as you read through the triggers, permissions, image build and scan, staging deployment, and production promotion.

Read the file top to bottom with this map.

Triggers: on: push: branches: [main] runs Job 1 on every push to main. workflow_dispatch adds a "Run workflow" button in GitHub with one checkbox, promote_to_production. Job 1 runs only when that box is off (if: ${{ !inputs.promote_to_production }}). Job 2 runs only when it's on. A push can never reach production by itself.

Permissions: id-token: write lets the job ask GitHub for a short-lived OIDC token, a signed statement that says "this run belongs to repository X". Google checks that statement against the pool from Stage 5 and hands back temporary credentials for clearpay-ci. No key file sits in GitHub secrets, so there's no key to leak.

Image tag: The tag is the short commit SHA from git rev-parse --short HEAD. Each image points to exactly one commit. The pipeline also pushes a moving tag, staging-latest, but deploys use the SHA.

Scan: Trivy is an open-source scanner that checks the packages inside an image against lists of known vulnerabilities. Each known vulnerability has an ID like CVE-2026-29786. The workflow sets:

• severity: CRITICAL,HIGH, so only serious findings count

• ignore-unfixed: true, so a finding with no patch available yet doesn't block the build

• exit-code: "1", so any remaining finding fails the job

Together they mean: fail if a serious vulnerability has a fix and this image doesn't have it. Nothing gets pushed if the scan fails.

Push: The job tags each image with the full registry path, us-east4-docker.pkg.dev/<project>/clearpay/api:<sha>, and pushes it. clearpay-ci has artifactregistry.writer from Stage 5, so the push works.

Deploy to staging: The step works inside k8s/overlays/staging:

1. It writes the two staging secrets from Secret Manager into secrets.staging.env on the runner. umask 077 makes the file readable only by the runner's user. The file is gitignored and dies with the runner.

2. sed replaces the PROJECT_ID placeholder in patch-serviceaccount.yaml and secretproviderclass.yaml with your real Project ID.

3. kustomize edit set image points the overlay at the image with this commit's SHA.

4. kubectl apply -k . sends the overlay to the cluster.

5. kubectl rollout status waits up to 180 seconds for the API and UI pods to become ready. If they never do, the job fails.

clearpay-ci has container.developer, which lets it deploy workloads but not change the cluster itself.

Promote to production: Job 2 is the only way the pipeline touches production:

• environment: production pauses the job until a named reviewer clicks Approve in GitHub.

• It doesn't build anything. It reads the image string that staging is running right now and deploys that exact image to production.

That rule is called "build once, promote the same artifact". The image that passed the Chapter 15 gate is byte for byte the image customers get. A rebuild could pull a newer package and ship something nobody tested.

Kustomize, used in both deploy steps, is the tool behind kubectl apply -k. It takes the shared files in k8s/base/ and applies small environment-specific changes from k8s/overlays/staging/ or k8s/overlays/production/, called overlays. Staging and production share one base and differ only where they must.

Steps

First, put this repository on GitHub under your own account, with main as the default branch.

Then tell Google which repository may use clearpay-ci. Open infra/gcp/terraform/05-identity/terraform.tfvars and set github_repo to your repository's owner/name, for example Osomudeya/migrate-api-aws-gke-devops. Don't paste the full https://github.com/... address, and no .git at the end. Apply Stage 5 again:

cd infra/gcp/terraform/05-identity
terraform plan
terraform apply

The plan changes the pool's attribute condition and the clearpay-ci link. It doesn't touch the cluster. If you already set the real name in Chapter 11, the plan shows no changes.

On GitHub, open your repository and click Settings. In the left menu, click Secrets and variables, then Actions. Open the Variables tab and click New repository variable for each of these three. Use the Repository section, not Environment.

These are variables, not secrets. None of them is a password. The pipeline needs no GitHub secrets at all.

When you copy the output of the two terraform output -raw commands, don't copy a % at the end. On a Mac, the terminal shows % when output has no line break after it. It's not part of the value. The provider value must end in clearpay-github-provider, and the service account must end in .iam.gserviceaccount.com.

Then, still in Settings, click Environments, then New environment. Name it production and click Configure environment. Tick Required reviewers, search for your own GitHub name, select it, and click Save protection rules.

Why: Job 2 is the only path to production. A required reviewer makes GitHub stop that job and wait until a person clicks Approve, so nothing reaches customers by accident. Job 2 waits for that approval in Chapter 19.

You'll see a Manage environment secrets button on this page. Don't click it. The production environment needs no secrets, because the pipeline gets its Google access through Workload Identity, not a stored password. If your repository already holds secrets from other projects, leave them where they are.

If GitHub doesn't offer Required reviewers, your repository may be private on a free plan. Make the repository public for this lab, or skip the reviewer. Without one, the production job runs as soon as you trigger it.

Next, check before you push. A pipeline that fails costs three minutes and gives a long log. A check costs ten seconds. First run the check from the end of Chapter 13 again. All six lines must print ok. Then check the repository name you gave Google:

REPO=$(grep '^github_repo' infra/gcp/terraform/05-identity/terraform.tfvars | cut -d'"' -f2)
case "$REPO" in
  https*|*.git|YOUR_GITHUB_USER/*) echo "MISSING github_repo must be owner/name. It is: $REPO" ;;
  */*) echo "ok   github_repo $REPO" ;;
  *)   echo "MISSING github_repo must be owner/name. It is: $REPO" ;;
esac

It must print ok and your own owner/name.

Now, trigger the pipeline:

git commit --allow-empty -m "trigger pipeline"
git push

Check

Open the Actions tab and click the running workflow. Watch the steps of build-scan-push finish one after another, each with a check mark: build, Scan API image, Scan UI image, Authenticate to GCP, push, Deploy to staging.

If a step turns red, find it here. Each one ends by telling you to re-run the workflow:

To re-run after a fix, open the Actions tab, click the failed run, and click Re-run all jobs at the top right. You don't need to push again.

The second job, promote-to-production, shows a gray circle with a slash. That's correct. GitHub skips it on every push. It only runs when you start it by hand in Chapter 19.

When Deploy to staging passes, check the cluster:

kubectl -n staging get pods

The API and UI pods show Running with READY 1/1.

Staging has no public address, and the load balancer still sends every customer to AWS. Don't open a browser for staging. Chapter 15 reaches it through a private tunnel.

If the Scan Fails

Never change exit-code: "1" to "0". That turns the security gate off.

The Dockerfiles are already patched, so the scan should pass. A new security hole can turn it red later. Open the red job. Trivy prints a table. Find the Target column and match it:

After any Dockerfile change, rebuild locally and confirm the user is still appuser:

docker build -t clearpay-api:test ./api
docker run --rm clearpay-api:test whoami

Commit, push to main, and watch Actions again.

If Authenticate to GCP Fails

The error looks like this:

The given credential is rejected by the attribute condition.

Google accepted the GitHub token and then rejected it. Stage 5 names a different repository, often the placeholder YOUR_GITHUB_USER/YOUR_REPO. The value must be exactly owner/name, like Osomudeya/migrate-api-aws-gke-devops. A full URL doesn't match, and neither does another person's repository. Repeat step 2 above with your exact owner/name. Then retry without changing the workflow:

git commit --allow-empty -m "retry pipeline after WIF repo lock"
git push

If Deploy to Staging Fails

Two messages mean the same thing: error: timed out waiting for the condition and error: deployment "clearpay-api" exceeded its progress deadline. The files applied, but the pods never became ready. Ask the cluster why:

kubectl -n staging get pods
kubectl -n staging describe pod -l app=clearpay-api

Then match what you see. The first two rows appear in the job log. The rest appear in the pod output:

Fix A. Give the pipeline access to the staging secrets:

cd infra/gcp/terraform/06-secrets
terraform plan
terraform apply

Fix B. Let the cluster read the registry. The machines pull images as the default Compute Engine service account, which needs artifactregistry.reader. The plan adds that one rule:

cd infra/gcp/terraform/04-registry
terraform plan
terraform apply

Then restart the staging pods and wait:

kubectl -n staging rollout restart deployment/clearpay-api deployment/clearpay-ui
kubectl -n staging rollout status deployment/clearpay-api --timeout=180s
kubectl -n staging rollout status deployment/clearpay-ui --timeout=180s

Fix C. Run one API pod in staging: The small lab cluster can't fit two. Scale down, then re-run the workflow:

kubectl -n staging scale deployment/clearpay-api --replicas=1

To read the API's logs without printing secrets:

kubectl -n staging logs deployment/clearpay-api --tail=20

Done When:

• You can explain what each of the two jobs in deploy.yml does

• The production environment exists with you as required reviewer

• Deploy to staging passed

• kubectl -n staging get pods shows the API and UI Running, READY 1/1

The pipeline stops there. It doesn't run the promotion gate. You do, in Chapter 15.

Chapter 15. Check Staging and Run the Promotion Gate

Week two, Wednesday, 10:00 a.m.

The pipeline finished without an error at 6:40 last night. You took a screenshot. Nobody at ClearPay cares about the screenshot.

Daniel stops at your desk and looks at the passing pipeline.

Daniel: "So GKE is healthy?"

You answer: "The pods are healthy. That doesn't prove a customer can send twenty dollars."

Maya has read the thread. She adds a line.

Maya: "Customers don't get this build until the promotion script passes. Running pods aren't enough for me."

Why This Matters

Kubernetes reports a pod as ready when its health check answers. That says the process is up and can reach the database. It says nothing about logins, payments, or double charges. The promotion gate checks those.

The promotion gate is a script that you run on your laptop. The pipeline doesn't run it. It's also not the gray promote-to-production job you saw in GitHub. That job comes later, in Chapter 19, and only after this gate passes.

How Staging Runs

GKE cluster "clearpay"   zone us-east4-a
│
├── namespace staging
│     Service clearpay-api :80
│     └── API pods   (HPA 1–3)
│
└── namespace production   (created in Chapter 19)
      Gateway clearpay-gke-edge   ← clearpay-gke-edge-ip
      HTTPRoute → Service :80
      └── API pods   (HPA 1–3)
            each pod: DB_POOL_SIZE=10

Daniel looks at the same diagram.

"How many API processes can production run?"

"Up to three on this cluster. That's the HPA ceiling in patch-hpa.yaml."

"And how many database connections can each one open?"

"Ten. Same as db.js on Monday."

"So what could the database see?"

You stop. 30 from GKE. Plus 10 from AWS while both are live. On a cluster that could scale to 8 pods, it would be 80 from GKE alone.

"That's the number I asked you to remember," Daniel says.

Three files are worth opening now:

• k8s/base/deployment.yaml: the health checks call /health/ready, and each API pod reserves 250m CPU and 256Mi memory.

• k8s/base/networkpolicy.yaml: limits which pods may talk to the API.

• k8s/overlays/staging/patch-deployment.yaml: staging's replica count and rollout settings.

Step 1. Check the Database Path from GKE

Run a one-line Node script inside a staging API pod. It opens a database connection and prints connected:

kubectl -n staging exec deployment/clearpay-api -- node -e \
  "require('./src/db').checkConnection().then(() => { console.log('connected'); process.exit(0); }).catch(e => { console.error(e); process.exit(1); })"

connected

Step 2. Check the Secrets Arrived

Kubernetes holds the synced Secret with two keys. describe lists key names and sizes, not values:

kubectl -n staging get secret clearpay-api-secrets
kubectl -n staging describe secret clearpay-api-secrets

The API process has the value, checked without printing it:

kubectl -n staging exec deployment/clearpay-api -- node -e 'if (!process.env.DATABASE_URL) process.exit(1); console.log("DATABASE_URL is set")'

DATABASE_URL is set

The script is in single quotes on purpose. The default Mac shell, zsh, treats a ! inside double quotes as a history command and stops with zsh: event not found.

The CSI mount shows the secret files by name:

kubectl -n staging exec deployment/clearpay-api -- ls /mnt/secrets-store

Look for DATABASE_URL and JWT_SECRET.

Step 3. Run the Promotion Gate

Staging is private. kubectl port-forward opens a tunnel so localhost:8080 on your laptop reaches the staging API. The & keeps the tunnel running in the background. From the repository root:

kubectl -n staging port-forward svc/clearpay-api 8080:80 &

Wait for this line before you continue:

Forwarding from 127.0.0.1:8080 -> 4000

Amara is already on the thread: "Run the replay. I want to see one debit."

Run the same gate you ran against AWS in Chapter 6. The four arguments are the tunnel URL, Alice's email, her password, and Bob's email:

./scripts/promotion-gate.sh http://localhost:8080 \
  alice@clearpay.dev password123 bob@clearpay.dev

PASS: health endpoint returns ok
PASS: readiness endpoint confirms DB connectivity
PASS: login succeeds and returns a token
PASS: replaying an idempotency key returns replayed=true
PASS: balance debited exactly once (100 cents), not twice

5 passed, 0 failed

GATE PASSED: safe to promote to production.

Stop the tunnel with kill %1.

Alice exists on GKE without a new seed because GKE reads the same RDS database as AWS.

Step 4. Confirm Customers Are Still on AWS

Staging has no public address:

kubectl -n staging get svc clearpay-api

NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
clearpay-api   ClusterIP   10.x.x.x       <none>        80/TCP    12m

ClusterIP with <none> means only the cluster can reach it.

The customer address still answers from AWS:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
curl -sk https://$INSTANCE_IP/health

{"status":"ok","served_by":{"platform":"aws",...}}

If you see gke here, stop. If the curl fails, AWS is down, which is a different problem from GKE taking traffic.

So both clouds are running, but only AWS is open to customers. Staging runs on its own inside GKE. The gate isn't what keeps it running. It only tests it, through your tunnel, which is the only door in.

Browser  →  api.clearpay.test  →  100 percent  →  AWS EC2  (serves everyone)

GKE staging  ←  your tunnel (port-forward)  ←  the gate and the load test
GKE production  (does not exist yet, Chapter 19)

Customers reach GKE only after Chapter 21, when the front door starts sending it real traffic.

If it Fails

Don't promote this build.

• Unexpected end of JSON input or Connection refused: the script ran before the tunnel was listening. If Forwarding from 127.0.0.1:8080 is already in the terminal, run only the script again. Don't start a second port-forward.

• Health passes but ready fails: the pod can't reach PostgreSQL. Go back to Chapter 13, Part B.

• Login fails: check the values you typed against the seed: alice@clearpay.dev, password123, bob@clearpay.dev.

• checkConnection prints a TLS or timeout error: same as ready failing. Chapter 13, Part B.

• The ls shows no files, or the pod is stuck in ContainerCreating: run kubectl -n staging describe pod <pod-name> and look for a Secret Manager permission error. The annotation on the staging clearpay-api service account must match the staging email from Stage 5, character for character. That's the email Stage 6 allowed to read the secrets.

Fix the cause, push through the pipeline if the code or files changed, and run the gate again.

Decision

Maya: "Are we ready?"

"The gate passed. Health, readiness, login, replay, and one debit. Amara's replay check passed. Customers are still on AWS."

Maya: "Then we can talk about production. Not before."

She looks at the terminal again. "Staging works. Now tell me whether we should keep sending GKE to that same Postgres over the internet."

Done When:

• checkConnection prints connected from a staging pod

• DATABASE_URL is set, and /mnt/secrets-store lists both secret names

• The gate prints 5 passed, 0 failed through the tunnel

• Staging is ClusterIP with no external IP, and the Elastic IP still answers aws

This is the checkpoint. Staging passed its gate, and GKE reaches the real database.

Next you prove it's safe, then prove it under load, before any customer traffic moves.

Part 5. How to Make GKE Production-Ready

Staging works. The promotion gate proved five things: the app runs, it reaches the database, a user can log in, a repeated payment charges once, and the balance drops by the right amount. That's a good result. It's not the same as being safe for customers.

Think of a new car that passed its first test drive. It starts and it turns. Before anyone rides in it, you still ask three more questions. Are the doors locked? Can you see the dashboard? What happens on a steep hill, or when a tire goes flat?

This part asks the same three questions about ClearPay:

These questions come now, on purpose. Customers are still on AWS, so if something breaks in this part, nobody notices and nobody loses money. Fixing it here costs you an hour. Fixing it after customers arrive costs you their trust, and every fix happens while people are watching.

Chapter 16. Run the Security Review

Wednesday, 2:00 p.m.

After lunch, Maya sends the next task. She has read the pipeline log.

Maya: "Before any customer touches GKE, I want a security check. I don't want to find a secret sitting in a file after launch."

Why This Matters

Every earlier check asked "does it work?". This one asks "what's exposed?". For a payments API, a review after launch comes too late.

Check 1. Traffic to the Database is Encrypted

The database connection crosses the public internet between clouds. Confirm the live staging process requires TLS, without printing the URL:

kubectl -n staging exec deployment/clearpay-api -- node -e "const u=process.env.DATABASE_URL||''; console.log(u.includes('sslmode=require') ? 'sslmode=require is set' : 'sslmode=require missing')"

sslmode=require is set

Check 2. Secret Rotation is Understood

api/src/routes/auth.js issues login tokens that last 12h. If you rotate JWT_SECRET, every token signed with the old one stops working, and users must log in again. For this lab that is acceptable. Write that sentence in your notes as the rotation policy. Don't automate rotation here.

Check 3. Audit Logs Exist

Google Cloud records who read secrets and who changed permissions, separate from the app's own logs. Confirm those audit entries exist:

gcloud logging read 'protoPayload.serviceName=("secretmanager.googleapis.com" OR "iam.googleapis.com")' --limit=5 --freshness=7d --format='table(timestamp,protoPayload.methodName)'

Look for method names that mention SecretManager or iam. Seeing only iam names, such as SetIAMPolicy, is fine. That proves Google is recording permission changes. The timestamps will differ.

Check 4. Each Identity Has Only What it Needs

Staging's account reads staging secrets. Production's account reads production secrets. The pipeline pushes images and deploys, and can't change the cluster.

PROJECT_ID=$(gcloud config get-value project)
gcloud secrets get-iam-policy clearpay-staging-database-url
gcloud secrets get-iam-policy clearpay-production-database-url
gcloud projects get-iam-policy "$PROJECT_ID" --flatten='bindings[].members' --filter='bindings.members:serviceAccount:clearpay-ci@' --format='table(bindings.role)'

What to look for:

• The staging secret lists clearpay-api-staging and clearpay-ci. The pipeline reads the staging pair during deploys.

• The production secret lists clearpay-api-production only.

• clearpay-ci has two project roles: artifactregistry.writer and container.developer.

If a production secret lists a staging account, or the pipeline has a role like owner or editor, stop and fix it in Stage 5 or Stage 6.

Done When:

• All four checks are written in your notes with the output you saw, not "probably fine"

Chapter 17. Watch ClearPay With Prometheus and Grafana (Optional)

Thursday, 9:30 a.m.

The security review is done. Daniel reads your notes, then turns his chair toward you.

Daniel: "Once we start moving traffic, I want to watch a graph instead of guessing. What will you look at?"

Why This Matters

Note that this chapter is optional. It doesn't touch AWS or GKE, and no later chapter needs it. Skip to Chapter 18 if you want. Do it if you want to learn how Prometheus and Grafana work.

The API exposes numbers at /metrics: request counts, latency, and database pool use. Prometheus collects those numbers every few seconds. Grafana draws them.

This chapter runs Prometheus and Grafana on your laptop with Docker Compose, pointed at the Chapter 1 API. It proves the dashboard works. It doesn't watch GKE or AWS. Those target files exist in observability/targets/ as .example files, but this lab doesn't use them. During the traffic shift, the command-line checks in Chapters 21 and 22 are the source of truth.

Google Cloud has its own error counter from Stage 7. That's a Cloud Logging metric in GCP, separate from this Grafana.

Now     localhost:4000  →  Prometheus  →  Grafana :3000
Later   AWS, GKE, load balancer  (observability/targets/*.example, optional)

Steps

Prometheus reads host.docker.internal:4000, which is the Chapter 1 API. If curl http://localhost:4000/health fails, start it:

docker compose up -d api postgres

Then start the monitoring stack:

cd observability
docker compose up

Open http://localhost:3000. Anonymous admin access is on for this laptop-only stack. Open the ClearPay - Migration Dashboard.

Check

Send the local API a few requests:

curl http://localhost:4000/health

The Request latency p95 by env panel shows a series named local. An empty panel means Prometheus can't reach port 4000.

Done When:

• The dashboard shows a local series

Chapter 18. Load-Test and Break Staging on Purpose

Thursday, 1:00 p.m.

Staging passed its gate, and you feel done. Daniel puts your Chapter 6 scorecard and the gate results side by side on his screen.

Daniel: "The gate only tested one login. What happens when twenty requests arrive at once, or when a pod dies?"

Run the Chapter 6 load test against staging, put the two scorecards side by side, then kill a pod and restart the Deployment while load is running.

Why This Matters

Customers will hit GKE with many requests at once, and Kubernetes will restart pods during updates and machine repairs. You want to see both happen now, with no customer involved.

The system under test runs on the normal node pool from Stage 3. If you ever run the load generator in the cloud, a cheap Spot VM is a good fit for it, since it can be interrupted without harm. The system you measure should never run on Spot, because an interruption would ruin the numbers.

Step 1. Load-Test Staging

Use two terminals, both at the repository root.

Terminal A holds the tunnel for this whole chapter. Leave it running:

kubectl -n staging port-forward svc/clearpay-api 8080:80

Look for Forwarding from 127.0.0.1:8080.

Terminal B: first confirm you reach GKE and that Postgres answers:

curl -s http://localhost:8080/health
curl -s http://localhost:8080/health/ready

health must include "platform":"gke". ready must include "status":"ready".

Then run the load test with 20 concurrent requests for 30 seconds:

./scripts/load-test.sh http://localhost:8080 \
  alice@clearpay.dev password123 20 30

Copy the scorecard into your notes, next to the AWS one. This lab's numbers:

                AWS (Chapter 6)   GKE staging (this chapter)
p95 latency:    745 ms            655 ms
Error rate:     0.0 %             0.0 %
Recovery time
after pod kill: n/a               (fill in after Step 3)

AWS ran with 10 concurrent requests and ok=372. GKE ran with 20 concurrent requests, ok=686, failed=0, and 22.9 requests per second.

GKE's p95 is a little lower than AWS's here. That might surprise you. Most of both numbers is the trip from your laptop to the cloud, and that trip is about the same size for both. Every database query on GKE also crosses from Google's us-east4 to AWS's us-east-1 through Cloud NAT, but that adds only a small part on top.

The two runs used different concurrency, 10 and 20, so treat the numbers as close, not as a race. The line that needs to stay clean is failed=0. Keep both latency numbers in mind. They come back at the end of the book.

Step 2. Read the served by Lines

Open the ClearPay dashboard on api.clearpay.test and look at Recent activity. Expect mostly aws lines and one or two gke lines. This lab looked like this:

-$1.00  served by aws/1.0.0        ← a send you made in the browser
-$1.00  served by gke/0.0.0-unset  ← the promotion gate from Chapter 15
-$5.00  served by aws/1.0.0        ← an older send

Each line names the process that wrote that transfer. Both clouds write to the same database, and the UI reads every row. Why so few gke lines?

• Your browser sends go to AWS, because the front door still sends 100 percent of traffic there.

• Only the promotion gate wrote through GKE, with a $1.00 payment through the tunnel. The load test only reads data, so it adds no lines.

A gke line doesn't mean the load balancer sent your page to GKE. It means a GKE pod handled that transfer.

The two names come from environment variables. aws/1.0.0 comes from the EC2 .env file. gke/0.0.0-unset comes from k8s/base/deployment.yaml, where APP_VERSION is a placeholder the pipeline doesn't change. The placeholder is expected, not a crash.

The black bar at the top shows the process answering right now. Recent activity shows history.

Step 3. Kill a Pod and Time the Recovery

Delete one API pod and time how long Kubernetes takes to replace it:

kubectl -n staging get pods -l app=clearpay-api
POD=$(kubectl -n staging get pod -l app=clearpay-api -o jsonpath='{.items[0].metadata.name}')
echo "deleting $POD"
START=$(date +%s)
kubectl -n staging delete pod "$POD"
kubectl -n staging wait --for=condition=Ready pod -l app=clearpay-api --timeout=120s
echo "recovery $(( $(date +%s) - START )) s"
kubectl -n staging get pods -l app=clearpay-api

kubectl delete pod waits until the old pod is gone, including a 5-second preStop pause in k8s/base/deployment.yaml. That pause lets in-flight requests finish before the process stops. The recovery line is the time from the delete until the new pod is ready. An earlier run of this lab printed 42 s. Yours will differ. Write your number in the table above.

If wait times out, run kubectl -n staging get pods and kubectl -n staging describe pod on the new pod name.

The Terminal A tunnel may drop when the pod it points at is deleted. If Terminal A exits, start the port-forward again before Step 4.

Step 4. Restart the Deployment Under Load

This step needs one change to scripts/load-test.sh. Without it, failed= reads 0 even when requests die. The script only counts requests that get an answer. When the pod restarts, the tunnel drops, requests get no answer, and the script stops that worker without counting a failure.

Open scripts/load-test.sh in your editor. Replace the whole hit() function with this one. Two things are new: 2>/dev/null || true on the curl line, and the if line at the end.

hit() {
  local path="$1" out code secs
  out=$(curl "${CURL_OPTS[@]}" -o /dev/null -w '%{http_code} %{time_total}' \
    -H "Authorization: Bearer $TOKEN" \
    "$BASE_URL$path" 2>/dev/null || true)
  code="${out%% *}"
  secs="${out##* }"
  echo "$code" >> "$CODES"
  node -e "process.stdout.write(String(Math.round(Number(process.argv[1]) * 1000)) + '\n')" "$secs" >> "$TIMES"
  if [ "$code" = "000" ]; then sleep 0.5; fi
}

000 is what curl reports when it got no answer. The short pause stops a failing worker from looping thousands of times a second.

Now run the load test in Terminal B. It runs for 40 seconds, so the restart fits inside it:

./scripts/load-test.sh http://localhost:8080 \
  alice@clearpay.dev password123 5 40

Before you press Enter on that command, type the restart commands in a third terminal and leave them ready. Within three seconds of starting the load test, press Enter in the third terminal:

kubectl -n staging rollout restart deployment/clearpay-api
kubectl -n staging rollout status deployment/clearpay-api
kubectl -n staging get events --sort-by=.lastTimestamp | tail

The events show old pods stopping and new pods starting, one at a time.

Staging lets one pod go down before its replacement starts (maxUnavailable: 1 and maxSurge: 0 in k8s/overlays/staging/patch-deployment.yaml). The small lab cluster has no room for an extra pod. So requests fail for a short time during this restart, and failed= above 0 is normal. Write down the count. Terminal A may print lost connection to pod. Stop it with Ctrl+C and start the tunnel again after the test.

Production is safer. Its base setting starts a new pod before an old one stops (maxSurge: 1 and maxUnavailable: 0).

Decision

Pass: failed=0 on the 30-second test, the replacement pod became ready after the delete, and rollout status finished. GKE's p95 being higher than AWS is still a pass on this path, as long as it stays under 3 times your AWS p95, the failure limit you wrote in Chapter 6.

Fail: 5xx errors in the 30-second test, a login internal_error, or a new pod that never becomes ready. Fix staging, push through the pipeline, and run this chapter again.

Done When:

• Both scorecards are in your notes side by side

• Pod recovery is timed

• The rolling restart finished while load was running, and you wrote down failed=

Stop the port-forward in Terminal A with Ctrl+C.

This is the checkpoint. GKE survived load and a pod kill. Every check so far happened with no customer in the path.

That changes now.

Part 6. How to Move Customers to GKE

This is the part where customers can land on GKE. Both clouds can serve a request. Every step from here asks one question: what evidence lets you move more traffic?

Chapter 19. Deploy Production and the GKE Gateway

Week three, Monday, 9:00 a.m.

This is the week customers can land on GKE. Nobody says it out loud. The office is a little quieter than usual.

So far, GKE only holds staging. Staging is a private copy that you reach through a tunnel. Customers will use a separate copy called production. This chapter builds that copy, and it connects the address customers will call.

Two things are missing:

1. The production namespace. A namespace is a separate area inside the cluster. Staging has one. Production doesn't have one yet, so no production pods exist. You will create it and put the same image in it that passed the staging gate.

2. Something behind the GKE edge IP. In Stage 8 you reserved a public address for GKE, called the edge IP. It works like a phone number you reserved but never connected: it rings, and nobody picks up. You will connect it to a Gateway. The Gateway answers each request and passes it to the production pods.

When you finish, GKE can answer real requests. You still don't move any traffic. The front door keeps sending 100 percent of customers to AWS until Chapter 21.

Why This Matters

The load balancer can only send traffic to GKE if something on GKE answers. That something is a Gateway: a Kubernetes object that tells Google to build a load balancer in front of a Service. Here it listens on clearpay-gke-edge-ip and forwards to the production API and UI.

The 08-edge load balancer reaches both clouds through Internet NEGs (network endpoint groups that point at an outside IP address). This type has no health checks. If a backend goes down, the load balancer keeps sending it traffic, and customers get 502 errors. Your protection is synthetic-health.sh, the decision gates, and rollback.sh.

Step 1. Turn on the Gateway API

The cluster from Stage 3 doesn't have it on:

gcloud container clusters update clearpay --zone us-east4-a \
  --gateway-api=standard

Wait until it finishes, then:

kubectl get gatewayclass

Look for gke-l7-global-external-managed. That's the class k8s/gateway/gke-gateway.yaml uses.

Step 2. Make Room on the Node Pool

The three e2-medium machines have about 940m of CPU each after GKE's own system pods. Staging and production together don't fit. Scale staging down to make room. Its images stay in Artifact Registry, and the gate result stays in your notes.

First write down the exact images staging runs. Production must run these same strings:

kubectl -n staging get deploy clearpay-api \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n staging get deploy clearpay-ui \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Then scale staging to zero:

kubectl -n staging scale deploy/clearpay-api --replicas=0
kubectl -n staging scale deploy/clearpay-ui --replicas=0

Don't push to main during the traffic shift. A push redeploys staging at full size and takes the CPU back.

If production API pods stay Pending with Insufficient cpu later, staging is still holding CPU. Don't raise the node pool's maximum unless you accept the cost of another machine.

Step 3. Deploy Production

Use the pipeline. It's the path Chapter 14 built for this moment.

With the pipeline:

1. On GitHub, open Actions, then Build, scan, and deploy in the left list. Start from this workflow page, not from a finished run. If you're looking at a single run, click the back arrow next to its name. The promote-to-production job on that old run shows "This job was skipped", and it stays skipped. You need a new run.

2. Click Run workflow. Keep the branch on main, tick promote_to_production, and click Run workflow.

3. The promote-to-production job waits with "Waiting for review". Click Review deployments, tick production, and click Approve and deploy.

4. Watch Find the currently-deployed staging images and Deploy those exact images to production finish with a check mark.

The job reads the image strings from the staging Deployment, even though staging now runs zero pods, and deploys those exact images.

By hand, if the pipeline isn't set up:

These are the same steps the job runs. PROJECT_ID in the files is a placeholder, so replace it the way the pipeline does. On a Mac, sed -i needs an empty '' after it:

cd k8s/overlays/production
PROJECT_ID=$(gcloud config get-value project)
sed -i '' "s/@PROJECT_ID.iam/@${PROJECT_ID}.iam/" patch-serviceaccount.yaml
sed -i '' "s#projects/PROJECT_ID/#projects/${PROJECT_ID}/#" secretproviderclass.yaml

On Linux, drop the '' after -i.

Point the overlay at the two images you wrote down in Step 2. Never use latest:

kustomize edit set image \
  REGION-docker.pkg.dev/PROJECT_ID/clearpay/api=STAGING_API_IMAGE
kustomize edit set image \
  REGION-docker.pkg.dev/PROJECT_ID/clearpay/ui=STAGING_UI_IMAGE
kubectl apply -k .
kubectl -n production rollout status deployment/clearpay-api --timeout=180s
kubectl -n production rollout status deployment/clearpay-ui --timeout=180s

Replace STAGING_API_IMAGE and STAGING_UI_IMAGE with the full strings from Step 2, in the form us-east4-docker.pkg.dev/.../api:<tag>.

Don't commit the files after sed and kustomize edit. They now contain your Project ID and a real tag. Run git checkout k8s/overlays/production from the repository root to undo them after the deploy.

Check Production

Return to the repository root. Wait until the pods are ready:

kubectl -n production get pods

API and UI pods show Running and READY 1/1. Then check the database from a production pod, not staging:

kubectl -n production exec deployment/clearpay-api -- node -e \
  "require('./src/db').checkConnection().then(() => { console.log('connected'); process.exit(0); }).catch(e => { console.error(e); process.exit(1); })"

connected

Step 4. Apply the GKE Gateway

Stage 8 reserved clearpay-gke-edge-ip and the certificate clearpay-gke-edge-cert. Now give them something to serve:

kubectl apply -f k8s/gateway/gke-gateway.yaml
kubectl apply -f k8s/gateway/httproute-production.yaml
kubectl apply -f k8s/gateway/healthcheck-api.yaml
kubectl -n production get gateway clearpay-gke-edge

Wait until PROGRAMMED shows True. That can take a few minutes while Google builds the load balancer.

The three files do three jobs:

• gke-gateway.yaml creates the Gateway on the reserved IP with the certificate.

• httproute-production.yaml sends requests to the production Service.

• healthcheck-api.yaml tells Google to health-check /health. Without it, Google checks /, the API answers 404, and requests fail with no healthy upstream.

Check the Gateway

Call GKE directly through its edge IP. --resolve makes curl send the hostname on the certificate while connecting to the GKE IP:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
EDGE=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw gke_edge_ip_address)
curl -k --resolve "$HOST:443:$EDGE" "https://$HOST/health"

Look for "platform":"gke". If curl prints no healthy upstream, wait a minute and retry. The health check needs time to mark the pods healthy.

Without --resolve, the same hostname still goes to AWS, because weights are 100 / 0.

Done When:

• kubectl get gatewayclass lists gke-l7-global-external-managed

• Production API and UI pods are Running, READY 1/1, with the staging image tags

• checkConnection prints connected from a production pod

• The Gateway shows PROGRAMMED True, and the edge IP returns "platform":"gke"

Chapter 20. Walk the Pre-Cutover Checklist

Monday, 3:00 p.m.

Production is up. Maya stops at your desk with her laptop under her arm.

Maya: "Before we move traffic, tell me exactly how we put customers back on AWS. Then show me it works."

Why the Rollback Drill Matters

scripts/rollback.sh does two things, in order:

1. It runs shift-traffic.sh 100 0, which sends all traffic back to AWS in seconds.

2. It connects to the AWS server over SSH, sets ROLLBACK_ACTIVE=true in its .env, and restarts the API. The UI then shows the rollback banner.

The dashboard has two different messages, on purpose:

• The new-version message is a small pop-up in the bottom right corner. It says "Noticed a change: this request ran on gke/... (previously aws/...)" and fades after five seconds. It appears when a different version answers.

• The rollback banner is a red-tinted strip across the top of the dashboard. It says "Rolled back to AWS, traffic has been restored while the GKE issue is investigated." It stays until you clear the flag.

Anyone looking at the app can tell "a new version shipped" apart from "something broke and we recovered".

The first time you run a rollback shouldn't be during a real incident, with customers failing, hoping the SSH key works. Run it now while nothing is wrong. Weights are already 100 / 0, so the traffic step changes nothing. The SSH step and the banner are what you test.

The Checklist

Tick each line with a command you ran, not from memory. Use your own Elastic IP and hostname.

Only four lines need a command now: GKE healthy, Rollback tested, AWS healthy, and Synthetic test running. The other five use results you already have, so read them and tick them. Chapters 21 and 22 don't repeat this checklist. They reuse only the synthetic test, which you leave running in its own terminal.

Pre-Cutover Checklist

[ ] GKE healthy
    kubectl -n production get pods
    (all Running, READY)

[ ] Database healthy
    the production checkConnection from Chapter 19 prints connected

[ ] Monitoring up (optional, only if you did Chapter 17)
    Grafana at http://localhost:3000 loads.
    It does not watch GKE. Do not block on an empty GKE series.

[ ] Rollback tested
    YOUR_ELASTIC_IP is your AWS server's public IP. Get it with:
    terraform -chdir=infra/aws/terraform output -raw instance_public_ip
    ./scripts/rollback.sh YOUR_ELASTIC_IP ~/.ssh/clearpay-deployer
    Open https://YOUR_ELASTIC_IP, log in, and look for the red strip
    at the top of the dashboard ("Rolled back to AWS ..."). Then clear it:
    ./scripts/clear-rollback-flag.sh YOUR_ELASTIC_IP ~/.ssh/clearpay-deployer

[ ] AWS healthy
    ./scripts/promotion-gate.sh https://YOUR_ELASTIC_IP \
      alice@clearpay.dev password123 bob@clearpay.dev
    5 passed, 0 failed

[ ] Synthetic test running
    ./scripts/synthetic-health.sh https://YOUR_PUBLIC_HOSTNAME
    Leave it running in its own terminal for the whole shift.
    YOUR_PUBLIC_HOSTNAME is api.clearpay.test. From the repository root, confirm with:
    terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname
    It prints one line per second: time, status code, latency.

[ ] Limits confirmed
    Chapter 6 limits, plus the Chapter 18 table:
    AWS p95 745 ms, GKE staging p95 655 ms, both 0.0 % errors, plus your pod recovery time from Chapter 18

[ ] Security review passed
    Chapter 16, all four checks written down

[ ] Cost ledger current
    infra/aws/COST.md and infra/gcp/COST.md, both clouds now billing

What the Rollback Drill Prints

The traffic weights are already 100 / 0, so step 1 changes nothing. Step 2 is the real test. It ends like this:

== Traffic split updated ==
"AWS: 100% / GKE: 0%"

== Step 2: flagging AWS as the active rollback target ==
AWS ROLLBACK_ACTIVE=true, service restarted.

== Rollback complete ==

The last lines of the script name old chapter numbers. Ignore them. The steps for a real rollback are in Chapter 23.

If step 2 stops with ssh: connect to host ... port 22: Operation timed out, your home IP has changed and the AWS firewall still allows the old one. Get your IP with curl -s ifconfig.me; echo. Put it plus /32 in ssh_allowed_cidr in infra/aws/terraform/terraform.tfvars. Run terraform apply in infra/aws/terraform, then run the script again.

Decision

Every line ticked, and the GKE edge IP returns "platform":"gke": go to Chapter 21.

Anything unticked: stop here. Don't run ./scripts/shift-traffic.sh 99 1 while the GKE edge IP is empty. One percent of customers would hit an address with nothing behind it.

Tell Maya: "Rollback is one script. It moves traffic in seconds and shows a banner. I ran it five minutes ago."

Done When:

• Every checklist line is ticked with output you saw

Chapter 21. Send 1 Percent of Traffic to GKE

Tuesday, 10:00 a.m.

The whole team is on one call. Chris has the support inbox open on his second screen. Amara has the transactions table open on hers. Daniel sits next to you with the load balancer page open.

Maya: "One percent to GKE. Watch the errors and the transfers. If they look wrong, send everything back."

Why 1 Percent?

One percent of requests. A request is one message from a browser to the API: a login, a balance check, a payment. Out of every 100 requests, about 99 go to AWS and about 1 goes to GKE. The load balancer picks for each request, not for each customer. So one customer can land on both clouds in a single visit.

One percent is enough to find a broken contract, such as a different JSON shape, a missing route, or a login that fails on one side. It's far too small to test capacity. Chapter 22 handles capacity.

Steps

./scripts/shift-traffic.sh 99 1

The script re-applies 08-edge with new weights on the load balancer's URL map. There's no DNS change and no deploy.

Now copy the same weights into infra/gcp/terraform/08-edge/terraform.tfvars:

aws_traffic_weight = 99
gke_traffic_weight = 1

The script passes the weights as command-line values. The next plain terraform apply in 08-edge reads the file. If the file still says 100 and 0, that apply would quietly move traffic back.

Confirm the file and the live state match:

grep traffic_weight infra/gcp/terraform/08-edge/terraform.tfvars
terraform -chdir=infra/gcp/terraform/08-edge output current_traffic_split

You should see aws_traffic_weight = 99, gke_traffic_weight = 1, and "AWS: 99% / GKE: 1%". If the file still says 100 and 0, fix it now.

Check: Did Any Request Reach GKE?

At 1 percent, almost every request still goes to AWS. Forty requests in a row that all say aws is normal. Loop until one says gke:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)
while true; do
  curl -sk --resolve "$HOST:443:$IP" "https://$HOST/health"
  echo
  sleep 0.2
done

Stop with Ctrl+C when you see "platform":"gke". One run in this lab took 200 requests: 197 aws and 3 gke.

To confirm GKE is still answering without waiting on chance, call its edge IP directly. This isn't a customer request:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
EDGE=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw gke_edge_ip_address)
curl -sk --resolve "$HOST:443:$EDGE" "https://$HOST/health"

Look for "status":"ok" and "platform":"gke". This lab printed:

{"status":"ok","served_by":{"platform":"gke","version":"0.0.0-unset","instance":"clearpay-api-6f9448c857-mj2j6","commit":"unset","rollback_active":false}}

Your instance will differ. If platform says aws, or the curl fails, GKE isn't answering. Don't raise the weight.

Check: Does a Login Work Through the Public Address?

From now on, one customer can land on AWS for one request and on GKE for the next. Both must accept the same request and give the same reply. If they differ, you want to find out now, at 1 percent, not later at 50 percent. Send a login through the public address:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)
curl -sk --resolve "$HOST:443:$IP" \
  -H 'Content-Type: application/json' \
  -d '{"email":"alice@clearpay.dev","password":"password123"}' \
  "https://$HOST/api/auth/login"

Look for "token" and "user" in the reply. If you see internal_error, stop and don't raise the weight.

Check: Use the App Like a Customer

Open https://api.clearpay.test and click through the certificate warning. Log in, check the balance, and send a small amount. Each new line under Recent activity says served by aws/1.0.0 or served by gke/0.0.0-unset. Only the newest line tells you about this request. Older lines keep the stamp from when they were written.

The black bar at the top will say AWS almost every time at 1 percent. Watch new activity lines and the loop, not the bar.

Glance at the synthetic-health.sh terminal. Every line should show status 200.

Decision

• The loop showed gke at least once, login returns a token, and synthetic health shows only 200: go to Chapter 22.

• Anything else: stop and investigate. Don't raise the weight. If customers are affected, run rollback.sh and go to Chapter 23.

Done When:

• Weights are 99 / 1 in both the file and current_traffic_split

• The loop printed "platform":"gke" at least once

• Login through the public address returns a token

Chapter 22. Raise Traffic to 100 Percent

Tuesday, 11:30 a.m., to Wednesday afternoon

One percent worked. That means 1 out of every 100 requests went to GKE. Now you turn the dial up, a little at a time. Nothing does it for you. You raise it by hand, and you look at the numbers before every step.

Check the same three things each time:

• The scorecard: how fast the app answers, and how many requests fail.

• The replica count: how many copies of the API are running.

• The connection math: whether the database still has room for every connection.

99 / 1     AWS ████████████████████░    GKE ░
90 / 10    AWS ██████████████████░░     GKE ██
75 / 25    AWS ███████████████░░░░░     GKE █████
50 / 50    AWS ██████████░░░░░░░░░░     GKE ██████████
25 / 75    AWS █████░░░░░░░░░░░░░░░     GKE ███████████████
10 / 90    AWS ██░░░░░░░░░░░░░░░░░░     GKE ██████████████████
 0 / 100   AWS                          GKE ████████████████████

Before 50 / 50: Do the Connection Math

Daniel, who is on the call, cuts in before you go to the halfway mark. "How many database connections can hit RDS if GKE scales all the way up while AWS is still live?"

Each API process can open up to DB_POOL_SIZE connections, ten by default. The pool opens them as load needs them, so the number stays low when traffic is light. At full load, every process can hold its full ten. The database has a hard limit, max_connections. When the total passes it, new connections are refused, requests wait, and customers see timeouts.

Get your database's limit. This runs psql on the AWS server, which already has it from Chapter 5:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
DATABASE_URL=$(terraform -chdir=infra/aws/terraform output -raw database_url)
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP "psql '$DATABASE_URL' -c 'SHOW max_connections;'"

Get GKE's ceiling from maxReplicas in k8s/overlays/production/patch-hpa.yaml. Then fill in this table with your numbers:

RDS max_connections:          ___
AWS, 1 process × 10:          10
GKE, maxReplicas × 10:        ___
Worst case, both at once:     10 + ___ = ___

This lab's filled-in table, so you can check yours:

RDS max_connections:          81
AWS, 1 process × 10:          10
GKE, maxReplicas × 10:        30    (3 pods × 10)
Worst case, both at once:     10 + 30 = 40

40 is under 81, so there's room to raise the traffic. Your max_connections may differ if your RDS instance is a different size. What matters is that your worst case stays below it.

Now run the same math for a bigger setup. A company with larger machines might run the HPA from 3 to 8 pods: 10 + 80 = 90 connections, against a small RDS instance that allows about 87.

Neither setting is wrong on its own. The pool size was set in the API code, and the pod ceiling was set in a Kubernetes file, by different people at different times. They only collide when both clouds run at full load at the same time, which is the exact situation a gradual migration creates.

If your worst case is above your max_connections, stop and read the fix options in Chapter 23 before you raise the weight.

Steps

Every stage has the same four moves. Do all four, then repeat with the next pair of numbers.

1. Shift: Run ./scripts/shift-traffic.sh with the new pair. The first number is the share for AWS, and the second is the share for GKE.

2. Save: Put the same two numbers in terraform.tfvars in 08-edge.

3. Check: Run the six checks in the next section. No single script does all of them. You use a few commands, plus two scripts: synthetic-health.sh, which is already running, and load-test.sh.

4. Decide: Every check passes: go to the next stage. Any check fails: stop.

The stages, in order:

Run one row at a time, and never run them all together.

Checks at Every Stage

1. The dial moved. The script's printout is Terraform's setting, not proof. Confirm the file matches:

grep traffic_weight infra/gcp/terraform/08-edge/terraform.tfvars
terraform -chdir=infra/gcp/terraform/08-edge output current_traffic_split

2. Real requests follow the dial. Wait three minutes after the shift, then count who answers:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)

for i in $(seq 1 40); do
  curl -sk --resolve "$HOST:443:$IP" "https://$HOST/health"
  echo
done | tee /tmp/clearpay-health.txt

grep -c '"platform":"aws"' /tmp/clearpay-health.txt
grep -c '"platform":"gke"' /tmp/clearpay-health.txt

The GKE count should rise as its weight rises. Forty requests are a rough sample, not a percentage proof. Google also takes a few minutes to apply a new split everywhere, so counting too early shows the old split. This lab saw:

90 / 10   80 requests:  75 aws,  5 gke
50 / 50   40 requests:  34 aws,  6 gke
25 / 75   40 requests:  17 aws, 23 gke
10 / 90   40 requests:   8 aws, 31 gke
 0 / 100  40 requests, 1 minute later:    3 aws, 37 gke
 0 / 100  40 requests, 90 seconds later:  0 aws, 40 gke

If the counts still don't follow the dial after ten minutes, ask Google which split it holds:

gcloud compute url-maps describe clearpay-migration-map | grep -B2 -A2 weight

At 0 / 100, the clearpay-gke-backend line shows weight: 100, and the AWS line shows no weight at all, because Google leaves out a zero. If the weights differ from your file, run terraform apply again in 08-edge.

3. GKE is healthy.

kubectl -n production get pods -l app=clearpay-api
kubectl -n production get hpa clearpay-api

Pods are Running and ready. The HPA line shows current CPU against the 70 percent target, and the current replica count. This lab stayed at 1 replica and 4 to 7 percent CPU through 25 / 75.

4. Logins still work. Run the Chapter 21 login curl again at every stage. A contract problem can appear only under a certain mix of traffic. Five failures in a row means stop, not raise.

5. Errors stay inside your limits. The synthetic-health.sh terminal should show only 200. If a line shows anything else, stop and don't raise.

6. The UI. Log in on https://api.clearpay.test, send money, and read the newest served by line.

The Decision at Every Stage

Maya asks the same question each time: "Are we ready for the next step?"

Answer with evidence: "The dial matches the file and GKE's share of requests rose. Pods are healthy, login works, and synthetic health is all 200. The worst-case connection count fits under the database limit."

If any check fails, don't raise. If customers are affected, go to Chapter 23 now.

Done When:

• Weights are 0 / 100 in the file and in current_traffic_split

• 40 requests in a row return "platform":"gke"

• The connection math is written down with your own numbers

• Synthetic health showed no sustained errors during any stage

Chapter 23. When the Numbers Go Bad

Wednesday, 4:12 p.m.

The last shift is done. Every customer is now on GKE, and the health check keeps printing 200, which means all is well. The support inbox has been quiet for the past hour. Amara checks the transactions table one more time.

"It looks boring," she says. Which is the best thing she could say.

You wait for someone to call it a successful migration.

Daniel is still looking at the deployment. He counts the API copies running in GKE, then looks at the database metrics.

"Can I tell you about the worst afternoon at my last job?"

Chris turns his chair around.

Daniel leans back.

"It was another migration for a different company, same basic problem though. We were moving customers from one cloud to another, and we'd reached about half of them by Friday afternoon. Traffic started climbing, so the new environment began adding more copies of the API."

He pauses for a moment.

"The old environment was still running too. It was serving the customers we hadn't moved yet."

You look at the deployment on the screen.

"So both environments were using the database?" "Yes. And every copy of the API needed database connections."

He points at the database on the diagram.

"That was where we got into trouble." Daniel reaches for a pen.

"Think about a restaurant with a fixed number of seats. The restaurant has enough seats for a certain number of people, but the number of seats doesn't change because more people arrive."

He draws two groups on the whiteboard.

"AWS was one group. GKE was the other. Both groups were using the same restaurant."

He added another group beside GKE.

"Then traffic increased. GKE added more copies of the API, and each new copy needed database connections. The old AWS copies were still holding their connections too."

He puts the pen down.

"The database had a limit. We kept increasing the number of connections going into it."

"What happened when you reached the limit?" you ask.

"Requests started waiting for connections. Then they started timing out."

He looks at you.

"The application itself wasn't obviously broken. If the pods were running and health checks were passing, the problem was capacity. We had more demand for database connections than the database could provide."

Chris looks back at the deployment.

"So you were watching the new environment, but the database was dealing with both environments."

"Exactly."

Daniel looks at the diagram again.

"That was the part we missed. We calculated what the new environment needed. We didn't calculate what the database needed while both environments were running."

He sits back down.

"We eventually moved the traffic back to the old environment and recovered. But the incident stayed with me."

You look at the GKE deployment. "What would you do differently now?"

"I'd work through the numbers before moving traffic."

Daniel turns his laptop toward you.

"Your lab is safe. You ran 3 copies against a database limit of 81 connections, and the workload never came close to exhausting it."

He opens another set of notes.

"My incident was different. We had up to 8 copies, and the database allowed 87 connections."

He looks at you. "So let's use those numbers."

You pull your chair closer. Daniel continues.

"I'm going to give you the events in the order they happened. You're on call. Don't assume you know the ending. Tell me what you would check and what you think is happening as each event comes in."

He opens a terminal.

The drill begins with 8 API copies available across the 2 environments and a database connection limit of 87.

This time, the numbers are going to matter.

Do you run anything in this chapter? Read it once, like a story with a lesson. Your lab had no incident, so there's nothing to fix. If you want proof that you're in the clear, run the two commands in Step 2 and compare them with "Your own cluster". If the logs show no errors and the HPA is at 1 replica, you're done. Go on to Chapter 24.

What it Looks Like

The first few minutes after the 50 / 50 shift look normal. Then the response times begin to creep up.

Amara notices it first in the team channel.

"A couple of transfers just timed out."

Daniel opens the production dashboard. The API has scaled to seven replicas, and the p95 latency is climbing.

Chris posts two customer reports about slow transfers.

Daniel looks at the graph again. "Is this affecting every request?"

"Not yet," Chris says. "A few transfers so far."

Amara checks the payment records. "Let's make sure those timeouts didn't result in duplicate charges."

Daniel looks at the database metrics. "Could be the database."

You don't know yet.

Nothing in the logs says, "The database is out of connections." The first signal is simply that requests are taking longer than they were a few minutes ago.

No error message says "the database is out of connections". All you see first is the slowest responses (p95 latency) getting slower. The status bar in the app (ui/src/components/StatusRibbon.jsx) switches to "Degraded. Investigating" only after several health checks fail in a row, so one slow request doesn't trigger it.

What You Know

Traffic is on both clouds. Latency is worse than the AWS baseline. The status bar flipped after several failed checks, not one blip.

What You Don't Know Yet

Whether this is the load balancer, the pods, the database, or a bad deploy. So don't assume. Gather info first.

Step 1. Stop Shifting

Don't run shift-traffic.sh toward GKE again until you understand what happened. Write down the time you noticed.

Record what you see before you do anything else: the actual error messages, the actual replica count, and the actual timestamp.

Every hour spent diagnosing at a stalled percentage is also an hour of dual-run cost. Add it to the cost ledger. Don't let it drop out of the running total because everyone's attention is elsewhere.

Step 2. Gather Evidence

Read the API's errors:

kubectl -n production logs deployment/clearpay-api --tail=100 | grep error

Look for words about the database: connection errors, "too many connections", or a timeout while waiting for a client from the pool. A 5xx error with no database words points somewhere else, such as the Gateway or a bad deploy.

Check how many pods Kubernetes is running:

kubectl -n production get hpa clearpay-api

Compare REPLICAS with maxReplicas. A climbing count means load went up. That's a clue, not yet a cause.

Then fill in the Chapter 22 table with the live numbers:

RDS max_connections:          87
AWS, 1 process × 10:          10
GKE, 7 replicas × 10:         70
Right now:                    80, close to the limit
GKE at maxReplicas 8:         80, worst case 90, over the limit

Your Own Cluster at 50 / 50

Run the same two commands on your cluster before you assume you're in Daniel's story. This lab at AWS 50 / GKE 50 showed:

HPA:       cpu 4%/70%, MINPODS 1, MAXPODS 3, REPLICAS 1
pods:      1 Running
logs:      no error lines in the last 100
RDS:       max_connections=81
           two sessions on database clearpay
status:    not Degraded

Worst case here is up to 40, under 81. Don't run rollback.sh unless your numbers are worse than your Chapter 6 limits.

If your HPA is still at 1 and the logs are quiet, write that down. That's the evidence. Stay with it.

How the Failure Builds

AWS alone would never hit this. GKE alone would never hit this. It happens only while both run at full load, and the migration plan keeps both running on purpose.

Tell Daniel: "Replica count climbed. The pool math puts us near the database limit. This is capacity, not Kubernetes."

Step 3. Roll Back

Maya: "What do you recommend?"

Compare the numbers with the limits you wrote in Chapter 6. Don't invent a new limit now.

"Put customers back on AWS. Errors are past our limit, customers are affected, and rollback is tested."

Maya: "Okay. Do it."

./scripts/rollback.sh <aws-instance-ip> ~/.ssh/clearpay-deployer

The diagram above shows two actions performed by rollback.sh. First, shift-traffic.sh 100 0 sends all customer traffic back to AWS and removes traffic from GKE. Second, the script connects to the AWS EC2 server over SSH, sets ROLLBACK_ACTIVE=true in the API's environment file, and restarts the API service. The API then reports AWS as the active platform and enables the distinct “Rolled back to AWS” banner in the ClearPay UI. These steps restore traffic and make the rollback visible to the operator.

Set terraform.tfvars in 08-edge back to 100 and 0 so a later apply doesn't undo the rollback.

Step 4. Confirm the Rollback Worked

curl -k https://api.clearpay.test/api/info

platform reads aws and rollback_active reads true. Refresh the UI: the red "Rolled back to AWS" banner appears, and the status bar goes back to All systems operational once enough health checks pass.

Then check each of these from evidence, not assumptions:

• Requests stopped failing (synthetic-health.sh shows 200 again)

• Latency is back near the Chapter 6 AWS numbers

• The database is consistent: a transfer from just before the rollback shows exactly one debit, not zero and not two

Amara: "I don't care that traffic is back on AWS. Did the transfer happen once?"

Answer from the database, through the transaction history, not from the load balancer. The idempotency key from Chapter 3 is what makes a retried transfer safe here, even when the first try went to GKE and the retry went to AWS.

Write down one number for the incident review: the minutes from "noticed" to "traffic back on AWS".

Step 5. Fix the Cause

Daniel: "What will you change so that number can't happen again?"

There are four real options, each with a cost:

• Lower DB_POOL_SIZE per pod: Fewer connections per process. Cheap and fast. Each pod can serve fewer database queries at the same time.

• Lower the HPA's maxReplicas in k8s/overlays/production/patch-hpa.yaml. Caps the total. GKE can no longer scale as far under a real traffic spike.

• Raise RDS max_connections with a custom parameter group. Each connection uses database memory, and a small instance like db.t3.micro can fail to start if you set it too high. Check AWS's guidance for your instance size first.

• Put a connection pooler in front of Postgres, such as PgBouncer or RDS Proxy. Many app connections share a few database connections. The best long-term fix, and the most work to set up.

This book doesn't pick one for you. Write down which one you chose and what you gave up.

For the smallest real change, lower the pool size. The math with 6 per pod: 8 × 6 = 48 from GKE, plus 10 from AWS, is 58. That fits under 87 even at the full ceiling.

In an emergency you can set it live:

kubectl -n production set env deployment/clearpay-api DB_POOL_SIZE=6

A live change leaves no record, and the next deploy erases it. Make the real change through the pipeline: set DB_POOL_SIZE in k8s/base/deployment.yaml, commit, and push to main. The pipeline builds, scans, and deploys staging. Load-test staging hard enough to push the HPA to its ceiling, using Chapter 18's steps with higher concurrency. A fix that only passes under light load proves nothing. Then run the promotion gate, and promote with the Chapter 19 pipeline job.

Step 6. Resume

Maya: "Are we ready for 10 percent?"

"Yes. The API is healthy, synthetic checks are clean, errors are within the limit, the pool math holds at the ceiling, and rollback is tested."

Daniel: "Agreed."

Maya: "Your call."

Clear the rollback flag first:

./scripts/clear-rollback-flag.sh <aws-instance-ip> ~/.ssh/clearpay-deployer

Refresh the UI. The rollback banner is gone. Leaving the flag on would keep the banner showing after recovery, which trains people to ignore it.

Then repeat Chapters 21 and 22 from the start:

./scripts/shift-traffic.sh 99 1
# observe, compare, record, decide

./scripts/shift-traffic.sh 90 10
# observe, compare, record, decide

./scripts/shift-traffic.sh 75 25
./scripts/shift-traffic.sh 50 50
./scripts/shift-traffic.sh 25 75
./scripts/shift-traffic.sh 10 90
./scripts/shift-traffic.sh 0 100

Watch the HPA and the error lines closely at the stage where the incident happened last time. That stage is where the fix proves itself.

Add a line to the cost ledger. A second pass through the shift means more hours with both clouds billing. Keep that cost visible instead of folding it into one "migration cost" number.

The Skills Behind This Chapter

The diagram above shows the incident response process from observation to recovery. Start by checking latency, the status bar, and error logs. Form a hypothesis, test it against the database connection limit, and use live measurements as evidence. If customers are affected, stop shifting traffic and roll back. Fix the cause through the pipeline, test under maximum expected load, and resume the migration only when the evidence supports it.

An incident isn't a hunt for the answer. It's the work of reducing uncertainty until you can make a safe decision. If customers are failing, the safe decision is to stop and roll back. If they aren't, the safe decision is to keep watching. Don't roll back because a number looks odd, and don't keep shifting because the pods look healthy.

This is the checkpoint. GKE is serving 100 percent.

AWS is a priced safety net now, not a leftover you forgot.

Part 7. How to Finish the Migration

GKE serves every customer. AWS still runs, and still bills, as a safety net. This part turns the cutover into evidence, puts a date on the safety net, and removes what you no longer need.

Chapter 24. Prove the Cutover With Numbers

Thursday, 9:00 a.m.

Maya has a meeting with the company leaders at ten. She stops at your desk with her laptop open to an empty page.

Maya: "The leaders weren't on any of our calls, and they'll ask one thing: did customers notice the move? I need a short summary I can put on the screen. How many requests went through, how many failed, how fast the app answers compared with AWS, and whether anyone saw an outage. Can you have it before ten?"

You nod. Every number she asked for comes from two scripts you already know.

Steps

Run the two checks that produce hard numbers against the public address, now at 100 percent GKE:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
./scripts/promotion-gate.sh "https://$HOST" \
  alice@clearpay.dev password123 bob@clearpay.dev

Then run load-test.sh against https://$HOST with the Chapter 6 settings, and copy the scorecard. Add the counts from your Chapter 22 loops and the synthetic-health.sh log.

The Evidence

Below is this lab's table. Yours will have different numbers:

Production requests (counted checks, not a full dashboard total):
  40 /health one minute after 0 / 100:  3 aws, 37 gke
  40 /health ninety seconds later:      0 aws, 40 gke
  load-test.sh 10 × 20s:                ok=310  failed=0
  promotion-gate.sh:                    5 passed, 0 failed
Successful requests:                 310 + 40 + 5 in those runs
Failed requests:                     0
5xx errors:                          0.0 %
p95 latency (GKE, 100%):             647 ms
  Chapter 6 AWS baseline was 164 ms.
  Chapter 18 GKE staging (port-forward) was 1157 ms.
Sustained customer-visible outage:   0 seconds

Local Prometheus didn't watch GKE, so this table uses counted checks. Don't invent a dashboard total you never measured.

Check the table against the limits you wrote in Chapter 6, before GKE existed:

• Outage limit: no stretch of 5xx errors above your limit. This lab had zero failed requests.

• Failure limit: p95 under your ceiling. This lab's 647 ms is under the example ceiling of 1500 ms, so it passes.

Passing isn't the whole story. 647 ms is about four times the AWS 164 ms. Customers never saw an outage, but every request got slower. Write that down plainly instead of hiding it. The last section of this book explains why.

Why the Table Doesn't Say "Zero failed requests, ever"

No real system running for hours, across traffic shifts and a possible rollback, can honestly promise that. Claiming it would weaken the one claim that matters: no sustained outage a customer could see.

A few failed requests inside the limit you set in advance is the honest version of "this went well". The standard is controlled and measured, not perfect.

Done When:

• Every line in your table holds a number you measured

• Each line is checked against your Chapter 6 limits, including any that failed

Chapter 25. Run on GKE and Set a Date for AWS

Friday, 10:00 a.m.

Maya comes back from her meeting with a question from the leaders.

Maya: "So what changed for the customer?"

You look at the numbers. Traffic is 100 percent GKE. Transfers succeed. Synthetic checks all pass. PostgreSQL is still on AWS. The EC2 server serves nobody.

"The address stayed the same and nobody saw an outage."

Maya: "Then this system is yours now. And I want a calendar date for turning off the old server."

Step 1. Watch it Run

Let production run for a real window before you commit to anything. For a real company that means days, not the minutes after the last shift.

This lab's 100 percent window lasted minutes, with a 5/5 gate and failed=0. Write that down. A weekend lab isn't a week of production.

Step 2. Check the Alert

Stage 7 created the alert policy ClearPay API - elevated error rate, which fires on more than five error lines in five minutes. It has no notification channel, so nobody gets told when it fires. An alert that reaches nobody isn't an alert.

For a real system, add a notification channel in Cloud Monitoring and trigger a test on staging, never by sending errors to customers. For this lab, write down the gap: the policy exists, and nobody gets paged.

Step 3. Review Capacity

Look at the HPA over the window:

kubectl -n production get hpa clearpay-api

This lab stayed at 1 replica and 4 to 7 percent CPU, even at 15.5 requests per second during the 100 percent load test. One pod was enough. Three e2-medium machines for one API pod is more than this traffic needs.

Google offers Committed Use Discounts (CUDs): a lower price in exchange for promising to use a set amount of machine capacity for one or three years.

Commit only after you have a stable usage history. A commitment based on the architecture you planned, instead of the usage you measured, is a guess with a contract attached. This lab bought none, because a few hours of GKE isn't a usage history.

Step 4. Write the Handover

Write this for someone who wasn't part of the migration. This lab's version:

Logs:       kubectl -n production logs deployment/clearpay-api
Dashboard:  http://localhost:3000 if observability/docker-compose.yml
            is up. GKE series may be empty.
Alert:      Cloud Monitoring "ClearPay API - elevated error rate"
            No notification channel.
Rollback:   scripts/rollback.sh, only while EC2 still exists.
Hostname:   terraform output public_hostname
            terraform output static_ip_address
            (yours, not a hostname copied from this book)

Step 5. Put a Date on AWS

scripts/rollback.sh works only while the EC2 server exists, and the server bills every day it exists. Decide how long it stays, write down why, and price it:

AWS daily cost:              ~$1.25/day (infra/aws/COST.md)
Retention window:            ___ days
Decommission date:           ___
Total safety-margin cost:    ___ days × $1.25 = $___

This lab used 0 extra days and retired AWS compute the same day 100 percent GKE was proven, because it's a teardown lab.

A real company might keep AWS for one or two weeks. A shorter window saves money. A longer one buys more confidence. The mistake is leaving it open as "keep it around just in case", which costs money every day and never gets revisited.

Write the decommission rules before the date arrives, so the decision doesn't get made under deadline pressure:

• a set number of days at 100 percent GKE with no rollback

• no alert fired for a GKE-caused problem in that window

• a named person signs off

This lab's rules: weights 0 / 100, gate 5/5, load test failed=0, HPA at 1, no rollback.

Last, check that nothing else still points at the Elastic IP directly: bookmarks, hardcoded URLs in other systems, or outside uptime monitors set up before the load balancer existed.

Done When:

• You watched production for a written, real time window

• The alert gap is written down

• Capacity is reviewed and the CUD decision is written down

• The handover note exists

• The retention window, decommission date, cost, and rules are written down

Chapter 26. Retire the AWS Server

Friday, 3:00 p.m.

The date has arrived, and the rules from Chapter 25 are met. Remove the EC2 server and keep the database. It was never part of this migration.

Steps

Perform your final check, from the repository root. GKE must pass the gate on the public address:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
./scripts/promotion-gate.sh "https://$HOST" \
  alice@clearpay.dev password123 bob@clearpay.dev

Then confirm that Terraform tracks the server you mean to remove:

cd infra/aws/terraform
terraform state show aws_instance.api   # confirm you have the right resource

Next, return to the repository root with cd ../../.., then remove the AWS compute resources only:

./scripts/teardown-aws-compute.sh
# Pass --yes to add -auto-approve after you have read a plan once.

After this step, rollback to AWS is gone. That's the point of the decision you made in Chapter 25.

Read the Destroy Plan

The script destroys only these:

• the EC2 instance

• the Elastic IP

• the security group rule that let the API reach the database

• the API security group

• the SSH key pair

• the CloudWatch alarm

RDS stays.

The database rule is its own Terraform resource, aws_security_group_rule.db_from_api, and the script removes it first. If that rule lived inside the database security group, AWS would refuse to delete the API security group while the database still referenced it.

A full terraform destroy in infra/aws/terraform would also delete PostgreSQL. Don't use it here.

Clean Up Credentials

• The SSH key pair in AWS is gone with the script. Delete the local private key ~/.ssh/clearpay-deployer if you used it only for this server.

• Remove anything else that pointed at the old server, such as saved hosts or scripts with the old IP.

• Remove any CI/CD secrets or variables that referenced the old server.

Update the Diagram

The dual-cloud diagram from Chapter 7 is now wrong. The new picture is GKE alone behind the load balancer, with the database still on AWS, as Chapter 7 said the project would end.

Done When:

• The gate passed on the public address before the teardown

• AWS compute is destroyed, and RDS still exists

• The local deploy key is removed or kept on purpose

• Your architecture diagram matches what runs

Chapter 27. Tear Down GCP and the Database

Friday, 5:30 p.m.

The office is emptying out. Two clouds are still billing you by the hour.

This is a lab. Once your notes are complete, remove everything that still bills: the Gateway load balancer, the GKE machines, Cloud NAT, the registry, and the RDS database.

Before tearing anything down, complete the migration record in Chapter 28. It asks you to document the final architecture, migration results, performance numbers, and total cost. If you still need evidence from the running system, collect it now. Once you destroy the infrastructure, those live measurements will no longer be available.

Steps

From the repository root:

./scripts/teardown-gcp.sh
# Pass --yes to skip the prompt and -auto-approve each stage.

The script:

1. Deletes the Gateway and the production and staging workloads. Kubernetes creates Google load balancer pieces for the Gateway, and Terraform doesn't know about them. Deleting the Gateway first lets Google remove them. Skip this and Terraform can get stuck on resources it didn't create.

2. Runs terraform destroy in reverse stage order: 08-edge → 07-observability → 06-secrets → 05-identity → 04-registry → 03-gke → 02-network → 01-foundation. Later stages depend on earlier ones, so they come down first.

Type yes at the script's prompt, then read and confirm each Terraform plan.

Namespaces can sit in Terminating for a few minutes. The script waits up to 180 seconds for each delete, then moves on. Destroying the cluster removes them anyway.

Then remove the database. The migration ended with it on AWS on purpose, but a lab shouldn't keep paying about $0.80 per day for it:

cd infra/aws/terraform
terraform destroy

Last, delete the api.clearpay.test line from /etc/hosts.

The book never created or deletes your GCP project. Don't delete a project unless it held only this lab.

Done When:

• kubectl get ns no longer shows staging or production, or the cluster is gone

• Every infra/gcp/terraform/0*-* stage is destroyed

• terraform destroy finished in infra/aws/terraform

• GCP and AWS billing show nothing from this lab still running

This is the checkpoint. The work is over.

Write down what actually happened, including the bill, before you forget.

Chapter 28. Write the Migration Record

Week four, Monday, 9:00 a.m.

Write down what happened, the architecture, the bill, and what you would change, before the details fade.

Part 1. The Architecture Record

Before:

User → nginx (TLS) → systemd-managed Node process → RDS Postgres

One EC2 server, one database, a default network nobody designed, secrets in a plain .env file, no autoscaling, and a deploy process run by hand.

During the migration:

After:

Then one short paragraph for each model:

• Traffic: weighted backends on a global external Application Load Balancer, changed by shift-traffic.sh. Not DNS, because cached DNS answers would have made rollback slow.

• Deploys: GitHub Actions builds, scans, and deploys every commit to staging. Production gets the exact staging image, only after the promotion gate and a reviewer's approval.

• Database: never moved. The same RDS instance from Chapter 4 stayed the single source of truth the whole time.

• Security: Workload Identity for both the pods and the pipeline. No downloaded service account key existed at any point. Secrets come from Secret Manager.

• Observability: the API's own JSON logs and /metrics endpoint, the same on AWS and GKE, plus a log-based error metric in Cloud Logging.

• Rollback: one script that moves the traffic weight and sets a flag on AWS, tested before the first shift.

Part 2. The Postmortem

The four of you go through the notes together. Maya opens with the question everyone half expects.

Maya: "Was Kubernetes the problem?"

"Not really. It did what we told it to do. The thing to watch was the database. Every copy of the API can open ten connections, GKE can run several copies, and AWS was still serving customers at the same time. Add those up and you get close to the database's limit.

On this small cluster we topped out at forty connections against eighty-one, so nothing broke. On a cluster that scales to eight copies, the same sum goes over the limit."

Daniel nods. "That's what I missed at my last company. All the numbers were there, and nobody put them side by side. You did it before any traffic moved."

Amara: "And the payments? Was every transfer charged once?"

"Yes. The idempotency key from Chapter 3 held through every shift. We even replayed a payment at 100 percent GKE, and there was no second charge."

Maya closes her laptop. "So what would we do differently next time?"

Write these sections:

What worked:

• Idempotency keys, built in before GKE existed, made it safe for a customer's retry to land on a different cloud than the first try.

• The staged Terraform meant each failure pointed at one stage.

• The pipeline meant production ran the exact image that passed the gate.

• Rollback was tested in Chapter 20, before it was ever needed.

What went wrong in this lab:

• Local Prometheus never watched GKE, so Chapter 24 used counted checks instead of a dashboard total.

• The alert policy has no notification channel.

• The small node pool couldn't run the pod counts a real company would use, so the connection-limit risk was checked on paper, not seen live.

• GKE's p95 was about four times AWS's.

The root cause: Two reasonable settings, DB_POOL_SIZE=10 and the HPA's maxReplicas, were chosen in different places. Nobody checked their product against the database limit until the migration ran both clouds at once. The fix isn't complicated once you find it. Finding it takes treating Chapter 23 as a real investigation instead of skipping ahead to an answer.

Decisions and their trade-offs:

• GKE Standard over Autopilot: full control of node pools and autoscaling, at the cost of having to understand and size them. This lab paid for three mostly idle machines. Autopilot might have hidden this connection-limit risk inside its own scheduling, for better or worse.

• Two environments instead of five: less to build and pay for, at the cost of staging sharing the production database. Fine at this scale, not for every company.

• A public database endpoint with an IP allowlist instead of a VPN: no setup time and no monthly VPN bill. The cost is that RDS has a public endpoint, even if it accepts only one IP and requires TLS.

• Leaving the database on AWS: a simpler, safer migration, at the cost of every query crossing between clouds.

So what would change in a second migration? Before any traffic moves, compute the worst-case connection count across every environment that shares the database. Compare it with the real limit. Make it a checklist item, not a live discovery.

Part 3. The Cost Postmortem

This lab's version:

Budget: about $25 for five days of GCP usage.

Actual dual-run: about $4.30 total, $1.25 AWS + $3.05 GCP. The GCP $3.05 was covered by free-trial credit, so the actual GCP charge was $0.

The migration stayed on schedule. No incident, rollback, or second shift was needed.

The cluster was over-provisioned: three GKE nodes supported one API pod, with CPU at 4–7%.

Standard GKE was chosen for the teaching value. Autopilot would have been cheaper for this workload.

No CUD was purchased.

Conclusion: The migration stayed well within budget, but the GKE setup used more capacity than the workload needed.

Fill in your own numbers from your cloud bills. This book can't do that part for you, and a guessed number would be the same false precision this whole project avoided.

Done When:

• The before, during, and after diagrams match what ran

• The postmortem states the root cause clearly

• The cost postmortem holds your real numbers

What Comes After This Project

What Question This Project Leaves Open

Chapter 13 flagged it and moved on, on purpose: at what point does keeping PostgreSQL in AWS cost more, in risk and in engineering time, than migrating it? The cross-cloud trip, the dependency on two providers, and the connection-pool pressure from Chapter 23 all sit on the same side of that scale.

Put two numbers from your notes side by side:

AWS, Chapter 6:             p95 164 ms
GKE at 100%, Chapter 24:    p95 647 ms

The API moved and customers never saw an outage. But every request got about four times slower.

The code didn't change. The load test did not change. What changed is distance. On AWS, the API and the database sat in the same data center. On GKE, every query leaves Google's us-east4, goes out through Cloud NAT, crosses the internet to AWS's us-east-1, and comes back. A single transfer runs several queries, so it pays that trip several times. The Google load balancer adds one more hop in front.

Maya reads the table. "Why is it slower?"

"The database stayed on AWS. Every query crosses between clouds."

"Was that a mistake?"

"No. Chapter 7 made the call to move the API alone, because moving two things at once would have made any failure much harder to trace. A problem could have come from the API, the network, or the database move itself. But the call has a price, and now we can measure it."

Maya: "Then that's the next project."

This project doesn't answer the open question. It was never supposed to. The next project moves PostgreSQL to Google Cloud, most likely with Google's Database Migration Service, which copies data continuously so the final switch takes minutes instead of hours. It will use the same method as this book: a baseline first, a written limit, small steps, a tested rollback, and evidence at every stage. When it's done, run the same load test and compare the same two numbers.

Again, that's a separate project, with its own book. This one ends here: ClearPay's API on GKE, its database still on AWS, exactly as Chapter 7 said it would.

What You Should Have at the End

• Working fintech API and minimal customer UI (api/, ui/)

• AWS deployment (infra/aws/)

• GCP architecture, staged Terraform (infra/gcp/terraform/)

• Docker images and a CI/CD pipeline that builds, scans, deploys staging, and promotes the tested image (.github/workflows/deploy.yml)

• Kubernetes manifests, two-environment Kustomize overlays (k8s/base/, k8s/overlays/)

• Database connectivity, validated and never migrated (Chapter 13)

• Secret management, upgraded from a .env file to Secret Manager (k8s/overlays/*/secretproviderclass.yaml, secretsync.yaml)

• A GKE Gateway behind a weighted load balancer (k8s/gateway/, infra/gcp/terraform/08-edge/)

• Observability dashboard on your laptop, if you did the optional Chapter 17 (observability/)

• Load-testing, promotion-gate, and synthetic health scripts (scripts/)

• Migration scorecard (Chapter 6, Chapter 18, Chapter 24)

• Cost ledger, start to finish (Chapter 6 through Chapter 28)

• Traffic-shifting procedure (scripts/shift-traffic.sh)

• Rollback procedure (scripts/rollback.sh, scripts/clear-rollback-flag.sh)

• Architecture diagrams (Chapters 4, 7, 9, 11, 13, 14, 19, 28)

• Migration runbook (this book)

• Technical postmortem (Chapter 28)

• Cost postmortem (Chapter 28)

The ClearPay API moved from one AWS EC2 server to GKE. PostgreSQL is still on AWS, on purpose. Customers kept one address while you shifted traffic, practiced a rollback, and finished the cutover.

Maya asked what changed for the customer. The address didn't. The balance didn't. Nobody saw an outage. Now you understand why that was the hard part, and why 647 milliseconds is the next one.

The lab covered how to prove each stage, how to read evidence, and how to undo a bad shift. Destroy anything you're still paying for.

Subscribe to my newsletter for new technical articles, project walkthroughs, and upcoming releases.

How to Discuss This Project in an Interview

I worked on a cloud migration project for a payments application called ClearPay. The application was running on AWS, and the goal was to move it to Google Cloud without disrupting access to the service.

The main challenge was making that move safely. Instead of switching everything at once, I designed a controlled migration where traffic could be moved gradually to the new environment. If something went wrong, I could redirect traffic back to the original system. I also kept the database on AWS to reduce the scope of the migration and avoid introducing unnecessary risk.

I set up the deployment process, security checks, and verification steps needed to confirm that the application was working correctly before completing the switch.

I then tested the final migration. The tests completed without failed requests, and the application didn't produce server errors during the final checks. I also identified a performance difference between the old and new environments that would need further investigation before considering the migration fully optimized.

The project demonstrates that I can plan and execute a cloud migration, manage the risks involved, verify the outcome, and identify issues that still need attention rather than simply declaring success because the application is running.

Interview tip: Be prepared to explain why you used weighted traffic routing, how you validated the application during cutover, what triggered a rollback, and why PostgreSQL remained on AWS.

Staging is shared. One environment, many pull requests. Someone deploys a half-finished feature to test it. Someone else deploys on top of that. A third person runs a database migration. By Thursday afternoon, nobody can tell which version of which branch is actually running, and the sentence "it works on staging" has stopped carrying any information at all.

The workarounds are familiar: a Slack channel where people claim staging like a meeting room, a rotating deploy schedule, or a second staging environment, then a third, each one drifting further from production configuration than the last.

There's a better pattern, and platform teams at companies like Vercel, Netlify, and Shopify have made it standard: give every pull request its own environment.

Open a PR, and a fresh, isolated copy of your application deploys automatically. Push another commit, and it updates. Merge or close the PR, and the whole thing deletes itself. No queue, no claiming, no cleanup, and no drift.

That's what we're building in this tutorial. Here's what it looks like from the reviewer's side. A bot comment appearing on the pull request with a live, working URL

What You'll Build

By the end of this handbook, you'll have a working system where:

• Opening a pull request triggers a GitHub Actions workflow that builds a container image tagged specifically for that PR

• The image is pushed to Google Artifact Registry

• A dedicated Kubernetes namespace (pr-42, pr-43, and so on) is created inside a GKE cluster, with resource quotas applied before anything deploys

• The application deploys into that namespace, completely isolated from every other PR's environment

• A shared Ingress controller routes pr-42.preview.yourdomain.com to that namespace, with a valid TLS certificate issued automatically

• A bot comments the live URL on the pull request, updating the same comment on every push

• Pushing new commits redeploys the environment with the updated code

• Closing or merging the PR deletes the namespace and everything inside it

• A scheduled job catches any environment the teardown missed

You'll also build the underlying infrastructure with Terraform, with three key design decisions, each with its own section.

1. No service account keys: GitHub Actions authenticates to Google Cloud through Workload Identity Federation. Short-lived OIDC tokens verified against a repository claim. Service account JSON keys sitting in GitHub secrets are one of the most common security problems in CI/CD pipelines today, and you don't need one.

2. One load balancer, not one per PR: A single shared Ingress controller routes every environment by hostname, and one wildcard DNS record covers every hostname that will ever exist. The naïve version provisions a cloud load balancer per environment, which gets expensive quickly.

3. Cleanup that assumes failure: Teardown runs when a PR closes. But a scheduled reaper also deletes namespaces whose PR is gone or that have outlived a maximum age, because even 95% effective automated cleanup still leaks, and leaks show up on the bill.

Here's the architecture we're working toward:

The diagram traces two separate paths through the same system.

The solid blue path is automation. A pull request triggers GitHub Actions, which authenticates to Google Cloud through Workload Identity Federation, pushes a PR-tagged image to Artifact Registry, and deploys into a dedicated namespace inside the GKE cluster.

Each namespace (pr-42, pr-43, and so on) holds its own Deployment, Service, and Pod, completely isolated from every other one.

Two more automated pieces sit alongside it: a bot that posts and updates the preview URL on the pull request, and a scheduled reaper that finds and deletes orphaned namespaces.

The dashed green path is public traffic. A browser request for a preview URL resolves through a single wildcard DNS record to one Google Cloud load balancer, which hands it to a shared NGINX Ingress controller. That controller reads the request's Host header and routes it to the right namespace. cert-manager runs beside it, requesting certificates from Let's Encrypt so each environment gets its own valid TLS certificate.

The important detail is what doesn't multiply. One load balancer, one DNS record, and one Ingress controller serve every environment, no matter how many pull requests are open.

Table of Contents

• What You'll Build

• Prerequisites

• Step 1: Set Up the Project Structure

• Step 2: Write the Application

• Step 3: Write the Dockerfile

• Step 4: Provision the Infrastructure with Terraform

• Step 5: Set Up Keyless Authentication with Workload Identity Federation

• Step 6: Apply the Terraform Configuration

• Step 7: Prove the Lifecycle Manually

• Step 8: Write the Kubernetes Manifests with Kustomize

• Step 9: Build the Spin-Up Workflow

• Step 10: Build the Teardown Workflow

• Step 11: Open a Pull Request and Watch It Work

• Step 12: Reserve a Static IP Address

• Step 13: Install the Ingress Controller

• Step 14: Point Wildcard DNS at the Load Balancer

• Step 15: Install cert-manager and Create a ClusterIssuer

• Step 16: Add an Ingress to the Base Manifests

• Step 17: Patch the Hostname Per Pull Request

• Step 18: Add Guardrails with ResourceQuota and LimitRange

• Step 19: Update the Deploy Workflow

• Step 20: Comment the URL on the Pull Request

• Step 21: Build the Reaper

• Step 22: Test the Whole Thing

• What This Costs

• Cleanup

• Conclusion

• Project File Structure

Prerequisites

Before you start, make sure you have the following ready:

• A Google Cloud project with billing enabled. If you don't have one, create it at console.cloud.google.com. Running this tutorial costs roughly $5–10 depending on how long you leave the cluster up. Don't skip the cleanup section at the end.

• gcloud CLI installed and authenticated. Run gcloud auth login to sign in, and gcloud config set project YOUR_PROJECT_ID to point it at your project.

• Terraform version 1.5 or later. Check with terraform version.

• kubectl installed. This is the CLI for interacting with Kubernetes clusters.

• Docker installed locally, for testing the build before you hand it to CI.

• A GitHub repository that you own. This needs to be a repository you can push branches to and open pull requests against, since that's the trigger for everything we build.

• Basic familiarity with Docker (images, layers, Dockerfile), Kubernetes (pods, deployments, services, namespaces), and GitHub Actions (workflows, jobs, steps). You don't need to be an expert, but you should know what these things are.

• Helm version 3, for installing the Ingress controller and cert-manager. Check with helm version.

• A domain name you control, with the ability to add DNS records. It doesn't matter where it's registered. You'll dedicate one subdomain to previews, like preview.yourdomain.com, and every environment lives beneath it.

A note on the domain requirement: this is the part of the project that can't be faked. Let's Encrypt needs to verify that you control the domain before it will issue a certificate, and it verifies this by making a real HTTP request from the public internet. A .local domain, an entry in /etc/hosts, or a domain you don't own won't work.

If you don't have a spare domain, a .dev or .xyz costs a few pounds a year and is worth having for exactly this kind of project.

One more thing worth bringing your attention to early: you'll also need the gke-gcloud-auth-plugin. Recent versions of gcloud prompt you to install it the first time you connect to a GKE cluster. You can install it ahead of time with:

gcloud components install gke-gcloud-auth-plugin

Step 1: Set Up the Project Structure

Let's lay out the directories first, so you have a clear mental model of where everything lives.

mkdir -p pr-preview-envs/{app,terraform,k8s/base,.github/workflows}
cd pr-preview-envs
git init

Here's what each directory holds:

• app/: the Go application and its Dockerfile

• terraform/: infrastructure as code: the GKE cluster, Artifact Registry, and Workload Identity Federation setup

• k8s/base/: the base Kubernetes manifests that every preview environment is built from

• .github/workflows/: the two GitHub Actions workflows that create and destroy environments

Connect this to a GitHub repository now, because the Workload Identity Federation setup in Step 5 needs to know your repository's full name (owner/repo):

git remote add origin https://github.com/YOUR_USERNAME/pr-preview-envs.git

Step 2: Write the Application

We need something to deploy. We'll use Go, for three reasons that matter to this project specifically.

First, Go compiles to a single statically-linked binary. There's no runtime to install and no interpreter, which means very small container images, and small images mean fast pushes and fast pulls, which matters a lot when you're building an image on every single commit to every open PR.

Second, Go's -ldflags -X mechanism lets us inject values into the binary at build time. We'll use this to bake the PR number and Git commit SHA directly into the application, so the running app can tell us exactly which build it is.

And lastly, build times are fast, and CI minutes aren't free.

Create app/main.go:

package main

import (
	"fmt"
	"net/http"
	"os"
)

// These values are empty at compile time and get injected by the
// linker via -ldflags -X during the Docker build. See the Dockerfile.
var (
	prNumber = "local"
	gitSha   = "dev"
)

func handler(w http.ResponseWriter, r *http.Request) {
	// Kubernetes injects these via the Downward API — see deployment.yaml
	namespace := os.Getenv("POD_NAMESPACE")
	podName := os.Getenv("POD_NAME")

	if namespace == "" {
		namespace = "not-in-kubernetes"
	}
	if podName == "" {
		podName, _ = os.Hostname()
	}

	fmt.Fprintf(w, "Preview environment is live\n\n")
	fmt.Fprintf(w, "PR number  : %s\n", prNumber)
	fmt.Fprintf(w, "Git SHA    : %s\n", gitSha)
	fmt.Fprintf(w, "Namespace  : %s\n", namespace)
	fmt.Fprintf(w, "Pod name   : %s\n", podName)
}

func healthz(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	fmt.Fprintln(w, "ok")
}

func main() {
	http.HandleFunc("/", handler)
	http.HandleFunc("/healthz", healthz)

	fmt.Println("Server starting on port 8080...")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		fmt.Fprintf(os.Stderr, "server error: %v\n", err)
		os.Exit(1)
	}
}

Initialize the Go module from inside app/:

cd app
go mod init preview-app

Let's walk through what this code does and why each piece is here.

var prNumber = "local" and var gitSha = "dev" are package-level string variables with default values. The defaults are what you'll see if you run the binary locally without going through the Docker build.

During the CI build, the Go linker overwrites them with the real PR number and commit SHA. This is the single most useful debugging feature in the whole project: when you hit a preview environment, the app itself tells you exactly which commit it was built from. There's no guessing.

os.Getenv("POD_NAMESPACE") and os.Getenv("POD_NAME") read environment variables that Kubernetes injects into the container using something called the Downward API. We'll configure that in the deployment manifest in Step 8.

The Downward API lets a pod read metadata about itself (its own name, namespace, and labels) without needing any Kubernetes API credentials. It's the cleanest way for an app to know where it's running.

The handler function is registered on the root path /. It writes the PR number, commit SHA, namespace, and pod name to the response. It's four pieces of information that together answer "what exactly am I looking at right now?"

The healthz function is registered on /healthz, and returns HTTP 200 with the text ok. Kubernetes uses this endpoint for liveness and readiness probes, which we'll wire up in the deployment manifest.

http.ListenAndServe(":8080", nil) starts the server on port 8080. If it fails to start, it prints the error and exits with a non-zero code, so Kubernetes knows the container is broken and can restart it.

Test it locally before moving on:

go run main.go

In a second terminal:

curl http://localhost:8080

You should see PR number : local and Git SHA : dev, since you're running outside the Docker build. That's the expected local behaviour.

Step 3: Write the Dockerfile

Create app/Dockerfile:

# -----------------------------------------------------------
# Stage 1: Build
# -----------------------------------------------------------
FROM golang:1.27-alpine AS builder
#Ensure the version matches what is in your go.mod file

# These are injected by the CI pipeline at build time
ARG PR_NUMBER=local
ARG GIT_SHA=dev

WORKDIR /app

# Copy go.mod first so Docker can cache the dependency layer
COPY go.mod .
RUN go mod download

COPY main.go .

# -w -s strips debug symbols (~30% smaller binary)
# -X overwrites package-level string variables at link time
RUN CGO_ENABLED=0 go build \
    -ldflags="-w -s -X main.prNumber=${PR_NUMBER} -X main.gitSha=${GIT_SHA}" \
    -o server main.go

# -----------------------------------------------------------
# Stage 2: Runtime
# -----------------------------------------------------------
FROM alpine:latest

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

WORKDIR /app
COPY --from=builder /app/server .

EXPOSE 8080
CMD ["./server"]

There are a few things happening here worth understanding properly.

1. The -ldflags -X Injection

-X main.prNumber=${PR_NUMBER}

The -X linker flag sets the value of a package-level string variable at link time. The syntax is -X importpath.name=value. Since our variables live in package main, the import path is just main.

This only works on string variables that are already declared in the source. You can't create new variables this way, and you can't inject into an int or a bool. That's why prNumber is a string in main.go even though it holds a number.

Why do this instead of just passing an environment variable through the deployment manifest? Because build metadata belongs to the image, not the deployment. If you pass it as an env var, someone can deploy image abc123 with an env var claiming it's commit def456, and the app will happily lie to you. Baking it into the binary means the image can only ever report what it actually is.

The CGO_ENABLED=0 Flag

By default, Go can link against C libraries. When it does, the resulting binary depends on the system's C library being present at runtime. Alpine uses musl instead of the more common glibc, and that mismatch is a classic source of "works in the builder stage, crashes in the runtime stage" errors.

Setting CGO_ENABLED=0 forces a pure-Go, fully static binary with no external library dependencies. It will run on any Linux system, including a scratch image with nothing in it at all.

COPY go.mod . Before COPY main.go .

Docker builds images layer by layer and caches each layer. By copying only go.mod first and running go mod download before copying the source code, we create a cached dependency layer.

On subsequent builds (and remember, this pipeline builds on every commit to every open PR), Docker skips the dependency download entirely as long as go.mod hasn't changed. On a real project with dozens of dependencies, this is the difference between a 20-second build and a 3-minute build, multiplied by every push your team makes.

The Non-Root User

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

Containers run as root by default. If an attacker finds a vulnerability in your application, running as root inside the container gives them significantly more room to escalate. Creating an unprivileged user and switching to it limits the blast radius. It costs you two lines, and there's no reason not to do it.

Build and test it locally:

docker build --build-arg PR_NUMBER=99 --build-arg GIT_SHA=abc1234 -t preview-app:test .
docker run -p 8080:8080 preview-app:test

Then curl http://localhost:8080 in another terminal. You should now see PR number : 99 and Git SHA : abc1234. The injection works.

Step 4: Provision the Infrastructure with Terraform

Now we'll build the cloud infrastructure. We're using Terraform rather than a series of gcloud commands for a reason that becomes obvious the moment something goes wrong: with Terraform, terraform destroy cleans up everything you created, in the right order, with no orphaned resources quietly billing you.

Create terraform/variables.tf:

variable "project_id" {
  description = "GCP project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "europe-west2"
}

variable "zone" {
  description = "GCP zone. A zonal cluster is cheaper than a regional one."
  type        = string
  default     = "europe-west2-a"
}

variable "cluster_name" {
  description = "Name of the GKE cluster"
  type        = string
  default     = "pr-preview-cluster"
}

variable "github_repo" {
  description = "GitHub repository in owner/repo format, e.g. amina/pr-preview-envs"
  type        = string
}

variable "machine_type" {
  description = "Node machine type"
  type        = string
  default     = "e2-standard-2"
}

variable "node_count" {
  description = "Number of nodes in the pool"
  type        = number
  default     = 2
}

Now create terraform/main.tf:

terraform {
  required_version = ">= 1.5"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

# Look up the project so we can reference its numeric project number,
# which Workload Identity Federation requires (it doesn't accept the
# human-readable project ID).
data "google_project" "current" {}

# -----------------------------------------------------------
# Enable required APIs
# -----------------------------------------------------------
resource "google_project_service" "apis" {
  for_each = toset([
    "container.googleapis.com",
    "artifactregistry.googleapis.com",
    "iamcredentials.googleapis.com",
    "sts.googleapis.com",
  ])

  service            = each.value
  disable_on_destroy = false
}

# -----------------------------------------------------------
# Artifact Registry — where preview images live
# -----------------------------------------------------------
resource "google_artifact_registry_repository" "preview" {
  location      = var.region
  repository_id = "pr-preview"
  description   = "Container images for PR preview environments"
  format        = "DOCKER"

  depends_on = [google_project_service.apis]
}

# -----------------------------------------------------------
# GKE cluster
# -----------------------------------------------------------
resource "google_container_cluster" "primary" {
  name     = var.cluster_name
  location = var.zone

  # Create the cluster with a throwaway default pool, then remove it
  # and manage nodes through a separate node pool resource. This is the
  # standard pattern — it lets you resize or upgrade the node pool
  # without recreating the cluster control plane.
  remove_default_node_pool = true
  initial_node_count       = 1

  networking_mode = "VPC_NATIVE"
  ip_allocation_policy {}

  # Set to true before you use this for anything real.
  deletion_protection = false

  depends_on = [google_project_service.apis]
}

resource "google_container_node_pool" "primary_nodes" {
  name       = "${var.cluster_name}-pool"
  location   = var.zone
  cluster    = google_container_cluster.primary.name
  node_count = var.node_count

  node_config {
    machine_type = var.machine_type
    disk_size_gb = 50

    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform",
    ]

    labels = {
      purpose = "pr-preview-envs"
    }
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }
}

There are a few things that're worth calling out here.

data "google_project" "current" isn't creating anything. It's a data source, which reads existing information from your provider. We need it because Workload Identity Federation identifies your project by its numeric project number, not the project ID you type in the console. Rather than making you look it up manually and paste it in, Terraform fetches it for you.

for_each on google_project_service: instead of writing four nearly identical resource blocks to enable four APIs, for_each iterates over a set and creates one resource per item. disable_on_destroy = false means terraform destroy won't turn these APIs off, which is what you want, because other things in your project might be using them.

remove_default_node_pool = true: GKE insists on creating a default node pool when you create a cluster. Setting this immediately deletes it, and we define our own node pool as a separate resource. The benefit is that node pool changes (resizing, changing machine type, upgrading) don't force a change to the cluster resource itself.

node_count = 2 with e2-standard-2: two nodes with 2 vCPU and 8GB each. Preview environments are small, but you'll have several running at once, plus GKE's own system pods. Two nodes give you comfortable headroom. If you plan to keep more than about eight preview environments alive simultaneously, bump this up.

deletion_protection = false: GKE added this as a guard against accidentally destroying production clusters. We're disabling it because this is a tutorial cluster you'll delete at the end. If you adapt this for real use, set it to true.

Step 5: Set Up Keyless Authentication with Workload Identity Federation

This is the section that matters most, so let's take it slowly.

The Problem With Service Account Keys

GitHub Actions needs permission to push images to Artifact Registry and deploy to your GKE cluster. The way most tutorials handle this is:

1. Create a service account in GCP

2. Generate a JSON key for it

3. Paste that JSON into a GitHub secret called GCP_SA_KEY

4. Load it in the workflow

This works, but it's also a security problem, and here's why:

First, the key never expires. A service account JSON key is a long-lived credential. It's valid until someone explicitly revokes it, which in practice means it's valid forever, because nobody remembers to rotate them.

Second, the key is a bearer token. Anyone who obtains that JSON file is your service account. No second factor, IP restriction, or expiry. Copy the file, and use it from anywhere in the world.

Third, the blast radius is enormous. GitHub secrets are exposed to workflows. A compromised third-party action, a malicious dependency in your build, or a workflow modified in a pull request can exfiltrate that secret. Once it's out, an attacker has persistent access to your cloud project.

This isn't theoretical. Leaked service account keys are one of the most common root causes in cloud security incidents.

How Workload Identity Federation Fixes It

Workload Identity Federation removes the long-lived key entirely. Here's the flow:

1. GitHub Actions generates a short-lived OIDC token for the running workflow. This token is signed by GitHub and contains verified claims: which repository is running, which branch, which workflow, and which actor.

2. Your workflow presents that token to Google Cloud's Security Token Service.

3. Google checks the token's signature against GitHub's public keys, and checks the claims against rules you configured, specifically that the token came from your repository and no other.

4. If everything matches, Google issues a short-lived access token, typically valid for about an hour.

5. The workflow uses that token, and it expires shortly after.

There's no secret stored anywhere. Nothing to leak, rotate, or revoke. If someone forks your repository and runs the same workflow, the OIDC token they generate will carry their repository name in its claims, Google's check will fail, and they get nothing.

Add this to terraform/main.tf:

# -----------------------------------------------------------
# Service account that GitHub Actions will impersonate
# -----------------------------------------------------------
resource "google_service_account" "github_actions" {
  account_id   = "github-actions-preview"
  display_name = "GitHub Actions — PR Preview Environments"
}

# Permission to push images to Artifact Registry
resource "google_project_iam_member" "artifact_writer" {
  project = var.project_id
  role    = "roles/artifactregistry.writer"
  member  = "serviceAccount:${google_service_account.github_actions.email}"
}

# Permission to create namespaces and deploy workloads in GKE.
# container.developer grants full access to Kubernetes API objects,
# but NOT permission to modify the cluster itself.
resource "google_project_iam_member" "gke_developer" {
  project = var.project_id
  role    = "roles/container.developer"
  member  = "serviceAccount:${google_service_account.github_actions.email}"
}

# -----------------------------------------------------------
# Workload Identity Federation
# -----------------------------------------------------------
resource "google_iam_workload_identity_pool" "github" {
  workload_identity_pool_id = "github-pool"
  display_name              = "GitHub Actions Pool"
  description               = "Identity pool for GitHub Actions OIDC"

  depends_on = [google_project_service.apis]
}

resource "google_iam_workload_identity_pool_provider" "github" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.github.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-provider"
  display_name                       = "GitHub OIDC Provider"

  # Map claims from the GitHub OIDC token to Google attributes
  attribute_mapping = {
    "google.subject"       = "assertion.sub"
    "attribute.actor"      = "assertion.actor"
    "attribute.repository" = "assertion.repository"
  }

  # CRITICAL: without this condition, ANY GitHub repository on the
  # internet could authenticate against your pool.
  attribute_condition = "assertion.repository == '${var.github_repo}'"

  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }
}

# Allow tokens from your specific repo to impersonate the service account
resource "google_service_account_iam_member" "github_impersonation" {
  service_account_id = google_service_account.github_actions.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "principalSet://iam.googleapis.com/projects/${data.google_project.current.number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.github.workload_identity_pool_id}/attribute.repository/${var.github_repo}"
}

Let's break down the parts that trip people up.

attribute_mapping: GitHub's OIDC token contains a set of claims. This block tells Google which claims to read and what to call them internally. assertion.sub is the token's subject (something like repo:owner/name:ref:refs/heads/main). assertion.repository is the full repository name. google.subject is required, while the others are optional attributes you can then write rules against.

attribute_condition: read this line twice, because getting it wrong is a serious vulnerability:

attribute_condition = "assertion.repository == '${var.github_repo}'"

Without this condition, your identity pool trusts the GitHub OIDC issuer, which means it trusts every repository on GitHub. Anyone who knows your pool's resource path (which isn't secret and appears in workflow files) could authenticate and impersonate your service account.

This condition narrows the trust to exactly one repository. Google now requires an attribute condition when you map assertion.repository, precisely because so many people got this wrong.

principalSet://...: this is the identity being granted the workloadIdentityUser role. Read it right to left: any principal in the github-pool whose attribute.repository equals your repo may impersonate this service account. Note that it uses data.google_project.current.number, which is the numeric project number, not the project ID. Using the project ID here produces an error that doesn't clearly explain what's wrong, and it's one of the most common mistakes with WIF.

Why roles/container.developer and not roles/container.admin? container.developer grants full access to Kubernetes API objects inside clusters: creating namespaces, applying deployments, deleting resources. container.admin additionally allows modifying the clusters themselves: resizing node pools, upgrading, deleting the cluster entirely.

Our workflow never needs to touch the cluster's own configuration, so it shouldn't have permission to. This is least privilege in practice: grant exactly what the job requires and nothing beyond it.

Now create terraform/outputs.tf:

output "cluster_name" {
  value = google_container_cluster.primary.name
}

output "cluster_zone" {
  value = var.zone
}

output "artifact_registry_url" {
  description = "Base URL for pushing images"
  value       = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.preview.repository_id}"
}

output "workload_identity_provider" {
  description = "Paste this into the GitHub Actions workflow"
  value       = "projects/${data.google_project.current.number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.github.workload_identity_pool_id}/providers/${google_iam_workload_identity_pool_provider.github.workload_identity_pool_provider_id}"
}

output "service_account_email" {
  description = "Paste this into the GitHub Actions workflow"
  value       = google_service_account.github_actions.email
}

output "get_credentials_command" {
  value = "gcloud container clusters get-credentials ${google_container_cluster.primary.name} --zone ${var.zone} --project ${var.project_id}"
}

Those outputs aren't decoration. workload_identity_provider and service_account_email are the two values you'll paste into your GitHub Actions workflow, and both are tedious to construct by hand.

Step 6: Apply the Terraform Configuration

From inside the terraform/ directory:

terraform init

This downloads the Google provider and sets up the backend. You should see "Terraform has been successfully initialized!"

Now preview what will be created:

terraform plan \
  -var="project_id=YOUR_PROJECT_ID" \
  -var="github_repo=YOUR_USERNAME/pr-preview-envs"

Read the plan output. Terraform tells you exactly what it intends to create before it creates anything. You should see roughly a dozen resources to add and none to change or destroy.

If that looks right:

terraform apply \
  -var="project_id=YOUR_PROJECT_ID" \
  -var="github_repo=YOUR_USERNAME/pr-preview-envs"

Type yes when prompted. Cluster creation takes about five minutes.

Tip: typing those -var flags repeatedly gets old fast. Create a file called terraform.tfvars containing project_id = "your-project-id" and github_repo = "you/pr-preview-envs", and Terraform picks it up automatically. Add it to .gitignore.

When it finishes, Terraform prints your outputs. Save these, as you'll need them in Step 9:

artifact_registry_url = "europe-west2-docker.pkg.dev/your-project/pr-preview"
service_account_email = "github-actions-preview@your-project.iam.gserviceaccount.com"
workload_identity_provider = "projects/123456789/locations/global/workloadIdentityPools/github-pool/providers/github-provider"

You can print them again any time with terraform output.

Connect kubectl to the new cluster using the command from the get_credentials_command output:

gcloud container clusters get-credentials pr-preview-cluster \
  --zone europe-west2-a \
  --project YOUR_PROJECT_ID

Verify:

kubectl get nodes

You should see two nodes in Ready state.

Step 7: Prove the Lifecycle Manually

Before automating anything, do it by hand once.

This isn't busywork. Automation is fundamentally just a script running commands you could have typed. If you don't know what the commands are, you can't debug the automation when it fails...and it will fail, usually at 11 pm before a demo. Ten minutes here saves hours later.

Create a namespace:

kubectl create namespace pr-test

Deploy a placeholder into it:

kubectl create deployment hello \
  --image=nginx:1.27-alpine \
  --namespace=pr-test

Check it:

kubectl get pods --namespace=pr-test

Now the important part – prove the isolation:

kubectl get pods --all-namespaces | grep hello

The pod exists only in pr-test. Nothing in default or any other namespace knows about it. That's the property the entire project is built on: a namespace is a boundary, and everything inside one PR's namespace is invisible to every other.

Now tear it down:

kubectl delete namespace pr-test

Watch what happens:

kubectl get namespace pr-test

For a few seconds, you'll see Terminating, then the namespace disappears entirely. Deleting a namespace deletes every resource inside it: deployments, pods, services, config maps, secrets, everything. One command, complete cleanup.

That single property is why namespace-per-PR is such a good fit for preview environments. Teardown isn't a careful sequence of deletions you have to get right. It's one command that can't leave anything behind.

Step 8: Write the Kubernetes Manifests with Kustomize

Every preview environment deploys the same application. The only differences between pr-42 and pr-43 are the namespace and the image tag.

This is exactly the problem Kustomize solves. You write the manifests once as a base, then apply small, declarative changes on top of it (an overlay) without templating, string substitution, or sed scripts mangling your YAML.

Kustomize is built into kubectl (that's the -k flag), so there's nothing extra to install.

Create k8s/base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: preview-app
  labels:
    app: preview-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: preview-app
  template:
    metadata:
      labels:
        app: preview-app
    spec:
      containers:
        - name: preview-app
          # This is a placeholder. The overlay replaces it with the
          # real registry path and PR-specific tag.
          image: preview-app
          ports:
            - containerPort: 8080
          env:
            # Downward API: the pod learns its own identity
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 5
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              cpu: "200m"
              memory: "128Mi"

Create k8s/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: preview-app
spec:
  selector:
    app: preview-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: ClusterIP

Create k8s/base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

There are some deliberate choices here.

replicas: 1: production runs multiple replicas for availability. A preview environment doesn't need high availability. One replica per environment, multiplied across every open PR, keeps your node count and your bill sane.

image: preview-app: a bare placeholder with no registry and no tag. It would never pull as-is. Kustomize's image transformer matches on this name and replaces it wholesale, so the base manifest never needs to know your registry URL.

The Downward API env block: this is where POD_NAMESPACE and POD_NAME come from. fieldRef reads a field from the pod's own spec at runtime and exposes it as an environment variable. No API calls, credentials, or service account needed.

type: ClusterIP: internal-only. Each LoadBalancer service provisions a real Google Cloud load balancer with a real hourly cost and a public IP. Doing that per PR would be expensive and wasteful. We'll reach the app with kubectl port-forward for now, and in Step 13 a single shared Ingress controller takes over, serving every environment through one load balancer.

Resource requests and limits: requests reserve capacity so the scheduler knows how much room a pod needs. limits cap what it can consume. Without limits, one runaway preview environment could starve every other one on the node. With many short-lived environments coming and going, this isn't optional.

Test the base renders correctly:

kubectl kustomize k8s/base

That prints the rendered YAML without applying it. This is useful for debugging.

Step 9: Build the Spin-Up Workflow

Now we connect everything.

Create .github/workflows/preview-deploy.yml:

name: Deploy PR Preview

on:
  pull_request:
    types: [opened, synchronize, reopened]

# If someone pushes three commits quickly, cancel the older runs.
# Without this, three deploys race each other into the same namespace
# and whichever finishes last wins — which may not be the newest commit.
concurrency:
  group: preview-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  id-token: write   # REQUIRED for Workload Identity Federation
  pull-requests: write

env:
  PROJECT_ID: YOUR_PROJECT_ID
  REGION: europe-west2
  CLUSTER_NAME: pr-preview-cluster
  CLUSTER_ZONE: europe-west2-a
  REPOSITORY: pr-preview
  IMAGE_NAME: preview-app

jobs:
  deploy:
    name: Build and deploy preview
    runs-on: ubuntu-latest

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set environment variables
        run: |
          PR_NUMBER=${{ github.event.pull_request.number }}
          SHORT_SHA=$(echo "${{ github.event.pull_request.head.sha }}" | cut -c1-7)

          echo "PR_NUMBER=${PR_NUMBER}"   >> $GITHUB_ENV
          echo "SHORT_SHA=${SHORT_SHA}"   >> $GITHUB_ENV
          echo "NAMESPACE=pr-${PR_NUMBER}" >> $GITHUB_ENV
          echo "IMAGE_TAG=pr-${PR_NUMBER}-${SHORT_SHA}" >> $GITHUB_ENV
          echo "IMAGE_BASE=${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.IMAGE_NAME }}" >> $GITHUB_ENV

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.WIF_SERVICE_ACCOUNT }}

      - name: Set up gcloud
        uses: google-github-actions/setup-gcloud@v2

      - name: Configure Docker for Artifact Registry
        run: gcloud auth configure-docker ${{ env.REGION }}-docker.pkg.dev --quiet

      - name: Build and push image
        run: |
          docker build \
            --build-arg PR_NUMBER="${PR_NUMBER}" \
            --build-arg GIT_SHA="${SHORT_SHA}" \
            -t "${IMAGE_BASE}:${IMAGE_TAG}" \
            ./app

          docker push "${IMAGE_BASE}:${IMAGE_TAG}"

      - name: Get GKE credentials
        uses: google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: ${{ env.CLUSTER_NAME }}
          location: ${{ env.CLUSTER_ZONE }}

      - name: Create namespace
        run: |
          # --dry-run=client | apply makes this idempotent.
          # Running it on an existing namespace is a no-op, not an error.
          kubectl create namespace "${NAMESPACE}" \
            --dry-run=client -o yaml | kubectl apply -f -

      - name: Generate Kustomize overlay
        run: |
          mkdir -p k8s/overlays/"${NAMESPACE}"
          cat > k8s/overlays/"${NAMESPACE}"/kustomization.yaml <<EOF
          apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization

          namespace: ${NAMESPACE}

          resources:
            - ../../base

          images:
            - name: preview-app
              newName: ${IMAGE_BASE}
              newTag: ${IMAGE_TAG}
          EOF

          echo "--- Rendered manifest ---"
          kubectl kustomize k8s/overlays/"${NAMESPACE}"

      - name: Deploy
        run: kubectl apply -k k8s/overlays/"${NAMESPACE}"

      - name: Wait for rollout
        run: |
          kubectl rollout status deployment/preview-app \
            --namespace="${NAMESPACE}" \
            --timeout=180s

      - name: Show deployment details
        run: |
          echo "Namespace: ${NAMESPACE}"
          kubectl get pods,svc --namespace="${NAMESPACE}"

Add the Two Secrets

Go to your repository on GitHub and go to Settings, Secrets and variables, Actions, and New repository secret. Add:

Neither of these is a credential. The provider path and service account email are identifiers, not secrets. They grant nothing on their own, because the OIDC token and the attribute condition are what actually control access. Storing them as secrets is just tidiness. Update PROJECT_ID and the region values in the workflow's env: block to match your setup.

Understanding the Workflow

on: pull_request: types: [opened, synchronize, reopened]: these are three distinct triggers.

• opened fires when the PR is first created.

• synchronize fires on every subsequent push to the branch, which is what keeps the environment current.

• reopened handles a PR that was closed and then reopened. Without it, that PR would have a torn-down environment and nothing to recreate it.

concurrency: this one saves real debugging time. Push three commits in quick succession, and you get three workflow runs targeting the same namespace. They race. The one that finishes last wins, and that's not necessarily the newest code. A fast early build can overtake a slow later one, leaving your preview environment silently running stale code while the PR shows green. Grouping by PR number with cancel-in-progress: true means only the newest run survives.

permissions: id-token: write: without this line, nothing else works. It's what allows the workflow to request an OIDC token from GitHub. Leave it out and google-github-actions/auth fails with an error about being unable to get an ID token. This is the single most common WIF setup mistake.

$GITHUB_ENV: writing KEY=value to this file makes the variable available in all subsequent steps. Not the current one, as a variable you write in a step can't be read later in that same step. It's why the setup happens in its own step at the top.

github.event.pull_request.head.sha: note this rather than github.sha. On pull_request events, GitHub checks out a merge commit. This is a temporary commit combining your branch with the base branch. github.sha refers to that merge commit, which doesn't exist in your repository's history and will confuse anyone who tries to look it up. head.sha is the actual last commit on the PR branch, which is what a developer expects to see.

--dry-run=client -o yaml | kubectl apply -f - is a small trick worth stealing. kubectl create namespace fails if the namespace already exists, which would break every push after the first. This pipeline generates the namespace YAML without contacting the cluster, then pipes it toapply, which creates it if it's missing and does nothing if it isn't. Idempotent by construction.

The heredoc overlay: instead of committing an overlay per PR (impossible, as you don't know PR numbers in advance), the workflow writes one at runtime and throws it away when the runner is destroyed. It's ephemeral configuration for an ephemeral environment.

kubectl rollout status is what makes the workflow honest. Without it, the job succeeds the moment kubectl apply is accepted by the API server, which happens before a single pod has started. A crash-looping image would still show a green check. rollout status blocks until pods are actually ready, or fails after 180 seconds. Green means working.

A Note on Forked Pull Requests

Something is worth knowing before you deploy this anywhere public: pull requests from forks do not receive id-token: write permission, by design. GitHub blocks it so a stranger can't open a PR that steals your cloud credentials.

For a private repository or a personal project, this doesn't affect you. For a public repository accepting outside contributions, you'd need a workflow_run-based pattern where a privileged workflow runs from the base repository after the untrusted build completes. That's beyond this handbook's scope, but if you're deploying this on an open-source project, don't skip it.

Step 10: Build the Teardown Workflow

Spin-up is only half the value. An environment that never disappears is just a slow leak in your cloud bill.

Create .github/workflows/preview-teardown.yml:

name: Tear Down PR Preview

on:
  pull_request:
    types: [closed]

permissions:
  contents: read
  id-token: write

env:
  PROJECT_ID: YOUR_PROJECT_ID
  CLUSTER_NAME: pr-preview-cluster
  CLUSTER_ZONE: europe-west2-a

jobs:
  teardown:
    name: Delete preview namespace
    runs-on: ubuntu-latest

    steps:
      - name: Set namespace
        run: echo "NAMESPACE=pr-${{ github.event.pull_request.number }}" >> $GITHUB_ENV

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.WIF_SERVICE_ACCOUNT }}

      - name: Get GKE credentials
        uses: google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: ${{ env.CLUSTER_NAME }}
          location: ${{ env.CLUSTER_ZONE }}

      - name: Delete namespace
        run: |
          echo "Deleting namespace ${NAMESPACE}"
          kubectl delete namespace "${NAMESPACE}" \
            --ignore-not-found=true \
            --wait=false

      - name: Confirm
        run: |
          echo "Preview environment for PR #${{ github.event.pull_request.number }} torn down."

A few details here matter more than they might seem.

First, types: [closed] covers merges too. GitHub doesn't have a separate merged event type. Merging a pull request fires closed with the payload field merged: true. So this single trigger handles both "merged successfully" and "abandoned and closed." Both should tear down the environment, so we don't need to distinguish between them.

Next, --ignore-not-found=true: if the namespace doesn't exist (perhaps the deploy failed, or someone deleted it manually), kubectl delete would normally exit with an error and mark the workflow red. That's a false alarm: the desired end state is "namespace doesn't exist," and it doesn't. This flag makes the step succeed when there's nothing to do.

Finally, --wait=false: namespace deletion is asynchronous. Kubernetes marks itTerminating, then works through deleting every resource inside it. Waiting for that to complete could hold your CI runner for a minute or more, for no benefit. Fire the delete and let the cluster finish on its own time.

Step 11: Open a Pull Request and Watch It Work

Commit everything and push:

git add .
git commit -m "Add PR preview environment infrastructure"
git push -u origin main

Now create a branch and make a visible change:

git checkout -b test-preview

Edit app/main.go. Change the first line of the handler output:

fmt.Fprintf(w, "Preview environment is live — testing the pipeline\n\n")

Commit and push:

git add .
git commit -m "Test preview environment"
git push -u origin test-preview

Open a pull request on GitHub, then go to the Actions tab. You should see "Deploy PR Preview" running.

Watch the steps in order: authentication succeeds without a key, the image builds and pushes, the namespace is created, Kustomize renders the overlay (you'll see the fully resolved manifest printed), the deployment applies, and rollout status waits until pods are ready.

Once it's green, verify from your terminal:

kubectl get namespaces | grep pr-

You should see your pr-1 (or whatever number your PR got).

kubectl get pods --namespace=pr-1

Now reach the application:

kubectl port-forward --namespace=pr-1 service/preview-app 8080:80

In a second terminal:

curl http://localhost:8080

You should see:

Preview environment is live — testing the pipeline

PR number  : 1
Git SHA    : a1b2c3d
Namespace  : pr-1
Pod name   : preview-app-7d9f8b6c4-xk2mp

The application is reporting the PR number and commit SHA that were baked into its binary at build time, and the namespace it discovered through the Downward API. Every one of those values was resolved by a different part of the pipeline, and they all agree. That's your proof the whole chain works.

Now push another commit to the same branch and watch the environment update in place.

Then close the pull request and watch the teardown workflow fire.

kubectl get namespace pr-1

Terminating, then gone.

What You Have, and What's Still Missing

Stop and take stock, because the hard infrastructure work is done. A pull request now produces a real, isolated environment with no manual steps, and closing it removes every trace. That's the whole lifecycle, automated.

But look again at how you reached it:

kubectl port-forward --namespace=pr-1 service/preview-app 8080:80

That requires kubectl installed, credentials for the cluster, permission to use them, and knowledge of which namespace maps to which PR. This is fine for the person who built the pipeline, but it's useless for the designer checking a spacing change, the product manager reviewing a copy edit, or the QA engineer clicking through a flow before sign-off.

The whole point of a preview environment is that anyone can look at the change. An environment only engineers can reach isn't really a preview environment. It's a debugging tool.

The rest of this tutorial closes that gap, and then makes the result safe to leave running. First, we'll set up a public HTTPS URL for every environment. Then we'll build the guardrails and go through the cleanup that stop it becoming a quiet line item on your cloud bill.

Step 12: Reserve a Static IP Address

Before installing anything, you'll need to reserve an external IP address.

This matters because when you install the Ingress controller, it creates a Kubernetes Service of type LoadBalancer, and Google Cloud provisions a real load balancer with a public IP. By default, that IP is ephemeral. If the Service is deleted and recreated, you get a different one.

Your wildcard DNS record points at that IP. If it changes, every preview environment breaks at once, and you won't find out until someone tells you the link is dead. Reserving the IP first means it belongs to your project, not to the load balancer, and it survives anything you do to the cluster.

Add this to terraform/main.tf:

# -----------------------------------------------------------
# Static external IP for the Ingress controller
# -----------------------------------------------------------
resource "google_compute_address" "ingress" {
  name         = "pr-preview-ingress-ip"
  region       = var.region
  address_type = "EXTERNAL"

  depends_on = [google_project_service.apis]
}

And add the output to terraform/outputs.tf:

output "ingress_ip" {
  description = "Point your wildcard DNS record at this address"
  value       = google_compute_address.ingress.address
}

You'll also want compute.googleapis.com in the APIs list if it isn't there already. Update the for_each set in your google_project_service resource:

  for_each = toset([
    "container.googleapis.com",
    "artifactregistry.googleapis.com",
    "iamcredentials.googleapis.com",
    "sts.googleapis.com",
    "compute.googleapis.com",
  ])

Apply it:

cd terraform
terraform apply \
  -var="project_id=YOUR_PROJECT_ID" \
  -var="github_repo=YOUR_USERNAME/pr-preview-envs"

Note the IP that comes back:

terraform output ingress_ip

Write it down. You'll use it twice in the next two steps.

Important: this must be a regional address in the same region as your cluster, which is what region = var.region gives you. Google Cloud also offers global addresses, used by global HTTP(S) load balancers.

The Service type LoadBalancer that ingress-nginx creates is a regional load balancer, and it can't bind a global address. Mixing these up produces a Service stuck in Pending forever with an error that doesn't clearly explain why.

Step 13: Install the Ingress Controller

An Ingress is a Kubernetes resource that describes HTTP routing rules: "requests for this hostname go to that Service." But the Ingress resource is only a description. Something has to actually read those rules and route traffic, and that something is an Ingress controller.

We'll use ingress-nginx, which is the community-maintained controller built on NGINX. It runs as pods in your cluster, watches the Kubernetes API for Ingress resources, and rewrites its own NGINX configuration whenever one changes.

The architectural point that makes this work for preview environments is that one controller serves every namespace. Each preview environment creates its own Ingress resource. The controller picks it up, adds a routing rule for that hostname, and sends traffic to the right Service. The controller doesn't care how many environments exist.

Contrast this with what we'd otherwise do. If each preview environment used a LoadBalancer Service, every PR would provision its own Google Cloud load balancer, each with an hourly charge and its own IP address that DNS would need to know about. Ten open PRs would mean ten load balancers. With a shared Ingress controller, ten open PRs mean one.

Install it with Helm:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.service.loadBalancerIP="YOUR_STATIC_IP" \
  --set controller.service.externalTrafficPolicy=Local

Replace YOUR_STATIC_IP with the address from Step 12.

Let's break down the flags before we go on:

--namespace ingress-nginx --create-namespace: the controller gets its own namespace, separate from any preview environment. It's shared infrastructure, and it should never be caught up in a preview environment's teardown.

controller.service.loadBalancerIP: tells GKE to bind the load balancer to the specific address you reserved, rather than allocating a new one. This is the line that makes your DNS record stable.

controller.service.externalTrafficPolicy=Local: preserves the client's real source IP. Without it, Kubernetes may route a request through an intermediate node, and by the time it reaches NGINX the source IP has been rewritten to a node's internal address. Your access logs would show cluster IPs instead of real visitors.

The tradeoff is that traffic only lands on nodes actually running a controller pod, which is fine here.

Watch for the external IP to be assigned:

kubectl get service ingress-nginx-controller \
  --namespace ingress-nginx \
  --watch

It shows <pending> for a minute or two while Google provisions the load balancer, then displays your static IP from Step 12. Press Ctrl+C once it appears.

Verify that the controller is healthy:

kubectl get pods --namespace ingress-nginx

Step 14: Point Wildcard DNS at the Load Balancer

This step takes about two minutes and is the piece that makes the whole pattern scale.

Go to wherever your domain's DNS is managed and create a single A record:

That's it. That's the entire DNS configuration for the project.

Here's why one record is enough. A wildcard DNS record matches any label in that position. *.preview.yourdomain.com resolves pr-1.preview.yourdomain.com, pr-42.preview.yourdomain.com, and pr-99999.preview.yourdomain.com, all to the same IP, all without anyone creating a record for them.

This is worth pausing on, because it's the design decision that keeps the system simple. The obvious approach would be to have the CI pipeline create a DNS record when a PR opens and delete it when the PR closes. That means DNS API credentials in your pipeline, another thing to fail, another thing to leak, and another source of orphaned resources.

The wildcard approach eliminates that entire category of work: DNS is configured once, manually, and never touched again.

Routing to the right environment doesn't happen in DNS. Every hostname resolves to the same load balancer, and the Ingress controller reads the Host header of each incoming request to decide which namespace's Service should handle it.

Verify the record has propagated:

dig +short anything-at-all.preview.yourdomain.com

You should get your static IP back. Note that the subdomain doesn't need to exist yet. That's the whole point of a wildcard.

If nothing comes back, wait a few minutes and try again. DNS propagation is usually fast with a low TTL, but it isn't instant.

Step 15: Install cert-manager and Create a ClusterIssuer

Now for HTTPS.

cert-manager is a Kubernetes controller that automates the full certificate lifecycle. It watches for Ingress resources annotated with an issuer, requests a certificate from a certificate authority, proves you control the domain, stores the certificate in a Kubernetes Secret, and renews it before it expires. Ther eare no cron jobs, manual renewal, or expired-certificate incident at 3 am.

Install it:

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.21.2 \
  --set crds.enabled=true

--set crds.enabled=true Installs cert-manager's Custom Resource Definitions: Certificate, Issuer, ClusterIssuer, CertificateRequest and others. These extend the Kubernetes API with new resource types. Without them, cert-manager has nothing to watch for. (If you're following an older tutorial that uses installCRDs=true, that flag was renamed in v1.15.)

Check it's running:

kubectl get pods --namespace cert-manager

You should see three pods: the controller, the webhook, and cainjector.

Create the ClusterIssuer

An Issuer tells cert-manager where to get certificates from. A ClusterIssuer is the same thing with cluster-wide scope, which is what we need. A namespaced Issuer would have to be recreated in every single preview namespace.

Create k8s/cluster/letsencrypt-staging.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # Let's Encrypt's staging environment. Certificates from this
    # server are NOT trusted by browsers, but the rate limits are
    # far more generous. Use this while debugging.
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: YOUR_EMAIL@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

And k8s/cluster/letsencrypt-prod.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: YOUR_EMAIL@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

Apply both:

kubectl apply -f k8s/cluster/letsencrypt-staging.yaml
kubectl apply -f k8s/cluster/letsencrypt-prod.yaml

Confirm they're ready:

kubectl get clusterissuer

Both should show READY: True.

Why Two Issuers?

This isn't redundancy. It's how you avoid a frustrating afternoon.

Let's Encrypt enforces strict rate limits on its production environment. The one that matters here is 50 certificates per registered domain per week. There's also a duplicate-certificate limit of 5 per week for the identical set of hostnames.

While you're setting this up, you'll likely misconfigure something. The Ingress annotation will be wrong, or the HTTP-01 challenge won't route correctly, or DNS won't have propagated. Each failed attempt burns against your quota, and once you hit the limit you're locked out for a week with no appeal.

The staging issuer has far looser limits. Its certificates are signed by an untrusted root, so browsers show a warning, but that warning is itself proof the pipeline works. If you see a certificate warning naming "Let's Encrypt Staging," everything functioned correctly, and the only remaining step is switching issuers.

Use letsencrypt-staging until the flow works end to end, then switch to letsencrypt-prod.

How the HTTP-01 Challenge Works

This is worth understanding, because it's what breaks when things go wrong.

When cert-manager requests a certificate for pr-42.preview.yourdomain.com, Let's Encrypt needs proof you control that hostname. With the HTTP-01 solver, the sequence is:

1. Let's Encrypt gives cert-manager a random token.

2. cert-manager creates a temporary pod, Service, and Ingress rule that serves that token at http://pr-42.preview.yourdomain.com/.well-known/acme-challenge/<token>.

3. Let's Encrypt makes a real HTTP request to that URL from the public internet.

4. If the token matches, domain control is proven and the certificate is issued.

5. cert-manager deletes the temporary resources and stores the certificate in a Secret.

Three things must be true for this to work, and they're the three things to check when it doesn't: DNS must resolve the hostname to your load balancer, port 80 must be reachable from the internet (don't redirect everything to HTTPS at the load balancer level. The challenge is plain HTTP), and the Ingress controller must be routing correctly.

A Note on Scale

HTTP-01 issues one certificate per hostname, and every preview environment has a unique hostname. That's one certificate per pull request.

Against Let's Encrypt's 50-per-week limit, that's comfortable for a solo project or a small team. An active team opening more than 50 PRs a week would hit the ceiling.

The production answer is to issue a single wildcard certificate for *.preview.yourdomain.com and share it across every namespace. Wildcards can't be issued over HTTP-01. They require the DNS-01 challenge, where cert-manager proves control by creating a TXT record instead of serving a file.

With Cloud DNS, cert-manager can do this automatically using a Google service account. You'd then replicate the resulting Secret into each preview namespace with a tool like reflector.

That's more moving parts than this handbook needs, but it's the right answer once your PR volume justifies it, and it's worth knowing that limit exists before you hit it.

Step 16: Add an Ingress to the Base Manifests

Now we'll give the application a route.

Create k8s/base/ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: preview-app
  annotations:
    # Tells cert-manager to request a certificate for the hosts
    # listed under spec.tls, using this ClusterIssuer.
    cert-manager.io/cluster-issuer: letsencrypt-staging
    # Redirect HTTP to HTTPS once the certificate exists.
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - preview-host-placeholder
      secretName: preview-app-tls
  rules:
    - host: preview-host-placeholder
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: preview-app
                port:
                  number: 80

Add it to k8s/base/kustomization.yaml:

You created this file back in Step 8 with two resources. Add the Ingress to it:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml

Here are the pieces that matter:

cert-manager.io/cluster-issuer: this single annotation is what triggers the entire certificate workflow. cert-manager watches for it, reads the hostnames from spec.tls, and does everything else. Note it's set to letsencrypt-staging for now. We'll switch to production once the flow is proven.

ingressClassName: nginx: identifies which controller should handle this Ingress. A cluster can run several controllers simultaneously, and this field prevents them from fighting over the same resource. Older tutorials use the kubernetes.io/ingress.class annotation, which is deprecated.

secretName: preview-app-tls: where cert-manager stores the issued certificate. It's created automatically in the same namespace as the Ingress, so each preview environment has its own.

preview-host-placeholder: deliberately not a valid hostname. It appears twice: once in spec.tls.hosts and once in spec.rules.host, and both get replaced per PR in the next step. Using something that obviously isn't a real domain means that if the patch ever fails, you get an immediate visible error rather than a subtle misroute.

pathType: Prefix with path: /: matches every path on the host. Preview environments serve the whole application, not a subpath.

Step 17: Patch the Hostname Per Pull Request

Each environment needs its own hostname substituted into two places in the Ingress. Kustomize handles this with a JSON 6902 patch. A precise operation that targets an exact path within a resource.

In Step 9, the workflow generated an overlay that set the namespace and image tag. We extend that generated overlay with a patch block. Here's what the complete generated kustomization.yaml looks like for PR 42:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: pr-42

resources:
  - ../../base

images:
  - name: preview-app
    newName: europe-west2-docker.pkg.dev/your-project/pr-preview/preview-app
    newTag: pr-42-a1b2c3d

patches:
  - target:
      kind: Ingress
      name: preview-app
    patch: |-
      - op: replace
        path: /spec/rules/0/host
        value: pr-42.preview.yourdomain.com
      - op: replace
        path: /spec/tls/0/hosts/0
        value: pr-42.preview.yourdomain.com

The patch syntax reads literally: in the resource matching kind: Ingress, name: preview-app, replace the value at /spec/rules/0/host, and replace the value at /spec/tls/0/hosts/0.

The 0 indices are array positions. /spec/rules/0 is the first (and only) rule. /spec/tls/0/hosts/0 is the first hostname in the first TLS block. If you later add a second hostname per environment, you'd add another operation targeting index 1.

Why a JSON 6902 patch rather than a strategic merge patch? Because strategic merge on an array of objects is ambiguous. Kubernetes has to guess whether you mean "replace this list" or "merge into an existing entry," and the answer depends on per-field merge keys. For replacing a single value at a known position, an explicit op: replace says exactly what you mean and fails loudly if the path doesn't exist.

The workflow generates this file at runtime, so you never commit a per-PR overlay. We'll write that step in Step 19.

Step 18: Add Guardrails with ResourceQuota and LimitRange

Before we wire up the workflow, let's stop a preview environment from being able to take down the cluster.

Right now nothing prevents a pull request from deploying a manifest requesting 32 CPUs, or scaling to 200 replicas, or creating a dozen LoadBalancer Services. Someone doesn't need bad intent for this to happen. A bad merge, a copied manifest, or a typo in a replica count does it just as effectively.

Two Kubernetes resources solve this, and they work together.

Create k8s/cluster/namespace-guardrails.yaml as a template:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: preview-quota
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
    pods: "10"
    services: "5"
    services.loadbalancers: "0"
    persistentvolumeclaims: "2"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: preview-limits
spec:
  limits:
    - type: Container
      # Applied to containers that don't specify their own
      defaultRequest:
        cpu: 50m
        memory: 64Mi
      default:
        cpu: 200m
        memory: 128Mi
      # Hard ceiling — a container asking for more is rejected
      max:
        cpu: "1"
        memory: 512Mi

What Each One Does

A ResourceQuota caps the total consumption of an entire namespace. Add up every pod's requests and limits. The sum can't exceed these numbers. A request that would push the namespace over its quota is rejected by the API server at admission time, before anything is scheduled.

The services.loadbalancers: "0" line is worth highlighting. It makes it structurally impossible for a preview environment to provision a Google Cloud load balancer, even if someone commits a manifest with type: LoadBalancer. That's a direct financial guardrail: the most expensive mistake this system could make is now unrepresentable.

A LimitRange operates on individual containers rather than the namespace total. It sets defaults for containers that don't specify resources, and a hard maximum that no single container may exceed.

Why They Must Be Deployed Together

Once a ResourceQuota that constrains requests.cpu exists in a namespace, every container in that namespace must declare resource requests. A pod without them is rejected outright, because Kubernetes can't verify it fits within the quota if it doesn't know what it's asking for.

That means adding a ResourceQuota on its own will break any manifest that omits resource specs, and most manifests found in the wild do omit them.

The LimitRange fixes this by supplying defaults. A container with no resource spec gets the defaultRequest and default values injected automatically at admission time, which satisfies the quota's requirement.

Deploy one without the other, and you get confusing failures. Deploy both and the system is forgiving of incomplete manifests while still being strictly bounded.

Sizing the Numbers

The values above suit a small application like ours. To adjust them, work backwards from your node capacity.

Two e2-standard-2 nodes give you 4 vCPU and 16GB total, minus roughly 15% that GKE reserves for system components. Call it 3.4 usable vCPU. With requests.cpu: "1" per namespace, three preview environments could theoretically fill the cluster.

Our application requests only 50m CPU, so in practice you'll fit far more than three. But the quota is a ceiling on what a namespace may request, and you should set it well below "one PR can consume everything."

If you expect ten concurrent environments, size the per-namespace quota at roughly a tenth of usable capacity, or add nodes.

Step 19: Update the Deploy Workflow

Here's the complete .github/workflows/preview-deploy.yml, with everything added in this section marked # NEW. Replace your Step 9 version with this one.

name: Deploy PR Preview

on:
  pull_request:
    types: [opened, synchronize, reopened]

concurrency:
  group: preview-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  id-token: write
  pull-requests: write

env:
  PROJECT_ID: YOUR_PROJECT_ID
  REGION: europe-west2
  CLUSTER_NAME: pr-preview-cluster
  CLUSTER_ZONE: europe-west2-a
  REPOSITORY: pr-preview
  IMAGE_NAME: preview-app
  # NEW: the base domain for all preview environments
  PREVIEW_DOMAIN: preview.yourdomain.com
  # NEW: switch to letsencrypt-prod once the flow is proven
  CLUSTER_ISSUER: letsencrypt-staging

jobs:
  deploy:
    name: Build and deploy preview
    runs-on: ubuntu-latest

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set environment variables
        run: |
          PR_NUMBER=${{ github.event.pull_request.number }}
          SHORT_SHA=$(echo "${{ github.event.pull_request.head.sha }}" | cut -c1-7)

          echo "PR_NUMBER=${PR_NUMBER}"    >> $GITHUB_ENV
          echo "SHORT_SHA=${SHORT_SHA}"    >> $GITHUB_ENV
          echo "NAMESPACE=pr-${PR_NUMBER}" >> $GITHUB_ENV
          echo "IMAGE_TAG=pr-${PR_NUMBER}-${SHORT_SHA}" >> $GITHUB_ENV
          echo "IMAGE_BASE=${{ env.REGION }}-docker.pkg.dev/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.IMAGE_NAME }}" >> $GITHUB_ENV
          # NEW: the hostname this environment will be reachable at
          echo "PREVIEW_HOST=pr-${PR_NUMBER}.${{ env.PREVIEW_DOMAIN }}" >> $GITHUB_ENV

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.WIF_SERVICE_ACCOUNT }}

      - name: Set up gcloud
        uses: google-github-actions/setup-gcloud@v2

      - name: Configure Docker for Artifact Registry
        run: gcloud auth configure-docker ${{ env.REGION }}-docker.pkg.dev --quiet

      - name: Build and push image
        run: |
          docker build \
            --build-arg PR_NUMBER="${PR_NUMBER}" \
            --build-arg GIT_SHA="${SHORT_SHA}" \
            -t "${IMAGE_BASE}:${IMAGE_TAG}" \
            ./app

          docker push "${IMAGE_BASE}:${IMAGE_TAG}"

      - name: Get GKE credentials
        uses: google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: ${{ env.CLUSTER_NAME }}
          location: ${{ env.CLUSTER_ZONE }}

      - name: Create namespace
        run: |
          kubectl create namespace "${NAMESPACE}" \
            --dry-run=client -o yaml | kubectl apply -f -

          # NEW: label it so the reaper knows this namespace is ours
          kubectl label namespace "${NAMESPACE}" \
            app.kubernetes.io/managed-by=pr-preview \
            preview.pr="${PR_NUMBER}" \
            --overwrite

      # NEW: apply the quota and limits before anything deploys
      - name: Apply namespace guardrails
        run: |
          kubectl apply -f k8s/cluster/namespace-guardrails.yaml \
            --namespace="${NAMESPACE}"

      - name: Generate Kustomize overlay
        run: |
          mkdir -p k8s/overlays/"${NAMESPACE}"
          cat > k8s/overlays/"${NAMESPACE}"/kustomization.yaml <<EOF
          apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization

          namespace: ${NAMESPACE}

          resources:
            - ../../base

          images:
            - name: preview-app
              newName: ${IMAGE_BASE}
              newTag: ${IMAGE_TAG}

          patches:
            - target:
                kind: Ingress
                name: preview-app
              patch: |-
                - op: replace
                  path: /spec/rules/0/host
                  value: ${PREVIEW_HOST}
                - op: replace
                  path: /spec/tls/0/hosts/0
                  value: ${PREVIEW_HOST}
                - op: replace
                  path: /metadata/annotations/cert-manager.io~1cluster-issuer
                  value: ${CLUSTER_ISSUER}
          EOF

          echo "--- Rendered manifest ---"
          kubectl kustomize k8s/overlays/"${NAMESPACE}"

      - name: Deploy
        run: kubectl apply -k k8s/overlays/"${NAMESPACE}"

      - name: Wait for rollout
        run: |
          kubectl rollout status deployment/preview-app \
            --namespace="${NAMESPACE}" \
            --timeout=180s

      # NEW: wait for the TLS certificate to be issued
      - name: Wait for certificate
        continue-on-error: true
        run: |
          echo "Waiting for TLS certificate for ${PREVIEW_HOST}..."
          kubectl wait certificate/preview-app-tls \
            --namespace="${NAMESPACE}" \
            --for=condition=Ready \
            --timeout=180s

      - name: Show deployment details
        run: |
          echo "Preview URL: https://${PREVIEW_HOST}"
          kubectl get pods,svc,ingress,certificate --namespace="${NAMESPACE}"

There are a few things to understand about the new steps.

The ~1 in the annotation patch path: this looks like a typo and isn't:

path: /metadata/annotations/cert-manager.io~1cluster-issuer

JSON Pointer paths use / as a separator, so a literal / inside a key has to be escaped. The escape sequence is ~1. (And ~ itself escapes to ~0.) Since the annotation key iscert-manager.io/cluster-issuer, the pointer becomes cert-manager.io~1cluster-issuer. Get this wrong, and the patch silently targets a path that doesn't exist.

Namespace labels: The app.kubernetes.io/managed-by=pr-preview label is what makes the reaper in Step 21 safe. A cleanup job that deletes namespaces matching a name pattern is one bad regex away from deleting something important. A job that only ever touches namespaces carrying a label it applied itself has a much smaller worst case.

Guardrails before deploy: The ResourceQuota must exist before the application deploys, not after. Applying it afterwards means the first deployment escapes it entirely, and you'd only find out when something exceeded a limit that was never enforced.

continue-on-error: true on the certificate wait: This is deliberate. Certificate issuance depends on Let's Encrypt, an external service with rate limits and occasional slowness. If issuance is delayed, the environment is still deployed and still works over HTTP.

So failing the entire workflow would be wrong. We note the problem and continue. The PR comment in the next step reflects whether the certificate is actually ready.

Step 20: Comment the URL on the Pull Request

The environment is live. Now it's time to tell people about it.

Add this step to the end of the deploy job:

      - name: Comment preview URL on PR
        uses: actions/github-script@v7
        with:
          script: |
            const host = process.env.PREVIEW_HOST;
            const sha = process.env.SHORT_SHA;
            const ns = process.env.NAMESPACE;

            // A hidden marker so we can find our own comment later
            const marker = '<!-- pr-preview-environment -->';

            const body = [
              marker,
              '### Preview environment is ready',
              '',
              `**https://${host}**`,
              '',
              `Commit \`${sha}\` · Namespace \`${ns}\``,
              '',
              '<sub>This environment updates on every push and is deleted when the PR closes.</sub>',
            ].join('\n');

            const { data: comments } = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
            });

            const existing = comments.find(c => c.body.includes(marker));

            if (existing) {
              await github.rest.issues.updateComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                comment_id: existing.id,
                body,
              });
            } else {
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.issue.number,
                body,
              });
            }

So why this approach?

actions/github-script is maintained by GitHub itself and gives you an authenticated Octokit client with no setup. There are third-party actions that post PR comments in fewer lines, but every third-party action in a workflow with write permissions is a supply chain dependency.

Given the effort we spent in Step 5 removing a long-lived credential, adding an unaudited action that can write to your repository would be an odd trade.

The hidden marker comment is the mechanism that makes this a sticky comment. <!-- pr-preview-environment --> renders as nothing in GitHub's markdown, but it's present in the comment body. On each run, we list the PR's comments, look for one containing the marker, and update it if found.

Without this, a PR with fifteen commits gets fifteen near-identical bot comments, and the thread becomes unreadable. Reviewers start ignoring bot comments entirely, which defeats the purpose. One comment that stays current is the difference between a useful tool and noise.

process.env.PREVIEW_HOST: values written to $GITHUB_ENV in earlier steps are available as environment variables in the script. This is cleaner than interpolating ${{ }} expressions directly into JavaScript, which can break in surprising ways if a value contains quotes or backticks.

context.issue.number works on pull_request events because GitHub treats pull requests as a specialised kind of issue. It's why we use the issues API to comment on a PR.

The step needs pull-requests: write, which the workflow already declares.

Step 21: Build the Reaper

Here's the uncomfortable truth about the teardown workflow from Part 1: it will eventually fail to run.

The failure modes are mundane. Someone closes a PR while GitHub Actions is having an incident. A workflow file gets edited on a branch in a way that breaks the teardown job. Someone deletes a branch without closing its PR. A repository is archived with PRs still open. GitHub's event delivery hiccups.

None of these are common. All of them leave a namespace running indefinitely, consuming cluster capacity and (if it happens enough) real money. Cleanup that works 95% of the time is still a leak.

The fix is a reaper: a scheduled job that periodically compares what's running against what should be running, and deletes the difference.

Create .github/workflows/preview-reaper.yml:

name: Reap Orphaned Preview Environments

on:
  schedule:
    # 03:00 UTC daily
    - cron: '0 3 * * *'
  # Allow manual runs from the Actions tab
  workflow_dispatch:

permissions:
  contents: read
  id-token: write
  pull-requests: read

env:
  PROJECT_ID: YOUR_PROJECT_ID
  CLUSTER_NAME: pr-preview-cluster
  CLUSTER_ZONE: europe-west2-a
  # Delete environments older than this even if the PR is still open
  MAX_AGE_DAYS: 7

jobs:
  reap:
    name: Delete orphaned namespaces
    runs-on: ubuntu-latest

    steps:
      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
          service_account: ${{ secrets.WIF_SERVICE_ACCOUNT }}

      - name: Get GKE credentials
        uses: google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: ${{ env.CLUSTER_NAME }}
          location: ${{ env.CLUSTER_ZONE }}

      - name: Reap
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          set -euo pipefail

          echo "Fetching open pull requests..."
          OPEN_PRS=$(gh pr list \
            --repo "${{ github.repository }}" \
            --state open \
            --limit 500 \
            --json number \
            --jq '.[].number')

          echo "Open PRs: ${OPEN_PRS:-none}"
          echo ""

          # Only consider namespaces we created ourselves.
          NAMESPACES=$(kubectl get namespaces \
            -l app.kubernetes.io/managed-by=pr-preview \
            -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.preview\.pr}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}')

          if [ -z "${NAMESPACES}" ]; then
            echo "No preview namespaces found. Nothing to do."
            exit 0
          fi

          NOW=$(date -u +%s)
          MAX_AGE_SECONDS=$(( MAX_AGE_DAYS * 86400 ))
          DELETED=0

          while IFS=$'\t' read -r NS PR_NUM CREATED; do
            [ -z "${NS}" ] && continue

            CREATED_EPOCH=$(date -u -d "${CREATED}" +%s)
            AGE=$(( NOW - CREATED_EPOCH ))
            AGE_DAYS=$(( AGE / 86400 ))

            REASON=""

            # Rule 1: the PR is no longer open
            if ! echo "${OPEN_PRS}" | grep -qx "${PR_NUM}"; then
              REASON="PR #${PR_NUM} is not open"
            # Rule 2: the environment has outlived MAX_AGE_DAYS
            elif [ "${AGE}" -gt "${MAX_AGE_SECONDS}" ]; then
              REASON="environment is ${AGE_DAYS} days old (max ${MAX_AGE_DAYS})"
            fi

            if [ -n "${REASON}" ]; then
              echo "DELETING ${NS} — ${REASON}"
              kubectl delete namespace "${NS}" \
                --ignore-not-found=true \
                --wait=false
              DELETED=$(( DELETED + 1 ))
            else
              echo "KEEPING  ${NS} — PR #${PR_NUM} open, ${AGE_DAYS}d old"
            fi
          done <<< "${NAMESPACES}"

          echo ""
          echo "Reaped ${DELETED} namespace(s)."

Where does GITHUB_TOKEN come from?

Unlike WIF_PROVIDER and WIF_SERVICE_ACCOUNT, which you created in Step 6, GITHUB_TOKEN isn't something you set up. You won't find it on the secrets page, and you shouldn't add it there.

GitHub mints a fresh installation token at the start of every workflow run, exposes it as secrets.GITHUB_TOKEN, and revokes it the moment the job finishes. It's scoped to the repository running the workflow and nothing else.

Understanding the Reaper

The reaper runs from GitHub Actions, not inside the cluster. A CronJob in Kubernetes would need a GitHub token stored as a cluster Secret to query open PRs. That's a long-lived credential sitting in the cluster, which is exactly what we avoided in Step 5.

A scheduled workflow already has GITHUB_TOKEN (scoped, automatic, short-lived) and already authenticates to GCP through Workload Identity Federation. No new secrets.

The label selector is the safety mechanism. The query is -l app.kubernetes.io/managed-by=pr-preview, not a name pattern like pr-*. This is a deliberate choice: a name-pattern match would happily delete a production namespace someone named pr-routing-service. The reaper can only ever see namespaces that the deploy workflow explicitly labelled, which means the set of things it could possibly delete is the set of things it created.

Be careful when you adapt this. Any cleanup automation should operate on a positive allowlist it controls, never a pattern that could match something it didn't create.

We also have two deletion rules, for two different problems. Rule 1 catches the actual orphan. A namespace whose PR is closed but whose teardown never ran. Rule 2 catches the stale environment: a PR that's been open for three weeks, hasn't been touched in two, and is quietly holding cluster capacity. Seven days is a reasonable default. If someone needs their environment back, they just push a commit, and it redeploys in a couple of minutes.

set -euo pipefail: -e exits on any command failure, -u errors on undefined variables, and -o pipefail makes a pipeline fail if any stage fails rather than only the last. In a script whose job is deleting things, silent failure is the worst outcome. If something goes wrong, stop immediately.

grep -qx: the -x flag requires the whole line to match. Without it, PR 4 would match against open PR 42 and the reaper would spare an orphaned namespace forever. It's a small flag, but a real bug.

workflow_dispatch lets you run the reaper manually from the Actions tab. Do that the first few times and read the output before trusting the schedule. The log prints KEEPING or DELETING with a reason for every namespace, so you can verify the logic matches your expectations before it runs unattended.

Step 22: Test the Whole Thing

Commit everything and push:

git add .
git commit -m "Add Ingress, TLS, PR comments, guardrails, and reaper"
git push

Create a branch, make a visible change, and open a pull request:

git checkout -b test-preview-urls
# edit app/main.go — change the greeting text
git add .
git commit -m "Test preview URLs"
git push -u origin test-preview-urls

Open the PR and watch the Actions tab.

When it finishes, the bot comment should appear with your URL. Open it.

Because you're still on the staging issuer, your browser will show a certificate warning. Something like "Your connection is not private," with an issuer of "Let's Encrypt Staging."

This is the success condition. It means DNS resolved, the load balancer routed, the Ingress matched the hostname, the HTTP-01 challenge completed, and a certificate was issued. Everything worked. The certificate simply comes from an untrusted test authority.

Click Advanced, pass through the warning, and confirm the application responds with the right PR number and namespace.

Check the certificate from the command line:

kubectl get certificate --namespace=pr-2 
#Ensure you use your own pr-number

You want READY: True. If it's False, inspect what's happening:

kubectl describe certificate preview-app-tls --namespace=pr-2
kubectl get challenges --all-namespaces

A pending challenge that never resolves almost always means DNS or port 80 reachability. Confirm with:

curl -I http://pr-2.preview.yourdomain.com/.well-known/acme-challenge/test

You want any HTTP response from NGINX. A 404 is fine, as it proves that the routing works. A connection timeout means the request isn't reaching your cluster.

Switch to Production Certificates

Once the staging flow works end to end, change one line in the workflow's env: block:

  CLUSTER_ISSUER: letsencrypt-prod

Make this change on your existing PR branch and push it. You don't need to close the pull request or merge to main first. You can also change the message in your main.go file to confirm everything works.

Push a new commit to the PR. The environment redeploys, cert-manager requests a real certificate, and this time the browser shows a padlock.

Open the URL in a private browsing window to check. Your browser cached an exception for the staging certificate against that hostname, and a normal tab may show a trusted-looking page that's really just a remembered override and still shows not secure.

Verify the Guardrails

Confirm the quota is active:

kubectl describe resourcequota preview-quota --namespace=pr-2

You'll see used versus hard limits for each resource. Then try to break it:

kubectl scale deployment/preview-app --replicas=50 --namespace=pr-2
kubectl get events --namespace=pr-2 | grep -i quota

The deployment will scale partway and then stop, with events explaining that it exceeded the quota. That's the guardrail doing its job: the namespace is bounded, and the rest of the cluster is unaffected.

Verify Teardown and the Reaper

Close the pull request. The teardown workflow fires and the namespace goes to Terminating.

Then test the reaper on a namespace it shouldn't touch. Create one manually without the label:

kubectl create namespace pr-9999

Run the reaper manually from the Actions tab.

It should ignore pr-9999 entirely, because that namespace carries no app.kubernetes.io/managed-by=pr-preview label. The log won't even mention it.

Clean it up yourself:

kubectl delete namespace pr-9999

That test matters more than it looks. You've just verified that your automated deletion job can't touch things it didn't create.

What This Costs

We should be concrete about this, since cost control is half the argument for this pattern.

Running continuously in europe-west2, here are the approximate monthly figures:

Call it roughly $190/month for the always-on infrastructure, regardless of how many preview environments you run.

The number that matters is what doesn't scale with PR count. One load balancer serves every environment. One DNS record covers every hostname. Certificates are free. The marginal cost of the eleventh preview environment is only the node capacity its pods consume, a few pence a day for an application this size.

Compare that with the naïve approach of a LoadBalancer Service per environment: ten open PRs would add roughly $180/month in load balancers alone, and that's before anyone forgets to clean one up. The services.loadbalancers: "0" line in the ResourceQuota exists specifically to make that mistake impossible.

There are two ways to cut the standing cost for a personal project. Use spot nodes, add spot = true to the node_config block in Terraform for roughly a 60–70% discount on node hours. Preview environments are the ideal spot workload: if a node is reclaimed, pods reschedule, and the worst case is a preview environment briefly restarting. Or scale the node pool to zero overnight with a scheduled workflow, since nobody reviews PRs at 3 am.

Cleanup

When you're finished:

# Delete every preview namespace
kubectl delete namespaces -l app.kubernetes.io/managed-by=pr-preview

# Remove the shared infrastructure
helm uninstall cert-manager --namespace cert-manager
helm uninstall ingress-nginx --namespace ingress-nginx
kubectl delete namespace cert-manager ingress-nginx

Uninstall the Helm releases before running terraform destroy. The Ingress controller's Service owns a Google Cloud load balancer that Terraform doesn't know about, because Kubernetes created it rather than Terraform. Destroying the cluster first can orphan that load balancer, leaving it billing quietly in a project with nothing else in it.

If You Plan to Rebuild This Later

Don't run a full terraform destroy. Tear down only the expensive parts:

cd terraform 
terraform destroy 
-var="project_id=YOUR_PROJECT_ID" 
-var="github_repo=YOUR_USERNAME/pr-preview-envs"

Here's why. When you delete a Workload Identity Pool, Google doesn't remove it immediately. It marks it DELETED and reserves the ID for 30 days. Terraform forgets about it, but Google hasn't. Recreating the stack within that window fails with:

Error: Error creating WorkloadIdentityPool: googleapi: Error 409: Requested entity already exists

Almost all of your cost is the cluster and its nodes. The identity pool, provider, service account, IAM bindings, static IP, and Artifact Registry repository cost nothing or pennies to leave in place. Rebuilding later is then just terraform apply, and your WIF_PROVIDER and WIF_SERVICE_ACCOUNT secrets in GitHub stay valid. No need to update them.

If you want to keep the cluster but stop paying for idle nodes, scale the node pool to zero instead:

gcloud container clusters resize pr-preview-cluster
--node-pool pr-preview-cluster-pool
--num-nodes 0
--zone europe-west2-a

You still pay for the GKE control plane, so this reduces cost rather than eliminating it.

If you're done for good:

cd terraform
terraform destroy
-var="project_id=YOUR_PROJECT_ID"
-var="github_repo=YOUR_USERNAME/pr-preview-envs" 

Then confirm nothing was left behind:

gcloud compute forwarding-rules list
gcloud compute addresses list

Both should be empty, or contain only resources belonging to other projects of yours. Don't forget to remove the wildcard DNS record too.

Conclusion

You've built something that most engineering teams either don't have or pay a vendor for.

You wrote a Go application that reports its own build metadata, injected at compile time through linker flags, so a running environment can always tell you exactly which commit it came from. You wrapped it in a multi-stage Dockerfile that produces a small, statically linked, non-root image

You provisioned a GKE cluster and an Artifact Registry repository with Terraform, which means the whole environment is reproducible and destroyable with a single command. You set up Workload Identity Federation so GitHub Actions authenticates to Google Cloud with short-lived OIDC tokens instead of a long-lived service account key, scoped to exactly one repository by an attribute condition.

You wrote base Kubernetes manifests once and used Kustomize overlays, generated at runtime, to deploy the same application into a different namespace with a different image tag and a different hostname for every pull request. You put a single shared Ingress controller in front of every environment and gave it a stable static IP, so one load balancer serves any number of previews. You pointed one wildcard DNS record at it, which means a new environment needs no DNS work at all: routing happens on the Host header inside the cluster. You added cert-manager so every environment gets a valid, auto-renewing TLS certificate, and made the pipeline announce itself with a sticky PR comment that updates in place.

You bounded each namespace with a ResourceQuota and LimitRange, including one line that makes the most expensive possible mistake structurally impossible. And you built a reaper that catches what the happy path misses, operating only on namespaces it can prove it created.

The result is that an isolated, production-shaped, publicly reachable environment is the default for every change, not something a developer has to request, claim in a Slack channel, or remember to clean up.

Where to Go From Here

There are a few directions you can explore. Here they are, roughly in order of how much they'd improve a real team's experience:

1. Seed data for each environment. The biggest limitation of what we built is that the application is stateless. Real applications need a database. The common patterns are a per-namespace ephemeral database seeded from a sanitised snapshot, or a shared database with per-environment schema isolation. This is the single hardest problem in preview environments and where most of the remaining engineering effort goes.

2. Run your test suite against the preview. Once the environment has a real URL, end-to-end tests can run against it in CI, a far stronger signal than tests against mocks.

3. Wildcard certificates via DNS-01. As covered in Step 15, this removes the Let's Encrypt rate limit ceiling once your PR volume approaches it.

4. Move to GitOps. Rather than the workflow running kubectl apply directly, have it commit the rendered manifests to a repository that Argo CD or Flux reconciles. The cluster's state then has a git history, and your CI pipeline no longer needs cluster write access at all.

5. Preview environment metrics. Hook the namespaces into the cost visibility tooling you'd use elsewhere, and you can answer "what do preview environments actually cost us per month" with a number rather than a shrug.

Project File Structure

pr-preview-envs/
├── app/
│   ├── main.go                      — Go HTTP server reporting build metadata
│   ├── go.mod                       — Go module definition
│   └── Dockerfile                   — Multi-stage build with ldflags injection
├── terraform/
│   ├── main.tf                      — GKE, Artifact Registry, WIF, static IP
│   ├── variables.tf                 — Input variables
│   └── outputs.tf                   — Cluster, WIF, and ingress IP values
├── k8s/
│   ├── base/
│   │   ├── deployment.yaml          — Deployment with probes and Downward API
│   │   ├── service.yaml             — ClusterIP Service
│   │   ├── ingress.yaml             — Ingress with cert-manager annotation
│   │   └── kustomization.yaml       — Base Kustomize config
│   └── cluster/
│       ├── letsencrypt-staging.yaml — Staging ClusterIssuer
│       ├── letsencrypt-prod.yaml    — Production ClusterIssuer
│       └── namespace-guardrails.yaml — ResourceQuota and LimitRange
└── .github/
    └── workflows/
        ├── preview-deploy.yml       — Build, deploy, certificate, PR comment
        ├── preview-teardown.yml     — Delete namespace on PR close
        └── preview-reaper.yml       — Scheduled orphan cleanup

All source files for this tutorial are available in the companion GitHub repository: https://github.com/Amiynarh/pr-preview-envs.

You'll investigate that situation in a disposable Kubernetes lab. Starting from a healthy application, you'll introduce three failures: an image that can't be pulled, a readiness probe that never passes, and a Pod that can't be scheduled. You'll identify the blocked revision, inspect its evidence, and verify recovery before moving on.

In the recorded image failure, the deployment reported both of these conditions. This table is derived from its JSON snapshot:

Those conditions answer different questions. The following experiments show why an application can still answer requests while its update is stuck.

Table of Contents

• How to Set Up the Lab

• How to Establish a Healthy Rollout

• How to Diagnose an Image-Pull Failure

• How to Distinguish Two Timeouts

• How to Restore the Healthy Configuration

• How to Diagnose Failed Readiness

• How to Diagnose a Placement Constraint

• How to Clean Up

• Conclusion

How to Set Up the Lab

You should be familiar with containers, Deployments, ReplicaSets, Pods, Services, and basic kubectl commands. The lab uses kind to run a single Kubernetes node inside a Docker container.

The recorded environment was:

The companion scripts require macOS ARM64 with Colima and kind 0.33.0. I didn't test Linux, Windows, and Intel machines. These memory figures describe the test machine, they're not minimum requirements.

You should already have Docker CLI, Colima, kind, and kubectl installed, with Colima running and internet access to Docker Hub.

Download the companion code here. Extract it into a fresh directory and open that directory in Terminal. Start Bash, which is the shell I'm using, then run setup:

bash
bash setup.sh

Setup creates the fcc-rollout-lab cluster and rollout-lab namespace. It writes credentials to ./kubeconfig, mounts the application source through an immutable ConfigMap, and creates the Deployment, Service, and an http-client Pod. No custom image build is needed.

It then verifies v1 and v2, saving snapshots and HTTP responses in an evidence archive. Setup refuses to overwrite an existing lab cluster or local kubeconfig. The node image and Python image are pinned by digest in the source.

After setup succeeds, define this function from the companion directory:

LAB_ROOT="$PWD"
k() {
  kubectl --kubeconfig "$LAB_ROOT/kubeconfig" \
    --context kind-fcc-rollout-lab \
    --namespace rollout-lab "$@"
}

The companion also includes failures.sh, an automated reproduction of the cases and checks. Follow the commands below for the manual walkthrough. The automated runner is an alternative way to reproduce the sequence.

Here, k is a Bash function that runs kubectl with this lab’s kubeconfig, context, and namespace already selected. For example, k get pods lists Pods in the lab. Run the definition above once in this Bash terminal, and keep using the same terminal and companion directory throughout the walkthrough.

How to Establish a Healthy Rollout

Before you break anything, establish that the rollout path and Service work. Setup performs a v1-to-v2 update by changing only the APP_VERSION environment variable in the Pod template.

The application is a small Python HTTP server. GET / returns its configured version and Pod name, and /ready returns HTTP 200. Every other path returns HTTP 404. The Pod name comes from metadata.name through the downward API.

Here's the routing logic from app/server.py:

if self.path == "/":
    status = 200
    payload = {
        "version": os.environ["APP_VERSION"],
        "pod": os.environ.get("POD_NAME", "local-test"),
    }
elif self.path == "/ready":
    status = 200
    payload = {"ready": True}
else:
    status = 404
    payload = {"error": "unknown path"}

The readiness probe checks /ready on port 8080. These settings come from the Deployment's spec in manifests/deployment.yaml. This is an excerpt, not a standalone manifest:

replicas: 2
minReadySeconds: 5
progressDeadlineSeconds: 180
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

With these rollout settings, the controller can add one replacement Pod. It waits for replacement availability before reducing the two healthy old replicas. A new Pod must remain Ready for five seconds to count as available. The progress deadline is 180 seconds.

maxUnavailable: 0 constrains this rollout. It doesn't protect the old Pods from unrelated failures. Terminating Pods can also remain visible beyond the active replica counts. These settings are teaching choices, not a production availability guarantee.

The recorded controls each completed successfully and returned ten correct Service responses. This table is derived from the two Deployment snapshots:

For example, the decoded response body of one recorded v2 request was:

{"version": "v2", "pod": "rollout-demo-5b4b45f68f-qrntz"}

Your Pod names will differ. Check your current state:

k get deployment rollout-demo -o json
k get pods -l app=rollout-demo -o wide

Before continuing, wait until exactly two application Pods remain, both Ready and neither terminating. The Deployment should have two updated, ready, and available replicas, with status.observedGeneration matching metadata.generation.

All subsequent failures leave APP_VERSION=v2. That makes the response's Pod name essential: the version alone can't distinguish the previous revision from the new one.

How to Diagnose an Image-Pull Failure

The first failure changes only the image reference to a deliberately missing tag in the public Python repository:

k apply -f failure-lab/manifests/image.json
k rollout status deployment/rollout-demo --timeout=10s

The short watch in the recorded run exited with code 1 and this error:

error: timed out waiting for the condition

This is expected for the exercise. Authentication, connection, or other errors require a different diagnosis. Don't treat every nonzero exit as the intended result.

Find the Blocked Revision

Inspect the Deployment and its related objects:

k get deployment rollout-demo
k get deployment rollout-demo -o json
k describe deployment rollout-demo
k get replicasets -l app=rollout-demo -o wide
k get pods -l app=rollout-demo -o wide

Compare metadata.generation with status.observedGeneration. If the latter is lower, the controller hasn't yet observed the current desired state. Recheck before interpreting its rollout status.

The recorded fault snapshot contained three replicas: one updated, two ready, and two available. A healthy-looking ready count can therefore come entirely from the previous revision.

kubectl describe deployment identifies the NewReplicaSet. Inspect that ReplicaSet and the unready Pod it owns. Replace both values below with names from your output:

NEW_RS='replace-with-the-new-replicaset-name'
BAD_POD='replace-with-its-unready-pod-name'
k get replicaset "$NEW_RS" -o json
k get pod "$BAD_POD" -o json
k describe pod "$BAD_POD"

In the ReplicaSet's metadata.ownerReferences, the entry with controller: true must name the Deployment's UID. In the Pod's owner references, the controller UID must match that ReplicaSet's UID. The new ReplicaSet's template matches the Deployment's current template apart from the generated template-hash label. Its revision annotation matches the Deployment's revision.

Labels narrow your search. The UID ownership chain establishes which objects belong together. See the owner-reference documentation for the underlying relationship.

Read the Failure Message

Get Events for this exact Pod instance:

POD_UID="$(
  k get pod "$BAD_POD" -o jsonpath='{.metadata.uid}'
)"
k get events --field-selector "involvedObject.uid=$POD_UID" \
  --sort-by=.metadata.creationTimestamp -o yaml

The image Pod was scheduled: PodScheduled=True, with nodeName set. Its container was waiting with ErrImagePull, followed by ImagePullBackOff between retries. The Pod phase remained Pending. The registry error ended with this exact text:

docker.io/library/python:fcc-rollout-missing-20260928: not found

That message identifies the missing tag. Image-pull errors can also result from registry credentials, throttling, DNS, or network problems. Diagnose the Event message rather than assuming every ImagePullBackOff has the same cause.

Also distinguish the kubectl STATUS display from the Pod's phase. ImagePullBackOff is a container waiting reason, while Pending is the phase. The Pod lifecycle reference explains the distinction.

Check Which Pods Serve Requests

Inspect the Service's EndpointSlices and send requests through the Service from inside the cluster:

k get endpointslices -l kubernetes.io/service-name=rollout-demo -o json
k exec --pod-running-timeout=30s --request-timeout=60s http-client -- \
  python -u /app/probe.py --url http://rollout-demo:8080/ \
  --expected-version v2 --count 10 --interval 0.2

The probe checks the status and version and records the responding Pod's name. Compare those names with the previous healthy ReplicaSet's Pods. Compare the EndpointSlice targetRef.uid values with their UIDs too.

In the recorded run, all ten responses came from the two previous-revision Pods. Their EndpointSlice entries were ready. The blocked image Pod was also listed, but its endpoint had ready: false and serving: false.

The new Pod couldn't become available, so the controller retained both healthy old replicas. The Service samples confirmed that those Pods answered. Ten successful requests establish what happened during those samples. They don't prove uninterrupted service between measurements.

How to Distinguish Two Timeouts

The first timeout came from your terminal. kubectl rollout status --timeout=10s stopped waiting after its client-side timeout. It didn't cancel the update or restore the previous configuration. The command reference describes that watch timeout.

The second belongs to the Deployment controller. Wait for its deadline reason, then inspect the full conditions:

k wait --for=jsonpath='{.status.conditions[?(@.type=="Progressing")].reason}'=ProgressDeadlineExceeded \
  deployment/rollout-demo --timeout=240s
k get deployment rollout-demo -o json

The 240-second timeout bounds this separate wait. Confirm that Progressing is False with reason ProgressDeadlineExceeded. The kubectl wait reference documents JSONPath waits.

In the original recording, the Progressing condition's last-update time changed from 04:57:14Z to 05:00:15Z: 181 seconds, with a configured deadline of 180. Controller timing isn't a promise of an exact wall-clock interval.

Available=True remained because the old Pods still supplied the required availability. Progressing=False reported the stalled update. Conversely, Progressing=True can also remain after a successful rollout, so always read its reason and the replica counts.

The Deployment controller reports the deadline without automatically rolling back. The missing image was still the desired image in the deadline snapshot.

Repeat the Service check now. All ten recorded requests after the deadline also came from the old Pods. Only the image case waits for the controller deadline in this tutorial.

How to Restore the Healthy Configuration

Recover now, before introducing another failure. Keep BAD_POD set to the failed Pod you just inspected:

k apply -f failure-lab/manifests/healthy.json
k rollout status deployment/rollout-demo --timeout=300s
k wait --for=delete "pod/$BAD_POD" --timeout=120s
k get deployment rollout-demo -o json
k get replicasets -l app=rollout-demo -o wide
k get pods -l app=rollout-demo -o wide
k get endpointslices -l kubernetes.io/service-name=rollout-demo -o json
k exec --pod-running-timeout=30s --request-timeout=60s http-client -- \
  python -u /app/probe.py --url http://rollout-demo:8080/ \
  --expected-version v2 --count 10 --interval 0.2

Check the whole recovery, not just the rollout command's success. The desired spec should match healthy.json. The controller must have observed it, with replicas, updated replicas, ready replicas, and available replicas all equal to two.

Exactly two application Pods should remain, both owned by the current ReplicaSet, Ready, and not terminating. Their UIDs should be the ready Service endpoints, and the HTTP samples should return v2 from those Pod names.

In the original experiment, recovery reused the existing healthy ReplicaSet and the same two Pods. Their template already matched the restored configuration. The failed ReplicaSet scaled to zero.

Revision annotations changed even though those healthy Pods remained. These values are from the recorded run:

Your numbers can differ. Restore the known healthy specification rather than relying on these example revision numbers.

How to Diagnose Failed Readiness

After verifying recovery, apply the second failure:

k apply -f failure-lab/manifests/readiness.json
k rollout status deployment/rollout-demo --timeout=10s

This changes only readinessProbe.httpGet.path, from /ready to /not-ready. The short watch times out again.

Repeat the inspection sequence from the image case. Identify the new ReplicaSet, reset NEW_RS and BAD_POD to this case's names, and query Events using the new Pod UID. This time the container runs, but the Pod isn't Ready.

The following values are derived from the recorded Pod snapshot:

The Pod's Event message explains why:

Readiness probe failed: HTTP probe failed with statuscode: 404

The app has no handler for /not-ready. HTTP probes accept responses from 200 through 399, so this 404 fails the readiness check. A failing readiness probe leaves the container running while marking it unready and continuing to probe. See the probe documentation.

The recording also had brief startup connection refused Events. Those alone wouldn't establish this fault: they can occur before a healthy server begins listening. The later 404 demonstrates that this server answered the request. Combined with the single changed field and the known routes, it identifies the wrong probe path.

The failed readiness check didn't restart the container. The captured restart count was zero. This case wasn't a crash loop.

Repeat the EndpointSlice and Service checks. The new Pod remained listed in the recorded EndpointSlice with ready: false, serving: false, and terminating: false. The two old Pods had ready endpoints, and all ten responses came from them.

This Service doesn't enable publishNotReadyAddresses. Endpoint readiness and termination settings matter, so avoid generalizing this observation to every Service configuration. The EndpointSlice documentation describes those conditions and exceptions.

For related background, freeCodeCamp's Kubernetes self-healing tutorial also explores readiness. Here, the failed probe explains why a new rollout can't finish.

Run the complete recovery sequence again, using this readiness Pod as BAD_POD. Verify healthy v2 before proceeding.

How to Diagnose a Placement Constraint

The third failure adds this selector to the Pod template:

nodeSelector:
  rollout-lab.example.com/placement: no-matching-node

Check that it matches no node, then apply it:

k get nodes -l rollout-lab.example.com/placement=no-matching-node
k apply -f failure-lab/manifests/scheduling.json
k rollout status deployment/rollout-demo --timeout=10s

If the first command lists a node, the intended unmatched-selector scenario doesn't hold. Inspect the label configuration before continuing.

Repeat the ownership and Event checks, updating NEW_RS, BAD_POD, and POD_UID for this case. The new Pod is Pending, just as in the image failure. But Pending covers both waiting for scheduling and waiting for container setup, including image downloads.

These are the recorded differences:

The scheduling Event said:

0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

No node satisfied the selector, so the Pod was never assigned to a kubelet. Image retrieval had not begun. Kubernetes requires nodes to match every specified nodeSelector label, as the node-selection documentation explains.

Repeat the Service checks. This unscheduled Pod had no Pod IP or EndpointSlice entry. The old Pods supplied all ten recorded responses.

Apply the complete recovery sequence once more, using this scheduling Pod as BAD_POD. Confirm healthy v2 and correct Service responses. The experiment tests a selector mismatch, not resource exhaustion or every possible scheduling failure.

How to Clean Up

After saving your evidence, delete the disposable lab:

bash cleanup.sh

The script selects Colima's Docker context and deletes fcc-rollout-lab, passing the lab's kubeconfig explicitly. It records the result and checks that other kind cluster names remain unchanged. This removes the lab workloads.

Evidence archives remain in the local companion directory. To run the tutorial again, use a fresh extraction after cleanup; setup intentionally refuses to reuse the old kubeconfig.

Conclusion

Three different faults stopped the new revision while the previous revision continued answering the recorded requests:

For your next stalled rollout, follow ownership from the Deployment to its ReplicaSet and Pod. Check whether the Pod was scheduled, whether the container started, and whether it became Ready. Use its conditions and Events to locate the failure, then use Service responses to establish which Pods actually answered.

Finally, distinguish a client watch timeout from a controller progress deadline, and verify the full recovery before drawing conclusions. These results came from a single-node local lab and finite HTTP samples. They don't establish production behavior or continuous availability.

The support channel has the same complaint arriving in six different tones. Is it down? Mine is just spinning. It worked this morning. Somebody has posted a screenshot of a loading indicator with no caption, which you suspect they're enjoying.

So you open the dashboard.

Nodes ready. Pods running. CPU at 20%. No restarts, nothing in CrashLoopBackOff, no alert fired. By every signal Kubernetes gives you, the system is healthy.

It is not healthy. Somebody is eleven seconds into waiting for the first word of an answer, and they're about to go back to doing it the old way, and they're not coming back.

That is the failure this piece is about. Not the outage. The quiet one, where everything is technically up and the product is still unusable.

Table of Contents

• Your First Instinct is Wrong

• You Are Not the Only One

• The Thing Nobody Told You When You Deployed It

• Where the Eleven Seconds Went

  • Stage One, Routing: Your Load Balancer is Guessing

  • Stage Two, The Queue: Nothing is Coming to Help

  • Stage Three, Prefill: The GPUs Are There, and Useless

  • Stage Four, Decode: Your Ingress is Cutting People Off

  • And Underneath All Four: Bursty Traffic and Nobody's Name on the Problem

• Meanwhile, the Platform Did Not Stand Still

• How This Got Here in the First Place

• What To Do Once the Fire is Out

  • Capacity

  • Performance

  • Placement

  • Resilience

  • Ownership

• Four Things Your Monitoring Still Isn't Telling You

• The Thing That is Still Unsettled

• Monday Morning

• Back to Thursday

Your First Instinct is Wrong

The instinct at that moment is to suspect the model. Wrong version, bad quantization, context too long, or that someone changed the system prompt.

It is almost never the model.

Walk one request end to end and you find four stages. It gets routed to a replica. It waits in a queue. The model reads the prompt. The model streams an answer. Only the last two involve the model doing anything you are paying for. The first two are queueing and routing, which is to say infrastructure.

When people say their model is slow, the model is usually fine. The wait is somewhere else.

You Are Not the Only One

It's worth knowing before you go hunting that this is the most common complaint in the field, not an unlucky configuration on your part.

A practitioner survey of 200 people running AI in production found that nearly half, 49.5%, named latency at peak load as their single hardest scaling problem. Not accuracy. Not hallucination. Not cost per token. Latency, at the exact moment people are trying to use the thing.

Two more figures from the same research are worth holding alongside it. 59.5% said running inference closer to users or decision points is critical or very important. 45.5% still serve from a single cloud region. People know proximity matters and have not managed to act on it, which tells you something about what multi-region GPU capacity costs to stand up.

The same research mentions organizations mandating sub-250ms response times alongside 99.9% availability. Read that carefully, because taken literally it is impossible. No meaningful LLM response completes in a quarter of a second. It has to mean time to first token (how long before the first word of the answer appears), and that is the right thing to hold yourself to anyway. Once tokens are flowing at a readable pace, people stop counting. The silence before the first one is what loses them.

The Thing Nobody Told You When You Deployed It

Here's the part that explains everything else.

A normal web request is short, small, and costs roughly what the last one cost. Kubernetes scheduling assumes that. Service load balancing assumes it. The Horizontal Pod Autoscaler assumes it. Your ingress controller's default timeout assumes it. Nobody wrote the assumption down, because for fifteen years it was simply true.

An LLM request breaks it in three places.

It runs in two phases with completely different profiles. Prefill: the model reads the entire prompt in one pass. Compute-bound, bursty, and this is what decides how long your user stares at nothing. Then decode: one token at a time, each needing the accumulated state of every token before it. That state is the KV cache. It lives in GPU memory next to the weights, and it grows as the answer grows.

So requests are not short, they can run for a minute. They do not cost the same, since a prompt with a document pasted into it can cost fifty times what a one-liner costs. And the genuinely scarce resource is GPU memory, which does not appear on a single dashboard you inherited.

That is why your monitoring said everything was fine. It was measuring the wrong machine.

Where the Eleven Seconds Went

Follow the request through the four stages and the failure modes fall out in order. These six are the ones practitioners name most often, and they map neatly onto the path.

Stage One, Routing: Your Load Balancer is Guessing

Round-robin stops being fair the moment request costs diverge. Three heavy prompts land on one pod while its neighbour handles one-liners. The loaded pod's queue grows, its tail latency climbs, and the fleet average still looks completely reasonable, which is why nobody noticed before today.

Long-lived connections make it worse. With HTTP/2 or keepalive, a client opens one connection and sends everything down it, and a Kubernetes Service balances per connection rather than per request. All that traffic pins to a single backend and stays there.

Then there is the cost most teams have never considered. If a replica already holds the prefix of this prompt in its KV cache, routing there skips real prefill work. Since most production prompts share a long system preamble, that is not an edge case, it is most of your traffic. Round-robin throws the benefit away at random, every time.

Stage Two, The Queue: Nothing is Coming to Help

Your autoscaler could add capacity. It will be late, and the reason is unglamorous: a new replica has to pull a multi-gigabyte serving image, fetch 20GB to 70GB of weights over the network, and load them into GPU memory. Minutes, not seconds, and that assumes a node with a free card already exists. If it does not, add provisioning time, and add whether your provider has stock in that zone today.

There is a second problem stacked on the first. The signal is usually wrong. CPU tells you nothing here, because the CPU is idle while the GPU works. GPU utilization is barely better: it reports that a kernel was executing during the sample window, and a server handling one request and a server handling sixty both read 100%. It cannot tell you whether anyone is waiting.

Queue depth can. vLLM already exposes running and waiting request counts as Prometheus metrics. Scale on those:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-server
spec:
  scaleTargetRef:
    name: llm-server
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(vllm:num_requests_waiting{app="llm-server"})
        threshold: "5"

Check the metric names against your vLLM version. Several have been renamed across releases, and the failure is silent.

Two values there are deliberate. The minimum of 2, because a cold start means you cannot sit at zero and expect to serve. The ten minute cooldown, because scaling down eagerly means paying that cold start again shortly afterwards.

Which leads to the uncomfortable conclusion about autoscaling inference. Reactive scaling is permanently late by however long your cold start is, and no trigger fixes that. The fix is to keep more spare capacity than feels comfortable. Run extra replicas at all times, scale up earlier than you think you need to, and scale down more slowly. This also means scale to zero isn't a good fit for user-facing inference. Save it for batch jobs, where nobody is sitting there waiting for a response.

Stage Three, Prefill: The GPUs Are There, and Useless

Kubernetes hands out CPU in millicores and GPUs whole. A pod asks for one card and gets the entire card, whether it needs 10% of it or all of it.

The waste is the obvious problem. Fragmentation is the one that ruins a Thursday. A 70B model in 16-bit needs roughly 140GB of weights, so a replica needs several GPUs on one node. You can have six free GPUs spread two-two-one-one across four nodes, and the replica sits in Pending indefinitely. Capacity on paper, none of it usable, and no alert, because nothing is broken.

Sharing a card gives three options, each with a catch you should know before choosing. Time-slicing is easy to enable and gives no memory isolation, so one greedy pod can take down its neighbour. Fine for development, not for anything customer-facing. MIG gives real hardware isolation but fixed slice geometry, which means predicting your workload mix before you have one. Dynamic Resource Allocation is the structurally correct answer and brings accelerator requests properly into the core Kubernetes API.

None of these are on by default. Each needs a deliberate decision from somebody who understands the workload, who is usually not in the room when the cluster gets built.

Stage Four, Decode: Your Ingress is Cutting People Off

Many ingress controllers ship with a 60 second read timeout and response buffering enabled.

The timeout terminates long generations mid-sentence. Users report that as a crash, and you cannot reproduce it, because your test prompt is short. The buffering holds tokens and releases them in clumps, which destroys the feel of streaming while the model is behaving perfectly.

Two config lines, written for a workload that no longer exists.

And Underneath All four: Bursty Traffic and Nobody's Name on the Problem

Internal AI tools don't get steady traffic. They get sudden spikes: when everyone starts work at 9am, in the hour after a company all-hands meeting, or when someone shares the link in a busy Slack channel and says it's actually quite good. A fixed replica count handles exactly one of those. Most teams set it once during a quiet week and never revisit it, so they get both waste and collapse on the same day.

Then the sixth failure mode, which is not technical at all, and which is the reason the other five survive for quarters.

Platform engineering owns the cluster. The cluster is green. From where they sit, the job is done and done well. Application developers can see latency is bad and cannot see why, because every cause sits in scheduling, scaling and routing that they neither control nor observe. MLOps owns the model artifact and almost nothing that determines how it performs at quarter past two on a Thursday.

Three teams, all telling the truth, and the problem living in the gap between their on-call rotas, where no alert is configured and no dashboard points.

Meanwhile, the Platform Did Not Stand Still

The encouraging part of this story is that the gaps above are known, and the fixes have been shipping.

Dynamic Resource Allocation brings accelerator hardware requests into the core Kubernetes API, so the scheduler can reason about the device rather than counting opaque units. Kueue adds job queueing and multi-tenant GPU quota alongside CPU and memory, which is how you stop a research job starving the endpoint your users depend on. Gateway API Inference Extension and llm-d add model-aware routing: request criticality, dynamic balancing from live model metrics, and awareness of cache state. That last one is the direct answer to the guessing load balancer in stage one.

Published benchmarks give a sense of scale: Kueue reducing multi-stage workload makespan by up to 15%, dynamic accelerator slicing cutting mean job completion time by 36%, and Gateway API plus llm-d improving tail time to first token by up to 90% under heavy load.

Treat those as headroom rather than forecast. Up to 90% is measured on a configuration chosen to demonstrate the improvement, and your gain depends entirely on how poor your baseline is. If you already run least-outstanding-requests balancing with a warm cache, expect far less. If you are on stock round-robin behind a default ingress, possibly something dramatic, because that baseline is genuinely bad.

The direction is what matters. The biggest published gain is in tail time to first token, which is precisely what half the surveyed practitioners named as their worst problem. Somebody built the fix for the thing people were complaining about.

How This Got Here in the First Place

Worth a short detour, because it explains why the tooling arrived late.

When generative AI landed in enterprise planning documents, the confident position was that Kubernetes could not hold it. Container orchestration was built for small fungible units of CPU and memory, cheap restarts, interchangeable replicas. AI needed scheduled specialized silicon carrying enormous state that takes minutes to place. A purpose-built platform was coming.

It never arrived. AI went into the microservices stack and Kubernetes absorbed it.

The survey data explains why. More than half of enterprises do not train models at all, and only 7% deploy a model on any given day. Meanwhile 82% of container users run Kubernetes in production, and 66% of organizations hosting generative AI already serve it there. The industry spent years designing for the workload almost nobody has, while everybody else quietly downloaded weights and put them behind an endpoint.

What actually converged was not a place to run AI. It was a standard way to run it, with the location left negotiable. Which is exactly why your problems today are placement, routing and capacity rather than anything model-shaped.

What To Do Once the Fire is Out

The path from working to reliable comes down to five things: capacity, performance, placement, resilience and ownership.

Capacity

Requests per second is meaningless when one request is 50 tokens and the next is 8,000. Plan in tokens per second and track prompt and completion separately, because they stress different parts of the system. Prompt tokens are a compute burst during prefill. Completion tokens are a sustained drip during decode that holds GPU memory for the life of the response.

Load test with your worst realistic prompt at your worst realistic concurrency, and plan from that number. A benchmark built on short prompts gives a reassuring figure that evaporates in week one.

On supply, treat GPUs as something you reserve ahead of time rather than request when needed. Keep a committed baseline and burst on top. Find out your provider's quota and regional stock position before the evening you need it, and use Kueue so one team cannot quietly consume the pool.

Performance

Two numbers matter most. Time to first token (TTFT) is how long a user waits between sending a prompt and seeing the first word of the answer. It decides whether people trust the product. Inter-token latency is the gap between each word after that, and it decides whether the answer feels smooth to read.

Track both at p95 and p99, not the average. These are percentiles. p95 is the time that 95% of requests come in under, so it shows what your slowest 1 in 20 users experience. p99 does the same for the slowest 1 in 100. An average can look healthy while those users wait far too long, and they're the ones who give up.

Next, tune the serving layer. This is the software that loads the model and handles requests, such as vLLM. It's where you'll get the biggest improvements for the least effort and cost. Continuous batching keeps the GPU busy without making early requests wait for a batch to fill. Quantization roughly halves memory footprint where the quality tradeoff is acceptable, and freed memory becomes concurrent requests. Set maximum context length to what your product actually needs rather than what the model supports. If the model handles 128K and your longest genuine prompt is 6K, you are reserving KV cache for a scenario that never occurs. One config line, and it can substantially raise effective concurrency.

Placement

Taint GPU nodes so general workloads cannot land on them and only inference pods with matching tolerations schedule there. A logging sidecar should never be the reason a model replica cannot place.

Keep weights close. Node-local NVMe or a zone-local cache turns a 40GB network pull into a fast local read, which feeds directly into cold start, autoscaling lag, and the peak-hour latency you spent Thursday afternoon on. This is the highest-leverage unglamorous fix on the list.

Respect topology. Cards connected by NVLink behave very differently from cards that merely share a PCIe bus, and for a sharded model that interconnect sits on the critical path of every token generated.

Then be honest about geography. If your users are in London and your GPUs are in Virginia, every token crosses an ocean, and a well-tuned model behind a long network path is still a slow product.

Resilience

Readiness should pass only when the model is loaded and genuinely able to serve. Use a startup probe with a generous failure threshold, otherwise liveness kills the pod partway through loading weights and you get a restart loop that presents as a mystery and costs an hour.

Set a PodDisruptionBudget so a routine node upgrade cannot remove half your replicas at once.

Raise the termination grace period past the 30 second default and add a preStop hook. A generation can easily run longer than that, so otherwise every rollout severs in-flight streams and your users experience deployments as random failure.

Shed load deliberately. Past a queue depth threshold, return a fast busy-try-again rather than accepting a request that will hang for two minutes. This feels wrong to engineers and is right for users. People forgive a quick honest retry. Nobody forgives a frozen screen.

Have somewhere to fall back to: a second region, a smaller model covering the common cases, or an external API at the edge of disaster. Given that 45.5% of practitioners run from a single region, this is the most commonly skipped item on the list, which makes it the most likely candidate for your next incident.

Ownership

Agree on the numbers and write them down: time to first token, inter-token latency, error rate, queue time. Put them on one dashboard showing cluster and model metrics side by side, and make sure both teams open the same link rather than maintaining separate versions of reality.

Split responsibility explicitly. Platform owns the ability to hit the target: capacity, scheduling, routing, scaling, placement. Application owns how the model is configured and called: context length, batching parameters, prompt size, retry behaviour. When the number slips, both show up, and the conversation starts from the same graph instead of two that disagree.

Four Things Your Monitoring Still Isn't Telling You

Latency, traffic, errors and saturation are still necessary and no longer sufficient. Four more belong on the board.

Token consumption rate, split prompt and completion, because that is your real unit of capacity. Cost per request by tenant and query type, because we run eight GPUs is not an answer to what a feature costs per user. Output quality, tracking hallucination rate and guardrail drift, because latency work that degrades quality is not a win. Model attribution, recording which checkpoint and which prompt version produced a response, because when quality moves you need to know what changed.

That last one connects to a shift worth noticing. Model registries have become peers to container registries, and prompts and agent configurations are increasingly versioned as GitOps artifacts in the same delivery pipelines as everything else.

The prompt is code now. It changes production behaviour, it can break things, and it should go through review, versioning and rollback like anything else that does. Plenty of teams are still editing prompts in a web console and wondering why quality shifted on a Tuesday.

The Thing That is Still Unsettled

The substrate has converged, and the speed of it is the evidence. A Kubernetes AI conformance programme launched in late 2025 with 18 platforms. Within roughly four months it had grown to 31 and extended to agentic workloads, covering the major hyperscalers, the enterprise and private cloud distributions, GPU neoclouds and edge networks. Companies that agree on almost nothing agreed on this.

The layer above has not converged at all. AI gateways, evaluation platforms, agent control planes, agent observability. All advancing faster in commercial products than in any standard body, and all sitting exactly where the interesting business logic is heading.

The risk is worth naming plainly. You can be perfectly portable at the Kubernetes layer and completely locked in at the layer where you actually build. Your pods will migrate between clouds beautifully while your agent definitions, eval suites and gateway policies stay exactly where they are, because there is nowhere standard to move them to.

That is not a reason to avoid those tools. They solve real problems today. It is a reason to know which of your components has an exit and which does not, and to make that a decision you wrote down rather than one you discover during a renewal negotiation.

Monday Morning

Start with one measurement. Time a cold start end to end, from pod created to first token served, and write the number down. That figure is your autoscaling floor, and your minimum replica count falls directly out of it.

Put time to first token and queue depth on your main dashboard at p99. Replace CPU-based autoscaling with a queue-depth trigger and minimums that respect the cold start. Check your ingress read timeout and buffering against a real long streaming response rather than a test prompt. Fix the termination grace period so deployments stop cutting people off.

Then, over the next few weeks: taint your GPU nodes, move weights to node-local or zone-local storage, set maximum context length to what you actually use, and find out whether you are still on round-robin. You probably are.

And before the quarter ends, decide who owns the end-to-end latency number, and say it out loud in a room with the other team present.

Back to Thursday

The dashboard was never lying. It was answering a different question.

It was telling you whether the containers were alive, which they were. It could not tell you that requests were queueing behind a badly routed batch, that no new replica was coming for four minutes, that six GPUs were free and unusable, or that the ingress was buffering tokens it should have been streaming. Nothing in the standard toolkit is pointed at any of that.

Getting a model to answer on Kubernetes takes an afternoon. Getting it to answer well when everyone logs in at once is a different discipline, and it looks far more like ordinary infrastructure engineering than the current conversation suggests. Scheduling, routing, capacity, ownership. We have been solving those since long before any of this carried the AI label. The skills are already in the building. They need pointing at the right metric.

A green cluster is where the work starts. What counts is what the person typing the prompt sees.

I hope you’ve enjoyed this and learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving stars on GitHub and endorsing me for relevant skills on LinkedIn.

Till the next one, happy exploring!

I've debugged Kubernetes networking issues at more than 10 companies. The same knowledge gaps appear every time. Engineers don't understand how ClusterIP works under the hood. They don't understand why Pods in different namespaces can talk to each other by default. And they don't understand what a CNI plugin actually does at the kernel level.

This tutorial is the fix. You'll learn how Kubernetes networking works from the bottom up: how Pod IPs are assigned and why they work across nodes, how kube-proxy implements ClusterIP using iptables rules, how Ingress controllers route external traffic through a single load balancer, how Network Policies enforce micro-segmentation for SOC2 compliance, and how Cilium uses eBPF to replace all of this with a faster, more observable, and more secure alternative.

By the end of this guide, you'll be able to debug "why can't my pod talk to that service?", implement default-deny Network Policies that satisfy SOC2 CC6.1, and choose the right CNI for your cluster with confidence.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: Pod IPs and the Container Network Model

• Part 2: Services — ClusterIP, NodePort, and LoadBalancer

• Part 3: Ingress — External Traffic Routing

• Part 4: Network Policies — Micro-Segmentation

• Part 5: CNI Comparison — Cilium vs Calico vs AWS VPC CNI

• Part 6: Service Mesh — Cilium vs Istio vs Linkerd

• Best Practices Summary

• Resources

What You'll Learn

• Pod IPs, the container network model, and how the CNI assigns addresses

• How kube-proxy implements ClusterIP with iptables and why eBPF is faster

• Ingress controllers: routing all external traffic through a single load balancer

• Network Policies: default-deny and per-service allow rules for zero-trust networking

• CNI comparison: Cilium vs Calico vs AWS VPC CNI and when to use each

• Service mesh: Cilium vs Istio vs Linkerd for mTLS and observability

Let's dive in.

Prerequisites

Before following along, you should have:

Knowledge:

• Basic Kubernetes familiarity: you can deploy a Pod and create a Service

• Basic Linux networking concepts: you know what an IP address and a port are

• A general understanding of what a load balancer does

Tools and access:

• A running Kubernetes cluster (EKS, GKE, or a local cluster via kind)

• kubectl configured and pointing at your cluster

• helm 3 installed (for Cilium installation in Part 4)

• For Part 4 onwards: Cilium installed on your cluster (helm install cilium cilium/cilium)

A note on CNI: Parts 1–3 apply to any Kubernetes cluster regardless of CNI. Parts 4–6 use Cilium-specific resources (CiliumNetworkPolicy, Hubble). If you're on a different CNI, the concepts are identical and only the YAML syntax differs.

Part 1: Pod IPs and the Container Network Model

1.1 Why Every Pod Gets Its Own IP

The Kubernetes networking model has one foundational rule: every Pod gets its own unique IP address, and every Pod can communicate with every other Pod using those IPs – without Network Address Translation (NAT).

This is different from how Docker works by default, where containers share the host network or use port mapping. In Kubernetes, there's no port mapping between pods. Pod A at IP 10.244.1.2 can directly reach Pod B at 10.244.2.3 across a different node, and the source IP is preserved.

Verify this for your cluster:

# List all pods across all namespaces with their IP addresses and node placement
kubectl get pods -o wide --all-namespaces

Expected output:

NAMESPACE     NAME                                READY   STATUS    IP            NODE
production    payment-api-5d6b8d8c4f-abc12        1/1     Running   10.244.1.2    node-1
production    user-api-5d6b8d8c4f-def34           1/1     Running   10.244.2.3    node-2
production    redis-master-0                      1/1     Running   10.244.1.4    node-1

Each pod has a unique IP. The payment-api on node-1 and the user-api on node-2 can reach each other directly at those IPs. Notice that the IPs come from the 10.244.0.0/16 CIDR: this is the Pod network, separate from the node network.

1.2 What the CNI Plugin Actually Does

The Container Network Interface (CNI) is the plugin responsible for making the Kubernetes networking model work. When a new Pod is scheduled on a node, the Kubernetes kubelet calls the CNI plugin, which performs four operations:

1. Creates a new network namespace for the Pod: an isolated networking environment

2. Creates a virtual Ethernet pair (veth): one end inside the Pod's namespace, one end on the node

3. Assigns an IP address from the cluster's Pod CIDR to the Pod's end of the veth pair

4. Adds routing rules so the node knows how to reach every Pod IP in the cluster

Without the CNI, pods would have no network connectivity. With it, the flat Pod network model becomes reality.

Check which CNI plugin is installed on your cluster:

# List the CNI binaries installed on a node
ls /opt/cni/bin/

Here are some common CNI plugins and when to use each:

1.3 Verifying Pod-to-Pod Communication

The most fundamental networking test: exec into one Pod and ping another by IP.

# Step 1: Get the IP of a target pod
TARGET_IP=$(kubectl get pod redis-master-0 -o jsonpath='{.status.podIP}')
echo "Target IP: $TARGET_IP"

# Step 2: Exec into another pod and ping the target
kubectl exec -it payment-api-5d6b8d8c4f-abc12 -- ping -c 3 $TARGET_IP

Expected output:

PING 10.244.1.4 (10.244.1.4): 56 data bytes
64 bytes from 10.244.1.4: icmp_seq=0 ttl=62 time=0.8ms
64 bytes from 10.244.1.4: icmp_seq=1 ttl=62 time=0.7ms
64 bytes from 10.244.1.4: icmp_seq=2 ttl=62 time=0.9ms

If this succeeds, the CNI is working correctly. If it fails, check whether a Network Policy is blocking ICMP traffic (Part 4 covers this).

The one rule to remember: every Pod gets an IP. Pods can communicate directly using those IPs. The CNI plugin makes both of these things true.

Part 2: Services — ClusterIP, NodePort, and LoadBalancer

2.1 The Problem: Pod IPs Are Not Stable

Pod IPs change every time a Pod restarts. If you deploy a new version of your payment API, the old Pods are deleted and new Pods are created with new IPs. Any service that was configured to call the old IPs now has dead references.

Here's the incorrect approach: hardcoding a Pod IP.

# Bad: Direct Pod IP in application configuration
# This IP will stop working the next time the database Pod restarts
apiVersion: v1
kind: Pod
metadata:
  name: payment-api
spec:
  containers:
  - name: api
    env:
    - name: DATABASE_HOST
      value: "10.244.1.4"  # Pod IP — will change on next restart

This is fragile in development and catastrophic in production. A routine Pod restart – from a node drain, an OOM kill, or a deployment rollout – will break any application that hardcoded the old IP.

2.2 How Services Solve the Stability Problem

A Kubernetes Service provides two things that Pod IPs can't: a stable IP address (the ClusterIP) that never changes as long as the Service exists, and a stable DNS name that other Pods can use regardless of the IP.

When you create a Service, Kubernetes assigns it a virtual ClusterIP from the service CIDR (for example, 10.100.0.0/16), creates a DNS record in CoreDNS as <service-name>.<namespace>.svc.cluster.local, and configures kube-proxy on every node to add iptables rules that load-balance traffic from the ClusterIP to the healthy Pod IPs behind it.

Here's the correct implementation: a ClusterIP Service.

# Good: ClusterIP Service provides a stable IP and DNS name
# redis.production.svc.cluster.local always resolves to 10.100.0.1
# regardless of which Redis pods are running behind it
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: production
spec:
  selector:
    app: redis
    role: master   # Only pods with these labels receive traffic
  ports:
  - port: 6379        # Port the Service listens on
    targetPort: 6379  # Port the Pod actually runs on
  type: ClusterIP     # Default: accessible only inside the cluster

How kube-proxy implements the load balancing using iptables: when a Service is created, kube-proxy adds iptables rules to every node in the cluster. These rules intercept traffic destined for the ClusterIP and redirect it to one of the healthy Pod IPs. Run this on a node to see the rules in action:

# View the iptables rules kube-proxy created for the redis Service
# Each KUBE-SEP entry represents one Pod endpoint
sudo iptables -t nat -L KUBE-SERVICES | grep redis

Expected output:

Chain KUBE-SVC-REDIS (1 references)
target          prot  source    destination
KUBE-SEP-AAA    all   anywhere  anywhere    /* production/redis */ statistic mode random probability 0.50
KUBE-SEP-BBB    all   anywhere  anywhere    /* production/redis */

Traffic to the Redis ClusterIP is distributed 50/50 between the two Pod endpoints via these iptables rules. When a Pod restarts and gets a new IP, kube-proxy updates the rules automatically.

2.3 When to Use Each Service Type

Verify a Service is routing traffic correctly:

# Describe a Service to see its endpoints (the actual Pod IPs behind it)
kubectl describe service redis -n production

Expected output:

Name:              redis
Namespace:         production
Type:              ClusterIP
IP:                10.100.0.1
Port:              6379/TCP
TargetPort:        6379/TCP
Endpoints:         10.244.1.4:6379,10.244.2.5:6379
Session Affinity:  None

If Endpoints shows <none>, the Service selector doesn't match any running Pods. This is the most common cause of "connection refused" errors in Kubernetes.

The one rule to remember is that pods should always connect to Service DNS names, never to Pod IPs. The Service handles stability, load balancing, and health checking automatically.

Part 3: Ingress — External Traffic Routing

3.1 The Problem: A LoadBalancer Service Per Microservice Is Expensive

Each LoadBalancer Service creates a dedicated cloud load balancer. On AWS, each Application Load Balancer costs approximately \(0.008/LCU-hour plus \)0.0225/hour base charge. That's roughly $16–27/month per load balancer.

At 20 microservices, that's $320–$540/month in load balancer charges alone, plus $0.008/LCU for each request processed.

Here's the incorrect approach with one LoadBalancer per microservice:

# Bad: This creates a new AWS ALB every time it is applied
# 20 microservices = 20 ALBs = $300-500/month before any traffic charges
apiVersion: v1
kind: Service
metadata:
  name: payment-api
spec:
  type: LoadBalancer   # Creates a dedicated ALB
  ports:
  - port: 80
    targetPort: 8080

3.2 How an Ingress Controller Solves This

An Ingress controller is a Pod running inside your cluster that watches for Ingress resources and programs a single external load balancer to route traffic to multiple Services based on the hostname and URL path.

The AWS Load Balancer Controller, for example, creates one ALB for all your Ingress resources and programs its listener rules to route api.company.com/payments to the payment Service and api.company.com/users to the user Service, all through the same load balancer.

Here's the correct implementation: one Ingress for all services.

# Good: One Ingress resource routes all external traffic
# One ALB is created total, regardless of how many services are listed
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shared-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
spec:
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /payments
        pathType: Prefix
        backend:
          service:
            name: payment-service
            port:
              number: 8080
      - path: /users
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 8080
  - host: dashboard.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboard-service
            port:
              number: 3000
  tls:
  - hosts:
    - api.company.com
    - dashboard.company.com
    secretName: tls-wildcard-cert

Verify the Ingress is provisioned and the ALB DNS name is assigned:

# Watch until the ADDRESS column shows the ALB DNS name (typically 2-3 minutes)
kubectl get ingress shared-ingress -n production -w

Expected output:

NAME             CLASS   HOSTS                                    ADDRESS                                              PORTS
shared-ingress   alb     api.company.com,dashboard.company.com   k8s-prod-sharedin-abc123.us-east-1.elb.amazonaws.com   80, 443

The cost difference:

The one rule to remember: one Ingress controller with path-based routing serves all your services through a single load balancer. The per-service LoadBalancer approach is for early prototyping only.

Part 4: Network Policies — Micro-Segmentation

4.1 The Default: Every Pod Can Talk to Every Other Pod

Out of the box, Kubernetes applies no network restrictions between Pods. A frontend Pod can make direct API calls to a database Pod. An analytics service can query the payment database. A compromised Pod can scan every other Pod in the cluster.

This isn't secure. For SOC2 CC6.1 (logical access controls), HIPAA, and most enterprise security frameworks, you need to be able to prove that network traffic is restricted to what's necessary.

Verify that unrestricted traffic is currently possible:

# Without Network Policies, this call from the frontend to the payment DB will succeed
# It should not be allowed in a secure cluster
kubectl exec -it frontend-pod -n production -- \
  curl http://payment-postgres.production.svc.cluster.local:5432

If this succeeds on your cluster, you have no network segmentation.

4.2 The Solution: Default-Deny with Cilium Network Policies

The correct approach is default-deny: block all traffic between Pods first, then explicitly allow only the specific communication paths that your application requires.

Step 1 — Apply the default-deny policy:

# This policy applies to all pods in the namespace (empty endpointSelector matches all)
# It blocks all ingress and egress traffic by default
# Warning: apply this and all pod-to-pod communication immediately stops
# Have your allow rules ready before applying this in production
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  description: "Block all inter-pod traffic by default — zero-trust baseline"
  endpointSelector: {}  # Matches all pods in this namespace
  ingress:
  - {}                  # Empty ingress rule = deny all inbound
  egress:
  - {}                  # Empty egress rule = deny all outbound

Applying the default-deny policy will break all pod-to-pod communication in the namespace immediately. Apply your allow rules (below) in the same kubectl apply command, or apply allow rules first.

Step 2 — Add namespace-level isolation:

# Allow pods to communicate within the same namespace
# Block cross-namespace traffic by default
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: production
spec:
  endpointSelector: {}
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: production
  egress:
  - toEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: production

Step 3 — Add per-service allow rules:

# Grant the payment service only the specific network access it needs
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-service-network-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  egress:
  # Allow: payment-service → postgres on port 5432
  - toEndpoints:
    - matchLabels:
        app: postgres-db
    toPorts:
    - ports:
      - port: "5432"
        protocol: TCP
  # Allow: payment-service → Stripe API externally
  - toFQDNs:
    - matchName: "api.stripe.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP

4.3 Using Hubble to Verify Policies and Collect SOC2 Evidence

Cilium includes Hubble, a network observability tool that shows you exactly which flows are being allowed and which are being dropped by your Network Policies. Hubble is your SOC2 evidence that network segmentation is operating correctly.

# Install the Hubble CLI
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
tar xzvf hubble-linux-amd64.tar.gz
sudo mv hubble /usr/local/bin/

# Port-forward to the Hubble relay
kubectl port-forward -n kube-system svc/hubble-relay 4245:80 &

# Show all flows in the production namespace from the last hour
hubble observe --namespace production --since 1h

# Show only dropped flows — proves policies are blocking unauthorised traffic
hubble observe --namespace production --verdict DROPPED --since 1h

Example Hubble output showing a blocked connection attempt:

Apr 19 03:17:41.234   DROPPED   TCP   10.244.1.5:52341 → 10.244.1.4:5432   policy-deny
Apr 19 03:17:41.235   ALLOWED   TCP   10.244.1.2:43211 → 10.244.1.4:5432   allow-same-namespace

The first line shows an unauthorized connection attempt blocked. The second shows a legitimate connection allowed. Export this log daily to your SOC2 evidence bucket.

The one rule to remember: default-deny is the zero-trust baseline. Then add explicit allow rules for every required communication path. Hubble gives you the evidence that it's working.

Part 5: CNI Comparison — Cilium vs Calico vs AWS VPC CNI

Choosing the right CNI is a decision that's difficult to reverse. Migrating between CNIs requires draining and replacing every node in the cluster. Make the decision once, for the right reasons.

Here's a real comparison across the capabilities that matter for production EKS clusters:

The recommendation matrix:

The one rule to remember: for SOC2 compliance and zero-trust networking on EKS, Cilium is the right choice. It provides pod-level isolation, Layer 7 policies, and Hubble flow logs that serve as audit evidence. These are capabilities no other CNI provides together.

Part 6: Service Mesh — Cilium vs Istio vs Linkerd

6.1 What a Service Mesh Provides

A service mesh adds three capabilities to your cluster's networking that Kubernetes doesn't provide natively.

mTLS (mutual TLS) encrypts communication between every pair of services and verifies both sides' identities. Without mTLS, traffic between your payment service and your database travels in plaintext inside the cluster.

Traffic observability tracks request rates, latency percentiles, and error rates for every service-to-service call, giving you a real-time performance map of your application.

Traffic management controls how traffic flows: retries on failure, timeouts, circuit breaking when a downstream service is degraded, and traffic splitting for canary deployments.

6.2 The Sidecar Problem

Traditional service meshes (like Istio and Linkerd) inject a sidecar proxy container into every Pod. This sidecar intercepts all network traffic and applies the mesh policies. The problem is resource overhead: Istio's Envoy sidecar adds approximately 128MB of memory and 5–10% latency overhead per Pod.

On a cluster with 200 Pods, Istio sidecars add 25.6GB of memory overhead and measurable latency to every service call.

Cilium solves this differently. It implements the service mesh at the kernel level using eBPF without any sidecar at all.

6.3 The Full Comparison

The recommendation matrix:

Enable Cilium's service mesh mode (no sidecars required):

# Upgrade your Cilium installation to enable service mesh features
helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set envoy.enabled=true \
  --set ingressController.enabled=true

# Verify the service mesh is active
cilium status | grep "Service Mesh"

The one rule to remember: Cilium gives you mTLS and SOC2 evidence with zero sidecar overhead. For teams that need advanced traffic management or complex canary release patterns, Istio provides more control at the cost of higher operational complexity.

Best Practices for Kubernetes Networking

✅ Do: Use Services, not Pod IPs. Pod IPs change on every restart. Service DNS names never change.

✅ Do: Use a single Ingress controller with path-based routing. One ALB serves all your services and saves $300–$400/month versus per-service LoadBalancer.

✅ Do: Implement default-deny Network Policies with Cilium. This is the technical control required by SOC2 CC6.1.

✅ Do: Use Hubble flow logs as SOC2 evidence. Export daily dropped-flow logs to your evidence bucket.

✅ Do: Enable mTLS with Cilium for encrypted service-to-service communication. No sidecar required.

✅ Do: Use topology-aware routing to keep traffic within the same Availability Zone and reduce cross-AZ data transfer costs.

❌ Don't: Create a LoadBalancer Service for every microservice. Use Ingress for external routing.

❌ Don't: Rely on Security Groups alone for pod-level isolation. Security Groups work at the node level. Any pod on a node shares the node's security group. Network Policies work at the pod level.

❌ Don't: Assume the default "allow all" pod networking is secure. Apply default-deny before your first enterprise customer asks for your network segmentation diagram.

Resources

• Cilium Documentation: Official Cilium installation guide, CiliumNetworkPolicy reference, and Hubble observability documentation

• Cilium Service Mesh Guide: How to enable mTLS and Layer 7 policies without sidecars

• Kubernetes Network Policy Documentation: The standard Kubernetes NetworkPolicy API reference

• AWS Load Balancer Controller: Official documentation for the Ingress controller that provisions AWS ALBs from Kubernetes Ingress resources

• Hubble CLI Installation: Install the Hubble CLI for observing Cilium network flows

• kube-proxy iptables mode: Kubernetes documentation explaining how kube-proxy implements Service routing using iptables

• Kubernetes CNI Plugin Specification: The CNI interface specification that all CNI plugins implement

• AWS VPC CNI Plugin GitHub: Source code and documentation for the default EKS networking plugin

• Companion Repository: CiliumNetworkPolicy manifests and Hubble evidence export scripts from this guide

Between those two moments runs a long chain of machinery: builds, artefacts, migrations, health checks, traffic shifts. Every production engineer depends on that chain, and many teams still build and operate it themselves.

Deployment infrastructure has quietly become operational overhead. It started as a technical necessity, something every team had to assemble because nothing else existed.

Today it is a second system your engineers maintain alongside the product, consuming on-call rotations, sprint capacity, and 2 a.m. attention that could go somewhere better.

In this article, we'll walk through each stage of a real production deployment: the build, the artefact it produces, database migrations, health checks, rolling updates, and rollbacks. Along the way, we'll look at why platform-as-a-service (PaaS) tools handle most of these steps for you, and what it costs a team to keep handling them itself.

Table of Contents

• The Build: Turning Code into Something That Can Run

• The Artefact: One Version, Frozen in Time

• Database Migrations: The Riskiest Step

• Health Checks: Proving the New Version Is Alive

• Rolling Updates: Replacing the Plane's Engine Mid-Flight

• Rollbacks: The Escape Hatch

• When You Don't Need a PaaS

• Should You Still Be Running This Yourself?

The Build: Turning Code into Something That Can Run

A deployment does not ship your source code as-is. It ships the result of a build. The build stage takes your code and turns it into something a server can run.

What this looks like depends on your stack. A Java or Go project gets compiled into a binary. A JavaScript front end gets bundled and minified. A Python app gets its dependencies resolved and pinned. In most modern setups, all of this gets packed into a container image, which is a frozen snapshot of your app plus everything it needs to run.

The build stage also runs your tests. Unit tests, linting, and security scans all happen here. If any of them fail, the deployment stops before it can touch production. This is the cheapest place to catch a bug. A failed build costs you a few minutes. A failed deployment can cost you customers.

Teams that run their own pipelines spend real effort here. They maintain build servers, cache dependencies, and debug flaky test runners.

None of that work ships a feature. It is pure upkeep, and it never ends. A PaaS bakes this whole stage into the platform. You push code, and the platform detects your language, builds it the same way every time, and fails fast when something is wrong. The build still happens. Your engineers just stop paying for it in hours.

The Artefact: One Version, Frozen in Time

The output of a build is called an artefact. It might be a container image, a compiled binary, or a zipped bundle. Whatever the format, the artefact has one job: to be exact. It represents one precise version of your app, frozen at one point in time.

This matters more than it sounds. The artefact that passed your tests must be the exact same one that reaches production. If you rebuild between testing and shipping, you risk shipping something slightly different. A dependency may have updated. A build flag may have changed. "It worked in staging" often means "we built it twice and got two different results."

Good pipelines build once and promote the same artefact through every stage. Artefacts get versioned and stored in a registry, so any version can be pulled and run again later. That stored history is also what makes rollbacks possible, which we will get to soon.

On a PaaS, artefact handling is standard practice by default. Every deploy produces a numbered release. The platform stores it, tracks it, and can restore it. You do not have to design a registry strategy, write promotion scripts, or assign an engineer to own them. The discipline is built in.

Database Migrations: The Riskiest Step

Before new code goes live, the database often has to change with it. Maybe the new version needs a new column or a new table. These changes are called migrations, and they are the most dangerous part of most deployments.

Why? Code is easy to replace. Data is not. If you deploy a bad code version, you can swap it out. If a migration corrupts or drops data, there may be no clean way back. Migrations also create a tricky window of time. For a few minutes, old code and new code may run against the same database at once. Both versions have to work with the schema during that window.

The safe pattern is to make migrations backwards-compatible. Add the new column first, deploy code that can handle both shapes, then clean up the old column in a later release. It takes more steps, but each step is safe on its own.

A PaaS cannot write your migrations for you. No tool can know what your data means. But a good platform gives migrations a defined place in the release process, runs them in order, and logs exactly what ran and when. That structure prevents the classic failure where someone runs a migration by hand and forgets to tell the team.

Health Checks: Proving the New Version Is Alive

Once the new version starts, the platform does not just trust it. It checks. A health check is a small endpoint in your app, often just a route that returns "OK." The platform calls it over and over. If the app answers, it is considered healthy. If it does not, the platform assumes something is wrong.

There are usually two kinds of checks. A readiness check asks, "Are you ready to receive traffic?" A liveness check asks, "Are you still working, or should I restart you?" The difference matters. An app can be alive but not ready, such as when it is still warming up a cache.

Health checks are the gatekeepers of a deployment. No traffic reaches a new version until it proves it can handle requests. Without them, you would be routing real users to an app that might still be crashing on startup.

Every serious PaaS runs health checks automatically. You define the endpoint, and the platform handles the polling, the timeouts, and the decisions. Teams that build this themselves tune all of those settings by hand, and they usually learn the right values through painful trial and error. That tuition is paid in engineering time, on a problem the industry solved years ago.

Rolling Updates: Replacing the Plane's Engine Mid-Flight

Here is the hard part. Your old version is serving live traffic right now. You need to replace it without dropping a single request. The most common answer is a rolling update.

It works like this. Say you have four copies of your app running. The platform starts one copy of the new version and waits for its health checks to pass. Then it shifts a slice of traffic to it and shuts down one old copy. It repeats this, one copy at a time, until only the new version remains. Users never notice, because at every moment there are enough healthy copies to serve everyone.

Some teams use variations of this idea. A blue-green deployment runs the full new version beside the old one, then flips all traffic at once. A canary release sends a tiny share of users to the new version first, watching for errors before going wider.

Doing this by hand means writing orchestration logic, managing load balancer rules, and handling every edge case where a step fails halfway. That is months of engineering effort to build and a permanent tax to maintain, all for behavior a PaaS ships as the default. On a platform, you get zero-downtime releases out of the box, not as a project your team has to staff.

Rollbacks: The Escape Hatch

Sometimes the new version passes every check and still breaks something real. An error rate climbs. A page loads blank. Now speed matters more than anything, and the fastest fix is rarely a new patch. It is a rollback: redeploying the previous artefact that you already know works.

This is why frozen, versioned artefacts are so important. A rollback is only fast if the old version is stored, tested, and ready to run. Teams that rebuild from an old commit under pressure are gambling at the worst possible time.

On most PaaS platforms, a rollback is one command or one click. The platform keeps your release history and can restore any previous version in seconds. That single feature has saved more on-call engineers' nights than perhaps any other.

When You Don't Need a PaaS

The case for handing deployment to a platform is strong, but it isn't universal. There are teams for whom owning the pipeline is not overhead; it is a deliberate and justified engineering decision.

When Compliance Demands It

Regulated industries like finance, healthcare, government, often operate under requirements that a standard PaaS cannot satisfy out of the box. Data residency rules may dictate exactly which physical infrastructure your builds touch. Audit requirements may demand a level of provenance and access logging that a managed platform doesn't expose.

Security controls may need to extend into the build environment itself, not just the runtime. In these contexts, the cost of owning the pipeline is real, but it is the cost of operating in that industry.

When Deployment is Your Product

If your company sells deployment infrastructure, a CI/CD platform, a release orchestration tool, an internal developer platform, then your pipeline is not overhead at all. It is the product.

The engineers maintaining it are doing product work, not distraction work. The same applies to platform engineering teams at large organizations whose explicit charter is to build and own the deployment layer for dozens of other internal teams. In both cases, the question of "why are we running this ourselves" has an obvious answer: because this is what we do.

When Your Infrastructure is Genuinely Unusual

Most PaaS platforms are optimized for stateless web services and standard container workloads. If your system falls outside that envelope, GPU clusters, real-time systems with strict latency requirements, hybrid on-premise and cloud deployments, hardware-in-the-loop testing, a general-purpose platform may simply not fit.

Shoehorning an unusual workload into a PaaS often produces more friction than building narrow, purpose-built deployment tooling around the specific constraints you actually have.

The common thread across all three cases is specificity. The teams that are right to own their pipelines can usually state clearly why a platform doesn't fit. If the answer is "we've always done it this way" or "we like having control," that's worth questioning. If the answer is "our compliance requirements mandate X" or "we sell this," that's a reason.

Should You Still Be Running This Yourself?

A PaaS does not make any of these steps disappear. The build still runs. Artefacts still get stored. Migrations still execute, health checks still poll, and traffic still shifts one copy at a time. Abstracting these mechanics does not eliminate them. It standardizes them, and pushes their maintenance onto a team whose entire product is deployment.

That is the question every product team should now ask plainly: why are we still building and operating this machinery ourselves? A decade ago, a custom pipeline was unavoidable. Today it is a choice, and for most teams it is the wrong one. Every hour spent debugging a flaky build agent, tuning a health check timeout, or patching orchestration scripts is an hour taken from the product your customers actually pay for. The pipeline does not differentiate you. It cannot. Your competitors' deploys work the same way yours do.

Know how the chain works, because on-call at 2 a.m. demands it. But knowing how it works is not a reason to own it. "We built our own deployment system" is not a badge of honor anymore. It is an admission that your team maintains a second product with no customers. Unless deployment infrastructure is your business, hand the machinery to a platform, and put your engineers back on the work only they can do.

Hope you enjoyed this article. You can connect with me on LinkedIn.

An operator extends the same pattern to resources Kubernetes doesn't know about natively, letting you manage custom, often external, systems the same declarative way you manage everything else in the cluster.

This guide is divided into four parts: what an operator actually is, the anatomy of one, building one from scratch, and preparing it for production.

Table of Contents

• Part 1: Introduction

  • What is an Operator?

  • Operator vs Controller vs CRD

  • Why Not Just a Helm Chart, a CronJob, or a Script?

• Part 2: Anatomy of an Operator

  • Custom Resource

  • Watching for Change

  • Manager

  • Reconciliation Loop

• Part 3: Building the Operator

  • Setup

  • Mock Provider

  • Defining the VirtualMachine CRD

  • Reconciler

  • Failure Handling & Retries

  • Finalizer

  • Predicate

  • Owned Resources

  • Cross-Resource Reconciliation

  • RBAC

• Part 4: Production & Deployment

  • Packaging & Deployment

  • Performance & Resilience

  • Security

  • Observability

• Next steps

Part 1: Introduction

What is an Operator?

Kubernetes works by comparing the state we describe against actual state. A controller acts to close the gap, whether that's the Deployment controller replacing a pod we killed or scaling down the ones we no longer want.

This loop of observe, compare, and act is called reconciliation. It means looking up a resource's desired and actual state, deciding what to do next, and recomputing that decision fresh on every run regardless of what changed.

That's what makes the loop resilient: it never has to trust that it saw every event, only that it gets called again.

An operator applies that exact same loop to a resource Kubernetes doesn't understand natively. We define a Custom Resource, describe our domain's desired state in it, and write a controller that knows how to reconcile that domain.

That's the whole concept. Everything else in this guide (informers, workqueues, finalizers, status conditions) exists to make that one loop reliable for a resource type Kubernetes only knows about because we defined it.

Operator vs Controller vs CRD

These three terms are often used interchangeably, but they describe three different layers of the same system.

• CRD (CustomResourceDefinition): a schema we register with the Kubernetes API server to teach it about a new resource type. On its own, a CRD does nothing. It only gives the API server a shape to store, validate, and serve.

• Controller: any piece of software running a reconciliation loop against a resource type, from the built-in Deployment controller to a custom controller reconciling a PostgresCluster.

• Operator: a controller, or a small set of controllers, that targets a custom resource and encodes enough domain-specific knowledge to manage its full lifecycle without a human: provisioning, upgrades, failure recovery, and so on.

Every operator is a controller, but not every controller is an operator. A CRD without a controller behind it is just a schema that nothing acts on.

Why Not Just a Helm Chart, a CronJob, or a Script?

A Helm chart renders a set of values into YAML and applies it once. It has no way to keep watching afterward. if a resource it created is deleted or drifts, Helm has no idea until we run helm upgrade again by hand.

A CronJob gives us a loop back, at the cost of granularity, staleness up to one interval, no state carried between runs, and no way for one CronJob to react to a status change another one made.

And a one-off script only acts when triggered, manually or by a CI pipeline, and does nothing about drift in between runs. It's also rarely written with retries and idempotency as first-class concerns.

Why an operator wins here:

An operator is event-driven and continuous. The API server notifies it the instant a custom resource is created, updated, or deleted, and it keeps reconciling for that resource's entire lifetime, not just at apply time. That matters most for state that takes time to converge, can fail partway through, and can drift after it's first created.

This comes at a cost, though. An operator is a long-running process with its own RBAC (Role-Based Access Control), failure modes, and observability surface. This is more to build and operate than a chart or a script.

If the problem really is rendering some YAML once, a Helm chart is the right tool. An operator earns its keep when the problem is keeping something continuously correct, which is what the rest of this guide builds toward.

Part 2: Anatomy of an Operator

Next, we'll look at the pieces that make that loop actually work: the Custom Resource itself, the machinery that notices when something changed, and the manager that runs it all, before going into the reconciliation loop in detail.

Custom Resource

Before a controller can reconcile anything, the API server needs to know the shape of what it's storing. Registering a CRD teaches it that shape.

Every Custom Resource carries the same identity fields every Kubernetes object already has (kind, name, namespace, labels, and so on), plus two fields that are entirely ours to define: a spec and a status. That split isn't a style choice. It maps directly to the reconciliation loop.

• Spec is desired state. Whoever creates or edits the resource writes it, and the controller only ever reads it.

• Status is observed state. It's written only by the controller, to record what it found and what it did.

A client that writes to status directly is working around the controller instead of through it, which is why status is usually served as its own subresource with separate permissions.

The last piece is registration. The API server and any client talking to it need a shared, agreed-upon way to encode and decode our type, so we register it once against a scheme before anything can use it. Without that, our type is just a definition nobody can serve. With it, the API server can store and serve it exactly the way it serves Pods or Deployments.

We'll see exactly what that registration looks like when we build one for real in Part 3.

Watching for Change

A reconciler doesn't poll the API server in a loop asking "did anything change yet?" Three pieces work together to avoid that.

An informer opens a long-lived watch against the API server and keeps a local, in-memory cache of every object of a given type, updating it as add, update, and delete events arrive.

A lister reads from that cache instead of the API server, so a reconciler checking "does this Resource already exist?" costs a local map lookup, not a network call.

A workqueue sits between the informer and the reconciler. When the informer sees a change, it doesn't call the reconciler directly. Instead, it enqueues a key, namespace, and name, not the object itself. Workers pull keys off the queue and reconcile them, and the queue deduplicates and rate-limits on our behalf, so ten rapid updates to the same object collapse into one pending item instead of ten redundant reconciles.

This is also why a reconciler receives a key and not an object. By the time a worker picks the key off the queue, the object may have changed again, so the reconciler always looks up the current state itself rather than trusting whatever triggered it.

Manager

The manager is the process that owns all of this: the shared cache the informers populate, the client the reconciler uses to read and write objects, and the health, readiness, and metrics endpoints the rest of the cluster uses to know the controller is alive. Every reconciler we register runs inside one manager.

If we run more than one replica of the same controller for availability, we don't want both replicas reconciling the same object at once and racing each other.

The manager coordinates this through leader election. Replicas compete for a lease, exactly one holds it and actively reconciles, and the rest sit idle until the leader stops renewing it.

We'll come back to this in practice in Part 4. For now it's enough to know the manager is what makes it possible.

Reconciliation Loop

This is the part that matters most. Once we understand this loop well, most of what an operator does is a variation on it.

A reconciler's entry point is called with just a namespace and a name, nothing else. No spec, status, or diff. The reconciler has to fetch the object itself, compare its spec against what it can observe of the actual state, and decide what to do. That constraint is deliberate, and it's the reason for everything below.

Idempotency

Because the reconciler only ever gets a key, and because it can be called any number of times for the same object (in a row, out of order, or after a long gap), it has to produce the same end result no matter how many times it runs. A reconciler that blindly calls create every time it runs breaks the moment it runs twice, since the second call fails against an object that already exists.

The fix is to always check current state before acting: create only if missing, update only if different, and delete only if it shouldn't exist.

Event-driven reconciliation

A reconcile is triggered by a watch event on the resource being reconciled, and by convention also on anything it owns or otherwise depends on. On top of that, most controllers set a periodic resync so the loop also runs on a schedule even with no watch event at all, which matters once state can drift for reasons a watch would never catch.

Requeues

Sometimes a single pass through reconcile can't finish the job, becuase the work it's waiting on is still in progress elsewhere. A reconciler can ask to be called again after a delay without treating this as a failure. This is how it polls something that takes time to converge, rather than blocking inside a single call.

Error handling

Returning an error does something similar: it requeues, but with exponential backoff instead of a fixed delay. So a persistently failing reconcile doesn't hammer whatever it's failing against.

It's worth distinguishing errors that are worth retrying (like a timeout or lock conflict) from ones that aren't (like a spec that will never be valid, which should be surfaced as a status condition instead of retried forever).

Drift correction

Put all of the above together and the loop is self-healing by construction. Because reconcile recomputes the full diff every time rather than reacting to what specifically changed, it doesn't matter whether the drift came from someone running kubectl edit, another controller, or the underlying system the resource represents changing state on its own. The next reconcile, whether triggered by a watch event or a resync, sees the same gap either way and closes it the same way.

Part 3: Building the Operator

Everything so far has been building toward this. We now know what an operator is, how the terms around it relate, and what pieces a reconciliation loop is made of.

Now we'll put all of it to use and build VMOperator. It's an operator that manages a VirtualMachine Custom Resource backed by a mock cloud provider, a small HTTP service we'll also write that stands in for a real one.

apiVersion: compute.example.com/v1
kind: VirtualMachine
spec:
  image: ubuntu-22.04
  cpu: 2
  memory: 4Gi
status:
  phase: Running
  id: vm-123

Say we want to represent a virtual machine in a Kubernetes-native way, kubectl apply a YAML file, and get a VM – all without touching a cloud console or a separate CLI.

That's the motivation behind VMOperator: something that lives entirely outside Kubernetes becomes just another object the cluster's own tooling (kubectl, RBAC, GitOps pipelines) already knows how to work with.

Setup

We'll need a local cluster, and kind is the easiest way to get one:

kind create cluster --name vmoperator

Beyond that, we'll be using Go in this part for the operator, kubectl pointed at the new cluster, and Python with Flask for the mock provider below.

pip install flask

Mock Provider

Before we write controller code, we need something for it to control. The mock provider is a small HTTP service with three endpoints:

• POST /vms to create one

• GET /vms/{id} to check on it

• DELETE /vms/{id} to remove it

backed by nothing more than a dict in memory.

Every VM it creates starts in Provisioning and flips to Running a few seconds later on its own, which is enough to force our reconciler to actually poll instead of assuming success.

We're writing this one in Python rather than Go. It has nothing to do with the operator's code, as this is only for mock purposes.

import random
import string
import threading
import time

from flask import Flask, jsonify, request

app = Flask(__name__)
vms = {}  # in-memory store, keyed by VM id

def provision(vm):
    time.sleep(5)  # simulate provisioning taking time
    vm["phase"] = "Running"

@app.post("/vms")
def create_vm():
    body = request.get_json()
    vm_id = "vm-" + "".join(random.choices(string.digits, k=6))
    vm = {"id": vm_id, "image": body["image"], "phase": "Provisioning"}
    vms[vm_id] = vm

    threading.Thread(target=provision, args=(vm,), daemon=True).start()  # flips to Running in the background

    return jsonify(vm)

@app.get("/vms/<vm_id>")
def get_vm(vm_id):
    vm = vms.get(vm_id)
    if vm is None:
        return "", 404
    return jsonify(vm)

@app.delete("/vms/<vm_id>")
def delete_vm(vm_id):
    vms.pop(vm_id, None)
    return "", 204

if __name__ == "__main__":
    app.run(port=8080, threaded=True)

We'll run this as its own process, alongside the cluster, listening on the port the operator will be configured to call. Nothing about it knows Kubernetes exists, which is the point: it's standing in for a real cloud API.

Defining the VirtualMachine CRD

With something to control, we can define what we're controlling. The VirtualMachine type follows exactly the spec and status split from Part 2:

type VirtualMachineSpec struct {
	Image  string `json:"image"`
	CPU    int    `json:"cpu"`
	Memory string `json:"memory"`
}

type VirtualMachineStatus struct {
	ID    string `json:"id,omitempty"`    // provider-assigned id, empty until first provisioned
	Phase string `json:"phase,omitempty"` // mirrors the provider's lifecycle phase
}

type VirtualMachine struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   VirtualMachineSpec   `json:"spec,omitempty"`
	Status VirtualMachineStatus `json:"status,omitempty"`
}

type VirtualMachineList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []VirtualMachine `json:"items"`
}

TypeMeta carries kind and apiVersion, the same two fields on every Kubernetes object, built-in or custom, that say what this thing is.

ListMeta is its counterpart for list types, resourceVersion and continue for pagination (instead of name/namespace). This is why VirtualMachineList embeds ListMeta next to its TypeMeta while VirtualMachine itself embeds ObjectMeta.

Every type we register needs to satisfy runtime.Object, which means implementing DeepCopyObject. This is normally generated for us, but since we're doing this by hand, here's what that generated code actually looks like for VirtualMachine. The rest follow the same mechanical pattern:

func (in *VirtualMachine) DeepCopyObject() runtime.Object {
	out := VirtualMachine{
		TypeMeta:   in.TypeMeta,
		ObjectMeta: *in.ObjectMeta.DeepCopy(), // ObjectMeta already knows how to copy itself
		Spec:       in.Spec,                   // no pointers or slices in Spec, a plain copy is safe
		Status:     in.Status,
	}
	return &out
}

And the CRD manifest that teaches the API server about it:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: virtualmachines.compute.example.com
spec:
  group: compute.example.com
  scope: Namespaced
  names:
    kind: VirtualMachine
    listKind: VirtualMachineList
    plural: virtualmachines
    singular: virtualmachine
    shortNames: [vm] # lets us type `kubectl get vm` instead of the full plural
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {} # splits status into its own subresource, see Part 2
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [image, cpu, memory]
              properties:
                image: { type: string }
                cpu: { type: integer }
                memory: { type: string }
            status:
              type: object
              properties:
                phase: { type: string }
                id: { type: string }

The subresources.status line matters. It's what makes status a separate subresource with its own update path. This is exactly the boundary we talked about in Part 2 between what a client can write and what only the controller can.

The names block is also what kubectl resolves against, kubectl get virtualmachines works because plural says so. shortNames is why kubectl get vm works too, the same way kubectl get po works for Pods.

Reconciler

The reconciler's job is small on paper, look at a VirtualMachine, and make sure a matching VM exists in the provider and its status reflects reality. We wrap the provider's HTTP API behind a small client so the reconciler itself stays readable:

func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var vm computev1.VirtualMachine
	if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err) // object was deleted, nothing left to do
	}

	if vm.Status.ID == "" {
		// no VM yet, this is the first time we've seen this object
		created, err := r.Provider.Create(ctx, vm.Spec.Image)
		if err != nil {
			return ctrl.Result{}, err
		}

		vm.Status.ID = created.ID
		vm.Status.Phase = created.Phase
		if err := r.Status().Update(ctx, &vm); err != nil {
			return ctrl.Result{}, err
		}

		return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // check back shortly instead of blocking here
	}

	// VM already exists, poll the provider for whatever it knows right now
	current, err := r.Provider.Get(ctx, vm.Status.ID)
	if err != nil {
		return ctrl.Result{}, err
	}

	vm.Status.Phase = current.Phase
	if err := r.Status().Update(ctx, &vm); err != nil {
		return ctrl.Result{}, err
	}

	if current.Phase != "Running" {
		return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // still provisioning, keep polling
	}

	return ctrl.Result{}, nil
}

Two things are worth calling out. First, this is only reachable at all because we've registered a watch on VirtualMachine. The API server tells us the moment one is created or edited, which is what triggers the first call.

Second, every branch ends by writing to vm.Status, mapping whatever the provider told us onto the resource. Kubernetes never talks to the provider directly. The only way anyone finds out a VM is running is because our reconciler wrote it into status.

Failure Handling & Retries

Notice the reconciler above never retries anything itself. When r.Provider.Create or r.Provider.Get fails (like because of a network blip or the mock provider not being up yet), it just returns the error. That's deliberate. Returning an error is how we ask controller-runtime to requeue with exponential backoff on our behalf. This means we don't need to hand-roll a retry loop, and a persistently unreachable provider doesn't get flooded with retries.

The one thing worth being careful about is treating every failure the same way. A timeout talking to the provider is worth retrying. A VirtualMachine whose spec.image the provider will never accept is not. Retrying that forever just produces a busy loop that never succeeds.

We'll leave surfacing that distinction through status conditions to the exercises. The reconciler above only has one failure mode to worry about, since the mock provider never rejects a request outright.

Finalizer

If we delete a VirtualMachine right now, Kubernetes removes the object and we're left with an orphaned VM the provider still thinks is running. A finalizer closes that gap: it's a string on the object that tells Kubernetes "don't actually delete this until I say so."

const vmFinalizer = "compute.example.com/vm-cleanup"

func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var vm computev1.VirtualMachine
	if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	if !vm.DeletionTimestamp.IsZero() {
		// being deleted, deprovision through the provider before letting it go
		if controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
			if vm.Status.ID != "" {
				if err := r.Provider.Delete(ctx, vm.Status.ID); err != nil {
					return ctrl.Result{}, err
				}
			}
			controllerutil.RemoveFinalizer(&vm, vmFinalizer) // safe to let the delete proceed now
			return ctrl.Result{}, r.Update(ctx, &vm)
		}
		return ctrl.Result{}, nil
	}

	if !controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
		controllerutil.AddFinalizer(&vm, vmFinalizer) // register before we ever provision anything
		if err := r.Update(ctx, &vm); err != nil {
			return ctrl.Result{}, err
		}
	}

	// ... provisioning logic from before
	return ctrl.Result{}, nil
}

A kubectl delete on a VirtualMachine with our finalizer present doesn't remove it. Rather, it sets deletionTimestamp and waits.

Our reconciler sees that on the next call, deprovisions the VM through the provider, and only then removes the finalizer. At this point Kubernetes finally deletes the object. If there's no finalizer, there's no guarantee that cleanup ever runs.

Predicate

There's a subtle bug already sitting in the reconciler above. Every time it calls r.Status().Update, that write is itself a change to the object. This triggers our own watch, which calls reconcile again.

Left alone, this doesn't spin forever, since we're recomputing the same status until it settles. But it's still wasted work reconciling in response to writes we made ourselves.

A predicate filters those events that actually enqueue a reconcile, before our code ever runs:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). // drop status only events
		Complete(r)
}

generation only increments when spec changes. Status updates don't touch it. GenerationChangedPredicate uses that to drop events where nothing but status moved, so our own writes stop retriggering us. Then we're back to reconciling only when something meaningful changed, or when we explicitly ask to be requeued.

Owned Resources

A VirtualMachine being Running somewhere isn't very useful on its own, so let's make the operator also create a Secret holding the VM's connection details in-cluster:

func (r *VirtualMachineReconciler) reconcileConnectionSecret(ctx context.Context, vm *computev1.VirtualMachine) error {
	secret := &corev1.Secret{
		ObjectMeta: metav1.ObjectMeta{
			Name:      vm.Name + "-connection",
			Namespace: vm.Namespace,
		},
		StringData: map[string]string{"id": vm.Status.ID},
	}

	if err := controllerutil.SetControllerReference(vm, secret, r.Scheme); err != nil {
		return err // ties the Secret's lifecycle to this VirtualMachine
	}

	return r.Patch(ctx, secret, client.Apply, client.ForceOwnership, client.FieldOwner("vmoperator")) // create or update, either way
}

SetControllerReference is what makes this an owned resource, it stamps an owner reference onto the Secret pointing back at the VirtualMachine.

Two things fall out of that for free. Deleting the VirtualMachine now cascades, Kubernetes garbage collects the Secret automatically, and there's no finalizer needed since it's an in-cluster object, not an external one.

And if we add Owns(&corev1.Secret{}) alongside For(&computev1.VirtualMachine{}) in SetupWithManager, an edit or deletion of the Secret itself re-triggers reconciliation of its owning VirtualMachine. So if someone deletes it by hand, we notice and recreate it.

The same pattern, SetControllerReference call, and Owns() registration creates a second owned resource: a Service fronting the VM in-cluster. That's two different resource kinds owned by one VirtualMachine, which is all multiple owned resources means in practice. There's nothing more to it than calling the same pattern twice for different types.

Cross-Resource Reconciliation

Every VirtualMachine so far talks to one hardcoded provider endpoint. Real deployments need that to be configurable, and it's rarely a one-off: fifty VirtualMachines in the same AWS account share the same endpoint and credentials, and a hundred more might live in Azure instead.

We could put an endpoint field directly on VirtualMachineSpec, but rotating a credential or fixing a typo would then mean editing every VirtualMachine that uses it, one at a time. Pulling that into its own object lets many VirtualMachines reference it by name instead, so a single edit propagates to all of them.

Now let's add a second, small CRD:

type ProviderConfigSpec struct {
	Endpoint string `json:"endpoint"`
}

And a providerRef field on VirtualMachineSpec pointing at one by name. The interesting part isn't the new type. It's what happens when a ProviderConfig changes.

A VirtualMachine doesn't watch ProviderConfig directly, and there's no owner reference between them, so a plain Owns() won't do it. Instead, we watch the type and map each event onto every VirtualMachine that references it:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
		Owns(&corev1.Secret{}).
		Owns(&corev1.Service{}).
		Watches(
			&computev1.ProviderConfig{}, // not owned, so Owns() won't catch its changes
			handler.EnqueueRequestsFromMapFunc(r.findVirtualMachinesForProviderConfig),
		).
		Complete(r)
}

func (r *VirtualMachineReconciler) findVirtualMachinesForProviderConfig(ctx context.Context, obj client.Object) []reconcile.Request {
	var vms computev1.VirtualMachineList
	if err := r.List(ctx, &vms, client.InNamespace(obj.GetNamespace())); err != nil {
		return nil
	}

	var requests []reconcile.Request
	for _, vm := range vms.Items {
		if vm.Spec.ProviderRef == obj.GetName() { // only re-enqueue VMs that actually reference this config
			requests = append(requests, reconcile.Request{NamespacedName: client.ObjectKeyFromObject(&vm)})
		}
	}
	return requests
}

This is cross-resource reconciliation: one resource's change causing a different resource type entirely to reconcile, connected only by a field value rather than ownership.

It's also where the theme of this whole project comes back around. ProviderConfig is what would hold real credentials and a real endpoint for AWS, Azure, or GCP in a production version of this operator. The mock provider is standing in for exactly that boundary.

RBAC

None of the above works without permission to act on it. The manifest just has to list what we actually touch: VirtualMachine and ProviderConfig objects, the VirtualMachine status subresource separately, and the Secret/Service objects we create:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vmoperator-manager-role
rules:
  - apiGroups: ['compute.example.com']
    resources: ['virtualmachines', 'providerconfigs']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
  - apiGroups: ['compute.example.com']
    resources: ['virtualmachines/status'] # separate rule, it's a separate subresource
    verbs: ['get', 'update', 'patch']
  - apiGroups: ['']
    resources: ['secrets', 'services']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vmoperator-manager-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vmoperator-manager-role
subjects:
  - kind: ServiceAccount
    name: vmoperator-controller-manager
    namespace: vmoperator-system

Note: VMOperator manages a resource entirely outside the cluster through a hand-rolled HTTP client, but it's not a novel pattern. Crossplane, AWS Controllers for Kubernetes, Cluster API, and cert-manager all reconcile external or non-Kubernetes state through CRDs the same way. These resources are worth reading once this pattern feels familiar.

Another note: we're keeping VMOperator's scope narrow on purpose. Resizing a running VM, stopping and restarting one, taking snapshots, and supporting more than one real provider behind ProviderConfig are all natural extensions of what's here, and a reasonable next step once the core loop feels solid.

Part 4: Production & Deployment

Now that VMOperator works, let's see how to package and deploy it and improve it for production.

Packaging & Deployment

Everything so far has run as a binary on our own machine. go run against whatever cluster kubectl happens to be pointed at.

A Deployment needs an image instead, so the operator gets a multi-stage Dockerfile: one stage to compile it, and a second, much smaller image to actually run it:

FROM golang:1.26 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /vmoperator ./cmd/manager

FROM gcr.io/distroless/static-debian12
COPY --from=build /vmoperator /vmoperator
USER 65532:65532 # nonroot, matches the security context on the Deployment below
ENTRYPOINT ["/vmoperator"]

The build stage has the full Go toolchain and every source file, none of which need to ship. The final image only has the compiled binary, which covers most of what a container security context later in this part would otherwise have to ask for. There's no shell to get a foothold in even before runAsNonRoot is set.

docker build -t registry.example.com/vmoperator:v0.1.0 .
docker push registry.example.com/vmoperator:v0.1.0

That image is what the Deployment manifest under config/manager/ actually references. With it pushed somewhere the cluster can pull from, the rest of the manifests can go on: the CRDs, RBAC, the operator's Deployment, and a Deployment and Service for the mock provider. So it's no longer something we run as a side process on our own machine either:

kubectl apply -f config/crd/
kubectl apply -f config/rbac/
kubectl apply -f config/manager/

Installing the CRDs before anything else matters. Otherwise the operator's Deployment will crash-loop if it starts and immediately (it tries to watch a resource type the API server has never heard of).

Schema changes are the part that hand-written manifests make us feel directly. Adding a field to VirtualMachineSpec is harmless, but existing objects just don't have it set. Renaming or restructuring one isn't: every stored VirtualMachine was serialized against the old shape.

The CRD's versions list is built for exactly this, as more than one version can be served at once. One is marked storage to say which shape objects are actually persisted as, and a conversion webhook translates between them when a client asks for a version that isn't the stored one.

We don't need this for VMOperator today, since v1 is the only version that's ever existed. But it's why the versions field was a list and not a single value from the very first manifest we wrote.

None of the above replaces a person running kubectl apply by hand forever. A CI pipeline that builds the operator's image, pushes it, and applies the manifests on merge to main is the natural next step. This is ordinary CI/CD, nothing operator-specific about it once the manifests themselves are in Git.

Performance & Resilience

By default, a controller only processes one reconcile at a time. That's fine while we're the only ones testing it, but with hundreds of VirtualMachine objects it means that most of them sit in the workqueue waiting their turn even though nothing about reconciling one blocks reconciling another.

MaxConcurrentReconciles raises that:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
		Owns(&corev1.Secret{}).
		Owns(&corev1.Service{}).
		Watches(&computev1.ProviderConfig{}, handler.EnqueueRequestsFromMapFunc(r.findVirtualMachinesForProviderConfig)).
		WithOptions(controller.Options{MaxConcurrentReconciles: 5}). // five VMs in flight instead of one
		Complete(r)
}

Caching only helps one side of this reconciler. Reading vm back from r.Get is already fast and local, as informers keep that in memory. But r.Provider.Get is a real HTTP round trip every single time, and there's no cache in front of it.

That asymmetry is worth sitting with, because it's the same one from Part 2: in-cluster reads are cheap because Kubernetes built the caching layer for us, and external reads are exactly as expensive as whatever's on the other end of the wire. We could add a short-lived cache in front of the provider client, but it comes with a real cost: a cached Running for a VM that just failed is a lie our status will repeat until the cache expires.

The provider not having a cache in front of it also means nothing is stopping us from hammering it. A burst of reconciles, say every VirtualMachine getting touched at once after a cluster restart, turns into a burst of HTTP calls with no coordination between them. Wrapping the client in a rate limiter caps that independently of whatever backoff the workqueue is already doing on failures:

type Client struct {
	baseURL string
	http    *http.Client
	limiter *rate.Limiter // shared across every reconcile using this client
}

func (c *Client) Create(ctx context.Context, image string) (*VM, error) {
	if err := c.limiter.Wait(ctx); err != nil {
		return nil, err
	}
	// ... existing HTTP call
}

Leader election is the other half of running more than one replica safely. We turned this down to a concept in Part 2, but in practice it's two fields on the manager:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
	LeaderElection:   true,
	LeaderElectionID: "vmoperator-leader",
})

With this set, every replica starts up, but only the one holding the lease actually reconciles. The rest sit ready to take over the moment it doesn't renew in time.

Concurrency also surfaces a race we glossed over in Part 3. Say the reconciler calls r.Provider.Create, the provider creates the VM and returns its id, and then the process crashes before r.Status().Update ever runs. vm.Status.ID is still empty, so the next reconcile sees an object with no VM yet and calls Create again. Now the provider has two VMs for one VirtualMachine.

Nothing about MaxConcurrentReconciles or leader election prevents this. It's a gap in the create step itself, and it only shows up once something can fail between the external call and the write that records it.

Closing it for real means the provider needs to accept an idempotency key, generated once and stored on the object before the first Create call, so a retried create recognizes that it already happened instead of making a second VM.

Security

The ClusterRole from Part 3 works, but it's broader than it needs to be. It grants every verb on secrets and services cluster-wide, when the operator only ever touches the ones it owns.

A tighter version scopes to a single namespace with Role/RoleBinding instead of ClusterRole/ClusterRoleBinding wherever VMOperator is only expected to run in one, and it drops verbs we never call. We never list or watch arbitrary Secrets outside our own, only the ones we create. This is also the RBAC the manifests applied in the previous section were referring to.

Credentials are the other gap. ProviderConfig currently holds a plaintext endpoint, and a real provider needs an API key alongside it, which has no business sitting in a CRD spec anyone with read access to the object can see. It belongs in a Secret, referenced by name instead of embedded:

type ProviderConfigSpec struct {
	Endpoint  string                      `json:"endpoint"`
	SecretRef corev1.LocalObjectReference `json:"secretRef"` // Secret holding the provider's API key
}

The reconciler resolves SecretRef at the point it builds the provider client, reads the key out of the Secret's data, and never logs it or writes it back to anything with wider read access, including the VirtualMachine's own status.

The last piece is the operator's own pod. A container security context that runs as a non-root user sets a read-only root filesystem, drops Linux capabilities it doesn't need, and shrinks what's possible if the binary itself is ever compromised. This is standard practice for any workload, not something specific to operators.

If we'd added an admission webhook anywhere in this guide, its certificates would belong here too. We didn't need one for VMOperator, so we'll leave that as a pointer rather than something to configure.

Observability

The manager exposes a Prometheus endpoint without us writing anything for it. Workqueue depth, reconcile duration, and reconcile error counts are already there per controller.

What isn't there automatically is anything about the provider, so we add a metric the same way any Go service would:

var providerCallDuration = prometheus.NewHistogramVec(
	prometheus.HistogramOpts{
		Name: "vmoperator_provider_call_duration_seconds",
		Help: "Duration of calls to the VM provider, by operation",
	},
	[]string{"operation"},
)

func init() {
	metrics.Registry.MustRegister(providerCallDuration) // shares the manager's existing /metrics endpoint
}

Wrapping each provider call with a timer around this turns "is the provider slow" from a question we'd have to guess at into one we can graph.

Logging benefits from the same instinct. log.FromContext(ctx) inside Reconcile already carries the VirtualMachine's name and namespace on every line if we set that up once in SetupWithManager. Adding vm.Status.ID to that logger right after it's set means every subsequent log line for that reconcile also carries the provider's own identifier for the VM. That one field is what makes it possible to grep a mock provider log and an operator log for the same request and find both sides of the same failure.

Next Steps

In this guide, you learned what a Kubernetes operator is, how to build one from scratch, and how to prepare it for production. You also learned about finalizers, predicates, owned resources, and cross-resource reconciliation along the way.

None of this is specific to managing VMs. The next operator, whatever it manages, is the same shape.

You can also review the resources below to keep learning:

• K8s custom resources

• client-go

• controller-runtime

• Docker docs

The Container Network Interface (CNI) is one of the great black boxes of Kubernetes. Most people who run clusters every day have never once looked inside it. They know the name of their CNI ("we run Calico," "we're on Cilium") the way you know the brand of the alternator in your car: as a label, not as a thing you actually understand.

It lives at the very bottom of the stack, beneath the kubelet, beneath your pods, quietly moving every single packet. And precisely because it never fails loudly on a good day, almost nobody learns what it does.

We won't run helm install cilium. We won't apply a single manifest. Instead, we'll wire up pod networking from scratch, feel exactly where it breaks the moment traffic tries to leave a physical machine, and fix it manually.

By the end, you'll understand it in your bones, not just in theory, why tools like Cilium exist and what they're really solving under the hood.

Who this is for:

• Developers, platform engineers, and SREs who use Kubernetes every day but quietly treat pod-to-pod networking as magic.

• Anyone who has ever watched a pod flip to Running and assumed the network "just works" and wants to know what's actually happening.

What you'll build with your own hands:

• Two isolated network namespaces wired together with a virtual cable (veth pair).

• A three-namespace virtual switch using a Linux bridge, the same trick legacy CNIs use on a single node.

• A deliberately broken two-node setup where a packet gets dropped on the floor, plus the manual fix that makes it work.

• A clear picture of the three jobs every CNI does, and why cloud providers force advanced CNIs like Cilium to use overlays and eBPF.

Note: Every command here needs a Linux host and root. Run this in a throwaway VM or lab environment, not on anything you care about. The whole point is to make a mess and learn from it.

Table of Contents

• Prerequisites

• The Illusion: Kubernetes Routes Zero Packets

• The Foundation: Virtual Ethernet (veth) Pairs

• How to Scale Locally with a Linux Bridge

• The Multi-Node Boundary Problem

• How to Fix It Manually with Direct Routing

• So What Is a CNI, Really?

• The Cloud Catch and Why Cilium Changes the Game

• Conclusion

Prerequisites

Before following along, you'll need:

• Two Linux VMs on the same network for the multi-node section so you can watch traffic cross a real machine boundary.

• The ip command from the iproute2 package (already installed on virtually every modern distro).

• A basic comfort with IP addresses, subnets, and the word "gateway." You don't need to be a network engineer.

• No Kubernetes. That's not a typo. We're going underneath Kubernetes on purpose.

The Illusion: Kubernetes Routes Zero Packets

Here's the uncomfortable truth most people never confront: Kubernetes can't route a single network packet.

Not one. Kubernetes is a orchestrator. It schedules pods, watches their health, and updates state in etcd. But when it comes to actually moving a packet from one container to another, it has zero built-in capability. None.

So how do your pods talk to each other? They rely completely on an external agent to wire up the virtual network plumbing on every node. That agent is the Container Network Interface (CNI). What the CNI does under the hood quietly, is what we would do ourselves to feel the pain and then the solution a CNI provides.

Here's the proof that it's load-bearing. Spin up a brand-new cluster with kubeadm and look at your nodes:

$ kubectl get nodes
NAME       STATUS     ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION                        CONTAINER-RUNTIME
no-cni     NotReady   control-plane,master   52s   v1.35.0   192.168.117.2   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6
worker-1   NotReady   <none>                 46s   v1.35.0   192.168.117.3   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6
worker-2   NotReady   <none>                 40s   v1.35.0   192.168.117.4   <none>        Ubuntu 24.04.3 LTS   7.0.11-orbstack-00360-gc9bc4d96ac70   containerd://2.1.6

NotReady. Every node. The control plane is healthy, etcd is up, the scheduler is alive, and the cluster still flatly refuses to be Ready. If you describe the node, it tells you precisely what's missing:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  Ready            False   Sat, 18 Jul 2026 10:25:51 +0200   Sat, 18 Jul 2026 10:25:20 +0200   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Read that again: your cluster is not Ready until you install a CNI. Not "mostly ready." Not "ready except for networking." NotReady, full stop. Until an external plugin shows up and takes responsibility for the packets Kubernetes itself refuses to touch.

A cluster without a CNI is a telephone exchange with no lines plugged in: every operator is present and ready, but not a single call is able to connect.

So what do most people do at this exact moment? They copy one line from a getting-started page:

kubectl apply -f https://.../calico.yaml

They watch the nodes flip to Ready, and they move on. That's the entire relationship most engineers have with the single component that makes their cluster work. They wing it. It works, so they never ask what "it" is.

This matters because "the network just works" is a dangerous story to tell yourself. The moment something breaks (a pod can't reach a service, cross-node traffic vanishes, a cloud migration mysteriously blackholes packets), you're standing in front of a system you never actually understood. So let's understand it. From the bottom up.

The Foundation: Virtual Ethernet (veth) Pairs

To understand container networking, you first have to understand how Linux isolates it.

When a container (or a Kubernetes pod) is created, the kernel wraps it in an isolated Network Namespace (netns). Think of a fresh network namespace as an island with no bridges to the mainland. By default it's completely blind to the outside world: no interfaces, no IP addresses, and no routing tables. It can't talk to anything, and nothing can talk to it.

So how do we get off the island? With a kernel primitive called a Virtual Ethernet (veth) pair.

A veth pair is a virtual network cable. Whatever packet enters one end immediately pops out the other end, even if the two ends live in different namespaces. Plug one end into the island and the other end into the mainland, and suddenly you have a connection.

Let's wire two isolated namespaces, red and blue, directly together.

# Step 1: Create the isolated network namespaces
sudo ip netns add red
sudo ip netns add blue

# Step 2: Create the virtual ethernet cable (veth pair)
sudo ip link add veth-red type veth peer name veth-blue

# Step 3: Move each end of the cable into its namespace
sudo ip link set veth-red netns red
sudo ip link set veth-blue netns blue

# Step 4: Assign IP addresses and bring the interfaces UP
sudo ip netns exec red ip addr add 10.0.0.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

sudo ip netns exec blue ip addr add 10.0.0.2/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

Now test the connection by pinging blue from inside red:

sudo ip netns exec red ping -c 2 10.0.0.2

In the end, it would look something like this:

• Isolation broken safely: The container transitions from an unreachable, isolated namespace (no IP, no routing) to an addressable endpoint (10.1.1.2) linked directly to the host network.

• Bi-directional traffic flow: Packets originating inside the container can reach external public IP networks, and incoming response packets from the internet can traverse back through the host's eth0 interface (192.168.1.10) directly into the container's veth pairs.

• Zero-latency in-kernel bridging: The veth pair (veth-island <--> veth-mainland) acts as a direct virtual pipe, allowing instant packet transit between distinct Linux network namespaces (netns) without requiring external physical hardware .

The verdict: the ping succeeds. You just manually wired two isolated environments together with nothing but a virtual cable.

But here's the problem with this approach: it scales horribly. It does not scale well because a veth pair is strictly point to point.

Following is the number of pairs need to be configured for the number of containers:

• 2 containers: 1 pair

• 3 containers: 3 pairs

• 4 containers: 6 cables

For ten, you'd need 45 cables to connect every pair.

• Explodes in cable count: The number of veth pairs grow quadratically as containers increase

• Complex to manage: Too many interfaces, routes and rules to configure and maintain.

This is the same reason data centers don't run a physical cable between every pair of servers. You need a switch.

How to Scale Locally with a Linux Bridge

When you need to connect more than two interfaces on a single host, you stop running cables between everything and plug everything into a central hub instead. In the Linux kernel, that hub is a Linux Bridge. It's a software Layer 2 virtual switch (you'll often see it named br0 or cni0).

A bridge does exactly what a physical switch does: it learns MAC addresses and forwards frames across every connected interface in the same broadcast domain.

The pattern changes slightly. Instead of connecting namespaces directly to each other, you attach one end of a veth pair to the namespace, and plug the other end into the host's bridge.

Let's tear down the old setup and build a three-namespace switch: red, blue, and green. They'll all share one broadcast domain.

# Clean up any previous configuration
sudo ip netns del red 2>/dev/null || true
sudo ip netns del blue 2>/dev/null || true
sudo ip netns del green 2>/dev/null || true
sudo ip link del br0 2>/dev/null || true

# Step 1: Create the host switch (bridge) and bring it up
sudo ip link add br0 type bridge
sudo ip link set br0 up

# Step 2: Wire namespace 1 (red) into the bridge
sudo ip netns add red
sudo ip link add veth-red type veth peer name veth-red-host
sudo ip link set veth-red netns red
sudo ip link set veth-red-host master br0
sudo ip link set veth-red-host up
sudo ip netns exec red ip addr add 10.0.0.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

# Step 3: Wire namespace 2 (blue) into the bridge
sudo ip netns add blue
sudo ip link add veth-blue type veth peer name veth-blue-host
sudo ip link set veth-blue netns blue
sudo ip link set veth-blue-host master br0
sudo ip link set veth-blue-host up
sudo ip netns exec blue ip addr add 10.0.0.2/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

# Step 4: Wire namespace 3 (green) into the bridge
sudo ip netns add green
sudo ip link add veth-green type veth peer name veth-green-host
sudo ip link set veth-green netns green
sudo ip link set veth-green-host master br0
sudo ip link set veth-green-host up
sudo ip netns exec green ip addr add 10.0.0.3/24 dev veth-green
sudo ip netns exec green ip link set veth-green up

Because all three namespaces are connected to the shared br0 device, they can communicate freely with each other across the virtual network switch. But how do they actually "find" each other on the network? This is where ARP comes in.

ARP stands for Address Resolution Protocol. It's a fundamental part of local networking. When one computer (or namespace, in our case) wants to talk to another using an IP address, it needs to discover the other computer's hardware address (called a MAC address) to actually send packets on the network.

ARP is the system that allows this to happen — it sends out a broadcast asking "Who has IP address X? Please tell me your MAC address," and the right system answers back.

Thanks to ARP, all the namespaces plugged into br0 can learn each other's MAC addresses automatically and send packets directly within their shared network segment. Let's prove it by pinging every pair, both ways:

# red reaches blue and green
sudo ip netns exec red ping -c 1 10.0.0.2
sudo ip netns exec red ping -c 1 10.0.0.3

# blue reaches red and green
sudo ip netns exec blue ping -c 1 10.0.0.1
sudo ip netns exec blue ping -c 1 10.0.0.3

# green reaches red and blue
sudo ip netns exec green ping -c 1 10.0.0.1
sudo ip netns exec green ping -c 1 10.0.0.2

All six pings succeed. And notice there isn't a single routing rule involved anywhere. Every namespace lives in the same 10.0.0.0/24 subnet on the same Layer 2 switch, so the kernel resolves the whole mesh with plain ARP.

This is exactly how legacy single-node CNIs (the old kubenet) operate. On one machine, it's clean and simple.

But Kubernetes is a distributed system designed to scale across thousands of physical machines. So here's the question that breaks everything: what happens when our namespaces need to leave the host?

The Multi-Node Boundary Problem

Everything so far has lived on one machine. Kubernetes doesn't. So let's do the honest thing: stand up two real VMs and watch the single-node trick fall apart. Don't take my word for it: build this and watch the packet die.

Here's the setup:

• VM 1 (host IP 10.1.44.216): home to the red namespace, pod subnet 10.0.1.0/24.

• VM 2 (host IP 10.1.44.178): home to the blue namespace, pod subnet 10.0.2.0/24.

Two things to notice before we start. First, each node gets its own pod subnet (10.0.1.0/24 on VM 1, 10.0.2.0/24 on VM 2) because if both nodes handed out 10.0.0.x addresses, you'd get IP collisions the instant two pods landed on the same number.

Second, because the subnets now differ, each namespace needs a gateway to route through, and that gateway is its own host's bridge.

Note: this is the cleanup-multinode.sh script that should only be used in case you make any errors while setting up the cross node routes and veth pairs.

#!/usr/bin/env bash
#
# cleanup-multinode.sh
# Tears down the manual multi-node CNI lab (bridge + namespaces + veth
# pairs + cross-node static routes) from "Build a Mental Model for
# Kubernetes CNI by Doing It Manually."
#
# Safe to run on BOTH VMs. Every step is idempotent: anything that was
# never created on this host is skipped instead of erroring out, so
# re-running it is harmless.
#
# Usage:  sudo ./cleanup-multinode.sh
#
set -u

if [[ $EUID -ne 0 ]]; then
  echo "This script needs root. Run:  sudo $0" >&2
  exit 1
fi

echo "==> Deleting network namespaces (this also destroys their veth pairs)..."
ip netns del red  2>/dev/null && echo "    - removed netns 'red'"  || true
ip netns del blue 2>/dev/null && echo "    - removed netns 'blue'" || true

echo "==> Removing any orphaned host-side veth interfaces..."
ip link del veth-red-host  2>/dev/null && echo "    - removed veth-red-host"  || true
ip link del veth-blue-host 2>/dev/null && echo "    - removed veth-blue-host" || true

echo "==> Deleting the bridge..."
ip link del br0 2>/dev/null && echo "    - removed bridge 'br0'" || true

echo "==> Removing cross-node static routes..."
ip route del 10.0.1.0/24 2>/dev/null && echo "    - removed route to 10.0.1.0/24" || true
ip route del 10.0.2.0/24 2>/dev/null && echo "    - removed route to 10.0.2.0/24" || true

echo "==> Disabling IP forwarding (non-persistent; resets on reboot anyway)..."
sysctl -w net.ipv4.ip_forward=0 >/dev/null

# --- Optional: undo the 'Common Gotchas' tweaks, ONLY if you applied them ---
# On a throwaway lab VM, leaving FORWARD at ACCEPT or rp_filter at 0 is
# usually harmless, so these are opt-in. Uncomment whatever you changed.
# sysctl -w net.ipv4.conf.all.rp_filter=1 >/dev/null
# iptables -P FORWARD DROP

echo
echo "==> Teardown complete. Verifying nothing is left behind:"
echo "--- namespaces (expect: no red/blue) ---"
out=$(ip netns list);                          echo "${out:-  (none)}"
echo "--- bridges (expect: no br0) ---"
out=$(ip -br link show type bridge 2>/dev/null); echo "${out:-  (none)}"
echo "--- lab routes (expect: none) ---"
out=$(ip route | grep -E '10\.0\.[12]\.0/24'); echo "${out:-  (none)}"

On VM 1 (10.1.44.216), build the bridge, wire up red, and turn the host into a router:

# Make the host a router so it can transit packets that aren't its own
sudo sysctl -w net.ipv4.ip_forward=1

# Build the bridge and give it a gateway IP for VM 1's pod subnet
sudo ip link add br0 type bridge
sudo ip addr add 10.0.1.254/24 dev br0
sudo ip link set br0 up

# Wire the red namespace into the bridge
sudo ip netns add red
sudo ip link add veth-red type veth peer name veth-red-host
sudo ip link set veth-red netns red
sudo ip link set veth-red-host master br0
sudo ip link set veth-red-host up
sudo ip netns exec red ip addr add 10.0.1.1/24 dev veth-red
sudo ip netns exec red ip link set veth-red up

# Point the namespace's default route at its bridge gateway
sudo ip netns exec red ip route add default via 10.0.1.254

On VM 2 (10.1.44.178), do the mirror image for blue:

sudo sysctl -w net.ipv4.ip_forward=1

sudo ip link add br0 type bridge
sudo ip addr add 10.0.2.254/24 dev br0
sudo ip link set br0 up

sudo ip netns add blue
sudo ip link add veth-blue type veth peer name veth-blue-host
sudo ip link set veth-blue netns blue
sudo ip link set veth-blue-host master br0
sudo ip link set veth-blue-host up
sudo ip netns exec blue ip addr add 10.0.2.1/24 dev veth-blue
sudo ip netns exec blue ip link set veth-blue up

sudo ip netns exec blue ip route add default via 10.0.2.254

Both hosts are routers now. Both namespaces are wired up. Ping blue on VM 2 from red on VM 1:

# On VM 1
sudo ip netns exec red ping -c 3 10.0.2.1

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.

--- 10.0.2.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2043ms

100% packet loss. The packet is dropped on the floor, exactly as promised, but now you've seen it with your own eyes.

Here's the part worth proving to yourself: the packet really does leave VM 1. It just never arrives at VM 2. Run tcpdump on both boxes and ping again:

# Detect your physical NIC once (enp1s0, ens3, eth0, ...)
NIC=$(ip route get 1.1.1.1 | grep -oP 'dev \K\S+')

# On VM 1: the echo requests march out the door
sudo tcpdump -ni "$NIC" icmp
IP 10.0.1.1 > 10.0.2.1: ICMP echo request, id 5, seq 1, length 64
IP 10.0.1.1 > 10.0.2.1: ICMP echo request, id 5, seq 2, length 64

# On VM 2: dead silence. Nothing ever shows up.
sudo tcpdump -ni "$NIC" icmp
(no output)

So where does it die? Follow the life and death of that packet:

To keep it even simpler, the flow is as follows:

Red Pod (10.0.1.1)
      │
      ▼
eth0
      │
      ▼
veth-red
      │
      ▼
br0 (10.0.1.254)
      │
      ▼
VM 1 Routing Table
(No route for 10.0.2.0/24)
      │
      ▼
Default Route (0.0.0.0/0)
      │
      ▼
Physical NIC (eth0)
      │
      ▼
LAN Gateway (192.168.1.1)
      │
      ▼
❌ No route for 10.0.2.0/24
(Packet Dropped)
      │
      ▼
VM 2 Never Receives the Packet

That's the multi-node boundary problem in one sentence: your per-node scripts are completely blind to the rest of the cluster's topology. VM 1 built its island, VM 2 built its island, and neither has any idea the other exists.

How to Fix It Manually with Direct Routing

The fix is almost insultingly small. VM 1 doesn't need a smarter network. It needs a map. We just have to tell each host one fact it's missing: "the other node's pod subnet lives behind the other node's physical IP." That's a single static route per side. Nothing gets rebuilt: the bridges, namespaces, and forwarding you set up a moment ago all stay exactly as they are.

On VM 1 (10.1.44.216), teach it where VM 2's pods live:

# VM 2's pods (10.0.2.0/24) are reachable via VM 2's physical IP
sudo ip route add 10.0.2.0/24 via 10.1.44.178

On VM 2 (10.1.44.178), teach it the way back:

# VM 1's pods (10.0.1.0/24) are reachable via VM 1's physical IP
sudo ip route add 10.0.1.0/24 via 10.1.44.216

That's it. Two lines. Re-run the exact same ping from red on VM 1:

sudo ip netns exec red ping -c 3 10.0.2.1

PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.
64 bytes from 10.0.2.1: icmp_seq=1 ttl=62 time=0.412 ms
64 bytes from 10.0.2.1: icmp_seq=2 ttl=62 time=0.388 ms
64 bytes from 10.0.2.1: icmp_seq=3 ttl=62 time=0.401 ms

--- 10.0.2.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

It works. (See ttl=62? Your packet started at 64 and lost one hop on each host it was forwarded through, proof it crossed two routers to get there.) The packet now completes the full journey:

The simpler flow looks like:

1. Pod (10.0.1.1)
        │
        ▼
2. veth-red
        │
        ▼
3. VM 1 br0 (10.0.1.254)
        │
        ▼
4. VM 1 Routing Table
   ✅ Static route:
   10.0.2.0/24 → 10.1.44.178
        │
        ▼
5. VM 1 Physical NIC
        │
        ▼
6. Direct Link
   VM 1 → VM 2
        │
        ▼
7. VM 2 Physical NIC
        │
        ▼
8. VM 2 Routing Table
   ✅ 10.0.2.0/24 is directly connected
        │
        ▼
9. VM 2 br0
        │
        ▼
10. Blue Pod (10.0.2.1)
        │
        ▼
✅ Reply follows the same path back

That one line changed everything. Instead of dumping the packet at your LAN's clueless gateway, VM 1 now hands it directly to VM 2, which knows exactly which local namespace owns 10.0.2.1. The reply follows the mirror route home. You just hand-built cross-node pod networking.

Now sit with how painful that was. Two nodes took a stack of careful commands and a hand-written route on each side.

Imagine a thousand nodes, pods being created and destroyed every second, each one needing a fresh IP and a route on every other node in the cluster. Doing that by hand isn't just tedious. It's impossible.

So What Is a CNI, Really?

Everything you just did by hand (the namespaces, the veth pairs, the bridges, the IP assignment, the routes) is exactly what a Container Network Interface automates dynamically, at scale, the instant a pod is scheduled.

When you apply a pod manifest, the CNI plugin intercepts the lifecycle event and performs three core jobs:

1. Namespace and interface provisioning: It creates the network namespace, generates the veth pair, and attaches it to the bridge (or its own datapath), cleanly, every time, with no fat-fingered typos.

2. IP Address Management (IPAM): It hands out unique, non-colliding subnets per node and leases an individual IP to every single container in the cluster. That "unique Pod CIDR per node" rule you set up manually? IPAM enforces it automatically.

3. Cluster-wide route distribution: It programs the routing so every node knows how to reach pods on every other node: the static routes you wrote by hand, generated and pushed everywhere, kept in sync as nodes and pods come and go.

That's the mental model. A CNI is the thing that does your dozen-command lab a thousand times a second and never makes a mistake.

The Cloud Catch and Why Cilium Changes the Game

Here's the part that surprises people. The manual direct-routing approach we just built works flawlessly in a bare-metal lab. In a modern public cloud (AWS, GCP, Azure), it breaks completely.

Why? Cloud providers don't let arbitrary IP addresses roam across their network fabric. Your 10.0.1.0/24 pod subnet means nothing to the VPC. Unless those IPs are explicitly registered through heavyweight cloud-controller API calls, the underlying network sees pod traffic as illegitimate and drops it: the exact failure from the boundary-problem section, except now the cloud itself is the thing saying "no."

This is where advanced CNIs like Cilium stop playing by the old rules. Instead of leaning on fragile Linux bridges and hand-written host routes, Cilium reaches for two much stronger mechanisms.

• Overlay networks (VXLAN / Geneve). Cilium takes your raw pod packet and encapsulates it, wrapping it inside an ordinary UDP packet addressed from Node 1's physical IP to Node 2's physical IP. To the cloud provider, it looks like completely normal node-to-node host traffic, so it sails straight through every VPC restriction. Your pod's real addresses are hidden inside the envelope.

• eBPF kernel programmability. Traditional CNIs push every packet through the full Linux bridge path and hundreds of sequential iptables rules: slow, and slower as your cluster grows. Cilium replaces that entire pipeline by loading compiled eBPF programs directly into the kernel at the network-interface level. Packets get short-circuited from the pod's veth straight toward the physical NIC, giving you near line-rate performance and deep security visibility for free.

Here's the whole progression in one table:

Look at that last column, then look at the last row. That's the entire value proposition of a CNI in two cells.

Conclusion

You didn't read about Kubernetes networking. You built it, broke it, and fixed it. Here's the mental model you now carry:

1. Kubernetes routes zero packets. It fully delegates the network to a CNI, and that CNI is doing real, physical plumbing on every node.

2. A veth pair is a virtual cable, and it's the atom of container networking: great for two endpoints, useless at scale.

3. A Linux bridge is a virtual switch that connects many namespaces on one host with nothing but Layer 2 and ARP. That's a single-node CNI in a nutshell.

4. The node boundary is where naïve networking dies. Different subnets and an unaware physical network mean cross-node packets get dropped until you teach every host how to route.

5. Static routes plus IP forwarding fix it manually, and doing that by hand for two nodes shows you instantly why nobody does it for a thousand.

6. A CNI automates three jobs: interface provisioning, IPAM, and cluster-wide route distribution.

7. The cloud breaks direct routing, which is precisely why Cilium leans on VXLAN/Geneve overlays and eBPF instead of bridges and iptables.

The next time a pod flips to Running and the network "just works," you'll know the truth: nothing just works. A CNI just did (silently, and at a scale you now truly respect) everything you just did by hand.

From here, the natural next step is to tear down these scripts, deploy Cilium into a real cluster, and watch eBPF orchestrate this entire topology automatically. Having felt the manual pain first, you'll actually appreciate the elegance.

If this helped you build a clearer picture of Kubernetes networking, come say hi:

• LinkedIn: linkedin.com/in/shubhamkatara

• YouTube: @kubesimplify

A developer needs a new staging environment, so they file a ticket. The platform team queues it.

Two weeks later, the environment exists. It's configured slightly differently from the last one, with a naming convention that doesn't match the production setup, missing the observability stack the previous environment had. The developer deploys. Something breaks. Nobody knows why.

The problem isn't the ticket queue. The problem is the absence of a platform: a paved road where developers can self-serve infrastructure, deployments, and environments that are consistent, auditable, and safe without requiring a platform engineer for every request.

An Internal Developer Platform (IDP) solves this. Not by removing platform engineers from the picture, but by shifting their work from executing individual requests to building the systems that execute those requests automatically.

This handbook builds a production-grade IDP from the three CNCF tools that form its core in 2026: Backstage as the developer portal and software catalog, ArgoCD as the GitOps continuous delivery engine, and Crossplane as the Kubernetes-native infrastructure control plane.

By the end, developers on your platform will be able to provision a cloud database, deploy an application to staging, and register a new service in the catalog — all without filing a single ticket.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: IDP Architecture — The Three-Layer Model

• Part 2: ArgoCD — The GitOps Foundation

• Part 3: Crossplane — Infrastructure as Kubernetes Resources

• Part 4: Backstage — The Developer Portal

• Part 5: Wiring It Together — The Golden Path

• Part 6: FinOps Integration — Cost Attribution on the IDP

• Part 7: The Platform Maturity Model — Measuring What You've Built

• Best Practices Summary

• Resources

What You'll Learn

• The three-layer IDP architecture and why each layer must be implemented in a specific order

• How to install and configure ArgoCD with ApplicationSets for multi-environment GitOps delivery

• How to define cloud infrastructure as Kubernetes custom resources using Crossplane Compositions

• How to deploy and configure Backstage with a software catalog and Software Templates

• How to wire Backstage, ArgoCD, and Crossplane together into a single self-service golden path

• How to implement cost attribution on your IDP so every resource provisioned through it carries team and cost center metadata

• How to measure your IDP's maturity using the CNCF Platform Engineering Maturity Model

Let's build it.

Prerequisites

Before following along, you should have:

Knowledge:

• Working familiarity with Kubernetes: you can deploy applications, write YAML manifests, and understand namespaces and RBAC

• Basic GitOps understanding: you know what "Git as source of truth" means in practice

• Comfort with Helm, Terraform HCL, and TypeScript at a reading level

• Understanding of AWS services: EKS, RDS, S3, IAM

Tools and access:

• An EKS cluster running Kubernetes 1.28 or later with at least 3 nodes (m5.xlarge or equivalent)

• kubectl configured and pointing at your cluster

• helm 3.12 or later installed

• AWS CLI v2 configured with admin-level permissions for the provisioning steps

• Node.js 18 or later and Yarn (for Backstage)

• A GitHub organisation you control (for the GitOps repositories and Backstage GitHub integration)

Companion repository:

git clone https://github.com/aayostem/platform-toolkit
cd platform-toolkit

The repository contains all manifests, Helm values files, Crossplane Compositions, and Backstage templates referenced in this guide. Each part maps to a directory in the repo.

Estimated time: The full implementation takes one to two days for an experienced platform engineer. Parts 1–3 can be completed in the morning and produce a working GitOps delivery layer.

Part 1: IDP Architecture — The Three-Layer Model

1.1 What an IDP Actually Is

An Internal Developer Platform isn't a tool. It's a product: a collection of tools, workflows, and abstractions that platform teams build and maintain so that application developers can move fast without managing infrastructure directly.

The distinction matters because it shapes every architectural decision. A tool is installed and configured. A product is designed for users, iterated based on feedback, and measured by whether those users actually adopt it. The platform teams that build the IDPs that developers love think like product managers, not system administrators.

The DORA 2025 report found that nearly 90% of enterprises now have some form of internal platform. But having a platform and having a platform that developers actually use are different things.

The survey found that developer satisfaction with internal platforms varied dramatically. And the gap between satisfied and unsatisfied teams correlated directly with whether the platform team treated the IDP as a product with a roadmap and user research, or as an infrastructure project with a ticket queue.

The three tools in this guide — Backstage, ArgoCD, and Crossplane — are the most widely adopted open-source stack for production IDPs in 2026. But the architecture that connects them matters as much as the tools themselves.

1.2 The Three-Layer Architecture

A production IDP has three distinct layers, each with a single responsibility:

Layer 1: Developer Interface (Backstage)
├── Software catalog — inventory of all services, APIs, and resources
├── Software Templates — self-service forms that trigger provisioning workflows
├── TechDocs — documentation co-located with each catalog entity
└── Plugins — integrations with ArgoCD, Kubernetes, PagerDuty, Grafana

Layer 2: Delivery Layer (ArgoCD)
├── GitOps sync — continuous reconciliation of cluster state to Git
├── ApplicationSets — multi-environment deployment from a single definition
├── Rollout management — progressive delivery with health checks
└── Audit trail — every deployment change linked to a Git commit

Layer 3: Infrastructure Layer (Crossplane)
├── Composite Resources — cloud resources defined as Kubernetes CRDs
├── Compositions — templates that expand a simple claim into full AWS infrastructure
├── ProviderConfigs — credentials and region configuration for each cloud provider
└── Usage tracking — every provisioned resource tagged with team and cost centre

The critical architectural rule: Backstage never talks directly to Kubernetes or cloud APIs. When a developer submits a Software Template in Backstage, the output is a Git commit — a YAML file representing a Crossplane claim or an ArgoCD Application manifest. ArgoCD picks up that commit and applies it to the cluster. Crossplane translates the cluster resource into actual cloud infrastructure.

This indirect path isn't complexity for complexity's sake. It means every infrastructure change is a Git commit, with an author, a timestamp, a pull request, and a review. The audit trail is automatic. The rollback mechanism is git revert.

Developer → Backstage Template → Git commit → ArgoCD → Crossplane → AWS
                                     ↑
                          Single source of truth
                          Full audit trail
                          Rollback = git revert

Here's what the incorrect alternative looks like — Backstage calling cloud APIs directly:

// Bad: Backstage template calling AWS SDK directly
// No audit trail, no rollback, no reconciliation loop
// If the call fails halfway, you have partial infrastructure with no record
import { S3Client, CreateBucketCommand } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "us-east-1" });
await client.send(new CreateBucketCommand({ Bucket: bucketName }));

And the correct approach — Backstage outputting a Crossplane claim to Git:

# Good: Backstage template output — a Crossplane claim committed to Git
# ArgoCD applies it, Crossplane reconciles it, AWS creates the bucket
# Every step is tracked, auditable, and reversible
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: S3Bucket
metadata:
  name: ${{ values.bucket_name }}
  namespace: ${{ values.team_namespace }}
  labels:
    team: ${{ values.team_name }}
    cost-centre: ${{ values.cost_centre }}
    environment: ${{ values.environment }}
spec:
  versioning: true
  encryption: AES256
  region: us-east-1

1.3 Implementation Order

Build in this order. Deviating from it creates integration problems that are difficult to debug:

Step 1: ArgoCD — the delivery foundation everything else depends on
Step 2: Crossplane — infrastructure control plane, delivered by ArgoCD
Step 3: Backstage — the portal, pointing at ArgoCD and Crossplane as backends
Step 4: Wire together — Software Templates that produce GitOps manifests
Step 5: FinOps layer — cost attribution metadata in every provisioned resource

Part 2: ArgoCD — The GitOps Foundation

ArgoCD is a declarative continuous delivery tool for Kubernetes that implements the GitOps pattern. If you haven't used a GitOps tool before, the core idea is simple: your Git repository is the single source of truth for what should be running in your cluster, and ArgoCD continuously reconciles actual cluster state to match it.

If a developer manually changes a resource in the cluster, ArgoCD detects the drift and resyncs from Git. If Git changes, ArgoCD applies the change to the cluster. Human intervention isn't required, and is actively discouraged — the goal is a cluster whose state is always fully explained by what's in Git.

ArgoCD is a CNCF Graduated project, meaning it's production-ready and widely used. It runs as a set of pods in your cluster with a web UI, a CLI, and a REST API. Everything you need to manage deployments across multiple environments lives in one place.

2.1 Installing ArgoCD

# Create the ArgoCD namespace
kubectl create namespace argocd

# Install ArgoCD using the official manifest
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all pods to be running before proceeding
kubectl wait --for=condition=Ready pods \
  --all -n argocd --timeout=300s

# Get the initial admin password
argocd_password=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

echo "ArgoCD initial password: $argocd_password"
echo "Save this somewhere secure before proceeding"

# Port-forward to access the ArgoCD UI locally
kubectl port-forward svc/argocd-server -n argocd 8080:443 &

# Login via CLI
argocd login localhost:8080 \
  --username admin \
  --password "$argocd_password" \
  --insecure

# Change the password immediately
argocd account update-password \
  --current-password "$argocd_password" \
  --new-password "your-secure-password"

2.2 Repository Structure for GitOps

The repository structure ArgoCD watches determines how you manage multiple environments. The pattern that scales best is environment-per-directory, with overlays managed by Kustomize.

Kustomize is a Kubernetes-native configuration management tool that lets you define a base configuration once and layer environment-specific overrides on top of it. This means your staging and production configurations share the same YAML structure but differ in replica counts, image tags, and resource limits.

gitops-repo/
├── apps/
│   ├── base/                    # Shared configuration across all environments
│   │   ├── payment-api/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── user-api/
│   │       ├── deployment.yaml
│   │       ├── service.yaml
│   │       └── kustomization.yaml
│   └── overlays/
│       ├── staging/             # Staging-specific overrides
│       │   ├── payment-api/
│       │   │   └── kustomization.yaml   # Override: 1 replica, staging image tag
│       │   └── kustomization.yaml
│       └── production/          # Production-specific overrides
│           ├── payment-api/
│           │   └── kustomization.yaml   # Override: 3 replicas, pinned image tag
│           └── kustomization.yaml
└── infrastructure/
    ├── crossplane/              # Crossplane installation and providers
    ├── monitoring/              # Prometheus, Grafana
    └── ingress/                 # NGINX or ALB ingress controller

2.3 ApplicationSets — Managing Multiple Environments

An ApplicationSet is an ArgoCD resource that generates multiple Application objects from a single template. Instead of creating one Application manifest per service per environment — which becomes unmanageable at scale — you define one ApplicationSet that covers all services across all environments. A matrix generator combines a list of environments with a Git directory scan to produce every combination automatically:

# applicationset-apps.yaml
# This single resource generates one ArgoCD Application
# for each combination of environment and application directory
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          # Generator 1: environments
          - list:
              elements:
                - environment: staging
                  cluster: https://staging.eks.cluster.local
                - environment: production
                  cluster: https://production.eks.cluster.local

          # Generator 2: application directories in the overlay
          - git:
              repoURL: https://github.com/your-org/gitops-repo
              revision: HEAD
              directories:
                - path: apps/overlays/{{environment}}/*

  template:
    metadata:
      name: "{{environment}}-{{path.basename}}"
      labels:
        environment: "{{environment}}"
        app: "{{path.basename}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/gitops-repo
        targetRevision: HEAD
        path: "apps/overlays/{{environment}}/{{path.basename}}"
      destination:
        server: "{{cluster}}"
        namespace: "{{path.basename}}"
      syncPolicy:
        automated:
          prune: true        # Delete resources removed from Git
          selfHeal: true     # Revert manual cluster changes
        syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=foreground

Verify the ApplicationSet is generating the expected Applications:

# List all generated Applications
kubectl get applications -n argocd

# Expected output: one Application per environment per app
# staging-payment-api    Synced    Healthy
# staging-user-api       Synced    Healthy
# production-payment-api Synced    Healthy
# production-user-api    Synced    Healthy

# Check sync status for a specific application
argocd app get staging-payment-api

2.4 ArgoCD RBAC for Platform Teams

In a multi-team IDP, different teams need different levels of access to ArgoCD. Application teams should be able to view and sync their own applications. Platform teams should have broader access. Nobody should have unrestricted cluster admin through ArgoCD.

The default policy is readonly — every authenticated user can see everything but change nothing:

# argocd-rbac-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Platform team: full access to all applications
    p, role:platform-team, applications, *, */*, allow
    p, role:platform-team, clusters, get, *, allow
    p, role:platform-team, repositories, *, *, allow

    # Application teams: sync and get their own namespace only
    p, role:app-team, applications, get, */staging-*, allow
    p, role:app-team, applications, sync, */staging-*, allow

    # Bind roles to GitHub teams
    g, your-org:platform-engineers, role:platform-team
    g, your-org:developers, role:app-team

  scopes: '[groups]'

Part 3: Crossplane — Infrastructure as Kubernetes Resources

Crossplane is a CNCF Graduated open-source framework that extends Kubernetes into a universal infrastructure control plane.

The core idea: instead of managing cloud resources with separate tools like Terraform or CloudFormation that live outside your cluster, you define cloud resources — RDS databases, S3 buckets, VPCs, IAM roles — as Kubernetes custom resource definitions.

Once you apply a Crossplane resource to the cluster, Crossplane's controllers take over and reconcile the desired state to the actual AWS state, exactly the way Kubernetes reconciles a Deployment to a set of running pods.

The key abstraction Crossplane adds on top of that is the Composite Resource. A platform team defines a high-level PostgreSQLDatabase type that abstracts over the thirty-plus configuration fields an actual RDS instance requires.

Developers interact with the simple type. Crossplane expands it into the full AWS resource configuration behind the scenes, applying the platform team's security and operational standards automatically — standards that developers can't bypass because they never see the underlying fields.

3.1 Installing Crossplane

Crossplane is delivered to your cluster by ArgoCD — the first integration between the two tools. By installing Crossplane through an ArgoCD Application rather than running helm install directly, you make Crossplane itself part of the GitOps-managed infrastructure. Any change to Crossplane's configuration goes through a Git commit and review:

# infrastructure/crossplane/application.yaml
# ArgoCD Application that installs Crossplane via Helm
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: crossplane
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.crossplane.io/stable
    chart: crossplane
    targetRevision: 1.15.0
    helm:
      values: |
        provider:
          packages:
            # AWS provider — manages all AWS resources
            - xpkg.upbound.io/upbound/provider-aws-s3:v1.2.0
            - xpkg.upbound.io/upbound/provider-aws-rds:v1.2.0
            - xpkg.upbound.io/upbound/provider-aws-iam:v1.2.0
  destination:
    server: https://kubernetes.default.svc
    namespace: crossplane-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

# Apply the ArgoCD Application — ArgoCD installs Crossplane
kubectl apply -f infrastructure/crossplane/application.yaml

# Watch Crossplane pods come up
kubectl get pods -n crossplane-system -w

# Verify providers are installed and healthy
kubectl get providers
# Expected:
# NAME                          INSTALLED   HEALTHY   PACKAGE
# upbound-provider-aws-s3       True        True      xpkg.upbound.io/...
# upbound-provider-aws-rds      True        True      xpkg.upbound.io/...

3.2 Provider Credentials

Crossplane needs AWS credentials to provision resources. The recommended approach for EKS is IAM Roles for Service Accounts (IRSA) — a mechanism that lets Kubernetes pods assume IAM roles directly without storing any credentials in the cluster.

The pod's Kubernetes service account is annotated with an IAM role ARN, and AWS automatically provides short-lived credentials when the pod makes API calls. No access keys, no secrets to rotate, and no credentials to accidentally expose:

# Create the IAM role for Crossplane with the necessary AWS permissions
aws iam create-role \
  --role-name CrossplaneProviderRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID:sub":
            "system:serviceaccount:crossplane-system:provider-aws"
        }
      }
    }]
  }'

# Attach the permissions policy (scope this to minimum required in production)
aws iam attach-role-policy \
  --role-name CrossplaneProviderRole \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

# provider-config.yaml
# Configure the AWS provider with IRSA — no static credentials
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: IRSA   # Use the IAM role attached to the provider service account

3.3 Defining a Composite Resource — PostgreSQL Database

This is where the IDP abstraction lives. The platform team defines two YAML files: the CompositeResourceDefinition (XRD), which specifies the shape of what developers can request, and the Composition, which specifies how that request expands into actual AWS resources with platform standards applied.

The XRD is the API contract with developers. Keep it simple — only fields developers genuinely need to control should appear here:

# xrd-postgresql.yaml
# Defines the PostgreSQLDatabase type that developers can request
# Developers never see the RDS-specific configuration below
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresqldatabases.platform.cloudfrugal.com
spec:
  group: platform.cloudfrugal.com
  names:
    kind: XPostgreSQLDatabase
    plural: xpostgresqldatabases
  claimNames:
    kind: PostgreSQLDatabase     # This is what developers create
    plural: postgresqldatabases
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                # Developer-facing fields only — simple and bounded
                storageGB:
                  type: integer
                  minimum: 20
                  maximum: 1000
                  description: "Storage in GB. Min 20, max 1000."
                instanceClass:
                  type: string
                  enum: ["small", "medium", "large"]
                  description: "small=db.t4g.medium, medium=db.r7g.large, large=db.r7g.2xlarge"
                environment:
                  type: string
                  enum: ["staging", "production"]

The Composition is the platform team's implementation. It maps the simple developer fields to the full RDS configuration and enforces platform standards that developers can't override:

# composition-postgresql.yaml
# Defines what a PostgreSQLDatabase claim expands into
# Platform standards (encryption, backup, deletion protection) are applied here
# Developers cannot override them — the platform enforces them
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: postgresql-aws-composition
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: platform.cloudfrugal.com/v1alpha1
    kind: XPostgreSQLDatabase

  resources:
    # The actual RDS instance — expanded from the simple developer claim
    - name: rds-instance
      base:
        apiVersion: rds.aws.upbound.io/v1beta1
        kind: Instance
        spec:
          forProvider:
            region: us-east-1
            engine: postgres
            engineVersion: "15.4"
            # Platform standards — always applied, not developer-configurable
            storageEncrypted: true           # Always encrypted
            backupRetentionPeriod: 7         # Always 7-day backup
            deletionProtection: true         # Always deletion-protected
            multiAZ: false                   # Overridden to true for production (see patches)
            dbSubnetGroupNameSelector:
              matchLabels:
                platform.cloudfrugal.com/subnet-group: private
      patches:
        # Map the developer's simple instanceClass to the actual RDS instance type
        - type: CombineFromComposite
          combine:
            variables:
              - fromFieldPath: spec.instanceClass
            strategy: string
            string:
              fmt: |
                %s
          toFieldPath: spec.forProvider.dbInstanceClass
          transforms:
            - type: map
              map:
                small:  db.t4g.medium
                medium: db.r7g.large
                large:  db.r7g.2xlarge

        # Enable Multi-AZ for production automatically
        - type: FromCompositeFieldPath
          fromFieldPath: spec.environment
          toFieldPath: spec.forProvider.multiAZ
          transforms:
            - type: map
              map:
                staging:    "false"
                production: "true"

        # Copy team labels from the claim to the RDS instance for cost attribution
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels
          toFieldPath: spec.forProvider.tags

A developer requesting a PostgreSQL database now writes this — nothing more:

# Developer creates this in their team's namespace
# No RDS knowledge required. No IAM configuration. No subnet group lookup.
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: PostgreSQLDatabase
metadata:
  name: payment-service-db
  namespace: payments-team
  labels:
    team: payments
    cost-centre: payments-engineering
    environment: staging
spec:
  storageGB: 100
  instanceClass: medium
  environment: staging

Crossplane reconciles this claim to a full RDS instance within minutes, with encryption, backup, and all platform standards applied automatically.

3.4 Verifying Crossplane Resource Provisioning

# Watch the claim status — it should transition to Ready=True
kubectl get postgresqldatabases -n payments-team -w

# Check the composite resource for detailed status
kubectl describe xpostgresqldatabases.platform.cloudfrugal.com

# Verify the actual AWS resource was created
aws rds describe-db-instances \
  --query 'DBInstances[?TagList[?Key==`team` && Value==`payments`]].[DBInstanceIdentifier,DBInstanceStatus]' \
  --output table

Part 4: Backstage — The Developer Portal

Backstage is a CNCF incubating open-source framework originally built by Spotify. It serves as the developer-facing interface of your IDP — the single place where developers discover services, request infrastructure, and find documentation, without needing to know which underlying system provides any of it.

Backstage provides three core capabilities:

1. A software catalog that inventories every service, API, library, and resource in your organisation

2. Software Templates that give developers self-service forms for provisioning infrastructure and scaffolding new services

3. TechDocs that co-locate documentation with the catalog entity it documents so that documentation is always findable from the same place as the service it covers.

Backstage is built in TypeScript with a React frontend and a Node.js backend. It's configured rather than installed: you create a Backstage app, configure it with your organisation's specifics, and deploy it to your cluster.

4.1 Creating and Configuring Backstage

# Create a new Backstage app
npx @backstage/create-app@latest

# When prompted:
# App name: platform-portal
# Choose SQLite for local development, PostgreSQL for production

cd platform-portal

Configure Backstage to connect to your ArgoCD instance and GitHub:

# app-config.production.yaml
app:
  title: Cloudfrugal Platform Portal
  baseUrl: https://platform.your-company.com

backend:
  baseUrl: https://platform.your-company.com
  database:
    client: pg
    connection:
      host: ${POSTGRES_HOST}
      port: 5432
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
      database: backstage

# GitHub integration for catalog discovery and template scaffolding
integrations:
  github:
    - host: github.com
      apps:
        - appId: ${GITHUB_APP_ID}
          webhookSecret: ${GITHUB_WEBHOOK_SECRET}
          clientId: ${GITHUB_CLIENT_ID}
          clientSecret: ${GITHUB_CLIENT_SECRET}
          privateKey: ${GITHUB_PRIVATE_KEY}

# ArgoCD plugin configuration
argocd:
  username: ${ARGOCD_USERNAME}
  password: ${ARGOCD_PASSWORD}
  appLocatorMethods:
    - type: 'config'
      instances:
        - name: main
          url: https://argocd.your-company.com

# Catalog auto-discovery — finds catalog-info.yaml files across your GitHub org
catalog:
  providers:
    github:
      your-org:
        organization: 'your-github-org'
        catalogPath: '/catalog-info.yaml'
        filters:
          branch: 'main'

4.2 The Software Catalog — Registering Services

Every service, API, library, and resource in your platform should be registered in the Backstage catalog via a catalog-info.yaml file committed to the service's repository. Backstage discovers these files automatically through the GitHub integration — no manual registration required once the file exists:

# catalog-info.yaml — committed to each service's repository root
# Backstage discovers this automatically via the GitHub integration
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-api
  title: Payment API
  description: "Core payment processing service. Handles transaction initiation, authorisation, and settlement."
  annotations:
    # Links ArgoCD to show deployment status in the Backstage UI
    argocd/app-name: production-payment-api
    # Links GitHub Actions workflow status
    github.com/project-slug: your-org/payment-api
    # Links Grafana dashboard for this service
    grafana/dashboard-selector: "title=Payment API"
    # Links PagerDuty on-call schedule
    pagerduty.com/service-id: P123456
  tags:
    - payments
    - typescript
    - critical
  links:
    - url: https://payment-api.docs.your-company.com
      title: Documentation
    - url: https://grafana.your-company.com/d/payment-api
      title: Grafana Dashboard
spec:
  type: service
  lifecycle: production
  owner: group:payments-team
  system: payment-platform
  dependsOn:
    - component:user-api
    - resource:payment-service-db
  providesApis:
    - payment-api-v2

4.3 Software Templates — Self-Service Infrastructure

A Software Template is a Backstage form that, when submitted, produces a Git commit. The commit contains whatever YAML, code, or configuration the template defines.

For infrastructure provisioning, the output is a Crossplane claim. For new service scaffolding, the output is a complete service skeleton committed to a new repository.

The key design decision: templates should create pull requests, not merge directly. The PR gives platform teams visibility, gives developers a review moment, and gives everyone an audit trail. Auto-merge policies can eliminate the review step for low-risk provisioning once you've built trust in the template's outputs:

# templates/postgresql-database/template.yaml
# This template gives developers a form to request a PostgreSQL database
# The output is a Crossplane PostgreSQLDatabase claim committed to the GitOps repo
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: postgresql-database
  title: PostgreSQL Database
  description: Provision a managed PostgreSQL database on AWS RDS. Encryption, backups, and deletion protection are configured automatically by the platform.
  tags:
    - database
    - postgresql
    - aws
spec:
  owner: group:platform-team
  type: infrastructure

  # The form developers fill out in the Backstage UI
  parameters:
    - title: Database Configuration
      required: [name, team, environment, storageGB, instanceClass]
      properties:
        name:
          title: Database Name
          type: string
          description: "Lowercase, hyphens only. E.g. payment-service-db"
          pattern: '^[a-z][a-z0-9-]*$'

        team:
          title: Owning Team
          type: string
          description: "Your team name. Used for cost attribution and ownership."
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: Group

        environment:
          title: Environment
          type: string
          enum: [staging, production]
          default: staging

        storageGB:
          title: Storage (GB)
          type: integer
          minimum: 20
          maximum: 1000
          default: 50

        instanceClass:
          title: Instance Size
          type: string
          enum: [small, medium, large]
          enumNames:
            - "Small (db.t4g.medium) — dev/staging workloads"
            - "Medium (db.r7g.large) — moderate production traffic"
            - "Large (db.r7g.2xlarge) — high-throughput production"
          default: small

  # What the template does when submitted
  steps:
    - id: generate-claim
      name: Generate Crossplane Claim
      action: fetch:template
      input:
        url: ./skeleton    # Contains the Crossplane claim YAML template
        values:
          name: ${{ parameters.name }}
          team: ${{ parameters.team | parseEntityRef | pick('name') }}
          environment: ${{ parameters.environment }}
          storageGB: ${{ parameters.storageGB }}
          instanceClass: ${{ parameters.instanceClass }}

    - id: create-pr
      name: Create Pull Request to GitOps Repo
      action: publish:github:pull-request
      input:
        repoUrl: github.com?repo=gitops-repo&owner=your-org
        title: "Platform: Provision PostgreSQL database ${{ parameters.name }} for ${{ parameters.team }}"
        branchName: "provision-db-${{ parameters.name }}-${{ '' | now }}"
        description: |
          Requesting PostgreSQL database provisioned by Crossplane.

          - **Name:** ${{ parameters.name }}
          - **Team:** ${{ parameters.team }}
          - **Environment:** ${{ parameters.environment }}
          - **Storage:** ${{ parameters.storageGB }}GB
          - **Instance:** ${{ parameters.instanceClass }}

          Approve this PR to trigger provisioning. ArgoCD will pick up the change and Crossplane will create the RDS instance within ~5 minutes of merge.
        sourcePath: ./skeleton

  output:
    links:
      - title: View Pull Request
        url: ${{ steps['create-pr'].output.remoteUrl }}
      - title: Track Provisioning in ArgoCD
        url: https://argocd.your-company.com/applications

The template skeleton directory contains the Crossplane claim with template variable placeholders:

# templates/postgresql-database/skeleton/databases/${{ values.name }}.yaml
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: PostgreSQLDatabase
metadata:
  name: ${{ values.name }}
  namespace: ${{ values.team }}-platform
  labels:
    team: ${{ values.team }}
    cost-centre: ${{ values.team }}-engineering
    environment: ${{ values.environment }}
    managed-by: backstage-scaffolder
spec:
  storageGB: ${{ values.storageGB }}
  instanceClass: ${{ values.instanceClass }}
  environment: ${{ values.environment }}

Part 5: Wiring It Together — The Golden Path

The Golden Path is the complete end-to-end workflow: a developer uses Backstage to request infrastructure, that request becomes a Git commit, ArgoCD applies the commit to the cluster, Crossplane provisions the actual AWS resource, and the result appears in both the Backstage catalog and the ArgoCD dashboard.

5.1 The Complete Flow

Developer fills form in Backstage
    ↓
Backstage Software Template renders the Crossplane claim YAML
    ↓
Backstage creates a Pull Request in the GitOps repository
    ↓
Platform engineer (or auto-merge policy) approves and merges the PR
    ↓
ArgoCD detects the new file in the GitOps repository
    ↓
ArgoCD applies the Crossplane claim to the cluster
    ↓
Crossplane reconciles the claim to an actual AWS RDS instance
    ↓
Developer receives the database endpoint via Kubernetes Secret
    ↓
Backstage catalog shows the new resource, owned by the requesting team

5.2 Surfacing Resource Status Back in Backstage

The Backstage Kubernetes plugin pulls live pod and resource status from your clusters and displays it on each catalog entity page. Developers can see whether their service is running, how many replicas are healthy, and whether the last deployment synced — without leaving Backstage or learning kubectl:

# Install the Kubernetes plugin packages
cd platform-portal
yarn --cwd packages/app add @backstage/plugin-kubernetes
yarn --cwd packages/backend add @backstage/plugin-kubernetes-backend

# app-config.production.yaml — add Kubernetes cluster configuration
kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'config'
      clusters:
        - name: production-eks
          url: ${PRODUCTION_CLUSTER_URL}
          authProvider: serviceAccount
          serviceAccountToken: ${PRODUCTION_SA_TOKEN}
          caData: ${PRODUCTION_CA_DATA}
        - name: staging-eks
          url: ${STAGING_CLUSTER_URL}
          authProvider: serviceAccount
          serviceAccountToken: ${STAGING_SA_TOKEN}
          caData: ${STAGING_CA_DATA}

Annotate each catalog entity to link it to its Kubernetes resources:

# In each service's catalog-info.yaml
annotations:
  backstage.io/kubernetes-label-selector: 'app=payment-api'
  backstage.io/kubernetes-namespace: payments-team

5.3 Installing the ArgoCD Plugin

The ArgoCD plugin shows deployment history and sync status directly in the Backstage entity page. When a developer opens the payment-api page in the catalog, they can see the last 10 deployments, the current sync state, and whether the application is healthy — all without opening the ArgoCD UI:

yarn --cwd packages/app add @roadiehq/backstage-plugin-argo-cd

// packages/app/src/components/catalog/EntityPage.tsx
import { EntityArgoCDOverviewCard } from '@roadiehq/backstage-plugin-argo-cd';

// Add to the service entity page layout
const serviceEntityPage = (
  <EntityLayout>
    <EntityLayout.Route path="/" title="Overview">
      <Grid container spacing={3}>
        <Grid item md={6}>
          <EntityAboutCard variant="gridItem" />
        </Grid>
        <Grid item md={6}>
          {/* ArgoCD deployment status — shows sync state and recent deployments */}
          <EntityArgoCDOverviewCard />
        </Grid>
      </Grid>
    </EntityLayout.Route>
  </EntityLayout>
);

Part 6: FinOps Integration — Cost Attribution on the IDP

An IDP that provisions resources without cost attribution creates a new problem: you now have automated infrastructure provisioning with no clear ownership of the bill it generates. Every resource created through the IDP must carry team and cost centre metadata from the moment it's provisioned.

6.1 Mandatory Labels on Every Crossplane Composition

The Crossplane Compositions are where cost attribution is enforced — not in the developer-facing claim, but in the platform layer that the developer can't bypass. These labels flow through to the actual AWS resource as tags, which means they appear in AWS Cost Explorer and can be used to build team-level cost reports:

# In every Composition, add mandatory cost attribution patches
patches:
  # These labels flow to the actual AWS resource as tags
  # They can't be omitted or overridden by the developer claim
  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[team]
    toFieldPath: spec.forProvider.tags[team]

  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[cost-centre]
    toFieldPath: spec.forProvider.tags[cost-centre]

  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[environment]
    toFieldPath: spec.forProvider.tags[environment]

  # Add a managed-by tag to identify all IDP-provisioned resources
  - type: FromCompositeFieldPath
    fromFieldPath: metadata.name
    toFieldPath: spec.forProvider.tags[managed-by]
    transforms:
      - type: string
        string:
          fmt: "idp-crossplane"

6.2 Cost Attribution Query

With mandatory tags on every resource, you can query actual cost by team directly from AWS Cost Explorer:

# Monthly cost breakdown by team — all IDP-provisioned resources
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \
  --granularity MONTHLY \
  --filter '{
    "Tags": {
      "Key": "managed-by",
      "Values": ["idp-crossplane"]
    }
  }' \
  --group-by Type=TAG,Key=team \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Team:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table

Every team that provisions resources through the IDP now has a line on the cost report with their name on it. This is the chargeback model that makes FinOps sustainable at platform scale — attribution is automatic, not manual.

Part 7: The Platform Maturity Model — Measuring What You've Built

The CNCF Platform Engineering Maturity Model defines five levels of platform maturity. Knowing where you sit helps you decide what to build next and communicate progress to engineering leadership.

A complete Backstage + ArgoCD + Crossplane implementation, with cost attribution and Software Templates covering your most common developer requests, puts you at Level 3. Moving to Level 4 requires adding SLO alerting on the platform's own health, running quarterly developer experience surveys, and producing a monthly cost-by-team report from the attribution tags.

The most common mistake at Level 3: building more features instead of measuring adoption. A platform that has 12 Software Templates but only 2 are regularly used hasn't reached Level 3 — it's reached Level 2 with more YAML. Measure which golden paths are used, interview developers who aren't using the portal, and fix the friction before adding capabilities.

Best Practices Summary

✅ Do: Build in order — ArgoCD first, then Crossplane, then Backstage. Each layer depends on the previous one.

✅ Do: Use Backstage as a Git commit generator, not as an infrastructure caller. All infrastructure changes must be auditable Git commits.

✅ Do: Apply cost attribution tags in the Crossplane Composition layer, not in the developer claim. Attribution that developers can bypass will be bypassed.

✅ Do: Start with two or three Software Templates and make them excellent before building more. Template adoption is your most important early metric.

✅ Do: Register every service in the Backstage catalog from day one. The catalog's value is proportional to its coverage.

✅ Do: Deliver Crossplane to your cluster via ArgoCD, not helm install. Everything the IDP manages should itself be managed by the IDP.

❌ Don't: Connect Backstage directly to cloud APIs. No audit trail, no rollback, no reconciliation.

❌ Don't: Give developers the Crossplane XRD directly. The Composition abstraction exists to hide RDS-specific configuration and enforce platform standards. Bypassing it defeats the purpose.

❌ Don't: Build the IDP in isolation and announce it as done. Platform engineering is product engineering. Schedule user interviews after the first two templates are live.

❌ Don't: Skip the ArgoCD RBAC configuration. An IDP that gives all developers cluster-admin through the delivery layer has created a security problem larger than the one it solved.

Resources

• Backstage Documentation — Official reference for plugin development, Software Templates, and catalog configuration

• Crossplane Documentation — CompositeResourceDefinition and Composition reference, provider installation guides

• ArgoCD Documentation — ApplicationSet generator reference, RBAC configuration, and sync policy options

• CNCF Platform Engineering Maturity Model — The maturity framework referenced in Part 7

• AWS Provider for Crossplane — Complete reference for all AWS resource types available through Crossplane

• Backstage Kubernetes Plugin — Setup guide for the Kubernetes resource visibility integration in Part 5

• FinOps Foundation — FinOps for Platform Engineering — Framework reference for the cost attribution model in Part 6

• Companion Repository — All manifests, Compositions, ApplicationSets, and Backstage templates from this guide

• 2025 DORA State of AI-assisted Software Development Report

Yesterday, 10.0.1.45 was your payment service. Today, after a rolling deployment, it's your logging agent. Your payment service is now at 10.0.1.89.

Kubernetes has already updated all the endpoints and service records — but your network policy has no idea. It silently allows traffic through based on an IP address that no longer belongs to the workload you intended to trust.

This is the workload identity problem. IP addresses aren't an identity, they're a location. And in a Kubernetes cluster, location changes constantly. Building security policy on top of IP addresses means your security posture silently degrades every time a pod is scheduled, rescheduled, or scaled.

The answer is cryptographic workload identity: every workload gets a certificate-backed identity that proves who it is, not where it is. Services authenticate each other using those certificates before exchanging any data. If the certificate doesn't match, the connection is refused, regardless of what IP address it came from.

This is what SPIFFE and SPIRE provide. And this is how Cilium enforces it using eBPF, without injecting a sidecar into every pod.

In this article you'll understand how the SPIFFE identity model works, deploy SPIRE to issue cryptographic identities to workloads, and use Cilium's built-in SPIRE integration to enforce mutual TLS between services without touching your application code.

Prerequisites

• Familiarity with Kubernetes RBAC and pod security — this handbook covers the foundations

• Familiarity with TLS certificates and Kubernetes Secrets — this handbook covers cert-manager and certificate concepts

• Helm 3 and the Cilium CLI installed

• A kind cluster — you'll create a fresh one with Cilium as the CNI in this article

• Patience: this is the most complex demo I've covered in this group of articles. SPIRE has more moving parts than anything else covered so far.

All demo files are in the companion GitHub repository.

Table of Contents

• Prerequisites

• The Workload Identity Problem

• How SPIFFE Works

  • SPIFFE IDs and Trust Domains

  • SVIDs: The Cryptographic Identity Document

  • The Trust Bundle

• How SPIRE Works

  • SPIRE Server and SPIRE Agent

  • Node Attestation

  • Workload Attestation

  • SVID Issuance and Rotation

• How Cilium Implements Mutual TLS with SPIFFE

• Demo 1 — Install Cilium with SPIRE Integration

  • Step 1: Install the Cilium CLI

  • Step 2: Create a kind cluster without a default CNI

  • Step 3: Install Cilium with SPIRE enabled

  • Step 4: Verify the installation

• Demo 2 — Enforce Mutual TLS with a CiliumNetworkPolicy

  • Step 1: Deploy a client and server

  • Step 2: Confirm traffic flows without authentication

  • Step 3: Apply a CiliumNetworkPolicy requiring mutual authentication

  • Step 4: Verify authenticated traffic still flows

  • Step 5: Observe the authentication with Hubble (optional)

  • Step 6: Verify that a pod without the matching label is blocked

  • Step 7: Check the workload entries in SPIRE

• Conclusion

• Cleanup (kind)

The Workload Identity Problem

The opening scenario isn't theoretical. In Kubernetes, pods are ephemeral. The scheduler can place a pod on any node, and a pod's IP address is assigned at scheduling time from the node's IP pool.

When a pod is deleted and recreated through a rolling deployment, a node drain, or an autoscaler event, it gets a new IP address. If you've written a NetworkPolicy that says, "allow traffic from this IP", that policy is now pointing at nothing, or worse, at a different workload.

Kubernetes service names help here for east-west traffic — a Service name resolves consistently regardless of which pods back it. But a NetworkPolicy based on a Service name is still a label selector match, not a cryptographic assertion. Any pod that can spoof the right labels can bypass it.

What you actually want is this: before service A sends a request to service B, service B proves its identity cryptographically. If service B can't prove it is who it claims to be, service A refuses the connection. This is mutual TLS, and the key question is: where do the identities come from?

SPIFFE answers that question.

How SPIFFE Works

SPIFFE — Secure Production Identity Framework for Everyone — is a CNCF standard that defines a model for workload identity. It doesn't implement anything by itself. It specifies the format of identities, the API for requesting them, and the trust model that makes them verifiable across services, clusters, and clouds. SPIRE is the reference implementation of that specification.

SPIFFE IDs and Trust Domains

A SPIFFE identity is a URI with a specific format:

spiffe://<trust-domain>/<workload-path>

The trust domain is a string that identifies the administrative boundary — typically your organisation, cluster, or environment. Everything within the same trust domain can verify each other's identities. Identities from different trust domains require explicit federation configuration.

Some concrete examples:

spiffe://payments.corp/ns/production/sa/checkout
spiffe://analytics.corp/ns/data/sa/pipeline-worker
spiffe://cluster.local/ns/monitoring/sa/prometheus

The path after the trust domain is arbitrary — it's defined by your SPIRE configuration and typically encodes the Kubernetes namespace and service account of the workload.

SVIDs: The Cryptographic Identity Document

An SVID — SPIFFE Verifiable Identity Document — is how a SPIFFE identity is materialised into something a service can actually use.

There are two SVID formats.

An X.509 SVID is a standard TLS certificate where the SPIFFE ID is embedded in the Subject Alternative Name (SAN) URI field. Because it's a standard X.509 certificate, any TLS library can use it without modification.

The workload presents this certificate in a TLS handshake, and the peer verifies the certificate was signed by a trusted SPIRE server. This is the format used for long-lived connections like gRPC streams.

A JWT SVID is a signed JSON Web Token containing the SPIFFE ID as a claim. It's suitable for request-based authentication over HTTP — pass it in an Authorization header, and the receiving service verifies the signature.

JWT SVIDs are shorter-lived than X.509 SVIDs and scoped to a specific audience to prevent token reuse across services.

For Cilium's mutual authentication, X.509 SVIDs are used. The rest of this article focuses on X.509.

The Trust Bundle

For service A to verify service B's certificate, service A needs to know which Certificate Authority signed it. In SPIFFE, this is called the trust bundle — the set of CA certificates that are trusted within a trust domain.

SPIRE makes the trust bundle available via the Workload API. When a workload requests its identity, it also receives the current trust bundle. When the SPIRE server rotates its CA, it distributes the new trust bundle to all agents, which push it to all workloads. Your application never has to manage trust bundles manually.

How SPIRE Works

SPIRE is the engine that issues SVIDs and manages the identity lifecycle. Understanding its architecture is what makes the Cilium integration make sense.

SPIRE Server and SPIRE Agent

SPIRE has two main components. The SPIRE Server is the central CA. It maintains a registry of workload entries (records that describe which SPIFFE IDs should be issued to which workloads). It issues SVIDs to agents on behalf of workloads, and it's the root of trust for the entire trust domain.

The SPIRE Agent runs on every node as a DaemonSet. It has two jobs. First, it proves to the SPIRE Server that it's running on a legitimate node. This is called node attestation. Second, it exposes the SPIFFE Workload API on a Unix socket on the node, which workloads use to request their SVIDs.

The agent caches SVIDs locally so that a temporary loss of connection to the SPIRE Server doesn't immediately break workload identity.

This split — central server, per-node agents — is deliberate. Workloads never contact the SPIRE Server directly. They only talk to the agent on their own node. The agent mediates all identity requests, which limits the blast radius if a node is compromised.

Node Attestation

When a SPIRE Agent starts up on a new node, it needs to prove its own identity to the SPIRE Server before it can serve identities to workloads. This is node attestation.

In Kubernetes, SPIRE uses PSAT — Projected Service Account Tokens — for node attestation. The agent presents a Kubernetes service account token that is projected specifically for the SPIRE server's audience. The SPIRE Server contacts the Kubernetes API to verify the token, confirms the agent is running in the expected namespace with the expected service account, and issues the agent its own SVID.

This is the reason SPIRE requires specific Kubernetes API flags. The kube-apiserver must be configured to support projected service account tokens with the right audience, which is why the kind cluster config in the demo below sets --api-audiences and --service-account-issuer.

Workload Attestation

Once a node has been attested, its agent can attest workloads. When a workload connects to the Workload API socket and requests an SVID, the agent collects facts about that workload (like its Kubernetes namespace, service account, pod name, and labels) by querying the Kubernetes API. It matches those facts against the workload entries registered in the SPIRE Server. If a matching entry exists, the agent issues the corresponding SVID.

A workload entry looks like this:

SPIFFE ID: spiffe://example.org/ns/production/sa/checkout
Parent ID: spiffe://example.org/spire/agent/k8s_psat/default/<node-uid>
Selectors:
  k8s:ns:production
  k8s:sa:checkout

The selectors describe the Kubernetes facts that must match. A pod running in the production namespace with service account checkout will receive the SPIFFE ID spiffe://example.org/ns/production/sa/checkout. Any other pod will not.

SVID Issuance and Rotation

SVIDs are short-lived by design. The default TTL for X.509 SVIDs in SPIRE is one hour. The SPIRE Agent automatically rotates them in the background — generating a new key pair, requesting a fresh SVID from the server, and making the new SVID available on the Workload API before the old one expires.

Workloads that use the Workload API directly or tools like the SPIFFE CSI driver get the new SVID transparently.

Short-lived credentials are the zero-trust way. If a workload's SVID is compromised, it's only valid for an hour. Compare that to a Kubernetes service account token, which was historically valid forever.

How Cilium Implements Mutual TLS with SPIFFE

Traditional approaches to service mesh mTLS (like Istio or Linkerd) inject a sidecar proxy into every pod. The proxy intercepts all traffic and handles the TLS handshake. The application has no idea TLS is happening. The sidecar adds memory overhead (roughly 50–100MB per pod for Envoy), an extra network hop on every request, and a complex certificate injection mechanism.

Cilium takes a different path. Rather than injecting a proxy, it handles authentication at the network layer using eBPF. The Cilium agent running on each node intercepts connections, performs the mutual TLS handshake using SPIFFE SVIDs, and enforces the authentication result — all in the kernel, without any user-space proxy.

The mechanism works like this. When pod A initiates a connection to pod B, the Cilium agent on pod A's node intercepts the connection. It retrieves pod A's SVID from the SPIRE Workload API. It checks whether there's a CiliumNetworkPolicy requiring mutual authentication for this connection. If there is, it performs a TLS handshake with the Cilium agent on pod B's node, presenting pod A's SVID and requesting pod B's SVID in return.

Both agents verify the SVID against the SPIRE trust bundle. If both SVIDs are valid and the policy allows the connection, it proceeds. If either SVID is invalid or missing, the connection is dropped.

The application on pod A receives data from the application on pod B. Neither application wrote any TLS code. Neither has a sidecar. The authentication happened entirely in the Cilium agents on their respective nodes.

In Cilium's model, the Cilium agent itself gets a SPIFFE identity from SPIRE. It acts as a delegate identity that can request SVIDs on behalf of workloads.

This is slightly different from the standalone SPIRE model where each workload requests its own SVID directly. The Cilium operator registers workload entries in SPIRE automatically based on the Kubernetes Identities it manages, so you don't need to manually create SPIRE entries for every pod.

Demo 1 — Install Cilium with SPIRE Integration

You'll create a kind cluster with Cilium as the CNI and enable its built-in SPIRE integration in a single Helm command.

Step 1: Install the Cilium CLI

# macOS
brew install cilium-cli

# Linux
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar -xzf cilium-linux-amd64.tar.gz -C /usr/local/bin

Step 2: Create a kind Cluster Without a Default CNI

kind's default CNI (kindnet) must be disabled so Cilium can take its place. Save this as kind-cilium.yaml:

# kind-cilium.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
networking:
  disableDefaultCNI: true   # Required: let Cilium be the CNI
  kubeProxyMode: none       # Cilium replaces kube-proxy too

kind create cluster --name k8s-mtls --config kind-cilium.yaml

The nodes will be in a NotReady state until Cilium is installed. This is expected because there's no CNI yet.

Step 3: Install Cilium with SPIRE Enabled

Because Step 2 set kubeProxyMode: none, Cilium has to play the kube-proxy role itself. That means its bootstrap pods can't reach the API server via the kubernetes Service ClusterIP, because nothing is routing it yet.

You have to pass the API server's real address up front. Grab the kind control-plane's IP from Docker:

API_SERVER_IP=$(docker inspect k8s-mtls-control-plane \
  --format='{{ .NetworkSettings.Networks.kind.IPAddress }}')
echo "API_SERVER_IP=$API_SERVER_IP"

Then install Cilium with SPIRE:

helm repo add cilium https://helm.cilium.io/
helm repo update

helm upgrade cilium cilium/cilium \
  --install \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=6443 \
  --set authentication.enabled=true \
  --set authentication.mutual.spire.enabled=true \
  --set authentication.mutual.spire.install.enabled=true \
  --set authentication.mutual.spire.install.server.dataStorage.enabled=false

A few of these flags are easy to miss but each is load-bearing:

• kubeProxyReplacement=true: Cilium installs its eBPF-based replacement for kube-proxy. Mandatory whenever the kind config sets kubeProxyMode: none.

• k8sServiceHost / k8sServicePort: direct API server address used during bootstrap, before Cilium can route the Service ClusterIP. On EKS/GKE/AKS you don't need this because kube-proxy is still present during install.

• authentication.enabled=true: required alongside authentication.mutual.spire.enabled=true. The chart's validate.yaml rejects the install with SPIRE integration requires .Values.authentication.enabled=true and .Values.authentication.mutual.spire.enabled=true if you set only the mutual flag.

• dataStorage.enabled=false: switches the SPIRE server from a PVC-backed datastore to in-memory. Fine for a lab cluster, but in production leave this enabled and ensure your cluster has PersistentVolume support.

Notice there's no --wait flag here. On a fresh cluster, --wait will appear to fail with context deadline exceeded because the install is racey by design. The SPIRE server has to schedule on a NotReady node thanks to its tolerations, then Cilium agents come up using SPIRE, then nodes flip to Ready. Let the install return immediately and watch the pods come up over the next ~2 minutes:

kubectl get pods -A -w

Step 4: Verify the Installation

cilium status --wait

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet              cilium-envoy       Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2

Three Cilium agents, one per node, including the control-plane (no taints in the kind config). Check the SPIRE components in the cilium-spire namespace:

kubectl get all -n cilium-spire

NAME                    READY   STATUS    RESTARTS   AGE
pod/spire-agent-2cpsr   1/1     Running   0          3m
pod/spire-agent-klhjx   1/1     Running   0          3m
pod/spire-agent-vhsnc   1/1     Running   0          3m
pod/spire-server-0      2/2     Running   0          3m

NAME                              TYPE        CLUSTER-IP    PORT(S)    AGE
service/spire-server              ClusterIP   10.96.x.x     8081/TCP   3m

NAME                          DESIRED   CURRENT   READY   AGE
daemonset.apps/spire-agent    3         3         3       3m

NAME                             READY   AGE
statefulset.apps/spire-server    1/1     3m

One SPIRE agent per node. The SPIRE server is a StatefulSet with two containers: the server itself plus the SPIRE controller manager, which automatically creates workload registration entries for Cilium identities.

Run a health check on the SPIRE server:

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server healthcheck

Server is healthy.

Verify the SPIRE agents have been attested:

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server agent list

Found 3 attested agents:

SPIFFE ID         : spiffe://spiffe.cilium/spire/agent/k8s_psat/default/<node-uid-1>
Attestation type  : k8s_psat
Expiration time   : 2026-05-17 21:08:47 +0000 UTC
Serial number     : 91532884191503307904684123063465502141
Can re-attest     : true

SPIFFE ID         : spiffe://spiffe.cilium/spire/agent/k8s_psat/default/<node-uid-2>
...

Three agents, one per node, all attested via Kubernetes PSAT. The SPIRE server trusts every node and will issue SVIDs to workloads running on them.

At this point the identity platform is fully in place, but nothing is using it yet. Demo 1 built the machinery that issues cryptographic identities. Demo 2, which we'll walk through next, puts that machinery to work, turning those SVIDs into an enforced mutual-TLS policy between two real services. Keep the cluster from Demo 1 running, as Demo 2 builds directly on it.

Demo 2 — Enforce Mutual TLS with a CiliumNetworkPolicy

Picking up in the same cluster from Demo 1, you'll deploy two services, enforce mutual authentication between them with a CiliumNetworkPolicy, verify that authenticated traffic flows, and confirm that unauthenticated connections are blocked.

Every request here is authenticated with the SVIDs that the SPIRE server you just verified hands out. These two demos are one continuous walkthrough, not standalone exercises.

Step 1: Deploy a Client and Server

This file contains both the server and the client — the client is a sleeping curl pod we'll use to exec into.

# echo-workloads.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
    spec:
      containers:
        - name: echo-server
          image: ealen/echo-server:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo-server
  namespace: default
spec:
  selector:
    app: echo-server
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-client
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-client
  template:
    metadata:
      labels:
        app: echo-client
    spec:
      containers:
        - name: client
          image: curlimages/curl:latest
          command: ["sleep", "infinity"]

kubectl apply -f echo-workloads.yaml
# kubectl rollout status only takes one resource at a time
kubectl rollout status deployment/echo-server -n default
kubectl rollout status deployment/echo-client -n default

Step 2: Confirm Traffic Flows Without Authentication

Before enforcing mTLS, confirm the client can reach the server:

CLIENT=$(kubectl get pod -l app=echo-client -o jsonpath='{.items[0].metadata.name}')
kubectl exec $CLIENT -- curl -s http://echo-server/

You should get a JSON response from the echo server. Traffic flows freely with no authentication.

Step 3: Apply a CiliumNetworkPolicy Requiring Mutual Authentication

Adding authentication.mode: required to a CiliumNetworkPolicy tells Cilium to enforce mutual TLS for matching traffic. Both sides of the connection must present a valid SPIFFE SVID:

# mtls-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: echo-server-mtls
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      app: echo-server
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: echo-client
      authentication:
        mode: required     # Require mutual TLS for this traffic

kubectl apply -f mtls-policy.yaml

Step 4: Verify Authenticated Traffic Still Flows

kubectl exec $CLIENT -- curl -s http://echo-server/

The connection succeeds. Cilium intercepted it, performed the SPIFFE mTLS handshake between the Cilium agents on both pods' nodes, verified both SVIDs, and allowed the traffic through. The application on the client sent a plain HTTP request and received a response — the mutual authentication happened transparently at the network layer.

Step 5: Observe the Authentication with Hubble (Optional)

Hubble is Cilium's observability layer. It needs its own CLI:

# macOS
brew install hubble

# Linux
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --remote-name-all \
  https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz
sudo tar -xzf hubble-linux-amd64.tar.gz -C /usr/local/bin

Enable Hubble in the cluster, then watch flows. cilium hubble enable deploys Hubble Relay and restarts the Cilium agents to switch on the Hubble server inside them, so wait for it to settle before port-forwarding. If you skip the wait, the port-forward connects before Relay is listening, then dies with connection reset by peer / rpc error … EOF:

cilium hubble enable
cilium status --wait          # wait for "Hubble Relay: OK" before continuing

cilium hubble port-forward &

# Watch flows for the echo-server (Ctrl-C to stop)
hubble observe --namespace default --pod echo-server --follow

Trigger another request in a second terminal:

kubectl exec $CLIENT -- curl -s http://echo-server/

In the Hubble output you'll see:

ℹ️  Hubble Relay is available at 127.0.0.1:4245
Jul  7 12:44:42.380: default/echo-client-86d446b8f-9bn5v:47500 (ID:2822) -> default/echo-server-7467b4b54d-5tvkz:80 (ID:30854) policy-verdict:none TRAFFIC_DIRECTION_UNKNOWN ALLOWED (TCP Flags: SYN)
Jul  7 12:44:42.380: default/echo-client-86d446b8f-9bn5v:47500 (ID:2822) -> default/echo-server-7467b4b54d-5tvkz:80 (ID:30854) to-endpoint FORWARDED (TCP Flags: SYN)
Jul  7 12:44:42.381: default/echo-client-86d446b8f-9bn5v:47500 (ID:2822) <- default/echo-server-7467b4b54d-5tvkz:80 (ID:30854) to-endpoint FORWARDED (TCP Flags: SYN, ACK)

The ALLOWED verdict with the policy-verdict reason confirms the CiliumNetworkPolicy matched and authentication was verified. No sidecar involved — this happened in the Cilium agents.

Prefer a graphical view? Enable the Hubble UI. Everything above is the API + terminal path (Relay on port 4245 backs the hubble CLI). Hubble also ships a web dashboard with a live service map — but it's a separate component that cilium hubble enable does not start by default:

# Add the UI (re-runs enable, keeps Relay, adds the hubble-ui deployment)
cilium hubble enable --ui

# Wait for it to be Ready before opening — same race as Relay. Skip this and
# `cilium hubble ui` fails with "connection refused" on port 8081, because the
# UI's frontend container isn't listening yet.
kubectl -n kube-system rollout status deployment/hubble-ui --timeout=90s

# Port-forwards hubble-ui and opens http://localhost:12000 in your browser
cilium hubble ui

Select the default namespace from the dropdown. That's where the demo pods and the policy live. The map is live: it renders edges from flows as they happen, so an idle namespace looks empty. Trigger a request to light it up:

kubectl exec $CLIENT -- curl -s http://echo-server/

You'll see a forwarded edge echo-client → echo-server. Click it (or open the flow table at the bottom) to read the policy-verdict: ALLOWED. Leave the UI open through Step 6. When you run the unauthorized-client test there, its connection shows up as a red dropped edge, the visual counterpart to the curl timeout.

The UI has three parts.

The service map at the top draws each workload identity as a box and each observed connection as an edge colored by verdict: echo-client → echo-server:80 is a solid green (forwarded) edge, while the box labelled default (that's the unauthorized pod, which carries only the namespace identity because it has no app label, so Hubble names it after that) reaches echo-server over a red dashed (dropped) line. The 🔒 lock on echo-server's → 80 TCP port marks that endpoint as mutually authenticated by the policy.

The flow table underneath logs one row per flow: source identity, destination identity, destination port, L7 info, Verdict, and timestamp. This lets you read both outcomes side by side, with echo-client → echo-server rows marked forwarded and default → echo-server rows marked dropped. This is the same allow/deny split as the CLI, one line per packet.

The top bar holds the namespace selector, a flow filter, the Any verdict / Visual toggle, and a live flows/s rate alongside the count of reporting nodes (3/3).

Step 6: Verify That a Pod Without the Matching Label is Blocked

Deploy a third pod without the echo-client label and try to reach the server:

# unauthorized-client.yaml
apiVersion: v1
kind: Pod
metadata:
  name: unauthorized
  namespace: default
spec:
  containers:
    - name: client
      image: curlimages/curl:latest
      command: ["sleep", "infinity"]

kubectl apply -f unauthorized-client.yaml
kubectl wait --for=condition=Ready pod/unauthorized --timeout=60s
kubectl exec unauthorized -- curl -sS --max-time 5 http://echo-server/

curl: (28) Connection timed out after 5000 milliseconds

The connection times out. The CiliumNetworkPolicy only permits ingress from pods with app: echo-client. A pod without that label gets no SVID match and no policy match. Cilium drops the traffic silently.

There are two gotchas to watch out for here. Run kubectl wait before exec. Run exec too soon after apply and you get container not found ("client") because the pod's container hasn't started yet.

And use curl -sS, not plain -s. With only -s, curl swallows the error text and you just see command terminated with exit code 28. That's the same result — 28 is curl's timeout code — but the -S restores the readable message. The fact that it times out (rather than "connection refused") is the signature of a policy drop: the packets are silently blackholed, not actively rejected. A refusal would return instantly with a different error.

Step 7: Check the Workload Entries in SPIRE

Cilium's SPIRE controller manager automatically created SPIFFE identities for the Cilium security identities in this cluster. You can see them:

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- \
  /opt/spire/bin/spire-server entry show \
  -selector cilium:mutual-auth

Each entry maps a Cilium security identity to a SPIFFE ID. The Cilium operator manages this registry automatically, so you never need to register workloads manually when using Cilium's built-in integration.

Conclusion

IP addresses are location, not identity. And in Kubernetes, location changes with every deployment, so any policy built on address matching silently degrades over time.

Cryptographic workload identity fixes that at the foundation. SPIFFE defines the model (a SPIFFE ID names a workload within a trust domain, an X.509 SVID materialises it into a certificate any TLS library can verify), and SPIRE implements it: the server is the CA and registry, while per-node agents attest via Kubernetes PSAT and issue short-lived, auto-rotating SVIDs.

Cilium wires that identity layer into the network. Add authentication.mode: required to a CiliumNetworkPolicy and its eBPF agents fetch both workloads' SVIDs, run the mutual TLS handshake, and enforce the verdict. There's no sidecar, no application changes, and near-zero overhead versus a service mesh. And you deployed the whole stack in a single Helm command: the complexity lives in the infrastructure, not in your code.

Cleanup (kind)

# Delete demo workloads
kubectl delete deployment echo-server echo-client -n default
kubectl delete service echo-server -n default
kubectl delete pod unauthorized -n default
kubectl delete ciliumnetworkpolicy echo-server-mtls -n default

# Uninstall Cilium (helm doesn't delete the cilium-spire namespace it created)
helm uninstall cilium -n kube-system
kubectl delete namespace cilium-spire

# Delete the cluster (easiest reset on kind)
kind delete cluster --name k8s-mtls

But what happens when the task actually takes time? Something like "go through the last three days of my emails, draft replies for anything urgent, and create Linear tickets for the engineering-related ones." That's not a quick job. It might take minutes, hours, or even longer. And this is just one example.

That's a full workflow, and the moment your server crashes or your process restarts, you lose everything. No retry, and no resume. You're starting from scratch.

That's the problem this article is about.

In this article, you'll build a durable background agent runtime that holds up under real conditions. Dispatch a task, walk away, and it gets done.

Under the hood, it runs on Kubernetes with KEDA autoscaling so workers scale to zero when idle and spin back up the moment work arrives. For crash recovery and durable execution we'll use Temporal, and for agentic capabilities and tool usage we'll use Composio.

What's Covered?

In this tutorial, you'll build a durable background agent runtime that runs on Kubernetes and scales based on actual workload. Here's what you'll learn along the way:

• What an agent loop is and how to build one with Claude and Composio

• How to make long-running agent tasks handle crashes using Temporal

• How to build a gateway that decouples task dispatch from execution

• How to containerize the worker and gateway with Docker

• How to deploy the full system to a local Kubernetes cluster

• How to autoscale workers to zero with KEDA based on queue depth

This gets into some advanced concepts, but follow along and you'll learn a lot along the way.

Table of Contents

• What's the Plan (the Architecture)

  • Dispatching a Task:

  • Running the Task

  • Scaling

• How to Set Up the Project

• Core Components in the Application

  • The Agent Loop

  • Making it Durable with Temporal

  • The Agent Gateway

  • Containerizing the Application

  • Deploying to Kubernetes

  • Autoscaling with KEDA

• Agent in Action

• Conclusion

What's the Plan (the Architecture)

Before diving into the code, it helps to understand how everything fits together.

The system is split into two distinct planes: a control plane that handles user-facing interactions (Next.js frontend), and an execution plane where the actual agent work happens. They never directly call each other, and that separation is intentional.

Here's the flow from start to finish:

Dispatching a Task

When a user submits a goal, the gateway first runs a pre-flight check to verify the required Composio tool connections are active for that user. If they are, it hands the task off to Temporal and returns immediately. The user doesn't wait around.

NOTE: You don't wait for the response to be back from the agent. It just happens all in the background. This isn't your regular chat app. You just launch the task and you forget.

Running the Task

Temporal puts the task on a queue and a worker pod picks it up. The worker runs the agent loop, LLM reasons over the goal, Composio executes the tools, and the result gets written back to Temporal. The frontend automatically polls the gateway for status updates so the user can see progress without doing anything.

Scaling

KEDA watches the Temporal queue depth and scales worker pods based on how much work is pending. When the queue is empty, workers scale down to zero. When tasks come in, they load back up. That's the beauty!

The reason the gateway never touches agent code is straightforward: agent tasks can take minutes, or even hours based on the work, and you don't want that in your API layer. Keeping them separate helps the control plane stay fast regardless of what's happening in the background.

Also, the application supports Linux CronJob-style task scheduling with no human involved. So, having a pre-flight check helps there, because failing fast at dispatch is much better than having a task silently fail because a tool connection was missing.

That's pretty much the high level architecture of our application. To put it simply:

• Kubernetes (k8s): Orchestration Layer

• KEDA: Auto-scaling Layer

• Temporal: Durability Layer

• Composio: Tool Layer

• Any LLM of your choice (in our case, Anthropic) = Reasoning layer

How to Set Up the Project

Before you start, make sure you have the following installed:

• Docker

• k3d (for running a local Kubernetes cluster)

• kubectl

• Helm

• Node.js and Python 3.11+

You'll also need API keys for Anthropic and Composio.

Start by cloning the repository:

git clone https://github.com/shricodev/kron-k8s-agent.git
cd kron-k8s-agent

Next, create the cluster, build the images, and load them in:

# Create the local cluster
k3d cluster create agent --wait

# Build both images and import them into the cluster
bash scripts/build-images.sh
bash scripts/load-images.sh

# Deploy Temporal (creates the temporal namespace, Postgres, and server)
kubectl apply -f infra/k8s/temporal/temporal-dev.yaml

Next, create the namespace and your secret. The secret has to exist before the app gets deployed, since the pods read their keys from it:

# Create the agent namespace
kubectl apply -f infra/k8s/00-namespace.yaml

# Create the secret with your keys (you're supposed to remove the placeholders with the actual values...)
kubectl create secret generic agent-secrets -n agent \
 --from-literal=ANTHROPIC_API_KEY=sk-ant-... \
 --from-literal=COMPOSIO_API_KEY=ak_... \
 --from-literal=JWT_SECRET=$(openssl rand -hex 32)