But you don't manage the servers it runs on or maintain the underlying infrastructure. You simply sign in and get to work. This is called Software as a Service (SaaS), and many people use it for years without knowing the term.

SaaS is just one of three major cloud service models. The other two, Infrastructure as a Service (IaaS) and Platform as a Service (PaaS), give you different levels of control over the infrastructure and software you build. Understanding how these models differ will help you make sense of cloud engineering concepts and choose the right approach for your projects.

In this article, we'll break down IaaS, PaaS, and SaaS, explain what you manage and what the cloud provider handles in each model, and learn how to choose between them for a real project.

What You'll Learn

By the end of this article, you'll be able to:

• Explain what IaaS, PaaS, and SaaS are and what each one manages for you
• Use the shared responsibility model to see where your job starts and ends
• Recognise real products in each category
• Choose the right model for a given project

Table of Contents

• Prerequisites
• What Is Cloud Computing?
• The Shared Responsibility Model
• Infrastructure as a Service (IaaS)
• Platform as a Service (PaaS)
• Software as a Service (SaaS)
• Beyond the Big Three: CaaS, FaaS, and Serverless
• How to Choose the Right Service Model
• Conclusion

Prerequisites

You don't need any cloud experience to follow this article. If you know what a computer, a server, and the internet are, you have everything you need.

If you're new to the cloud, read my article What is the Cloud and Cloud Computing? first. It explains what the cloud is and how it differs from running your own hardware, and this article builds on that foundation.

What Is Cloud Computing?

Cloud computing is a model that lets you use computing resources, such as servers, storage, and networking, on demand over the internet. YOu pay only for what you use.

These resources sit in data centres that are owned and run by cloud service providers, or CSPs for short. The best-known CSPs are Amazon Web Services (AWS), Microsoft Azure, and Google Cloud.

This is just a short recap to make sure you understand the key concepts. My article above covers it in full if you want or need more info.

What Does On-Premises Mean?

Before the cloud, most organisations ran their systems on-premises. On-premises means the hardware sits in a building the organisation owns or rents. This could be a server room in their office, with its own staff looking after it.

In this setup, you're responsible for everything. This includes buying servers, setting up the network, installing and updating software, and keeping the room powered and cool.

For example, if a small business wanted to launch an online shop, it would need to buy a server, install the software, and pay someone to keep it running. If the shop grew faster than expected, it would need to buy more hardware.

What Changes When You Move to the Cloud?

The cloud moves some of that work to the provider. The physical servers still exist, but someone else owns and maintains them, and you rent what you need.

What differs from one cloud service to the next is how much of the work the provider takes on for you. The cloud service models describe this, and there are three main ones:

• Infrastructure as a Service (or IaaS): Here the provider gives you virtual servers, storage, and networking. Then you manage everything you put on them.
• Platform as a Service (or PaaS): Here, the provider also manages the operating system and the tools your code needs to run. You bring your application and your data.
• Software as a Service (or SaaS): In this model, the provider gives you a finished application. You sign in, use it, and manage your users and data.

So a cloud service model describes how the work is split between you, the customer, and the provider, that is which parts of the system you manage and which parts the provider manages for you.

The Shared Responsibility Model

The shared responsibility model defines which parts of a cloud system the provider manages and which parts you, as the customer, manage. To understand it, you first need to see the parts that make up a running application.

The Layers of a Computing Stack

A computing stack is the set of layers that work together to run an application. Each layer depends on the ones beneath it. From the bottom up, we have:

• Networking: the physical cables, switches, and routers that connect servers to each other and to the internet.
• Storage: the physical drives where files and databases are saved.
• Servers: the physical computers that provide processing power (CPU) and memory (RAM).
• Virtualisation: software that splits one physical server into several virtual machines, or VMs for short. A VM behaves like a separate computer even though it shares hardware with other VMs.
• Operating system: the base software, such as Linux or Windows, that manages the hardware and lets other programs run.
• Middleware: software that connects your application to the outside world and to other services, such as a web server that receives requests from browsers.
• Runtime: the environment your code needs to run, such as Node.js for JavaScript or the Python interpreter for Python.
• Application: the software itself, such as an online shop or an email client.
• Data: the information the application stores and processes, such as customer records, orders, and emails.

Who Manages What in Each Model

AWS describes the split as security “of” the cloud and security “in” the cloud. The provider manages the security of the cloud. This covers the data centres, physical hardware, physical network, and the virtualisation layer.

But security in the cloud is the customer's job. It covers what you put into the cloud and how you configure it. This includes your data, your users and their permissions, and the settings of the services you use.

Infrastructure as a Service (IaaS)

Infrastructure as a Service is a cloud model where the provider gives you the basic building blocks of computing. This includes virtual servers, storage, and networking.

How IaaS Works

With IaaS, you request a virtual machine from the provider. You choose how many CPUs and how much memory it has, which operating system it runs, and which network it connects to. Within minutes, the VM is running, and you can connect to it.

From that point, the server is yours to manage. You install your software, apply operating system updates, configure the firewall, and keep your application running.

Before IaaS, getting a new server meant ordering hardware and waiting weeks for installation. With IaaS, the same server is ready in minutes, and you can delete it when you no longer need it.

Just remember: IaaS removes the hardware burden, but it doesn't remove the server burden.

With IaaS, you usually pay for each hour or second a server is running, plus the storage you keep and outbound data transfer.

Real-World Examples include:

• Amazon EC2 (Elastic Compute Cloud) on AWS
• Google Compute Engine on Google Cloud
• Azure Virtual Machines on Microsoft Azure

Each of these gives you virtual machines. They work alongside other IaaS building blocks, such as virtual disks (Amazon EBS), storage for files and backups (Amazon S3), and private networks (Amazon VPC).

When to Use IaaS

IaaS is the right choice when you need control over the operating system and everything on it, especially when:

• You need a specific operating system version or software that a managed platform doesn't support.
• You need to meet security or compliance rules that require you to configure and audit the server yourself.
• You're moving an existing application from your own servers to the cloud without rewriting it. This is often called “lift and shift”.
• You need development and test environments that you create and delete often.

Tradeoffs

As with any system, there are some tradeoffs you should be aware of before selecting this model.

First, you manage the operating system. You handle patching, security updates, and backups, and a missed update can create a security gap.

You also need to plan for scaling. The provider gives you tools to add servers automatically, but you have to set them up and test them.

In addition, you may need to acquire some new skills. Running servers well requires knowledge of Linux or Windows administration, networking, and security.

And you also pay for idle time. A server that runs all night with no traffic still costs money.

Platform as a Service (PaaS)

Platform as a Service is a cloud model where the provider gives you a ready environment to run your code. The provider manages the servers, operating system, middleware, and runtime. The customer brings the application code and data.

How PaaS Works

With IaaS, you still have to prepare a server before your application can run. This includes updating the operating system, setting up a web server, installing the right language version, and keeping it all maintained.

PaaS removes this burden. With PaaS, you tell the platform which language your application uses, then send your code, usually by pushing to Git or uploading a package. The platform builds your application, runs it on managed servers, and gives you a web address. When traffic grows, it can run more copies of your application based on rules you set.

You still decide how much memory your application gets, which settings it uses, and who can deploy it. You just no longer have to log in to servers to patch and maintain them.

With PaaS, you usually pay for the resources your application uses, such as the size and number of running instances, billed by the hour or month.

Real-World Examples include:

• Heroku
• AWS Elastic Beanstalk
• Google App Engine
• Azure App Service

There are some products that sit between categories. For example, Firebase from Google is often called a PaaS, but it's more accurately a Backend as a Service, or BaaS. A BaaS gives you ready-made backend features, such as a database, user authentication flow, and file storage, that your app calls directly. You don't deploy a whole application to it the way you would on Heroku, for example.

Another is Amazon RDS (Relational Database Service), which is a managed database. AWS runs the database software and backups, while you manage the tables, data, and access. It works like PaaS, but for databases.

When to Use PaaS

• You want to launch a web application or API quickly and focus on code.
• Your team is small and has nobody dedicated to managing servers.
• Your application uses a common language the platform supports, such as Node.js, Python, Java, or .NET.
• You want deployments, scaling, and monitoring without building them yourself.

Tradeoffs

The first tradeoff here is that you have less control. You can't install just any software you want or change operating system settings. Also, if the platform doesn't support something, you have to work around it.

You also risk vendor lock-in. Vendor lock-in means that moving to another provider becomes hard because your application depends on features only one platform offers.

Also, you may pay more at scale. As traffic grows, PaaS can cost more than running the same workload on servers you manage because you pay for the convenience.

And you need to work within platform limits, such as maximum request times or supported language versions.

Finally, you still own your code’s security, including the third-party libraries it uses (and all the risks that come with that).

Software as a Service (SaaS)

Software as a Service is a cloud model where the provider delivers a complete application over the internet. You access it through a web browser or an app, and the provider manages everything needed to keep it running.

How SaaS Works

Before SaaS, using business software meant buying a license, installing it on every computer, and updating each one for every new version. For example, if an organisation wanted its own email system, it would need its own email server, someone to run it, and a backup plan.

This is different with SaaS. The provider runs the application on its own infrastructure, and every customer uses the same centrally updated version. You create an account, add your users, and start working. New features and security fixes arrive without you installing anything.

Your work moves from running software to configuring it. You decide who gets an account, what each person can access, and how files are shared.

With SaaS, you usually pay per user per month or per year, often in tiers with more features at higher levels.

Real-World Examples include:

• Gmail and Google Workspace, for email, documents, and file storage
• Microsoft 365, for Outlook, Word, Excel, and Teams
• Slack, for team messaging
• Zoom, for video meetings
• Dropbox, for file storage and sharing
• Salesforce, for managing customers and sales

When to Use SaaS

• The software solves a common problem that many organisations share, such as email, documents, chat, or accounting.
• You don't want to build or run the software yourself.
• Your team needs access from anywhere, on any device.
• You need the tool working this week, not in six months.

For example, say a non-governmental organisation with twenty staff who need professional email, shared documents, and video calls. It has no IT team. With Google Workspace or Microsoft 365, one administrator can set up accounts for every staff member in a day, while the provider handles the servers, updates, and spam filtering.

Tradeoffs

With this option, you have the least control. You get the features the provider builds, and if you need something it doesn't offer, you wait or look elsewhere.

Also, your data lives with the provider. You depend on their security, so you need to know where your data is stored and how you can export it if you leave.

Your costs could also easily grow with your team.

Further, you depend on your internet connection. When connectivity is unreliable, a browser-based tool can become unavailable when you need it most.

And you have limited customisation options. Connecting the tool to your other systems depends on what the provider allows.

Beyond the Big Three: CaaS, FaaS, and Serverless

IaaS, PaaS, and SaaS are the main models, but there are also options like CaaS, FaaS, and serverless. Let’s look at what each means and how it works.

Containers as a Service (CaaS)

A container packages your application with everything it needs to run, including its runtime, libraries, and settings. Because everything is inside the package, a container runs the same way on your laptop as it does on a server in the cloud. Docker is the most common tool for building containers.

Containers as a Service, or CaaS, is a model where the provider runs and manages containers for you. You give it your container images, and it starts them, restarts them when they fail, and spreads them across servers. The software that makes these decisions is called a container orchestrator, and Kubernetes is the most widely used one.

CaaS sits between IaaS and PaaS. You have more control than on a PaaS, because you decide exactly what goes into each container, but the provider handles much of the work of running them.

Some common examples include:

• Amazon ECS (Elastic Container Service), which is AWS’s own orchestrator
• Amazon EKS (Elastic Kubernetes Service), which is managed Kubernetes on AWS
• Google Kubernetes Engine (GKE), which is managed Kubernetes on Google Cloud

With ECS and EKS, you may still manage the virtual machines your containers run on, unless you use AWS Fargate, which runs containers without you managing any servers.

Functions as a Service (FaaS)

Functions as a Service, or FaaS, is a model where you upload small pieces of code called functions, and the provider runs each one only when an event triggers it. An event is something that happens, such as a web request arriving, a file being uploaded, or a scheduled time being reached.

You don't keep a server running while you wait for work. When the event happens, the provider starts your function, runs it, and stops it. You pay per request and for the compute time each run uses, so you pay nothing for compute when nothing happens.

For example, say that an online shop needs to create a small thumbnail every time a seller uploads a product photo. With IaaS, you keep a server running all day to watch for uploads, even at 3 a.m. when nobody is uploading. With FaaS, the upload itself triggers a function that creates the thumbnail, and you pay only for the seconds it runs.

Some common examples include:

• AWS Lambda
• Azure Functions
• Google Cloud Run functions, previously called Google Cloud Functions

FaaS has trade-offs, too. Each run has a time limit, and an idle function can take a moment longer to start. This is called a cold start.

Where Serverless Fits

Serverless describes cloud services where you never set up or manage servers, the service scales automatically with demand, and you pay only for what you use rather than for idle capacity.

Serverless isn't a fourth model on the stack. It describes how a service is operated and billed, not which layers you manage. FaaS is serverless, but so are many other managed services, such as Amazon S3 for file storage, Amazon DynamoDB for databases, and AWS Fargate for containers.

Google Cloud Run functions, which are CaaS, also fit into the serverless category.

How to Choose the Right Service Model

No single model is best for every project. The right choice depends on what you're building, who's building it, and how much control you need.

To help you decide, you can work through these questions in order:

• Does software that does this job already exist? If yes, start with SaaS.
• Are you building your own application in a common language? If yes, look at PaaS first, or FaaS for small tasks that run in response to events.
• Is your application already packaged in containers? If yes, CaaS or a serverless container platform may suit you.
• Do you need control over the operating system, custom software, or a strict compliance setup? If yes, choose IaaS.
• Is your traffic steady or unpredictable? Unpredictable traffic often suits serverless pay-per-use pricing.

Start with the most managed model that meets your needs, and move down the stack only when you have a clear reason.

Most production systems combine multiple cloud service models, each serving a different purpose.

Wrapping Up

Each service model draws the line between your work and the provider’s at a different point on the stack. IaaS gives you virtual infrastructure, and you manage the operating system and everything above it. PaaS gives you a platform, and you manage your code and data. SaaS gives you a finished application, and you manage your users, data, and settings.

CaaS and FaaS sit between these models, and serverless describes how you operate and pay. In all of them, you (the customer) remain responsible for your data and who can access it.

Thank you for reading!

You’ll move the UI and API to Google Kubernetes Engine without changing the public URL customers already use, gradually shifting traffic from AWS to GKE a few percent at a time.

By the end, you’ll have run the app locally, stood up AWS, built GKE in eight Terraform stages, tested the new environment, shifted production traffic to GKE, completed the cutover, and torn down what you no longer need.

The image below shows the middle of the migration, with both clouds live, customers still using one URL, and the database remaining on AWS.

Around that request path, three more systems do their jobs:

Table of Contents

• Before You Start

  • What You Need Installed

  • Skills You Need

  • What This Costs

  • How This Book Works

  • The Team

• Part 1. How to Run ClearPay and Put It on AWS

  • Chapter 1. Run ClearPay on Your Laptop

  • Chapter 2. Read the Code That Matters

  • Chapter 3. Prove a Retry Never Charges Twice

  • Chapter 4. Build the AWS Server With Terraform

  • Chapter 5. Deploy ClearPay to the AWS Server

  • Chapter 6. Record the AWS Baseline

• Part 2. How to Plan the Move

  • Chapter 7. Read the Migration Assignment

  • Chapter 8. Map What ClearPay Depends On

  • Chapter 9. Understand the GCP Design

• Part 3. How to Build Google Cloud With Terraform

  • Chapter 10. Set Upgcloudand Pick Your Project

  • Chapter 11. Build GCP in Eight Terraform Stages

• Part 4. How to Get ClearPay Onto GKE Staging

  • Chapter 12. Check the API Image

  • Chapter 13. Prepare the Cluster for ClearPay

  • Chapter 14. Build the CI/CD Pipeline

  • Chapter 15. Check Staging and Run the Promotion Gate

• Part 5. How to Make GKE Production-Ready

  • Chapter 16. Run the Security Review

  • Chapter 17. Watch ClearPay With Prometheus and Grafana (Optional)

  • Chapter 18. Load-Test and Break Staging on Purpose

• Part 6. How to Move Customers to GKE

  • Chapter 19. Deploy Production and the GKE Gateway

  • Chapter 20. Walk the Pre-Cutover Checklist

  • Chapter 21. Send 1 Percent of Traffic to GKE

  • Chapter 22. Raise Traffic to 100 Percent

  • Chapter 23. When the Numbers Go Bad

• Part 7. How to Finish the Migration

  • Chapter 24. Prove the Cutover With Numbers

  • Chapter 25. Run on GKE and Set a Date for AWS

  • Chapter 26. Retire the AWS Server

  • Chapter 27. Tear Down GCP and the Database

  • Chapter 28. Write the Migration Record

• What Comes After This Project

  • What Question This Project Leaves Open

• What You Should Have at the End

• How to Discuss This Project in an Interview

Before You Start

What You Need Installed

Install each tool when its chapter needs it. You don't need everything on day one.

On Windows, use Git Bash or WSL for every command in this book. The commands are written for a Bash shell.

Skills You Need

Prior Kubernetes, Terraform, GCP, or load-balancer experience isn't required. Don't buy a domain. On AWS, you open https:// plus the Elastic IP. The browser warns about a self-signed certificate. Click through it and you get the same ClearPay login as on your laptop. On GCP you add one /etc/hosts line so https://api.clearpay.test points at the load balancer, unless you turn on Cloud DNS.

What This Costs

AWS costs about $1.25 per day. GCP costs about $1.60 to $3 per day once it's up. A weekend run of the whole book costs a few dollars.

Neither environment should run for a month. If you stop for more than a couple of days, tear both down:

./scripts/teardown-gcp.sh
cd infra/aws/terraform && terraform destroy

Two files hold the detailed numbers: infra/aws/COST.md and infra/gcp/COST.md. The book sends you to those two files and no other documents.

How This Book Works

Clone the repository and run every command from its root folder unless a step says cd somewhere else:

git clone https://github.com/Osomudeya/migrate-api-aws-gke-devops.git

cd migrate-api-aws-gke-devops

The repository already holds the app, the Terraform, the Kubernetes files, the pipeline, and the scripts.

You won't write new code. Your work is to run each step, read what the system prints, and decide whether to continue.

This book reads like a story where you join a small payments company in week one, and every chapter is a task someone on the team hands you.

Each chapter opens with a time stamp and a scene, then follows the same loop:

Most Important Rule

Don't run a command you don't understand. If the book hasn't told you why yet, keep reading, because the "why" is the point.

This book has two kinds of steps.

The first simply asks whether the step worked, such as whether the file copied or the container started. If it did, continue.

The second asks you to think about what happened. Once the command finishes, read the output and understand what it means before moving to the next chapter.

The Chapters

Read the chapters in order and skip none. Chapter 1 doesn't ask you to install Kubernetes or open GCP. Those come later.

If Something Doesn't Work

Stay in that chapter. Use its "If it fails" steps. Don't skip ahead. A problem you carry forward gets harder to find, because the next chapter adds another cloud on top of it.

The job isn't to memorize Kubernetes or Terraform. It's to learn how to work through a real migration: understand what's running, make decisions based on what you find, verify the result, and know what to do when something breaks.

How Long This Takes

This might take more than one day. If you'll be away for more than a day or two, destroy the AWS and GCP resources so you're not billed for them while it's idle. The commands are in "What this costs" above.

The Team

This lab starts as Zee, a junior DevOps engineer on the ClearPay team, week one. You play Zee.

• Maya is the engineering manager. She asks for evidence.

• Daniel is a senior engineer. He pairs with you on servers, Kubernetes, and capacity.

• Amara maintains the payments API. She cares about one thing: no customer is charged twice.

• Chris handles customer support. He hears first when transfers fail.

Your job is to move the payments API from one AWS EC2 server to GKE without a customer-visible outage, while PostgreSQL stays on AWS.

Part 1. How to Run ClearPay and Put It on AWS

A migration needs a starting point you trust. In this part, you run ClearPay on your laptop, deploy it to AWS the way the company runs it today, and write down how fast it is. Every later chapter compares GKE against those numbers.

Chapter 1. Run ClearPay on Your Laptop

Week one, Monday, 9:05 a.m.

Your laptop is new. The Wi-Fi password is on a sticky note. The first message in your inbox is from Maya, the engineering manager, and it's short.

Maya: "Get ClearPay running on your laptop first. Log in, send $20, and then we'll talk about moving the API to GKE."

The code is in the repository: api/, ui/, and db/. Nothing is running yet.

Why This Matters

Later chapters need a picture of a good login and a good $20 send. When something fails on AWS or GKE, you compare it with this.

What You're About to Run

ClearPay has three parts, and each part has one job.

• The UI is what a customer sees and taps: the login page, the balance, and the Send button. UI stands for user interface.

• The API is the part that does the work. It has no screen. The UI sends it a message, such as "send $20 to Bob". The API checks the message, does the work, and sends back a reply, such as "done". API stands for application programming interface. Think of it as a door with rules. The UI asks in a fixed way, and the API answers in a fixed way.

• The database is where ClearPay writes everything down: every account, balance, and payment. PostgreSQL is the database program ClearPay uses.

• A server is a computer that stays on all day and waits for requests. The API runs on one.

A café works the same way. The menu is the UI. The waiter is the API. The kitchen and its records are the database. A customer never walks into the kitchen. The customer tells the waiter what they want.

This chapter runs all three on your laptop. Later chapters move the UI and the API to Google Cloud. The database stays on AWS.

Steps

Start Docker Desktop. Don't install Node or PostgreSQL for this chapter. They run inside the containers.

From the folder that holds docker-compose.yml:

docker compose up --build

The first build takes a couple of minutes. This terminal keeps printing logs and doesn't give you a prompt back until you press Ctrl+C. Leave it running.

Wait until all three of these lines appear. The names on the left may start with your folder name:

Container <folder>-postgres-1 Healthy
api-1       | {"message":"api_started", ...}
ui-1        |   ➜  Local:   http://localhost:5173/

Healthy means Postgres is up. api_started means the API is up. The localhost:5173 line means the UI is up.

Open a second terminal in the same folder. Create two demo accounts:

docker compose exec api node db/seed.js

This runs the seed script directly. The API image has no npm, on purpose, because the security scan in Chapter 14 flags it.

Expected output:

seeded: alice@clearpay.dev (password: password123)
seeded: bob@clearpay.dev (password: password123)

If you see skip (already exists): alice@clearpay.dev, the accounts exist from an earlier run. That's fine.

This is what's now running:

Check

In the second terminal:

curl localhost:4000/health

Here's the expected output. Your instance value will be a container ID:

{"status":"ok","served_by":{"platform":"aws","version":"1.0.0","instance":"14b8d5d767ea","commit":"local","rollback_active":false}}

"status":"ok" is the check. served_by names the platform that answered the request. It says aws on your laptop because docker-compose.yml sets PLATFORM: aws. Later, GKE answers with gke, and that one field tells you which cloud handled each request.

Now open http://localhost:5173 and sign in. Seed already created Alice, so don't register her:

• Email: alice@clearpay.dev

• Password: password123

Send $20 to bob@clearpay.dev. The transfer appears under Recent activity as Sent with a short ID, for example Sent · a1b2c3d4.

Look at two places on this dashboard. You'll watch them for the rest of the book:

1. Each line under Recent activity ends with served by aws/1.0.0. That's the platform and version of the API process that wrote the transfer. It's stored with the transfer in the database.

2. The black bar at the top says All systems operational when /health/ready succeeds. That means the API process is up and can reach Postgres. It says nothing about logins. A wrong password shows a red error on the form, while the bar keeps saying All systems operational.

After Chapter 5, you open the same dashboard on the AWS address. Same login, same two places.

If it Fails:

• failed to connect to the docker API or docker.sock: Docker Desktop isn't running. Start it, wait until it is ready, and run docker compose up --build again.

• email already registered on the Register page: Alice already exists. Go back to Sign in.

• invalid credentials: the password is the full string password123. If that still fails, run the seed command again. A skip (already exists) line means the account is there.

• docker compose up exits right away: run docker compose logs api in the second terminal. On a slow first boot the API can start before Postgres accepts connections. Run docker compose up again.

• Port 4000 or 5173 already in use: another program on your machine holds that port. Stop it, or change the port mapping in docker-compose.yml.

Decision

If the transfer went through, the application works. Continue.

If it didn't, fix it here, on the laptop. AWS and GKE are harder to debug. On this machine, the answer is one docker compose logs command away.

Done When:

• docker compose up runs without errors

• The seed reports two accounts, created or skipped

• You sent $20 in the UI and saw it under Recent activity

Chapter 2. Read the Code That Matters

Monday, 11:30 a.m.

The $20 went through. You're still looking at the transfer line when a chair rolls up next to yours. Daniel, the senior engineer at the next desk, has his coffee and no hurry.

Daniel looks over at your screen. "You don't need to memorize every line. Know which file does which job, so you can open it fast during the migration."

Why This Matters

Every later chapter touches one of these four files. Five minutes now saves you a search under pressure later.

Steps

Open these files in this order:

1. api/src/index.js starts the API and connects every route: health checks, /api/info, login, and transactions.

2. api/src/servedBy.js is 17 lines long. It reads PLATFORM, APP_VERSION, and ROLLBACK_ACTIVE from environment variables. The dashboard uses it to print served by aws/1.0.0.

3. api/src/routes/transactions.js handles sending money. Find the idempotency_key check near the top. It runs before anything else.

4. api/src/db.js connects the API to Postgres. Find DB_POOL_SIZE || '10'.

Daniel stops you on the last file.

"A connection is an open line to the database. One API process can hold up to ten of those lines at the same time. Remember that number."

What You Now Know

index.js                 → starts the API and connects the routes
servedBy.js              → says which cloud answered (aws or gke)
routes/transactions.js   → sends money, checks the idempotency key first
db.js                    → talks to Postgres, up to 10 connections per process

servedBy.js fills the dashboard line served by aws/1.0.0. That's which cloud answered, not who logged in.

Today there's one API process, so ten connections is the whole pool. On GKE, several copies of the API run at once. Kubernetes calls each copy a pod. Each pod gets its own pool of ten, and every pod shares the same database. Chapter 22 comes back to that math.

Done When:

• You can say, without looking, which file sets the connection limit and which file names the platform

Chapter 3. Prove a Retry Never Charges Twice

Monday, 2:10 p.m.

After lunch, you get a message from someone you haven't met yet. Amara maintains the payments API. Maya told her you have ClearPay running on your laptop, so she skips the greeting.

Amara: "Send $20 twice with the same key. Alice's balance should drop by $20, not $40."

Why This Matters

Alice sends $20 to Bob. She taps Send, and the screen freezes. She doesn't know if the payment worked, so she taps Send again.

Without protection, the API sees two requests and takes $20 two times. Alice loses $40 for one payment.

The idempotency_key stops this. "Idempotent" means safe to repeat: doing it twice gives the same result as doing it once.

The key is a label the app puts on each payment, for example round-a.

1. The first request arrives with the label round-a. The API takes $20 and saves the label.

2. The second request arrives with the same label. The API sees round-a again. It knows this is the same payment, so it takes no money. It sends back the first result.

Same label means same payment.

This matters more during the migration. Sometimes the first tap goes to AWS and the retry goes to GKE. Those are two different servers, but they use one database. GKE looks in that database, finds round-a, and doesn't charge again. One shared database keeps the customer safe on both clouds.

Use curl for this chapter, not the Send money button. The UI gives every click a new key, so every click is a new transfer.

For a deeper walk-through of idempotency, see PayFlow: A Production-Grade Fintech System for Learning Real DevOps.

Steps

Stay in the second terminal, in the folder with docker-compose.yml. Run one command at a time and read the output before the next one.

Log in as Alice. This stores her login token in a shell variable called TOKEN. A token is proof that Alice signed in. Later commands send it with $TOKEN.

TOKEN=$(curl -s -X POST localhost:4000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"alice@clearpay.dev","password":"password123"}' \
  | python3 -c "import sys, json; print(json.load(sys.stdin)['token'])")

This prints nothing when it works. Run echo $TOKEN to confirm. You should see a long string.

Then read Alice's account. Write down the balance_cents number.

curl -s localhost:4000/api/accounts/me -H "Authorization: Bearer $TOKEN"

This is the expected shape. Your numbers and id will differ:

{"account":{"id":"...","balance_cents":"94000","currency":"USD"},"served_by":{"platform":"aws","version":"1.0.0","instance":"...","commit":"local","rollback_active":false}}

94000 means $940.00. If you haven't sent anything yet, you'll see 100000 ($1,000.00). If the command says missing_token or invalid_token, run step 1 again.

Now pick a label you haven't used yet:

LABEL=round-a

Send $20 to Bob. 2000 is $20.00 in cents.

curl -s -X POST localhost:4000/api/transactions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"to_email\":\"bob@clearpay.dev\",\"amount_cents\":2000,\"idempotency_key\":\"$LABEL\"}"

Look for "replayed" in the reply:

• "replayed":false means this send took $20. Continue.

• "replayed":true means this label was used before, probably in an earlier run. No money moved. Set LABEL=round-b and send again until you see "replayed":false.

Send the exact same request again. Don't change LABEL.

curl -s -X POST localhost:4000/api/transactions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"to_email\":\"bob@clearpay.dev\",\"amount_cents\":2000,\"idempotency_key\":\"$LABEL\"}"

Now you should see "replayed":true, and the transaction id matches the first send.

Read the account again:

curl -s localhost:4000/api/accounts/me -H "Authorization: Bearer $TOKEN"

Check

balance_cents is 2000 lower than the number you wrote down, not 4000 lower. If you started at 94000, you now see 92000.

If it Fails:

• echo $TOKEN is empty: run the login command again.

• The second send still shows "replayed":false: LABEL changed between the two sends. Run echo $LABEL, then send again.

• Both sends showed "replayed":true: the balance won't change. Pick a new LABEL and repeat steps 3 and 4.

Done When:

• One send returned "replayed":false and the next returned "replayed":true for the same label

• balance_cents fell by 2000, not 4000

Chapter 4. Build the AWS Server With Terraform

Tuesday, 9:15 a.m.

Maya stops at your desk on her way to a meeting. Yesterday you told her that ClearPay runs on your laptop and that a retried payment charges only once.

Maya: "Good. Now, localhost isn't a baseline. Get ClearPay onto AWS and prove a customer can complete a transaction there. Then we can talk about leaving AWS. Daniel will work with you on this one."

Daniel rolls his chair over from the next desk.

Daniel: "One rule before you start. Don't skip terraform plan."

Why This Matters

This book starts from the setup many small companies run: one Linux server with no Docker on it.

systemd, the Linux service manager, starts the Node API when the server boots. nginx sits in front of it and handles HTTPS on port 443. PostgreSQL runs on Amazon RDS, the managed database service.

There's no AWS load balancer. A load balancer spreads traffic across several servers, and this setup has one. Adding one would add an hourly bill for no gain. The numbers are in infra/aws/COST.md.

The file api/Dockerfile exists, but only your laptop uses it. The AWS server never runs Docker.

Daniel: "The EC2 server won't have Docker. systemd starts Node on boot, next to nginx. That's the server you're inheriting."

The diagram above shows how a customer request reaches ClearPay on AWS. The Elastic IP provides a fixed public address for the EC2 server. Nginx handles HTTPS traffic on port 443, serves the frontend files, and forwards API requests to the Node.js process managed by systemd on port 4000. The API reads and writes payment data in PostgreSQL on Amazon RDS, which has no public endpoint at this stage.

Two AWS terms appear in this chapter:

• A VPC is the private network your servers sit in. AWS gives every account a default one. This lab uses it instead of building its own.

• A security group is AWS's firewall. It lists which ports may reach a server. This lab allows SSH from your IP only, HTTPS from anywhere, and Postgres only from the API server.

Steps

First confirm your laptop is ready. Each of these must work:

docker --version          # still needed later
aws sts get-caller-identity
terraform version         # 1.5 or newer
ssh-keygen -t ed25519 -f ~/.ssh/clearpay-deployer -N ""

aws sts get-caller-identity prints your AWS account. If it errors, run aws configure with an access key from an IAM user that can create EC2, RDS, VPCs, and CloudWatch alarms. An admin user on a throwaway account also works.

From the repository root:

cd infra/aws/terraform
cp terraform.tfvars.example terraform.tfvars

Open terraform.tfvars and change two values:

• ssh_allowed_cidr: your public IP address followed by /32. Find your IP with curl ifconfig.me.

• ssh_public_key_path: the path to your key's .pub file.

Then:

terraform init
terraform plan
terraform apply

Read the plan before you type yes. This first run should only create things: EC2, RDS, and security groups. If the plan says it will destroy anything, stop and don't apply.

This is what Terraform builds. The server is empty until Chapter 5:

The diagram shows the AWS infrastructure Terraform creates: an EC2 Ubuntu server with an Elastic IP, security groups controlling SSH and HTTPS access, an RDS PostgreSQL database accessible only from the API server, and a CloudWatch status check. Both resources sit inside the default VPC. The EC2 server is still empty. Chapter 5 deploys ClearPay to it.

A note on cost: the Elastic IP costs about $0.12 per day even while the server runs, because AWS bills every public IPv4 address (AWS public IPv4 charge).

Check

terraform output instance_public_ip

This prints a public IPv4 address. That is how you reach the server. You do not need a domain, so Route 53 stays off (manage_dns = false).

If it Fails:

• Error: InvalidKeyPair.Duplicate: a key pair named clearpay-deployer already exists in this AWS region from an earlier attempt. Delete it in the AWS console, or rename it in infra/aws/terraform/ec2.tf.

• Error: UnauthorizedOperation: your AWS user lacks permission for one resource type. The error names the operation. Add that one permission instead of reaching for full admin.

Decision

The infrastructure exists. A server with nothing running on it isn't done yet. Continue to Chapter 5.

Done When:

• terraform apply finished with no errors

• terraform output instance_public_ip prints a real IP

Chapter 5. Deploy ClearPay to the AWS Server

Tuesday, 1:40 p.m.

Terraform apply finished before lunch. AWS now shows a server and a database, but nothing is serving customers yet.

Terraform built an empty server and a database. Nothing serves customers yet.

You open a terminal to start the deploy. Daniel pulls his chair over and sits down next to you.

Daniel: "Terraform finishing only tells us the server exists. Let's get the app on it and send a real request."

Why This Matters

An empty server isn't a migration source. Chapter 6 asks whether a real customer request works on AWS. That question has no answer until the login page and the API are up.

Steps

Start from the repository root. If you're still in infra/aws/terraform, run cd ../../.. first.

Start by installing Node, nginx, and certbot on the server:

cd infra/aws/deploy
INSTANCE_IP=$(terraform -chdir=../terraform output -raw instance_public_ip)
echo "$INSTANCE_IP"
# Write that IP down. A new terminal will not have INSTANCE_IP.

ssh -o StrictHostKeyChecking=accept-new \
  -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP 'bash -s' < bootstrap.sh

You should see == ClearPay AWS bootstrap ==, then a few minutes of apt-get output. bootstrap.sh is short. Read it.

The step worked when the output ends with these lines:

== Bootstrap complete ==
Node: v20.20.2
Self-signed TLS is in /etc/letsencrypt/live/clearpay, use curl -k.

Your Node version may differ. Lines that start with debconf: unable to initialize frontend are harmless. They appear because the SSH session has no interactive screen. The rows of + and * come from OpenSSL creating the self-signed certificate.

If you see no output at all, SSH didn't connect. Your home IP may have changed since Chapter 4, so the security group still allows the old one. Check your IP with curl -s ifconfig.me; echo, put it plus /32 in terraform.tfvars as ssh_allowed_cidr, run terraform apply in infra/aws/terraform, and retry.

Next, write the .env file the API needs and deploy the app. The database URL comes from Terraform. The JWT secret signs login tokens and is generated fresh:

DATABASE_URL=$(terraform -chdir=../terraform output -raw database_url)

cat > /tmp/clearpay.env <<EOF
DATABASE_URL=$DATABASE_URL
JWT_SECRET=$(openssl rand -hex 32)
PLATFORM=aws
APP_VERSION=1.0.0
GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo local)
ROLLBACK_ACTIVE=false
PORT=4000
EOF

./deploy.sh "$INSTANCE_IP" ~/.ssh/clearpay-deployer /tmp/clearpay.env
rm /tmp/clearpay.env

deploy.sh builds the React UI and copies it to /opt/clearpay-ui. It copies the API to /opt/clearpay-api, installs its packages, writes .env, installs the systemd service and the nginx config from infra/aws/deploy/, and restarts both.

Now load the database schema and seed Alice and Bob. These commands run on the server, not your laptop. RDS has no public endpoint yet (publicly_accessible = false in infra/aws/terraform/rds.tf), so your laptop can't reach it.

Your laptop can't reach the database directly. It uses the EC2 server as a middleman:

The diagram above shows why your laptop can't connect directly to the private RDS database. SSH access to the EC2 server is allowed, and the server is allowed to connect to RDS within the AWS network. Direct access from your laptop to the database is blocked. You use the EC2 server as the entry point for database access.

Each command below starts with ssh ... ubuntu@$INSTANCE_IP "...". That means: log in to the EC2 server and run the part in quotes there. So psql and the seed run on EC2, and EC2 talks to RDS for you. You see the result later in the browser: you log in as Alice, and EC2 reads her account from RDS.

scp -i ~/.ssh/clearpay-deployer ../../../db/schema.sql \
  ubuntu@$INSTANCE_IP:/tmp/schema.sql

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo apt-get install -y postgresql-client -qq && psql '$DATABASE_URL' -f /tmp/schema.sql"

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "cd /opt/clearpay-api && sudo -u clearpay npm run seed"

The seed is the same script from Chapter 1. This time it writes to RDS.

Check

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP 'sudo systemctl is-active clearpay-api'

Expected output: active.

Then from your laptop:

curl -k https://$INSTANCE_IP/health

Look for "status":"ok" and "platform":"aws". The -k flag tells curl to accept the self-signed certificate that bootstrap.sh created.

Open https://<your instance IP> in a browser. Click through the "Not Secure" warning. That warning comes from the self-signed certificate. You should see the same ClearPay login as on your laptop. Sign in as Alice with password123. The dashboard shows her balance and served by aws/1.0.0.

Daniel: "The health check only tells us the app is running. Before we write down a baseline, I want to see a transfer go through on this IP."

Send $20 to Bob in the browser.

If it Fails:

• systemctl is-active returns failed: SSH in and run sudo journalctl -u clearpay-api -n 50. The usual cause is a DATABASE_URL that doesn't match RDS. Compare it with terraform output database_url, character by character.

• seed_failed with SELF_SIGNED_CERT_IN_CHAIN: psql already worked, so the database is fine. Newer versions of the Node pg library treat sslmode=require as full certificate checking. Copy the current api/src/pgConfig.js, api/src/db.js, and api/db/seed.js onto the server, restart clearpay-api, and run only the seed command again. Don't run schema.sql again.

• nginx returns 502 Bad Gateway: nginx is running but the API behind it is not. Check the systemd status above first.

• The browser shows Cannot GET /: nginx still has an old API-only config. Run ./deploy.sh "$INSTANCE_IP" ~/.ssh/clearpay-deployer with no third argument. That keeps the .env already on the server. Don't seed again.

• invalid credentials for Alice: the browser probably filled in a saved password. Clear the box, type password123, and try again.

Done When:

• systemctl is-active clearpay-api prints active

• curl -k https://$INSTANCE_IP/health returns "status":"ok"

• You sent $20 in the browser on the AWS address

Chapter 6. Record the AWS Baseline

Wednesday, 10:00 a.m.

You message Maya: "health is ok, and Alice signed in on the Elastic IP." Her reply comes back in under a minute.

Maya: "Good. Now run the five checks against that IP. Send me the terminal output, not a screenshot."

A second message follows: "Then load-test it and send me the p95. I can't compare GKE against 'it works'."

Why This Matters

This chapter gives you two things. You'll use both for the rest of the book.

1. A pass from the promotion gate. The promotion gate is a short script. It tests five things every payment app must do: the app is running, it can reach the database, a user can log in, a repeated payment charges only once, and the balance drops by the right amount. If all five pass, the app works.

In Chapter 15 you run the same script on GKE. If GKE passes the same test AWS passed, GKE does the job, too.

2. A scorecard. The load test sends many requests at the same time, like many customers using the app at once. Then it writes down how fast the app answered and how many requests failed. That's how fast AWS is today. Later you run the same test on GKE and compare the two.

Think of timing a runner before they change shoes. Without the first time, the second time means nothing. This chapter records the first time.

The scripts in this chapter use Node on your laptop to read JSON. Install Node.js 20 or newer now if you haven't already.

Steps

Start from the repository root. If you're still in infra/aws/deploy, run cd ../../.. first.

Run the promotion gate against AWS:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)

./scripts/promotion-gate.sh https://$INSTANCE_IP \
  alice@clearpay.dev password123 bob@clearpay.dev

Use https://. With http://, nginx redirects and the script looks like it hung.

The gate checks five things, in order:

1. The process is up (/health).

2. It can reach the database (/health/ready).

3. Alice can log in.

4. A replayed payment key returns replayed=true.

5. Alice's balance drops once.

Expected output:

PASS: health endpoint returns ok
PASS: readiness endpoint confirms DB connectivity
PASS: login succeeds and returns a token
PASS: replaying an idempotency key returns replayed=true
PASS: balance debited exactly once (100 cents), not twice

5 passed, 0 failed

GATE PASSED: safe to promote to production.

The last line talks about production. There's no production to promote to yet. It's the same script you run on GKE later, where that line means what it says.

Now load-test AWS. The script logs in and hits /api/accounts/me and /api/transactions, which both query Postgres. Don't load-test /health. The API answers /health by itself, straight away. It never asks the database anything. A real customer request has to ask the database and wait for the answer, and that wait is most of the time:

/health                  laptop → API → reply                    (fast, no database)
/api/accounts/me         laptop → API → database → API → reply   (the real work)

Load-testing /health times only the fast half. Think of asking a shop worker "are you open?" versus "do you have this shoe in size 42?" Only the second question makes them walk to the storeroom.

./scripts/load-test.sh https://$INSTANCE_IP \
  alice@clearpay.dev password123 10 30

The last two numbers are concurrency (10 requests at once) and duration (30 seconds).

Check

The script prints a scorecard. This is the one this lab printed:

== DB load test: https://184.194.120.157  concurrency=10  duration=30s ==
Endpoints: GET /api/accounts/me  GET /api/transactions


ok=372  failed=0

Migration Scorecard, this run

p50 latency:          673 ms
p95 latency:          745 ms
p99 latency:          1326 ms
Throughput:           12.4 req/s
4xx rate:             0.0 %
5xx rate:             0.0 %
Transaction success:  100.0 %

Your scorecard will differ. Your IP address, request counts, and milliseconds depend on where you are. The load test runs on your laptop, so every request travels from your home to AWS in us-east-1 and back.

A reader close to that region sees much lower numbers than a reader on another continent. Neither is wrong. What matters is that you compare GKE against your own scorecard, measured from the same place.

The run passes when you see failed=0, 5xx rate at 0.0 %, and Transaction success at 100.0 %.

• p95 latency means 95 percent of requests finished faster than this number. It shows what slow customers feel, which an average hides.

• 4xx means the client sent a bad request, such as a missing token.

• 5xx means the server failed.

You copy the whole scorecard into a notes file as printed, and send the same block to Maya. Her answer comes back fast: "Don't round it to look tidier. I will compare GKE to this page."

Set Your Limits Now

Write these two lines under the scorecard, with numbers:

Outage:   5xx rate above 1 % for more than 2 minutes in a row
Failure:  p95 more than 3 times your AWS p95 for more than 5 minutes in a row

The failure limit uses your own AWS p95, because every reader's baseline is different. With this lab's 745 ms, the limit is about 2,235 ms. These are example limits. Pick your own, but pick them now, before GKE exists. A limit you set after you see the GKE numbers is a limit you bent to fit them.

Start the cost ledger with one line:

Cost ledger, entry 1:  AWS about $1.25/day (infra/aws/COST.md)

If it Fails:

Stay here until all five checks pass. A problem you carry into the migration comes back later with two clouds involved, and is much harder to find.

• A check fails on readiness: the API can't reach RDS. Go back to Chapter 5's "If it fails" list.

• node: command not found: install Node.js on your laptop.

Decision

The gate passed. This EC2 server is the baseline. Every GKE number in this book gets compared against this scorecard.

A minute later Maya adds one more line: "Good. Those are my numbers now."

Done When:

• The gate printed 5 passed, 0 failed against the AWS IP

• The load test printed failed=0, and the scorecard is in your notes

• Your outage and failure limits are written down with numbers

Part 2. How to Plan the Move

Tell Maya the gate passed and send her the scorecard.

She writes back: "The next job is GKE. I need the API running there, and I need customers to keep sending money the whole time you move it. Leave Postgres on AWS. The database isn't in this assignment."

No cloud gets touched in this part. You read, map, and decide.

Chapter 7. Read the Migration Assignment

Wednesday, 4:00 p.m.

Maya forwards the assignment with one line on top: "Read all of it before you open Terraform."

The Assignment: Migrate the ClearPay API from AWS EC2 to GKE.

ClearPay runs on one EC2 server. It works, but it has no autoscaling, no staged rollout, and a deploy process that lives in one person's head. You ran that process by hand in Chapter 5, so you know what it lacks.

The business wants the API on Kubernetes: a staging environment, automated deploys, and room to grow. Again, the database isn't part of this assignment.

Current Architecture

ClearPay lives in one AWS account, in the default VPC from Chapter 4: one EC2 server and one RDS database.

What Can't Change

• The database. Moving Postgres is a separate, later project.

• The API contract. The same paths return the same JSON: /health, /api/auth/login, /api/transactions. A browser or script that works on AWS must work on GKE with no changes.

• Customers see no lasting outage at any point.

Acceptance Criteria

Done means this, in order:

1. GKE serves all production traffic, and it stays that way. Errors and latency stay inside the limits you wrote in Chapter 6.

2. While AWS still exists, rollback to AWS still works.

3. Then you retire AWS, after GKE has proven itself.

What moves and what stays:

The diagram shows the migration from AWS to GCP. The ClearPay UI and API move from the EC2 server to Google Kubernetes Engine (GKE), where staging and production run. PostgreSQL remains on Amazon RDS in AWS, so both environments use the same database during the migration.

The whole book follows one path:

AWS only  →  AWS and GKE together  →  GKE only, then EC2 gone

The middle step is where the work happens. Both clouds run the same app, both talk to the same database, and traffic moves between them in small steps.

How Traffic Will Move

Today, customers type the Elastic IP and every request goes to AWS.

1. You put a Google load balancer in front, and customers switch to its address once. They keep using that address for the rest of the migration.

2. The load balancer then sends some requests to AWS and the rest to GKE. It starts with 100% AWS and 0% GKE.

3. You raise the GKE share by hand with scripts/shift-traffic.sh, a few steps at a time, checking numbers after each step.

4. If something goes wrong on GKE, you send 100% of the traffic back to AWS with shift-traffic.sh 100 0. The rollback takes seconds.

The diagram above shows how customers reach ClearPay through one public address while the UI and API run on both AWS and GKE during migration. Both environments connect to the same PostgreSQL database on Amazon RDS in AWS. Keeping the database in one place avoids migrating payment data alongside the application.

Why not use DNS to split traffic? DNS is the lookup that turns a name into an IP address. Laptops and internet providers cache the answer.

After you change a DNS record, some customers keep going to the old place for minutes or hours. A rollback that takes an hour isn't a rollback. The load balancer changes the split in seconds.

Rollback Requirement

At every stage of the shift, going back to 100 percent AWS must be fast and safe. Not a redeploy. Not a DNS change that takes an hour to spread. Chapter 20 tests exactly that, before the first shift.

What "Zero Downtime" Means Here

It doesn't mean zero failed requests, ever. No real system can promise that.

It means no customer sees a lasting outage. A few failed requests are allowed if they stay under the limits you wrote in Chapter 6.

Why the API Moves First

If the API and the database move together and something breaks, the fault could be in either one, or in how they interact. Moving only the API keeps the question small: can this API run on GKE without hurting customers?

That's an answerable question. "Can you move everything at once safely?" is much harder to answer, and much worse to debug live.

Why the Database Stays on AWS

Chapter 13 shows what connecting across clouds involves. For now: PostgreSQL doesn't move in this project.

A future, separate project handles that, most likely with Google's Database Migration Service, continuous replication, and its own cutover. This project ends with the API on GKE and the database still on AWS. That's the plan, not an unfinished loose end.

Why Both Clouds Must Run at the Same Time

The rollback plan depends on it. You send a little traffic to GKE and keep AWS ready. If GKE fails, AWS takes everything back. That only works while AWS still exists.

Set a Migration Budget Before Touching GCP

Do this now, before Chapter 9. Write four numbers in your notes. They decide how much this move may cost you.

Plan your spending the way you plan money for a trip:

1. What AWS costs today. About $1.25 a day, from infra/aws/COST.md.

2. What GCP will cost per day. You don't know yet. Write "TBD" (to be decided). Chapter 9 works it out. infra/gcp/COST.md puts it at about $1.60 to $5 a day.

3. What the move costs while both clouds run. During the move you pay AWS and GCP at the same time. That is $1.25 plus the GCP daily cost, for every day of the move.

4. Your budget. The most you will spend on the whole move. Pick it now, before you spend anything.

This lab's numbers:

Current AWS cost:   $1.25/day
Expected GCP cost:  about $3/day (Chapter 9 checks this)
Dual-run cost:      $1.25 + $3 = $4.25/day while both run
Migration budget:   5 days × $4.25 = about $21. Round up to $25.

Keep a running total as you go. Chapters 19 to 25 add the real dual-run days. Chapter 28 asks one question: did you stay under your budget? If the move took longer, for example because of a rollback, the total shows exactly what that cost.

Migration phases

Done When:

• You can explain to Maya, in two sentences, what moves, what stays, and how you undo a bad step

Chapter 8. Map What ClearPay Depends On

Thursday, 9:00 a.m.

Before anyone stands up GKE, the team needs a written map of what ClearPay depends on. This system has run by hand since Tuesday. Nothing on paper says what must keep working during the move.

Maya: "Before we stand up another cloud, write down what this system depends on."

Why This Matters

A migration without a map moves the wrong thing, or misses something that breaks the day you switch. Answer each question below from the repository, not from memory.

1. Is the app ready for containers?

Yes. api/Dockerfile has worked since Chapter 1. What changes is its role. On your laptop it was a convenience. On GKE it becomes the thing production runs.

2. Where do the secrets live?

Open infra/aws/deploy/systemd/clearpay-api.service. It loads EnvironmentFile=/opt/clearpay-api/.env. That file isn't in Git. It sits on the EC2 server, where deploy.sh copied it in Chapter 5.

Every secret the API needs is in one plain text file on one server. Note that. Don't open it to print secrets. GKE will read the same values from Google Secret Manager instead.

3. What outside systems does it call?

One: PostgreSQL. The connection stays inside the AWS default VPC, and RDS has no public endpoint. There's no third-party API. Confirm that by reading api/src/.

4. Does the server remember users between requests?

No. Open api/src/index.js and api/src/routes/transactions.js. Each request carries a JWT (a signed login token), and all account data lives in Postgres. The server holds nothing in memory.

That's why AWS and GKE can split requests later. A user can log in on AWS and send money on GKE, because both clouds check the same token secret and read the same database.

5. Can a retry charge twice?

No. Chapter 3 proved the idempotency key works. It must keep working when a first try lands on AWS and the retry lands on GKE.

The Map

Write these five answers in your notes:

Container ready:   yes, api/Dockerfile
Secrets:           one .env file on the EC2 server
Outside systems:   PostgreSQL only
Server state:      none, JWT + database
Retry safety:      idempotency key, proven in Chapter 3

What Actually Needs to Move

Only the UI and the API: off the EC2 server, onto GKE. Postgres stays on AWS.

Browsers and scripts still hit the same URLs and get the same JSON back. A retry still must not take money twice. Those rules don't change.

Done when

• All five answers are written down, each pointing at a file

Chapter 9. Understand the GCP Design

Thursday, 2:30 p.m.

The Terraform for Google Cloud is already written. Somebody made two big decisions in it before you joined. Maya wants to know whether you can defend them. She writes in the team channel.

Maya: "Open infra/gcp/COST.md and infra/gcp/terraform/03-gke. Then tell me why this repo uses GKE Standard, not Autopilot. I need a reason I can defend."

Daniel, who sits at the next desk, has infra/gcp/terraform/03-gke/main.tf open. He points at a block named primary. It's the node pool: the group of virtual machines GKE runs the app on. This one holds one to three e2-medium machines.

Daniel: "If we keep these machines, how many copies of the API can run on them at once?"

This chapter doesn't run Terraform. The files are already written. You read the decisions in them.

Standard or Autopilot

GKE is Google's managed Kubernetes. It runs in two modes:

• Standard: you choose the virtual machines, grouped in a node pool. Google bills those machines whether the app is busy or idle.

• Autopilot: Google hides the machines. You say how much CPU and memory each container needs, and Google bills that.

This repo uses Standard. The reason isn't cost. It keeps node pools, machine sizes, and autoscaling visible, and those are what this book teaches. Autopilot would hide them.

Work out both costs yourself. Standard bills the machines. One e2-medium running all day is about $1.44 in infra/gcp/COST.md, plus Cloud NAT.

Autopilot bills the CPU and memory each pod reserves. k8s/base/deployment.yaml reserves 250m CPU (a quarter of one CPU core) and 256Mi of memory per API pod. Using the Autopilot rates in infra/gcp/COST.md, three API pods cost roughly:

0.25 vCPU × 3 × $0.0445/hour × 24  ≈  $0.80 / day
0.25 GiB  × 3 × $0.0049/hour × 24  ≈  $0.09 / day

That's about $0.89 a day for three API pods, before the UI pods. Autopilot can cost less than one Standard machine. The lab still uses Standard so you can see the node pool.

Tell Maya both numbers and the reason.

How Many API Processes

AWS runs one API process. On GKE, a HorizontalPodAutoscaler (HPA) adds or removes API pods based on CPU use. Each pod is one API process with its own pool of ten database connections.

This lab's cluster is three small e2-medium machines. The production HPA in k8s/overlays/production/patch-hpa.yaml runs between 1 and 3 API pods, because three machines of that size can't fit more alongside GKE's own system pods.

Answer Daniel: "One to three API pods on GKE, plus one on AWS while both run. Each can open ten connections."

A company with bigger machines would set a higher ceiling, such as 3 to 8 pods. Chapter 22 shows why that ceiling needs checking against the database limit before traffic moves.

A Designed Network

On AWS, this lab used the default VPC that AWS created for you. Nobody designed it.

On GCP, infra/gcp/terraform/02-network builds its own VPC with a named subnet and Cloud NAT. The cluster's machines get private addresses only. They reach the internet, and later the AWS database, through Cloud NAT, which gives all of them one shared public IP.

Region

01-foundation/variables.tf sets the region to us-east4, the Google region closest to AWS's us-east-1. Every database query from GKE crosses between the two clouds, so a short distance keeps that trip short. If your AWS setup isn't in us-east-1, change this before you build anything.

The Map

The left side is what exists. The right side is what Part 3 builds.

The diagram above maps the migration from AWS to GCP. Customers use one public address, managed by the GCP load balancer, which splits traffic between the existing AWS EC2 server and GKE. GKE runs the UI and API in private nodes, with Cloud NAT providing outbound internet access. Both environments connect to the same PostgreSQL database on AWS. Traffic shifts gradually to GKE while the database stays in place.

You'll meet three public IP addresses, in this order:

1. The Elastic IP. That's the EC2 server. Customers use it today.

2. The GKE Gateway IP. The load balancer sends GKE's share of traffic to it. Customers never type it.

3. The load balancer IP from 08-edge. Customers switch to this address once, and the traffic split happens behind it.

If You Need to Pause AWS

Part 3 doesn't need AWS running every minute. AWS is needed again in Chapter 11 (to copy the database URL into Secret Manager) and Chapter 13 (to open the database to GKE).

Don't stop the EC2 server and leave RDS running. RDS is most of the daily AWS bill, a stopped RDS restarts itself after seven days, and the Elastic IP bills either way.

If you continue within a day or two, leave AWS up. If you stop for several days, destroy it:

cd infra/aws/terraform && terraform destroy

To come back, run terraform apply in that folder and repeat Chapter 5. You get a new Elastic IP and a new DATABASE_URL. Your Chapter 6 scorecard still counts as the baseline.

Done When:

• You told Maya the Standard and Autopilot daily numbers and why the lab uses Standard

• You told Daniel how many API processes can run, and how many connections each can open

Part 3. How to Build Google Cloud With Terraform

There's a working AWS app and a scorecard. Google Cloud isn't set up yet. There's no network, no cluster, and no registry until you build them.

The UI and the API will run on that cluster later. Postgres stays on AWS.
So the order is: build an empty place in GCP first, then put the app there, then teach the app how to reach RDS across the internet.

This part is only the empty place. Customers don't move yet.

Chapter 10. Set Up gcloud and Pick Your Project

Friday, 8:45 a.m.

Google Cloud is a blank page. Before you build anything, your laptop needs to know which project to build in, and who you are.

Get gcloud, kubectl, and Terraform pointed at the right Google project, with billing on.

Steps

gcloud is the Google Cloud command-line tool. Install it, then close the terminal and open a new one.

On a Mac:

brew install --cask google-cloud-sdk

On Windows, download the installer from Install the Google Cloud CLI, tick the box that adds gcloud to PATH, and reopen Git Bash. On Linux, follow the Linux tab on the same page. gcloud version should print a version.

If kubectl version --client fails after gcloud works, install kubectl and the plugin it uses to log in to GKE:

gcloud components install kubectl
gcloud components install gke-gcloud-auth-plugin

The plugin must be on your PATH. If a later kubectl get nodes says gke-gcloud-auth-plugin not found, run this and open a new terminal:

echo 'export PATH="$(gcloud info --format="value(installation.sdk_root)")/bin:$PATH"' >> ~/.zshrc

That line is for zsh, the default Mac shell. On Linux with Bash, change ~/.zshrc to ~/.bashrc. On Windows, the installer usually handles PATH. Reopen Git Bash after the install.

Now log in and pick the project:

gcloud auth login --update-adc
gcloud projects list
gcloud config set project YOUR_PROJECT_ID
gcloud billing projects describe YOUR_PROJECT_ID
terraform version
kubectl version --client

--update-adc matters. Plain gcloud auth login only lets gcloud use your account. Terraform is a separate program that looks for a file called Application Default Credentials. The flag writes that file in the same browser login. On the Google page, click Allow and don't close the tab early.

Which ID to Use

gcloud projects list prints two columns that look alike:

• NAME is a label, and many projects share the default label My First Project.

• PROJECT_ID is unique, for example my-first-project-123456.

Copy the PROJECT_ID, never the name. Paste it wherever the commands say YOUR_PROJECT_ID.

If you have several projects, pick one with billing on. Run gcloud billing projects describe with each ID until the output includes billingEnabled: true.

You can also find the ID in the Google Cloud Console: click the project name at the top of the page and read the ID column.

Check

gcloud config get-value project

This prints your Project ID. If it prints [None], run the gcloud config set project command again.

If it Fails:

• Terraform later says could not find default credentials: you logged in without the flag. Run:

gcloud auth application-default login

• That command says cloud-platform scope is required but not consented: the browser closed before you clicked Allow. Run gcloud auth login --update-adc again and click Allow.

Today is only a gcloud login. The name api.clearpay.test shows up later, when Stage 8 builds the load balancer. Skip it until then.

Done When:

• gcloud config get-value project prints your Project ID

• gcloud billing projects describe shows billingEnabled: true

• kubectl version --client prints a version

Message Maya: "gcloud is logged in. The project is set. Billing is on. Ready to start GCP."

Maya: "Build one stage at a time. Tell me when each one is verified. If you run all eight at once and the third one breaks, you won't know which one to fix."

Chapter 11. Build GCP in Eight Terraform Stages

Friday, 10:30 a.m., to Monday of week two

GCP is built in eight Terraform stages. Each stage creates one layer of the destination environment.

Run them in order. After each stage, verify the result before moving to the next.

Build the Google side of the migration: APIs, network, cluster, image registry, identities, secrets, error alerting, and the public load balancer.

Why Eight Stages?

The Terraform lives in eight folders, 01-foundation through 08-edge. Each folder has its own state file and reads only the earlier stages it needs. If 03-gke fails, the problem is in the cluster, not somewhere in one 1,500-line file.

This isn't "run Terraform eight times". The work is to build eight things and prove each one exists before the next one needs it.

State stays local on your laptop, which is fine for one person. A team would move each stage's state to a Google Cloud Storage bucket before a second person runs apply.

The diagram above shows how Terraform builds GCP infrastructure in eight stages. It enables the required APIs, creates the network and private GKE cluster with Cloud NAT, then sets up the image registry, identities, secrets, error monitoring, and public load balancer. Cloud NAT provides outbound connectivity for private nodes, including access to AWS RDS later in the migration.

How Every Stage Runs

Each stage uses the same three commands inside its folder:

terraform init
terraform plan
terraform apply

Read the plan before you apply. Every stage below lists the plan count to expect, for example Plan: 7 to add, 0 to change, 0 to destroy. If the count differs, or the plan wants to destroy something, stop and read it before typing yes.

terraform apply asks Do you want to perform these actions?. Type yes and press Enter.

Folder paths below start from the repository root. If you're already inside infra/gcp/terraform/0X-..., use cd ../0Y-... to move to the next stage.

For the "In the console" checks: pick your project at the top of the page, and set the region to us-east4 when a page has a region menu. An empty list often means the menu is still on us-central1.

Stage 1: Turn on the Google APIs (01-foundation)

Google Cloud services are off by default in a new project. This stage turns on ten of them: GKE, Artifact Registry, Secret Manager, IAM, logging, and others. If you skip it, 03-gke fails later with a permission error that points nowhere useful.

cd infra/gcp/terraform/01-foundation
cp terraform.tfvars.example terraform.tfvars

If terraform.tfvars already exists with your Project ID, skip the copy. A second copy overwrites your edit.

Open terraform.tfvars and replace clearpay-migration-demo with your Project ID. Then run:

terraform init
terraform plan
terraform apply

Expected plan: google_project_service ten times, and no cluster.

Plan: 10 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + project_id = "YOUR_PROJECT_ID"
  + region     = "us-east4"

If the plan prints clearpay-migration-demo, fix terraform.tfvars first. Secret Manager and Service Networking can sit on Still creating for about half a minute during apply.

Check:

gcloud services list --enabled --filter='config.name:container.googleapis.com'

NAME                      TITLE
container.googleapis.com  Kubernetes Engine API

If this prints nothing, the API isn't on for the project gcloud is looking at. Don't go to Stage 2 yet. Check these three causes, in order:

1. gcloud points at a different project. Run gcloud config get-value project. It must print the same Project ID you put in terraform.tfvars. If it doesn't, run gcloud config set project YOUR_PROJECT_ID and check again.

2. The apply didn't finish. Run terraform plan in 01-foundation again. A finished stage prints No changes. If the plan still wants to add resources, run terraform apply and type yes.

3. Google hasn't caught up yet. A newly enabled API can take a minute to show in the list. Wait one minute and run the check again.

Message Maya: "the first stage applied, and Kubernetes Engine API shows as enabled."

Maya: "Good. Network next. Nothing else."

Stage 2: Build the Network (02-network)

A VPC is a private network inside your Google project. Think of it as the neighborhood the cluster will live in, and the subnet as one street in it. This stage creates:

• clearpay-vpc, the private network

• clearpay-gke-subnet, the address range the cluster's machines use, with two extra ranges named pods and services reserved for Stage 3

• clearpay-router and clearpay-nat, the Cloud NAT setup

• clearpay-nat-ip, the one public IP that all cluster machines share when they call out

• two firewall rules so machines inside can talk to each other and Google's health checks can reach them

This folder has no terraform.tfvars. It reads your Project ID from Stage 1's state file.

cd infra/gcp/terraform/02-network

Run:

terraform init
terraform plan
terraform apply

Expected plan:

Plan: 7 to add, 0 to change, 0 to destroy.

Expected end of apply:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

nat_ip_address = "x.x.x.x"
network_name = "clearpay-vpc"
subnet_name = "clearpay-gke-subnet"

Check:

gcloud compute networks describe clearpay-vpc
terraform output nat_ip_address

Look for name: clearpay-vpc, a subnetworks line ending in clearpay-gke-subnet, and autoCreateSubnetworks: false. That last line means this network was designed, not generated.

Write the NAT IP down. Chapter 13 opens the AWS database to exactly this address.

In the console: VPC networks shows clearpay-vpc. Cloud NAT shows clearpay-nat on clearpay-router. IP addresses shows clearpay-nat-ip.

If the plan can't read ../01-foundation/terraform.tfstate, Stage 1 isn't done.

Message Maya: "clearpay-vpc and Cloud NAT exist. The NAT IP is in your notes."

Daniel sees the NAT IP in your notes.

Daniel: "Every call our pods make to the outside world leaves from that one address. That includes calls to the database on AWS. Later we tell the AWS database to accept connections from this address and nothing else."

That happens in Chapter 13. Keep the NAT IP in your notes until then.

Stage 3: Create the GKE Cluster (03-gke)

This stage creates a Standard cluster named clearpay in zone us-east4-a, with a node pool named primary of one to three e2-medium machines. The machines have private IPs only and use Cloud NAT to reach out. The control plane, Google's managed API server for the cluster, has a public endpoint so kubectl on your laptop can reach it.

GKE first creates a default node pool, then Terraform deletes it and creates primary, which you control.

cd infra/gcp/terraform/03-gke

Run:

terraform init
terraform plan
terraform apply

Expected plan: the cluster and the primary node pool.

Plan: 2 to add, 0 to change, 0 to destroy.

The apply sits on Still creating for seven to ten minutes. Don't press Ctrl+C.

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

cluster_name = "clearpay"
get_credentials_command = "gcloud container clusters get-credentials clearpay --zone us-east4-a --project YOUR_PROJECT_ID"

cluster_endpoint prints <sensitive>. You don't need it.

Point kubectl at the new cluster:

$(terraform output -raw get_credentials_command)

A good run prints kubeconfig entry generated for clearpay.

Check:

kubectl get nodes

NAME                                 STATUS   ROLES    AGE   VERSION
gke-clearpay-primary-........        Ready    <none>   2m    v1.xx

Ready means the machine is on and joined the cluster. ClearPay isn't on it yet.

If the apply mentions container.googleapis.com, go back to Stage 1. If it mentions the subnet or the pods or services ranges, go back to Stage 2. Don't edit 03-gke to hide those errors.

If kubectl says gke-gcloud-auth-plugin not found, the cluster is fine. Don't destroy it. Install the plugin and fix PATH for this terminal:

gcloud components install gke-gcloud-auth-plugin
export PATH="$(gcloud info --format='value(installation.sdk_root)')/bin:$PATH"
kubectl get nodes

In the console: Kubernetes clusters shows clearpay running. VM instances shows a gke-clearpay-primary- machine with an internal IP and no external IP.

Message Maya: "the cluster is up. One node is Ready. Nothing of ours runs on it yet."

The apply took nine minutes. You watched every one of them.

Stage 4: Create the Image Registry (04-registry)

A container image is a packaged copy of the app. GKE downloads images from a registry. This stage creates a private Docker registry in Artifact Registry, named clearpay, in us-east4. It stays empty until the pipeline pushes to it in Chapter 14.

cd infra/gcp/terraform/04-registry

Run:

terraform init
terraform plan
terraform apply

The two resources are the registry and a rule that lets the cluster's machines pull images from it.

Plan: 2 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

repository_url = "us-east4-docker.pkg.dev/YOUR_PROJECT_ID/clearpay"

Check:

gcloud artifacts repositories describe clearpay --location=us-east4

Look for format: DOCKER, a name ending in /repositories/clearpay, and Repository Size: 0.000MB. An empty registry is correct today. You'll also see SCANNING_DISABLED. The pipeline scans images with Trivy instead, in Chapter 14.

Message Maya: "the image registry exists. It's empty."

Stage 5: Create Identities with No Key Files (05-identity)

Two things will need Google permissions later: the ClearPay pods on GKE, and GitHub Actions when it pushes images and deploys.

The old way is to download a service account key file. A key file is a password on disk. If it leaks, anyone can use it. This lab never creates one.

Instead, this stage creates three Google service accounts (identities with permissions) and lets named callers borrow them for short periods:

• clearpay-api-staging, borrowed by pods in the staging namespace

• clearpay-api-production, borrowed by pods in the production namespace

• clearpay-ci, borrowed by GitHub Actions through a Workload Identity pool named clearpay-github-pool

Workload Identity is the Google feature that links a Kubernetes service account, or a GitHub repository, to a Google service account without a key.

Think of each service account as a badge. Nothing wears a badge today. Later, Google lets a named pod or a named repository wear one for a short time, and no key file ever exists.

Namespaces are separate areas inside one cluster. Staging and production share the cluster's machines but not their identities, so a staging pod can't read production's secrets. A real company would usually run separate clusters, often in separate projects. This lab uses one cluster to keep the bill small.

cd infra/gcp/terraform/05-identity
cp terraform.tfvars.example terraform.tfvars

The only value in the file is github_repo. If you already have your GitHub copy of this repository, put its owner/name there now, with no .git at the end. If not, leave YOUR_GITHUB_USER/YOUR_REPO. Chapter 14 tells you when to set it and apply this stage again.

Run:

terraform init
terraform plan
terraform apply

Expected plan:

Plan: 10 to add, 0 to change, 0 to destroy.

The ten are the three service accounts, the GitHub pool and its provider, and the permission links between them.

Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

ci_service_account_email = "clearpay-ci@YOUR_PROJECT_ID.iam.gserviceaccount.com"
service_account_emails = {
  "production" = "clearpay-api-production@YOUR_PROJECT_ID.iam.gserviceaccount.com"
  "staging" = "clearpay-api-staging@YOUR_PROJECT_ID.iam.gserviceaccount.com"
}

Check:

gcloud iam service-accounts list --filter='email~clearpay-'

Look for three emails: clearpay-api-staging, clearpay-api-production, and clearpay-ci.

In the console: Service accounts lists the three. Workload Identity Federation shows clearpay-github-pool with provider clearpay-github-provider.

If apply fails with Error 409: Requested entity already exists on the pool, you ran this stage before. Google keeps a deleted pool for 30 days and blocks its name. Restore it with gcloud iam workload-identity-pools undelete clearpay-github-pool --location=global,
then run terraform import google_iam_workload_identity_pool.github projects/YOUR_PROJECT_ID/locations/global/workloadIdentityPools/clearpay-github-pool and apply again.

If the same error then names the provider, restore it with gcloud iam workload-identity-pools providers undelete clearpay-github-provider --workload-identity-pool=clearpay-github-pool --location=global,
then run terraform import google_iam_workload_identity_pool_provider.github projects/YOUR_PROJECT_ID/locations/global/workloadIdentityPools/clearpay-github-pool/providers/clearpay-github-provider and apply again.

Message Maya: "three service accounts exist. Nothing uses them yet."

Maya: "No key files?"

"None."

Stage 6: Create the Secrets (06-secrets)

ClearPay on GKE needs two secrets: the database URL and the JWT signing key. Both already exist on AWS. This stage creates four secrets in Secret Manager, one pair per namespace:

• clearpay-staging-jwt-secret

• clearpay-staging-database-url

• clearpay-production-jwt-secret

• clearpay-production-database-url

The staging service account can read only the staging pair. The production service account can read only the production pair.

Think of each secret as a locked drawer. Each badge from Stage 5 opens only its own drawers.

This lab puts the same values in both pairs. Both environments use the same RDS, and a login on AWS must still work when the next request lands on GKE. A real company would give each environment its own values.

Terraform creates the secrets with a placeholder value, REPLACE_ME. Terraform is allowed to create the secrets but not to invent their values. So the order is: apply first, then copy the real values in by hand.

cd infra/gcp/terraform/06-secrets

Run:

terraform init
terraform plan
terraform apply

Terraform creates fourteen resources:

• four secrets

• four first versions

• four access rules for the API service accounts

• two rules that let the pipeline's clearpay-ci account read the staging pair

Plan: 14 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Now copy the AWS values in. AWS must be running.

Run these from the repository root, the top folder of the project, not from 06-secrets and not from infra/aws/terraform. You're still in 06-secrets, so go up four folders first:

cd ../../../..

ls should now show api, infra, k8s, and scripts.

You don't need to move into the AWS folder. terraform -chdir=infra/aws/terraform reads the AWS values from where you stand. Each command pipes a value straight into Secret Manager, so nothing prints on screen.

The database URL, once for each namespace:

terraform -chdir=infra/aws/terraform output -raw database_url \
  | gcloud secrets versions add clearpay-staging-database-url --data-file=-
terraform -chdir=infra/aws/terraform output -raw database_url \
  | gcloud secrets versions add clearpay-production-database-url --data-file=-

A good run prints Created version [2] and doesn't print the URL.

The JWT secret, read from the .env file on the AWS server:

INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo grep '^JWT_SECRET=' /opt/clearpay-api/.env | cut -d= -f2-" \
  | gcloud secrets versions add clearpay-staging-jwt-secret --data-file=-
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo grep '^JWT_SECRET=' /opt/clearpay-api/.env | cut -d= -f2-" \
  | gcloud secrets versions add clearpay-production-jwt-secret --data-file=-

A later terraform apply in this folder won't put REPLACE_ME back. The Terraform file tells it to leave the value alone after the first apply.

Check:

gcloud secrets list --filter='name~clearpay-'

Then prove no secret still holds the placeholder. This loop prints ok or STILL_PLACEHOLDER and never prints a secret:

for s in clearpay-staging-database-url clearpay-staging-jwt-secret \
         clearpay-production-database-url clearpay-production-jwt-secret; do
  val=$(gcloud secrets versions access latest --secret="$s")
  if [ "$val" = "REPLACE_ME" ]; then echo "STILL_PLACEHOLDER $s"; else echo "ok $s"; fi
done

Look for four ok lines. If one says STILL_PLACEHOLDER, run its copy command again.

In the console: Secret Manager lists the four secrets, each with version 2 as the latest. Don't click "Show secret value" on a shared screen.

Message Maya: "the four secrets hold the AWS values. Nothing was printed on screen or pasted into a chat."

Stage 7: Count API Errors (07-observability)

Every time something happens in the API, it writes a short note called a log line. A login writes one. A payment writes one. When something goes wrong, the note says "level":"error". A failed payment writes this:

{"level":"error","message":"transaction_failed","timestamp":"2026-09-29T10:15:02.000Z","error":"..."}

On GKE, Google Cloud collects every one of these notes. This stage builds two things on top of them:

1. A counter, named clearpay-api-error-count. It goes up by one each time an error note appears.

2. An alarm, named ClearPay API - elevated error rate. It switches on when the counter goes up more than five times in five minutes. One error can be bad luck. Five in five minutes means something is broken.

The alarm has no phone number or email attached yet. When it switches on, the Google Cloud console shows it, but nobody gets a message. You can add an email later in the console.

The counter stays at zero for now, because ClearPay doesn't run on GKE yet. Zero is the correct result today.

cd infra/gcp/terraform/07-observability

Run:

terraform init
terraform plan
terraform apply

Expected plan:

Plan: 2 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

log_metric_name = "clearpay-api-error-count"

If apply stops with Error 404: Cannot find metric(s) that match type, nothing is broken. Google needs a few minutes to register a brand-new counter before an alarm can watch it. The counter was created. Wait five minutes and run terraform apply again. This time the plan shows 1 to add, only the alarm.

Check:

gcloud logging metrics describe clearpay-api-error-count

Look for name: clearpay-api-error-count and a filter that mentions clearpay-api and jsonPayload.level="error".

Message Maya: "the error counter exists and reads zero. No app runs on GKE yet."

Maya: "Last one. This is the address customers will use, so take your time."

Stage 8. Build the Public Load Balancer (08-edge)

Stages 1 to 7 built the house. This stage builds the front door: the address customers keep for the rest of the migration. It's a Google global load balancer with a traffic dial behind it: a percentage to AWS, the rest to GKE. Today it sends 100 percent to AWS.

This stage creates:

• clearpay-api-ip, the public address customers will use

• a self-signed certificate so HTTPS works

• the traffic split, set to AWS 100, GKE 0

• clearpay-gke-edge-ip, a second IP for GKE's side. The load balancer uses it. Customers never do.

The diagram below shows how the GCP load balancer directs customer traffic between AWS and GKE using configurable traffic percentages. Customers connect through one public IP. AWS receives traffic through its Elastic IP, while GKE receives traffic through a separate edge IP that customers never access directly. Both environments connect to the same PostgreSQL database on AWS. The GKE cluster is empty at this stage.

cd infra/gcp/terraform/08-edge
cp terraform.tfvars.example terraform.tfvars

Skip the copy if the file exists. Put your Elastic IP in aws_backend_ip. Get it from a second terminal at the repository root:

terraform -chdir=infra/aws/terraform output -raw instance_public_ip

Keep the weights at 100 and 0:

aws_traffic_weight = 100
gke_traffic_weight = 0

Run:

terraform init
terraform plan
terraform apply

The apply takes about two minutes.

Plan: 17 to add, 0 to change, 0 to destroy.

Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

Outputs:

current_traffic_split = "AWS: 100% / GKE: 0%"
static_ip_address = "x.x.x.x"
gke_edge_ip_address = "y.y.y.y"
hosts_file_line = "x.x.x.x api.clearpay.test"

Write both IPs down. static_ip_address is the customer address. gke_edge_ip_address is not.

Give the Address a Name

You don't own a domain, so you point the name api.clearpay.test at the load balancer in your computer's hosts file.

You're already in 08-edge after the apply, so skip the cd line below. Run it only if you opened a new terminal at the repository root:

cd infra/gcp/terraform/08-edge
LINE=$(terraform output -raw hosts_file_line)
echo "LINE is: [$LINE]"

If LINE is empty, stop. If it looks like x.x.x.x api.clearpay.test, add it to the hosts file.

On a Mac:

echo "$LINE" | sudo tee -a /etc/hosts
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

On Linux, run only the first line. On Windows, add the same IP and api.clearpay.test to C:\Windows\System32\drivers\etc\hosts in an editor opened as Administrator.

Don't run sudo cd. cd is built into the shell, so sudo can't change your folder.

Confirm the line is there:

grep api.clearpay.test /etc/hosts

The name ends in .test on purpose. .test is reserved for testing, so Chrome lets you click through a self-signed certificate warning. Don't change lab_hostname to a .dev name. Chrome forces strict HTTPS on every .dev site and hides the button to continue.

Check

curl -k https://api.clearpay.test/health

Expected output:

{"status":"ok","served_by":{"platform":"aws","version":"1.0.0","instance":"ip-172-31-64-73","commit":"504e0cd","rollback_active":false}}

Your instance and commit values will differ. "platform":"aws" is the check. The request went to the new Google address, and the load balancer passed it to AWS, because the weights are 100 and 0.

A new Google load balancer can take a few minutes to start. If you don't see the JSON yet, you may see one of these first:

• Connection reset. The load balancer is still starting. Wait a minute and run the same curl again.

• A Google page titled 404 Not Found. The load balancer answered but hasn't reached AWS yet. Confirm AWS itself is up:

curl -k https://YOUR_ELASTIC_IP/health

If AWS answers, wait a few minutes and try api.clearpay.test again. Stage 8 is done even if this is still slow. Chapter 20 checks it again.

When the JSON appears, open https://api.clearpay.test in a browser. In Chrome, click Advanced, then "Proceed to api.clearpay.test (unsafe)". You see the ClearPay login, served by AWS through the new address. If the browser says NXDOMAIN, the hosts line is missing or the cache wasn't flushed.

In the console: IP addresses shows clearpay-api-ip and clearpay-gke-edge-ip. Load balancing shows clearpay-migration-forwarding-rule with your Elastic IP as a backend.

Done When:

• Stage 1: container.googleapis.com is enabled

• Stage 2: terraform output nat_ip_address prints an IP, and you wrote it down

• Stage 3: kubectl get nodes shows at least one Ready node

• Stage 4: the clearpay registry exists

• Stage 5: three clearpay- service accounts exist

• Stage 6: the loop prints four ok lines

• Stage 7: the error metric exists

• Stage 8: /etc/hosts has api.clearpay.test, and weights are 100 / 0

Tell Maya: "all eight stages are verified. Weights are 100 and 0. Nothing runs on GKE yet."

Daniel: "The cluster is empty. Next we put ClearPay on it."

Part 4. How to Get ClearPay Onto GKE Staging

Maya stops by to look at the new address in your browser. She reads the dashboard line and smiles.

Maya: "Nice work on the front door. But look, it still says aws. Customers can reach the new address, and every request still lands on the old server. The cluster on GKE is empty, so nobody can send $20 there yet."

She points at your terminal.

Maya: "Next step is to get ClearPay onto that cluster. One thing I care about: let the pipeline do it, not a person typing commands. A pipeline does it the same way every time, and we can see exactly what it did."

https://api.clearpay.test already works in your browser, so it's easy to think ClearPay is on GKE. It's not. That address is Google's new front door, the load balancer from Stage 8. With the weights at 100 and 0, the front door sends every request to AWS. The served by aws/1.0.0 line on the dashboard proves it. The GKE cluster itself is still empty.

Browser → api.clearpay.test (Google load balancer) → 100% → AWS EC2 → RDS
                                                  →   0% → GKE (empty)

In this part the pipeline puts ClearPay on GKE staging, and the promotion gate from Chapter 6 decides whether staging is good enough. Staging is a private copy of the app inside the cluster. No customer reaches it.

Chapter 12. Check the API Image

Week two, Tuesday, 9:00 a.m.

The cluster is built and empty. Before anything goes on it, Daniel pulls the API Dockerfile up on your screen.

Daniel: "We're about to put this on a cluster. Show me the container doesn't run as root."

Prove the API image is small, holds no secrets, and does not run as root.

Why This Matters

A process that runs as root inside a container has more power than it needs if an attacker breaks in. Production images should run as a regular user.

Steps

Open api/Dockerfile. It has two stages:

1. The first stage, FROM node:22-slim AS deps, installs only production packages with npm install --omit=dev.

2. The second stage starts again from node:22-slim. It applies Debian security updates and deletes npm and yarn, because the running API only needs node src/index.js. It creates appuser with user ID 1001 and copies in node_modules, src, and db. The last lines switch to USER appuser.

Notice what the image doesn't contain: DATABASE_URL and JWT_SECRET. Those arrive at run time from Secret Manager.

Docker Desktop must be running. From the repository root:

docker build -t clearpay-api:test ./api
docker run --rm clearpay-api:test whoami

Check

The second command prints:

appuser

If it Fails:

• It prints root: the USER appuser line is missing. Fix api/Dockerfile and run both commands again.

• Docker can't connect: start Docker Desktop and retry. The cluster is fine.

Done When:

• whoami prints appuser

Chapter 13. Prepare the Cluster for ClearPay

Tuesday, 11:00 a.m.

The image check is done. You open GitHub, ready to run the pipeline. Daniel looks at your screen and shakes his head.

Daniel: "The first deploy will fail twice, for reasons that have nothing to do with the pipeline. Fix those first."

Before the first deploy, the cluster needs two things it doesn't have yet:

1. A way for pods to read secrets from Secret Manager.

2. A network path to the database on AWS.

Without the first, Kubernetes rejects the staging files. Without the second, the pods start but never become ready.

Part A. Turn on the Secret Manager Add-ons

Daniel: "The pods should read the Secret Manager values from Stage 6. If they don't, write down why they still use a file."

The staging and production files in k8s/overlays/ read the database URL and JWT secret from Secret Manager. They use two GKE add-ons:

• The Secret Manager CSI driver mounts secrets as files inside the pod.

• Secret Sync copies them into a normal Kubernetes Secret named clearpay-api-secrets. The Deployment loads that Secret into the API's environment variables.

  

The pod proves who it is through Workload Identity, using the service account from Stage 5. No key file exists anywhere. The API code doesn't change. It still reads plain environment variables, as it does on AWS.

Turn on both add-ons. gcloud needs two separate commands, and each takes a few minutes:

gcloud container clusters update clearpay --zone us-east4-a \
  --enable-secret-manager
gcloud container clusters update clearpay --zone us-east4-a \
  --enable-secret-sync

These run here, not in 03-gke, so the cluster stage stays small and you see the moment secret access is added.

Leave PROJECT_ID as it is in k8s/overlays/staging/secretproviderclass.yaml and patch-serviceaccount.yaml. The pipeline replaces it with your real Project ID at deploy time. Don't type your ID into those files.

Part B. Open the AWS Database to GKE

Don't describe this as "connecting GKE to the database". That phrasing suggests the database moves too, and it doesn't. This step lets GKE reach and use the real, unmoved AWS database, safely.

The cluster's machines have no public IP. Their outbound traffic leaves through Cloud NAT with one public address, the NAT IP from Stage 2. RDS will accept connections from exactly that address on port 5432, and the connection string already requires TLS (sslmode=require).

GKE Pod  (clearpay-api)
   │
   ▼
Private node  (no public IP)
   │
   ▼
Cloud NAT  (clearpay-nat-ip)
   │
   │  public egress, one /32
   ▼
AWS security group  (additional_db_ingress_cidrs)
   │  port 5432 + SSL
   ▼
RDS PostgreSQL 16

This is one of three ways to connect two clouds:

To compare them for your own system:

(expected monthly GB) × (price per GB on that path) + fixed monthly cost of that option

ClearPay sends small queries, so a VPN would cost more each month than the data is worth. This lab uses the allowlist. A busier system might choose differently.

Get the NAT IP:

cd infra/gcp/terraform/02-network
terraform output nat_ip_address

Put it in infra/aws/terraform/terraform.tfvars:

db_publicly_accessible      = true
additional_db_ingress_cidrs = ["<the NAT IP>/32"]

Then apply the AWS stack:

cd infra/aws/terraform
terraform plan
terraform apply

The plan adds a new security group rule for the NAT IP and makes RDS publicly reachable. It must not remove the rule that lets the EC2 server reach RDS. Those are two separate rules in Terraform: aws_security_group_rule.db_from_nat for GKE and aws_security_group_rule.db_from_api for EC2.

Check

AWS must still reach its database. From the repository root:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
curl -k https://$INSTANCE_IP/health/ready

{"status":"ready", ... "platform":"aws" ...}

If this prints not_ready, or the AWS login page says internal error, the database stopped accepting the EC2 server. Stop here and read the Terraform plan again. Fix AWS before you touch GKE, because AWS still serves every customer.

Check Before You Go On

Chapter 14 pushes code and starts the pipeline. If any step above was skipped, the pipeline fails two chapters later with an error that doesn't say which step you missed. Run this check now instead. It prints ok or MISSING for each thing the pipeline needs. Run it from the repository root:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
NAT_IP=$(terraform -chdir=infra/gcp/terraform/02-network output -raw nat_ip_address)
DB_HOST=$(terraform -chdir=infra/aws/terraform output -raw db_endpoint | cut -d: -f1)

grep -q '^db_publicly_accessible *= *true' infra/aws/terraform/terraform.tfvars \
  && echo "ok   RDS public setting" \
  || echo "MISSING db_publicly_accessible = true in infra/aws/terraform/terraform.tfvars"
grep -q "$NAT_IP/32" infra/aws/terraform/terraform.tfvars \
  && echo "ok   NAT IP in the allowlist" \
  || echo "MISSING $NAT_IP/32 in additional_db_ingress_cidrs"
kubectl get csidriver secrets-store-gke.csi.k8s.io >/dev/null 2>&1 \
  && echo "ok   Secret Manager driver" \
  || echo "MISSING Secret Manager driver: run the two gcloud commands in Part A"
kubectl get crd secretsyncs.secret-sync.gke.io >/dev/null 2>&1 \
  && echo "ok   Secret Sync" \
  || echo "MISSING Secret Sync: run the two gcloud commands in Part A"
curl -sk https://$INSTANCE_IP/health/ready | grep -q '"status":"ready"' \
  && echo "ok   AWS still reaches its database" \
  || echo "MISSING AWS cannot reach its database: read the Terraform plan again"

The last check tests the real path from GKE to the database. It starts a small throwaway pod, asks the database port if it's accepting connections, and deletes the pod. The first run takes about a minute, because Kubernetes downloads the image:

kubectl run dbcheck --rm -i --restart=Never --image=postgres:16 -- \
  pg_isready -h "$DB_HOST" -p 5432 -t 10 \
  && echo "ok   GKE reaches the database" \
  || echo "MISSING GKE cannot reach the database"

You need six ok lines and no MISSING. Each MISSING line names the fix. Apply it, then run the check again. Don't start Chapter 14 until all six print ok.

If the last line says MISSING but the first two say ok, RDS may still be applying the change. Wait two minutes and run that command again.

Done When:

• Both gcloud container clusters update commands finished

• RDS allows the NAT IP, and curl -k https://$INSTANCE_IP/health/ready still prints ready

• The check above printed six ok lines and no MISSING

Chapter 14. Build the CI/CD Pipeline

Tuesday, 3:00 p.m.

The cluster is ready for its first deploy. Maya reads the team channel and adds one rule.

Maya: "From now on, let's not run kubectl apply on staging by hand. The pipeline should build the image, scan it, push it, and deploy it, the same way every time."

The workflow file is already written: .github/workflows/deploy.yml. Your job is to understand it, connect it to Google Cloud, and watch it deploy staging.

Why a Pipeline?

When a person deploys by hand, each deploy is slightly different. Someone skips the scan because it's late. Someone deploys from a laptop with uncommitted changes. When something breaks, nobody can say what ran.

A pipeline removes those differences. Every image comes from a commit, passes the same scan, and reaches the cluster through the same steps. The promotion gate in Chapter 15 means something because it checks an image built that way.

How the Pipeline Works

The workflow has two jobs:

Open .github/workflows/deploy.yml in your repository and keep it open alongside this diagram. The diagram maps the workflow's two jobs and their sequence. The file contains the actual GitHub Actions configuration. Follow both together as you read through the triggers, permissions, image build and scan, staging deployment, and production promotion.

Read the file top to bottom with this map.

Triggers: on: push: branches: [main] runs Job 1 on every push to main. workflow_dispatch adds a "Run workflow" button in GitHub with one checkbox, promote_to_production. Job 1 runs only when that box is off (if: ${{ !inputs.promote_to_production }}). Job 2 runs only when it's on. A push can never reach production by itself.

Permissions: id-token: write lets the job ask GitHub for a short-lived OIDC token, a signed statement that says "this run belongs to repository X". Google checks that statement against the pool from Stage 5 and hands back temporary credentials for clearpay-ci. No key file sits in GitHub secrets, so there's no key to leak.

Image tag: The tag is the short commit SHA from git rev-parse --short HEAD. Each image points to exactly one commit. The pipeline also pushes a moving tag, staging-latest, but deploys use the SHA.

Scan: Trivy is an open-source scanner that checks the packages inside an image against lists of known vulnerabilities. Each known vulnerability has an ID like CVE-2026-29786. The workflow sets:

• severity: CRITICAL,HIGH, so only serious findings count

• ignore-unfixed: true, so a finding with no patch available yet doesn't block the build

• exit-code: "1", so any remaining finding fails the job

Together they mean: fail if a serious vulnerability has a fix and this image doesn't have it. Nothing gets pushed if the scan fails.

Push: The job tags each image with the full registry path, us-east4-docker.pkg.dev/<project>/clearpay/api:<sha>, and pushes it. clearpay-ci has artifactregistry.writer from Stage 5, so the push works.

Deploy to staging: The step works inside k8s/overlays/staging:

1. It writes the two staging secrets from Secret Manager into secrets.staging.env on the runner. umask 077 makes the file readable only by the runner's user. The file is gitignored and dies with the runner.

2. sed replaces the PROJECT_ID placeholder in patch-serviceaccount.yaml and secretproviderclass.yaml with your real Project ID.

3. kustomize edit set image points the overlay at the image with this commit's SHA.

4. kubectl apply -k . sends the overlay to the cluster.

5. kubectl rollout status waits up to 180 seconds for the API and UI pods to become ready. If they never do, the job fails.

clearpay-ci has container.developer, which lets it deploy workloads but not change the cluster itself.

Promote to production: Job 2 is the only way the pipeline touches production:

• environment: production pauses the job until a named reviewer clicks Approve in GitHub.

• It doesn't build anything. It reads the image string that staging is running right now and deploys that exact image to production.

That rule is called "build once, promote the same artifact". The image that passed the Chapter 15 gate is byte for byte the image customers get. A rebuild could pull a newer package and ship something nobody tested.

Kustomize, used in both deploy steps, is the tool behind kubectl apply -k. It takes the shared files in k8s/base/ and applies small environment-specific changes from k8s/overlays/staging/ or k8s/overlays/production/, called overlays. Staging and production share one base and differ only where they must.

Steps

First, put this repository on GitHub under your own account, with main as the default branch.

Then tell Google which repository may use clearpay-ci. Open infra/gcp/terraform/05-identity/terraform.tfvars and set github_repo to your repository's owner/name, for example Osomudeya/migrate-api-aws-gke-devops. Don't paste the full https://github.com/... address, and no .git at the end. Apply Stage 5 again:

cd infra/gcp/terraform/05-identity
terraform plan
terraform apply

The plan changes the pool's attribute condition and the clearpay-ci link. It doesn't touch the cluster. If you already set the real name in Chapter 11, the plan shows no changes.

On GitHub, open your repository and click Settings. In the left menu, click Secrets and variables, then Actions. Open the Variables tab and click New repository variable for each of these three. Use the Repository section, not Environment.

These are variables, not secrets. None of them is a password. The pipeline needs no GitHub secrets at all.

When you copy the output of the two terraform output -raw commands, don't copy a % at the end. On a Mac, the terminal shows % when output has no line break after it. It's not part of the value. The provider value must end in clearpay-github-provider, and the service account must end in .iam.gserviceaccount.com.

Then, still in Settings, click Environments, then New environment. Name it production and click Configure environment. Tick Required reviewers, search for your own GitHub name, select it, and click Save protection rules.

Why: Job 2 is the only path to production. A required reviewer makes GitHub stop that job and wait until a person clicks Approve, so nothing reaches customers by accident. Job 2 waits for that approval in Chapter 19.

You'll see a Manage environment secrets button on this page. Don't click it. The production environment needs no secrets, because the pipeline gets its Google access through Workload Identity, not a stored password. If your repository already holds secrets from other projects, leave them where they are.

If GitHub doesn't offer Required reviewers, your repository may be private on a free plan. Make the repository public for this lab, or skip the reviewer. Without one, the production job runs as soon as you trigger it.

Next, check before you push. A pipeline that fails costs three minutes and gives a long log. A check costs ten seconds. First run the check from the end of Chapter 13 again. All six lines must print ok. Then check the repository name you gave Google:

REPO=$(grep '^github_repo' infra/gcp/terraform/05-identity/terraform.tfvars | cut -d'"' -f2)
case "$REPO" in
  https*|*.git|YOUR_GITHUB_USER/*) echo "MISSING github_repo must be owner/name. It is: $REPO" ;;
  */*) echo "ok   github_repo $REPO" ;;
  *)   echo "MISSING github_repo must be owner/name. It is: $REPO" ;;
esac

It must print ok and your own owner/name.

Now, trigger the pipeline:

git commit --allow-empty -m "trigger pipeline"
git push

Check

Open the Actions tab and click the running workflow. Watch the steps of build-scan-push finish one after another, each with a check mark: build, Scan API image, Scan UI image, Authenticate to GCP, push, Deploy to staging.

If a step turns red, find it here. Each one ends by telling you to re-run the workflow:

To re-run after a fix, open the Actions tab, click the failed run, and click Re-run all jobs at the top right. You don't need to push again.

The second job, promote-to-production, shows a gray circle with a slash. That's correct. GitHub skips it on every push. It only runs when you start it by hand in Chapter 19.

When Deploy to staging passes, check the cluster:

kubectl -n staging get pods

The API and UI pods show Running with READY 1/1.

Staging has no public address, and the load balancer still sends every customer to AWS. Don't open a browser for staging. Chapter 15 reaches it through a private tunnel.

If the Scan Fails

Never change exit-code: "1" to "0". That turns the security gate off.

The Dockerfiles are already patched, so the scan should pass. A new security hole can turn it red later. Open the red job. Trivy prints a table. Find the Target column and match it:

After any Dockerfile change, rebuild locally and confirm the user is still appuser:

docker build -t clearpay-api:test ./api
docker run --rm clearpay-api:test whoami

Commit, push to main, and watch Actions again.

If Authenticate to GCP Fails

The error looks like this:

The given credential is rejected by the attribute condition.

Google accepted the GitHub token and then rejected it. Stage 5 names a different repository, often the placeholder YOUR_GITHUB_USER/YOUR_REPO. The value must be exactly owner/name, like Osomudeya/migrate-api-aws-gke-devops. A full URL doesn't match, and neither does another person's repository. Repeat step 2 above with your exact owner/name. Then retry without changing the workflow:

git commit --allow-empty -m "retry pipeline after WIF repo lock"
git push

If Deploy to Staging Fails

Two messages mean the same thing: error: timed out waiting for the condition and error: deployment "clearpay-api" exceeded its progress deadline. The files applied, but the pods never became ready. Ask the cluster why:

kubectl -n staging get pods
kubectl -n staging describe pod -l app=clearpay-api

Then match what you see. The first two rows appear in the job log. The rest appear in the pod output:

Fix A. Give the pipeline access to the staging secrets:

cd infra/gcp/terraform/06-secrets
terraform plan
terraform apply

Fix B. Let the cluster read the registry. The machines pull images as the default Compute Engine service account, which needs artifactregistry.reader. The plan adds that one rule:

cd infra/gcp/terraform/04-registry
terraform plan
terraform apply

Then restart the staging pods and wait:

kubectl -n staging rollout restart deployment/clearpay-api deployment/clearpay-ui
kubectl -n staging rollout status deployment/clearpay-api --timeout=180s
kubectl -n staging rollout status deployment/clearpay-ui --timeout=180s

Fix C. Run one API pod in staging: The small lab cluster can't fit two. Scale down, then re-run the workflow:

kubectl -n staging scale deployment/clearpay-api --replicas=1

To read the API's logs without printing secrets:

kubectl -n staging logs deployment/clearpay-api --tail=20

Done When:

• You can explain what each of the two jobs in deploy.yml does

• The production environment exists with you as required reviewer

• Deploy to staging passed

• kubectl -n staging get pods shows the API and UI Running, READY 1/1

The pipeline stops there. It doesn't run the promotion gate. You do, in Chapter 15.

Chapter 15. Check Staging and Run the Promotion Gate

Week two, Wednesday, 10:00 a.m.

The pipeline finished without an error at 6:40 last night. You took a screenshot. Nobody at ClearPay cares about the screenshot.

Daniel stops at your desk and looks at the passing pipeline.

Daniel: "So GKE is healthy?"

You answer: "The pods are healthy. That doesn't prove a customer can send twenty dollars."

Maya has read the thread. She adds a line.

Maya: "Customers don't get this build until the promotion script passes. Running pods aren't enough for me."

Why This Matters

Kubernetes reports a pod as ready when its health check answers. That says the process is up and can reach the database. It says nothing about logins, payments, or double charges. The promotion gate checks those.

The promotion gate is a script that you run on your laptop. The pipeline doesn't run it. It's also not the gray promote-to-production job you saw in GitHub. That job comes later, in Chapter 19, and only after this gate passes.

How Staging Runs

GKE cluster "clearpay"   zone us-east4-a
│
├── namespace staging
│     Service clearpay-api :80
│     └── API pods   (HPA 1–3)
│
└── namespace production   (created in Chapter 19)
      Gateway clearpay-gke-edge   ← clearpay-gke-edge-ip
      HTTPRoute → Service :80
      └── API pods   (HPA 1–3)
            each pod: DB_POOL_SIZE=10

Daniel looks at the same diagram.

"How many API processes can production run?"

"Up to three on this cluster. That's the HPA ceiling in patch-hpa.yaml."

"And how many database connections can each one open?"

"Ten. Same as db.js on Monday."

"So what could the database see?"

You stop. 30 from GKE. Plus 10 from AWS while both are live. On a cluster that could scale to 8 pods, it would be 80 from GKE alone.

"That's the number I asked you to remember," Daniel says.

Three files are worth opening now:

• k8s/base/deployment.yaml: the health checks call /health/ready, and each API pod reserves 250m CPU and 256Mi memory.

• k8s/base/networkpolicy.yaml: limits which pods may talk to the API.

• k8s/overlays/staging/patch-deployment.yaml: staging's replica count and rollout settings.

Step 1. Check the Database Path from GKE

Run a one-line Node script inside a staging API pod. It opens a database connection and prints connected:

kubectl -n staging exec deployment/clearpay-api -- node -e \
  "require('./src/db').checkConnection().then(() => { console.log('connected'); process.exit(0); }).catch(e => { console.error(e); process.exit(1); })"

connected

Step 2. Check the Secrets Arrived

Kubernetes holds the synced Secret with two keys. describe lists key names and sizes, not values:

kubectl -n staging get secret clearpay-api-secrets
kubectl -n staging describe secret clearpay-api-secrets

The API process has the value, checked without printing it:

kubectl -n staging exec deployment/clearpay-api -- node -e 'if (!process.env.DATABASE_URL) process.exit(1); console.log("DATABASE_URL is set")'

DATABASE_URL is set

The script is in single quotes on purpose. The default Mac shell, zsh, treats a ! inside double quotes as a history command and stops with zsh: event not found.

The CSI mount shows the secret files by name:

kubectl -n staging exec deployment/clearpay-api -- ls /mnt/secrets-store

Look for DATABASE_URL and JWT_SECRET.

Step 3. Run the Promotion Gate

Staging is private. kubectl port-forward opens a tunnel so localhost:8080 on your laptop reaches the staging API. The & keeps the tunnel running in the background. From the repository root:

kubectl -n staging port-forward svc/clearpay-api 8080:80 &

Wait for this line before you continue:

Forwarding from 127.0.0.1:8080 -> 4000

Amara is already on the thread: "Run the replay. I want to see one debit."

Run the same gate you ran against AWS in Chapter 6. The four arguments are the tunnel URL, Alice's email, her password, and Bob's email:

./scripts/promotion-gate.sh http://localhost:8080 \
  alice@clearpay.dev password123 bob@clearpay.dev

PASS: health endpoint returns ok
PASS: readiness endpoint confirms DB connectivity
PASS: login succeeds and returns a token
PASS: replaying an idempotency key returns replayed=true
PASS: balance debited exactly once (100 cents), not twice

5 passed, 0 failed

GATE PASSED: safe to promote to production.

Stop the tunnel with kill %1.

Alice exists on GKE without a new seed because GKE reads the same RDS database as AWS.

Step 4. Confirm Customers Are Still on AWS

Staging has no public address:

kubectl -n staging get svc clearpay-api

NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
clearpay-api   ClusterIP   10.x.x.x       <none>        80/TCP    12m

ClusterIP with <none> means only the cluster can reach it.

The customer address still answers from AWS:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
curl -sk https://$INSTANCE_IP/health

{"status":"ok","served_by":{"platform":"aws",...}}

If you see gke here, stop. If the curl fails, AWS is down, which is a different problem from GKE taking traffic.

So both clouds are running, but only AWS is open to customers. Staging runs on its own inside GKE. The gate isn't what keeps it running. It only tests it, through your tunnel, which is the only door in.

Browser  →  api.clearpay.test  →  100 percent  →  AWS EC2  (serves everyone)

GKE staging  ←  your tunnel (port-forward)  ←  the gate and the load test
GKE production  (does not exist yet, Chapter 19)

Customers reach GKE only after Chapter 21, when the front door starts sending it real traffic.

If it Fails

Don't promote this build.

• Unexpected end of JSON input or Connection refused: the script ran before the tunnel was listening. If Forwarding from 127.0.0.1:8080 is already in the terminal, run only the script again. Don't start a second port-forward.

• Health passes but ready fails: the pod can't reach PostgreSQL. Go back to Chapter 13, Part B.

• Login fails: check the values you typed against the seed: alice@clearpay.dev, password123, bob@clearpay.dev.

• checkConnection prints a TLS or timeout error: same as ready failing. Chapter 13, Part B.

• The ls shows no files, or the pod is stuck in ContainerCreating: run kubectl -n staging describe pod <pod-name> and look for a Secret Manager permission error. The annotation on the staging clearpay-api service account must match the staging email from Stage 5, character for character. That's the email Stage 6 allowed to read the secrets.

Fix the cause, push through the pipeline if the code or files changed, and run the gate again.

Decision

Maya: "Are we ready?"

"The gate passed. Health, readiness, login, replay, and one debit. Amara's replay check passed. Customers are still on AWS."

Maya: "Then we can talk about production. Not before."

She looks at the terminal again. "Staging works. Now tell me whether we should keep sending GKE to that same Postgres over the internet."

Done When:

• checkConnection prints connected from a staging pod

• DATABASE_URL is set, and /mnt/secrets-store lists both secret names

• The gate prints 5 passed, 0 failed through the tunnel

• Staging is ClusterIP with no external IP, and the Elastic IP still answers aws

This is the checkpoint. Staging passed its gate, and GKE reaches the real database.

Next you prove it's safe, then prove it under load, before any customer traffic moves.

Part 5. How to Make GKE Production-Ready

Staging works. The promotion gate proved five things: the app runs, it reaches the database, a user can log in, a repeated payment charges once, and the balance drops by the right amount. That's a good result. It's not the same as being safe for customers.

Think of a new car that passed its first test drive. It starts and it turns. Before anyone rides in it, you still ask three more questions. Are the doors locked? Can you see the dashboard? What happens on a steep hill, or when a tire goes flat?

This part asks the same three questions about ClearPay:

These questions come now, on purpose. Customers are still on AWS, so if something breaks in this part, nobody notices and nobody loses money. Fixing it here costs you an hour. Fixing it after customers arrive costs you their trust, and every fix happens while people are watching.

Chapter 16. Run the Security Review

Wednesday, 2:00 p.m.

After lunch, Maya sends the next task. She has read the pipeline log.

Maya: "Before any customer touches GKE, I want a security check. I don't want to find a secret sitting in a file after launch."

Why This Matters

Every earlier check asked "does it work?". This one asks "what's exposed?". For a payments API, a review after launch comes too late.

Check 1. Traffic to the Database is Encrypted

The database connection crosses the public internet between clouds. Confirm the live staging process requires TLS, without printing the URL:

kubectl -n staging exec deployment/clearpay-api -- node -e "const u=process.env.DATABASE_URL||''; console.log(u.includes('sslmode=require') ? 'sslmode=require is set' : 'sslmode=require missing')"

sslmode=require is set

Check 2. Secret Rotation is Understood

api/src/routes/auth.js issues login tokens that last 12h. If you rotate JWT_SECRET, every token signed with the old one stops working, and users must log in again. For this lab that is acceptable. Write that sentence in your notes as the rotation policy. Don't automate rotation here.

Check 3. Audit Logs Exist

Google Cloud records who read secrets and who changed permissions, separate from the app's own logs. Confirm those audit entries exist:

gcloud logging read 'protoPayload.serviceName=("secretmanager.googleapis.com" OR "iam.googleapis.com")' --limit=5 --freshness=7d --format='table(timestamp,protoPayload.methodName)'

Look for method names that mention SecretManager or iam. Seeing only iam names, such as SetIAMPolicy, is fine. That proves Google is recording permission changes. The timestamps will differ.

Check 4. Each Identity Has Only What it Needs

Staging's account reads staging secrets. Production's account reads production secrets. The pipeline pushes images and deploys, and can't change the cluster.

PROJECT_ID=$(gcloud config get-value project)
gcloud secrets get-iam-policy clearpay-staging-database-url
gcloud secrets get-iam-policy clearpay-production-database-url
gcloud projects get-iam-policy "$PROJECT_ID" --flatten='bindings[].members' --filter='bindings.members:serviceAccount:clearpay-ci@' --format='table(bindings.role)'

What to look for:

• The staging secret lists clearpay-api-staging and clearpay-ci. The pipeline reads the staging pair during deploys.

• The production secret lists clearpay-api-production only.

• clearpay-ci has two project roles: artifactregistry.writer and container.developer.

If a production secret lists a staging account, or the pipeline has a role like owner or editor, stop and fix it in Stage 5 or Stage 6.

Done When:

• All four checks are written in your notes with the output you saw, not "probably fine"

Chapter 17. Watch ClearPay With Prometheus and Grafana (Optional)

Thursday, 9:30 a.m.

The security review is done. Daniel reads your notes, then turns his chair toward you.

Daniel: "Once we start moving traffic, I want to watch a graph instead of guessing. What will you look at?"

Why This Matters

Note that this chapter is optional. It doesn't touch AWS or GKE, and no later chapter needs it. Skip to Chapter 18 if you want. Do it if you want to learn how Prometheus and Grafana work.

The API exposes numbers at /metrics: request counts, latency, and database pool use. Prometheus collects those numbers every few seconds. Grafana draws them.

This chapter runs Prometheus and Grafana on your laptop with Docker Compose, pointed at the Chapter 1 API. It proves the dashboard works. It doesn't watch GKE or AWS. Those target files exist in observability/targets/ as .example files, but this lab doesn't use them. During the traffic shift, the command-line checks in Chapters 21 and 22 are the source of truth.

Google Cloud has its own error counter from Stage 7. That's a Cloud Logging metric in GCP, separate from this Grafana.

Now     localhost:4000  →  Prometheus  →  Grafana :3000
Later   AWS, GKE, load balancer  (observability/targets/*.example, optional)

Steps

Prometheus reads host.docker.internal:4000, which is the Chapter 1 API. If curl http://localhost:4000/health fails, start it:

docker compose up -d api postgres

Then start the monitoring stack:

cd observability
docker compose up

Open http://localhost:3000. Anonymous admin access is on for this laptop-only stack. Open the ClearPay - Migration Dashboard.

Check

Send the local API a few requests:

curl http://localhost:4000/health

The Request latency p95 by env panel shows a series named local. An empty panel means Prometheus can't reach port 4000.

Done When:

• The dashboard shows a local series

Chapter 18. Load-Test and Break Staging on Purpose

Thursday, 1:00 p.m.

Staging passed its gate, and you feel done. Daniel puts your Chapter 6 scorecard and the gate results side by side on his screen.

Daniel: "The gate only tested one login. What happens when twenty requests arrive at once, or when a pod dies?"

Run the Chapter 6 load test against staging, put the two scorecards side by side, then kill a pod and restart the Deployment while load is running.

Why This Matters

Customers will hit GKE with many requests at once, and Kubernetes will restart pods during updates and machine repairs. You want to see both happen now, with no customer involved.

The system under test runs on the normal node pool from Stage 3. If you ever run the load generator in the cloud, a cheap Spot VM is a good fit for it, since it can be interrupted without harm. The system you measure should never run on Spot, because an interruption would ruin the numbers.

Step 1. Load-Test Staging

Use two terminals, both at the repository root.

Terminal A holds the tunnel for this whole chapter. Leave it running:

kubectl -n staging port-forward svc/clearpay-api 8080:80

Look for Forwarding from 127.0.0.1:8080.

Terminal B: first confirm you reach GKE and that Postgres answers:

curl -s http://localhost:8080/health
curl -s http://localhost:8080/health/ready

health must include "platform":"gke". ready must include "status":"ready".

Then run the load test with 20 concurrent requests for 30 seconds:

./scripts/load-test.sh http://localhost:8080 \
  alice@clearpay.dev password123 20 30

Copy the scorecard into your notes, next to the AWS one. This lab's numbers:

                AWS (Chapter 6)   GKE staging (this chapter)
p95 latency:    745 ms            655 ms
Error rate:     0.0 %             0.0 %
Recovery time
after pod kill: n/a               (fill in after Step 3)

AWS ran with 10 concurrent requests and ok=372. GKE ran with 20 concurrent requests, ok=686, failed=0, and 22.9 requests per second.

GKE's p95 is a little lower than AWS's here. That might surprise you. Most of both numbers is the trip from your laptop to the cloud, and that trip is about the same size for both. Every database query on GKE also crosses from Google's us-east4 to AWS's us-east-1 through Cloud NAT, but that adds only a small part on top.

The two runs used different concurrency, 10 and 20, so treat the numbers as close, not as a race. The line that needs to stay clean is failed=0. Keep both latency numbers in mind. They come back at the end of the book.

Step 2. Read the served by Lines

Open the ClearPay dashboard on api.clearpay.test and look at Recent activity. Expect mostly aws lines and one or two gke lines. This lab looked like this:

-$1.00  served by aws/1.0.0        ← a send you made in the browser
-$1.00  served by gke/0.0.0-unset  ← the promotion gate from Chapter 15
-$5.00  served by aws/1.0.0        ← an older send

Each line names the process that wrote that transfer. Both clouds write to the same database, and the UI reads every row. Why so few gke lines?

• Your browser sends go to AWS, because the front door still sends 100 percent of traffic there.

• Only the promotion gate wrote through GKE, with a $1.00 payment through the tunnel. The load test only reads data, so it adds no lines.

A gke line doesn't mean the load balancer sent your page to GKE. It means a GKE pod handled that transfer.

The two names come from environment variables. aws/1.0.0 comes from the EC2 .env file. gke/0.0.0-unset comes from k8s/base/deployment.yaml, where APP_VERSION is a placeholder the pipeline doesn't change. The placeholder is expected, not a crash.

The black bar at the top shows the process answering right now. Recent activity shows history.

Step 3. Kill a Pod and Time the Recovery

Delete one API pod and time how long Kubernetes takes to replace it:

kubectl -n staging get pods -l app=clearpay-api
POD=$(kubectl -n staging get pod -l app=clearpay-api -o jsonpath='{.items[0].metadata.name}')
echo "deleting $POD"
START=$(date +%s)
kubectl -n staging delete pod "$POD"
kubectl -n staging wait --for=condition=Ready pod -l app=clearpay-api --timeout=120s
echo "recovery $(( $(date +%s) - START )) s"
kubectl -n staging get pods -l app=clearpay-api

kubectl delete pod waits until the old pod is gone, including a 5-second preStop pause in k8s/base/deployment.yaml. That pause lets in-flight requests finish before the process stops. The recovery line is the time from the delete until the new pod is ready. An earlier run of this lab printed 42 s. Yours will differ. Write your number in the table above.

If wait times out, run kubectl -n staging get pods and kubectl -n staging describe pod on the new pod name.

The Terminal A tunnel may drop when the pod it points at is deleted. If Terminal A exits, start the port-forward again before Step 4.

Step 4. Restart the Deployment Under Load

This step needs one change to scripts/load-test.sh. Without it, failed= reads 0 even when requests die. The script only counts requests that get an answer. When the pod restarts, the tunnel drops, requests get no answer, and the script stops that worker without counting a failure.

Open scripts/load-test.sh in your editor. Replace the whole hit() function with this one. Two things are new: 2>/dev/null || true on the curl line, and the if line at the end.

hit() {
  local path="$1" out code secs
  out=$(curl "${CURL_OPTS[@]}" -o /dev/null -w '%{http_code} %{time_total}' \
    -H "Authorization: Bearer $TOKEN" \
    "$BASE_URL$path" 2>/dev/null || true)
  code="${out%% *}"
  secs="${out##* }"
  echo "$code" >> "$CODES"
  node -e "process.stdout.write(String(Math.round(Number(process.argv[1]) * 1000)) + '\n')" "$secs" >> "$TIMES"
  if [ "$code" = "000" ]; then sleep 0.5; fi
}

000 is what curl reports when it got no answer. The short pause stops a failing worker from looping thousands of times a second.

Now run the load test in Terminal B. It runs for 40 seconds, so the restart fits inside it:

./scripts/load-test.sh http://localhost:8080 \
  alice@clearpay.dev password123 5 40

Before you press Enter on that command, type the restart commands in a third terminal and leave them ready. Within three seconds of starting the load test, press Enter in the third terminal:

kubectl -n staging rollout restart deployment/clearpay-api
kubectl -n staging rollout status deployment/clearpay-api
kubectl -n staging get events --sort-by=.lastTimestamp | tail

The events show old pods stopping and new pods starting, one at a time.

Staging lets one pod go down before its replacement starts (maxUnavailable: 1 and maxSurge: 0 in k8s/overlays/staging/patch-deployment.yaml). The small lab cluster has no room for an extra pod. So requests fail for a short time during this restart, and failed= above 0 is normal. Write down the count. Terminal A may print lost connection to pod. Stop it with Ctrl+C and start the tunnel again after the test.

Production is safer. Its base setting starts a new pod before an old one stops (maxSurge: 1 and maxUnavailable: 0).

Decision

Pass: failed=0 on the 30-second test, the replacement pod became ready after the delete, and rollout status finished. GKE's p95 being higher than AWS is still a pass on this path, as long as it stays under 3 times your AWS p95, the failure limit you wrote in Chapter 6.

Fail: 5xx errors in the 30-second test, a login internal_error, or a new pod that never becomes ready. Fix staging, push through the pipeline, and run this chapter again.

Done When:

• Both scorecards are in your notes side by side

• Pod recovery is timed

• The rolling restart finished while load was running, and you wrote down failed=

Stop the port-forward in Terminal A with Ctrl+C.

This is the checkpoint. GKE survived load and a pod kill. Every check so far happened with no customer in the path.

That changes now.

Part 6. How to Move Customers to GKE

This is the part where customers can land on GKE. Both clouds can serve a request. Every step from here asks one question: what evidence lets you move more traffic?

Chapter 19. Deploy Production and the GKE Gateway

Week three, Monday, 9:00 a.m.

This is the week customers can land on GKE. Nobody says it out loud. The office is a little quieter than usual.

So far, GKE only holds staging. Staging is a private copy that you reach through a tunnel. Customers will use a separate copy called production. This chapter builds that copy, and it connects the address customers will call.

Two things are missing:

1. The production namespace. A namespace is a separate area inside the cluster. Staging has one. Production doesn't have one yet, so no production pods exist. You will create it and put the same image in it that passed the staging gate.

2. Something behind the GKE edge IP. In Stage 8 you reserved a public address for GKE, called the edge IP. It works like a phone number you reserved but never connected: it rings, and nobody picks up. You will connect it to a Gateway. The Gateway answers each request and passes it to the production pods.

When you finish, GKE can answer real requests. You still don't move any traffic. The front door keeps sending 100 percent of customers to AWS until Chapter 21.

Why This Matters

The load balancer can only send traffic to GKE if something on GKE answers. That something is a Gateway: a Kubernetes object that tells Google to build a load balancer in front of a Service. Here it listens on clearpay-gke-edge-ip and forwards to the production API and UI.

The 08-edge load balancer reaches both clouds through Internet NEGs (network endpoint groups that point at an outside IP address). This type has no health checks. If a backend goes down, the load balancer keeps sending it traffic, and customers get 502 errors. Your protection is synthetic-health.sh, the decision gates, and rollback.sh.

Step 1. Turn on the Gateway API

The cluster from Stage 3 doesn't have it on:

gcloud container clusters update clearpay --zone us-east4-a \
  --gateway-api=standard

Wait until it finishes, then:

kubectl get gatewayclass

Look for gke-l7-global-external-managed. That's the class k8s/gateway/gke-gateway.yaml uses.

Step 2. Make Room on the Node Pool

The three e2-medium machines have about 940m of CPU each after GKE's own system pods. Staging and production together don't fit. Scale staging down to make room. Its images stay in Artifact Registry, and the gate result stays in your notes.

First write down the exact images staging runs. Production must run these same strings:

kubectl -n staging get deploy clearpay-api \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n staging get deploy clearpay-ui \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Then scale staging to zero:

kubectl -n staging scale deploy/clearpay-api --replicas=0
kubectl -n staging scale deploy/clearpay-ui --replicas=0

Don't push to main during the traffic shift. A push redeploys staging at full size and takes the CPU back.

If production API pods stay Pending with Insufficient cpu later, staging is still holding CPU. Don't raise the node pool's maximum unless you accept the cost of another machine.

Step 3. Deploy Production

Use the pipeline. It's the path Chapter 14 built for this moment.

With the pipeline:

1. On GitHub, open Actions, then Build, scan, and deploy in the left list. Start from this workflow page, not from a finished run. If you're looking at a single run, click the back arrow next to its name. The promote-to-production job on that old run shows "This job was skipped", and it stays skipped. You need a new run.

2. Click Run workflow. Keep the branch on main, tick promote_to_production, and click Run workflow.

3. The promote-to-production job waits with "Waiting for review". Click Review deployments, tick production, and click Approve and deploy.

4. Watch Find the currently-deployed staging images and Deploy those exact images to production finish with a check mark.

The job reads the image strings from the staging Deployment, even though staging now runs zero pods, and deploys those exact images.

By hand, if the pipeline isn't set up:

These are the same steps the job runs. PROJECT_ID in the files is a placeholder, so replace it the way the pipeline does. On a Mac, sed -i needs an empty '' after it:

cd k8s/overlays/production
PROJECT_ID=$(gcloud config get-value project)
sed -i '' "s/@PROJECT_ID.iam/@${PROJECT_ID}.iam/" patch-serviceaccount.yaml
sed -i '' "s#projects/PROJECT_ID/#projects/${PROJECT_ID}/#" secretproviderclass.yaml

On Linux, drop the '' after -i.

Point the overlay at the two images you wrote down in Step 2. Never use latest:

kustomize edit set image \
  REGION-docker.pkg.dev/PROJECT_ID/clearpay/api=STAGING_API_IMAGE
kustomize edit set image \
  REGION-docker.pkg.dev/PROJECT_ID/clearpay/ui=STAGING_UI_IMAGE
kubectl apply -k .
kubectl -n production rollout status deployment/clearpay-api --timeout=180s
kubectl -n production rollout status deployment/clearpay-ui --timeout=180s

Replace STAGING_API_IMAGE and STAGING_UI_IMAGE with the full strings from Step 2, in the form us-east4-docker.pkg.dev/.../api:<tag>.

Don't commit the files after sed and kustomize edit. They now contain your Project ID and a real tag. Run git checkout k8s/overlays/production from the repository root to undo them after the deploy.

Check Production

Return to the repository root. Wait until the pods are ready:

kubectl -n production get pods

API and UI pods show Running and READY 1/1. Then check the database from a production pod, not staging:

kubectl -n production exec deployment/clearpay-api -- node -e \
  "require('./src/db').checkConnection().then(() => { console.log('connected'); process.exit(0); }).catch(e => { console.error(e); process.exit(1); })"

connected

Step 4. Apply the GKE Gateway

Stage 8 reserved clearpay-gke-edge-ip and the certificate clearpay-gke-edge-cert. Now give them something to serve:

kubectl apply -f k8s/gateway/gke-gateway.yaml
kubectl apply -f k8s/gateway/httproute-production.yaml
kubectl apply -f k8s/gateway/healthcheck-api.yaml
kubectl -n production get gateway clearpay-gke-edge

Wait until PROGRAMMED shows True. That can take a few minutes while Google builds the load balancer.

The three files do three jobs:

• gke-gateway.yaml creates the Gateway on the reserved IP with the certificate.

• httproute-production.yaml sends requests to the production Service.

• healthcheck-api.yaml tells Google to health-check /health. Without it, Google checks /, the API answers 404, and requests fail with no healthy upstream.

Check the Gateway

Call GKE directly through its edge IP. --resolve makes curl send the hostname on the certificate while connecting to the GKE IP:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
EDGE=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw gke_edge_ip_address)
curl -k --resolve "$HOST:443:$EDGE" "https://$HOST/health"

Look for "platform":"gke". If curl prints no healthy upstream, wait a minute and retry. The health check needs time to mark the pods healthy.

Without --resolve, the same hostname still goes to AWS, because weights are 100 / 0.

Done When:

• kubectl get gatewayclass lists gke-l7-global-external-managed

• Production API and UI pods are Running, READY 1/1, with the staging image tags

• checkConnection prints connected from a production pod

• The Gateway shows PROGRAMMED True, and the edge IP returns "platform":"gke"

Chapter 20. Walk the Pre-Cutover Checklist

Monday, 3:00 p.m.

Production is up. Maya stops at your desk with her laptop under her arm.

Maya: "Before we move traffic, tell me exactly how we put customers back on AWS. Then show me it works."

Why the Rollback Drill Matters

scripts/rollback.sh does two things, in order:

1. It runs shift-traffic.sh 100 0, which sends all traffic back to AWS in seconds.

2. It connects to the AWS server over SSH, sets ROLLBACK_ACTIVE=true in its .env, and restarts the API. The UI then shows the rollback banner.

The dashboard has two different messages, on purpose:

• The new-version message is a small pop-up in the bottom right corner. It says "Noticed a change: this request ran on gke/... (previously aws/...)" and fades after five seconds. It appears when a different version answers.

• The rollback banner is a red-tinted strip across the top of the dashboard. It says "Rolled back to AWS, traffic has been restored while the GKE issue is investigated." It stays until you clear the flag.

Anyone looking at the app can tell "a new version shipped" apart from "something broke and we recovered".

The first time you run a rollback shouldn't be during a real incident, with customers failing, hoping the SSH key works. Run it now while nothing is wrong. Weights are already 100 / 0, so the traffic step changes nothing. The SSH step and the banner are what you test.

The Checklist

Tick each line with a command you ran, not from memory. Use your own Elastic IP and hostname.

Only four lines need a command now: GKE healthy, Rollback tested, AWS healthy, and Synthetic test running. The other five use results you already have, so read them and tick them. Chapters 21 and 22 don't repeat this checklist. They reuse only the synthetic test, which you leave running in its own terminal.

Pre-Cutover Checklist

[ ] GKE healthy
    kubectl -n production get pods
    (all Running, READY)

[ ] Database healthy
    the production checkConnection from Chapter 19 prints connected

[ ] Monitoring up (optional, only if you did Chapter 17)
    Grafana at http://localhost:3000 loads.
    It does not watch GKE. Do not block on an empty GKE series.

[ ] Rollback tested
    YOUR_ELASTIC_IP is your AWS server's public IP. Get it with:
    terraform -chdir=infra/aws/terraform output -raw instance_public_ip
    ./scripts/rollback.sh YOUR_ELASTIC_IP ~/.ssh/clearpay-deployer
    Open https://YOUR_ELASTIC_IP, log in, and look for the red strip
    at the top of the dashboard ("Rolled back to AWS ..."). Then clear it:
    ./scripts/clear-rollback-flag.sh YOUR_ELASTIC_IP ~/.ssh/clearpay-deployer

[ ] AWS healthy
    ./scripts/promotion-gate.sh https://YOUR_ELASTIC_IP \
      alice@clearpay.dev password123 bob@clearpay.dev
    5 passed, 0 failed

[ ] Synthetic test running
    ./scripts/synthetic-health.sh https://YOUR_PUBLIC_HOSTNAME
    Leave it running in its own terminal for the whole shift.
    YOUR_PUBLIC_HOSTNAME is api.clearpay.test. From the repository root, confirm with:
    terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname
    It prints one line per second: time, status code, latency.

[ ] Limits confirmed
    Chapter 6 limits, plus the Chapter 18 table:
    AWS p95 745 ms, GKE staging p95 655 ms, both 0.0 % errors, plus your pod recovery time from Chapter 18

[ ] Security review passed
    Chapter 16, all four checks written down

[ ] Cost ledger current
    infra/aws/COST.md and infra/gcp/COST.md, both clouds now billing

What the Rollback Drill Prints

The traffic weights are already 100 / 0, so step 1 changes nothing. Step 2 is the real test. It ends like this:

== Traffic split updated ==
"AWS: 100% / GKE: 0%"

== Step 2: flagging AWS as the active rollback target ==
AWS ROLLBACK_ACTIVE=true, service restarted.

== Rollback complete ==

The last lines of the script name old chapter numbers. Ignore them. The steps for a real rollback are in Chapter 23.

If step 2 stops with ssh: connect to host ... port 22: Operation timed out, your home IP has changed and the AWS firewall still allows the old one. Get your IP with curl -s ifconfig.me; echo. Put it plus /32 in ssh_allowed_cidr in infra/aws/terraform/terraform.tfvars. Run terraform apply in infra/aws/terraform, then run the script again.

Decision

Every line ticked, and the GKE edge IP returns "platform":"gke": go to Chapter 21.

Anything unticked: stop here. Don't run ./scripts/shift-traffic.sh 99 1 while the GKE edge IP is empty. One percent of customers would hit an address with nothing behind it.

Tell Maya: "Rollback is one script. It moves traffic in seconds and shows a banner. I ran it five minutes ago."

Done When:

• Every checklist line is ticked with output you saw

Chapter 21. Send 1 Percent of Traffic to GKE

Tuesday, 10:00 a.m.

The whole team is on one call. Chris has the support inbox open on his second screen. Amara has the transactions table open on hers. Daniel sits next to you with the load balancer page open.

Maya: "One percent to GKE. Watch the errors and the transfers. If they look wrong, send everything back."

Why 1 Percent?

One percent of requests. A request is one message from a browser to the API: a login, a balance check, a payment. Out of every 100 requests, about 99 go to AWS and about 1 goes to GKE. The load balancer picks for each request, not for each customer. So one customer can land on both clouds in a single visit.

One percent is enough to find a broken contract, such as a different JSON shape, a missing route, or a login that fails on one side. It's far too small to test capacity. Chapter 22 handles capacity.

Steps

./scripts/shift-traffic.sh 99 1

The script re-applies 08-edge with new weights on the load balancer's URL map. There's no DNS change and no deploy.

Now copy the same weights into infra/gcp/terraform/08-edge/terraform.tfvars:

aws_traffic_weight = 99
gke_traffic_weight = 1

The script passes the weights as command-line values. The next plain terraform apply in 08-edge reads the file. If the file still says 100 and 0, that apply would quietly move traffic back.

Confirm the file and the live state match:

grep traffic_weight infra/gcp/terraform/08-edge/terraform.tfvars
terraform -chdir=infra/gcp/terraform/08-edge output current_traffic_split

You should see aws_traffic_weight = 99, gke_traffic_weight = 1, and "AWS: 99% / GKE: 1%". If the file still says 100 and 0, fix it now.

Check: Did Any Request Reach GKE?

At 1 percent, almost every request still goes to AWS. Forty requests in a row that all say aws is normal. Loop until one says gke:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)
while true; do
  curl -sk --resolve "$HOST:443:$IP" "https://$HOST/health"
  echo
  sleep 0.2
done

Stop with Ctrl+C when you see "platform":"gke". One run in this lab took 200 requests: 197 aws and 3 gke.

To confirm GKE is still answering without waiting on chance, call its edge IP directly. This isn't a customer request:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
EDGE=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw gke_edge_ip_address)
curl -sk --resolve "$HOST:443:$EDGE" "https://$HOST/health"

Look for "status":"ok" and "platform":"gke". This lab printed:

{"status":"ok","served_by":{"platform":"gke","version":"0.0.0-unset","instance":"clearpay-api-6f9448c857-mj2j6","commit":"unset","rollback_active":false}}

Your instance will differ. If platform says aws, or the curl fails, GKE isn't answering. Don't raise the weight.

Check: Does a Login Work Through the Public Address?

From now on, one customer can land on AWS for one request and on GKE for the next. Both must accept the same request and give the same reply. If they differ, you want to find out now, at 1 percent, not later at 50 percent. Send a login through the public address:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)
curl -sk --resolve "$HOST:443:$IP" \
  -H 'Content-Type: application/json' \
  -d '{"email":"alice@clearpay.dev","password":"password123"}' \
  "https://$HOST/api/auth/login"

Look for "token" and "user" in the reply. If you see internal_error, stop and don't raise the weight.

Check: Use the App Like a Customer

Open https://api.clearpay.test and click through the certificate warning. Log in, check the balance, and send a small amount. Each new line under Recent activity says served by aws/1.0.0 or served by gke/0.0.0-unset. Only the newest line tells you about this request. Older lines keep the stamp from when they were written.

The black bar at the top will say AWS almost every time at 1 percent. Watch new activity lines and the loop, not the bar.

Glance at the synthetic-health.sh terminal. Every line should show status 200.

Decision

• The loop showed gke at least once, login returns a token, and synthetic health shows only 200: go to Chapter 22.

• Anything else: stop and investigate. Don't raise the weight. If customers are affected, run rollback.sh and go to Chapter 23.

Done When:

• Weights are 99 / 1 in both the file and current_traffic_split

• The loop printed "platform":"gke" at least once

• Login through the public address returns a token

Chapter 22. Raise Traffic to 100 Percent

Tuesday, 11:30 a.m., to Wednesday afternoon

One percent worked. That means 1 out of every 100 requests went to GKE. Now you turn the dial up, a little at a time. Nothing does it for you. You raise it by hand, and you look at the numbers before every step.

Check the same three things each time:

• The scorecard: how fast the app answers, and how many requests fail.

• The replica count: how many copies of the API are running.

• The connection math: whether the database still has room for every connection.

99 / 1     AWS ████████████████████░    GKE ░
90 / 10    AWS ██████████████████░░     GKE ██
75 / 25    AWS ███████████████░░░░░     GKE █████
50 / 50    AWS ██████████░░░░░░░░░░     GKE ██████████
25 / 75    AWS █████░░░░░░░░░░░░░░░     GKE ███████████████
10 / 90    AWS ██░░░░░░░░░░░░░░░░░░     GKE ██████████████████
 0 / 100   AWS                          GKE ████████████████████

Before 50 / 50: Do the Connection Math

Daniel, who is on the call, cuts in before you go to the halfway mark. "How many database connections can hit RDS if GKE scales all the way up while AWS is still live?"

Each API process can open up to DB_POOL_SIZE connections, ten by default. The pool opens them as load needs them, so the number stays low when traffic is light. At full load, every process can hold its full ten. The database has a hard limit, max_connections. When the total passes it, new connections are refused, requests wait, and customers see timeouts.

Get your database's limit. This runs psql on the AWS server, which already has it from Chapter 5:

export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
DATABASE_URL=$(terraform -chdir=infra/aws/terraform output -raw database_url)
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP "psql '$DATABASE_URL' -c 'SHOW max_connections;'"

Get GKE's ceiling from maxReplicas in k8s/overlays/production/patch-hpa.yaml. Then fill in this table with your numbers:

RDS max_connections:          ___
AWS, 1 process × 10:          10
GKE, maxReplicas × 10:        ___
Worst case, both at once:     10 + ___ = ___

This lab's filled-in table, so you can check yours:

RDS max_connections:          81
AWS, 1 process × 10:          10
GKE, maxReplicas × 10:        30    (3 pods × 10)
Worst case, both at once:     10 + 30 = 40

40 is under 81, so there's room to raise the traffic. Your max_connections may differ if your RDS instance is a different size. What matters is that your worst case stays below it.

Now run the same math for a bigger setup. A company with larger machines might run the HPA from 3 to 8 pods: 10 + 80 = 90 connections, against a small RDS instance that allows about 87.

Neither setting is wrong on its own. The pool size was set in the API code, and the pod ceiling was set in a Kubernetes file, by different people at different times. They only collide when both clouds run at full load at the same time, which is the exact situation a gradual migration creates.

If your worst case is above your max_connections, stop and read the fix options in Chapter 23 before you raise the weight.

Steps

Every stage has the same four moves. Do all four, then repeat with the next pair of numbers.

1. Shift: Run ./scripts/shift-traffic.sh with the new pair. The first number is the share for AWS, and the second is the share for GKE.

2. Save: Put the same two numbers in terraform.tfvars in 08-edge.

3. Check: Run the six checks in the next section. No single script does all of them. You use a few commands, plus two scripts: synthetic-health.sh, which is already running, and load-test.sh.

4. Decide: Every check passes: go to the next stage. Any check fails: stop.

The stages, in order:

Run one row at a time, and never run them all together.

Checks at Every Stage

1. The dial moved. The script's printout is Terraform's setting, not proof. Confirm the file matches:

grep traffic_weight infra/gcp/terraform/08-edge/terraform.tfvars
terraform -chdir=infra/gcp/terraform/08-edge output current_traffic_split

2. Real requests follow the dial. Wait three minutes after the shift, then count who answers:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)

for i in $(seq 1 40); do
  curl -sk --resolve "$HOST:443:$IP" "https://$HOST/health"
  echo
done | tee /tmp/clearpay-health.txt

grep -c '"platform":"aws"' /tmp/clearpay-health.txt
grep -c '"platform":"gke"' /tmp/clearpay-health.txt

The GKE count should rise as its weight rises. Forty requests are a rough sample, not a percentage proof. Google also takes a few minutes to apply a new split everywhere, so counting too early shows the old split. This lab saw:

90 / 10   80 requests:  75 aws,  5 gke
50 / 50   40 requests:  34 aws,  6 gke
25 / 75   40 requests:  17 aws, 23 gke
10 / 90   40 requests:   8 aws, 31 gke
 0 / 100  40 requests, 1 minute later:    3 aws, 37 gke
 0 / 100  40 requests, 90 seconds later:  0 aws, 40 gke

If the counts still don't follow the dial after ten minutes, ask Google which split it holds:

gcloud compute url-maps describe clearpay-migration-map | grep -B2 -A2 weight

At 0 / 100, the clearpay-gke-backend line shows weight: 100, and the AWS line shows no weight at all, because Google leaves out a zero. If the weights differ from your file, run terraform apply again in 08-edge.

3. GKE is healthy.

kubectl -n production get pods -l app=clearpay-api
kubectl -n production get hpa clearpay-api

Pods are Running and ready. The HPA line shows current CPU against the 70 percent target, and the current replica count. This lab stayed at 1 replica and 4 to 7 percent CPU through 25 / 75.

4. Logins still work. Run the Chapter 21 login curl again at every stage. A contract problem can appear only under a certain mix of traffic. Five failures in a row means stop, not raise.

5. Errors stay inside your limits. The synthetic-health.sh terminal should show only 200. If a line shows anything else, stop and don't raise.

6. The UI. Log in on https://api.clearpay.test, send money, and read the newest served by line.

The Decision at Every Stage

Maya asks the same question each time: "Are we ready for the next step?"

Answer with evidence: "The dial matches the file and GKE's share of requests rose. Pods are healthy, login works, and synthetic health is all 200. The worst-case connection count fits under the database limit."

If any check fails, don't raise. If customers are affected, go to Chapter 23 now.

Done When:

• Weights are 0 / 100 in the file and in current_traffic_split

• 40 requests in a row return "platform":"gke"

• The connection math is written down with your own numbers

• Synthetic health showed no sustained errors during any stage

Chapter 23. When the Numbers Go Bad

Wednesday, 4:12 p.m.

The last shift is done. Every customer is now on GKE, and the health check keeps printing 200, which means all is well. The support inbox has been quiet for the past hour. Amara checks the transactions table one more time.

"It looks boring," she says. Which is the best thing she could say.

You wait for someone to call it a successful migration.

Daniel is still looking at the deployment. He counts the API copies running in GKE, then looks at the database metrics.

"Can I tell you about the worst afternoon at my last job?"

Chris turns his chair around.

Daniel leans back.

"It was another migration for a different company, same basic problem though. We were moving customers from one cloud to another, and we'd reached about half of them by Friday afternoon. Traffic started climbing, so the new environment began adding more copies of the API."

He pauses for a moment.

"The old environment was still running too. It was serving the customers we hadn't moved yet."

You look at the deployment on the screen.

"So both environments were using the database?" "Yes. And every copy of the API needed database connections."

He points at the database on the diagram.

"That was where we got into trouble." Daniel reaches for a pen.

"Think about a restaurant with a fixed number of seats. The restaurant has enough seats for a certain number of people, but the number of seats doesn't change because more people arrive."

He draws two groups on the whiteboard.

"AWS was one group. GKE was the other. Both groups were using the same restaurant."

He added another group beside GKE.

"Then traffic increased. GKE added more copies of the API, and each new copy needed database connections. The old AWS copies were still holding their connections too."

He puts the pen down.

"The database had a limit. We kept increasing the number of connections going into it."

"What happened when you reached the limit?" you ask.

"Requests started waiting for connections. Then they started timing out."

He looks at you.

"The application itself wasn't obviously broken. If the pods were running and health checks were passing, the problem was capacity. We had more demand for database connections than the database could provide."

Chris looks back at the deployment.

"So you were watching the new environment, but the database was dealing with both environments."

"Exactly."

Daniel looks at the diagram again.

"That was the part we missed. We calculated what the new environment needed. We didn't calculate what the database needed while both environments were running."

He sits back down.

"We eventually moved the traffic back to the old environment and recovered. But the incident stayed with me."

You look at the GKE deployment. "What would you do differently now?"

"I'd work through the numbers before moving traffic."

Daniel turns his laptop toward you.

"Your lab is safe. You ran 3 copies against a database limit of 81 connections, and the workload never came close to exhausting it."

He opens another set of notes.

"My incident was different. We had up to 8 copies, and the database allowed 87 connections."

He looks at you. "So let's use those numbers."

You pull your chair closer. Daniel continues.

"I'm going to give you the events in the order they happened. You're on call. Don't assume you know the ending. Tell me what you would check and what you think is happening as each event comes in."

He opens a terminal.

The drill begins with 8 API copies available across the 2 environments and a database connection limit of 87.

This time, the numbers are going to matter.

Do you run anything in this chapter? Read it once, like a story with a lesson. Your lab had no incident, so there's nothing to fix. If you want proof that you're in the clear, run the two commands in Step 2 and compare them with "Your own cluster". If the logs show no errors and the HPA is at 1 replica, you're done. Go on to Chapter 24.

What it Looks Like

The first few minutes after the 50 / 50 shift look normal. Then the response times begin to creep up.

Amara notices it first in the team channel.

"A couple of transfers just timed out."

Daniel opens the production dashboard. The API has scaled to seven replicas, and the p95 latency is climbing.

Chris posts two customer reports about slow transfers.

Daniel looks at the graph again. "Is this affecting every request?"

"Not yet," Chris says. "A few transfers so far."

Amara checks the payment records. "Let's make sure those timeouts didn't result in duplicate charges."

Daniel looks at the database metrics. "Could be the database."

You don't know yet.

Nothing in the logs says, "The database is out of connections." The first signal is simply that requests are taking longer than they were a few minutes ago.

No error message says "the database is out of connections". All you see first is the slowest responses (p95 latency) getting slower. The status bar in the app (ui/src/components/StatusRibbon.jsx) switches to "Degraded. Investigating" only after several health checks fail in a row, so one slow request doesn't trigger it.

What You Know

Traffic is on both clouds. Latency is worse than the AWS baseline. The status bar flipped after several failed checks, not one blip.

What You Don't Know Yet

Whether this is the load balancer, the pods, the database, or a bad deploy. So don't assume. Gather info first.

Step 1. Stop Shifting

Don't run shift-traffic.sh toward GKE again until you understand what happened. Write down the time you noticed.

Record what you see before you do anything else: the actual error messages, the actual replica count, and the actual timestamp.

Every hour spent diagnosing at a stalled percentage is also an hour of dual-run cost. Add it to the cost ledger. Don't let it drop out of the running total because everyone's attention is elsewhere.

Step 2. Gather Evidence

Read the API's errors:

kubectl -n production logs deployment/clearpay-api --tail=100 | grep error

Look for words about the database: connection errors, "too many connections", or a timeout while waiting for a client from the pool. A 5xx error with no database words points somewhere else, such as the Gateway or a bad deploy.

Check how many pods Kubernetes is running:

kubectl -n production get hpa clearpay-api

Compare REPLICAS with maxReplicas. A climbing count means load went up. That's a clue, not yet a cause.

Then fill in the Chapter 22 table with the live numbers:

RDS max_connections:          87
AWS, 1 process × 10:          10
GKE, 7 replicas × 10:         70
Right now:                    80, close to the limit
GKE at maxReplicas 8:         80, worst case 90, over the limit

Your Own Cluster at 50 / 50

Run the same two commands on your cluster before you assume you're in Daniel's story. This lab at AWS 50 / GKE 50 showed:

HPA:       cpu 4%/70%, MINPODS 1, MAXPODS 3, REPLICAS 1
pods:      1 Running
logs:      no error lines in the last 100
RDS:       max_connections=81
           two sessions on database clearpay
status:    not Degraded

Worst case here is up to 40, under 81. Don't run rollback.sh unless your numbers are worse than your Chapter 6 limits.

If your HPA is still at 1 and the logs are quiet, write that down. That's the evidence. Stay with it.

How the Failure Builds

AWS alone would never hit this. GKE alone would never hit this. It happens only while both run at full load, and the migration plan keeps both running on purpose.

Tell Daniel: "Replica count climbed. The pool math puts us near the database limit. This is capacity, not Kubernetes."

Step 3. Roll Back

Maya: "What do you recommend?"

Compare the numbers with the limits you wrote in Chapter 6. Don't invent a new limit now.

"Put customers back on AWS. Errors are past our limit, customers are affected, and rollback is tested."

Maya: "Okay. Do it."

./scripts/rollback.sh <aws-instance-ip> ~/.ssh/clearpay-deployer

The diagram above shows two actions performed by rollback.sh. First, shift-traffic.sh 100 0 sends all customer traffic back to AWS and removes traffic from GKE. Second, the script connects to the AWS EC2 server over SSH, sets ROLLBACK_ACTIVE=true in the API's environment file, and restarts the API service. The API then reports AWS as the active platform and enables the distinct “Rolled back to AWS” banner in the ClearPay UI. These steps restore traffic and make the rollback visible to the operator.

Set terraform.tfvars in 08-edge back to 100 and 0 so a later apply doesn't undo the rollback.

Step 4. Confirm the Rollback Worked

curl -k https://api.clearpay.test/api/info

platform reads aws and rollback_active reads true. Refresh the UI: the red "Rolled back to AWS" banner appears, and the status bar goes back to All systems operational once enough health checks pass.

Then check each of these from evidence, not assumptions:

• Requests stopped failing (synthetic-health.sh shows 200 again)

• Latency is back near the Chapter 6 AWS numbers

• The database is consistent: a transfer from just before the rollback shows exactly one debit, not zero and not two

Amara: "I don't care that traffic is back on AWS. Did the transfer happen once?"

Answer from the database, through the transaction history, not from the load balancer. The idempotency key from Chapter 3 is what makes a retried transfer safe here, even when the first try went to GKE and the retry went to AWS.

Write down one number for the incident review: the minutes from "noticed" to "traffic back on AWS".

Step 5. Fix the Cause

Daniel: "What will you change so that number can't happen again?"

There are four real options, each with a cost:

• Lower DB_POOL_SIZE per pod: Fewer connections per process. Cheap and fast. Each pod can serve fewer database queries at the same time.

• Lower the HPA's maxReplicas in k8s/overlays/production/patch-hpa.yaml. Caps the total. GKE can no longer scale as far under a real traffic spike.

• Raise RDS max_connections with a custom parameter group. Each connection uses database memory, and a small instance like db.t3.micro can fail to start if you set it too high. Check AWS's guidance for your instance size first.

• Put a connection pooler in front of Postgres, such as PgBouncer or RDS Proxy. Many app connections share a few database connections. The best long-term fix, and the most work to set up.

This book doesn't pick one for you. Write down which one you chose and what you gave up.

For the smallest real change, lower the pool size. The math with 6 per pod: 8 × 6 = 48 from GKE, plus 10 from AWS, is 58. That fits under 87 even at the full ceiling.

In an emergency you can set it live:

kubectl -n production set env deployment/clearpay-api DB_POOL_SIZE=6

A live change leaves no record, and the next deploy erases it. Make the real change through the pipeline: set DB_POOL_SIZE in k8s/base/deployment.yaml, commit, and push to main. The pipeline builds, scans, and deploys staging. Load-test staging hard enough to push the HPA to its ceiling, using Chapter 18's steps with higher concurrency. A fix that only passes under light load proves nothing. Then run the promotion gate, and promote with the Chapter 19 pipeline job.

Step 6. Resume

Maya: "Are we ready for 10 percent?"

"Yes. The API is healthy, synthetic checks are clean, errors are within the limit, the pool math holds at the ceiling, and rollback is tested."

Daniel: "Agreed."

Maya: "Your call."

Clear the rollback flag first:

./scripts/clear-rollback-flag.sh <aws-instance-ip> ~/.ssh/clearpay-deployer

Refresh the UI. The rollback banner is gone. Leaving the flag on would keep the banner showing after recovery, which trains people to ignore it.

Then repeat Chapters 21 and 22 from the start:

./scripts/shift-traffic.sh 99 1
# observe, compare, record, decide

./scripts/shift-traffic.sh 90 10
# observe, compare, record, decide

./scripts/shift-traffic.sh 75 25
./scripts/shift-traffic.sh 50 50
./scripts/shift-traffic.sh 25 75
./scripts/shift-traffic.sh 10 90
./scripts/shift-traffic.sh 0 100

Watch the HPA and the error lines closely at the stage where the incident happened last time. That stage is where the fix proves itself.

Add a line to the cost ledger. A second pass through the shift means more hours with both clouds billing. Keep that cost visible instead of folding it into one "migration cost" number.

The Skills Behind This Chapter

The diagram above shows the incident response process from observation to recovery. Start by checking latency, the status bar, and error logs. Form a hypothesis, test it against the database connection limit, and use live measurements as evidence. If customers are affected, stop shifting traffic and roll back. Fix the cause through the pipeline, test under maximum expected load, and resume the migration only when the evidence supports it.

An incident isn't a hunt for the answer. It's the work of reducing uncertainty until you can make a safe decision. If customers are failing, the safe decision is to stop and roll back. If they aren't, the safe decision is to keep watching. Don't roll back because a number looks odd, and don't keep shifting because the pods look healthy.

This is the checkpoint. GKE is serving 100 percent.

AWS is a priced safety net now, not a leftover you forgot.

Part 7. How to Finish the Migration

GKE serves every customer. AWS still runs, and still bills, as a safety net. This part turns the cutover into evidence, puts a date on the safety net, and removes what you no longer need.

Chapter 24. Prove the Cutover With Numbers

Thursday, 9:00 a.m.

Maya has a meeting with the company leaders at ten. She stops at your desk with her laptop open to an empty page.

Maya: "The leaders weren't on any of our calls, and they'll ask one thing: did customers notice the move? I need a short summary I can put on the screen. How many requests went through, how many failed, how fast the app answers compared with AWS, and whether anyone saw an outage. Can you have it before ten?"

You nod. Every number she asked for comes from two scripts you already know.

Steps

Run the two checks that produce hard numbers against the public address, now at 100 percent GKE:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
./scripts/promotion-gate.sh "https://$HOST" \
  alice@clearpay.dev password123 bob@clearpay.dev

Then run load-test.sh against https://$HOST with the Chapter 6 settings, and copy the scorecard. Add the counts from your Chapter 22 loops and the synthetic-health.sh log.

The Evidence

Below is this lab's table. Yours will have different numbers:

Production requests (counted checks, not a full dashboard total):
  40 /health one minute after 0 / 100:  3 aws, 37 gke
  40 /health ninety seconds later:      0 aws, 40 gke
  load-test.sh 10 × 20s:                ok=310  failed=0
  promotion-gate.sh:                    5 passed, 0 failed
Successful requests:                 310 + 40 + 5 in those runs
Failed requests:                     0
5xx errors:                          0.0 %
p95 latency (GKE, 100%):             647 ms
  Chapter 6 AWS baseline was 164 ms.
  Chapter 18 GKE staging (port-forward) was 1157 ms.
Sustained customer-visible outage:   0 seconds

Local Prometheus didn't watch GKE, so this table uses counted checks. Don't invent a dashboard total you never measured.

Check the table against the limits you wrote in Chapter 6, before GKE existed:

• Outage limit: no stretch of 5xx errors above your limit. This lab had zero failed requests.

• Failure limit: p95 under your ceiling. This lab's 647 ms is under the example ceiling of 1500 ms, so it passes.

Passing isn't the whole story. 647 ms is about four times the AWS 164 ms. Customers never saw an outage, but every request got slower. Write that down plainly instead of hiding it. The last section of this book explains why.

Why the Table Doesn't Say "Zero failed requests, ever"

No real system running for hours, across traffic shifts and a possible rollback, can honestly promise that. Claiming it would weaken the one claim that matters: no sustained outage a customer could see.

A few failed requests inside the limit you set in advance is the honest version of "this went well". The standard is controlled and measured, not perfect.

Done When:

• Every line in your table holds a number you measured

• Each line is checked against your Chapter 6 limits, including any that failed

Chapter 25. Run on GKE and Set a Date for AWS

Friday, 10:00 a.m.

Maya comes back from her meeting with a question from the leaders.

Maya: "So what changed for the customer?"

You look at the numbers. Traffic is 100 percent GKE. Transfers succeed. Synthetic checks all pass. PostgreSQL is still on AWS. The EC2 server serves nobody.

"The address stayed the same and nobody saw an outage."

Maya: "Then this system is yours now. And I want a calendar date for turning off the old server."

Step 1. Watch it Run

Let production run for a real window before you commit to anything. For a real company that means days, not the minutes after the last shift.

This lab's 100 percent window lasted minutes, with a 5/5 gate and failed=0. Write that down. A weekend lab isn't a week of production.

Step 2. Check the Alert

Stage 7 created the alert policy ClearPay API - elevated error rate, which fires on more than five error lines in five minutes. It has no notification channel, so nobody gets told when it fires. An alert that reaches nobody isn't an alert.

For a real system, add a notification channel in Cloud Monitoring and trigger a test on staging, never by sending errors to customers. For this lab, write down the gap: the policy exists, and nobody gets paged.

Step 3. Review Capacity

Look at the HPA over the window:

kubectl -n production get hpa clearpay-api

This lab stayed at 1 replica and 4 to 7 percent CPU, even at 15.5 requests per second during the 100 percent load test. One pod was enough. Three e2-medium machines for one API pod is more than this traffic needs.

Google offers Committed Use Discounts (CUDs): a lower price in exchange for promising to use a set amount of machine capacity for one or three years.

Commit only after you have a stable usage history. A commitment based on the architecture you planned, instead of the usage you measured, is a guess with a contract attached. This lab bought none, because a few hours of GKE isn't a usage history.

Step 4. Write the Handover

Write this for someone who wasn't part of the migration. This lab's version:

Logs:       kubectl -n production logs deployment/clearpay-api
Dashboard:  http://localhost:3000 if observability/docker-compose.yml
            is up. GKE series may be empty.
Alert:      Cloud Monitoring "ClearPay API - elevated error rate"
            No notification channel.
Rollback:   scripts/rollback.sh, only while EC2 still exists.
Hostname:   terraform output public_hostname
            terraform output static_ip_address
            (yours, not a hostname copied from this book)

Step 5. Put a Date on AWS

scripts/rollback.sh works only while the EC2 server exists, and the server bills every day it exists. Decide how long it stays, write down why, and price it:

AWS daily cost:              ~$1.25/day (infra/aws/COST.md)
Retention window:            ___ days
Decommission date:           ___
Total safety-margin cost:    ___ days × $1.25 = $___

This lab used 0 extra days and retired AWS compute the same day 100 percent GKE was proven, because it's a teardown lab.

A real company might keep AWS for one or two weeks. A shorter window saves money. A longer one buys more confidence. The mistake is leaving it open as "keep it around just in case", which costs money every day and never gets revisited.

Write the decommission rules before the date arrives, so the decision doesn't get made under deadline pressure:

• a set number of days at 100 percent GKE with no rollback

• no alert fired for a GKE-caused problem in that window

• a named person signs off

This lab's rules: weights 0 / 100, gate 5/5, load test failed=0, HPA at 1, no rollback.

Last, check that nothing else still points at the Elastic IP directly: bookmarks, hardcoded URLs in other systems, or outside uptime monitors set up before the load balancer existed.

Done When:

• You watched production for a written, real time window

• The alert gap is written down

• Capacity is reviewed and the CUD decision is written down

• The handover note exists

• The retention window, decommission date, cost, and rules are written down

Chapter 26. Retire the AWS Server

Friday, 3:00 p.m.

The date has arrived, and the rules from Chapter 25 are met. Remove the EC2 server and keep the database. It was never part of this migration.

Steps

Perform your final check, from the repository root. GKE must pass the gate on the public address:

HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
./scripts/promotion-gate.sh "https://$HOST" \
  alice@clearpay.dev password123 bob@clearpay.dev

Then confirm that Terraform tracks the server you mean to remove:

cd infra/aws/terraform
terraform state show aws_instance.api   # confirm you have the right resource

Next, return to the repository root with cd ../../.., then remove the AWS compute resources only:

./scripts/teardown-aws-compute.sh
# Pass --yes to add -auto-approve after you have read a plan once.

After this step, rollback to AWS is gone. That's the point of the decision you made in Chapter 25.

Read the Destroy Plan

The script destroys only these:

• the EC2 instance

• the Elastic IP

• the security group rule that let the API reach the database

• the API security group

• the SSH key pair

• the CloudWatch alarm

RDS stays.

The database rule is its own Terraform resource, aws_security_group_rule.db_from_api, and the script removes it first. If that rule lived inside the database security group, AWS would refuse to delete the API security group while the database still referenced it.

A full terraform destroy in infra/aws/terraform would also delete PostgreSQL. Don't use it here.

Clean Up Credentials

• The SSH key pair in AWS is gone with the script. Delete the local private key ~/.ssh/clearpay-deployer if you used it only for this server.

• Remove anything else that pointed at the old server, such as saved hosts or scripts with the old IP.

• Remove any CI/CD secrets or variables that referenced the old server.

Update the Diagram

The dual-cloud diagram from Chapter 7 is now wrong. The new picture is GKE alone behind the load balancer, with the database still on AWS, as Chapter 7 said the project would end.

Done When:

• The gate passed on the public address before the teardown

• AWS compute is destroyed, and RDS still exists

• The local deploy key is removed or kept on purpose

• Your architecture diagram matches what runs

Chapter 27. Tear Down GCP and the Database

Friday, 5:30 p.m.

The office is emptying out. Two clouds are still billing you by the hour.

This is a lab. Once your notes are complete, remove everything that still bills: the Gateway load balancer, the GKE machines, Cloud NAT, the registry, and the RDS database.

Before tearing anything down, complete the migration record in Chapter 28. It asks you to document the final architecture, migration results, performance numbers, and total cost. If you still need evidence from the running system, collect it now. Once you destroy the infrastructure, those live measurements will no longer be available.

Steps

From the repository root:

./scripts/teardown-gcp.sh
# Pass --yes to skip the prompt and -auto-approve each stage.

The script:

1. Deletes the Gateway and the production and staging workloads. Kubernetes creates Google load balancer pieces for the Gateway, and Terraform doesn't know about them. Deleting the Gateway first lets Google remove them. Skip this and Terraform can get stuck on resources it didn't create.

2. Runs terraform destroy in reverse stage order: 08-edge → 07-observability → 06-secrets → 05-identity → 04-registry → 03-gke → 02-network → 01-foundation. Later stages depend on earlier ones, so they come down first.

Type yes at the script's prompt, then read and confirm each Terraform plan.

Namespaces can sit in Terminating for a few minutes. The script waits up to 180 seconds for each delete, then moves on. Destroying the cluster removes them anyway.

Then remove the database. The migration ended with it on AWS on purpose, but a lab shouldn't keep paying about $0.80 per day for it:

cd infra/aws/terraform
terraform destroy

Last, delete the api.clearpay.test line from /etc/hosts.

The book never created or deletes your GCP project. Don't delete a project unless it held only this lab.

Done When:

• kubectl get ns no longer shows staging or production, or the cluster is gone

• Every infra/gcp/terraform/0*-* stage is destroyed

• terraform destroy finished in infra/aws/terraform

• GCP and AWS billing show nothing from this lab still running

This is the checkpoint. The work is over.

Write down what actually happened, including the bill, before you forget.

Chapter 28. Write the Migration Record

Week four, Monday, 9:00 a.m.

Write down what happened, the architecture, the bill, and what you would change, before the details fade.

Part 1. The Architecture Record

Before:

User → nginx (TLS) → systemd-managed Node process → RDS Postgres

One EC2 server, one database, a default network nobody designed, secrets in a plain .env file, no autoscaling, and a deploy process run by hand.

During the migration:

After:

Then one short paragraph for each model:

• Traffic: weighted backends on a global external Application Load Balancer, changed by shift-traffic.sh. Not DNS, because cached DNS answers would have made rollback slow.

• Deploys: GitHub Actions builds, scans, and deploys every commit to staging. Production gets the exact staging image, only after the promotion gate and a reviewer's approval.

• Database: never moved. The same RDS instance from Chapter 4 stayed the single source of truth the whole time.

• Security: Workload Identity for both the pods and the pipeline. No downloaded service account key existed at any point. Secrets come from Secret Manager.

• Observability: the API's own JSON logs and /metrics endpoint, the same on AWS and GKE, plus a log-based error metric in Cloud Logging.

• Rollback: one script that moves the traffic weight and sets a flag on AWS, tested before the first shift.

Part 2. The Postmortem

The four of you go through the notes together. Maya opens with the question everyone half expects.

Maya: "Was Kubernetes the problem?"

"Not really. It did what we told it to do. The thing to watch was the database. Every copy of the API can open ten connections, GKE can run several copies, and AWS was still serving customers at the same time. Add those up and you get close to the database's limit.

On this small cluster we topped out at forty connections against eighty-one, so nothing broke. On a cluster that scales to eight copies, the same sum goes over the limit."

Daniel nods. "That's what I missed at my last company. All the numbers were there, and nobody put them side by side. You did it before any traffic moved."

Amara: "And the payments? Was every transfer charged once?"

"Yes. The idempotency key from Chapter 3 held through every shift. We even replayed a payment at 100 percent GKE, and there was no second charge."

Maya closes her laptop. "So what would we do differently next time?"

Write these sections:

What worked:

• Idempotency keys, built in before GKE existed, made it safe for a customer's retry to land on a different cloud than the first try.

• The staged Terraform meant each failure pointed at one stage.

• The pipeline meant production ran the exact image that passed the gate.

• Rollback was tested in Chapter 20, before it was ever needed.

What went wrong in this lab:

• Local Prometheus never watched GKE, so Chapter 24 used counted checks instead of a dashboard total.

• The alert policy has no notification channel.

• The small node pool couldn't run the pod counts a real company would use, so the connection-limit risk was checked on paper, not seen live.

• GKE's p95 was about four times AWS's.

The root cause: Two reasonable settings, DB_POOL_SIZE=10 and the HPA's maxReplicas, were chosen in different places. Nobody checked their product against the database limit until the migration ran both clouds at once. The fix isn't complicated once you find it. Finding it takes treating Chapter 23 as a real investigation instead of skipping ahead to an answer.

Decisions and their trade-offs:

• GKE Standard over Autopilot: full control of node pools and autoscaling, at the cost of having to understand and size them. This lab paid for three mostly idle machines. Autopilot might have hidden this connection-limit risk inside its own scheduling, for better or worse.

• Two environments instead of five: less to build and pay for, at the cost of staging sharing the production database. Fine at this scale, not for every company.

• A public database endpoint with an IP allowlist instead of a VPN: no setup time and no monthly VPN bill. The cost is that RDS has a public endpoint, even if it accepts only one IP and requires TLS.

• Leaving the database on AWS: a simpler, safer migration, at the cost of every query crossing between clouds.

So what would change in a second migration? Before any traffic moves, compute the worst-case connection count across every environment that shares the database. Compare it with the real limit. Make it a checklist item, not a live discovery.

Part 3. The Cost Postmortem

This lab's version:

Budget: about $25 for five days of GCP usage.

Actual dual-run: about $4.30 total, $1.25 AWS + $3.05 GCP. The GCP $3.05 was covered by free-trial credit, so the actual GCP charge was $0.

The migration stayed on schedule. No incident, rollback, or second shift was needed.

The cluster was over-provisioned: three GKE nodes supported one API pod, with CPU at 4–7%.

Standard GKE was chosen for the teaching value. Autopilot would have been cheaper for this workload.

No CUD was purchased.

Conclusion: The migration stayed well within budget, but the GKE setup used more capacity than the workload needed.

Fill in your own numbers from your cloud bills. This book can't do that part for you, and a guessed number would be the same false precision this whole project avoided.

Done When:

• The before, during, and after diagrams match what ran

• The postmortem states the root cause clearly

• The cost postmortem holds your real numbers

What Comes After This Project

What Question This Project Leaves Open

Chapter 13 flagged it and moved on, on purpose: at what point does keeping PostgreSQL in AWS cost more, in risk and in engineering time, than migrating it? The cross-cloud trip, the dependency on two providers, and the connection-pool pressure from Chapter 23 all sit on the same side of that scale.

Put two numbers from your notes side by side:

AWS, Chapter 6:             p95 164 ms
GKE at 100%, Chapter 24:    p95 647 ms

The API moved and customers never saw an outage. But every request got about four times slower.

The code didn't change. The load test did not change. What changed is distance. On AWS, the API and the database sat in the same data center. On GKE, every query leaves Google's us-east4, goes out through Cloud NAT, crosses the internet to AWS's us-east-1, and comes back. A single transfer runs several queries, so it pays that trip several times. The Google load balancer adds one more hop in front.

Maya reads the table. "Why is it slower?"

"The database stayed on AWS. Every query crosses between clouds."

"Was that a mistake?"

"No. Chapter 7 made the call to move the API alone, because moving two things at once would have made any failure much harder to trace. A problem could have come from the API, the network, or the database move itself. But the call has a price, and now we can measure it."

Maya: "Then that's the next project."

This project doesn't answer the open question. It was never supposed to. The next project moves PostgreSQL to Google Cloud, most likely with Google's Database Migration Service, which copies data continuously so the final switch takes minutes instead of hours. It will use the same method as this book: a baseline first, a written limit, small steps, a tested rollback, and evidence at every stage. When it's done, run the same load test and compare the same two numbers.

Again, that's a separate project, with its own book. This one ends here: ClearPay's API on GKE, its database still on AWS, exactly as Chapter 7 said it would.

What You Should Have at the End

• Working fintech API and minimal customer UI (api/, ui/)

• AWS deployment (infra/aws/)

• GCP architecture, staged Terraform (infra/gcp/terraform/)

• Docker images and a CI/CD pipeline that builds, scans, deploys staging, and promotes the tested image (.github/workflows/deploy.yml)

• Kubernetes manifests, two-environment Kustomize overlays (k8s/base/, k8s/overlays/)

• Database connectivity, validated and never migrated (Chapter 13)

• Secret management, upgraded from a .env file to Secret Manager (k8s/overlays/*/secretproviderclass.yaml, secretsync.yaml)

• A GKE Gateway behind a weighted load balancer (k8s/gateway/, infra/gcp/terraform/08-edge/)

• Observability dashboard on your laptop, if you did the optional Chapter 17 (observability/)

• Load-testing, promotion-gate, and synthetic health scripts (scripts/)

• Migration scorecard (Chapter 6, Chapter 18, Chapter 24)

• Cost ledger, start to finish (Chapter 6 through Chapter 28)

• Traffic-shifting procedure (scripts/shift-traffic.sh)

• Rollback procedure (scripts/rollback.sh, scripts/clear-rollback-flag.sh)

• Architecture diagrams (Chapters 4, 7, 9, 11, 13, 14, 19, 28)

• Migration runbook (this book)

• Technical postmortem (Chapter 28)

• Cost postmortem (Chapter 28)

The ClearPay API moved from one AWS EC2 server to GKE. PostgreSQL is still on AWS, on purpose. Customers kept one address while you shifted traffic, practiced a rollback, and finished the cutover.

Maya asked what changed for the customer. The address didn't. The balance didn't. Nobody saw an outage. Now you understand why that was the hard part, and why 647 milliseconds is the next one.

The lab covered how to prove each stage, how to read evidence, and how to undo a bad shift. Destroy anything you're still paying for.

Subscribe to my newsletter for new technical articles, project walkthroughs, and upcoming releases.

How to Discuss This Project in an Interview

I worked on a cloud migration project for a payments application called ClearPay. The application was running on AWS, and the goal was to move it to Google Cloud without disrupting access to the service.

The main challenge was making that move safely. Instead of switching everything at once, I designed a controlled migration where traffic could be moved gradually to the new environment. If something went wrong, I could redirect traffic back to the original system. I also kept the database on AWS to reduce the scope of the migration and avoid introducing unnecessary risk.

I set up the deployment process, security checks, and verification steps needed to confirm that the application was working correctly before completing the switch.

I then tested the final migration. The tests completed without failed requests, and the application didn't produce server errors during the final checks. I also identified a performance difference between the old and new environments that would need further investigation before considering the migration fully optimized.

The project demonstrates that I can plan and execute a cloud migration, manage the risks involved, verify the outcome, and identify issues that still need attention rather than simply declaring success because the application is running.

Interview tip: Be prepared to explain why you used weighted traffic routing, how you validated the application during cutover, what triggered a rollback, and why PostgreSQL remained on AWS.

This guide is designed to end that cycle by replacing it with something more useful: a systematic, service-by-service approach to knowing exactly what you're spending, why you're spending it, catching problems before they become invoices, and reducing costs without guesswork.

It's organised as a reference you can return to. Each part is complete on its own: you can go straight to the RDS section if that's your current problem, or follow the guide start to finish if you're building a FinOps practice from scratch. Every command is runnable, and every script is deployable.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: Monitoring — Know Where Every Dollar Goes

• Part 2: Alerting — Catch Spikes Before They Become Invoices

• Part 3: Optimisation by Service

• Part 4: The 30-Day Optimisation Sprint

• Best Practices Summary

• Resources

What You'll Learn

• How to set up Cost and Usage Report querying with Athena, the foundation of all serious cost analysis

• The five dashboards every engineering team needs, built with queries you can run today

• A three-tier alerting strategy that catches cost spikes without creating alert fatigue

• Service-specific optimisation playbooks for EC2, S3, Lambda, RDS, DynamoDB, and data transfer

• A concrete 30-day sprint that produces measurable savings in the first month

Let's build this from the ground up.

Prerequisites

Before following this guide, you should have:

Knowledge:

• Working familiarity with AWS services: EC2, S3, RDS, Lambda, and VPC

• Comfort reading Python and SQL

• Basic understanding of how IAM policies and roles work

Access:

• AWS account with billing access. The IAM user or role you work with needs ce:GetCostAndUsage, ec2:Describe*, rds:Describe*, and s3:GetBucketLifecycleConfiguration permissions.

• AWS CLI v2 configured

• Athena access (for the CUR queries in Part 1)

Setup:

• Enable Cost Explorer if it isn't already. It's free and required for most commands in this guide:

aws ce enable-cost-explorer --region us-east-1

• The Cost and Usage Report (CUR) should be configured and exporting to an S3 bucket. If it isn't yet, the AWS CUR setup guide walks through the process. Give the report 24 hours after setup to generate the first data file.

Part 1: Monitoring — Know Where Every Dollar Goes

1.1 The Cost and Usage Report — Your Source of Truth

Cost Explorer shows you service-level totals. It's useful for trends, but it isn't sufficient for root cause analysis. When you need to know which specific resource is responsible for a $12,000/month line item, you need the Cost and Usage Report queried through Athena.

Create the Athena table over your CUR data:

-- Run this once in Athena after your CUR starts generating data
-- Replace 'your-cur-bucket' and 'your-prefix' with your actual values

CREATE EXTERNAL TABLE IF NOT EXISTS cur_database.billing (
    bill_billing_period_start_date  STRING,
    bill_payer_account_id           STRING,
    line_item_usage_start_date      STRING,
    line_item_resource_id           STRING,
    line_item_usage_type            STRING,
    line_item_usage_amount          DOUBLE,
    line_item_unblended_cost        DOUBLE,
    product_servicecode             STRING,
    product_instance_type           STRING,
    product_region                  STRING,
    resource_tags_user_environment  STRING,
    resource_tags_user_team         STRING,
    resource_tags_user_service      STRING,
    resource_tags_user_owner        STRING
)
PARTITIONED BY (year STRING, month STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LOCATION 's3://your-cur-bucket/your-prefix/'
TBLPROPERTIES ('skip.header.line.count'='1');

MSCK REPAIR TABLE cur_database.billing;

Here are the three queries you should run on day one:

-- Query 1: Top 20 resources by cost this month
-- Run this first. It tells you where to focus.
SELECT
    line_item_resource_id,
    product_servicecode,
    resource_tags_user_team     AS team,
    resource_tags_user_service  AS service,
    SUM(line_item_unblended_cost) AS total_cost_usd
FROM cur_database.billing
WHERE line_item_usage_start_date >= DATE_FORMAT(
    DATE_TRUNC('month', CURRENT_DATE), '%Y-%m-%d'
)
  AND line_item_unblended_cost > 0
GROUP BY 1, 2, 3, 4
ORDER BY total_cost_usd DESC
LIMIT 20;

-- Query 2: Week-over-week cost growth by service
-- Identifies which services are growing fastest — these need investigation
WITH weekly AS (
    SELECT
        DATE_TRUNC('week', CAST(line_item_usage_start_date AS DATE)) AS week,
        product_servicecode,
        SUM(line_item_unblended_cost) AS cost
    FROM cur_database.billing
    WHERE line_item_usage_start_date >=
          DATE_FORMAT(DATE_ADD('day', -42, CURRENT_DATE), '%Y-%m-%d')
    GROUP BY 1, 2
)
SELECT
    curr.product_servicecode AS service,
    ROUND(prev.cost, 2)      AS prev_week_cost,
    ROUND(curr.cost, 2)      AS curr_week_cost,
    ROUND(
        100.0 * (curr.cost - prev.cost) / NULLIF(prev.cost, 0),
        1
    ) AS pct_change
FROM weekly curr
JOIN weekly prev
  ON curr.product_servicecode = prev.product_servicecode
  AND curr.week = DATE_ADD('week', 1, prev.week)
WHERE curr.week = DATE_TRUNC('week', CURRENT_DATE)
  AND ABS((curr.cost - prev.cost) / NULLIF(prev.cost, 0)) > 0.20
ORDER BY pct_change DESC;

-- Query 3: Resources running with no usage (candidates for shutdown)
-- Finds resources that incurred cost but had zero usage quantity
-- in the past 7 days — strong signal for idle or orphaned resources
SELECT
    line_item_resource_id,
    product_servicecode,
    resource_tags_user_owner    AS owner,
    resource_tags_user_team     AS team,
    SUM(line_item_unblended_cost) AS cost_past_7_days
FROM cur_database.billing
WHERE line_item_usage_start_date >=
      DATE_FORMAT(DATE_ADD('day', -7, CURRENT_DATE), '%Y-%m-%d')
  AND line_item_usage_amount = 0
  AND line_item_unblended_cost > 5
GROUP BY 1, 2, 3, 4
ORDER BY cost_past_7_days DESC
LIMIT 30;

1.2 The Five Essential Cost Dashboards

These five views cover the monitoring needs of most engineering teams. Each is built from queries you can run immediately, no third-party tool required.

Dashboard 1: Executive Summary (for weekly leadership updates)

# executive_summary.py
import boto3
from datetime import datetime, timedelta

ce = boto3.client('ce')


def weekly_summary():
    today     = datetime.now()
    start_mtd = today.replace(day=1).strftime('%Y-%m-%d')
    today_str = today.strftime('%Y-%m-%d')

    # Month-to-date spend
    mtd = ce.get_cost_and_usage(
        TimePeriod={'Start': start_mtd, 'End': today_str},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )
    mtd_spend = float(
        mtd['ResultsByTime'][0]['Total']['UnblendedCost']['Amount']
    )

    # End-of-month forecast
    forecast = ce.get_cost_forecast(
        TimePeriod={
            'Start': today_str,
            'End':   (today.replace(day=28) + timedelta(days=4)).replace(day=1).strftime('%Y-%m-%d'),
        },
        Metric='UNBLENDED_COST',
        Granularity='MONTHLY'
    )
    eom_forecast = float(forecast['Total']['Amount']) + mtd_spend

    # Top 5 services
    by_service = ce.get_cost_and_usage(
        TimePeriod={'Start': start_mtd, 'End': today_str},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[{'Type': 'DIMENSION', 'Key': 'SERVICE'}]
    )
    services = sorted(
        [
            (g['Keys'][0], float(g['Metrics']['UnblendedCost']['Amount']))
            for g in by_service['ResultsByTime'][0]['Groups']
        ],
        key=lambda x: x[1],
        reverse=True
    )[:5]

    print(f"\n{'─'*48}")
    print(f"  AWS Cost Summary — {today.strftime('%B %Y')}")
    print(f"{'─'*48}")
    print(f"  Month-to-date:     ${mtd_spend:>12,.2f}")
    print(f"  End-of-month est:  ${eom_forecast:>12,.2f}")
    print(f"\n  Top 5 Services:")
    for name, cost in services:
        short = name.replace('Amazon ', '').replace('AWS ', '')
        print(f"    {short:<32} ${cost:>9,.2f}")
    print(f"{'─'*48}\n")


weekly_summary()

Dashboard 2: Team Cost Breakdown (for engineering leads)

# team_breakdown.py
import boto3
from datetime import datetime

ce = boto3.client('ce')


def team_breakdown():
    start = datetime.now().replace(day=1).strftime('%Y-%m-%d')
    end   = datetime.now().strftime('%Y-%m-%d')

    response = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[
            {'Type': 'TAG',       'Key': 'Team'},
            {'Type': 'DIMENSION', 'Key': 'SERVICE'},
        ]
    )

    by_team = {}
    for group in response['ResultsByTime'][0].get('Groups', []):
        team_raw = group['Keys'][0]
        team     = team_raw.replace('Team$', '') if team_raw else 'untagged'
        service  = group['Keys'][1]
        cost     = float(group['Metrics']['UnblendedCost']['Amount'])

        if team not in by_team:
            by_team[team] = {'total': 0.0, 'by_service': {}}
        by_team[team]['total'] += cost
        by_team[team]['by_service'][service] = (
            by_team[team]['by_service'].get(service, 0.0) + cost
        )

    total_bill = sum(d['total'] for d in by_team.values())

    print(f"\n{'─'*58}")
    print(f"  Team Cost Breakdown — MTD {datetime.now().strftime('%Y-%m-%d')}")
    print(f"  Total: ${total_bill:,.2f}")
    print(f"{'─'*58}")

    for team, data in sorted(by_team.items(), key=lambda x: x[1]['total'], reverse=True):
        pct = (data['total'] / total_bill * 100) if total_bill else 0
        print(f"\n  {team:<20}  ${data['total']:>10,.2f}  ({pct:.1f}%)")
        top3 = sorted(data['by_service'].items(), key=lambda x: x[1], reverse=True)[:3]
        for svc, cost in top3:
            short = svc.replace('Amazon ', '').replace('AWS ', '')
            print(f"    └─ {short:<30} ${cost:>8,.2f}")

    print()


team_breakdown()

Dashboard 3: Waste Detection (for weekly cleanup reviews):

# waste_detector.py
import boto3
from datetime import datetime, timezone, timedelta

ec2  = boto3.client('ec2')
elbv2 = boto3.client('elbv2')
cw   = boto3.client('cloudwatch')


def detect_waste():
    report = {'items': [], 'total_monthly_waste': 0.0}

    # Unattached EBS volumes
    for vol in ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )['Volumes']:
        age  = (datetime.now(timezone.utc) - vol['CreateTime']).days
        cost = round(vol['Size'] * 0.08, 2)
        tags = {t['Key']: t['Value'] for t in vol.get('Tags', [])}
        report['items'].append({
            'type':         'Unattached EBS Volume',
            'id':           vol['VolumeId'],
            'detail':       f"{vol['Size']}GB — {age} days old",
            'owner':        tags.get('Owner', '—'),
            'monthly_cost': cost,
        })
        report['total_monthly_waste'] += cost

    # Unassociated Elastic IPs
    for addr in ec2.describe_addresses()['Addresses']:
        if 'AssociationId' not in addr:
            report['items'].append({
                'type':         'Unassociated Elastic IP',
                'id':           addr.get('AllocationId', ''),
                'detail':       addr['PublicIp'],
                'owner':        '—',
                'monthly_cost': 3.60,
            })
            report['total_monthly_waste'] += 3.60

    # Idle load balancers (fewer than 100 requests in 7 days)
    for lb in elbv2.describe_load_balancers()['LoadBalancers']:
        metrics = cw.get_metric_statistics(
            Namespace='AWS/ApplicationELB',
            MetricName='RequestCount',
            Dimensions=[{'Name': 'LoadBalancer',
                         'Value': lb['LoadBalancerArn'].split(':loadbalancer/')[-1]}],
            StartTime=datetime.now() - timedelta(days=7),
            EndTime=datetime.now(),
            Period=604800,
            Statistics=['Sum']
        )['Datapoints']
        total_requests = metrics[0]['Sum'] if metrics else 0

        if total_requests < 100:
            report['items'].append({
                'type':         'Idle Load Balancer',
                'id':           lb['LoadBalancerName'],
                'detail':       f"{int(total_requests)} requests in 7 days",
                'owner':        '—',
                'monthly_cost': 22.0,
            })
            report['total_monthly_waste'] += 22.0

    print(f"\n  Waste Detection Report — {datetime.now().strftime('%Y-%m-%d')}")
    print(f"  Estimated monthly waste: ${report['total_monthly_waste']:.2f}\n")

    for item in sorted(report['items'], key=lambda x: x['monthly_cost'], reverse=True)[:20]:
        print(f"  [{item['type']}]")
        print(f"    ID:     {item['id']}")
        print(f"    Detail: {item['detail']}")
        print(f"    Owner:  {item['owner']}")
        print(f"    Cost:   ${item['monthly_cost']:.2f}/month\n")

    return report


detect_waste()

1.3 Tagging Strategy — The Foundation of All Attribution

Every cost attribution model depends on tags. Teams that skip tagging build dashboards that show totals without explanations. The discipline is in making tagging structural rather than procedural: enforced by infrastructure code, not by reminders in Confluence.

Required tag set:

# terraform/variables.tf
variable "mandatory_tags" {
  description = "Tags applied to every resource in this account"
  type        = map(string)

  validation {
    condition = alltrue([
      contains(keys(var.mandatory_tags), "Environment"),
      contains(keys(var.mandatory_tags), "Team"),
      contains(keys(var.mandatory_tags), "Owner"),
      contains(keys(var.mandatory_tags), "Service"),
    ])
    error_message = "mandatory_tags must include Environment, Team, Owner, and Service."
  }
}

locals {
  common_tags = merge(var.mandatory_tags, {
    ManagedBy    = "terraform"
    LastModified = timestamp()
  })
}

resource "aws_instance" "api_server" {
  ami           = data.aws_ami.amazon_linux_2023.id
  instance_type = "t3.medium"
  tags          = merge(local.common_tags, {Name = "api-server-${var.environment}"})
}

Find and report untagged resources weekly:

#!/usr/bin/env bash
# find_untagged.sh

echo "Untagged EC2 instances (missing Team tag):"
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[?!not_null(Tags[?Key=='Team'].Value|[0])].[InstanceId,InstanceType,LaunchTime]" \
  --output table

echo "Untagged RDS instances:"
aws rds describe-db-instances \
  --query "DBInstances[?!not_null(TagList[?Key=='Team'].Value|[0])].DBInstanceIdentifier" \
  --output table

Part 2: Alerting — Catch Spikes Before They Become Invoices

The typical discovery timeline without proactive alerting: a cost spike happens on the 5th, the monthly invoice arrives on the 20th, someone notices on the 22nd, investigation begins on the 23rd, and two weeks of billed waste can't be recovered. With proactive alerting, discovery happens within hours.

2.1 The Three-Tier Alert Structure

Alert fatigue is as damaging as no alerting. The three-tier model keeps signal high by routing different severity levels to different channels with different response expectations.

# alert_router.py
import boto3
import json
import urllib.request
from enum import Enum

SLACK_INFO_WEBHOOK  = 'https://hooks.slack.com/services/INFO/WEBHOOK'
SLACK_ALERT_WEBHOOK = 'https://hooks.slack.com/services/ALERT/WEBHOOK'
SNS_CRITICAL_TOPIC  = 'arn:aws:sns:us-east-1:YOUR_ACCOUNT:cost-critical'


class AlertTier(Enum):
    INFO     = 1
    WARNING  = 2
    CRITICAL = 3


def route_alert(tier: AlertTier, subject: str, message: str):
    """Send an alert to the appropriate channel for its severity tier."""
    icons   = {AlertTier.INFO: ':information_source:',
               AlertTier.WARNING: ':warning:', AlertTier.CRITICAL: ':rotating_light:'}
    payload = {'text': f"{icons[tier]} *{subject}*\n{message}"}

    if tier == AlertTier.INFO:
        _post_slack(SLACK_INFO_WEBHOOK, payload)
    elif tier == AlertTier.WARNING:
        _post_slack(SLACK_ALERT_WEBHOOK, payload)
        _send_sns(SNS_CRITICAL_TOPIC, subject, f"WARNING: {message}")
    elif tier == AlertTier.CRITICAL:
        _post_slack(SLACK_ALERT_WEBHOOK, payload)
        _send_sns(SNS_CRITICAL_TOPIC, subject, f"CRITICAL: {message}")


def _post_slack(webhook: str, payload: dict):
    req = urllib.request.Request(
        webhook,
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json'}
    )
    urllib.request.urlopen(req)


def _send_sns(topic_arn: str, subject: str, message: str):
    sns = boto3.client('sns')
    sns.publish(TopicArn=topic_arn, Subject=subject[:100], Message=message)

The three tiers and what they respond to: Tier 1 INFO goes to a Slack informational channel for daily cost summaries, weekly trend reports, and tag compliance updates. No action required.

Tier 2 WARNING goes to a Slack alert channel plus email for budget above 75% utilisation, 25% week-over-week increases, and expiring Savings Plans. Acknowledge within 24 hours.

Tier 3 CRITICAL goes to PagerDuty plus SMS for budget above 90% utilisation, 100% increase in 24 hours, crypto mining detected, and projected overspend above 120% of plan. Investigate within 1 hour.

2.2 Real-Time Budget Monitor

AWS Budgets sends alerts once daily by default. A daily window means a cost spike that begins at 08:00 isn't caught until the next day's alert fires. The Lambda below runs hourly and checks both absolute budget utilisation and hour-over-hour rate of change.

# budget_monitor.py
# Lambda triggered by EventBridge every hour

import boto3
from datetime import datetime, timedelta
from alert_router import route_alert, AlertTier

ce      = boto3.client('ce')
budgets = boto3.client('budgets', region_name='us-east-1')
ACCOUNT_ID  = boto3.client('sts').get_caller_identity()['Account']
BUDGET_NAME = 'monthly-infrastructure'


def get_mtd_spend() -> float:
    start = datetime.now().replace(day=1).strftime('%Y-%m-%d')
    end   = datetime.now().strftime('%Y-%m-%d')
    r = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )
    return float(r['ResultsByTime'][0]['Total']['UnblendedCost']['Amount'])


def get_budget_limit() -> float:
    r = budgets.describe_budget(AccountId=ACCOUNT_ID, BudgetName=BUDGET_NAME)
    return float(r['Budget']['BudgetLimit']['Amount'])


def get_hourly_costs(hours: int = 4) -> list:
    """Return hourly cost totals for the last N hours."""
    end   = datetime.now()
    start = end - timedelta(hours=hours)
    r = ce.get_cost_and_usage(
        TimePeriod={'Start': start.strftime('%Y-%m-%d'), 'End': end.strftime('%Y-%m-%d')},
        Granularity='HOURLY',
        Metrics=['UnblendedCost']
    )
    return [
        float(period['Total']['UnblendedCost']['Amount'])
        for period in r['ResultsByTime']
    ]


def lambda_handler(event, context):
    mtd_spend    = get_mtd_spend()
    budget_limit = get_budget_limit()
    utilisation  = mtd_spend / budget_limit * 100

    days_elapsed  = datetime.now().day
    projected_eom = (mtd_spend / days_elapsed) * 30
    projected_pct = projected_eom / budget_limit * 100

    if utilisation >= 90:
        route_alert(
            AlertTier.CRITICAL,
            f'Budget at {utilisation:.0f}%',
            f'MTD spend ${mtd_spend:,.2f} is {utilisation:.0f}% of ${budget_limit:,.0f} budget. '
            f'Projected EOM: ${projected_eom:,.2f}.'
        )
    elif utilisation >= 75:
        route_alert(
            AlertTier.WARNING,
            f'Budget at {utilisation:.0f}%',
            f'MTD spend ${mtd_spend:,.2f} is {utilisation:.0f}% of ${budget_limit:,.0f} budget. '
            f'Projected EOM: ${projected_eom:,.2f}.'
        )

    # Check hourly spike
    hourly = get_hourly_costs(hours=4)
    if len(hourly) >= 2:
        last_hour = hourly[-1]
        prev_avg  = sum(hourly[:-1]) / len(hourly[:-1])
        if prev_avg > 0.10 and last_hour > prev_avg * 1.5:
            route_alert(
                AlertTier.WARNING,
                'Hourly cost spike detected',
                f'Last hour: ${last_hour:.2f} vs prior 3-hour avg ${prev_avg:.2f} '
                f'(+{(last_hour/prev_avg - 1)*100:.0f}%)'
            )

    return {
        'mtd_spend':       round(mtd_spend, 2),
        'utilisation_pct': round(utilisation, 1),
        'projected_eom':   round(projected_eom, 2),
    }

Part 3: Optimisation by Service

3.1 EC2 — Seven Levers in Priority Order

EC2 is the largest line item in most AWS accounts and the one with the most optimisation options. Work through these levers in order, as each one lowers the baseline that the next lever acts on.

Lever 1: Find truly idle instances (CPU below 1% for 14 days).

# ec2_idle_finder.py
import boto3
from datetime import datetime, timedelta

ec2 = boto3.client('ec2')
cw  = boto3.client('cloudwatch')


def find_idle_instances(avg_cpu_threshold: float = 1.0, days: int = 14):
    instances = [
        inst
        for r in ec2.describe_instances(
            Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
        )['Reservations']
        for inst in r['Instances']
    ]

    idle = []
    for inst in instances:
        iid   = inst['InstanceId']
        stats = cw.get_metric_statistics(
            Namespace='AWS/EC2',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'InstanceId', 'Value': iid}],
            StartTime=datetime.utcnow() - timedelta(days=days),
            EndTime=datetime.utcnow(),
            Period=days * 86400,
            Statistics=['Average']
        )['Datapoints']

        avg_cpu = stats[0]['Average'] if stats else 0.0
        if avg_cpu < avg_cpu_threshold:
            tags = {t['Key']: t['Value'] for t in inst.get('Tags', [])}
            idle.append({
                'instance_id':   iid,
                'instance_type': inst['InstanceType'],
                'avg_cpu':       round(avg_cpu, 2),
                'environment':   tags.get('Environment', '—'),
                'owner':         tags.get('Owner', '—'),
            })

    return sorted(idle, key=lambda x: x['avg_cpu'])


for inst in find_idle_instances():
    print(f"  {inst['instance_id']}  {inst['instance_type']}  "
          f"{inst['avg_cpu']}% CPU  env:{inst['environment']}  owner:{inst['owner']}")

Lever 2: Right-size over-provisioned instances (CPU below 20%, sustained).

Use the same script with avg_cpu_threshold=20.0. These are right-sizing candidates, not shutdown candidates.

Lever 3: Schedule dev and staging shutdowns using EventBridge rules on AutoShutdown=true tagged instances.

Lever 4: Purchase Savings Plans only after completing levers 1–3.

Lever 5: Migrate to Graviton (20% cheaper, same performance for most workloads).

Lever 6: Use Spot for fault-tolerant batch and development workloads.

Lever 7: Migrate containerised workloads to EKS with Karpenter for automatic bin-packing.

Spot savings estimate:

# spot_price_analyser.py
import boto3

ec2 = boto3.client('ec2')


def spot_savings_estimate(instance_type: str) -> dict:
    spot_history = ec2.describe_spot_price_history(
        InstanceTypes=[instance_type],
        ProductDescriptions=['Linux/UNIX'],
        MaxResults=1
    )['SpotPriceHistory']
    spot_price = float(spot_history[0]['SpotPrice']) if spot_history else 0

    on_demand_approx = {
        't3.medium': 0.0416, 'm5.large': 0.096,
        'c5.xlarge': 0.17,   'r5.2xlarge': 0.504,
    }
    od_price    = on_demand_approx.get(instance_type, 0)
    savings_pct = ((od_price - spot_price) / od_price * 100) if od_price else 0

    return {
        'instance_type': instance_type,
        'spot_price':    round(spot_price, 4),
        'on_demand':     od_price,
        'savings_pct':   round(savings_pct, 1),
        'monthly_spot':  round(spot_price * 730, 2),
        'monthly_od':    round(od_price * 730, 2),
    }


for itype in ['t3.medium', 'm5.large', 'c5.xlarge']:
    r = spot_savings_estimate(itype)
    print(f"  {r['instance_type']:<15} Spot: ${r['spot_price']}/hr  "
          f"OD: ${r['on_demand']}/hr  Savings: {r['savings_pct']}%")

3.2 S3 — Lifecycle Policies and Storage Class Selection

S3 optimisation has two components: moving infrequently accessed data to cheaper storage classes via lifecycle policies, and eliminating waste patterns like incomplete multipart uploads.

# s3_lifecycle_applier.py
import boto3

s3 = boto3.client('s3')

LOG_POLICY = {
    'Rules': [{
        'ID': 'standard-tiering',
        'Status': 'Enabled',
        'Filter': {'Prefix': ''},
        'Transitions': [
            {'Days': 30,  'StorageClass': 'STANDARD_IA'},
            {'Days': 90,  'StorageClass': 'GLACIER_IR'},
            {'Days': 365, 'StorageClass': 'DEEP_ARCHIVE'},
        ],
        'Expiration': {'Days': 2555},
        'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 7},
    }]
}

TEMP_POLICY = {
    'Rules': [{
        'ID': 'temp-data-retention',
        'Status': 'Enabled',
        'Filter': {'Prefix': ''},
        'Expiration': {'Days': 30},
        'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 1},
    }]
}

for bucket in s3.list_buckets()['Buckets']:
    name = bucket['Name']
    try:
        s3.get_bucket_lifecycle_configuration(Bucket=name)
        print(f"  {name} — policy already exists, skipping")
    except s3.exceptions.ClientError:
        policy = TEMP_POLICY if any(k in name for k in ['temp', 'build', 'cache']) else LOG_POLICY
        s3.put_bucket_lifecycle_configuration(
            Bucket=name, LifecycleConfiguration=policy
        )
        print(f"  {name} — applied {'TEMP' if policy is TEMP_POLICY else 'LOG'} policy")

3.3 RDS — Five Optimisation Levels

Find over-provisioned RDS instances:

# rds_rightsizer.py
import boto3
from datetime import datetime, timedelta

rds = boto3.client('rds')
cw  = boto3.client('cloudwatch')


def find_oversized_rds():
    instances  = rds.describe_db_instances()['DBInstances']
    candidates = []

    for inst in instances:
        iid    = inst['DBInstanceIdentifier']
        iclass = inst['DBInstanceClass']

        stats = cw.get_metric_statistics(
            Namespace='AWS/RDS',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'DBInstanceIdentifier', 'Value': iid}],
            StartTime=datetime.utcnow() - timedelta(days=14),
            EndTime=datetime.utcnow(),
            Period=1209600,
            Statistics=['Average', 'Maximum']
        )['Datapoints']

        if not stats:
            continue

        avg_cpu = stats[0]['Average']
        max_cpu = stats[0]['Maximum']

        if avg_cpu < 20 and max_cpu < 50:
            candidates.append({
                'id':       iid,
                'class':    iclass,
                'avg_cpu':  round(avg_cpu, 1),
                'max_cpu':  round(max_cpu, 1),
                'engine':   inst['Engine'],
            })

    return candidates


for c in find_oversized_rds():
    print(f"  {c['id']}  {c['class']}  avg:{c['avg_cpu']}%  max:{c['max_cpu']}%  engine:{c['engine']}")

3.4 DynamoDB — On-Demand vs Provisioned Decision

On-demand is convenient but can be 3–5× more expensive than provisioned for predictable workloads. Provisioned with auto-scaling covers most cases at significantly lower cost.

# dynamodb_mode_advisor.py
import boto3
from datetime import datetime, timedelta

dynamodb = boto3.client('dynamodb')
cw       = boto3.client('cloudwatch')


def analyse_table_billing(table_name: str) -> dict:
    table = dynamodb.describe_table(TableName=table_name)['Table']
    mode  = table.get('BillingModeSummary', {}).get('BillingMode', 'PROVISIONED')

    stats = {}
    for metric in ['ConsumedReadCapacityUnits', 'ConsumedWriteCapacityUnits']:
        data = cw.get_metric_statistics(
            Namespace='AWS/DynamoDB',
            MetricName=metric,
            Dimensions=[{'Name': 'TableName', 'Value': table_name}],
            StartTime=datetime.utcnow() - timedelta(days=30),
            EndTime=datetime.utcnow(),
            Period=86400,
            Statistics=['Average', 'Maximum']
        )['Datapoints']
        if data:
            avg  = sum(d['Average'] for d in data) / len(data)
            peak = max(d['Maximum'] for d in data)
            stats[metric] = {'avg': round(avg, 1), 'peak': round(peak, 1)}

    if mode == 'PAY_PER_REQUEST':
        avg_rcu = stats.get('ConsumedReadCapacityUnits', {}).get('avg', 0)
        avg_wcu = stats.get('ConsumedWriteCapacityUnits', {}).get('avg', 0)

        if avg_rcu > 2000 or avg_wcu > 500:
            return {
                'table':          table_name,
                'current_mode':   'PAY_PER_REQUEST',
                'recommendation': 'Switch to PROVISIONED with auto-scaling',
                'reason': f'Avg {avg_rcu:.0f} RCU/s and {avg_wcu:.0f} WCU/s — predictable pattern',
            }

    return {'table': table_name, 'current_mode': mode, 'recommendation': 'No change needed'}


for table in dynamodb.list_tables()['TableNames']:
    r = analyse_table_billing(table)
    if r['recommendation'] != 'No change needed':
        print(f"  {r['table']}: {r['recommendation']}")

3.5 Data Transfer — The Three Main Waste Patterns

Data transfer charges are often the most confusing line item on an AWS bill. Here are the the three main patterns and their fixes:

Cross-AZ traffic (most common, most fixable): services in different AZs incur $0.01/GB in each direction. The fix is to use topology-aware routing on Kubernetes Services or ensure your application tier and database tier use the same AZ placement.

NAT Gateway charges for internal AWS traffic: S3, ECR, DynamoDB, and SQS traffic that routes through NAT Gateway incurs $0.045/GB. The fix: VPC endpoints eliminate this entirely.

Inter-region replication that compliance doesn't require: audit your S3 replication rules quarterly against actual compliance requirements.

# Find S3 buckets with active replication
for bucket in $(aws s3api list-buckets --query 'Buckets[*].Name' --output text); do
    result=$(aws s3api get-bucket-replication --bucket "$bucket" 2>&1)
    if ! echo "$result" | grep -q "ReplicationConfigurationNotFoundError"; then
        echo "  $bucket — replication active, verify compliance requirement"
    fi
done

Part 4: The 30-Day Optimisation Sprint

This sprint produces measurable savings in the first month. It's designed for a single engineer with two to four hours per week of dedicated FinOps time.

Week 1 – Visibility: enable CUR, set up the Athena table, run the three day-one queries, screenshot the results as your baseline, tag 100% of running EC2 and RDS instances, and identify your top three cost drivers with a documented hypothesis for each.

Week 2 – Quick wins: deploy the orphaned resource reporter Lambda, apply S3 lifecycle policies to your three largest buckets, run the idle instance finder and stop anything below 1% average CPU with no owner objection, and add VPC endpoints for S3, ECR, and DynamoDB.

Week 3 – Right-sizing: run the EC2 rightsizing analyser, downsize the three highest-confidence candidates (non-production first), run the RDS rightsizing script, and find read replicas serving minimal traffic and decommission them.

Week 4 – Alerting and automation: deploy the hourly budget monitor Lambda, configure the three-tier alert routing, set up the weekly waste reporter, add the Infracost GitHub Action to your infrastructure repository, and schedule a monthly 30-minute FinOps review meeting.

Expected outcome after 30 days: 15–25% reduction in monthly AWS spend, documented evidence of every change, and a recurring process that prevents the same waste from accumulating again.

Best Practices Summary

✅ Do: Set up CUR + Athena before any other monitoring. Cost Explorer is a starting point, while CUR is the source of truth.

✅ Do: Enforce tagging in Terraform or CloudFormation. Process-based tagging decays, but infrastructure-enforced tagging is permanent.

✅ Do: Run the idle instance finder and waste reporter weekly. Waste accumulates continuously. A weekly report keeps the pile small.

✅ Do: Use the three-tier alert model. One alert channel with everything in it creates fatigue and gets muted.

✅ Do: Work through the EC2 optimisation levers in order. Right-sizing before Savings Plans prevents locking in waste at a discount.

✅ Do: Check DynamoDB billing mode against actual usage patterns quarterly.

❌ Don't: Delete untagged resources without investigation. Untagged doesn't mean unused, it means unclaimed.

❌ Don't: Apply aggressive S3 lifecycle policies without auditing access patterns first. Glacier retrieval fees can exceed Standard storage costs if data is accessed more frequently than expected.

❌ Don't: Run the waste reporter Lambda with auto-deletion enabled on its first deployment. Run in report-only mode for two weeks to validate the output before adding deletion logic.

Resources

• AWS Cost and Usage Report Data Dictionary: Column reference for all CUR Athena queries in this guide

• AWS Cost Explorer API Reference: Full reference for the Python boto3 cost queries

• AWS Compute Optimizer: ML-powered right-sizing recommendations, useful as a cross-check against the manual analyser scripts

• Amazon DynamoDB Pricing: The definitive reference for the provisioned vs on-demand cost calculation in Section 3.4

• AWS Instance Scheduler: The official AWS solution for tag-based EC2 and RDS scheduling

• FinOps Foundation Framework: The practitioner framework that defines the Inform, Optimise, Operate cycle this guide implements

• Companion Repository: All scripts, Lambda functions, and Terraform modules from this guide

Among these metrics, histograms are particularly useful. They show how often values fall into different ranges, like HTTP request durations, database query times, or queue processing latencies.

Unlike simple averages, histograms show the full picture: you can see how many requests are fast, how many are slow, and where the occasional outliers occur that might be silently degrading the user experience.

In payment systems, for instance, a sudden spike in transactions can expose hidden bottlenecks. Most of the requests might complete quickly, but a small percentage of slow transactions can ripple through the system, impacting retries, failures, and overall throughput. Histograms help identify these issues early by showing how values are distributed and highlighting outliers that averages obscure.

But not all backends understand Prometheus metrics natively. Many modern observability platforms prefer OTLP (OpenTelemetry Protocol). Forwarding Prometheus metrics without converting them can lead to incomplete or misinterpreted data. That’s why we need a pipline to scrape, transform, and export histograms into OTLP so that your observability pipeline remains consistent and actionable.

In this article, we'll use the OpenTelemetry Collector to scrape Prometheus histograms from application /metrics endpoints, map them to the OpenTelemetry Histogram data model, and export them to our observability backend using OTLP. The Collector acts as a bridge that preserves data fidelity while ensuring compatibility with your monitoring platform.

To make this concrete, we'll use a small FastAPI application that simulates payment transactions. It exposes two Prometheus metrics: payment_transaction_duration_seconds (a histogram tracking how long each transaction takes) and payment_transactions_total (a counter of completed transactions). You can follow along with your own instrumented application, as anything exposing Prometheus metrics at /metrics will work the same way. This is the metric we'll follow from the application all the way to the observability backend.

What We'll Cover:

• Prerequisites

• 1. How to Scrape Metrics with Prometheus Receiver

• 2. Transforming Prometheus Histograms

• 3. Exporting Metrics via OTLP

• 4. Putting the Pipeline Together

• 5. Running the OpenTelemetry Collector

  • 5.1 Set Up SigNoz Cloud

  • 5.2 Start the FastAPI Application

  • 5.3 Start the Collector

  • 5.4 Generate Test Transactions

  • 5.5 Confirm Backend Receipt

  • 5.6 Troubleshooting

• Conclusion

Prerequisites

Before you begin, make sure you have:

• Docker installed

• An application exposing Prometheus metrics through a /metrics endpoint

• An OTLP-compatible observability backend

• Basic knowledge of Prometheus metrics

• Basic knowledge of YAML

• Basic familiarity with Docker and OpenTelemetry

You don't need advanced OpenTelemetry knowledge to follow this tutorial. I'll walk through the Prometheus histogram and show what happens to it as it moves through the OpenTelemetry Collector.

1. How to Scrape Metrics with Prometheus Receiver

First, we'll collect metrics from the application. The demo application exposes its Prometheus metrics at the /metrics endpoint, and the Prometheus receiver periodically scrapes this endpoint and ingests the metrics into the observability pipeline. You can also inspect /metrics directly to see the data before the Collector reads it.

Configuration setup example:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: payment-demo
          scrape_interval: 15s
          static_configs:
            - targets: ["payment-api:8080"]

What's happening behind the scenes here:

scrape_interval controls how often the Collector scrapes the target. Here, we're using 15 seconds; adjust it based on how frequently you need metric updates and the load your application can handle.

In high-throughput systems like payment platforms or real-time processing services, the scrape interval becomes particularly important. Setting it too long may cause you to miss short-lived performance issues or transient errors, while setting it too short risks overwhelming the service with scraping requests or generating excessive network traffic.

targets lists the specific endpoints that expose Prometheus metrics. You can add multiple targets when you need to scrape more than one service.

Finally, job_name is an identifier that helps group metrics logically, making them easier to manage and analyze once they reach your backend.

2. Transforming Prometheus Histograms

Once the metrics are scraped, the next step is transformation. Prometheus exposes histograms as multiple time series:

• _bucket shows how many requests fall below a certain duration.

• _sum is the total of all observed durations.

• _count is the number of observations.

The payment_transaction_duration_seconds histogram records how long each payment transaction takes. Prometheus exposes it as _bucket, _count, and _sum series. When the Prometheus receiver collects these metrics, it converts them into the OpenTelemetry Histogram data model, which can then be exported through OTLP while preserving the information needed to analyze transaction latency and calculate percentiles.

Without histograms, averages obscure latency distributions. If most requests complete in 50ms but 5% take 2+ seconds, the average of 150ms masks that serious performance issue. Histograms capture the complete picture by recording how many observations fall into each latency bucket.

Keeping the distribution allows you to identify changes in latency and investigate issues such as slow database queries, overloaded services, or delays from downstream dependencies.

Actual Prometheus Histogram

This is how your actual /metrics output would look, for example:

And here's the actual OpenTelemetry Histogram representation:

The output above shows the same transaction-duration data represented as an OpenTelemetry Histogram. Instead of three separate Prometheus series, the Collector now has one histogram containing the count, sum, explicit bucket boundaries, and bucket counts.

3. Exporting Metrics via OTLP

Once the metrics have been scraped and processed, the Collector sends them to your observability backend using OTLP. The OTLP exporter sends the processed metrics, including histograms, counters, and gauges, to the backend.

exporters:
  otlp:
    endpoint: "otlp.backend.example.com:4317"
    tls:
      insecure: false

endpoint specifies the backend address for receiving OTLP metrics. This typically points to a central observability platform that aggregates metrics from multiple services across your infrastructure.

tls ensures secure data transmission between the collector and your backend. Set insecure: true only when intentionally connecting to an endpoint that does not use TLS, such as some local development setups

4. Putting the Pipeline Together

We've looked at each part of the pipeline individually. Now let's connect them and follow a metric from the application all the way to the backend.

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp]

This is the complete path our payment_transaction_duration_seconds metric follows, from the FastAPI application to the observability backend. The diagram above shows this flow. The next section walks through actually running it and connecting to SigNoz

Pipeline flow:

The above architecture diagram shows a FastAPI application exposing metrics through /metrics, the OpenTelemetry Collector scraping them with the Prometheus receiver, processing the metrics, and exporting them through OTLP to an observability backend.

5. Running the OpenTelemetry Collector

We'll run the complete pipeline and verify that the transaction metrics make it from the application to SigNoz. Follow these steps (which I'll walk you through in detail below):

1. Set up SigNoz Cloud: Configure the endpoint and ingestion key that the Collector will use.

2. Start the FastAPI application: The application exposes the Prometheus metrics at /metrics.

3. Start the OpenTelemetry Collector: The Collector begins scraping the application's /metrics endpoint using the Prometheus receiver.

4. Generate test transactions: Send requests to the application to create transaction-duration measurements.

5. Verify the Collector and backend: Check the Collector logs to confirm that the pipeline is running, then open SigNoz and verify that the payment_transaction_duration_seconds metric has arrived.

6. Troubleshooting

5.1 Set Up SigNoz Cloud

SigNoz provides the observability backend that will receive the metrics exported by the OpenTelemetry Collector. For this demo, we'll use SigNoz Cloud as the observability backend, so there's nothing to install locally.

First, sign up for a free account at signoz.io. In the SigNoz Cloud dashboard, go to Settings → Ingestion Keys. The page shows your Ingestion URL, Region, and Ingestion Key.

Add these to a .env file in your project directory:

SIGNOZ_INGESTION_KEY=your-real-key
SIGNOZ_OTLP_ENDPOINT=ingest.<your-region>.signoz.cloud:443

Treat the ingestion key like a password and never commit it or share it publicly.

Reference those variables in your Collector configuration:

exporters:
  otlp:
    endpoint: "${SIGNOZ_OTLP_ENDPOINT}"
    tls:
      insecure: false
    headers:
      signoz-ingestion-key: "${SIGNOZ_INGESTION_KEY}"

Then add .env to your .gitignore so the key never gets committed:

echo ".env" >> .gitignore

5.2 Start the FastAPI Application.

Start the application on port 8080:

uvicorn app.main:app --reload --port 8080

Verify that the application is running:

curl http://localhost:8080/

Now inspect the Prometheus metrics:

curl -Ls http://localhost:8080/metrics

The /metrics endpoint exposes the application's Prometheus metrics, including the payment_transaction_duration_seconds histogram that the Collector will scrape.

5.3 Start the Collector

With the application running and the Collector configuration in place, start the Collector:

docker compose up --build

Check the Collector logs to confirm that the pipeline is running and that metrics are being processed.

This builds the payment-api image, starts both containers on the shared telemetry network, and the Collector immediately begins scraping /metrics from the application using the Prometheus receiver, reading your SigNoz ingestion key and endpoint from .env automatically.

5.4 Generate Test Transactions

Generate transactions with different processing times:

curl -X POST "http://localhost:8080/transactions?delay_ms=50"

curl -X POST "http://localhost:8080/transactions?delay_ms=250"

curl -X POST "http://localhost:8080/transactions?delay_ms=2500"

Generate additional requests if you want more observations in the histogram.

These requests create transaction-duration observations that are recorded by the payment_transaction_duration_seconds histogram. The Collector picks up the updated metric during its next scrape.

5.5 Confirm Backend Receipt

Open SigNoz Cloud and search for:

payment_transaction_duration_seconds

The metric should be represented as a histogram, with its bucket distribution _bucket, _sum, and _count available to the backend rather than appearing as unrelated Prometheus series.

5.6 Troubleshooting

If metrics aren't flowing correctly, start by checking the log:

docker logs <collector-container>

Look for connection errors, authentication failures, failed scrape attempts, or configuration errors.

Also verify that:

• The FastAPI application is running.

• /metrics is accessible.

• The Collector can reach the application.

• The SigNoz endpoint and ingestion key are correct.

• The .env variables are available to the Collector.

• The receiver and exporter names match the pipeline configuration.

Use the --dry-run flag if available to validate before deployment.

Conclusion

Prometheus histograms provide a useful view of transaction latency by showing how observations are distributed across different buckets rather than reducing them to a single average.

In this tutorial, we followed payment_transaction_duration_seconds from a FastAPI application's /metrics endpoint through the OpenTelemetry Collector and into SigNoz.

The Prometheus receiver mapped the _bucket, _count, and _sum series into the OpenTelemetry Histogram data model, preserving the distribution of transaction durations for analysis in the backend.

This allows the same metric to move from a Prometheus-instrumented application into an OTLP-based observability platform without manually reconstructing the histogram.

While these advantages have fueled rapid cloud adoption, they've also introduced new security challenges that many organisations fail to recognise until it's too late.

Unlike traditional data centers, cloud environments are dynamic. New virtual machines, databases, storage accounts, containers, and serverless functions can appear and disappear throughout the day.

As cloud infrastructure grows, so does the number of potential security gaps. Many of these vulnerabilities remain hidden because they stem from configuration errors rather than software flaws.

This is where cloud security posture management becomes essential. Instead of relying on periodic security reviews, organisations can continuously monitor their cloud environments, identify risks, and remediate issues before they become security incidents.

For each of the five risks below, I'll walk through one concrete example of how the problem actually shows up in a real environment, and the specific steps a team would take to fix it. The examples are deliberately provider-neutral, because the same patterns appear across every major cloud platform under different names.

Understanding these hidden risks is the first step toward building a stronger and more resilient cloud environment.

What We'll Cover

• Why Cloud Infrastructure Creates New Security Challenges

• Hidden Risk #1: Cloud Misconfigurations

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #2: Excessive Permissions

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #3: Forgotten Cloud Resources

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #4: Compliance Drift

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #5: Limited Visibility Across Multi-Cloud Environments

  • What this looks like in practice

  • How to mitigate it

• Why Manual Security Reviews Are No Longer Enough

• How Cloud Security Posture Management Strengthens Cloud Security

• Building a Stronger Cloud Security Strategy

• Conclusion

Why Cloud Infrastructure Creates New Security Challenges

Modern cloud environments are far more complex than the on-premises infrastructure many security teams were built to protect. A single organisation may use Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) simultaneously. Each provider offers hundreds of services, each with its own security settings and access controls.

Development teams also create new cloud resources at an unprecedented pace. Infrastructure as Code, automated deployment pipelines, and container orchestration platforms enable provisioning entire environments in minutes. While this improves productivity, it also increases the likelihood of configuration errors.

The shared responsibility model adds another layer of complexity. Cloud providers secure the underlying infrastructure, but customers remain responsible for securing their applications, identities, storage, networks, and data. Many organisations mistakenly assume their cloud provider handles every aspect of security, leaving critical resources exposed.

Hidden Risk #1: Cloud Misconfigurations

Misconfigurations remain one of the leading causes of cloud security incidents. A single incorrect setting can expose sensitive information to the public internet or give attackers access to critical systems.

Examples include publicly accessible storage buckets, databases that allow unrestricted network access, disabled encryption settings, and security groups with overly permissive firewall rules. These mistakes often occur during rapid deployments when security reviews are skipped or delayed.

The challenge is that cloud environments change constantly. Even if an environment is secure today, a new deployment tomorrow may introduce a vulnerability without anyone noticing.

What This Looks Like in Practice

A product team needs to share a set of report images with an external design agency. The quickest path is to create an object storage bucket and set its access policy to allow read access from any identity, rather than setting up temporary signed URLs or a guest account. The share works, the agency delivers, and the ticket is closed.

Six weeks later, the same bucket is reused as the destination for a nightly export job. Nobody rechecks the access policy, because the bucket already exists and "works." The export contains customer names, email addresses, and internal order IDs, and it's now readable by anyone who can guess or enumerate the bucket name. Automated scanners that crawl for open buckets will typically find it within days.

Nothing was hacked here. No credential was stolen and no software was exploited. A setting that was reasonable for one purpose was silently inherited by another.

How to Mitigate it

1. Turn on the account-level public access block: Every major provider offers an org- or account-wide setting that overrides individual resource policies and prevents public exposure of storage. Enable it as the default, and treat any exception as a change that requires approval.

2. Scan the infrastructure code before it's applied, not after: Add a policy-as-code check to your deployment pipeline so a template that grants public read on a bucket fails the build. This is where you catch the problem for free, before it exists in the environment.

3. Replace public access with time-limited access: For genuine external sharing, use pre-signed URLs with an expiry measured in hours, or a scoped guest identity. This gives the same convenience without leaving a permanent hole.

4. Alert on the change, not just the state: Configure your posture management tool to fire when a bucket's access policy changes to allow anonymous principals, and route it to the team that owns the resource. Detecting the drift within minutes is what separates a near miss from a breach notification.

Cloud security posture management continuously scans cloud resources for configuration issues and alerts security teams when risks appear. Instead of waiting for a scheduled audit, organisations gain visibility into problems as they occur.

Hidden Risk #2: Excessive Permissions

Identity and Access Management (IAM) is one of the most important parts of cloud security. Unfortunately, it's also one of the most common sources of risk.

Many organisations grant users, applications, or service accounts more permissions than they actually need. Over time, these excessive privileges accumulate as employees change roles, applications evolve, and temporary permissions become permanent.

If an attacker compromises an account with broad privileges, the damage can spread quickly across the cloud environment. Sensitive data, production systems, and administrative resources may all become accessible.

What This Looks Like in Practice

A team sets up a deployment pipeline. The pipeline needs to create compute instances, update a container registry, and write to a couple of storage locations. Working out the exact permission set takes time, and the first three attempts fail with cryptic access-denied errors. Under deadline pressure, someone attaches the built-in administrator role to the pipeline's service identity "just to unblock it," with the intention of narrowing it later.

Later never comes. The pipeline now holds full administrative rights across the account, and its credentials live in a CI/CD system that dozens of engineers can configure. Anyone who can modify a build file, or who compromises a single developer's account on that CI system, can execute arbitrary commands with administrator privileges like read every database, create new identities, or disable logging.

The same pattern appears with people. An engineer moves from platform to frontend work, keeps the old production access "in case something breaks," and two years later has accumulated permissions spanning four teams they no longer belong to.

How to Mitigate it

1. Pull the last-accessed data before you touch anything: Every major provider records which services an identity has actually used and when. Start by listing every role with administrative privileges and check what it has genuinely called in the past 90 days. In most environments the real usage is a small fraction of what's granted.

2. Generate a scoped policy from observed activity: Use the provider's policy-generation tooling (or your posture management platform) to produce a candidate policy based on that access history, then review it manually before applying. Applying the principle of least privilege is far easier when you start from evidence rather than guesswork.

3. Roll it out in audit mode first: Apply the narrowed policy to a non-production copy of the pipeline, or run it alongside the permissive one with denial logging enabled, so you find the missing permissions without breaking deployments.

4. Set an expiry on elevated access: Temporary administrative rights should be granted through a just-in-time mechanism that expires automatically, so the "narrow it later" step happens whether or not anyone remembers.

5. Watch for privilege escalation paths, not just broad roles: A role that can't do much directly but can create identities, attach policies, or pass a privileged role to a new resource is effectively an administrator. Posture management tools flag these chains, which are almost impossible to spot by reading policies one at a time.

Hidden Risk #3: Forgotten Cloud Resources

Cloud environments are rarely static. Development teams frequently create temporary virtual machines, test databases, storage accounts, and networking components while building new features. Once projects are completed, these resources are often forgotten rather than removed.

Although these abandoned resources may no longer serve a business purpose, they continue to consume cloud resources and may still contain sensitive information. Some may even remain connected to production networks.

These overlooked assets create unnecessary attack surfaces. Attackers often look for neglected systems because they're less likely to receive security updates or active monitoring.

What This Looks Like in Practice

Ahead of a major release, a team spins up a load-testing environment: a handful of compute instances, a database restored from a production snapshot so the test data is realistic, and a permissive network rule opening administrative access from any address, because the testers are working from home and nobody wants to maintain an IP allowlist for a two-week project.

The release ships. The environment stays. A year on, those instances are running an operating system image that has missed four rounds of patches, the database still contains a year-old copy of real customer records, and the administrative port is still open to the internet. It doesn't appear on any architecture diagram, it isn't in the patching rota, and no alerting is configured against it, because it was never meant to exist for more than a fortnight.

This is one of the most attractive targets in a cloud estate: real data, no monitoring, and a known-vulnerable software version.

How to Mitigate it

1. Build an inventory that doesn't depend on memory: Query every region in every account for running compute, databases, storage, and snapshots, including the regions you believe you don't use, which is where forgotten resources most often hide. Deploying application security tools across both active and forgotten resources helps ensure vulnerabilities are caught before they can be exploited.

2. Sort by last activity, then by exposure: Rank resources by how recently they saw network traffic or authentication, and cross-reference against public network reachability. A resource that's both idle and internet-facing is the highest-priority item on the list, every time.

3. Require an owner tag at creation: Enforce a policy that rejects the creation of any resource without owner and environment tags. Untagged resources become the queue you work through, and the queue stops growing.

4. Give non-production resources a default lifetime: Attach an expiry tag to anything created in a test or sandbox account, and run an automated job that stops resources past their date and deletes them after a grace period. Make deletion the default and continuation the thing that requires a decision.

5. Treat production data in test environments as an incident: Restoring a production snapshot into a lower environment should trigger an alert. If realistic data is genuinely needed, mask it as part of the restore process.

Continuous asset discovery is another important capability of cloud security posture management. By maintaining an up-to-date inventory of cloud resources, organisations can quickly identify unused assets and remove them before they become security liabilities.

Hidden Risk #4: Compliance Drift

Meeting regulatory requirements isn't a one-time project. Standards such as ISO 27001, SOC 2, PCI DSS, HIPAA, and GDPR require organisations to maintain secure configurations over time.

As cloud environments evolve, once-compliant systems can gradually drift away from required security baselines. A disabled encryption setting, an open network port, or missing activity logs may create compliance violations without anyone noticing.

Manual compliance reviews often occur quarterly or annually, leaving long periods during which risks remain undetected.

What This Looks Like in Practice

An organisation passes its SOC 2 audit. Part of the evidence is that API activity logging is enabled across all accounts and all regions, with logs written to a dedicated, immutable archive.

Four months later, an engineer investigating a surprise bill notices that log storage is one of the larger line items. Most of the volume comes from high-frequency read events in a development account. They disable logging in that account to cut costs. It's a sensible-looking decision made with incomplete context, and it's approved in a routine change ticket by someone who doesn't know the account is in the audit scope.

Nothing breaks and no alert fires. The gap surfaces eight months later during audit fieldwork, when the assessor asks for a continuous log record for the period and the organisation can't produce one. There's no way to backfill it. The finding stands, and the same gap means that if an incident had occurred in that account, there would be no record of who did what.

How to Mitigate it

1. Map each control to a machine-checkable rule: "Activity logging is enabled everywhere" becomes a specific check: a multi-region trail exists in every account, log file validation is on, and the destination has retention and object-lock configured. If a control can't be expressed as a check, it can't be monitored continuously.

2. Make the critical controls structurally hard to disable: Deploy logging from the organisation level rather than per account, and apply a service control policy that denies the disable and delete actions to everyone except a break-glass identity. This turns a one-click mistake into a deliberate act.

3. Run the checks continuously and alert on transitions: The valuable signal isn't the monthly compliance score, it's the moment a control moves from compliant to non-compliant. Route that event to a channel someone actually reads, with the account, the resource, and the identity that made the change.

4. Add a compliance-scope check to your change process: Change tickets touching accounts in audit scope should surface that fact automatically, so the approver knows what they're approving.

5. Keep the evidence as you go: Store the continuous check results with timestamps. Audit preparation then becomes an export rather than a scramble, and you find gaps months before an assessor does.

Cloud security posture management continuously evaluates cloud environments against security frameworks and compliance standards. This allows organisations to detect violations early, simplify audit preparation, and reduce regulatory risk.

Hidden Risk #5: Limited Visibility Across Multi-Cloud Environments

Many organisations now operate workloads across multiple cloud providers. Development teams may choose different platforms based on cost, performance, or specialised services.

While this approach offers flexibility, it also creates fragmented security visibility. Each cloud provider has its own monitoring tools, dashboards, and security controls. Security teams may struggle to understand the overall security posture of the organisation.

Without centralised visibility, important risks can easily go unnoticed. One cloud environment may enforce strong security policies while another contains exposed resources and outdated configurations.

What This Looks Like in Practice

An organisation runs its core platform on one cloud provider, where the security team has spent two years building guardrails: preventative policies, automated scanning, and an on-call rota that responds to findings.

Meanwhile, the data science group adopts a second provider for its managed machine learning tooling. The account is opened on a corporate card, funded through a departmental budget, and never onboarded into the security programme. There's no reason for anyone to notice, no ticket crosses the security team's desk, because from the data team's perspective nothing about their work has changed.

That second environment ends up hosting training datasets derived from production records. Its storage defaults are different from the primary provider's, the equivalent of "block public access" isn't switched on, and the terminology doesn't map cleanly onto the checks the security team already wrote. The organisation's security dashboard shows a healthy posture, and it's accurate for the 70% of the estate it can see.

The failure isn't a weak control. It's that a whole environment is outside the boundary of the thing doing the measuring.

How to Mitigate it

1. Start with a billing reconciliation, not a technical scan: Ask finance for every cloud vendor the organisation pays, then compare that list against the accounts your security tooling monitors. Shadow cloud environments show up in expense reports long before they show up in a scanner.

2. Write your baseline in terms of outcomes, not provider features: Define controls as "object storage must not be readable by anonymous principals" rather than naming a specific provider's setting. One statement then maps onto every platform, and the standard doesn't fragment.

3. Normalise findings into a single queue: Pull results from each provider into one place with consistent severity scoring, so a critical exposure in the smallest environment ranks above a low-severity finding in the largest. Severity should reflect business impact, not which dashboard raised it.

4. Make onboarding a condition of account creation: Require any new cloud account, on any provider, to be enrolled in central monitoring before it can hold production or production-derived data. Pair the requirement with a fast, low-friction path so teams aren't tempted to route around it.

5. Re-run the discovery on a schedule: New accounts appear continuously. Repeat the billing reconciliation quarterly, because the environment you don't know about is by definition the one you'll never get an alert for.

Cloud security posture management provides a unified view across multiple cloud providers, allowing organisations to monitor security consistently regardless of where workloads are deployed.

Why Manual Security Reviews Are No Longer Enough

Traditional security audits were designed for relatively stable infrastructure. Cloud environments operate very differently.

Infrastructure changes can occur hundreds or even thousands of times each day through automated deployment pipelines. New services are launched continuously, configurations are updated frequently, and development teams release software at a rapid pace.

Human reviewers can't keep up with this level of change. By the time a manual assessment is completed, the cloud environment may already look completely different.

Notice what the five examples above have in common. In every case, the environment was compliant at some point in the past, and a single reasonable-looking decision moved it out of compliance without producing an error, an outage, or a complaint. A bucket policy was reused. A role was widened to unblock a deployment. A test environment outlived its project. Logging was disabled to control cost. An account was opened outside the security boundary.

None of these would be caught by a quarterly review, because by the time a reviewer arrives, the change is months old and looks like part of the furniture. Automation has become essential. Continuous monitoring allows organisations to identify new risks immediately instead of discovering them weeks or months later.

How Cloud Security Posture Management Strengthens Cloud Security

Cloud security posture management provides continuous visibility into cloud environments by automatically evaluating resources against security best practices and organisational policies.

Rather than replacing existing security tools, it complements them by focusing on configuration security, governance, compliance, and risk reduction.

A modern cloud security posture management platform can discover cloud assets, monitor configuration changes, identify excessive permissions, detect publicly exposed resources, validate compliance requirements, and prioritise security findings based on business impact.

Many platforms also support automated remediation. Instead of simply reporting an issue, they can trigger workflows that correct insecure configurations or notify the appropriate teams before vulnerabilities are exploited.

A word of caution on automated remediation: start it in report-only mode. A rule that automatically closes public access is exactly right for a storage bucket holding internal exports and exactly wrong for one serving a public website's static assets. Run new rules in observation mode for a few weeks, review what they would have changed, and only then let them act , beginning with the resource types where a false positive is cheap.

This proactive approach allows organisations to reduce risk without slowing down software development.

Building a Stronger Cloud Security Strategy

Technology alone can't eliminate cloud security risks. Organisations must also establish strong governance processes and promote security awareness across development, operations, and security teams.

Security should become part of the software development lifecycle rather than an activity performed after deployment. Infrastructure templates should follow secure defaults, access permissions should be reviewed regularly, and cloud resources should be continuously monitored throughout their lifecycle.

If you're starting from scratch, the sequence that tends to work is: inventory first, since you can't secure what you can't see; then public exposure, because that's where the shortest path from mistake to breach runs; then identity, because over-permissioned roles determine how far an attacker gets once inside; and finally compliance mapping, which is far easier once the first three are under control.

Regular security training also helps reduce human error. Developers who understand cloud security best practices are less likely to introduce risky configurations during deployment. It's worth noting that in every example in this article, the person who introduced the risk was doing their job competently under normal constraints. Training works best when it's framed that way, and when the secure path is also the convenient one.

Combining secure development practices with cloud security posture management creates multiple layers of protection that significantly reduce the likelihood of security incidents.

Conclusion

Cloud infrastructure offers tremendous flexibility, but it also introduces risks that are often difficult to detect through traditional security practices. Misconfigured resources, excessive permissions, forgotten assets, compliance drift, and fragmented visibility can quietly weaken an organisation's security posture without triggering immediate alarms.

As cloud environments continue to grow in size and complexity, continuous monitoring has become a necessity rather than a luxury. Cloud security posture management enables organisations to identify vulnerabilities, maintain compliance, and respond to security risks before they lead to costly breaches.

The organisations that succeed in the cloud aren't those that eliminate every possible risk. They're the ones who continuously understand their environment, detect changes as they happen, and act quickly to maintain a secure cloud infrastructure.

In a landscape where new resources are created every day, maintaining visibility into your cloud security posture is one of the most effective ways to stay ahead of evolving threats.

Hope you enjoyed this article. You can connect with me on LinkedIn.

A developer needs a new staging environment, so they file a ticket. The platform team queues it.

Two weeks later, the environment exists. It's configured slightly differently from the last one, with a naming convention that doesn't match the production setup, missing the observability stack the previous environment had. The developer deploys. Something breaks. Nobody knows why.

The problem isn't the ticket queue. The problem is the absence of a platform: a paved road where developers can self-serve infrastructure, deployments, and environments that are consistent, auditable, and safe without requiring a platform engineer for every request.

An Internal Developer Platform (IDP) solves this. Not by removing platform engineers from the picture, but by shifting their work from executing individual requests to building the systems that execute those requests automatically.

This handbook builds a production-grade IDP from the three CNCF tools that form its core in 2026: Backstage as the developer portal and software catalog, ArgoCD as the GitOps continuous delivery engine, and Crossplane as the Kubernetes-native infrastructure control plane.

By the end, developers on your platform will be able to provision a cloud database, deploy an application to staging, and register a new service in the catalog — all without filing a single ticket.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: IDP Architecture — The Three-Layer Model

• Part 2: ArgoCD — The GitOps Foundation

• Part 3: Crossplane — Infrastructure as Kubernetes Resources

• Part 4: Backstage — The Developer Portal

• Part 5: Wiring It Together — The Golden Path

• Part 6: FinOps Integration — Cost Attribution on the IDP

• Part 7: The Platform Maturity Model — Measuring What You've Built

• Best Practices Summary

• Resources

What You'll Learn

• The three-layer IDP architecture and why each layer must be implemented in a specific order

• How to install and configure ArgoCD with ApplicationSets for multi-environment GitOps delivery

• How to define cloud infrastructure as Kubernetes custom resources using Crossplane Compositions

• How to deploy and configure Backstage with a software catalog and Software Templates

• How to wire Backstage, ArgoCD, and Crossplane together into a single self-service golden path

• How to implement cost attribution on your IDP so every resource provisioned through it carries team and cost center metadata

• How to measure your IDP's maturity using the CNCF Platform Engineering Maturity Model

Let's build it.

Prerequisites

Before following along, you should have:

Knowledge:

• Working familiarity with Kubernetes: you can deploy applications, write YAML manifests, and understand namespaces and RBAC

• Basic GitOps understanding: you know what "Git as source of truth" means in practice

• Comfort with Helm, Terraform HCL, and TypeScript at a reading level

• Understanding of AWS services: EKS, RDS, S3, IAM

Tools and access:

• An EKS cluster running Kubernetes 1.28 or later with at least 3 nodes (m5.xlarge or equivalent)

• kubectl configured and pointing at your cluster

• helm 3.12 or later installed

• AWS CLI v2 configured with admin-level permissions for the provisioning steps

• Node.js 18 or later and Yarn (for Backstage)

• A GitHub organisation you control (for the GitOps repositories and Backstage GitHub integration)

Companion repository:

git clone https://github.com/aayostem/platform-toolkit
cd platform-toolkit

The repository contains all manifests, Helm values files, Crossplane Compositions, and Backstage templates referenced in this guide. Each part maps to a directory in the repo.

Estimated time: The full implementation takes one to two days for an experienced platform engineer. Parts 1–3 can be completed in the morning and produce a working GitOps delivery layer.

Part 1: IDP Architecture — The Three-Layer Model

1.1 What an IDP Actually Is

An Internal Developer Platform isn't a tool. It's a product: a collection of tools, workflows, and abstractions that platform teams build and maintain so that application developers can move fast without managing infrastructure directly.

The distinction matters because it shapes every architectural decision. A tool is installed and configured. A product is designed for users, iterated based on feedback, and measured by whether those users actually adopt it. The platform teams that build the IDPs that developers love think like product managers, not system administrators.

The DORA 2025 report found that nearly 90% of enterprises now have some form of internal platform. But having a platform and having a platform that developers actually use are different things.

The survey found that developer satisfaction with internal platforms varied dramatically. And the gap between satisfied and unsatisfied teams correlated directly with whether the platform team treated the IDP as a product with a roadmap and user research, or as an infrastructure project with a ticket queue.

The three tools in this guide — Backstage, ArgoCD, and Crossplane — are the most widely adopted open-source stack for production IDPs in 2026. But the architecture that connects them matters as much as the tools themselves.

1.2 The Three-Layer Architecture

A production IDP has three distinct layers, each with a single responsibility:

Layer 1: Developer Interface (Backstage)
├── Software catalog — inventory of all services, APIs, and resources
├── Software Templates — self-service forms that trigger provisioning workflows
├── TechDocs — documentation co-located with each catalog entity
└── Plugins — integrations with ArgoCD, Kubernetes, PagerDuty, Grafana

Layer 2: Delivery Layer (ArgoCD)
├── GitOps sync — continuous reconciliation of cluster state to Git
├── ApplicationSets — multi-environment deployment from a single definition
├── Rollout management — progressive delivery with health checks
└── Audit trail — every deployment change linked to a Git commit

Layer 3: Infrastructure Layer (Crossplane)
├── Composite Resources — cloud resources defined as Kubernetes CRDs
├── Compositions — templates that expand a simple claim into full AWS infrastructure
├── ProviderConfigs — credentials and region configuration for each cloud provider
└── Usage tracking — every provisioned resource tagged with team and cost centre

The critical architectural rule: Backstage never talks directly to Kubernetes or cloud APIs. When a developer submits a Software Template in Backstage, the output is a Git commit — a YAML file representing a Crossplane claim or an ArgoCD Application manifest. ArgoCD picks up that commit and applies it to the cluster. Crossplane translates the cluster resource into actual cloud infrastructure.

This indirect path isn't complexity for complexity's sake. It means every infrastructure change is a Git commit, with an author, a timestamp, a pull request, and a review. The audit trail is automatic. The rollback mechanism is git revert.

Developer → Backstage Template → Git commit → ArgoCD → Crossplane → AWS
                                     ↑
                          Single source of truth
                          Full audit trail
                          Rollback = git revert

Here's what the incorrect alternative looks like — Backstage calling cloud APIs directly:

// Bad: Backstage template calling AWS SDK directly
// No audit trail, no rollback, no reconciliation loop
// If the call fails halfway, you have partial infrastructure with no record
import { S3Client, CreateBucketCommand } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "us-east-1" });
await client.send(new CreateBucketCommand({ Bucket: bucketName }));

And the correct approach — Backstage outputting a Crossplane claim to Git:

# Good: Backstage template output — a Crossplane claim committed to Git
# ArgoCD applies it, Crossplane reconciles it, AWS creates the bucket
# Every step is tracked, auditable, and reversible
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: S3Bucket
metadata:
  name: ${{ values.bucket_name }}
  namespace: ${{ values.team_namespace }}
  labels:
    team: ${{ values.team_name }}
    cost-centre: ${{ values.cost_centre }}
    environment: ${{ values.environment }}
spec:
  versioning: true
  encryption: AES256
  region: us-east-1

1.3 Implementation Order

Build in this order. Deviating from it creates integration problems that are difficult to debug:

Step 1: ArgoCD — the delivery foundation everything else depends on
Step 2: Crossplane — infrastructure control plane, delivered by ArgoCD
Step 3: Backstage — the portal, pointing at ArgoCD and Crossplane as backends
Step 4: Wire together — Software Templates that produce GitOps manifests
Step 5: FinOps layer — cost attribution metadata in every provisioned resource

Part 2: ArgoCD — The GitOps Foundation

ArgoCD is a declarative continuous delivery tool for Kubernetes that implements the GitOps pattern. If you haven't used a GitOps tool before, the core idea is simple: your Git repository is the single source of truth for what should be running in your cluster, and ArgoCD continuously reconciles actual cluster state to match it.

If a developer manually changes a resource in the cluster, ArgoCD detects the drift and resyncs from Git. If Git changes, ArgoCD applies the change to the cluster. Human intervention isn't required, and is actively discouraged — the goal is a cluster whose state is always fully explained by what's in Git.

ArgoCD is a CNCF Graduated project, meaning it's production-ready and widely used. It runs as a set of pods in your cluster with a web UI, a CLI, and a REST API. Everything you need to manage deployments across multiple environments lives in one place.

2.1 Installing ArgoCD

# Create the ArgoCD namespace
kubectl create namespace argocd

# Install ArgoCD using the official manifest
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all pods to be running before proceeding
kubectl wait --for=condition=Ready pods \
  --all -n argocd --timeout=300s

# Get the initial admin password
argocd_password=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

echo "ArgoCD initial password: $argocd_password"
echo "Save this somewhere secure before proceeding"

# Port-forward to access the ArgoCD UI locally
kubectl port-forward svc/argocd-server -n argocd 8080:443 &

# Login via CLI
argocd login localhost:8080 \
  --username admin \
  --password "$argocd_password" \
  --insecure

# Change the password immediately
argocd account update-password \
  --current-password "$argocd_password" \
  --new-password "your-secure-password"

2.2 Repository Structure for GitOps

The repository structure ArgoCD watches determines how you manage multiple environments. The pattern that scales best is environment-per-directory, with overlays managed by Kustomize.

Kustomize is a Kubernetes-native configuration management tool that lets you define a base configuration once and layer environment-specific overrides on top of it. This means your staging and production configurations share the same YAML structure but differ in replica counts, image tags, and resource limits.

gitops-repo/
├── apps/
│   ├── base/                    # Shared configuration across all environments
│   │   ├── payment-api/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── user-api/
│   │       ├── deployment.yaml
│   │       ├── service.yaml
│   │       └── kustomization.yaml
│   └── overlays/
│       ├── staging/             # Staging-specific overrides
│       │   ├── payment-api/
│       │   │   └── kustomization.yaml   # Override: 1 replica, staging image tag
│       │   └── kustomization.yaml
│       └── production/          # Production-specific overrides
│           ├── payment-api/
│           │   └── kustomization.yaml   # Override: 3 replicas, pinned image tag
│           └── kustomization.yaml
└── infrastructure/
    ├── crossplane/              # Crossplane installation and providers
    ├── monitoring/              # Prometheus, Grafana
    └── ingress/                 # NGINX or ALB ingress controller

2.3 ApplicationSets — Managing Multiple Environments

An ApplicationSet is an ArgoCD resource that generates multiple Application objects from a single template. Instead of creating one Application manifest per service per environment — which becomes unmanageable at scale — you define one ApplicationSet that covers all services across all environments. A matrix generator combines a list of environments with a Git directory scan to produce every combination automatically:

# applicationset-apps.yaml
# This single resource generates one ArgoCD Application
# for each combination of environment and application directory
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          # Generator 1: environments
          - list:
              elements:
                - environment: staging
                  cluster: https://staging.eks.cluster.local
                - environment: production
                  cluster: https://production.eks.cluster.local

          # Generator 2: application directories in the overlay
          - git:
              repoURL: https://github.com/your-org/gitops-repo
              revision: HEAD
              directories:
                - path: apps/overlays/{{environment}}/*

  template:
    metadata:
      name: "{{environment}}-{{path.basename}}"
      labels:
        environment: "{{environment}}"
        app: "{{path.basename}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/gitops-repo
        targetRevision: HEAD
        path: "apps/overlays/{{environment}}/{{path.basename}}"
      destination:
        server: "{{cluster}}"
        namespace: "{{path.basename}}"
      syncPolicy:
        automated:
          prune: true        # Delete resources removed from Git
          selfHeal: true     # Revert manual cluster changes
        syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=foreground

Verify the ApplicationSet is generating the expected Applications:

# List all generated Applications
kubectl get applications -n argocd

# Expected output: one Application per environment per app
# staging-payment-api    Synced    Healthy
# staging-user-api       Synced    Healthy
# production-payment-api Synced    Healthy
# production-user-api    Synced    Healthy

# Check sync status for a specific application
argocd app get staging-payment-api

2.4 ArgoCD RBAC for Platform Teams

In a multi-team IDP, different teams need different levels of access to ArgoCD. Application teams should be able to view and sync their own applications. Platform teams should have broader access. Nobody should have unrestricted cluster admin through ArgoCD.

The default policy is readonly — every authenticated user can see everything but change nothing:

# argocd-rbac-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    # Platform team: full access to all applications
    p, role:platform-team, applications, *, */*, allow
    p, role:platform-team, clusters, get, *, allow
    p, role:platform-team, repositories, *, *, allow

    # Application teams: sync and get their own namespace only
    p, role:app-team, applications, get, */staging-*, allow
    p, role:app-team, applications, sync, */staging-*, allow

    # Bind roles to GitHub teams
    g, your-org:platform-engineers, role:platform-team
    g, your-org:developers, role:app-team

  scopes: '[groups]'

Part 3: Crossplane — Infrastructure as Kubernetes Resources

Crossplane is a CNCF Graduated open-source framework that extends Kubernetes into a universal infrastructure control plane.

The core idea: instead of managing cloud resources with separate tools like Terraform or CloudFormation that live outside your cluster, you define cloud resources — RDS databases, S3 buckets, VPCs, IAM roles — as Kubernetes custom resource definitions.

Once you apply a Crossplane resource to the cluster, Crossplane's controllers take over and reconcile the desired state to the actual AWS state, exactly the way Kubernetes reconciles a Deployment to a set of running pods.

The key abstraction Crossplane adds on top of that is the Composite Resource. A platform team defines a high-level PostgreSQLDatabase type that abstracts over the thirty-plus configuration fields an actual RDS instance requires.

Developers interact with the simple type. Crossplane expands it into the full AWS resource configuration behind the scenes, applying the platform team's security and operational standards automatically — standards that developers can't bypass because they never see the underlying fields.

3.1 Installing Crossplane

Crossplane is delivered to your cluster by ArgoCD — the first integration between the two tools. By installing Crossplane through an ArgoCD Application rather than running helm install directly, you make Crossplane itself part of the GitOps-managed infrastructure. Any change to Crossplane's configuration goes through a Git commit and review:

# infrastructure/crossplane/application.yaml
# ArgoCD Application that installs Crossplane via Helm
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: crossplane
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.crossplane.io/stable
    chart: crossplane
    targetRevision: 1.15.0
    helm:
      values: |
        provider:
          packages:
            # AWS provider — manages all AWS resources
            - xpkg.upbound.io/upbound/provider-aws-s3:v1.2.0
            - xpkg.upbound.io/upbound/provider-aws-rds:v1.2.0
            - xpkg.upbound.io/upbound/provider-aws-iam:v1.2.0
  destination:
    server: https://kubernetes.default.svc
    namespace: crossplane-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

# Apply the ArgoCD Application — ArgoCD installs Crossplane
kubectl apply -f infrastructure/crossplane/application.yaml

# Watch Crossplane pods come up
kubectl get pods -n crossplane-system -w

# Verify providers are installed and healthy
kubectl get providers
# Expected:
# NAME                          INSTALLED   HEALTHY   PACKAGE
# upbound-provider-aws-s3       True        True      xpkg.upbound.io/...
# upbound-provider-aws-rds      True        True      xpkg.upbound.io/...

3.2 Provider Credentials

Crossplane needs AWS credentials to provision resources. The recommended approach for EKS is IAM Roles for Service Accounts (IRSA) — a mechanism that lets Kubernetes pods assume IAM roles directly without storing any credentials in the cluster.

The pod's Kubernetes service account is annotated with an IAM role ARN, and AWS automatically provides short-lived credentials when the pod makes API calls. No access keys, no secrets to rotate, and no credentials to accidentally expose:

# Create the IAM role for Crossplane with the necessary AWS permissions
aws iam create-role \
  --role-name CrossplaneProviderRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/YOUR_OIDC_ID:sub":
            "system:serviceaccount:crossplane-system:provider-aws"
        }
      }
    }]
  }'

# Attach the permissions policy (scope this to minimum required in production)
aws iam attach-role-policy \
  --role-name CrossplaneProviderRole \
  --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

# provider-config.yaml
# Configure the AWS provider with IRSA — no static credentials
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: IRSA   # Use the IAM role attached to the provider service account

3.3 Defining a Composite Resource — PostgreSQL Database

This is where the IDP abstraction lives. The platform team defines two YAML files: the CompositeResourceDefinition (XRD), which specifies the shape of what developers can request, and the Composition, which specifies how that request expands into actual AWS resources with platform standards applied.

The XRD is the API contract with developers. Keep it simple — only fields developers genuinely need to control should appear here:

# xrd-postgresql.yaml
# Defines the PostgreSQLDatabase type that developers can request
# Developers never see the RDS-specific configuration below
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresqldatabases.platform.cloudfrugal.com
spec:
  group: platform.cloudfrugal.com
  names:
    kind: XPostgreSQLDatabase
    plural: xpostgresqldatabases
  claimNames:
    kind: PostgreSQLDatabase     # This is what developers create
    plural: postgresqldatabases
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                # Developer-facing fields only — simple and bounded
                storageGB:
                  type: integer
                  minimum: 20
                  maximum: 1000
                  description: "Storage in GB. Min 20, max 1000."
                instanceClass:
                  type: string
                  enum: ["small", "medium", "large"]
                  description: "small=db.t4g.medium, medium=db.r7g.large, large=db.r7g.2xlarge"
                environment:
                  type: string
                  enum: ["staging", "production"]

The Composition is the platform team's implementation. It maps the simple developer fields to the full RDS configuration and enforces platform standards that developers can't override:

# composition-postgresql.yaml
# Defines what a PostgreSQLDatabase claim expands into
# Platform standards (encryption, backup, deletion protection) are applied here
# Developers cannot override them — the platform enforces them
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: postgresql-aws-composition
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: platform.cloudfrugal.com/v1alpha1
    kind: XPostgreSQLDatabase

  resources:
    # The actual RDS instance — expanded from the simple developer claim
    - name: rds-instance
      base:
        apiVersion: rds.aws.upbound.io/v1beta1
        kind: Instance
        spec:
          forProvider:
            region: us-east-1
            engine: postgres
            engineVersion: "15.4"
            # Platform standards — always applied, not developer-configurable
            storageEncrypted: true           # Always encrypted
            backupRetentionPeriod: 7         # Always 7-day backup
            deletionProtection: true         # Always deletion-protected
            multiAZ: false                   # Overridden to true for production (see patches)
            dbSubnetGroupNameSelector:
              matchLabels:
                platform.cloudfrugal.com/subnet-group: private
      patches:
        # Map the developer's simple instanceClass to the actual RDS instance type
        - type: CombineFromComposite
          combine:
            variables:
              - fromFieldPath: spec.instanceClass
            strategy: string
            string:
              fmt: |
                %s
          toFieldPath: spec.forProvider.dbInstanceClass
          transforms:
            - type: map
              map:
                small:  db.t4g.medium
                medium: db.r7g.large
                large:  db.r7g.2xlarge

        # Enable Multi-AZ for production automatically
        - type: FromCompositeFieldPath
          fromFieldPath: spec.environment
          toFieldPath: spec.forProvider.multiAZ
          transforms:
            - type: map
              map:
                staging:    "false"
                production: "true"

        # Copy team labels from the claim to the RDS instance for cost attribution
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels
          toFieldPath: spec.forProvider.tags

A developer requesting a PostgreSQL database now writes this — nothing more:

# Developer creates this in their team's namespace
# No RDS knowledge required. No IAM configuration. No subnet group lookup.
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: PostgreSQLDatabase
metadata:
  name: payment-service-db
  namespace: payments-team
  labels:
    team: payments
    cost-centre: payments-engineering
    environment: staging
spec:
  storageGB: 100
  instanceClass: medium
  environment: staging

Crossplane reconciles this claim to a full RDS instance within minutes, with encryption, backup, and all platform standards applied automatically.

3.4 Verifying Crossplane Resource Provisioning

# Watch the claim status — it should transition to Ready=True
kubectl get postgresqldatabases -n payments-team -w

# Check the composite resource for detailed status
kubectl describe xpostgresqldatabases.platform.cloudfrugal.com

# Verify the actual AWS resource was created
aws rds describe-db-instances \
  --query 'DBInstances[?TagList[?Key==`team` && Value==`payments`]].[DBInstanceIdentifier,DBInstanceStatus]' \
  --output table

Part 4: Backstage — The Developer Portal

Backstage is a CNCF incubating open-source framework originally built by Spotify. It serves as the developer-facing interface of your IDP — the single place where developers discover services, request infrastructure, and find documentation, without needing to know which underlying system provides any of it.

Backstage provides three core capabilities:

1. A software catalog that inventories every service, API, library, and resource in your organisation

2. Software Templates that give developers self-service forms for provisioning infrastructure and scaffolding new services

3. TechDocs that co-locate documentation with the catalog entity it documents so that documentation is always findable from the same place as the service it covers.

Backstage is built in TypeScript with a React frontend and a Node.js backend. It's configured rather than installed: you create a Backstage app, configure it with your organisation's specifics, and deploy it to your cluster.

4.1 Creating and Configuring Backstage

# Create a new Backstage app
npx @backstage/create-app@latest

# When prompted:
# App name: platform-portal
# Choose SQLite for local development, PostgreSQL for production

cd platform-portal

Configure Backstage to connect to your ArgoCD instance and GitHub:

# app-config.production.yaml
app:
  title: Cloudfrugal Platform Portal
  baseUrl: https://platform.your-company.com

backend:
  baseUrl: https://platform.your-company.com
  database:
    client: pg
    connection:
      host: ${POSTGRES_HOST}
      port: 5432
      user: ${POSTGRES_USER}
      password: ${POSTGRES_PASSWORD}
      database: backstage

# GitHub integration for catalog discovery and template scaffolding
integrations:
  github:
    - host: github.com
      apps:
        - appId: ${GITHUB_APP_ID}
          webhookSecret: ${GITHUB_WEBHOOK_SECRET}
          clientId: ${GITHUB_CLIENT_ID}
          clientSecret: ${GITHUB_CLIENT_SECRET}
          privateKey: ${GITHUB_PRIVATE_KEY}

# ArgoCD plugin configuration
argocd:
  username: ${ARGOCD_USERNAME}
  password: ${ARGOCD_PASSWORD}
  appLocatorMethods:
    - type: 'config'
      instances:
        - name: main
          url: https://argocd.your-company.com

# Catalog auto-discovery — finds catalog-info.yaml files across your GitHub org
catalog:
  providers:
    github:
      your-org:
        organization: 'your-github-org'
        catalogPath: '/catalog-info.yaml'
        filters:
          branch: 'main'

4.2 The Software Catalog — Registering Services

Every service, API, library, and resource in your platform should be registered in the Backstage catalog via a catalog-info.yaml file committed to the service's repository. Backstage discovers these files automatically through the GitHub integration — no manual registration required once the file exists:

# catalog-info.yaml — committed to each service's repository root
# Backstage discovers this automatically via the GitHub integration
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-api
  title: Payment API
  description: "Core payment processing service. Handles transaction initiation, authorisation, and settlement."
  annotations:
    # Links ArgoCD to show deployment status in the Backstage UI
    argocd/app-name: production-payment-api
    # Links GitHub Actions workflow status
    github.com/project-slug: your-org/payment-api
    # Links Grafana dashboard for this service
    grafana/dashboard-selector: "title=Payment API"
    # Links PagerDuty on-call schedule
    pagerduty.com/service-id: P123456
  tags:
    - payments
    - typescript
    - critical
  links:
    - url: https://payment-api.docs.your-company.com
      title: Documentation
    - url: https://grafana.your-company.com/d/payment-api
      title: Grafana Dashboard
spec:
  type: service
  lifecycle: production
  owner: group:payments-team
  system: payment-platform
  dependsOn:
    - component:user-api
    - resource:payment-service-db
  providesApis:
    - payment-api-v2

4.3 Software Templates — Self-Service Infrastructure

A Software Template is a Backstage form that, when submitted, produces a Git commit. The commit contains whatever YAML, code, or configuration the template defines.

For infrastructure provisioning, the output is a Crossplane claim. For new service scaffolding, the output is a complete service skeleton committed to a new repository.

The key design decision: templates should create pull requests, not merge directly. The PR gives platform teams visibility, gives developers a review moment, and gives everyone an audit trail. Auto-merge policies can eliminate the review step for low-risk provisioning once you've built trust in the template's outputs:

# templates/postgresql-database/template.yaml
# This template gives developers a form to request a PostgreSQL database
# The output is a Crossplane PostgreSQLDatabase claim committed to the GitOps repo
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: postgresql-database
  title: PostgreSQL Database
  description: Provision a managed PostgreSQL database on AWS RDS. Encryption, backups, and deletion protection are configured automatically by the platform.
  tags:
    - database
    - postgresql
    - aws
spec:
  owner: group:platform-team
  type: infrastructure

  # The form developers fill out in the Backstage UI
  parameters:
    - title: Database Configuration
      required: [name, team, environment, storageGB, instanceClass]
      properties:
        name:
          title: Database Name
          type: string
          description: "Lowercase, hyphens only. E.g. payment-service-db"
          pattern: '^[a-z][a-z0-9-]*$'

        team:
          title: Owning Team
          type: string
          description: "Your team name. Used for cost attribution and ownership."
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: Group

        environment:
          title: Environment
          type: string
          enum: [staging, production]
          default: staging

        storageGB:
          title: Storage (GB)
          type: integer
          minimum: 20
          maximum: 1000
          default: 50

        instanceClass:
          title: Instance Size
          type: string
          enum: [small, medium, large]
          enumNames:
            - "Small (db.t4g.medium) — dev/staging workloads"
            - "Medium (db.r7g.large) — moderate production traffic"
            - "Large (db.r7g.2xlarge) — high-throughput production"
          default: small

  # What the template does when submitted
  steps:
    - id: generate-claim
      name: Generate Crossplane Claim
      action: fetch:template
      input:
        url: ./skeleton    # Contains the Crossplane claim YAML template
        values:
          name: ${{ parameters.name }}
          team: ${{ parameters.team | parseEntityRef | pick('name') }}
          environment: ${{ parameters.environment }}
          storageGB: ${{ parameters.storageGB }}
          instanceClass: ${{ parameters.instanceClass }}

    - id: create-pr
      name: Create Pull Request to GitOps Repo
      action: publish:github:pull-request
      input:
        repoUrl: github.com?repo=gitops-repo&owner=your-org
        title: "Platform: Provision PostgreSQL database ${{ parameters.name }} for ${{ parameters.team }}"
        branchName: "provision-db-${{ parameters.name }}-${{ '' | now }}"
        description: |
          Requesting PostgreSQL database provisioned by Crossplane.

          - **Name:** ${{ parameters.name }}
          - **Team:** ${{ parameters.team }}
          - **Environment:** ${{ parameters.environment }}
          - **Storage:** ${{ parameters.storageGB }}GB
          - **Instance:** ${{ parameters.instanceClass }}

          Approve this PR to trigger provisioning. ArgoCD will pick up the change and Crossplane will create the RDS instance within ~5 minutes of merge.
        sourcePath: ./skeleton

  output:
    links:
      - title: View Pull Request
        url: ${{ steps['create-pr'].output.remoteUrl }}
      - title: Track Provisioning in ArgoCD
        url: https://argocd.your-company.com/applications

The template skeleton directory contains the Crossplane claim with template variable placeholders:

# templates/postgresql-database/skeleton/databases/${{ values.name }}.yaml
apiVersion: platform.cloudfrugal.com/v1alpha1
kind: PostgreSQLDatabase
metadata:
  name: ${{ values.name }}
  namespace: ${{ values.team }}-platform
  labels:
    team: ${{ values.team }}
    cost-centre: ${{ values.team }}-engineering
    environment: ${{ values.environment }}
    managed-by: backstage-scaffolder
spec:
  storageGB: ${{ values.storageGB }}
  instanceClass: ${{ values.instanceClass }}
  environment: ${{ values.environment }}

Part 5: Wiring It Together — The Golden Path

The Golden Path is the complete end-to-end workflow: a developer uses Backstage to request infrastructure, that request becomes a Git commit, ArgoCD applies the commit to the cluster, Crossplane provisions the actual AWS resource, and the result appears in both the Backstage catalog and the ArgoCD dashboard.

5.1 The Complete Flow

Developer fills form in Backstage
    ↓
Backstage Software Template renders the Crossplane claim YAML
    ↓
Backstage creates a Pull Request in the GitOps repository
    ↓
Platform engineer (or auto-merge policy) approves and merges the PR
    ↓
ArgoCD detects the new file in the GitOps repository
    ↓
ArgoCD applies the Crossplane claim to the cluster
    ↓
Crossplane reconciles the claim to an actual AWS RDS instance
    ↓
Developer receives the database endpoint via Kubernetes Secret
    ↓
Backstage catalog shows the new resource, owned by the requesting team

5.2 Surfacing Resource Status Back in Backstage

The Backstage Kubernetes plugin pulls live pod and resource status from your clusters and displays it on each catalog entity page. Developers can see whether their service is running, how many replicas are healthy, and whether the last deployment synced — without leaving Backstage or learning kubectl:

# Install the Kubernetes plugin packages
cd platform-portal
yarn --cwd packages/app add @backstage/plugin-kubernetes
yarn --cwd packages/backend add @backstage/plugin-kubernetes-backend

# app-config.production.yaml — add Kubernetes cluster configuration
kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'config'
      clusters:
        - name: production-eks
          url: ${PRODUCTION_CLUSTER_URL}
          authProvider: serviceAccount
          serviceAccountToken: ${PRODUCTION_SA_TOKEN}
          caData: ${PRODUCTION_CA_DATA}
        - name: staging-eks
          url: ${STAGING_CLUSTER_URL}
          authProvider: serviceAccount
          serviceAccountToken: ${STAGING_SA_TOKEN}
          caData: ${STAGING_CA_DATA}

Annotate each catalog entity to link it to its Kubernetes resources:

# In each service's catalog-info.yaml
annotations:
  backstage.io/kubernetes-label-selector: 'app=payment-api'
  backstage.io/kubernetes-namespace: payments-team

5.3 Installing the ArgoCD Plugin

The ArgoCD plugin shows deployment history and sync status directly in the Backstage entity page. When a developer opens the payment-api page in the catalog, they can see the last 10 deployments, the current sync state, and whether the application is healthy — all without opening the ArgoCD UI:

yarn --cwd packages/app add @roadiehq/backstage-plugin-argo-cd

// packages/app/src/components/catalog/EntityPage.tsx
import { EntityArgoCDOverviewCard } from '@roadiehq/backstage-plugin-argo-cd';

// Add to the service entity page layout
const serviceEntityPage = (
  <EntityLayout>
    <EntityLayout.Route path="/" title="Overview">
      <Grid container spacing={3}>
        <Grid item md={6}>
          <EntityAboutCard variant="gridItem" />
        </Grid>
        <Grid item md={6}>
          {/* ArgoCD deployment status — shows sync state and recent deployments */}
          <EntityArgoCDOverviewCard />
        </Grid>
      </Grid>
    </EntityLayout.Route>
  </EntityLayout>
);

Part 6: FinOps Integration — Cost Attribution on the IDP

An IDP that provisions resources without cost attribution creates a new problem: you now have automated infrastructure provisioning with no clear ownership of the bill it generates. Every resource created through the IDP must carry team and cost centre metadata from the moment it's provisioned.

6.1 Mandatory Labels on Every Crossplane Composition

The Crossplane Compositions are where cost attribution is enforced — not in the developer-facing claim, but in the platform layer that the developer can't bypass. These labels flow through to the actual AWS resource as tags, which means they appear in AWS Cost Explorer and can be used to build team-level cost reports:

# In every Composition, add mandatory cost attribution patches
patches:
  # These labels flow to the actual AWS resource as tags
  # They can't be omitted or overridden by the developer claim
  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[team]
    toFieldPath: spec.forProvider.tags[team]

  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[cost-centre]
    toFieldPath: spec.forProvider.tags[cost-centre]

  - type: FromCompositeFieldPath
    fromFieldPath: metadata.labels[environment]
    toFieldPath: spec.forProvider.tags[environment]

  # Add a managed-by tag to identify all IDP-provisioned resources
  - type: FromCompositeFieldPath
    fromFieldPath: metadata.name
    toFieldPath: spec.forProvider.tags[managed-by]
    transforms:
      - type: string
        string:
          fmt: "idp-crossplane"

6.2 Cost Attribution Query

With mandatory tags on every resource, you can query actual cost by team directly from AWS Cost Explorer:

# Monthly cost breakdown by team — all IDP-provisioned resources
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \
  --granularity MONTHLY \
  --filter '{
    "Tags": {
      "Key": "managed-by",
      "Values": ["idp-crossplane"]
    }
  }' \
  --group-by Type=TAG,Key=team \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Team:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table

Every team that provisions resources through the IDP now has a line on the cost report with their name on it. This is the chargeback model that makes FinOps sustainable at platform scale — attribution is automatic, not manual.

Part 7: The Platform Maturity Model — Measuring What You've Built

The CNCF Platform Engineering Maturity Model defines five levels of platform maturity. Knowing where you sit helps you decide what to build next and communicate progress to engineering leadership.

A complete Backstage + ArgoCD + Crossplane implementation, with cost attribution and Software Templates covering your most common developer requests, puts you at Level 3. Moving to Level 4 requires adding SLO alerting on the platform's own health, running quarterly developer experience surveys, and producing a monthly cost-by-team report from the attribution tags.

The most common mistake at Level 3: building more features instead of measuring adoption. A platform that has 12 Software Templates but only 2 are regularly used hasn't reached Level 3 — it's reached Level 2 with more YAML. Measure which golden paths are used, interview developers who aren't using the portal, and fix the friction before adding capabilities.

Best Practices Summary

✅ Do: Build in order — ArgoCD first, then Crossplane, then Backstage. Each layer depends on the previous one.

✅ Do: Use Backstage as a Git commit generator, not as an infrastructure caller. All infrastructure changes must be auditable Git commits.

✅ Do: Apply cost attribution tags in the Crossplane Composition layer, not in the developer claim. Attribution that developers can bypass will be bypassed.

✅ Do: Start with two or three Software Templates and make them excellent before building more. Template adoption is your most important early metric.

✅ Do: Register every service in the Backstage catalog from day one. The catalog's value is proportional to its coverage.

✅ Do: Deliver Crossplane to your cluster via ArgoCD, not helm install. Everything the IDP manages should itself be managed by the IDP.

❌ Don't: Connect Backstage directly to cloud APIs. No audit trail, no rollback, no reconciliation.

❌ Don't: Give developers the Crossplane XRD directly. The Composition abstraction exists to hide RDS-specific configuration and enforce platform standards. Bypassing it defeats the purpose.

❌ Don't: Build the IDP in isolation and announce it as done. Platform engineering is product engineering. Schedule user interviews after the first two templates are live.

❌ Don't: Skip the ArgoCD RBAC configuration. An IDP that gives all developers cluster-admin through the delivery layer has created a security problem larger than the one it solved.

Resources

• Backstage Documentation — Official reference for plugin development, Software Templates, and catalog configuration

• Crossplane Documentation — CompositeResourceDefinition and Composition reference, provider installation guides

• ArgoCD Documentation — ApplicationSet generator reference, RBAC configuration, and sync policy options

• CNCF Platform Engineering Maturity Model — The maturity framework referenced in Part 7

• AWS Provider for Crossplane — Complete reference for all AWS resource types available through Crossplane

• Backstage Kubernetes Plugin — Setup guide for the Kubernetes resource visibility integration in Part 5

• FinOps Foundation — FinOps for Platform Engineering — Framework reference for the cost attribution model in Part 6

• Companion Repository — All manifests, Compositions, ApplicationSets, and Backstage templates from this guide

• 2025 DORA State of AI-assisted Software Development Report

It changed deployment speed, infrastructure management, and the way engineering teams operate. It also changed the security landscape.

Applications that once lived on a few static servers now run across containers, Kubernetes clusters, APIs, serverless functions, and multiple cloud providers.

Many organisations moved from a handful of assets to thousands in only a few years. Yet while infrastructure evolved rapidly, penetration testing models often stayed the same.

The result is a growing mismatch. Traditional pentesting approaches were designed for environments that changed slowly. Cloud environments don't work that way.

Systems spin up and disappear within minutes. New code reaches production many times per day. Infrastructure is increasingly dynamic and distributed.

The problem isn't that traditional pentesting stopped being useful. The problem is that it stopped being enough.

In this article, you'll learn why traditional penetration testing struggles in modern cloud environments, how cloud infrastructure changes the security model, and how organisations are moving toward continuous security validation.

We'll also look at what continuous pentesting means in practice and how automation and human expertise work together.

Prerequisites: A basic understanding of cloud computing concepts such as virtual machines, containers, APIs, and CI/CD pipelines will help, but no prior penetration testing experience is required.

What We'll Cover:

• Traditional Pentesting Was Built for Stable Environments

• Infrastructure Growth Creates an Explosion of Attack Surface

• Multi-Cloud Makes Visibility Even Harder

• Speed Creates Security Gaps

• Cloud Infrastructure Is Temporary by Design

• Security Teams Need More Than Reports

• The Shift Toward Continuous Pentesting

• Cloud Changed the Rules

Traditional Pentesting Was Built for Stable Environments

For years, pentesting followed a familiar cycle. Companies defined the scope, hired security specialists, conducted an assessment, received a report, addressed the findings, and repeated the process months later.

That process worked well in traditional environments. Infrastructure was relatively static. Applications changed less frequently. Production systems remained predictable enough that a point-in-time assessment could provide value for an extended period.

A financial institution may have deployed major releases every quarter. An enterprise application might only change several times each year. Under those conditions, a pentest represented a useful snapshot of risk.

Cloud environments broke that assumption.

Today, a company running on cloud platforms like Microsoft Azure or Amazon Web Services can deploy hundreds of changes in a single week. Infrastructure teams use automation tools to create environments instantly. Engineering organisations rely on microservices that continuously evolve.

By the time a pentest report arrives, parts of the environment may already be different.

Security teams are trying to defend a moving target.

Infrastructure Growth Creates an Explosion of Attack Surface

Cloud systems rarely become simpler as organisations grow. The opposite usually happens.

A small startup may begin with a few virtual machines and a database. A larger organisation eventually accumulates APIs, serverless workloads, container clusters, identity systems, third-party integrations, CI/CD pipelines, and regional deployments.

Every new service introduces new security questions.

• Who has access?

• What permissions exist?

• Which APIs are exposed externally?

• Which workloads communicate internally?

• Where are secrets stored?

• What changed last week?

Answering those questions manually becomes increasingly difficult.

The challenge isn't simply the number of assets. It's their rate of change.

Traditional pentesting was designed around known systems and a defined scope. Cloud environments continuously generate new scope.

That difference matters.

Security teams may successfully test what exists today while missing what appears tomorrow.

Multi-Cloud Makes Visibility Even Harder

Many organisations no longer operate within a single environment.

Different teams may deploy workloads across cloud platforms for cost, capability, or business reasons. Development teams often make independent technology decisions. Acquisitions introduce entirely new infrastructure stacks.

As a result, environments become fragmented.

An organisation might run applications in AWS, analytics workloads in Azure, and internal systems elsewhere. Each environment introduces different security models, logging systems, identity controls, and operational practices.

Consistency becomes difficult.

Security teams now face a visibility problem as much as a testing problem.

The challenge is no longer just finding vulnerabilities. The challenge is knowing where testing should happen in the first place.

Large enterprises frequently discover forgotten environments, abandoned APIs, unused assets, or infrastructure that security teams never knew existed.

Traditional pentesting assumes complete visibility. But cloud environments often provide the opposite.

Speed Creates Security Gaps

Engineering organisations optimise for delivery speed. And that decision makes sense. Faster iteration creates business value.

Modern deployment systems, supported by tools from companies like GitHub and New Relic, help teams release features quickly and continuously monitor applications.

But speed creates unintended side effects.

Security processes built around manual reviews often become bottlenecks. When development teams deploy ten times each day, security teams can't manually assess every change.

This creates difficult tradeoffs: either security slows releases or releases move ahead without sufficient validation.

Neither outcome works well.

Organisations often discover a hidden reality: scaling software delivery doesn't automatically scale security operations.

The old process eventually breaks under volume.

Cloud Infrastructure Is Temporary by Design

Traditional systems generally remained active for long periods.

Cloud infrastructure behaves differently.

Containers may run briefly before replacement. Autoscaling systems create resources during peak traffic and remove them later. Development environments appear and disappear continuously.

Some assets may only exist for hours. Others may live for minutes.

This creates a serious challenge for scheduled assessments. A pentest performed on Monday might never examine infrastructure created on Wednesday.

The concept of testing a fixed environment becomes harder when the environment itself changes constantly.

Security teams increasingly need continuous awareness rather than periodic review.

The question shifts from "Did we test this?" toward "How do we know what changed?"

That's a very different operating model.

Security Teams Need More Than Reports

Traditional pentesting often ends with a report.

The report identifies findings and severity levels. Engineering teams then review and prioritise remediation work.

This approach creates delays. Findings become disconnected from operational systems. Teams manually transfer issues into workflows. Security and engineering often operate separately.

Modern engineering organisations increasingly expect security to integrate directly into development processes.

Security findings need context, ownership, and prioritisation. And most importantly, they need to fit naturally into how engineering teams already work.

A PDF delivered weeks later doesn't align well with continuous deployment environments.

Security increasingly behaves like an engineering discipline rather than an isolated review function.

The Shift Toward Continuous Pentesting

Continuous pentesting represents a shift in how organisations approach offensive security. Rather than treating penetration testing as a scheduled activity performed a few times each year, many teams now view security validation as an ongoing process that keeps pace with continuously changing infrastructure.

This approach combines continuous visibility with automation to monitor the current state of cloud environments. Instead of asking whether an assessment happened last quarter, security teams ask whether they have an accurate, real-time understanding of their attack surface.

That means continuously collecting security signals from across the environment. These signals include newly deployed internet-facing services, changes to identity and access management (IAM) permissions, vulnerable container images, exposed secrets, misconfigured storage buckets, infrastructure-as-code changes, dependency vulnerabilities, and unusual authentication or network activity.

By monitoring these signals as infrastructure evolves, teams can detect security issues soon after they appear rather than waiting for the next scheduled assessment.

Many of these checks are automated. Cloud security platforms, vulnerability scanners, infrastructure-as-code analyzers, and CI/CD pipelines continuously discover new assets, scan configurations, identify common vulnerabilities, detect exposed credentials, and monitor changes that could increase an organisation's attack surface.

Instead of producing isolated findings, modern security platforms correlate information from multiple sources to highlight issues that are most likely to represent genuine risk.

This doesn't eliminate the need for human expertise. Experienced security professionals remain essential for validating whether a vulnerability is actually exploitable, understanding business context, chaining together multiple weaknesses into realistic attack paths, prioritising remediation efforts, and performing deep manual assessments that automated tools cannot replicate.

The difference is that repetitive, high-volume work increasingly becomes automated, allowing security teams to spend less time discovering obvious issues and more time investigating the complex risks that require human judgment.

Platforms such as XBOW reflect this broader shift. As cloud environments become larger and more dynamic, organisations increasingly need systems that continuously validate changing infrastructure and provide ongoing visibility into their security posture rather than relying solely on periodic assessment cycles.

The objective isn't to replace people. It's to enable security professionals to focus their expertise where it delivers the most value while automation handles the scale and speed of modern cloud infrastructure.

Cloud Changed the Rules

The central challenge isn't that security teams suddenly became less effective. The environment changed.

Traditional pentesting evolved in a world of stable infrastructure, predictable deployments, and relatively fixed boundaries.

Cloud systems operate differently. Infrastructure changes continuously. Assets appear and disappear rapidly. Development cycles accelerate. Scope expands faster than manual processes can handle.

The security practices that worked well ten years ago are colliding with modern infrastructure realities.

Organisations that recognise this shift early are changing how they think about security operations. They're moving away from isolated assessments and toward continuous visibility.

Because in cloud environments, risk is no longer static. So security can't be static either.

Hope you enjoyed this article. You can connect with me on LinkedIn.

When you learn Terraform, you work with a single repository, state file, and a single environment. You run terraform apply from your laptop, and your infrastructure is provisioned.

That model works fine until the day you join a company and realize engineers rarely apply to production from a laptop.
A lot of what you see will not match what you practiced.

This article explains how large engineering teams actually run Terraform, the repositories, workflows, ownership rules, and what goes wrong without them.

You'll learn how enterprise teams structure repositories and state files, how they store and version reusable modules through GitHub, why infrastructure changes move to production through pipelines, how they catch changes that happen outside of Terraform, and how they recover when things go wrong.

Every practice here exists because a team hit a specific wall and built something to get past it.

Prerequisites

You should be comfortable with Terraform before reading this.
You should also know how Git pull requests and branch merging work.

This is not a Terraform introduction, it is about what happens after you have learned the basics and start sharing infrastructure with other engineers.

Table of Contents

• How State Corruption Happens

• Why State File Gets Treated Like a Production Database

• How Enterprise Teams Structure Their Terraform Repositories

• How Teams Split State Files to Protect Each Other

• Why Some Teams Prefer Directories Over Workspaces for Production

• How Teams Share Infrastructure Through Modules on GitHub

• How Teams Version and Release Terraform Modules

• How Teams Maintain Terraform Modules at Scale

• How Teams Share Data Between State Files

• How Infrastructure Changes Actually Move to Production

• How Teams Detect Infrastructure Drift

• How Teams Recover When State Goes Wrong

• Conclusion

How State Corruption Happens

The state file is how Terraform tracks what it has built. It remembers every resource, every ID, and every configuration value. When it gets out of sync with what actually exists in the cloud, that's state corruption.

It gets blamed for a lot of things. But engineers who have dealt with it in production know it usually traces back to one of a handful of situations, each with a different cause and a different fix.

Two Engineers Run terraform apply at the Same Time

Before understanding this one, you need to understand something about how Terraform works.

When you run terraform apply, two things happen separately:

First, Terraform talks to AWS, and the resource gets created in the cloud. Second, Terraform updates the state file to record what was just built.

These are two different systems. AWS holds the real infrastructure, and the state file is Terraform's notebook about it. If anything interrupts the process between step one and step two, they fall out of sync.

Now here's what happens when two engineers apply at the same time without locking:

Sarah opens the state file and starts adding a subnet. Marcus opens the same state file at the same moment and starts updating a NAT gateway. Both are working from the same starting copy.

Sarah finishes first. Her apply creates the subnet in AWS and updates the state file to record it.

Marcus finishes second. His apply updates the NAT gateway in AWS. Terraform then updates the state file using the version of state Marcus read when his apply started.

That version didn't include Sarah's subnet, so the updated state no longer contains a record of it.

The subnet exists in AWS. But Terraform's notebook no longer has a record of it. The next terraform plan thinks the subnet was never created and proposes building it again.

State locking prevents this. Sarah's apply acquires a lock before it starts. When Marcus tries to apply, Terraform makes him wait.

After Sarah finishes, Terraform updates the state file and releases the lock. Marcus then runs against the updated state, so both the subnet and NAT gateway changes are recorded correctly.

An Apply Gets Interrupted

A GitHub Actions pipeline is applying changes to the payments infrastructure, adding three new security group rules and a database parameter group. Halfway through, the pipeline runner hits its 60-minute timeout limit, and the job gets killed.

Here's what the apply actually managed to do before dying:

The terminal image above shows three security group rules completing successfully before the pipeline hits its 60-minute runtime limit. The runner is then terminated. The database parameter group never finishes creating, and the state file update never runs because the job died first.

Security group rule 1  → created ✓
Security group rule 2  → created ✓
Security group rule 3  → created ✓
Database parameter     → not created ✗
State file update      → never wrote (job died first)

The three security group rules now exist in AWS. The problem is that the pipeline died before Terraform could finish updating the state file. AWS knows the rules exist. Terraform's state file does not.

At this point, reality and the state file no longer match.

Fortunately, this is usually easy to recover from. When the pipeline runs again, Terraform checks what already exists in AWS. It sees the three security group rules and doesn't try to create them again. It then creates the database parameter group that never got built.

The second run completes successfully and the state file catches up.

This works because Terraform is idempotent, running the same configuration again moves infrastructure toward the desired state rather than blindly creating everything from scratch.

One small complication remains: the state lock.

If the pipeline was interrupted while holding a lock, Terraform may still think another apply is running. The next pipeline run fails immediately with an error like this:

The terminal above shows terraform apply failing because the previous job left a state lock behind. The error includes the lock ID, the path to the state file, and the name of the process that acquired it. Terraform refuses to proceed until the lock is released or manually cleared.

Before clearing the lock, make sure no Terraform apply is still running.

Open your CI/CD system. GitHub Actions, GitLab CI, Jenkins, or whatever your team uses and check the pipeline history for that environment:

The GitHub Actions pipeline history image above shows four recent runs. terraform-plan completed successfully. Two terraform-apply jobs show as cancelled and timed out, both flagged as lock may be stale. A fourth terraform-apply job is currently in progress, and this one shouldn't be unlocked until it finishes.

If the previous apply was cancelled or timed out, the lock is stale. Clear it with terraform force-unlock plus the lock ID from the error. The pipeline then runs normally.

Only force-unlock when you're certain nothing is actively running. Clearing a live lock lets two applies write to the same state at the same time, which is exactly the problem locking was built to prevent.

Someone Runs a Terraform State Command in the Wrong Environment

A database engineer is cleaning up an old test database in the staging environment.

The database still exists in AWS, but Terraform should stop managing it. To do that, the engineer uses terraform state rm.

This command doesn't delete anything in AWS. It only removes Terraform's record of the resource from the state file. Think of it as telling Terraform: "forget this resource exists, but leave it running."

The engineer intends to run it against staging:

Intended:  staging state       → forget the old test database

But they're working in the wrong directory. They run it against production instead.

Actual:    production state    → forget the live payments database

Nothing gets deleted. The production database is still running in AWS. But Terraform has now forgotten it exists.

Now Terraform and reality disagree. The next terraform plan sees a database defined in the code but missing from the state file, so it assumes the database doesn't exist and proposes creating a new one.

If nobody catches it in the plan output, Terraform creates a second production database alongside the original: two databases running in production, neither fully managed, and a very expensive mess to untangle.

terraform state rm, terraform import, and terraform state mv make immediate changes to the state file with no confirmation prompt. Run them from the wrong directory, the wrong workspace, or with the wrong resource address and you change the wrong state in seconds.

Two Teams Manage the Same Resource

The networking team owns a security group that controls access to the payments database. When a new microservice needs database access, a payments engineer has two options: ask the networking team to add a new rule, or manage the security group themselves.

They choose the second option. The engineer imports the existing security group into the payments state file and adds a rule for Microservice C.
From that moment, both teams think they own the same security group.

The problem is that Terraform does exactly what each state file tells it to do. The networking state says the security group should allow A and B. The payments state says it should allow A, B, and Microservice C. Both can't be true at the same time.

When the payments team applies their state, Microservice C gets access. But later that night, the networking pipeline runs. Terraform reads the networking state, sees only A and B, and updates the security group to match. Microservice C's rule disappears silently.

No errors are seen and both pipelines pass, which is exactly what makes this so hard to debug. Terraform isn't broken, it's receiving conflicting instructions from two different state files and doing exactly what each one says.

This isn't something to be fixed with Terraform commands. It's an ownership decision that should have been made before anyone ran an import. If the payments team had submitted a pull request to the networking repository asking them to add the rule, one team would own the security group, one state file would manage it, and the conflict could never have happened.

Why State File Gets Treated Like a Production Database

The state file looks like bookkeeping: a record of what Terraform created. The reason teams treat it differently is that it often contains secrets.

The state file stores sensitive values in plaintext. Database passwords, API keys, connection strings – if those values were passed to a Terraform resource during an apply, they're now sitting in the state file. Even if you marked the variable as sensitive in your Terraform code, the value still lands in the state file. Terraform needs it there to compute diffs on future plans.

That means: whoever can read the state file can read your database password.

In large organizations, engineers typically don't have direct access to the production state bucket. Instead, Terraform runs through a CI/CD pipeline that assumes a dedicated IAM role with permission to read and write the state bucket and perform applies. Engineers interact with infrastructure through pull requests and plan output, not by touching the state bucket directly.

This separation reduces risk and creates an audit trail. Every state change is performed by the pipeline and logged, making it straightforward to trace what changed and when.

How Enterprise Teams Structure Their Terraform Repositories

When you join a large engineering organization, the first thing you notice is the number of repositories. You might expect one repository for all infrastructure, but what you find is dozens.

The structure maps directly to ownership. Each repository belongs to one team, and that team is responsible for everything in it. A typical layout looks like this:

The diagram shows two types of repositories. The first type belongs to the platform team and contains reusable modules: things like VPC configurations, database templates, and security group patterns. These repositories don't create production resources directly.

The second type belongs to individual product teams, such as the payments team or the auth team. These repositories call the platform modules and use them to build their actual infrastructure. A mistake in a product team repository affects only that team. A mistake in a shared platform module can affect every team that depends on it.

The key thing to understand here is that the platform team repositories don't create production resources. They create reusable modules that the product teams call when building their actual infrastructure.

That distinction matters because some repositories are used by one team, while others are shared by everyone.

A mistake in a product team's repository usually affects only that team. A mistake in a shared module can affect every team that depends on it.

The diagram illustrates why shared repositories carry more risk than product-specific ones. A bug in the payments-infra repository affects only the payments team. A bug in the terraform-aws-postgres module affects every team that uses it to provision databases. A bug in the terraform-policies repository affects every pipeline in the company. The wider the module is shared, the larger the blast radius when something goes wrong.

This is why experienced engineers pay close attention to shared modules and policy repositories.

If the payments team's infrastructure breaks, the problem is probably in the payments repository.

If five different teams start seeing the same issue at the same time, the shared modules and policy repositories become the first place to investigate.

How Teams Split State Files to Protect Each Other

A single state file managing everything, VPC, Kubernetes cluster, databases, monitoring, is fine when one person is running things, but quickly becomes a problem when multiple teams share it.

Three specific problems emerge.

1. Blast radius: If the networking configuration and the database configuration live in the same state file, a bad networking apply can accidentally affect database resources that had nothing to do with the change. Separate state files keep failures contained.

2. Deployment speed: Networking infrastructure might change a few times a year. Applications might deploy dozens of times a day. If they share a state file, teams end up waiting on each other's locks.

3. Ownership conflicts: When multiple teams share a state file, one team can change something the other team depends on without realizing it.

The solution is to split state along ownership boundaries. A structure that addresses all three problems looks like this:

The structure image above shows one state file per domain under a production folder.

• networking handles VPC, subnets, routing, and NAT gateways.

• identity handles IAM roles, policies, and service accounts.

• platform handles the Kubernetes cluster, node pools, and add-ons.

• database handles RDS instances, Redis clusters, and backups.

• security handles security groups, WAF rules, and certificates.

• monitoring handles Prometheus, Grafana, and alerting pipelines.

• payments handles payment service infrastructure.

production/
  networking/terraform.tfstate   → VPC, subnets, routing, NAT gateways
  identity/terraform.tfstate     → IAM roles, policies, service accounts
  platform/terraform.tfstate     → Kubernetes cluster, node pools, add-ons
  database/terraform.tfstate     → RDS instances, Redis clusters, backups
  security/terraform.tfstate     → Security groups, WAF rules, certificates
  monitoring/terraform.tfstate   → Prometheus, Grafana, alerting pipelines
  payments/terraform.tfstate     → Payment service infrastructure

This is one example, not a universal standard. Larger organizations often split further. The principle is the same: one owning team per state file, one pipeline, one blast radius.

The rule is simple: every resource belongs to one state file. If the networking team owns a security group, it stays in the networking state. Other teams can reference it as a data source, but they don't import it into their own state.
That is what prevents the ownership collision described in the first section.

Why Some Teams Prefer Directories Over Workspaces for Production

Terraform CLI workspaces let you manage multiple environments like dev, staging, and production from a single directory. Each workspace gets its own state file, but they all share the same .tf configuration files.

infra/
  main.tf          ← same code runs for ALL environments
  variables.tf

  terraform.tfstate.d/
    dev/
    staging/
    production/    ← separate state, same code

The workspace approach keeps all environments in one directory called infra. It contains a single main.tf file that runs for all environments. State is stored separately under terraform.tfstate.d with folders for dev, staging, and production, but all three share the same code.

You switch environments with terraform workspace select production, then apply.

The risk is that switching workspaces is a manual step. If the wrong workspace is active, changes meant for staging can end up in production.

Many teams prefer separate directories for long-lived environments:

environments/
  dev/
    main.tf      ← its own code path
    backend.tf   ← points to the dev state bucket
  staging/
    main.tf      ← its own code path
    backend.tf   ← points to the staging state bucket
  production/
    main.tf      ← its own code path
    backend.tf   ← points to the production state bucket

The directory approach gives each environment its own folder under environments. Dev, staging, and production each have their own main.tf with a separate code path, and their own backend.tf pointing to a different state bucket. The environments are completely separate from each other.

To apply against production, you have to be in the production directory. Each environment has its own state, backend, and execution path.

The tradeoff is duplication. Teams usually solve that with shared modules, so each environment directory contains only environment-specific configuration.

Workspaces are still useful for short-lived environments such as feature branches, preview deployments, and temporary test infrastructure.

How Teams Share Infrastructure Through Modules on GitHub

When 30 teams each need a PostgreSQL database, two things happen.

Without a shared standard, every team writes their own database configuration. Six months later, a security audit runs across all environments and finds that:

The diagram shows what a security audit found when four teams each wrote their own database configuration independently.

Team A set backup_retention_period = 0, meaning their database was never backed up. Team B set storage_encrypted = false, leaving data in plaintext. Team C passed an empty tags = {}, so there was no cost tracking. Team D set deletion_protection = false, leaving the database one accident away from permanent data loss.

Nobody skipped those things on purpose, there was just no shared standard.

With a shared module, the platform team writes a postgres module once. They encode every organizational requirement into it: encryption on, 7-day backups, monitoring alarms, required tags, deletion protection enabled. They publish it to a GitHub repository called terraform-aws-postgres.

Every team that needs a database now writes this:

module "payments_db" {
  source         = "git::ssh://github.company.com/platform/terraform-aws-postgres.git?ref=v2.1.0"
  name           = "payments"
  environment    = "production"
  instance_class = "db.m5.large"
}

Four inputs. Everything else is handled by the module.

Large organizations usually expose approved modules through an internal registry so engineers can discover and version them without browsing GitHub repositories. Instead of the full Git URL, the reference becomes:

module "payments_db" {
  source  = "app.terraform.io/mycompany/postgres/aws"
  version = "~> 2.1"
}

HCP Terraform and Terraform Enterprise both include a private registry that connects to GitHub, watches for version tags on module repositories, and publishes new versions automatically.

How Teams Version and Release Terraform Modules

The ?ref=v2.1.0 in a module source URL isn't decoration. At the scale of 40 teams sharing one module, it's the thing that prevents a well-intentioned change from becoming a company-wide incident.

Without version pinning, the payments team references the Postgres module from main meaning whatever the latest code is at any given moment. The module owners rename an output variable from db_endpoint to database_endpoint to match a new naming convention. The next time any team runs terraform init, they pull that change. Their configuration still references db_endpoint.

Plans break:

payments-infra                        → plan fails
analytics-infra                       → plan fails
auth-infra                            → plan fails
reporting-infra                       → plan fails

Version pinning prevents this. The payments team stays on v2.1.0. The module owners release v2.2.0 with the renamed output and write a changelog. Teams upgrade when they're ready, after testing in staging. Nobody's pipeline breaks without warning.

The versioning convention is called semantic versioning:

v2.1.1  → patch:  bug fix. Safe to upgrade. Nothing to change in your code.
v2.2.0  → minor:  new optional feature. Safe to upgrade. Nothing to change.
v3.0.0  → major:  breaking change. Read the changelog. Update your code first.

The table shows three version types. A patch version like v2.1.1 means a bug fix, safe to upgrade with nothing to change in your code. A minor version like v2.2.0 means a new optional feature, also safe to upgrade with nothing to change. A major version like v3.0.0 means a breaking change, so you need to read the changelog and update your code before upgrading.

How Teams Maintain Terraform Modules at Scale

Building a Terraform module takes an afternoon, bit maintaining it for two years is a different job entirely.

A networking engineer needs a VPC module. The platform team has one, but their backlog is full. So the engineer creates a slightly different version. Three months later, another team does the same. Then another. Now this exists:

terraform-aws-vpc           ← original, maintained by platform team
terraform-aws-vpc-v2        ← created by the app team, author unknown
terraform-aws-vpc-shared    ← no idea which environments use this
terraform-aws-vpc-prod      ← unclear if this was ever different from the original

No one created a module graveyard on purpose. It grew one "I'll just make a quick variation" at a time. Each variant has slightly different security settings, different tagging, different defaults. When a compliance audit requires all VPCs to enable flow logging, the team has to investigate four different modules to figure out which environments are compliant.

Teams that avoid this treat their modules like shared services: named owner, contributions through pull requests, breaking changes in major versions with a migration guide, and deprecated modules with a retirement date. A CODEOWNERS file routes every pull request to the right reviewer automatically.

Organizations that skip this end up with modules that nobody owns, nobody wants to touch, and nobody is sure can be safely removed.

How Teams Share Data Between State Files

Once infrastructure is split into separate state files, a practical problem surfaces: teams need information from each other's infrastructure. The platform team's Kubernetes state needs the VPC ID from the networking team's state. The database state needs subnet IDs. The payments state needs the database endpoint.

Two patterns exist for solving this.

Reading Another Team's State Outputs

The terraform_remote_state data source lets one state read the outputs of another. The networking team marks their VPC ID and subnet IDs as outputs. The database team reads those outputs and uses them to place databases in the right subnets.

Networking state
  └── outputs: vpc_id, private_subnet_ids
                          ↓
               Database state reads them
               └── places RDS in the right subnets

This works, but there's a limitation. Reading another team's state requires full read access to their entire state file, not just the outputs you want. State files contain database passwords and API keys in plaintext. More dependencies means more teams reading each other's secrets.

Looking Up Resources Directly From the Cloud

The alternative, and the one HashiCorp now recommends, is to look up resources through the cloud provider's API instead of reading another team's state:

data "aws_vpc" "main" {
  tags = {
    Name        = "production-vpc"
    Environment = "production"
  }
}

No cross-team state access needed, and each team's state stays isolated. The tradeoff is consistent tagging: the networking team has to tag their VPC in a way the database team can reliably search for, which forces teams to agree on naming conventions early.

Many teams use both. Remote state for a small number of trusted, tightly coupled dependencies. Cloud data sources for everything broader.

How Infrastructure Changes Actually Move to Production

In large organizations managing production Terraform at scale, changes don't come from someone's laptop. Applying directly from a local machine requires production cloud credentials sitting on that machine, a security risk and leaves no audit trail if something breaks.

Instead, production changes move through a pipeline. Every change goes through a pull request in GitHub, and the pipeline does the work:

Engineer opens a pull request
        ↓
Pipeline: terraform validate + fmt check
        ↓
Pipeline: security scan (Checkov, tfsec, or similar)
        ↓
Pipeline: terraform plan → posts the full output as a comment on the PR
        ↓
Reviewer reads the plan output (not just the code)
        ↓
Required reviewers approve (enforced by CODEOWNERS + branch protection)
        ↓
Merge triggers the apply pipeline
        ↓
Pipeline: acquires state lock → applies → releases lock → logs result

The diagram above shows eight steps in order. An engineer opens a pull request. The pipeline runs terraform validate and a format check. A security scan runs using Checkov, tfsec, or similar. The pipeline runs terraform plan and posts the output as a comment on the pull request. A reviewer reads the full plan output. Required reviewers approve, enforced by CODEOWNERS and branch protection rules. Merging triggers the apply pipeline. The pipeline acquires the state lock, applies the changes, releases the lock, and logs the result.

The part that surprises engineers when they first encounter this is that the reviewer isn't approving the code. They're approving the plan output and the list of exactly what will be created, changed, or destroyed in the cloud.

A code change can look completely harmless and produce a destructive plan. Changing one database parameter might force a resource replacement, meaning Terraform destroys the current database and creates a new one. Seeing this in the plan output before the PR merges:

# aws_db_instance.payments must be replaced
-/+ resource "aws_db_instance" "payments" {

The image above shows a plan output that aws_db_instance.payments must be replaced, meaning Terraform will destroy the existing database and create a new one, not update it in place.

Catching that before merge is the entire point of reviewing the plan. Not the code.

How CODEOWNERS Enforces Who Reviews What

Earlier, we talked about module ownership. A VPC module might belong to the platform team, while database infrastructure belongs to the database team.

The challenge is making sure changes are actually reviewed by the people who own them.

GitHub solves this with a feature called CODEOWNERS. It lets a repository define which team is responsible for which directories. When someone opens a pull request that touches those files, GitHub automatically requests reviews from the correct team.

For example, if an engineer modifies the PostgreSQL module, GitHub can automatically require approval from the platform team before the change can be merged.

Without CODEOWNERS, engineers have to remember who owns which parts of the infrastructure.

CODEOWNERS makes ownership explicit and automatically requests reviews from the right team.

How Teams Detect Infrastructure Drift

Drift is the diff between what Terraform says should exist and what actually exists in the cloud.

Here's the scenario that produces drift more reliably than anything else:

Monday 3:00 AM  Production database CPU spikes. Outage.
Monday 3:15 AM  Engineer resizes database in AWS console: db.m5.large → db.m5.4xlarge
Monday 3:20 AM  Incident resolved. Engineer goes to sleep.
Monday 3:21 AM  Terraform state file: still says db.m5.large

The incident is forgotten, the ticket is closed, and life moves on.

Three months later, a routine Terraform apply runs. Terraform sees db.m5.large in the configuration but finds db.m5.4xlarge running in AWS. From Terraform's perspective, the database is larger than it should be, so the plan proposes changing it back.

Nobody notices the change in the plan output. The apply goes through, the database is downsized, and users begin reporting slow queries. The team spends hours investigating before eventually tracing the issue back to a Terraform change that reverted the emergency fix from months earlier.

Teams that handle this well run scheduled terraform plan jobs against every production state. If terraform plan exits with code 2, differences were found and an alert fires. The team then decides whether to apply to restore declared state or update the configuration to match reality. Either way, the change is visible and deliberate. Invisible drift always gets worse.

How Teams Recover When State Goes Wrong

State is recoverable in almost every situation, as long as the team set things up correctly before the incident happened.

The teams that recover in twenty minutes instead of three days aren't the ones with the deepest Terraform expertise. They're the ones who prepared.

Step 1: Pull a Backup Before Touching Anything.

terraform state pull > backup-$(date +%Y%m%d-%H%M%S).json

This saves the current state to a local file. Whatever you try next, you have a starting point to return to.

Step 2: Run terraform plan and Look at What it Proposes.

If Terraform proposes destroying resources that still exist in the cloud, the state is behind reality. If it proposes creating resources that already exist, reality is ahead of the state. Either way, the plan output tells you which direction the mismatch runs.

Step 3: Restore from S3 Versioning if the State is Corrupted.

Every write to a versioned S3 bucket saves a new version automatically. If the state file is corrupted or wrong, list the previous versions, download the last known good one, and push it back:

# List previous versions
aws s3api list-object-versions \
  --bucket mycompany-terraform-state \
  --prefix production/database/terraform.tfstate

# Download a specific version
aws s3api get-object \
  --bucket mycompany-terraform-state \
  --key production/database/terraform.tfstate \
  --version-id "the-version-id-here" \
  recovered-state.json

# Push it back
terraform state push recovered-state.json

Run terraform plan after restoring to confirm it looks correct before running any apply.

Step 4: Clear a Stale Lock if the Pipeline is Blocked.

If a lock was never released after a failed apply, clear it:

terraform force-unlock LOCK_ID

Only do this after confirming no apply is actively running. Clearing a live lock corrupts the state.

Step 5: Re-import Resources That Fell Out of State.

If a resource exists in the cloud but Terraform no longer knows about it — because of an accidental terraform state rm — bring it back without recreating it:

terraform import aws_db_instance.payments db-ABCD1234EFGH5678

Run terraform plan after importing to confirm no unexpected changes are proposed.

Conclusion

Every practice in this article traces back to a specific problem teams ran into as Terraform usage grew.

State locking prevents engineers from overwriting each other's changes.
State splitting reduces blast radius. Module versioning prevents shared infrastructure from breaking unexpectedly. Drift detection catches changes made outside Terraform. CODEOWNERS ensures the right people review the right changes.

Different problems with different solutions. But they all point to the same underlying theme which is ownership.

As teams grow, many Terraform problems have less to do with infrastructure and more to do with ownership.

State collisions happen when multiple people can modify the same state.
Module sprawl happens when nobody is responsible for maintaining a shared standard.

Drift becomes dangerous when changes are made without anyone taking ownership of bringing Terraform and reality back into alignment. Even review bottlenecks often trace back to uncertainty about who should approve what.

Understanding this changes how you read an unfamiliar Terraform repository.

Dozens of small state files aren't necessarily over-engineering. They're often ownership boundaries. A CODEOWNERS file is not bureaucracy. It's an ownership map. A pipeline that posts plan output on a pull request isn't just automation, it's a review process built around infrastructure consequences rather than code.

The infrastructure matters. But as teams grow, ownership is what keeps the system understandable.

I write about DevOps engineering, production systems, and the things tutorials do not cover weekly. If this was useful, please join the newsletter.
If you enjoyed reading this, we can also connect on Linkedin.

I've audited EKS clusters at more than 10 companies. The same waste patterns appear every time: over-provisioned nodes, cross-AZ data transfer, idle EBS volumes, and so on. And the most expensive mistake of all: buying compute commitments before rightsizing.

This handbook is the fix. I've used this 7-step playbook to reduce EKS costs by 50–60% at every company where I've implemented it. There are no product code changes, and no downtime. Just infrastructure optimization executed in the right order.

By the end of this guide, you'll know how to right-size pod resource requests, implement Karpenter for intelligent bin-packing and Spot diversification, migrate compatible workloads to Graviton for 20% cheaper compute, and eliminate NAT Gateway charges entirely with VPC endpoints.

All Terraform modules, NodePool templates, and automation scripts referenced in this guide are available in the companion repository at github.com/aayostem/eks-cost-optimization. The repo includes ready-to-deploy configurations for every step so you can move from reading to implementing in the same afternoon.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: The Baseline — Where Your EKS Money Is Going

• Part 2: Right-Sizing Pod Resource Requests

• Part 3: Karpenter for Bin-Packing and Spot Diversification

• Part 4: Graviton Migration

• Part 5: VPC Endpoints for Data Transfer

• Part 6: EBS Volume Optimisation

• Part 7: Load Balancer Consolidation

• The Complete 7-Step Sequence

• Best Practices Summary

• Resources

What You'll Learn

• How to right-size pod resource requests using VPA recommendations

• The complete Karpenter setup with Spot diversification and automatic consolidation

• Graviton3 migration for all non-GPU workloads

• VPC endpoints to eliminate NAT Gateway data transfer charges

• EBS gp2 to gp3 migration — 20% cheaper with zero performance loss

• Load balancer consolidation with shared Ingress

• The 7-step sequence that maximises ROI — and why the order isn't optional

Let's dive in.

Prerequisites

Before following along, you should have:

Knowledge:

• Working familiarity with Kubernetes — you can deploy an application and inspect pods

• Basic AWS knowledge — you understand EC2 instance types, VPCs, and EBS volumes

• Comfort reading Terraform HCL and Kubernetes YAML

Tools and access:

• An existing EKS cluster running Kubernetes 1.27 or later

• kubectl configured and pointing at your cluster

• AWS CLI v2 installed and authenticated with appropriate permissions

• Helm 3 installed (for Karpenter and Kubecost)

• Metrics Server installed in your cluster

Companion repository: Clone the repo before starting. It contains all YAML, Terraform, and shell scripts referenced in this guide:

git clone https://github.com/aayostem/eks-cost-optimization
cd eks-cost-optimization

Estimated savings: For a cluster running at $85,000/month with typical over-provisioning, expect $40,000 to $55,000/month in savings after completing all 7 steps. Smaller clusters under $10,000/month typically see 40–50% reduction.

Part 1: The Baseline — Where Your EKS Money Is Going

1.1 The Typical EKS Cost Breakdown

Before touching anything, you need to know exactly where the money is going. Optimising the wrong category first is how teams waste weeks of engineering time and see no meaningful reduction.

Here's what a typical $85,000/month EKS cluster looks like when you break it down:

Compute and Data Transfer together represent 79% of the bill and account for 90% of the correctable waste. Those are the targets.

Run this command to see your own breakdown before starting anything:

# Pull last month's cost breakdown by service
# Save this output — it becomes your before number
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \
  --granularity MONTHLY \
  --group-by Type=DIMENSION,Key=SERVICE \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table | sort -k3 -rn

Screenshot the output and save it. You'll compare against it after each step to verify actual savings before moving to the next one.

1.2 The Most Expensive Mistake: Wrong Optimisation Order

Here's what most teams do when they get a large AWS bill:

1. Buy Savings Plans immediately, locking in waste at a 30% discount

2. Then implement Karpenter, discovering they've over-committed the wrong instance family

3. Then migrate to Graviton, discovering their Savings Plan doesn't cover ARM instances

The result: a 12–36 month commitment paying for waste they could have eliminated in three weeks.

The correct sequence is:

Step 1: Right-size pod requests        ← Always first
Step 2: Implement Karpenter            ← Dynamic provisioning on rightsized requests
Step 3: Enable Spot for non-prod       ← Karpenter handles fallback automatically
Step 4: Migrate to Graviton            ← Karpenter makes this seamless
Step 5: Add VPC endpoints              ← Eliminate data transfer charges
Step 6: Optimise EBS volumes           ← Quick win, run alongside other steps
Step 7: Consolidate load balancers     ← Final structural cleanup

Then, and only then, buy Savings Plans — against the optimised baseline you've just established.

The one rule: optimise first, then commit. Every step before the Savings Plan purchase reduces what you're locking in for 1–3 years.

Part 2: Right-Sizing Pod Resource Requests

2.1 Why Over-Provisioned Requests Are So Expensive

Kubernetes schedules pods based on resource requests — not actual usage. A pod that requests 2 vCPUs and 4GB of memory requires a node with that capacity available, regardless of whether the pod is actually using it.

Here's the incorrect approach with the requests set to worst-case peak estimates:

# Bad: Resource requests set during initial deployment, never revisited
# This pod actually uses 250m CPU and 512Mi memory on average
resources:
  requests:
    cpu: "2"        # 8x more than actual usage
    memory: "4Gi"   # 8x more than actual usage
  limits:
    cpu: "4"
    memory: "8Gi"

When every pod is over-requested by 8x, your cluster needs 8x more nodes than your workloads actually require. That's where the 61% compute line in your bill comes from.

First, verify actual usage before changing anything:

# Install Metrics Server if not already running
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Check actual CPU and memory usage per pod
# Compare these numbers against your current resource requests
kubectl top pods --all-namespaces --sort-by=cpu

Expected output showing the typical gap:

NAMESPACE     NAME                    CPU(cores)   MEMORY(bytes)
production    payment-api-xxx         25m          128Mi
production    user-api-xxx            15m          96Mi
production    notification-svc-xxx    5m           64Mi
staging       worker-xxx              10m          256Mi

If your pods are requesting 2 CPU cores each but using 25m–15m cores in practice, you have a 50–80x over-request ratio. Every node in your cluster is mostly empty space you're paying for.

2.2 Using the Vertical Pod Autoscaler for Recommendations

The Vertical Pod Autoscaler (VPA) is a Kubernetes component that analyses historical CPU and memory usage for each deployment and recommends optimal resource requests. You use it in recommendation-only mode first — it tells you what to set without changing anything automatically, so you can review and apply the changes yourself with full control.

Here's the correct implementation:

# Good: VPA in recommendation-only mode
# Watches your pod's actual usage for 24+ hours, then recommends right-sized requests
# updateMode: "Off" means it only recommends — it never restarts your pods
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: payment-api-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  updatePolicy:
    updateMode: "Off"   # Recommendation only — you apply manually after review
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      minAllowed:
        cpu: "100m"     # VPA will never recommend below this floor
        memory: "256Mi"
      maxAllowed:
        cpu: "2"        # VPA will never recommend above this ceiling
        memory: "4Gi"

Install VPA and retrieve recommendations:

# Install VPA components
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/download/vertical-pod-autoscaler-1.0.0/vpa-v1.0.0.yaml

# Apply the VPA manifest for each deployment you want to right-size
kubectl apply -f vpa/payment-api-vpa.yaml

# Wait 24 hours for VPA to collect usage data, then check recommendations
kubectl describe vpa payment-api-vpa -n production

What a VPA recommendation looks like:

Recommendation:
  Container Recommendations:
    Container Name: payment-api
    Lower Bound:
      cpu:     50m
      memory:  128Mi
    Target:
      cpu:     250m      ← Set your requests to this value
      memory:  512Mi     ← Set your requests to this value
    Upper Bound:
      cpu:     500m
      memory:  1Gi

Apply the recommendation to your deployment:

# Good: Right-sized requests based on VPA Target recommendation
resources:
  requests:
    cpu: "250m"     # Down from 2000m — an 8x reduction
    memory: "512Mi" # Down from 4096Mi — an 8x reduction
  limits:
    cpu: "500m"     # 2x the request — headroom for genuine spikes
    memory: "1Gi"   # 2x the request

All VPA manifests for common deployment types are in vpa/ in the companion repo.

2.3 The ROI of Right-Sizing

Verify the improvement after applying recommendations:

# Check cluster-level utilisation after right-sizing
# Target: 60–75% CPU and memory utilisation across nodes
kubectl top nodes

Part 3: Karpenter for Bin-Packing and Spot Diversification

Karpenter is an open-source Kubernetes node provisioner built by AWS and donated to the CNCF.

Where the default Kubernetes Cluster Autoscaler scales pre-configured node groups up and down, Karpenter watches the actual resource requests of pending pods and provisions exactly the right EC2 instance type to satisfy them — selecting dynamically from thousands of available instance families rather than the two or three you pre-configured. It also continuously monitors running nodes for underutilisation and consolidates workloads onto fewer nodes, terminating the empty ones automatically.

The result is a cluster that is always sized to what your workloads actually need right now, not what you anticipated at setup time.

3.1 The Ceiling with Cluster Autoscaler

Cluster Autoscaler works with pre-defined node groups. You configure which instance types are available and it scales those groups up and down.

The limitation is that it can only provision instances from the types you pre-configured. It can't dynamically select the right instance type based on what the workload actually needs right now.

Here's the incorrect approach using static node groups:

# Bad: Two static node groups, each over-provisioning against worst-case scenarios
# CPU-optimised group runs even when workloads are memory-bound
# Memory-optimised group runs even when workloads are CPU-bound
eksctl create nodegroup \
  --cluster my-cluster \
  --name cpu-optimized \
  --instance-types c5.2xlarge \
  --nodes-min 5 --nodes-max 20

eksctl create nodegroup \
  --cluster my-cluster \
  --name memory-optimized \
  --instance-types r5.2xlarge \
  --nodes-min 3 --nodes-max 10

You're provisioning for the worst case in each family simultaneously. At any given moment, one group is underutilised while the other is scaling. Neither is right.

3.2 How Karpenter Solves This

Karpenter watches the actual resource requests of pending pods and provisions exactly the right instance type to fit them. It selects from thousands of available instance types, not just the two you pre-configured. It also consolidates running workloads onto fewer nodes when utilisation drops, automatically terminating underutilised nodes.

Here's the correct implementation:

# Good: Karpenter NodePool
# Karpenter selects the optimal instance type based on pending pod requirements
# Tries Spot first, falls back to On-Demand automatically when Spot isn't available
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        # Allow both x86 and ARM (Graviton) — Karpenter picks the cheaper option
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        # Try Spot first, fall back to On-Demand if unavailable
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        # Exclude families with poor price-to-performance ratio
        - key: karpenter.k8s.aws/instance-family
          operator: NotIn
          values: ["t2", "t3a"]
  limits:
    cpu: "1000"
    memory: "4000Gi"
  disruption:
    # Remove underutilised nodes and reschedule their pods automatically
    consolidationPolicy: WhenUnderutilized
    # Recycle nodes after 30 days to ensure fresh, patched AMIs
    expireAfter: 720h

What each setting does:

• consolidationPolicy: WhenUnderutilized: Karpenter continuously monitors node utilisation and removes underused nodes, moving their pods elsewhere. Your node count decreases automatically as load drops without any manual intervention.

• expireAfter: 720h: Nodes older than 30 days are gracefully replaced, ensuring your infrastructure always runs the latest EKS-optimised AMI with current security patches.

• values: ["spot", "on-demand"]: Karpenter attempts Spot capacity first. If Spot is unavailable for the requested instance type, it falls back to On-Demand with no alerts and no manual action required.

Migrating from Cluster Autoscaler safely:

# Step 1: Install Karpenter alongside Cluster Autoscaler — do not remove CAS yet
helm repo add karpenter https://charts.karpenter.sh
helm install karpenter karpenter/karpenter \
  --namespace karpenter \
  --create-namespace \
  --set settings.clusterName=your-cluster-name

# Step 2: Apply NodePool and NodeClass configuration
kubectl apply -f karpenter/nodepool.yaml
kubectl apply -f karpenter/nodeclass.yaml

# Step 3: Taint existing legacy nodes so new pods schedule on Karpenter nodes
# This migrates workloads gradually — zero downtime
kubectl taint nodes -l eks.amazonaws.com/nodegroup=cpu-optimized \
  group=legacy:NoSchedule

# Step 4: Watch pods reschedule to Karpenter-managed nodes over the next hour
kubectl get pods -o wide --all-namespaces | grep -v legacy

# Step 5: After 30 days of stable operation, remove the old node groups
eksctl delete nodegroup --cluster my-cluster --name cpu-optimized
eksctl delete nodegroup --cluster my-cluster --name memory-optimized

Ready-to-deploy NodePool and NodeClass templates are in karpenter/ in the companion repo.

3.3 Spot Instances for Non-Production Workloads

Staging and development workloads don't need the reliability guarantees of On-Demand instances. Moving them to Spot saves 60–90% on those node costs. Karpenter handles Spot interruptions by rescheduling pods automatically. For stateless workloads, interruptions are invisible to users.

# Good: Spot-only NodePool for staging environments
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: staging-spot
spec:
  template:
    metadata:
      labels:
        billing/environment: staging
    spec:
      taints:
        - key: environment
          value: staging
          effect: NoSchedule  # Only pods that tolerate this taint schedule here
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]   # Spot only for non-production
  disruption:
    consolidationPolicy: WhenUnderutilized

3.4 The ROI of Karpenter and Spot

Part 4: Graviton Migration

AWS Graviton is Amazon's own ARM-based processor family, available across EC2 instance types with names ending in g — m7g, c7g, r7g, and so on.

Graviton instances are priced approximately 20% lower than equivalent Intel or AMD x86 instances. For most server-side workloads — Node.js, Python, Go, Java — they also deliver 20–40% better performance per dollar because the processor architecture is optimised specifically for these workload types.

You don't change your application code to use Graviton. You change the architecture flag in your container image build and the node selector in your Kubernetes deployment.

4.1 Why Graviton Reduces Cost Without Reducing Performance

The first question to answer before migrating is whether your container images support ARM64. Most official images from Docker Hub ship as multi-architecture images. Your own application images need to be built for both architectures explicitly.

Check whether your images support ARM64:

# Check if an image has an ARM64 manifest
docker manifest inspect your-registry/your-app:latest | jq '.manifests[].platform'

Expected output for a multi-arch image:

{"architecture": "amd64", "os": "linux"},
{"architecture": "arm64", "os": "linux", "variant": "v8"}

If arm64 appears, the image is ready. If not, you need to build and push a multi-arch image first.

Build and push a multi-architecture image:

# Build for both x86 and ARM in a single command using Docker Buildx
docker buildx create --use --name multi-arch-builder

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag your-registry/your-app:latest \
  --push \
  .

4.2 Migrating Workloads to Graviton

With Karpenter already installed, Graviton migration is a single label change on your deployment. Karpenter provisions the appropriate ARM64 node automatically.

Here's the correct implementation:

# Good: nodeSelector directs the pod to Graviton nodes
# Karpenter provisions an arm64 node if one isn't already available
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64   # Schedule exclusively on Graviton nodes
      containers:
        - name: api
          image: your-registry/payment-api:latest  # Must be multi-arch

Migrate gradually, starting with stateless services:

# Step 1: Migrate one stateless service and monitor for 48 hours
kubectl patch deployment payment-api \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/arch":"arm64"}}}}}'

# Step 2: Watch for errors in the first 30 minutes
kubectl logs -l app=payment-api --tail=100 -f

# Step 3: Verify the pod is running on a Graviton node
# The NODE column should show a Graviton instance type (m7g, c7g, r7g)
kubectl get pods -l app=payment-api -o wide

# Step 4: After 48 hours of stable operation, migrate the next service

There are some situations where you shouldn't migrate to Graviton: GPU workloads, applications with native x86 binary dependencies, or any workload where you haven't yet built multi-arch images.

4.3 The ROI of Graviton

Part 5: VPC Endpoints for Data Transfer

5.1 The NAT Gateway Tax

Every byte that travels from your EKS pods to an AWS service — S3, DynamoDB, ECR, SQS — goes through a NAT Gateway if you haven't configured VPC endpoints. NAT Gateway charges $0.045 per GB of data processed.

A busy EKS cluster pulling container images from ECR, writing to S3, and polling SQS queues can process hundreds of terabytes per month through NAT Gateway — generating thousands of dollars in charges for traffic that never actually left the AWS network.

Measure your current NAT Gateway cost before adding endpoints:

# Get last month's NAT Gateway data processing charges
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \
  --granularity DAILY \
  --filter '{
    "Dimensions": {
      "Key": "USAGE_TYPE",
      "Values": ["NATGateway-Bytes"]
    }
  }' \
  --metrics UnblendedCost \
  --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.UnblendedCost.Amount}' \
  --output table

5.2 VPC Endpoints — The Fix That Takes 30 Minutes

A VPC endpoint creates a private connection between your VPC and an AWS service, routing traffic through the AWS backbone without touching the NAT Gateway. The data transfer becomes free. Each endpoint costs approximately $0.01/hour — roughly $7.20/month — far less than the NAT Gateway processing charges it replaces.

Here's the complete implementation for the four most common EKS traffic destinations:

# Get your VPC ID and primary route table ID first
VPC_ID=$(aws eks describe-cluster --name your-cluster \
  --query 'cluster.resourcesVpcConfig.vpcId' --output text)

ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=$VPC_ID Name=association.main,Values=true \
  --query 'RouteTables[0].RouteTableId' --output text)

echo "VPC: $VPC_ID | Route Table: $ROUTE_TABLE_ID"

# S3 gateway endpoint — free to create, eliminates all S3 traffic through NAT
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids $ROUTE_TABLE_ID

# DynamoDB gateway endpoint — also free, same mechanism as S3
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.us-east-1.dynamodb \
  --route-table-ids $ROUTE_TABLE_ID

# ECR API interface endpoint — eliminates NAT charges on image pulls
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.ecr.api \
  --subnet-ids $(aws ec2 describe-subnets \
    --filters Name=vpc-id,Values=$VPC_ID Name=tag:Tier,Values=private \
    --query 'Subnets[*].SubnetId' --output text)

# ECR Docker endpoint — required alongside ECR API for complete image pull coverage
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.ecr.dkr \
  --subnet-ids $(aws ec2 describe-subnets \
    --filters Name=vpc-id,Values=$VPC_ID Name=tag:Tier,Values=private \
    --query 'Subnets[*].SubnetId' --output text)

The Terraform module that creates all four endpoints in a single apply is in terraform/vpc-endpoints/ in the companion repo.

Verify that the endpoints are routing traffic correctly:

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=$VPC_ID \
  --query 'VpcEndpoints[*].{Service:ServiceName,State:State,Type:VpcEndpointType}' \
  --output table
# Expected: all endpoints showing State=available

5.3 The ROI of VPC Endpoints

Part 6: EBS Volume Optimisation

6.1 The gp2 to gp3 Migration

EBS gp2 volumes price their IOPS based on storage size — 3 IOPS per GB, with a 100 IOPS minimum. EBS gp3 volumes provide 3,000 IOPS baseline regardless of size, and cost 20% less per GB. The migration runs online with no downtime.

Find and migrate all gp2 volumes:

# Step 1: List all gp2 volumes and their sizes
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].{ID:VolumeId,Size:Size,State:State}' \
  --output table

# Step 2: Migrate each gp2 volume to gp3 — no instance stop required
# The modify operation runs online while the volume stays attached and in use
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].VolumeId' \
  --output text | tr '\t' '\n' | while read vol; do
    echo "Migrating $vol from gp2 to gp3..."
    aws ec2 modify-volume \
      --volume-id $vol \
      --volume-type gp3
done

# Step 3: Verify all volumes are now gp3
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].VolumeId' \
  --output text
# Expected: empty output — zero gp2 volumes remaining

6.2 Finding and Removing Orphaned Volumes and Snapshots

When Kubernetes PersistentVolumeClaims are deleted, the underlying EBS volumes sometimes aren't cleaned up. They keep running — and billing — indefinitely.

# Find unattached EBS volumes — status=available means not attached to any instance
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[*].{ID:VolumeId,Size:Size,Created:CreateTime}' \
  --output table

# Find EBS snapshots older than 90 days
aws ec2 describe-snapshots \
  --owner-ids self \
  --query "Snapshots[?StartTime<='$(date -d '90 days ago' --iso-8601=seconds)'].[SnapshotId,StartTime,VolumeSize]" \
  --output table

Before deleting any snapshot, cross-reference with your RDS automated backup schedule to confirm it's not the only backup for a production database.

6.3 The ROI of EBS Optimisation

Part 7: Load Balancer Consolidation

7.1 The Problem — One Load Balancer Per Service

Many teams create a separate LoadBalancer Service for every microservice. On AWS, each Application Load Balancer costs approximately $16.20/month base charge plus $0.008/LCU-hour for traffic processed. At 20 microservices, that's $324/month before a single request is processed.

Here's the incorrect approach:

# Bad: This creates a dedicated AWS ALB every time it's applied
# 20 microservices = 20 ALBs = $324+/month before any traffic charges
apiVersion: v1
kind: Service
metadata:
  name: payment-api
spec:
  type: LoadBalancer   # Creates a dedicated ALB
  ports:
  - port: 80
    targetPort: 8080

7.2 The Fix — Shared Ingress Controller

An Ingress controller is a Kubernetes component that runs as a pod inside your cluster and programs a single external load balancer to route traffic to multiple services based on hostname and URL path. Instead of one AWS Application Load Balancer per microservice, you get one ALB total — with path-based routing directing each request to the right backend service. The result is the same routing behaviour at a fraction of the cost.

Here's the correct implementation:

# Good: One Ingress resource routes all external traffic
# The AWS Load Balancer Controller creates one ALB for all services listed here
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shared-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
spec:
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /payments
        pathType: Prefix
        backend:
          service:
            name: payment-service
            port:
              number: 8080
      - path: /users
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 8080
  - host: dashboard.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboard-service
            port:
              number: 3000
  tls:
  - hosts:
    - api.company.com
    - dashboard.company.com
    secretName: tls-wildcard-cert

Verify the Ingress is provisioned and the ALB DNS name is assigned:

# Watch until the ADDRESS column shows the ALB DNS name (typically 2–3 minutes)
kubectl get ingress shared-ingress -n production -w

The cost difference:

The shared Ingress manifest is in k8s/ingress/ in the companion repo.

The Complete 7-Step Sequence

Annual saving at this rate: $597,588. Engineering time required: one engineer, one sprint per step.

Best Practices for EKS Cost Optimisation

✅ Do: Right-size pod resource requests before any other optimisation. Every subsequent step depends on accurate requests.

✅ Do: Implement Karpenter with consolidationPolicy: WhenUnderutilized. Let it continuously optimise your node count automatically.

✅ Do: Move staging and development workloads to Spot. 60–90% savings for workloads that tolerate interruption.

✅ Do: Migrate compatible workloads to Graviton. Most web services and APIs run without code changes.

✅ Do: Add VPC endpoints for S3, DynamoDB, and ECR before reviewing data transfer costs.

✅ Do: Migrate gp2 volumes to gp3. It's online, zero downtime, and immediately 20% cheaper.

✅ Do: Use a single shared Ingress controller for all external traffic instead of per-service load balancers.

❌ Don't: Buy Savings Plans before completing steps 1–6. You'll lock in waste for 1–3 years.

❌ Don't: Use static node groups with Cluster Autoscaler when your workload mix changes. Karpenter handles this dynamically.

❌ Don't: Run staging and development environments on On-Demand instances. Spot interruptions are manageable, but the cost difference is not.

Resources

• Karpenter Documentation — Official NodePool configuration reference and installation guide

• AWS Graviton Getting Started Guide — Language-specific compatibility notes and migration guidance from AWS

• Vertical Pod Autoscaler GitHub — VPA installation and configuration documentation

• AWS VPC Endpoints Documentation — Complete list of available VPC endpoints and configuration options

• EBS Volume Modification Documentation — AWS guide for online volume type migration with zero downtime

• AWS Load Balancer Controller — Official documentation for the Ingress controller that provisions AWS ALBs

• AWS Cost Explorer API Reference — Full reference for the cost breakdown commands used throughout this guide

• EKS Best Practices Guide — Cost Optimisation — AWS's official EKS cost optimisation framework

• Companion Repository — All Terraform modules, NodePool templates, VPA manifests, and automation scripts from this guide

Nobody told me. The bill just grew quietly in the background while I was proud of the feature I shipped. A Lambda function that called an external enrichment API on every user event. Clean code. Solid tests. Thirty-two million events that month. At $0.0007 per API call.

My engineering manager forwarded the invoice with two words: "Please explain."

That was the moment I discovered FinOps — not from a conference talk or a certification course, but from the specific shame of having written expensive code and not knowing it until the damage was done.

This roadmap is what I needed that day. A complete, honest guide to transforming from an engineer who builds things that work into an engineer who builds things that work and cost what they should. By the end of this guide, you'll have the skills, the scripts, and the vocabulary to talk about cloud spend the way a CFO and a CTO both want to hear.

Table of Contents

• What You'll Learn

• Prerequisites

• The Four Stages Overview

• Stage 1: The Cost-Aware Engineer — Months 1 to 3

• Stage 2: The Optimisation Specialist — Months 4 to 8

• Stage 3: The Automation Architect — Months 9 to 15

• Stage 4: The Cloud Financial Manager — Months 16 to 24

• Essential Tools and Certifications

• Your 90-Day Action Plan

• Best Practices Summary

• Resources

What You'll Learn

• How to read your AWS bill as an engineer, not as a passive observer

• The exact tagging strategy that makes cost attribution possible

• How to right-size EC2 and RDS instances using CloudWatch data you already have

• The correct sequence for purchasing Savings Plans — and why sequence matters more than the discount percentage

• How to build automated cleanup systems for orphaned resources

• How to present cloud cost findings to engineering leadership with data that drives decisions

• The chargeback and showback models that make cost accountability stick

Let's begin.

Prerequisites

Before following this roadmap, you should have some skills and tools ready to go.

Knowledge:

• You can deploy an application to AWS (EC2, Lambda, or containers)

• You understand basic AWS services: S3, RDS, EC2, VPC, IAM

• You're comfortable reading Python and writing simple bash scripts

• You know what a pull request is and have gone through at least one code review

Access:

• Read-only access to your AWS billing console and Cost Explorer

• AWS CLI v2 configured with at least ReadOnlyAccess policy attached

• Python 3.9 or later for running the audit scripts in this guide

Mindset: You don't need to be a finance expert. But you do need to be willing to look at numbers that might be uncomfortable. Every engineer I've worked with who became excellent at FinOps had one thing in common: they were willing to be the person who asked "but what does this cost?" in a room where nobody else wanted to.

Estimated time: This roadmap covers 24 months of deliberate skill-building. You can absorb the reading in a few evenings. The practice is the 24 months.

The Four Stages Overview

Before going deep, here's the complete picture of where you're going:

Stage 1 — Cost-Aware Engineer (Months 1–3)
├── Read your cloud bill and understand it
├── Tag every resource with meaningful metadata
├── Identify your top 5 cost drivers
└── Block your first expensive PR with cost justification

Stage 2 — Optimisation Specialist (Months 4–8)
├── Right-size every over-provisioned resource
├── Implement storage lifecycle policies
├── Move non-production to Spot instances
└── Purchase your first Savings Plan in the right order

Stage 3 — Automation Architect (Months 9–15)
├── Build automated cleanup for orphaned resources
├── Add cost estimation to your CI/CD pipeline
├── Create cost-aware auto-scaling triggers
└── Deploy a self-service FinOps dashboard

Stage 4 — Cloud Financial Manager (Months 16–24)
├── Lead monthly FinOps reviews with engineering leadership
├── Build chargeback models for departments
├── Negotiate enterprise agreements with AWS
└── Forecast cloud spend within 5% variance

The reason this is a 24-month journey and not a weekend project: each stage builds on the previous one. Engineers who jump straight to Savings Plans without rightsizing first end up paying discounted prices for waste. Engineers who build dashboards before tagging get beautiful charts with no actionable data. The sequence isn't arbitrary.

Stage 1: The Cost-Aware Engineer — Months 1 to 3

1.1 Reading the Bill Like an Engineer, Not an Accountant

The default AWS Cost Explorer view shows you service-level totals. That's accounting. What you need is engineering-level decomposition: which specific resources cost money, what business function they serve, and whether each dollar is justified.

Start by pulling a proper breakdown:

# Pull last month's cost breakdown grouped by service
# Run this before touching any optimisation — this is your baseline
aws ce get-cost-and-usage \
  --time-period Start=\((date -d 'last month' +%Y-%m-01),End=\)(date +%Y-%m-01) \
  --granularity MONTHLY \
  --group-by Type=DIMENSION,Key=SERVICE \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table | sort -k3 -rn

Save the output. Name the file aws-baseline-YYYY-MM.txt. You'll compare every future month against this number. Without a baseline, you can't measure progress — and without measurable progress, you can't make the case to leadership that the work is worth engineering time.

Three questions for every service in your top 5:

Most engineers stop at "what is this service?" and never reach the useful question. Here's the framework I use when I first audit an account:

The first question is whether you know what specific business function this service is performing. Not the product name, the function. "S3" isn't an answer. "Storing unprocessed video uploads that sit for 90 days before anyone watches them" is an answer.

The second question is whether the cost is growing, stable, or shrinking when you look at the past three months. A stable \(12,000/month is a different problem from a \)12,000/month line that was $4,000 six months ago.

The third question is what percentage of your total bill this service represents. Optimising a 1% line item while a 40% line item runs unchecked is a common time-wasting trap.

1.2 The Tagging Strategy That Actually Survives

Here's the honest truth about tagging: most tagging strategies die within six months because they're designed for reporting rather than for engineers. Engineers don't tag things well when they're moving fast. The solution isn't to demand more discipline. Instead, it's to make tagging enforced at the infrastructure layer.

Here's the minimal viable tag set (the six tags that cover 90% of attribution needs):

# These six tags enable cost attribution, accountability, and automated remediation
# Add these to every resource in your AWS account — EC2, RDS, S3, Lambda, everything

Environment: "production" | "staging" | "dev"
Team: "platform" | "backend" | "data" | "ml"
Service: "payment-api" | "fraud-detection" | "user-service"
Owner: "ayo@cloudfrugal.com"     # Person responsible for this resource
CostCenter: "engineering"         # For chargeback reporting
AutoShutdown: "true" | "false"    # Enables automated remediation

Enforce tags at the Terraform level so they can't be skipped:

# variables.tf
# Add this to your Terraform root module
# Any plan that creates a resource without these tags will fail validation

variable "required_tags" {
  description = "Tags required on every resource in this account"
  type = map(string)
  
  validation {
    condition = contains(keys(var.required_tags), "Environment") &&
                contains(keys(var.required_tags), "Team") &&
                contains(keys(var.required_tags), "Owner")
    error_message = "required_tags must include Environment, Team, and Owner."
  }
}

# Apply in every resource
resource "aws_instance" "app_server" {
  ami           = data.aws_ami.amazon_linux.id
  instance_type = "t3.medium"

  tags = merge(var.required_tags, {
    Name    = "app-server-${var.environment}"
    Service = "payment-api"
  })
}

Find everything that's currently untagged:

# List EC2 instances missing the Team tag
# Run this weekly until you hit zero results
aws ec2 describe-instances \
  --query "Reservations[].Instances[?!not_null(Tags[?Key=='Team'].Value | [0])].[InstanceId, InstanceType, State.Name]" \
  --output table

Once you start finding untagged resources, you'll discover a pattern: the oldest resources in the account are the least tagged, and they're often the most expensive. An EC2 instance from 2021 that predates your tagging policy is exactly the kind of thing that generates a $3,000/month line item nobody can explain.

1.3 The Cost-Aware Code Review

The most underused FinOps practice in engineering teams is reviewing code changes for cost implications before they merge. It takes thirty seconds per PR once you build the habit, and it prevents the kind of problem that opened this guide: the expensive feature that nobody priced before shipping.

Add this section to your PR template:

## Cost Impact (required for infrastructure and data changes)

- [ ] This change does not affect cloud resource usage
- [ ] New API calls introduced: estimated cost per call $______, calls/month ______
- [ ] New data storage: estimated monthly delta $______
- [ ] Cross-region data transfer introduced: yes / no
- [ ] New external service dependency with per-call pricing: yes / no

If any box other than the first is checked, add a cost estimate before requesting review.

The discipline is in making cost estimation a first-class review concern, not an afterthought that gets caught by the finance team on the 15th of the month.

Stage 1 Outcomes

By the end of month 3, you should have a baseline cost breakdown on file, 100% tag coverage on active resources, identified your top 5 cost drivers with specific reduction targets, and blocked at least one expensive PR with a cost justification that held up in review.

Stage 2: The Optimisation Specialist — Months 4 to 8

2.1 Right-Sizing: The 80/20 of Cloud Savings

The single most reliable source of cloud waste I find in every account I audit is over-provisioned compute.

The pattern is consistent: an engineer provisions an instance at a size that handles their anticipated peak load, the peak never quite materialises at the expected scale, and nobody revisits the instance size because there's no automatic signal that says "this machine is 75% empty."

Make sure you verify actual utilisation before changing anything:

# rightsize_analyzer.py
# Finds EC2 instances running below 20% average CPU for 14 days
# These are right-sizing candidates — not automatic deletions

import boto3
from datetime import datetime, timedelta

def find_oversized_instances(region='us-east-1'):
    """
    Returns instances with average CPU below 20% for the last 14 days.
    Low CPU alone doesn't mean right-size — check memory too if CW agent installed.
    """
    ec2 = boto3.client('ec2', region_name=region)
    cw  = boto3.client('cloudwatch', region_name=region)

    reservations = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
    )['Reservations']

    candidates = []

    for r in reservations:
        for inst in r['Instances']:
            iid  = inst['InstanceId']
            itype = inst['InstanceType']
            tags = {t['Key']: t['Value'] for t in inst.get('Tags', [])}

            # Pull 14-day average CPU from CloudWatch
            stats = cw.get_metric_statistics(
                Namespace='AWS/EC2',
                MetricName='CPUUtilization',
                Dimensions=[{'Name': 'InstanceId', 'Value': iid}],
                StartTime=datetime.utcnow() - timedelta(days=14),
                EndTime=datetime.utcnow(),
                Period=1209600,   # One 14-day period
                Statistics=['Average']
            )['Datapoints']

            avg_cpu = stats[0]['Average'] if stats else 0.0

            if avg_cpu < 20.0:
                candidates.append({
                    'instance_id':  iid,
                    'instance_type': itype,
                    'avg_cpu_pct':  round(avg_cpu, 1),
                    'environment':  tags.get('Environment', 'unknown'),
                    'owner':        tags.get('Owner', 'unknown'),
                    'team':         tags.get('Team', 'unknown'),
                })

    return sorted(candidates, key=lambda x: x['avg_cpu_pct'])

if __name__ == '__main__':
    results = find_oversized_instances()
    print(f"\nFound {len(results)} right-sizing candidates:\n")
    for r in results:
        print(f"  {r['instance_id']} ({r['instance_type']}) — "
              f"{r['avg_cpu_pct']}% avg CPU — "
              f"owner: {r['owner']}")

A word of caution: CPU utilisation below 20% is a signal, not a verdict. Some workloads are memory-intensive or I/O-bound and will show low CPU while being correctly sized. Before acting on any right-sizing recommendation, check memory utilisation (requires the CloudWatch agent) and network I/O patterns alongside CPU.

2.2 Storage Tiering: Stop Paying Retail for Cold Data

S3 Standard costs \(0.023 per GB per month. S3 Glacier Deep Archive costs \)0.00099 per GB per month. The difference is a factor of 23. If you have data that you last accessed six months ago and you're keeping it in S3 Standard because nobody set up lifecycle policies, you're paying 23x more than necessary.

The complete S3 lifecycle policy for engineering teams:

{
  "Rules": [
    {
      "ID": "application-logs-lifecycle",
      "Status": "Enabled",
      "Filter": {"Prefix": "logs/"},
      "Transitions": [
        {"Days": 30,  "StorageClass": "STANDARD_IA"},
        {"Days": 90,  "StorageClass": "GLACIER_IR"},
        {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
      ],
      "Expiration": {"Days": 2555},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
    },
    {
      "ID": "training-checkpoints-lifecycle",
      "Status": "Enabled",
      "Filter": {"Prefix": "ml-checkpoints/"},
      "Transitions": [
        {"Days": 7,  "StorageClass": "STANDARD_IA"},
        {"Days": 30, "StorageClass": "GLACIER_IR"}
      ],
      "Expiration": {"Days": 90}
    }
  ]
}

# Apply the lifecycle policy to a bucket
aws s3api put-bucket-lifecycle-configuration \
  --bucket your-logs-bucket \
  --lifecycle-configuration file://lifecycle.json

# Verify it applied correctly
aws s3api get-bucket-lifecycle-configuration \
  --bucket your-logs-bucket

2.3 Savings Plans: The Sequence Is Everything

A Savings Plan is a commitment to spend a minimum dollar amount per hour on AWS compute for one or three years, in exchange for discounts of 30–70% off On-Demand rates. The discount is real. The trap is buying before optimising.

The wrong order: You have a \(50,000/month EC2 bill. You buy a Savings Plan covering \)35,000/hour. Then you implement right-sizing and Spot instances — and your actual spend drops to \(22,000/month. You've committed to paying \)35,000/month for 12 months against a need of \(22,000. You're paying \)13,000/month for compute you don't use, at a 30% discount. Congratulations on your discounted waste.

The right order:

Month 1-2: Right-size all instances using VPA and CloudWatch data
Month 3:   Move staging and development to Spot instances
Month 4:   Migrate compatible workloads to Graviton (20% cheaper)
Month 5:   Add VPC endpoints to eliminate NAT Gateway charges
Month 6:   THEN look at your steady-state On-Demand spend
Month 6+:  Purchase Savings Plans covering 70% of that optimised baseline

Calculate what to commit to:

# Get your On-Demand EC2 spend for the last 30 days
# This is your rightsized baseline — the number to commit against
aws ce get-cost-and-usage \
  --time-period Start=\((date -d '30 days ago' +%Y-%m-%d),End=\)(date +%Y-%m-%d) \
  --granularity DAILY \
  --filter '{
    "And": [
      {"Dimensions": {"Key": "SERVICE",       "Values": ["Amazon Elastic Compute Cloud - Compute"]}},
      {"Dimensions": {"Key": "PURCHASE_TYPE", "Values": ["On-Demand"]}}
    ]
  }' \
  --metrics UnblendedCost \
  --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.UnblendedCost.Amount}' \
  --output table

# Get AWS's own recommendation for what to commit
aws savingsplans get-savings-plans-purchase-recommendation \
  --savings-plans-type COMPUTE_SP \
  --term-in-years ONE_YEAR \
  --payment-option NO_UPFRONT \
  --lookback-period-in-days THIRTY_DAYS

Stage 3: The Automation Architect — Months 9 to 15

3.1 The Orphaned Resource Problem — And Why It Never Fixes Itself

Orphaned resources are the cloud equivalent of a gym membership you forgot to cancel. They exist, they charge you, but nobody notices until the annual audit.

The root cause isn't laziness. It's the absence of lifecycle management at the infrastructure layer. When an engineer spins up an EC2 instance for a one-week experiment and then leaves the company, there's no automatic signal that the instance is now orphaned. It sits there, billing $140/month, until someone hunts it down.

The fix is a weekly automated audit that surfaces candidates for deletion and notifies the registered owner, not a process change that depends on engineers remembering to clean up.

# orphan_reporter.py
# Runs every Sunday via EventBridge → Lambda
# Posts a Slack report of orphaned resources for human review
# DOES NOT auto-delete — deletion requires a human decision

import boto3
import json
import urllib.request
from datetime import datetime, timedelta, timezone

SLACK_WEBHOOK = 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
UNATTACHED_VOLUME_AGE_DAYS = 14
SNAPSHOT_AGE_DAYS = 90


def find_orphaned_resources():
    ec2 = boto3.client('ec2')
    report = {'monthly_waste_usd': 0, 'items': []}

    # Unattached EBS volumes
    for vol in ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )['Volumes']:
        age = (datetime.now(timezone.utc) - vol['CreateTime']).days
        if age >= UNATTACHED_VOLUME_AGE_DAYS:
            cost = round(vol['Size'] * 0.08, 2)  # gp3 rate
            tags = {t['Key']: t['Value'] for t in vol.get('Tags', [])}
            report['items'].append({
                'type':  'Unattached EBS Volume',
                'id':    vol['VolumeId'],
                'detail': f"{vol['Size']}GB {vol['VolumeType']} — {age} days old",
                'owner': tags.get('Owner', 'unknown'),
                'monthly_cost_usd': cost,
            })
            report['monthly_waste_usd'] += cost

    # Unassociated Elastic IPs
    for addr in ec2.describe_addresses()['Addresses']:
        if 'AssociationId' not in addr:
            report['items'].append({
                'type':  'Unassociated Elastic IP',
                'id':    addr.get('AllocationId', addr['PublicIp']),
                'detail': addr['PublicIp'],
                'owner': 'unknown',
                'monthly_cost_usd': 3.60,
            })
            report['monthly_waste_usd'] += 3.60

    # Old snapshots
    cutoff = (datetime.now(timezone.utc) - timedelta(days=SNAPSHOT_AGE_DAYS)).isoformat()
    for snap in ec2.describe_snapshots(OwnerIds=['self'])['Snapshots']:
        if snap['StartTime'].isoformat() < cutoff:
            cost = round(snap.get('VolumeSize', 0) * 0.05, 2)
            report['items'].append({
                'type':  f'Snapshot ({SNAPSHOT_AGE_DAYS}+ days old)',
                'id':    snap['SnapshotId'],
                'detail': f"Created {snap['StartTime'].strftime('%Y-%m-%d')}",
                'owner': 'unknown',
                'monthly_cost_usd': cost,
            })
            report['monthly_waste_usd'] += cost

    return report


def post_to_slack(report):
    lines = [
        f":money_with_wings: *Weekly Orphaned Resource Report*",
        f"Found *{len(report['items'])} orphaned resources* "
        f"costing *${report['monthly_waste_usd']:.2f}/month*\n",
    ]
    for item in report['items'][:20]:  # Cap at 20 lines to stay readable
        lines.append(
            f"• `{item['type']}` {item['id']} — {item['detail']} "
            f"— *${item['monthly_cost_usd']:.2f}/mo* — owner: {item['owner']}"
        )
    lines.append("\nReview and delete anything no longer needed.")

    req = urllib.request.Request(
        SLACK_WEBHOOK,
        data=json.dumps({'text': '\n'.join(lines)}).encode(),
        headers={'Content-Type': 'application/json'}
    )
    urllib.request.urlopen(req)


def lambda_handler(event, context):
    report = find_orphaned_resources()
    post_to_slack(report)
    return {
        'items_found': len(report['items']),
        'monthly_waste': report['monthly_waste_usd'],
    }

3.2 Cost Estimation in Your CI/CD Pipeline

The goal is to catch expensive infrastructure changes at the PR stage — before they deploy and before they generate a billing surprise.

# .github/workflows/cost-check.yml
# Runs on any PR that touches infrastructure files
# Uses Infracost to estimate the monthly cost delta

name: Infrastructure Cost Check

on:
  pull_request:
    paths:
      - 'terraform/**'
      - 'infrastructure/**'
      - '*.tf'

jobs:
  cost-estimate:
    name: Estimate monthly cost change
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Setup Infracost
        uses: infracost/actions/setup@v3
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Generate cost estimate
        run: |
          infracost breakdown \
            --path terraform/ \
            --format json \
            --out-file /tmp/infracost.json

      - name: Post cost diff to PR
        uses: infracost/actions/comment@v3
        with:
          path: /tmp/infracost.json
          behavior: update

      - name: Block if monthly increase exceeds threshold
        run: |
          MONTHLY_DELTA=$(cat /tmp/infracost.json | \
            jq '.projects[0].diff.totalMonthlyCost' | tr -d '"')

          echo "Estimated monthly cost change: \$$MONTHLY_DELTA"

          # Fail the PR if this change adds more than $500/month
          python3 -c "
          import sys
          delta = float('$MONTHLY_DELTA')
          if delta > 500:
              print(f'PR blocked: estimated +\\({delta:.2f}/month exceeds \\)500 threshold')
              sys.exit(1)
          else:
              print(f'Cost check passed: estimated +\${delta:.2f}/month')
          "

Stage 4: The Cloud Financial Manager — Months 16 to 24

4.1 Leading FinOps Reviews with Executives

By month 16, you have the data. What changes at Stage 4 is the audience. You're no longer presenting to engineers who understand instance types and NAT Gateway pricing. You're presenting to a CTO who wants to know if the infrastructure investment is proportional to the business value it produces, and a CFO who wants to know when the line will stop going up.

The vocabulary shift is simple but important. You stop saying "we right-sized our EC2 instances" and start saying "we reduced our infrastructure unit cost by 28% while maintaining the same request throughput." You stop saying "we eliminated NAT Gateway charges" and start saying "we closed a $6,400/month gap between what we were paying and what was necessary."

The metric that anchors every executive FinOps conversation is cost per business unit. Not total bill (cost per API call, cost per user, cost per transaction, cost per model inference). That ratio tells the story of whether your infrastructure efficiency is improving as the business scales.

# unit_economics.py
# Calculate cost per transaction — the metric that matters to leadership

import boto3
from datetime import datetime, timedelta

def calculate_cost_per_transaction(service_name, transaction_count, days_back=30):
    """
    Returns cost per transaction for a given service over the last N days.
    transaction_count: total transactions for the same period (from your metrics)
    """
    ce = boto3.client('ce')

    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': (datetime.now() - timedelta(days=days_back)).strftime('%Y-%m-%d'),
            'End':   datetime.now().strftime('%Y-%m-%d'),
        },
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        Filter={
            'Tags': {
                'Key':    'Service',
                'Values': [service_name]
            }
        }
    )

    total_cost = sum(
        float(period['Total']['UnblendedCost']['Amount'])
        for period in response['ResultsByTime']
    )

    cost_per_txn = total_cost / transaction_count if transaction_count > 0 else 0

    return {
        'service':           service_name,
        'period_days':       days_back,
        'total_cost_usd':    round(total_cost, 2),
        'transactions':      transaction_count,
        'cost_per_txn_usd':  round(cost_per_txn, 6),
    }


# Example: payment service processed 4.2M transactions this month
result = calculate_cost_per_transaction('payment-api', 4_200_000)
print(f"Cost per transaction: ${result['cost_per_txn_usd']:.6f}")
print(f"Total infrastructure cost: ${result['total_cost_usd']:,.2f}")

4.2 The Chargeback and Showback Models

Chargeback means actually billing departments for their cloud usage. Showback means showing departments their usage costs without the internal billing transfer. Both create the same outcome: engineers start caring about what they consume because someone they work with is paying attention to it.

# showback_report.py
# Generates monthly cost-by-team report for distribution to engineering leads

import boto3
from datetime import datetime

def generate_team_showback():
    ce = boto3.client('ce')

    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': datetime.now().replace(day=1).strftime('%Y-%m-%d'),
            'End':   datetime.now().strftime('%Y-%m-%d'),
        },
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[
            {'Type': 'TAG',       'Key': 'Team'},
            {'Type': 'DIMENSION', 'Key': 'SERVICE'},
        ]
    )

    by_team = {}
    for group in response['ResultsByTime'][0].get('Groups', []):
        team    = group['Keys'][0].replace('Team$', '') or 'untagged'
        service = group['Keys'][1]
        cost    = float(group['Metrics']['UnblendedCost']['Amount'])

        if team not in by_team:
            by_team[team] = {'total': 0, 'services': {}}
        by_team[team]['total'] += cost
        by_team[team]['services'][service] = round(cost, 2)

    # Print sorted by total cost descending
    print(f"\n{'='*52}")
    print(f"  Month-to-Date Cloud Spend by Team")
    print(f"  Generated: {datetime.now().strftime('%Y-%m-%d')}")
    print(f"{'='*52}\n")

    for team, data in sorted(by_team.items(), key=lambda x: x[1]['total'], reverse=True):
        print(f"  {team:<20} ${data['total']:>10,.2f}/month")
        top_services = sorted(data['services'].items(), key=lambda x: x[1], reverse=True)[:3]
        for svc, cost in top_services:
            print(f"    └─ {svc:<30} ${cost:>8,.2f}")
    print()

generate_team_showback()

Essential Tools and Certifications

The tools that matter at each stage of this roadmap:

The one certification worth your time: FinOps Certified Practitioner from the FinOps Foundation. It takes 20 hours to prepare and $300 to sit. It signals to hiring managers and clients that you understand the discipline formally — which matters when you're the person leading FinOps conversations at the executive level.

Your 90-Day Action Plan

Month 1 — Foundation:

Enable Cost Explorer if it isn't already on. Pull the baseline command from Section 1.1 and save the output. Run the untagged resource query from Section 1.2 and document how many resources are missing tags. Find your top three cost drivers. Present the findings to your engineering manager — not as a problem, but as an opportunity with a dollar figure attached.

Month 2 — Quick Wins:

Run the rightsizing analyser from Section 2.1 on your EC2 fleet. Downsize the three highest-confidence candidates. Apply S3 lifecycle policies to your two largest buckets. Create VPC endpoints for S3, ECR, and DynamoDB. Estimate the savings from each action and document them against your baseline.

Month 3 — Automation and Habits:

Deploy the orphan reporter Lambda on a Sunday schedule. Add the cost check GitHub Action to your infrastructure repository. Start a monthly FinOps review meeting — even if it's just you and one other engineer. Build the habit before you need the audience.

Best Practices Summary

✅ Do: Establish a cost baseline before any optimisation. The number is meaningless without a comparison point.

✅ Do: Right-size before buying Savings Plans. Always. The sequence changes the outcome.

✅ Do: Enforce tagging at the infrastructure layer — Terraform or CloudFormation — not as a process reminder.

✅ Do: Move staging and development to Spot instances. The interruption rate is manageable, while the 70% cost difference is not.

✅ Do: Add VPC endpoints for S3, ECR, and DynamoDB before reviewing data transfer costs. It's a 30-minute fix for a multi-thousand-dollar line item.

✅ Do: Present cost findings as cost-per-business-metric, not as total bill. "We reduced cost per transaction from \(0.0021 to \)0.0013" is a business result. "$38,000/month reduction" is an accounting result.

❌ Don't: Buy Savings Plans on an unoptimised baseline. You'll lock in discounted waste.

❌ Don't: Build FinOps dashboards before tagging is complete. Beautiful charts with no attribution data answer no questions.

❌ Don't: Run orphaned resource cleanup without human review first. Run in report-only mode for two weeks, verify the candidates are genuinely orphaned, then add deletion logic.

Resources

• FinOps Foundation Framework — The practitioner framework that defines the Inform, Optimise, and Operate cycle this roadmap is built on

• AWS Cost Explorer API Reference — Full reference for the cost query commands used throughout this guide

• AWS Compute Optimizer — AWS's own rightsizing recommendation service; complements the manual analysis in Stage 2

• Infracost Documentation — Setup guide for the PR-level cost estimation tool in Stage 3

• FinOps Certified Practitioner Exam — The certification referenced in the tools section

• AWS Savings Plans Documentation — The authoritative reference on commitment types, coverage rules, and purchase strategy

• Companion Repository — All scripts from this guide, including the rightsizing analyser, orphan reporter, and showback report generator

Ayobami Adejumo is a senior platform engineer and FinOps consultant. He has audited AWS infrastructure for 20+ Series A and Series B companies. He is an active FinOps Foundation Supporter

Kubernetes can seem intimidating at first, but Amazon EKS (Elastic Kubernetes Service) makes it much more approachable, especially when you have a step-by-step guide to follow.

In this tutorial, we'll walk through exactly how to get a Spring Boot application with a MySQL database up and running on Amazon EKS. I'll take you from from containerizing your app to connecting it to a managed database, all the way to accessing it live in the cloud. Let’s get started.

Table of Contents

• Prerequisites

• Application Overview

• What is Amazon EKS?

• How to Deploy a Spring Boot App with MySQL on Amazon EKS

  • Step 1: Create the VPC

  • Step 2: Set Up the MySQL Database in a Private Subnet

  • Step 3: Deploy EC2 Instance in a Public Subnet

  • Step 4: Create SSH Tunneling for the Database

  • Step 5: Set Up a Simple SpringBoot Application Development

  • Step 6: Configure SpringBoot App for Database

  • Step 7: Dockerize the Spring Boot Application

  • Step 8: Push the Image to Elastic Container Registry (ECR)

  • Step 9: Implement AWS App Load Balancer

  • Step 10: Create a Cluster in EKS

  • Step 11: Install AWS Load Balancing

  • Step 12: Create and Deploy Kubernetes

  • Step 13: Delete Cluster

• Conclusion

Prerequisites

Before you begin, ensure you have the following:

• Basic knowledge of AWS (AWS Console access).

• Basic knowledge of containerization.

• Working knowledge of Kubernetes.

• Basic knowledge of databases.

• Helm installed

• Kubectl installed

• Eksctl installed

• An IDE

Application Overview

The application runs inside an AWS VPC spread across two availability zones for high availability. When a user makes a request, it flows through an Internet Gateway into an AWS Application Load Balancer sitting in the public subnet, which handles incoming traffic via an Ingress rule.

The Load Balancer routes requests to the App Service, which distributes them across multiple App Pods running inside AWS EKS (Elastic Kubernetes Service) in the private subnets.

The Docker images for these pods are pulled from AWS ECR (Elastic Container Registry). For data persistence, the app pods connect to Amazon RDS MySQL databases through a MySQL External Service, with an RDS instance in each availability zone to ensure redundancy.

A NAT Gateway in the public subnet allows the private resources to make outbound internet calls without being directly exposed to the internet.

What is Amazon EKS?

If you've ever tried to manage containers manually, you already know it can get messy pretty quickly, tracking which containers are running, restarting ones that crash, scaling up when traffic spikes... It's a lot.

That's exactly the problem Kubernetes was built to solve. It automates the deployment, scaling, and management of containerized applications. But setting up and maintaining your own Kubernetes cluster from scratch? That's a whole other challenge.

That's where Amazon EKS comes in. EKS is a fully managed Kubernetes service provided by AWS, which means AWS handles the heavy lifting of setting up, securing, and maintaining the Kubernetes control plane for you. You just focus on deploying your application.

How to Deploy a Spring Boot App with MySQL on Amazon EKS

In this section, we’ll cover the steps to follow in deploying your SpringBoot application with MySQL on Amazon EKS.

Step 1: Create the VPC

To create a VPC, log in to the AWS IAM Console and search for “VPC,” then click create VPC.

Select the "VPC and more option:, and give your VPC a name for your project, for example, spring-demo. Set the IPv4 CIDR block to 10.4.0.0/16. For the NAT gateway configuration, select Zonal, then In 1 AZ.

Select None for VPC endpoints configuration. Next, click Create VPC, then click View VPC. This takes you to the VPC resource map.

Step 2: Set Up the MySQL Database in a Private Subnet

First, you need to create the security group for the MySQL and EC2 instance deployment. To do that, navigate to EC2 > Security Groups. For the inbound rule, select Type: All traffic and Source: Anywhere-IPv4. Then click Create security group.

Next, we’ll create the subnet group for the database. To do that, navigate to Aurora and RDS > Subnet groups and click Create DB subnet group. Next, configure the DB subnet to include:

• Name: private-subnet-db

• Description: private-subnet-db

• VPC: Select VPC

• Add subnets: Choose us-east-1a and us-east-1b as the availability zones, then select the private and public subnets

Click Create**.**

Now, navigate to Databases, click Create database, and select Full configuration. Select MySQL as the engine type.

Select the Free tier when choosing a sample template. Next, give your DB a username and a strong password. Choose db.t3.micro as the instance type.

Select your VPC and associated private subnet. Now, uncheck the "Enable auto minor version upgrade" option in the Additional configuration section and click Create database.

While our database initializes, let's create a key pair for the EC2 instance, which will be launched in a public subnet. To do that, navigate to EC2 > Network & Security > Key Pairs and click Create key pair.

Give your key pair a name, for example, ece-db-key-pair. Leave everything else as-is and click Create key pair. This automatically downloads the key-pair into your local machine.

Step 3: Deploy EC2 Instance in a Public Subnet

Now it’s time to create an EC2 instance. To do this, navigate to EC2 > Instances and click Launch instances. Select the key pair you just created in the Key pair section.

Next, in the Network section, select the VPC created earlier for the project. For Auto-assign public IP, choose Enable. Next, choose the Select existing security group option and select the all-access-sg security group created earlier. Next, click Launch instance.

Step 4: Create SSH Tunneling for the Database

For this step, go into your terminal and navigate to the folder where your key pair is downloaded. Run the ls command, and you should see your key pair there.

Next, you need to change the permission of the key pair file. Use the command below:

chmod 0400 ece-db-key-pair.pem 

Now, run the SSH tunneling command below:

ssh -i <YOUR-KEY-PAIR>.pem -f -N -L <LOCAL-PORT>:<YOUR-RDS-ENDPOINT>:<RDS-PORT> <EC2-USERNAME>@<YOUR-EC2-PUBLIC-DNS> -v

• <YOUR-KEY-PAIR>.pem: the name of your downloaded key pair file

• <LOCAL-PORT>:  the port on your laptop (3306 for MySQL, 5432 for PostgreSQL)

• <YOUR-RDS-ENDPOINT>: found in AWS Console > RDS > Your database > Connectivity & Security > Endpoint

• <RDS-PORT>: same as local port (3306 for MySQL, 5432 for PostgreSQL)

• <EC2-USERNAME>: usually ec2-user for Amazon Linux, ubuntu for Ubuntu

• <YOUR-EC2-PUBLIC-DNS>: found in AWS Console > EC2 > Your instance > Public IPv4 DNS

This command lets your laptop or local machine talk directly to your remote database, as if the database were sitting on your own computer.

After running this command, you can open a database tool (like MySQL Workbench, DBeaver, or TablePlus) on your laptop and connect to:

• Host: localhost

• Port: 3306

For this tutorial, I’ll be using the community version of DBeaver. You can use other similar tools, but if you prefer to use the same tool for the purpose of this guide, you can install the community version from the official DBeaver download page.

After download and installation, open the DBeaver client and click the Connect to a database icon in the top-left corner of the app.

Select MySQL and click Next. On the next window, enter your database username and password, and set Server Host to 127.0.0.1.

Click Test Connection.

You should see a window appear on your screen, indicating that the connection is successful.

Click OK and Finish.

Now, on the left panel, you should see your connection. Expand it to see the database structure.

Now, you have successfully created SSH tunneling for your database.

Troubleshooting

While attempting to test the database connection, I initially ran into a “Plugin 'mysql_native_password' is not loaded” error. If you encounter this error, follow the steps below to fix it.

1. On the Connection Settings window, navigate to the Driver properties tab.

2. Look for allowPublicKeyRetrieval and set it to FALSE.

3. Navigate back to the Main tab and click Test Connection.

Everything should work fine now.

Step 5: Set Up a Simple SpringBoot Application Development

To get started, head over to the Spring Initializr website. Rename Artifact to “springboot-mysql-eks”. Then click ADD DEPENDENCIES… to add dependencies for the REST APIs. Search for the following dependencies:

• Spring Web: Build web apps, including RESTful applications using Spring MVC. Uses Apache Tomcat as the default embedded container.

• Spring Data JPA: Persist data in SQL stores with the Java Persistence API using Spring Data and Hibernate.

• IBM DB2 Driver: A JDBC driver that provides access to IBM DB2.

• Lombok: A Java annotation library that helps to reduce boilerplate code.

Next, click GENERATE at the bottom center of the page. This action downloads a zip file to your local machine. Open this file in an IDE, such as VSCode or IntelliJ IDEA. For this tutorial, I use VSCode. In the build.gradle file, you can see all the added dependencies:

dependencies {
   implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
   implementation 'org.springframework.boot:spring-boot-starter-webmvc'
   compileOnly 'org.projectlombok:lombok'
   runtimeOnly 'com.ibm.db2:jcc'
   annotationProcessor 'org.projectlombok:lombok'
   testImplementation 'org.springframework.boot:spring-boot-starter-data-jpa-test'
   testImplementation 'org.springframework.boot:spring-boot-starter-webmvc-test'
   testCompileOnly 'org.projectlombok:lombok'
   testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
   testAnnotationProcessor 'org.projectlombok:lombok'
}

What we're building

The Spring Boot app is a currency exchange rate and conversion app:

We'll be inserting the exchange data into the database table.

To continue with this tutorial, you can clone the project repo here to save time.

In main > java > com.. > model > ExchangeRate, you’ll see the code below:

package com.example.springbootmysqleks.model;

import jakarta.persistence.*;
import lombok.Getter;
import lombok.Setter;

import java.sql.Date;

@Getter
@Setter
@Entity
@Table(name = "exchange-rate")
public class ExchangeRate {
   @Id
   @GeneratedValue(strategy=GenerationType.AUTO)
   private Integer transactionId;
   private String sourceCurrency;
   private String targetCurrency;
   private double amount;
   private Date lastUpdated;
}

This class is essentially a blueprint for storing currency exchange rate data in our database. It uses the libraries and dependencies added earlier. Lombok handles all the repetitive getter/setter boilerplate so you don't have to write it yourself, while JPA annotations like @Entity and @Table tell Spring, "hey, this class maps to a database table called exchange-rate."

Inside the class, there are five fields that become database columns:

• A self-incrementing transactionId as the primary key.

• sourceCurrency and targetCurrency to track which currencies are being converted,

• The amount holding the actual exchange rate

• lastUpdated date, so you always know how fresh your data is.

To store the data, create a repository file in main > java > com.. > repository > ExchangeRateRepository:

package com.example.springbootmysqleks.repository;

import com.example.springbootmysqleks.model.ExchangeRate;
import org.springframework.data.jpa.repository.JpaRepository;

public interface ExchangeRateRepository extends JpaRepository<ExchangeRate, Integer> {
   ExchangeRate findBySourceCurrencyAndTargetCurrency(String sourceCurrency, String targetCurrency);
}

This file acts as the middleman between your code and the database. By simply extending JpaRepository, you instantly get a whole suite of built-in database operations (like save, delete, findAll, and so on) completely for free, without writing a single SQL query.

The interface is typed to work with the ExchangeRate model we just looked at, using Integer as the primary key type.

The one custom method, findBySourceCurrencyAndTargetCurrency, is where Spring's magic really shines. Just by following a naming convention, Spring automatically figures out the SQL query it needs to run, so you can look up an exchange rate by simply passing in two currency codes like "USD" and "EUR" without writing any query logic yourself.

To use the findBySourceCurrencyAndTargetCurrency method, create a service file in main > java > com.. > service > ExchangeRateService with the code below:

package com.example.springbootmysqleks.service;

import com.example.springbootmysqleks.model.ExchangeRate;
import com.example.springbootmysqleks.repository.ExchangeRateRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class ExchangeRateService {

   @Autowired
   private ExchangeRateRepository exchangeRateRepository;

   public ExchangeRate addExchangeRate(ExchangeRate exchangeRate) {
       return exchangeRateRepository.save(exchangeRate);
   }

   public double getAmount(String sourceCurrency, String targetCurrency) {
       ExchangeRate exchangeRate =  exchangeRateRepository.findBySourceCurrencyAndTargetCurrency(sourceCurrency, targetCurrency);
       return exchangeRate == null ? 0 : exchangeRate.getAmount();
   }
}

Here, we created a @Service class that interacts with the repository.

The class has two methods, the addExchangeRate, which simply takes an ExchangeRate object and saves it to the database, and getAmount, which takes a source and target currency, uses our custom repository method to look up the matching record, and then either returns the exchange rate amount or a safe default of 0 if no record is found.

That little ternary check (exchangeRate == null ? 0 : exchangeRate.getAmount()) ensures the app doesn't crash if you query a currency pair that doesn't exist in the database yet.

In main > java > com.. > controller > ExchangeRateService, we have the following code:

package com.example.springbootmysqleks.controller;

import com.example.springbootmysqleks.model.ExchangeRate;
import com.example.springbootmysqleks.service.ExchangeRateService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@RestController
public class ExchangeRateController {

   @Autowired
   ExchangeRateService exchangeRateService;

   @GetMapping("/getAmount")
   public double getAmount(@RequestParam String sourceCurrency, @RequestParam String targetCurrency) {
       return exchangeRateService.getAmount(sourceCurrency, targetCurrency);
   }

   @PostMapping("/addExchangeRate")
   public ExchangeRate addExchangeRate(@RequestBody ExchangeRate exchangeRate) {
       return exchangeRateService.addExchangeRate(exchangeRate);
   }

   @GetMapping("/")
   public String getHealth() {
       return "up";
   }

}

The @RestController annotation tells Spring this class will be serving up REST API endpoints, and again @Autowired wires in the service layer automatically.

There are three endpoints:

1. a GET request to /getAmount that accepts sourceCurrency and targetCurrency as query parameters and returns the exchange rate amount

2. a POST request to /addExchangeRate that accepts a full ExchangeRate object as a JSON body and saves it to the database

3. and finally a simple health check endpoint at / that just returns "up",  which is a common pattern in cloud deployments to let load balancers and orchestration tools know the app is alive and running.

Step 6: Configure SpringBoot App for Database

Now, it’s time to configure the application for the database. Navigate to src > main > resources > application.properties, and you should see this:

spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.url=jdbc:mysql://\({MYSQL_HOSTNAME}:\){MYSQL_PORT}/${MYSQL_DATABASE}?createDatabaseIfNotExist=true
spring.datasource.username=${MYSQL_USERNAME}
spring.datasource.password=${MYSQL_PASSWORD}

spring.jpa.hibernate.ddl-auto=update

spring.jpa.show-sql: true

These are the configurations that allow your app to connect with the database.

• spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver: The driver class for the MySQL database.

• spring.datasource.url=jdbc:mysql://\({MYSQL_HOSTNAME}:\){MYSQL_PORT}/${MYSQL_DATABASE}?createDatabaseIfNotExist=true: This is the data source URL in which we are using the MySQL hostname (127.0.0.1), port name, and database name.

• spring.datasource.username=${MYSQL_USERNAME}: your database user name.

• spring.datasource.password=${MYSQL_PASSWORD}: your database password.

One thing to note: the process of configuring environment variables with your actual credentials varies depending on the IDE you're using. If you're using IntelliJ IDEA, this process is pretty straightforward. If you're using VS Code, the process is different.

To configure your actual credentials for the env variables, create a .vscode/launch.json file in your project root folder and paste in the following configuration:

{
 "version": "0.2.0",
 "configurations": [
   {
     "type": "java",
     "name": "Spring Boot App",
     "request": "launch",
     "mainClass": "com.example.springbootmysqleks.SpringbootMysqlEksApplication",
     "projectName": "springboot-mysql-eks",
     "env": {
       "MYSQL_HOSTNAME": "localhost",
       "MYSQL_PORT": "3306",
       "MYSQL_DATABASE": "exchangedb",
       "MYSQL_USERNAME": "root",
       "MYSQL_PASSWORD": "CHANGE_ME"
     }
   }
 ]
}

Configure the credentials to use your actual credentials.

Now, when you run the app, you should be able to see the created exchangedb table in DBeaver:

Use an API testing tool like Postman to send a POST request to the database:

Next, run the select * from exchange_rate er script in the exchangedb SQL script editor:

At the bottom of the editor, you should see the created table from the Postman request.

Now, run a GET request to the endpoint below:

http://localhost:8080/getAmount?sourceCurrency=USD&targetCurrency=EUR&transactionId=1

You should get a 200 OK response with the currency exchange value, for example, 0.93.

Step 7: Dockerize the SpringBoot Application

To Dockerize your application, create a file named Dockerfile and paste in the configuration below:

FROM eclipse-temurin:17-jre-jammy
WORKDIR /app
COPY build/libs/springboot-mysql-eks.jar /app
EXPOSE 8080
CMD ["java", "-jar", "springboot-mysql-eks.jar"]

Our Dockerfile starts by pulling the lightweight eclipse-temurin:17-jre-jammy base image to keep things lean, then sets /app as the working directory inside the container. It copies our compiled Spring Boot JAR file from the local build/libs/ folder into that directory, exposes port 8080 for incoming traffic, and finally runs the app with java -jar when the container starts up.

Next, build the app to create the .jar file. To do that, run the command below:

./gradlew clean assemble 

You should get a successful build output as shown below:

Navigate to build > the libs folder. You’ll see the springboot-mysql-eks file created.

If you run into an “operation couldn’t be completed.” error, try running the export commands to fix this issue. If you’re using a Mac, then run the command below:

brew install openjdk@21

Next, run the export commands:

export JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home

export PATH=\(JAVA_HOME/bin:\)PATH

Then run the ./gradlew clean assemble command again.

Step 8: Push the Image to Elastic Container Registry (ECR)

In this next step, we’ll create an Amazon ECR and push our image to the registry.

To get started, head back into your AWS Console and search for “ECR”. On the ECR page, click Create**.** Then, enter a repository name, for example, “springboot-mysql-eks.” Next, click Create.

Next, select the repo and click View push commands at the top of the page. This presents a window with a bunch of commands you can use to push your image to the registry. Open your terminal and run these commands. You'll need to ensure Docker is running on your local machine before running the commands.

After running the commands, you should see that your image has been successfully pushed to the registry.

Step 9: Implement AWS App Load Balancer

Before getting started with this step, make sure you check out the installation steps and link to additional AWS documentation in the project README. This will help you follow along.

Now, to get started, create a new folder in your root directory named cluster . This is where you'll download the AWS IAM policy for the load balancer. To download the policy, go into your terminal and cd into cluster, then run the command below:

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.14.1/docs/install/iam_policy.json

This command is gotten from the AWS documentation. Now, when you go to the folder, you’ll see an iam_policy.json file automatically generated.

Next, apply the IAM policy using the command below:

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

You should get an output like this in your terminal:

This shows that the IAM policy has been successfully created. To confirm this, head over to the IAM section in your console, navigate to Policies**,** and search for “AWSLoad…”. You should see the policy created there.

The next step is creating the Kubernetes service account. But before that, you need to tag your public and private subnets as described in this documentation.

Now, head over to the VPC dashboard, navigate to Subnets, click into a subnet, and navigate to Tags. Then, click Manage tags.

Click Add new tag, then enter the key/pair value in the documentation.

Step 10: Create a Cluster in EKS

To create a Kubernetes cluster on EKS, you need the eksctl CLI. Follow the instructions in the AWS eksctl documentation to install the CLI. Next, you need a config file schema to create the cluster. To use this schema, create a new file called cluster.yaml in the cluster folder.

Next, paste in the following configurations:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: spring-test-cluster
  region: us-east-1
  version: "1.30"

vpc:
  id: "<your-vpc-id>"
  subnets:
    private:
      us-east-1a:
        id: "<your-private-subnet-1a-id>" # spring-demo-subnet-private1-us-east-1a
      us-east-1b:
        id: "<your-private-subnet-1b-id>" # spring-demo-subnet-private2-us-east-1b
    public:
      us-east-1a:
        id: "<your-public-subnet-1a-id>" # spring-demo-subnet-public1-us-east-1a
      us-east-1b:
        id: "<your-public-subnet-1b-id>" # spring-demo-subnet-public2-us-east-1b

nodeGroups:
  - name: ng-1
    labels: { role: backend }
    instanceType: t2.micro
    desiredCapacity: 3
    minSize: 3
    maxSize: 5
    privateNetworking: true
    ssh:
      allow: true
      publicKeyName: <your-ec2-key-name>
    iam:
      withAddonPolicies:
        imageBuilder: true
        awsLoadBalancerController: true
        autoScaler: true
iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      attachPolicyARNs:
        - arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy

Th ClusterConfig file is used by eksctl to create our EKS cluster called spring-test-cluster in the us-east-1 region, running Kubernetes version 1.30. It plugs into our existing VPC, placing the worker nodes across private subnets in two availability zones us-east-1a and us-east-1b) for high availability, while keeping public subnets available for the load balancer.

The node group spins up t2.micro EC2 instances with a desired count of 3 (scaling up to 5 if needed), all with private networking enabled for security. It also sets up the necessary IAM permissions for the AWS Load Balancer Controller, Auto Scaler, and ECR image access so our cluster has everything it needs to manage traffic and pull our Docker images automatically.

Now, after updating your configuration with your credentials, run the command below:

eksctl create cluster -f cluster.yaml

This creates the cluster. You should see an output like this on your terminal:

Now, in your AWS console, navigate to CloudFormation, and you’ll see your cluster creation process in progress.

Now, when you go into the EC2 instance page, you should see the three nodes created.

Step 11: Install AWS Load Balancing

The next step is installing a load balancer for our application. To get started, run the command below:

 kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"

This installs custom resource definitions (CRDs) for our controller. Next, run the command below to add the Helm chart repo.

helm repo add eks https://aws.github.io/eks-charts

Update your local repo to ensure you have the most recent charts:

helm repo update eks

Next, install the Helm chart:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --version 1.14.0

Next, verify that the controller is installed:

kubectl get deployment -n kube-system aws-load-balancer-controller

You should see this on your terminal:

This indicates that your controller is ready.

Step 12: Create and Deploy Kubernetes

To get started, you'll first need to create a Kubernetes manifest file. For that, we’ll use Helm Chart.

helm create ytchart

The command above creates a folder named ytchart with the templates for the components. In this folder, you need to make some configurations for your use case. First, navigate to ytchart > templates and delete the serviceaccount.yaml file, since we already created the service account earlier.

Next, go to values.yaml and make the following changes:

• For repository, navigate to the ECR service page on the AWS Console and copy the image URI.

• Tag is latest.

• Set database name

mysql:
 databaseName: exchangedb

• Change service account creation to false.

• Scroll down a bit more and change the service type to NodePort and port to 8080.

You also need to store the database username and password using secrets. Navigate to the templates folder and go into the file named secrets.yaml. Here, set your database username and password, then comment out the liveness and readiness probe in deployment.yaml.

Next, we’ll create a service to connect to the database. To do that, navigate to the mysql.yaml file, then for externalName. Navigate to the RDS service page on the AWS console and copy the database endpoint.

Now, in the deployment.yaml file, paste in the following configuration:

          env:
            - name: SPRING_DATASOURCE_URL
              value: jdbc:mysql://spring-mysql:3306/{{ .Values.mysql.databaseName }}?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useUnicode=true&useSSL=false&allowPublicKeyRetrieval=true
            - name: SPRING_DATASOURCE_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mysql-username
                  key: username
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-root-password
                  key: password

You have successfully created environment variables to secure your database credentials.

In the ingress.yaml file, paste in the following configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "spring-microservice-ingress"
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/load-balancer-name: spring-alb-test
  labels:
    app: spring-microservice
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ include "ytchart.fullname" . }}
                port:
                  number: 8080

This is your configuration for the ingress service.

Run the command below to see all your configuration values:

helm template ytchart/

Next, run the command below to deploy the chart:

helm install mychart ytchart

You should see an output like this on your terminal:

Now, when you run kubectl get all, you should see this:

Now, navigate to EC2 > Load balancers, copy the DNS name, and enter it into a browser. You should see the “up” text. This indicates that your application is working properly.

Now, when you call the API using the DNS URL as such:

http://spring-alb-test-260424558.us-east-1.elb.amazonaws.com/addExchangeRate

You should get a 200 OK response. Congratulations, you have successfully deployed a SpringBoot app in Kubernetes!

Step 13: Delete Cluster

If you’re familiar with AWS and the cloud, you should already be aware of how costly it can be to leave resources running for extended periods, especially when you’re not using them actively.

Now that we've come to the end of this tutorial, it’s time to delete the resources.

These are the resources to delete:

• RDS database.

• Cluster using the command eksctl delete cluster -f cluster.yaml.

• Navigate to VPC and delete the NAT Gateway

Conclusion

Deploying a Spring Boot application with MySQL on Amazon EKS involves a lot of moving parts, but each step builds logically on the last.

In this tutorial, you've gone from setting up a VPC and provisioning a managed database to containerizing your app, pushing it to ECR, and finally orchestrating everything with Kubernetes and an Application Load Balancer.

What you get is a production-grade setup with high availability, private networking, secure credential management, and auto-scaling built in. This is the kind of infrastructure that would take significant manual effort to replicate without managed services like EKS and RDS.

As a next step, consider adding HTTPS support via AWS Certificate Manager, setting up horizontal pod autoscaling, or integrating a CI/CD pipeline to automate future deployments. And remember to clean up your AWS resources when you're done experimenting. Your wallet will thank you.

Startup environments make this worse. The pressure to ship fast, the small team sizes, and the absence of senior engineers to review your decisions means mistakes happen quietly until they become outages, data loss events, or security incidents that cost the company thousands of dollars and weeks of recovery time.

This article is a direct breakdown of the ten most costly DevOps mistakes engineers make early in their careers at startups. For each mistake, you will get the real-world scenario, the business impact, and the concrete fix you can apply immediately.

Whether you are setting up your first production environment or auditing an existing one, this guide will help you build systems that are reliable, secure, and aligned with what the business actually needs.

Table of Contents

• Who This Article Is For

• Why Startups Are a Different Environment

• Mistake 1: Deploying Without Understanding What You're Deploying

• Mistake 2: Using Production as a Development Environment

• Mistake 3: Hardcoding Secrets and Credentials

• Mistake 4: Overengineering for Problems You Don't Have Yet

• Mistake 5: No Observability Before Launch

• Mistake 6: Treating Security as a Final Step

• Mistake 7: Manual Deployments in Production

• Mistake 8: No Disaster Recovery Plan

• Mistake 9: No Documentation or Runbooks

• Mistake 10: Solving Technical Problems Without Understanding the Business

• The System Thinking Framework Every DevOps Engineer Needs

• Your Production Readiness Checklist

• Conclusion

Who This Article Is For

• Early-career DevOps and cloud engineers who are building or maintaining production infrastructure at a startup.

• Backend developers who have recently taken on DevOps responsibilities.

• Engineers joining a startup who want to understand what operational discipline actually looks like in a fast-moving environment.

You do not need to be an expert in any specific tool to follow this article. The focus is on decision-making patterns and operational discipline, not tool configuration.

Why Startups Are a Different Environment

Before getting into the mistakes, you have to understand why startups produce them in the first place.

In a large company, you typically have dedicated security engineers, an SRE team, a platform team, and multiple reviewers for every infrastructure change. In a startup, you mostly likely have one engineer responsible for all of that simultaneously.

This creates four specific pressure points:

1. Speed pressure. The business needs features shipped now. Operational discipline gets treated as optional because nobody is watching closely yet.

2. Budget constraints. Every infrastructure decision has a direct impact on company runway. Engineers optimize for the cheapest option rather than the most reliable one.

3. Absent guardrails. There is no senior engineer reviewing your Terraform plans. There is no security audit before launch. The absence of immediate consequences can make bad decisions feel like good ones.

4. Constantly changing requirements. The architecture you design today may need to support a completely different product in six months. None of these pressures are excuses for poor decisions. But understanding them helps you see why the following mistakes happen so consistently.

Mistake 1: Deploying Without Understanding What You're Deploying

The Scenario

A junior engineer is asked to deploy the company's Node.js API to AWS. They find a tutorial for Elastic Beanstalk, follow it, and it works. Two weeks later, traffic increases. They try to scale "the same way as in the tutorial." The application goes down. They cannot debug it because they never understood what the deployment was actually doing.

The Business Impact

When production breaks and the person who deployed the system cannot explain how it works, diagnosis takes hours instead of minutes. The longer the incident runs, the higher the cost in customer trust, team morale, and potentially direct revenue loss.

The Fix

Before you deploy anything to production, you should be able to answer these five questions in writing:

1. What compute type is running my code? (EC2, Lambda, Fargate, container?)

2. How does a new version replace the old one? (Rolling? Blue/green? All-at-once?)

3. Where does configuration and secrets come from? (SSM? Secrets Manager? Environment file?)

4. What downstream services depend on this? (Database connections? Other APIs? Cache?)

5. How do I roll back in under five minutes if this breaks?

If you cannot answer all five, do not deploy until you can. The tutorial that got it running is not the documentation for how it operates.

"It is better to spend two hours understanding a system before deploying it than two days debugging it after something breaks."

Personally, when learning a new technology, tool, or implementing something I have not worked with before, I usually focus on three core questions: What, Why, and How.

• The first question is: What is this technology or concept about?
  This helps me build a solid foundation by doing deep research, studying the official documentation, understanding the core principles, and sometimes even learning the history behind the tool or technology. I believe having a well-grounded understanding before implementation is very important.

• The second question is: Why do we need it?
  I try to understand the value the technology brings, why it should be implemented, what problem it solves, and how it benefits the team or organization. This helps me make informed technical decisions instead of just implementing tools without understanding their purpose.

• The third question is: How should it be implemented?
  There are usually multiple approaches to solving a problem or implementing a technology, so I focus on understanding the best and most practical approach based on the use case and expected outcome.

This structured approach has helped me learn new technologies quickly, adapt fast, and implement solutions effectively in real-world environments.

Mistake 2: Using Production as a Development Environment

The Scenario

To save time, an engineer tests a new deployment script directly in the production AWS account. They accidentally run a command that terminates the production database instance. Automated backups exist but were misconfigured. Six hours of customer data is unrecoverable.

This scenario happens more often than you would expect. The reasoning is always the same: "It will only take a minute."

The Business Impact

A single test-in-production incident can result in data loss, hours of downtime, and a customer communication crisis. In a startup, that can permanently damage the company's reputation before it has had the chance to build one.

The Fix

You need at minimum three separate environments and ideally three separate AWS accounts:

Using separate AWS accounts (not just separate VPCs) gives you account-level isolation. A permission error in the dev account cannot accidentally touch production infrastructure at the API level.

Infrastructure as Code (Terraform or CloudFormation) makes this affordable, you write the configuration once and apply it three times with different variable files.

# terraform/environments/prod/main.tf
module "app" {
  source      = "../../modules/app"
  environment = "production"
  instance_type = "t3.medium"
  db_instance_class = "db.t3.medium"
  multi_az          = true
}

# terraform/environments/staging/main.tf
module "app" {
  source      = "../../modules/app"
  environment = "staging"
  instance_type = "t3.small"
  db_instance_class = "db.t3.small"
  multi_az          = false
}

The module is the same. The environment-specific variables are different. Separate environments are not a luxury, they are the minimum operating standard for any team running real software.

Mistake 3: Hardcoding Secrets and Credentials

The Scenario

A new engineer joins a startup and clones the repository. Inside they find a .env file committed to Git containing the production database password, the Stripe secret key, and an AWS access key with admin permissions. The repository has been public for six months.

GitHub's automated secret scanning never triggered because the secrets were inside a .env file rather than raw in the code. The credentials had been valid and actively used for over six months.

The Business Impact

Automated scanners run by attackers find exposed credentials within minutes of them being pushed to a public repository. A single exposed AWS access key with admin permissions can result in:

• Crypto-mining workloads generating thousands of dollars in cloud bills overnight

• Complete exfiltration of customer data from every S3 bucket

• Privilege escalation: the attacker creates new admin users and locks you out of your own account

• AWS account suspension while the investigation runs

According to GitHub's annual security report, millions of secrets are exposed in public repositories every year. The average time to detect a compromised cloud credential is 197 days.

The Fix

Step 1: Never commit secrets to Git. Not temporarily. Not in a branch. Not in a private repository.

Step 2: Add .gitignore before you create the first file. Check in the .gitignore with the first line of code before any .env files exist.

# .gitignore
.env
.env.*
*.pem
*.key
secrets/

Step 3: Use AWS Secrets Manager or SSM Parameter Store for all production secrets. Your application reads secrets at runtime:

# Python example — fetch secret at runtime, never at build time
import boto3
import json
 
def get_secret(secret_name: str, region: str = "us-east-1") -> dict:
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])
 
# Usage
db_config = get_secret("prod/myapp/database")
DATABASE_URL = db_config["connection_string"]

Step 4: Scan your existing repositories immediately. You may already have a problem:

# Install trufflehog to scan for exposed secrets in your repo history
pip install trufflehog
 
# Scan the entire commit history of your repository
trufflehog git file://.
 
# Or scan a remote GitHub repo
trufflehog github --repo https://github.com/your-org/your-repo

Step 5: Add a pre-commit hook to prevent future accidents:

pip install pre-commit

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/awslabs/git-secrets
    rev: master
    hooks:
      - id: git-secrets
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets

pre-commit install
# Now the hook runs before every commit and blocks detected secrets

There is no recovery from a publicly exposed database password. The fix takes ten minutes upfront. The incident takes weeks.

Mistake 4: Overengineering for Problems You Don't Have Yet

The Scenario

A five-person startup with 200 users decides to build a microservices architecture on Kubernetes because "Netflix uses it." They spend three months setting up Kubernetes, Istio service mesh, ArgoCD, Vault, Prometheus, and Grafana. Their product has not shipped a new feature in three months. A competitor with a monolith on a single EC2 instance shipped twelve new features in the same period.

The Business Impact

Every layer of infrastructure you add is a layer that can break, a layer that requires expertise to operate, and a layer that slows down every future change. Kubernetes is the right answer for organizations with the scale and team size to operate it. For a five-person startup, it is an expensive distraction.

Premature complexity does not just cost engineering time. It costs the competitive advantage that speed provides in the early stage.

The Fix

Match your infrastructure to your actual stage:

The question to ask before every infrastructure decision is: "What specific, measurable problem does this solve today that my current setup cannot solve?"

Amazon, Netflix, and Uber did not start with microservices. They started with monoliths and extracted services only when the monolith became the actual bottleneck. You are not Netflix. You are solving the problems in front of you today.

Use managed services wherever possible, RDS instead of self-hosted Postgres, Fargate instead of self-managed Kubernetes, ElastiCache instead of self-hosted Redis. Managed services let your team focus on the product instead of the infrastructure.

Mistake 5: No Observability Before Launch

The Scenario

A startup's checkout flow breaks on a Friday evening. Users are abandoning their carts and the company is losing revenue. The DevOps engineer finds out 45 minutes later because a customer sent a direct message to the CEO on Twitter.

The engineer has no dashboards, no log aggregation, and no alerting. They SSH into the production server and scroll through raw log files. Two hours later, they find the issue: a database connection pool was exhausted by a memory leak introduced in that morning's deployment.

Business Impact

Without observability:

• You find out about production problems from users, not from your systems

• Incidents take 10x longer to resolve because diagnosis is guesswork

• You cannot tell whether a deployment improved or degraded performance

• You have no data for making better architecture decisions

The Fix

Implement the four golden signals before any service goes to production. These come from Google's Site Reliability Engineering book:

1. Latency: How long requests take to complete (p50, p95, p99)

2. Traffic: How many requests per second the system is handling

3. Errors: The rate of failed requests (5xx responses per minute)

4. Saturation: How close the system is to its limits (CPU, memory, connection pool)

Here is a minimal CloudWatch alarm setup using the AWS CLI:

# Alert when error rate exceeds 1% for 5 consecutive minutes

aws cloudwatch put-metric-alarm \
  --alarm-name "high-error-rate-production" \
  --alarm-description "Error rate exceeded 1% for 5 minutes" \
  --metric-name "5XXError" \
  --namespace "AWS/ApplicationELB" \
  --statistic "Average" \
  --period 60 \
  --evaluation-periods 5 \
  --threshold 0.01 \
  --comparison-operator "GreaterThanOrEqualToThreshold" \
  --alarm-actions "arn:aws:sns:us-east-1:123456789:pagerduty-production" \
  --dimensions Name=LoadBalancer,Value=app/my-alb/1234567890abcdef

Every application should also expose a /health endpoint that returns 200 OK when healthy:

# FastAPI example

from fastapi import FastAPI
from sqlalchemy import text
 
app = FastAPI()
 
@app.get("/health")
async def health_check():
    # Check database connectivity
    try:
        db.execute(text("SELECT 1"))
        db_status = "healthy"
    except Exception:
        db_status = "unhealthy"
 
    return {
        "status": "healthy" if db_status == "healthy" else "degraded",
        "database": db_status,
        "version": os.getenv("APP_VERSION", "unknown")
    }

Your load balancer checks this endpoint. Your uptime monitor checks it. You check it after every deployment.

You do not get to say a system is working unless you have data to prove it. "Nobody complained" is not the same as "nothing is broken."

Mistake 6: Treating Security as a Final Step

The Scenario

A startup rushes to launch their MVP. Security reviews are "planned for after launch." Six months later, a potential enterprise customer requires a security audit before signing a contract. The audit reveals:

• S3 buckets publicly accessible by default

• EC2 instances with port 22 open to 0.0.0.0/0

• IAM users with AdministratorAccess for the entire team

• No encryption on the database at rest

• JWT secrets hardcoded in environment variables The audit fails. The enterprise deal worth $120,000 annually is lost. Remediation takes four weeks of engineering time.

The Business Impact

Security debt is the most expensive technical debt you can accumulate. Unlike performance debt that degrades gradually, security vulnerabilities cause sudden, catastrophic events: data breaches, ransomware, account takeovers, and regulatory fines. At a startup, any one of these can end the company.

The Fix

Apply these six security controls before the first line of production code ships:

1. Principle of Least Privilege every IAM role gets only what it needs:

One of the most common security mistakes in AWS is granting roles more permissions than they need either out of convenience (s3:*) or uncertainty about what the service actually requires. This creates unnecessary risk: if a role is compromised, the attacker inherits every permission you granted.

The fix is simple: look at what your service actually does, then write a policy that allows exactly that.

If your app uploads and reads files from a specific S3 bucket, the policy should say exactly that:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-app-uploads/*"
    }
  ]
}

Notice the Resource is scoped to my-app-uploads/* not all S3 buckets. And the Action list covers only GetObject and PutObject not DeleteObject, not s3:*. If the service gets compromised, the attacker can read and write to that one bucket. That is it. The rest of your account is untouched.

2. Block all S3 public access by default:

AWS S3 buckets are private by default when created but that can be overridden at the bucket level, the object level, or through a bucket policy. Misconfigured S3 buckets are one of the most common causes of data breaches, and they are almost always accidental.

The safest approach is to enable the "Block Public Access" setting at the account level, which overrides all other settings and prevents any bucket from being made public even if someone tries:

aws s3api put-public-access-block \
  --bucket my-app-bucket \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Run this for every bucket you create. Better yet, enable it at the AWS account level so it applies automatically to all future buckets by default.

3. Never open SSH to the internet, use AWS Systems Manager Session Manager instead:

Port 22 open to 0.0.0.0/0 is an attack surface that exists on thousands of AWS instances right now. Brute-force bots scan the internet continuously looking for open SSH ports. Even with a strong key, the exposure is unnecessary because AWS provides a better alternative.

AWS Systems Manager Session Manager gives you full shell access to any EC2 instance without opening a single inbound port on the security group. There is no port to scan, no port to attack, and every session is logged automatically to CloudTrail:

# Start a session on an EC2 instance without port 22 open
aws ssm start-session --target i-0123456789abcdef0

To use Session Manager, the EC2 instance needs the SSM Agent installed (included by default on Amazon Linux 2 and Ubuntu 20.04+) and an IAM instance profile with the AmazonSSMManagedInstanceCore policy attached. Once that is set up, you can close port 22 on the security group entirely.

4. Enable MFA for all IAM users and enforce it via policy:

A leaked IAM username and password with no MFA is a fully compromised account. Multi-factor authentication is the single most effective control against credential theft, and it costs nothing to enable.

Enforce it through an IAM policy that denies all actions when MFA is not present, except the actions needed to set up MFA in the first place. This means even if a set of credentials is stolen, the attacker cannot do anything without the second factor.

The AWS documentation provides the Complete Deny Without MFA Policy, attach it to every IAM user or group in your account. This is a one-time setup that permanently raises your account's security baseline.

5. Enable CloudTrail in all regions:

Without CloudTrail, you have no record of who did what in your AWS account. If a credential is compromised, you cannot investigate what the attacker accessed. If an engineer accidentally deletes a resource, you cannot trace it. You are operating blind.

CloudTrail logs every AWS API call who made it, from which IP, at what time, and what the response was. Enable it across all regions so activity in regions you do not actively use is also captured:

aws cloudtrail create-trail \
  --name production-audit-trail \
  --s3-bucket-name my-cloudtrail-logs \
  --is-multi-region-trail \
  --enable-log-file-validation

The --enable-log-file-validation flag generates a digest file for each log that lets you verify the log has not been tampered with, this is important if you ever need to use these logs in a security investigation or compliance audit. Once this is running, every AssumeRole, every DeleteBucket, and every RunInstances call in your account is permanently recorded.

6. Run AWS Security Hub from day one:

Most teams only discover security misconfigurations after a breach or a compliance audit. Security Hub inverts this, it continuously scans your AWS environment against industry-standard frameworks (CIS AWS Foundations Benchmark, AWS Foundational Security Best Practices) and surfaces findings before they become incidents.

Enabling it takes a single command:

aws securityhub enable-security-hub

Within minutes, Security Hub gives your account a compliance score and a prioritized list of findings. A finding might tell you that a security group has port 22 open to the world, that an S3 bucket has logging disabled, or that root account credentials were recently used. Each finding includes the affected resource and a remediation guide.

Treat every Security Hub finding the same way you treat a production bug: assign it a priority, assign an owner, and close it. A finding sitting unaddressed for 30 days is a known vulnerability you chose to leave open.

Mistake 7: Manual Deployments in Production

The Scenario

A startup's deployment process is documented in a Notion page that is four months out of date. It involves SSH-ing into the server, running git pull, running npm install, and restarting the PM2 process. Different engineers do it slightly differently. One engineer, rushing a late-night release, skips npm install. The application starts crashing because a new dependency is missing.

The Business Impact

Manual deployment processes are inherently unreliable. Humans under pressure skip steps, perform steps in the wrong order, and remember procedures differently. Every manual step in a production deployment process is a scheduled incident waiting for the right moment of stress.

The Fix

If a deployment step is performed manually more than twice, it needs to be automated. Here is a minimal but complete GitHub Actions deployment workflow for an ECS Fargate service:

# .github/workflows/deploy.yml
name: Deploy to Production
 
on:
  push:
    branches:
      - main
 
permissions:
  id-token: write   # Required for OIDC authentication with AWS
  contents: read
 
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
 
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
          aws-region: us-east-1
 
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2
 
      - name: Build and push Docker image
        id: build
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t \(ECR_REGISTRY/my-app:\)IMAGE_TAG .
          docker push \(ECR_REGISTRY/my-app:\)IMAGE_TAG
          echo "image=\(ECR_REGISTRY/my-app:\)IMAGE_TAG" >> $GITHUB_OUTPUT
 
      - name: Deploy to Amazon ECS
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1
        with:
          task-definition: task-definition.json
          service: my-app-service
          cluster: production
          wait-for-service-stability: true

Notice wait-for-service-stability: true. Without this, the workflow reports success the moment ECS accepts the new task definition before the containers are actually healthy. With it, the workflow fails if the new containers crash. You want to know immediately, not discover it from user reports thirty minutes later.

Mistake 8: No Disaster Recovery Plan

The Scenario

A startup's production database runs on a single RDS instance with no Multi-AZ configuration. Automated backups are enabled but have never been tested. The EBS volume backing the instance fails. AWS provisions a new instance from the last snapshot, which is 18 hours old. 18 hours of customer data is permanently lost.

The startup had no disaster recovery plan, no tested recovery procedure, and no communication template ready for customers.

The Business Impact

The question is not whether your infrastructure will fail. It will fail. Every database, every server, every availability zone experiences failures. The question is whether you have a tested plan for when it does.

Data loss of any magnitude is serious. For startups that handle financial data, healthcare data, or anything under GDPR, even partial data loss can trigger regulatory consequences.

The Fix

Define your RTO and RPO before you design anything:

• RTO (Recovery Time Objective): How long can the business survive without this system? A payment API might have an RTO of 15 minutes. An internal analytics dashboard might have an RTO of 4 hours.

• RPO (Recovery Point Objective): How much data loss is acceptable? Zero means real-time replication. One hour means hourly snapshots are sufficient. This directly determines your backup frequency and architecture.

Enable RDS Multi-AZ for all production databases:

# Terraform
resource "aws_db_instance" "production" {
  identifier        = "prod-postgres"
  engine            = "postgres"
  engine_version    = "15.4"
  instance_class    = "db.t3.medium"
  allocated_storage = 100
 
  # Multi-AZ: automatic failover to standby in a different AZ
  # No data loss. Automatic failover in ~60-120 seconds.
  multi_az = true
 
  # Encryption at rest — non-negotiable
  storage_encrypted = true
 
  # Automated backups with 7-day retention
  backup_retention_period = 7
  backup_window           = "03:00-04:00"
 
  # Enable deletion protection in production
  deletion_protection = true
 
  tags = {
    Environment = "production"
  }
}

Test your backups on a schedule. Create a monthly calendar event: "Restore production backup to staging and verify data integrity." An untested backup is not a backup, it is a hope.

# Restore a snapshot to a test instance and verify
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier recovery-test \
  --db-snapshot-identifier rds:prod-postgres-2025-01-15 \
  --db-instance-class db.t3.medium \
  --no-multi-az
 
# Connect and verify row counts
psql -h recovery-test.xxxx.rds.amazonaws.com -U admin -d mydb \
  -c "SELECT COUNT(*) FROM users; SELECT COUNT(*) FROM orders;"

For official guidance on RDS backup and restore, refer to the AWS RDS Backup and Restore documentation.

Mistake 9: No Documentation or Runbooks

The Scenario

The startup's most experienced DevOps engineer takes two weeks of vacation. On day three of their holiday, the staging environment goes down. Nobody else knows how it was built, the engineer set it up manually over six months with no documentation, no Terraform, no notes. The team spends four days trying to reconstruct the environment from memory and guesswork. The engineer gets messages on their vacation every day. When they return, they rebuild the environment in four hours.

The Business Impact

Undocumented infrastructure creates single points of failure not in your systems, but in your team. It makes onboarding new engineers take weeks instead of hours. It makes incident response depend on specific people being available. When that person leaves the company, the knowledge walks out with them.

The Fix

Documentation for an engineering team means three specific things:

1. Infrastructure as Code is the highest form of documentation. The Terraform that defines your infrastructure IS the documentation for what exists and how it is configured. If something is not in code, it should not exist in production.

2. A runbook for every operational task. A runbook is a step-by-step procedure written well enough that someone in their first week at the company can follow it during an incident:

# Runbook: Production Database Connection Exhaustion
 
## Symptoms
- Application logs: "too many connections" errors
- 500 error rate spike on database-dependent endpoints
- pg_stat_activity shows max connections reached
 
## Diagnosis
# Check current connection count
psql -h \(DB_HOST -U \)DB_USER -c "SELECT COUNT(*) FROM pg_stat_activity;"
 
# See connections by application
psql -h \(DB_HOST -U \)DB_USER \
  -c "SELECT application_name, COUNT(*) FROM pg_stat_activity GROUP BY 1 ORDER BY 2 DESC;"

## Resolution
1. Identify and restart the service causing the connection leak
2. If immediate relief needed: kill idle connections older than 10 minutes
3. Long-term: review connection pool settings in application config

## Escalation
If unresolved in 30 minutes: page the on-call backend engineer.

1. An architecture README in every repository. Every engineer who clones your repository should be able to understand what it does, how to run it locally, how to deploy it, and what it depends on without asking anyone.

Mistake 10: Solving Technical Problems Without Understanding the Business

The Scenario

A startup is experiencing slow page loads. A DevOps engineer decides to solve it by migrating to Kubernetes with horizontal pod auto-scaling. The migration takes six weeks. Page loads improve slightly. But 80% of the slowness was caused by unoptimized database queries that had nothing to do with the infrastructure layer. The six-week migration solved 20% of the problem.

The Business Impact

Technical solutions to misdiagnosed problems are extraordinarily expensive. Every hour spent building the wrong solution is an hour not spent on the right one. Infrastructure is a tool for delivering business outcomes not an end in itself.

The Fix

Before making any infrastructure decision, answer these four questions:

1. What is the actual, measured bottleneck? Instrument before you act. The bottleneck is almost never where you assumed it was.

2. What does success look like, and how will you measure it? "Pages are faster" is not measurable. "p95 page load time drops below 1.2 seconds" is measurable.

3. What is the full cost of this solution? Time to implement, ongoing operational burden, team learning curve. Is this cost justified by the measured impact?

4. Can a simpler solution solve 80% of the problem in 20% of the time?

Always profile and measure before you rebuild:

# Check slow queries in PostgreSQL before any infrastructure changes
psql -h \(DB_HOST -U \)DB_USER -d $DB_NAME -c "
SELECT
  query,
  calls,
  total_exec_time / calls AS avg_ms,
  rows / calls AS avg_rows
FROM pg_stat_statements
ORDER BY avg_ms DESC
LIMIT 10;
"

Nine times out of ten, slow applications have slow queries, missing indexes, or an N+1 query problem, none of which require a new infrastructure layer to fix.

The System Thinking Framework Every DevOps Engineer Needs

Most of the mistakes above share a common root cause: the engineer was thinking about one component in isolation instead of the full system.

A system thinker asks six questions before making any change in production:

This is not a checklist you run through slowly. It is a thinking habit that becomes automatic with practice. Senior engineers do not spend more time on deployments than junior engineers do, they spend their time on different things, and this is one of them.

Your Production Readiness Checklist

Use this checklist before any production system goes live. Mark each item as done, in progress, or not yet started.

Infrastructure

• Infrastructure is defined as code (Terraform or CloudFormation) and version-controlled in Git

• Separate dev, staging, and production environments exist with separate credentials

• All production changes go through an automated CI/CD pipeline, no manual SSH deployments

• You can rebuild the entire production environment from code in under two hours

Security

• No secrets, credentials, or API keys exist in any Git repository

• All production secrets are in Secrets Manager or SSM Parameter Store

• All IAM roles follow the principle of least privilege

• S3 buckets have public access blocked by default

• Port 22 is not open to 0.0.0.0/0 on any security group

• CloudTrail is enabled in all regions

• All IAM users have MFA enabled

• AWS Security Hub is enabled and findings are reviewed weekly

Observability

• Every service has a /health endpoint that monitoring checks continuously

• Alerts fire within five minutes of a production error rate spike

• Dashboards exist showing latency, error rate, and resource utilization

• Logs are centralized and searchable, not scattered across individual servers

Reliability

• Production database has Multi-AZ enabled

• Backup restoration has been tested in the last 30 days

• Written runbooks exist for the three most likely failure scenarios

• RTO and RPO requirements are documented and the architecture meets them

Documentation

• Every repository has a README explaining what it does and how to deploy it

• A new engineer could understand the production architecture from documentation alone

• No single engineer holds critical knowledge that lives only in their head

Conclusion

None of the mistakes in this article require rare misfortune to experience. They are the predictable result of decisions that feel reasonable under startup pressure but accumulate into real operational risk over time.

The good news is that every single one of them is preventable with the right awareness and the right habits applied early.

You do not need a perfect infrastructure from day one. You need a correct one: version-controlled, automated, observable, secure, and documented. Start with that foundation. Add complexity only when a specific, measured problem requires it. Always connect technical decisions to business outcomes.

The goal of DevOps in a startup is not to build impressive infrastructure. It is to build reliable systems that support product growth safely, efficiently, and sustainably and to make sure that when something does break, you can recover faster than anyone notices.

Want to Go Deeper?

If this article resonated with you, The Startup DevOps Field Guide covers these principles in full depth with complete infrastructure blueprints, security frameworks, CI/CD pipeline templates, and the end-to-end decision-making playbook for engineers building DevOps practices in startup environments from scratch.

It is written specifically for the engineer who wants to do this right from the beginning not the one rebuilding everything after the first major incident.

But it has always carried a cost that rarely gets discussed openly. That cost isn't just money, though a DynamoDB table with on-demand billing adds up across multiple teams and environments.

The real cost is complexity. Every new AWS environment needs both resources provisioned before Terraform can manage anything else. Every engineer who sets up their first Terraform backend has to understand why two completely different AWS services are responsible for what is logically one thing: storing and protecting state. And every incident involving a stuck lock has required someone to manually delete a record from DynamoDB to unblock the team.

In November 2024, AWS announced that S3 now supports native object locking for Terraform state files, meaning DynamoDB is no longer required for state locking. Terraform 1.10 added support for this feature, and it's now generally available.

In this tutorial, you'll learn:

• What S3 native locking is and how it works

• How to set it up from scratch if you're starting a new project

• How to migrate an existing S3 + DynamoDB setup to S3 native locking safely

• How to verify locking is working and handle edge cases

By the end, you'll have a simpler, cleaner Terraform backend with one fewer AWS resource to manage.

Table of Contents

• What Is Terraform State Locking?

• What Is S3 Native State Locking?

• How S3 Native Locking Compares to the S3 + DynamoDB Approach

• Prerequisites

• Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

  • Step 1: Create the S3 Bucket with Versioning and Encryption

  • Step 2: Configure the Terraform Backend with Native Locking

  • Step 3: Initialize and Verify

• Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

  • Step 1: Verify Your Current Setup

  • Step 2: Enable Object Lock on the Existing S3 Bucket

  • Step 3: Update the Terraform Backend Configuration

  • Step 4: Reinitialize Terraform

  • Step 5: Verify the Migration

  • Step 6: Clean Up the DynamoDB Table

• How to Verify That Locking Is Working

• How to Handle a Stuck Lock

• Rollback Plan: If Something Goes Wrong

• Security Best Practices for Your State Bucket

• Conclusion

• References

What is Terraform State Locking?

Before looking at the new approach, it helps to understand what state locking is solving.

Terraform stores everything it knows about your infrastructure in a state file – a JSON document that maps your configuration to real AWS resources. When you run terraform apply, Terraform reads this file, calculates the difference between the current state and your configuration, and makes the necessary changes.

The problem arises when two engineers or two CI/CD pipelines run and try to apply changes at the same time. If both read the state file simultaneously, calculate changes independently, and both try to write back, you get a race condition. The second write overwrites changes from the first, and your state is now out of sync with reality. This is a serious problem that can cause resources to be untracked, doubled, or destroyed unexpectedly.

State locking solves this by creating a lock when any operation starts that could modify state. If a lock already exists, Terraform refuses to proceed and reports who holds the lock and when it was acquired. Only one operation can hold the lock at a time. When the operation completes, the lock is released.

Terraform Run A                 State File / Lock                Terraform Run B
(User 1)                         (S3/DynamoDB)                   (User 2)

   |                                   |                            |
   |------- 1. Acquire Lock ---------->|                            |
   |                                   |                            |
   |<------ 2. Lock Granted -----------|                            |
   |                                   |                            |
   |                                   |------- 3. Acquire Lock --->|
   |            [PROCESSING]           |                            |
   |      (Modifying Infrastructure)   |<------ 4. Lock Denied -----|
   |                                   |        (Wait / Retry)      |
   |                                   |                            |
   |------- 5. Release Lock ---------->|                            |
   |                                   |                            |
   |           [COMPLETED]             |<------ 6. Lock Granted ----|
   |                                   |                            |
   |                                   |       [PROCESSING]         |
   |                                   | (Modifying Infrastructure) |              
   |                                   |                            |

What Is S3 Native State Locking?

Previously, Terraform's S3 backend used a DynamoDB table as the locking mechanism. When a lock was needed, Terraform wrote a record to DynamoDB with a LockID primary key. DynamoDB's conditional writes guaranteed that only one process could create that record, which is what made the locking atomic.

S3 native locking uses S3 Object Lock instead. S3 Object Lock is an S3 feature originally designed to enforce WORM (Write Once, Read Many) compliance for regulatory requirements. AWS extended this capability to support Terraform's state locking workflow.

When S3 native locking is enabled in your Terraform backend:

1. Terraform writes your state to an .tfstate object in S3 (as before)

2. To acquire a lock, Terraform uses S3's conditional write operations – specifically the if-none-match conditional header to create a lock file atomically

3. If the lock file already exists, S3 rejects the write, and Terraform reports that a lock is held

4. When the operation completes, Terraform deletes the lock file to release the lock.

The key difference from DynamoDB: the entire locking mechanism lives inside S3. No second service. No second set of IAM permissions. No second resource to provision.

Note: This feature requires Terraform version 1.10.0 or later and an S3 bucket with Object Lock enabled. Object Lock must be enabled at bucket creation time. You can't enable it on an existing bucket through the console or CLI. But there is a supported workaround for existing buckets, which we'll cover in Part 2.

How S3 Native Locking Compares to the S3 + DynamoDB Approach

The functional behavior from Terraform's perspective is identical. Locking works the same way. The lock information displayed when a lock is held has the same structure. The only difference is what happens under the hood.

Prerequisites

Before you start, make sure you have the following in place:

• Terraform 1.10.0 or later installed. Check your version:

terraform version

If you need to upgrade, follow the official upgrade guide.

• AWS CLI installed and configured with credentials that have permission to create and manage S3 buckets.

aws --version
aws sts get-caller-identity   # confirm you're authenticated

• IAM permissions to perform the following S3 actions:

  • s3:CreateBucket

  • s3:PutBucketVersioning

  • s3:PutBucketEncryption

  • s3:PutObjectLegalHold

  • s3:PutObjectRetention

  • s3:GetObject

  • s3:PutObject

  • s3:DeleteObject

  • s3:ListBucket

• For the migration path: access to your existing Terraform project and the S3 bucket and DynamoDB table currently in use.

Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

Follow this section if you're starting a new Terraform project and want to use S3 native locking from the beginning.

Step 1: Create the S3 Bucket with Versioning and Encryption

Object Lock must be enabled at bucket creation time. You can't add it afterward through the standard console flow. Create the bucket using the AWS CLI with Object Lock enabled:

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

Note: For regions other than us-east-1, add the --create-bucket-configuration flag.

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket

Now enable versioning on the bucket. Versioning is required alongside Object Lock and allows Terraform to recover previous state versions if something goes wrong:

aws s3api put-bucket-versioning \
  --bucket your-project-terraform-state \
  --versioning-configuration Status=Enabled

Enable server-side encryption so your state files are encrypted at rest:

aws s3api put-bucket-encryption \
  --bucket your-project-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

Block all public access to the bucket. A Terraform state file contains resource IDs, IP addresses, and potentially sensitive values. It should never be publicly accessible:

aws s3api put-public-access-block \
  --bucket your-project-terraform-state \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Verify the bucket configuration:

# Confirm Object Lock is enabled
aws s3api get-object-lock-configuration \
  --bucket your-project-terraform-state
 
# Confirm versioning is enabled
aws s3api get-bucket-versioning \
  --bucket your-project-terraform-state
 
# Confirm encryption is configured
aws s3api get-bucket-encryption \
  --bucket your-project-terraform-state

Expected output for the Object Lock check:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Step 2: Configure the Terraform Backend with Native Locking

In your Terraform project, create or update your backend.tf file:

terraform {
  backend "s3" {
    bucket = "your-project-terraform-state"
    key    = "production/terraform.tfstate"
    region = "us-east-1"
 
    # Enable S3 native state locking
    # Requires Terraform 1.10.0+ and a bucket with Object Lock enabled
    use_lockfile = true
 
    # Encryption at rest
    encrypt = true
  }
}

The critical difference from the old configuration is the use_lockfile = true parameter. Notice what is absent: there's no dynamodb_table argument. No DynamoDB table. No second service.

Here's a direct comparison of the old and new configurations:

Old configuration (S3 + DynamoDB):

terraform {
  backend "s3" {
    bucket         = "your-project-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # this goes away
  }
}

New configuration (S3 native locking):

terraform {
  backend "s3" {
    bucket       = "your-project-terraform-state"
    key          = "production/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # this replaces dynamodb_table
  }
}

Step 3: Initialize and Verify

Run terraform init to initialize the backend:

terraform init

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
 
Terraform has been successfully initialized!

Run a plan to confirm everything is working end-to-end:

terraform plan

If locking is working, you'll see a brief pause while Terraform acquires the lock before the plan output appears. You'll also see the lock information if you look at the S3 bucket – a .tflock file will appear temporarily alongside your state file during the operation and disappear when it completes.

Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

Follow this section if you have an existing Terraform setup using an S3 bucket and DynamoDB table for state locking, and you want to migrate to S3 native locking.

Important: Migration requires a maintenance window or at minimum a period where no Terraform operations are running. You're changing the backend configuration, which means all team members and CI/CD pipelines must stop running terraform plan or terraform apply during the migration. The migration itself takes under 10 minutes.

Step 1: Verify Your Current Setup

Before making any changes, document your existing backend configuration and confirm the state file is accessible:

# Confirm your state file is in S3
aws s3 ls s3://your-existing-bucket/path/to/terraform.tfstate
 
# Confirm the DynamoDB table exists
aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table \
  --query 'Table.TableStatus'

Check your current backend.tf and note the exact values:

# Your current backend.tf - note these values before changing anything
terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"       # note this
    key            = "path/to/terraform.tfstate"   # note this
    region         = "us-east-1"                   # note this
    encrypt        = true
    dynamodb_table = "your-dynamodb-lock-table"    # this will be removed
  }
}

Run one final plan to confirm the current state is clean and there are no unexpected changes pending:

terraform plan

If the plan shows no changes, you're in a safe state to proceed.

Step 2: Enable Object Lock on the Existing S3 Bucket

This is the most important step in the migration. Object Lock can't normally be enabled on an existing bucket. It's a setting that must be configured at creation time.

But AWS provides a way to enable Object Lock on an existing bucket through a support request or through a direct API call that's not exposed in the standard console UI. AWS has officially documented this path for the Terraform migration use case.

Run the following AWS CLI command to enable Object Lock on your existing bucket:

aws s3api put-object-lock-configuration \
  --bucket your-existing-bucket \
  --object-lock-configuration '{"ObjectLockEnabled": "Enabled"}'

Note: This command enables Object Lock in governance mode with no default retention, meaning it enables the locking capability without setting a default retention period on all objects. This is exactly what Terraform's native locking needs: the ability to create and delete lock files, not permanent object retention.

Verify Object Lock is now enabled:

aws s3api get-object-lock-configuration \
  --bucket your-existing-bucket

Expected output:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Also verify that versioning is already enabled (it should be if you are running a production Terraform setup):

aws s3api get-bucket-versioning \
  --bucket your-existing-bucket

Expected output:

{
    "Status": "Enabled"
}

If versioning isn't enabled, enable it before proceeding:

aws s3api put-bucket-versioning \
  --bucket your-existing-bucket \
  --versioning-configuration Status=Enabled

Step 3: Update the Terraform Backend Configuration

Update your backend.tf to remove the dynamodb_table argument and add use_lockfile = true:

terraform {
  backend "s3" {
    bucket = "your-existing-bucket"
    key    = "path/to/terraform.tfstate"
    region = "us-east-1"
    encrypt = true
 
    # Add this:
    use_lockfile = true
 
    # Remove this line entirely:
    # dynamodb_table = "your-dynamodb-lock-table"
  }
}

Your updated backend.tf should look like this:

terraform {
  backend "s3" {
    bucket       = "your-existing-bucket"
    key          = "path/to/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Step 4: Reinitialize Terraform

Run terraform init with the -reconfigure flag. This flag tells Terraform that the backend configuration has changed intentionally and to reinitialize without prompting you to copy state (the state is already in the same bucket):

terraform init -reconfigure

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
 
Terraform has been successfully initialized!

If you see an error here: The most common cause is that Object Lock wasn't successfully enabled on the bucket. Re-run the verification from Step 2 before proceeding.

Step 5: Verify the Migration

Run a plan to confirm Terraform is working correctly with the new backend configuration:

terraform plan

The plan should:

• Complete successfully

• Show the same result as the plan you ran in Step 1 (no changes, or the same changes as before)

• NOT mention DynamoDB anywhere in its output

To confirm that locking is actually using S3 instead of DynamoDB, open a second terminal and run a plan while the first one is running. You should see the second terminal output a lock error that mentions S3, not DynamoDB:

╷
│ Error: Error acquiring the state lock
│
│Error message: operation error S3: PutObject, https response       error StatusCode: 409,
│ RequestID: ..., api error Conflict: Object lock already exists for this key.
│
│ Lock Info:
│   ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
│   Path:      your-existing-bucket/path/to/terraform.tfstate.tflock
│   Operation: OperationTypePlan
│   Who:       user@hostname
│   Version:   1.10.0
│   Created:   2026-05-06 14:22:01 UTC
│   Info:
╵

The Path field shows .tfstate.tflock, a file in your S3 bucket, not a DynamoDB record. This confirms that locking is now handled entirely by S3.

Step 6: Clean Up the DynamoDB Table

Once you've confirmed the migration is working correctly and your team has run at least one successful plan and apply cycle using the new backend, you can remove the DynamoDB table.

Wait at least 24-48 hours before deleting the DynamoDB table if you have CI/CD pipelines or multiple team members. This gives time to catch any pipeline that wasn't updated with the new backend configuration.

When you're ready, delete the DynamoDB table:

aws dynamodb delete-table \
  --table-name your-dynamodb-lock-table

Confirm the deletion:

aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table

Expected output:

An error occurred (ResourceNotFoundException) when calling the DescribeTable operation:
Requested resource not found

This error confirms that the table is gone. The migration is complete.

If you provisioned the DynamoDB table using Terraform (which is the recommended pattern), remove the resource from your Terraform configuration and run terraform apply to destroy it via Terraform rather than the CLI directly. This keeps your state clean:

# Remove this entire block from your Terraform configuration:
resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
 
  attribute {
    name = "LockID"
    type = "S"
  }
}

After removing the block, run:

terraform apply

Terraform will detect that the DynamoDB table resource has been removed from configuration and will destroy the table.

How to Verify That Locking Is Working

After completing either the fresh setup or the migration, use this procedure to independently verify that locking is functioning correctly.

Method 1: Observe the lock file during an operation

In one terminal, start a long-running plan against a configuration with many resources:

terraform plan

While it's running, in a second terminal, check for the lock file in S3:

aws s3 ls s3://your-bucket/path/to/ | grep tflock

You should see a file like:

2026-05-06 14:22:01        512 terraform.tfstate.tflock