<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
    <channel>
        
        <title>
            <![CDATA[ AWS - freeCodeCamp.org ]]>
        </title>
        <description>
            <![CDATA[ Browse thousands of programming tutorials written by experts. Learn Web Development, Data Science, DevOps, Security, and get developer career advice. ]]>
        </description>
        <link>https://www.freecodecamp.org/news/</link>
        <image>
            <url>https://cdn.freecodecamp.org/universal/favicons/favicon.png</url>
            <title>
                <![CDATA[ AWS - freeCodeCamp.org ]]>
            </title>
            <link>https://www.freecodecamp.org/news/</link>
        </image>
        <generator>Eleventy</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:31:46 +0000</lastBuildDate>
        <atom:link href="https://www.freecodecamp.org/news/tag/aws/rss.xml" rel="self" type="application/rss+xml" />
        <ttl>60</ttl>
        
            <item>
                <title>
                    <![CDATA[ How to Migrate an Application from AWS to GKE [Full Book] ]]>
                </title>
                <description>
                    <![CDATA[ This guided lab walks you through migrating a small payments app, with a React UI and Express API running on a single AWS EC2 instance, while PostgreSQL stays on AWS. You’ll move the UI and API to Goo ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-migrate-an-application-from-aws-to-gke-full-book/</link>
                <guid isPermaLink="false">6ad245772c56514e6fd78db7</guid>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Cloud Computing ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Cloud Migration ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Kubernetes ]]>
                    </category>
                
                    <category>
                        <![CDATA[ gke ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ book ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Osomudeya Zudonu ]]>
                </dc:creator>
                <pubDate>Fri, 16 Oct 2026 15:40:39 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/8a1803ba-c9c9-496d-b2c4-58392cc655eb.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>This guided lab walks you through migrating a small payments app, with a React UI and Express API running on a single AWS EC2 instance, while PostgreSQL stays on AWS.</p>
<p>You’ll move the UI and API to Google Kubernetes Engine without changing the public URL customers already use, gradually shifting traffic from AWS to GKE a few percent at a time.</p>
<p>By the end, you’ll have run the app locally, stood up AWS, built GKE in eight Terraform stages, tested the new environment, shifted production traffic to GKE, completed the cutover, and torn down what you no longer need.</p>
<p>The image below shows the middle of the migration, with both clouds live, customers still using one URL, and the database remaining on AWS.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/a72eb451-8e75-4125-ba27-070c73054436.png" alt="Image showing the middle of the migration, with both clouds live, customers still using one URL, and the database remaining on AWS." style="display: block;" width="600" height="400" loading="lazy">

<p>Around that request path, three more systems do their jobs:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/8abff4c2-ba02-41d0-907f-5c167d485d31.png" alt="Image demonstrating the request path" style="display: block;" width="600" height="400" loading="lazy">

<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-before-you-start">Before You Start</a></p>
<ul>
<li><p><a href="#heading-what-you-need-installed">What You Need Installed</a></p>
</li>
<li><p><a href="#heading-skills-you-need">Skills You Need</a></p>
</li>
<li><p><a href="#heading-what-this-costs">What This Costs</a></p>
</li>
<li><p><a href="#heading-how-this-book-works">How This Book Works</a></p>
</li>
<li><p><a href="#heading-the-team">The Team</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-1-how-to-run-clearpay-and-put-it-on-aws">Part 1. How to Run ClearPay and Put It on AWS</a></p>
<ul>
<li><p><a href="#heading-chapter-1-run-clearpay-on-your-laptop">Chapter 1. Run ClearPay on Your Laptop</a></p>
</li>
<li><p><a href="#heading-chapter-2-read-the-code-that-matters">Chapter 2. Read the Code That Matters</a></p>
</li>
<li><p><a href="#heading-chapter-3-prove-a-retry-never-charges-twice">Chapter 3. Prove a Retry Never Charges Twice</a></p>
</li>
<li><p><a href="#heading-chapter-4-build-the-aws-server-with-terraform">Chapter 4. Build the AWS Server With Terraform</a></p>
</li>
<li><p><a href="#heading-chapter-5-deploy-clearpay-to-the-aws-server">Chapter 5. Deploy ClearPay to the AWS Server</a></p>
</li>
<li><p><a href="#heading-chapter-6-record-the-aws-baseline">Chapter 6. Record the AWS Baseline</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-2-how-to-plan-the-move">Part 2. How to Plan the Move</a></p>
<ul>
<li><p><a href="#heading-chapter-7-read-the-migration-assignment">Chapter 7. Read the Migration Assignment</a></p>
</li>
<li><p><a href="#heading-chapter-8-map-what-clearpay-depends-on">Chapter 8. Map What ClearPay Depends On</a></p>
</li>
<li><p><a href="#heading-chapter-9-understand-the-gcp-design">Chapter 9. Understand the GCP Design</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-3-how-to-build-google-cloud-with-terraform">Part 3. How to Build Google Cloud With Terraform</a></p>
<ul>
<li><p><a href="#heading-chapter-10-set-up-gcloud-and-pick-your-project">Chapter 10. Set Upgcloudand Pick Your Project</a></p>
</li>
<li><p><a href="#heading-chapter-11-build-gcp-in-eight-terraform-stages">Chapter 11. Build GCP in Eight Terraform Stages</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-4-how-to-get-clearpay-onto-gke-staging">Part 4. How to Get ClearPay Onto GKE Staging</a></p>
<ul>
<li><p><a href="#heading-chapter-12-check-the-api-image">Chapter 12. Check the API Image</a></p>
</li>
<li><p><a href="#heading-chapter-13-prepare-the-cluster-for-clearpay">Chapter 13. Prepare the Cluster for ClearPay</a></p>
</li>
<li><p><a href="#heading-chapter-14-build-the-cicd-pipeline">Chapter 14. Build the CI/CD Pipeline</a></p>
</li>
<li><p><a href="#heading-chapter-15-check-staging-and-run-the-promotion-gate">Chapter 15. Check Staging and Run the Promotion Gate</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-5-how-to-make-gke-production-ready">Part 5. How to Make GKE Production-Ready</a></p>
<ul>
<li><p><a href="#heading-chapter-16-run-the-security-review">Chapter 16. Run the Security Review</a></p>
</li>
<li><p><a href="#heading-chapter-17-watch-clearpay-with-prometheus-and-grafana-optional">Chapter 17. Watch ClearPay With Prometheus and Grafana (Optional)</a></p>
</li>
<li><p><a href="#heading-chapter-18-load-test-and-break-staging-on-purpose">Chapter 18. Load-Test and Break Staging on Purpose</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-6-how-to-move-customers-to-gke">Part 6. How to Move Customers to GKE</a></p>
<ul>
<li><p><a href="#heading-chapter-19-deploy-production-and-the-gke-gateway">Chapter 19. Deploy Production and the GKE Gateway</a></p>
</li>
<li><p><a href="#heading-chapter-20-walk-the-pre-cutover-checklist">Chapter 20. Walk the Pre-Cutover Checklist</a></p>
</li>
<li><p><a href="#heading-chapter-21-send-1-percent-of-traffic-to-gke">Chapter 21. Send 1 Percent of Traffic to GKE</a></p>
</li>
<li><p><a href="#heading-chapter-22-raise-traffic-to-100-percent">Chapter 22. Raise Traffic to 100 Percent</a></p>
</li>
<li><p><a href="#heading-chapter-23-when-the-numbers-go-bad">Chapter 23. When the Numbers Go Bad</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-7-how-to-finish-the-migration">Part 7. How to Finish the Migration</a></p>
<ul>
<li><p><a href="#heading-chapter-24-prove-the-cutover-with-numbers">Chapter 24. Prove the Cutover With Numbers</a></p>
</li>
<li><p><a href="#heading-chapter-25-run-on-gke-and-set-a-date-for-aws">Chapter 25. Run on GKE and Set a Date for AWS</a></p>
</li>
<li><p><a href="#heading-chapter-26-retire-the-aws-server">Chapter 26. Retire the AWS Server</a></p>
</li>
<li><p><a href="#heading-chapter-27-tear-down-gcp-and-the-database">Chapter 27. Tear Down GCP and the Database</a></p>
</li>
<li><p><a href="#heading-chapter-28-write-the-migration-record">Chapter 28. Write the Migration Record</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-what-comes-after-this-project">What Comes After This Project</a></p>
<ul>
<li><a href="#heading-what-question-this-project-leaves-open">What Question This Project Leaves Open</a></li>
</ul>
</li>
<li><p><a href="#heading-what-you-should-have-at-the-end">What You Should Have at the End</a></p>
</li>
<li><p><a href="#heading-how-to-discuss-this-project-in-an-interview">How to Discuss This Project in an Interview</a></p>
</li>
</ul>
<h2 id="heading-before-you-start">Before You Start</h2>
<h3 id="heading-what-you-need-installed">What You Need Installed</h3>
<p>Install each tool when its chapter needs it. You don't need everything on day one.</p>
<table>
<thead>
<tr>
<th>Chapter</th>
<th>Tools</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Docker Desktop (or Docker Engine with Compose on Linux), Git, and <code>python3</code></td>
</tr>
<tr>
<td>4</td>
<td>An AWS account, the AWS CLI (run <code>aws configure</code> once), Terraform 1.5 or newer, <code>rsync</code>, and an SSH key</td>
</tr>
<tr>
<td>6</td>
<td>Node.js 20 or newer on your laptop. The check scripts in <code>scripts/</code> use <code>node</code> to read JSON</td>
</tr>
<tr>
<td>10</td>
<td>A GCP project with billing turned on, the <code>gcloud</code> CLI, and <code>kubectl</code></td>
</tr>
<tr>
<td>14</td>
<td>A GitHub account and your own copy of this repository on GitHub</td>
</tr>
<tr>
<td>19</td>
<td><code>kustomize</code> (install with <code>brew install kustomize</code> on a Mac, or see <a href="https://kubectl.docs.kubernetes.io/installation/kustomize/">kubectl.docs.kubernetes.io</a>)</td>
</tr>
</tbody></table>
<p>On Windows, use Git Bash or WSL for every command in this book. The commands are written for a Bash shell.</p>
<h3 id="heading-skills-you-need">Skills You Need</h3>
<p>Prior Kubernetes, Terraform, GCP, or load-balancer experience isn't required. Don't buy a domain. On AWS, you open <code>https://</code> plus the Elastic IP. The browser warns about a self-signed certificate. Click through it and you get the same ClearPay login as on your laptop. On GCP you add one <code>/etc/hosts</code> line so <code>https://api.clearpay.test</code> points at the load balancer, unless you turn on Cloud DNS.</p>
<h3 id="heading-what-this-costs">What This Costs</h3>
<p>AWS costs about $1.25 per day. GCP costs about $1.60 to $3 per day once it's up. A weekend run of the whole book costs a few dollars.</p>
<p>Neither environment should run for a month. If you stop for more than a couple of days, tear both down:</p>
<pre><code class="language-bash">./scripts/teardown-gcp.sh
cd infra/aws/terraform &amp;&amp; terraform destroy
</code></pre>
<p>Two files hold the detailed numbers: <a href="https://github.com/Osomudeya/migrate-api-aws-gke-devops/blob/main/infra/aws/COST.md">infra/aws/</a><a href="http://COST.md">COST.md</a> and <a href="https://github.com/Osomudeya/migrate-api-aws-gke-devops/blob/main/infra/gcp/COST.md">infra/gcp/</a><a href="http://COST.md">COST.md</a>. The book sends you to those two files and no other documents.</p>
<h3 id="heading-how-this-book-works">How This Book Works</h3>
<p>Clone the repository and run every command from its root folder unless a step says <code>cd</code> somewhere else:</p>
<pre><code class="language-bash">git clone https://github.com/Osomudeya/migrate-api-aws-gke-devops.git

cd migrate-api-aws-gke-devops
</code></pre>
<p>The repository already holds the app, the Terraform, the Kubernetes files, the pipeline, and the scripts.</p>
<p>You won't write new code. Your work is to run each step, read what the system prints, and decide whether to continue.</p>
<p>This book reads like a story where you join a small payments company in week one, and every chapter is a task someone on the team hands you.</p>
<p>Each chapter opens with a time stamp and a scene, then follows the same loop:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/6fd0931a-5d4d-44ff-8531-74ce76b75023.png" alt="Flow diagram of the loop the book follows" style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-most-important-rule">Most Important Rule</h4>
<p>Don't run a command you don't understand. If the book hasn't told you why yet, keep reading, because the "why" is the point.</p>
<p>This book has two kinds of steps.</p>
<p>The first simply asks whether the step worked, such as whether the file copied or the container started. If it did, continue.</p>
<p>The second asks you to think about what happened. Once the command finishes, read the output and understand what it means before moving to the next chapter.</p>
<h4 id="heading-the-chapters">The Chapters</h4>
<p>Read the chapters in order and skip none. Chapter 1 doesn't ask you to install Kubernetes or open GCP. Those come later.</p>
<table>
<thead>
<tr>
<th>Part</th>
<th>Chapters</th>
<th>What happens</th>
</tr>
</thead>
<tbody><tr>
<td>1. Run ClearPay and Put It on AWS</td>
<td>1–6</td>
<td>You run the app on your laptop, deploy it to one AWS server, and write down how fast it is</td>
</tr>
<tr>
<td>2. Plan the Move</td>
<td>7–9</td>
<td>You read the assignment, map what the app depends on, and read the GCP design</td>
</tr>
<tr>
<td>3. Build Google Cloud With Terraform</td>
<td>10–11</td>
<td>You build GCP in eight Terraform stages, one at a time</td>
</tr>
<tr>
<td>4. Get ClearPay Onto GKE Staging</td>
<td>12–15</td>
<td>The pipeline puts ClearPay on GKE staging, and the promotion gate checks it</td>
</tr>
<tr>
<td>5. Make GKE Production-Ready</td>
<td>16–18</td>
<td>You review security, set up a dashboard, and break staging on purpose</td>
</tr>
<tr>
<td>6. Move Customers to GKE</td>
<td>19–23</td>
<td>You deploy production and move traffic from 1 percent to 100 percent</td>
</tr>
<tr>
<td>7. Finish the Migration</td>
<td>24–28</td>
<td>You prove the cutover, retire AWS, tear everything down, and write it up</td>
</tr>
</tbody></table>
<h4 id="heading-if-something-doesnt-work">If Something Doesn't Work</h4>
<p>Stay in that chapter. Use its <em>"If it fails"</em> steps. Don't skip ahead. A problem you carry forward gets harder to find, because the next chapter adds another cloud on top of it.</p>
<p>The job isn't to memorize Kubernetes or Terraform. It's to learn how to work through a real migration: understand what's running, make decisions based on what you find, verify the result, and know what to do when something breaks.</p>
<h4 id="heading-how-long-this-takes">How Long This Takes</h4>
<p>This might take more than one day. If you'll be away for more than a day or two, destroy the AWS and GCP resources so you're not billed for them while it's idle. The commands are in "What this costs" above.</p>
<h3 id="heading-the-team">The Team</h3>
<p>This lab starts as Zee, a junior DevOps engineer on the ClearPay team, week one. You play Zee.</p>
<ul>
<li><p>Maya is the engineering manager. She asks for evidence.</p>
</li>
<li><p>Daniel is a senior engineer. He pairs with you on servers, Kubernetes, and capacity.</p>
</li>
<li><p>Amara maintains the payments API. She cares about one thing: no customer is charged twice.</p>
</li>
<li><p>Chris handles customer support. He hears first when transfers fail.</p>
</li>
</ul>
<p>Your job is to move the payments API from one AWS EC2 server to GKE without a customer-visible outage, while PostgreSQL stays on AWS.</p>
<h2 id="heading-part-1-how-to-run-clearpay-and-put-it-on-aws">Part 1. How to Run ClearPay and Put It on AWS</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/b613b3b6-75c8-4b75-9cca-02750c9bab8a.png" alt="what next?" style="display: block;" width="600" height="400" loading="lazy">

<p>A migration needs a starting point you trust. In this part, you run ClearPay on your laptop, deploy it to AWS the way the company runs it today, and write down how fast it is. Every later chapter compares GKE against those numbers.</p>
<h3 id="heading-chapter-1-run-clearpay-on-your-laptop">Chapter 1. Run ClearPay on Your Laptop</h3>
<p><em>Week one, Monday, 9:05 a.m.</em></p>
<p>Your laptop is new. The Wi-Fi password is on a sticky note. The first message in your inbox is from Maya, the engineering manager, and it's short.</p>
<p>Maya: "Get ClearPay running on your laptop first. Log in, send $20, and then we'll talk about moving the API to GKE."</p>
<p>The code is in the repository: <code>api/</code>, <code>ui/</code>, and <code>db/</code>. Nothing is running yet.</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>Later chapters need a picture of a good login and a good $20 send. When something fails on AWS or GKE, you compare it with this.</p>
<h4 id="heading-what-youre-about-to-run">What You're About to Run</h4>
<p>ClearPay has three parts, and each part has one job.</p>
<ul>
<li><p>The <strong>UI</strong> is what a customer sees and taps: the login page, the balance, and the Send button. UI stands for user interface.</p>
</li>
<li><p>The <strong>API</strong> is the part that does the work. It has no screen. The UI sends it a message, such as "send $20 to Bob". The API checks the message, does the work, and sends back a reply, such as "done". API stands for application programming interface. Think of it as a door with rules. The UI asks in a fixed way, and the API answers in a fixed way.</p>
</li>
<li><p>The <strong>database</strong> is where ClearPay writes everything down: every account, balance, and payment. PostgreSQL is the database program ClearPay uses.</p>
</li>
<li><p>A <strong>server</strong> is a computer that stays on all day and waits for requests. The API runs on one.</p>
</li>
</ul>
<p>A café works the same way. The menu is the UI. The waiter is the API. The kitchen and its records are the database. A customer never walks into the kitchen. The customer tells the waiter what they want.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e734e16e-0bec-4f5e-b542-03b2cecc1980.png" alt="image demonstrating the flow of the application" style="display: block;" width="600" height="400" loading="lazy">

<p>This chapter runs all three on your laptop. Later chapters move the UI and the API to Google Cloud. The database stays on AWS.</p>
<h4 id="heading-steps">Steps</h4>
<p>Start Docker Desktop. Don't install Node or PostgreSQL for this chapter. They run inside the containers.</p>
<p>From the folder that holds <code>docker-compose.yml</code>:</p>
<pre><code class="language-bash">docker compose up --build
</code></pre>
<p>The first build takes a couple of minutes. This terminal keeps printing logs and doesn't give you a prompt back until you press Ctrl+C. Leave it running.</p>
<p>Wait until all three of these lines appear. The names on the left may start with your folder name:</p>
<pre><code class="language-plaintext">Container &lt;folder&gt;-postgres-1 Healthy
api-1       | {"message":"api_started", ...}
ui-1        |   ➜  Local:   http://localhost:5173/
</code></pre>
<p><code>Healthy</code> means Postgres is up. <code>api_started</code> means the API is up. The <code>localhost:5173</code> line means the UI is up.</p>
<p>Open a second terminal in the same folder. Create two demo accounts:</p>
<pre><code class="language-bash">docker compose exec api node db/seed.js
</code></pre>
<p>This runs the seed script directly. The API image has no npm, on purpose, because the security scan in Chapter 14 flags it.</p>
<p>Expected output:</p>
<pre><code class="language-plaintext">seeded: alice@clearpay.dev (password: password123)
seeded: bob@clearpay.dev (password: password123)
</code></pre>
<p>If you see <code>skip (already exists): alice@clearpay.dev</code>, the accounts exist from an earlier run. That's fine.</p>
<p><strong>This is what's now running:</strong></p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3dfe597d-2949-4ce8-beae-285d3a5a13f2.png" alt="Flow diagram of what we have running at this moment: Browser, React UI, Express API, and Postgres" style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-check">Check</h4>
<p>In the second terminal:</p>
<pre><code class="language-bash">curl localhost:4000/health
</code></pre>
<p>Here's the expected output. Your <code>instance</code> value will be a container ID:</p>
<pre><code class="language-json">{"status":"ok","served_by":{"platform":"aws","version":"1.0.0","instance":"14b8d5d767ea","commit":"local","rollback_active":false}}
</code></pre>
<p><code>"status":"ok"</code> is the check. <code>served_by</code> names the platform that answered the request. It says <code>aws</code> on your laptop because <code>docker-compose.yml</code> sets <code>PLATFORM: aws</code>. Later, GKE answers with <code>gke</code>, and that one field tells you which cloud handled each request.</p>
<p>Now open <a href="http://localhost:5173">http://localhost:5173</a> and sign in. Seed already created Alice, so don't register her:</p>
<ul>
<li><p>Email: <code>alice@clearpay.dev</code></p>
</li>
<li><p>Password: <code>password123</code></p>
</li>
</ul>
<p>Send $20 to <code>bob@clearpay.dev</code>. The transfer appears under Recent activity as <code>Sent</code> with a short ID, for example <code>Sent · a1b2c3d4</code>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/fff6c8f3-be7d-4bb3-a525-403f424af320.png" alt="Image showing the UI of Our APP" style="display: block;" width="600" height="400" loading="lazy">

<p>Look at two places on this dashboard. You'll watch them for the rest of the book:</p>
<ol>
<li><p>Each line under Recent activity ends with <code>served by aws/1.0.0</code>. That's the platform and version of the API process that wrote the transfer. It's stored with the transfer in the database.</p>
</li>
<li><p>The black bar at the top says <code>All systems operational</code> when <code>/health/ready</code> succeeds. That means the API process is up and can reach Postgres. It says nothing about logins. A wrong password shows a red error on the form, while the bar keeps saying <code>All systems operational</code>.</p>
</li>
</ol>
<p>After Chapter 5, you open the same dashboard on the AWS address. Same login, same two places.</p>
<h4 id="heading-if-it-fails">If it Fails:</h4>
<ul>
<li><p><code>failed to connect to the docker API</code> or <code>docker.sock</code>: Docker Desktop isn't running. Start it, wait until it is ready, and run <code>docker compose up --build</code> again.</p>
</li>
<li><p><code>email already registered</code> on the Register page: Alice already exists. Go back to Sign in.</p>
</li>
<li><p><code>invalid credentials</code>: the password is the full string <code>password123</code>. If that still fails, run the seed command again. A <code>skip (already exists)</code> line means the account is there.</p>
</li>
<li><p><code>docker compose up</code> exits right away: run <code>docker compose logs api</code> in the second terminal. On a slow first boot the API can start before Postgres accepts connections. Run <code>docker compose up</code> again.</p>
</li>
<li><p>Port <code>4000</code> or <code>5173</code> already in use: another program on your machine holds that port. Stop it, or change the port mapping in <code>docker-compose.yml</code>.</p>
</li>
</ul>
<h4 id="heading-decision">Decision</h4>
<p>If the transfer went through, the application works. Continue.</p>
<p>If it didn't, fix it here, on the laptop. AWS and GKE are harder to debug. On this machine, the answer is one <code>docker compose logs</code> command away.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p><code>docker compose up</code> runs without errors</p>
</li>
<li><p>The seed reports two accounts, created or skipped</p>
</li>
<li><p>You sent $20 in the UI and saw it under Recent activity</p>
</li>
</ul>
<h3 id="heading-chapter-2-read-the-code-that-matters">Chapter 2. Read the Code That Matters</h3>
<p><em>Monday, 11:30 a.m.</em></p>
<p>The $20 went through. You're still looking at the transfer line when a chair rolls up next to yours. Daniel, the senior engineer at the next desk, has his coffee and no hurry.</p>
<p>Daniel looks over at your screen. "You don't need to memorize every line. Know which file does which job, so you can open it fast during the migration."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>Every later chapter touches one of these four files. Five minutes now saves you a search under pressure later.</p>
<h4 id="heading-steps">Steps</h4>
<p>Open these files in this order:</p>
<ol>
<li><p><code>api/src/index.js</code> starts the API and connects every route: health checks, <code>/api/info</code>, login, and transactions.</p>
</li>
<li><p><code>api/src/servedBy.js</code> is 17 lines long. It reads <code>PLATFORM</code>, <code>APP_VERSION</code>, and <code>ROLLBACK_ACTIVE</code> from environment variables. The dashboard uses it to print <code>served by aws/1.0.0</code>.</p>
</li>
<li><p><code>api/src/routes/transactions.js</code> handles sending money. Find the <code>idempotency_key</code> check near the top. It runs before anything else.</p>
</li>
<li><p><code>api/src/db.js</code> connects the API to Postgres. Find <code>DB_POOL_SIZE || '10'</code>.</p>
</li>
</ol>
<p>Daniel stops you on the last file.</p>
<p>"A connection is an open line to the database. One API process can hold up to ten of those lines at the same time. Remember that number."</p>
<h4 id="heading-what-you-now-know">What You Now Know</h4>
<pre><code class="language-plaintext">index.js                 → starts the API and connects the routes
servedBy.js              → says which cloud answered (aws or gke)
routes/transactions.js   → sends money, checks the idempotency key first
db.js                    → talks to Postgres, up to 10 connections per process
</code></pre>
<p><code>servedBy.js</code> fills the dashboard line <code>served by aws/1.0.0</code>. That's which cloud answered, not who logged in.</p>
<p>Today there's one API process, so ten connections is the whole pool. On GKE, several copies of the API run at once. Kubernetes calls each copy a pod. Each pod gets its own pool of ten, and every pod shares the same database. Chapter 22 comes back to that math.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li>You can say, without looking, which file sets the connection limit and which file names the platform</li>
</ul>
<h3 id="heading-chapter-3-prove-a-retry-never-charges-twice">Chapter 3. Prove a Retry Never Charges Twice</h3>
<p><em>Monday, 2:10 p.m.</em></p>
<p>After lunch, you get a message from someone you haven't met yet. Amara maintains the payments API. Maya told her you have ClearPay running on your laptop, so she skips the greeting.</p>
<p>Amara: "Send $20 twice with the same key. Alice's balance should drop by $20, not $40."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>Alice sends $20 to Bob. She taps Send, and the screen freezes. She doesn't know if the payment worked, so she taps Send again.</p>
<p>Without protection, the API sees two requests and takes $20 two times. Alice loses $40 for one payment.</p>
<p>The <code>idempotency_key</code> stops this. "Idempotent" means safe to repeat: doing it twice gives the same result as doing it once.</p>
<p>The key is a label the app puts on each payment, for example <code>round-a</code>.</p>
<ol>
<li><p>The first request arrives with the label <code>round-a</code>. The API takes $20 and saves the label.</p>
</li>
<li><p>The second request arrives with the same label. The API sees <code>round-a</code> again. It knows this is the same payment, so it takes no money. It sends back the first result.</p>
</li>
</ol>
<p>Same label means same payment.</p>
<p>This matters more during the migration. Sometimes the first tap goes to AWS and the retry goes to GKE. Those are two different servers, but they use one database. GKE looks in that database, finds <code>round-a</code>, and doesn't charge again. One shared database keeps the customer safe on both clouds.</p>
<p>Use <code>curl</code> for this chapter, not the Send money button. The UI gives every click a new key, so every click is a new transfer.</p>
<p>For a deeper walk-through of idempotency, see <a href="https://osomudeya.gumroad.com/l/payflow">PayFlow: A Production-Grade Fintech System for Learning Real DevOps</a>.</p>
<h4 id="heading-steps">Steps</h4>
<p>Stay in the second terminal, in the folder with <code>docker-compose.yml</code>. Run one command at a time and read the output before the next one.</p>
<p>Log in as Alice. This stores her login token in a shell variable called <code>TOKEN</code>. A token is proof that Alice signed in. Later commands send it with <code>$TOKEN</code>.</p>
<pre><code class="language-bash">TOKEN=$(curl -s -X POST localhost:4000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"alice@clearpay.dev","password":"password123"}' \
  | python3 -c "import sys, json; print(json.load(sys.stdin)['token'])")
</code></pre>
<p>This prints nothing when it works. Run <code>echo $TOKEN</code> to confirm. You should see a long string.</p>
<p>Then read Alice's account. Write down the <code>balance_cents</code> number.</p>
<pre><code class="language-bash">curl -s localhost:4000/api/accounts/me -H "Authorization: Bearer $TOKEN"
</code></pre>
<p>This is the expected shape. Your numbers and <code>id</code> will differ:</p>
<pre><code class="language-json">{"account":{"id":"...","balance_cents":"94000","currency":"USD"},"served_by":{"platform":"aws","version":"1.0.0","instance":"...","commit":"local","rollback_active":false}}
</code></pre>
<p><code>94000</code> means $940.00. If you haven't sent anything yet, you'll see <code>100000</code> ($1,000.00). If the command says <code>missing_token</code> or <code>invalid_token</code>, run step 1 again.</p>
<p>Now pick a label you haven't used yet:</p>
<pre><code class="language-bash">LABEL=round-a
</code></pre>
<p>Send $20 to Bob. <code>2000</code> is $20.00 in cents.</p>
<pre><code class="language-bash">curl -s -X POST localhost:4000/api/transactions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"to_email\":\"bob@clearpay.dev\",\"amount_cents\":2000,\"idempotency_key\":\"$LABEL\"}"
</code></pre>
<p>Look for <code>"replayed"</code> in the reply:</p>
<ul>
<li><p><code>"replayed":false</code> means this send took $20. Continue.</p>
</li>
<li><p><code>"replayed":true</code> means this label was used before, probably in an earlier run. No money moved. Set <code>LABEL=round-b</code> and send again until you see <code>"replayed":false</code>.</p>
</li>
</ul>
<p>Send the exact same request again. Don't change <code>LABEL</code>.</p>
<pre><code class="language-bash">curl -s -X POST localhost:4000/api/transactions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"to_email\":\"bob@clearpay.dev\",\"amount_cents\":2000,\"idempotency_key\":\"$LABEL\"}"
</code></pre>
<p>Now you should see <code>"replayed":true</code>, and the transaction <code>id</code> matches the first send.</p>
<p>Read the account again:</p>
<pre><code class="language-bash">curl -s localhost:4000/api/accounts/me -H "Authorization: Bearer $TOKEN"
</code></pre>
<h4 id="heading-check">Check</h4>
<p><code>balance_cents</code> is 2000 lower than the number you wrote down, not 4000 lower. If you started at <code>94000</code>, you now see <code>92000</code>.</p>
<h4 id="heading-if-it-fails">If it Fails:</h4>
<ul>
<li><p><code>echo $TOKEN</code> is empty: run the login command again.</p>
</li>
<li><p>The second send still shows <code>"replayed":false</code>: <code>LABEL</code> changed between the two sends. Run <code>echo $LABEL</code>, then send again.</p>
</li>
<li><p>Both sends showed <code>"replayed":true</code>: the balance won't change. Pick a new <code>LABEL</code> and repeat steps 3 and 4.</p>
</li>
</ul>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>One send returned <code>"replayed":false</code> and the next returned <code>"replayed":true</code> for the same label</p>
</li>
<li><p><code>balance_cents</code> fell by 2000, not 4000</p>
</li>
</ul>
<h3 id="heading-chapter-4-build-the-aws-server-with-terraform">Chapter 4. Build the AWS Server With Terraform</h3>
<p><em>Tuesday, 9:15 a.m.</em></p>
<p>Maya stops at your desk on her way to a meeting. Yesterday you told her that ClearPay runs on your laptop and that a retried payment charges only once.</p>
<p>Maya: "Good. Now, localhost isn't a baseline. Get ClearPay onto AWS and prove a customer can complete a transaction there. Then we can talk about leaving AWS. Daniel will work with you on this one."</p>
<p>Daniel rolls his chair over from the next desk.</p>
<p>Daniel: "One rule before you start. Don't skip <code>terraform plan</code>."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>This book starts from the setup many small companies run: one Linux server with no Docker on it.</p>
<p>systemd, the Linux service manager, starts the Node API when the server boots. nginx sits in front of it and handles HTTPS on port 443. PostgreSQL runs on Amazon RDS, the managed database service.</p>
<p>There's no AWS load balancer. A load balancer spreads traffic across several servers, and this setup has one. Adding one would add an hourly bill for no gain. The numbers are in <code>infra/aws/COST.md</code>.</p>
<p>The file <code>api/Dockerfile</code> exists, but only your laptop uses it. The AWS server never runs Docker.</p>
<p>Daniel: "The EC2 server won't have Docker. systemd starts Node on boot, next to nginx. That's the server you're inheriting."</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/30d53ba8-7337-4be6-8b45-3d37955315b2.png" alt="Architecture diagram showing ClearPay running on AWS. Users connect through an Elastic IP to nginx on port 443 inside an EC2 server. Nginx serves the frontend files and forwards API requests to a Node.js process managed by systemd on port 4000. The API connects to PostgreSQL 16 on Amazon RDS, which has no public endpoint." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram above shows how a customer request reaches ClearPay on AWS. The Elastic IP provides a fixed public address for the EC2 server. Nginx handles HTTPS traffic on port 443, serves the frontend files, and forwards API requests to the Node.js process managed by systemd on port 4000. The API reads and writes payment data in PostgreSQL on Amazon RDS, which has no public endpoint at this stage.</p>
<p>Two AWS terms appear in this chapter:</p>
<ul>
<li><p>A VPC is the private network your servers sit in. AWS gives every account a default one. This lab uses it instead of building its own.</p>
</li>
<li><p>A security group is AWS's firewall. It lists which ports may reach a server. This lab allows SSH from your IP only, HTTPS from anywhere, and Postgres only from the API server.</p>
</li>
</ul>
<h4 id="heading-steps">Steps</h4>
<p>First confirm your laptop is ready. Each of these must work:</p>
<pre><code class="language-bash">docker --version          # still needed later
aws sts get-caller-identity
terraform version         # 1.5 or newer
ssh-keygen -t ed25519 -f ~/.ssh/clearpay-deployer -N ""
</code></pre>
<p><code>aws sts get-caller-identity</code> prints your AWS account. If it errors, run <code>aws configure</code> with an access key from an IAM user that can create EC2, RDS, VPCs, and CloudWatch alarms. An admin user on a throwaway account also works.</p>
<p>From the repository root:</p>
<pre><code class="language-bash">cd infra/aws/terraform
cp terraform.tfvars.example terraform.tfvars
</code></pre>
<p>Open <code>terraform.tfvars</code> and change two values:</p>
<ul>
<li><p><code>ssh_allowed_cidr</code>: your public IP address followed by <code>/32</code>. Find your IP with <code>curl ifconfig.me</code>.</p>
</li>
<li><p><code>ssh_public_key_path</code>: the path to your key's <code>.pub</code> file.</p>
</li>
</ul>
<p>Then:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Read the plan before you type <code>yes</code>. This first run should only create things: EC2, RDS, and security groups. If the plan says it will destroy anything, stop and don't apply.</p>
<p>This is what Terraform builds. The server is empty until Chapter 5:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/02de98fe-808d-4a5a-8e96-aa9b4fcb5a18.png" alt="AWS infrastructure diagram showing an Elastic IP pointing to an empty EC2 Ubuntu server inside the default VPC. Its security group allows SSH from a specified IP address and ports 80 and 443 from anywhere. CloudWatch monitors the instance status. An RDS PostgreSQL 16 database accepts connections on port 5432 only from the EC2 server's security group." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram shows the AWS infrastructure Terraform creates: an EC2 Ubuntu server with an Elastic IP, security groups controlling SSH and HTTPS access, an RDS PostgreSQL database accessible only from the API server, and a CloudWatch status check. Both resources sit inside the default VPC. The EC2 server is still empty. Chapter 5 deploys ClearPay to it.</p>
<p><strong>A note on cost:</strong> the Elastic IP costs about $0.12 per day even while the server runs, because AWS bills every public IPv4 address (<a href="https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/">AWS public IPv4 charge</a>).</p>
<h4 id="heading-check">Check</h4>
<pre><code class="language-bash">terraform output instance_public_ip
</code></pre>
<p>This prints a public IPv4 address. That is how you reach the server. You do not need a domain, so Route 53 stays off (<code>manage_dns = false</code>).</p>
<h4 id="heading-if-it-fails">If it Fails:</h4>
<ul>
<li><p><code>Error: InvalidKeyPair.Duplicate</code>: a key pair named <code>clearpay-deployer</code> already exists in this AWS region from an earlier attempt. Delete it in the AWS console, or rename it in <code>infra/aws/terraform/ec2.tf</code>.</p>
</li>
<li><p><code>Error: UnauthorizedOperation</code>: your AWS user lacks permission for one resource type. The error names the operation. Add that one permission instead of reaching for full admin.</p>
</li>
</ul>
<h4 id="heading-decision">Decision</h4>
<p>The infrastructure exists. A server with nothing running on it isn't done yet. Continue to Chapter 5.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p><code>terraform apply</code> finished with no errors</p>
</li>
<li><p><code>terraform output instance_public_ip</code> prints a real IP</p>
</li>
</ul>
<h3 id="heading-chapter-5-deploy-clearpay-to-the-aws-server">Chapter 5. Deploy ClearPay to the AWS Server</h3>
<p><em>Tuesday, 1:40 p.m.</em></p>
<p>Terraform apply finished before lunch. AWS now shows a server and a database, but nothing is serving customers yet.</p>
<p>Terraform built an empty server and a database. Nothing serves customers yet.</p>
<p>You open a terminal to start the deploy. Daniel pulls his chair over and sits down next to you.</p>
<p>Daniel: "Terraform finishing only tells us the server exists. Let's get the app on it and send a real request."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>An empty server isn't a migration source. Chapter 6 asks whether a real customer request works on AWS. That question has no answer until the login page and the API are up.</p>
<h4 id="heading-steps">Steps</h4>
<p>Start from the repository root. If you're still in <code>infra/aws/terraform</code>, run <code>cd ../../..</code> first.</p>
<p>Start by installing Node, nginx, and certbot on the server:</p>
<pre><code class="language-bash">cd infra/aws/deploy
INSTANCE_IP=$(terraform -chdir=../terraform output -raw instance_public_ip)
echo "$INSTANCE_IP"
# Write that IP down. A new terminal will not have INSTANCE_IP.

ssh -o StrictHostKeyChecking=accept-new \
  -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP 'bash -s' &lt; bootstrap.sh
</code></pre>
<p>You should see <code>== ClearPay AWS bootstrap ==</code>, then a few minutes of <code>apt-get</code> output. <a href="http://bootstrap.sh">bootstrap.sh</a> is short. Read it.</p>
<p>The step worked when the output ends with these lines:</p>
<pre><code class="language-plaintext">== Bootstrap complete ==
Node: v20.20.2
Self-signed TLS is in /etc/letsencrypt/live/clearpay, use curl -k.
</code></pre>
<p>Your Node version may differ. Lines that start with <code>debconf: unable to initialize frontend</code> are harmless. They appear because the SSH session has no interactive screen. The rows of <code>+</code> and <code>*</code> come from OpenSSL creating the self-signed certificate.</p>
<p>If you see no output at all, SSH didn't connect. Your home IP may have changed since Chapter 4, so the security group still allows the old one. Check your IP with <code>curl -s ifconfig.me; echo</code>, put it plus <code>/32</code> in <code>terraform.tfvars</code> as <code>ssh_allowed_cidr</code>, run <code>terraform apply</code> in <code>infra/aws/terraform</code>, and retry.</p>
<p>Next, write the <code>.env</code> file the API needs and deploy the app. The database URL comes from Terraform. The JWT secret signs login tokens and is generated fresh:</p>
<pre><code class="language-bash">DATABASE_URL=$(terraform -chdir=../terraform output -raw database_url)

cat &gt; /tmp/clearpay.env &lt;&lt;EOF
DATABASE_URL=$DATABASE_URL
JWT_SECRET=$(openssl rand -hex 32)
PLATFORM=aws
APP_VERSION=1.0.0
GIT_COMMIT=$(git rev-parse --short HEAD 2&gt;/dev/null || echo local)
ROLLBACK_ACTIVE=false
PORT=4000
EOF

./deploy.sh "$INSTANCE_IP" ~/.ssh/clearpay-deployer /tmp/clearpay.env
rm /tmp/clearpay.env
</code></pre>
<p><code>deploy.sh</code> builds the React UI and copies it to <code>/opt/clearpay-ui</code>. It copies the API to <code>/opt/clearpay-api</code>, installs its packages, writes <code>.env</code>, installs the systemd service and the nginx config from <code>infra/aws/deploy/</code>, and restarts both.</p>
<p>Now load the database schema and seed Alice and Bob. These commands run on the server, not your laptop. RDS has no public endpoint yet (<code>publicly_accessible = false</code> in <code>infra/aws/terraform/rds.tf</code>), so your laptop can't reach it.</p>
<p>Your laptop can't reach the database directly. It uses the EC2 server as a middleman:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/fc6cf623-39f3-43f9-8c4b-82cc275a361f.png" alt="Two-part diagram showing database access rules. The top panel shows SSH access from a laptop to an EC2 server, which is allowed to connect to an RDS database within the same private AWS network. The bottom panel shows a direct connection from the laptop to RDS blocked by a red X." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram above shows why your laptop can't connect directly to the private RDS database. SSH access to the EC2 server is allowed, and the server is allowed to connect to RDS within the AWS network. Direct access from your laptop to the database is blocked. You use the EC2 server as the entry point for database access.</p>
<p>Each command below starts with <code>ssh ... ubuntu@$INSTANCE_IP "..."</code>. That means: log in to the EC2 server and run the part in quotes there. So <code>psql</code> and the seed run on EC2, and EC2 talks to RDS for you. You see the result later in the browser: you log in as Alice, and EC2 reads her account from RDS.</p>
<pre><code class="language-bash">scp -i ~/.ssh/clearpay-deployer ../../../db/schema.sql \
  ubuntu@$INSTANCE_IP:/tmp/schema.sql

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo apt-get install -y postgresql-client -qq &amp;&amp; psql '$DATABASE_URL' -f /tmp/schema.sql"

ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "cd /opt/clearpay-api &amp;&amp; sudo -u clearpay npm run seed"
</code></pre>
<p>The seed is the same script from Chapter 1. This time it writes to RDS.</p>
<h4 id="heading-check">Check</h4>
<pre><code class="language-bash">ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP 'sudo systemctl is-active clearpay-api'
</code></pre>
<p>Expected output: <code>active</code>.</p>
<p>Then from your laptop:</p>
<pre><code class="language-bash">curl -k https://$INSTANCE_IP/health
</code></pre>
<p>Look for <code>"status":"ok"</code> and <code>"platform":"aws"</code>. The <code>-k</code> flag tells curl to accept the self-signed certificate that <code>bootstrap.sh</code> created.</p>
<p>Open <code>https://&lt;your instance IP&gt;</code> in a browser. Click through the "Not Secure" warning. That warning comes from the self-signed certificate. You should see the same ClearPay login as on your laptop. Sign in as Alice with <code>password123</code>. The dashboard shows her balance and <code>served by aws/1.0.0</code>.</p>
<p>Daniel: "The health check only tells us the app is running. Before we write down a baseline, I want to see a transfer go through on this IP."</p>
<p>Send $20 to Bob in the browser.</p>
<h4 id="heading-if-it-fails">If it Fails:</h4>
<ul>
<li><p><code>systemctl is-active</code> returns <code>failed</code>: SSH in and run <code>sudo journalctl -u clearpay-api -n 50</code>. The usual cause is a <code>DATABASE_URL</code> that doesn't match RDS. Compare it with <code>terraform output database_url</code>, character by character.</p>
</li>
<li><p><code>seed_failed</code> with <code>SELF_SIGNED_CERT_IN_CHAIN</code>: <code>psql</code> already worked, so the database is fine. Newer versions of the Node <code>pg</code> library treat <code>sslmode=require</code> as full certificate checking. Copy the current <code>api/src/pgConfig.js</code>, <code>api/src/db.js</code>, and <code>api/db/seed.js</code> onto the server, restart <code>clearpay-api</code>, and run only the seed command again. Don't run <code>schema.sql</code> again.</p>
</li>
<li><p>nginx returns <code>502 Bad Gateway</code>: nginx is running but the API behind it is not. Check the systemd status above first.</p>
</li>
<li><p>The browser shows <code>Cannot GET /</code>: nginx still has an old API-only config. Run <code>./deploy.sh "$INSTANCE_IP" ~/.ssh/clearpay-deployer</code> with no third argument. That keeps the <code>.env</code> already on the server. Don't seed again.</p>
</li>
<li><p><code>invalid credentials</code> for Alice: the browser probably filled in a saved password. Clear the box, type <code>password123</code>, and try again.</p>
</li>
</ul>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p><code>systemctl is-active clearpay-api</code> prints <code>active</code></p>
</li>
<li><p><code>curl -k https://$INSTANCE_IP/health</code> returns <code>"status":"ok"</code></p>
</li>
<li><p>You sent $20 in the browser on the AWS address</p>
</li>
</ul>
<h3 id="heading-chapter-6-record-the-aws-baseline">Chapter 6. Record the AWS Baseline</h3>
<p><em>Wednesday, 10:00 a.m.</em></p>
<p>You message Maya: "health is ok, and Alice signed in on the Elastic IP." Her reply comes back in under a minute.</p>
<p>Maya: "Good. Now run the five checks against that IP. Send me the terminal output, not a screenshot."</p>
<p>A second message follows: "Then load-test it and send me the p95. I can't compare GKE against 'it works'."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>This chapter gives you two things. You'll use both for the rest of the book.</p>
<p><strong>1. A pass from the promotion gate.</strong> The promotion gate is a short script. It tests five things every payment app must do: the app is running, it can reach the database, a user can log in, a repeated payment charges only once, and the balance drops by the right amount. If all five pass, the app works.</p>
<p>In Chapter 15 you run the same script on GKE. If GKE passes the same test AWS passed, GKE does the job, too.</p>
<p><strong>2. A scorecard.</strong> The load test sends many requests at the same time, like many customers using the app at once. Then it writes down how fast the app answered and how many requests failed. That's how fast AWS is today. Later you run the same test on GKE and compare the two.</p>
<p>Think of timing a runner before they change shoes. Without the first time, the second time means nothing. This chapter records the first time.</p>
<p>The scripts in this chapter use Node on your laptop to read JSON. Install Node.js 20 or newer now if you haven't already.</p>
<h4 id="heading-steps">Steps</h4>
<p>Start from the repository root. If you're still in <code>infra/aws/deploy</code>, run <code>cd ../../..</code> first.</p>
<p>Run the promotion gate against AWS:</p>
<pre><code class="language-bash">export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)

./scripts/promotion-gate.sh https://$INSTANCE_IP \
  alice@clearpay.dev password123 bob@clearpay.dev
</code></pre>
<p>Use <code>https://</code>. With <code>http://</code>, nginx redirects and the script looks like it hung.</p>
<p>The gate checks five things, in order:</p>
<ol>
<li><p>The process is up (<code>/health</code>).</p>
</li>
<li><p>It can reach the database (<code>/health/ready</code>).</p>
</li>
<li><p>Alice can log in.</p>
</li>
<li><p>A replayed payment key returns <code>replayed=true</code>.</p>
</li>
<li><p>Alice's balance drops once.</p>
</li>
</ol>
<p>Expected output:</p>
<pre><code class="language-plaintext">PASS: health endpoint returns ok
PASS: readiness endpoint confirms DB connectivity
PASS: login succeeds and returns a token
PASS: replaying an idempotency key returns replayed=true
PASS: balance debited exactly once (100 cents), not twice

5 passed, 0 failed

GATE PASSED: safe to promote to production.
</code></pre>
<p>The last line talks about production. There's no production to promote to yet. It's the same script you run on GKE later, where that line means what it says.</p>
<p>Now load-test AWS. The script logs in and hits <code>/api/accounts/me</code> and <code>/api/transactions</code>, which both query Postgres. Don't load-test <code>/health</code>. The API answers <code>/health</code> by itself, straight away. It never asks the database anything. A real customer request has to ask the database and wait for the answer, and that wait is most of the time:</p>
<pre><code class="language-plaintext">/health                  laptop → API → reply                    (fast, no database)
/api/accounts/me         laptop → API → database → API → reply   (the real work)
</code></pre>
<p>Load-testing <code>/health</code> times only the fast half. Think of asking a shop worker "are you open?" versus "do you have this shoe in size 42?" Only the second question makes them walk to the storeroom.</p>
<pre><code class="language-bash">./scripts/load-test.sh https://$INSTANCE_IP \
  alice@clearpay.dev password123 10 30
</code></pre>
<p>The last two numbers are concurrency (10 requests at once) and duration (30 seconds).</p>
<h4 id="heading-check">Check</h4>
<p>The script prints a scorecard. This is the one this lab printed:</p>
<pre><code class="language-plaintext">== DB load test: https://184.194.120.157  concurrency=10  duration=30s ==
Endpoints: GET /api/accounts/me  GET /api/transactions


ok=372  failed=0

Migration Scorecard, this run

p50 latency:          673 ms
p95 latency:          745 ms
p99 latency:          1326 ms
Throughput:           12.4 req/s
4xx rate:             0.0 %
5xx rate:             0.0 %
Transaction success:  100.0 %
</code></pre>
<p>Your scorecard will differ. Your IP address, request counts, and milliseconds depend on where you are. The load test runs on your laptop, so every request travels from your home to AWS in <code>us-east-1</code> and back.</p>
<p>A reader close to that region sees much lower numbers than a reader on another continent. Neither is wrong. What matters is that you compare GKE against your own scorecard, measured from the same place.</p>
<p>The run passes when you see <code>failed=0</code>, <code>5xx rate</code> at <code>0.0 %</code>, and <code>Transaction success</code> at <code>100.0 %</code>.</p>
<ul>
<li><p>p95 latency means 95 percent of requests finished faster than this number. It shows what slow customers feel, which an average hides.</p>
</li>
<li><p><code>4xx</code> means the client sent a bad request, such as a missing token.</p>
</li>
<li><p><code>5xx</code> means the server failed.</p>
</li>
</ul>
<p>You copy the whole scorecard into a notes file as printed, and send the same block to Maya. Her answer comes back fast: "Don't round it to look tidier. I will compare GKE to this page."</p>
<h4 id="heading-set-your-limits-now">Set Your Limits Now</h4>
<p>Write these two lines under the scorecard, with numbers:</p>
<pre><code class="language-plaintext">Outage:   5xx rate above 1 % for more than 2 minutes in a row
Failure:  p95 more than 3 times your AWS p95 for more than 5 minutes in a row
</code></pre>
<p>The failure limit uses your own AWS p95, because every reader's baseline is different. With this lab's 745 ms, the limit is about 2,235 ms. These are example limits. Pick your own, but pick them now, before GKE exists. A limit you set after you see the GKE numbers is a limit you bent to fit them.</p>
<p>Start the cost ledger with one line:</p>
<pre><code class="language-plaintext">Cost ledger, entry 1:  AWS about $1.25/day (infra/aws/COST.md)
</code></pre>
<h4 id="heading-if-it-fails">If it Fails:</h4>
<p>Stay here until all five checks pass. A problem you carry into the migration comes back later with two clouds involved, and is much harder to find.</p>
<ul>
<li><p>A check fails on readiness: the API can't reach RDS. Go back to Chapter 5's "If it fails" list.</p>
</li>
<li><p><code>node: command not found</code>: install Node.js on your laptop.</p>
</li>
</ul>
<h4 id="heading-decision">Decision</h4>
<p>The gate passed. This EC2 server is the baseline. Every GKE number in this book gets compared against this scorecard.</p>
<p>A minute later Maya adds one more line: "Good. Those are my numbers now."</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>The gate printed <code>5 passed, 0 failed</code> against the AWS IP</p>
</li>
<li><p>The load test printed <code>failed=0</code>, and the scorecard is in your notes</p>
</li>
<li><p>Your outage and failure limits are written down with numbers</p>
</li>
</ul>
<h2 id="heading-part-2-how-to-plan-the-move">Part 2. How to Plan the Move</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/23833564-68ec-469e-a404-cf2daca2d22c.png" alt="23833564-68ec-469e-a404-cf2daca2d22c" style="display: block;" width="600" height="400" loading="lazy">

<p>Tell Maya the gate passed and send her the scorecard.</p>
<p>She writes back: "The next job is GKE. I need the API running there, and I need customers to keep sending money the whole time you move it. Leave Postgres on AWS. The database isn't in this assignment."</p>
<p>No cloud gets touched in this part. You read, map, and decide.</p>
<h3 id="heading-chapter-7-read-the-migration-assignment">Chapter 7. Read the Migration Assignment</h3>
<p><em>Wednesday, 4:00 p.m.</em></p>
<p>Maya forwards the assignment with one line on top: "Read all of it before you open Terraform."</p>
<h4 id="heading-the-assignment-migrate-the-clearpay-api-from-aws-ec2-to-gke">The Assignment: Migrate the ClearPay API from AWS EC2 to GKE.</h4>
<p>ClearPay runs on one EC2 server. It works, but it has no autoscaling, no staged rollout, and a deploy process that lives in one person's head. You ran that process by hand in Chapter 5, so you know what it lacks.</p>
<p>The business wants the API on Kubernetes: a staging environment, automated deploys, and room to grow. Again, the database isn't part of this assignment.</p>
<h4 id="heading-current-architecture">Current Architecture</h4>
<p>ClearPay lives in one AWS account, in the default VPC from Chapter 4: one EC2 server and one RDS database.</p>
<h4 id="heading-what-cant-change">What Can't Change</h4>
<ul>
<li><p>The database. Moving Postgres is a separate, later project.</p>
</li>
<li><p>The API contract. The same paths return the same JSON: <code>/health</code>, <code>/api/auth/login</code>, <code>/api/transactions</code>. A browser or script that works on AWS must work on GKE with no changes.</p>
</li>
<li><p>Customers see no lasting outage at any point.</p>
</li>
</ul>
<h4 id="heading-acceptance-criteria">Acceptance Criteria</h4>
<p>Done means this, in order:</p>
<ol>
<li><p>GKE serves all production traffic, and it stays that way. Errors and latency stay inside the limits you wrote in Chapter 6.</p>
</li>
<li><p>While AWS still exists, rollback to AWS still works.</p>
</li>
<li><p>Then you retire AWS, after GKE has proven itself.</p>
</li>
</ol>
<h4 id="heading-what-moves-and-what-stays">What moves and what stays:</h4>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/680819f1-1c81-49b3-9b04-4cfc2aec9cad.png" alt="Migration diagram showing the ClearPay UI and API moving from an AWS EC2 server to Google Kubernetes Engine (GKE) on GCP. GKE hosts staging and production. PostgreSQL remains on Amazon RDS in AWS and is shared by both environments." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram shows the migration from AWS to GCP. The ClearPay UI and API move from the EC2 server to Google Kubernetes Engine (GKE), where staging and production run. PostgreSQL remains on Amazon RDS in AWS, so both environments use the same database during the migration.</p>
<p>The whole book follows one path:</p>
<pre><code class="language-plaintext">AWS only  →  AWS and GKE together  →  GKE only, then EC2 gone
</code></pre>
<p>The middle step is where the work happens. Both clouds run the same app, both talk to the same database, and traffic moves between them in small steps.</p>
<h4 id="heading-how-traffic-will-move">How Traffic Will Move</h4>
<p>Today, customers type the Elastic IP and every request goes to AWS.</p>
<ol>
<li><p>You put a Google load balancer in front, and customers switch to its address once. They keep using that address for the rest of the migration.</p>
</li>
<li><p>The load balancer then sends some requests to AWS and the rest to GKE. It starts with <strong>100% AWS and 0% GKE</strong>.</p>
</li>
<li><p>You raise the GKE share by hand with <code>scripts/shift-traffic.sh</code>, a few steps at a time, checking numbers after each step.</p>
</li>
<li><p>If something goes wrong on GKE, you send <strong>100% of the traffic back to AWS</strong> with <code>shift-traffic.sh 100 0</code>. The rollback takes seconds.</p>
</li>
</ol>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e1df65a6-b164-403e-935b-f7bbd6500682.png" alt="Architecture diagram showing users connecting through one public address to the ClearPay UI and API running on AWS and Google Kubernetes Engine (GKE). Both environments connect to the same PostgreSQL database on Amazon RDS in AWS." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram above shows how customers reach ClearPay through one public address while the UI and API run on both AWS and GKE during migration. Both environments connect to the same PostgreSQL database on Amazon RDS in AWS. Keeping the database in one place avoids migrating payment data alongside the application.</p>
<p>Why not use DNS to split traffic? DNS is the lookup that turns a name into an IP address. Laptops and internet providers cache the answer.</p>
<p>After you change a DNS record, some customers keep going to the old place for minutes or hours. A rollback that takes an hour isn't a rollback. The load balancer changes the split in seconds.</p>
<h4 id="heading-rollback-requirement">Rollback Requirement</h4>
<p>At every stage of the shift, going back to 100 percent AWS must be fast and safe. Not a redeploy. Not a DNS change that takes an hour to spread. Chapter 20 tests exactly that, before the first shift.</p>
<h4 id="heading-what-zero-downtime-means-here">What "Zero Downtime" Means Here</h4>
<p>It doesn't mean zero failed requests, ever. No real system can promise that.</p>
<p>It means no customer sees a lasting outage. A few failed requests are allowed if they stay under the limits you wrote in Chapter 6.</p>
<h4 id="heading-why-the-api-moves-first">Why the API Moves First</h4>
<p>If the API and the database move together and something breaks, the fault could be in either one, or in how they interact. Moving only the API keeps the question small: can this API run on GKE without hurting customers?</p>
<p>That's an answerable question. "Can you move everything at once safely?" is much harder to answer, and much worse to debug live.</p>
<h4 id="heading-why-the-database-stays-on-aws">Why the Database Stays on AWS</h4>
<p>Chapter 13 shows what connecting across clouds involves. For now: PostgreSQL doesn't move in this project.</p>
<p>A future, separate project handles that, most likely with Google's Database Migration Service, continuous replication, and its own cutover. This project ends with the API on GKE and the database still on AWS. That's the plan, not an unfinished loose end.</p>
<h4 id="heading-why-both-clouds-must-run-at-the-same-time">Why Both Clouds Must Run at the Same Time</h4>
<p>The rollback plan depends on it. You send a little traffic to GKE and keep AWS ready. If GKE fails, AWS takes everything back. That only works while AWS still exists.</p>
<h4 id="heading-set-a-migration-budget-before-touching-gcp">Set a Migration Budget Before Touching GCP</h4>
<p><strong>Do this now, before Chapter 9.</strong> Write four numbers in your notes. They decide how much this move may cost you.</p>
<p>Plan your spending the way you plan money for a trip:</p>
<ol>
<li><p><strong>What AWS costs today.</strong> About $1.25 a day, from <code>infra/aws/COST.md</code>.</p>
</li>
<li><p><strong>What GCP will cost per day.</strong> You don't know yet. Write "TBD" (to be decided). Chapter 9 works it out. <code>infra/gcp/COST.md</code> puts it at about $1.60 to $5 a day.</p>
</li>
<li><p><strong>What the move costs while both clouds run.</strong> During the move you pay AWS and GCP at the same time. That is $1.25 plus the GCP daily cost, for every day of the move.</p>
</li>
<li><p><strong>Your budget.</strong> The most you will spend on the whole move. Pick it now, before you spend anything.</p>
</li>
</ol>
<p><strong>This lab's numbers:</strong></p>
<pre><code class="language-plaintext">Current AWS cost:   $1.25/day
Expected GCP cost:  about $3/day (Chapter 9 checks this)
Dual-run cost:      $1.25 + $3 = $4.25/day while both run
Migration budget:   5 days × $4.25 = about $21. Round up to $25.
</code></pre>
<p>Keep a running total as you go. Chapters 19 to 25 add the real dual-run days. Chapter 28 asks one question: did you stay under your budget? If the move took longer, for example because of a rollback, the total shows exactly what that cost.</p>
<h4 id="heading-migration-phases">Migration phases</h4>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/055183da-5e97-451b-a341-917bd5e1d198.png" alt="Flow diagram showing 12 migration phases in sequence: investigate, design, provision, containerize, deploy staging, promote to production, validate, shift traffic gradually, handle failures, complete the cutover, retire AWS, and review. Arrows connect the phases in order across three rows." style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-done-when">Done When:</h4>
<ul>
<li>You can explain to Maya, in two sentences, what moves, what stays, and how you undo a bad step</li>
</ul>
<h3 id="heading-chapter-8-map-what-clearpay-depends-on">Chapter 8. Map What ClearPay Depends On</h3>
<p><em>Thursday, 9:00 a.m.</em></p>
<p>Before anyone stands up GKE, the team needs a written map of what ClearPay depends on. This system has run by hand since Tuesday. Nothing on paper says what must keep working during the move.</p>
<p>Maya: "Before we stand up another cloud, write down what this system depends on."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>A migration without a map moves the wrong thing, or misses something that breaks the day you switch. Answer each question below from the repository, not from memory.</p>
<h4 id="heading-1-is-the-app-ready-for-containers">1. Is the app ready for containers?</h4>
<p>Yes. <code>api/Dockerfile</code> has worked since Chapter 1. What changes is its role. On your laptop it was a convenience. On GKE it becomes the thing production runs.</p>
<h4 id="heading-2-where-do-the-secrets-live">2. Where do the secrets live?</h4>
<p>Open <code>infra/aws/deploy/systemd/clearpay-api.service</code>. It loads <code>EnvironmentFile=/opt/clearpay-api/.env</code>. That file isn't in Git. It sits on the EC2 server, where <code>deploy.sh</code> copied it in Chapter 5.</p>
<p>Every secret the API needs is in one plain text file on one server. Note that. Don't open it to print secrets. GKE will read the same values from Google Secret Manager instead.</p>
<h4 id="heading-3-what-outside-systems-does-it-call">3. What outside systems does it call?</h4>
<p>One: PostgreSQL. The connection stays inside the AWS default VPC, and RDS has no public endpoint. There's no third-party API. Confirm that by reading <code>api/src/</code>.</p>
<h4 id="heading-4-does-the-server-remember-users-between-requests">4. Does the server remember users between requests?</h4>
<p>No. Open <code>api/src/index.js</code> and <code>api/src/routes/transactions.js</code>. Each request carries a JWT (a signed login token), and all account data lives in Postgres. The server holds nothing in memory.</p>
<p>That's why AWS and GKE can split requests later. A user can log in on AWS and send money on GKE, because both clouds check the same token secret and read the same database.</p>
<h4 id="heading-5-can-a-retry-charge-twice">5. Can a retry charge twice?</h4>
<p>No. Chapter 3 proved the idempotency key works. It must keep working when a first try lands on AWS and the retry lands on GKE.</p>
<h4 id="heading-the-map">The Map</h4>
<p>Write these five answers in your notes:</p>
<pre><code class="language-plaintext">Container ready:   yes, api/Dockerfile
Secrets:           one .env file on the EC2 server
Outside systems:   PostgreSQL only
Server state:      none, JWT + database
Retry safety:      idempotency key, proven in Chapter 3
</code></pre>
<h4 id="heading-what-actually-needs-to-move">What Actually Needs to Move</h4>
<p>Only the UI and the API: off the EC2 server, onto GKE. Postgres stays on AWS.</p>
<p>Browsers and scripts still hit the same URLs and get the same JSON back. A retry still must not take money twice. Those rules don't change.</p>
<h4 id="heading-done-when"><strong>Done when</strong></h4>
<ul>
<li>All five answers are written down, each pointing at a file</li>
</ul>
<h3 id="heading-chapter-9-understand-the-gcp-design">Chapter 9. Understand the GCP Design</h3>
<p><em>Thursday, 2:30 p.m.</em></p>
<p>The Terraform for Google Cloud is already written. Somebody made two big decisions in it before you joined. Maya wants to know whether you can defend them. She writes in the team channel.</p>
<p>Maya: "Open <code>infra/gcp/COST.md</code> and <code>infra/gcp/terraform/03-gke</code>. Then tell me why this repo uses GKE Standard, not Autopilot. I need a reason I can defend."</p>
<p>Daniel, who sits at the next desk, has <code>infra/gcp/terraform/03-gke/main.tf</code> open. He points at a block named <code>primary</code>. It's the node pool: the group of virtual machines GKE runs the app on. This one holds one to three <code>e2-medium</code> machines.</p>
<p>Daniel: "If we keep these machines, how many copies of the API can run on them at once?"</p>
<p>This chapter doesn't run Terraform. The files are already written. You read the decisions in them.</p>
<h4 id="heading-standard-or-autopilot">Standard or Autopilot</h4>
<p>GKE is Google's managed Kubernetes. It runs in two modes:</p>
<ul>
<li><p><strong>Standard</strong>: you choose the virtual machines, grouped in a node pool. Google bills those machines whether the app is busy or idle.</p>
</li>
<li><p><strong>Autopilot</strong>: Google hides the machines. You say how much CPU and memory each container needs, and Google bills that.</p>
</li>
</ul>
<p>This repo uses Standard. The reason isn't cost. It keeps node pools, machine sizes, and autoscaling visible, and those are what this book teaches. Autopilot would hide them.</p>
<p>Work out both costs yourself. Standard bills the machines. One <code>e2-medium</code> running all day is about $1.44 in <code>infra/gcp/COST.md</code>, plus Cloud NAT.</p>
<p>Autopilot bills the CPU and memory each pod reserves. <code>k8s/base/deployment.yaml</code> reserves <code>250m</code> CPU (a quarter of one CPU core) and <code>256Mi</code> of memory per API pod. Using the Autopilot rates in <code>infra/gcp/COST.md</code>, three API pods cost roughly:</p>
<pre><code class="language-plaintext">0.25 vCPU × 3 × $0.0445/hour × 24  ≈  $0.80 / day
0.25 GiB  × 3 × $0.0049/hour × 24  ≈  $0.09 / day
</code></pre>
<p>That's about $0.89 a day for three API pods, before the UI pods. Autopilot can cost less than one Standard machine. The lab still uses Standard so you can see the node pool.</p>
<p>Tell Maya both numbers and the reason.</p>
<h4 id="heading-how-many-api-processes">How Many API Processes</h4>
<p>AWS runs one API process. On GKE, a HorizontalPodAutoscaler (HPA) adds or removes API pods based on CPU use. Each pod is one API process with its own pool of ten database connections.</p>
<p>This lab's cluster is three small <code>e2-medium</code> machines. The production HPA in <code>k8s/overlays/production/patch-hpa.yaml</code> runs between 1 and 3 API pods, because three machines of that size can't fit more alongside GKE's own system pods.</p>
<p>Answer Daniel: "One to three API pods on GKE, plus one on AWS while both run. Each can open ten connections."</p>
<p>A company with bigger machines would set a higher ceiling, such as 3 to 8 pods. Chapter 22 shows why that ceiling needs checking against the database limit before traffic moves.</p>
<h4 id="heading-a-designed-network">A Designed Network</h4>
<p>On AWS, this lab used the default VPC that AWS created for you. Nobody designed it.</p>
<p>On GCP, <code>infra/gcp/terraform/02-network</code> builds its own VPC with a named subnet and Cloud NAT. The cluster's machines get private addresses only. They reach the internet, and later the AWS database, through Cloud NAT, which gives all of them one shared public IP.</p>
<h4 id="heading-region">Region</h4>
<p><code>01-foundation/variables.tf</code> sets the region to <code>us-east4</code>, the Google region closest to AWS's <code>us-east-1</code>. Every database query from GKE crosses between the two clouds, so a short distance keeps that trip short. If your AWS setup isn't in <code>us-east-1</code>, change this before you build anything.</p>
<h4 id="heading-the-map">The Map</h4>
<p>The left side is what exists. The right side is what Part 3 builds.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/d52f2aab-d663-4081-a645-52871d564020.png" alt="Migration architecture diagram showing customers reaching a single public load balancer in GCP, which distributes traffic between an AWS EC2 server running nginx, the UI and API, and GKE in GCP running UI and API pods on private nodes. Cloud NAT provides outbound access from GKE. Both environments connect to the same PostgreSQL 16 database on AWS. Traffic shifts gradually from AWS to GKE." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram above maps the migration from AWS to GCP. Customers use one public address, managed by the GCP load balancer, which splits traffic between the existing AWS EC2 server and GKE. GKE runs the UI and API in private nodes, with Cloud NAT providing outbound internet access. Both environments connect to the same PostgreSQL database on AWS. Traffic shifts gradually to GKE while the database stays in place.</p>
<p>You'll meet three public IP addresses, in this order:</p>
<ol>
<li><p>The Elastic IP. That's the EC2 server. Customers use it today.</p>
</li>
<li><p>The GKE Gateway IP. The load balancer sends GKE's share of traffic to it. Customers never type it.</p>
</li>
<li><p>The load balancer IP from <code>08-edge</code>. Customers switch to this address once, and the traffic split happens behind it.</p>
</li>
</ol>
<h4 id="heading-if-you-need-to-pause-aws">If You Need to Pause AWS</h4>
<p>Part 3 doesn't need AWS running every minute. AWS is needed again in Chapter 11 (to copy the database URL into Secret Manager) and Chapter 13 (to open the database to GKE).</p>
<p>Don't stop the EC2 server and leave RDS running. RDS is most of the daily AWS bill, a stopped RDS restarts itself after seven days, and the Elastic IP bills either way.</p>
<p>If you continue within a day or two, leave AWS up. If you stop for several days, destroy it:</p>
<pre><code class="language-bash">cd infra/aws/terraform &amp;&amp; terraform destroy
</code></pre>
<p>To come back, run <code>terraform apply</code> in that folder and repeat Chapter 5. You get a new Elastic IP and a new <code>DATABASE_URL</code>. Your Chapter 6 scorecard still counts as the baseline.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>You told Maya the Standard and Autopilot daily numbers and why the lab uses Standard</p>
</li>
<li><p>You told Daniel how many API processes can run, and how many connections each can open</p>
</li>
</ul>
<h2 id="heading-part-3-how-to-build-google-cloud-with-terraform">Part 3. How to Build Google Cloud With Terraform</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/2c0327bb-ec77-4ed5-9dd4-f0ac575e150c.png" alt="Image describing what we have done, where we are and what next." style="display: block;" width="600" height="400" loading="lazy">

<p>There's a working AWS app and a scorecard. Google Cloud isn't set up yet. There's no network, no cluster, and no registry until you build them.</p>
<p>The UI and the API will run on that cluster later. Postgres stays on AWS.<br>So the order is: build an empty place in GCP first, then put the app there, then teach the app how to reach RDS across the internet.</p>
<p>This part is only the empty place. Customers don't move yet.</p>
<h3 id="heading-chapter-10-set-up-gcloud-and-pick-your-project">Chapter 10. Set Up <code>gcloud</code> and Pick Your Project</h3>
<p><em>Friday, 8:45 a.m.</em></p>
<p>Google Cloud is a blank page. Before you build anything, your laptop needs to know which project to build in, and who you are.</p>
<p>Get <code>gcloud</code>, <code>kubectl</code>, and Terraform pointed at the right Google project, with billing on.</p>
<h4 id="heading-steps">Steps</h4>
<p><code>gcloud</code> is the Google Cloud command-line tool. Install it, then close the terminal and open a new one.</p>
<p>On a Mac:</p>
<pre><code class="language-bash">brew install --cask google-cloud-sdk
</code></pre>
<p>On Windows, download the installer from <a href="https://cloud.google.com/sdk/docs/install">Install the Google Cloud CLI</a>, tick the box that adds <code>gcloud</code> to PATH, and reopen Git Bash. On Linux, follow the Linux tab on the same page. <code>gcloud version</code> should print a version.</p>
<p>If <code>kubectl version --client</code> fails after <code>gcloud</code> works, install <code>kubectl</code> and the plugin it uses to log in to GKE:</p>
<pre><code class="language-bash">gcloud components install kubectl
gcloud components install gke-gcloud-auth-plugin
</code></pre>
<p>The plugin must be on your PATH. If a later <code>kubectl get nodes</code> says <code>gke-gcloud-auth-plugin not found</code>, run this and open a new terminal:</p>
<pre><code class="language-bash">echo 'export PATH="$(gcloud info --format="value(installation.sdk_root)")/bin:$PATH"' &gt;&gt; ~/.zshrc
</code></pre>
<p>That line is for zsh, the default Mac shell. On Linux with Bash, change <code>~/.zshrc</code> to <code>~/.bashrc</code>. On Windows, the installer usually handles PATH. Reopen Git Bash after the install.</p>
<p>Now log in and pick the project:</p>
<pre><code class="language-bash">gcloud auth login --update-adc
gcloud projects list
gcloud config set project YOUR_PROJECT_ID
gcloud billing projects describe YOUR_PROJECT_ID
terraform version
kubectl version --client
</code></pre>
<p><code>--update-adc</code> matters. Plain <code>gcloud auth login</code> only lets <code>gcloud</code> use your account. Terraform is a separate program that looks for a file called Application Default Credentials. The flag writes that file in the same browser login. On the Google page, click Allow and don't close the tab early.</p>
<h4 id="heading-which-id-to-use">Which ID to Use</h4>
<p><code>gcloud projects list</code> prints two columns that look alike:</p>
<ul>
<li><p><code>NAME</code> is a label, and many projects share the default label <code>My First Project</code>.</p>
</li>
<li><p><code>PROJECT_ID</code> is unique, for example <code>my-first-project-123456</code>.</p>
</li>
</ul>
<p>Copy the <code>PROJECT_ID</code>, never the name. Paste it wherever the commands say <code>YOUR_PROJECT_ID</code>.</p>
<p>If you have several projects, pick one with billing on. Run <code>gcloud billing projects describe</code> with each ID until the output includes <code>billingEnabled: true</code>.</p>
<p>You can also find the ID in the <a href="https://console.cloud.google.com">Google Cloud Console</a>: click the project name at the top of the page and read the ID column.</p>
<h4 id="heading-check">Check</h4>
<pre><code class="language-bash">gcloud config get-value project
</code></pre>
<p>This prints your Project ID. If it prints <code>[None]</code>, run the <code>gcloud config set project</code> command again.</p>
<h4 id="heading-if-it-fails">If it Fails:</h4>
<ul>
<li>Terraform later says <code>could not find default credentials</code>: you logged in without the flag. Run:</li>
</ul>
<pre><code class="language-bash">gcloud auth application-default login
</code></pre>
<ul>
<li>That command says <code>cloud-platform scope is required but not consented</code>: the browser closed before you clicked Allow. Run <code>gcloud auth login --update-adc</code> again and click Allow.</li>
</ul>
<p>Today is only a <code>gcloud</code> login. The name <code>api.clearpay.test</code> shows up later, when Stage 8 builds the load balancer. Skip it until then.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p><code>gcloud config get-value project</code> prints your Project ID</p>
</li>
<li><p><code>gcloud billing projects describe</code> shows <code>billingEnabled: true</code></p>
</li>
<li><p><code>kubectl version --client</code> prints a version</p>
</li>
</ul>
<p>Message Maya: "gcloud is logged in. The project is set. Billing is on. Ready to start GCP."</p>
<p>Maya: "Build one stage at a time. Tell me when each one is verified. If you run all eight at once and the third one breaks, you won't know which one to fix."</p>
<h3 id="heading-chapter-11-build-gcp-in-eight-terraform-stages">Chapter 11. Build GCP in Eight Terraform Stages</h3>
<p><em>Friday, 10:30 a.m., to Monday of week two</em></p>
<p>GCP is built in eight Terraform stages. Each stage creates one layer of the destination environment.</p>
<p>Run them in order. After each stage, verify the result before moving to the next.</p>
<p>Build the Google side of the migration: APIs, network, cluster, image registry, identities, secrets, error alerting, and the public load balancer.</p>
<h4 id="heading-why-eight-stages">Why Eight Stages?</h4>
<p>The Terraform lives in eight folders, <code>01-foundation</code> through <code>08-edge</code>. Each folder has its own state file and reads only the earlier stages it needs. If <code>03-gke</code> fails, the problem is in the cluster, not somewhere in one 1,500-line file.</p>
<p>This isn't "run Terraform eight times". The work is to build eight things and prove each one exists before the next one needs it.</p>
<p>State stays local on your laptop, which is fine for one person. A team would move each stage's state to a Google Cloud Storage bucket before a second person runs <code>apply</code>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/78d0dd12-38ee-4390-a0be-0eecab7f662e.png" alt="Diagram of eight Terraform stages inside a GCP project. Stage 1 enables APIs. Stage 2 creates the network and subnet. Stage 3 creates a private GKE cluster and Cloud NAT with a NAT IP. Stage 4 creates the image registry. Stage 5 configures identities. Stage 6 configures secrets. Stage 7 sets up error metrics and alerts. Stage 8 creates the public load balancer. The network provides internet access and later connectivity to AWS RDS." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram above shows how Terraform builds GCP infrastructure in eight stages. It enables the required APIs, creates the network and private GKE cluster with Cloud NAT, then sets up the image registry, identities, secrets, error monitoring, and public load balancer. Cloud NAT provides outbound connectivity for private nodes, including access to AWS RDS later in the migration.</p>
<h4 id="heading-how-every-stage-runs">How Every Stage Runs</h4>
<p>Each stage uses the same three commands inside its folder:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Read the plan before you apply. Every stage below lists the plan count to expect, for example <code>Plan: 7 to add, 0 to change, 0 to destroy</code>. If the count differs, or the plan wants to destroy something, stop and read it before typing <code>yes</code>.</p>
<p><code>terraform apply</code> asks <code>Do you want to perform these actions?</code>. Type <code>yes</code> and press Enter.</p>
<p>Folder paths below start from the repository root. If you're already inside <code>infra/gcp/terraform/0X-...</code>, use <code>cd ../0Y-...</code> to move to the next stage.</p>
<p>For the "In the console" checks: pick your project at the top of the page, and set the region to <code>us-east4</code> when a page has a region menu. An empty list often means the menu is still on <code>us-central1</code>.</p>
<h4 id="heading-stage-1-turn-on-the-google-apis-01-foundation">Stage 1: Turn on the Google APIs (<code>01-foundation</code>)</h4>
<p>Google Cloud services are off by default in a new project. This stage turns on ten of them: GKE, Artifact Registry, Secret Manager, IAM, logging, and others. If you skip it, <code>03-gke</code> fails later with a permission error that points nowhere useful.</p>
<pre><code class="language-bash">cd infra/gcp/terraform/01-foundation
cp terraform.tfvars.example terraform.tfvars
</code></pre>
<p>If <code>terraform.tfvars</code> already exists with your Project ID, skip the copy. A second copy overwrites your edit.</p>
<p>Open <code>terraform.tfvars</code> and replace <code>clearpay-migration-demo</code> with your Project ID. Then run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Expected plan: <code>google_project_service</code> ten times, and no cluster.</p>
<pre><code class="language-plaintext">Plan: 10 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + project_id = "YOUR_PROJECT_ID"
  + region     = "us-east4"
</code></pre>
<p>If the plan prints <code>clearpay-migration-demo</code>, fix <code>terraform.tfvars</code> first. Secret Manager and Service Networking can sit on <code>Still creating</code> for about half a minute during apply.</p>
<p>Check:</p>
<pre><code class="language-bash">gcloud services list --enabled --filter='config.name:container.googleapis.com'
</code></pre>
<pre><code class="language-plaintext">NAME                      TITLE
container.googleapis.com  Kubernetes Engine API
</code></pre>
<p>If this prints nothing, the API isn't on for the project <code>gcloud</code> is looking at. Don't go to Stage 2 yet. Check these three causes, in order:</p>
<ol>
<li><p><code>gcloud</code> <strong>points at a different project.</strong> Run <code>gcloud config get-value project</code>. It must print the same Project ID you put in <code>terraform.tfvars</code>. If it doesn't, run <code>gcloud config set project YOUR_PROJECT_ID</code> and check again.</p>
</li>
<li><p><strong>The apply didn't finish.</strong> Run <code>terraform plan</code> in <code>01-foundation</code> again. A finished stage prints <code>No changes</code>. If the plan still wants to add resources, run <code>terraform apply</code> and type <code>yes</code>.</p>
</li>
<li><p><strong>Google hasn't caught up yet.</strong> A newly enabled API can take a minute to show in the list. Wait one minute and run the check again.</p>
</li>
</ol>
<p>Message Maya: "the first stage applied, and Kubernetes Engine API shows as enabled."</p>
<p>Maya: "Good. Network next. Nothing else."</p>
<h4 id="heading-stage-2-build-the-network-02-network">Stage 2: Build the Network (<code>02-network</code>)</h4>
<p>A VPC is a private network inside your Google project. Think of it as the neighborhood the cluster will live in, and the subnet as one street in it. This stage creates:</p>
<ul>
<li><p><code>clearpay-vpc</code>, the private network</p>
</li>
<li><p><code>clearpay-gke-subnet</code>, the address range the cluster's machines use, with two extra ranges named <code>pods</code> and <code>services</code> reserved for Stage 3</p>
</li>
<li><p><code>clearpay-router</code> and <code>clearpay-nat</code>, the Cloud NAT setup</p>
</li>
<li><p><code>clearpay-nat-ip</code>, the one public IP that all cluster machines share when they call out</p>
</li>
<li><p>two firewall rules so machines inside can talk to each other and Google's health checks can reach them</p>
</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/4934e0f5-ed7f-4de7-879d-9e04cec275e1.png" alt="The diagram shows the GCP network setup. Private machines run inside a subnet within the  and have no public IP addresses. Cloud NAT provides outbound internet access through the static IP . The network also supports connectivity to the AWS RDS database later in the migration." style="display: block;" width="600" height="400" loading="lazy">

<p>This folder has no <code>terraform.tfvars</code>. It reads your Project ID from Stage 1's state file.</p>
<pre><code class="language-bash">cd infra/gcp/terraform/02-network
</code></pre>
<p>Run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Expected plan:</p>
<pre><code class="language-plaintext">Plan: 7 to add, 0 to change, 0 to destroy.
</code></pre>
<p>Expected end of apply:</p>
<pre><code class="language-plaintext">Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

nat_ip_address = "x.x.x.x"
network_name = "clearpay-vpc"
subnet_name = "clearpay-gke-subnet"
</code></pre>
<p>Check:</p>
<pre><code class="language-bash">gcloud compute networks describe clearpay-vpc
terraform output nat_ip_address
</code></pre>
<p>Look for <code>name: clearpay-vpc</code>, a <code>subnetworks</code> line ending in <code>clearpay-gke-subnet</code>, and <code>autoCreateSubnetworks: false</code>. That last line means this network was designed, not generated.</p>
<p>Write the NAT IP down. Chapter 13 opens the AWS database to exactly this address.</p>
<p>In the console: <a href="https://console.cloud.google.com/networking/networks/list">VPC networks</a> shows <code>clearpay-vpc</code>. <a href="https://console.cloud.google.com/net-services/nat/list">Cloud NAT</a> shows <code>clearpay-nat</code> on <code>clearpay-router</code>. <a href="https://console.cloud.google.com/networking/addresses/list">IP addresses</a> shows <code>clearpay-nat-ip</code>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/fe7ffa07-ae26-4095-91a3-d9a62708c7e5.png" alt="image showing cloud NAT on gcp console" style="display: block;" width="600" height="400" loading="lazy">

<p>If the plan can't read <code>../01-foundation/terraform.tfstate</code>, Stage 1 isn't done.</p>
<p>Message Maya: "<code>clearpay-vpc</code> and Cloud NAT exist. The NAT IP is in your notes."</p>
<p>Daniel sees the NAT IP in your notes.</p>
<p>Daniel: "Every call our pods make to the outside world leaves from that one address. That includes calls to the database on AWS. Later we tell the AWS database to accept connections from this address and nothing else."</p>
<p>That happens in Chapter 13. Keep the NAT IP in your notes until then.</p>
<h4 id="heading-stage-3-create-the-gke-cluster-03-gke">Stage 3: Create the GKE Cluster (<code>03-gke</code>)</h4>
<p>This stage creates a Standard cluster named <code>clearpay</code> in zone <code>us-east4-a</code>, with a node pool named <code>primary</code> of one to three <code>e2-medium</code> machines. The machines have private IPs only and use Cloud NAT to reach out. The control plane, Google's managed API server for the cluster, has a public endpoint so <code>kubectl</code> on your laptop can reach it.</p>
<p>GKE first creates a default node pool, then Terraform deletes it and creates <code>primary</code>, which you control.</p>
<pre><code class="language-bash">cd infra/gcp/terraform/03-gke
</code></pre>
<p>Run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Expected plan: the cluster and the <code>primary</code> node pool.</p>
<pre><code class="language-plaintext">Plan: 2 to add, 0 to change, 0 to destroy.
</code></pre>
<p>The apply sits on <code>Still creating</code> for seven to ten minutes. Don't press Ctrl+C.</p>
<pre><code class="language-plaintext">Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

cluster_name = "clearpay"
get_credentials_command = "gcloud container clusters get-credentials clearpay --zone us-east4-a --project YOUR_PROJECT_ID"
</code></pre>
<p><code>cluster_endpoint</code> prints <code>&lt;sensitive&gt;</code>. You don't need it.</p>
<p>Point <code>kubectl</code> at the new cluster:</p>
<pre><code class="language-bash">$(terraform output -raw get_credentials_command)
</code></pre>
<p>A good run prints <code>kubeconfig entry generated for clearpay</code>.</p>
<p>Check:</p>
<pre><code class="language-bash">kubectl get nodes
</code></pre>
<pre><code class="language-plaintext">NAME                                 STATUS   ROLES    AGE   VERSION
gke-clearpay-primary-........        Ready    &lt;none&gt;   2m    v1.xx
</code></pre>
<p><code>Ready</code> means the machine is on and joined the cluster. ClearPay isn't on it yet.</p>
<p>If the apply mentions <code>container.googleapis.com</code>, go back to Stage 1. If it mentions the subnet or the <code>pods</code> or <code>services</code> ranges, go back to Stage 2. Don't edit <code>03-gke</code> to hide those errors.</p>
<p>If <code>kubectl</code> says <code>gke-gcloud-auth-plugin not found</code>, the cluster is fine. Don't destroy it. Install the plugin and fix PATH for this terminal:</p>
<pre><code class="language-bash">gcloud components install gke-gcloud-auth-plugin
export PATH="$(gcloud info --format='value(installation.sdk_root)')/bin:$PATH"
kubectl get nodes
</code></pre>
<p>In the console: <a href="https://console.cloud.google.com/kubernetes/list/overview">Kubernetes clusters</a> shows <code>clearpay</code> running. <a href="https://console.cloud.google.com/compute/instances">VM instances</a> shows a <code>gke-clearpay-primary-</code> machine with an internal IP and no external IP.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/c4ad49c3-5313-48d9-906d-626ffa23be02.png" alt="image showing gke created in the console" style="display: block;" width="600" height="400" loading="lazy">

<p>Message Maya: "the cluster is up. One node is Ready. Nothing of ours runs on it yet."</p>
<p>The apply took nine minutes. You watched every one of them.</p>
<h4 id="heading-stage-4-create-the-image-registry-04-registry">Stage 4: Create the Image Registry (<code>04-registry</code>)</h4>
<p>A container image is a packaged copy of the app. GKE downloads images from a registry. This stage creates a private Docker registry in Artifact Registry, named <code>clearpay</code>, in <code>us-east4</code>. It stays empty until the pipeline pushes to it in Chapter 14.</p>
<pre><code class="language-bash">cd infra/gcp/terraform/04-registry
</code></pre>
<p>Run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>The two resources are the registry and a rule that lets the cluster's machines pull images from it.</p>
<pre><code class="language-plaintext">Plan: 2 to add, 0 to change, 0 to destroy.
</code></pre>
<pre><code class="language-plaintext">Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

repository_url = "us-east4-docker.pkg.dev/YOUR_PROJECT_ID/clearpay"
</code></pre>
<p>Check:</p>
<pre><code class="language-bash">gcloud artifacts repositories describe clearpay --location=us-east4
</code></pre>
<p>Look for <code>format: DOCKER</code>, a <code>name</code> ending in <code>/repositories/clearpay</code>, and <code>Repository Size: 0.000MB</code>. An empty registry is correct today. You'll also see <code>SCANNING_DISABLED</code>. The pipeline scans images with Trivy instead, in Chapter 14.</p>
<p>Message Maya: "the image registry exists. It's empty."</p>
<h4 id="heading-stage-5-create-identities-with-no-key-files-05-identity">Stage 5: Create Identities with No Key Files (<code>05-identity</code>)</h4>
<p>Two things will need Google permissions later: the ClearPay pods on GKE, and GitHub Actions when it pushes images and deploys.</p>
<p>The old way is to download a service account key file. A key file is a password on disk. If it leaks, anyone can use it. This lab never creates one.</p>
<p>Instead, this stage creates three Google service accounts (identities with permissions) and lets named callers borrow them for short periods:</p>
<ul>
<li><p><code>clearpay-api-staging</code>, borrowed by pods in the <code>staging</code> namespace</p>
</li>
<li><p><code>clearpay-api-production</code>, borrowed by pods in the <code>production</code> namespace</p>
</li>
<li><p><code>clearpay-ci</code>, borrowed by GitHub Actions through a Workload Identity pool named <code>clearpay-github-pool</code></p>
</li>
</ul>
<p>Workload Identity is the Google feature that links a Kubernetes service account, or a GitHub repository, to a Google service account without a key.</p>
<p>Think of each service account as a badge. Nothing wears a badge today. Later, Google lets a named pod or a named repository wear one for a short time, and no key file ever exists.</p>
<p>Namespaces are separate areas inside one cluster. Staging and production share the cluster's machines but not their identities, so a staging pod can't read production's secrets. A real company would usually run separate clusters, often in separate projects. This lab uses one cluster to keep the bill small.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/9a167f0d-f157-4dd5-81dd-877eb25bc2d6.png" alt="Diagram showing one GKE cluster named  with separate staging and production namespaces. Staging uses the  service account, while production uses . GitHub Actions uses the separate  service account for CI/CD operations." style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-bash">cd infra/gcp/terraform/05-identity
cp terraform.tfvars.example terraform.tfvars
</code></pre>
<p>The only value in the file is <code>github_repo</code>. If you already have your GitHub copy of this repository, put its <code>owner/name</code> there now, with no <code>.git</code> at the end. If not, leave <code>YOUR_GITHUB_USER/YOUR_REPO</code>. Chapter 14 tells you when to set it and apply this stage again.</p>
<p>Run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Expected plan:</p>
<pre><code class="language-plaintext">Plan: 10 to add, 0 to change, 0 to destroy.
</code></pre>
<p>The ten are the three service accounts, the GitHub pool and its provider, and the permission links between them.</p>
<pre><code class="language-plaintext">Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

ci_service_account_email = "clearpay-ci@YOUR_PROJECT_ID.iam.gserviceaccount.com"
service_account_emails = {
  "production" = "clearpay-api-production@YOUR_PROJECT_ID.iam.gserviceaccount.com"
  "staging" = "clearpay-api-staging@YOUR_PROJECT_ID.iam.gserviceaccount.com"
}
</code></pre>
<p>Check:</p>
<pre><code class="language-bash">gcloud iam service-accounts list --filter='email~clearpay-'
</code></pre>
<p>Look for three emails: <code>clearpay-api-staging</code>, <code>clearpay-api-production</code>, and <code>clearpay-ci</code>.</p>
<p>In the console: <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">Service accounts</a> lists the three. <a href="https://console.cloud.google.com/iam-admin/workload-identity-pools">Workload Identity Federation</a> shows <code>clearpay-github-pool</code> with provider <code>clearpay-github-provider</code>.</p>
<p><strong>If apply fails</strong> with <code>Error 409: Requested entity already exists</code> on the pool, you ran this stage before. Google keeps a deleted pool for 30 days and blocks its name. Restore it with <code>gcloud iam workload-identity-pools undelete clearpay-github-pool --location=global</code>,<br>then run <code>terraform import google_iam_workload_identity_pool.github projects/YOUR_PROJECT_ID/locations/global/workloadIdentityPools/clearpay-github-pool</code> and apply again.</p>
<p>If the same error then names the provider, restore it with <code>gcloud iam workload-identity-pools providers undelete clearpay-github-provider --workload-identity-pool=clearpay-github-pool --location=global</code>,<br>then run <code>terraform import google_iam_workload_identity_pool_provider.github projects/YOUR_PROJECT_ID/locations/global/workloadIdentityPools/clearpay-github-pool/providers/clearpay-github-provider</code> and apply again.</p>
<p>Message Maya: "three service accounts exist. Nothing uses them yet."</p>
<p>Maya: "No key files?"</p>
<p>"None."</p>
<h4 id="heading-stage-6-create-the-secrets-06-secrets">Stage 6: Create the Secrets (<code>06-secrets</code>)</h4>
<p>ClearPay on GKE needs two secrets: the database URL and the JWT signing key. Both already exist on AWS. This stage creates four secrets in Secret Manager, one pair per namespace:</p>
<ul>
<li><p><code>clearpay-staging-jwt-secret</code></p>
</li>
<li><p><code>clearpay-staging-database-url</code></p>
</li>
<li><p><code>clearpay-production-jwt-secret</code></p>
</li>
<li><p><code>clearpay-production-database-url</code></p>
</li>
</ul>
<p>The staging service account can read only the staging pair. The production service account can read only the production pair.</p>
<p>Think of each secret as a locked drawer. Each badge from Stage 5 opens only its own drawers.</p>
<p>This lab puts the same values in both pairs. Both environments use the same RDS, and a login on AWS must still work when the next request lands on GKE. A real company would give each environment its own values.</p>
<p>Terraform creates the secrets with a placeholder value, <code>REPLACE_ME</code>. Terraform is allowed to create the secrets but not to invent their values. So the order is: apply first, then copy the real values in by hand.</p>
<pre><code class="language-bash">cd infra/gcp/terraform/06-secrets
</code></pre>
<p>Run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Terraform creates fourteen resources:</p>
<ul>
<li><p>four secrets</p>
</li>
<li><p>four first versions</p>
</li>
<li><p>four access rules for the API service accounts</p>
</li>
<li><p>two rules that let the pipeline's <code>clearpay-ci</code> account read the staging pair</p>
</li>
</ul>
<pre><code class="language-plaintext">Plan: 14 to add, 0 to change, 0 to destroy.
</code></pre>
<pre><code class="language-plaintext">Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
</code></pre>
<p>Now copy the AWS values in. AWS must be running.</p>
<p>Run these from the repository root, the top folder of the project, not from <code>06-secrets</code> and not from <code>infra/aws/terraform</code>. You're still in <code>06-secrets</code>, so go up four folders first:</p>
<pre><code class="language-bash">cd ../../../..
</code></pre>
<p><code>ls</code> should now show <code>api</code>, <code>infra</code>, <code>k8s</code>, and <code>scripts</code>.</p>
<p>You don't need to move into the AWS folder. <code>terraform -chdir=infra/aws/terraform</code> reads the AWS values from where you stand. Each command pipes a value straight into Secret Manager, so nothing prints on screen.</p>
<p>The database URL, once for each namespace:</p>
<pre><code class="language-bash">terraform -chdir=infra/aws/terraform output -raw database_url \
  | gcloud secrets versions add clearpay-staging-database-url --data-file=-
terraform -chdir=infra/aws/terraform output -raw database_url \
  | gcloud secrets versions add clearpay-production-database-url --data-file=-
</code></pre>
<p>A good run prints <code>Created version [2]</code> and doesn't print the URL.</p>
<p>The JWT secret, read from the <code>.env</code> file on the AWS server:</p>
<pre><code class="language-bash">INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo grep '^JWT_SECRET=' /opt/clearpay-api/.env | cut -d= -f2-" \
  | gcloud secrets versions add clearpay-staging-jwt-secret --data-file=-
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP \
  "sudo grep '^JWT_SECRET=' /opt/clearpay-api/.env | cut -d= -f2-" \
  | gcloud secrets versions add clearpay-production-jwt-secret --data-file=-
</code></pre>
<p>A later <code>terraform apply</code> in this folder won't put <code>REPLACE_ME</code> back. The Terraform file tells it to leave the value alone after the first apply.</p>
<p>Check:</p>
<pre><code class="language-bash">gcloud secrets list --filter='name~clearpay-'
</code></pre>
<p>Then prove no secret still holds the placeholder. This loop prints <code>ok</code> or <code>STILL_PLACEHOLDER</code> and never prints a secret:</p>
<pre><code class="language-bash">for s in clearpay-staging-database-url clearpay-staging-jwt-secret \
         clearpay-production-database-url clearpay-production-jwt-secret; do
  val=$(gcloud secrets versions access latest --secret="$s")
  if [ "$val" = "REPLACE_ME" ]; then echo "STILL_PLACEHOLDER $s"; else echo "ok $s"; fi
done
</code></pre>
<p>Look for four <code>ok</code> lines. If one says <code>STILL_PLACEHOLDER</code>, run its copy command again.</p>
<p>In the console: <a href="https://console.cloud.google.com/security/secret-manager">Secret Manager</a> lists the four secrets, each with version 2 as the latest. Don't click "Show secret value" on a shared screen.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/783c7aba-0a57-45bb-a629-5a76f4b942f7.png" alt="image showing list of secrets created on ur gcp console" style="display: block;" width="600" height="400" loading="lazy">

<p>Message Maya: "the four secrets hold the AWS values. Nothing was printed on screen or pasted into a chat."</p>
<h4 id="heading-stage-7-count-api-errors-07-observability">Stage 7: Count API Errors (<code>07-observability</code>)</h4>
<p>Every time something happens in the API, it writes a short note called a log line. A login writes one. A payment writes one. When something goes wrong, the note says <code>"level":"error"</code>. A failed payment writes this:</p>
<pre><code class="language-plaintext">{"level":"error","message":"transaction_failed","timestamp":"2026-09-29T10:15:02.000Z","error":"..."}
</code></pre>
<p>On GKE, Google Cloud collects every one of these notes. This stage builds two things on top of them:</p>
<ol>
<li><p><strong>A counter</strong>, named <code>clearpay-api-error-count</code>. It goes up by one each time an error note appears.</p>
</li>
<li><p><strong>An alarm</strong>, named <code>ClearPay API - elevated error rate</code>. It switches on when the counter goes up more than five times in five minutes. One error can be bad luck. Five in five minutes means something is broken.</p>
</li>
</ol>
<p>The alarm has no phone number or email attached yet. When it switches on, the Google Cloud console shows it, but nobody gets a message. You can add an email later in the console.</p>
<p>The counter stays at zero for now, because ClearPay doesn't run on GKE yet. Zero is the correct result today.</p>
<pre><code class="language-bash">cd infra/gcp/terraform/07-observability
</code></pre>
<p>Run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>Expected plan:</p>
<pre><code class="language-plaintext">Plan: 2 to add, 0 to change, 0 to destroy.
</code></pre>
<pre><code class="language-plaintext">Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

log_metric_name = "clearpay-api-error-count"
</code></pre>
<p>If apply stops with <code>Error 404: Cannot find metric(s) that match type</code>, nothing is broken. Google needs a few minutes to register a brand-new counter before an alarm can watch it. The counter was created. Wait five minutes and run <code>terraform apply</code> again. This time the plan shows <code>1 to add</code>, only the alarm.</p>
<p>Check:</p>
<pre><code class="language-bash">gcloud logging metrics describe clearpay-api-error-count
</code></pre>
<p>Look for <code>name: clearpay-api-error-count</code> and a filter that mentions <code>clearpay-api</code> and <code>jsonPayload.level="error"</code>.</p>
<p>Message Maya: "the error counter exists and reads zero. No app runs on GKE yet."</p>
<p>Maya: "Last one. This is the address customers will use, so take your time."</p>
<h4 id="heading-stage-8-build-the-public-load-balancer-08-edge">Stage 8. Build the Public Load Balancer (<code>08-edge</code>)</h4>
<p>Stages 1 to 7 built the house. This stage builds the front door: the address customers keep for the rest of the migration. It's a Google global load balancer with a traffic dial behind it: a percentage to AWS, the rest to GKE. Today it sends 100 percent to AWS.</p>
<p>This stage creates:</p>
<ul>
<li><p><code>clearpay-api-ip</code>, the public address customers will use</p>
</li>
<li><p>a self-signed certificate so HTTPS works</p>
</li>
<li><p>the traffic split, set to AWS 100, GKE 0</p>
</li>
<li><p><code>clearpay-gke-edge-ip</code>, a second IP for GKE's side. The load balancer uses it. Customers never do.</p>
</li>
</ul>
<p>The diagram below shows how the GCP load balancer directs customer traffic between AWS and GKE using configurable traffic percentages. Customers connect through one public IP. AWS receives traffic through its Elastic IP, while GKE receives traffic through a separate edge IP that customers never access directly. Both environments connect to the same PostgreSQL database on AWS. The GKE cluster is empty at this stage.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/83038470-79f5-4488-be93-4e205dfbfeed.png" alt="Architecture diagram showing customers connecting to a GCP load balancer through one public IP. The load balancer splits traffic between an AWS EC2 server, reached through its Elastic IP, and a GKE cluster, reached through a separate GCP edge IP. Both environments connect to the same PostgreSQL database on AWS. The GKE cluster is empty at this stage." style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-bash">cd infra/gcp/terraform/08-edge
cp terraform.tfvars.example terraform.tfvars
</code></pre>
<p>Skip the copy if the file exists. Put your Elastic IP in <code>aws_backend_ip</code>. Get it from a second terminal at the repository root:</p>
<pre><code class="language-bash">terraform -chdir=infra/aws/terraform output -raw instance_public_ip
</code></pre>
<p>Keep the weights at 100 and 0:</p>
<pre><code class="language-plaintext">aws_traffic_weight = 100
gke_traffic_weight = 0
</code></pre>
<p>Run:</p>
<pre><code class="language-bash">terraform init
terraform plan
terraform apply
</code></pre>
<p>The apply takes about two minutes.</p>
<pre><code class="language-plaintext">Plan: 17 to add, 0 to change, 0 to destroy.
</code></pre>
<pre><code class="language-plaintext">Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

Outputs:

current_traffic_split = "AWS: 100% / GKE: 0%"
static_ip_address = "x.x.x.x"
gke_edge_ip_address = "y.y.y.y"
hosts_file_line = "x.x.x.x api.clearpay.test"
</code></pre>
<p>Write both IPs down. <code>static_ip_address</code> is the customer address. <code>gke_edge_ip_address</code> is not.</p>
<h4 id="heading-give-the-address-a-name">Give the Address a Name</h4>
<p>You don't own a domain, so you point the name <code>api.clearpay.test</code> at the load balancer in your computer's hosts file.</p>
<p>You're already in <code>08-edge</code> after the apply, so skip the <code>cd</code> line below. Run it only if you opened a new terminal at the repository root:</p>
<pre><code class="language-bash">cd infra/gcp/terraform/08-edge
LINE=$(terraform output -raw hosts_file_line)
echo "LINE is: [$LINE]"
</code></pre>
<p>If <code>LINE</code> is empty, stop. If it looks like <code>x.x.x.x api.clearpay.test</code>, add it to the hosts file.</p>
<p>On a Mac:</p>
<pre><code class="language-bash">echo "$LINE" | sudo tee -a /etc/hosts
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder
</code></pre>
<p>On Linux, run only the first line. On Windows, add the same IP and <code>api.clearpay.test</code> to <code>C:\Windows\System32\drivers\etc\hosts</code> in an editor opened as Administrator.</p>
<p>Don't run <code>sudo cd</code>. <code>cd</code> is built into the shell, so <code>sudo</code> can't change your folder.</p>
<p>Confirm the line is there:</p>
<pre><code class="language-bash">grep api.clearpay.test /etc/hosts
</code></pre>
<p>The name ends in <code>.test</code> on purpose. <code>.test</code> is reserved for testing, so Chrome lets you click through a self-signed certificate warning. Don't change <code>lab_hostname</code> to a <code>.dev</code> name. Chrome forces strict HTTPS on every <code>.dev</code> site and hides the button to continue.</p>
<h4 id="heading-check">Check</h4>
<pre><code class="language-bash">curl -k https://api.clearpay.test/health
</code></pre>
<p>Expected output:</p>
<pre><code class="language-plaintext">{"status":"ok","served_by":{"platform":"aws","version":"1.0.0","instance":"ip-172-31-64-73","commit":"504e0cd","rollback_active":false}}
</code></pre>
<p>Your <code>instance</code> and <code>commit</code> values will differ. <code>"platform":"aws"</code> is the check. The request went to the new Google address, and the load balancer passed it to AWS, because the weights are 100 and 0.</p>
<p>A new Google load balancer can take a few minutes to start. If you don't see the JSON yet, you may see one of these first:</p>
<ul>
<li><p><code>Connection reset</code>. The load balancer is still starting. Wait a minute and run the same curl again.</p>
</li>
<li><p>A Google page titled <code>404 Not Found</code>. The load balancer answered but hasn't reached AWS yet. Confirm AWS itself is up:</p>
</li>
</ul>
<pre><code class="language-bash">curl -k https://YOUR_ELASTIC_IP/health
</code></pre>
<p>If AWS answers, wait a few minutes and try <code>api.clearpay.test</code> again. Stage 8 is done even if this is still slow. Chapter 20 checks it again.</p>
<p>When the JSON appears, open <a href="https://api.clearpay.test">https://api.clearpay.test</a> in a browser. In Chrome, click Advanced, then "Proceed to api.clearpay.test (unsafe)". You see the ClearPay login, served by AWS through the new address. If the browser says <code>NXDOMAIN</code>, the hosts line is missing or the cache wasn't flushed.</p>
<p><strong>In the console:</strong> <a href="https://console.cloud.google.com/networking/addresses/list">IP addresses</a> shows <code>clearpay-api-ip</code> and <code>clearpay-gke-edge-ip</code>. <a href="https://console.cloud.google.com/net-services/loadbalancing/list/loadBalancers">Load balancing</a> shows <code>clearpay-migration-forwarding-rule</code> with your Elastic IP as a backend.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/561ee96e-d795-4de1-a45a-d288f374c660.png" alt="image showing loadbalancer map on gcp" style="display: block;" width="600" height="400" loading="lazy">

<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/8f175baf-a8f5-498c-9a00-793da73e654d.png" alt="image showing IP addresses on gcp console" style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>Stage 1: <code>container.googleapis.com</code> is enabled</p>
</li>
<li><p>Stage 2: <code>terraform output nat_ip_address</code> prints an IP, and you wrote it down</p>
</li>
<li><p>Stage 3: <code>kubectl get nodes</code> shows at least one <code>Ready</code> node</p>
</li>
<li><p>Stage 4: the <code>clearpay</code> registry exists</p>
</li>
<li><p>Stage 5: three <code>clearpay-</code> service accounts exist</p>
</li>
<li><p>Stage 6: the loop prints four <code>ok</code> lines</p>
</li>
<li><p>Stage 7: the error metric exists</p>
</li>
<li><p>Stage 8: <code>/etc/hosts</code> has <code>api.clearpay.test</code>, and weights are 100 / 0</p>
</li>
</ul>
<p>Tell Maya: "all eight stages are verified. Weights are 100 and 0. Nothing runs on GKE yet."</p>
<p>Daniel: "The cluster is empty. Next we put ClearPay on it."</p>
<h2 id="heading-part-4-how-to-get-clearpay-onto-gke-staging">Part 4. How to Get ClearPay Onto GKE Staging</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/4a852432-a65f-4e7d-ad72-18f628051930.png" alt="4a852432-a65f-4e7d-ad72-18f628051930" style="display: block;" width="600" height="400" loading="lazy">

<p>Maya stops by to look at the new address in your browser. She reads the dashboard line and smiles.</p>
<p>Maya: "Nice work on the front door. But look, it still says <code>aws</code>. Customers can reach the new address, and every request still lands on the old server. The cluster on GKE is empty, so nobody can send $20 there yet."</p>
<p>She points at your terminal.</p>
<p>Maya: "Next step is to get ClearPay onto that cluster. One thing I care about: let the pipeline do it, not a person typing commands. A pipeline does it the same way every time, and we can see exactly what it did."</p>
<p><code>https://api.clearpay.test</code> already works in your browser, so it's easy to think ClearPay is on GKE. It's not. That address is Google's new front door, the load balancer from Stage 8. With the weights at 100 and 0, the front door sends every request to AWS. The <code>served by aws/1.0.0</code> line on the dashboard proves it. The GKE cluster itself is still empty.</p>
<pre><code class="language-plaintext">Browser → api.clearpay.test (Google load balancer) → 100% → AWS EC2 → RDS
                                                  →   0% → GKE (empty)
</code></pre>
<p>In this part the pipeline puts ClearPay on GKE staging, and the promotion gate from Chapter 6 decides whether staging is good enough. Staging is a private copy of the app inside the cluster. No customer reaches it.</p>
<h3 id="heading-chapter-12-check-the-api-image">Chapter 12. Check the API Image</h3>
<p><em>Week two, Tuesday, 9:00 a.m.</em></p>
<p>The cluster is built and empty. Before anything goes on it, Daniel pulls the API Dockerfile up on your screen.</p>
<p>Daniel: "We're about to put this on a cluster. Show me the container doesn't run as root."</p>
<p>Prove the API image is small, holds no secrets, and does not run as <code>root</code>.</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>A process that runs as <code>root</code> inside a container has more power than it needs if an attacker breaks in. Production images should run as a regular user.</p>
<h4 id="heading-steps">Steps</h4>
<p>Open <code>api/Dockerfile</code>. It has two stages:</p>
<ol>
<li><p>The first stage, <code>FROM node:22-slim AS deps</code>, installs only production packages with <code>npm install --omit=dev</code>.</p>
</li>
<li><p>The second stage starts again from <code>node:22-slim</code>. It applies Debian security updates and deletes npm and yarn, because the running API only needs <code>node src/index.js</code>. It creates <code>appuser</code> with user ID 1001 and copies in <code>node_modules</code>, <code>src</code>, and <code>db</code>. The last lines switch to <code>USER appuser</code>.</p>
</li>
</ol>
<p>Notice what the image doesn't contain: <code>DATABASE_URL</code> and <code>JWT_SECRET</code>. Those arrive at run time from Secret Manager.</p>
<p>Docker Desktop must be running. From the repository root:</p>
<pre><code class="language-bash">docker build -t clearpay-api:test ./api
docker run --rm clearpay-api:test whoami
</code></pre>
<h4 id="heading-check">Check</h4>
<p>The second command prints:</p>
<pre><code class="language-plaintext">appuser
</code></pre>
<h4 id="heading-if-it-fails">If it Fails:</h4>
<ul>
<li><p>It prints <code>root</code>: the <code>USER appuser</code> line is missing. Fix <code>api/Dockerfile</code> and run both commands again.</p>
</li>
<li><p>Docker can't connect: start Docker Desktop and retry. The cluster is fine.</p>
</li>
</ul>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><code>whoami</code> prints <code>appuser</code></li>
</ul>
<h3 id="heading-chapter-13-prepare-the-cluster-for-clearpay">Chapter 13. Prepare the Cluster for ClearPay</h3>
<p><em>Tuesday, 11:00 a.m.</em></p>
<p>The image check is done. You open GitHub, ready to run the pipeline. Daniel looks at your screen and shakes his head.</p>
<p>Daniel: "The first deploy will fail twice, for reasons that have nothing to do with the pipeline. Fix those first."</p>
<p>Before the first deploy, the cluster needs two things it doesn't have yet:</p>
<ol>
<li><p>A way for pods to read secrets from Secret Manager.</p>
</li>
<li><p>A network path to the database on AWS.</p>
</li>
</ol>
<p>Without the first, Kubernetes rejects the staging files. Without the second, the pods start but never become ready.</p>
<h4 id="heading-part-a-turn-on-the-secret-manager-add-ons">Part A. Turn on the Secret Manager Add-ons</h4>
<p>Daniel: "The pods should read the Secret Manager values from Stage 6. If they don't, write down why they still use a file."</p>
<p>The staging and production files in <code>k8s/overlays/</code> read the database URL and JWT secret from Secret Manager. They use two GKE add-ons:</p>
<ul>
<li><p>The Secret Manager CSI driver mounts secrets as files inside the pod.</p>
</li>
<li><p>Secret Sync copies them into a normal Kubernetes Secret named <code>clearpay-api-secrets</code>. The Deployment loads that Secret into the API's environment variables.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/60d4dd57-e960-441e-a86f-243c3b2b1b69.png" alt="Image showing secret manager flow" style="display: block;" width="600" height="400" loading="lazy"></li>
</ul>
<p>The pod proves who it is through Workload Identity, using the service account from Stage 5. No key file exists anywhere. The API code doesn't change. It still reads plain environment variables, as it does on AWS.</p>
<p>Turn on both add-ons. <code>gcloud</code> needs two separate commands, and each takes a few minutes:</p>
<pre><code class="language-bash">gcloud container clusters update clearpay --zone us-east4-a \
  --enable-secret-manager
gcloud container clusters update clearpay --zone us-east4-a \
  --enable-secret-sync
</code></pre>
<p>These run here, not in <code>03-gke</code>, so the cluster stage stays small and you see the moment secret access is added.</p>
<p>Leave <code>PROJECT_ID</code> as it is in <code>k8s/overlays/staging/secretproviderclass.yaml</code> and <code>patch-serviceaccount.yaml</code>. The pipeline replaces it with your real Project ID at deploy time. Don't type your ID into those files.</p>
<h4 id="heading-part-b-open-the-aws-database-to-gke">Part B. Open the AWS Database to GKE</h4>
<p>Don't describe this as "connecting GKE to the database". That phrasing suggests the database moves too, and it doesn't. This step lets GKE reach and use the real, unmoved AWS database, safely.</p>
<p>The cluster's machines have no public IP. Their outbound traffic leaves through Cloud NAT with one public address, the NAT IP from Stage 2. RDS will accept connections from exactly that address on port 5432, and the connection string already requires TLS (<code>sslmode=require</code>).</p>
<pre><code class="language-plaintext">GKE Pod  (clearpay-api)
   │
   ▼
Private node  (no public IP)
   │
   ▼
Cloud NAT  (clearpay-nat-ip)
   │
   │  public egress, one /32
   ▼
AWS security group  (additional_db_ingress_cidrs)
   │  port 5432 + SSL
   ▼
RDS PostgreSQL 16
</code></pre>
<p>This is one of three ways to connect two clouds:</p>
<table>
<thead>
<tr>
<th>Option</th>
<th>Setup</th>
<th>Ongoing</th>
<th>What it is</th>
</tr>
</thead>
<tbody><tr>
<td>Public endpoint + IP allowlist</td>
<td>A few Terraform variables</td>
<td>$0 extra (normal internet egress)</td>
<td>RDS gets a public IP. The security group allows only the Cloud NAT <code>/32</code>. TLS is required.</td>
</tr>
<tr>
<td>Cloud VPN (site-to-site)</td>
<td>Hours: gateways on both clouds</td>
<td>About $36 per month on GCP plus about $36 per month on AWS per tunnel (<a href="https://cloud.google.com/vpc/network-pricing#vpn">GCP VPN pricing</a>)</td>
<td>A private path. RDS has no public endpoint. Not in this repository.</td>
</tr>
<tr>
<td>Partner Cross-Cloud Interconnect for AWS</td>
<td>Real provisioning lead time</td>
<td>Fixed 10 or 100 Gbps tiers (<a href="https://cloud.google.com/network-connectivity/docs/interconnect/concepts/cross-cloud">Cross-Cloud Interconnect</a>)</td>
<td>A dedicated private link for heavy traffic. Not in this repository.</td>
</tr>
</tbody></table>
<p>To compare them for your own system:</p>
<pre><code class="language-plaintext">(expected monthly GB) × (price per GB on that path) + fixed monthly cost of that option
</code></pre>
<p>ClearPay sends small queries, so a VPN would cost more each month than the data is worth. This lab uses the allowlist. A busier system might choose differently.</p>
<p>Get the NAT IP:</p>
<pre><code class="language-bash">cd infra/gcp/terraform/02-network
terraform output nat_ip_address
</code></pre>
<p>Put it in <code>infra/aws/terraform/terraform.tfvars</code>:</p>
<pre><code class="language-plaintext">db_publicly_accessible      = true
additional_db_ingress_cidrs = ["&lt;the NAT IP&gt;/32"]
</code></pre>
<p>Then apply the AWS stack:</p>
<pre><code class="language-bash">cd infra/aws/terraform
terraform plan
terraform apply
</code></pre>
<p>The plan adds a new security group rule for the NAT IP and makes RDS publicly reachable. It must not remove the rule that lets the EC2 server reach RDS. Those are two separate rules in Terraform: <code>aws_security_group_rule.db_from_nat</code> for GKE and <code>aws_security_group_rule.db_from_api</code> for EC2.</p>
<h4 id="heading-check">Check</h4>
<p>AWS must still reach its database. From the repository root:</p>
<pre><code class="language-bash">export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
curl -k https://$INSTANCE_IP/health/ready
</code></pre>
<pre><code class="language-plaintext">{"status":"ready", ... "platform":"aws" ...}
</code></pre>
<p>If this prints <code>not_ready</code>, or the AWS login page says <code>internal error</code>, the database stopped accepting the EC2 server. Stop here and read the Terraform plan again. Fix AWS before you touch GKE, because AWS still serves every customer.</p>
<h4 id="heading-check-before-you-go-on">Check Before You Go On</h4>
<p>Chapter 14 pushes code and starts the pipeline. If any step above was skipped, the pipeline fails two chapters later with an error that doesn't say which step you missed. Run this check now instead. It prints <code>ok</code> or <code>MISSING</code> for each thing the pipeline needs. Run it from the repository root:</p>
<pre><code class="language-bash">export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
NAT_IP=$(terraform -chdir=infra/gcp/terraform/02-network output -raw nat_ip_address)
DB_HOST=$(terraform -chdir=infra/aws/terraform output -raw db_endpoint | cut -d: -f1)

grep -q '^db_publicly_accessible *= *true' infra/aws/terraform/terraform.tfvars \
  &amp;&amp; echo "ok   RDS public setting" \
  || echo "MISSING db_publicly_accessible = true in infra/aws/terraform/terraform.tfvars"
grep -q "$NAT_IP/32" infra/aws/terraform/terraform.tfvars \
  &amp;&amp; echo "ok   NAT IP in the allowlist" \
  || echo "MISSING $NAT_IP/32 in additional_db_ingress_cidrs"
kubectl get csidriver secrets-store-gke.csi.k8s.io &gt;/dev/null 2&gt;&amp;1 \
  &amp;&amp; echo "ok   Secret Manager driver" \
  || echo "MISSING Secret Manager driver: run the two gcloud commands in Part A"
kubectl get crd secretsyncs.secret-sync.gke.io &gt;/dev/null 2&gt;&amp;1 \
  &amp;&amp; echo "ok   Secret Sync" \
  || echo "MISSING Secret Sync: run the two gcloud commands in Part A"
curl -sk https://$INSTANCE_IP/health/ready | grep -q '"status":"ready"' \
  &amp;&amp; echo "ok   AWS still reaches its database" \
  || echo "MISSING AWS cannot reach its database: read the Terraform plan again"
</code></pre>
<p>The last check tests the real path from GKE to the database. It starts a small throwaway pod, asks the database port if it's accepting connections, and deletes the pod. The first run takes about a minute, because Kubernetes downloads the image:</p>
<pre><code class="language-bash">kubectl run dbcheck --rm -i --restart=Never --image=postgres:16 -- \
  pg_isready -h "$DB_HOST" -p 5432 -t 10 \
  &amp;&amp; echo "ok   GKE reaches the database" \
  || echo "MISSING GKE cannot reach the database"
</code></pre>
<p>You need six <code>ok</code> lines and no <code>MISSING</code>. Each <code>MISSING</code> line names the fix. Apply it, then run the check again. Don't start Chapter 14 until all six print <code>ok</code>.</p>
<p>If the last line says <code>MISSING</code> but the first two say <code>ok</code>, RDS may still be applying the change. Wait two minutes and run that command again.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>Both <code>gcloud container clusters update</code> commands finished</p>
</li>
<li><p>RDS allows the NAT IP, and <code>curl -k https://$INSTANCE_IP/health/ready</code> still prints <code>ready</code></p>
</li>
<li><p>The check above printed six <code>ok</code> lines and no <code>MISSING</code></p>
</li>
</ul>
<h3 id="heading-chapter-14-build-the-cicd-pipeline">Chapter 14. Build the CI/CD Pipeline</h3>
<p><em>Tuesday, 3:00 p.m.</em></p>
<p>The cluster is ready for its first deploy. Maya reads the team channel and adds one rule.</p>
<p>Maya: "From now on, let's not run <code>kubectl apply</code> on staging by hand. The pipeline should build the image, scan it, push it, and deploy it, the same way every time."</p>
<p>The workflow file is already written: <code>.github/workflows/deploy.yml</code>. Your job is to understand it, connect it to Google Cloud, and watch it deploy staging.</p>
<h4 id="heading-why-a-pipeline">Why a Pipeline?</h4>
<p>When a person deploys by hand, each deploy is slightly different. Someone skips the scan because it's late. Someone deploys from a laptop with uncommitted changes. When something breaks, nobody can say what ran.</p>
<p>A pipeline removes those differences. Every image comes from a commit, passes the same scan, and reaches the cluster through the same steps. The promotion gate in Chapter 15 means something because it checks an image built that way.</p>
<h4 id="heading-how-the-pipeline-works">How the Pipeline Works</h4>
<p>The workflow has two jobs:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/75a5ecc9-a8b7-4458-ba10-e7d8ea1b33bd.png" alt="Image showing cicd flow" style="display: block;" width="600" height="400" loading="lazy">

<p>Open <code>.github/workflows/deploy.yml</code> in your <a href="https://github.com/Osomudeya/migrate-api-aws-gke-devops/blob/main/.github/workflows/deploy.yml">repository</a> and keep it open alongside this diagram. The diagram maps the workflow's two jobs and their sequence. The file contains the actual GitHub Actions configuration. Follow both together as you read through the triggers, permissions, image build and scan, staging deployment, and production promotion.</p>
<p>Read the file top to bottom with this map.</p>
<p><strong>Triggers:</strong> <code>on: push: branches: [main]</code> runs Job 1 on every push to <code>main</code>. <code>workflow_dispatch</code> adds a "Run workflow" button in GitHub with one checkbox, <code>promote_to_production</code>. Job 1 runs only when that box is off (<code>if: ${{ !inputs.promote_to_production }}</code>). Job 2 runs only when it's on. A push can never reach production by itself.</p>
<p><strong>Permissions:</strong> <code>id-token: write</code> lets the job ask GitHub for a short-lived OIDC token, a signed statement that says "this run belongs to repository X". Google checks that statement against the pool from Stage 5 and hands back temporary credentials for <code>clearpay-ci</code>. No key file sits in GitHub secrets, so there's no key to leak.</p>
<p><strong>Image tag:</strong> The tag is the short commit SHA from <code>git rev-parse --short HEAD</code>. Each image points to exactly one commit. The pipeline also pushes a moving tag, <code>staging-latest</code>, but deploys use the SHA.</p>
<p><strong>Scan:</strong> Trivy is an open-source scanner that checks the packages inside an image against lists of known vulnerabilities. Each known vulnerability has an ID like <code>CVE-2026-29786</code>. The workflow sets:</p>
<ul>
<li><p><code>severity: CRITICAL,HIGH</code>, so only serious findings count</p>
</li>
<li><p><code>ignore-unfixed: true</code>, so a finding with no patch available yet doesn't block the build</p>
</li>
<li><p><code>exit-code: "1"</code>, so any remaining finding fails the job</p>
</li>
</ul>
<p>Together they mean: fail if a serious vulnerability has a fix and this image doesn't have it. Nothing gets pushed if the scan fails.</p>
<p><strong>Push:</strong> The job tags each image with the full registry path, <code>us-east4-docker.pkg.dev/&lt;project&gt;/clearpay/api:&lt;sha&gt;</code>, and pushes it. <code>clearpay-ci</code> has <code>artifactregistry.writer</code> from Stage 5, so the push works.</p>
<p><strong>Deploy to staging:</strong> The step works inside <code>k8s/overlays/staging</code>:</p>
<ol>
<li><p>It writes the two staging secrets from Secret Manager into <code>secrets.staging.env</code> on the runner. <code>umask 077</code> makes the file readable only by the runner's user. The file is gitignored and dies with the runner.</p>
</li>
<li><p><code>sed</code> replaces the <code>PROJECT_ID</code> placeholder in <code>patch-serviceaccount.yaml</code> and <code>secretproviderclass.yaml</code> with your real Project ID.</p>
</li>
<li><p><code>kustomize edit set image</code> points the overlay at the image with this commit's SHA.</p>
</li>
<li><p><code>kubectl apply -k .</code> sends the overlay to the cluster.</p>
</li>
<li><p><code>kubectl rollout status</code> waits up to 180 seconds for the API and UI pods to become ready. If they never do, the job fails.</p>
</li>
</ol>
<p><code>clearpay-ci</code> has <code>container.developer</code>, which lets it deploy workloads but not change the cluster itself.</p>
<p><strong>Promote to production:</strong> Job 2 is the only way the pipeline touches production:</p>
<ul>
<li><p><code>environment: production</code> pauses the job until a named reviewer clicks Approve in GitHub.</p>
</li>
<li><p>It doesn't build anything. It reads the image string that staging is running right now and deploys that exact image to production.</p>
</li>
</ul>
<p>That rule is called "build once, promote the same artifact". The image that passed the Chapter 15 gate is byte for byte the image customers get. A rebuild could pull a newer package and ship something nobody tested.</p>
<p>Kustomize, used in both deploy steps, is the tool behind <code>kubectl apply -k</code>. It takes the shared files in <code>k8s/base/</code> and applies small environment-specific changes from <code>k8s/overlays/staging/</code> or <code>k8s/overlays/production/</code>, called overlays. Staging and production share one base and differ only where they must.</p>
<h4 id="heading-steps">Steps</h4>
<p>First, put this repository on GitHub under your own account, with <code>main</code> as the default branch.</p>
<p>Then tell Google which repository may use <code>clearpay-ci</code>. Open <code>infra/gcp/terraform/05-identity/terraform.tfvars</code> and set <code>github_repo</code> to your repository's <code>owner/name</code>, for example <code>Osomudeya/migrate-api-aws-gke-devops</code>. Don't paste the full <code>https://github.com/...</code> address, and no <code>.git</code> at the end. Apply Stage 5 again:</p>
<pre><code class="language-bash">cd infra/gcp/terraform/05-identity
terraform plan
terraform apply
</code></pre>
<p>The plan changes the pool's attribute condition and the <code>clearpay-ci</code> link. It doesn't touch the cluster. If you already set the real name in Chapter 11, the plan shows no changes.</p>
<p>On GitHub, open your repository and click Settings. In the left menu, click Secrets and variables, then Actions. Open the Variables tab and click New repository variable for each of these three. Use the Repository section, not Environment.</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td><code>GCP_PROJECT_ID</code></td>
<td>your GCP Project ID</td>
</tr>
<tr>
<td><code>GCP_WORKLOAD_IDENTITY_PROVIDER</code></td>
<td>the output of <code>terraform -chdir=infra/gcp/terraform/05-identity output -raw github_actions_workload_identity_provider</code></td>
</tr>
<tr>
<td><code>GCP_CI_SERVICE_ACCOUNT</code></td>
<td>the output of <code>terraform -chdir=infra/gcp/terraform/05-identity output -raw ci_service_account_email</code></td>
</tr>
</tbody></table>
<p>These are variables, not secrets. None of them is a password. The pipeline needs no GitHub secrets at all.</p>
<p>When you copy the output of the two <code>terraform output -raw</code> commands, don't copy a <code>%</code> at the end. On a Mac, the terminal shows <code>%</code> when output has no line break after it. It's not part of the value. The provider value must end in <code>clearpay-github-provider</code>, and the service account must end in <code>.iam.gserviceaccount.com</code>.</p>
<p>Then, still in Settings, click Environments, then New environment. Name it <code>production</code> and click Configure environment. Tick Required reviewers, search for your own GitHub name, select it, and click Save protection rules.</p>
<p>Why: Job 2 is the only path to production. A required reviewer makes GitHub stop that job and wait until a person clicks Approve, so nothing reaches customers by accident. Job 2 waits for that approval in Chapter 19.</p>
<p>You'll see a Manage environment secrets button on this page. Don't click it. The <code>production</code> environment needs no secrets, because the pipeline gets its Google access through Workload Identity, not a stored password. If your repository already holds secrets from other projects, leave them where they are.</p>
<p>If GitHub doesn't offer <strong>Required reviewers</strong>, your repository may be private on a free plan. Make the repository public for this lab, or skip the reviewer. Without one, the production job runs as soon as you trigger it.</p>
<p>Next, check before you push. A pipeline that fails costs three minutes and gives a long log. A check costs ten seconds. First run the check from the end of Chapter 13 again. All six lines must print <code>ok</code>. Then check the repository name you gave Google:</p>
<pre><code class="language-bash">REPO=$(grep '^github_repo' infra/gcp/terraform/05-identity/terraform.tfvars | cut -d'"' -f2)
case "$REPO" in
  https*|*.git|YOUR_GITHUB_USER/*) echo "MISSING github_repo must be owner/name. It is: $REPO" ;;
  */*) echo "ok   github_repo $REPO" ;;
  *)   echo "MISSING github_repo must be owner/name. It is: $REPO" ;;
esac
</code></pre>
<p>It must print <code>ok</code> and your own <code>owner/name</code>.</p>
<p>Now, trigger the pipeline:</p>
<pre><code class="language-bash">git commit --allow-empty -m "trigger pipeline"
git push
</code></pre>
<h4 id="heading-check">Check</h4>
<p>Open the <strong>Actions</strong> tab and click the running workflow. Watch the steps of <code>build-scan-push</code> finish one after another, each with a check mark: build, <strong>Scan API image</strong>, <strong>Scan UI image</strong>, <strong>Authenticate to GCP</strong>, push, <strong>Deploy to staging</strong>.</p>
<p>If a step turns red, find it here. Each one ends by telling you to re-run the workflow:</p>
<table>
<thead>
<tr>
<th>Red step</th>
<th>Go to</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Scan API image</strong> or <strong>Scan UI image</strong></td>
<td>"If the scan fails"</td>
</tr>
<tr>
<td><strong>Authenticate to GCP</strong></td>
<td>"If Authenticate to GCP fails"</td>
</tr>
<tr>
<td><strong>Deploy to staging</strong></td>
<td>"If Deploy to staging fails"</td>
</tr>
</tbody></table>
<p>To re-run after a fix, open the <strong>Actions</strong> tab, click the failed run, and click <strong>Re-run all jobs</strong> at the top right. You don't need to push again.</p>
<p>The second job, <strong>promote-to-production</strong>, shows a gray circle with a slash. That's correct. GitHub skips it on every push. It only runs when you start it by hand in Chapter 19.</p>
<p>When <strong>Deploy to staging</strong> passes, check the cluster:</p>
<pre><code class="language-bash">kubectl -n staging get pods
</code></pre>
<p>The API and UI pods show <code>Running</code> with <code>READY 1/1</code>.</p>
<p>Staging has no public address, and the load balancer still sends every customer to AWS. Don't open a browser for staging. Chapter 15 reaches it through a private tunnel.</p>
<h4 id="heading-if-the-scan-fails">If the Scan Fails</h4>
<p>Never change <code>exit-code: "1"</code> to <code>"0"</code>. That turns the security gate off.</p>
<p>The Dockerfiles are already patched, so the scan should pass. A new security hole can turn it red later. Open the red job. Trivy prints a table. Find the <strong>Target</strong> column and match it:</p>
<table>
<thead>
<tr>
<th>Target column</th>
<th>What it means</th>
<th>What to do</th>
</tr>
</thead>
<tbody><tr>
<td>The operating system, like <code>debian 12.x</code>, with a version in the Fixed column</td>
<td>The base image is behind</td>
<td>Rebuild. If it stays red, move <code>FROM node:22-slim</code> to the newest <code>22.x-slim</code> tag on <a href="https://hub.docker.com/_/node/tags?name=slim">Docker Hub</a>.</td>
</tr>
<tr>
<td><code>usr/local/lib/node_modules/npm</code></td>
<td>npm ships the package</td>
<td>The Dockerfile deletes npm. Check that the delete line is still there.</td>
</tr>
<tr>
<td><code>app/node_modules</code> or <code>package-lock.json</code></td>
<td>One of ClearPay's packages is behind</td>
<td>Bump it in <code>api/package.json</code> or <code>ui/package.json</code></td>
</tr>
<tr>
<td>Alpine packages, in <strong>Scan UI image</strong>, with a Fixed version</td>
<td>The UI base image is behind</td>
<td>Add <code>RUN apk upgrade --no-cache</code> to the nginx stage of <code>ui/Dockerfile.prod</code></td>
</tr>
</tbody></table>
<p>After any Dockerfile change, rebuild locally and confirm the user is still <code>appuser</code>:</p>
<pre><code class="language-bash">docker build -t clearpay-api:test ./api
docker run --rm clearpay-api:test whoami
</code></pre>
<p>Commit, push to <code>main</code>, and watch Actions again.</p>
<h4 id="heading-if-authenticate-to-gcp-fails">If Authenticate to GCP Fails</h4>
<p>The error looks like this:</p>
<pre><code class="language-plaintext">The given credential is rejected by the attribute condition.
</code></pre>
<p>Google accepted the GitHub token and then rejected it. Stage 5 names a different repository, often the placeholder <code>YOUR_GITHUB_USER/YOUR_REPO</code>. The value must be exactly <code>owner/name</code>, like <code>Osomudeya/migrate-api-aws-gke-devops</code>. A full URL doesn't match, and neither does another person's repository. Repeat step 2 above with your exact <code>owner/name</code>. Then retry without changing the workflow:</p>
<pre><code class="language-bash">git commit --allow-empty -m "retry pipeline after WIF repo lock"
git push
</code></pre>
<h4 id="heading-if-deploy-to-staging-fails">If Deploy to Staging Fails</h4>
<p>Two messages mean the same thing: <code>error: timed out waiting for the condition</code> and <code>error: deployment "clearpay-api" exceeded its progress deadline</code>. The files applied, but the pods never became ready. Ask the cluster why:</p>
<pre><code class="language-bash">kubectl -n staging get pods
kubectl -n staging describe pod -l app=clearpay-api
</code></pre>
<p>Then match what you see. The first two rows appear in the job log. The rest appear in the pod output:</p>
<table>
<thead>
<tr>
<th>What you see</th>
<th>What it means</th>
<th>What to do</th>
</tr>
</thead>
<tbody><tr>
<td><code>no matches for kind "SecretProviderClass"</code> or <code>"SecretSync"</code></td>
<td>The Secret Manager add-ons are off</td>
<td>Run the two <code>gcloud</code> commands from Chapter 13, Part A. Re-run the workflow.</td>
</tr>
<tr>
<td><code>Permission denied</code> on <code>gcloud secrets versions access</code></td>
<td>The pipeline account can't read the staging secrets</td>
<td>Apply Stage 6 again (fix A below)</td>
</tr>
<tr>
<td><code>ImagePullBackOff</code> with <code>403 Forbidden</code></td>
<td>The cluster's machines can't read the registry</td>
<td>Apply Stage 4 again, then restart the pods (fix B below)</td>
</tr>
<tr>
<td><code>Insufficient cpu</code></td>
<td>The cluster is growing, or it's full</td>
<td>Run <code>kubectl get nodes</code>. Fewer than three machines: wait a few minutes. Three machines: scale staging to one API pod (fix C below).</td>
</tr>
<tr>
<td><code>FailedMount</code> with <code>driver ... not found</code>, or <code>secret "clearpay-api-secrets" not found</code></td>
<td>A new machine is still starting, or Secret Sync hasn't written the secret yet</td>
<td>Wait two minutes. The pod retries on its own. Check with <code>kubectl -n staging get secret clearpay-api-secrets</code>.</td>
</tr>
<tr>
<td><code>Running</code> but <code>READY 0/1</code>, with <code>Connection terminated due to connection timeout</code> or <code>Readiness probe failed</code></td>
<td>The pod can't reach PostgreSQL</td>
<td>Run the check at the end of Chapter 13. The <code>MISSING</code> line shows what's not done. Chapter 13, Part B is the usual cause. The pods become ready on their own at the next health check.</td>
</tr>
</tbody></table>
<p><strong>Fix A. Give the pipeline access to the staging secrets:</strong></p>
<pre><code class="language-bash">cd infra/gcp/terraform/06-secrets
terraform plan
terraform apply
</code></pre>
<p><strong>Fix B. Let the cluster read the registry.</strong> The machines pull images as the default Compute Engine service account, which needs <code>artifactregistry.reader</code>. The plan adds that one rule:</p>
<pre><code class="language-bash">cd infra/gcp/terraform/04-registry
terraform plan
terraform apply
</code></pre>
<p>Then restart the staging pods and wait:</p>
<pre><code class="language-bash">kubectl -n staging rollout restart deployment/clearpay-api deployment/clearpay-ui
kubectl -n staging rollout status deployment/clearpay-api --timeout=180s
kubectl -n staging rollout status deployment/clearpay-ui --timeout=180s
</code></pre>
<p><strong>Fix C. Run one API pod in staging:</strong> The small lab cluster can't fit two. Scale down, then re-run the workflow:</p>
<pre><code class="language-bash">kubectl -n staging scale deployment/clearpay-api --replicas=1
</code></pre>
<p>To read the API's logs without printing secrets:</p>
<pre><code class="language-bash">kubectl -n staging logs deployment/clearpay-api --tail=20
</code></pre>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>You can explain what each of the two jobs in <code>deploy.yml</code> does</p>
</li>
<li><p>The <code>production</code> environment exists with you as required reviewer</p>
</li>
<li><p><strong>Deploy to staging</strong> passed</p>
</li>
<li><p><code>kubectl -n staging get pods</code> shows the API and UI <code>Running</code>, <code>READY 1/1</code></p>
</li>
</ul>
<p>The pipeline stops there. It doesn't run the promotion gate. You do, in Chapter 15.</p>
<h3 id="heading-chapter-15-check-staging-and-run-the-promotion-gate">Chapter 15. Check Staging and Run the Promotion Gate</h3>
<p><em>Week two, Wednesday, 10:00 a.m.</em></p>
<p>The pipeline finished without an error at 6:40 last night. You took a screenshot. Nobody at ClearPay cares about the screenshot.</p>
<p>Daniel stops at your desk and looks at the passing pipeline.</p>
<p>Daniel: "So GKE is healthy?"</p>
<p>You answer: "The pods are healthy. That doesn't prove a customer can send twenty dollars."</p>
<p>Maya has read the thread. She adds a line.</p>
<p>Maya: "Customers don't get this build until the promotion script passes. Running pods aren't enough for me."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>Kubernetes reports a pod as ready when its health check answers. That says the process is up and can reach the database. It says nothing about logins, payments, or double charges. The promotion gate checks those.</p>
<p>The promotion gate is a script that you run on your laptop. The pipeline doesn't run it. It's also not the gray <strong>promote-to-production</strong> job you saw in GitHub. That job comes later, in Chapter 19, and only after this gate passes.</p>
<h4 id="heading-how-staging-runs">How Staging Runs</h4>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/36bb447a-2782-40a2-bf7b-588fb04f8dae.png" alt="Image showing the cluster architecture for gcp" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-plaintext">GKE cluster "clearpay"   zone us-east4-a
│
├── namespace staging
│     Service clearpay-api :80
│     └── API pods   (HPA 1–3)
│
└── namespace production   (created in Chapter 19)
      Gateway clearpay-gke-edge   ← clearpay-gke-edge-ip
      HTTPRoute → Service :80
      └── API pods   (HPA 1–3)
            each pod: DB_POOL_SIZE=10
</code></pre>
<p>Daniel looks at the same diagram.</p>
<p>"How many API processes can production run?"</p>
<p>"Up to three on this cluster. That's the HPA ceiling in <code>patch-hpa.yaml</code>."</p>
<p>"And how many database connections can each one open?"</p>
<p>"Ten. Same as <code>db.js</code> on Monday."</p>
<p>"So what could the database see?"</p>
<p>You stop. 30 from GKE. Plus 10 from AWS while both are live. On a cluster that could scale to 8 pods, it would be 80 from GKE alone.</p>
<p>"That's the number I asked you to remember," Daniel says.</p>
<p>Three files are worth opening now:</p>
<ul>
<li><p><code>k8s/base/deployment.yaml</code>: the health checks call <code>/health/ready</code>, and each API pod reserves <code>250m</code> CPU and <code>256Mi</code> memory.</p>
</li>
<li><p><code>k8s/base/networkpolicy.yaml</code>: limits which pods may talk to the API.</p>
</li>
<li><p><code>k8s/overlays/staging/patch-deployment.yaml</code>: staging's replica count and rollout settings.</p>
</li>
</ul>
<h4 id="heading-step-1-check-the-database-path-from-gke">Step 1. Check the Database Path from GKE</h4>
<p>Run a one-line Node script inside a staging API pod. It opens a database connection and prints <code>connected</code>:</p>
<pre><code class="language-bash">kubectl -n staging exec deployment/clearpay-api -- node -e \
  "require('./src/db').checkConnection().then(() =&gt; { console.log('connected'); process.exit(0); }).catch(e =&gt; { console.error(e); process.exit(1); })"
</code></pre>
<pre><code class="language-plaintext">connected
</code></pre>
<h4 id="heading-step-2-check-the-secrets-arrived">Step 2. Check the Secrets Arrived</h4>
<p>Kubernetes holds the synced Secret with two keys. <code>describe</code> lists key names and sizes, not values:</p>
<pre><code class="language-bash">kubectl -n staging get secret clearpay-api-secrets
kubectl -n staging describe secret clearpay-api-secrets
</code></pre>
<p>The API process has the value, checked without printing it:</p>
<pre><code class="language-bash">kubectl -n staging exec deployment/clearpay-api -- node -e 'if (!process.env.DATABASE_URL) process.exit(1); console.log("DATABASE_URL is set")'
</code></pre>
<pre><code class="language-plaintext">DATABASE_URL is set
</code></pre>
<p>The script is in single quotes on purpose. The default Mac shell, zsh, treats a <code>!</code> inside double quotes as a history command and stops with <code>zsh: event not found</code>.</p>
<p>The CSI mount shows the secret files by name:</p>
<pre><code class="language-bash">kubectl -n staging exec deployment/clearpay-api -- ls /mnt/secrets-store
</code></pre>
<p>Look for <code>DATABASE_URL</code> and <code>JWT_SECRET</code>.</p>
<h4 id="heading-step-3-run-the-promotion-gate">Step 3. Run the Promotion Gate</h4>
<p>Staging is private. <code>kubectl port-forward</code> opens a tunnel so <code>localhost:8080</code> on your laptop reaches the staging API. The <code>&amp;</code> keeps the tunnel running in the background. From the repository root:</p>
<pre><code class="language-bash">kubectl -n staging port-forward svc/clearpay-api 8080:80 &amp;
</code></pre>
<p>Wait for this line before you continue:</p>
<pre><code class="language-plaintext">Forwarding from 127.0.0.1:8080 -&gt; 4000
</code></pre>
<p>Amara is already on the thread: "Run the replay. I want to see one debit."</p>
<p>Run the same gate you ran against AWS in Chapter 6. The four arguments are the tunnel URL, Alice's email, her password, and Bob's email:</p>
<pre><code class="language-bash">./scripts/promotion-gate.sh http://localhost:8080 \
  alice@clearpay.dev password123 bob@clearpay.dev
</code></pre>
<pre><code class="language-plaintext">PASS: health endpoint returns ok
PASS: readiness endpoint confirms DB connectivity
PASS: login succeeds and returns a token
PASS: replaying an idempotency key returns replayed=true
PASS: balance debited exactly once (100 cents), not twice

5 passed, 0 failed

GATE PASSED: safe to promote to production.
</code></pre>
<p>Stop the tunnel with <code>kill %1</code>.</p>
<p>Alice exists on GKE without a new seed because GKE reads the same RDS database as AWS.</p>
<h4 id="heading-step-4-confirm-customers-are-still-on-aws">Step 4. Confirm Customers Are Still on AWS</h4>
<p>Staging has no public address:</p>
<pre><code class="language-bash">kubectl -n staging get svc clearpay-api
</code></pre>
<pre><code class="language-plaintext">NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
clearpay-api   ClusterIP   10.x.x.x       &lt;none&gt;        80/TCP    12m
</code></pre>
<p><code>ClusterIP</code> with <code>&lt;none&gt;</code> means only the cluster can reach it.</p>
<p>The customer address still answers from AWS:</p>
<pre><code class="language-bash">export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
curl -sk https://$INSTANCE_IP/health
</code></pre>
<pre><code class="language-plaintext">{"status":"ok","served_by":{"platform":"aws",...}}
</code></pre>
<p>If you see <code>gke</code> here, stop. If the curl fails, AWS is down, which is a different problem from GKE taking traffic.</p>
<p>So both clouds are running, but only AWS is open to customers. Staging runs on its own inside GKE. The gate isn't what keeps it running. It only tests it, through your tunnel, which is the only door in.</p>
<pre><code class="language-plaintext">Browser  →  api.clearpay.test  →  100 percent  →  AWS EC2  (serves everyone)

GKE staging  ←  your tunnel (port-forward)  ←  the gate and the load test
GKE production  (does not exist yet, Chapter 19)
</code></pre>
<p>Customers reach GKE only after Chapter 21, when the front door starts sending it real traffic.</p>
<h4 id="heading-if-it-fails">If it Fails</h4>
<p>Don't promote this build.</p>
<ul>
<li><p><code>Unexpected end of JSON input</code> or <code>Connection refused</code>: the script ran before the tunnel was listening. If <code>Forwarding from 127.0.0.1:8080</code> is already in the terminal, run only the script again. Don't start a second port-forward.</p>
</li>
<li><p>Health passes but ready fails: the pod can't reach PostgreSQL. Go back to Chapter 13, Part B.</p>
</li>
<li><p>Login fails: check the values you typed against the seed: <code>alice@clearpay.dev</code>, <code>password123</code>, <code>bob@clearpay.dev</code>.</p>
</li>
<li><p><code>checkConnection</code> prints a TLS or timeout error: same as ready failing. Chapter 13, Part B.</p>
</li>
<li><p>The <code>ls</code> shows no files, or the pod is stuck in <code>ContainerCreating</code>: run <code>kubectl -n staging describe pod &lt;pod-name&gt;</code> and look for a Secret Manager permission error. The annotation on the staging <code>clearpay-api</code> service account must match the staging email from Stage 5, character for character. That's the email Stage 6 allowed to read the secrets.</p>
</li>
</ul>
<p>Fix the cause, push through the pipeline if the code or files changed, and run the gate again.</p>
<h4 id="heading-decision">Decision</h4>
<p>Maya: "Are we ready?"</p>
<p>"The gate passed. Health, readiness, login, replay, and one debit. Amara's replay check passed. Customers are still on AWS."</p>
<p>Maya: "Then we can talk about production. Not before."</p>
<p>She looks at the terminal again. "Staging works. Now tell me whether we should keep sending GKE to that same Postgres over the internet."</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p><code>checkConnection</code> prints <code>connected</code> from a staging pod</p>
</li>
<li><p><code>DATABASE_URL is set</code>, and <code>/mnt/secrets-store</code> lists both secret names</p>
</li>
<li><p>The gate prints <code>5 passed, 0 failed</code> through the tunnel</p>
</li>
<li><p>Staging is <code>ClusterIP</code> with no external IP, and the Elastic IP still answers <code>aws</code></p>
</li>
</ul>
<p>This is the checkpoint. Staging passed its gate, and GKE reaches the real database.</p>
<p>Next you prove it's safe, then prove it under load, before any customer traffic moves.</p>
<h2 id="heading-part-5-how-to-make-gke-production-ready">Part 5. How to Make GKE Production-Ready</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/c6f87779-a415-4b5c-b069-465f9be13f40.png" alt="Image describing what we have done, where we are and what next." style="display: block;" width="600" height="400" loading="lazy">

<p>Staging works. The promotion gate proved five things: the app runs, it reaches the database, a user can log in, a repeated payment charges once, and the balance drops by the right amount. That's a good result. It's not the same as being safe for customers.</p>
<p>Think of a new car that passed its first test drive. It starts and it turns. Before anyone rides in it, you still ask three more questions. Are the doors locked? Can you see the dashboard? What happens on a steep hill, or when a tire goes flat?</p>
<p>This part asks the same three questions about ClearPay:</p>
<table>
<thead>
<tr>
<th>Question</th>
<th>Chapter</th>
<th>What you check</th>
</tr>
</thead>
<tbody><tr>
<td>Is anything exposed?</td>
<td>16</td>
<td>Who can read the secrets. Whether the database connection is encrypted. Whether each account has only the permissions it needs.</td>
</tr>
<tr>
<td>Can we see what's happening?</td>
<td>17 (optional)</td>
<td>A practice dashboard on your laptop that shows how fast the app answers and how many requests fail. It teaches the tools. During the traffic shift you watch with command-line checks.</td>
</tr>
<tr>
<td>What happens under pressure?</td>
<td>18</td>
<td>Many requests at once, like a busy afternoon. A pod that dies in the middle of work. A restart while customers are using the app.</td>
</tr>
</tbody></table>
<p>These questions come now, on purpose. Customers are still on AWS, so if something breaks in this part, nobody notices and nobody loses money. Fixing it here costs you an hour. Fixing it after customers arrive costs you their trust, and every fix happens while people are watching.</p>
<h3 id="heading-chapter-16-run-the-security-review">Chapter 16. Run the Security Review</h3>
<p><em>Wednesday, 2:00 p.m.</em></p>
<p>After lunch, Maya sends the next task. She has read the pipeline log.</p>
<p>Maya: "Before any customer touches GKE, I want a security check. I don't want to find a secret sitting in a file after launch."</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>Every earlier check asked "does it work?". This one asks "what's exposed?". For a payments API, a review after launch comes too late.</p>
<h4 id="heading-check-1-traffic-to-the-database-is-encrypted">Check 1. Traffic to the Database is Encrypted</h4>
<p>The database connection crosses the public internet between clouds. Confirm the live staging process requires TLS, without printing the URL:</p>
<pre><code class="language-bash">kubectl -n staging exec deployment/clearpay-api -- node -e "const u=process.env.DATABASE_URL||''; console.log(u.includes('sslmode=require') ? 'sslmode=require is set' : 'sslmode=require missing')"
</code></pre>
<pre><code class="language-plaintext">sslmode=require is set
</code></pre>
<h4 id="heading-check-2-secret-rotation-is-understood">Check 2. Secret Rotation is Understood</h4>
<p><code>api/src/routes/auth.js</code> issues login tokens that last <code>12h</code>. If you rotate <code>JWT_SECRET</code>, every token signed with the old one stops working, and users must log in again. For this lab that is acceptable. Write that sentence in your notes as the rotation policy. Don't automate rotation here.</p>
<h4 id="heading-check-3-audit-logs-exist">Check 3. Audit Logs Exist</h4>
<p>Google Cloud records who read secrets and who changed permissions, separate from the app's own logs. Confirm those audit entries exist:</p>
<pre><code class="language-bash">gcloud logging read 'protoPayload.serviceName=("secretmanager.googleapis.com" OR "iam.googleapis.com")' --limit=5 --freshness=7d --format='table(timestamp,protoPayload.methodName)'
</code></pre>
<p>Look for method names that mention <code>SecretManager</code> or <code>iam</code>. Seeing only <code>iam</code> names, such as <code>SetIAMPolicy</code>, is fine. That proves Google is recording permission changes. The timestamps will differ.</p>
<h4 id="heading-check-4-each-identity-has-only-what-it-needs">Check 4. Each Identity Has Only What it Needs</h4>
<p>Staging's account reads staging secrets. Production's account reads production secrets. The pipeline pushes images and deploys, and can't change the cluster.</p>
<pre><code class="language-bash">PROJECT_ID=$(gcloud config get-value project)
gcloud secrets get-iam-policy clearpay-staging-database-url
gcloud secrets get-iam-policy clearpay-production-database-url
gcloud projects get-iam-policy "$PROJECT_ID" --flatten='bindings[].members' --filter='bindings.members:serviceAccount:clearpay-ci@' --format='table(bindings.role)'
</code></pre>
<p>What to look for:</p>
<ul>
<li><p>The staging secret lists <code>clearpay-api-staging</code> and <code>clearpay-ci</code>. The pipeline reads the staging pair during deploys.</p>
</li>
<li><p>The production secret lists <code>clearpay-api-production</code> only.</p>
</li>
<li><p><code>clearpay-ci</code> has two project roles: <code>artifactregistry.writer</code> and <code>container.developer</code>.</p>
</li>
</ul>
<p>If a production secret lists a staging account, or the pipeline has a role like <code>owner</code> or <code>editor</code>, stop and fix it in Stage 5 or Stage 6.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li>All four checks are written in your notes with the output you saw, not "probably fine"</li>
</ul>
<h3 id="heading-chapter-17-watch-clearpay-with-prometheus-and-grafana-optional">Chapter 17. Watch ClearPay With Prometheus and Grafana (Optional)</h3>
<p><em>Thursday, 9:30 a.m.</em></p>
<p>The security review is done. Daniel reads your notes, then turns his chair toward you.</p>
<p>Daniel: "Once we start moving traffic, I want to watch a graph instead of guessing. What will you look at?"</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p><strong>Note that this chapter is optional.</strong> It doesn't touch AWS or GKE, and no later chapter needs it. Skip to Chapter 18 if you want. Do it if you want to learn how Prometheus and Grafana work.</p>
<p>The API exposes numbers at <code>/metrics</code>: request counts, latency, and database pool use. Prometheus collects those numbers every few seconds. Grafana draws them.</p>
<p>This chapter runs Prometheus and Grafana on your laptop with Docker Compose, pointed at the Chapter 1 API. It proves the dashboard works. It doesn't watch GKE or AWS. Those target files exist in <code>observability/targets/</code> as <code>.example</code> files, but this lab doesn't use them. During the traffic shift, the command-line checks in Chapters 21 and 22 are the source of truth.</p>
<p>Google Cloud has its own error counter from Stage 7. That's a Cloud Logging metric in GCP, separate from this Grafana.</p>
<pre><code class="language-plaintext">Now     localhost:4000  →  Prometheus  →  Grafana :3000
Later   AWS, GKE, load balancer  (observability/targets/*.example, optional)
</code></pre>
<h4 id="heading-steps">Steps</h4>
<p>Prometheus reads <code>host.docker.internal:4000</code>, which is the Chapter 1 API. If <code>curl http://localhost:4000/health</code> fails, start it:</p>
<pre><code class="language-bash">docker compose up -d api postgres
</code></pre>
<p>Then start the monitoring stack:</p>
<pre><code class="language-bash">cd observability
docker compose up
</code></pre>
<p>Open <a href="http://localhost:3000">http://localhost:3000</a>. Anonymous admin access is on for this laptop-only stack. Open the <strong>ClearPay - Migration Dashboard</strong>.</p>
<h4 id="heading-check">Check</h4>
<p>Send the local API a few requests:</p>
<pre><code class="language-bash">curl http://localhost:4000/health
</code></pre>
<p>The <strong>Request latency p95 by env</strong> panel shows a series named <code>local</code>. An empty panel means Prometheus can't reach port 4000.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li>The dashboard shows a <code>local</code> series</li>
</ul>
<h3 id="heading-chapter-18-load-test-and-break-staging-on-purpose">Chapter 18. Load-Test and Break Staging on Purpose</h3>
<p><em>Thursday, 1:00 p.m.</em></p>
<p>Staging passed its gate, and you feel done. Daniel puts your Chapter 6 scorecard and the gate results side by side on his screen.</p>
<p>Daniel: "The gate only tested one login. What happens when twenty requests arrive at once, or when a pod dies?"</p>
<p>Run the Chapter 6 load test against staging, put the two scorecards side by side, then kill a pod and restart the Deployment while load is running.</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>Customers will hit GKE with many requests at once, and Kubernetes will restart pods during updates and machine repairs. You want to see both happen now, with no customer involved.</p>
<p>The system under test runs on the normal node pool from Stage 3. If you ever run the load generator in the cloud, a cheap Spot VM is a good fit for it, since it can be interrupted without harm. The system you measure should never run on Spot, because an interruption would ruin the numbers.</p>
<h4 id="heading-step-1-load-test-staging">Step 1. Load-Test Staging</h4>
<p>Use two terminals, both at the repository root.</p>
<p><strong>Terminal A</strong> holds the tunnel for this whole chapter. Leave it running:</p>
<pre><code class="language-bash">kubectl -n staging port-forward svc/clearpay-api 8080:80
</code></pre>
<p>Look for <code>Forwarding from 127.0.0.1:8080</code>.</p>
<p><strong>Terminal B</strong>: first confirm you reach GKE and that Postgres answers:</p>
<pre><code class="language-bash">curl -s http://localhost:8080/health
curl -s http://localhost:8080/health/ready
</code></pre>
<p><code>health</code> must include <code>"platform":"gke"</code>. <code>ready</code> must include <code>"status":"ready"</code>.</p>
<p>Then run the load test with 20 concurrent requests for 30 seconds:</p>
<pre><code class="language-bash">./scripts/load-test.sh http://localhost:8080 \
  alice@clearpay.dev password123 20 30
</code></pre>
<p>Copy the scorecard into your notes, next to the AWS one. This lab's numbers:</p>
<pre><code class="language-plaintext">                AWS (Chapter 6)   GKE staging (this chapter)
p95 latency:    745 ms            655 ms
Error rate:     0.0 %             0.0 %
Recovery time
after pod kill: n/a               (fill in after Step 3)
</code></pre>
<p>AWS ran with 10 concurrent requests and <code>ok=372</code>. GKE ran with 20 concurrent requests, <code>ok=686</code>, <code>failed=0</code>, and 22.9 requests per second.</p>
<p>GKE's p95 is a little lower than AWS's here. That might surprise you. Most of both numbers is the trip from your laptop to the cloud, and that trip is about the same size for both. Every database query on GKE also crosses from Google's <code>us-east4</code> to AWS's <code>us-east-1</code> through Cloud NAT, but that adds only a small part on top.</p>
<p>The two runs used different concurrency, 10 and 20, so treat the numbers as close, not as a race. The line that needs to stay clean is <code>failed=0</code>. Keep both latency numbers in mind. They come back at the end of the book.</p>
<h4 id="heading-step-2-read-the-served-by-lines">Step 2. Read the <code>served by</code> Lines</h4>
<p>Open the ClearPay dashboard on <code>api.clearpay.test</code> and look at Recent activity. Expect mostly <code>aws</code> lines and one or two <code>gke</code> lines. This lab looked like this:</p>
<pre><code class="language-plaintext">-$1.00  served by aws/1.0.0        ← a send you made in the browser
-$1.00  served by gke/0.0.0-unset  ← the promotion gate from Chapter 15
-$5.00  served by aws/1.0.0        ← an older send
</code></pre>
<p>Each line names the process that wrote that transfer. Both clouds write to the same database, and the UI reads every row. Why so few <code>gke</code> lines?</p>
<ul>
<li><p>Your browser sends go to AWS, because the front door still sends 100 percent of traffic there.</p>
</li>
<li><p>Only the promotion gate wrote through GKE, with a $1.00 payment through the tunnel. The load test only reads data, so it adds no lines.</p>
</li>
</ul>
<p>A <code>gke</code> line doesn't mean the load balancer sent your page to GKE. It means a GKE pod handled that transfer.</p>
<p>The two names come from environment variables. <code>aws/1.0.0</code> comes from the EC2 <code>.env</code> file. <code>gke/0.0.0-unset</code> comes from <code>k8s/base/deployment.yaml</code>, where <code>APP_VERSION</code> is a placeholder the pipeline doesn't change. The placeholder is expected, not a crash.</p>
<p>The black bar at the top shows the process answering right now. Recent activity shows history.</p>
<h4 id="heading-step-3-kill-a-pod-and-time-the-recovery">Step 3. Kill a Pod and Time the Recovery</h4>
<p>Delete one API pod and time how long Kubernetes takes to replace it:</p>
<pre><code class="language-bash">kubectl -n staging get pods -l app=clearpay-api
POD=$(kubectl -n staging get pod -l app=clearpay-api -o jsonpath='{.items[0].metadata.name}')
echo "deleting $POD"
START=$(date +%s)
kubectl -n staging delete pod "$POD"
kubectl -n staging wait --for=condition=Ready pod -l app=clearpay-api --timeout=120s
echo "recovery $(( $(date +%s) - START )) s"
kubectl -n staging get pods -l app=clearpay-api
</code></pre>
<p><code>kubectl delete pod</code> waits until the old pod is gone, including a 5-second <code>preStop</code> pause in <code>k8s/base/deployment.yaml</code>. That pause lets in-flight requests finish before the process stops. The <code>recovery</code> line is the time from the delete until the new pod is ready. An earlier run of this lab printed <code>42 s</code>. Yours will differ. Write your number in the table above.</p>
<p>If <code>wait</code> times out, run <code>kubectl -n staging get pods</code> and <code>kubectl -n staging describe pod</code> on the new pod name.</p>
<p>The Terminal A tunnel may drop when the pod it points at is deleted. If Terminal A exits, start the port-forward again before Step 4.</p>
<h4 id="heading-step-4-restart-the-deployment-under-load">Step 4. Restart the Deployment Under Load</h4>
<p>This step needs one change to <code>scripts/load-test.sh</code>. Without it, <code>failed=</code> reads 0 even when requests die. The script only counts requests that get an answer. When the pod restarts, the tunnel drops, requests get no answer, and the script stops that worker without counting a failure.</p>
<p>Open <code>scripts/load-test.sh</code> in your editor. Replace the whole <code>hit()</code> function with this one. Two things are new: <code>2&gt;/dev/null || true</code> on the curl line, and the <code>if</code> line at the end.</p>
<pre><code class="language-bash">hit() {
  local path="$1" out code secs
  out=$(curl "${CURL_OPTS[@]}" -o /dev/null -w '%{http_code} %{time_total}' \
    -H "Authorization: Bearer $TOKEN" \
    "$BASE_URL$path" 2&gt;/dev/null || true)
  code="${out%% *}"
  secs="${out##* }"
  echo "$code" &gt;&gt; "$CODES"
  node -e "process.stdout.write(String(Math.round(Number(process.argv[1]) * 1000)) + '\n')" "$secs" &gt;&gt; "$TIMES"
  if [ "$code" = "000" ]; then sleep 0.5; fi
}
</code></pre>
<p><code>000</code> is what curl reports when it got no answer. The short pause stops a failing worker from looping thousands of times a second.</p>
<p>Now run the load test in Terminal B. It runs for 40 seconds, so the restart fits inside it:</p>
<pre><code class="language-bash">./scripts/load-test.sh http://localhost:8080 \
  alice@clearpay.dev password123 5 40
</code></pre>
<p>Before you press Enter on that command, type the restart commands in a third terminal and leave them ready. Within three seconds of starting the load test, press Enter in the third terminal:</p>
<pre><code class="language-bash">kubectl -n staging rollout restart deployment/clearpay-api
kubectl -n staging rollout status deployment/clearpay-api
kubectl -n staging get events --sort-by=.lastTimestamp | tail
</code></pre>
<p>The events show old pods stopping and new pods starting, one at a time.</p>
<p>Staging lets one pod go down before its replacement starts (<code>maxUnavailable: 1</code> and <code>maxSurge: 0</code> in <code>k8s/overlays/staging/patch-deployment.yaml</code>). The small lab cluster has no room for an extra pod. So requests fail for a short time during this restart, and <code>failed=</code> above 0 is normal. Write down the count. Terminal A may print <code>lost connection to pod</code>. Stop it with Ctrl+C and start the tunnel again after the test.</p>
<p>Production is safer. Its base setting starts a new pod before an old one stops (<code>maxSurge: 1</code> and <code>maxUnavailable: 0</code>).</p>
<h4 id="heading-decision">Decision</h4>
<p>Pass: <code>failed=0</code> on the 30-second test, the replacement pod became ready after the delete, and <code>rollout status</code> finished. GKE's p95 being higher than AWS is still a pass on this path, as long as it stays under 3 times your AWS p95, the failure limit you wrote in Chapter 6.</p>
<p>Fail: 5xx errors in the 30-second test, a login <code>internal_error</code>, or a new pod that never becomes ready. Fix staging, push through the pipeline, and run this chapter again.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>Both scorecards are in your notes side by side</p>
</li>
<li><p>Pod recovery is timed</p>
</li>
<li><p>The rolling restart finished while load was running, and you wrote down <code>failed=</code></p>
</li>
</ul>
<p>Stop the port-forward in Terminal A with Ctrl+C.</p>
<p>This is the checkpoint. GKE survived load and a pod kill. Every check so far happened with no customer in the path.</p>
<p>That changes now.</p>
<h2 id="heading-part-6-how-to-move-customers-to-gke">Part 6. How to Move Customers to GKE</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ca5b0eba-5b2e-4999-bf7f-67c80bba426f.png" alt="Image describing what we have done, where we are and what next." style="display: block;" width="600" height="400" loading="lazy">

<p>This is the part where customers can land on GKE. Both clouds can serve a request. Every step from here asks one question: what evidence lets you move more traffic?</p>
<h3 id="heading-chapter-19-deploy-production-and-the-gke-gateway">Chapter 19. Deploy Production and the GKE Gateway</h3>
<p><em>Week three, Monday, 9:00 a.m.</em></p>
<p>This is the week customers can land on GKE. Nobody says it out loud. The office is a little quieter than usual.</p>
<p>So far, GKE only holds staging. Staging is a private copy that you reach through a tunnel. Customers will use a separate copy called production. This chapter builds that copy, and it connects the address customers will call.</p>
<p>Two things are missing:</p>
<ol>
<li><p><strong>The production namespace.</strong> A namespace is a separate area inside the cluster. Staging has one. Production doesn't have one yet, so no production pods exist. You will create it and put the same image in it that passed the staging gate.</p>
</li>
<li><p><strong>Something behind the GKE edge IP.</strong> In Stage 8 you reserved a public address for GKE, called the edge IP. It works like a phone number you reserved but never connected: it rings, and nobody picks up. You will connect it to a Gateway. The Gateway answers each request and passes it to the production pods.</p>
</li>
</ol>
<p>When you finish, GKE can answer real requests. You still don't move any traffic. The front door keeps sending 100 percent of customers to AWS until Chapter 21.</p>
<h4 id="heading-why-this-matters">Why This Matters</h4>
<p>The load balancer can only send traffic to GKE if something on GKE answers. That something is a Gateway: a Kubernetes object that tells Google to build a load balancer in front of a Service. Here it listens on <code>clearpay-gke-edge-ip</code> and forwards to the production API and UI.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/34561bea-1e1e-4c3a-af90-0760e25ce0d3.png" alt="image showing the cloud architecture flow diagram" style="display: block;" width="600" height="400" loading="lazy">

<p>The <code>08-edge</code> load balancer reaches both clouds through Internet NEGs (network endpoint groups that point at an outside IP address). This type has no health checks. If a backend goes down, the load balancer keeps sending it traffic, and customers get 502 errors. Your protection is <code>synthetic-health.sh</code>, the decision gates, and <code>rollback.sh</code>.</p>
<h4 id="heading-step-1-turn-on-the-gateway-api">Step 1. Turn on the Gateway API</h4>
<p>The cluster from Stage 3 doesn't have it on:</p>
<pre><code class="language-bash">gcloud container clusters update clearpay --zone us-east4-a \
  --gateway-api=standard
</code></pre>
<p>Wait until it finishes, then:</p>
<pre><code class="language-bash">kubectl get gatewayclass
</code></pre>
<p>Look for <code>gke-l7-global-external-managed</code>. That's the class <code>k8s/gateway/gke-gateway.yaml</code> uses.</p>
<h4 id="heading-step-2-make-room-on-the-node-pool">Step 2. Make Room on the Node Pool</h4>
<p>The three <code>e2-medium</code> machines have about 940m of CPU each after GKE's own system pods. Staging and production together don't fit. Scale staging down to make room. Its images stay in Artifact Registry, and the gate result stays in your notes.</p>
<p>First write down the exact images staging runs. Production must run these same strings:</p>
<pre><code class="language-bash">kubectl -n staging get deploy clearpay-api \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n staging get deploy clearpay-ui \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
</code></pre>
<p>Then scale staging to zero:</p>
<pre><code class="language-bash">kubectl -n staging scale deploy/clearpay-api --replicas=0
kubectl -n staging scale deploy/clearpay-ui --replicas=0
</code></pre>
<p>Don't push to <code>main</code> during the traffic shift. A push redeploys staging at full size and takes the CPU back.</p>
<p>If production API pods stay <code>Pending</code> with <code>Insufficient cpu</code> later, staging is still holding CPU. Don't raise the node pool's maximum unless you accept the cost of another machine.</p>
<h4 id="heading-step-3-deploy-production">Step 3. Deploy Production</h4>
<p>Use the pipeline. It's the path Chapter 14 built for this moment.</p>
<p><strong>With the pipeline:</strong></p>
<ol>
<li><p>On GitHub, open Actions, then Build, scan, and deploy in the left list. Start from this workflow page, not from a finished run. If you're looking at a single run, click the back arrow next to its name. The <code>promote-to-production</code> job on that old run shows "This job was skipped", and it stays skipped. You need a new run.</p>
</li>
<li><p>Click Run workflow. Keep the branch on <code>main</code>, tick promote_to_production, and click Run workflow.</p>
</li>
<li><p>The <code>promote-to-production</code> job waits with "Waiting for review". Click Review deployments, tick <code>production</code>, and click Approve and deploy.</p>
</li>
<li><p>Watch Find the currently-deployed staging images and Deploy those exact images to production finish with a check mark.</p>
</li>
</ol>
<p>The job reads the image strings from the staging Deployment, even though staging now runs zero pods, and deploys those exact images.</p>
<p>By hand, if the pipeline isn't set up:</p>
<p>These are the same steps the job runs. <code>PROJECT_ID</code> in the files is a placeholder, so replace it the way the pipeline does. On a Mac, <code>sed -i</code> needs an empty <code>''</code> after it:</p>
<pre><code class="language-bash">cd k8s/overlays/production
PROJECT_ID=$(gcloud config get-value project)
sed -i '' "s/@PROJECT_ID.iam/@${PROJECT_ID}.iam/" patch-serviceaccount.yaml
sed -i '' "s#projects/PROJECT_ID/#projects/${PROJECT_ID}/#" secretproviderclass.yaml
</code></pre>
<p>On Linux, drop the <code>''</code> after <code>-i</code>.</p>
<p>Point the overlay at the two images you wrote down in Step 2. Never use <code>latest</code>:</p>
<pre><code class="language-bash">kustomize edit set image \
  REGION-docker.pkg.dev/PROJECT_ID/clearpay/api=STAGING_API_IMAGE
kustomize edit set image \
  REGION-docker.pkg.dev/PROJECT_ID/clearpay/ui=STAGING_UI_IMAGE
kubectl apply -k .
kubectl -n production rollout status deployment/clearpay-api --timeout=180s
kubectl -n production rollout status deployment/clearpay-ui --timeout=180s
</code></pre>
<p>Replace <code>STAGING_API_IMAGE</code> and <code>STAGING_UI_IMAGE</code> with the full strings from Step 2, in the form <code>us-east4-docker.pkg.dev/.../api:&lt;tag&gt;</code>.</p>
<p>Don't commit the files after <code>sed</code> and <code>kustomize edit</code>. They now contain your Project ID and a real tag. Run <code>git checkout k8s/overlays/production</code> from the repository root to undo them after the deploy.</p>
<h4 id="heading-check-production">Check Production</h4>
<p>Return to the repository root. Wait until the pods are ready:</p>
<pre><code class="language-bash">kubectl -n production get pods
</code></pre>
<p>API and UI pods show <code>Running</code> and <code>READY 1/1</code>. Then check the database from a production pod, not staging:</p>
<pre><code class="language-bash">kubectl -n production exec deployment/clearpay-api -- node -e \
  "require('./src/db').checkConnection().then(() =&gt; { console.log('connected'); process.exit(0); }).catch(e =&gt; { console.error(e); process.exit(1); })"
</code></pre>
<pre><code class="language-plaintext">connected
</code></pre>
<h4 id="heading-step-4-apply-the-gke-gateway">Step 4. Apply the GKE Gateway</h4>
<p>Stage 8 reserved <code>clearpay-gke-edge-ip</code> and the certificate <code>clearpay-gke-edge-cert</code>. Now give them something to serve:</p>
<pre><code class="language-bash">kubectl apply -f k8s/gateway/gke-gateway.yaml
kubectl apply -f k8s/gateway/httproute-production.yaml
kubectl apply -f k8s/gateway/healthcheck-api.yaml
kubectl -n production get gateway clearpay-gke-edge
</code></pre>
<p>Wait until <code>PROGRAMMED</code> shows <code>True</code>. That can take a few minutes while Google builds the load balancer.</p>
<p>The three files do three jobs:</p>
<ul>
<li><p><code>gke-gateway.yaml</code> creates the Gateway on the reserved IP with the certificate.</p>
</li>
<li><p><code>httproute-production.yaml</code> sends requests to the production Service.</p>
</li>
<li><p><code>healthcheck-api.yaml</code> tells Google to health-check <code>/health</code>. Without it, Google checks <code>/</code>, the API answers 404, and requests fail with <code>no healthy upstream</code>.</p>
</li>
</ul>
<h4 id="heading-check-the-gateway">Check the Gateway</h4>
<p>Call GKE directly through its edge IP. <code>--resolve</code> makes curl send the hostname on the certificate while connecting to the GKE IP:</p>
<pre><code class="language-bash">HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
EDGE=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw gke_edge_ip_address)
curl -k --resolve "$HOST:443:$EDGE" "https://$HOST/health"
</code></pre>
<p>Look for <code>"platform":"gke"</code>. If curl prints <code>no healthy upstream</code>, wait a minute and retry. The health check needs time to mark the pods healthy.</p>
<p>Without <code>--resolve</code>, the same hostname still goes to AWS, because weights are 100 / 0.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p><code>kubectl get gatewayclass</code> lists <code>gke-l7-global-external-managed</code></p>
</li>
<li><p>Production API and UI pods are <code>Running</code>, <code>READY 1/1</code>, with the staging image tags</p>
</li>
<li><p><code>checkConnection</code> prints <code>connected</code> from a production pod</p>
</li>
<li><p>The Gateway shows <code>PROGRAMMED</code> <code>True</code>, and the edge IP returns <code>"platform":"gke"</code></p>
</li>
</ul>
<h3 id="heading-chapter-20-walk-the-pre-cutover-checklist">Chapter 20. Walk the Pre-Cutover Checklist</h3>
<p><em>Monday, 3:00 p.m.</em></p>
<p>Production is up. Maya stops at your desk with her laptop under her arm.</p>
<p>Maya: "Before we move traffic, tell me exactly how we put customers back on AWS. Then show me it works."</p>
<h4 id="heading-why-the-rollback-drill-matters">Why the Rollback Drill Matters</h4>
<p><code>scripts/rollback.sh</code> does two things, in order:</p>
<ol>
<li><p>It runs <code>shift-traffic.sh 100 0</code>, which sends all traffic back to AWS in seconds.</p>
</li>
<li><p>It connects to the AWS server over SSH, sets <code>ROLLBACK_ACTIVE=true</code> in its <code>.env</code>, and restarts the API. The UI then shows the rollback banner.</p>
</li>
</ol>
<p>The dashboard has two different messages, on purpose:</p>
<ul>
<li><p><strong>The new-version message</strong> is a small pop-up in the bottom right corner. It says "Noticed a change: this request ran on gke/... (previously aws/...)" and fades after five seconds. It appears when a different version answers.</p>
</li>
<li><p><strong>The rollback banner</strong> is a red-tinted strip across the top of the dashboard. It says "Rolled back to AWS, traffic has been restored while the GKE issue is investigated." It stays until you clear the flag.</p>
</li>
</ul>
<p>Anyone looking at the app can tell "a new version shipped" apart from "something broke and we recovered".</p>
<p>The first time you run a rollback shouldn't be during a real incident, with customers failing, hoping the SSH key works. Run it now while nothing is wrong. Weights are already 100 / 0, so the traffic step changes nothing. The SSH step and the banner are what you test.</p>
<h4 id="heading-the-checklist">The Checklist</h4>
<p>Tick each line with a command you ran, not from memory. Use your own Elastic IP and hostname.</p>
<p>Only four lines need a command now: <strong>GKE healthy</strong>, <strong>Rollback tested</strong>, <strong>AWS healthy</strong>, and <strong>Synthetic test running</strong>. The other five use results you already have, so read them and tick them. Chapters 21 and 22 don't repeat this checklist. They reuse only the synthetic test, which you leave running in its own terminal.</p>
<pre><code class="language-plaintext">Pre-Cutover Checklist

[ ] GKE healthy
    kubectl -n production get pods
    (all Running, READY)

[ ] Database healthy
    the production checkConnection from Chapter 19 prints connected

[ ] Monitoring up (optional, only if you did Chapter 17)
    Grafana at http://localhost:3000 loads.
    It does not watch GKE. Do not block on an empty GKE series.

[ ] Rollback tested
    YOUR_ELASTIC_IP is your AWS server's public IP. Get it with:
    terraform -chdir=infra/aws/terraform output -raw instance_public_ip
    ./scripts/rollback.sh YOUR_ELASTIC_IP ~/.ssh/clearpay-deployer
    Open https://YOUR_ELASTIC_IP, log in, and look for the red strip
    at the top of the dashboard ("Rolled back to AWS ..."). Then clear it:
    ./scripts/clear-rollback-flag.sh YOUR_ELASTIC_IP ~/.ssh/clearpay-deployer

[ ] AWS healthy
    ./scripts/promotion-gate.sh https://YOUR_ELASTIC_IP \
      alice@clearpay.dev password123 bob@clearpay.dev
    5 passed, 0 failed

[ ] Synthetic test running
    ./scripts/synthetic-health.sh https://YOUR_PUBLIC_HOSTNAME
    Leave it running in its own terminal for the whole shift.
    YOUR_PUBLIC_HOSTNAME is api.clearpay.test. From the repository root, confirm with:
    terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname
    It prints one line per second: time, status code, latency.

[ ] Limits confirmed
    Chapter 6 limits, plus the Chapter 18 table:
    AWS p95 745 ms, GKE staging p95 655 ms, both 0.0 % errors, plus your pod recovery time from Chapter 18

[ ] Security review passed
    Chapter 16, all four checks written down

[ ] Cost ledger current
    infra/aws/COST.md and infra/gcp/COST.md, both clouds now billing
</code></pre>
<h4 id="heading-what-the-rollback-drill-prints">What the Rollback Drill Prints</h4>
<p>The traffic weights are already 100 / 0, so step 1 changes nothing. Step 2 is the real test. It ends like this:</p>
<pre><code class="language-plaintext">== Traffic split updated ==
"AWS: 100% / GKE: 0%"

== Step 2: flagging AWS as the active rollback target ==
AWS ROLLBACK_ACTIVE=true, service restarted.

== Rollback complete ==
</code></pre>
<p>The last lines of the script name old chapter numbers. Ignore them. The steps for a real rollback are in Chapter 23.</p>
<p>If step 2 stops with <code>ssh: connect to host ... port 22: Operation timed out</code>, your home IP has changed and the AWS firewall still allows the old one. Get your IP with <code>curl -s ifconfig.me; echo</code>. Put it plus <code>/32</code> in <code>ssh_allowed_cidr</code> in <code>infra/aws/terraform/terraform.tfvars</code>. Run <code>terraform apply</code> in <code>infra/aws/terraform</code>, then run the script again.</p>
<h4 id="heading-decision">Decision</h4>
<p>Every line ticked, and the GKE edge IP returns <code>"platform":"gke"</code>: go to Chapter 21.</p>
<p>Anything unticked: stop here. Don't run <code>./scripts/shift-traffic.sh 99 1</code> while the GKE edge IP is empty. One percent of customers would hit an address with nothing behind it.</p>
<p>Tell Maya: "Rollback is one script. It moves traffic in seconds and shows a banner. I ran it five minutes ago."</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li>Every checklist line is ticked with output you saw</li>
</ul>
<h3 id="heading-chapter-21-send-1-percent-of-traffic-to-gke">Chapter 21. Send 1 Percent of Traffic to GKE</h3>
<p><em>Tuesday, 10:00 a.m.</em></p>
<p>The whole team is on one call. Chris has the support inbox open on his second screen. Amara has the transactions table open on hers. Daniel sits next to you with the load balancer page open.</p>
<p>Maya: "One percent to GKE. Watch the errors and the transfers. If they look wrong, send everything back."</p>
<h4 id="heading-why-1-percent">Why 1 Percent?</h4>
<p>One percent of requests. A request is one message from a browser to the API: a login, a balance check, a payment. Out of every 100 requests, about 99 go to AWS and about 1 goes to GKE. The load balancer picks for each request, not for each customer. So one customer can land on both clouds in a single visit.</p>
<p>One percent is enough to find a broken contract, such as a different JSON shape, a missing route, or a login that fails on one side. It's far too small to test capacity. Chapter 22 handles capacity.</p>
<h4 id="heading-steps">Steps</h4>
<pre><code class="language-bash">./scripts/shift-traffic.sh 99 1
</code></pre>
<p>The script re-applies <code>08-edge</code> with new weights on the load balancer's URL map. There's no DNS change and no deploy.</p>
<p>Now copy the same weights into <code>infra/gcp/terraform/08-edge/terraform.tfvars</code>:</p>
<pre><code class="language-plaintext">aws_traffic_weight = 99
gke_traffic_weight = 1
</code></pre>
<p>The script passes the weights as command-line values. The next plain <code>terraform apply</code> in <code>08-edge</code> reads the file. If the file still says 100 and 0, that apply would quietly move traffic back.</p>
<p>Confirm the file and the live state match:</p>
<pre><code class="language-bash">grep traffic_weight infra/gcp/terraform/08-edge/terraform.tfvars
terraform -chdir=infra/gcp/terraform/08-edge output current_traffic_split
</code></pre>
<p>You should see <code>aws_traffic_weight = 99</code>, <code>gke_traffic_weight = 1</code>, and <code>"AWS: 99% / GKE: 1%"</code>. If the file still says 100 and 0, fix it now.</p>
<h4 id="heading-check-did-any-request-reach-gke">Check: Did Any Request Reach GKE?</h4>
<p>At 1 percent, almost every request still goes to AWS. Forty requests in a row that all say <code>aws</code> is normal. Loop until one says <code>gke</code>:</p>
<pre><code class="language-bash">HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)
while true; do
  curl -sk --resolve "$HOST:443:$IP" "https://$HOST/health"
  echo
  sleep 0.2
done
</code></pre>
<p>Stop with Ctrl+C when you see <code>"platform":"gke"</code>. One run in this lab took 200 requests: 197 <code>aws</code> and 3 <code>gke</code>.</p>
<p>To confirm GKE is still answering without waiting on chance, call its edge IP directly. This isn't a customer request:</p>
<pre><code class="language-bash">HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
EDGE=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw gke_edge_ip_address)
curl -sk --resolve "$HOST:443:$EDGE" "https://$HOST/health"
</code></pre>
<p>Look for <code>"status":"ok"</code> and <code>"platform":"gke"</code>. This lab printed:</p>
<pre><code class="language-plaintext">{"status":"ok","served_by":{"platform":"gke","version":"0.0.0-unset","instance":"clearpay-api-6f9448c857-mj2j6","commit":"unset","rollback_active":false}}
</code></pre>
<p>Your <code>instance</code> will differ. If <code>platform</code> says <code>aws</code>, or the curl fails, GKE isn't answering. Don't raise the weight.</p>
<h4 id="heading-check-does-a-login-work-through-the-public-address">Check: Does a Login Work Through the Public Address?</h4>
<p>From now on, one customer can land on AWS for one request and on GKE for the next. Both must accept the same request and give the same reply. If they differ, you want to find out now, at 1 percent, not later at 50 percent. Send a login through the public address:</p>
<pre><code class="language-bash">HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)
curl -sk --resolve "$HOST:443:$IP" \
  -H 'Content-Type: application/json' \
  -d '{"email":"alice@clearpay.dev","password":"password123"}' \
  "https://$HOST/api/auth/login"
</code></pre>
<p>Look for <code>"token"</code> and <code>"user"</code> in the reply. If you see <code>internal_error</code>, stop and don't raise the weight.</p>
<h4 id="heading-check-use-the-app-like-a-customer">Check: Use the App Like a Customer</h4>
<p>Open <a href="https://api.clearpay.test">https://api.clearpay.test</a> and click through the certificate warning. Log in, check the balance, and send a small amount. Each new line under Recent activity says <code>served by aws/1.0.0</code> or <code>served by gke/0.0.0-unset</code>. Only the newest line tells you about this request. Older lines keep the stamp from when they were written.</p>
<p>The black bar at the top will say AWS almost every time at 1 percent. Watch new activity lines and the loop, not the bar.</p>
<p>Glance at the <code>synthetic-health.sh</code> terminal. Every line should show status <code>200</code>.</p>
<h4 id="heading-decision">Decision</h4>
<ul>
<li><p>The loop showed <code>gke</code> at least once, login returns a token, and synthetic health shows only <code>200</code>: go to Chapter 22.</p>
</li>
<li><p>Anything else: stop and investigate. Don't raise the weight. If customers are affected, run <code>rollback.sh</code> and go to Chapter 23.</p>
</li>
</ul>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>Weights are 99 / 1 in both the file and <code>current_traffic_split</code></p>
</li>
<li><p>The loop printed <code>"platform":"gke"</code> at least once</p>
</li>
<li><p>Login through the public address returns a token</p>
</li>
</ul>
<h3 id="heading-chapter-22-raise-traffic-to-100-percent">Chapter 22. Raise Traffic to 100 Percent</h3>
<p><em>Tuesday, 11:30 a.m., to Wednesday afternoon</em></p>
<p>One percent worked. That means 1 out of every 100 requests went to GKE. Now you turn the dial up, a little at a time. Nothing does it for you. You raise it by hand, and you look at the numbers before every step.</p>
<p>Check the same three things each time:</p>
<ul>
<li><p><strong>The scorecard:</strong> how fast the app answers, and how many requests fail.</p>
</li>
<li><p><strong>The replica count:</strong> how many copies of the API are running.</p>
</li>
<li><p><strong>The connection math:</strong> whether the database still has room for every connection.</p>
</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/66432691-1878-4296-b6b4-634dab51bb3e.png" alt="Image showing the traffic migration split" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-plaintext">99 / 1     AWS ████████████████████░    GKE ░
90 / 10    AWS ██████████████████░░     GKE ██
75 / 25    AWS ███████████████░░░░░     GKE █████
50 / 50    AWS ██████████░░░░░░░░░░     GKE ██████████
25 / 75    AWS █████░░░░░░░░░░░░░░░     GKE ███████████████
10 / 90    AWS ██░░░░░░░░░░░░░░░░░░     GKE ██████████████████
 0 / 100   AWS                          GKE ████████████████████
</code></pre>
<h4 id="heading-before-50-50-do-the-connection-math">Before 50 / 50: Do the Connection Math</h4>
<p>Daniel, who is on the call, cuts in before you go to the halfway mark. "How many database connections can hit RDS if GKE scales all the way up while AWS is still live?"</p>
<p>Each API process can open up to <code>DB_POOL_SIZE</code> connections, ten by default. The pool opens them as load needs them, so the number stays low when traffic is light. At full load, every process can hold its full ten. The database has a hard limit, <code>max_connections</code>. When the total passes it, new connections are refused, requests wait, and customers see timeouts.</p>
<p>Get your database's limit. This runs <code>psql</code> on the AWS server, which already has it from Chapter 5:</p>
<pre><code class="language-bash">export INSTANCE_IP=$(terraform -chdir=infra/aws/terraform output -raw instance_public_ip)
DATABASE_URL=$(terraform -chdir=infra/aws/terraform output -raw database_url)
ssh -i ~/.ssh/clearpay-deployer ubuntu@$INSTANCE_IP "psql '$DATABASE_URL' -c 'SHOW max_connections;'"
</code></pre>
<p>Get GKE's ceiling from <code>maxReplicas</code> in <code>k8s/overlays/production/patch-hpa.yaml</code>. Then fill in this table with your numbers:</p>
<pre><code class="language-plaintext">RDS max_connections:          ___
AWS, 1 process × 10:          10
GKE, maxReplicas × 10:        ___
Worst case, both at once:     10 + ___ = ___
</code></pre>
<p>This lab's filled-in table, so you can check yours:</p>
<pre><code class="language-plaintext">RDS max_connections:          81
AWS, 1 process × 10:          10
GKE, maxReplicas × 10:        30    (3 pods × 10)
Worst case, both at once:     10 + 30 = 40
</code></pre>
<p>40 is under 81, so there's room to raise the traffic. Your <code>max_connections</code> may differ if your RDS instance is a different size. What matters is that your worst case stays below it.</p>
<p>Now run the same math for a bigger setup. A company with larger machines might run the HPA from 3 to 8 pods: 10 + 80 = 90 connections, against a small RDS instance that allows about 87.</p>
<p>Neither setting is wrong on its own. The pool size was set in the API code, and the pod ceiling was set in a Kubernetes file, by different people at different times. They only collide when both clouds run at full load at the same time, which is the exact situation a gradual migration creates.</p>
<p>If your worst case is above your <code>max_connections</code>, stop and read the fix options in Chapter 23 before you raise the weight.</p>
<h4 id="heading-steps">Steps</h4>
<p>Every stage has the same four moves. Do all four, then repeat with the next pair of numbers.</p>
<ol>
<li><p><strong>Shift:</strong> Run <code>./scripts/shift-traffic.sh</code> with the new pair. The first number is the share for AWS, and the second is the share for GKE.</p>
</li>
<li><p><strong>Save:</strong> Put the same two numbers in <code>terraform.tfvars</code> in <code>08-edge</code>.</p>
</li>
<li><p><strong>Check:</strong> Run the six checks in the next section. No single script does all of them. You use a few commands, plus two scripts: <code>synthetic-health.sh</code>, which is already running, and <code>load-test.sh</code>.</p>
</li>
<li><p><strong>Decide:</strong> Every check passes: go to the next stage. Any check fails: stop.</p>
</li>
</ol>
<p>The stages, in order:</p>
<table>
<thead>
<tr>
<th>Stage</th>
<th>Command</th>
<th>AWS gets</th>
<th>GKE gets</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td><code>./scripts/shift-traffic.sh 90 10</code></td>
<td>90 %</td>
<td>10 %</td>
</tr>
<tr>
<td>2</td>
<td><code>./scripts/shift-traffic.sh 75 25</code></td>
<td>75 %</td>
<td>25 %</td>
</tr>
<tr>
<td>3</td>
<td><code>./scripts/shift-traffic.sh 50 50</code></td>
<td>50 %</td>
<td>50 %</td>
</tr>
<tr>
<td>4</td>
<td><code>./scripts/shift-traffic.sh 25 75</code></td>
<td>25 %</td>
<td>75 %</td>
</tr>
<tr>
<td>5</td>
<td><code>./scripts/shift-traffic.sh 10 90</code></td>
<td>10 %</td>
<td>90 %</td>
</tr>
<tr>
<td>6</td>
<td><code>./scripts/shift-traffic.sh 0 100</code></td>
<td>0 %</td>
<td>100 %</td>
</tr>
</tbody></table>
<p>Run one row at a time, and never run them all together.</p>
<h4 id="heading-checks-at-every-stage">Checks at Every Stage</h4>
<p><strong>1. The dial moved.</strong> The script's printout is Terraform's setting, not proof. Confirm the file matches:</p>
<pre><code class="language-bash">grep traffic_weight infra/gcp/terraform/08-edge/terraform.tfvars
terraform -chdir=infra/gcp/terraform/08-edge output current_traffic_split
</code></pre>
<p><strong>2. Real requests follow the dial.</strong> Wait three minutes after the shift, then count who answers:</p>
<pre><code class="language-bash">HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
IP=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw static_ip_address)

for i in $(seq 1 40); do
  curl -sk --resolve "$HOST:443:$IP" "https://$HOST/health"
  echo
done | tee /tmp/clearpay-health.txt

grep -c '"platform":"aws"' /tmp/clearpay-health.txt
grep -c '"platform":"gke"' /tmp/clearpay-health.txt
</code></pre>
<p>The GKE count should rise as its weight rises. Forty requests are a rough sample, not a percentage proof. Google also takes a few minutes to apply a new split everywhere, so counting too early shows the old split. This lab saw:</p>
<pre><code class="language-plaintext">90 / 10   80 requests:  75 aws,  5 gke
50 / 50   40 requests:  34 aws,  6 gke
25 / 75   40 requests:  17 aws, 23 gke
10 / 90   40 requests:   8 aws, 31 gke
 0 / 100  40 requests, 1 minute later:    3 aws, 37 gke
 0 / 100  40 requests, 90 seconds later:  0 aws, 40 gke
</code></pre>
<p>If the counts still don't follow the dial after ten minutes, ask Google which split it holds:</p>
<pre><code class="language-bash">gcloud compute url-maps describe clearpay-migration-map | grep -B2 -A2 weight
</code></pre>
<p>At 0 / 100, the <code>clearpay-gke-backend</code> line shows <code>weight: 100</code>, and the AWS line shows no weight at all, because Google leaves out a zero. If the weights differ from your file, run <code>terraform apply</code> again in <code>08-edge</code>.</p>
<p><strong>3. GKE is healthy.</strong></p>
<pre><code class="language-bash">kubectl -n production get pods -l app=clearpay-api
kubectl -n production get hpa clearpay-api
</code></pre>
<p>Pods are <code>Running</code> and ready. The HPA line shows current CPU against the 70 percent target, and the current replica count. This lab stayed at 1 replica and 4 to 7 percent CPU through 25 / 75.</p>
<p><strong>4. Logins still work.</strong> Run the Chapter 21 login curl again at every stage. A contract problem can appear only under a certain mix of traffic. Five failures in a row means stop, not raise.</p>
<p><strong>5. Errors stay inside your limits.</strong> The <code>synthetic-health.sh</code> terminal should show only <code>200</code>. If a line shows anything else, stop and don't raise.</p>
<p><strong>6. The UI.</strong> Log in on <a href="https://api.clearpay.test">https://api.clearpay.test</a>, send money, and read the newest <code>served by</code> line.</p>
<h4 id="heading-the-decision-at-every-stage">The Decision at Every Stage</h4>
<p>Maya asks the same question each time: "Are we ready for the next step?"</p>
<p>Answer with evidence: "The dial matches the file and GKE's share of requests rose. Pods are healthy, login works, and synthetic health is all 200. The worst-case connection count fits under the database limit."</p>
<p>If any check fails, don't raise. If customers are affected, go to Chapter 23 now.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>Weights are 0 / 100 in the file and in <code>current_traffic_split</code></p>
</li>
<li><p>40 requests in a row return <code>"platform":"gke"</code></p>
</li>
<li><p>The connection math is written down with your own numbers</p>
</li>
<li><p>Synthetic health showed no sustained errors during any stage</p>
</li>
</ul>
<h3 id="heading-chapter-23-when-the-numbers-go-bad">Chapter 23. When the Numbers Go Bad</h3>
<p><em>Wednesday, 4:12 p.m.</em></p>
<p>The last shift is done. Every customer is now on GKE, and the health check keeps printing 200, which means all is well. The support inbox has been quiet for the past hour. Amara checks the transactions table one more time.</p>
<p>"It looks boring," she says. Which is the best thing she could say.</p>
<p>You wait for someone to call it a successful migration.</p>
<p>Daniel is still looking at the deployment. He counts the API copies running in GKE, then looks at the database metrics.</p>
<p><em>"Can I tell you about the worst afternoon at my last job?"</em></p>
<p>Chris turns his chair around.</p>
<p>Daniel leans back.</p>
<p>"It was another migration for a different company, same basic problem though. We were moving customers from one cloud to another, and we'd reached about half of them by Friday afternoon. Traffic started climbing, so the new environment began adding more copies of the API."</p>
<p>He pauses for a moment.</p>
<p>"The old environment was still running too. It was serving the customers we hadn't moved yet."</p>
<p>You look at the deployment on the screen.</p>
<p>"So both environments were using the database?" "Yes. And every copy of the API needed database connections."</p>
<p>He points at the database on the diagram.</p>
<p>"That was where we got into trouble." Daniel reaches for a pen.</p>
<p>"Think about a restaurant with a fixed number of seats. The restaurant has enough seats for a certain number of people, but the number of seats doesn't change because more people arrive."</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/c89a67f5-b226-4401-8a4f-186edacddddb.png" alt="Daniel describing his situation to the team" style="display: block;" width="600" height="400" loading="lazy">

<p>He draws two groups on the whiteboard.</p>
<p>"AWS was one group. GKE was the other. Both groups were using the same restaurant."</p>
<p>He added another group beside GKE.</p>
<p>"Then traffic increased. GKE added more copies of the API, and each new copy needed database connections. The old AWS copies were still holding their connections too."</p>
<p>He puts the pen down.</p>
<p>"The database had a limit. We kept increasing the number of connections going into it."</p>
<p>"What happened when you reached the limit?" you ask.</p>
<p>"Requests started waiting for connections. Then they started timing out."</p>
<p>He looks at you.</p>
<p>"The application itself wasn't obviously broken. If the pods were running and health checks were passing, the problem was capacity. We had more demand for database connections than the database could provide."</p>
<p>Chris looks back at the deployment.</p>
<p>"So you were watching the new environment, but the database was dealing with both environments."</p>
<p>"Exactly."</p>
<p>Daniel looks at the diagram again.</p>
<p>"That was the part we missed. We calculated what the new environment needed. We didn't calculate what the database needed while both environments were running."</p>
<p>He sits back down.</p>
<p>"We eventually moved the traffic back to the old environment and recovered. But the incident stayed with me."</p>
<p>You look at the GKE deployment. "What would you do differently now?"</p>
<p>"I'd work through the numbers before moving traffic."</p>
<p>Daniel turns his laptop toward you.</p>
<p>"Your lab is safe. You ran 3 copies against a database limit of 81 connections, and the workload never came close to exhausting it."</p>
<p>He opens another set of notes.</p>
<p>"My incident was different. We had up to 8 copies, and the database allowed 87 connections."</p>
<p>He looks at you. "So let's use those numbers."</p>
<p>You pull your chair closer. Daniel continues.</p>
<p>"I'm going to give you the events in the order they happened. You're on call. Don't assume you know the ending. Tell me what you would check and what you think is happening as each event comes in."</p>
<p>He opens a terminal.</p>
<p>The drill begins with 8 API copies available across the 2 environments and a database connection limit of 87.</p>
<p>This time, the numbers are going to matter.</p>
<p><strong>Do you run anything in this chapter?</strong> Read it once, like a story with a lesson. Your lab had no incident, so there's nothing to fix. If you want proof that you're in the clear, run the two commands in Step 2 and compare them with "Your own cluster". If the logs show no errors and the HPA is at 1 replica, you're done. Go on to Chapter 24.</p>
<h4 id="heading-what-it-looks-like">What it Looks Like</h4>
<p>The first few minutes after the 50 / 50 shift look normal. Then the response times begin to creep up.</p>
<p>Amara notices it first in the team channel.</p>
<p>"A couple of transfers just timed out."</p>
<p>Daniel opens the production dashboard. The API has scaled to seven replicas, and the p95 latency is climbing.</p>
<p>Chris posts two customer reports about slow transfers.</p>
<p>Daniel looks at the graph again. "Is this affecting every request?"</p>
<p>"Not yet," Chris says. "A few transfers so far."</p>
<p>Amara checks the payment records. "Let's make sure those timeouts didn't result in duplicate charges."</p>
<p>Daniel looks at the database metrics. "Could be the database."</p>
<p>You don't know yet.</p>
<p>Nothing in the logs says, "The database is out of connections." The first signal is simply that requests are taking longer than they were a few minutes ago.</p>
<p>No error message says "the database is out of connections". All you see first is the slowest responses (p95 latency) getting slower. The status bar in the app (ui/src/components/StatusRibbon.jsx) switches to "Degraded. Investigating" only after several health checks fail in a row, so one slow request doesn't trigger it.</p>
<h4 id="heading-what-you-know">What You Know</h4>
<p>Traffic is on both clouds. Latency is worse than the AWS baseline. The status bar flipped after several failed checks, not one blip.</p>
<h4 id="heading-what-you-dont-know-yet">What You Don't Know Yet</h4>
<p>Whether this is the load balancer, the pods, the database, or a bad deploy. So don't assume. Gather info first.</p>
<h4 id="heading-step-1-stop-shifting">Step 1. Stop Shifting</h4>
<p>Don't run <code>shift-traffic.sh</code> toward GKE again until you understand what happened. Write down the time you noticed.</p>
<p>Record what you see before you do anything else: the actual error messages, the actual replica count, and the actual timestamp.</p>
<p>Every hour spent diagnosing at a stalled percentage is also an hour of dual-run cost. Add it to the cost ledger. Don't let it drop out of the running total because everyone's attention is elsewhere.</p>
<h4 id="heading-step-2-gather-evidence">Step 2. Gather Evidence</h4>
<p>Read the API's errors:</p>
<pre><code class="language-bash">kubectl -n production logs deployment/clearpay-api --tail=100 | grep error
</code></pre>
<p>Look for words about the database: connection errors, "too many connections", or a timeout while waiting for a client from the pool. A 5xx error with no database words points somewhere else, such as the Gateway or a bad deploy.</p>
<p>Check how many pods Kubernetes is running:</p>
<pre><code class="language-bash">kubectl -n production get hpa clearpay-api
</code></pre>
<p>Compare <code>REPLICAS</code> with <code>maxReplicas</code>. A climbing count means load went up. That's a clue, not yet a cause.</p>
<p>Then fill in the Chapter 22 table with the live numbers:</p>
<pre><code class="language-plaintext">RDS max_connections:          87
AWS, 1 process × 10:          10
GKE, 7 replicas × 10:         70
Right now:                    80, close to the limit
GKE at maxReplicas 8:         80, worst case 90, over the limit
</code></pre>
<h4 id="heading-your-own-cluster-at-50-50">Your Own Cluster at 50 / 50</h4>
<p>Run the same two commands on your cluster before you assume you're in Daniel's story. This lab at AWS 50 / GKE 50 showed:</p>
<pre><code class="language-plaintext">HPA:       cpu 4%/70%, MINPODS 1, MAXPODS 3, REPLICAS 1
pods:      1 Running
logs:      no error lines in the last 100
RDS:       max_connections=81
           two sessions on database clearpay
status:    not Degraded
</code></pre>
<p>Worst case here is up to 40, under 81. Don't run <code>rollback.sh</code> unless your numbers are worse than your Chapter 6 limits.</p>
<p>If your HPA is still at 1 and the logs are quiet, write that down. That's the evidence. Stay with it.</p>
<h4 id="heading-how-the-failure-builds">How the Failure Builds</h4>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/5d5bc823-4b2a-40c0-8060-7adb2376a8b6.png" alt="Image diagram describing how the failure builds" style="display: block;" width="600" height="400" loading="lazy">

<p>AWS alone would never hit this. GKE alone would never hit this. It happens only while both run at full load, and the migration plan keeps both running on purpose.</p>
<p>Tell Daniel: "Replica count climbed. The pool math puts us near the database limit. This is capacity, not Kubernetes."</p>
<h4 id="heading-step-3-roll-back">Step 3. Roll Back</h4>
<p>Maya: "What do you recommend?"</p>
<p>Compare the numbers with the limits you wrote in Chapter 6. Don't invent a new limit now.</p>
<p>"Put customers back on AWS. Errors are past our limit, customers are affected, and rollback is tested."</p>
<p>Maya: "Okay. Do it."</p>
<pre><code class="language-bash">./scripts/rollback.sh &lt;aws-instance-ip&gt; ~/.ssh/clearpay-deployer
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e996fa61-cf03-4f11-abaf-1e0e57e9622a.png" alt="Image demonstrating instant rollback" style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram above shows two actions performed by <code>rollback.sh</code>. First, <code>shift-traffic.sh</code> <code>100 0</code> sends all customer traffic back to AWS and removes traffic from GKE. Second, the script connects to the AWS EC2 server over SSH, sets <code>ROLLBACK_ACTIVE=true</code> in the API's environment file, and restarts the API service. The API then reports AWS as the active platform and enables the distinct “Rolled back to AWS” banner in the ClearPay UI. These steps restore traffic and make the rollback visible to the operator.</p>
<p>Set <code>terraform.tfvars</code> in <code>08-edge</code> back to 100 and 0 so a later apply doesn't undo the rollback.</p>
<h4 id="heading-step-4-confirm-the-rollback-worked">Step 4. Confirm the Rollback Worked</h4>
<pre><code class="language-bash">curl -k https://api.clearpay.test/api/info
</code></pre>
<p><code>platform</code> reads <code>aws</code> and <code>rollback_active</code> reads <code>true</code>. Refresh the UI: the red "Rolled back to AWS" banner appears, and the status bar goes back to <code>All systems operational</code> once enough health checks pass.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/da9f3303-1efa-4d13-9fee-e058ee67bbaf.png" alt="ClearPay dashboard showing a red banner stating that traffic has been restored to AWS while the GKE issue is investigated. The status bar reads “All systems operational.” The page displays an account balance of $965.00, a form to send money, and recent transactions marked as served by AWS." style="display: block;" width="600" height="400" loading="lazy">

<p>Then check each of these from evidence, not assumptions:</p>
<ul>
<li><p>Requests stopped failing (<a href="http://synthetic-health.sh">synthetic-health.sh</a> shows 200 again)</p>
</li>
<li><p>Latency is back near the Chapter 6 AWS numbers</p>
</li>
<li><p>The database is consistent: a transfer from just before the rollback shows exactly one debit, not zero and not two</p>
</li>
</ul>
<p>Amara: "I don't care that traffic is back on AWS. Did the transfer happen once?"</p>
<p>Answer from the database, through the transaction history, not from the load balancer. The idempotency key from Chapter 3 is what makes a retried transfer safe here, even when the first try went to GKE and the retry went to AWS.</p>
<p>Write down one number for the incident review: the minutes from "noticed" to "traffic back on AWS".</p>
<h4 id="heading-step-5-fix-the-cause">Step 5. Fix the Cause</h4>
<p>Daniel: "What will you change so that number can't happen again?"</p>
<p>There are four real options, each with a cost:</p>
<ul>
<li><p><strong>Lower</strong> <code>DB_POOL_SIZE</code> <strong>per pod:</strong> Fewer connections per process. Cheap and fast. Each pod can serve fewer database queries at the same time.</p>
</li>
<li><p><strong>Lower the HPA's</strong> <code>maxReplicas</code> in <code>k8s/overlays/production/patch-hpa.yaml</code>. Caps the total. GKE can no longer scale as far under a real traffic spike.</p>
</li>
<li><p><strong>Raise RDS</strong> <code>max_connections</code> with a custom parameter group. Each connection uses database memory, and a small instance like <code>db.t3.micro</code> can fail to start if you set it too high. Check AWS's guidance for your instance size first.</p>
</li>
<li><p><strong>Put a connection pooler in front of Postgres</strong>, such as PgBouncer or RDS Proxy. Many app connections share a few database connections. The best long-term fix, and the most work to set up.</p>
</li>
</ul>
<p>This book doesn't pick one for you. Write down which one you chose and what you gave up.</p>
<p>For the smallest real change, lower the pool size. The math with 6 per pod: 8 × 6 = 48 from GKE, plus 10 from AWS, is 58. That fits under 87 even at the full ceiling.</p>
<p>In an emergency you can set it live:</p>
<pre><code class="language-bash">kubectl -n production set env deployment/clearpay-api DB_POOL_SIZE=6
</code></pre>
<p>A live change leaves no record, and the next deploy erases it. Make the real change through the pipeline: set <code>DB_POOL_SIZE</code> in <code>k8s/base/deployment.yaml</code>, commit, and push to <code>main</code>. The pipeline builds, scans, and deploys staging. Load-test staging hard enough to push the HPA to its ceiling, using Chapter 18's steps with higher concurrency. A fix that only passes under light load proves nothing. Then run the promotion gate, and promote with the Chapter 19 pipeline job.</p>
<h4 id="heading-step-6-resume">Step 6. Resume</h4>
<p>Maya: "Are we ready for 10 percent?"</p>
<p>"Yes. The API is healthy, synthetic checks are clean, errors are within the limit, the pool math holds at the ceiling, and rollback is tested."</p>
<p>Daniel: "Agreed."</p>
<p>Maya: "Your call."</p>
<p>Clear the rollback flag first:</p>
<pre><code class="language-bash">./scripts/clear-rollback-flag.sh &lt;aws-instance-ip&gt; ~/.ssh/clearpay-deployer
</code></pre>
<p>Refresh the UI. The rollback banner is gone. Leaving the flag on would keep the banner showing after recovery, which trains people to ignore it.</p>
<p>Then repeat Chapters 21 and 22 from the start:</p>
<pre><code class="language-bash">./scripts/shift-traffic.sh 99 1
# observe, compare, record, decide

./scripts/shift-traffic.sh 90 10
# observe, compare, record, decide

./scripts/shift-traffic.sh 75 25
./scripts/shift-traffic.sh 50 50
./scripts/shift-traffic.sh 25 75
./scripts/shift-traffic.sh 10 90
./scripts/shift-traffic.sh 0 100
</code></pre>
<p>Watch the HPA and the error lines closely at the stage where the incident happened last time. That stage is where the fix proves itself.</p>
<p>Add a line to the cost ledger. A second pass through the shift means more hours with both clouds billing. Keep that cost visible instead of folding it into one "migration cost" number.</p>
<h4 id="heading-the-skills-behind-this-chapter">The Skills Behind This Chapter</h4>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e0c6ec10-3f2b-4689-93a6-80539e82d727.png" alt="The incident response flow starts by observing latency, the status bar, and error logs. You form a hypothesis that AWS and GKE together are exhausting PostgreSQL connections, then test the combined replica count and connection pool sizes against the database limit. You record the live numbers as evidence. If customers are affected, stop shifting traffic and roll back. Fix the cause through the deployment pipeline and test the fix at maximum expected load. Resume the migration only after the evidence supports it." style="display: block;" width="600" height="400" loading="lazy">

<p>The diagram above shows the incident response process from observation to recovery. Start by checking latency, the status bar, and error logs. Form a hypothesis, test it against the database connection limit, and use live measurements as evidence. If customers are affected, stop shifting traffic and roll back. Fix the cause through the pipeline, test under maximum expected load, and resume the migration only when the evidence supports it.</p>
<p>An incident isn't a hunt for the answer. It's the work of reducing uncertainty until you can make a safe decision. If customers are failing, the safe decision is to stop and roll back. If they aren't, the safe decision is to keep watching. Don't roll back because a number looks odd, and don't keep shifting because the pods look healthy.</p>
<p>This is the checkpoint. GKE is serving 100 percent.</p>
<p>AWS is a priced safety net now, not a leftover you forgot.</p>
<h2 id="heading-part-7-how-to-finish-the-migration">Part 7. How to Finish the Migration</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/048f42c0-5eda-4d41-b518-2110ec08f856.png" alt="Image describing what we have done, where we are and what next." style="display: block;" width="600" height="400" loading="lazy">

<p>GKE serves every customer. AWS still runs, and still bills, as a safety net. This part turns the cutover into evidence, puts a date on the safety net, and removes what you no longer need.</p>
<h3 id="heading-chapter-24-prove-the-cutover-with-numbers">Chapter 24. Prove the Cutover With Numbers</h3>
<p><em>Thursday, 9:00 a.m.</em></p>
<p>Maya has a meeting with the company leaders at ten. She stops at your desk with her laptop open to an empty page.</p>
<p>Maya: "The leaders weren't on any of our calls, and they'll ask one thing: did customers notice the move? I need a short summary I can put on the screen. How many requests went through, how many failed, how fast the app answers compared with AWS, and whether anyone saw an outage. Can you have it before ten?"</p>
<p>You nod. Every number she asked for comes from two scripts you already know.</p>
<h4 id="heading-steps">Steps</h4>
<p>Run the two checks that produce hard numbers against the public address, now at 100 percent GKE:</p>
<pre><code class="language-bash">HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
./scripts/promotion-gate.sh "https://$HOST" \
  alice@clearpay.dev password123 bob@clearpay.dev
</code></pre>
<p>Then run <code>load-test.sh</code> against <code>https://$HOST</code> with the Chapter 6 settings, and copy the scorecard. Add the counts from your Chapter 22 loops and the <code>synthetic-health.sh</code> log.</p>
<h4 id="heading-the-evidence">The Evidence</h4>
<p>Below is this lab's table. Yours will have different numbers:</p>
<pre><code class="language-plaintext">Production requests (counted checks, not a full dashboard total):
  40 /health one minute after 0 / 100:  3 aws, 37 gke
  40 /health ninety seconds later:      0 aws, 40 gke
  load-test.sh 10 × 20s:                ok=310  failed=0
  promotion-gate.sh:                    5 passed, 0 failed
Successful requests:                 310 + 40 + 5 in those runs
Failed requests:                     0
5xx errors:                          0.0 %
p95 latency (GKE, 100%):             647 ms
  Chapter 6 AWS baseline was 164 ms.
  Chapter 18 GKE staging (port-forward) was 1157 ms.
Sustained customer-visible outage:   0 seconds
</code></pre>
<p>Local Prometheus didn't watch GKE, so this table uses counted checks. Don't invent a dashboard total you never measured.</p>
<p>Check the table against the limits you wrote in Chapter 6, before GKE existed:</p>
<ul>
<li><p>Outage limit: no stretch of 5xx errors above your limit. This lab had zero failed requests.</p>
</li>
<li><p>Failure limit: p95 under your ceiling. This lab's 647 ms is under the example ceiling of 1500 ms, so it passes.</p>
</li>
</ul>
<p>Passing isn't the whole story. 647 ms is about four times the AWS 164 ms. Customers never saw an outage, but every request got slower. Write that down plainly instead of hiding it. The last section of this book explains why.</p>
<h4 id="heading-why-the-table-doesnt-say-zero-failed-requests-ever">Why the Table Doesn't Say "Zero failed requests, ever"</h4>
<p>No real system running for hours, across traffic shifts and a possible rollback, can honestly promise that. Claiming it would weaken the one claim that matters: no sustained outage a customer could see.</p>
<p>A few failed requests inside the limit you set in advance is the honest version of "this went well". The standard is controlled and measured, not perfect.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>Every line in your table holds a number you measured</p>
</li>
<li><p>Each line is checked against your Chapter 6 limits, including any that failed</p>
</li>
</ul>
<h3 id="heading-chapter-25-run-on-gke-and-set-a-date-for-aws">Chapter 25. Run on GKE and Set a Date for AWS</h3>
<p><em>Friday, 10:00 a.m.</em></p>
<p>Maya comes back from her meeting with a question from the leaders.</p>
<p>Maya: "So what changed for the customer?"</p>
<p>You look at the numbers. Traffic is 100 percent GKE. Transfers succeed. Synthetic checks all pass. PostgreSQL is still on AWS. The EC2 server serves nobody.</p>
<p>"The address stayed the same and nobody saw an outage."</p>
<p>Maya: "Then this system is yours now. And I want a calendar date for turning off the old server."</p>
<h4 id="heading-step-1-watch-it-run">Step 1. Watch it Run</h4>
<p>Let production run for a real window before you commit to anything. For a real company that means days, not the minutes after the last shift.</p>
<p>This lab's 100 percent window lasted minutes, with a 5/5 gate and <code>failed=0</code>. Write that down. A weekend lab isn't a week of production.</p>
<h4 id="heading-step-2-check-the-alert">Step 2. Check the Alert</h4>
<p>Stage 7 created the alert policy <code>ClearPay API - elevated error rate</code>, which fires on more than five error lines in five minutes. It has no notification channel, so nobody gets told when it fires. An alert that reaches nobody isn't an alert.</p>
<p>For a real system, add a notification channel in Cloud Monitoring and trigger a test on staging, never by sending errors to customers. For this lab, write down the gap: the policy exists, and nobody gets paged.</p>
<h4 id="heading-step-3-review-capacity">Step 3. Review Capacity</h4>
<p>Look at the HPA over the window:</p>
<pre><code class="language-bash">kubectl -n production get hpa clearpay-api
</code></pre>
<p>This lab stayed at 1 replica and 4 to 7 percent CPU, even at 15.5 requests per second during the 100 percent load test. One pod was enough. Three <code>e2-medium</code> machines for one API pod is more than this traffic needs.</p>
<p>Google offers Committed Use Discounts (CUDs): a lower price in exchange for promising to use a set amount of machine capacity for one or three years.</p>
<p>Commit only after you have a stable usage history. A commitment based on the architecture you planned, instead of the usage you measured, is a guess with a contract attached. This lab bought none, because a few hours of GKE isn't a usage history.</p>
<h4 id="heading-step-4-write-the-handover">Step 4. Write the Handover</h4>
<p>Write this for someone who wasn't part of the migration. This lab's version:</p>
<pre><code class="language-plaintext">Logs:       kubectl -n production logs deployment/clearpay-api
Dashboard:  http://localhost:3000 if observability/docker-compose.yml
            is up. GKE series may be empty.
Alert:      Cloud Monitoring "ClearPay API - elevated error rate"
            No notification channel.
Rollback:   scripts/rollback.sh, only while EC2 still exists.
Hostname:   terraform output public_hostname
            terraform output static_ip_address
            (yours, not a hostname copied from this book)
</code></pre>
<h4 id="heading-step-5-put-a-date-on-aws">Step 5. Put a Date on AWS</h4>
<p><code>scripts/rollback.sh</code> works only while the EC2 server exists, and the server bills every day it exists. Decide how long it stays, write down why, and price it:</p>
<pre><code class="language-plaintext">AWS daily cost:              ~$1.25/day (infra/aws/COST.md)
Retention window:            ___ days
Decommission date:           ___
Total safety-margin cost:    ___ days × $1.25 = $___
</code></pre>
<p>This lab used 0 extra days and retired AWS compute the same day 100 percent GKE was proven, because it's a teardown lab.</p>
<p>A real company might keep AWS for one or two weeks. A shorter window saves money. A longer one buys more confidence. The mistake is leaving it open as "keep it around just in case", which costs money every day and never gets revisited.</p>
<p>Write the decommission rules before the date arrives, so the decision doesn't get made under deadline pressure:</p>
<ul>
<li><p>a set number of days at 100 percent GKE with no rollback</p>
</li>
<li><p>no alert fired for a GKE-caused problem in that window</p>
</li>
<li><p>a named person signs off</p>
</li>
</ul>
<p><strong>This lab's rules:</strong> weights 0 / 100, gate 5/5, load test <code>failed=0</code>, HPA at 1, no rollback.</p>
<p>Last, check that nothing else still points at the Elastic IP directly: bookmarks, hardcoded URLs in other systems, or outside uptime monitors set up before the load balancer existed.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>You watched production for a written, real time window</p>
</li>
<li><p>The alert gap is written down</p>
</li>
<li><p>Capacity is reviewed and the CUD decision is written down</p>
</li>
<li><p>The handover note exists</p>
</li>
<li><p>The retention window, decommission date, cost, and rules are written down</p>
</li>
</ul>
<h3 id="heading-chapter-26-retire-the-aws-server">Chapter 26. Retire the AWS Server</h3>
<p><em>Friday, 3:00 p.m.</em></p>
<p>The date has arrived, and the rules from Chapter 25 are met. Remove the EC2 server and keep the database. It was never part of this migration.</p>
<h4 id="heading-steps">Steps</h4>
<p>Perform your final check, from the repository root. GKE must pass the gate on the public address:</p>
<pre><code class="language-bash">HOST=$(terraform -chdir=infra/gcp/terraform/08-edge output -raw public_hostname)
./scripts/promotion-gate.sh "https://$HOST" \
  alice@clearpay.dev password123 bob@clearpay.dev
</code></pre>
<p>Then confirm that Terraform tracks the server you mean to remove:</p>
<pre><code class="language-bash">cd infra/aws/terraform
terraform state show aws_instance.api   # confirm you have the right resource
</code></pre>
<p>Next, return to the repository root with <code>cd ../../..</code>, then remove the AWS compute resources only:</p>
<pre><code class="language-bash">./scripts/teardown-aws-compute.sh
# Pass --yes to add -auto-approve after you have read a plan once.
</code></pre>
<p>After this step, rollback to AWS is gone. That's the point of the decision you made in Chapter 25.</p>
<h4 id="heading-read-the-destroy-plan">Read the Destroy Plan</h4>
<p>The script destroys only these:</p>
<ul>
<li><p>the EC2 instance</p>
</li>
<li><p>the Elastic IP</p>
</li>
<li><p>the security group rule that let the API reach the database</p>
</li>
<li><p>the API security group</p>
</li>
<li><p>the SSH key pair</p>
</li>
<li><p>the CloudWatch alarm</p>
</li>
</ul>
<p>RDS stays.</p>
<p>The database rule is its own Terraform resource, <code>aws_security_group_rule.db_from_api</code>, and the script removes it first. If that rule lived inside the database security group, AWS would refuse to delete the API security group while the database still referenced it.</p>
<p>A full <code>terraform destroy</code> in <code>infra/aws/terraform</code> would also delete PostgreSQL. Don't use it here.</p>
<h4 id="heading-clean-up-credentials">Clean Up Credentials</h4>
<ul>
<li><p>The SSH key pair in AWS is gone with the script. Delete the local private key <code>~/.ssh/clearpay-deployer</code> if you used it only for this server.</p>
</li>
<li><p>Remove anything else that pointed at the old server, such as saved hosts or scripts with the old IP.</p>
</li>
<li><p>Remove any CI/CD secrets or variables that referenced the old server.</p>
</li>
</ul>
<h4 id="heading-update-the-diagram">Update the Diagram</h4>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/7e4308e9-0c42-4246-a2f5-9c550c9fb26b.png" alt="updated diagram of of our architecture" style="display: block;" width="600" height="400" loading="lazy">

<p>The dual-cloud diagram from Chapter 7 is now wrong. The new picture is GKE alone behind the load balancer, with the database still on AWS, as Chapter 7 said the project would end.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>The gate passed on the public address before the teardown</p>
</li>
<li><p>AWS compute is destroyed, and RDS still exists</p>
</li>
<li><p>The local deploy key is removed or kept on purpose</p>
</li>
<li><p>Your architecture diagram matches what runs</p>
</li>
</ul>
<h3 id="heading-chapter-27-tear-down-gcp-and-the-database">Chapter 27. Tear Down GCP and the Database</h3>
<p><em>Friday, 5:30 p.m.</em></p>
<p>The office is emptying out. Two clouds are still billing you by the hour.</p>
<p>This is a lab. Once your notes are complete, remove everything that still bills: the Gateway load balancer, the GKE machines, Cloud NAT, the registry, and the RDS database.</p>
<p>Before tearing anything down, complete the migration record in Chapter 28. It asks you to document the final architecture, migration results, performance numbers, and total cost. If you still need evidence from the running system, collect it now. Once you destroy the infrastructure, those live measurements will no longer be available.</p>
<h4 id="heading-steps">Steps</h4>
<p>From the repository root:</p>
<pre><code class="language-bash">./scripts/teardown-gcp.sh
# Pass --yes to skip the prompt and -auto-approve each stage.
</code></pre>
<p><strong>The script:</strong></p>
<ol>
<li><p>Deletes the Gateway and the production and staging workloads. Kubernetes creates Google load balancer pieces for the Gateway, and Terraform doesn't know about them. Deleting the Gateway first lets Google remove them. Skip this and Terraform can get stuck on resources it didn't create.</p>
</li>
<li><p>Runs <code>terraform destroy</code> in reverse stage order: <code>08-edge</code> → <code>07-observability</code> → <code>06-secrets</code> → <code>05-identity</code> → <code>04-registry</code> → <code>03-gke</code> → <code>02-network</code> → <code>01-foundation</code>. Later stages depend on earlier ones, so they come down first.</p>
</li>
</ol>
<p>Type <code>yes</code> at the script's prompt, then read and confirm each Terraform plan.</p>
<p>Namespaces can sit in <code>Terminating</code> for a few minutes. The script waits up to 180 seconds for each delete, then moves on. Destroying the cluster removes them anyway.</p>
<p>Then remove the database. The migration ended with it on AWS on purpose, but a lab shouldn't keep paying about $0.80 per day for it:</p>
<pre><code class="language-bash">cd infra/aws/terraform
terraform destroy
</code></pre>
<p>Last, delete the <code>api.clearpay.test</code> line from <code>/etc/hosts</code>.</p>
<p>The book never created or deletes your GCP project. Don't delete a project unless it held only this lab.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p><code>kubectl get ns</code> no longer shows <code>staging</code> or <code>production</code>, or the cluster is gone</p>
</li>
<li><p>Every <code>infra/gcp/terraform/0*-*</code> stage is destroyed</p>
</li>
<li><p><code>terraform destroy</code> finished in <code>infra/aws/terraform</code></p>
</li>
<li><p>GCP and AWS billing show nothing from this lab still running</p>
</li>
</ul>
<p>This is the checkpoint. The work is over.</p>
<p>Write down what actually happened, including the bill, before you forget.</p>
<h3 id="heading-chapter-28-write-the-migration-record">Chapter 28. Write the Migration Record</h3>
<p><em>Week four, Monday, 9:00 a.m.</em></p>
<p>Write down what happened, the architecture, the bill, and what you would change, before the details fade.</p>
<h4 id="heading-part-1-the-architecture-record">Part 1. The Architecture Record</h4>
<p><strong>Before:</strong></p>
<pre><code class="language-plaintext">User → nginx (TLS) → systemd-managed Node process → RDS Postgres
</code></pre>
<p>One EC2 server, one database, a default network nobody designed, secrets in a plain <code>.env</code> file, no autoscaling, and a deploy process run by hand.</p>
<p><strong>During the migration:</strong></p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/1b0d833a-1077-4d01-9ab3-c259aaf5be8a.png" alt="Architecture diagram during the migration period" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>After:</strong></p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/2dcd551e-b47e-4390-a5d4-2cf4a1368581.png" alt="architecture diagram after the migration" style="display: block;" width="600" height="400" loading="lazy">

<p>Then one short paragraph for each model:</p>
<ul>
<li><p><strong>Traffic</strong>: weighted backends on a global external Application Load Balancer, changed by <code>shift-traffic.sh</code>. Not DNS, because cached DNS answers would have made rollback slow.</p>
</li>
<li><p><strong>Deploys</strong>: GitHub Actions builds, scans, and deploys every commit to staging. Production gets the exact staging image, only after the promotion gate and a reviewer's approval.</p>
</li>
<li><p><strong>Database</strong>: never moved. The same RDS instance from Chapter 4 stayed the single source of truth the whole time.</p>
</li>
<li><p><strong>Security</strong>: Workload Identity for both the pods and the pipeline. No downloaded service account key existed at any point. Secrets come from Secret Manager.</p>
</li>
<li><p><strong>Observability</strong>: the API's own JSON logs and <code>/metrics</code> endpoint, the same on AWS and GKE, plus a log-based error metric in Cloud Logging.</p>
</li>
<li><p><strong>Rollback</strong>: one script that moves the traffic weight and sets a flag on AWS, tested before the first shift.</p>
</li>
</ul>
<h4 id="heading-part-2-the-postmortem">Part 2. The Postmortem</h4>
<p>The four of you go through the notes together. Maya opens with the question everyone half expects.</p>
<p>Maya: "Was Kubernetes the problem?"</p>
<p>"Not really. It did what we told it to do. The thing to watch was the database. Every copy of the API can open ten connections, GKE can run several copies, and AWS was still serving customers at the same time. Add those up and you get close to the database's limit.</p>
<p>On this small cluster we topped out at forty connections against eighty-one, so nothing broke. On a cluster that scales to eight copies, the same sum goes over the limit."</p>
<p>Daniel nods. "That's what I missed at my last company. All the numbers were there, and nobody put them side by side. You did it before any traffic moved."</p>
<p>Amara: "And the payments? Was every transfer charged once?"</p>
<p>"Yes. The idempotency key from Chapter 3 held through every shift. We even replayed a payment at 100 percent GKE, and there was no second charge."</p>
<p>Maya closes her laptop. "So what would we do differently next time?"</p>
<p>Write these sections:</p>
<p><strong>What worked:</strong></p>
<ul>
<li><p>Idempotency keys, built in before GKE existed, made it safe for a customer's retry to land on a different cloud than the first try.</p>
</li>
<li><p>The staged Terraform meant each failure pointed at one stage.</p>
</li>
<li><p>The pipeline meant production ran the exact image that passed the gate.</p>
</li>
<li><p>Rollback was tested in Chapter 20, before it was ever needed.</p>
</li>
</ul>
<p><strong>What went wrong in this lab:</strong></p>
<ul>
<li><p>Local Prometheus never watched GKE, so Chapter 24 used counted checks instead of a dashboard total.</p>
</li>
<li><p>The alert policy has no notification channel.</p>
</li>
<li><p>The small node pool couldn't run the pod counts a real company would use, so the connection-limit risk was checked on paper, not seen live.</p>
</li>
<li><p>GKE's p95 was about four times AWS's.</p>
</li>
</ul>
<p><strong>The root cause:</strong> Two reasonable settings, <code>DB_POOL_SIZE=10</code> and the HPA's <code>maxReplicas</code>, were chosen in different places. Nobody checked their product against the database limit until the migration ran both clouds at once. The fix isn't complicated once you find it. Finding it takes treating Chapter 23 as a real investigation instead of skipping ahead to an answer.</p>
<p><strong>Decisions and their trade-offs:</strong></p>
<ul>
<li><p>GKE Standard over Autopilot: full control of node pools and autoscaling, at the cost of having to understand and size them. This lab paid for three mostly idle machines. Autopilot might have hidden this connection-limit risk inside its own scheduling, for better or worse.</p>
</li>
<li><p>Two environments instead of five: less to build and pay for, at the cost of staging sharing the production database. Fine at this scale, not for every company.</p>
</li>
<li><p>A public database endpoint with an IP allowlist instead of a VPN: no setup time and no monthly VPN bill. The cost is that RDS has a public endpoint, even if it accepts only one IP and requires TLS.</p>
</li>
<li><p>Leaving the database on AWS: a simpler, safer migration, at the cost of every query crossing between clouds.</p>
</li>
</ul>
<p>So what would change in a second migration? Before any traffic moves, compute the worst-case connection count across every environment that shares the database. Compare it with the real limit. Make it a checklist item, not a live discovery.</p>
<h4 id="heading-part-3-the-cost-postmortem">Part 3. The Cost Postmortem</h4>
<p>This lab's version:</p>
<p><strong>Budget:</strong> about $25 for five days of GCP usage.</p>
<p><strong>Actual dual-run:</strong> about $4.30 total, $1.25 AWS + $3.05 GCP. The GCP $3.05 was covered by free-trial credit, so the actual GCP charge was $0.</p>
<p>The migration stayed on schedule. No incident, rollback, or second shift was needed.</p>
<p><strong>The cluster was over-provisioned:</strong> three GKE nodes supported one API pod, with CPU at 4–7%.</p>
<p>Standard GKE was chosen for the teaching value. Autopilot would have been cheaper for this workload.</p>
<p>No CUD was purchased.</p>
<p><strong>Conclusion:</strong> The migration stayed well within budget, but the GKE setup used more capacity than the workload needed.</p>
<p>Fill in your own numbers from your cloud bills. This book can't do that part for you, and a guessed number would be the same false precision this whole project avoided.</p>
<h4 id="heading-done-when">Done When:</h4>
<ul>
<li><p>The before, during, and after diagrams match what ran</p>
</li>
<li><p>The postmortem states the root cause clearly</p>
</li>
<li><p>The cost postmortem holds your real numbers</p>
</li>
</ul>
<h2 id="heading-what-comes-after-this-project">What Comes After This Project</h2>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f6161aec-ff05-4e2b-8a21-6b9c108ea6bc.png" alt="Image describing what we have done, where we are and what next." style="display: block;" width="600" height="400" loading="lazy">

<h3 id="heading-what-question-this-project-leaves-open">What Question This Project Leaves Open</h3>
<p>Chapter 13 flagged it and moved on, on purpose: at what point does keeping PostgreSQL in AWS cost more, in risk and in engineering time, than migrating it? The cross-cloud trip, the dependency on two providers, and the connection-pool pressure from Chapter 23 all sit on the same side of that scale.</p>
<p>Put two numbers from your notes side by side:</p>
<pre><code class="language-plaintext">AWS, Chapter 6:             p95 164 ms
GKE at 100%, Chapter 24:    p95 647 ms
</code></pre>
<p>The API moved and customers never saw an outage. But every request got about four times slower.</p>
<p>The code didn't change. The load test did not change. What changed is distance. On AWS, the API and the database sat in the same data center. On GKE, every query leaves Google's <code>us-east4</code>, goes out through Cloud NAT, crosses the internet to AWS's <code>us-east-1</code>, and comes back. A single transfer runs several queries, so it pays that trip several times. The Google load balancer adds one more hop in front.</p>
<p>Maya reads the table. "Why is it slower?"</p>
<p>"The database stayed on AWS. Every query crosses between clouds."</p>
<p>"Was that a mistake?"</p>
<p>"No. Chapter 7 made the call to move the API alone, because moving two things at once would have made any failure much harder to trace. A problem could have come from the API, the network, or the database move itself. But the call has a price, and now we can measure it."</p>
<p>Maya: "Then that's the next project."</p>
<p>This project doesn't answer the open question. It was never supposed to. The next project moves PostgreSQL to Google Cloud, most likely with Google's Database Migration Service, which copies data continuously so the final switch takes minutes instead of hours. It will use the same method as this book: a baseline first, a written limit, small steps, a tested rollback, and evidence at every stage. When it's done, run the same load test and compare the same two numbers.</p>
<p>Again, that's a separate project, with its own book. This one ends here: ClearPay's API on GKE, its database still on AWS, exactly as Chapter 7 said it would.</p>
<h2 id="heading-what-you-should-have-at-the-end">What You Should Have at the End</h2>
<ul>
<li><p>Working fintech API and minimal customer UI (<code>api/</code>, <code>ui/</code>)</p>
</li>
<li><p>AWS deployment (<code>infra/aws/</code>)</p>
</li>
<li><p>GCP architecture, staged Terraform (<code>infra/gcp/terraform/</code>)</p>
</li>
<li><p>Docker images and a CI/CD pipeline that builds, scans, deploys staging, and promotes the tested image (<code>.github/workflows/deploy.yml</code>)</p>
</li>
<li><p>Kubernetes manifests, two-environment Kustomize overlays (<code>k8s/base/</code>, <code>k8s/overlays/</code>)</p>
</li>
<li><p>Database connectivity, validated and never migrated (Chapter 13)</p>
</li>
<li><p>Secret management, upgraded from a <code>.env</code> file to Secret Manager (<code>k8s/overlays/*/secretproviderclass.yaml</code>, <code>secretsync.yaml</code>)</p>
</li>
<li><p>A GKE Gateway behind a weighted load balancer (<code>k8s/gateway/</code>, <code>infra/gcp/terraform/08-edge/</code>)</p>
</li>
<li><p>Observability dashboard on your laptop, if you did the optional Chapter 17 (<code>observability/</code>)</p>
</li>
<li><p>Load-testing, promotion-gate, and synthetic health scripts (<code>scripts/</code>)</p>
</li>
<li><p>Migration scorecard (Chapter 6, Chapter 18, Chapter 24)</p>
</li>
<li><p>Cost ledger, start to finish (Chapter 6 through Chapter 28)</p>
</li>
<li><p>Traffic-shifting procedure (<code>scripts/shift-traffic.sh</code>)</p>
</li>
<li><p>Rollback procedure (<code>scripts/rollback.sh</code>, <code>scripts/clear-rollback-flag.sh</code>)</p>
</li>
<li><p>Architecture diagrams (Chapters 4, 7, 9, 11, 13, 14, 19, 28)</p>
</li>
<li><p>Migration runbook (this book)</p>
</li>
<li><p>Technical postmortem (Chapter 28)</p>
</li>
<li><p>Cost postmortem (Chapter 28)</p>
</li>
</ul>
<p>The ClearPay API moved from one AWS EC2 server to GKE. PostgreSQL is still on AWS, on purpose. Customers kept one address while you shifted traffic, practiced a rollback, and finished the cutover.</p>
<p>Maya asked what changed for the customer. The address didn't. The balance didn't. Nobody saw an outage. Now you understand why that was the hard part, and why 647 milliseconds is the next one.</p>
<p>The lab covered how to prove each stage, how to read evidence, and how to undo a bad shift. Destroy anything you're still paying for.</p>
<p>Subscribe to my <a href="https://osomudeya.kit.com/23db7ca59f">newsletter</a> for new technical articles, project walkthroughs, and upcoming releases.</p>
<h2 id="heading-how-to-discuss-this-project-in-an-interview"><strong>How to Discuss This Project in an Interview</strong></h2>
<p>I worked on a cloud migration project for a payments application called ClearPay. The application was running on AWS, and the goal was to move it to Google Cloud without disrupting access to the service.</p>
<p>The main challenge was making that move safely. Instead of switching everything at once, I designed a controlled migration where traffic could be moved gradually to the new environment. If something went wrong, I could redirect traffic back to the original system. I also kept the database on AWS to reduce the scope of the migration and avoid introducing unnecessary risk.</p>
<p>I set up the deployment process, security checks, and verification steps needed to confirm that the application was working correctly before completing the switch.</p>
<p>I then tested the final migration. The tests completed without failed requests, and the application didn't produce server errors during the final checks. I also identified a performance difference between the old and new environments that would need further investigation before considering the migration fully optimized.</p>
<p>The project demonstrates that I can plan and execute a cloud migration, manage the risks involved, verify the outcome, and identify issues that still need attention rather than simply declaring success because the application is running.</p>
<p><strong>Interview tip:</strong> Be prepared to explain why you used weighted traffic routing, how you validated the application during cutover, what triggered a rollback, and why PostgreSQL remained on AWS.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Build Your Own AI App Builder Like Lovable with Next.js, AWS and Sandboxes ]]>
                </title>
                <description>
                    <![CDATA[ Tools like Lovable, Bolt, and v0 feel a bit like magic the first time you use them. Honestly, I was shocked the first time I saw something like that...and you get to do all that from a chat window! It ]]>
                </description>
                <link>https://www.freecodecamp.org/news/build-your-own-ai-app-builder-with-next-js-aws-and-sandboxes/</link>
                <guid isPermaLink="false">6abf9aa749124019f564574f</guid>
                
                    <category>
                        <![CDATA[ AI ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ lovable ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Sandbox ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Shrijal Acharya ]]>
                </dc:creator>
                <pubDate>Fri, 02 Oct 2026 11:51:03 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/32b191cd-400c-44ea-bc95-8815897f3f82.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Tools like Lovable, Bolt, and v0 feel a bit like magic the first time you use them. Honestly, I was shocked the first time I saw something like that...and you get to do all that from a chat window! It was just wild. I had this whole existential crisis the first time I tried Lovable.</p>
<p>But have you ever wondered what's actually happening under the hood? Like, how is it even possible to have one app set up a whole other app that's ready to test, share, download, and so on?</p>
<p>Somewhere, an AI model is writing code. That's clear. But how? That code needs to be installed, built, and run. And the part I was a bit skeptical about was that nobody even seems to read that code nowadays before it runs. It could be broken, it could be slow, or it could even try to do something it shouldn't like wipe out your whole system with <code>rm -rf</code>. We've had such things happen from AI, so we can't be 100% sure.</p>
<p>That's the problem this article addresses.</p>
<p>Here, you'll build your own Lovable-style AI app builder. This isn't going to be a clone. Rather, we'll dive into the logic behind Lovable to see how it really works. We'll keep the UI pretty basic.</p>
<p>For our project, a user will be able to describe an app in plain English, and an AI agent will write the code inside an isolated cloud sandbox. The AI will fixe its own errors, and then show a live preview. From there, the user can keep building on top of it through chat, roll back to any version, and publish the finished app with a single click.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>This tutorial gets into some more advanced concepts, so before you start, it’ll help if you’re comfortable with:</p>
<ul>
<li><p>JavaScript or TypeScript</p>
</li>
<li><p>React and basic Next.js concepts</p>
</li>
<li><p>Node.js and a bit of working with APIs</p>
</li>
<li><p>Git and basic version control</p>
</li>
<li><p>Docker</p>
</li>
<li><p>Postgres</p>
</li>
<li><p>AWS is optional. You can follow the entire tutorial locally using MinIO instead.</p>
</li>
</ul>
<p>To run the project yourself, you’ll also need:</p>
<ul>
<li><p>Node.js 22+</p>
</li>
<li><p>pnpm</p>
</li>
<li><p>Docker installed locally</p>
</li>
<li><p>An API key for a sandbox provider (here we'll use Tensorlake Sandboxes)</p>
</li>
<li><p>An Anthropic (recommended) or OpenAI API key</p>
</li>
</ul>
<p>You don’t need to be an expert in any of these. A basic understanding is enough, and we’ll go through the important parts as we build.</p>
<h2 id="heading-whats-covered-here">What's Covered Here:</h2>
<p>In this tutorial, you'll build the whole thing from scratch. Here's what you'll learn along the way:</p>
<ul>
<li><p>Why AI generated code needs a sandbox</p>
</li>
<li><p>How to get a fresh sandbox ready in about 4 seconds instead of 30+ using memory snapshots</p>
</li>
<li><p>How to build an agent loop that writes code, checks its own work, and fixes its own errors</p>
</li>
<li><p>How to stream what the agent is doing to the browser in real time (and not lose it on a page refresh)</p>
</li>
<li><p>How to show a live preview with hot reload through your own gateway</p>
</li>
<li><p>How to version every change with Git without ever putting a token inside the sandbox</p>
</li>
<li><p>How to put idle sandboxes to sleep, wake them up when needed, and publish the final app</p>
</li>
</ul>
<p>This gets into some advanced concepts, but follow along and you'll learn a lot along the way. I definitely did while building it. 😉</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-whats-the-plan-the-architecture">What's the Plan (the Architecture)</a></p>
</li>
<li><p><a href="#heading-why-do-we-need-a-sandbox">Why Do We Need a Sandbox?</a></p>
</li>
<li><p><a href="#heading-how-to-set-up-the-project">How to Set Up the Project</a></p>
</li>
<li><p><a href="#heading-core-components-in-the-application">Core Components in the Application</a></p>
<ul>
<li><p><a href="#heading-starting-a-sandbox-fast">Starting a Sandbox Fast</a></p>
</li>
<li><p><a href="#heading-the-agent-loop">The Agent Loop</a></p>
</li>
<li><p><a href="#heading-letting-the-agent-check-its-own-work">Letting the Agent Check Its Own Work</a></p>
</li>
<li><p><a href="#heading-streaming-progress-to-the-browser">Streaming Progress to the Browser</a></p>
</li>
<li><p><a href="#heading-the-live-preview-gateway">The Live Preview Gateway</a></p>
</li>
<li><p><a href="#heading-versions-with-git">Versions with Git</a></p>
</li>
<li><p><a href="#heading-sleeping-waking-and-sharing-the-sandboxes">Sleeping, Waking, and Sharing the Sandboxes</a></p>
</li>
<li><p><a href="#heading-publishing-the-app">Publishing the App</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-app-builder-in-action">App Builder in Action</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
</ul>
<h2 id="heading-whats-the-plan-the-architecture">What's the Plan (the Architecture)</h2>
<p>Before diving into the code, it helps to understand how everything fits together, because there are quite a few concepts worth understanding earlier.</p>
<p>The app is split into three processes and a couple of pieces of infrastructure. One thing I was very strict about from the start is that the web app should never run the AI agent itself, and the agent should never run inside the sandbox. You'll see why that matters in a bit.</p>
<img src="https://cdn.hashnode.com/uploads/covers/641fd8b0be4ca15b2ad2a590/c3274cbe-ac2d-4fbc-ab45-b0a2899a99c9.png" alt="AI app builder architecture using Next.js, Tensorlake sandboxes, Postgres, LLM workers, Git, and AWS S3." style="display: block;" width="2580" height="1635" loading="lazy">

<p>Here's the flow from start to finish:</p>
<h3 id="heading-sending-a-prompt">Sending a Prompt</h3>
<p>When a user types a prompt, the web app (Next.js) saves it, creates a "run" in Postgres, puts a job on a queue, and returns right away. So the user isn't sitting there waiting on some HTTP request while the agent does its thing.</p>
<h3 id="heading-running-the-agent">Running the Agent</h3>
<p>A worker process picks up that job. First it makes sure the project has a running sandbox, and then it starts the agent loop. The LLM decides what to do, and every tool call it makes (write a file, run a command, install a package and all) actually happens inside the sandbox.</p>
<p>Every single step is also saved to the database as an event, and that's how the browser gets them live.</p>
<h3 id="heading-showing-the-preview">Showing the Preview</h3>
<p>The generated app runs its own Vite dev server inside the sandbox. We have a small gateway service that proxies <code>http://&lt;project-id&gt;.preview.localhost:4000</code> to that dev server, including the WebSocket that Vite uses for hot reload.</p>
<p>So as the agent edits files, the preview just updates by itself, which honestly still feels kinda cool every time I see it.</p>
<h3 id="heading-saving-and-publishing">Saving and Publishing</h3>
<p>Once the agent is done, the worker commits the changes as a new version. And when the user clicks Publish, the app gets built and the static files are uploaded to S3, so the published site keeps working even when the sandbox is asleep.</p>
<p>That's pretty much the high level architecture of our application. To put it simply:</p>
<ul>
<li><p><strong>Next.js:</strong> UI and API layer</p>
</li>
<li><p><strong>Worker (Node.js +</strong> <code>pg-boss</code><strong>):</strong> Agent layer</p>
</li>
<li><p><strong>Gateway (Node.js proxy):</strong> Preview layer</p>
</li>
<li><p><strong>Postgres:</strong> State and job queue layer</p>
</li>
<li><p><strong>Cloud sandbox:</strong> Execution layer</p>
</li>
<li><p><strong>S3 (MinIO locally):</strong> Storage layer</p>
</li>
<li><p><strong>Any LLM of your choice (Claude or GPT in our case):</strong> Reasoning layer</p>
</li>
</ul>
<h2 id="heading-why-do-we-need-a-sandbox">Why Do We Need a Sandbox?</h2>
<p>If you think about it, the whole product is basically "run code that nobody has reviewed." The agent writes it, <code>npm install</code> pulls in packages whose install scripts can run pretty much anything, and then a dev server starts executing all of it. There's no way I'm running that on my own server, and you shouldn't either.</p>
<p>So every project gets its own isolated sandbox, which is basically a small VM in the cloud. When I was picking a sandbox provider for this, these are the things I actually needed:</p>
<ul>
<li><p><strong>Suspend and resume:</strong> Most projects sit idle most of the time. I wanted to put them to sleep and wake them back up in a couple of seconds, with their memory intact.</p>
</li>
<li><p><strong>Files, commands, and a terminal:</strong> The agent needs to read and write files and run commands, and it's nice to give the user a real shell too.</p>
</li>
</ul>
<p>💁 There's much more to check on when considering using something like this on prod, but these were my only hard requirements.</p>
<p>For this, I'm using <a href="https://tensorlake.ai">Tensorlake</a> sandboxes. To be clear, there's no specific reason to use this particular one. E2B, Daytona, Modal, or even your own Firecracker setup would all work, and you're free to choose whatever you prefer. I've just been using it for a few projects already, and for sandboxes it works perfectly, especially for a use case like ours.</p>
<p><strong>Note:</strong> All the sandbox code lives in a single package (<code>packages/sandbox</code>). So if you want to switch providers, that's pretty much the only place you need to touch. The rest of the app doesn't even know which provider it's talking to.</p>
<h2 id="heading-how-to-set-up-the-project">How to Set Up the Project</h2>
<p>Before you start, make sure you have the following installed:</p>
<ul>
<li><p>Node.js 22 or newer</p>
</li>
<li><p>pnpm</p>
</li>
<li><p>Docker (for Postgres and MinIO if you plan to test it locally first)</p>
</li>
</ul>
<p>You'll also need API keys for your sandbox provider and for an LLM (<a href="https://console.anthropic.com">Anthropic</a> or <a href="https://platform.openai.com">OpenAI</a>, whichever you like).</p>
<p>Start by cloning the repository and installing the dependencies:</p>
<pre><code class="language-bash">git clone https://github.com/shricodev/lovable-build-tensorlake-aws.git
cd lovable-build-tensorlake-aws
pnpm install
</code></pre>
<p>Next, create your environment file and fill in the keys:</p>
<pre><code class="language-bash">cp .env.example .env
# Add your sandbox and LLM keys, and generate AUTH_SECRET with:
openssl rand -base64 32
</code></pre>
<p>Now start the local infrastructure, create the database tables, and set up the storage bucket:</p>
<pre><code class="language-bash"># Start Postgres and MinIO
pnpm infra:up

# Create the database tables
pnpm db:migrate

# Create the bucket and lock it down
pnpm s3:setup
</code></pre>
<p>Then build the base snapshot that every new project starts from (more on this in a bit). It takes about 45 seconds:</p>
<pre><code class="language-bash">pnpm sandbox:build-base
</code></pre>
<p>Finally, start everything:</p>
<pre><code class="language-bash"># web on :3000, preview gateway on :4000
pnpm dev
</code></pre>
<p>Open <code>http://localhost:3000</code>, sign in, and describe an app. That's it! 🎉</p>
<p><strong>Note:</strong> The app supports GitHub sign in, but it also has a simple dev login so you can try it out before creating a GitHub OAuth app. Don't worry, the dev login is always disabled in production.</p>
<h2 id="heading-core-components-in-the-application">Core Components in the Application</h2>
<p>The project is huge. Walking through every single line would turn this into an hours long read, so instead I'll focus on the core components that actually make the system work. Things like the dashboard, sign in, and the code editor are pretty standard stuff, so I'll skip those.</p>
<p><strong>Note:</strong> This means that the code snippets below are trimmed down to the important parts. You can find the complete code in the repository.</p>
<h3 id="heading-starting-a-sandbox-fast">Starting a Sandbox Fast</h3>
<p>Every generated app starts from the same template: Vite, React, TypeScript, Tailwind, and a few common libraries. The naÏve way of doing it looks something like this for every new project:</p>
<ol>
<li><p>Create a fresh sandbox</p>
</li>
<li><p>Upload the template</p>
</li>
<li><p>Run <code>npm install</code></p>
</li>
<li><p>Start the dev server</p>
</li>
</ol>
<p>And that works, but it took <strong>33.4 seconds</strong> before the preview was even reachable in my tests. Most of that (about 23 seconds) was just <code>npm install</code> running on a single vCPU. I don't know about you, but I'm not staring at a spinner for that long every time I start a new project.</p>
<p>The fix is to do all of that <strong>once</strong>, and save the result as a memory snapshot. A memory snapshot captures the files, the RAM, and the running processes. So when you restore it, the dev server is already running. Nothing has to boot again.</p>
<p>Here's the core of the base snapshot builder:</p>
<pre><code class="language-ts">export async function buildBaseSnapshot(log: Logger) {
  // Cold path, done once: create, upload template, npm install, git init,
  // verify it builds, start the dev server and warm up Vite's cache.
  const { ps } = await coldCreateFromTemplate({
    name: `base-${Date.now()}`,
    log,
    verify: true,
  });

  try {
    // Memory checkpoint: files + RAM + running processes.
    const snapshotId = await ps.checkpoint();
    writeBaseSnapshot({ snapshotId, createdAt: new Date().toISOString() });
  } finally {
    await ps.terminate();
  }
}
</code></pre>
<p>With that in place, creating a sandbox for a new project is just a restore, and then we lock it down:</p>
<pre><code class="language-ts">static async createFromSnapshot(opts: { snapshotId: string; name: string; log: Logger }) {
  const sb = await Sandbox.create({ snapshotId: opts.snapshotId, name: opts.name, timeoutSecs: 600 });
  const ps = new ProjectSandbox(sb, opts.log);

  await sb.update({
    exposedPorts: [5173],              // the Vite dev server, reachable through the proxy
    allowUnauthenticatedAccess: false, // the port URL is never public
    network: {
      allowInternetAccess: true,
      allowOut: ["registry.npmjs.org"], // npm and nothing else
      denyOut: [],
    },
  });
  return ps;
}
</code></pre>
<p>A few things worth noting here:</p>
<ul>
<li><p><code>exposedPorts</code> makes the dev server reachable through the provider's proxy, but only if you have our API key. We'll use that later in the gateway.</p>
</li>
<li><p>The <code>network</code> block is an allow list. Once <code>allowOut</code> has an entry in it, everything else is blocked.</p>
<p>💁 I actually tested this with <code>example.com</code>, <code>github.com</code>, and the cloud metadata IP, and all of them were blocked while npm still worked just fine.</p>
</li>
<li><p>There are no secrets in the sandbox environment at all. No LLM keys, no database URL, no AWS keys, nothing. So even if a generated app or some prompt injection tries to steal something, there's simply nothing in there to steal.</p>
</li>
</ul>
<p>Here are the numbers I got, measured all the way until the preview actually loads:</p>
<table>
<thead>
<tr>
<th>Path</th>
<th>Time</th>
</tr>
</thead>
<tbody><tr>
<td>Cold (create, install, start dev server)</td>
<td>33.4s</td>
</tr>
<tr>
<td>Restore from the memory snapshot</td>
<td>4.0s</td>
</tr>
<tr>
<td>Wake a sleeping sandbox</td>
<td>2.4s</td>
</tr>
</tbody></table>
<p>That's roughly 8 times faster. How cool is that? 😎</p>
<h3 id="heading-the-agent-loop">The Agent Loop</h3>
<p>The agent loop is the brain of the entire system. Every time a user sends a prompt, this is what runs.</p>
<p>The idea is simple even if the implementation isn't. You give the LLM a goal and some tools, let it call them, feed the results back, and repeat until it's done.</p>
<p>These are the tools the agent gets:</p>
<ul>
<li><p><code>list_files</code>, <code>read_file</code>, <code>write_file</code>, <code>edit_file</code>, and <code>delete_file</code></p>
</li>
<li><p><code>run_command</code> for quick checks like <code>npx tsc --noEmit</code></p>
</li>
<li><p><code>install_packages</code> for adding npm packages</p>
</li>
<li><p><code>get_dev_server_logs</code> and <code>get_browser_errors</code> for debugging</p>
</li>
<li><p><code>finish</code>, which the agent calls when it thinks it's done</p>
</li>
</ul>
<p>Each tool is just a small file with a Zod schema and a <code>run</code> function. Here's <code>edit_file</code> for example:</p>
<pre><code class="language-ts">export const editFile = defineTool({
  name: "edit_file",
  description:
    "Replace one exact snippet in a file. `search` must match exactly and occur exactly once.",
  schema: z.object({
    path: z.string(),
    search: z.string().min(1),
    replace: z.string(),
  }),
  async run({ path, search, replace }, { sandbox }) {
    const text = await sandbox.readFile(path);
    const count = text.split(search).length - 1;
    if (count !== 1) {
      return {
        isError: true,
        content: `search text occurs ${count} times in ${path}`,
      };
    }
    await sandbox.writeFile(
      path,
      text.replace(search, () =&gt; replace),
    );
    return { content: `Edited ${path}`, changedFiles: [path] };
  },
});
</code></pre>
<p>The Zod schema is doing two jobs here. It gets converted to JSON Schema for the LLM (with <code>z.toJSONSchema</code>), and it also validates whatever the model sends back before anything touches the sandbox. If the input is invalid, the error just goes back to the model instead of crashing the whole run.</p>
<p>Then the actual loop runs:</p>
<pre><code class="language-ts">while (true) {
  if (signal.aborted) return result("cancelled");

  const res = await llm.chat({
    system: SYSTEM_PROMPT,
    messages,
    tools,
    signal,
    onText,
  });
  messages.push(res.message);

  const results = [];
  for (const call of res.message.toolCalls) {
    const out = await executeTool(call.name, call.input, ctx); // validate + run
    results.push({
      type: "tool_result",
      toolCallId: call.id,
      content: out.content,
      isError: out.isError,
    });
    if (out.finish) finishCalled = true;
  }

  if (!finishCalled) {
    messages.push({ role: "user", content: results });
    continue;
  }

  // The agent says it's done. Now we check. (next section)
}
</code></pre>
<p>The <code>llm.chat</code> call is a small wrapper I wrote that supports both Anthropic and OpenAI with the same interface, so you can switch models from a dropdown in the UI.</p>
<p>On the Anthropic side, prompt caching is turned on, and to be honest, it does most of the heavy lifting on cost. A typical turn reads over 120K tokens, and almost all of them come straight from the cache.</p>
<h4 id="heading-keeping-file-paths-safe">Keeping File Paths Safe</h4>
<p>Now this one's a bit sneaky, and I only found it because I was poking around. The sandbox file API blocks paths with <code>..</code> in them, but it happily <strong>follows symlinks</strong>. So if the generated code creates a <code>leak.txt</code> that points to <code>/etc/passwd</code>, reading <code>leak.txt</code> gives you back the password file. Not great.</p>
<p>So every file tool resolves the real path inside the sandbox before touching anything:</p>
<pre><code class="language-ts">async safePath(relPath: string) {
  const abs = resolveProjectPath(relPath); // rejects "..", absolute paths, NUL bytes
  const r = await this.sb.run("realpath", { args: ["-m", "--", abs] });
  const real = r.stdout.trim();
  if (!isInsideApp(real)) throw new SandboxPathError(relPath, "resolves outside the project");
  return real;
}
</code></pre>
<p><code>realpath -m</code> follows every symlink and tells you where the path actually points to. If that ends up outside the project folder, the call just fails.</p>
<h4 id="heading-keeping-the-context-small">Keeping the Context Small</h4>
<p>Instead of replaying every past tool call on each new prompt, every turn starts a fresh conversation with:</p>
<ul>
<li><p>a one line summary of each earlier turn (what the user asked, and what the agent did)</p>
</li>
<li><p>the file tree and the list of installed packages</p>
</li>
<li><p>the current <code>src/App.tsx</code></p>
</li>
</ul>
<p>The agent reads anything else it needs with its tools. This keeps the cost of a turn pretty much flat, even after a project has had 20 prompts.</p>
<h3 id="heading-letting-the-agent-check-its-own-work">Letting the Agent Check Its Own Work</h3>
<p>LLMs are really weird. They'll happily tell you everything works when the build is basically on fire. So when the agent calls <code>finish</code>, we don't just take its word for it. We run three checks inside the sandbox:</p>
<pre><code class="language-ts">export async function runChecks(sandbox: ProjectSandbox) {
  const tsc = await sandbox.exec("npx tsc --noEmit -p . 2&gt;&amp;1", {
    timeoutSecs: 120,
  });
  const build = await sandbox.exec(
    "npx vite build --outDir /tmp/build --emptyOutDir --logLevel error 2&gt;&amp;1",
    { timeoutSecs: 180 },
  );
  const render = await renderCheck(sandbox); // renders the app once in a fake DOM

  return {
    ok: tsc.exitCode === 0 &amp;&amp; build.exitCode === 0 &amp;&amp; render.ok,
    typecheck: { ok: tsc.exitCode === 0, output: tsc.stdout },
    build: { ok: build.exitCode === 0, output: build.stdout },
    render,
  };
}
</code></pre>
<p>The first check runs TypeScript’s type checker to catch type errors without generating any files. The second runs a full Vite production build to make sure the app can actually compile successfully.</p>
<p>The third one is where it gets interesting. A lot of bugs only show up at runtime, stuff like <code>Cannot read properties of undefined (reading 'map')</code>. The usual answer to this is a headless browser, but that's heavy and slow in a small sandbox, and I really didn't want to go down that road.</p>
<p>So instead, the template ships a tiny script that renders the app once with <a href="https://github.com/capricorn86/happy-dom"><code>happy-dom</code></a> (a fake DOM for Node.js) and Vite's <code>ssrLoadModule</code>:</p>
<pre><code class="language-js">GlobalRegistrator.register({ url: "http://localhost:5173/" });
document.body.innerHTML = '&lt;div id="root"&gt;&lt;/div&gt;';
console.error = (...args) =&gt; errors.push(args.join(" "));

await server.ssrLoadModule("/src/main.tsx"); // runs the real app entry
await new Promise((r) =&gt; setTimeout(r, 1500)); // let React render

const rendered = document.getElementById("root").innerHTML.trim().length &gt; 0;
console.log(
  JSON.stringify({ ok: rendered &amp;&amp; errors.length === 0, rendered, errors }),
);
</code></pre>
<p>It takes about 2 seconds, and it caught the <code>undefined.map</code> crash along with the exact line in <code>App.tsx</code>. Noiceee!</p>
<p>If any of the checks fail, the errors go straight back to the agent as a new message, and it gets another round to fix them:</p>
<pre><code class="language-ts">lastCheck = await runChecks(sandbox);
if (lastCheck.ok) return result("succeeded");
if (healRounds &gt;= maxHeal)
  return result("failed", { error: describeFailures(lastCheck) });

healRounds++;
messages.push({
  role: "user",
  content: [
    ...results,
    {
      type: "text",
      text: `Verification failed. Fix these problems, then call finish again.\n\n${describeFailures(lastCheck)}`,
    },
  ],
});
</code></pre>
<p>It stops after 3 rounds by default. If it's still broken after that, the user gets an honest "I couldn't finish this one" with the actual errors.</p>
<h3 id="heading-streaming-progress-to-the-browser">Streaming Progress to the Browser</h3>
<p>The agent runs in the worker, but the user is looking at the browser. So somehow every step ("Wrote <code>src/App.tsx</code>", "Ran <code>npx tsc --noEmit</code>", "Checks passed" and all) has to get from one to the other as it happens.</p>
<p>The worker writes each step as a row in a <code>run_events</code> table and then fires a Postgres <code>NOTIFY</code>:</p>
<pre><code class="language-ts">export async function appendEvent(
  db: Db,
  e: { projectId: string; runId?: string; type: string; payload: unknown },
) {
  const [row] = await db
    .insert(runEvents)
    .values(e)
    .returning({ id: runEvents.id });
  await db.$client.notify(
    "events",
    JSON.stringify({ projectId: e.projectId, id: row.id }),
  );
  return row.id;
}
</code></pre>
<p>On the web side, a Next.js route handler streams these events to the browser with Server Sent Events. When the browser connects, it first replays everything from the active run, and then it just keeps listening for new rows:</p>
<pre><code class="language-ts">const pump = async () =&gt; {
  const rows = await db
    .select()
    .from(runEvents)
    .where(and(eq(runEvents.projectId, project.id), gt(runEvents.id, cursor)))
    .orderBy(asc(runEvents.id));

  for (const r of rows) {
    cursor = r.id;
    send(`id: ${r.id}\ndata: ${JSON.stringify(r)}\n\n`);
  }
};

bus.on(project.id, pump); // fired by a single LISTEN connection per process
pump(); // replay first
</code></pre>
<p>Since all the events live in the database, refreshing the page in the middle of a run doesn't lose anything. The browser reconnects, replays the run so far, and continues from where it left off. The Stop button works the same way, just in reverse. The web app sends a <code>NOTIFY</code> with the run ID, and whichever worker is holding that run aborts it.</p>
<p><strong>Note:</strong> Streamed text from the LLM comes in token by token, and saving every token would mean hundreds of rows per turn. So the worker buffers it and writes one row every 250ms instead. I also had a small bug here where events landed out of order because each write was its own promise. Pushing every write through a single promise chain fixed it.</p>
<h3 id="heading-the-live-preview-gateway">The Live Preview Gateway</h3>
<p>The generated app's dev server runs inside the sandbox on port 5173, and the provider exposes it at a URL like <code>https://5173-&lt;sandbox-id&gt;.sandbox.example</code>. Now, you could just put that URL in an iframe and call it a day, but there are two problems with that:</p>
<ol>
<li><p>To load it, the browser would need our sandbox API key. Yeah, that's clearly not happening.</p>
</li>
<li><p>And if we made it public instead, anyone who guessed the URL could open it, and we'd have no control over who sees what.</p>
</li>
</ol>
<p>So we put our own small gateway in front of it. It maps <code>&lt;project-id&gt;.preview.localhost:4000</code> to the right sandbox and adds the API key on the server side:</p>
<pre><code class="language-ts">const proxy = createProxyServer({ changeOrigin: true, secure: true, ws: true });

const server = http.createServer(async (req, res) =&gt; {
  const projectId = HOST_RE.exec(req.headers.host ?? "")?.[1];
  const target = await resolve(projectId); // project -&gt; sandbox, cached for a few seconds
  if (!target.sandboxId) return send(res, noPreviewPage());

  delete req.headers.cookie; // never forward the visitor's credentials
  delete req.headers.authorization;
  proxy.web(req, res, {
    target: previewUrlFor(target.sandboxId),
    headers: { authorization: `Bearer ${API_KEY}` },
  });
});

// Vite's hot reload runs over a WebSocket, so upgrades get proxied too.
server.on("upgrade", async (req, socket, head) =&gt; {
  const target = await resolve(HOST_RE.exec(req.headers.host ?? "")?.[1]);
  proxy.ws(req, socket, head, {
    target: previewUrlFor(target.sandboxId).replace(/^https/, "wss"),
    headers: { authorization: `Bearer ${API_KEY}` },
  });
});
</code></pre>
<p>That <code>upgrade</code> handler is what makes the preview feel alive. Whenever the agent edits a file, Vite pushes the change over the WebSocket, and the preview updates without a reload.</p>
<p>There's also another reason for having the gateway that's pretty easy to miss. The preview runs on a <strong>different origin</strong> (<code>*.preview.localhost</code>) than the main app (<code>localhost:3000</code>). So a generated app can never read the main app's cookies or call its API as the logged in user. And since <code>*.localhost</code> resolves to <code>127.0.0.1</code> in modern browsers, you don't even need to touch your hosts file for this.</p>
<p><strong>Note:</strong> The template also injects a tiny script into the preview that listens for <code>window.onerror</code>, and posts them to the parent window. The workspace then forwards those to the backend, and that's where the agent's <code>get_browser_errors</code> tool gets real runtime errors from.</p>
<h3 id="heading-versions-with-git">Versions with Git</h3>
<p>Every finished prompt becomes a version. So, the user can pretty much revert to a specific "prompt", more like <code>git reset</code>.</p>
<p>For this, plain old Git inside the sandbox works great. The base snapshot already has a repo with the template as the first commit, and after each successful turn, the worker commits everything:</p>
<pre><code class="language-ts">export async function commitAll(ps: ProjectSandbox, message: string) {
  const out = await git(
    ps,
    `git add -A
if git diff --cached --quiet; then
  echo NOCHANGE
else
  git commit -q -m "$MSG"
  git rev-parse HEAD
  git show --name-only --format= HEAD
fi`,
    { MSG: message },
  );
  if (out.trim() === "NOCHANGE") return null;
  const [sha, ...files] = out.trim().split("\n");
  return { sha, files };
}
</code></pre>
<p><strong>Note:</strong> You might be wondering why there's no <code>exit 0</code> in there. Commands run under <code>bash -l</code>, and an explicit <code>exit</code> in a login shell runs <code>~/.bash_logout</code>, whose last command failed on this image and turned my exit code into a failure. This one honestly took me way longer to figure out. 😭</p>
<p>Restoring never rewrites history. It makes the files match the old commit exactly, and then commits that as a brand new version on top. So "restore version 1" creates version 5, and you can still go back to version 4 whenever you want. Nothing gets lost.</p>
<h4 id="heading-keeping-the-history-durable-without-tokens-in-the-sandbox">Keeping the History Durable (Without Tokens in the Sandbox)</h4>
<p>Git inside the sandbox is great, but it only lives as long as the sandbox does. So after each version, the history also gets pushed to a hosted Git repository.</p>
<p>The straightforward way to do this would be to give the sandbox a Git token and just run <code>git push</code>. But the tokens I had access to were scoped to the <strong>whole project</strong>, not a single repo. Putting one of those inside a sandbox full of untrusted code? Yeah, no thanks.</p>
<p>So the sandbox never pushes anything. It creates a <code>git bundle</code> (basically the entire repo in a single file), the worker reads that file out, and the worker does the push itself:</p>
<pre><code class="language-ts">export async function pushToHostedGit(
  ps: ProjectSandbox,
  repo: string,
  dataDir: string,
) {
  await git(ps, "git bundle create -q /tmp/repo.bundle main");
  const bundle = await ps.sb.readFile("/tmp/repo.bundle");

  const mirror = join(dataDir, "git", `${repo}.git`); // a bare repo on the worker
  writeFileSync(join(mirror, "incoming.bundle"), bundle);
  await run("git", [
    "-C",
    mirror,
    "fetch",
    "-q",
    "--force",
    "incoming.bundle",
    "+refs/heads/main:refs/heads/main",
  ]);

  const cred = await repos.credential(repo); // short lived token, only on the worker
  await run("git", [
    "-C",
    mirror,
    "-c",
    `http.extraHeader=Authorization: Basic ${basic(cred)}`,
    "push",
    "-q",
    "--force",
    url,
    "main",
  ]);
}
</code></pre>
<p>A nice side effect of this is that it also powers <strong>remix</strong>. When someone copies a shared project, the worker loads the source project's bundle from the hosted repo into a brand new sandbox, and the original sandbox doesn't even have to wake up for it.</p>
<h3 id="heading-sleeping-waking-and-sharing-the-sandboxes">Sleeping, Waking, and Sharing the Sandboxes</h3>
<p>A running sandbox costs money even when nobody's working on it. So there's a small job that runs every minute and suspends any sandbox that hasn't had an agent run for 10 minutes. Suspending keeps the memory, so the dev server comes back exactly as it was.</p>
<p>Waking up happens in the gateway. If someone opens the preview of a sleeping project, the gateway shows a small "Waking up your app..." page that keeps refreshing itself, and resumes the sandbox in the background:</p>
<pre><code class="language-ts">if (isPage &amp;&amp; target.status === "suspended") {
  const outcome = wake(projectId, target, log); // deduplicated per project
  const done = await Promise.race([outcome, timeout(4000, "pending")]);
  if (done === "busy") return send(res, busyPage());
  if (done !== "running") return send(res, wakingPage()); // auto refreshes
}
</code></pre>
<p>In my tests, a sleeping sandbox woke up in about 3 seconds, and the app loaded right after. 🎊</p>
<h4 id="heading-sharing-a-limited-number-of-sandboxes">Sharing a Limited Number of Sandboxes</h4>
<p>Most sandbox providers limit how many sandboxes you can run at the same time, especially on a free plan. So I added a small concurrency checker, and it all happens in Postgres with an advisory lock, so two workers never end up making the same decision at the same time:</p>
<pre><code class="language-ts">async function decide(db: Db, projectId: string) {
  return db.transaction(async (tx) =&gt; {
    await tx.execute(sql`select pg_advisory_xact_lock(${LOCK_KEY})`);

    const live = await runningSandboxesOldestFirst(tx);
    if (live.some((s) =&gt; s.projectId === projectId)) return { kind: "ok" };
    if (live.length &lt; concurrencyLimit()) return { kind: "ok" };

    // Full. Free a slot by suspending the least recently used idle sandbox.
    const victim = live.find((s) =&gt; !busyProjects.has(s.projectId));
    if (victim) return { kind: "evict", sandboxId: victim.sandboxId };

    return { kind: "wait", position }; // everything is busy, just wait...
  });
}
</code></pre>
<p>To test this, I set the limit to 1 and sent a prompt to one project while another project's sandbox was just sitting idle. The idle one went to sleep, and the new one took its slot. Then I sent a prompt to the first project while the second was still working, and the UI showed "Waiting for a free sandbox, #1 in line" until the slot freed up. If you have a bigger plan, you just raise the limit in <code>.env</code> and nothing else changes.</p>
<h3 id="heading-publishing-the-app">Publishing the App</h3>
<p>The preview is great while you're building, but you don't want your published app to depend on a sandbox that goes to sleep every 10 minutes. 🫩</p>
<p>So publishing builds the app once and turns it into plain static files:</p>
<pre><code class="language-ts">export async function publish(project: Project) {
  const ps = await projectSandbox(project.id); // wakes it if needed
  const build = await ps.exec(
    "rm -rf dist &amp;&amp; npx vite build --outDir dist --emptyOutDir 2&gt;&amp;1",
    { timeoutSecs: 180 },
  );
  if (build.exitCode !== 0)
    throw new HttpError(422, `The build failed:\n${build.stdout.slice(-1500)}`);

  const files = await listFiles(ps, "dist");
  const prefix = `published/${slug}/${versionId}/`;

  for (const f of files) {
    const bytes = await ps.readBytes(`dist/${f}`);
    await storage.put(prefix + f, bytes, contentTypeFor(f), cacheControlFor(f));
  }

  await savePublishedSite({ projectId: project.id, slug, s3Prefix: prefix });
  return { url: publishedUrl(slug) };
}
</code></pre>
<p>The gateway then serves those files from S3 at <code>http://&lt;slug&gt;.app.localhost:4000</code>. Files that Vite hashes (like <code>assets/index-CScgwd68.js</code>) get cached forever, and <code>index.html</code> always gets revalidated, so updates show up right away. Any path without a file extension falls back to <code>index.html</code>, so client side routing works as well.</p>
<p>And published sites get their own subdomain on purpose. If they lived under the main app's domain, a published app's JavaScript could call your API with the cookies of whoever is looking at it, and you really don't want that. 😺</p>
<p>In my tests, publishing took about 6 seconds, and the published site loaded in 4ms with the sandbox asleep, because it never touches the sandbox at all.</p>
<h2 id="heading-app-builder-in-action">App Builder in Action</h2>
<p>Here's a quick demo of the app builder in action:</p>
<div class="embed-wrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/hE96nLJo_fc" style="aspect-ratio: 16 / 9; width: 100%; height: auto;" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen="" loading="lazy"></iframe></div>

<p>I also ran a small eval out of curiosity. It sends the same three prompts (a habit tracker, a kanban board, and an expense dashboard with charts) to two different models, each in a fresh sandbox:</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Passed (build + render)</th>
<th>Avg time</th>
</tr>
</thead>
<tbody><tr>
<td>Claude Sonnet 5</td>
<td>3/3</td>
<td>135s</td>
</tr>
<tr>
<td>GPT 5.5</td>
<td>3/3</td>
<td>92s</td>
</tr>
</tbody></table>
<p>All six apps built and rendered on the first check, without needing a single fix round, which honestly surprised me a bit. On Claude, each app cost somewhere around $0.15 to $0.20.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>So, what do you think of the project? This was truly one of the most fun projects I've worked on in a while since this AI stuff has taken over raw coding. 🤦‍♂️</p>
<p>When you use tools like Lovable, it's easy to think it's all about the prompt and the model. But once you build one yourself, you realize most of the work is everything around the model, like where the code runs, how fast it starts, how you show it to the user, and how you keep it from doing something it shouldn't.</p>
<p>If there's one thing I'd want you to take away from this, it's the sandbox part. Treat AI generated code as untrusted, give it its own small machine with no secrets and almost no network access. Memory snapshots and suspend/resume then take care of making it fast and cheap.</p>
<p>There's still a lot of room to extend this. You could add a small backend (like Hono and SQLite) to the template so users can build full stack apps, let users click an element in the preview to edit exactly that component, or generate a few design variations of the same prompt side by side. I'm just too exhausted to implement that right now. I'll leave it up to you. ✌️</p>
<p>The foundation is there. The rest is just building on top of it.</p>
<p>You can find the complete source code here: <a href="https://github.com/shricodev/lovable-build-tensorlake-aws">shricodev/lovable-build-tensorlake-aws</a></p>
<p>So, that's it for this article. Thank you so much for reading! See you next time. 🫡</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ AWS Cloud Cost Monitoring, Alerting, and Optimization: A Guide for Devs ]]>
                </title>
                <description>
                    <![CDATA[ There's a common conversation that happens in engineering teams every month. Someone forwards a screenshot of the AWS bill. The number is higher than last month. Everyone nods and agrees it should be  ]]>
                </description>
                <link>https://www.freecodecamp.org/news/aws-cloud-cost-monitoring-alerting-and-optimization-a-guide-for-devs/</link>
                <guid isPermaLink="false">6a9ee93ad24de200149086c9</guid>
                
                    <category>
                        <![CDATA[ Cloud Computing ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ monitoring ]]>
                    </category>
                
                    <category>
                        <![CDATA[ optimization ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Ayobami Adejumo ]]>
                </dc:creator>
                <pubDate>Mon, 07 Sep 2026 16:41:30 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/01c5b407-d719-404c-a207-495ae6dcaa53.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>There's a common conversation that happens in engineering teams every month. Someone forwards a screenshot of the AWS bill. The number is higher than last month. Everyone nods and agrees it should be lower. Nothing specific gets decided, and the cycle repeats.</p>
<p>This guide is designed to end that cycle by replacing it with something more useful: a systematic, service-by-service approach to knowing exactly what you're spending, why you're spending it, catching problems before they become invoices, and reducing costs without guesswork.</p>
<p>It's organised as a reference you can return to. Each part is complete on its own: you can go straight to the RDS section if that's your current problem, or follow the guide start to finish if you're building a FinOps practice from scratch. Every command is runnable, and every script is deployable.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-part-1-monitoring-know-where-every-dollar-goes">Part 1: Monitoring — Know Where Every Dollar Goes</a></p>
</li>
<li><p><a href="#heading-part-2-alerting-catch-spikes-before-they-become-invoices">Part 2: Alerting — Catch Spikes Before They Become Invoices</a></p>
</li>
<li><p><a href="#heading-part-3-optimisation-by-service">Part 3: Optimisation by Service</a></p>
</li>
<li><p><a href="#heading-part-4-the-30-day-optimisation-sprint">Part 4: The 30-Day Optimisation Sprint</a></p>
</li>
<li><p><a href="#heading-best-practices-summary">Best Practices Summary</a></p>
</li>
<li><p><a href="#heading-resources">Resources</a></p>
</li>
</ul>
<h2 id="heading-what-youll-learn">What You'll Learn</h2>
<ul>
<li><p>How to set up Cost and Usage Report querying with Athena, the foundation of all serious cost analysis</p>
</li>
<li><p>The five dashboards every engineering team needs, built with queries you can run today</p>
</li>
<li><p>A three-tier alerting strategy that catches cost spikes without creating alert fatigue</p>
</li>
<li><p>Service-specific optimisation playbooks for EC2, S3, Lambda, RDS, DynamoDB, and data transfer</p>
</li>
<li><p>A concrete 30-day sprint that produces measurable savings in the first month</p>
</li>
</ul>
<p>Let's build this from the ground up.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>Before following this guide, you should have:</p>
<p><strong>Knowledge:</strong></p>
<ul>
<li><p>Working familiarity with AWS services: EC2, S3, RDS, Lambda, and VPC</p>
</li>
<li><p>Comfort reading Python and SQL</p>
</li>
<li><p>Basic understanding of how IAM policies and roles work</p>
</li>
</ul>
<p><strong>Access:</strong></p>
<ul>
<li><p>AWS account with billing access. The IAM user or role you work with needs <code>ce:GetCostAndUsage</code>, <code>ec2:Describe*</code>, <code>rds:Describe*</code>, and <code>s3:GetBucketLifecycleConfiguration</code> permissions.</p>
</li>
<li><p>AWS CLI v2 configured</p>
</li>
<li><p>Athena access (for the CUR queries in Part 1)</p>
</li>
</ul>
<p><strong>Setup:</strong></p>
<ul>
<li>Enable Cost Explorer if it isn't already. It's free and required for most commands in this guide:</li>
</ul>
<pre><code class="language-bash">aws ce enable-cost-explorer --region us-east-1
</code></pre>
<ul>
<li>The Cost and Usage Report (CUR) should be configured and exporting to an S3 bucket. If it isn't yet, the <a href="https://docs.aws.amazon.com/cur/latest/userguide/cur-create.html">AWS CUR setup guide</a> walks through the process. Give the report 24 hours after setup to generate the first data file.</li>
</ul>
<h2 id="heading-part-1-monitoring-know-where-every-dollar-goes">Part 1: Monitoring — Know Where Every Dollar Goes</h2>
<h3 id="heading-11-the-cost-and-usage-report-your-source-of-truth">1.1 The Cost and Usage Report — Your Source of Truth</h3>
<p>Cost Explorer shows you service-level totals. It's useful for trends, but it isn't sufficient for root cause analysis. When you need to know which specific resource is responsible for a $12,000/month line item, you need the Cost and Usage Report queried through Athena.</p>
<p>Create the Athena table over your CUR data:</p>
<pre><code class="language-sql">-- Run this once in Athena after your CUR starts generating data
-- Replace 'your-cur-bucket' and 'your-prefix' with your actual values

CREATE EXTERNAL TABLE IF NOT EXISTS cur_database.billing (
    bill_billing_period_start_date  STRING,
    bill_payer_account_id           STRING,
    line_item_usage_start_date      STRING,
    line_item_resource_id           STRING,
    line_item_usage_type            STRING,
    line_item_usage_amount          DOUBLE,
    line_item_unblended_cost        DOUBLE,
    product_servicecode             STRING,
    product_instance_type           STRING,
    product_region                  STRING,
    resource_tags_user_environment  STRING,
    resource_tags_user_team         STRING,
    resource_tags_user_service      STRING,
    resource_tags_user_owner        STRING
)
PARTITIONED BY (year STRING, month STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LOCATION 's3://your-cur-bucket/your-prefix/'
TBLPROPERTIES ('skip.header.line.count'='1');

MSCK REPAIR TABLE cur_database.billing;
</code></pre>
<p>Here are the three queries you should run on day one:</p>
<pre><code class="language-sql">-- Query 1: Top 20 resources by cost this month
-- Run this first. It tells you where to focus.
SELECT
    line_item_resource_id,
    product_servicecode,
    resource_tags_user_team     AS team,
    resource_tags_user_service  AS service,
    SUM(line_item_unblended_cost) AS total_cost_usd
FROM cur_database.billing
WHERE line_item_usage_start_date &gt;= DATE_FORMAT(
    DATE_TRUNC('month', CURRENT_DATE), '%Y-%m-%d'
)
  AND line_item_unblended_cost &gt; 0
GROUP BY 1, 2, 3, 4
ORDER BY total_cost_usd DESC
LIMIT 20;
</code></pre>
<pre><code class="language-sql">-- Query 2: Week-over-week cost growth by service
-- Identifies which services are growing fastest — these need investigation
WITH weekly AS (
    SELECT
        DATE_TRUNC('week', CAST(line_item_usage_start_date AS DATE)) AS week,
        product_servicecode,
        SUM(line_item_unblended_cost) AS cost
    FROM cur_database.billing
    WHERE line_item_usage_start_date &gt;=
          DATE_FORMAT(DATE_ADD('day', -42, CURRENT_DATE), '%Y-%m-%d')
    GROUP BY 1, 2
)
SELECT
    curr.product_servicecode AS service,
    ROUND(prev.cost, 2)      AS prev_week_cost,
    ROUND(curr.cost, 2)      AS curr_week_cost,
    ROUND(
        100.0 * (curr.cost - prev.cost) / NULLIF(prev.cost, 0),
        1
    ) AS pct_change
FROM weekly curr
JOIN weekly prev
  ON curr.product_servicecode = prev.product_servicecode
  AND curr.week = DATE_ADD('week', 1, prev.week)
WHERE curr.week = DATE_TRUNC('week', CURRENT_DATE)
  AND ABS((curr.cost - prev.cost) / NULLIF(prev.cost, 0)) &gt; 0.20
ORDER BY pct_change DESC;
</code></pre>
<pre><code class="language-sql">-- Query 3: Resources running with no usage (candidates for shutdown)
-- Finds resources that incurred cost but had zero usage quantity
-- in the past 7 days — strong signal for idle or orphaned resources
SELECT
    line_item_resource_id,
    product_servicecode,
    resource_tags_user_owner    AS owner,
    resource_tags_user_team     AS team,
    SUM(line_item_unblended_cost) AS cost_past_7_days
FROM cur_database.billing
WHERE line_item_usage_start_date &gt;=
      DATE_FORMAT(DATE_ADD('day', -7, CURRENT_DATE), '%Y-%m-%d')
  AND line_item_usage_amount = 0
  AND line_item_unblended_cost &gt; 5
GROUP BY 1, 2, 3, 4
ORDER BY cost_past_7_days DESC
LIMIT 30;
</code></pre>
<h3 id="heading-12-the-five-essential-cost-dashboards">1.2 The Five Essential Cost Dashboards</h3>
<p>These five views cover the monitoring needs of most engineering teams. Each is built from queries you can run immediately, no third-party tool required.</p>
<h4 id="heading-dashboard-1-executive-summary-for-weekly-leadership-updates">Dashboard 1: Executive Summary (for weekly leadership updates)</h4>
<pre><code class="language-python"># executive_summary.py
import boto3
from datetime import datetime, timedelta

ce = boto3.client('ce')


def weekly_summary():
    today     = datetime.now()
    start_mtd = today.replace(day=1).strftime('%Y-%m-%d')
    today_str = today.strftime('%Y-%m-%d')

    # Month-to-date spend
    mtd = ce.get_cost_and_usage(
        TimePeriod={'Start': start_mtd, 'End': today_str},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )
    mtd_spend = float(
        mtd['ResultsByTime'][0]['Total']['UnblendedCost']['Amount']
    )

    # End-of-month forecast
    forecast = ce.get_cost_forecast(
        TimePeriod={
            'Start': today_str,
            'End':   (today.replace(day=28) + timedelta(days=4)).replace(day=1).strftime('%Y-%m-%d'),
        },
        Metric='UNBLENDED_COST',
        Granularity='MONTHLY'
    )
    eom_forecast = float(forecast['Total']['Amount']) + mtd_spend

    # Top 5 services
    by_service = ce.get_cost_and_usage(
        TimePeriod={'Start': start_mtd, 'End': today_str},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[{'Type': 'DIMENSION', 'Key': 'SERVICE'}]
    )
    services = sorted(
        [
            (g['Keys'][0], float(g['Metrics']['UnblendedCost']['Amount']))
            for g in by_service['ResultsByTime'][0]['Groups']
        ],
        key=lambda x: x[1],
        reverse=True
    )[:5]

    print(f"\n{'─'*48}")
    print(f"  AWS Cost Summary — {today.strftime('%B %Y')}")
    print(f"{'─'*48}")
    print(f"  Month-to-date:     ${mtd_spend:&gt;12,.2f}")
    print(f"  End-of-month est:  ${eom_forecast:&gt;12,.2f}")
    print(f"\n  Top 5 Services:")
    for name, cost in services:
        short = name.replace('Amazon ', '').replace('AWS ', '')
        print(f"    {short:&lt;32} ${cost:&gt;9,.2f}")
    print(f"{'─'*48}\n")


weekly_summary()
</code></pre>
<h4 id="heading-dashboard-2-team-cost-breakdown-for-engineering-leads">Dashboard 2: Team Cost Breakdown (for engineering leads)</h4>
<pre><code class="language-python"># team_breakdown.py
import boto3
from datetime import datetime

ce = boto3.client('ce')


def team_breakdown():
    start = datetime.now().replace(day=1).strftime('%Y-%m-%d')
    end   = datetime.now().strftime('%Y-%m-%d')

    response = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[
            {'Type': 'TAG',       'Key': 'Team'},
            {'Type': 'DIMENSION', 'Key': 'SERVICE'},
        ]
    )

    by_team = {}
    for group in response['ResultsByTime'][0].get('Groups', []):
        team_raw = group['Keys'][0]
        team     = team_raw.replace('Team$', '') if team_raw else 'untagged'
        service  = group['Keys'][1]
        cost     = float(group['Metrics']['UnblendedCost']['Amount'])

        if team not in by_team:
            by_team[team] = {'total': 0.0, 'by_service': {}}
        by_team[team]['total'] += cost
        by_team[team]['by_service'][service] = (
            by_team[team]['by_service'].get(service, 0.0) + cost
        )

    total_bill = sum(d['total'] for d in by_team.values())

    print(f"\n{'─'*58}")
    print(f"  Team Cost Breakdown — MTD {datetime.now().strftime('%Y-%m-%d')}")
    print(f"  Total: ${total_bill:,.2f}")
    print(f"{'─'*58}")

    for team, data in sorted(by_team.items(), key=lambda x: x[1]['total'], reverse=True):
        pct = (data['total'] / total_bill * 100) if total_bill else 0
        print(f"\n  {team:&lt;20}  ${data['total']:&gt;10,.2f}  ({pct:.1f}%)")
        top3 = sorted(data['by_service'].items(), key=lambda x: x[1], reverse=True)[:3]
        for svc, cost in top3:
            short = svc.replace('Amazon ', '').replace('AWS ', '')
            print(f"    └─ {short:&lt;30} ${cost:&gt;8,.2f}")

    print()


team_breakdown()
</code></pre>
<h4 id="heading-dashboard-3-waste-detection-for-weekly-cleanup-reviews">Dashboard 3: Waste Detection (for weekly cleanup reviews):</h4>
<pre><code class="language-python"># waste_detector.py
import boto3
from datetime import datetime, timezone, timedelta

ec2  = boto3.client('ec2')
elbv2 = boto3.client('elbv2')
cw   = boto3.client('cloudwatch')


def detect_waste():
    report = {'items': [], 'total_monthly_waste': 0.0}

    # Unattached EBS volumes
    for vol in ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )['Volumes']:
        age  = (datetime.now(timezone.utc) - vol['CreateTime']).days
        cost = round(vol['Size'] * 0.08, 2)
        tags = {t['Key']: t['Value'] for t in vol.get('Tags', [])}
        report['items'].append({
            'type':         'Unattached EBS Volume',
            'id':           vol['VolumeId'],
            'detail':       f"{vol['Size']}GB — {age} days old",
            'owner':        tags.get('Owner', '—'),
            'monthly_cost': cost,
        })
        report['total_monthly_waste'] += cost

    # Unassociated Elastic IPs
    for addr in ec2.describe_addresses()['Addresses']:
        if 'AssociationId' not in addr:
            report['items'].append({
                'type':         'Unassociated Elastic IP',
                'id':           addr.get('AllocationId', ''),
                'detail':       addr['PublicIp'],
                'owner':        '—',
                'monthly_cost': 3.60,
            })
            report['total_monthly_waste'] += 3.60

    # Idle load balancers (fewer than 100 requests in 7 days)
    for lb in elbv2.describe_load_balancers()['LoadBalancers']:
        metrics = cw.get_metric_statistics(
            Namespace='AWS/ApplicationELB',
            MetricName='RequestCount',
            Dimensions=[{'Name': 'LoadBalancer',
                         'Value': lb['LoadBalancerArn'].split(':loadbalancer/')[-1]}],
            StartTime=datetime.now() - timedelta(days=7),
            EndTime=datetime.now(),
            Period=604800,
            Statistics=['Sum']
        )['Datapoints']
        total_requests = metrics[0]['Sum'] if metrics else 0

        if total_requests &lt; 100:
            report['items'].append({
                'type':         'Idle Load Balancer',
                'id':           lb['LoadBalancerName'],
                'detail':       f"{int(total_requests)} requests in 7 days",
                'owner':        '—',
                'monthly_cost': 22.0,
            })
            report['total_monthly_waste'] += 22.0

    print(f"\n  Waste Detection Report — {datetime.now().strftime('%Y-%m-%d')}")
    print(f"  Estimated monthly waste: ${report['total_monthly_waste']:.2f}\n")

    for item in sorted(report['items'], key=lambda x: x['monthly_cost'], reverse=True)[:20]:
        print(f"  [{item['type']}]")
        print(f"    ID:     {item['id']}")
        print(f"    Detail: {item['detail']}")
        print(f"    Owner:  {item['owner']}")
        print(f"    Cost:   ${item['monthly_cost']:.2f}/month\n")

    return report


detect_waste()
</code></pre>
<h3 id="heading-13-tagging-strategy-the-foundation-of-all-attribution">1.3 Tagging Strategy — The Foundation of All Attribution</h3>
<p>Every cost attribution model depends on tags. Teams that skip tagging build dashboards that show totals without explanations. The discipline is in making tagging structural rather than procedural: enforced by infrastructure code, not by reminders in Confluence.</p>
<p>Required tag set:</p>
<pre><code class="language-hcl"># terraform/variables.tf
variable "mandatory_tags" {
  description = "Tags applied to every resource in this account"
  type        = map(string)

  validation {
    condition = alltrue([
      contains(keys(var.mandatory_tags), "Environment"),
      contains(keys(var.mandatory_tags), "Team"),
      contains(keys(var.mandatory_tags), "Owner"),
      contains(keys(var.mandatory_tags), "Service"),
    ])
    error_message = "mandatory_tags must include Environment, Team, Owner, and Service."
  }
}

locals {
  common_tags = merge(var.mandatory_tags, {
    ManagedBy    = "terraform"
    LastModified = timestamp()
  })
}

resource "aws_instance" "api_server" {
  ami           = data.aws_ami.amazon_linux_2023.id
  instance_type = "t3.medium"
  tags          = merge(local.common_tags, {Name = "api-server-${var.environment}"})
}
</code></pre>
<p>Find and report untagged resources weekly:</p>
<pre><code class="language-bash">#!/usr/bin/env bash
# find_untagged.sh

echo "Untagged EC2 instances (missing Team tag):"
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[?!not_null(Tags[?Key=='Team'].Value|[0])].[InstanceId,InstanceType,LaunchTime]" \
  --output table

echo "Untagged RDS instances:"
aws rds describe-db-instances \
  --query "DBInstances[?!not_null(TagList[?Key=='Team'].Value|[0])].DBInstanceIdentifier" \
  --output table
</code></pre>
<h2 id="heading-part-2-alerting-catch-spikes-before-they-become-invoices">Part 2: Alerting — Catch Spikes Before They Become Invoices</h2>
<p>The typical discovery timeline without proactive alerting: a cost spike happens on the 5th, the monthly invoice arrives on the 20th, someone notices on the 22nd, investigation begins on the 23rd, and two weeks of billed waste can't be recovered. With proactive alerting, discovery happens within hours.</p>
<h3 id="heading-21-the-three-tier-alert-structure">2.1 The Three-Tier Alert Structure</h3>
<p>Alert fatigue is as damaging as no alerting. The three-tier model keeps signal high by routing different severity levels to different channels with different response expectations.</p>
<pre><code class="language-python"># alert_router.py
import boto3
import json
import urllib.request
from enum import Enum

SLACK_INFO_WEBHOOK  = 'https://hooks.slack.com/services/INFO/WEBHOOK'
SLACK_ALERT_WEBHOOK = 'https://hooks.slack.com/services/ALERT/WEBHOOK'
SNS_CRITICAL_TOPIC  = 'arn:aws:sns:us-east-1:YOUR_ACCOUNT:cost-critical'


class AlertTier(Enum):
    INFO     = 1
    WARNING  = 2
    CRITICAL = 3


def route_alert(tier: AlertTier, subject: str, message: str):
    """Send an alert to the appropriate channel for its severity tier."""
    icons   = {AlertTier.INFO: ':information_source:',
               AlertTier.WARNING: ':warning:', AlertTier.CRITICAL: ':rotating_light:'}
    payload = {'text': f"{icons[tier]} *{subject}*\n{message}"}

    if tier == AlertTier.INFO:
        _post_slack(SLACK_INFO_WEBHOOK, payload)
    elif tier == AlertTier.WARNING:
        _post_slack(SLACK_ALERT_WEBHOOK, payload)
        _send_sns(SNS_CRITICAL_TOPIC, subject, f"WARNING: {message}")
    elif tier == AlertTier.CRITICAL:
        _post_slack(SLACK_ALERT_WEBHOOK, payload)
        _send_sns(SNS_CRITICAL_TOPIC, subject, f"CRITICAL: {message}")


def _post_slack(webhook: str, payload: dict):
    req = urllib.request.Request(
        webhook,
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json'}
    )
    urllib.request.urlopen(req)


def _send_sns(topic_arn: str, subject: str, message: str):
    sns = boto3.client('sns')
    sns.publish(TopicArn=topic_arn, Subject=subject[:100], Message=message)
</code></pre>
<p>The three tiers and what they respond to: Tier 1 INFO goes to a Slack informational channel for daily cost summaries, weekly trend reports, and tag compliance updates. No action required.</p>
<p>Tier 2 WARNING goes to a Slack alert channel plus email for budget above 75% utilisation, 25% week-over-week increases, and expiring Savings Plans. Acknowledge within 24 hours.</p>
<p>Tier 3 CRITICAL goes to PagerDuty plus SMS for budget above 90% utilisation, 100% increase in 24 hours, crypto mining detected, and projected overspend above 120% of plan. Investigate within 1 hour.</p>
<h3 id="heading-22-real-time-budget-monitor">2.2 Real-Time Budget Monitor</h3>
<p>AWS Budgets sends alerts once daily by default. A daily window means a cost spike that begins at 08:00 isn't caught until the next day's alert fires. The Lambda below runs hourly and checks both absolute budget utilisation and hour-over-hour rate of change.</p>
<pre><code class="language-python"># budget_monitor.py
# Lambda triggered by EventBridge every hour

import boto3
from datetime import datetime, timedelta
from alert_router import route_alert, AlertTier

ce      = boto3.client('ce')
budgets = boto3.client('budgets', region_name='us-east-1')
ACCOUNT_ID  = boto3.client('sts').get_caller_identity()['Account']
BUDGET_NAME = 'monthly-infrastructure'


def get_mtd_spend() -&gt; float:
    start = datetime.now().replace(day=1).strftime('%Y-%m-%d')
    end   = datetime.now().strftime('%Y-%m-%d')
    r = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )
    return float(r['ResultsByTime'][0]['Total']['UnblendedCost']['Amount'])


def get_budget_limit() -&gt; float:
    r = budgets.describe_budget(AccountId=ACCOUNT_ID, BudgetName=BUDGET_NAME)
    return float(r['Budget']['BudgetLimit']['Amount'])


def get_hourly_costs(hours: int = 4) -&gt; list:
    """Return hourly cost totals for the last N hours."""
    end   = datetime.now()
    start = end - timedelta(hours=hours)
    r = ce.get_cost_and_usage(
        TimePeriod={'Start': start.strftime('%Y-%m-%d'), 'End': end.strftime('%Y-%m-%d')},
        Granularity='HOURLY',
        Metrics=['UnblendedCost']
    )
    return [
        float(period['Total']['UnblendedCost']['Amount'])
        for period in r['ResultsByTime']
    ]


def lambda_handler(event, context):
    mtd_spend    = get_mtd_spend()
    budget_limit = get_budget_limit()
    utilisation  = mtd_spend / budget_limit * 100

    days_elapsed  = datetime.now().day
    projected_eom = (mtd_spend / days_elapsed) * 30
    projected_pct = projected_eom / budget_limit * 100

    if utilisation &gt;= 90:
        route_alert(
            AlertTier.CRITICAL,
            f'Budget at {utilisation:.0f}%',
            f'MTD spend ${mtd_spend:,.2f} is {utilisation:.0f}% of ${budget_limit:,.0f} budget. '
            f'Projected EOM: ${projected_eom:,.2f}.'
        )
    elif utilisation &gt;= 75:
        route_alert(
            AlertTier.WARNING,
            f'Budget at {utilisation:.0f}%',
            f'MTD spend ${mtd_spend:,.2f} is {utilisation:.0f}% of ${budget_limit:,.0f} budget. '
            f'Projected EOM: ${projected_eom:,.2f}.'
        )

    # Check hourly spike
    hourly = get_hourly_costs(hours=4)
    if len(hourly) &gt;= 2:
        last_hour = hourly[-1]
        prev_avg  = sum(hourly[:-1]) / len(hourly[:-1])
        if prev_avg &gt; 0.10 and last_hour &gt; prev_avg * 1.5:
            route_alert(
                AlertTier.WARNING,
                'Hourly cost spike detected',
                f'Last hour: ${last_hour:.2f} vs prior 3-hour avg ${prev_avg:.2f} '
                f'(+{(last_hour/prev_avg - 1)*100:.0f}%)'
            )

    return {
        'mtd_spend':       round(mtd_spend, 2),
        'utilisation_pct': round(utilisation, 1),
        'projected_eom':   round(projected_eom, 2),
    }
</code></pre>
<h2 id="heading-part-3-optimisation-by-service">Part 3: Optimisation by Service</h2>
<h3 id="heading-31-ec2-seven-levers-in-priority-order">3.1 EC2 — Seven Levers in Priority Order</h3>
<p>EC2 is the largest line item in most AWS accounts and the one with the most optimisation options. Work through these levers in order, as each one lowers the baseline that the next lever acts on.</p>
<p>Lever 1: Find truly idle instances (CPU below 1% for 14 days).</p>
<pre><code class="language-python"># ec2_idle_finder.py
import boto3
from datetime import datetime, timedelta

ec2 = boto3.client('ec2')
cw  = boto3.client('cloudwatch')


def find_idle_instances(avg_cpu_threshold: float = 1.0, days: int = 14):
    instances = [
        inst
        for r in ec2.describe_instances(
            Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
        )['Reservations']
        for inst in r['Instances']
    ]

    idle = []
    for inst in instances:
        iid   = inst['InstanceId']
        stats = cw.get_metric_statistics(
            Namespace='AWS/EC2',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'InstanceId', 'Value': iid}],
            StartTime=datetime.utcnow() - timedelta(days=days),
            EndTime=datetime.utcnow(),
            Period=days * 86400,
            Statistics=['Average']
        )['Datapoints']

        avg_cpu = stats[0]['Average'] if stats else 0.0
        if avg_cpu &lt; avg_cpu_threshold:
            tags = {t['Key']: t['Value'] for t in inst.get('Tags', [])}
            idle.append({
                'instance_id':   iid,
                'instance_type': inst['InstanceType'],
                'avg_cpu':       round(avg_cpu, 2),
                'environment':   tags.get('Environment', '—'),
                'owner':         tags.get('Owner', '—'),
            })

    return sorted(idle, key=lambda x: x['avg_cpu'])


for inst in find_idle_instances():
    print(f"  {inst['instance_id']}  {inst['instance_type']}  "
          f"{inst['avg_cpu']}% CPU  env:{inst['environment']}  owner:{inst['owner']}")
</code></pre>
<p>Lever 2: Right-size over-provisioned instances (CPU below 20%, sustained).</p>
<p>Use the same script with <code>avg_cpu_threshold=20.0</code>. These are right-sizing candidates, not shutdown candidates.</p>
<p>Lever 3: Schedule dev and staging shutdowns using EventBridge rules on <code>AutoShutdown=true</code> tagged instances.</p>
<p>Lever 4: Purchase Savings Plans only after completing levers 1–3.</p>
<p>Lever 5: Migrate to Graviton (20% cheaper, same performance for most workloads).</p>
<p>Lever 6: Use Spot for fault-tolerant batch and development workloads.</p>
<p>Lever 7: Migrate containerised workloads to EKS with Karpenter for automatic bin-packing.</p>
<p>Spot savings estimate:</p>
<pre><code class="language-python"># spot_price_analyser.py
import boto3

ec2 = boto3.client('ec2')


def spot_savings_estimate(instance_type: str) -&gt; dict:
    spot_history = ec2.describe_spot_price_history(
        InstanceTypes=[instance_type],
        ProductDescriptions=['Linux/UNIX'],
        MaxResults=1
    )['SpotPriceHistory']
    spot_price = float(spot_history[0]['SpotPrice']) if spot_history else 0

    on_demand_approx = {
        't3.medium': 0.0416, 'm5.large': 0.096,
        'c5.xlarge': 0.17,   'r5.2xlarge': 0.504,
    }
    od_price    = on_demand_approx.get(instance_type, 0)
    savings_pct = ((od_price - spot_price) / od_price * 100) if od_price else 0

    return {
        'instance_type': instance_type,
        'spot_price':    round(spot_price, 4),
        'on_demand':     od_price,
        'savings_pct':   round(savings_pct, 1),
        'monthly_spot':  round(spot_price * 730, 2),
        'monthly_od':    round(od_price * 730, 2),
    }


for itype in ['t3.medium', 'm5.large', 'c5.xlarge']:
    r = spot_savings_estimate(itype)
    print(f"  {r['instance_type']:&lt;15} Spot: ${r['spot_price']}/hr  "
          f"OD: ${r['on_demand']}/hr  Savings: {r['savings_pct']}%")
</code></pre>
<h3 id="heading-32-s3-lifecycle-policies-and-storage-class-selection">3.2 S3 — Lifecycle Policies and Storage Class Selection</h3>
<p>S3 optimisation has two components: moving infrequently accessed data to cheaper storage classes via lifecycle policies, and eliminating waste patterns like incomplete multipart uploads.</p>
<pre><code class="language-python"># s3_lifecycle_applier.py
import boto3

s3 = boto3.client('s3')

LOG_POLICY = {
    'Rules': [{
        'ID': 'standard-tiering',
        'Status': 'Enabled',
        'Filter': {'Prefix': ''},
        'Transitions': [
            {'Days': 30,  'StorageClass': 'STANDARD_IA'},
            {'Days': 90,  'StorageClass': 'GLACIER_IR'},
            {'Days': 365, 'StorageClass': 'DEEP_ARCHIVE'},
        ],
        'Expiration': {'Days': 2555},
        'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 7},
    }]
}

TEMP_POLICY = {
    'Rules': [{
        'ID': 'temp-data-retention',
        'Status': 'Enabled',
        'Filter': {'Prefix': ''},
        'Expiration': {'Days': 30},
        'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 1},
    }]
}

for bucket in s3.list_buckets()['Buckets']:
    name = bucket['Name']
    try:
        s3.get_bucket_lifecycle_configuration(Bucket=name)
        print(f"  {name} — policy already exists, skipping")
    except s3.exceptions.ClientError:
        policy = TEMP_POLICY if any(k in name for k in ['temp', 'build', 'cache']) else LOG_POLICY
        s3.put_bucket_lifecycle_configuration(
            Bucket=name, LifecycleConfiguration=policy
        )
        print(f"  {name} — applied {'TEMP' if policy is TEMP_POLICY else 'LOG'} policy")
</code></pre>
<h3 id="heading-33-rds-five-optimisation-levels">3.3 RDS — Five Optimisation Levels</h3>
<table>
<thead>
<tr>
<th>Level</th>
<th>Action</th>
<th>Typical Saving</th>
<th>Risk</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Delete unused read replicas</td>
<td>30–50% of replica cost</td>
<td>Low</td>
</tr>
<tr>
<td>2</td>
<td>Reduce backup retention to compliance minimum</td>
<td>20–30% of storage cost</td>
<td>Low</td>
</tr>
<tr>
<td>3</td>
<td>Right-size instance class (CPU below 20% sustained)</td>
<td>20–40% of compute</td>
<td>Medium</td>
</tr>
<tr>
<td>4</td>
<td>Purchase Reserved Instances for production</td>
<td>30–60% of compute</td>
<td>Low</td>
</tr>
<tr>
<td>5</td>
<td>Migrate variable-load DBs to Aurora Serverless v2</td>
<td>40–70% total</td>
<td>High effort</td>
</tr>
</tbody></table>
<p>Find over-provisioned RDS instances:</p>
<pre><code class="language-python"># rds_rightsizer.py
import boto3
from datetime import datetime, timedelta

rds = boto3.client('rds')
cw  = boto3.client('cloudwatch')


def find_oversized_rds():
    instances  = rds.describe_db_instances()['DBInstances']
    candidates = []

    for inst in instances:
        iid    = inst['DBInstanceIdentifier']
        iclass = inst['DBInstanceClass']

        stats = cw.get_metric_statistics(
            Namespace='AWS/RDS',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'DBInstanceIdentifier', 'Value': iid}],
            StartTime=datetime.utcnow() - timedelta(days=14),
            EndTime=datetime.utcnow(),
            Period=1209600,
            Statistics=['Average', 'Maximum']
        )['Datapoints']

        if not stats:
            continue

        avg_cpu = stats[0]['Average']
        max_cpu = stats[0]['Maximum']

        if avg_cpu &lt; 20 and max_cpu &lt; 50:
            candidates.append({
                'id':       iid,
                'class':    iclass,
                'avg_cpu':  round(avg_cpu, 1),
                'max_cpu':  round(max_cpu, 1),
                'engine':   inst['Engine'],
            })

    return candidates


for c in find_oversized_rds():
    print(f"  {c['id']}  {c['class']}  avg:{c['avg_cpu']}%  max:{c['max_cpu']}%  engine:{c['engine']}")
</code></pre>
<h3 id="heading-34-dynamodb-on-demand-vs-provisioned-decision">3.4 DynamoDB — On-Demand vs Provisioned Decision</h3>
<p>On-demand is convenient but can be 3–5× more expensive than provisioned for predictable workloads. Provisioned with auto-scaling covers most cases at significantly lower cost.</p>
<pre><code class="language-python"># dynamodb_mode_advisor.py
import boto3
from datetime import datetime, timedelta

dynamodb = boto3.client('dynamodb')
cw       = boto3.client('cloudwatch')


def analyse_table_billing(table_name: str) -&gt; dict:
    table = dynamodb.describe_table(TableName=table_name)['Table']
    mode  = table.get('BillingModeSummary', {}).get('BillingMode', 'PROVISIONED')

    stats = {}
    for metric in ['ConsumedReadCapacityUnits', 'ConsumedWriteCapacityUnits']:
        data = cw.get_metric_statistics(
            Namespace='AWS/DynamoDB',
            MetricName=metric,
            Dimensions=[{'Name': 'TableName', 'Value': table_name}],
            StartTime=datetime.utcnow() - timedelta(days=30),
            EndTime=datetime.utcnow(),
            Period=86400,
            Statistics=['Average', 'Maximum']
        )['Datapoints']
        if data:
            avg  = sum(d['Average'] for d in data) / len(data)
            peak = max(d['Maximum'] for d in data)
            stats[metric] = {'avg': round(avg, 1), 'peak': round(peak, 1)}

    if mode == 'PAY_PER_REQUEST':
        avg_rcu = stats.get('ConsumedReadCapacityUnits', {}).get('avg', 0)
        avg_wcu = stats.get('ConsumedWriteCapacityUnits', {}).get('avg', 0)

        if avg_rcu &gt; 2000 or avg_wcu &gt; 500:
            return {
                'table':          table_name,
                'current_mode':   'PAY_PER_REQUEST',
                'recommendation': 'Switch to PROVISIONED with auto-scaling',
                'reason': f'Avg {avg_rcu:.0f} RCU/s and {avg_wcu:.0f} WCU/s — predictable pattern',
            }

    return {'table': table_name, 'current_mode': mode, 'recommendation': 'No change needed'}


for table in dynamodb.list_tables()['TableNames']:
    r = analyse_table_billing(table)
    if r['recommendation'] != 'No change needed':
        print(f"  {r['table']}: {r['recommendation']}")
</code></pre>
<h3 id="heading-35-data-transfer-the-three-main-waste-patterns">3.5 Data Transfer — The Three Main Waste Patterns</h3>
<p>Data transfer charges are often the most confusing line item on an AWS bill. Here are the the three main patterns and their fixes:</p>
<p>Cross-AZ traffic (most common, most fixable): services in different AZs incur $0.01/GB in each direction. The fix is to use topology-aware routing on Kubernetes Services or ensure your application tier and database tier use the same AZ placement.</p>
<p>NAT Gateway charges for internal AWS traffic: S3, ECR, DynamoDB, and SQS traffic that routes through NAT Gateway incurs $0.045/GB. The fix: VPC endpoints eliminate this entirely.</p>
<p>Inter-region replication that compliance doesn't require: audit your S3 replication rules quarterly against actual compliance requirements.</p>
<pre><code class="language-bash"># Find S3 buckets with active replication
for bucket in $(aws s3api list-buckets --query 'Buckets[*].Name' --output text); do
    result=$(aws s3api get-bucket-replication --bucket "$bucket" 2&gt;&amp;1)
    if ! echo "$result" | grep -q "ReplicationConfigurationNotFoundError"; then
        echo "  $bucket — replication active, verify compliance requirement"
    fi
done
</code></pre>
<h2 id="heading-part-4-the-30-day-optimisation-sprint">Part 4: The 30-Day Optimisation Sprint</h2>
<p>This sprint produces measurable savings in the first month. It's designed for a single engineer with two to four hours per week of dedicated FinOps time.</p>
<p>Week 1 – Visibility: enable CUR, set up the Athena table, run the three day-one queries, screenshot the results as your baseline, tag 100% of running EC2 and RDS instances, and identify your top three cost drivers with a documented hypothesis for each.</p>
<p>Week 2 – Quick wins: deploy the orphaned resource reporter Lambda, apply S3 lifecycle policies to your three largest buckets, run the idle instance finder and stop anything below 1% average CPU with no owner objection, and add VPC endpoints for S3, ECR, and DynamoDB.</p>
<p>Week 3 – Right-sizing: run the EC2 rightsizing analyser, downsize the three highest-confidence candidates (non-production first), run the RDS rightsizing script, and find read replicas serving minimal traffic and decommission them.</p>
<p>Week 4 – Alerting and automation: deploy the hourly budget monitor Lambda, configure the three-tier alert routing, set up the weekly waste reporter, add the Infracost GitHub Action to your infrastructure repository, and schedule a monthly 30-minute FinOps review meeting.</p>
<p>Expected outcome after 30 days: 15–25% reduction in monthly AWS spend, documented evidence of every change, and a recurring process that prevents the same waste from accumulating again.</p>
<h2 id="heading-best-practices-summary">Best Practices Summary</h2>
<p>✅ <strong>Do:</strong> Set up CUR + Athena before any other monitoring. Cost Explorer is a starting point, while CUR is the source of truth.</p>
<p>✅ <strong>Do:</strong> Enforce tagging in Terraform or CloudFormation. Process-based tagging decays, but infrastructure-enforced tagging is permanent.</p>
<p>✅ <strong>Do:</strong> Run the idle instance finder and waste reporter weekly. Waste accumulates continuously. A weekly report keeps the pile small.</p>
<p>✅ <strong>Do:</strong> Use the three-tier alert model. One alert channel with everything in it creates fatigue and gets muted.</p>
<p>✅ <strong>Do:</strong> Work through the EC2 optimisation levers in order. Right-sizing before Savings Plans prevents locking in waste at a discount.</p>
<p>✅ <strong>Do:</strong> Check DynamoDB billing mode against actual usage patterns quarterly.</p>
<p>❌ <strong>Don't:</strong> Delete untagged resources without investigation. Untagged doesn't mean unused, it means unclaimed.</p>
<p>❌ <strong>Don't:</strong> Apply aggressive S3 lifecycle policies without auditing access patterns first. Glacier retrieval fees can exceed Standard storage costs if data is accessed more frequently than expected.</p>
<p>❌ <strong>Don't:</strong> Run the waste reporter Lambda with auto-deletion enabled on its first deployment. Run in report-only mode for two weeks to validate the output before adding deletion logic.</p>
<h2 id="heading-resources">Resources</h2>
<ul>
<li><p><a href="https://docs.aws.amazon.com/cur/latest/userguide/data-dictionary.html"><strong>AWS Cost and Usage Report Data Dictionary</strong></a>: Column reference for all CUR Athena queries in this guide</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/cost-management/latest/APIReference/"><strong>AWS Cost Explorer API Reference</strong></a>: Full reference for the Python boto3 cost queries</p>
</li>
<li><p><a href="https://aws.amazon.com/compute-optimizer/"><strong>AWS Compute Optimizer</strong></a>: ML-powered right-sizing recommendations, useful as a cross-check against the manual analyser scripts</p>
</li>
<li><p><a href="https://aws.amazon.com/dynamodb/pricing/"><strong>Amazon DynamoDB Pricing</strong></a>: The definitive reference for the provisioned vs on-demand cost calculation in Section 3.4</p>
</li>
<li><p><a href="https://aws.amazon.com/solutions/implementations/instance-scheduler-on-aws/"><strong>AWS Instance Scheduler</strong></a>: The official AWS solution for tag-based EC2 and RDS scheduling</p>
</li>
<li><p><a href="https://www.finops.org/framework/"><strong>FinOps Foundation Framework</strong></a>: The practitioner framework that defines the Inform, Optimise, Operate cycle this guide implements</p>
</li>
<li><p><a href="https://github.com/aayostem/platform-toolkit"><strong>Companion Repository</strong></a>: All scripts, Lambda functions, and Terraform modules from this guide</p>
</li>
</ul>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Fix the Dual-Write Problem in Node.js with the Outbox Pattern ]]>
                </title>
                <description>
                    <![CDATA[ Imagine you're building an e-commerce platform where placing an order needs to trigger several things at once: the warehouse has to be told to prepare the shipment, the email service has to send a con ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-fix-the-dual-write-problem-in-node-js-with-the-outbox-pattern/</link>
                <guid isPermaLink="false">6a736f87fcec1e65edd2a703</guid>
                
                    <category>
                        <![CDATA[ Node.js ]]>
                    </category>
                
                    <category>
                        <![CDATA[ design patterns ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Docker ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Gabor Koos ]]>
                </dc:creator>
                <pubDate>Wed, 05 Aug 2026 17:14:47 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/bcec9aaf-d418-4e5a-b8aa-f3c75b35f482.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Imagine you're building an e-commerce platform where placing an order needs to trigger several things at once: the warehouse has to be told to prepare the shipment, the email service has to send a confirmation, and the fraud checker has to review the transaction.</p>
<p>The order service handles the checkout, saves the order to its database, and then publishes an <code>order.created</code> event to a message queue so every downstream system can react independently.</p>
<p>This is a common and reasonable design, but it has a reliability problem that's easy to miss until something goes wrong in production.</p>
<p>When a customer places an order and the payment goes through, the application needs to do two things: save the order to the database and publish the event to the queue. These are two separate writes to two separate systems, and there's no way to make them share a single atomic transaction. If the process crashes, the network hiccups, or a deployment rolls out between the two writes, one side commits and the other does not. The order sits confirmed on the customer's screen while the warehouse has no idea it exists.</p>
<p>The <a href="https://microservices.io/patterns/data/transactional-outbox.html">transactional outbox pattern</a> is the standard solution to this problem. In this article, we'll build it from scratch in Node.js, using PostgreSQL for the order service database, SQS for the queue, and DynamoDB as the fulfillment service's database. For local development, we'll use <a href="https://floci.io">floci</a>, a free open-source AWS emulator that runs all three with a single Docker container.</p>
<h2 id="heading-what-well-cover">What We'll Cover</h2>
<ul>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-the-problem-with-two-writes">The Problem with Two Writes</a></p>
</li>
<li><p><a href="#heading-the-outbox-pattern">The Outbox Pattern</a></p>
</li>
<li><p><a href="#heading-what-well-build">What We'll Build</a></p>
</li>
<li><p><a href="#heading-project-setup">Project Setup</a></p>
</li>
<li><p><a href="#heading-database-schema">Database Schema</a></p>
</li>
<li><p><a href="#heading-the-request-handler">The Request Handler</a></p>
</li>
<li><p><a href="#heading-the-relay-worker">The Relay Worker</a></p>
</li>
<li><p><a href="#heading-the-consumer">The Consumer</a></p>
</li>
<li><p><a href="#heading-running-the-whole-thing">Running the Whole Thing</a></p>
</li>
<li><p><a href="#heading-going-to-production">Going to Production</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
</ul>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>To follow along, you should be comfortable with:</p>
<ul>
<li><p>Node.js and async/await</p>
</li>
<li><p>Database transactions (BEGIN, COMMIT, ROLLBACK)</p>
</li>
<li><p>The general concept of a message queue</p>
</li>
</ul>
<p>You don't need prior experience with AWS, SQS, or DynamoDB. We'll be running everything locally.</p>
<p>You will need Node.js 20 or later and Docker installed on your machine.</p>
<h2 id="heading-the-problem-with-two-writes">The Problem with Two Writes</h2>
<p>The order service scenario from the intro is one place this problem appears, but the same pattern comes up in many other contexts.</p>
<p>A user registers and the app inserts their account record, then sends a message to trigger the welcome email and the onboarding workflow. A file is uploaded and the API writes the metadata to the database, then publishes a message to kick off a processing worker for virus scanning or thumbnail generation. A payment webhook arrives, the handler records it in the database, then notifies downstream services that the payment is confirmed.</p>
<p>In every case, the application needs two writes to succeed together: one to the database and one to a queue or external system. If the second one is lost, the first one has no way of knowing.</p>
<p>If you want a deeper look at what database transactions actually guarantee and where they stop helping, see <a href="https://blog.gaborkoos.com/posts/2026-08-01-Beyond-Happy-Path-Engineering-Databases/">Beyond Happy Path Engineering: Databases</a>.</p>
<p>The naïve implementation looks straightforward:</p>
<pre><code class="language-js">await db.query('INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2)', [customerId, amountCents]);
await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify({ customerId, amountCents }) }));
</code></pre>
<p>The database write happens first, then the queue write. Under normal conditions this works fine. The problem is what happens when something goes wrong between the two.</p>
<p>If the process crashes, runs out of memory, or gets killed mid-deployment after the database write but before <code>sqs.send</code> is called, the order record exists in the database but no event is ever published. The warehouse, email service, and fraud checker never find out the order happened. From the customer's perspective the order went through. From every downstream system's perspective it doesn't exist.</p>
<p>The failure can also go the other way. If <code>sqs.send</code> succeeds but the database write is later rolled back due to a constraint violation or an error in a subsequent step, you've published an event for an order that doesn't actually exist. A consumer acting on that event may try to fulfill an order with no corresponding record, or charge a customer for something that was never saved.</p>
<p>There's also a timing window even when both writes eventually succeed. Between the database commit and the successful <code>sqs.send</code>, a consumer that queries the database after receiving the event may not find the order yet, depending on transaction isolation and replication lag. These are two separate systems with no shared transaction boundary, and no amount of careful sequencing fully closes the gap.</p>
<p>These aren't edge cases that only happen under extraordinary circumstances. Deploys restart processes mid-request. Out-of-memory kills happen without warning. Networks drop connections at any point. Any of these can interrupt the two-write sequence, and the result is a system that's silently inconsistent with no error logged and no alert fired.</p>
<p>A variation I've seen a few times that looks safer but is actually worse is wrapping both operations in a database transaction:</p>
<pre><code class="language-js">// PLEASE DO NOT EVER DO THIS
const client = await pool.connect();
await client.query('BEGIN');
await client.query('INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2)', [customerId, amountCents]);
await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify({ customerId, amountCents }) }));
await client.query('COMMIT');
</code></pre>
<p>The intent is to make the two writes feel like a unit, but a database transaction has no authority over SQS. The transaction can only roll back database operations. If <code>sqs.send</code> succeeds and then <code>COMMIT</code> fails, the message is already in the queue and can't be taken back. If the process crashes after <code>COMMIT</code> but before the function returns, the transaction committed and the message was sent, but the caller may retry, potentially inserting a duplicate order.</p>
<p>Beyond the correctness problems, this pattern holds an open database connection and any row locks for the entire duration of the SQS network call. SQS is normally fast, but under load, retries, or a degraded queue, that call can take seconds. Every other request trying to read or write the same rows has to wait. In a busy application, this is a reliable way to exhaust the connection pool and bring down unrelated parts of the service.</p>
<h2 id="heading-the-outbox-pattern">The Outbox Pattern</h2>
<p>The core idea is to stop treating the queue publish as a second write that happens after the database write, and instead make it part of the same database transaction.</p>
<p>Rather than calling <code>sqs.send</code> directly, the application inserts a row into an <code>outbox</code> table in the same transaction as the business record. A separate relay process reads the outbox table and publishes the messages to SQS. On the other end, a consumer receives the messages and writes to its own data store. In our case that is a fulfillment service writing to DynamoDB, completely separate from the order service's PostgreSQL database.</p>
<p>If the transaction rolls back for any reason, the outbox row disappears with it. There's no orphaned message in the queue because the message was never sent. If the application crashes after committing but before the relay runs, the outbox row is still there with <code>status='pending'</code>, and the relay will pick it up on its next iteration.</p>
<p>The only guarantee the pattern relies on is the one the database already provides: atomicity within a single transaction.</p>
<p>The relay worker is responsible for the eventual delivery guarantee. It runs on an interval, selects pending rows, publishes them to SQS, and marks them as sent only after SQS confirms receipt. If the relay crashes mid-run, it will reprocess the same rows on the next iteration, which means SQS may receive some messages more than once.</p>
<p>That's why the consumer needs to be <strong>idempotent</strong>: it must handle receiving the same message twice without creating duplicate fulfillment records. We'll cover how to implement that when we build the consumer.</p>
<p>This separation of concerns is what makes the pattern practical. The request handler commits one atomic database transaction and returns. The relay handles the network call to SQS asynchronously, at its own pace, with its own retry logic, without holding database connections open or blocking request handling. The consumer is fully decoupled from the order service and owns its own data store.</p>
<p>The diagram below illustrates the flow:</p>
<img src="https://cdn.hashnode.com/uploads/covers/68b08746916c71e1ed2db58e/ab0620f0-65c6-43f1-a406-00bfd4880cdc.svg" alt="Diagram: outbox pattern flow" style="display: block;" width="600" height="400" loading="lazy">

<h2 id="heading-what-well-build">What We'll Build</h2>
<p>Now let's see the whole thing in practice. We'll implement a simple order placement API. When a customer sends a request to place an order, the order service saves it to PostgreSQL and inserts a row into the outbox table, all in one atomic transaction. A relay worker wakes up periodically, reads the pending outbox rows, and publishes each one as a message to SQS. A separate fulfillment service receives those messages from the queue and creates fulfillment records in DynamoDB.</p>
<p>By the end, you'll have an HTTP endpoint you can call, and you'll be able to verify that placing an order triggers the creation of a fulfillment record in a completely separate database, owned by a completely separate service, without either service ever talking to the other directly.</p>
<p>You can find the complete working code at <a href="https://github.com/gkoos/article-outbox">github.com/gkoos/article-outbox</a>.</p>
<h2 id="heading-project-setup">Project Setup</h2>
<p>Before you can run any code, you need to get floci running so you have local instances of PostgreSQL, SQS, and DynamoDB. You'll also need Node.js 20 or later and Docker installed.</p>
<p>Start by cloning the repository and installing dependencies:</p>
<pre><code class="language-bash">git clone https://github.com/gkoos/article-outbox
cd article-outbox
npm install
</code></pre>
<p>Next, start floci. This command pulls the latest floci image and starts a Docker container that exposes a local AWS API endpoint (make sure Docker is running):</p>
<pre><code class="language-bash">npm run floci:start
</code></pre>
<p>On Linux and macOS, this just works. On Windows with Docker Desktop, <strong>you need to edit the</strong> <code>floci:start</code> <strong>script in your</strong> <code>package.json</code> <strong>to change the Docker socket mount from</strong> <code>/var/run/docker.sock</code> <strong>to</strong> <code>//var/run/docker.sock</code>.</p>
<p>The floci container is now listening on port 4566 and can spin up RDS (PostgreSQL), SQS, and DynamoDB instances on demand.</p>
<p>Now provision the AWS resources with a single setup command:</p>
<pre><code class="language-bash">npm run setup
</code></pre>
<p>This script creates an RDS PostgreSQL database instance, an SQS queue named <code>orders</code>, and a DynamoDB table named <code>fulfillments</code>. It waits for RDS to become available and then writes a <code>.env</code> file with the correct connection details. The environment variables <code>PG_PORT</code>, <code>SQS_QUEUE_URL</code>, and <code>DYNAMODB_TABLE_NAME</code> now point to the local emulated services.</p>
<p>Finally, create the PostgreSQL tables:</p>
<pre><code class="language-bash">npm run migrate
</code></pre>
<p>This creates the <code>orders</code> table and the <code>outbox</code> table in PostgreSQL. You now have a fully functional local environment ready to build against.</p>
<h2 id="heading-database-schema">Database Schema</h2>
<p>The two tables are simple. <code>orders</code> holds the business records: each order has a customer ID, an amount in cents, and a timestamp. The <code>outbox</code> table is the heart of the pattern: it's where the application writes the event that needs to be published.</p>
<pre><code class="language-sql">CREATE TABLE orders (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  customer_id TEXT NOT NULL,
  amount_cents INTEGER NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  created_at TIMESTAMPTZ DEFAULT now(),
  sent_at TIMESTAMPTZ
);

CREATE INDEX ON outbox (status, created_at) WHERE status = 'pending';
</code></pre>
<p>The <code>orders</code> table needs nothing special. The <code>outbox</code> table stores the event metadata: what type of event it is (<code>event_type</code>), what data it contains (<code>payload</code> as JSON), and whether it has been sent yet (<code>status</code>).</p>
<p>The status starts as <code>pending</code>. When the relay publishes it to SQS, it will mark it as <code>sent</code> and record the timestamp. The index on <code>(status, created_at) WHERE status = 'pending'</code> lets the relay quickly find the next batch of unsent events without scanning the entire table.</p>
<h2 id="heading-the-request-handler">The Request Handler</h2>
<p>This is where the pattern starts. The request handler receives an HTTP POST, inserts an order into the database, inserts a corresponding row into the outbox table, and commits everything in a single atomic transaction. The key insight is that neither write succeeds unless both succeed.</p>
<pre><code class="language-js">const client = await pool.connect();
try {
  await client.query('BEGIN');

  // Insert the order record
  const { rows } = await client.query(
    'INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2) RETURNING *',
    [customerId, amountCents]
  );
  const order = rows[0];

  // Insert the outbox record in the same transaction
  await client.query(
    `INSERT INTO outbox (event_type, payload)
     VALUES ($1, $2)`,
    ['order.created', JSON.stringify({ orderId: order.id, customerId: order.customer_id, amountCents: order.amount_cents, createdAt: order.created_at })],
  );

  await client.query('COMMIT');
  res.status(201).json(order);
} catch (err) {
  await client.query('ROLLBACK');
  next(err);
} finally {
  client.release();
}
</code></pre>
<p>The handler gets <code>customerId</code> and <code>amountCents</code> from the request body, starts an explicit transaction with <code>BEGIN</code>, and inserts the order. Then it inserts an outbox row with the order data as the payload.</p>
<p>Everything commits atomically. If anything fails, everything rolls back and the client gets an error. If the process crashes between the commit and the response, the client won't get a 201, but the order and the outbox row are still safely committed to the database and the relay will eventually pick it up. The handler doesn't call SQS at all. That is the relay's job.</p>
<h2 id="heading-the-relay-worker">The Relay Worker</h2>
<p>The relay worker is a separate process that polls the outbox table every second and publishes pending rows to SQS. It runs independently of the HTTP server and has no shared state with it.</p>
<pre><code class="language-js">async function relay() {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    const { rows } = await client.query(`
      SELECT *
      FROM outbox
      WHERE status = 'pending'
      ORDER BY created_at
      LIMIT 10
      FOR UPDATE SKIP LOCKED -- prevents multiple relays from processing the same rows
    `);

    for (const row of rows) {
      await sqsClient.send(new SendMessageCommand({
        QueueUrl: QUEUE_URL,
        MessageBody: JSON.stringify(row.payload),
        MessageAttributes: {
          EventType: { DataType: 'String', StringValue: row.event_type },
        },
      }));

      await client.query(
        `UPDATE outbox SET status = 'sent', sent_at = now() WHERE id = $1`,
        [row.id],
      );
    }

    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    console.error('Relay error:', err.message);
  } finally {
    client.release();
  }
}

setInterval(relay, 1000);
</code></pre>
<p><code>FOR UPDATE SKIP LOCKED</code> is the key to running multiple relay instances safely: when a relay picks up a batch of rows, it locks them. Any other relay instance trying to select the same rows will skip them and move to the next available ones, so you never get two relays publishing the same message from the same run.</p>
<p>The relay marks each row as <code>sent</code> only after <code>sqsClient.send</code> returns. If the relay crashes after sending to SQS but before updating the row, the row stays <code>pending</code> and the relay will resend it on the next iteration.</p>
<p>Note that the <code>UPDATE</code> happens inside the same transaction as the <code>SELECT FOR UPDATE</code>, so if the relay crashes mid-batch, the entire batch rolls back and all rows in it will be retried, including any that were already successfully sent to SQS.</p>
<p>The at-least-once delivery guarantee applies at the batch level, not the individual row level. You can read about this problem in <a href="https://blog.gaborkoos.com/posts/2026-07-01-Beyond-Happy-Path-Engineering-the-Network/">Beyond Happy Path Engineering: the Network</a>: when a response is lost, the caller can't know whether the operation succeeded, so it retries, and the receiver may see the same request twice. This means the consumer may see the same message more than once, which is why idempotency matters on the consumer side.</p>
<h2 id="heading-the-consumer">The Consumer</h2>
<p>The consumer is a completely separate service. It knows nothing about the order service's PostgreSQL database. Its only input is the SQS queue, and its only output is the DynamoDB <code>fulfillments</code> table. This is the point of the pattern: the two services are decoupled by the queue, and each owns its own data store.</p>
<p>As we saw earlier, because SQS delivers at least once (meaning a message might be delivered more than once), the consumer must be idempotent. The <code>PutItem</code> call uses a <code>ConditionExpression</code> that makes the write a no-op if a fulfillment record for that order already exists, so redelivered messages are handled safely.</p>
<pre><code class="language-js">async function consume() {
  const { Messages } = await sqsClient.send(new ReceiveMessageCommand({
    QueueUrl:              QUEUE_URL,
    WaitTimeSeconds:       20,   // long-poll: wait up to 20s for messages
    MaxNumberOfMessages:   10,
    MessageAttributeNames: ['All'],
  }));

  for (const msg of Messages ?? []) {
    const event = JSON.parse(msg.Body);

    try {
      await dynamoClient.send(new PutItemCommand({
        TableName: 'fulfillments',
        Item: {
          orderId:     { S: event.orderId },
          customerId:  { S: event.customerId },
          amountCents: { N: String(event.amountCents) },
          status:      { S: 'received' },
          createdAt:   { S: new Date().toISOString() },
        },
        ConditionExpression: 'attribute_not_exists(orderId)', // idempotency check
      }));
    } catch (err) {
      if (err.name !== 'ConditionalCheckFailedException') throw err;
      // already processed, safe to continue
    }

    // delete the message only after the write succeeds (or was already done)
    await sqsClient.send(new DeleteMessageCommand({
      QueueUrl:      QUEUE_URL,
      ReceiptHandle: msg.ReceiptHandle,
    }));
  }
}
</code></pre>
<p><code>ConditionExpression: 'attribute_not_exists(orderId)'</code> tells DynamoDB to reject the write if a record with that <code>orderId</code> already exists. When that happens, DynamoDB throws a <code>ConditionalCheckFailedException</code>. The consumer catches that specific error and ignores it, then deletes the message from the queue and moves on. Any other error is rethrown and the message stays in the queue to be retried.</p>
<p>The <code>DeleteMessage</code> call happens after the DynamoDB write, not before. If the process crashes between the write and the delete, SQS will redeliver the message and the condition check will handle it. If the process crashes before the write, the message stays in the queue and will be processed normally on the next delivery.</p>
<h2 id="heading-running-the-whole-thing">Running the Whole Thing</h2>
<p>With floci running and the resources provisioned, open three terminal tabs and start each process:</p>
<pre><code class="language-bash">node src/server.js    # the order API on port 3000
node src/relay.js     # the outbox relay
node src/consumer.js  # the fulfillment consumer
</code></pre>
<p>Now place an order:</p>
<pre><code class="language-bash">curl -X POST localhost:3000/orders \
  -H 'Content-Type: application/json' \
  -d '{"customerId":"c1","amountCents":4999}'
</code></pre>
<p>You should get back a 201 with the new order record:</p>
<pre><code class="language-bash">{"id":"1768d35b-083d-45f1-adb5-4063d8d7fcab","customer_id":"c1","amount_cents":4999,"created_at":"2026-07-30T20:27:10.628Z"}
</code></pre>
<p>Within a second the relay will pick up the outbox row and publish it to SQS. The consumer will receive the message and write a fulfillment record to DynamoDB. The repo includes a convenience script to verify this:</p>
<pre><code class="language-bash">npm run check
</code></pre>
<p>You should see a fulfillment record with the <code>orderId</code> from the order you just placed:</p>
<pre><code class="language-bash">{
  orderId: 'c335640e-bc4a-47e4-afed-484c95fbd6d3',
  customerId: 'c1',
  amountCents: '4999',
  status: 'received',
  createdAt: '2026-07-30T19:02:54.929Z'
}
</code></pre>
<h2 id="heading-going-to-production">Going to Production</h2>
<p>Because the local setup uses floci to emulate AWS, switching to real AWS requires no code changes at all. The AWS SDK reads the endpoint from <code>AWS_ENDPOINT_URL</code> in the environment. In production, you simply don't set that variable and the SDK talks to real AWS using the credentials and region from the standard environment variables (<code>AWS_REGION</code>, <code>AWS_ACCESS_KEY_ID</code>, <code>AWS_SECRET_ACCESS_KEY</code>, or an IAM role if you are running on EC2 or ECS).</p>
<p>Running multiple relay instances is safe out of the box because of <code>FOR UPDATE SKIP LOCKED</code>. You can scale the relay horizontally and each instance will pick up a different set of rows without duplicating messages.</p>
<p>One thing worth adding before going to production is handling permanent failures in the relay. Right now the relay only uses <code>pending</code> and <code>sent</code>. You should add a <code>failed</code> status and a retry counter: after a row has failed N times, mark it <code>failed</code> and stop retrying it. Then configure a dead-letter queue on the <code>orders</code> SQS queue as well, so that messages the consumer can't process after the maximum number of retries land somewhere you can inspect rather than disappearing silently.</p>
<p>For high-throughput systems where polling latency matters, <a href="https://en.wikipedia.org/wiki/Change_data_capture">change data capture</a> (CDC) is a common alternative to the polling relay. Tools like <a href="https://debezium.io/">Debezium</a> read directly from the PostgreSQL write-ahead log and publish changes to <a href="https://kafka.apache.org/">Kafka</a> or SQS without any polling delay. The outbox table and the consumer stay exactly the same, only the relay is replaced.</p>
<p>This is a bigger operational commitment than a polling worker, so polling is the right starting point for most systems.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>The dual-write problem is easy to overlook because the naïve implementation works correctly most of the time. It only fails in the gaps between two separate system writes, and those gaps only become visible when something goes wrong at exactly the wrong moment. By the time you notice it in production, data is already inconsistent and there is no clean way to recover.</p>
<p>The transactional outbox pattern closes that gap at the database level. The outbox row is part of the same atomic commit as the business record, so the two are always in sync. The relay handles the network call to SQS independently, with its own retry logic, without touching the request lifecycle. The consumer handles at-least-once delivery with a single condition check on the write.</p>
<p>Each piece is simple on its own, and together they give you reliable, decoupled event delivery without distributed transactions.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Implement HIPAA Technical Safeguards on AWS [Full Handbook] ]]>
                </title>
                <description>
                    <![CDATA[ Before I had ever heard the term "HIPAA audit", I spent three days helping a healthcare SaaS startup fix a single misconfigured S3 bucket. Not a breach — nothing was accessed. But the bucket was publi ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-implement-hipaa-technical-safeguards-on-aws-full-handbook/</link>
                <guid isPermaLink="false">6a71ec76a6c3ed946b473d26</guid>
                
                    <category>
                        <![CDATA[ healthcare ]]>
                    </category>
                
                    <category>
                        <![CDATA[ handbook ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Ayobami Adejumo ]]>
                </dc:creator>
                <pubDate>Tue, 04 Aug 2026 13:43:18 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/a83a4b13-be16-4bbf-904c-5fa81fdefd51.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Before I had ever heard the term "HIPAA audit", I spent three days helping a healthcare SaaS startup fix a single misconfigured S3 bucket. Not a breach — nothing was accessed. But the bucket was publicly listable, it contained patient appointment records, and the CEO had received a message from a security researcher at 11 PM on a Friday.</p>
<p>The fine never came. The legal fees did. The remediation work did. The reputational conversations with enterprise customers who asked pointed questions for the next six months definitely did.</p>
<p>HIPAA isn't abstract compliance overhead. It's a specific set of technical requirements that translate directly into infrastructure decisions. Get them right and you build a system that earns enterprise healthcare contracts. Get them wrong and you spend your fundraising runway on lawyers instead of engineers.</p>
<p>This handbook gives you the complete technical implementation: every safeguard mapped to its regulation clause, production-ready AWS infrastructure code, and the specific evidence each auditor will ask for. By the time you finish, you'll be able to answer every technical question in a HIPAA audit without looking anything up.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-part-1-understanding-hipaa-technical-safeguards">Part 1: Understanding HIPAA Technical Safeguards</a></p>
</li>
<li><p><a href="#heading-part-2-access-control-164312a1">Part 2: Access Control — §164.312(a)(1)</a></p>
</li>
<li><p><a href="#heading-part-3-audit-controls-164312b">Part 3: Audit Controls — §164.312(b)</a></p>
</li>
<li><p><a href="#heading-part-4-integrity-controls-164312c1">Part 4: Integrity Controls — §164.312(c)(1)</a></p>
</li>
<li><p><a href="#heading-part-5-transmission-security-164312e1">Part 5: Transmission Security — §164.312(e)(1)</a></p>
</li>
<li><p><a href="#heading-part-6-aws-network-architecture-for-hipaa">Part 6: AWS Network Architecture for HIPAA</a></p>
</li>
<li><p><a href="#heading-part-7-aws-services-covered-by-baa">Part 7: AWS Services Covered by BAA</a></p>
</li>
<li><p><a href="#heading-part-8-continuous-compliance-monitoring">Part 8: Continuous Compliance Monitoring</a></p>
</li>
<li><p><a href="#heading-part-9-the-pre-audit-checklist">Part 9: The Pre-Audit Checklist</a></p>
</li>
<li><p><a href="#heading-best-practices-summary">Best Practices Summary</a></p>
</li>
<li><p><a href="#heading-resources">Resources</a></p>
</li>
</ul>
<h2 id="heading-what-youll-learn">What You'll Learn</h2>
<ul>
<li><p>The five HIPAA Technical Safeguards and exactly which AWS infrastructure decisions each one governs</p>
</li>
<li><p>How to implement unique user identification and automatic logoff with production-ready code</p>
</li>
<li><p>How to build an immutable, tamper-evident audit log using hash chaining and S3 Object Lock</p>
</li>
<li><p>How to implement envelope encryption for ePHI fields using AWS KMS</p>
</li>
<li><p>The TLS configuration that satisfies HIPAA transmission security requirements</p>
</li>
<li><p>The complete VPC architecture that satisfies facility access control requirements</p>
</li>
<li><p>How to run automated HIPAA compliance scans and maintain continuous audit readiness</p>
</li>
<li><p>The specific evidence your auditor will request for each control</p>
</li>
</ul>
<p>Let's build it properly.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>Before following this guide, you should have:</p>
<p><strong>Knowledge:</strong></p>
<ul>
<li><p>Intermediate AWS experience — you've deployed applications on EC2 or ECS, worked with RDS, and understand VPCs and IAM roles</p>
</li>
<li><p>Comfort reading Python and Terraform HCL</p>
</li>
<li><p>Basic understanding of cryptography concepts — you know what symmetric encryption, asymmetric encryption, and hash functions are at a conceptual level</p>
</li>
</ul>
<p><strong>Legal prerequisite — sign the BAA first:</strong> Before writing a single line of HIPAA-related infrastructure code, your organisation must have a signed Business Associate Agreement (BAA) with AWS. You can accept the AWS BAA through the AWS Artifact console. Without a signed BAA, using AWS to process ePHI isn't HIPAA-compliant regardless of how well-engineered your technical controls are.</p>
<p><strong>Tools:</strong></p>
<ul>
<li><p>Terraform 1.5 or later</p>
</li>
<li><p>AWS CLI v2 configured</p>
</li>
<li><p>Python 3.10 or later with <code>boto3</code>, <code>cryptography</code>, and <code>pyjwt</code> installed</p>
</li>
</ul>
<p><strong>Important scope note:</strong> This guide covers the Technical Safeguards defined in 45 CFR §164.312. HIPAA compliance also requires Administrative Safeguards (§164.308) and Physical Safeguards (§164.310). The technical controls in this guide are necessary but not sufficient — they must be accompanied by documented policies, workforce training, and a formal risk assessment.</p>
<h2 id="heading-part-1-understanding-hipaa-technical-safeguards">Part 1: Understanding HIPAA Technical Safeguards</h2>
<h3 id="heading-11-what-the-five-safeguards-actually-require">1.1 What the Five Safeguards Actually Require</h3>
<p>HIPAA's Security Rule defines five categories of Technical Safeguards. Each maps to specific engineering decisions:</p>
<table>
<thead>
<tr>
<th>Safeguard</th>
<th>Regulation</th>
<th>What It Requires</th>
<th>AWS Implementation</th>
</tr>
</thead>
<tbody><tr>
<td>Access Control</td>
<td>§164.312(a)(1)</td>
<td>Unique user IDs, emergency access, auto-logoff, encryption</td>
<td>IAM, Cognito, KMS, session management</td>
</tr>
<tr>
<td>Audit Controls</td>
<td>§164.312(b)</td>
<td>Record and examine all ePHI access activity</td>
<td>CloudTrail, CloudWatch, Kinesis, S3 Object Lock</td>
</tr>
<tr>
<td>Integrity</td>
<td>§164.312(c)(1)</td>
<td>Prevent and detect improper alteration or destruction</td>
<td>Hash chaining, digital signatures, deletion protection</td>
</tr>
<tr>
<td>Transmission Security</td>
<td>§164.312(e)(1)</td>
<td>Encrypt ePHI during transmission</td>
<td>TLS 1.2+, mTLS, API Gateway, ALB policy</td>
</tr>
<tr>
<td>Facility Access</td>
<td>§164.310(a)(1)</td>
<td>Limit physical and logical access</td>
<td>VPC architecture, security groups, NACLs</td>
</tr>
</tbody></table>
<h3 id="heading-12-the-compliance-evidence-distinction">1.2 The Compliance-Evidence Distinction</h3>
<p>The most important concept in practical HIPAA engineering: compliance and evidence of compliance are different things, and auditors care about both.</p>
<p>Compliance means your encryption is correctly configured. Evidence of compliance means you have a CloudTrail log showing the KMS key rotation, a command output showing <code>StorageEncrypted: true</code> on every RDS instance, and a dated screenshot of the configuration that you can produce when asked.</p>
<p>Every section of this guide ends with an evidence collection command. Run each one. Save the outputs. Name the files with the date and the control they demonstrate. When your auditor asks "can you show me that your RDS instances are encrypted at rest?", you hand them a file rather than running a command in the room.</p>
<h3 id="heading-13-protected-health-information-know-your-scope">1.3 Protected Health Information — Know Your Scope</h3>
<p>HIPAA compliance begins with knowing what data in your system constitutes ePHI. The 18 HIPAA identifiers that must be protected:</p>
<pre><code class="language-python"># phi_classifier.py
# Reference: the 18 HIPAA identifiers

HIPAA_IDENTIFIERS = [
    'full_name', 'first_name', 'last_name',
    'geographic_subdivision',     # Any subdivision smaller than state
    'date_of_birth', 'admission_date', 'discharge_date', 'death_date',
    'phone_number', 'fax_number', 'email_address',
    'social_security_number', 'medical_record_number',
    'health_plan_beneficiary_number', 'account_number',
    'certificate_license_number', 'vehicle_identifier',
    'device_identifier', 'web_url', 'ip_address',
    'biometric_identifier', 'full_face_photo',
]

# Any data record containing one or more of these identifiers
# combined with health information is ePHI and subject to HIPAA.
</code></pre>
<h2 id="heading-part-2-access-control-164312a1">Part 2: Access Control — §164.312(a)(1)</h2>
<p>§164.312(a)(1) requires four specific implementation specifications: unique user identification, emergency access procedures, automatic logoff, and encryption and decryption.</p>
<h3 id="heading-21-unique-user-identification">2.1 Unique User Identification</h3>
<p>The requirement: every user with access to ePHI must have a unique identifier. Shared accounts violate this requirement. The reason this matters in practice: when an audit or security incident occurs, investigators need to know exactly who accessed which patient record and when. If three nurses share a login, that trail disappears. A unique identifier per user means every ePHI access event is attributable to a specific, named individual — which is what HIPAA's audit controls require and what your legal team will need if something goes wrong.</p>
<p>The code below implements this by assigning every user a UUID v4 at creation time — a randomly generated identifier that is unique across your entire system and is never reused, even after the user's account is deleted. When a user is removed, the account is soft-deleted: the UUID stays in the database and in all historical audit logs, so you can always reconstruct who accessed what. The authentication method also enforces MFA by default and tracks failed login attempts, automatically suspending accounts after five consecutive failures to satisfy HIPAA's requirements around access control and account management.</p>
<pre><code class="language-python"># user_identity_service.py
# HIPAA-compliant user identity management

import uuid
import hashlib
import hmac
import os
from datetime import datetime, timezone
from dataclasses import dataclass
from typing import Optional


@dataclass
class HIPAAUser:
    user_id:     str   # UUID — never reused
    email:       str
    role:        str   # Clinical / Administrative / Engineering
    department:  str
    status:      str   # ACTIVE / SUSPENDED / DELETED
    mfa_enabled: bool
    created_at:  str
    last_login:  Optional[str] = None
    failed_attempts: int = 0


class UserIdentityService:
    """
    Implements HIPAA §164.312(a)(1)(i) — Unique User Identification.

    Key properties:
    - Every user gets a UUID v4 that is unique and never reused
    - Accounts are soft-deleted — the UUID is preserved in audit logs
      permanently, even after the user leaves
    - All authentication events are logged with user_id, timestamp, and outcome
    """

    MAX_FAILED_ATTEMPTS = 5

    def __init__(self, db, audit_logger):
        self.db    = db
        self.audit = audit_logger

    def create_user(self, email: str, role: str, department: str,
                    created_by: str) -&gt; HIPAAUser:
        """Create a user with a unique, non-reusable identifier."""
        user = HIPAAUser(
            user_id=str(uuid.uuid4()),
            email=email.strip().lower(),
            role=role,
            department=department,
            status='ACTIVE',
            mfa_enabled=True,
            created_at=datetime.now(timezone.utc).isoformat(),
        )
        self.db.save_user(user)
        self.audit.log(
            event_type='USER_CREATED',
            actor_id=created_by,
            subject_id=user.user_id,
            details={'role': role, 'department': department}
        )
        return user

    def authenticate(self, email: str, password: str, mfa_token: str) -&gt; Optional[str]:
        """Authenticate a user. Returns a JWT access token on success."""
        user = self.db.find_user_by_email(email.strip().lower())

        if not user:
            self.audit.log(
                event_type='AUTH_FAILURE',
                actor_id=None,
                subject_id=None,
                details={'reason': 'unknown_email',
                         'email_hash': hashlib.sha256(email.encode()).hexdigest()[:16]}
            )
            return None

        if user.status != 'ACTIVE':
            self.audit.log(
                event_type='AUTH_FAILURE',
                actor_id=user.user_id,
                subject_id=user.user_id,
                details={'reason': f'account_{user.status.lower()}'}
            )
            return None

        if not self._verify_password(password, self.db.get_password_hash(user.user_id)):
            user.failed_attempts += 1
            if user.failed_attempts &gt;= self.MAX_FAILED_ATTEMPTS:
                user.status = 'SUSPENDED'
                self.audit.log(
                    event_type='ACCOUNT_SUSPENDED',
                    actor_id='system',
                    subject_id=user.user_id,
                    details={'reason': 'max_failed_attempts',
                             'attempts': user.failed_attempts}
                )
            self.db.save_user(user)
            return None

        user.failed_attempts = 0
        user.last_login = datetime.now(timezone.utc).isoformat()
        self.db.save_user(user)

        token = self._issue_jwt(user)
        self.audit.log(
            event_type='AUTH_SUCCESS',
            actor_id=user.user_id,
            subject_id=user.user_id,
            details={'token_jti': self._extract_jti(token)}
        )
        return token

    @staticmethod
    def _verify_password(plaintext: str, stored_hash: str) -&gt; bool:
        candidate = hashlib.pbkdf2_hmac(
            'sha256', plaintext.encode(), b'salt', 100_000
        ).hex()
        return hmac.compare_digest(candidate, stored_hash)

    def _issue_jwt(self, user: HIPAAUser) -&gt; str:
        import jwt
        return jwt.encode(
            {
                'sub':  user.user_id,
                'role': user.role,
                'jti':  str(uuid.uuid4()),
                'exp':  int(datetime.now(timezone.utc).timestamp()) + 900,
                'iat':  int(datetime.now(timezone.utc).timestamp()),
            },
            os.environ['JWT_PRIVATE_KEY'],
            algorithm='RS256'
        )

    @staticmethod
    def _extract_jti(token: str) -&gt; str:
        import jwt
        return jwt.decode(token, options={'verify_signature': False})['jti']
</code></pre>
<p>The two evidence queries below confirm to an auditor that your system enforces uniqueness: the first shows that every <code>user_id</code> in the database is distinct (no duplicates, no shared credentials), and the second shows that every active user has MFA enabled — both of which are specific requirements auditors check for under this clause.</p>
<p>Evidence for auditors — §164.312(a)(1)(i):</p>
<pre><code class="language-bash"># Show that no shared accounts exist
psql $DATABASE_URL -c "
    SELECT COUNT(*) AS total_users,
           COUNT(DISTINCT user_id) AS unique_ids,
           COUNT(CASE WHEN status='ACTIVE' THEN 1 END) AS active_users
    FROM hipaa_users;
"

# Show MFA is enabled for all active users — expected: 0
psql $DATABASE_URL -c "
    SELECT COUNT(*) FROM hipaa_users
    WHERE status = 'ACTIVE' AND mfa_enabled = FALSE;
"
</code></pre>
<h3 id="heading-22-automatic-logoff-164312a2iii">2.2 Automatic Logoff — §164.312(a)(2)(iii)</h3>
<p>The requirement: implement electronic procedures that terminate an electronic session after a predetermined time of inactivity. Most healthcare applications use 15 minutes.</p>
<p>The reason for this control is straightforward: clinical environments involve shared workstations. A nurse logs in to check a patient record, gets called away, and leaves the browser open. Without automatic logoff, the next person to sit at that workstation has full access to ePHI under someone else's credentials. The control is about protecting against the reality of how healthcare teams actually work, not just against malicious actors.</p>
<p>The code below implements automatic logoff using short-lived JWTs. When a user authenticates, they receive a token that expires after 15 minutes of inactivity — each API call implicitly resets that window by issuing a new token. There's also an absolute 8-hour session ceiling: regardless of activity, a user must re-authenticate after eight hours. This prevents a session from staying open indefinitely if a user simply leaves a tab running in the background. The <code>validate_session</code> decorator is applied to every endpoint that touches ePHI, so no access path can bypass these checks.</p>
<pre><code class="language-python"># session_manager.py
# Implements §164.312(a)(2)(iii) — Automatic Logoff

import os
import uuid
from datetime import datetime, timezone
from functools import wraps
from flask import request, g, jsonify
import jwt

INACTIVITY_TIMEOUT_SECONDS = 15 * 60    # 15 minutes
ABSOLUTE_SESSION_SECONDS   = 8 * 60 * 60  # 8 hours maximum


def validate_session(f):
    """Decorator for endpoints that access ePHI."""
    @wraps(f)
    def decorated(*args, **kwargs):
        auth_header = request.headers.get('Authorization', '')
        if not auth_header.startswith('Bearer '):
            return jsonify({'error': 'MISSING_TOKEN'}), 401

        token = auth_header[7:]

        try:
            payload = jwt.decode(
                token,
                os.environ['JWT_PUBLIC_KEY'],
                algorithms=['RS256']
            )
        except jwt.ExpiredSignatureError:
            return jsonify({
                'error':   'SESSION_EXPIRED',
                'message': 'Your session has expired due to inactivity. Please log in again.',
                'code':    'INACTIVITY_TIMEOUT'
            }), 401
        except jwt.InvalidTokenError as e:
            return jsonify({'error': 'INVALID_TOKEN', 'detail': str(e)}), 401

        # Check absolute session age
        issued_at   = payload.get('session_start', payload['iat'])
        session_age = datetime.now(timezone.utc).timestamp() - issued_at

        if session_age &gt; ABSOLUTE_SESSION_SECONDS:
            return jsonify({
                'error':   'SESSION_EXPIRED',
                'message': 'Your session has exceeded the 8-hour limit. Please log in again.',
                'code':    'ABSOLUTE_TIMEOUT'
            }), 401

        g.user_id = payload['sub']
        g.role    = payload.get('role')
        g.jti     = payload.get('jti')
        return f(*args, **kwargs)

    return decorated


def issue_access_token(user_id: str, role: str, session_start: int = None) -&gt; str:
    """Issue a 15-minute access token with an 8-hour absolute session limit."""
    now = int(datetime.now(timezone.utc).timestamp())
    return jwt.encode(
        {
            'sub':           user_id,
            'role':          role,
            'jti':           str(uuid.uuid4()),
            'iat':           now,
            'exp':           now + INACTIVITY_TIMEOUT_SECONDS,
            'session_start': session_start or now,
        },
        os.environ['JWT_PRIVATE_KEY'],
        algorithm='RS256'
    )
</code></pre>
<h3 id="heading-23-encryption-at-rest-164312a2iv">2.3 Encryption at Rest — §164.312(a)(2)(iv)</h3>
<p>The requirement: implement a mechanism to encrypt and decrypt ePHI.</p>
<p>Encryption at rest means that if someone gains physical access to your storage media — a hard drive, a backup tape, an S3 object — they can't read the data without the encryption key. On AWS, this protection comes in two layers. The first layer is storage-level encryption, where the database or storage service automatically encrypts every byte written to disk. The second, more powerful layer is field-level encryption, where individual sensitive values are encrypted by your application before they're even handed to the database — so a database administrator with full SQL access still can't read patient SSNs or diagnoses without the application key.</p>
<p>Layer 1 — storage encryption (Terraform):</p>
<p>The Terraform below provisions an RDS instance with a customer-managed KMS key. Using a customer-managed key rather than the AWS default key matters for two reasons: it gives you proof of key ownership (auditors will ask for the KMS key ARN), and it enables automatic key rotation, which replaces the cryptographic material annually without any disruption to your application. The <code>enable_key_rotation = true</code> setting automates this entirely — you don't need to touch the configuration again, and the key stays current.</p>
<pre><code class="language-hcl"># rds_hipaa.tf — HIPAA-compliant RDS with customer-managed KMS key

resource "aws_kms_key" "rds" {
  description             = "Customer-managed KMS key for RDS ePHI encryption"
  enable_key_rotation     = true
  deletion_window_in_days = 30

  tags = {
    Purpose     = "HIPAA-ePHI-encryption"
    Environment = "production"
    Control     = "164.312(a)(2)(iv)"
  }
}

resource "aws_db_instance" "hipaa_postgres" {
  identifier     = "hipaa-production-db"
  engine         = "postgres"
  engine_version = "15.4"
  instance_class = "db.r7g.large"

  storage_encrypted = true
  kms_key_id        = aws_kms_key.rds.arn

  backup_retention_period = 30
  deletion_protection     = true
  skip_final_snapshot     = false
  final_snapshot_identifier = "hipaa-production-db-final-snapshot"

  db_subnet_group_name   = aws_db_subnet_group.hipaa.name
  vpc_security_group_ids = [aws_security_group.rds.id]
  publicly_accessible    = false

  tags = {
    DataClassification = "ePHI"
    HIPAAControl       = "164.312(a)(2)(iv)"
  }
}
</code></pre>
<p>Layer 2 — field-level application encryption for highest-sensitivity data:</p>
<p>Storage encryption protects you if someone steals a disk. Field-level encryption protects you from authorized users who have legitimate database access but shouldn't be able to read raw patient data. The <code>FieldEncryption</code> class below implements envelope encryption: AWS KMS generates a unique data key for each field value, that key is used to encrypt the plaintext, and only the encrypted version of the key is stored. Even if an attacker extracts your entire database, every encrypted field requires a separate KMS API call to decrypt — which is logged, rate-limited, and requires the correct IAM permissions. The encryption context ties each ciphertext to its purpose and owner, so a key decrypted for one patient's record can't be reused for another.</p>
<pre><code class="language-python"># field_encryption.py
# Envelope encryption using AWS KMS

import boto3
import base64
from cryptography.fernet import Fernet
from typing import Optional

kms     = boto3.client('kms')
KMS_KEY = 'alias/hipaa-rds-ephi'


class FieldEncryption:
    """
    Envelope encryption for ePHI fields.

    How it works:
    1. AWS KMS generates a data key (plaintext + encrypted copy)
    2. The plaintext data key encrypts the field value using Fernet (AES-128-CBC)
    3. Only the encrypted data key is stored alongside the ciphertext
    4. To decrypt: KMS decrypts the stored data key, then Fernet decrypts the value
    5. A database admin with direct SQL access sees only base64 ciphertext
    """

    def encrypt(self, plaintext: str, context: dict) -&gt; Optional[dict]:
        if not plaintext:
            return None

        data_key = kms.generate_data_key(
            KeyId=KMS_KEY,
            KeySpec='AES_256',
            EncryptionContext=context
        )

        fernet    = Fernet(data_key['Plaintext'])
        ciphertext = fernet.encrypt(plaintext.encode('utf-8'))

        return {
            'ciphertext':         base64.b64encode(ciphertext).decode(),
            'encrypted_data_key': base64.b64encode(data_key['CiphertextBlob']).decode(),
            'encryption_context': context,
        }

    def decrypt(self, payload: dict) -&gt; Optional[str]:
        if not payload:
            return None

        decrypted_key = kms.decrypt(
            CiphertextBlob=base64.b64decode(payload['encrypted_data_key']),
            EncryptionContext=payload['encryption_context']
        )

        fernet    = Fernet(decrypted_key['Plaintext'])
        plaintext = fernet.decrypt(base64.b64decode(payload['ciphertext']))
        return plaintext.decode('utf-8')
</code></pre>
<p>Evidence for auditors — §164.312(a)(2)(iv):</p>
<pre><code class="language-bash"># Verify RDS storage encryption
aws rds describe-db-instances \
  --db-instance-identifier hipaa-production-db \
  --query 'DBInstances[0].{Encrypted:StorageEncrypted,KMSKey:KmsKeyId}' \
  --output table

# Verify KMS key rotation is enabled — expected: true
aws kms get-key-rotation-status \
  --key-id alias/hipaa-rds-ephi \
  --query 'KeyRotationEnabled'
</code></pre>
<h2 id="heading-part-3-audit-controls-164312b">Part 3: Audit Controls — §164.312(b)</h2>
<p>§164.312(b) requires implementing hardware, software, and procedural mechanisms that record and examine activity in information systems that contain or use ePHI. Every access. Every modification. Every deletion. Logged, immutable, and retainable for six years.</p>
<h3 id="heading-31-the-audit-log-schema">3.1 The Audit Log Schema</h3>
<p>Every ePHI-related event must answer five questions: who did it, what did they do, when did they do it, to which record, and from where.</p>
<p>The <code>log_ephi_event</code> function below is the central audit mechanism for your application. Every time a user reads, updates, or deletes a patient record, this function is called before the response is returned. It captures the actor's identity, IP address, browser, and session ID alongside the action, resource, and timestamp — and then adds something more powerful: a cryptographic chain. Each log entry includes the SHA-256 hash of the previous entry. This means if anyone tampers with a log entry — even a single character — every subsequent hash in the chain becomes invalid, making tampering detectable. The entries are then streamed to Kinesis, which fans them out to S3 for long-term storage. One critical rule: the <code>details</code> dictionary must never contain ePHI values, only field names. Log that a SSN field was accessed, not what the SSN was.</p>
<pre><code class="language-python"># audit_logger.py
# Implements §164.312(b) — Audit Controls

import hashlib
import json
import uuid
import boto3
from datetime import datetime, timezone
from typing import Any, Optional

kinesis = boto3.client('kinesis')
STREAM  = 'hipaa-audit-events'

_last_hash = '0' * 64  # Chain starts with 64 zeros


def log_ephi_event(
    event_type:  str,
    actor_id:    Optional[str],
    patient_id:  Optional[str],
    resource:    Optional[str],
    action:      str,
    details:     dict,
    request_ctx: dict = None,
) -&gt; str:
    """
    Log a HIPAA-relevant event.
    Never include PHI values in the details dict — field names only.
    """
    global _last_hash

    entry = {
        'actor_id':       actor_id,
        'actor_ip':       (request_ctx or {}).get('ip'),
        'actor_ua':       (request_ctx or {}).get('user_agent'),
        'session_id':     (request_ctx or {}).get('session_id'),
        'event_type':     event_type,
        'action':         action,
        'resource':       resource,
        'details':        details,
        'patient_id':     patient_id,
        'timestamp':      datetime.now(timezone.utc).isoformat(),
        'service':        'healthcare-api',
        'environment':    'production',
        'previous_hash':  _last_hash,
        'log_id':         str(uuid.uuid4()),
    }

    canonical   = json.dumps(entry, sort_keys=True)
    entry_hash  = hashlib.sha256(canonical.encode()).hexdigest()
    entry['log_hash'] = entry_hash
    _last_hash  = entry_hash

    kinesis.put_record(
        StreamName=STREAM,
        Data=json.dumps(entry),
        PartitionKey=actor_id or 'system'
    )

    return entry_hash


def log_phi_read(actor_id: str, patient_id: str, resource: str,
                 purpose: str, request_ctx: dict = None):
    return log_ephi_event(
        event_type='PHI_ACCESS',
        actor_id=actor_id,
        patient_id=patient_id,
        resource=resource,
        action='READ',
        details={'purpose': purpose},
        request_ctx=request_ctx,
    )


def log_phi_update(actor_id: str, patient_id: str, resource: str,
                   fields_changed: list, request_ctx: dict = None):
    return log_ephi_event(
        event_type='PHI_UPDATE',
        actor_id=actor_id,
        patient_id=patient_id,
        resource=resource,
        action='UPDATE',
        details={'fields_changed': fields_changed},  # Field names only — NOT values
        request_ctx=request_ctx,
    )
</code></pre>
<h3 id="heading-32-immutable-log-storage-with-s3-object-lock">3.2 Immutable Log Storage with S3 Object Lock</h3>
<p>Writing logs to S3 isn't enough on its own — logs stored in a standard S3 bucket can be deleted, which would let someone cover their tracks after a breach. S3 Object Lock in COMPLIANCE mode solves this by making every object in the bucket permanently immutable for the retention period you specify. In COMPLIANCE mode, not even the AWS root account can delete the objects before the retention period expires. The bucket below is configured with a 2,190-day (six-year) retention period, which satisfies HIPAA's documentation retention requirement. Versioning is also enabled so that even if a write operation partially overwrites an object, the original version is preserved.</p>
<pre><code class="language-hcl"># audit_log_bucket.tf
# S3 bucket with Object Lock in COMPLIANCE mode
# Logs cannot be deleted or modified by anyone — including root

resource "aws_s3_bucket" "audit_logs" {
  bucket              = "hipaa-audit-logs-${data.aws_caller_identity.current.account_id}"
  object_lock_enabled = true

  tags = {
    DataClassification = "audit-log"
    HIPAAControl       = "164.312(b)"
    RetentionYears     = "6"
  }
}

resource "aws_s3_bucket_versioning" "audit_logs" {
  bucket = aws_s3_bucket.audit_logs.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_object_lock_configuration" "audit_logs" {
  bucket = aws_s3_bucket.audit_logs.id
  rule {
    default_retention {
      mode = "COMPLIANCE"  # Nobody can delete — not even root
      days = 2190          # 6 years = 2,190 days
    }
  }
}

resource "aws_s3_bucket_public_access_block" "audit_logs" {
  bucket                  = aws_s3_bucket.audit_logs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
</code></pre>
<p>Evidence for auditors — §164.312(b):</p>
<pre><code class="language-bash"># Verify Object Lock is enabled in COMPLIANCE mode
aws s3api get-object-lock-configuration \
  --bucket hipaa-audit-logs-YOUR_ACCOUNT_ID \
  --query 'ObjectLockConfiguration'
# Expected: Mode=COMPLIANCE, Days=2190
</code></pre>
<h2 id="heading-part-4-integrity-controls-164312c1">Part 4: Integrity Controls — §164.312(c)(1)</h2>
<p>§164.312(c)(1) requires implementing policies and procedures to protect ePHI from improper alteration or destruction.</p>
<p>The integrity control solves a specific problem: how do you know a patient record hasn't been modified after it was written? Storage encryption protects data from being read by unauthorised parties, but it doesn't protect against an authorised user — a database administrator, a compromised internal account — silently editing a record. Digital signatures do. When a record is created, <code>sign_record</code> produces a cryptographic signature using a KMS asymmetric key. That signature is stored alongside the record. The <code>verify_record</code> function can then confirm at any point that the record's content exactly matches what was signed — any modification, even a single character, produces a different signature that fails verification. The weekly integrity scan calls <code>verify_record</code> on every ePHI record in the specified table and fires an SNS alert for any that fail, creating a continuous tamper-detection mechanism.</p>
<pre><code class="language-python"># integrity_service.py
# Digitally signs each ePHI record using KMS asymmetric key

import boto3
import base64
import json
from datetime import datetime, timezone

kms         = boto3.client('kms')
SIGNING_KEY = 'alias/hipaa-record-signing'


def sign_record(record: dict) -&gt; str:
    """Digitally sign a patient record. Store the signature alongside the record."""
    canonical = json.dumps(record, sort_keys=True)
    response  = kms.sign(
        KeyId=SIGNING_KEY,
        Message=canonical.encode(),
        MessageType='RAW',
        SigningAlgorithm='RSASSA_PKCS1_V1_5_SHA_256'
    )
    return base64.b64encode(response['Signature']).decode()


def verify_record(record: dict, signature: str) -&gt; bool:
    """Verify a patient record hasn't been altered since signing."""
    canonical = json.dumps(record, sort_keys=True)
    try:
        kms.verify(
            KeyId=SIGNING_KEY,
            Message=canonical.encode(),
            MessageType='RAW',
            Signature=base64.b64decode(signature),
            SigningAlgorithm='RSASSA_PKCS1_V1_5_SHA_256'
        )
        return True
    except kms.exceptions.KMSInvalidSignatureException:
        return False


def weekly_integrity_scan(db, table_name: str) -&gt; dict:
    """
    Scheduled job: verify the digital signature on every ePHI record.
    Any record that fails verification is flagged as potentially tampered.
    Run weekly as required by §164.312(c)(1) policy.
    """
    total    = 0
    failures = []

    for record_id, record, signature in db.iterate_records_with_signatures(table_name):
        total += 1
        if not verify_record(record, signature):
            failures.append({
                'record_id':   record_id,
                'table':       table_name,
                'detected_at': datetime.now(timezone.utc).isoformat(),
            })

    result = {
        'scan_date':          datetime.now(timezone.utc).isoformat(),
        'table':              table_name,
        'records_checked':    total,
        'integrity_failures': len(failures),
        'failed_records':     failures,
    }

    if failures:
        sns = boto3.client('sns')
        sns.publish(
            TopicArn='arn:aws:sns:us-east-1:YOUR_ACCOUNT:hipaa-integrity-alerts',
            Subject=f'INTEGRITY FAILURE: {len(failures)} records in {table_name}',
            Message=json.dumps(result, indent=2)
        )

    return result
</code></pre>
<h2 id="heading-part-5-transmission-security-164312e1">Part 5: Transmission Security — §164.312(e)(1)</h2>
<p>§164.312(e)(1) requires protecting ePHI during transmission by implementing technical security measures to guard against unauthorized access. The minimum TLS version required is TLS 1.2. TLS 1.3 is recommended. SSL, TLS 1.0, and TLS 1.1 are not acceptable.</p>
<p>Application Load Balancer SSL policy (Terraform):</p>
<p>The ALB is the entry point for all external traffic to your HIPAA application, so it's the first place to enforce TLS requirements. The Terraform resource below configures the HTTPS listener with the <code>ELBSecurityPolicy-TLS13-1-2-2021-06</code> policy — this is AWS's policy name for a configuration that accepts TLS 1.2 and TLS 1.3 connections while rejecting all older protocols and weak cipher suites. The HTTP listener is configured separately to redirect all port 80 traffic to port 443 with a permanent 301 redirect, ensuring no ePHI can ever be transmitted unencrypted even if a client accidentally connects over HTTP.</p>
<pre><code class="language-hcl"># alb_hipaa.tf

resource "aws_alb_listener" "hipaa_https" {
  load_balancer_arn = aws_alb.hipaa.arn
  port              = 443
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = aws_acm_certificate.hipaa.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_alb_target_group.hipaa_api.arn
  }
}

# Redirect all HTTP traffic to HTTPS
resource "aws_alb_listener" "hipaa_http_redirect" {
  load_balancer_arn = aws_alb.hipaa.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type = "redirect"
    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"
    }
  }
}
</code></pre>
<p>nginx TLS configuration for direct deployments:</p>
<p>If your application servers handle TLS termination directly — rather than offloading to the ALB — the nginx configuration below enforces the same standards at the server level. The <code>ssl_protocols</code> directive explicitly lists only TLSv1.2 and TLSv1.3, which means nginx will reject any connection attempt using an older protocol. The <code>ssl_ciphers</code> list specifies only ECDHE-based cipher suites with AES-GCM or ChaCha20-Poly1305 — these provide forward secrecy, meaning that even if your private key is later compromised, past session recordings can't be decrypted. The <code>Strict-Transport-Security</code> header with a two-year max-age instructs browsers to always use HTTPS for this domain, even if a user types the HTTP URL. <code>ssl_session_tickets off</code> prevents a class of attack where session ticket keys could be used to decrypt past sessions.</p>
<pre><code class="language-nginx"># /etc/nginx/conf.d/hipaa-tls.conf

server {
    listen 443 ssl http2;
    server_name api.your-healthcare-app.com;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
    ssl_prefer_server_ciphers off;

    # HTTP Strict Transport Security — 2 years
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    ssl_stapling        on;
    ssl_stapling_verify on;
    ssl_session_tickets off;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;
}
</code></pre>
<p>Evidence for auditors — §164.312(e)(1):</p>
<pre><code class="language-bash"># Verify TLS 1.1 is rejected
openssl s_client \
  -connect api.your-healthcare-app.com:443 \
  -tls1_1 2&gt;&amp;1 | grep -E "CONNECTED|handshake failure"
# Expected: handshake failure

# Verify TLS 1.2 succeeds
openssl s_client \
  -connect api.your-healthcare-app.com:443 \
  -tls1_2 2&gt;&amp;1 | grep "CONNECTED"

# Check ALB SSL policy
aws elbv2 describe-listeners \
  --load-balancer-arn YOUR_ALB_ARN \
  --query 'Listeners[*].{Port:Port,SslPolicy:SslPolicy}' \
  --output table
</code></pre>
<h2 id="heading-part-6-aws-network-architecture-for-hipaa">Part 6: AWS Network Architecture for HIPAA</h2>
<p>§164.310(a)(1) (Physical Facility Access Controls) is interpreted in cloud environments as logical network access control — the VPC architecture that isolates ePHI processing from other workloads.</p>
<p>The three-tier VPC below implements network segmentation as a hard boundary around ePHI. The public subnets hold only load balancers — nothing that processes or stores patient data is publicly reachable. The private app subnets hold your API servers, which can receive traffic from the load balancers but have no direct internet path in or out. The private data subnets hold RDS and <code>ElastiCache</code>, which can only receive traffic from the app tier's security group — not from the internet, not from the public subnets, and not from any other source. This means a compromised load balancer cannot directly reach the database: it can only reach the application servers, which apply their own authentication layer before talking to the database.</p>
<p>The VPC endpoints for S3 and KMS ensure that ePHI-related traffic to those services travels through AWS's internal network rather than the public internet. The VPC Flow Logs capture all accepted and rejected network traffic, which gives you the network-level audit trail that complements your application-level audit logs.</p>
<pre><code class="language-hcl"># vpc_hipaa.tf — Three-tier VPC architecture

resource "aws_vpc" "hipaa" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name         = "hipaa-production-vpc"
    DataClass    = "ePHI"
    HIPAAControl = "164.310(a)(1)"
  }
}

# Public subnets — load balancers only, no ePHI
resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.hipaa.id
  cidr_block        = "10.0.${count.index + 1}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {Name = "hipaa-public-${count.index + 1}", DataClass = "none"}
}

# Private app subnets — API servers, no direct internet access
resource "aws_subnet" "private_app" {
  count             = 2
  vpc_id            = aws_vpc.hipaa.id
  cidr_block        = "10.0.${count.index + 10}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {Name = "hipaa-private-app-${count.index + 1}", DataClass = "ePHI-processing"}
}

# Private data subnets — RDS, ElastiCache
resource "aws_subnet" "private_data" {
  count             = 2
  vpc_id            = aws_vpc.hipaa.id
  cidr_block        = "10.0.${count.index + 20}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {Name = "hipaa-private-data-${count.index + 1}", DataClass = "ePHI-storage"}
}

# VPC endpoints — AWS services without internet traversal
# ePHI must not traverse the public internet even within AWS
resource "aws_vpc_endpoint" "s3" {
  vpc_id          = aws_vpc.hipaa.id
  service_name    = "com.amazonaws.${var.region}.s3"
  route_table_ids = [aws_route_table.private.id]
}

resource "aws_vpc_endpoint" "kms" {
  vpc_id              = aws_vpc.hipaa.id
  service_name        = "com.amazonaws.${var.region}.kms"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private_app[*].id
  security_group_ids  = [aws_security_group.vpce.id]
  private_dns_enabled = true
}

# Security groups — least-privilege access
resource "aws_security_group" "rds" {
  name   = "hipaa-rds-sg"
  vpc_id = aws_vpc.hipaa.id

  ingress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id]
    description     = "PostgreSQL from app tier only — no direct external access"
  }
}

# VPC Flow Logs — network audit trail
resource "aws_flow_log" "hipaa" {
  vpc_id          = aws_vpc.hipaa.id
  traffic_type    = "ALL"
  iam_role_arn    = aws_iam_role.flow_logs.arn
  log_destination = aws_cloudwatch_log_group.vpc_flow_logs.arn

  tags = {HIPAAControl = "164.310(a)(1)", Retention = "365-days"}
}
</code></pre>
<h2 id="heading-part-7-aws-services-covered-by-baa">Part 7: AWS Services Covered by BAA</h2>
<p>Not every AWS service is covered by the AWS Business Associate Agreement. Using a non-BAA service to process ePHI is a HIPAA violation.</p>
<table>
<thead>
<tr>
<th>AWS Service</th>
<th>BAA Covered</th>
<th>Notes</th>
</tr>
</thead>
<tbody><tr>
<td>EC2</td>
<td>Yes</td>
<td>Encrypt EBS volumes at creation</td>
</tr>
<tr>
<td>RDS (all engines)</td>
<td>Yes</td>
<td>Enable storage encryption — not default</td>
</tr>
<tr>
<td>S3</td>
<td>Yes</td>
<td>Enforce encryption in bucket policy. Block public access</td>
</tr>
<tr>
<td>Lambda</td>
<td>Yes</td>
<td>Environment variables must not contain PHI values</td>
</tr>
<tr>
<td>EKS</td>
<td>Yes</td>
<td>Encrypt etcd. Use private cluster endpoint</td>
</tr>
<tr>
<td>API Gateway</td>
<td>Yes</td>
<td>Enable CloudTrail logging</td>
</tr>
<tr>
<td>KMS</td>
<td>Yes</td>
<td>Required for all encryption in this guide</td>
</tr>
<tr>
<td>CloudTrail</td>
<td>Yes</td>
<td>Enable in all regions, encrypt logs</td>
</tr>
<tr>
<td>CloudWatch Logs</td>
<td>Yes</td>
<td>Encrypt log groups. Logs may contain ePHI</td>
</tr>
<tr>
<td>Kinesis Data Streams</td>
<td>Yes</td>
<td>Used for audit log fan-out</td>
</tr>
<tr>
<td>SNS</td>
<td>Yes</td>
<td>Encrypt topics</td>
</tr>
<tr>
<td>SQS</td>
<td>Yes</td>
<td>Encrypt queues</td>
</tr>
<tr>
<td>Secrets Manager</td>
<td>Yes</td>
<td>Preferred for rotating credentials</td>
</tr>
</tbody></table>
<p>Services not covered by default BAA — do not use for ePHI: Amazon Connect (requires separate agreement), some Amazon Comprehend Medical features (check current BAA), and third-party marketplace products.</p>
<h2 id="heading-part-8-continuous-compliance-monitoring">Part 8: Continuous Compliance Monitoring</h2>
<p>HIPAA compliance isn't a state you achieve once — it's a condition you maintain continuously. Configuration drift is one of the most common causes of HIPAA findings in audits: an engineer spins up a new RDS instance without encryption, a developer creates an S3 bucket without blocking public access, a log group accumulates without a retention policy. None of these are malicious. They're the normal entropy of a growing engineering team.</p>
<p>The scanner below is designed to run as a daily Lambda function. It checks your AWS account against the most common HIPAA technical control failures and writes structured findings to S3 as a dated evidence file. Each finding maps to a specific regulation clause, has a severity level (CRITICAL or HIGH), and names the exact resource that's out of compliance. Running this daily means you catch drift within 24 hours rather than discovering it during an audit.</p>
<pre><code class="language-python"># compliance_scanner.py
# Daily Lambda job — runs all HIPAA compliance checks

import boto3
import json
from datetime import datetime, timezone

ec2 = boto3.client('ec2')
rds = boto3.client('rds')
s3  = boto3.client('s3')
ct  = boto3.client('cloudtrail')
gd  = boto3.client('guardduty')


def scan_all() -&gt; dict:
    """Run all HIPAA compliance checks. Returns structured findings."""
    findings = []

    # §164.312(a)(2)(iv) — Check: all RDS instances encrypted
    for inst in rds.describe_db_instances()['DBInstances']:
        if not inst.get('StorageEncrypted'):
            findings.append({
                'control':  '164.312(a)(2)(iv)',
                'severity': 'CRITICAL',
                'resource': inst['DBInstanceIdentifier'],
                'finding':  'RDS instance not encrypted at rest',
            })

    # §164.312(a)(2)(iv) — Check: all EBS volumes encrypted
    for vol in ec2.describe_volumes()['Volumes']:
        if not vol.get('Encrypted'):
            findings.append({
                'control':  '164.312(a)(2)(iv)',
                'severity': 'HIGH',
                'resource': vol['VolumeId'],
                'finding':  'EBS volume not encrypted',
            })

    # §164.312(a)(2)(iv) — Check: S3 buckets block public access
    for bucket in s3.list_buckets()['Buckets']:
        name = bucket['Name']
        try:
            pab = s3.get_public_access_block(Bucket=name)[
                'PublicAccessBlockConfiguration'
            ]
            if not all([pab.get('BlockPublicAcls'), pab.get('BlockPublicPolicy'),
                        pab.get('IgnorePublicAcls'), pab.get('RestrictPublicBuckets')]):
                findings.append({
                    'control':  '164.312(a)(2)(iv)',
                    'severity': 'CRITICAL',
                    'resource': f's3://{name}',
                    'finding':  'S3 bucket public access not fully blocked',
                })
        except s3.exceptions.NoSuchPublicAccessBlockConfiguration:
            findings.append({
                'control':  '164.312(a)(2)(iv)',
                'severity': 'CRITICAL',
                'resource': f's3://{name}',
                'finding':  'S3 bucket has no public access block configuration',
            })

    # §164.312(b) — Check: CloudTrail multi-region enabled
    trails      = ct.describe_trails()['trailList']
    multi_region = [t for t in trails if t.get('IsMultiRegionTrail')]
    if not multi_region:
        findings.append({
            'control':  '164.312(b)',
            'severity': 'CRITICAL',
            'resource': 'CloudTrail',
            'finding':  'No multi-region CloudTrail — ePHI access events may not be logged',
        })

    # §164.312(b) — Check: GuardDuty enabled
    detectors = gd.list_detectors().get('DetectorIds', [])
    if not detectors:
        findings.append({
            'control':  '164.312(b)',
            'severity': 'HIGH',
            'resource': 'GuardDuty',
            'finding':  'GuardDuty not enabled — threat detection inactive',
        })

    result = {
        'scan_timestamp':    datetime.now(timezone.utc).isoformat(),
        'total_findings':    len(findings),
        'critical_findings': sum(1 for f in findings if f['severity'] == 'CRITICAL'),
        'high_findings':     sum(1 for f in findings if f['severity'] == 'HIGH'),
        'findings':          findings,
        'compliant':         len(findings) == 0,
    }

    # Save to S3 as dated evidence file
    evidence_s3 = boto3.client('s3')
    date_str    = datetime.now(timezone.utc).strftime('%Y/%m/%d')
    evidence_s3.put_object(
        Bucket='hipaa-compliance-evidence',
        Key=f'scans/{date_str}/compliance_scan.json',
        Body=json.dumps(result, indent=2),
        ContentType='application/json',
    )

    return result


def lambda_handler(event, context):
    result = scan_all()
    print(f"Scan complete: {result['total_findings']} findings, compliant={result['compliant']}")
    return result
</code></pre>
<h2 id="heading-part-9-the-pre-audit-checklist">Part 9: The Pre-Audit Checklist</h2>
<p>Run this script 30 days before any HIPAA audit. It queries your live AWS account across five control categories and writes the output of each check to a dated directory of evidence files. Each file is named after the specific regulation clause it demonstrates, so when an auditor asks for evidence of a particular control, you hand them a file rather than running a command in the room.</p>
<p>Here's what each check collects and what the output looks like:</p>
<p>The RDS encryption check queries every database instance in your account and produces a table showing the instance identifier, whether storage encryption is enabled (true or false), and the KMS key ARN. A HIPAA-compliant account has <code>StorageEncrypted: true</code> on every row.</p>
<p>The KMS rotation check queries every key with "hipaa" in its alias and confirms that <code>KeyRotationEnabled</code> is true for each one. If any key shows false, that's an audit finding under §164.312(a)(2)(iv).</p>
<p>The CloudTrail check returns the trail name, whether it's multi-region (must be true), and whether the trail logs are encrypted with a KMS key. Both properties are required.</p>
<p>The ALB TLS check shows the listener port, protocol, and SSL policy name for every load balancer listener. Auditors look for the policy name to confirm that deprecated TLS versions are disabled.</p>
<p>The VPC endpoint check lists every VPC endpoint in your account with its service name, state, and type. For a HIPAA account, you expect to see at minimum S3 and KMS endpoints in the <code>available</code> state.</p>
<pre><code class="language-bash">#!/usr/bin/env bash
# pre_audit_evidence_collector.sh

EVIDENCE_DIR="hipaa-evidence-$(date +%Y-%m-%d)"
mkdir -p "$EVIDENCE_DIR"

echo "Collecting HIPAA compliance evidence..."

# §164.312(a)(2)(iv) — Encryption at rest
aws rds describe-db-instances \
  --query 'DBInstances[*].{ID:DBInstanceIdentifier,Encrypted:StorageEncrypted,KMS:KmsKeyId}' \
  --output table &gt; "$EVIDENCE_DIR/164-312-a-2-iv-rds-encryption.txt"

aws kms list-aliases \
  --query 'Aliases[?contains(AliasName,`hipaa`)].AliasName' \
  --output text | xargs -I{} aws kms get-key-rotation-status --key-id {} \
  &gt;&gt; "$EVIDENCE_DIR/164-312-a-2-iv-kms-rotation.txt"

# §164.312(b) — Audit Controls
aws cloudtrail describe-trails \
  --query 'trailList[*].{Name:Name,MultiRegion:IsMultiRegionTrail,Encrypted:KMSKeyId}' \
  --output table &gt; "$EVIDENCE_DIR/164-312-b-cloudtrail-config.txt"

# §164.312(e)(1) — Transmission Security
aws elbv2 describe-listeners \
  --load-balancer-arn $(aws elbv2 describe-load-balancers \
    --query 'LoadBalancers[0].LoadBalancerArn' --output text) \
  --query 'Listeners[*].{Port:Port,Protocol:Protocol,SslPolicy:SslPolicy}' \
  --output table &gt; "$EVIDENCE_DIR/164-312-e-1-tls-config.txt"

# §164.310(a)(1) — Network Access Controls
aws ec2 describe-vpc-endpoints \
  --query 'VpcEndpoints[*].{Service:ServiceName,State:State,Type:VpcEndpointType}' \
  --output table &gt; "$EVIDENCE_DIR/164-310-a-1-vpc-endpoints.txt"

echo "Evidence collection complete. Files saved to: $EVIDENCE_DIR/"
ls "$EVIDENCE_DIR/"
</code></pre>
<h2 id="heading-best-practices-summary">Best Practices Summary</h2>
<p><strong>Do:</strong> Sign the AWS BAA before writing any HIPAA infrastructure code. The technical controls are invalid without the legal agreement.</p>
<p><strong>Do:</strong> Use customer-managed KMS keys with automatic rotation. AWS-managed keys are acceptable but don't give you proof of key material control that enterprise healthcare auditors will ask for.</p>
<p><strong>Do:</strong> Implement field-level encryption for the highest-sensitivity ePHI fields (SSN, diagnosis, treatment notes). Storage encryption alone doesn't protect against authorized users with direct database access.</p>
<p><strong>Do:</strong> Enable S3 Object Lock in COMPLIANCE mode for audit logs. GOVERNANCE mode allows deletion by privileged users. COMPLIANCE mode doesn't allow deletion by anyone, including root.</p>
<p><strong>Do:</strong> Run the pre-audit evidence collector monthly, not just before audits. Continuous evidence collection means you're always 30 days away from audit-ready.</p>
<p><strong>Do:</strong> Use VPC endpoints for all AWS service communication. ePHI must not traverse the public internet even when both source and destination are within AWS.</p>
<p><strong>Don't:</strong> Log PHI values in CloudWatch or application logs. Log that a field was accessed, not what it contained.</p>
<p><strong>Don't:</strong> Use shared IAM credentials across multiple engineers or automation systems. Every entity that accesses ePHI must have a unique, auditable identity.</p>
<p><strong>Don't:</strong> Assume that being inside a VPC means a workload is isolated. Security groups are the actual enforcement boundary — a misconfigured security group that allows 0.0.0.0/0 on port 5432 exposes your RDS instance regardless of VPC placement.</p>
<h2 id="heading-resources">Resources</h2>
<ul>
<li><p><a href="https://www.hhs.gov/hipaa/for-professionals/security/index.html"><strong>HHS HIPAA Security Rule</strong></a> — The primary source for all Technical Safeguard requirements cited in this guide</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/whitepapers/latest/architecting-hipaa-security-and-compliance-on-aws/architecting-hipaa-security-and-compliance-on-aws.html"><strong>AWS HIPAA Compliance Reference</strong></a> — AWS's official HIPAA whitepaper — required reading before building on the patterns in this guide</p>
</li>
<li><p><a href="https://aws.amazon.com/artifact/"><strong>AWS Artifact — BAA Download</strong></a> — Where to accept the AWS Business Associate Agreement</p>
</li>
<li><p><a href="https://aws.amazon.com/compliance/hipaa-eligible-services-reference/"><strong>AWS Services in Scope for HIPAA</strong></a> — The current, definitive list of BAA-covered services</p>
</li>
<li><p><a href="https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-111.pdf"><strong>NIST SP 800-111 — Storage Encryption</strong></a> — NIST guidance on storage encryption that informs HIPAA implementation best practices</p>
</li>
<li><p><a href="https://www.hhs.gov/hipaa/for-professionals/compliance-enforcement/audit/protocol/index.html"><strong>OCR HIPAA Audit Protocol</strong></a> — The exact audit protocol OCR uses — reading this tells you precisely what auditors look for</p>
</li>
<li><p><a href="https://github.com/aayostem/platform-toolkit"><strong>Companion Repository</strong></a> — All Terraform modules, Python scripts, and evidence collection scripts from this guide</p>
</li>
</ul>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Build a Production-Ready DevSecOps Platform from Homelab to AWS [Full Book] ]]>
                </title>
                <description>
                    <![CDATA[ In this book, you'll build a fintech transaction ledger from scratch and progressively transform it into a production-ready DevSecOps platform. You'll also deploy it on AWS. The app processes credit a ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-build-a-production-ready-devsecops-platform-from-homelab-to-aws-full-book/</link>
                <guid isPermaLink="false">6a67a3c3a26e578cabe00161</guid>
                
                    <category>
                        <![CDATA[ DevSecOps ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Security ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ book ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Osomudeya Zudonu ]]>
                </dc:creator>
                <pubDate>Mon, 27 Jul 2026 18:30:27 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/f545c4cf-df83-4c56-a196-7b57458de9da.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>In this book, you'll build a fintech transaction ledger from scratch and progressively transform it into a production-ready DevSecOps platform. You'll also deploy it on AWS.</p>
<p>The app processes credit and debit transactions, fires compliance alerts, and stores everything in a database. You'll build the infrastructure around it yourself: automation, scanning, policy enforcement, secrets management, threat detection, and observability.</p>
<p>By the time you're finished, you'll be able to talk your way through every decision in an interview because you made each one here.</p>
<p>This guide doesn't give you a pre-built solution. It makes you feel out and understand each problem before it introduces the tool that solves it.</p>
<p>All the code, manifests, scripts, and stage-by-stage READMEs live in the companion repository. Make sure you clone it before you start:</p>
<pre><code class="language-bash">git clone https://github.com/Osomudeya/clearledger.git
cd clearledger
</code></pre>
<p>Everything in this book refers to files inside that repo.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>You'll need these tools installed on your machine before Stage 0:</p>
<ul>
<li><strong>Multipass:</strong> creates a lightweight Ubuntu VM so Kubernetes has enough resources</li>
<li><strong>kubectl:</strong> talks to your Kubernetes cluster from your terminal</li>
<li><strong>Helm:</strong> installs apps into Kubernetes</li>
<li><strong>Docker Desktop:</strong> builds container images</li>
<li><strong>jq:</strong> formats JSON output so it's readable</li>
</ul>
<p>You'll also need free accounts on GitHub and Docker Hub.</p>
<p>And you should be comfortable with the following knowledge and skills:</p>
<ul>
<li><strong>Basic Linux command line:</strong> navigating directories, reading files, running scripts</li>
<li><strong>Git:</strong> clone, commit, push</li>
<li>What a container is and roughly how Docker builds one</li>
</ul>
<p>You don't need prior Kubernetes, security, or cloud experience. This guide builds that from Stage 0.</p>
<p>Your machine needs at least 24 GB of RAM, 6 CPU cores, and 80 GB of free disk space. See <a href="#heading-how-to-set-up-your-machine">How to Set Up Your Machine</a> for the exact install commands.</p>
<p>The companion repo is at <a href="https://github.com/Osomudeya/clearledger">github.com/Osomudeya/clearledger</a>. Star it, clone it, then continue.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><a href="#heading-prerequisites">Prerequisites</a></li>
<li><a href="#heading-what-you-are-building">What You Are Building</a></li>
<li><a href="#heading-how-to-work-through-this-lab">How to Work Through This Lab</a></li>
<li><a href="#heading-tools-you-will-use">Tools You Will Use</a></li>
<li><a href="#heading-how-to-choose-your-path">How to Choose Your Path</a></li>
<li><a href="#heading-how-to-save-your-progress">How to Save Your Progress</a></li>
<li><a href="#heading-who-this-is-for">Who This Is For</a></li>
<li><a href="#heading-how-to-set-up-your-machine">How to Set Up Your Machine</a></li>
<li><a href="#heading-how-to-start-the-lab">How to Start the Lab</a></li>
<li><a href="#heading-how-to-manage-disk-space">How to Manage Disk Space</a></li>
<li><a href="#heading-how-to-try-the-app-without-kubernetes">How to Try the App Without Kubernetes</a></li>
<li><a href="#heading-how-to-configure-local-domain-names">How to Configure Local Domain Names</a></li>
<li><a href="#heading-stage-0-the-running-system">Stage 0 — The Running System</a></li>
<li><a href="#heading-stage-1-ci-pipeline-github-actions-self-hosted-runner">Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)</a></li>
<li><a href="#heading-stage-2-gitops-with-argocd">Stage 2 — GitOps with ArgoCD</a></li>
<li><a href="#heading-stage-3-security-gates">Stage 3 — Security Gates</a></li>
<li><a href="#heading-stage-4-admission-control-kyverno">Stage 4 — Admission Control (Kyverno)</a></li>
<li><a href="#heading-stage-5-secrets-management-vault">Stage 5: Secrets Management (Vault)</a></li>
<li><a href="#heading-stage-6-runtime-security-falco">Stage 6 — Runtime Security (Falco)</a></li>
<li><a href="#heading-stage-65-chaos-engineering-optional">Stage 6.5 — Chaos Engineering (Optional)</a></li>
<li><a href="#heading-stage-7-security-observability">Stage 7 — Security Observability</a></li>
<li><a href="#heading-stage-75-opentelemetry-optional">Stage 7.5 — OpenTelemetry (Optional)</a></li>
<li><a href="#heading-stage-8-aws-migration">Stage 8 — AWS Migration</a></li>
<li><a href="#heading-troubleshooting-see-troubleshootingmd">Troubleshooting (see troubleshooting.md)</a></li>
<li><a href="#heading-compliance-reference">Compliance Reference</a></li>
<li><a href="#heading-interview-preparation">Interview Preparation</a></li>
<li><a href="#heading-aws-cost-reference">AWS Cost Reference</a></li>
<li><a href="#heading-conclusion">Conclusion</a></li>
</ul>
<h2 id="heading-what-you-are-building">What You Are Building</h2>
<p>ClearLedger is a fintech transaction ledger built with three FastAPI microservices, PostgreSQL, Redis, and a web frontend.</p>
<p>Users can register, sign in, record credit and debit transactions, view their account balance, and receive compliance alerts whenever a transaction exceeds a predefined threshold.</p>
<p>The application is intentionally simple. Its purpose isn't to teach fintech. It gives you a realistic system that you'll secure and operate like a production platform.</p>
<p>The project consists of four components:</p>
<ul>
<li><strong>auth-service:</strong> Handles user registration, login, and JWT authentication.</li>
<li><strong>ledger-service:</strong> Processes transactions, maintains account balances, and stores transaction history.</li>
<li><strong>notification-service:</strong> Listens for large transactions through Redis and generates compliance alerts.</li>
<li><strong>frontend:</strong> A web interface for logging in, viewing balances, submitting transactions, and reviewing alerts.</li>
</ul>
<p>By the end of this book, every one of these services will still exist. What changes is how they are built, deployed, secured, and operated.</p>
<p>The application is simply the vehicle. DevSecOps is the destination.</p>
<h3 id="heading-how-the-platform-evolves">How the Platform Evolves</h3>
<p>You won't install every tool on day one. Instead, the platform grows the same way production systems usually do: a problem appears first, then a solution is introduced.</p>
<p>You'll begin with a manually deployed Kubernetes application. From there, each stage solves one real operational problem.</p>
<p><strong>Stage 0: Raw Kubernetes</strong></p>
<p>You'll deploy and run the application manually, which lets you understand the system before introducing automation.</p>
<p><strong>Stage 1: Continuous Integration</strong></p>
<p>Building container images becomes automatic whenever code is pushed, eliminating manual build steps.</p>
<p><strong>Stage 2: GitOps</strong></p>
<p>Deployments are no longer done with kubectl. Git becomes the single source of truth, preventing configuration drift.</p>
<p><strong>Stage 3: Security Gates</strong></p>
<p>Every commit passes through security scanning so vulnerable code, secrets, and misconfigurations are stopped before deployment.</p>
<p><strong>Stage 4: Admission Control</strong></p>
<p>Even if something bypasses the pipeline, Kubernetes policies prevent insecure workloads from entering the cluster.</p>
<p><strong>Stage 5: Secrets Management</strong></p>
<p>Application credentials move out of Kubernetes Secrets into Vault, removing sensitive data from Git and cluster storage.</p>
<p><strong>Stage 6: Runtime Security</strong></p>
<p>Falco continuously watches running containers and detects suspicious behavior after deployment.</p>
<p><strong>Stage 6.5 (Optional): Chaos Engineering</strong></p>
<p>Failures are introduced deliberately to verify that the platform can recover instead of simply detecting problems.</p>
<p><strong>Stage 7: Observability</strong></p>
<p>Metrics, logs, and dashboards provide visibility into the health, performance, and security of the platform.</p>
<p><strong>Stage 7.5 (Optional): OpenTelemetry</strong></p>
<p>Distributed tracing follows requests across every service, revealing how a single transaction moves through the system.</p>
<p><strong>Stage 8: AWS Migration</strong></p>
<p>The same architecture is deployed on AWS using EKS, ECR, RDS, and an Application Load Balancer without changing how the application itself works.</p>
<p>If you simply want to explore the application before touching Kubernetes, an optional Docker Compose stack lets you run everything locally on your machine.</p>
<p>The guiding principle of this book is simple: every stage makes you feel the problem before introducing the tool that solves it.</p>
<h2 id="heading-how-to-work-through-this-lab">How to Work Through This Lab</h2>
<p>Throughout this process of understanding each problem before introducing the tool that solves it, three habits will carry you through every stage.</p>
<ol>
<li><strong>Read first before you run:</strong> The paragraphs before each command explain <em>why</em> you're running it. Skipping them means you can reproduce the steps but not explain them, and explaining them is what gets you hired. The commands are proof you understand.</li>
<li><strong>Choose with a reason:</strong> Every tool here solves a specific problem. Why use Vault instead of Kubernetes Secrets? Why split code and manifests into two repos? Don't just follow the steps: ask <em>what breaks if we skip this?</em> If you understand the problem, you'll remember the solution.</li>
<li><strong>Go in order and verify every checkpoint:</strong> Each stage depends on the one before it. When you hit an issue, read the error. Getting stuck and debugging is part of the learning: employers want to hear "I hit X error and fixed it by doing Y."</li>
</ol>
<p>At every ✋ Hands-on checkpoint:</p>
<ol>
<li>Run the command.</li>
<li>Compare your output with Expected.</li>
<li>If it doesn't match, fix it before continuing.</li>
<li>When <code>make check-N</code> passes: <code>make snapshot STAGE=N &amp;&amp; make snapshots</code>. Only continue after you see <code>clearledger.stageN</code>.</li>
</ol>
<p>Avoid these mistakes:</p>
<ul>
<li>Don't skip a checkpoint because it passed before.</li>
<li>Don't run <code>make restore</code> without checking available snapshots first.</li>
<li>Replace <code>your-username</code> with your real Docker Hub or GitHub username everywhere it appears.</li>
<li>Run runner commands inside the VM (prompt shows <code>ubuntu@clearledger</code>), not on your Mac.</li>
</ul>
<p>Take screenshots at each <strong>portfolio checkpoint</strong>. These moments become your evidence: proof that the platform runs, detects, blocks, syncs, and observes real activity.</p>
<h2 id="heading-tools-you-will-use">Tools You Will Use</h2>
<p>Come back to the below table when a new name appears and you wonder <em>why now</em>. Each entry is one line: what it does and when it appears.</p>
<p><strong>On your laptop:</strong></p>
<ul>
<li>Multipass creates the Ubuntu VM.</li>
<li>Docker builds images.</li>
<li><code>make</code> wraps long commands into <code>make setup</code> / <code>make check-N</code>.</li>
<li><code>/etc/hosts</code> entries like <code>clearledger.local</code> let your browser reach the cluster.</li>
</ul>
<p><strong>The app:</strong></p>
<ul>
<li>Three Python APIs (auth, ledger, notifications) + a web frontend.</li>
<li>Postgres stores data</li>
<li>Redis lets ledger publish alerts without calling notification directly</li>
<li>nginx ingress routes browser traffic to the right service.</li>
</ul>
<table>
<thead>
<tr>
<th>Tool</th>
<th>One-line role</th>
<th>Stage</th>
</tr>
</thead>
<tbody><tr>
<td>MicroK8s / kubectl</td>
<td>Kubernetes cluster inside the VM: <code>kubectl</code> talks to it.</td>
<td>0</td>
</tr>
<tr>
<td><code>clearledger</code> (repo)</td>
<td>App code + CI workflow: what you build.</td>
<td>1</td>
</tr>
<tr>
<td><code>clearledger-infra</code> (repo)</td>
<td>Kubernetes YAML only: what the cluster should run. CI updates it, ArgoCD deploys it.</td>
<td>1</td>
</tr>
<tr>
<td>GitHub Actions + self-hosted runner</td>
<td>Builds images and updates infra repo on every push. Runner lives in the VM to reach the local cluster.</td>
<td>1</td>
</tr>
<tr>
<td>ArgoCD</td>
<td>Watches <code>clearledger-infra</code>, syncs the cluster to match Git, reverts unauthorized changes.</td>
<td>2</td>
</tr>
<tr>
<td>Gitleaks</td>
<td>Blocks commits that contain secrets (API keys, tokens).</td>
<td>3</td>
</tr>
<tr>
<td>Semgrep</td>
<td>SAST: catches unsafe Python patterns (injection, hardcoded credentials).</td>
<td>3</td>
</tr>
<tr>
<td>Checkov</td>
<td>IaC scanning: misconfigs in Dockerfiles and Kubernetes YAML.</td>
<td>3</td>
</tr>
<tr>
<td>Trivy</td>
<td>Image scanning: known CVEs in pip/npm packages and the built container.</td>
<td>3</td>
</tr>
<tr>
<td>Syft + Grype</td>
<td>SBOM generation and vulnerability check on the artifact itself.</td>
<td>3</td>
</tr>
<tr>
<td>Cosign</td>
<td>Signs container images: Stage 4 rejects unsigned ones at deploy time.</td>
<td>3</td>
</tr>
<tr>
<td>Kyverno</td>
<td>Admission control: blocks non-compliant pods at the cluster gate (root containers, missing limits, unsigned images).</td>
<td>4</td>
</tr>
<tr>
<td>Vault</td>
<td>Stores credentials outside Git and etcd: injects them into pods via a sidecar at startup.</td>
<td>5</td>
</tr>
<tr>
<td>Falco</td>
<td>eBPF runtime detection: alerts when a shell starts or a sensitive file is read inside a running container.</td>
<td>6</td>
</tr>
<tr>
<td>Network policies</td>
<td>Kubernetes firewall between pods: limits blast radius if one service is compromised.</td>
<td>6</td>
</tr>
<tr>
<td>LitmusChaos</td>
<td>Kills pods deliberately to prove the app recovers (optional).</td>
<td>6.5</td>
</tr>
<tr>
<td>Prometheus / Grafana / Loki</td>
<td>Metrics, dashboards, and log search: turns security events into evidence.</td>
<td>7</td>
</tr>
<tr>
<td>OpenTelemetry + Tempo</td>
<td>Distributed traces: shows where one request spent its time across services (optional).</td>
<td>7.5</td>
</tr>
<tr>
<td>Terraform / EKS / ECR / RDS</td>
<td>Infrastructure as code for the AWS migration: same app, cloud-managed backing services.</td>
<td>8</td>
</tr>
</tbody></table>
<p>Each stage adds a new security layer. The tools aren't interchangeable: scanners check your code and images before deployment, ArgoCD keeps the cluster synced to Git, Vault handles secrets, Kyverno blocks unsafe workloads before they run, and Falco watches for suspicious behavior after they're running.</p>
<p>That's why the order matters: you're building defense in depth, one layer at a time.</p>
<h2 id="heading-how-to-choose-your-path">How to Choose Your Path</h2>
<p>Pick one path from your host RAM before you provision a cluster. Switching mid-lab after OOM kills or disk pressure wastes a day, so choose upfront.</p>
<table>
<thead>
<tr>
<th>Your situation</th>
<th>Path</th>
<th>What you get</th>
</tr>
</thead>
<tbody><tr>
<td><strong>8 GB RAM</strong>, or unsure this laptop can carry the lab</td>
<td><strong>Docker Compose first</strong></td>
<td>The real app: register, post a transaction, see the compliance alert fire. Then decide on a cluster. <code>make integration-up</code> · <a href="#heading-how-to-try-the-app-without-kubernetes">Local integration stack</a></td>
</tr>
<tr>
<td><strong>16 GB RAM</strong> on the host</td>
<td><strong>Lite local cluster</strong> (Stages 0–5)</td>
<td><strong>Running on one VM:</strong> This setup includes Kubernetes, CI/CD, GitOps, security checks, admission control, and Vault. To use fewer resources, edit <code>scripts/setup-cluster.local.env</code> before running <code>make setup</code>.</td>
</tr>
<tr>
<td><strong>Under 16 GB</strong> host RAM and you need Kubernetes, or you want all 8 stages</td>
<td><strong>Cloud VM</strong></td>
<td>Provision a remote machine (4–8 vCPU, 16–32 GB RAM), clone the repo, run the lab there, <code>make teardown</code> when done. Stages 6.5 / 7 / 7.5 (chaos + full observability) need 24 GB on the host, use this path if your laptop cannot spare that.</td>
</tr>
</tbody></table>
<p>The default path in this guide assumes 24 GB+ RAM and the full local VM (Before You Start). If that's not you, start from the row that matches your machine.</p>
<h2 id="heading-how-to-save-your-progress">How to Save Your Progress</h2>
<p><strong>Mac + Multipass only:</strong> <code>make snapshot</code> and <code>make restore</code> require Multipass. If you're using Linux without Multipass, skip snapshots and use Path B if something goes wrong.</p>
<p>This lab takes several days to complete.</p>
<p>Your source code lives on your computer, so rebuilding or deleting the VM doesn't delete your Git repository, commits, manifests, or configuration files.</p>
<p>The VM stores your running environment, including deployed pods, Vault secrets, Postgres data, and Grafana dashboards.</p>
<h3 id="heading-save-your-progress">Save Your Progress</h3>
<p>After completing each stage, create a snapshot before moving on. For example:</p>
<pre><code class="language-bash">make snapshot STAGE=7
make snapshots
</code></pre>
<p>Always run <code>make snapshots</code> to confirm the snapshot was created.</p>
<h3 id="heading-restore-your-progress">Restore Your Progress</h3>
<p>If the VM becomes unusable after a while, restore the latest working snapshot:</p>
<pre><code class="language-bash">make snapshots
make restore STAGE=7

export KUBECONFIG=~/.kube/clearledger-config
make check-7
</code></pre>
<h3 id="heading-what-happens-if-the-vm-breaks">What Happens If the VM Breaks?</h3>
<p>You keep:</p>
<ul>
<li>Your Git repository</li>
<li>Your commits</li>
<li><code>.env</code></li>
<li><code>setup-cluster.local.env</code></li>
<li><code>clearledger-infra</code> on GitHub</li>
</ul>
<p>You lose anything stored inside the VM after your last snapshot, including:</p>
<ul>
<li>Running pods</li>
<li>Vault secrets</li>
<li>Postgres data</li>
<li>Grafana and Loki data</li>
</ul>
<p>That's why it's a good idea to create a snapshot after every completed stage.</p>
<h4 id="heading-path-a-you-have-a-snapshot-recommended">Path A: You Have a Snapshot (Recommended)</h4>
<p>Restore the latest working snapshot and continue from that stage.</p>
<pre><code class="language-bash">make snapshots
make restore STAGE=6

export KUBECONFIG=~/.kube/clearledger-config
make check-6
</code></pre>
<h4 id="heading-path-b-no-snapshot">Path B: No Snapshot</h4>
<p>Rebuild the lab.</p>
<pre><code class="language-bash">make teardown
make setup

export KUBECONFIG=~/.kube/clearledger-config
</code></pre>
<p>Your Git repositories are still intact, but the Kubernetes cluster starts empty. Continue the book from the stage you had reached and rebuild the platform from there.</p>
<p>If you run into problems such as disk space issues, failed snapshots, Mac sleep or restart problems, Vault authentication errors, or pods stuck in CrashLoopBackOff, see <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a> for detailed recovery steps.</p>
<h2 id="heading-who-this-is-for">Who This Is For</h2>
<p><strong>Junior DevOps (0–2 yrs):</strong> do every stage in order. Don't skip the pain point sections. Expect Stage 0–2 to take a full day each, Stages 3–7 half a day each, Stage 8 a few hours. That's normal, so don't rush.</p>
<p><strong>Mid-level DevOps (2–4 yrs):</strong> skim Stages 0–2 to understand the app, focus time on Stages 3–7 where the security layers are.</p>
<p><strong>Interview preparation:</strong> complete through Stage 4, then read <code>docs/interview-prep.md</code>. The questions are based on exactly what's in this lab.</p>
<h2 id="heading-how-to-set-up-your-machine">How to Set Up Your Machine</h2>
<p>Requirements are in <a href="#heading-prerequisites">Prerequisites</a> above. Confirm 24 GB RAM, 6 CPU cores, and 80 GB free disk before installing.</p>
<h3 id="heading-install-the-required-tools">Install the Required Tools</h3>
<table>
<thead>
<tr>
<th>Tool</th>
<th>What it does</th>
<th>macOS</th>
<th>Linux</th>
<th>Windows</th>
</tr>
</thead>
<tbody><tr>
<td>Multipass</td>
<td>Creates lightweight Ubuntu VMs on your laptop</td>
<td><code>brew install --cask multipass</code></td>
<td><code>sudo snap install multipass</code></td>
<td><a href="https://multipass.run/install">multipass.run/install</a></td>
</tr>
<tr>
<td>kubectl</td>
<td>Talks to your Kubernetes cluster from your terminal</td>
<td><code>brew install kubectl</code></td>
<td><code>sudo snap install kubectl --classic</code></td>
<td><code>winget install Kubernetes.kubectl</code></td>
</tr>
<tr>
<td>Helm</td>
<td>Package manager for Kubernetes (like apt/brew but for cluster apps)</td>
<td><code>brew install helm</code></td>
<td><code>sudo snap install helm --classic</code></td>
<td><code>winget install Helm.Helm</code></td>
</tr>
<tr>
<td>Docker Desktop</td>
<td>Builds container images on your machine</td>
<td><a href="https://docs.docker.com/desktop/">docker.com</a></td>
<td><a href="https://docs.docker.com/engine/install/">docker.com</a></td>
<td><a href="https://docs.docker.com/desktop/">docker.com</a></td>
</tr>
<tr>
<td>jq</td>
<td>Formats JSON output so you can read it</td>
<td><code>brew install jq</code></td>
<td><code>sudo apt install jq</code></td>
<td><code>winget install jqlang.jq</code></td>
</tr>
</tbody></table>
<p><strong>Windows users:</strong> Run all commands inside WSL2 Ubuntu. Don't use PowerShell for this lab because the setup uses <code>make</code> and Bash scripts.</p>
<p>Verify everything before continuing:</p>
<pre><code class="language-bash">multipass --version
kubectl version --client
helm version
docker --version
jq --version
</code></pre>
<p>If any command fails, install the missing tool before continuing.</p>
<h2 id="heading-how-to-start-the-lab">How to Start the Lab</h2>
<p>The main lab path starts at <a href="#heading-stage-0-the-running-system">Stage 0: the Running System</a>.</p>
<p>After you have run the setup once step by step, you can use this shortcut next time:</p>
<pre><code class="language-bash">make setup
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes
</code></pre>
<p>Expected: one node named <code>clearledger</code> with STATUS <code>Ready</code>.</p>
<p><code>make setup</code> provisions the Multipass VM, installs MicroK8s, applies disk-safety caps, and updates <code>/etc/hosts</code>. Takes 3–5 minutes.</p>
<h2 id="heading-how-to-manage-disk-space">How to Manage Disk Space</h2>
<p>The lab runs on a single-node MicroK8s VM with a fixed disk (80 GB by default). Over days or weeks (especially after CI builds, Helm upgrades, and Stage 7 observability) container images, logs, and journald can fill the root filesystem. Pods then fail with <code>Evicted</code>, <code>ImagePullBackOff</code>, or mysterious <code>Pending</code> states.</p>
<p><code>make setup</code> applies preventive caps automatically (log rotation, image GC thresholds, journald cap). See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">Disk health in troubleshooting.md</a> for the full table and recovery steps.</p>
<p><strong>Check disk health:</strong></p>
<pre><code class="language-bash">make doctor    # PASS / WARN / FAIL + PVC and Prometheus TSDB sizes
</code></pre>
<p><strong>Clean up unused files inside the VM without deleting app data:</strong></p>
<pre><code class="language-bash">make reclaim
</code></pre>
<p>If <code>make doctor</code> still reports FAIL after reclaim, you may need <code>make teardown &amp;&amp; make setup</code> and restore from a snapshot. Full guidance: <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md. VM disk full</a>.</p>
<h2 id="heading-how-to-try-the-app-without-kubernetes">How to Try the App Without Kubernetes</h2>
<p>If your machine doesn't have enough resources for Kubernetes, you can run ClearLedger with Docker Compose.</p>
<pre><code class="language-bash">docker compose -f docker-compose.integration.yml up --build -d
</code></pre>
<p>Open <a href="http://localhost:3000">http://localhost:3000</a>.</p>
<p>When you're ready, stop the stack and continue with Stage 0.</p>
<pre><code class="language-bash">docker compose -f docker-compose.integration.yml down
</code></pre>
<h3 id="heading-how-to-sign-in-for-the-first-time">How to Sign In for the First Time</h3>
<p>First, you'll need to register. The database starts empty after each fresh <code>up</code> (or <code>down -v</code>). Use a real-looking email (Pydantic rejects <code>@*.local</code>), for example <code>test@clearledger.io</code> for an email and <code>SecurePass123</code> for a password.</p>
<p>Then sign in with the same credentials.</p>
<p>Wrong password shows <em>Incorrect email or password</em>. If you see a stale error, hard-refresh or run <code>localStorage.removeItem('cl_token')</code> in the browser console.</p>
<h3 id="heading-how-to-run-the-demo-flow">How to Run the Demo Flow</h3>
<p>First, register and sign in at <a href="http://localhost:3000">http://localhost:3000</a>. Submit a few credits and debits (for example, Salary +$5000, Rent −$1200).</p>
<p>Then confirm the balance updates and history lists entries.</p>
<p>Now submit a transaction <strong>≥ $10,000</strong>: the Alerts panel should show <code>LARGE_TRANSACTION</code>.</p>
<p>Here's an optional smoke test against the same base URL:</p>
<pre><code class="language-bash">BASE_URL=http://localhost:3000 bash scripts/dast/smoke.sh
</code></pre>
<h2 id="heading-how-to-configure-local-domain-names">How to Configure Local Domain Names</h2>
<p>Add the ClearLedger hostnames to your hosts file.</p>
<h3 id="heading-macos-or-linux-with-multipass">macOS or Linux with Multipass</h3>
<p>Run:</p>
<pre><code class="language-bash">sudo bash scripts/setup-hosts.sh
</code></pre>
<p>Or do it manually:</p>
<pre><code class="language-bash">VMIP=$(multipass info clearledger | grep IPv4 | awk '{print $2}')

echo "$VMIP  clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts
</code></pre>
<p>Verify after Stage 0:</p>
<pre><code class="language-bash">curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health
</code></pre>
<p>Expected: <code>200</code>.</p>
<h3 id="heading-wsl2">WSL2</h3>
<p>Find your WSL IP:</p>
<pre><code class="language-bash">ip -4 addr show eth0 | grep inet
</code></pre>
<p>Use the IP shown (or <code>127.0.0.1</code> if it works on your machine), then add it to <code>/etc/hosts</code>:</p>
<pre><code class="language-bash">LAB_IP=&lt;YOUR_IP&gt;

echo "$LAB_IP  clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts
</code></pre>
<p>If you use Chrome or Edge on Windows instead of inside WSL, add the same line to:</p>
<p><code>C:\Windows\System32\drivers\etc\hosts</code></p>
<p>Verify:</p>
<pre><code class="language-bash">curl http://clearledger.local/auth/health
</code></pre>
<h2 id="heading-stage-0-the-running-system">Stage 0 — The Running System</h2>
<p><strong>Starting point:</strong> Nothing is deployed yet, so you're about to build a Kubernetes cluster and deploy ClearLedger manually.</p>
<p><strong>Goal:</strong> By the end of this stage, ClearLedger will be running on Kubernetes. You'll be able to register a user, submit transactions, and see compliance alerts, all deployed by hand, with no automation.</p>
<p>Every deployment, update, and fix is manual. That's intentional. Before automating a platform, you need to understand how it works without automation.</p>
<h3 id="heading-01-provision-the-cluster">0.1: Provision the Cluster</h3>
<p>Next you'll be creating a virtual machine on your laptop that runs its own Kubernetes cluster. Think of it as a miniature data center inside your computer.</p>
<p>Multipass creates lightweight Ubuntu VMs. MicroK8s is a minimal Kubernetes distribution that runs inside that VM. Together they give you a real cluster without needing cloud resources.</p>
<p><strong>Recommended: one command (do this):</strong></p>
<pre><code class="language-bash">make setup
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes
</code></pre>
<p>Expected:</p>
<pre><code class="language-plaintext">NAME          STATUS   ROLES    AGE   VERSION
clearledger   Ready    &lt;none&gt;   2m    v1.29.x
</code></pre>
<p><code>make setup</code> runs <code>scripts/setup-cluster.sh</code> (VM + MicroK8s + disk-safety caps) and <code>scripts/setup-hosts.sh</code> (<code>/etc/hosts</code> entries). It takes 3–5 minutes.</p>
<p>Disk-safety (log rotation, image GC thresholds, journald cap) is configured automatically. See Disk health in <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a> for more info.</p>
<p>If STATUS is <code>NotReady</code>, wait 60 seconds and try again.</p>
<p>Here's the manual setup (only if <code>make setup</code> failed and you need to debug step by step):</p>
<pre><code class="language-bash">multipass launch \
  --name clearledger \
  --cpus 6 --memory 12G --disk 80G \
  22.04
</code></pre>
<p>Get the VM IP (needed for <code>/etc/hosts</code>):</p>
<pre><code class="language-bash">multipass info clearledger | grep IPv4
</code></pre>
<p>Add hosts entries. See the <a href="#heading-how-to-configure-local-domain-names">Domain Names</a> section above, or run <code>sudo bash scripts/setup-hosts.sh</code>.</p>
<pre><code class="language-bash">multipass shell clearledger
</code></pre>
<p>Inside the VM:</p>
<pre><code class="language-bash">sudo snap install microk8s --classic --channel=1.29/stable
sudo usermod -aG microk8s ubuntu &amp;&amp; newgrp microk8s
microk8s enable dns ingress storage helm3 rbac
echo "alias kubectl='microk8s kubectl'" &gt;&gt; ~/.bashrc
echo "alias helm='microk8s helm3'" &gt;&gt; ~/.bashrc
source ~/.bashrc
kubectl get nodes
exit   # back to your host machine
</code></pre>
<p>Connect kubectl from your host:</p>
<pre><code class="language-bash">multipass exec clearledger -- microk8s config &gt; ~/.kube/clearledger-config
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes
</code></pre>
<h3 id="heading-02-understand-the-application-before-deploying-it">0.2: Understand the Application Before Deploying it</h3>
<p>Open these files before running a single <code>kubectl</code> command. Reading the code first builds context that makes everything else make sense.</p>
<table>
<thead>
<tr>
<th>File</th>
<th>What it does</th>
</tr>
</thead>
<tbody><tr>
<td><a href="../app/auth-service/main.py"><code>app/auth-service/main.py</code></a></td>
<td>Register, login, verify JWT</td>
</tr>
<tr>
<td><a href="../app/ledger-service/main.py"><code>app/ledger-service/main.py</code></a></td>
<td>Transactions, balance, calls auth-service to verify every request</td>
</tr>
<tr>
<td><a href="../app/notification-service/main.py"><code>app/notification-service/main.py</code></a></td>
<td>Subscribes to Redis, fires alerts when amount ≥ $10,000</td>
</tr>
<tr>
<td><a href="../app/frontend/src/app.js"><code>app/frontend/src/app.js</code></a></td>
<td>SPA: calls the same API as the curl commands</td>
</tr>
<tr>
<td><a href="../app/auth-service/Dockerfile"><code>app/auth-service/Dockerfile</code></a></td>
<td>Non-root user, pinned base image, HEALTHCHECK</td>
</tr>
</tbody></table>
<p>Notice this line in every Dockerfile: <code>USER appuser</code>. It means the image is designed to run as a normal user instead of root. The Kubernetes manifests also set <code>runAsNonRoot: true</code>. Later, in Stage 4, Kyverno enforces that rule and rejects pods that don't declare they run as non-root. Your app is prepared early so it passes that policy later.</p>
<p>Also look at <a href="../infra/manifests/auth-service/secret.yaml"><code>infra/manifests/auth-service/secret.yaml</code></a>. The database password is <code>changeme-stage0</code> encoded in base64. Decode it:</p>
<pre><code class="language-bash">echo "Y2hhbmdlbWUtc3RhZ2Uw" | base64 -d
# changeme-stage0
</code></pre>
<p>That password is sitting in a YAML file anyone with repo access can read. base64 is encoding, not encryption. It's trivially reversible. Remember this moment. It's why Stage 5 exists.</p>
<h3 id="heading-03-docker-hub-setup">0.3: Docker Hub Setup</h3>
<p>You need a container registry: a place to store the built images so the cluster can pull them. Docker Hub is the simplest option. You'll replace it with a private registry (ECR) in Stage 8.</p>
<p>Create four public repositories on Docker Hub (free account, <a href="http://hub.docker.com">hub.docker.com</a>):</p>
<ol>
<li>Go to <a href="http://hub.docker.com"><code>hub.docker.com</code></a></li>
<li>Click <strong>Create repository</strong></li>
<li>Choose your Docker Hub username as the namespace</li>
<li>Enter one repository name from the list below</li>
<li>Set visibility to <strong>Public</strong></li>
<li>Click <strong>Create</strong></li>
<li>Repeat for all four services</li>
</ol>
<pre><code class="language-plaintext">YOUR_USERNAME/clearledger-auth-service
YOUR_USERNAME/clearledger-ledger-service
YOUR_USERNAME/clearledger-notification-service
YOUR_USERNAME/clearledger-frontend
</code></pre>
<p>Next, generate an access token. Go to <a href="http://hub.docker.com">hub.docker.com</a>, then Account Settings, Security, and New Access Token (Read/Write/Delete). Save it. You won't see it again.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/85562991-e4e7-4d17-8d91-4c4ec2f60114.png" alt="image screenshot guide describing where and how to create access token" style="display: block;" width="600" height="400" loading="lazy">

<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ad8a99a9-c6e4-41ac-a1c0-6799f731782c.png" alt="Image screenshot showing how to create access token" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-bash">docker login
# Username: your Docker Hub username
# Password: the access token (NOT your account password)
</code></pre>
<p>Build and push all four services:</p>
<pre><code class="language-bash"># Replace your-username with your Docker Hub username, the same string everywhere in this lab
export DOCKER_USERNAME=your-username
echo "Using DOCKER_USERNAME=$DOCKER_USERNAME"
</code></pre>
<p><strong>✋ Hands-on checkpoint: Docker Hub username</strong></p>
<pre><code class="language-bash"># Must print your real username, not the literal text "your-username"
echo "$DOCKER_USERNAME"
</code></pre>
<p>Expected: one line with your Docker Hub name (for example, <code>veeno-demo</code>). If you see <code>your-username</code> instead, stop and fix <code>export</code> before building.</p>
<p>Build and push all four services:</p>
<pre><code class="language-bash">docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.1.0 ./app/auth-service
docker build -t $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0 ./app/ledger-service
docker build -t $DOCKER_USERNAME/clearledger-notification-service:v0.1.0 ./app/notification-service
docker build -t $DOCKER_USERNAME/clearledger-frontend:v0.1.0 ./app/frontend

# Push

docker push $DOCKER_USERNAME/clearledger-auth-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-notification-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-frontend:v0.1.0
</code></pre>
<p><strong>✋ Hands-on checkpoint: images on Docker Hub</strong></p>
<p>Open <a href="http://hub.docker.com">hub.docker.com</a> and go to your profile, then <strong>Repositories</strong>. Then confirm that all four <code>clearledger-*</code> repos exist and each shows tag <code>v0.1.0</code>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/fb07dc5f-9db2-4819-810c-89b76b01a0e1.png" alt="screenshot image confirming what docker image repo looks like when done" style="display: block;" width="600" height="400" loading="lazy">

<p>On your laptop, run:</p>
<pre><code class="language-bash">docker pull $DOCKER_USERNAME/clearledger-auth-service:v0.1.0
</code></pre>
<p>Expected: <code>Status: Downloaded newer image</code> or <code>Image is up to date</code>, not <code>repository does not exist</code> or <code>denied</code>.</p>
<h3 id="heading-04-look-at-the-manifests-before-applying-them">0.4: Look at the Manifests Before Applying Them</h3>
<p>Kubernetes uses <strong>manifest</strong> files (YAML) to describe the resources it should create. Instead of clicking buttons, you declare the desired state, and Kubernetes creates it.</p>
<p>Before deploying ClearLedger, take a quick look at these manifests:</p>
<ul>
<li><code>infra/manifests/namespace.yaml</code>: Creates the <code>clearledger</code> namespace.</li>
<li><code>infra/manifests/postgres/</code>: Deploys PostgreSQL.</li>
<li><code>infra/manifests/redis/redis.yaml</code>: Deploys Redis.</li>
<li><code>infra/manifests/auth-service/</code>: Deploys the authentication service.</li>
<li><code>infra/manifests/ledger-service/</code>: Deploys the ledger service.</li>
<li><code>infra/manifests/notification-service/</code>: Deploys the notification service.</li>
<li><code>infra/manifests/frontend/</code>: Deploys the web application.</li>
<li><code>infra/manifests/ingress.yaml</code>: Makes the application available at <code>clearledger.local</code>.</li>
<li><code>infra/manifests/rbac/rbac.yaml</code>: Defines who may do what inside the cluster.</li>
</ul>
<p>You don't need to understand every field yet. The goal is simply to see how the application is described before Kubernetes creates it.</p>
<p>You'll understand how Ingress routing and RBAC work in the two optional sections after §0.6. For now, just see how the app is described before Kubernetes creates it.</p>
<h3 id="heading-05-deploy-clearledger-layer-by-layer">0.5: Deploy ClearLedger (Layer by Layer)</h3>
<p>Deploy in <strong>six layers</strong>. Finish each layer before starting the next. Run <code>kubectl get pods -n clearledger</code> after layers 2, 3, and 6 to confirm progress.</p>
<p>Set a short path variable and confirm your username is still set:</p>
<pre><code class="language-bash">export DOCKER_USERNAME=your-username   # skip if already set in §0.3
STAGE0=stages/stage-0-raw-kubernetes/infra/manifests
</code></pre>
<h4 id="heading-051-layer-1-namespace-and-rbac">0.5.1 — Layer 1: Namespace and RBAC</h4>
<p>Nothing else can be created until the namespace exists. RBAC also must exist before workloads reference ServiceAccounts.</p>
<pre><code class="language-bash">kubectl apply -f infra/manifests/namespace.yaml
kubectl apply -f infra/manifests/rbac/rbac.yaml
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get namespace clearledger
kubectl get serviceaccount -n clearledger
# Expected: auth-service, ledger-service, notification-service, clearledger-viewer
</code></pre>
<h4 id="heading-052-layer-2-postgresql">0.5.2 — Layer 2: PostgreSQL</h4>
<p>Database must be running before auth-service or ledger-service start. Both services connect to Postgres on startup to run migrations and serve requests, and they'll crash-loop if the database isn't there yet.</p>
<pre><code class="language-bash">kubectl apply -f infra/manifests/postgres/postgres-secret.yaml
kubectl apply -f infra/manifests/postgres/postgres.yaml

kubectl wait --for=condition=ready pod -l app=postgres \
  -n clearledger --timeout=120s
</code></pre>
<p>Expected after <code>kubectl apply</code>:</p>
<pre><code class="language-plaintext">secret/postgres-secret created
persistentvolumeclaim/postgres-pvc created
statefulset.apps/postgres created
service/postgres created
</code></pre>
<p>Expected when <code>kubectl wait</code> succeeds: the command exits with no output (exit code 0). If it times out, see <strong>If Postgres stays Pending</strong> below before continuing.</p>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pods -n clearledger -l app=postgres
kubectl get pvc -n clearledger
</code></pre>
<p>Expected:</p>
<pre><code class="language-plaintext">NAME         READY   STATUS    RESTARTS   AGE
postgres-0   1/1     Running   0          45s

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
postgres-pvc   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   5Gi        RWO            microk8s-hostpath   45s
</code></pre>
<p><strong>If Postgres stays Pending</strong> (<code>kubectl wait</code> times out, pod shows <code>0/1 Pending</code>, PVC shows <code>Pending</code>):</p>
<p>Postgres needs a <strong>PersistentVolumeClaim</strong>: disk space on the cluster. MicroK8s provides that through the <code>hostpath-storage</code> addon. If <code>make setup</code> was interrupted or you used manual setup without <code>microk8s enable storage</code>, the PVC has nothing to bind to and the pod never schedules.</p>
<p>Check the events: you'll usually see something like:</p>
<pre><code class="language-plaintext">Warning  FailedScheduling  ...  pod has unbound immediate PersistentVolumeClaims
Normal   FailedBinding     ...  no persistent volumes available for this claim and no storage class is set
</code></pre>
<p>Fix it on the VM, then restart the postgres pod. Run this <strong>from your host</strong>: the same command on macOS, Linux, or Windows PowerShell (Multipass is installed on the host. It executes inside the VM for you):</p>
<pre><code class="language-bash"># Enable storage (and ingress/rbac if make setup skipped them)
multipass exec clearledger -- microk8s enable storage ingress rbac

# Confirm a default StorageClass exists
kubectl get storageclass
# Expected: microk8s-hostpath (default)

# Kick the pod so it reschedules against the new storage class
kubectl delete pod postgres-0 -n clearledger

kubectl wait --for=condition=ready pod -l app=postgres \
  -n clearledger --timeout=120s
kubectl get pods -n clearledger -l app=postgres
# Expected: postgres-0   1/1   Running
</code></pre>
<p>Don't continue to auth-service or ledger-service until Postgres is <code>Running</code>. They will crash-loop without a database.</p>
<h4 id="heading-053-layer-3-redis">0.5.3 — Layer 3: Redis</h4>
<p><strong>Why Redis is here (a quick scenario):</strong> Imagine a customer posts a $15,000 debit. Ledger-service saves it to Postgres, then publishes a message to Redis: <em>"large transaction, user X, amount 15000."</em> Notification-service is listening on that channel. It picks up the message and records a compliance alert: the one you'll see in the UI later when you curl <code>/notifications/alerts</code>.</p>
<p>Ledger-service and notification-service don't call each other directly. Redis sits in the middle as a <strong>message bus</strong>: ledger publishes, notification subscribes. That's why Redis must be running before you deploy notification-service (and why you deploy it now, alongside Postgres, before the app layer).</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f099d734-8b73-483d-a28b-16cd7703703b.png" alt="flow daigram image explaining how redis works" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-bash">kubectl apply -f infra/manifests/redis/redis.yaml
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pods -n clearledger -l app=redis
</code></pre>
<p>Expected:</p>
<pre><code class="language-plaintext">NAME                     READY   STATUS    RESTARTS   AGE
redis-xxxxxxxxxx-xxxxx   1/1     Running   0          30s
</code></pre>
<h4 id="heading-054-layer-4-application-secrets">0.5.4 — Layer 4: Application secrets</h4>
<p>Credentials live in Kubernetes Secrets for Stage 0 (Stage 5 moves them to Vault).</p>
<pre><code class="language-bash">kubectl apply -f infra/manifests/auth-service/secret.yaml
kubectl apply -f infra/manifests/ledger-service/secret.yaml
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get secrets -n clearledger | grep -E 'auth-service|ledger-service'
</code></pre>
<p>Expected (AGE will differ, <strong>DATA</strong> counts must match):</p>
<pre><code class="language-plaintext">auth-service-secret     Opaque   2      64s
ledger-service-secret   Opaque   1      8s
</code></pre>
<p><code>auth-service-secret</code> holds two keys (<code>database_url</code>, <code>jwt_secret</code>). <code>ledger-service-secret</code> holds one (<code>database_url</code>). Stage 5 replaces these with Vault, for now they live in the cluster as Kubernetes Secrets.</p>
<h4 id="heading-055-layer-5-application-workloads">0.5.5 — Layer 5: Application workloads</h4>
<p>You're about to start the four app services: auth, ledger, notification, and frontend. Postgres, Redis, and the Secrets from the last two layers are already in place. Now Kubernetes needs to pull your Docker Hub images and run them as pods.</p>
<p><strong>Two files per service (mostly):</strong> A Deployment tells Kubernetes <em>which container image to run</em> and <em>how many copies</em>. A <strong>Service</strong> gives that app a stable name inside the cluster (for example, <code>auth-service</code> so ledger can find auth without knowing pod IP addresses). You apply the Deployment first, then the Service.</p>
<p>So why are we using the <code>sed</code> command below? The deployment YAML files in Git contain a placeholder: literally the text <code>DOCKER_USERNAME</code>, because everyone's Docker Hub username is different. You already set yours in §0.3 (<code>export DOCKER_USERNAME=YOUR_DOCKERHUB_USERNAME</code>). The <code>sed</code> line swaps that placeholder for your real username on the fly, as the manifest is sent to Kubernetes. You never edit the file in Git. If you skip <code>sed</code> and apply the raw file, Kubernetes tries to pull an image called <code>DOCKER_USERNAME/clearledger-auth-service</code>, which doesn't exist.</p>
<p>Why do we use the Stage 0 folder? This repo has more than one copy of the Kubernetes manifests. For this manual deployment, use <code>stages/stage-0-raw-kubernetes/infra/manifests/</code>. Those files are prepared for Stage 0 and contain the <code>DOCKER_USERNAME</code> placeholder that the commands below replace. Don't use <code>infra/manifests/</code> yet, as those files are for the GitOps stages later.</p>
<p>Deploy each service in order. Run these from the repo root with <code>DOCKER_USERNAME</code> still exported:</p>
<p><strong>1. auth-service</strong>: login and registration</p>
<pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/auth-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/auth-service/service.yaml
</code></pre>
<p><strong>2. ledger-service</strong>: transactions and balance (needs Postgres + the secret you created in §0.5.4)</p>
<pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/ledger-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/ledger-service/service.yaml
</code></pre>
<p><strong>3. notification-service</strong>: listens on Redis for large-transaction alerts (no database secret in this one)</p>
<pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/notification-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/notification-service/service.yaml
</code></pre>
<p><strong>4. frontend</strong>: the web UI (Deployment and Service are in one file here)</p>
<pre><code class="language-bash">sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/frontend/deployment.yaml" | kubectl apply -f -
</code></pre>
<p><strong>Verify</strong> (all app pods should reach <code>Running</code>: auth and ledger may take ~30s while they connect to Postgres):</p>
<pre><code class="language-bash">kubectl get pods -n clearledger
</code></pre>
<p>Expected. You should see Postgres and Redis from earlier layers plus new pods for each app (exact pod names vary):</p>
<pre><code class="language-plaintext">NAME                                      READY   STATUS    RESTARTS   AGE
postgres-0                                1/1     Running   0          15m
redis-xxxxxxxxxx-xxxxx                    1/1     Running   0          10m
auth-service-xxxxxxxxxx-xxxxx             1/1     Running   0          45s
auth-service-xxxxxxxxxx-xxxxx             1/1     Running   0          45s
ledger-service-xxxxxxxxxx-xxxxx           1/1     Running   0          40s
ledger-service-xxxxxxxxxx-xxxxx           1/1     Running   0          40s
notification-service-xxxxxxxxxx-xxxxx     1/1     Running   0          35s
frontend-xxxxxxxxxx-xxxxx                 1/1     Running   0          30s
</code></pre>
<p>If auth-service or ledger-service is <code>CrashLoopBackOff</code>, check the logs:</p>
<pre><code class="language-bash">kubectl logs -n clearledger deploy/auth-service --tail=20
</code></pre>
<p><strong>Common cause:</strong> you applied <code>infra/manifests/*/deployment.yaml</code> instead of the Stage 0 files above: logs may show <code>DATABASE_URL is not set</code>. Re-run the <code>sed</code> + <code>kubectl apply</code> commands in this section.</p>
<p><strong>✋ Hands-on checkpoint: workloads before ingress</strong></p>
<pre><code class="language-bash">kubectl get deployment -n clearledger
kubectl get pods -n clearledger --field-selector=status.phase!=Running
</code></pre>
<p>Expected: four Deployments (<code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>frontend</code>) with <code>READY</code> matching desired replicas (auth and ledger show <code>2/2</code>). The second command prints <strong>nothing</strong>: no pods stuck in Pending or CrashLoopBackOff.</p>
<h4 id="heading-056-layer-6-ingress">0.5.6 — Layer 6: Ingress</h4>
<p>Exposes the cluster to <code>http://clearledger.local</code>.</p>
<pre><code class="language-bash">kubectl apply -f infra/manifests/ingress.yaml
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get ingress -n clearledger
curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/
# Expected: 200
</code></pre>
<h4 id="heading-057-watch-until-stable">0.5.7: Watch until stable</h4>
<pre><code class="language-bash">kubectl get pods -n clearledger -w
</code></pre>
<p>Expected final state (press Ctrl+C to stop watching once all pods show <code>Running</code>):</p>
<pre><code class="language-plaintext">NAME                                  READY   STATUS    RESTARTS
auth-service-xxx                      1/1     Running   0
auth-service-yyy                      1/1     Running   0
frontend-xxx                          1/1     Running   0
ledger-service-xxx                    1/1     Running   0
ledger-service-yyy                    1/1     Running   0
notification-service-xxx              1/1     Running   0
postgres-0                            1/1     Running   0
redis-xxx                             1/1     Running   0
</code></pre>
<p>Pod stuck in <code>Pending</code> or <code>CrashLoopBackOff</code>? These two commands show you what went wrong:</p>
<pre><code class="language-bash">kubectl describe pod POD_NAME -n clearledger
kubectl logs POD_NAME -n clearledger --previous
</code></pre>
<h3 id="heading-06-verify-the-running-system">0.6: Verify the Running System</h3>
<p>Use <strong>one test account</strong> for both browser and curl so nothing conflicts:</p>
<table>
<thead>
<tr>
<th>Field</th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>Email</td>
<td><code>test@clearledger.io</code></td>
</tr>
<tr>
<td>Password</td>
<td><code>SecurePass123</code></td>
</tr>
</tbody></table>
<p>If you already registered in the browser with a <strong>different</strong> password, either sign in with that password or pick a new email: the curl commands below must use the <strong>same</strong> email and password you actually registered with.</p>
<h4 id="heading-browser-verification-recommended">Browser verification (recommended):</h4>
<p>Open <code>http://clearledger.local</code> in your browser. You should see the ClearLedger login screen.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ba678edb-ed82-4063-b2e3-304bfe31e27c.png" alt="clearledger login screen UI screenshot" style="display: block;" width="600" height="400" loading="lazy">

<p>Click <strong>Register</strong> and create an account with <code>test@clearledger.io</code> / <code>SecurePass123</code> (same as the curl block below. Pydantic rejects obviously fake emails like <code>test@test.com</code>).</p>
<p>Sign in with that email and password. On first login the dashboard auto-seeds demo transactions. Wait a few seconds for them to appear:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/4bf4bd15-ad3b-466c-ba46-8539c2e1536c.png" alt="screenshot of clearledger UI after login" style="display: block;" width="600" height="400" loading="lazy">

<p>Look at the <strong>Current Balance</strong> card. It should show a dollar amount with a sparkline chart.</p>
<p>Look at <strong>Transaction History</strong>. You should see entries like "Salary (Acme Corp", "Rent) May 2026", and so on.</p>
<p>And look at the <strong>Alerts</strong> panel at the bottom. You should see <code>LARGE_TRANSACTION</code> alerts with a red badge. Two of the demo transactions exceed $10,000, which triggers the compliance alert automatically.</p>
<p>Then submit your own transaction over $10,000 and watch the alert count increase in real time.</p>
<p><strong>What to look for:</strong></p>
<ul>
<li>Balance updates immediately after each transaction</li>
<li>Credits show as green <code>+$</code> amounts, debits show as red <code>−$</code> amounts</li>
<li>The Alerts badge count increases when you submit a transaction ≥ $10,000</li>
<li>Each alert shows the amount, direction, and timestamp</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/05690dde-58de-4453-9a3f-8eee09b33293.png" alt="screenshot of clearledger UI after login and making transactions" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>Take a screenshot of the dashboard showing transactions and at least one alert.</strong> This is the first piece of your portfolio.</p>
<p><strong>Alternatively via curl</strong> (same account: useful if the browser is not cooperating):</p>
<pre><code class="language-bash"># Register (skip if you already registered in the browser with the same email)
curl -s -X POST http://clearledger.local/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' | jq .
</code></pre>
<p>Expected: <code>{"user_id":"...","email":"test@clearledger.io"}</code>, or an error that the email is already registered (fine if you used the browser first).</p>
<pre><code class="language-bash"># Login — save the token (must match the password you registered with)
TOKEN=$(curl -s -X POST http://clearledger.local/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' \
  | jq -r .access_token)
echo "Token: ${TOKEN:0:30}..."
</code></pre>
<p>If <code>TOKEN</code> is empty or login returns <code>401</code>, your browser password doesn't match: re-register with the table above or use your actual password in the <code>-d</code> JSON.</p>
<pre><code class="language-bash"># Create a large transaction (triggers notification alert)
curl -s -X POST http://clearledger.local/ledger/transactions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"amount":15000,"direction":"debit","description":"Property payment"}' | jq .
</code></pre>
<p>Expected: a transaction object with <code>id</code>, <code>amount: 15000</code>, <code>direction: "debit"</code>:</p>
<pre><code class="language-bash"># Check balance
curl -s http://clearledger.local/ledger/balance \
  -H "Authorization: Bearer $TOKEN" | jq .
</code></pre>
<pre><code class="language-bash"># Confirm the notification alert fired
curl -s http://clearledger.local/notifications/alerts | jq .
</code></pre>
<p>Expected (curl-only path, no browser demo seed): at least one alert for the $15,000 transaction, for example, <code>{"total":1,"alerts":[{"type":"LARGE_TRANSACTION","amount":15000,...}]}</code>. If you already used the browser, <code>total</code> may be <strong>3 or more</strong> (two demo alerts plus yours), that is also correct.</p>
<p><strong>If you see</strong> <code>{"detail":"Unauthorized"}</code><strong>:</strong> your token has expired. JWTs are short-lived for security. This is intentional. Re-run the login command above to get a fresh token, then retry the failed command.</p>
<p>This only affects the <code>$TOKEN</code> variable in your current terminal session. If you open a new terminal, you need to run the login command again because <code>$TOKEN</code> doesn't persist across sessions.</p>
<pre><code class="language-bash">make check-0
</code></pre>
<h3 id="heading-understanding-ingress-optional">Understanding Ingress (Optional)</h3>
<p>Read this after §0.6 if you want to understand how <code>clearledger.local</code> reaches your pods.</p>
<p>Your cluster runs four application services: frontend, auth-service, ledger-service, and notification-service. Each has an internal <strong>Service</strong> address inside the cluster, but none are reachable from your browser until an <strong>Ingress</strong> routes external traffic.</p>
<p>The Ingress is the front door. When a request hits <code>clearledger.local</code>, Kubernetes looks at the URL path and forwards to the right service. Requests to <code>/auth</code> go to auth-service, <code>/ledger</code> to ledger-service, <code>/notifications</code> to notification-service, and <code>/</code> to the frontend.</p>
<p>Open <a href="./infra/manifests/ingress.yaml"><code>infra/manifests/ingress.yaml</code></a> and read the comments. The API paths use a <strong>rewrite:</strong> <code>/auth/login</code> becomes <code>/login</code> before it reaches auth-service, so backend routes stay simple.</p>
<p>You'll add more hostnames later (<code>grafana.local</code>, <code>argocd.local</code>, and so on): each gets its own Ingress manifest in a later stage. This file is only the ClearLedger app.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3307ca55-75d4-41be-a161-e741d2349b5e.png" alt="flow diagram explaining ingress, how it works." style="display: block;" width="600" height="400" loading="lazy">

<h3 id="heading-understanding-rbac-optional">Understanding RBAC (Optional)</h3>
<p>Ingress controls traffic coming from outside the cluster. RBAC controls permissions inside the cluster.</p>
<p>This file creates identities and permissions for the <code>clearledger</code> namespace.</p>
<p>Open <a href="../infra/manifests/rbac/rbac.yaml"><code>infra/manifests/rbac/rbac.yaml</code></a>. The comments at the top mirror this walkthrough.</p>
<p>A <strong>ServiceAccount</strong> is an identity for a pod. For example, <code>auth-service</code>, <code>ledger-service</code>, and <code>notification-service</code> each get their own identity.</p>
<p>A <strong>Role</strong> says what that identity is allowed to do. In your repo, the app roles are very limited: they can only <code>get</code> and <code>list</code> Kubernetes Endpoints. They can't read Secrets, delete pods, create resources, or access other namespaces.</p>
<p>A <strong>RoleBinding</strong> connects the identity to the permissions. Without the RoleBinding, the Role exists but no pod receives those permissions.</p>
<p>The <code>clearledger-viewer</code> ServiceAccount is for read-only debugging. It can inspect pods, services, endpoints, events, and configmaps, but it can't read Secrets.</p>
<p>The default ServiceAccount is bound to a role with zero permissions. That way, if a pod forgets to set <code>serviceAccountName</code>, it falls back to an identity that can do nothing.</p>
<p>The point is least privilege: even if a pod is compromised, Kubernetes doesn't hand it broad cluster access.</p>
<h3 id="heading-07-why-manual-deploys-cant-be-trusted">0.7: Why Manual Deploys Can't Be Trusted</h3>
<p>In Stage 0, you built and deployed the app by hand. Now you'll make one small code change and deploy it again. This shows the problem with manual deployments: they're hard to track, hard to roll back, and hard to prove. Stages 1 and 2 fix that with CI and GitOps.</p>
<h4 id="heading-step-1-make-a-visible-change">Step 1: Make a visible change.</h4>
<p>Open <code>app/auth-service/main.py</code> and find the <code>/health</code> endpoint. Change the return value so you can tell the new version is running:</p>
<pre><code class="language-python"># Before
return {"status": "ok", "service": settings.service_name}

# After — add a version field
return {"status": "ok", "service": settings.service_name, "version": "0.2.0"}
</code></pre>
<p>Save the file. This simulates a developer shipping a small fix.</p>
<h4 id="heading-step-2-build-push-and-deploy-by-hand">Step 2: Build, push, and deploy by hand.</h4>
<pre><code class="language-bash">docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.2.0 ./app/auth-service

docker push $DOCKER_USERNAME/clearledger-auth-service:v0.2.0
kubectl set image deployment/auth-service \
  auth-service=$DOCKER_USERNAME/clearledger-auth-service:v0.2.0 \
  -n clearledger
</code></pre>
<p>Wait about 30 seconds for Kubernetes to pull the new image and restart the pods:</p>
<pre><code class="language-bash">kubectl rollout status deployment/auth-service -n clearledger
</code></pre>
<h4 id="heading-step-3-verify-your-change-is-live">Step 3: Verify your change is live.</h4>
<pre><code class="language-bash">curl -s http://clearledger.local/auth/health | jq .
</code></pre>
<p>Expected: <code>{"status":"ok","service":"auth-service","version":"0.2.0"}</code></p>
<p>If you still see the old response without <code>"version"</code>, wait a few more seconds and retry. Kubernetes is still rolling out the new pods.</p>
<h4 id="heading-step-4-notice-what-manual-deploy-doesnt-give-you">Step 4: Notice what manual deploy doesn't give you.</h4>
<p>You deployed a change. It works. But think about what just happened:</p>
<ul>
<li><strong>Who deployed this?</strong> There's no record. You ran <code>kubectl</code> from your laptop. If three people have cluster access, no one knows who changed what.</li>
<li><strong>What changed?</strong> The only evidence is the Docker Hub tag <code>v0.2.0</code>. Nothing links that tag to a specific commit or code review.</li>
<li><strong>What if</strong> <code>v0.2.0</code> <strong>is broken?</strong> You would need to remember the previous tag, then run <code>kubectl set image</code> again to roll back. What if you don't remember the tag? What if the previous image was deleted?</li>
<li><strong>What if someone else runs</strong> <code>kubectl apply</code> <strong>with</strong> <code>v0.1.0</code> <strong>while you're pushing</strong> <code>v0.2.0</code><strong>?</strong> The cluster silently reverts to the old version. No error. No notification. You think your fix is live, but it's not.</li>
<li><strong>Where is the audit trail?</strong> Nowhere. In a regulated environment (banking, healthcare, government), you need proof of who deployed what and when. Right now you have nothing.</li>
</ul>
<p>Manual deploys can work for a demo. But they don't hold up for a team or a regulated environment. Keep these gaps in mind. They're why the next stages exist.</p>
<h4 id="heading-step-5-revert-your-change-before-continuing">Step 5: Revert your change before continuing.</h4>
<p>Undo the health endpoint change in <code>app/auth-service/main.py</code> (remove <code>"version": "0.2.0"</code>). Don't rebuild: the cluster will keep running <code>v0.2.0</code> for now, and Stage 1 will take over image management.</p>
<p>Stage 1 automates the build. Stage 2 fixes the deployment.</p>
<h3 id="heading-what-you-learned-in-stage-0">What You Learned in Stage 0</h3>
<ul>
<li>How to provision a local Kubernetes cluster with Multipass and MicroK8s</li>
<li>How Kubernetes manifests describe the desired state of your system</li>
<li>How an Ingress routes external traffic to internal services</li>
<li>How to build, push, and deploy container images manually</li>
<li><strong>Why manual deploys can't be trusted</strong>: no audit trail, no rollback, no consistency</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Deployed a multi-service application to Kubernetes by hand: namespace, RBAC, a StatefulSet database, Deployments, Services, and path-based Ingress routing, and can explain why each layer deploys in that order.</p>
</blockquote>
<p><code>make snapshot STAGE=0 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage0</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<h2 id="heading-stage-1-ci-pipeline-github-actions-self-hosted-runner">Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)</h2>
<p>In Stage 0 you built and deployed by hand. Stage 1 automates the build: a <code>git push</code> runs a pipeline that builds images, scans them, pushes to Docker Hub, and records the new tag in <code>clearledger-infra</code>.</p>
<p><strong>Goal:</strong> every push to GitHub automatically builds images, pushes them to Docker Hub, and updates image tags in <code>clearledger-infra</code>.</p>
<p><strong>Am I ready for Stage 1?</strong></p>
<p>Run these <strong>yourself</strong> before §1.1:</p>
<pre><code class="language-plaintext">make check-0
echo "$DOCKER_USERNAME"    # must not be empty or "your-username"
curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health
</code></pre>
<p>Expected: health check green, <code>echo</code> prints your Docker Hub user, and curl prints <code>200</code>.</p>
<p>What you'll need for this section:</p>
<ul>
<li>Docker Hub account with four clearledger- repositories (see <a href="http://QUICKSTART.md">QUICKSTART.md</a> §1b)</li>
<li>GitHub account: you can create repos and personal access tokens</li>
<li>~2–4 hours for runner install + first green pipeline (this is the hardest stage for beginners)</li>
<li>Done when: make check-1 passes and you manually confirmed the five items in §1.7 below. Then save: make snapshot STAGE=1 → make snapshots (confirm clearledger.stage1).</li>
</ul>
<h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3>
<p>In Stage 0, your laptop was the deployment system.</p>
<p>You typed <code>docker build</code>, <code>docker push</code>, and <code>kubectl set image</code> yourself. That worked for a demo, but it's not how teams should ship software.</p>
<p>Manual builds create too many unanswered questions:</p>
<ul>
<li>Did this image come from the latest code?</li>
<li>Did someone build it from a dirty working tree?</li>
<li>Did the build work the same way on another machine?</li>
<li>Which commit produced the image currently running?</li>
<li>Who pushed the image, and when?</li>
</ul>
<p><strong>CI (Continuous Integration)</strong> fixes the build side of that problem. It means that every time code is pushed, an automated system builds, checks, and packages it the same way.</p>
<p>Think of CI as a factory line:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/5a9a4283-a77c-493e-85c4-e457b6ab00c9.png" alt="flow diagram explain gihub ci flow" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-text">Developer pushes code
        ↓
GitHub detects the push
        ↓
GitHub Actions starts the pipeline
        ↓
Runner executes the jobs
        ↓
Docker images are built and pushed
        ↓
Infra manifests are updated with the new image tags  (in clearledger-infra — §1.3)
</code></pre>
<p>The important idea is that the build no longer depends on your laptop. Your laptop writes code and the pipeline produces the release artifact.</p>
<p>A CI system has three parts:</p>
<ol>
<li><strong>Pipeline host</strong>: the control plane. It notices a push and decides which workflow to run. In this lab, that's <strong>GitHub Actions</strong>.</li>
<li><strong>Pipeline file</strong>: the instructions. It's a YAML file at <code>.github/workflows/ci.yaml</code> that says what jobs to run.</li>
<li><strong>Runner</strong>: the worker machine. It actually executes the commands in the pipeline.</li>
</ol>
<p>GitHub Actions normally uses GitHub-hosted runners in the cloud. In this lab, that's not enough. Your Kubernetes cluster lives inside a local Multipass VM and GitHub's cloud runner can't reach it. You also need the runner inside the VM to build Docker images using the local Docker daemon.</p>
<p>So you install a self-hosted runner inside the VM. It connects outbound to GitHub, waits for work, then executes pipeline jobs locally where it can reach everything.</p>
<p>Two repos, <code>clearledger</code> (code + CI) and <code>clearledger-infra</code> (Kubernetes YAML only). You'll create the second in §1.3.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f7c69215-31c8-4806-b899-15caeace9485.png" alt="flow chart demonstrating self hosted github flow" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-text">GitHub — clearledger (app repo)
  stores your code
  starts the workflow on git push
        ↓
Self-hosted runner (inside Multipass VM)
  builds Docker images
  pushes images to Docker Hub
  updates image tags in clearledger-infra  ← you create this in §1.3
        ↓
GitHub — clearledger-infra (infra repo)
  stores Kubernetes YAML with the new image tags
  ArgoCD watches this repo in Stage 2 (not yet)
</code></pre>
<p>For Stages 1–7, the lab uses <code>.github/workflows/ci.yaml</code> with your self-hosted runner. It builds images, pushes them to Docker Hub, and updates <code>clearledger-infra</code>. Stage 8 adds a separate AWS workflow, <code>.github/workflows/ci-aws.yaml</code>, which pushes to ECR instead. You don't need to configure the AWS workflow until you reach Stage 8.</p>
<h3 id="heading-11-push-the-app-repo-to-github-not-clearledger-infra-yet">1.1: Push the App Repo to GitHub (Not <code>clearledger-infra</code> Yet)</h3>
<p>This step is <strong>repo #1,</strong> <code>clearledger</code> (application code + CI workflow). You're pushing the clone on your laptop: the same folder where you ran Stage 0 (<code>make setup</code>, <code>kubectl apply</code>, and so on).</p>
<p><code>clearledger-infra</code> comes later in §1.3. That second repo holds Kubernetes manifests only. Don't create it here.</p>
<p>First, put the application repo somewhere GitHub Actions can see it.</p>
<p>Go to GitHub and then New Repository:</p>
<ul>
<li>Repository name: <code>clearledger</code> (exact name, not <code>clearledger-infra</code>)</li>
<li>Visibility: <strong>Public or Private</strong>. Both work with the self-hosted runner and GitHub Actions. ArgoCD never reads this repo (see <a href="#heading-private-repos-what-syncs-where">Private repos: what syncs where</a> in §1.3).</li>
<li>Do <strong>not</strong> initialize with a README or <code>.gitignore</code></li>
</ul>
<p>The repo already has those files locally. If GitHub creates its own, your first push may fail because the histories don't match.</p>
<p>Run from your <strong>local</strong> <code>clearledger</code> <strong>project root</strong> on your laptop (where <code>app/</code>, <code>infra/</code>, and <code>.github/workflows/ci.yaml</code> live):</p>
<pre><code class="language-bash">cd ~/Desktop/clearledger   # your clone path
git remote add origin https://github.com/YOUR_USERNAME/clearledger.git
git branch -M main
git push -u origin main
</code></pre>
<p>If <code>git remote add</code> fails because <code>origin</code> already exists:</p>
<pre><code class="language-bash">git remote -v
git remote set-url origin https://github.com/YOUR_USERNAME/clearledger.git
git push -u origin main
</code></pre>
<p>Verify in the browser: <code>https://github.com/YOUR_USERNAME/clearledger</code>.</p>
<p>You should see <code>app/</code>, <code>infra/manifests/</code>, <code>docs/</code>, and <code>.github/workflows/ci.yaml</code>. That confirms GitHub can trigger the pipeline on your next push.</p>
<p><strong>What you proved:</strong> the <strong>app repo</strong> is on GitHub. CI will run from here. Deployment manifests for GitOps land in <code>clearledger-infra</code> in §1.3.</p>
<h3 id="heading-12-install-the-self-hosted-runner-inside-the-vm">1.2: Install the Self-Hosted Runner Inside the VM</h3>
<p>The workflow file tells GitHub <em>what</em> to run. The runner is <em>where</em> it runs.</p>
<p>This lab uses a self-hosted runner because your infrastructure is local. GitHub's cloud servers can't reach your MicroK8s cluster or Docker daemon inside the Multipass VM. The runner solves that by living inside the VM. It connects to GitHub to pick up jobs, then executes everything locally.</p>
<p>If the runner is missing or offline, the pipeline can't execute. The workflow may sit queued, or it may fail because no matching runner is available.</p>
<h4 id="heading-step-1-open-githubs-runner-setup-page-keep-this-tab-open">Step 1: Open GitHub’s runner setup page (keep this tab open)</h4>
<p>GitHub gives you a full copy-paste install guide on one page. Use it: don’t hunt for URLs or tokens elsewhere.</p>
<ol>
<li>Open <code>https://github.com/YOUR_USERNAME/clearledger</code></li>
<li>Go to Settings, Actions, Runners, and New self-hosted runner</li>
<li>Select Linux and x64</li>
</ol>
<p>The page title should look like: <strong>Add new self-hosted runner · YOUR_USERNAME/clearledger</strong>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e0aa537c-a2b2-4ed7-8b48-f686a012ea8d.png" alt="e0aa537c-a2b2-4ed7-8b48-f686a012ea8d" style="display: block;" width="600" height="400" loading="lazy">

<p>That page has three sections you'll use:</p>
<table>
<thead>
<tr>
<th>Section on GitHub</th>
<th>What to do with it</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Download</strong></td>
<td>Copy the <code>mkdir</code>, <code>curl</code>, and <code>tar</code> commands into the VM in Step 4 (same versions as below)</td>
</tr>
<tr>
<td><strong>Configure</strong></td>
<td>Copy the <strong>token</strong> from the <code>./config.sh ... --token ...</code> line: do <strong>not</strong> run GitHub’s <code>./config.sh</code> as-is</td>
</tr>
<tr>
<td><strong>Using your self-hosted runner</strong></td>
<td>Ignore for now, the lab workflow needs the <code>clearledger</code> label (Step 4)</td>
</tr>
</tbody></table>
<p>Scroll to <strong>Configure</strong>. You'll see something like:</p>
<pre><code class="language-bash">./config.sh --url https://github.com/YOUR_USERNAME/clearledger --token AXXXXXXXXXXXXXXXXXXXXXXXXX
./run.sh
</code></pre>
<p>The token is the long string after <code>--token</code> (starts with <code>A</code>, about 26 characters). Copy only that string.</p>
<p>Keep this tab open until Step 4 finishes: the token expires in about <strong>1 hour</strong>. If it expires, click New self-hosted runner again for a fresh token.</p>
<h4 id="heading-step-2-enter-the-vm">Step 2: Enter the VM</h4>
<p><code>multipass shell clearledger</code></p>
<p>After this command, your prompt should look like <code>ubuntu@clearledger:~$</code>. That means you are inside the Ubuntu VM. If your prompt still shows your Mac username or MacBook name, you're still on your host machine and the runner setup will fail.</p>
<p>Continue only when your prompt shows <code>ubuntu@clearledger</code>.</p>
<p>Everything from Step 3 onwards runs inside the VM, not on your Mac.</p>
<h4 id="heading-step-3-install-docker-inside-the-vm">Step 3: Install Docker inside the VM</h4>
<p>The runner will build Docker images. That means Docker must exist where the runner runs.</p>
<pre><code class="language-bash">curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker ubuntu
newgrp docker

docker --version
</code></pre>
<p>Expected: Docker prints a version number (for example, <code>Docker version 29.x.x</code>).</p>
<p><strong>Verify Docker works for the</strong> <code>ubuntu</code> <strong>user now</strong>: the runner doesn't exist yet (Step 4 creates <code>~/actions-runner</code>):</p>
<pre><code class="language-bash">docker ps
</code></pre>
<p>Expected: a table header (CONTAINER ID, IMAGE, …), even if no containers are listed. <strong>Not</strong> <code>permission denied while trying to connect to the Docker API</code>.</p>
<p>If <code>docker ps</code> fails with permission denied, the <code>docker</code> group has not applied yet. Run <code>newgrp docker</code> again, or log out of the VM (<code>exit</code>) and <code>multipass shell clearledger</code> back in, then retry <code>docker ps</code>.</p>
<p><strong>What you proved:</strong> the VM can run Docker without Docker Desktop on your Mac. Continue to Step 4 to install the runner.</p>
<h4 id="heading-step-4-install-and-register-the-runner">Step 4: Install and register the runner</h4>
<p>Still inside the VM (<code>ubuntu@clearledger</code> prompt):</p>
<p><strong>Download:</strong> you can copy the commands from the <strong>Download</strong> section on GitHub’s runner page (Step 1), or run the block below. They should match. Paste into the VM, not your Mac.</p>
<p><strong>Configure:</strong> use the lab command below, not GitHub’s <code>./config.sh</code> line. Paste your token from Step 1 and replace <code>YOUR_USERNAME</code>.</p>
<pre><code class="language-bash">mkdir -p ~/actions-runner &amp;&amp; cd ~/actions-runner

curl -o actions-runner-linux-x64-2.335.1.tar.gz -L \
  https://github.com/actions/runner/releases/download/v2.335.1/actions-runner-linux-x64-2.335.1.tar.gz

tar xzf ./actions-runner-linux-x64-2.335.1.tar.gz

./config.sh \
  --url https://github.com/YOUR_USERNAME/clearledger \
  --token YOUR_RUNNER_TOKEN \
  --name clearledger-runner \
  --labels clearledger,self-hosted,linux \
  --work _work \
  --unattended

sudo ./svc.sh install
sudo ./svc.sh start
</code></pre>
<p>Do <strong>not</strong> run GitHub’s <code>./run.sh</code> for day-to-day use: the lab uses <code>sudo ./svc.sh</code> so the runner survives VM reboots. GitHub shows <code>./run.sh</code> for a quick test only.</p>
<p>Expected after <code>./config.sh</code>: <code>Runner successfully added</code> (or similar). If you see Invalid token or Expired token, go back to Step 1 in the browser and copy a fresh token.</p>
<p>The <code>clearledger</code> label is required GitHub’s default <code>./config.sh</code> on the setup page doesn't add it. The workflow uses:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f8fa25ff-81da-4759-b021-293c244add7c.png" alt="image showing where to add the label in github ui for the runner" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-yaml">runs-on: [self-hosted, clearledger]
</code></pre>
<p>GitHub schedules jobs by runner labels, not by runner name. A runner named <code>clearledger</code> without the <code>clearledger</code> label will stay online but jobs will remain queued with <code>Waiting for a runner to pick up this job</code>.</p>
<p>What those last two commands mean:</p>
<pre><code class="language-text">sudo ./svc.sh install
  Registers the runner with systemd inside the VM.
  Without this, `sudo ./svc.sh status` says: not installed.

sudo ./svc.sh start
  Starts the runner service in the background.
  After this, it keeps running even when you close the terminal.
</code></pre>
<p>Check it locally from the same folder, still inside the VM:</p>
<pre><code class="language-bash">cd ~/actions-runner
sudo ./svc.sh status
</code></pre>
<p>Expected: the service is installed and running.</p>
<p>If <code>docker ps</code> worked in Step 3 but a CI job later fails with Docker socket permission denied, the runner probably started before the <code>docker</code> group applied. Restart it after Step 4 (only when <code>~/actions-runner</code> exists):</p>
<pre><code class="language-bash">cd ~/actions-runner
sudo ./svc.sh stop
sudo ./svc.sh start
docker ps    # must work without sudo
</code></pre>
<p>Or, if you started the runner manually with <code>./run.sh</code> instead of systemd:</p>
<pre><code class="language-bash">cd ~/actions-runner
pkill -f "Runner.Listener|Runner.Worker|./run.sh" || true
nohup ./run.sh &gt; _diag/manual-runner.log 2&gt;&amp;1 &amp;
docker ps
</code></pre>
<p>If you see this:</p>
<pre><code class="language-text">not installed
</code></pre>
<p>then <code>sudo ./svc.sh install</code> didn't run successfully. Run:</p>
<pre><code class="language-bash">cd ~/actions-runner
sudo ./svc.sh install
sudo ./svc.sh start
sudo ./svc.sh status
</code></pre>
<p>If <code>install</code> fails, rerun <code>./config.sh</code> with a fresh GitHub runner token, then run the install/start commands again.</p>
<h4 id="heading-step-5-exit-the-vm">Step 5: Exit the VM</h4>
<pre><code class="language-bash">exit
</code></pre>
<h4 id="heading-step-6-verify-the-runner-is-connected">Step 6: Verify the runner is connected</h4>
<p>Go to <a href="http://github.com/YOUR">github.com/YOUR</a>_USERNAME/clearledger then to Settings, Actions, and Runners.</p>
<p>You should see <code>clearledger-runner</code> with a green dot and status <strong>Idle</strong>. Open the runner details and confirm the labels include:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/f47d23ff-fecf-4789-b8ee-3be4191c1c3a.png" alt="screenshot image of github ui shpwing runner status as &quot;idle&quot; green" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-text">self-hosted
Linux
X64
clearledger
</code></pre>
<p>If <code>clearledger</code> is missing, add it in the runner settings before rerunning the workflow. The runner name alone is not enough.</p>
<p><strong>✋ Hands-on checkpoint: runner ready for jobs</strong></p>
<p>Still on GitHub, Settings, Actions, and Runners, confirm:</p>
<table>
<thead>
<tr>
<th>Field</th>
<th>Expected</th>
</tr>
</thead>
<tbody><tr>
<td>Status</td>
<td><strong>Idle</strong> (green)</td>
</tr>
<tr>
<td>Labels</td>
<td>includes <code>self-hosted</code> <strong>and</strong> <code>clearledger</code></td>
</tr>
<tr>
<td>OS</td>
<td>Linux</td>
</tr>
</tbody></table>
<p>Then trigger a dry run from your laptop:</p>
<pre><code class="language-bash">git commit --allow-empty -m "test: verify runner picks up jobs"
git push
</code></pre>
<p>Open <code>https://github.com/YOUR_USERNAME/clearledger/actions</code>. Within 30 seconds a workflow run should show Queued then In progress, not stuck on “Waiting for a runner.” If it waits more than 2 minutes, the labels are wrong. Edit the runner on GitHub and add <code>clearledger</code>.</p>
<p><strong>If it shows Offline:</strong></p>
<pre><code class="language-bash">multipass exec clearledger -- sudo systemctl status actions.runner.*.service
multipass exec clearledger -- journalctl -u actions.runner.*.service --lines=50
</code></pre>
<p><strong>What you proved:</strong> GitHub can now send work into your local lab environment.</p>
<h3 id="heading-13-create-the-infra-repo-on-github">1.3: Create the Infra Repo on GitHub</h3>
<p>Now separate <strong>application code</strong> from <strong>deployment state</strong>. Stage 1 introduces a second GitHub repository alongside the <code>clearledger</code> app repo you pushed in §1.1.</p>
<p>You'll use two repositories for the rest of the lab:</p>
<table>
<thead>
<tr>
<th>Repo</th>
<th>What lives there</th>
<th>Who changes it</th>
<th>Why it exists</th>
</tr>
</thead>
<tbody><tr>
<td><code>clearledger</code></td>
<td>App source code, Dockerfiles, tests, <code>.github/workflows/ci.yaml</code>, lab docs</td>
<td>You, the developer</td>
<td>This is where code changes start</td>
</tr>
<tr>
<td><code>clearledger-infra</code></td>
<td>Kubernetes manifests only: <code>deployment.yaml</code>, <code>service.yaml</code>, ingress, secrets templates</td>
<td>The CI pipeline, then ArgoCD reads it</td>
<td>This is the desired state of the cluster</td>
</tr>
</tbody></table>
<p>Think of <code>clearledger</code> as the question <em>“What is the application?”</em>. Python services, Dockerfiles, tests, and the CI workflow. Think of <code>clearledger-infra</code> as <em>“What exact version should be running in Kubernetes right now?”</em>. Deployments, Services, ingress rules, and the image tags that point at Docker Hub.</p>
<p>Teams split these on purpose. If you edit <code>README.md</code> in <code>clearledger</code>, that is a documentation change. It shouldn't trigger a deployment.<br>If you change <code>auth-service</code> code, the pipeline builds a new image (for example tag <code>abc123</code>) and, only after scans pass, records that tag in <code>clearledger-infra</code>:</p>
<pre><code class="language-yaml">image: $DOCKER_USERNAME/clearledger-auth-service:abc123
</code></pre>
<p>That line is a deployment contract: Git now says the cluster <em>should</em> run <code>abc123</code>. In Stage 1, the cluster doesn't change yet (and you'll prove that in §1.6).<br>In Stage 2, ArgoCD watches <code>clearledger-infra</code>, compares Git to what is running, and syncs the cluster when they differ. The app repo is where work begins. The infra repo is what production is supposed to look like.</p>
<h4 id="heading-private-repos-what-syncs-where">Private repos: what syncs where</h4>
<p>This lab uses two GitHub repos. <code>clearledger</code> is your main project repo: app code, CI pipeline, docs, policies, and lab files. This repo can be private.</p>
<p><code>clearledger-infra</code> contains only Kubernetes manifests. ArgoCD watches this repo and uses it to deploy the app. For beginners, make this repo public so ArgoCD can read it without extra authentication.</p>
<p>The flow looks like this:</p>
<pre><code class="language-text">clearledger
app code + infra/manifests/
        ↓
CI copies infra/manifests/
        ↓
clearledger-infra
Kubernetes manifests only
        ↓
ArgoCD syncs from this repo
        ↓
Kubernetes cluster
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/4aa15b2c-c746-4711-aa64-704a9d3eada2.png" alt="flow chart explain how both repos work" style="display: block;" width="600" height="400" loading="lazy">

<p>ArgoCD doesn't read the main <code>clearledger</code> repo. It only reads <code>clearledger-infra</code>. If <code>clearledger</code> is private, that is fine. If <code>clearledger-infra</code> is private, you must give ArgoCD GitHub credentials later. If you do not, ArgoCD may show <code>ComparisonError</code>.</p>
<p>Create the infra repo on GitHub:</p>
<ol>
<li>Go to GitHub and then <strong>New Repository</strong></li>
<li>Name it <code>clearledger-infra</code></li>
<li>Choose <strong>Public</strong></li>
<li>Don't add a README</li>
<li>Click <strong>Create</strong></li>
</ol>
<p>Later, the CI pipeline will update <code>clearledger-infra</code> automatically. In Stage 1, the pipeline doesn't run <code>kubectl apply</code> – it updates Git. In Stage 2, ArgoCD reads that Git repo and applies it to the cluster.</p>
<p><strong>Before pushing:</strong> set your Docker Hub username in Kustomize (image tags are resolved here, not in deployment YAML):</p>
<pre><code class="language-bash"># Replace YOUR_DOCKERHUB_USERNAME with the same value as $DOCKER_USERNAME from §0.3
sed -i.bak "s/YOUR_DOCKERHUB_USERNAME/${DOCKER_USERNAME}/g" infra/manifests/kustomization.yaml
rm -f infra/manifests/kustomization.yaml.bak
</code></pre>
<p>Push only the Kubernetes manifests from <code>infra/manifests/</code> (not everything under <code>infra/</code>):</p>
<pre><code class="language-bash">mkdir -p /tmp/clearledger-infra
cp -r infra/manifests /tmp/clearledger-infra/
cd /tmp/clearledger-infra
git init
git remote add origin https://github.com/YOUR_USERNAME/clearledger-infra.git
git add . &amp;&amp; git commit -m "feat: initial manifests" &amp;&amp; git push -u origin main
cd -
</code></pre>
<p><strong>✋ Hands-on checkpoint: infra repo on GitHub (do this before §1.4)</strong></p>
<p>On your laptop:</p>
<pre><code class="language-bash">grep "docker.io/${DOCKER_USERNAME}/" infra/manifests/kustomization.yaml | wc -l
grep YOUR_DOCKERHUB_USERNAME infra/manifests/kustomization.yaml || echo "OK: placeholder replaced"
</code></pre>
<p>Expected: first command prints <code>4</code> (four image lines). Second prints <code>OK: placeholder replaced</code>, not four lines still saying <code>YOUR_DOCKERHUB_USERNAME</code>.</p>
<p>In the browser, open <code>https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests</code> and confirm <strong>with your eyes</strong>:</p>
<table>
<thead>
<tr>
<th>File / folder</th>
<th>Must exist</th>
</tr>
</thead>
<tbody><tr>
<td><code>kustomization.yaml</code></td>
<td>Yes. Open it: <code>newName:</code> lines use <strong>your</strong> Docker Hub user</td>
</tr>
<tr>
<td><code>auth-service/secret.yaml</code></td>
<td>Yes. Stages 2–4 need this until Stage 5</td>
</tr>
<tr>
<td><code>ledger-service/secret.yaml</code></td>
<td>Yes</td>
</tr>
<tr>
<td><code>auth-service/deployment.yaml</code></td>
<td>Yes. Open it: must contain <code>secretKeyRef</code>, <strong>not</strong> <code>vault.hashicorp.com</code></td>
</tr>
<tr>
<td><code>netpol/</code></td>
<td><strong>No</strong>. If present, delete the folder on GitHub before Stage 2</td>
</tr>
<tr>
<td><code>vault/</code></td>
<td><strong>No</strong>. Vault rotation is Stage 5 only</td>
</tr>
</tbody></table>
<p><strong>Which folders matter?</strong> You only pushed <code>infra/manifests/</code> to GitHub, that's correct. Everything else in this repo stays local for now.</p>
<p>Some manifests for later stages (network policies, Vault extras) live under <code>infra/deferred-by-stage/</code> in the <code>clearledger</code> repo. You'll apply those by hand when you reach that stage. Do <strong>not</strong> copy that folder into <code>clearledger-infra</code>, or ArgoCD will deploy things too early.</p>
<p>You might notice <code>stages/stage-1-ci-pipeline/</code> has no copy of the manifests. That is normal: the lab doesn't duplicate YAML there. The canonical copy is <code>infra/manifests/</code> in this repo, and the live GitOps copy is <code>clearledger-infra</code> on GitHub.</p>
<p><strong>What you proved:</strong> Kubernetes config now has its own repo and Git history, separate from application code. CI will update <code>clearledger-infra</code> after each build, and your app repo stays for code and the pipeline file.</p>
<h3 id="heading-14-set-up-github-secrets">1.4: Set up GitHub Secrets</h3>
<p>Go to <code>github.com/YOUR_USERNAME/clearledger</code> and then Settings, Secrets and variables, Actions, and New repository secret.</p>
<p>The workflow needs credentials for Docker Hub, GitHub, and image signing:</p>
<ul>
<li>Docker Hub, so it can push images.</li>
<li>GitHub, so it can push image tag updates into <code>clearledger-infra</code>.</li>
<li>Cosign, so it can sign the images after pushing them.</li>
</ul>
<p>Do <strong>not</strong> paste these values into YAML files. Store them as GitHub Actions secrets.</p>
<h4 id="heading-secret-1-dockerusername">Secret 1, <code>DOCKER_USERNAME</code></h4>
<p>This is just your Docker Hub username.</p>
<p>Example:</p>
<pre><code class="language-text">veeno-demo
</code></pre>
<p>Get it from Docker Hub: <a href="http://hub.docker.com">hub.docker.com</a>, profile menu, Account Settings.</p>
<h4 id="heading-secret-2-dockerpassword">Secret 2, <code>DOCKER_PASSWORD</code></h4>
<p>This should be a Docker Hub <strong>access token</strong>, not your normal Docker Hub password.</p>
<p>Create it here:</p>
<pre><code class="language-text">hub.docker.com
→ Account Settings
→ Security
→ New Access Token
→ Description: clearledger-github-actions
→ Access permissions: Read, Write, Delete or Read/Write
→ Generate
</code></pre>
<p>Copy the token immediately. Docker Hub only shows it once.</p>
<h4 id="heading-secret-3-infrarepotoken">Secret 3, <code>INFRA_REPO_TOKEN</code></h4>
<p>This is a GitHub Personal Access Token (PAT). The pipeline uses it to push commits to the second repo, <code>clearledger-infra</code>.</p>
<p>Create it here:</p>
<pre><code class="language-text">GitHub profile settings
→ Settings
→ Developer settings
→ Personal access tokens
→ Tokens (classic)
→ Click "Generate new token"
→ Choose "Generate new token (classic)"
→ If GitHub asks for your password or 2FA, complete it
→ Note: clearledger-infra-ci
→ Expiration: choose a lab-friendly value
→ Select scope: repo
   This allows the pipeline to push to clearledger-infra.
→ Generate token
</code></pre>
<p>Copy the token immediately. GitHub only shows it once.</p>
<p>For this lab, <code>repo</code> scope is the simplest option. In production, you would use tighter permissions, such as a fine-grained token limited to only <code>clearledger-infra</code>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/5a0d06bf-570a-4360-aadb-038b7ae7ed4e.png" alt="screenshot of docker ui showing where to set up PAT" style="display: block;" width="600" height="400" loading="lazy">

<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/b9e7f0c8-882c-4d15-b67b-d2ff38289836.png" alt="screenshot of docker ui showing where to set up token scope" style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-secrets-4-and-5-cosignprivatekey-and-cosignpassword">Secrets 4 and 5, <code>COSIGN_PRIVATE_KEY</code> and <code>COSIGN_PASSWORD</code></h4>
<p>Cosign signs container images after the pipeline pushes them to Docker Hub. Later, Stage 4 uses the public key with Kyverno so the cluster can verify that images came from your trusted pipeline.</p>
<p>Generate the key pair on your host machine, not inside the Multipass VM:</p>
<pre><code class="language-bash"># macOS: brew install cosign
# Linux/WSL2: curl -sSL -o cosign https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 &amp;&amp; chmod +x cosign &amp;&amp; sudo mv cosign /usr/local/bin/
cosign generate-key-pair
</code></pre>
<p>This creates:</p>
<pre><code class="language-text">cosign.key   # private key — never commit this
cosign.pub   # public key — keep for later Kyverno verification
</code></pre>
<p>When Cosign asks for a password, enter one and save it in your password manager. If you already generated a key without a password, regenerate it with a password for this lab.</p>
<p>Add these five secrets to the <code>clearledger</code> repo, not <code>clearledger-infra</code>:</p>
<table>
<thead>
<tr>
<th>Secret name</th>
<th>Value</th>
<th>Purpose</th>
</tr>
</thead>
<tbody><tr>
<td><code>DOCKER_USERNAME</code></td>
<td>Your Docker Hub username</td>
<td>Pipeline logs in to push images</td>
</tr>
<tr>
<td><code>DOCKER_PASSWORD</code></td>
<td>Your Docker Hub access token</td>
<td>Pipeline authenticates with Docker Hub</td>
</tr>
<tr>
<td><code>INFRA_REPO_TOKEN</code></td>
<td>The GitHub PAT from above</td>
<td>Pipeline pushes image tag updates to clearledger-infra</td>
</tr>
<tr>
<td><code>COSIGN_PRIVATE_KEY</code></td>
<td>Contents of <code>cosign.key</code></td>
<td>Pipeline signs pushed container images</td>
</tr>
<tr>
<td><code>COSIGN_PASSWORD</code></td>
<td>Password used when creating the Cosign key</td>
<td>Unlocks the private key during signing</td>
</tr>
</tbody></table>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/638009a9-844b-4bcb-9701-6312ec18d5c6.png" alt="screenshot of github UI showing my repository secrets" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>Repository variables (not secrets)</strong> (optional) toggles for later stages. Add under <strong>Settings, Secrets and variables, Actions, Variables</strong>:</p>
<table>
<thead>
<tr>
<th>Variable</th>
<th>Stage 1</th>
<th>When to enable</th>
</tr>
</thead>
<tbody><tr>
<td><code>ENABLE_ARGOCD_SYNC</code></td>
<td>Leave <strong>unset</strong></td>
<td><strong>Stage 2</strong> — after ArgoCD’s first sync is healthy (see <a href="#heading-how-to-enable-the-ci-to-argocd-handoff">Enable CI → ArgoCD handoff</a>)</td>
</tr>
<tr>
<td><code>ENABLE_DAST</code></td>
<td>Leave <strong>unset</strong></td>
<td><strong>Stage 3</strong> — after the app is live at <code>clearledger.local</code> (see <a href="#heading-enable-dast-optional-after-stage-2">Enable DAST</a>)</td>
</tr>
</tbody></table>
<p>Don't add either variable in Stage 1. If you set them now, CI will try to refresh ArgoCD or run ZAP before the cluster is ready, and the pipeline output gets harder to read. The guide calls out the exact moment to turn each one on – you only need to remember that both exist.</p>
<p><strong>What you proved:</strong> the pipeline can authenticate to external systems without hardcoding credentials in the repo.</p>
<h3 id="heading-15-understand-the-pipeline-before-activating-it">1.5: Understand the Pipeline Before Activating it</h3>
<p>Don't treat the workflow file as magic. Open <code>.github/workflows/ci.yaml</code> and read it before you run it.</p>
<p>The pipeline has two responsibilities:</p>
<ol>
<li>Prove the code and images are safe enough to publish.</li>
<li>Update the infra repo with the new image tags.</li>
</ol>
<p>Here's the security flow first:</p>
<pre><code class="language-text">Developer pushes code to GitHub
        ↓
GitHub Actions starts workflow
        ↓
Self-hosted runner inside the Multipass VM picks up the job
        ↓
1. Scan secrets (Gitleaks)
        ↓
2. Run code security scans (Semgrep) + IaC scan (Checkov) — parallel
        ↓
3. Prepare scanners (install Trivy/Syft/Grype/Cosign once; refresh Trivy DB once)
        ↓
4. BUILD: docker build all four services (local tags only; nothing hits Docker Hub yet)
        ↓
5. SCAN: Trivy on all images; Syft + Grype SBOM on auth-service; upload evidence
        ↓
6. PUBLISH: push to Docker Hub + Cosign sign (only if scan passed)
        ↓
7. UPDATE MANIFESTS: commit new image tags to clearledger-infra
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/0cea04bf-e166-4fd3-9eb5-1a664e427206.png" alt="visual image of the cicd security flow pattern" style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-build-scan-publish-prod-style-gates">Build, scan, publish (prod-style gates)</h4>
<p>Real teams never push first and scan later. The pipeline separates three concerns into three jobs in <code>.github/workflows/ci.yaml</code>:</p>
<table>
<thead>
<tr>
<th>Job</th>
<th>What it does</th>
<th>If it fails…</th>
</tr>
</thead>
<tbody><tr>
<td><code>build-images</code></td>
<td><code>docker build</code> all services with tag <code>${{ github.sha }}</code></td>
<td>No registry pollution, images never left the runner</td>
</tr>
<tr>
<td><code>scan-images</code></td>
<td>Trivy (all 4 images); Syft + Grype (auth only)</td>
<td>Publish is skipped: bad images never reach Docker Hub</td>
</tr>
<tr>
<td><code>publish-images</code></td>
<td>Runs <code>scripts/ci-publish-image.sh</code> tag, push, Cosign sign</td>
<td>Only runs after scan passes</td>
</tr>
</tbody></table>
<p>You do <strong>not</strong> run <code>scripts/ci-publish-image.sh</code> yourself before pushing code. GitHub Actions checks out the repo and calls it inside <code>publish-images</code>.</p>
<p><strong>Why can</strong> <code>build-images</code> <strong>and</strong> <code>scan-images</code> <strong>be separate jobs?</strong> Each job is a fresh checkout on GitHub-hosted runners. They don't share a disk. On <strong>your</strong> self-hosted runner, all three jobs run on the <strong>same Multipass VM</strong> and use the <strong>same Docker engine</strong>.</p>
<p>Job 1 runs <code>docker build</code> and leaves the images on that machine. Job 2 runs Trivy against those same local images: no upload, no download. Job 3 pushes to Docker Hub only if the scan passed.</p>
<p>That's a practical lab setup: one persistent build machine with Docker installed, like a dedicated CI worker in a real office. In <strong>Stage 8 (AWS)</strong>, the pipeline uses GitHub-hosted runners instead: there, <code>build-images</code> saves the images to a file (<code>images.tar</code>) and passes that file to the next job as a workflow artifact, because those runners are throwaway VMs with no shared Docker cache.</p>
<p>Then comes the GitOps handoff:</p>
<pre><code class="language-text">Secure images now exist in Docker Hub
        ↓
Runner checks out clearledger-infra from GitHub
        ↓
Deployment YAML image tags are updated
        ↓
Runner commits and pushes back to clearledger-infra
        ↓
Stage 1 ends here
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3be226da-a7d5-4231-8110-c37f1b8bfdce.png" alt="visual image of the cicd security flow pattern and github handoff journey" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>Here's how the image tag ties to your code:</strong> every pipeline run is triggered by a git commit. GitHub gives that commit a unique ID called the <strong>SHA</strong> (a long hex string like <code>a1b2c3d4e5f6789…</code>). The workflow sets <code>IMAGE_TAG</code> to that SHA and uses it everywhere:</p>
<ol>
<li><strong>Build:</strong> <code>docker build -t clearledger-auth-service:a1b2c3d4…</code></li>
<li><strong>Publish:</strong> push to Docker Hub as <code>YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4…</code></li>
<li><strong>Update manifests:</strong> <code>kustomize edit set image …:a1b2c3d4…</code> in <code>clearledger-infra</code></li>
<li><strong>Commit message:</strong> <code>ci: deploy a1b2c3d4… — all gates passed</code></li>
</ol>
<p>If production is running <code>YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4</code>, you can copy that <code>a1b2c3d4</code> tag, open GitHub, and instantly find the exact commit that built that image. There's no guessing and no wondering if <code>latest</code> changed. Every deployed image points back to one specific version of the code, making rollbacks and debugging much easier.</p>
<h4 id="heading-the-kustomize-placeholder">The Kustomize placeholder</h4>
<p><code>auth-service/deployment.yaml</code> uses a label instead of a real image address:</p>
<pre><code class="language-yaml">image: clearledger/auth-service:gitops
</code></pre>
<p>That label isn't on Docker Hub. It tells Kustomize where to substitute. The real address lives in <code>kustomization.yaml</code>:</p>
<pre><code class="language-yaml">images:
  - name: clearledger/auth-service          # matches the label above
    newName: docker.io/YOUR_DOCKERHUB_USERNAME/clearledger-auth-service
    newTag: abc123def456…                   # real commit SHA — CI writes this
</code></pre>
<p>When ArgoCD deploys, <code>kustomize build</code> swaps the label for the full address.</p>
<p>You edit <code>kustomization.yaml</code> once in §1.3 to set your Docker Hub username in <code>newName:</code>. After that, CI writes <code>newTag:</code> automatically on every green push. You never touch it by hand.</p>
<h4 id="heading-stage-1-ci-updates-github-not-the-cluster">Stage 1: CI updates GitHub, not the cluster</h4>
<p>After a green pipeline run, three things are true:</p>
<ul>
<li>New images exist on Docker Hub</li>
<li><code>clearledger-infra</code> on GitHub has new SHAs in <code>kustomization.yaml</code></li>
<li>Your Kubernetes cluster is <strong>unchanged</strong>. Still running whatever Stage 0 left there</li>
</ul>
<p>CI never runs <code>kubectl apply</code>. It only commits to <code>clearledger-infra</code>. That's the whole Stage 1 lesson: build and scan are automated, but <strong>deploy</strong> is not: yet. Stage 2 installs ArgoCD, which reads <code>clearledger-infra</code> and updates the cluster for you.</p>
<p><strong>Kubernetes Checkov</strong> runs in Stage 1 but does <strong>not</strong> block the pipeline. It uploads findings so you can see hardening work ahead. Stage 4 turns those kinds of rules into cluster enforcement with Kyverno.</p>
<p>Jobs run on your self-hosted runner (<code>runs-on: [self-hosted, clearledger]</code>). Both <code>ENABLE_ARGOCD_SYNC</code> and <code>ENABLE_DAST</code> are unset in Stage 1. See §1.4 for when each gets flipped.</p>
<p>If a job fails, start with <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md"><code>docs/troubleshooting.md</code></a> before editing the workflow.</p>
<h4 id="heading-stage-1-security-posture-what-blocks-vs-what-waits">Stage 1 security posture: what blocks vs what waits</h4>
<p>Note that stage 1 is not “security off.” Some gates stop the pipeline while others run for evidence and tighten in later stages.</p>
<p><strong>Blocks the pipeline today:</strong></p>
<ul>
<li>Gitleaks (secrets in Git)</li>
<li>Semgrep (SAST on Python)</li>
<li>Checkov on Dockerfiles</li>
<li>Trivy (fixable HIGH/CRITICAL CVEs in images)</li>
<li>Grype on auth-service SBOM (fixable HIGH+)</li>
<li>Manifest update to <code>clearledger-infra</code> (must succeed)</li>
</ul>
<p><strong>Runs but doesn't block yet:</strong></p>
<ul>
<li>Checkov on Kubernetes manifests – enforced in <strong>Stage 4</strong> (Kyverno)</li>
<li>Cosign sign + SLSA attest – enforced in <strong>Stage 4</strong> (unsigned images rejected)</li>
<li>Syft SBOM generation – supply-chain evidence. You'll purposely break gates in <strong>Stage 3.</strong></li>
<li>ArgoCD refresh – <strong>Stage 2</strong> (<code>ENABLE_ARGOCD_SYNC=true</code>)</li>
<li>DAST / ZAP – <strong>Stage 3</strong> (<code>ENABLE_DAST=true</code>)</li>
</ul>
<p><strong>If you forget which stage fixes what</strong>, search this guide for “Stage 1 security posture” or follow the stage order: Stage 3 breaks gates on purpose, Stage 4 connects Checkov findings to Kyverno, Stage 5 moves secrets off Git, Stage 6 adds runtime detection, Stage 7 adds monitoring dashboards.</p>
<p>Run <code>make check-3</code> and <code>make check-4</code> after those stages to confirm hardening landed.</p>
<p><strong>Design intent:</strong> Stage 1 proves CI can build, scan, push, and update Git without you touching Docker manually. Later stages turn evidence into enforcement. The relaxations here are deliberate.</p>
<h3 id="heading-16-activate-the-pipeline">1.6: Activate the Pipeline</h3>
<p><strong>Run this in the</strong> <code>clearledger</code> <strong>app repo, not</strong> <code>clearledger-infra</code><strong>.</strong></p>
<p>§1.3 created <code>clearledger-infra</code> with only Kubernetes manifests. It has no <code>.github/workflows/</code> and no pipeline. If your shell prompt says <code>clearledger-infra</code>, or you used <code>/tmp/clearledger-infra</code>, you're in the wrong place.</p>
<pre><code class="language-bash">cd /path/to/clearledger    # the app repo you pushed in §1.1

git remote -v              # must show .../clearledger.git — NOT clearledger-infra

ls .github/workflows/ci.yaml   # must exist before you commit
</code></pre>
<p>The pipeline file already lives at <code>.github/workflows/ci.yaml</code>. Push any small change to <code>clearledger</code> on <code>main</code>:</p>
<pre><code class="language-bash">echo "# Pipeline activated $(date)" &gt;&gt; README.md
git add README.md
git commit -m "ci: activate GitHub Actions pipeline"
git push origin main
</code></pre>
<p>Watch the run at: <code>https://github.com/YOUR_USERNAME/clearledger/actions</code> (app repo Actions tab, not the infra repo).</p>
<p>When the pipeline succeeds, it updates <code>clearledger-infra</code> for you. You don't need to push anything to the infra repo by hand for this step.</p>
<p>Expected Output: all jobs green in about 8 minutes.</p>
<pre><code class="language-plaintext">✓ Build + Scan auth-service
✓ Build + Scan ledger-service
✓ Build + Scan notification-service
✓ Build + Scan frontend
✓ Update manifests → GitHub
</code></pre>
<p>DAST and the ArgoCD refresh step show as <strong>skipped</strong>: this is expected, because Both toggles are unset until later (see §1.4).</p>
<p><strong>Note:</strong> this lab includes <code>.gitleaksignore</code> because some intentional demo secrets are already present in Git history. Gitleaks still runs normally. The ignore file only suppresses known lab fingerprints. Don't add new findings to it unless you've confirmed they're intentional test data.</p>
<p>Click into the job logs and look for the story. Don't just wait for green:</p>
<ul>
<li>Docker login succeeded</li>
<li>Each service image built and pushed to Docker Hub</li>
<li><code>clearledger-infra</code> was checked out</li>
<li>Deployment YAMLs were updated with the new SHA tag</li>
<li>A commit was pushed back to <code>clearledger-infra</code></li>
</ul>
<p>After the pipeline succeeds, open <code>https://github.com/YOUR_USERNAME/clearledger-infra</code> and look at the deployment manifests. The image tags should now use the current commit SHA.</p>
<p>Now check the cluster:</p>
<pre><code class="language-bash">kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}' &amp;&amp; echo
</code></pre>
<p>You may still see the old image. That's expected. This is the most important learning in Stage 1:</p>
<pre><code class="language-text">GitHub pipeline succeeded.
Docker Hub has new images.
clearledger-infra has new image tags.
The Kubernetes cluster did not update automatically.
</code></pre>
<p>That's not a failure. It's the deployment gap. Stage 1 automated the build, but no controller is watching the infra repo yet. Stage 2 installs ArgoCD to close that gap.</p>
<h3 id="heading-17-hands-on-checkpoint-prove-stage-1-is-really-done">1.7 — Hands-on Checkpoint: Prove Stage 1 is Really Done</h3>
<p>Don't rely on a green workflow badge alone. Run each check yourself:</p>
<h4 id="heading-1-infra-repo-still-has-app-secrets-critical-for-stage-2">1. Infra repo still has app secrets (critical for Stage 2)</h4>
<p>Open <code>https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests/auth-service</code>, <code>secret.yaml</code> must be visible.</p>
<p>On your laptop:</p>
<pre><code class="language-bash">git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra
grep secretKeyRef /tmp/verify-infra/manifests/auth-service/deployment.yaml
grep secret.yaml /tmp/verify-infra/manifests/kustomization.yaml
rm -rf /tmp/verify-infra
</code></pre>
<p>Expected: <code>secretKeyRef</code> in deployment output. kustomization lists <code>auth-service/secret.yaml</code> and <code>ledger-service/secret.yaml</code>. If secrets are missing, re-push §1.3 manifests before Stage 2.</p>
<h4 id="heading-2-kustomize-image-tags-updated-by-ci">2. Kustomize image tags updated by CI</h4>
<pre><code class="language-bash">git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra
grep newTag /tmp/verify-infra/manifests/kustomization.yaml
rm -rf /tmp/verify-infra
</code></pre>
<p>Expected: <code>newTag</code> is a 40-character git SHA (or your commit hash), not still <code>v0.1.0</code> only: unless you haven't pushed since §0.3.</p>
<h4 id="heading-3-docker-hub-has-signed-images-from-this-pipeline">3. Docker Hub has signed images from this pipeline</h4>
<p>Open <a href="http://hub.docker.com">hub.docker.com</a> then <code>clearledger-auth-service</code> then <strong>Tags</strong>. The latest tag should match the SHA from step 2.</p>
<h4 id="heading-4-cluster-unchanged-deployment-gap-intentional">4. (Cluster unchanged (deployment gap) intentional)</h4>
<pre><code class="language-bash">kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}' &amp;&amp; echo
</code></pre>
<p>Expected: still your <strong>Stage 0</strong> tag (for example, <code>veeno-demo/clearledger-auth-service:v0.1.0</code>), not the new SHA. That proves CI didn't touch the cluster.</p>
<h4 id="heading-5-runner-still-idle">5. Runner still idle</h4>
<p>GitHub, Settings, Actions, Runners, <code>clearledger-runner</code>, <strong>Idle</strong>.</p>
<pre><code class="language-bash">make check-1
</code></pre>
<p>All five pass, onto Stage 2.</p>
<h3 id="heading-what-you-learned-in-stage-1">What You Learned in Stage 1</h3>
<ul>
<li><strong>CI removes your laptop from the build process.</strong> Builds become repeatable, visible, and tied to Git commits.</li>
<li><strong>A runner is the worker, not the pipeline itself.</strong> GitHub schedules the job, the self-hosted runner executes it inside your VM.</li>
<li><strong>Artifacts and desired state are different things.</strong> Docker Hub stores built images. <code>clearledger-infra</code> on GitHub stores the Kubernetes manifests that say which image should run.</li>
<li><strong>Good pipelines don't secretly mutate clusters.</strong> This pipeline updates Git instead of running <code>kubectl</code>.</li>
<li><strong>The gap that remains:</strong> the infra repo changed, but the cluster didn't. Someone still has to apply the change manually. Stage 2 fixes that with GitOps.</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Built a CI pipeline on a self-hosted GitHub Actions runner that builds and pushes container images on every push, and can debug a workflow that fails before any job is created.</p>
</blockquote>
<p><code>make snapshot STAGE=1 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage1</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<h2 id="heading-stage-2-gitops-with-argocd">Stage 2 — GitOps with ArgoCD</h2>
<p>From this point on, Git is in charge. Whatever is written in the infrastructure repository is what should be running. If someone changes the cluster by hand, ArgoCD notices the difference and changes it back to match Git.</p>
<p><strong>Goal:</strong> Install ArgoCD so it watches <code>clearledger-infra</code> and deploys changes to the cluster. The CI pipeline only updates the Git repository, it never connects to Kubernetes or runs <code>kubectl</code> commands.</p>
<p>Here's a more conversational, compressed version:</p>
<h3 id="heading-am-i-ready-for-stage-2">Am I ready for Stage 2?</h3>
<p>Before moving on, finish <strong>§1.6</strong>, then run:</p>
<pre><code class="language-bash">make check-1

grep secretKeyRef infra/manifests/auth-service/deployment.yaml

grep vault.hashicorp infra/manifests/auth-service/deployment.yaml &amp;&amp; echo "STOP: Vault annotations present" || echo "OK"
</code></pre>
<p>You should see:</p>
<ul>
<li><code>check-1</code> passes</li>
<li><code>secretKeyRef</code> is present</li>
<li><code>OK</code> (no Vault annotations yet)</li>
</ul>
<p>Quick checklist:</p>
<ul>
<li><code>clearledger-infra</code> contains <code>auth-service/secret.yaml</code> and <code>ledger-service/secret.yaml</code></li>
<li>Your self-hosted runner is <strong>Idle</strong> with the <code>clearledger</code> label</li>
<li><code>ENABLE_ARGOCD_SYNC</code> isn't set yet (you'll enable it after installing ArgoCD)</li>
</ul>
<p>You're done with Stage 2 when <code>make check-2</code> passes and <a href="http://argocd.local"><code>http://argocd.local</code></a> shows ArgoCD syncing <code>clearledger</code>.</p>
<p>Finally, save your progress:</p>
<pre><code class="language-bash">make snapshot STAGE=2
make snapshots
</code></pre>
<p>Confirm that <code>clearledger.stage2</code> appears in the snapshot list.</p>
<h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3>
<p><strong>The gap from Stage 1:</strong> CI already builds images and updates <code>clearledger-infra</code>. The cluster didn't change until someone ran <code>kubectl</code>. This stage closes that last step.</p>
<table>
<thead>
<tr>
<th>Who</th>
<th>Job</th>
</tr>
</thead>
<tbody><tr>
<td><strong>CI</strong> (Stage 1)</td>
<td>Build → scan → push images → update image tags in <code>clearledger-infra</code></td>
</tr>
<tr>
<td><strong>ArgoCD</strong> (Stage 2)</td>
<td>Watch <code>clearledger-infra</code> → apply manifests → cluster runs what Git says</td>
</tr>
</tbody></table>
<pre><code class="language-text">push code → CI updates clearledger-infra → ArgoCD syncs cluster
</code></pre>
<h3 id="heading-pre-sync-checklist-run-before-argocd-app-sync">Pre-sync Checklist: Run Before <code>argocd app sync</code></h3>
<p>ArgoCD applies whatever is in <code>clearledger-infra</code>. Wrong content causes red pods. Re-run the §1.7 checkpoint table to confirm GitHub-side content is still correct, then verify the laptop side:</p>
<pre><code class="language-bash"># Application manifest must point at YOUR infra repo
grep repoURL stages/stage-2-gitops/argocd/clearledger-app.yaml

# Stage 0 workloads still healthy before ArgoCD takes over
kubectl get pods -n clearledger
curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health
</code></pre>
<p>Expected: <code>repoURL</code> contains your GitHub username, all app pods <code>Running</code>, curl <code>200</code>. Only when both pass should you install ArgoCD and sync below.</p>
<pre><code class="language-bash">kubectl create namespace argocd 2&gt;/dev/null || true

kubectl apply -n argocd --server-side --force-conflicts -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=argocd-server -n argocd --timeout=180s
</code></pre>
<p><strong>Why</strong> <code>--server-side --force-conflicts</code><strong>?</strong> Argo CD ships a very large <code>applicationsets.argoproj.io</code> CRD. A normal <code>kubectl apply</code> tries to stash the whole thing in an annotation, hits a 256 KiB limit, and errors with <code>metadata.annotations: Too long</code>. Server-side apply avoids that. It's <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/">how Argo CD expects you to install</a>.</p>
<p>Get the admin password:</p>
<pre><code class="language-bash">kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d &amp;&amp; echo
</code></pre>
<h4 id="heading-configure-argo-cd-for-your-nginx-ingress">Configure Argo CD for your NGINX ingress</h4>
<p>The browser talks HTTPS to ingress and ingress talks plain HTTP to the Argo CD server. Without this, the UI often breaks with <code>503</code> or <code>ERR_TOO_MANY_REDIRECTS</code> on live-update URLs (<code>/api/v1/stream/*</code>).</p>
<pre><code class="language-bash">kubectl apply -f stages/stage-2-gitops/infra/argocd-cmd-params.yaml

kubectl apply -f stages/stage-2-gitops/infra/argocd-ingress.yaml

kubectl rollout restart deployment/argocd-server -n argocd

kubectl rollout status deployment/argocd-server -n argocd --timeout=180s
</code></pre>
<p><strong>Expected in</strong> <code>argocd-cmd-params-cm</code><strong>:</strong> <code>server.insecure: "true"</code>, <code>server.grpc.web: "true"</code>, <code>server.url: https://argocd.local</code>.</p>
<p>Open <code>https://argocd.local</code>. Login: <code>admin</code> and the password from above. Accept the self-signed certificate warning if the browser shows one.</p>
<p><strong>Expected:</strong> The Applications page loads. In the browser console (F12 Console), you shouldn't see <code>401</code> or <code>ERR_HTTP2_PROTOCOL_ERROR</code>. If the UI looks fine in a normal window, you're done: incognito isn't required.</p>
<p><strong>If login fails with</strong> <code>401 Unauthorized</code> (often after a config change or a bad earlier login), try a private/incognito window or clear site data for <code>argocd.local</code>, then log in again. Still stuck? See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md. ArgoCD</a>.</p>
<p>Connect ArgoCD to the infra repo and apply the Application manifest:</p>
<h4 id="heading-1-edit-stagesstage-2-gitopsargocdclearledger-appyaml">1. Edit <code>stages/stage-2-gitops/argocd/clearledger-app.yaml</code></h4>
<p>Set <code>spec.source.repoURL</code> to your infra repo (your GitHub username, not <code>git config user.name</code>).</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/56020fa9-77fa-4d60-bcc5-bbd06b6c809f.png" alt="photo of manifest file pointing to what to change." style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-2-connect-argocd-to-your-infrastructure-repository">2. Connect ArgoCD to your infrastructure repository:</h4>
<p>This gives ArgoCD permission to watch <code>clearledger-infra</code> for new commits. Whenever the deployment manifests change, ArgoCD will update the cluster automatically.</p>
<pre><code class="language-bash"># macOS: brew install argocd
argocd login argocd.local --username admin --password YOUR_PASSWORD --insecure --grpc-web

# Public repo
argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git --grpc-web

# Private repo — PAT from Stage 1 §1.4 (you saved it as GitHub secret INFRA_REPO_TOKEN)
export INFRA_REPO_TOKEN='ghp_...'   # paste here; GitHub only shows it once at creation
argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git \
  --username git --password "$INFRA_REPO_TOKEN" --grpc-web
</code></pre>
<p><strong>Verify that Argo CD can reach the repo</strong> (do this before applying the Application):</p>
<pre><code class="language-bash">argocd repo list --grpc-web
</code></pre>
<p>Look for your <code>clearledger-infra</code> URL with <strong>TYPE</strong> <code>git</code> and connection Successful. If it shows Failed or the repo is missing, Argo CD can't sync. Fix credentials before Stage 4 or any stage that depends on GitOps.</p>
<p>After a VM restore or Argo CD reinstall, you may need to run <code>argocd repo add</code> again (credentials are stored in the cluster, not in Git).</p>
<h4 id="heading-3-apply-and-sync">3. Apply and sync:</h4>
<pre><code class="language-bash">kubectl apply -f stages/stage-2-gitops/argocd/clearledger-app.yaml

argocd app sync clearledger --grpc-web
</code></pre>
<h3 id="heading-how-to-read-the-argo-cd-ui">How to Read the Argo CD UI</h3>
<p>After sync, open the <strong>clearledger</strong> application in the tree view. Three badges at the top tell you almost everything:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/2db73788-ee70-450d-b5ed-00835d50d180.png" alt="screenshot shwoing argocd UI" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>APP HEALTH: Healthy</strong>. Kubernetes thinks the workloads are running. Pods are up (or still starting if it says Progressing).</p>
<p><strong>SYNC STATUS: Synced</strong>: the cluster matches <code>clearledger-infra</code> on GitHub at the commit shown (for example, <code>main (2c88aa1)</code>). Git is the source of truth and Argo CD applied it.</p>
<p><strong>LAST SYNC: Succeeded</strong>: the most recent apply from Git worked. If this failed, click it for the error.</p>
<p>The resource tree below is the same app broken into pieces: namespace, secrets, services, deployments, ingress, and so on Green checkmarks = applied from Git. Click any box (for example, <code>deploy/auth-service</code>) then <strong>Live Manifest</strong> vs <strong>Desired</strong> to see what Argo CD thinks should run.</p>
<p><strong>Quick "is the app actually working?" test</strong> (outside Argo CD):</p>
<pre><code class="language-bash">curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health
</code></pre>
<p><code>200</code> = the app is reachable end-to-end, not only "green in Argo CD."</p>
<p><strong>When something is wrong:</strong> HEALTH goes <strong>Degraded</strong> or <strong>Progressing</strong> for a long time, SYNC goes <strong>OutOfSync</strong>, and a resource in the tree turns <strong>red</strong>. Click that resource and then <strong>Events</strong> or <strong>Logs</strong>. The kubectl checks below double-check the same thing from the terminal.</p>
<p>Confirm ArgoCD is watching all workloads (not only ingress):</p>
<pre><code class="language-bash">argocd app resources clearledger --grpc-web | grep Deployment
</code></pre>
<p><strong>Pass looks like your output:</strong></p>
<pre><code class="language-text">apps    Deployment    clearledger    auth-service            No
apps    Deployment    clearledger    frontend                No
apps    Deployment    clearledger    ledger-service          No
apps    Deployment    clearledger    notification-service  No
apps    Deployment    clearledger    redis                   No
</code></pre>
<p>This command shows the Deployments that ArgoCD is managing for ClearLedger. You should see <code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>frontend</code>, and <code>redis</code>. That means ArgoCD reads the full <code>kustomization.yaml</code> from <code>clearledger-infra/manifests</code>, not just one file.</p>
<p>The last column is <code>ORPHANED</code>. <code>No</code> is good. It means ArgoCD knows this resource belongs to the ClearLedger app. You only need to worry if one of the Deployments is missing, or if ArgoCD shows <code>OutOfSync</code>, <code>Degraded</code>, or red resources in the UI.</p>
<p><strong>✋ Hands-on checkpoint: first sync healthy</strong></p>
<p>Run these four checks. Pass looks like this:</p>
<pre><code class="language-bash">kubectl get pods -n clearledger
# Every app pod 1/1 Running (postgres/redis may show older RESTARTS from VM reboots — OK)

kubectl get application clearledger -n argocd \
  -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'
# sync=Synced health=Healthy

curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health
# 200

kubectl logs -n clearledger deploy/auth-service --tail=5 2&gt;/dev/null | head -3
# Lines like: GET /health HTTP/1.1" 200 OK
# Bad sign: DATABASE_URL is not set
</code></pre>
<p>If all four pass then, Stage 2 first sync is done. Continue to Enable CI, and ArgoCD handoff below, then <code>make check-2</code> and <code>make snapshot STAGE=2</code>.</p>
<h3 id="heading-how-to-enable-the-ci-to-argocd-handoff">How to Enable the CI to ArgoCD Handoff</h3>
<p>In Stage 1, the pipeline updated <code>clearledger-infra</code>, but it didn't update the cluster. That was intentional.</p>
<p>Now ArgoCD is installed, so you can let the pipeline tell ArgoCD to check for changes after each successful run.</p>
<p>In GitHub, open your <code>clearledger</code> repo and go to Settings, Secrets and variables, Actions, Variables, and then New repository variable.</p>
<p>Add:</p>
<table>
<thead>
<tr>
<th><strong>Name</strong></th>
<th><strong>Value</strong></th>
</tr>
</thead>
<tbody><tr>
<td><code>ENABLE_ARGOCD_SYNC</code></td>
<td><code>true</code></td>
</tr>
</tbody></table>
<p>From now on, a green pipeline does two things:</p>
<ol>
<li>Updates <code>clearledger-infra</code> with the new image tag</li>
<li>Asks ArgoCD to sync the cluster</li>
</ol>
<p>If the pipeline can't trigger ArgoCD immediately, that's usually okay. ArgoCD checks <code>clearledger-infra</code> on its own every few minutes, so it should still pick up the new Git change.</p>
<p>Leave <code>ENABLE_DAST</code> unset for now. You enable that in Stage 3 after the app is stable at <code>clearledger.local</code>.</p>
<h3 id="heading-if-the-argocd-ui-shows-red-pods-or-progressing-read-this-before-the-screenshot">If the Argocd UI Shows Red Pods or "Progressing" (Read This Before the Screenshot)</h3>
<p>This is a common first-sync surprise, not a broken install.</p>
<h4 id="heading-why-it-happens-in-stage-2">Why it happens in Stage 2</h4>
<p>ArgoCD syncs whatever is in <code>clearledger-infra</code>. Deployments must use <code>secretKeyRef</code> (Stages 2–4), not Vault injection. If your infra repo has Vault annotations from an older lab copy, auth/ledger crash with <code>DATABASE_URL is not set</code> until Stage 5.</p>
<p>Network policies belong to Stage 6. In the main <code>clearledger</code> repo, they live in <code>infra/deferred-by-stage/stage-6-runtime-security/netpol/</code>, not in <code>infra/manifests/</code>. Don't copy them into <code>clearledger-infra</code> during Stage 2.</p>
<p>If <code>manifests/netpol/</code> is still in your <code>clearledger-infra</code> repo on GitHub (from an older copy of the lab), ArgoCD will keep applying it. Those policies use <strong>default-deny</strong> and break DNS for new pods, so you see red <strong>0/1</strong> pods and <strong>Progressing</strong> health.</p>
<h4 id="heading-fix-for-stage-2">Fix for Stage 2</h4>
<p>Do <strong>both</strong> steps. Deleting only in the cluster is not enough: ArgoCD recreates policies from Git on the next sync.</p>
<p><strong>Step 1: remove from</strong> <code>clearledger-infra</code> <strong>on GitHub</strong></p>
<p>Delete the folder <code>manifests/netpol/</code> and commit: <code>chore: defer network policies to Stage 6</code>.</p>
<p><strong>Step 2: sync and restart</strong></p>
<pre><code class="language-bash">argocd app sync clearledger --grpc-web
kubectl delete networkpolicy -n clearledger --all   # safe once Git no longer has netpol
kubectl rollout restart deployment/auth-service deployment/ledger-service -n clearledger
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
</code></pre>
<p>Network policies stay in <code>clearledger</code> under <code>infra/deferred-by-stage/</code> until you apply them in Stage 6.</p>
<p>When that looks good, continue below.</p>
<p>When ArgoCD finishes syncing, open the <code>clearledger</code> app in the ArgoCD UI. You should see green <code>Healthy</code> and <code>Synced</code> badges. The app should point to your <code>clearledger-infra</code> repo, use the <code>manifests</code> path, and deploy into the <code>clearledger</code> namespace.</p>
<p>Then open the app tile. The resource tree should show your deployments, services, and ingress with no red resources.</p>
<p>You can confirm the same thing from the terminal:</p>
<p><code>argocd app get clearledger --grpc-web</code></p>
<p>Look for <code>Sync Status: Synced</code> and <code>Health Status: Healthy</code>.</p>
<h3 id="heading-argocd-stuck-outofsync">ArgoCD stuck OutOfSync</h3>
<p><strong>Normal path:</strong> CI copies full manifests + updates Kustomize tags, ArgoCD auto-syncs within ~3 minutes.</p>
<p><strong>If still OutOfSync after 10+ minutes:</strong></p>
<pre><code class="language-bash">make fix-argocd
</code></pre>
<p>This re-syncs canonical manifests to <code>clearledger-infra</code> (Kustomize SHAs preserved), re-applies the Application, and triggers a hard refresh. <strong>Don't</strong> <code>kubectl apply</code> deployments: fix Git, let ArgoCD sync.</p>
<pre><code class="language-bash">kubectl annotate application clearledger -n argocd 

argocd.argoproj.io/refresh=hard --overwrite

argocd app sync clearledger --grpc-web --prune

kubectl get application clearledger -n argocd -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'
</code></pre>
<p><strong>Take a screenshot of that view</strong>: the app tile or the resource tree is fine. That’s your portfolio proof that GitOps is actually running.</p>
<h3 id="heading-prove-argocd-self-healing">Prove ArgoCD Self-Healing</h3>
<p>Now prove that Git is the source of truth.</p>
<p>In this demo, you'll change the running cluster by hand. You will <strong>not</strong> change Git. ArgoCD should notice that the cluster no longer matches <code>clearledger-infra</code>, then change it back.</p>
<p>Before you start, make sure the app is healthy and ArgoCD is managing the deployments:</p>
<pre><code class="language-bash">argocd app resources clearledger --grpc-web | grep Deployment
</code></pre>
<p>Manually change the auth-service image in the cluster:</p>
<pre><code class="language-bash"># Manually change the image in the cluster only (Git stays the same)
kubectl set image deployment/auth-service \
  auth-service=$DOCKER_USERNAME/clearledger-auth-service:fake-tag \
  -n clearledger
</code></pre>
<p>Check ArgoCD:</p>
<pre><code class="language-bash"># ArgoCD should flip to OutOfSync within a minute or two
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
</code></pre>
<p>Wait for ArgoCD to fix the cluster. The fake image tag may briefly cause an image pull error. That's expected in this demo.</p>
<pre><code class="language-bash"># Wait for selfHeal (default sync interval is ~3 minutes)
sleep 180
</code></pre>
<p>Confirm the image was changed back to the Git version:</p>
<pre><code class="language-bash"># Cluster image should match clearledger-infra again — Git was never edited
kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
</code></pre>
<p>If the image changed back, ArgoCD self-healing worked. You changed the cluster by hand, but ArgoCD restored it to match <code>clearledger-infra</code>.</p>
<p>That's GitOps: Git says what should run, and ArgoCD keeps the cluster matching Git.</p>
<pre><code class="language-bash">make check-2
</code></pre>
<h3 id="heading-how-to-roll-back-a-bad-deploy">How to Roll Back a Bad Deploy</h3>
<p>You just proved that ArgoCD reverts unauthorized cluster changes. Now flip it: <strong>what if you pushed a bad commit yourself?</strong> GitOps rollback isn't a button. It's a Git operation. This section explains why, shows you both methods, and has you practice each one before you need them under pressure.</p>
<h4 id="heading-how-you-know-you-need-to-roll-back">How you know you need to roll back</h4>
<p>These symptoms appearing within minutes of a push to <code>clearledger-infra</code> point at a bad commit:</p>
<ul>
<li>Pods stuck in <code>CrashLoopBackOff</code> or <code>Error</code>. Check with <code>kubectl get pods -n clearledger</code>.</li>
<li><code>kubectl logs &lt;pod&gt; -n clearledger --previous</code> shows startup errors that weren't there before.</li>
<li>ArgoCD health flips from <code>Healthy</code> to <code>Degraded</code> or stays on <code>Progressing</code>. Check with <code>argocd app get clearledger --grpc-web</code>.</li>
<li>The app returns 5xx errors or login stops working. Check with <code>curl -I http://clearledger.local/health</code>.</li>
</ul>
<p>If this happens right after a push, roll back first. Once the app is stable again, investigate the bad commit.</p>
<h4 id="heading-why-argocd-rollback-isnt-just-a-button">Why ArgoCD rollback isn't just a button</h4>
<p>ArgoCD has a rollback button in the UI and an <code>argocd app rollback</code> command. Both work. But only if you understand the interaction with <code>selfHeal</code>.</p>
<p>Your Application (<code>stages/stage-2-gitops/argocd/clearledger-app.yaml</code>) is configured with:</p>
<pre><code class="language-yaml">syncPolicy:
  automated:
    selfHeal: true
</code></pre>
<p>ArgoCD keeps the cluster matched to Git. In this lab, Git means <code>clearledger-infra</code>.</p>
<p>If someone changes the cluster by hand, ArgoCD treats that as drift and changes it back to match Git.</p>
<p>This also affects rollback. The ArgoCD UI rollback changes the cluster, but it doesn't change Git. If <code>clearledger-infra</code> still points to the bad version, and self-heal can bring the bad version back.</p>
<p>The safer GitOps rollback is to change Git with <code>git revert</code> in <code>clearledger-infra</code>. Then ArgoCD syncs the cluster to the reverted, good version.</p>
<p>If you need an emergency UI rollback, turn off auto-sync first, roll back in ArgoCD, then fix Git afterward.</p>
<h4 id="heading-method-1-git-revert-preferred-always-try-this-first">Method 1: Git revert (preferred, always try this first)</h4>
<p>This is the GitOps way. You don't touch the cluster. You change Git, and ArgoCD syncs the fix.</p>
<p><strong>When to use:</strong> You have a few minutes and can identify the bad commit in <code>clearledger-infra</code>.</p>
<p><strong>How it works:</strong></p>
<pre><code class="language-plaintext">Bad commit pushed to clearledger-infra
        ↓
ArgoCD auto-synced it (cluster is now broken)
        ↓
You run: git revert &lt;bad-commit&gt; &amp;&amp; git push
        ↓
ArgoCD auto-syncs the revert (cluster is fixed, selfHeal works with you)
        ↓
Git history shows the bad deploy AND the revert, full audit trail
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/2371fab2-b05d-4453-ac28-80465a95a88f.png" alt="demo showing how argocd works and the flow" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>Step-by-step:</strong></p>
<pre><code class="language-bash"># 1. Go to your clearledger-infra repo (wherever you cloned it)
cd ~/clearledger-infra  # adjust path if you cloned elsewhere
git pull                # make sure you are up to date

# 2. Find the bad commit
git log --oneline -10

# Output looks like:
# abc1234 update ledger-service image to v1.4.0   ← this broke prod
# def5678 update auth-service image to v1.3.1     ← was fine
# 9a1b2c3 add vault rotation cronjob

# 3. Revert it — this creates a NEW commit, it does not delete history
git revert abc1234 --no-edit

# 4. Push — ArgoCD picks it up automatically within ~3 minutes
git push

# 5. Confirm the cluster recovered
kubectl get pods -n clearledger
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy
</code></pre>
<p>This method is preferred because it fixes the source of truth: <code>clearledger-infra</code>.</p>
<p>After you push the revert, ArgoCD sees the new Git state and syncs the cluster to it. Nothing fights you because Git and the cluster are supposed to match.</p>
<p>It also leaves a clear history. Git shows the bad deploy, the revert, who made both changes, and when they happened. That's easier to debug, easier to review, and better for compliance.</p>
<h4 id="heading-method-2-emergency-argocd-rollback-when-the-cluster-is-on-fire">Method 2: Emergency ArgoCD rollback (when the cluster is on fire)</h4>
<p>Use this if the cluster is broken right now and you don't have time to push a Git fix. It pins the cluster to a previous known good deployment immediately. You'll still fix Git afterward. This isn't a permanent fix.</p>
<p><strong>When to use:</strong> Incident in progress. Pods are crashing, users are affected, and you need the cluster back to a known good state in under 30 seconds.</p>
<p><strong>Before you start:</strong> confirm your ArgoCD CLI session is still valid. If it expired, re-login first: an expired session will silently fail every command below.</p>
<blockquote>
<pre><code class="language-bash">argocd account get-user-info --grpc-web
# If you see "Unauthenticated", re-login:
ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

argocd login argocd.local --username admin --password "$ARGOCD_PASSWORD" \
  --insecure --grpc-web
</code></pre>
</blockquote>
<p><strong>Step 1: Disable auto-sync</strong> (critical). Skip this and selfHeal will undo your rollback within 3 minutes.</p>
<pre><code class="language-bash">argocd app set clearledger --sync-policy none --grpc-web
# Confirm: automated sync is now off
argocd app get clearledger --grpc-web | grep "Sync Policy"
# Expected: Sync Policy: &lt;none&gt;
</code></pre>
<p><strong>Step 2: Find the last known-good deployment ID</strong></p>
<pre><code class="language-bash">argocd app history clearledger --grpc-web

# Output looks like:
# ID   DATE                           REVISION
# 9    2026-06-05 10:12:00 +0000 UTC  abc1234  ← bad deploy (current)
# 8    2026-06-04 14:46:06 +0000 UTC  def5678  ← known good
# 7    2026-06-01 20:53:19 +0000 UTC  9a1b2c3

# Or check via kubectl (no argocd CLI needed):
kubectl get application clearledger -n argocd \
  -o jsonpath='{range .status.history[*]}{.id}{"\t"}{.deployedAt}{"\t"}{.revision}{"\n"}{end}'
</code></pre>
<p>Use the ID (the number on the left), not the SHA.</p>
<p><strong>Step 3: Roll back to the good ID</strong></p>
<pre><code class="language-bash">argocd app rollback clearledger 8 --grpc-web
</code></pre>
<p><strong>Step 4: Confirm the cluster is stable</strong></p>
<pre><code class="language-bash">kubectl get pods -n clearledger
# All pods should be Running

argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Sync Status:   OutOfSync  ← expected — cluster is at rev 8, Git is still at the bad HEAD
# Health Status: Healthy    ← this is what matters right now
</code></pre>
<p><code>OutOfSync</code> is correct and expected at this point. The cluster is running the old good revision. Git still has the bad commit. You'll fix that next.</p>
<p><strong>Step 5: Fix Git (don't leave it broken)</strong></p>
<pre><code class="language-bash">cd ~/clearledger-infra
git pull
git revert &lt;bad-commit-sha&gt; --no-edit
git push
</code></pre>
<p><strong>Step 6: Re-enable auto-sync</strong></p>
<pre><code class="language-bash">argocd app set clearledger \
  --sync-policy automated \
  --self-heal \
  --auto-prune \
  --grpc-web

# Trigger an immediate sync so you do not wait for the next auto-check
argocd app sync clearledger --grpc-web

# Confirm everything is clean
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy
</code></pre>
<p><strong>Never leave auto-sync disabled longer than the incident.</strong> It's your drift-detection and tamper-evidence mechanism: without it, unauthorized <code>kubectl</code> changes go undetected. Re-enable it the moment you push the Git fix.</p>
<h4 id="heading-practise-the-rollback-now-before-you-need-it-under-pressure">Practise the rollback now (before you need it under pressure)</h4>
<p>Don't wait for a real incident to run this for the first time. The steps below simulate a bad image tag deploy and walk you through Method 1 (the preferred path).</p>
<p><strong>Step 1: Push a bad image tag to</strong> <code>clearledger-infra</code></p>
<pre><code class="language-bash">cd ~/clearledger-infra
git pull

# Edit manifests/notification-service/deployment.yaml
# Change the image tag to a tag that does not exist, e.g.:
#   image: docker.io/$DOCKER_USERNAME/clearledger-notification-service:broken-tag

# Commit and push it
git add manifests/notification-service/deployment.yaml
git commit -m "test: simulate bad deploy with nonexistent image tag"
git push
</code></pre>
<p><strong>Step 2: Watch ArgoCD sync the bad state</strong></p>
<pre><code class="language-bash"># Give ArgoCD ~3 minutes to pick it up, or trigger immediately:
argocd app sync clearledger --grpc-web

# Watch the notification-service pod fail
kubectl get pods -n clearledger -w
# You will see: notification-service pod stuck in ImagePullBackOff or ErrImagePull
</code></pre>
<p><strong>Step 3: Roll back using Method 1</strong></p>
<pre><code class="language-bash">cd ~/clearledger-infra

# Revert the bad commit
git revert HEAD --no-edit
git push

# ArgoCD will auto-sync — or trigger it:
argocd app sync clearledger --grpc-web

# Watch pods recover
kubectl get pods -n clearledger -w
# notification-service should return to Running
</code></pre>
<p><strong>Step 4: Verify</strong></p>
<pre><code class="language-bash">argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy

kubectl get pods -n clearledger
# All pods Running, no ImagePullBackOff
</code></pre>
<p>You have now practised a rollback end-to-end. The <code>git revert</code> commit is permanently in the infra repo's history: a real audit record of a simulated recovery.</p>
<h4 id="heading-quick-reference">Quick reference</h4>
<p><strong>Use Method 1 (git revert) when:</strong></p>
<ul>
<li>A bad image tag or manifest was pushed to <code>clearledger-infra</code> and you have a few minutes</li>
<li>Any config change in the infra repo caused pods to break</li>
<li>This is almost always the right answer. It's fast, safe, and leaves a clean audit trail.</li>
</ul>
<p><strong>Use Method 2 (emergency ArgoCD rollback) when:</strong></p>
<ul>
<li>The cluster is broken right now, users are affected, and you need it stable in under 30 seconds</li>
<li>You're not yet sure which commit caused the problem and need time to investigate: roll back to stabilise, then use <code>git log</code> to find the culprit, then fix forward with Method 1</li>
</ul>
<p><strong>Neither method applies</strong> when a pod is crashing but nothing was pushed to the infra repo recently. This isn't a rollback problem. Check <code>kubectl logs</code>, Vault connectivity, and network policies instead.</p>
<p><code>revisionHistoryLimit: 10</code> in <code>stages/stage-2-gitops/argocd/clearledger-app.yaml</code> means ArgoCD always has 10 previous deployments available for emergency rollback. Increase it if your release cadence is high.</p>
<h3 id="heading-what-you-learned-in-stage-2">What You Learned in Stage 2</h3>
<ul>
<li>What GitOps means: Git is the single source of truth, and a tool enforces it</li>
<li>What ArgoCD does: watches Git, compares it to the cluster, corrects drift automatically</li>
<li>How the full flow works now: push code, CI builds image, CI updates infra repo, and ArgoCD syncs cluster.</li>
<li>No one runs <code>kubectl</code> to deploy anymore. The pipeline updates Git, ArgoCD does the rest.</li>
<li><strong>How to roll back safely:</strong> <code>git revert</code> in the infra repo is the correct answer, while ArgoCD emergency rollback is the break-glass option. You must disable auto-sync first or selfHeal will silently undo it.</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Implemented GitOps with ArgoCD so cluster state is driven from Git, with drift detection, auto-sync, and a Git-based rollback of a bad deploy.</p>
</blockquote>
<p><code>make snapshot STAGE=2 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage2</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<h2 id="heading-stage-3-security-gates">Stage 3 — Security Gates</h2>
<p>Every push runs security checks. Some failures stop the pipeline right away. Others you learn from now and enforce in the cluster later (Stage 4).</p>
<p><strong>Goal:</strong> understand six scanners: what each one looks at, what it catches, and how to read a failure. You'll break each gate on purpose (§3.4) so a failed CI job isn't a surprise.</p>
<p><strong>Ready for Stage 3?</strong></p>
<ul>
<li><code>make check-2</code> passes</li>
<li><code>ENABLE_ARGOCD_SYNC=true</code> on GitHub (you set this in Stage 2)</li>
<li><code>ENABLE_DAST</code> still <strong>unset</strong> (turn on later in this stage if you want)</li>
<li>Argo CD at <code>http://argocd.local</code> shows <strong>Synced</strong></li>
<li>Optional: skim <a href="#heading-stage-1-security-posture-what-blocks-vs-what-waits">Stage 1 security posture</a>. Stage 1 already ran many of these tools</li>
</ul>
<p><strong>Done when:</strong> <code>make check-3</code> passes and you triggered each gate once (§3.4). Then <code>make snapshot STAGE=3</code> and <code>make snapshots</code>.</p>
<h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3>
<p>One tool isn't enough. Each scanner guards a different layer:</p>
<ul>
<li><strong>Gitleaks</strong>: secrets in code or Git history (API keys, tokens)</li>
<li><strong>Semgrep (SAST)</strong>: bugs in your Python/JS source (injection, unsafe patterns)</li>
<li><strong>Trivy (SCA + images)</strong>: finds known security vulnerabilities (called CVEs) in your Python/Node.js packages and Docker images. A CVE (Common Vulnerabilities and Exposures) is a publicly tracked software security flaw with a unique identifier.</li>
<li><strong>Checkov (IaC)</strong>: misconfigurations in Dockerfiles, Kubernetes manifests, and Stage 8 Terraform.</li>
<li><strong>Cosign</strong>: proves images were built and signed by your pipeline</li>
</ul>
<p>What blocks CI today: secrets, bad code (SAST), vulnerable images, Dockerfile issues on production images.</p>
<p>What waits for later: Some Kubernetes issues are only reported in Stage 1. They show you what still needs hardening. In Stage 4, Kyverno turns the important rules into real cluster enforcement, so unsafe workloads are blocked before they run.</p>
<p>When you <code>git commit</code>, hooks on your laptop can scan first (pre-commit). When you <code>git push</code>, GitHub Actions scans again on the runner. Same idea twice: catch mistakes before they waste a 10-minute pipeline. Pre-commit is optional to install, as CI always runs on push either way.</p>
<h3 id="heading-enable-dast-optional-after-stage-2">Enable DAST (Optional After Stage 2)</h3>
<p>DAST (Dynamic Application Security Testing) scans the <strong>running</strong> app at <code>http://clearledger.local</code>. It was off in Stages 1–2 on purpose: Stage 1 never deployed to the cluster, and Stage 2 was about getting GitOps healthy first.</p>
<p>If <code>make check-2</code> passes and <code>curl http://clearledger.local/auth/health</code> returns <code>200</code>, you can turn DAST on:</p>
<p>Go to GitHub and into your <code>clearledger</code> repo. Then go to <strong>Settings, Secrets and variables, Actions, Variables</strong>, and <strong>New repository variable</strong>:</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td><code>ENABLE_DAST</code></td>
<td><code>true</code></td>
</tr>
</tbody></table>
<p>Push a small commit (or re-run the last workflow on <code>main</code>). The <strong>DAST (OWASP ZAP + fintech API tests)</strong> job should run instead of <strong>skipped</strong>. A failed ZAP scan is a real finding to investigate. Skipped before this step only means the toggle was off.</p>
<h3 id="heading-31-install-pre-commit-hooks">3.1: Install Pre-commit Hooks</h3>
<pre><code class="language-bash"># macOS (Homebrew — avoids PEP 668 "externally-managed-environment" from pip3):
brew install pre-commit

# Linux/WSL2:
# sudo apt install -y pre-commit
# or: python3 -m pip install --user pre-commit

pre-commit install
pre-commit run --all-files
</code></pre>
<p>If a hook fails, read the error first. Gitleaks and Ruff should pass before you commit. Some YAML or Terraform hook issues may come from later-stage files. If that happens, continue with the stage instructions and use <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md"><code>troubleshooting.md</code></a> for Gitleaks or CI scanner failures.</p>
<p>Test it catches secrets locally before CI does:</p>
<pre><code class="language-bash">echo 'AWS_SECRET = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' &gt;&gt; app/auth-service/main.py
git add app/auth-service/main.py &amp;&amp; git commit -m "test"

# Gitleaks fires and blocks the commit — see "What you should see" below

git restore --staged app/auth-service/main.py

git checkout app/auth-service/main.py
</code></pre>
<p>The commit was blocked before it even reached Git. If the pre-commit hook wasn't installed, that fake AWS key would be in your Git history permanently (even if you delete the line later, Git remembers).</p>
<p><strong>If you already did Cosign in Stage 1,</strong> that counts. Stage 3 doesn't require regenerating keys. Confirm that <code>infra/cosign.pub</code> exists and GitHub has <code>COSIGN_PRIVATE_KEY</code> + <code>COSIGN_PASSWORD</code>. Stage 4 turns signing into <strong>enforcement</strong> at the cluster gate.</p>
<p><strong>✋ Hands-on checkpoint: pre-commit actually blocks a secret</strong></p>
<p>Installed-but-not-wired is the classic silent failure. Prove the hooks fire:</p>
<pre><code class="language-bash">echo 'AWS_SECRET='"$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')" &gt; leak-test.env

git add leak-test.env

pre-commit run --all-files; echo "exit=$?"

git reset leak-test.env &gt;/dev/null; rm -f leak-test.env
</code></pre>
<p><strong>Expected:</strong> the secret-scanning hook <strong>fails</strong> the run (<code>exit=1</code>) and flags <code>leak-test.env</code>. If <code>exit=0</code>, your hooks are installed but not catching anything: re-run <code>pre-commit install</code> and confirm <code>.git/hooks/pre-commit</code> exists.</p>
<p>If you skip this, commits sail through unscanned and you'll believe Stage 3 is protecting you when it's not.</p>
<h3 id="heading-32-generate-cosign-keys">3.2: Generate Cosign Keys</h3>
<p>If you created Cosign keys in Stage 1 (§1.4), skip generation: go straight to inserting <code>cosign.pub</code> into the Kyverno policy and adding the GitHub secrets below.</p>
<p><strong>Cosign</strong> signs your Docker images with a cryptographic key. When you deploy to the cluster, Kyverno (Stage 4) can verify the signature and reject any image that wasn't signed by your pipeline. This prevents someone from pushing a malicious image to your Docker Hub and having the cluster run it.</p>
<pre><code class="language-bash"># macOS: brew install cosign

# Linux/WSL2: curl -O -L https://github.com/sigstore/cosign/releases/download/v2.2.4/cosign-linux-amd64 &amp;&amp; chmod +x cosign-linux-amd64 &amp;&amp; sudo mv cosign-linux-amd64 /usr/local/bin/cosign

cosign generate-key-pair   # enter a password when prompted
</code></pre>
<p>This creates two files: <code>cosign.key</code> (private, used by the pipeline to sign) and <code>cosign.pub</code> (public, used by Kyverno to verify).</p>
<p>Insert your public key into the Kyverno policy (replace the placeholder block in <code>infra/policies/require-signed-images.yaml</code> with the contents of <code>cosign.pub</code>).</p>
<p>Add secrets to GitHub (<a href="http://github.com/YOUR">github.com/YOUR</a>_USERNAME/clearledger → Settings → Secrets and variables → Actions):</p>
<table>
<thead>
<tr>
<th>Secret</th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td><code>COSIGN_PRIVATE_KEY</code></td>
<td>Contents of <code>cosign.key</code></td>
</tr>
<tr>
<td><code>COSIGN_PASSWORD</code></td>
<td>The password you entered when generating keys</td>
</tr>
</tbody></table>
<p><strong>✋ Hands-on checkpoint: Cosign keys are ready</strong></p>
<p>Stage 4 uses <a href="http://cosign.pub"><code>cosign.pub</code></a> to verify signed images. Before you continue, confirm the key files exist and the private key isn't tracked by Git:</p>
<pre><code class="language-bash">test -f cosign.key &amp;&amp; echo "private key present"
test -f cosign.pub &amp;&amp; echo "public key present"
grep -q "BEGIN PUBLIC KEY" cosign.pub &amp;&amp; echo "public key valid"
git check-ignore cosign.key &amp;&amp; echo "private key correctly ignored"
</code></pre>
<p><strong>Expected:</strong> all four lines should print.</p>
<p>If <code>git check-ignore cosign.key</code> prints nothing, add <code>cosign.key</code> to <code>.gitignore</code> before committing anything. The private key must stay out of Git.</p>
<p>Don't skip this check. Stage 4 needs the public key for the Kyverno image-signing policy, and the private key must remain local.</p>
<h3 id="heading-33-activate-the-full-security-pipeline">3.3: Activate the Full Security Pipeline</h3>
<p>The security gates are already in <code>.github/workflows/ci.yaml</code>. Push any change to trigger the full pipeline:</p>
<pre><code class="language-bash">git add . &amp;&amp; git commit -m "ci: full DevSecOps pipeline" &amp;&amp; git push origin main
</code></pre>
<h3 id="heading-34-break-each-gate-on-purpose">3.4: Break Each Gate on Purpose</h3>
<p>For each gate, you'll want to break something on purpose, read how the tool reports it, revert, and confirm green again. Try the local command first, then push once if you want a screenshot on GitHub Actions.</p>
<pre><code class="language-bash"># 1. Break it   2. Run locally or push   3. Read the failure
# 4. git checkout -- path/to/file   5. pre-commit run --all-files (optional)   6. git push
</code></pre>
<p>Start with <strong>Gate 1</strong> end-to-end before the others.</p>
<h4 id="heading-gate-1-gitleaks-secrets">Gate 1: Gitleaks (secrets)</h4>
<p><strong>Inject:</strong> hardcoded AWS key in any Python file.</p>
<p>The goal is to prove the secret scanner works.</p>
<p>This command adds a fake AWS-looking key to <code>app/auth-service/main.py</code>:</p>
<pre><code class="language-bash">echo 'AWS_KEY = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' &gt;&gt; app/auth-service/main.py

git add app/auth-service/main.py &amp;&amp; git commit -m "test: trigger gitleaks"
# pre-commit blocks this commit locally — that is the test.
# For a CI screenshot only: git commit --no-verify -m "test: trigger gitleaks" &amp;&amp; git push
</code></pre>
<p><strong>Done looks like this (terminal: pre-commit):</strong></p>
<pre><code class="language-text">🔑 Secrets scan (Gitleaks)...............................................Failed
- hook id: gitleaks
- exit code: 1

Finding:     AWS_KEY = "REDACTED"
RuleID:      aws-access-token
File:        app/auth-service/main.py
Line:        316
</code></pre>
<p><strong>Expected:</strong> the commit should fail. Gitleaks should report one secret finding in <code>app/auth-service/</code><a href="http://main.py"><code>main.py</code></a>.</p>
<p>That failure is good. It means the local pre-commit hook caught the secret before it reached Git.</p>
<p><strong>Revert:</strong></p>
<pre><code class="language-bash">git restore --staged app/auth-service/main.py 2&gt;/dev/null
git checkout app/auth-service/main.py
pre-commit run gitleaks --all-files   # → Passed
</code></pre>
<h4 id="heading-gate-2-semgrep-sast">Gate 2: Semgrep (SAST)</h4>
<p><strong>Local dry-run</strong> (no repo change):</p>
<pre><code class="language-bash">python3 -m venv /tmp/sec-gates-venv &amp;&amp; /tmp/sec-gates-venv/bin/pip install semgrep
cat &gt; /tmp/semgrep-bad.py &lt;&lt; 'EOF'
import subprocess
from fastapi import Request
def bad(request: Request):
    subprocess.run(request.query_params.get("cmd"), shell=True)
EOF
/tmp/sec-gates-venv/bin/semgrep \
  --config=p/python --config=p/security-audit --config=p/owasp-top-ten --error \
  /tmp/semgrep-bad.py
</code></pre>
<p><strong>Break CI</strong>: add a temporary file Semgrep will scan, commit, and push:</p>
<pre><code class="language-bash">cat &gt; app/auth-service/gate_test_semgrep.py &lt;&lt; 'EOF'
import subprocess
from fastapi import Request
def bad(request: Request):
    subprocess.run(request.query_params.get("cmd"), shell=True)
EOF

git add app/auth-service/gate_test_semgrep.py &amp;&amp; git commit -m "test: trigger semgrep" &amp;&amp; git push
</code></pre>
<p><strong>Expected result:</strong> Semgrep reports <code>subprocess-shell-true</code> as <code>Blocking</code>. The <code>SAST (Semgrep)</code> job turns red, and the image build jobs don't run.</p>
<p><strong>Revert:</strong></p>
<pre><code class="language-bash">rm -f app/auth-service/gate_test_semgrep.py
git add -A &amp;&amp; git commit -m "revert: semgrep gate test" &amp;&amp; git push
</code></pre>
<h4 id="heading-gate-3-checkov-iac-dockerfile">Gate 3: Checkov (IaC / Dockerfile)</h4>
<p>Checkov scans Dockerfiles and Kubernetes manifests for unsafe configuration.</p>
<p>First, run a local demo. This removes the <code>HEALTHCHECK</code> from a copied Dockerfile and shows how Checkov reports it:</p>
<pre><code class="language-bash">python3 -m venv /tmp/sec-gates-venv &amp;&amp; /tmp/sec-gates-venv/bin/pip install checkov
sed '/^HEALTHCHECK/,+1d' app/auth-service/Dockerfile &gt; /tmp/Dockerfile-nohc
mkdir -p /tmp/checkov-demo/app/auth-service
cp /tmp/Dockerfile-nohc /tmp/checkov-demo/app/auth-service/Dockerfile
/tmp/sec-gates-venv/bin/checkov --directory /tmp/checkov-demo --framework dockerfile
</code></pre>
<p>Now trigger a Checkov finding in CI by exposing SSH port <code>22</code> in the auth-service Dockerfile:</p>
<pre><code class="language-bash">echo 'EXPOSE 22' &gt;&gt; app/auth-service/Dockerfile
git add app/auth-service/Dockerfile &amp;&amp; git commit -m "test: trigger checkov" &amp;&amp; git push
</code></pre>
<p><strong>Expected result:</strong> the Checkov log or artifact should show <code>CKV_DOCKER_1</code>, which means an SSH port was exposed.</p>
<p>The <code>IaC Scan (Checkov)</code> job may or may not turn red, depending on the severity Checkov assigns. That's okay for this exercise. The goal is to find and understand the Checkov result.</p>
<p>If you need a screenshot of a failed GitHub Actions job, use Gate 1, Gate 2, or Gate 4. Those are designed to turn the workflow red. Checkov is mainly for reading the finding, so it may stay green.</p>
<p><strong>Revert:</strong></p>
<pre><code class="language-bash">git checkout app/auth-service/Dockerfile
git commit -am "revert: checkov gate test" &amp;&amp; git push
</code></pre>
<h4 id="heading-gate-4-trivy-image-cves">Gate 4: Trivy (image CVEs)</h4>
<p><strong>Local dry-run</strong>: scan an old base image (no build):</p>
<pre><code class="language-bash">trivy image --exit-code 1 --severity CRITICAL,HIGH --ignore-unfixed python:3.8-slim
</code></pre>
<p><strong>Break CI</strong>: pin an old base in the Dockerfile, push, wait for <code>Scan images</code>:</p>
<pre><code class="language-bash">sed -i.bak 's/FROM python:3.13-slim/FROM python:3.8-slim/' app/auth-service/Dockerfile
git add app/auth-service/Dockerfile &amp;&amp; git commit -m "test: trigger trivy" &amp;&amp; git push
</code></pre>
<p><strong>Pass:</strong> <code>Scan images</code> → <strong>Trivy scan all images</strong> exits 1 with a CVE table (<code>HIGH</code> / <code>CRITICAL</code>). <code>Publish images</code> and <code>Update Manifests</code> are skipped.</p>
<p><strong>Revert:</strong></p>
<pre><code class="language-bash">git checkout app/auth-service/Dockerfile
git commit -am "revert: trivy gate test" &amp;&amp; git push
</code></pre>
<h3 id="heading-35-when-a-scan-fails-on-a-cve-you-didnt-inject">3.5: When a Scan Fails on a CVE You Didn't Inject</h3>
<p>§3.4 is deliberate. This section is for the other case where you push normal code, but the image scan fails because a new vulnerability was found.<br>That's normal. CVE databases update all the time. Don't weaken the scan. Fix the vulnerable package or image.</p>
<p>First, find the real CVE. In GitHub Actions, open <strong>Scan images</strong> then go to <strong>Trivy scan all images</strong> and look for the table with:</p>
<ul>
<li>Package</li>
<li>CVE</li>
<li>Installed version</li>
<li>Fixed version You can also download the artifact:</li>
</ul>
<p><strong>Ignore this red herring</strong> at the bottom of the log:</p>
<pre><code class="language-text">Version 0.71.2 of Trivy is now available
Error: Process completed with exit code 1.
</code></pre>
<p>The version notice doesn't fail the job. A fixable HIGH/CRITICAL CVE does. Don't add <code>--skip-version-check</code> to “fix” it.</p>
<p><strong>Instead, fix it with:</strong></p>
<ul>
<li><strong>pip package</strong>: bump to the Fixed Version in <code>requirements.txt</code> (example: <code>python-multipart==0.0.30</code> for CVE-2026-53539). Apply the same bump to sibling services if they share that pin.</li>
<li><strong>OS package</strong>: newer base image or a targeted <code>apt</code>/<code>apk</code> upgrade in the Dockerfile.</li>
<li><strong>No stable fix yet</strong> documented exception only: add the CVE to <code>.trivyignore</code> and <code>.grype.yaml</code> with a comment (see <code>CVE-2026-7210</code>).</li>
</ul>
<p>Don't remove <code>--exit-code 1</code>, lower the severity rule, or disable scanning. For help, see <a href="troubleshooting.md#trivy-version-x-is-now-available-notice-not-a-scan-failure">Trivy version notice</a> and <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">Trivy blocks Python service images</a>.</p>
<h3 id="heading-finish-stage-3">Finish Stage 3</h3>
<p>For screenshots, use one clear failed gate:</p>
<ul>
<li>Gitleaks: <code>Secrets Scan</code></li>
<li>Semgrep: <code>SAST</code></li>
<li>Trivy: <code>Scan images</code></li>
<li>Checkov: look for <code>CKV_*</code> in the log or artifact. The job may stay green</li>
</ul>
<p>After each test in §3.4, undo the test change, push the revert, and confirm the workflow is green again. One red GitHub Actions screenshot is enough for your portfolio.</p>
<p><strong>Run the stage check:</strong></p>
<pre><code class="language-bash">make check-3   # must end: All checks passed. Ready for the next stage.
</code></pre>
<p><strong>Expected:</strong> <code>All checks passed. Ready for the next stage.</code></p>
<p>You should also have triggered at least one gate in §3.4. A local Gitleaks failure counts.</p>
<p><code>ENABLE_DAST=true</code> is optional. You only need it if you want to run ZAP later.</p>
<p><strong>Not required yet:</strong> Checkov blocking Kubernetes manifests or Cosign blocking deployments. Stage 4 turns those into cluster enforcement with Kyverno.</p>
<p>Next, save your progress:</p>
<pre><code class="language-bash">make snapshot STAGE=3 &amp;&amp; make snapshots
</code></pre>
<h2 id="heading-stage-4-admission-control-kyverno">Stage 4 — Admission Control (Kyverno)</h2>
<p>Even if CI passes, the cluster can still refuse.</p>
<p>CI scans your code and images before they reach GitOps, but it can't watch everything that happens inside the cluster. Someone with <code>kubectl</code> access could apply a manifest directly.</p>
<p>A Helm chart you install might create pods that violate your security standards. Those paths never hit the pipeline, which is why Stage 4 adds admission control: a checkpoint built into Kubernetes itself.</p>
<p>Every time something tries to create or update a resource, the request passes through admission webhooks before it takes effect. If a webhook rejects the request, the resource is never created.</p>
<p><strong>Kyverno</strong> is a Kubernetes-native policy engine that uses those webhooks. You write policies as YAML files (not application code), and Kyverno enforces them on every matching resource in the cluster, for example, rejecting any pod that runs as root or requiring CPU and memory limits on every container.</p>
<p>The difference from CI is timing: CI scans <em>before</em> code ships, while Kyverno enforces at the <em>cluster gate</em>. Together they give you two layers of defense.</p>
<p>Your goal in this stage is to install Kyverno, apply the policies in <code>infra/policies/</code>, and prove in §4.4 that non-compliant pods are denied before the container runtime ever sees them.</p>
<p>Before you start, make sure the foundation from earlier stages is still solid: <code>make check-3</code> should pass (pre-commit hooks and CI security gates are active), <code>infra/cosign.pub</code> should exist from Stage 3, and ArgoCD should still be syncing so the app responds at <code>http://clearledger.local</code>. If any of those are red, fix them first: Kyverno sits on top of a healthy cluster, not a broken one.</p>
<p>You're done with Stage 4 when all three break-it scenarios in §4.4 are denied and <code>make check-4</code> passes.</p>
<p><strong>What changes from Stage 3 is enforcement, not scanning.</strong> In CI, Checkov reported Kubernetes misconfigurations but didn't block the pipeline. Kyverno now stops those same classes of problems at the cluster gate.</p>
<p>Cosign has been signing your images since Stage 1. Kyverno now <em>requires</em> that signature before a ClearLedger image can deploy. This is where <a href="#heading-stage-1-security-posture-what-blocks-vs-what-waits">Stage 1 evidence becomes enforcement</a>. See that section if you want the full map of what blocked in Stage 1 versus what waited for Stage 4.</p>
<p>Start with §4.1 to install Kyverno. If the install, policies, break-it scenarios, or <code>make check-4</code> fail, read <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md"><code>troubleshooting.md</code></a> and <strong>Stage 4: Admission Control (Kyverno)</strong> before changing Helm values or policy YAML.</p>
<h3 id="heading-what-kyverno-enforces">What Kyverno Enforces</h3>
<p>All policy files live in <code>infra/policies/</code>. Kyverno itself is installed via Helm using <code>stages/stage-4-admission-control/infra/kyverno/values.yaml</code>.</p>
<table>
<thead>
<tr>
<th>Policy</th>
<th>What it enforces</th>
<th>Framework</th>
</tr>
</thead>
<tbody><tr>
<td><code>disallow-root-containers</code></td>
<td><code>runAsNonRoot: true</code></td>
<td>CIS K8s 5.2.6</td>
</tr>
<tr>
<td><code>require-resource-limits</code></td>
<td>CPU/memory requests and limits</td>
<td>CIS K8s 5.2.4</td>
</tr>
<tr>
<td><code>disallow-privilege-escalation</code></td>
<td><code>allowPrivilegeEscalation: false</code></td>
<td>CIS K8s 5.2.5</td>
</tr>
<tr>
<td><code>drop-all-capabilities</code></td>
<td><code>capabilities.drop: [ALL]</code></td>
<td>CIS K8s 5.2.7</td>
</tr>
<tr>
<td><code>require-signed-images</code></td>
<td>Cosign signature on ClearLedger images</td>
<td>SLSA Level 2</td>
</tr>
</tbody></table>
<h3 id="heading-platform-stability-from-stage-4-onward">Platform Stability: From Stage 4 Onward</h3>
<p>From Stage 4 on, you're running more controllers on a single-node VM. Kyverno, storage provisioners, and later Prometheus and Loki. A pod can show <code>Running</code> while it's actually crash-looping in the background.</p>
<p>When platform pods (Kyverno controllers, <code>hostpath-provisioner</code>, the Prometheus operator, and similar) accumulate high <code>RESTARTS</code>, the API server starts timing out, <code>kubectl</code> feels flaky, and you can waste days debugging the wrong component because the app pods look fine.</p>
<p>After every stage from here on, give the cluster about ten minutes to settle, then run the stage health check:</p>
<pre><code class="language-bash">bash scripts/health-check.sh &lt;stage&gt;    # for example, 4, 7, 7.5
# or the Makefile shortcut:
make check-4
</code></pre>
<p>The script ends with a Platform stability section that flags pods with suspicious restart counts. You can also scan the worst offenders yourself. This lists the fifteen pods with the highest restart counts cluster-wide, which is useful when something feels slow but you're not sure which namespace is struggling:</p>
<pre><code class="language-bash">kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount' \
  | tail -15
</code></pre>
<p><strong>The gate:</strong> Kyverno controllers and other platform pods should show <strong>RESTARTS under 5</strong> after the stage settles. If any platform pod is climbing past 10, stop and fix it with the documented Helm values or <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a>. Don't <code>kubectl patch</code> around it and move on. A stable platform layer is a prerequisite for every stage that follows.</p>
<h3 id="heading-41-install-kyverno">4.1: Install Kyverno</h3>
<pre><code class="language-bash">helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm upgrade --install kyverno kyverno/kyverno \
  --version 3.2.8 \
  --namespace kyverno \
  --create-namespace \
  -f stages/stage-4-admission-control/infra/kyverno/values.yaml \
  --wait --timeout=600s
</code></pre>
<p>The values file does three important things for the lab:</p>
<ol>
<li><strong>Disables cleanup CronJobs</strong>: older Kyverno charts pull <code>bitnami/kubectl</code>, which was removed from Docker Hub and causes <code>ImagePullBackOff</code> on cleanup pods.</li>
<li><strong>Points Helm hooks at</strong> <code>bitnamilegacy/kubectl</code>, so future <code>helm uninstall</code> doesn't hang on a missing image.</li>
<li><strong>Extends liveness probe timeouts</strong>: the default <code>timeoutSeconds: 5, failureThreshold: 2</code> is too tight for a loaded single-node VM. Under CPU pressure, the health endpoint can take &gt;5s to respond, which triggers a restart cascade that saturates the node and makes the API server intermittently unreachable. The values file sets <code>timeoutSeconds: 30, failureThreshold: 5</code> so Kyverno survives load spikes without crash-looping.</li>
</ol>
<p><strong>What you should see:</strong></p>
<pre><code class="language-yaml">Release "kyverno" does not exist. Installing it now.
NAME: kyverno
NAMESPACE: kyverno
STATUS: deployed
...
Kyverno version: v1.12.6
</code></pre>
<p>Verify all four controllers are running (first pull can take several minutes on a slow connection):</p>
<pre><code class="language-markdown">kubectl get pods -n kyverno
</code></pre>
<pre><code class="language-plaintext">NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-bd685cd4b-f6kl6     1/1     Running   0          2m
kyverno-background-controller-66fcfc6d87-59wgt   1/1     Running   0          2m
kyverno-cleanup-controller-5c5bf8bc6b-7kspq      1/1     Running   0          2m
kyverno-reports-controller-5cdd6f4c48-qf5wc      1/1     Running   0          2m
</code></pre>
<p>If pods stay in <code>ContainerCreating</code> for a long time, the node is still pulling images from <code>ghcr.io/kyverno</code>. Wait. Don't start a second Helm install on top of a partial one.</p>
<h4 id="heading-stability-gate-kyverno-install-only-before-42">Stability gate: Kyverno install only (before §4.2):</h4>
<p>Before continuing, make sure the Kyverno pods are healthy:</p>
<pre><code class="language-bash">kubectl get pods -n kyverno
</code></pre>
<p><strong>Expected:</strong> the Kyverno controller pods show <code>1/1 Running</code>, with low restart counts such as <code>0</code>, <code>1</code>, or <code>2</code>, and the restart count isn't increasing.</p>
<p>Don't run <code>make check-4</code> yet. That check also looks for the policies you apply later in §4.3, so it may fail at this point even if Kyverno installed correctly.</p>
<h3 id="heading-42-confirm-your-cosign-public-key-is-in-the-policy">4.2: Confirm your Cosign Public Key is in the Policy</h3>
<p>Stage 3 created <code>infra/cosign.pub</code>. Kyverno uses that same key to verify image signatures when a pod is created. The policy file ships with a placeholder. You must replace it with your key before applying policies in §4.3.</p>
<h4 id="heading-step-1-show-your-key-run-from-the-repo-root-on-the-vm">Step 1: Show your key (run from the repo root on the VM)</h4>
<pre><code class="language-bash">cd ~/clearledger    # or wherever you cloned the repo
cat infra/cosign.pub
</code></pre>
<p>You should see three lines: <code>-----BEGIN PUBLIC KEY-----</code>, a long base64 line, and <code>-----END PUBLIC KEY-----</code>. Copy that whole block (you'll paste it in the next step).</p>
<h4 id="heading-step-2-paste-the-key-into-the-policy">Step 2: Paste the key into the policy</h4>
<p>Open <code>infra/policies/require-signed-images.yaml</code> in your editor (<code>nano</code>, <code>vim</code>, or VS Code).</p>
<p>Find this line:</p>
<pre><code class="language-yaml">                      PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE
</code></pre>
<p>Delete <strong>only</strong> that placeholder line and paste the three lines from <code>cosign.pub</code> in its place. The result should look like this (your base64 line will differ):</p>
<pre><code class="language-yaml">                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                     JFkwEwYHKoZIzj0CAQYIKoFIzj0DAQcDQgZEI...
                      -----END PUBLIC KEY-----
</code></pre>
<p>Save the file. Keep the pasted key indented under <code>publicKeys: |-</code>. The <code>BEGIN PUBLIC KEY</code> and <code>END PUBLIC KEY</code> lines should have spaces before them, just like the base64 line between them.</p>
<h4 id="heading-step-3-verify-three-quick-checks">Step 3: Verify (three quick checks)</h4>
<p>Run these one at a time from the repo root:</p>
<pre><code class="language-bash"># Check A — placeholder must be gone
grep PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE infra/policies/require-signed-images.yaml \
  &amp;&amp; echo "❌ FAIL: placeholder still in file — edit and save again" \
  || echo "✓ OK: placeholder removed"
</code></pre>
<pre><code class="language-bash"># Check B — key block must be present exactly once
grep -c "BEGIN PUBLIC KEY" infra/policies/require-signed-images.yaml
</code></pre>
<p>Expected output for Check B: <code>1</code> (if you see <code>0</code>, the key was not pasted. If <code>2</code>, you pasted it twice).</p>
<pre><code class="language-bash"># Check C — policy key must match cosign.pub byte-for-byte
diff infra/cosign.pub \
  &lt;(sed -n '/-----BEGIN PUBLIC KEY-----/,/-----END PUBLIC KEY-----/p' \
      infra/policies/require-signed-images.yaml | sed 's/^[[:space:]]*//')
</code></pre>
<p>Expected output for Check C: <strong>nothing</strong>. No diff lines means the keys match. If <code>diff</code> prints differences, open the policy file and fix the paste.</p>
<p>If all three passed, continue to §4.3.</p>
<p><strong>If you skip this</strong>, Scenario 3 in §4.4 fails in a confusing way: unsigned images may slip through, or signed pods may be rejected because Kyverno is checking against the wrong key.</p>
<h3 id="heading-43-apply-the-five-core-policies">4.3: Apply the Five Core Policies</h3>
<p>Now you'll apply the five policies that map to CIS controls. Don't apply <code>verify-slsa-provenance.yaml</code> yet. It's an optional SLSA attestation policy (Audit mode) for a later enhancement.</p>
<p>Stage 4 applies <code>infra/policies/require-signed-images.yaml</code>. This policy uses <code>failurePolicy: Fail</code>, so if Kyverno can't verify an image signature, the pod is blocked instead of allowed. The ECR policy with <code>failurePolicy: Ignore</code> is for Stage 8, not this step.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/67a638f5-65b8-41d2-be68-babe6c7b8c99.png" alt="screenshot image showing infra policy Yaml file failurePolicy as &quot;Fail&quot;" style="display: block;" width="600" height="400" loading="lazy">

<pre><code class="language-bash">kubectl apply \
  -f infra/policies/disallow-root.yaml \
  -f infra/policies/disallow-privilege-escalation.yaml \
  -f infra/policies/drop-all-capabilities.yaml \
  -f infra/policies/require-resource-limits.yaml \
  -f infra/policies/require-signed-images.yaml
</code></pre>
<p>Wait a few seconds, then confirm all policies show <code>READY: True</code> and <code>VALIDATE ACTION: Enforce</code>:</p>
<pre><code class="language-bash">kubectl get clusterpolicy
</code></pre>
<pre><code class="language-plaintext">NAME                            ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE
disallow-privilege-escalation   true        true         Enforce           True    10s
disallow-root-containers        true        true         Enforce           True    10s
drop-all-capabilities           true        true         Enforce           True    10s
require-resource-limits         true        true         Enforce           True    10s
require-signed-images           true        false        Enforce           True    10s
</code></pre>
<p>If <code>READY</code> stays empty, check Kyverno logs: <code>kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller --tail=50</code>.</p>
<h3 id="heading-44-breaking-it-on-purpose">4.4: Breaking it on Purpose</h3>
<p>Now you'll test the policies by trying to create bad pods.</p>
<p>These pods are supposed to fail. That's the point.</p>
<p>CI tools like Checkov warn you in a report. Kyverno goes further: it blocks unsafe pods before Kubernetes runs them.</p>
<p>For each test, read the error message, as it should tell you which policy blocked the pod and what field was wrong. That error message is your proof that admission control is working.</p>
<table>
<thead>
<tr>
<th>Scenario</th>
<th>What you simulate</th>
<th>Policy under test</th>
<th>Success looks like</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Attacker applies a bare pod (no hardening)</td>
<td>Root, caps, privilege, limits</td>
<td>Four policies fire, pod <code>NotFound</code></td>
</tr>
<tr>
<td>2</td>
<td>Developer fixes securityContext but forgets limits</td>
<td>Resource limits only</td>
<td>One policy fires, pod <code>NotFound</code></td>
</tr>
<tr>
<td>3</td>
<td>Attacker pushes unsigned image to Docker Hub</td>
<td>Cosign signature</td>
<td><code>require-signed-images</code> denies, pod <code>NotFound</code></td>
</tr>
</tbody></table>
<h4 id="heading-scenario-1-root-container-no-securitycontext">Scenario 1: root container (no securityContext)</h4>
<p><strong>What you're simulating:</strong> Someone with <code>kubectl</code> access bypasses CI and applies a minimal pod: no <code>securityContext</code>, no resource limits.</p>
<p>This is exactly what Stage 1 Checkov flagged as evidence. Stage 4 now blocks it.</p>
<p><strong>What's wrong with this manifest:</strong> The container has only a name and image. It will run as root by default, keep all Linux capabilities, and has no CPU/memory bounds.</p>
<pre><code class="language-bash">cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: root-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
EOF
</code></pre>
<p><strong>What you should see:</strong></p>
<pre><code class="language-yaml">Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/root-test was blocked due to the following policies

disallow-privilege-escalation:
  check-allowPrivilegeEscalation: 'validation error: allowPrivilegeEscalation must
    be set to false. rule check-allowPrivilegeEscalation failed at path /spec/containers/0/securityContext/'
disallow-root-containers:
  check-runAsNonRoot: |-
    validation error: Root containers are blocked in the clearledger namespace. Set securityContext.runAsNonRoot: true on the pod or container.
    . rule check-runAsNonRoot failed at path /spec/containers/0/securityContext/
drop-all-capabilities:
  check-capabilities: 'validation error: All containers must drop ALL capabilities.
    rule check-capabilities failed at path /spec/containers/0/securityContext/'
require-resource-limits:
  check-resources: 'validation error: Resource requests and limits are required for
    all containers. rule check-resources failed at path /spec/containers/0/resources/limits/'
</code></pre>
<p><strong>How to read this output:</strong></p>
<p>The important line is:</p>
<pre><code class="language-text">resource Pod/clearledger/root-test was blocked due to the following policies
</code></pre>
<p>That means Kyverno stopped the pod before it was created.</p>
<p>Under that line, Kyverno lists every policy the pod failed. For example:</p>
<pre><code class="language-text">disallow-root-containers:
  check-runAsNonRoot:
</code></pre>
<p>This means the pod failed the <code>disallow-root-containers</code> policy, specifically the <code>check-runAsNonRoot</code> rule. The fix is also shown in the message:</p>
<pre><code class="language-text">Set securityContext.runAsNonRoot: true
</code></pre>
<p>The same pattern applies to the other policies:</p>
<ul>
<li><code>disallow-privilege-escalation</code> means the pod didn't set <code>allowPrivilegeEscalation: false</code></li>
<li><code>drop-all-capabilities</code> means the pod didn't drop Linux capabilities with <code>capabilities.drop: [ALL]</code></li>
<li><code>require-resource-limits</code> means the pod didn't set CPU and memory requests/limits</li>
</ul>
<p>The <code>path</code> part tells you where Kubernetes expected the missing setting. For example, <code>/spec/containers/0/securityContext/</code> means: look inside the pod spec, then the first container, then its <code>securityContext</code>.</p>
<p>And <code>/spec/containers/0/resources/limits/</code> means: look inside the first container's resource limits.</p>
<p>So this one bad pod failed four controls at once. That's the lesson: Kyverno doesn't just say "no." It tells you which policy failed and where to fix the YAML.</p>
<p><strong>Verify enforcement worked:</strong></p>
<pre><code class="language-bash">kubectl get pod root-test -n clearledger
# Error from server (NotFound): pods "root-test" not found
</code></pre>
<p>If you see a pod in <code>Running</code> or <code>Pending</code>, policies aren't enforcing: re-check that <code>kubectl get clusterpolicy</code> shows all five <code>READY: True</code>.</p>
<p><strong>Take a screenshot.</strong> This is portfolio evidence for CIS Kubernetes Benchmark 5.2.6: enforced, not just configured.</p>
<h4 id="heading-scenario-2-missing-resource-limits">Scenario 2: missing resource limits</h4>
<p><strong>What you're simulating:</strong> A developer who read the securityContext requirements and fixed root/caps/privilege. But skipped resource limits.</p>
<p>This is common in real teams: “we hardened the container” but forgot CPU/memory bounds.</p>
<p><strong>What's wrong with this manifest:</strong> <code>securityContext</code> is correct, but there's no <code>resources.requests</code> or <code>resources.limits</code>. A container without limits can starve other workloads on the node.</p>
<pre><code class="language-bash">cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nolimits-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
EOF
</code></pre>
<p><strong>What you should see:</strong></p>
<pre><code class="language-plaintext">Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/nolimits-test was blocked due to the following policies

require-resource-limits:
  check-resources: 'validation error: Resource requests and limits are required for
    all containers. rule check-resources failed at path /spec/containers/0/resources/limits/'
</code></pre>
<p><strong>Key observation:</strong> Only one policy fires this time: the securityContext fields satisfied the other four rules. Kyverno evaluates rules independently. Each container property is a separate gate.</p>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pod nolimits-test -n clearledger
# Error from server (NotFound): pods "nolimits-test" not found
</code></pre>
<h4 id="heading-scenario-3-unsigned-clearledger-image">Scenario 3: unsigned ClearLedger image</h4>
<p><strong>What you're simulating:</strong> A supply-chain attack: someone pushes a malicious image to Docker Hub under your repo name (<code>clearledger-auth-service</code>) without going through your signed CI pipeline. Stage 3 made Cosign signing possible, while Stage 4 makes it mandatory at the cluster gate.</p>
<p><strong>Why this setup is needed:</strong> Kyverno checks image signatures against the image in Docker Hub, not against images on your laptop. The test image tag must exist in Docker Hub first.</p>
<p>If you use a fake tag like <code>:unsigned</code> that was never pushed, Kubernetes may fail later with <code>ImagePullBackOff</code>. That only means the image can't be pulled; it doesn't prove Kyverno blocked an unsigned image.</p>
<h4 id="heading-step-1-push-a-deliberately-unsigned-test-image-one-time">Step 1: push a deliberately unsigned test image (one-time):</h4>
<pre><code class="language-bash">export DOCKER_USERNAME=your-dockerhub-username

docker pull nginx:alpine
docker tag nginx:alpine ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
docker push ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test

# Must fail — proves the image has no Cosign signature from your pipeline key:
cosign verify --key infra/cosign.pub \
  index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
# Error: no signatures found
</code></pre>
<h4 id="heading-step-2-try-to-deploy-it-with-a-compliant-pod-spec">Step 2: try to deploy it with a compliant pod spec:</h4>
<p>The pod manifest is fully hardened (securityContext + limits) so only the signature policy can fail. Use <code>index.docker.io/</code> in the image URL: on Kyverno 1.12, <code>docker.io/...</code> may not trigger <code>verifyImages</code> matching.</p>
<pre><code class="language-bash">cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
      resources:
        requests:
          memory: "64Mi"
          cpu: "50m"
        limits:
          memory: "128Mi"
          cpu: "200m"
EOF
</code></pre>
<p><strong>What you should see:</strong></p>
<pre><code class="language-plaintext">Error from server: error when creating "STDIN": admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/unsigned-test was blocked due to the following policies

require-signed-images:
  verify-cosign-signature: 'failed to verify image index.docker.io/veeno-demo/clearledger-auth-service:unsigned-test:
    .attestors[0].entries[0].keys: no signatures found'
</code></pre>
<p><strong>How to read this output:</strong></p>
<ul>
<li>Note the webhook name is <code>mutate.kyverno.svc-fail</code>, not <code>validate</code>: image verification runs in Kyverno’s mutate pass (digest + signature check) before the pod is admitted.</li>
<li><code>no signatures found</code> means Kyverno reached Docker Hub, found the image, and confirmed it was <strong>not</strong> signed with your <code>infra/cosign.pub</code> key.</li>
<li>The pod never exists: the attacker can't get a shell even if the image is pullable.</li>
</ul>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pod unsigned-test -n clearledger
# Error from server (NotFound): pods "unsigned-test" not found
</code></pre>
<p><strong>What you should NOT see</strong> (these mean the test didn't prove signature enforcement):</p>
<table>
<thead>
<tr>
<th>Symptom</th>
<th>What went wrong</th>
</tr>
</thead>
<tbody><tr>
<td>Pod created, then <code>ImagePullBackOff</code></td>
<td>Tag does not exist on Docker Hub, complete Step 1 first</td>
</tr>
<tr>
<td>Pod created and <code>Running</code></td>
<td>Image used <code>docker.io/...</code> instead of <code>index.docker.io/...</code></td>
</tr>
<tr>
<td>No <code>require-signed-images</code> in the error</td>
<td>Policy not applied, or <code>cosign.pub</code> not embedded in the policy YAML</td>
</tr>
</tbody></table>
<h4 id="heading-contrast-signed-image-is-allowed">Contrast: signed image is allowed:</h4>
<p>The previous test used an unsigned image, so Kyverno blocked it.</p>
<p>Your real ClearLedger images should be signed by the CI pipeline. If the pod also follows the security rules, Kyverno allows it to run.</p>
<p>You can check the image currently used by <code>auth-service</code>:</p>
<pre><code class="language-bash"># Your deployed tag (signed in CI) should start if spec is compliant:
kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
# docker.io/veeno-demo/clearledger-auth-service:v0.1.0
</code></pre>
<p><strong>Example output:</strong></p>
<p><code>docker.io/veeno-demo/clearledger-auth-service:v0.1.0</code></p>
<p>Pods that were already running before the policies were applied will keep running. The important test is what happens when Kubernetes creates a new pod. New pods using signed ClearLedger images should pass Kyverno verification.</p>
<p>Take a screenshot of the Scenario 3 denial. It proves the cluster blocks unsigned images, not just that CI signs images.</p>
<h3 id="heading-45-verify-clearledger-still-works">4.5: Verify ClearLedger Still Works</h3>
<p>Kyverno enforces on new pod creation. Existing deployments that already passed admission (or were synced before policies existed) keep running. Confirm your app pods are healthy:</p>
<pre><code class="language-bash">kubectl get pods -n clearledger
</code></pre>
<pre><code class="language-plaintext">NAME                                    READY   STATUS    RESTARTS   AGE
auth-service-...                        1/1     Running   0          ...
frontend-...                            1/1     Running   0          ...
ledger-service-...                      1/1     Running   0          ...
notification-service-...                1/1     Running   0          ...
postgres-0                              1/1     Running   0          ...
redis-...                               1/1     Running   0          ...
</code></pre>
<p>If ingress is configured:</p>
<pre><code class="language-bash">curl -s http://clearledger.local/auth/health | jq .
# {"status": "ok", "service": "auth-service"}
</code></pre>
<p>ArgoCD should still show <strong>Synced</strong> and <strong>Healthy</strong>: GitOps and admission control work together, not against each other.</p>
<h3 id="heading-46-policy-exceptions-when-a-legitimate-workload-needs-a-bypass">4.6: Policy Exceptions (When a Legitimate Workload Needs a Bypass)</h3>
<p>Kyverno blocks every pod that violates a policy. But what happens when a legitimate workload needs to bypass a specific rule?</p>
<p>PostgreSQL is the example. The official Postgres Alpine image uses a specific internal user (UID 70) to manage its data directory. The <code>disallow-root-containers</code> policy requires every pod to set <code>runAsNonRoot: true</code>.</p>
<p>Postgres does set that. But if Kyverno is configured to also check specific UID ranges, or if the pod's security context doesn't satisfy the rule for any reason, Kyverno blocks it. The database can't start, and the entire application fails.</p>
<p>You can't weaken the policy cluster-wide to accommodate one database. That would let every pod bypass the rule. Instead, you create a <strong>PolicyException</strong>: a targeted exemption for exactly the pods that need it.</p>
<p>Open <a href="../infra/policies/exceptions/postgres-root-exception.yaml"><code>infra/policies/exceptions/postgres-root-exception.yaml</code></a> and read the comments. Here's what each section does:</p>
<p><strong>The</strong> <code>spec.exceptions</code> <strong>block</strong> identifies which policy and rule to bypass:</p>
<pre><code class="language-yaml">exceptions:
  - policyName: disallow-root-containers
    ruleNames:
      - check-runAsNonRoot
</code></pre>
<p>This says: "skip only the <code>check-runAsNonRoot</code> rule from the <code>disallow-root-containers</code> policy." Every other rule in that policy (and every other policy in the cluster) still enforces normally.</p>
<p><strong>The</strong> <code>spec.match</code> <strong>block</strong> limits which resources get the exception:</p>
<pre><code class="language-yaml">match:
  any:
    - resources:
        kinds:
          - Pod
        namespaces:
          - clearledger
        names:
          - postgres-*
</code></pre>
<p>Only pods named <code>postgres-*</code> (matching <code>postgres-0</code>, <code>postgres-1</code>, and so on), only in the <code>clearledger</code> namespace, only for the <code>Pod</code> resource kind. Everything else in the cluster still follows the strict policy.</p>
<p><strong>The annotations</strong> are documentation for your team and auditors:</p>
<pre><code class="language-yaml">annotations:
  reason: "Postgres alpine image requires UID 70 for data directory ownership"
  approved-by: "platform-team"
  review-date: "2026-01-01"
</code></pre>
<p>These have no technical effect: Kyverno ignores them. They exist so that six months from now, when someone asks "why does Postgres bypass this rule?", the answer is right there in the file.</p>
<p><strong>The rules for safe exceptions:</strong></p>
<ol>
<li><strong>Scope narrowly</strong>: target the exact resource that needs it, nothing more</li>
<li><strong>Commit to Git</strong>: the exception is reviewed in a pull request, tracked in version history, and auditable</li>
<li><strong>Never weaken the policy itself</strong>: the rule stays strict for everything else</li>
<li><strong>Review periodically</strong>: exceptions should be temporary if possible, and re-evaluated on a schedule</li>
</ol>
<p>Apply the exception <strong>only if</strong> Kyverno blocks your Postgres pods:</p>
<pre><code class="language-bash">kubectl apply -f infra/policies/exceptions/postgres-root-exception.yaml
</code></pre>
<p>Verify Kyverno still blocks other non-compliant pods (same denial as Scenario 1):</p>
<pre><code class="language-bash">cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: another-root-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
EOF
</code></pre>
<h3 id="heading-47-cis-benchmark-evidence-kube-bench">4.7: CIS Benchmark Evidence <code>kube-bench</code>)</h3>
<p>You already installed Kyverno and proved it blocks unsafe pods.</p>
<p>This step is different. <code>kube-bench</code> doesn't block pods and doesn't change the cluster. It only checks the Kubernetes node against the CIS benchmark and saves evidence.</p>
<p>Think of the difference like this:</p>
<table>
<thead>
<tr>
<th>Tool</th>
<th>What it checks</th>
<th>Question it answers</th>
</tr>
</thead>
<tbody><tr>
<td>Kyverno</td>
<td>Pods and workloads</td>
<td>"Is this pod allowed to run?"</td>
</tr>
<tr>
<td>kube-bench</td>
<td>Kubernetes node settings</td>
<td>"Is this Kubernetes node hardened?"</td>
</tr>
</tbody></table>
<p>Both are useful, but only Kyverno blocks workloads in this lab.</p>
<p>Run kube-bench:</p>
<pre><code class="language-bash">bash stages/stage-4-admission-control/scripts/run-kube-bench.sh
</code></pre>
<p>The script runs kube-bench as a Kubernetes Job and saves the report here:</p>
<pre><code class="language-text">stages/stage-4-admission-control/scripts/kube-bench-report.json
</code></pre>
<p>It also compares the result against this baseline:</p>
<pre><code class="language-text">stages/stage-4-admission-control/scripts/kube-bench-baseline.json
</code></pre>
<p>On MicroK8s, you'll see many <code>FAIL</code> and <code>WARN</code> lines. That's expected. The lab isn't asking you to fix every CIS warning on a single-node local VM.</p>
<p>What matters is the final result.</p>
<p>Pass looks like this:</p>
<pre><code class="language-text">kube-bench: 1 FAIL control(s) present (documented in baseline — no regressions).
kube-bench: no regressions vs baseline.
</code></pre>
<p>That means the known MicroK8s issues are documented, and your cluster didn't get worse.</p>
<p>If you see <code>REGRESSION</code> or <code>make check-4</code> fails on kube-bench, stop and investigate before Stage 5.</p>
<p>Optional: confirm the report file exists:</p>
<pre><code class="language-bash">ls -la stages/stage-4-admission-control/scripts/kube-bench-report.json
</code></pre>
<p>In production, you would either fix the CIS failures or document approved exceptions. In this lab, the baseline records the expected MicroK8s state.</p>
<h3 id="heading-48-health-check">4.8: Health Check</h3>
<pre><code class="language-bash">make check-4
</code></pre>
<p><strong>What you should see:</strong></p>
<pre><code class="language-plaintext">▶ Stage 4 — Admission Control (Kyverno)
  ✓ Kyverno is running
  ✓ Policy disallow-root-containers — Enforce mode
  ✓ Policy require-resource-limits — Enforce mode
  ✓ Policy require-signed-images — Enforce mode
  ✓ Policy disallow-privilege-escalation — Enforce mode
  ✓ Policy drop-all-capabilities — Enforce mode
  ✓ Kyverno correctly rejects pods without securityContext
  ✓ kube-bench baseline exists (...)

All checks passed. Ready for the next stage.
</code></pre>
<p>If kube-bench reports regressions, run the script manually and update the baseline after reviewing. That diff is audit evidence.</p>
<p>If Kyverno install, policies, break-it scenarios, or <code>make check-4</code> fail, see <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md. Stage 4</a>.</p>
<h3 id="heading-stage-4-complete-done-checklist-move-to-stage-5">Stage 4 Complete: Done Checklist (Move to Stage 5)</h3>
<p>You're <strong>done with Stage 4</strong> when all of these are true:</p>
<table>
<thead>
<tr>
<th>#</th>
<th>Check</th>
<th>How to verify</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Kyverno running</td>
<td><code>kubectl get pods -n kyverno</code> — four controllers <code>Running</code></td>
</tr>
<tr>
<td>2</td>
<td>Policies applied</td>
<td><code>kubectl get clusterpolicy</code> — five policies, <code>READY: True</code>, <code>Enforce</code></td>
</tr>
<tr>
<td>3</td>
<td>Root pod blocked</td>
<td>Scenario 1 denial in terminal (screenshot for portfolio)</td>
</tr>
<tr>
<td>4</td>
<td>Unsigned image blocked</td>
<td>Scenario 3 denial — push <code>unsigned-test</code> tag first, use <code>index.docker.io/</code></td>
</tr>
<tr>
<td>5</td>
<td>App still healthy</td>
<td><code>kubectl get pods -n clearledger</code> — all app pods <code>Running</code></td>
</tr>
<tr>
<td>6</td>
<td>Health check green</td>
<td><code>make check-4</code> ends with <code>All checks passed. Ready for the next stage.</code></td>
</tr>
</tbody></table>
<p><strong>Portfolio screenshots (optional):</strong> root-pod denial (§4.4 Scenario 1), unsigned-image denial (§4.4 Scenario 3), and <code>kubectl get clusterpolicy</code> showing five <code>Enforce</code> policies.</p>
<p>Not yet: SLSA attestation (optional), Vault secrets (Stage 5), network policies (Stage 6). Passwords still live in Kubernetes Secrets. Stage 5 moves them into Vault.</p>
<h3 id="heading-what-you-learned-in-stage-4">What You Learned in Stage 4</h3>
<ul>
<li>The difference between CI scanning (before merge) and admission control (at the cluster gate)</li>
<li>What Kyverno is: a policy engine that intercepts every Kubernetes API request</li>
<li>That enforcement means the bad resource never exists, not "we detected it after the fact"</li>
<li>How to read a Kyverno denial: policy name → rule name → JSON path that failed</li>
<li>How to write and apply cluster-wide security policies as YAML</li>
<li>How to scope a PolicyException without weakening the policy for everyone else</li>
<li>That operational issues (Helm, image pulls, registry URL format) affect whether controls actually fire</li>
<li><strong>Why both CI and admission control are needed:</strong> CI catches problems in your code while Kyverno catches everything else that touches the cluster</li>
<li><strong>Evidence beats configuration:</strong> a policy file in Git means nothing: the break-it denials are proof CIS controls are enforced, not just documented.</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Enforced admission control with Kyverno: blocking root containers, privilege escalation, unsigned images, and missing resource limits at deploy time: mapped to CIS Kubernetes benchmarks.</p>
</blockquote>
<p><code>make snapshot STAGE=4 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage4</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<h2 id="heading-stage-5-secrets-management-vault">Stage 5: Secrets Management (Vault)</h2>
<p>By the end of this stage, sensitive values no longer live in Git or in etcd-backed Kubernetes Secrets: Vault holds them centrally and injects them into pods only when they start.</p>
<p><strong>Your goal:</strong> remove <code>auth-service-secret</code> and <code>ledger-service-secret</code> from the cluster.</p>
<p>Login and API calls must still work because Vault injects credentials at pod startup. That's the moment secrets management clicks.</p>
<p><strong>Before you start</strong>, confirm Stage 4 is solid: <code>make check-4</code> passes, all five Kyverno policies are enforcing, and the app responds at <code>http://clearledger.local</code>. Fix any crash-looping pods before installing Vault.</p>
<h3 id="heading-what-changes-in-this-stage">What Changes in This Stage</h3>
<p>Right now, database passwords and JWT keys sit in <code>secret.yaml</code> files on GitHub and in Kubernetes Secrets inside the cluster. In Stage 5 you move those values into <strong>HashiCorp Vault</strong> and teach the app to read them a different way.</p>
<p>When an auth or ledger pod starts, the <strong>Vault agent injector</strong> adds a small sidecar container. That sidecar logs into Vault using the pod’s own service account, fetches the password and JWT, and writes them as files under <code>/vault/secrets/</code>.</p>
<p>Your app already knows how to read those paths. It's the same data that used to arrive via <code>secretKeyRef</code>, just delivered at runtime instead of pulled from a Kubernetes Secret object.</p>
<p>Once migration is complete, sensitive values live in <strong>Vault</strong> (the long-term store) and briefly on the <strong>pod filesystem</strong> while the container runs. They're not in Git anymore. You remove <code>secret.yaml</code> from <code>clearledger-infra</code> and ArgoCD syncs deployments that point at Vault instead.</p>
<p>To load Vault the first time, you copy a template to a local <code>.env</code> file (§5.1). That file is gitignored. You run <code>seed-vault-secrets.sh</code> once to copy those values into Vault.</p>
<p>Real secret values aren't written into committed scripts. The scripts read secrets from your local <code>.env</code> file or from your terminal, so passwords and tokens stay out of Git.</p>
<h3 id="heading-do-the-steps-in-this-order">Do the Steps in This Order</h3>
<p>Each step depends on the one before it. Skipping ahead is the most common way to get red auth/ledger pods that look like a broken app but really mean “Vault is not ready yet.”</p>
<ol>
<li><strong>§5.1</strong>: copy <code>stages/stage-5-secrets-management/.env.example</code> to <code>.env</code>, then fill it with your cluster passwords</li>
<li><strong>§5.2</strong>: install Vault and the agent injector with Helm</li>
<li><strong>§5.3</strong>. Run <code>setup.sh</code>, then <code>seed-vault-secrets.sh</code> (passwords now live in Vault)</li>
<li><strong>§5.4</strong>: push Vault-enabled deployments to <code>clearledger-infra</code>. Let ArgoCD sync.</li>
<li><strong>§5.5</strong>. Wait for <strong>2/2</strong> pods (app + Vault sidecar), then delete the old Kubernetes Secrets</li>
<li><strong>§5.5b</strong>: ArgoCD <strong>Synced / Healthy</strong> (after secret delete. OutOfSync before delete is normal)</li>
<li><strong>§5.6</strong>. Confirm login works and credentials appear under <code>/vault/secrets/</code> inside the pod</li>
</ol>
<p>Start at <strong>§5.1</strong>. If anything fails, read <code>troubleshooting.md.</code> before changing manifests.</p>
<h3 id="heading-51-create-env-local-only-never-commit">5.1: Create <code>.env</code> (Local Only, Never Commit)</h3>
<p>This file holds two things: a dev Vault root token for Helm (§5.2), and the passwords you'll load into Vault in §5.3.</p>
<p>It stays on your machine only. Never commit it. The <code>SEED_*</code> values must match what the app uses today so login still works after you delete Kubernetes Secrets later.</p>
<p>Two different files. <strong>Don't mix them up:</strong></p>
<table>
<thead>
<tr>
<th>File</th>
<th>What it is</th>
</tr>
</thead>
<tbody><tr>
<td><code>stages/stage-5-secrets-management/.env.example</code></td>
<td>Blank template in the repo (empty fields). Copy this in step 1.</td>
</tr>
<tr>
<td><code>stages/stage-5-secrets-management/.env</code></td>
<td>Your real file (gitignored). You create it and fill it in steps 2–3.</td>
</tr>
</tbody></table>
<p>The sample block at the bottom of this section is only a picture of what a completed <code>.env</code> looks like: don't copy those placeholder passwords unless they happen to match your cluster.</p>
<h4 id="heading-step-1-copy-the-template-to-env">Step 1: copy the template to <code>.env</code></h4>
<pre><code class="language-bash">cp stages/stage-5-secrets-management/.env.example \
   stages/stage-5-secrets-management/.env
</code></pre>
<p>That gives you a file with empty <code>VAULT_TOKEN=</code> and <code>SEED_*=</code> lines. Open it in your editor for steps 2–3.</p>
<h4 id="heading-step-2-read-the-current-passwords-from-the-cluster">Step 2: read the current passwords from the cluster</h4>
<p>Run these from the repo root. Each command prints one value: copy the output into <code>.env</code> in step 3.</p>
<pre><code class="language-bash"># → paste as SEED_AUTH_DATABASE_URL
kubectl get secret auth-service-secret -n clearledger \
  -o jsonpath='{.data.database_url}' | base64 -d; echo

# → paste as SEED_AUTH_JWT_SECRET
kubectl get secret auth-service-secret -n clearledger \
  -o jsonpath='{.data.jwt_secret}' | base64 -d; echo

# → paste as SEED_LEDGER_DATABASE_URL
kubectl get secret ledger-service-secret -n clearledger \
  -o jsonpath='{.data.database_url}' | base64 -d; echo
</code></pre>
<h4 id="heading-step-3-fill-in-env">Step 3: fill in <code>.env</code></h4>
<table>
<thead>
<tr>
<th>Variable</th>
<th>What to put</th>
</tr>
</thead>
<tbody><tr>
<td><code>VAULT_TOKEN</code></td>
<td>Any dev-only string you choose (for example, <code>my-dev-root-token</code>): same value in §5.2 Helm install</td>
</tr>
<tr>
<td><code>SEED_AUTH_DATABASE_URL</code></td>
<td>Output of first command above</td>
</tr>
<tr>
<td><code>SEED_AUTH_JWT_SECRET</code></td>
<td>Output of second command</td>
</tr>
<tr>
<td><code>SEED_LEDGER_DATABASE_URL</code></td>
<td>Output of third command</td>
</tr>
</tbody></table>
<p><strong>Sample only: shape of a completed</strong> <code>.env</code> (use your kubectl output from step 2, not these example strings unless they match):</p>
<pre><code class="language-text">VAULT_TOKEN=my-dev-root-token
SEED_AUTH_DATABASE_URL=postgresql://clearledger:changeme-stage0@postgres:5432/clearledger
SEED_AUTH_JWT_SECRET=stage0-jwt-secret-change-in-production
SEED_LEDGER_DATABASE_URL=postgresql://clearledger:changeme-stage0@postgres:5432/clearledger
</code></pre>
<p>If <code>auth-service-secret</code> is already deleted (you skipped ahead: recover like this):</p>
<pre><code class="language-bash"># Database URL from Postgres bootstrap secret (lab default password is often changeme-stage0)
PG_PASS=$(kubectl get secret postgres-secret -n clearledger \
  -o jsonpath='{.data.password}' | base64 -d)
echo "postgresql://clearledger:${PG_PASS}@postgres:5432/clearledger"
# Use that line for both SEED_AUTH_DATABASE_URL and SEED_LEDGER_DATABASE_URL

# JWT: same value you used at Stage 0, or read from Vault if you already seeded:
kubectl exec -n vault vault-0 -- vault kv get -field=jwt_secret clearledger/auth-service 2&gt;/dev/null \
  || echo "(set SEED_AUTH_JWT_SECRET manually — must match tokens already issued)"
</code></pre>
<p>Continue to <strong>§5.2</strong> once <code>.env</code> has all four variables set.</p>
<h3 id="heading-52-install-vault-and-the-agent-injector">5.2: Install Vault and the Agent Injector</h3>
<pre><code class="language-bash">set -a &amp;&amp; source stages/stage-5-secrets-management/.env &amp;&amp; set +a

helm repo add hashicorp https://helm.releases.hashicorp.com &amp;&amp; helm repo update

# First install:
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set server.dev.enabled=true \
  --set server.dev.devRootToken="${VAULT_TOKEN}" \
  --set ui.enabled=true \
  --set injector.enabled=true

# If helm install fails with "cannot re-use a name", use upgrade instead:
# helm upgrade --install vault hashicorp/vault \
#   --namespace vault --create-namespace \
#   --set server.dev.enabled=true \
#   --set server.dev.devRootToken="${VAULT_TOKEN}" \
#   --set ui.enabled=true \
#   --set injector.enabled=true

kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=vault -n vault --timeout=120s
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=vault-agent-injector -n vault --timeout=120s

kubectl apply -f stages/stage-5-secrets-management/infra/vault-ingress.yaml
</code></pre>
<p>Open <a href="http://vault.local"><code>http://vault.local</code></a> in your browser. Log in with the value you set as <code>VAULT_TOKEN</code> in <code>stages/stage-5-secrets-management/.env</code>. For example, if your <code>.env</code> has <code>VAULT_TOKEN=my-dev-root-token</code>, use <code>my-dev-root-token</code> as the Vault login token.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/8a2b0002-2ee3-4b04-89f0-e43da18fc9b9.png" alt="screenshot showing vault UI" style="display: block;" width="600" height="400" loading="lazy">

<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ec5fdb0b-7258-4e41-aeb5-adaa8754dbf2.png" alt="screenshot showing vault UI" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>Verify: list Vault pods:</strong></p>
<pre><code class="language-bash">kubectl get pods -n vault
</code></pre>
<p><strong>Expected: Vault pods:</strong></p>
<pre><code class="language-text">NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                1/1     Running   0          1m
vault-agent-injector-8d6b668b4-xxxxx   1/1     Running   0          1m
</code></pre>
<p><strong>If</strong> <code>helm install</code> <strong>fails with “cannot re-use a name”</strong>: Vault is already installed. Use the <code>helm upgrade --install</code> block above.</p>
<h3 id="heading-53-configure-vault-platform-seed-kv">5.3: Configure Vault (Platform + Seed KV)</h3>
<p>Run both scripts in order. Each reads <code>VAULT_TOKEN</code> from your <code>.env</code>.</p>
<pre><code class="language-bash">bash stages/stage-5-secrets-management/infra/vault/setup.sh
bash stages/stage-5-secrets-management/infra/vault/seed-vault-secrets.sh
</code></pre>
<p><code>setup.sh</code>: prepares Vault for the cluster: Kubernetes auth, the KV secret store, policies, and roles so auth/ledger pods <em>can</em> fetch secrets later. It doesn't write your database passwords yet and nothing goes to Git.</p>
<p><code>seed-vault-secrets.sh</code>: takes the <code>SEED_*</code> lines from <code>.env</code> and stores them in Vault at <code>clearledger/data/auth-service</code> and <code>clearledger/data/ledger-service</code>. It doesn't echo those values to the terminal.</p>
<p>Re-running either script is safe for the lab.</p>
<p><strong>Expected,</strong> <code>setup.sh</code> <strong>(tail):</strong></p>
<pre><code class="language-text">==&gt; Enabling Kubernetes auth method...
==&gt; Configuring Kubernetes auth...
==&gt; Enabling KV secrets engine...
==&gt; Creating Vault policies...
==&gt; Creating Kubernetes auth roles...
==&gt; Applying RBAC + ServiceAccounts...

✓ Vault platform setup complete (no secrets written yet).
  Next: bash stages/stage-5-secrets-management/infra/vault/seed-vault-secrets.sh
</code></pre>
<p><strong>Expected,</strong> <code>seed-vault-secrets.sh</code><strong>:</strong></p>
<pre><code class="language-text">==&gt; Logging into Vault...
==&gt; Writing secrets to Vault KV (values are not printed)...
======== Secret Path ========
clearledger/data/auth-service
======= Metadata =======
Key                Value
---                -----
created_time       2026-06-01T15:31:53.538991153Z
version            1
✓ Secrets stored at clearledger/data/auth-service and clearledger/data/ledger-service
</code></pre>
<p><strong>Verify metadata only</strong> (no secret values printed):</p>
<pre><code class="language-bash">kubectl exec -n vault vault-0 -- vault kv metadata get clearledger/auth-service
</code></pre>
<pre><code class="language-text">Key                     Value
---                     -----
cas_required            false
created_time            2026-06-01T15:31:53.538991153Z
current_version         1
delete_version_after    0s
max_versions            0
oldest_version          0
updated_time            2026-06-01T15:31:53.538991153Z
</code></pre>
<h3 id="heading-54-gitops-update-clearledger-infra-fixes-argocd-outofsync">5.4: GitOps: Update <code>clearledger-infra</code> (Fixes ArgoCD OutOfSync)</h3>
<p>ArgoCD deploys from your <code>clearledger-infra</code> GitHub repo, not from the main <code>clearledger</code> app repo where you're working now. You edit manifests here first, then copy the same changes to <code>clearledger-infra</code> so ArgoCD can sync them. Work slowly and verify after each sub-step.</p>
<h4 id="heading-54a-update-manifests-in-the-app-repo-clearledger">5.4a. Update manifests in the app repo (<code>clearledger</code>)</h4>
<pre><code class="language-bash">cp stages/stage-5-secrets-management/infra/manifests/auth-service/deployment.yaml \
   infra/manifests/auth-service/deployment.yaml

cp stages/stage-5-secrets-management/infra/manifests/ledger-service/deployment.yaml \
   infra/manifests/ledger-service/deployment.yaml\

mkdir -p infra/manifests/vault

cp infra/deferred-by-stage/stage-5-secrets-management/vault/rotation-cronjob.yaml \
   infra/manifests/vault/rotation-cronjob.yaml

rm -f infra/manifests/auth-service/secret.yaml infra/manifests/ledger-service/secret.yaml
</code></pre>
<h4 id="heading-54b-edit-inframanifestskustomizationyaml-by-hand">5.4b. Edit <code>infra/manifests/kustomization.yaml</code> by hand</h4>
<p>Open the file in your editor. In the <code>resources:</code> list:</p>
<ul>
<li><p><strong>Remove</strong> the app secret entries: delete these two lines, or comment them out with <code>#</code> (both work, as Kustomize ignores <code>#</code> lines):</p>
<pre><code class="language-yaml">- auth-service/secret.yaml
- ledger-service/secret.yaml
</code></pre>
</li>
<li><p><strong>Add</strong> this line (with the other resources):</p>
<pre><code class="language-yaml">- vault/rotation-cronjob.yaml
</code></pre>
</li>
</ul>
<p>Leave <code>postgres/postgres-secret.yaml</code>, that is Postgres bootstrap only, not app credentials.</p>
<p>Save. Verify:</p>
<pre><code class="language-bash"># Active (uncommented) app secret lines must be gone — postgres-secret is OK
grep -E '^[[:space:]]*-[[:space:]]+(auth-service|ledger-service)/secret\.yaml' \
  infra/manifests/kustomization.yaml &amp;&amp; echo "STOP: app secrets still active" || echo "OK"

grep vault/rotation-cronjob.yaml infra/manifests/kustomization.yaml
grep vault.hashicorp infra/manifests/auth-service/deployment.yaml | head -1
kustomize build infra/manifests &gt;/dev/null &amp;&amp; echo "OK: kustomize build"
</code></pre>
<p>Expected: <code>OK</code>, rotation cronjob listed, first line shows <code>vault.hashicorp.com/agent-inject</code>, kustomize build succeeds.</p>
<p>Commit in the <strong>app</strong> repo when ready: <code>git add infra/manifests &amp;&amp; git commit -m "feat(stage-5): Vault deployments in canonical manifests"</code>.</p>
<h4 id="heading-54c-push-the-same-changes-to-clearledger-infra">5.4c. Push the same changes to <code>clearledger-infra</code></h4>
<pre><code class="language-bash">git clone https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/clearledger-infra
</code></pre>
<p>If clone fails with <code>destination path '/tmp/clearledger-infra' already exists</code> (you cloned in §1.3 or an earlier step), reuse that folder. Don't clone again:</p>
<pre><code class="language-bash">cd /tmp/clearledger-infra &amp;&amp; git pull &amp;&amp; cd -
</code></pre>
<p>Or start fresh: <code>rm -rf /tmp/clearledger-infra</code> then run <code>git clone</code> again.</p>
<p><strong>Run the</strong> <code>cp</code> <strong>commands from the main</strong> <code>clearledger</code> <strong>app repo</strong>, not from <code>/tmp/clearledger-infra</code>. Your shell prompt should say <code>clearledger</code>, not <code>clearledger-infra</code>. The source path <code>infra/manifests/...</code> only exists in the app repo.</p>
<pre><code class="language-bash">cd ~/clearledger    # main app repo — adjust path if yours differs

cp infra/manifests/auth-service/deployment.yaml /tmp/clearledger-infra/manifests/auth-service/
cp infra/manifests/ledger-service/deployment.yaml /tmp/clearledger-infra/manifests/ledger-service/
mkdir -p /tmp/clearledger-infra/manifests/vault
cp infra/manifests/vault/rotation-cronjob.yaml /tmp/clearledger-infra/manifests/vault/
cp infra/manifests/kustomization.yaml /tmp/clearledger-infra/manifests/kustomization.yaml
rm -f /tmp/clearledger-infra/manifests/auth-service/secret.yaml
rm -f /tmp/clearledger-infra/manifests/ledger-service/secret.yaml

cd /tmp/clearledger-infra
git add -A
git status
git commit -m "feat(stage-5): Vault injection; remove app secrets from GitOps"
git push
cd -
</code></pre>
<p><strong>✋ Hands-on checkpoint. Stage 5 GitOps landed</strong></p>
<pre><code class="language-bash">git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-s5
test ! -f /tmp/verify-s5/manifests/auth-service/secret.yaml &amp;&amp; echo "OK: app secret removed from Git"
grep vault.hashicorp /tmp/verify-s5/manifests/auth-service/deployment.yaml | head -1
grep vault/rotation-cronjob.yaml /tmp/verify-s5/manifests/kustomization.yaml
rm -rf /tmp/verify-s5
</code></pre>
<p>Expected: <code>OK</code>, Vault annotation present, rotation job in kustomization.</p>
<p><strong>Expected,</strong> <code>git status</code> <strong>before commit (step 5.4c):</strong></p>
<pre><code class="language-text">modified:   manifests/auth-service/deployment.yaml
modified:   manifests/ledger-service/deployment.yaml
modified:   manifests/kustomization.yaml
new file:   manifests/vault/rotation-cronjob.yaml
deleted:    manifests/auth-service/secret.yaml
deleted:    manifests/ledger-service/secret.yaml
</code></pre>
<p>After <code>git push</code>, ArgoCD will roll out Vault-enabled deployments automatically. <strong>Continue to §5.5</strong>. Don't expect <strong>Synced</strong> yet, as app secrets are still in the cluster until you delete them there.</p>
<p><strong>Common rollout failures:</strong></p>
<table>
<thead>
<tr>
<th>Symptom</th>
<th>Fix</th>
</tr>
</thead>
<tbody><tr>
<td><code>Duplicate value: "vault-secrets"</code></td>
<td>Do <strong>not</strong> declare a <code>vault-secrets</code> volume in <code>deployment.yaml</code>: the injector creates it</td>
</tr>
<tr>
<td><code>Service appeared 2 times</code></td>
<td>Keep <code>Service</code> only in <code>service.yaml</code>, not at the bottom of <code>deployment.yaml</code></td>
</tr>
<tr>
<td>Kyverno <code>containers/0</code> <code>runAsNonRoot</code></td>
<td>Add <code>runAsNonRoot: true</code> on the <strong>app</strong> container <code>securityContext</code>, not only on <code>spec.securityContext</code></td>
</tr>
<tr>
<td>Pods stuck <code>1/1</code> (no sidecar)</td>
<td>Confirm <code>injector.enabled=true</code> and deployment has <code>vault.hashicorp.com/agent-inject: "true"</code></td>
</tr>
<tr>
<td><code>permission denied</code> in vault-agent-init</td>
<td>Run <code>setup.sh</code> : K8s auth role not bound to service account</td>
</tr>
<tr>
<td>ArgoCD <strong>Sync failed</strong> on <code>CronJob/vault-secret-rotation</code></td>
<td>Kyverno blocked the job: <code>infra/manifests/vault/rotation-cronjob.yaml</code> must include <code>runAsNonRoot</code>, <code>allowPrivilegeEscalation: false</code>, <code>capabilities.drop: [ALL]</code>, and CPU/memory limits. Push fix to <code>clearledger-infra</code>.</td>
</tr>
</tbody></table>
<h3 id="heading-55-wait-for-vault-injected-pods-then-delete-k8s-app-secrets">5.5: Wait for Vault-injected Pods, Then Delete K8s App Secrets</h3>
<p><strong>Wait until auth/ledger show Vault sidecars</strong> (<code>READY 2/2</code> = app + vault-agent):</p>
<pre><code class="language-bash">kubectl get pods -n clearledger -l app=auth-service
kubectl get pods -n clearledger -l app=ledger-service
</code></pre>
<p><strong>Expected:</strong></p>
<pre><code class="language-text">NAME                            READY   STATUS    RESTARTS   AGE
auth-service-5756d9fcb9-bmdlr   2/2     Running   0          2m
auth-service-5756d9fcb9-jtgss   2/2     Running   0          2m
</code></pre>
<p>Inspect sidecar pulled secrets (init container logs):</p>
<pre><code class="language-bash">kubectl logs -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c vault-agent-init
# ... Authentication successful, rendering templates ...
</code></pre>
<p><strong>Only after pods are 2/2</strong>, delete app Secrets:</p>
<pre><code class="language-bash">kubectl delete secret auth-service-secret ledger-service-secret -n clearledger
</code></pre>
<p><strong>Expected: secrets remaining:</strong></p>
<pre><code class="language-bash">kubectl get secret -n clearledger
</code></pre>
<pre><code class="language-text">NAME              TYPE     DATA   AGE
postgres-secret   Opaque   2      6d
</code></pre>
<p><code>postgres-secret</code> is Postgres bootstrap only, not app credentials. That stays until you harden Postgres separately.</p>
<p>If delete says <code>NotFound</code>: secrets were already removed. Continue to §5.6.</p>
<h3 id="heading-55b-argocd-should-be-synced-after-secret-delete">5.5b: ArgoCD Should Be Synced After Secret Delete</h3>
<p>Run this after §5.5, not right after §5.4. Before you delete app Secrets, OutOfSync is normal. Git no longer lists <code>auth-service-secret</code> / <code>ledger-service-secret</code>, but they still exist in the cluster until you delete them in the step above.</p>
<pre><code class="language-bash">kubectl get application clearledger -n argocd \
  -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'
</code></pre>
<p><strong>Before secret delete:</strong> expect <code>sync=OutOfSync health=Healthy</code> or <code>Progressing</code> while Vault pods roll out. That's fine if auth/ledger are <strong>2/2</strong>.</p>
<p><strong>After secret delete</strong>, hard-refresh and sync if still OutOfSync:</p>
<pre><code class="language-bash">kubectl annotate application clearledger -n argocd argocd.argoproj.io/refresh=hard --overwrite
argocd app sync clearledger --grpc-web --prune
</code></pre>
<p>If sync says <strong>another operation is already in progress</strong>, wait a minute: ArgoCD auto-sync is already running.</p>
<p>Wait until:</p>
<pre><code class="language-bash">kubectl get application clearledger -n argocd \
  -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'
# Synced Healthy
</code></pre>
<p>Don't update app deployments with <code>kubectl apply</code> after ArgoCD is managing them. ArgoCD keeps the cluster matched to <code>clearledger-infra</code>. If you change a deployment by hand, ArgoCD may revert it. For Stage 5, update the manifests in Git and let ArgoCD sync the Vault-enabled deployments.</p>
<h3 id="heading-56-login-and-injected-files">5.6: Login and Injected Files</h3>
<pre><code class="language-bash">kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- ls /vault/secrets/
</code></pre>
<pre><code class="language-text">database_url
jwt_secret
</code></pre>
<pre><code class="language-bash">curl -s -X POST http://clearledger.local/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' | jq .
</code></pre>
<p><strong>Expected:</strong></p>
<pre><code class="language-json">{
  "access_token": "&lt;jwt-returned-by-auth-service&gt;",
  "token_type": "bearer"
}
</code></pre>
<p><strong>Take a screenshot:</strong> working login JSON + <code>kubectl get secret -n clearledger</code> showing no <code>auth-service-secret</code> / <code>ledger-service-secret</code>.</p>
<h3 id="heading-57-health-check">5.7: Health Check</h3>
<pre><code class="language-bash">make check-5
</code></pre>
<p><strong>What you should see:</strong></p>
<blockquote>
<p><code>make check-5</code> re-runs Stage 4 checks first, that is expected. Look for the Stage 5 block below to confirm Vault is working.</p>
</blockquote>
<pre><code class="language-text">▶ Stage 4: Admission Control (Kyverno)
  ✓ Kyverno is running
  ✓ Policy disallow-root-containers — Enforce mode
  ...
  ✓ kube-bench matches baseline (no new FAIL regressions)

▶ Stage 5: Secrets Management (Vault)
  ✓ Vault pod is running
  ✓ Vault agent injector is running
  ✓ Vault is unsealed
  ✓ Vault Kubernetes auth method is enabled
  ✓ auth-service-secret removed — Vault is the secret source
  ✓ Vault injected /vault/secrets/database_url into auth-service

All checks passed. Ready for the next stage.
</code></pre>
<p>If Vault injection or ArgoCD sync fails, see <code>troubleshooting.md</code>.</p>
<h3 id="heading-stage-5-is-done-checklist-before-moving-to-stage-6">Stage 5 is Done: Checklist Before Moving to Stage 6</h3>
<table>
<thead>
<tr>
<th>#</th>
<th>Check</th>
<th>How to verify</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Secrets in Vault only</td>
<td><code>vault kv metadata get clearledger/auth-service</code> shows <code>current_version &gt;= 1</code></td>
</tr>
<tr>
<td>2</td>
<td>No app secrets in infra Git</td>
<td><code>secret.yaml</code> absent from <code>clearledger-infra/manifests/auth-service/</code> and <code>ledger-service/</code></td>
</tr>
<tr>
<td>3</td>
<td>ArgoCD synced</td>
<td><code>Synced Healthy</code> on Application <code>clearledger</code></td>
</tr>
<tr>
<td>4</td>
<td>K8s app secrets deleted</td>
<td><code>kubectl get secret -n clearledger</code> no auth/ledger app secrets</td>
</tr>
<tr>
<td>5</td>
<td>Injection works</td>
<td>Auth pods <code>2/2</code>; <code>ls /vault/secrets/</code> shows <code>database_url</code>, <code>jwt_secret</code></td>
</tr>
<tr>
<td>6</td>
<td>App works</td>
<td>Login curl returns <code>access_token</code></td>
</tr>
<tr>
<td>7</td>
<td>Health check</td>
<td><code>make check-5</code> ends with <code>All checks passed. Ready for the next stage.</code></td>
</tr>
</tbody></table>
<p>Stage 5 moves app credentials out of Git and Kubernetes Secrets. It doesn't make Vault production-grade yet. This lab still uses Vault dev mode, not HA or auto-unseal.</p>
<p>Also, a running pod can still read the files under <code>/vault/secrets/</code> because the app needs those credentials to work. That's normal. Stage 6 adds Falco so you can detect suspicious runtime access.</p>
<h3 id="heading-what-you-learned-in-stage-5">What You Learned in Stage 5</h3>
<ul>
<li>Kubernetes Secrets aren't enough for real secret management.</li>
<li>Vault now stores the app credentials.</li>
<li><code>.env</code> was only used locally to load the first secrets into Vault. It's never committed.</li>
<li>Vault injects secrets into the pod when the app starts.</li>
<li><code>clearledger-infra</code> must stop storing <code>secret.yaml</code>, because ArgoCD deploys from that repo.</li>
<li>The order matters: install Vault, seed secrets, update GitOps, wait for healthy pods, then delete old Kubernetes Secrets.</li>
</ul>
<p><strong>What you can now say in an interview:</strong></p>
<blockquote>
<p>I replaced Kubernetes Secrets with HashiCorp Vault agent injection, removed app credentials from Git and Kubernetes Secrets, and verified the app still worked after Vault injected the credentials at runtime.</p>
</blockquote>
<p>Save your progress:</p>
<pre><code class="language-bash">make snapshot STAGE=5 &amp;&amp; make snapshots
</code></pre>
<p>Confirm <code>clearledger.stage5</code> appears in the snapshot list.</p>
<h2 id="heading-stage-6-runtime-security-falco">Stage 6 — Runtime Security (Falco)</h2>
<p>Stages 1–5 secured what gets deployed and how secrets are stored. Stage 6 watches what happens inside running containers after they start.</p>
<p>Your goal is to learn what runtime security catches and why it matters, then prove it by triggering a Falco alert and reading it the way an on-call engineer would.</p>
<p>CI, Kyverno, and Vault all act before or at pod startup. Falco fills the gap they leave open. It watches what running software actually does inside the container. That's the layer incident response and forensics care about, not just another chart to install.</p>
<p><strong>Before you start Stage 6:</strong></p>
<ul>
<li><code>make check-5</code> passes</li>
<li>Login and transactions still work at <code>http://clearledger.local</code></li>
<li>Platform pods have low restart counts</li>
</ul>
<p>You're done with Stage 6 when:</p>
<ul>
<li>You trigger at least one Falco alert</li>
<li>You apply the network policies</li>
<li><code>make check-6</code> passes</li>
</ul>
<p>Then save your VM:</p>
<pre><code class="language-bash">make snapshot STAGE=6
make snapshots
</code></pre>
<h3 id="heading-do-the-steps-in-this-order">Do the Steps in This Order</h3>
<p>Each step depends on the one before it. Don't run <code>make check-6</code> until §6.4. It checks network policies you haven't applied yet.</p>
<ol>
<li><strong>§6.1:</strong> <code>bash stages/stage-6-runtime-security/scripts/install-falco.sh</code>. Confirm <code>falco-*</code> pods <code>2/2 Running</code> and custom rules loaded.</li>
<li><strong>§6.2:</strong> <code>make demo-6</code>: read <code>✓ Runtime detection confirmed</code> in the terminal</li>
<li><strong>§6.3</strong> (optional) manual break-it scenarios (skip if <code>make demo-6</code> already worked)</li>
<li><strong>§6.4:</strong> <code>kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml</code>. Confirm <code>curl http://clearledger.local/</code> returns 200.</li>
<li><strong>§6.6:</strong> <code>make check-6</code></li>
</ol>
<p>Start at <strong>§6.1</strong>. If anything fails, see <code>troubleshooting.md</code>.</p>
<p><strong>Optional reading:</strong> <a href="#heading-how-stage-6-fits-the-full-stack-optional-reading">How Stage 6 fits the full stack</a>: why Falco and netpol exist and how they differ from Stages 3–5.</p>
<h3 id="heading-if-you-get-stuck-in-stage-6">If You Get Stuck in Stage 6</h3>
<p>Stage 6 has three jobs:</p>
<ol>
<li>Install Falco</li>
<li>Trigger one test alert</li>
<li>Apply network policies</li>
</ol>
<p>Don't worry about every row in the Falco UI. The UI may show noise. You pass the Falco part when you can find one alert from your demo, either in the terminal or in the UI.</p>
<p>For the portfolio screenshot, open:</p>
<p><code>http://falco.local</code></p>
<p>Login:</p>
<ul>
<li>Username: <code>admin</code></li>
<li>Password: <code>admin</code></li>
</ul>
<p>Take a screenshot only after your demo alert appears.</p>
<p><strong>Common stuck points</strong></p>
<table>
<thead>
<tr>
<th>You think…</th>
<th>What is actually true</th>
</tr>
</thead>
<tbody><tr>
<td>“The UI shows 200+ Critical alerts, maybe I broke something”</td>
<td>No. <code>postgres-0</code> reads <code>/etc/passwd</code> on a loop and Falco flags it. Ignore those rows.</td>
</tr>
<tr>
<td>“I can't find my demo alert”</td>
<td>Search the UI with <strong>Cmd+F →</strong> <code>Shell Spawned</code>, or use the <strong>terminal grep</strong> in step 4 above. If grep shows <code>auth-service</code> + <code>id &amp;&amp; exit</code>, you passed.</td>
</tr>
<tr>
<td>“<code>make check-6</code> failed on NetworkPolicy”</td>
<td>You ran the check <strong>before §6.4</strong>. Apply netpol first, then re-run.</td>
</tr>
<tr>
<td>“§6.3 vs §6.2 — which do I run?”</td>
<td>Run <code>make demo-6</code> <strong>(§6.2)</strong> only. §6.3 is the same attacks as manual commands. Skip it if demo-6 already worked.</td>
</tr>
<tr>
<td>“What is Shell Spawned?”</td>
<td>Falco saw a <code>sh</code> <strong>process start</strong> inside <code>auth-service</code>. That's suspicious in production. In the lab, <strong>you</strong> caused it on purpose. See §6.2.</td>
</tr>
<tr>
<td>“Scenario 4 hangs or exit 137”</td>
<td>Old <code>wget</code> command + <strong>Terminating</strong> pod. Skip Scenario 4 or use the <strong>python3</strong> command in §6.4. Checkpoint + <code>make check-6</code> is enough.</td>
</tr>
</tbody></table>
<h3 id="heading-61-install-falco-and-falcosidekick-ui">6.1: Install Falco and Falcosidekick UI</h3>
<pre><code class="language-bash">bash stages/stage-6-runtime-security/scripts/install-falco.sh
</code></pre>
<p>This runs <code>helm upgrade --install</code> with <code>modern_ebpf</code>, enables Falcosidekick + Web UI, enables the <strong>k8s-metacollector</strong> (<code>collectors.kubernetes.enabled: true</code>) so custom rules can match <code>k8smeta.ns.name = clearledger</code>, loads rules from <code>infra/falco/clearledger-rules-content.yaml</code>, applies the rules ConfigMap and ingress.</p>
<p>If Falco is already installed, the script is safe to re-run (upgrade).</p>
<p><strong>Verify Falco pods:</strong></p>
<pre><code class="language-bash">kubectl get pods -n falco
</code></pre>
<p><strong>Expected output:</strong></p>
<pre><code class="language-text">NAME                                      READY   STATUS    RESTARTS   AGE
falco-w4fh6                               2/2     Running   0          2m
falco-falcosidekick-...                   1/1     Running   0          2m
falco-falcosidekick-ui-...                1/1     Running   0          2m
falco-falcosidekick-ui-redis-0            1/1     Running   0          2m
</code></pre>
<p>The Falco DaemonSet should show <strong>2/2 Running</strong>. Sidekick, UI, and Redis pods should each show <strong>1/1 Running</strong>. Pod name suffixes on your cluster will differ from the example.</p>
<p>Open <code>http://falco.local</code>. You'll see the Falcosidekick UI. Log in with the chart defaults:</p>
<table>
<thead>
<tr>
<th>Field</th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Login</strong></td>
<td><code>admin</code></td>
</tr>
<tr>
<td><strong>Password</strong></td>
<td><code>admin</code></td>
</tr>
</tbody></table>
<p>To read the credentials from the cluster instead of trusting the lab defaults:</p>
<pre><code class="language-bash">kubectl get secret falco-falcosidekick-ui -n falco \
  -o jsonpath='{.data.FALCOSIDEKICK_UI_USER}' | base64 -d &amp;&amp; echo
# admin:admin
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/eeac67ee-182a-403d-806d-328e1fdcd8b7.png" alt="Screenshot fo Falco UI" style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-falcosidekick-ui-quick-orientation">Falcosidekick UI: quick orientation</h4>
<p>After login you land on the <strong>Events</strong> tab. The table can look busy before you run any demo, which is normal.</p>
<ul>
<li><strong>Rule</strong>: detection name (what fired)</li>
<li><strong>Priority</strong>: <strong>Critical</strong> / <strong>Warning</strong> / <strong>Notice</strong> (focus on Critical and Warning for this lab)</li>
<li><strong>Output</strong>: pod name, file, or command details</li>
<li><strong>Tags</strong>: look for <code>clearledger</code> on lab alerts</li>
</ul>
<p><strong>Background noise you can ignore:</strong> Notice rows from ArgoCD. <strong>Critical</strong> <strong>Sensitive File Read</strong> rows from <code>postgres-0</code> reading <code>/etc/passwd</code> (repeats every few seconds). Your demo alert is different. See §6.2.</p>
<p><strong>Verify custom rules loaded</strong> (do this before §6.2):</p>
<pre><code class="language-bash">kubectl get pods -n falco                                    # Falco pod 2/2 Running
kubectl get configmap clearledger-falco-rules -n falco
kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=200 \
  | grep 'rules.d/clearledger_rules'
</code></pre>
<p><strong>Expected:</strong> <code>clearledger_rules.yaml | schema validation: ok</code></p>
<p>An empty grep with <code>--tail=30</code> alone isn't a failure. Use <code>--tail=200</code>. If you see <code>LOAD_ERR_COMPILE_CONDITION</code>, see <code>troubleshooting.md</code>.</p>
<p>If rules didn't load, §6.2 and §6.3 will look like they passed when nothing fired.</p>
<h3 id="heading-62-guided-demo-make-demo-6">6.2: Guided Demo (<code>make demo-6</code>)</h3>
<p>Run this <strong>after</strong> §6.1 (Falco installed, rules verified, UI opens at <code>http://falco.local</code>).</p>
<pre><code class="language-bash">make demo-6
# or:
bash stages/stage-6-runtime-security/scripts/demo-falco-alerts.sh
</code></pre>
<h4 id="heading-what-the-demo-script-does">What the demo script does</h4>
<p>The demo proves Falco can detect suspicious activity inside a running container.</p>
<p>The script checks that Falco is running, opens <code>http://falco.local</code>, and waits while you log in with:</p>
<ul>
<li>Username: <code>admin</code></li>
<li>Password: <code>admin</code></li>
</ul>
<p>Then it runs this test command inside the <code>auth-service</code> container:</p>
<pre><code class="language-bash">kubectl exec -n clearledger \
  auth-service-&lt;pod-suffix&gt; \
  -c auth-service -- /bin/sh -c 'id &amp;&amp; exit'
</code></pre>
<p>The script picks the real pod name for you.</p>
<p><strong>Non-interactive</strong> (CI or no Enter prompts): <code>SKIP_PROMPT=1 make demo-6</code>.</p>
<p>This starts a shell inside the app container. That's suspicious in production because app containers should run the app, not open shells. Falco should detect it and create an alert called:</p>
<p><code>Shell Spawned in ClearLedger Container</code></p>
<p>When the script prints:</p>
<pre><code class="language-text">✓ Runtime detection confirmed
</code></pre>
<p>refresh the Falco UI.</p>
<p>Look for a <strong>Critical</strong> alert with:</p>
<ul>
<li>Rule: <code>Shell Spawned in ClearLedger Container</code></li>
<li>Pod: <code>auth-service-...</code></li>
<li>Command: <code>sh -c id &amp;&amp; exit</code></li>
</ul>
<p>Ignore alerts from <code>postgres-0</code>, especially <code>Sensitive File Read</code>. Those are background noise for this lab.</p>
<p>If the UI is noisy, search the page for <code>Shell Spawned</code> or check from the terminal:</p>
<pre><code class="language-bash">kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=500 \
  | grep 'Shell Spawned'
</code></pre>
<p>For your screenshot, capture the <code>Shell Spawned</code> alert for <code>auth-service</code>.</p>
<h3 id="heading-63-break-it-scenarios-manual-optional">6.3: Break-it Scenarios (Manual, Optional)</h3>
<p>These are the same detections as §6.2, but you run each command yourself. Skip this section if you already completed <code>make demo-6</code>.</p>
<table>
<thead>
<tr>
<th>Rule name</th>
<th>You trigger it by…</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Shell Spawned in ClearLedger Container</strong></td>
<td>Scenario 1 — <code>kubectl exec … /bin/sh</code></td>
</tr>
<tr>
<td><strong>Sensitive File Read in ClearLedger</strong></td>
<td>Scenario 2 — <code>cat /etc/passwd</code></td>
</tr>
<tr>
<td><strong>Package Manager / Outbound connection</strong></td>
<td>Scenario 3 — <code>wget</code> or <code>curl</code></td>
</tr>
</tbody></table>
<p>After each command, refresh <code>http://falco.local</code> or use the find methods in §6.2.</p>
<h4 id="heading-scenario-1-shell-in-a-running-pod-command-injection-simulation">Scenario 1 – Shell in a running pod (command injection simulation):</h4>
<pre><code class="language-bash">kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- /bin/sh -c "id &amp;&amp; exit"
</code></pre>
<p><strong>Expected in Falco UI / logs</strong> (within ~10 seconds):</p>
<pre><code class="language-text">CRITICAL: Shell spawned in ClearLedger container
  user=... container=auth-service pod=auth-service-... cmd=sh -c id &amp;&amp; exit
</code></pre>
<p><strong>What this means:</strong> Stage 4 allowed the pod (it is compliant). Stage 6 detected <em>behavior inside</em> the pod: exactly what an attacker would do after command injection.</p>
<p><strong>If you see no alert:</strong> confirm the exec used <code>-c auth-service</code> (not the vault-agent sidecar), rules show <code>schema validation: ok</code>, and the pod image name contains <code>clearledger</code>.</p>
<h4 id="heading-scenario-2-read-a-sensitive-file-reconnaissance">Scenario 2 – Read a sensitive file (reconnaissance):</h4>
<pre><code class="language-bash">kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- cat /etc/passwd
</code></pre>
<p><strong>Expected:</strong></p>
<pre><code class="language-text">CRITICAL: Sensitive file read in ClearLedger
  file=/etc/passwd container=auth-service pod=auth-service-...
</code></pre>
<h4 id="heading-scenario-3-download-tool-at-runtime-optional">Scenario 3 – Download tool at runtime (optional):</h4>
<pre><code class="language-bash">kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- sh -c "wget -q ifconfig.me -O - 2&gt;/dev/null || true"
</code></pre>
<p>May fire Package manager executed and/or Unexpected outbound connection (WARNING).</p>
<p>Take screenshots of Scenarios 1 and 2: portfolio evidence for runtime detection.</p>
<h3 id="heading-64-apply-network-policies-zero-trust-segmentation">6.4: Apply Network Policies (Zero-trust Segmentation)</h3>
<p>Network policies are firewall rules between pods. Apply them after the Falco demo.</p>
<p><code>make check-6</code> checks for these policies, so run it only after this section. The <code>default-deny-all</code> policy blocks traffic by default.</p>
<p>The <code>allow-*</code> policies open only the paths ClearLedger needs to work. Falco detects suspicious behavior. Network policies limit where a pod can connect.</p>
<p><strong>Apply:</strong></p>
<pre><code class="language-bash">kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml
kubectl get networkpolicy -n clearledger
</code></pre>
<p><strong>Expected:</strong> seven policies: <code>default-deny-all</code> plus six <code>allow-*</code> (<code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>postgres</code>, <code>redis</code>, <code>frontend</code>).</p>
<p>Verify the app still works:</p>
<pre><code class="language-bash">curl -s http://clearledger.local/auth/health | jq .
# {"status":"ok","service":"auth-service"}

curl -s http://clearledger.local/notifications/health | jq .
# {"status":"ok",...}
</code></pre>
<p><strong>Checkpoint (required)</strong>: proves netpol didn't break the real app:</p>
<pre><code class="language-bash">kubectl get networkpolicy -n clearledger
curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/
kubectl get pods -n clearledger --field-selector=status.phase!=Running
</code></pre>
<table>
<thead>
<tr>
<th>Result</th>
<th>Meaning</th>
</tr>
</thead>
<tbody><tr>
<td>Seven policies listed</td>
<td>Netpol applied</td>
</tr>
<tr>
<td><code>200</code> from curl</td>
<td>Users can still reach the app through ingress</td>
</tr>
<tr>
<td>Third command prints <strong>nothing</strong></td>
<td>No crashed pods</td>
</tr>
</tbody></table>
<p>If auth or ledger start restarting after netpol, egress rules are too strict. See <code>troubleshooting.md</code>.</p>
<h4 id="heading-scenario-4-blocked-cross-service-traffic-optional">Scenario 4 – blocked cross-service traffic (optional)</h4>
<p>Skip if the checkpoint passed and you plan to run <code>make check-6</code>. This proves ledger can't call notification directly (no allow rule for that path). Failure to connect is success.</p>
<p><strong>Don't use the old</strong> <code>wget</code> <strong>one-liner</strong>: the ledger image has no <code>wget</code>/<code>curl</code>, and <code>head -1</code> can pick a Terminating pod (exec hangs or exit <strong>137</strong>).</p>
<pre><code class="language-bash">LEDGER_POD=$(kubectl get pods -n clearledger -l app=ledger-service --no-headers \
  | awk '$2=="2/2" &amp;&amp; $3=="Running" {print $1; exit}')

echo "Using pod: $LEDGER_POD"

kubectl exec -n clearledger "$LEDGER_POD" -c ledger-service -- python3 -c "
import urllib.request
try:
    urllib.request.urlopen('http://notification-service/', timeout=5)
    print('UNEXPECTED: connection succeeded')
except Exception as e:
    print('BLOCKED (expected):', e)
"
</code></pre>
<p><strong>Expected:</strong></p>
<pre><code class="language-text">BLOCKED (expected): &lt;urlopen error timed out&gt;
</code></pre>
<p>or <code>Connection refused</code>, <strong>not</strong> <code>UNEXPECTED: connection succeeded</code>.</p>
<h3 id="heading-66-health-check">6.6: Health Check</h3>
<p>Run this <strong>after §6.4</strong> (network policies). It confirms Falco, custom rules, and netpol are installed. It does <strong>not</strong> prove an alert fired (that is §6.2).</p>
<pre><code class="language-bash">make check-6
</code></pre>
<p><strong>What you should see:</strong></p>
<pre><code class="language-text">▶ Stage 6 — Runtime Security (Falco)
  ✓ Falco DaemonSet: 1/1 nodes
  ✓ ClearLedger custom Falco rules ConfigMap exists
  ✓ NetworkPolicy default-deny-all exists
  ✓ NetworkPolicy allow-auth-service exists
  ✓ NetworkPolicy allow-ledger-service exists
  ✓ NetworkPolicy allow-notification-service exists
  ✓ auth-service reachable after network policies
  ✓ notification-service reachable after network policies

All checks passed. Ready for the next stage.
</code></pre>
<h3 id="heading-how-stage-6-fits-the-full-stack-optional-reading">How Stage 6 fits the full stack (optional reading)</h3>
<p>Each stage guards a different point in the lifecycle. Stages 1 through 5 work before or during pod startup. Stage 6 watches what happens inside a container that is already running.</p>
<ul>
<li><strong>In Stage 3,</strong> CI catches bad code and images on <code>git push</code>.</li>
<li><strong>Stage 4,</strong> Kyverno blocks bad pods at admission.</li>
<li><strong>Stage 5,</strong> Vault injects secrets at startup.</li>
<li><strong>Stage 6,</strong> Falco watches syscalls after the pod is running (shell spawns, sensitive file reads).</li>
<li><strong>Stage 6,</strong> Network policies filter pod-to-pod traffic.</li>
</ul>
<p>They answer three different questions: Kyverno asks whether this pod may be created. Falco asks what the pod is doing right now. Network policies ask who the pod may talk to.</p>
<p>Falco doesn't replace CI or Kyverno. If you skip Stages 3–5, Falco can still alert, but you already shipped vulnerable code and secrets in Git.</p>
<h3 id="heading-stage-6-is-complete-proceed-to-stage-65-or-7">Stage 6 is Complete. Proceed to Stage 6.5 or 7.</h3>
<table>
<thead>
<tr>
<th>#</th>
<th>Check</th>
<th>How to verify</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Falco running</td>
<td><code>kubectl get pods -n falco</code> — DaemonSet <code>2/2</code></td>
</tr>
<tr>
<td>2</td>
<td>Custom rules loaded</td>
<td>`kubectl logs -n falco -l <a href="http://app.kubernetes.io/name=falco">app.kubernetes.io/name=falco</a> -c falco --tail=200</td>
</tr>
<tr>
<td>3</td>
<td>Shell alert fired <strong>and you read it</strong></td>
<td><code>make demo-6</code> → Critical row with <code>cmd=sh -c id &amp;&amp; exit</code>, pod <code>auth-service-…</code> — §6.2</td>
</tr>
<tr>
<td>4</td>
<td>Network policies applied</td>
<td><code>kubectl get networkpolicy -n clearledger</code> — §6.4</td>
</tr>
<tr>
<td>5</td>
<td>App still healthy</td>
<td><code>curl</code> auth + notification health return 200</td>
</tr>
<tr>
<td>6</td>
<td>Health check</td>
<td><code>make check-6</code> green — §6.6</td>
</tr>
</tbody></table>
<p><strong>Portfolio screenshots (optional):</strong> shell-in-container alert · sensitive-file read alert in Falco UI.</p>
<p>What comes next: Stage 6 gives you Falco alerts and basic network policies. You can refine the network policies later. Stage 6.5 is optional chaos testing with Litmus, and Stage 7 adds Grafana dashboards so you can see security events over time.</p>
<h3 id="heading-what-you-learned-in-stage-6">What You Learned in Stage 6</h3>
<ul>
<li>What runtime security catches that CI and admission control can't: threats inside running containers</li>
<li>What Falco is: eBPF syscall monitoring with custom YAML rules</li>
<li>What network policies are: Kubernetes firewall rules between pods</li>
<li>How to trigger and interpret alerts: incident response skills</li>
<li><strong>The full stack:</strong> code scanning, admission control, secrets management, runtime detection which leads to (next) observability</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Deployed Falco for runtime threat detection with custom rules, and can trigger and read an alert for a shell-in-container or sensitive-file read the way an on-call engineer would.</p>
</blockquote>
<p><code>make snapshot STAGE=6 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage6</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<h2 id="heading-stage-65-chaos-engineering-optional">Stage 6.5 — Chaos Engineering (Optional)</h2>
<p><strong>Most learners skip this.</strong> If Stage 6 is done and <code>make check-6</code> passes, jump straight to <a href="#heading-stage-7-security-observability">Stage 7</a>. Nothing in Stages 7–8 requires Litmus.</p>
<p><strong>If you want chaos/resilience (~1 hour):</strong> LitmusChaos deletes one <code>auth-service</code> pod and proves <code>/auth/health</code> stays <strong>200</strong> while Kubernetes replaces it.</p>
<h3 id="heading-do-the-steps-in-this-order">Do the Steps in This Order</h3>
<table>
<thead>
<tr>
<th>Step</th>
<th>Section</th>
<th>What you do</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td><a href="#heading-650-before-you-start-auth-pods-must-be-22">§6.5.0</a></td>
<td><code>make fix-65-prereqs</code> — auth pods <strong>2/2 Ready</strong></td>
</tr>
<tr>
<td>2</td>
<td><a href="#heading-651-install-litmuschaos-operator-ui-cluster-connection">§6.5.1</a></td>
<td><code>bash ...install-litmus.sh</code> — UI shows <strong>Active 1</strong></td>
</tr>
<tr>
<td>3</td>
<td><a href="#heading-652-run-your-first-experiment-pod-delete">§6.5.2</a></td>
<td>UI: Pod-delete experiment + <code>curl</code> stays 200</td>
</tr>
<tr>
<td>4</td>
<td><a href="#heading-657-health-check">§6.5.7</a></td>
<td><code>make check-65</code>, snapshot</td>
</tr>
</tbody></table>
<p><strong>Optional:</strong> <a href="#heading-653-same-experiment-from-the-terminal-make-demo-65-optional">§6.5.3</a>: same test via <code>make demo-65</code> (terminal path) instead of the UI wizard.</p>
<h3 id="heading-650-before-you-start-auth-pods-must-be-22">6.5.0: Before You Start (Auth Pods Must be 2/2)</h3>
<p>Chaos deletes pods. If replacements fail to start, you debug CrashLoopBackOff instead of learning resilience.</p>
<pre><code class="language-bash">export GITHUB_OWNER=YOUR_GITHUB_USERNAME   # required — without this, fix-argocd breaks ArgoCD repoURL
make fix-65-prereqs
kubectl get pods -n clearledger -l app=auth-service
</code></pre>
<p><strong>Pass:</strong> two pods, both <strong>2/2 Ready</strong>. Don't install Litmus until this is true.</p>
<p><strong>If something fails:</strong></p>
<table>
<thead>
<tr>
<th>Symptom</th>
<th>Fix</th>
</tr>
</thead>
<tbody><tr>
<td>ArgoCD <strong>ComparisonError</strong> after <code>fix-65-prereqs</code></td>
<td><code>kubectl apply -f stages/stage-2-gitops/argocd/clearledger-app.yaml</code></td>
</tr>
<tr>
<td>Auth <strong>Init:0/1</strong>, Vault <code>permission denied</code></td>
<td>Re-run Stage 5 <code>setup.sh</code> + <code>seed-vault-secrets.sh</code>, delete auth/ledger pods</td>
</tr>
<tr>
<td>Auth <strong>1/2</strong> or postgres timeout</td>
<td><code>make fix-65-prereqs</code> again (adds netpol + startup probes)</td>
</tr>
</tbody></table>
<h3 id="heading-651-install-litmuschaos-operator-ui-cluster-connection">6.5.1: Install LitmusChaos (Operator, UI, Cluster Connection)</h3>
<pre><code class="language-bash">bash stages/stage-6.5-chaos-engineering/scripts/install-litmus.sh
kubectl get pods -n litmus
open http://litmus.local    # login: admin / litmus
</code></pre>
<p><strong>Pass before §6.5.2:</strong> Overview shows Infrastructures: Active 1 (not 0, not Pending).</p>
<p><strong>Verify pods:</strong></p>
<pre><code class="language-bash">kubectl get pods -n litmus
# litmus-core, chaos frontend/server, mongodb, subscriber — all Running
</code></pre>
<h4 id="heading-if-overview-shows-0-infrastructures-or-pending">If Overview shows 0 infrastructures or PENDING</h4>
<p>The UI is empty until a subscriber agent connects your cluster:</p>
<pre><code class="language-bash">export LITMUS_PASSWORD='litmus'   # only if you changed the default
bash stages/stage-6.5-chaos-engineering/scripts/connect-litmus-infra.sh
</code></pre>
<p>Hard-refresh the browser. Start at <a href="http://litmus.local"><strong>http://litmus.local</strong></a> only, not old <code>/account/.../settings</code> bookmarks.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/69212567-ae17-4f8f-b60d-7c3aac1592b8.png" alt="screenshot showing litmus ui" style="display: block;" width="600" height="400" loading="lazy">

<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/66cfe3f9-99b5-4c1b-bfed-aa90fbcca2e8.png" alt="screenshot showing litmus ui" style="display: block;" width="600" height="400" loading="lazy">

<h4 id="heading-ui-navigation-click-order-for-652">UI navigation (click order for §6.5.2)</h4>
<ol>
<li><strong>Overview</strong>: Confirm <strong>Active 1</strong></li>
<li><strong>ChaosHubs</strong>, <strong>Pod Delete</strong>, <strong>Launch Experiment</strong></li>
<li><strong>Chaos Experiments</strong>: watch <strong>Running to Completed</strong></li>
</ol>
<p>Left nav: <strong>Overview</strong>, <strong>Environments</strong>, <strong>ChaosHub</strong>, <strong>Chaos Experiments</strong>. Skip <strong>Resilience Probes</strong> and deep <strong>Settings</strong> URLs for this lab.</p>
<h3 id="heading-652-run-your-first-experiment-pod-delete">6.5.2: Run Your First Experiment (Pod Delete)</h3>
<p><strong>Goal:</strong> Kill one <code>auth-service</code> pod and prove <code>/auth/health</code> stays <strong>200</strong>.</p>
<p><strong>Before you click Run in the UI</strong>, open two terminals:</p>
<pre><code class="language-bash"># Terminal A — watch pods
kubectl get pods -n clearledger -l app=auth-service -w

# Terminal B — watch health every 5 seconds
while true; do
  date +%H:%M:%S
  curl -s -o /dev/null -w "health=%{http_code}\n" http://clearledger.local/auth/health
  sleep 5
done
</code></pre>
<p><strong>In the UI (</strong><code>http://litmus.local</code><strong>):</strong> Left nav → ChaosHubs → Pod Delete card → Launch Experiment.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ca183e75-6680-4743-9ffe-03a144e09e13.png" alt="screenshot showing litmus ui" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>Litmus UI note:</strong> ChaosCenter labels change between versions (for example, “Tune fault”, “Target selection”, “Chaos Experiment”). Match fields by <strong>concept</strong>, not exact button text. Accept wizard defaults unless the table below lists a value.</p>
<p>In the Litmus UI, open:</p>
<p><code>ChaosHubs</code> → <code>Pod Delete</code> → <code>Launch Experiment</code></p>
<p>This opens the experiment wizard. Use these values when the wizard asks for them:</p>
<ul>
<li>Infrastructure: <code>clearledger-cluster</code> and it must be <code>Active</code></li>
<li>Namespace: <code>clearledger</code></li>
<li>Target label: <code>app=auth-service</code></li>
<li>Target kind: <code>Deployment</code></li>
<li>Pods affected: <code>50%</code></li>
<li>Duration: <code>30</code> seconds</li>
<li>Fault/experiment name: <code>pod-delete</code></li>
</ul>
<p>Finish the wizard with Save or Create, then click Run. Don't choose <strong>Schedule</strong>.</p>
<p><strong>What success looks like:</strong></p>
<table>
<thead>
<tr>
<th>Where</th>
<th>Good sign</th>
</tr>
</thead>
<tbody><tr>
<td>Terminal A</td>
<td>One pod <strong>Terminating</strong>, then back to <strong>2/2 Ready</strong></td>
</tr>
<tr>
<td>Terminal B</td>
<td><code>health=200</code> even while one pod is down</td>
</tr>
<tr>
<td>Litmus UI</td>
<td>Experiment <strong>Running → Completed</strong></td>
</tr>
</tbody></table>
<p><strong>Prefer terminal over UI?</strong> Skip the wizard and run <a href="#heading-653-same-experiment-from-the-terminal-make-demo-65-optional">§6.5.3</a> (<code>make demo-65</code>) instead.</p>
<h3 id="heading-653-same-experiment-from-the-terminal-make-demo-65-optional">6.5.3 — Same experiment from the terminal (<code>make demo-65</code>) — optional</h3>
<p>Use this if you want to run the pod-delete test without clicking through the Litmus UI.</p>
<p>Make sure auth pods are healthy first:</p>
<pre><code class="language-bash">make fix-65-prereqs
</code></pre>
<p>Then run the demo:</p>
<pre><code class="language-bash">make demo-65
</code></pre>
<p>The script applies the <code>auth-service-pod-delete</code> ChaosEngine in the <code>litmus</code> namespace. Litmus deletes one <code>auth-service</code> pod, Kubernetes replaces it, and the script checks that <code>/auth/health</code> keeps returning <code>200</code>.</p>
<p>After it finishes, verify the result:</p>
<pre><code class="language-bash">kubectl get chaosresult -n litmus
kubectl get pods -n clearledger -l app=auth-service
</code></pre>
<p>You passed if the script ends with <code>PASS</code>, the <code>ChaosResult</code> is <code>Completed / Pass</code>, and two <code>auth-service</code> pods are running again.</p>
<p>You can also see the run in the Litmus UI: <strong>Chaos Experiments</strong> → refresh → open the latest run.</p>
<p>If new auth pods get stuck in <code>Init:0/1</code>, re-apply the Stage 6 network policies:</p>
<pre><code class="language-bash">kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml
</code></pre>
<h3 id="heading-653a-real-output-examples-verified-on-the-lab-cluster">6.5.3a: Real Output Examples (Verified on the Lab Cluster)</h3>
<p>These samples were captured from a working cluster after <code>make fix-65-prereqs</code>, <code>make connect-litmus</code>, and <code>make demo-65</code>.</p>
<h4 id="heading-make-check-65"><code>make check-65</code></h4>
<pre><code class="language-text">▶ Stage 6.5 — Chaos Engineering (LitmusChaos)
  ✓ litmus namespace exists
  ✓ litmus-admin ServiceAccount exists in litmus
  ✓ pod-delete ChaosExperiment installed in litmus
  ✓ Litmus chaos operator is running
  ✓ Litmus ChaosCenter reachable at http://litmus.local
  ✓ Litmus subscriber running (UI connected to cluster)
  ✓ auth-service healthy (baseline before chaos)
  ✓ auth-service has 2/2 Ready replicas (stable for chaos)
  ✓ allow-postgres NetworkPolicy exists (Stage 6 fix)

All checks passed. Ready for the next stage.
</code></pre>
<h4 id="heading-make-demo-65-captured-from-a-real-run-2026-06-01"><code>make demo-65</code> captured from a real run (2026-06-01)</h4>
<pre><code class="language-text">Stage 6.5 — auth-service pod-delete

Preflight: 2 auth-service pods Running

Applying ChaosEngine auth-service-pod-delete (namespace litmus)

Watching http://clearledger.local/auth/health

  10s  health=200  pods=2
  20s  health=200  pods=1
  30s  health=200  pods=1
  40s  health=200  pods=2
  50s  health=200  pods=2
  60s  health=200  pods=2

Result:
  ChaosResult: Completed / Pass
  Recovery:    2 auth-service pod(s) Running
  Health:      6/6 checks returned 200

PASS
</code></pre>
<p>If health lines show <code>000</code>, run <code>bash scripts/setup-hosts.sh</code> on your Mac and re-run. The script also tries <code>multipass exec clearledger -- curl</code> when the VM is present.</p>
<h4 id="heading-terminal-b-health-loop-expected-output">Terminal B (health loop, expected output)</h4>
<pre><code class="language-text">22:05:01
health=200
22:05:06
health=200
22:05:11
health=200
</code></pre>
<p>Pod count may show <strong>1</strong> while the replacement pod is starting, which is expected.</p>
<h4 id="heading-terminal-a-during-chaos-kubectl-get-pods-w">Terminal A during chaos (<code>kubectl get pods -w</code>)</h4>
<pre><code class="language-text">NAME                            READY   STATUS        RESTARTS   AGE
auth-service-84cc988c4d-hdb45   2/2     Running       0          67m
auth-service-84cc988c4d-b59sj   2/2     Terminating   0          15m    ← killed
auth-service-84cc988c4d-dxz9q   0/2     Pending       0          0s     ← replacement
auth-service-84cc988c4d-dxz9q   0/2     Init:0/1      0          2s
auth-service-84cc988c4d-dxz9q   2/2     Running       0          90s
</code></pre>
<h4 id="heading-after-demo-verify">After demo: verify</h4>
<pre><code class="language-bash">kubectl get chaosresult -n litmus
# auth-service-pod-delete-pod-delete   Completed   Pass

kubectl get pods -n clearledger -l app=auth-service
# auth-service-84cc988c4d-xxxxx   2/2   Running
# auth-service-84cc988c4d-yyyyy   2/2   Running

kubectl get cm subscriber-config -n litmus -o jsonpath='{.data.IS_INFRA_CONFIRMED}'
# true
</code></pre>
<h4 id="heading-subscriber-connected-infrastructure-active-in-ui">Subscriber connected (infrastructure Active in UI)</h4>
<pre><code class="language-text">kubectl logs -n litmus -l app.kubernetes.io/name=subscriber --tail=3
level=info msg="AgentID: a63c2a2c-... has been confirmed"
level=info msg="Server connection established, Listening...."
</code></pre>
<h3 id="heading-654-understand-the-yaml-files-read-before-running">6.5.4: Understand the YAML Files (Read Before Running)</h3>
<p>Each file is a <code>ChaosEngine</code>: a request to Litmus: “run experiment X against app Y for Z seconds.”</p>
<h4 id="heading-litmus-installyaml"><code>litmus-install.yaml</code></h4>
<p>This creates the <code>litmus</code> namespace only. Platform workloads live here, separate from <code>clearledger</code> app pods.</p>
<h4 id="heading-litmus-rbacyaml"><code>litmus-rbac.yaml</code></h4>
<table>
<thead>
<tr>
<th>Resource</th>
<th>What it does</th>
</tr>
</thead>
<tbody><tr>
<td><code>ServiceAccount litmus-admin</code> (namespace <code>litmus</code>)</td>
<td>Identity for Litmus runner pods</td>
</tr>
<tr>
<td><code>ClusterRoleBinding → cluster-admin</code></td>
<td>Allows deleting pods / injecting faults in <code>clearledger</code> (lab simplification. Production would use least-privilege)</td>
</tr>
</tbody></table>
<h4 id="heading-auth-service-pod-deleteyaml-experiment-1-used-by-demo"><code>auth-service-pod-delete.yaml</code> (Experiment 1: used by demo)</h4>
<pre><code class="language-yaml">metadata:
  namespace: litmus          # engine lives here (Kyverno-safe)
spec:
  appinfo:
    appns: clearledger       # target app namespace
    applabel: app=auth-service
    appkind: deployment
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: PODS_AFFECTED_PERC
              value: "50"    # 50% of 2 replicas = 1 pod killed
            - name: TOTAL_CHAOS_DURATION
              value: "30"    # chaos window in seconds
</code></pre>
<p>What happens when applied:</p>
<ol>
<li>Operator reads <code>ChaosEngine</code> and creates <code>auth-service-pod-delete-runner</code> pod in <code>litmus</code></li>
<li>Runner selects one <code>auth-service</code> pod in <code>clearledger</code> and sends SIGTERM / delete</li>
<li>Kubernetes Deployment controller sees 1/2 replicas and schedules a replacement pod</li>
<li>Service routes traffic to the <strong>surviving</strong> replica during recovery</li>
<li><code>ChaosResult</code> CR records pass/fail from Litmus’s perspective</li>
</ol>
<h4 id="heading-ledger-service-network-latencyyaml-experiment-2-manual"><code>ledger-service-network-latency.yaml</code> (Experiment 2 — manual)</h4>
<p>Adds <strong>2000 ms</strong> network latency to <code>ledger-service</code> pods for 60 seconds. Proves timeouts return <strong>503</strong> instead of hanging the UI.</p>
<h4 id="heading-notification-service-memory-hogyaml-experiment-3-manual"><code>notification-service-memory-hog.yaml</code> (Experiment 3 — manual)</h4>
<p>Fills <strong>80%</strong> of pod memory limit for 60 seconds. Proves OOMKill + restart behavior.</p>
<p><strong>Never apply all three at once.</strong> Run one experiment, verify recovery, then the next.</p>
<h3 id="heading-655-after-the-demo-what-to-look-for-do-not-skip">6.5.5: After the Demo, What to Look For (Do Not Skip)</h3>
<p><strong>1. During chaos: availability</strong></p>
<table>
<thead>
<tr>
<th>Signal</th>
<th>Good</th>
<th>Bad</th>
</tr>
</thead>
<tbody><tr>
<td><code>curl http://clearledger.local/auth/health</code></td>
<td><strong>200</strong> while one pod is down</td>
<td>502/503/timeout</td>
</tr>
<tr>
<td><code>kubectl get pods -l app=auth-service</code></td>
<td>1 Running + 1 Init/Pending (replacement starting)</td>
<td>0 Running</td>
</tr>
</tbody></table>
<p><strong>2. After chaos: recovery</strong></p>
<table>
<thead>
<tr>
<th>Signal</th>
<th>Good</th>
<th>Bad</th>
</tr>
</thead>
<tbody><tr>
<td>Pod count</td>
<td>2/2 <strong>Ready</strong> (may take 1–2 min — Vault agent init)</td>
<td>Stuck at 1 replica</td>
</tr>
<tr>
<td>Events</td>
<td><code>Killing</code> then <code>Scheduled</code> / <code>Started</code> on new pod</td>
<td>Repeated CrashLoopBackOff</td>
</tr>
<tr>
<td>ArgoCD</td>
<td>Synced</td>
<td>—</td>
</tr>
</tbody></table>
<p><strong>3. Litmus</strong> <code>ChaosResult</code> <strong>verdict</strong></p>
<pre><code class="language-bash">kubectl get chaosresult -n litmus
</code></pre>
<p><strong>Your pass criteria:</strong></p>
<ul>
<li><code>/auth/health</code> returned <strong>200</strong> at least once during the chaos window</li>
<li>A pod was <strong>Killed</strong> (see events)</li>
<li>Deployment returned to <strong>2 replicas</strong></li>
</ul>
<h3 id="heading-656-manual-experiments-after-experiment-1-succeeds">6.5.6 Manual Experiments (After Experiment 1 Succeeds)</h3>
<p>Wait until both auth-service pods show <strong>2/2 Ready</strong>, then run <strong>one</strong> experiment at a time:</p>
<pre><code class="language-bash"># Experiment 2 — 2s network latency on ledger-service (60s)
kubectl delete chaosengine ledger-service-network-latency -n litmus --ignore-not-found
kubectl apply -f stages/stage-6.5-chaos-engineering/infra/chaos/ledger-service-network-latency.yaml

# Experiment 3 — memory pressure on notification-service (60s)
kubectl delete chaosengine notification-service-memory-hog -n litmus --ignore-not-found
kubectl apply -f stages/stage-6.5-chaos-engineering/infra/chaos/notification-service-memory-hog.yaml
</code></pre>
<table>
<thead>
<tr>
<th>Experiment</th>
<th>File</th>
<th>What to verify</th>
</tr>
</thead>
<tbody><tr>
<td>Pod delete</td>
<td><code>auth-service-pod-delete.yaml</code></td>
<td>Health 200 during kill, 2 replicas after</td>
</tr>
<tr>
<td>Network latency</td>
<td><code>ledger-service-network-latency.yaml</code></td>
<td>API returns 503/timeout, not infinite hang</td>
</tr>
<tr>
<td>Memory hog</td>
<td><code>notification-service-memory-hog.yaml</code></td>
<td>Pod OOMKills and restarts, Redis subscription recovers</td>
</tr>
</tbody></table>
<p>Clean up an experiment:</p>
<pre><code class="language-bash">kubectl delete chaosengine auth-service-pod-delete -n litmus
</code></pre>
<h3 id="heading-657-health-check">6.5.7: Health Check</h3>
<pre><code class="language-bash">make check-65
</code></pre>
<p><strong>Expected:</strong> see full sample in <a href="#heading-653a-real-output-examples-verified-on-the-lab-cluster">§6.5.3a</a> (<code>make check-65</code> block). Minimum:</p>
<pre><code class="language-text">▶ Stage 6.5 Chaos Engineering (LitmusChaos)
  ✓ Litmus subscriber running (UI connected to cluster)
  ✓ auth-service has 2/2 Ready replicas (stable for chaos)
  ...
All checks passed. Ready for the next stage.
</code></pre>
<h3 id="heading-stage-65-complete-checklist">Stage 6.5 Complete: Checklist</h3>
<table>
<thead>
<tr>
<th>#</th>
<th>Check</th>
<th>How to verify</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Litmus operator running</td>
<td><code>kubectl get pods -n litmus</code> — <code>litmus-*</code> Running</td>
</tr>
<tr>
<td>2</td>
<td>Experiments installed</td>
<td><code>kubectl get chaosexperiment pod-delete -n litmus</code></td>
</tr>
<tr>
<td>3</td>
<td>Pod-delete demo run</td>
<td><code>make demo-65</code> health 200 during chaos</td>
</tr>
<tr>
<td>4</td>
<td>Recovery observed</td>
<td>2 auth-service replicas Ready. Killing/Scheduled events</td>
</tr>
<tr>
<td>5</td>
<td>Evidence saved</td>
<td>Terminal output from <code>run-chaos.sh</code> (DORA artifact)</td>
</tr>
<tr>
<td>6</td>
<td>Health check</td>
<td><code>make check-65</code> green</td>
</tr>
<tr>
<td>7</td>
<td>UI infrastructure connected</td>
<td>Overview → <strong>Active: 1</strong> (§6.5.2)</td>
</tr>
</tbody></table>
<h3 id="heading-what-you-learned-in-stage-65">What You Learned in Stage 6.5</h3>
<ul>
<li><strong>Detection ≠ resilience</strong>: Falco alerts don't prove HA</li>
<li><strong>Replicas + Services + probes</strong>: why <code>replicas: 2</code> isn't cosmetic</li>
<li><strong>ChaosEngine YAML</strong>: declarative failure injection as code</li>
<li><strong>Platform vs app namespaces</strong>: Kyverno blocks chaos runners in <code>clearledger</code>, engines run in <code>litmus</code></li>
<li><strong>MTTR</strong>: time from pod kill to 2/2 Ready again (Stage 7 graphs this)</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Ran chaos experiments with LitmusChaos (pod-delete, network latency, memory pressure) to prove the system recovers, and can distinguish detection from resilience.</p>
</blockquote>
<p><code>make snapshot STAGE=65 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage65</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<h2 id="heading-stage-7-security-observability">Stage 7 — Security Observability</h2>
<p>Security you can't measure, you can't prove.</p>
<p>The goal here is to understand how metrics, logs, and dashboards fit together. Then prove it by running commands in the terminal, watching the same events appear in Grafana, and explaining what each panel means.</p>
<p>This stage is not “install Grafana and move on.” Stage 7 isn't complete until your dashboards show real Kyverno violations and Falco alerts that you triggered in §7.4: plus portfolio screenshots (§7.6). <code>make check-7</code> only proves the stack is up; it does not prove you can detect security events.</p>
<p><strong>Before you start:</strong> <code>make check-6</code> should pass (Stage 6.5 is optional. Skip is fine). Check the VM is not overloaded: <code>multipass exec clearledger -- uptime</code>. If you ran Stage 6.5, do <a href="#heading-70-free-node-resources-scale-down-litmus">§7.0</a> first to scale Litmus down. Plan about half a day. This is the heaviest stage on a single-node VM.</p>
<p>You'll be done when §7.6 is complete: dashboards show your Kyverno denial and Falco alert, not empty panels. Then <code>make check-7</code> (§7.7), <code>make snapshot STAGE=7</code>, and <code>make snapshots</code> (confirm <code>clearledger.stage7</code>).</p>
<p><strong>Already installed?</strong> If <code>kubectl get pods -n monitoring</code> shows Grafana <strong>3/3</strong> and Loki <strong>1/1</strong>, skip §7.1. Start at §7.2 (verify the stack), then §7.4 (hands-on lab).</p>
<h3 id="heading-what-you-need-to-know-first">What You Need to Know First</h3>
<p>Up to now, each stage had its own window into the cluster. Stage 3 gave you CI scan results in GitHub Actions. Stage 4 showed Kyverno blocking a bad deploy in the terminal. Stage 6 gave you Falco alerts in its UI, and you could always run <code>kubectl logs</code> on a pod. Those views are useful, but they are scattered.</p>
<p>Stage 7 brings them together in one place: <strong>Grafana</strong>. Instead of jumping between five different tools, you open a dashboard and see whether security events, policy violations, and app health are happening over time.</p>
<h4 id="heading-the-three-tools-youre-installing">The three tools you're installing</h4>
<p><strong>Prometheus</strong> collects numbers from the cluster: things like “how many Kyverno denials in the last hour” or “how many HTTP requests per second.” It checks those numbers every 15–30 seconds and keeps a history you can graph.</p>
<p><strong>Loki</strong> collects log lines: the same kind of text you see from <code>kubectl logs</code>, but from many pods at once. Falco alerts, failed login attempts, and application errors all land here so you can search them later.</p>
<p><strong>Grafana</strong> is the web UI where charts and tables pull data from Prometheus and Loki. This is what you would show an auditor: not a one-off terminal screenshot, but proof that you can find and measure events after they happen.</p>
<p>Prometheus doesn't magically know what to collect. ServiceMonitors and PodMonitors are small config objects that point it at the right targets.<br>If Kyverno has no monitor, the Kyverno dashboard stays empty even when Kyverno is working fine. The same applies to application request rates. Those panels stay blank until §7.5, when metrics-enabled images are deployed through GitOps.</p>
<p>Logs follow a similar path. <strong>Promtail</strong> reads container logs and sends them to Loki. If Loki isn't running, Grafana log panels show “No data” even though <code>kubectl logs</code> still works on individual pods.</p>
<h4 id="heading-how-this-connects-to-what-you-already-built">How this connects to what you already built</h4>
<p>When you blocked a bad <code>kubectl apply</code> in Stage 4, Kyverno recorded that denial. In Stage 7, that shows up on the <strong>Kyverno Policy Violations</strong> dashboard (via Prometheus).</p>
<p>When you triggered a shell inside a pod in Stage 6, Falco wrote an alert. In Stage 7, that appears on the <strong>Security Event Timeline</strong> (via Loki).</p>
<p>When ClearLedger handles HTTP traffic or a failed login, those events feed the <strong>Service Health</strong> dashboards (Loki and Prometheus together).</p>
<p>Vault (Stage 5) and network policies (Stage 6) don't always have their own panel, but they still matter: fewer secrets in Git and blocked pod traffic show up indirectly in a healthier, quieter cluster.</p>
<h4 id="heading-what-youll-do-in-this-stage">What you'll do in this stage</h4>
<p>You'll run a command in the terminal (for example, a Kyverno violation or a Falco trigger) and then wait a short time while Prometheus or Loki ingests the event. Within about 15–90 seconds, the matching Grafana panel should update.</p>
<p>That's the whole point of observability for security: the terminal proves the event happened once, while the dashboard proves you can <strong>detect and measure</strong> it later without being logged into the cluster at that exact moment.</p>
<h3 id="heading-70-free-node-resources-scale-down-litmus">7.0: Free Node Resources (Scale Down Litmus)</h3>
<p>Stage 6.5 is complete. You don't need the Litmus UI, MongoDB, or chaos operator running while Prometheus, Loki, and Grafana start. They compete for the same CPUs on a single-node lab VM (6 by default, see <code>scripts/setup-cluster.sh</code>).</p>
<p>Scaling Litmus to zero frees ~500–800MB RAM and reduces CPU churn before the observability install.</p>
<pre><code class="language-bash">kubectl scale deployment,statefulset -n litmus --replicas=0 --all
kubectl get pods -n litmus
# Expected: no Running pods (Succeeded job pods from chaos experiments are OK)
multipass exec clearledger -- uptime
# Expected: load average (1m) ideally below ~8 before continuing
</code></pre>
<p>You can scale Litmus back up later if you want to re-run chaos experiments (<code>bash stages/stage-6.5-chaos-engineering/scripts/install-litmus.sh</code>). For Stages 7–7.5, keep it scaled down.</p>
<h3 id="heading-71-install-the-observability-stack">7.1: Install the Observability Stack</h3>
<p><strong>This is safe to run more than once.</strong> The script checks what's already installed. If Grafana, Prometheus, and Loki are healthy, it skips the heavy install and only updates dashboards and scrape configs. Running it again after a partial failure won't duplicate or break a working stack.</p>
<p>Only add <code>FORCE=1</code> if something is genuinely stuck, for example you edited the Helm values files and need a full reinstall, or Loki keeps crashing in a restart loop:</p>
<pre><code class="language-bash">FORCE=1 bash stages/stage-7-observability/scripts/install-observability.sh
</code></pre>
<p>On a first-time install, use the plain command in Step 1 below. Don't use <code>FORCE=1</code> unless the troubleshooting section tells you to.</p>
<p><strong>macOS, Linux, and WSL2:</strong> <code>FORCE=1 bash ...</code> works as written.</p>
<p><strong>Native Windows PowerShell</strong> doesn't use that syntax.</p>
<p>Run the lab inside <strong>WSL2 Ubuntu</strong> (recommended), or set the variable first: <code>$env:FORCE=1; bash stages/stage-7-observability/scripts/install-observability.sh</code>.</p>
<p><strong>Step 1: install</strong> (wait until the script prints <code>✓ Stage 7 installed.</code>):</p>
<pre><code class="language-bash">bash stages/stage-7-observability/scripts/install-observability.sh
</code></pre>
<h4 id="heading-if-you-see-waiting-for-falco-during-the-stage-7-install-thats-expected">If you see “Waiting for Falco” during the Stage 7 install, that's expected.</h4>
<p>You already installed Falco in Stage 6. Stage 7 is not adding a second Falco. It is making sure the existing Falco setup can feed logs and metrics into the observability stack.</p>
<p>The flow is:</p>
<ul>
<li>Falco still runs in the <code>falco</code> namespace.</li>
<li>Promtail sends Falco logs to Loki.</li>
<li>Grafana reads those logs from Loki.</li>
<li>The Security Event Timeline dashboard shows the Falco alerts.</li>
</ul>
<p>Right after install, the Grafana panels may be empty. That's normal. You need to trigger a new alert in §7.4 before the dashboard has something fresh to show.</p>
<p><strong>Step 2. Check pods</strong> (run this after Step 1 finishes):</p>
<pre><code class="language-bash">kubectl get pods -n monitoring
</code></pre>
<p>You want something like this (pod name suffixes vary):</p>
<pre><code class="language-text">NAME                                              READY   STATUS    RESTARTS   AGE
kube-prometheus-stack-grafana-....                3/3     Running   0          5m
kube-prometheus-stack-prometheus-....             2/2     Running   0          5m
loki-0                                            1/1     Running   0          5m
loki-promtail-....                                1/1     Running   0          5m
</code></pre>
<p>Grafana must show <strong>3/3</strong> Ready (not 2/3). Loki must show <strong>1/1</strong>. If pods are still <code>Pending</code> or <code>ContainerCreating</code>, wait a few minutes and run <code>kubectl get pods -n monitoring</code> again.</p>
<p><strong>Expected – Loki healthy:</strong></p>
<pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- http://127.0.0.1:3100/ready
</code></pre>
<pre><code class="language-text">ready
</code></pre>
<p><strong>Expected – Grafana can reach Loki (same path log panels use):</strong></p>
<pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- --timeout=5 http://loki:3100/ready
</code></pre>
<pre><code class="language-text">ready
</code></pre>
<p><strong>Expected – Grafana UI reachable:</strong></p>
<pre><code class="language-bash">curl -sI http://grafana.local | head -n 1
</code></pre>
<pre><code class="language-text">HTTP/1.1 302 Found
</code></pre>
<p>Log into <a href="http://grafana.local"><strong>http://grafana.local</strong></a><strong>:</strong> <code>admin</code> / <code>admin123</code></p>
<p>Empty panels right after install are <strong>normal</strong>. You haven't generated events yet. Continue to §7.2–§7.4.</p>
<p>If Helm fails: wait 30s, then <code>FORCE=1 bash stages/stage-7-observability/scripts/install-observability.sh</code>. See <code>troubleshooting.md. Stage 7</code>.</p>
<p><strong>✋ Hands-on checkpoint: confirm Loki and dashboards are ready</strong></p>
<p>Before you open Grafana, confirm the logging stack and dashboards actually installed.</p>
<p>On a single-node VM, Grafana can look fine while Loki is crash-looping or the ClearLedger dashboards never loaded. If you skip this check, you may spend the rest of Stage 7 debugging empty panels.</p>
<p><strong>Run:</strong></p>
<pre><code class="language-bash">kubectl get pods -n monitoring
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki \
  -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}{"\n"}'
kubectl get configmap -n monitoring -l clearledger_dashboard=1 --no-headers | wc -l
</code></pre>
<p><strong>Expected:</strong></p>
<ul>
<li>All monitoring pods are <code>Running</code></li>
<li>Grafana shows <code>3/3</code> Ready</li>
<li>Loki shows <code>1/1</code> Ready</li>
<li>Loki restart count is <code>0</code>, or low and not climbing</li>
<li>The dashboard count is <code>6</code></li>
</ul>
<p>If Loki keeps restarting or the dashboard count is <code>0</code>, stop here and fix the install before continuing. Empty Grafana panels usually mean Loki or the dashboards are missing, not that the security events failed.</p>
<h3 id="heading-72-verify-prometheus-loki-and-grafana-before-opening-dashboards">7.2: Verify Prometheus, Loki, and Grafana (before opening dashboards)</h3>
<p>Run these three checks so you know which layer is broken if a panel is empty.</p>
<h4 id="heading-check-1-prometheus-has-kyverno-metrics">Check 1: Prometheus has Kyverno metrics</h4>
<pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=kyverno_admission_requests_total' 2&gt;/dev/null \
  | head -c 400
</code></pre>
<p>(Prometheus runs as a StatefulSet pod, not a Deployment. This query goes through Grafana to the Prometheus Service.)</p>
<p><strong>Expected:</strong> JSON with <code>"status":"success"</code> and a <code>"metric"</code> block (values may be <code>0</code> until you trigger a violation in §7.4).</p>
<p>If you see <code>"status":"success"</code> but <code>"result":[]</code>, Prometheus is up but Kyverno hasn't recorded admissions yet. That's fine before the lab.</p>
<h4 id="heading-check-2-loki-has-falco-logs">Check 2: Loki has Falco logs</h4>
<pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- \
  'http://127.0.0.1:3100/loki/api/v1/labels' 2&gt;/dev/null | head -c 300
</code></pre>
<p><strong>Expected:</strong> JSON listing labels such as <code>"namespace"</code> (and after Falco events, you'll see <code>"falco"</code> in label values).</p>
<p>Quick log search (may return empty lines until §7.4 Exercise B):</p>
<pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- \
  'http://127.0.0.1:3100/loki/api/v1/query?query=%7Bnamespace%3D%22falco%22%7D&amp;limit=3' 2&gt;/dev/null \
  | head -c 500
</code></pre>
<p><strong>Expected:</strong> <code>"status":"success"</code>. <code>"result":[]</code> means no Falco lines in Loki yet, not a broken Loki.</p>
<h4 id="heading-check-3-grafana-imported-clearledger-dashboards">Check 3: Grafana imported ClearLedger dashboards</h4>
<pre><code class="language-bash">curl -s -u admin:admin123 'http://grafana.local/api/search?tag=clearledger' | jq -r '.[].title'
</code></pre>
<p><strong>Expected: six titles:</strong></p>
<pre><code class="language-text">ClearLedger - Compliance Posture
ClearLedger - DORA Metrics
ClearLedger - Kubernetes Audit Log Analysis
ClearLedger - Kyverno Policy Violations
ClearLedger - Security Event Timeline
ClearLedger - Service Health + Auth Security
</code></pre>
<p>Or in the UI: go. to<strong>Dashboards</strong> then filter tag <code>clearledger</code>. You should see exactly these six (no missing names).</p>
<h3 id="heading-73-your-first-10-minutes-in-grafana">7.3: Your First 10 Minutes in Grafana</h3>
<p>This section is only a tour. You're not proving anything yet.</p>
<p><strong>Rule for all of Stage 7:</strong> an empty panel usually means no events have happened in the selected time range, not that Grafana is broken. You create the real events in §7.4.</p>
<h4 id="heading-step-1-open-grafana">Step 1: Open Grafana</h4>
<p>Go to <code>http://grafana.local</code> and log in:</p>
<ul>
<li>Username: <code>admin</code></li>
<li>Password: <code>admin123</code></li>
</ul>
<h4 id="heading-step-2-set-the-time-range">Step 2: Set the Time Range</h4>
<p>In the top-right corner, choose <strong>Last 15 minutes</strong>.</p>
<p>Keep this setting for all of Stage 7. Wider ranges like <strong>Last 24 hours</strong> can overload Loki on a single-node lab VM.</p>
<h4 id="heading-step-3-open-dashboards-one-at-a-time">Step 3: Open Dashboards One at a Time</h4>
<p>Open one dashboard, look around, then move to the next. Don't open all six at once.</p>
<ol>
<li><a href="http://grafana.local/d/clearledger-kyverno-violations">Kyverno Policy Violations</a>: policy blocks from Stage 4.</li>
<li><a href="http://grafana.local/d/clearledger-security-events">Security Event Timeline</a>: Falco alerts from Stage 6. You may see old <code>postgres</code> noise in the log table.</li>
<li><a href="http://grafana.local/d/clearledger-service-health">Service Health + Auth</a>: app traffic and login attempts.</li>
<li><a href="http://grafana.local/d/clearledger-compliance">Compliance Posture</a>: summary view for auditors. Skim it and come back after §7.4.</li>
<li><a href="http://grafana.local/d/clearledger-audit-logs">Audit Log Analysis</a>: empty on MicroK8s by design (audit pipeline not enabled by default).</li>
<li><a href="http://grafana.local/d/clearledger-dora-metrics">DORA Metrics</a>: deploy-frequency charts. Needs multiple CI runs to accumulate data. May show blank on first look. Optional.</li>
</ol>
<p>Use the short dashboard links in this guide. Avoid old bookmarked URLs with long random slugs.</p>
<p>You can also find them in Grafana: go to <strong>Dashboards</strong> then search tag <code>clearledger</code>.</p>
<h4 id="heading-step-4-how-to-read-what-you-see">Step 4: How to Read What You See</h4>
<p>Grafana panels pull data from two places:</p>
<ul>
<li><strong>Prometheus</strong> shows numbers over time, like Kyverno violation counts and request rates</li>
<li><strong>Loki</strong> shows log lines, like Falco alerts and auth-service messages</li>
</ul>
<p>A big number panel asks: did this count go above zero?</p>
<p>A line chart asks: was there a spike after I ran something?</p>
<p>A logs panel shows the actual text, like rule names, <code>CRITICAL</code>, or <code>Failed login attempt</code>.</p>
<p>If only log panels show <code>connection refused</code>, check Loki again in §7.1.</p>
<p>If number panels work but log panels fail, the problem is likely Loki, not Grafana itself.</p>
<h4 id="heading-step-5-move-on">Step 5: Move On</h4>
<p>Open dashboards 1–3, then continue to §7.4.</p>
<p>That's where you'll run commands in the terminal and watch the panels update with real security events.</p>
<h3 id="heading-74-hands-on-lab-terminal-dashboard-proof">7.4: Hands-on Lab: Terminal → Dashboard Proof</h3>
<p>This is the core learning section. For each exercise: run the command, wait, then confirm in Grafana.</p>
<p><strong>Timing:</strong> wait <strong>30–90 seconds</strong> after each command for Prometheus scrape and Loki ingestion.</p>
<h3 id="heading-two-ways-to-do-this-lab">Two Ways to Do This Lab</h3>
<h4 id="heading-option-1-follow-the-exercises-below-recommended-for-learning">Option 1: follow the exercises below (recommended for learning)</h4>
<p>Run each command yourself, then check Grafana. That's Exercise A, B, and C.</p>
<h4 id="heading-option-2-use-the-guided-script">Option 2: use the guided script</h4>
<p>The script runs the same steps and pauses so you can check Grafana between them:</p>
<pre><code class="language-bash">bash stages/stage-7-observability/scripts/generate-dashboard-data.sh
</code></pre>
<p>Or:</p>
<pre><code class="language-bash">make demo-7
</code></pre>
<p>Both commands do the same thing. The script will say things like “Press Enter after you checked the Kyverno dashboard.” Switch to Grafana, look at the panel, then come back and press Enter.</p>
<p><strong>Want it to run without pauses?</strong> (faster, less hand-holding)</p>
<pre><code class="language-bash">SKIP_PROMPT=1 make demo-7
</code></pre>
<p>Use Option 1 if you want to understand each step. Use Option 2 if you want a walkthrough. Use <code>SKIP_PROMPT=1</code> if you just want the data generated quickly.</p>
<h4 id="heading-exercise-a-kyverno-block-prometheus-kyverno-dashboard">Exercise A: Kyverno block → Prometheus → Kyverno dashboard</h4>
<p><strong>Terminal</strong>: apply a pod that violates Stage 4 policy (runs as root):</p>
<pre><code class="language-bash">cat &lt;&lt;'YAML' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: stage7-kyverno-lab
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
YAML
</code></pre>
<p><strong>How to know it worked:</strong></p>
<p>You're testing whether Kyverno <strong>blocks</strong> a deliberately bad pod. Success means the pod <strong>never gets created</strong>.</p>
<p><strong>Pass. You should see:</strong></p>
<ul>
<li>The terminal prints <code>Error from server</code> and <code>denied the request</code></li>
<li>The exact policy names in the error don't matter. Your output might list one rule or several (<code>disallow-root-containers</code>, <code>require-resource-limits</code>, <code>drop-all-capabilities</code>, …). More lines just means more rules failed, that's still a pass.</li>
<li>The pod name never shows up in the cluster:</li>
</ul>
<pre><code class="language-bash">kubectl get pods -n clearledger | grep stage7-kyverno-lab
</code></pre>
<p><strong>Expected:</strong> no output.</p>
<p><strong>If it fails: stop and fix Stage 4 first</strong></p>
<ul>
<li>The command ends quietly with <code>created</code> (no error)</li>
<li><code>kubectl get pods -n clearledger</code> shows <code>stage7-kyverno-lab</code></li>
</ul>
<p>That means Kyverno let a root pod through. Run <code>make check-4</code> before continuing Stage 7.</p>
<p><strong>Example of a passing terminal</strong> (yours may list more policies):</p>
<pre><code class="language-text">Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc" denied the request:
policy disallow-root-containers/validate-run-as-non-root fail: Running as root is not allowed
</code></pre>
<p><strong>Confirm Prometheus saw it</strong> (optional but useful if Grafana is empty):</p>
<pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=kyverno_admission_requests_total{request_allowed="false"}' 2&gt;/dev/null \
  | grep -o '"value":\[[^]]*\]' | head -3
</code></pre>
<p><strong>Expected:</strong> a <code>"value"</code> entry with a recent Unix timestamp and a number <strong>greater than 0</strong> (for example <code>"value":[..., "1"]</code>). If you see this, Kyverno and Prometheus are working even when Grafana panels say <strong>No data</strong>.</p>
<p><strong>Grafana</strong>: open <a href="http://grafana.local/d/clearledger-kyverno-violations?from=now-15m&amp;to=now">Kyverno Policy Violations</a>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/7f5cf039-f2de-41d6-b71e-f5af2fc3ccab.png" alt="screenshot of Kyverno Policy Violations" style="display: block;" width="600" height="400" loading="lazy">

<p><strong>What you're proving:</strong> the terminal denial showed up in Grafana. You don't need every panel to light up. You need <strong>one clear sign</strong> that Kyverno blocks are being counted.</p>
<p><strong>Step 1: quick sanity check (top row, left to right)</strong></p>
<ol>
<li><strong>Policy Violations (time range)</strong>: big number. Pass: shows 1 or more. Fail: says No data.</li>
<li><strong>Violations (time range)</strong>: same idea, second counter. Pass: 1 or more.</li>
<li><strong>Active Kyverno Rules</strong> — usually 18. If this number shows up, Grafana can talk to Prometheus. That is good even when the first two panels are still empty.</li>
</ol>
<p><strong>Step 2: if the top two numbers work, skim the charts</strong></p>
<ul>
<li><strong>Violation Rate by Resource Kind</strong> (middle chart): look for a bump labeled Pod around the time you ran <code>kubectl apply</code>.</li>
<li><strong>Top Blocked Resource Types</strong> (bottom-left table): look for a Pod row.</li>
<li><strong>Violations by Namespace (trend)</strong> (bottom-right chart): look for a bump for clearledger.</li>
</ul>
<p>Charts can lag. A big number &gt; 0 in Step 1 is enough to move on. The charts are bonus proof for §7.6 screenshots.</p>
<p><strong>If the top two panels say "No data" but the terminal denial worked:</strong></p>
<p>This is common. Those panels count <strong>new</strong> denials during the time range, not the total ever recorded. One denial sometimes lands in Prometheus before Grafana's counter moves.</p>
<p>Try this:</p>
<ol>
<li>Run the same <code>kubectl apply</code> command again (denied again, that is expected).</li>
<li>Wait 60 seconds.</li>
<li>Click Refresh (circular arrow, top-right).</li>
</ol>
<p>After a second denial you should see 2 in the top stat panels. Screenshot that for §7.6.</p>
<p><strong>Still empty? Use Explore as backup proof:</strong></p>
<ol>
<li>Grafana left menu → Explore</li>
<li>Datasource: Prometheus</li>
<li>Paste: <code>sum(kyverno_admission_requests_total{request_allowed="false"})</code></li>
<li>Click Run query</li>
</ol>
<p><strong>Pass:</strong> the result is 1 or 2.</p>
<p>A screenshot of the terminal denial plus Explore showing a number &gt; 0 counts as portfolio proof even if the dashboard stats stay slow.</p>
<h4 id="heading-exercise-b-falco-shell-loki-security-event-timeline">Exercise B: Falco shell → Loki → Security Event Timeline</h4>
<p><strong>What you're doing (same idea as Exercise A):</strong></p>
<ul>
<li><strong>Exercise A:</strong> you did something bad, Kyverno blocked it, and the Grafana <strong>Kyverno</strong> dashboard updated.</li>
<li><strong>Exercise B:</strong> you do something suspicious inside a running pod, Falco detects it, and Grafana <strong>Security Event Timeline</strong> updates.</li>
</ul>
<p>You already did this in Stage 6 (<code>make demo-6</code>). Here you do it again and prove the alert shows up in Grafana, not only in <code>http://falco.local</code>.</p>
<p><strong>The story in one line:</strong> pretend you're an attacker who got shell access inside <code>auth-service</code>: Falco should scream, and the scream should appear on the timeline dashboard.</p>
<p><strong>Step 1: trigger the alert (terminal way)</strong></p>
<p>You're pretending an attacker got into <code>auth-service</code> and ran a quick command (<code>id</code>) to see who they're logged in as. That's suspicious. Falco is supposed to catch it.</p>
<p>The block below is three commands in order. Copy-paste the whole block:</p>
<pre><code class="language-bash">AUTH_POD=$(kubectl get pod -n clearledger -l app=auth-service \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
echo "Using pod: $AUTH_POD"
kubectl exec -n clearledger "$AUTH_POD" -c auth-service -- /bin/sh -c 'id &amp;&amp; exit'
</code></pre>
<p>What each line does:</p>
<ol>
<li><strong>Line 1</strong>: finds the name of a running <code>auth-service</code> pod and saves it in <code>AUTH_POD</code>.</li>
<li><strong>Line 2</strong>: prints that name so you can see it worked (not empty).</li>
<li><strong>Line 3</strong>: runs <code>/bin/sh -c 'id &amp;&amp; exit'</code> <strong>inside</strong> that pod. This is the fake “attack.” Falco watches for shells like this.</li>
</ol>
<p><strong>Pass: you only need these two lines in the output:</strong></p>
<pre><code class="language-text">Using pod: auth-service-77b7d9cd99-xxxxx
uid=1000 gid=1000 groups=1000
</code></pre>
<ul>
<li>First line: a real pod name (not blank).</li>
<li>Second line: the <code>id</code> command ran inside the container.</li>
</ul>
<p>That's Step 1 done. The pod is still running. You didn't break anything.</p>
<p><strong>Fail: stop and fix before Step 2:</strong></p>
<ul>
<li><code>error: Internal error</code> or <code>container not found</code></li>
<li><code>Using pod:</code> with nothing after it</li>
</ul>
<p>Run <code>kubectl get pods -n clearledger -l app=auth-service</code> and retry when one pod shows <strong>Running</strong>.</p>
<p><strong>Step 2: Confirm Falco saw it (terminal, right away)</strong></p>
<p>The Falco log is one long JSON line. Don't try to read the whole thing. Run:</p>
<pre><code class="language-bash">kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=50 | grep -i 'Shell spawned'
</code></pre>
<p><strong>Pass. You should see one short phrase somewhere in the line:</strong></p>
<pre><code class="language-text">Shell spawned in ClearLedger container ... pod=auth-service-... cmd=sh -c id &amp;&amp; exit
</code></pre>
<p>Or the rule name:</p>
<pre><code class="language-text">"rule":"Shell Spawned in ClearLedger Container"
</code></pre>
<p><strong>That one grep hit means Exercise B worked in the terminal.</strong> Screenshot this line for your portfolio.</p>
<p><strong>Ignore:</strong></p>
<ul>
<li><code>Defaulted container "falco" out of: ...</code>: normal kubectl noise</li>
<li>Lines about <code>postgres-0</code> and <code>/etc/passwd</code>: background noise from Stage 6, not your test</li>
<li>The rest of the JSON (<code>output_fields</code>, <code>k8smeta</code>, and so on). You don't need to parse it</li>
</ul>
<p><strong>If grep prints nothing:</strong> run Step 1 again, wait 5 seconds, then re-run the grep.</p>
<p><strong>Step 3: Confirm Loki stored it (wait ~60 seconds first)</strong></p>
<p>The story so far:</p>
<ul>
<li><strong>Step 1</strong>: you triggered the alert inside <code>auth-service</code></li>
<li><strong>Step 2</strong>: Falco wrote the alert to its own logs ✓</li>
</ul>
<p><strong>Step 3 asks:</strong> did that log line make it into <strong>Loki</strong> which is Grafana's log database?</p>
<p>Falco doesn't talk to Grafana directly. Promtail copies Falco's logs into Loki. That copy takes 60–90 seconds. Wait after Step 1, then run this check.</p>
<p><strong>What this command does:</strong></p>
<p>"Search Loki for Falco logs that contain <code>Shell spawned</code>, then show only lines that also mention <code>auth-service</code>."</p>
<pre><code class="language-bash">kubectl exec -n monitoring loki-0 -- wget -qO- \
  'http://127.0.0.1:3100/loki/api/v1/query?query=%7Bnamespace%3D%22falco%22%2Ccontainer%3D%22falco%22%7D%20%7C%3D%20%22Shell%20spawned%22&amp;limit=3' 2&gt;/dev/null \
  | grep -i 'auth-service'
</code></pre>
<p><strong>Pass:</strong> you see a line with both <code>auth-service</code> and <code>Shell spawned</code>. That means Loki has your alert and Grafana can show it.</p>
<p><strong>Fail (misleading pass):</strong> you grep for <code>ClearLedger</code> alone and get a hit from <code>postgres-0</code> reading <code>/etc/passwd</code>. That's background noise from Stage 6, not your shell test. Always look for <code>auth-service</code>.</p>
<p><strong>Empty output?</strong> That's OK. If Step 2 passed, <strong>continue to Step 4</strong>. Promtail may still be catching up, or the JSON is too long for this quick grep. Grafana often shows the alert even when this command prints nothing.</p>
<p><strong>Step 4: open Grafana</strong></p>
<p>Open <a href="http://grafana.local/d/clearledger-security-events?from=now-1h&amp;to=now">Security Event Timeline</a>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/c8f78ce4-8857-485b-9e4a-77c83bb95bd9.png" alt="screenshot of security timeline dashboard" style="display: block;" width="600" height="400" loading="lazy">

<p>This is the right dashboard. The title at the top should say <strong>ClearLedger - Security Event Timeline.</strong></p>
<p><strong>Before you look at panels:</strong></p>
<ol>
<li>Time range: <strong>Last 1 hour</strong> (top-right)</li>
<li>Auto-refresh: Off</li>
<li>Re-run Step 1 if your shell command was more than a few minutes ago</li>
<li>Wait 90 seconds, then click Refresh</li>
</ol>
<p><strong>What you'll probably see (and this is normal):</strong></p>
<ul>
<li><strong>CRITICAL Alerts (1h)</strong>: a big number like <strong>1.08 K</strong>. That is mostly <code>postgres-0</code> reading <code>/etc/passwd</code> on a loop (Stage 6 background noise). It does <strong>not</strong> mean you failed.</li>
<li><strong>Alerts by Rule Name</strong> (pie chart): dominated by <strong>Sensitive File Read in ClearLedger</strong>. Also normal.</li>
<li><strong>Recent CRITICAL / WARNING Events</strong>: lots of Postgres rows. Your shell alert is in there, but buried.</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/577e14fa-205d-462a-beb2-7a6a08514295.png" alt="anothre screenshot showing security even timeline grafana dashboard" style="display: block;" width="600" height="400" loading="lazy">

<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/e3a363af-676b-47a6-a61a-fe5ee8b7c130.png" alt="e3a363af-676b-47a6-a61a-fe5ee8b7c130" style="display: block;" width="600" height="400" loading="lazy">

<p>The top timeline (<strong>Falco Alerts by Priority - Timeline</strong>) may say <strong>No data</strong>. That's a known quirk. Don't panic, just use the log panel and browser search instead.</p>
<p><strong>How to find <em>your</em> alert (on this dashboard):</strong></p>
<ol>
<li>Stay on <strong>ClearLedger - Security Event Timeline</strong> — not Explore, not Tempo.</li>
<li>Click inside <strong>Recent CRITICAL / WARNING Events</strong> (the log list on the right).</li>
<li>Press <strong>Cmd+F</strong> (Mac) or <strong>Ctrl+F</strong> (Windows/Linux).</li>
<li>Search for <code>auth-service</code> or <code>Shell spawned</code>.</li>
</ol>
<p>If the search finds a row mentioning your pod and <strong>Shell spawned</strong>, screenshot it.</p>
<p><strong>Wrong place (common mistake):</strong> Grafana <strong>Explore</strong> with datasource <strong>Tempo</strong> showing <code>ledger-service</code> traces. That's <strong>Stage 7.5</strong> (OpenTelemetry), not Exercise B. Tempo shows request traces, not Falco security alerts.</p>
<p><strong>Pass for Exercise B (pick one):</strong></p>
<ol>
<li><strong>Best:</strong> Step 2 terminal grep shows <code>Shell spawned</code> <strong>and</strong> the <strong>Security Event Timeline</strong> log search finds <code>auth-service</code> / <code>Shell spawned</code> — screenshot both.</li>
<li><strong>Also fine:</strong> Step 2 grep screenshot <strong>plus</strong> the <strong>Security Event Timeline</strong> dashboard with <strong>CRITICAL Alerts (1h)</strong> showing a number (proves that Falco → Loki → Grafana works, even if your shell row is buried in postgres noise).</li>
<li><strong>Fallback (only if the dashboard search fails):</strong> Step 2 grep <strong>plus</strong> Grafana <strong>Explore</strong> with datasource <strong>Loki</strong> (not Tempo):<ul>
<li>Left menu, go to <strong>Explore</strong></li>
<li>Top-left datasource dropdown: choose <strong>Loki</strong></li>
<li>Query: <code>{namespace="falco", container="falco"} |= "Shell spawned"</code></li>
<li>Click <strong>Run query</strong></li>
<li>Look for a line with <code>auth-service</code></li>
</ul>
</li>
</ol>
<p>Screenshot for §7.6.</p>
<h4 id="heading-exercise-c-failed-login-loki-and-service-health">Exercise C: Failed login, Loki, and Service Health</h4>
<p><strong>The story:</strong> someone is guessing passwords on your login API.<br>You send ten bad login attempts from the terminal. <code>auth-service</code> writes <code>Failed login attempt</code> to its logs. Grafana <strong>Service Health + Auth Security</strong> should show the count go up.</p>
<p>Same pattern as A and B: terminal action, then logs, then dashboard.</p>
<p><strong>Step 1: send bad login attempts (terminal)</strong></p>
<p>Copy-paste the whole block:</p>
<pre><code class="language-bash">for i in $(seq 1 10); do
  curl -s http://clearledger.local/auth/health &gt;/dev/null
  curl -s -X POST http://clearledger.local/auth/login \
    -H 'Content-Type: application/json' \
    -d '{"email":"lab-attacker@evil.com","password":"wrong"}' &gt;/dev/null
done
echo "done"
</code></pre>
<p><strong>Pass:</strong> the only output you need is:</p>
<pre><code class="language-text">done
</code></pre>
<p>No output from the <code>curl</code> lines is normal. The loop hits <code>/auth/health</code> (keeps the app warm) and <code>/auth/login</code> with a wrong password ten times.</p>
<p><strong>Fail:</strong> <code>curl: (6) Could not resolve host</code>. Run <code>bash scripts/setup-hosts.sh</code> on your Mac. <code>curl: (7) Failed to connect</code>. Check <code>kubectl get pods -n clearledger -l app=auth-service</code>.</p>
<p><strong>Step 2: Confirm auth-service logged it</strong></p>
<pre><code class="language-bash">kubectl logs -n clearledger -l app=auth-service --tail=30 | grep -i 'Failed login' | tail -3
</code></pre>
<p><strong>Pass. You should see lines like:</strong></p>
<pre><code class="language-text">Failed login attempt for email: lab-attacker@evil.com
</code></pre>
<p>You may see several lines (one per failed attempt). One line is enough. Screenshot this for your portfolio.</p>
<p><strong>If grep prints nothing:</strong> wait 10 seconds and run again. If still empty, check the auth pod is Running: <code>kubectl get pods -n clearledger -l app=auth-service</code>.</p>
<p><strong>Step 3: open Grafana (wait ~60 seconds after Step 1)</strong></p>
<p>Open <a href="http://grafana.local/d/clearledger-service-health?from=now-1h&amp;to=now">Service Health + Auth Security</a>.</p>
<p><strong>This is the right dashboard.</strong> The title should say <strong>ClearLedger - Service Health + Auth Security</strong>.</p>
<ol>
<li>Time range: <strong>Last 1 hour</strong></li>
<li>Auto-refresh: <strong>Off</strong></li>
<li>Click <strong>Refresh</strong> once</li>
</ol>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/a6df00f0-3f4a-4951-ab5f-3efb498beaa1.png" alt="screenshot showing Service Health + Auth Security grafana dashboard" style="display: block;" width="600" height="400" loading="lazy">

<p>What to check (only these matter for Exercise C):</p>
<ol>
<li><strong>Failed Login Attempts (1h)</strong>: big number. <strong>Pass:</strong> <strong>&gt; 0</strong>. This is your main proof.</li>
<li><strong>Failed Login Log Stream</strong>: log lines in the panel. <strong>Pass:</strong> lines with <code>Failed login attempt</code> or <code>lab-attacker@evil.com</code>. Use <strong>Cmd+F</strong> inside the panel if needed.</li>
</ol>
<p>Panels you can ignore if empty:</p>
<ul>
<li><strong>Successful Logins</strong>: fine at <strong>0</strong> (you only sent bad passwords)</li>
<li><strong>Request Rate by Service</strong>: may be empty until §7.5 metrics images. Not required for Exercise C.</li>
</ul>
<p>You pass Exercise C when you have <strong>two screenshots:</strong></p>
<p><strong>Screenshot 1 (required):</strong> your Step 2 terminal output showing <code>Failed login attempt for lab-attacker@evil.com</code>. This proves the app logged the bad logins.</p>
<p><strong>Screenshot 2 (pick one of these):</strong></p>
<ul>
<li><strong>Option A:</strong> the <strong>Failed Login Attempts (1h)</strong> panel showing a number greater than zero (for example <strong>10</strong>). This proves Grafana counted the failures.</li>
<li><strong>Option B:</strong> the <strong>Failed Login Log Stream</strong> panel showing a line with <code>lab-attacker@evil.com</code>. Use this if the big number panel is still empty but the log stream has your email.</li>
</ul>
<p>You need Screenshot 1 and either Option A or Option B. That's enough for §7.6.</p>
<h4 id="heading-exercise-d-compliance-dashboard-the-auditor-summary">Exercise D: Compliance dashboard (the auditor summary)</h4>
<p><strong>What you're doing:</strong> open one dashboard that rolls up Exercises A, B, and C. This is the “show the auditor” view: admission control + runtime detection + application security in one screen.</p>
<p><strong>When:</strong> only after you finished A, B, and C.</p>
<p><strong>Step 1: open the dashboard</strong></p>
<p><a href="http://grafana.local/d/clearledger-compliance?from=now-1h&amp;to=now">Compliance Posture</a></p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/060714bc-c7c3-4d53-ad0e-d6d642970550.png" alt="screenshot of grafana Compliance Posture dashboard" style="display: block;" width="600" height="400" loading="lazy">

<p>Set <strong>Last 1 hour</strong>, auto-refresh <strong>Off</strong>, click <strong>Refresh</strong>.</p>
<p><strong>Step 2: Check the top row stats</strong></p>
<table>
<thead>
<tr>
<th>Stat on dashboard</th>
<th>Came from</th>
<th>Pass</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Policy Violations</strong></td>
<td>Exercise A (Kyverno)</td>
<td><strong>&gt; 0</strong></td>
</tr>
<tr>
<td><strong>Runtime Threats</strong></td>
<td>Exercise B (Falco)</td>
<td><strong>&gt; 0</strong> (postgres noise counts — that is OK)</td>
</tr>
<tr>
<td><strong>Failed Auth Attempts</strong></td>
<td>Exercise C (bad logins)</td>
<td><strong>&gt; 0</strong></td>
</tr>
</tbody></table>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ba6d0d41-bc88-4d8b-a93b-a1b78f99f4da.png" alt="screenshot of grafana Compliance Posture dashboard" style="display: block;" width="1068" height="1058" loading="lazy">

<p>All three don't need to be huge numbers. They just need to be <strong>above zero</strong> after your tests.</p>
<p><strong>If one stat is still 0:</strong> re-run that exercise (A, B, or C), wait 90 seconds, refresh. Policy Violations may need a second Kyverno denial like Exercise A.</p>
<p>This is screenshot #3 for §7.6: the single frame that proves defense-in-depth.</p>
<p><strong>✋ Hands-on checkpoint: are you actually done with Stage 7?</strong></p>
<p>Installing Grafana isn't the goal. Detection is: you triggered real events and can see them on dashboards.</p>
<p><strong>Optional terminal check (proves Grafana is wired up):</strong></p>
<pre><code class="language-bash">curl -s -u admin:admin123 'http://grafana.local/api/search?tag=clearledger' | jq -r '.[].title'

curl -s -u admin:admin123 'http://grafana.local/api/datasources' | jq -r '.[].name'
</code></pre>
<p>First command, you should see six dashboard names:</p>
<ul>
<li>ClearLedger - Kyverno Policy Violations</li>
<li>ClearLedger - Security Event Timeline</li>
<li>ClearLedger - Service Health + Auth Security</li>
<li>ClearLedger - Compliance Posture</li>
<li>ClearLedger - Kubernetes Audit Log Analysis</li>
<li>ClearLedger - DORA Metrics</li>
</ul>
<p>Second command, you should see at least:</p>
<ul>
<li>Prometheus</li>
<li>Loki</li>
</ul>
<p><strong>What does NOT mean you're done:</strong></p>
<p><code>make check-7</code> only checks that monitoring pods are running. Green output there does <strong>not</strong> replace §7.4.</p>
<p><strong>What DOES mean you are done:</strong></p>
<p>You ran Exercises A, B, and C in §7.4 and saved the §7.6 screenshots:</p>
<ol>
<li>Kyverno denial (terminal + dashboard)</li>
<li>Falco shell alert (terminal + Security Event Timeline)</li>
<li>Failed logins (terminal + Service Health)</li>
<li>Compliance Posture summary (all three stats above zero)</li>
</ol>
<p>If you have those four screenshots, Stage 7 is complete.</p>
<h3 id="heading-75-fill-in-the-request-rate-chart-optional">7.5: Fill in the Request Rate Chart (Optional)</h3>
<p><strong>This is not required for Stage 7.</strong> Exercises A–C and §7.6 screenshots don't need this section. Skip it if you're happy moving on.</p>
<p>Also, this is not the same as Stage 7.5 (OpenTelemetry/Tempo). This subsection is only about the <strong>Request Rate by Service</strong> chart on the Service Health dashboard.</p>
<h4 id="heading-what-this-section-is-for">What this section is for:</h4>
<p>On Service Health + Auth Security, the Failed Login panels work from logs (Loki). The Request Rate by Service chart needs something different: app pods must expose a <code>/metrics</code> endpoint so Prometheus can scrape request counts.</p>
<p>The code is already in the repo (<code>app/*/prom_metrics.py</code>). Prometheus is already configured to scrape it (<code>clearledger-podmonitor.yaml</code>). The usual problem: your cluster is still running older images from before that code was in your build.</p>
<h4 id="heading-step-1-check-if-you-already-have-metrics-30-seconds">Step 1: check if you already have metrics (30 seconds)</h4>
<p>Run this first. If it passes, skip the rest of §7.5.</p>
<pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=http_requests_total' 2&gt;/dev/null \
  | grep -o '"__name__":"http_requests_total"' | head -1
</code></pre>
<p><strong>Pass:</strong> prints <code>"__name__":"http_requests_total"</code>. Open Service Health, refresh, and the Request Rate by Service chart should already have lines.</p>
<p><strong>No output:</strong> continue to Step 2.</p>
<h4 id="heading-step-2-deploy-images-that-expose-metrics">Step 2: deploy images that expose <code>/metrics</code></h4>
<p>Pick one path.</p>
<p><strong>Path A: GitOps (if you have been using CI/CD since Stage 1–2)</strong></p>
<ol>
<li>Push a commit to <code>main</code> on your app repo.</li>
<li>Wait for CI to build new images and update <code>clearledger-infra</code>.</li>
<li>Wait for ArgoCD to show <strong>Synced</strong> and <strong>Healthy</strong> on the clearledger app.</li>
<li>Go to Step 3.</li>
</ol>
<p><strong>Path B: lab shortcut (faster, local only)</strong></p>
<pre><code class="language-bash">export DOCKER_USERNAME=your-dockerhub-user
bash stages/stage-7-observability/scripts/build-metrics-images.sh
</code></pre>
<p>This builds, pushes, and rolls out metrics-enabled images for all three services.</p>
<p><strong>Heads-up:</strong> ArgoCD self-heal may revert these image tags within a few minutes if <code>clearledger-infra</code> still points at older tags. That's fine for a quick lab demo. For a lasting fix, use Path A or update the infra repo (see §2 rollback notes).</p>
<h4 id="heading-step-3-verify-metrics-landed-60-seconds-after-rollout">Step 3: verify metrics landed (~60 seconds after rollout)</h4>
<pre><code class="language-bash">kubectl exec -n clearledger deploy/auth-service -c auth-service -- \
  wget -qO- http://127.0.0.1:8000/metrics 2&gt;/dev/null | head -5
</code></pre>
<p><strong>Pass:</strong> lines starting with <code># HELP</code> or <code>http_requests_total</code>.</p>
<p>Then confirm Prometheus sees them:</p>
<pre><code class="language-bash">kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=http_requests_total' 2&gt;/dev/null \
  | grep -o '"__name__":"http_requests_total"' | head -1
</code></pre>
<p><strong>Pass:</strong> <code>"__name__":"http_requests_total"</code></p>
<p>Generate a little traffic (re-run the Exercise C curl loop or hit <code>http://clearledger.local/auth/health</code> a few times), wait 60 seconds, then open <strong>Service Health + Auth Security</strong> and refresh. <strong>Request Rate by Service</strong> should show lines for <code>auth-service</code>, <code>ledger-service</code>, or <code>notification-service</code>.</p>
<h4 id="heading-when-to-stop">When to stop:</h4>
<ul>
<li><strong>Request Rate still empty but Failed Login panels work?</strong> You're done with Stage 7. Request Rate is a nice-to-have.</li>
<li><strong>Prometheus query passes but chart empty?</strong> Widen time range to <strong>Last 1 hour</strong>, generate traffic, wait 60s, refresh.</li>
</ul>
<h3 id="heading-76-wrap-up-stage-7-screenshots-done-check">7.6: Wrap up Stage 7 (Screenshots + Done Check)</h3>
<p>You're almost done. This section is just about saving proof, then moving on.</p>
<h4 id="heading-are-you-actually-finished">Are you actually finished?</h4>
<p>Opening Grafana and seeing six dashboards isn't enough. <code>make check-7</code> passing isn't enough either. That only proves pods are running.</p>
<p>You're done when you ran §7.4, waited for the panels to update, and saved three screenshots from your cluster.</p>
<p>If the panels are empty or only show old Postgres noise, go back to §7.4 first.</p>
<p><strong>Before each screenshot:</strong> set time range to <strong>Last 15 minutes</strong> (or <strong>Last 1 hour</strong> for Exercise B). Include the time picker and panel titles in the frame.</p>
<p><strong>Screenshot 1: Falco alert (Exercise B)</strong></p>
<p>Open <a href="http://grafana.local/d/clearledger-security-events">Security Event Timeline</a>.</p>
<p>Capture <strong>Recent CRITICAL / WARNING Events</strong> with a row that mentions <code>Shell spawned</code> or <code>auth-service</code>. If postgres rows bury it, use Cmd+F inside the log panel, that still counts.</p>
<p><strong>Screenshot 2: Kyverno denial (Exercise A)</strong></p>
<p>Open <a href="http://grafana.local/d/clearledger-kyverno-violations">Kyverno Policy Violations</a>.</p>
<p>Capture <strong>Policy Violations (time range)</strong> or <strong>Violations (time range)</strong> showing a number of 1 or more.</p>
<p><strong>Screenshot 3: Compliance summary (Exercise D)</strong></p>
<p>Open <a href="http://grafana.local/d/clearledger-compliance">Compliance Posture</a>.</p>
<p>Capture the top row with all three stats above zero: <strong>Policy Violations</strong>, <strong>Runtime Threats</strong>, and <strong>Failed Auth Attempts</strong>.</p>
<p><strong>Screenshot 4 (optional): Failed logins (Exercise C)</strong></p>
<p>Open <a href="http://grafana.local/d/clearledger-service-health">Service Health + Auth Security</a>.</p>
<p>Capture <strong>Failed Login Attempts (1h)</strong> above zero, or <strong>Failed Login Log Stream</strong> showing <code>lab-attacker@evil.com</code>.</p>
<p>Save files somewhere sensible, like <code>docs/evidence/stage-7-screenshot-1-falco.png</code>. Name them so you know what each proves.</p>
<p><strong>Final check:</strong> run <code>make check-7</code> (§7.7), save your VM, and you can claim Stage 7.</p>
<h3 id="heading-77-verify">7.7: Verify</h3>
<pre><code class="language-bash">make check-7
</code></pre>
<p><strong>Expected:</strong></p>
<pre><code class="language-text">▶ Stage 7 — Observability (Grafana + Prometheus + Loki)
  ✓ Prometheus is running
  ✓ Grafana reachable (http://grafana.local or in-cluster health OK)
  ✓ Loki pod is running (0 restarts)
  ✓ Loki reachable from Grafana (http://loki:3100/ready)
  ✓ ClearLedger alerting rules exist
  ✓ ClearLedger dashboards imported (6 found)
</code></pre>
<p>Warnings about Loki restarts or missing dashboards: fix with §7.1 before claiming Stage 7 complete.</p>
<p><strong>Save your VM</strong> after §7.6 and <code>make check-7</code>. See the block at the end of Stage 7 below.</p>
<h3 id="heading-78-what-broke-lab-notes-interview-talking-points">7.8: What Broke (Lab Notes + Interview Talking Points)</h3>
<p><strong>The stack in one sentence:</strong> Prometheus stores numbers (metrics), Loki stores log lines, and Grafana displays both visually. Nothing appears until something actually happens in the cluster.</p>
<h4 id="heading-what-tripped-you-up-in-the-lab">What tripped you up in the lab</h4>
<ol>
<li><strong>Empty dashboards right after install:</strong> Normal. Grafana doesn't create events. You trigger them in §7.4 (Kyverno denial, Falco shell, failed logins).</li>
<li><strong>Loki slow or refresh stuck on “Cancel”:</strong> Falco logs are huge. <strong>Last 24 hours</strong> overloads a small cluster. Use <strong>Last 1 hour</strong>, one dashboard at a time, and wait ~10 seconds.</li>
<li><code>make check-7</code> passed but panels still empty <em>(lab checklist only, not an interview topic)</em>: The health check confirms Prometheus/Loki/Grafana pods are up. It does <strong>not</strong> mean events exist. You still need §7.4 + §7.6 before you snapshot and move on.</li>
</ol>
<h4 id="heading-if-someone-asks-about-this-in-an-interview">If someone asks about this in an interview</h4>
<p><strong>Empty dashboards?</strong> Grafana only shows what already happened. No event in the time range means an empty panel. That's normal until you trigger something.</p>
<p><strong>Loki slow on a small cluster?</strong> Falco logs are huge. We kept time ranges short (15 minutes, not 24 hours) and opened one dashboard at a time. Same trade-off you would make in prod on limited hardware.</p>
<p><strong>How did you prove it worked?</strong> I ran the attacks myself: denied a bad pod, spawned a shell in a running container, and sent failed logins. Then I checked Grafana and screenshot the matching panels. Terminal action first, dashboard proof second.</p>
<p><strong>Short version you can say out loud:</strong></p>
<blockquote>
<p>"I connected Kyverno and Falco into Grafana. To prove it, I triggered a policy block and a runtime alert, then showed both on security dashboards. On a single-node lab, Loki got slow with wide time ranges, so we kept queries tight."</p>
</blockquote>
<p>Pipeline problems from earlier stages (Trivy, Kyverno, image tags, and so on) are in <code>docs/troubleshooting.md</code> — not something you need to rehearse for Stage 7.</p>
<h3 id="heading-79-if-panels-look-wrong-after-a-repo-update">7.9: If Panels Look Wrong After a Repo Update</h3>
<p>Re-apply dashboards, then generate real events (§7.4, not fake data):</p>
<pre><code class="language-bash">bash stages/stage-7-observability/scripts/install-observability.sh
# Then run Exercises A–C from §7.4 (Kyverno denial, Falco shell, failed logins)
</code></pre>
<p>Open Grafana at <strong>Last 1 hour</strong>, wait ~30–60s after each exercise, and refresh once. The §7.4 exercises cover expected appearance for each dashboard.</p>
<h3 id="heading-what-you-learned-in-stage-7">What You Learned in Stage 7</h3>
<ul>
<li><strong>Prometheus</strong> proves countable security events (Kyverno denials, HTTP rates)</li>
<li><strong>Loki</strong> proves forensic detail (Falco JSON, auth log lines)</li>
<li><strong>Grafana</strong> is the narrative layer, not a second install step after the lab</li>
<li>You can trace: terminal action, backend signal, panel update</li>
<li>ServiceMonitors / PodMonitors are what connect Stages 4–6 to charts</li>
<li>Empty dashboards mean “no events yet” or “wrong time range”, not “broken security”</li>
<li>Compliance posture is how you answer an auditor in one screen</li>
<li>Network policies must explicitly allow the <code>monitoring</code> namespace to reach app pods on port 8000, otherwise PodMonitor scrapes silently fail with <code>context deadline exceeded</code></li>
<li>Kubernetes Audit Log dashboard is empty on MicroK8s by design: the API server audit pipeline (audit-policy → file → Promtail → Loki) isn't enabled by default</li>
<li>Request Rate requires the full chain: app image with <code>/metrics</code>, PodMonitor, and network policy: any one missing means the panel stays empty</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Built security observability with Prometheus, Loki, and Grafana (dashboards correlating Kyverno violations, Falco alerts, and DORA metrics) and can prove a security event end-to-end from terminal to dashboard.</p>
</blockquote>
<h4 id="heading-stage-7-done-checklist">Stage 7 done checklist:</h4>
<ul>
<li><code>make check-7</code> → 6/6 ✓ (Stage 6.5 Litmus failure is expected: scaled down for memory)</li>
<li><code>http://grafana.local/d/clearledger-kyverno-violations</code>. Violations stat &gt; 0</li>
<li><code>http://grafana.local/d/clearledger-security-events</code>. CRITICAL Falco alert visible</li>
<li><code>http://grafana.local/d/clearledger-compliance</code>. Policy Violations + Runtime Threats + Failed Auth Attempts all &gt; 0</li>
<li><code>http://grafana.local/d/clearledger-service-health</code>. Failed Login Attempts &gt; 0. Request Rate &gt; 0 only if you did §7.5</li>
<li>Portfolio screenshots 1–3 saved</li>
</ul>
<p><code>make snapshot STAGE=7 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage7</code>. <strong>Don't skip this</strong>. Stage 7 is heavy, and disk pressure is common. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<p>After a Mac reboot or sleep, auth/ledger pods may show <strong>Unknown</strong> or <strong>Init:0/1</strong> even though the cluster is up (see <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md) Mac reboot</a>).</p>
<h2 id="heading-stage-75-opentelemetry-optional">Stage 7.5 — OpenTelemetry (Optional)</h2>
<p><strong>You can skip this whole stage.</strong> Stage 7 (metrics + logs) is enough to finish the homelab and move to Stage 8.</p>
<p>Only do Stage 7.5 if you want distributed traces for your portfolio or interviews, and your VM has spare RAM (about 1.5 Gi free).</p>
<h3 id="heading-what-you-are-adding">What You Are Adding</h3>
<p>Stage 7 answers: <em>did something happen?</em> (Kyverno blocked a pod, Falco saw a shell, or login failed.)</p>
<p>Traces answer: <em>what steps ran on this one request, and how long did each take?</em></p>
<ul>
<li><strong>Metrics</strong>: how many requests, how many errors</li>
<li><strong>Logs</strong>: what the app printed in its log file, such as errors, warnings, login failures)</li>
<li><strong>Traces</strong>: ledger-service called auth-service (12ms), then Postgres (8ms)</li>
</ul>
<p>In this stage, you send one real transaction, then open that request in Grafana Explore (Tempo). You'll see each step listed with its timing: ledger-service, auth-service, Postgres.</p>
<h3 id="heading-before-you-start">Before You Start</h3>
<ol>
<li>Finish Stage 7: §7.4 exercises done, §7.6 screenshots saved, <code>SKIP_CHAOS_CHECK=1 make check-7</code> passes.</li>
<li>Check VM memory: <code>multipass exec clearledger -- free -h</code> , want about 1.5 Gi free.</li>
<li>If you ran Stage 6.5 Litmus, scale it down first (§7.0).</li>
</ol>
<p>You're done when you see the full request trace in Grafana Explore (Tempo datasource) and <code>make check-75</code> passes. Then <code>make snapshot STAGE=75</code>.</p>
<h3 id="heading-ignore-this-warning-in-app-logs">Ignore This Warning in App Logs</h3>
<p>Since Stage 7 you may see:</p>
<pre><code class="language-plaintext">WARNING: Transient error StatusCode.UNAVAILABLE encountered while exporting traces
</code></pre>
<p>That is harmless. The apps are already set up to send trace data, but the receiver isn't installed until §7.5.3.</p>
<p>Your apps still work fine, the trace data just gets thrown away. Installing the collector in §7.5.3 makes the warning go away.</p>
<h3 id="heading-how-tracing-is-wired">How Tracing is Wired</h3>
<ol>
<li>Your apps send trace data when a request runs</li>
<li>OTel Collector receives it (port 4317) and passes it along</li>
<li>Grafana Tempo stores it</li>
<li>Grafana Explore (Tempo selected) is where you look at one request step by step</li>
</ol>
<p>Apps talk to the collector only, not to Tempo directly. That way you can change where traces are stored later without rebuilding the apps.</p>
<h3 id="heading-751-check-memory-and-load">7.5.1: Check Memory and Load</h3>
<p>Tempo needs ~300MB. Confirm headroom before installing:</p>
<pre><code class="language-bash">multipass exec clearledger -- free -h    # want ~1.5Gi available
multipass exec clearledger -- uptime      # load should be reasonable for your CPU count
SKIP_CHAOS_CHECK=1 bash scripts/health-check.sh 7
</code></pre>
<p>If Litmus is still running from Stage 6.5, scale it down first (§7.0):</p>
<pre><code class="language-bash">kubectl get pods -n litmus --field-selector=status.phase=Running
# Expected: no resources found
</code></pre>
<h3 id="heading-752-install-grafana-tempo">7.5.2: Install Grafana Tempo</h3>
<p>Tempo is the trace storage backend. Install it into the <code>monitoring</code> namespace next to Prometheus and Loki:</p>
<pre><code class="language-bash">helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install tempo grafana/tempo \
  --namespace monitoring \
  --set tempo.storage.trace.backend=local \
  --set tempo.storage.trace.local.path=/var/tempo \
  --set persistence.enabled=true \
  --set persistence.size=5Gi \
  --wait
</code></pre>
<p><strong>Verify Tempo is running:</strong></p>
<pre><code class="language-bash">kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
# Expected: tempo-0   1/1   Running
</code></pre>
<pre><code class="language-bash">kubectl exec -n monitoring tempo-0 -- wget -qO- http://localhost:3200/ready
# Expected: ready
</code></pre>
<h3 id="heading-753-deploy-otel-collector-and-wire-grafana">7.5.3: Deploy OTel Collector and Wire Grafana</h3>
<p>This applies the OTel Collector (receives spans from app pods) and registers Tempo as a Grafana datasource automatically via the sidecar:</p>
<pre><code class="language-bash">kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/otel-collector.yaml
kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/grafana-datasource-tempo.yaml
</code></pre>
<p><strong>Verify the collector is running:</strong></p>
<pre><code class="language-bash">kubectl get pods -n monitoring -l app=otel-collector
# Expected: otel-collector-xxxxx   1/1   Running
</code></pre>
<p><strong>Verify the collector started (not trace receipt yet):</strong></p>
<p>Apps push spans to the collector over OTLP: the collector doesn't scrape pods. At this step you're only confirming that it's listening.</p>
<pre><code class="language-bash">kubectl logs -n monitoring deploy/otel-collector --tail=15
# Expected:
#   Starting GRPC server ... endpoint: 0.0.0.0:4317
#   Starting HTTP server ... endpoint: 0.0.0.0:4318
#   Everything is ready. Begin running and processing data.
# No crash loops or repeated errors.
</code></pre>
<p>Proof that traces are actually flowing comes later: after you generate traffic in §7.5.6, check collector logs for span export lines from the <code>debug</code> exporter, then confirm the trace in Grafana Tempo (§7.5.7).</p>
<h3 id="heading-754-enable-prometheus-remote-write-receiver">7.5.4: Enable Prometheus Remote Write Receiver</h3>
<p>The OTel Collector also forwards OTel metrics to Prometheus via remote write. Prometheus needs to accept them:</p>
<pre><code class="language-bash">helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f stages/stage-7-observability/infra/helm/kube-prometheus-stack-values.yaml \
  --wait
</code></pre>
<p>This applies the <code>enableRemoteWriteReceiver: true</code> setting added to the Helm values in Stage 7.5. Wait for Prometheus to restart (about 60 seconds).</p>
<h3 id="heading-755-verify-app-pods-connect-to-the-collector">7.5.5: Verify App Pods Connect to the Collector</h3>
<p>The deployments in <code>clearledger-infra</code> already have <code>OTEL_EXPORTER_OTLP_ENDPOINT</code> set. Once the collector is running, the pods auto-connect.</p>
<p>Confirm that the OTEL warnings are gone:</p>
<pre><code class="language-bash">kubectl logs -n clearledger deploy/ledger-service -c ledger-service --tail=20 2&gt;/dev/null \
  | grep -v "opentelemetry\|otlp\|Transient" | tail -10
# Expected: only INFO request logs, no WARNING: Transient error
</code></pre>
<p>If warnings persist, the network policy may not have port 4317 egress. Apply the latest policies:</p>
<pre><code class="language-bash">kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml
</code></pre>
<h3 id="heading-756-generate-a-trace">7.5.6: Generate a Trace</h3>
<p>Now create a transaction and watch it flow through the system:</p>
<pre><code class="language-bash"># Step 1: register (skip if already registered)
curl -s -X POST http://clearledger.local/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"trace-demo@clearledger.io","password":"TracePass123"}' | python3 -m json.tool

# Step 2: login and grab the token
TOKEN=$(curl -s -X POST http://clearledger.local/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"trace-demo@clearledger.io","password":"TracePass123"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
echo "Token acquired: ${TOKEN:0:20}..."

# Step 3: create a transaction (this is the request you will trace)
curl -s -X POST http://clearledger.local/ledger/transactions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"amount": 5000, "direction": "credit"}' | python3 -m json.tool
</code></pre>
<p><strong>Verify the collector received spans:</strong></p>
<pre><code class="language-bash">kubectl logs -n monitoring deploy/otel-collector --tail=30 \
  | grep -iE "Traces|spans|ResourceSpans" || echo "No span lines yet — see §7.5.5 (OTEL env / netpol)"
# Expected after a successful transaction: debug exporter lines mentioning exported traces/spans
</code></pre>
<h3 id="heading-757-view-the-trace-in-grafana">7.5.7: View the Trace in Grafana</h3>
<p>Open <a href="http://grafana.local"><strong>http://grafana.local</strong></a> and go to the left sidebar <strong>Explore</strong> (compass icon).</p>
<h4 id="heading-step-1-select-tempo-and-open-search">Step 1: Select Tempo and open Search</h4>
<p>At the top of the query pane:</p>
<ol>
<li>Datasource dropdown (orange <strong>T</strong> logo) → <strong>Tempo</strong></li>
<li>Query row labeled A (Tempo) → three tabs: Search | TraceQL | Service Graph</li>
<li>Click <strong>Search</strong>. This shows dropdown filters. <strong>TraceQL</strong> is a text box only. If you land there with nothing typed you get <code>0 series returned</code>.</li>
</ol>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/b005b7ed-ad35-4c2f-8ca4-7d685717754f.png" alt="screenshot of grafana showing tempo and ledger service" style="display: block;" width="1158" height="408" loading="lazy">

<h4 id="heading-step-2-filter-by-service">Step 2: Filter by service</h4>
<p>In the <strong>Search</strong> tab:</p>
<ul>
<li><strong>Service Name</strong> → type or select <code>ledger-service</code></li>
<li>Leave Span Name, Status, Duration, and Tags empty for now</li>
<li>Grafana shows the query it will run: <code>{resource.service.name="ledger-service"}</code></li>
</ul>
<p>Set the time range (top-right clock icon) to <strong>Last 15 minutes</strong> so your §7.5.6 transaction is included.</p>
<h4 id="heading-step-3-run-the-query">Step 3: Run the query</h4>
<p>Grafana Explore has <strong>no “Run query” button</strong>: results appear automatically after selecting a service. If the table stays empty, use the <strong>blue refresh button</strong> top-right of the pane.</p>
<h4 id="heading-step-4-open-the-trace-waterfall">Step 4: Open the trace waterfall</h4>
<p>Below the query editor, find <strong>Table - Traces</strong>. You should see at least one row like:</p>
<table>
<thead>
<tr>
<th>Column</th>
<th>Example</th>
</tr>
</thead>
<tbody><tr>
<td>Trace ID</td>
<td><code>5730edf3…</code> (blue link)</td>
</tr>
<tr>
<td>Start time</td>
<td>when you ran the <code>curl</code></td>
</tr>
<tr>
<td>Service</td>
<td><code>ledger-service</code></td>
</tr>
<tr>
<td>Name</td>
<td><code>POST /transactions</code></td>
</tr>
<tr>
<td>Duration</td>
<td>~200ms (yours may differ)</td>
</tr>
</tbody></table>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ac4aad25-4ca4-4ffc-9fb6-25c30f0214bd.png" alt="screenshot of grafana showing tempo and ledger service and query result" style="display: block;" width="1179" height="1055" loading="lazy">

<p><strong>Click the Trace ID link.</strong> The right panel opens the trace detail view.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/d3c15dc0-e4b3-49e4-93e6-28b49feab6eb.png" alt="screenshot of grafana showing tempo and ledger service and query results" style="display: block;" width="1226" height="1289" loading="lazy">

<h4 id="heading-what-the-trace-detail-view-shows">What the trace detail view shows</h4>
<p>Header: <code>ledger-service: POST /transactions</code></p>
<ul>
<li><strong>Trace ID</strong>: unique ID for this request</li>
<li><strong>Duration</strong>: total end-to-end time</li>
<li><strong>Services</strong>: <code>2</code> (<code>ledger-service</code> and <code>auth-service</code> for a normal transaction)</li>
</ul>
<p>Expand spans in the timeline:</p>
<pre><code class="language-plaintext">ledger-service   POST /transactions          (~total duration)
  ├── auth-service   GET /verify             ← JWT check over HTTP
  ├── ledger-service INSERT / sqlalchemy    ← Postgres write
  └── (optional) redis PUBLISH              ← only if amount ≥ notification threshold
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/0b69babc-90c6-4cdc-a0bf-bad012854a36.jpg" alt="trace transaction flow" style="display: block;" width="1536" height="957" loading="lazy">

<p><strong>Reading the trace detail screen:</strong></p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3a72b4a4-2243-403d-9f03-ebc3b22e5337.png" alt="Tempo trace detail: ledger-service transaction with auth-service verify step." style="display: block;" width="1226" height="1289" loading="lazy">

<p>Each row is one step in the request (Grafana calls it a <em>span</em>). The colored bar on the right shows <strong>how long that step took</strong>. That's the <em>span bar</em>. A longer bar = more time spent on that step.</p>
<p>Click a row or its bar to open the details panel on the right. You'll see two kinds of metadata:</p>
<ul>
<li><strong>Span attributes</strong>: what happened in <em>this step</em>.<br>Examples: HTTP method (<code>POST</code>, <code>GET</code>), status code (<code>200</code>), or SQL text on a database step. In your trace you might see <code>asgi.event.type: http.request</code> on the FastAPI receive step.</li>
<li><strong>Resource attributes</strong>: <em>where</em> the step ran.<br>Examples: <code>service.name: ledger-service</code>, <code>k8s.cluster.name: clearledger</code>, <code>deployment.environment: production</code>.</li>
</ul>
<p>Quick mental model: span attributes = what the step did. Resource attributes = which service produced it.</p>
<p><strong>Connecting traces to logs:</strong> once you have a step selected, the Logs tab will take you straight to the matching Loki log lines for that pod at the same moment in time.</p>
<p><strong>Screenshot this trace detail view</strong>: portfolio proof for Stage 7.5.</p>
<h4 id="heading-traceql-alternative">TraceQL alternative</h4>
<p>If you prefer the text box, Switch to the <strong>TraceQL</strong> tab, and paste:</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/679d687d-1a2e-474b-8f7e-672f3ab2eb2b.png" alt="screenshot showing direction for where traceql button is" style="display: block;" width="1157" height="263" loading="lazy">

<pre><code class="language-traceql">{ resource.service.name = "ledger-service" }
</code></pre>
<h4 id="heading-if-the-table-is-empty">If the table is empty</h4>
<p><strong>If TraceQL says</strong> <code>0 series returned</code>: Use the <strong>Search</strong> tab instead, or paste the TraceQL query from above into the TraceQL tab.</p>
<p><strong>If search tab has no rows:</strong> Widen the time range to <strong>Last 15 minutes</strong>, re-run the transaction curl from §7.5.6, wait a few seconds, and refresh.</p>
<p><strong>If grafana can't connect to Tempo:</strong> The datasource URL needs port <strong>3200</strong>. Re-apply the datasource and restart Grafana:</p>
<pre><code class="language-bash">kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/grafana-datasource-tempo.yaml
kubectl rollout restart deployment/kube-prometheus-stack-grafana -n monitoring
</code></pre>
<p><strong>If collector logs show no trace data:</strong> Work through §7.5.5: usually the OTEL environment variables or network policy blocking port 4317.</p>
<h3 id="heading-757b-understand-when-a-trace-happens">7.5.7b: Understand When a Trace Happens</h3>
<p>You ran one curl command in §7.5.6. Grafana shows every place that single request traveled.</p>
<p>Think of it like tracking a package:</p>
<ol>
<li><strong>You</strong> sent <code>POST /transactions</code> to <strong>ledger-service</strong></li>
<li><strong>ledger-service</strong> asked <strong>auth-service</strong>: "is this user logged in?"</li>
<li><strong>ledger-service</strong> saved the row to the <strong>database</strong></li>
<li><strong>redis</strong> only runs if the amount is <strong>big</strong> (10,000 or more)</li>
</ol>
<p>Each of those is a row you see in the Tempo detail screen. You're not looking at four separate requests. It's <strong>one</strong> request with multiple stops.</p>
<p><strong>Why do I see both ledger-service and auth-service?</strong></p>
<p>Because ledger had to call auth before it could save the transaction. Grafana groups those stops into one trip so you can see the full path, not just the first hop.</p>
<p><strong>Why did my demo have no Redis row?</strong></p>
<p>You used <code>"amount": 5000</code>. The app only talks to Redis when the amount is 10,000 or higher. So seeing ledger + auth + database but no Redis is correct.</p>
<p>Want to see Redis? Run §7.5.6 again with <code>"amount": 15000</code> and search Tempo again.</p>
<p><strong>Optional: connect it to the code</strong></p>
<p>Open <code>app/ledger-service/main.py</code>, find <code>create_transaction</code>, and read top to bottom. The Tempo rows follow that function in order: check the user, save to the database, and maybe notify Redis.</p>
<p><strong>Optional: same request, three tools</strong></p>
<p>At the time you ran the curl:</p>
<ul>
<li><strong>Tempo</strong> (this stage): which services ran and how long each took</li>
<li><strong>Loki</strong> (Stage 7): what the apps wrote in their log files</li>
<li><strong>Prometheus</strong> (Stage 7): how many requests happened around that time</li>
</ul>
<p>Same moment, three different views. You already used Loki and Prometheus in Stage 7.</p>
<h3 id="heading-758-verify">7.5.8: Verify</h3>
<pre><code class="language-bash">make check-75
</code></pre>
<p>Expected output:</p>
<pre><code class="language-text">▶ Stage 7.5 — OpenTelemetry (Distributed Tracing)
  ✓ OTel Collector is running (1 replica(s))
  ✓ Grafana Tempo datasource ConfigMap exists
  ✓ Tempo is running
  ✓ auth-service has OTEL_EXPORTER_OTLP_ENDPOINT set
</code></pre>
<p><strong>If you see a warning instead:</strong></p>
<pre><code class="language-text">⚠ OTel env vars not found on auth-service, redeploy with updated manifests
</code></pre>
<p><code>check-75</code> looks for <code>OTEL_EXPORTER_OTLP_ENDPOINT</code> in the deployment manifest. Older Stage 5 manifests may not list it even though tracing works: the Python apps default to <code>http://otel-collector.monitoring.svc.cluster.local:4317</code> when the env var is missing.</p>
<p>You can proceed if collector logs show spans and Tempo shows your trace. To clear the warning, apply only the app deployments (not the whole kustomize tree: Kyverno may block redis/postgres patches):</p>
<pre><code class="language-bash">kubectl apply -f infra/manifests/auth-service/deployment.yaml
kubectl apply -f infra/manifests/ledger-service/deployment.yaml
kubectl rollout restart deployment/auth-service deployment/ledger-service -n clearledger
make check-75
</code></pre>
<p><strong>Save your VM</strong> after <code>make check-75</code>. See the block at the end of Stage 7.5 below.</p>
<h3 id="heading-what-you-learned">What You Learned</h3>
<p>Stage 7 gave you metrics (how busy?) and logs (what was printed?). Stage 7.5 adds traces (for one slow request, which step took the time?).</p>
<p>In the lab you proved it with one <code>POST /transactions</code> curl. In production the idea is the same: a user hits an API, the request crosses multiple services, and you need to see that full path in one place.</p>
<h4 id="heading-if-someone-asks-in-an-interview">If someone asks in an interview:</h4>
<p><strong>Why traces at all?</strong> Metrics might tell you p99 latency doubled. Logs might show an error on one pod. Traces tell you <em>which downstream call</em> in the chain caused the delay, auth, database, cache, or third-party API, without guessing.</p>
<p><strong>How did you implement it?</strong> We instrumented the services with OpenTelemetry, sent telemetry to a collector, and stored traces in Grafana Tempo. Apps talk to the collector, not directly to the backend, so we can change storage later without redeploying every service.</p>
<p><strong>What would you do in an incident?</strong> Find a slow or failing trace ID (from logs, metrics, or an alert), open it in Tempo, walk the call chain service by service, see where time stacked up, then jump to logs for that service at the same timestamp. That's faster than tailing logs on five pods and hoping they line up.</p>
<p><strong>Short version you can say out loud:</strong></p>
<blockquote>
<p>"We use the three pillars together: Prometheus for rates and errors, Loki for log detail, and Tempo for request-level debugging across microservices. When latency spikes, I start from a trace, identify the slow hop, often a database or downstream API, and correlate back to logs and metrics for that service."</p>
</blockquote>
<p><code>make check-75 &amp;&amp; make snapshot STAGE=75 &amp;&amp; make snapshots</code>. Confirm <code>clearledger.stage75</code>. See <a href="#heading-how-to-save-your-progress">How to Save Your Progress</a>.</p>
<h2 id="heading-stage-8-aws-migration">Stage 8 — AWS Migration</h2>
<p>Your goal here is to run the same ClearLedger app on AWS instead of your laptop VM.</p>
<p>You're not rewriting the application. Stages 0–7 built containers on Kubernetes with GitOps, Kyverno, secrets, and observability. Stage 8 changes where it runs. You keep the same images, the same ArgoCD workflow, and the same security policies. Only the cloud services underneath change (MicroK8s → EKS, Vault → Secrets Manager, and so on).</p>
<ul>
<li><strong>Homelab:</strong> MicroK8s, Postgres in a pod, dev Vault, Docker Hub, <code>clearledger.local</code></li>
<li><strong>AWS:</strong> EKS, RDS, Secrets Manager, ECR, ALB hostname</li>
</ul>
<p><strong>Am I ready for Stage 8?</strong></p>
<ul>
<li>Homelab complete through Stage 7 (Stage 7.5 optional)</li>
<li>make check-7 passes (and make check-75 if you did traces)</li>
<li>AWS account with billing alerts enabled. make aws-up creates billable resources</li>
<li>Skim §8.2 so you know what make aws-up does (even if you use the quick path)</li>
</ul>
<p><strong>Done when</strong> the app is reachable on the AWS ALB, ArgoCD syncing, and you run <code>make aws-down</code> when finished to stop charges.</p>
<h3 id="heading-what-make-aws-up-gives-you">What <code>make aws-up</code> Gives You</h3>
<p>This is a <strong>demo stack</strong>: production-<em>shaped</em>, but not production-<em>ready</em>. It has HTTP only (no TLS cert).</p>
<p>Stage 7 observability is installed automatically. CI still runs Gitleaks, Semgrep, Checkov, Trivy, and Cosign.</p>
<p>For real production, you would add HTTPS (see <a href="https://github.com/Osomudeya/clearledger/blob/main/stages/stage-8-aws-migration/manifests/ingress-aws-https.example.yaml"><code>ingress-aws-https.example.yaml</code></a>), staging before promote, and alert routing. Those are documented but not applied by the spinup script.</p>
<p><strong>GitOps rule:</strong> after bootstrap, don't <code>kubectl apply</code> app Deployments by hand. ArgoCD owns the cluster (Stage 2). Push manifest changes to Git and let ArgoCD sync.</p>
<h3 id="heading-secrets-on-aws">Secrets on AWS</h3>
<p>On the homelab, Vault wrote secret files into the pod. On AWS, secrets live in <strong>AWS Secrets Manager</strong> (created by Terraform). Your app still needs them as environment variables like <code>DATABASE_URL</code>.</p>
<p><strong>ESO (default in this lab)</strong>: the simple mental model:</p>
<ol>
<li>Terraform stores the real password in AWS Secrets Manager (for example <code>clearledger/auth-service</code>)</li>
<li>External Secrets Operator (ESO) watches that AWS secret</li>
<li>ESO copies it into a normal Kubernetes Secret inside the cluster (for example <code>auth-service-secret</code>)</li>
<li>Your deployment reads <code>DATABASE_URL</code> from that Kubernetes Secret, same as Stage 0, but the values come from AWS instead of a YAML file in Git</li>
</ol>
<p>You never put passwords in Git. ESO keeps the Kubernetes Secret in sync with Secrets Manager.</p>
<p><strong>CSI (optional, §8.5 exercise)</strong>: same AWS secrets but different delivery: mounted as <strong>files</strong> at <code>/mnt/secrets/*</code> instead of env vars. This is closer to how Vault worked on the homelab.</p>
<p><strong>IRSA</strong>: how ESO is allowed to read Secrets Manager without storing AWS access keys in the cluster. AWS trusts a Kubernetes service account instead.</p>
<p>IRSA lets AWS trust a Kubernetes ServiceAccount, no <code>AWS_ACCESS_KEY_ID</code> in Git or in the cluster.</p>
<p>Details here: <a href="https://github.com/Osomudeya/clearledger/tree/main/stages/stage-8-aws-migration/docs"><code>stages/stage-8-aws-migration/docs/secrets-patterns.md</code></a>.</p>
<h3 id="heading-81-two-ways-through-stage-8">8.1: Two Ways Through Stage 8</h3>
<p><strong>Quick path (~45–60 min):</strong> edit <code>terraform/secrets.tf</code> (replace <code>CHANGE_ME_BEFORE_APPLY</code>), then:</p>
<pre><code class="language-bash">make aws-up    # runs stages/stage-8-aws-migration/scripts/aws-spinup.sh
make aws-down  # destroys billable resources when you are done
</code></pre>
<p>Read §8.2 afterward so you know what ran.</p>
<p><strong>Manual path (§8.3):</strong> run Terraform, ECR push, ArgoCD, Kyverno, ESO, and deploy yourself. Use this when learning, interviewing, or debugging a failed spinup.</p>
<p>Don't skip §8.2–§8.5 if you only ran <code>make aws-up</code>. Otherwise you won't know what Terraform, ESO, or ArgoCD each did.</p>
<p>Before your first Stage 8 push, read <a href="#heading-ci-routing-stages-17-vs-stage-8">§8: CI routing and <code>CLEARLEDGER_CI_TARGET</code></a> and set <code>CLEARLEDGER_CI_TARGET=aws</code> only after Terraform succeeds, not while you are still on Stages 1–7.</p>
<h3 id="heading-82-what-make-aws-up-runs">8.2: What <code>make aws-up</code> Runs</h3>
<p>The spinup script runs 15 steps in order:</p>
<p><strong>Setup (1–6)</strong>: Check tools and AWS login; <code>terraform apply</code> (VPC, EKS, RDS, ECR, Secrets Manager, GuardDuty, CloudTrail, IAM), confirm security services, build and push images to ECR, patch <code>manifests/kustomization.yaml</code> with your registry and git SHA, and configure <code>kubectl</code> for EKS.</p>
<p><strong>Platform (7–12)</strong>: install ArgoCD; Kyverno + cluster policies, Falco, External Secrets Operator + IRSA service accounts, CSI secrets driver, and Stage 7 observability stack.</p>
<p><strong>Deploy (13–15)</strong>: ArgoCD app <code>clearledger-aws</code> syncs <code>stages/stage-8-aws-migration/manifests/</code>, wait for ALB hostname, and print URL and tear-down reminder.</p>
<p>After the script finishes, open the printed <code>http://&lt;alb-dns&gt;/</code> in your browser (ClearLedger login UI), or follow <a href="#heading-when-to-open-what-checkpoint-map">§8.3. When to open what</a> for Argo CD and Grafana port-forwards.</p>
<p>Default app deploy uses ESO for secrets. CSI is also installed so you can try file mounts in §8.5 without extra setup.</p>
<p><strong>Terraform layout</strong>: there's no <code>terraform.tf</code> file. The <code>terraform {}</code> block (version, providers, optional S3 backend) is at the top of <code>main.tf</code>. Resources are split by topic: <code>vpc.tf</code>, <code>eks.tf</code>, <code>rds.tf</code>, <code>ecr.tf</code>, <code>alb.tf</code>, <code>iam.tf</code>, <code>secrets.tf</code>, <code>security.tf</code>.</p>
<p>Run all commands from <code>stages/stage-8-aws-migration/terraform/</code>.</p>
<h3 id="heading-83-manual-walkthrough">8.3: Manual Walkthrough</h3>
<p>Go to <strong>Before you start</strong> in this section and run the manual steps from <strong>Step A</strong> yourself at least once instead of <code>make aws-up</code>. Paths are from the repo root.</p>
<p>Commands install things, while UIs prove they work. Homelab Stages 2 and 7 already taught you to open Argo CD and Grafana in a browser. Stage 8 is the same idea.</p>
<p>But on AWS there's no <code>clearledger.local</code> or <code>grafana.local</code> in <code>/etc/hosts</code>. You use port-forward for control-plane UIs and the public ALB hostname for the app.</p>
<h4 id="heading-when-to-open-what-checkpoint-map">When to open what (checkpoint map)</h4>
<p><code>make aws-up</code> runs fifteen steps. You don't need every UI open at once, just know when to look and what success looks like as the script moves along.</p>
<p>First, Terraform builds the AWS foundation. When step 2 finishes, open the <strong>AWS Console</strong> and confirm the cluster, registry, and database exist before any pods run: EKS <code>clearledger</code> is <strong>Active</strong>, ECR has four repos including <code>frontend</code> (empty is fine for now), and RDS <code>clearledger-postgres</code> is <strong>Available</strong>.</p>
<p>See <a href="#heading-aws-console-after-step-2">AWS Console (after step 2)</a> for the walkthrough.</p>
<p>Next come container images. After step 4, or after CI — AWS (ECR + OIDC) goes green in GitHub Actions, check ECR: each repo should list your git SHA tag. That's what ArgoCD will pull when the app deploys.</p>
<p>Around step 7 the script installs Argo CD. Port-forward to the UI and confirm the login page loads. You won't see the app yet. You're only checking that GitOps is reachable. Details: <a href="#heading-step-13-watch-argocd-sync-ui-cli">Argo CD UI</a>.</p>
<p>Step 12 adds observability. Port-forward to Grafana, log in, and confirm the six ClearLedger dashboards are listed. Panels can stay empty until you generate events. This is the same as Stage 7 on the homelab.</p>
<p>Step 13 applies the <code>clearledger-aws</code> app. Go back to Argo CD → <strong>Applications</strong> → <code>clearledger-aws</code>. You want Synced, Healthy, and running pods for auth, ledger, and notification.</p>
<p>Step 14 exposes the app on a public URL. Open <code>http://&lt;alb-dns&gt;/</code> in your browser: you should see the same ClearLedger login UI as homelab <code>clearledger.local</code>, served from the ALB with no <code>/etc/hosts</code> entry.<br>Use <code>/auth/health</code> and the other health URLs when you want a quick API check from the terminal.</p>
<p>See <a href="#heading-step-15-open-the-app-in-your-browser">ALB — first time the app is public</a>.</p>
<p>If you want extra confirmation, the optional check is <strong>EC2 → Load Balancers →</strong> <code>clearledger</code>: status <strong>Active</strong>, with healthy targets for frontend and the API services.</p>
<p>On AWS the app is four services behind one ALB: the frontend at <code>/</code> (login, dashboard, transactions) and the three APIs at <code>/auth</code>, <code>/ledger</code>, and <code>/notifications</code>.</p>
<p>Your portfolio screenshot for Stage 8 is the ALB URL showing the UI, like <code>http://clearledger-xxxxxxxxxx.eu-west-1.elb.amazonaws.com</code> with the ClearLedger login or dashboard visible.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/411a7ae0-8e7e-4a19-9267-e207f78ece93.png" alt="screenshot of clearledger ui with ALB Url" style="display: block;" width="1348" height="364" loading="lazy">

<p>For Argo CD and Grafana, keep a dedicated terminal running <code>kubectl port-forward</code> while the browser tab is open. <code>Ctrl+C</code> closes the tunnel.</p>
<h4 id="heading-before-you-start">Before you start</h4>
<p><strong>Step A: set real passwords in</strong> <code>secrets.tf</code></p>
<p>Open <code>stages/stage-8-aws-migration/terraform/secrets.tf</code> and search for the literal text <code>CHANGE_ME_BEFORE_APPLY</code>. It appears four times in the file (Postgres password, JWT secret, and two database URLs). Replace every occurrence:</p>
<ul>
<li><strong>Postgres password</strong>: pick a strong password (same value in all three places that reference it)</li>
<li><strong>JWT secret</strong>: run <code>openssl rand -base64 64</code> and paste the output</li>
</ul>
<p><code>make aws-up</code> will <strong>refuse to run</strong> if any <code>CHANGE_ME_BEFORE_APPLY</code> text is still in that file.</p>
<p><strong>Step B: terminal checks</strong></p>
<pre><code class="language-bash">aws sts get-caller-identity
terraform --version

# REQUIRED before first terraform apply; GitHub Actions OIDC (ci-aws.yaml) reads this at apply time:
cp stages/stage-8-aws-migration/terraform/terraform.tfvars.example \
   stages/stage-8-aws-migration/terraform/terraform.tfvars
# Edit terraform.tfvars: github_owner = "YOUR_GITHUB_USERNAME"   # your GitHub user or org, not a placeholder

terraform -chdir=stages/stage-8-aws-migration/terraform validate
# Fails with "Set github_owner in terraform.tfvars" until you replace YOUR_GITHUB_USERNAME
</code></pre>
<p><strong>Don't run</strong> <code>terraform apply</code> <strong>until</strong> <code>github_owner</code> <strong>is set.</strong> If you apply with the placeholder, AWS creates IAM role <code>clearledger-github-actions-ecr</code> with trust <code>repo:YOUR_GITHUB_USERNAME/...</code>. CI then fails at <strong>Publish images → ECR</strong> with <code>Not authorized to perform sts:AssumeRoleWithWebIdentity</code>.</p>
<p>Fix: edit <code>terraform.tfvars</code> → <code>terraform apply</code> again → verify with <code>aws iam get-role</code> below then <strong>Re-run failed jobs</strong> on the failed Actions run (not the full pipeline).</p>
<h4 id="heading-steps-12-terraform">Steps 1–2: Terraform</h4>
<pre><code class="language-bash">cd stages/stage-8-aws-migration/terraform
terraform init -upgrade
terraform apply

# Save outputs:
terraform output -raw ecr_registry_url
terraform output -raw github_actions_ecr_role_arn
terraform output -raw eso_role_arn
terraform output -raw auth_service_irsa_role_arn
terraform output -raw kubeconfig_command
cd ../../..
</code></pre>
<h4 id="heading-aws-console-after-step-2">AWS Console after step 2.</h4>
<p>Confirm Terraform created resources before you touch the cluster:</p>
<ol>
<li><strong>EKS</strong> → Clusters → <code>clearledger</code> → <strong>Status: Active</strong>, <strong>3 nodes</strong></li>
<li><strong>ECR</strong> → Repositories → <code>clearledger/auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, <code>frontend</code> (0 images until step 4 or CI)</li>
<li><strong>RDS</strong> → Databases → <code>clearledger-postgres</code> → <strong>Available</strong></li>
</ol>
<p><strong>Verify GitHub can push to ECR (only if you plan to use AWS CI later)</strong></p>
<p>GitHub Actions needs permission to push images to your AWS account. Terraform creates an IAM role for that, but only if you set your real GitHub username in <code>terraform.tfvars</code> before <code>terraform apply</code>.</p>
<p>Check it worked:</p>
<pre><code class="language-bash">aws iam get-role --role-name clearledger-github-actions-ecr \
  --query 'Role.AssumeRolePolicyDocument.Statement[0].Condition.StringEquals."token.actions.githubusercontent.com:sub"' \
  --output text
</code></pre>
<p><strong>Good:</strong> <code>repo:your-real-username/clearledger:environment:production</code></p>
<p><strong>Bad:</strong> <code>repo:YOUR_GITHUB_USERNAME/clearledger:...</code> you forgot to edit <code>terraform.tfvars</code>.</p>
<p>Fix the file, run <code>terraform apply</code> again, then in GitHub go to Actions and then the failed CI, AWS (ECR + OIDC) run → click Re-run failed jobs. That retries only the push step. You don't need to rebuild and rescan everything.</p>
<p>Skip this whole block if you are only using <code>make aws-up</code> for now and not enabling AWS CI yet.</p>
<p><strong>When do ECR repos appear?</strong></p>
<p>During <code>terraform apply</code> <strong>(step 2)</strong>, not when you <code>docker push</code>. Terraform creates <strong>empty</strong> image repositories: <code>clearledger/auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>, and <code>frontend</code>, so seeing 0 images right after apply is normal.</p>
<p>Images land later in step 4 (manual <code>docker push</code>) or when GitHub Actions CI succeeds.</p>
<p><strong>Set your AWS CLI region to</strong> <code>eu-west-1</code></p>
<p>Everything in this lab lives in eu-west-1 (Ireland). If your CLI defaults to <code>us-east-1</code>, commands will say resources are missing even though they exist:</p>
<pre><code class="language-bash">aws configure set region eu-west-1
aws configure get region   # expect: eu-west-1
</code></pre>
<h4 id="heading-steps-34-security-services-ecr-images">Steps 3–4: Security services + ECR images</h4>
<pre><code class="language-bash">AWS_REGION=eu-west-1   # or rely on aws configure set region above

# Step 3: verify security services (must pass --region eu-west-1)
aws guardduty list-detectors --region "${AWS_REGION}"
# Expect: DetectorIds: ["&lt;id&gt;"]  — empty [] means wrong region, not "not created"

aws cloudtrail get-trail-status --name clearledger-trail --region "${AWS_REGION}"
# Expect: IsLogging: true
# Error "Unknown trail ... us-east-1" → you forgot --region eu-west-1

# Step 4: build and push images to the ECR repos Terraform already created
ECR_REGISTRY=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ecr_registry_url)
AUTH_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw auth_service_ecr_url)
LEDGER_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ledger_service_ecr_url)
NOTIFY_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw notification_service_ecr_url)
TAG=$(git rev-parse --short HEAD)

aws ecr get-login-password --region "${AWS_REGION}" \
  | docker login --username AWS --password-stdin "${ECR_REGISTRY}"

docker build -t "${AUTH_ECR}:${TAG}" app/auth-service &amp;&amp; docker push "${AUTH_ECR}:${TAG}"
docker build -t "${LEDGER_ECR}:${TAG}" app/ledger-service &amp;&amp; docker push "${LEDGER_ECR}:${TAG}"
docker build -t "${NOTIFY_ECR}:${TAG}" app/notification-service &amp;&amp; docker push "${NOTIFY_ECR}:${TAG}"

# Confirm images landed (optional)
aws ecr describe-images --repository-name clearledger/auth-service --region "${AWS_REGION}" \
  --query 'imageDetails[*].imageTags' --output table
</code></pre>
<p><strong>ECR console (after step 4 or green CI)</strong>: open each repository and go. tothe Images tab. You should see tags matching your git commit SHA. If repos are empty, ArgoCD will show <code>ImagePullBackOff</code> later.</p>
<p><strong>GitHub Actions (if using CI instead of manual push)</strong>: repo → Actions → workflow CI. AWS (ECR + OIDC).</p>
<p>If all jobs are green, publish the images. ECR succeeded. This is the supply-chain proof before deploy.</p>
<h4 id="heading-step-5-gitops-source-of-truth">Step 5: GitOps source of truth</h4>
<p>Patch placeholders in <code>kustomization.yaml</code> (same <code>sed</code> as <code>aws-spinup.sh</code> step 5):</p>
<pre><code class="language-bash">AWS_REGION=eu-west-1
ECR_REGISTRY=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ecr_registry_url)
TAG=$(git rev-parse --short HEAD)
KUST=stages/stage-8-aws-migration/manifests/kustomization.yaml

sed -i.bak \
  -e "s|REPLACE_ECR_REGISTRY|${ECR_REGISTRY}|g" \
  -e "s|REPLACE_IMAGE_TAG|${TAG}|g" \
  "${KUST}"
rm -f "${KUST}.bak"

# Region in ESO + CSI manifests (only if not eu-west-1)
if [[ "${AWS_REGION}" != "eu-west-1" ]]; then
  sed -i.bak "s|region: eu-west-1|region: ${AWS_REGION}|g" \
    stages/stage-8-aws-migration/manifests/external-secrets.yaml \
    stages/stage-8-aws-migration/manifests/csi/auth-service-spc.yaml \
    stages/stage-8-aws-migration/manifests/csi/ledger-service-spc.yaml
  rm -f stages/stage-8-aws-migration/manifests/external-secrets.yaml.bak \
        stages/stage-8-aws-migration/manifests/csi/*.bak 2&gt;/dev/null || true
fi

# Verify before commit
grep -E 'newName:|newTag:' "${KUST}"
# Expect: YOUR_AWS_ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/clearledger/... and your git SHA

git add stages/stage-8-aws-migration/manifests/kustomization.yaml
git commit -m "stage8: ECR images ${TAG}"
git push
</code></pre>
<p>Also fix the ArgoCD Application repo URL once (replace with your GitHub username):</p>
<pre><code class="language-bash"># Example: YOUR_GITHUB_USERNAME/clearledger — check: git remote get-url origin
sed -i.bak 's|YOUR_GITHUB_USERNAME|YOUR_ACTUAL_GITHUB_USER|g' \
  stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml
rm -f stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml.bak
</code></pre>
<h4 id="heading-step-6-cluster-access-terraform-outputs">Step 6: Cluster access + Terraform outputs</h4>
<p>Run from the repo root. Set the CLI region first (EKS and IAM outputs are regional), then kubeconfig, then export IRSA role ARNs: steps 9–10 need them.</p>
<pre><code class="language-bash">aws configure set region eu-west-1

eval "$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw kubeconfig_command)"
kubectl get nodes

export AWS_REGION=eu-west-1
export ESO_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw eso_role_arn)
export FALCO_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw falco_role_arn)
export REPLACE_AUTH_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw auth_service_irsa_role_arn)
export REPLACE_LEDGER_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ledger_service_irsa_role_arn)
export REPLACE_NOTIFICATION_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw notification_service_irsa_role_arn)

# Sanity check (all should print ARNs, not empty)
echo "ESO:      ${ESO_ROLE_ARN}"
echo "Falco:    ${FALCO_ROLE_ARN}"
echo "Auth IRSA: ${REPLACE_AUTH_IRSA_ROLE_ARN}"
</code></pre>
<h4 id="heading-steps-712-platform-stack-on-the-cluster">Steps 7–12: Platform stack on the cluster</h4>
<p>You finished steps 1–6 (AWS exists, images in ECR, <code>kubectl</code> works). Now for steps 7-12 you'll install the platform stack, the same components as <code>aws-spinup.sh</code>, but you run the commands from the sections below, not the script.</p>
<p>For each step, run the Install code block, then run the Verify block right under it. Don't move to the next step until you see Running pods (or a ClusterPolicy list). “Command finished with no output” isn't enough.</p>
<table>
<thead>
<tr>
<th>Step</th>
<th>Namespace</th>
<th>What you are installing</th>
<th>Rough pod count</th>
</tr>
</thead>
<tbody><tr>
<td>7</td>
<td><code>argocd</code></td>
<td>GitOps controller</td>
<td>~7 pods</td>
</tr>
<tr>
<td>8</td>
<td><code>kyverno</code></td>
<td>Admission policies</td>
<td>~4 pods + ClusterPolicies</td>
</tr>
<tr>
<td>9</td>
<td><code>falco</code></td>
<td>Runtime detection</td>
<td>1 DaemonSet pod <strong>per node</strong> (3 on this cluster)</td>
</tr>
<tr>
<td>10</td>
<td><code>external-secrets</code> + <code>clearledger</code></td>
<td>ESO + IRSA ServiceAccounts</td>
<td>~3 ESO pods + 3 ServiceAccounts</td>
</tr>
<tr>
<td>11</td>
<td><code>kube-system</code> + <code>clearledger</code></td>
<td>CSI driver + AWS provider</td>
<td>3 driver + 3 provider (one per node)</td>
</tr>
<tr>
<td>12</td>
<td><code>monitoring</code></td>
<td>Prometheus, Grafana, Loki</td>
<td>~10+ pods</td>
</tr>
</tbody></table>
<p>Steps 13–15 (deploy app, wait for ALB, verify UI) come after step 12 below.</p>
<h4 id="heading-step-7-argocd">Step 7: ArgoCD</h4>
<pre><code class="language-bash">kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl rollout status deployment/argocd-server -n argocd --timeout=180s
</code></pre>
<p><strong>Verify what got created:</strong></p>
<pre><code class="language-bash">kubectl get pods -n argocd
kubectl get svc -n argocd
kubectl get deploy -n argocd
</code></pre>
<p><strong>Expected:</strong> <code>argocd-server</code>, <code>argocd-repo-server</code>, <code>argocd-application-controller</code>, and so on: most pods <strong>Running</strong> <strong>1/1</strong> or <strong>2/2</strong>. <code>argocd-server</code> Service exposes port 443.</p>
<p><strong>UI (optional now, required after step 13):</strong> new terminal, leave running. Use any free local port (<code>8081</code> if <code>8080</code> is in use):</p>
<pre><code class="language-bash">kubectl port-forward svc/argocd-server -n argocd 8080:443
# Or if 8080 is taken:
# kubectl port-forward svc/argocd-server -n argocd 8081:443
# https://localhost:8080 (or 8081)  user: admin
kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d; echo
</code></pre>
<p>Applications list is empty until step 13. That's normal.</p>
<h4 id="heading-step-8-kyverno-policies">Step 8: Kyverno + policies</h4>
<p><code>cosign.pub</code> / <code>infra/cosign.pub</code> are gitignored (private key must never commit; public key is learner-specific).<br>The repo ships example keys in <code>require-signed-images.yaml</code> / <code>require-signed-images-ecr.yaml</code>.<br>If you regenerated keys in Stage 3, sync your local public key into policies before apply:</p>
<pre><code class="language-bash"># infra/cosign.pub exists locally but is gitignored — safe to copy into committed policy YAMLs
bash scripts/embed-cosign-pub-in-policies.sh
diff infra/cosign.pub &lt;(grep -A3 'BEGIN PUBLIC KEY' infra/policies/require-signed-images-ecr.yaml | grep -v publicKeys)
</code></pre>
<pre><code class="language-bash">helm repo add kyverno https://kyverno.github.io/kyverno/ --force-update
helm upgrade --install kyverno kyverno/kyverno \
  --namespace kyverno --create-namespace \
  -f stages/stage-4-admission-control/infra/kyverno/values.yaml \
  --set admissionController.replicas=1 \
  --wait --timeout=180s
kubectl apply -f infra/policies/
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pods -n kyverno
kubectl get clusterpolicy
kubectl get clusterpolicy require-signed-images-ecr -o jsonpath='{.spec.rules[0].verifyImages[0].attestors[0].entries[0].keys.publicKeys}' | head -3
</code></pre>
<p><strong>Expected:</strong> admission-controller, background-controller, cleanup-controller, reports-controller pods Running.</p>
<p><code>kubectl get clusterpolicy</code> lists 6+ policies including <code>require-signed-images-ecr</code>, <code>disallow-root-containers</code>, and so on. The <code>publicKeys</code> output must show <code>-----BEGIN PUBLIC KEY-----</code>, not <code>PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE</code> (Kyverno treats a placeholder as a file path and blocks all deploys).</p>
<p><code>require-signed-images-ecr</code> defaults to Audit until CI Cosign-signs ECR images (<code>COSIGN_PRIVATE_KEY</code> + <code>COSIGN_PASSWORD</code> in GitHub). Unsigned images still deploy. Signed-image enforcement is optional later.</p>
<p>If <code>verify-slsa-provenance</code> fails to apply (Audit + <code>mutateDigest</code>), set <code>mutateDigest: false</code> in that file, or skip it. It's optional for Stage 8.</p>
<h4 id="heading-step-9-falco">Step 9: Falco</h4>
<pre><code class="language-bash">helm repo add falcosecurity https://falcosecurity.github.io/charts --force-update
helm upgrade --install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  -f stages/stage-6-runtime-security/infra/falco/helm-values.yaml \
  --set driver.kind=modern_ebpf \
  --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${FALCO_ROLE_ARN}" \
  --wait --timeout=300s
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pods -n falco -o wide
kubectl get daemonset -n falco
kubectl get sa falco -n falco -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'; echo
</code></pre>
<p><strong>Expected:</strong> Falco DaemonSet with DESIRED = number of nodes (3). Each pod <strong>Running</strong>. ServiceAccount annotation shows your <code>FALCO_ROLE_ARN</code>.</p>
<h4 id="heading-step-10-external-secrets-operator-irsa-serviceaccounts">Step 10: External Secrets Operator + IRSA ServiceAccounts</h4>
<pre><code class="language-bash">helm repo add external-secrets https://charts.external-secrets.io --force-update
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${ESO_ROLE_ARN}" \
  --wait --timeout=180s
kubectl apply -f stages/stage-8-aws-migration/manifests/resources/namespace.yaml
envsubst &lt; stages/stage-8-aws-migration/manifests/clearledger-serviceaccounts.yaml | kubectl apply -f -
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pods -n external-secrets
kubectl get sa -n external-secrets external-secrets -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'; echo
kubectl get sa -n clearledger
</code></pre>
<p><strong>Expected:</strong> <code>external-secrets</code> deployment <strong>Running</strong> (often 3 containers / 1 pod). Three ServiceAccounts in <code>clearledger</code>: <code>auth-service</code>, <code>ledger-service</code>, <code>notification-service</code>: each with an <code>eks.amazonaws.com/role-arn</code> annotation. No app pods yet (ArgoCD deploys those in step 13).</p>
<p><strong>Step 11: CSI driver + SecretProviderClasses</strong></p>
<pre><code class="language-bash">bash stages/stage-8-aws-migration/scripts/install-csi-secrets.sh
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pods -n kube-system | grep -E 'secrets-store|provider-aws'
kubectl get secretproviderclass -n clearledger
helm list -n kube-system | grep -E 'csi-secrets|secrets-provider'
</code></pre>
<p><strong>Expected:</strong> CSI driver pods <strong>3/3 Running</strong> (one per node). AWS provider pods <strong>1/1 Running</strong> per node. Two <code>SecretProviderClass</code> objects in <code>clearledger</code>. Helm shows <code>csi-secrets-store</code> and/or <code>secrets-provider-aws</code> <strong>deployed</strong>.</p>
<p>If Helm reports <code>meta.helm.sh/release-name</code> conflicts, re-run the script. It installs the AWS provider without duplicating the driver chart.</p>
<h4 id="heading-step-12-observability">Step 12: Observability</h4>
<pre><code class="language-bash">bash stages/stage-7-observability/scripts/install-observability.sh
</code></pre>
<p><strong>Verify:</strong></p>
<pre><code class="language-bash">kubectl get pods -n monitoring
kubectl get svc -n monitoring | grep -E 'grafana|prometheus|loki'
kubectl get configmap -n monitoring -l grafana_dashboard=1 --no-headers | wc -l
</code></pre>
<p><strong>Expected:</strong> Grafana <strong>3/3 Running</strong>, Prometheus and Loki pods <strong>Running</strong>. ConfigMap count for dashboards is <strong>6</strong> (ClearLedger dashboards). Script prints <code>http://grafana.local</code>: on EKS use port-forward instead:</p>
<pre><code class="language-bash"># New terminal — keep running
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80
# http://localhost:3000  admin / admin123
# http://localhost:3000/dashboards?tag=clearledger
</code></pre>
<p>Panels may show <strong>No data</strong> until you trigger events (§7.4 exercises work on this cluster too).</p>
<p><strong>Platform stack summary</strong>: quick sanity check before step 13:</p>
<pre><code class="language-bash">for ns in argocd kyverno falco external-secrets monitoring clearledger; do
  echo "=== ${ns} ==="
  kubectl get pods -n "${ns}" --no-headers 2&gt;/dev/null | awk '{print $3}' | sort | uniq -c || echo "(no pods yet)"
done
kubectl get clusterpolicy --no-headers | wc -l | xargs echo "ClusterPolicies:"
kubectl get secretproviderclass -n clearledger --no-headers | wc -l | xargs echo "SecretProviderClasses:"
</code></pre>
<p><strong>Expected:</strong> every namespace shows only <code>Running</code> (or <code>Completed</code> for jobs). <code>clearledger</code> may be empty until ArgoCD syncs. ClusterPolicies ≥ 6. SecretProviderClasses = 2.</p>
<p><strong>EKS API timeout on namespace create?</strong> You may see <code>Unexpected error when reading response body</code> / <code>context deadline exceeded</code> and still get <code>namespace/argocd created</code>. That's a <strong>transient client timeout</strong> talking to the EKS API (first request, slow network, or control plane catching up), not a failed create. Confirm with <code>kubectl get namespace argocd</code> and continue. If commands keep timing out, retry once or run <code>kubectl cluster-info</code> to verify connectivity.</p>
<h4 id="heading-steps-1314-deploy-via-argocd-see-the-alb">Steps 13–14: Deploy via ArgoCD + see the ALB</h4>
<p>The app YAMLs under <code>stages/stage-8-aws-migration/manifests/</code> aren't applied by hand. Step 13 tells Argo CD to sync Git. Argo CD then creates Deployments, Services, Ingress, and the rest.</p>
<p><strong>Repo access first</strong></p>
<p>If your GitHub repo is private, add a PAT in Argo CD → Settings → Repositories. If you made the repo public, refresh the app, <code>ComparisonError: authentication required</code> should clear.</p>
<p><strong>If sync still fails</strong>, check the usual causes:</p>
<ul>
<li><code>external-secrets.io/v1beta1</code> <strong>not found</strong>, your cluster has a newer ESO API. Push <code>external-secrets.yaml</code> with <code>apiVersion: external-secrets.io/v1</code>.</li>
<li><strong>Kyverno complains about</strong> <code>PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE</code> , run <code>bash scripts/embed-cosign-pub-in-policies.sh</code>, then <code>kubectl apply -f infra/policies/</code>.</li>
<li><code>SecretSyncedError</code> <strong>on auth,</strong> <code>database_url</code> <strong>or</strong> <code>jwt_secret</code> <strong>not found</strong> — the AWS secret <code>clearledger/auth-service</code> must contain both keys (Terraform writes them in <code>secrets.tf</code>). Re-run <code>terraform apply</code> after fixing <code>CHANGE_ME_BEFORE_APPLY</code> values, or check the secret in the AWS console.</li>
<li><strong>Pods stuck</strong> <code>Pending</code> <strong>or “too many pods”</strong> the lab nodes are small. Scale the node group in Terraform or lower replica counts in the manifests.</li>
</ul>
<p>Register the app:</p>
<pre><code class="language-bash">kubectl apply -f stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml
</code></pre>
<p>Watch Argo CD until <code>clearledger-aws</code> is <strong>Synced</strong> and <strong>Healthy</strong>. That's when app pods appear in <code>clearledger</code>.</p>
<h4 id="heading-step-13-watch-argocd-sync-ui-cli">Step 13: Watch ArgoCD sync (UI + CLI)</h4>
<p>Open the Argo CD browser tab you kept open (port-forward from step 7).</p>
<pre><code class="language-plaintext">https://localhost:8080        ← or 8081 if 8080 was busy
</code></pre>
<p>Click <code>clearledger-aws</code>. Wait for <strong>Healthy + Synced</strong> (2–5 minutes on first deploy). You can watch the same info from the terminal without touching the browser:</p>
<pre><code class="language-bash">kubectl get application clearledger-aws -n argocd -w
# Ctrl-C when HEALTH STATUS shows Healthy
</code></pre>
<p>While that's settling, watch pods start up in a second terminal:</p>
<pre><code class="language-bash">kubectl get pods -n clearledger -w
# All pods should reach 1/1 Running within 2 minutes
# Ctrl-C when everything is Running
</code></pre>
<h4 id="heading-step-14-get-your-public-app-url-alb">Step 14: Get your public app URL (ALB)</h4>
<p>AWS takes 2–5 minutes after ArgoCD syncs to provision the load balancer.<br>Run this and wait until the ADDRESS column fills in:</p>
<pre><code class="language-bash">kubectl get ingress clearledger-ingress -n clearledger -w
# ADDRESS is empty at first, then shows something like:
# clearledger-xxxxxxxxxx.eu-west-1.elb.amazonaws.com
# Ctrl-C once the hostname appears
</code></pre>
<p>Export the URL for the steps below:</p>
<pre><code class="language-bash">export ALB_DNS=$(kubectl get ingress clearledger-ingress -n clearledger \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "Your app is live at: http://${ALB_DNS}"
</code></pre>
<p><strong>Still empty after 10 minutes?</strong> See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a> for ALB/ingress recovery steps.</p>
<h4 id="heading-step-15-open-the-app-in-your-browser">Step 15: Open the app in your browser</h4>
<p>Paste the ALB root URL into your browser. No DNS entry, port-forward, or VPN:</p>
<pre><code class="language-plaintext">http://clearledger-xxxxxxxxxx.eu-west-1.elb.amazonaws.com/
</code></pre>
<p>You should see the ClearLedger login screen (same SPA as homelab <code>clearledger.local</code>). Register or log in, submit a transaction, and confirm the dashboard loads. That's your Stage 8 portfolio screenshot.</p>
<p><strong>Quick API health checks</strong> (terminal or browser):</p>
<pre><code class="language-bash">curl -fsS "http://${ALB_DNS}/auth/health" &amp;&amp; echo
curl -fsS "http://${ALB_DNS}/ledger/health" &amp;&amp; echo
curl -fsS "http://${ALB_DNS}/notifications/health" &amp;&amp; echo
</code></pre>
<p>Each should return JSON like <code>{"status":"ok","service":"auth-service"}</code>.</p>
<h4 id="heading-step-16-verify-in-the-aws-console-optional-but-recommended">Step 16: Verify in the AWS Console (optional but recommended)</h4>
<p>This is what the deployed stack looks like from AWS side:</p>
<table>
<thead>
<tr>
<th>Console location</th>
<th>What to look for</th>
</tr>
</thead>
<tbody><tr>
<td><strong>EC2 → Load Balancers</strong></td>
<td>A load balancer named <code>clearledger-…</code> with state <strong>Active</strong></td>
</tr>
<tr>
<td><strong>EC2 → Target Groups</strong></td>
<td>Two or three target groups, all targets showing <strong>healthy</strong></td>
</tr>
<tr>
<td><strong>ECR → Repositories</strong></td>
<td><code>clearledger/auth-service</code>, <code>clearledger/ledger-service</code>, <code>clearledger/notification-service</code>, <code>clearledger/frontend</code> — each with a recently pushed image tag</td>
</tr>
<tr>
<td><strong>EKS → Clusters → clearledger → Workloads</strong></td>
<td>Your pods shown as Running in the <code>clearledger</code> namespace</td>
</tr>
<tr>
<td><strong>Secrets Manager</strong></td>
<td><code>clearledger/auth-service</code>, <code>clearledger/ledger-service</code>, <code>clearledger/postgres</code> — all present</td>
</tr>
</tbody></table>
<p><strong>502/503 from the ALB?</strong> The load balancer is up but the pods aren't healthy yet, or the secrets haven't synced. Check: <code>kubectl get pods -n clearledger</code> (all <code>1/1 Running</code>?) and <code>kubectl get externalsecret -n clearledger</code> (both <code>SecretSynced True</code>?).</p>
<p><strong>✋ Hands-on checkpoint: app is publicly reachable</strong></p>
<pre><code class="language-bash"># All three must print {"status":"ok",...}
curl -fsS "http://${ALB_DNS}/auth/health"         &amp;&amp; echo
curl -fsS "http://${ALB_DNS}/ledger/health"        &amp;&amp; echo
curl -fsS "http://${ALB_DNS}/notifications/health" &amp;&amp; echo

# All pods Running
kubectl get pods -n clearledger

# Nothing printed here = all pods Running (non-Running pods would show)
kubectl get pods -n clearledger --field-selector=status.phase!=Running
</code></pre>
<p><code>ImagePullBackOff</code> in the pod list means ECR images aren't there yet. Check GitHub Actions and re-run the workflow. A <code>502</code> from the health URL means the pod isn't ready yet. Wait 30 seconds and retry.</p>
<h3 id="heading-84-verify-eso-default-secret-path">8.4: Verify ESO (Default Secret Path)</h3>
<p>After Argo CD syncs, confirm External Secrets Operator copied values from AWS Secrets Manager into normal Kubernetes Secrets:</p>
<pre><code class="language-bash">kubectl get externalsecret,secret -n clearledger
kubectl describe externalsecret auth-service-secret -n clearledger | grep -A6 "Conditions:"
kubectl get pods -n clearledger -l app=auth-service
kubectl exec -n clearledger deploy/auth-service -c auth-service -- env | grep DATABASE_URL
</code></pre>
<h4 id="heading-command-1-externalsecrets-secrets">Command 1: ExternalSecrets + Secrets</h4>
<p>You should see two ExternalSecrets and two matching Secrets (auth has 2 keys, ledger has 1):</p>
<pre><code class="language-plaintext">NAME                                                     STORE                 REFRESH INTERVAL   STATUS         READY
externalsecret.external-secrets.io/auth-service-secret   aws-secrets-manager   1h                 SecretSynced   True
externalsecret.external-secrets.io/ledger-service-secret aws-secrets-manager   1h                 SecretSynced   True

NAME                         TYPE     DATA   AGE
secret/auth-service-secret   Opaque   2      3m
secret/ledger-service-secret Opaque   1      3m
</code></pre>
<p><code>STATUS</code> must be <strong>SecretSynced</strong> and <strong>READY</strong> must be <strong>True</strong>. If you see <code>SecretSyncedError</code>, stop here and fix IRSA before §8.5.</p>
<h4 id="heading-command-2-describe-auth-externalsecret">Command 2: describe auth ExternalSecret</h4>
<p>Look for <code>Reason: SecretSynced</code> and <code>Status: True</code>:</p>
<pre><code class="language-plaintext">  Conditions:
    Last Transition Time:   2026-07-10T22:15:00Z
    Message:                Secret was synced
    Reason:                 SecretSynced
    Status:                 True
    Type:                   Ready
</code></pre>
<h4 id="heading-command-3-auth-pods-running">Command 3: auth pods running</h4>
<pre><code class="language-plaintext">NAME                            READY   STATUS    RESTARTS   AGE
auth-service-xxxxxxxxxx-xxxxx   1/1     Running   0          2m
auth-service-xxxxxxxxxx-xxxxx   1/1     Running   0          2m
</code></pre>
<p>Both replicas <strong>1/1 Running</strong>. If pods are <code>CrashLoopBackOff</code> or <code>CreateContainerConfigError</code>, the K8s Secret may be missing or empty.</p>
<h4 id="heading-command-4-databaseurl-is-an-env-var-eso-path-not-a-file-path">Command 4: DATABASE_URL is an env var (ESO path), not a file path</h4>
<pre><code class="language-plaintext">DATABASE_URL=postgresql://clearledger:*****@clearledger-postgres.xxxxx.eu-west-1.rds.amazonaws.com:5432/clearledger
</code></pre>
<p>Good: a <code>postgresql://...</code> connection string (password shown as <code>*****</code> or your real password).</p>
<p>Bad for this section: <code>/mnt/secrets/database_url</code> that means CSI file mounts (§8.5), not the default ESO env-var path.</p>
<p>You can also spot-check the secret exists without printing values:</p>
<pre><code class="language-bash">kubectl get secret auth-service-secret -n clearledger -o jsonpath='{.data}' | grep -o 'database_url\|jwt_secret'
# Expect: database_url and jwt_secret (two keys)
</code></pre>
<p>If <code>SecretSynced=False</code>, check ESO logs and IRSA:</p>
<pre><code class="language-bash">kubectl logs -n external-secrets deploy/external-secrets -c external-secrets | tail -30
kubectl get sa auth-service -n clearledger -o yaml | grep role-arn
</code></pre>
<p><strong>✋ Hands-on checkpoint. External Secrets actually synced from AWS</strong></p>
<pre><code class="language-bash">kubectl get externalsecret -n clearledger
kubectl get secret -n clearledger
</code></pre>
<p>Expected: <code>auth-service-secret</code> and <code>ledger-service-secret</code> each show <code>SecretSynced</code> / Ready <code>True</code>. The matching Kubernetes Secrets exist in <code>clearledger</code>. A <code>SecretSyncedError</code> means IRSA/IAM can't reach Secrets Manager: fix the role binding before §8.5.</p>
<p>If you skip this, §8.5 (CSI driver) builds on working secret access, and a silent IAM failure here surfaces as an unrelated-looking pod error two sections later.</p>
<h3 id="heading-85-hands-on-csi-driver-file-mounts">8.5: Hands-on, CSI Driver (File Mounts)</h3>
<p>The default pods already use ESO: secrets arrive as environment variables from a Kubernetes Secret object. This exercise switches <code>auth-service</code> to the CSI path instead: secrets are mounted as plain files under <code>/mnt/secrets/</code>, and the app reads them from disk. It's the same code path the homelab uses with Vault (<code>DATABASE_URL_FILE</code> / <code>JWT_SECRET_FILE</code>).</p>
<p>CSI was already installed at spinup step 11, so there's nothing extra to install.</p>
<h4 id="heading-step-1-confirm-csi-is-running">Step 1: Confirm CSI is running</h4>
<pre><code class="language-bash">kubectl get pods -n kube-system -l app=secrets-store-csi-driver
kubectl get secretproviderclass -n clearledger
</code></pre>
<p>You should see one CSI driver pod per node, and two <code>SecretProviderClass</code> objects: one for auth-service and one for ledger-service.</p>
<h4 id="heading-step-2-swap-the-deployment-in-git">Step 2: swap the deployment in Git</h4>
<p>Open <code>stages/stage-8-aws-migration/manifests/kustomization.yaml</code> and change one line:</p>
<pre><code class="language-yaml"># Before
  - deployments/auth-service.yaml

# After
  - deployments/auth-service-csi.yaml
</code></pre>
<p>Commit and push, then sync:</p>
<pre><code class="language-bash">argocd app sync clearledger-aws
kubectl rollout status deployment/auth-service -n clearledger
</code></pre>
<p>ArgoCD will roll out a new auth-service pod with the CSI volume attached.</p>
<h4 id="heading-step-3-confirm-the-files-are-there">Step 3: Confirm the files are there</h4>
<pre><code class="language-bash"># Find the new pod
kubectl get pod -n clearledger -l secrets=csi

# List the mounted secret files
kubectl exec -n clearledger deploy/auth-service -- ls /mnt/secrets

# Check the database URL was written correctly
kubectl exec -n clearledger deploy/auth-service -- cat /mnt/secrets/database_url

# Confirm the service is still healthy
curl -s "http://$(kubectl get ingress clearledger-ingress -n clearledger \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')/auth/health"
</code></pre>
<p>You should see <code>database_url</code> and <code>jwt_secret</code> listed as files, and the health check should return <code>{"status":"ok"}</code>.</p>
<p><strong>ESO vs CSI: what actually changed?</strong></p>
<p>Both paths read the same passwords from AWS Secrets Manager. Only the delivery method changes.</p>
<p><strong>ESO (default, what you verified in §8.4)</strong></p>
<p>Think of ESO as a copy clerk that runs in the cluster:</p>
<ol>
<li>ESO has its own AWS permission (IAM role).</li>
<li>It reads <code>clearledger/auth-service</code> from Secrets Manager.</li>
<li>It copies the values into a normal Kubernetes Secret named <code>auth-service-secret</code>.</li>
<li>The auth pod reads <code>DATABASE_URL</code> and <code>JWT_SECRET</code> as <strong>environment variables.</strong></li>
</ol>
<p>The password lives briefly inside the cluster as a Kubernetes Secret object.</p>
<p><strong>CSI (this exercise, file mounts)</strong></p>
<p>Think of CSI as the pod picking up secrets itself when it starts:</p>
<ol>
<li>The auth-service pod has its own AWS permission (IRSA on its ServiceAccount).</li>
<li>When the pod starts, the CSI driver asks Secrets Manager for the values.</li>
<li>They appear as files under <code>/mnt/secrets/</code> (<code>database_url</code>, <code>jwt_secret</code>).</li>
<li>The pod is told <code>DATABASE_URL_FILE=/mnt/secrets/database_url</code> , it reads from disk, not from a copied K8s Secret.</li>
</ol>
<p>No Kubernetes Secret copy is created for those values on this path.</p>
<p><strong>Why does the same app code work for both?</strong></p>
<p><code>app/auth-service/main.py</code> uses a small helper <code>_read_secret()</code>:</p>
<ul>
<li>If <code>DATABASE_URL_FILE</code> points to a file that exists → read the file (CSI or homelab Vault).</li>
<li>Otherwise → read <code>DATABASE_URL</code> directly (ESO / Stage 0–4).</li>
</ul>
<p>Same image, same code: you only change which deployment YAML Argo CD syncs.</p>
<p><strong>To switch back to ESO:</strong> in <code>kustomization.yaml</code>, change <code>auth-service-csi.yaml</code> back to <code>auth-service.yaml</code>, commit, push, and <code>argocd app sync clearledger-aws</code>.</p>
<p><strong>Terraform</strong> provisions all the AWS resources (VPC, EKS, RDS, ECR, Secrets Manager, and IAM roles) from <code>.tf</code> files in <code>stages/stage-8-aws-migration/terraform/</code>.</p>
<h3 id="heading-two-oidc-ideas-in-stage-8">Two OIDC Ideas in Stage 8</h3>
<p>Stage 8 uses OIDC in two different places. They sound similar, but they solve different problems.</p>
<p><strong>GitHub Actions OIDC</strong> lets the CI pipeline push images to ECR without storing long-lived AWS keys in GitHub. When a job runs, GitHub mints a short-lived token that proves the job's identity. AWS trusts that token and hands back temporary credentials: enough to push images and nothing else.</p>
<p><strong>IRSA</strong> does the same thing, but for pods running inside EKS. Instead of a GitHub token, the pod presents its Kubernetes ServiceAccount token. AWS trusts the EKS cluster's OIDC provider, verifies the token, and returns temporary credentials scoped to exactly what that pod needs.</p>
<p>It helps to see what each one says:</p>
<pre><code class="language-text">GitHub Actions OIDC:
  Pipeline says → "I am a job in the production environment of YOUR_USERNAME/clearledger"
  AWS replies   → "Here are credentials to push to ECR, valid for one hour"

IRSA:
  Pod says   → "I am the auth-service ServiceAccount in the clearledger namespace"
  AWS replies → "Here are credentials to read only the auth-service secret, valid for one hour"
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/ac83e9bf-dbfd-42b3-bba8-0bbf327b03c5.png" alt="Image flow diagram showing difference between OIDC and IRSA" style="display: block;" width="1536" height="1024" loading="lazy">

<p>The key is what is <em>not</em> stored anywhere:</p>
<pre><code class="language-text">No AWS_ACCESS_KEY_ID in GitHub Secrets
No AWS_SECRET_ACCESS_KEY in GitHub Secrets
No AWS keys inside Kubernetes Secrets
</code></pre>
<p>Terraform creates the role <code>clearledger-github-actions-ecr</code> and wires up the trust policies for both. The pipeline in <code>.github/workflows/ci-aws.yaml</code> assumes that role, pushes images to ECR, and updates <code>kustomization.yaml</code>. ArgoCD picks up the change and deploys the new images.</p>
<h3 id="heading-ci-routing-stages-17-vs-stage-8">CI Routing: Stages 1–7 vs Stage 8</h3>
<p>The repo ships two workflow files. You don't need both running at the same time.</p>
<p><code>ci.yaml</code> is the homelab pipeline from Stages 1–7. It runs on your self-hosted Multipass VM, pushes images to Docker Hub, and updates your <code>clearledger-infra</code> GitOps repo. This is the default: nothing to configure.</p>
<p><code>ci-aws.yaml</code> is the AWS pipeline for Stage 8. It runs on GitHub-hosted <code>ubuntu-latest</code> runners, pushes images to ECR, and updates <code>kustomization.yaml</code> directly in this repo. It only activates when you set the repo variable <code>CLEARLEDGER_CI_TARGET=aws</code>.</p>
<p><strong>If you're on Stages 1–7, do nothing.</strong> The <code>CLEARLEDGER_CI_TARGET</code> variable is unset by default, so every push runs <code>ci.yaml</code> on your self-hosted runner as normal. The AWS workflow file exists in the repo but its jobs are skipped.</p>
<p><strong>Do not set</strong> <code>CLEARLEDGER_CI_TARGET=aws</code> <strong>until your EKS cluster is running.</strong></p>
<p>If you set it early, <code>ci.yaml</code> stops running on push (no more Docker Hub builds), and <code>ci-aws.yaml</code> will fail immediately because there's no ECR, no OIDC role, and no AWS infrastructure yet. If you accidentally set it, delete the variable: GitHub → repo <strong>Settings</strong> → <strong>Secrets and variables</strong> → <strong>Actions</strong> → <strong>Variables</strong> → delete <code>CLEARLEDGER_CI_TARGET</code>.</p>
<p><strong>Enabling AWS CI (do this after</strong> <code>terraform apply</code> <strong>completes)</strong></p>
<p>You need three repository variables and one secret in a <code>production</code> environment.</p>
<p>First, set the variables: replace <code>YOUR_USERNAME</code> with your GitHub username:</p>
<pre><code class="language-bash">gh variable set CLEARLEDGER_CI_TARGET --body aws --repo YOUR_USERNAME/clearledger

gh variable set AWS_ACCOUNT_ID --body "$(aws sts get-caller-identity --query Account --output text)" --repo YOUR_USERNAME/clearledger

gh variable set AWS_REGION --body eu-west-1 --repo YOUR_USERNAME/clearledger
</code></pre>
<p>Then create the <code>production</code> environment and add the OIDC role ARN as a secret:</p>
<pre><code class="language-bash"># Create the environment first, gh secret set returns 404 if it does not exist
gh api --method PUT "repos/YOUR_USERNAME/clearledger/environments/production"

gh secret set AWS_ACTIONS_ROLE_ARN \
  --env production \
  --body "$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw github_actions_ecr_role_arn)" \
  --repo YOUR_USERNAME/clearledger
</code></pre>
<p><strong>Note</strong>: GitHub blocks secret names that start with <code>GITHUB_</code>. Use <code>AWS_ACTIONS_ROLE_ARN</code>, not <code>GITHUB_ACTIONS_ROLE_ARN</code>.</p>
<p>Also make sure <code>github_owner</code> is set correctly in <code>terraform.tfvars</code> (see <code>terraform.tfvars.example</code>) before running <code>terraform apply</code>. This wires up the OIDC trust policy so AWS will accept tokens from your specific GitHub account.</p>
<p>Once <code>CLEARLEDGER_CI_TARGET=aws</code> is set, every push to <code>main</code> runs the AWS pipeline: Gitleaks → Semgrep → Checkov → build → Trivy scan → ECR push → kustomization update. The homelab <code>ci.yaml</code> is skipped.</p>
<p><strong>If CI fails at the ECR push step:</strong></p>
<p>The most common failure is <code>Not authorized to perform sts:AssumeRoleWithWebIdentity</code>. This means the IAM role trust policy still has a placeholder <code>YOUR_GITHUB_USERNAME</code> in the <code>:sub</code> condition. Fix it by setting <code>github_owner</code> in <code>terraform.tfvars</code> and running <code>terraform apply</code> again, then re-run only the failed job (not the whole pipeline: the earlier scan steps already passed).</p>
<pre><code class="language-text">GitHub → Actions → failed run → Re-run failed jobs
</code></pre>
<p>If you see <code>404</code> when running <code>gh secret set</code>, the <code>production</code> environment doesn't exist yet. Run the <code>gh api --method PUT</code> command above first.</p>
<p><strong>Re-run after fixing OIDC:</strong> failed jobs only, not the full pipeline. Earlier gates (Gitleaks, build, scan) already passed, and their artifacts are still in the workflow run. Use Re-run all jobs only if you changed app code or want a clean scan from scratch.</p>
<h3 id="heading-production-hardening-checklist">Production Hardening Checklist</h3>
<p>The lab architecture is production-style, but a real production setup needs extra guardrails. Add these before you describe it as production-ready.</p>
<h4 id="heading-1-protect-the-main-branches">1. Protect the main branches</h4>
<p>Protect both GitHub repos:</p>
<pre><code class="language-text">github.com/YOUR_GITHUB_USERNAME/clearledger
github.com/YOUR_GITHUB_USERNAME/clearledger-infra
</code></pre>
<p>Go to each repo:</p>
<pre><code class="language-text">Settings
→ Rules
→ Rulesets
→ New ruleset
→ Branch targeting: main
</code></pre>
<p>Enable:</p>
<pre><code class="language-text">Require a pull request before merging
Require approvals
Require status checks to pass
Require branches to be up to date before merging
Block force pushes
Block branch deletion
</code></pre>
<p>Why this matters: nobody should push straight to the code repo or the GitOps repo in production. A bad direct push to <code>clearledger-infra</code> is a direct deployment request.</p>
<h4 id="heading-2-use-github-environments-with-approvals">2. Use GitHub Environments with approvals</h4>
<p>Create a protected environment:</p>
<pre><code class="language-text">clearledger repo
→ Settings
→ Environments
→ New environment
→ Name: production
→ Required reviewers: add yourself or the team
→ Deployment branches: main only
</code></pre>
<p>The AWS workflow uses:</p>
<pre><code class="language-yaml">environment: production
</code></pre>
<p>That means GitHub pauses the AWS deployment until an approved reviewer allows it. This creates a real promotion gate instead of "every push deploys to prod."</p>
<h4 id="heading-3-prefer-fine-grained-tokens-or-a-github-app">3. Prefer fine-grained tokens or a GitHub App</h4>
<p>For the basic lab, <code>INFRA_REPO_TOKEN</code> can be a classic PAT. For production, tighten it.</p>
<p>Better option:</p>
<pre><code class="language-text">Fine-grained personal access token
→ Repository access: only YOUR_GITHUB_USERNAME/clearledger-infra
→ Permissions:
   Contents: Read and write
   Metadata: Read
</code></pre>
<p>Best option for teams: use a GitHub App installed only on <code>clearledger-infra</code>, with permission to write contents. That gives better audit logs and easier rotation than a personal token.</p>
<p>Store <code>INFRA_REPO_TOKEN</code> as a production environment secret, not a general repository secret:</p>
<pre><code class="language-text">clearledger
→ Settings
→ Environments
→ production
→ Environment secrets
→ INFRA_REPO_TOKEN
</code></pre>
<h4 id="heading-4-lock-aws-oidc-to-the-production-environment">4. Lock AWS OIDC to the production environment</h4>
<p>This isn't a shell command. It's a trust rule Terraform writes into AWS when you run <code>terraform apply</code>.</p>
<p>In <code>iam.tf</code>, the IAM role <code>clearledger-github-actions-ecr</code> only accepts GitHub tokens whose subject claim matches:</p>
<pre><code class="language-text">repo:YOUR_GITHUB_USERNAME/clearledger:environment:production
</code></pre>
<p>Only GitHub Actions jobs running in the <code>production</code> environment of your <code>clearledger</code> repo can assume the ECR push role. A random branch, fork, or workflow without that environment can't get AWS credentials.</p>
<p><strong>What you do:</strong></p>
<ol>
<li>Set <code>github_owner</code> in <code>terraform.tfvars</code>, then <code>terraform apply</code> (Stage 8 step 2).</li>
<li>On GitHub: <strong>Settings → Environments → production.</strong> Create it if missing, and add protection rules if you want.</li>
<li>Add environment secret <code>AWS_ACTIONS_ROLE_ARN</code> = <code>terraform output -raw github_actions_ecr_role_arn</code>.</li>
<li><code>ci-aws.yaml</code> already sets <code>environment: production</code> on the ECR jobs, that is what makes GitHub mint a matching token.</li>
</ol>
<p><strong>Verify the rule exists (optional):</strong></p>
<pre><code class="language-bash">aws iam get-role --role-name clearledger-github-actions-ecr \
  --query 'Role.AssumeRolePolicyDocument.Statement[0].Condition.StringEquals."token.actions.githubusercontent.com:sub"' \
  --output text
</code></pre>
<p>Expect: <code>repo:your-username/clearledger:environment:production</code></p>
<p>On the manual Stage 8 path you can skip GitHub CI entirely, this lock only matters when you enable CI, AWS (ECR + OIDC).</p>
<h4 id="heading-5-staging-before-production-promote-dont-rebuild">5. Staging before production (promote, don't rebuild)</h4>
<p><strong>Lab flow:</strong> push to <code>main</code> → <code>ci-aws.yaml</code> builds and scans → CI updates <code>stages/stage-8-aws-migration/manifests/kustomization.yaml</code> with the new image tag → Argo CD syncs <code>clearledger-aws</code>.</p>
<p>Homelab Stages 1–7 still use <code>clearledger-infra</code> and Docker Hub. Stage 8 AWS uses the in-repo kustomize path.</p>
<p>Real production adds a staging step in the middle: build the image once, deploy that same tag or digest to staging, run smoke tests or get manual approval, then promote to production, without building again.</p>
<p>Why? Because, if you rebuild for prod, you might ship different code than what passed staging. The safe pattern is one artifact, tested once, promoted twice.</p>
<pre><code class="language-text">Build once (one image SHA)
  → deploy to staging
  → test / approve
  → deploy the same SHA to production
</code></pre>
<h4 id="heading-6-use-private-networking-where-possible">6. Use private networking where possible</h4>
<p>For production AWS:</p>
<pre><code class="language-text">- EKS nodes in private subnets
- RDS in private subnets
- Private EKS API endpoint, or restricted public endpoint
- Security groups scoped to required ports only
- ALB public only if the app is public
- No SSH-based deployment path
</code></pre>
<p>The pipeline should talk to AWS APIs through IAM/OIDC and deploy through GitOps. It shouldn't SSH into EC2 instances.</p>
<h4 id="heading-7-store-terraform-state-remotely">7. Store Terraform state remotely</h4>
<p>Local Terraform state is fine for a lab. Production should use encrypted remote state:</p>
<pre><code class="language-text">- S3 bucket for terraform.tfstate
- DynamoDB table for state locking
- SSE encryption enabled
- Bucket versioning enabled
- Public access blocked
</code></pre>
<p>The Terraform backend block is already included in <code>stages/stage-8-aws-migration/terraform/main.tf</code> as a commented template.<br>Uncomment it after you create the S3 bucket and DynamoDB lock table.</p>
<h4 id="heading-production-ready-summary">Production-ready summary:</h4>
<pre><code class="language-text">- CI builds and proves the artifact.
- GitHub Environments approve production.
- OIDC gives short-lived AWS credentials.
- ECR stores immutable images.
- kustomization.yaml (Stage 8 path) records desired state.
- ArgoCD clearledger-aws deploys from Git.
- No SSH. No static AWS keys. No direct kubectl from CI.
</code></pre>
<p>Open the URL. ClearLedger is running on AWS. Same architecture, same security layers, just new infrastructure.</p>
<img src="https://cdn.hashnode.com/uploads/covers/698d563262d4ce66226a844a/3e6a283d-add1-467b-83da-97c1915f0b92.png" alt="screenshot of clearledger UI running on EKS with ALB URL" style="display: block;" width="1473" height="1269" loading="lazy">

<p><strong>Destroy when done.</strong> This stops all charges:</p>
<pre><code class="language-bash">make aws-down
</code></pre>
<p>See <code>stages/stage-8-aws-migration/README.md</code> for the full walkthrough and cost reference.</p>
<h3 id="heading-what-you-learned-in-stage-8">What You Learned in Stage 8</h3>
<ul>
<li>That containerized applications are portable: the same code runs on your laptop and on AWS</li>
<li>What Terraform does: declares infrastructure as code so environments are reproducible</li>
<li>What changes in a cloud migration (managed services, IAM, networking) and what does not (application code, CI logic, security policies)</li>
<li>Three AWS secret delivery paths: ESO (default), CSI file mounts (§8.5), vs Vault on homelab</li>
<li>AWS-specific security services: GuardDuty (threat detection), CloudTrail (API audit), GitHub Actions OIDC (pipeline AWS auth without long-lived keys), and IRSA (pod-level IAM without long-lived credentials)</li>
</ul>
<p><strong>What you can now put on your CV / say in an interview:</strong></p>
<blockquote>
<p>Migrated the same architecture to AWS (EKS, ECR, RDS, ALB, with secrets via External Secrets Operator and IRSA) provisioned by Terraform, without rewriting the application.</p>
</blockquote>
<p><strong>When you're done on AWS, tear down to stop charges:</strong></p>
<pre><code class="language-bash">make aws-down
</code></pre>
<p>Your homelab VM is separate. If you plan to return to it, you should already have a snapshot from Stage 7 (<code>make snapshots</code> to confirm). See <a href="#heading-how-to-save-your-progress">Saving your progress</a>.</p>
<h2 id="heading-troubleshooting-see-troubleshootingmd">Troubleshooting (See <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md</a>)</h2>
<p><strong>Pod stuck in Pending:</strong></p>
<pre><code class="language-bash">kubectl describe pod POD_NAME -n clearledger
# Insufficient memory/cpu → reduce resource requests
# Image pull error → check Docker Hub repo name and credentials
</code></pre>
<p><strong>Kyverno blocking a deployment:</strong></p>
<pre><code class="language-bash">kubectl get events -n clearledger --sort-by='.lastTimestamp' | tail -10
kubectl get policyreport -n clearledger -o yaml
</code></pre>
<p><strong>Vault agent not injecting secrets:</strong></p>
<pre><code class="language-bash">kubectl logs POD_NAME -n clearledger -c vault-agent-init
kubectl exec -n vault vault-0 -- vault read auth/kubernetes/role/auth-service
</code></pre>
<p><strong>Falco not firing alerts:</strong></p>
<pre><code class="language-bash">kubectl logs -n falco daemonset/falco | grep -i error | tail -20
</code></pre>
<p><strong>ArgoCD shows OutOfSync:</strong></p>
<pre><code class="language-bash">argocd app sync clearledger --force
argocd app get clearledger
kubectl get events -n clearledger --sort-by='.lastTimestamp'
</code></pre>
<p><strong>clearledger.local not resolving:</strong></p>
<pre><code class="language-bash">multipass info clearledger | grep IPv4
grep clearledger /etc/hosts
# If the IP changed, update /etc/hosts
</code></pre>
<p><strong>VM disk full or pods Evicted (disk pressure):</strong></p>
<pre><code class="language-bash">make doctor     # PASS / WARN / FAIL + PVC and Prometheus TSDB sizes
make reclaim    # safe reclaim — unused images + journald only (not PVCs)
</code></pre>
<p>If still FAIL after reclaim, tear down and recreate: <code>make teardown &amp;&amp; make setup</code>. Full guidance: <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">troubleshooting.md: disk health</a> and <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/troubleshooting.md">VM disk full</a>.</p>
<h2 id="heading-compliance-reference">Compliance Reference</h2>
<p>Every control maps to at least one framework. Full mapping: <a href="compliance-mapping.md"><code>docs/compliance-mapping.md</code></a>.</p>
<table>
<thead>
<tr>
<th>Control</th>
<th>Tool</th>
<th>Stage</th>
<th>PCI-DSS</th>
<th>SOC2</th>
<th>CIS K8s</th>
</tr>
</thead>
<tbody><tr>
<td>Secrets detection</td>
<td>Gitleaks</td>
<td>3</td>
<td>6.2</td>
<td>CC8.1</td>
<td>—</td>
</tr>
<tr>
<td>SAST</td>
<td>Semgrep</td>
<td>3</td>
<td>6.3.2</td>
<td>CC7.1</td>
<td>—</td>
</tr>
<tr>
<td>Dependency scan</td>
<td>Trivy SCA</td>
<td>3</td>
<td>6.3.3</td>
<td>CC7.1</td>
<td>—</td>
</tr>
<tr>
<td>IaC scan</td>
<td>Checkov</td>
<td>3</td>
<td>6.3.1</td>
<td>CC6.1</td>
<td>—</td>
</tr>
<tr>
<td>Image signing</td>
<td>Cosign</td>
<td>3</td>
<td>6.3</td>
<td>CC6.1</td>
<td>—</td>
</tr>
<tr>
<td>SBOM generation</td>
<td>Syft</td>
<td>3</td>
<td>6.3.3</td>
<td>CC6.1</td>
<td>—</td>
</tr>
<tr>
<td>Non-root containers</td>
<td>Kyverno</td>
<td>4</td>
<td>6.5</td>
<td>CC6.3</td>
<td>5.2.6</td>
</tr>
<tr>
<td>Resource limits</td>
<td>Kyverno</td>
<td>4</td>
<td>—</td>
<td>A1.1</td>
<td>5.2.4</td>
</tr>
<tr>
<td>No privilege escalation</td>
<td>Kyverno</td>
<td>4</td>
<td>6.5</td>
<td>CC6.3</td>
<td>5.2.5</td>
</tr>
<tr>
<td>Secrets management</td>
<td>Vault</td>
<td>5</td>
<td>3.5</td>
<td>CC6.1</td>
<td>—</td>
</tr>
<tr>
<td>Runtime detection</td>
<td>Falco</td>
<td>6</td>
<td>10.7</td>
<td>CC7.2</td>
<td>—</td>
</tr>
<tr>
<td>Network segmentation</td>
<td>NetworkPolicy</td>
<td>6</td>
<td>1.3</td>
<td>CC6.6</td>
<td>5.3.2</td>
</tr>
<tr>
<td>Security observability</td>
<td>Grafana</td>
<td>7</td>
<td>10.6</td>
<td>CC7.2</td>
<td>—</td>
</tr>
<tr>
<td>DORA metrics</td>
<td>ArgoCD + Grafana</td>
<td>7</td>
<td>—</td>
<td>—</td>
<td>—</td>
</tr>
<tr>
<td>Account threat detection</td>
<td>GuardDuty</td>
<td>8</td>
<td>10.6</td>
<td>CC7.2</td>
<td>—</td>
</tr>
<tr>
<td>API audit trail</td>
<td>CloudTrail</td>
<td>8</td>
<td>10.2</td>
<td>CC7.3</td>
<td>—</td>
</tr>
</tbody></table>
<p><strong>EU DORA (Digital Operational Resilience Act):</strong> applies to EU financial entities since January 2025. ClearLedger maps to all five DORA pillars. Full mapping in <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/compliance-mapping.md"><code>docs/compliance-mapping.md</code></a>.</p>
<h2 id="heading-interview-preparation">Interview Preparation</h2>
<p>Full weak/strong answers: <a href="https://github.com/Osomudeya/clearledger/blob/main/docs/interview-prep.md"><code>docs/interview-prep.md</code></a></p>
<p>Practice these as you finish each stage:</p>
<p><strong>Stage 0:</strong> How does traffic reach your services in Kubernetes? What breaks first when deployment is manual?</p>
<p><strong>Stage 1:</strong> How do you prove what image is deployed for a given commit? What stops a developer bypassing CI?</p>
<p><strong>Stage 2:</strong> What does GitOps mean mechanically? How do you prove drift is corrected automatically?</p>
<p><strong>Stage 3:</strong> Difference between SAST, IaC scanning, and image scanning? Where do you draw the line for fail-on severity?</p>
<p><strong>Stage 4:</strong> What is admission control and why is it different from CI? How would you safely introduce a policy exception?</p>
<p><strong>Stage 5:</strong> Why are Kubernetes Secrets not "secret management"? How do you rotate secrets with minimal downtime risk?</p>
<p><strong>Stage 6:</strong> What does runtime detection catch that CI and admission can't? What is your first response to a shell-spawn alert?</p>
<p><strong>Stage 7:</strong> What's the difference between a dashboard and an alert? How do you produce audit evidence, not just claims?</p>
<p><strong>Stage 8:</strong> What actually changes when you move to EKS? What shouldn't change? How does IRSA reduce risk?</p>
<h2 id="heading-aws-cost-reference">AWS Cost Reference</h2>
<p>Default Stage 8 sizes (eu-west-1, approximate):</p>
<table>
<thead>
<tr>
<th>Resource</th>
<th>Monthly (8h/day)</th>
<th>Monthly (24/7)</th>
</tr>
</thead>
<tbody><tr>
<td>EKS control plane</td>
<td>~$24</td>
<td>~$73</td>
</tr>
<tr>
<td>3× t3.medium nodes</td>
<td>~$30</td>
<td>~$92</td>
</tr>
<tr>
<td>NAT Gateway</td>
<td>~$11</td>
<td>~$33</td>
</tr>
<tr>
<td>RDS db.t3.micro</td>
<td>~$4</td>
<td>~$13</td>
</tr>
<tr>
<td>ALB</td>
<td>~$2</td>
<td>~$6</td>
</tr>
<tr>
<td>GuardDuty + CloudTrail</td>
<td>~$2</td>
<td>~$5</td>
</tr>
<tr>
<td><strong>Total estimate</strong></td>
<td><strong>~$73</strong></td>
<td><strong>~$222</strong></td>
</tr>
</tbody></table>
<p>Always destroy when not in use:</p>
<pre><code class="language-bash">make aws-down
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>You've now built a fintech application and layered eight security and reliability controls on top of it: all from a laptop.</p>
<p>You started with raw Kubernetes and manual deploys in Stage 0. You added a CI pipeline that builds, scans, and signs images automatically in Stage 1. You connected Git to the cluster with ArgoCD in Stage 2. You gated every push with SAST, IaC, and image scanning in Stage 3. You blocked bad workloads at the cluster boundary with Kyverno in Stage 4. You moved credentials out of Git and Kubernetes secrets into Vault in Stage 5. You added runtime threat detection with Falco and network segmentation in Stage 6. You built observability dashboards that produce audit evidence in Stage 7. And you migrated the whole thing to AWS in Stage 8.</p>
<p>None of these stages is a toy exercise. Each one represents a real problem that real teams hit in production. You felt the pain, then built the solution. That's the difference between reading about DevSecOps and being able to do it.</p>
<p>Take your screenshots, update your CV with the specific tools and outcomes, and use the interview prep section when you need to talk through the decisions you made. You built every one of them.</p>
<p><em>If you found this guide helpful, share it with someone breaking into DevOps or DevSecOps and</em> <a href="https://www.linkedin.com/in/osomudeya-zudonu-17290b124"><em>connect on LinkedIn</em></a><em>.</em></p>
<p><em>I also post DevOps walkthroughs and interview tips for getting hired; follow or</em> <a href="https://osomudeya.kit.com/23db7ca59f"><em>subscribe there</em></a> <em>if you want more.</em></p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ My Team's Experience Moving from AWS to a PaaS ]]>
                </title>
                <description>
                    <![CDATA[ Most product teams assume infrastructure ownership is simply part of building software. We did too. It wasn’t until we measured how much engineering time was disappearing into operational work that we ]]>
                </description>
                <link>https://www.freecodecamp.org/news/my-team-s-experience-moving-from-aws-to-a-paas/</link>
                <guid isPermaLink="false">6a442bf6f75ef9bd6e0bfc7f</guid>
                
                    <category>
                        <![CDATA[ infrastructure ]]>
                    </category>
                
                    <category>
                        <![CDATA[ PaaS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Manish Shivanandhan ]]>
                </dc:creator>
                <pubDate>Tue, 30 Jun 2026 20:49:58 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/01d97a10-ea77-49e7-a8d9-1471683948b7.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Most product teams assume infrastructure ownership is simply part of building software. We did too. It wasn’t until we measured how much engineering time was disappearing into operational work that we realised how expensive that assumption had become.</p>
<p>During a quarterly planning session, one of our engineers asked a question nobody on the team had thought to ask directly before: “How much of our time is actually going into infrastructure, versus building things people use?”</p>
<p>It wasn’t a rhetorical question. We pulled up our sprint history, our incident logs, and our calendars, and tried to answer it honestly.</p>
<p>We were a 7-person internal tooling team inside a larger enterprise organisation. Our mandate was straightforward: make other teams across the company faster through workflow automation, internal dashboards, and integrations between internal systems.</p>
<p>Our <a href="https://aws.amazon.com/">Amazon Web Services (AWS)</a> environment wasn’t poorly built. It was, by most standards, mature infrastructure. Containerised services on ECS, automated deployments through GitHub Actions, CloudWatch observability, and properly scoped IAM roles across environments. Nothing about it would have raised concerns in an architecture review.</p>
<p>What it cost us wasn’t visible on an invoice. It was visible in calendars, in context-switching, and in how often “infrastructure work” quietly displaced the backlog we were actually accountable for.</p>
<p>That conversation eventually led us to evaluate and migrate to <a href="http://sevalla.com/">Sevalla</a>, a Platform-as-a-Service infrastructure control for operational simplicity. The migration took three weeks. The effects were measurable within a month.</p>
<p>In this article, we'll walk through what our AWS setup looked like before migrating, what the migration process actually involved, the specific metrics that changed afterwards, and the trade-offs we accepted along the way.</p>
<h3 id="heading-what-well-cover">What We'll Cover:</h3>
<ul>
<li><p><a href="#heading-before-the-migration">Before the Migration</a></p>
</li>
<li><p><a href="#heading-the-number-that-started-the-conversation">The Number That Started the Conversation</a></p>
</li>
<li><p><a href="#heading-the-deployment-process-what-reasonably-automated-actually-meant">The Deployment Process: What “Reasonably Automated” Actually&nbsp;Meant</a></p>
</li>
<li><p><a href="#heading-what-the-migration-actually-involved">What the Migration Actually&nbsp;Involved</a></p>
</li>
<li><p><a href="#heading-what-changed-after-the-migration">What Changed After the Migration</a></p>
<ul>
<li><p><a href="#heading-deployment-time-dropped-from-12-minutes-to-3-minutes">Deployment time dropped from ~12 minutes to ~3&nbsp;minutes</a></p>
</li>
<li><p><a href="#heading-any-engineer-could-deploy-confidently-on-day-one">Any engineer could deploy confidently on day&nbsp;one</a></p>
</li>
<li><p><a href="#heading-rollbacks-went-from-a-12-minute-manual-process-to-a-30-second-action">Rollbacks went from a 12-minute manual process to a 30-second action</a></p>
</li>
<li><p><a href="#heading-infrastructure-maintenance-time-dropped-to-approximately-23-hours-per-week">Infrastructure maintenance time dropped to approximately 2–3 hours per&nbsp;week</a></p>
</li>
<li><p><a href="#heading-log-visibility-improved-without-any-additional-tooling">Log visibility improved without any additional tooling</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-what-we-gave-up">What We Gave&nbsp;Up</a></p>
</li>
<li><p><a href="#heading-the-actual-lesson">The Actual&nbsp;Lesson</a></p>
</li>
</ul>
<h2 id="heading-before-the-migration">Before the Migration</h2>
<p>Our AWS setup was respectable. We weren’t running something embarrassingly manual. We had:</p>
<ul>
<li><p>ECS for container orchestration</p>
</li>
<li><p>RDS for databases</p>
</li>
<li><p>CloudWatch for logs and metrics</p>
</li>
<li><p>A CI/CD pipeline through GitHub Actions</p>
</li>
<li><p>IAM roles managed across environments</p>
</li>
<li><p><a href="https://aws.amazon.com/cloudformation/">CloudFormation</a> templates maintained by one senior engineer</p>
</li>
</ul>
<p>It worked. Deployments were automated. The system was stable.</p>
<p>The problem wasn’t that anything was broken. The problem was what it cost us to keep it running smoothly.</p>
<h2 id="heading-the-number-that-started-the-conversation">The Number That Started the Conversation</h2>
<p>During a quarterly planning session, we tried to honestly account for where engineering time was going.</p>
<p>We estimated that across the team, roughly 12–15 hours per week were being spent on infrastructure-related work that wasn’t directly delivering value to internal users. This included:</p>
<ul>
<li><p>Deployment pipeline maintenance and debugging (~4 hrs/week)</p>
</li>
<li><p>CloudWatch log investigation and alert tuning (~3 hrs/week)</p>
</li>
<li><p>IAM permissions management and access reviews (~2 hrs/week)</p>
</li>
<li><p>Dependency updates, security patches for infrastructure components (~2 hrs/week)</p>
</li>
<li><p>Ad hoc incidents, environment drift, cost anomaly investigations (~3–4 hrs/week)</p>
</li>
</ul>
<p>At a fully-loaded engineer cost, 12–15 hours per week is the equivalent of roughly one-third of a full-time engineer, every week, spent on keeping the lights on rather than building anything.</p>
<p>For a team whose backlog was already longer than we could realistically tackle, that number was hard to justify.</p>
<h2 id="heading-the-deployment-process-what-reasonably-automated-actually-meant">The Deployment Process: What “Reasonably Automated” Actually&nbsp;Meant</h2>
<p>Our deployment pipeline was good by most standards. Push to <code>main</code>, GitHub Actions triggered a build, pushed an image to ECR, and updated the ECS service. On a good day, a deployment took about 12 minutes from merge to live.</p>
<p>But “reasonably automated” came with caveats.</p>
<p>Only one engineer fully understood the pipeline. If something failed mid-deployment, like a task definition mismatch, an IAM permission error, or a CloudFormation stack conflict, most of the team would either wait for that engineer or spend significant time reading AWS documentation to diagnose it themselves.</p>
<p>Rollbacks were manual. There was no clean one-click rollback. Rolling back meant redeploying the previous image tag, which required knowing what that tag was, triggering the pipeline again, and waiting another 12 minutes. In an incident, those 12 minutes mattered.</p>
<p>Environment parity was fragile. We had staging and production environments. Keeping them consistent required discipline and periodic reconciliation. Configuration drift happened more than we’d like to admit, and it occasionally caused releases to behave differently in production than they had in staging.</p>
<p>New team members couldn’t deploy confidently. Onboarding a new engineer to the deployment process took the better part of a day, and most new hires remained hesitant to trigger deployments independently for weeks. The pipeline was automated, but the knowledge wasn’t.</p>
<h2 id="heading-what-the-migration-actually-involved">What the Migration Actually&nbsp;Involved</h2>
<p>We moved over the course of about three weeks, migrating services incrementally rather than cutting over all at once.</p>
<p>The largest time investment was translating our environment variable configuration and secrets management from AWS Parameter Store into Sevalla’s environment configuration. That took roughly half a day.</p>
<p>The CI/CD migration was straightforward. We replaced our ECS deployment step with Sevalla’s Git-connected deployment. The GitHub integration picked up our repository directly.</p>
<p>Database migration was the most careful part. We ran both databases in parallel for two weeks, verified data consistency, then cut over DNS. There was no data loss, and no downtime.</p>
<p>Total migration effort across the team: approximately 40 hours spread over three weeks, mostly concentrated in two engineers.</p>
<h2 id="heading-what-changed-after-the-migration">What Changed After the Migration</h2>
<h3 id="heading-deployment-time-dropped-from-12-minutes-to-3-minutes">Deployment time dropped from ~12 minutes to ~3&nbsp;minutes</h3>
<p>This wasn’t the most important change, but it was the most immediately visible one. Faster deployments meant faster feedback loops. A fix could be in production and verified within minutes rather than waiting out a build cycle.</p>
<p>Over a typical week with 8–10 deployments, that’s roughly 90 minutes of cumulative waiting time recovered, per week.</p>
<h3 id="heading-any-engineer-could-deploy-confidently-on-day-one">Any engineer could deploy confidently on day&nbsp;one</h3>
<p>This was the change that mattered most operationally. The deployment process became visible, documented by the interface itself, and required no specialist knowledge to operate. A new engineer joining the team could deploy their first change independently on their first day.</p>
<p>The informal “deployment gatekeeper” role that had quietly formed around our most AWS-experienced engineer effectively dissolved.</p>
<h3 id="heading-rollbacks-went-from-a-12-minute-manual-process-to-a-30-second-action">Rollbacks went from a 12-minute manual process to a 30-second action</h3>
<p>Every deployment in Sevalla retains a one-click rollback to the previous build. During the first month after migration, we used this twice: once for a regression we caught quickly, and once during a failed database migration we immediately needed to reverse.</p>
<p>Both incidents that previously would have required hours of manual intervention were resolved in under a minute.</p>
<h3 id="heading-infrastructure-maintenance-time-dropped-to-approximately-23-hours-per-week">Infrastructure maintenance time dropped to approximately 2–3 hours per&nbsp;week</h3>
<p>We no longer maintain IAM roles, CloudWatch alerts, CloudFormation templates, or ECS task definitions. The infrastructure surface area we own shrank dramatically.</p>
<p>Our estimate of 12–15 hours per week of infrastructure work fell to roughly 2–3 hours per week . It now involves  primarily monitoring application behaviour and reviewing build logs. That’s a recovery of approximately 10 hours per week of engineering time redirected toward the actual backlog.</p>
<p>Over a quarter, that’s roughly 130 hours, or about three full working weeks, returned to product work.</p>
<p>Looking back, we had quietly become a platform team. Not because we intended to, but because every infrastructure decision created more infrastructure to own.</p>
<h3 id="heading-log-visibility-improved-without-any-additional-tooling">Log visibility improved without any additional tooling</h3>
<p>One outcome we didn’t anticipate: production visibility got better even though we invested less in it.</p>
<p>On AWS, meaningful log analysis required CloudWatch Insights queries, proper log group configuration, and knowing where to look. Useful observability required deliberate setup effort.</p>
<p>On Sevalla, build logs, runtime logs, and deployment history are accessible directly from the dashboard without configuration. When something went wrong in production, the time from “something is broken” to “here is what happened” dropped from 10–20 minutes of searching across tools to under 2 minutes in most cases.</p>
<h2 id="heading-what-we-gave-up">What We Gave&nbsp;Up</h2>
<p>Intellectual honesty requires listing the trade-offs.</p>
<p>First, we have less infrastructure flexibility. If we needed custom networking topology, specialised compute instances, or fine-grained storage configuration, Sevalla wouldn't cover those requirements. For an internal tooling team, none of those needs has materialised. But they could.</p>
<p>Also, some AWS-native integrations required reworking. We used a few Lambda functions that had to be refactored into services. That added some migration complexity we hadn’t fully anticipated.</p>
<h2 id="heading-the-actual-lesson">The Actual&nbsp;Lesson</h2>
<p>The migration confirmed something that’s easy to miss when you’re inside it: the cost of infrastructure ownership for a product team isn’t primarily the cloud bill. It’s the engineering attention.</p>
<p>For our team, 10 hours per week of recovered time across a 7-person team meant a 28% increase in capacity available for work that users actually care about. That’s not a marginal improvement. It’s a meaningful change in what the team can realistically ship.</p>
<p>That outcome isn’t specific to Sevalla. Any infrastructure simplification that genuinely reduces operational burden would produce a similar result.</p>
<p>The question worth asking isn’t whether your team <em>can</em> manage infrastructure. It’s whether managing infrastructure is the best use of the engineering capacity you have.</p>
<p>For an internal tooling team whose value is measured entirely by what it ships, not by how it deploys, the answer, for us, was clearly no.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ The EKS Cost Optimization Handbook: Reduce Your AWS Bill by 60% Using Karpenter and Rightsizing ]]>
                </title>
                <description>
                    <![CDATA[ This handbook is a complete guide to the 7-step playbook that took one EKS bill from $85,000/month to $34,000/month — without touching a single line of product code. I've audited EKS clusters at more  ]]>
                </description>
                <link>https://www.freecodecamp.org/news/eks-cost-optimization-reduce-your-aws-bill-using-karpenter-and-rightsizing/</link>
                <guid isPermaLink="false">6a396515fa8e37864960ddb6</guid>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ optimization ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Cloud Computing ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Ayobami Adejumo ]]>
                </dc:creator>
                <pubDate>Mon, 22 Jun 2026 16:38:45 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/cd12d552-bcf2-466a-a98e-7674c436afaa.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>This handbook is a complete guide to the 7-step playbook that took one EKS bill from <code>$85,000</code>/month to <code>$34,000</code>/month — without touching a single line of product code.</p>
<p>I've audited EKS clusters at more than 10 companies. The same waste patterns appear every time: over-provisioned nodes, cross-AZ data transfer, idle EBS volumes, and so on. And the most expensive mistake of all: buying compute commitments before rightsizing.</p>
<p>This handbook is the fix. I've used this 7-step playbook to reduce EKS costs by 50–60% at every company where I've implemented it. There are no product code changes, and no downtime. Just infrastructure optimization executed in the right order.</p>
<p>By the end of this guide, you'll know how to right-size pod resource requests, implement Karpenter for intelligent bin-packing and Spot diversification, migrate compatible workloads to Graviton for 20% cheaper compute, and eliminate NAT Gateway charges entirely with VPC endpoints.</p>
<p>All Terraform modules, NodePool templates, and automation scripts referenced in this guide are available in the companion repository at <a href="https://github.com/aayostem/eks-cost-optimization">github.com/aayostem/eks-cost-optimization</a>. The repo includes ready-to-deploy configurations for every step so you can move from reading to implementing in the same afternoon.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-part-1-the-baseline-where-your-eks-money-is-going">Part 1: The Baseline — Where Your EKS Money Is Going</a></p>
</li>
<li><p><a href="#heading-part-2-right-sizing-pod-resource-requests">Part 2: Right-Sizing Pod Resource Requests</a></p>
</li>
<li><p><a href="#heading-part-3-karpenter-for-bin-packing-and-spot-diversification">Part 3: Karpenter for Bin-Packing and Spot Diversification</a></p>
</li>
<li><p><a href="#heading-part-4-graviton-migration">Part 4: Graviton Migration</a></p>
</li>
<li><p><a href="#heading-part-5-vpc-endpoints-for-data-transfer">Part 5: VPC Endpoints for Data Transfer</a></p>
</li>
<li><p><a href="#heading-part-6-ebs-volume-optimisation">Part 6: EBS Volume Optimisation</a></p>
</li>
<li><p><a href="#heading-part-7-load-balancer-consolidation">Part 7: Load Balancer Consolidation</a></p>
</li>
<li><p><a href="#heading-the-complete-7-step-sequence">The Complete 7-Step Sequence</a></p>
</li>
<li><p><a href="#heading-best-practices-for-eks-cost-optimisation">Best Practices Summary</a></p>
</li>
<li><p><a href="#heading-resources">Resources</a></p>
</li>
</ul>
<h2 id="heading-what-youll-learn">What You'll Learn</h2>
<ul>
<li><p>How to right-size pod resource requests using VPA recommendations</p>
</li>
<li><p>The complete Karpenter setup with Spot diversification and automatic consolidation</p>
</li>
<li><p>Graviton3 migration for all non-GPU workloads</p>
</li>
<li><p>VPC endpoints to eliminate NAT Gateway data transfer charges</p>
</li>
<li><p>EBS gp2 to gp3 migration — 20% cheaper with zero performance loss</p>
</li>
<li><p>Load balancer consolidation with shared Ingress</p>
</li>
<li><p>The 7-step sequence that maximises ROI — and why the order isn't optional</p>
</li>
</ul>
<p>Let's dive in.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>Before following along, you should have:</p>
<p><strong>Knowledge:</strong></p>
<ul>
<li><p>Working familiarity with Kubernetes — you can deploy an application and inspect pods</p>
</li>
<li><p>Basic AWS knowledge — you understand EC2 instance types, VPCs, and EBS volumes</p>
</li>
<li><p>Comfort reading Terraform HCL and Kubernetes YAML</p>
</li>
</ul>
<p><strong>Tools and access:</strong></p>
<ul>
<li><p>An existing EKS cluster running Kubernetes 1.27 or later</p>
</li>
<li><p><code>kubectl</code> configured and pointing at your cluster</p>
</li>
<li><p>AWS CLI v2 installed and authenticated with appropriate permissions</p>
</li>
<li><p>Helm 3 installed (for Karpenter and Kubecost)</p>
</li>
<li><p><a href="https://github.com/kubernetes-sigs/metrics-server">Metrics Server</a> installed in your cluster</p>
</li>
</ul>
<p><strong>Companion repository:</strong> Clone the repo before starting. It contains all YAML, Terraform, and shell scripts referenced in this guide:</p>
<pre><code class="language-bash">git clone https://github.com/aayostem/eks-cost-optimization
cd eks-cost-optimization
</code></pre>
<p><strong>Estimated savings:</strong> For a cluster running at <code>$85,000</code>/month with typical over-provisioning, expect <code>$40,000</code> to <code>$55,000</code>/month in savings after completing all 7 steps. Smaller clusters under <code>$10,000</code>/month typically see 40–50% reduction.</p>
<h2 id="heading-part-1-the-baseline-where-your-eks-money-is-going">Part 1: The Baseline — Where Your EKS Money Is Going</h2>
<h3 id="heading-11-the-typical-eks-cost-breakdown">1.1 The Typical EKS Cost Breakdown</h3>
<p>Before touching anything, you need to know exactly where the money is going. Optimising the wrong category first is how teams waste weeks of engineering time and see no meaningful reduction.</p>
<p>Here's what a typical <code>$85,000</code>/month EKS cluster looks like when you break it down:</p>
<table>
<thead>
<tr>
<th>Category</th>
<th>Monthly Cost</th>
<th>Percentage</th>
<th>Waste Potential</th>
</tr>
</thead>
<tbody><tr>
<td>Compute (EC2 nodes)</td>
<td>$52,000</td>
<td>61%</td>
<td>High — over-provisioning, wrong instance types</td>
</tr>
<tr>
<td>Data Transfer</td>
<td>$15,300</td>
<td>18%</td>
<td>Very High — cross-AZ and NAT Gateway charges</td>
</tr>
<tr>
<td>Storage (EBS volumes)</td>
<td>$10,200</td>
<td>12%</td>
<td>Medium — unattached volumes and gp2 vs gp3</td>
</tr>
<tr>
<td>Load Balancers</td>
<td>$4,250</td>
<td>5%</td>
<td>Low to Medium — single-service ALBs</td>
</tr>
<tr>
<td>EKS Control Plane</td>
<td>$72</td>
<td>&lt;1%</td>
<td>None — this is a fixed cost</td>
</tr>
<tr>
<td>Other</td>
<td>$3,178</td>
<td>4%</td>
<td>Low</td>
</tr>
</tbody></table>
<p>Compute and Data Transfer together represent 79% of the bill and account for 90% of the correctable waste. Those are the targets.</p>
<p>Run this command to see your own breakdown before starting anything:</p>
<pre><code class="language-bash"># Pull last month's cost breakdown by service
# Save this output — it becomes your before number
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \
  --granularity MONTHLY \
  --group-by Type=DIMENSION,Key=SERVICE \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table | sort -k3 -rn
</code></pre>
<p>Screenshot the output and save it. You'll compare against it after each step to verify actual savings before moving to the next one.</p>
<h3 id="heading-12-the-most-expensive-mistake-wrong-optimisation-order">1.2 The Most Expensive Mistake: Wrong Optimisation Order</h3>
<p>Here's what most teams do when they get a large AWS bill:</p>
<ol>
<li><p>Buy Savings Plans immediately, locking in waste at a 30% discount</p>
</li>
<li><p>Then implement Karpenter, discovering they've over-committed the wrong instance family</p>
</li>
<li><p>Then migrate to Graviton, discovering their Savings Plan doesn't cover ARM instances</p>
</li>
</ol>
<p>The result: a 12–36 month commitment paying for waste they could have eliminated in three weeks.</p>
<p>The correct sequence is:</p>
<pre><code class="language-plaintext">Step 1: Right-size pod requests        ← Always first
Step 2: Implement Karpenter            ← Dynamic provisioning on rightsized requests
Step 3: Enable Spot for non-prod       ← Karpenter handles fallback automatically
Step 4: Migrate to Graviton            ← Karpenter makes this seamless
Step 5: Add VPC endpoints              ← Eliminate data transfer charges
Step 6: Optimise EBS volumes           ← Quick win, run alongside other steps
Step 7: Consolidate load balancers     ← Final structural cleanup
</code></pre>
<p>Then, and only then, buy Savings Plans — against the optimised baseline you've just established.</p>
<p>The one rule: optimise first, then commit. Every step before the Savings Plan purchase reduces what you're locking in for 1–3 years.</p>
<h2 id="heading-part-2-right-sizing-pod-resource-requests">Part 2: Right-Sizing Pod Resource Requests</h2>
<h3 id="heading-21-why-over-provisioned-requests-are-so-expensive">2.1 Why Over-Provisioned Requests Are So Expensive</h3>
<p>Kubernetes schedules pods based on resource <em>requests</em> — not actual usage. A pod that requests 2 vCPUs and 4GB of memory requires a node with that capacity available, regardless of whether the pod is actually using it.</p>
<p>Here's the incorrect approach with the requests set to worst-case peak estimates:</p>
<pre><code class="language-yaml"># Bad: Resource requests set during initial deployment, never revisited
# This pod actually uses 250m CPU and 512Mi memory on average
resources:
  requests:
    cpu: "2"        # 8x more than actual usage
    memory: "4Gi"   # 8x more than actual usage
  limits:
    cpu: "4"
    memory: "8Gi"
</code></pre>
<p>When every pod is over-requested by 8x, your cluster needs 8x more nodes than your workloads actually require. That's where the 61% compute line in your bill comes from.</p>
<p>First, verify actual usage before changing anything:</p>
<pre><code class="language-bash"># Install Metrics Server if not already running
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Check actual CPU and memory usage per pod
# Compare these numbers against your current resource requests
kubectl top pods --all-namespaces --sort-by=cpu
</code></pre>
<p>Expected output showing the typical gap:</p>
<pre><code class="language-plaintext">NAMESPACE     NAME                    CPU(cores)   MEMORY(bytes)
production    payment-api-xxx         25m          128Mi
production    user-api-xxx            15m          96Mi
production    notification-svc-xxx    5m           64Mi
staging       worker-xxx              10m          256Mi
</code></pre>
<p>If your pods are requesting 2 CPU cores each but using 25m–15m cores in practice, you have a 50–80x over-request ratio. Every node in your cluster is mostly empty space you're paying for.</p>
<h3 id="heading-22-using-the-vertical-pod-autoscaler-for-recommendations">2.2 Using the Vertical Pod Autoscaler for Recommendations</h3>
<p>The Vertical Pod Autoscaler (VPA) is a Kubernetes component that analyses historical CPU and memory usage for each deployment and recommends optimal resource requests. You use it in recommendation-only mode first — it tells you what to set without changing anything automatically, so you can review and apply the changes yourself with full control.</p>
<p>Here's the correct implementation:</p>
<pre><code class="language-yaml"># Good: VPA in recommendation-only mode
# Watches your pod's actual usage for 24+ hours, then recommends right-sized requests
# updateMode: "Off" means it only recommends — it never restarts your pods
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: payment-api-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  updatePolicy:
    updateMode: "Off"   # Recommendation only — you apply manually after review
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      minAllowed:
        cpu: "100m"     # VPA will never recommend below this floor
        memory: "256Mi"
      maxAllowed:
        cpu: "2"        # VPA will never recommend above this ceiling
        memory: "4Gi"
</code></pre>
<p>Install VPA and retrieve recommendations:</p>
<pre><code class="language-bash"># Install VPA components
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/download/vertical-pod-autoscaler-1.0.0/vpa-v1.0.0.yaml

# Apply the VPA manifest for each deployment you want to right-size
kubectl apply -f vpa/payment-api-vpa.yaml

# Wait 24 hours for VPA to collect usage data, then check recommendations
kubectl describe vpa payment-api-vpa -n production
</code></pre>
<p>What a VPA recommendation looks like:</p>
<pre><code class="language-plaintext">Recommendation:
  Container Recommendations:
    Container Name: payment-api
    Lower Bound:
      cpu:     50m
      memory:  128Mi
    Target:
      cpu:     250m      ← Set your requests to this value
      memory:  512Mi     ← Set your requests to this value
    Upper Bound:
      cpu:     500m
      memory:  1Gi
</code></pre>
<p>Apply the recommendation to your deployment:</p>
<pre><code class="language-yaml"># Good: Right-sized requests based on VPA Target recommendation
resources:
  requests:
    cpu: "250m"     # Down from 2000m — an 8x reduction
    memory: "512Mi" # Down from 4096Mi — an 8x reduction
  limits:
    cpu: "500m"     # 2x the request — headroom for genuine spikes
    memory: "1Gi"   # 2x the request
</code></pre>
<p>All VPA manifests for common deployment types are in <code>vpa/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/vpa">companion repo</a>.</p>
<h3 id="heading-23-the-roi-of-right-sizing">2.3 The ROI of Right-Sizing</h3>
<table>
<thead>
<tr>
<th>Metric</th>
<th>Before</th>
<th>After</th>
<th>Improvement</th>
</tr>
</thead>
<tbody><tr>
<td>Average CPU utilisation</td>
<td>18%</td>
<td>65%</td>
<td>+47 percentage points</td>
</tr>
<tr>
<td>Node count required</td>
<td>42</td>
<td>28</td>
<td>-33%</td>
</tr>
<tr>
<td>Monthly compute cost</td>
<td>$52,000</td>
<td>$36,400</td>
<td>-$15,600/month</td>
</tr>
</tbody></table>
<p>Verify the improvement after applying recommendations:</p>
<pre><code class="language-bash"># Check cluster-level utilisation after right-sizing
# Target: 60–75% CPU and memory utilisation across nodes
kubectl top nodes
</code></pre>
<h2 id="heading-part-3-karpenter-for-bin-packing-and-spot-diversification">Part 3: Karpenter for Bin-Packing and Spot Diversification</h2>
<p>Karpenter is an open-source Kubernetes node provisioner built by AWS and donated to the CNCF.</p>
<p>Where the default Kubernetes Cluster Autoscaler scales pre-configured node groups up and down, Karpenter watches the actual resource requests of pending pods and provisions exactly the right EC2 instance type to satisfy them — selecting dynamically from thousands of available instance families rather than the two or three you pre-configured. It also continuously monitors running nodes for underutilisation and consolidates workloads onto fewer nodes, terminating the empty ones automatically.</p>
<p>The result is a cluster that is always sized to what your workloads actually need right now, not what you anticipated at setup time.</p>
<h3 id="heading-31-the-ceiling-with-cluster-autoscaler">3.1 The Ceiling with Cluster Autoscaler</h3>
<p>Cluster Autoscaler works with pre-defined node groups. You configure which instance types are available and it scales those groups up and down.</p>
<p>The limitation is that it can only provision instances from the types you pre-configured. It can't dynamically select the right instance type based on what the workload actually needs right now.</p>
<p>Here's the incorrect approach using static node groups:</p>
<pre><code class="language-bash"># Bad: Two static node groups, each over-provisioning against worst-case scenarios
# CPU-optimised group runs even when workloads are memory-bound
# Memory-optimised group runs even when workloads are CPU-bound
eksctl create nodegroup \
  --cluster my-cluster \
  --name cpu-optimized \
  --instance-types c5.2xlarge \
  --nodes-min 5 --nodes-max 20

eksctl create nodegroup \
  --cluster my-cluster \
  --name memory-optimized \
  --instance-types r5.2xlarge \
  --nodes-min 3 --nodes-max 10
</code></pre>
<p>You're provisioning for the worst case in each family simultaneously. At any given moment, one group is underutilised while the other is scaling. Neither is right.</p>
<h3 id="heading-32-how-karpenter-solves-this">3.2 How Karpenter Solves This</h3>
<p>Karpenter watches the actual resource requests of pending pods and provisions exactly the right instance type to fit them. It selects from thousands of available instance types, not just the two you pre-configured. It also consolidates running workloads onto fewer nodes when utilisation drops, automatically terminating underutilised nodes.</p>
<p>Here's the correct implementation:</p>
<pre><code class="language-yaml"># Good: Karpenter NodePool
# Karpenter selects the optimal instance type based on pending pod requirements
# Tries Spot first, falls back to On-Demand automatically when Spot isn't available
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        # Allow both x86 and ARM (Graviton) — Karpenter picks the cheaper option
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        # Try Spot first, fall back to On-Demand if unavailable
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        # Exclude families with poor price-to-performance ratio
        - key: karpenter.k8s.aws/instance-family
          operator: NotIn
          values: ["t2", "t3a"]
  limits:
    cpu: "1000"
    memory: "4000Gi"
  disruption:
    # Remove underutilised nodes and reschedule their pods automatically
    consolidationPolicy: WhenUnderutilized
    # Recycle nodes after 30 days to ensure fresh, patched AMIs
    expireAfter: 720h
</code></pre>
<p>What each setting does:</p>
<ul>
<li><p><code>consolidationPolicy: WhenUnderutilized</code>: Karpenter continuously monitors node utilisation and removes underused nodes, moving their pods elsewhere. Your node count decreases automatically as load drops without any manual intervention.</p>
</li>
<li><p><code>expireAfter: 720h</code>: Nodes older than 30 days are gracefully replaced, ensuring your infrastructure always runs the latest EKS-optimised AMI with current security patches.</p>
</li>
<li><p><code>values: ["spot", "on-demand"]</code>: Karpenter attempts Spot capacity first. If Spot is unavailable for the requested instance type, it falls back to On-Demand with no alerts and no manual action required.</p>
</li>
</ul>
<p>Migrating from Cluster Autoscaler safely:</p>
<pre><code class="language-bash"># Step 1: Install Karpenter alongside Cluster Autoscaler — do not remove CAS yet
helm repo add karpenter https://charts.karpenter.sh
helm install karpenter karpenter/karpenter \
  --namespace karpenter \
  --create-namespace \
  --set settings.clusterName=your-cluster-name

# Step 2: Apply NodePool and NodeClass configuration
kubectl apply -f karpenter/nodepool.yaml
kubectl apply -f karpenter/nodeclass.yaml

# Step 3: Taint existing legacy nodes so new pods schedule on Karpenter nodes
# This migrates workloads gradually — zero downtime
kubectl taint nodes -l eks.amazonaws.com/nodegroup=cpu-optimized \
  group=legacy:NoSchedule

# Step 4: Watch pods reschedule to Karpenter-managed nodes over the next hour
kubectl get pods -o wide --all-namespaces | grep -v legacy

# Step 5: After 30 days of stable operation, remove the old node groups
eksctl delete nodegroup --cluster my-cluster --name cpu-optimized
eksctl delete nodegroup --cluster my-cluster --name memory-optimized
</code></pre>
<p>Ready-to-deploy NodePool and NodeClass templates are in <code>karpenter/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/karpenter">companion repo</a>.</p>
<h3 id="heading-33-spot-instances-for-non-production-workloads">3.3 Spot Instances for Non-Production Workloads</h3>
<p>Staging and development workloads don't need the reliability guarantees of On-Demand instances. Moving them to Spot saves 60–90% on those node costs. Karpenter handles Spot interruptions by rescheduling pods automatically. For stateless workloads, interruptions are invisible to users.</p>
<pre><code class="language-yaml"># Good: Spot-only NodePool for staging environments
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: staging-spot
spec:
  template:
    metadata:
      labels:
        billing/environment: staging
    spec:
      taints:
        - key: environment
          value: staging
          effect: NoSchedule  # Only pods that tolerate this taint schedule here
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]   # Spot only for non-production
  disruption:
    consolidationPolicy: WhenUnderutilized
</code></pre>
<h3 id="heading-34-the-roi-of-karpenter-and-spot">3.4 The ROI of Karpenter and Spot</h3>
<table>
<thead>
<tr>
<th>Metric</th>
<th>Before (Cluster Autoscaler)</th>
<th>After (Karpenter + Spot)</th>
<th>Improvement</th>
</tr>
</thead>
<tbody><tr>
<td>Average node count</td>
<td>28</td>
<td>18</td>
<td>-36%</td>
</tr>
<tr>
<td>Average CPU utilisation</td>
<td>65%</td>
<td>82%</td>
<td>+17 percentage points</td>
</tr>
<tr>
<td>Staging environment cost</td>
<td>$8,000/month</td>
<td>$2,400/month</td>
<td>-70%</td>
</tr>
<tr>
<td>Scale-up time for new pods</td>
<td>3–5 minutes</td>
<td>30–60 seconds</td>
<td>-80%</td>
</tr>
</tbody></table>
<h2 id="heading-part-4-graviton-migration">Part 4: Graviton Migration</h2>
<p>AWS Graviton is Amazon's own ARM-based processor family, available across EC2 instance types with names ending in <code>g</code> — <code>m7g</code>, <code>c7g</code>, <code>r7g</code>, and so on.</p>
<p>Graviton instances are priced approximately 20% lower than equivalent Intel or AMD x86 instances. For most server-side workloads — Node.js, Python, Go, Java — they also deliver 20–40% better performance per dollar because the processor architecture is optimised specifically for these workload types.</p>
<p>You don't change your application code to use Graviton. You change the architecture flag in your container image build and the node selector in your Kubernetes deployment.</p>
<h3 id="heading-41-why-graviton-reduces-cost-without-reducing-performance">4.1 Why Graviton Reduces Cost Without Reducing Performance</h3>
<p>The first question to answer before migrating is whether your container images support ARM64. Most official images from Docker Hub ship as multi-architecture images. Your own application images need to be built for both architectures explicitly.</p>
<p>Check whether your images support ARM64:</p>
<pre><code class="language-bash"># Check if an image has an ARM64 manifest
docker manifest inspect your-registry/your-app:latest | jq '.manifests[].platform'
</code></pre>
<p>Expected output for a multi-arch image:</p>
<pre><code class="language-json">{"architecture": "amd64", "os": "linux"},
{"architecture": "arm64", "os": "linux", "variant": "v8"}
</code></pre>
<p>If <code>arm64</code> appears, the image is ready. If not, you need to build and push a multi-arch image first.</p>
<p>Build and push a multi-architecture image:</p>
<pre><code class="language-bash"># Build for both x86 and ARM in a single command using Docker Buildx
docker buildx create --use --name multi-arch-builder

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag your-registry/your-app:latest \
  --push \
  .
</code></pre>
<h3 id="heading-42-migrating-workloads-to-graviton">4.2 Migrating Workloads to Graviton</h3>
<p>With Karpenter already installed, Graviton migration is a single label change on your deployment. Karpenter provisions the appropriate ARM64 node automatically.</p>
<p>Here's the correct implementation:</p>
<pre><code class="language-yaml"># Good: nodeSelector directs the pod to Graviton nodes
# Karpenter provisions an arm64 node if one isn't already available
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64   # Schedule exclusively on Graviton nodes
      containers:
        - name: api
          image: your-registry/payment-api:latest  # Must be multi-arch
</code></pre>
<p>Migrate gradually, starting with stateless services:</p>
<pre><code class="language-bash"># Step 1: Migrate one stateless service and monitor for 48 hours
kubectl patch deployment payment-api \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/arch":"arm64"}}}}}'

# Step 2: Watch for errors in the first 30 minutes
kubectl logs -l app=payment-api --tail=100 -f

# Step 3: Verify the pod is running on a Graviton node
# The NODE column should show a Graviton instance type (m7g, c7g, r7g)
kubectl get pods -l app=payment-api -o wide

# Step 4: After 48 hours of stable operation, migrate the next service
</code></pre>
<p>There are some situations where you shouldn't migrate to Graviton: GPU workloads, applications with native x86 binary dependencies, or any workload where you haven't yet built multi-arch images.</p>
<h3 id="heading-43-the-roi-of-graviton">4.3 The ROI of Graviton</h3>
<table>
<thead>
<tr>
<th>Workload Type</th>
<th>x86 Monthly Cost</th>
<th>Graviton Monthly Cost</th>
<th>Saving</th>
</tr>
</thead>
<tbody><tr>
<td>Web services (Node.js, Python)</td>
<td>$18,000</td>
<td>$14,400</td>
<td>$3,600/month</td>
</tr>
<tr>
<td>Data processing</td>
<td>$12,000</td>
<td>$9,600</td>
<td>$2,400/month</td>
</tr>
<tr>
<td>API services (Go, Java)</td>
<td>$8,000</td>
<td>$6,400</td>
<td>$1,600/month</td>
</tr>
<tr>
<td><strong>Total</strong></td>
<td><strong>$38,000</strong></td>
<td><strong>$30,400</strong></td>
<td><strong>$7,600/month</strong></td>
</tr>
</tbody></table>
<h2 id="heading-part-5-vpc-endpoints-for-data-transfer">Part 5: VPC Endpoints for Data Transfer</h2>
<h3 id="heading-51-the-nat-gateway-tax">5.1 The NAT Gateway Tax</h3>
<p>Every byte that travels from your EKS pods to an AWS service — S3, DynamoDB, ECR, SQS — goes through a NAT Gateway if you haven't configured VPC endpoints. NAT Gateway charges <code>$0.045</code> per GB of data processed.</p>
<p>A busy EKS cluster pulling container images from ECR, writing to S3, and polling SQS queues can process hundreds of terabytes per month through NAT Gateway — generating thousands of dollars in charges for traffic that never actually left the AWS network.</p>
<p>Measure your current NAT Gateway cost before adding endpoints:</p>
<pre><code class="language-bash"># Get last month's NAT Gateway data processing charges
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'last month' +%Y-%m-01),End=$(date +%Y-%m-01) \
  --granularity DAILY \
  --filter '{
    "Dimensions": {
      "Key": "USAGE_TYPE",
      "Values": ["NATGateway-Bytes"]
    }
  }' \
  --metrics UnblendedCost \
  --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.UnblendedCost.Amount}' \
  --output table
</code></pre>
<h3 id="heading-52-vpc-endpoints-the-fix-that-takes-30-minutes">5.2 VPC Endpoints — The Fix That Takes 30 Minutes</h3>
<p>A VPC endpoint creates a private connection between your VPC and an AWS service, routing traffic through the AWS backbone without touching the NAT Gateway. The data transfer becomes free. Each endpoint costs approximately <code>$0.01</code>/hour — roughly <code>$7.20</code>/month — far less than the NAT Gateway processing charges it replaces.</p>
<p>Here's the complete implementation for the four most common EKS traffic destinations:</p>
<pre><code class="language-bash"># Get your VPC ID and primary route table ID first
VPC_ID=$(aws eks describe-cluster --name your-cluster \
  --query 'cluster.resourcesVpcConfig.vpcId' --output text)

ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=$VPC_ID Name=association.main,Values=true \
  --query 'RouteTables[0].RouteTableId' --output text)

echo "VPC: $VPC_ID | Route Table: $ROUTE_TABLE_ID"

# S3 gateway endpoint — free to create, eliminates all S3 traffic through NAT
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids $ROUTE_TABLE_ID

# DynamoDB gateway endpoint — also free, same mechanism as S3
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.us-east-1.dynamodb \
  --route-table-ids $ROUTE_TABLE_ID

# ECR API interface endpoint — eliminates NAT charges on image pulls
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.ecr.api \
  --subnet-ids $(aws ec2 describe-subnets \
    --filters Name=vpc-id,Values=$VPC_ID Name=tag:Tier,Values=private \
    --query 'Subnets[*].SubnetId' --output text)

# ECR Docker endpoint — required alongside ECR API for complete image pull coverage
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.ecr.dkr \
  --subnet-ids $(aws ec2 describe-subnets \
    --filters Name=vpc-id,Values=$VPC_ID Name=tag:Tier,Values=private \
    --query 'Subnets[*].SubnetId' --output text)
</code></pre>
<p>The Terraform module that creates all four endpoints in a single <code>apply</code> is in <code>terraform/vpc-endpoints/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/terraform/vpc-endpoints">companion repo</a>.</p>
<p>Verify that the endpoints are routing traffic correctly:</p>
<pre><code class="language-bash">aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=$VPC_ID \
  --query 'VpcEndpoints[*].{Service:ServiceName,State:State,Type:VpcEndpointType}' \
  --output table
# Expected: all endpoints showing State=available
</code></pre>
<h3 id="heading-53-the-roi-of-vpc-endpoints">5.3 The ROI of VPC Endpoints</h3>
<table>
<thead>
<tr>
<th>Service</th>
<th>Before (Through NAT)</th>
<th>After (VPC Endpoint)</th>
<th>Monthly Saving</th>
</tr>
</thead>
<tbody><tr>
<td>S3 data transfer</td>
<td>$4,500</td>
<td>$0</td>
<td>$4,500</td>
</tr>
<tr>
<td>ECR image pulls</td>
<td>$800</td>
<td>$0</td>
<td>$800</td>
</tr>
<tr>
<td>DynamoDB queries</td>
<td>$1,200</td>
<td>$0</td>
<td>$1,200</td>
</tr>
<tr>
<td>Endpoint cost</td>
<td>—</td>
<td>$29 (4 endpoints)</td>
<td>-$29</td>
</tr>
<tr>
<td><strong>Net saving</strong></td>
<td></td>
<td></td>
<td><strong>$6,471/month</strong></td>
</tr>
</tbody></table>
<h2 id="heading-part-6-ebs-volume-optimisation">Part 6: EBS Volume Optimisation</h2>
<h3 id="heading-61-the-gp2-to-gp3-migration">6.1 The gp2 to gp3 Migration</h3>
<p>EBS gp2 volumes price their IOPS based on storage size — 3 IOPS per GB, with a 100 IOPS minimum. EBS gp3 volumes provide 3,000 IOPS baseline regardless of size, and cost 20% less per GB. The migration runs online with no downtime.</p>
<p>Find and migrate all gp2 volumes:</p>
<pre><code class="language-bash"># Step 1: List all gp2 volumes and their sizes
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].{ID:VolumeId,Size:Size,State:State}' \
  --output table

# Step 2: Migrate each gp2 volume to gp3 — no instance stop required
# The modify operation runs online while the volume stays attached and in use
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].VolumeId' \
  --output text | tr '\t' '\n' | while read vol; do
    echo "Migrating $vol from gp2 to gp3..."
    aws ec2 modify-volume \
      --volume-id $vol \
      --volume-type gp3
done

# Step 3: Verify all volumes are now gp3
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].VolumeId' \
  --output text
# Expected: empty output — zero gp2 volumes remaining
</code></pre>
<h3 id="heading-62-finding-and-removing-orphaned-volumes-and-snapshots">6.2 Finding and Removing Orphaned Volumes and Snapshots</h3>
<p>When Kubernetes PersistentVolumeClaims are deleted, the underlying EBS volumes sometimes aren't cleaned up. They keep running — and billing — indefinitely.</p>
<pre><code class="language-bash"># Find unattached EBS volumes — status=available means not attached to any instance
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[*].{ID:VolumeId,Size:Size,Created:CreateTime}' \
  --output table

# Find EBS snapshots older than 90 days
aws ec2 describe-snapshots \
  --owner-ids self \
  --query "Snapshots[?StartTime&lt;='$(date -d '90 days ago' --iso-8601=seconds)'].[SnapshotId,StartTime,VolumeSize]" \
  --output table
</code></pre>
<p>Before deleting any snapshot, cross-reference with your RDS automated backup schedule to confirm it's not the only backup for a production database.</p>
<h3 id="heading-63-the-roi-of-ebs-optimisation">6.3 The ROI of EBS Optimisation</h3>
<table>
<thead>
<tr>
<th>Resource</th>
<th>Before</th>
<th>After</th>
<th>Monthly Saving</th>
</tr>
</thead>
<tbody><tr>
<td>gp2 → gp3 migration (1TB total)</td>
<td>$102</td>
<td>$72</td>
<td>$30</td>
</tr>
<tr>
<td>Unattached volumes removed (50 × 100GB)</td>
<td>$500</td>
<td>$0</td>
<td>$500</td>
</tr>
<tr>
<td>Old snapshots cleaned (500GB)</td>
<td>$25</td>
<td>$0</td>
<td>$25</td>
</tr>
<tr>
<td><strong>Total</strong></td>
<td><strong>$627</strong></td>
<td><strong>$72</strong></td>
<td><strong>$555/month</strong></td>
</tr>
</tbody></table>
<h2 id="heading-part-7-load-balancer-consolidation">Part 7: Load Balancer Consolidation</h2>
<h3 id="heading-71-the-problem-one-load-balancer-per-service">7.1 The Problem — One Load Balancer Per Service</h3>
<p>Many teams create a separate <code>LoadBalancer</code> Service for every microservice. On AWS, each Application Load Balancer costs approximately <code>$16.20</code>/month base charge plus <code>$0.008</code>/LCU-hour for traffic processed. At 20 microservices, that's <code>$324</code>/month before a single request is processed.</p>
<p>Here's the incorrect approach:</p>
<pre><code class="language-yaml"># Bad: This creates a dedicated AWS ALB every time it's applied
# 20 microservices = 20 ALBs = $324+/month before any traffic charges
apiVersion: v1
kind: Service
metadata:
  name: payment-api
spec:
  type: LoadBalancer   # Creates a dedicated ALB
  ports:
  - port: 80
    targetPort: 8080
</code></pre>
<h3 id="heading-72-the-fix-shared-ingress-controller">7.2 The Fix — Shared Ingress Controller</h3>
<p>An Ingress controller is a Kubernetes component that runs as a pod inside your cluster and programs a single external load balancer to route traffic to multiple services based on hostname and URL path. Instead of one AWS Application Load Balancer per microservice, you get one ALB total — with path-based routing directing each request to the right backend service. The result is the same routing behaviour at a fraction of the cost.</p>
<p>Here's the correct implementation:</p>
<pre><code class="language-yaml"># Good: One Ingress resource routes all external traffic
# The AWS Load Balancer Controller creates one ALB for all services listed here
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shared-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
spec:
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /payments
        pathType: Prefix
        backend:
          service:
            name: payment-service
            port:
              number: 8080
      - path: /users
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 8080
  - host: dashboard.company.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboard-service
            port:
              number: 3000
  tls:
  - hosts:
    - api.company.com
    - dashboard.company.com
    secretName: tls-wildcard-cert
</code></pre>
<p>Verify the Ingress is provisioned and the ALB DNS name is assigned:</p>
<pre><code class="language-bash"># Watch until the ADDRESS column shows the ALB DNS name (typically 2–3 minutes)
kubectl get ingress shared-ingress -n production -w
</code></pre>
<p>The cost difference:</p>
<table>
<thead>
<tr>
<th>Approach</th>
<th>Load balancers</th>
<th>Monthly cost</th>
</tr>
</thead>
<tbody><tr>
<td>LoadBalancer Service per microservice (20 services)</td>
<td>20 ALBs</td>
<td>~$400/month</td>
</tr>
<tr>
<td>Single Ingress controller</td>
<td>1 ALB</td>
<td>~$27/month</td>
</tr>
<tr>
<td><strong>Monthly saving</strong></td>
<td></td>
<td><strong>~$373/month</strong></td>
</tr>
</tbody></table>
<p>The shared Ingress manifest is in <code>k8s/ingress/</code> in the <a href="https://github.com/aayostem/eks-cost-optimization/tree/main/k8s/ingress">companion repo</a>.</p>
<h2 id="heading-the-complete-7-step-sequence">The Complete 7-Step Sequence</h2>
<table>
<thead>
<tr>
<th>Step</th>
<th>Action</th>
<th>Time to Implement</th>
<th>Expected Monthly Saving</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Right-size pod resource requests (VPA)</td>
<td>1 week</td>
<td>$15,600</td>
</tr>
<tr>
<td>2</td>
<td>Install Karpenter with consolidation</td>
<td>1 week</td>
<td>$8,400</td>
</tr>
<tr>
<td>3</td>
<td>Move staging and dev to Spot</td>
<td>1 week</td>
<td>$11,200</td>
</tr>
<tr>
<td>4</td>
<td>Migrate compatible workloads to Graviton</td>
<td>2 weeks</td>
<td>$7,600</td>
</tr>
<tr>
<td>5</td>
<td>Add VPC endpoints for S3, ECR, DynamoDB</td>
<td>1 day</td>
<td>$6,471</td>
</tr>
<tr>
<td>6</td>
<td>Migrate gp2 to gp3 and delete orphaned volumes</td>
<td>1 day</td>
<td>$555</td>
</tr>
<tr>
<td>7</td>
<td>Consolidate load balancers with shared Ingress</td>
<td>1 day</td>
<td>$373</td>
</tr>
<tr>
<td><strong>Total</strong></td>
<td></td>
<td><strong>3–4 weeks</strong></td>
<td><strong>$49,799/month</strong></td>
</tr>
</tbody></table>
<p>Annual saving at this rate: <code>$597,588</code>. Engineering time required: one engineer, one sprint per step.</p>
<h2 id="heading-best-practices-for-eks-cost-optimisation">Best Practices for EKS Cost Optimisation</h2>
<p>✅ <strong>Do:</strong> Right-size pod resource requests before any other optimisation. Every subsequent step depends on accurate requests.</p>
<p>✅ <strong>Do:</strong> Implement Karpenter with <code>consolidationPolicy: WhenUnderutilized</code>. Let it continuously optimise your node count automatically.</p>
<p>✅ <strong>Do:</strong> Move staging and development workloads to Spot. 60–90% savings for workloads that tolerate interruption.</p>
<p>✅ <strong>Do:</strong> Migrate compatible workloads to Graviton. Most web services and APIs run without code changes.</p>
<p>✅ <strong>Do:</strong> Add VPC endpoints for S3, DynamoDB, and ECR before reviewing data transfer costs.</p>
<p>✅ <strong>Do:</strong> Migrate gp2 volumes to gp3. It's online, zero downtime, and immediately 20% cheaper.</p>
<p>✅ <strong>Do:</strong> Use a single shared Ingress controller for all external traffic instead of per-service load balancers.</p>
<p>❌ <strong>Don't:</strong> Buy Savings Plans before completing steps 1–6. You'll lock in waste for 1–3 years.</p>
<p>❌ <strong>Don't:</strong> Use static node groups with Cluster Autoscaler when your workload mix changes. Karpenter handles this dynamically.</p>
<p>❌ <strong>Don't:</strong> Run staging and development environments on On-Demand instances. Spot interruptions are manageable, but the cost difference is not.</p>
<h2 id="heading-resources">Resources</h2>
<ul>
<li><p><a href="https://karpenter.sh/docs/"><strong>Karpenter Documentation</strong></a> — Official NodePool configuration reference and installation guide</p>
</li>
<li><p><a href="https://github.com/aws/aws-graviton-getting-started"><strong>AWS Graviton Getting Started Guide</strong></a> — Language-specific compatibility notes and migration guidance from AWS</p>
</li>
<li><p><a href="https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler"><strong>Vertical Pod Autoscaler GitHub</strong></a> — VPA installation and configuration documentation</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html"><strong>AWS VPC Endpoints Documentation</strong></a> — Complete list of available VPC endpoints and configuration options</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html"><strong>EBS Volume Modification Documentation</strong></a> — AWS guide for online volume type migration with zero downtime</p>
</li>
<li><p><a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/"><strong>AWS Load Balancer Controller</strong></a> — Official documentation for the Ingress controller that provisions AWS ALBs</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/cost-management/latest/APIReference/API_GetCostAndUsage.html"><strong>AWS Cost Explorer API Reference</strong></a> — Full reference for the cost breakdown commands used throughout this guide</p>
</li>
<li><p><a href="https://aws.github.io/aws-eks-best-practices/cost_optimization/cfm_framework/"><strong>EKS Best Practices Guide — Cost Optimisation</strong></a> — AWS's official EKS cost optimisation framework</p>
</li>
<li><p><a href="https://github.com/aayostem/eks-cost-optimization"><strong>Companion Repository</strong></a> — All Terraform modules, NodePool templates, VPA manifests, and automation scripts from this guide</p>
</li>
</ul>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ The 2026 FinOps Roadmap: From Cost-Blind Engineer to Cloud Financial Manager ]]>
                </title>
                <description>
                    <![CDATA[ My first AWS bill was $23,000. I had been working at the company for three weeks. Nobody told me. The bill just grew quietly in the background while I was proud of the feature I shipped. A Lambda func ]]>
                </description>
                <link>https://www.freecodecamp.org/news/the-2026-finops-roadmap-from-cost-blind-engineer-to-cloud-financial-manager/</link>
                <guid isPermaLink="false">6a30894af07f26c8d93079b8</guid>
                
                    <category>
                        <![CDATA[ Cloud Computing ]]>
                    </category>
                
                    <category>
                        <![CDATA[ finops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Roadmap ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Ayobami Adejumo ]]>
                </dc:creator>
                <pubDate>Mon, 15 Jun 2026 23:22:50 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/365d29dc-738d-4c21-a9a5-8f818c36cc95.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>My first AWS bill was $23,000. I had been working at the company for three weeks.</p>
<p>Nobody told me. The bill just grew quietly in the background while I was proud of the feature I shipped. A Lambda function that called an external enrichment API on every user event. Clean code. Solid tests. Thirty-two million events that month. At $0.0007 per API call.</p>
<p>My engineering manager forwarded the invoice with two words: "Please explain."</p>
<p>That was the moment I discovered FinOps — not from a conference talk or a certification course, but from the specific shame of having written expensive code and not knowing it until the damage was done.</p>
<p>This roadmap is what I needed that day. A complete, honest guide to transforming from an engineer who builds things that work into an engineer who builds things that work <em>and</em> cost what they should. By the end of this guide, you'll have the skills, the scripts, and the vocabulary to talk about cloud spend the way a CFO and a CTO both want to hear.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-the-four-stages-overview">The Four Stages Overview</a></p>
</li>
<li><p><a href="#heading-stage-1-the-cost-aware-engineer-months-1-to-3">Stage 1: The Cost-Aware Engineer — Months 1 to 3</a></p>
</li>
<li><p><a href="#heading-stage-2-the-optimisation-specialist-months-4-to-8">Stage 2: The Optimisation Specialist — Months 4 to 8</a></p>
</li>
<li><p><a href="#heading-stage-3-the-automation-architect-months-9-to-15">Stage 3: The Automation Architect — Months 9 to 15</a></p>
</li>
<li><p><a href="#heading-stage-4-the-cloud-financial-manager-months-16-to-24">Stage 4: The Cloud Financial Manager — Months 16 to 24</a></p>
</li>
<li><p><a href="#heading-essential-tools-and-certifications">Essential Tools and Certifications</a></p>
</li>
<li><p><a href="#heading-your-90-day-action-plan">Your 90-Day Action Plan</a></p>
</li>
<li><p><a href="#heading-best-practices-summary">Best Practices Summary</a></p>
</li>
<li><p><a href="#heading-resources">Resources</a></p>
</li>
</ul>
<h2 id="heading-what-youll-learn">What You'll Learn</h2>
<ul>
<li><p>How to read your AWS bill as an engineer, not as a passive observer</p>
</li>
<li><p>The exact tagging strategy that makes cost attribution possible</p>
</li>
<li><p>How to right-size EC2 and RDS instances using CloudWatch data you already have</p>
</li>
<li><p>The correct sequence for purchasing Savings Plans — and why sequence matters more than the discount percentage</p>
</li>
<li><p>How to build automated cleanup systems for orphaned resources</p>
</li>
<li><p>How to present cloud cost findings to engineering leadership with data that drives decisions</p>
</li>
<li><p>The chargeback and showback models that make cost accountability stick</p>
</li>
</ul>
<p>Let's begin.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>Before following this roadmap, you should have some skills and tools ready to go.</p>
<p><strong>Knowledge:</strong></p>
<ul>
<li><p>You can deploy an application to AWS (EC2, Lambda, or containers)</p>
</li>
<li><p>You understand basic AWS services: S3, RDS, EC2, VPC, IAM</p>
</li>
<li><p>You're comfortable reading Python and writing simple bash scripts</p>
</li>
<li><p>You know what a pull request is and have gone through at least one code review</p>
</li>
</ul>
<p><strong>Access:</strong></p>
<ul>
<li><p>Read-only access to your AWS billing console and Cost Explorer</p>
</li>
<li><p>AWS CLI v2 configured with at least <code>ReadOnlyAccess</code> policy attached</p>
</li>
<li><p>Python 3.9 or later for running the audit scripts in this guide</p>
</li>
</ul>
<p><strong>Mindset:</strong> You don't need to be a finance expert. But you do need to be willing to look at numbers that might be uncomfortable. Every engineer I've worked with who became excellent at FinOps had one thing in common: they were willing to be the person who asked "but what does this cost?" in a room where nobody else wanted to.</p>
<p><strong>Estimated time:</strong> This roadmap covers 24 months of deliberate skill-building. You can absorb the reading in a few evenings. The practice is the 24 months.</p>
<h2 id="heading-the-four-stages-overview">The Four Stages Overview</h2>
<p>Before going deep, here's the complete picture of where you're going:</p>
<pre><code class="language-plaintext">Stage 1 — Cost-Aware Engineer (Months 1–3)
├── Read your cloud bill and understand it
├── Tag every resource with meaningful metadata
├── Identify your top 5 cost drivers
└── Block your first expensive PR with cost justification

Stage 2 — Optimisation Specialist (Months 4–8)
├── Right-size every over-provisioned resource
├── Implement storage lifecycle policies
├── Move non-production to Spot instances
└── Purchase your first Savings Plan in the right order

Stage 3 — Automation Architect (Months 9–15)
├── Build automated cleanup for orphaned resources
├── Add cost estimation to your CI/CD pipeline
├── Create cost-aware auto-scaling triggers
└── Deploy a self-service FinOps dashboard

Stage 4 — Cloud Financial Manager (Months 16–24)
├── Lead monthly FinOps reviews with engineering leadership
├── Build chargeback models for departments
├── Negotiate enterprise agreements with AWS
└── Forecast cloud spend within 5% variance
</code></pre>
<p>The reason this is a 24-month journey and not a weekend project: each stage builds on the previous one. Engineers who jump straight to Savings Plans without rightsizing first end up paying discounted prices for waste. Engineers who build dashboards before tagging get beautiful charts with no actionable data. The sequence isn't arbitrary.</p>
<h2 id="heading-stage-1-the-cost-aware-engineer-months-1-to-3">Stage 1: The Cost-Aware Engineer — Months 1 to 3</h2>
<h3 id="heading-11-reading-the-bill-like-an-engineer-not-an-accountant">1.1 Reading the Bill Like an Engineer, Not an Accountant</h3>
<p>The default AWS Cost Explorer view shows you service-level totals. That's accounting. What you need is engineering-level decomposition: which specific resources cost money, what business function they serve, and whether each dollar is justified.</p>
<p>Start by pulling a proper breakdown:</p>
<pre><code class="language-bash"># Pull last month's cost breakdown grouped by service
# Run this before touching any optimisation — this is your baseline
aws ce get-cost-and-usage \
  --time-period Start=\((date -d 'last month' +%Y-%m-01),End=\)(date +%Y-%m-01) \
  --granularity MONTHLY \
  --group-by Type=DIMENSION,Key=SERVICE \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table | sort -k3 -rn
</code></pre>
<p>Save the output. Name the file <code>aws-baseline-YYYY-MM.txt</code>. You'll compare every future month against this number. Without a baseline, you can't measure progress — and without measurable progress, you can't make the case to leadership that the work is worth engineering time.</p>
<h4 id="heading-three-questions-for-every-service-in-your-top-5">Three questions for every service in your top 5:</h4>
<p>Most engineers stop at "what is this service?" and never reach the useful question. Here's the framework I use when I first audit an account:</p>
<p>The first question is whether you know what specific business function this service is performing. Not the product name, the function. "S3" isn't an answer. "Storing unprocessed video uploads that sit for 90 days before anyone watches them" is an answer.</p>
<p>The second question is whether the cost is growing, stable, or shrinking when you look at the past three months. A stable \(12,000/month is a different problem from a \)12,000/month line that was $4,000 six months ago.</p>
<p>The third question is what percentage of your total bill this service represents. Optimising a 1% line item while a 40% line item runs unchecked is a common time-wasting trap.</p>
<h3 id="heading-12-the-tagging-strategy-that-actually-survives">1.2 The Tagging Strategy That Actually Survives</h3>
<p>Here's the honest truth about tagging: most tagging strategies die within six months because they're designed for reporting rather than for engineers. Engineers don't tag things well when they're moving fast. The solution isn't to demand more discipline. Instead, it's to make tagging enforced at the infrastructure layer.</p>
<p>Here's the minimal viable tag set (the six tags that cover 90% of attribution needs):</p>
<pre><code class="language-yaml"># These six tags enable cost attribution, accountability, and automated remediation
# Add these to every resource in your AWS account — EC2, RDS, S3, Lambda, everything

Environment: "production" | "staging" | "dev"
Team: "platform" | "backend" | "data" | "ml"
Service: "payment-api" | "fraud-detection" | "user-service"
Owner: "ayo@cloudfrugal.com"     # Person responsible for this resource
CostCenter: "engineering"         # For chargeback reporting
AutoShutdown: "true" | "false"    # Enables automated remediation
</code></pre>
<p>Enforce tags at the Terraform level so they can't be skipped:</p>
<pre><code class="language-hcl"># variables.tf
# Add this to your Terraform root module
# Any plan that creates a resource without these tags will fail validation

variable "required_tags" {
  description = "Tags required on every resource in this account"
  type = map(string)
  
  validation {
    condition = contains(keys(var.required_tags), "Environment") &amp;&amp;
                contains(keys(var.required_tags), "Team") &amp;&amp;
                contains(keys(var.required_tags), "Owner")
    error_message = "required_tags must include Environment, Team, and Owner."
  }
}

# Apply in every resource
resource "aws_instance" "app_server" {
  ami           = data.aws_ami.amazon_linux.id
  instance_type = "t3.medium"

  tags = merge(var.required_tags, {
    Name    = "app-server-${var.environment}"
    Service = "payment-api"
  })
}
</code></pre>
<p>Find everything that's currently untagged:</p>
<pre><code class="language-bash"># List EC2 instances missing the Team tag
# Run this weekly until you hit zero results
aws ec2 describe-instances \
  --query "Reservations[].Instances[?!not_null(Tags[?Key=='Team'].Value | [0])].[InstanceId, InstanceType, State.Name]" \
  --output table
</code></pre>
<p>Once you start finding untagged resources, you'll discover a pattern: the oldest resources in the account are the least tagged, and they're often the most expensive. An EC2 instance from 2021 that predates your tagging policy is exactly the kind of thing that generates a $3,000/month line item nobody can explain.</p>
<h3 id="heading-13-the-cost-aware-code-review">1.3 The Cost-Aware Code Review</h3>
<p>The most underused FinOps practice in engineering teams is reviewing code changes for cost implications before they merge. It takes thirty seconds per PR once you build the habit, and it prevents the kind of problem that opened this guide: the expensive feature that nobody priced before shipping.</p>
<p>Add this section to your PR template:</p>
<pre><code class="language-markdown">## Cost Impact (required for infrastructure and data changes)

- [ ] This change does not affect cloud resource usage
- [ ] New API calls introduced: estimated cost per call $______, calls/month ______
- [ ] New data storage: estimated monthly delta $______
- [ ] Cross-region data transfer introduced: yes / no
- [ ] New external service dependency with per-call pricing: yes / no

If any box other than the first is checked, add a cost estimate before requesting review.
</code></pre>
<p>The discipline is in making cost estimation a first-class review concern, not an afterthought that gets caught by the finance team on the 15th of the month.</p>
<h3 id="heading-stage-1-outcomes">Stage 1 Outcomes</h3>
<p>By the end of month 3, you should have a baseline cost breakdown on file, 100% tag coverage on active resources, identified your top 5 cost drivers with specific reduction targets, and blocked at least one expensive PR with a cost justification that held up in review.</p>
<h2 id="heading-stage-2-the-optimisation-specialist-months-4-to-8">Stage 2: The Optimisation Specialist — Months 4 to 8</h2>
<h3 id="heading-21-right-sizing-the-8020-of-cloud-savings">2.1 Right-Sizing: The 80/20 of Cloud Savings</h3>
<p>The single most reliable source of cloud waste I find in every account I audit is over-provisioned compute.</p>
<p>The pattern is consistent: an engineer provisions an instance at a size that handles their anticipated peak load, the peak never quite materialises at the expected scale, and nobody revisits the instance size because there's no automatic signal that says "this machine is 75% empty."</p>
<p>Make sure you verify actual utilisation before changing anything:</p>
<pre><code class="language-python"># rightsize_analyzer.py
# Finds EC2 instances running below 20% average CPU for 14 days
# These are right-sizing candidates — not automatic deletions

import boto3
from datetime import datetime, timedelta

def find_oversized_instances(region='us-east-1'):
    """
    Returns instances with average CPU below 20% for the last 14 days.
    Low CPU alone doesn't mean right-size — check memory too if CW agent installed.
    """
    ec2 = boto3.client('ec2', region_name=region)
    cw  = boto3.client('cloudwatch', region_name=region)

    reservations = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
    )['Reservations']

    candidates = []

    for r in reservations:
        for inst in r['Instances']:
            iid  = inst['InstanceId']
            itype = inst['InstanceType']
            tags = {t['Key']: t['Value'] for t in inst.get('Tags', [])}

            # Pull 14-day average CPU from CloudWatch
            stats = cw.get_metric_statistics(
                Namespace='AWS/EC2',
                MetricName='CPUUtilization',
                Dimensions=[{'Name': 'InstanceId', 'Value': iid}],
                StartTime=datetime.utcnow() - timedelta(days=14),
                EndTime=datetime.utcnow(),
                Period=1209600,   # One 14-day period
                Statistics=['Average']
            )['Datapoints']

            avg_cpu = stats[0]['Average'] if stats else 0.0

            if avg_cpu &lt; 20.0:
                candidates.append({
                    'instance_id':  iid,
                    'instance_type': itype,
                    'avg_cpu_pct':  round(avg_cpu, 1),
                    'environment':  tags.get('Environment', 'unknown'),
                    'owner':        tags.get('Owner', 'unknown'),
                    'team':         tags.get('Team', 'unknown'),
                })

    return sorted(candidates, key=lambda x: x['avg_cpu_pct'])

if __name__ == '__main__':
    results = find_oversized_instances()
    print(f"\nFound {len(results)} right-sizing candidates:\n")
    for r in results:
        print(f"  {r['instance_id']} ({r['instance_type']}) — "
              f"{r['avg_cpu_pct']}% avg CPU — "
              f"owner: {r['owner']}")
</code></pre>
<p>A word of caution: CPU utilisation below 20% is a signal, not a verdict. Some workloads are memory-intensive or I/O-bound and will show low CPU while being correctly sized. Before acting on any right-sizing recommendation, check memory utilisation (requires the CloudWatch agent) and network I/O patterns alongside CPU.</p>
<h3 id="heading-22-storage-tiering-stop-paying-retail-for-cold-data">2.2 Storage Tiering: Stop Paying Retail for Cold Data</h3>
<p>S3 Standard costs \(0.023 per GB per month. S3 Glacier Deep Archive costs \)0.00099 per GB per month. The difference is a factor of 23. If you have data that you last accessed six months ago and you're keeping it in S3 Standard because nobody set up lifecycle policies, you're paying 23x more than necessary.</p>
<p><strong>The complete S3 lifecycle policy for engineering teams:</strong></p>
<pre><code class="language-json">{
  "Rules": [
    {
      "ID": "application-logs-lifecycle",
      "Status": "Enabled",
      "Filter": {"Prefix": "logs/"},
      "Transitions": [
        {"Days": 30,  "StorageClass": "STANDARD_IA"},
        {"Days": 90,  "StorageClass": "GLACIER_IR"},
        {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
      ],
      "Expiration": {"Days": 2555},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
    },
    {
      "ID": "training-checkpoints-lifecycle",
      "Status": "Enabled",
      "Filter": {"Prefix": "ml-checkpoints/"},
      "Transitions": [
        {"Days": 7,  "StorageClass": "STANDARD_IA"},
        {"Days": 30, "StorageClass": "GLACIER_IR"}
      ],
      "Expiration": {"Days": 90}
    }
  ]
}
</code></pre>
<pre><code class="language-bash"># Apply the lifecycle policy to a bucket
aws s3api put-bucket-lifecycle-configuration \
  --bucket your-logs-bucket \
  --lifecycle-configuration file://lifecycle.json

# Verify it applied correctly
aws s3api get-bucket-lifecycle-configuration \
  --bucket your-logs-bucket
</code></pre>
<h3 id="heading-23-savings-plans-the-sequence-is-everything">2.3 Savings Plans: The Sequence Is Everything</h3>
<p>A Savings Plan is a commitment to spend a minimum dollar amount per hour on AWS compute for one or three years, in exchange for discounts of 30–70% off On-Demand rates. The discount is real. The trap is buying before optimising.</p>
<p><strong>The wrong order:</strong> You have a \(50,000/month EC2 bill. You buy a Savings Plan covering \)35,000/hour. Then you implement right-sizing and Spot instances — and your actual spend drops to \(22,000/month. You've committed to paying \)35,000/month for 12 months against a need of \(22,000. You're paying \)13,000/month for compute you don't use, at a 30% discount. Congratulations on your discounted waste.</p>
<p><strong>The right order:</strong></p>
<pre><code class="language-plaintext">Month 1-2: Right-size all instances using VPA and CloudWatch data
Month 3:   Move staging and development to Spot instances
Month 4:   Migrate compatible workloads to Graviton (20% cheaper)
Month 5:   Add VPC endpoints to eliminate NAT Gateway charges
Month 6:   THEN look at your steady-state On-Demand spend
Month 6+:  Purchase Savings Plans covering 70% of that optimised baseline
</code></pre>
<p><strong>Calculate what to commit to:</strong></p>
<pre><code class="language-bash"># Get your On-Demand EC2 spend for the last 30 days
# This is your rightsized baseline — the number to commit against
aws ce get-cost-and-usage \
  --time-period Start=\((date -d '30 days ago' +%Y-%m-%d),End=\)(date +%Y-%m-%d) \
  --granularity DAILY \
  --filter '{
    "And": [
      {"Dimensions": {"Key": "SERVICE",       "Values": ["Amazon Elastic Compute Cloud - Compute"]}},
      {"Dimensions": {"Key": "PURCHASE_TYPE", "Values": ["On-Demand"]}}
    ]
  }' \
  --metrics UnblendedCost \
  --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.UnblendedCost.Amount}' \
  --output table

# Get AWS's own recommendation for what to commit
aws savingsplans get-savings-plans-purchase-recommendation \
  --savings-plans-type COMPUTE_SP \
  --term-in-years ONE_YEAR \
  --payment-option NO_UPFRONT \
  --lookback-period-in-days THIRTY_DAYS
</code></pre>
<h2 id="heading-stage-3-the-automation-architect-months-9-to-15">Stage 3: The Automation Architect — Months 9 to 15</h2>
<h3 id="heading-31-the-orphaned-resource-problem-and-why-it-never-fixes-itself">3.1 The Orphaned Resource Problem — And Why It Never Fixes Itself</h3>
<p>Orphaned resources are the cloud equivalent of a gym membership you forgot to cancel. They exist, they charge you, but nobody notices until the annual audit.</p>
<p>The root cause isn't laziness. It's the absence of lifecycle management at the infrastructure layer. When an engineer spins up an EC2 instance for a one-week experiment and then leaves the company, there's no automatic signal that the instance is now orphaned. It sits there, billing $140/month, until someone hunts it down.</p>
<p>The fix is a weekly automated audit that surfaces candidates for deletion and notifies the registered owner, not a process change that depends on engineers remembering to clean up.</p>
<pre><code class="language-python"># orphan_reporter.py
# Runs every Sunday via EventBridge → Lambda
# Posts a Slack report of orphaned resources for human review
# DOES NOT auto-delete — deletion requires a human decision

import boto3
import json
import urllib.request
from datetime import datetime, timedelta, timezone

SLACK_WEBHOOK = 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
UNATTACHED_VOLUME_AGE_DAYS = 14
SNAPSHOT_AGE_DAYS = 90


def find_orphaned_resources():
    ec2 = boto3.client('ec2')
    report = {'monthly_waste_usd': 0, 'items': []}

    # Unattached EBS volumes
    for vol in ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )['Volumes']:
        age = (datetime.now(timezone.utc) - vol['CreateTime']).days
        if age &gt;= UNATTACHED_VOLUME_AGE_DAYS:
            cost = round(vol['Size'] * 0.08, 2)  # gp3 rate
            tags = {t['Key']: t['Value'] for t in vol.get('Tags', [])}
            report['items'].append({
                'type':  'Unattached EBS Volume',
                'id':    vol['VolumeId'],
                'detail': f"{vol['Size']}GB {vol['VolumeType']} — {age} days old",
                'owner': tags.get('Owner', 'unknown'),
                'monthly_cost_usd': cost,
            })
            report['monthly_waste_usd'] += cost

    # Unassociated Elastic IPs
    for addr in ec2.describe_addresses()['Addresses']:
        if 'AssociationId' not in addr:
            report['items'].append({
                'type':  'Unassociated Elastic IP',
                'id':    addr.get('AllocationId', addr['PublicIp']),
                'detail': addr['PublicIp'],
                'owner': 'unknown',
                'monthly_cost_usd': 3.60,
            })
            report['monthly_waste_usd'] += 3.60

    # Old snapshots
    cutoff = (datetime.now(timezone.utc) - timedelta(days=SNAPSHOT_AGE_DAYS)).isoformat()
    for snap in ec2.describe_snapshots(OwnerIds=['self'])['Snapshots']:
        if snap['StartTime'].isoformat() &lt; cutoff:
            cost = round(snap.get('VolumeSize', 0) * 0.05, 2)
            report['items'].append({
                'type':  f'Snapshot ({SNAPSHOT_AGE_DAYS}+ days old)',
                'id':    snap['SnapshotId'],
                'detail': f"Created {snap['StartTime'].strftime('%Y-%m-%d')}",
                'owner': 'unknown',
                'monthly_cost_usd': cost,
            })
            report['monthly_waste_usd'] += cost

    return report


def post_to_slack(report):
    lines = [
        f":money_with_wings: *Weekly Orphaned Resource Report*",
        f"Found *{len(report['items'])} orphaned resources* "
        f"costing *${report['monthly_waste_usd']:.2f}/month*\n",
    ]
    for item in report['items'][:20]:  # Cap at 20 lines to stay readable
        lines.append(
            f"• `{item['type']}` {item['id']} — {item['detail']} "
            f"— *${item['monthly_cost_usd']:.2f}/mo* — owner: {item['owner']}"
        )
    lines.append("\nReview and delete anything no longer needed.")

    req = urllib.request.Request(
        SLACK_WEBHOOK,
        data=json.dumps({'text': '\n'.join(lines)}).encode(),
        headers={'Content-Type': 'application/json'}
    )
    urllib.request.urlopen(req)


def lambda_handler(event, context):
    report = find_orphaned_resources()
    post_to_slack(report)
    return {
        'items_found': len(report['items']),
        'monthly_waste': report['monthly_waste_usd'],
    }
</code></pre>
<h3 id="heading-32-cost-estimation-in-your-cicd-pipeline">3.2 Cost Estimation in Your CI/CD Pipeline</h3>
<p>The goal is to catch expensive infrastructure changes at the PR stage — before they deploy and before they generate a billing surprise.</p>
<pre><code class="language-yaml"># .github/workflows/cost-check.yml
# Runs on any PR that touches infrastructure files
# Uses Infracost to estimate the monthly cost delta

name: Infrastructure Cost Check

on:
  pull_request:
    paths:
      - 'terraform/**'
      - 'infrastructure/**'
      - '*.tf'

jobs:
  cost-estimate:
    name: Estimate monthly cost change
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Setup Infracost
        uses: infracost/actions/setup@v3
        with:
          api-key: ${{ secrets.INFRACOST_API_KEY }}

      - name: Generate cost estimate
        run: |
          infracost breakdown \
            --path terraform/ \
            --format json \
            --out-file /tmp/infracost.json

      - name: Post cost diff to PR
        uses: infracost/actions/comment@v3
        with:
          path: /tmp/infracost.json
          behavior: update

      - name: Block if monthly increase exceeds threshold
        run: |
          MONTHLY_DELTA=$(cat /tmp/infracost.json | \
            jq '.projects[0].diff.totalMonthlyCost' | tr -d '"')

          echo "Estimated monthly cost change: \$$MONTHLY_DELTA"

          # Fail the PR if this change adds more than $500/month
          python3 -c "
          import sys
          delta = float('$MONTHLY_DELTA')
          if delta &gt; 500:
              print(f'PR blocked: estimated +\\({delta:.2f}/month exceeds \\)500 threshold')
              sys.exit(1)
          else:
              print(f'Cost check passed: estimated +\${delta:.2f}/month')
          "
</code></pre>
<h2 id="heading-stage-4-the-cloud-financial-manager-months-16-to-24">Stage 4: The Cloud Financial Manager — Months 16 to 24</h2>
<h3 id="heading-41-leading-finops-reviews-with-executives">4.1 Leading FinOps Reviews with Executives</h3>
<p>By month 16, you have the data. What changes at Stage 4 is the audience. You're no longer presenting to engineers who understand instance types and NAT Gateway pricing. You're presenting to a CTO who wants to know if the infrastructure investment is proportional to the business value it produces, and a CFO who wants to know when the line will stop going up.</p>
<p>The vocabulary shift is simple but important. You stop saying "we right-sized our EC2 instances" and start saying "we reduced our infrastructure unit cost by 28% while maintaining the same request throughput." You stop saying "we eliminated NAT Gateway charges" and start saying "we closed a $6,400/month gap between what we were paying and what was necessary."</p>
<p>The metric that anchors every executive FinOps conversation is cost per business unit. Not total bill (cost per API call, cost per user, cost per transaction, cost per model inference). That ratio tells the story of whether your infrastructure efficiency is improving as the business scales.</p>
<pre><code class="language-python"># unit_economics.py
# Calculate cost per transaction — the metric that matters to leadership

import boto3
from datetime import datetime, timedelta

def calculate_cost_per_transaction(service_name, transaction_count, days_back=30):
    """
    Returns cost per transaction for a given service over the last N days.
    transaction_count: total transactions for the same period (from your metrics)
    """
    ce = boto3.client('ce')

    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': (datetime.now() - timedelta(days=days_back)).strftime('%Y-%m-%d'),
            'End':   datetime.now().strftime('%Y-%m-%d'),
        },
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        Filter={
            'Tags': {
                'Key':    'Service',
                'Values': [service_name]
            }
        }
    )

    total_cost = sum(
        float(period['Total']['UnblendedCost']['Amount'])
        for period in response['ResultsByTime']
    )

    cost_per_txn = total_cost / transaction_count if transaction_count &gt; 0 else 0

    return {
        'service':           service_name,
        'period_days':       days_back,
        'total_cost_usd':    round(total_cost, 2),
        'transactions':      transaction_count,
        'cost_per_txn_usd':  round(cost_per_txn, 6),
    }


# Example: payment service processed 4.2M transactions this month
result = calculate_cost_per_transaction('payment-api', 4_200_000)
print(f"Cost per transaction: ${result['cost_per_txn_usd']:.6f}")
print(f"Total infrastructure cost: ${result['total_cost_usd']:,.2f}")
</code></pre>
<h3 id="heading-42-the-chargeback-and-showback-models">4.2 The Chargeback and Showback Models</h3>
<p>Chargeback means actually billing departments for their cloud usage. Showback means showing departments their usage costs without the internal billing transfer. Both create the same outcome: engineers start caring about what they consume because someone they work with is paying attention to it.</p>
<pre><code class="language-python"># showback_report.py
# Generates monthly cost-by-team report for distribution to engineering leads

import boto3
from datetime import datetime

def generate_team_showback():
    ce = boto3.client('ce')

    response = ce.get_cost_and_usage(
        TimePeriod={
            'Start': datetime.now().replace(day=1).strftime('%Y-%m-%d'),
            'End':   datetime.now().strftime('%Y-%m-%d'),
        },
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[
            {'Type': 'TAG',       'Key': 'Team'},
            {'Type': 'DIMENSION', 'Key': 'SERVICE'},
        ]
    )

    by_team = {}
    for group in response['ResultsByTime'][0].get('Groups', []):
        team    = group['Keys'][0].replace('Team$', '') or 'untagged'
        service = group['Keys'][1]
        cost    = float(group['Metrics']['UnblendedCost']['Amount'])

        if team not in by_team:
            by_team[team] = {'total': 0, 'services': {}}
        by_team[team]['total'] += cost
        by_team[team]['services'][service] = round(cost, 2)

    # Print sorted by total cost descending
    print(f"\n{'='*52}")
    print(f"  Month-to-Date Cloud Spend by Team")
    print(f"  Generated: {datetime.now().strftime('%Y-%m-%d')}")
    print(f"{'='*52}\n")

    for team, data in sorted(by_team.items(), key=lambda x: x[1]['total'], reverse=True):
        print(f"  {team:&lt;20} ${data['total']:&gt;10,.2f}/month")
        top_services = sorted(data['services'].items(), key=lambda x: x[1], reverse=True)[:3]
        for svc, cost in top_services:
            print(f"    └─ {svc:&lt;30} ${cost:&gt;8,.2f}")
    print()

generate_team_showback()
</code></pre>
<h2 id="heading-essential-tools-and-certifications">Essential Tools and Certifications</h2>
<p>The tools that matter at each stage of this roadmap:</p>
<table>
<thead>
<tr>
<th>Stage</th>
<th>Tool</th>
<th>Why It Matters</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>AWS Cost Explorer</td>
<td>Free, built-in, the starting point for all cost analysis</td>
</tr>
<tr>
<td>1</td>
<td>AWS CLI <code>ce</code> commands</td>
<td>Scriptable cost queries — dashboards can't be automated</td>
</tr>
<tr>
<td>2</td>
<td>AWS Compute Optimizer</td>
<td>ML-powered rightsizing recommendations for EC2 and RDS</td>
</tr>
<tr>
<td>2</td>
<td>VPA (Kubernetes)</td>
<td>Pod-level rightsizing recommendations using actual usage</td>
</tr>
<tr>
<td>3</td>
<td>Infracost</td>
<td>PR-level cost estimation for Terraform changes</td>
</tr>
<tr>
<td>3</td>
<td>AWS Budgets</td>
<td>Proactive alerts — catches problems before the monthly invoice</td>
</tr>
<tr>
<td>4</td>
<td>AWS Cost and Usage Report + Athena</td>
<td>SQL-level billing analysis at any granularity</td>
</tr>
<tr>
<td>4</td>
<td>CloudHealth or Vantage</td>
<td>Multi-account, multi-cloud cost management</td>
</tr>
</tbody></table>
<p><strong>The one certification worth your time:</strong> FinOps Certified Practitioner from the FinOps Foundation. It takes 20 hours to prepare and $300 to sit. It signals to hiring managers and clients that you understand the discipline formally — which matters when you're the person leading FinOps conversations at the executive level.</p>
<h2 id="heading-your-90-day-action-plan">Your 90-Day Action Plan</h2>
<h3 id="heading-month-1-foundation">Month 1 — Foundation:</h3>
<p>Enable Cost Explorer if it isn't already on. Pull the baseline command from Section 1.1 and save the output. Run the untagged resource query from Section 1.2 and document how many resources are missing tags. Find your top three cost drivers. Present the findings to your engineering manager — not as a problem, but as an opportunity with a dollar figure attached.</p>
<h3 id="heading-month-2-quick-wins">Month 2 — Quick Wins:</h3>
<p>Run the rightsizing analyser from Section 2.1 on your EC2 fleet. Downsize the three highest-confidence candidates. Apply S3 lifecycle policies to your two largest buckets. Create VPC endpoints for S3, ECR, and DynamoDB. Estimate the savings from each action and document them against your baseline.</p>
<h3 id="heading-month-3-automation-and-habits">Month 3 — Automation and Habits:</h3>
<p>Deploy the orphan reporter Lambda on a Sunday schedule. Add the cost check GitHub Action to your infrastructure repository. Start a monthly FinOps review meeting — even if it's just you and one other engineer. Build the habit before you need the audience.</p>
<h2 id="heading-best-practices-summary">Best Practices Summary</h2>
<p>✅ <strong>Do:</strong> Establish a cost baseline before any optimisation. The number is meaningless without a comparison point.</p>
<p>✅ <strong>Do:</strong> Right-size before buying Savings Plans. Always. The sequence changes the outcome.</p>
<p>✅ <strong>Do:</strong> Enforce tagging at the infrastructure layer — Terraform or CloudFormation — not as a process reminder.</p>
<p>✅ <strong>Do:</strong> Move staging and development to Spot instances. The interruption rate is manageable, while the 70% cost difference is not.</p>
<p>✅ <strong>Do:</strong> Add VPC endpoints for S3, ECR, and DynamoDB before reviewing data transfer costs. It's a 30-minute fix for a multi-thousand-dollar line item.</p>
<p>✅ <strong>Do:</strong> Present cost findings as cost-per-business-metric, not as total bill. "We reduced cost per transaction from \(0.0021 to \)0.0013" is a business result. "$38,000/month reduction" is an accounting result.</p>
<p>❌ <strong>Don't:</strong> Buy Savings Plans on an unoptimised baseline. You'll lock in discounted waste.</p>
<p>❌ <strong>Don't:</strong> Build FinOps dashboards before tagging is complete. Beautiful charts with no attribution data answer no questions.</p>
<p>❌ <strong>Don't:</strong> Run orphaned resource cleanup without human review first. Run in report-only mode for two weeks, verify the candidates are genuinely orphaned, then add deletion logic.</p>
<h2 id="heading-resources">Resources</h2>
<ul>
<li><p><a href="https://www.finops.org/framework/"><strong>FinOps Foundation Framework</strong></a> — The practitioner framework that defines the Inform, Optimise, and Operate cycle this roadmap is built on</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/cost-management/latest/APIReference/API_GetCostAndUsage.html"><strong>AWS Cost Explorer API Reference</strong></a> — Full reference for the cost query commands used throughout this guide</p>
</li>
<li><p><a href="https://aws.amazon.com/compute-optimizer/"><strong>AWS Compute Optimizer</strong></a> — AWS's own rightsizing recommendation service; complements the manual analysis in Stage 2</p>
</li>
<li><p><a href="https://www.infracost.io/docs/"><strong>Infracost Documentation</strong></a> — Setup guide for the PR-level cost estimation tool in Stage 3</p>
</li>
<li><p><a href="https://learn.finops.org/path/finops-certified-practitioner"><strong>FinOps Certified Practitioner Exam</strong></a> — The certification referenced in the tools section</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/savingsplans/latest/userguide/what-is-savings-plans.html"><strong>AWS Savings Plans Documentation</strong></a> — The authoritative reference on commitment types, coverage rules, and purchase strategy</p>
</li>
<li><p><a href="https://github.com/aayostem"><strong>Companion Repository</strong></a> — All scripts from this guide, including the rightsizing analyser, orphan reporter, and showback report generator</p>
</li>
</ul>
<p><a href="https://github.com/aayostem"><em>Ayobami Adejumo</em></a> <em>is a senior platform engineer and FinOps consultant. He has audited AWS infrastructure for 20+ Series A and Series B companies. He is an active FinOps Foundation Supporter</em></p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ The AWS FinOps Guide for Series A Startups: The 8 Cost Patterns That Appear After Product-Market Fit ]]>
                </title>
                <description>
                    <![CDATA[ You raised your Series A. Engineering hired fast. Features shipped faster. And somewhere between month six and month twelve, someone forwarded you an AWS Cost Explorer screenshot with a line that only ]]>
                </description>
                <link>https://www.freecodecamp.org/news/the-aws-finops-guide-for-series-a-startups/</link>
                <guid isPermaLink="false">6a1f046fcf96043972a575f0</guid>
                
                    <category>
                        <![CDATA[ startup ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ finops ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Ayobami Adejumo ]]>
                </dc:creator>
                <pubDate>Tue, 02 Jun 2026 16:27:27 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/e4bbaeaf-810e-4ebb-9c81-d2183cac6df6.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>You raised your Series A. Engineering hired fast. Features shipped faster. And somewhere between month six and month twelve, someone forwarded you an AWS Cost Explorer screenshot with a line that only goes up.</p>
<p>That line isn't random. It follows a pattern. The same eight patterns, at the same growth stage, at almost every company I've audited.</p>
<p>This guide names all eight, shows you exactly where to look, and gives you the fix for each one. By the time you finish reading, you'll know which leaks are draining your runway — and what to do about them this week.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-who-this-guide-is-for">Who This Guide Is For</a></p>
</li>
<li><p><a href="#heading-before-you-start-establish-your-baseline">Before You Start: Establish Your Baseline</a></p>
</li>
<li><p><a href="#heading-pattern-1-the-new-hire-experiment-tax">Pattern 1: The New Hire Experiment Tax</a></p>
</li>
<li><p><a href="#heading-pattern-2-staging-environment-proliferation">Pattern 2: Staging Environment Proliferation</a></p>
</li>
<li><p><a href="#heading-pattern-3-the-nat-gateway-tax">Pattern 3: The NAT Gateway Tax</a></p>
</li>
<li><p><a href="#heading-pattern-4-the-savings-plan-timing-mistake">Pattern 4: The Savings Plan Timing Mistake</a></p>
</li>
<li><p><a href="#heading-pattern-5-cross-az-data-transfer">Pattern 5: Cross-AZ Data Transfer</a></p>
</li>
<li><p><a href="#heading-pattern-6-the-gp2-volume-trap">Pattern 6: The gp2 Volume Trap</a></p>
</li>
<li><p><a href="#heading-pattern-7-the-infinite-log-trap">Pattern 7: The Infinite Log Trap</a></p>
</li>
<li><p><a href="#heading-pattern-8-the-orphaned-resource-collector">Pattern 8: The Orphaned Resource Collector</a></p>
</li>
<li><p><a href="#heading-the-full-savings-summary">The Full Savings Summary</a></p>
</li>
<li><p><a href="#heading-what-to-do-this-week">What to Do This Week</a></p>
</li>
<li><p><a href="#heading-resources">Resources</a></p>
</li>
</ul>
<h2 id="heading-who-this-guide-is-for">Who This Guide Is For</h2>
<p>This guide is written for engineers, CTOs, and technical co-founders at Series A companies — typically 15 to 80 engineers, AWS bills between \(20,000 and \)150,000 per month, and a finance team that has recently started paying attention to the infrastructure line.</p>
<p>You don't need a dedicated FinOps team. You need one engineer, one afternoon per week, and the eight patterns in this guide.</p>
<p><strong>What you should have before starting:</strong></p>
<ul>
<li><p>AWS account access with Cost Explorer enabled</p>
</li>
<li><p>AWS CLI v2 configured (<code>aws configure</code>)</p>
</li>
<li><p>Basic familiarity with EC2, RDS, EBS, and S3</p>
</li>
<li><p>A Cost Explorer bookmark — you will use it constantly</p>
</li>
</ul>
<p><strong>Estimated time to complete all fixes:</strong> 8–20 engineering hours spread across two sprints. The reading takes around 20 minutes. The highest-ROI fix (Pattern 3) takes about 30 minutes.</p>
<h2 id="heading-before-you-start-establish-your-baseline">Before You Start: Establish Your Baseline</h2>
<p>Don't skip this step. Optimization without a baseline is just guessing. Run this command before touching anything:</p>
<pre><code class="language-bash"># Pull last month's AWS cost breakdown by service
# This becomes your before number — save it somewhere
aws ce get-cost-and-usage \
  --time-period Start=\((date -d 'last month' +%Y-%m-01),End=\)(date +%Y-%m-01) \
  --granularity MONTHLY \
  --group-by Type=DIMENSION,Key=SERVICE \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.UnblendedCost.Amount}' \
  --output table | sort -k3 -rn
</code></pre>
<p>Then screenshot the output. Name the file <code>aws-baseline-YYYY-MM.png</code>. You'll compare against this after each fix to verify actual savings.</p>
<p>The typical breakdown at Series A looks like this:</p>
<table>
<thead>
<tr>
<th>AWS Service</th>
<th>% of Bill</th>
<th>Waste Potential</th>
</tr>
</thead>
<tbody><tr>
<td>EC2 (compute)</td>
<td>45–55%</td>
<td>High</td>
</tr>
<tr>
<td>Data Transfer</td>
<td>15–20%</td>
<td>Very High</td>
</tr>
<tr>
<td>RDS</td>
<td>10–15%</td>
<td>Medium</td>
</tr>
<tr>
<td>EBS</td>
<td>8–12%</td>
<td>Medium</td>
</tr>
<tr>
<td>CloudWatch</td>
<td>3–6%</td>
<td>Medium</td>
</tr>
<tr>
<td>Load Balancers</td>
<td>3–5%</td>
<td>Low</td>
</tr>
</tbody></table>
<p>Now let's go through each pattern.</p>
<h2 id="heading-pattern-1-the-new-hire-experiment-tax">Pattern 1: The New Hire Experiment Tax</h2>
<p>Every engineering hire needs a development environment. This is expected. What's not expected is what happens after the feature ships: nothing.</p>
<p>The environment keeps running. At \(0.192/hour for an m5.xlarge, a forgotten dev environment costs \)138/month. Ten engineers who each forgot one environment is $1,380/month — for infrastructure that's doing precisely nothing.</p>
<p>This pattern accelerates after a Series A because hiring moves fast. A new engineer joins on Monday, spins up an EC2, an RDS, and a namespace in the dev cluster, ships the feature by Friday, and moves to the next ticket. The environment isn't on anyone's radar. There's no off-boarding process for dev resources.</p>
<p><strong>What the waste looks like:</strong></p>
<pre><code class="language-text">Dev environment for Alice (feature/payment-flow):
  EC2 m5.xlarge — last CPU activity: 23 days ago
  RDS db.t3.medium — last connection: 19 days ago
  EKS namespace — last pod scheduled: 15 days ago
  Monthly cost: $187
  Status: running
</code></pre>
<p><strong>Finding it:</strong></p>
<pre><code class="language-bash"># Find EC2 instances with average CPU below 5% for the last 14 days
# These are idle instances — candidates for shutdown or termination
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --period 1209600 \
  --statistics Average \
  --start-time $(date -d '14 days ago' --iso-8601=seconds) \
  --end-time $(date --iso-8601=seconds) \
  --dimensions Name=InstanceId,Value=YOUR_INSTANCE_ID \
  --query 'Datapoints[*].{Average:Average}' \
  --output table
</code></pre>
<h3 id="heading-the-fix-an-automatic-idle-instance-stopper">The Fix — an Automatic Idle Instance Stopper:</h3>
<p>The Lambda below runs every night at 22:00. It checks every EC2 instance tagged <code>Environment=dev</code> for CPU utilisation over the past seven days. Any instance averaging below 5% gets stopped automatically. An SNS notification goes to the engineer's email before the stop happens, giving them a chance to override it by adding a <code>KeepAlive=true</code> tag.</p>
<pre><code class="language-python"># idle_environment_stopper.py
# Deploy as a Lambda function triggered by EventBridge on schedule: cron(0 22 * * ? *)
# This stops idle dev environments before they run through the night and weekend

import boto3
from datetime import datetime, timedelta, timezone

ec2 = boto3.client('ec2')
cloudwatch = boto3.client('cloudwatch')
sns = boto3.client('sns')

IDLE_CPU_THRESHOLD = 5.0      # Stop instances below this average CPU %
IDLE_DAYS = 7                  # Look back 7 days of CloudWatch data
SNS_TOPIC_ARN = 'arn:aws:sns:us-east-1:YOUR_ACCOUNT:dev-environment-alerts'

def get_average_cpu(instance_id):
    """Return the 7-day average CPU utilisation for an EC2 instance."""
    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/EC2',
        MetricName='CPUUtilization',
        Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
        StartTime=datetime.now(timezone.utc) - timedelta(days=IDLE_DAYS),
        EndTime=datetime.now(timezone.utc),
        Period=604800,  # One 7-day period
        Statistics=['Average']
    )
    datapoints = response.get('Datapoints', [])
    return datapoints[0]['Average'] if datapoints else 0.0

def lambda_handler(event, context):
    """Stop idle dev instances and notify their owners."""
    
    # Find all running dev instances
    response = ec2.describe_instances(
        Filters=[
            {'Name': 'instance-state-name', 'Values': ['running']},
            {'Name': 'tag:Environment', 'Values': ['dev', 'development']},
        ]
    )

    stopped = []
    skipped = []

    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            instance_id = instance['InstanceId']
            tags = {t['Key']: t['Value'] for t in instance.get('Tags', [])}

            # Skip instances explicitly marked to keep alive
            if tags.get('KeepAlive', '').lower() == 'true':
                skipped.append(instance_id)
                continue

            avg_cpu = get_average_cpu(instance_id)

            if avg_cpu &lt; IDLE_CPU_THRESHOLD:
                # Notify the owner before stopping
                owner = tags.get('Owner', 'unknown')
                sns.publish(
                    TopicArn=SNS_TOPIC_ARN,
                    Subject=f'Dev environment stopped: {instance_id}',
                    Message=(
                        f'Instance {instance_id} (Owner: {owner}) had {avg_cpu:.1f}% average CPU '
                        f'over {IDLE_DAYS} days and has been stopped.\n\n'
                        f'To prevent this, add the tag: KeepAlive=true\n'
                        f'To restart: aws ec2 start-instances --instance-ids {instance_id}'
                    )
                )
                ec2.stop_instances(InstanceIds=[instance_id])
                stopped.append({'id': instance_id, 'owner': owner, 'avg_cpu': avg_cpu})

    print(f"Stopped {len(stopped)} idle instances. Skipped {len(skipped)} keep-alive instances.")
    return {'stopped': stopped, 'skipped': skipped}
</code></pre>
<p><strong>Monthly savings:</strong> \(1,000–\)2,000 depending on team size and how long the pattern has been running.</p>
<h2 id="heading-pattern-2-staging-environment-proliferation">Pattern 2: Staging Environment Proliferation</h2>
<p>Staging starts as one environment. Then the frontend team needs their own because the backend team keeps breaking theirs. Then the ML team needs isolated compute. Then QA needs a stable environment for integration tests.</p>
<p>Before anyone noticed, you have four staging environments running 24/7 — each one idle for 16 hours of every day.</p>
<p>The waste isn't in the existence of the environments. It's in the schedule. Staging environments don't need to run at 3am.</p>
<p><strong>What the waste looks like:</strong></p>
<pre><code class="language-text">staging-frontend:   $250/month   Used: Mon-Fri 09:00-18:00
staging-backend:    $250/month   Used: Mon-Fri 09:00-18:00
staging-ml:         $250/month   Used: Mon-Fri 10:00-17:00
staging-qa:         $250/month   Used: Mon-Fri 09:00-17:00
Total:            $1,000/month   Running: 24 hours/day, 7 days/week
Actual usage:        ~35%        You are paying 100%
</code></pre>
<p><strong>Finding it:</strong></p>
<pre><code class="language-bash"># Find EKS node groups tagged as staging with their current status
aws eks list-nodegroups --cluster-name your-cluster-name --output table

# Check EC2 instances tagged staging and their launch time
# Any instance running &gt; 30 days with no weekend stop schedule is a candidate
aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=staging" "Name=instance-state-name,Values=running" \
  --query 'Reservations[*].Instances[*].{ID:InstanceId,Type:InstanceType,Launch:LaunchTime}' \
  --output table
</code></pre>
<h3 id="heading-the-fix-scheduled-start-and-stop-with-aws-instance-scheduler">The Fix — Scheduled Start and Stop with AWS Instance Scheduler:</h3>
<pre><code class="language-bash"># Option 1: Tag-based scheduling with AWS Instance Scheduler (CloudFormation solution)
# Add these tags to your staging EC2 instances and RDS clusters:
# Schedule: office-hours
# This starts instances at 08:00 and stops them at 20:00 Mon-Fri
# Weekend: completely off

# Option 2: Quick Lambda-based solution — stop all staging at 20:00 weekdays
aws events put-rule \
  --schedule-expression "cron(0 20 ? * MON-FRI *)" \
  --name stop-staging-environments \
  --state ENABLED

# The stop Lambda — same pattern as Pattern 1 but targets staging tag
# Add a corresponding start rule at 07:30 Mon-Fri
</code></pre>
<h3 id="heading-consolidation-in-addition-to-scheduling">Consolidation in Addition to Scheduling</h3>
<p>If frontend and backend share a database schema, consolidate them into one shared staging environment with namespace-level isolation. The combined cost is lower than two separate environments:</p>
<pre><code class="language-yaml"># One shared staging cluster with namespace isolation
# frontend-staging and backend-staging share nodes via Karpenter
# but are isolated by namespace-level network policies
apiVersion: v1
kind: Namespace
metadata:
  name: staging-frontend
  labels:
    environment: staging
    team: frontend
---
apiVersion: v1
kind: Namespace
metadata:
  name: staging-backend
  labels:
    environment: staging
    team: backend
</code></pre>
<p><strong>The math:</strong></p>
<table>
<thead>
<tr>
<th>Scenario</th>
<th>Monthly cost</th>
</tr>
</thead>
<tbody><tr>
<td>Before: 4 environments, always on</td>
<td>$1,000</td>
</tr>
<tr>
<td>After: 2 consolidated environments, office hours only</td>
<td>$290</td>
</tr>
<tr>
<td>Monthly savings</td>
<td>$710</td>
</tr>
</tbody></table>
<h2 id="heading-pattern-3-the-nat-gateway-tax">Pattern 3: The NAT Gateway Tax</h2>
<p>NAT Gateway is the most consistently underestimated line item on every AWS bill I've audited. It charges $0.045 per GB of data processed — and in EKS clusters, a staggering amount of traffic flows through it by default.</p>
<p>Every pod that pulls a container image from ECR goes through NAT Gateway. Every Lambda that writes to S3 goes through NAT Gateway. Every service that polls SQS, queries DynamoDB, or calls the Secrets Manager API goes through NAT Gateway — unless you have configured VPC endpoints.</p>
<p>VPC endpoints create a private connection between your VPC and the AWS service. Traffic routes through the AWS backbone instead of NAT Gateway. The data transfer becomes free.</p>
<p><strong>What the waste looks like:</strong></p>
<pre><code class="language-bash"># Run this to see your current NAT Gateway data processing bill
aws ce get-cost-and-usage \
  --time-period Start=\((date -d 'last month' +%Y-%m-01),End=\)(date +%Y-%m-01) \
  --granularity MONTHLY \
  --filter '{
    "Dimensions": {
      "Key": "USAGE_TYPE",
      "Values": ["NatGateway-Bytes", "NatGateway-Hours"]
    }
  }' \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Total.UnblendedCost.Amount' \
  --output text
</code></pre>
<p>If this number is above \(200, you have a NAT Gateway problem. At most Series A companies running EKS, it is between \)800 and $6,000.</p>
<h3 id="heading-the-fix-vpc-endpoints-for-the-four-highest-traffic-aws-services">The Fix — VPC Endpoints for the Four Highest-traffic AWS Services:</h3>
<pre><code class="language-bash"># Get your VPC ID and route table ID first
VPC_ID=$(aws ec2 describe-vpcs \
  --filters "Name=tag:Name,Values=your-vpc-name" \
  --query 'Vpcs[0].VpcId' --output text)

ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=$VPC_ID" "Name=association.main,Values=true" \
  --query 'RouteTables[0].RouteTableId' --output text)

# S3 gateway endpoint — free to create, eliminates all S3 NAT charges
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids $ROUTE_TABLE_ID

# DynamoDB gateway endpoint — also free
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --service-name com.amazonaws.us-east-1.dynamodb \
  --route-table-ids $ROUTE_TABLE_ID

# ECR API endpoint — eliminates NAT charges on every container pull
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.ecr.api \
  --subnet-ids $(aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Tier,Values=private" \
    --query 'Subnets[*].SubnetId' --output text)

# ECR Docker endpoint — required alongside ECR API for image pulls
aws ec2 create-vpc-endpoint \
  --vpc-id $VPC_ID \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.ecr.dkr \
  --subnet-ids $(aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Tier,Values=private" \
    --query 'Subnets[*].SubnetId' --output text)
</code></pre>
<p>When explaining this to your CFO, call it the NAT tax. They understand taxes. "We're paying a $0.045/GB tax on internal network traffic that we can eliminate in 30 minutes" lands better than "data processing bytes."</p>
<p><strong>Monthly savings:</strong> \(2,000–\)8,000 depending on your container pull frequency and S3 usage.</p>
<h2 id="heading-pattern-4-the-savings-plan-timing-mistake">Pattern 4: The Savings Plan Timing Mistake</h2>
<p>A Savings Plan is a commitment to spend a fixed dollar amount per hour on AWS compute for one or three years in exchange for a 30–70% discount. The math is attractive. The timing is where teams go wrong.</p>
<p>When the bill gets large, the instinct is to commit. Buy the Savings Plan, reduce the bill, show the CFO. The problem: if you haven't rightsized first, you're committing to pay for waste at a discount. When you rightsize later, your actual spend drops below your commitment — and you pay for compute you're not using.</p>
<p><strong>What wrong order looks like:</strong></p>
<pre><code class="language-text">Step 1: AWS bill is $100,000/month
Step 2: Buy $70,000/hour Savings Plan commitment
Step 3: Rightsize instances — actual spend drops to $60,000
Step 4: Savings Plan covers \(70,000 but you only use \)60,000
Step 5: You pay $28,000/month for compute you do not use
         (Savings Plan discount applied to the overage)
         
Net result: You locked in waste for 12 months
</code></pre>
<p><strong>What right order looks like:</strong></p>
<pre><code class="language-text">Step 1: Rightsize instances — spend drops from \(100,000 to \)60,000
Step 2: Add Spot for staging — spend drops from \(60,000 to \)45,000
Step 3: Migrate compatible workloads to Graviton — spend drops to $36,000
Step 4: NOW buy a Savings Plan covering $25,000/month (70% of steady-state)
Step 5: Effective monthly cost: \(12,500 for committed + \)11,000 on-demand = $23,500

Net result: $76,500/month saved versus the original bill
</code></pre>
<p>How to check what you should commit to:</p>
<pre><code class="language-bash"># View your last 30 days of EC2 On-Demand spend
# This is your rightsized baseline — what you actually use after optimisation
aws ce get-cost-and-usage \
  --time-period Start=\((date -d '30 days ago' +%Y-%m-%d),End=\)(date +%Y-%m-%d) \
  --granularity DAILY \
  --filter '{
    "And": [
      {"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Elastic Compute Cloud - Compute"]}},
      {"Dimensions": {"Key": "PURCHASE_TYPE", "Values": ["On-Demand"]}}
    ]
  }' \
  --metrics UnblendedCost \
  --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.UnblendedCost.Amount}' \
  --output table

# Get AWS's own Savings Plan recommendation based on your usage
aws savingsplans get-savings-plans-purchase-recommendation \
  --savings-plans-type COMPUTE_SP \
  --term-in-years ONE_YEAR \
  --payment-option NO_UPFRONT \
  --lookback-period-in-days THIRTY_DAYS
</code></pre>
<p>As a rule, commit to 60–70% of your steady-state On-Demand spend after optimisation. Leave 30–40% flexible. Never commit on the unoptimised baseline.</p>
<p><strong>Monthly savings:</strong> \(5,000–\)15,000 depending on compute spend. This is the pattern with the highest single-action ROI when sequenced correctly.</p>
<h2 id="heading-pattern-5-cross-az-data-transfer">Pattern 5: Cross-AZ Data Transfer</h2>
<p>AWS charges \(0.01 per GB in each direction when data crosses an Availability Zone boundary. \)0.01 sounds negligible. It's not — because AZ boundaries are crossed constantly in distributed systems, and the charge is bidirectional.</p>
<p>The most common scenario: your application pods are scheduled across multiple AZs (as they should be for resilience), but your database is pinned to one AZ. Every database query from a pod in a different AZ costs \(0.01/GB going to the database and \)0.01/GB coming back. At 100GB of database traffic per day, that's \(60/month. At 1TB per day, it is \)600/month.</p>
<p><strong>What the waste looks like:</strong></p>
<pre><code class="language-bash"># Check current cross-AZ data transfer charges
aws ce get-cost-and-usage \
  --time-period Start=\((date -d 'last month' +%Y-%m-01),End=\)(date +%Y-%m-01) \
  --granularity MONTHLY \
  --filter '{"Dimensions": {"Key": "USAGE_TYPE", "Values": ["DataTransfer-Regional-Bytes"]}}'  \
  --metrics UnblendedCost \
  --query 'ResultsByTime[0].Total.UnblendedCost.Amount' \
  --output text
</code></pre>
<p>How to find which pods are causing the cross-AZ traffic:</p>
<pre><code class="language-bash"># Check which AZ your database RDS instance is in
aws rds describe-db-instances \
  --query 'DBInstances[*].{ID:DBInstanceIdentifier,AZ:AvailabilityZone}' \
  --output table

# Check which AZs your application pods are running in
kubectl get pods -o wide -n production | awk '{print $7}' | sort | uniq -c
</code></pre>
<p>If your RDS is in <code>us-east-1a</code> and 60% of your pods are in <code>us-east-1b</code> and <code>us-east-1c</code>, you have a cross-AZ traffic problem.</p>
<h3 id="heading-the-fix-topology-aware-routing">The Fix — Topology-aware Routing:</h3>
<pre><code class="language-yaml"># topology-aware-routing.yaml
# This tells Kubernetes to prefer scheduling pods in the same AZ
# as the node making the request — keeping traffic local

apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: production
  annotations:
    # Route traffic to pods in the same AZ as the caller when possible
    service.kubernetes.io/topology-mode: "Auto"
spec:
  selector:
    app: payment-api
  ports:
  - port: 8080
    targetPort: 8080
</code></pre>
<pre><code class="language-yaml"># For pods themselves — spread across AZs but prefer local
# topologySpreadConstraints ensures even distribution
# while topology-aware routing keeps traffic within AZs

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: payment-api
</code></pre>
<p>For database traffic specifically, consider migrating from single-AZ RDS to Aurora, which handles AZ routing internally. Your application connects to one endpoint and Aurora routes internally — no cross-AZ charge from the application layer.</p>
<p><strong>Monthly savings:</strong> \(500–\)6,000 depending on database query volume and AZ distribution of your pods.</p>
<h2 id="heading-pattern-6-the-gp2-volume-trap">Pattern 6: The gp2 Volume Trap</h2>
<p>In 2014, AWS launched gp2 EBS volumes. In 2020, they launched gp3 — cheaper, faster, and with better baseline performance. In 2026, most Series A companies are still running gp2.</p>
<p>The difference: gp2 costs \(0.10/GB/month and provides 3 IOPS per GB (100 IOPS minimum). gp3 costs \)0.08/GB/month and provides 3,000 IOPS baseline regardless of size. gp3 is 20% cheaper and 10x faster on IOPS for most volume sizes. The migration is online — it runs while the volume is attached and in use.</p>
<p><strong>Finding all your gp2 volumes:</strong></p>
<pre><code class="language-bash"># List every gp2 volume in your account with its size and monthly cost
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].{
    ID:VolumeId,
    Size:Size,
    State:State,
    MonthlyCost_USD:Size
  }' \
  --output table

# Count the total: number of volumes and combined GB
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'length(Volumes)' --output text

aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'sum(Volumes[*].Size)' --output text
</code></pre>
<h3 id="heading-the-fix-migrate-all-gp2-to-gp3-in-one-script">The Fix — Migrate All gp2 to gp3 in One Script:</h3>
<pre><code class="language-bash">#!/bin/bash
# migrate_gp2_to_gp3.sh
# Migrates all gp2 volumes to gp3. Online operation — no downtime.
# Each modification runs asynchronously; the volume stays available throughout.

echo "Starting gp2 to gp3 migration..."

# Get all gp2 volume IDs
VOLUMES=$(aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[*].VolumeId' \
  --output text)

COUNT=0
for VOL_ID in $VOLUMES; do
  echo "Migrating $VOL_ID to gp3..."
  aws ec2 modify-volume \
    --volume-id $VOL_ID \
    --volume-type gp3 \
    --no-cli-pager
  COUNT=$((COUNT + 1))
done

echo "Migration initiated for $COUNT volumes."
echo "Modifications run online — no downtime. Monitor progress:"
echo "aws ec2 describe-volumes-modifications --query 'VolumesModifications[*].{ID:VolumeId,State:ModificationState}'"
</code></pre>
<p><strong>Verify completion:</strong></p>
<pre><code class="language-bash"># Check that no gp2 volumes remain
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'length(Volumes)' \
  --output text
# Expected: 0
</code></pre>
<p><strong>Monthly savings:</strong> 20% of your total EBS spend. At \(10,000/month in EBS, that's \)2,000 saved for 30 minutes of work.</p>
<h2 id="heading-pattern-7-the-infinite-log-trap">Pattern 7: The Infinite Log Trap</h2>
<p>CloudWatch log groups have a default retention policy of "Never expire." Every log group created without an explicit retention setting accumulates logs indefinitely. For a busy Series A company, this means you're storing debug logs from 2022 that nobody has opened since the sprint review they were created for.</p>
<p>The cost compounds quietly. CloudWatch charges \(0.03/GB/month for log storage and \)0.50/GB for log ingestion. A cluster generating 50GB of logs per day ingests \(25/day — \)750/month — and then stores those logs forever at an increasing monthly cost.</p>
<p><strong>Finding log groups with no retention policy:</strong></p>
<pre><code class="language-bash"># List all log groups with their retention settings
# Any group showing "retentionInDays: null" is infinite — it never expires
aws logs describe-log-groups \
  --query 'logGroups[*].{Name:logGroupName,RetentionDays:retentionInDays,StoredBytes:storedBytes}' \
  --output table | grep -E "(None|null)"

# Count how many log groups have no retention set
aws logs describe-log-groups \
  --query 'length(logGroups[?retentionInDays==`null`])' \
  --output text
</code></pre>
<h3 id="heading-the-fix-set-retention-policies-in-bulk">The Fix — Set Retention Policies in Bulk:</h3>
<p>Different log types have different compliance requirements. Debug logs don't need to be kept. Audit logs might need 365 days. The table below gives sensible defaults:</p>
<table>
<thead>
<tr>
<th>Log Type</th>
<th>Recommended Retention</th>
<th>Reason</th>
</tr>
</thead>
<tbody><tr>
<td>Application debug logs</td>
<td>14 days</td>
<td>Only useful for active debugging</td>
</tr>
<tr>
<td>Application error logs</td>
<td>90 days</td>
<td>Post-incident investigation window</td>
</tr>
<tr>
<td>Access logs</td>
<td>30 days</td>
<td>Security review window</td>
</tr>
<tr>
<td>CloudTrail audit logs</td>
<td>365 days</td>
<td>SOC2 evidence requirement</td>
</tr>
<tr>
<td>VPC Flow Logs</td>
<td>90 days</td>
<td>Security investigation window</td>
</tr>
</tbody></table>
<pre><code class="language-bash">#!/bin/bash
# set_log_retention.sh
# Sets 30-day retention on all log groups that have no policy set
# Adjust the retention period per log group type as needed

echo "Setting retention policies on log groups with no expiry..."

# Get all log groups with no retention
aws logs describe-log-groups \
  --query 'logGroups[?retentionInDays==`null`].logGroupName' \
  --output text | tr '\t' '\n' | while read LOG_GROUP; do

  # Skip CloudTrail logs — these need longer retention for SOC2
  if echo "$LOG_GROUP" | grep -qi "cloudtrail"; then
    echo "Skipping CloudTrail log group: $LOG_GROUP"
    aws logs put-retention-policy \
      --log-group-name "$LOG_GROUP" \
      --retention-in-days 365
    continue
  fi

  # Set 30-day retention on all other log groups
  echo "Setting 30-day retention on: $LOG_GROUP"
  aws logs put-retention-policy \
    --log-group-name "$LOG_GROUP" \
    --retention-in-days 30
done

echo "Done. Logs older than their retention period will be deleted automatically by CloudWatch."
</code></pre>
<p><strong>Monthly savings:</strong> \(500–\)2,000 on storage costs. The ingestion cost reduction kicks in immediately when noisy debug logging is reduced. The storage cost reduction compounds over 30–90 days as old logs expire.</p>
<h2 id="heading-pattern-8-the-orphaned-resource-collector">Pattern 8: The Orphaned Resource Collector</h2>
<p>Every departed engineer leaves a trail. An EBS volume attached to a terminated instance. An Elastic IP allocated but not associated. A load balancer fronting a service that was deprecated in Q3. Old snapshots from an RDS instance that was replaced. None of these are intentional, but all of them are billed.</p>
<p>The fix is a weekly audit. Not a manual investigation — an automated script that runs every Sunday night, finds orphaned resources, and sends a Slack message with a list of candidates for deletion.</p>
<p><strong>Finding the orphans:</strong></p>
<pre><code class="language-bash"># Unattached EBS volumes — you are paying for storage with nothing in it
aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[*].{
    ID:VolumeId,
    Size:Size,
    Created:CreateTime,
    MonthlyCost:Size
  }' \
  --output table

# Unassociated Elastic IPs — $3.60/month each when not attached to a running instance
aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' \
  --output table

# Old snapshots — created more than 90 days ago, no longer needed
aws ec2 describe-snapshots \
  --owner-ids self \
  --query "Snapshots[?StartTime&lt;='$(date -d '90 days ago' --iso-8601=seconds)'].[SnapshotId,StartTime,VolumeSize]" \
  --output table

# Idle load balancers — active but routing zero traffic
aws elbv2 describe-load-balancers \
  --query 'LoadBalancers[*].{ARN:LoadBalancerArn,DNS:DNSName,State:State.Code}' \
  --output table
</code></pre>
<p><strong>The weekly cleanup Lambda:</strong></p>
<pre><code class="language-python"># orphan_resource_reporter.py
# Runs every Sunday at 20:00 via EventBridge
# Reports orphaned resources to Slack — does NOT auto-delete
# Deletion requires a human decision. The Lambda surfaces the candidates.

import boto3
import json
import urllib.request
from datetime import datetime, timedelta, timezone

SLACK_WEBHOOK_URL = 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

def get_orphaned_resources():
    """Collect all orphaned AWS resources and their estimated monthly costs."""
    ec2 = boto3.client('ec2')
    elbv2 = boto3.client('elbv2')
    report = {'total_monthly_waste': 0, 'resources': []}

    # Unattached EBS volumes ($0.08/GB/month for gp3)
    volumes = ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )['Volumes']
    for vol in volumes:
        monthly_cost = round(vol['Size'] * 0.08, 2)
        report['resources'].append({
            'type': 'Unattached EBS Volume',
            'id': vol['VolumeId'],
            'detail': f"{vol['Size']}GB {vol['VolumeType']}",
            'monthly_cost': monthly_cost
        })
        report['total_monthly_waste'] += monthly_cost

    # Unassociated Elastic IPs ($3.60/month each)
    addresses = ec2.describe_addresses()['Addresses']
    for addr in addresses:
        if 'AssociationId' not in addr:
            report['resources'].append({
                'type': 'Unassociated Elastic IP',
                'id': addr['AllocationId'],
                'detail': addr['PublicIp'],
                'monthly_cost': 3.60
            })
            report['total_monthly_waste'] += 3.60

    # Snapshots older than 90 days
    cutoff = (datetime.now(timezone.utc) - timedelta(days=90)).isoformat()
    snapshots = ec2.describe_snapshots(OwnerIds=['self'])['Snapshots']
    old_snapshots = [s for s in snapshots if s['StartTime'].isoformat() &lt; cutoff]
    for snap in old_snapshots:
        monthly_cost = round(snap.get('VolumeSize', 0) * 0.05, 2)
        report['resources'].append({
            'type': 'Old Snapshot (90+ days)',
            'id': snap['SnapshotId'],
            'detail': f"Created {snap['StartTime'].strftime('%Y-%m-%d')}",
            'monthly_cost': monthly_cost
        })
        report['total_monthly_waste'] += monthly_cost

    return report

def post_to_slack(report):
    """Send the orphaned resource report to Slack."""
    resource_lines = '\n'.join([
        f"• {r['type']} `{r['id']}` — {r['detail']} — *${r['monthly_cost']}/month*"
        for r in report['resources']
    ])

    message = {
        'text': (
            f":money_with_wings: *Weekly Orphaned Resource Report*\n\n"
            f"Found *{len(report['resources'])} orphaned resources* "
            f"costing *${report['total_monthly_waste']:.2f}/month*\n\n"
            f"{resource_lines}\n\n"
            f"Review and delete resources that are no longer needed."
        )
    }
    
    req = urllib.request.Request(
        SLACK_WEBHOOK_URL,
        data=json.dumps(message).encode(),
        headers={'Content-Type': 'application/json'}
    )
    urllib.request.urlopen(req)

def lambda_handler(event, context):
    report = get_orphaned_resources()
    post_to_slack(report)
    return {
        'resources_found': len(report['resources']),
        'monthly_waste': report['total_monthly_waste']
    }
</code></pre>
<p><strong>Monthly savings:</strong> \(500–\)2,000. Every departed engineer typically leaves \(50–\)200 in orphaned resources. At a team of 30 with 30% annual turnover, that compounds quickly.</p>
<h2 id="heading-the-full-savings-summary">The Full Savings Summary</h2>
<table>
<thead>
<tr>
<th>Pattern</th>
<th>Monthly Saving</th>
<th>Time to Fix</th>
<th>Difficulty</th>
</tr>
</thead>
<tbody><tr>
<td>1. New hire experiment tax</td>
<td>\(1,000–\)2,000</td>
<td>2 hours (Lambda)</td>
<td>Medium</td>
</tr>
<tr>
<td>2. Staging proliferation</td>
<td>\(600–\)800</td>
<td>3 hours (scheduling)</td>
<td>Low</td>
</tr>
<tr>
<td>3. NAT Gateway tax</td>
<td>\(2,000–\)8,000</td>
<td>30 minutes</td>
<td>Low</td>
</tr>
<tr>
<td>4. Savings Plan timing</td>
<td>\(5,000–\)15,000</td>
<td>One decision</td>
<td>Low</td>
</tr>
<tr>
<td>5. Cross-AZ data transfer</td>
<td>\(500–\)6,000</td>
<td>2 hours</td>
<td>Medium</td>
</tr>
<tr>
<td>6. gp2 volume trap</td>
<td>\(1,000–\)5,000</td>
<td>30 minutes (script)</td>
<td>Low</td>
</tr>
<tr>
<td>7. Infinite log trap</td>
<td>\(500–\)2,000</td>
<td>1 hour (script)</td>
<td>Low</td>
</tr>
<tr>
<td>8. Orphaned resources</td>
<td>\(500–\)2,000</td>
<td>2 hours (Lambda)</td>
<td>Low</td>
</tr>
<tr>
<td><strong>Total potential</strong></td>
<td><strong>\(11,100–\)40,800/month</strong></td>
<td></td>
<td></td>
</tr>
</tbody></table>
<h2 id="heading-what-to-do-this-week">What to Do This Week</h2>
<p>Don't fix all eight this week. Prioritise by ROI per hour of engineering time:</p>
<p><strong>Day 1 (30 minutes):</strong> Pattern 3 — NAT Gateway endpoints. Highest ROI per minute of any fix in this guide. One command creates the S3 endpoint. Done.</p>
<p><strong>Day 2 (30 minutes):</strong> Pattern 6 — gp2 to gp3 migration. Run the script. Check the output. Done.</p>
<p><strong>Day 3 (1 hour):</strong> Pattern 7 — log retention policies. Run the bulk retention script. Done.</p>
<p><strong>Day 4 (2 hours):</strong> Patterns 1 and 8 — deploy both Lambdas. They run automatically from here.</p>
<p><strong>Next sprint:</strong> Pattern 2 (staging schedule), Pattern 5 (topology-aware routing), and Pattern 4 (run the rightsizing cycle first, then evaluate Savings Plans).</p>
<p>Open Cost Explorer after each fix. Compare against your baseline screenshot from the start of this guide. The line should start going down.</p>
<h2 id="heading-resources">Resources</h2>
<ul>
<li><p><a href="https://www.finops.org/framework/"><strong>FinOps Foundation Framework</strong></a> — The practitioner framework this guide contributes to, covering Inform, Optimize, and Operate phases of cloud cost management</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/cost-management/latest/APIReference/API_GetCostAndUsage.html"><strong>AWS Cost Explorer API Reference</strong></a> — Full reference for the <code>get-cost-and-usage</code> command used throughout this guide</p>
</li>
<li><p><a href="https://aws.amazon.com/compute-optimizer/"><strong>AWS Compute Optimizer</strong></a> — AWS's own rightsizing recommendation service, used alongside the patterns in this guide for EC2 and EBS recommendations</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html"><strong>AWS VPC Endpoints Documentation</strong></a> — Complete list of available VPC endpoints for Pattern 3</p>
</li>
<li><p><a href="https://aws.amazon.com/solutions/implementations/instance-scheduler-on-aws/"><strong>AWS Instance Scheduler Solution</strong></a> — The AWS-maintained CloudFormation solution for Pattern 2 environment scheduling</p>
</li>
<li><p><a href="https://karpenter.sh/docs/"><strong>Karpenter Documentation</strong></a> — For teams ready to go beyond these 8 patterns into dynamic node provisioning and Spot diversification</p>
</li>
<li><p><a href="https://www.finops.org/resources/"><strong>FinOps Foundation Asset Library</strong></a> — The community asset library where practical scripts like the ones in this guide are contributed and maintained by practitioners</p>
</li>
</ul>
<p><a href="https://github.com/aayostem"><em>Ayobami Adejumo</em></a> <em>is a senior platform engineer and FinOps specialist. He has audited AWS infrastructure for 30+ Series A companies and contributes practical tooling to the FinOps Foundation Asset Library.</em></p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ Common DevOps Mistakes and How to Avoid Them — Tips for Startups ]]>
                </title>
                <description>
                    <![CDATA[ Most DevOps engineers don't fail because they lack knowledge about tools. They fail because nobody told them what not to do before they got into production. Startup environments make this worse. The p ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-avoid-devops-mistakes/</link>
                <guid isPermaLink="false">6a060c22baf09db7a6253878</guid>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Cloud Computing ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ startup ]]>
                    </category>
                
                    <category>
                        <![CDATA[ tips ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Tolani Akintayo ]]>
                </dc:creator>
                <pubDate>Thu, 14 May 2026 17:53:38 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5fc16e412cae9c5b190b6cdd/6fcabd5e-272f-4f1d-b035-8241896e8296.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Most DevOps engineers don't fail because they lack knowledge about tools. They fail because nobody told them what <em>not</em> to do before they got into production.</p>
<p>Startup environments make this worse. The pressure to ship fast, the small team sizes, and the absence of senior engineers to review your decisions means mistakes happen quietly until they become outages, data loss events, or security incidents that cost the company thousands of dollars and weeks of recovery time.</p>
<p>This article is a direct breakdown of the ten most costly DevOps mistakes engineers make early in their careers at startups. For each mistake, you will get the real-world scenario, the business impact, and the concrete fix you can apply immediately.</p>
<p>Whether you are setting up your first production environment or auditing an existing one, this guide will help you build systems that are reliable, secure, and aligned with what the business actually needs.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-who-this-article-is-for">Who This Article Is For</a></p>
</li>
<li><p><a href="#heading-why-startups-are-a-different-environment">Why Startups Are a Different Environment</a></p>
</li>
<li><p><a href="#heading-mistake-1-deploying-without-understanding-what-youre-deploying">Mistake 1: Deploying Without Understanding What You're Deploying</a></p>
</li>
<li><p><a href="#heading-mistake-2-using-production-as-a-development-environment">Mistake 2: Using Production as a Development Environment</a></p>
</li>
<li><p><a href="#heading-mistake-3-hardcoding-secrets-and-credentials">Mistake 3: Hardcoding Secrets and Credentials</a></p>
</li>
<li><p><a href="#heading-mistake-4-overengineering-for-problems-you-dont-have-yet">Mistake 4: Overengineering for Problems You Don't Have Yet</a></p>
</li>
<li><p><a href="#heading-mistake-5-no-observability-before-launch">Mistake 5: No Observability Before Launch</a></p>
</li>
<li><p><a href="#heading-mistake-6-treating-security-as-a-final-step">Mistake 6: Treating Security as a Final Step</a></p>
</li>
<li><p><a href="#heading-mistake-7-manual-deployments-in-production">Mistake 7: Manual Deployments in Production</a></p>
</li>
<li><p><a href="#heading-mistake-8-no-disaster-recovery-plan">Mistake 8: No Disaster Recovery Plan</a></p>
</li>
<li><p><a href="#heading-mistake-9-no-documentation-or-runbooks">Mistake 9: No Documentation or Runbooks</a></p>
</li>
<li><p><a href="#heading-mistake-10-solving-technical-problems-without-understanding-the-business">Mistake 10: Solving Technical Problems Without Understanding the Business</a></p>
</li>
<li><p><a href="#heading-the-system-thinking-framework-every-devops-engineer-needs">The System Thinking Framework Every DevOps Engineer Needs</a></p>
</li>
<li><p><a href="#heading-your-production-readiness-checklist">Your Production Readiness Checklist</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
</ul>
<h2 id="heading-who-this-article-is-for">Who This Article Is For</h2>
<ul>
<li><p><strong>Early-career DevOps and cloud engineers</strong> who are building or maintaining production infrastructure at a startup.</p>
</li>
<li><p><strong>Backend developers</strong> who have recently taken on DevOps responsibilities.</p>
</li>
<li><p><strong>Engineers joining a startup</strong> who want to understand what operational discipline actually looks like in a fast-moving environment.</p>
</li>
</ul>
<p>You do not need to be an expert in any specific tool to follow this article. The focus is on decision-making patterns and operational discipline, not tool configuration.</p>
<h2 id="heading-why-startups-are-a-different-environment">Why Startups Are a Different Environment</h2>
<p>Before getting into the mistakes, you have to understand why startups produce them in the first place.</p>
<img src="https://cdn.hashnode.com/uploads/covers/65a5bfab4c73b29396c0b895/f9bec1fa-8938-4144-b934-9e5af4edf4ad.svg" alt="diagram showing the startup DevOps reality, a single engineer handling infra, CI/CD, security, monitoring, and deployment pipelines simultaneously" style="display: block;" width="680" height="506" loading="lazy">

<p>In a large company, you typically have dedicated security engineers, an SRE team, a platform team, and multiple reviewers for every infrastructure change. In a startup, you mostly likely have one engineer responsible for all of that simultaneously.</p>
<p>This creates four specific pressure points:</p>
<ol>
<li><p><strong>Speed pressure.</strong> The business needs features shipped now. Operational discipline gets treated as optional because nobody is watching closely yet.</p>
</li>
<li><p><strong>Budget constraints.</strong> Every infrastructure decision has a direct impact on company runway. Engineers optimize for the cheapest option rather than the most reliable one.</p>
</li>
<li><p><strong>Absent guardrails.</strong> There is no senior engineer reviewing your Terraform plans. There is no security audit before launch. The absence of immediate consequences can make bad decisions feel like good ones.</p>
</li>
<li><p><strong>Constantly changing requirements.</strong> The architecture you design today may need to support a completely different product in six months. None of these pressures are excuses for poor decisions. But understanding them helps you see why the following mistakes happen so consistently.</p>
</li>
</ol>
<h2 id="heading-mistake-1-deploying-without-understanding-what-youre-deploying">Mistake 1: Deploying Without Understanding What You're Deploying</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A junior engineer is asked to deploy the company's Node.js API to AWS. They find a tutorial for Elastic Beanstalk, follow it, and it works. Two weeks later, traffic increases. They try to scale "the same way as in the tutorial." The application goes down. They cannot debug it because they never understood what the deployment was actually doing.</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>When production breaks and the person who deployed the system cannot explain how it works, diagnosis takes hours instead of minutes. The longer the incident runs, the higher the cost in customer trust, team morale, and potentially direct revenue loss.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p>Before you deploy anything to production, you should be able to answer these five questions in writing:</p>
<ol>
<li><p><strong>What compute type is running my code?</strong> (EC2, Lambda, Fargate, container?)</p>
</li>
<li><p><strong>How does a new version replace the old one?</strong> (Rolling? Blue/green? All-at-once?)</p>
</li>
<li><p><strong>Where does configuration and secrets come from?</strong> (SSM? Secrets Manager? Environment file?)</p>
</li>
<li><p><strong>What downstream services depend on this?</strong> (Database connections? Other APIs? Cache?)</p>
</li>
<li><p><strong>How do I roll back in under five minutes if this breaks?</strong></p>
</li>
</ol>
<p>If you cannot answer all five, do not deploy until you can. The tutorial that got it running is not the documentation for how it operates.</p>
<blockquote>
<p>"It is better to spend two hours understanding a system before deploying it than two days debugging it after something breaks."</p>
</blockquote>
<p>Personally, when learning a new technology, tool, or implementing something I have not worked with before, I usually focus on three core questions: What, Why, and How.</p>
<ul>
<li><p><strong>The first question is: What is this technology or concept about?</strong><br>This helps me build a solid foundation by doing deep research, studying the official documentation, understanding the core principles, and sometimes even learning the history behind the tool or technology. I believe having a well-grounded understanding before implementation is very important.</p>
</li>
<li><p><strong>The second question is: Why do we need it?</strong><br>I try to understand the value the technology brings, why it should be implemented, what problem it solves, and how it benefits the team or organization. This helps me make informed technical decisions instead of just implementing tools without understanding their purpose.</p>
</li>
<li><p><strong>The third question is: How should it be implemented?</strong><br>There are usually multiple approaches to solving a problem or implementing a technology, so I focus on understanding the best and most practical approach based on the use case and expected outcome.</p>
</li>
</ul>
<p>This structured approach has helped me learn new technologies quickly, adapt fast, and implement solutions effectively in real-world environments.</p>
<h2 id="heading-mistake-2-using-production-as-a-development-environment">Mistake 2: Using Production as a Development Environment</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>To save time, an engineer tests a new deployment script directly in the production AWS account. They accidentally run a command that terminates the production database instance. Automated backups exist but were misconfigured. Six hours of customer data is unrecoverable.</p>
<p>This scenario happens more often than you would expect. The reasoning is always the same: "It will only take a minute."</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>A single test-in-production incident can result in data loss, hours of downtime, and a customer communication crisis. In a startup, that can permanently damage the company's reputation before it has had the chance to build one.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p>You need at minimum three separate environments and ideally three separate AWS accounts:</p>
<table>
<thead>
<tr>
<th>Environment</th>
<th>Purpose</th>
<th>Access Level</th>
</tr>
</thead>
<tbody><tr>
<td><strong>dev</strong></td>
<td>Break things freely. No real data.</td>
<td>Engineers have broad access</td>
</tr>
<tr>
<td><strong>staging</strong></td>
<td>Mirror of production. Final verification.</td>
<td>Controlled access</td>
</tr>
<tr>
<td><strong>production</strong></td>
<td>Real customers. Real data.</td>
<td>MFA required. No manual deployments.</td>
</tr>
</tbody></table>
<p>Using separate AWS accounts (not just separate VPCs) gives you account-level isolation. A permission error in the dev account cannot accidentally touch production infrastructure at the API level.</p>
<p>Infrastructure as Code (Terraform or CloudFormation) makes this affordable, you write the configuration once and apply it three times with different variable files.</p>
<pre><code class="language-hcl"># terraform/environments/prod/main.tf
module "app" {
  source      = "../../modules/app"
  environment = "production"
  instance_type = "t3.medium"
  db_instance_class = "db.t3.medium"
  multi_az          = true
}
</code></pre>
<pre><code class="language-hcl"># terraform/environments/staging/main.tf
module "app" {
  source      = "../../modules/app"
  environment = "staging"
  instance_type = "t3.small"
  db_instance_class = "db.t3.small"
  multi_az          = false
}
</code></pre>
<p>The module is the same. The environment-specific variables are different. Separate environments are not a luxury, they are the minimum operating standard for any team running real software.</p>
<h2 id="heading-mistake-3-hardcoding-secrets-and-credentials">Mistake 3: Hardcoding Secrets and Credentials</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A new engineer joins a startup and clones the repository. Inside they find a <code>.env</code> file committed to Git containing the production database password, the Stripe secret key, and an AWS access key with admin permissions. The repository has been public for six months.</p>
<p>GitHub's automated secret scanning never triggered because the secrets were inside a <code>.env</code> file rather than raw in the code. The credentials had been valid and actively used for over six months.</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>Automated scanners run by attackers find exposed credentials within minutes of them being pushed to a public repository. A single exposed AWS access key with admin permissions can result in:</p>
<ul>
<li><p>Crypto-mining workloads generating thousands of dollars in cloud bills overnight</p>
</li>
<li><p>Complete exfiltration of customer data from every S3 bucket</p>
</li>
<li><p>Privilege escalation: the attacker creates new admin users and locks you out of your own account</p>
</li>
<li><p>AWS account suspension while the investigation runs</p>
</li>
</ul>
<p>According to <a href="https://github.blog/security/vulnerability-research/securing-millions-of-developers-together/">GitHub's annual security report</a>, millions of secrets are exposed in public repositories every year. The average time to detect a compromised cloud credential is 197 days.</p>
<h2 id="heading-the-fix">The Fix</h2>
<p><strong>Step 1: Never commit secrets to Git.</strong> Not temporarily. Not in a branch. Not in a private repository.</p>
<p><strong>Step 2: Add</strong> <code>.gitignore</code> <strong>before you create the first file.</strong> Check in the <code>.gitignore</code> with the first line of code before any <code>.env</code> files exist.</p>
<pre><code class="language-gitignore"># .gitignore
.env
.env.*
*.pem
*.key
secrets/
</code></pre>
<p><strong>Step 3: Use AWS Secrets Manager or SSM Parameter Store for all production secrets.</strong> Your application reads secrets at runtime:</p>
<pre><code class="language-python"># Python example — fetch secret at runtime, never at build time
import boto3
import json
 
def get_secret(secret_name: str, region: str = "us-east-1") -&gt; dict:
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])
 
# Usage
db_config = get_secret("prod/myapp/database")
DATABASE_URL = db_config["connection_string"]
</code></pre>
<p><strong>Step 4: Scan your existing repositories immediately.</strong> You may already have a problem:</p>
<pre><code class="language-bash"># Install trufflehog to scan for exposed secrets in your repo history
pip install trufflehog
 
# Scan the entire commit history of your repository
trufflehog git file://.
 
# Or scan a remote GitHub repo
trufflehog github --repo https://github.com/your-org/your-repo
</code></pre>
<p><strong>Step 5: Add a pre-commit hook to prevent future accidents:</strong></p>
<pre><code class="language-bash">pip install pre-commit
</code></pre>
<pre><code class="language-yaml"># .pre-commit-config.yaml
repos:
  - repo: https://github.com/awslabs/git-secrets
    rev: master
    hooks:
      - id: git-secrets
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
</code></pre>
<pre><code class="language-bash">pre-commit install
# Now the hook runs before every commit and blocks detected secrets
</code></pre>
<p>There is no recovery from a publicly exposed database password. The fix takes ten minutes upfront. The incident takes weeks.</p>
<h2 id="heading-mistake-4-overengineering-for-problems-you-dont-have-yet">Mistake 4: Overengineering for Problems You Don't Have Yet</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A five-person startup with 200 users decides to build a microservices architecture on Kubernetes because "Netflix uses it." They spend three months setting up Kubernetes, Istio service mesh, ArgoCD, Vault, Prometheus, and Grafana. Their product has not shipped a new feature in three months. A competitor with a monolith on a single EC2 instance shipped twelve new features in the same period.</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>Every layer of infrastructure you add is a layer that can break, a layer that requires expertise to operate, and a layer that slows down every future change. Kubernetes is the right answer for organizations with the scale and team size to operate it. For a five-person startup, it is an expensive distraction.</p>
<p>Premature complexity does not just cost engineering time. It costs the competitive advantage that speed provides in the early stage.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p>Match your infrastructure to your actual stage:</p>
<table>
<thead>
<tr>
<th>Scale</th>
<th>Right Infrastructure</th>
<th>Cost Range</th>
</tr>
</thead>
<tbody><tr>
<td><strong>1–1,000 users</strong></td>
<td>Single EC2 + RDS + Nginx reverse proxy</td>
<td>$20–50/month</td>
</tr>
<tr>
<td><strong>1K–50K users</strong></td>
<td>Auto-scaling group, RDS Multi-AZ, ALB, basic CI/CD</td>
<td>$200-500/month</td>
</tr>
<tr>
<td><strong>50K–500K users</strong></td>
<td>ECS Fargate, RDS read replicas, ElastiCache, full observability</td>
<td>$1K-5K/month</td>
</tr>
<tr>
<td><strong>500K+ users</strong></td>
<td>Multi-region, managed Kubernetes, dedicated SRE</td>
<td>$10K+/month</td>
</tr>
</tbody></table>
<p>The question to ask before every infrastructure decision is: <strong>"What specific, measurable problem does this solve today that my current setup cannot solve?"</strong></p>
<p>Amazon, Netflix, and Uber did not start with microservices. They started with monoliths and extracted services only when the monolith became the actual bottleneck. You are not Netflix. You are solving the problems in front of you today.</p>
<p>Use managed services wherever possible, RDS instead of self-hosted Postgres, Fargate instead of self-managed Kubernetes, ElastiCache instead of self-hosted Redis. Managed services let your team focus on the product instead of the infrastructure.</p>
<h2 id="heading-mistake-5-no-observability-before-launch">Mistake 5: No Observability Before Launch</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A startup's checkout flow breaks on a Friday evening. Users are abandoning their carts and the company is losing revenue. The DevOps engineer finds out 45 minutes later because a customer sent a direct message to the CEO on Twitter.</p>
<p>The engineer has no dashboards, no log aggregation, and no alerting. They SSH into the production server and scroll through raw log files. Two hours later, they find the issue: a database connection pool was exhausted by a memory leak introduced in that morning's deployment.</p>
<h3 id="heading-business-impact">Business Impact</h3>
<p>Without observability:</p>
<ul>
<li><p>You find out about production problems from users, not from your systems</p>
</li>
<li><p>Incidents take 10x longer to resolve because diagnosis is guesswork</p>
</li>
<li><p>You cannot tell whether a deployment improved or degraded performance</p>
</li>
<li><p>You have no data for making better architecture decisions</p>
</li>
</ul>
<h3 id="heading-the-fix">The Fix</h3>
<p>Implement the four golden signals before any service goes to production. These come from <a href="https://sre.google/sre-book/monitoring-distributed-systems/">Google's Site Reliability Engineering book</a>:</p>
<ol>
<li><p><strong>Latency</strong>: How long requests take to complete (p50, p95, p99)</p>
</li>
<li><p><strong>Traffic</strong>: How many requests per second the system is handling</p>
</li>
<li><p><strong>Errors</strong>: The rate of failed requests (5xx responses per minute)</p>
</li>
<li><p><strong>Saturation</strong>: How close the system is to its limits (CPU, memory, connection pool)</p>
</li>
</ol>
<p>Here is a minimal CloudWatch alarm setup using the AWS CLI:</p>
<pre><code class="language-shell"># Alert when error rate exceeds 1% for 5 consecutive minutes

aws cloudwatch put-metric-alarm \
  --alarm-name "high-error-rate-production" \
  --alarm-description "Error rate exceeded 1% for 5 minutes" \
  --metric-name "5XXError" \
  --namespace "AWS/ApplicationELB" \
  --statistic "Average" \
  --period 60 \
  --evaluation-periods 5 \
  --threshold 0.01 \
  --comparison-operator "GreaterThanOrEqualToThreshold" \
  --alarm-actions "arn:aws:sns:us-east-1:123456789:pagerduty-production" \
  --dimensions Name=LoadBalancer,Value=app/my-alb/1234567890abcdef
</code></pre>
<p>Every application should also expose a <code>/health</code> endpoint that returns <code>200 OK</code> when healthy:</p>
<pre><code class="language-python"># FastAPI example

from fastapi import FastAPI
from sqlalchemy import text
 
app = FastAPI()
 
@app.get("/health")
async def health_check():
    # Check database connectivity
    try:
        db.execute(text("SELECT 1"))
        db_status = "healthy"
    except Exception:
        db_status = "unhealthy"
 
    return {
        "status": "healthy" if db_status == "healthy" else "degraded",
        "database": db_status,
        "version": os.getenv("APP_VERSION", "unknown")
    }
</code></pre>
<p>Your load balancer checks this endpoint. Your uptime monitor checks it. You check it after every deployment.</p>
<blockquote>
<p>You do not get to say a system is working unless you have data to prove it. "Nobody complained" is not the same as "nothing is broken."</p>
</blockquote>
<h2 id="heading-mistake-6-treating-security-as-a-final-step">Mistake 6: Treating Security as a Final Step</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A startup rushes to launch their MVP. Security reviews are "planned for after launch." Six months later, a potential enterprise customer requires a security audit before signing a contract. The audit reveals:</p>
<ul>
<li><p>S3 buckets publicly accessible by default</p>
</li>
<li><p>EC2 instances with port 22 open to <code>0.0.0.0/0</code></p>
</li>
<li><p>IAM users with <code>AdministratorAccess</code> for the entire team</p>
</li>
<li><p>No encryption on the database at rest</p>
</li>
<li><p>JWT secrets hardcoded in environment variables The audit fails. The enterprise deal worth $120,000 annually is lost. Remediation takes four weeks of engineering time.</p>
</li>
</ul>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>Security debt is the most expensive technical debt you can accumulate. Unlike performance debt that degrades gradually, security vulnerabilities cause sudden, catastrophic events: data breaches, ransomware, account takeovers, and regulatory fines. At a startup, any one of these can end the company.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p>Apply these six security controls before the first line of production code ships:</p>
<p><strong>1. Principle of Least Privilege every IAM role gets only what it needs:</strong></p>
<p>One of the most common security mistakes in AWS is granting roles more permissions than they need either out of convenience (<code>s3:*</code>) or uncertainty about what the service actually requires. This creates unnecessary risk: if a role is compromised, the attacker inherits every permission you granted.</p>
<p>The fix is simple: look at what your service actually does, then write a policy that allows exactly that.</p>
<p>If your app uploads and reads files from a specific S3 bucket, the policy should say exactly that:</p>
<pre><code class="language-json">{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-app-uploads/*"
    }
  ]
}
</code></pre>
<p>Notice the <code>Resource</code> is scoped to <code>my-app-uploads/*</code> not all S3 buckets. And the <code>Action</code> list covers only <code>GetObject</code> and <code>PutObject</code> not <code>DeleteObject</code>, not <code>s3:*</code>. If the service gets compromised, the attacker can read and write to that one bucket. That is it. The rest of your account is untouched.</p>
<p><strong>2. Block all S3 public access by default:</strong></p>
<p>AWS S3 buckets are private by default when created but that can be overridden at the bucket level, the object level, or through a bucket policy. Misconfigured S3 buckets are one of the most common causes of data breaches, and they are almost always accidental.</p>
<p>The safest approach is to enable the "Block Public Access" setting at the account level, which overrides all other settings and prevents any bucket from being made public even if someone tries:</p>
<pre><code class="language-bash">aws s3api put-public-access-block \
  --bucket my-app-bucket \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
</code></pre>
<p>Run this for every bucket you create. Better yet, enable it at the AWS account level so it applies automatically to all future buckets by default.</p>
<p><strong>3. Never open SSH to the internet, use AWS Systems Manager Session Manager instead:</strong></p>
<p>Port 22 open to <code>0.0.0.0/0</code> is an attack surface that exists on thousands of AWS instances right now. Brute-force bots scan the internet continuously looking for open SSH ports. Even with a strong key, the exposure is unnecessary because AWS provides a better alternative.</p>
<p>AWS Systems Manager Session Manager gives you full shell access to any EC2 instance without opening a single inbound port on the security group. There is no port to scan, no port to attack, and every session is logged automatically to CloudTrail:</p>
<pre><code class="language-bash"># Start a session on an EC2 instance without port 22 open
aws ssm start-session --target i-0123456789abcdef0
</code></pre>
<p>To use Session Manager, the EC2 instance needs the SSM Agent installed (included by default on Amazon Linux 2 and Ubuntu 20.04+) and an IAM instance profile with the <code>AmazonSSMManagedInstanceCore</code> policy attached. Once that is set up, you can close port 22 on the security group entirely.</p>
<p><strong>4. Enable MFA for all IAM users and enforce it via policy:</strong></p>
<p>A leaked IAM username and password with no MFA is a fully compromised account. Multi-factor authentication is the single most effective control against credential theft, and it costs nothing to enable.</p>
<p>Enforce it through an IAM policy that denies all actions when MFA is not present, except the actions needed to set up MFA in the first place. This means even if a set of credentials is stolen, the attacker cannot do anything without the second factor.</p>
<p>The AWS documentation provides the <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_users-self-manage-mfa-and-creds.html">Complete Deny Without MFA Policy</a>, attach it to every IAM user or group in your account. This is a one-time setup that permanently raises your account's security baseline.</p>
<p><strong>5. Enable CloudTrail in all regions:</strong></p>
<p>Without CloudTrail, you have no record of who did what in your AWS account. If a credential is compromised, you cannot investigate what the attacker accessed. If an engineer accidentally deletes a resource, you cannot trace it. You are operating blind.</p>
<p>CloudTrail logs every AWS API call who made it, from which IP, at what time, and what the response was. Enable it across all regions so activity in regions you do not actively use is also captured:</p>
<pre><code class="language-bash">aws cloudtrail create-trail \
  --name production-audit-trail \
  --s3-bucket-name my-cloudtrail-logs \
  --is-multi-region-trail \
  --enable-log-file-validation
</code></pre>
<p>The <code>--enable-log-file-validation</code> flag generates a digest file for each log that lets you verify the log has not been tampered with, this is important if you ever need to use these logs in a security investigation or compliance audit. Once this is running, every <code>AssumeRole</code>, every <code>DeleteBucket</code>, and every <code>RunInstances</code> call in your account is permanently recorded.</p>
<p><strong>6. Run AWS Security Hub from day one:</strong></p>
<p>Most teams only discover security misconfigurations after a breach or a compliance audit. Security Hub inverts this, it continuously scans your AWS environment against industry-standard frameworks (CIS AWS Foundations Benchmark, AWS Foundational Security Best Practices) and surfaces findings before they become incidents.</p>
<p>Enabling it takes a single command:</p>
<pre><code class="language-bash">aws securityhub enable-security-hub
</code></pre>
<p>Within minutes, Security Hub gives your account a compliance score and a prioritized list of findings. A finding might tell you that a security group has port 22 open to the world, that an S3 bucket has logging disabled, or that root account credentials were recently used. Each finding includes the affected resource and a remediation guide.</p>
<p>Treat every Security Hub finding the same way you treat a production bug: assign it a priority, assign an owner, and close it. A finding sitting unaddressed for 30 days is a known vulnerability you chose to leave open.</p>
<h2 id="heading-mistake-7-manual-deployments-in-production">Mistake 7: Manual Deployments in Production</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A startup's deployment process is documented in a Notion page that is four months out of date. It involves SSH-ing into the server, running <code>git pull</code>, running <code>npm install</code>, and restarting the PM2 process. Different engineers do it slightly differently. One engineer, rushing a late-night release, skips <code>npm install</code>. The application starts crashing because a new dependency is missing.</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>Manual deployment processes are inherently unreliable. Humans under pressure skip steps, perform steps in the wrong order, and remember procedures differently. Every manual step in a production deployment process is a scheduled incident waiting for the right moment of stress.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p>If a deployment step is performed manually more than twice, it needs to be automated. Here is a minimal but complete GitHub Actions deployment workflow for an ECS Fargate service:</p>
<pre><code class="language-yaml"># .github/workflows/deploy.yml
name: Deploy to Production
 
on:
  push:
    branches:
      - main
 
permissions:
  id-token: write   # Required for OIDC authentication with AWS
  contents: read
 
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
 
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
          aws-region: us-east-1
 
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2
 
      - name: Build and push Docker image
        id: build
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t \(ECR_REGISTRY/my-app:\)IMAGE_TAG .
          docker push \(ECR_REGISTRY/my-app:\)IMAGE_TAG
          echo "image=\(ECR_REGISTRY/my-app:\)IMAGE_TAG" &gt;&gt; $GITHUB_OUTPUT
 
      - name: Deploy to Amazon ECS
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1
        with:
          task-definition: task-definition.json
          service: my-app-service
          cluster: production
          wait-for-service-stability: true
</code></pre>
<p>Notice <code>wait-for-service-stability: true</code>. Without this, the workflow reports success the moment ECS accepts the new task definition before the containers are actually healthy. With it, the workflow fails if the new containers crash. You want to know immediately, not discover it from user reports thirty minutes later.</p>
<h2 id="heading-mistake-8-no-disaster-recovery-plan">Mistake 8: No Disaster Recovery Plan</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A startup's production database runs on a single RDS instance with no Multi-AZ configuration. Automated backups are enabled but have never been tested. The EBS volume backing the instance fails. AWS provisions a new instance from the last snapshot, which is 18 hours old. 18 hours of customer data is permanently lost.</p>
<p>The startup had no disaster recovery plan, no tested recovery procedure, and no communication template ready for customers.</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>The question is not whether your infrastructure will fail. It will fail. Every database, every server, every availability zone experiences failures. The question is whether you have a tested plan for when it does.</p>
<p>Data loss of any magnitude is serious. For startups that handle financial data, healthcare data, or anything under GDPR, even partial data loss can trigger regulatory consequences.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p><strong>Define your RTO and RPO before you design anything:</strong></p>
<ul>
<li><p><strong>RTO (Recovery Time Objective):</strong> How long can the business survive without this system? A payment API might have an RTO of 15 minutes. An internal analytics dashboard might have an RTO of 4 hours.</p>
</li>
<li><p><strong>RPO (Recovery Point Objective):</strong> How much data loss is acceptable? Zero means real-time replication. One hour means hourly snapshots are sufficient. This directly determines your backup frequency and architecture.</p>
</li>
</ul>
<p><strong>Enable RDS Multi-AZ for all production databases:</strong></p>
<pre><code class="language-hcl"># Terraform
resource "aws_db_instance" "production" {
  identifier        = "prod-postgres"
  engine            = "postgres"
  engine_version    = "15.4"
  instance_class    = "db.t3.medium"
  allocated_storage = 100
 
  # Multi-AZ: automatic failover to standby in a different AZ
  # No data loss. Automatic failover in ~60-120 seconds.
  multi_az = true
 
  # Encryption at rest — non-negotiable
  storage_encrypted = true
 
  # Automated backups with 7-day retention
  backup_retention_period = 7
  backup_window           = "03:00-04:00"
 
  # Enable deletion protection in production
  deletion_protection = true
 
  tags = {
    Environment = "production"
  }
}
</code></pre>
<p><strong>Test your backups on a schedule.</strong> Create a monthly calendar event: "Restore production backup to staging and verify data integrity." An untested backup is not a backup, it is a hope.</p>
<pre><code class="language-bash"># Restore a snapshot to a test instance and verify
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier recovery-test \
  --db-snapshot-identifier rds:prod-postgres-2025-01-15 \
  --db-instance-class db.t3.medium \
  --no-multi-az
 
# Connect and verify row counts
psql -h recovery-test.xxxx.rds.amazonaws.com -U admin -d mydb \
  -c "SELECT COUNT(*) FROM users; SELECT COUNT(*) FROM orders;"
</code></pre>
<p>For official guidance on RDS backup and restore, refer to the <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html">AWS RDS Backup and Restore documentation</a>.</p>
<h2 id="heading-mistake-9-no-documentation-or-runbooks">Mistake 9: No Documentation or Runbooks</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>The startup's most experienced DevOps engineer takes two weeks of vacation. On day three of their holiday, the staging environment goes down. Nobody else knows how it was built, the engineer set it up manually over six months with no documentation, no Terraform, no notes. The team spends four days trying to reconstruct the environment from memory and guesswork. The engineer gets messages on their vacation every day. When they return, they rebuild the environment in four hours.</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>Undocumented infrastructure creates single points of failure not in your systems, but in your team. It makes onboarding new engineers take weeks instead of hours. It makes incident response depend on specific people being available. When that person leaves the company, the knowledge walks out with them.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p>Documentation for an engineering team means three specific things:</p>
<ol>
<li><p><strong>Infrastructure as Code is the highest form of documentation.</strong> The Terraform that defines your infrastructure IS the documentation for what exists and how it is configured. If something is not in code, it should not exist in production.</p>
</li>
<li><p><strong>A runbook for every operational task.</strong> A runbook is a step-by-step procedure written well enough that someone in their first week at the company can follow it during an incident:</p>
</li>
</ol>
<pre><code class="language-markdown"># Runbook: Production Database Connection Exhaustion
 
## Symptoms
- Application logs: "too many connections" errors
- 500 error rate spike on database-dependent endpoints
- pg_stat_activity shows max connections reached
 
## Diagnosis
# Check current connection count
psql -h \(DB_HOST -U \)DB_USER -c "SELECT COUNT(*) FROM pg_stat_activity;"
 
# See connections by application
psql -h \(DB_HOST -U \)DB_USER \
  -c "SELECT application_name, COUNT(*) FROM pg_stat_activity GROUP BY 1 ORDER BY 2 DESC;"

## Resolution
1. Identify and restart the service causing the connection leak
2. If immediate relief needed: kill idle connections older than 10 minutes
3. Long-term: review connection pool settings in application config

## Escalation
If unresolved in 30 minutes: page the on-call backend engineer.
</code></pre>
<ol>
<li><strong>An architecture README in every repository.</strong> Every engineer who clones your repository should be able to understand what it does, how to run it locally, how to deploy it, and what it depends on without asking anyone.</li>
</ol>
<h2 id="heading-mistake-10-solving-technical-problems-without-understanding-the-business">Mistake 10: Solving Technical Problems Without Understanding the Business</h2>
<h3 id="heading-the-scenario">The Scenario</h3>
<p>A startup is experiencing slow page loads. A DevOps engineer decides to solve it by migrating to Kubernetes with horizontal pod auto-scaling. The migration takes six weeks. Page loads improve slightly. But 80% of the slowness was caused by unoptimized database queries that had nothing to do with the infrastructure layer. The six-week migration solved 20% of the problem.</p>
<h3 id="heading-the-business-impact">The Business Impact</h3>
<p>Technical solutions to misdiagnosed problems are extraordinarily expensive. Every hour spent building the wrong solution is an hour not spent on the right one. Infrastructure is a tool for delivering business outcomes not an end in itself.</p>
<h3 id="heading-the-fix">The Fix</h3>
<p>Before making any infrastructure decision, answer these four questions:</p>
<ol>
<li><p><strong>What is the actual, measured bottleneck?</strong> Instrument before you act. The bottleneck is almost never where you assumed it was.</p>
</li>
<li><p><strong>What does success look like, and how will you measure it?</strong> "Pages are faster" is not measurable. "p95 page load time drops below 1.2 seconds" is measurable.</p>
</li>
<li><p><strong>What is the full cost of this solution?</strong> Time to implement, ongoing operational burden, team learning curve. Is this cost justified by the measured impact?</p>
</li>
<li><p><strong>Can a simpler solution solve 80% of the problem in 20% of the time?</strong></p>
</li>
</ol>
<p>Always profile and measure before you rebuild:</p>
<pre><code class="language-bash"># Check slow queries in PostgreSQL before any infrastructure changes
psql -h \(DB_HOST -U \)DB_USER -d $DB_NAME -c "
SELECT
  query,
  calls,
  total_exec_time / calls AS avg_ms,
  rows / calls AS avg_rows
FROM pg_stat_statements
ORDER BY avg_ms DESC
LIMIT 10;
"
</code></pre>
<p>Nine times out of ten, slow applications have slow queries, missing indexes, or an N+1 query problem, none of which require a new infrastructure layer to fix.</p>
<h2 id="heading-the-system-thinking-framework-every-devops-engineer-needs">The System Thinking Framework Every DevOps Engineer Needs</h2>
<p>Most of the mistakes above share a common root cause: the engineer was thinking about one component in isolation instead of the full system.</p>
<img src="https://cdn.hashnode.com/uploads/covers/65a5bfab4c73b29396c0b895/b33035a6-448f-419b-b293-206b7b775594.jpg" alt="A diagram showing a request flowing through a full system: user → CDN → load balancer → application servers → cache → database → logs/monitoring" style="display: block;" width="544" height="650" loading="lazy">

<p>A system thinker asks six questions before making any change in production:</p>
<table>
<thead>
<tr>
<th>Question</th>
<th>Why You Ask It</th>
</tr>
</thead>
<tbody><tr>
<td><strong>What does this change?</strong></td>
<td>List every configuration, file, or service that will be different.</td>
</tr>
<tr>
<td><strong>What does this depend on?</strong></td>
<td>What must be true upstream for this component to work correctly?</td>
</tr>
<tr>
<td><strong>What depends on this?</strong></td>
<td>What downstream systems are affected if this changes or fails?</td>
</tr>
<tr>
<td><strong>What is the failure mode?</strong></td>
<td>Does this fail loudly (500 errors) or silently (wrong data)?</td>
</tr>
<tr>
<td><strong>What is the rollback path?</strong></td>
<td>How do you reverse this in under five minutes?</td>
</tr>
<tr>
<td><strong>What does healthy look like after the change?</strong></td>
<td>What metrics confirm everything is working correctly?</td>
</tr>
</tbody></table>
<p>This is not a checklist you run through slowly. It is a thinking habit that becomes automatic with practice. Senior engineers do not spend more time on deployments than junior engineers do, they spend their time on different things, and this is one of them.</p>
<h2 id="heading-your-production-readiness-checklist">Your Production Readiness Checklist</h2>
<p>Use this checklist before any production system goes live. Mark each item as done, in progress, or not yet started.</p>
<h3 id="heading-infrastructure">Infrastructure</h3>
<ul>
<li><p>Infrastructure is defined as code (Terraform or CloudFormation) and version-controlled in Git</p>
</li>
<li><p>Separate dev, staging, and production environments exist with separate credentials</p>
</li>
<li><p>All production changes go through an automated CI/CD pipeline, no manual SSH deployments</p>
</li>
<li><p>You can rebuild the entire production environment from code in under two hours</p>
</li>
</ul>
<h3 id="heading-security">Security</h3>
<ul>
<li><p>No secrets, credentials, or API keys exist in any Git repository</p>
</li>
<li><p>All production secrets are in Secrets Manager or SSM Parameter Store</p>
</li>
<li><p>All IAM roles follow the principle of least privilege</p>
</li>
<li><p>S3 buckets have public access blocked by default</p>
</li>
<li><p>Port 22 is not open to <code>0.0.0.0/0</code> on any security group</p>
</li>
<li><p>CloudTrail is enabled in all regions</p>
</li>
<li><p>All IAM users have MFA enabled</p>
</li>
<li><p>AWS Security Hub is enabled and findings are reviewed weekly</p>
</li>
</ul>
<h3 id="heading-observability">Observability</h3>
<ul>
<li><p>Every service has a <code>/health</code> endpoint that monitoring checks continuously</p>
</li>
<li><p>Alerts fire within five minutes of a production error rate spike</p>
</li>
<li><p>Dashboards exist showing latency, error rate, and resource utilization</p>
</li>
<li><p>Logs are centralized and searchable, not scattered across individual servers</p>
</li>
</ul>
<h3 id="heading-reliability">Reliability</h3>
<ul>
<li><p>Production database has Multi-AZ enabled</p>
</li>
<li><p>Backup restoration has been tested in the last 30 days</p>
</li>
<li><p>Written runbooks exist for the three most likely failure scenarios</p>
</li>
<li><p>RTO and RPO requirements are documented and the architecture meets them</p>
</li>
</ul>
<h3 id="heading-documentation">Documentation</h3>
<ul>
<li><p>Every repository has a README explaining what it does and how to deploy it</p>
</li>
<li><p>A new engineer could understand the production architecture from documentation alone</p>
</li>
<li><p>No single engineer holds critical knowledge that lives only in their head</p>
</li>
</ul>
<h2 id="heading-conclusion">Conclusion</h2>
<p>None of the mistakes in this article require rare misfortune to experience. They are the predictable result of decisions that feel reasonable under startup pressure but accumulate into real operational risk over time.</p>
<p>The good news is that every single one of them is preventable with the right awareness and the right habits applied early.</p>
<p>You do not need a perfect infrastructure from day one. You need a correct one: version-controlled, automated, observable, secure, and documented. Start with that foundation. Add complexity only when a specific, measured problem requires it. Always connect technical decisions to business outcomes.</p>
<p>The goal of DevOps in a startup is not to build impressive infrastructure. It is to build reliable systems that support product growth safely, efficiently, and sustainably and to make sure that when something does break, you can recover faster than anyone notices.</p>
<h2 id="heading-want-to-go-deeper">Want to Go Deeper?</h2>
<p>If this article resonated with you, <a href="https://coachli.co/tolani-akintayo/PR-H4oQS"><strong>The Startup DevOps Field Guide</strong></a> covers these principles in full depth with complete infrastructure blueprints, security frameworks, CI/CD pipeline templates, and the end-to-end decision-making playbook for engineers building DevOps practices in startup environments from scratch.</p>
<p>It is written specifically for the engineer who wants to do this right from the beginning not the one rebuilding everything after the first major incident.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ AWS Certified Cloud Practitioner Study Course – Pass the Exam With This Free 14-Hour Course ]]>
                </title>
                <description>
                    <![CDATA[ Passing the AWS Certified Cloud Practitioner Exam is one of the first steps to a career in cloud development. And freeCodeCamp just published a free 14-hour course that will help you prepare for the e ]]>
                </description>
                <link>https://www.freecodecamp.org/news/aws-certified-cloud-practitioner-study-course-pass-the-exam-with-this-free-13-hour-course/</link>
                <guid isPermaLink="false">66b200a5276d158502db2eb1</guid>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ youtube ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Beau Carnes ]]>
                </dc:creator>
                <pubDate>Thu, 14 May 2026 13:00:00 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5f68e7df6dfc523d0a894e7c/f493e192-b126-4291-8884-d2e2ff621df4.jpg" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Passing the AWS Certified Cloud Practitioner Exam is one of the first steps to a career in cloud development. And freeCodeCamp just published a free 14-hour course that will help you prepare for the exam.</p>
<p>This course has been updated for 2026.</p>
<p>This exam mostly deals with cloud computing concepts. Even if you are new to coding, you should be able to prepare for this exam and earn the AWS certification. Andrew Brown created this course. He is a popular instructor and the CEO of ExamPro.</p>
<h2 id="heading-what-is-the-aws-certified-cloud-practitioner">What is the AWS Certified Cloud Practitioner?</h2>
<p>The Certified Cloud Practitioner is the entry-level AWS certification that goes through:</p>
<ul>
<li><p>The cloud fundamentals, for example Cloud Concepts, Cloud Architecture, and Cloud Deployment Models</p>
</li>
<li><p>A close look at the AWS Core Services</p>
</li>
<li><p>A quick look at the vast amount of AWS services</p>
</li>
<li><p>Identity, Security, and Governance of the Cloud</p>
</li>
<li><p>Billing, Pricing, and Support of AWS Services</p>
</li>
</ul>
<p>The course code is CLF-C02 but its commonly referred to as the CCP.</p>
<p>Amazon Web Services is the leading Cloud Service Provider (CSP) in the world and the AWS Certified Cloud Practitioner is the most common starting point for people breaking into the cloud industry.</p>
<p>Consider the AWS Certified Cloud Practitioner if:</p>
<ul>
<li><p>You are new to cloud and need to learn the fundamentals</p>
</li>
<li><p>You are in the executive, management, or sales level and need to acquire strategic information about cloud for adoption or migration</p>
</li>
<li><p>You are a Senior Cloud Engineer or Solutions Architect who needs to reset or refresh your AWS knowledge after working for multiple years</p>
</li>
</ul>
<p>No matter your path towards a cloud role, the AWS Certified Cloud Practitioner provides fundamental knowledge that you shouldn't skip.</p>
<p>Here are all the sections in this comprehensive course:</p>
<h3 id="heading-introduction">Introduction</h3>
<p>🎤 Is Certified Cloud Practitioner right for me?<br>🎤 Exam Guide<br>🎤 Practice Exam Sample<br>🎤 Case Study Question Type<br>🎤 Validators</p>
<h3 id="heading-cloud-concepts">Cloud Concepts</h3>
<p>🎤 What is Cloud Computing<br>🎤 Evolution of Cloud Hosting<br>🎤 What is Amazon<br>🎤 What is AWS<br>🎤 What is a Cloud Service Provider<br>🎤 Landscape of CSPs<br>🎤 Gartner Magic Quadrant for Cloud<br>🎤 Common Cloud Services<br>🎤 AWS Technology Overview<br>🎤 AWS Services Preview<br>🎤 Evolution of Computing<br>🎤 Types of Cloud Computing<br>🎤 Cloud Computing Deployment Models<br>🎤 Deployment Model Use Cases</p>
<h3 id="heading-getting-started">Getting Started</h3>
<p>🎤 Create an AWS Account<br>🎤 Create IAM User<br>🎤 AWS Region Selector<br>🎤 Overbilling Story<br>🎤 AWS Budgets<br>🎤 AWS Free Tier<br>🎤 Billing Alarm<br>🎤 Turning on MFA</p>
<h3 id="heading-digital-transformation">Digital Transformation</h3>
<p>🎤 Innovation Waves<br>🎤 Burning Platform<br>🎤 Digital Transformation Checklist<br>🎤 Evolution of Computing Power<br>🎤 Amazon Braket</p>
<h3 id="heading-the-benefits-of-cloud">The Benefits of Cloud</h3>
<p>🎤 The Benefits of the Cloud<br>🎤 The Six Advantages of Cloud<br>🎤 The Six Advantages of Cloud Doc Reference<br>🎤 The Seven Advantages of Cloud</p>
<h3 id="heading-global-infrastructure">Global Infrastructure</h3>
<p>🎤 AWS Global Infrastructure Overview<br>🎤 AWS Global Infrastructure Follow Along<br>🎤 Regions<br>🎤 Regional vs Global Services<br>🎤 Availability Zones AZs<br>🎤 Regions and AZ Visualized<br>🎤 Selecting Regions and Azs Follow Along<br>🎤 Fault Tolerance<br>🎤 AWS Global Network<br>🎤 Points of Presence PoP<br>🎤 Tier 1<br>🎤 AWS Services using PoPs<br>🎤 AWS Direct Connect<br>🎤 Direct Connect Locations<br>🎤 AWS Local Zones<br>🎤 Wavelength Zones<br>🎤 Data Residency<br>🎤 AWS for Government<br>🎤 GovCloud<br>🎤 AWS in China<br>🎤 AWS in China Follow Along<br>🎤 Sustainability<br>🎤 Sustainability Follow Along<br>🎤 AWS Ground Station<br>🎤 AWS Outposts</p>
<h3 id="heading-cloud-architecture">Cloud Architecture</h3>
<p>🎤 Cloud Architecture Terminologies<br>🎤 High Availability<br>🎤 High Scalability<br>🎤 High Elasticity<br>🎤 Fault Tolerance<br>🎤 High Durability<br>🎤 Business Continuity Plan<br>🎤 Disaster Recovery Options<br>🎤 RTO Visualized<br>🎤 RPO Visualized<br>🎤 Architectural diagram example<br>🎤 HA Follow Along</p>
<h3 id="heading-management-and-developer-tools">Management and Developer Tools</h3>
<p>🎤 AWS API<br>🎤 AWS API Follow Along<br>🎤 AWS Management Console<br>🎤 AWS Management Console Follow Along<br>🎤 Service Console<br>🎤 Service Console Follow Along<br>🎤 AWS Account ID<br>🎤 AWS Account ID Follow Along<br>🎤 AWS Tools for PowerShell<br>🎤 AWS Tools for Powershell Follow Along<br>🎤 Amazon Resource Names<br>🎤 ARN Follow Along<br>🎤 AWS CLI<br>🎤 AWS CLI Follow Along<br>🎤 AWS SDK<br>🎤 AWS SDK Follow Along<br>🎤 AWS CloudShell<br>🎤 Infrastructure as Code<br>🎤 CloudFormation<br>🎤 CloudFormation Follow Along<br>🎤 CDK<br>🎤 CDK Follow Along<br>🎤 AWS Toolkit for VSCode<br>🎤 Access Keys<br>🎤 Access Keys Follow Along<br>🎤 AWS Documentation<br>🎤 AWS Documentation Follow Along</p>
<h3 id="heading-shared-responsibility-model">Shared Responsibility Model</h3>
<p>🎤 Introduction to Shared Responsibility Model<br>🎤 AWS Shared Responsibility Model<br>🎤 Types of Cloud Responsibilities<br>🎤 Shared Responsibility for Compute<br>🎤 Shared Responsibility Model Alternate<br>🎤 Shared Responsibility Model Architecture</p>
<h3 id="heading-compute">Compute</h3>
<p>🎤 EC2 Overview<br>🎤 VMs Containers and Serverless<br>🎤 Compute Follow Along<br>🎤 High Performance Computing HPC<br>🎤 HPC Follow Along<br>🎤 Edge and Hybrid<br>🎤 Edge Computing Follow Along<br>🎤 Cost Capacity Management</p>
<h3 id="heading-storage-services">Storage Services</h3>
<p>🎤 Types of Storage Services<br>🎤 Introduction to S3<br>🎤 S3 Storage Classes<br>🎤 AWS Snow Family<br>🎤 Storage Services<br>🎤 S3 Follow Along<br>🎤 EBS Follow Along<br>🎤 EFS Follow Along<br>🎤 Snow Family Follow Along</p>
<h3 id="heading-databases">Databases</h3>
<p>🎤 What is a database<br>🎤 What is a data warehouse<br>🎤 What is a key value store<br>🎤 What is a document database<br>🎤 NoSQL Database Services<br>🎤 Relational Database Services<br>🎤 Other Database Services<br>🎤 DynamoDB Follow Along<br>🎤 RDS Follow Along<br>🎤 Redshift Follow Along</p>
<h3 id="heading-networking">Networking</h3>
<p>🎤 Cloud Native Networking Services<br>🎤 Enterprise Hybrid Networking Services<br>🎤 Virtual Private Cloud VPC Subnets<br>🎤 Security Groups vs NACLs<br>🎤 Security Groups vs NACLs Follow Along<br>🎤 AWS CloudFront</p>
<h3 id="heading-ec2">EC2</h3>
<p>🎤 Introduction to EC2<br>🎤 EC2 Instance Families<br>🎤 EC2 Instance Types<br>🎤 Dedicated Host vs Dedicated Instances<br>🎤 EC2 Tenancy<br>🎤 Launch an EC2 SSH and Sessions Manager<br>🎤 Elastic IP<br>🎤 AMI and Launch Template<br>🎤 Launch an ASG<br>🎤 Launch an ALB<br>🎤 Cleanup</p>
<h3 id="heading-ec2-pricing-models">EC2 Pricing Models</h3>
<p>🎤 Ec2 Pricing Models<br>🎤 On Demand<br>🎤 Reserved Instances<br>🎤 RI Attributes<br>🎤 Regional and Zonal RI<br>🎤 RI Limits<br>🎤 Capacity Reservations<br>🎤 Standard vs Convertible RI<br>🎤 RI Marketplace<br>🎤 Spot Instances<br>🎤 Dedicated Instances<br>🎤 Savings Plan</p>
<h3 id="heading-identity">Identity</h3>
<p>🎤 Zero Trust Model<br>🎤 Zero Trust on AWS<br>🎤 Zero Trust on AWS with Third Parties<br>🎤 Directory Service<br>🎤 Active Directory<br>🎤 Identity Providers<br>🎤 Single Sign On<br>🎤 LDAP<br>🎤 Multi Factor Authenication<br>🎤 Security Keys<br>🎤 AWS IAM<br>🎤 Anatomy of an IAM Policy<br>🎤 IAM Policies Follow Along<br>🎤 Principle of Least Priivilege<br>🎤 AWS Account Root User<br>🎤 AWS SSO</p>
<h3 id="heading-application-integration">Application Integration</h3>
<p>🎤 Introduction to Application Integration<br>🎤 Queueing and SQS<br>🎤 Streaming and Kinesis<br>🎤 Pub Sub and SNS<br>🎤 API Gateway and Amazon API Gateway<br>🎤 State Machines and AWS Step Functions<br>🎤 Event Bus and Amazon Event Bridge<br>🎤 Application Integration Services</p>
<h3 id="heading-containers">Containers</h3>
<p>🎤 VMs vs Containers<br>🎤 What are Microservices<br>🎤 Kuberenetes<br>🎤 Docker<br>🎤 Podman<br>🎤 Container Services</p>
<h3 id="heading-governance">Governance</h3>
<p>🎤 Organizations and Accounts<br>🎤 AWS Control Tower<br>🎤 AWS Config<br>🎤 AWS Config FollowAlong<br>🎤 AWS Quick Starts<br>🎤 AWS QuickStarts Follow Along<br>🎤 Tagging<br>🎤 Tag Name Follow Along<br>🎤 Resource Groups<br>🎤 Resource Groups Follow Along<br>🎤 Business Centric Services</p>
<h3 id="heading-provisioning">Provisioning</h3>
<p>🎤 Provisioning Services<br>🎤 AWS Elastic Beanstalk<br>🎤 AWS Elastic Beanstalk Follow Along</p>
<h3 id="heading-serverless">Serverless</h3>
<p>🎤 What is Serverless<br>🎤 Serverless Services</p>
<h3 id="heading-windows-on-aws">Windows on AWS</h3>
<p>🎤 Windows on AWS<br>🎤 EC2 Windows Follow Along<br>🎤 AWS License Manager</p>
<h3 id="heading-logging">Logging</h3>
<p>🎤 Logging Services<br>🎤 AWS Cloud Trail<br>🎤 CloudWatch Alarm<br>🎤 Anatomy of an Alarm<br>🎤 Log Streams and Events<br>🎤 Log Insights<br>🎤 CloudWatch Metrics<br>🎤 AWS CloudTrail Follow Along</p>
<h3 id="heading-ml-ai-bigdata">ML AI BigData</h3>
<p>🎤 Introduction to ML and AI<br>🎤 AI and ML Services<br>🎤 BigData and Analytics Services<br>🎤 Amazon QuickSight<br>🎤 QuickSight Follow Along<br>🎤 Machine Learning and AI Services Extended<br>🎤 Generative AI<br>🎤 ML and DL Frameworks and Tools<br>🎤 Apache MXNet<br>🎤 What is Intel<br>🎤 Intel Xeon Scalable and Intel Gaudi<br>🎤 What is a GPU<br>🎤 What is CUDA</p>
<h3 id="heading-aws-well-architected-framework">AWS Well Architected Framework</h3>
<p>🎤 AWS Well Architected Framework<br>🎤 General Defintions<br>🎤 On Architecture<br>🎤 Amazon Leadership Principles<br>🎤 General Design Principles<br>🎤 Anatomy of a Pillar<br>🎤 Operational Excellence<br>🎤 Security<br>🎤 Reliability<br>🎤 Performance Efficiency<br>🎤 Cost Optimization<br>🎤 AWS Well Architected Tool<br>🎤 Well Architected Framework and Tool Follow Along<br>🎤 AWS Architecture Center</p>
<h3 id="heading-tco-and-migration">TCO and Migration</h3>
<p>🎤 Total Cost of Ownership TCO<br>🎤 CAPEX vs OPEX<br>🎤 Shifting IT Personnel<br>🎤 AWS Pricing Calculator<br>🎤 AWS Pricing Calculator Follow Along<br>🎤 Migration Evaluator<br>🎤 VM Import Export<br>🎤 Database Migration Service<br>🎤 Cloud Adoption Framework</p>
<h3 id="heading-billing-and-pricing">Billing and Pricing</h3>
<p>🎤 AWS Free Services<br>🎤 AWS Support Plans<br>🎤 Technical Account Manager<br>🎤 AWS Support Follow Along<br>🎤 AWS Marketplace<br>🎤 AWS Marketplace Follow Along<br>🎤 Consolidated Billing<br>🎤 Consolidated Billing Volume Discounts<br>🎤 AWS Trusted Advisor<br>🎤 AWS Trusted Advisor Follow Along<br>🎤 SLAs<br>🎤 AWS SLA Examples<br>🎤 AWS SLA Follow Along<br>🎤 Service Health Dashboard<br>🎤 AWS Personal Health Dashboard<br>🎤 AWS Abuse<br>🎤 AWS Abuse Report Follow Along<br>🎤 AWS Free Tier<br>🎤 AWS Credits<br>🎤 AWS Partner Network<br>🎤 AWS Budgets<br>🎤 AWS Budget Reports<br>🎤 AWS Cost and Usage Reports<br>🎤 Cost Allocation Tags<br>🎤 Billing Alarms<br>🎤 AWS Cost Explorer<br>🎤 AWS Cost Explorer Follow Along<br>🎤 Programmatic Pricing APIs<br>🎤 AWS Savings Plan Follow Along</p>
<h3 id="heading-security">Security</h3>
<p>🎤 Defense In Depth<br>🎤 CIA Triad<br>🎤 Vulnerabilities<br>🎤 Encryption<br>🎤 Cyphers<br>🎤 Cryptographic Keys<br>🎤 Hashing and Salting<br>🎤 Digital Signatures and Signing<br>🎤 In Transit vs At Rest Encryption<br>🎤 Compliance Programs<br>🎤 AWS Compliance Programs Follow Along<br>🎤 Pen Testing<br>🎤 Pen Testing Follow Along<br>🎤 AWS Artifact<br>🎤 AWS Artifact Follow Along<br>🎤 AWS Inspector<br>🎤 DDoS<br>🎤 AWS Shield<br>🎤 AWS Guard Duty<br>🎤 AWS Guard Duty Follow Along<br>🎤 Amazon Macie<br>🎤 AWS VPN<br>🎤 AWS WAF<br>🎤 AWS WAF Follow Along<br>🎤 Hardware Security Module<br>🎤 AWS KMS<br>🎤 AWS KMS Follow Along<br>🎤 CloudHSM</p>
<h3 id="heading-variation-study">Variation Study</h3>
<p>🎤 Know Your Initialisms<br>🎤 AWS Config AWS AppConfig<br>🎤 SNS vs SQS<br>🎤 SNS vs SES vs PinPoint vs Workmail<br>🎤 Amazon Inspector vs AWS Trusted Advisor<br>🎤 Connect Named Services<br>🎤 Elastic Transcoder vs MediaConvert<br>🎤 AWS Artifact vs Amazon Inspector<br>🎤 ELB variants</p>
<p>You can watch the entire <a href="https://youtu.be/7HKot-brXFE">course on the freeCodeCamp.org</a> (14-hour course).</p>
<div class="embed-wrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/7HKot-brXFE" style="aspect-ratio: 16 / 9; width: 100%; height: auto;" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen="" loading="lazy"></iframe></div>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Migrate to S3 Native State Locking in Terraform ]]>
                </title>
                <description>
                    <![CDATA[ If you've been running Terraform on AWS for any length of time, you know the setup: an S3 bucket for state storage, a DynamoDB table for state locking, and a handful of IAM policies tying them togethe ]]>
                </description>
                <link>https://www.freecodecamp.org/news/how-to-migrate-to-s3-native-state-locking-in-terraform/</link>
                <guid isPermaLink="false">69fd19239f93a850a430069b</guid>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Terraform ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Cloud Computing ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Infrastructure as code ]]>
                    </category>
                
                    <category>
                        <![CDATA[ S3 ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Tolani Akintayo ]]>
                </dc:creator>
                <pubDate>Thu, 07 May 2026 22:58:43 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/9619ad45-15c5-4be7-9221-ed4b76bc2b24.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>If you've been running Terraform on AWS for any length of time, you know the setup: an S3 bucket for state storage, a DynamoDB table for state locking, and a handful of IAM policies tying them together. It works. It has worked for years.</p>
<p>But it has always carried a cost that rarely gets discussed openly. That cost isn't just money, though a DynamoDB table with on-demand billing adds up across multiple teams and environments.</p>
<p>The real cost is complexity. Every new AWS environment needs both resources provisioned before Terraform can manage anything else. Every engineer who sets up their first Terraform backend has to understand why two completely different AWS services are responsible for what is logically one thing: storing and protecting state. And every incident involving a stuck lock has required someone to manually delete a record from DynamoDB to unblock the team.</p>
<p>In November 2024, AWS announced that S3 now supports native object locking for Terraform state files, meaning <strong>DynamoDB is no longer required for state locking</strong>. Terraform 1.10 added support for this feature, and it's now generally available.</p>
<p>In this tutorial, you'll learn:</p>
<ul>
<li><p>What S3 native locking is and how it works</p>
</li>
<li><p>How to set it up from scratch if you're starting a new project</p>
</li>
<li><p>How to migrate an existing S3 + DynamoDB setup to S3 native locking safely</p>
</li>
<li><p>How to verify locking is working and handle edge cases</p>
</li>
</ul>
<p>By the end, you'll have a simpler, cleaner Terraform backend with one fewer AWS resource to manage.</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ul>
<li><p><a href="#heading-what-is-terraform-state-locking">What Is Terraform State Locking?</a></p>
</li>
<li><p><a href="#heading-what-is-s3-native-state-locking">What Is S3 Native State Locking?</a></p>
</li>
<li><p><a href="#heading-how-s3-native-locking-compares-to-the-s3-dynamodb-approach">How S3 Native Locking Compares to the S3 + DynamoDB Approach</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-part-1-fresh-setup-how-to-configure-s3-native-locking-from-scratch">Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch</a></p>
<ul>
<li><p><a href="#heading-step-1-create-the-s3-bucket-with-versioning-and-encryption">Step 1: Create the S3 Bucket with Versioning and Encryption</a></p>
</li>
<li><p><a href="#heading-step-2-configure-the-terraform-backend-with-native-locking">Step 2: Configure the Terraform Backend with Native Locking</a></p>
</li>
<li><p><a href="#heading-step-3-initialize-and-verify">Step 3: Initialize and Verify</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-part-2-migration-how-to-move-from-s3-dynamodb-to-s3-native-locking">Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking</a></p>
<ul>
<li><p><a href="#heading-step-1-verify-your-current-setup">Step 1: Verify Your Current Setup</a></p>
</li>
<li><p><a href="#heading-step-2-enable-object-lock-on-the-existing-s3-bucket">Step 2: Enable Object Lock on the Existing S3 Bucket</a></p>
</li>
<li><p><a href="#heading-step-3-update-the-terraform-backend-configuration">Step 3: Update the Terraform Backend Configuration</a></p>
</li>
<li><p><a href="#heading-step-4-reinitialize-terraform">Step 4: Reinitialize Terraform</a></p>
</li>
<li><p><a href="#heading-step-5-verify-the-migration">Step 5: Verify the Migration</a></p>
</li>
<li><p><a href="#heading-step-6-clean-up-the-dynamodb-table">Step 6: Clean Up the DynamoDB Table</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-how-to-verify-that-locking-is-working">How to Verify That Locking Is Working</a></p>
</li>
<li><p><a href="#heading-how-to-handle-a-stuck-lock">How to Handle a Stuck Lock</a></p>
</li>
<li><p><a href="#heading-rollback-plan-if-something-goes-wrong">Rollback Plan: If Something Goes Wrong</a></p>
</li>
<li><p><a href="#heading-security-best-practices-for-your-state-bucket">Security Best Practices for Your State Bucket</a></p>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
</li>
<li><p><a href="#heading-references">References</a></p>
</li>
</ul>
<h2 id="heading-what-is-terraform-state-locking">What is Terraform State Locking?</h2>
<p>Before looking at the new approach, it helps to understand what state locking is solving.</p>
<p>Terraform stores everything it knows about your infrastructure in a <strong>state file</strong> – a JSON document that maps your configuration to real AWS resources. When you run <code>terraform apply</code>, Terraform reads this file, calculates the difference between the current state and your configuration, and makes the necessary changes.</p>
<p>The problem arises when two engineers or two CI/CD pipelines run and try to apply changes at the same time. If both read the state file simultaneously, calculate changes independently, and both try to write back, you get a <strong>race condition</strong>. The second write overwrites changes from the first, and your state is now out of sync with reality. This is a serious problem that can cause resources to be untracked, doubled, or destroyed unexpectedly.</p>
<p><strong>State locking</strong> solves this by creating a lock when any operation starts that could modify state. If a lock already exists, Terraform refuses to proceed and reports who holds the lock and when it was acquired. Only one operation can hold the lock at a time. When the operation completes, the lock is released.</p>
<pre><code class="language-plaintext">Terraform Run A                 State File / Lock                Terraform Run B
(User 1)                         (S3/DynamoDB)                   (User 2)

   |                                   |                            |
   |------- 1. Acquire Lock ----------&gt;|                            |
   |                                   |                            |
   |&lt;------ 2. Lock Granted -----------|                            |
   |                                   |                            |
   |                                   |------- 3. Acquire Lock ---&gt;|
   |            [PROCESSING]           |                            |
   |      (Modifying Infrastructure)   |&lt;------ 4. Lock Denied -----|
   |                                   |        (Wait / Retry)      |
   |                                   |                            |
   |------- 5. Release Lock ----------&gt;|                            |
   |                                   |                            |
   |           [COMPLETED]             |&lt;------ 6. Lock Granted ----|
   |                                   |                            |
   |                                   |       [PROCESSING]         |
   |                                   | (Modifying Infrastructure) |              
   |                                   |                            |
</code></pre>
<h2 id="heading-what-is-s3-native-state-locking">What Is S3 Native State Locking?</h2>
<p>Previously, Terraform's S3 backend used a DynamoDB table as the locking mechanism. When a lock was needed, Terraform wrote a record to DynamoDB with a <code>LockID</code> primary key. DynamoDB's conditional writes guaranteed that only one process could create that record, which is what made the locking atomic.</p>
<p>S3 native locking uses <strong>S3 Object Lock</strong> instead. S3 Object Lock is an S3 feature originally designed to enforce WORM (Write Once, Read Many) compliance for regulatory requirements. AWS extended this capability to support Terraform's state locking workflow.</p>
<p>When S3 native locking is enabled in your Terraform backend:</p>
<ol>
<li><p>Terraform writes your state to an <code>.tfstate</code> object in S3 (as before)</p>
</li>
<li><p>To acquire a lock, Terraform uses <strong>S3's conditional write operations</strong> – specifically the <code>if-none-match</code> conditional header to create a lock file atomically</p>
</li>
<li><p>If the lock file already exists, S3 rejects the write, and Terraform reports that a lock is held</p>
</li>
<li><p>When the operation completes, Terraform deletes the lock file to release the lock.</p>
</li>
</ol>
<p>The key difference from DynamoDB: the entire locking mechanism lives inside S3. No second service. No second set of IAM permissions. No second resource to provision.</p>
<p><strong>Note:</strong> This feature requires Terraform version <strong>1.10.0 or later</strong> and an S3 bucket with <strong>Object Lock enabled</strong>. Object Lock must be enabled at bucket creation time. You can't enable it on an existing bucket through the console or CLI. But there is a supported workaround for existing buckets, which we'll cover in Part 2.</p>
<h2 id="heading-how-s3-native-locking-compares-to-the-s3-dynamodb-approach">How S3 Native Locking Compares to the S3 + DynamoDB Approach</h2>
<table>
<thead>
<tr>
<th><strong>Aspect</strong></th>
<th><strong>S3 + DynamoDB (Old)</strong></th>
<th><strong>S3 Native Locking (New)</strong></th>
</tr>
</thead>
<tbody><tr>
<td><strong>AWS services required</strong></td>
<td>S3 + DynamoDB</td>
<td>S3 only</td>
</tr>
<tr>
<td><strong>IAM permissions needed</strong></td>
<td>S3 + DynamoDB permissions</td>
<td>S3 permissions only</td>
</tr>
<tr>
<td><strong>Terraform version</strong></td>
<td>Any</td>
<td>1.10.0 or later</td>
</tr>
<tr>
<td><strong>Setup complexity</strong></td>
<td>Two resources, two IAM scopes</td>
<td>One resource</td>
</tr>
<tr>
<td><strong>Stuck lock resolution</strong></td>
<td>Delete DynamoDB record</td>
<td>Delete S3 lock file</td>
</tr>
<tr>
<td><strong>Cost</strong></td>
<td>S3 storage + DynamoDB on-demand</td>
<td>S3 storage only</td>
</tr>
<tr>
<td><strong>Object Lock requirement</strong></td>
<td>Not required</td>
<td>Required on S3 bucket</td>
</tr>
<tr>
<td><strong>Locking mechanism</strong></td>
<td>DynamoDB conditional writes</td>
<td>S3 conditional writes (<code>if-none-match</code>)</td>
</tr>
<tr>
<td><strong>State versioning</strong></td>
<td>S3 Versioning (recommended)</td>
<td>S3 Versioning (required for full safety)</td>
</tr>
</tbody></table>
<p>The functional behavior from Terraform's perspective is identical. Locking works the same way. The lock information displayed when a lock is held has the same structure. The only difference is what happens under the hood.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>Before you start, make sure you have the following in place:</p>
<ul>
<li><strong>Terraform 1.10.0 or later</strong> installed. Check your version:</li>
</ul>
<pre><code class="language-shell">terraform version
</code></pre>
<p>If you need to upgrade, follow the <a href="https://developer.hashicorp.com/terraform/install">official upgrade guide</a>.</p>
<ul>
<li><strong>AWS CLI</strong> installed and configured with credentials that have permission to create and manage S3 buckets.</li>
</ul>
<pre><code class="language-shell">aws --version
aws sts get-caller-identity   # confirm you're authenticated
</code></pre>
<ul>
<li><p><strong>IAM permissions</strong> to perform the following S3 actions:</p>
<ul>
<li><p><code>s3:CreateBucket</code></p>
</li>
<li><p><code>s3:PutBucketVersioning</code></p>
</li>
<li><p><code>s3:PutBucketEncryption</code></p>
</li>
<li><p><code>s3:PutObjectLegalHold</code></p>
</li>
<li><p><code>s3:PutObjectRetention</code></p>
</li>
<li><p><code>s3:GetObject</code></p>
</li>
<li><p><code>s3:PutObject</code></p>
</li>
<li><p><code>s3:DeleteObject</code></p>
</li>
<li><p><code>s3:ListBucket</code></p>
</li>
</ul>
</li>
<li><p>For the <strong>migration path</strong>: access to your existing Terraform project and the S3 bucket and DynamoDB table currently in use.</p>
</li>
</ul>
<h2 id="heading-part-1-fresh-setup-how-to-configure-s3-native-locking-from-scratch">Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch</h2>
<p>Follow this section if you're starting a new Terraform project and want to use S3 native locking from the beginning.</p>
<h3 id="heading-step-1-create-the-s3-bucket-with-versioning-and-encryption">Step 1: Create the S3 Bucket with Versioning and Encryption</h3>
<p>Object Lock <strong>must be enabled at bucket creation time</strong>. You can't add it afterward through the standard console flow. Create the bucket using the AWS CLI with Object Lock enabled:</p>
<pre><code class="language-shell">aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region us-east-1 \
  --object-lock-enabled-for-bucket
</code></pre>
<p><strong>Note:</strong> For regions other than <code>us-east-1</code>, add the <code>--create-bucket-configuration</code> flag.</p>
<pre><code class="language-shell">aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket
</code></pre>
<p>Now enable versioning on the bucket. Versioning is required alongside Object Lock and allows Terraform to recover previous state versions if something goes wrong:</p>
<pre><code class="language-shell">aws s3api put-bucket-versioning \
  --bucket your-project-terraform-state \
  --versioning-configuration Status=Enabled
</code></pre>
<p>Enable server-side encryption so your state files are encrypted at rest:</p>
<pre><code class="language-shell">aws s3api put-bucket-encryption \
  --bucket your-project-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'
</code></pre>
<p>Block all public access to the bucket. A Terraform state file contains resource IDs, IP addresses, and potentially sensitive values. It should never be publicly accessible:</p>
<pre><code class="language-shell">aws s3api put-public-access-block \
  --bucket your-project-terraform-state \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
</code></pre>
<p>Verify the bucket configuration:</p>
<pre><code class="language-shell"># Confirm Object Lock is enabled
aws s3api get-object-lock-configuration \
  --bucket your-project-terraform-state
 
# Confirm versioning is enabled
aws s3api get-bucket-versioning \
  --bucket your-project-terraform-state
 
# Confirm encryption is configured
aws s3api get-bucket-encryption \
  --bucket your-project-terraform-state
</code></pre>
<p>Expected output for the Object Lock check:</p>
<pre><code class="language-json">{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/65a5bfab4c73b29396c0b895/2b2e56cf-687f-4932-a61e-ed7cc33ea6f1.png" alt="Terminal showing AWS CLI verification commands confirming S3 bucket is configured correctly with Object Lock, versioning, and encryption enabled" style="display: block;" width="1120" height="616" loading="lazy">

<h3 id="heading-step-2-configure-the-terraform-backend-with-native-locking">Step 2: Configure the Terraform Backend with Native Locking</h3>
<p>In your Terraform project, create or update your <code>backend.tf</code> file:</p>
<pre><code class="language-hcl">terraform {
  backend "s3" {
    bucket = "your-project-terraform-state"
    key    = "production/terraform.tfstate"
    region = "us-east-1"
 
    # Enable S3 native state locking
    # Requires Terraform 1.10.0+ and a bucket with Object Lock enabled
    use_lockfile = true
 
    # Encryption at rest
    encrypt = true
  }
}
</code></pre>
<p>The critical difference from the old configuration is the <code>use_lockfile = true</code> parameter. Notice what is <strong>absent</strong>: there's no <code>dynamodb_table</code> argument. No DynamoDB table. No second service.</p>
<p>Here's a direct comparison of the old and new configurations:</p>
<p><strong>Old configuration (S3 + DynamoDB):</strong></p>
<pre><code class="language-hcl">terraform {
  backend "s3" {
    bucket         = "your-project-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # this goes away
  }
}
</code></pre>
<p><strong>New configuration (S3 native locking):</strong></p>
<pre><code class="language-hcl">terraform {
  backend "s3" {
    bucket       = "your-project-terraform-state"
    key          = "production/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # this replaces dynamodb_table
  }
}
</code></pre>
<h3 id="heading-step-3-initialize-and-verify">Step 3: Initialize and Verify</h3>
<p>Run <code>terraform init</code> to initialize the backend:</p>
<pre><code class="language-shell">terraform init
</code></pre>
<p>Expected output:</p>
<pre><code class="language-plaintext">Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
 
Terraform has been successfully initialized!
</code></pre>
<p>Run a plan to confirm everything is working end-to-end:</p>
<pre><code class="language-shell">terraform plan
</code></pre>
<p>If locking is working, you'll see a brief pause while Terraform acquires the lock before the plan output appears. You'll also see the lock information if you look at the S3 bucket&nbsp;– a <code>.tflock</code> file will appear temporarily alongside your state file during the operation and disappear when it completes.</p>
<h2 id="heading-part-2-migration-how-to-move-from-s3-dynamodb-to-s3-native-locking">Part 2: Migration&nbsp;– How to Move from S3 + DynamoDB to S3 Native Locking</h2>
<p>Follow this section if you have an <strong>existing Terraform setup</strong> using an S3 bucket and DynamoDB table for state locking, and you want to migrate to S3 native locking.</p>
<p><strong>Important:</strong> Migration requires a maintenance window or at minimum a period where no Terraform operations are running. You're changing the backend configuration, which means <strong>all team members and CI/CD pipelines must stop running</strong> <code>terraform plan</code> <strong>or</strong> <code>terraform apply</code> <strong>during the migration</strong>. The migration itself takes under 10 minutes.</p>
<h3 id="heading-step-1-verify-your-current-setup">Step 1: Verify Your Current Setup</h3>
<p>Before making any changes, document your existing backend configuration and confirm the state file is accessible:</p>
<pre><code class="language-shell"># Confirm your state file is in S3
aws s3 ls s3://your-existing-bucket/path/to/terraform.tfstate
 
# Confirm the DynamoDB table exists
aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table \
  --query 'Table.TableStatus'
</code></pre>
<p>Check your current <code>backend.tf</code> and note the exact values:</p>
<pre><code class="language-shell"># Your current backend.tf - note these values before changing anything
terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"       # note this
    key            = "path/to/terraform.tfstate"   # note this
    region         = "us-east-1"                   # note this
    encrypt        = true
    dynamodb_table = "your-dynamodb-lock-table"    # this will be removed
  }
}
</code></pre>
<p>Run one final plan to confirm the current state is clean and there are no unexpected changes pending:</p>
<pre><code class="language-shell">terraform plan
</code></pre>
<p>If the plan shows no changes, you're in a safe state to proceed.</p>
<h3 id="heading-step-2-enable-object-lock-on-the-existing-s3-bucket">Step 2: Enable Object Lock on the Existing S3 Bucket</h3>
<p>This is the most important step in the migration. Object Lock can't normally be enabled on an existing bucket. It's a setting that must be configured at creation time.</p>
<p>But AWS provides a way to enable Object Lock on an existing bucket through a support request or through a direct API call that's not exposed in the standard console UI. AWS has officially documented this path for the Terraform migration use case.</p>
<p>Run the following AWS CLI command to enable Object Lock on your <strong>existing</strong> bucket:</p>
<pre><code class="language-bash">aws s3api put-object-lock-configuration \
  --bucket your-existing-bucket \
  --object-lock-configuration '{"ObjectLockEnabled": "Enabled"}'
</code></pre>
<p><strong>Note:</strong> This command enables Object Lock in <strong>governance mode with no default retention</strong>, meaning it enables the locking capability without setting a default retention period on all objects. This is exactly what Terraform's native locking needs: the ability to create and delete lock files, not permanent object retention.</p>
<p>Verify Object Lock is now enabled:</p>
<pre><code class="language-shell">aws s3api get-object-lock-configuration \
  --bucket your-existing-bucket
</code></pre>
<p>Expected output:</p>
<pre><code class="language-json">{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}
</code></pre>
<p>Also verify that versioning is already enabled (it should be if you are running a production Terraform setup):</p>
<pre><code class="language-shell">aws s3api get-bucket-versioning \
  --bucket your-existing-bucket
</code></pre>
<p>Expected output:</p>
<pre><code class="language-json">{
    "Status": "Enabled"
}
</code></pre>
<p>If versioning isn't enabled, enable it before proceeding:</p>
<pre><code class="language-shell">aws s3api put-bucket-versioning \
  --bucket your-existing-bucket \
  --versioning-configuration Status=Enabled
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/65a5bfab4c73b29396c0b895/cd17df01-3d0a-4f93-9250-3f51627e91c8.png" alt="Terminal output showing successful Object Lock enablement on an existing S3 bucket using the AWS CLI" style="display: block;" width="1204" height="320" loading="lazy">

<h3 id="heading-step-3-update-the-terraform-backend-configuration">Step 3: Update the Terraform Backend Configuration</h3>
<p>Update your <code>backend.tf</code> to remove the <code>dynamodb_table</code> argument and add <code>use_lockfile = true</code>:</p>
<pre><code class="language-hcl">terraform {
  backend "s3" {
    bucket = "your-existing-bucket"
    key    = "path/to/terraform.tfstate"
    region = "us-east-1"
    encrypt = true
 
    # Add this:
    use_lockfile = true
 
    # Remove this line entirely:
    # dynamodb_table = "your-dynamodb-lock-table"
  }
}
</code></pre>
<p>Your updated <code>backend.tf</code> should look like this:</p>
<pre><code class="language-hcl">terraform {
  backend "s3" {
    bucket       = "your-existing-bucket"
    key          = "path/to/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}
</code></pre>
<h3 id="heading-step-4-reinitialize-terraform">Step 4: Reinitialize Terraform</h3>
<p>Run <code>terraform init</code> with the <code>-reconfigure</code> flag. This flag tells Terraform that the backend configuration has changed intentionally and to reinitialize without prompting you to copy state (the state is already in the same bucket):</p>
<pre><code class="language-shell">terraform init -reconfigure
</code></pre>
<p>Expected output:</p>
<pre><code class="language-plaintext">Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
 
Terraform has been successfully initialized!
</code></pre>
<p><strong>If you see an error here:</strong> The most common cause is that Object Lock wasn't successfully enabled on the bucket. Re-run the verification from Step 2 before proceeding.</p>
<h3 id="heading-step-5-verify-the-migration">Step 5: Verify the Migration</h3>
<p>Run a plan to confirm Terraform is working correctly with the new backend configuration:</p>
<pre><code class="language-shell">terraform plan
</code></pre>
<p>The plan should:</p>
<ul>
<li><p>Complete successfully</p>
</li>
<li><p>Show the same result as the plan you ran in Step 1 (no changes, or the same changes as before)</p>
</li>
<li><p>NOT mention DynamoDB anywhere in its output</p>
</li>
</ul>
<p>To confirm that locking is actually using S3 instead of DynamoDB, open a second terminal and run a plan while the first one is running. You should see the second terminal output a lock error that mentions S3, not DynamoDB:</p>
<pre><code class="language-plaintext">╷
│ Error: Error acquiring the state lock
│
│Error message: operation error S3: PutObject, https response       error StatusCode: 409,
│ RequestID: ..., api error Conflict: Object lock already exists for this key.
│
│ Lock Info:
│   ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
│   Path:      your-existing-bucket/path/to/terraform.tfstate.tflock
│   Operation: OperationTypePlan
│   Who:       user@hostname
│   Version:   1.10.0
│   Created:   2026-05-06 14:22:01 UTC
│   Info:
╵
</code></pre>
<p>The <code>Path</code> field shows <code>.tfstate.tflock</code>, a file in your S3 bucket, not a DynamoDB record. This confirms that locking is now handled entirely by S3.</p>
<img src="https://cdn.hashnode.com/uploads/covers/65a5bfab4c73b29396c0b895/e9abb703-af6e-429c-83bb-2ea2dac43a3a.png" alt="Two terminals showing concurrent terraform plan commands, the second one displays a lock error confirming S3 native locking is working" style="display: block;" width="1264" height="539" loading="lazy">

<h3 id="heading-step-6-clean-up-the-dynamodb-table">Step 6: Clean Up the DynamoDB Table</h3>
<p>Once you've confirmed the migration is working correctly and your team has run at least one successful <code>plan</code> and <code>apply</code> cycle using the new backend, you can remove the DynamoDB table.</p>
<p><strong>Wait at least 24-48 hours before deleting the DynamoDB table</strong> if you have CI/CD pipelines or multiple team members. This gives time to catch any pipeline that wasn't updated with the new backend configuration.</p>
<p>When you're ready, delete the DynamoDB table:</p>
<pre><code class="language-shell">aws dynamodb delete-table \
  --table-name your-dynamodb-lock-table
</code></pre>
<p>Confirm the deletion:</p>
<pre><code class="language-shell">aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table
</code></pre>
<p>Expected output:</p>
<pre><code class="language-plaintext">An error occurred (ResourceNotFoundException) when calling the DescribeTable operation:
Requested resource not found
</code></pre>
<p>This error confirms that the table is gone. The migration is complete.</p>
<p>If you provisioned the DynamoDB table using Terraform (which is the recommended pattern), remove the resource from your Terraform configuration and run <code>terraform apply</code> to destroy it via Terraform rather than the CLI directly. This keeps your state clean:</p>
<pre><code class="language-hcl"># Remove this entire block from your Terraform configuration:
resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
 
  attribute {
    name = "LockID"
    type = "S"
  }
}
</code></pre>
<p>After removing the block, run:</p>
<pre><code class="language-bash">terraform apply
</code></pre>
<p>Terraform will detect that the DynamoDB table resource has been removed from configuration and will destroy the table.</p>
<h2 id="heading-how-to-verify-that-locking-is-working">How to Verify That Locking Is Working</h2>
<p>After completing either the fresh setup or the migration, use this procedure to independently verify that locking is functioning correctly.</p>
<h3 id="heading-method-1-observe-the-lock-file-during-an-operation">Method 1: Observe the lock file during an operation</h3>
<p>In one terminal, start a long-running plan against a configuration with many resources:</p>
<pre><code class="language-shell">terraform plan
</code></pre>
<p>While it's running, in a second terminal, check for the lock file in S3:</p>
<pre><code class="language-shell">aws s3 ls s3://your-bucket/path/to/ | grep tflock
</code></pre>
<p>You should see a file like:</p>
<pre><code class="language-plaintext">2026-05-06 14:22:01        512 terraform.tfstate.tflock
</code></pre>
<p>After the plan completes, run the same command again. The <code>.tflock</code> file should be gone.</p>
<h3 id="heading-method-2-read-the-lock-file-contents">Method 2: Read the lock file contents</h3>
<p>While a plan is running, download and read the lock file to see its contents:</p>
<pre><code class="language-shell">aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/current.lock &amp;&amp; cat /tmp/current.lock
</code></pre>
<p>Expected output (formatted for readability):</p>
<pre><code class="language-json">{
  "ID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "Operation": "OperationTypePlan",
  "Info": "",
  "Who": "tolani@dev-machine",
  "Version": "1.10.0",
  "Created": "2026-05-06T14:22:01.123456789Z",
  "Path": "your-bucket/path/to/terraform.tfstate"
}
</code></pre>
<p>This is the same lock information that Terraform displays when a lock is held. It's now a JSON file in S3 rather than a record in DynamoDB.</p>
<h2 id="heading-how-to-handle-a-stuck-lock">How to Handle a Stuck Lock</h2>
<p>With the DynamoDB backend, resolving a stuck lock meant deleting a record from the DynamoDB table. With S3 native locking, it means deleting the <code>.tflock</code> file from S3.</p>
<p>A lock can get stuck if:</p>
<ul>
<li><p>A <code>terraform apply</code> or <code>plan</code> process was killed mid-execution</p>
</li>
<li><p>A CI/CD pipeline runner crashed during a Terraform operation</p>
</li>
<li><p>A network interruption prevented the lock release from completing</p>
</li>
</ul>
<p>Here's how you can check for a stuck lock:</p>
<pre><code class="language-shell">aws s3 ls s3://your-bucket/path/to/ | grep tflock
</code></pre>
<p>If a <code>.tflock</code> file exists and no Terraform operation is currently running, it is a stuck lock.</p>
<p>You can also read the lock to understand who held it:</p>
<pre><code class="language-shell">aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/stuck.lock &amp;&amp; cat /tmp/stuck.lock
</code></pre>
<p>This tells you who (<code>Who</code> field) was running the operation, what operation it was (<code>Operation</code> field), and when it was acquired (<code>Created</code> field).</p>
<p>And you can force-unlock using Terraform like this:</p>
<pre><code class="language-shell">terraform force-unlock LOCK-ID
</code></pre>
<p>Replace <code>LOCK-ID</code> with the <code>ID</code> value from the lock file contents. For example:</p>
<pre><code class="language-shell">terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890
</code></pre>
<p>Terraform will confirm:</p>
<pre><code class="language-plaintext">Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may be still be in use. Only 'yes' will be accepted to confirm.
 
  Enter a value: yes
 
Terraform state has been successfully unlocked!
</code></pre>
<p>An alternative is to delete the lock file directly via CLI. If <code>terraform force-unlock</code> doesn't work (for example, because you are running in a CI environment without Terraform available), delete the lock file directly:</p>
<pre><code class="language-shell">aws s3 rm s3://your-bucket/path/to/terraform.tfstate.tflock
</code></pre>
<p><strong>Only delete the lock file if you are certain no Terraform operation is currently running.</strong> Deleting a lock that is actively held by a running operation will allow a second concurrent operation to start, which is exactly the race condition locking is designed to prevent.</p>
<h2 id="heading-rollback-plan-if-something-goes-wrong">Rollback Plan: If Something Goes Wrong</h2>
<p>If you encounter problems after migrating, you can roll back to the S3 + DynamoDB setup with these steps.</p>
<p><strong>Step 1: Stop all Terraform operations</strong> in your team and CI/CD pipelines.</p>
<p><strong>Step 2: Recreate the DynamoDB table</strong> if you already deleted it:</p>
<pre><code class="language-shell">aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST
</code></pre>
<p><strong>Step 3: Revert</strong> <code>backend.tf</code> to the previous configuration:</p>
<pre><code class="language-hcl">terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"
    key            = "path/to/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # restored
    # Remove: use_lockfile = true
  }
}
</code></pre>
<p><strong>Step 4: Reinitialize:</strong></p>
<pre><code class="language-shell">terraform init -reconfigure
</code></pre>
<p><strong>Step 5: Verify:</strong></p>
<pre><code class="language-shell">terraform plan
</code></pre>
<p>The state file hasn't moved, so there's no data loss during a rollback. The only change is which locking mechanism Terraform uses.</p>
<p><strong>Note:</strong> Object Lock being enabled on the S3 bucket doesn't prevent the rollback. Object Lock and DynamoDB locking can coexist, Object Lock simply adds a capability to the bucket. Using <code>dynamodb_table</code> in your backend config tells Terraform to use DynamoDB regardless of whether Object Lock is enabled on the bucket.</p>
<h2 id="heading-security-best-practices-for-your-state-bucket">Security Best Practices for Your State Bucket</h2>
<p>Migrating to S3 native locking is a good opportunity to review the overall security configuration of your state bucket. Here are the practices every production Terraform state bucket should implement:</p>
<h3 id="heading-enable-versioning-required">Enable Versioning (Required)</h3>
<p>Versioning is a hard requirement for S3 native locking to work safely. It ensures that if a state file is accidentally overwritten or corrupted, you can restore a previous version.</p>
<pre><code class="language-shell">aws s3api put-bucket-versioning \
  --bucket your-state-bucket \
  --versioning-configuration Status=Enabled
</code></pre>
<h3 id="heading-block-all-public-access-non-negotiable">Block All Public Access (Non-Negotiable)</h3>
<p>Your state file contains resource ARNs, IP addresses, and may contain sensitive values passed through Terraform variables. It must never be publicly accessible.</p>
<pre><code class="language-shell">aws s3api put-public-access-block \
  --bucket your-state-bucket \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
</code></pre>
<h3 id="heading-enable-server-side-encryption">Enable Server-Side Encryption</h3>
<p>Always encrypt state files at rest. AES256 is the minimum. If your organization requires KMS key management:</p>
<pre><code class="language-shell">aws s3api put-bucket-encryption \
  --bucket your-state-bucket \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'
</code></pre>
<h3 id="heading-apply-least-privilege-iam-permissions">Apply Least-Privilege IAM Permissions</h3>
<p>The role or user that Terraform uses to access the state bucket should have only the permissions it needs. Here's a minimal IAM policy for S3 native locking:</p>
<pre><code class="language-json">{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TerraformStateAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-state-bucket",
        "arn:aws:s3:::your-state-bucket/*"
      ]
    },
    {
      "Sid": "TerraformStateLocking",
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectLegalHold",
        "s3:PutObjectLegalHold",
        "s3:GetObjectRetention",
        "s3:PutObjectRetention"
      ],
      "Resource": "arn:aws:s3:::your-state-bucket/*.tflock"
    }
  ]
}
</code></pre>
<p>Notice what is absent: there are no DynamoDB permissions. This is a cleaner, smaller permission set than the old approach required.</p>
<h3 id="heading-enable-access-logging">Enable Access Logging</h3>
<p>Log all access to your state bucket in CloudTrail or S3 server access logs. This gives you an audit trail of every time state was read, written, or locked:</p>
<pre><code class="language-shell">aws s3api put-bucket-logging \
  --bucket your-state-bucket \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "your-logging-bucket",
      "TargetPrefix": "terraform-state-access/"
    }
  }'
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>AWS S3 native state locking removes the need for a DynamoDB table from your Terraform backend setup. The result is simpler infrastructure, a smaller IAM permission surface, and one fewer service to provision, monitor, and pay for across every environment your team manages.</p>
<p>Here's a summary of what you accomplished:</p>
<ul>
<li><p>Understood what state locking is and why it's required for safe Terraform operations</p>
</li>
<li><p>Compared S3 native locking to the existing S3 + DynamoDB approach</p>
</li>
<li><p>Set up a fresh Terraform backend using S3 native locking with correct bucket configuration</p>
</li>
<li><p>Migrated an existing backend from S3 + DynamoDB to S3 native locking safely</p>
</li>
<li><p>Learned how to verify locking, handle stuck locks, and roll back if needed</p>
</li>
<li><p>Applied security best practices to the state bucket</p>
</li>
</ul>
<p>This pattern – using S3 native locking – is the recommended approach for all new Terraform projects on AWS going forward. If you're managing a large estate with multiple Terraform backends, consider automating the migration using a script or Terraform module that applies the pattern across all your state buckets.</p>
<p><em>If you are building or optimizing cloud infrastructure for a startup and want a complete reference for production-ready Terraform modules, CI/CD pipeline patterns, and infrastructure runbooks, check out</em> <a href="https://coachli.co/tolani-akintayo/PR-H4oQS">The Startup DevOps Field Guide</a><em>. It covers the full lifecycle of AWS infrastructure from initial setup to production reliability.</em></p>
<h2 id="heading-references">References</h2>
<ul>
<li><p><a href="https://developer.hashicorp.com/terraform/language/backend/s3#use_lockfile">HashiCorp - S3 Backend Configuration: use_lockfile</a></p>
</li>
<li><p><a href="https://github.com/hashicorp/terraform/releases/tag/v1.10.0">HashiCorp: Terraform 1.10 Release Notes</a></p>
</li>
<li><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html">AWS Docs: S3 Object Lock Overview</a></p>
</li>
<li><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectLockConfiguration.html">AWS Docs: PutObjectLockConfiguration API</a></p>
</li>
<li><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html">AWS Docs: S3 Conditional Writes</a></p>
</li>
<li><p><a href="https://developer.hashicorp.com/terraform/language/state/locking">HashiCorp: Backend State Locking</a></p>
</li>
<li><p><a href="https://developer.hashicorp.com/terraform/cli/commands/force-unlock">HashiCorp: terraform force-unlock Command</a></p>
</li>
<li><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html">AWS Docs: Enabling S3 Versioning</a></p>
</li>
<li><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html">AWS Docs: S3 Server-Side Encryption</a></p>
</li>
</ul>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ The Complete SOC 2 Type II Implementation Handbook for Engineers: A Month-by-Month Roadmap with Real Commands ]]>
                </title>
                <description>
                    <![CDATA[ If your team is preparing for a SOC 2 Type II review, this handbook is for you. It's a self-contained guide to the exact 90-day timeline, 14 critical controls, and evidence collection infrastructure t ]]>
                </description>
                <link>https://www.freecodecamp.org/news/the-complete-soc-2-type-ii-implementation-guide-for-engineers/</link>
                <guid isPermaLink="false">69fa364da386d7f121c468af</guid>
                
                    <category>
                        <![CDATA[ SOC ]]>
                    </category>
                
                    <category>
                        <![CDATA[ compliance  ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                    <category>
                        <![CDATA[ cloud security ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Ayobami Adejumo ]]>
                </dc:creator>
                <pubDate>Tue, 05 May 2026 18:26:21 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/83d83215-5d73-49f6-a745-d9c6cd0c33f8.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>If your team is preparing for a SOC 2 Type II review, this handbook is for you. It's a self-contained guide to the exact 90-day timeline, 14 critical controls, and evidence collection infrastructure that auditors actually check.</p>
<p>Everyone publishes the controls list. But nobody publishes the week-by-week engineering calendar you'll need to follow to make sure your ducks are in a row.</p>
<p>Here is the exact 90-day timeline — including the mistakes that add 60 days (and how to avoid them).</p>
<h2 id="heading-table-of-contents">Table of Contents</h2>
<ol>
<li><p><a href="#heading-what-youll-learn">What You'll Learn</a></p>
</li>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-weeks-1-2-the-scope-decision">Weeks 1–2: The Scope Decision</a></p>
</li>
<li><p><a href="#heading-weeks-3-6-the-14-controls-that-must-be-active-on-day-1">Weeks 3–6: The 14 Controls That Must Be Active on Day 1</a></p>
</li>
<li><p><a href="#heading-weeks-7-10-the-evidence-collection-infrastructure">Weeks 7–10: The Evidence Collection Infrastructure</a></p>
</li>
<li><p><a href="#heading-weeks-11-14-auditor-selection-and-readiness-assessment">Weeks 11–14: Auditor Selection and Readiness Assessment</a></p>
</li>
<li><p><a href="#heading-weeks-15-18-the-observation-period">Weeks 15–18: The Observation Period</a></p>
</li>
<li><p><a href="#heading-the-90-day-soc2-timeline-at-a-glance">The 90-Day SOC2 Timeline at a Glance</a></p>
</li>
<li><p><a href="#heading-whats-next">What's Next</a></p>
</li>
<li><p><a href="#heading-resources">Resources</a></p>
</li>
</ol>
<h2 id="heading-what-youll-learn">What You'll Learn</h2>
<p>By the end of this guide, you'll know:</p>
<ul>
<li><p>How to scope your SOC2 boundary correctly — the decision that determines everything else</p>
</li>
<li><p>The 14 controls that must be active on day 1 of your observation period</p>
</li>
<li><p>How to build evidence collection infrastructure that runs automatically</p>
</li>
<li><p>How to choose an auditor and run a readiness assessment</p>
</li>
<li><p>What happens during the observation period and how to close gaps without restarting the clock</p>
</li>
</ul>
<p>Let's dive in.</p>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>Before following along, you should have:</p>
<p><strong>Knowledge:</strong></p>
<ul>
<li><p>Basic understanding of AWS services (EC2, RDS, S3, IAM, VPC)</p>
</li>
<li><p>Familiarity with Terraform or another infrastructure as code tool</p>
</li>
<li><p>Comfort reading GitHub Actions YAML workflows</p>
</li>
<li><p>A general understanding of what SOC2 is — if you are starting from scratch, read the <a href="https://www.aicpa-cima.com/resources/landing/system-and-organization-controls-soc-suite-of-services">AICPA's SOC2 overview</a> first</p>
</li>
</ul>
<p><strong>Tools and access:</strong></p>
<ul>
<li><p>An AWS account with administrator access</p>
</li>
<li><p>A GitHub organisation with admin rights</p>
</li>
<li><p>Terraform installed (v1.0 or later)</p>
</li>
<li><p>Python 3.8 or later (for the evidence collector Lambda)</p>
</li>
<li><p>A compliance automation platform — <a href="https://www.vanta.com/">Vanta</a> or <a href="https://drata.com/">Drata</a> — connected to your AWS account and GitHub organisation</p>
</li>
</ul>
<p><strong>Estimated time:</strong> 90 days end-to-end, with active engineering work of approximately 8–12 hours per week in the first six weeks, tapering to 2–4 hours per week during the observation period.</p>
<h2 id="heading-weeks-12-the-scope-decision-what-is-in-and-out-of-your-soc2-boundary">Weeks 1–2: The Scope Decision — What Is In and Out of Your SOC2 Boundary</h2>
<h3 id="heading-what-most-teams-get-wrong">What Most Teams Get Wrong</h3>
<p>Most teams scope their SOC2 boundary too broadly. They include every AWS account, every service, every environment. This is a mistake — and here is exactly why.</p>
<p>A broader scope means more controls to implement, more evidence to collect, and more systems the auditor will examine.</p>
<p>Every system inside your boundary must satisfy all 14 controls. Including your development sandbox means your engineers' experimental environments must have GuardDuty enabled, CloudTrail logging, and branch-protected deployments. That adds weeks of work and months of evidence collection for systems that pose no risk to your customers.</p>
<p>A correctly bounded scope means you include only the systems that store, process, or transmit customer data — and you prove that everything else cannot reach those systems.</p>
<p><strong>Bad scope (over-inclusive):</strong></p>
<pre><code class="language-plaintext">Entire AWS Organization
├── Production (in scope)
├── Staging (in scope)
├── Development (in scope)
├── Sandbox (in scope)
└── CI/CD (in scope)
</code></pre>
<p><strong>Good scope (correctly bounded):</strong></p>
<pre><code class="language-plaintext">SOC2 Boundary
├── Production AWS Account (in scope)
├── Production EKS Cluster (in scope)
├── Production RDS (in scope)
└── Everything else (OUT of scope — proven by network segmentation)
</code></pre>
<p>The correctly bounded scope works because it draws the tightest defensible line around the systems that actually handle customer data. Everything outside that line is excluded — not by assumption, but by technical controls that prevent those systems from reaching anything inside the boundary.</p>
<h3 id="heading-the-scope-decision-framework">The Scope Decision Framework</h3>
<p>For every system in your infrastructure, ask these four questions:</p>
<table>
<thead>
<tr>
<th>Question</th>
<th>If YES</th>
<th>If NO</th>
</tr>
</thead>
<tbody><tr>
<td>Does this system store, process, or transmit customer data?</td>
<td>✅ In scope</td>
<td>❌ Out of scope</td>
</tr>
<tr>
<td>Does this system affect the availability of customer-facing services?</td>
<td>✅ In scope</td>
<td>❌ Out of scope</td>
</tr>
<tr>
<td>Does this system have access to production credentials?</td>
<td>✅ In scope</td>
<td>❌ Out of scope</td>
</tr>
<tr>
<td>Can a compromise of this system lead to a customer data breach?</td>
<td>✅ In scope</td>
<td>❌ Out of scope</td>
</tr>
</tbody></table>
<p>Any system where the answer to even one question is yes belongs inside your boundary.</p>
<h3 id="heading-network-segmentation-the-technical-proof-that-your-boundary-holds">Network Segmentation — The Technical Proof That Your Boundary Holds</h3>
<p>Network segmentation is the practice of dividing your infrastructure into isolated zones so that systems in one zone can't communicate with systems in another unless you explicitly allow it.</p>
<p>In the context of SOC2, it's the technical control that proves your out-of-scope systems genuinely can't reach your in-scope systems — not just by policy, but by infrastructure enforcement.</p>
<p>Without network segmentation, the SOC2 auditor can't trust that your boundary is real. A developer in your sandbox environment who can query your production database means the sandbox is effectively in scope, regardless of what your diagram says.</p>
<p>Here's the Terraform that implements network segmentation between your production and non-production environments. The network access control list (NACL) blocks all inbound traffic from the broader private IP range (10.0.0.0/8) into your in-scope production VPC, while the explicit <code>aws_vpc_peering_connection</code> comment documents the deliberate decision not to peer environments:</p>
<pre><code class="language-hcl"># This account has NO VPC peering to non-production environments.
# The absence of peering is itself the segmentation control.
# Do NOT add peering connections to this account without SOC2 scope review.

resource "aws_network_acl" "deny_non_production" {
  vpc_id = aws_vpc.production.id

  # Block all inbound traffic from non-production IP ranges
  ingress {
    rule_no    = 100
    action     = "deny"
    from_port  = 0
    to_port    = 0
    protocol   = "-1"
    cidr_block = "10.0.0.0/8"
  }

  # Allow legitimate inbound traffic (HTTPS from internet)
  ingress {
    rule_no    = 200
    action     = "allow"
    from_port  = 443
    to_port    = 443
    protocol   = "tcp"
    cidr_block = "0.0.0.0/0"
  }

  # Allow all outbound (tighten this per your architecture)
  egress {
    rule_no    = 100
    action     = "allow"
    from_port  = 0
    to_port    = 0
    protocol   = "-1"
    cidr_block = "0.0.0.0/0"
  }

  tags = {
    Name        = "production-nacl"
    Environment = "production"
    Purpose     = "SOC2 network segmentation"
  }
}
</code></pre>
<p>Verify the segmentation with this command after applying the Terraform:</p>
<pre><code class="language-bash"># Confirm no VPC peering connections exist from production to non-production
aws ec2 describe-vpc-peering-connections \
  --filters Name=status-code,Values=active \
  --query 'VpcPeeringConnections[*].{ID:VpcPeeringConnectionId,Requester:RequesterVpcInfo.VpcId,Accepter:AccepterVpcInfo.VpcId}' \
  --output table
</code></pre>
<h3 id="heading-the-deliverable-your-soc2-boundary-diagram">The Deliverable: Your SOC2 Boundary Diagram</h3>
<p>At the end of weeks 1–2, you need a boundary diagram — a visual document that shows every in-scope system, every out-of-scope system, and the segmentation controls between them.</p>
<p>Here is what the diagram should contain:</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d00d5be466e2b76263a583/29dfe0c8-f455-44af-8562-8d088f8a111a.png" alt="29dfe0c8-f455-44af-8562-8d088f8a111a" style="display: block;" width="611" height="686" loading="lazy">

<p>Include every AWS service, every data flow arrow, and a label on the segmentation control. This diagram becomes your primary scope evidence and is typically the first thing an auditor asks for.</p>
<h2 id="heading-weeks-36-the-14-controls-that-must-be-active-on-day-1">Weeks 3–6: The 14 Controls That Must Be Active on Day 1</h2>
<p>These 14 controls must be implemented and actively collecting evidence from day 1 of your observation period. If you add any of them late, the observation period clock for that control restarts from the implementation date — not from day 1 of the audit period.</p>
<p>Think of the observation period as a surveillance camera recording your infrastructure. The auditor watches the footage later. If the camera was not on when a specific event occurred, that event has no record — and the SOC2 control for it has a gap.</p>
<h3 id="heading-control-1-mfa-enforcement-cc66">Control 1: MFA Enforcement (CC6.6)</h3>
<p>Multi-Factor Authentication (MFA) requires a user to verify their identity using two independent factors — something they know (a password) and something they have (a phone or hardware key). Without MFA, a stolen password is sufficient to access your production systems.</p>
<p>SOC2 CC6.6 requires that access to systems is restricted to authorized users. MFA is the technical control that makes "authorized" meaningful. Without it, any password compromise is a production access event.</p>
<p>To implement MFA, you can use AWS IAM Identity Center (formerly SSO) connected to your identity provider (Okta, Google Workspace, or Azure AD). MFA is then enforced at the identity provider level — any user without MFA enrolled can't authenticate, regardless of which AWS service they're trying to reach.</p>
<pre><code class="language-hcl"># IAM Identity Center configuration — MFA is enforced at the IdP level.
# No IAM user has direct console or CLI access.
# All access goes through SSO sessions (8-hour expiry by default).

resource "aws_ssoadmin_instance_access_control_attributes" "mfa" {
  instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]

  attribute {
    key = "email"
    value {
      source = ["$${path:email}"]
    }
  }
}
</code></pre>
<p>You can verify that no IAM users retain direct console access (which would bypass MFA):</p>
<pre><code class="language-bash"># Any user listed here has direct console access bypassing SSO — investigate immediately
aws iam list-users \
  --query 'Users[?PasswordLastUsed!=`null`].[UserName,PasswordLastUsed]' \
  --output table
</code></pre>
<h3 id="heading-control-2-infrastructure-as-code-cc81">Control 2: Infrastructure as Code (CC8.1)</h3>
<p>Infrastructure as Code (IaC) means defining your cloud infrastructure in version-controlled code files (Terraform, Pulumi, or AWS CDK) rather than creating resources manually through the AWS console. Every infrastructure change is proposed in a pull request, reviewed by a colleague, and applied through an automated pipeline.</p>
<p>SOC2 CC8.1 covers change management — the requirement that every change to your production environment is documented, reviewed, and approved. Manual console changes produce no audit trail. If an engineer opens the AWS console and creates a security group without going through Terraform, that change is invisible to your SOC2 auditor. IaC makes every change reviewable and traceable.</p>
<p>Now let's see how to implement IaC here. This GitHub Actions workflow applies Terraform only from the main branch, after a pull request has been reviewed and approved. The workflow creates an immutable record of every infrastructure change:</p>
<pre><code class="language-yaml"># .github/workflows/terraform-apply.yml
name: Terraform Apply (Production)
on:
  push:
    branches: [main]
    paths: ['terraform/**']

permissions:
  id-token: write   # Required for AWS OIDC authentication
  contents: read

jobs:
  apply:
    name: Apply Infrastructure Changes
    runs-on: ubuntu-latest
    environment: production  # Requires manual approval for production

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Configure AWS credentials (OIDC — no long-lived keys)
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/terraform-apply
          aws-region: us-east-1

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: "1.6.0"

      - name: Terraform Plan
        run: |
          terraform init
          terraform plan -out=tfplan -input=false

      - name: Terraform Apply
        run: terraform apply -input=false tfplan
</code></pre>
<p>SOC2 evidence this produces: A GitHub Actions run log for every infrastructure change, showing who triggered it (the pull request author), when it was applied, and what changed.</p>
<h3 id="heading-control-3-cloudtrail-enabled-cc71">Control 3: CloudTrail Enabled (CC7.1)</h3>
<p>AWS CloudTrail is a service that records every API call made in your AWS account — who called it, when, from which IP address, and whether it succeeded. Think of it as the complete audit log of everything that has ever happened in your AWS environment.</p>
<p>SOC2 CC7.1 requires monitoring for security events. CloudTrail is the foundational logging layer — without it, you can't detect unauthorized access, investigate incidents, or prove to an auditor that your controls were operating as intended. An auditor who can't see historical AWS API activity can't verify that your access controls were enforced during the observation period.</p>
<p>To implement it, you'll want to enable multi-region CloudTrail so that activity in every AWS region is captured, including global services like IAM. You can ship logs to an S3 bucket with Object Lock enabled (Control 3 in the evidence collection section covers this) so logs can't be modified or deleted:</p>
<pre><code class="language-bash"># Enable CloudTrail with log file validation and multi-region coverage
aws cloudtrail create-trail \
  --name production-audit-trail \
  --s3-bucket-name your-cloudtrail-logs-bucket \
  --is-multi-region-trail \
  --enable-log-file-validation \
  --include-global-service-events

# Start the trail (creation alone does not start logging)
aws cloudtrail start-logging --name production-audit-trail

# Verify the trail is active and logging
aws cloudtrail get-trail-status --name production-audit-trail \
  --query '{IsLogging:IsLogging,LatestDeliveryTime:LatestDeliveryTime}'
</code></pre>
<h3 id="heading-control-4-guardduty-enabled-cc72">Control 4: GuardDuty Enabled (CC7.2)</h3>
<p>AWS GuardDuty is a threat detection service that analyses your CloudTrail logs, VPC Flow Logs, and DNS logs. It uses machine learning to identify suspicious behaviour — things like an EC2 instance communicating with a known malware server, an IAM user logging in from an unusual country, or unusual API call patterns that indicate credential theft.</p>
<p>SOC2 CC7.2 requires the use of detection tools to identify potential security events. GuardDuty is the monitoring layer that tells you when something anomalous is happening, not just what happened after the fact. Without it, you would only discover a compromise when the damage is done.</p>
<p>Here's the implementation:</p>
<pre><code class="language-bash"># Enable GuardDuty — findings published every 15 minutes for active threats
aws guardduty create-detector \
  --enable \
  --finding-publishing-frequency FIFTEEN_MINUTES

# Verify GuardDuty is active
aws guardduty list-detectors --query 'DetectorIds' --output table
</code></pre>
<p>You can set up an EventBridge rule to route CRITICAL and HIGH severity GuardDuty findings to your incident response channel immediately. A finding sitting unreviewed for 90 days is a qualified SOC2 finding.</p>
<h3 id="heading-control-5-vpc-flow-logs-cc61">Control 5: VPC Flow Logs (CC6.1)</h3>
<p>VPC Flow Logs capture information about the IP traffic flowing through your Virtual Private Cloud — every accepted and rejected connection, including source IP, destination IP, port, protocol, and whether the traffic was allowed or denied. They are the network-level audit trail that CloudTrail doesn't provide.</p>
<p>SOC2 CC6.1 requires logical access controls and monitoring. VPC Flow Logs let you verify that your network segmentation is actually working (traffic you denied is showing as rejected in the logs), detect unexpected communication between services, and investigate security events at the network layer.</p>
<pre><code class="language-bash"># Create an IAM role for VPC Flow Logs to deliver to CloudWatch
aws iam create-role \
  --role-name vpc-flow-logs-role \
  --assume-role-policy-document '{
    "Version":"2012-10-17",
    "Statement":[{
      "Effect":"Allow",
      "Principal":{"Service":"vpc-flow-logs.amazonaws.com"},
      "Action":"sts:AssumeRole"
    }]
  }'

# Enable VPC Flow Logs for all traffic (ACCEPT and REJECT)
aws ec2 create-flow-logs \
  --resource-ids vpc-YOUR_PRODUCTION_VPC_ID \
  --resource-type VPC \
  --traffic-type ALL \
  --log-group-name /aws/vpc/flow-logs/production \
  --deliver-log-permission-arn arn:aws:iam::YOUR_ACCOUNT_ID:role/vpc-flow-logs-role

# Verify flow logs are active
aws ec2 describe-flow-logs \
  --filter Name=resource-id,Values=vpc-YOUR_PRODUCTION_VPC_ID \
  --query 'FlowLogs[*].{Status:FlowLogStatus,LogGroup:LogGroupName}'
</code></pre>
<h3 id="heading-control-6-secrets-manager-cc67">Control 6: Secrets Manager (CC6.7)</h3>
<p>Secrets management means storing credentials (database passwords, API keys, certificates, and other sensitive configuration values) in a dedicated, access-controlled service (like AWS Secrets Manager or HashiCorp Vault) rather than in <code>.env</code> files, GitHub repository secrets, or hardcoded in application code.</p>
<p>SOC2 CC6.7 requires protecting sensitive system components from unauthorized access. A secret stored in an <code>.env</code> file committed to a repository is accessible to every developer with repo access, every CI/CD runner, and every engineer who has ever cloned the repo — including those who have since left the company.</p>
<p>A Secrets Manager provides centralised storage, access logging, automatic rotation, and fine-grained IAM permissions so only specific services can retrieve specific secrets.</p>
<p>Let's look at the implementation — storing and rotating a secret:</p>
<pre><code class="language-bash"># Store a database credential with automatic 90-day rotation
aws secretsmanager create-secret \
  --name production/postgresql/credentials \
  --description "Production PostgreSQL credentials — rotated every 90 days" \
  --secret-string '{
    "username": "app_user",
    "password": "REPLACE_WITH_STRONG_PASSWORD",
    "host": "your-rds-endpoint.us-east-1.rds.amazonaws.com",
    "port": 5432,
    "dbname": "production"
  }'

# Enable automatic rotation every 90 days
aws secretsmanager rotate-secret \
  --secret-id production/postgresql/credentials \
  --rotation-rules AutomaticallyAfterDays=90
</code></pre>
<p>How your application retrieves the secret at runtime (no hardcoded credentials):</p>
<pre><code class="language-python"># Good: secret retrieved at runtime from Secrets Manager
import boto3
import json

def get_db_credentials():
    client = boto3.client('secretsmanager', region_name='us-east-1')
    response = client.get_secret_value(SecretId='production/postgresql/credentials')
    return json.loads(response['SecretString'])

# Bad: secret hardcoded in application code or .env file
DB_PASSWORD = "my_database_password_123"  # Never do this
</code></pre>
<p>The access log in CloudTrail records every time a secret is retrieved, by which IAM role, at what time. That log is your SOC2 evidence that secrets access is controlled and auditable.</p>
<h3 id="heading-control-7-ebs-encryption-cc61">Control 7: EBS Encryption (CC6.1)</h3>
<p>EBS (Elastic Block Store) encryption ensures that the persistent disks attached to your EC2 instances and used by your RDS databases are encrypted at rest using AES-256. If an AWS employee or an attacker gained physical access to the storage hardware, the data would be unreadable without the encryption key.</p>
<p>SOC2 CC6.1 requires protecting information assets from unauthorised access. Encryption at rest is the control that protects data in the event of physical storage compromise or an improperly decommissioned disk. Enabling it account-wide means every new EBS volume is encrypted automatically, including RDS storage, EKS node volumes, and EC2 instance root volumes.</p>
<pre><code class="language-bash"># Enable EBS encryption by default for all new volumes in this region
aws ec2 enable-ebs-encryption-by-default

# Verify it is enabled
aws ec2 get-ebs-encryption-by-default \
  --query 'EbsEncryptionByDefault'
# Expected output: true

# Check existing volumes — any showing false need to be migrated
aws ec2 describe-volumes \
  --query 'Volumes[?Encrypted==`false`].[VolumeId,Size,VolumeType]' \
  --output table
</code></pre>
<p>Any existing unencrypted volumes must be snapshot-and-replaced. The process: create a snapshot of the unencrypted volume, create a new encrypted volume from the snapshot, and swap it into the instance.</p>
<h3 id="heading-control-8-s3-block-public-access-cc61">Control 8: S3 Block Public Access (CC6.1)</h3>
<p>Amazon S3 buckets can be configured to allow public access — meaning anyone on the internet can read their contents without authentication. Block Public Access is an account-level and bucket-level setting that prevents any bucket from being made public, regardless of the bucket's own policy.</p>
<p>A misconfigured S3 bucket is one of the most common causes of data breaches in cloud environments. Block Public Access at the account level means a developer can't accidentally expose a bucket containing customer data, even if they set the wrong bucket policy. It's a guardrail, not just a policy.</p>
<pre><code class="language-bash"># Block public access at the AWS account level — applies to all buckets
aws s3control put-public-access-block \
  --account-id YOUR_ACCOUNT_ID \
  --public-access-block-configuration \
    BlockPublicAcls=true,\
    IgnorePublicAcls=true,\
    BlockPublicPolicy=true,\
    RestrictPublicBuckets=true

# Verify account-level setting is active
aws s3control get-public-access-block \
  --account-id YOUR_ACCOUNT_ID

# Scan for any buckets that have public access enabled (should be zero)
aws s3api list-buckets --query 'Buckets[*].Name' --output text | \
  tr '\t' '\n' | while read bucket; do
    result=\((aws s3api get-public-access-block --bucket "\)bucket" 2&gt;/dev/null)
    if echo "$result" | grep -q '"BlockPublicAcls": false'; then
      echo "WARNING: $bucket has public access not fully blocked"
    fi
  done
</code></pre>
<h3 id="heading-control-9-branch-protection-cc81">Control 9: Branch Protection (CC8.1)</h3>
<p>Branch protection is a GitHub setting that prevents engineers from pushing code directly to your main branch without going through a pull request that has been reviewed and approved by at least one other team member. It also requires your CI pipeline to pass before any code can be merged.</p>
<p>SOC2 CC8.1 requires change management — the requirement that every change to production systems is documented, reviewed, and approved. Without branch protection, an engineer can push directly to main, which deploys directly to production through your CI/CD pipeline, with no review and no audit trail. Branch protection is the technical enforcement of your change management policy.</p>
<p>The critical setting that most teams miss: the "Do not allow bypassing the above settings" option must be enabled. Without it, administrators can bypass branch protection — and a SOC2 auditor will flag this as a gap because it means your change management control can be circumvented.</p>
<pre><code class="language-yaml"># .github/settings.yml — enforces branch protection via code
# Requires the settings GitHub App: https://github.com/apps/settings

branches:
  - name: main
    protection:
      required_pull_request_reviews:
        required_approving_review_count: 1
        dismiss_stale_reviews: true
        require_code_owner_reviews: false
      required_status_checks:
        strict: true
        contexts:
          - "CI / test"
          - "Security / trivy-scan"
      enforce_admins: true         # Admins cannot bypass — this is critical
      restrictions: null           # No push restriction beyond the above
      allow_force_pushes: false
      allow_deletions: false
</code></pre>
<p>Here's how you can verify that branch protection is enforced and admins can't bypass it:</p>
<pre><code class="language-bash"># Returns the branch protection rules including enforce_admins status
curl -H "Authorization: token YOUR_GITHUB_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/YOUR_REPO/branches/main/protection \
  | jq '{enforce_admins: .enforce_admins.enabled, required_reviews: .required_pull_request_reviews.required_approving_review_count}'
</code></pre>
<h3 id="heading-control-10-container-image-scanning-cc74">Control 10: Container Image Scanning (CC7.4)</h3>
<p>Container image scanning analyses your Docker images before deployment to identify known security vulnerabilities (CVEs) in the operating system packages and application dependencies they contain.</p>
<p>Trivy is an open-source scanner that checks the base image (Ubuntu, Alpine, and so on), all installed OS packages, and language-specific dependencies (npm, pip, Go modules) against the National Vulnerability Database.</p>
<p>SOC2 CC7.4 requires monitoring and identifying vulnerabilities. Every container you deploy contains a base image with OS packages — and those packages regularly receive CVE disclosures. A critical CVE left unpatched for 90 days in a production container is a SOC2 finding. Automated scanning in CI means every image is checked before it can deploy.</p>
<pre><code class="language-yaml"># .github/workflows/security-scan.yml
name: Security Scan
on: [push, pull_request]

jobs:
  trivy-scan:
    name: Container Vulnerability Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build container image
        run: docker build -t app:${{ github.sha }} .

      - name: Scan image for vulnerabilities
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: app:${{ github.sha }}
          format: sarif
          output: trivy-results.sarif
          severity: CRITICAL,HIGH
          exit-code: 1          # Fail the pipeline on CRITICAL or HIGH findings

      - name: Upload results to GitHub Security tab
        uses: github/codeql-action/upload-sarif@v2
        if: always()            # Upload even if scan found issues
        with:
          sarif_file: trivy-results.sarif
</code></pre>
<p>The scanner looks for:</p>
<ul>
<li><p>CVEs in base image OS packages (for example, a critical OpenSSL vulnerability in your Ubuntu base)</p>
</li>
<li><p>Vulnerable versions of application dependencies (a known RCE in an npm package your app uses)</p>
</li>
<li><p>Misconfigurations in the Dockerfile itself (running as root, using <code>latest</code> tags)</p>
</li>
</ul>
<p>Results appear in the GitHub Security tab for your repository, giving you a historical record of every scan — which is your SOC2 evidence.</p>
<h3 id="heading-control-11-incident-response-plan-cc92">Control 11: Incident Response Plan (CC9.2)</h3>
<p>An incident response plan is a written, tested procedure that defines exactly what your team does when a security event occurs — from the moment an alert fires through to customer notification and post-incident review.</p>
<p>SOC2 CC9.2 requires that you have a documented process for responding to security events and that you've tested it. The auditor will ask for the written runbook and evidence that a tabletop exercise (a simulated incident walkthrough) has been conducted within the observation period.</p>
<p>Your incident response runbook must include:</p>
<ol>
<li><p><strong>Severity classification:</strong> Definitions of P1 (production down, customer data at risk), P2 (degraded service, potential risk), and P3 (minor issue, no customer impact) — and the response SLA for each.</p>
</li>
<li><p><strong>Escalation path:</strong> Exactly who gets paged at each severity level, with contact details. Not "the on-call engineer" — specific names and a backup if the first person doesn't respond within 10 minutes.</p>
</li>
<li><p><strong>First 15 minutes:</strong> The specific steps to take immediately — isolate the affected system, assess the scope, notify the incident channel, begin the timeline log.</p>
</li>
<li><p><strong>Communication templates:</strong> Pre-written Slack messages, customer email templates, and regulatory notification templates (GDPR requires notification within 72 hours, HIPAA within 60 days).</p>
</li>
<li><p><strong>Post-incident review:</strong> The blameless postmortem process, the <a href="https://www.freecodecamp.org/news/from-symptoms-to-root-cause-how-to-use-the-5-whys-technique/">5-why</a> root cause analysis template, and the action item tracking process.</p>
</li>
</ol>
<p>Conduct a tabletop exercise at least once during your observation period: gather your engineering team for 45 minutes, simulate a realistic scenario (for example, "an AWS access key was committed to a public GitHub repo"), and walk through the runbook together. Document the meeting date, attendees, scenario, gaps found, and remediation actions. This document is your evidence.</p>
<h3 id="heading-control-12-access-reviews-cc63">Control 12: Access Reviews (CC6.3)</h3>
<p>An access review is a quarterly audit of who has access to what in your production systems — AWS accounts, GitHub repositories, production databases, and every SaaS tool that touches customer data. You verify that every person on the list still works at the company and still needs the access their role grants them.</p>
<p>SOC2 CC6.3 requires that access is revoked when it's no longer needed. Former employees who retain access to production AWS accounts represent a genuine security risk and a definitive SOC2 finding.</p>
<p>In every access review I've conducted, at least 3–5 former employees or contractors still had active access they should not.</p>
<p>The quarterly access review checklist:</p>
<pre><code class="language-bash"># 1. IAM users — list all with their last login date
aws iam generate-credential-report
aws iam get-credential-report --output text --query Content \
  | base64 --decode | cut -d',' -f1,5 | column -t -s ','

# 2. IAM roles — find roles that have not been used in 90+ days
aws iam get-account-authorization-details \
  --query 'RoleDetailList[*].{Role:RoleName,LastUsed:RoleLastUsed.LastUsedDate}' \
  --output table

# 3. Verify AWS SSO user list matches your current employee list
aws identitystore list-users \
  --identity-store-id YOUR_IDENTITY_STORE_ID \
  --query 'Users[*].{Name:DisplayName,Email:Emails[0].Value}' \
  --output table
</code></pre>
<p>Cross-reference the output against your current employee list in your HR system. Document every change made — access removed, permissions reduced, accounts disabled. The documented changes are the evidence that the review was conducted meaningfully, not just as a checkbox exercise.</p>
<h3 id="heading-control-13-backup-verification-cc95">Control 13: Backup Verification (CC9.5)</h3>
<p>Backup verification is the process of actually restoring your backups to confirm they work — not just confirming that backups are being created. A backup that has never been tested doesn't exist from a recovery perspective.</p>
<p>SOC2 CC9.5 requires that recovery procedures are tested. If your production database is corrupted and you discover for the first time during the incident that your automated RDS snapshots can't be restored, you have both a disaster recovery failure and a SOC2 finding.</p>
<p>How to test your RDS backup:</p>
<pre><code class="language-bash"># Step 1: Find your most recent production snapshot
aws rds describe-db-snapshots \
  --db-instance-identifier your-production-db \
  --query 'sort_by(DBSnapshots, &amp;SnapshotCreateTime)[-1].DBSnapshotIdentifier' \
  --output text

# Step 2: Restore the snapshot to a test instance
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier backup-verification-test \
  --db-snapshot-identifier YOUR_SNAPSHOT_ID \
  --db-instance-class db.t3.medium \
  --no-publicly-accessible \
  --tags Key=Purpose,Value=backup-verification Key=Environment,Value=test

# Step 3: Wait for the restore to complete (typically 5–15 minutes)
aws rds wait db-instance-available \
  --db-instance-identifier backup-verification-test

# Step 4: Connect and verify data integrity (spot check key tables)
# Run this against the restored instance
psql -h RESTORED_INSTANCE_ENDPOINT -U your_user -d your_database \
  -c "SELECT COUNT(*) FROM users; SELECT MAX(created_at) FROM orders;"

# Step 5: Document the test result and delete the test instance
aws rds delete-db-instance \
  --db-instance-identifier backup-verification-test \
  --skip-final-snapshot
</code></pre>
<p>Document the test date, the snapshot used, the restore time, the data verification query results, and who conducted the test. Run this quarterly at minimum. This documentation is your SOC2 evidence for CC9.5.</p>
<h3 id="heading-control-14-change-management-log-cc81">Control 14: Change Management Log (CC8.1)</h3>
<p>A change management log is the auditable record of every change made to your production environment — what changed, who approved it, and when it was applied.</p>
<p>SOC2 CC8.1 requires that changes to your production environment are authorized and documented. With IaC and GitOps in place, you already have two separate sources of immutable change history that together satisfy this control.</p>
<p><strong>GitHub Pull Request history</strong> provides the record of every code and infrastructure change: who opened the PR, who reviewed and approved it, what the CI status was, and when it was merged. This is your change management log for application and infrastructure changes.</p>
<p><strong>ArgoCD sync history</strong> provides the record of every deployment to your Kubernetes cluster: which application was synced, from which Git commit, at what time, and whether the sync succeeded.</p>
<p>To export the ArgoCD sync history as evidence:</p>
<pre><code class="language-bash"># Export ArgoCD application sync history as JSON evidence
argocd app history YOUR_APP_NAME --output json &gt; argocd-sync-history-$(date +%Y%m).json

# Upload to your SOC2 evidence bucket
aws s3 cp argocd-sync-history-$(date +%Y%m).json \
  s3://your-soc2-evidence-bucket/change-management/$(date +%Y/%m)/

# For each deployment, the evidence contains:
# - App name, deployed revision (Git commit SHA)
# - Deployment timestamp
# - Initiating user or automated sync
# - Success/failure status
</code></pre>
<p>Together, the GitHub PR history and the ArgoCD sync history give the auditor a complete, tamper-evident record of every change to your production environment during the observation period.</p>
<h2 id="heading-weeks-710-the-evidence-collection-infrastructure">Weeks 7–10: The Evidence Collection Infrastructure</h2>
<p>Evidence is the difference between passing and failing SOC2.</p>
<p>You might be wondering: what exactly is evidence? In SOC2 terms, evidence is the documentation that proves a specific control was operating correctly during a specific point in time within the observation period. A policy document says you will do something. Evidence proves you did it — and that you did it continuously, not just the week before the audit.</p>
<p>For example:</p>
<ul>
<li><p>For MFA enforcement (Control 1), evidence is a screenshot of your IAM Identity Center MFA settings taken at a specific date during the observation period, combined with an IAM credential report showing zero IAM users with console access.</p>
</li>
<li><p>For GuardDuty (Control 4), evidence is the GuardDuty console screenshot showing active detectors, plus your documented response to any findings during the period.</p>
</li>
<li><p>For access reviews (Control 12), evidence is the completed access review document with dates, names, and specific access changes made.</p>
</li>
</ul>
<p>The challenge is collecting this evidence continuously across 3–12 months without spending hundreds of hours on manual work. The solution is automated evidence collection infrastructure.</p>
<h3 id="heading-the-evidence-bucket-tamper-proof-storage-for-your-audit-evidence">The Evidence Bucket — Tamper-Proof Storage for Your Audit Evidence</h3>
<p>The evidence bucket is an S3 bucket with Object Lock enabled in GOVERNANCE mode. Object Lock prevents any object from being deleted or modified for the retention period you specify — in this case, 365 days. This means once a piece of evidence is uploaded, it can't be altered, even by a user with administrator access (without explicitly overriding the lock, which itself creates an audit trail).</p>
<p>This tamper-evident property is what gives the auditor confidence that the evidence was not created or modified after the fact.</p>
<pre><code class="language-hcl"># terraform/soc2-evidence-bucket.tf

resource "aws_s3_bucket" "soc2_evidence" {
  bucket = "\({var.company_name}-soc2-evidence-\){var.environment}"
}

# Block all public access to the evidence bucket
resource "aws_s3_bucket_public_access_block" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Enable versioning so overwrites create new versions, not replacements
resource "aws_s3_bucket_versioning" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Object Lock in GOVERNANCE mode — objects cannot be deleted for 365 days
resource "aws_s3_bucket_object_lock_configuration" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id

  rule {
    default_retention {
      mode = "GOVERNANCE"
      days = 365
    }
  }
}

# Encrypt all evidence at rest
resource "aws_s3_bucket_server_side_encryption_configuration" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}
</code></pre>
<h3 id="heading-the-daily-evidence-collector-lambda">The Daily Evidence Collector Lambda</h3>
<p>This Lambda function runs automatically every day and exports the status of each critical control to a time-stamped JSON file in the evidence bucket. Over your 3–12 month observation period, it creates a daily record proving that your controls were active and operating.</p>
<p>The function checks seven controls automatically: CloudTrail status, GuardDuty status, VPC Flow Logs, S3 public access block, EBS encryption, MFA compliance, and GuardDuty finding count. Each daily snapshot is uploaded with Object Lock enabled so it can't be modified.</p>
<pre><code class="language-python"># lambda/evidence-collector/handler.py

import boto3
import json
from datetime import datetime, timedelta, timezone

def lambda_handler(event, context):
    """
    Daily SOC2 evidence collector.
    Runs at 00:00 UTC every day via EventBridge scheduler.
    Exports control status to S3 evidence bucket with Object Lock.
    """
    evidence = {
        'collection_timestamp': datetime.now(timezone.utc).isoformat(),
        'collection_date': datetime.now(timezone.utc).strftime('%Y-%m-%d'),
        'account_id': boto3.client('sts').get_caller_identity()['Account'],
        'controls': {}
    }

    # Control 3: CloudTrail status
    cloudtrail = boto3.client('cloudtrail')
    trails = cloudtrail.describe_trails(includeShadowTrails=False)['trailList']
    multi_region_trails = [t for t in trails if t.get('IsMultiRegionTrail')]
    evidence['controls']['cloudtrail'] = {
        'status': 'PASS' if multi_region_trails else 'FAIL',
        'detail': f"{len(multi_region_trails)} multi-region trail(s) active",
        'trails': [t['Name'] for t in multi_region_trails]
    }

    # Control 4: GuardDuty status
    guardduty = boto3.client('guardduty')
    detectors = guardduty.list_detectors()['DetectorIds']
    unresolved_critical = 0
    for detector_id in detectors:
        findings = guardduty.list_findings(
            DetectorId=detector_id,
            FindingCriteria={
                'Criterion': {
                    'severity': {'Gte': 7},  # HIGH and CRITICAL only
                    'service.archived': {'Eq': ['false']}
                }
            }
        )
        unresolved_critical += len(findings['FindingIds'])

    evidence['controls']['guardduty'] = {
        'status': 'PASS' if detectors else 'FAIL',
        'detail': f"{len(detectors)} detector(s) active, {unresolved_critical} unresolved HIGH/CRITICAL findings",
        'unresolved_high_critical': unresolved_critical
    }

    # Control 5: VPC Flow Logs
    ec2 = boto3.client('ec2')
    flow_logs = ec2.describe_flow_logs(
        Filters=[{'Name': 'resource-type', 'Values': ['VPC']},
                 {'Name': 'flow-log-status', 'Values': ['ACTIVE']}]
    )['FlowLogs']
    evidence['controls']['vpc_flow_logs'] = {
        'status': 'PASS' if flow_logs else 'FAIL',
        'detail': f"{len(flow_logs)} active VPC flow log(s)",
        'active_flow_logs': len(flow_logs)
    }

    # Control 7: EBS encryption by default
    ebs_encryption = ec2.get_ebs_encryption_by_default()['EbsEncryptionByDefault']
    evidence['controls']['ebs_encryption_by_default'] = {
        'status': 'PASS' if ebs_encryption else 'FAIL',
        'detail': 'EBS encryption by default is enabled' if ebs_encryption else 'EBS encryption by default is NOT enabled'
    }

    # Control 8: S3 Block Public Access (account level)
    s3control = boto3.client('s3control')
    account_id = boto3.client('sts').get_caller_identity()['Account']
    try:
        pab = s3control.get_public_access_block(AccountId=account_id)['PublicAccessBlockConfiguration']
        all_blocked = all([pab['BlockPublicAcls'], pab['IgnorePublicAcls'],
                           pab['BlockPublicPolicy'], pab['RestrictPublicBuckets']])
        evidence['controls']['s3_block_public_access'] = {
            'status': 'PASS' if all_blocked else 'FAIL',
            'detail': 'All four S3 Block Public Access settings enabled' if all_blocked else 'One or more S3 Block Public Access settings not enabled',
            'configuration': pab
        }
    except Exception as e:
        evidence['controls']['s3_block_public_access'] = {'status': 'FAIL', 'detail': str(e)}

    # Upload evidence to S3 with Object Lock
    s3 = boto3.client('s3')
    evidence_key = f"daily/{evidence['collection_date']}/control-status.json"
    lock_until = datetime.now(timezone.utc) + timedelta(days=365)

    s3.put_object(
        Bucket='YOUR_EVIDENCE_BUCKET_NAME',
        Key=evidence_key,
        Body=json.dumps(evidence, indent=2),
        ContentType='application/json',
        ObjectLockMode='GOVERNANCE',
        ObjectLockRetainUntilDate=lock_until
    )

    # Alert if any control fails
    failed_controls = [k for k, v in evidence['controls'].items() if v['status'] == 'FAIL']
    if failed_controls:
        sns = boto3.client('sns')
        sns.publish(
            TopicArn='YOUR_ALERT_TOPIC_ARN',
            Subject=f'SOC2 Control Failure Detected — {evidence["collection_date"]}',
            Message=f'The following controls failed their daily check:\n\n{json.dumps(failed_controls, indent=2)}'
        )

    return {
        'statusCode': 200,
        'controls_checked': len(evidence['controls']),
        'controls_failed': len(failed_controls),
        'evidence_location': f"s3://YOUR_EVIDENCE_BUCKET_NAME/{evidence_key}"
    }
</code></pre>
<h3 id="heading-the-github-actions-evidence-workflow">The GitHub Actions Evidence Workflow</h3>
<p>This workflow runs daily and captures evidence that can't be automated through AWS APIs — GitHub-level controls like branch protection status, recent pull request activity, and CI pipeline results. It exports these as JSON files to the same evidence bucket.</p>
<pre><code class="language-yaml"># .github/workflows/soc2-evidence.yml
name: SOC2 Evidence Collection
on:
  schedule:
    - cron: '0 1 * * *'   # 01:00 UTC daily (after the Lambda runs at 00:00)
  workflow_dispatch:        # Allow manual trigger when needed

permissions:
  contents: read

jobs:
  collect-github-evidence:
    name: Collect GitHub Control Evidence
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/evidence-collector
          aws-region: us-east-1

      - name: Collect branch protection status
        run: |
          DATE=$(date +%Y-%m-%d)
          mkdir -p evidence/github

          # Export branch protection rules for main
          curl -s -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
            "https://api.github.com/repos/${{ github.repository }}/branches/main/protection" \
            | jq '{
                date: "'$DATE'",
                enforce_admins: .enforce_admins.enabled,
                required_reviews: .required_pull_request_reviews.required_approving_review_count,
                required_status_checks: .required_status_checks.contexts,
                allow_force_pushes: .allow_force_pushes.enabled
              }' &gt; evidence/github/branch-protection-$DATE.json

          echo "Branch protection evidence collected"
          cat evidence/github/branch-protection-$DATE.json

      - name: Upload evidence to S3
        run: |
          DATE=$(date +%Y-%m-%d)
          aws s3 sync evidence/ \
            s3://\({{ secrets.SOC2_EVIDENCE_BUCKET }}/daily/\)DATE/github/ \
            --no-progress
          echo "Evidence uploaded: s3://\({{ secrets.SOC2_EVIDENCE_BUCKET }}/daily/\)DATE/github/"
</code></pre>
<h2 id="heading-weeks-1114-auditor-selection-and-readiness-assessment">Weeks 11–14: Auditor Selection and Readiness Assessment</h2>
<h3 id="heading-how-to-choose-a-soc2-auditor">How to Choose a SOC2 Auditor</h3>
<p>Selecting the right auditor is more consequential than most teams realize. SOC2 audits are conducted by CPA firms — specifically, firms licensed to issue SOC reports. The right firm has experience with cloud-native, SaaS companies your size. The wrong firm could apply enterprise audit frameworks to a seed-stage startup and generate findings based on controls that aren't appropriate to your context.</p>
<p>Here is what to look for and what to watch out for:</p>
<h4 id="heading-experience-matters-more-than-brand">Experience matters more than brand</h4>
<p>A large Big Four firm isn't necessarily better than a specialist boutique auditor for a 20-person SaaS company.</p>
<p>Ask specifically: "How many SOC2 audits have you completed in the last 12 months for SaaS companies between 10 and 50 employees?" You want a firm where this is common, not exceptional.</p>
<h4 id="heading-verify-familiarity-with-your-compliance-tool">Verify familiarity with your compliance tool</h4>
<p>If you're using Vanta or Drata, confirm that the auditor has experience with evidence produced by those platforms. Some auditors prefer to collect evidence directly and are unfamiliar with automated evidence exports. An auditor who doesn't trust your Vanta evidence will ask you to re-collect everything manually.</p>
<h4 id="heading-understand-what-type-ii-actually-costs">Understand what Type II actually costs</h4>
<p>For a Series A SaaS company, expect \(15,000–\)30,000 for a SOC2 Type II audit with a 3-month observation period. A quote below \(10,000 often means the auditor is cutting corners on the review depth. A quote above \)50,000 for a small company typically means the firm is applying enterprise pricing to a startup engagement.</p>
<h4 id="heading-get-references-from-similar-companies">Get references from similar companies</h4>
<p>Ask the auditor for two or three references from SaaS companies they've audited in the last year. Call those references and ask: did the auditor understand cloud infrastructure? Were the findings reasonable? How was the communication during the review?</p>
<p>Here's a summary table of some things to watch out for:</p>
<table>
<thead>
<tr>
<th>Criteria</th>
<th>What to Look For</th>
<th>Red Flag</th>
</tr>
</thead>
<tbody><tr>
<td>Experience</td>
<td>5+ years, 20+ SaaS audits annually</td>
<td>"We have completed several SOC2 audits" (vague)</td>
</tr>
<tr>
<td>Tool familiarity</td>
<td>Has reviewed Vanta/Drata evidence before</td>
<td>Requires manual re-collection of automated evidence</td>
</tr>
<tr>
<td>Company size fit</td>
<td>Has audited companies your size</td>
<td>Only lists enterprise clients as references</td>
</tr>
<tr>
<td>Cost (Type II)</td>
<td>\(15K–\)30K for a 20-person company</td>
<td>Under \(10K or over \)50K without clear justification</td>
</tr>
<tr>
<td>References</td>
<td>Can provide SaaS company contacts to call</td>
<td>Cannot provide references</td>
</tr>
</tbody></table>
<h3 id="heading-how-to-run-a-readiness-assessment-mock-audit">How to Run a Readiness Assessment (Mock Audit)</h3>
<p>A readiness assessment is a self-conducted simulation of the real audit, run 2–4 weeks before you engage the auditor. Its purpose is to find and close gaps before the auditor finds them, because gaps found in a mock audit cost you a week of remediation time, while gaps found in the real audit cost you a conditional report and a re-review.</p>
<p>You can run the readiness assessment yourself or hire a consultant to run it. The consultant approach is more valuable because an independent reviewer will find gaps you have rationalised away.</p>
<p>The process:</p>
<ol>
<li><p><strong>Step 1:</strong> Work through every control in the checklist below and attempt to produce the evidence that an auditor would request.</p>
</li>
<li><p><strong>Step 2:</strong> For every control where you can't produce clear, timestamped evidence: that's a gap. Document it.</p>
</li>
<li><p><strong>Step 3:</strong> Prioritise gaps by type. Evidence gaps (missing evidence for an active control) require evidence collection infrastructure fixes. Control gaps (a control that isn't implemented) require engineering work.</p>
</li>
<li><p><strong>Step 4:</strong> Close all gaps before engaging the real auditor.</p>
</li>
</ol>
<table>
<thead>
<tr>
<th>Control</th>
<th>Evidence Required</th>
<th>How to Verify</th>
<th>Ready?</th>
</tr>
</thead>
<tbody><tr>
<td>MFA enforced</td>
<td>IAM credential report + SSO MFA policy screenshot</td>
<td><code>aws iam get-credential-report</code></td>
<td>⬜</td>
</tr>
<tr>
<td>CloudTrail active</td>
<td>Trail status + S3 delivery confirmation</td>
<td><code>aws cloudtrail get-trail-status</code></td>
<td>⬜</td>
</tr>
<tr>
<td>GuardDuty active</td>
<td>Detector list + finding review log</td>
<td><code>aws guardduty list-detectors</code></td>
<td>⬜</td>
</tr>
<tr>
<td>VPC Flow Logs</td>
<td>Active flow log list + sample log entries</td>
<td><code>aws ec2 describe-flow-logs</code></td>
<td>⬜</td>
</tr>
<tr>
<td>Secrets in Secrets Manager</td>
<td>Secret list + rotation policy confirmation</td>
<td><code>aws secretsmanager list-secrets</code></td>
<td>⬜</td>
</tr>
<tr>
<td>EBS encryption by default</td>
<td>Account-level encryption setting</td>
<td><code>aws ec2 get-ebs-encryption-by-default</code></td>
<td>⬜</td>
</tr>
<tr>
<td>S3 Block Public Access</td>
<td>Account-level PAB configuration</td>
<td><code>aws s3control get-public-access-block</code></td>
<td>⬜</td>
</tr>
<tr>
<td>Branch protection (no admin bypass)</td>
<td>GitHub branch protection API response</td>
<td>GitHub API or Settings UI</td>
<td>⬜</td>
</tr>
<tr>
<td>Trivy scanning in CI</td>
<td>GitHub Actions run history showing scans</td>
<td>GitHub Actions logs</td>
<td>⬜</td>
</tr>
<tr>
<td>Incident response runbook</td>
<td>Written runbook + tabletop exercise notes with date</td>
<td>Document review</td>
<td>⬜</td>
</tr>
<tr>
<td>Access review</td>
<td>Quarterly review document with specific changes made</td>
<td>Document review</td>
<td>⬜</td>
</tr>
<tr>
<td>Backup test</td>
<td>RDS restore log + data verification results</td>
<td>Document review</td>
<td>⬜</td>
</tr>
<tr>
<td>Change management log</td>
<td>GitHub PR history + ArgoCD sync history</td>
<td>GitHub and ArgoCD</td>
<td>⬜</td>
</tr>
</tbody></table>
<p><strong>The one thing most teams skip:</strong> Running the readiness assessment against their own evidence bucket. Pull a random day's evidence from the daily Lambda export and verify that it's complete, timestamped, and accurately reflects the control status on that day.</p>
<p>If the evidence file for December 14th shows GuardDuty as PASS but GuardDuty was actually disabled that day, the auditor will find the discrepancy in the AWS account history — and that's a qualified finding.</p>
<h2 id="heading-weeks-1518-the-observation-period">Weeks 15–18: The Observation Period</h2>
<h3 id="heading-how-the-auditor-observes-your-controls">How the Auditor Observes Your Controls</h3>
<p>The SOC2 auditor doesn't physically visit your office or sit inside your AWS console watching your infrastructure in real time. The audit is a remote, documentation-based process conducted entirely through evidence review.</p>
<p>Here is how it actually works:</p>
<p>First, the auditor provides a list of evidence requests — typically 80–150 items for a Type II audit. You upload the evidence to a shared portal (the auditor provides this — it is usually a secure document sharing platform). The auditor reviews the evidence, asks follow-up questions, and identifies gaps where evidence is missing or a control wasn't operating as described.</p>
<p>For automated controls like CloudTrail and GuardDuty, the evidence is your daily Lambda exports — the auditor spot-checks a sample of daily snapshots across the observation period to verify the controls were consistently active.</p>
<p>For manual controls like access reviews and backup tests, the evidence is the documents you produced when you ran those processes.</p>
<p>The practical implication: the auditor is trusting your evidence. This is why the Object Lock on your evidence bucket matters. It proves to the auditor that the evidence was generated at the time it claims to have been generated and hasn't been modified since.</p>
<h3 id="heading-what-the-auditor-reviews-over-the-observation-period">What the Auditor Reviews Over the Observation Period</h3>
<table>
<thead>
<tr>
<th>What They Check</th>
<th>How Often</th>
<th>What They Are Looking For</th>
</tr>
</thead>
<tbody><tr>
<td>CloudTrail logs</td>
<td>Spot check monthly</td>
<td>Manual console changes that bypassed IaC, gaps in log delivery</td>
</tr>
<tr>
<td>GuardDuty findings</td>
<td>Review quarterly summary</td>
<td>HIGH or CRITICAL findings not remediated within your documented SLA</td>
</tr>
<tr>
<td>Access review completion</td>
<td>Verify each quarterly cycle</td>
<td>Reviews skipped, reviews with no access changes despite employee turnover</td>
</tr>
<tr>
<td>Incident response tests</td>
<td>Verify annually</td>
<td>No tabletop exercise conducted during the observation period</td>
</tr>
<tr>
<td>Evidence collection</td>
<td>Verify continuous coverage</td>
<td>Gaps in daily evidence exports, missing evidence for specific dates</td>
</tr>
<tr>
<td>Change management log</td>
<td>Sample PR/sync history</td>
<td>Deployments with no associated pull request or review</td>
</tr>
</tbody></table>
<h3 id="heading-what-triggers-a-finding">What Triggers a Finding</h3>
<p>A SOC2 finding is the auditor's documented conclusion that a control wasn't operating effectively during the observation period. Findings range from observations (minor issues that don't affect the audit opinion) to qualified opinions (material failures that result in a qualified rather than unqualified report).</p>
<p>Understanding what triggers findings — and which ones restart the observation period — is critical for managing your audit timeline.</p>
<p><strong>Control gaps</strong> occur when a required control isn't implemented or was disabled during the observation period. If you discover in month 2 that MFA wasn't enforced on one IAM user for the first three weeks, you must document the remediation and demonstrate the gap was closed.</p>
<p>Whether this restarts your observation period depends on how long the gap lasted and how the auditor assesses the risk — but a gap of less than 30 days that's immediately remediated and documented typically doesn't restart the clock.</p>
<p><strong>Evidence gaps</strong> are more serious. If your daily Lambda evidence collector failed for two weeks and produced no evidence exports, you have a two-week window with no documented proof that your controls were operating. The auditor can't verify controls they can't see evidence for.</p>
<p>Evidence gaps almost always require extending the observation period because there's no way to retroactively produce evidence for a period that wasn't recorded.</p>
<p><strong>Process failures</strong> occur when a manual control wasn't executed as documented. The most common is an access review that was skipped. Like control gaps, these can typically be remediated without restarting the clock if they're documented promptly and the remediation is clear.</p>
<p><strong>Unpatched critical CVEs</strong> are a special case. If Trivy identifies a CRITICAL vulnerability in a production container and it remains unpatched for more than your documented remediation SLA (typically 30 days for critical, 90 days for high), this is a qualified finding that the auditor will note in the report.</p>
<h3 id="heading-how-to-close-gaps-without-restarting-the-clock">How to Close Gaps Without Restarting the Clock</h3>
<p>When you discover a gap during the observation period:</p>
<p><strong>For control gaps:</strong></p>
<pre><code class="language-plaintext">1. Fix the control immediately — don't wait
2. Document the fix: screenshot, PR link, or CLI command output with timestamp
3. Note the gap date range in your audit log: "Control gap: 2024-03-10 to 2024-03-14 (4 days). Root cause: [X]. Remediated: [Y]. No customer data accessed during gap period."
4. Notify your auditor proactively — they will find it anyway; proactive disclosure is better than defensive explanation
5. The observation period doesn't restart if the gap was short-lived and promptly remediated
</code></pre>
<p><strong>For evidence gaps:</strong></p>
<pre><code class="language-plaintext">1. Fix the evidence collection infrastructure immediately
2. Understand that you can't retroactively generate evidence for the gap period
3. The observation period for affected controls effectively restarts from the date evidence collection resumed
4. If the gap is early in your observation period, you may be able to extend the period rather than restart — discuss with your auditor
</code></pre>
<p><strong>The pro tip:</strong> Set up a CloudWatch alarm that triggers if the evidence Lambda fails to deliver to S3 on schedule. A missing daily evidence file is caught within 24 hours, not discovered during the audit review.</p>
<h2 id="heading-the-90-day-soc2-timeline-at-a-glance">The 90-Day SOC2 Timeline at a Glance</h2>
<table>
<thead>
<tr>
<th>Weeks</th>
<th>Focus</th>
<th>Key Deliverables</th>
<th>Common Mistake</th>
</tr>
</thead>
<tbody><tr>
<td>1–2</td>
<td>Scope</td>
<td>Boundary diagram, network segmentation Terraform</td>
<td>Over-scoping to include dev and staging</td>
</tr>
<tr>
<td>3–6</td>
<td>Controls</td>
<td>14 controls implemented and collecting evidence</td>
<td>Starting controls after the observation period begins</td>
</tr>
<tr>
<td>7–10</td>
<td>Evidence</td>
<td>S3 evidence bucket, Lambda daily collector, GitHub Actions workflow</td>
<td>Manual evidence collection with inevitable gaps</td>
</tr>
<tr>
<td>11–14</td>
<td>Readiness</td>
<td>Mock audit, gap remediation, auditor selected</td>
<td>Skipping the mock audit</td>
</tr>
<tr>
<td>15–18</td>
<td>Observation</td>
<td>Daily evidence, quarterly reviews, incident response test</td>
<td>Discovering evidence gaps during the audit rather than before</td>
</tr>
</tbody></table>
<h2 id="heading-whats-next">What's Next?</h2>
<p>Start with Week 1. Define your SOC2 boundary. Apply the four-question framework to every system in your infrastructure. Draw the diagram in Excalidraw. Document the network segmentation controls.</p>
<p>Then implement the 14 controls in order, starting with MFA and CloudTrail — the two that most commonly fail audits when they're missing.</p>
<p>Then build your evidence collection infrastructure before the observation period starts. The automated Lambda and GitHub Actions workflow are the difference between a smooth audit and a 60-day extension.</p>
<p>One thing to remember: SOC2 is 20% controls, 30% evidence, and 50% continuous operation. Start early. Automate everything. Run a mock audit before you call the real one.</p>
<h2 id="heading-resources">Resources</h2>
<p>The following resources are referenced throughout this guide:</p>
<ul>
<li><p><a href="https://www.aicpa-cima.com/resources/landing/system-and-organization-controls-soc-suite-of-services"><strong>AICPA SOC2 Overview</strong></a> — The official SOC2 documentation from the American Institute of CPAs, including the Trust Service Criteria</p>
</li>
<li><p><a href="https://www.vanta.com/"><strong>Vanta</strong></a> — Compliance automation platform that connects to AWS and GitHub to automate evidence collection and track control status</p>
</li>
<li><p><a href="https://drata.com/"><strong>Drata</strong></a> — Alternative compliance automation platform with similar capabilities to Vanta</p>
</li>
<li><p><a href="https://github.com/aquasecurity/trivy"><strong>Trivy by Aqua Security</strong></a> — Open-source container and filesystem vulnerability scanner used in Control 10</p>
</li>
<li><p><a href="https://excalidraw.com/"><strong>Excalidraw</strong></a> — Free, open-source diagram tool for creating the SOC2 boundary diagram</p>
</li>
<li><p><a href="https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html"><strong>AWS IAM Identity Center documentation</strong></a> — Official AWS documentation for setting up SSO and MFA enforcement</p>
</li>
<li><p><a href="https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches"><strong>GitHub branch protection documentation</strong></a> — Official GitHub documentation for configuring branch protection rules</p>
</li>
<li><p><a href="https://argo-cd.readthedocs.io/"><strong>ArgoCD documentation</strong></a> — Official ArgoCD documentation for GitOps deployment and sync history</p>
</li>
</ul>
<p><a href="https://github.com/aayostem">Ayobami Adejumo</a> <em>is a senior platform engineer and FinOps specialist. He writes about SOC2 compliance engineering, Kubernetes cost optimization, and platform engineering.</em></p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ How to Deploy a Serverless Spam Classifier Using Scikit-Learn, AWS Lambda, & API Gateway ]]>
                </title>
                <description>
                    <![CDATA[ In today's digital world, spam is no longer just an annoyance - it's a growing security threat. To combat this, developers often turn to machine learning to build intelligent filters that can distingu ]]>
                </description>
                <link>https://www.freecodecamp.org/news/deploying-serverless-spam-classifier/</link>
                <guid isPermaLink="false">69f2e347b18c978233780179</guid>
                
                    <category>
                        <![CDATA[ Machine Learning ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ serverless ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AWS ]]>
                    </category>
                
                    <category>
                        <![CDATA[ MathJax ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Data Architecture ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Rakshath Naik ]]>
                </dc:creator>
                <pubDate>Thu, 30 Apr 2026 05:06:15 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/08672d22-a4df-4b99-8ef7-fffd18f5dc07.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>In today's digital world, spam is no longer just an annoyance - it's a growing security threat. To combat this, developers often turn to machine learning to build intelligent filters that can distinguish legitimate emails from malicious ones.</p>
<p>While building a machine learning model in a notebook is relatively straightforward, the real challenge lies in the last mile: deploying that model into a scalable, production-ready system that users can actually interact with.</p>
<p>In this project, I built an end-to-end serverless spam classifier, combining Scikit-learn for model development with AWS Lambda, Amazon S3, and Amazon API Gateway for deployment. The result is a lightweight, scalable API that can classify messages in real time.</p>
<p>The system is designed to be modular and cost-efficient, allowing the model to be retrained and updated independently without affecting the live API. From detecting "free iPhone" scams to identifying phishing attempts, this project demonstrates how to bridge the gap between machine learning experimentation and real-world deployment.</p>
<h3 id="heading-table-of-contents">Table of&nbsp;Contents</h3>
<ul>
<li><p><a href="#heading-1-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-2-building-the-brain-the-model">Building the Brain: The Model</a></p>
</li>
<li><p><a href="#heading-3-deploying-the-model-to-aws">Deploying the Model to AWS</a></p>
</li>
<li><p><a href="#heading-4-how-to-run-the-project-locally">How to Run The Project Locally</a></p>
</li>
<li><p><a href="#heading-5-our-project-architecture">Our Project Architecture</a></p>
</li>
<li><p><a href="#heading-6-conclusion-the-power-of-serverless-ai">Conclusion: The Power of Serverless AI</a></p>
</li>
<li><p><a href="#heading-7-acknowledgment-references">Acknowledgment / References</a></p>
</li>
</ul>
<h2 id="heading-1-prerequisites">1. Prerequisites</h2>
<ol>
<li><p><strong>Fundamental skills:</strong> Basic proficiency in Python and understanding of Machine Learning concepts like classification.</p>
</li>
<li><p><strong>AWS account:</strong> Access to an AWS account with permissions for Lambda, S3, and API Gateway.</p>
</li>
<li><p><strong>Environment:</strong> Python 3.11 installed, along with libraries like scikit-learn, pandas, and joblib.</p>
</li>
<li><p><strong>AWS CLI:</strong> Configured on your local machine for file uploads.</p>
</li>
<li><p><strong>HuggingFace account:</strong> You can directly download the model from my account.</p>
</li>
</ol>
<h2 id="heading-2-building-the-brain-the-model">2. Building the Brain: The&nbsp;Model</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6942c2903c5d674e359eaf1e/b43af198-1472-4914-9469-6cd5ca5384e2.png" alt="Demonstrational image to show the brain of AI." style="display: block;" width="1000" height="563" loading="lazy">

<p><em>Photo by</em> <a href="https://unsplash.com/@steve_j?utm_source=medium&amp;utm_medium=referral"><em>Steve A Johnson</em></a> <em>on</em> <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral"><em>Unsplash</em></a></p>
<p>At the heart of this project lies a supervised learning approach. Instead of simply specifying which words are considered spam, we'll provide the computer with a dataset and an algorithm, enabling it to learn and identify spam patterns on its own.</p>
<h3 id="heading-1-vectorization-turning-text-into-math">1. Vectorization: Turning Text into&nbsp;Math</h3>
<p>Machine Learning models can't <strong>read</strong> text. They require numerical input. To solve this, we used the <a href="https://www.freecodecamp.org/news/how-to-extract-keywords-from-text-with-tf-idf-and-pythons-scikit-learn-b2a0f3d7e667/">TF-IDF</a> (Term Frequency-Inverse Document Frequency) Vectorizer.</p>
<pre><code class="language-python">feature_extraction = TfidfVectorizer(min_df=1, stop_words='english', lowercase=True)
X_train_features = feature_extraction.fit_transform(X_train
</code></pre>
<p>Here's the mathematical formula:</p>
<p>$$w_{i,j} = tf_{i,j} \times \log \left( \frac{N}{df_i} \right)$$</p>
<p>TF-IDF term definitions:</p>
<ul>
<li><p><strong>wᵢ,ⱼ (Weight):</strong> The final importance score of a specific word in a document.</p>
</li>
<li><p><strong>tfᵢ,ⱼ (Term Frequency):</strong> How often a word appears in a single email.</p>
</li>
<li><p><strong>N (Total Documents):</strong> The total count of all emails in your dataset.</p>
</li>
<li><p><strong>dfᵢ (Document Frequency):</strong> The number of different emails that contain this specific word.</p>
</li>
<li><p><strong>log(N/dfᵢ) (IDF):</strong> A penalty that lowers the score of common words like <strong>the</strong> or <strong>is</strong> that appear everywhere.</p>
</li>
</ul>
<p>It cleans the data by removing common words, converts all text to lowercase for consistency, and assigns more importance to rare and meaningful words while giving less importance to frequently used words.</p>
<h3 id="heading-2-training-the-logistic-regression-engine">2. Training: The Logistic Regression Engine</h3>
<p>We'll use <strong>Logistic Regression</strong> here, a classification algorithm that predicts the probability of an outcome.</p>
<p>In this stage, we feed our vectorized training data into the Logistic Regression algorithm. The goal is to establish a mathematical relationship between specific word weights and the <strong>Spam</strong> or <strong>Ham</strong> label.</p>
<p>During training, the model iteratively adjusts its internal parameters to minimize error, eventually learning that words like winner or free correlate highly with spam, while conversational language correlates with legitimate messages.</p>
<pre><code class="language-python">model = LogisticRegression()
model.fit(X_train_features, Y_train)
</code></pre>
<p>In our case, it calculates the probability that an email belongs to spam or HAM.</p>
<p>The algorithm uses the Sigmoid function to map any real-valued number into a value between 0 and 1.</p>
<p>$$P(y=1|x) = \frac{1}{1 + e^{-(z)}}$$</p>
<p>where z = β₀ + β₁x₁ +&nbsp;… + βₙxₙ.</p>
<h3 id="heading-3-evaluation-testing-the-intelligence">3. Evaluation: Testing the Intelligence</h3>
<p>After training, we need to verify if the brain actually works on data it hasn't seen before.</p>
<pre><code class="language-python">prediction_on_test_data = model.predict(X_test_features)
accuracy_on_test_data = accuracy_score(Y_test, prediction_on_test_data)
</code></pre>
<p>By comparing the model’s predictions against the actual labels in our test set, we calculate an Accuracy Score. This gives us the confidence that the model is ready for the real world (achieving ~94% accuracy in our tests).</p>
<h3 id="heading-4-exporting-the-logic-serialization">4. Exporting the Logic (Serialization)</h3>
<p>To move this brain from our local Python environment to the AWS Cloud, we'll use Joblib to save our work into binary files (.pkl).</p>
<pre><code class="language-python">joblib.dump(model, 'spam_model.pkl')
joblib.dump(feature_extraction, 'vectorizer.pkl')
</code></pre>
<p>We use the Pickle format because it allows us to freeze complex Python objects (mathematical weights and word mappings) into a portable binary format that can be instantly re-animated in the cloud.</p>
<p>We need the Vectorizer to translate new user text into the exact numerical coordinates the Model was trained to understand. Using one without the other is like having a key but no lock.</p>
<p>The trained Logistic Regression model and TF-IDF vectorizer are openly available for the community on Hugging Face here: <a href="https://huggingface.co/rakshath1/mail-spam-detector">Get the model on HuggingFace</a>.</p>
<h2 id="heading-3-deploying-the-model-to-aws">3. Deploying the Model to&nbsp;AWS</h2>
<p>Training a model is science, while deploying it is engineering. To make this classifier accessible to the world, we'll use a serverless stack that scales automatically and incurs nearly no maintenance costs.</p>
<h3 id="heading-1-model-storage-amazon-s3">1. Model Storage: Amazon&nbsp;S3</h3>
<p>First, we'll uploade our&nbsp;.pkl files to an S3 bucket. By decoupling the model from the code, we can update the AI's intelligence (simply by overwriting the file in S3) without redeploying the backend code. It makes the system highly maintainable.</p>
<h3 id="heading-2-the-production-backend-aws-lambda">2. The Production Backend: AWS&nbsp;Lambda</h3>
<p>To make the AI accessible, we'll move from a local script to a Serverless Cloud Architecture. This ensures the model is always available without the cost of a 24/7 server.</p>
<p>The deployment environment is AWS Lambda (Python 3.11). Since Lambda is a lightweight environment, it doesn't include Scikit-Learn or Joblib. To provide these, we'll download and store them in our S3 bucket and import them through the layers.</p>
<p><strong>Commands in AWS CLI:</strong></p>
<pre><code class="language-python">
# 1. Create a workspace
mkdir ml_layer &amp;&amp; cd ml_layer

# 2. Install scikit-learn and its dependencies into a folder
pip install \
    --platform manylinux2014_x86_64 \
    --target=python/lib/python3.11/site-packages \
    --implementation cp \
    --python-version 3.11 \
    --only-binary=:all: \
    scikit-learn joblib

# 3. Zip the folder
zip -r sklearn_lib.zip python

# 4. Upload to S3 (Using AWS CLI)
aws s3 cp sklearn_lib.zip s3://YOUR-BUCKET-NAME/
</code></pre>
<p>We store the Scikit-Learn library as a ZIP in S3 to bypass the AWS Lambda deployment package size limit. This allows the function to dynamically load heavy dependencies only when needed without bloating the core code.</p>
<p><strong>The Lambda Function:</strong></p>
<pre><code class="language-python">
import json
import boto3
import os
import sys
from io import BytesIO

# Ensures the custom Lambda layer(containing sklearn/joblib)
sys.path.append('/opt/python')

try:
    import joblib
except ImportError:
    # Fallback for specific Scikit-Learn distributions
    from sklearn.utils import _joblib as joblib

# Initialize S3 client
s3 = boto3.client('s3')

# Use placeholders for the article so readers can insert their own values
BUCKET_NAME = 'YOUR_S3_BUCKET_NAME' 
MODEL_KEY = 'spam_model.pkl'
VECTORIZER_KEY = 'vectorizer.pkl'

# Global variables for 'Warm Start' caching (improves performance by keeping model in RAM)
model = None
vectorizer = None

def load_model():
    """Downloads model files from S3 only if they aren't already in RAM"""
    global model, vectorizer
    if model is None or vectorizer is None:
        try:
            # 1. Load the Logistic Regression Model from S3
            m_obj = s3.get_object(Bucket=BUCKET_NAME, Key=MODEL_KEY)
            model = joblib.load(BytesIO(m_obj['Body'].read()))
            
            # 2. Load the TF-IDF Vectorizer directly from S3
            v_obj = s3.get_object(Bucket=BUCKET_NAME, Key=VECTORIZER_KEY)
            vectorizer = joblib.load(BytesIO(v_obj['Body'].read()))
        except Exception as e:
            raise Exception(f"Failed to load .pkl files from S3: {str(e)}")

def lambda_handler(event, context):
    try:
        # Ensure model and vectorizer are ready before processing
        load_model()
        
        # Handles both direct Lambda tests and API Gateway POST requests
        body = event.get('body', event)
        if isinstance(body, str):
            body = json.loads(body)
            
        text = body.get('text', '')
            
        if not text:
            return {
                'statusCode': 400,
                'body': json.dumps({'error': 'No text provided.'})
              }

        # 1. Transform input text to numeric features using the trained Vectorizer
        data_vec = vectorizer.transform([text])
        
        # 2. Predict using the Logistic Regression Model 
        prediction = int(model.predict(data_vec)[0])
        
      # 3. Map numeric result to human-readable label
        result_label = "HAM" if prediction == 1 else "SPAM"
        
        # RESPONSE WITH CORS
        return {
            'statusCode': 200,
            'headers': {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Origin': '*' # needed for cross-domain web integration
            },
            'body': json.dumps({
                'status': 'success',
                'classification': result_label,
                'input_text': text
            })
        }
        
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps({'error_message': f"Inference Error: {str(e)}"})
        }
</code></pre>
<p>Key features of the Lambda function:</p>
<ol>
<li><p><strong>Warm start caching:</strong> By defining the model and vectorizer variables outside the lambda_handler, we store them in the container's memory. This significantly reduces cold start latency for subsequent requests.</p>
</li>
<li><p><strong>Dynamic dependency loading:</strong> The <strong>sys.path.append('/opt/python')</strong> line allows us to import heavy libraries from S3/Layers without exceeding the upload limit.</p>
</li>
<li><p><strong>Bimodal input handling:</strong> The function is designed to handle both direct JSON testing from the AWS console and stringified payloads sent via API Gateway.</p>
</li>
</ol>
<h3 id="heading-3-the-api-gateway-the-bridge-to-the-web">3. The API Gateway - The Bridge to the&nbsp;Web</h3>
<img src="https://cdn.hashnode.com/uploads/covers/6942c2903c5d674e359eaf1e/8aa3e8d7-569a-4dd5-a6ac-184922474952.png" alt="Demonstrational image to show the API Gateway." style="display: block;" width="1000" height="563" loading="lazy">

<p>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></p>
<h4 id="heading-creating-the-rest-api">Creating the REST API</h4>
<p>Next we'll create a REST API with a single POST method. Why POST, you might be wondering? Well, we need to securely send a JSON payload containing the user’s text message to our model.</p>
<ol>
<li><p>First navigate to the Amazon API Gateway console and select Create API -&gt; REST API.</p>
</li>
<li><p>Give your API a name, such as EmailSpamPredictor-API, and set the Endpoint Type to Regional.</p>
</li>
<li><p>Then in the left sidebar, click Resources and enter a resource name (e.g: <strong>/ predict</strong> as entered by me)</p>
</li>
<li><p>Next click the create method and select POST and then select Lambda Function for integration type</p>
</li>
<li><p>Ensure Lambda Proxy integration is enabled (this allows the full request to pass through to your code).</p>
</li>
</ol>
<p><strong>The CORS Configuration (The Troubleshooting Hub)</strong><br>This is where many developers encounter the dreaded <strong>Connection Error</strong>. Since our API is hosted on AWS, and if your front-end is on a separate website, the browser’s Same-Origin Policy will block the request by default.</p>
<p>To fix this, we'll enable <strong>CORS:</strong></p>
<ol>
<li><p><strong>Access-Control-Allow-Origin:</strong> Set to * (or specifically to your domain) to tell the browser that the API is allowed to talk to your front-end.</p>
</li>
<li><p><strong>The OPTIONS method:</strong> API Gateway creates an OPTIONS method automatically. This handles the Preflight request where the browser asks, “Are you allowed to receive data from me?” before sending the actual text.</p>
</li>
<li><p><strong>Access-Control-Allow-Headers:</strong> In the screenshot, you'll notice headers like Content-Type and Authorization are allowed. This ensures that when our JavaScript fetch() call sets the content type to application/json, the API Gateway doesn't reject it.</p>
</li>
</ol>
<img src="https://cdn.hashnode.com/uploads/covers/6942c2903c5d674e359eaf1e/cf5c87c6-f374-4dda-8001-77a0aab52672.png" alt="Image illustrates the CORS configuration for our project. " style="display: block;" width="1487" height="617" loading="lazy">

<p>Image illustrates the CORS configuration for our project. (Image by author)</p>
<h4 id="heading-deployment-stages">Deployment Stages</h4>
<p>Once the API is deployed to a production stage, AWS generates a permanent Invoke URL. This acts as the public gateway to our model and typically follows this structure: <a href="https://%5Bapi-id%5D.execute-api.%5Bregion%5D.amazonaws.com/prod/classify">https://[api-id].execute-api.[region].amazonaws.com/prod/classify</a>.</p>
<h4 id="heading-connecting-the-frontend-the-javascript-layer">Connecting the Frontend (The JavaScript Layer)</h4>
<p>With the API live, we can now write a simple JavaScript function to talk to our model. This script runs whenever a user clicks the <strong>Analyze</strong> button on your site.</p>
<pre><code class="language-python">
async function checkSpam() {
    const message = document.getElementById("userInput").value;
    const apiUrl = "YOUR_API_GATEWAY_INVOKE_URL";

    try {
        const response = await fetch(apiUrl, {
            method: "POST",
            headers: {
                "Content-Type": "application/json"
            },
            body: JSON.stringify({ "text": message })
        });

        const data = await response.json();
        
        // Display result on the webpage
        const resultElement = document.getElementById("result");
        resultElement.innerText = `Prediction: ${data.classification}`;
        resultElement.style.color = data.classification === "SPAM" ? "red" : "green";

    } catch (error) {
        console.error("Error:", error);
        alert("Could not connect to the Spam Detector API.");
    }
}
</code></pre>
<h2 id="heading-4-how-to-run-the-project-locally">4. How to Run The Project&nbsp;Locally</h2>
<p>You can store the front-end as an HTML file. Once it's ready, you shouldn’t just double-click the&nbsp;.html file. Opening it as a <strong>file</strong> in your browser can cause security restrictions. Instead, you should host it using a simple local server.</p>
<p><strong>Step 1:</strong> Open the terminal or Command Prompt.</p>
<p><strong>Step 2:</strong> Navigate to your project folder</p>
<pre><code class="language-shell">cd [PATH_TO_YOUR_FOLDER]
</code></pre>
<p><strong>Step 3:</strong> Start a local Python web server.</p>
<pre><code class="language-shell">python -m http.server 8000
</code></pre>
<p><strong>Step 4:</strong> Access the application.</p>
<p>Open your browser and navigate to:<br><a href="http://localhost:8000/your-file-name.html">http://localhost:8000/your-file-name.html</a></p>
<p><strong>Watch the Demo:</strong></p>
<div class="embed-wrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/q2X_azntmzY" style="aspect-ratio: 16 / 9; width: 100%; height: auto;" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen="" loading="lazy"></iframe></div>

<h2 id="heading-5-our-project-architecture">5. Our Project Architecture</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6942c2903c5d674e359eaf1e/c17673d4-5dd0-43dc-8e8d-3015bcd31864.png" alt="Image showing the Architecture Diagram of our Project." style="display: block;" width="1000" height="563" loading="lazy">

<p>The image illustrates the architecture of our project (Building a Serverless Spam Classifier). It shows the process that takes place from the client input to the final model output. (Image by Author)</p>
<ol>
<li><p><strong>Client Front-End Interaction:</strong> The process starts on the far left. A user interacts with the web interface (for example, a website or a desktop app). They input text like <strong>WIN free iPhone now</strong> and trigger a request.</p>
</li>
<li><p><strong>The Entry Point: API Gateway:</strong> The request hits the Amazon API Gateway, which acts as the <strong>security guard</strong> and translator.&nbsp;<br><strong>(a)</strong> CORS OPTIONS handles the pre-flight handshake to ensure the browser has permission to talk to the AWS cloud.&nbsp;<br><strong>(b)</strong> Classification Request (POST) routes the actual message data to your backend logic.</p>
</li>
<li><p><strong>The Engine: AWS Lambda (Python 3.11):</strong>&nbsp;The central “<strong>lightbulb</strong>” represents your Lambda function. This is where the code you wrote lives. It doesn’t run 24/7 – it only wakes up when a request arrives.</p>
</li>
<li><p><strong>Storage &amp; Retrieval: S3 Bucket:</strong> Since Lambda is lightweight, it doesn’t store your heavy Machine Learning files internally.<br><strong>Dependency and Model Download:</strong> The function reaches out to the S3 Bucket to pull in the sklearn_<a href="http://lib.zip">lib.zip</a> (the engine) and the&nbsp;.pkl files (the intelligence).&nbsp;<br><strong>Required Dependency and Model:</strong> These assets are loaded into the Lambda’s temporary memory to prepare for the prediction.</p>
</li>
<li><p><strong>The Inference Pipeline:</strong>&nbsp;Inside the Lambda, a three-step mathematical cycle occurs:<br><strong>(a) Text Vectorizer:</strong> Translates the words into numbers.<br><strong>(b) Logistic Regression:</strong> Calculates the probability of spam based on those numbers.<br><strong>(c) Label:</strong> Assigns a final result (Spam or Ham).</p>
</li>
<li><p><strong>The Result Delivery:</strong> The result is sent back through the API Gateway, including the necessary CORS Headers to ensure the browser accepts it. The front-end then updates to show the “<strong>Result: SPAM</strong>” with a visual indicator.</p>
</li>
</ol>
<h2 id="heading-6-conclusion-the-power-of-serverless-ai">6. Conclusion: The Power of Serverless AI</h2>
<p>By merging the mathematical simplicity of Logistic Regression with the industrial strength of AWS Serverless Architecture, we have transformed a static Python script into a globally accessible, scalable API.</p>
<p>This project demonstrates that you don’t need a massive budget or a 24/7 dedicated server to deploy high-quality Machine Learning.</p>
<p>Using the S3-to-Lambda workaround allowed us to bypass common storage hurdles, ensuring that our Brain (the model) and its Muscle (Scikit-Learn) could function seamlessly within the cloud’s ephemeral environment. It bridges the gap between experimentation and real-world applications, making AI systems practical, efficient, and accessible.</p>
<h2 id="heading-7-acknowledgment-references">7. Acknowledgment / References</h2>
<ul>
<li><p>Pre-trained spam classification model: View on Hugging Face (<a href="https://huggingface.co/rakshath1/mail-spam-detector"><strong>rakshath1/mail-spam-detector · Hugging Face</strong></a><strong>)</strong></p>
</li>
<li><p>Scikit-learn <a href="https://scikit-learn.org/stable/api/index.html?utm_source=chatgpt.com">Documentation</a></p>
</li>
<li><p>AWS Lambda <a href="https://docs.aws.amazon.com/lambda/latest/api/welcome.html?utm_source=chatgpt.com">Documentation</a></p>
</li>
<li><p>Amazon S3 <a href="https://aws.amazon.com/documentation-overview/s3/">Documentation</a></p>
</li>
<li><p>Amazon API Gateway <a href="https://docs.aws.amazon.com/apigateway/">Documentation</a></p>
</li>
</ul>
<h3 id="heading-connect-with-me">Connect With Me</h3>
<ul>
<li><p><a href="https://medium.com/@rakshathnaik62">Medium</a></p>
</li>
<li><p><a href="https://www.linkedin.com/in/rakshath-/">LinkedIN</a></p>
</li>
</ul>
<p><strong>You may also like</strong></p>
<ol>
<li><p><a href="https://qubrica.com/python-polars-v-s-pandas-libraries-comparison/">How Polars overtook Pandas</a></p>
</li>
<li><p><a href="https://qubrica.com/devops-is-dead-platform-engineering-2026/"><strong>DevOps is Dead. Long Live Platform Engineering</strong></a></p>
</li>
</ol>
 ]]>
                </content:encoded>
            </item>
        
    </channel>
</rss>
