But have you ever wondered what's actually happening under the hood? Like, how is it even possible to have one app set up a whole other app that's ready to test, share, download, and so on?

Somewhere, an AI model is writing code. That's clear. But how? That code needs to be installed, built, and run. And the part I was a bit skeptical about was that nobody even seems to read that code nowadays before it runs. It could be broken, it could be slow, or it could even try to do something it shouldn't like wipe out your whole system with rm -rf. We've had such things happen from AI, so we can't be 100% sure.

That's the problem this article addresses.

Here, you'll build your own Lovable-style AI app builder. This isn't going to be a clone. Rather, we'll dive into the logic behind Lovable to see how it really works. We'll keep the UI pretty basic.

For our project, a user will be able to describe an app in plain English, and an AI agent will write the code inside an isolated cloud sandbox. The AI will fixe its own errors, and then show a live preview. From there, the user can keep building on top of it through chat, roll back to any version, and publish the finished app with a single click.

Prerequisites

This tutorial gets into some more advanced concepts, so before you start, it’ll help if you’re comfortable with:

• JavaScript or TypeScript

• React and basic Next.js concepts

• Node.js and a bit of working with APIs

• Git and basic version control

• Docker

• Postgres

• AWS is optional. You can follow the entire tutorial locally using MinIO instead.

To run the project yourself, you’ll also need:

• Node.js 22+

• pnpm

• Docker installed locally

• An API key for a sandbox provider (here we'll use Tensorlake Sandboxes)

• An Anthropic (recommended) or OpenAI API key

You don’t need to be an expert in any of these. A basic understanding is enough, and we’ll go through the important parts as we build.

What's Covered Here:

In this tutorial, you'll build the whole thing from scratch. Here's what you'll learn along the way:

• Why AI generated code needs a sandbox

• How to get a fresh sandbox ready in about 4 seconds instead of 30+ using memory snapshots

• How to build an agent loop that writes code, checks its own work, and fixes its own errors

• How to stream what the agent is doing to the browser in real time (and not lose it on a page refresh)

• How to show a live preview with hot reload through your own gateway

• How to version every change with Git without ever putting a token inside the sandbox

• How to put idle sandboxes to sleep, wake them up when needed, and publish the final app

This gets into some advanced concepts, but follow along and you'll learn a lot along the way. I definitely did while building it. 😉

Table of Contents

• What's the Plan (the Architecture)

• Why Do We Need a Sandbox?

• How to Set Up the Project

• Core Components in the Application

  • Starting a Sandbox Fast

  • The Agent Loop

  • Letting the Agent Check Its Own Work

  • Streaming Progress to the Browser

  • The Live Preview Gateway

  • Versions with Git

  • Sleeping, Waking, and Sharing the Sandboxes

  • Publishing the App

• App Builder in Action

• Conclusion

What's the Plan (the Architecture)

Before diving into the code, it helps to understand how everything fits together, because there are quite a few concepts worth understanding earlier.

The app is split into three processes and a couple of pieces of infrastructure. One thing I was very strict about from the start is that the web app should never run the AI agent itself, and the agent should never run inside the sandbox. You'll see why that matters in a bit.

Here's the flow from start to finish:

Sending a Prompt

When a user types a prompt, the web app (Next.js) saves it, creates a "run" in Postgres, puts a job on a queue, and returns right away. So the user isn't sitting there waiting on some HTTP request while the agent does its thing.

Running the Agent

A worker process picks up that job. First it makes sure the project has a running sandbox, and then it starts the agent loop. The LLM decides what to do, and every tool call it makes (write a file, run a command, install a package and all) actually happens inside the sandbox.

Every single step is also saved to the database as an event, and that's how the browser gets them live.

Showing the Preview

The generated app runs its own Vite dev server inside the sandbox. We have a small gateway service that proxies http://<project-id>.preview.localhost:4000 to that dev server, including the WebSocket that Vite uses for hot reload.

So as the agent edits files, the preview just updates by itself, which honestly still feels kinda cool every time I see it.

Saving and Publishing

Once the agent is done, the worker commits the changes as a new version. And when the user clicks Publish, the app gets built and the static files are uploaded to S3, so the published site keeps working even when the sandbox is asleep.

That's pretty much the high level architecture of our application. To put it simply:

• Next.js: UI and API layer

• Worker (Node.js + pg-boss): Agent layer

• Gateway (Node.js proxy): Preview layer

• Postgres: State and job queue layer

• Cloud sandbox: Execution layer

• S3 (MinIO locally): Storage layer

• Any LLM of your choice (Claude or GPT in our case): Reasoning layer

Why Do We Need a Sandbox?

If you think about it, the whole product is basically "run code that nobody has reviewed." The agent writes it, npm install pulls in packages whose install scripts can run pretty much anything, and then a dev server starts executing all of it. There's no way I'm running that on my own server, and you shouldn't either.

So every project gets its own isolated sandbox, which is basically a small VM in the cloud. When I was picking a sandbox provider for this, these are the things I actually needed:

• Suspend and resume: Most projects sit idle most of the time. I wanted to put them to sleep and wake them back up in a couple of seconds, with their memory intact.

• Files, commands, and a terminal: The agent needs to read and write files and run commands, and it's nice to give the user a real shell too.

💁 There's much more to check on when considering using something like this on prod, but these were my only hard requirements.

For this, I'm using Tensorlake sandboxes. To be clear, there's no specific reason to use this particular one. E2B, Daytona, Modal, or even your own Firecracker setup would all work, and you're free to choose whatever you prefer. I've just been using it for a few projects already, and for sandboxes it works perfectly, especially for a use case like ours.

Note: All the sandbox code lives in a single package (packages/sandbox). So if you want to switch providers, that's pretty much the only place you need to touch. The rest of the app doesn't even know which provider it's talking to.

How to Set Up the Project

Before you start, make sure you have the following installed:

• Node.js 22 or newer

• pnpm

• Docker (for Postgres and MinIO if you plan to test it locally first)

You'll also need API keys for your sandbox provider and for an LLM (Anthropic or OpenAI, whichever you like).

Start by cloning the repository and installing the dependencies:

git clone https://github.com/shricodev/lovable-build-tensorlake-aws.git
cd lovable-build-tensorlake-aws
pnpm install

Next, create your environment file and fill in the keys:

cp .env.example .env
# Add your sandbox and LLM keys, and generate AUTH_SECRET with:
openssl rand -base64 32

Now start the local infrastructure, create the database tables, and set up the storage bucket:

# Start Postgres and MinIO
pnpm infra:up

# Create the database tables
pnpm db:migrate

# Create the bucket and lock it down
pnpm s3:setup

Then build the base snapshot that every new project starts from (more on this in a bit). It takes about 45 seconds:

pnpm sandbox:build-base

Finally, start everything:

# web on :3000, preview gateway on :4000
pnpm dev

Open http://localhost:3000, sign in, and describe an app. That's it! 🎉

Note: The app supports GitHub sign in, but it also has a simple dev login so you can try it out before creating a GitHub OAuth app. Don't worry, the dev login is always disabled in production.

Core Components in the Application

The project is huge. Walking through every single line would turn this into an hours long read, so instead I'll focus on the core components that actually make the system work. Things like the dashboard, sign in, and the code editor are pretty standard stuff, so I'll skip those.

Note: This means that the code snippets below are trimmed down to the important parts. You can find the complete code in the repository.

Starting a Sandbox Fast

Every generated app starts from the same template: Vite, React, TypeScript, Tailwind, and a few common libraries. The naÏve way of doing it looks something like this for every new project:

1. Create a fresh sandbox

2. Upload the template

3. Run npm install

4. Start the dev server

And that works, but it took 33.4 seconds before the preview was even reachable in my tests. Most of that (about 23 seconds) was just npm install running on a single vCPU. I don't know about you, but I'm not staring at a spinner for that long every time I start a new project.

The fix is to do all of that once, and save the result as a memory snapshot. A memory snapshot captures the files, the RAM, and the running processes. So when you restore it, the dev server is already running. Nothing has to boot again.

Here's the core of the base snapshot builder:

export async function buildBaseSnapshot(log: Logger) {
  // Cold path, done once: create, upload template, npm install, git init,
  // verify it builds, start the dev server and warm up Vite's cache.
  const { ps } = await coldCreateFromTemplate({
    name: `base-${Date.now()}`,
    log,
    verify: true,
  });

  try {
    // Memory checkpoint: files + RAM + running processes.
    const snapshotId = await ps.checkpoint();
    writeBaseSnapshot({ snapshotId, createdAt: new Date().toISOString() });
  } finally {
    await ps.terminate();
  }
}

With that in place, creating a sandbox for a new project is just a restore, and then we lock it down:

static async createFromSnapshot(opts: { snapshotId: string; name: string; log: Logger }) {
  const sb = await Sandbox.create({ snapshotId: opts.snapshotId, name: opts.name, timeoutSecs: 600 });
  const ps = new ProjectSandbox(sb, opts.log);

  await sb.update({
    exposedPorts: [5173],              // the Vite dev server, reachable through the proxy
    allowUnauthenticatedAccess: false, // the port URL is never public
    network: {
      allowInternetAccess: true,
      allowOut: ["registry.npmjs.org"], // npm and nothing else
      denyOut: [],
    },
  });
  return ps;
}

A few things worth noting here:

• exposedPorts makes the dev server reachable through the provider's proxy, but only if you have our API key. We'll use that later in the gateway.

• The network block is an allow list. Once allowOut has an entry in it, everything else is blocked.

  💁 I actually tested this with example.com, github.com, and the cloud metadata IP, and all of them were blocked while npm still worked just fine.

• There are no secrets in the sandbox environment at all. No LLM keys, no database URL, no AWS keys, nothing. So even if a generated app or some prompt injection tries to steal something, there's simply nothing in there to steal.

Here are the numbers I got, measured all the way until the preview actually loads:

That's roughly 8 times faster. How cool is that? 😎

The Agent Loop

The agent loop is the brain of the entire system. Every time a user sends a prompt, this is what runs.

The idea is simple even if the implementation isn't. You give the LLM a goal and some tools, let it call them, feed the results back, and repeat until it's done.

These are the tools the agent gets:

• list_files, read_file, write_file, edit_file, and delete_file

• run_command for quick checks like npx tsc --noEmit

• install_packages for adding npm packages

• get_dev_server_logs and get_browser_errors for debugging

• finish, which the agent calls when it thinks it's done

Each tool is just a small file with a Zod schema and a run function. Here's edit_file for example:

export const editFile = defineTool({
  name: "edit_file",
  description:
    "Replace one exact snippet in a file. `search` must match exactly and occur exactly once.",
  schema: z.object({
    path: z.string(),
    search: z.string().min(1),
    replace: z.string(),
  }),
  async run({ path, search, replace }, { sandbox }) {
    const text = await sandbox.readFile(path);
    const count = text.split(search).length - 1;
    if (count !== 1) {
      return {
        isError: true,
        content: `search text occurs ${count} times in ${path}`,
      };
    }
    await sandbox.writeFile(
      path,
      text.replace(search, () => replace),
    );
    return { content: `Edited ${path}`, changedFiles: [path] };
  },
});

The Zod schema is doing two jobs here. It gets converted to JSON Schema for the LLM (with z.toJSONSchema), and it also validates whatever the model sends back before anything touches the sandbox. If the input is invalid, the error just goes back to the model instead of crashing the whole run.

Then the actual loop runs:

while (true) {
  if (signal.aborted) return result("cancelled");

  const res = await llm.chat({
    system: SYSTEM_PROMPT,
    messages,
    tools,
    signal,
    onText,
  });
  messages.push(res.message);

  const results = [];
  for (const call of res.message.toolCalls) {
    const out = await executeTool(call.name, call.input, ctx); // validate + run
    results.push({
      type: "tool_result",
      toolCallId: call.id,
      content: out.content,
      isError: out.isError,
    });
    if (out.finish) finishCalled = true;
  }

  if (!finishCalled) {
    messages.push({ role: "user", content: results });
    continue;
  }

  // The agent says it's done. Now we check. (next section)
}

The llm.chat call is a small wrapper I wrote that supports both Anthropic and OpenAI with the same interface, so you can switch models from a dropdown in the UI.

On the Anthropic side, prompt caching is turned on, and to be honest, it does most of the heavy lifting on cost. A typical turn reads over 120K tokens, and almost all of them come straight from the cache.

Keeping File Paths Safe

Now this one's a bit sneaky, and I only found it because I was poking around. The sandbox file API blocks paths with .. in them, but it happily follows symlinks. So if the generated code creates a leak.txt that points to /etc/passwd, reading leak.txt gives you back the password file. Not great.

So every file tool resolves the real path inside the sandbox before touching anything:

async safePath(relPath: string) {
  const abs = resolveProjectPath(relPath); // rejects "..", absolute paths, NUL bytes
  const r = await this.sb.run("realpath", { args: ["-m", "--", abs] });
  const real = r.stdout.trim();
  if (!isInsideApp(real)) throw new SandboxPathError(relPath, "resolves outside the project");
  return real;
}

realpath -m follows every symlink and tells you where the path actually points to. If that ends up outside the project folder, the call just fails.

Keeping the Context Small

Instead of replaying every past tool call on each new prompt, every turn starts a fresh conversation with:

• a one line summary of each earlier turn (what the user asked, and what the agent did)

• the file tree and the list of installed packages

• the current src/App.tsx

The agent reads anything else it needs with its tools. This keeps the cost of a turn pretty much flat, even after a project has had 20 prompts.

Letting the Agent Check Its Own Work

LLMs are really weird. They'll happily tell you everything works when the build is basically on fire. So when the agent calls finish, we don't just take its word for it. We run three checks inside the sandbox:

export async function runChecks(sandbox: ProjectSandbox) {
  const tsc = await sandbox.exec("npx tsc --noEmit -p . 2>&1", {
    timeoutSecs: 120,
  });
  const build = await sandbox.exec(
    "npx vite build --outDir /tmp/build --emptyOutDir --logLevel error 2>&1",
    { timeoutSecs: 180 },
  );
  const render = await renderCheck(sandbox); // renders the app once in a fake DOM

  return {
    ok: tsc.exitCode === 0 && build.exitCode === 0 && render.ok,
    typecheck: { ok: tsc.exitCode === 0, output: tsc.stdout },
    build: { ok: build.exitCode === 0, output: build.stdout },
    render,
  };
}

The first check runs TypeScript’s type checker to catch type errors without generating any files. The second runs a full Vite production build to make sure the app can actually compile successfully.

The third one is where it gets interesting. A lot of bugs only show up at runtime, stuff like Cannot read properties of undefined (reading 'map'). The usual answer to this is a headless browser, but that's heavy and slow in a small sandbox, and I really didn't want to go down that road.

So instead, the template ships a tiny script that renders the app once with happy-dom (a fake DOM for Node.js) and Vite's ssrLoadModule:

GlobalRegistrator.register({ url: "http://localhost:5173/" });
document.body.innerHTML = '<div id="root"></div>';
console.error = (...args) => errors.push(args.join(" "));

await server.ssrLoadModule("/src/main.tsx"); // runs the real app entry
await new Promise((r) => setTimeout(r, 1500)); // let React render

const rendered = document.getElementById("root").innerHTML.trim().length > 0;
console.log(
  JSON.stringify({ ok: rendered && errors.length === 0, rendered, errors }),
);

It takes about 2 seconds, and it caught the undefined.map crash along with the exact line in App.tsx. Noiceee!

If any of the checks fail, the errors go straight back to the agent as a new message, and it gets another round to fix them:

lastCheck = await runChecks(sandbox);
if (lastCheck.ok) return result("succeeded");
if (healRounds >= maxHeal)
  return result("failed", { error: describeFailures(lastCheck) });

healRounds++;
messages.push({
  role: "user",
  content: [
    ...results,
    {
      type: "text",
      text: `Verification failed. Fix these problems, then call finish again.\n\n${describeFailures(lastCheck)}`,
    },
  ],
});

It stops after 3 rounds by default. If it's still broken after that, the user gets an honest "I couldn't finish this one" with the actual errors.

Streaming Progress to the Browser

The agent runs in the worker, but the user is looking at the browser. So somehow every step ("Wrote src/App.tsx", "Ran npx tsc --noEmit", "Checks passed" and all) has to get from one to the other as it happens.

The worker writes each step as a row in a run_events table and then fires a Postgres NOTIFY:

export async function appendEvent(
  db: Db,
  e: { projectId: string; runId?: string; type: string; payload: unknown },
) {
  const [row] = await db
    .insert(runEvents)
    .values(e)
    .returning({ id: runEvents.id });
  await db.$client.notify(
    "events",
    JSON.stringify({ projectId: e.projectId, id: row.id }),
  );
  return row.id;
}

On the web side, a Next.js route handler streams these events to the browser with Server Sent Events. When the browser connects, it first replays everything from the active run, and then it just keeps listening for new rows:

const pump = async () => {
  const rows = await db
    .select()
    .from(runEvents)
    .where(and(eq(runEvents.projectId, project.id), gt(runEvents.id, cursor)))
    .orderBy(asc(runEvents.id));

  for (const r of rows) {
    cursor = r.id;
    send(`id: ${r.id}\ndata: ${JSON.stringify(r)}\n\n`);
  }
};

bus.on(project.id, pump); // fired by a single LISTEN connection per process
pump(); // replay first

Since all the events live in the database, refreshing the page in the middle of a run doesn't lose anything. The browser reconnects, replays the run so far, and continues from where it left off. The Stop button works the same way, just in reverse. The web app sends a NOTIFY with the run ID, and whichever worker is holding that run aborts it.

Note: Streamed text from the LLM comes in token by token, and saving every token would mean hundreds of rows per turn. So the worker buffers it and writes one row every 250ms instead. I also had a small bug here where events landed out of order because each write was its own promise. Pushing every write through a single promise chain fixed it.

The Live Preview Gateway

The generated app's dev server runs inside the sandbox on port 5173, and the provider exposes it at a URL like https://5173-<sandbox-id>.sandbox.example. Now, you could just put that URL in an iframe and call it a day, but there are two problems with that:

1. To load it, the browser would need our sandbox API key. Yeah, that's clearly not happening.

2. And if we made it public instead, anyone who guessed the URL could open it, and we'd have no control over who sees what.

So we put our own small gateway in front of it. It maps <project-id>.preview.localhost:4000 to the right sandbox and adds the API key on the server side:

const proxy = createProxyServer({ changeOrigin: true, secure: true, ws: true });

const server = http.createServer(async (req, res) => {
  const projectId = HOST_RE.exec(req.headers.host ?? "")?.[1];
  const target = await resolve(projectId); // project -> sandbox, cached for a few seconds
  if (!target.sandboxId) return send(res, noPreviewPage());

  delete req.headers.cookie; // never forward the visitor's credentials
  delete req.headers.authorization;
  proxy.web(req, res, {
    target: previewUrlFor(target.sandboxId),
    headers: { authorization: `Bearer ${API_KEY}` },
  });
});

// Vite's hot reload runs over a WebSocket, so upgrades get proxied too.
server.on("upgrade", async (req, socket, head) => {
  const target = await resolve(HOST_RE.exec(req.headers.host ?? "")?.[1]);
  proxy.ws(req, socket, head, {
    target: previewUrlFor(target.sandboxId).replace(/^https/, "wss"),
    headers: { authorization: `Bearer ${API_KEY}` },
  });
});

That upgrade handler is what makes the preview feel alive. Whenever the agent edits a file, Vite pushes the change over the WebSocket, and the preview updates without a reload.

There's also another reason for having the gateway that's pretty easy to miss. The preview runs on a different origin (*.preview.localhost) than the main app (localhost:3000). So a generated app can never read the main app's cookies or call its API as the logged in user. And since *.localhost resolves to 127.0.0.1 in modern browsers, you don't even need to touch your hosts file for this.

Note: The template also injects a tiny script into the preview that listens for window.onerror, and posts them to the parent window. The workspace then forwards those to the backend, and that's where the agent's get_browser_errors tool gets real runtime errors from.

Versions with Git

Every finished prompt becomes a version. So, the user can pretty much revert to a specific "prompt", more like git reset.

For this, plain old Git inside the sandbox works great. The base snapshot already has a repo with the template as the first commit, and after each successful turn, the worker commits everything:

export async function commitAll(ps: ProjectSandbox, message: string) {
  const out = await git(
    ps,
    `git add -A
if git diff --cached --quiet; then
  echo NOCHANGE
else
  git commit -q -m "$MSG"
  git rev-parse HEAD
  git show --name-only --format= HEAD
fi`,
    { MSG: message },
  );
  if (out.trim() === "NOCHANGE") return null;
  const [sha, ...files] = out.trim().split("\n");
  return { sha, files };
}

Note: You might be wondering why there's no exit 0 in there. Commands run under bash -l, and an explicit exit in a login shell runs ~/.bash_logout, whose last command failed on this image and turned my exit code into a failure. This one honestly took me way longer to figure out. 😭

Restoring never rewrites history. It makes the files match the old commit exactly, and then commits that as a brand new version on top. So "restore version 1" creates version 5, and you can still go back to version 4 whenever you want. Nothing gets lost.

Keeping the History Durable (Without Tokens in the Sandbox)

Git inside the sandbox is great, but it only lives as long as the sandbox does. So after each version, the history also gets pushed to a hosted Git repository.

The straightforward way to do this would be to give the sandbox a Git token and just run git push. But the tokens I had access to were scoped to the whole project, not a single repo. Putting one of those inside a sandbox full of untrusted code? Yeah, no thanks.

So the sandbox never pushes anything. It creates a git bundle (basically the entire repo in a single file), the worker reads that file out, and the worker does the push itself:

export async function pushToHostedGit(
  ps: ProjectSandbox,
  repo: string,
  dataDir: string,
) {
  await git(ps, "git bundle create -q /tmp/repo.bundle main");
  const bundle = await ps.sb.readFile("/tmp/repo.bundle");

  const mirror = join(dataDir, "git", `${repo}.git`); // a bare repo on the worker
  writeFileSync(join(mirror, "incoming.bundle"), bundle);
  await run("git", [
    "-C",
    mirror,
    "fetch",
    "-q",
    "--force",
    "incoming.bundle",
    "+refs/heads/main:refs/heads/main",
  ]);

  const cred = await repos.credential(repo); // short lived token, only on the worker
  await run("git", [
    "-C",
    mirror,
    "-c",
    `http.extraHeader=Authorization: Basic ${basic(cred)}`,
    "push",
    "-q",
    "--force",
    url,
    "main",
  ]);
}

A nice side effect of this is that it also powers remix. When someone copies a shared project, the worker loads the source project's bundle from the hosted repo into a brand new sandbox, and the original sandbox doesn't even have to wake up for it.

Sleeping, Waking, and Sharing the Sandboxes

A running sandbox costs money even when nobody's working on it. So there's a small job that runs every minute and suspends any sandbox that hasn't had an agent run for 10 minutes. Suspending keeps the memory, so the dev server comes back exactly as it was.

Waking up happens in the gateway. If someone opens the preview of a sleeping project, the gateway shows a small "Waking up your app..." page that keeps refreshing itself, and resumes the sandbox in the background:

if (isPage && target.status === "suspended") {
  const outcome = wake(projectId, target, log); // deduplicated per project
  const done = await Promise.race([outcome, timeout(4000, "pending")]);
  if (done === "busy") return send(res, busyPage());
  if (done !== "running") return send(res, wakingPage()); // auto refreshes
}

In my tests, a sleeping sandbox woke up in about 3 seconds, and the app loaded right after. 🎊

Sharing a Limited Number of Sandboxes

Most sandbox providers limit how many sandboxes you can run at the same time, especially on a free plan. So I added a small concurrency checker, and it all happens in Postgres with an advisory lock, so two workers never end up making the same decision at the same time:

async function decide(db: Db, projectId: string) {
  return db.transaction(async (tx) => {
    await tx.execute(sql`select pg_advisory_xact_lock(${LOCK_KEY})`);

    const live = await runningSandboxesOldestFirst(tx);
    if (live.some((s) => s.projectId === projectId)) return { kind: "ok" };
    if (live.length < concurrencyLimit()) return { kind: "ok" };

    // Full. Free a slot by suspending the least recently used idle sandbox.
    const victim = live.find((s) => !busyProjects.has(s.projectId));
    if (victim) return { kind: "evict", sandboxId: victim.sandboxId };

    return { kind: "wait", position }; // everything is busy, just wait...
  });
}

To test this, I set the limit to 1 and sent a prompt to one project while another project's sandbox was just sitting idle. The idle one went to sleep, and the new one took its slot. Then I sent a prompt to the first project while the second was still working, and the UI showed "Waiting for a free sandbox, #1 in line" until the slot freed up. If you have a bigger plan, you just raise the limit in .env and nothing else changes.

Publishing the App

The preview is great while you're building, but you don't want your published app to depend on a sandbox that goes to sleep every 10 minutes. 🫩

So publishing builds the app once and turns it into plain static files:

export async function publish(project: Project) {
  const ps = await projectSandbox(project.id); // wakes it if needed
  const build = await ps.exec(
    "rm -rf dist && npx vite build --outDir dist --emptyOutDir 2>&1",
    { timeoutSecs: 180 },
  );
  if (build.exitCode !== 0)
    throw new HttpError(422, `The build failed:\n${build.stdout.slice(-1500)}`);

  const files = await listFiles(ps, "dist");
  const prefix = `published/${slug}/${versionId}/`;

  for (const f of files) {
    const bytes = await ps.readBytes(`dist/${f}`);
    await storage.put(prefix + f, bytes, contentTypeFor(f), cacheControlFor(f));
  }

  await savePublishedSite({ projectId: project.id, slug, s3Prefix: prefix });
  return { url: publishedUrl(slug) };
}

The gateway then serves those files from S3 at http://<slug>.app.localhost:4000. Files that Vite hashes (like assets/index-CScgwd68.js) get cached forever, and index.html always gets revalidated, so updates show up right away. Any path without a file extension falls back to index.html, so client side routing works as well.

And published sites get their own subdomain on purpose. If they lived under the main app's domain, a published app's JavaScript could call your API with the cookies of whoever is looking at it, and you really don't want that. 😺

In my tests, publishing took about 6 seconds, and the published site loaded in 4ms with the sandbox asleep, because it never touches the sandbox at all.

App Builder in Action

Here's a quick demo of the app builder in action:

I also ran a small eval out of curiosity. It sends the same three prompts (a habit tracker, a kanban board, and an expense dashboard with charts) to two different models, each in a fresh sandbox:

All six apps built and rendered on the first check, without needing a single fix round, which honestly surprised me a bit. On Claude, each app cost somewhere around $0.15 to $0.20.

Conclusion

So, what do you think of the project? This was truly one of the most fun projects I've worked on in a while since this AI stuff has taken over raw coding. 🤦‍♂️

When you use tools like Lovable, it's easy to think it's all about the prompt and the model. But once you build one yourself, you realize most of the work is everything around the model, like where the code runs, how fast it starts, how you show it to the user, and how you keep it from doing something it shouldn't.

If there's one thing I'd want you to take away from this, it's the sandbox part. Treat AI generated code as untrusted, give it its own small machine with no secrets and almost no network access. Memory snapshots and suspend/resume then take care of making it fast and cheap.

There's still a lot of room to extend this. You could add a small backend (like Hono and SQLite) to the template so users can build full stack apps, let users click an element in the preview to edit exactly that component, or generate a few design variations of the same prompt side by side. I'm just too exhausted to implement that right now. I'll leave it up to you. ✌️

The foundation is there. The rest is just building on top of it.

You can find the complete source code here: shricodev/lovable-build-tensorlake-aws

So, that's it for this article. Thank you so much for reading! See you next time. 🫡

This guide is designed to end that cycle by replacing it with something more useful: a systematic, service-by-service approach to knowing exactly what you're spending, why you're spending it, catching problems before they become invoices, and reducing costs without guesswork.

It's organised as a reference you can return to. Each part is complete on its own: you can go straight to the RDS section if that's your current problem, or follow the guide start to finish if you're building a FinOps practice from scratch. Every command is runnable, and every script is deployable.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: Monitoring — Know Where Every Dollar Goes

• Part 2: Alerting — Catch Spikes Before They Become Invoices

• Part 3: Optimisation by Service

• Part 4: The 30-Day Optimisation Sprint

• Best Practices Summary

• Resources

What You'll Learn

• How to set up Cost and Usage Report querying with Athena, the foundation of all serious cost analysis

• The five dashboards every engineering team needs, built with queries you can run today

• A three-tier alerting strategy that catches cost spikes without creating alert fatigue

• Service-specific optimisation playbooks for EC2, S3, Lambda, RDS, DynamoDB, and data transfer

• A concrete 30-day sprint that produces measurable savings in the first month

Let's build this from the ground up.

Prerequisites

Before following this guide, you should have:

Knowledge:

• Working familiarity with AWS services: EC2, S3, RDS, Lambda, and VPC

• Comfort reading Python and SQL

• Basic understanding of how IAM policies and roles work

Access:

• AWS account with billing access. The IAM user or role you work with needs ce:GetCostAndUsage, ec2:Describe*, rds:Describe*, and s3:GetBucketLifecycleConfiguration permissions.

• AWS CLI v2 configured

• Athena access (for the CUR queries in Part 1)

Setup:

• Enable Cost Explorer if it isn't already. It's free and required for most commands in this guide:

aws ce enable-cost-explorer --region us-east-1

• The Cost and Usage Report (CUR) should be configured and exporting to an S3 bucket. If it isn't yet, the AWS CUR setup guide walks through the process. Give the report 24 hours after setup to generate the first data file.

Part 1: Monitoring — Know Where Every Dollar Goes

1.1 The Cost and Usage Report — Your Source of Truth

Cost Explorer shows you service-level totals. It's useful for trends, but it isn't sufficient for root cause analysis. When you need to know which specific resource is responsible for a $12,000/month line item, you need the Cost and Usage Report queried through Athena.

Create the Athena table over your CUR data:

-- Run this once in Athena after your CUR starts generating data
-- Replace 'your-cur-bucket' and 'your-prefix' with your actual values

CREATE EXTERNAL TABLE IF NOT EXISTS cur_database.billing (
    bill_billing_period_start_date  STRING,
    bill_payer_account_id           STRING,
    line_item_usage_start_date      STRING,
    line_item_resource_id           STRING,
    line_item_usage_type            STRING,
    line_item_usage_amount          DOUBLE,
    line_item_unblended_cost        DOUBLE,
    product_servicecode             STRING,
    product_instance_type           STRING,
    product_region                  STRING,
    resource_tags_user_environment  STRING,
    resource_tags_user_team         STRING,
    resource_tags_user_service      STRING,
    resource_tags_user_owner        STRING
)
PARTITIONED BY (year STRING, month STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LOCATION 's3://your-cur-bucket/your-prefix/'
TBLPROPERTIES ('skip.header.line.count'='1');

MSCK REPAIR TABLE cur_database.billing;

Here are the three queries you should run on day one:

-- Query 1: Top 20 resources by cost this month
-- Run this first. It tells you where to focus.
SELECT
    line_item_resource_id,
    product_servicecode,
    resource_tags_user_team     AS team,
    resource_tags_user_service  AS service,
    SUM(line_item_unblended_cost) AS total_cost_usd
FROM cur_database.billing
WHERE line_item_usage_start_date >= DATE_FORMAT(
    DATE_TRUNC('month', CURRENT_DATE), '%Y-%m-%d'
)
  AND line_item_unblended_cost > 0
GROUP BY 1, 2, 3, 4
ORDER BY total_cost_usd DESC
LIMIT 20;

-- Query 2: Week-over-week cost growth by service
-- Identifies which services are growing fastest — these need investigation
WITH weekly AS (
    SELECT
        DATE_TRUNC('week', CAST(line_item_usage_start_date AS DATE)) AS week,
        product_servicecode,
        SUM(line_item_unblended_cost) AS cost
    FROM cur_database.billing
    WHERE line_item_usage_start_date >=
          DATE_FORMAT(DATE_ADD('day', -42, CURRENT_DATE), '%Y-%m-%d')
    GROUP BY 1, 2
)
SELECT
    curr.product_servicecode AS service,
    ROUND(prev.cost, 2)      AS prev_week_cost,
    ROUND(curr.cost, 2)      AS curr_week_cost,
    ROUND(
        100.0 * (curr.cost - prev.cost) / NULLIF(prev.cost, 0),
        1
    ) AS pct_change
FROM weekly curr
JOIN weekly prev
  ON curr.product_servicecode = prev.product_servicecode
  AND curr.week = DATE_ADD('week', 1, prev.week)
WHERE curr.week = DATE_TRUNC('week', CURRENT_DATE)
  AND ABS((curr.cost - prev.cost) / NULLIF(prev.cost, 0)) > 0.20
ORDER BY pct_change DESC;

-- Query 3: Resources running with no usage (candidates for shutdown)
-- Finds resources that incurred cost but had zero usage quantity
-- in the past 7 days — strong signal for idle or orphaned resources
SELECT
    line_item_resource_id,
    product_servicecode,
    resource_tags_user_owner    AS owner,
    resource_tags_user_team     AS team,
    SUM(line_item_unblended_cost) AS cost_past_7_days
FROM cur_database.billing
WHERE line_item_usage_start_date >=
      DATE_FORMAT(DATE_ADD('day', -7, CURRENT_DATE), '%Y-%m-%d')
  AND line_item_usage_amount = 0
  AND line_item_unblended_cost > 5
GROUP BY 1, 2, 3, 4
ORDER BY cost_past_7_days DESC
LIMIT 30;

1.2 The Five Essential Cost Dashboards

These five views cover the monitoring needs of most engineering teams. Each is built from queries you can run immediately, no third-party tool required.

Dashboard 1: Executive Summary (for weekly leadership updates)

# executive_summary.py
import boto3
from datetime import datetime, timedelta

ce = boto3.client('ce')


def weekly_summary():
    today     = datetime.now()
    start_mtd = today.replace(day=1).strftime('%Y-%m-%d')
    today_str = today.strftime('%Y-%m-%d')

    # Month-to-date spend
    mtd = ce.get_cost_and_usage(
        TimePeriod={'Start': start_mtd, 'End': today_str},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )
    mtd_spend = float(
        mtd['ResultsByTime'][0]['Total']['UnblendedCost']['Amount']
    )

    # End-of-month forecast
    forecast = ce.get_cost_forecast(
        TimePeriod={
            'Start': today_str,
            'End':   (today.replace(day=28) + timedelta(days=4)).replace(day=1).strftime('%Y-%m-%d'),
        },
        Metric='UNBLENDED_COST',
        Granularity='MONTHLY'
    )
    eom_forecast = float(forecast['Total']['Amount']) + mtd_spend

    # Top 5 services
    by_service = ce.get_cost_and_usage(
        TimePeriod={'Start': start_mtd, 'End': today_str},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[{'Type': 'DIMENSION', 'Key': 'SERVICE'}]
    )
    services = sorted(
        [
            (g['Keys'][0], float(g['Metrics']['UnblendedCost']['Amount']))
            for g in by_service['ResultsByTime'][0]['Groups']
        ],
        key=lambda x: x[1],
        reverse=True
    )[:5]

    print(f"\n{'─'*48}")
    print(f"  AWS Cost Summary — {today.strftime('%B %Y')}")
    print(f"{'─'*48}")
    print(f"  Month-to-date:     ${mtd_spend:>12,.2f}")
    print(f"  End-of-month est:  ${eom_forecast:>12,.2f}")
    print(f"\n  Top 5 Services:")
    for name, cost in services:
        short = name.replace('Amazon ', '').replace('AWS ', '')
        print(f"    {short:<32} ${cost:>9,.2f}")
    print(f"{'─'*48}\n")


weekly_summary()

Dashboard 2: Team Cost Breakdown (for engineering leads)

# team_breakdown.py
import boto3
from datetime import datetime

ce = boto3.client('ce')


def team_breakdown():
    start = datetime.now().replace(day=1).strftime('%Y-%m-%d')
    end   = datetime.now().strftime('%Y-%m-%d')

    response = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[
            {'Type': 'TAG',       'Key': 'Team'},
            {'Type': 'DIMENSION', 'Key': 'SERVICE'},
        ]
    )

    by_team = {}
    for group in response['ResultsByTime'][0].get('Groups', []):
        team_raw = group['Keys'][0]
        team     = team_raw.replace('Team$', '') if team_raw else 'untagged'
        service  = group['Keys'][1]
        cost     = float(group['Metrics']['UnblendedCost']['Amount'])

        if team not in by_team:
            by_team[team] = {'total': 0.0, 'by_service': {}}
        by_team[team]['total'] += cost
        by_team[team]['by_service'][service] = (
            by_team[team]['by_service'].get(service, 0.0) + cost
        )

    total_bill = sum(d['total'] for d in by_team.values())

    print(f"\n{'─'*58}")
    print(f"  Team Cost Breakdown — MTD {datetime.now().strftime('%Y-%m-%d')}")
    print(f"  Total: ${total_bill:,.2f}")
    print(f"{'─'*58}")

    for team, data in sorted(by_team.items(), key=lambda x: x[1]['total'], reverse=True):
        pct = (data['total'] / total_bill * 100) if total_bill else 0
        print(f"\n  {team:<20}  ${data['total']:>10,.2f}  ({pct:.1f}%)")
        top3 = sorted(data['by_service'].items(), key=lambda x: x[1], reverse=True)[:3]
        for svc, cost in top3:
            short = svc.replace('Amazon ', '').replace('AWS ', '')
            print(f"    └─ {short:<30} ${cost:>8,.2f}")

    print()


team_breakdown()

Dashboard 3: Waste Detection (for weekly cleanup reviews):

# waste_detector.py
import boto3
from datetime import datetime, timezone, timedelta

ec2  = boto3.client('ec2')
elbv2 = boto3.client('elbv2')
cw   = boto3.client('cloudwatch')


def detect_waste():
    report = {'items': [], 'total_monthly_waste': 0.0}

    # Unattached EBS volumes
    for vol in ec2.describe_volumes(
        Filters=[{'Name': 'status', 'Values': ['available']}]
    )['Volumes']:
        age  = (datetime.now(timezone.utc) - vol['CreateTime']).days
        cost = round(vol['Size'] * 0.08, 2)
        tags = {t['Key']: t['Value'] for t in vol.get('Tags', [])}
        report['items'].append({
            'type':         'Unattached EBS Volume',
            'id':           vol['VolumeId'],
            'detail':       f"{vol['Size']}GB — {age} days old",
            'owner':        tags.get('Owner', '—'),
            'monthly_cost': cost,
        })
        report['total_monthly_waste'] += cost

    # Unassociated Elastic IPs
    for addr in ec2.describe_addresses()['Addresses']:
        if 'AssociationId' not in addr:
            report['items'].append({
                'type':         'Unassociated Elastic IP',
                'id':           addr.get('AllocationId', ''),
                'detail':       addr['PublicIp'],
                'owner':        '—',
                'monthly_cost': 3.60,
            })
            report['total_monthly_waste'] += 3.60

    # Idle load balancers (fewer than 100 requests in 7 days)
    for lb in elbv2.describe_load_balancers()['LoadBalancers']:
        metrics = cw.get_metric_statistics(
            Namespace='AWS/ApplicationELB',
            MetricName='RequestCount',
            Dimensions=[{'Name': 'LoadBalancer',
                         'Value': lb['LoadBalancerArn'].split(':loadbalancer/')[-1]}],
            StartTime=datetime.now() - timedelta(days=7),
            EndTime=datetime.now(),
            Period=604800,
            Statistics=['Sum']
        )['Datapoints']
        total_requests = metrics[0]['Sum'] if metrics else 0

        if total_requests < 100:
            report['items'].append({
                'type':         'Idle Load Balancer',
                'id':           lb['LoadBalancerName'],
                'detail':       f"{int(total_requests)} requests in 7 days",
                'owner':        '—',
                'monthly_cost': 22.0,
            })
            report['total_monthly_waste'] += 22.0

    print(f"\n  Waste Detection Report — {datetime.now().strftime('%Y-%m-%d')}")
    print(f"  Estimated monthly waste: ${report['total_monthly_waste']:.2f}\n")

    for item in sorted(report['items'], key=lambda x: x['monthly_cost'], reverse=True)[:20]:
        print(f"  [{item['type']}]")
        print(f"    ID:     {item['id']}")
        print(f"    Detail: {item['detail']}")
        print(f"    Owner:  {item['owner']}")
        print(f"    Cost:   ${item['monthly_cost']:.2f}/month\n")

    return report


detect_waste()

1.3 Tagging Strategy — The Foundation of All Attribution

Every cost attribution model depends on tags. Teams that skip tagging build dashboards that show totals without explanations. The discipline is in making tagging structural rather than procedural: enforced by infrastructure code, not by reminders in Confluence.

Required tag set:

# terraform/variables.tf
variable "mandatory_tags" {
  description = "Tags applied to every resource in this account"
  type        = map(string)

  validation {
    condition = alltrue([
      contains(keys(var.mandatory_tags), "Environment"),
      contains(keys(var.mandatory_tags), "Team"),
      contains(keys(var.mandatory_tags), "Owner"),
      contains(keys(var.mandatory_tags), "Service"),
    ])
    error_message = "mandatory_tags must include Environment, Team, Owner, and Service."
  }
}

locals {
  common_tags = merge(var.mandatory_tags, {
    ManagedBy    = "terraform"
    LastModified = timestamp()
  })
}

resource "aws_instance" "api_server" {
  ami           = data.aws_ami.amazon_linux_2023.id
  instance_type = "t3.medium"
  tags          = merge(local.common_tags, {Name = "api-server-${var.environment}"})
}

Find and report untagged resources weekly:

#!/usr/bin/env bash
# find_untagged.sh

echo "Untagged EC2 instances (missing Team tag):"
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[?!not_null(Tags[?Key=='Team'].Value|[0])].[InstanceId,InstanceType,LaunchTime]" \
  --output table

echo "Untagged RDS instances:"
aws rds describe-db-instances \
  --query "DBInstances[?!not_null(TagList[?Key=='Team'].Value|[0])].DBInstanceIdentifier" \
  --output table

Part 2: Alerting — Catch Spikes Before They Become Invoices

The typical discovery timeline without proactive alerting: a cost spike happens on the 5th, the monthly invoice arrives on the 20th, someone notices on the 22nd, investigation begins on the 23rd, and two weeks of billed waste can't be recovered. With proactive alerting, discovery happens within hours.

2.1 The Three-Tier Alert Structure

Alert fatigue is as damaging as no alerting. The three-tier model keeps signal high by routing different severity levels to different channels with different response expectations.

# alert_router.py
import boto3
import json
import urllib.request
from enum import Enum

SLACK_INFO_WEBHOOK  = 'https://hooks.slack.com/services/INFO/WEBHOOK'
SLACK_ALERT_WEBHOOK = 'https://hooks.slack.com/services/ALERT/WEBHOOK'
SNS_CRITICAL_TOPIC  = 'arn:aws:sns:us-east-1:YOUR_ACCOUNT:cost-critical'


class AlertTier(Enum):
    INFO     = 1
    WARNING  = 2
    CRITICAL = 3


def route_alert(tier: AlertTier, subject: str, message: str):
    """Send an alert to the appropriate channel for its severity tier."""
    icons   = {AlertTier.INFO: ':information_source:',
               AlertTier.WARNING: ':warning:', AlertTier.CRITICAL: ':rotating_light:'}
    payload = {'text': f"{icons[tier]} *{subject}*\n{message}"}

    if tier == AlertTier.INFO:
        _post_slack(SLACK_INFO_WEBHOOK, payload)
    elif tier == AlertTier.WARNING:
        _post_slack(SLACK_ALERT_WEBHOOK, payload)
        _send_sns(SNS_CRITICAL_TOPIC, subject, f"WARNING: {message}")
    elif tier == AlertTier.CRITICAL:
        _post_slack(SLACK_ALERT_WEBHOOK, payload)
        _send_sns(SNS_CRITICAL_TOPIC, subject, f"CRITICAL: {message}")


def _post_slack(webhook: str, payload: dict):
    req = urllib.request.Request(
        webhook,
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json'}
    )
    urllib.request.urlopen(req)


def _send_sns(topic_arn: str, subject: str, message: str):
    sns = boto3.client('sns')
    sns.publish(TopicArn=topic_arn, Subject=subject[:100], Message=message)

The three tiers and what they respond to: Tier 1 INFO goes to a Slack informational channel for daily cost summaries, weekly trend reports, and tag compliance updates. No action required.

Tier 2 WARNING goes to a Slack alert channel plus email for budget above 75% utilisation, 25% week-over-week increases, and expiring Savings Plans. Acknowledge within 24 hours.

Tier 3 CRITICAL goes to PagerDuty plus SMS for budget above 90% utilisation, 100% increase in 24 hours, crypto mining detected, and projected overspend above 120% of plan. Investigate within 1 hour.

2.2 Real-Time Budget Monitor

AWS Budgets sends alerts once daily by default. A daily window means a cost spike that begins at 08:00 isn't caught until the next day's alert fires. The Lambda below runs hourly and checks both absolute budget utilisation and hour-over-hour rate of change.

# budget_monitor.py
# Lambda triggered by EventBridge every hour

import boto3
from datetime import datetime, timedelta
from alert_router import route_alert, AlertTier

ce      = boto3.client('ce')
budgets = boto3.client('budgets', region_name='us-east-1')
ACCOUNT_ID  = boto3.client('sts').get_caller_identity()['Account']
BUDGET_NAME = 'monthly-infrastructure'


def get_mtd_spend() -> float:
    start = datetime.now().replace(day=1).strftime('%Y-%m-%d')
    end   = datetime.now().strftime('%Y-%m-%d')
    r = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )
    return float(r['ResultsByTime'][0]['Total']['UnblendedCost']['Amount'])


def get_budget_limit() -> float:
    r = budgets.describe_budget(AccountId=ACCOUNT_ID, BudgetName=BUDGET_NAME)
    return float(r['Budget']['BudgetLimit']['Amount'])


def get_hourly_costs(hours: int = 4) -> list:
    """Return hourly cost totals for the last N hours."""
    end   = datetime.now()
    start = end - timedelta(hours=hours)
    r = ce.get_cost_and_usage(
        TimePeriod={'Start': start.strftime('%Y-%m-%d'), 'End': end.strftime('%Y-%m-%d')},
        Granularity='HOURLY',
        Metrics=['UnblendedCost']
    )
    return [
        float(period['Total']['UnblendedCost']['Amount'])
        for period in r['ResultsByTime']
    ]


def lambda_handler(event, context):
    mtd_spend    = get_mtd_spend()
    budget_limit = get_budget_limit()
    utilisation  = mtd_spend / budget_limit * 100

    days_elapsed  = datetime.now().day
    projected_eom = (mtd_spend / days_elapsed) * 30
    projected_pct = projected_eom / budget_limit * 100

    if utilisation >= 90:
        route_alert(
            AlertTier.CRITICAL,
            f'Budget at {utilisation:.0f}%',
            f'MTD spend ${mtd_spend:,.2f} is {utilisation:.0f}% of ${budget_limit:,.0f} budget. '
            f'Projected EOM: ${projected_eom:,.2f}.'
        )
    elif utilisation >= 75:
        route_alert(
            AlertTier.WARNING,
            f'Budget at {utilisation:.0f}%',
            f'MTD spend ${mtd_spend:,.2f} is {utilisation:.0f}% of ${budget_limit:,.0f} budget. '
            f'Projected EOM: ${projected_eom:,.2f}.'
        )

    # Check hourly spike
    hourly = get_hourly_costs(hours=4)
    if len(hourly) >= 2:
        last_hour = hourly[-1]
        prev_avg  = sum(hourly[:-1]) / len(hourly[:-1])
        if prev_avg > 0.10 and last_hour > prev_avg * 1.5:
            route_alert(
                AlertTier.WARNING,
                'Hourly cost spike detected',
                f'Last hour: ${last_hour:.2f} vs prior 3-hour avg ${prev_avg:.2f} '
                f'(+{(last_hour/prev_avg - 1)*100:.0f}%)'
            )

    return {
        'mtd_spend':       round(mtd_spend, 2),
        'utilisation_pct': round(utilisation, 1),
        'projected_eom':   round(projected_eom, 2),
    }

Part 3: Optimisation by Service

3.1 EC2 — Seven Levers in Priority Order

EC2 is the largest line item in most AWS accounts and the one with the most optimisation options. Work through these levers in order, as each one lowers the baseline that the next lever acts on.

Lever 1: Find truly idle instances (CPU below 1% for 14 days).

# ec2_idle_finder.py
import boto3
from datetime import datetime, timedelta

ec2 = boto3.client('ec2')
cw  = boto3.client('cloudwatch')


def find_idle_instances(avg_cpu_threshold: float = 1.0, days: int = 14):
    instances = [
        inst
        for r in ec2.describe_instances(
            Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
        )['Reservations']
        for inst in r['Instances']
    ]

    idle = []
    for inst in instances:
        iid   = inst['InstanceId']
        stats = cw.get_metric_statistics(
            Namespace='AWS/EC2',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'InstanceId', 'Value': iid}],
            StartTime=datetime.utcnow() - timedelta(days=days),
            EndTime=datetime.utcnow(),
            Period=days * 86400,
            Statistics=['Average']
        )['Datapoints']

        avg_cpu = stats[0]['Average'] if stats else 0.0
        if avg_cpu < avg_cpu_threshold:
            tags = {t['Key']: t['Value'] for t in inst.get('Tags', [])}
            idle.append({
                'instance_id':   iid,
                'instance_type': inst['InstanceType'],
                'avg_cpu':       round(avg_cpu, 2),
                'environment':   tags.get('Environment', '—'),
                'owner':         tags.get('Owner', '—'),
            })

    return sorted(idle, key=lambda x: x['avg_cpu'])


for inst in find_idle_instances():
    print(f"  {inst['instance_id']}  {inst['instance_type']}  "
          f"{inst['avg_cpu']}% CPU  env:{inst['environment']}  owner:{inst['owner']}")

Lever 2: Right-size over-provisioned instances (CPU below 20%, sustained).

Use the same script with avg_cpu_threshold=20.0. These are right-sizing candidates, not shutdown candidates.

Lever 3: Schedule dev and staging shutdowns using EventBridge rules on AutoShutdown=true tagged instances.

Lever 4: Purchase Savings Plans only after completing levers 1–3.

Lever 5: Migrate to Graviton (20% cheaper, same performance for most workloads).

Lever 6: Use Spot for fault-tolerant batch and development workloads.

Lever 7: Migrate containerised workloads to EKS with Karpenter for automatic bin-packing.

Spot savings estimate:

# spot_price_analyser.py
import boto3

ec2 = boto3.client('ec2')


def spot_savings_estimate(instance_type: str) -> dict:
    spot_history = ec2.describe_spot_price_history(
        InstanceTypes=[instance_type],
        ProductDescriptions=['Linux/UNIX'],
        MaxResults=1
    )['SpotPriceHistory']
    spot_price = float(spot_history[0]['SpotPrice']) if spot_history else 0

    on_demand_approx = {
        't3.medium': 0.0416, 'm5.large': 0.096,
        'c5.xlarge': 0.17,   'r5.2xlarge': 0.504,
    }
    od_price    = on_demand_approx.get(instance_type, 0)
    savings_pct = ((od_price - spot_price) / od_price * 100) if od_price else 0

    return {
        'instance_type': instance_type,
        'spot_price':    round(spot_price, 4),
        'on_demand':     od_price,
        'savings_pct':   round(savings_pct, 1),
        'monthly_spot':  round(spot_price * 730, 2),
        'monthly_od':    round(od_price * 730, 2),
    }


for itype in ['t3.medium', 'm5.large', 'c5.xlarge']:
    r = spot_savings_estimate(itype)
    print(f"  {r['instance_type']:<15} Spot: ${r['spot_price']}/hr  "
          f"OD: ${r['on_demand']}/hr  Savings: {r['savings_pct']}%")

3.2 S3 — Lifecycle Policies and Storage Class Selection

S3 optimisation has two components: moving infrequently accessed data to cheaper storage classes via lifecycle policies, and eliminating waste patterns like incomplete multipart uploads.

# s3_lifecycle_applier.py
import boto3

s3 = boto3.client('s3')

LOG_POLICY = {
    'Rules': [{
        'ID': 'standard-tiering',
        'Status': 'Enabled',
        'Filter': {'Prefix': ''},
        'Transitions': [
            {'Days': 30,  'StorageClass': 'STANDARD_IA'},
            {'Days': 90,  'StorageClass': 'GLACIER_IR'},
            {'Days': 365, 'StorageClass': 'DEEP_ARCHIVE'},
        ],
        'Expiration': {'Days': 2555},
        'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 7},
    }]
}

TEMP_POLICY = {
    'Rules': [{
        'ID': 'temp-data-retention',
        'Status': 'Enabled',
        'Filter': {'Prefix': ''},
        'Expiration': {'Days': 30},
        'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 1},
    }]
}

for bucket in s3.list_buckets()['Buckets']:
    name = bucket['Name']
    try:
        s3.get_bucket_lifecycle_configuration(Bucket=name)
        print(f"  {name} — policy already exists, skipping")
    except s3.exceptions.ClientError:
        policy = TEMP_POLICY if any(k in name for k in ['temp', 'build', 'cache']) else LOG_POLICY
        s3.put_bucket_lifecycle_configuration(
            Bucket=name, LifecycleConfiguration=policy
        )
        print(f"  {name} — applied {'TEMP' if policy is TEMP_POLICY else 'LOG'} policy")

3.3 RDS — Five Optimisation Levels

Find over-provisioned RDS instances:

# rds_rightsizer.py
import boto3
from datetime import datetime, timedelta

rds = boto3.client('rds')
cw  = boto3.client('cloudwatch')


def find_oversized_rds():
    instances  = rds.describe_db_instances()['DBInstances']
    candidates = []

    for inst in instances:
        iid    = inst['DBInstanceIdentifier']
        iclass = inst['DBInstanceClass']

        stats = cw.get_metric_statistics(
            Namespace='AWS/RDS',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'DBInstanceIdentifier', 'Value': iid}],
            StartTime=datetime.utcnow() - timedelta(days=14),
            EndTime=datetime.utcnow(),
            Period=1209600,
            Statistics=['Average', 'Maximum']
        )['Datapoints']

        if not stats:
            continue

        avg_cpu = stats[0]['Average']
        max_cpu = stats[0]['Maximum']

        if avg_cpu < 20 and max_cpu < 50:
            candidates.append({
                'id':       iid,
                'class':    iclass,
                'avg_cpu':  round(avg_cpu, 1),
                'max_cpu':  round(max_cpu, 1),
                'engine':   inst['Engine'],
            })

    return candidates


for c in find_oversized_rds():
    print(f"  {c['id']}  {c['class']}  avg:{c['avg_cpu']}%  max:{c['max_cpu']}%  engine:{c['engine']}")

3.4 DynamoDB — On-Demand vs Provisioned Decision

On-demand is convenient but can be 3–5× more expensive than provisioned for predictable workloads. Provisioned with auto-scaling covers most cases at significantly lower cost.

# dynamodb_mode_advisor.py
import boto3
from datetime import datetime, timedelta

dynamodb = boto3.client('dynamodb')
cw       = boto3.client('cloudwatch')


def analyse_table_billing(table_name: str) -> dict:
    table = dynamodb.describe_table(TableName=table_name)['Table']
    mode  = table.get('BillingModeSummary', {}).get('BillingMode', 'PROVISIONED')

    stats = {}
    for metric in ['ConsumedReadCapacityUnits', 'ConsumedWriteCapacityUnits']:
        data = cw.get_metric_statistics(
            Namespace='AWS/DynamoDB',
            MetricName=metric,
            Dimensions=[{'Name': 'TableName', 'Value': table_name}],
            StartTime=datetime.utcnow() - timedelta(days=30),
            EndTime=datetime.utcnow(),
            Period=86400,
            Statistics=['Average', 'Maximum']
        )['Datapoints']
        if data:
            avg  = sum(d['Average'] for d in data) / len(data)
            peak = max(d['Maximum'] for d in data)
            stats[metric] = {'avg': round(avg, 1), 'peak': round(peak, 1)}

    if mode == 'PAY_PER_REQUEST':
        avg_rcu = stats.get('ConsumedReadCapacityUnits', {}).get('avg', 0)
        avg_wcu = stats.get('ConsumedWriteCapacityUnits', {}).get('avg', 0)

        if avg_rcu > 2000 or avg_wcu > 500:
            return {
                'table':          table_name,
                'current_mode':   'PAY_PER_REQUEST',
                'recommendation': 'Switch to PROVISIONED with auto-scaling',
                'reason': f'Avg {avg_rcu:.0f} RCU/s and {avg_wcu:.0f} WCU/s — predictable pattern',
            }

    return {'table': table_name, 'current_mode': mode, 'recommendation': 'No change needed'}


for table in dynamodb.list_tables()['TableNames']:
    r = analyse_table_billing(table)
    if r['recommendation'] != 'No change needed':
        print(f"  {r['table']}: {r['recommendation']}")

3.5 Data Transfer — The Three Main Waste Patterns

Data transfer charges are often the most confusing line item on an AWS bill. Here are the the three main patterns and their fixes:

Cross-AZ traffic (most common, most fixable): services in different AZs incur $0.01/GB in each direction. The fix is to use topology-aware routing on Kubernetes Services or ensure your application tier and database tier use the same AZ placement.

NAT Gateway charges for internal AWS traffic: S3, ECR, DynamoDB, and SQS traffic that routes through NAT Gateway incurs $0.045/GB. The fix: VPC endpoints eliminate this entirely.

Inter-region replication that compliance doesn't require: audit your S3 replication rules quarterly against actual compliance requirements.

# Find S3 buckets with active replication
for bucket in $(aws s3api list-buckets --query 'Buckets[*].Name' --output text); do
    result=$(aws s3api get-bucket-replication --bucket "$bucket" 2>&1)
    if ! echo "$result" | grep -q "ReplicationConfigurationNotFoundError"; then
        echo "  $bucket — replication active, verify compliance requirement"
    fi
done

Part 4: The 30-Day Optimisation Sprint

This sprint produces measurable savings in the first month. It's designed for a single engineer with two to four hours per week of dedicated FinOps time.

Week 1 – Visibility: enable CUR, set up the Athena table, run the three day-one queries, screenshot the results as your baseline, tag 100% of running EC2 and RDS instances, and identify your top three cost drivers with a documented hypothesis for each.

Week 2 – Quick wins: deploy the orphaned resource reporter Lambda, apply S3 lifecycle policies to your three largest buckets, run the idle instance finder and stop anything below 1% average CPU with no owner objection, and add VPC endpoints for S3, ECR, and DynamoDB.

Week 3 – Right-sizing: run the EC2 rightsizing analyser, downsize the three highest-confidence candidates (non-production first), run the RDS rightsizing script, and find read replicas serving minimal traffic and decommission them.

Week 4 – Alerting and automation: deploy the hourly budget monitor Lambda, configure the three-tier alert routing, set up the weekly waste reporter, add the Infracost GitHub Action to your infrastructure repository, and schedule a monthly 30-minute FinOps review meeting.

Expected outcome after 30 days: 15–25% reduction in monthly AWS spend, documented evidence of every change, and a recurring process that prevents the same waste from accumulating again.

Best Practices Summary

✅ Do: Set up CUR + Athena before any other monitoring. Cost Explorer is a starting point, while CUR is the source of truth.

✅ Do: Enforce tagging in Terraform or CloudFormation. Process-based tagging decays, but infrastructure-enforced tagging is permanent.

✅ Do: Run the idle instance finder and waste reporter weekly. Waste accumulates continuously. A weekly report keeps the pile small.

✅ Do: Use the three-tier alert model. One alert channel with everything in it creates fatigue and gets muted.

✅ Do: Work through the EC2 optimisation levers in order. Right-sizing before Savings Plans prevents locking in waste at a discount.

✅ Do: Check DynamoDB billing mode against actual usage patterns quarterly.

❌ Don't: Delete untagged resources without investigation. Untagged doesn't mean unused, it means unclaimed.

❌ Don't: Apply aggressive S3 lifecycle policies without auditing access patterns first. Glacier retrieval fees can exceed Standard storage costs if data is accessed more frequently than expected.

❌ Don't: Run the waste reporter Lambda with auto-deletion enabled on its first deployment. Run in report-only mode for two weeks to validate the output before adding deletion logic.

Resources

• AWS Cost and Usage Report Data Dictionary: Column reference for all CUR Athena queries in this guide

• AWS Cost Explorer API Reference: Full reference for the Python boto3 cost queries

• AWS Compute Optimizer: ML-powered right-sizing recommendations, useful as a cross-check against the manual analyser scripts

• Amazon DynamoDB Pricing: The definitive reference for the provisioned vs on-demand cost calculation in Section 3.4

• AWS Instance Scheduler: The official AWS solution for tag-based EC2 and RDS scheduling

• FinOps Foundation Framework: The practitioner framework that defines the Inform, Optimise, Operate cycle this guide implements

• Companion Repository: All scripts, Lambda functions, and Terraform modules from this guide

The order service handles the checkout, saves the order to its database, and then publishes an order.created event to a message queue so every downstream system can react independently.

This is a common and reasonable design, but it has a reliability problem that's easy to miss until something goes wrong in production.

When a customer places an order and the payment goes through, the application needs to do two things: save the order to the database and publish the event to the queue. These are two separate writes to two separate systems, and there's no way to make them share a single atomic transaction. If the process crashes, the network hiccups, or a deployment rolls out between the two writes, one side commits and the other does not. The order sits confirmed on the customer's screen while the warehouse has no idea it exists.

The transactional outbox pattern is the standard solution to this problem. In this article, we'll build it from scratch in Node.js, using PostgreSQL for the order service database, SQS for the queue, and DynamoDB as the fulfillment service's database. For local development, we'll use floci, a free open-source AWS emulator that runs all three with a single Docker container.

What We'll Cover

• Prerequisites

• The Problem with Two Writes

• The Outbox Pattern

• What We'll Build

• Project Setup

• Database Schema

• The Request Handler

• The Relay Worker

• The Consumer

• Running the Whole Thing

• Going to Production

• Conclusion

Prerequisites

To follow along, you should be comfortable with:

• Node.js and async/await

• Database transactions (BEGIN, COMMIT, ROLLBACK)

• The general concept of a message queue

You don't need prior experience with AWS, SQS, or DynamoDB. We'll be running everything locally.

You will need Node.js 20 or later and Docker installed on your machine.

The Problem with Two Writes

The order service scenario from the intro is one place this problem appears, but the same pattern comes up in many other contexts.

A user registers and the app inserts their account record, then sends a message to trigger the welcome email and the onboarding workflow. A file is uploaded and the API writes the metadata to the database, then publishes a message to kick off a processing worker for virus scanning or thumbnail generation. A payment webhook arrives, the handler records it in the database, then notifies downstream services that the payment is confirmed.

In every case, the application needs two writes to succeed together: one to the database and one to a queue or external system. If the second one is lost, the first one has no way of knowing.

If you want a deeper look at what database transactions actually guarantee and where they stop helping, see Beyond Happy Path Engineering: Databases.

The naïve implementation looks straightforward:

await db.query('INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2)', [customerId, amountCents]);
await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify({ customerId, amountCents }) }));

The database write happens first, then the queue write. Under normal conditions this works fine. The problem is what happens when something goes wrong between the two.

If the process crashes, runs out of memory, or gets killed mid-deployment after the database write but before sqs.send is called, the order record exists in the database but no event is ever published. The warehouse, email service, and fraud checker never find out the order happened. From the customer's perspective the order went through. From every downstream system's perspective it doesn't exist.

The failure can also go the other way. If sqs.send succeeds but the database write is later rolled back due to a constraint violation or an error in a subsequent step, you've published an event for an order that doesn't actually exist. A consumer acting on that event may try to fulfill an order with no corresponding record, or charge a customer for something that was never saved.

There's also a timing window even when both writes eventually succeed. Between the database commit and the successful sqs.send, a consumer that queries the database after receiving the event may not find the order yet, depending on transaction isolation and replication lag. These are two separate systems with no shared transaction boundary, and no amount of careful sequencing fully closes the gap.

These aren't edge cases that only happen under extraordinary circumstances. Deploys restart processes mid-request. Out-of-memory kills happen without warning. Networks drop connections at any point. Any of these can interrupt the two-write sequence, and the result is a system that's silently inconsistent with no error logged and no alert fired.

A variation I've seen a few times that looks safer but is actually worse is wrapping both operations in a database transaction:

// PLEASE DO NOT EVER DO THIS
const client = await pool.connect();
await client.query('BEGIN');
await client.query('INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2)', [customerId, amountCents]);
await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify({ customerId, amountCents }) }));
await client.query('COMMIT');

The intent is to make the two writes feel like a unit, but a database transaction has no authority over SQS. The transaction can only roll back database operations. If sqs.send succeeds and then COMMIT fails, the message is already in the queue and can't be taken back. If the process crashes after COMMIT but before the function returns, the transaction committed and the message was sent, but the caller may retry, potentially inserting a duplicate order.

Beyond the correctness problems, this pattern holds an open database connection and any row locks for the entire duration of the SQS network call. SQS is normally fast, but under load, retries, or a degraded queue, that call can take seconds. Every other request trying to read or write the same rows has to wait. In a busy application, this is a reliable way to exhaust the connection pool and bring down unrelated parts of the service.

The Outbox Pattern

The core idea is to stop treating the queue publish as a second write that happens after the database write, and instead make it part of the same database transaction.

Rather than calling sqs.send directly, the application inserts a row into an outbox table in the same transaction as the business record. A separate relay process reads the outbox table and publishes the messages to SQS. On the other end, a consumer receives the messages and writes to its own data store. In our case that is a fulfillment service writing to DynamoDB, completely separate from the order service's PostgreSQL database.

If the transaction rolls back for any reason, the outbox row disappears with it. There's no orphaned message in the queue because the message was never sent. If the application crashes after committing but before the relay runs, the outbox row is still there with status='pending', and the relay will pick it up on its next iteration.

The only guarantee the pattern relies on is the one the database already provides: atomicity within a single transaction.

The relay worker is responsible for the eventual delivery guarantee. It runs on an interval, selects pending rows, publishes them to SQS, and marks them as sent only after SQS confirms receipt. If the relay crashes mid-run, it will reprocess the same rows on the next iteration, which means SQS may receive some messages more than once.

That's why the consumer needs to be idempotent: it must handle receiving the same message twice without creating duplicate fulfillment records. We'll cover how to implement that when we build the consumer.

This separation of concerns is what makes the pattern practical. The request handler commits one atomic database transaction and returns. The relay handles the network call to SQS asynchronously, at its own pace, with its own retry logic, without holding database connections open or blocking request handling. The consumer is fully decoupled from the order service and owns its own data store.

The diagram below illustrates the flow:

What We'll Build

Now let's see the whole thing in practice. We'll implement a simple order placement API. When a customer sends a request to place an order, the order service saves it to PostgreSQL and inserts a row into the outbox table, all in one atomic transaction. A relay worker wakes up periodically, reads the pending outbox rows, and publishes each one as a message to SQS. A separate fulfillment service receives those messages from the queue and creates fulfillment records in DynamoDB.

By the end, you'll have an HTTP endpoint you can call, and you'll be able to verify that placing an order triggers the creation of a fulfillment record in a completely separate database, owned by a completely separate service, without either service ever talking to the other directly.

You can find the complete working code at github.com/gkoos/article-outbox.

Project Setup

Before you can run any code, you need to get floci running so you have local instances of PostgreSQL, SQS, and DynamoDB. You'll also need Node.js 20 or later and Docker installed.

Start by cloning the repository and installing dependencies:

git clone https://github.com/gkoos/article-outbox
cd article-outbox
npm install

Next, start floci. This command pulls the latest floci image and starts a Docker container that exposes a local AWS API endpoint (make sure Docker is running):

npm run floci:start

On Linux and macOS, this just works. On Windows with Docker Desktop, you need to edit the floci:start script in your package.json to change the Docker socket mount from /var/run/docker.sock to //var/run/docker.sock.

The floci container is now listening on port 4566 and can spin up RDS (PostgreSQL), SQS, and DynamoDB instances on demand.

Now provision the AWS resources with a single setup command:

npm run setup

This script creates an RDS PostgreSQL database instance, an SQS queue named orders, and a DynamoDB table named fulfillments. It waits for RDS to become available and then writes a .env file with the correct connection details. The environment variables PG_PORT, SQS_QUEUE_URL, and DYNAMODB_TABLE_NAME now point to the local emulated services.

Finally, create the PostgreSQL tables:

npm run migrate

This creates the orders table and the outbox table in PostgreSQL. You now have a fully functional local environment ready to build against.

Database Schema

The two tables are simple. orders holds the business records: each order has a customer ID, an amount in cents, and a timestamp. The outbox table is the heart of the pattern: it's where the application writes the event that needs to be published.

CREATE TABLE orders (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  customer_id TEXT NOT NULL,
  amount_cents INTEGER NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  created_at TIMESTAMPTZ DEFAULT now(),
  sent_at TIMESTAMPTZ
);

CREATE INDEX ON outbox (status, created_at) WHERE status = 'pending';

The orders table needs nothing special. The outbox table stores the event metadata: what type of event it is (event_type), what data it contains (payload as JSON), and whether it has been sent yet (status).

The status starts as pending. When the relay publishes it to SQS, it will mark it as sent and record the timestamp. The index on (status, created_at) WHERE status = 'pending' lets the relay quickly find the next batch of unsent events without scanning the entire table.

The Request Handler

This is where the pattern starts. The request handler receives an HTTP POST, inserts an order into the database, inserts a corresponding row into the outbox table, and commits everything in a single atomic transaction. The key insight is that neither write succeeds unless both succeed.

const client = await pool.connect();
try {
  await client.query('BEGIN');

  // Insert the order record
  const { rows } = await client.query(
    'INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2) RETURNING *',
    [customerId, amountCents]
  );
  const order = rows[0];

  // Insert the outbox record in the same transaction
  await client.query(
    `INSERT INTO outbox (event_type, payload)
     VALUES ($1, $2)`,
    ['order.created', JSON.stringify({ orderId: order.id, customerId: order.customer_id, amountCents: order.amount_cents, createdAt: order.created_at })],
  );

  await client.query('COMMIT');
  res.status(201).json(order);
} catch (err) {
  await client.query('ROLLBACK');
  next(err);
} finally {
  client.release();
}

The handler gets customerId and amountCents from the request body, starts an explicit transaction with BEGIN, and inserts the order. Then it inserts an outbox row with the order data as the payload.

Everything commits atomically. If anything fails, everything rolls back and the client gets an error. If the process crashes between the commit and the response, the client won't get a 201, but the order and the outbox row are still safely committed to the database and the relay will eventually pick it up. The handler doesn't call SQS at all. That is the relay's job.

The Relay Worker

The relay worker is a separate process that polls the outbox table every second and publishes pending rows to SQS. It runs independently of the HTTP server and has no shared state with it.

async function relay() {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    const { rows } = await client.query(`
      SELECT *
      FROM outbox
      WHERE status = 'pending'
      ORDER BY created_at
      LIMIT 10
      FOR UPDATE SKIP LOCKED -- prevents multiple relays from processing the same rows
    `);

    for (const row of rows) {
      await sqsClient.send(new SendMessageCommand({
        QueueUrl: QUEUE_URL,
        MessageBody: JSON.stringify(row.payload),
        MessageAttributes: {
          EventType: { DataType: 'String', StringValue: row.event_type },
        },
      }));

      await client.query(
        `UPDATE outbox SET status = 'sent', sent_at = now() WHERE id = $1`,
        [row.id],
      );
    }

    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    console.error('Relay error:', err.message);
  } finally {
    client.release();
  }
}

setInterval(relay, 1000);

FOR UPDATE SKIP LOCKED is the key to running multiple relay instances safely: when a relay picks up a batch of rows, it locks them. Any other relay instance trying to select the same rows will skip them and move to the next available ones, so you never get two relays publishing the same message from the same run.

The relay marks each row as sent only after sqsClient.send returns. If the relay crashes after sending to SQS but before updating the row, the row stays pending and the relay will resend it on the next iteration.

Note that the UPDATE happens inside the same transaction as the SELECT FOR UPDATE, so if the relay crashes mid-batch, the entire batch rolls back and all rows in it will be retried, including any that were already successfully sent to SQS.

The at-least-once delivery guarantee applies at the batch level, not the individual row level. You can read about this problem in Beyond Happy Path Engineering: the Network: when a response is lost, the caller can't know whether the operation succeeded, so it retries, and the receiver may see the same request twice. This means the consumer may see the same message more than once, which is why idempotency matters on the consumer side.

The Consumer

The consumer is a completely separate service. It knows nothing about the order service's PostgreSQL database. Its only input is the SQS queue, and its only output is the DynamoDB fulfillments table. This is the point of the pattern: the two services are decoupled by the queue, and each owns its own data store.

As we saw earlier, because SQS delivers at least once (meaning a message might be delivered more than once), the consumer must be idempotent. The PutItem call uses a ConditionExpression that makes the write a no-op if a fulfillment record for that order already exists, so redelivered messages are handled safely.

async function consume() {
  const { Messages } = await sqsClient.send(new ReceiveMessageCommand({
    QueueUrl:              QUEUE_URL,
    WaitTimeSeconds:       20,   // long-poll: wait up to 20s for messages
    MaxNumberOfMessages:   10,
    MessageAttributeNames: ['All'],
  }));

  for (const msg of Messages ?? []) {
    const event = JSON.parse(msg.Body);

    try {
      await dynamoClient.send(new PutItemCommand({
        TableName: 'fulfillments',
        Item: {
          orderId:     { S: event.orderId },
          customerId:  { S: event.customerId },
          amountCents: { N: String(event.amountCents) },
          status:      { S: 'received' },
          createdAt:   { S: new Date().toISOString() },
        },
        ConditionExpression: 'attribute_not_exists(orderId)', // idempotency check
      }));
    } catch (err) {
      if (err.name !== 'ConditionalCheckFailedException') throw err;
      // already processed, safe to continue
    }

    // delete the message only after the write succeeds (or was already done)
    await sqsClient.send(new DeleteMessageCommand({
      QueueUrl:      QUEUE_URL,
      ReceiptHandle: msg.ReceiptHandle,
    }));
  }
}

ConditionExpression: 'attribute_not_exists(orderId)' tells DynamoDB to reject the write if a record with that orderId already exists. When that happens, DynamoDB throws a ConditionalCheckFailedException. The consumer catches that specific error and ignores it, then deletes the message from the queue and moves on. Any other error is rethrown and the message stays in the queue to be retried.

The DeleteMessage call happens after the DynamoDB write, not before. If the process crashes between the write and the delete, SQS will redeliver the message and the condition check will handle it. If the process crashes before the write, the message stays in the queue and will be processed normally on the next delivery.

Running the Whole Thing

With floci running and the resources provisioned, open three terminal tabs and start each process:

node src/server.js    # the order API on port 3000
node src/relay.js     # the outbox relay
node src/consumer.js  # the fulfillment consumer

Now place an order:

curl -X POST localhost:3000/orders \
  -H 'Content-Type: application/json' \
  -d '{"customerId":"c1","amountCents":4999}'

You should get back a 201 with the new order record:

{"id":"1768d35b-083d-45f1-adb5-4063d8d7fcab","customer_id":"c1","amount_cents":4999,"created_at":"2026-07-30T20:27:10.628Z"}

Within a second the relay will pick up the outbox row and publish it to SQS. The consumer will receive the message and write a fulfillment record to DynamoDB. The repo includes a convenience script to verify this:

npm run check

You should see a fulfillment record with the orderId from the order you just placed:

{
  orderId: 'c335640e-bc4a-47e4-afed-484c95fbd6d3',
  customerId: 'c1',
  amountCents: '4999',
  status: 'received',
  createdAt: '2026-07-30T19:02:54.929Z'
}

Going to Production

Because the local setup uses floci to emulate AWS, switching to real AWS requires no code changes at all. The AWS SDK reads the endpoint from AWS_ENDPOINT_URL in the environment. In production, you simply don't set that variable and the SDK talks to real AWS using the credentials and region from the standard environment variables (AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, or an IAM role if you are running on EC2 or ECS).

Running multiple relay instances is safe out of the box because of FOR UPDATE SKIP LOCKED. You can scale the relay horizontally and each instance will pick up a different set of rows without duplicating messages.

One thing worth adding before going to production is handling permanent failures in the relay. Right now the relay only uses pending and sent. You should add a failed status and a retry counter: after a row has failed N times, mark it failed and stop retrying it. Then configure a dead-letter queue on the orders SQS queue as well, so that messages the consumer can't process after the maximum number of retries land somewhere you can inspect rather than disappearing silently.

For high-throughput systems where polling latency matters, change data capture (CDC) is a common alternative to the polling relay. Tools like Debezium read directly from the PostgreSQL write-ahead log and publish changes to Kafka or SQS without any polling delay. The outbox table and the consumer stay exactly the same, only the relay is replaced.

This is a bigger operational commitment than a polling worker, so polling is the right starting point for most systems.

Conclusion

The dual-write problem is easy to overlook because the naïve implementation works correctly most of the time. It only fails in the gaps between two separate system writes, and those gaps only become visible when something goes wrong at exactly the wrong moment. By the time you notice it in production, data is already inconsistent and there is no clean way to recover.

The transactional outbox pattern closes that gap at the database level. The outbox row is part of the same atomic commit as the business record, so the two are always in sync. The relay handles the network call to SQS independently, with its own retry logic, without touching the request lifecycle. The consumer handles at-least-once delivery with a single condition check on the write.

Each piece is simple on its own, and together they give you reliable, decoupled event delivery without distributed transactions.

The fine never came. The legal fees did. The remediation work did. The reputational conversations with enterprise customers who asked pointed questions for the next six months definitely did.

HIPAA isn't abstract compliance overhead. It's a specific set of technical requirements that translate directly into infrastructure decisions. Get them right and you build a system that earns enterprise healthcare contracts. Get them wrong and you spend your fundraising runway on lawyers instead of engineers.

This handbook gives you the complete technical implementation: every safeguard mapped to its regulation clause, production-ready AWS infrastructure code, and the specific evidence each auditor will ask for. By the time you finish, you'll be able to answer every technical question in a HIPAA audit without looking anything up.

Table of Contents

• What You'll Learn

• Prerequisites

• Part 1: Understanding HIPAA Technical Safeguards

• Part 2: Access Control — §164.312(a)(1)

• Part 3: Audit Controls — §164.312(b)

• Part 4: Integrity Controls — §164.312(c)(1)

• Part 5: Transmission Security — §164.312(e)(1)

• Part 6: AWS Network Architecture for HIPAA

• Part 7: AWS Services Covered by BAA

• Part 8: Continuous Compliance Monitoring

• Part 9: The Pre-Audit Checklist

• Best Practices Summary

• Resources

What You'll Learn

• The five HIPAA Technical Safeguards and exactly which AWS infrastructure decisions each one governs

• How to implement unique user identification and automatic logoff with production-ready code

• How to build an immutable, tamper-evident audit log using hash chaining and S3 Object Lock

• How to implement envelope encryption for ePHI fields using AWS KMS

• The TLS configuration that satisfies HIPAA transmission security requirements

• The complete VPC architecture that satisfies facility access control requirements

• How to run automated HIPAA compliance scans and maintain continuous audit readiness

• The specific evidence your auditor will request for each control

Let's build it properly.

Prerequisites

Before following this guide, you should have:

Knowledge:

• Intermediate AWS experience — you've deployed applications on EC2 or ECS, worked with RDS, and understand VPCs and IAM roles

• Comfort reading Python and Terraform HCL

• Basic understanding of cryptography concepts — you know what symmetric encryption, asymmetric encryption, and hash functions are at a conceptual level

Legal prerequisite — sign the BAA first: Before writing a single line of HIPAA-related infrastructure code, your organisation must have a signed Business Associate Agreement (BAA) with AWS. You can accept the AWS BAA through the AWS Artifact console. Without a signed BAA, using AWS to process ePHI isn't HIPAA-compliant regardless of how well-engineered your technical controls are.

Tools:

• Terraform 1.5 or later

• AWS CLI v2 configured

• Python 3.10 or later with boto3, cryptography, and pyjwt installed

Important scope note: This guide covers the Technical Safeguards defined in 45 CFR §164.312. HIPAA compliance also requires Administrative Safeguards (§164.308) and Physical Safeguards (§164.310). The technical controls in this guide are necessary but not sufficient — they must be accompanied by documented policies, workforce training, and a formal risk assessment.

Part 1: Understanding HIPAA Technical Safeguards

1.1 What the Five Safeguards Actually Require

HIPAA's Security Rule defines five categories of Technical Safeguards. Each maps to specific engineering decisions:

1.2 The Compliance-Evidence Distinction

The most important concept in practical HIPAA engineering: compliance and evidence of compliance are different things, and auditors care about both.

Compliance means your encryption is correctly configured. Evidence of compliance means you have a CloudTrail log showing the KMS key rotation, a command output showing StorageEncrypted: true on every RDS instance, and a dated screenshot of the configuration that you can produce when asked.

Every section of this guide ends with an evidence collection command. Run each one. Save the outputs. Name the files with the date and the control they demonstrate. When your auditor asks "can you show me that your RDS instances are encrypted at rest?", you hand them a file rather than running a command in the room.

1.3 Protected Health Information — Know Your Scope

HIPAA compliance begins with knowing what data in your system constitutes ePHI. The 18 HIPAA identifiers that must be protected:

# phi_classifier.py
# Reference: the 18 HIPAA identifiers

HIPAA_IDENTIFIERS = [
    'full_name', 'first_name', 'last_name',
    'geographic_subdivision',     # Any subdivision smaller than state
    'date_of_birth', 'admission_date', 'discharge_date', 'death_date',
    'phone_number', 'fax_number', 'email_address',
    'social_security_number', 'medical_record_number',
    'health_plan_beneficiary_number', 'account_number',
    'certificate_license_number', 'vehicle_identifier',
    'device_identifier', 'web_url', 'ip_address',
    'biometric_identifier', 'full_face_photo',
]

# Any data record containing one or more of these identifiers
# combined with health information is ePHI and subject to HIPAA.

Part 2: Access Control — §164.312(a)(1)

§164.312(a)(1) requires four specific implementation specifications: unique user identification, emergency access procedures, automatic logoff, and encryption and decryption.

2.1 Unique User Identification

The requirement: every user with access to ePHI must have a unique identifier. Shared accounts violate this requirement. The reason this matters in practice: when an audit or security incident occurs, investigators need to know exactly who accessed which patient record and when. If three nurses share a login, that trail disappears. A unique identifier per user means every ePHI access event is attributable to a specific, named individual — which is what HIPAA's audit controls require and what your legal team will need if something goes wrong.

The code below implements this by assigning every user a UUID v4 at creation time — a randomly generated identifier that is unique across your entire system and is never reused, even after the user's account is deleted. When a user is removed, the account is soft-deleted: the UUID stays in the database and in all historical audit logs, so you can always reconstruct who accessed what. The authentication method also enforces MFA by default and tracks failed login attempts, automatically suspending accounts after five consecutive failures to satisfy HIPAA's requirements around access control and account management.

# user_identity_service.py
# HIPAA-compliant user identity management

import uuid
import hashlib
import hmac
import os
from datetime import datetime, timezone
from dataclasses import dataclass
from typing import Optional


@dataclass
class HIPAAUser:
    user_id:     str   # UUID — never reused
    email:       str
    role:        str   # Clinical / Administrative / Engineering
    department:  str
    status:      str   # ACTIVE / SUSPENDED / DELETED
    mfa_enabled: bool
    created_at:  str
    last_login:  Optional[str] = None
    failed_attempts: int = 0


class UserIdentityService:
    """
    Implements HIPAA §164.312(a)(1)(i) — Unique User Identification.

    Key properties:
    - Every user gets a UUID v4 that is unique and never reused
    - Accounts are soft-deleted — the UUID is preserved in audit logs
      permanently, even after the user leaves
    - All authentication events are logged with user_id, timestamp, and outcome
    """

    MAX_FAILED_ATTEMPTS = 5

    def __init__(self, db, audit_logger):
        self.db    = db
        self.audit = audit_logger

    def create_user(self, email: str, role: str, department: str,
                    created_by: str) -> HIPAAUser:
        """Create a user with a unique, non-reusable identifier."""
        user = HIPAAUser(
            user_id=str(uuid.uuid4()),
            email=email.strip().lower(),
            role=role,
            department=department,
            status='ACTIVE',
            mfa_enabled=True,
            created_at=datetime.now(timezone.utc).isoformat(),
        )
        self.db.save_user(user)
        self.audit.log(
            event_type='USER_CREATED',
            actor_id=created_by,
            subject_id=user.user_id,
            details={'role': role, 'department': department}
        )
        return user

    def authenticate(self, email: str, password: str, mfa_token: str) -> Optional[str]:
        """Authenticate a user. Returns a JWT access token on success."""
        user = self.db.find_user_by_email(email.strip().lower())

        if not user:
            self.audit.log(
                event_type='AUTH_FAILURE',
                actor_id=None,
                subject_id=None,
                details={'reason': 'unknown_email',
                         'email_hash': hashlib.sha256(email.encode()).hexdigest()[:16]}
            )
            return None

        if user.status != 'ACTIVE':
            self.audit.log(
                event_type='AUTH_FAILURE',
                actor_id=user.user_id,
                subject_id=user.user_id,
                details={'reason': f'account_{user.status.lower()}'}
            )
            return None

        if not self._verify_password(password, self.db.get_password_hash(user.user_id)):
            user.failed_attempts += 1
            if user.failed_attempts >= self.MAX_FAILED_ATTEMPTS:
                user.status = 'SUSPENDED'
                self.audit.log(
                    event_type='ACCOUNT_SUSPENDED',
                    actor_id='system',
                    subject_id=user.user_id,
                    details={'reason': 'max_failed_attempts',
                             'attempts': user.failed_attempts}
                )
            self.db.save_user(user)
            return None

        user.failed_attempts = 0
        user.last_login = datetime.now(timezone.utc).isoformat()
        self.db.save_user(user)

        token = self._issue_jwt(user)
        self.audit.log(
            event_type='AUTH_SUCCESS',
            actor_id=user.user_id,
            subject_id=user.user_id,
            details={'token_jti': self._extract_jti(token)}
        )
        return token

    @staticmethod
    def _verify_password(plaintext: str, stored_hash: str) -> bool:
        candidate = hashlib.pbkdf2_hmac(
            'sha256', plaintext.encode(), b'salt', 100_000
        ).hex()
        return hmac.compare_digest(candidate, stored_hash)

    def _issue_jwt(self, user: HIPAAUser) -> str:
        import jwt
        return jwt.encode(
            {
                'sub':  user.user_id,
                'role': user.role,
                'jti':  str(uuid.uuid4()),
                'exp':  int(datetime.now(timezone.utc).timestamp()) + 900,
                'iat':  int(datetime.now(timezone.utc).timestamp()),
            },
            os.environ['JWT_PRIVATE_KEY'],
            algorithm='RS256'
        )

    @staticmethod
    def _extract_jti(token: str) -> str:
        import jwt
        return jwt.decode(token, options={'verify_signature': False})['jti']

The two evidence queries below confirm to an auditor that your system enforces uniqueness: the first shows that every user_id in the database is distinct (no duplicates, no shared credentials), and the second shows that every active user has MFA enabled — both of which are specific requirements auditors check for under this clause.

Evidence for auditors — §164.312(a)(1)(i):

# Show that no shared accounts exist
psql $DATABASE_URL -c "
    SELECT COUNT(*) AS total_users,
           COUNT(DISTINCT user_id) AS unique_ids,
           COUNT(CASE WHEN status='ACTIVE' THEN 1 END) AS active_users
    FROM hipaa_users;
"

# Show MFA is enabled for all active users — expected: 0
psql $DATABASE_URL -c "
    SELECT COUNT(*) FROM hipaa_users
    WHERE status = 'ACTIVE' AND mfa_enabled = FALSE;
"

2.2 Automatic Logoff — §164.312(a)(2)(iii)

The requirement: implement electronic procedures that terminate an electronic session after a predetermined time of inactivity. Most healthcare applications use 15 minutes.

The reason for this control is straightforward: clinical environments involve shared workstations. A nurse logs in to check a patient record, gets called away, and leaves the browser open. Without automatic logoff, the next person to sit at that workstation has full access to ePHI under someone else's credentials. The control is about protecting against the reality of how healthcare teams actually work, not just against malicious actors.

The code below implements automatic logoff using short-lived JWTs. When a user authenticates, they receive a token that expires after 15 minutes of inactivity — each API call implicitly resets that window by issuing a new token. There's also an absolute 8-hour session ceiling: regardless of activity, a user must re-authenticate after eight hours. This prevents a session from staying open indefinitely if a user simply leaves a tab running in the background. The validate_session decorator is applied to every endpoint that touches ePHI, so no access path can bypass these checks.

# session_manager.py
# Implements §164.312(a)(2)(iii) — Automatic Logoff

import os
import uuid
from datetime import datetime, timezone
from functools import wraps
from flask import request, g, jsonify
import jwt

INACTIVITY_TIMEOUT_SECONDS = 15 * 60    # 15 minutes
ABSOLUTE_SESSION_SECONDS   = 8 * 60 * 60  # 8 hours maximum


def validate_session(f):
    """Decorator for endpoints that access ePHI."""
    @wraps(f)
    def decorated(*args, **kwargs):
        auth_header = request.headers.get('Authorization', '')
        if not auth_header.startswith('Bearer '):
            return jsonify({'error': 'MISSING_TOKEN'}), 401

        token = auth_header[7:]

        try:
            payload = jwt.decode(
                token,
                os.environ['JWT_PUBLIC_KEY'],
                algorithms=['RS256']
            )
        except jwt.ExpiredSignatureError:
            return jsonify({
                'error':   'SESSION_EXPIRED',
                'message': 'Your session has expired due to inactivity. Please log in again.',
                'code':    'INACTIVITY_TIMEOUT'
            }), 401
        except jwt.InvalidTokenError as e:
            return jsonify({'error': 'INVALID_TOKEN', 'detail': str(e)}), 401

        # Check absolute session age
        issued_at   = payload.get('session_start', payload['iat'])
        session_age = datetime.now(timezone.utc).timestamp() - issued_at

        if session_age > ABSOLUTE_SESSION_SECONDS:
            return jsonify({
                'error':   'SESSION_EXPIRED',
                'message': 'Your session has exceeded the 8-hour limit. Please log in again.',
                'code':    'ABSOLUTE_TIMEOUT'
            }), 401

        g.user_id = payload['sub']
        g.role    = payload.get('role')
        g.jti     = payload.get('jti')
        return f(*args, **kwargs)

    return decorated


def issue_access_token(user_id: str, role: str, session_start: int = None) -> str:
    """Issue a 15-minute access token with an 8-hour absolute session limit."""
    now = int(datetime.now(timezone.utc).timestamp())
    return jwt.encode(
        {
            'sub':           user_id,
            'role':          role,
            'jti':           str(uuid.uuid4()),
            'iat':           now,
            'exp':           now + INACTIVITY_TIMEOUT_SECONDS,
            'session_start': session_start or now,
        },
        os.environ['JWT_PRIVATE_KEY'],
        algorithm='RS256'
    )

2.3 Encryption at Rest — §164.312(a)(2)(iv)

The requirement: implement a mechanism to encrypt and decrypt ePHI.

Encryption at rest means that if someone gains physical access to your storage media — a hard drive, a backup tape, an S3 object — they can't read the data without the encryption key. On AWS, this protection comes in two layers. The first layer is storage-level encryption, where the database or storage service automatically encrypts every byte written to disk. The second, more powerful layer is field-level encryption, where individual sensitive values are encrypted by your application before they're even handed to the database — so a database administrator with full SQL access still can't read patient SSNs or diagnoses without the application key.

Layer 1 — storage encryption (Terraform):

The Terraform below provisions an RDS instance with a customer-managed KMS key. Using a customer-managed key rather than the AWS default key matters for two reasons: it gives you proof of key ownership (auditors will ask for the KMS key ARN), and it enables automatic key rotation, which replaces the cryptographic material annually without any disruption to your application. The enable_key_rotation = true setting automates this entirely — you don't need to touch the configuration again, and the key stays current.

# rds_hipaa.tf — HIPAA-compliant RDS with customer-managed KMS key

resource "aws_kms_key" "rds" {
  description             = "Customer-managed KMS key for RDS ePHI encryption"
  enable_key_rotation     = true
  deletion_window_in_days = 30

  tags = {
    Purpose     = "HIPAA-ePHI-encryption"
    Environment = "production"
    Control     = "164.312(a)(2)(iv)"
  }
}

resource "aws_db_instance" "hipaa_postgres" {
  identifier     = "hipaa-production-db"
  engine         = "postgres"
  engine_version = "15.4"
  instance_class = "db.r7g.large"

  storage_encrypted = true
  kms_key_id        = aws_kms_key.rds.arn

  backup_retention_period = 30
  deletion_protection     = true
  skip_final_snapshot     = false
  final_snapshot_identifier = "hipaa-production-db-final-snapshot"

  db_subnet_group_name   = aws_db_subnet_group.hipaa.name
  vpc_security_group_ids = [aws_security_group.rds.id]
  publicly_accessible    = false

  tags = {
    DataClassification = "ePHI"
    HIPAAControl       = "164.312(a)(2)(iv)"
  }
}

Layer 2 — field-level application encryption for highest-sensitivity data:

Storage encryption protects you if someone steals a disk. Field-level encryption protects you from authorized users who have legitimate database access but shouldn't be able to read raw patient data. The FieldEncryption class below implements envelope encryption: AWS KMS generates a unique data key for each field value, that key is used to encrypt the plaintext, and only the encrypted version of the key is stored. Even if an attacker extracts your entire database, every encrypted field requires a separate KMS API call to decrypt — which is logged, rate-limited, and requires the correct IAM permissions. The encryption context ties each ciphertext to its purpose and owner, so a key decrypted for one patient's record can't be reused for another.

# field_encryption.py
# Envelope encryption using AWS KMS

import boto3
import base64
from cryptography.fernet import Fernet
from typing import Optional

kms     = boto3.client('kms')
KMS_KEY = 'alias/hipaa-rds-ephi'


class FieldEncryption:
    """
    Envelope encryption for ePHI fields.

    How it works:
    1. AWS KMS generates a data key (plaintext + encrypted copy)
    2. The plaintext data key encrypts the field value using Fernet (AES-128-CBC)
    3. Only the encrypted data key is stored alongside the ciphertext
    4. To decrypt: KMS decrypts the stored data key, then Fernet decrypts the value
    5. A database admin with direct SQL access sees only base64 ciphertext
    """

    def encrypt(self, plaintext: str, context: dict) -> Optional[dict]:
        if not plaintext:
            return None

        data_key = kms.generate_data_key(
            KeyId=KMS_KEY,
            KeySpec='AES_256',
            EncryptionContext=context
        )

        fernet    = Fernet(data_key['Plaintext'])
        ciphertext = fernet.encrypt(plaintext.encode('utf-8'))

        return {
            'ciphertext':         base64.b64encode(ciphertext).decode(),
            'encrypted_data_key': base64.b64encode(data_key['CiphertextBlob']).decode(),
            'encryption_context': context,
        }

    def decrypt(self, payload: dict) -> Optional[str]:
        if not payload:
            return None

        decrypted_key = kms.decrypt(
            CiphertextBlob=base64.b64decode(payload['encrypted_data_key']),
            EncryptionContext=payload['encryption_context']
        )

        fernet    = Fernet(decrypted_key['Plaintext'])
        plaintext = fernet.decrypt(base64.b64decode(payload['ciphertext']))
        return plaintext.decode('utf-8')

Evidence for auditors — §164.312(a)(2)(iv):

# Verify RDS storage encryption
aws rds describe-db-instances \
  --db-instance-identifier hipaa-production-db \
  --query 'DBInstances[0].{Encrypted:StorageEncrypted,KMSKey:KmsKeyId}' \
  --output table

# Verify KMS key rotation is enabled — expected: true
aws kms get-key-rotation-status \
  --key-id alias/hipaa-rds-ephi \
  --query 'KeyRotationEnabled'

Part 3: Audit Controls — §164.312(b)

§164.312(b) requires implementing hardware, software, and procedural mechanisms that record and examine activity in information systems that contain or use ePHI. Every access. Every modification. Every deletion. Logged, immutable, and retainable for six years.

3.1 The Audit Log Schema

Every ePHI-related event must answer five questions: who did it, what did they do, when did they do it, to which record, and from where.

The log_ephi_event function below is the central audit mechanism for your application. Every time a user reads, updates, or deletes a patient record, this function is called before the response is returned. It captures the actor's identity, IP address, browser, and session ID alongside the action, resource, and timestamp — and then adds something more powerful: a cryptographic chain. Each log entry includes the SHA-256 hash of the previous entry. This means if anyone tampers with a log entry — even a single character — every subsequent hash in the chain becomes invalid, making tampering detectable. The entries are then streamed to Kinesis, which fans them out to S3 for long-term storage. One critical rule: the details dictionary must never contain ePHI values, only field names. Log that a SSN field was accessed, not what the SSN was.

# audit_logger.py
# Implements §164.312(b) — Audit Controls

import hashlib
import json
import uuid
import boto3
from datetime import datetime, timezone
from typing import Any, Optional

kinesis = boto3.client('kinesis')
STREAM  = 'hipaa-audit-events'

_last_hash = '0' * 64  # Chain starts with 64 zeros


def log_ephi_event(
    event_type:  str,
    actor_id:    Optional[str],
    patient_id:  Optional[str],
    resource:    Optional[str],
    action:      str,
    details:     dict,
    request_ctx: dict = None,
) -> str:
    """
    Log a HIPAA-relevant event.
    Never include PHI values in the details dict — field names only.
    """
    global _last_hash

    entry = {
        'actor_id':       actor_id,
        'actor_ip':       (request_ctx or {}).get('ip'),
        'actor_ua':       (request_ctx or {}).get('user_agent'),
        'session_id':     (request_ctx or {}).get('session_id'),
        'event_type':     event_type,
        'action':         action,
        'resource':       resource,
        'details':        details,
        'patient_id':     patient_id,
        'timestamp':      datetime.now(timezone.utc).isoformat(),
        'service':        'healthcare-api',
        'environment':    'production',
        'previous_hash':  _last_hash,
        'log_id':         str(uuid.uuid4()),
    }

    canonical   = json.dumps(entry, sort_keys=True)
    entry_hash  = hashlib.sha256(canonical.encode()).hexdigest()
    entry['log_hash'] = entry_hash
    _last_hash  = entry_hash

    kinesis.put_record(
        StreamName=STREAM,
        Data=json.dumps(entry),
        PartitionKey=actor_id or 'system'
    )

    return entry_hash


def log_phi_read(actor_id: str, patient_id: str, resource: str,
                 purpose: str, request_ctx: dict = None):
    return log_ephi_event(
        event_type='PHI_ACCESS',
        actor_id=actor_id,
        patient_id=patient_id,
        resource=resource,
        action='READ',
        details={'purpose': purpose},
        request_ctx=request_ctx,
    )


def log_phi_update(actor_id: str, patient_id: str, resource: str,
                   fields_changed: list, request_ctx: dict = None):
    return log_ephi_event(
        event_type='PHI_UPDATE',
        actor_id=actor_id,
        patient_id=patient_id,
        resource=resource,
        action='UPDATE',
        details={'fields_changed': fields_changed},  # Field names only — NOT values
        request_ctx=request_ctx,
    )

3.2 Immutable Log Storage with S3 Object Lock

Writing logs to S3 isn't enough on its own — logs stored in a standard S3 bucket can be deleted, which would let someone cover their tracks after a breach. S3 Object Lock in COMPLIANCE mode solves this by making every object in the bucket permanently immutable for the retention period you specify. In COMPLIANCE mode, not even the AWS root account can delete the objects before the retention period expires. The bucket below is configured with a 2,190-day (six-year) retention period, which satisfies HIPAA's documentation retention requirement. Versioning is also enabled so that even if a write operation partially overwrites an object, the original version is preserved.

# audit_log_bucket.tf
# S3 bucket with Object Lock in COMPLIANCE mode
# Logs cannot be deleted or modified by anyone — including root

resource "aws_s3_bucket" "audit_logs" {
  bucket              = "hipaa-audit-logs-${data.aws_caller_identity.current.account_id}"
  object_lock_enabled = true

  tags = {
    DataClassification = "audit-log"
    HIPAAControl       = "164.312(b)"
    RetentionYears     = "6"
  }
}

resource "aws_s3_bucket_versioning" "audit_logs" {
  bucket = aws_s3_bucket.audit_logs.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_object_lock_configuration" "audit_logs" {
  bucket = aws_s3_bucket.audit_logs.id
  rule {
    default_retention {
      mode = "COMPLIANCE"  # Nobody can delete — not even root
      days = 2190          # 6 years = 2,190 days
    }
  }
}

resource "aws_s3_bucket_public_access_block" "audit_logs" {
  bucket                  = aws_s3_bucket.audit_logs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Evidence for auditors — §164.312(b):

# Verify Object Lock is enabled in COMPLIANCE mode
aws s3api get-object-lock-configuration \
  --bucket hipaa-audit-logs-YOUR_ACCOUNT_ID \
  --query 'ObjectLockConfiguration'
# Expected: Mode=COMPLIANCE, Days=2190

Part 4: Integrity Controls — §164.312(c)(1)

§164.312(c)(1) requires implementing policies and procedures to protect ePHI from improper alteration or destruction.

The integrity control solves a specific problem: how do you know a patient record hasn't been modified after it was written? Storage encryption protects data from being read by unauthorised parties, but it doesn't protect against an authorised user — a database administrator, a compromised internal account — silently editing a record. Digital signatures do. When a record is created, sign_record produces a cryptographic signature using a KMS asymmetric key. That signature is stored alongside the record. The verify_record function can then confirm at any point that the record's content exactly matches what was signed — any modification, even a single character, produces a different signature that fails verification. The weekly integrity scan calls verify_record on every ePHI record in the specified table and fires an SNS alert for any that fail, creating a continuous tamper-detection mechanism.

# integrity_service.py
# Digitally signs each ePHI record using KMS asymmetric key

import boto3
import base64
import json
from datetime import datetime, timezone

kms         = boto3.client('kms')
SIGNING_KEY = 'alias/hipaa-record-signing'


def sign_record(record: dict) -> str:
    """Digitally sign a patient record. Store the signature alongside the record."""
    canonical = json.dumps(record, sort_keys=True)
    response  = kms.sign(
        KeyId=SIGNING_KEY,
        Message=canonical.encode(),
        MessageType='RAW',
        SigningAlgorithm='RSASSA_PKCS1_V1_5_SHA_256'
    )
    return base64.b64encode(response['Signature']).decode()


def verify_record(record: dict, signature: str) -> bool:
    """Verify a patient record hasn't been altered since signing."""
    canonical = json.dumps(record, sort_keys=True)
    try:
        kms.verify(
            KeyId=SIGNING_KEY,
            Message=canonical.encode(),
            MessageType='RAW',
            Signature=base64.b64decode(signature),
            SigningAlgorithm='RSASSA_PKCS1_V1_5_SHA_256'
        )
        return True
    except kms.exceptions.KMSInvalidSignatureException:
        return False


def weekly_integrity_scan(db, table_name: str) -> dict:
    """
    Scheduled job: verify the digital signature on every ePHI record.
    Any record that fails verification is flagged as potentially tampered.
    Run weekly as required by §164.312(c)(1) policy.
    """
    total    = 0
    failures = []

    for record_id, record, signature in db.iterate_records_with_signatures(table_name):
        total += 1
        if not verify_record(record, signature):
            failures.append({
                'record_id':   record_id,
                'table':       table_name,
                'detected_at': datetime.now(timezone.utc).isoformat(),
            })

    result = {
        'scan_date':          datetime.now(timezone.utc).isoformat(),
        'table':              table_name,
        'records_checked':    total,
        'integrity_failures': len(failures),
        'failed_records':     failures,
    }

    if failures:
        sns = boto3.client('sns')
        sns.publish(
            TopicArn='arn:aws:sns:us-east-1:YOUR_ACCOUNT:hipaa-integrity-alerts',
            Subject=f'INTEGRITY FAILURE: {len(failures)} records in {table_name}',
            Message=json.dumps(result, indent=2)
        )

    return result

Part 5: Transmission Security — §164.312(e)(1)

§164.312(e)(1) requires protecting ePHI during transmission by implementing technical security measures to guard against unauthorized access. The minimum TLS version required is TLS 1.2. TLS 1.3 is recommended. SSL, TLS 1.0, and TLS 1.1 are not acceptable.

Application Load Balancer SSL policy (Terraform):

The ALB is the entry point for all external traffic to your HIPAA application, so it's the first place to enforce TLS requirements. The Terraform resource below configures the HTTPS listener with the ELBSecurityPolicy-TLS13-1-2-2021-06 policy — this is AWS's policy name for a configuration that accepts TLS 1.2 and TLS 1.3 connections while rejecting all older protocols and weak cipher suites. The HTTP listener is configured separately to redirect all port 80 traffic to port 443 with a permanent 301 redirect, ensuring no ePHI can ever be transmitted unencrypted even if a client accidentally connects over HTTP.

# alb_hipaa.tf

resource "aws_alb_listener" "hipaa_https" {
  load_balancer_arn = aws_alb.hipaa.arn
  port              = 443
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = aws_acm_certificate.hipaa.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_alb_target_group.hipaa_api.arn
  }
}

# Redirect all HTTP traffic to HTTPS
resource "aws_alb_listener" "hipaa_http_redirect" {
  load_balancer_arn = aws_alb.hipaa.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type = "redirect"
    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"
    }
  }
}

nginx TLS configuration for direct deployments:

If your application servers handle TLS termination directly — rather than offloading to the ALB — the nginx configuration below enforces the same standards at the server level. The ssl_protocols directive explicitly lists only TLSv1.2 and TLSv1.3, which means nginx will reject any connection attempt using an older protocol. The ssl_ciphers list specifies only ECDHE-based cipher suites with AES-GCM or ChaCha20-Poly1305 — these provide forward secrecy, meaning that even if your private key is later compromised, past session recordings can't be decrypted. The Strict-Transport-Security header with a two-year max-age instructs browsers to always use HTTPS for this domain, even if a user types the HTTP URL. ssl_session_tickets off prevents a class of attack where session ticket keys could be used to decrypt past sessions.

# /etc/nginx/conf.d/hipaa-tls.conf

server {
    listen 443 ssl http2;
    server_name api.your-healthcare-app.com;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
    ssl_prefer_server_ciphers off;

    # HTTP Strict Transport Security — 2 years
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    ssl_stapling        on;
    ssl_stapling_verify on;
    ssl_session_tickets off;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;
}

Evidence for auditors — §164.312(e)(1):

# Verify TLS 1.1 is rejected
openssl s_client \
  -connect api.your-healthcare-app.com:443 \
  -tls1_1 2>&1 | grep -E "CONNECTED|handshake failure"
# Expected: handshake failure

# Verify TLS 1.2 succeeds
openssl s_client \
  -connect api.your-healthcare-app.com:443 \
  -tls1_2 2>&1 | grep "CONNECTED"

# Check ALB SSL policy
aws elbv2 describe-listeners \
  --load-balancer-arn YOUR_ALB_ARN \
  --query 'Listeners[*].{Port:Port,SslPolicy:SslPolicy}' \
  --output table

Part 6: AWS Network Architecture for HIPAA

§164.310(a)(1) (Physical Facility Access Controls) is interpreted in cloud environments as logical network access control — the VPC architecture that isolates ePHI processing from other workloads.

The three-tier VPC below implements network segmentation as a hard boundary around ePHI. The public subnets hold only load balancers — nothing that processes or stores patient data is publicly reachable. The private app subnets hold your API servers, which can receive traffic from the load balancers but have no direct internet path in or out. The private data subnets hold RDS and ElastiCache, which can only receive traffic from the app tier's security group — not from the internet, not from the public subnets, and not from any other source. This means a compromised load balancer cannot directly reach the database: it can only reach the application servers, which apply their own authentication layer before talking to the database.

The VPC endpoints for S3 and KMS ensure that ePHI-related traffic to those services travels through AWS's internal network rather than the public internet. The VPC Flow Logs capture all accepted and rejected network traffic, which gives you the network-level audit trail that complements your application-level audit logs.

# vpc_hipaa.tf — Three-tier VPC architecture

resource "aws_vpc" "hipaa" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name         = "hipaa-production-vpc"
    DataClass    = "ePHI"
    HIPAAControl = "164.310(a)(1)"
  }
}

# Public subnets — load balancers only, no ePHI
resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.hipaa.id
  cidr_block        = "10.0.${count.index + 1}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {Name = "hipaa-public-${count.index + 1}", DataClass = "none"}
}

# Private app subnets — API servers, no direct internet access
resource "aws_subnet" "private_app" {
  count             = 2
  vpc_id            = aws_vpc.hipaa.id
  cidr_block        = "10.0.${count.index + 10}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {Name = "hipaa-private-app-${count.index + 1}", DataClass = "ePHI-processing"}
}

# Private data subnets — RDS, ElastiCache
resource "aws_subnet" "private_data" {
  count             = 2
  vpc_id            = aws_vpc.hipaa.id
  cidr_block        = "10.0.${count.index + 20}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  tags = {Name = "hipaa-private-data-${count.index + 1}", DataClass = "ePHI-storage"}
}

# VPC endpoints — AWS services without internet traversal
# ePHI must not traverse the public internet even within AWS
resource "aws_vpc_endpoint" "s3" {
  vpc_id          = aws_vpc.hipaa.id
  service_name    = "com.amazonaws.${var.region}.s3"
  route_table_ids = [aws_route_table.private.id]
}

resource "aws_vpc_endpoint" "kms" {
  vpc_id              = aws_vpc.hipaa.id
  service_name        = "com.amazonaws.${var.region}.kms"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private_app[*].id
  security_group_ids  = [aws_security_group.vpce.id]
  private_dns_enabled = true
}

# Security groups — least-privilege access
resource "aws_security_group" "rds" {
  name   = "hipaa-rds-sg"
  vpc_id = aws_vpc.hipaa.id

  ingress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id]
    description     = "PostgreSQL from app tier only — no direct external access"
  }
}

# VPC Flow Logs — network audit trail
resource "aws_flow_log" "hipaa" {
  vpc_id          = aws_vpc.hipaa.id
  traffic_type    = "ALL"
  iam_role_arn    = aws_iam_role.flow_logs.arn
  log_destination = aws_cloudwatch_log_group.vpc_flow_logs.arn

  tags = {HIPAAControl = "164.310(a)(1)", Retention = "365-days"}
}

Part 7: AWS Services Covered by BAA

Not every AWS service is covered by the AWS Business Associate Agreement. Using a non-BAA service to process ePHI is a HIPAA violation.

Services not covered by default BAA — do not use for ePHI: Amazon Connect (requires separate agreement), some Amazon Comprehend Medical features (check current BAA), and third-party marketplace products.

Part 8: Continuous Compliance Monitoring

HIPAA compliance isn't a state you achieve once — it's a condition you maintain continuously. Configuration drift is one of the most common causes of HIPAA findings in audits: an engineer spins up a new RDS instance without encryption, a developer creates an S3 bucket without blocking public access, a log group accumulates without a retention policy. None of these are malicious. They're the normal entropy of a growing engineering team.

The scanner below is designed to run as a daily Lambda function. It checks your AWS account against the most common HIPAA technical control failures and writes structured findings to S3 as a dated evidence file. Each finding maps to a specific regulation clause, has a severity level (CRITICAL or HIGH), and names the exact resource that's out of compliance. Running this daily means you catch drift within 24 hours rather than discovering it during an audit.

# compliance_scanner.py
# Daily Lambda job — runs all HIPAA compliance checks

import boto3
import json
from datetime import datetime, timezone

ec2 = boto3.client('ec2')
rds = boto3.client('rds')
s3  = boto3.client('s3')
ct  = boto3.client('cloudtrail')
gd  = boto3.client('guardduty')


def scan_all() -> dict:
    """Run all HIPAA compliance checks. Returns structured findings."""
    findings = []

    # §164.312(a)(2)(iv) — Check: all RDS instances encrypted
    for inst in rds.describe_db_instances()['DBInstances']:
        if not inst.get('StorageEncrypted'):
            findings.append({
                'control':  '164.312(a)(2)(iv)',
                'severity': 'CRITICAL',
                'resource': inst['DBInstanceIdentifier'],
                'finding':  'RDS instance not encrypted at rest',
            })

    # §164.312(a)(2)(iv) — Check: all EBS volumes encrypted
    for vol in ec2.describe_volumes()['Volumes']:
        if not vol.get('Encrypted'):
            findings.append({
                'control':  '164.312(a)(2)(iv)',
                'severity': 'HIGH',
                'resource': vol['VolumeId'],
                'finding':  'EBS volume not encrypted',
            })

    # §164.312(a)(2)(iv) — Check: S3 buckets block public access
    for bucket in s3.list_buckets()['Buckets']:
        name = bucket['Name']
        try:
            pab = s3.get_public_access_block(Bucket=name)[
                'PublicAccessBlockConfiguration'
            ]
            if not all([pab.get('BlockPublicAcls'), pab.get('BlockPublicPolicy'),
                        pab.get('IgnorePublicAcls'), pab.get('RestrictPublicBuckets')]):
                findings.append({
                    'control':  '164.312(a)(2)(iv)',
                    'severity': 'CRITICAL',
                    'resource': f's3://{name}',
                    'finding':  'S3 bucket public access not fully blocked',
                })
        except s3.exceptions.NoSuchPublicAccessBlockConfiguration:
            findings.append({
                'control':  '164.312(a)(2)(iv)',
                'severity': 'CRITICAL',
                'resource': f's3://{name}',
                'finding':  'S3 bucket has no public access block configuration',
            })

    # §164.312(b) — Check: CloudTrail multi-region enabled
    trails      = ct.describe_trails()['trailList']
    multi_region = [t for t in trails if t.get('IsMultiRegionTrail')]
    if not multi_region:
        findings.append({
            'control':  '164.312(b)',
            'severity': 'CRITICAL',
            'resource': 'CloudTrail',
            'finding':  'No multi-region CloudTrail — ePHI access events may not be logged',
        })

    # §164.312(b) — Check: GuardDuty enabled
    detectors = gd.list_detectors().get('DetectorIds', [])
    if not detectors:
        findings.append({
            'control':  '164.312(b)',
            'severity': 'HIGH',
            'resource': 'GuardDuty',
            'finding':  'GuardDuty not enabled — threat detection inactive',
        })

    result = {
        'scan_timestamp':    datetime.now(timezone.utc).isoformat(),
        'total_findings':    len(findings),
        'critical_findings': sum(1 for f in findings if f['severity'] == 'CRITICAL'),
        'high_findings':     sum(1 for f in findings if f['severity'] == 'HIGH'),
        'findings':          findings,
        'compliant':         len(findings) == 0,
    }

    # Save to S3 as dated evidence file
    evidence_s3 = boto3.client('s3')
    date_str    = datetime.now(timezone.utc).strftime('%Y/%m/%d')
    evidence_s3.put_object(
        Bucket='hipaa-compliance-evidence',
        Key=f'scans/{date_str}/compliance_scan.json',
        Body=json.dumps(result, indent=2),
        ContentType='application/json',
    )

    return result


def lambda_handler(event, context):
    result = scan_all()
    print(f"Scan complete: {result['total_findings']} findings, compliant={result['compliant']}")
    return result

Part 9: The Pre-Audit Checklist

Run this script 30 days before any HIPAA audit. It queries your live AWS account across five control categories and writes the output of each check to a dated directory of evidence files. Each file is named after the specific regulation clause it demonstrates, so when an auditor asks for evidence of a particular control, you hand them a file rather than running a command in the room.

Here's what each check collects and what the output looks like:

The RDS encryption check queries every database instance in your account and produces a table showing the instance identifier, whether storage encryption is enabled (true or false), and the KMS key ARN. A HIPAA-compliant account has StorageEncrypted: true on every row.

The KMS rotation check queries every key with "hipaa" in its alias and confirms that KeyRotationEnabled is true for each one. If any key shows false, that's an audit finding under §164.312(a)(2)(iv).

The CloudTrail check returns the trail name, whether it's multi-region (must be true), and whether the trail logs are encrypted with a KMS key. Both properties are required.

The ALB TLS check shows the listener port, protocol, and SSL policy name for every load balancer listener. Auditors look for the policy name to confirm that deprecated TLS versions are disabled.

The VPC endpoint check lists every VPC endpoint in your account with its service name, state, and type. For a HIPAA account, you expect to see at minimum S3 and KMS endpoints in the available state.

#!/usr/bin/env bash
# pre_audit_evidence_collector.sh

EVIDENCE_DIR="hipaa-evidence-$(date +%Y-%m-%d)"
mkdir -p "$EVIDENCE_DIR"

echo "Collecting HIPAA compliance evidence..."

# §164.312(a)(2)(iv) — Encryption at rest
aws rds describe-db-instances \
  --query 'DBInstances[*].{ID:DBInstanceIdentifier,Encrypted:StorageEncrypted,KMS:KmsKeyId}' \
  --output table > "$EVIDENCE_DIR/164-312-a-2-iv-rds-encryption.txt"

aws kms list-aliases \
  --query 'Aliases[?contains(AliasName,`hipaa`)].AliasName' \
  --output text | xargs -I{} aws kms get-key-rotation-status --key-id {} \
  >> "$EVIDENCE_DIR/164-312-a-2-iv-kms-rotation.txt"

# §164.312(b) — Audit Controls
aws cloudtrail describe-trails \
  --query 'trailList[*].{Name:Name,MultiRegion:IsMultiRegionTrail,Encrypted:KMSKeyId}' \
  --output table > "$EVIDENCE_DIR/164-312-b-cloudtrail-config.txt"

# §164.312(e)(1) — Transmission Security
aws elbv2 describe-listeners \
  --load-balancer-arn $(aws elbv2 describe-load-balancers \
    --query 'LoadBalancers[0].LoadBalancerArn' --output text) \
  --query 'Listeners[*].{Port:Port,Protocol:Protocol,SslPolicy:SslPolicy}' \
  --output table > "$EVIDENCE_DIR/164-312-e-1-tls-config.txt"

# §164.310(a)(1) — Network Access Controls
aws ec2 describe-vpc-endpoints \
  --query 'VpcEndpoints[*].{Service:ServiceName,State:State,Type:VpcEndpointType}' \
  --output table > "$EVIDENCE_DIR/164-310-a-1-vpc-endpoints.txt"

echo "Evidence collection complete. Files saved to: $EVIDENCE_DIR/"
ls "$EVIDENCE_DIR/"

Best Practices Summary

Do: Sign the AWS BAA before writing any HIPAA infrastructure code. The technical controls are invalid without the legal agreement.

Do: Use customer-managed KMS keys with automatic rotation. AWS-managed keys are acceptable but don't give you proof of key material control that enterprise healthcare auditors will ask for.

Do: Implement field-level encryption for the highest-sensitivity ePHI fields (SSN, diagnosis, treatment notes). Storage encryption alone doesn't protect against authorized users with direct database access.

Do: Enable S3 Object Lock in COMPLIANCE mode for audit logs. GOVERNANCE mode allows deletion by privileged users. COMPLIANCE mode doesn't allow deletion by anyone, including root.

Do: Run the pre-audit evidence collector monthly, not just before audits. Continuous evidence collection means you're always 30 days away from audit-ready.

Do: Use VPC endpoints for all AWS service communication. ePHI must not traverse the public internet even when both source and destination are within AWS.

Don't: Log PHI values in CloudWatch or application logs. Log that a field was accessed, not what it contained.

Don't: Use shared IAM credentials across multiple engineers or automation systems. Every entity that accesses ePHI must have a unique, auditable identity.

Don't: Assume that being inside a VPC means a workload is isolated. Security groups are the actual enforcement boundary — a misconfigured security group that allows 0.0.0.0/0 on port 5432 exposes your RDS instance regardless of VPC placement.

Resources

• HHS HIPAA Security Rule — The primary source for all Technical Safeguard requirements cited in this guide

• AWS HIPAA Compliance Reference — AWS's official HIPAA whitepaper — required reading before building on the patterns in this guide

• AWS Artifact — BAA Download — Where to accept the AWS Business Associate Agreement

• AWS Services in Scope for HIPAA — The current, definitive list of BAA-covered services

• NIST SP 800-111 — Storage Encryption — NIST guidance on storage encryption that informs HIPAA implementation best practices

• OCR HIPAA Audit Protocol — The exact audit protocol OCR uses — reading this tells you precisely what auditors look for

• Companion Repository — All Terraform modules, Python scripts, and evidence collection scripts from this guide

The app processes credit and debit transactions, fires compliance alerts, and stores everything in a database. You'll build the infrastructure around it yourself: automation, scanning, policy enforcement, secrets management, threat detection, and observability.

By the time you're finished, you'll be able to talk through every decision in an interview because you made each one.

This guide doesn't hand you a pre-built solution. It makes you feel out. and understand each problem before introducing the tool that solves it.

All the code, manifests, scripts, and stage-by-stage READMEs live in the companion repository. Clone it before you start:

git clone https://github.com/Osomudeya/clearledger.git
cd clearledger

Everything in this book refers to files inside that repo.

Prerequisites

You'll need these tools installed on your machine before Stage 0:

• Multipass: creates a lightweight Ubuntu VM so Kubernetes has enough resources

• kubectl: talks to your Kubernetes cluster from your terminal

• Helm: installs apps into Kubernetes

• Docker Desktop: builds container images

• jq: formats JSON output so it's readable

You'll also need free accounts on GitHub and Docker Hub.

And you should be comfortable with the following knowledge and skills:

• Basic Linux command line: navigating directories, reading files, running scripts

• Git: clone, commit, push

• What a container is and roughly how Docker builds one

You don't need prior Kubernetes, security, or cloud experience. This guide builds that from Stage 0.

Your machine needs at least 24 GB of RAM, 6 CPU cores, and 80 GB of free disk space. See How to Set Up Your Machine for the exact install commands.

The companion repo is at github.com/Osomudeya/clearledger. Star it, clone it, then continue.

Table of Contents

• Prerequisites

• What You Are Building

• How to Work Through This Lab

• Tools You Will Use

• How to Choose Your Path

• How to Save Your Progress

• Who This Is For

• How to Set Up Your Machine

• How to Start the Lab

• How to Manage Disk Space

• How to Try the App Without Kubernetes

• How to Configure Local Domain Names

• Stage 0 — The Running System

• Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)

• Stage 2 — GitOps with ArgoCD

• Stage 3 — Security Gates

• Stage 4 — Admission Control (Kyverno)

• Stage 5: Secrets Management (Vault)

• Stage 6 — Runtime Security (Falco)

• Stage 6.5 — Chaos Engineering (Optional)

• Stage 7 — Security Observability

• Stage 7.5 — OpenTelemetry (Optional)

• Stage 8 — AWS Migration

• Troubleshooting (see troubleshooting.md)

• Compliance Reference

• Interview Preparation

• AWS Cost Reference

• Conclusion

What You Are Building

ClearLedger is a fintech transaction ledger built with three FastAPI microservices, PostgreSQL, Redis, and a web frontend.

Users can register, sign in, record credit and debit transactions, view their account balance, and receive compliance alerts whenever a transaction exceeds a predefined threshold.

The application is intentionally simple. Its purpose isn't to teach fintech. It gives you a realistic system that you'll secure and operate like a production platform.

The project consists of four components:

• auth-service: Handles user registration, login, and JWT authentication.

• ledger-service: Processes transactions, maintains account balances, and stores transaction history.

• notification-service: Listens for large transactions through Redis and generates compliance alerts.

• frontend: A web interface for logging in, viewing balances, submitting transactions, and reviewing alerts.

By the end of this book, every one of these services will still exist. What changes is how they are built, deployed, secured, and operated.

The application is simply the vehicle. DevSecOps is the destination.

How the Platform Evolves

You won't install every tool on day one. Instead, the platform grows the same way production systems usually do: a problem appears first, then a solution is introduced.

You'll begin with a manually deployed Kubernetes application. From there, each stage solves one real operational problem.

Stage 0: Raw Kubernetes

You'll deploy and run the application manually, which lets you understand the system before introducing automation.

Stage 1: Continuous Integration

Building container images becomes automatic whenever code is pushed, eliminating manual build steps.

Stage 2: GitOps

Deployments are no longer done with kubectl. Git becomes the single source of truth, preventing configuration drift.

Stage 3: Security Gates

Every commit passes through security scanning so vulnerable code, secrets, and misconfigurations are stopped before deployment.

Stage 4: Admission Control

Even if something bypasses the pipeline, Kubernetes policies prevent insecure workloads from entering the cluster.

Stage 5: Secrets Management

Application credentials move out of Kubernetes Secrets into Vault, removing sensitive data from Git and cluster storage.

Stage 6: Runtime Security

Falco continuously watches running containers and detects suspicious behavior after deployment.

Stage 6.5 (Optional): Chaos Engineering

Failures are introduced deliberately to verify that the platform can recover instead of simply detecting problems.

Stage 7: Observability

Metrics, logs, and dashboards provide visibility into the health, performance, and security of the platform.

Stage 7.5 (Optional): OpenTelemetry

Distributed tracing follows requests across every service, revealing how a single transaction moves through the system.

Stage 8: AWS Migration

The same architecture is deployed on AWS using EKS, ECR, RDS, and an Application Load Balancer without changing how the application itself works.

If you simply want to explore the application before touching Kubernetes, an optional Docker Compose stack lets you run everything locally on your machine.

The guiding principle of this book is simple: every stage makes you feel the problem before introducing the tool that solves it.

How to Work Through This Lab

Throughout this process of understanding each problem before introducing the tool that solves it, three habits will carry you through every stage.

1. Read first before you run: The paragraphs before each command explain why you're running it. Skipping them means you can reproduce the steps but not explain them, and explaining them is what gets you hired. The commands are proof you understand.

2. Choose with a reason: Every tool here solves a specific problem. Why use Vault instead of Kubernetes Secrets? Why split code and manifests into two repos? Don't just follow the steps: ask what breaks if we skip this? If you understand the problem, you'll remember the solution.

3. Go in order and verify every checkpoint: Each stage depends on the one before it. When you hit an issue, read the error. Getting stuck and debugging is part of the learning: employers want to hear "I hit X error and fixed it by doing Y."

At every ✋ Hands-on checkpoint:

1. Run the command.

2. Compare your output with Expected.

3. If it doesn't match, fix it before continuing.

4. When make check-N passes: make snapshot STAGE=N && make snapshots. Only continue after you see clearledger.stageN.

Avoid these mistakes:

• Don't skip a checkpoint because it passed before.

• Don't run make restore without checking available snapshots first.

• Replace your-username with your real Docker Hub or GitHub username everywhere it appears.

• Run runner commands inside the VM (prompt shows ubuntu@clearledger), not on your Mac.

Take screenshots at each portfolio checkpoint. These moments become your evidence: proof that the platform runs, detects, blocks, syncs, and observes real activity.

Tools You Will Use

Come back to the below table when a new name appears and you wonder why now. Each entry is one line: what it does and when it appears.

On your laptop:

• Multipass creates the Ubuntu VM.

• Docker builds images.

• make wraps long commands into make setup / make check-N.

• /etc/hosts entries like clearledger.local let your browser reach the cluster.

The app:

• Three Python APIs (auth, ledger, notifications) + a web frontend.

• Postgres stores data

• Redis lets ledger publish alerts without calling notification directly

• nginx ingress routes browser traffic to the right service.

Each stage adds a new security layer. The tools aren't interchangeable: scanners check your code and images before deployment, ArgoCD keeps the cluster synced to Git, Vault handles secrets, Kyverno blocks unsafe workloads before they run, and Falco watches for suspicious behavior after they're running.

That's why the order matters: you're building defense in depth, one layer at a time.

How to Choose Your Path

Pick one path from your host RAM before you provision a cluster. Switching mid-lab after OOM kills or disk pressure wastes a day, so choose upfront.

The default path in this guide assumes 24 GB+ RAM and the full local VM (Before You Start). If that's not you, start from the row that matches your machine.

How to Save Your Progress

Mac + Multipass only: make snapshot and make restore require Multipass. If you're using Linux without Multipass, skip snapshots and use Path B if something goes wrong.

This lab takes several days to complete.

Your source code lives on your computer, so rebuilding or deleting the VM doesn't delete your Git repository, commits, manifests, or configuration files.

The VM stores your running environment, including deployed pods, Vault secrets, Postgres data, and Grafana dashboards.

Save Your Progress

After completing each stage, create a snapshot before moving on. For example:

make snapshot STAGE=7
make snapshots

Always run make snapshots to confirm the snapshot was created.

Restore Your Progress

If the VM becomes unusable after a while, restore the latest working snapshot:

make snapshots
make restore STAGE=7

export KUBECONFIG=~/.kube/clearledger-config
make check-7

What Happens If the VM Breaks?

You keep:

• Your Git repository

• Your commits

• .env

• setup-cluster.local.env

• clearledger-infra on GitHub

You lose anything stored inside the VM after your last snapshot, including:

• Running pods

• Vault secrets

• Postgres data

• Grafana and Loki data

That's why it's a good idea to create a snapshot after every completed stage.

Path A: You Have a Snapshot (Recommended)

Restore the latest working snapshot and continue from that stage.

make snapshots
make restore STAGE=6

export KUBECONFIG=~/.kube/clearledger-config
make check-6

Path B: No Snapshot

Rebuild the lab.

make teardown
make setup

export KUBECONFIG=~/.kube/clearledger-config

Your Git repositories are still intact, but the Kubernetes cluster starts empty. Continue the book from the stage you had reached and rebuild the platform from there.

If you run into problems such as disk space issues, failed snapshots, Mac sleep or restart problems, Vault authentication errors, or pods stuck in CrashLoopBackOff, see troubleshooting.md for detailed recovery steps.

Who This Is For

Junior DevOps (0–2 yrs): do every stage in order. Don't skip the pain point sections. Expect Stage 0–2 to take a full day each, Stages 3–7 half a day each, Stage 8 a few hours. That's normal, so don't rush.

Mid-level DevOps (2–4 yrs): skim Stages 0–2 to understand the app, focus time on Stages 3–7 where the security layers are.

Interview preparation: complete through Stage 4, then read docs/interview-prep.md. The questions are based on exactly what's in this lab.

How to Set Up Your Machine

Requirements are in Prerequisites above. Confirm 24 GB RAM, 6 CPU cores, and 80 GB free disk before installing.

Install the Required Tools

Windows users: Run all commands inside WSL2 Ubuntu. Don't use PowerShell for this lab because the setup uses make and Bash scripts.

Verify everything before continuing:

multipass --version
kubectl version --client
helm version
docker --version
jq --version

If any command fails, install the missing tool before continuing.

How to Start the Lab

The main lab path starts at Stage 0: the Running System.

After you have run the setup once step by step, you can use this shortcut next time:

make setup
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes

Expected: one node named clearledger with STATUS Ready.

make setup provisions the Multipass VM, installs MicroK8s, applies disk-safety caps, and updates /etc/hosts. Takes 3–5 minutes.

How to Manage Disk Space

The lab runs on a single-node MicroK8s VM with a fixed disk (80 GB by default). Over days or weeks (especially after CI builds, Helm upgrades, and Stage 7 observability) container images, logs, and journald can fill the root filesystem. Pods then fail with Evicted, ImagePullBackOff, or mysterious Pending states.

make setup applies preventive caps automatically (log rotation, image GC thresholds, journald cap). See Disk health in troubleshooting.md for the full table and recovery steps.

Check disk health:

make doctor    # PASS / WARN / FAIL + PVC and Prometheus TSDB sizes

Clean up unused files inside the VM without deleting app data:

make reclaim

If make doctor still reports FAIL after reclaim, you may need make teardown && make setup and restore from a snapshot. Full guidance: troubleshooting.md. VM disk full.

How to Try the App Without Kubernetes

If your machine doesn't have enough resources for Kubernetes, you can run ClearLedger with Docker Compose.

docker compose -f docker-compose.integration.yml up --build -d

Open http://localhost:3000.

When you're ready, stop the stack and continue with Stage 0.

docker compose -f docker-compose.integration.yml down

How to Sign In for the First Time

First, you'll need to register. The database starts empty after each fresh up (or down -v). Use a real-looking email (Pydantic rejects @*.local), for example test@clearledger.io for an email and SecurePass123 for a password.

Then sign in with the same credentials.

Wrong password shows Incorrect email or password. If you see a stale error, hard-refresh or run localStorage.removeItem('cl_token') in the browser console.

How to Run the Demo Flow

First, register and sign in at http://localhost:3000. Submit a few credits and debits (for example, Salary +$5000, Rent −$1200).

Then confirm the balance updates and history lists entries.

Now submit a transaction ≥ $10,000: the Alerts panel should show LARGE_TRANSACTION.

Here's an optional smoke test against the same base URL:

BASE_URL=http://localhost:3000 bash scripts/dast/smoke.sh

How to Configure Local Domain Names

Add the ClearLedger hostnames to your hosts file.

macOS or Linux with Multipass

Run:

sudo bash scripts/setup-hosts.sh

Or do it manually:

VMIP=$(multipass info clearledger | grep IPv4 | awk '{print $2}')

echo "$VMIP  clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts

Verify after Stage 0:

curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health

Expected: 200.

WSL2

Find your WSL IP:

ip -4 addr show eth0 | grep inet

Use the IP shown (or 127.0.0.1 if it works on your machine), then add it to /etc/hosts:

LAB_IP=<YOUR_IP>

echo "$LAB_IP  clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts

If you use Chrome or Edge on Windows instead of inside WSL, add the same line to:

C:\Windows\System32\drivers\etc\hosts

Verify:

curl http://clearledger.local/auth/health

Stage 0 — The Running System

Starting point: Nothing is deployed yet, so you're about to build a Kubernetes cluster and deploy ClearLedger manually.

Goal: By the end of this stage, ClearLedger will be running on Kubernetes. You'll be able to register a user, submit transactions, and see compliance alerts, all deployed by hand, with no automation.

Every deployment, update, and fix is manual. That's intentional. Before automating a platform, you need to understand how it works without automation.

0.1: Provision the Cluster

Next you'll be creating a virtual machine on your laptop that runs its own Kubernetes cluster. Think of it as a miniature data center inside your computer.

Multipass creates lightweight Ubuntu VMs. MicroK8s is a minimal Kubernetes distribution that runs inside that VM. Together they give you a real cluster without needing cloud resources.

Recommended: one command (do this):

make setup
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes

Expected:

NAME          STATUS   ROLES    AGE   VERSION
clearledger   Ready    <none>   2m    v1.29.x

make setup runs scripts/setup-cluster.sh (VM + MicroK8s + disk-safety caps) and scripts/setup-hosts.sh (/etc/hosts entries). It takes 3–5 minutes.

Disk-safety (log rotation, image GC thresholds, journald cap) is configured automatically. See Disk health in troubleshooting.md for more info.

If STATUS is NotReady, wait 60 seconds and try again.

Here's the manual setup (only if make setup failed and you need to debug step by step):

multipass launch \
  --name clearledger \
  --cpus 6 --memory 12G --disk 80G \
  22.04

Get the VM IP (needed for /etc/hosts):

multipass info clearledger | grep IPv4

Add hosts entries. See the Domain Names section above, or run sudo bash scripts/setup-hosts.sh.

multipass shell clearledger

Inside the VM:

sudo snap install microk8s --classic --channel=1.29/stable
sudo usermod -aG microk8s ubuntu && newgrp microk8s
microk8s enable dns ingress storage helm3 rbac
echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
echo "alias helm='microk8s helm3'" >> ~/.bashrc
source ~/.bashrc
kubectl get nodes
exit   # back to your host machine

Connect kubectl from your host:

multipass exec clearledger -- microk8s config > ~/.kube/clearledger-config
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes

0.2: Understand the Application Before Deploying it

Open these files before running a single kubectl command. Reading the code first builds context that makes everything else make sense.

Notice this line in every Dockerfile: USER appuser. It means the image is designed to run as a normal user instead of root. The Kubernetes manifests also set runAsNonRoot: true. Later, in Stage 4, Kyverno enforces that rule and rejects pods that don't declare they run as non-root. Your app is prepared early so it passes that policy later.

Also look at infra/manifests/auth-service/secret.yaml. The database password is changeme-stage0 encoded in base64. Decode it:

echo "Y2hhbmdlbWUtc3RhZ2Uw" | base64 -d
# changeme-stage0

That password is sitting in a YAML file anyone with repo access can read. base64 is encoding, not encryption. It's trivially reversible. Remember this moment. It's why Stage 5 exists.

0.3: Docker Hub Setup

You need a container registry: a place to store the built images so the cluster can pull them. Docker Hub is the simplest option. You'll replace it with a private registry (ECR) in Stage 8.

Create four public repositories on Docker Hub (free account, hub.docker.com):

1. Go to hub.docker.com

2. Click Create repository

3. Choose your Docker Hub username as the namespace

4. Enter one repository name from the list below

5. Set visibility to Public

6. Click Create

7. Repeat for all four services

YOUR_USERNAME/clearledger-auth-service
YOUR_USERNAME/clearledger-ledger-service
YOUR_USERNAME/clearledger-notification-service
YOUR_USERNAME/clearledger-frontend

Next, generate an access token. Go to hub.docker.com, then Account Settings, Security, and New Access Token (Read/Write/Delete). Save it. You won't see it again.

docker login
# Username: your Docker Hub username
# Password: the access token (NOT your account password)

Build and push all four services:

# Replace your-username with your Docker Hub username, the same string everywhere in this lab
export DOCKER_USERNAME=your-username
echo "Using DOCKER_USERNAME=$DOCKER_USERNAME"

✋ Hands-on checkpoint: Docker Hub username

# Must print your real username, not the literal text "your-username"
echo "$DOCKER_USERNAME"

Expected: one line with your Docker Hub name (for example, veeno-demo). If you see your-username instead, stop and fix export before building.

Build and push all four services:

docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.1.0 ./app/auth-service
docker build -t $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0 ./app/ledger-service
docker build -t $DOCKER_USERNAME/clearledger-notification-service:v0.1.0 ./app/notification-service
docker build -t $DOCKER_USERNAME/clearledger-frontend:v0.1.0 ./app/frontend

# Push

docker push $DOCKER_USERNAME/clearledger-auth-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-notification-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-frontend:v0.1.0

✋ Hands-on checkpoint: images on Docker Hub

Open hub.docker.com and go to your profile, then Repositories. Then confirm that all four clearledger-* repos exist and each shows tag v0.1.0.

On your laptop, run:

docker pull $DOCKER_USERNAME/clearledger-auth-service:v0.1.0

Expected: Status: Downloaded newer image or Image is up to date, not repository does not exist or denied.

0.4: Look at the Manifests Before Applying Them

Kubernetes uses manifest files (YAML) to describe the resources it should create. Instead of clicking buttons, you declare the desired state, and Kubernetes creates it.

Before deploying ClearLedger, take a quick look at these manifests:

• infra/manifests/namespace.yaml: Creates the clearledger namespace.

• infra/manifests/postgres/: Deploys PostgreSQL.

• infra/manifests/redis/redis.yaml: Deploys Redis.

• infra/manifests/auth-service/: Deploys the authentication service.

• infra/manifests/ledger-service/: Deploys the ledger service.

• infra/manifests/notification-service/: Deploys the notification service.

• infra/manifests/frontend/: Deploys the web application.

• infra/manifests/ingress.yaml: Makes the application available at clearledger.local.

• infra/manifests/rbac/rbac.yaml: Defines who may do what inside the cluster.

You don't need to understand every field yet. The goal is simply to see how the application is described before Kubernetes creates it.

You'll understand how Ingress routing and RBAC work in the two optional sections after §0.6. For now, just see how the app is described before Kubernetes creates it.

0.5: Deploy ClearLedger (Layer by Layer)

Deploy in six layers. Finish each layer before starting the next. Run kubectl get pods -n clearledger after layers 2, 3, and 6 to confirm progress.

Set a short path variable and confirm your username is still set:

export DOCKER_USERNAME=your-username   # skip if already set in §0.3
STAGE0=stages/stage-0-raw-kubernetes/infra/manifests

0.5.1 — Layer 1: Namespace and RBAC

Nothing else can be created until the namespace exists. RBAC also must exist before workloads reference ServiceAccounts.

kubectl apply -f infra/manifests/namespace.yaml
kubectl apply -f infra/manifests/rbac/rbac.yaml

Verify:

kubectl get namespace clearledger
kubectl get serviceaccount -n clearledger
# Expected: auth-service, ledger-service, notification-service, clearledger-viewer

0.5.2 — Layer 2: PostgreSQL

Database must be running before auth-service or ledger-service start. Both services connect to Postgres on startup to run migrations and serve requests, and they'll crash-loop if the database isn't there yet.

kubectl apply -f infra/manifests/postgres/postgres-secret.yaml
kubectl apply -f infra/manifests/postgres/postgres.yaml

kubectl wait --for=condition=ready pod -l app=postgres \
  -n clearledger --timeout=120s

Expected after kubectl apply:

secret/postgres-secret created
persistentvolumeclaim/postgres-pvc created
statefulset.apps/postgres created
service/postgres created

Expected when kubectl wait succeeds: the command exits with no output (exit code 0). If it times out, see If Postgres stays Pending below before continuing.

Verify:

kubectl get pods -n clearledger -l app=postgres
kubectl get pvc -n clearledger

Expected:

NAME         READY   STATUS    RESTARTS   AGE
postgres-0   1/1     Running   0          45s

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
postgres-pvc   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   5Gi        RWO            microk8s-hostpath   45s

If Postgres stays Pending (kubectl wait times out, pod shows 0/1 Pending, PVC shows Pending):

Postgres needs a PersistentVolumeClaim: disk space on the cluster. MicroK8s provides that through the hostpath-storage addon. If make setup was interrupted or you used manual setup without microk8s enable storage, the PVC has nothing to bind to and the pod never schedules.

Check the events: you'll usually see something like:

Warning  FailedScheduling  ...  pod has unbound immediate PersistentVolumeClaims
Normal   FailedBinding     ...  no persistent volumes available for this claim and no storage class is set

Fix it on the VM, then restart the postgres pod. Run this from your host: the same command on macOS, Linux, or Windows PowerShell (Multipass is installed on the host. It executes inside the VM for you):

# Enable storage (and ingress/rbac if make setup skipped them)
multipass exec clearledger -- microk8s enable storage ingress rbac

# Confirm a default StorageClass exists
kubectl get storageclass
# Expected: microk8s-hostpath (default)

# Kick the pod so it reschedules against the new storage class
kubectl delete pod postgres-0 -n clearledger

kubectl wait --for=condition=ready pod -l app=postgres \
  -n clearledger --timeout=120s
kubectl get pods -n clearledger -l app=postgres
# Expected: postgres-0   1/1   Running

Don't continue to auth-service or ledger-service until Postgres is Running. They will crash-loop without a database.

0.5.3 — Layer 3: Redis

Why Redis is here (a quick scenario): Imagine a customer posts a $15,000 debit. Ledger-service saves it to Postgres, then publishes a message to Redis: "large transaction, user X, amount 15000." Notification-service is listening on that channel. It picks up the message and records a compliance alert: the one you'll see in the UI later when you curl /notifications/alerts.

Ledger-service and notification-service don't call each other directly. Redis sits in the middle as a message bus: ledger publishes, notification subscribes. That's why Redis must be running before you deploy notification-service (and why you deploy it now, alongside Postgres, before the app layer).

kubectl apply -f infra/manifests/redis/redis.yaml

Verify:

kubectl get pods -n clearledger -l app=redis

Expected:

NAME                     READY   STATUS    RESTARTS   AGE
redis-xxxxxxxxxx-xxxxx   1/1     Running   0          30s

0.5.4 — Layer 4: Application secrets

Credentials live in Kubernetes Secrets for Stage 0 (Stage 5 moves them to Vault).

kubectl apply -f infra/manifests/auth-service/secret.yaml
kubectl apply -f infra/manifests/ledger-service/secret.yaml

Verify:

kubectl get secrets -n clearledger | grep -E 'auth-service|ledger-service'

Expected (AGE will differ, DATA counts must match):

auth-service-secret     Opaque   2      64s
ledger-service-secret   Opaque   1      8s

auth-service-secret holds two keys (database_url, jwt_secret). ledger-service-secret holds one (database_url). Stage 5 replaces these with Vault, for now they live in the cluster as Kubernetes Secrets.

0.5.5 — Layer 5: Application workloads

You're about to start the four app services: auth, ledger, notification, and frontend. Postgres, Redis, and the Secrets from the last two layers are already in place. Now Kubernetes needs to pull your Docker Hub images and run them as pods.

Two files per service (mostly): A Deployment tells Kubernetes which container image to run and how many copies. A Service gives that app a stable name inside the cluster (for example, auth-service so ledger can find auth without knowing pod IP addresses). You apply the Deployment first, then the Service.

So why are we using the sed command below? The deployment YAML files in Git contain a placeholder: literally the text DOCKER_USERNAME, because everyone's Docker Hub username is different. You already set yours in §0.3 (export DOCKER_USERNAME=YOUR_DOCKERHUB_USERNAME). The sed line swaps that placeholder for your real username on the fly, as the manifest is sent to Kubernetes. You never edit the file in Git. If you skip sed and apply the raw file, Kubernetes tries to pull an image called DOCKER_USERNAME/clearledger-auth-service, which doesn't exist.

Why do we use the Stage 0 folder? This repo has more than one copy of the Kubernetes manifests. For this manual deployment, use stages/stage-0-raw-kubernetes/infra/manifests/. Those files are prepared for Stage 0 and contain the DOCKER_USERNAME placeholder that the commands below replace. Don't use infra/manifests/ yet, as those files are for the GitOps stages later.

Deploy each service in order. Run these from the repo root with DOCKER_USERNAME still exported:

1. auth-service: login and registration

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/auth-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/auth-service/service.yaml

2. ledger-service: transactions and balance (needs Postgres + the secret you created in §0.5.4)

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/ledger-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/ledger-service/service.yaml

3. notification-service: listens on Redis for large-transaction alerts (no database secret in this one)

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/notification-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/notification-service/service.yaml

4. frontend: the web UI (Deployment and Service are in one file here)

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/frontend/deployment.yaml" | kubectl apply -f -

Verify (all app pods should reach Running: auth and ledger may take ~30s while they connect to Postgres):

kubectl get pods -n clearledger

Expected. You should see Postgres and Redis from earlier layers plus new pods for each app (exact pod names vary):

NAME                                      READY   STATUS    RESTARTS   AGE
postgres-0                                1/1     Running   0          15m
redis-xxxxxxxxxx-xxxxx                    1/1     Running   0          10m
auth-service-xxxxxxxxxx-xxxxx             1/1     Running   0          45s
auth-service-xxxxxxxxxx-xxxxx             1/1     Running   0          45s
ledger-service-xxxxxxxxxx-xxxxx           1/1     Running   0          40s
ledger-service-xxxxxxxxxx-xxxxx           1/1     Running   0          40s
notification-service-xxxxxxxxxx-xxxxx     1/1     Running   0          35s
frontend-xxxxxxxxxx-xxxxx                 1/1     Running   0          30s

If auth-service or ledger-service is CrashLoopBackOff, check the logs:

kubectl logs -n clearledger deploy/auth-service --tail=20

Common cause: you applied infra/manifests/*/deployment.yaml instead of the Stage 0 files above: logs may show DATABASE_URL is not set. Re-run the sed + kubectl apply commands in this section.

✋ Hands-on checkpoint: workloads before ingress

kubectl get deployment -n clearledger
kubectl get pods -n clearledger --field-selector=status.phase!=Running

Expected: four Deployments (auth-service, ledger-service, notification-service, frontend) with READY matching desired replicas (auth and ledger show 2/2). The second command prints nothing: no pods stuck in Pending or CrashLoopBackOff.

0.5.6 — Layer 6: Ingress

Exposes the cluster to http://clearledger.local.

kubectl apply -f infra/manifests/ingress.yaml

Verify:

kubectl get ingress -n clearledger
curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/
# Expected: 200

0.5.7: Watch until stable

kubectl get pods -n clearledger -w

Expected final state (press Ctrl+C to stop watching once all pods show Running):

NAME                                  READY   STATUS    RESTARTS
auth-service-xxx                      1/1     Running   0
auth-service-yyy                      1/1     Running   0
frontend-xxx                          1/1     Running   0
ledger-service-xxx                    1/1     Running   0
ledger-service-yyy                    1/1     Running   0
notification-service-xxx              1/1     Running   0
postgres-0                            1/1     Running   0
redis-xxx                             1/1     Running   0

Pod stuck in Pending or CrashLoopBackOff? These two commands show you what went wrong:

kubectl describe pod POD_NAME -n clearledger
kubectl logs POD_NAME -n clearledger --previous

0.6: Verify the Running System

Use one test account for both browser and curl so nothing conflicts:

If you already registered in the browser with a different password, either sign in with that password or pick a new email: the curl commands below must use the same email and password you actually registered with.

Browser verification (recommended):

Open http://clearledger.local in your browser. You should see the ClearLedger login screen.

Click Register and create an account with test@clearledger.io / SecurePass123 (same as the curl block below. Pydantic rejects obviously fake emails like test@test.com).

Sign in with that email and password. On first login the dashboard auto-seeds demo transactions. Wait a few seconds for them to appear:

Look at the Current Balance card. It should show a dollar amount with a sparkline chart.

Look at Transaction History. You should see entries like "Salary (Acme Corp", "Rent) May 2026", and so on.

And look at the Alerts panel at the bottom. You should see LARGE_TRANSACTION alerts with a red badge. Two of the demo transactions exceed $10,000, which triggers the compliance alert automatically.

Then submit your own transaction over $10,000 and watch the alert count increase in real time.

What to look for:

• Balance updates immediately after each transaction

• Credits show as green +$ amounts, debits show as red −$ amounts

• The Alerts badge count increases when you submit a transaction ≥ $10,000

• Each alert shows the amount, direction, and timestamp

Take a screenshot of the dashboard showing transactions and at least one alert. This is the first piece of your portfolio.

Alternatively via curl (same account: useful if the browser is not cooperating):

# Register (skip if you already registered in the browser with the same email)
curl -s -X POST http://clearledger.local/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' | jq .

Expected: {"user_id":"...","email":"test@clearledger.io"}, or an error that the email is already registered (fine if you used the browser first).

# Login — save the token (must match the password you registered with)
TOKEN=$(curl -s -X POST http://clearledger.local/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' \
  | jq -r .access_token)
echo "Token: ${TOKEN:0:30}..."

If TOKEN is empty or login returns 401, your browser password doesn't match: re-register with the table above or use your actual password in the -d JSON.

# Create a large transaction (triggers notification alert)
curl -s -X POST http://clearledger.local/ledger/transactions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"amount":15000,"direction":"debit","description":"Property payment"}' | jq .

Expected: a transaction object with id, amount: 15000, direction: "debit":

# Check balance
curl -s http://clearledger.local/ledger/balance \
  -H "Authorization: Bearer $TOKEN" | jq .

# Confirm the notification alert fired
curl -s http://clearledger.local/notifications/alerts | jq .

Expected (curl-only path, no browser demo seed): at least one alert for the $15,000 transaction, for example, {"total":1,"alerts":[{"type":"LARGE_TRANSACTION","amount":15000,...}]}. If you already used the browser, total may be 3 or more (two demo alerts plus yours), that is also correct.

If you see {"detail":"Unauthorized"}: your token has expired. JWTs are short-lived for security. This is intentional. Re-run the login command above to get a fresh token, then retry the failed command.

This only affects the $TOKEN variable in your current terminal session. If you open a new terminal, you need to run the login command again because $TOKEN doesn't persist across sessions.

make check-0

Understanding Ingress (Optional)

Read this after §0.6 if you want to understand how clearledger.local reaches your pods.

Your cluster runs four application services: frontend, auth-service, ledger-service, and notification-service. Each has an internal Service address inside the cluster, but none are reachable from your browser until an Ingress routes external traffic.

The Ingress is the front door. When a request hits clearledger.local, Kubernetes looks at the URL path and forwards to the right service. Requests to /auth go to auth-service, /ledger to ledger-service, /notifications to notification-service, and / to the frontend.

Open infra/manifests/ingress.yaml and read the comments. The API paths use a rewrite: /auth/login becomes /login before it reaches auth-service, so backend routes stay simple.

You'll add more hostnames later (grafana.local, argocd.local, and so on): each gets its own Ingress manifest in a later stage. This file is only the ClearLedger app.

Understanding RBAC (Optional)

Ingress controls traffic coming from outside the cluster. RBAC controls permissions inside the cluster.

This file creates identities and permissions for the clearledger namespace.

Open infra/manifests/rbac/rbac.yaml. The comments at the top mirror this walkthrough.

A ServiceAccount is an identity for a pod. For example, auth-service, ledger-service, and notification-service each get their own identity.

A Role says what that identity is allowed to do. In your repo, the app roles are very limited: they can only get and list Kubernetes Endpoints. They can't read Secrets, delete pods, create resources, or access other namespaces.

A RoleBinding connects the identity to the permissions. Without the RoleBinding, the Role exists but no pod receives those permissions.

The clearledger-viewer ServiceAccount is for read-only debugging. It can inspect pods, services, endpoints, events, and configmaps, but it can't read Secrets.

The default ServiceAccount is bound to a role with zero permissions. That way, if a pod forgets to set serviceAccountName, it falls back to an identity that can do nothing.

The point is least privilege: even if a pod is compromised, Kubernetes doesn't hand it broad cluster access.

0.7: Why Manual Deploys Can't Be Trusted

In Stage 0, you built and deployed the app by hand. Now you'll make one small code change and deploy it again. This shows the problem with manual deployments: they're hard to track, hard to roll back, and hard to prove. Stages 1 and 2 fix that with CI and GitOps.

Step 1: Make a visible change.

Open app/auth-service/main.py and find the /health endpoint. Change the return value so you can tell the new version is running:

# Before
return {"status": "ok", "service": settings.service_name}

# After — add a version field
return {"status": "ok", "service": settings.service_name, "version": "0.2.0"}

Save the file. This simulates a developer shipping a small fix.

Step 2: Build, push, and deploy by hand.

docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.2.0 ./app/auth-service

docker push $DOCKER_USERNAME/clearledger-auth-service:v0.2.0
kubectl set image deployment/auth-service \
  auth-service=$DOCKER_USERNAME/clearledger-auth-service:v0.2.0 \
  -n clearledger

Wait about 30 seconds for Kubernetes to pull the new image and restart the pods:

kubectl rollout status deployment/auth-service -n clearledger

Step 3: Verify your change is live.

curl -s http://clearledger.local/auth/health | jq .

Expected: {"status":"ok","service":"auth-service","version":"0.2.0"}

If you still see the old response without "version", wait a few more seconds and retry. Kubernetes is still rolling out the new pods.

Step 4: Notice what manual deploy doesn't give you.

You deployed a change. It works. But think about what just happened:

• Who deployed this? There's no record. You ran kubectl from your laptop. If three people have cluster access, no one knows who changed what.

• What changed? The only evidence is the Docker Hub tag v0.2.0. Nothing links that tag to a specific commit or code review.

• What if v0.2.0 is broken? You would need to remember the previous tag, then run kubectl set image again to roll back. What if you don't remember the tag? What if the previous image was deleted?

• What if someone else runs kubectl apply with v0.1.0 while you're pushing v0.2.0? The cluster silently reverts to the old version. No error. No notification. You think your fix is live, but it's not.

• Where is the audit trail? Nowhere. In a regulated environment (banking, healthcare, government), you need proof of who deployed what and when. Right now you have nothing.

Manual deploys can work for a demo. But they don't hold up for a team or a regulated environment. Keep these gaps in mind. They're why the next stages exist.

Step 5: Revert your change before continuing.

Undo the health endpoint change in app/auth-service/main.py (remove "version": "0.2.0"). Don't rebuild: the cluster will keep running v0.2.0 for now, and Stage 1 will take over image management.

Stage 1 automates the build. Stage 2 fixes the deployment.

What You Learned in Stage 0

• How to provision a local Kubernetes cluster with Multipass and MicroK8s

• How Kubernetes manifests describe the desired state of your system

• How an Ingress routes external traffic to internal services

• How to build, push, and deploy container images manually

• Why manual deploys can't be trusted: no audit trail, no rollback, no consistency

What you can now put on your CV / say in an interview:

Deployed a multi-service application to Kubernetes by hand: namespace, RBAC, a StatefulSet database, Deployments, Services, and path-based Ingress routing, and can explain why each layer deploys in that order.

make snapshot STAGE=0 && make snapshots. Confirm clearledger.stage0. See How to Save Your Progress.

Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)

In Stage 0 you built and deployed by hand. Stage 1 automates the build: a git push runs a pipeline that builds images, scans them, pushes to Docker Hub, and records the new tag in clearledger-infra.

Goal: every push to GitHub automatically builds images, pushes them to Docker Hub, and updates image tags in clearledger-infra.

Am I ready for Stage 1?

Run these yourself before §1.1:

make check-0
echo "$DOCKER_USERNAME"    # must not be empty or "your-username"
curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health

Expected: health check green, echo prints your Docker Hub user, and curl prints 200.

What you'll need for this section:

• Docker Hub account with four clearledger- repositories (see QUICKSTART.md §1b)

• GitHub account: you can create repos and personal access tokens

• ~2–4 hours for runner install + first green pipeline (this is the hardest stage for beginners)

• Done when: make check-1 passes and you manually confirmed the five items in §1.7 below. Then save: make snapshot STAGE=1 → make snapshots (confirm clearledger.stage1).

What You Need to Know First

In Stage 0, your laptop was the deployment system.

You typed docker build, docker push, and kubectl set image yourself. That worked for a demo, but it's not how teams should ship software.

Manual builds create too many unanswered questions:

• Did this image come from the latest code?

• Did someone build it from a dirty working tree?

• Did the build work the same way on another machine?

• Which commit produced the image currently running?

• Who pushed the image, and when?

CI (Continuous Integration) fixes the build side of that problem. It means that every time code is pushed, an automated system builds, checks, and packages it the same way.

Think of CI as a factory line:

Developer pushes code
        ↓
GitHub detects the push
        ↓
GitHub Actions starts the pipeline
        ↓
Runner executes the jobs
        ↓
Docker images are built and pushed
        ↓
Infra manifests are updated with the new image tags  (in clearledger-infra — §1.3)

The important idea is that the build no longer depends on your laptop. Your laptop writes code and the pipeline produces the release artifact.

A CI system has three parts:

1. Pipeline host: the control plane. It notices a push and decides which workflow to run. In this lab, that's GitHub Actions.

2. Pipeline file: the instructions. It's a YAML file at .github/workflows/ci.yaml that says what jobs to run.

3. Runner: the worker machine. It actually executes the commands in the pipeline.

GitHub Actions normally uses GitHub-hosted runners in the cloud. In this lab, that's not enough. Your Kubernetes cluster lives inside a local Multipass VM and GitHub's cloud runner can't reach it. You also need the runner inside the VM to build Docker images using the local Docker daemon.

So you install a self-hosted runner inside the VM. It connects outbound to GitHub, waits for work, then executes pipeline jobs locally where it can reach everything.

Two repos, clearledger (code + CI) and clearledger-infra (Kubernetes YAML only). You'll create the second in §1.3.

GitHub — clearledger (app repo)
  stores your code
  starts the workflow on git push
        ↓
Self-hosted runner (inside Multipass VM)
  builds Docker images
  pushes images to Docker Hub
  updates image tags in clearledger-infra  ← you create this in §1.3
        ↓
GitHub — clearledger-infra (infra repo)
  stores Kubernetes YAML with the new image tags
  ArgoCD watches this repo in Stage 2 (not yet)

For Stages 1–7, the lab uses .github/workflows/ci.yaml with your self-hosted runner. It builds images, pushes them to Docker Hub, and updates clearledger-infra. Stage 8 adds a separate AWS workflow, .github/workflows/ci-aws.yaml, which pushes to ECR instead. You don't need to configure the AWS workflow until you reach Stage 8.

1.1: Push the App Repo to GitHub (Not clearledger-infra Yet)

This step is repo #1, clearledger (application code + CI workflow). You're pushing the clone on your laptop: the same folder where you ran Stage 0 (make setup, kubectl apply, and so on).

clearledger-infra comes later in §1.3. That second repo holds Kubernetes manifests only. Don't create it here.

First, put the application repo somewhere GitHub Actions can see it.

Go to GitHub and then New Repository:

• Repository name: clearledger (exact name, not clearledger-infra)

• Visibility: Public or Private. Both work with the self-hosted runner and GitHub Actions. ArgoCD never reads this repo (see Private repos: what syncs where in §1.3).

• Do not initialize with a README or .gitignore

The repo already has those files locally. If GitHub creates its own, your first push may fail because the histories don't match.

Run from your local clearledger project root on your laptop (where app/, infra/, and .github/workflows/ci.yaml live):

cd ~/Desktop/clearledger   # your clone path
git remote add origin https://github.com/YOUR_USERNAME/clearledger.git
git branch -M main
git push -u origin main

If git remote add fails because origin already exists:

git remote -v
git remote set-url origin https://github.com/YOUR_USERNAME/clearledger.git
git push -u origin main

Verify in the browser: https://github.com/YOUR_USERNAME/clearledger.

You should see app/, infra/manifests/, docs/, and .github/workflows/ci.yaml. That confirms GitHub can trigger the pipeline on your next push.

What you proved: the app repo is on GitHub. CI will run from here. Deployment manifests for GitOps land in clearledger-infra in §1.3.

1.2: Install the Self-Hosted Runner Inside the VM

The workflow file tells GitHub what to run. The runner is where it runs.

This lab uses a self-hosted runner because your infrastructure is local. GitHub's cloud servers can't reach your MicroK8s cluster or Docker daemon inside the Multipass VM. The runner solves that by living inside the VM. It connects to GitHub to pick up jobs, then executes everything locally.

If the runner is missing or offline, the pipeline can't execute. The workflow may sit queued, or it may fail because no matching runner is available.

Step 1: Open GitHub’s runner setup page (keep this tab open)

GitHub gives you a full copy-paste install guide on one page. Use it: don’t hunt for URLs or tokens elsewhere.

1. Open https://github.com/YOUR_USERNAME/clearledger

2. Go to Settings, Actions, Runners, and New self-hosted runner

3. Select Linux and x64

The page title should look like: Add new self-hosted runner · YOUR_USERNAME/clearledger.

That page has three sections you'll use:

Scroll to Configure. You'll see something like:

./config.sh --url https://github.com/YOUR_USERNAME/clearledger --token AXXXXXXXXXXXXXXXXXXXXXXXXX
./run.sh

The token is the long string after --token (starts with A, about 26 characters). Copy only that string.

Keep this tab open until Step 4 finishes: the token expires in about 1 hour. If it expires, click New self-hosted runner again for a fresh token.

Step 2: Enter the VM

multipass shell clearledger

After this command, your prompt should look like ubuntu@clearledger:~$. That means you are inside the Ubuntu VM. If your prompt still shows your Mac username or MacBook name, you're still on your host machine and the runner setup will fail.

Continue only when your prompt shows ubuntu@clearledger.

Everything from Step 3 onwards runs inside the VM, not on your Mac.

Step 3: Install Docker inside the VM

The runner will build Docker images. That means Docker must exist where the runner runs.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker ubuntu
newgrp docker

docker --version

Expected: Docker prints a version number (for example, Docker version 29.x.x).

Verify Docker works for the ubuntu user now: the runner doesn't exist yet (Step 4 creates ~/actions-runner):

docker ps

Expected: a table header (CONTAINER ID, IMAGE, …), even if no containers are listed. Not permission denied while trying to connect to the Docker API.

If docker ps fails with permission denied, the docker group has not applied yet. Run newgrp docker again, or log out of the VM (exit) and multipass shell clearledger back in, then retry docker ps.

What you proved: the VM can run Docker without Docker Desktop on your Mac. Continue to Step 4 to install the runner.

Step 4: Install and register the runner

Still inside the VM (ubuntu@clearledger prompt):

Download: you can copy the commands from the Download section on GitHub’s runner page (Step 1), or run the block below. They should match. Paste into the VM, not your Mac.

Configure: use the lab command below, not GitHub’s ./config.sh line. Paste your token from Step 1 and replace YOUR_USERNAME.

mkdir -p ~/actions-runner && cd ~/actions-runner

curl -o actions-runner-linux-x64-2.335.1.tar.gz -L \
  https://github.com/actions/runner/releases/download/v2.335.1/actions-runner-linux-x64-2.335.1.tar.gz

tar xzf ./actions-runner-linux-x64-2.335.1.tar.gz

./config.sh \
  --url https://github.com/YOUR_USERNAME/clearledger \
  --token YOUR_RUNNER_TOKEN \
  --name clearledger-runner \
  --labels clearledger,self-hosted,linux \
  --work _work \
  --unattended

sudo ./svc.sh install
sudo ./svc.sh start

Do not run GitHub’s ./run.sh for day-to-day use: the lab uses sudo ./svc.sh so the runner survives VM reboots. GitHub shows ./run.sh for a quick test only.

Expected after ./config.sh: Runner successfully added (or similar). If you see Invalid token or Expired token, go back to Step 1 in the browser and copy a fresh token.

The clearledger label is required GitHub’s default ./config.sh on the setup page doesn't add it. The workflow uses:

runs-on: [self-hosted, clearledger]

GitHub schedules jobs by runner labels, not by runner name. A runner named clearledger without the clearledger label will stay online but jobs will remain queued with Waiting for a runner to pick up this job.

What those last two commands mean:

sudo ./svc.sh install
  Registers the runner with systemd inside the VM.
  Without this, `sudo ./svc.sh status` says: not installed.

sudo ./svc.sh start
  Starts the runner service in the background.
  After this, it keeps running even when you close the terminal.

Check it locally from the same folder, still inside the VM:

cd ~/actions-runner
sudo ./svc.sh status

Expected: the service is installed and running.

If docker ps worked in Step 3 but a CI job later fails with Docker socket permission denied, the runner probably started before the docker group applied. Restart it after Step 4 (only when ~/actions-runner exists):

cd ~/actions-runner
sudo ./svc.sh stop
sudo ./svc.sh start
docker ps    # must work without sudo

Or, if you started the runner manually with ./run.sh instead of systemd:

cd ~/actions-runner
pkill -f "Runner.Listener|Runner.Worker|./run.sh" || true
nohup ./run.sh > _diag/manual-runner.log 2>&1 &
docker ps

If you see this:

not installed

then sudo ./svc.sh install didn't run successfully. Run:

cd ~/actions-runner
sudo ./svc.sh install
sudo ./svc.sh start
sudo ./svc.sh status

If install fails, rerun ./config.sh with a fresh GitHub runner token, then run the install/start commands again.

Step 5: Exit the VM

exit

Step 6: Verify the runner is connected

Go to github.com/YOUR_USERNAME/clearledger then to Settings, Actions, and Runners.

You should see clearledger-runner with a green dot and status Idle. Open the runner details and confirm the labels include:

self-hosted
Linux
X64
clearledger

If clearledger is missing, add it in the runner settings before rerunning the workflow. The runner name alone is not enough.

✋ Hands-on checkpoint: runner ready for jobs

Still on GitHub, Settings, Actions, and Runners, confirm:

Then trigger a dry run from your laptop:

git commit --allow-empty -m "test: verify runner picks up jobs"
git push

Open https://github.com/YOUR_USERNAME/clearledger/actions. Within 30 seconds a workflow run should show Queued then In progress, not stuck on “Waiting for a runner.” If it waits more than 2 minutes, the labels are wrong. Edit the runner on GitHub and add clearledger.

If it shows Offline:

multipass exec clearledger -- sudo systemctl status actions.runner.*.service
multipass exec clearledger -- journalctl -u actions.runner.*.service --lines=50

What you proved: GitHub can now send work into your local lab environment.

1.3: Create the Infra Repo on GitHub

Now separate application code from deployment state. Stage 1 introduces a second GitHub repository alongside the clearledger app repo you pushed in §1.1.

You'll use two repositories for the rest of the lab:

Think of clearledger as the question “What is the application?”. Python services, Dockerfiles, tests, and the CI workflow. Think of clearledger-infra as “What exact version should be running in Kubernetes right now?”. Deployments, Services, ingress rules, and the image tags that point at Docker Hub.

Teams split these on purpose. If you edit README.md in clearledger, that is a documentation change. It shouldn't trigger a deployment.
If you change auth-service code, the pipeline builds a new image (for example tag abc123) and, only after scans pass, records that tag in clearledger-infra:

image: $DOCKER_USERNAME/clearledger-auth-service:abc123

That line is a deployment contract: Git now says the cluster should run abc123. In Stage 1, the cluster doesn't change yet (and you'll prove that in §1.6).
In Stage 2, ArgoCD watches clearledger-infra, compares Git to what is running, and syncs the cluster when they differ. The app repo is where work begins. The infra repo is what production is supposed to look like.

Private repos: what syncs where

This lab uses two GitHub repos. clearledger is your main project repo: app code, CI pipeline, docs, policies, and lab files. This repo can be private.

clearledger-infra contains only Kubernetes manifests. ArgoCD watches this repo and uses it to deploy the app. For beginners, make this repo public so ArgoCD can read it without extra authentication.

The flow looks like this:

clearledger
app code + infra/manifests/
        ↓
CI copies infra/manifests/
        ↓
clearledger-infra
Kubernetes manifests only
        ↓
ArgoCD syncs from this repo
        ↓
Kubernetes cluster

ArgoCD doesn't read the main clearledger repo. It only reads clearledger-infra. If clearledger is private, that is fine. If clearledger-infra is private, you must give ArgoCD GitHub credentials later. If you do not, ArgoCD may show ComparisonError.

Create the infra repo on GitHub:

1. Go to GitHub and then New Repository

2. Name it clearledger-infra

3. Choose Public

4. Don't add a README

5. Click Create

Later, the CI pipeline will update clearledger-infra automatically. In Stage 1, the pipeline doesn't run kubectl apply – it updates Git. In Stage 2, ArgoCD reads that Git repo and applies it to the cluster.

Before pushing: set your Docker Hub username in Kustomize (image tags are resolved here, not in deployment YAML):

# Replace YOUR_DOCKERHUB_USERNAME with the same value as $DOCKER_USERNAME from §0.3
sed -i.bak "s/YOUR_DOCKERHUB_USERNAME/${DOCKER_USERNAME}/g" infra/manifests/kustomization.yaml
rm -f infra/manifests/kustomization.yaml.bak

Push only the Kubernetes manifests from infra/manifests/ (not everything under infra/):

mkdir -p /tmp/clearledger-infra
cp -r infra/manifests /tmp/clearledger-infra/
cd /tmp/clearledger-infra
git init
git remote add origin https://github.com/YOUR_USERNAME/clearledger-infra.git
git add . && git commit -m "feat: initial manifests" && git push -u origin main
cd -

✋ Hands-on checkpoint: infra repo on GitHub (do this before §1.4)

On your laptop:

grep "docker.io/${DOCKER_USERNAME}/" infra/manifests/kustomization.yaml | wc -l
grep YOUR_DOCKERHUB_USERNAME infra/manifests/kustomization.yaml || echo "OK: placeholder replaced"

Expected: first command prints 4 (four image lines). Second prints OK: placeholder replaced, not four lines still saying YOUR_DOCKERHUB_USERNAME.

In the browser, open https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests and confirm with your eyes:

Which folders matter? You only pushed infra/manifests/ to GitHub, that's correct. Everything else in this repo stays local for now.

Some manifests for later stages (network policies, Vault extras) live under infra/deferred-by-stage/ in the clearledger repo. You'll apply those by hand when you reach that stage. Do not copy that folder into clearledger-infra, or ArgoCD will deploy things too early.

You might notice stages/stage-1-ci-pipeline/ has no copy of the manifests. That is normal: the lab doesn't duplicate YAML there. The canonical copy is infra/manifests/ in this repo, and the live GitOps copy is clearledger-infra on GitHub.

What you proved: Kubernetes config now has its own repo and Git history, separate from application code. CI will update clearledger-infra after each build, and your app repo stays for code and the pipeline file.

1.4: Set up GitHub Secrets

Go to github.com/YOUR_USERNAME/clearledger and then Settings, Secrets and variables, Actions, and New repository secret.

The workflow needs credentials for Docker Hub, GitHub, and image signing:

• Docker Hub, so it can push images.

• GitHub, so it can push image tag updates into clearledger-infra.

• Cosign, so it can sign the images after pushing them.

Do not paste these values into YAML files. Store them as GitHub Actions secrets.

Secret 1, DOCKER_USERNAME

This is just your Docker Hub username.

Example:

veeno-demo

Get it from Docker Hub: hub.docker.com, profile menu, Account Settings.

Secret 2, DOCKER_PASSWORD

This should be a Docker Hub access token, not your normal Docker Hub password.

Create it here:

hub.docker.com
→ Account Settings
→ Security
→ New Access Token
→ Description: clearledger-github-actions
→ Access permissions: Read, Write, Delete or Read/Write
→ Generate

Copy the token immediately. Docker Hub only shows it once.

Secret 3, INFRA_REPO_TOKEN

This is a GitHub Personal Access Token (PAT). The pipeline uses it to push commits to the second repo, clearledger-infra.

Create it here:

GitHub profile settings
→ Settings
→ Developer settings
→ Personal access tokens
→ Tokens (classic)
→ Click "Generate new token"
→ Choose "Generate new token (classic)"
→ If GitHub asks for your password or 2FA, complete it
→ Note: clearledger-infra-ci
→ Expiration: choose a lab-friendly value
→ Select scope: repo
   This allows the pipeline to push to clearledger-infra.
→ Generate token

Copy the token immediately. GitHub only shows it once.

For this lab, repo scope is the simplest option. In production, you would use tighter permissions, such as a fine-grained token limited to only clearledger-infra.

Secrets 4 and 5, COSIGN_PRIVATE_KEY and COSIGN_PASSWORD

Cosign signs container images after the pipeline pushes them to Docker Hub. Later, Stage 4 uses the public key with Kyverno so the cluster can verify that images came from your trusted pipeline.

Generate the key pair on your host machine, not inside the Multipass VM:

# macOS: brew install cosign
# Linux/WSL2: curl -sSL -o cosign https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 && chmod +x cosign && sudo mv cosign /usr/local/bin/
cosign generate-key-pair

This creates:

cosign.key   # private key — never commit this
cosign.pub   # public key — keep for later Kyverno verification

When Cosign asks for a password, enter one and save it in your password manager. If you already generated a key without a password, regenerate it with a password for this lab.

Add these five secrets to the clearledger repo, not clearledger-infra:

Repository variables (not secrets) (optional) toggles for later stages. Add under Settings, Secrets and variables, Actions, Variables:

Don't add either variable in Stage 1. If you set them now, CI will try to refresh ArgoCD or run ZAP before the cluster is ready, and the pipeline output gets harder to read. The guide calls out the exact moment to turn each one on – you only need to remember that both exist.

What you proved: the pipeline can authenticate to external systems without hardcoding credentials in the repo.

1.5: Understand the Pipeline Before Activating it

Don't treat the workflow file as magic. Open .github/workflows/ci.yaml and read it before you run it.

The pipeline has two responsibilities:

1. Prove the code and images are safe enough to publish.

2. Update the infra repo with the new image tags.

Here's the security flow first:

Developer pushes code to GitHub
        ↓
GitHub Actions starts workflow
        ↓
Self-hosted runner inside the Multipass VM picks up the job
        ↓
1. Scan secrets (Gitleaks)
        ↓
2. Run code security scans (Semgrep) + IaC scan (Checkov) — parallel
        ↓
3. Prepare scanners (install Trivy/Syft/Grype/Cosign once; refresh Trivy DB once)
        ↓
4. BUILD: docker build all four services (local tags only; nothing hits Docker Hub yet)
        ↓
5. SCAN: Trivy on all images; Syft + Grype SBOM on auth-service; upload evidence
        ↓
6. PUBLISH: push to Docker Hub + Cosign sign (only if scan passed)
        ↓
7. UPDATE MANIFESTS: commit new image tags to clearledger-infra

Build, scan, publish (prod-style gates)

Real teams never push first and scan later. The pipeline separates three concerns into three jobs in .github/workflows/ci.yaml:

You do not run scripts/ci-publish-image.sh yourself before pushing code. GitHub Actions checks out the repo and calls it inside publish-images.

Why can build-images and scan-images be separate jobs? Each job is a fresh checkout on GitHub-hosted runners. They don't share a disk. On your self-hosted runner, all three jobs run on the same Multipass VM and use the same Docker engine.

Job 1 runs docker build and leaves the images on that machine. Job 2 runs Trivy against those same local images: no upload, no download. Job 3 pushes to Docker Hub only if the scan passed.

That's a practical lab setup: one persistent build machine with Docker installed, like a dedicated CI worker in a real office. In Stage 8 (AWS), the pipeline uses GitHub-hosted runners instead: there, build-images saves the images to a file (images.tar) and passes that file to the next job as a workflow artifact, because those runners are throwaway VMs with no shared Docker cache.

Then comes the GitOps handoff:

Secure images now exist in Docker Hub
        ↓
Runner checks out clearledger-infra from GitHub
        ↓
Deployment YAML image tags are updated
        ↓
Runner commits and pushes back to clearledger-infra
        ↓
Stage 1 ends here

Here's how the image tag ties to your code: every pipeline run is triggered by a git commit. GitHub gives that commit a unique ID called the SHA (a long hex string like a1b2c3d4e5f6789…). The workflow sets IMAGE_TAG to that SHA and uses it everywhere:

1. Build: docker build -t clearledger-auth-service:a1b2c3d4…

2. Publish: push to Docker Hub as YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4…

3. Update manifests: kustomize edit set image …:a1b2c3d4… in clearledger-infra

4. Commit message: ci: deploy a1b2c3d4… — all gates passed

If production is running YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4, you can copy that a1b2c3d4 tag, open GitHub, and instantly find the exact commit that built that image. There's no guessing and no wondering if latest changed. Every deployed image points back to one specific version of the code, making rollbacks and debugging much easier.

The Kustomize placeholder

auth-service/deployment.yaml uses a label instead of a real image address:

image: clearledger/auth-service:gitops

That label isn't on Docker Hub. It tells Kustomize where to substitute. The real address lives in kustomization.yaml:

images:
  - name: clearledger/auth-service          # matches the label above
    newName: docker.io/YOUR_DOCKERHUB_USERNAME/clearledger-auth-service
    newTag: abc123def456…                   # real commit SHA — CI writes this

When ArgoCD deploys, kustomize build swaps the label for the full address.

You edit kustomization.yaml once in §1.3 to set your Docker Hub username in newName:. After that, CI writes newTag: automatically on every green push. You never touch it by hand.

Stage 1: CI updates GitHub, not the cluster

After a green pipeline run, three things are true:

• New images exist on Docker Hub

• clearledger-infra on GitHub has new SHAs in kustomization.yaml

• Your Kubernetes cluster is unchanged. Still running whatever Stage 0 left there

CI never runs kubectl apply. It only commits to clearledger-infra. That's the whole Stage 1 lesson: build and scan are automated, but deploy is not: yet. Stage 2 installs ArgoCD, which reads clearledger-infra and updates the cluster for you.

Kubernetes Checkov runs in Stage 1 but does not block the pipeline. It uploads findings so you can see hardening work ahead. Stage 4 turns those kinds of rules into cluster enforcement with Kyverno.

Jobs run on your self-hosted runner (runs-on: [self-hosted, clearledger]). Both ENABLE_ARGOCD_SYNC and ENABLE_DAST are unset in Stage 1. See §1.4 for when each gets flipped.

If a job fails, start with docs/troubleshooting.md before editing the workflow.

Stage 1 security posture: what blocks vs what waits

Note that stage 1 is not “security off.” Some gates stop the pipeline while others run for evidence and tighten in later stages.

Blocks the pipeline today:

• Gitleaks (secrets in Git)

• Semgrep (SAST on Python)

• Checkov on Dockerfiles

• Trivy (fixable HIGH/CRITICAL CVEs in images)

• Grype on auth-service SBOM (fixable HIGH+)

• Manifest update to clearledger-infra (must succeed)

Runs but doesn't block yet:

• Checkov on Kubernetes manifests – enforced in Stage 4 (Kyverno)

• Cosign sign + SLSA attest – enforced in Stage 4 (unsigned images rejected)

• Syft SBOM generation – supply-chain evidence. You'll purposely break gates in Stage 3.

• ArgoCD refresh – Stage 2 (ENABLE_ARGOCD_SYNC=true)

• DAST / ZAP – Stage 3 (ENABLE_DAST=true)

If you forget which stage fixes what, search this guide for “Stage 1 security posture” or follow the stage order: Stage 3 breaks gates on purpose, Stage 4 connects Checkov findings to Kyverno, Stage 5 moves secrets off Git, Stage 6 adds runtime detection, Stage 7 adds monitoring dashboards.

Run make check-3 and make check-4 after those stages to confirm hardening landed.

Design intent: Stage 1 proves CI can build, scan, push, and update Git without you touching Docker manually. Later stages turn evidence into enforcement. The relaxations here are deliberate.

1.6: Activate the Pipeline

Run this in the clearledger app repo, not clearledger-infra.

§1.3 created clearledger-infra with only Kubernetes manifests. It has no .github/workflows/ and no pipeline. If your shell prompt says clearledger-infra, or you used /tmp/clearledger-infra, you're in the wrong place.

cd /path/to/clearledger    # the app repo you pushed in §1.1

git remote -v              # must show .../clearledger.git — NOT clearledger-infra

ls .github/workflows/ci.yaml   # must exist before you commit

The pipeline file already lives at .github/workflows/ci.yaml. Push any small change to clearledger on main:

echo "# Pipeline activated $(date)" >> README.md
git add README.md
git commit -m "ci: activate GitHub Actions pipeline"
git push origin main

Watch the run at: https://github.com/YOUR_USERNAME/clearledger/actions (app repo Actions tab, not the infra repo).

When the pipeline succeeds, it updates clearledger-infra for you. You don't need to push anything to the infra repo by hand for this step.

Expected Output: all jobs green in about 8 minutes.

✓ Build + Scan auth-service
✓ Build + Scan ledger-service
✓ Build + Scan notification-service
✓ Build + Scan frontend
✓ Update manifests → GitHub

DAST and the ArgoCD refresh step show as skipped: this is expected, because Both toggles are unset until later (see §1.4).

Note: this lab includes .gitleaksignore because some intentional demo secrets are already present in Git history. Gitleaks still runs normally. The ignore file only suppresses known lab fingerprints. Don't add new findings to it unless you've confirmed they're intentional test data.

Click into the job logs and look for the story. Don't just wait for green:

• Docker login succeeded

• Each service image built and pushed to Docker Hub

• clearledger-infra was checked out

• Deployment YAMLs were updated with the new SHA tag

• A commit was pushed back to clearledger-infra

After the pipeline succeeds, open https://github.com/YOUR_USERNAME/clearledger-infra and look at the deployment manifests. The image tags should now use the current commit SHA.

Now check the cluster:

kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}' && echo

You may still see the old image. That's expected. This is the most important learning in Stage 1:

GitHub pipeline succeeded.
Docker Hub has new images.
clearledger-infra has new image tags.
The Kubernetes cluster did not update automatically.

That's not a failure. It's the deployment gap. Stage 1 automated the build, but no controller is watching the infra repo yet. Stage 2 installs ArgoCD to close that gap.

1.7 — Hands-on Checkpoint: Prove Stage 1 is Really Done

Don't rely on a green workflow badge alone. Run each check yourself:

1. Infra repo still has app secrets (critical for Stage 2)

Open https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests/auth-service, secret.yaml must be visible.

On your laptop:

git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra
grep secretKeyRef /tmp/verify-infra/manifests/auth-service/deployment.yaml
grep secret.yaml /tmp/verify-infra/manifests/kustomization.yaml
rm -rf /tmp/verify-infra

Expected: secretKeyRef in deployment output. kustomization lists auth-service/secret.yaml and ledger-service/secret.yaml. If secrets are missing, re-push §1.3 manifests before Stage 2.

2. Kustomize image tags updated by CI

git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra
grep newTag /tmp/verify-infra/manifests/kustomization.yaml
rm -rf /tmp/verify-infra

Expected: newTag is a 40-character git SHA (or your commit hash), not still v0.1.0 only: unless you haven't pushed since §0.3.

3. Docker Hub has signed images from this pipeline

Open hub.docker.com then clearledger-auth-service then Tags. The latest tag should match the SHA from step 2.

4. (Cluster unchanged (deployment gap) intentional)

kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}' && echo

Expected: still your Stage 0 tag (for example, veeno-demo/clearledger-auth-service:v0.1.0), not the new SHA. That proves CI didn't touch the cluster.

5. Runner still idle

GitHub, Settings, Actions, Runners, clearledger-runner, Idle.

make check-1

All five pass, onto Stage 2.

What You Learned in Stage 1

• CI removes your laptop from the build process. Builds become repeatable, visible, and tied to Git commits.

• A runner is the worker, not the pipeline itself. GitHub schedules the job, the self-hosted runner executes it inside your VM.

• Artifacts and desired state are different things. Docker Hub stores built images. clearledger-infra on GitHub stores the Kubernetes manifests that say which image should run.

• Good pipelines don't secretly mutate clusters. This pipeline updates Git instead of running kubectl.

• The gap that remains: the infra repo changed, but the cluster didn't. Someone still has to apply the change manually. Stage 2 fixes that with GitOps.

What you can now put on your CV / say in an interview:

Built a CI pipeline on a self-hosted GitHub Actions runner that builds and pushes container images on every push, and can debug a workflow that fails before any job is created.

make snapshot STAGE=1 && make snapshots. Confirm clearledger.stage1. See How to Save Your Progress.

Stage 2 — GitOps with ArgoCD

From this point on, Git is in charge. Whatever is written in the infrastructure repository is what should be running. If someone changes the cluster by hand, ArgoCD notices the difference and changes it back to match Git.

Goal: Install ArgoCD so it watches clearledger-infra and deploys changes to the cluster. The CI pipeline only updates the Git repository, it never connects to Kubernetes or runs kubectl commands.

Here's a more conversational, compressed version:

Am I ready for Stage 2?

Before moving on, finish §1.6, then run:

make check-1

grep secretKeyRef infra/manifests/auth-service/deployment.yaml

grep vault.hashicorp infra/manifests/auth-service/deployment.yaml && echo "STOP: Vault annotations present" || echo "OK"

You should see:

• check-1 passes

• secretKeyRef is present

• OK (no Vault annotations yet)

Quick checklist:

• clearledger-infra contains auth-service/secret.yaml and ledger-service/secret.yaml

• Your self-hosted runner is Idle with the clearledger label

• ENABLE_ARGOCD_SYNC isn't set yet (you'll enable it after installing ArgoCD)

You're done with Stage 2 when make check-2 passes and http://argocd.local shows ArgoCD syncing clearledger.

Finally, save your progress:

make snapshot STAGE=2
make snapshots

Confirm that clearledger.stage2 appears in the snapshot list.

What You Need to Know First

The gap from Stage 1: CI already builds images and updates clearledger-infra. The cluster didn't change until someone ran kubectl. This stage closes that last step.

push code → CI updates clearledger-infra → ArgoCD syncs cluster

Pre-sync Checklist: Run Before argocd app sync

ArgoCD applies whatever is in clearledger-infra. Wrong content causes red pods. Re-run the §1.7 checkpoint table to confirm GitHub-side content is still correct, then verify the laptop side:

# Application manifest must point at YOUR infra repo
grep repoURL stages/stage-2-gitops/argocd/clearledger-app.yaml

# Stage 0 workloads still healthy before ArgoCD takes over
kubectl get pods -n clearledger
curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health

Expected: repoURL contains your GitHub username, all app pods Running, curl 200. Only when both pass should you install ArgoCD and sync below.

kubectl create namespace argocd 2>/dev/null || true

kubectl apply -n argocd --server-side --force-conflicts -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=argocd-server -n argocd --timeout=180s

Why --server-side --force-conflicts? Argo CD ships a very large applicationsets.argoproj.io CRD. A normal kubectl apply tries to stash the whole thing in an annotation, hits a 256 KiB limit, and errors with metadata.annotations: Too long. Server-side apply avoids that. It's how Argo CD expects you to install.

Get the admin password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d && echo

Configure Argo CD for your NGINX ingress

The browser talks HTTPS to ingress and ingress talks plain HTTP to the Argo CD server. Without this, the UI often breaks with 503 or ERR_TOO_MANY_REDIRECTS on live-update URLs (/api/v1/stream/*).

kubectl apply -f stages/stage-2-gitops/infra/argocd-cmd-params.yaml

kubectl apply -f stages/stage-2-gitops/infra/argocd-ingress.yaml

kubectl rollout restart deployment/argocd-server -n argocd

kubectl rollout status deployment/argocd-server -n argocd --timeout=180s

Expected in argocd-cmd-params-cm: server.insecure: "true", server.grpc.web: "true", server.url: https://argocd.local.

Open https://argocd.local. Login: admin and the password from above. Accept the self-signed certificate warning if the browser shows one.

Expected: The Applications page loads. In the browser console (F12 Console), you shouldn't see 401 or ERR_HTTP2_PROTOCOL_ERROR. If the UI looks fine in a normal window, you're done: incognito isn't required.

If login fails with 401 Unauthorized (often after a config change or a bad earlier login), try a private/incognito window or clear site data for argocd.local, then log in again. Still stuck? See troubleshooting.md. ArgoCD.

Connect ArgoCD to the infra repo and apply the Application manifest:

1. Edit stages/stage-2-gitops/argocd/clearledger-app.yaml

Set spec.source.repoURL to your infra repo (your GitHub username, not git config user.name).

2. Connect ArgoCD to your infrastructure repository:

This gives ArgoCD permission to watch clearledger-infra for new commits. Whenever the deployment manifests change, ArgoCD will update the cluster automatically.

# macOS: brew install argocd
argocd login argocd.local --username admin --password YOUR_PASSWORD --insecure --grpc-web

# Public repo
argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git --grpc-web

# Private repo — PAT from Stage 1 §1.4 (you saved it as GitHub secret INFRA_REPO_TOKEN)
export INFRA_REPO_TOKEN='ghp_...'   # paste here; GitHub only shows it once at creation
argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git \
  --username git --password "$INFRA_REPO_TOKEN" --grpc-web

Verify that Argo CD can reach the repo (do this before applying the Application):

argocd repo list --grpc-web

Look for your clearledger-infra URL with TYPE git and connection Successful. If it shows Failed or the repo is missing, Argo CD can't sync. Fix credentials before Stage 4 or any stage that depends on GitOps.

After a VM restore or Argo CD reinstall, you may need to run argocd repo add again (credentials are stored in the cluster, not in Git).

3. Apply and sync:

kubectl apply -f stages/stage-2-gitops/argocd/clearledger-app.yaml

argocd app sync clearledger --grpc-web

How to Read the Argo CD UI

After sync, open the clearledger application in the tree view. Three badges at the top tell you almost everything:

APP HEALTH: Healthy. Kubernetes thinks the workloads are running. Pods are up (or still starting if it says Progressing).

SYNC STATUS: Synced: the cluster matches clearledger-infra on GitHub at the commit shown (for example, main (2c88aa1)). Git is the source of truth and Argo CD applied it.

LAST SYNC: Succeeded: the most recent apply from Git worked. If this failed, click it for the error.

The resource tree below is the same app broken into pieces: namespace, secrets, services, deployments, ingress, and so on Green checkmarks = applied from Git. Click any box (for example, deploy/auth-service) then Live Manifest vs Desired to see what Argo CD thinks should run.

Quick "is the app actually working?" test (outside Argo CD):

curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health

200 = the app is reachable end-to-end, not only "green in Argo CD."

When something is wrong: HEALTH goes Degraded or Progressing for a long time, SYNC goes OutOfSync, and a resource in the tree turns red. Click that resource and then Events or Logs. The kubectl checks below double-check the same thing from the terminal.

Confirm ArgoCD is watching all workloads (not only ingress):

argocd app resources clearledger --grpc-web | grep Deployment

Pass looks like your output:

apps    Deployment    clearledger    auth-service            No
apps    Deployment    clearledger    frontend                No
apps    Deployment    clearledger    ledger-service          No
apps    Deployment    clearledger    notification-service  No
apps    Deployment    clearledger    redis                   No

This command shows the Deployments that ArgoCD is managing for ClearLedger. You should see auth-service, ledger-service, notification-service, frontend, and redis. That means ArgoCD reads the full kustomization.yaml from clearledger-infra/manifests, not just one file.

The last column is ORPHANED. No is good. It means ArgoCD knows this resource belongs to the ClearLedger app. You only need to worry if one of the Deployments is missing, or if ArgoCD shows OutOfSync, Degraded, or red resources in the UI.

✋ Hands-on checkpoint: first sync healthy

Run these four checks. Pass looks like this:

kubectl get pods -n clearledger
# Every app pod 1/1 Running (postgres/redis may show older RESTARTS from VM reboots — OK)

kubectl get application clearledger -n argocd \
  -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'
# sync=Synced health=Healthy

curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health
# 200

kubectl logs -n clearledger deploy/auth-service --tail=5 2>/dev/null | head -3
# Lines like: GET /health HTTP/1.1" 200 OK
# Bad sign: DATABASE_URL is not set

If all four pass then, Stage 2 first sync is done. Continue to Enable CI, and ArgoCD handoff below, then make check-2 and make snapshot STAGE=2.

How to Enable the CI to ArgoCD Handoff

In Stage 1, the pipeline updated clearledger-infra, but it didn't update the cluster. That was intentional.

Now ArgoCD is installed, so you can let the pipeline tell ArgoCD to check for changes after each successful run.

In GitHub, open your clearledger repo and go to Settings, Secrets and variables, Actions, Variables, and then New repository variable.

Add:

From now on, a green pipeline does two things:

1. Updates clearledger-infra with the new image tag

2. Asks ArgoCD to sync the cluster

If the pipeline can't trigger ArgoCD immediately, that's usually okay. ArgoCD checks clearledger-infra on its own every few minutes, so it should still pick up the new Git change.

Leave ENABLE_DAST unset for now. You enable that in Stage 3 after the app is stable at clearledger.local.

If the Argocd UI Shows Red Pods or "Progressing" (Read This Before the Screenshot)

This is a common first-sync surprise, not a broken install.

Why it happens in Stage 2

ArgoCD syncs whatever is in clearledger-infra. Deployments must use secretKeyRef (Stages 2–4), not Vault injection. If your infra repo has Vault annotations from an older lab copy, auth/ledger crash with DATABASE_URL is not set until Stage 5.

Network policies belong to Stage 6. In the main clearledger repo, they live in infra/deferred-by-stage/stage-6-runtime-security/netpol/, not in infra/manifests/. Don't copy them into clearledger-infra during Stage 2.

If manifests/netpol/ is still in your clearledger-infra repo on GitHub (from an older copy of the lab), ArgoCD will keep applying it. Those policies use default-deny and break DNS for new pods, so you see red 0/1 pods and Progressing health.

Fix for Stage 2

Do both steps. Deleting only in the cluster is not enough: ArgoCD recreates policies from Git on the next sync.

Step 1: remove from clearledger-infra on GitHub

Delete the folder manifests/netpol/ and commit: chore: defer network policies to Stage 6.

Step 2: sync and restart

argocd app sync clearledger --grpc-web
kubectl delete networkpolicy -n clearledger --all   # safe once Git no longer has netpol
kubectl rollout restart deployment/auth-service deployment/ledger-service -n clearledger
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"

Network policies stay in clearledger under infra/deferred-by-stage/ until you apply them in Stage 6.

When that looks good, continue below.

When ArgoCD finishes syncing, open the clearledger app in the ArgoCD UI. You should see green Healthy and Synced badges. The app should point to your clearledger-infra repo, use the manifests path, and deploy into the clearledger namespace.

Then open the app tile. The resource tree should show your deployments, services, and ingress with no red resources.

You can confirm the same thing from the terminal:

argocd app get clearledger --grpc-web

Look for Sync Status: Synced and Health Status: Healthy.

ArgoCD stuck OutOfSync

Normal path: CI copies full manifests + updates Kustomize tags, ArgoCD auto-syncs within ~3 minutes.

If still OutOfSync after 10+ minutes:

make fix-argocd

This re-syncs canonical manifests to clearledger-infra (Kustomize SHAs preserved), re-applies the Application, and triggers a hard refresh. Don't kubectl apply deployments: fix Git, let ArgoCD sync.

kubectl annotate application clearledger -n argocd 

argocd.argoproj.io/refresh=hard --overwrite

argocd app sync clearledger --grpc-web --prune

kubectl get application clearledger -n argocd -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'

Take a screenshot of that view: the app tile or the resource tree is fine. That’s your portfolio proof that GitOps is actually running.

Prove ArgoCD Self-Healing

Now prove that Git is the source of truth.

In this demo, you'll change the running cluster by hand. You will not change Git. ArgoCD should notice that the cluster no longer matches clearledger-infra, then change it back.

Before you start, make sure the app is healthy and ArgoCD is managing the deployments:

argocd app resources clearledger --grpc-web | grep Deployment

Manually change the auth-service image in the cluster:

# Manually change the image in the cluster only (Git stays the same)
kubectl set image deployment/auth-service \
  auth-service=$DOCKER_USERNAME/clearledger-auth-service:fake-tag \
  -n clearledger

Check ArgoCD:

# ArgoCD should flip to OutOfSync within a minute or two
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"

Wait for ArgoCD to fix the cluster. The fake image tag may briefly cause an image pull error. That's expected in this demo.

# Wait for selfHeal (default sync interval is ~3 minutes)
sleep 180

Confirm the image was changed back to the Git version:

# Cluster image should match clearledger-infra again — Git was never edited
kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

If the image changed back, ArgoCD self-healing worked. You changed the cluster by hand, but ArgoCD restored it to match clearledger-infra.

That's GitOps: Git says what should run, and ArgoCD keeps the cluster matching Git.

make check-2

How to Roll Back a Bad Deploy

You just proved that ArgoCD reverts unauthorized cluster changes. Now flip it: what if you pushed a bad commit yourself? GitOps rollback isn't a button. It's a Git operation. This section explains why, shows you both methods, and has you practice each one before you need them under pressure.

How you know you need to roll back

These symptoms appearing within minutes of a push to clearledger-infra point at a bad commit:

• Pods stuck in CrashLoopBackOff or Error. Check with kubectl get pods -n clearledger.

• kubectl logs <pod> -n clearledger --previous shows startup errors that weren't there before.

• ArgoCD health flips from Healthy to Degraded or stays on Progressing. Check with argocd app get clearledger --grpc-web.

• The app returns 5xx errors or login stops working. Check with curl -I http://clearledger.local/health.

If this happens right after a push, roll back first. Once the app is stable again, investigate the bad commit.

Why ArgoCD rollback isn't just a button

ArgoCD has a rollback button in the UI and an argocd app rollback command. Both work. But only if you understand the interaction with selfHeal.

Your Application (stages/stage-2-gitops/argocd/clearledger-app.yaml) is configured with:

syncPolicy:
  automated:
    selfHeal: true

ArgoCD keeps the cluster matched to Git. In this lab, Git means clearledger-infra.

If someone changes the cluster by hand, ArgoCD treats that as drift and changes it back to match Git.

This also affects rollback. The ArgoCD UI rollback changes the cluster, but it doesn't change Git. If clearledger-infra still points to the bad version, and self-heal can bring the bad version back.

The safer GitOps rollback is to change Git with git revert in clearledger-infra. Then ArgoCD syncs the cluster to the reverted, good version.

If you need an emergency UI rollback, turn off auto-sync first, roll back in ArgoCD, then fix Git afterward.

Method 1: Git revert (preferred, always try this first)

This is the GitOps way. You don't touch the cluster. You change Git, and ArgoCD syncs the fix.

When to use: You have a few minutes and can identify the bad commit in clearledger-infra.

How it works:

Bad commit pushed to clearledger-infra
        ↓
ArgoCD auto-synced it (cluster is now broken)
        ↓
You run: git revert <bad-commit> && git push
        ↓
ArgoCD auto-syncs the revert (cluster is fixed, selfHeal works with you)
        ↓
Git history shows the bad deploy AND the revert, full audit trail

Step-by-step:

# 1. Go to your clearledger-infra repo (wherever you cloned it)
cd ~/clearledger-infra  # adjust path if you cloned elsewhere
git pull                # make sure you are up to date

# 2. Find the bad commit
git log --oneline -10

# Output looks like:
# abc1234 update ledger-service image to v1.4.0   ← this broke prod
# def5678 update auth-service image to v1.3.1     ← was fine
# 9a1b2c3 add vault rotation cronjob

# 3. Revert it — this creates a NEW commit, it does not delete history
git revert abc1234 --no-edit

# 4. Push — ArgoCD picks it up automatically within ~3 minutes
git push

# 5. Confirm the cluster recovered
kubectl get pods -n clearledger
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy

This method is preferred because it fixes the source of truth: clearledger-infra.

After you push the revert, ArgoCD sees the new Git state and syncs the cluster to it. Nothing fights you because Git and the cluster are supposed to match.

It also leaves a clear history. Git shows the bad deploy, the revert, who made both changes, and when they happened. That's easier to debug, easier to review, and better for compliance.

Method 2: Emergency ArgoCD rollback (when the cluster is on fire)

Use this if the cluster is broken right now and you don't have time to push a Git fix. It pins the cluster to a previous known good deployment immediately. You'll still fix Git afterward. This isn't a permanent fix.

When to use: Incident in progress. Pods are crashing, users are affected, and you need the cluster back to a known good state in under 30 seconds.

Before you start: confirm your ArgoCD CLI session is still valid. If it expired, re-login first: an expired session will silently fail every command below.

argocd account get-user-info --grpc-web
# If you see "Unauthenticated", re-login:
ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

argocd login argocd.local --username admin --password "$ARGOCD_PASSWORD" \
  --insecure --grpc-web

Step 1: Disable auto-sync (critical). Skip this and selfHeal will undo your rollback within 3 minutes.

argocd app set clearledger --sync-policy none --grpc-web
# Confirm: automated sync is now off
argocd app get clearledger --grpc-web | grep "Sync Policy"
# Expected: Sync Policy: <none>

Step 2: Find the last known-good deployment ID

argocd app history clearledger --grpc-web

# Output looks like:
# ID   DATE                           REVISION
# 9    2026-06-05 10:12:00 +0000 UTC  abc1234  ← bad deploy (current)
# 8    2026-06-04 14:46:06 +0000 UTC  def5678  ← known good
# 7    2026-06-01 20:53:19 +0000 UTC  9a1b2c3

# Or check via kubectl (no argocd CLI needed):
kubectl get application clearledger -n argocd \
  -o jsonpath='{range .status.history[*]}{.id}{"\t"}{.deployedAt}{"\t"}{.revision}{"\n"}{end}'

Use the ID (the number on the left), not the SHA.

Step 3: Roll back to the good ID

argocd app rollback clearledger 8 --grpc-web

Step 4: Confirm the cluster is stable

kubectl get pods -n clearledger
# All pods should be Running

argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Sync Status:   OutOfSync  ← expected — cluster is at rev 8, Git is still at the bad HEAD
# Health Status: Healthy    ← this is what matters right now

OutOfSync is correct and expected at this point. The cluster is running the old good revision. Git still has the bad commit. You'll fix that next.

Step 5: Fix Git (don't leave it broken)

cd ~/clearledger-infra
git pull
git revert <bad-commit-sha> --no-edit
git push

Step 6: Re-enable auto-sync

argocd app set clearledger \
  --sync-policy automated \
  --self-heal \
  --auto-prune \
  --grpc-web

# Trigger an immediate sync so you do not wait for the next auto-check
argocd app sync clearledger --grpc-web

# Confirm everything is clean
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy

Never leave auto-sync disabled longer than the incident. It's your drift-detection and tamper-evidence mechanism: without it, unauthorized kubectl changes go undetected. Re-enable it the moment you push the Git fix.

Practise the rollback now (before you need it under pressure)

Don't wait for a real incident to run this for the first time. The steps below simulate a bad image tag deploy and walk you through Method 1 (the preferred path).

Step 1: Push a bad image tag to clearledger-infra

cd ~/clearledger-infra
git pull

# Edit manifests/notification-service/deployment.yaml
# Change the image tag to a tag that does not exist, e.g.:
#   image: docker.io/$DOCKER_USERNAME/clearledger-notification-service:broken-tag

# Commit and push it
git add manifests/notification-service/deployment.yaml
git commit -m "test: simulate bad deploy with nonexistent image tag"
git push

Step 2: Watch ArgoCD sync the bad state

# Give ArgoCD ~3 minutes to pick it up, or trigger immediately:
argocd app sync clearledger --grpc-web

# Watch the notification-service pod fail
kubectl get pods -n clearledger -w
# You will see: notification-service pod stuck in ImagePullBackOff or ErrImagePull

Step 3: Roll back using Method 1

cd ~/clearledger-infra

# Revert the bad commit
git revert HEAD --no-edit
git push

# ArgoCD will auto-sync — or trigger it:
argocd app sync clearledger --grpc-web

# Watch pods recover
kubectl get pods -n clearledger -w
# notification-service should return to Running

Step 4: Verify

argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy

kubectl get pods -n clearledger
# All pods Running, no ImagePullBackOff

You have now practised a rollback end-to-end. The git revert commit is permanently in the infra repo's history: a real audit record of a simulated recovery.

Quick reference

Use Method 1 (git revert) when:

• A bad image tag or manifest was pushed to clearledger-infra and you have a few minutes

• Any config change in the infra repo caused pods to break

• This is almost always the right answer. It's fast, safe, and leaves a clean audit trail.

Use Method 2 (emergency ArgoCD rollback) when:

• The cluster is broken right now, users are affected, and you need it stable in under 30 seconds

• You're not yet sure which commit caused the problem and need time to investigate: roll back to stabilise, then use git log to find the culprit, then fix forward with Method 1

Neither method applies when a pod is crashing but nothing was pushed to the infra repo recently. This isn't a rollback problem. Check kubectl logs, Vault connectivity, and network policies instead.

revisionHistoryLimit: 10 in stages/stage-2-gitops/argocd/clearledger-app.yaml means ArgoCD always has 10 previous deployments available for emergency rollback. Increase it if your release cadence is high.

What You Learned in Stage 2

• What GitOps means: Git is the single source of truth, and a tool enforces it

• What ArgoCD does: watches Git, compares it to the cluster, corrects drift automatically

• How the full flow works now: push code, CI builds image, CI updates infra repo, and ArgoCD syncs cluster.

• No one runs kubectl to deploy anymore. The pipeline updates Git, ArgoCD does the rest.

• How to roll back safely: git revert in the infra repo is the correct answer, while ArgoCD emergency rollback is the break-glass option. You must disable auto-sync first or selfHeal will silently undo it.

What you can now put on your CV / say in an interview:

Implemented GitOps with ArgoCD so cluster state is driven from Git, with drift detection, auto-sync, and a Git-based rollback of a bad deploy.

make snapshot STAGE=2 && make snapshots. Confirm clearledger.stage2. See How to Save Your Progress.

Stage 3 — Security Gates

Every push runs security checks. Some failures stop the pipeline right away. Others you learn from now and enforce in the cluster later (Stage 4).

Goal: understand six scanners: what each one looks at, what it catches, and how to read a failure. You'll break each gate on purpose (§3.4) so a failed CI job isn't a surprise.

Ready for Stage 3?

• make check-2 passes

• ENABLE_ARGOCD_SYNC=true on GitHub (you set this in Stage 2)

• ENABLE_DAST still unset (turn on later in this stage if you want)

• Argo CD at http://argocd.local shows Synced

• Optional: skim Stage 1 security posture. Stage 1 already ran many of these tools

Done when: make check-3 passes and you triggered each gate once (§3.4). Then make snapshot STAGE=3 and make snapshots.

What You Need to Know First

One tool isn't enough. Each scanner guards a different layer:

• Gitleaks: secrets in code or Git history (API keys, tokens)

• Semgrep (SAST): bugs in your Python/JS source (injection, unsafe patterns)

• Trivy (SCA + images): finds known security vulnerabilities (called CVEs) in your Python/Node.js packages and Docker images. A CVE (Common Vulnerabilities and Exposures) is a publicly tracked software security flaw with a unique identifier.

• Checkov (IaC): misconfigurations in Dockerfiles, Kubernetes manifests, and Stage 8 Terraform.

• Cosign: proves images were built and signed by your pipeline

What blocks CI today: secrets, bad code (SAST), vulnerable images, Dockerfile issues on production images.

What waits for later: Some Kubernetes issues are only reported in Stage 1. They show you what still needs hardening. In Stage 4, Kyverno turns the important rules into real cluster enforcement, so unsafe workloads are blocked before they run.

When you git commit, hooks on your laptop can scan first (pre-commit). When you git push, GitHub Actions scans again on the runner. Same idea twice: catch mistakes before they waste a 10-minute pipeline. Pre-commit is optional to install, as CI always runs on push either way.

Enable DAST (Optional After Stage 2)

DAST (Dynamic Application Security Testing) scans the running app at http://clearledger.local. It was off in Stages 1–2 on purpose: Stage 1 never deployed to the cluster, and Stage 2 was about getting GitOps healthy first.

If make check-2 passes and curl http://clearledger.local/auth/health returns 200, you can turn DAST on:

Go to GitHub and into your clearledger repo. Then go to Settings, Secrets and variables, Actions, Variables, and New repository variable:

Push a small commit (or re-run the last workflow on main). The DAST (OWASP ZAP + fintech API tests) job should run instead of skipped. A failed ZAP scan is a real finding to investigate. Skipped before this step only means the toggle was off.

3.1: Install Pre-commit Hooks

# macOS (Homebrew — avoids PEP 668 "externally-managed-environment" from pip3):
brew install pre-commit

# Linux/WSL2:
# sudo apt install -y pre-commit
# or: python3 -m pip install --user pre-commit

pre-commit install
pre-commit run --all-files

If a hook fails, read the error first. Gitleaks and Ruff should pass before you commit. Some YAML or Terraform hook issues may come from later-stage files. If that happens, continue with the stage instructions and use troubleshooting.md for Gitleaks or CI scanner failures.

Test it catches secrets locally before CI does:

echo 'AWS_SECRET = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' >> app/auth-service/main.py
git add app/auth-service/main.py && git commit -m "test"

# Gitleaks fires and blocks the commit — see "What you should see" below

git restore --staged app/auth-service/main.py

git checkout app/auth-service/main.py

The commit was blocked before it even reached Git. If the pre-commit hook wasn't installed, that fake AWS key would be in your Git history permanently (even if you delete the line later, Git remembers).

If you already did Cosign in Stage 1, that counts. Stage 3 doesn't require regenerating keys. Confirm that infra/cosign.pub exists and GitHub has COSIGN_PRIVATE_KEY + COSIGN_PASSWORD. Stage 4 turns signing into enforcement at the cluster gate.

✋ Hands-on checkpoint: pre-commit actually blocks a secret

Installed-but-not-wired is the classic silent failure. Prove the hooks fire:

echo 'AWS_SECRET='"$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')" > leak-test.env

git add leak-test.env

pre-commit run --all-files; echo "exit=$?"

git reset leak-test.env >/dev/null; rm -f leak-test.env

Expected: the secret-scanning hook fails the run (exit=1) and flags leak-test.env. If exit=0, your hooks are installed but not catching anything: re-run pre-commit install and confirm .git/hooks/pre-commit exists.

If you skip this, commits sail through unscanned and you'll believe Stage 3 is protecting you when it's not.

3.2: Generate Cosign Keys

If you created Cosign keys in Stage 1 (§1.4), skip generation: go straight to inserting cosign.pub into the Kyverno policy and adding the GitHub secrets below.

Cosign signs your Docker images with a cryptographic key. When you deploy to the cluster, Kyverno (Stage 4) can verify the signature and reject any image that wasn't signed by your pipeline. This prevents someone from pushing a malicious image to your Docker Hub and having the cluster run it.

# macOS: brew install cosign

# Linux/WSL2: curl -O -L https://github.com/sigstore/cosign/releases/download/v2.2.4/cosign-linux-amd64 && chmod +x cosign-linux-amd64 && sudo mv cosign-linux-amd64 /usr/local/bin/cosign

cosign generate-key-pair   # enter a password when prompted

This creates two files: cosign.key (private, used by the pipeline to sign) and cosign.pub (public, used by Kyverno to verify).

Insert your public key into the Kyverno policy (replace the placeholder block in infra/policies/require-signed-images.yaml with the contents of cosign.pub).

Add secrets to GitHub (github.com/YOUR_USERNAME/clearledger → Settings → Secrets and variables → Actions):

✋ Hands-on checkpoint: Cosign keys are ready

Stage 4 uses cosign.pub to verify signed images. Before you continue, confirm the key files exist and the private key isn't tracked by Git:

test -f cosign.key && echo "private key present"
test -f cosign.pub && echo "public key present"
grep -q "BEGIN PUBLIC KEY" cosign.pub && echo "public key valid"
git check-ignore cosign.key && echo "private key correctly ignored"

Expected: all four lines should print.

If git check-ignore cosign.key prints nothing, add cosign.key to .gitignore before committing anything. The private key must stay out of Git.

Don't skip this check. Stage 4 needs the public key for the Kyverno image-signing policy, and the private key must remain local.

3.3: Activate the Full Security Pipeline

The security gates are already in .github/workflows/ci.yaml. Push any change to trigger the full pipeline:

git add . && git commit -m "ci: full DevSecOps pipeline" && git push origin main

3.4: Break Each Gate on Purpose

For each gate, you'll want to break something on purpose, read how the tool reports it, revert, and confirm green again. Try the local command first, then push once if you want a screenshot on GitHub Actions.

# 1. Break it   2. Run locally or push   3. Read the failure
# 4. git checkout -- path/to/file   5. pre-commit run --all-files (optional)   6. git push

Start with Gate 1 end-to-end before the others.

Gate 1: Gitleaks (secrets)

Inject: hardcoded AWS key in any Python file.

The goal is to prove the secret scanner works.

This command adds a fake AWS-looking key to app/auth-service/main.py:

echo 'AWS_KEY = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' >> app/auth-service/main.py

git add app/auth-service/main.py && git commit -m "test: trigger gitleaks"
# pre-commit blocks this commit locally — that is the test.
# For a CI screenshot only: git commit --no-verify -m "test: trigger gitleaks" && git push

Done looks like this (terminal: pre-commit):

🔑 Secrets scan (Gitleaks)...............................................Failed
- hook id: gitleaks
- exit code: 1

Finding:     AWS_KEY = "REDACTED"
RuleID:      aws-access-token
File:        app/auth-service/main.py
Line:        316

Expected: the commit should fail. Gitleaks should report one secret finding in app/auth-service/main.py.

That failure is good. It means the local pre-commit hook caught the secret before it reached Git.

Revert:

git restore --staged app/auth-service/main.py 2>/dev/null
git checkout app/auth-service/main.py
pre-commit run gitleaks --all-files   # → Passed

Gate 2: Semgrep (SAST)

Local dry-run (no repo change):

python3 -m venv /tmp/sec-gates-venv && /tmp/sec-gates-venv/bin/pip install semgrep
cat > /tmp/semgrep-bad.py << 'EOF'
import subprocess
from fastapi import Request
def bad(request: Request):
    subprocess.run(request.query_params.get("cmd"), shell=True)
EOF
/tmp/sec-gates-venv/bin/semgrep \
  --config=p/python --config=p/security-audit --config=p/owasp-top-ten --error \
  /tmp/semgrep-bad.py

Break CI: add a temporary file Semgrep will scan, commit, and push:

cat > app/auth-service/gate_test_semgrep.py << 'EOF'
import subprocess
from fastapi import Request
def bad(request: Request):
    subprocess.run(request.query_params.get("cmd"), shell=True)
EOF

git add app/auth-service/gate_test_semgrep.py && git commit -m "test: trigger semgrep" && git push

Expected result: Semgrep reports subprocess-shell-true as Blocking. The SAST (Semgrep) job turns red, and the image build jobs don't run.

Revert:

rm -f app/auth-service/gate_test_semgrep.py
git add -A && git commit -m "revert: semgrep gate test" && git push

Gate 3: Checkov (IaC / Dockerfile)

Checkov scans Dockerfiles and Kubernetes manifests for unsafe configuration.

First, run a local demo. This removes the HEALTHCHECK from a copied Dockerfile and shows how Checkov reports it:

python3 -m venv /tmp/sec-gates-venv && /tmp/sec-gates-venv/bin/pip install checkov
sed '/^HEALTHCHECK/,+1d' app/auth-service/Dockerfile > /tmp/Dockerfile-nohc
mkdir -p /tmp/checkov-demo/app/auth-service
cp /tmp/Dockerfile-nohc /tmp/checkov-demo/app/auth-service/Dockerfile
/tmp/sec-gates-venv/bin/checkov --directory /tmp/checkov-demo --framework dockerfile

Now trigger a Checkov finding in CI by exposing SSH port 22 in the auth-service Dockerfile:

echo 'EXPOSE 22' >> app/auth-service/Dockerfile
git add app/auth-service/Dockerfile && git commit -m "test: trigger checkov" && git push

Expected result: the Checkov log or artifact should show CKV_DOCKER_1, which means an SSH port was exposed.

The IaC Scan (Checkov) job may or may not turn red, depending on the severity Checkov assigns. That's okay for this exercise. The goal is to find and understand the Checkov result.

If you need a screenshot of a failed GitHub Actions job, use Gate 1, Gate 2, or Gate 4. Those are designed to turn the workflow red. Checkov is mainly for reading the finding, so it may stay green.

Revert:

git checkout app/auth-service/Dockerfile
git commit -am "revert: checkov gate test" && git push

Gate 4: Trivy (image CVEs)

Local dry-run: scan an old base image (no build):

trivy image --exit-code 1 --severity CRITICAL,HIGH --ignore-unfixed python:3.8-slim

Break CI: pin an old base in the Dockerfile, push, wait for Scan images:

sed -i.bak 's/FROM python:3.13-slim/FROM python:3.8-slim/' app/auth-service/Dockerfile
git add app/auth-service/Dockerfile && git commit -m "test: trigger trivy" && git push

Pass: Scan images → Trivy scan all images exits 1 with a CVE table (HIGH / CRITICAL). Publish images and Update Manifests are skipped.

Revert:

git checkout app/auth-service/Dockerfile
git commit -am "revert: trivy gate test" && git push

3.5: When a Scan Fails on a CVE You Didn't Inject

§3.4 is deliberate. This section is for the other case where you push normal code, but the image scan fails because a new vulnerability was found.
That's normal. CVE databases update all the time. Don't weaken the scan. Fix the vulnerable package or image.

First, find the real CVE. In GitHub Actions, open Scan images then go to Trivy scan all images and look for the table with:

• Package

• CVE

• Installed version

• Fixed version You can also download the artifact:

Ignore this red herring at the bottom of the log:

Version 0.71.2 of Trivy is now available
Error: Process completed with exit code 1.

The version notice doesn't fail the job. A fixable HIGH/CRITICAL CVE does. Don't add --skip-version-check to “fix” it.

Instead, fix it with:

• pip package: bump to the Fixed Version in requirements.txt (example: python-multipart==0.0.30 for CVE-2026-53539). Apply the same bump to sibling services if they share that pin.

• OS package: newer base image or a targeted apt/apk upgrade in the Dockerfile.

• No stable fix yet documented exception only: add the CVE to .trivyignore and .grype.yaml with a comment (see CVE-2026-7210).

Don't remove --exit-code 1, lower the severity rule, or disable scanning. For help, see Trivy version notice and Trivy blocks Python service images.

Finish Stage 3

For screenshots, use one clear failed gate:

• Gitleaks: Secrets Scan

• Semgrep: SAST

• Trivy: Scan images

• Checkov: look for CKV_* in the log or artifact. The job may stay green

After each test in §3.4, undo the test change, push the revert, and confirm the workflow is green again. One red GitHub Actions screenshot is enough for your portfolio.

Run the stage check:

make check-3   # must end: All checks passed. Ready for the next stage.

Expected: All checks passed. Ready for the next stage.

You should also have triggered at least one gate in §3.4. A local Gitleaks failure counts.

ENABLE_DAST=true is optional. You only need it if you want to run ZAP later.

Not required yet: Checkov blocking Kubernetes manifests or Cosign blocking deployments. Stage 4 turns those into cluster enforcement with Kyverno.

Next, save your progress:

make snapshot STAGE=3 && make snapshots

Stage 4 — Admission Control (Kyverno)

Even if CI passes, the cluster can still refuse.

CI scans your code and images before they reach GitOps, but it can't watch everything that happens inside the cluster. Someone with kubectl access could apply a manifest directly.

A Helm chart you install might create pods that violate your security standards. Those paths never hit the pipeline, which is why Stage 4 adds admission control: a checkpoint built into Kubernetes itself.

Every time something tries to create or update a resource, the request passes through admission webhooks before it takes effect. If a webhook rejects the request, the resource is never created.

Kyverno is a Kubernetes-native policy engine that uses those webhooks. You write policies as YAML files (not application code), and Kyverno enforces them on every matching resource in the cluster, for example, rejecting any pod that runs as root or requiring CPU and memory limits on every container.

The difference from CI is timing: CI scans before code ships, while Kyverno enforces at the cluster gate. Together they give you two layers of defense.

Your goal in this stage is to install Kyverno, apply the policies in infra/policies/, and prove in §4.4 that non-compliant pods are denied before the container runtime ever sees them.

Before you start, make sure the foundation from earlier stages is still solid: make check-3 should pass (pre-commit hooks and CI security gates are active), infra/cosign.pub should exist from Stage 3, and ArgoCD should still be syncing so the app responds at http://clearledger.local. If any of those are red, fix them first: Kyverno sits on top of a healthy cluster, not a broken one.

You're done with Stage 4 when all three break-it scenarios in §4.4 are denied and make check-4 passes.

What changes from Stage 3 is enforcement, not scanning. In CI, Checkov reported Kubernetes misconfigurations but didn't block the pipeline. Kyverno now stops those same classes of problems at the cluster gate.

Cosign has been signing your images since Stage 1. Kyverno now requires that signature before a ClearLedger image can deploy. This is where Stage 1 evidence becomes enforcement. See that section if you want the full map of what blocked in Stage 1 versus what waited for Stage 4.

Start with §4.1 to install Kyverno. If the install, policies, break-it scenarios, or make check-4 fail, read troubleshooting.md and Stage 4: Admission Control (Kyverno) before changing Helm values or policy YAML.

What Kyverno Enforces

All policy files live in infra/policies/. Kyverno itself is installed via Helm using stages/stage-4-admission-control/infra/kyverno/values.yaml.

Platform Stability: From Stage 4 Onward

From Stage 4 on, you're running more controllers on a single-node VM. Kyverno, storage provisioners, and later Prometheus and Loki. A pod can show Running while it's actually crash-looping in the background.

When platform pods (Kyverno controllers, hostpath-provisioner, the Prometheus operator, and similar) accumulate high RESTARTS, the API server starts timing out, kubectl feels flaky, and you can waste days debugging the wrong component because the app pods look fine.

After every stage from here on, give the cluster about ten minutes to settle, then run the stage health check:

bash scripts/health-check.sh <stage>    # for example, 4, 7, 7.5
# or the Makefile shortcut:
make check-4

The script ends with a Platform stability section that flags pods with suspicious restart counts. You can also scan the worst offenders yourself. This lists the fifteen pods with the highest restart counts cluster-wide, which is useful when something feels slow but you're not sure which namespace is struggling:

kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount' \
  | tail -15

The gate: Kyverno controllers and other platform pods should show RESTARTS under 5 after the stage settles. If any platform pod is climbing past 10, stop and fix it with the documented Helm values or troubleshooting.md. Don't kubectl patch around it and move on. A stable platform layer is a prerequisite for every stage that follows.

4.1: Install Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm upgrade --install kyverno kyverno/kyverno \
  --version 3.2.8 \
  --namespace kyverno \
  --create-namespace \
  -f stages/stage-4-admission-control/infra/kyverno/values.yaml \
  --wait --timeout=600s

The values file does three important things for the lab:

1. Disables cleanup CronJobs: older Kyverno charts pull bitnami/kubectl, which was removed from Docker Hub and causes ImagePullBackOff on cleanup pods.

2. Points Helm hooks at bitnamilegacy/kubectl, so future helm uninstall doesn't hang on a missing image.

3. Extends liveness probe timeouts: the default timeoutSeconds: 5, failureThreshold: 2 is too tight for a loaded single-node VM. Under CPU pressure, the health endpoint can take >5s to respond, which triggers a restart cascade that saturates the node and makes the API server intermittently unreachable. The values file sets timeoutSeconds: 30, failureThreshold: 5 so Kyverno survives load spikes without crash-looping.

What you should see:

Release "kyverno" does not exist. Installing it now.
NAME: kyverno
NAMESPACE: kyverno
STATUS: deployed
...
Kyverno version: v1.12.6

Verify all four controllers are running (first pull can take several minutes on a slow connection):

kubectl get pods -n kyverno

NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-bd685cd4b-f6kl6     1/1     Running   0          2m
kyverno-background-controller-66fcfc6d87-59wgt   1/1     Running   0          2m
kyverno-cleanup-controller-5c5bf8bc6b-7kspq      1/1     Running   0          2m
kyverno-reports-controller-5cdd6f4c48-qf5wc      1/1     Running   0          2m

If pods stay in ContainerCreating for a long time, the node is still pulling images from ghcr.io/kyverno. Wait. Don't start a second Helm install on top of a partial one.

Stability gate: Kyverno install only (before §4.2):

Before continuing, make sure the Kyverno pods are healthy:

kubectl get pods -n kyverno

Expected: the Kyverno controller pods show 1/1 Running, with low restart counts such as 0, 1, or 2, and the restart count isn't increasing.

Don't run make check-4 yet. That check also looks for the policies you apply later in §4.3, so it may fail at this point even if Kyverno installed correctly.

4.2: Confirm your Cosign Public Key is in the Policy

Stage 3 created infra/cosign.pub. Kyverno uses that same key to verify image signatures when a pod is created. The policy file ships with a placeholder. You must replace it with your key before applying policies in §4.3.

Step 1: Show your key (run from the repo root on the VM)

cd ~/clearledger    # or wherever you cloned the repo
cat infra/cosign.pub

You should see three lines: -----BEGIN PUBLIC KEY-----, a long base64 line, and -----END PUBLIC KEY-----. Copy that whole block (you'll paste it in the next step).

Step 2: Paste the key into the policy

Open infra/policies/require-signed-images.yaml in your editor (nano, vim, or VS Code).

Find this line:

                      PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE

Delete only that placeholder line and paste the three lines from cosign.pub in its place. The result should look like this (your base64 line will differ):

                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                     JFkwEwYHKoZIzj0CAQYIKoFIzj0DAQcDQgZEI...
                      -----END PUBLIC KEY-----

Save the file. Keep the pasted key indented under publicKeys: |-. The BEGIN PUBLIC KEY and END PUBLIC KEY lines should have spaces before them, just like the base64 line between them.

Step 3: Verify (three quick checks)

Run these one at a time from the repo root:

# Check A — placeholder must be gone
grep PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE infra/policies/require-signed-images.yaml \
  && echo "❌ FAIL: placeholder still in file — edit and save again" \
  || echo "✓ OK: placeholder removed"

# Check B — key block must be present exactly once
grep -c "BEGIN PUBLIC KEY" infra/policies/require-signed-images.yaml

Expected output for Check B: 1 (if you see 0, the key was not pasted. If 2, you pasted it twice).

# Check C — policy key must match cosign.pub byte-for-byte
diff infra/cosign.pub \
  <(sed -n '/-----BEGIN PUBLIC KEY-----/,/-----END PUBLIC KEY-----/p' \
      infra/policies/require-signed-images.yaml | sed 's/^[[:space:]]*//')

Expected output for Check C: nothing. No diff lines means the keys match. If diff prints differences, open the policy file and fix the paste.

If all three passed, continue to §4.3.

If you skip this, Scenario 3 in §4.4 fails in a confusing way: unsigned images may slip through, or signed pods may be rejected because Kyverno is checking against the wrong key.

4.3: Apply the Five Core Policies

Now you'll apply the five policies that map to CIS controls. Don't apply verify-slsa-provenance.yaml yet. It's an optional SLSA attestation policy (Audit mode) for a later enhancement.

Stage 4 applies infra/policies/require-signed-images.yaml. This policy uses failurePolicy: Fail, so if Kyverno can't verify an image signature, the pod is blocked instead of allowed. The ECR policy with failurePolicy: Ignore is for Stage 8, not this step.

kubectl apply \
  -f infra/policies/disallow-root.yaml \
  -f infra/policies/disallow-privilege-escalation.yaml \
  -f infra/policies/drop-all-capabilities.yaml \
  -f infra/policies/require-resource-limits.yaml \
  -f infra/policies/require-signed-images.yaml

Wait a few seconds, then confirm all policies show READY: True and VALIDATE ACTION: Enforce:

kubectl get clusterpolicy

NAME                            ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE
disallow-privilege-escalation   true        true         Enforce           True    10s
disallow-root-containers        true        true         Enforce           True    10s
drop-all-capabilities           true        true         Enforce           True    10s
require-resource-limits         true        true         Enforce           True    10s
require-signed-images           true        false        Enforce           True    10s

If READY stays empty, check Kyverno logs: kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller --tail=50.

4.4: Breaking it on Purpose

Now you'll test the policies by trying to create bad pods.

These pods are supposed to fail. That's the point.

CI tools like Checkov warn you in a report. Kyverno goes further: it blocks unsafe pods before Kubernetes runs them.

For each test, read the error message, as it should tell you which policy blocked the pod and what field was wrong. That error message is your proof that admission control is working.

Scenario 1: root container (no securityContext)

What you're simulating: Someone with kubectl access bypasses CI and applies a minimal pod: no securityContext, no resource limits.

This is exactly what Stage 1 Checkov flagged as evidence. Stage 4 now blocks it.

What's wrong with this manifest: The container has only a name and image. It will run as root by default, keep all Linux capabilities, and has no CPU/memory bounds.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: root-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
EOF

What you should see:

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/root-test was blocked due to the following policies

disallow-privilege-escalation:
  check-allowPrivilegeEscalation: 'validation error: allowPrivilegeEscalation must
    be set to false. rule check-allowPrivilegeEscalation failed at path /spec/containers/0/securityContext/'
disallow-root-containers:
  check-runAsNonRoot: |-
    validation error: Root containers are blocked in the clearledger namespace. Set securityContext.runAsNonRoot: true on the pod or container.
    . rule check-runAsNonRoot failed at path /spec/containers/0/securityContext/
drop-all-capabilities:
  check-capabilities: 'validation error: All containers must drop ALL capabilities.
    rule check-capabilities failed at path /spec/containers/0/securityContext/'
require-resource-limits:
  check-resources: 'validation error: Resource requests and limits are required for
    all containers. rule check-resources failed at path /spec/containers/0/resources/limits/'

How to read this output:

The important line is:

resource Pod/clearledger/root-test was blocked due to the following policies

That means Kyverno stopped the pod before it was created.

Under that line, Kyverno lists every policy the pod failed. For example:

disallow-root-containers:
  check-runAsNonRoot:

This means the pod failed the disallow-root-containers policy, specifically the check-runAsNonRoot rule. The fix is also shown in the message:

Set securityContext.runAsNonRoot: true

The same pattern applies to the other policies:

• disallow-privilege-escalation means the pod didn't set allowPrivilegeEscalation: false

• drop-all-capabilities means the pod didn't drop Linux capabilities with capabilities.drop: [ALL]

• require-resource-limits means the pod didn't set CPU and memory requests/limits

The path part tells you where Kubernetes expected the missing setting. For example, /spec/containers/0/securityContext/ means: look inside the pod spec, then the first container, then its securityContext.

And /spec/containers/0/resources/limits/ means: look inside the first container's resource limits.

So this one bad pod failed four controls at once. That's the lesson: Kyverno doesn't just say "no." It tells you which policy failed and where to fix the YAML.

Verify enforcement worked:

kubectl get pod root-test -n clearledger
# Error from server (NotFound): pods "root-test" not found

If you see a pod in Running or Pending, policies aren't enforcing: re-check that kubectl get clusterpolicy shows all five READY: True.

Take a screenshot. This is portfolio evidence for CIS Kubernetes Benchmark 5.2.6: enforced, not just configured.

Scenario 2: missing resource limits

What you're simulating: A developer who read the securityContext requirements and fixed root/caps/privilege. But skipped resource limits.

This is common in real teams: “we hardened the container” but forgot CPU/memory bounds.

What's wrong with this manifest: securityContext is correct, but there's no resources.requests or resources.limits. A container without limits can starve other workloads on the node.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nolimits-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
EOF

What you should see:

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/nolimits-test was blocked due to the following policies

require-resource-limits:
  check-resources: 'validation error: Resource requests and limits are required for
    all containers. rule check-resources failed at path /spec/containers/0/resources/limits/'

Key observation: Only one policy fires this time: the securityContext fields satisfied the other four rules. Kyverno evaluates rules independently. Each container property is a separate gate.

Verify:

kubectl get pod nolimits-test -n clearledger
# Error from server (NotFound): pods "nolimits-test" not found

Scenario 3: unsigned ClearLedger image

What you're simulating: A supply-chain attack: someone pushes a malicious image to Docker Hub under your repo name (clearledger-auth-service) without going through your signed CI pipeline. Stage 3 made Cosign signing possible, while Stage 4 makes it mandatory at the cluster gate.

Why this setup is needed: Kyverno checks image signatures against the image in Docker Hub, not against images on your laptop. The test image tag must exist in Docker Hub first.

If you use a fake tag like :unsigned that was never pushed, Kubernetes may fail later with ImagePullBackOff. That only means the image can't be pulled; it doesn't prove Kyverno blocked an unsigned image.

Step 1: push a deliberately unsigned test image (one-time):

export DOCKER_USERNAME=your-dockerhub-username

docker pull nginx:alpine
docker tag nginx:alpine ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
docker push ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test

# Must fail — proves the image has no Cosign signature from your pipeline key:
cosign verify --key infra/cosign.pub \
  index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
# Error: no signatures found

Step 2: try to deploy it with a compliant pod spec:

The pod manifest is fully hardened (securityContext + limits) so only the signature policy can fail. Use index.docker.io/ in the image URL: on Kyverno 1.12, docker.io/... may not trigger verifyImages matching.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
      resources:
        requests:
          memory: "64Mi"
          cpu: "50m"
        limits:
          memory: "128Mi"
          cpu: "200m"
EOF

What you should see:

Error from server: error when creating "STDIN": admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/unsigned-test was blocked due to the following policies

require-signed-images:
  verify-cosign-signature: 'failed to verify image index.docker.io/veeno-demo/clearledger-auth-service:unsigned-test:
    .attestors[0].entries[0].keys: no signatures found'

How to read this output:

• Note the webhook name is mutate.kyverno.svc-fail, not validate: image verification runs in Kyverno’s mutate pass (digest + signature check) before the pod is admitted.

• no signatures found means Kyverno reached Docker Hub, found the image, and confirmed it was not signed with your infra/cosign.pub key.

• The pod never exists: the attacker can't get a shell even if the image is pullable.

Verify:

kubectl get pod unsigned-test -n clearledger
# Error from server (NotFound): pods "unsigned-test" not found

What you should NOT see (these mean the test didn't prove signature enforcement):

Contrast: signed image is allowed:

The previous test used an unsigned image, so Kyverno blocked it.

Your real ClearLedger images should be signed by the CI pipeline. If the pod also follows the security rules, Kyverno allows it to run.

You can check the image currently used by auth-service:

# Your deployed tag (signed in CI) should start if spec is compliant:
kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
# docker.io/veeno-demo/clearledger-auth-service:v0.1.0

Example output:

docker.io/veeno-demo/clearledger-auth-service:v0.1.0

Pods that were already running before the policies were applied will keep running. The important test is what happens when Kubernetes creates a new pod. New pods using signed ClearLedger images should pass Kyverno verification.

Take a screenshot of the Scenario 3 denial. It proves the cluster blocks unsigned images, not just that CI signs images.

4.5: Verify ClearLedger Still Works

Kyverno enforces on new pod creation. Existing deployments that already passed admission (or were synced before policies existed) keep running. Confirm your app pods are healthy:

kubectl get pods -n clearledger

NAME                                    READY   STATUS    RESTARTS   AGE
auth-service-...                        1/1     Running   0          ...
frontend-...                            1/1     Running   0          ...
ledger-service-...                      1/1     Running   0          ...
notification-service-...                1/1     Running   0          ...
postgres-0                              1/1     Running   0          ...
redis-...                               1/1     Running   0          ...

If ingress is configured:

curl -s http://clearledger.local/auth/health | jq .
# {"status": "ok", "service": "auth-service"}

ArgoCD should still show Synced and Healthy: GitOps and admission control work together, not against each other.

4.6: Policy Exceptions (When a Legitimate Workload Needs a Bypass)

Kyverno blocks every pod that violates a policy. But what happens when a legitimate workload needs to bypass a specific rule?

PostgreSQL is the example. The official Postgres Alpine image uses a specific internal user (UID 70) to manage its data directory. The disallow-root-containers policy requires every pod to set runAsNonRoot: true.

Postgres does set that. But if Kyverno is configured to also check specific UID ranges, or if the pod's security context doesn't satisfy the rule for any reason, Kyverno blocks it. The database can't start, and the entire application fails.

You can't weaken the policy cluster-wide to accommodate one database. That would let every pod bypass the rule. Instead, you create a PolicyException: a targeted exemption for exactly the pods that need it.

Open infra/policies/exceptions/postgres-root-exception.yaml and read the comments. Here's what each section does:

The spec.exceptions block identifies which policy and rule to bypass:

exceptions:
  - policyName: disallow-root-containers
    ruleNames:
      - check-runAsNonRoot

This says: "skip only the check-runAsNonRoot rule from the disallow-root-containers policy." Every other rule in that policy (and every other policy in the cluster) still enforces normally.

The spec.match block limits which resources get the exception:

match:
  any:
    - resources:
        kinds:
          - Pod
        namespaces:
          - clearledger
        names:
          - postgres-*

Only pods named postgres-* (matching postgres-0, postgres-1, and so on), only in the clearledger namespace, only for the Pod resource kind. Everything else in the cluster still follows the strict policy.

The annotations are documentation for your team and auditors:

annotations:
  reason: "Postgres alpine image requires UID 70 for data directory ownership"
  approved-by: "platform-team"
  review-date: "2026-01-01"

These have no technical effect: Kyverno ignores them. They exist so that six months from now, when someone asks "why does Postgres bypass this rule?", the answer is right there in the file.

The rules for safe exceptions:

1. Scope narrowly: target the exact resource that needs it, nothing more

2. Commit to Git: the exception is reviewed in a pull request, tracked in version history, and auditable

3. Never weaken the policy itself: the rule stays strict for everything else

4. Review periodically: exceptions should be temporary if possible, and re-evaluated on a schedule

Apply the exception only if Kyverno blocks your Postgres pods:

kubectl apply -f infra/policies/exceptions/postgres-root-exception.yaml

Verify Kyverno still blocks other non-compliant pods (same denial as Scenario 1):

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: another-root-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
EOF

4.7: CIS Benchmark Evidence kube-bench)

You already installed Kyverno and proved it blocks unsafe pods.

This step is different. kube-bench doesn't block pods and doesn't change the cluster. It only checks the Kubernetes node against the CIS benchmark and saves evidence.

Think of the difference like this:

Both are useful, but only Kyverno blocks workloads in this lab.

Run kube-bench:

bash stages/stage-4-admission-control/scripts/run-kube-bench.sh

The script runs kube-bench as a Kubernetes Job and saves the report here:

stages/stage-4-admission-control/scripts/kube-bench-report.json

It also compares the result against this baseline:

stages/stage-4-admission-control/scripts/kube-bench-baseline.json

On MicroK8s, you'll see many FAIL and WARN lines. That's expected. The lab isn't asking you to fix every CIS warning on a single-node local VM.

What matters is the final result.

Pass looks like this:

kube-bench: 1 FAIL control(s) present (documented in baseline — no regressions).
kube-bench: no regressions vs baseline.

That means the known MicroK8s issues are documented, and your cluster didn't get worse.

If you see REGRESSION or make check-4 fails on kube-bench, stop and investigate before Stage 5.

Optional: confirm the report file exists:

ls -la stages/stage-4-admission-control/scripts/kube-bench-report.json

In production, you would either fix the CIS failures or document approved exceptions. In this lab, the baseline records the expected MicroK8s state.

4.8: Health Check

make check-4

What you should see:

▶ Stage 4 — Admission Control (Kyverno)
  ✓ Kyverno is running
  ✓ Policy disallow-root-containers — Enforce mode
  ✓ Policy require-resource-limits — Enforce mode
  ✓ Policy require-signed-images — Enforce mode
  ✓ Policy disallow-privilege-escalation — Enforce mode
  ✓ Policy drop-all-capabilities — Enforce mode
  ✓ Kyverno correctly rejects pods without securityContext
  ✓ kube-bench baseline exists (...)

All checks passed. Ready for the next stage.

If kube-bench reports regressions, run the script manually and update the baseline after reviewing. That diff is audit evidence.

If Kyverno install, policies, break-it scenarios, or make check-4 fail, see troubleshooting.md. Stage 4.

Stage 4 Complete: Done Checklist (Move to Stage 5)

You're done with Stage 4 when all of these are true:

Portfolio screenshots (optional): root-pod denial (§4.4 Scenario 1), unsigned-image denial (§4.4 Scenario 3), and kubectl get clusterpolicy showing five Enforce policies.

Not yet: SLSA attestation (optional), Vault secrets (Stage 5), network policies (Stage 6). Passwords still live in Kubernetes Secrets. Stage 5 moves them into Vault.

What You Learned in Stage 4

• The difference between CI scanning (before merge) and admission control (at the cluster gate)

• What Kyverno is: a policy engine that intercepts every Kubernetes API request

• That enforcement means the bad resource never exists, not "we detected it after the fact"

• How to read a Kyverno denial: policy name → rule name → JSON path that failed

• How to write and apply cluster-wide security policies as YAML

• How to scope a PolicyException without weakening the policy for everyone else

• That operational issues (Helm, image pulls, registry URL format) affect whether controls actually fire

• Why both CI and admission control are needed: CI catches problems in your code while Kyverno catches everything else that touches the cluster

• Evidence beats configuration: a policy file in Git means nothing: the break-it denials are proof CIS controls are enforced, not just documented.

What you can now put on your CV / say in an interview:

Enforced admission control with Kyverno: blocking root containers, privilege escalation, unsigned images, and missing resource limits at deploy time: mapped to CIS Kubernetes benchmarks.

make snapshot STAGE=4 && make snapshots. Confirm clearledger.stage4. See How to Save Your Progress.

Stage 5: Secrets Management (Vault)

By the end of this stage, sensitive values no longer live in Git or in etcd-backed Kubernetes Secrets: Vault holds them centrally and injects them into pods only when they start.

Your goal: remove auth-service-secret and ledger-service-secret from the cluster.

Login and API calls must still work because Vault injects credentials at pod startup. That's the moment secrets management clicks.

Before you start, confirm Stage 4 is solid: make check-4 passes, all five Kyverno policies are enforcing, and the app responds at http://clearledger.local. Fix any crash-looping pods before installing Vault.

What Changes in This Stage

Right now, database passwords and JWT keys sit in secret.yaml files on GitHub and in Kubernetes Secrets inside the cluster. In Stage 5 you move those values into HashiCorp Vault and teach the app to read them a different way.

When an auth or ledger pod starts, the Vault agent injector adds a small sidecar container. That sidecar logs into Vault using the pod’s own service account, fetches the password and JWT, and writes them as files under /vault/secrets/.

Your app already knows how to read those paths. It's the same data that used to arrive via secretKeyRef, just delivered at runtime instead of pulled from a Kubernetes Secret object.

Once migration is complete, sensitive values live in Vault (the long-term store) and briefly on the pod filesystem while the container runs. They're not in Git anymore. You remove secret.yaml from clearledger-infra and ArgoCD syncs deployments that point at Vault instead.

To load Vault the first time, you copy a template to a local .env file (§5.1). That file is gitignored. You run seed-vault-secrets.sh once to copy those values into Vault.

Real secret values aren't written into committed scripts. The scripts read secrets from your local .env file or from your terminal, so passwords and tokens stay out of Git.

Do the Steps in This Order

Each step depends on the one before it. Skipping ahead is the most common way to get red auth/ledger pods that look like a broken app but really mean “Vault is not ready yet.”

1. §5.1: copy stages/stage-5-secrets-management/.env.example to .env, then fill it with your cluster passwords

2. §5.2: install Vault and the agent injector with Helm

3. §5.3. Run setup.sh, then seed-vault-secrets.sh (passwords now live in Vault)

4. §5.4: push Vault-enabled deployments to clearledger-infra. Let ArgoCD sync.

5. §5.5. Wait for 2/2 pods (app + Vault sidecar), then delete the old Kubernetes Secrets

6. §5.5b: ArgoCD Synced / Healthy (after secret delete. OutOfSync before delete is normal)

7. §5.6. Confirm login works and credentials appear under /vault/secrets/ inside the pod

Start at §5.1. If anything fails, read troubleshooting.md. before changing manifests.

5.1: Create .env (Local Only, Never Commit)

This file holds two things: a dev Vault root token for Helm (§5.2), and the passwords you'll load into Vault in §5.3.

It stays on your machine only. Never commit it. The SEED_* values must match what the app uses today so login still works after you delete Kubernetes Secrets later.

Two different files. Don't mix them up:

The sample block at the bottom of this section is only a picture of what a completed .env looks like: don't copy those placeholder passwords unless they happen to match your cluster.

Step 1: copy the template to .env

cp stages/stage-5-secrets-management/.env.example \
   stages/stage-5-secrets-management/.env

That gives you a file with empty VAULT_TOKEN= and SEED_*= lines. Open it in your editor for steps 2–3.

Step 2: read the current passwords from the cluster

Run these from the repo root. Each command prints one value: copy the output into .env in step 3.

# → paste as SEED_AUTH_DATABASE_URL
kubectl get secret auth-service-secret -n clearledger \
  -o jsonpath='{.data.database_url}' | base64 -d; echo

# → paste as SEED_AUTH_JWT_SECRET
kubectl get secret auth-service-secret -n clearledger \
  -o jsonpath='{.data.jwt_secret}' | base64 -d; echo

# → paste as SEED_LEDGER_DATABASE_URL
kubectl get secret ledger-service-secret -n clearledger \
  -o jsonpath='{.data.database_url}' | base64 -d; echo

Step 3: fill in .env

Sample only: shape of a completed .env (use your kubectl output from step 2, not these example strings unless they match):

VAULT_TOKEN=my-dev-root-token
SEED_AUTH_DATABASE_URL=postgresql://clearledger:changeme-stage0@postgres:5432/clearledger
SEED_AUTH_JWT_SECRET=stage0-jwt-secret-change-in-production
SEED_LEDGER_DATABASE_URL=postgresql://clearledger:changeme-stage0@postgres:5432/clearledger

If auth-service-secret is already deleted (you skipped ahead: recover like this):

# Database URL from Postgres bootstrap secret (lab default password is often changeme-stage0)
PG_PASS=$(kubectl get secret postgres-secret -n clearledger \
  -o jsonpath='{.data.password}' | base64 -d)
echo "postgresql://clearledger:${PG_PASS}@postgres:5432/clearledger"
# Use that line for both SEED_AUTH_DATABASE_URL and SEED_LEDGER_DATABASE_URL

# JWT: same value you used at Stage 0, or read from Vault if you already seeded:
kubectl exec -n vault vault-0 -- vault kv get -field=jwt_secret clearledger/auth-service 2>/dev/null \
  || echo "(set SEED_AUTH_JWT_SECRET manually — must match tokens already issued)"

Continue to §5.2 once .env has all four variables set.

5.2: Install Vault and the Agent Injector

set -a && source stages/stage-5-secrets-management/.env && set +a

helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update

# First install:
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set server.dev.enabled=true \
  --set server.dev.devRootToken="${VAULT_TOKEN}" \
  --set ui.enabled=true \
  --set injector.enabled=true

# If helm install fails with "cannot re-use a name", use upgrade instead:
# helm upgrade --install vault hashicorp/vault \
#   --namespace vault --create-namespace \
#   --set server.dev.enabled=true \
#   --set server.dev.devRootToken="${VAULT_TOKEN}" \
#   --set ui.enabled=true \
#   --set injector.enabled=true

kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=vault -n vault --timeout=120s
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=vault-agent-injector -n vault --timeout=120s

kubectl apply -f stages/stage-5-secrets-management/infra/vault-ingress.yaml

Open http://vault.local in your browser. Log in with the value you set as VAULT_TOKEN in stages/stage-5-secrets-management/.env. For example, if your .env has VAULT_TOKEN=my-dev-root-token, use my-dev-root-token as the Vault login token.

Verify: list Vault pods:

kubectl get pods -n vault

Expected: Vault pods:

NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                1/1     Running   0          1m
vault-agent-injector-8d6b668b4-xxxxx   1/1     Running   0          1m

If helm install fails with “cannot re-use a name”: Vault is already installed. Use the helm upgrade --install block above.

5.3: Configure Vault (Platform + Seed KV)

Run both scripts in order. Each reads VAULT_TOKEN from your .env.

bash stages/stage-5-secrets-management/infra/vault/setup.sh
bash stages/stage-5-secrets-management/infra/vault/seed-vault-secrets.sh

setup.sh: prepares Vault for the cluster: Kubernetes auth, the KV secret store, policies, and roles so auth/ledger pods can fetch secrets later. It doesn't write your database passwords yet and nothing goes to Git.

seed-vault-secrets.sh: takes the SEED_* lines from .env and stores them in Vault at clearledger/data/auth-service and clearledger/data/ledger-service. It doesn't echo those values to the terminal.

Re-running either script is safe for the lab.

Expected, setup.sh (tail):

==> Enabling Kubernetes auth method...
==> Configuring Kubernetes auth...
==> Enabling KV secrets engine...
==> Creating Vault policies...
==> Creating Kubernetes auth roles...
==> Applying RBAC + ServiceAccounts...

✓ Vault platform setup complete (no secrets written yet).
  Next: bash stages/stage-5-secrets-management/infra/vault/seed-vault-secrets.sh

Expected, seed-vault-secrets.sh:

==> Logging into Vault...
==> Writing secrets to Vault KV (values are not printed)...
======== Secret Path ========
clearledger/data/auth-service
======= Metadata =======
Key                Value
---                -----
created_time       2026-06-01T15:31:53.538991153Z
version            1
✓ Secrets stored at clearledger/data/auth-service and clearledger/data/ledger-service

Verify metadata only (no secret values printed):

kubectl exec -n vault vault-0 -- vault kv metadata get clearledger/auth-service

Key                     Value
---                     -----
cas_required            false
created_time            2026-06-01T15:31:53.538991153Z
current_version         1
delete_version_after    0s
max_versions            0
oldest_version          0
updated_time            2026-06-01T15:31:53.538991153Z

5.4: GitOps: Update clearledger-infra (Fixes ArgoCD OutOfSync)

ArgoCD deploys from your clearledger-infra GitHub repo, not from the main clearledger app repo where you're working now. You edit manifests here first, then copy the same changes to clearledger-infra so ArgoCD can sync them. Work slowly and verify after each sub-step.

5.4a. Update manifests in the app repo (clearledger)

cp stages/stage-5-secrets-management/infra/manifests/auth-service/deployment.yaml \
   infra/manifests/auth-service/deployment.yaml

cp stages/stage-5-secrets-management/infra/manifests/ledger-service/deployment.yaml \
   infra/manifests/ledger-service/deployment.yaml\

mkdir -p infra/manifests/vault

cp infra/deferred-by-stage/stage-5-secrets-management/vault/rotation-cronjob.yaml \
   infra/manifests/vault/rotation-cronjob.yaml

rm -f infra/manifests/auth-service/secret.yaml infra/manifests/ledger-service/secret.yaml

5.4b. Edit infra/manifests/kustomization.yaml by hand

Open the file in your editor. In the resources: list:

• Remove the app secret entries: delete these two lines, or comment them out with # (both work, as Kustomize ignores # lines):

  - auth-service/secret.yaml
  - ledger-service/secret.yaml
  

• Add this line (with the other resources):

  - vault/rotation-cronjob.yaml
  

Leave postgres/postgres-secret.yaml, that is Postgres bootstrap only, not app credentials.

Save. Verify:

# Active (uncommented) app secret lines must be gone — postgres-secret is OK
grep -E '^[[:space:]]*-[[:space:]]+(auth-service|ledger-service)/secret\.yaml' \
  infra/manifests/kustomization.yaml && echo "STOP: app secrets still active" || echo "OK"

grep vault/rotation-cronjob.yaml infra/manifests/kustomization.yaml
grep vault.hashicorp infra/manifests/auth-service/deployment.yaml | head -1
kustomize build infra/manifests >/dev/null && echo "OK: kustomize build"

Expected: OK, rotation cronjob listed, first line shows vault.hashicorp.com/agent-inject, kustomize build succeeds.

Commit in the app repo when ready: git add infra/manifests && git commit -m "feat(stage-5): Vault deployments in canonical manifests".

5.4c. Push the same changes to clearledger-infra

git clone https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/clearledger-infra

If clone fails with destination path '/tmp/clearledger-infra' already exists (you cloned in §1.3 or an earlier step), reuse that folder. Don't clone again:

cd /tmp/clearledger-infra && git pull && cd -

Or start fresh: rm -rf /tmp/clearledger-infra then run git clone again.

Run the cp commands from the main clearledger app repo, not from /tmp/clearledger-infra. Your shell prompt should say clearledger, not clearledger-infra. The source path infra/manifests/... only exists in the app repo.

cd ~/clearledger    # main app repo — adjust path if yours differs

cp infra/manifests/auth-service/deployment.yaml /tmp/clearledger-infra/manifests/auth-service/
cp infra/manifests/ledger-service/deployment.yaml /tmp/clearledger-infra/manifests/ledger-service/
mkdir -p /tmp/clearledger-infra/manifests/vault
cp infra/manifests/vault/rotation-cronjob.yaml /tmp/clearledger-infra/manifests/vault/
cp infra/manifests/kustomization.yaml /tmp/clearledger-infra/manifests/kustomization.yaml
rm -f /tmp/clearledger-infra/manifests/auth-service/secret.yaml
rm -f /tmp/clearledger-infra/manifests/ledger-service/secret.yaml

cd /tmp/clearledger-infra
git add -A
git status
git commit -m "feat(stage-5): Vault injection; remove app secrets from GitOps"
git push
cd -

✋ Hands-on checkpoint. Stage 5 GitOps landed

git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-s5
test ! -f /tmp/verify-s5/manifests/auth-service/secret.yaml && echo "OK: app secret removed from Git"
grep vault.hashicorp /tmp/verify-s5/manifests/auth-service/deployment.yaml | head -1
grep vault/rotation-cronjob.yaml /tmp/verify-s5/manifests/kustomization.yaml
rm -rf /tmp/verify-s5

Expected: OK, Vault annotation present, rotation job in kustomization.

Expected, git status before commit (step 5.4c):

modified:   manifests/auth-service/deployment.yaml
modified:   manifests/ledger-service/deployment.yaml
modified:   manifests/kustomization.yaml
new file:   manifests/vault/rotation-cronjob.yaml
deleted:    manifests/auth-service/secret.yaml
deleted:    manifests/ledger-service/secret.yaml

After git push, ArgoCD will roll out Vault-enabled deployments automatically. Continue to §5.5. Don't expect Synced yet, as app secrets are still in the cluster until you delete them there.

Common rollout failures:

5.5: Wait for Vault-injected Pods, Then Delete K8s App Secrets

Wait until auth/ledger show Vault sidecars (READY 2/2 = app + vault-agent):

kubectl get pods -n clearledger -l app=auth-service
kubectl get pods -n clearledger -l app=ledger-service

Expected:

NAME                            READY   STATUS    RESTARTS   AGE
auth-service-5756d9fcb9-bmdlr   2/2     Running   0          2m
auth-service-5756d9fcb9-jtgss   2/2     Running   0          2m

Inspect sidecar pulled secrets (init container logs):

kubectl logs -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c vault-agent-init
# ... Authentication successful, rendering templates ...

Only after pods are 2/2, delete app Secrets:

kubectl delete secret auth-service-secret ledger-service-secret -n clearledger

Expected: secrets remaining:

kubectl get secret -n clearledger

NAME              TYPE     DATA   AGE
postgres-secret   Opaque   2      6d

postgres-secret is Postgres bootstrap only, not app credentials. That stays until you harden Postgres separately.

If delete says NotFound: secrets were already removed. Continue to §5.6.

5.5b: ArgoCD Should Be Synced After Secret Delete

Run this after §5.5, not right after §5.4. Before you delete app Secrets, OutOfSync is normal. Git no longer lists auth-service-secret / ledger-service-secret, but they still exist in the cluster until you delete them in the step above.

kubectl get application clearledger -n argocd \
  -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'

Before secret delete: expect sync=OutOfSync health=Healthy or Progressing while Vault pods roll out. That's fine if auth/ledger are 2/2.

After secret delete, hard-refresh and sync if still OutOfSync:

kubectl annotate application clearledger -n argocd argocd.argoproj.io/refresh=hard --overwrite
argocd app sync clearledger --grpc-web --prune

If sync says another operation is already in progress, wait a minute: ArgoCD auto-sync is already running.

Wait until:

kubectl get application clearledger -n argocd \
  -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'
# Synced Healthy

Don't update app deployments with kubectl apply after ArgoCD is managing them. ArgoCD keeps the cluster matched to clearledger-infra. If you change a deployment by hand, ArgoCD may revert it. For Stage 5, update the manifests in Git and let ArgoCD sync the Vault-enabled deployments.

5.6: Login and Injected Files

kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- ls /vault/secrets/

database_url
jwt_secret

curl -s -X POST http://clearledger.local/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' | jq .

Expected:

{
  "access_token": "<jwt-returned-by-auth-service>",
  "token_type": "bearer"
}

Take a screenshot: working login JSON + kubectl get secret -n clearledger showing no auth-service-secret / ledger-service-secret.

5.7: Health Check

make check-5

What you should see:

make check-5 re-runs Stage 4 checks first, that is expected. Look for the Stage 5 block below to confirm Vault is working.

▶ Stage 4: Admission Control (Kyverno)
  ✓ Kyverno is running
  ✓ Policy disallow-root-containers — Enforce mode
  ...
  ✓ kube-bench matches baseline (no new FAIL regressions)

▶ Stage 5: Secrets Management (Vault)
  ✓ Vault pod is running
  ✓ Vault agent injector is running
  ✓ Vault is unsealed
  ✓ Vault Kubernetes auth method is enabled
  ✓ auth-service-secret removed — Vault is the secret source
  ✓ Vault injected /vault/secrets/database_url into auth-service

All checks passed. Ready for the next stage.

If Vault injection or ArgoCD sync fails, see troubleshooting.md.

Stage 5 is Done: Checklist Before Moving to Stage 6

Stage 5 moves app credentials out of Git and Kubernetes Secrets. It doesn't make Vault production-grade yet. This lab still uses Vault dev mode, not HA or auto-unseal.

Also, a running pod can still read the files under /vault/secrets/ because the app needs those credentials to work. That's normal. Stage 6 adds Falco so you can detect suspicious runtime access.

What You Learned in Stage 5

• Kubernetes Secrets aren't enough for real secret management.

• Vault now stores the app credentials.

• .env was only used locally to load the first secrets into Vault. It's never committed.

• Vault injects secrets into the pod when the app starts.

• clearledger-infra must stop storing secret.yaml, because ArgoCD deploys from that repo.

• The order matters: install Vault, seed secrets, update GitOps, wait for healthy pods, then delete old Kubernetes Secrets.

What you can now say in an interview:

I replaced Kubernetes Secrets with HashiCorp Vault agent injection, removed app credentials from Git and Kubernetes Secrets, and verified the app still worked after Vault injected the credentials at runtime.

Save your progress:

make snapshot STAGE=5 && make snapshots

Confirm clearledger.stage5 appears in the snapshot list.

Stage 6 — Runtime Security (Falco)

Stages 1–5 secured what gets deployed and how secrets are stored. Stage 6 watches what happens inside running containers after they start.

Your goal is to learn what runtime security catches and why it matters, then prove it by triggering a Falco alert and reading it the way an on-call engineer would.

CI, Kyverno, and Vault all act before or at pod startup. Falco fills the gap they leave open. It watches what running software actually does inside the container. That's the layer incident response and forensics care about, not just another chart to install.

Before you start Stage 6:

• make check-5 passes

• Login and transactions still work at http://clearledger.local

• Platform pods have low restart counts

You're done with Stage 6 when:

• You trigger at least one Falco alert

• You apply the network policies

• make check-6 passes

Then save your VM:

make snapshot STAGE=6
make snapshots

Do the Steps in This Order

Each step depends on the one before it. Don't run make check-6 until §6.4. It checks network policies you haven't applied yet.

1. §6.1: bash stages/stage-6-runtime-security/scripts/install-falco.sh. Confirm falco-* pods 2/2 Running and custom rules loaded.

2. §6.2: make demo-6: read ✓ Runtime detection confirmed in the terminal

3. §6.3 (optional) manual break-it scenarios (skip if make demo-6 already worked)

4. §6.4: kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml. Confirm curl http://clearledger.local/ returns 200.

5. §6.6: make check-6

Start at §6.1. If anything fails, see troubleshooting.md.

Optional reading: How Stage 6 fits the full stack: why Falco and netpol exist and how they differ from Stages 3–5.

If You Get Stuck in Stage 6

Stage 6 has three jobs:

1. Install Falco

2. Trigger one test alert

3. Apply network policies

Don't worry about every row in the Falco UI. The UI may show noise. You pass the Falco part when you can find one alert from your demo, either in the terminal or in the UI.

For the portfolio screenshot, open:

http://falco.local

Login:

• Username: admin

• Password: admin

Take a screenshot only after your demo alert appears.

Common stuck points

6.1: Install Falco and Falcosidekick UI

bash stages/stage-6-runtime-security/scripts/install-falco.sh

This runs helm upgrade --install with modern_ebpf, enables Falcosidekick + Web UI, enables the k8s-metacollector (collectors.kubernetes.enabled: true) so custom rules can match k8smeta.ns.name = clearledger, loads rules from infra/falco/clearledger-rules-content.yaml, applies the rules ConfigMap and ingress.

If Falco is already installed, the script is safe to re-run (upgrade).

Verify Falco pods:

kubectl get pods -n falco

Expected output:

NAME                                      READY   STATUS    RESTARTS   AGE
falco-w4fh6                               2/2     Running   0          2m
falco-falcosidekick-...                   1/1     Running   0          2m
falco-falcosidekick-ui-...                1/1     Running   0          2m
falco-falcosidekick-ui-redis-0            1/1     Running   0          2m

The Falco DaemonSet should show 2/2 Running. Sidekick, UI, and Redis pods should each show 1/1 Running. Pod name suffixes on your cluster will differ from the example.

Open http://falco.local. You'll see the Falcosidekick UI. Log in with the chart defaults:

To read the credentials from the cluster instead of trusting the lab defaults:

kubectl get secret falco-falcosidekick-ui -n falco \
  -o jsonpath='{.data.FALCOSIDEKICK_UI_USER}' | base64 -d && echo
# admin:admin

Falcosidekick UI: quick orientation

After login you land on the Events tab. The table can look busy before you run any demo, which is normal.

• Rule: detection name (what fired)

• Priority: Critical / Warning / Notice (focus on Critical and Warning for this lab)

• Output: pod name, file, or command details

• Tags: look for clearledger on lab alerts

Background noise you can ignore: Notice rows from ArgoCD. Critical Sensitive File Read rows from postgres-0 reading /etc/passwd (repeats every few seconds). Your demo alert is different. See §6.2.

Verify custom rules loaded (do this before §6.2):

kubectl get pods -n falco                                    # Falco pod 2/2 Running
kubectl get configmap clearledger-falco-rules -n falco
kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=200 \
  | grep 'rules.d/clearledger_rules'

Expected: clearledger_rules.yaml | schema validation: ok

An empty grep with --tail=30 alone isn't a failure. Use --tail=200. If you see LOAD_ERR_COMPILE_CONDITION, see troubleshooting.md.

If rules didn't load, §6.2 and §6.3 will look like they passed when nothing fired.

6.2: Guided Demo (make demo-6)

Run this after §6.1 (Falco installed, rules verified, UI opens at http://falco.local).

make demo-6
# or:
bash stages/stage-6-runtime-security/scripts/demo-falco-alerts.sh

What the demo script does

The demo proves Falco can detect suspicious activity inside a running container.

The script checks that Falco is running, opens http://falco.local, and waits while you log in with:

• Username: admin

• Password: admin

Then it runs this test command inside the auth-service container:

kubectl exec -n clearledger \
  auth-service-<pod-suffix> \
  -c auth-service -- /bin/sh -c 'id && exit'

The script picks the real pod name for you.

Non-interactive (CI or no Enter prompts): SKIP_PROMPT=1 make demo-6.

This starts a shell inside the app container. That's suspicious in production because app containers should run the app, not open shells. Falco should detect it and create an alert called:

Shell Spawned in ClearLedger Container

When the script prints:

✓ Runtime detection confirmed

refresh the Falco UI.

Look for a Critical alert with:

• Rule: Shell Spawned in ClearLedger Container

• Pod: auth-service-...

• Command: sh -c id && exit

Ignore alerts from postgres-0, especially Sensitive File Read. Those are background noise for this lab.

If the UI is noisy, search the page for Shell Spawned or check from the terminal:

kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=500 \
  | grep 'Shell Spawned'

For your screenshot, capture the Shell Spawned alert for auth-service.

6.3: Break-it Scenarios (Manual, Optional)

These are the same detections as §6.2, but you run each command yourself. Skip this section if you already completed make demo-6.

After each command, refresh http://falco.local or use the find methods in §6.2.

Scenario 1 – Shell in a running pod (command injection simulation):

kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- /bin/sh -c "id && exit"

Expected in Falco UI / logs (within ~10 seconds):

CRITICAL: Shell spawned in ClearLedger container
  user=... container=auth-service pod=auth-service-... cmd=sh -c id && exit

What this means: Stage 4 allowed the pod (it is compliant). Stage 6 detected behavior inside the pod: exactly what an attacker would do after command injection.

If you see no alert: confirm the exec used -c auth-service (not the vault-agent sidecar), rules show schema validation: ok, and the pod image name contains clearledger.

Scenario 2 – Read a sensitive file (reconnaissance):

kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- cat /etc/passwd

Expected:

CRITICAL: Sensitive file read in ClearLedger
  file=/etc/passwd container=auth-service pod=auth-service-...

Scenario 3 – Download tool at runtime (optional):

kubectl exec -n clearledger \
  $(kubectl get pod -n clearledger -l app=auth-service -o name | head -1) \
  -c auth-service -- sh -c "wget -q ifconfig.me -O - 2>/dev/null || true"

May fire Package manager executed and/or Unexpected outbound connection (WARNING).

Take screenshots of Scenarios 1 and 2: portfolio evidence for runtime detection.

6.4: Apply Network Policies (Zero-trust Segmentation)

Network policies are firewall rules between pods. Apply them after the Falco demo.

make check-6 checks for these policies, so run it only after this section. The default-deny-all policy blocks traffic by default.

The allow-* policies open only the paths ClearLedger needs to work. Falco detects suspicious behavior. Network policies limit where a pod can connect.

Apply:

kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml
kubectl get networkpolicy -n clearledger

Expected: seven policies: default-deny-all plus six allow-* (auth-service, ledger-service, notification-service, postgres, redis, frontend).

Verify the app still works:

curl -s http://clearledger.local/auth/health | jq .
# {"status":"ok","service":"auth-service"}

curl -s http://clearledger.local/notifications/health | jq .
# {"status":"ok",...}

Checkpoint (required): proves netpol didn't break the real app:

kubectl get networkpolicy -n clearledger
curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/
kubectl get pods -n clearledger --field-selector=status.phase!=Running

If auth or ledger start restarting after netpol, egress rules are too strict. See troubleshooting.md.

Scenario 4 – blocked cross-service traffic (optional)

Skip if the checkpoint passed and you plan to run make check-6. This proves ledger can't call notification directly (no allow rule for that path). Failure to connect is success.

Don't use the old wget one-liner: the ledger image has no wget/curl, and head -1 can pick a Terminating pod (exec hangs or exit 137).

LEDGER_POD=$(kubectl get pods -n clearledger -l app=ledger-service --no-headers \
  | awk '$2=="2/2" && $3=="Running" {print $1; exit}')

echo "Using pod: $LEDGER_POD"

kubectl exec -n clearledger "$LEDGER_POD" -c ledger-service -- python3 -c "
import urllib.request
try:
    urllib.request.urlopen('http://notification-service/', timeout=5)
    print('UNEXPECTED: connection succeeded')
except Exception as e:
    print('BLOCKED (expected):', e)
"

Expected:

BLOCKED (expected): <urlopen error timed out>

or Connection refused, not UNEXPECTED: connection succeeded.

6.6: Health Check

Run this after §6.4 (network policies). It confirms Falco, custom rules, and netpol are installed. It does not prove an alert fired (that is §6.2).

make check-6

What you should see:

▶ Stage 6 — Runtime Security (Falco)
  ✓ Falco DaemonSet: 1/1 nodes
  ✓ ClearLedger custom Falco rules ConfigMap exists
  ✓ NetworkPolicy default-deny-all exists
  ✓ NetworkPolicy allow-auth-service exists
  ✓ NetworkPolicy allow-ledger-service exists
  ✓ NetworkPolicy allow-notification-service exists
  ✓ auth-service reachable after network policies
  ✓ notification-service reachable after network policies

All checks passed. Ready for the next stage.

How Stage 6 fits the full stack (optional reading)

Each stage guards a different point in the lifecycle. Stages 1 through 5 work before or during pod startup. Stage 6 watches what happens inside a container that is already running.

• In Stage 3, CI catches bad code and images on git push.

• Stage 4, Kyverno blocks bad pods at admission.

• Stage 5, Vault injects secrets at startup.

• Stage 6, Falco watches syscalls after the pod is running (shell spawns, sensitive file reads).

• Stage 6, Network policies filter pod-to-pod traffic.

They answer three different questions: Kyverno asks whether this pod may be created. Falco asks what the pod is doing right now. Network policies ask who the pod may talk to.

Falco doesn't replace CI or Kyverno. If you skip Stages 3–5, Falco can still alert, but you already shipped vulnerable code and secrets in Git.

Stage 6 is Complete. Proceed to Stage 6.5 or 7.

Portfolio screenshots (optional): shell-in-container alert · sensitive-file read alert in Falco UI.

What comes next: Stage 6 gives you Falco alerts and basic network policies. You can refine the network policies later. Stage 6.5 is optional chaos testing with Litmus, and Stage 7 adds Grafana dashboards so you can see security events over time.

What You Learned in Stage 6

• What runtime security catches that CI and admission control can't: threats inside running containers

• What Falco is: eBPF syscall monitoring with custom YAML rules

• What network policies are: Kubernetes firewall rules between pods

• How to trigger and interpret alerts: incident response skills

• The full stack: code scanning, admission control, secrets management, runtime detection which leads to (next) observability

What you can now put on your CV / say in an interview:

Deployed Falco for runtime threat detection with custom rules, and can trigger and read an alert for a shell-in-container or sensitive-file read the way an on-call engineer would.

make snapshot STAGE=6 && make snapshots. Confirm clearledger.stage6. See How to Save Your Progress.

Stage 6.5 — Chaos Engineering (Optional)

Most learners skip this. If Stage 6 is done and make check-6 passes, jump straight to Stage 7. Nothing in Stages 7–8 requires Litmus.

If you want chaos/resilience (~1 hour): LitmusChaos deletes one auth-service pod and proves /auth/health stays 200 while Kubernetes replaces it.

Do the Steps in This Order

Optional: §6.5.3: same test via make demo-65 (terminal path) instead of the UI wizard.

6.5.0: Before You Start (Auth Pods Must be 2/2)

Chaos deletes pods. If replacements fail to start, you debug CrashLoopBackOff instead of learning resilience.

export GITHUB_OWNER=YOUR_GITHUB_USERNAME   # required — without this, fix-argocd breaks ArgoCD repoURL
make fix-65-prereqs
kubectl get pods -n clearledger -l app=auth-service

Pass: two pods, both 2/2 Ready. Don't install Litmus until this is true.

If something fails:

6.5.1: Install LitmusChaos (Operator, UI, Cluster Connection)

bash stages/stage-6.5-chaos-engineering/scripts/install-litmus.sh
kubectl get pods -n litmus
open http://litmus.local    # login: admin / litmus

Pass before §6.5.2: Overview shows Infrastructures: Active 1 (not 0, not Pending).

Verify pods:

kubectl get pods -n litmus
# litmus-core, chaos frontend/server, mongodb, subscriber — all Running

If Overview shows 0 infrastructures or PENDING

The UI is empty until a subscriber agent connects your cluster:

export LITMUS_PASSWORD='litmus'   # only if you changed the default
bash stages/stage-6.5-chaos-engineering/scripts/connect-litmus-infra.sh

Hard-refresh the browser. Start at http://litmus.local only, not old /account/.../settings bookmarks.

UI navigation (click order for §6.5.2)

1. Overview: Confirm Active 1

2. ChaosHubs, Pod Delete, Launch Experiment

3. Chaos Experiments: watch Running to Completed

Left nav: Overview, Environments, ChaosHub, Chaos Experiments. Skip Resilience Probes and deep Settings URLs for this lab.

6.5.2: Run Your First Experiment (Pod Delete)

Goal: Kill one auth-service pod and prove /auth/health stays 200.

Before you click Run in the UI, open two terminals:

# Terminal A — watch pods
kubectl get pods -n clearledger -l app=auth-service -w

# Terminal B — watch health every 5 seconds
while true; do
  date +%H:%M:%S
  curl -s -o /dev/null -w "health=%{http_code}\n" http://clearledger.local/auth/health
  sleep 5
done

In the UI (http://litmus.local): Left nav → ChaosHubs → Pod Delete card → Launch Experiment.

Litmus UI note: ChaosCenter labels change between versions (for example, “Tune fault”, “Target selection”, “Chaos Experiment”). Match fields by concept, not exact button text. Accept wizard defaults unless the table below lists a value.

In the Litmus UI, open:

ChaosHubs → Pod Delete → Launch Experiment

This opens the experiment wizard. Use these values when the wizard asks for them:

• Infrastructure: clearledger-cluster and it must be Active

• Namespace: clearledger

• Target label: app=auth-service

• Target kind: Deployment

• Pods affected: 50%

• Duration: 30 seconds

• Fault/experiment name: pod-delete

Finish the wizard with Save or Create, then click Run. Don't choose Schedule.

What success looks like:

Prefer terminal over UI? Skip the wizard and run §6.5.3 (make demo-65) instead.

6.5.3 — Same experiment from the terminal (make demo-65) — optional

Use this if you want to run the pod-delete test without clicking through the Litmus UI.

Make sure auth pods are healthy first:

make fix-65-prereqs

Then run the demo:

make demo-65

The script applies the auth-service-pod-delete ChaosEngine in the litmus namespace. Litmus deletes one auth-service pod, Kubernetes replaces it, and the script checks that /auth/health keeps returning 200.

After it finishes, verify the result:

kubectl get chaosresult -n litmus
kubectl get pods -n clearledger -l app=auth-service

You passed if the script ends with PASS, the ChaosResult is Completed / Pass, and two auth-service pods are running again.

You can also see the run in the Litmus UI: Chaos Experiments → refresh → open the latest run.

If new auth pods get stuck in Init:0/1, re-apply the Stage 6 network policies:

kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml

6.5.3a: Real Output Examples (Verified on the Lab Cluster)

These samples were captured from a working cluster after make fix-65-prereqs, make connect-litmus, and make demo-65.

make check-65

▶ Stage 6.5 — Chaos Engineering (LitmusChaos)
  ✓ litmus namespace exists
  ✓ litmus-admin ServiceAccount exists in litmus
  ✓ pod-delete ChaosExperiment installed in litmus
  ✓ Litmus chaos operator is running
  ✓ Litmus ChaosCenter reachable at http://litmus.local
  ✓ Litmus subscriber running (UI connected to cluster)
  ✓ auth-service healthy (baseline before chaos)
  ✓ auth-service has 2/2 Ready replicas (stable for chaos)
  ✓ allow-postgres NetworkPolicy exists (Stage 6 fix)

All checks passed. Ready for the next stage.

make demo-65 captured from a real run (2026-06-01)

Stage 6.5 — auth-service pod-delete

Preflight: 2 auth-service pods Running

Applying ChaosEngine auth-service-pod-delete (namespace litmus)

Watching http://clearledger.local/auth/health

  10s  health=200  pods=2
  20s  health=200  pods=1
  30s  health=200  pods=1
  40s  health=200  pods=2
  50s  health=200  pods=2
  60s  health=200  pods=2

Result:
  ChaosResult: Completed / Pass
  Recovery:    2 auth-service pod(s) Running
  Health:      6/6 checks returned 200

PASS

If health lines show 000, run bash scripts/setup-hosts.sh on your Mac and re-run. The script also tries multipass exec clearledger -- curl when the VM is present.

Terminal B (health loop, expected output)

22:05:01
health=200
22:05:06
health=200
22:05:11
health=200

Pod count may show 1 while the replacement pod is starting, which is expected.

Terminal A during chaos (kubectl get pods -w)

NAME                            READY   STATUS        RESTARTS   AGE
auth-service-84cc988c4d-hdb45   2/2     Running       0          67m
auth-service-84cc988c4d-b59sj   2/2     Terminating   0          15m    ← killed
auth-service-84cc988c4d-dxz9q   0/2     Pending       0          0s     ← replacement
auth-service-84cc988c4d-dxz9q   0/2     Init:0/1      0          2s
auth-service-84cc988c4d-dxz9q   2/2     Running       0          90s

After demo: verify

kubectl get chaosresult -n litmus
# auth-service-pod-delete-pod-delete   Completed   Pass

kubectl get pods -n clearledger -l app=auth-service
# auth-service-84cc988c4d-xxxxx   2/2   Running
# auth-service-84cc988c4d-yyyyy   2/2   Running

kubectl get cm subscriber-config -n litmus -o jsonpath='{.data.IS_INFRA_CONFIRMED}'
# true

Subscriber connected (infrastructure Active in UI)

kubectl logs -n litmus -l app.kubernetes.io/name=subscriber --tail=3
level=info msg="AgentID: a63c2a2c-... has been confirmed"
level=info msg="Server connection established, Listening...."

6.5.4: Understand the YAML Files (Read Before Running)

Each file is a ChaosEngine: a request to Litmus: “run experiment X against app Y for Z seconds.”

litmus-install.yaml

This creates the litmus namespace only. Platform workloads live here, separate from clearledger app pods.

litmus-rbac.yaml

auth-service-pod-delete.yaml (Experiment 1: used by demo)

metadata:
  namespace: litmus          # engine lives here (Kyverno-safe)
spec:
  appinfo:
    appns: clearledger       # target app namespace
    applabel: app=auth-service
    appkind: deployment
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: PODS_AFFECTED_PERC
              value: "50"    # 50% of 2 replicas = 1 pod killed
            - name: TOTAL_CHAOS_DURATION
              value: "30"    # chaos window in seconds

What happens when applied:

1. Operator reads ChaosEngine and creates auth-service-pod-delete-runner pod in litmus

2. Runner selects one auth-service pod in clearledger and sends SIGTERM / delete

3. Kubernetes Deployment controller sees 1/2 replicas and schedules a replacement pod

4. Service routes traffic to the surviving replica during recovery

5. ChaosResult CR records pass/fail from Litmus’s perspective

ledger-service-network-latency.yaml (Experiment 2 — manual)

Adds 2000 ms network latency to ledger-service pods for 60 seconds. Proves timeouts return 503 instead of hanging the UI.

notification-service-memory-hog.yaml (Experiment 3 — manual)

Fills 80% of pod memory limit for 60 seconds. Proves OOMKill + restart behavior.

Never apply all three at once. Run one experiment, verify recovery, then the next.

6.5.5: After the Demo, What to Look For (Do Not Skip)

1. During chaos: availability

2. After chaos: recovery

3. Litmus ChaosResult verdict

kubectl get chaosresult -n litmus

Your pass criteria:

• /auth/health returned 200 at least once during the chaos window

• A pod was Killed (see events)

• Deployment returned to 2 replicas

6.5.6 Manual Experiments (After Experiment 1 Succeeds)

Wait until both auth-service pods show 2/2 Ready, then run one experiment at a time:

# Experiment 2 — 2s network latency on ledger-service (60s)
kubectl delete chaosengine ledger-service-network-latency -n litmus --ignore-not-found
kubectl apply -f stages/stage-6.5-chaos-engineering/infra/chaos/ledger-service-network-latency.yaml

# Experiment 3 — memory pressure on notification-service (60s)
kubectl delete chaosengine notification-service-memory-hog -n litmus --ignore-not-found
kubectl apply -f stages/stage-6.5-chaos-engineering/infra/chaos/notification-service-memory-hog.yaml

Clean up an experiment:

kubectl delete chaosengine auth-service-pod-delete -n litmus

6.5.7: Health Check

make check-65

Expected: see full sample in §6.5.3a (make check-65 block). Minimum:

▶ Stage 6.5 Chaos Engineering (LitmusChaos)
  ✓ Litmus subscriber running (UI connected to cluster)
  ✓ auth-service has 2/2 Ready replicas (stable for chaos)
  ...
All checks passed. Ready for the next stage.

Stage 6.5 Complete: Checklist

What You Learned in Stage 6.5

• Detection ≠ resilience: Falco alerts don't prove HA

• Replicas + Services + probes: why replicas: 2 isn't cosmetic

• ChaosEngine YAML: declarative failure injection as code

• Platform vs app namespaces: Kyverno blocks chaos runners in clearledger, engines run in litmus

• MTTR: time from pod kill to 2/2 Ready again (Stage 7 graphs this)

What you can now put on your CV / say in an interview:

Ran chaos experiments with LitmusChaos (pod-delete, network latency, memory pressure) to prove the system recovers, and can distinguish detection from resilience.

make snapshot STAGE=65 && make snapshots. Confirm clearledger.stage65. See How to Save Your Progress.

Stage 7 — Security Observability

Security you can't measure, you can't prove.

The goal here is to understand how metrics, logs, and dashboards fit together. Then prove it by running commands in the terminal, watching the same events appear in Grafana, and explaining what each panel means.

This stage is not “install Grafana and move on.” Stage 7 isn't complete until your dashboards show real Kyverno violations and Falco alerts that you triggered in §7.4: plus portfolio screenshots (§7.6). make check-7 only proves the stack is up; it does not prove you can detect security events.

Before you start: make check-6 should pass (Stage 6.5 is optional. Skip is fine). Check the VM is not overloaded: multipass exec clearledger -- uptime. If you ran Stage 6.5, do §7.0 first to scale Litmus down. Plan about half a day. This is the heaviest stage on a single-node VM.

You'll be done when §7.6 is complete: dashboards show your Kyverno denial and Falco alert, not empty panels. Then make check-7 (§7.7), make snapshot STAGE=7, and make snapshots (confirm clearledger.stage7).

Already installed? If kubectl get pods -n monitoring shows Grafana 3/3 and Loki 1/1, skip §7.1. Start at §7.2 (verify the stack), then §7.4 (hands-on lab).

What You Need to Know First

Up to now, each stage had its own window into the cluster. Stage 3 gave you CI scan results in GitHub Actions. Stage 4 showed Kyverno blocking a bad deploy in the terminal. Stage 6 gave you Falco alerts in its UI, and you could always run kubectl logs on a pod. Those views are useful, but they are scattered.

Stage 7 brings them together in one place: Grafana. Instead of jumping between five different tools, you open a dashboard and see whether security events, policy violations, and app health are happening over time.

The three tools you're installing

Prometheus collects numbers from the cluster: things like “how many Kyverno denials in the last hour” or “how many HTTP requests per second.” It checks those numbers every 15–30 seconds and keeps a history you can graph.

Loki collects log lines: the same kind of text you see from kubectl logs, but from many pods at once. Falco alerts, failed login attempts, and application errors all land here so you can search them later.

Grafana is the web UI where charts and tables pull data from Prometheus and Loki. This is what you would show an auditor: not a one-off terminal screenshot, but proof that you can find and measure events after they happen.

Prometheus doesn't magically know what to collect. ServiceMonitors and PodMonitors are small config objects that point it at the right targets.
If Kyverno has no monitor, the Kyverno dashboard stays empty even when Kyverno is working fine. The same applies to application request rates. Those panels stay blank until §7.5, when metrics-enabled images are deployed through GitOps.

Logs follow a similar path. Promtail reads container logs and sends them to Loki. If Loki isn't running, Grafana log panels show “No data” even though kubectl logs still works on individual pods.

How this connects to what you already built

When you blocked a bad kubectl apply in Stage 4, Kyverno recorded that denial. In Stage 7, that shows up on the Kyverno Policy Violations dashboard (via Prometheus).

When you triggered a shell inside a pod in Stage 6, Falco wrote an alert. In Stage 7, that appears on the Security Event Timeline (via Loki).

When ClearLedger handles HTTP traffic or a failed login, those events feed the Service Health dashboards (Loki and Prometheus together).

Vault (Stage 5) and network policies (Stage 6) don't always have their own panel, but they still matter: fewer secrets in Git and blocked pod traffic show up indirectly in a healthier, quieter cluster.

What you'll do in this stage

You'll run a command in the terminal (for example, a Kyverno violation or a Falco trigger) and then wait a short time while Prometheus or Loki ingests the event. Within about 15–90 seconds, the matching Grafana panel should update.

That's the whole point of observability for security: the terminal proves the event happened once, while the dashboard proves you can detect and measure it later without being logged into the cluster at that exact moment.

7.0: Free Node Resources (Scale Down Litmus)

Stage 6.5 is complete. You don't need the Litmus UI, MongoDB, or chaos operator running while Prometheus, Loki, and Grafana start. They compete for the same CPUs on a single-node lab VM (6 by default, see scripts/setup-cluster.sh).

Scaling Litmus to zero frees ~500–800MB RAM and reduces CPU churn before the observability install.

kubectl scale deployment,statefulset -n litmus --replicas=0 --all
kubectl get pods -n litmus
# Expected: no Running pods (Succeeded job pods from chaos experiments are OK)
multipass exec clearledger -- uptime
# Expected: load average (1m) ideally below ~8 before continuing

You can scale Litmus back up later if you want to re-run chaos experiments (bash stages/stage-6.5-chaos-engineering/scripts/install-litmus.sh). For Stages 7–7.5, keep it scaled down.

7.1: Install the Observability Stack

This is safe to run more than once. The script checks what's already installed. If Grafana, Prometheus, and Loki are healthy, it skips the heavy install and only updates dashboards and scrape configs. Running it again after a partial failure won't duplicate or break a working stack.

Only add FORCE=1 if something is genuinely stuck, for example you edited the Helm values files and need a full reinstall, or Loki keeps crashing in a restart loop:

FORCE=1 bash stages/stage-7-observability/scripts/install-observability.sh

On a first-time install, use the plain command in Step 1 below. Don't use FORCE=1 unless the troubleshooting section tells you to.

macOS, Linux, and WSL2: FORCE=1 bash ... works as written.

Native Windows PowerShell doesn't use that syntax.

Run the lab inside WSL2 Ubuntu (recommended), or set the variable first: $env:FORCE=1; bash stages/stage-7-observability/scripts/install-observability.sh.

Step 1: install (wait until the script prints ✓ Stage 7 installed.):

bash stages/stage-7-observability/scripts/install-observability.sh

If you see “Waiting for Falco” during the Stage 7 install, that's expected.

You already installed Falco in Stage 6. Stage 7 is not adding a second Falco. It is making sure the existing Falco setup can feed logs and metrics into the observability stack.

The flow is:

• Falco still runs in the falco namespace.

• Promtail sends Falco logs to Loki.

• Grafana reads those logs from Loki.

• The Security Event Timeline dashboard shows the Falco alerts.

Right after install, the Grafana panels may be empty. That's normal. You need to trigger a new alert in §7.4 before the dashboard has something fresh to show.

Step 2. Check pods (run this after Step 1 finishes):

kubectl get pods -n monitoring

You want something like this (pod name suffixes vary):

NAME                                              READY   STATUS    RESTARTS   AGE
kube-prometheus-stack-grafana-....                3/3     Running   0          5m
kube-prometheus-stack-prometheus-....             2/2     Running   0          5m
loki-0                                            1/1     Running   0          5m
loki-promtail-....                                1/1     Running   0          5m

Grafana must show 3/3 Ready (not 2/3). Loki must show 1/1. If pods are still Pending or ContainerCreating, wait a few minutes and run kubectl get pods -n monitoring again.

Expected – Loki healthy:

kubectl exec -n monitoring loki-0 -- wget -qO- http://127.0.0.1:3100/ready

ready

Expected – Grafana can reach Loki (same path log panels use):

kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- --timeout=5 http://loki:3100/ready

ready

Expected – Grafana UI reachable:

curl -sI http://grafana.local | head -n 1

HTTP/1.1 302 Found

Log into http://grafana.local: admin / admin123

Empty panels right after install are normal. You haven't generated events yet. Continue to §7.2–§7.4.

If Helm fails: wait 30s, then FORCE=1 bash stages/stage-7-observability/scripts/install-observability.sh. See troubleshooting.md. Stage 7.

✋ Hands-on checkpoint: confirm Loki and dashboards are ready

Before you open Grafana, confirm the logging stack and dashboards actually installed.

On a single-node VM, Grafana can look fine while Loki is crash-looping or the ClearLedger dashboards never loaded. If you skip this check, you may spend the rest of Stage 7 debugging empty panels.

Run:

kubectl get pods -n monitoring
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki \
  -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}{"\n"}'
kubectl get configmap -n monitoring -l clearledger_dashboard=1 --no-headers | wc -l

Expected:

• All monitoring pods are Running

• Grafana shows 3/3 Ready

• Loki shows 1/1 Ready

• Loki restart count is 0, or low and not climbing

• The dashboard count is 6

If Loki keeps restarting or the dashboard count is 0, stop here and fix the install before continuing. Empty Grafana panels usually mean Loki or the dashboards are missing, not that the security events failed.

7.2: Verify Prometheus, Loki, and Grafana (before opening dashboards)

Run these three checks so you know which layer is broken if a panel is empty.

Check 1: Prometheus has Kyverno metrics

kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=kyverno_admission_requests_total' 2>/dev/null \
  | head -c 400

(Prometheus runs as a StatefulSet pod, not a Deployment. This query goes through Grafana to the Prometheus Service.)

Expected: JSON with "status":"success" and a "metric" block (values may be 0 until you trigger a violation in §7.4).

If you see "status":"success" but "result":[], Prometheus is up but Kyverno hasn't recorded admissions yet. That's fine before the lab.

Check 2: Loki has Falco logs

kubectl exec -n monitoring loki-0 -- wget -qO- \
  'http://127.0.0.1:3100/loki/api/v1/labels' 2>/dev/null | head -c 300

Expected: JSON listing labels such as "namespace" (and after Falco events, you'll see "falco" in label values).

Quick log search (may return empty lines until §7.4 Exercise B):

kubectl exec -n monitoring loki-0 -- wget -qO- \
  'http://127.0.0.1:3100/loki/api/v1/query?query=%7Bnamespace%3D%22falco%22%7D&limit=3' 2>/dev/null \
  | head -c 500

Expected: "status":"success". "result":[] means no Falco lines in Loki yet, not a broken Loki.

Check 3: Grafana imported ClearLedger dashboards

curl -s -u admin:admin123 'http://grafana.local/api/search?tag=clearledger' | jq -r '.[].title'

Expected: six titles:

ClearLedger - Compliance Posture
ClearLedger - DORA Metrics
ClearLedger - Kubernetes Audit Log Analysis
ClearLedger - Kyverno Policy Violations
ClearLedger - Security Event Timeline
ClearLedger - Service Health + Auth Security

Or in the UI: go. toDashboards then filter tag clearledger. You should see exactly these six (no missing names).

7.3: Your First 10 Minutes in Grafana

This section is only a tour. You're not proving anything yet.

Rule for all of Stage 7: an empty panel usually means no events have happened in the selected time range, not that Grafana is broken. You create the real events in §7.4.

Step 1: Open Grafana

Go to http://grafana.local and log in:

• Username: admin

• Password: admin123

Step 2: Set the Time Range

In the top-right corner, choose Last 15 minutes.

Keep this setting for all of Stage 7. Wider ranges like Last 24 hours can overload Loki on a single-node lab VM.

Step 3: Open Dashboards One at a Time

Open one dashboard, look around, then move to the next. Don't open all six at once.

1. Kyverno Policy Violations: policy blocks from Stage 4.

2. Security Event Timeline: Falco alerts from Stage 6. You may see old postgres noise in the log table.

3. Service Health + Auth: app traffic and login attempts.

4. Compliance Posture: summary view for auditors. Skim it and come back after §7.4.

5. Audit Log Analysis: empty on MicroK8s by design (audit pipeline not enabled by default).

6. DORA Metrics: deploy-frequency charts. Needs multiple CI runs to accumulate data. May show blank on first look. Optional.

Use the short dashboard links in this guide. Avoid old bookmarked URLs with long random slugs.

You can also find them in Grafana: go to Dashboards then search tag clearledger.

Step 4: How to Read What You See

Grafana panels pull data from two places:

• Prometheus shows numbers over time, like Kyverno violation counts and request rates

• Loki shows log lines, like Falco alerts and auth-service messages

A big number panel asks: did this count go above zero?

A line chart asks: was there a spike after I ran something?

A logs panel shows the actual text, like rule names, CRITICAL, or Failed login attempt.

If only log panels show connection refused, check Loki again in §7.1.

If number panels work but log panels fail, the problem is likely Loki, not Grafana itself.

Step 5: Move On

Open dashboards 1–3, then continue to §7.4.

That's where you'll run commands in the terminal and watch the panels update with real security events.

7.4: Hands-on Lab: Terminal → Dashboard Proof

This is the core learning section. For each exercise: run the command, wait, then confirm in Grafana.

Timing: wait 30–90 seconds after each command for Prometheus scrape and Loki ingestion.

Two Ways to Do This Lab

Option 1: follow the exercises below (recommended for learning)

Run each command yourself, then check Grafana. That's Exercise A, B, and C.

Option 2: use the guided script

The script runs the same steps and pauses so you can check Grafana between them:

bash stages/stage-7-observability/scripts/generate-dashboard-data.sh

Or:

make demo-7

Both commands do the same thing. The script will say things like “Press Enter after you checked the Kyverno dashboard.” Switch to Grafana, look at the panel, then come back and press Enter.

Want it to run without pauses? (faster, less hand-holding)

SKIP_PROMPT=1 make demo-7

Use Option 1 if you want to understand each step. Use Option 2 if you want a walkthrough. Use SKIP_PROMPT=1 if you just want the data generated quickly.

Exercise A: Kyverno block → Prometheus → Kyverno dashboard

Terminal: apply a pod that violates Stage 4 policy (runs as root):

cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: stage7-kyverno-lab
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
YAML

How to know it worked:

You're testing whether Kyverno blocks a deliberately bad pod. Success means the pod never gets created.

Pass. You should see:

• The terminal prints Error from server and denied the request

• The exact policy names in the error don't matter. Your output might list one rule or several (disallow-root-containers, require-resource-limits, drop-all-capabilities, …). More lines just means more rules failed, that's still a pass.

• The pod name never shows up in the cluster:

kubectl get pods -n clearledger | grep stage7-kyverno-lab

Expected: no output.

If it fails: stop and fix Stage 4 first

• The command ends quietly with created (no error)

• kubectl get pods -n clearledger shows stage7-kyverno-lab

That means Kyverno let a root pod through. Run make check-4 before continuing Stage 7.

Example of a passing terminal (yours may list more policies):

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc" denied the request:
policy disallow-root-containers/validate-run-as-non-root fail: Running as root is not allowed

Confirm Prometheus saw it (optional but useful if Grafana is empty):

kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=kyverno_admission_requests_total{request_allowed="false"}' 2>/dev/null \
  | grep -o '"value":\[[^]]*\]' | head -3

Expected: a "value" entry with a recent Unix timestamp and a number greater than 0 (for example "value":[..., "1"]). If you see this, Kyverno and Prometheus are working even when Grafana panels say No data.

Grafana: open Kyverno Policy Violations.

What you're proving: the terminal denial showed up in Grafana. You don't need every panel to light up. You need one clear sign that Kyverno blocks are being counted.

Step 1: quick sanity check (top row, left to right)

1. Policy Violations (time range): big number. Pass: shows 1 or more. Fail: says No data.

2. Violations (time range): same idea, second counter. Pass: 1 or more.

3. Active Kyverno Rules — usually 18. If this number shows up, Grafana can talk to Prometheus. That is good even when the first two panels are still empty.

Step 2: if the top two numbers work, skim the charts

• Violation Rate by Resource Kind (middle chart): look for a bump labeled Pod around the time you ran kubectl apply.

• Top Blocked Resource Types (bottom-left table): look for a Pod row.

• Violations by Namespace (trend) (bottom-right chart): look for a bump for clearledger.

Charts can lag. A big number > 0 in Step 1 is enough to move on. The charts are bonus proof for §7.6 screenshots.

If the top two panels say "No data" but the terminal denial worked:

This is common. Those panels count new denials during the time range, not the total ever recorded. One denial sometimes lands in Prometheus before Grafana's counter moves.

Try this:

1. Run the same kubectl apply command again (denied again, that is expected).

2. Wait 60 seconds.

3. Click Refresh (circular arrow, top-right).

After a second denial you should see 2 in the top stat panels. Screenshot that for §7.6.

Still empty? Use Explore as backup proof:

1. Grafana left menu → Explore

2. Datasource: Prometheus

3. Paste: sum(kyverno_admission_requests_total{request_allowed="false"})

4. Click Run query

Pass: the result is 1 or 2.

A screenshot of the terminal denial plus Explore showing a number > 0 counts as portfolio proof even if the dashboard stats stay slow.

Exercise B: Falco shell → Loki → Security Event Timeline

What you're doing (same idea as Exercise A):

• Exercise A: you did something bad, Kyverno blocked it, and the Grafana Kyverno dashboard updated.

• Exercise B: you do something suspicious inside a running pod, Falco detects it, and Grafana Security Event Timeline updates.

You already did this in Stage 6 (make demo-6). Here you do it again and prove the alert shows up in Grafana, not only in http://falco.local.

The story in one line: pretend you're an attacker who got shell access inside auth-service: Falco should scream, and the scream should appear on the timeline dashboard.

Step 1: trigger the alert (terminal way)

You're pretending an attacker got into auth-service and ran a quick command (id) to see who they're logged in as. That's suspicious. Falco is supposed to catch it.

The block below is three commands in order. Copy-paste the whole block:

AUTH_POD=$(kubectl get pod -n clearledger -l app=auth-service \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
echo "Using pod: $AUTH_POD"
kubectl exec -n clearledger "$AUTH_POD" -c auth-service -- /bin/sh -c 'id && exit'

What each line does:

1. Line 1: finds the name of a running auth-service pod and saves it in AUTH_POD.

2. Line 2: prints that name so you can see it worked (not empty).

3. Line 3: runs /bin/sh -c 'id && exit' inside that pod. This is the fake “attack.” Falco watches for shells like this.

Pass: you only need these two lines in the output:

Using pod: auth-service-77b7d9cd99-xxxxx
uid=1000 gid=1000 groups=1000

• First line: a real pod name (not blank).

• Second line: the id command ran inside the container.

That's Step 1 done. The pod is still running. You didn't break anything.

Fail: stop and fix before Step 2:

• error: Internal error or container not found

• Using pod: with nothing after it

Run kubectl get pods -n clearledger -l app=auth-service and retry when one pod shows Running.

Step 2: Confirm Falco saw it (terminal, right away)

The Falco log is one long JSON line. Don't try to read the whole thing. Run:

kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=50 | grep -i 'Shell spawned'

Pass. You should see one short phrase somewhere in the line:

Shell spawned in ClearLedger container ... pod=auth-service-... cmd=sh -c id && exit

Or the rule name:

"rule":"Shell Spawned in ClearLedger Container"

That one grep hit means Exercise B worked in the terminal. Screenshot this line for your portfolio.

Ignore:

• Defaulted container "falco" out of: ...: normal kubectl noise

• Lines about postgres-0 and /etc/passwd: background noise from Stage 6, not your test

• The rest of the JSON (output_fields, k8smeta, and so on). You don't need to parse it

If grep prints nothing: run Step 1 again, wait 5 seconds, then re-run the grep.

Step 3: Confirm Loki stored it (wait ~60 seconds first)

The story so far:

• Step 1: you triggered the alert inside auth-service

• Step 2: Falco wrote the alert to its own logs ✓

Step 3 asks: did that log line make it into Loki which is Grafana's log database?

Falco doesn't talk to Grafana directly. Promtail copies Falco's logs into Loki. That copy takes 60–90 seconds. Wait after Step 1, then run this check.

What this command does:

"Search Loki for Falco logs that contain Shell spawned, then show only lines that also mention auth-service."

kubectl exec -n monitoring loki-0 -- wget -qO- \
  'http://127.0.0.1:3100/loki/api/v1/query?query=%7Bnamespace%3D%22falco%22%2Ccontainer%3D%22falco%22%7D%20%7C%3D%20%22Shell%20spawned%22&limit=3' 2>/dev/null \
  | grep -i 'auth-service'

Pass: you see a line with both auth-service and Shell spawned. That means Loki has your alert and Grafana can show it.

Fail (misleading pass): you grep for ClearLedger alone and get a hit from postgres-0 reading /etc/passwd. That's background noise from Stage 6, not your shell test. Always look for auth-service.

Empty output? That's OK. If Step 2 passed, continue to Step 4. Promtail may still be catching up, or the JSON is too long for this quick grep. Grafana often shows the alert even when this command prints nothing.

Step 4: open Grafana

Open Security Event Timeline.

This is the right dashboard. The title at the top should say ClearLedger - Security Event Timeline.

Before you look at panels:

1. Time range: Last 1 hour (top-right)

2. Auto-refresh: Off

3. Re-run Step 1 if your shell command was more than a few minutes ago

4. Wait 90 seconds, then click Refresh

What you'll probably see (and this is normal):

• CRITICAL Alerts (1h): a big number like 1.08 K. That is mostly postgres-0 reading /etc/passwd on a loop (Stage 6 background noise). It does not mean you failed.

• Alerts by Rule Name (pie chart): dominated by Sensitive File Read in ClearLedger. Also normal.

• Recent CRITICAL / WARNING Events: lots of Postgres rows. Your shell alert is in there, but buried.

The top timeline (Falco Alerts by Priority - Timeline) may say No data. That's a known quirk. Don't panic, just use the log panel and browser search instead.

How to find your alert (on this dashboard):

1. Stay on ClearLedger - Security Event Timeline — not Explore, not Tempo.

2. Click inside Recent CRITICAL / WARNING Events (the log list on the right).

3. Press Cmd+F (Mac) or Ctrl+F (Windows/Linux).

4. Search for auth-service or Shell spawned.

If the search finds a row mentioning your pod and Shell spawned, screenshot it.

Wrong place (common mistake): Grafana Explore with datasource Tempo showing ledger-service traces. That's Stage 7.5 (OpenTelemetry), not Exercise B. Tempo shows request traces, not Falco security alerts.

Pass for Exercise B (pick one):

1. Best: Step 2 terminal grep shows Shell spawned and the Security Event Timeline log search finds auth-service / Shell spawned — screenshot both.

2. Also fine: Step 2 grep screenshot plus the Security Event Timeline dashboard with CRITICAL Alerts (1h) showing a number (proves that Falco → Loki → Grafana works, even if your shell row is buried in postgres noise).

3. Fallback (only if the dashboard search fails): Step 2 grep plus Grafana Explore with datasource Loki (not Tempo):

   • Left menu, go to Explore

   • Top-left datasource dropdown: choose Loki

   • Query: {namespace="falco", container="falco"} |= "Shell spawned"

   • Click Run query

   • Look for a line with auth-service

Screenshot for §7.6.

Exercise C: Failed login, Loki, and Service Health

The story: someone is guessing passwords on your login API.
You send ten bad login attempts from the terminal. auth-service writes Failed login attempt to its logs. Grafana Service Health + Auth Security should show the count go up.

Same pattern as A and B: terminal action, then logs, then dashboard.

Step 1: send bad login attempts (terminal)

Copy-paste the whole block:

for i in $(seq 1 10); do
  curl -s http://clearledger.local/auth/health >/dev/null
  curl -s -X POST http://clearledger.local/auth/login \
    -H 'Content-Type: application/json' \
    -d '{"email":"lab-attacker@evil.com","password":"wrong"}' >/dev/null
done
echo "done"

Pass: the only output you need is:

done

No output from the curl lines is normal. The loop hits /auth/health (keeps the app warm) and /auth/login with a wrong password ten times.

Fail: curl: (6) Could not resolve host. Run bash scripts/setup-hosts.sh on your Mac. curl: (7) Failed to connect. Check kubectl get pods -n clearledger -l app=auth-service.

Step 2: Confirm auth-service logged it

kubectl logs -n clearledger -l app=auth-service --tail=30 | grep -i 'Failed login' | tail -3

Pass. You should see lines like:

Failed login attempt for email: lab-attacker@evil.com

You may see several lines (one per failed attempt). One line is enough. Screenshot this for your portfolio.

If grep prints nothing: wait 10 seconds and run again. If still empty, check the auth pod is Running: kubectl get pods -n clearledger -l app=auth-service.

Step 3: open Grafana (wait ~60 seconds after Step 1)

Open Service Health + Auth Security.

This is the right dashboard. The title should say ClearLedger - Service Health + Auth Security.

1. Time range: Last 1 hour

2. Auto-refresh: Off

3. Click Refresh once

What to check (only these matter for Exercise C):

1. Failed Login Attempts (1h): big number. Pass: > 0. This is your main proof.

2. Failed Login Log Stream: log lines in the panel. Pass: lines with Failed login attempt or lab-attacker@evil.com. Use Cmd+F inside the panel if needed.

Panels you can ignore if empty:

• Successful Logins: fine at 0 (you only sent bad passwords)

• Request Rate by Service: may be empty until §7.5 metrics images. Not required for Exercise C.

You pass Exercise C when you have two screenshots:

Screenshot 1 (required): your Step 2 terminal output showing Failed login attempt for lab-attacker@evil.com. This proves the app logged the bad logins.

Screenshot 2 (pick one of these):

• Option A: the Failed Login Attempts (1h) panel showing a number greater than zero (for example 10). This proves Grafana counted the failures.

• Option B: the Failed Login Log Stream panel showing a line with lab-attacker@evil.com. Use this if the big number panel is still empty but the log stream has your email.

You need Screenshot 1 and either Option A or Option B. That's enough for §7.6.

Exercise D: Compliance dashboard (the auditor summary)

What you're doing: open one dashboard that rolls up Exercises A, B, and C. This is the “show the auditor” view: admission control + runtime detection + application security in one screen.

When: only after you finished A, B, and C.

Step 1: open the dashboard

Compliance Posture

Set Last 1 hour, auto-refresh Off, click Refresh.

Step 2: Check the top row stats

All three don't need to be huge numbers. They just need to be above zero after your tests.

If one stat is still 0: re-run that exercise (A, B, or C), wait 90 seconds, refresh. Policy Violations may need a second Kyverno denial like Exercise A.

This is screenshot #3 for §7.6: the single frame that proves defense-in-depth.

✋ Hands-on checkpoint: are you actually done with Stage 7?

Installing Grafana isn't the goal. Detection is: you triggered real events and can see them on dashboards.

Optional terminal check (proves Grafana is wired up):

curl -s -u admin:admin123 'http://grafana.local/api/search?tag=clearledger' | jq -r '.[].title'

curl -s -u admin:admin123 'http://grafana.local/api/datasources' | jq -r '.[].name'

First command, you should see six dashboard names:

• ClearLedger - Kyverno Policy Violations

• ClearLedger - Security Event Timeline

• ClearLedger - Service Health + Auth Security

• ClearLedger - Compliance Posture

• ClearLedger - Kubernetes Audit Log Analysis

• ClearLedger - DORA Metrics

Second command, you should see at least:

• Prometheus

• Loki

What does NOT mean you're done:

make check-7 only checks that monitoring pods are running. Green output there does not replace §7.4.

What DOES mean you are done:

You ran Exercises A, B, and C in §7.4 and saved the §7.6 screenshots:

1. Kyverno denial (terminal + dashboard)

2. Falco shell alert (terminal + Security Event Timeline)

3. Failed logins (terminal + Service Health)

4. Compliance Posture summary (all three stats above zero)

If you have those four screenshots, Stage 7 is complete.

7.5: Fill in the Request Rate Chart (Optional)

This is not required for Stage 7. Exercises A–C and §7.6 screenshots don't need this section. Skip it if you're happy moving on.

Also, this is not the same as Stage 7.5 (OpenTelemetry/Tempo). This subsection is only about the Request Rate by Service chart on the Service Health dashboard.

What this section is for:

On Service Health + Auth Security, the Failed Login panels work from logs (Loki). The Request Rate by Service chart needs something different: app pods must expose a /metrics endpoint so Prometheus can scrape request counts.

The code is already in the repo (app/*/prom_metrics.py). Prometheus is already configured to scrape it (clearledger-podmonitor.yaml). The usual problem: your cluster is still running older images from before that code was in your build.

Step 1: check if you already have metrics (30 seconds)

Run this first. If it passes, skip the rest of §7.5.

kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=http_requests_total' 2>/dev/null \
  | grep -o '"__name__":"http_requests_total"' | head -1

Pass: prints "__name__":"http_requests_total". Open Service Health, refresh, and the Request Rate by Service chart should already have lines.

No output: continue to Step 2.

Step 2: deploy images that expose /metrics

Pick one path.

Path A: GitOps (if you have been using CI/CD since Stage 1–2)

1. Push a commit to main on your app repo.

2. Wait for CI to build new images and update clearledger-infra.

3. Wait for ArgoCD to show Synced and Healthy on the clearledger app.

4. Go to Step 3.

Path B: lab shortcut (faster, local only)

export DOCKER_USERNAME=your-dockerhub-user
bash stages/stage-7-observability/scripts/build-metrics-images.sh

This builds, pushes, and rolls out metrics-enabled images for all three services.

Heads-up: ArgoCD self-heal may revert these image tags within a few minutes if clearledger-infra still points at older tags. That's fine for a quick lab demo. For a lasting fix, use Path A or update the infra repo (see §2 rollback notes).

Step 3: verify metrics landed (~60 seconds after rollout)

kubectl exec -n clearledger deploy/auth-service -c auth-service -- \
  wget -qO- http://127.0.0.1:8000/metrics 2>/dev/null | head -5

Pass: lines starting with # HELP or http_requests_total.

Then confirm Prometheus sees them:

kubectl exec -n monitoring deploy/kube-prometheus-stack-grafana -c grafana -- \
  wget -qO- 'http://kube-prometheus-stack-prometheus.monitoring:9090/api/v1/query?query=http_requests_total' 2>/dev/null \
  | grep -o '"__name__":"http_requests_total"' | head -1

Pass: "__name__":"http_requests_total"

Generate a little traffic (re-run the Exercise C curl loop or hit http://clearledger.local/auth/health a few times), wait 60 seconds, then open Service Health + Auth Security and refresh. Request Rate by Service should show lines for auth-service, ledger-service, or notification-service.

When to stop:

• Request Rate still empty but Failed Login panels work? You're done with Stage 7. Request Rate is a nice-to-have.

• Prometheus query passes but chart empty? Widen time range to Last 1 hour, generate traffic, wait 60s, refresh.

7.6: Wrap up Stage 7 (Screenshots + Done Check)

You're almost done. This section is just about saving proof, then moving on.

Are you actually finished?

Opening Grafana and seeing six dashboards isn't enough. make check-7 passing isn't enough either. That only proves pods are running.

You're done when you ran §7.4, waited for the panels to update, and saved three screenshots from your cluster.

If the panels are empty or only show old Postgres noise, go back to §7.4 first.

Before each screenshot: set time range to Last 15 minutes (or Last 1 hour for Exercise B). Include the time picker and panel titles in the frame.

Screenshot 1: Falco alert (Exercise B)

Open Security Event Timeline.

Capture Recent CRITICAL / WARNING Events with a row that mentions Shell spawned or auth-service. If postgres rows bury it, use Cmd+F inside the log panel, that still counts.

Screenshot 2: Kyverno denial (Exercise A)

Open Kyverno Policy Violations.

Capture Policy Violations (time range) or Violations (time range) showing a number of 1 or more.

Screenshot 3: Compliance summary (Exercise D)

Open Compliance Posture.

Capture the top row with all three stats above zero: Policy Violations, Runtime Threats, and Failed Auth Attempts.

Screenshot 4 (optional): Failed logins (Exercise C)

Open Service Health + Auth Security.

Capture Failed Login Attempts (1h) above zero, or Failed Login Log Stream showing lab-attacker@evil.com.

Save files somewhere sensible, like docs/evidence/stage-7-screenshot-1-falco.png. Name them so you know what each proves.

Final check: run make check-7 (§7.7), save your VM, and you can claim Stage 7.

7.7: Verify

make check-7

Expected:

▶ Stage 7 — Observability (Grafana + Prometheus + Loki)
  ✓ Prometheus is running
  ✓ Grafana reachable (http://grafana.local or in-cluster health OK)
  ✓ Loki pod is running (0 restarts)
  ✓ Loki reachable from Grafana (http://loki:3100/ready)
  ✓ ClearLedger alerting rules exist
  ✓ ClearLedger dashboards imported (6 found)

Warnings about Loki restarts or missing dashboards: fix with §7.1 before claiming Stage 7 complete.

Save your VM after §7.6 and make check-7. See the block at the end of Stage 7 below.

7.8: What Broke (Lab Notes + Interview Talking Points)

The stack in one sentence: Prometheus stores numbers (metrics), Loki stores log lines, and Grafana displays both visually. Nothing appears until something actually happens in the cluster.

What tripped you up in the lab

1. Empty dashboards right after install: Normal. Grafana doesn't create events. You trigger them in §7.4 (Kyverno denial, Falco shell, failed logins).

2. Loki slow or refresh stuck on “Cancel”: Falco logs are huge. Last 24 hours overloads a small cluster. Use Last 1 hour, one dashboard at a time, and wait ~10 seconds.

3. make check-7 passed but panels still empty (lab checklist only, not an interview topic): The health check confirms Prometheus/Loki/Grafana pods are up. It does not mean events exist. You still need §7.4 + §7.6 before you snapshot and move on.

If someone asks about this in an interview

Empty dashboards? Grafana only shows what already happened. No event in the time range means an empty panel. That's normal until you trigger something.

Loki slow on a small cluster? Falco logs are huge. We kept time ranges short (15 minutes, not 24 hours) and opened one dashboard at a time. Same trade-off you would make in prod on limited hardware.

How did you prove it worked? I ran the attacks myself: denied a bad pod, spawned a shell in a running container, and sent failed logins. Then I checked Grafana and screenshot the matching panels. Terminal action first, dashboard proof second.

Short version you can say out loud:

"I connected Kyverno and Falco into Grafana. To prove it, I triggered a policy block and a runtime alert, then showed both on security dashboards. On a single-node lab, Loki got slow with wide time ranges, so we kept queries tight."

Pipeline problems from earlier stages (Trivy, Kyverno, image tags, and so on) are in docs/troubleshooting.md — not something you need to rehearse for Stage 7.

7.9: If Panels Look Wrong After a Repo Update

Re-apply dashboards, then generate real events (§7.4, not fake data):

bash stages/stage-7-observability/scripts/install-observability.sh
# Then run Exercises A–C from §7.4 (Kyverno denial, Falco shell, failed logins)

Open Grafana at Last 1 hour, wait ~30–60s after each exercise, and refresh once. The §7.4 exercises cover expected appearance for each dashboard.

What You Learned in Stage 7

• Prometheus proves countable security events (Kyverno denials, HTTP rates)

• Loki proves forensic detail (Falco JSON, auth log lines)

• Grafana is the narrative layer, not a second install step after the lab

• You can trace: terminal action, backend signal, panel update

• ServiceMonitors / PodMonitors are what connect Stages 4–6 to charts

• Empty dashboards mean “no events yet” or “wrong time range”, not “broken security”

• Compliance posture is how you answer an auditor in one screen

• Network policies must explicitly allow the monitoring namespace to reach app pods on port 8000, otherwise PodMonitor scrapes silently fail with context deadline exceeded

• Kubernetes Audit Log dashboard is empty on MicroK8s by design: the API server audit pipeline (audit-policy → file → Promtail → Loki) isn't enabled by default

• Request Rate requires the full chain: app image with /metrics, PodMonitor, and network policy: any one missing means the panel stays empty

What you can now put on your CV / say in an interview:

Built security observability with Prometheus, Loki, and Grafana (dashboards correlating Kyverno violations, Falco alerts, and DORA metrics) and can prove a security event end-to-end from terminal to dashboard.

Stage 7 done checklist:

• make check-7 → 6/6 ✓ (Stage 6.5 Litmus failure is expected: scaled down for memory)

• http://grafana.local/d/clearledger-kyverno-violations. Violations stat > 0

• http://grafana.local/d/clearledger-security-events. CRITICAL Falco alert visible

• http://grafana.local/d/clearledger-compliance. Policy Violations + Runtime Threats + Failed Auth Attempts all > 0

• http://grafana.local/d/clearledger-service-health. Failed Login Attempts > 0. Request Rate > 0 only if you did §7.5

• Portfolio screenshots 1–3 saved

make snapshot STAGE=7 && make snapshots. Confirm clearledger.stage7. Don't skip this. Stage 7 is heavy, and disk pressure is common. See How to Save Your Progress.

After a Mac reboot or sleep, auth/ledger pods may show Unknown or Init:0/1 even though the cluster is up (see troubleshooting.md) Mac reboot).

Stage 7.5 — OpenTelemetry (Optional)

You can skip this whole stage. Stage 7 (metrics + logs) is enough to finish the homelab and move to Stage 8.

Only do Stage 7.5 if you want distributed traces for your portfolio or interviews, and your VM has spare RAM (about 1.5 Gi free).

What You Are Adding

Stage 7 answers: did something happen? (Kyverno blocked a pod, Falco saw a shell, or login failed.)

Traces answer: what steps ran on this one request, and how long did each take?

• Metrics: how many requests, how many errors

• Logs: what the app printed in its log file, such as errors, warnings, login failures)

• Traces: ledger-service called auth-service (12ms), then Postgres (8ms)

In this stage, you send one real transaction, then open that request in Grafana Explore (Tempo). You'll see each step listed with its timing: ledger-service, auth-service, Postgres.

Before You Start

1. Finish Stage 7: §7.4 exercises done, §7.6 screenshots saved, SKIP_CHAOS_CHECK=1 make check-7 passes.

2. Check VM memory: multipass exec clearledger -- free -h , want about 1.5 Gi free.

3. If you ran Stage 6.5 Litmus, scale it down first (§7.0).

You're done when you see the full request trace in Grafana Explore (Tempo datasource) and make check-75 passes. Then make snapshot STAGE=75.

Ignore This Warning in App Logs

Since Stage 7 you may see:

WARNING: Transient error StatusCode.UNAVAILABLE encountered while exporting traces

That is harmless. The apps are already set up to send trace data, but the receiver isn't installed until §7.5.3.

Your apps still work fine, the trace data just gets thrown away. Installing the collector in §7.5.3 makes the warning go away.

How Tracing is Wired

1. Your apps send trace data when a request runs

2. OTel Collector receives it (port 4317) and passes it along

3. Grafana Tempo stores it

4. Grafana Explore (Tempo selected) is where you look at one request step by step

Apps talk to the collector only, not to Tempo directly. That way you can change where traces are stored later without rebuilding the apps.

7.5.1: Check Memory and Load

Tempo needs ~300MB. Confirm headroom before installing:

multipass exec clearledger -- free -h    # want ~1.5Gi available
multipass exec clearledger -- uptime      # load should be reasonable for your CPU count
SKIP_CHAOS_CHECK=1 bash scripts/health-check.sh 7

If Litmus is still running from Stage 6.5, scale it down first (§7.0):

kubectl get pods -n litmus --field-selector=status.phase=Running
# Expected: no resources found

7.5.2: Install Grafana Tempo

Tempo is the trace storage backend. Install it into the monitoring namespace next to Prometheus and Loki:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install tempo grafana/tempo \
  --namespace monitoring \
  --set tempo.storage.trace.backend=local \
  --set tempo.storage.trace.local.path=/var/tempo \
  --set persistence.enabled=true \
  --set persistence.size=5Gi \
  --wait

Verify Tempo is running:

kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
# Expected: tempo-0   1/1   Running

kubectl exec -n monitoring tempo-0 -- wget -qO- http://localhost:3200/ready
# Expected: ready

7.5.3: Deploy OTel Collector and Wire Grafana

This applies the OTel Collector (receives spans from app pods) and registers Tempo as a Grafana datasource automatically via the sidecar:

kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/otel-collector.yaml
kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/grafana-datasource-tempo.yaml

Verify the collector is running:

kubectl get pods -n monitoring -l app=otel-collector
# Expected: otel-collector-xxxxx   1/1   Running

Verify the collector started (not trace receipt yet):

Apps push spans to the collector over OTLP: the collector doesn't scrape pods. At this step you're only confirming that it's listening.

kubectl logs -n monitoring deploy/otel-collector --tail=15
# Expected:
#   Starting GRPC server ... endpoint: 0.0.0.0:4317
#   Starting HTTP server ... endpoint: 0.0.0.0:4318
#   Everything is ready. Begin running and processing data.
# No crash loops or repeated errors.

Proof that traces are actually flowing comes later: after you generate traffic in §7.5.6, check collector logs for span export lines from the debug exporter, then confirm the trace in Grafana Tempo (§7.5.7).

7.5.4: Enable Prometheus Remote Write Receiver

The OTel Collector also forwards OTel metrics to Prometheus via remote write. Prometheus needs to accept them:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f stages/stage-7-observability/infra/helm/kube-prometheus-stack-values.yaml \
  --wait

This applies the enableRemoteWriteReceiver: true setting added to the Helm values in Stage 7.5. Wait for Prometheus to restart (about 60 seconds).

7.5.5: Verify App Pods Connect to the Collector

The deployments in clearledger-infra already have OTEL_EXPORTER_OTLP_ENDPOINT set. Once the collector is running, the pods auto-connect.

Confirm that the OTEL warnings are gone:

kubectl logs -n clearledger deploy/ledger-service -c ledger-service --tail=20 2>/dev/null \
  | grep -v "opentelemetry\|otlp\|Transient" | tail -10
# Expected: only INFO request logs, no WARNING: Transient error

If warnings persist, the network policy may not have port 4317 egress. Apply the latest policies:

kubectl apply -f infra/deferred-by-stage/stage-6-runtime-security/netpol/network-policies.yaml

7.5.6: Generate a Trace

Now create a transaction and watch it flow through the system:

# Step 1: register (skip if already registered)
curl -s -X POST http://clearledger.local/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"trace-demo@clearledger.io","password":"TracePass123"}' | python3 -m json.tool

# Step 2: login and grab the token
TOKEN=$(curl -s -X POST http://clearledger.local/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"trace-demo@clearledger.io","password":"TracePass123"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
echo "Token acquired: ${TOKEN:0:20}..."

# Step 3: create a transaction (this is the request you will trace)
curl -s -X POST http://clearledger.local/ledger/transactions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"amount": 5000, "direction": "credit"}' | python3 -m json.tool

Verify the collector received spans:

kubectl logs -n monitoring deploy/otel-collector --tail=30 \
  | grep -iE "Traces|spans|ResourceSpans" || echo "No span lines yet — see §7.5.5 (OTEL env / netpol)"
# Expected after a successful transaction: debug exporter lines mentioning exported traces/spans

7.5.7: View the Trace in Grafana

Open http://grafana.local and go to the left sidebar Explore (compass icon).

Step 1: Select Tempo and open Search

At the top of the query pane:

1. Datasource dropdown (orange T logo) → Tempo

2. Query row labeled A (Tempo) → three tabs: Search | TraceQL | Service Graph

3. Click Search. This shows dropdown filters. TraceQL is a text box only. If you land there with nothing typed you get 0 series returned.

Step 2: Filter by service

In the Search tab:

• Service Name → type or select ledger-service

• Leave Span Name, Status, Duration, and Tags empty for now

• Grafana shows the query it will run: {resource.service.name="ledger-service"}

Set the time range (top-right clock icon) to Last 15 minutes so your §7.5.6 transaction is included.

Step 3: Run the query

Grafana Explore has no “Run query” button: results appear automatically after selecting a service. If the table stays empty, use the blue refresh button top-right of the pane.

Step 4: Open the trace waterfall

Below the query editor, find Table - Traces. You should see at least one row like:

Click the Trace ID link. The right panel opens the trace detail view.

What the trace detail view shows

Header: ledger-service: POST /transactions

• Trace ID: unique ID for this request

• Duration: total end-to-end time

• Services: 2 (ledger-service and auth-service for a normal transaction)

Expand spans in the timeline:

ledger-service   POST /transactions          (~total duration)
  ├── auth-service   GET /verify             ← JWT check over HTTP
  ├── ledger-service INSERT / sqlalchemy    ← Postgres write
  └── (optional) redis PUBLISH              ← only if amount ≥ notification threshold

Reading the trace detail screen:

Each row is one step in the request (Grafana calls it a span). The colored bar on the right shows how long that step took. That's the span bar. A longer bar = more time spent on that step.

Click a row or its bar to open the details panel on the right. You'll see two kinds of metadata:

• Span attributes: what happened in this step.
  Examples: HTTP method (POST, GET), status code (200), or SQL text on a database step. In your trace you might see asgi.event.type: http.request on the FastAPI receive step.

• Resource attributes: where the step ran.
  Examples: service.name: ledger-service, k8s.cluster.name: clearledger, deployment.environment: production.

Quick mental model: span attributes = what the step did. Resource attributes = which service produced it.

Connecting traces to logs: once you have a step selected, the Logs tab will take you straight to the matching Loki log lines for that pod at the same moment in time.

Screenshot this trace detail view: portfolio proof for Stage 7.5.

TraceQL alternative

If you prefer the text box, Switch to the TraceQL tab, and paste:

{ resource.service.name = "ledger-service" }

If the table is empty

If TraceQL says 0 series returned: Use the Search tab instead, or paste the TraceQL query from above into the TraceQL tab.

If search tab has no rows: Widen the time range to Last 15 minutes, re-run the transaction curl from §7.5.6, wait a few seconds, and refresh.

If grafana can't connect to Tempo: The datasource URL needs port 3200. Re-apply the datasource and restart Grafana:

kubectl apply -f stages/stage-7.5-opentelemetry/infra/otel/grafana-datasource-tempo.yaml
kubectl rollout restart deployment/kube-prometheus-stack-grafana -n monitoring

If collector logs show no trace data: Work through §7.5.5: usually the OTEL environment variables or network policy blocking port 4317.

7.5.7b: Understand When a Trace Happens

You ran one curl command in §7.5.6. Grafana shows every place that single request traveled.

Think of it like tracking a package:

1. You sent POST /transactions to ledger-service

2. ledger-service asked auth-service: "is this user logged in?"

3. ledger-service saved the row to the database

4. redis only runs if the amount is big (10,000 or more)

Each of those is a row you see in the Tempo detail screen. You're not looking at four separate requests. It's one request with multiple stops.

Why do I see both ledger-service and auth-service?

Because ledger had to call auth before it could save the transaction. Grafana groups those stops into one trip so you can see the full path, not just the first hop.

Why did my demo have no Redis row?

You used "amount": 5000. The app only talks to Redis when the amount is 10,000 or higher. So seeing ledger + auth + database but no Redis is correct.

Want to see Redis? Run §7.5.6 again with "amount": 15000 and search Tempo again.

Optional: connect it to the code

Open app/ledger-service/main.py, find create_transaction, and read top to bottom. The Tempo rows follow that function in order: check the user, save to the database, and maybe notify Redis.

Optional: same request, three tools

At the time you ran the curl:

• Tempo (this stage): which services ran and how long each took

• Loki (Stage 7): what the apps wrote in their log files

• Prometheus (Stage 7): how many requests happened around that time

Same moment, three different views. You already used Loki and Prometheus in Stage 7.

7.5.8: Verify

make check-75

Expected output:

▶ Stage 7.5 — OpenTelemetry (Distributed Tracing)
  ✓ OTel Collector is running (1 replica(s))
  ✓ Grafana Tempo datasource ConfigMap exists
  ✓ Tempo is running
  ✓ auth-service has OTEL_EXPORTER_OTLP_ENDPOINT set

If you see a warning instead:

⚠ OTel env vars not found on auth-service, redeploy with updated manifests

check-75 looks for OTEL_EXPORTER_OTLP_ENDPOINT in the deployment manifest. Older Stage 5 manifests may not list it even though tracing works: the Python apps default to http://otel-collector.monitoring.svc.cluster.local:4317 when the env var is missing.

You can proceed if collector logs show spans and Tempo shows your trace. To clear the warning, apply only the app deployments (not the whole kustomize tree: Kyverno may block redis/postgres patches):

kubectl apply -f infra/manifests/auth-service/deployment.yaml
kubectl apply -f infra/manifests/ledger-service/deployment.yaml
kubectl rollout restart deployment/auth-service deployment/ledger-service -n clearledger
make check-75

Save your VM after make check-75. See the block at the end of Stage 7.5 below.

What You Learned

Stage 7 gave you metrics (how busy?) and logs (what was printed?). Stage 7.5 adds traces (for one slow request, which step took the time?).

In the lab you proved it with one POST /transactions curl. In production the idea is the same: a user hits an API, the request crosses multiple services, and you need to see that full path in one place.

If someone asks in an interview:

Why traces at all? Metrics might tell you p99 latency doubled. Logs might show an error on one pod. Traces tell you which downstream call in the chain caused the delay, auth, database, cache, or third-party API, without guessing.

How did you implement it? We instrumented the services with OpenTelemetry, sent telemetry to a collector, and stored traces in Grafana Tempo. Apps talk to the collector, not directly to the backend, so we can change storage later without redeploying every service.

What would you do in an incident? Find a slow or failing trace ID (from logs, metrics, or an alert), open it in Tempo, walk the call chain service by service, see where time stacked up, then jump to logs for that service at the same timestamp. That's faster than tailing logs on five pods and hoping they line up.

Short version you can say out loud:

"We use the three pillars together: Prometheus for rates and errors, Loki for log detail, and Tempo for request-level debugging across microservices. When latency spikes, I start from a trace, identify the slow hop, often a database or downstream API, and correlate back to logs and metrics for that service."

make check-75 && make snapshot STAGE=75 && make snapshots. Confirm clearledger.stage75. See How to Save Your Progress.

Stage 8 — AWS Migration

Your goal here is to run the same ClearLedger app on AWS instead of your laptop VM.

You're not rewriting the application. Stages 0–7 built containers on Kubernetes with GitOps, Kyverno, secrets, and observability. Stage 8 changes where it runs. You keep the same images, the same ArgoCD workflow, and the same security policies. Only the cloud services underneath change (MicroK8s → EKS, Vault → Secrets Manager, and so on).

• Homelab: MicroK8s, Postgres in a pod, dev Vault, Docker Hub, clearledger.local

• AWS: EKS, RDS, Secrets Manager, ECR, ALB hostname

Am I ready for Stage 8?

• Homelab complete through Stage 7 (Stage 7.5 optional)

• make check-7 passes (and make check-75 if you did traces)

• AWS account with billing alerts enabled. make aws-up creates billable resources

• Skim §8.2 so you know what make aws-up does (even if you use the quick path)

Done when the app is reachable on the AWS ALB, ArgoCD syncing, and you run make aws-down when finished to stop charges.

What make aws-up Gives You

This is a demo stack: production-shaped, but not production-ready. It has HTTP only (no TLS cert).

Stage 7 observability is installed automatically. CI still runs Gitleaks, Semgrep, Checkov, Trivy, and Cosign.

For real production, you would add HTTPS (see ingress-aws-https.example.yaml), staging before promote, and alert routing. Those are documented but not applied by the spinup script.

GitOps rule: after bootstrap, don't kubectl apply app Deployments by hand. ArgoCD owns the cluster (Stage 2). Push manifest changes to Git and let ArgoCD sync.

Secrets on AWS

On the homelab, Vault wrote secret files into the pod. On AWS, secrets live in AWS Secrets Manager (created by Terraform). Your app still needs them as environment variables like DATABASE_URL.

ESO (default in this lab): the simple mental model:

1. Terraform stores the real password in AWS Secrets Manager (for example clearledger/auth-service)

2. External Secrets Operator (ESO) watches that AWS secret

3. ESO copies it into a normal Kubernetes Secret inside the cluster (for example auth-service-secret)

4. Your deployment reads DATABASE_URL from that Kubernetes Secret, same as Stage 0, but the values come from AWS instead of a YAML file in Git

You never put passwords in Git. ESO keeps the Kubernetes Secret in sync with Secrets Manager.

CSI (optional, §8.5 exercise): same AWS secrets but different delivery: mounted as files at /mnt/secrets/* instead of env vars. This is closer to how Vault worked on the homelab.

IRSA: how ESO is allowed to read Secrets Manager without storing AWS access keys in the cluster. AWS trusts a Kubernetes service account instead.

IRSA lets AWS trust a Kubernetes ServiceAccount, no AWS_ACCESS_KEY_ID in Git or in the cluster.

Details here: stages/stage-8-aws-migration/docs/secrets-patterns.md.

8.1: Two Ways Through Stage 8

Quick path (~45–60 min): edit terraform/secrets.tf (replace CHANGE_ME_BEFORE_APPLY), then:

make aws-up    # runs stages/stage-8-aws-migration/scripts/aws-spinup.sh
make aws-down  # destroys billable resources when you are done

Read §8.2 afterward so you know what ran.

Manual path (§8.3): run Terraform, ECR push, ArgoCD, Kyverno, ESO, and deploy yourself. Use this when learning, interviewing, or debugging a failed spinup.

Don't skip §8.2–§8.5 if you only ran make aws-up. Otherwise you won't know what Terraform, ESO, or ArgoCD each did.

Before your first Stage 8 push, read §8: CI routing and CLEARLEDGER_CI_TARGET and set CLEARLEDGER_CI_TARGET=aws only after Terraform succeeds, not while you are still on Stages 1–7.

8.2: What make aws-up Runs

The spinup script runs 15 steps in order:

Setup (1–6): Check tools and AWS login; terraform apply (VPC, EKS, RDS, ECR, Secrets Manager, GuardDuty, CloudTrail, IAM), confirm security services, build and push images to ECR, patch manifests/kustomization.yaml with your registry and git SHA, and configure kubectl for EKS.

Platform (7–12): install ArgoCD; Kyverno + cluster policies, Falco, External Secrets Operator + IRSA service accounts, CSI secrets driver, and Stage 7 observability stack.

Deploy (13–15): ArgoCD app clearledger-aws syncs stages/stage-8-aws-migration/manifests/, wait for ALB hostname, and print URL and tear-down reminder.

After the script finishes, open the printed http://<alb-dns>/ in your browser (ClearLedger login UI), or follow §8.3. When to open what for Argo CD and Grafana port-forwards.

Default app deploy uses ESO for secrets. CSI is also installed so you can try file mounts in §8.5 without extra setup.

Terraform layout: there's no terraform.tf file. The terraform {} block (version, providers, optional S3 backend) is at the top of main.tf. Resources are split by topic: vpc.tf, eks.tf, rds.tf, ecr.tf, alb.tf, iam.tf, secrets.tf, security.tf.

Run all commands from stages/stage-8-aws-migration/terraform/.

8.3: Manual Walkthrough

Go to Before you start in this section and run the manual steps from Step A yourself at least once instead of make aws-up. Paths are from the repo root.

Commands install things, while UIs prove they work. Homelab Stages 2 and 7 already taught you to open Argo CD and Grafana in a browser. Stage 8 is the same idea.

But on AWS there's no clearledger.local or grafana.local in /etc/hosts. You use port-forward for control-plane UIs and the public ALB hostname for the app.

When to open what (checkpoint map)

make aws-up runs fifteen steps. You don't need every UI open at once, just know when to look and what success looks like as the script moves along.

First, Terraform builds the AWS foundation. When step 2 finishes, open the AWS Console and confirm the cluster, registry, and database exist before any pods run: EKS clearledger is Active, ECR has four repos including frontend (empty is fine for now), and RDS clearledger-postgres is Available.

See AWS Console (after step 2) for the walkthrough.

Next come container images. After step 4, or after CI — AWS (ECR + OIDC) goes green in GitHub Actions, check ECR: each repo should list your git SHA tag. That's what ArgoCD will pull when the app deploys.

Around step 7 the script installs Argo CD. Port-forward to the UI and confirm the login page loads. You won't see the app yet. You're only checking that GitOps is reachable. Details: Argo CD UI.

Step 12 adds observability. Port-forward to Grafana, log in, and confirm the six ClearLedger dashboards are listed. Panels can stay empty until you generate events. This is the same as Stage 7 on the homelab.

Step 13 applies the clearledger-aws app. Go back to Argo CD → Applications → clearledger-aws. You want Synced, Healthy, and running pods for auth, ledger, and notification.

Step 14 exposes the app on a public URL. Open http://<alb-dns>/ in your browser: you should see the same ClearLedger login UI as homelab clearledger.local, served from the ALB with no /etc/hosts entry.
Use /auth/health and the other health URLs when you want a quick API check from the terminal.

See ALB — first time the app is public.

If you want extra confirmation, the optional check is EC2 → Load Balancers → clearledger: status Active, with healthy targets for frontend and the API services.

On AWS the app is four services behind one ALB: the frontend at / (login, dashboard, transactions) and the three APIs at /auth, /ledger, and /notifications.

Your portfolio screenshot for Stage 8 is the ALB URL showing the UI, like http://clearledger-xxxxxxxxxx.eu-west-1.elb.amazonaws.com with the ClearLedger login or dashboard visible.

For Argo CD and Grafana, keep a dedicated terminal running kubectl port-forward while the browser tab is open. Ctrl+C closes the tunnel.

Before you start

Step A: set real passwords in secrets.tf

Open stages/stage-8-aws-migration/terraform/secrets.tf and search for the literal text CHANGE_ME_BEFORE_APPLY. It appears four times in the file (Postgres password, JWT secret, and two database URLs). Replace every occurrence:

• Postgres password: pick a strong password (same value in all three places that reference it)

• JWT secret: run openssl rand -base64 64 and paste the output

make aws-up will refuse to run if any CHANGE_ME_BEFORE_APPLY text is still in that file.

Step B: terminal checks

aws sts get-caller-identity
terraform --version

# REQUIRED before first terraform apply; GitHub Actions OIDC (ci-aws.yaml) reads this at apply time:
cp stages/stage-8-aws-migration/terraform/terraform.tfvars.example \
   stages/stage-8-aws-migration/terraform/terraform.tfvars
# Edit terraform.tfvars: github_owner = "YOUR_GITHUB_USERNAME"   # your GitHub user or org, not a placeholder

terraform -chdir=stages/stage-8-aws-migration/terraform validate
# Fails with "Set github_owner in terraform.tfvars" until you replace YOUR_GITHUB_USERNAME

Don't run terraform apply until github_owner is set. If you apply with the placeholder, AWS creates IAM role clearledger-github-actions-ecr with trust repo:YOUR_GITHUB_USERNAME/.... CI then fails at Publish images → ECR with Not authorized to perform sts:AssumeRoleWithWebIdentity.

Fix: edit terraform.tfvars → terraform apply again → verify with aws iam get-role below then Re-run failed jobs on the failed Actions run (not the full pipeline).

Steps 1–2: Terraform

cd stages/stage-8-aws-migration/terraform
terraform init -upgrade
terraform apply

# Save outputs:
terraform output -raw ecr_registry_url
terraform output -raw github_actions_ecr_role_arn
terraform output -raw eso_role_arn
terraform output -raw auth_service_irsa_role_arn
terraform output -raw kubeconfig_command
cd ../../..

AWS Console after step 2.

Confirm Terraform created resources before you touch the cluster:

1. EKS → Clusters → clearledger → Status: Active, 3 nodes

2. ECR → Repositories → clearledger/auth-service, ledger-service, notification-service, frontend (0 images until step 4 or CI)

3. RDS → Databases → clearledger-postgres → Available

Verify GitHub can push to ECR (only if you plan to use AWS CI later)

GitHub Actions needs permission to push images to your AWS account. Terraform creates an IAM role for that, but only if you set your real GitHub username in terraform.tfvars before terraform apply.

Check it worked:

aws iam get-role --role-name clearledger-github-actions-ecr \
  --query 'Role.AssumeRolePolicyDocument.Statement[0].Condition.StringEquals."token.actions.githubusercontent.com:sub"' \
  --output text

Good: repo:your-real-username/clearledger:environment:production

Bad: repo:YOUR_GITHUB_USERNAME/clearledger:... you forgot to edit terraform.tfvars.

Fix the file, run terraform apply again, then in GitHub go to Actions and then the failed CI, AWS (ECR + OIDC) run → click Re-run failed jobs. That retries only the push step. You don't need to rebuild and rescan everything.

Skip this whole block if you are only using make aws-up for now and not enabling AWS CI yet.

When do ECR repos appear?

During terraform apply (step 2), not when you docker push. Terraform creates empty image repositories: clearledger/auth-service, ledger-service, notification-service, and frontend, so seeing 0 images right after apply is normal.

Images land later in step 4 (manual docker push) or when GitHub Actions CI succeeds.

Set your AWS CLI region to eu-west-1

Everything in this lab lives in eu-west-1 (Ireland). If your CLI defaults to us-east-1, commands will say resources are missing even though they exist:

aws configure set region eu-west-1
aws configure get region   # expect: eu-west-1

Steps 3–4: Security services + ECR images

AWS_REGION=eu-west-1   # or rely on aws configure set region above

# Step 3: verify security services (must pass --region eu-west-1)
aws guardduty list-detectors --region "${AWS_REGION}"
# Expect: DetectorIds: ["<id>"]  — empty [] means wrong region, not "not created"

aws cloudtrail get-trail-status --name clearledger-trail --region "${AWS_REGION}"
# Expect: IsLogging: true
# Error "Unknown trail ... us-east-1" → you forgot --region eu-west-1

# Step 4: build and push images to the ECR repos Terraform already created
ECR_REGISTRY=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ecr_registry_url)
AUTH_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw auth_service_ecr_url)
LEDGER_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ledger_service_ecr_url)
NOTIFY_ECR=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw notification_service_ecr_url)
TAG=$(git rev-parse --short HEAD)

aws ecr get-login-password --region "${AWS_REGION}" \
  | docker login --username AWS --password-stdin "${ECR_REGISTRY}"

docker build -t "${AUTH_ECR}:${TAG}" app/auth-service && docker push "${AUTH_ECR}:${TAG}"
docker build -t "${LEDGER_ECR}:${TAG}" app/ledger-service && docker push "${LEDGER_ECR}:${TAG}"
docker build -t "${NOTIFY_ECR}:${TAG}" app/notification-service && docker push "${NOTIFY_ECR}:${TAG}"

# Confirm images landed (optional)
aws ecr describe-images --repository-name clearledger/auth-service --region "${AWS_REGION}" \
  --query 'imageDetails[*].imageTags' --output table

ECR console (after step 4 or green CI): open each repository and go. tothe Images tab. You should see tags matching your git commit SHA. If repos are empty, ArgoCD will show ImagePullBackOff later.

GitHub Actions (if using CI instead of manual push): repo → Actions → workflow CI. AWS (ECR + OIDC).

If all jobs are green, publish the images. ECR succeeded. This is the supply-chain proof before deploy.

Step 5: GitOps source of truth

Patch placeholders in kustomization.yaml (same sed as aws-spinup.sh step 5):

AWS_REGION=eu-west-1
ECR_REGISTRY=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ecr_registry_url)
TAG=$(git rev-parse --short HEAD)
KUST=stages/stage-8-aws-migration/manifests/kustomization.yaml

sed -i.bak \
  -e "s|REPLACE_ECR_REGISTRY|${ECR_REGISTRY}|g" \
  -e "s|REPLACE_IMAGE_TAG|${TAG}|g" \
  "${KUST}"
rm -f "${KUST}.bak"

# Region in ESO + CSI manifests (only if not eu-west-1)
if [[ "${AWS_REGION}" != "eu-west-1" ]]; then
  sed -i.bak "s|region: eu-west-1|region: ${AWS_REGION}|g" \
    stages/stage-8-aws-migration/manifests/external-secrets.yaml \
    stages/stage-8-aws-migration/manifests/csi/auth-service-spc.yaml \
    stages/stage-8-aws-migration/manifests/csi/ledger-service-spc.yaml
  rm -f stages/stage-8-aws-migration/manifests/external-secrets.yaml.bak \
        stages/stage-8-aws-migration/manifests/csi/*.bak 2>/dev/null || true
fi

# Verify before commit
grep -E 'newName:|newTag:' "${KUST}"
# Expect: YOUR_AWS_ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/clearledger/... and your git SHA

git add stages/stage-8-aws-migration/manifests/kustomization.yaml
git commit -m "stage8: ECR images ${TAG}"
git push

Also fix the ArgoCD Application repo URL once (replace with your GitHub username):

# Example: YOUR_GITHUB_USERNAME/clearledger — check: git remote get-url origin
sed -i.bak 's|YOUR_GITHUB_USERNAME|YOUR_ACTUAL_GITHUB_USER|g' \
  stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml
rm -f stages/stage-8-aws-migration/argocd/clearledger-aws-app.yaml.bak

Step 6: Cluster access + Terraform outputs

Run from the repo root. Set the CLI region first (EKS and IAM outputs are regional), then kubeconfig, then export IRSA role ARNs: steps 9–10 need them.

aws configure set region eu-west-1

eval "$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw kubeconfig_command)"
kubectl get nodes

export AWS_REGION=eu-west-1
export ESO_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw eso_role_arn)
export FALCO_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw falco_role_arn)
export REPLACE_AUTH_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw auth_service_irsa_role_arn)
export REPLACE_LEDGER_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw ledger_service_irsa_role_arn)
export REPLACE_NOTIFICATION_IRSA_ROLE_ARN=$(terraform -chdir=stages/stage-8-aws-migration/terraform output -raw notification_service_irsa_role_arn)

# Sanity check (all should print ARNs, not empty)
echo "ESO:      ${ESO_ROLE_ARN}"
echo "Falco:    ${FALCO_ROLE_ARN}"
echo "Auth IRSA: ${REPLACE_AUTH_IRSA_ROLE_ARN}"

Steps 7–12: Platform stack on the cluster

You finished steps 1–6 (AWS exists, images in ECR, kubectl works). Now for steps 7-12 you'll install the platform stack, the same components as aws-spinup.sh, but you run the commands from the sections below, not the script.

For each step, run the Install code block, then run the Verify block right under it. Don't move to the next step until you see Running pods (or a ClusterPolicy list). “Command finished with no output” isn't enough.