ORMs like Sequelize, Prisma, TypeORM, and Knex parameterize the queries they build for you automatically. That part works.

The problem shows up in the parts of your app where you step outside that safe path: a raw query the ORM's query builder can't express cleanly, a dynamic sort column, or a single helper function reached for without a second thought.

Those are exactly the places attackers go looking first, and they're easy for developers to miss precisely because the rest of the codebase feels protected.

Four of those patterns are the focus here. Each one gets a working example that triggers the bug, a breakdown of why it's exploitable, and a rewritten version that closes it. I also include notes on exactly what moved and why.

You don't need a security background to follow along. You just need enough comfort with Node.js, SQL, and at least one ORM to read the code.

Prerequisites

The examples assume you're comfortable with:

• The basics of Node.js and Express

• Writing basic SQL queries

• Basic use of an ORM (all code examples use Sequelize, but the patterns apply to Prisma, TypeORM, Knex, and others too)

• How an HTTP request/response cycle works

Note on code examples: Throughout this guide, sequelize is assumed to already be an initialized Sequelize instance, with QueryTypes and Op imported from 'sequelize' alongside it, and User/Product are Sequelize models defined elsewhere in the app. Examples use Sequelize v6+ syntax (the current major version) since some of the APIs referenced here changed between major versions. You can adapt the syntax to the version and ORM you use.

Table of Contents

• 1. Raw Query Escape Hatches

• 2. Identifiers That Can't Be Parameterized

• 3. Second-Order Injection Through Stored Data

• 4. Raw SQL Smuggled Into Normal ORM Calls

• Summary

1. Raw Query Escape Hatches

Every major ORM ships an "escape hatch" for queries the query builder can't express cleanly: sequelize.query(), Prisma's $queryRawUnsafe, or TypeORM's query(). They exist for good reasons, like complex joins, window functions, and vendor-specific SQL.

The trouble starts when developers treat that escape hatch like the rest of the ORM, and interpolate user input directly into the string it builds.

How to Identify This Vulnerability in Your Code

Here's a typical filtered report endpoint:

// Express.js - Vulnerable
app.get('/reports', async (req, res) => {
  const { region } = req.query;
  const results = await sequelize.query(
    `SELECT * FROM sales WHERE region = '${region}'`,
    { type: QueryTypes.SELECT }
  );
  res.json(results);
});

The query builder never sees this string. It's handed straight to the database driver exactly as written.

Why This Matters

A request like ?region=' OR '1'='1 turns the query into SELECT * FROM sales WHERE region = '' OR '1'='1', which is always true. Every row in the table comes back, regardless of region.

Worse, because this code lives inside an ORM-based project, it often doesn't get the same scrutiny a raw mysql.query() call would in a non-ORM codebase. Reviewers assume the ORM already handled it.

How to Fix This Vulnerability

Pass values through the replacement/binding mechanism the raw-query API already provides, instead of building the string yourself:

// Express.js - Secure
app.get('/reports', async (req, res) => {
  const { region } = req.query;
  const results = await sequelize.query(
    'SELECT * FROM sales WHERE region = :region',
    {
      replacements: { region },
      type: QueryTypes.SELECT
    }
  );
  res.json(results);
});

The fix isn't avoiding raw queries entirely, because sometimes you genuinely need them. It's never building the SQL string by hand when a binding mechanism is sitting right there.

Never concatenate user input into a raw query string. Use the replacement/binding API your raw-query method already provides.

2. Identifiers That Can't Be Parameterized

Parameterized queries protect values. They don't protect identifiers like table names, column names, or ORDER BY direction. Those have to be part of the SQL string itself.

That's why a dynamic sort feature is one of the most common places injection sneaks back into otherwise well-written ORM code.

How to Identify This Vulnerability in Your Code

Here's a typical sortable list endpoint:

// Express.js - Vulnerable
app.get('/users', async (req, res) => {
  const { sortBy = 'created_at' } = req.query;
  const users = await sequelize.query(
    `SELECT * FROM users ORDER BY ${sortBy}`,
    { type: QueryTypes.SELECT }
  );
  res.json(users);
});

sortBy goes straight from the query string into the ORDER BY clause, with nothing in between.

Why This Matters

sortBy looks like a harmless UI convenience until someone sends created_at; DROP TABLE users; --. Whether that exact payload runs depends on your database driver: Postgres's pg driver executes stacked statements like this by default, while MySQL's mysql2 blocks them unless multipleStatements: true is explicitly set.

Either way, the underlying problem is the same: you have arbitrary SQL sitting in a position that was only ever meant to hold a column name.

How to Fix This Vulnerability

Since identifiers can't be bound as parameters, the only safe option is an allowlist:

// Express.js - Secure
const ALLOWED_SORT_COLUMNS = ['created_at', 'name', 'email'];

app.get('/users', async (req, res) => {
  const { sortBy = 'created_at' } = req.query;
  const column = ALLOWED_SORT_COLUMNS.includes(sortBy) ? sortBy : 'created_at';

  const users = await sequelize.query(
    `SELECT * FROM users ORDER BY ${column}`,
    { type: QueryTypes.SELECT }
  );
  res.json(users);
});

Never pass user input through to an identifier position, even after "sanitizing" it first. Sanitization for identifiers is much easier to get wrong than for values.

Identifiers can't be parameterized. If user input decides a column or table name, allowlist it, don't sanitize it.

3. Second-Order Injection Through Stored Data

This one catches teams off guard because the input was parameterized...the first time it was written to the database.

The injection happens later, when that already-stored value gets reused inside a different, unparameterized query.

How to Identify This Vulnerability in Your Code

Here's a registration flow, and elsewhere in the codebase, an admin search feature:

// Express.js - Vulnerable

// Step 1: user registration, correctly parameterized
app.post('/register', async (req, res) => {
  await User.create({ username: req.body.username });
  res.json({ success: true });
});

// Step 2: somewhere else in the codebase, an admin search feature
app.get('/admin/search', async (req, res) => {
  const user = await User.findByPk(req.params.id);
  const results = await sequelize.query(
    `SELECT * FROM audit_log WHERE actor = '${user.username}'`,
    { type: QueryTypes.SELECT }
  );
  res.json(results);
});

Step 1 is completely safe on its own. Step 2 is where things go wrong.

Why This Matters

A username like admin' OR '1'='1 passes through step 1 without any issue. Sequelize's create() parameterizes it, so it's stored exactly as typed. It sits in the database looking completely normal.

The payload only fires when step 2 pulls that value back out and drops it straight into a raw query string. "It's already in our own database" is not the same thing as "it's safe". It's still attacker-controlled if it originated from user input anywhere upstream.

How to Fix This Vulnerability

Same fix as pattern #1. You bind the value instead of concatenating it:

// Express.js - Secure

// Step 1 (registration) is unchanged — it was already parameterized

app.get('/admin/search', async (req, res) => {
  const user = await User.findByPk(req.params.id);
  const results = await sequelize.query(
    'SELECT * FROM audit_log WHERE actor = :actor',
    {
      replacements: { actor: user.username },
      type: QueryTypes.SELECT
    }
  );
  res.json(results);
});

The bug here is harder to spot than pattern #1, because the tainted data crosses a database round-trip before it becomes dangerous.

Data from your own database isn't automatically safe. If it originated from user input anywhere upstream, it still needs to be parameterized everywhere it's used.

4. Raw SQL Smuggled Into Normal ORM Calls

Pattern #1 covered the obvious raw-query escape hatch, a method you reach for on purpose when you need real SQL. This one is sneakier: injection hiding inside a call that looks completely ORM-managed. It's not raw SQL, right? It's just one helper function among many.

How to Identify This Vulnerability in Your Code

Here's a typical filtered product listing:

// Express.js - Vulnerable
app.get('/products', async (req, res) => {
  const { minPrice } = req.query;
  const products = await Product.findAll({
    where: sequelize.literal(`price > ${minPrice}`)
  });
  res.json(products);
});

Product.findAll() looks like a fully safe, parameterized ORM call...right up until sequelize.literal() shows up inside it.

Why This Matters

literal() tells Sequelize "don't touch this, insert it into the SQL exactly as written." Anything interpolated into it is exactly as injectable as pattern #1, just wearing a normal-looking ORM method as a disguise.

Send ?minPrice=0 OR 1=1 and the WHERE clause becomes unconditionally true: every row comes back, price filter or not. No stacked statement needed this time. It's a single boolean expression, so it works the same regardless of database driver.

How to Fix This Vulnerability

Stop reaching for literal(). Sequelize's own operator API already covers this case:

// Express.js - Secure
app.get('/products', async (req, res) => {
  const minPrice = Number(req.query.minPrice);

  if (!Number.isFinite(minPrice)) {
    return res.status(400).json({ error: 'minPrice must be a number' });
  }

  const products = await Product.findAll({
    where: { price: { [Op.gt]: minPrice } }
  });
  res.json(products);
});

Op.gt, Op.between, Op.in, and the rest of Sequelize's operator API already cover the vast majority of cases people reach for literal() for, and they parameterize correctly by default. Validating that minPrice is actually a number, before it ever reaches the query, closes the gap even if a future refactor reintroduces literal() somewhere else.

Grep for literal(, fn(, and your ORM's other raw escape hatches everywhere they appear in the codebase, not just inside the queries that are clearly "raw."

Summary

For a quick recap, here's how the four patterns line up:

None of these four bugs takes a skilled attacker to find. Each one is just a value that never made it into the ORM's parameterization path.

What's worth making part of your workflow: parameterize values, allowlist identifiers, treat data pulled from your own database as no safer than a fresh request, and treat literal()/raw() calls as exactly as dangerous as a dedicated raw-query method (no matter how safe the surrounding code looks).

Fixing these gaps is only half the picture. Knowing how someone actually goes looking for them (and chains them together during a real assessment) is the half most developers never see.

If you're curious about that offensive perspective, this deep dive on SQL injection through ORMs walks through exactly how these four patterns get exploited step by step in a live penetration test.

That reads like a milestone, but it's the reason you have a problem.

The same report tested more than a hundred models and found the average security pass rate at 56%, "barely changed from 55% in the first report", with roughly 44% of generation tasks introducing a risky vulnerability.

Functional correctness and security turn out to be separate problems, and only one of them is close to solved.

That result is neither an outlier nor new. At IEEE Security and Privacy in 2022, a team at NYU Tandon ran GitHub Copilot through 89 security-relevant scenarios, generated 1,689 programs, and found roughly 40% of them vulnerable to something on MITRE's CWE Top 25. The paper was later selected as a Communications of the ACM research highlight.

In November 2024, Georgetown's Center for Security and Emerging Technology evaluated five LLMs and reported that almost half the snippets they produced contained bugs that could lead to exploitation. Four years, four independent teams, four methodologies, and the same answer each time.

At ACM CCS in 2023, Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh at Stanford put the developers into the experiment: 47 participants, five security-related programming tasks, three languages, with 33 given an AI assistant and 14 not. The assisted group wrote significantly less secure code, and was more likely to believe the code it wrote was secure.

It's a small study, and it explains why the problem doesn't correct itself: the mechanism that would normally catch this (a developer looking harder at code that worries them) is the exact mechanism the tooling switches off.

Those studies all measure application code. But infrastructure code is the harder case, because a bad security group never fails: it works exactly as written, serving traffic to whoever asks, and the only thing that objects is a person reading a diff.

I can't review that volume by reading it, and neither can anybody else. What I can do is write the rules down in a form a computer checks on every change, which is what Policy as Code means.

In this handbook, I walk you through building that check. We'll point it at a real vulnerable repository, watch the obvious version of it clear five of the nine violations sitting in front of it, and then fix it.

By the end, you'll know how to:

• Write a Rego policy against the JSON that terraform show -json produces.

• Test a policy the way you test application code, with fixtures and a coverage report.

• Build a command-line gate with an exit-code contract that a CI pipeline can trust.

• Block non-compliant workloads at Kubernetes admission time using CEL.

• Have a model write a policy and let opa check and your own tests decide whether to keep it.

• Authorise an AI agent's tool calls from the same policy engine.

Table of Contents

• Prerequisites

• Key Terms in Plain English

• Step 1: Fetch Real Infrastructure to Test Against

• Step 2: Write the Tests Before the Policy

• Step 3: Write the Policy Until the Tests Pass

• Step 4: Point It at the Real Plan

• Step 5: Turn the Verdict into an Exit Code

• Step 6: Enforce at Admission Time

• Step 7: Let a Model Write the Policy

• Step 8: Govern the Agent Itself

• Step 9: What I Got Wrong

• Limits of the Check

• Conclusion

Prerequisites

You need:

• A terminal and a working python3 (3.10 or newer).

• jq, for reading JSON at the command line.

• About 700 MB of disk, because the AWS Terraform provider is large.

• An Anthropic API key, but only for Step 7. Every other step runs offline.

mkdir policy-lab && cd policy-lab
python3 -m venv .venv
source .venv/bin/activate
pip install anthropic

curl -L -o opa https://openpolicyagent.org/downloads/v1.20.2/opa_darwin_arm64_static
chmod +x opa && sudo mv opa /usr/local/bin/

curl -L -o tf.zip https://releases.hashicorp.com/terraform/1.14.2/terraform_1.14.2_darwin_arm64.zip
unzip tf.zip && sudo mv terraform /usr/local/bin/

On Windows, activate the environment with .venv\Scripts\activate, and swap the two download URLs for opa_windows_amd64.exe and terraform_1.14.2_windows_amd64.zip.

I ran everything below on OPA 1.20.2, Terraform 1.14.2, and AWS provider 6.x, on macOS. The policy syntax is stable across OPA 1.x.

If you're on OPA 0.x, every rule here needs import rego.v1 added at the top, and I would upgrade instead. The violation counts depend on the AWS provider version only through the shape of the plan JSON, which has been stable since provider 5.

Key Terms in Plain English

• Policy as Code: a rule your organisation has already agreed on, written as a program that takes a proposed change and returns a decision.

• Rego: the query language Open Policy Agent evaluates. It's declarative: a rule body is a list of conditions that must all hold.

• Plan JSON: the machine-readable description of what Terraform is about to do, produced by terraform show -json. This is what the policy reads, so your .tf files never reach it.

• Admission control: the point inside the Kubernetes API server where an object can be rejected before it's stored.

• CEL: Common Expression Language, the small expression language Kubernetes evaluates natively inside the API server, with no webhook to deploy.

• False clearance: a resource the policy passed that it should have failed. Nobody ever notices one, so it goes unmeasured unless you go looking for it.

The same judgement happens in three places, and only the middle one is specific to Kubernetes.

Step 1: Fetch Real Infrastructure to Test Against

I didn't want to invent a vulnerable Terraform file, because inventing one means inventing the bug, and then the policy only catches the bug I planted. So I went looking for code somebody else had written and published.

TerraGoat is a deliberately vulnerable Terraform repository published by Bridgecrew. Fetch its EC2 module at a pinned commit:

SHA=729f8da62c6a85ce4af5ad3d123de97776d954c4
curl -s "https://raw.githubusercontent.com/bridgecrewio/terragoat/$SHA/terraform/aws/ec2.tf" \
  | sed -n '77,96p'

resource "aws_security_group" "web-node" {
  # security group is open to the world in SSH port
  name        = "${local.resource_prefix.value}-sg"
  description = "${local.resource_prefix.value} Security Group"
  vpc_id      = aws_vpc.web_vpc.id

  ingress {
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }

The comment on line two is TerraGoat's own, and port 22 open to the world is the finding it points at.

TerraGoat's module won't initialise on modern Terraform, because it still declares type = "string" in quotes, which Terraform 0.12 deprecated and 1.x rejects. So I lifted the resource into a minimal module of my own, replacing only the two references to TerraGoat's internal locals.

Create main.tf:

terraform {
  required_version = ">= 1.9"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

# Mock credentials. This configuration is only ever planned, never applied,
# so the provider must not try to reach AWS.
provider "aws" {
  region                      = "us-west-2"
  access_key                  = "mock"
  secret_key                  = "mock"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

resource "aws_vpc" "web_vpc" {
  cidr_block = "10.0.0.0/16"
}

# Verbatim from bridgecrewio/terragoat, terraform/aws/ec2.tf, commit 729f8da.
# Only the two references to TerraGoat's own locals are replaced with literals.
resource "aws_security_group" "web-node" {
  name        = "terragoat-sg"
  description = "terragoat Security Group"
  vpc_id      = aws_vpc.web_vpc.id

  ingress {
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  egress {
    from_port = 0
    to_port   = 0
    protocol  = "-1"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  depends_on = [aws_vpc.web_vpc]
  tags = {
    git_commit           = "d68d2897add9bc2203a5ed0632a5cdd8ff8cefb0"
    git_file             = "terraform/aws/ec2.tf"
    git_last_modified_at = "2020-06-16 14:46:24"
    git_org              = "bridgecrewio"
    git_repo             = "terragoat"
  }
}

Produce the plan JSON:

terraform init
terraform plan -out=tfplan.binary
terraform show -json tfplan.binary > plan.json

The mock credentials matter: terraform plan on a create-only configuration never calls AWS, so with skip_credentials_validation and its three siblings the provider won't try to authenticate, and nothing is ever applied.

Step 2: Write the Tests Before the Policy

The rule I wanted was: no security group may expose an administrative port to the public internet.

That sounds like one line of code, and the tests are where I pin down why it's not. Create policy/network_test.rego:

package terraform.network_test

import data.terraform.network

plan(resources) := {"resource_changes": resources}

security_group(ingress) := {
	"address": "aws_security_group.web",
	"type": "aws_security_group",
	"change": {"actions": ["create"], "after": {"ingress": [ingress]}},
}

test_denies_ssh_open_to_the_world if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": 22,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

# A from_port equality check would miss this. The range check does not.
test_denies_wide_open_port_range if {
	fixture := plan([security_group({
		"from_port": 0,
		"to_port": 65535,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 4 with input as fixture
}

test_denies_ipv6_route_to_the_world if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": 22,
		"protocol": "tcp",
		"ipv6_cidr_blocks": ["::/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

test_denies_standalone_ingress_rule if {
	fixture := plan([{
		"address": "aws_vpc_security_group_ingress_rule.ssh",
		"type": "aws_vpc_security_group_ingress_rule",
		"change": {"actions": ["create"], "after": {
			"from_port": 22,
			"to_port": 22,
			"ip_protocol": "tcp",
			"cidr_ipv4": "0.0.0.0/0",
			"cidr_ipv6": null,
		}},
	}])

	count(network.deny) == 1 with input as fixture
}

test_denies_deprecated_standalone_rule if {
	fixture := plan([{
		"address": "aws_security_group_rule.ssh",
		"type": "aws_security_group_rule",
		"change": {"actions": ["create"], "after": {
			"type": "ingress",
			"from_port": 22,
			"to_port": 22,
			"cidr_blocks": ["0.0.0.0/0"],
		}},
	}])

	count(network.deny) == 1 with input as fixture
}

test_allows_ssh_from_a_private_range if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": 22,
		"protocol": "tcp",
		"cidr_blocks": ["10.0.0.0/8"],
	})])

	count(network.deny) == 0 with input as fixture
}

test_allows_https_from_the_world if {
	fixture := plan([security_group({
		"from_port": 443,
		"to_port": 443,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 0 with input as fixture
}

# An all-protocols rule opens every port, whatever its port fields say.
# The first version of this test asserted the opposite and hid the bug.
test_denies_all_protocols_rule_open_to_the_world if {
	fixture := plan([security_group({
		"from_port": 0,
		"to_port": 0,
		"protocol": "-1",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 4 with input as fixture
}

# Ports the policy cannot read are reported, never passed.
test_reports_a_world_open_rule_with_unreadable_ports if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": null,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

test_reports_string_ports if {
	fixture := plan([security_group({
		"from_port": "22",
		"to_port": "22",
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

# Unreadable ports on a rule that is not open to the world stay quiet.
test_ignores_unreadable_ports_on_a_private_range if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": null,
		"protocol": "tcp",
		"cidr_blocks": ["10.0.0.0/8"],
	})])

	count(network.deny) == 0 with input as fixture
}

# `considered` drives the gate's pass-or-vacuous decision, so it needs
# tests of its own even though it takes no part in the judgement.
test_considers_every_ingress_bearing_type if {
	fixture := plan([
		security_group({}),
		{"address": "aws_vpc_security_group_ingress_rule.a", "type": "aws_vpc_security_group_ingress_rule", "change": {"actions": ["create"], "after": {}}},
		{"address": "aws_security_group_rule.b", "type": "aws_security_group_rule", "change": {"actions": ["create"], "after": {}}},
	])

	count(network.considered) == 3 with input as fixture
}

test_does_not_consider_unrelated_types if {
	fixture := plan([{
		"address": "aws_vpc.main",
		"type": "aws_vpc",
		"change": {"actions": ["create"], "after": {}},
	}])

	count(network.considered) == 0 with input as fixture
}

Here's what that file encodes:

1. with input as fixture swaps in a fake plan for one expression, which is how a policy is tested without a cloud account.

2. test_denies_wide_open_port_range expects four violations, one per administrative port, because an ingress rule describes a range. A 0-65535 rule opens SSH exactly as wide as an explicit port 22 rule while sailing past an equality check.

3. Three tests cover three other shapes Terraform uses for the same idea: the IPv6 field, the modern standalone aws_vpc_security_group_ingress_rule, and the deprecated aws_security_group_rule. I didn't write these first, and Step 9 explains where they came from.

4. The two test_allows_ cases matter as much as the denials. A policy that rejects everything passes every deny test and is worthless.

5. The last three tests arrived after the policy was already "finished", and Step 9 explains where they came from. An all-protocols rule opens every port whatever its port fields say, and a rule whose ports the policy can't read has to be reported.

Step 3: Write the Policy Until the Tests Pass

Terraform describes ingress in four shapes, so the policy normalises all four into one set and then judges that set once. Create policy/network.rego:

# METADATA
# title: No admin port is reachable from the public internet
# description: |
#   Terraform describes ingress in four different shapes. Each one is
#   normalised into a single `exposures` set first, so the judgement below
#   is written once and a new shape only costs one more helper rule.
#
#   Two things here are deliberate rather than incidental. An all-protocols
#   rule covers every port whatever its port fields say, and a rule whose
#   ports this policy cannot read is reported rather than passed.
package terraform.network

admin_ports := {22, 3389, 3306, 5432}

public_cidrs := {"0.0.0.0/0", "::/0"}

# The AWS provider writes from_port 0 and to_port 0 for an all-protocols
# rule, which opens every port, so the port fields cannot be read literally.
all_protocols := {"-1", "all"}

# Shape 1 and 2: inline ingress blocks, IPv4 and IPv6.
exposures contains exposure if {
	some resource in input.resource_changes
	resource.type == "aws_security_group"
	some ingress in resource.change.after.ingress
	some field in ["cidr_blocks", "ipv6_cidr_blocks"]
	some cidr in object.get(ingress, field, [])
	exposure := {
		"address": resource.address,
		"protocol": object.get(ingress, "protocol", ""),
		"from_port": object.get(ingress, "from_port", null),
		"to_port": object.get(ingress, "to_port", null),
		"cidr": cidr,
	}
}

# Shape 3: the standalone rule the AWS provider has recommended since v5.
exposures contains exposure if {
	some resource in input.resource_changes
	resource.type == "aws_vpc_security_group_ingress_rule"
	some field in ["cidr_ipv4", "cidr_ipv6"]
	cidr := object.get(resource.change.after, field, null)
	is_string(cidr)
	exposure := {
		"address": resource.address,
		"protocol": object.get(resource.change.after, "ip_protocol", ""),
		"from_port": object.get(resource.change.after, "from_port", null),
		"to_port": object.get(resource.change.after, "to_port", null),
		"cidr": cidr,
	}
}

# Shape 4: the deprecated standalone rule, still in most existing estates.
exposures contains exposure if {
	some resource in input.resource_changes
	resource.type == "aws_security_group_rule"
	resource.change.after.type == "ingress"
	some cidr in object.get(resource.change.after, "cidr_blocks", [])
	exposure := {
		"address": resource.address,
		"protocol": object.get(resource.change.after, "protocol", ""),
		"from_port": object.get(resource.change.after, "from_port", null),
		"to_port": object.get(resource.change.after, "to_port", null),
		"cidr": cidr,
	}
}

# The ports a rule really covers. Undefined when the policy cannot tell.
covered_ports(exposure) := [0, 65535] if {
	exposure.protocol in all_protocols
}

covered_ports(exposure) := [exposure.from_port, exposure.to_port] if {
	not exposure.protocol in all_protocols
	is_number(exposure.from_port)
	is_number(exposure.to_port)
}

deny contains msg if {
	some exposure in exposures
	exposure.cidr in public_cidrs

	# A rule covers a port if that port falls inside [from_port, to_port].
	range := covered_ports(exposure)
	some port in admin_ports
	port >= range[0]
	port <= range[1]

	msg := sprintf(
		"%s: ingress rule exposes port %d to %s",
		[exposure.address, port, exposure.cidr],
	)
}

# A rule open to the world whose ports this policy cannot read is reported.
# Passing it would be the policy guessing in the permissive direction.
deny contains msg if {
	some exposure in exposures
	exposure.cidr in public_cidrs
	not covered_ports(exposure)

	msg := sprintf(
		"%s: ingress rule to %s has ports this policy cannot evaluate (%v to %v)",
		[exposure.address, exposure.cidr, exposure.from_port, exposure.to_port],
	)
}

# Addresses this policy knows how to inspect. The gate uses this to tell
# "nothing violated" apart from "nothing examined".
considered contains resource.address if {
	some resource in input.resource_changes
	resource.type in {
		"aws_security_group",
		"aws_vpc_security_group_ingress_rule",
		"aws_security_group_rule",
	}
}

Reading that from the top:

1. A rule body in Rego is a conjunction. Every line must hold, and some ... in lines iterate, so OPA explores every combination of resource, ingress rule, field, and port.

2. Three separate exposures rules define one set between them, which Rego calls an incremental definition. Adding a fifth shape later costs one more block and changes nothing below it.

3. object.get(ingress, field, []) returns an empty list when a field is absent, so an IPv4-only rule doesn't error when the policy looks for ipv6_cidr_blocks.

4. covered_ports is the safety valve here, because an all-protocols rule reports 0 to 0 in the plan while opening every port, so the port fields can't be read literally, and a rule whose ports are null or strings leaves the function undefined, which the second deny rule turns into a violation.

5. considered isn't part of the judgement. It records which resources this policy can speak about at all, which Step 5 uses to avoid reporting a pass it hasn't earned.

Run it:

opa test policy
opa check --strict policy
opa fmt --diff policy

PASS: 20/20

Add -v to opa test for a line per test.

opa check --strict catches unsafe variables and shadowed imports, while opa fmt --diff prints nothing when the formatting is already canonical. OPA formats Rego with tabs. Both belong in CI, ahead of everything else.

The second policy is ownership tagging, so create policy/tags.rego:

# METADATA
# title: Every managed resource carries ownership tags
# description: |
#   Terraform emits `tags: null` for a resource with no tags at all, so a
#   policy that reaches into `after.tags` skips exactly the resources with
#   the worst tagging. `tags_of` coerces that null to an empty object.
package terraform.tags

required_tags := {"owner", "cost-center", "data-classification"}

# Resource types that genuinely cannot carry tags.
untaggable := {"aws_iam_policy_attachment", "aws_route_table_association"}

in_scope contains resource if {
	some resource in input.resource_changes
	some action in resource.change.actions
	action in {"create", "update"}
	not resource.type in untaggable
}

tags_of(resource) := tags if {
	tags := resource.change.after.tags
	is_object(tags)
} else := {}

deny contains msg if {
	some resource in in_scope
	some tag in required_tags
	value := object.get(tags_of(resource), tag, "")
	trim_space(value) == ""
	msg := sprintf("%s: missing required tag %q", [resource.address, tag])
}

considered contains resource.address if {
	some resource in in_scope
}

Here's why those two lines look the way they do:

1. trim_space(value) == "" does the check. It has to, because in Rego only false and undefined are falsy, so an empty string is truthy. A bare existence check happily accepts owner = "", which is compliance theatre of exactly the kind a tagging policy exists to stop.

2. tags_of, with its else := {} branch, is the fix for a bug I wrote and only found in Step 9.

Step 4: Point It at the Real Plan

Evaluate one package against the TerraGoat plan:

opa eval --data policy --input plan.json --format pretty 'data.terraform.network.deny'

[
  "aws_security_group.web-node: ingress rule exposes port 22 to 0.0.0.0/0"
]

The policy reports one violation, and stays quiet about port 80. It's open to the world in the same resource, because a public web server is the point of a public web server. A check that flags both is a check people learn to ignore.

Running one command per package doesn't scale, and Rego can aggregate across a namespace in a single query:

opa eval --data policy --input plan.json --format pretty \
  'union({v | v := data.terraform[_].deny})'

[
  "aws_security_group.web-node: ingress rule exposes port 22 to 0.0.0.0/0",
  "aws_security_group.web-node: missing required tag \"cost-center\"",
  "aws_security_group.web-node: missing required tag \"data-classification\"",
  "aws_security_group.web-node: missing required tag \"owner\"",
  "aws_vpc.web_vpc: missing required tag \"cost-center\"",
  "aws_vpc.web_vpc: missing required tag \"data-classification\"",
  "aws_vpc.web_vpc: missing required tag \"owner\""
]

{v | v := data.terraform[_].deny} is a comprehension that collects the deny set from every package under data.terraform, and union flattens them. Drop a new policy file into that namespace and it's picked up with no change to the command.

Step 5: Turn the Verdict into an Exit Code

A CI gate communicates through its exit status, and conflating two kinds of failure into one code is how a broken pipeline passes for a month. This tool uses the following:

The fourth row is the one most gates get wrong. If a plan contains no resource any policy knows about, reporting a pass claims an assurance the run can't give. It gets its own verdict name and it doesn't exit 0.

Create policy_gate.py:

"""Evaluate a Terraform plan against a directory of Rego policies."""

import argparse
import json
import pathlib
import shutil
import subprocess
import sys

PASS, FAIL, BROKEN = 0, 1, 2

DENY_QUERY = "union({v | v := data.terraform[_].deny})"
CONSIDERED_QUERY = "union({v | v := data.terraform[_].considered})"


def die(message: str) -> None:
    print(f"policy-gate: {message}", file=sys.stderr)
    sys.exit(BROKEN)


def load_plan(path: pathlib.Path) -> dict:
    try:
        text = path.read_text()
    except OSError as exc:
        die(f"cannot read {path}: {exc.strerror}")
    try:
        return json.loads(text)
    except json.JSONDecodeError as exc:
        die(f"{path}:{exc.lineno}:{exc.colno}: invalid JSON: {exc.msg}")


def query(opa: str, policy_dirs: list[pathlib.Path], plan: pathlib.Path, expr: str) -> list:
    command = [opa, "eval", "--input", str(plan), "--format", "raw"]
    for directory in policy_dirs:
        command += ["--data", str(directory)]
    command.append(expr)

    result = subprocess.run(command, capture_output=True, text=True)
    if result.returncode != 0:
        die(f"opa failed: {result.stderr.strip() or result.stdout.strip()}")
    values = json.loads(result.stdout)
    # A rule that yields anything but strings is a policy bug, and sorting a
    # mixed list would surface it as an unrelated TypeError.
    for value in values:
        if not isinstance(value, str):
            die(f"{expr} produced a {type(value).__name__}; "
                "deny and considered rules must yield strings")
    return values


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--plan", required=True, type=pathlib.Path,
                        help="JSON from `terraform show -json`")
    parser.add_argument("--policy", required=True, nargs="+", type=pathlib.Path,
                        help="one or more directories of .rego files")
    parser.add_argument("--opa", default="opa", help="path to the opa binary")
    args = parser.parse_args()

    if shutil.which(args.opa) is None:
        die(f"{args.opa} is not on PATH")
    for directory in args.policy:
        if not directory.is_dir():
            die(f"{directory} is not a directory")

    plan = load_plan(args.plan)
    if "resource_changes" not in plan:
        die(f"{args.plan} has no resource_changes key; is it a Terraform plan?")

    considered = query(args.opa, args.policy, args.plan, CONSIDERED_QUERY)
    if not considered:
        # Reporting a pass here would claim an assurance the run cannot give.
        print(f"VACUOUS: no policy examined any of the "
              f"{len(plan['resource_changes'])} planned resource(s)", file=sys.stderr)
        return BROKEN

    violations = sorted(query(args.opa, args.policy, args.plan, DENY_QUERY))
    if violations:
        print(f"FAIL: {len(violations)} violation(s) "
              f"across {len(considered)} examined resource(s)", file=sys.stderr)
        for violation in violations:
            print(f"  - {violation}", file=sys.stderr)
        return FAIL

    print(f"PASS: {len(considered)} resource(s) examined, no violations")
    return PASS


if __name__ == "__main__":
    sys.exit(main())

Here's what the script does:

1. argparse marks --plan and --policy as required=True, and --policy takes nargs="+", so an empty policy list raises an error at parse time.

2. load_plan reports the line and column of a JSON syntax error, because json.JSONDecodeError carries lineno and colno and a gate that says only "invalid JSON" wastes somebody's afternoon.

3. Every failure path routes through die, which always exits 2, because a missing opa, an unreadable file, and a plan with no resource_changes key are all failures of the tool itself.

4. The considered query runs before the deny query. If nothing was examined, the run ends at VACUOUS and never gets the chance to print a pass.

5. Violations are sorted, so the same plan produces byte-identical output on every run and a diff of two CI logs means something.

Seven violations across the two resources in this plan, and an exit code CI can act on.

The same tool on an empty plan, where a gate answering "pass" would be lying.

The GitHub Actions workflow tests the policies before it uses them to judge anything:

name: policy

on: [pull_request]

jobs:
  policy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - name: Install OPA
        run: |
          curl -L -o /usr/local/bin/opa \
            https://openpolicyagent.org/downloads/v1.20.2/opa_linux_amd64_static
          chmod +x /usr/local/bin/opa

      # The policies are code. Lint and test them before trusting them.
      - name: Check policy syntax
        run: opa check --strict policy

      - name: Verify formatting
        run: opa fmt --fail --diff policy

      - name: Test policies
        run: opa test policy --verbose --coverage --format json > coverage.json

      # Only now does anything get judged.
      - name: Evaluate Terraform plan
        run: python3 policy_gate.py --plan plan.json --policy policy

Roll this out with the gate reporting only, for a fortnight, before you let it block. A policy that looks obviously correct will fail on something structural in your real estate, and you would rather find that out from a log line than from a blocked release.

Step 6: Enforce at Admission Time

The gate in Step 5 checks what you intended to deploy. It doesn't see a kubectl apply from somebody's laptop, a vendor's Helm chart, or an operator creating Pods on its own schedule. For those, you need admission control, and Kubernetes now has it built in.

ValidatingAdmissionPolicy has been generally available since v1.30, evaluating CEL inside the API server with no webhook to deploy or keep alive:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-trusted-registry
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  variables:
    # A Pod has three container lists. A policy that reads only
    # spec.containers is bypassed by moving the image to an initContainer.
    - name: allImages
      expression: >-
        object.spec.containers.map(c, c.image) +
        object.spec.?initContainers.orValue([]).map(c, c.image) +
        object.spec.?ephemeralContainers.orValue([]).map(c, c.image)
  validations:
    - expression: >-
        variables.allImages.all(i, i.startsWith('registry.internal.example.com/'))
      messageExpression: >-
        'images must come from registry.internal.example.com: ' +
        variables.allImages.filter(i,
          !i.startsWith('registry.internal.example.com/')).join(', ')
      reason: Forbidden

The policy does nothing until a binding activates it, which is what lets you pilot on one namespace:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-trusted-registry-binding
spec:
  policyName: require-trusted-registry
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        policy.example.com/enforce: "true"

Set validationActions: ["Warn", "Audit"], label one namespace, watch for a week, and then switch to ["Deny"] and widen the selector.

The initContainers handling matters, because it's the most common way an image-provenance policy gets bypassed, and the ? optional-field syntax with .orValue([]) is how you read a list that may be absent without the whole expression erroring.

I couldn't apply these two manifests, because I had no cluster to hand. They're checked against the v1 reference schema, and no real API server has admitted them, so treat them as a starting point and roll them out in Warn mode (which you should be doing anyway).

Two other engines are in wide production use, starting with Kyverno. It graduated in the CNCF in March 2026 with production use at Bloomberg, Coinbase, Deutsche Telekom, LinkedIn, and Spotify. Its policies are written in YAML, so a platform team needs no new language, and it handles generation, image-signature verification, and cleanup that built-in policies leave alone.

OPA Gatekeeper is the right answer when you want one Rego codebase covering Kubernetes and Terraform and CI, which is the position this tutorial builds toward. Mutation is now built in too: MutatingAdmissionPolicy became stable in v1.36.

Step 7: Let a Model Write the Policy

Policies are tedious, and models are good at tedious. So the obvious move is to have the model write them.

There's a catch that you can measure yourself in about a minute, and I do exactly that at the end of this step: a great deal of the Rego in public training data is Rego v0, the dialect that stopped parsing when OPA 1.0 shipped in January 2025. A model reaching for the most common pattern it has seen reaches for a dialect the current parser rejects.

A 2025 preprint from a group at the University of Calabria, ARPaCCino, reports the same effect on a Terraform case study: asked for Rego with no tools, Qwen3-30B and GPT-4o each produced 0 of 5 syntactically correct policies. Adding retrieval over the OPA documentation changed nothing. Giving the model a loop that could run opa check and read the errors took those to 4 of 5 and 5 of 5.

Those counts come from one small case study, so treat the direction as the durable part of the result.

A feedback loop is what fixed it, and the loop costs nothing, because you already built it out of opa check --strict, opa fmt, and opa test.

So build the loop with one inversion that makes it trustworthy. I write the tests, and the model writes the policy. Test fixtures are concrete and cheap to review, since you read a JSON blob and say "yes, that should be rejected" in three seconds. Rego with nested comprehensions takes real effort to read and is easy to misread. Put the human where review is cheap, and let the machine work where its output can be checked mechanically.

Create policy_forge.py:

"""Generate a Rego policy from a rule in English, and keep it only if the toolchain agrees."""

import argparse
import pathlib
import re
import subprocess
import sys
import tempfile

WRITTEN, REJECTED, BROKEN = 0, 1, 2

SYSTEM = """You write Open Policy Agent policies in Rego v1 (OPA 1.0+).

Rules:
- Use `if` on every rule body and `contains` for multi-value rules.
- Do not emit `import rego.v1`; it is redundant on OPA 1.0+.
- The input is the JSON from `terraform show -json`.
- Return one ```rego block and nothing else."""


def extract_rego(reply: str) -> str:
    blocks = re.findall(r"```rego\n(.*?)```", reply, re.DOTALL)
    if not blocks:
        raise ValueError("model returned no rego block")
    if len(blocks) > 1:
        raise ValueError(f"model returned {len(blocks)} rego blocks; expected one")
    return blocks[0]


def verify(opa: str, policy: str, tests: pathlib.Path) -> tuple[bool, str]:
    with tempfile.TemporaryDirectory() as tmp:
        bundle = pathlib.Path(tmp)
        # The tests keep their own name; the policy gets one that cannot
        # collide with it, whatever the caller named the test file.
        (bundle / "candidate_policy.rego").write_text(policy)
        (bundle / tests.name).write_text(tests.read_text())

        for command in ([opa, "check", "--strict"], [opa, "test"]):
            result = subprocess.run(command + [str(bundle)], capture_output=True, text=True)
            if result.returncode != 0:
                return False, (result.stdout + result.stderr).strip()
    return True, "opa check and opa test both passed"


def forge(rule: str, tests: pathlib.Path, ask, opa: str, attempts: int) -> str:
    transcript = [{
        "role": "user",
        "content": (
            f"Write a Rego policy for this rule:\n\n{rule}\n\n"
            f"It must satisfy these tests:\n\n```rego\n{tests.read_text()}```"
        ),
    }]

    for attempt in range(1, attempts + 1):
        reply = ask(transcript)
        policy = extract_rego(reply)
        ok, output = verify(opa, policy, tests)
        headline = next(iter(output.splitlines()), "no output from the toolchain")
        print(f"attempt {attempt}: {'PASS' if ok else 'FAIL'} - {headline}", file=sys.stderr)
        if ok:
            return policy
        transcript += [
            {"role": "assistant", "content": reply},
            {"role": "user", "content": f"The toolchain rejected that:\n\n{output}\n\nFix it."},
        ]

    raise RuntimeError(f"no policy survived {attempts} attempts")


def claude(model: str):
    import anthropic

    client = anthropic.Anthropic()

    def ask(transcript: list[dict]) -> str:
        response = client.messages.create(
            model=model,
            max_tokens=16000,
            system=[{
                "type": "text",
                "text": SYSTEM,
                "cache_control": {"type": "ephemeral"},
            }],
            thinking={"type": "adaptive"},
            messages=transcript,
        )
        return "".join(b.text for b in response.content if b.type == "text")

    return ask


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--rule", required=True, help="the policy, in one English sentence")
    parser.add_argument("--tests", required=True, type=pathlib.Path,
                        help="a _test.rego file you wrote by hand")
    parser.add_argument("--out", required=True, type=pathlib.Path,
                        help="where to write the policy, only if it passes")
    parser.add_argument("--model", default="claude-opus-5")
    parser.add_argument("--attempts", type=int, default=4)
    parser.add_argument("--opa", default="opa")
    args = parser.parse_args()

    if args.attempts < 1:
        print("policy-forge: --attempts must be at least 1", file=sys.stderr)
        return BROKEN
    if not args.tests.is_file():
        print(f"policy-forge: {args.tests} does not exist", file=sys.stderr)
        return BROKEN

    try:
        policy = forge(args.rule, args.tests, claude(args.model), args.opa, args.attempts)
    except RuntimeError as exc:
        print(f"policy-forge: {exc}; nothing written", file=sys.stderr)
        return REJECTED
    except ValueError as exc:
        print(f"policy-forge: {exc}", file=sys.stderr)
        return BROKEN

    args.out.write_text(policy)
    print(f"policy-forge: verified policy written to {args.out}", file=sys.stderr)
    return WRITTEN


if __name__ == "__main__":
    sys.exit(main())

export ANTHROPIC_API_KEY=...
python3 policy_forge.py \
  --rule "No security group may expose an administrative port to the public internet." \
  --tests policy/network_test.rego \
  --out policy/generated.rego

Here's what the loop guarantees:

1. Verification runs as a subprocess: the model is never asked whether its policy is correct. opa check and opa test decide, and their exit codes are the only evidence the loop accepts.

2. Failures go back as raw tool output, never summarised: compiler errors and test failures are the highest-signal feedback a model can receive, and paraphrasing throws away the part that helps.

3. Nothing reaches disk until it passes: forge either returns a verified policy or raises, so there's no path where an unverified policy lands in the repository just because the retry budget ran out.

I drove the loop with a scripted model so the result is reproducible without an API key. The three replies were a realistic v0-syntax policy, a realistic-but-wrong v1 policy, and the policy from Step 3:

attempt 1: FAIL - 2 errors occurred during loading:
attempt 2: FAIL - policy/network_test.rego:63:
attempt 3: PASS - opa check and opa test both passed

Attempt 1 was Rego v0, the deny[msg] { ... } form, which stopped parsing when OPA 1.0 shipped in January 2025. And it's overwhelmingly what public training data contains. opa check --strict rejected it before it reached a test.

Attempt 2 was valid Rego v1. It would have passed review from most engineers, and it still scored only 4 of 8 on the suite. This is because it compared ingress.from_port against admin_ports directly, ignored the range, and read only cidr_blocks. opa check had no complaint, because the code was perfectly well-formed.

A well-formed policy can still be the wrong policy, and the only thing in this loop that knows what you wanted is the test suite you wrote.

A repair loop isn't monotonic, because each attempt is a fresh generation conditioned on an error message, with nothing carrying forward what already worked, so attempt four can lose a property attempt three had. Nothing in this design detects that, because the only thing being checked is the test suite you wrote.

Cap the retries, keep the suite growing, and treat every generated policy as a pull request that somebody approves before it merges.

Step 8: Govern the Agent Itself

An AI agent is also an actor, and it calls tools, so every tool call becomes an authorisation decision that something has to make.

The industry converged on this quickly: Amazon Bedrock AgentCore Policy reached general availability in March 2026, evaluating agent tool calls at the gateway in Cedar. The common open-source pattern is an OPA sidecar in front of an MCP tool gateway.

Research is pushing the same boundary harder: a 2026 preprint from the University of Washington group behind Defects4J, Solver-Aided Verification of Policy Compliance in Tool-Augmented LLM Agents (Winston, Winston, and Just), compiles natural-language policies into SMT constraints and blocks non-compliant calls with the Z3 solver.

They share one claim: a policy in the system prompt isn't enforcement. Enforcement is an interceptor sitting in the call path that can return "no" and stop the call from happening.

Create agent/authz.rego:

# METADATA
# title: Agent tool-call authorisation
# description: |
#   Evaluated once per tool call, before the tool runs. The decision has
#   three values rather than two, because an agent worth deploying will
#   sometimes need to do something that a human, not the policy, should
#   approve.
package agent.authz

tool_grants := {
	"support": {"search_orders", "read_customer", "issue_refund"},
	"analytics": {"search_orders", "run_query"},
}

write_tools := {"issue_refund", "run_query"}

refund_ceiling_cents := 10000

# An unmapped role, an unknown tool or a malformed input all land here.
default decision := {"effect": "deny", "reasons": ["no matching grant"]}

decision := {"effect": effect_for(reasons), "reasons": reasons} if {
	count(granted) > 0
	reasons := escalations
}

granted contains role if {
	some role in input.agent.roles
	input.tool in object.get(tool_grants, role, set())
}

effect_for(reasons) := "allow" if count(reasons) == 0

effect_for(reasons) := "require_approval" if count(reasons) > 0

escalations contains reason if {
	input.tool in write_tools
	not input.session.human_in_loop
	reason := sprintf("%q writes state and the session is unattended", [input.tool])
}

# A refund with no readable amount cannot be checked against the ceiling,
# so it escalates. Silence here would clear the exact call an attacker
# would craft.
escalations contains reason if {
	input.tool == "issue_refund"
	not positive_amount
	reason := "refund amount is missing, unreadable, or not positive"
}

# A negative amount is a charge wearing a refund's name.
positive_amount if {
	amount := object.get(input, ["arguments", "amount_cents"], null)
	is_number(amount)
	amount > 0
}

escalations contains reason if {
	input.tool == "issue_refund"
	amount := object.get(input, ["arguments", "amount_cents"], null)
	is_number(amount)
	amount > refund_ceiling_cents
	reason := sprintf(
		"refund of %d cents exceeds the %d cent ceiling",
		[amount, refund_ceiling_cents],
	)
}

# Keyword matching is a coarse guard, and it is here to show the shape of an
# argument-level rule. Anything holding real data wants a SQL parser: this
# catches `DROP TABLE` and misses a statement that spells it another way.
destructive_sql := `(?i)\b(drop|truncate|delete|alter|grant|revoke)\b`

escalations contains reason if {
	input.tool == "run_query"
	regex.match(destructive_sql, object.get(input, ["arguments", "statement"], ""))
	reason := "statement contains a destructive SQL keyword"
}

The decision vocabulary is closed, and I'll state it plainly here:

Here's why the policy is shaped that way:

1. default decision is deny: an unrecognised tool, a role you forgot to map, or a malformed input all end up there. A policy that defaults to allow fails open on exactly the inputs nobody anticipated, which is the set an attacker picks from.

2. Three values: binary authorisation forces a choice between blocking useful work and permitting dangerous work, and the third value is what makes a high-autonomy agent tolerable.

3. Reasons come back as a set: every applicable reason is collected. When somebody gets an approval prompt at three in the morning, "refund of 250000 cents exceeds the 10000 cent ceiling" tells them what to do. "Policy violation" does not.

4. Arguments are inspected too: issue_refund is routine at £5 and serious at £2,500, so tool-name granularity is far too coarse for agents, given that the agent chooses the arguments.

Fifteen tests cover the decision table, including an agent with an empty role list, a refund with no amount at all, and a query that hides DROP behind a newline:

opa test agent -v

PASS: 15/15

Serve it and try a call:

opa run --server --addr localhost:8181 agent/

curl -s localhost:8181/v1/data/agent/authz/decision \
  -d '{"input":{"agent":{"roles":["support"]},"tool":"issue_refund",
       "arguments":{"amount_cents":250000},"session":{"human_in_loop":true}}}' | jq .result

{
  "effect": "require_approval",
  "reasons": [
    "refund of 250000 cents exceeds the 10000 cent ceiling"
  ]
}

The policy is inert until something refuses to proceed on its answer. That's the client:

import json
import urllib.request

OPA_URL = "http://localhost:8181/v1/data/agent/authz/decision"


class PolicyDenied(Exception):
    pass


class ApprovalRequired(Exception):
    pass


def authorize(agent, tool, arguments, session):
    payload = json.dumps({"input": {
        "agent": agent, "tool": tool,
        "arguments": arguments, "session": session,
    }}).encode()
    req = urllib.request.Request(
        OPA_URL, data=payload, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=2) as resp:
        body = json.load(resp)

    # OPA returns {} with a 200 when a query matches nothing. Fail closed.
    decision = body.get("result", {"effect": "deny", "reasons": ["policy unavailable"]})

    if decision["effect"] == "deny":
        raise PolicyDenied("; ".join(decision["reasons"]))
    if decision["effect"] == "require_approval":
        raise ApprovalRequired("; ".join(decision["reasons"]))
    return decision

search_orders    -> ALLOWED
issue_refund     -> NEEDS APPROVAL (refund of 250000 cents exceeds the 10000 cent ceiling)
delete_account   -> DENIED (no matching grant)

Note body.get("result", ...): OPA returns {} with a 200 status when a query matches nothing, so a bare body["result"] raises KeyError, and depending on how your agent framework handles exceptions that may fail open. Every layer defaults to deny, including the parsing.

Call authorize() from your framework's tool-execution hook, before the tool function runs. It's about fifteen lines, and it turns a system prompt's polite suggestions into an actual boundary.

Step 9: What I Got Wrong

Steps 2 and 3 show the finished policies. I reached for something simpler first (the version most tutorials stop at), and the gap between that and what you have just read is the most useful thing here.

The Tagging Policy Skipped the Worst Resources

My naïve in_scope rule ended with resource.change.after.tags, which reads as "only resources that have tags".

What it actually does is worse than that, because Terraform emits tags: null for a resource with no tags at all, and an undefined lookup makes the rule body fail, so the resource drops out of scope entirely.

The TerraGoat plan has two resources: the security group carries five git_* tags and no ownership tags, while the VPC carries nothing at all.

jq -r '.resource_changes[] | "\(.address): tags=\(.change.after.tags | type)"' plan.json

aws_security_group.web-node: tags=object
aws_vpc.web_vpc: tags=null

opa eval --data naive  --input plan.json --format pretty 'count(data.terraform.tags.deny)'
opa eval --data policy --input plan.json --format pretty 'count(data.terraform.tags.deny)'

3
6

The three it missed were all on the completely untagged resource, so the policy flagged the resource with some tags and silently cleared the one with none.

tags_of with its else := {} branch is the fix, and it's three lines.

The Network Policy Read One of Four Shapes

My naïve network policy read aws_security_group and cidr_blocks, which is what every tutorial shows, but Terraform has four ways to express the same ingress rule.

![Diagram titled "Four ways Terraform describes one ingress rule". Four boxes are shown. Top left, aws_security_group.ingress[].cidr_blocks, filled pale blue and labelled read. The other three are outlined in red with red hatching and labelled not read: aws_security_group.ingress[].ipv6_cidr_blocks, aws_vpc_security_group_ingress_rule.cidr_ipv4, and the deprecated aws_security_group_rule. A key states that solid blue fill means the policy looks here and red hatch means it does not.](https://cdn.hashnode.com/uploads/covers/5f3a74bfc4d5973f55c91c8c/ab61d36d-78bf-45f7-a073-e7013216e3e1.png align="center")

One rule, four encodings, and the naïve policy read only the top-left one.

I planned a second real configuration with three security groups that open SSH to the world using the shapes the policy didn't read. Both versions are in code/, so this reproduces:

opa eval --data naive --input plan-evasion.json \
  --format pretty 'data.terraform.network.deny'

[]

That is three publicly reachable SSH ports, zero violations, and a gate that would have printed PASS and exited 0.

A Test That Was Holding a Hole Open

The port range had a second problem, and my own test suite was protecting it. I had written a case called test_tolerates_null_ports, asserting that an ingress rule with protocol: "-1" produced no violations, on the reasoning that a comparison against a null port shouldn't crash the policy.

An all-protocols rule opens every port, and the AWS provider records it as from_port: 0 and to_port: 0. The the range check read it literally as the single port zero, so the most permissive rule in AWS scored clean.

The plan is in code/plan-all-protocols.json, and it's one resource:

jq -c '.resource_changes[] | select(.type=="aws_security_group")
       | .change.after.ingress[0] | {protocol,from_port,to_port,cidr_blocks}' \
   plan-all-protocols.json
opa eval --data naive --input plan-all-protocols.json \
   --format pretty 'data.terraform.network.deny'

{"protocol":"-1","from_port":0,"to_port":0,"cidr_blocks":["0.0.0.0/0"]}
[]

Every protocol and every port, open to the whole internet, and a test I wrote on purpose certified it as fine. The fix is covered_ports, which maps an all-protocols rule onto the full range and goes undefined for ports it can't read, with a second deny rule that reports the undefined case. That test is gone and four took its place:

opa eval --data policy --input plan-all-protocols.json --format pretty 'data.terraform.network.deny'

[
  "aws_security_group.wide_open: ingress rule exposes port 22 to 0.0.0.0/0",
  "aws_security_group.wide_open: ingress rule exposes port 3306 to 0.0.0.0/0",
  "aws_security_group.wide_open: ingress rule exposes port 3389 to 0.0.0.0/0",
  "aws_security_group.wide_open: ingress rule exposes port 5432 to 0.0.0.0/0"
]

Step 7 argues that the test suite is the only artifact in the loop that knows what you wanted. This is the cost of that property: a test that's wrong is a specification that's wrong, and nothing downstream of it will argue.

What the Numbers Actually Were

Both versions score 1.000 on the plan I designed the policy against. The gap only appears on the plan I did not.

Across both real plans, the naïve policies found 4 of the 9 violations present. They scored 1.000 on the TerraGoat security group, which is the case I had in mind while writing them, and 0.000 and 0.500 on the two cases I did not.

The Part That Genuinely Surprised Me

I assumed test coverage would have caught this, and it doesn't. I reconstructed the naïve tagging policy with the two tests I originally wrote for it:

opa test . --coverage --format json | jq '{overall: .coverage}'

{
  "overall": 100
}

The naïve policy scored 100% coverage, passed 2 of 2 tests, and cleared a resource that carried no tags at all.

Coverage measures which lines of a policy your tests executed, and says nothing at all about which shapes of input you failed to imagine. For policy code, this is the entire failure mode. Coverage is worth reporting, and the evidence that a policy actually works comes from running it against infrastructure you didn't write.

Limits of the Check

Here's what this gate still can't do, because its false clearances matter more than its catches.

1. Unknown Values Are Invisible

Terraform marks anything it can't resolve until apply time as unknown, which appears as null in the plan JSON alongside an after_unknown map. A policy reading change.after.some_field doesn't fire when that field is unknown.

This bites hardest on cross-resource rules, where "every bucket has a public access block" is genuinely hard at plan time, because the block references a bucket ID that's usually (known after apply).

2. It Sees the Plan, and Only the Plan

Anything applied outside the pipeline, changed in a console, or drifted since creation stays invisible to it. Plan-time checks and admission-time checks have different blind spots, and both leave work for a periodic scan of deployed state.

3. Coverage of Controls Isn't Measurable from Inside

A hundred green checks say nothing about the rules nobody wrote. Keep the mapping from your control requirements to your policy files somewhere explicit and audit it on a schedule, because the gate can't tell you what it was never asked.

4. The CIDR List is an Exact Match

public_cidrs holds 0.0.0.0/0 and ::/0 and nothing else, so a rule opening 0.0.0.0/1 reaches half the internet and passes. Widening it means deciding which prefix lengths count as public and carving out RFC 1918 space, and that decision belongs to your organisation.

5. Four Shapes is What I Found

The exposures set covers the four encodings I went looking for, and AWS offers more. Security group references (security_groups), prefix lists, and self rules are all ways to reach a port that this policy doesn't model, and it never sees them. I would expect a fifth shape to turn up the first time this runs against a large estate.

6. Regulatory Dates Move

If you're building toward the EU AI Act, the Digital Omnibus published in July 2026 pushed Annex III high-risk obligations from 2 August 2026 to 2 December 2027, and Annex I obligations to 2 August 2028, while the Article 50 transparency duties kept their original 2 August 2026 date.

Encode the controls, and look the dates up in the official timeline every time you need one, including when a blog post from last quarter tells you otherwise (this one included).

Conclusion

Models produce infrastructure code that parses almost every time, is secure about 56% of the time, and arrives faster than anybody can read it. Manual review stopped being a real control somewhere in that gap. The rules were always meant to be executable, and the volume is what finally forced the issue.

In this tutorial, you:

• Pulled a real vulnerable security group from TerraGoat at commit 729f8da and planned it with Terraform 1.14.2.

• Wrote 20 policy tests covering four Terraform encodings of one ingress rule, and got them to pass on OPA 1.20.2.

• Caught 7 real violations in the TerraGoat plan, while correctly ignoring port 80.

• Built a gate with a four-verdict contract that exits 2 when it examined nothing.

• Watched the naïve version find 4 of 9 violations while reporting 100% test coverage, and hardened it to find 9 of 9.

• Wired a generation loop where opa check and your own tests decide what reaches disk.

• Authorised agent tool calls from the same engine, with 15 tests and a default of deny.

My policies handled the cases I wrote them for and missed two I hadn't imagined, and every signal available to me (tests passing, coverage at 100%, and a clean opa check) agreed they were fine. The one thing that disagreed was infrastructure somebody else had written. Point your policies at code you didn't write, early, and keep the failures.

All the code, the policies, the plan JSON, and the scripts that build the figures are in the code/ directory alongside this handbook. The figures regenerate with python3 build/make_images.py and python3 build/make_terminals.py. The terminal screenshots re-run their commands at build time, so they can't drift from the truth.

Third-party Actions are dependencies. A floating tag behaves like an unpinned package dependency. If the reference changes, the workflow can execute different code without any change to your repository.

This article shows you how to prevent poisoned continuous integration (CI) dependencies by pinning Actions to full commit SHAs, validating those pins against release tags, restricting permitted actions, and automating reviewed updates with Dependabot.

Who this is for: Platform and DevSecOps engineers securing GitHub Actions workflows.

Prerequisites:

• Repository admin access

• Workflows using uses: org/action@ref syntax

The Quick Reference:

Here's what you'll learn how to do here:

• Pin every uses: reference to a full commit SHA, not a moving tag such as @v4.

• Validate that each recorded SHA matches the release you reviewed before approving it.

• Maintain an allowlist of permitted actions at the organization or repository level.

• Enable Dependabot for GitHub Actions version bumps with review.

• Prefer official or verified creators. Mirror critical actions internally if needed.

• Verify pins in pull request checks before merge.

Table of Contents

• The Quick Reference

• Why Floating Tags Fail

• Pin Actions to Commit SHAs

• Validate the Commit SHAs

• Enable Dependabot for Actions

• Allowlist Actions

• Enforce Pins in Pull Requests

• Vet Third-Party Actions

• How to Verify This Works

• When This Breaks Down

• Conclusion

• References

Why Floating Tags Fail

GitHub Actions lets you reference a dependency with a branch name, version tag, or commit SHA. These references don't provide the same level of stability. A branch can change at any time, and a version tag can be moved to a different commit after you review it. A commit SHA identifies one specific revision.

The table below compares the common reference styles and shows why a moving tag creates a supply chain risk:

The first three references fail because the name doesn't permanently identify the code that GitHub will execute. A full SHA solves that specific problem by making the workflow change only when someone changes the reference in the repository.

Key idea: CI pipelines deserve the same dependency discipline as application code.

Pin Actions to Commit SHAs

Pinning an Action means replacing its branch or version tag with the full SHA of the commit you reviewed. GitHub then checks out that exact revision, even if the publisher later moves the original tag.

This first example is bad because both references use mutable version tags. The workflow may run different Action code later without a corresponding change in your repository:

- uses: actions/checkout@v4
- uses: actions/setup-node@v4

This second example is good because each reference uses a full 40-character commit SHA. The comments preserve the readable release versions for maintainers, but GitHub uses the SHA rather than the comments:

- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/setup-node@4992456781334f2795ae9dd169f5041797d85689 # v4.4.0

Find the SHA on the action's GitHub release page or:

gh api repos/actions/checkout/git/ref/tags/v4.2.2 --jq '.object.sha'

Use the exact release tag you reviewed. Add a comment with the human-readable version for maintainability. The comment helps a reviewer, but it's not part of the security control. GitHub executes the commit identified by the SHA.

Don't use a shortened SHA. A full 40-character SHA makes accidental collisions and ambiguous reviews less likely.

Validate the Commit SHAs

A full SHA protects the workflow from a tag moving after you merge it, but you still need to validate the SHA before approving it. The release tag and commit should be reviewed together. This check compares the commit behind the reviewed tag with the commit recorded in the workflow:

#!/usr/bin/env bash
set -euo pipefail

repository="actions/checkout"
release_tag="v4.2.2"
expected_sha="11bd71901bbe5b1630ceea73d27597364c9af683"

actual_sha="$({
  git ls-remote "https://github.com/${repository}.git" \
    "refs/tags/${release_tag}" "refs/tags/${release_tag}^{}"
} | awk -v tag="refs/tags/${release_tag}" '
  $2 == tag "^{}" { print $1; found = 1 }
  $2 == tag { fallback = $1 }
  END { if (!found) print fallback }
')"

if [[ "${actual_sha}" != "${expected_sha}" ]]; then
  printf 'SHA mismatch for %s %s\n' "${repository}" "${release_tag}" >&2
  printf 'Expected: %s\nActual:   %s\n' "${expected_sha}" "${actual_sha}" >&2
  exit 1
fi

printf 'Validated %s@%s\n' "${repository}" "${expected_sha}"

This is release-reference validation, not a claim that a repository is trustworthy. Read the action source, review its permissions, and record why you approved the dependency. For higher-assurance environments, mirror critical actions internally and validate the mirrored artifact through your normal repository controls.

Enable Dependabot for Actions

Dependabot is GitHub's automated dependency-update service. For GitHub Actions, it checks the workflow references for newer releases and opens pull requests that update them. This gives you a reviewable place to inspect and approve an Action update instead of changing pins manually or using floating tags.

Create .github/dependabot.yml:

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
    groups:
      actions:
        patterns:
          - "*"

Dependabot opens pull requests when pinned SHAs have newer releases. Review and merge like application dependencies.

Don't configure Dependabot to merge these updates automatically until you have a review process for action changes. An update can contain new permissions, changed scripts, or a new transitive dependency.

For each update pull request, compare the old and new commit, inspect the action's release notes, and review changes to action.yml JavaScript bundles, shell scripts, and workflow permissions. Confirm that the new commit belongs to the release you intended to adopt. Run the workflow in a test repository or environment when the action handles deployment, publishing, credentials, or other high-impact operations.

The goal isn't to reject every update. The goal is to make the trust decision visible and repeatable. A reviewer should be able to answer three questions before merging: what code changed, why is the new version needed, and what permissions can that code use?

Allowlist Actions

An Action allowlist is a repository or organization policy that limits which publishers and repositories workflows are allowed to call. It reduces the chance that a contributor introduces an unknown or unreviewed Action into a trusted pipeline.

Organization administrators can configure it at Settings → Actions → Policies → Allow specified actions. Choose the narrowest policy that matches your workflows, then add only the Action repositories your teams have reviewed.

Example allowlist patterns:

actions/checkout@*
actions/setup-node@*
actions/cache@*
docker/*
my-org/*

With this policy, Actions that don't match the approved patterns are denied. Repositories under the organization inherit the policy, subject to the organization's GitHub plan and repository settings.

For a single repo without an org policy, use Settings → Actions → General → Allow actions created by GitHub, and select non-GitHub actions and restrict to verified creators only.

An allowlist limits which action identities may run. It doesn't replace SHA pinning. A permitted action should still be referenced by a reviewed full SHA in every workflow.

Enforce Pins in Pull Requests

Run a workflow linter in CI and add a repository check that fails when a workflow introduces a non-SHA reference. Pin the checker itself before using it in a protected workflow. Replace the placeholder below with the full commit SHA you verified for the actionlint release you selected:

name: actionlint

on:
  pull_request:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
      - uses: rhysd/actionlint@03d0035246f3e81f36aed592ffb4bebf33a03106 # v1.7.7
        with:
          args: -color

Resolve the release tag, review the commit, and record the full 40-character SHA just as you did for actions/checkout.

You can also use a small check to catch common floating references. Treat this as a backup signal, not a YAML parser or a complete supply chain policy:

      - name: Reject floating action tags
        run: |
          if grep -RInE '^[[:space:]]*-?[[:space:]]*uses:[[:space:]]*[^#]+@(main|master|v[0-9]+)([[:space:]]|$)' .github/workflows/; then
            echo "Floating action refs found. Pin to full SHA."
            exit 1
          fi

The check should run on pull requests and on protected branches. A pull request can pass this test and still contain a malicious commit, so combine it with code review, the allowlist, and the SHA validation process.

Vet Third-Party Actions

Before adding a community action to the allowlist:

1. Read the action's action.yml and entry script.

2. Check star history, maintainer reputation, and open security issues.

3. Pin SHA and fork to my-org/action-name if the action is critical but externally maintained.

Also inspect the permissions available to the workflow. An action that can read repository secrets or write releases deserves more scrutiny than an action that only formats a file. Set the workflow's top-level permissions to the minimum required, then grant additional permissions to individual jobs only when needed.

How to Verify This Works

1. Run the SHA validation script for every action you approve and expect all comparisons to pass.

2. Open a test pull request that changes an action to@main, and confirm the CI gate fails.

3. Add a trailing comment after a floating tag and confirm your parser or linter still rejects it.

4. Confirm organization policy blocks a disallowed action in a test workflow.

5. Confirm Dependabot opens an action update pull request and that the change receives normal review.

6. Review the workflow run permissions and verify that the action can't access credentials it doesn't need.

When This Breaks Down

1. Emergency patches: SHA pinning slows hotfixes. Dependabot plus on-call review is safer than temporarily using a floating tag.

2. Composite actions in private repos: pin internal actions the same way as public ones.

3. Reusable workflows: pin the reusable workflow ref to SHA as well as steps inside it.

4. Annotated tags and mirrors: resolve annotated tags to their commit before recording a SHA, and validate internal mirrors through your own change-control process.

5. False sense of safety: pinning without reading code still trusts the maintainer at pin time. Combine pinning with allowlists, least-privilege permissions, and review.

Conclusion

In this tutorial, you learned how to prevent poisoned CI dependencies by pinning GitHub Actions to reviewed commit SHAs, validating those references, allowlisting trusted actions, enabling Dependabot updates, and enforcing pins in pull request checks.

The result is a layered control: a reviewed immutable reference, a restricted set of permitted actions, automated update proposals, and a pull request check that catches regressions before merge.

References

• GitHub Actions, Security hardening

• GitHub, Dependabot version updates for Actions

• StepSecurity, Pin GitHub Actions

• OpenSSF, Source compromise defenses

A lightweight way to shift security left is to run static application security testing (SAST) locally through Git pre-commit hooks.

In this guide, you'll install the DevSkim CLI, wire it into pre-commit so it lints staged files, add a dedicated secret scanner alongside it, verify the setup with a deliberate failure, and enforce the same checks in CI.

What We'll Cover:

• Why Pre-Commit Security Scanning?

• What Is DevSkim?

• Prerequisites

• Create a Pre-Commit Configuration

• Install the Git Hook

• Verify the Setup With a Deliberate Failure

• Example: Catching a Hardcoded Secret

• Keep Hooks Fast and Focused

• Handling False Positives

• Add Security Checks to CI Too

• Practical Rollout Tips

• Final Thoughts

Why Pre-Commit Security Scanning?

Pre-commit hooks run automatically before Git creates a commit. They're useful for catching hardcoded passwords, tokens, and connection strings, insecure cryptographic patterns, weak input validation, risky file or process handling, and other language-specific security anti-patterns.

The goal isn't to replace CI security scanning or manual reviews. It's to give developers fast, actionable feedback at the earliest practical point.

A healthy model looks like this:

Developer workstation
  -> Pre-commit security checks
  -> Pull request review
  -> CI/CD SAST and dependency scanning
  -> Runtime monitoring and vulnerability management

What Is DevSkim?

DevSkim is a security linter created by Microsoft. It ships as IDE extensions and a cross-platform CLI, and it flags insecure coding patterns using a configurable rule set.

It works well as a developer-facing guardrail because it scans files quickly and explains why a pattern may be risky. Its rules cover dangerous API usage, weak cryptography, insecure deserialization, weak TLS or certificate validation, and potential command injection risks.

One distinction matters before you rely on it: DevSkim is a security linter, not a secret scanner. It bundles a few generic credential rules, but those are regex patterns aimed at long, token-like values.

Teams typically pair DevSkim with a dedicated secret-scanning tool such as Gitleaks, detect-secrets, or TruffleHog. Gitleaks and detect-secrets both publish official pre-commit hooks, so either slots into this workflow easily. This guide uses Gitleaks.

As with any security scanner, findings need context. A warning isn't automatically a vulnerability, but it's worth reviewing before the code moves downstream.

Prerequisites

You'll need Git, Python 3, and the .NET SDK, which DevSkim's CLI is distributed through.

Install pre-commit:

pip install pre-commit

Install the DevSkim CLI as a .NET global tool:

dotnet tool install --global Microsoft.CST.DevSkim.CLI

If you prefer not to install the .NET SDK, platform-specific binaries are available on the DevSkim releases page.

Confirm both installations in a new terminal, so it picks up the updated PATH:

pre-commit --version
devskim --version

For inline feedback while you type, also install the DevSkim VS Code extension. The CLI is what the Git hook uses.

Create a Pre-Commit Configuration

DevSkim's CLI takes a single source path after -I, while pre-commit appends every staged filename to the command it runs. Passing devskim analyze directly therefore breaks as soon as a commit touches more than one file.

The simplest fix is a small wrapper that invokes DevSkim once per filename. Create scripts/run-devskim.py in your repository:

import subprocess
import sys

exit_code = 0

for filename in sys.argv[1:]:
    result = subprocess.run(["devskim", "analyze", "-I", filename])
    exit_code = exit_code or result.returncode

sys.exit(exit_code)

Now create .pre-commit-config.yaml at the root of your repository:

repos:
  - repo: local
    hooks:
      - id: devskim
        name: DevSkim security lint
        entry: python scripts/run-devskim.py
        language: system
        types_or: [python, javascript, typescript, json, yaml]
        pass_filenames: true

  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1
    hooks:
      - id: gitleaks

The first hook runs DevSkim against the staged files that match the listed types. The second adds Gitleaks for the secret detection DevSkim isn't designed to handle.

If you would rather keep things simpler and scan the whole repository on every commit, drop the wrapper and use a single fixed path instead:

      - id: devskim
        name: DevSkim security lint
        entry: devskim analyze -I .
        language: system
        pass_filenames: false

That version is easier to reason about, but it gets slower as the repository grows. Staged-file scanning is the better default once a project has real history.

Install the Git Hook

Run this once from the repository root:

pre-commit install

Now, every git commit triggers the configured checks.

You can run the hooks manually against the entire codebase:

pre-commit run --all-files

That's particularly useful when introducing the tool to an existing project. Expect initial findings. Older repositories often contain patterns that predate current security standards.

Verify the Setup With a Deliberate Failure

Before trusting the hook, you'll want to confirm that it actually blocks something. Create a file called insecure-example.js with a weak hashing call:

const crypto = require("crypto");

const hash = crypto.createHash("md5").update("password").digest("hex");

Stage it and attempt a commit:

git add insecure-example.js
git commit -m "Test security hooks"

The commit should be rejected. pre-commit prints a status line per hook, followed by the failing hook's ID, its exit code, and the scanner's own findings:

DevSkim security lint....................................................Failed
- hook id: devskim
- exit code: 1

[DevSkim output listing the weak hash finding appears here]

DevSkim's finding text includes a rule ID, a severity label, and the file and line, and the exact formatting varies by version and ruleset. What matters is the non-zero exit code and the blocked commit.

If the hook passes unexpectedly, run the scan directly to isolate the problem:

devskim analyze -I insecure-example.js

If that reports the finding but the hook does not, the issue is in your .pre-commit-config.yaml rather than in DevSkim.

Example: Catching a Hardcoded Secret

Secrets are where the linter and the secret scanner divide. DevSkim's generic credential rules are regex-based and target long, token-like values, so a readable placeholder string will usually slip past them. A value shaped like a real key is a more realistic test:

const apiKey = "4f2a9c1e7b6d3a8f0c5e9b2d7a41c6";

Stage the change and run the hooks:

git add config.js
pre-commit run

This is the case the secret scanner is built for. Gitleaks applies entropy and pattern checks tuned for credentials, so it's the hook you should expect to block a value like this, and its status line will report a failure and a non-zero exit code. Treat DevSkim's result here as a bonus rather than the control you rely on, which is exactly why the two tools sit side by side.

The fix is to move the value out of source control:

const apiKey = process.env.PAYMENT_API_KEY;

The real value should be stored in an approved secret manager, such as Azure Key Vault, GitHub Actions Secrets, or another managed platform.

This small shift matters. Removing a secret after it has been committed is harder, because Git history, clones, CI logs, and deployment systems may already contain it.

Once you have confirmed both hooks behave as expected, delete the test files. Never use a real credential to test a scanner.

Keep Hooks Fast and Focused

Developers will bypass slow hooks. A good local security scan should complete quickly and focus on high-confidence findings.

Start by scanning only staged or changed files locally, then reserve broader scans for CI. Block high-severity findings such as exposed secrets and critical insecure API usage first. Lower-confidence findings can be reported for review while the team tunes noisy rules and documents justified suppressions.

Handling False Positives

False positives are expected in static analysis. The wrong response is disabling the scanner entirely. Verify whether a finding is exploitable and fix it when it represents real risk. When an exception is justified, use a narrowly scoped suppression, record the reason, and review it periodically.

Avoid broad exclusions such as skipping entire directories unless there's a clear technical reason. Broad exclusions tend to become blind spots.

Add Security Checks to CI Too

Pre-commit hooks improve developer feedback, but they aren't enforcement. You can skip hooks with:

git commit --no-verify

That's why the same or equivalent security checks should run in CI. Here's a GitHub Actions workflow that runs both tools on every pull request. Save it as .github/workflows/security.yml:

name: Security Checks

on:
  pull_request:
  push:
    branches: [main]

jobs:
  scan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: "8.0.x"

      - name: Install DevSkim
        run: dotnet tool install --global Microsoft.CST.DevSkim.CLI

      - name: Run DevSkim
        run: devskim analyze -I . -f sarif -O devskim.sarif

      - name: Upload DevSkim results
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: devskim.sarif

      - name: Run Gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

A few details here are worth calling out. fetch-depth: 0 gives Gitleaks the full history it needs to scan past commits rather than just the tip. The SARIF upload sends DevSkim findings to the repository's Security tab, so results are reviewable in GitHub instead of buried in build logs. Microsoft also publishes a DevSkim Action if you prefer not to manage the CLI install yourself.

From there, use pre-commit for fast feedback on changed code, pull request pipelines for SAST, secret scanning, and dependency checks, main-branch builds for full scans and reporting, and release pipelines for gates on critical findings. This layered approach balances developer experience with governance.

Practical Rollout Tips

Start with one or two repositories and baseline existing findings before enforcing new rules. Initially, block only high-confidence, high-impact issues, then share examples of findings and fixes with developers. Track recurring patterns to guide secure coding training, and measure adoption and time-to-remediation instead of focusing only on finding counts.

The aim is to help developers make secure choices by default, not create another tool that gets ignored.

Final Thoughts

Security tooling is most valuable when it appears at the moment a developer can act on it. Pre-commit hooks make secure coding feedback immediate, while CI provides the enforcement and broader coverage needed for production systems.

DevSkim is a useful lightweight entry point for teams building a shift-left security practice, and pairing it with a dedicated secret scanner covers the gap a linter is not built to fill. Start small, tune the rules, keep scans fast, and support the workflow with CI validation and secure coding guidance.

The best security control is often the one that prevents the issue from being committed in the first place.

That world is gone. Today, a single attack can steal your passwords, lock up your photos, watch what you type, and hide inside software you trust.

The bigger problem is volume. The AV-TEST Institute records over 450,000 new malicious programs every single day. No security team can review that many files by hand. So the job has moved to machines, and antivirus software's decision-making has changed with it.

In this article, we'll look at how signature scanning worked and why it started to fail. We'll also cover what machine learning adds, how behaviour tracking catches ransomware while it runs, how cloud threat data turns every device into a sensor, and where AI still gets things wrong.

What We'll Cover:

• Signature Scanning Worked Until Malware Learned to Change

• Why Today's Malware is So Hard to Spot

• What Machine Learning Adds to the Picture

• Watching What Software Does, Not Just What it Looks Like

• Stopping Ransomware While it's Still Running

• The Cloud Turns Every Device into a Sensor

• Catching the Attack Before Malware Ever Lands

• Where AI Still Gets it Wrong

• Layers Beat Any Single Trick

• What This Means For You

Signature Scanning Worked Until Malware Learned to Change

Early antivirus software used signatures. A signature is a small pattern taken from a known bad file, a bit like a fingerprint.

Researchers found a piece of malware, pulled out its pattern, and added it to a database. Your antivirus downloaded that database and compared it against every file on your disk. A match meant the file was blocked.

This was fast, cheap, and easy to trust. It also had one big weakness.

Attackers figured out that they only had to change the file a little. A new name, some padding, a different way of packing the code, and the fingerprint no longer matched. The old signature was suddenly useless.

That created a gap. A new threat would spread for hours or days before anyone wrote a signature for it. Attackers now build thousands of tiny variations of the same program on purpose, making it a losing race to write one signature per version.

Signatures are still worth keeping. They catch known threats in milliseconds. They just can't be the only thing standing between you and an attack.

Why Today's Malware is So Hard to Spot

Modern malware tries hard to look boring.

It may sit quiet for days before doing anything. It may arrive as a harmless-looking script and download the real payload later. Some of it never writes a file to disk at all, which is why Microsoft groups these as fileless threats.

Worse, plenty of attacks use tools that are already on your computer. An attacker who gets access can run commands through PowerShell, a normal Windows administration tool. The LOLBAS project catalogues hundreds of trusted Windows programs that can be abused this way.

Nothing malicious is being installed here. A trusted tool is simply being used for the wrong reason. A file scanner has almost nothing to grab onto.

So the question security software asks has changed. It's no longer just "have I seen this file before?" It's "what is this program actually doing?"

What Machine Learning Adds to the Picture

Machine learning doesn't give software a sixth sense. It gives it a way to make a judgment call from evidence.

A model is trained on huge sets of files, both safe and harmful. Over time, it learns which traits tend to show up in each group. It learns about file structure, how the code is packed, which system calls it makes, what it talks to over the network, and how it interacts with other programs.

When something new arrives, the model weighs those traits and estimates the risk. It never saw this exact file, but it has seen the shape of the problem before.

This matters most for variants. Attackers often rewrite the surface of their code and keep the guts the same. Signatures miss the family resemblance. A model trained on behaviour and structure often catches it.

Watching What Software Does, Not Just What it Looks Like

The biggest shift in antivirus protection is the move from scanning files to watching actions.

Picture an unknown program starting up on a laptop. Within seconds, it opens hundreds of documents, rewrites each one, changes their file extensions, deletes the recovery copies, and calls out to a server nobody recognises.

There may be no signature for it anywhere. But that pattern is ransomware, and it's unmistakable.

Microsoft describes this approach in its documentation on behavioral blocking and containment, where machine learning models score a chain of actions rather than a single file. Individual steps can look innocent. Read together, they tell a story.

The advantage is timing. The software doesn't need to have met this exact malware before. It only needs to notice the shape of the attack early enough to cut it off.

Stopping Ransomware While it's Still Running

Ransomware shows why this matters more than any other threat type.

Known ransomware families get caught by signatures without trouble. But a brand new variant will often slip straight past them. That's the whole point of building a new variant.

Behavior monitoring gives you a second chance. The software watches for rapid file changes across many folders, attempts to delete backups or shadow copies, and processes trying to shut off security tools. Federal guidance on the CISA StopRansomware hub leans on the same signals, alongside offline backups you can actually restore from.

Once enough warning signs stack up, the software can kill the process or pull the device off the network. Even a partial save matters here. Stopping an attack after fifty encrypted files is a very different day than stopping it after fifty thousand.

Products aimed at everyday users are moving the same way. The next-gen antivirus from NordVPN blocks malicious downloads and scam pages before they reach the device, which shows how consumer tools have widened past plain file matching.

The Cloud Turns Every Device into a Sensor

Everything so far has been about what security software examines: files first, then behaviour. The other big change is where that examination happens.

Signature scanning ran start to finish on your machine. Your antivirus pulled down a database, compared files against it locally, and that was the entire decision.

Modern protection splits the work instead. Cheap checks stay on the device so they're instant, and the harder calls get handed to the vendor's cloud, which can see far more than any one laptop ever will.

Here's the actual sequence: First, the agent installed on your device records security-relevant events: process launches, parent-child process relationships, registry edits, outbound connections, file hashes. When it meets something it can't classify on its own, it sends metadata about it (typically the hash, the file's structural traits, and the surrounding activity, rather than the whole document) to the vendor's backend over an encrypted channel.

Microsoft documents this handoff for Defender in its notes on cloud protection, where the local client queries the cloud service for a verdict and briefly holds the file while it waits for an answer.

The backend does two jobs at once. Automated systems compare the submission against what millions of other devices have reported and score it with models far too large to ship to a laptop.

If the picture is clear, a verdict comes back in under a second with no human involved. If it isn't, the case escalates to the vendor's threat research and security operations teams, the analysts employed specifically to hunt for clusters like this.

That's what makes the "few thousand devices" signal worth something. If the same unfamiliar binary, or the same odd process chain, appears across thousands of unrelated organisations inside an hour, no single one of them would notice. The backend sees the cluster immediately and flags it for a human to open up. Analysts pull samples, detonate them in a sandbox, confirm what the code does, and write a detection for it.

Then the loop closes. Confirmed cases become labelled training data, which is exactly what the next generation of models needs. The output goes out in two speeds: new indicators like hashes, domains and behavioural rules reach every protected device within minutes, while retrained models follow on a slower cycle of days or weeks.

Either way, the person targeted next is protected by what your device reported, and nobody had to wait for the next big database download.

Catching the Attack Before Malware Ever Lands

Not every threat arrives as a program. Most start with a message.

Phishing is still the front door. The APWG counted more than one million phishing attacks in a single quarter, and the fake pages are getting harder to eyeball. Attackers copy bank branding, login screens, and delivery notices closely enough to fool careful people.

AI helps by checking the things humans skip: how old the domain is, whether the link redirects somewhere odd, and whether the page matches a known scam kit.

Blocking a page like that stops the attack one step earlier. No download, no file to scan, and no cleanup.

Where AI Still Gets it Wrong

AI brings real problems along with the benefits, and it's worth being clear about them.

False positives are the everyday one. Unusual isn't the same as malicious, and software that blocks a legitimate app because it looked odd trains people to switch protection off.

Speed is another. All this analysis has to happen without making the machine feel slow.

Then there's the arms race. Attackers study these models too. NIST's report on adversarial machine learning lays out how models get poisoned during training or fooled at the moment of decision. A model is a target, not a fortress.

Privacy deserves a mention as well. Cloud analysis means some information about your files and connections leaves your device. It's fair to ask any vendor what they collect and how long they keep it.

Layers Beat Any Single Trick

None of this replaces what came before. The strongest setups stack methods on purpose.

Signatures handle known malware instantly. Reputation checks block bad sites and untrusted programs. Behaviour monitoring catches suspicious activity as it happens. Machine learning fills the gap for threats nobody has named yet.

The MITRE ATT&CK framework is useful here because it maps out attacker techniques so teams can see which layer covers which step.

Layers also give the software context. A file with a clean history that starts acting strangely deserves a closer look. A file already known to be malware doesn't need any analysis at all.

What This Means For You

Antivirus software has moved a long way from matching files against a list.

Signatures still earn their place, but they only answer one question. AI and behaviour monitoring answer a better one: what is this software doing right now, and does it make sense?

The practical takeaway is short. Good protection today is less about recognizing bad files and more about noticing bad behavior quickly. If you're choosing security software, ask whether it watches activity or only scans files, and check whether it blocks dangerous sites and downloads before they arrive.

Attacks keep getting faster and more automated. The defenses have to work the same way, and that's the real reason AI ended up at the centre of malware detection.

Hope you enjoyed this article. You can connect with me on LinkedIn.

That's exciting. It's also a responsibility.

Lovable can help you move quickly, experiment with ideas, and create useful software. But speed shouldn't replace careful thinking. A generated app can contain security problems, confusing user experiences, inaccurate information, or code that works in a demonstration but falls apart in real life.

In this guide, you'll learn practical ways to use Lovable while keeping security, privacy, accessibility, and user safety in mind. We'll cover how to write clearer prompts, protect sensitive information, test authentication and authorization, validate user input, work with realistic test data, review AI-generated code, and decide when an application is ready to share.

By the end, you'll have a simple workflow for building with Lovable more responsibly without giving up the speed and creativity that make AI-powered development useful.

What We'll Cover:

• What Is Lovable?

• Why Responsible Use Matters

• Start With a Clear and Straightforward Idea

• Do Not Enter Sensitive Information Unnecessarily

• Protect Secrets With Environment Variables

• Understand What Your App Does

• Build Security Into Your Prompts

• Test Authentication and Authorization Separately

• Validate All User Input

• Be Careful With Generated Dependencies

• Design for Accessibility

• Avoid Dark Patterns

• Handle Errors Effectively

• Be Honest About AI-Generated Features

• Protect Personal Data

• Respect Copyright and Ownership

• Test With Realistic but Fake Data

• Test Before You Share the App

• Ask Lovable to Review Its Own Work

• Learn From the Generated Code

• Use Lovable for Prototyping Without Pretending It Is Production-Ready

• Create a Simple Responsible Development Workflow

• A Responsible Prompt Template

• The Golden Rule of AI App Building

• Final Thoughts

What Is Lovable?

Lovable is an AI-powered app-building platform that allows you to describe an application using natural language. Instead of writing every line of code manually, you can explain what you want and let the tool generate parts of the interface, functionality, and application structure.

For example, you might write:

Create a task manager with user accounts, a dashboard, task categories, due dates, and a button for marking tasks as complete.

Lovable may then generate a starting point that you can review, test, and improve.

The key phrase is “starting point.” AI-generated software isn't automatically finished software. Think of Lovable as a very fast coding partner that needs clear instructions, thoughtful reviews, and occasional reminders not to put a banana-shaped button in the middle of your login form.

Why Responsible Use Matters

AI app builders make software development more accessible, but accessibility comes with responsibility. When you create an app, you're making decisions that can affect real people.

Your application might collect names, email addresses, messages, payment details, health information, or location data. It might make recommendations, display important information, or control access to something valuable.

A small mistake can create large problems.

Responsible development helps you:

• Protect user information

• Reduce security risks

• Avoid misleading users

• Create accessible experiences

• Respect copyright and ownership

• Test your application before sharing it

• Understand the code and services your app uses

• Make decisions that are fair and explainable

You don't need to be a security expert to use Lovable responsibly. But you do need to slow down long enough to ask good questions.

Start With a Clear and Straightforward Idea

Before asking Lovable to build an app, describe the problem you want to solve.

A vague prompt such as this:

Build a cool productivity app.

leaves a lot of room for confusion.

A clearer prompt might look like this:

Build a simple productivity app for students. Users should be able to create tasks, assign a due date, mark tasks as complete, and filter tasks by status. Use plain language, a calm color palette, and a layout that works well on phones and desktop screens.

A strong prompt usually explains:

• Who the app is for

• What problem it solves

• What users should be able to do

• What information the app stores

• What the interface should feel like

• What the app should not do

• What platform or screen sizes it should support

Clear instructions make it easier to review the result. They also reduce the chance that the AI invents unnecessary features that make your project more complicated than a group project with twelve shared spreadsheets.

Don't Enter Sensitive Information Unnecessarily

When working with an AI-powered development tool, avoid including sensitive information in prompts unless it's genuinely necessary and handled through an appropriate process.

Don't paste in:

• Passwords

• Private API keys

• Authentication tokens

• Credit card numbers

• Personal identification numbers

• Private customer records

• Confidential business documents

• Unreleased product details

• Medical records

• Private conversations

Use placeholders instead:

Use a placeholder for the payment provider API key.

Or:

Connect to an email service using an environment variable named EMAIL\_API\_KEY. Do not hardcode the key in the source code.

A placeholder keeps your project easier to share, review, and maintain. It also prevents the classic “I accidentally published a secret to the internet” plot twist.

Protect Secrets With Environment Variables

Secrets shouldn't be placed directly in frontend code or committed to a public repository.

A safer pattern is to use environment variables:

const apiKey = process.env.API\_KEY;

For a client-side application, be especially careful. Environment variables used in browser code may be visible to users. A secret that must remain private should usually be used on a secure server or through a protected backend service.

Never use this pattern:

const apiKey = "your-real-secret-key";

Use a placeholder during development:

const apiKey = process.env.API\_KEY || "";

Then configure the real value through the appropriate secret-management system for your hosting platform.

Before deploying, search your project for common secret patterns such as:

API\_KEYSECRETTOKENPASSWORDPRIVATE\_KEY

Finding a suspicious value doesn't always mean it's a secret, but it's worth checking.

Understand What Your App Does

Don't publish an application that you can't explain at a basic level.

You should know:

• What data the app collects

• Where that data is stored

• Which external services receive the data

• Who can view or modify the data

• How users delete their accounts or information

• Which parts of the app require authentication

• What happens when a request fails

• What happens when a user enters unexpected input

You don't need to understand every line immediately. But you should understand the major building blocks.

If Lovable generates code that you don't understand, ask it to explain a specific section:

Explain how user authentication works in this project. Identify where sessions are created, how access is checked, and what could go wrong if authentication is misconfigured.

You can also ask:

List all external services used by this application and explain what data each service receives.

Explanations are useful, but they aren't proof that the code is safe. Treat them as a map, not a magical safety certificate.

Build Security Into Your Prompts

Security should be part of the original request, not an emergency patch added after someone discovers that every user can view every account.

Include security requirements in your prompts:

Only authenticated users should be able to access the dashboard. Users must only be able to view and edit their own tasks. Validate all form inputs, display safe error messages, and never expose secrets in frontend code.

For an administrative area, you might write:

Create an admin section that is available only to users with an admin role. Check authorization on the server for every admin action instead of relying only on hiding buttons in the interface.

For user-generated content:

Allow users to submit comments, but sanitize and safely render the content to reduce cross-site scripting risks. Limit comment length and reject empty submissions.

Detailed prompts help the generated application start from better assumptions.

Test Authentication and Authorization Separately

Authentication answers the question, “Who are you?”, while authorization answers the question, “What are you allowed to do?”

These are different.

A user may be successfully logged in but still not be allowed to view another user’s private records. A responsible application checks both.

Test cases should include:

1. A logged-out visitor tries to open a private page.

2. A regular user tries to open an administrator page.

3. A user tries to access another user's record by changing an identifier in the URL.

4. A user submits a request without the required session information

5. A user logs out and them presses the browser's back button.

Don't rely only on hiding navigation links. A hidden button isn't a security system. If a user can still call a backend endpoint directly, the application may be vulnerable.

For example, suppose your application has a page at /admin that should only be available to administrators. You could test authentication and authorization separately like this:

Authentication Test

Log out of the application and then try to open /admin directly. The application should redirect you to the login page or return an appropriate unauthorized response.

Then log in with a valid account and confirm that the application recognizes the authenticated session.

Authorization Test

Log in with a normal user account that doesn't have an admin role. Then try to open /admin directly instead of using the navigation menu. The application should deny access.

Try the same test against the backend endpoint used by an admin action. Confirm that the server also rejects the request.

You can also test whether changing an identifier in a URL or request allows one user to access another user's information. The important part is to verify the behavior from the user's perspective and, where possible, confirm that the server is enforcing the permission rather than simply hiding parts of the interface.

Validate All User Input

Users will enter unexpected information. Sometimes this happens by accident. Sometimes it happens because users are testing the boundaries of your application. Occasionally, it happens because someone has decided that a username should be 4,000 characters long and contain seventeen emojis.

Validate input on the client for a better user experience and on the server for security.

Examples of validation include:

• Required fields

• Maximum and minimum lengths

• Valid email formats

• Allowed file types

• Maximum file sizes

• Valid dates

• Acceptable numeric ranges

• Safe content handling

A frontend check might look like this:

if (username.trim().length < 3) {
     showError("Username must be at least 3 characters long.");
     return;
}                

But don't assume that frontend validation is enough. A user can bypass browser checks by sending requests directly to your backend.

The server should validate the data again before storing or processing it.

Server-side validation means treating everything received from the browser as untrusted input. The server should check that the submitted data has the expected type, format, length, and range before using it. It should also reject unexpected fields or values when appropriate.

For example, if an API accepts a username and age, the server could verify that the username is a non-empty string within the allowed length and that the age is a number within the application's acceptable range. If the request fails validation, the server should reject it rather than storing or processing the invalid data.

You can ask Lovable to help create these checks and generate test cases:

• Add server-side validation for every field in this form.

• Reject missing, incorrectly formatted, oversized, or out-of-range values before they're stored or processed.

• Then create tests for valid input, missing fields, invalid formats, boundary values, and unexpected input.

AI-generated tests can be useful, but don't rely on them as the only verification. Run the tests yourself and manually try important edge cases as well. The goal is to use AI to speed up the work while keeping human judgment involved in checking whether the validation actually protects the application.

Be Careful With Generated Dependencies

AI-generated projects may use libraries, packages, plugins, and external services. These tools can be helpful, but each dependency adds another piece to understand and maintain.

Ask Lovable:

List the main packages used in this project and explain why each one is needed.

You can also ask:

Identify dependencies that are unnecessary for the current features and suggest a simpler alternative.

Fewer dependencies can mean:

• Less code to maintain

• Fewer security updates

• Smaller application size

• Fewer compatibility problems

• Easier debugging

You don't need to remove every package. Just avoid collecting dependencies like digital souvenirs.

Design for Accessibility

An application isn't truly successful if many people can't use it.

Ask Lovable to include accessibility from the beginning:

Make the interface accessible. Use semantic HTML, keyboard navigation, visible focus states, descriptive labels, sufficient color contrast, and accessible error messages.

Check whether:

• Buttons have clear names

• Form inputs have labels

• Keyboard users can reach every interactive element

• Focus indicators are visible

• Text has enough contrast

• Images have useful alternative text

• Error messages explain how to fix a problem

• The layout works at different screen sizes

• Content remains usable when text is enlarged

Avoid using color as the only way to communicate meaning. For example, don't show errors only with a red border. Add text such as:

Email address is required.

Accessibility isn't just a compliance task. It usually makes the application easier for everyone to use.

Avoid Dark Patterns

A responsible app should help users make informed choices. It shouldn't trick them into doing something they didn't intend.

Avoid:

• Preselected marketing consent

• Hidden cancellation links

• Confusing double negatives

• Misleading buttons

• Fake countdown timers

• Notifications that look like system warnings

• Subscriptions that are easy to start but difficult to stop

• Important information hidden in tiny text

Use clear labels:

Delete account

is better than:

Continue

when the action permanently deletes an account.

For destructive actions, provide a confirmation step that clearly explains what will happen:

This will permanently delete your account and all saved tasks. This action cannot be undone.

Good design respects the user’s ability to choose.

Handle Errors Effectively

Every application experiences errors. Networks fail. Services go offline. Users close tabs at inconvenient moments. Servers occasionally decide to take an unscheduled vacation.

Don't display vague or misleading messages such as:

Something went wrong.

when you can provide useful guidance.

Better:

We could not save your task because the connection was interrupted. Check your internet connection and try again.

For developers, log enough information to investigate the problem without exposing sensitive data:

try {  await saveTask(task);} catch (error) {  console.error("Task save failed", {    operation: "create\_task",    message: error.message  });
  showError("Your task could not be saved. Please try again.");}

Avoid sending passwords, tokens, private messages, or personal records into logs.

Be Honest About AI-Generated Features

If your application uses AI to generate text, recommendations, summaries, images, or decisions, users should understand that the output may be wrong.

Use clear language:

This summary was generated automatically and may contain mistakes. Review it before sharing.

Avoid presenting AI-generated information as guaranteed fact, especially in areas such as:

• Health

• Finance

• Education

• Employment

• Legal information

• Safety

• Personal identity

• News and public information

Give users ways to correct, reject, or report problematic output. If an AI feature affects important decisions, provide human review whenever possible.

Protect Personal Data

Collect only the information your app actually needs.

If a task manager only needs an email address for account recovery, it probably doesn't need a user’s home address, phone number, favorite color, and childhood nickname.

Before adding a data field, ask:

Why do we need this information?

Then ask:

What could happen if this information were exposed?

Good data practices include:

• Collecting less information

• Explaining why information is needed

• Restricting access

• Deleting information when it's no longer necessary

• Avoiding unnecessary analytics

• Protecting data during transmission and storage

• Giving users meaningful control over their information

Data isn't free just because a form field is free to add.

Respect Copyright and Ownership

Don't ask Lovable to copy an existing product exactly, reproduce copyrighted artwork, or imitate a brand in a way that could confuse users.

Instead, describe the qualities you want:

Create a clean project-management interface with a left sidebar, clear status labels, and a spacious layout. Use original styling and avoid copying any specific company's branding.

Be careful with:

• Images

• Logos

• Icons

• Fonts

• Code snippets

• Written content

• Product names

• Brand colors

• User-generated material

Use assets that you created, licensed, or are allowed to use. When in doubt, choose an original design.

Test With Realistic but Fake Data

Use fictional data during development:

Name: Jordan 
ExampleEmail: jordan@example.test
testOrder ID: TEST-1001

Don't use real customer records just because they're convenient.

Create test cases for:

• Empty states

• Long names

• Very long text

• Invalid email addresses

• Duplicate records

• Missing images

• Slow connections

• Failed requests

• Expired sessions

• Multiple users

• Different screen sizes

• Keyboard-only navigation

Fake data helps you test realistic behavior without exposing real people’s information.

You can create fake data yourself by using clearly fictional names, addresses, email addresses, identifiers, and other values that can't be mistaken for real customer information. For larger datasets, you can also use a reputable fake-data generator or ask Lovable to create a dataset specifically for testing.

For example, you could ask:

Create 100 fictional user records for testing. Use clearly fake names and email addresses under `example.test`. Include different account types, missing optional fields, long names, and other edge cases. Do not use real people's information.

Review generated data before using it, especially if you obtain it from an external source. Avoid datasets containing real personal information unless you have a legitimate reason, appropriate authorization, and proper safeguards. When possible, use synthetic data designed specifically for testing so that realistic application behavior can be tested without exposing real people's information.

Test Before You Share the App

Before showing your project to others, follow a basic release checklist.

1. The app works on mobile and desktop screens.
2. Forms validate input correctly.
3. Authentication behaves as expected.
4. Users can't access data belonging to other users.
5. Secrets aren't included in frontend code.
6. Error messages are clear and safe.
7. Keyboard navigation works.
8. Important buttons have clear labels.
9. Empty states are understandable.
10. Loading states are visible.
11. Destructive actions require confirmation.
12. Test data doesn't contain real personal information.
13. External services are configured correctly.
14. The production environment uses secure settings.
15. The app has been tested after the final changes.

A checklist may feel less exciting than clicking a shiny “Publish” button, but it's much more exciting than explaining to users why the app deleted everything.

Ask Lovable to Review Its Own Work

AI tools can help with review tasks when given specific instructions.

Try prompts such as:

Review this application for authentication and authorization problems. Identify any route, database query, or API endpoint that may expose data to the wrong user.

Review the forms for missing validation, unclear error messages, and accessibility problems.

Review the project for hardcoded secrets, unsafe logging, and sensitive information that might appear in the browser.

Review the application for mobile layout problems and explain the changes you recommend.

Don't accept the review blindly. Compare the suggestions with your own testing and, for serious applications, get help from an experienced developer or security professional.

Learn From the Generated Code

Using Lovable responsibly doesn't mean avoiding AI-generated code. It means using the tool as an opportunity to learn.

When you receive a result, ask:

Explain this function in beginner-friendly language.

Show me a simpler version of this code.

What assumptions does this implementation make?

What are the possible failure cases?

How would this code behave with two users at the same time?

Try changing one small part manually. Read the error messages. Compare the before-and-after versions. Over time, the generated code will become less mysterious.

The goal isn't to memorize every programming concept immediately. The goal is to become confident enough to ask better questions and recognize risky answers.

Use Lovable for Prototyping Without Pretending It's Production-Ready

Lovable is excellent for exploring ideas quickly.

You can use it to:

• Test a product concept

• Build a portfolio project

• Create a prototype for user feedback

• Learn how web applications are structured

• Experiment with interfaces

• Build an internal tool

• Turn a rough idea into something people can react to

A prototype may not have the same security, reliability, monitoring, documentation, and scalability requirements as a public production application.

Be honest about the stage of your project. Use labels such as "Prototype", "Demo", "Work in Progress", and so on.

Don't treat a prototype like a finished product simply because it has a nice gradient and a button that says “Launch.”

Create a Simple Responsible Development Workflow

A practical workflow might look like this:

1. Define the problem

2. Identify the users

3. Decide what information the app needs

4. Write a clear prompt

5. Generate a small feature

6. Review the result

7. Test normal and unexpected behavior

8. Fix security and accessibility problems

9. Repeat for the next feature

10. Test the complete app

11. Remove test data and secrets

12. Document important decisions

13. Deploy only when the app is ready for its intended audience

This process isn't slow. It's controlled. The fastest path is often the one that avoids rebuilding the entire application after discovering that the foundation was made of optimism and unvalidated form fields.

A Responsible Prompt Template

You can use this template when asking Lovable to create a feature:

Build [feature] for [type of user].

The goal is to [explain the problem being solved].

Users should be able to:
- [action one]
- [action two]
- [action three]

The application should:
- Validate all user input.
- Protect authenticated routes.
- Ensure users can access only data they are authorized to access.
- Avoid hardcoded secrets.
- Use clear loading and error states.
- Support keyboard navigation.
- Work on mobile and desktop screens.
- Use accessible labels and sufficient color contrast.

Do not:
- Collect unnecessary personal information.
- Expose private data.
- Add unrelated features.
- Change existing authentication behavior without explaining the change.

After building the feature, explain:
- Which files changed.
- What data is stored.
- Which external services are used.
- What security risks remain.
- How I should test the feature.

This template encourages Lovable to think about more than appearance.

The Golden Rule of AI App Building

If an AI-generated feature affects another person, review it as if you will be the person affected.

Would you want your data stored there?

Would you understand what the app is doing?

Would you be able to correct a mistake?

Would you know how to delete your information?

Would you feel comfortable using the application on a phone, with a keyboard, or with a slow internet connection?

Would you trust the app if you knew how it was built?

These questions turn responsible development from an abstract idea into a practical habit.

Final Thoughts

Lovable can make app development more approachable, faster, and more fun. It can help beginners build their first projects and help experienced developers explore ideas without spending hours creating every screen from scratch.

But responsible use requires more than generating attractive interfaces. Write clear prompts. Protect secrets. Collect less data. Validate inputs. Test permissions. Design for accessibility. Respect ownership. Explain AI-generated features. Review the code. Keep people involved in important decisions.

The best AI-built applications aren't the ones created with the fewest clicks. They're the ones built with curiosity, care, and enough testing to survive contact with real users.

Use Lovable to move faster, but use your judgment to decide where you're going.

Happy coding!

The good news is that you don't need to be a security expert to fix this. Most of the work takes an afternoon. After that, privacy becomes a habit rather than a project. This guide walks through the steps in the order that gives you the biggest gain for the least effort.

In this article, you'll learn how to improve your online privacy without making browsing complicated. We'll cover the most important steps, from choosing a privacy-friendly browser and blocking trackers to using a VPN, securing your search history and DNS, and cleaning up your online accounts.

You'll also learn what these tools can and can't protect, so you can build a practical privacy setup that works for everyday browsing.

What We'll Cover:

• Start With Your Browser

• Block Trackers and Ads

• Hide Your IP Address With a VPN

• Fix Your Search and DNS

• Clean Up Your Accounts

• Know What Each Step Cannot Do

• A Simple Plan

Start With Your Browser

Your browser is the front door. If it leaks, nothing else matters much.

Chrome is fast, but it belongs to an advertising company. Firefox and Brave block trackers by default and are built with privacy in mind. Safari also does a decent job on Apple devices.

If you want to switch, Firefox is the easiest move. It imports your bookmarks and saved passwords in a couple of clicks.

If you stay with Chrome, change a few settings. Turn off third-party cookies. Turn off the ad topics feature, which lets Chrome guess your interests and share them with sites.

Set your browser to clear cookies when you close it. You'll sign in to sites more often, but trackers will lose their memory of you every day.

Also check what your browser is telling the world right now. The Electronic Frontier Foundation runs a free test called Cover Your Tracks. It shows you how unique your browser looks to a tracking company.

Run it before you make changes, then run it again after. The difference is a good reality check.

Block Trackers and Ads

Most tracking happens through scripts loaded from other companies. A page may look like one site, but it can quietly load code from ten different firms.

A content blocker stops those scripts before they run. uBlock Origin is the best-known free option and works in Firefox. It blocks ads too, which makes pages load faster and use less data. Privacy Badger, also from the EFF, takes a different approach. It watches which companies follow you across sites and blocks the ones that do.

Don't install multiple blockers at once. Two is plenty. More than that causes broken pages and slow browsing, and the extra protection is small.

Be careful about which extensions you trust. An extension can read everything on every page you visit. Stick to well-known tools with a clear privacy policy and a long track record. If an extension is free, tiny, and asks for wide permissions, skip it.

Hide Your IP Address With a VPN

Blocking trackers helps, but your internet provider still sees every site you visit. In many countries, providers are allowed to sell that data. Public Wi-Fi at a hotel or airport is worse, because other people on the network may be watching, too.

A virtual private network fixes this. It wraps your traffic in an encrypted tunnel and sends it through a server run by the VPN company. Your provider sees only that you connected to the VPN. Websites see the VPN server's address instead of yours.

If you want protection only inside your browser, a browser extension is the simplest route. The CyberGhost Chrome add-on is one example. It routes your Chrome traffic through an encrypted proxy, lets you pick a country, and blocks some trackers along the way. Anything outside Chrome, like a mail client or a game, still uses your normal connection. That's fine for casual privacy and it keeps your other apps fast.

One warning about free VPNs: running servers costs money. If a VPN is free and shows no clear business model, your data may be the product. Pick a paid service with a published no-logs audit, or use a free plan from a company you already trust.

Fix Your Search and DNS

Your search history is the most personal thing you own. It holds your health worries, your money problems, and your plans.

Google keeps that history tied to your account unless you tell it not to. Visit My Activity and set auto-delete to three months. While you're there, look at what has been saved. Most people find the amount surprising.

You can also switch your default search engine. DuckDuckGo and Startpage don't build a profile from your searches. Results are good enough for daily use, and you can still fall back to Google when you need it.

Then there's DNS, the system that turns a web address into a number your computer can reach. By default your provider handles this, so it sees every domain you visit even when the traffic itself is encrypted. Switching to an encrypted DNS service such as Cloudflare's 1.1.1.1 takes about two minutes and closes that gap. Most browsers now have a setting for secure DNS, so you may not even need to touch your system settings.

Clean Up Your Accounts

Privacy isn't only about traffic. It's also about the data already sitting on other people's servers.

Start by finding out where your details have leaked. The free site Have I Been Pwned tells you which breaches include your email address. If your address shows up in an old breach, change that password right away, and change it anywhere else you reused it.

Reused passwords are the single biggest risk most people carry. A password manager solves it. It creates a long random password for every site and remembers them all for you. Turn on two-factor authentication for your email, your bank, and any account tied to money. Email matters most, because whoever controls your email can reset everything else.

Finally, use different email addresses for different parts of your life. Many providers let you create aliases. One address for banking, one for shopping, and one for newsletters. When an alias starts getting spam, you know exactly who sold your data, and you can shut it off without changing your real address.

Know What Each Step Can't Do

It helps to be clear about limits, so you don't feel safer than you are.

A VPN hides your traffic from your provider, but it doesn't hide you from a site you log in to. If you sign in to a social network, that network knows it's you no matter which country your traffic seems to come from.

Blocking trackers reduces profiling, but browser fingerprinting can still identify you through your unique mix of settings. Only tools like the Tor Browser, which makes everyone look the same, defeat that fully. Tor is slower, so most people save it for the moments that really matter.

And nothing protects information you hand over yourself. If you post your holiday plans in public, no setting will pull them back.

A Simple Plan

If you only do three things, do these. Install a good content blocker, turn on a password manager and two-factor authentication, and use a VPN whenever you're on a network you don't control.

Then set a reminder for six months from now. Check your breach report, review the extensions you've installed, and delete the accounts you no longer use. Privacy isn't a wall you build once. It's closer to housekeeping. A little attention now and then keeps the mess from piling up, and it costs far less than cleaning up after your data has already escaped.

Hope you enjoyed this article. You can connect with me on LinkedIn.

You run:

git add .
git commit -m "Fix API integration"
git push

A few minutes later, you notice something strange. Your API usage has suddenly increased. Maybe there are unexpected requests, new cloud resources, or even a bill that looks much larger than expected.

Then you find it:

const apiKey = "sk_live_123456789";

Your API key is sitting in a Git repository.

This situation is stressful, but it's fixable.

The most important rule is:

If an API key has been committed to Git, assume it has been copied and compromised, even if you delete it immediately.

Deleting the key from the latest version of your file doesn't make the old key safe. Git keeps previous versions of files in its history, and exposed credentials can be discovered by automated scanners.

In this guide, you'll learn the following:

• What Is an API Key?

• The Emergency Response: What to Do First

• Step 1: Revoke or Rotate the Leaked Key

• Step 2: Investigate Suspicious Activity

• Step 3: Remove the Secret From Your Current Code

• Step 4: Use a.envFile for Local Development

• Step 5: Create a Safe.env.example

• Step 6: Determine Whether the Secret Is Still in Git History

• When Do You Need to Rewrite Git History?

• Step 7: Remove the Secret From Git History

• Step 8: Verify That the Secret Is Gone

• Step 9: Push the Cleaned History Carefully

• Step 10: Replace the Credential Everywhere

• Step 11: Restrict the Replacement Key

• What About Frontend Applications?

• Environment Variables vs Secret Managers

• Add Secret Scanning to Your Workflow

• Use Git Hooks as an Extra Safety Net

• Review Your Staged Diff Before Committing

• Common Mistakes Developers Make

• A Complete API-Key Incident Checklist

• A Secure Project Structure

We'll use this basic workflow throughout the article:

Invalidate → Investigate → Remove → Replace → Prevent

Let's start with what an API key actually is before we get to the most important part: what to do right now after a key is exposed.

What Is an API Key?

An API key is a credential that allows an application to communicate with another service.

For example, an application might use an API key to access:

• A weather service

• A payment provider

• A mapping service

• An artificial intelligence API

• A cloud platform

• A database

• An email provider

• A private company API

A key might look something like this:

const apiKey = "your-real-api-key";

Or it might appear in a configuration file:

{
  "apiKey": "your-real-api-key",
  "databasePassword": "your-real-password"
}

API keys are often called secrets because possessing one may allow someone to make requests, access data, create resources, or generate charges on your account.

Not every API key is equally sensitive. Some services provide browser keys that are intentionally visible to users. Those keys should still have appropriate restrictions, quotas, and permissions.

As a general rule:

If a credential can access private data, create resources, modify records, or generate charges, it shouldn't be stored directly in your source code.

The Emergency Response: What to Do First

When you discover a leaked credential, a common reaction is to delete the key from the file and push another commit.

Don't start there.

Your first priority is to make the leaked credential useless.

Use this order of operations:

1. Invalidate the leaked credential
2. Investigate suspicious activity
3. Remove the secret from your code
4. Replace it with a new credential
5. Clean the Git history if necessary
6. Verify the cleanup
7. Add protections against future leaks

Think of an API key like a house key that was dropped in a crowded street.

Deleting a picture of the key doesn't matter if someone already picked up the physical key.

Change the lock first.

Step 1: Revoke or Rotate the Leaked Key

Go to the dashboard of the service that issued the credential.

Depending on the provider, you may see options such as:

• Revoke

• Delete

• Disable

• Rotate

• Regenerate

• Create new key

If the provider supports key rotation, create a replacement credential before disabling the old one if possible. This can reduce application downtime while you update your configuration.

The important thing is that the original credential must no longer be usable.

Do not reuse the leaked key. Don't rename it. Don't encode it. Don't move it to another file and assume it is safe. Don't assume nobody saw it.

Treat it as compromised.

Step 2: Investigate Suspicious Activity

After disabling the credential, check the provider's usage dashboard and logs.

Look for things such as:

• Sudden spikes in requests

• Requests from unfamiliar locations

• Unexpected database queries

• New cloud resources

• Changes to permissions

• Unexpected downloads

• Unusual payment activity

• New deployments

• Requests at times when your application was inactive

If the credential had broad permissions, assume that anything within its permission scope MAY have been accessed or modified.

For example, if a cloud credential could create virtual machines, check whether unexpected machines were created.

If a credential could access a database, review:

• Authentication logs

• Read operations

• Write operations

• Deleted records

• Exported data

• Newly created accounts

• Permission changes

Also check your billing information if the credential could generate usage-based charges.

Write down what you discover. A simple timeline can help:

10:15 - API key committed
10:23 - Repository pushed publicly
10:41 - Unusual usage detected
10:45 - Key revoked
11:00 - Logs reviewed
11:30 - Replacement key deployed
12:00 - Git history cleaned

This can be especially useful if you need to report the incident to a team or service provider.

Step 3: Remove the Secret From Your Current Code

Once the original credential has been disabled, remove it from your working files.

This is unsafe:

const apiKey = "your-real-api-key";

Instead, load the credential from the environment:

const apiKey = process.env.API_KEY;

if (!apiKey) {
  throw new Error("API_KEY is not configured");
}

In Python:

import os

api_key = os.environ.get("API_KEY")

if not api_key:
    raise RuntimeError("API_KEY is not configured")

The important idea is simple:

Source code → environment variable → secret value

instead of:

Source code → hardcoded secret

Environment variables aren't the only way to manage secrets, but they are a common and practical solution for local development and many deployment environments.

Step 4: Use a .env File for Local Development

For local development, you can store environment variables in a .env file.

For example:

API_KEY=your-local-development-key
DATABASE_URL=your-local-database-url

A Node.js project can load these values with a package such as dotenv.

Install it with:

npm install dotenv

Then:

import "dotenv/config";

const apiKey = process.env.API_KEY;

The important part is that the .env file normally should not be committed to Git.

Add it to .gitignore:

# Environment files
.env
.env.*
!.env.example

# Credential files
*.pem
*.key
credentials.json
service-account.json

# Local development files
.DS_Store

But there's an important detail here: the .gitignore does NOT remove files that Git is already tracking.

If .env has already been committed, adding it to .gitignore won't erase it from Git.

You can stop tracking the file while keeping it on your computer:

git rm --cached .env

Then commit the .gitignore change:

git add .gitignore
git commit -m "Ignore local environment files"

But remember: this only removes the file from future commits. It does not remove the secret from previous commits.

That's where Git history comes in.

Step 5: Create a Safe .env.example

Other developers still need to know which environment variables the application requires.

Instead of committing .env, create .env.example:

API_KEY=
DATABASE_URL=
PORT=3000
LOG_LEVEL=info

This file contains variable names rather than real credentials, so it can be committed to the repository.

You can also provide comments:

# Required API credential
API_KEY=

# PostgreSQL connection string
DATABASE_URL=

# Optional application port
PORT=3000

A new developer can then copy the file:

cp .env.example .env

and provide their own values.

Use clearly fake placeholders in examples:

API_KEY=replace-me-with-your-own-key

Avoid putting realistic-looking production credentials into .env.example.

Step 6: Determine Whether the Secret Is Still in Git History

This is one of the most important parts of fixing a leaked credential.

Suppose your Git history looks like this:

Commit A: Add API key to config.js
Commit B: Update API integration
Commit C: Delete API key

Even though Commit C no longer contains the key, Commit A still does.

Git remembers previous versions of your files.

You can inspect the history of a file with:

git log --all -- config.js

To display a file from an older commit:

git show COMMIT_ID:config.js

You can also search Git history for a known leaked value:

git log --all -S"your-leaked-key" --oneline

If you know the secret was committed, you should assume that it exists somewhere in the repository's history until you've verified otherwise.

When Do You Need to Rewrite Git History?

Not every accidental secret requires a history rewrite. Consider these situations:

The Secret Was Never Committed

If the secret exists only in your working directory and was never committed, you generally don't need to rewrite history.

Remove it, add the appropriate file to .gitignore, and continue.

The Secret Was Committed Locally But Never Pushed

If the secret exists in local commits but hasn't been shared with a remote repository, you may be able to clean up those commits before pushing.

The Secret Was Pushed to a Remote Repository

Treat the credential as compromised. Revoke or rotate it immediately.

Then determine whether removing the secret from the repository's history is appropriate.

The Repository Was Public

Assume that someone or something may already have copied the secret.

This is why revocation comes before Git cleanup.

The Secret Was in a Private Repository

A private repository is safer than a public repository, but it isn't a secret vault.

Credentials can still escape through:

• Compromised accounts

• Contractors

• Integrations

• CI logs

• Forks

• Backups

• Screenshots

• Copied code

• Pull requests

So the safest rule remains:

Never intentionally commit credentials to Git, even in a private repository.

Step 7: Remove the Secret From Git History

If the credential was committed, you may need to remove it from the repository's history.

Before rewriting history, create a backup:

git clone --mirror https://github.com/your-username/your-repository.git repository-backup.git

A mirror clone includes branches and tags, which makes it useful for recovery if something goes wrong.

Option 1: Remove an Entire File

If the secret was stored in a file such as .env, you can remove that file from the entire history:

git filter-repo --path .env --invert-paths

For a file inside a directory:

git filter-repo --path config/production.json --invert-paths

This removes the file from the repository's rewritten history.

Option 2: Replace a Secret Inside a File

Sometimes you need to keep the file but remove the secret from previous versions.

Create a temporary replacements file:

Then run:

git filter-repo --replace-text replacements.txt

You can replace the value with a placeholder:

your-leaked-key==>YOUR_API_KEY_HERE

Be extremely careful with replacements.txt. It contains the original secret, so do not commit it.

Delete it after the cleanup:

rm replacements.txt

On Windows PowerShell:

Remove-Item replacements.txt

For multiple secrets:

old-api-key==>REMOVED_API_KEY
old-database-password==>REMOVED_DATABASE_PASSWORD
old-token==>REMOVED_TOKEN

Then:

git filter-repo --replace-text replacements.txt

Test the cleanup on your backup clone first.

Step 8: Verify That the Secret Is Gone

Never assume the cleanup worked just because the command completed successfully.

Search for the known leaked value again:

git log --all -S"your-leaked-key" --oneline

You can also inspect relevant files and commits:

git log --all -- config.js

and:

git show COMMIT_ID:config.js

If your repository uses branches and tags, make sure you aren't checking only the branch you currently have checked out.

You should also inspect other locations where the secret may have appeared, including pull requests, CI/CD logs, build artifacts, release files, Docker images, package releases, documentation, issue comments, and screenshots

Remember:

Rewriting your repository doesn't erase copies that already exist somewhere else.

That's another reason why the original credential must be revoked.

Step 9: Push the Cleaned History Carefully

Once you've verified the cleanup, you may need to push the rewritten history:

git push --force --all origin
git push --force --tags origin

Important Warning

Force-pushing rewritten history is disruptive. It changes commit hashes and can affect collaborators who have existing clones of the repository.

Before doing this on a shared project:

1. Tell your collaborators.

2. Make sure everyone understands that history is being rewritten.

3. Coordinate the cleanup.

4. Follow your organization's incident-response process if one exists.

After the rewrite, collaborators may need to reclone the repository:

git clone https://github.com/your-username/your-repository.git

They shouldn't blindly merge their old repository history back into the cleaned repository.

Step 10: Replace the Credential Everywhere

Now create or use the replacement credential. Update every environment where the application runs. Common locations include:

• Local development

• Testing

• Staging

• Production

• Docker containers

• Kubernetes secrets

• CI/CD systems

• Hosting platforms

• Scheduled jobs

• Serverless functions

A common mistake is updating production but forgetting the deployment pipeline.

For example, your local application may work because .env contains the new key, while your CI/CD system still contains the old one.

Make a checklist:

1. Local development
2. Automated tests
3. Staging
4. Production
5. CI/CD variables
6. Docker configuration
7. Cloud deployment settings
8. Scheduled scripts
9. Serverless functions

After updating the credential, test the application in each important environment.

Step 11: Restrict the Replacement Key

Replacing a leaked credential is only part of the solution.

The new credential should have only the permissions it actually needs.

Useful restrictions can include:

• Read-only permissions

• Specific API scopes

• Allowed IP addresses

• Allowed domains

• Environment-specific access

• Request quotas

• Rate limits

• Expiration dates

For example, a weather application may only need permission to read weather data.

It shouldn't have permission to manage users, modify billing, or delete unrelated resources.

This is the principle of least privilege:

Give each credential the smallest amount of access necessary to perform its job.

It's also a good idea to use different credentials for different environments:

local-development-key
testing-key
staging-key
production-key

That way, a development credential leak doesn't automatically expose production resources.

What About Frontend Applications?

This is where API-key security gets confusing.

Frontend code runs on the user's device.

That means users can inspect it.

For example:

const apiKey = "browser-key";

A user can inspect the JavaScript bundle, browser developer tools, or network requests and potentially see the value.

Some services intentionally provide browser API keys that are designed to be publicly visible.

Those keys should still be restricted by things such as:

• Allowed domains

• Website origins

• API operations

• Usage quotas

• Referrer restrictions

• Time limits

But a truly private credential should never be placed in browser code.

Instead of:

fetch("https://private-api.example.com/data", {
  headers: {
    Authorization: "Bearer private-secret-token"
  }
});

have the browser call your own backend:

fetch("/api/data");

Then the backend communicates with the private service:

const response = await fetch(
  "https://private-api.example.com/data",
  {
    headers: {
      Authorization: `Bearer ${process.env.PRIVATE_API_TOKEN}`
    }
  }
);

The backend can then return only the information the browser is allowed to receive.

The important distinction is:

Public/browser credential
        ↓
Can be visible, but should be restricted

Private credential
        ↓
Must remain on a trusted backend or secret-management system

Environment Variables vs Secret Managers

Environment variables are useful, but they're not a universal secret-management solution.

For a small application or local development environment, something like:

API_KEY=your-secret

may be perfectly reasonable.

For larger production systems, you may want a dedicated secret manager.

A secret-management system can provide features such as:

• Centralized credential storage

• Access controls

• Auditing

• Credential rotation

• Versioning

• Separation between environments

• Integration with deployment systems

The important idea is that your source code shouldn't be responsible for storing production secrets.

Instead:

Application
    ↓
Secret management system
    ↓
Credential

rather than:

Application
    ↓
Hardcoded production credential

Which solution you use depends on the size and requirements of your project.

Add Secret Scanning to Your Workflow

Humans are excellent programmers and occasionally terrible search engines.

Automated secret scanning can catch credentials before they make it into a repository.

Popular tools include:

• Gitleaks

• TruffleHog

• detect-secrets

• Pre-commit hooks

• Git hosting secret scanning

• CI security scanners

For example, you can run Gitleaks locally:

gitleaks detect --source . --verbose

You can also integrate secret scanning into CI.

A basic GitHub Actions workflow might look like this:

name: Secret Scan

on:
  push:
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Scan for secrets
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Review the documentation for your chosen tool and pin versions according to your project's security practices.

Secret scanners can produce false positives, so you may need to configure exceptions for safe test values.

Be careful with allowlists, though. An overly broad exception can hide a real credential.

Use Git Hooks as an Extra Safety Net

You can also scan files before they're committed.

For example, a simple pre-commit script could search for suspicious words:

#!/usr/bin/env bash

if grep -RniE "api[_-]?key|password|secret|token|private[_-]?key" . \
  --exclude-dir=.git \
  --exclude=".env.example"; then

  echo "Possible secret detected. Commit cancelled."
  exit 1
fi

This isn't a complete security scanner, but it can catch obvious mistakes.

For stronger protection, use a dedicated secret-scanning tool through a pre-commit framework.

The goal isn't to make committing miserable. The goal is to make accidentally publishing a credential harder.

Review Your Staged Diff Before Committing

One of the simplest security habits you can develop is checking what you're actually about to commit.

First:

git status

Then stage only the files you intend to commit:

git add src/api.js README.md

Now inspect the staged changes:

git diff --cached

Look for:

• API keys

• Passwords

• Tokens

• Private URLs

• Internal hostnames

• Customer data

• Debug output

• Personal information

• Private certificates

Only commit after the staged diff looks correct:

git commit -m "Load API key from environment"

Be cautious with:

git add .

It can stage files you never intended to publish, including .env files, database exports, generated files, or local configuration.

Common Mistakes Developers Make

Mistake 1: "I Deleted It, So It's Fine"

Deleting a secret from the current version of a file doesn't delete it from Git history.

Correct response: Revoke the credential and clean the repository history when appropriate.

Mistake 2: "The Repository Is Private"

Private repositories aren't vaults.

Credentials can still escape through compromised accounts, integrations, CI logs, forks, backups, or copied code.

Correct response: Don't commit secrets even to private repositories.

Mistake 3: "I'll Just Encode It"

These don't make a credential secret:

const key = atob("c29tZS1rZXk=");

or:

const key = "some-" + "secret-" + "value";

Encoding, splitting, renaming, or hiding a credential doesn't protect it.

If your application can reconstruct the credential, someone analyzing the application may be able to do the same.

Mistake 4: Logging the Secret

Don't do this:

console.log(process.env.API_KEY);

Logs can be stored by your terminal, CI system, hosting provider, monitoring platform, or cloud service.

Instead:

console.log(
  "API key configured:",
  Boolean(process.env.API_KEY)
);

If you absolutely need to inspect a value during debugging, avoid printing the full credential.

For example:

function maskSecret(value) {
  if (!value) return "not configured";
  if (value.length <= 8) return "********";

  return `${value.slice(0, 4)}...${value.slice(-4)}`;
}

console.log(maskSecret(process.env.API_KEY));

Even masked credentials should be handled carefully.

Mistake 5: Using the Same Credential Everywhere

If local development, testing, staging, and production all use the same credential, one leak can affect everything.

Correct response: Use separate credentials with separate permissions.

Mistake 6: Cleaning Only the Current Branch

A secret can remain in:

• Old branches

• Tags

• Pull requests

• Other references

Correct response: Consider the entire repository when investigating and cleaning a leaked credential.

Mistake 7: Forgetting Build Artifacts

A secret might also appear in:

• Compiled JavaScript bundles

• Docker images

• Downloadable releases

• Published packages

• Generated documentation

Correct response: Revoke the credential and identify affected artifacts that may need to be removed or replaced.

A Complete API-Key Incident Checklist

If you discover that you've exposed an API key, use this checklist:

1. Revoke or rotate the leaked key
2. Create a replacement credential
3. Restrict the replacement credential
4. Review provider logs
5. Review billing and usage
6. Check for unauthorized resources
7. Remove the key from current files
8. Add secret files to .gitignore
9. Create or update .env.example
10. Search Git history
11. Check branches and tags
12. Remove the secret from Git history if necessary
13. Verify the old secret is gone
14. Force-push cleaned history if appropriate
15. Check pull requests and forks
16. Check CI and deployment logs
17. Update local configuration
18. Update staging configuration
19. Update production configuration
20. Update CI/CD secrets
21. Run a secret scanner
22. Document the incident
23. Add preventive security checks

The exact steps will depend on your provider and project, but the order matters: Invalidate first. Clean up second.

A Secure Project Structure

A simple Node.js project might look like this:

my-project/
├── src/
│   └── api.js
├── .env
├── .env.example
├── .gitignore
├── package.json
└── README.md

The local .env file contains the actual development value:

API_KEY=your-local-key

The .env.example file contains no real credential:

API_KEY=replace-me-with-your-own-key

The application reads the environment variable:

import "dotenv/config";

const apiKey = process.env.API_KEY;

if (!apiKey) {
  throw new Error("Missing API_KEY environment variable");
}

export async function getData() {
  const response = await fetch(
    "https://api.example.com/data",
    {
      headers: {
        Authorization: `Bearer ${apiKey}`
      }
    }
  );

  if (!response.ok) {
    throw new Error(
      `API request failed: ${response.status}`
    );
  }

  return response.json();
}

And .gitignore keeps the local environment file out of future commits:

.env
.env.*
!.env.example

node_modules/

Finally, your README can explain the setup without exposing credentials:

Step 1: Copy the example environment file on your bash cp .env.example .env

Step 2: Add your own API key to .env.

And last but not least, start your application!

npm start

Final Thoughts

Leaking an API key doesn't mean you're a terrible developer. It just means your development workflow needs better guardrails.

The important thing is knowing how to respond quickly and how to prevent the same mistake from happening again.

Remember the emergency formula:

Invalidate → Investigate → Remove → Replace → Prevent

The important thing is knowing how to respond quickly and how to prevent the same mistake from happening again.

• Invalidate the leaked credential so it can no longer be used.

• Investigate your logs, usage, and billing to determine whether it was abused.

• Remove the secret from your current code and, when necessary, from Git history.

• Replace it with a new credential that has only the permissions it needs.

• Prevent future leaks with environment variables, secret managers, secret scanning, and careful Git practices.

Git is excellent at remembering your project's history. That's useful when you accidentally delete an important function. But it's much less useful when that history contains a password.

So keep your code public when appropriate. And keep your secrets somewhere else.

Happy coding!

While these advantages have fueled rapid cloud adoption, they've also introduced new security challenges that many organisations fail to recognise until it's too late.

Unlike traditional data centers, cloud environments are dynamic. New virtual machines, databases, storage accounts, containers, and serverless functions can appear and disappear throughout the day.

As cloud infrastructure grows, so does the number of potential security gaps. Many of these vulnerabilities remain hidden because they stem from configuration errors rather than software flaws.

This is where cloud security posture management becomes essential. Instead of relying on periodic security reviews, organisations can continuously monitor their cloud environments, identify risks, and remediate issues before they become security incidents.

For each of the five risks below, I'll walk through one concrete example of how the problem actually shows up in a real environment, and the specific steps a team would take to fix it. The examples are deliberately provider-neutral, because the same patterns appear across every major cloud platform under different names.

Understanding these hidden risks is the first step toward building a stronger and more resilient cloud environment.

What We'll Cover

• Why Cloud Infrastructure Creates New Security Challenges

• Hidden Risk #1: Cloud Misconfigurations

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #2: Excessive Permissions

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #3: Forgotten Cloud Resources

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #4: Compliance Drift

  • What this looks like in practice

  • How to mitigate it

• Hidden Risk #5: Limited Visibility Across Multi-Cloud Environments

  • What this looks like in practice

  • How to mitigate it

• Why Manual Security Reviews Are No Longer Enough

• How Cloud Security Posture Management Strengthens Cloud Security

• Building a Stronger Cloud Security Strategy

• Conclusion

Why Cloud Infrastructure Creates New Security Challenges

Modern cloud environments are far more complex than the on-premises infrastructure many security teams were built to protect. A single organisation may use Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) simultaneously. Each provider offers hundreds of services, each with its own security settings and access controls.

Development teams also create new cloud resources at an unprecedented pace. Infrastructure as Code, automated deployment pipelines, and container orchestration platforms enable provisioning entire environments in minutes. While this improves productivity, it also increases the likelihood of configuration errors.

The shared responsibility model adds another layer of complexity. Cloud providers secure the underlying infrastructure, but customers remain responsible for securing their applications, identities, storage, networks, and data. Many organisations mistakenly assume their cloud provider handles every aspect of security, leaving critical resources exposed.

Hidden Risk #1: Cloud Misconfigurations

Misconfigurations remain one of the leading causes of cloud security incidents. A single incorrect setting can expose sensitive information to the public internet or give attackers access to critical systems.

Examples include publicly accessible storage buckets, databases that allow unrestricted network access, disabled encryption settings, and security groups with overly permissive firewall rules. These mistakes often occur during rapid deployments when security reviews are skipped or delayed.

The challenge is that cloud environments change constantly. Even if an environment is secure today, a new deployment tomorrow may introduce a vulnerability without anyone noticing.

What This Looks Like in Practice

A product team needs to share a set of report images with an external design agency. The quickest path is to create an object storage bucket and set its access policy to allow read access from any identity, rather than setting up temporary signed URLs or a guest account. The share works, the agency delivers, and the ticket is closed.

Six weeks later, the same bucket is reused as the destination for a nightly export job. Nobody rechecks the access policy, because the bucket already exists and "works." The export contains customer names, email addresses, and internal order IDs, and it's now readable by anyone who can guess or enumerate the bucket name. Automated scanners that crawl for open buckets will typically find it within days.

Nothing was hacked here. No credential was stolen and no software was exploited. A setting that was reasonable for one purpose was silently inherited by another.

How to Mitigate it

1. Turn on the account-level public access block: Every major provider offers an org- or account-wide setting that overrides individual resource policies and prevents public exposure of storage. Enable it as the default, and treat any exception as a change that requires approval.

2. Scan the infrastructure code before it's applied, not after: Add a policy-as-code check to your deployment pipeline so a template that grants public read on a bucket fails the build. This is where you catch the problem for free, before it exists in the environment.

3. Replace public access with time-limited access: For genuine external sharing, use pre-signed URLs with an expiry measured in hours, or a scoped guest identity. This gives the same convenience without leaving a permanent hole.

4. Alert on the change, not just the state: Configure your posture management tool to fire when a bucket's access policy changes to allow anonymous principals, and route it to the team that owns the resource. Detecting the drift within minutes is what separates a near miss from a breach notification.

Cloud security posture management continuously scans cloud resources for configuration issues and alerts security teams when risks appear. Instead of waiting for a scheduled audit, organisations gain visibility into problems as they occur.

Hidden Risk #2: Excessive Permissions

Identity and Access Management (IAM) is one of the most important parts of cloud security. Unfortunately, it's also one of the most common sources of risk.

Many organisations grant users, applications, or service accounts more permissions than they actually need. Over time, these excessive privileges accumulate as employees change roles, applications evolve, and temporary permissions become permanent.

If an attacker compromises an account with broad privileges, the damage can spread quickly across the cloud environment. Sensitive data, production systems, and administrative resources may all become accessible.

What This Looks Like in Practice

A team sets up a deployment pipeline. The pipeline needs to create compute instances, update a container registry, and write to a couple of storage locations. Working out the exact permission set takes time, and the first three attempts fail with cryptic access-denied errors. Under deadline pressure, someone attaches the built-in administrator role to the pipeline's service identity "just to unblock it," with the intention of narrowing it later.

Later never comes. The pipeline now holds full administrative rights across the account, and its credentials live in a CI/CD system that dozens of engineers can configure. Anyone who can modify a build file, or who compromises a single developer's account on that CI system, can execute arbitrary commands with administrator privileges like read every database, create new identities, or disable logging.

The same pattern appears with people. An engineer moves from platform to frontend work, keeps the old production access "in case something breaks," and two years later has accumulated permissions spanning four teams they no longer belong to.

How to Mitigate it

1. Pull the last-accessed data before you touch anything: Every major provider records which services an identity has actually used and when. Start by listing every role with administrative privileges and check what it has genuinely called in the past 90 days. In most environments the real usage is a small fraction of what's granted.

2. Generate a scoped policy from observed activity: Use the provider's policy-generation tooling (or your posture management platform) to produce a candidate policy based on that access history, then review it manually before applying. Applying the principle of least privilege is far easier when you start from evidence rather than guesswork.

3. Roll it out in audit mode first: Apply the narrowed policy to a non-production copy of the pipeline, or run it alongside the permissive one with denial logging enabled, so you find the missing permissions without breaking deployments.

4. Set an expiry on elevated access: Temporary administrative rights should be granted through a just-in-time mechanism that expires automatically, so the "narrow it later" step happens whether or not anyone remembers.

5. Watch for privilege escalation paths, not just broad roles: A role that can't do much directly but can create identities, attach policies, or pass a privileged role to a new resource is effectively an administrator. Posture management tools flag these chains, which are almost impossible to spot by reading policies one at a time.

Hidden Risk #3: Forgotten Cloud Resources

Cloud environments are rarely static. Development teams frequently create temporary virtual machines, test databases, storage accounts, and networking components while building new features. Once projects are completed, these resources are often forgotten rather than removed.

Although these abandoned resources may no longer serve a business purpose, they continue to consume cloud resources and may still contain sensitive information. Some may even remain connected to production networks.

These overlooked assets create unnecessary attack surfaces. Attackers often look for neglected systems because they're less likely to receive security updates or active monitoring.

What This Looks Like in Practice

Ahead of a major release, a team spins up a load-testing environment: a handful of compute instances, a database restored from a production snapshot so the test data is realistic, and a permissive network rule opening administrative access from any address, because the testers are working from home and nobody wants to maintain an IP allowlist for a two-week project.

The release ships. The environment stays. A year on, those instances are running an operating system image that has missed four rounds of patches, the database still contains a year-old copy of real customer records, and the administrative port is still open to the internet. It doesn't appear on any architecture diagram, it isn't in the patching rota, and no alerting is configured against it, because it was never meant to exist for more than a fortnight.

This is one of the most attractive targets in a cloud estate: real data, no monitoring, and a known-vulnerable software version.

How to Mitigate it

1. Build an inventory that doesn't depend on memory: Query every region in every account for running compute, databases, storage, and snapshots, including the regions you believe you don't use, which is where forgotten resources most often hide. Deploying application security tools across both active and forgotten resources helps ensure vulnerabilities are caught before they can be exploited.

2. Sort by last activity, then by exposure: Rank resources by how recently they saw network traffic or authentication, and cross-reference against public network reachability. A resource that's both idle and internet-facing is the highest-priority item on the list, every time.

3. Require an owner tag at creation: Enforce a policy that rejects the creation of any resource without owner and environment tags. Untagged resources become the queue you work through, and the queue stops growing.

4. Give non-production resources a default lifetime: Attach an expiry tag to anything created in a test or sandbox account, and run an automated job that stops resources past their date and deletes them after a grace period. Make deletion the default and continuation the thing that requires a decision.

5. Treat production data in test environments as an incident: Restoring a production snapshot into a lower environment should trigger an alert. If realistic data is genuinely needed, mask it as part of the restore process.

Continuous asset discovery is another important capability of cloud security posture management. By maintaining an up-to-date inventory of cloud resources, organisations can quickly identify unused assets and remove them before they become security liabilities.

Hidden Risk #4: Compliance Drift

Meeting regulatory requirements isn't a one-time project. Standards such as ISO 27001, SOC 2, PCI DSS, HIPAA, and GDPR require organisations to maintain secure configurations over time.

As cloud environments evolve, once-compliant systems can gradually drift away from required security baselines. A disabled encryption setting, an open network port, or missing activity logs may create compliance violations without anyone noticing.

Manual compliance reviews often occur quarterly or annually, leaving long periods during which risks remain undetected.

What This Looks Like in Practice

An organisation passes its SOC 2 audit. Part of the evidence is that API activity logging is enabled across all accounts and all regions, with logs written to a dedicated, immutable archive.

Four months later, an engineer investigating a surprise bill notices that log storage is one of the larger line items. Most of the volume comes from high-frequency read events in a development account. They disable logging in that account to cut costs. It's a sensible-looking decision made with incomplete context, and it's approved in a routine change ticket by someone who doesn't know the account is in the audit scope.

Nothing breaks and no alert fires. The gap surfaces eight months later during audit fieldwork, when the assessor asks for a continuous log record for the period and the organisation can't produce one. There's no way to backfill it. The finding stands, and the same gap means that if an incident had occurred in that account, there would be no record of who did what.

How to Mitigate it

1. Map each control to a machine-checkable rule: "Activity logging is enabled everywhere" becomes a specific check: a multi-region trail exists in every account, log file validation is on, and the destination has retention and object-lock configured. If a control can't be expressed as a check, it can't be monitored continuously.

2. Make the critical controls structurally hard to disable: Deploy logging from the organisation level rather than per account, and apply a service control policy that denies the disable and delete actions to everyone except a break-glass identity. This turns a one-click mistake into a deliberate act.

3. Run the checks continuously and alert on transitions: The valuable signal isn't the monthly compliance score, it's the moment a control moves from compliant to non-compliant. Route that event to a channel someone actually reads, with the account, the resource, and the identity that made the change.

4. Add a compliance-scope check to your change process: Change tickets touching accounts in audit scope should surface that fact automatically, so the approver knows what they're approving.

5. Keep the evidence as you go: Store the continuous check results with timestamps. Audit preparation then becomes an export rather than a scramble, and you find gaps months before an assessor does.

Cloud security posture management continuously evaluates cloud environments against security frameworks and compliance standards. This allows organisations to detect violations early, simplify audit preparation, and reduce regulatory risk.

Hidden Risk #5: Limited Visibility Across Multi-Cloud Environments

Many organisations now operate workloads across multiple cloud providers. Development teams may choose different platforms based on cost, performance, or specialised services.

While this approach offers flexibility, it also creates fragmented security visibility. Each cloud provider has its own monitoring tools, dashboards, and security controls. Security teams may struggle to understand the overall security posture of the organisation.

Without centralised visibility, important risks can easily go unnoticed. One cloud environment may enforce strong security policies while another contains exposed resources and outdated configurations.

What This Looks Like in Practice

An organisation runs its core platform on one cloud provider, where the security team has spent two years building guardrails: preventative policies, automated scanning, and an on-call rota that responds to findings.

Meanwhile, the data science group adopts a second provider for its managed machine learning tooling. The account is opened on a corporate card, funded through a departmental budget, and never onboarded into the security programme. There's no reason for anyone to notice, no ticket crosses the security team's desk, because from the data team's perspective nothing about their work has changed.

That second environment ends up hosting training datasets derived from production records. Its storage defaults are different from the primary provider's, the equivalent of "block public access" isn't switched on, and the terminology doesn't map cleanly onto the checks the security team already wrote. The organisation's security dashboard shows a healthy posture, and it's accurate for the 70% of the estate it can see.

The failure isn't a weak control. It's that a whole environment is outside the boundary of the thing doing the measuring.

How to Mitigate it

1. Start with a billing reconciliation, not a technical scan: Ask finance for every cloud vendor the organisation pays, then compare that list against the accounts your security tooling monitors. Shadow cloud environments show up in expense reports long before they show up in a scanner.

2. Write your baseline in terms of outcomes, not provider features: Define controls as "object storage must not be readable by anonymous principals" rather than naming a specific provider's setting. One statement then maps onto every platform, and the standard doesn't fragment.

3. Normalise findings into a single queue: Pull results from each provider into one place with consistent severity scoring, so a critical exposure in the smallest environment ranks above a low-severity finding in the largest. Severity should reflect business impact, not which dashboard raised it.

4. Make onboarding a condition of account creation: Require any new cloud account, on any provider, to be enrolled in central monitoring before it can hold production or production-derived data. Pair the requirement with a fast, low-friction path so teams aren't tempted to route around it.

5. Re-run the discovery on a schedule: New accounts appear continuously. Repeat the billing reconciliation quarterly, because the environment you don't know about is by definition the one you'll never get an alert for.

Cloud security posture management provides a unified view across multiple cloud providers, allowing organisations to monitor security consistently regardless of where workloads are deployed.

Why Manual Security Reviews Are No Longer Enough

Traditional security audits were designed for relatively stable infrastructure. Cloud environments operate very differently.

Infrastructure changes can occur hundreds or even thousands of times each day through automated deployment pipelines. New services are launched continuously, configurations are updated frequently, and development teams release software at a rapid pace.

Human reviewers can't keep up with this level of change. By the time a manual assessment is completed, the cloud environment may already look completely different.

Notice what the five examples above have in common. In every case, the environment was compliant at some point in the past, and a single reasonable-looking decision moved it out of compliance without producing an error, an outage, or a complaint. A bucket policy was reused. A role was widened to unblock a deployment. A test environment outlived its project. Logging was disabled to control cost. An account was opened outside the security boundary.

None of these would be caught by a quarterly review, because by the time a reviewer arrives, the change is months old and looks like part of the furniture. Automation has become essential. Continuous monitoring allows organisations to identify new risks immediately instead of discovering them weeks or months later.

How Cloud Security Posture Management Strengthens Cloud Security

Cloud security posture management provides continuous visibility into cloud environments by automatically evaluating resources against security best practices and organisational policies.

Rather than replacing existing security tools, it complements them by focusing on configuration security, governance, compliance, and risk reduction.

A modern cloud security posture management platform can discover cloud assets, monitor configuration changes, identify excessive permissions, detect publicly exposed resources, validate compliance requirements, and prioritise security findings based on business impact.

Many platforms also support automated remediation. Instead of simply reporting an issue, they can trigger workflows that correct insecure configurations or notify the appropriate teams before vulnerabilities are exploited.

A word of caution on automated remediation: start it in report-only mode. A rule that automatically closes public access is exactly right for a storage bucket holding internal exports and exactly wrong for one serving a public website's static assets. Run new rules in observation mode for a few weeks, review what they would have changed, and only then let them act , beginning with the resource types where a false positive is cheap.

This proactive approach allows organisations to reduce risk without slowing down software development.

Building a Stronger Cloud Security Strategy

Technology alone can't eliminate cloud security risks. Organisations must also establish strong governance processes and promote security awareness across development, operations, and security teams.

Security should become part of the software development lifecycle rather than an activity performed after deployment. Infrastructure templates should follow secure defaults, access permissions should be reviewed regularly, and cloud resources should be continuously monitored throughout their lifecycle.

If you're starting from scratch, the sequence that tends to work is: inventory first, since you can't secure what you can't see; then public exposure, because that's where the shortest path from mistake to breach runs; then identity, because over-permissioned roles determine how far an attacker gets once inside; and finally compliance mapping, which is far easier once the first three are under control.

Regular security training also helps reduce human error. Developers who understand cloud security best practices are less likely to introduce risky configurations during deployment. It's worth noting that in every example in this article, the person who introduced the risk was doing their job competently under normal constraints. Training works best when it's framed that way, and when the secure path is also the convenient one.

Combining secure development practices with cloud security posture management creates multiple layers of protection that significantly reduce the likelihood of security incidents.

Conclusion

Cloud infrastructure offers tremendous flexibility, but it also introduces risks that are often difficult to detect through traditional security practices. Misconfigured resources, excessive permissions, forgotten assets, compliance drift, and fragmented visibility can quietly weaken an organisation's security posture without triggering immediate alarms.

As cloud environments continue to grow in size and complexity, continuous monitoring has become a necessity rather than a luxury. Cloud security posture management enables organisations to identify vulnerabilities, maintain compliance, and respond to security risks before they lead to costly breaches.

The organisations that succeed in the cloud aren't those that eliminate every possible risk. They're the ones who continuously understand their environment, detect changes as they happen, and act quickly to maintain a secure cloud infrastructure.

In a landscape where new resources are created every day, maintaining visibility into your cloud security posture is one of the most effective ways to stay ahead of evolving threats.

Hope you enjoyed this article. You can connect with me on LinkedIn.

In my previous guide, How OAuth 2.0 Works: A Practical Guide for Backend Developers, I briefly touched on the mysterious state parameter and noted that its core purpose is protecting authorization flows against CSRF attacks.

At the time, we treated CSRF as a quick prerequisite concept. Today, we're taking a much deeper dive.

Perhaps you were building a REST API in Spring Boot, ran into unexpected HTTP 403 Forbidden errors on every POST request, and "fixed" it by adding .csrf(csrf -> csrf.disable()) to your Security Filter Chain.

Most tutorials treat CSRF as a checkbox item or a framework toggle. They immediately jump to code:

// What most tutorials show on line 1:
http.csrf(Customizer.withDefaults());

Starting with framework configuration hides how web security actually operates. Spring Security doesn't invent security rules out of thin air. It responds to the fundamental mechanics of web browsers, HTTP protocols, and cookies.

In this handbook, we'll take a bottom-up, first-principles approach. We won't talk about Spring Security until we've thoroughly explored browsers, HTTP headers, session management, and the underlying mechanics of Cross-Site Request Forgery.

By the end of this guide, you'll understand:

• Why browsers automatically attach credentials to outgoing requests.

• Why that automatic behavior creates a fundamental vulnerability.

• Why attackers never need to steal or read your cookies to exploit CSRF.

• Why Same Origin Policy (SOP) and CORS don't prevent CSRF.

• How modern defenses, from CSRF Tokens to SameSite cookies, work under the hood.

• How Spring Security implements these defenses internally and how to configure them effectively.

Let’s begin by stripping away frameworks and looking at how the web actually works.

Table of Contents

• The Problem Before CSRF

• Why Browsers Automatically Send Cookies

• When Automatic Cookies Become Dangerous

• Visualize the Attack

• Why the Browser Isn't Broken

• Same Origin Policy (SOP)

• Why CORS Does NOT Prevent CSRF

• Safe Methods and State Mutation

• CSRF Tokens (Synchronizer Token Pattern)

• Double Submit Cookie Pattern

• SameSite Cookies

• Origin and Referer Headers

• JWT and CSRF: The Token Storage Dilemma

• Spring Security CSRF Internals

• Implement CSRF Protection Yourself

• Testing CSRF Protections

• Common Misconceptions

• Production Best Practices Checklist

• Final Summary & Defense Matrix

The Problem Before CSRF

To understand security, we must first understand state.

The Hypertext Transfer Protocol (HTTP) is inherently stateless. This means that if Alice sends an HTTP request to travelbuddy.com (our example) at 10:00 AM, and sends another HTTP request to travelbuddy.com at 10:01 AM, the server treats those two requests as completely isolated, unrelated events.

Without a mechanism to remember Alice between requests, Alice would have to send her username and password inside every single HTTP request she makes. That would be horrific for both user experience and performance.

Before session mechanisms were standard, developers tried passing credentials via query parameters or basic authentication headers on every click. This led to credential exposure in server logs, browser histories, and URL shares.

How Do Sessions and Cookies Solve This?

To solve this, web engineers introduced the concept of Server-Side Sessions and HTTP Cookies.

When Alice logs into TravelBuddy by sending her username and password via a POST request to https://travelbuddy.com/login, the server verifies her credentials. Instead of asking Alice to log in again on the next page, the server creates a Session in its memory (or in a database/Redis cache) and assigns it a unique, unpredictable identifier: a Session ID.

The server then sends this Session ID back to Alice’s browser using a special HTTP response header: Set-Cookie.

HTTP/1.1 200 OK
Content-Type: text/html
Set-Cookie: JSESSIONID=abc123xyz789; Path=/; Secure; HttpOnly

When Alice’s browser receives this response, it sees the Set-Cookie header. It extracts JSESSIONID=abc123xyz789 and stores it inside its internal storage unit: the Browser Cookie Jar.

Now, Alice is "logged in". The server remembers her via that session record, and the browser holds the key (JSESSIONID) to that session.

Why Browsers Automatically Send Cookies

Now we arrive at the pivotal design choice made in the early days of the web.

Once the browser stores JSESSIONID=abc123xyz789 in its cookie jar for the domain travelbuddy.com, how does that cookie get sent back to the server on subsequent requests?

Does the developer have to write custom JavaScript to attach the cookie? No.

Browsers are explicitly designed to handle cookie management automatically.

The Request Lifecycle and Automatic Cookie Attachment

Every time Alice's browser prepares an HTTP request to https://travelbuddy.com (whether caused by Alice clicking a link, submitting an HTML form, or JavaScript triggering a fetch() call), the browser follows this exact process:

1. URL Inspection: The browser examines the destination URL (for example, https://travelbuddy.com/api/connections).

2. Cookie Jar Lookup: The browser scans its cookie jar for any stored cookies whose domain and path match travelbuddy.com.

3. Validation Check: It verifies if the cookie has expired, and if flags like Secure (requires HTTPS) are respected.

4. Header Injection: If valid cookies match, the browser automatically injects a Cookie header into the outgoing HTTP request payload.

Here's what the outgoing request looks like as it leaves Alice's machine:

POST /api/connections/add HTTP/1.1
Host: travelbuddy.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
Accept: text/html,application/xhtml+xml
Cookie: JSESSIONID=abc123xyz789
Content-Type: application/x-www-form-urlencoded

service=SkyScanner

Notice something critical: Neither Alice nor any custom frontend JavaScript explicitly attached Cookie: JSESSIONID=abc123xyz789.

The browser's internal engine attached it automatically before sending the byte stream across the network. From the server's perspective, receiving Cookie: JSESSIONID=abc123xyz789 is proof that the request originated from an authenticated session belonging to Alice.

This automatic behavior is convenient. It makes web browsing seamless across page reloads and link navigation. But as we'll soon see, this convenience leaves a backdoor wide open.

When Automatic Cookies Become Dangerous

Is automatic cookie inclusion a vulnerability by itself?

No. If Alice only visits travelbuddy.com, automatic cookie inclusion works exactly as intended.

The vulnerability emerges because of a simple web reality: Alice visits multiple websites in the same browser session.

Enter evil.com

Suppose Alice is logged into TravelBuddy in Tab 1. Her session cookie (JSESSIONID=abc123xyz789) sits safely inside her browser's cookie jar for travelbuddy.com.

In Tab 2, Alice visits an unrelated website: https://evil.com (perhaps she clicked a link in a phishing email or a forum post).

evil.com is controlled by an attacker. The attacker knows that TravelBuddy has a feature located at POST [https://travelbuddy.com/api/connections/add that connects third-party services. The attacker wants to trick Alice into connecting the attacker's malicious service to her account.

The attacker embeds the following hidden HTML form inside the HTML page served by evil.com:

<!-- Hosted on https://evil.com/win-a-car.html -->
<!DOCTYPE html>
<html>
<body>
  <h1>You won a free trip! Click below to claim.</h1>
  
  <!-- Hidden Form targeting TravelBuddy -->
  <form id="maliciousForm" action="https://travelbuddy.com/api/connections/add" method="POST">
    <input type="hidden" name="service" value="MaliciousAttackerService" />
  </form>

  <script>
    // Automatically submit the form as soon as the page loads
    document.getElementById('maliciousForm').submit();
  </script>
</body>
</html>

Walkthrough of the Attack Execution

Let's trace step-by-step what happens when Alice opens https://evil.com/win-a-car.html:

1. Alice's browser fetches and parses HTML from evil.com.

2. The browser encounters the <script> tag and executes document.getElementById('maliciousForm').submit().

3. The browser prepares an outgoing POST request targeting https://travelbuddy.com/api/connections/add.

4. The browser looks at the target destination: travelbuddy.com.

5. The browser checks its Cookie Jar: "Do I have any active cookies for travelbuddy.com?"

6. Yes! It finds JSESSIONID=abc123xyz789 (Alice's active session cookie from Tab 1).

7. The browser automatically injects Cookie: JSESSIONID=abc123xyz789 into the outgoing request payload heading to travelbuddy.com.

8. The request lands on the TravelBuddy Spring Boot backend server.

The Server's Perspective

Here's what the TravelBuddy backend sees when processing the request:

POST /api/connections/add HTTP/1.1
Host: travelbuddy.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID=abc123xyz789

service=MaliciousAttackerService

The TravelBuddy server checks the Cookie header. It validates JSESSIONID=abc123xyz789 against its session store. The session is valid: it belongs to Alice!

The server assumes: "Alice sent a POST request to add MaliciousAttackerService. She is authenticated, so I will grant this request."

The server updates Alice's account state. MaliciousAttackerService is now connected to her profile.

The Core Realization of CSRF

Take a step back and examine what just happened:

1. The attacker NEVER saw or stole Alice’s session cookie. The attacker on evil.com can't read cookies belonging to travelbuddy.com due to browser isolation rules.

2. The attacker did NOT break encryption. HTTPS was active the entire time.

3. The attacker simply induced Alice's browser to make a request. The browser, faithfully executing its automatic cookie attachment rules, provided the credentials on behalf of the attacker. You could say the attacker got caught with their hand in Alice's cookie jar!

This is Cross-Site Request Forgery in action: An attacker tricks a victim's browser into executing an unwanted, state-changing HTTP request to a trusted site where the victim is currently authenticated.

Visualize the Attack

Visualizing the interaction between Alice, the browser, evil.com, and TravelBuddy makes the underlying request flow clear.

1. The Complete CSRF Sequence

The attack unfolds across three distinct phases involving four main actors: Alice, her web browser, the TravelBuddy backend server, and the attacker site running on evil.com.

In the first phase, Alice authenticates with TravelBuddy. She submits her login credentials through her browser, which sends a POST request to the TravelBuddy backend. The backend verifies her credentials and responds with an HTTP 200 OK status alongside a Set-Cookie header containing JSESSIONID=abc123xyz.

Upon receiving this response, Alice's browser automatically saves this session identifier inside its cookie jar for the travelbuddy.com domain.

In the second phase, the attacker sets a trap. While keeping her TravelBuddy tab active, Alice opens a second browser tab and visits evil.com. Her browser requests the page win-a-car.html from evil.com. In response, evil.com serves an HTML document containing an invisible form targeting TravelBuddy, paired with an embedded JavaScript script designed to trigger immediately upon loading.

In the final phase, the attack executes automatically. The malicious JavaScript on evil.com calls form.submit(), commanding the browser to send a POST request to https://travelbuddy.com/api/connections/add.

Before sending the request across the network, the browser checks its cookie jar for any cookies matching travelbuddy.com. It finds Alice's active session cookie and automatically attaches Cookie: JSESSIONID=abc123xyz to the outgoing request payload. The TravelBuddy server receives the request, inspects the valid session cookie, assumes Alice intended to perform this action, and attaches the attacker's service to her account.

2. Browser Decision Tree during Outgoing Request

When any request is fired, the browser follows a decision path regarding cookie attachment:

This diagram outlines the automatic evaluation loop executed by a browser whenever an HTTP request is triggered from any tab or script.

The process begins as soon as an outgoing HTTP request is initiated. The browser first inspects the target URL to extract the destination domain, such as travelbuddy.com. Once the domain is identified, the browser queries its internal cookie storage to check whether any cookies are mapped to that target domain. If no matching cookies exist, the browser immediately skips credential attachment and dispatches the raw HTTP request across the network.

If matching cookies are found, the browser evaluates their validity. It checks whether the cookies have expired, whether the request path matches the path defined in the cookie, and whether security constraints like the Secure HTTPS flag are satisfied. If any validation check fails, the cookie is discarded, and the request proceeds without credentials. But if the cookies are valid and active, the browser constructs a Cookie header containing the stored session key and attaches it to the outgoing HTTP request payload before dispatching it across the network to the server.

3. Session and Cookie Lifecycle State Diagram

This state diagram tracks how a user moves between secure, authenticated, and vulnerable conditions during a web session.

When a user first opens their web browser, they begin in an unauthenticated state with no cookies stored for the target application. Submitting valid credentials via a login form transitions the user into an authenticated state. Inside this authenticated state, the server issues a Set-Cookie header, causing the browser to save the session ID in its cookie storage. For every subsequent request directed to that application, the browser automatically attaches the cookie while keeping the user logged in.

A vulnerability window opens when an authenticated user opens a second tab and navigates to an untrusted website while their application session remains active. This action shifts the browser context into a state vulnerable to Cross-Site Request Forgery. If the untrusted site fires a cross-site request back to the original application, the browser's automatic cookie attachment mechanism triggers, executing an unauthorized state change on the server. The cycle ends only when the user logs out or the server session expires, returning the client to the initial unauthenticated state.

Why the Browser Isn't Broken

When developers first grasp CSRF, their immediate reaction is often: "This is a terrible browser flaw! Why don't browser vendors fix this by disabling automatic cookie sending entirely?"

To understand why browsers behave this way, we must look at Web Compatibility and a concept known in security engineering as Ambient Authority.

The Principle of Ambient Authority

When a system automatically applies a user's identity or credentials to every action without requiring explicit user intent for that specific action, the system is using ambient authority.

HTTP cookies are an ambient credential. If you're logged in, every request carrying a destination URL automatically includes your credential.

Why Browser Vendors Don't Just "Fix" It

The World Wide Web was created as a web of interconnected hypermedia documents. Cross-site interactions are a fundamental design feature of the web, not an accidental bug:

• Images and assets: When news.com embeds an image hosted on cdn.com, your browser makes a cross-site request to cdn.com.

• Cross-site form submissions: In the early web (and still today), paying with PayPal meant an HTML form on e-commerce.com submitted data directly to paypal.com.

• Hyperlinks: Clicking a link on google.com takes you to wikipedia.org via a cross-site GET request.

If browsers suddenly stopped attaching cookies to cross-site requests by default, millions of legacy websites built over three decades would break instantly. Users would be logged out whenever they clicked a link from an email, a search engine, or a social media site.

Browser vendors prioritize backward compatibility. Rather than removing cross-site capabilities, they introduced configurable security boundaries that developers can opt into.

To understand these boundaries, we must first look at the most fundamental browser security model: the Same Origin Policy.

Same Origin Policy (SOP)

Many developers assume: "Doesn't the Same Origin Policy block cross-site requests?"

This is one of the most common misunderstandings in web development. Let's clarify what the Same Origin Policy actually is and what it does.

Defining an Origin

An Origin in web security is defined by three components:

1. Scheme (Protocol, for example, http vs https)

2. Host (Domain, for example, travelbuddy.com)

3. Port (for example, :80, :443, :8080)

Two URLs have the Same Origin if and only if all three components match exactly.

What SOP Protects vs. What SOP Allows

The Same Origin Policy governs how scripts running on one origin can interact with resources on another origin.

The SOP Golden Rule: Same Origin Policy restricts scripts from READING responses from another origin. Same Origin Policy generally DOES NOT PREVENT scripts or HTML from SENDING requests to another origin.

Let's emphasize this distinction:

Sending a request: evil.com can create an HTML form like this: <form action="https://travelbuddy.com/api/delete" method="POST">. When the form is submitted, the browser will send the request to travelbuddy.com. The backend will process the request and mutate the database state.

Reading the response: JavaScript running on evil.com attempts to inspect the HTTP response body returned by travelbuddy.com. The browser blocks JavaScript from reading that data because evil.com and travelbuddy.com are different origins.

Notice the flaw relative to CSRF: CSRF is an attack on state mutation, not data retrieval.

The attacker on evil.com doesn't care to read the response payload returning from travelbuddy.com. Their goal was simply to trigger the action on the server. Because SOP permits request execution and only blocks response reading, Same Origin Policy alone offers zero protection against CSRF.

Why CORS Does NOT Prevent CSRF

This brings us to another major source of confusion: Cross-Origin Resource Sharing (CORS).

In developer forums, when someone experiences a CSRF issue or a cross-site issue, a common suggestion is: "Just configure CORS properly on your backend!"

Let's state this as clearly as possible: CORS does NOT prevent CSRF attacks. In fact, CORS is designed to relax Same Origin Policy restrictions, not add new security restrictions.

Reading vs. Sending Revisited

Remember: SOP blocks cross-origin reading by default.

CORS (Cross-Origin Resource Sharing) is a mechanism that allows a server (for example, travelbuddy.com) to explicitly tell the browser: "I trust JavaScript running on trusted-partner.com. You may allow trusted-partner.com to read my responses."

CORS is an opt-in mechanism to allow cross-origin reading. Disabling or improperly configuring CORS doesn't stop a browser from sending a forged request.

Simple Requests vs. Preflighted Requests

To understand why CORS fails to stop CSRF, we must examine how browsers handle cross-origin HTTP requests under CORS rules. Browsers divide cross-origin requests into two categories:

1. Simple Requests

2. Preflighted Requests

1. Simple Requests

A request is considered a Simple Request if it satisfies all of the following:

• Uses HTTP methods: GET, HEAD, or POST.

• Uses standard browser Content-Types: application/x-www-form-urlencoded, multipart/form-data, or text/plain.

• Doesn't set custom HTTP headers (like X-Requested-With or Authorization).

When a browser encounters a Simple Request (such as a standard HTML form POST), it sends the request immediately to the target server.

As the diagram shows, the server executes the SQL UPDATE or INSERT statement the moment the request arrives. By the time the browser evaluates CORS headers on the returning response, the state mutation on the server has already happened.

2. Preflighted Requests

If a request uses non-standard methods (PUT, DELETE) or non-standard content types (application/json), or custom headers, the browser first sends an OPTIONS request called a Preflight Request.

Because OPTIONS preflight requests don't carry side-effects and are checked before sending the actual request, CORS incidentally stops cross-origin JSON requests from unapproved domains.

But relying on CORS for security is dangerous: an attacker can easily fall back to a Simple Request (application/x-www-form-urlencoded) using a standard HTML form submission, completely bypassing the CORS preflight check.

Safe Methods and State Mutation

Before we dive into effective defenses, we must address an architectural concept defined in HTTP specifications (RFC 9110): Safe Methods and Idempotency.

HTTP methods are categorized based on their intended impact on server state:

• Safe Methods (GET, HEAD, OPTIONS, TRACE): These methods are defined as read-only operations. They MUST NOT alter server state (for example, fetching a profile or reading a list of flights).

• Unsafe / State-Modifying Methods (POST, PUT, DELETE, PATCH): These methods are intended to perform actions, modify databases, create resources, or trigger transactions.

The Developer Crime: State-Changing GET Requests

Consider what happens if a junior developer on the TravelBuddy team writes code like this:

// ❌ DANGEROUS CODE: State mutation via GET request
@GetMapping("/api/connections/delete")
public String deleteConnection(@RequestParam String serviceId, HttpSession session) {
    User user = (User) session.getAttribute("user");
    connectionService.deleteForUser(user, serviceId);
    return "redirect:/dashboard";
}

Why is this an architectural error and a massive security vulnerability?

Because an attacker on evil.com doesn't even need an HTML form or JavaScript to trigger a GET request. They can trigger a GET request using simple HTML element tags:

<!-- Hosted on evil.com -->
<img src="https://travelbuddy.com/api/connections/delete?serviceId=SkyScanner" width="0" height="0" />

When Alice's browser parses the HTML from evil.com, it encounters the <img> tag. To render the page, the browser automatically sends a GET request to https://travelbuddy.com/api/connections/delete?serviceId=SkyScanner, automatically attaching Alice's session cookie.

The backend receives the GET request, executes connectionService.deleteForUser(...), and wipes Alice's integration!

Rule #1 of Web Security

GET requests MUST ALWAYS be safe and read-only. Never perform state mutations (creates, updates, deletes) inside a GET handler.

Enforcing safe GET requests is the foundation of web security. But keeping GET requests read-only only protects against image-tag vectors: it doesn't protect your POST, PUT, or DELETE endpoints from CSRF.

For state-modifying requests, we need specialized defenses.

CSRF Tokens (Synchronizer Token Pattern)

Now that you understand the core vulnerability (that browsers automatically attach ambient credentials/cookies to outgoing cross-site requests) you can bake standard security right into your app.

What Problem Existed Before CSRF Tokens?

Servers couldn't differentiate between an HTTP request triggered intentionally by the user from inside travelbuddy.com's real user interface and one forged by evil.com that caused the browser to automatically attach the user's cookies.

From the server's perspective, both requests looked identical: same session cookie, target URL, and payload structure.

How Do CSRF Tokens Solve This?

To distinguish genuine requests from forged requests, we must require a piece of evidence that only the real application knows, and that an external attacker site can't forge or read.

This defense is known as the Synchronizer Token Pattern (or CSRF Token).

How the Synchronizer Token Pattern Works

1. Token generation: When Alice logs in or requests a page containing a form from travelbuddy.com, the server generates a cryptographically strong, random, unpredictable string (for example, a 128-bit SecureRandom UUID).

2. Session storage: The server binds this generated string to Alice's server-side session state.

3. Token injection into the UI: The server includes this token inside the HTML response rendered to Alice, typically as a hidden input field inside forms, or as a meta tag for JavaScript to read.

4. Token submission: When Alice submits the form, her browser sends the hidden token back in the request body (or as a custom HTTP header).

5. Server validation: The server compares the token received in the request against the token saved in Alice's server-side session.

   • If the tokens match: Request is Genuine. Process it.

   • If the tokens don't match (or the token is missing): Request is Forged. Reject with HTTP 403 Forbidden!

HTML Form Example

Here is how TravelBuddy renders a protected form:

<!-- Rendered by TravelBuddy at https://travelbuddy.com/connect-service -->
<form action="/api/connections/add" method="POST">
  <!-- Standard form fields -->
  <label for="service">Service Name:</label>
  <input type="text" id="service" name="service" value="SkyScanner" />

  <!-- Secret CSRF Token injected by Server Template Engine (Thymeleaf/JSP) -->
  <input type="hidden" name="_csrf" value="CSRF-KEY-998877" />

  <button type="submit">Submit</button>
</form>

When submitted, the raw HTTP request looks like this:

POST /api/connections/add HTTP/1.1
Host: travelbuddy.com
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID=abc123xyz789

service=SkyScanner&_csrf=CSRF-KEY-998877

Why Attackers Can't Forge the CSRF Token

Now let's trace what happens when evil.com tries to forge this request:

1. evil.com builds an auto-submitting form targeting https://travelbuddy.com/api/connections/add.

2. To succeed, evil.com must include _csrf=CSRF-KEY-998877 in its form payload.

3. How can evil.com get CSRF-KEY-998877?

   • Can evil.com guess it? No. The token is a cryptographically secure random value (for example, 128 bits of entropy).

   • Can evil.com make an AJAX GET request to travelbuddy.com to read the HTML form and extract the token? No! Because Same Origin Policy (SOP) blocks evil.com JavaScript from reading the response contents of travelbuddy.com.

Because the attacker can't read the page from travelbuddy.com, they can't extract the valid token. When evil.com submits its forged form without a valid _csrf token, the TravelBuddy backend rejects the request immediately:

HTTP/1.1 403 Forbidden
Content-Type: application/json

{
  "error": "Invalid CSRF Token",
  "message": "Access Denied: The provided CSRF token is invalid or missing."
}

Double Submit Cookie Pattern

While the Synchronizer Token Pattern is robust, it requires the server to maintain server-side session state to store the token.

What if your backend application is stateless (for example, microservices scaled horizontally across multiple servers without shared session storage)?

Enter the Double Submit Cookie Pattern.

How Double Submit Cookie Works

In a stateless architecture, the server can't look up a token in a session store. Instead, it relies on cryptographic and domain-isolation properties:

1. Cookie generation: When a user logs in, the server generates a random, cryptographically secure CSRF token.

2. Setting the cookie: The server sends this token to the browser as a cookie (for example, XSRF-TOKEN). Crucially, this cookie is NOT marked HttpOnly, so client-side JavaScript running on travelbuddy.com can read it.

3. Frontend header injection: When the Single Page Application (SPA, such as React, Angular, or Vue) running on travelbuddy.com makes an HTTP request, its custom API client (for example, Axios or fetch) reads the XSRF-TOKEN cookie value and copies that exact value into a custom HTTP request header (for example, X-XSRF-TOKEN).

4. Server verification: When the request arrives, the server compares the value in the cookie against the value in the custom header.

If Cookie Value == Header Value, the request is valid.

Why Double Submit Cookie Works against Cross-Site Attackers

Suppose Alice visits evil.com:

1. evil.com triggers a cross-site request to travelbuddy.com.

2. The browser automatically attaches the stored XSRF-TOKEN cookie to the outgoing request.

3. But evil.com must also set the custom header X-XSRF-TOKEN with a matching value.

4. Can evil.com read the XSRF-TOKEN cookie to copy its value into the header? No! Browsers strictly prevent evil.com from reading cookies set by travelbuddy.com.

5. Can evil.com write custom headers on a cross-site request? No! Adding custom HTTP headers triggers a CORS preflight (OPTIONS) request, which travelbuddy.com will reject for evil.com.

Since evil.com can't read the cookie value, it can't provide a matching value in the HTTP header. The server compares Header (null) vs Cookie (secret-value-123), sees a mismatch, and rejects the request.

SameSite Cookies

For over two decades, developers relied entirely on CSRF tokens. Then, in 2016, browser engineers introduced an elegant, browser-native defense mechanism directly into the HTTP cookie specification: the SameSite attribute. This defense really takes the biscuit when it comes to simplicity.

What Problem Existed Before SameSite?

Cookies were strictly cross-site by default. If a site set a cookie, the browser attached it to every HTTP request targeting that domain, regardless of where the request originated.

How SameSite Solves This

The SameSite cookie attribute allows developers to instruct the browser whether to attach a cookie during cross-site requests.

Syntax in HTTP response:

Set-Cookie: JSESSIONID=abc123xyz789; Path=/; Secure; HttpOnly; SameSite=Lax

SameSite accepts three values: Strict, Lax, and None.

Deep Dive into SameSite Modes

1. SameSite=Strict

This is the most secure setting. The browser never attaches the cookie on any cross-site request.

Let's say that Alice is logged into TravelBuddy (SameSite=Strict). She clicks a link on twitter.com pointing to https://travelbuddy.com/dashboard.

Because the navigation originated from a cross-site source (twitter.com), the browser omits the JSESSIONID cookie. Alice lands on TravelBuddy appearing logged out.

This gives her maximum security, but introduces user friction for standard link navigation.

2. SameSite=Lax (Modern Browser Default)

Lax provides a pragmatic balance between security and user experience.

• Top-level navigations (GET): If Alice clicks a link on twitter.com to open https://travelbuddy.com/dashboard, the browser includes the cookie. Alice stays logged in!

• State-modifying / cross-site requests (POST, PUT, DELETE or <img> tags): If evil.com submits a cross-site POST form to travelbuddy.com, the browser blocks and strips the cookie.

/* Cross-site POST request from evil.com targeting travelbuddy.com */
POST /api/connections/add HTTP/1.1
Host: travelbuddy.com
User-Agent: Mozilla/5.0
/* Cookie header is STRIPPED by browser because SameSite=Lax! */

service=MaliciousService

Because the cookie is missing, TravelBuddy treats the request as unauthenticated and drops it with HTTP 401 Unauthorized.

3. SameSite=None

Disables SameSite restrictions entirely. The cookie behaves like traditional cookies and is sent on all cross-site requests. Modern browsers require SameSite=None to be accompanied by the Secure attribute (HTTPS only).

Is SameSite=Lax a Complete Replacement for CSRF Tokens?

Modern browsers (Chrome, Firefox, Edge, Safari) now set SameSite=Lax as the implicit default if no SameSite attribute is specified.

This doesn't mean CSRF tokens are dead. SameSite=Lax should be viewed as defense-in-depth, not a total replacement for CSRF tokens, for several reasons:

1. Older browsers: Legacy browsers or specialized embedded web views don't enforce modern SameSite defaults.

2. Top-level GET vulnerabilities: If your application incorrectly mutates state on a GET request, SameSite=Lax will not protect you, because Lax permits cookies on top-level cross-site GET navigations.

3. Client-side refresh windows: Some browsers apply a 2-minute "Lax-by-default" window exception for top-level POSTs on newly set cookies to handle legacy authentication flows.

Origin and Referer Headers

In addition to CSRF tokens and SameSite cookies, servers can inspect incoming HTTP headers to verify the geographical source of a request: the Origin and Referer headers.

Understanding the Headers

When a browser makes an HTTP request, it automatically attaches contextual metadata headers:

• Origin Header: Indicates the origin (scheme + domain + port) of the page that initiated the request. For example: Origin: https://evil.com

• Referer Header: Contains the full URL of the exact web page that initiated the request. For example: Referer: https://evil.com/win-a-car.html

Server-Side Validation Logic

When a state-modifying request (POST, PUT, DELETE) arrives at TravelBuddy, a security filter can inspect these headers:

// Conceptual Origin/Referer Checking Logic
public boolean isValidRequest(HttpServletRequest request) {
    String origin = request.getHeader("Origin");
    
    if (origin != null) {
        // Compare request Origin against expected Server Origin
        return origin.equals("https://travelbuddy.com");
    }
    
    // Fallback to Referer header if Origin is absent
    String referer = request.getHeader("Referer");
    if (referer != null) {
        return referer.startsWith("https://travelbuddy.com/");
    }
    
    // If both headers are missing, drop or handle cautiously
    return false;
}

Limitations of Origin/Referer Verification

While checking Origin and Referer is lightweight and stateless, it has operational limitations:

1. Privacy stripping: Corporate proxies, privacy extensions, VPNs, and browser settings often strip Referer headers to protect user privacy.

2. Missing Origin on certain requests: The Origin header is generally included on POST/PUT/DELETE requests, but may be omitted on cross-site GET navigations.

3. Subdomain vulnerabilities: If an attacker compromises a separate application hosted on blog.travelbuddy.com, an origin check verifying *.travelbuddy.com might accept the forged request.

JWT and CSRF: The Token Storage Dilemma

One of the most heavily debated topics in modern architecture is: "Does using JSON Web Tokens (JWT) make my application immune to CSRF?"

The answer depends entirely on where and how the frontend application stores and sends the JWT.

Let's evaluate the two primary JWT storage strategies.

Strategy A: Storing JWT in localStorage or sessionStorage

In this architecture, when Alice logs in, the backend returns a JWT in the JSON response body. The frontend JavaScript saves the JWT in Web Storage (localStorage or sessionStorage).

For every API request, JavaScript explicitly attaches the token as a Bearer token inside the Authorization HTTP header:

POST /api/connections/add HTTP/1.1
Host: travelbuddy.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
Content-Type: application/json

{"service": "SkyScanner"}

Is Strategy A Vulnerable to CSRF?

No: strategy A is completely immune to CSRF.

Why? Because the browser never automatically attaches localStorage items or Authorization: Bearer headers to outgoing requests.

If Alice visits evil.com, evil.com can send a request to travelbuddy.com. But because evil.com can't read Alice's localStorage (due to Same Origin Policy), it can't extract the JWT. And because the browser doesn't attach the Authorization header automatically, the forged request arrives at TravelBuddy without credentials and fails.

The Catch: XSS Vulnerability

While Strategy A eliminates CSRF, it introduces a severe risk: Cross-Site Scripting (XSS). Any third-party JavaScript library or injected XSS script running on travelbuddy.com can execute localStorage.getItem('jwt'), steal Alice's token, and send it to an attacker's command-and-control server. Once stolen, the token can be used from anywhere in the world.

Strategy B: Storing JWT in an HttpOnly Cookie

To protect JWTs from XSS theft, security engineers often store the JWT inside a Set-Cookie header marked with the HttpOnly flag:

Set-Cookie: jwt_token=eyJhbGciOi...; Path=/; HttpOnly; Secure; SameSite=Lax

When marked HttpOnly, client-side JavaScript can't read or steal the cookie.

Is Strategy B Vulnerable to CSRF?

Yes: strategy B is vulnerable to CSRF unless explicitly defended.

Why? Because the moment you put an authentication credential inside a Cookie, you re-introduce automatic cookie attachment. The browser treats a JWT cookie exactly like a session cookie.

If evil.com triggers a cross-site request to travelbuddy.com, the browser automatically attaches Cookie: jwt_token=eyJhbGciOi....

Summary Matrix: JWT Storage Trade-offs

OAuth State Parameter & Login CSRF

In the introduction, I mentioned that OAuth 2.0 uses a state parameter to protect against CSRF. Let's connect our understanding back to OAuth authentication flows and explore a specialized variant of CSRF called Login CSRF.

What is Login CSRF?

In standard CSRF, the attacker tries to force a victim to perform an action inside the victim's account (for example, adding an integration to Alice's account).

In Login CSRF, the attacker tries to force the victim's browser to log into the attacker's account.

How Login CSRF Works

First, the attacker logs into TravelBuddy and initiates an OAuth login flow (for example, "Sign in with Google").

Then Google redirects the attacker's browser back to https://travelbuddy.com/login/oauth2/code/google?code=ATTACKER_AUTHORIZATION_CODE.

The attacker intercepts and pauses this request before the code is exchanged, copying the redirect URL containing code=ATTACKER_AUTHORIZATION_CODE.

Next, the attacker crafts a link or malicious page on evil.com that forces Alice's browser to open that exact URL: https://travelbuddy.com/login/oauth2/code/google?code=ATTACKER_AUTHORIZATION_CODE.

Alice's browser executes the request. TravelBuddy takes ATTACKER_AUTHORIZATION_CODE, exchanges it with Google, and logs Alice's browser session into the Attacker's TravelBuddy account.

Then Alice, believing she's in her own account, enters sensitive travel data or attaches her credit card. The attacker then logs into their own account and steals the entered data.

How the OAuth state Parameter Prevents Login CSRF

To prevent Login CSRF, OAuth 2.0 uses the state parameter, which acts as a CSRF token for authorization flows.

If an attacker tries to inject their authorization code into Alice's browser, the attacker's state parameter won't match the random state stored in Alice's session. TravelBuddy rejects the callback, stopping Login CSRF.

Comparison Table: CSRF Token vs OAuth State vs PKCE

Spring Security CSRF Internals

Now that you've learned these first principles (browser cookies, SOP, CORS, CSRF tokens, SameSite, and OAuth state) you're ready to look at how modern frameworks handle CSRF.

We'll analyze Spring Security (Spring Boot 3.x / 4 architecture, using Java 21).

The Mechanics: CsrfFilter

Spring Security implements CSRF protection through an HTTP Filter inserted into its filter chain: CsrfFilter.

Spring Security CSRF Key Architecture Components

Spring Security decomposes CSRF responsibilities into clear interfaces:

1. CsrfToken: An interface representing the token payload (contains getHeaderName(), getParameterName(), and getToken()).

2. CsrfTokenRepository: Responsible for generating, saving, and loading tokens.

   • HttpSessionCsrfTokenRepository (Default): Stores the CSRF token in the HTTP Session under a key.

   • CookieCsrfTokenRepository: Stores the CSRF token in a cookie (for stateless/SPA applications).

3. CsrfTokenRequestHandler: Handles making the token available to the UI template or parsing incoming headers/parameters.

   • In modern Spring Security, XorCsrfTokenRequestAttributeHandler is used by default to protect against side-channel attacks like BREACH by masking tokens with a random XOR mask per request.

4. Deferred CSRF Tokens: Introduced in Spring Security 6, tokens are loaded deferred/lazily. Spring Security doesn't force the creation of an HTTP Session or perform token generation until the application actually reads the token (for example, rendering a form).

Modern Spring Security Configuration (Spring Boot 3.x / 4)

Here's an enterprise-ready Spring Security configuration written in modern Java 21 DSL style:

package com.travelbuddy.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.web.SecurityFilterChain;
import org.springframework.security.web.csrf.CookieCsrfTokenRepository;
import org.springframework.security.web.csrf.XorCsrfTokenRequestAttributeHandler;

@Configuration
@EnableWebSecurity
public class SecurityConfig {

    @Bean
    public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
            .authorizeHttpRequests(auth -> auth
                .requestMatchers("/public/**", "/login", "/register").permitAll()
                .anyRequest().authenticated()
            )
            .formLogin(form -> form
                .loginPage("/login")
                .defaultSuccessUrl("/dashboard", true)
            )
            // Configure CSRF explicitly using modern Lambda DSL
            .csrf(csrf -> csrf
                .csrfTokenRepository(CookieCsrfTokenRepository.withHttpOnlyFalse())
                .csrfTokenRequestHandler(new XorCsrfTokenRequestAttributeHandler())
                .ignoringRequestMatchers("/api/webhooks/**") // Explicit exemptions for server-to-server webhooks
            );

        return http.build();
    }
}

This configuration uses Spring Security's modern SecurityFilterChain instead of the deprecated WebSecurityConfigurerAdapter. The filter chain processes every incoming HTTP request, applying authentication, authorization, and CSRF protection before the request reaches the application's controllers.

The authorizeHttpRequests() method defines the authorization rules. Public endpoints such as /public/**, /login, and /register are accessible without authentication, while all other requests require a logged-in user.

CSRF protection is enabled using CookieCsrfTokenRepository.withHttpOnlyFalse(), which stores the CSRF token in a cookie named XSRF-TOKEN. Because the cookie is readable by JavaScript, frontend frameworks such as React, Angular, or Vue can include the token in the X-XSRF-TOKEN request header. Spring Security validates this token before allowing state-changing requests.

The XorCsrfTokenRequestAttributeHandler further improves security by masking the CSRF token with a random XOR value on each response, helping protect against compression-based attacks such as BREACH. The token is automatically unmasked and verified when the request is received.

Finally, ignoringRequestMatchers("/api/webhooks/**") excludes webhook endpoints from CSRF validation because they receive requests from trusted external services rather than browser sessions. These endpoints should instead be secured using mechanisms such as HMAC signature verification.

Implement CSRF Protection Yourself

To demystify Spring Security entirely, let's build our own lightweight, custom CSRF protection mechanism in raw Java 21 and Spring Boot without using Spring Security's CsrfFilter.

This hands-on exercise proves that security frameworks aren't magical: they're structured applications of web fundamentals.

Step 1: Create a Custom CSRF Filter

package com.travelbuddy.security;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import jakarta.servlet.http.HttpSession;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

import java.io.IOException;
import java.security.SecureRandom;
import java.util.Base64;
import java.util.Set;

@Component
public class CustomCsrfFilter extends OncePerRequestFilter {

    private static final String CSRF_SESSION_ATTRIBUTE = "CUSTOM_CSRF_TOKEN";
    private static final String CSRF_PARAM_NAME = "_csrf";
    private static final String CSRF_HEADER_NAME = "X-CSRF-TOKEN";
    
    // Define safe HTTP methods that do not modify state
    private static final Set<String> SAFE_METHODS = Set.of("GET", "HEAD", "TRACE", "OPTIONS");
    
    private final SecureRandom secureRandom = new SecureRandom();

    @Override
    protected void doFilterInternal(HttpServletRequest request, 
                                    HttpServletResponse response, 
                                    FilterChain filterChain) throws ServletException, IOException {

        HttpSession session = request.getSession(true);

        // 1. Ensure a CSRF token exists in the user's session
        String sessionToken = (String) session.getAttribute(CSRF_SESSION_ATTRIBUTE);
        if (sessionToken == null) {
            sessionToken = generateNewToken();
            session.setAttribute(CSRF_SESSION_ATTRIBUTE, sessionToken);
        }

        // Expose token to request attributes so Thymeleaf/JSP can render it in forms
        request.setAttribute("csrfToken", sessionToken);

        // 2. Check if the incoming request method is SAFE
        if (SAFE_METHODS.contains(request.getMethod())) {
            // Safe request: Allow execution to proceed
            filterChain.doFilter(request, response);
            return;
        }

        // 3. Unsafe request (POST, PUT, DELETE): Extract actual token from Header or Parameter
        String actualToken = request.getHeader(CSRF_HEADER_NAME);
        if (actualToken == null || actualToken.isBlank()) {
            actualToken = request.getParameter(CSRF_PARAM_NAME);
        }

        // 4. Validate Token
        if (actualToken != null && actualToken.equals(sessionToken)) {
            // Token matches! Proceed to controller handler
            filterChain.doFilter(request, response);
        } else {
            // Token missing or mismatched! Reject forged request
            response.setStatus(HttpServletResponse.SC_FORBIDDEN);
            response.setContentType("application/json");
            response.getWriter().write("""
                {
                    "error": "Forbidden",
                    "message": "Custom CSRF Filter: Invalid or missing CSRF token."
                }
                """);
        }
    }

    private String generateNewToken() {
        byte[] randomBytes = new byte[32];
        secureRandom.nextBytes(randomBytes);
        return Base64.getUrlEncoder().withoutPadding().encodeToString(randomBytes);
    }
}

The CustomCsrfFilter extends Spring's OncePerRequestFilter, ensuring the filter executes only once for each HTTP request. When a request arrives, it checks the user's session for a CSRF token. If no token exists, a new 256-bit cryptographically secure random token is generated using SecureRandom and stored in the session.

The filter then exposes the token as a request attribute using request.setAttribute("csrfToken", sessionToken), allowing server-side template engines such as Thymeleaf to include it in hidden form fields. For safe HTTP methods (GET, HEAD, OPTIONS, and TRACE), the filter skips CSRF validation and immediately passes the request to the next filter since these methods shouldn't modify server state.

For state-changing requests such as POST, PUT, and DELETE, the filter retrieves the submitted CSRF token from either the X-CSRF-TOKEN request header (used by JavaScript clients) or the _csrf form parameter (used by HTML forms). It then compares this value with the token stored in the user's session. If the tokens match, the request proceeds normally. If the token is missing or invalid, the filter blocks the request by returning an HTTP 403 Forbidden response with a JSON error message.

Step 2: Register the Custom Filter

package com.travelbuddy.config;

import com.travelbuddy.security.CustomCsrfFilter;
import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class WebFilterConfig {

    @Bean
    public FilterRegistrationBean<CustomCsrfFilter> loggingFilter(CustomCsrfFilter filter) {
        FilterRegistrationBean<CustomCsrfFilter> registrationBean = new FilterRegistrationBean<>();
        registrationBean.setFilter(filter);
        registrationBean.addUrlPatterns("/api/*"); // Protect API endpoints
        return registrationBean;
    }
}

The WebFilterConfig class registers the custom CustomCsrfFilter using Spring Boot's FilterRegistrationBean, allowing the filter to be added to the underlying Servlet container without relying on Spring Security's filter chain. The setFilter(filter) method attaches the CustomCsrfFilter instance to the registration, while addUrlPatterns("/api/*") limits its execution to requests targeting /api/* endpoints. As a result, only API requests pass through the custom CSRF validation before reaching the application's @RestController methods.

Compare Custom Filter vs. Spring Security's CsrfFilter

Building this filter manually shows that Spring Security isn't magic. It performs the exact steps we built: checking HTTP methods, extracting tokens, and comparing request attributes against stored session state.

Testing CSRF Protections

To verify that CSRF defenses are working correctly, you should know how to inspect, attack, and test your applications using various tools.

1. Browser DevTools Inspection

Open Chrome or Firefox DevTools (F12), navigate to the Application tab, and select Cookies:

• Inspect JSESSIONID: Verify that HttpOnly and Secure flags are set.

• Inspect SameSite column: Verify whether Lax or Strict is active.

In the Network tab, inspect a submitted POST request payload:

• Look for _csrf under Form Data, or X-XSRF-TOKEN under Request Headers.

2. Testing via curl

Let's attempt a forged request using command-line curl.

Test Attempt A: Submit POST without CSRF Token (Simulating Attacker)

curl -i -X POST https://travelbuddy.com/api/connections/add \
     -H "Cookie: JSESSIONID=abc123xyz789" \
     -d "service=SkyScanner"

Expected Response:

HTTP/1.1 403 Forbidden
Content-Type: application/json

{"error":"Forbidden","message":"Invalid CSRF Token"}

Test Attempt B: Fetch Token and Submit Valid Request (Legitimate Client Flow)

# Step 1: Fetch session cookie and CSRF token from page
curl -i -c cookies.txt https://travelbuddy.com/connect-service

# Step 2: Extract token value from HTML, then submit POST request with Cookie + Token
curl -i -b cookies.txt -X POST https://travelbuddy.com/api/connections/add \
     -H "X-CSRF-TOKEN: CSRF-KEY-998877" \
     -d "service=SkyScanner"

Expected Response:

HTTP/1.1 200 OK
Content-Type: application/json

{"status":"success","message":"Service connected successfully"}

3. Why Postman Can Mislead Developers

Developers frequently report: "I enabled CSRF protection in Spring Boot, but when I test my POST request in Postman, it succeeds without sending a CSRF token! Why?"

Postman is an API client, not a web browser. When you run a request in Postman, Postman doesn't maintain a cross-site sandbox, nor does it enforce Same Origin Policy or automatic ambient cookie injection unless explicitly configured.

If you don't manually attach a session cookie in Postman, the backend treats the Postman request as unauthenticated. If you use Postman's Interceptor cookie sync, Postman acts like a client explicitly sending parameters. Postman tests API contracts, but it doesn't simulate the browser's ambient authorization rules.

4. Automated Integration Testing with Spring Security Test

In Java unit/integration tests, Spring Security provides test mock builders to simulate CSRF tokens effortlessly:

package com.travelbuddy.controller;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.security.test.context.support.WithMockUser;
import org.springframework.test.web.servlet.MockMvc;

import static org.springframework.security.test.web.servlet.request.SecurityMockMvcRequestPostProcessors.csrf;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@SpringBootTest
@AutoConfigureMockMvc
class ConnectionControllerTest {

    @Autowired
    private MockMvc mockMvc;

    @Test
    @WithMockUser(username = "alice")
    void addConnection_WithoutCsrf_ShouldReturn403Forbidden() throws Exception {
        mockMvc.perform(post("/api/connections/add")
                .param("service", "SkyScanner"))
                .andExpect(status().isForbidden());
    }

    @Test
    @WithMockUser(username = "alice")
    void addConnection_WithCsrf_ShouldSucceed() throws Exception {
        mockMvc.perform(post("/api/connections/add")
                .param("service", "SkyScanner")
                .with(csrf())) // Injects a valid mock CSRF token into request
                .andExpect(status().isOk());
    }
}

Common Misconceptions

Let's dispel the seven most persistent myths surrounding CSRF.

Myth 1: "CSRF and XSS are the same thing."

Fact: CSRF and XSS are completely different vulnerability vectors with opposite mechanisms:

• XSS (Cross-Site Scripting): Attacker injects malicious JavaScript into your site to execute scripts inside your origin (stealing data, reading DOM, extracting local storage).

• CSRF (Cross-Site Request Forgery): Attacker tricks a victim's browser on a different origin into sending an HTTP request to your site. The attacker cannot read your site's DOM or steal cookies.

Myth 2: "HTTPS prevents CSRF attacks."

Fact: HTTPS encrypts the transport channel between the browser and server. It prevents wiretapping and man-in-the-middle attacks. But in a CSRF attack, the browser itself sends encrypted, valid HTTPS requests. Encrypting the pipe doesn't stop the browser from sending a forged request down that pipe.

Myth 3: "Our app requires authentication, so we're safe from CSRF."

Fact: Authentication is what enables CSRF. CSRF specifically targets authenticated users because the browser automatically attaches their authenticated session cookies.

Myth 4: "Our API uses JWTs, so we don't have to worry about CSRF."

Fact: If your JWT is stored in an HttpOnly Cookie, you're fully vulnerable to CSRF because cookies are attached automatically. CSRF is a function of credential transmission mechanism (cookies), not credential payload structure (JWT vs Session ID).

Myth 5: "CORS blocks cross-site attacks."

Fact: CORS controls response reading, not request execution. Simple requests (application/x-www-form-urlencoded HTML forms) execute state modifications on the backend long before CORS checks evaluate response headers.

Myth 6: "SameSite=Lax makes CSRF tokens obsolete."

Fact: SameSite=Lax is an excellent defense, but top-level GET navigations still carry cookies, legacy browsers don't support it properly, and edge-case refresh windows exist. CSRF tokens remain necessary as defense-in-depth.

Myth 7: "Attackers can read our CSRF token from the HTML form."

Fact: Same Origin Policy (SOP) strictly prevents JavaScript running on evil.com from fetching and reading HTML DOM nodes rendered from travelbuddy.com.

Production Best Practices Checklist

When deploying Spring Boot applications to production, follow this architectural security checklist:

1. Identify Your Architecture Type

• Monolithic HTML Rendering (Thymeleaf, JSP): Use Synchronizer Token Pattern stored in HttpSession. Ensure all HTML forms include _csrf hidden fields.

• Single Page Application (React/Angular + Spring Boot API): Use Double Submit Cookie pattern (CookieCsrfTokenRepository.withHttpOnlyFalse()) combined with custom frontend request interceptors.

• Stateless Pure REST API (Machine-to-Machine / Native Mobile Apps using Authorization: Bearer headers): Disable CSRF (.csrf(csrf -> csrf.disable())), because clients explicitly manage non-cookie tokens.

2. Cookie Security Flags

Ensure every authentication cookie sets these attributes:

• Secure = true (HTTPS only)

• HttpOnly = true (Prevents XSS token theft)

• SameSite = Lax or Strict (Browser-native cross-site blocking)

3. Keep GET Requests Read-Only

Audit your codebase to ensure no @GetMapping or HttpServletRequest.getMethod().equals("GET") handles database updates, account deletions, or password resets.

4. Cross-Origin Defense Layers

Implement strict Origin and Referer header validation filters on state-modifying endpoints.

Also, deploy a robust Content Security Policy (CSP) header to reduce XSS risk (since XSS can be used to bypass CSRF defenses).

5. Webhooks and External Callbacks

For server-to-server endpoints (such as Stripe or GitHub webhooks):

• Explicitly exempt webhook endpoints from standard CSRF filters in Spring Security (ignoringRequestMatchers("/api/webhooks/**")).

• Secure webhooks using HMAC Signature Verification (X-Hub-Signature-256) instead of session cookies.

Final Summary & Defense Matrix

Cross-Site Request Forgery (CSRF) isn't a bug in browser design. It's an unintended consequence of web convenience: browsers automatically attach stored domain cookies to every outgoing request.

When an attacker tricks a user into visiting a malicious origin (evil.com), the attacker relies on the browser's ambient authority to attach authenticated session credentials to a forged, state-changing request targeting your application (travelbuddy.com).

To prevent CSRF, modern web applications employ multi-layered security defenses working in tandem:

Comprehensive Defense Matrix

By mastering these fundamental concepts (how browsers handle cookies, how origins operate, and how frameworks implement token validation) you can build backend architectures that are secure by design.

The app processes credit and debit transactions, fires compliance alerts, and stores everything in a database. You'll build the infrastructure around it yourself: automation, scanning, policy enforcement, secrets management, threat detection, and observability.

By the time you're finished, you'll be able to talk through every decision in an interview because you made each one.

This guide doesn't hand you a pre-built solution. It makes you feel out. and understand each problem before introducing the tool that solves it.

All the code, manifests, scripts, and stage-by-stage READMEs live in the companion repository. Clone it before you start:

git clone https://github.com/Osomudeya/clearledger.git
cd clearledger

Everything in this book refers to files inside that repo.

Prerequisites

You'll need these tools installed on your machine before Stage 0:

• Multipass: creates a lightweight Ubuntu VM so Kubernetes has enough resources

• kubectl: talks to your Kubernetes cluster from your terminal

• Helm: installs apps into Kubernetes

• Docker Desktop: builds container images

• jq: formats JSON output so it's readable

You'll also need free accounts on GitHub and Docker Hub.

And you should be comfortable with the following knowledge and skills:

• Basic Linux command line: navigating directories, reading files, running scripts

• Git: clone, commit, push

• What a container is and roughly how Docker builds one

You don't need prior Kubernetes, security, or cloud experience. This guide builds that from Stage 0.

Your machine needs at least 24 GB of RAM, 6 CPU cores, and 80 GB of free disk space. See How to Set Up Your Machine for the exact install commands.

The companion repo is at github.com/Osomudeya/clearledger. Star it, clone it, then continue.

Table of Contents

• Prerequisites

• What You Are Building

• How to Work Through This Lab

• Tools You Will Use

• How to Choose Your Path

• How to Save Your Progress

• Who This Is For

• How to Set Up Your Machine

• How to Start the Lab

• How to Manage Disk Space

• How to Try the App Without Kubernetes

• How to Configure Local Domain Names

• Stage 0 — The Running System

• Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)

• Stage 2 — GitOps with ArgoCD

• Stage 3 — Security Gates

• Stage 4 — Admission Control (Kyverno)

• Stage 5: Secrets Management (Vault)

• Stage 6 — Runtime Security (Falco)

• Stage 6.5 — Chaos Engineering (Optional)

• Stage 7 — Security Observability

• Stage 7.5 — OpenTelemetry (Optional)

• Stage 8 — AWS Migration

• Troubleshooting (see troubleshooting.md)

• Compliance Reference

• Interview Preparation

• AWS Cost Reference

• Conclusion

What You Are Building

ClearLedger is a fintech transaction ledger built with three FastAPI microservices, PostgreSQL, Redis, and a web frontend.

Users can register, sign in, record credit and debit transactions, view their account balance, and receive compliance alerts whenever a transaction exceeds a predefined threshold.

The application is intentionally simple. Its purpose isn't to teach fintech. It gives you a realistic system that you'll secure and operate like a production platform.

The project consists of four components:

• auth-service: Handles user registration, login, and JWT authentication.

• ledger-service: Processes transactions, maintains account balances, and stores transaction history.

• notification-service: Listens for large transactions through Redis and generates compliance alerts.

• frontend: A web interface for logging in, viewing balances, submitting transactions, and reviewing alerts.

By the end of this book, every one of these services will still exist. What changes is how they are built, deployed, secured, and operated.

The application is simply the vehicle. DevSecOps is the destination.

How the Platform Evolves

You won't install every tool on day one. Instead, the platform grows the same way production systems usually do: a problem appears first, then a solution is introduced.

You'll begin with a manually deployed Kubernetes application. From there, each stage solves one real operational problem.

Stage 0: Raw Kubernetes

You'll deploy and run the application manually, which lets you understand the system before introducing automation.

Stage 1: Continuous Integration

Building container images becomes automatic whenever code is pushed, eliminating manual build steps.

Stage 2: GitOps

Deployments are no longer done with kubectl. Git becomes the single source of truth, preventing configuration drift.

Stage 3: Security Gates

Every commit passes through security scanning so vulnerable code, secrets, and misconfigurations are stopped before deployment.

Stage 4: Admission Control

Even if something bypasses the pipeline, Kubernetes policies prevent insecure workloads from entering the cluster.

Stage 5: Secrets Management

Application credentials move out of Kubernetes Secrets into Vault, removing sensitive data from Git and cluster storage.

Stage 6: Runtime Security

Falco continuously watches running containers and detects suspicious behavior after deployment.

Stage 6.5 (Optional): Chaos Engineering

Failures are introduced deliberately to verify that the platform can recover instead of simply detecting problems.

Stage 7: Observability

Metrics, logs, and dashboards provide visibility into the health, performance, and security of the platform.

Stage 7.5 (Optional): OpenTelemetry

Distributed tracing follows requests across every service, revealing how a single transaction moves through the system.

Stage 8: AWS Migration

The same architecture is deployed on AWS using EKS, ECR, RDS, and an Application Load Balancer without changing how the application itself works.

If you simply want to explore the application before touching Kubernetes, an optional Docker Compose stack lets you run everything locally on your machine.

The guiding principle of this book is simple: every stage makes you feel the problem before introducing the tool that solves it.

How to Work Through This Lab

Throughout this process of understanding each problem before introducing the tool that solves it, three habits will carry you through every stage.

1. Read first before you run: The paragraphs before each command explain why you're running it. Skipping them means you can reproduce the steps but not explain them, and explaining them is what gets you hired. The commands are proof you understand.

2. Choose with a reason: Every tool here solves a specific problem. Why use Vault instead of Kubernetes Secrets? Why split code and manifests into two repos? Don't just follow the steps: ask what breaks if we skip this? If you understand the problem, you'll remember the solution.

3. Go in order and verify every checkpoint: Each stage depends on the one before it. When you hit an issue, read the error. Getting stuck and debugging is part of the learning: employers want to hear "I hit X error and fixed it by doing Y."

At every ✋ Hands-on checkpoint:

1. Run the command.

2. Compare your output with Expected.

3. If it doesn't match, fix it before continuing.

4. When make check-N passes: make snapshot STAGE=N && make snapshots. Only continue after you see clearledger.stageN.

Avoid these mistakes:

• Don't skip a checkpoint because it passed before.

• Don't run make restore without checking available snapshots first.

• Replace your-username with your real Docker Hub or GitHub username everywhere it appears.

• Run runner commands inside the VM (prompt shows ubuntu@clearledger), not on your Mac.

Take screenshots at each portfolio checkpoint. These moments become your evidence: proof that the platform runs, detects, blocks, syncs, and observes real activity.

Tools You Will Use

Come back to the below table when a new name appears and you wonder why now. Each entry is one line: what it does and when it appears.

On your laptop:

• Multipass creates the Ubuntu VM.

• Docker builds images.

• make wraps long commands into make setup / make check-N.

• /etc/hosts entries like clearledger.local let your browser reach the cluster.

The app:

• Three Python APIs (auth, ledger, notifications) + a web frontend.

• Postgres stores data

• Redis lets ledger publish alerts without calling notification directly

• nginx ingress routes browser traffic to the right service.

Each stage adds a new security layer. The tools aren't interchangeable: scanners check your code and images before deployment, ArgoCD keeps the cluster synced to Git, Vault handles secrets, Kyverno blocks unsafe workloads before they run, and Falco watches for suspicious behavior after they're running.

That's why the order matters: you're building defense in depth, one layer at a time.

How to Choose Your Path

Pick one path from your host RAM before you provision a cluster. Switching mid-lab after OOM kills or disk pressure wastes a day, so choose upfront.

The default path in this guide assumes 24 GB+ RAM and the full local VM (Before You Start). If that's not you, start from the row that matches your machine.

How to Save Your Progress

Mac + Multipass only: make snapshot and make restore require Multipass. If you're using Linux without Multipass, skip snapshots and use Path B if something goes wrong.

This lab takes several days to complete.

Your source code lives on your computer, so rebuilding or deleting the VM doesn't delete your Git repository, commits, manifests, or configuration files.

The VM stores your running environment, including deployed pods, Vault secrets, Postgres data, and Grafana dashboards.

Save Your Progress

After completing each stage, create a snapshot before moving on. For example:

make snapshot STAGE=7
make snapshots

Always run make snapshots to confirm the snapshot was created.

Restore Your Progress

If the VM becomes unusable after a while, restore the latest working snapshot:

make snapshots
make restore STAGE=7

export KUBECONFIG=~/.kube/clearledger-config
make check-7

What Happens If the VM Breaks?

You keep:

• Your Git repository

• Your commits

• .env

• setup-cluster.local.env

• clearledger-infra on GitHub

You lose anything stored inside the VM after your last snapshot, including:

• Running pods

• Vault secrets

• Postgres data

• Grafana and Loki data

That's why it's a good idea to create a snapshot after every completed stage.

Path A: You Have a Snapshot (Recommended)

Restore the latest working snapshot and continue from that stage.

make snapshots
make restore STAGE=6

export KUBECONFIG=~/.kube/clearledger-config
make check-6

Path B: No Snapshot

Rebuild the lab.

make teardown
make setup

export KUBECONFIG=~/.kube/clearledger-config

Your Git repositories are still intact, but the Kubernetes cluster starts empty. Continue the book from the stage you had reached and rebuild the platform from there.

If you run into problems such as disk space issues, failed snapshots, Mac sleep or restart problems, Vault authentication errors, or pods stuck in CrashLoopBackOff, see troubleshooting.md for detailed recovery steps.

Who This Is For

Junior DevOps (0–2 yrs): do every stage in order. Don't skip the pain point sections. Expect Stage 0–2 to take a full day each, Stages 3–7 half a day each, Stage 8 a few hours. That's normal, so don't rush.

Mid-level DevOps (2–4 yrs): skim Stages 0–2 to understand the app, focus time on Stages 3–7 where the security layers are.

Interview preparation: complete through Stage 4, then read docs/interview-prep.md. The questions are based on exactly what's in this lab.

How to Set Up Your Machine

Requirements are in Prerequisites above. Confirm 24 GB RAM, 6 CPU cores, and 80 GB free disk before installing.

Install the Required Tools

Windows users: Run all commands inside WSL2 Ubuntu. Don't use PowerShell for this lab because the setup uses make and Bash scripts.

Verify everything before continuing:

multipass --version
kubectl version --client
helm version
docker --version
jq --version

If any command fails, install the missing tool before continuing.

How to Start the Lab

The main lab path starts at Stage 0: the Running System.

After you have run the setup once step by step, you can use this shortcut next time:

make setup
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes

Expected: one node named clearledger with STATUS Ready.

make setup provisions the Multipass VM, installs MicroK8s, applies disk-safety caps, and updates /etc/hosts. Takes 3–5 minutes.

How to Manage Disk Space

The lab runs on a single-node MicroK8s VM with a fixed disk (80 GB by default). Over days or weeks (especially after CI builds, Helm upgrades, and Stage 7 observability) container images, logs, and journald can fill the root filesystem. Pods then fail with Evicted, ImagePullBackOff, or mysterious Pending states.

make setup applies preventive caps automatically (log rotation, image GC thresholds, journald cap). See Disk health in troubleshooting.md for the full table and recovery steps.

Check disk health:

make doctor    # PASS / WARN / FAIL + PVC and Prometheus TSDB sizes

Clean up unused files inside the VM without deleting app data:

make reclaim

If make doctor still reports FAIL after reclaim, you may need make teardown && make setup and restore from a snapshot. Full guidance: troubleshooting.md. VM disk full.

How to Try the App Without Kubernetes

If your machine doesn't have enough resources for Kubernetes, you can run ClearLedger with Docker Compose.

docker compose -f docker-compose.integration.yml up --build -d

Open http://localhost:3000.

When you're ready, stop the stack and continue with Stage 0.

docker compose -f docker-compose.integration.yml down

How to Sign In for the First Time

First, you'll need to register. The database starts empty after each fresh up (or down -v). Use a real-looking email (Pydantic rejects @*.local), for example test@clearledger.io for an email and SecurePass123 for a password.

Then sign in with the same credentials.

Wrong password shows Incorrect email or password. If you see a stale error, hard-refresh or run localStorage.removeItem('cl_token') in the browser console.

How to Run the Demo Flow

First, register and sign in at http://localhost:3000. Submit a few credits and debits (for example, Salary +$5000, Rent −$1200).

Then confirm the balance updates and history lists entries.

Now submit a transaction ≥ $10,000: the Alerts panel should show LARGE_TRANSACTION.

Here's an optional smoke test against the same base URL:

BASE_URL=http://localhost:3000 bash scripts/dast/smoke.sh

How to Configure Local Domain Names

Add the ClearLedger hostnames to your hosts file.

macOS or Linux with Multipass

Run:

sudo bash scripts/setup-hosts.sh

Or do it manually:

VMIP=$(multipass info clearledger | grep IPv4 | awk '{print $2}')

echo "$VMIP  clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts

Verify after Stage 0:

curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health

Expected: 200.

WSL2

Find your WSL IP:

ip -4 addr show eth0 | grep inet

Use the IP shown (or 127.0.0.1 if it works on your machine), then add it to /etc/hosts:

LAB_IP=<YOUR_IP>

echo "$LAB_IP  clearledger.local argocd.local grafana.local vault.local falco.local litmus.local" | sudo tee -a /etc/hosts

If you use Chrome or Edge on Windows instead of inside WSL, add the same line to:

C:\Windows\System32\drivers\etc\hosts

Verify:

curl http://clearledger.local/auth/health

Stage 0 — The Running System

Starting point: Nothing is deployed yet, so you're about to build a Kubernetes cluster and deploy ClearLedger manually.

Goal: By the end of this stage, ClearLedger will be running on Kubernetes. You'll be able to register a user, submit transactions, and see compliance alerts, all deployed by hand, with no automation.

Every deployment, update, and fix is manual. That's intentional. Before automating a platform, you need to understand how it works without automation.

0.1: Provision the Cluster

Next you'll be creating a virtual machine on your laptop that runs its own Kubernetes cluster. Think of it as a miniature data center inside your computer.

Multipass creates lightweight Ubuntu VMs. MicroK8s is a minimal Kubernetes distribution that runs inside that VM. Together they give you a real cluster without needing cloud resources.

Recommended: one command (do this):

make setup
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes

Expected:

NAME          STATUS   ROLES    AGE   VERSION
clearledger   Ready    <none>   2m    v1.29.x

make setup runs scripts/setup-cluster.sh (VM + MicroK8s + disk-safety caps) and scripts/setup-hosts.sh (/etc/hosts entries). It takes 3–5 minutes.

Disk-safety (log rotation, image GC thresholds, journald cap) is configured automatically. See Disk health in troubleshooting.md for more info.

If STATUS is NotReady, wait 60 seconds and try again.

Here's the manual setup (only if make setup failed and you need to debug step by step):

multipass launch \
  --name clearledger \
  --cpus 6 --memory 12G --disk 80G \
  22.04

Get the VM IP (needed for /etc/hosts):

multipass info clearledger | grep IPv4

Add hosts entries. See the Domain Names section above, or run sudo bash scripts/setup-hosts.sh.

multipass shell clearledger

Inside the VM:

sudo snap install microk8s --classic --channel=1.29/stable
sudo usermod -aG microk8s ubuntu && newgrp microk8s
microk8s enable dns ingress storage helm3 rbac
echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
echo "alias helm='microk8s helm3'" >> ~/.bashrc
source ~/.bashrc
kubectl get nodes
exit   # back to your host machine

Connect kubectl from your host:

multipass exec clearledger -- microk8s config > ~/.kube/clearledger-config
export KUBECONFIG=~/.kube/clearledger-config
kubectl get nodes

0.2: Understand the Application Before Deploying it

Open these files before running a single kubectl command. Reading the code first builds context that makes everything else make sense.

Notice this line in every Dockerfile: USER appuser. It means the image is designed to run as a normal user instead of root. The Kubernetes manifests also set runAsNonRoot: true. Later, in Stage 4, Kyverno enforces that rule and rejects pods that don't declare they run as non-root. Your app is prepared early so it passes that policy later.

Also look at infra/manifests/auth-service/secret.yaml. The database password is changeme-stage0 encoded in base64. Decode it:

echo "Y2hhbmdlbWUtc3RhZ2Uw" | base64 -d
# changeme-stage0

That password is sitting in a YAML file anyone with repo access can read. base64 is encoding, not encryption. It's trivially reversible. Remember this moment. It's why Stage 5 exists.

0.3: Docker Hub Setup

You need a container registry: a place to store the built images so the cluster can pull them. Docker Hub is the simplest option. You'll replace it with a private registry (ECR) in Stage 8.

Create four public repositories on Docker Hub (free account, hub.docker.com):

1. Go to hub.docker.com

2. Click Create repository

3. Choose your Docker Hub username as the namespace

4. Enter one repository name from the list below

5. Set visibility to Public

6. Click Create

7. Repeat for all four services

YOUR_USERNAME/clearledger-auth-service
YOUR_USERNAME/clearledger-ledger-service
YOUR_USERNAME/clearledger-notification-service
YOUR_USERNAME/clearledger-frontend

Next, generate an access token. Go to hub.docker.com, then Account Settings, Security, and New Access Token (Read/Write/Delete). Save it. You won't see it again.

docker login
# Username: your Docker Hub username
# Password: the access token (NOT your account password)

Build and push all four services:

# Replace your-username with your Docker Hub username, the same string everywhere in this lab
export DOCKER_USERNAME=your-username
echo "Using DOCKER_USERNAME=$DOCKER_USERNAME"

✋ Hands-on checkpoint: Docker Hub username

# Must print your real username, not the literal text "your-username"
echo "$DOCKER_USERNAME"

Expected: one line with your Docker Hub name (for example, veeno-demo). If you see your-username instead, stop and fix export before building.

Build and push all four services:

docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.1.0 ./app/auth-service
docker build -t $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0 ./app/ledger-service
docker build -t $DOCKER_USERNAME/clearledger-notification-service:v0.1.0 ./app/notification-service
docker build -t $DOCKER_USERNAME/clearledger-frontend:v0.1.0 ./app/frontend

# Push

docker push $DOCKER_USERNAME/clearledger-auth-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-ledger-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-notification-service:v0.1.0
docker push $DOCKER_USERNAME/clearledger-frontend:v0.1.0

✋ Hands-on checkpoint: images on Docker Hub

Open hub.docker.com and go to your profile, then Repositories. Then confirm that all four clearledger-* repos exist and each shows tag v0.1.0.

On your laptop, run:

docker pull $DOCKER_USERNAME/clearledger-auth-service:v0.1.0

Expected: Status: Downloaded newer image or Image is up to date, not repository does not exist or denied.

0.4: Look at the Manifests Before Applying Them

Kubernetes uses manifest files (YAML) to describe the resources it should create. Instead of clicking buttons, you declare the desired state, and Kubernetes creates it.

Before deploying ClearLedger, take a quick look at these manifests:

• infra/manifests/namespace.yaml: Creates the clearledger namespace.

• infra/manifests/postgres/: Deploys PostgreSQL.

• infra/manifests/redis/redis.yaml: Deploys Redis.

• infra/manifests/auth-service/: Deploys the authentication service.

• infra/manifests/ledger-service/: Deploys the ledger service.

• infra/manifests/notification-service/: Deploys the notification service.

• infra/manifests/frontend/: Deploys the web application.

• infra/manifests/ingress.yaml: Makes the application available at clearledger.local.

• infra/manifests/rbac/rbac.yaml: Defines who may do what inside the cluster.

You don't need to understand every field yet. The goal is simply to see how the application is described before Kubernetes creates it.

You'll understand how Ingress routing and RBAC work in the two optional sections after §0.6. For now, just see how the app is described before Kubernetes creates it.

0.5: Deploy ClearLedger (Layer by Layer)

Deploy in six layers. Finish each layer before starting the next. Run kubectl get pods -n clearledger after layers 2, 3, and 6 to confirm progress.

Set a short path variable and confirm your username is still set:

export DOCKER_USERNAME=your-username   # skip if already set in §0.3
STAGE0=stages/stage-0-raw-kubernetes/infra/manifests

0.5.1 — Layer 1: Namespace and RBAC

Nothing else can be created until the namespace exists. RBAC also must exist before workloads reference ServiceAccounts.

kubectl apply -f infra/manifests/namespace.yaml
kubectl apply -f infra/manifests/rbac/rbac.yaml

Verify:

kubectl get namespace clearledger
kubectl get serviceaccount -n clearledger
# Expected: auth-service, ledger-service, notification-service, clearledger-viewer

0.5.2 — Layer 2: PostgreSQL

Database must be running before auth-service or ledger-service start. Both services connect to Postgres on startup to run migrations and serve requests, and they'll crash-loop if the database isn't there yet.

kubectl apply -f infra/manifests/postgres/postgres-secret.yaml
kubectl apply -f infra/manifests/postgres/postgres.yaml

kubectl wait --for=condition=ready pod -l app=postgres \
  -n clearledger --timeout=120s

Expected after kubectl apply:

secret/postgres-secret created
persistentvolumeclaim/postgres-pvc created
statefulset.apps/postgres created
service/postgres created

Expected when kubectl wait succeeds: the command exits with no output (exit code 0). If it times out, see If Postgres stays Pending below before continuing.

Verify:

kubectl get pods -n clearledger -l app=postgres
kubectl get pvc -n clearledger

Expected:

NAME         READY   STATUS    RESTARTS   AGE
postgres-0   1/1     Running   0          45s

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
postgres-pvc   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   5Gi        RWO            microk8s-hostpath   45s

If Postgres stays Pending (kubectl wait times out, pod shows 0/1 Pending, PVC shows Pending):

Postgres needs a PersistentVolumeClaim: disk space on the cluster. MicroK8s provides that through the hostpath-storage addon. If make setup was interrupted or you used manual setup without microk8s enable storage, the PVC has nothing to bind to and the pod never schedules.

Check the events: you'll usually see something like:

Warning  FailedScheduling  ...  pod has unbound immediate PersistentVolumeClaims
Normal   FailedBinding     ...  no persistent volumes available for this claim and no storage class is set

Fix it on the VM, then restart the postgres pod. Run this from your host: the same command on macOS, Linux, or Windows PowerShell (Multipass is installed on the host. It executes inside the VM for you):

# Enable storage (and ingress/rbac if make setup skipped them)
multipass exec clearledger -- microk8s enable storage ingress rbac

# Confirm a default StorageClass exists
kubectl get storageclass
# Expected: microk8s-hostpath (default)

# Kick the pod so it reschedules against the new storage class
kubectl delete pod postgres-0 -n clearledger

kubectl wait --for=condition=ready pod -l app=postgres \
  -n clearledger --timeout=120s
kubectl get pods -n clearledger -l app=postgres
# Expected: postgres-0   1/1   Running

Don't continue to auth-service or ledger-service until Postgres is Running. They will crash-loop without a database.

0.5.3 — Layer 3: Redis

Why Redis is here (a quick scenario): Imagine a customer posts a $15,000 debit. Ledger-service saves it to Postgres, then publishes a message to Redis: "large transaction, user X, amount 15000." Notification-service is listening on that channel. It picks up the message and records a compliance alert: the one you'll see in the UI later when you curl /notifications/alerts.

Ledger-service and notification-service don't call each other directly. Redis sits in the middle as a message bus: ledger publishes, notification subscribes. That's why Redis must be running before you deploy notification-service (and why you deploy it now, alongside Postgres, before the app layer).

kubectl apply -f infra/manifests/redis/redis.yaml

Verify:

kubectl get pods -n clearledger -l app=redis

Expected:

NAME                     READY   STATUS    RESTARTS   AGE
redis-xxxxxxxxxx-xxxxx   1/1     Running   0          30s

0.5.4 — Layer 4: Application secrets

Credentials live in Kubernetes Secrets for Stage 0 (Stage 5 moves them to Vault).

kubectl apply -f infra/manifests/auth-service/secret.yaml
kubectl apply -f infra/manifests/ledger-service/secret.yaml

Verify:

kubectl get secrets -n clearledger | grep -E 'auth-service|ledger-service'

Expected (AGE will differ, DATA counts must match):

auth-service-secret     Opaque   2      64s
ledger-service-secret   Opaque   1      8s

auth-service-secret holds two keys (database_url, jwt_secret). ledger-service-secret holds one (database_url). Stage 5 replaces these with Vault, for now they live in the cluster as Kubernetes Secrets.

0.5.5 — Layer 5: Application workloads

You're about to start the four app services: auth, ledger, notification, and frontend. Postgres, Redis, and the Secrets from the last two layers are already in place. Now Kubernetes needs to pull your Docker Hub images and run them as pods.

Two files per service (mostly): A Deployment tells Kubernetes which container image to run and how many copies. A Service gives that app a stable name inside the cluster (for example, auth-service so ledger can find auth without knowing pod IP addresses). You apply the Deployment first, then the Service.

So why are we using the sed command below? The deployment YAML files in Git contain a placeholder: literally the text DOCKER_USERNAME, because everyone's Docker Hub username is different. You already set yours in §0.3 (export DOCKER_USERNAME=YOUR_DOCKERHUB_USERNAME). The sed line swaps that placeholder for your real username on the fly, as the manifest is sent to Kubernetes. You never edit the file in Git. If you skip sed and apply the raw file, Kubernetes tries to pull an image called DOCKER_USERNAME/clearledger-auth-service, which doesn't exist.

Why do we use the Stage 0 folder? This repo has more than one copy of the Kubernetes manifests. For this manual deployment, use stages/stage-0-raw-kubernetes/infra/manifests/. Those files are prepared for Stage 0 and contain the DOCKER_USERNAME placeholder that the commands below replace. Don't use infra/manifests/ yet, as those files are for the GitOps stages later.

Deploy each service in order. Run these from the repo root with DOCKER_USERNAME still exported:

1. auth-service: login and registration

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/auth-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/auth-service/service.yaml

2. ledger-service: transactions and balance (needs Postgres + the secret you created in §0.5.4)

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/ledger-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/ledger-service/service.yaml

3. notification-service: listens on Redis for large-transaction alerts (no database secret in this one)

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/notification-service/deployment.yaml" | kubectl apply -f -
kubectl apply -f infra/manifests/notification-service/service.yaml

4. frontend: the web UI (Deployment and Service are in one file here)

sed "s|DOCKER_USERNAME|${DOCKER_USERNAME}|g" \
  "$STAGE0/frontend/deployment.yaml" | kubectl apply -f -

Verify (all app pods should reach Running: auth and ledger may take ~30s while they connect to Postgres):

kubectl get pods -n clearledger

Expected. You should see Postgres and Redis from earlier layers plus new pods for each app (exact pod names vary):

NAME                                      READY   STATUS    RESTARTS   AGE
postgres-0                                1/1     Running   0          15m
redis-xxxxxxxxxx-xxxxx                    1/1     Running   0          10m
auth-service-xxxxxxxxxx-xxxxx             1/1     Running   0          45s
auth-service-xxxxxxxxxx-xxxxx             1/1     Running   0          45s
ledger-service-xxxxxxxxxx-xxxxx           1/1     Running   0          40s
ledger-service-xxxxxxxxxx-xxxxx           1/1     Running   0          40s
notification-service-xxxxxxxxxx-xxxxx     1/1     Running   0          35s
frontend-xxxxxxxxxx-xxxxx                 1/1     Running   0          30s

If auth-service or ledger-service is CrashLoopBackOff, check the logs:

kubectl logs -n clearledger deploy/auth-service --tail=20

Common cause: you applied infra/manifests/*/deployment.yaml instead of the Stage 0 files above: logs may show DATABASE_URL is not set. Re-run the sed + kubectl apply commands in this section.

✋ Hands-on checkpoint: workloads before ingress

kubectl get deployment -n clearledger
kubectl get pods -n clearledger --field-selector=status.phase!=Running

Expected: four Deployments (auth-service, ledger-service, notification-service, frontend) with READY matching desired replicas (auth and ledger show 2/2). The second command prints nothing: no pods stuck in Pending or CrashLoopBackOff.

0.5.6 — Layer 6: Ingress

Exposes the cluster to http://clearledger.local.

kubectl apply -f infra/manifests/ingress.yaml

Verify:

kubectl get ingress -n clearledger
curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/
# Expected: 200

0.5.7: Watch until stable

kubectl get pods -n clearledger -w

Expected final state (press Ctrl+C to stop watching once all pods show Running):

NAME                                  READY   STATUS    RESTARTS
auth-service-xxx                      1/1     Running   0
auth-service-yyy                      1/1     Running   0
frontend-xxx                          1/1     Running   0
ledger-service-xxx                    1/1     Running   0
ledger-service-yyy                    1/1     Running   0
notification-service-xxx              1/1     Running   0
postgres-0                            1/1     Running   0
redis-xxx                             1/1     Running   0

Pod stuck in Pending or CrashLoopBackOff? These two commands show you what went wrong:

kubectl describe pod POD_NAME -n clearledger
kubectl logs POD_NAME -n clearledger --previous

0.6: Verify the Running System

Use one test account for both browser and curl so nothing conflicts:

If you already registered in the browser with a different password, either sign in with that password or pick a new email: the curl commands below must use the same email and password you actually registered with.

Browser verification (recommended):

Open http://clearledger.local in your browser. You should see the ClearLedger login screen.

Click Register and create an account with test@clearledger.io / SecurePass123 (same as the curl block below. Pydantic rejects obviously fake emails like test@test.com).

Sign in with that email and password. On first login the dashboard auto-seeds demo transactions. Wait a few seconds for them to appear:

Look at the Current Balance card. It should show a dollar amount with a sparkline chart.

Look at Transaction History. You should see entries like "Salary (Acme Corp", "Rent) May 2026", and so on.

And look at the Alerts panel at the bottom. You should see LARGE_TRANSACTION alerts with a red badge. Two of the demo transactions exceed $10,000, which triggers the compliance alert automatically.

Then submit your own transaction over $10,000 and watch the alert count increase in real time.

What to look for:

• Balance updates immediately after each transaction

• Credits show as green +$ amounts, debits show as red −$ amounts

• The Alerts badge count increases when you submit a transaction ≥ $10,000

• Each alert shows the amount, direction, and timestamp

Take a screenshot of the dashboard showing transactions and at least one alert. This is the first piece of your portfolio.

Alternatively via curl (same account: useful if the browser is not cooperating):

# Register (skip if you already registered in the browser with the same email)
curl -s -X POST http://clearledger.local/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' | jq .

Expected: {"user_id":"...","email":"test@clearledger.io"}, or an error that the email is already registered (fine if you used the browser first).

# Login — save the token (must match the password you registered with)
TOKEN=$(curl -s -X POST http://clearledger.local/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@clearledger.io","password":"SecurePass123"}' \
  | jq -r .access_token)
echo "Token: ${TOKEN:0:30}..."

If TOKEN is empty or login returns 401, your browser password doesn't match: re-register with the table above or use your actual password in the -d JSON.

# Create a large transaction (triggers notification alert)
curl -s -X POST http://clearledger.local/ledger/transactions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"amount":15000,"direction":"debit","description":"Property payment"}' | jq .

Expected: a transaction object with id, amount: 15000, direction: "debit":

# Check balance
curl -s http://clearledger.local/ledger/balance \
  -H "Authorization: Bearer $TOKEN" | jq .

# Confirm the notification alert fired
curl -s http://clearledger.local/notifications/alerts | jq .

Expected (curl-only path, no browser demo seed): at least one alert for the $15,000 transaction, for example, {"total":1,"alerts":[{"type":"LARGE_TRANSACTION","amount":15000,...}]}. If you already used the browser, total may be 3 or more (two demo alerts plus yours), that is also correct.

If you see {"detail":"Unauthorized"}: your token has expired. JWTs are short-lived for security. This is intentional. Re-run the login command above to get a fresh token, then retry the failed command.

This only affects the $TOKEN variable in your current terminal session. If you open a new terminal, you need to run the login command again because $TOKEN doesn't persist across sessions.

make check-0

Understanding Ingress (Optional)

Read this after §0.6 if you want to understand how clearledger.local reaches your pods.

Your cluster runs four application services: frontend, auth-service, ledger-service, and notification-service. Each has an internal Service address inside the cluster, but none are reachable from your browser until an Ingress routes external traffic.

The Ingress is the front door. When a request hits clearledger.local, Kubernetes looks at the URL path and forwards to the right service. Requests to /auth go to auth-service, /ledger to ledger-service, /notifications to notification-service, and / to the frontend.

Open infra/manifests/ingress.yaml and read the comments. The API paths use a rewrite: /auth/login becomes /login before it reaches auth-service, so backend routes stay simple.

You'll add more hostnames later (grafana.local, argocd.local, and so on): each gets its own Ingress manifest in a later stage. This file is only the ClearLedger app.

Understanding RBAC (Optional)

Ingress controls traffic coming from outside the cluster. RBAC controls permissions inside the cluster.

This file creates identities and permissions for the clearledger namespace.

Open infra/manifests/rbac/rbac.yaml. The comments at the top mirror this walkthrough.

A ServiceAccount is an identity for a pod. For example, auth-service, ledger-service, and notification-service each get their own identity.

A Role says what that identity is allowed to do. In your repo, the app roles are very limited: they can only get and list Kubernetes Endpoints. They can't read Secrets, delete pods, create resources, or access other namespaces.

A RoleBinding connects the identity to the permissions. Without the RoleBinding, the Role exists but no pod receives those permissions.

The clearledger-viewer ServiceAccount is for read-only debugging. It can inspect pods, services, endpoints, events, and configmaps, but it can't read Secrets.

The default ServiceAccount is bound to a role with zero permissions. That way, if a pod forgets to set serviceAccountName, it falls back to an identity that can do nothing.

The point is least privilege: even if a pod is compromised, Kubernetes doesn't hand it broad cluster access.

0.7: Why Manual Deploys Can't Be Trusted

In Stage 0, you built and deployed the app by hand. Now you'll make one small code change and deploy it again. This shows the problem with manual deployments: they're hard to track, hard to roll back, and hard to prove. Stages 1 and 2 fix that with CI and GitOps.

Step 1: Make a visible change.

Open app/auth-service/main.py and find the /health endpoint. Change the return value so you can tell the new version is running:

# Before
return {"status": "ok", "service": settings.service_name}

# After — add a version field
return {"status": "ok", "service": settings.service_name, "version": "0.2.0"}

Save the file. This simulates a developer shipping a small fix.

Step 2: Build, push, and deploy by hand.

docker build -t $DOCKER_USERNAME/clearledger-auth-service:v0.2.0 ./app/auth-service

docker push $DOCKER_USERNAME/clearledger-auth-service:v0.2.0
kubectl set image deployment/auth-service \
  auth-service=$DOCKER_USERNAME/clearledger-auth-service:v0.2.0 \
  -n clearledger

Wait about 30 seconds for Kubernetes to pull the new image and restart the pods:

kubectl rollout status deployment/auth-service -n clearledger

Step 3: Verify your change is live.

curl -s http://clearledger.local/auth/health | jq .

Expected: {"status":"ok","service":"auth-service","version":"0.2.0"}

If you still see the old response without "version", wait a few more seconds and retry. Kubernetes is still rolling out the new pods.

Step 4: Notice what manual deploy doesn't give you.

You deployed a change. It works. But think about what just happened:

• Who deployed this? There's no record. You ran kubectl from your laptop. If three people have cluster access, no one knows who changed what.

• What changed? The only evidence is the Docker Hub tag v0.2.0. Nothing links that tag to a specific commit or code review.

• What if v0.2.0 is broken? You would need to remember the previous tag, then run kubectl set image again to roll back. What if you don't remember the tag? What if the previous image was deleted?

• What if someone else runs kubectl apply with v0.1.0 while you're pushing v0.2.0? The cluster silently reverts to the old version. No error. No notification. You think your fix is live, but it's not.

• Where is the audit trail? Nowhere. In a regulated environment (banking, healthcare, government), you need proof of who deployed what and when. Right now you have nothing.

Manual deploys can work for a demo. But they don't hold up for a team or a regulated environment. Keep these gaps in mind. They're why the next stages exist.

Step 5: Revert your change before continuing.

Undo the health endpoint change in app/auth-service/main.py (remove "version": "0.2.0"). Don't rebuild: the cluster will keep running v0.2.0 for now, and Stage 1 will take over image management.

Stage 1 automates the build. Stage 2 fixes the deployment.

What You Learned in Stage 0

• How to provision a local Kubernetes cluster with Multipass and MicroK8s

• How Kubernetes manifests describe the desired state of your system

• How an Ingress routes external traffic to internal services

• How to build, push, and deploy container images manually

• Why manual deploys can't be trusted: no audit trail, no rollback, no consistency

What you can now put on your CV / say in an interview:

Deployed a multi-service application to Kubernetes by hand: namespace, RBAC, a StatefulSet database, Deployments, Services, and path-based Ingress routing, and can explain why each layer deploys in that order.

make snapshot STAGE=0 && make snapshots. Confirm clearledger.stage0. See How to Save Your Progress.

Stage 1 — CI Pipeline (GitHub Actions + Self-Hosted Runner)

In Stage 0 you built and deployed by hand. Stage 1 automates the build: a git push runs a pipeline that builds images, scans them, pushes to Docker Hub, and records the new tag in clearledger-infra.

Goal: every push to GitHub automatically builds images, pushes them to Docker Hub, and updates image tags in clearledger-infra.

Am I ready for Stage 1?

Run these yourself before §1.1:

make check-0
echo "$DOCKER_USERNAME"    # must not be empty or "your-username"
curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health

Expected: health check green, echo prints your Docker Hub user, and curl prints 200.

What you'll need for this section:

• Docker Hub account with four clearledger- repositories (see QUICKSTART.md §1b)

• GitHub account: you can create repos and personal access tokens

• ~2–4 hours for runner install + first green pipeline (this is the hardest stage for beginners)

• Done when: make check-1 passes and you manually confirmed the five items in §1.7 below. Then save: make snapshot STAGE=1 → make snapshots (confirm clearledger.stage1).

What You Need to Know First

In Stage 0, your laptop was the deployment system.

You typed docker build, docker push, and kubectl set image yourself. That worked for a demo, but it's not how teams should ship software.

Manual builds create too many unanswered questions:

• Did this image come from the latest code?

• Did someone build it from a dirty working tree?

• Did the build work the same way on another machine?

• Which commit produced the image currently running?

• Who pushed the image, and when?

CI (Continuous Integration) fixes the build side of that problem. It means that every time code is pushed, an automated system builds, checks, and packages it the same way.

Think of CI as a factory line:

Developer pushes code
        ↓
GitHub detects the push
        ↓
GitHub Actions starts the pipeline
        ↓
Runner executes the jobs
        ↓
Docker images are built and pushed
        ↓
Infra manifests are updated with the new image tags  (in clearledger-infra — §1.3)

The important idea is that the build no longer depends on your laptop. Your laptop writes code and the pipeline produces the release artifact.

A CI system has three parts:

1. Pipeline host: the control plane. It notices a push and decides which workflow to run. In this lab, that's GitHub Actions.

2. Pipeline file: the instructions. It's a YAML file at .github/workflows/ci.yaml that says what jobs to run.

3. Runner: the worker machine. It actually executes the commands in the pipeline.

GitHub Actions normally uses GitHub-hosted runners in the cloud. In this lab, that's not enough. Your Kubernetes cluster lives inside a local Multipass VM and GitHub's cloud runner can't reach it. You also need the runner inside the VM to build Docker images using the local Docker daemon.

So you install a self-hosted runner inside the VM. It connects outbound to GitHub, waits for work, then executes pipeline jobs locally where it can reach everything.

Two repos, clearledger (code + CI) and clearledger-infra (Kubernetes YAML only). You'll create the second in §1.3.

GitHub — clearledger (app repo)
  stores your code
  starts the workflow on git push
        ↓
Self-hosted runner (inside Multipass VM)
  builds Docker images
  pushes images to Docker Hub
  updates image tags in clearledger-infra  ← you create this in §1.3
        ↓
GitHub — clearledger-infra (infra repo)
  stores Kubernetes YAML with the new image tags
  ArgoCD watches this repo in Stage 2 (not yet)

For Stages 1–7, the lab uses .github/workflows/ci.yaml with your self-hosted runner. It builds images, pushes them to Docker Hub, and updates clearledger-infra. Stage 8 adds a separate AWS workflow, .github/workflows/ci-aws.yaml, which pushes to ECR instead. You don't need to configure the AWS workflow until you reach Stage 8.

1.1: Push the App Repo to GitHub (Not clearledger-infra Yet)

This step is repo #1, clearledger (application code + CI workflow). You're pushing the clone on your laptop: the same folder where you ran Stage 0 (make setup, kubectl apply, and so on).

clearledger-infra comes later in §1.3. That second repo holds Kubernetes manifests only. Don't create it here.

First, put the application repo somewhere GitHub Actions can see it.

Go to GitHub and then New Repository:

• Repository name: clearledger (exact name, not clearledger-infra)

• Visibility: Public or Private. Both work with the self-hosted runner and GitHub Actions. ArgoCD never reads this repo (see Private repos: what syncs where in §1.3).

• Do not initialize with a README or .gitignore

The repo already has those files locally. If GitHub creates its own, your first push may fail because the histories don't match.

Run from your local clearledger project root on your laptop (where app/, infra/, and .github/workflows/ci.yaml live):

cd ~/Desktop/clearledger   # your clone path
git remote add origin https://github.com/YOUR_USERNAME/clearledger.git
git branch -M main
git push -u origin main

If git remote add fails because origin already exists:

git remote -v
git remote set-url origin https://github.com/YOUR_USERNAME/clearledger.git
git push -u origin main

Verify in the browser: https://github.com/YOUR_USERNAME/clearledger.

You should see app/, infra/manifests/, docs/, and .github/workflows/ci.yaml. That confirms GitHub can trigger the pipeline on your next push.

What you proved: the app repo is on GitHub. CI will run from here. Deployment manifests for GitOps land in clearledger-infra in §1.3.

1.2: Install the Self-Hosted Runner Inside the VM

The workflow file tells GitHub what to run. The runner is where it runs.

This lab uses a self-hosted runner because your infrastructure is local. GitHub's cloud servers can't reach your MicroK8s cluster or Docker daemon inside the Multipass VM. The runner solves that by living inside the VM. It connects to GitHub to pick up jobs, then executes everything locally.

If the runner is missing or offline, the pipeline can't execute. The workflow may sit queued, or it may fail because no matching runner is available.

Step 1: Open GitHub’s runner setup page (keep this tab open)

GitHub gives you a full copy-paste install guide on one page. Use it: don’t hunt for URLs or tokens elsewhere.

1. Open https://github.com/YOUR_USERNAME/clearledger

2. Go to Settings, Actions, Runners, and New self-hosted runner

3. Select Linux and x64

The page title should look like: Add new self-hosted runner · YOUR_USERNAME/clearledger.

That page has three sections you'll use:

Scroll to Configure. You'll see something like:

./config.sh --url https://github.com/YOUR_USERNAME/clearledger --token AXXXXXXXXXXXXXXXXXXXXXXXXX
./run.sh

The token is the long string after --token (starts with A, about 26 characters). Copy only that string.

Keep this tab open until Step 4 finishes: the token expires in about 1 hour. If it expires, click New self-hosted runner again for a fresh token.

Step 2: Enter the VM

multipass shell clearledger

After this command, your prompt should look like ubuntu@clearledger:~$. That means you are inside the Ubuntu VM. If your prompt still shows your Mac username or MacBook name, you're still on your host machine and the runner setup will fail.

Continue only when your prompt shows ubuntu@clearledger.

Everything from Step 3 onwards runs inside the VM, not on your Mac.

Step 3: Install Docker inside the VM

The runner will build Docker images. That means Docker must exist where the runner runs.

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker ubuntu
newgrp docker

docker --version

Expected: Docker prints a version number (for example, Docker version 29.x.x).

Verify Docker works for the ubuntu user now: the runner doesn't exist yet (Step 4 creates ~/actions-runner):

docker ps

Expected: a table header (CONTAINER ID, IMAGE, …), even if no containers are listed. Not permission denied while trying to connect to the Docker API.

If docker ps fails with permission denied, the docker group has not applied yet. Run newgrp docker again, or log out of the VM (exit) and multipass shell clearledger back in, then retry docker ps.

What you proved: the VM can run Docker without Docker Desktop on your Mac. Continue to Step 4 to install the runner.

Step 4: Install and register the runner

Still inside the VM (ubuntu@clearledger prompt):

Download: you can copy the commands from the Download section on GitHub’s runner page (Step 1), or run the block below. They should match. Paste into the VM, not your Mac.

Configure: use the lab command below, not GitHub’s ./config.sh line. Paste your token from Step 1 and replace YOUR_USERNAME.

mkdir -p ~/actions-runner && cd ~/actions-runner

curl -o actions-runner-linux-x64-2.335.1.tar.gz -L \
  https://github.com/actions/runner/releases/download/v2.335.1/actions-runner-linux-x64-2.335.1.tar.gz

tar xzf ./actions-runner-linux-x64-2.335.1.tar.gz

./config.sh \
  --url https://github.com/YOUR_USERNAME/clearledger \
  --token YOUR_RUNNER_TOKEN \
  --name clearledger-runner \
  --labels clearledger,self-hosted,linux \
  --work _work \
  --unattended

sudo ./svc.sh install
sudo ./svc.sh start

Do not run GitHub’s ./run.sh for day-to-day use: the lab uses sudo ./svc.sh so the runner survives VM reboots. GitHub shows ./run.sh for a quick test only.

Expected after ./config.sh: Runner successfully added (or similar). If you see Invalid token or Expired token, go back to Step 1 in the browser and copy a fresh token.

The clearledger label is required GitHub’s default ./config.sh on the setup page doesn't add it. The workflow uses:

runs-on: [self-hosted, clearledger]

GitHub schedules jobs by runner labels, not by runner name. A runner named clearledger without the clearledger label will stay online but jobs will remain queued with Waiting for a runner to pick up this job.

What those last two commands mean:

sudo ./svc.sh install
  Registers the runner with systemd inside the VM.
  Without this, `sudo ./svc.sh status` says: not installed.

sudo ./svc.sh start
  Starts the runner service in the background.
  After this, it keeps running even when you close the terminal.

Check it locally from the same folder, still inside the VM:

cd ~/actions-runner
sudo ./svc.sh status

Expected: the service is installed and running.

If docker ps worked in Step 3 but a CI job later fails with Docker socket permission denied, the runner probably started before the docker group applied. Restart it after Step 4 (only when ~/actions-runner exists):

cd ~/actions-runner
sudo ./svc.sh stop
sudo ./svc.sh start
docker ps    # must work without sudo

Or, if you started the runner manually with ./run.sh instead of systemd:

cd ~/actions-runner
pkill -f "Runner.Listener|Runner.Worker|./run.sh" || true
nohup ./run.sh > _diag/manual-runner.log 2>&1 &
docker ps

If you see this:

not installed

then sudo ./svc.sh install didn't run successfully. Run:

cd ~/actions-runner
sudo ./svc.sh install
sudo ./svc.sh start
sudo ./svc.sh status

If install fails, rerun ./config.sh with a fresh GitHub runner token, then run the install/start commands again.

Step 5: Exit the VM

exit

Step 6: Verify the runner is connected

Go to github.com/YOUR_USERNAME/clearledger then to Settings, Actions, and Runners.

You should see clearledger-runner with a green dot and status Idle. Open the runner details and confirm the labels include:

self-hosted
Linux
X64
clearledger

If clearledger is missing, add it in the runner settings before rerunning the workflow. The runner name alone is not enough.

✋ Hands-on checkpoint: runner ready for jobs

Still on GitHub, Settings, Actions, and Runners, confirm:

Then trigger a dry run from your laptop:

git commit --allow-empty -m "test: verify runner picks up jobs"
git push

Open https://github.com/YOUR_USERNAME/clearledger/actions. Within 30 seconds a workflow run should show Queued then In progress, not stuck on “Waiting for a runner.” If it waits more than 2 minutes, the labels are wrong. Edit the runner on GitHub and add clearledger.

If it shows Offline:

multipass exec clearledger -- sudo systemctl status actions.runner.*.service
multipass exec clearledger -- journalctl -u actions.runner.*.service --lines=50

What you proved: GitHub can now send work into your local lab environment.

1.3: Create the Infra Repo on GitHub

Now separate application code from deployment state. Stage 1 introduces a second GitHub repository alongside the clearledger app repo you pushed in §1.1.

You'll use two repositories for the rest of the lab:

Think of clearledger as the question “What is the application?”. Python services, Dockerfiles, tests, and the CI workflow. Think of clearledger-infra as “What exact version should be running in Kubernetes right now?”. Deployments, Services, ingress rules, and the image tags that point at Docker Hub.

Teams split these on purpose. If you edit README.md in clearledger, that is a documentation change. It shouldn't trigger a deployment.
If you change auth-service code, the pipeline builds a new image (for example tag abc123) and, only after scans pass, records that tag in clearledger-infra:

image: $DOCKER_USERNAME/clearledger-auth-service:abc123

That line is a deployment contract: Git now says the cluster should run abc123. In Stage 1, the cluster doesn't change yet (and you'll prove that in §1.6).
In Stage 2, ArgoCD watches clearledger-infra, compares Git to what is running, and syncs the cluster when they differ. The app repo is where work begins. The infra repo is what production is supposed to look like.

Private repos: what syncs where

This lab uses two GitHub repos. clearledger is your main project repo: app code, CI pipeline, docs, policies, and lab files. This repo can be private.

clearledger-infra contains only Kubernetes manifests. ArgoCD watches this repo and uses it to deploy the app. For beginners, make this repo public so ArgoCD can read it without extra authentication.

The flow looks like this:

clearledger
app code + infra/manifests/
        ↓
CI copies infra/manifests/
        ↓
clearledger-infra
Kubernetes manifests only
        ↓
ArgoCD syncs from this repo
        ↓
Kubernetes cluster

ArgoCD doesn't read the main clearledger repo. It only reads clearledger-infra. If clearledger is private, that is fine. If clearledger-infra is private, you must give ArgoCD GitHub credentials later. If you do not, ArgoCD may show ComparisonError.

Create the infra repo on GitHub:

1. Go to GitHub and then New Repository

2. Name it clearledger-infra

3. Choose Public

4. Don't add a README

5. Click Create

Later, the CI pipeline will update clearledger-infra automatically. In Stage 1, the pipeline doesn't run kubectl apply – it updates Git. In Stage 2, ArgoCD reads that Git repo and applies it to the cluster.

Before pushing: set your Docker Hub username in Kustomize (image tags are resolved here, not in deployment YAML):

# Replace YOUR_DOCKERHUB_USERNAME with the same value as $DOCKER_USERNAME from §0.3
sed -i.bak "s/YOUR_DOCKERHUB_USERNAME/${DOCKER_USERNAME}/g" infra/manifests/kustomization.yaml
rm -f infra/manifests/kustomization.yaml.bak

Push only the Kubernetes manifests from infra/manifests/ (not everything under infra/):

mkdir -p /tmp/clearledger-infra
cp -r infra/manifests /tmp/clearledger-infra/
cd /tmp/clearledger-infra
git init
git remote add origin https://github.com/YOUR_USERNAME/clearledger-infra.git
git add . && git commit -m "feat: initial manifests" && git push -u origin main
cd -

✋ Hands-on checkpoint: infra repo on GitHub (do this before §1.4)

On your laptop:

grep "docker.io/${DOCKER_USERNAME}/" infra/manifests/kustomization.yaml | wc -l
grep YOUR_DOCKERHUB_USERNAME infra/manifests/kustomization.yaml || echo "OK: placeholder replaced"

Expected: first command prints 4 (four image lines). Second prints OK: placeholder replaced, not four lines still saying YOUR_DOCKERHUB_USERNAME.

In the browser, open https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests and confirm with your eyes:

Which folders matter? You only pushed infra/manifests/ to GitHub, that's correct. Everything else in this repo stays local for now.

Some manifests for later stages (network policies, Vault extras) live under infra/deferred-by-stage/ in the clearledger repo. You'll apply those by hand when you reach that stage. Do not copy that folder into clearledger-infra, or ArgoCD will deploy things too early.

You might notice stages/stage-1-ci-pipeline/ has no copy of the manifests. That is normal: the lab doesn't duplicate YAML there. The canonical copy is infra/manifests/ in this repo, and the live GitOps copy is clearledger-infra on GitHub.

What you proved: Kubernetes config now has its own repo and Git history, separate from application code. CI will update clearledger-infra after each build, and your app repo stays for code and the pipeline file.

1.4: Set up GitHub Secrets

Go to github.com/YOUR_USERNAME/clearledger and then Settings, Secrets and variables, Actions, and New repository secret.

The workflow needs credentials for Docker Hub, GitHub, and image signing:

• Docker Hub, so it can push images.

• GitHub, so it can push image tag updates into clearledger-infra.

• Cosign, so it can sign the images after pushing them.

Do not paste these values into YAML files. Store them as GitHub Actions secrets.

Secret 1, DOCKER_USERNAME

This is just your Docker Hub username.

Example:

veeno-demo

Get it from Docker Hub: hub.docker.com, profile menu, Account Settings.

Secret 2, DOCKER_PASSWORD

This should be a Docker Hub access token, not your normal Docker Hub password.

Create it here:

hub.docker.com
→ Account Settings
→ Security
→ New Access Token
→ Description: clearledger-github-actions
→ Access permissions: Read, Write, Delete or Read/Write
→ Generate

Copy the token immediately. Docker Hub only shows it once.

Secret 3, INFRA_REPO_TOKEN

This is a GitHub Personal Access Token (PAT). The pipeline uses it to push commits to the second repo, clearledger-infra.

Create it here:

GitHub profile settings
→ Settings
→ Developer settings
→ Personal access tokens
→ Tokens (classic)
→ Click "Generate new token"
→ Choose "Generate new token (classic)"
→ If GitHub asks for your password or 2FA, complete it
→ Note: clearledger-infra-ci
→ Expiration: choose a lab-friendly value
→ Select scope: repo
   This allows the pipeline to push to clearledger-infra.
→ Generate token

Copy the token immediately. GitHub only shows it once.

For this lab, repo scope is the simplest option. In production, you would use tighter permissions, such as a fine-grained token limited to only clearledger-infra.

Secrets 4 and 5, COSIGN_PRIVATE_KEY and COSIGN_PASSWORD

Cosign signs container images after the pipeline pushes them to Docker Hub. Later, Stage 4 uses the public key with Kyverno so the cluster can verify that images came from your trusted pipeline.

Generate the key pair on your host machine, not inside the Multipass VM:

# macOS: brew install cosign
# Linux/WSL2: curl -sSL -o cosign https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 && chmod +x cosign && sudo mv cosign /usr/local/bin/
cosign generate-key-pair

This creates:

cosign.key   # private key — never commit this
cosign.pub   # public key — keep for later Kyverno verification

When Cosign asks for a password, enter one and save it in your password manager. If you already generated a key without a password, regenerate it with a password for this lab.

Add these five secrets to the clearledger repo, not clearledger-infra:

Repository variables (not secrets) (optional) toggles for later stages. Add under Settings, Secrets and variables, Actions, Variables:

Don't add either variable in Stage 1. If you set them now, CI will try to refresh ArgoCD or run ZAP before the cluster is ready, and the pipeline output gets harder to read. The guide calls out the exact moment to turn each one on – you only need to remember that both exist.

What you proved: the pipeline can authenticate to external systems without hardcoding credentials in the repo.

1.5: Understand the Pipeline Before Activating it

Don't treat the workflow file as magic. Open .github/workflows/ci.yaml and read it before you run it.

The pipeline has two responsibilities:

1. Prove the code and images are safe enough to publish.

2. Update the infra repo with the new image tags.

Here's the security flow first:

Developer pushes code to GitHub
        ↓
GitHub Actions starts workflow
        ↓
Self-hosted runner inside the Multipass VM picks up the job
        ↓
1. Scan secrets (Gitleaks)
        ↓
2. Run code security scans (Semgrep) + IaC scan (Checkov) — parallel
        ↓
3. Prepare scanners (install Trivy/Syft/Grype/Cosign once; refresh Trivy DB once)
        ↓
4. BUILD: docker build all four services (local tags only; nothing hits Docker Hub yet)
        ↓
5. SCAN: Trivy on all images; Syft + Grype SBOM on auth-service; upload evidence
        ↓
6. PUBLISH: push to Docker Hub + Cosign sign (only if scan passed)
        ↓
7. UPDATE MANIFESTS: commit new image tags to clearledger-infra

Build, scan, publish (prod-style gates)

Real teams never push first and scan later. The pipeline separates three concerns into three jobs in .github/workflows/ci.yaml:

You do not run scripts/ci-publish-image.sh yourself before pushing code. GitHub Actions checks out the repo and calls it inside publish-images.

Why can build-images and scan-images be separate jobs? Each job is a fresh checkout on GitHub-hosted runners. They don't share a disk. On your self-hosted runner, all three jobs run on the same Multipass VM and use the same Docker engine.

Job 1 runs docker build and leaves the images on that machine. Job 2 runs Trivy against those same local images: no upload, no download. Job 3 pushes to Docker Hub only if the scan passed.

That's a practical lab setup: one persistent build machine with Docker installed, like a dedicated CI worker in a real office. In Stage 8 (AWS), the pipeline uses GitHub-hosted runners instead: there, build-images saves the images to a file (images.tar) and passes that file to the next job as a workflow artifact, because those runners are throwaway VMs with no shared Docker cache.

Then comes the GitOps handoff:

Secure images now exist in Docker Hub
        ↓
Runner checks out clearledger-infra from GitHub
        ↓
Deployment YAML image tags are updated
        ↓
Runner commits and pushes back to clearledger-infra
        ↓
Stage 1 ends here

Here's how the image tag ties to your code: every pipeline run is triggered by a git commit. GitHub gives that commit a unique ID called the SHA (a long hex string like a1b2c3d4e5f6789…). The workflow sets IMAGE_TAG to that SHA and uses it everywhere:

1. Build: docker build -t clearledger-auth-service:a1b2c3d4…

2. Publish: push to Docker Hub as YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4…

3. Update manifests: kustomize edit set image …:a1b2c3d4… in clearledger-infra

4. Commit message: ci: deploy a1b2c3d4… — all gates passed

If production is running YOUR_DOCKERHUB_USERNAME/clearledger-auth-service:a1b2c3d4, you can copy that a1b2c3d4 tag, open GitHub, and instantly find the exact commit that built that image. There's no guessing and no wondering if latest changed. Every deployed image points back to one specific version of the code, making rollbacks and debugging much easier.

The Kustomize placeholder

auth-service/deployment.yaml uses a label instead of a real image address:

image: clearledger/auth-service:gitops

That label isn't on Docker Hub. It tells Kustomize where to substitute. The real address lives in kustomization.yaml:

images:
  - name: clearledger/auth-service          # matches the label above
    newName: docker.io/YOUR_DOCKERHUB_USERNAME/clearledger-auth-service
    newTag: abc123def456…                   # real commit SHA — CI writes this

When ArgoCD deploys, kustomize build swaps the label for the full address.

You edit kustomization.yaml once in §1.3 to set your Docker Hub username in newName:. After that, CI writes newTag: automatically on every green push. You never touch it by hand.

Stage 1: CI updates GitHub, not the cluster

After a green pipeline run, three things are true:

• New images exist on Docker Hub

• clearledger-infra on GitHub has new SHAs in kustomization.yaml

• Your Kubernetes cluster is unchanged. Still running whatever Stage 0 left there

CI never runs kubectl apply. It only commits to clearledger-infra. That's the whole Stage 1 lesson: build and scan are automated, but deploy is not: yet. Stage 2 installs ArgoCD, which reads clearledger-infra and updates the cluster for you.

Kubernetes Checkov runs in Stage 1 but does not block the pipeline. It uploads findings so you can see hardening work ahead. Stage 4 turns those kinds of rules into cluster enforcement with Kyverno.

Jobs run on your self-hosted runner (runs-on: [self-hosted, clearledger]). Both ENABLE_ARGOCD_SYNC and ENABLE_DAST are unset in Stage 1. See §1.4 for when each gets flipped.

If a job fails, start with docs/troubleshooting.md before editing the workflow.

Stage 1 security posture: what blocks vs what waits

Note that stage 1 is not “security off.” Some gates stop the pipeline while others run for evidence and tighten in later stages.

Blocks the pipeline today:

• Gitleaks (secrets in Git)

• Semgrep (SAST on Python)

• Checkov on Dockerfiles

• Trivy (fixable HIGH/CRITICAL CVEs in images)

• Grype on auth-service SBOM (fixable HIGH+)

• Manifest update to clearledger-infra (must succeed)

Runs but doesn't block yet:

• Checkov on Kubernetes manifests – enforced in Stage 4 (Kyverno)

• Cosign sign + SLSA attest – enforced in Stage 4 (unsigned images rejected)

• Syft SBOM generation – supply-chain evidence. You'll purposely break gates in Stage 3.

• ArgoCD refresh – Stage 2 (ENABLE_ARGOCD_SYNC=true)

• DAST / ZAP – Stage 3 (ENABLE_DAST=true)

If you forget which stage fixes what, search this guide for “Stage 1 security posture” or follow the stage order: Stage 3 breaks gates on purpose, Stage 4 connects Checkov findings to Kyverno, Stage 5 moves secrets off Git, Stage 6 adds runtime detection, Stage 7 adds monitoring dashboards.

Run make check-3 and make check-4 after those stages to confirm hardening landed.

Design intent: Stage 1 proves CI can build, scan, push, and update Git without you touching Docker manually. Later stages turn evidence into enforcement. The relaxations here are deliberate.

1.6: Activate the Pipeline

Run this in the clearledger app repo, not clearledger-infra.

§1.3 created clearledger-infra with only Kubernetes manifests. It has no .github/workflows/ and no pipeline. If your shell prompt says clearledger-infra, or you used /tmp/clearledger-infra, you're in the wrong place.

cd /path/to/clearledger    # the app repo you pushed in §1.1

git remote -v              # must show .../clearledger.git — NOT clearledger-infra

ls .github/workflows/ci.yaml   # must exist before you commit

The pipeline file already lives at .github/workflows/ci.yaml. Push any small change to clearledger on main:

echo "# Pipeline activated $(date)" >> README.md
git add README.md
git commit -m "ci: activate GitHub Actions pipeline"
git push origin main

Watch the run at: https://github.com/YOUR_USERNAME/clearledger/actions (app repo Actions tab, not the infra repo).

When the pipeline succeeds, it updates clearledger-infra for you. You don't need to push anything to the infra repo by hand for this step.

Expected Output: all jobs green in about 8 minutes.

✓ Build + Scan auth-service
✓ Build + Scan ledger-service
✓ Build + Scan notification-service
✓ Build + Scan frontend
✓ Update manifests → GitHub

DAST and the ArgoCD refresh step show as skipped: this is expected, because Both toggles are unset until later (see §1.4).

Note: this lab includes .gitleaksignore because some intentional demo secrets are already present in Git history. Gitleaks still runs normally. The ignore file only suppresses known lab fingerprints. Don't add new findings to it unless you've confirmed they're intentional test data.

Click into the job logs and look for the story. Don't just wait for green:

• Docker login succeeded

• Each service image built and pushed to Docker Hub

• clearledger-infra was checked out

• Deployment YAMLs were updated with the new SHA tag

• A commit was pushed back to clearledger-infra

After the pipeline succeeds, open https://github.com/YOUR_USERNAME/clearledger-infra and look at the deployment manifests. The image tags should now use the current commit SHA.

Now check the cluster:

kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}' && echo

You may still see the old image. That's expected. This is the most important learning in Stage 1:

GitHub pipeline succeeded.
Docker Hub has new images.
clearledger-infra has new image tags.
The Kubernetes cluster did not update automatically.

That's not a failure. It's the deployment gap. Stage 1 automated the build, but no controller is watching the infra repo yet. Stage 2 installs ArgoCD to close that gap.

1.7 — Hands-on Checkpoint: Prove Stage 1 is Really Done

Don't rely on a green workflow badge alone. Run each check yourself:

1. Infra repo still has app secrets (critical for Stage 2)

Open https://github.com/YOUR_USERNAME/clearledger-infra/tree/main/manifests/auth-service, secret.yaml must be visible.

On your laptop:

git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra
grep secretKeyRef /tmp/verify-infra/manifests/auth-service/deployment.yaml
grep secret.yaml /tmp/verify-infra/manifests/kustomization.yaml
rm -rf /tmp/verify-infra

Expected: secretKeyRef in deployment output. kustomization lists auth-service/secret.yaml and ledger-service/secret.yaml. If secrets are missing, re-push §1.3 manifests before Stage 2.

2. Kustomize image tags updated by CI

git clone --depth 1 https://github.com/YOUR_USERNAME/clearledger-infra.git /tmp/verify-infra
grep newTag /tmp/verify-infra/manifests/kustomization.yaml
rm -rf /tmp/verify-infra

Expected: newTag is a 40-character git SHA (or your commit hash), not still v0.1.0 only: unless you haven't pushed since §0.3.

3. Docker Hub has signed images from this pipeline

Open hub.docker.com then clearledger-auth-service then Tags. The latest tag should match the SHA from step 2.

4. (Cluster unchanged (deployment gap) intentional)

kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}' && echo

Expected: still your Stage 0 tag (for example, veeno-demo/clearledger-auth-service:v0.1.0), not the new SHA. That proves CI didn't touch the cluster.

5. Runner still idle

GitHub, Settings, Actions, Runners, clearledger-runner, Idle.

make check-1

All five pass, onto Stage 2.

What You Learned in Stage 1

• CI removes your laptop from the build process. Builds become repeatable, visible, and tied to Git commits.

• A runner is the worker, not the pipeline itself. GitHub schedules the job, the self-hosted runner executes it inside your VM.

• Artifacts and desired state are different things. Docker Hub stores built images. clearledger-infra on GitHub stores the Kubernetes manifests that say which image should run.

• Good pipelines don't secretly mutate clusters. This pipeline updates Git instead of running kubectl.

• The gap that remains: the infra repo changed, but the cluster didn't. Someone still has to apply the change manually. Stage 2 fixes that with GitOps.

What you can now put on your CV / say in an interview:

Built a CI pipeline on a self-hosted GitHub Actions runner that builds and pushes container images on every push, and can debug a workflow that fails before any job is created.

make snapshot STAGE=1 && make snapshots. Confirm clearledger.stage1. See How to Save Your Progress.

Stage 2 — GitOps with ArgoCD

From this point on, Git is in charge. Whatever is written in the infrastructure repository is what should be running. If someone changes the cluster by hand, ArgoCD notices the difference and changes it back to match Git.

Goal: Install ArgoCD so it watches clearledger-infra and deploys changes to the cluster. The CI pipeline only updates the Git repository, it never connects to Kubernetes or runs kubectl commands.

Here's a more conversational, compressed version:

Am I ready for Stage 2?

Before moving on, finish §1.6, then run:

make check-1

grep secretKeyRef infra/manifests/auth-service/deployment.yaml

grep vault.hashicorp infra/manifests/auth-service/deployment.yaml && echo "STOP: Vault annotations present" || echo "OK"

You should see:

• check-1 passes

• secretKeyRef is present

• OK (no Vault annotations yet)

Quick checklist:

• clearledger-infra contains auth-service/secret.yaml and ledger-service/secret.yaml

• Your self-hosted runner is Idle with the clearledger label

• ENABLE_ARGOCD_SYNC isn't set yet (you'll enable it after installing ArgoCD)

You're done with Stage 2 when make check-2 passes and http://argocd.local shows ArgoCD syncing clearledger.

Finally, save your progress:

make snapshot STAGE=2
make snapshots

Confirm that clearledger.stage2 appears in the snapshot list.

What You Need to Know First

The gap from Stage 1: CI already builds images and updates clearledger-infra. The cluster didn't change until someone ran kubectl. This stage closes that last step.

push code → CI updates clearledger-infra → ArgoCD syncs cluster

Pre-sync Checklist: Run Before argocd app sync

ArgoCD applies whatever is in clearledger-infra. Wrong content causes red pods. Re-run the §1.7 checkpoint table to confirm GitHub-side content is still correct, then verify the laptop side:

# Application manifest must point at YOUR infra repo
grep repoURL stages/stage-2-gitops/argocd/clearledger-app.yaml

# Stage 0 workloads still healthy before ArgoCD takes over
kubectl get pods -n clearledger
curl -s -o /dev/null -w "%{http_code}" http://clearledger.local/auth/health

Expected: repoURL contains your GitHub username, all app pods Running, curl 200. Only when both pass should you install ArgoCD and sync below.

kubectl create namespace argocd 2>/dev/null || true

kubectl apply -n argocd --server-side --force-conflicts -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=argocd-server -n argocd --timeout=180s

Why --server-side --force-conflicts? Argo CD ships a very large applicationsets.argoproj.io CRD. A normal kubectl apply tries to stash the whole thing in an annotation, hits a 256 KiB limit, and errors with metadata.annotations: Too long. Server-side apply avoids that. It's how Argo CD expects you to install.

Get the admin password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d && echo

Configure Argo CD for your NGINX ingress

The browser talks HTTPS to ingress and ingress talks plain HTTP to the Argo CD server. Without this, the UI often breaks with 503 or ERR_TOO_MANY_REDIRECTS on live-update URLs (/api/v1/stream/*).

kubectl apply -f stages/stage-2-gitops/infra/argocd-cmd-params.yaml

kubectl apply -f stages/stage-2-gitops/infra/argocd-ingress.yaml

kubectl rollout restart deployment/argocd-server -n argocd

kubectl rollout status deployment/argocd-server -n argocd --timeout=180s

Expected in argocd-cmd-params-cm: server.insecure: "true", server.grpc.web: "true", server.url: https://argocd.local.

Open https://argocd.local. Login: admin and the password from above. Accept the self-signed certificate warning if the browser shows one.

Expected: The Applications page loads. In the browser console (F12 Console), you shouldn't see 401 or ERR_HTTP2_PROTOCOL_ERROR. If the UI looks fine in a normal window, you're done: incognito isn't required.

If login fails with 401 Unauthorized (often after a config change or a bad earlier login), try a private/incognito window or clear site data for argocd.local, then log in again. Still stuck? See troubleshooting.md. ArgoCD.

Connect ArgoCD to the infra repo and apply the Application manifest:

1. Edit stages/stage-2-gitops/argocd/clearledger-app.yaml

Set spec.source.repoURL to your infra repo (your GitHub username, not git config user.name).

2. Connect ArgoCD to your infrastructure repository:

This gives ArgoCD permission to watch clearledger-infra for new commits. Whenever the deployment manifests change, ArgoCD will update the cluster automatically.

# macOS: brew install argocd
argocd login argocd.local --username admin --password YOUR_PASSWORD --insecure --grpc-web

# Public repo
argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git --grpc-web

# Private repo — PAT from Stage 1 §1.4 (you saved it as GitHub secret INFRA_REPO_TOKEN)
export INFRA_REPO_TOKEN='ghp_...'   # paste here; GitHub only shows it once at creation
argocd repo add https://github.com/YOUR_USERNAME/clearledger-infra.git \
  --username git --password "$INFRA_REPO_TOKEN" --grpc-web

Verify that Argo CD can reach the repo (do this before applying the Application):

argocd repo list --grpc-web

Look for your clearledger-infra URL with TYPE git and connection Successful. If it shows Failed or the repo is missing, Argo CD can't sync. Fix credentials before Stage 4 or any stage that depends on GitOps.

After a VM restore or Argo CD reinstall, you may need to run argocd repo add again (credentials are stored in the cluster, not in Git).

3. Apply and sync:

kubectl apply -f stages/stage-2-gitops/argocd/clearledger-app.yaml

argocd app sync clearledger --grpc-web

How to Read the Argo CD UI

After sync, open the clearledger application in the tree view. Three badges at the top tell you almost everything:

APP HEALTH: Healthy. Kubernetes thinks the workloads are running. Pods are up (or still starting if it says Progressing).

SYNC STATUS: Synced: the cluster matches clearledger-infra on GitHub at the commit shown (for example, main (2c88aa1)). Git is the source of truth and Argo CD applied it.

LAST SYNC: Succeeded: the most recent apply from Git worked. If this failed, click it for the error.

The resource tree below is the same app broken into pieces: namespace, secrets, services, deployments, ingress, and so on Green checkmarks = applied from Git. Click any box (for example, deploy/auth-service) then Live Manifest vs Desired to see what Argo CD thinks should run.

Quick "is the app actually working?" test (outside Argo CD):

curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health

200 = the app is reachable end-to-end, not only "green in Argo CD."

When something is wrong: HEALTH goes Degraded or Progressing for a long time, SYNC goes OutOfSync, and a resource in the tree turns red. Click that resource and then Events or Logs. The kubectl checks below double-check the same thing from the terminal.

Confirm ArgoCD is watching all workloads (not only ingress):

argocd app resources clearledger --grpc-web | grep Deployment

Pass looks like your output:

apps    Deployment    clearledger    auth-service            No
apps    Deployment    clearledger    frontend                No
apps    Deployment    clearledger    ledger-service          No
apps    Deployment    clearledger    notification-service  No
apps    Deployment    clearledger    redis                   No

This command shows the Deployments that ArgoCD is managing for ClearLedger. You should see auth-service, ledger-service, notification-service, frontend, and redis. That means ArgoCD reads the full kustomization.yaml from clearledger-infra/manifests, not just one file.

The last column is ORPHANED. No is good. It means ArgoCD knows this resource belongs to the ClearLedger app. You only need to worry if one of the Deployments is missing, or if ArgoCD shows OutOfSync, Degraded, or red resources in the UI.

✋ Hands-on checkpoint: first sync healthy

Run these four checks. Pass looks like this:

kubectl get pods -n clearledger
# Every app pod 1/1 Running (postgres/redis may show older RESTARTS from VM reboots — OK)

kubectl get application clearledger -n argocd \
  -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'
# sync=Synced health=Healthy

curl -s -o /dev/null -w "%{http_code}\n" http://clearledger.local/auth/health
# 200

kubectl logs -n clearledger deploy/auth-service --tail=5 2>/dev/null | head -3
# Lines like: GET /health HTTP/1.1" 200 OK
# Bad sign: DATABASE_URL is not set

If all four pass then, Stage 2 first sync is done. Continue to Enable CI, and ArgoCD handoff below, then make check-2 and make snapshot STAGE=2.

How to Enable the CI to ArgoCD Handoff

In Stage 1, the pipeline updated clearledger-infra, but it didn't update the cluster. That was intentional.

Now ArgoCD is installed, so you can let the pipeline tell ArgoCD to check for changes after each successful run.

In GitHub, open your clearledger repo and go to Settings, Secrets and variables, Actions, Variables, and then New repository variable.

Add:

From now on, a green pipeline does two things:

1. Updates clearledger-infra with the new image tag

2. Asks ArgoCD to sync the cluster

If the pipeline can't trigger ArgoCD immediately, that's usually okay. ArgoCD checks clearledger-infra on its own every few minutes, so it should still pick up the new Git change.

Leave ENABLE_DAST unset for now. You enable that in Stage 3 after the app is stable at clearledger.local.

If the Argocd UI Shows Red Pods or "Progressing" (Read This Before the Screenshot)

This is a common first-sync surprise, not a broken install.

Why it happens in Stage 2

ArgoCD syncs whatever is in clearledger-infra. Deployments must use secretKeyRef (Stages 2–4), not Vault injection. If your infra repo has Vault annotations from an older lab copy, auth/ledger crash with DATABASE_URL is not set until Stage 5.

Network policies belong to Stage 6. In the main clearledger repo, they live in infra/deferred-by-stage/stage-6-runtime-security/netpol/, not in infra/manifests/. Don't copy them into clearledger-infra during Stage 2.

If manifests/netpol/ is still in your clearledger-infra repo on GitHub (from an older copy of the lab), ArgoCD will keep applying it. Those policies use default-deny and break DNS for new pods, so you see red 0/1 pods and Progressing health.

Fix for Stage 2

Do both steps. Deleting only in the cluster is not enough: ArgoCD recreates policies from Git on the next sync.

Step 1: remove from clearledger-infra on GitHub

Delete the folder manifests/netpol/ and commit: chore: defer network policies to Stage 6.

Step 2: sync and restart

argocd app sync clearledger --grpc-web
kubectl delete networkpolicy -n clearledger --all   # safe once Git no longer has netpol
kubectl rollout restart deployment/auth-service deployment/ledger-service -n clearledger
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"

Network policies stay in clearledger under infra/deferred-by-stage/ until you apply them in Stage 6.

When that looks good, continue below.

When ArgoCD finishes syncing, open the clearledger app in the ArgoCD UI. You should see green Healthy and Synced badges. The app should point to your clearledger-infra repo, use the manifests path, and deploy into the clearledger namespace.

Then open the app tile. The resource tree should show your deployments, services, and ingress with no red resources.

You can confirm the same thing from the terminal:

argocd app get clearledger --grpc-web

Look for Sync Status: Synced and Health Status: Healthy.

ArgoCD stuck OutOfSync

Normal path: CI copies full manifests + updates Kustomize tags, ArgoCD auto-syncs within ~3 minutes.

If still OutOfSync after 10+ minutes:

make fix-argocd

This re-syncs canonical manifests to clearledger-infra (Kustomize SHAs preserved), re-applies the Application, and triggers a hard refresh. Don't kubectl apply deployments: fix Git, let ArgoCD sync.

kubectl annotate application clearledger -n argocd 

argocd.argoproj.io/refresh=hard --overwrite

argocd app sync clearledger --grpc-web --prune

kubectl get application clearledger -n argocd -o jsonpath='sync={.status.sync.status} health={.status.health.status}{"\n"}'

Take a screenshot of that view: the app tile or the resource tree is fine. That’s your portfolio proof that GitOps is actually running.

Prove ArgoCD Self-Healing

Now prove that Git is the source of truth.

In this demo, you'll change the running cluster by hand. You will not change Git. ArgoCD should notice that the cluster no longer matches clearledger-infra, then change it back.

Before you start, make sure the app is healthy and ArgoCD is managing the deployments:

argocd app resources clearledger --grpc-web | grep Deployment

Manually change the auth-service image in the cluster:

# Manually change the image in the cluster only (Git stays the same)
kubectl set image deployment/auth-service \
  auth-service=$DOCKER_USERNAME/clearledger-auth-service:fake-tag \
  -n clearledger

Check ArgoCD:

# ArgoCD should flip to OutOfSync within a minute or two
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"

Wait for ArgoCD to fix the cluster. The fake image tag may briefly cause an image pull error. That's expected in this demo.

# Wait for selfHeal (default sync interval is ~3 minutes)
sleep 180

Confirm the image was changed back to the Git version:

# Cluster image should match clearledger-infra again — Git was never edited
kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

If the image changed back, ArgoCD self-healing worked. You changed the cluster by hand, but ArgoCD restored it to match clearledger-infra.

That's GitOps: Git says what should run, and ArgoCD keeps the cluster matching Git.

make check-2

How to Roll Back a Bad Deploy

You just proved that ArgoCD reverts unauthorized cluster changes. Now flip it: what if you pushed a bad commit yourself? GitOps rollback isn't a button. It's a Git operation. This section explains why, shows you both methods, and has you practice each one before you need them under pressure.

How you know you need to roll back

These symptoms appearing within minutes of a push to clearledger-infra point at a bad commit:

• Pods stuck in CrashLoopBackOff or Error. Check with kubectl get pods -n clearledger.

• kubectl logs <pod> -n clearledger --previous shows startup errors that weren't there before.

• ArgoCD health flips from Healthy to Degraded or stays on Progressing. Check with argocd app get clearledger --grpc-web.

• The app returns 5xx errors or login stops working. Check with curl -I http://clearledger.local/health.

If this happens right after a push, roll back first. Once the app is stable again, investigate the bad commit.

Why ArgoCD rollback isn't just a button

ArgoCD has a rollback button in the UI and an argocd app rollback command. Both work. But only if you understand the interaction with selfHeal.

Your Application (stages/stage-2-gitops/argocd/clearledger-app.yaml) is configured with:

syncPolicy:
  automated:
    selfHeal: true

ArgoCD keeps the cluster matched to Git. In this lab, Git means clearledger-infra.

If someone changes the cluster by hand, ArgoCD treats that as drift and changes it back to match Git.

This also affects rollback. The ArgoCD UI rollback changes the cluster, but it doesn't change Git. If clearledger-infra still points to the bad version, and self-heal can bring the bad version back.

The safer GitOps rollback is to change Git with git revert in clearledger-infra. Then ArgoCD syncs the cluster to the reverted, good version.

If you need an emergency UI rollback, turn off auto-sync first, roll back in ArgoCD, then fix Git afterward.

Method 1: Git revert (preferred, always try this first)

This is the GitOps way. You don't touch the cluster. You change Git, and ArgoCD syncs the fix.

When to use: You have a few minutes and can identify the bad commit in clearledger-infra.

How it works:

Bad commit pushed to clearledger-infra
        ↓
ArgoCD auto-synced it (cluster is now broken)
        ↓
You run: git revert <bad-commit> && git push
        ↓
ArgoCD auto-syncs the revert (cluster is fixed, selfHeal works with you)
        ↓
Git history shows the bad deploy AND the revert, full audit trail

Step-by-step:

# 1. Go to your clearledger-infra repo (wherever you cloned it)
cd ~/clearledger-infra  # adjust path if you cloned elsewhere
git pull                # make sure you are up to date

# 2. Find the bad commit
git log --oneline -10

# Output looks like:
# abc1234 update ledger-service image to v1.4.0   ← this broke prod
# def5678 update auth-service image to v1.3.1     ← was fine
# 9a1b2c3 add vault rotation cronjob

# 3. Revert it — this creates a NEW commit, it does not delete history
git revert abc1234 --no-edit

# 4. Push — ArgoCD picks it up automatically within ~3 minutes
git push

# 5. Confirm the cluster recovered
kubectl get pods -n clearledger
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy

This method is preferred because it fixes the source of truth: clearledger-infra.

After you push the revert, ArgoCD sees the new Git state and syncs the cluster to it. Nothing fights you because Git and the cluster are supposed to match.

It also leaves a clear history. Git shows the bad deploy, the revert, who made both changes, and when they happened. That's easier to debug, easier to review, and better for compliance.

Method 2: Emergency ArgoCD rollback (when the cluster is on fire)

Use this if the cluster is broken right now and you don't have time to push a Git fix. It pins the cluster to a previous known good deployment immediately. You'll still fix Git afterward. This isn't a permanent fix.

When to use: Incident in progress. Pods are crashing, users are affected, and you need the cluster back to a known good state in under 30 seconds.

Before you start: confirm your ArgoCD CLI session is still valid. If it expired, re-login first: an expired session will silently fail every command below.

argocd account get-user-info --grpc-web
# If you see "Unauthenticated", re-login:
ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d)

argocd login argocd.local --username admin --password "$ARGOCD_PASSWORD" \
  --insecure --grpc-web

Step 1: Disable auto-sync (critical). Skip this and selfHeal will undo your rollback within 3 minutes.

argocd app set clearledger --sync-policy none --grpc-web
# Confirm: automated sync is now off
argocd app get clearledger --grpc-web | grep "Sync Policy"
# Expected: Sync Policy: <none>

Step 2: Find the last known-good deployment ID

argocd app history clearledger --grpc-web

# Output looks like:
# ID   DATE                           REVISION
# 9    2026-06-05 10:12:00 +0000 UTC  abc1234  ← bad deploy (current)
# 8    2026-06-04 14:46:06 +0000 UTC  def5678  ← known good
# 7    2026-06-01 20:53:19 +0000 UTC  9a1b2c3

# Or check via kubectl (no argocd CLI needed):
kubectl get application clearledger -n argocd \
  -o jsonpath='{range .status.history[*]}{.id}{"\t"}{.deployedAt}{"\t"}{.revision}{"\n"}{end}'

Use the ID (the number on the left), not the SHA.

Step 3: Roll back to the good ID

argocd app rollback clearledger 8 --grpc-web

Step 4: Confirm the cluster is stable

kubectl get pods -n clearledger
# All pods should be Running

argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Sync Status:   OutOfSync  ← expected — cluster is at rev 8, Git is still at the bad HEAD
# Health Status: Healthy    ← this is what matters right now

OutOfSync is correct and expected at this point. The cluster is running the old good revision. Git still has the bad commit. You'll fix that next.

Step 5: Fix Git (don't leave it broken)

cd ~/clearledger-infra
git pull
git revert <bad-commit-sha> --no-edit
git push

Step 6: Re-enable auto-sync

argocd app set clearledger \
  --sync-policy automated \
  --self-heal \
  --auto-prune \
  --grpc-web

# Trigger an immediate sync so you do not wait for the next auto-check
argocd app sync clearledger --grpc-web

# Confirm everything is clean
argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy

Never leave auto-sync disabled longer than the incident. It's your drift-detection and tamper-evidence mechanism: without it, unauthorized kubectl changes go undetected. Re-enable it the moment you push the Git fix.

Practise the rollback now (before you need it under pressure)

Don't wait for a real incident to run this for the first time. The steps below simulate a bad image tag deploy and walk you through Method 1 (the preferred path).

Step 1: Push a bad image tag to clearledger-infra

cd ~/clearledger-infra
git pull

# Edit manifests/notification-service/deployment.yaml
# Change the image tag to a tag that does not exist, e.g.:
#   image: docker.io/$DOCKER_USERNAME/clearledger-notification-service:broken-tag

# Commit and push it
git add manifests/notification-service/deployment.yaml
git commit -m "test: simulate bad deploy with nonexistent image tag"
git push

Step 2: Watch ArgoCD sync the bad state

# Give ArgoCD ~3 minutes to pick it up, or trigger immediately:
argocd app sync clearledger --grpc-web

# Watch the notification-service pod fail
kubectl get pods -n clearledger -w
# You will see: notification-service pod stuck in ImagePullBackOff or ErrImagePull

Step 3: Roll back using Method 1

cd ~/clearledger-infra

# Revert the bad commit
git revert HEAD --no-edit
git push

# ArgoCD will auto-sync — or trigger it:
argocd app sync clearledger --grpc-web

# Watch pods recover
kubectl get pods -n clearledger -w
# notification-service should return to Running

Step 4: Verify

argocd app get clearledger --grpc-web | grep -E "Sync Status|Health Status"
# Expected: Sync Status: Synced, Health Status: Healthy

kubectl get pods -n clearledger
# All pods Running, no ImagePullBackOff

You have now practised a rollback end-to-end. The git revert commit is permanently in the infra repo's history: a real audit record of a simulated recovery.

Quick reference

Use Method 1 (git revert) when:

• A bad image tag or manifest was pushed to clearledger-infra and you have a few minutes

• Any config change in the infra repo caused pods to break

• This is almost always the right answer. It's fast, safe, and leaves a clean audit trail.

Use Method 2 (emergency ArgoCD rollback) when:

• The cluster is broken right now, users are affected, and you need it stable in under 30 seconds

• You're not yet sure which commit caused the problem and need time to investigate: roll back to stabilise, then use git log to find the culprit, then fix forward with Method 1

Neither method applies when a pod is crashing but nothing was pushed to the infra repo recently. This isn't a rollback problem. Check kubectl logs, Vault connectivity, and network policies instead.

revisionHistoryLimit: 10 in stages/stage-2-gitops/argocd/clearledger-app.yaml means ArgoCD always has 10 previous deployments available for emergency rollback. Increase it if your release cadence is high.

What You Learned in Stage 2

• What GitOps means: Git is the single source of truth, and a tool enforces it

• What ArgoCD does: watches Git, compares it to the cluster, corrects drift automatically

• How the full flow works now: push code, CI builds image, CI updates infra repo, and ArgoCD syncs cluster.

• No one runs kubectl to deploy anymore. The pipeline updates Git, ArgoCD does the rest.

• How to roll back safely: git revert in the infra repo is the correct answer, while ArgoCD emergency rollback is the break-glass option. You must disable auto-sync first or selfHeal will silently undo it.

What you can now put on your CV / say in an interview:

Implemented GitOps with ArgoCD so cluster state is driven from Git, with drift detection, auto-sync, and a Git-based rollback of a bad deploy.

make snapshot STAGE=2 && make snapshots. Confirm clearledger.stage2. See How to Save Your Progress.

Stage 3 — Security Gates

Every push runs security checks. Some failures stop the pipeline right away. Others you learn from now and enforce in the cluster later (Stage 4).

Goal: understand six scanners: what each one looks at, what it catches, and how to read a failure. You'll break each gate on purpose (§3.4) so a failed CI job isn't a surprise.

Ready for Stage 3?

• make check-2 passes

• ENABLE_ARGOCD_SYNC=true on GitHub (you set this in Stage 2)

• ENABLE_DAST still unset (turn on later in this stage if you want)

• Argo CD at http://argocd.local shows Synced

• Optional: skim Stage 1 security posture. Stage 1 already ran many of these tools

Done when: make check-3 passes and you triggered each gate once (§3.4). Then make snapshot STAGE=3 and make snapshots.

What You Need to Know First

One tool isn't enough. Each scanner guards a different layer:

• Gitleaks: secrets in code or Git history (API keys, tokens)

• Semgrep (SAST): bugs in your Python/JS source (injection, unsafe patterns)

• Trivy (SCA + images): finds known security vulnerabilities (called CVEs) in your Python/Node.js packages and Docker images. A CVE (Common Vulnerabilities and Exposures) is a publicly tracked software security flaw with a unique identifier.

• Checkov (IaC): misconfigurations in Dockerfiles, Kubernetes manifests, and Stage 8 Terraform.

• Cosign: proves images were built and signed by your pipeline

What blocks CI today: secrets, bad code (SAST), vulnerable images, Dockerfile issues on production images.

What waits for later: Some Kubernetes issues are only reported in Stage 1. They show you what still needs hardening. In Stage 4, Kyverno turns the important rules into real cluster enforcement, so unsafe workloads are blocked before they run.

When you git commit, hooks on your laptop can scan first (pre-commit). When you git push, GitHub Actions scans again on the runner. Same idea twice: catch mistakes before they waste a 10-minute pipeline. Pre-commit is optional to install, as CI always runs on push either way.

Enable DAST (Optional After Stage 2)

DAST (Dynamic Application Security Testing) scans the running app at http://clearledger.local. It was off in Stages 1–2 on purpose: Stage 1 never deployed to the cluster, and Stage 2 was about getting GitOps healthy first.

If make check-2 passes and curl http://clearledger.local/auth/health returns 200, you can turn DAST on:

Go to GitHub and into your clearledger repo. Then go to Settings, Secrets and variables, Actions, Variables, and New repository variable:

Push a small commit (or re-run the last workflow on main). The DAST (OWASP ZAP + fintech API tests) job should run instead of skipped. A failed ZAP scan is a real finding to investigate. Skipped before this step only means the toggle was off.

3.1: Install Pre-commit Hooks

# macOS (Homebrew — avoids PEP 668 "externally-managed-environment" from pip3):
brew install pre-commit

# Linux/WSL2:
# sudo apt install -y pre-commit
# or: python3 -m pip install --user pre-commit

pre-commit install
pre-commit run --all-files

If a hook fails, read the error first. Gitleaks and Ruff should pass before you commit. Some YAML or Terraform hook issues may come from later-stage files. If that happens, continue with the stage instructions and use troubleshooting.md for Gitleaks or CI scanner failures.

Test it catches secrets locally before CI does:

echo 'AWS_SECRET = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' >> app/auth-service/main.py
git add app/auth-service/main.py && git commit -m "test"

# Gitleaks fires and blocks the commit — see "What you should see" below

git restore --staged app/auth-service/main.py

git checkout app/auth-service/main.py

The commit was blocked before it even reached Git. If the pre-commit hook wasn't installed, that fake AWS key would be in your Git history permanently (even if you delete the line later, Git remembers).

If you already did Cosign in Stage 1, that counts. Stage 3 doesn't require regenerating keys. Confirm that infra/cosign.pub exists and GitHub has COSIGN_PRIVATE_KEY + COSIGN_PASSWORD. Stage 4 turns signing into enforcement at the cluster gate.

✋ Hands-on checkpoint: pre-commit actually blocks a secret

Installed-but-not-wired is the classic silent failure. Prove the hooks fire:

echo 'AWS_SECRET='"$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')" > leak-test.env

git add leak-test.env

pre-commit run --all-files; echo "exit=$?"

git reset leak-test.env >/dev/null; rm -f leak-test.env

Expected: the secret-scanning hook fails the run (exit=1) and flags leak-test.env. If exit=0, your hooks are installed but not catching anything: re-run pre-commit install and confirm .git/hooks/pre-commit exists.

If you skip this, commits sail through unscanned and you'll believe Stage 3 is protecting you when it's not.

3.2: Generate Cosign Keys

If you created Cosign keys in Stage 1 (§1.4), skip generation: go straight to inserting cosign.pub into the Kyverno policy and adding the GitHub secrets below.

Cosign signs your Docker images with a cryptographic key. When you deploy to the cluster, Kyverno (Stage 4) can verify the signature and reject any image that wasn't signed by your pipeline. This prevents someone from pushing a malicious image to your Docker Hub and having the cluster run it.

# macOS: brew install cosign

# Linux/WSL2: curl -O -L https://github.com/sigstore/cosign/releases/download/v2.2.4/cosign-linux-amd64 && chmod +x cosign-linux-amd64 && sudo mv cosign-linux-amd64 /usr/local/bin/cosign

cosign generate-key-pair   # enter a password when prompted

This creates two files: cosign.key (private, used by the pipeline to sign) and cosign.pub (public, used by Kyverno to verify).

Insert your public key into the Kyverno policy (replace the placeholder block in infra/policies/require-signed-images.yaml with the contents of cosign.pub).

Add secrets to GitHub (github.com/YOUR_USERNAME/clearledger → Settings → Secrets and variables → Actions):

✋ Hands-on checkpoint: Cosign keys are ready

Stage 4 uses cosign.pub to verify signed images. Before you continue, confirm the key files exist and the private key isn't tracked by Git:

test -f cosign.key && echo "private key present"
test -f cosign.pub && echo "public key present"
grep -q "BEGIN PUBLIC KEY" cosign.pub && echo "public key valid"
git check-ignore cosign.key && echo "private key correctly ignored"

Expected: all four lines should print.

If git check-ignore cosign.key prints nothing, add cosign.key to .gitignore before committing anything. The private key must stay out of Git.

Don't skip this check. Stage 4 needs the public key for the Kyverno image-signing policy, and the private key must remain local.

3.3: Activate the Full Security Pipeline

The security gates are already in .github/workflows/ci.yaml. Push any change to trigger the full pipeline:

git add . && git commit -m "ci: full DevSecOps pipeline" && git push origin main

3.4: Break Each Gate on Purpose

For each gate, you'll want to break something on purpose, read how the tool reports it, revert, and confirm green again. Try the local command first, then push once if you want a screenshot on GitHub Actions.

# 1. Break it   2. Run locally or push   3. Read the failure
# 4. git checkout -- path/to/file   5. pre-commit run --all-files (optional)   6. git push

Start with Gate 1 end-to-end before the others.

Gate 1: Gitleaks (secrets)

Inject: hardcoded AWS key in any Python file.

The goal is to prove the secret scanner works.

This command adds a fake AWS-looking key to app/auth-service/main.py:

echo 'AWS_KEY = "'$(printf '%s%s' 'AKIA' 'IOSFODNN7EXAMPLE')'"' >> app/auth-service/main.py

git add app/auth-service/main.py && git commit -m "test: trigger gitleaks"
# pre-commit blocks this commit locally — that is the test.
# For a CI screenshot only: git commit --no-verify -m "test: trigger gitleaks" && git push

Done looks like this (terminal: pre-commit):

🔑 Secrets scan (Gitleaks)...............................................Failed
- hook id: gitleaks
- exit code: 1

Finding:     AWS_KEY = "REDACTED"
RuleID:      aws-access-token
File:        app/auth-service/main.py
Line:        316

Expected: the commit should fail. Gitleaks should report one secret finding in app/auth-service/main.py.

That failure is good. It means the local pre-commit hook caught the secret before it reached Git.

Revert:

git restore --staged app/auth-service/main.py 2>/dev/null
git checkout app/auth-service/main.py
pre-commit run gitleaks --all-files   # → Passed

Gate 2: Semgrep (SAST)

Local dry-run (no repo change):

python3 -m venv /tmp/sec-gates-venv && /tmp/sec-gates-venv/bin/pip install semgrep
cat > /tmp/semgrep-bad.py << 'EOF'
import subprocess
from fastapi import Request
def bad(request: Request):
    subprocess.run(request.query_params.get("cmd"), shell=True)
EOF
/tmp/sec-gates-venv/bin/semgrep \
  --config=p/python --config=p/security-audit --config=p/owasp-top-ten --error \
  /tmp/semgrep-bad.py

Break CI: add a temporary file Semgrep will scan, commit, and push:

cat > app/auth-service/gate_test_semgrep.py << 'EOF'
import subprocess
from fastapi import Request
def bad(request: Request):
    subprocess.run(request.query_params.get("cmd"), shell=True)
EOF

git add app/auth-service/gate_test_semgrep.py && git commit -m "test: trigger semgrep" && git push

Expected result: Semgrep reports subprocess-shell-true as Blocking. The SAST (Semgrep) job turns red, and the image build jobs don't run.

Revert:

rm -f app/auth-service/gate_test_semgrep.py
git add -A && git commit -m "revert: semgrep gate test" && git push

Gate 3: Checkov (IaC / Dockerfile)

Checkov scans Dockerfiles and Kubernetes manifests for unsafe configuration.

First, run a local demo. This removes the HEALTHCHECK from a copied Dockerfile and shows how Checkov reports it:

python3 -m venv /tmp/sec-gates-venv && /tmp/sec-gates-venv/bin/pip install checkov
sed '/^HEALTHCHECK/,+1d' app/auth-service/Dockerfile > /tmp/Dockerfile-nohc
mkdir -p /tmp/checkov-demo/app/auth-service
cp /tmp/Dockerfile-nohc /tmp/checkov-demo/app/auth-service/Dockerfile
/tmp/sec-gates-venv/bin/checkov --directory /tmp/checkov-demo --framework dockerfile

Now trigger a Checkov finding in CI by exposing SSH port 22 in the auth-service Dockerfile:

echo 'EXPOSE 22' >> app/auth-service/Dockerfile
git add app/auth-service/Dockerfile && git commit -m "test: trigger checkov" && git push

Expected result: the Checkov log or artifact should show CKV_DOCKER_1, which means an SSH port was exposed.

The IaC Scan (Checkov) job may or may not turn red, depending on the severity Checkov assigns. That's okay for this exercise. The goal is to find and understand the Checkov result.

If you need a screenshot of a failed GitHub Actions job, use Gate 1, Gate 2, or Gate 4. Those are designed to turn the workflow red. Checkov is mainly for reading the finding, so it may stay green.

Revert:

git checkout app/auth-service/Dockerfile
git commit -am "revert: checkov gate test" && git push

Gate 4: Trivy (image CVEs)

Local dry-run: scan an old base image (no build):

trivy image --exit-code 1 --severity CRITICAL,HIGH --ignore-unfixed python:3.8-slim

Break CI: pin an old base in the Dockerfile, push, wait for Scan images:

sed -i.bak 's/FROM python:3.13-slim/FROM python:3.8-slim/' app/auth-service/Dockerfile
git add app/auth-service/Dockerfile && git commit -m "test: trigger trivy" && git push

Pass: Scan images → Trivy scan all images exits 1 with a CVE table (HIGH / CRITICAL). Publish images and Update Manifests are skipped.

Revert:

git checkout app/auth-service/Dockerfile
git commit -am "revert: trivy gate test" && git push

3.5: When a Scan Fails on a CVE You Didn't Inject

§3.4 is deliberate. This section is for the other case where you push normal code, but the image scan fails because a new vulnerability was found.
That's normal. CVE databases update all the time. Don't weaken the scan. Fix the vulnerable package or image.

First, find the real CVE. In GitHub Actions, open Scan images then go to Trivy scan all images and look for the table with:

• Package

• CVE

• Installed version

• Fixed version You can also download the artifact:

Ignore this red herring at the bottom of the log:

Version 0.71.2 of Trivy is now available
Error: Process completed with exit code 1.

The version notice doesn't fail the job. A fixable HIGH/CRITICAL CVE does. Don't add --skip-version-check to “fix” it.

Instead, fix it with:

• pip package: bump to the Fixed Version in requirements.txt (example: python-multipart==0.0.30 for CVE-2026-53539). Apply the same bump to sibling services if they share that pin.

• OS package: newer base image or a targeted apt/apk upgrade in the Dockerfile.

• No stable fix yet documented exception only: add the CVE to .trivyignore and .grype.yaml with a comment (see CVE-2026-7210).

Don't remove --exit-code 1, lower the severity rule, or disable scanning. For help, see Trivy version notice and Trivy blocks Python service images.

Finish Stage 3

For screenshots, use one clear failed gate:

• Gitleaks: Secrets Scan

• Semgrep: SAST

• Trivy: Scan images

• Checkov: look for CKV_* in the log or artifact. The job may stay green

After each test in §3.4, undo the test change, push the revert, and confirm the workflow is green again. One red GitHub Actions screenshot is enough for your portfolio.

Run the stage check:

make check-3   # must end: All checks passed. Ready for the next stage.

Expected: All checks passed. Ready for the next stage.

You should also have triggered at least one gate in §3.4. A local Gitleaks failure counts.

ENABLE_DAST=true is optional. You only need it if you want to run ZAP later.

Not required yet: Checkov blocking Kubernetes manifests or Cosign blocking deployments. Stage 4 turns those into cluster enforcement with Kyverno.

Next, save your progress:

make snapshot STAGE=3 && make snapshots

Stage 4 — Admission Control (Kyverno)

Even if CI passes, the cluster can still refuse.

CI scans your code and images before they reach GitOps, but it can't watch everything that happens inside the cluster. Someone with kubectl access could apply a manifest directly.

A Helm chart you install might create pods that violate your security standards. Those paths never hit the pipeline, which is why Stage 4 adds admission control: a checkpoint built into Kubernetes itself.

Every time something tries to create or update a resource, the request passes through admission webhooks before it takes effect. If a webhook rejects the request, the resource is never created.

Kyverno is a Kubernetes-native policy engine that uses those webhooks. You write policies as YAML files (not application code), and Kyverno enforces them on every matching resource in the cluster, for example, rejecting any pod that runs as root or requiring CPU and memory limits on every container.

The difference from CI is timing: CI scans before code ships, while Kyverno enforces at the cluster gate. Together they give you two layers of defense.

Your goal in this stage is to install Kyverno, apply the policies in infra/policies/, and prove in §4.4 that non-compliant pods are denied before the container runtime ever sees them.

Before you start, make sure the foundation from earlier stages is still solid: make check-3 should pass (pre-commit hooks and CI security gates are active), infra/cosign.pub should exist from Stage 3, and ArgoCD should still be syncing so the app responds at http://clearledger.local. If any of those are red, fix them first: Kyverno sits on top of a healthy cluster, not a broken one.

You're done with Stage 4 when all three break-it scenarios in §4.4 are denied and make check-4 passes.

What changes from Stage 3 is enforcement, not scanning. In CI, Checkov reported Kubernetes misconfigurations but didn't block the pipeline. Kyverno now stops those same classes of problems at the cluster gate.

Cosign has been signing your images since Stage 1. Kyverno now requires that signature before a ClearLedger image can deploy. This is where Stage 1 evidence becomes enforcement. See that section if you want the full map of what blocked in Stage 1 versus what waited for Stage 4.

Start with §4.1 to install Kyverno. If the install, policies, break-it scenarios, or make check-4 fail, read troubleshooting.md and Stage 4: Admission Control (Kyverno) before changing Helm values or policy YAML.

What Kyverno Enforces

All policy files live in infra/policies/. Kyverno itself is installed via Helm using stages/stage-4-admission-control/infra/kyverno/values.yaml.

Platform Stability: From Stage 4 Onward

From Stage 4 on, you're running more controllers on a single-node VM. Kyverno, storage provisioners, and later Prometheus and Loki. A pod can show Running while it's actually crash-looping in the background.

When platform pods (Kyverno controllers, hostpath-provisioner, the Prometheus operator, and similar) accumulate high RESTARTS, the API server starts timing out, kubectl feels flaky, and you can waste days debugging the wrong component because the app pods look fine.

After every stage from here on, give the cluster about ten minutes to settle, then run the stage health check:

bash scripts/health-check.sh <stage>    # for example, 4, 7, 7.5
# or the Makefile shortcut:
make check-4

The script ends with a Platform stability section that flags pods with suspicious restart counts. You can also scan the worst offenders yourself. This lists the fifteen pods with the highest restart counts cluster-wide, which is useful when something feels slow but you're not sure which namespace is struggling:

kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount' \
  | tail -15

The gate: Kyverno controllers and other platform pods should show RESTARTS under 5 after the stage settles. If any platform pod is climbing past 10, stop and fix it with the documented Helm values or troubleshooting.md. Don't kubectl patch around it and move on. A stable platform layer is a prerequisite for every stage that follows.

4.1: Install Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm upgrade --install kyverno kyverno/kyverno \
  --version 3.2.8 \
  --namespace kyverno \
  --create-namespace \
  -f stages/stage-4-admission-control/infra/kyverno/values.yaml \
  --wait --timeout=600s

The values file does three important things for the lab:

1. Disables cleanup CronJobs: older Kyverno charts pull bitnami/kubectl, which was removed from Docker Hub and causes ImagePullBackOff on cleanup pods.

2. Points Helm hooks at bitnamilegacy/kubectl, so future helm uninstall doesn't hang on a missing image.

3. Extends liveness probe timeouts: the default timeoutSeconds: 5, failureThreshold: 2 is too tight for a loaded single-node VM. Under CPU pressure, the health endpoint can take >5s to respond, which triggers a restart cascade that saturates the node and makes the API server intermittently unreachable. The values file sets timeoutSeconds: 30, failureThreshold: 5 so Kyverno survives load spikes without crash-looping.

What you should see:

Release "kyverno" does not exist. Installing it now.
NAME: kyverno
NAMESPACE: kyverno
STATUS: deployed
...
Kyverno version: v1.12.6

Verify all four controllers are running (first pull can take several minutes on a slow connection):

kubectl get pods -n kyverno

NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-bd685cd4b-f6kl6     1/1     Running   0          2m
kyverno-background-controller-66fcfc6d87-59wgt   1/1     Running   0          2m
kyverno-cleanup-controller-5c5bf8bc6b-7kspq      1/1     Running   0          2m
kyverno-reports-controller-5cdd6f4c48-qf5wc      1/1     Running   0          2m

If pods stay in ContainerCreating for a long time, the node is still pulling images from ghcr.io/kyverno. Wait. Don't start a second Helm install on top of a partial one.

Stability gate: Kyverno install only (before §4.2):

Before continuing, make sure the Kyverno pods are healthy:

kubectl get pods -n kyverno

Expected: the Kyverno controller pods show 1/1 Running, with low restart counts such as 0, 1, or 2, and the restart count isn't increasing.

Don't run make check-4 yet. That check also looks for the policies you apply later in §4.3, so it may fail at this point even if Kyverno installed correctly.

4.2: Confirm your Cosign Public Key is in the Policy

Stage 3 created infra/cosign.pub. Kyverno uses that same key to verify image signatures when a pod is created. The policy file ships with a placeholder. You must replace it with your key before applying policies in §4.3.

Step 1: Show your key (run from the repo root on the VM)

cd ~/clearledger    # or wherever you cloned the repo
cat infra/cosign.pub

You should see three lines: -----BEGIN PUBLIC KEY-----, a long base64 line, and -----END PUBLIC KEY-----. Copy that whole block (you'll paste it in the next step).

Step 2: Paste the key into the policy

Open infra/policies/require-signed-images.yaml in your editor (nano, vim, or VS Code).

Find this line:

                      PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE

Delete only that placeholder line and paste the three lines from cosign.pub in its place. The result should look like this (your base64 line will differ):

                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                     JFkwEwYHKoZIzj0CAQYIKoFIzj0DAQcDQgZEI...
                      -----END PUBLIC KEY-----

Save the file. Keep the pasted key indented under publicKeys: |-. The BEGIN PUBLIC KEY and END PUBLIC KEY lines should have spaces before them, just like the base64 line between them.

Step 3: Verify (three quick checks)

Run these one at a time from the repo root:

# Check A — placeholder must be gone
grep PASTE_YOUR_COSIGN_PUBLIC_KEY_HERE infra/policies/require-signed-images.yaml \
  && echo "❌ FAIL: placeholder still in file — edit and save again" \
  || echo "✓ OK: placeholder removed"

# Check B — key block must be present exactly once
grep -c "BEGIN PUBLIC KEY" infra/policies/require-signed-images.yaml

Expected output for Check B: 1 (if you see 0, the key was not pasted. If 2, you pasted it twice).

# Check C — policy key must match cosign.pub byte-for-byte
diff infra/cosign.pub \
  <(sed -n '/-----BEGIN PUBLIC KEY-----/,/-----END PUBLIC KEY-----/p' \
      infra/policies/require-signed-images.yaml | sed 's/^[[:space:]]*//')

Expected output for Check C: nothing. No diff lines means the keys match. If diff prints differences, open the policy file and fix the paste.

If all three passed, continue to §4.3.

If you skip this, Scenario 3 in §4.4 fails in a confusing way: unsigned images may slip through, or signed pods may be rejected because Kyverno is checking against the wrong key.

4.3: Apply the Five Core Policies

Now you'll apply the five policies that map to CIS controls. Don't apply verify-slsa-provenance.yaml yet. It's an optional SLSA attestation policy (Audit mode) for a later enhancement.

Stage 4 applies infra/policies/require-signed-images.yaml. This policy uses failurePolicy: Fail, so if Kyverno can't verify an image signature, the pod is blocked instead of allowed. The ECR policy with failurePolicy: Ignore is for Stage 8, not this step.

kubectl apply \
  -f infra/policies/disallow-root.yaml \
  -f infra/policies/disallow-privilege-escalation.yaml \
  -f infra/policies/drop-all-capabilities.yaml \
  -f infra/policies/require-resource-limits.yaml \
  -f infra/policies/require-signed-images.yaml

Wait a few seconds, then confirm all policies show READY: True and VALIDATE ACTION: Enforce:

kubectl get clusterpolicy

NAME                            ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE
disallow-privilege-escalation   true        true         Enforce           True    10s
disallow-root-containers        true        true         Enforce           True    10s
drop-all-capabilities           true        true         Enforce           True    10s
require-resource-limits         true        true         Enforce           True    10s
require-signed-images           true        false        Enforce           True    10s

If READY stays empty, check Kyverno logs: kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller --tail=50.

4.4: Breaking it on Purpose

Now you'll test the policies by trying to create bad pods.

These pods are supposed to fail. That's the point.

CI tools like Checkov warn you in a report. Kyverno goes further: it blocks unsafe pods before Kubernetes runs them.

For each test, read the error message, as it should tell you which policy blocked the pod and what field was wrong. That error message is your proof that admission control is working.

Scenario 1: root container (no securityContext)

What you're simulating: Someone with kubectl access bypasses CI and applies a minimal pod: no securityContext, no resource limits.

This is exactly what Stage 1 Checkov flagged as evidence. Stage 4 now blocks it.

What's wrong with this manifest: The container has only a name and image. It will run as root by default, keep all Linux capabilities, and has no CPU/memory bounds.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: root-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
EOF

What you should see:

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/root-test was blocked due to the following policies

disallow-privilege-escalation:
  check-allowPrivilegeEscalation: 'validation error: allowPrivilegeEscalation must
    be set to false. rule check-allowPrivilegeEscalation failed at path /spec/containers/0/securityContext/'
disallow-root-containers:
  check-runAsNonRoot: |-
    validation error: Root containers are blocked in the clearledger namespace. Set securityContext.runAsNonRoot: true on the pod or container.
    . rule check-runAsNonRoot failed at path /spec/containers/0/securityContext/
drop-all-capabilities:
  check-capabilities: 'validation error: All containers must drop ALL capabilities.
    rule check-capabilities failed at path /spec/containers/0/securityContext/'
require-resource-limits:
  check-resources: 'validation error: Resource requests and limits are required for
    all containers. rule check-resources failed at path /spec/containers/0/resources/limits/'

How to read this output:

The important line is:

resource Pod/clearledger/root-test was blocked due to the following policies

That means Kyverno stopped the pod before it was created.

Under that line, Kyverno lists every policy the pod failed. For example:

disallow-root-containers:
  check-runAsNonRoot:

This means the pod failed the disallow-root-containers policy, specifically the check-runAsNonRoot rule. The fix is also shown in the message:

Set securityContext.runAsNonRoot: true

The same pattern applies to the other policies:

• disallow-privilege-escalation means the pod didn't set allowPrivilegeEscalation: false

• drop-all-capabilities means the pod didn't drop Linux capabilities with capabilities.drop: [ALL]

• require-resource-limits means the pod didn't set CPU and memory requests/limits

The path part tells you where Kubernetes expected the missing setting. For example, /spec/containers/0/securityContext/ means: look inside the pod spec, then the first container, then its securityContext.

And /spec/containers/0/resources/limits/ means: look inside the first container's resource limits.

So this one bad pod failed four controls at once. That's the lesson: Kyverno doesn't just say "no." It tells you which policy failed and where to fix the YAML.

Verify enforcement worked:

kubectl get pod root-test -n clearledger
# Error from server (NotFound): pods "root-test" not found

If you see a pod in Running or Pending, policies aren't enforcing: re-check that kubectl get clusterpolicy shows all five READY: True.

Take a screenshot. This is portfolio evidence for CIS Kubernetes Benchmark 5.2.6: enforced, not just configured.

Scenario 2: missing resource limits

What you're simulating: A developer who read the securityContext requirements and fixed root/caps/privilege. But skipped resource limits.

This is common in real teams: “we hardened the container” but forgot CPU/memory bounds.

What's wrong with this manifest: securityContext is correct, but there's no resources.requests or resources.limits. A container without limits can starve other workloads on the node.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nolimits-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
EOF

What you should see:

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/nolimits-test was blocked due to the following policies

require-resource-limits:
  check-resources: 'validation error: Resource requests and limits are required for
    all containers. rule check-resources failed at path /spec/containers/0/resources/limits/'

Key observation: Only one policy fires this time: the securityContext fields satisfied the other four rules. Kyverno evaluates rules independently. Each container property is a separate gate.

Verify:

kubectl get pod nolimits-test -n clearledger
# Error from server (NotFound): pods "nolimits-test" not found

Scenario 3: unsigned ClearLedger image

What you're simulating: A supply-chain attack: someone pushes a malicious image to Docker Hub under your repo name (clearledger-auth-service) without going through your signed CI pipeline. Stage 3 made Cosign signing possible, while Stage 4 makes it mandatory at the cluster gate.

Why this setup is needed: Kyverno checks image signatures against the image in Docker Hub, not against images on your laptop. The test image tag must exist in Docker Hub first.

If you use a fake tag like :unsigned that was never pushed, Kubernetes may fail later with ImagePullBackOff. That only means the image can't be pulled; it doesn't prove Kyverno blocked an unsigned image.

Step 1: push a deliberately unsigned test image (one-time):

export DOCKER_USERNAME=your-dockerhub-username

docker pull nginx:alpine
docker tag nginx:alpine ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
docker push ${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test

# Must fail — proves the image has no Cosign signature from your pipeline key:
cosign verify --key infra/cosign.pub \
  index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
# Error: no signatures found

Step 2: try to deploy it with a compliant pod spec:

The pod manifest is fully hardened (securityContext + limits) so only the signature policy can fail. Use index.docker.io/ in the image URL: on Kyverno 1.12, docker.io/... may not trigger verifyImages matching.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: index.docker.io/${DOCKER_USERNAME}/clearledger-auth-service:unsigned-test
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
      resources:
        requests:
          memory: "64Mi"
          cpu: "50m"
        limits:
          memory: "128Mi"
          cpu: "200m"
EOF

What you should see:

Error from server: error when creating "STDIN": admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Pod/clearledger/unsigned-test was blocked due to the following policies

require-signed-images:
  verify-cosign-signature: 'failed to verify image index.docker.io/veeno-demo/clearledger-auth-service:unsigned-test:
    .attestors[0].entries[0].keys: no signatures found'

How to read this output:

• Note the webhook name is mutate.kyverno.svc-fail, not validate: image verification runs in Kyverno’s mutate pass (digest + signature check) before the pod is admitted.

• no signatures found means Kyverno reached Docker Hub, found the image, and confirmed it was not signed with your infra/cosign.pub key.

• The pod never exists: the attacker can't get a shell even if the image is pullable.

Verify:

kubectl get pod unsigned-test -n clearledger
# Error from server (NotFound): pods "unsigned-test" not found

What you should NOT see (these mean the test didn't prove signature enforcement):

Contrast: signed image is allowed:

The previous test used an unsigned image, so Kyverno blocked it.

Your real ClearLedger images should be signed by the CI pipeline. If the pod also follows the security rules, Kyverno allows it to run.

You can check the image currently used by auth-service:

# Your deployed tag (signed in CI) should start if spec is compliant:
kubectl get deployment auth-service -n clearledger \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
# docker.io/veeno-demo/clearledger-auth-service:v0.1.0

Example output:

docker.io/veeno-demo/clearledger-auth-service:v0.1.0

Pods that were already running before the policies were applied will keep running. The important test is what happens when Kubernetes creates a new pod. New pods using signed ClearLedger images should pass Kyverno verification.

Take a screenshot of the Scenario 3 denial. It proves the cluster blocks unsigned images, not just that CI signs images.

4.5: Verify ClearLedger Still Works

Kyverno enforces on new pod creation. Existing deployments that already passed admission (or were synced before policies existed) keep running. Confirm your app pods are healthy:

kubectl get pods -n clearledger

NAME                                    READY   STATUS    RESTARTS   AGE
auth-service-...                        1/1     Running   0          ...
frontend-...                            1/1     Running   0          ...
ledger-service-...                      1/1     Running   0          ...
notification-service-...                1/1     Running   0          ...
postgres-0                              1/1     Running   0          ...
redis-...                               1/1     Running   0          ...

If ingress is configured:

curl -s http://clearledger.local/auth/health | jq .
# {"status": "ok", "service": "auth-service"}

ArgoCD should still show Synced and Healthy: GitOps and admission control work together, not against each other.

4.6: Policy Exceptions (When a Legitimate Workload Needs a Bypass)

Kyverno blocks every pod that violates a policy. But what happens when a legitimate workload needs to bypass a specific rule?

PostgreSQL is the example. The official Postgres Alpine image uses a specific internal user (UID 70) to manage its data directory. The disallow-root-containers policy requires every pod to set runAsNonRoot: true.

Postgres does set that. But if Kyverno is configured to also check specific UID ranges, or if the pod's security context doesn't satisfy the rule for any reason, Kyverno blocks it. The database can't start, and the entire application fails.

You can't weaken the policy cluster-wide to accommodate one database. That would let every pod bypass the rule. Instead, you create a PolicyException: a targeted exemption for exactly the pods that need it.

Open infra/policies/exceptions/postgres-root-exception.yaml and read the comments. Here's what each section does:

The spec.exceptions block identifies which policy and rule to bypass:

exceptions:
  - policyName: disallow-root-containers
    ruleNames:
      - check-runAsNonRoot

This says: "skip only the check-runAsNonRoot rule from the disallow-root-containers policy." Every other rule in that policy (and every other policy in the cluster) still enforces normally.

The spec.match block limits which resources get the exception:

match:
  any:
    - resources:
        kinds:
          - Pod
        namespaces:
          - clearledger
        names:
          - postgres-*

Only pods named postgres-* (matching postgres-0, postgres-1, and so on), only in the clearledger namespace, only for the Pod resource kind. Everything else in the cluster still follows the strict policy.

The annotations are documentation for your team and auditors:

annotations:
  reason: "Postgres alpine image requires UID 70 for data directory ownership"
  approved-by: "platform-team"
  review-date: "2026-01-01"

These have no technical effect: Kyverno ignores them. They exist so that six months from now, when someone asks "why does Postgres bypass this rule?", the answer is right there in the file.

The rules for safe exceptions:

1. Scope narrowly: target the exact resource that needs it, nothing more

2. Commit to Git: the exception is reviewed in a pull request, tracked in version history, and auditable

3. Never weaken the policy itself: the rule stays strict for everything else

4. Review periodically: exceptions should be temporary if possible, and re-evaluated on a schedule

Apply the exception only if Kyverno blocks your Postgres pods:

kubectl apply -f infra/policies/exceptions/postgres-root-exception.yaml

Verify Kyverno still blocks other non-compliant pods (same denial as Scenario 1):

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: another-root-test
  namespace: clearledger
spec:
  containers:
    - name: test
      image: nginx:alpine
EOF

4.7: CIS Benchmark Evidence kube-bench)

You already installed Kyverno and proved it blocks unsafe pods.

This step is different. kube-bench doesn't block pods and doesn't change the cluster. It only checks the Kubernetes node against the CIS benchmark and saves evidence.

Think of the difference like this:

Both are useful, but only Kyverno blocks workloads in this lab.

Run kube-bench:

bash stages/stage-4-admission-control/scripts/run-kube-bench.sh

The script runs kube-bench as a Kubernetes Job and saves the report here:

stages/stage-4-admission-control/scripts/kube-bench-report.json

It also compares the result against this baseline:

stages/stage-4-admission-control/scripts/kube-bench-baseline.json

On MicroK8s, you'll see many FAIL and WARN lines. That's expected. The lab isn't asking you to fix every CIS warning on a single-node local VM.

What matters is the final result.

Pass looks like this:

kube-bench: 1 FAIL control(s) present (documented in baseline — no regressions).
kube-bench: no regressions vs baseline.

That means the known MicroK8s issues are documented, and your cluster didn't get worse.

If you see REGRESSION or make check-4 fails on kube-bench, stop and investigate before Stage 5.

Optional: confirm the report file exists:

ls -la stages/stage-4-admission-control/scripts/kube-bench-report.json

In production, you would either fix the CIS failures or document approved exceptions. In this lab, the baseline records the expected MicroK8s state.

4.8: Health Check

make check-4

What you should see:

▶ Stage 4 — Admission Control (Kyverno)
  ✓ Kyverno is running
  ✓ Policy disallow-root-containers — Enforce mode
  ✓ Policy require-resource-limits — Enforce mode
  ✓ Policy require-signed-images — Enforce mode
  ✓ Policy disallow-privilege-escalation — Enforce mode
  ✓ Policy drop-all-capabilities — Enforce mode
  ✓ Kyverno correctly rejects pods without securityContext
  ✓ kube-bench baseline exists (...)

All checks passed. Ready for the next stage.

If kube-bench reports regressions, run the script manually and update the baseline after reviewing. That diff is audit evidence.

If Kyverno install, policies, break-it scenarios, or make check-4 fail, see troubleshooting.md. Stage 4.

Stage 4 Complete: Done Checklist (Move to Stage 5)

You're done with Stage 4 when all of these are true: