That reads like a milestone, but it's the reason you have a problem.

The same report tested more than a hundred models and found the average security pass rate at 56%, "barely changed from 55% in the first report", with roughly 44% of generation tasks introducing a risky vulnerability.

Functional correctness and security turn out to be separate problems, and only one of them is close to solved.

That result is neither an outlier nor new. At IEEE Security and Privacy in 2022, a team at NYU Tandon ran GitHub Copilot through 89 security-relevant scenarios, generated 1,689 programs, and found roughly 40% of them vulnerable to something on MITRE's CWE Top 25. The paper was later selected as a Communications of the ACM research highlight.

In November 2024, Georgetown's Center for Security and Emerging Technology evaluated five LLMs and reported that almost half the snippets they produced contained bugs that could lead to exploitation. Four years, four independent teams, four methodologies, and the same answer each time.

At ACM CCS in 2023, Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh at Stanford put the developers into the experiment: 47 participants, five security-related programming tasks, three languages, with 33 given an AI assistant and 14 not. The assisted group wrote significantly less secure code, and was more likely to believe the code it wrote was secure.

It's a small study, and it explains why the problem doesn't correct itself: the mechanism that would normally catch this (a developer looking harder at code that worries them) is the exact mechanism the tooling switches off.

Those studies all measure application code. But infrastructure code is the harder case, because a bad security group never fails: it works exactly as written, serving traffic to whoever asks, and the only thing that objects is a person reading a diff.

I can't review that volume by reading it, and neither can anybody else. What I can do is write the rules down in a form a computer checks on every change, which is what Policy as Code means.

In this handbook, I walk you through building that check. We'll point it at a real vulnerable repository, watch the obvious version of it clear five of the nine violations sitting in front of it, and then fix it.

By the end, you'll know how to:

• Write a Rego policy against the JSON that terraform show -json produces.

• Test a policy the way you test application code, with fixtures and a coverage report.

• Build a command-line gate with an exit-code contract that a CI pipeline can trust.

• Block non-compliant workloads at Kubernetes admission time using CEL.

• Have a model write a policy and let opa check and your own tests decide whether to keep it.

• Authorise an AI agent's tool calls from the same policy engine.

Table of Contents

• Prerequisites

• Key Terms in Plain English

• Step 1: Fetch Real Infrastructure to Test Against

• Step 2: Write the Tests Before the Policy

• Step 3: Write the Policy Until the Tests Pass

• Step 4: Point It at the Real Plan

• Step 5: Turn the Verdict into an Exit Code

• Step 6: Enforce at Admission Time

• Step 7: Let a Model Write the Policy

• Step 8: Govern the Agent Itself

• Step 9: What I Got Wrong

• Limits of the Check

• Conclusion

Prerequisites

You need:

• A terminal and a working python3 (3.10 or newer).

• jq, for reading JSON at the command line.

• About 700 MB of disk, because the AWS Terraform provider is large.

• An Anthropic API key, but only for Step 7. Every other step runs offline.

mkdir policy-lab && cd policy-lab
python3 -m venv .venv
source .venv/bin/activate
pip install anthropic

curl -L -o opa https://openpolicyagent.org/downloads/v1.20.2/opa_darwin_arm64_static
chmod +x opa && sudo mv opa /usr/local/bin/

curl -L -o tf.zip https://releases.hashicorp.com/terraform/1.14.2/terraform_1.14.2_darwin_arm64.zip
unzip tf.zip && sudo mv terraform /usr/local/bin/

On Windows, activate the environment with .venv\Scripts\activate, and swap the two download URLs for opa_windows_amd64.exe and terraform_1.14.2_windows_amd64.zip.

I ran everything below on OPA 1.20.2, Terraform 1.14.2, and AWS provider 6.x, on macOS. The policy syntax is stable across OPA 1.x.

If you're on OPA 0.x, every rule here needs import rego.v1 added at the top, and I would upgrade instead. The violation counts depend on the AWS provider version only through the shape of the plan JSON, which has been stable since provider 5.

Key Terms in Plain English

• Policy as Code: a rule your organisation has already agreed on, written as a program that takes a proposed change and returns a decision.

• Rego: the query language Open Policy Agent evaluates. It's declarative: a rule body is a list of conditions that must all hold.

• Plan JSON: the machine-readable description of what Terraform is about to do, produced by terraform show -json. This is what the policy reads, so your .tf files never reach it.

• Admission control: the point inside the Kubernetes API server where an object can be rejected before it's stored.

• CEL: Common Expression Language, the small expression language Kubernetes evaluates natively inside the API server, with no webhook to deploy.

• False clearance: a resource the policy passed that it should have failed. Nobody ever notices one, so it goes unmeasured unless you go looking for it.

The same judgement happens in three places, and only the middle one is specific to Kubernetes.

Step 1: Fetch Real Infrastructure to Test Against

I didn't want to invent a vulnerable Terraform file, because inventing one means inventing the bug, and then the policy only catches the bug I planted. So I went looking for code somebody else had written and published.

TerraGoat is a deliberately vulnerable Terraform repository published by Bridgecrew. Fetch its EC2 module at a pinned commit:

SHA=729f8da62c6a85ce4af5ad3d123de97776d954c4
curl -s "https://raw.githubusercontent.com/bridgecrewio/terragoat/$SHA/terraform/aws/ec2.tf" \
  | sed -n '77,96p'

resource "aws_security_group" "web-node" {
  # security group is open to the world in SSH port
  name        = "${local.resource_prefix.value}-sg"
  description = "${local.resource_prefix.value} Security Group"
  vpc_id      = aws_vpc.web_vpc.id

  ingress {
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }

The comment on line two is TerraGoat's own, and port 22 open to the world is the finding it points at.

TerraGoat's module won't initialise on modern Terraform, because it still declares type = "string" in quotes, which Terraform 0.12 deprecated and 1.x rejects. So I lifted the resource into a minimal module of my own, replacing only the two references to TerraGoat's internal locals.

Create main.tf:

terraform {
  required_version = ">= 1.9"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

# Mock credentials. This configuration is only ever planned, never applied,
# so the provider must not try to reach AWS.
provider "aws" {
  region                      = "us-west-2"
  access_key                  = "mock"
  secret_key                  = "mock"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  skip_region_validation      = true
}

resource "aws_vpc" "web_vpc" {
  cidr_block = "10.0.0.0/16"
}

# Verbatim from bridgecrewio/terragoat, terraform/aws/ec2.tf, commit 729f8da.
# Only the two references to TerraGoat's own locals are replaced with literals.
resource "aws_security_group" "web-node" {
  name        = "terragoat-sg"
  description = "terragoat Security Group"
  vpc_id      = aws_vpc.web_vpc.id

  ingress {
    from_port = 80
    to_port   = 80
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  egress {
    from_port = 0
    to_port   = 0
    protocol  = "-1"
    cidr_blocks = [
    "0.0.0.0/0"]
  }
  depends_on = [aws_vpc.web_vpc]
  tags = {
    git_commit           = "d68d2897add9bc2203a5ed0632a5cdd8ff8cefb0"
    git_file             = "terraform/aws/ec2.tf"
    git_last_modified_at = "2020-06-16 14:46:24"
    git_org              = "bridgecrewio"
    git_repo             = "terragoat"
  }
}

Produce the plan JSON:

terraform init
terraform plan -out=tfplan.binary
terraform show -json tfplan.binary > plan.json

The mock credentials matter: terraform plan on a create-only configuration never calls AWS, so with skip_credentials_validation and its three siblings the provider won't try to authenticate, and nothing is ever applied.

Step 2: Write the Tests Before the Policy

The rule I wanted was: no security group may expose an administrative port to the public internet.

That sounds like one line of code, and the tests are where I pin down why it's not. Create policy/network_test.rego:

package terraform.network_test

import data.terraform.network

plan(resources) := {"resource_changes": resources}

security_group(ingress) := {
	"address": "aws_security_group.web",
	"type": "aws_security_group",
	"change": {"actions": ["create"], "after": {"ingress": [ingress]}},
}

test_denies_ssh_open_to_the_world if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": 22,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

# A from_port equality check would miss this. The range check does not.
test_denies_wide_open_port_range if {
	fixture := plan([security_group({
		"from_port": 0,
		"to_port": 65535,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 4 with input as fixture
}

test_denies_ipv6_route_to_the_world if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": 22,
		"protocol": "tcp",
		"ipv6_cidr_blocks": ["::/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

test_denies_standalone_ingress_rule if {
	fixture := plan([{
		"address": "aws_vpc_security_group_ingress_rule.ssh",
		"type": "aws_vpc_security_group_ingress_rule",
		"change": {"actions": ["create"], "after": {
			"from_port": 22,
			"to_port": 22,
			"ip_protocol": "tcp",
			"cidr_ipv4": "0.0.0.0/0",
			"cidr_ipv6": null,
		}},
	}])

	count(network.deny) == 1 with input as fixture
}

test_denies_deprecated_standalone_rule if {
	fixture := plan([{
		"address": "aws_security_group_rule.ssh",
		"type": "aws_security_group_rule",
		"change": {"actions": ["create"], "after": {
			"type": "ingress",
			"from_port": 22,
			"to_port": 22,
			"cidr_blocks": ["0.0.0.0/0"],
		}},
	}])

	count(network.deny) == 1 with input as fixture
}

test_allows_ssh_from_a_private_range if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": 22,
		"protocol": "tcp",
		"cidr_blocks": ["10.0.0.0/8"],
	})])

	count(network.deny) == 0 with input as fixture
}

test_allows_https_from_the_world if {
	fixture := plan([security_group({
		"from_port": 443,
		"to_port": 443,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 0 with input as fixture
}

# An all-protocols rule opens every port, whatever its port fields say.
# The first version of this test asserted the opposite and hid the bug.
test_denies_all_protocols_rule_open_to_the_world if {
	fixture := plan([security_group({
		"from_port": 0,
		"to_port": 0,
		"protocol": "-1",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 4 with input as fixture
}

# Ports the policy cannot read are reported, never passed.
test_reports_a_world_open_rule_with_unreadable_ports if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": null,
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

test_reports_string_ports if {
	fixture := plan([security_group({
		"from_port": "22",
		"to_port": "22",
		"protocol": "tcp",
		"cidr_blocks": ["0.0.0.0/0"],
	})])

	count(network.deny) == 1 with input as fixture
}

# Unreadable ports on a rule that is not open to the world stay quiet.
test_ignores_unreadable_ports_on_a_private_range if {
	fixture := plan([security_group({
		"from_port": 22,
		"to_port": null,
		"protocol": "tcp",
		"cidr_blocks": ["10.0.0.0/8"],
	})])

	count(network.deny) == 0 with input as fixture
}

# `considered` drives the gate's pass-or-vacuous decision, so it needs
# tests of its own even though it takes no part in the judgement.
test_considers_every_ingress_bearing_type if {
	fixture := plan([
		security_group({}),
		{"address": "aws_vpc_security_group_ingress_rule.a", "type": "aws_vpc_security_group_ingress_rule", "change": {"actions": ["create"], "after": {}}},
		{"address": "aws_security_group_rule.b", "type": "aws_security_group_rule", "change": {"actions": ["create"], "after": {}}},
	])

	count(network.considered) == 3 with input as fixture
}

test_does_not_consider_unrelated_types if {
	fixture := plan([{
		"address": "aws_vpc.main",
		"type": "aws_vpc",
		"change": {"actions": ["create"], "after": {}},
	}])

	count(network.considered) == 0 with input as fixture
}

Here's what that file encodes:

1. with input as fixture swaps in a fake plan for one expression, which is how a policy is tested without a cloud account.

2. test_denies_wide_open_port_range expects four violations, one per administrative port, because an ingress rule describes a range. A 0-65535 rule opens SSH exactly as wide as an explicit port 22 rule while sailing past an equality check.

3. Three tests cover three other shapes Terraform uses for the same idea: the IPv6 field, the modern standalone aws_vpc_security_group_ingress_rule, and the deprecated aws_security_group_rule. I didn't write these first, and Step 9 explains where they came from.

4. The two test_allows_ cases matter as much as the denials. A policy that rejects everything passes every deny test and is worthless.

5. The last three tests arrived after the policy was already "finished", and Step 9 explains where they came from. An all-protocols rule opens every port whatever its port fields say, and a rule whose ports the policy can't read has to be reported.

Step 3: Write the Policy Until the Tests Pass

Terraform describes ingress in four shapes, so the policy normalises all four into one set and then judges that set once. Create policy/network.rego:

# METADATA
# title: No admin port is reachable from the public internet
# description: |
#   Terraform describes ingress in four different shapes. Each one is
#   normalised into a single `exposures` set first, so the judgement below
#   is written once and a new shape only costs one more helper rule.
#
#   Two things here are deliberate rather than incidental. An all-protocols
#   rule covers every port whatever its port fields say, and a rule whose
#   ports this policy cannot read is reported rather than passed.
package terraform.network

admin_ports := {22, 3389, 3306, 5432}

public_cidrs := {"0.0.0.0/0", "::/0"}

# The AWS provider writes from_port 0 and to_port 0 for an all-protocols
# rule, which opens every port, so the port fields cannot be read literally.
all_protocols := {"-1", "all"}

# Shape 1 and 2: inline ingress blocks, IPv4 and IPv6.
exposures contains exposure if {
	some resource in input.resource_changes
	resource.type == "aws_security_group"
	some ingress in resource.change.after.ingress
	some field in ["cidr_blocks", "ipv6_cidr_blocks"]
	some cidr in object.get(ingress, field, [])
	exposure := {
		"address": resource.address,
		"protocol": object.get(ingress, "protocol", ""),
		"from_port": object.get(ingress, "from_port", null),
		"to_port": object.get(ingress, "to_port", null),
		"cidr": cidr,
	}
}

# Shape 3: the standalone rule the AWS provider has recommended since v5.
exposures contains exposure if {
	some resource in input.resource_changes
	resource.type == "aws_vpc_security_group_ingress_rule"
	some field in ["cidr_ipv4", "cidr_ipv6"]
	cidr := object.get(resource.change.after, field, null)
	is_string(cidr)
	exposure := {
		"address": resource.address,
		"protocol": object.get(resource.change.after, "ip_protocol", ""),
		"from_port": object.get(resource.change.after, "from_port", null),
		"to_port": object.get(resource.change.after, "to_port", null),
		"cidr": cidr,
	}
}

# Shape 4: the deprecated standalone rule, still in most existing estates.
exposures contains exposure if {
	some resource in input.resource_changes
	resource.type == "aws_security_group_rule"
	resource.change.after.type == "ingress"
	some cidr in object.get(resource.change.after, "cidr_blocks", [])
	exposure := {
		"address": resource.address,
		"protocol": object.get(resource.change.after, "protocol", ""),
		"from_port": object.get(resource.change.after, "from_port", null),
		"to_port": object.get(resource.change.after, "to_port", null),
		"cidr": cidr,
	}
}

# The ports a rule really covers. Undefined when the policy cannot tell.
covered_ports(exposure) := [0, 65535] if {
	exposure.protocol in all_protocols
}

covered_ports(exposure) := [exposure.from_port, exposure.to_port] if {
	not exposure.protocol in all_protocols
	is_number(exposure.from_port)
	is_number(exposure.to_port)
}

deny contains msg if {
	some exposure in exposures
	exposure.cidr in public_cidrs

	# A rule covers a port if that port falls inside [from_port, to_port].
	range := covered_ports(exposure)
	some port in admin_ports
	port >= range[0]
	port <= range[1]

	msg := sprintf(
		"%s: ingress rule exposes port %d to %s",
		[exposure.address, port, exposure.cidr],
	)
}

# A rule open to the world whose ports this policy cannot read is reported.
# Passing it would be the policy guessing in the permissive direction.
deny contains msg if {
	some exposure in exposures
	exposure.cidr in public_cidrs
	not covered_ports(exposure)

	msg := sprintf(
		"%s: ingress rule to %s has ports this policy cannot evaluate (%v to %v)",
		[exposure.address, exposure.cidr, exposure.from_port, exposure.to_port],
	)
}

# Addresses this policy knows how to inspect. The gate uses this to tell
# "nothing violated" apart from "nothing examined".
considered contains resource.address if {
	some resource in input.resource_changes
	resource.type in {
		"aws_security_group",
		"aws_vpc_security_group_ingress_rule",
		"aws_security_group_rule",
	}
}

Reading that from the top:

1. A rule body in Rego is a conjunction. Every line must hold, and some ... in lines iterate, so OPA explores every combination of resource, ingress rule, field, and port.

2. Three separate exposures rules define one set between them, which Rego calls an incremental definition. Adding a fifth shape later costs one more block and changes nothing below it.

3. object.get(ingress, field, []) returns an empty list when a field is absent, so an IPv4-only rule doesn't error when the policy looks for ipv6_cidr_blocks.

4. covered_ports is the safety valve here, because an all-protocols rule reports 0 to 0 in the plan while opening every port, so the port fields can't be read literally, and a rule whose ports are null or strings leaves the function undefined, which the second deny rule turns into a violation.

5. considered isn't part of the judgement. It records which resources this policy can speak about at all, which Step 5 uses to avoid reporting a pass it hasn't earned.

Run it:

opa test policy
opa check --strict policy
opa fmt --diff policy

PASS: 20/20

Add -v to opa test for a line per test.

opa check --strict catches unsafe variables and shadowed imports, while opa fmt --diff prints nothing when the formatting is already canonical. OPA formats Rego with tabs. Both belong in CI, ahead of everything else.

The second policy is ownership tagging, so create policy/tags.rego:

# METADATA
# title: Every managed resource carries ownership tags
# description: |
#   Terraform emits `tags: null` for a resource with no tags at all, so a
#   policy that reaches into `after.tags` skips exactly the resources with
#   the worst tagging. `tags_of` coerces that null to an empty object.
package terraform.tags

required_tags := {"owner", "cost-center", "data-classification"}

# Resource types that genuinely cannot carry tags.
untaggable := {"aws_iam_policy_attachment", "aws_route_table_association"}

in_scope contains resource if {
	some resource in input.resource_changes
	some action in resource.change.actions
	action in {"create", "update"}
	not resource.type in untaggable
}

tags_of(resource) := tags if {
	tags := resource.change.after.tags
	is_object(tags)
} else := {}

deny contains msg if {
	some resource in in_scope
	some tag in required_tags
	value := object.get(tags_of(resource), tag, "")
	trim_space(value) == ""
	msg := sprintf("%s: missing required tag %q", [resource.address, tag])
}

considered contains resource.address if {
	some resource in in_scope
}

Here's why those two lines look the way they do:

1. trim_space(value) == "" does the check. It has to, because in Rego only false and undefined are falsy, so an empty string is truthy. A bare existence check happily accepts owner = "", which is compliance theatre of exactly the kind a tagging policy exists to stop.

2. tags_of, with its else := {} branch, is the fix for a bug I wrote and only found in Step 9.

Step 4: Point It at the Real Plan

Evaluate one package against the TerraGoat plan:

opa eval --data policy --input plan.json --format pretty 'data.terraform.network.deny'

[
  "aws_security_group.web-node: ingress rule exposes port 22 to 0.0.0.0/0"
]

The policy reports one violation, and stays quiet about port 80. It's open to the world in the same resource, because a public web server is the point of a public web server. A check that flags both is a check people learn to ignore.

Running one command per package doesn't scale, and Rego can aggregate across a namespace in a single query:

opa eval --data policy --input plan.json --format pretty \
  'union({v | v := data.terraform[_].deny})'

[
  "aws_security_group.web-node: ingress rule exposes port 22 to 0.0.0.0/0",
  "aws_security_group.web-node: missing required tag \"cost-center\"",
  "aws_security_group.web-node: missing required tag \"data-classification\"",
  "aws_security_group.web-node: missing required tag \"owner\"",
  "aws_vpc.web_vpc: missing required tag \"cost-center\"",
  "aws_vpc.web_vpc: missing required tag \"data-classification\"",
  "aws_vpc.web_vpc: missing required tag \"owner\""
]

{v | v := data.terraform[_].deny} is a comprehension that collects the deny set from every package under data.terraform, and union flattens them. Drop a new policy file into that namespace and it's picked up with no change to the command.

Step 5: Turn the Verdict into an Exit Code

A CI gate communicates through its exit status, and conflating two kinds of failure into one code is how a broken pipeline passes for a month. This tool uses the following:

The fourth row is the one most gates get wrong. If a plan contains no resource any policy knows about, reporting a pass claims an assurance the run can't give. It gets its own verdict name and it doesn't exit 0.

Create policy_gate.py:

"""Evaluate a Terraform plan against a directory of Rego policies."""

import argparse
import json
import pathlib
import shutil
import subprocess
import sys

PASS, FAIL, BROKEN = 0, 1, 2

DENY_QUERY = "union({v | v := data.terraform[_].deny})"
CONSIDERED_QUERY = "union({v | v := data.terraform[_].considered})"


def die(message: str) -> None:
    print(f"policy-gate: {message}", file=sys.stderr)
    sys.exit(BROKEN)


def load_plan(path: pathlib.Path) -> dict:
    try:
        text = path.read_text()
    except OSError as exc:
        die(f"cannot read {path}: {exc.strerror}")
    try:
        return json.loads(text)
    except json.JSONDecodeError as exc:
        die(f"{path}:{exc.lineno}:{exc.colno}: invalid JSON: {exc.msg}")


def query(opa: str, policy_dirs: list[pathlib.Path], plan: pathlib.Path, expr: str) -> list:
    command = [opa, "eval", "--input", str(plan), "--format", "raw"]
    for directory in policy_dirs:
        command += ["--data", str(directory)]
    command.append(expr)

    result = subprocess.run(command, capture_output=True, text=True)
    if result.returncode != 0:
        die(f"opa failed: {result.stderr.strip() or result.stdout.strip()}")
    values = json.loads(result.stdout)
    # A rule that yields anything but strings is a policy bug, and sorting a
    # mixed list would surface it as an unrelated TypeError.
    for value in values:
        if not isinstance(value, str):
            die(f"{expr} produced a {type(value).__name__}; "
                "deny and considered rules must yield strings")
    return values


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--plan", required=True, type=pathlib.Path,
                        help="JSON from `terraform show -json`")
    parser.add_argument("--policy", required=True, nargs="+", type=pathlib.Path,
                        help="one or more directories of .rego files")
    parser.add_argument("--opa", default="opa", help="path to the opa binary")
    args = parser.parse_args()

    if shutil.which(args.opa) is None:
        die(f"{args.opa} is not on PATH")
    for directory in args.policy:
        if not directory.is_dir():
            die(f"{directory} is not a directory")

    plan = load_plan(args.plan)
    if "resource_changes" not in plan:
        die(f"{args.plan} has no resource_changes key; is it a Terraform plan?")

    considered = query(args.opa, args.policy, args.plan, CONSIDERED_QUERY)
    if not considered:
        # Reporting a pass here would claim an assurance the run cannot give.
        print(f"VACUOUS: no policy examined any of the "
              f"{len(plan['resource_changes'])} planned resource(s)", file=sys.stderr)
        return BROKEN

    violations = sorted(query(args.opa, args.policy, args.plan, DENY_QUERY))
    if violations:
        print(f"FAIL: {len(violations)} violation(s) "
              f"across {len(considered)} examined resource(s)", file=sys.stderr)
        for violation in violations:
            print(f"  - {violation}", file=sys.stderr)
        return FAIL

    print(f"PASS: {len(considered)} resource(s) examined, no violations")
    return PASS


if __name__ == "__main__":
    sys.exit(main())

Here's what the script does:

1. argparse marks --plan and --policy as required=True, and --policy takes nargs="+", so an empty policy list raises an error at parse time.

2. load_plan reports the line and column of a JSON syntax error, because json.JSONDecodeError carries lineno and colno and a gate that says only "invalid JSON" wastes somebody's afternoon.

3. Every failure path routes through die, which always exits 2, because a missing opa, an unreadable file, and a plan with no resource_changes key are all failures of the tool itself.

4. The considered query runs before the deny query. If nothing was examined, the run ends at VACUOUS and never gets the chance to print a pass.

5. Violations are sorted, so the same plan produces byte-identical output on every run and a diff of two CI logs means something.

Seven violations across the two resources in this plan, and an exit code CI can act on.

The same tool on an empty plan, where a gate answering "pass" would be lying.

The GitHub Actions workflow tests the policies before it uses them to judge anything:

name: policy

on: [pull_request]

jobs:
  policy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - name: Install OPA
        run: |
          curl -L -o /usr/local/bin/opa \
            https://openpolicyagent.org/downloads/v1.20.2/opa_linux_amd64_static
          chmod +x /usr/local/bin/opa

      # The policies are code. Lint and test them before trusting them.
      - name: Check policy syntax
        run: opa check --strict policy

      - name: Verify formatting
        run: opa fmt --fail --diff policy

      - name: Test policies
        run: opa test policy --verbose --coverage --format json > coverage.json

      # Only now does anything get judged.
      - name: Evaluate Terraform plan
        run: python3 policy_gate.py --plan plan.json --policy policy

Roll this out with the gate reporting only, for a fortnight, before you let it block. A policy that looks obviously correct will fail on something structural in your real estate, and you would rather find that out from a log line than from a blocked release.

Step 6: Enforce at Admission Time

The gate in Step 5 checks what you intended to deploy. It doesn't see a kubectl apply from somebody's laptop, a vendor's Helm chart, or an operator creating Pods on its own schedule. For those, you need admission control, and Kubernetes now has it built in.

ValidatingAdmissionPolicy has been generally available since v1.30, evaluating CEL inside the API server with no webhook to deploy or keep alive:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-trusted-registry
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  variables:
    # A Pod has three container lists. A policy that reads only
    # spec.containers is bypassed by moving the image to an initContainer.
    - name: allImages
      expression: >-
        object.spec.containers.map(c, c.image) +
        object.spec.?initContainers.orValue([]).map(c, c.image) +
        object.spec.?ephemeralContainers.orValue([]).map(c, c.image)
  validations:
    - expression: >-
        variables.allImages.all(i, i.startsWith('registry.internal.example.com/'))
      messageExpression: >-
        'images must come from registry.internal.example.com: ' +
        variables.allImages.filter(i,
          !i.startsWith('registry.internal.example.com/')).join(', ')
      reason: Forbidden

The policy does nothing until a binding activates it, which is what lets you pilot on one namespace:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-trusted-registry-binding
spec:
  policyName: require-trusted-registry
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchLabels:
        policy.example.com/enforce: "true"

Set validationActions: ["Warn", "Audit"], label one namespace, watch for a week, and then switch to ["Deny"] and widen the selector.

The initContainers handling matters, because it's the most common way an image-provenance policy gets bypassed, and the ? optional-field syntax with .orValue([]) is how you read a list that may be absent without the whole expression erroring.

I couldn't apply these two manifests, because I had no cluster to hand. They're checked against the v1 reference schema, and no real API server has admitted them, so treat them as a starting point and roll them out in Warn mode (which you should be doing anyway).

Two other engines are in wide production use, starting with Kyverno. It graduated in the CNCF in March 2026 with production use at Bloomberg, Coinbase, Deutsche Telekom, LinkedIn, and Spotify. Its policies are written in YAML, so a platform team needs no new language, and it handles generation, image-signature verification, and cleanup that built-in policies leave alone.

OPA Gatekeeper is the right answer when you want one Rego codebase covering Kubernetes and Terraform and CI, which is the position this tutorial builds toward. Mutation is now built in too: MutatingAdmissionPolicy became stable in v1.36.

Step 7: Let a Model Write the Policy

Policies are tedious, and models are good at tedious. So the obvious move is to have the model write them.

There's a catch that you can measure yourself in about a minute, and I do exactly that at the end of this step: a great deal of the Rego in public training data is Rego v0, the dialect that stopped parsing when OPA 1.0 shipped in January 2025. A model reaching for the most common pattern it has seen reaches for a dialect the current parser rejects.

A 2025 preprint from a group at the University of Calabria, ARPaCCino, reports the same effect on a Terraform case study: asked for Rego with no tools, Qwen3-30B and GPT-4o each produced 0 of 5 syntactically correct policies. Adding retrieval over the OPA documentation changed nothing. Giving the model a loop that could run opa check and read the errors took those to 4 of 5 and 5 of 5.

Those counts come from one small case study, so treat the direction as the durable part of the result.

A feedback loop is what fixed it, and the loop costs nothing, because you already built it out of opa check --strict, opa fmt, and opa test.

So build the loop with one inversion that makes it trustworthy. I write the tests, and the model writes the policy. Test fixtures are concrete and cheap to review, since you read a JSON blob and say "yes, that should be rejected" in three seconds. Rego with nested comprehensions takes real effort to read and is easy to misread. Put the human where review is cheap, and let the machine work where its output can be checked mechanically.

Create policy_forge.py:

"""Generate a Rego policy from a rule in English, and keep it only if the toolchain agrees."""

import argparse
import pathlib
import re
import subprocess
import sys
import tempfile

WRITTEN, REJECTED, BROKEN = 0, 1, 2

SYSTEM = """You write Open Policy Agent policies in Rego v1 (OPA 1.0+).

Rules:
- Use `if` on every rule body and `contains` for multi-value rules.
- Do not emit `import rego.v1`; it is redundant on OPA 1.0+.
- The input is the JSON from `terraform show -json`.
- Return one ```rego block and nothing else."""


def extract_rego(reply: str) -> str:
    blocks = re.findall(r"```rego\n(.*?)```", reply, re.DOTALL)
    if not blocks:
        raise ValueError("model returned no rego block")
    if len(blocks) > 1:
        raise ValueError(f"model returned {len(blocks)} rego blocks; expected one")
    return blocks[0]


def verify(opa: str, policy: str, tests: pathlib.Path) -> tuple[bool, str]:
    with tempfile.TemporaryDirectory() as tmp:
        bundle = pathlib.Path(tmp)
        # The tests keep their own name; the policy gets one that cannot
        # collide with it, whatever the caller named the test file.
        (bundle / "candidate_policy.rego").write_text(policy)
        (bundle / tests.name).write_text(tests.read_text())

        for command in ([opa, "check", "--strict"], [opa, "test"]):
            result = subprocess.run(command + [str(bundle)], capture_output=True, text=True)
            if result.returncode != 0:
                return False, (result.stdout + result.stderr).strip()
    return True, "opa check and opa test both passed"


def forge(rule: str, tests: pathlib.Path, ask, opa: str, attempts: int) -> str:
    transcript = [{
        "role": "user",
        "content": (
            f"Write a Rego policy for this rule:\n\n{rule}\n\n"
            f"It must satisfy these tests:\n\n```rego\n{tests.read_text()}```"
        ),
    }]

    for attempt in range(1, attempts + 1):
        reply = ask(transcript)
        policy = extract_rego(reply)
        ok, output = verify(opa, policy, tests)
        headline = next(iter(output.splitlines()), "no output from the toolchain")
        print(f"attempt {attempt}: {'PASS' if ok else 'FAIL'} - {headline}", file=sys.stderr)
        if ok:
            return policy
        transcript += [
            {"role": "assistant", "content": reply},
            {"role": "user", "content": f"The toolchain rejected that:\n\n{output}\n\nFix it."},
        ]

    raise RuntimeError(f"no policy survived {attempts} attempts")


def claude(model: str):
    import anthropic

    client = anthropic.Anthropic()

    def ask(transcript: list[dict]) -> str:
        response = client.messages.create(
            model=model,
            max_tokens=16000,
            system=[{
                "type": "text",
                "text": SYSTEM,
                "cache_control": {"type": "ephemeral"},
            }],
            thinking={"type": "adaptive"},
            messages=transcript,
        )
        return "".join(b.text for b in response.content if b.type == "text")

    return ask


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--rule", required=True, help="the policy, in one English sentence")
    parser.add_argument("--tests", required=True, type=pathlib.Path,
                        help="a _test.rego file you wrote by hand")
    parser.add_argument("--out", required=True, type=pathlib.Path,
                        help="where to write the policy, only if it passes")
    parser.add_argument("--model", default="claude-opus-5")
    parser.add_argument("--attempts", type=int, default=4)
    parser.add_argument("--opa", default="opa")
    args = parser.parse_args()

    if args.attempts < 1:
        print("policy-forge: --attempts must be at least 1", file=sys.stderr)
        return BROKEN
    if not args.tests.is_file():
        print(f"policy-forge: {args.tests} does not exist", file=sys.stderr)
        return BROKEN

    try:
        policy = forge(args.rule, args.tests, claude(args.model), args.opa, args.attempts)
    except RuntimeError as exc:
        print(f"policy-forge: {exc}; nothing written", file=sys.stderr)
        return REJECTED
    except ValueError as exc:
        print(f"policy-forge: {exc}", file=sys.stderr)
        return BROKEN

    args.out.write_text(policy)
    print(f"policy-forge: verified policy written to {args.out}", file=sys.stderr)
    return WRITTEN


if __name__ == "__main__":
    sys.exit(main())

export ANTHROPIC_API_KEY=...
python3 policy_forge.py \
  --rule "No security group may expose an administrative port to the public internet." \
  --tests policy/network_test.rego \
  --out policy/generated.rego

Here's what the loop guarantees:

1. Verification runs as a subprocess: the model is never asked whether its policy is correct. opa check and opa test decide, and their exit codes are the only evidence the loop accepts.

2. Failures go back as raw tool output, never summarised: compiler errors and test failures are the highest-signal feedback a model can receive, and paraphrasing throws away the part that helps.

3. Nothing reaches disk until it passes: forge either returns a verified policy or raises, so there's no path where an unverified policy lands in the repository just because the retry budget ran out.

I drove the loop with a scripted model so the result is reproducible without an API key. The three replies were a realistic v0-syntax policy, a realistic-but-wrong v1 policy, and the policy from Step 3:

attempt 1: FAIL - 2 errors occurred during loading:
attempt 2: FAIL - policy/network_test.rego:63:
attempt 3: PASS - opa check and opa test both passed

Attempt 1 was Rego v0, the deny[msg] { ... } form, which stopped parsing when OPA 1.0 shipped in January 2025. And it's overwhelmingly what public training data contains. opa check --strict rejected it before it reached a test.

Attempt 2 was valid Rego v1. It would have passed review from most engineers, and it still scored only 4 of 8 on the suite. This is because it compared ingress.from_port against admin_ports directly, ignored the range, and read only cidr_blocks. opa check had no complaint, because the code was perfectly well-formed.

A well-formed policy can still be the wrong policy, and the only thing in this loop that knows what you wanted is the test suite you wrote.

A repair loop isn't monotonic, because each attempt is a fresh generation conditioned on an error message, with nothing carrying forward what already worked, so attempt four can lose a property attempt three had. Nothing in this design detects that, because the only thing being checked is the test suite you wrote.

Cap the retries, keep the suite growing, and treat every generated policy as a pull request that somebody approves before it merges.

Step 8: Govern the Agent Itself

An AI agent is also an actor, and it calls tools, so every tool call becomes an authorisation decision that something has to make.

The industry converged on this quickly: Amazon Bedrock AgentCore Policy reached general availability in March 2026, evaluating agent tool calls at the gateway in Cedar. The common open-source pattern is an OPA sidecar in front of an MCP tool gateway.

Research is pushing the same boundary harder: a 2026 preprint from the University of Washington group behind Defects4J, Solver-Aided Verification of Policy Compliance in Tool-Augmented LLM Agents (Winston, Winston, and Just), compiles natural-language policies into SMT constraints and blocks non-compliant calls with the Z3 solver.

They share one claim: a policy in the system prompt isn't enforcement. Enforcement is an interceptor sitting in the call path that can return "no" and stop the call from happening.

Create agent/authz.rego:

# METADATA
# title: Agent tool-call authorisation
# description: |
#   Evaluated once per tool call, before the tool runs. The decision has
#   three values rather than two, because an agent worth deploying will
#   sometimes need to do something that a human, not the policy, should
#   approve.
package agent.authz

tool_grants := {
	"support": {"search_orders", "read_customer", "issue_refund"},
	"analytics": {"search_orders", "run_query"},
}

write_tools := {"issue_refund", "run_query"}

refund_ceiling_cents := 10000

# An unmapped role, an unknown tool or a malformed input all land here.
default decision := {"effect": "deny", "reasons": ["no matching grant"]}

decision := {"effect": effect_for(reasons), "reasons": reasons} if {
	count(granted) > 0
	reasons := escalations
}

granted contains role if {
	some role in input.agent.roles
	input.tool in object.get(tool_grants, role, set())
}

effect_for(reasons) := "allow" if count(reasons) == 0

effect_for(reasons) := "require_approval" if count(reasons) > 0

escalations contains reason if {
	input.tool in write_tools
	not input.session.human_in_loop
	reason := sprintf("%q writes state and the session is unattended", [input.tool])
}

# A refund with no readable amount cannot be checked against the ceiling,
# so it escalates. Silence here would clear the exact call an attacker
# would craft.
escalations contains reason if {
	input.tool == "issue_refund"
	not positive_amount
	reason := "refund amount is missing, unreadable, or not positive"
}

# A negative amount is a charge wearing a refund's name.
positive_amount if {
	amount := object.get(input, ["arguments", "amount_cents"], null)
	is_number(amount)
	amount > 0
}

escalations contains reason if {
	input.tool == "issue_refund"
	amount := object.get(input, ["arguments", "amount_cents"], null)
	is_number(amount)
	amount > refund_ceiling_cents
	reason := sprintf(
		"refund of %d cents exceeds the %d cent ceiling",
		[amount, refund_ceiling_cents],
	)
}

# Keyword matching is a coarse guard, and it is here to show the shape of an
# argument-level rule. Anything holding real data wants a SQL parser: this
# catches `DROP TABLE` and misses a statement that spells it another way.
destructive_sql := `(?i)\b(drop|truncate|delete|alter|grant|revoke)\b`

escalations contains reason if {
	input.tool == "run_query"
	regex.match(destructive_sql, object.get(input, ["arguments", "statement"], ""))
	reason := "statement contains a destructive SQL keyword"
}

The decision vocabulary is closed, and I'll state it plainly here:

Here's why the policy is shaped that way:

1. default decision is deny: an unrecognised tool, a role you forgot to map, or a malformed input all end up there. A policy that defaults to allow fails open on exactly the inputs nobody anticipated, which is the set an attacker picks from.

2. Three values: binary authorisation forces a choice between blocking useful work and permitting dangerous work, and the third value is what makes a high-autonomy agent tolerable.

3. Reasons come back as a set: every applicable reason is collected. When somebody gets an approval prompt at three in the morning, "refund of 250000 cents exceeds the 10000 cent ceiling" tells them what to do. "Policy violation" does not.

4. Arguments are inspected too: issue_refund is routine at £5 and serious at £2,500, so tool-name granularity is far too coarse for agents, given that the agent chooses the arguments.

Fifteen tests cover the decision table, including an agent with an empty role list, a refund with no amount at all, and a query that hides DROP behind a newline:

opa test agent -v

PASS: 15/15

Serve it and try a call:

opa run --server --addr localhost:8181 agent/

curl -s localhost:8181/v1/data/agent/authz/decision \
  -d '{"input":{"agent":{"roles":["support"]},"tool":"issue_refund",
       "arguments":{"amount_cents":250000},"session":{"human_in_loop":true}}}' | jq .result

{
  "effect": "require_approval",
  "reasons": [
    "refund of 250000 cents exceeds the 10000 cent ceiling"
  ]
}

The policy is inert until something refuses to proceed on its answer. That's the client:

import json
import urllib.request

OPA_URL = "http://localhost:8181/v1/data/agent/authz/decision"


class PolicyDenied(Exception):
    pass


class ApprovalRequired(Exception):
    pass


def authorize(agent, tool, arguments, session):
    payload = json.dumps({"input": {
        "agent": agent, "tool": tool,
        "arguments": arguments, "session": session,
    }}).encode()
    req = urllib.request.Request(
        OPA_URL, data=payload, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=2) as resp:
        body = json.load(resp)

    # OPA returns {} with a 200 when a query matches nothing. Fail closed.
    decision = body.get("result", {"effect": "deny", "reasons": ["policy unavailable"]})

    if decision["effect"] == "deny":
        raise PolicyDenied("; ".join(decision["reasons"]))
    if decision["effect"] == "require_approval":
        raise ApprovalRequired("; ".join(decision["reasons"]))
    return decision

search_orders    -> ALLOWED
issue_refund     -> NEEDS APPROVAL (refund of 250000 cents exceeds the 10000 cent ceiling)
delete_account   -> DENIED (no matching grant)

Note body.get("result", ...): OPA returns {} with a 200 status when a query matches nothing, so a bare body["result"] raises KeyError, and depending on how your agent framework handles exceptions that may fail open. Every layer defaults to deny, including the parsing.

Call authorize() from your framework's tool-execution hook, before the tool function runs. It's about fifteen lines, and it turns a system prompt's polite suggestions into an actual boundary.

Step 9: What I Got Wrong

Steps 2 and 3 show the finished policies. I reached for something simpler first (the version most tutorials stop at), and the gap between that and what you have just read is the most useful thing here.

The Tagging Policy Skipped the Worst Resources

My naïve in_scope rule ended with resource.change.after.tags, which reads as "only resources that have tags".

What it actually does is worse than that, because Terraform emits tags: null for a resource with no tags at all, and an undefined lookup makes the rule body fail, so the resource drops out of scope entirely.

The TerraGoat plan has two resources: the security group carries five git_* tags and no ownership tags, while the VPC carries nothing at all.

jq -r '.resource_changes[] | "\(.address): tags=\(.change.after.tags | type)"' plan.json

aws_security_group.web-node: tags=object
aws_vpc.web_vpc: tags=null

opa eval --data naive  --input plan.json --format pretty 'count(data.terraform.tags.deny)'
opa eval --data policy --input plan.json --format pretty 'count(data.terraform.tags.deny)'

3
6

The three it missed were all on the completely untagged resource, so the policy flagged the resource with some tags and silently cleared the one with none.

tags_of with its else := {} branch is the fix, and it's three lines.

The Network Policy Read One of Four Shapes

My naïve network policy read aws_security_group and cidr_blocks, which is what every tutorial shows, but Terraform has four ways to express the same ingress rule.

![Diagram titled "Four ways Terraform describes one ingress rule". Four boxes are shown. Top left, aws_security_group.ingress[].cidr_blocks, filled pale blue and labelled read. The other three are outlined in red with red hatching and labelled not read: aws_security_group.ingress[].ipv6_cidr_blocks, aws_vpc_security_group_ingress_rule.cidr_ipv4, and the deprecated aws_security_group_rule. A key states that solid blue fill means the policy looks here and red hatch means it does not.](https://cdn.hashnode.com/uploads/covers/5f3a74bfc4d5973f55c91c8c/ab61d36d-78bf-45f7-a073-e7013216e3e1.png align="center")

One rule, four encodings, and the naïve policy read only the top-left one.

I planned a second real configuration with three security groups that open SSH to the world using the shapes the policy didn't read. Both versions are in code/, so this reproduces:

opa eval --data naive --input plan-evasion.json \
  --format pretty 'data.terraform.network.deny'

[]

That is three publicly reachable SSH ports, zero violations, and a gate that would have printed PASS and exited 0.

A Test That Was Holding a Hole Open

The port range had a second problem, and my own test suite was protecting it. I had written a case called test_tolerates_null_ports, asserting that an ingress rule with protocol: "-1" produced no violations, on the reasoning that a comparison against a null port shouldn't crash the policy.

An all-protocols rule opens every port, and the AWS provider records it as from_port: 0 and to_port: 0. The the range check read it literally as the single port zero, so the most permissive rule in AWS scored clean.

The plan is in code/plan-all-protocols.json, and it's one resource:

jq -c '.resource_changes[] | select(.type=="aws_security_group")
       | .change.after.ingress[0] | {protocol,from_port,to_port,cidr_blocks}' \
   plan-all-protocols.json
opa eval --data naive --input plan-all-protocols.json \
   --format pretty 'data.terraform.network.deny'

{"protocol":"-1","from_port":0,"to_port":0,"cidr_blocks":["0.0.0.0/0"]}
[]

Every protocol and every port, open to the whole internet, and a test I wrote on purpose certified it as fine. The fix is covered_ports, which maps an all-protocols rule onto the full range and goes undefined for ports it can't read, with a second deny rule that reports the undefined case. That test is gone and four took its place:

opa eval --data policy --input plan-all-protocols.json --format pretty 'data.terraform.network.deny'

[
  "aws_security_group.wide_open: ingress rule exposes port 22 to 0.0.0.0/0",
  "aws_security_group.wide_open: ingress rule exposes port 3306 to 0.0.0.0/0",
  "aws_security_group.wide_open: ingress rule exposes port 3389 to 0.0.0.0/0",
  "aws_security_group.wide_open: ingress rule exposes port 5432 to 0.0.0.0/0"
]

Step 7 argues that the test suite is the only artifact in the loop that knows what you wanted. This is the cost of that property: a test that's wrong is a specification that's wrong, and nothing downstream of it will argue.

What the Numbers Actually Were

Both versions score 1.000 on the plan I designed the policy against. The gap only appears on the plan I did not.

Across both real plans, the naïve policies found 4 of the 9 violations present. They scored 1.000 on the TerraGoat security group, which is the case I had in mind while writing them, and 0.000 and 0.500 on the two cases I did not.

The Part That Genuinely Surprised Me

I assumed test coverage would have caught this, and it doesn't. I reconstructed the naïve tagging policy with the two tests I originally wrote for it:

opa test . --coverage --format json | jq '{overall: .coverage}'

{
  "overall": 100
}

The naïve policy scored 100% coverage, passed 2 of 2 tests, and cleared a resource that carried no tags at all.

Coverage measures which lines of a policy your tests executed, and says nothing at all about which shapes of input you failed to imagine. For policy code, this is the entire failure mode. Coverage is worth reporting, and the evidence that a policy actually works comes from running it against infrastructure you didn't write.

Limits of the Check

Here's what this gate still can't do, because its false clearances matter more than its catches.

1. Unknown Values Are Invisible

Terraform marks anything it can't resolve until apply time as unknown, which appears as null in the plan JSON alongside an after_unknown map. A policy reading change.after.some_field doesn't fire when that field is unknown.

This bites hardest on cross-resource rules, where "every bucket has a public access block" is genuinely hard at plan time, because the block references a bucket ID that's usually (known after apply).

2. It Sees the Plan, and Only the Plan

Anything applied outside the pipeline, changed in a console, or drifted since creation stays invisible to it. Plan-time checks and admission-time checks have different blind spots, and both leave work for a periodic scan of deployed state.

3. Coverage of Controls Isn't Measurable from Inside

A hundred green checks say nothing about the rules nobody wrote. Keep the mapping from your control requirements to your policy files somewhere explicit and audit it on a schedule, because the gate can't tell you what it was never asked.

4. The CIDR List is an Exact Match

public_cidrs holds 0.0.0.0/0 and ::/0 and nothing else, so a rule opening 0.0.0.0/1 reaches half the internet and passes. Widening it means deciding which prefix lengths count as public and carving out RFC 1918 space, and that decision belongs to your organisation.

5. Four Shapes is What I Found

The exposures set covers the four encodings I went looking for, and AWS offers more. Security group references (security_groups), prefix lists, and self rules are all ways to reach a port that this policy doesn't model, and it never sees them. I would expect a fifth shape to turn up the first time this runs against a large estate.

6. Regulatory Dates Move

If you're building toward the EU AI Act, the Digital Omnibus published in July 2026 pushed Annex III high-risk obligations from 2 August 2026 to 2 December 2027, and Annex I obligations to 2 August 2028, while the Article 50 transparency duties kept their original 2 August 2026 date.

Encode the controls, and look the dates up in the official timeline every time you need one, including when a blog post from last quarter tells you otherwise (this one included).

Conclusion

Models produce infrastructure code that parses almost every time, is secure about 56% of the time, and arrives faster than anybody can read it. Manual review stopped being a real control somewhere in that gap. The rules were always meant to be executable, and the volume is what finally forced the issue.

In this tutorial, you:

• Pulled a real vulnerable security group from TerraGoat at commit 729f8da and planned it with Terraform 1.14.2.

• Wrote 20 policy tests covering four Terraform encodings of one ingress rule, and got them to pass on OPA 1.20.2.

• Caught 7 real violations in the TerraGoat plan, while correctly ignoring port 80.

• Built a gate with a four-verdict contract that exits 2 when it examined nothing.

• Watched the naïve version find 4 of 9 violations while reporting 100% test coverage, and hardened it to find 9 of 9.

• Wired a generation loop where opa check and your own tests decide what reaches disk.

• Authorised agent tool calls from the same engine, with 15 tests and a default of deny.

My policies handled the cases I wrote them for and missed two I hadn't imagined, and every signal available to me (tests passing, coverage at 100%, and a clean opa check) agreed they were fine. The one thing that disagreed was infrastructure somebody else had written. Point your policies at code you didn't write, early, and keep the failures.

All the code, the policies, the plan JSON, and the scripts that build the figures are in the code/ directory alongside this handbook. The figures regenerate with python3 build/make_images.py and python3 build/make_terminals.py. The terminal screenshots re-run their commands at build time, so they can't drift from the truth.

That score looked like a success, but the model had learned very little about the real world. CDC had calculated the sixth column from the other five, so the model simply worked out CDC's formula.

Data scientists call this problem target leakage, and it happens when the inputs you give a model already contain the answer in some form.

Leakage like this hides easily in public data, because a large share of public data is calculated from other public data. A government index might be built from survey columns, and a second index might be built from the first one. Agencies explain these recipes in their methodology PDFs, yet data catalogues rarely store them in a form a computer can check.

In this tutorial, you'll write that record yourself and then build a small Python tool that reads it. The tool works like the dependency checker inside a package manager: you tell it what you want to predict and which columns you plan to use, and it refuses any column that sits on a derivation path to or from your target.

By the end, you'll know how to:

• reproduce a real leak using live CDC data and scikit-learn

• describe what a dataset was built from in a small YAML file called a manifest

• walk that graph with breadth-first search and depth-first search

• make a checking tool that fails loudly on typos, broken files, and empty inputs

• run the check automatically on every push with GitHub Actions

Table of Contents

• Prerequisites

• Key Terms in Plain English

• Step 1: See the Leak for Yourself

• Step 2: Understand Why Public Data Leaks

• Step 3: Borrow an Idea from Package Managers

• Step 4: Write the Dependency Manifest

• Step 5: Build the Linter

• Step 6: Run the Linter on Real Cases

• Step 7: Make the Linter Fail Loudly on Bad Input

• Step 8: Run the Check Automatically in CI

• Step 9: Learn from My Mistakes

• Going Further with the Full Tool

• Conclusion

Prerequisites

To follow along, you'll need:

• Python 3.10 or newer

• a basic idea of what a pandas DataFrame is

• a terminal where you can run commands

• about 7 MB of free disk space for the CDC data file

Create a fresh project folder with a virtual environment inside it, so these libraries stay separate from the rest of your system. Then install the three libraries this tutorial uses:

mkdir leak-tutorial
cd leak-tutorial
python3 -m venv .venv
source .venv/bin/activate
pip install pandas scikit-learn pyyaml

On Windows, run .venv\Scripts\activate in place of the source line, and type python wherever this article says python3. Run every command in this tutorial from inside the leak-tutorial folder.

Every file you build in this tutorial is also in the companion repository, derives-from-tutorial, so you can compare your work against it if you get stuck.

The full version of the tool lives in a public GitHub repository, and I link to it at the end of the article.

Key Terms in Plain English

Here are five words that come up again and again in this tutorial:

• Target: the column you want your model to predict.

• Covariate: a column you feed into the model to help it predict the target (many people call these features).

• R² (R-squared): a score that tells you how closely a model's predictions match the real values. A score of 1.0 means a perfect match, and a score near 0 means the model explains very little.

• Census tract: a small area of the United States that usually holds about 4,000 people, roughly the size of a neighbourhood.

• Cross-validation: a fair way to test a model. You split the data into five parts, train on four, test on the fifth, and repeat until every part has had a turn as the test set.

Step 1: See the Leak for Yourself

The US Centers for Disease Control and Prevention (CDC) publishes the Social Vulnerability Index, or SVI. Emergency planners use it to find communities that may need extra help during a flood, a heatwave, or a disease outbreak.

CDC builds the SVI in layers. It starts with 16 columns from the American Community Survey (ACS), a large survey run by the US Census Bureau. Each column is a percentage, such as the share of people living in poverty or the share of households with zero vehicles.

CDC groups those 16 columns into four themes and ranks every census tract within each theme. It then combines the four theme ranks into one overall rank called RPL_THEMES.

How CDC builds the SVI: 16 ACS survey columns feed four theme ranks, and the four theme ranks feed the one overall rank. Every yellow box is calculated from the boxes below it.

Here's the detail that matters for this tutorial: CDC ships the raw ACS columns and the finished ranks together in the same CSV file. That makes it very easy to grab both and put them into one model.

Download the California file:

curl -L -o California.csv https://svi.cdc.gov/Documents/Data/2022/csv/states/California.csv

I use curl here because some Python installs on macOS fail to verify the website's security certificate when they download files directly.

Now create a file called leak_demo.py:

"""leak_demo.py: predict a published index from the columns it was built from."""
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import KFold, cross_val_score

# CDC marks missing values as -999, so turn those into proper blanks.
df = pd.read_csv("California.csv", low_memory=False).replace(-999, float("nan"))


def score(inputs, target):
    data = df[inputs + [target]].dropna()
    model = HistGradientBoostingRegressor(random_state=0)
    folds = KFold(n_splits=5, shuffle=True, random_state=0)
    r2 = cross_val_score(model, data[inputs], data[target],
                         cv=folds, scoring="r2").mean()
    print(f"{target:<11} from {len(inputs)} column(s)  "
          f"tracts={len(data)}  R2 = {r2:.3f}")


# Theme 1 is built from exactly these five columns.
score(["EP_POV150", "EP_UNEMP", "EP_HBURD", "EP_NOHSDP", "EP_UNINSUR"],
      "RPL_THEME1")

# EP_NOINT ships in the same file, and CDC leaves it out of the index.
score(["EP_NOINT"], "RPL_THEMES")

Here's what the script does:

1. It loads the CSV and turns CDC's -999 markers into blank values, because CDC uses -999 to flag a missing value.

2. The score function trains a gradient boosting model, which is a strong and popular choice for tables of numbers, and it measures R² with five-fold cross-validation.

3. The first call predicts Theme 1 using the exact five columns CDC used to build Theme 1.

4. The second call predicts the overall rank using EP_NOINT, the share of households lacking a broadband internet subscription. CDC includes this column in the same file and leaves it out of the index.

Run it:

python3 leak_demo.py

The output of leak_demo.py. Theme 1, predicted from the five columns CDC built it from, scores R² = 0.998. The overall rank, predicted from a column CDC leaves out of the index, scores 0.384.

The first score is 0.998, which means the model rebuilt CDC's Theme 1 almost perfectly. CDC's formula is a fixed recipe, and the model had every ingredient.

The second score is 0.384. EP_NOINT sits beside the index in the file, and its score shows the size of an ordinary link between two related measures.

Now imagine a paper that reports R² = 0.998 for predicting social vulnerability. That number would look like a breakthrough, yet it would only show that the model had found CDC's recipe.

The scores in this article came from scikit-learn 1.8.0. They stay the same to three decimal places across scikit-learn 1.3.2 to 1.9.0, so your run should match.

Step 2: Understand Why Public Data Leaks

The SVI example is easy to spot because the inputs and the index sit in one file. Most real cases are harder, because the chain runs across several agencies.

Here's one real chain that crosses three organisations. FEMA's National Risk Index (NRI) includes a social vulnerability score. According to FEMA's technical documentation (version 1.20, December 2025), that score comes from the Census Bureau's Community Resilience Estimates. The Census Bureau builds those estimates from ACS survey data.

So a FEMA risk score and an ACS column can sit at two ends of one chain, even though they come from different agencies and different websites.

To see why computers miss this, you need to know about two kinds of history a number can have:

• Provenance answers the question "Where did this number arrive from?" For example, a value came from California.csv, which came from svi.cdc.gov.

• Derivation answers the question "What was this number calculated from?" For example, Theme 1 was calculated from five ACS columns.

Two kinds of history a number can have. Provenance, on the left, records the file and the website a value arrived from. Derivation, on the right, records the five ACS columns Theme 1 was calculated from.

Most data catalogues store provenance well, and Google's Data Commons is a good example: it defines provenance as "the physical unit of an import", which tells you the file a number came in. Derivation usually lives only in PDF methodology documents written for humans.

So when an automated pipeline searches for helpful covariates, it can happily collect columns that the target was built from. The pipeline sees high scores and keeps those columns.

Step 3: Borrow an Idea from Package Managers

Software developers solved a very similar problem long ago.

When you run pip install requests, pip reads a list of what requests depends on, then what those packages depend on, and so on down the tree. Because every dependency is written down, pip can spot trouble anywhere in the tree before it installs anything.

The diagram below puts that tree beside the data version of the same problem.

The same shape twice. On the left, pip's dependency tree: urllib3 and certifi feed requests, which feeds my-app. On the right, the data version: ACS.EP_UNEMP goes into a model that predicts FEMA_NRI.risk_score, and the same column also climbs through two other products into that risk score.

Public data needs the same kind of record. In the right-hand half of the diagram above, ACS.EP_UNEMP (the unemployment rate) goes into the model as a covariate. The same column also climbs up through two other products into FEMA_NRI.risk_score, which is the target. The column sits at both ends of the loop.

In computer science, this kind of diagram is a graph. Each box is a node, and each arrow is an edge. In a data graph, following the arrows always leads you upward and away from where you started, so the graph is a directed acyclic graph, or DAG for short. "Acyclic" means the arrows form zero loops.

Throughout this article, every arrow points from an ingredient to the product made from it.

Two family words help describe positions in the graph:

• An ancestor of a node is anything you reach by following arrows backwards from it, at any distance. The ACS columns are ancestors of the SVI.

• A descendant of a node is anything you reach by following arrows forwards from it. The SVI is a descendant of the ACS columns.

Your leak check then becomes one simple rule: every covariate must stay clear of the target's ancestors and descendants.

Step 4: Write the Dependency Manifest

A manifest is a file that lists every product and what each one was built from. You'll use YAML here because people can read and edit it easily.

Here's how one measured product looks:

ACS.EP_UNEMP:
  label: Unemployment rate
  measurementBasis: measured
  derivesFrom: []

The empty list in derivesFrom: [] records zero parents, because this product comes straight from a survey.

And here is a product built from another product:

FEMA_NRI.social_vulnerability:
  label: FEMA National Risk Index, social vulnerability
  measurementBasis: composite
  derivesFrom:
    - {variable: CENSUS_CRE.social_vulnerability, relation: identity, confidence: documented}

Each entry in derivesFrom is one edge in the graph, and every edge carries three facts:

• variable holds the name of the parent product, and that name must match a product defined elsewhere in the file.

• relation describes how the parent was used.

• confidence records how sure you are about the edge.

These are the five relations:

These are the three confidence levels:

The confidence field matters more than it first appears. A lineage graph full of guesses would recreate the same problem it aims to solve, so each edge should say how much evidence stands behind it.

Each product also has a measurementBasis:

This field records something public catalogues usually leave out: whether a number was counted or predicted. A census count and a random forest prediction look identical in a spreadsheet, yet they're very different kinds of evidence.

Now create mini-manifest.yaml with the content below. It's a trimmed slice of the full manifest with 11 products from real US data infrastructure, and each product keeps a few of its real edges so the file stays short.

# A small slice of derivation-manifest.yaml, used in the tutorial.
schema: derives-from/0.2

products:

  # ---- measured: counted or surveyed directly
  ACS.EP_POV150:
    label: Population below 150% of the poverty line
    measurementBasis: measured
    derivesFrom: []

  ACS.EP_UNEMP:
    label: Unemployment rate
    measurementBasis: measured
    derivesFrom: []

  ACS.EP_NOVEH:
    label: Households with zero vehicles
    measurementBasis: measured
    derivesFrom: []

  SAT.chirps_rainfall:
    label: CHIRPS satellite rainfall
    measurementBasis: measured
    derivesFrom: []

  NVSS.mortality:
    label: Death certificate records
    measurementBasis: measured
    derivesFrom: []

  # ---- built from other products
  CENSUS_CRE.social_vulnerability:
    label: Census Community Resilience Estimates, social vulnerability
    measurementBasis: modelled
    derivesFrom:
      - {variable: ACS.EP_POV150, relation: modelled_from, confidence: documented}
      - {variable: ACS.EP_UNEMP,  relation: modelled_from, confidence: documented}
      - {variable: ACS.EP_NOVEH,  relation: modelled_from, confidence: documented}

  FEMA_NRI.social_vulnerability:
    label: FEMA National Risk Index, social vulnerability
    measurementBasis: composite
    derivesFrom:
      - {variable: CENSUS_CRE.social_vulnerability, relation: identity, confidence: documented}

  HVRI.bric:
    label: Baseline Resilience Indicators for Communities
    measurementBasis: composite
    derivesFrom:
      - {variable: ACS.EP_UNEMP, relation: component, confidence: documented}
      - {variable: ACS.EP_NOVEH, relation: component, confidence: documented}

  FEMA_NRI.community_resilience:
    label: FEMA National Risk Index, community resilience
    measurementBasis: composite
    derivesFrom:
      - {variable: HVRI.bric, relation: identity, confidence: documented}

  FEMA_NRI.expected_annual_loss:
    label: FEMA National Risk Index, expected annual loss
    measurementBasis: modelled
    derivesFrom: []

  FEMA_NRI.risk_score:
    label: FEMA National Risk Index, overall risk score
    measurementBasis: composite
    derivesFrom:
      - {variable: FEMA_NRI.expected_annual_loss, relation: component, confidence: certain}
      - {variable: FEMA_NRI.social_vulnerability, relation: component, confidence: certain}
      - {variable: FEMA_NRI.community_resilience, relation: component, confidence: certain}

Step 5: Build the Linter

A linter is a tool that reads something and warns you about problems before they cause harm. Code linters such as Flake8 read source code, while this linter reads your manifest and your list of covariates.

Create a file called mini_lint.py. You'll build it in six parts, and the finished file stays under 200 lines.

Part 1: Load YAML and Refuse Duplicate Keys

"""mini_lint.py: refuse covariates that sit on a derivation path to or from the target."""
import argparse
import sys
from collections import deque
from itertools import combinations

import yaml

RANK = {"certain": 3, "documented": 2, "inferred": 1}
DETERMINISTIC = {"component", "identity", "denominator"}


def fail(message):
    """Exit code 2 means the manifest or the command itself is broken."""
    print(message, file=sys.stderr)
    sys.exit(2)


# ---------------------------------------------------------------- step 1
class StrictLoader(yaml.SafeLoader):
    """A YAML loader that stops on a repeated key."""


def refuse_duplicates(loader, node, deep=False):
    seen = {}
    for key_node, _ in node.value:
        key = loader.construct_object(key_node, deep=deep)
        line = key_node.start_mark.line + 1
        if key in seen:
            fail(f"manifest error: key {key!r} appears twice "
                 f"(line {seen[key]} and line {line})")
        seen[key] = line
    return loader.construct_mapping(node, deep=deep)


StrictLoader.add_constructor(
    yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, refuse_duplicates)

RANK turns confidence words into numbers, so the tool can find the weakest edge in a route. DETERMINISTIC lists the relations that are pure arithmetic.

fail prints a message and exits with code 2. Later in the tutorial, you'll see why code 2 must stay separate from code 1.

The loader deals with a sneaky YAML behaviour. If a key appears twice in the same block, PyYAML quietly keeps the last copy and throws the first one away. In a manifest, that can erase every edge of a product, and the tool would then see zero routes and happily clear a leaky covariate.

refuse_duplicates runs every time PyYAML builds a mapping (a Python dictionary). It walks through the keys, remembers the line number of each one, and stops the program as soon as a key repeats.

Part 2: Read Products and Edges

# ---------------------------------------------------------------- step 2
def load(path):
    try:
        with open(path) as fh:
            products = yaml.load(fh, StrictLoader)["products"]
    except (OSError, yaml.YAMLError, KeyError, TypeError) as e:
        fail(f"manifest error: unable to read {path}: {e}")

    edges = {}
    for name, product in products.items():
        edges[name] = []
        for e in product.get("derivesFrom") or []:
            if RANK.get(e.get("confidence")) is None:
                fail(f"manifest error: {name} has an edge with "
                     f"confidence {e.get('confidence')!r}")
            edges[name].append((e["variable"], e["relation"], e["confidence"]))
    return products, edges

load opens the file with the strict loader. If anything goes wrong while reading, such as a bad path or broken YAML, it calls fail.

It then builds a dictionary called edges. For each product name, it stores a list of (parent, relation, confidence) tuples. For example:

edges["FEMA_NRI.social_vulnerability"]
# [("CENSUS_CRE.social_vulnerability", "identity", "documented")]

It also checks that every confidence value is one of the three allowed words. A typo such as documneted would otherwise slip through and break the ranking later.

Part 3: Check the Manifest Before Trusting It

# ---------------------------------------------------------------- step 3
def undefined_names(products, edges):
    mentioned = {parent for rows in edges.values() for parent, _, _ in rows}
    return sorted(mentioned - set(products))


def find_cycle(edges):
    state = {}

    def visit(node, stack):
        state[node] = "open"
        stack.append(node)
        for parent, _, _ in edges.get(node, []):
            if state.get(parent) == "open":
                return stack[stack.index(parent):] + [parent]
            if parent in state:
                continue
            cycle = visit(parent, stack)
            if cycle:
                return cycle
        stack.pop()
        state[node] = "closed"
        return None

    for node in edges:
        if node in state:
            continue
        cycle = visit(node, [])
        if cycle:
            return cycle
    return None

A typo in a parent name, such as HVRI.brick in place of HVRI.bric, creates an edge that points at a product the file lacks. The traversal would stop at that dead end, and every covariate beyond it would look safe.

undefined_names collects every parent mentioned in any edge and subtracts the set of defined products. Anything left over is either a typo or a product you forgot to add.

find_cycle makes sure the manifest really is a DAG. A product built from itself is impossible in real data, and a loop would send the route finder around in circles forever.

The function uses depth-first search with two labels. When the search enters a node, it marks that node open. When it has finished exploring everything above the node, it marks it closed. If the search reaches a node that's still open, it has walked in a circle, and the function returns that circle so you can see it.

Part 4: Walk the Graph

# ---------------------------------------------------------------- step 4
def ancestors(edges, node):
    """Every product that `node` was built from, at any distance."""
    found = set()
    queue = deque([node])
    while queue:
        current = queue.popleft()
        for parent, _, _ in edges.get(current, []):
            if parent in found:
                continue
            found.add(parent)
            queue.append(parent)
    return found


def routes(edges, start, goal):
    """Every path from start up to goal. Safe because step 3 ruled out cycles."""
    found = []
    for parent, relation, confidence in edges.get(start, []):
        step = (parent, relation, confidence)
        if parent == goal:
            found.append([step])
        else:
            for rest in routes(edges, parent, goal):
                found.append([step] + rest)
    return found


def describe(start, route):
    chain = " -> ".join([start] + [parent for parent, _, _ in route])
    weakest = min(route, key=lambda step: RANK[step[2]])[2]
    arithmetic = all(rel in DETERMINISTIC for _, rel, _ in route)
    kind = "deterministic" if arithmetic else "statistical"
    return [chain, f"{kind}, weakest link: {weakest}"]

ancestors uses breadth-first search (BFS), so picture a queue at a ticket counter: you put the starting product in the queue. On each turn, you take the product at the front, look up its parents, and add each parent you have yet to see to the back of the queue. When the queue is empty, the found set holds every ancestor at every distance.

The found set also stops the search from visiting the same product twice. That matters because many products share parents.

ancestors tells you whether a covariate is upstream, and routes tells you how it gets there.

routes uses depth-first search with recursion. For each parent of start, it checks whether that parent is the goal. If it is, that single step is a complete route. Otherwise, the function calls itself to find every route from the parent to the goal, then puts the current step on the front of each one.

The function returns every route, and that choice is deliberate. An earlier version of my full tool reported only the shortest route, so the report showed whichever route had the fewest hops, even when a longer route rested on stronger evidence.

The recursion is safe here only because Part 3 already confirmed that the graph is a DAG.

describe turns a route into two readable lines. The first line is the chain of names. The second line says whether the route is deterministic (arithmetic at every step) or statistical (at least one model in the chain), and it names the weakest confidence level along the route, since a chain is only as strong as its weakest link.

Part 5: The Audit

The audit looks for three shapes in the graph:

Each panel shows one shape and the verdict it produces: an arrow running into the target (FAIL), an arrow running out of the target (FAIL), and two covariates hanging off one shared input (REVIEW).

• Ancestor: the covariate went into the target, directly or through other products. This is the classic leak, so the tool reports it as an error.

• Descendant: the target went into the covariate. Predicting a parent from its own child leaks just as badly, so this is also an error.

• Shared ancestor: two covariates came from the same input. This is a softer problem, because the pair carries overlapping information, so the tool raises a warning for a person to review.

# ---------------------------------------------------------------- step 5
def audit(products, edges, target, covariates):
    findings = []

    unknown = [n for n in [target, *covariates] if products.get(n) is None]
    if unknown:
        return [("ERROR", f"unknown name: {n}", ["check the spelling"])
                for n in unknown]

    target_ancestors = ancestors(edges, target)
    for cov in covariates:
        if cov in target_ancestors:
            found = routes(edges, target, cov)
            details = [line for r in found for line in describe(target, r)]
            findings.append(("ERROR", f"{cov} is an ancestor of the target "
                                      f"({len(found)} route(s))", details))
        if target in ancestors(edges, cov):
            found = routes(edges, cov, target)
            details = [line for r in found for line in describe(cov, r)]
            findings.append(("ERROR", f"{cov} is a descendant of the target "
                                      f"({len(found)} route(s))", details))

    for a, b in combinations(covariates, 2):
        if a in ancestors(edges, b) or b in ancestors(edges, a):
            findings.append(("ERROR", f"{a} and {b}: one is built from the other", []))
        elif ancestors(edges, a) & ancestors(edges, b):
            shared = sorted(ancestors(edges, a) & ancestors(edges, b))
            findings.append(("WARN", f"{a} and {b} share ancestors", shared))
    return findings

The audit starts with name checks: if you misspell a covariate, the tool reports an error straight away, because it holds zero information about a name outside the manifest, and calling that name safe would be a guess.

Next, it computes the target's ancestors once and tests each covariate against that set. It also computes each covariate's ancestors to see whether the target appears among them, which is how it catches descendants.

Finally, combinations from the itertools module produces every pair of covariates. If one covariate is built from the other, that's an error. If the pair shares any ancestor, that's a warning.

Part 6: Verdicts and Exit Codes

# ---------------------------------------------------------------- step 6
def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--manifest", default="mini-manifest.yaml")
    parser.add_argument("--target", required=True)
    parser.add_argument("--covariates", nargs="+", required=True)
    args = parser.parse_args()

    products, edges = load(args.manifest)
    missing = undefined_names(products, edges)
    if missing:
        fail(f"manifest error: undefined names: {', '.join(missing)}")
    cycle = find_cycle(edges)
    if cycle:
        fail(f"manifest error: cycle: {' -> '.join(cycle)}")

    findings = audit(products, edges, args.target, args.covariates)
    severities = {severity for severity, _, _ in findings}
    if "ERROR" in severities:
        verdict = "FAIL"
    elif "WARN" in severities:
        verdict = "REVIEW"
    elif ancestors(edges, args.target):
        verdict = "PASS"
    else:
        verdict = "UNTRACED"

    basis = products.get(args.target, {}).get("measurementBasis", "unknown")
    print(f"target      {args.target}  [{basis}]")
    print(f"covariates  {', '.join(args.covariates)}")
    print(f"verdict     {verdict}\n")
    for severity, message, details in findings:
        print(f"  {severity:<5} {message}")
        for line in details:
            print(f"        {line}")

    sys.exit(1 if verdict == "FAIL" else 0)


if __name__ == "__main__":
    main()

main reads the command-line flags, loads the manifest, runs both self-checks, and then runs the audit. It turns the findings into one of four verdicts:

A broken manifest or a malformed command exits with code 2.

PASS and UNTRACED deserve a closer look. PASS means the tool walked a real family tree and found every covariate outside it. UNTRACED means the manifest holds an empty family tree for the target, so the walk had zero steps to take. Calling that a pass would flatter the tool, so it gets its own name.

The exit codes matter just as much. Exit code 1 means the check ran and found a leak, while exit code 2 means the check itself is broken. A CI pipeline needs to tell these two apart, because a leak asks you to change your covariates and a broken manifest asks you to fix the file.

Step 6: Run the Linter on Real Cases

Case 1: FEMA's Risk Score

FEMA's composite risk score multiplies Expected Annual Loss by a community risk factor. That factor is built from a social vulnerability score and a community resilience score, and both of those reach back to ACS survey columns through different organisations.

FEMA_NRI.risk_score and everything it was built from. Two routes, drawn in blue and red, both start at the same ACS unemployment column: one climbs through the Census Bureau's resilience estimates, the other through HVRI's BRIC index.

Suppose you want to predict the risk score using the poverty rate and the unemployment rate:

python3 mini_lint.py --target FEMA_NRI.risk_score --covariates ACS.EP_POV150 ACS.EP_UNEMP

The verdict is FAIL. The poverty rate reaches the target by one route, and the unemployment rate reaches it by two, one statistical and one deterministic.

The verdict is FAIL, with exit code 1.

ACS.EP_POV150 reaches the target by one route. ACS.EP_UNEMP reaches it by two.

The first unemployment route passes through the Census model, so the tool labels it statistical. The second route passes through HVRI's BRIC index, where unemployment is a direct ingredient, so the tool labels it deterministic.

Try tracing both routes by eye in a spreadsheet of column names and you'll quickly see why a graph helps. The traversal finds both in a fraction of a second.

Case 2: A Descendant

Now flip the direction and suppose you want to predict the Census score using FEMA's republished copy of it as a covariate:

python3 mini_lint.py --target CENSUS_CRE.social_vulnerability --covariates FEMA_NRI.social_vulnerability

The flipped case, and another FAIL. FEMA's republished copy of the Census score is a descendant of the target by one deterministic route.

FEMA's score is built directly from the Census score, so using it as an input hands the model the answer. The tool catches this as a descendant.

Case 3: REVIEW, PASS, and UNTRACED

Here are three runs that should come back clean or nearly clean:

python3 mini_lint.py --target NVSS.mortality --covariates FEMA_NRI.social_vulnerability HVRI.bric
python3 mini_lint.py --target FEMA_NRI.risk_score --covariates SAT.chirps_rainfall
python3 mini_lint.py --target NVSS.mortality --covariates SAT.chirps_rainfall

Three cleaner runs: REVIEW for the pair that shares two ACS parents, PASS for satellite rainfall against the risk score, and UNTRACED for a target with zero listed parents.

The first run returns REVIEW, because the two covariates share two ACS parents and overlap in what they tell the model.

The second run returns PASS, because the risk score has a traced family tree and satellite rainfall sits outside it.

The third run returns UNTRACED, because death certificate records are a direct count with zero listed parents.

These quiet results matter as much as the failures. A checker that raised an alarm on every input would be useless, so a good test set always includes cases that should pass.

Step 7: Make the Linter Fail Loudly on Bad Input

A safety tool earns trust by failing clearly. The worst outcome for a leak checker is a green PASS on a check that quietly skipped its work, and there are three common ways that can happen.

Trap 1: Typos in Names

To try this, copy mini-manifest.yaml to typo-manifest.yaml and change HVRI.bric to HVRI.brick inside the FEMA_NRI.community_resilience entry. Then run these two commands:

python3 mini_lint.py --target FEMA_NRI.risk_score --covariates ACS.EP_POV15
python3 mini_lint.py --manifest typo-manifest.yaml --target FEMA_NRI.risk_score --covariates ACS.EP_UNEMP

Two typos, two exit codes. The misspelled covariate becomes a FAIL with exit code 1, and the misspelled parent inside the manifest becomes a manifest error with exit code 2.

A misspelled covariate (ACS.EP_POV15) becomes a FAIL with exit code 1. A misspelled parent inside the manifest (HVRI.brick) becomes a manifest error with exit code 2. Both stop the run before any traversal happens.

Trap 2: Duplicate YAML Keys

Make another copy of the manifest called sneaky-manifest.yaml, then add one extra line at the very end of the file, inside the FEMA_NRI.risk_score block:

    derivesFrom: []

The risk score now has two derivesFrom keys. Create peek.py to see what plain PyYAML does with that:

import yaml

with open("sneaky-manifest.yaml") as fh:
    doc = yaml.safe_load(fh)

print(doc["products"]["FEMA_NRI.risk_score"]["derivesFrom"])

Now run peek.py, and then run the linter on the same file:

python3 peek.py
python3 mini_lint.py --manifest sneaky-manifest.yaml --target FEMA_NRI.risk_score --covariates ACS.EP_UNEMP

The duplicate key, seen two ways. Plain yaml.safe_load prints an empty list and says nothing, while the strict loader names the repeated key, both line numbers, and exits with code 2.

Plain yaml.safe_load returns an empty list, because PyYAML kept the second key, threw away all three real edges, and stayed silent about it. A linter built on that loader would find zero routes and let every leaky covariate through.

The strict loader stops with exit code 2 and points at both line numbers.

Trap 3: An Empty Covariate List

The third trap is easy to overlook. In CI, you might build the covariate list from a file or a shell variable. If that file is empty or the variable name has a typo, the command ends up with zero covariates.

An earlier version of my full tool accepted that and printed PASS, which is a clean bill of health for a check that skipped all its work.

python3 mini_lint.py --target FEMA_NRI.risk_score --covariates

An empty --covariates list now stops the run at argparse, before any traversal, with exit code 2.

In mini_lint.py, nargs="+" tells argparse that --covariates needs at least one value, and required=True makes the flag itself mandatory. An empty list now turns the pipeline red with exit code 2.

Step 8: Run the Check Automatically in CI

CI (continuous integration) runs checks for you every time you push code. This GitHub Actions workflow runs the linter on every push and every pull request. Save it as .github/workflows/lineage.yml in a repository that holds mini_lint.py and mini-manifest.yaml at its root:

name: lineage-check

on: [push, pull_request]

jobs:
  lint-lineage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - name: Check covariates against the target's lineage
        run: |
          python3 mini_lint.py \
            --target FEMA_NRI.risk_score \
            --covariates $(cat features.txt)

Put your covariate names in a file called features.txt at the root of the repository, one name per line:

SAT.chirps_rainfall

The $(cat features.txt) part pastes those names into the command, and with this file the job passes and turns green. If someone later adds a leaky covariate such as ACS.EP_UNEMP, the job exits with code 1 and turns red. If someone empties features.txt by accident, argparse exits with code 2, and the job turns red as well.

REVIEW and UNTRACED also exit with code 0. If you want your pipeline to stop on those as well, change the last line of main so that every verdict other than PASS exits with code 1.

The companion repository runs this exact workflow, and you can see its results on the repository's Actions tab.

Step 9: Learn from My Mistakes

Building the full manifest taught me that a linter is only as good as the graph it reads. I got the SVI wrong twice, in opposite directions, and both mistakes came from the same habit of trusting the columns in a file over the methodology behind it.

Mistake 1: The SVI California file contains 24 columns whose names start with EP_, but CDC ranks only 16 of them into the index. My first manifest counted all 24 and recorded EP_NOINT (broadband subscriptions) as an ingredient of the index. That column sits in the file, and CDC leaves it out of the ranking, so I removed the edge.

Mistake 2: I then over-corrected and listed all eight unranked columns as safe bystanders that ship beside the index. Seven of those eight are race and ethnicity columns. When I checked the raw counts, those seven added up to E_MINRTY exactly, and the largest difference across all 9,109 tracts was zero.

EP_MINRTY is the single input to Theme 3, and Theme 3 feeds the overall index.

Why those seven columns aren't bystanders. They sum exactly to EP_MINRTY, which feeds Theme 3 one hop up, which feeds the overall index one hop above that. EP_NOINT, in the dashed box, ships in the same file and stays outside the index.

So those seven columns are ancestors of the index, two hops up. My own linter had been clearing them as safe covariates for an SVI target, which is exactly the kind of false clearance the tool exists to prevent.

Cross-validated R² for each product, predicted from the columns listed beside it. Every product predicted from its own inputs scores 0.987 or higher, while EP_NOINT, which only ships beside the index, reaches 0.384.

The chart makes the difference plain: the seven race and ethnicity columns rebuild Theme 3 at R² = 0.992, while EP_NOINT alone reaches only 0.384 against the overall index.

I now follow a stricter rule before I mark any column as safe. I read the methodology first, and then I test the relationship in the data.

The full manifest records the result with a field called coPublishedNonInputs, which lists columns that ship in the same file as an index and take zero part in computing it. For the SVI, EP_NOINT is now the only entry.

Going Further with the Full Tool

The mini linter in this tutorial covers the core ideas. The full project adds:

• a manifest with 60 products and 75 derivation edges across US and global data, including CDC PLACES, FEMA's National Risk Index, WorldPop, AlphaEarth satellite embeddings, and WFP's HungerMap LIVE

• a written evidence note for every product that carries edges, plus a correction field wherever an earlier claim turned out to be wrong

• a --graph mode that prints the whole derivation graph

• a built-in suite of eight real audit cases

• a reproduce_svi.py script that checks every R² figure in this article against the live CDC file

• a pinned Dockerfile, so the figures reproduce exactly

To try it:

git clone https://github.com/Adeniyikayodee/dependency_manifest.git
cd dependency_manifest
python3 lint_lineage.py
python3 lint_lineage.py --graph
python3 reproduce_svi.py

The full tool on the complete manifest: 60 products, 75 derivation edges, and eight audits that come back as 5 FAIL, 1 REVIEW, 1 PASS, and 1 UNTRACED.

The manifest is clear about its limits. It covers 60 products out of an estimated 400 or more official composite indices worldwide, and four of its edges are still marked inferred. The first audit of the file found six errors, and four of them sat in edges I had already labelled certain or documented.

The people best placed to write this kind of record are the agencies themselves, since they already describe their methods in PDF form. Two new fields on a public data schema, derivesFrom and measurementBasis, would give every producer a place to store what they already know.

If you work with public data, you can help by adding products you know well, or by checking the edges marked inferred against their source documents.

Conclusion

Target leakage in public data hides inside the recipes that agencies use to build their indices. A model can score close to perfect by rediscovering one of those recipes, and that score says very little about the real world.

In this tutorial, you:

• rebuilt CDC's Theme 1 at R² = 0.998 from its own five input columns

• separated provenance (where a number arrived from) from derivation (what it was calculated from)

• wrote a YAML manifest that records derivation edges with a relation and a confidence level

• built a linter that uses breadth-first search to find ancestors and depth-first search to list every route

• made the linter fail loudly on typos, duplicate YAML keys, cycles, and empty covariate lists

• wired the check into GitHub Actions with clear exit codes

Before you trust a high score on public data, ask yourself what your target was built from. Once you write the answer down, a few lines of Python can check it every time you train a model.

The code from this tutorial lives in derives-from-tutorial, and you can find the full tool, the manifest, and the reproduction script in the main dependency_manifest repository. The project is archived on Zenodo with the DOI 10.5281/zenodo.22274757, and you are free to use it under the CC0 licence.

Sources

• CDC/ATSDR Social Vulnerability Index: https://svi.cdc.gov/

• FEMA National Risk Index Technical Documentation v1.20, December 2025: https://www.fema.gov/sites/default/files/documents/fema_national-risk-index_technical-documentation.pdf

• Census Bureau Community Resilience Estimates: https://www.census.gov/programs-surveys/community-resilience-estimates.html

• CDC PLACES methodology, Preventing Chronic Disease, 2022: https://www.cdc.gov/pcd/issues/2022/21_0459.htm

• Data Commons data model: https://docs.datacommons.org/data_model.html

Then you read the diff, and you find that it pulled in a validation library that's not in your package.json, it wrote the test in Jest even though your team moved to the Node test runner last spring, and it reached into the database from inside the route handler because it had no way of knowing that every other handler in the codebase delegates to a service.

The code runs, the tests it wrote pass, but you still have to rewrite most of it.

None of that is a reasoning failure on the model's part. It produced a sensible solution to the problem as it understood it, but it understood the problem badly because nobody told it how this particular codebase works.

Your conventions live in your team's heads, in code review comments, and in decisions made eighteen months ago that nobody wrote down. The agent can't see any of that, so it falls back on the average of every repository it has ever been trained on, which is exactly what you got.

The fix isn't a longer prompt, since you would have to retype it every session and your teammates would each write a different version of it. The fix is a set of files that live in the repository, load automatically, and are maintained the same way you maintain code.

This tutorial shows you how to structure those files, how to keep one source of truth across the four or five formats the different tools expect, and, most importantly, how to stop them from quietly going out of date. After all, a context file that describes a codebase you deleted six months ago is worse than no context file at all.

Everything here is built on a companion repository you can clone and run: github.com/Adeniyikayodee/MCF. It has no dependencies, so Node 20 or newer is all you need.

Table of Contents

• What You Need Before You Start

• Why the Context Window is the Real Constraint

• The Three Layers

• Picking a Format Without Maintaining Four Copies

• Writing the Root File

• Scoping Rules to a Directory

• Pointing Instead of Inlining

• Making Context Files Verifiable

• Give the Agent Something to Verify Against

• Checking Whether it Actually Worked

• Keeping the Files Healthy

• Mistakes Worth Avoiding

• Where to Start

What You Need Before You Start

You should be comfortable with Git and a terminal, you should have Node 20 or newer installed, and you should have used at least one coding agent such as Claude Code, Cursor, GitHub Copilot, or Codex on a real project.

You don't need to know anything about how models work internally, since everything in this tutorial is about files on disk.

Why the Context Window is the Real Constraint

Everything an agent knows while it works on your task lives in one buffer called the context window. That buffer holds the system prompt, your conversation, every file the agent opened, every command it ran, and every stack trace those commands printed.

But it's important to know that it's finite, and it fills up faster than most people expect. A single debugging session, for example, can burn tens of thousands of tokens before the agent has written a line of code.

The part that matters for this tutorial is what happens as that buffer fills. Anthropic's engineering team describes an effect they call context rot, where a model's ability to retrieve a specific instruction degrades as the token count climbs. The model isn't ignoring you out of stubbornness, it's working with an attention budget that gets thinner as more material competes for it.

That single fact overturns the intuition most people bring to context files. Writing more feels safer, because you've covered more cases and left less to chance. But every line you add competes with every other line for a finite amount of attention.

The Claude Code documentation puts the consequence plainly, noting that a bloated instructions file causes the agent to ignore the rules inside it. Also, it notes that the symptom of an over-long file is the agent repeatedly breaking a rule you've clearly written down.

Here's roughly how a session budget gets spent on a real task:

system prompt and tool definitions        ~12,000 tokens
context files loaded at startup            ~4,800 tokens
three source files the agent opened        ~9,000 tokens
one test run with a stack trace            ~3,500 tokens

The 4,800 token context file in that list is competing with the stack trace the agent needs to read in order to fix the bug. A 600 token file that names the right paths would leave room for the agent to go and read the code itself, which it's very good at.

Context files are a budget allocation problem before they're a documentation problem, and almost every improvement in this tutorial comes from taking that seriously.

The Three Layers

The structure that works treats context as three distinct layers with different costs.

The always loaded layer is a single file at the root of your repository that the agent reads at the start of every session, whether the task is a typo fix or a migration. You pay for this file on every single request, so it holds only what applies to every task in the repository. It also stays small enough that you could read it aloud in under a minute.

The scoped layer is made up of nested files that load only when the agent works inside a particular directory. Rules about your API layer sit in src/AGENTS.md, so a task that only touches the frontend never pays for them.

The on demand layer is ordinary documentation that the root file points at by path rather than inlining. A path costs a handful of tokens while the document behind it might cost two thousand, so the agent spends that budget only when the task actually calls for it.

This mirrors how a new engineer works, since they don't memorise your architecture document on day one. They remember that it exists and go and read it when they need it.

The finished layout in the companion repository looks like this:

MCF/
├── AGENTS.md                          always loaded, budgeted
├── CLAUDE.md                          generated from AGENTS.md
├── .github/copilot-instructions.md    generated from AGENTS.md
├── .cursor/rules/testing.mdc          glob scoped, hand written
├── .claude/
│   ├── settings.json                  hook that runs the context linter
│   └── skills/add-endpoint/SKILL.md   workflow, loaded on demand
├── docs/
│   ├── architecture.md
│   ├── testing.md
│   └── decisions/0001-in-memory-store.md
├── scripts/
│   ├── context-lint.mjs
│   └── sync-context.mjs
├── src/
│   ├── AGENTS.md                      scoped to the source tree
│   ├── api/tasks.js
│   ├── services/tasks.js
│   ├── lib/validate.js
│   ├── router.js
│   └── server.js
└── tests/

Picking a Format Without Maintaining Four Copies

Every vendor picked a different filename for the same idea, which is annoying but manageable once you decide which one is the source of truth.

AGENTS.md is the closest thing to a shared convention. It's plain Markdown with no required schema, its governance sits with the Agentic AI Foundation under the Linux Foundation, and it's read natively by Claude Code, Codex, Cursor, Copilot, Gemini CLI, Aider, Windsurf, Zed, and a long list of others.

Nested files are part of the spec, the file closest to the code being edited takes precedence, and anything you type directly into the chat overrides all of it.

The tool-specific formats still exist alongside it. Claude Code reads CLAUDE.md, walks up the directory tree concatenating every one it finds, and resolves @path/to/file imports. Cursor uses .mdc files inside .cursor/rules/ with YAML frontmatter that can scope a rule to a glob such as tests/**/*.js, which makes it the most expressive of the formats and also the least portable, since nothing outside Cursor reads it. GitHub Copilot, for its part, reads a single .github/copilot-instructions.md at the repository root.

The practical answer is to write AGENTS.md once, generate the rest from it, and hand write a separate file only when a tool offers something the shared format can't express. In practice, this means Cursor's glob scoping. You can do the generating with symlinks:

ln -s AGENTS.md CLAUDE.md

Symlinks are the shortest path, though they cause trouble for contributors on Windows and for some CI checkout configurations, so the companion repository uses a small script instead. The script writes a banner into every file it generates, which stops a well-meaning teammate from editing the copy and losing their work on the next sync:

// scripts/sync-context.mjs
const banner = `<!-- Generated from ${SOURCE} by \`npm run sync:context\`. Edit ${SOURCE} instead. -->`;

export const targets = [
  // Claude Code resolves @path imports, so its file stays a pointer plus what is specific to it.
  { path: 'CLAUDE.md', render: () => `${banner}\n\n@${SOURCE}\n\n${CLAUDE_EXTRAS}` },
  // Copilot has no import syntax, so the source is inlined.
  { path: '.github/copilot-instructions.md', render: (source) => `${banner}\n\n${source}` },
];

Because Claude Code resolves imports, its generated file stays a pointer plus the handful of instructions that only make sense for that tool, which keeps it at around 130 tokens rather than duplicating the whole thing:

<!-- Generated from AGENTS.md by `npm run sync:context`. Edit AGENTS.md instead. -->

@AGENTS.md

## Claude Code specific

- Use plan mode for any change that touches more than three files, and skip it for a one line fix.
- Delegate codebase exploration to a subagent so the findings come back summarised rather than as
  a hundred file reads in the main context.

Running the script regenerates both files, and running it again does nothing. This is what you want from something a hook or a CI job will call repeatedly:

Writing the Root File

This is where most of the value is, and it's also where most people go wrong, because the instinct is to write everything down.

Use one editing test on every line you're tempted to add: would removing this line cause the agent to make a mistake? If the answer is no, the line is costing you attention budget and buying you nothing, so cut it. Applied honestly, that test removes most of what people put in these files.

Here's the kind of file the test is designed to catch:

# AGENTS.md

## About this project
This project is a REST API for managing tasks. It was originally built in 2023 by the platform
team and has since been maintained by the core services group. The codebase is written in modern
JavaScript using ES modules.

## Code style
- Use meaningful variable names
- Write clean, maintainable code
- Follow the DRY principle
- Use const instead of var
- Add comments where the code is complex

## Structure
- `src/server.js` contains the server
- `src/router.js` contains the router
- `src/api/tasks.js` contains the task handlers
- `src/services/tasks.js` contains the task service

Every line there fails the test. The model already knows what const is for, it can see that the file called router.js contains the router, and knowing which team owned the code in 2023 won't change a single decision it makes.

Meanwhile the one thing an agent genuinely can't work out on its own, which is that this project deliberately has no dependencies, is nowhere in the file.

This is the version that ships in the companion repository:

# AGENTS.md

Task API used as the worked example for a tutorial on managing context files. This file is the
single source of truth for agent instructions, and `CLAUDE.md` plus
`.github/copilot-instructions.md` are generated from it by `npm run sync:context`, so edit this
file and never the generated ones.

## Commands

- Install: nothing to install, the project has zero dependencies
- Run the tests: `npm test`
- Start the server on port 3000: `npm start`
- Check the context files: `npm run lint:context`
- Regenerate the tool specific context files: `npm run sync:context`

## Conventions that are not obvious from the code

- The test runner is the Node built in runner invoked through `node --test`, so do not add Jest,
  Vitest, or any other test dependency to this repository.
- This project stays dependency-free on purpose, so solve problems with the Node standard library
  rather than by adding a package.
- Handlers in `src/api/` return `{ data }` or `{ error: { code, message } }` and never choose an
  HTTP status, because `src/router.js` owns the mapping from error code to status.
- Handlers never touch the store directly, so any logic that reads or writes tasks belongs in
  `src/services/tasks.js`.
- The store is module level state that survives between test cases, so any test file that creates
  a task has to call `resetTasks()` in a `beforeEach` hook.

## Definition of done

Run `npm test` and `npm run lint:context` before you report a task as finished, and paste the
output rather than asserting that it passed.

## Where to look

- Architecture and request flow: `docs/architecture.md`
- Testing conventions and how to add a case: `docs/testing.md`
- Why the store is in memory: `docs/decisions/0001-in-memory-store.md`
- Rules that apply only to the API layer: `src/AGENTS.md`

Notice what each section is doing. The commands are there because an agent can't reliably guess your script names, and guessing wrong costs a failed run. The conventions are all things that are either invisible from reading the code or actively contradicted by what the model would otherwise assume, and each one states the reason, since a rule with a reason attached survives situations the rule author didn't anticipate. The last section is nothing but paths, which is the on demand layer doing its job.

Rough guidance on what earns its place:

Getting the Altitude Right

There's a second way to write a bad rule, which is to pitch it at the wrong level of specificity. Anthropic's guidance frames this as finding the right altitude, sitting between hardcoded logic that shatters on the first case it didn't anticipate, and vague encouragement that gives the model nothing to act on.

Too rigid, and it breaks on the first handler that does not fit:
- Every route handler must be exactly 40 lines and call validate() on line 3.

Too vague, and it changes nothing about what the agent does:
- Write clean, maintainable code.

Right altitude:
- Route handlers parse and validate input, then delegate to a function in `src/services/`.
  Handlers do not touch the store directly. See `src/api/tasks.js` for the pattern to copy.

The third version tells the agent the shape of the rule, the boundary it must not cross, and where to find a worked example, which is roughly what you would tell a competent new hire on their first day.

Scoping Rules to a Directory

Anything that only matters inside one part of the tree belongs in a nested file, and the test for whether a rule qualifies is simple: would a developer working in a different directory ever need to know this? If not, move it down.

<!-- src/AGENTS.md -->
# Source layer

Rules below apply to everything under `src/`, and they sit on top of the root `AGENTS.md` rather
than replacing it.

## Adding an endpoint

1. Add the handler to `src/api/tasks.js` following the shape the neighbouring handlers use.
2. Add one entry to the `routes` array in `src/router.js` with its success status.
3. Add a case to `tests/api.test.js` that covers the success path and the failure path.

## Validation

Validators live in `src/lib/validate.js`, they return an array of problem strings rather than
throwing, and they report every failing field instead of stopping at the first one, so a caller can
show the user all of their mistakes at once.

That validation rule is a good example of something worth writing down, because the code alone doesn't explain itself. An agent reading src/lib/validate.js sees a function returning an array and has no way to know whether that's a deliberate convention or an accident of one implementation, so it might reasonably throw an exception in the next validator it writes:

// src/lib/validate.js
export function validateTaskInput(input) {
  if (typeof input !== 'object' || input === null || Array.isArray(input)) {
    return ['body must be a JSON object'];
  }

  const problems = [];

  if (typeof input.title !== 'string' || input.title.trim() === '') {
    problems.push('title is required and must be a non-empty string');
  } else if (input.title.length > TITLE_MAX) {
    problems.push(`title must be ${TITLE_MAX} characters or fewer`);
  }

  if (input.done !== undefined && typeof input.done !== 'boolean') {
    problems.push('done must be a boolean when present');
  }

  return problems;
}

Pointing Instead of Inlining

The Where to look section of the root file is the cheapest thing in this whole setup. Four lines of paths cost almost nothing to load, and behind them sit several thousand tokens of architecture notes, testing conventions, and decision records that the agent pulls in only when a task needs them.

Architecture decision records are the natural home for the reasoning that would otherwise bloat your root file. The companion repository has one explaining why the task store is a plain Map rather than a database, and its most useful paragraph is the last one:

An agent working here should not add a database, an ORM, or a persistence layer unless the task
explicitly asks for one, and should treat the missing persistence as a deliberate choice rather than
a gap to fill.

Without that, an agent asked to "make the API production ready" will helpfully add Postgres. With it, the agent knows the absence is intentional and asks before changing it. That sentence costs you nothing until the day it saves you an afternoon.

The same logic applies to workflows that only come up occasionally. A step by step procedure for adding an endpoint is genuinely useful, and it would be dead weight in a file loaded on every task, so it lives in a skill file that loads when someone actually asks for an endpoint:

---
name: add-endpoint
description: Add a new endpoint to the task API following the layering this repository uses
---

# Add an endpoint

This workflow loads only when someone asks for a new endpoint, which is why it lives here instead
of in `AGENTS.md` where every session would pay for it.

Read `docs/architecture.md` first if you have not already, then work through these steps in order.

1. Decide which layer owns the new behaviour. Anything that reads or writes tasks belongs in
   `src/services/tasks.js`, and anything about request shape belongs in `src/api/tasks.js`.
2. Add or extend a validator in `src/lib/validate.js` if the endpoint accepts input, returning an
   array of problem strings so the handler can report every failure at once.
3. Add the handler to `src/api/tasks.js`, returning `{ data }` on success and
   `{ error: { code, message } }` on failure, and using an existing error code where one fits.
4. Register the route in the `routes` array in `src/router.js` with the success status it should
   return, and add the error code to `STATUS_BY_ERROR_CODE` if you introduced a new one.
5. Add at least one success case and one failure case to `tests/api.test.js`.
6. Run `npm test` and `npm run lint:context`, then paste both outputs into your summary.

Do not add a dependency, do not introduce a persistence layer, and do not set a status code inside
a handler.

Making Context Files Verifiable

Everything so far is fairly standard advice, and on its own it has a short shelf life. Context files rot for exactly the same reason documentation rots, which is that nothing breaks when they're wrong. You rename src/services/task.js to src/services/tasks.js, and your context file keeps confidently pointing at a path that no longer exists. You delete the typecheck script, and six months later an agent burns two turns trying to run it. Nobody notices either of those, because nothing in your pipeline is checking.

So put a check in the pipeline and let it fail. The companion repository has a linter in scripts/context-lint.mjs that runs four checks, and it's about 150 lines of dependency-free JavaScript that you can adapt to your own repository in an afternoon.

The first check is a token budget on every file that loads at startup:

// Loaded at the start of every session whether the task needs them or not. When one of these keeps
// pushing against its ceiling, move the detail into docs/ and leave a path behind.
const ALWAYS_LOADED = [
  { path: 'AGENTS.md', budget: 800 },
  { path: 'CLAUDE.md', budget: 300 },
  { path: '.github/copilot-instructions.md', budget: 900 },
  { path: 'src/AGENTS.md', budget: 400 },
];

// Rough average for English prose. Precision is not the point, catching a file that doubled is.
const CHARS_PER_TOKEN = 4;

const estimateTokens = (text) => Math.ceil(text.length / CHARS_PER_TOKEN);

Four characters per token is an approximation rather than a real tokenizer count, and it runs a little optimistic on code heavy files. This is fine because the number you care about is the ceiling. A file creeping from 400 tokens to 800 is the signal, and being off by 8% on the absolute figure changes nothing about how you respond to it.

The second and third checks read your context files as prose and verify that the things they mention are real. Anything in single backticks that looks like a path has to exist on disk, and any npm script has to exist in package.json:

// Fenced blocks are stripped first so an example inside a snippet is never read as a real reference.
function inlineCodeSpans(text) {
  const prose = text.replace(/```[\s\S]*?```/g, '');
  return [...prose.matchAll(/`([^`\n]+)`/g)].map((match) => match[1].trim());
}

for (const span of spans) {
  if (looksLikePath(span)) {
    if (!existsSync(join(ROOT, span))) {
      problems.push(`${file} points at a path that does not exist: ${span}`);
    }
    continue;
  }

  const script = span.match(/^npm run ([\w:-]+)$/) ?? span.match(/^npm (test|start)$/);

  if (script && !scripts.includes(script[1])) {
    problems.push(`${file} mentions an npm script that is not in package.json: ${span}`);
  }
}

Stripping fenced code blocks before scanning matters more than it looks, since your documentation is full of illustrative examples that were never meant to be real references, and a linter that fails on those gets switched off within a week.

The fourth check reruns the sync script in a dry run mode and fails if any generated file no longer matches AGENTS.md, which catches the teammate who edited CLAUDE.md directly despite the banner.

On a healthy repository the whole thing takes well under a second:

The interesting output is what happens when something rots. Adding one plausible looking line to AGENTS.md that mentions a script that was deleted and a file that was renamed produces this:

The script exits with a non-zero status, so wiring it into CI takes four lines and means the files can't drift quietly:

# .github/workflows/ci.yml
      - name: Run the test suite
        run: npm test

      # The context files are checked on every pull request, which is what stops them from
      # drifting away from the code they describe.
      - name: Check the context files
        run: npm run lint:context

This is the part I would keep if I had to throw away everything else in this tutorial. A mediocre context file that's verifiably true beats a beautifully written one that describes last year's architecture, because the agent has no way to tell the difference and will act on both with equal confidence.

Give the Agent Something to Verify Against

There's one more line in that root file worth dwelling on, and it's the definition of done.

An agent stops when the work looks finished, and without a check it can run for itself, "looks finished" is the only signal available to it, which quietly makes you the verification loop. Every mistake then waits for you to notice it.

Naming a command that returns a pass or a fail converts that into something the agent can act on by itself, so it writes the code, runs the check, reads the result, and keeps going until the check passes.

That's why Run npm test and npm run lint:context before you report a task as finished does more for output quality than any amount of style guidance you could write. Asking the agent to paste the output rather than assert success matters too, since reviewing evidence takes you a few seconds and re-running the verification yourself takes minutes.

Instructions in a context file are advice, though, and advice gets lost as the context fills. When something must happen every single time without exception, use a hook, which runs a script at a fixed point in the agent's loop and can't be talked out of it:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "npm run lint:context --silent"
          }
        ]
      }
    ]
  }
}

The rule of thumb is that anything advisory belongs in prose, and anything mandatory belongs in a hook or in CI.

Checking Whether it Actually Worked

You shouldn't take any of this on faith, and there's a cheap way to test it on your own repository.

Pick a task with an obviously correct shape, write the prompt down so it stays identical across runs, and run it twice: once on your current branch, and once on a branch where you have deleted the context files. In the companion repository a good candidate is "add a GET /tasks/count endpoint that returns the number of open tasks, with tests".

Then compare the two runs on four points. Did the tests pass without you intervening? How many corrections did you have to make? Did the code follow the existing layering, or did it reach into the store from the handler? Did any new dependency appear?

This is a sample of one rather than a benchmark, and you should treat it as such. But it's enough to tell you whether your files are pulling their weight, and it makes it very obvious which specific rule was missing when something goes wrong.

Keeping the Files Healthy

Treat these files the way you treat code, which means reviewing them when something breaks rather than on a schedule.

Two diagnostics will cover most of the situations you run into. If the agent keeps violating a rule that's written down, the file is almost certainly too long and the rule is getting lost in the noise. Prune aggressively rather than adding emphasis.

If the agent asks you a question that the file already answers, the wording is ambiguous, so rewrite that line rather than adding a second one next to it.

Beyond that, delete any rule the agent already follows without being told, since the model's defaults improve with every release and a rule that was necessary last year may be dead weight now.

Watch the token budget in the linter output as a rough health metric, because a file that keeps creeping toward its ceiling is telling you that detail needs to move into docs/.

Mistakes Worth Avoiding

The most common failure is the kitchen sink file, where every convention anyone ever mentioned gets appended until the file is three thousand tokens and the agent follows roughly half of it. The fix is the removal test applied without sentiment.

The second is duplicating your README into your context file, which doubles the cost of every session while adding nothing, since the two documents have different audiences and the agent can read the README when it needs to.

The third is documenting things the model can see for itself. The giveaway is any line that describes what a file contains rather than what you expect an agent to do about it.

The fourth is writing rules that can't be verified, such as asking for readable code or good performance, which sound reasonable and give the agent no way to tell whether it has complied.

The fifth, and the one that gets teams eventually, is letting each tool keep its own hand-maintained copy. They start out identical, they diverge within a month, and then Cursor and Claude Code are working from contradictory instructions in the same repository. Generate the copies, and check the generation in CI.

Where to Start

If you only do one thing after reading this, run a token estimate on the context file you already have, and then read it line by line asking whether removing each line would cause a mistake. Most people cut somewhere between a third and a half of the file on the first pass, and notice the agent following the remainder more reliably.

After that, add the pointers so your documentation becomes reachable without being expensive, and put the linter in CI so the whole thing stays honest as the codebase moves underneath it.

The full setup, including the linter, the sync script, the hook, and the CI workflow, is at github.com/Adeniyikayodee/MCF. Clone it, run npm run lint:context to watch it pass, then break something in AGENTS.md and watch it fail.

You can adapt the linter to your own conventions rather than copying it verbatim, since the checks worth running are the ones that match the ways your particular repository tends to drift.

Fork the repository if you want your own copy to experiment in, since a fork gives you a branch point you can modify freely without losing the ability to pull later changes back in. If you would rather be told when those changes land, use the Watch button next to Fork and choose releases or all activity, because that's the control that actually sends you notifications while a fork only captures the code as it stands on the day you take it.

Further Reading

• Effective context engineering for AI agents, Anthropic

• Best practices for Claude Code, Anthropic

• The AGENTS.md convention

Imagine a feature that promises to enhance user experience but that ends up resonating with only a small subset of users. This scenario underscores the importance of precision in feature rollouts.

Fortunately, feature flagging management tools like Flagsmith can help with granular segmentation. This process helps your team make sure that new features are introduced to the most relevant audiences. Granular Segmentation makes it easier to understand your user base, leading to higher engagement and satisfaction.

In this article, we will be focusing on the concept of granular user segmentation and its significance in enhancing feature rollouts. We’ll also explore some best practices, pitfalls to avoid, and will look at how Flagsmith facilitates granular segmentation with feature flags.

Here’s what we’ll cover:

1. What is Flagsmith?

2. What is Granular Segmentation?

3. How Feature Flags Enable Granular Segmentation?

4. How to Implement Granular Segmentation in Flagsmith

5. Benefits of Granular Segmentation for User Engagement

6. Conclusion

What is Flagsmith?

Flagsmith is an open-source feature management platform that helps teams control feature rollouts with precision. It helps developers toggle features on or off for specific users, environments, or groups without redeploying code.

Ideal for A/B testing, phased rollouts, and remote configuration, Flagsmith ensures real-time adjustments and seamless feature delivery. With flexible deployment options – hosted, private cloud, or on-premises – it adapts to the needs of organizations of all sizes.

The Importance of Granular Segmentation

What is Granular Segmentation?

Granular segmentation is a process in which the user base is divided into groups based on unique attributes such as behavior demographics or engagement levels. These groups can be identified as segments of users of a platform, and each segment is based on several traits that help teams tailor feature rollouts to meet the needs of each segment.

This granular level of control over rollouts helps product teams release features that resonate with their user base. This creates a more personalized experience for the end user that can improve the effectiveness of the feature.

Now, let’s discuss what kind of an impact feature rollouts might have.

Impact of Feature Rollouts

The advantages of granular segmentation in feature rollouts include:

• Targeted relevance: Features are delivered to users who will benefit most from them, making the updates more relevant and useful. This targeted approach increases the likelihood of user engagement.

• Optimized user experience: Because of this targeted approach, businesses can prevent rollouts of features that overwhelm their users in any way. This means that users would receive updates according to their interests, leading to a better user experience.

• Higher adoption rates: All this would also lead to higher adoption rates. An increased adoption rate is a sign of good engagement from a business’s users as well as of business growth

• Less risk when rolling out new features: Segmenting your user base and releasing new features to, say, a select 10% of users reduces risk. Teams can see how features do with those users and adjust accordingly before rolling out to the next segment. Or they can roll back if the impact is negative, which helps them avoid incidents like the latest high-publicity one we saw with CrowdStrike.

To put things into perspective, let’s discuss an example of an e-commerce store.

The MART Example

The MART is an online store that sells various products. They want to introduce an AI-powered recommendation engine, but only to a subset of their user base that shows less engagement on the platform buying products. AI-powered recommendation engines would target this user segment to generate more sales from the platform and increase business growth.

Here we see the concept of segmentation in practice where a feature is dedicatedly exposed to a user group's explicit attributes, thus leading to increased relevance and user satisfaction.

If the feature proves to be successful with the targeted segment, the next phase would be to expand its availability to other user groups.

How Feature Flags Enable Granular Segmentation

You can integrate Flagsmith into your development workflow by using SDKs. The user segmentation adds a layer of granular control to the product teams with over-feature releases. This control helps product teams to minimize the risk of degradation of a new feature. They can leverage the GUI to interact with Flagsmith and roll out/roll back features according to their needs.

What are Segments in Feature Rollouts?

A segment is a subset of identities, defined by a set of rules based on traits associated with identities. So a single identity can be a part of many segments and is associated with an environment, such as staging or production.

You might be wondering – how can product teams use segments in their feature rollouts?

You can use segments to create ‘overrides’ on any number of features in your application. This allows you to control the state and/or value of a feature for a selection of your users, as defined by the segment.

Now that you understand segments, let’s discuss what key features allow you to use detailed user segmentation.

• User attributes: Flagsmith allows you to define and manage user attributes, such as location, behavior, subscription levels, or platform activity. These are attributes you can use to create highly specific user segments.

• Segment definitions: You can create custom segment definitions from these user attributes. For instance, you can define a segment for users who have been highly active on the platform since last month or users who live in a different region than most of your user base. This granularity ensures that you can target features to the most relevant user groups.

• Dynamic targeting: Dynamic targeting can help you adjust feature rollouts on the basis of user attributes. This means that you can progressively roll out features to segments of users, monitor their performance, and make adjustments to the feature accordingly.

Flexibility and Control

Flexibility and control are a rare combination when it comes to such tools, but with Flagsmith you get the best of both worlds. User segments and feature management ensure you have precision and control over your feature rollouts:

• Granular control: Multiple segment creation and control access are available in Flagsmith with a variety of criteria, allowing feature rollouts that cater to specific user needs.

• Analytics and feedback: Analytics and feedback are an integral part of the feature testing loop. They provide tracking of how different segments interact with new features. It’s invaluable for understanding the user’s behavior on the platform which helps you make informed decisions for further rollouts.

So now you’ve learned what segments are, what you can do with them, and how segments help in granular control over rollouts. Now, let’s move on and see how you can implement segmentation using Flagsmith.

How to Implement Granular Segmentation in Flagsmith

Set Up Flagsmith in Your Project

You can integrate Flagsmith into your application using the available SDKs for the language of your choice. For example, to integrate the SDK in Node.js, you’ll first need to install the npm package as follows:

npm i flagsmith-nodejs --save

After installing the package, you will use the following code to initialize Flagsmith in your project:

const Flagsmith = require('flagsmith-nodejs');
const flagsmith = new Flagsmith({ environmentKey: 'FLAGSMITH_SERVER_SIDE_ENVIRONMENT_KEY',});

Once it’s integrated, configure your Flagsmith instance by creating a new project. We’ll go through this below.

How to Create Identities and Define User Traits and Segments

Now you’ll need to create the identities and traits you want to use for segmentation. These could include user profile information, behavior metrics, or any other relevant data.

So, let’s create a user named John Doe.

Now click on the created user and define a trait country.

In Flagsmith, creating a trait country involves defining a user attribute that specifies their geographic location. Traits are key-value pairs assigned to user identities, allowing for precise segmentation. For example, you can define the "country" trait with values like "USA," "Canada," or "Germany."

This enables product teams to create segments based on location and target feature rollouts accordingly. For instance, a feature can be activated only for users with the "country" trait set to "USA," facilitating controlled and region-specific rollouts.

Next, you’ll create some segments. You’ll use the Flagsmith dashboard to create custom segments based on these attributes. For example, you can create a segment for users who are from the USA. Define your segment, for example (western_users ), as below:

How to Create and Manage Feature Flags

Create a feature flag called ai_recommendation_engine in Flagsmith for the features you want to roll out. Each flag represents a specific feature or configuration option that can be toggled on or off.

Next, assign your feature flags to the segment you created. For instance, if you have a recommendation engine, you can target it specifically to users that match the segment created in the previous step. Use the Flagsmith dashboard to set these targeting rules and manage feature flag settings.

How to Target Segments for Rollouts

After configuring Flagsmith and setting up your segments and traits, you can start rolling out features to your defined segments.

First, you’ll want to do gradual rollouts. Using the percentage split operator, you can initially release the feature to a small percentage of users within the segment. Based on performance and feedback, you can gradually expand the rollout to a larger portion of the segment or additional segments, ensuring a controlled and data-driven approach.

Second, monitoring is a crucial part of feature rollouts and Flagsmith can help you with its analytics tools. You can track the performance of your feature flags and user segments, monitor how different segments interact with the new features, and make adjustments as needed.

For example, you might decide to increase the rollout percentage or adjust segment definitions based on user feedback.

Some best practices:

• Start small: To test out segmentation, it’s a good idea to start small and create well-defined segments to test new features. This will help you gather valuable feedback and will prevent you from being overwhelmed in case of degraded performance or a rollback scenario.

• Use data: Analytical tools are a great help in gathering data on how different segments interact with your features. You can use this data to refine your targeting and improve the user experience.

• Iterate: You’ll likely make better decisions after several iterations. So remember that you should iterate your segmentation and rollouts based on metrics and user feedback.

Some common pitfalls:

• Overlapping segments: Distinction between segmentations is the key to avoiding conflicts between feature targeting. Always be careful while defining segments for your user groups.

• Ignoring feedback: The greatest mistake a product team can make is to overlook early user feedback. Early feedback is crucial for identifying issues and making informed decisions about a feature rollout.

By following these steps and best practices, you can effectively use this granular segmentation approach, ensuring that your feature rollouts are targeted, relevant, and successful.

Benefits of Granular Segmentation for User Engagement

Improved User Satisfaction

Granular segmentation helps your users out, as it gives them specifically personalized features according to their needs and inclinations. You can build more personalized experiences by aiming certain features at particular users that match their behavior or interest.

For example, a fitness app might launch an update that contains a workout feature for those users who have shown interest in strength building, rather than for all users. This targeted approach ensures that users receive updates related to and suitable to them, which leads to a positive experience, increased satisfaction, and better recognition of your product.

Increased Engagement

When users get features or updates that are targeted toward their specific needs, it’s more likely that they’ll engage with that feature. Granular segmentation helps maximize engagement by providing users with upgrades that are pertinent to their interests and usage patterns.

For example, an e-commerce platform could propose a new recommendation system and try it out on users who recurrently browse specific categories. This relevant targeting will likely increase the probability that those users will respond to those recommendations, leading to increased engagement and potentially higher conversions.

Enhanced Feature Adoption

Targeting specific segments of users with features that address their needs should lead to higher adoption rates. Presenting new features to users who are very likely to benefit from them, you increase the probability of these features being adopted and utilized.

For example, a software company introducing a new improved analytics tool would likely target power users who consistently use analytics features. After those users provide positive feedback and adopt the tool, it can be deployed on other segments. Then the team can be confident that the feature is approved and effective.

Data-Driven Insights

Granular segmentation offers valuable insights into how various user groups engage with new features. Analyzing this data can provide you insights into the behavior and inclinations your users as well as the overall impact of your features.

For example, you might realize that users are responsive to new features in a specific segment compared to other segments. Such information helps you refine your feature strategy, making rational decisions regarding future launches, and enhancing user engagement across different segments.

Optimized Resource Allocation

Centering on targeted segments lets you allocate resources more effectively. instead of investing in a broad, one-size-fits-all approach, you can direct your initiative towards segments that are likely to benefit from and engage with new features. This optimized allocation assures that your resources are utilized efficiently, leading to positive outcomes and a higher return on investment.

By leveraging granular segmentation, you can enhance user engagement, improve feature adoption, and gain valuable insights, all of which contribute to a more successful and user-centric feature rollout strategy.

Conclusion

In this article, we discussed the power of granular user segmentation in driving successful feature rollouts, highlighting how it can improve user satisfaction, engagement, and adoption rates. We also explored how Flagsmith enables this approach, offering tools to manage and target features with precision.

By leveraging these strategies, you can ensure that your product updates are more relevant and impactful. If you're interested in optimizing your feature rollouts, consider exploring Flagsmith’s capabilities to start making data-driven decisions.

In an era where agile development is the go-to practice for quick feature releases and feedback, it's easy for security and compliance to get overlooked. This may be securing CI/CD pipelines, protecting user data, or managing access to feature flagging in production environments.

Though the tech industry is aware that strong security measures and compliance practices are essential in DevOps, sometimes these measures take a back seat behind the need to get code shipped to production.

Because of this, it's crucial that you understand the importance of security and compliance in DevOps. You should also learn how your team can leverage popular security concepts like Roll-Based Access Control (RBAC), which is what we'll focus on here.

This is useful not only for DevOps teams (as a compliant practice for assigning access to teams), but also as a full-fledged feature that SaaS platforms such as Flagsmith (an open source feature flagging platform) provide to their customers.

But you might ask – why is RBAC important? Well, we'll cover that and more in this tutorial.

Table of Contents:

1. Why is RBAC Important?
2. Feature Flagging and Flagsmith
3. Understanding Users, Groups, Roles, and Permissions
   – Create the project
   – Create the group
   – Create an Editor role
   – Assign permissions to the Editor role
   – Assign the Editor role to a group
   – Test the assigned permissions
4. Use Case: the Chaos at "NetGlobal Solutions"
   – How to mitigate the issue
5. How to Establish a Standard DevOps Process
6. Wrapping Up

Why is RBAC Important?

Keeping it simple, role-based access control is a fundamental requirement for DevOps. If you don't implement RBAC, your user's data is at a much greater risk of compromise. This can lead to financial losses and damage to your site's reputation.

So to reduce the attack surface and avoid damages, DevOps teams should leverage all security mechanisms at hand, such as RBAC. RBAC is pretty much what it sounds like: giving people on a team permissions based on the role they play within the organization.

This helps secure not only the lifecycle of a feature released in production but also how it is managed after the release – should it be enabled or disabled? Which users should be able to use it? And who will be responsible for performing this operation?

This is where feature flagging and the need for a platform that manages these security concerns come into play.

What is feature flagging anyway and why is RBAC important to manage it?

Feature Flagging and Flagsmith

Feature flagging is a software development concept that involves enabling or disabling a feature independent of redeploys or source code changes.

Feature flags are conditional statements in the application code that determine which section of code to execute on runtime based on a boolean value. They help deploy new features to production and provide granular control over their visibility to a user base or deployment environment.

You can configure feature flag values by various methods, such as config files, request headers, or from the database. This means that there's a certain amount of necessary developer intervention in this process. And anyone with access to the code or database can enable or disable a feature in production.

Although companies have compliance practices to handle feature flagging, there is always a chance of someone doing something that could harm production (intentionally or unintentionally).

So, to give you and your team better visibility and fine-grained control over feature toggling, feature flagging tools such as Flagsmith help you address such security and permission issues.

But you might be wondering – how does Flagsmith manage permissions and users? Enter RBAC, which is built into the Flagsmith toolset. This makes it easier for larger teams to collaborate across projects and environments, and for companies to manage access. Let's dive deeper to understand how it works.

Understanding Users, Groups, Roles, and Permissions

Flagsmith has two primary roles in the RBAC system: organization administrator and simple users.

Organization administrators enjoy the privilege of having superuser capabilities, whereas regular users require explicit permission to access the needed resources. Flagsmith also lets you control permissions for numerous users in a group. This makes access management easier for larger teams that are divided up based on responsibilities.

There are also certain roles in Flagsmith RBAC that can have a permission set attached to them. This enables people with these roles to access particular features of Flagsmith.

Roles come in handy especially when you need to assign bulk permissions to a group. In this case, you can create a role, assign permissions, and attach it to a group.

If we look at permissions from a macro level, we can see that Flagsmith has divided the permission set into three different levels: Organization, Project, and Environment.

For instance, we can manage permission sets for users to create projects at the organizational level. At the project level we can get access to environment creation, audit logs, and feature and segment management. Lastly, we can manage feature flags, segments, identities, and more on the environment level.

Before we move on to a case study, let's create an arbitrary project on Flagsmith and give permissions to a user so they can start creating feature flags for the project. We will perform the following actions:

• Create a project inside an organization
• Create a group “front_end_devs” and add users to this group
• Create an editor role
• Assign organization, project, and environment level permissions
• Assign the role to the newly created group
• Login in with the user account to test assigned permissions

Create the Project

Click on the Create Project button after logging in with an owner account. We'll name this project Dev Test.

Create the project

Create the Group

Navigate to the Members tab, click Create Group, then fill in the required details.

Enter the information for creating a group

We created a group “front_end_devs” and made John Smith the group admin. Consider this as an inline permission while creating a group. John can now manage the users inside this group.

Create an Editor Role

Click on Roles next to Groups, and create a role called “Editor”.

Creating an editor role

We created the role called Editor, so now let’s assign the appropriate permissions to this role. The Editor will be given permissions at all three levels – Organization, Project, and Environment.

Assign Permissions to the Editor Role

To assign permissions, click the name of the role you just created. You will see a sidebar on the right side with a permissions tab. We'll assign permissions for all levels, starting from the Organization level.

Assigning permissions starting at the Organization level

This Editor role can now create projects in this organization and manage the groups and their members in this organization.

Next, we give it Project level permissions. Click on the project name under the Project tab, and do as shown in the screenshot below:

Assigning project-level permissions

As you can see, we assigned this role two project-level permissions: View Project, and Create Feature. So any user or a set of users with this role will only be able to view a project and create a feature.

Now for the most granular permissions, which are at the environment level. Click on the Environment tab, choose the Dev Test project from the dropdown, and click on the Development Environment.

Handling Environment level permissions (the most granular)

As you can see, we assigned this role two environment-level permissions: View Environment, and Update Feature State. Any user or a set of users with this role will only be able to view an environment and update its feature state value in the project.

Assign the Editor Role to a Group

Now we will assign this role to the “front_end_devs” group. To do this, select the editor role, go to the Members tab, then click the text “Assigned Groups”. Enter the name of your group in the search bar, and select it.

Searching for groups to which to assign the Editor role

Test the Assigned Permissions

After going through the above steps, users in the “front_end_devs” group should only be able to perform the following operations

• Create a new project and manage its groups (Organizational level). Being the creator of a new project, the policies assigned to that user for another project will not be applicable here.
• Create and Delete Features in Dev Test Project
• Update feature state values in the Dev Test Project

Now we log in from John Smith’s account, a testing user from the “front_end_devs” group, to verify the assigned permissions are working properly.

First, we will check the organizational level permissions which lets us create a new project.

Checking organizational level permissions

You can see that the user was successfully able to create a new project. And since he is the creator of this project, John is the admin and has full authority to manage it.

Now we can test the permissions for the Dev Test project. Just click on the project name from the left-hand dropdown list and switch between projects. You will surely notice that in the left menu bar, you only have access to the development environment. This is due to the environment-level permissions we assigned to the Editor role.

To create the feature just click on the Create feature button, give the feature a name, and enter a value of your choice.

Creating a feature

You will see something like this after the feature creation:

_After creating the feature called johns_feature_

Now you should have a basic understanding of how Flagsmith’s RBAC handles permission assignments and manages users, groups, and roles.

To help you get a holistic view of things and understand how implementing Flagsmith for feature flagging can minimize chaotic incidents in production, let's consider a use case.

Use case: The Chaos at “NetGlobal Solutions”

Let's hypothesize that there's a company called NetGlobal Solutions, a global giant in the network industry. They provide various networking solutions to their customers, such as CDN, DNS management, geo-location, cloud cybersecurity, and DDoS mitigation.

They decided to introduce a new service, NetGlobal load balancing (a solution to manage huge amounts of web traffic) for their customers.

NetGlobal policy dictates that a feature should be tested for at least 3 months with only 10% of their customers before fully exposing it to the rest of the users. So they decided to use feature flagging to test it in production with 10% of their customers – let's say 10,000 considering the large size of their customer base.

The feature flag values are passed down to their code base from a central database table isolated from any relationships with other tables. The table has a boolean value that manages the visibility of the new feature, and Devin, the team lead for developing this feature, is responsible for its management and stability.

So, the time comes and Devin releases the feature in production. Two months go by and thousands of customers are using their load balancing service for their projects.

A dev from Devin’s team, while working on the prod database, accidentally changes the feature flag value in the table. Due to this mistake, the load balancing service instantly goes down and users start to face traffic loss on their sites.

The monitoring system triggers an alarm and the dev and DevOps teams spring into action. In about 10-15 minutes, they find the problem and resolve the issue. But because the user base is huge and the usage of the feature at the user end was quite technical, an impactful loss already happened.

How to Mitigate the Issue

Now, let's consider how Devin's team could've mitigated this incident if they'd been using Flagsmith to create feature flagging. We'll also look at how its RBAC would have helped to secure the flag value access.

• Flag value management: By using Flagsmith’s SDK in the application code, the flag values could have been managed and passed to the application with clear visibility.
• Audit control: By using Flagsmith’s audit log, the team could've had better accountability and transparency concerning changes made to feature flags.
• RBAC: it would have restricted access to unauthorized developers so that they couldn't change the feature flag values and provided granular control to the team lead, release managers, or DevOps engineers.

This hypothetical use case helps us get a basic understanding of the significance of a feature flagging tool for production releases. It also shows us why RBAC plays an important role in managing permissions and hierarchy in an ecosystem to help your team avoid downtime incidents.

The key takeaway here is that it's important to establish a standard DevOps process and choose DevOps tools that become a compliant part of feature release and management.

How to Establish a Standard DevOps Process

A standard DevOps process should be set in place for the lifecycle of a feature. It should address all the steps from the build stage to the production release.

Most importantly, in the pursuit of quick releases, your team shouldn't ignore the importance of securing this process as I mentioned at the beginning of this article.

A basic example of a standard DevOps process would start with establishing a strong Continuous Integration workflow with the following steps:

1. Build and Scan: Building artifacts and vulnerability scanning before pushing to artifact hubs.
2. Perform Unit, End-to-End, and Integration testing: Writing unit, end-to-end, and integration tests is paramount to testing the functionality of the application builds.

Then, you'd want to establish a solid Continuous Deployment workflow with the following steps:

1. Separation of environments: Separate dev, staging, and production environments for environment-specific testing.
2. Rollout method selection: Select rollout strategies depending on your needs, such as Rolling Updates, A/B Testing, Canary Deployments, and Feature Flagging.

Next, you should implement a robust monitoring mechanism for applications by leveraging monitoring systems such as Prometheus for monitoring, Grafana for visualization, and Grafana On Call for incident/on-call management tool.

And finally, after creating the above mechanism, the last step would be to use the provided RBAC systems in place. You'd start from cloud platforms and move on to DevOps tools being used implement the concept of least privilege on all levels and add them as a part of DevOps compliance practices.

Wrapping up

In this article, we discussed the importance of RBAC in the world of DevOps and how teams can leverage it in the industry to secure production environments.

We also discussed what feature flagging is, its importance for feature releases, and how it leverages RBAC to manage user permissions.

For a better understanding, we discussed a use case in which we saw how implementing Flagsmith could have saved a downtime incident, and how a DevOps compliance process could give strength to feature releases in production.

For AWS cloud development, the built-in choice for infrastructure as code is AWS CloudFormation.

Using IaC, developers can manage a project’s infrastructure efficiently, allowing them to easily configure and maintain changes within a project’s architecture and resources.

There are numerous IaC tools available such as Ansible, Puppet, Chef, and Terraform.

But for this guide, we will use CloudFormation, which was made specifically for AWS resources.

What You Will Learn in This Tutorial

After going through this tutorial, you will understand how to maintain your resources within one software file.

In addition to this, you will learn the benefits related to speed that Infrastructure as Code brings to the table. Without IaC, the time and cost of manual deployment of various infrastructures can be much greater compared to maintaining infrastructure as software.

In this article, we will consider an example. It will demonstrate manually provisioning resources vs deploying a CloudFormation script to create a serverless Lambda function and REST API on AWS.

Services We'll Use in This Tutorial

We will use the following services to implement Infrastructure as Code in AWS:

For those who are new to AWS, it's good to have some knowledge of it to understand the article. So, you can follow along with me by creating an account on AWS here and making sure you have AWS CLI installed to work with the example.

Overview of the Example

For the article, we will be implementing a REST API with an API gateway. It will be integrated with a serverless backend Lambda function that handles POST and gets requests made by our API.

The first step will show you how to manually build and deploy these resources using the AWS console. The second step will show you how to automate the process using CloudFormation.

How to Deploy Manually

In manual deployment, we will work inside the AWS console. It is a bit hard to track changes while working outside the local IDE, especially for large-scale projects.

In the first step, we will create a Lambda function.

If you want your Lambda function to work with some other service like Comprehend, you must give permissions for that service. So, make sure to create a role with these permissions.

Following is our Lambda function that will return “Hello World” when integrated with the API gateway.

import json

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello World!')
    }

Now that we have configured our Lambda function, the next step is to create a REST API to interact with the Lambda function.

For this, go to Amazon API Gateway, Click create API, and select REST API from the provided options.

Now we will integrate Lambda with the API. For this, create a GET method from the actions menu and point our REST API to the Lambda function.

We are in a position to deploy and test our API for its proper integration with Lambda. Select any stage name you want – for this example, I'm using “prod”.

After deploying the API, you can see a URL on the “prod” stage. Hitting this URL will trigger the Lambda function. As we have returned “Hello World” from our Lambda function, so you should be able to see the desired result.

How to Deploy with CloudFormation

Up to this point, we have seen how manual deployment works, which usually takes a few minutes.

But let’s imagine that we have more than one API, method, and more than one developer working on them. In this scenario, tracking all the resources and changes would be challenging.

So, in this section, we will use AWS CloudFormation instead. It will give flexibility to the developers, allowing them to adjust their Infrastructure with a simple script.

How Does CloudFormation Work?

We will use the YAML file to provision and declare these resources and deploy them to AWS to create a CloudFormation stack. CloudFormation is a stack that contains all the resources required for the project.

We will be using the SAM template as described above in the services section. It is an abstraction of CloudFormation to build serverless applications with less YAML code.

For those who don’t know about YAML, you can think about it like JSON. But CloudFormation uses both of these file formats.

In the first step, we head to our local IDE and write the same Lambda function as we did in the AWS console.

helloworld.py:

import json

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello World!')
    }

Next, we will create a template.yaml file containing our infrastructure. We will define our Lambda function and API Gateway in this file.

To build this file, we need to add some information that is common to all SAM templates.

template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: First CloudFormation template

Now, we have to add “Globals” to this CloudFormation template.yaml file. Globals are the common configs for the resources that you are going to deploy. Globals allow you to declare information globally for a specific resource type rather than specifying it again and again for different resources.

template.yaml:

Globals:
    #Common to all Lambda functions you create
    Function:
      MemorySize: 128
      Runtime: python3.6
      Timeout: 5

We define have to define the Resources tag in our template.yaml file. The Lambda function and REST API will come under this tag.

template.yaml:

Resources:

    ##Lambda and API GW Integrated
    helloworld:
        Type: AWS::Serverless::Function
        Properties:
          #filename.functionname
          Handler: helloworld.lambda_handler

          #REST API created
          Events:
            PostAdd:
              Type: Api
              Properties:
                Path: /helloworld
                Method: get

In the above code, we define parameters for creating the Lambda function. For the event, we create a REST API that triggers the Lambda function.

Note: There is an array of parameters like CodeURI and description that you can specify for your serverless function. The best way to create a template file is to go through the CloudFormation docs and see the parameters available for your specified resource/service.

How to Deploy the Template File

We can deploy our template.yaml file using the following two AWS CLI commands:

##s3 bucket stores our sam template which we need to deploy
aws cloudformation package --template-file template.yaml --output-template-file sam-template.yaml --s3-bucket helloworld-sam

After running the above command, you will be able to see a SAM template file. We will use this file in the second command below.

In this command, give your appropriate path to the sam-template.yaml file:

#Deploy stack
#point to template file created by a previous command and a stack name as well as your region you're deploying

aws cloudformation deploy --template-file /path to sam-template.yaml file --stack-name test-stack --capabilities CAPABILITY_IAM --region us-east-1

After executing both of these commands, you will see the stack created in the CLI. You can verify it using CloudFormation in the console.

Here you will see all the resources provisioned through the code created and deployed using the template.yaml file.

You can click on API and access the URL to check the output as we did for the manual deployment.

Wrapping Up

That’s it – you have successfully implemented infrastructure as code in AWS using CloudFormation.

I hope this article has been helpful for anyone wanting to understand implementing infrastructure as code in AWS.

Connect with me on LinkedIn and Twitter

Hasta la vista!

The technologies usually used for blockchains are purpose-driven, and you can use them for other projects as well.

The examples in this article can readily handle heavy loads and remain responsive and quick to execute user requests.

Because the cryptocurrency industry is growing fast, a lot of countries are setting rules around how everything should be managed. As a result, I will adhere to certain regulations and take into account certain details, such as the characteristics of the blockchain technology.

For instance, you can have overloads and performance problems with blockchain technology. And as the market for cryptocurrencies and blockchain technology expands, new products are more likely to appeal to a wide range of active blockchain technology users.

Because of this, I had to find a way to prevent the program from becoming overloaded in the event of a significant increase in users.

Tutorial Prerequisites

For this walkthrough, these are the following technologies we will be using. You should be familiar with them:

• Node.js (specifically, the NestJS framework) for backend development. Nest.js forces you to use a modular structure where each feature can be isolated and easily connected/disconnected to/from other modules. Nest supports TypeScript out-of-the-box.
• PostgreSQL is the database we'll use to collect data.
• Kafka JS serves incoming loads and establishes communication between microservices.
• Helm charts and Kubernetes (k8s) for deployment. These tools will enable easy deployment of scalable microservices infrastructure on any cloud platform (we will use AWS EKS)

Also, this article assumes you have a decent level of knowledge about Kubernetes, Helm, and Node. Let's dive in.

Development Method

Our primary goal for the first phase is to divide our application into microservices.

In addition to helping with service communication, load balancing with Kafka enables the one-by-one processing of input data. When we have many customers, processing times may increase, but at least the service will continue and be preserved.

Additionally, if the cluster has enough resources, we can spawn extra consumers for the group that manages particular events. This reduces delays and accelerates task processing.

In this situation, we will develop six different microservices:

1. Admin Microservice – We'll use the admin microservice for all administrative panel logic, which should be isolated from user-facing functionality.
2. Core Microservice – Logic pertaining to users and their accounts is contained in the core microservice. Identification, gifts, charts, profiles, and so on. However, this microservice does not carry out the duties of a financial service, such as processing payments and exchanging currency.
3. Payment Microservice – A financial service called a "payment microservice" includes logic for trade, exchange, and withdrawal transactions. There will be integrations with CEX and DeFi solutions.
4. Email and notification service – This microservice is in charge of informing the user of emails, push notifications, and other types of alerts. It contains a separate Kafka queue for requests from other microservices to send users emails or notifications.
5. Cron Tasks – A microservice called Cron Tasks Service transmits predetermined events for task processing. Microservices don't carry out tasks on their own. Holding such a microservice helps prevent skipping cron job iterations when, for instance, the processing service is down due to deployment or a breakdown. The event will remain in a queue as it waits to be executed.
6. Webhooks Microservice – The goal of the webhooks microservice is to prevent any events from external APIs that may be very significant and contain transaction statuses or other vital data from being missed. Such events are processed after being queued up (based on the sender API).

Now let's see how to make these microservices using Nest.js.

For the Kafka messages broker, you'll need to create configuration options. In order to store the shared modules and configurations of all microservices, we will establish a shared resources folder.

Microservices Configuration Options

Production apps must have configuration. The configuration is crucial for understanding what your production application consumes as you build out a microservice application. It is usually recommended practice to keep configuration settings distinct from your code when developing microservices.

import { ClientProviderOptions, Transport } from '@nestjs/microservices';

import CONFIG from '@application-config';

import { ConsumerGroups, ProjectMicroservices } from './microservices.enum';

const { BROKER_HOST, BROKER_PORT } = CONFIG.KAFKA;




export const PRODUCER_CONFIG = (name: ProjectMicroservices): ClientProviderOptions => ({

 name,

 transport: Transport.KAFKA,

 options: {

   client: {

     brokers: [${BROKER_HOST}:${BROKER_PORT}],

   },

 }

});



export const CONSUMER_CONFIG = (groupId: ConsumerGroups) => ({

 transport: Transport.KAFKA,

 options: {

   client: {

     brokers: [${BROKER_HOST}:${BROKER_PORT}],

   },

   consumer: {

     groupId

   }

 }

});

Let's link our microservice for the admin panel to Kafka in consumer mode. We can detect and manage events from topics thanks to it.

Make the app operate in microservice mode so that events can be consumed like this:

app.connectMicroservice(CONSUMER_CONFIG(ConsumerGroups.ADMIN));

 await app.startAllMicroservices();

We can see that groupId is included in the consumer configuration. It's a crucial choice that will enable customers from the same group to get events from topics and share them with one another to process them more quickly.

For instance, we can use autoscaling to launch more pods to divide loading between them and speed up the process double if our microservice receives events more quickly than it can process them.

Consumers must be included in the group for this to work, and after scaling, spawned pods will also be included. They won't have to handle the same subject events from several Kafka partitions because they can share loading.

Let's look at an illustration of how we can use Nest to capture and handle Kafka events.

Consumer Controller

import { Controller } from '@nestjs/common';

import { Ctx, KafkaContext, MessagePattern, EventPattern, Payload } from '@nestjs/microservices';




@Controller('consumer')

export class ConsumerController {

 @MessagePattern('hero')

 readMessage(@Payload() message: any, @Ctx() context: KafkaContext) {

   return message;

 }




 @EventPattern('event-hero')

 sendNotif(data) {

   console.log(data);

 }

}

Customers can operate in two modes. It accepts events, processes them without delivering a response (EventPattern decorator), or, after processing an event, returns the response to the producer (MessagePattern decorator).

Since it doesn't contain any additional source code layers to enable request/response functionality, EventPattern is preferable and you should choose it wherever possible.

Who Are the Producers?

We must supply producer configuration for a module that will be in charge of transmitting events in order to link producers.

Producer Connection

import { Module } from '@nestjs/common';

import DatabaseModule from '@shared/database/database.module';

import { ClientsModule } from '@nestjs/microservices';

import { ProducerController } from './producer.controller';

import { PRODUCER_CONFIG } from '@shared/microservices/microservices.config';

import { ProjectMicroservices } from '@shared/microservices/microservices.enum';




@Module({

 imports: [

   DatabaseModule,

   ClientsModule.register([PRODUCER_CONFIG(ProjectMicroservices.ADMIN)]),

 ],

 controllers: [ProducerController],

 providers: [],

})

export class ProducerModule {}

Event-based producer

import { Controller, Get, Inject } from '@nestjs/common';

import { ClientKafka } from '@nestjs/microservices';

import { ProjectMicroservices } from '@shared/microservices/microservices.enum';




@Controller('producer')

export class ProducerController {

 constructor(

   @Inject(ProjectMicroservices.ADMIN)

   private readonly client: ClientKafka,

 ) {}




 @Get()

 async getHello() {

   this.client.emit('event-hero', { msg: 'Event Based'});

 }

}

Request/response-based producer

import { Controller, Get, Inject } from '@nestjs/common';

import { ClientKafka } from '@nestjs/microservices';

import { ProjectMicroservices } from '@shared/microservices/microservices.enum';




@Controller('producer')

export class ProducerController {

 constructor(

   @Inject(ProjectMicroservices.ADMIN)

   private readonly client: ClientKafka,

 ) {}




 async onModuleInit() {

   // Need to subscribe to a topic

   // to make the response receiving from Kafka microservice possible

   this.client.subscribeToResponseOf('hero');

   await this.client.connect();

 }




 @Get()

 async getHello() {

   const responseBased = this.client.send('hero', { msg: 'Response Based' });

      return responseBased;

 }

}

Each microservice has the option of operating in one of the two modes—producer or consumer—or in both modes simultaneously (mixed).

Microservices typically employ mixed mode for load balancing, producing events to the subject, and consuming them while equally splitting the load.

For each microservice, we'll use a Kubernetes setup based on Helm chart templates.

There are several configuration files in the template:

• Hpa (horizontal pod autoscaler)
• Ingress controller
• Service
• Deployment

We'll examine each configuration file separately (without Helm templating).

How to deploy the admin-API

apiVersion: apps/v1

kind: Deployment

metadata:

 name: admin-api

spec:

 replicas: 1

 selector:

   matchLabels:

     app: admin-api

 template:

   metadata:

     labels:

       app: admin-api

   spec:

     containers:

     - name: admin-api

       Image: xxx208926xxx.dkr.ecr.us-east-1.amazonaws.com/project-name/stage/admin-api

       resources:

         requests:

           cpu: 250m

           memory: 512Mi

         limits:

           cpu: 250m

           memory: 512Mi

       ports:

         - containerPort: 80

       env:

         - name: NODE_ENV

           value: production




         - name: APP_PORT

           value: "80"

You can include more minimal configurations, such as resource limitations, health check configurations, update strategies, and so on in a deployment.

Admin-API service

---

apiVersion: v1

kind: Service

metadata:

 name: admin-api

spec:

 selector:

   app: admin-api

 ports:

   - name: admin-api-port

     port: 80

     targetPort: 80

     protocol: TCP

 type: NodePort

To use this service, we must make it available to the public. Let's utilise SSL setup to leverage a secure HTTPS connection and expose our app via a load balancer.

On our cluster, we must deploy a load balancer controller. The most widely used answer is as follows: Load Balancer Controller for AWS.

Next, we must set up ingress with the following settings:

Admin-API ingress controller

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

 namespace: default

 name: admin-api-ingress

 annotations:

   alb.ingress.kubernetes.io/load-balancer-name: admin-api-alb

   alb.ingress.kubernetes.io/ip-address-type: ipv4

   alb.ingress.kubernetes.io/tags: Environment=production,Kind=application

   alb.ingress.kubernetes.io/scheme: internet-facing

   alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:xxxxxxxx:certificate/xxxxxxxxxx

   alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'

   alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS

   alb.ingress.kubernetes.io/healthcheck-path: /healthcheck

   alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'

   alb.ingress.kubernetes.io/ssl-redirect: '443'

   alb.ingress.kubernetes.io/group.name: admin-api

spec:

 ingressClassName: alb

 rules:

   - host: example.com

     http:

       paths:

         - path: /*

           pathType: ImplementationSpecific

           backend:

             service:

               name: admin-api

               port:

                 number: 80

Once this configuration has been applied, a new alb load balancer will be formed. We must construct a domain with the name we specified in the 'host' option and direct traffic to our load balancer from this host.

Admin-API autoscaling configuration

apiVersion: autoscaling/v2beta1

kind: HorizontalPodAutoscaler

metadata:

 name: admin-api-hpa

spec:

 scaleTargetRef:

   apiVersion: apps/v1

   kind: Deployment

   name: admin-api

 minReplicas: 1

 maxReplicas: 2

 metrics:

   - type: Resource

     resource:

       name: cpu

       targetAverageUtilization: 90

How Does Helm Come into the Picture?

When we want to make our k8s infrastructure less complex, Helm is quite helpful. Without this tool, running it on a cluster requires writing numerous YML files.

Additionally, we must consider the relationships among applications, labels, names, and so on. Helm, on the other hand, can simplify things. It functions similarly to a package manager, allowing us to make an app template, prepare it using short commands, and then launch it.

Let's create our templates using Helm.

Admin-API deployment (Helm chart)

apiVersion: apps/v1

kind: Deployment

metadata:

 name: {{ .Values.appName }}

spec:

 replicas: {{ .Values.replicas }}

 selector:

   matchLabels:

     app: {{ .Values.appName }}

 template:

   metadata:

     labels:

       app: {{ .Values.appName }}

   spec:

     containers:

     - name: {{ .Values.appName }}

       image: {{ .Values.image.repository }}:{{ .Values.image.tag }}"

       imagePullPolicy: {{ .Values.image.pullPolicy }}

       ports:

       - containerPort: {{ .Values.internalPort }}

       {{- with .Values.env }}

       env: {{ tpl (. | toYaml) $ | nindent 12 }}

       {{- end }}

Admin-API service (Helm chart)

apiVersion: v1

kind: Service

metadata:

 name: {{ .Values.global.appName }}

spec:

 selector:

   app: {{ .Values.global.appName }}

 ports:

   - name: {{ .Values.global.appName }}-port

     port: {{ .Values.externalPort }}

     targetPort: {{ .Values.internalPort }}

     protocol: TCP

 type: NodePort

Admin-API ingress (Helm chart)

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

 namespace: default

 name: ingress

 annotations:

   alb.ingress.kubernetes.io/load-balancer-name: {{ .Values.ingress.loadBalancerName }}

   alb.ingress.kubernetes.io/ip-address-type: ipv4

   alb.ingress.kubernetes.io/tags: {{ .Values.ingress.tags }}

   alb.ingress.kubernetes.io/scheme: internet-facing

   alb.ingress.kubernetes.io/certificate-arn: {{ .Values.ingress.certificateArn }}

   alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'

   alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS

   alb.ingress.kubernetes.io/healthcheck-path: {{ .Values.ingress.healthcheckPath }}

   alb.ingress.kubernetes.io/healthcheck-interval-seconds: {{ .Values.ingress.healthcheckIntervalSeconds }}

   alb.ingress.kubernetes.io/ssl-redirect: '443'

   alb.ingress.kubernetes.io/group.name: {{ .Values.ingress.loadBalancerGroup }}

spec:

 ingressClassName: alb

 rules:

   - host: {{ .Values.adminApi.domain }}

     http:

       paths:

         - path: {{ .Values.adminApi.path }}

           pathType: ImplementationSpecific

           backend:

             service:

               name: {{ .Values.adminApi.appName }}

               port:

                 number: {{ .Values.adminApi.externalPort }}

Admin-API autoscaling configuration (Helm chart)

{{- if .Values.autoscaling.enabled }}

apiVersion: autoscaling/v2beta1

kind: HorizontalPodAutoscaler

metadata:

 name: {{ include "ks.fullname" . }}

 labels:

   {{- include "ks.labels" . | nindent 4 }}

spec:

 scaleTargetRef:

   apiVersion: apps/v1

   kind: Deployment

   name: {{ include "ks.fullname" . }}

 minReplicas: {{ .Values.autoscaling.minReplicas }}

 maxReplicas: {{ .Values.autoscaling.maxReplicas }}

 metrics:

 {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}

   - type: Resource

     resource:

       name: cpu

       targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}

 {{- end }}

 {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}

   - type: Resource

     resource:

       name: memory

       targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}

 {{- end }}

{{- end }}

The "values.yml," "values-dev.yml," and "values-stage.yml" files contain the values for the templates. The environment will determine which of them is used.

Let's look at few samples of dev env values.

Admin-API Helm values-stage.yml file

env: stage

appName: admin-api

domain: admin-api.xxxx.com

path: /*

internalPort: '80'

externalPort: '80'




replicas: 1

image:

 repository: xxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/admin-api

 pullPolicy: Always

 tag: latest




ingress:

 loadBalancerName: project-microservices-alb

 tags: Environment=stage,Kind=application

 certificateArn: arn:aws:acm:us-east-2:xxxxxxxxx:certificate/xxxxxx

 healthcheckPath: /healthcheck

 healthcheckIntervalSeconds: '15'

 loadBalancerGroup: project-microservices




autoscaling:

 enabled: false

 minReplicas: 1

 maxReplicas: 100

 targetCPUUtilizationPercentage: 80




env:

 - name: NODE_ENV

   value: stage




 - name: ADMIN_PORT

   value: "80

We must upgrade the chart and restart our deployment in order for the configuration to take effect on the cluster.

Let's investigate the GitHub Actions steps in question.

How to apply Helm configuration in GitHub Actions

GitHub actions are CI/CD services from GitHub. They provide straightforward work processes arranged as Yaml files which run configurable blocks of code based on GitHub events. Since they are integrated into GitHub, they reduce significantly the overhead in getting a CI/CD pipeline setup.

 - name: Admin image build and push

       run: |

         docker build -t project-admin-api -f Dockerfile.admin .

         docker tag project-admin-api ${{ env.AWS_ECR_REGISTRY }}/project/${{ env.ENV }}/admin-api:latest

         docker push ${{ env.AWS_ECR_REGISTRY }}/project/${{ env.ENV }}/admin-api:latest




     - name: Helm upgrade admin-api

       uses: koslib/helm-eks-action@master

       env:

         KUBE_CONFIG_DATA: ${{ env.KUBE_CONFIG_DATA }}

       with:

         command: helm upgrade --install admin-api -n project-${{ env.ENV }} charts/admin-api/ -f charts/admin-api/values-${{ env.ENV }}.yaml




     - name: Deploy admin-api image

       uses: kodermax/kubectl-aws-eks@master

       env:

         KUBE_CONFIG_DATA: ${{ env.KUBE_CONFIG_DATA }}

       with:

         args: rollout restart deployment/admin-api-project-admin-api --namespace=project-${{ env.ENV }}

Summary

In this article, we looked at infrastructure-building and Kubernetes cluster deployment steps for microservices. By using straightforward examples and avoiding further complexity with full configurations, I hope it was relatively easy to grasp.

Connect with me on LinkedIn and Twitter

Hasta la vista!

Prerequisites

To get the most out of this article, you will need an understanding of the following concepts:

• Node.js setup and installation
• How to build APIs with Node
• How to use the Postman tool
• How JavaScript async/await works
• How to work with a basic Redis application

What API Optimization Actually Means

Optimization involves improving the response time of your API. The shorter the response time is, the faster the API will be.

The tips I will share in this article will help you reduce response time, lower latency, manage errors and throughput, and minimize CPU and memory usage.

How to Optimize Node.js APIs

1. Always Use Asynchronous Functions

Async functions are like the heart of JavaScript. So, the best we can do to optimize CPU usage is to write asynchronous functions to perform nonblocking I/O operations.

I/O operations include the processes which perform read and write data operations. It can be the database, cloud storage, or any local storage disk on which the I/O operations are performed.

Using asynchronous functions in an application that heavily uses I/O operations will improve it. This is because the CPU will be able to handle multiple requests simultaneously due to non-blocking I/O, while one of these requests is making an Input/Output operation.

Here's an example:

var fs = require('fs');
// Performing a blocking I/O
var file = fs.readFileSync('/etc/passwd');
console.log(file);
// Performing a non-blocking I/O
fs.readFile('/etc/passwd', function(err, file) {
    if (err) return err;
    console.log(file);
});

• We use the fs Node package to work with files.
• readFileSync() is synchronous and blocks execution until finished.
• readFile() is asynchronous and returns immediately while things function in the background.

2. Avoid Sessions and Cookies in APIs, and Send Only Data in the API Response.

You use cookies and sessions to store temporary states in the server. They cost a lot for servers.

Now, stateless APIs are common and provide JWT, OAuth, and other authentication mechanisms. These authentication tokens are kept on the client side and protect the servers to manage the state.

JWT is a JSON-based security token for API Authentication. JWTs can be seen but they're not modifiable once they're sent. JWT is just serialized, not encrypted. OAuth is not an API or a service – rather, it's an open standard for authorization. OAuth is a standard set of steps for obtaining a token.

Also, don’t waste your time in making your Node.js server serve static files. Use NGINX and Apache instead, as they work far better than Node for this purpose.

While building APIs in Node, don’t send the full HTML page in the response of the API. Node servers work better when only data is sent by the API. Generally, this kind of application works with JSON data.

3. Optimize Database Queries

Query optimization is an essential part of building optimized APIs in Node. Especially in larger applications, you'll need to query databases many times. So, a bad query can reduce the overall performance of the application.

Indexing is an approach to optimize the performance of a database by minimizing the number of disk accesses required when a query is processed. It is a data structure technique that is used to quickly locate and access the data in a database. Indexes are created using a few database columns.

Let's say we have a DB schema without indexing and the database contains 1 million records. A simple find query will go through a larger number of records to find the matching one compared to the schema with indexing.

• Query without indexing:

> db.user.find({email: 'ofan@skyshi.com'}).explain("executionStats")

• Query with indexing:

> db.getCollection("user").createIndex({ "email": 1 }, { "name": "email_1", "unique": true })
{
 "createdCollectionAutomatically" : false,
 "numIndexesBefore" : 1,
 "numIndexesAfter" : 2,
 "ok" : 1
}

There is a huge difference in the number of documents scanned ~ 1038:

4. Optimize APIs with PM2 Clustering

PM2 is a production process manager designed for Node.js applications. It has a built-in load balancer and allows the application to run as multiple processes without code modifications.

Application downtime is almost zero using PM2. Overall, PM2 can really improve the performance and concurrency of your API.

Deploy the code on production and run the following command to see how the PM2 cluster has scaled on all available CPUs:

pm2 start  app.js -i 0

5. Reduce TTFB (Time to First Byte)

Time to the first byte is a measurement used as an indication of the responsiveness of a web server or other network resource. TTFB measures the duration from the user or client making an HTTP request to the first byte of the page being received by the client's browser.

It is unlikely that the page all users are accessing on the web browser loads within 100ms. This is simply because of the physical distance between the server and the users.

Here, we can reduce the Time to First Byte by using a CDN and caching content in local data centers across the globe. This helps users access the content with minimal latency. Cloudflare is one of the CDN solutions you can use to start with.

6. Use Error Scripts with Logging

The best way to monitor the proper functioning of your APIs is to keep track of their activity. This is where logging the data comes into play.

A common example of logging is printing out the logs to the console (using console.log()).

More efficient logging modules as compared to console.log are Morgan, Buyan, and Winston. Here, I’ll go with the example of Winston.

How to log with Winston – features

• Provides 4 custom levels that we can use such as info, error, verbose, debug, silly, and warn.
• Supports querying the logs
• Simple profiling
• You can use multiple transports of the same type
• Catches and logs uncaughtException

You can set up Winston with the following command:

npm install winston --save

And here's a basic configuration of Winston for logging:

const winston = require('winston');

let logger = new winston.Logger({
  transports: [
    new winston.transports.File({
      level: 'verbose',
      timestamp: new Date(),
      filename: 'filelog-verbose.log',
      json: false,
    }),
    new winston.transports.File({
      level: 'error',
      timestamp: new Date(),
      filename: 'filelog-error.log',
      json: false,
    })
  ]
});

logger.stream = {
  write: function(message, encoding) {
    logger.info(message);
  }
};

7. Use HTTP/2 Instead of HTTP

In addition to these techniques, we can also apply some other techniques like using HTTP/2 over HTTP, as it has the following advantages:

• Multiplexing
• Header compression
• Server push
• Binary format

It focuses on the performance and issues that the previous version of HTTP has. It makes web browsing faster and easier and consumes less bandwidth.

8. Run Tasks in Parallel

Use async.js to help you run tasks. Parallelizing tasks has a great impact on the performance of your API. It reduces latency and minimizes blocking operations.

Parallel means running multiple things at the same time. However, when you run things in parallel, you don’t need to control the execution sequence of the program.

Here's a simple example using async parallel with an array:

const async = require("async");
// an example using an object instead of an array
async.parallel({
  task1: function(callback) {
    setTimeout(function() {
      console.log('Task One');
      callback(null, 1);
    }, 200);
  },
  task2: function(callback) {
    setTimeout(function() {
      console.log('Task Two');
      callback(null, 2);
    }, 100);
    }
}, function(err, results) {
  console.log(results);
  // results now equals to: {task2: 2, task1: 1}
});

In this example, we used async.js to execute the two tasks in asynchronous mode. Task 1 requires 200 ms to complete, but task 2 does not wait for its completion – it executes at its specified delay of 100ms.

Parallelizing tasks has a great impact on the performance of API. It reduces latency and minimizes blocking operations.

9. Use Redis to Cache the App

Redis is the advanced version of Memcached. It optimizes the APIs response time by storing and retrieving the data from the main memory of the server. It increases the performance of the database queries which also reduces access latency.

In the following code snippets, we have called the APIs without and with Redis, respectively, and compared the response time.

There is a huge difference in the response time ~ 899.37ms:

Here's Node without Redis:

'use strict';

//Define all dependencies needed
const express = require('express');
const responseTime = require('response-time')
const axios = require('axios');

//Load Express Framework
var app = express();

//Create a middleware that adds a X-Response-Time header to responses.
app.use(responseTime());

const getBook = (req, res) => {
  let isbn = req.query.isbn;
  let url = `https://www.googleapis.com/books/v1/volumes?q=isbn:${isbn}`;
  axios.get(url)
    .then(response => {
      let book = response.data.items
      res.send(book);
    })
    .catch(err => {
      res.send('The book you are looking for is not found !!!');
    });
};

app.get('/book', getBook);

app.listen(3000, function() {
  console.log('Your node is running on port 3000 !!!')
});

And here's Node with Redis:

'use strict';

//Define all dependencies needed
const express = require('express');
const responseTime = require('response-time')
const axios = require('axios');
const redis = require('redis');
const client = redis.createClient();

//Load Express Framework
var app = express();

//Create a middleware that adds a X-Response-Time header to responses.
app.use(responseTime());

const getBook = (req, res) => {
  let isbn = req.query.isbn;
  let url = `https://www.googleapis.com/books/v1/volumes?q=isbn:${isbn}`;
  return axios.get(url)
    .then(response => {
      let book = response.data.items;
      // Set the string-key:isbn in our cache. With the contents of the cache : title
      // Set cache expiration to 1 hour (60 minutes)
      client.setex(isbn, 3600, JSON.stringify(book));

      res.send(book);
    })
    .catch(err => {
      res.send('The book you are looking for is not found !!!');
    });
};

const getCache = (req, res) => {
  let isbn = req.query.isbn;
  //Check the cache data from the server redis
  client.get(isbn, (err, result) => {
    if (result) {
      res.send(result);
    } else {
      getBook(req, res);
    }
  });
}
app.get('/book', getCache);

app.listen(3000, function() {
  console.log('Your node is running on port 3000 !!!')
)};

Conclusion

In this guide, we have learned how can we optimize the response time of Node.js APIs.

JavaScript depends heavily on functions. So, using async functions can make the script faster and non-blocking.

Other than this, we used cache memory (Redis), database indexing, TTFB and PM2 clustering to enhance the response times.

Lastly, keep in mind that it's important to pay attention to the security of the routes and make sure they're as optimized as possible. We cannot compromise a quick API response over a security loophole. So, you should keep all your standard security checks while building optimized APIs in Node.

Connect with me on LinkedIn.

Hasta la vista!