When you learn Terraform, you work with a single repository, state file, and a single environment. You run terraform apply from your laptop, and your infrastructure is provisioned.

That model works fine until the day you join a company and realize engineers rarely apply to production from a laptop.
A lot of what you see will not match what you practiced.

This article explains how large engineering teams actually run Terraform, the repositories, workflows, ownership rules, and what goes wrong without them.

You'll learn how enterprise teams structure repositories and state files, how they store and version reusable modules through GitHub, why infrastructure changes move to production through pipelines, how they catch changes that happen outside of Terraform, and how they recover when things go wrong.

Every practice here exists because a team hit a specific wall and built something to get past it.

Prerequisites

You should be comfortable with Terraform before reading this.
You should also know how Git pull requests and branch merging work.

This is not a Terraform introduction, it is about what happens after you have learned the basics and start sharing infrastructure with other engineers.

Table of Contents

• How State Corruption Happens

• Why State File Gets Treated Like a Production Database

• How Enterprise Teams Structure Their Terraform Repositories

• How Teams Split State Files to Protect Each Other

• Why Some Teams Prefer Directories Over Workspaces for Production

• How Teams Share Infrastructure Through Modules on GitHub

• How Teams Version and Release Terraform Modules

• How Teams Maintain Terraform Modules at Scale

• How Teams Share Data Between State Files

• How Infrastructure Changes Actually Move to Production

• How Teams Detect Infrastructure Drift

• How Teams Recover When State Goes Wrong

• Conclusion

How State Corruption Happens

The state file is how Terraform tracks what it has built. It remembers every resource, every ID, and every configuration value. When it gets out of sync with what actually exists in the cloud, that's state corruption.

It gets blamed for a lot of things. But engineers who have dealt with it in production know it usually traces back to one of a handful of situations, each with a different cause and a different fix.

Two Engineers Run terraform apply at the Same Time

Before understanding this one, you need to understand something about how Terraform works.

When you run terraform apply, two things happen separately:

First, Terraform talks to AWS, and the resource gets created in the cloud. Second, Terraform updates the state file to record what was just built.

These are two different systems. AWS holds the real infrastructure, and the state file is Terraform's notebook about it. If anything interrupts the process between step one and step two, they fall out of sync.

Now here's what happens when two engineers apply at the same time without locking:

Sarah opens the state file and starts adding a subnet. Marcus opens the same state file at the same moment and starts updating a NAT gateway. Both are working from the same starting copy.

Sarah finishes first. Her apply creates the subnet in AWS and updates the state file to record it.

Marcus finishes second. His apply updates the NAT gateway in AWS. Terraform then updates the state file using the version of state Marcus read when his apply started.

That version didn't include Sarah's subnet, so the updated state no longer contains a record of it.

The subnet exists in AWS. But Terraform's notebook no longer has a record of it. The next terraform plan thinks the subnet was never created and proposes building it again.

State locking prevents this. Sarah's apply acquires a lock before it starts. When Marcus tries to apply, Terraform makes him wait.

After Sarah finishes, Terraform updates the state file and releases the lock. Marcus then runs against the updated state, so both the subnet and NAT gateway changes are recorded correctly.

An Apply Gets Interrupted

A GitHub Actions pipeline is applying changes to the payments infrastructure, adding three new security group rules and a database parameter group. Halfway through, the pipeline runner hits its 60-minute timeout limit, and the job gets killed.

Here's what the apply actually managed to do before dying:

The terminal image above shows three security group rules completing successfully before the pipeline hits its 60-minute runtime limit. The runner is then terminated. The database parameter group never finishes creating, and the state file update never runs because the job died first.

Security group rule 1  → created ✓
Security group rule 2  → created ✓
Security group rule 3  → created ✓
Database parameter     → not created ✗
State file update      → never wrote (job died first)

The three security group rules now exist in AWS. The problem is that the pipeline died before Terraform could finish updating the state file. AWS knows the rules exist. Terraform's state file does not.

At this point, reality and the state file no longer match.

Fortunately, this is usually easy to recover from. When the pipeline runs again, Terraform checks what already exists in AWS. It sees the three security group rules and doesn't try to create them again. It then creates the database parameter group that never got built.

The second run completes successfully and the state file catches up.

This works because Terraform is idempotent, running the same configuration again moves infrastructure toward the desired state rather than blindly creating everything from scratch.

One small complication remains: the state lock.

If the pipeline was interrupted while holding a lock, Terraform may still think another apply is running. The next pipeline run fails immediately with an error like this:

The terminal above shows terraform apply failing because the previous job left a state lock behind. The error includes the lock ID, the path to the state file, and the name of the process that acquired it. Terraform refuses to proceed until the lock is released or manually cleared.

Before clearing the lock, make sure no Terraform apply is still running.

Open your CI/CD system. GitHub Actions, GitLab CI, Jenkins, or whatever your team uses and check the pipeline history for that environment:

The GitHub Actions pipeline history image above shows four recent runs. terraform-plan completed successfully. Two terraform-apply jobs show as cancelled and timed out, both flagged as lock may be stale. A fourth terraform-apply job is currently in progress, and this one shouldn't be unlocked until it finishes.

If the previous apply was cancelled or timed out, the lock is stale. Clear it with terraform force-unlock plus the lock ID from the error. The pipeline then runs normally.

Only force-unlock when you're certain nothing is actively running. Clearing a live lock lets two applies write to the same state at the same time, which is exactly the problem locking was built to prevent.

Someone Runs a Terraform State Command in the Wrong Environment

A database engineer is cleaning up an old test database in the staging environment.

The database still exists in AWS, but Terraform should stop managing it. To do that, the engineer uses terraform state rm.

This command doesn't delete anything in AWS. It only removes Terraform's record of the resource from the state file. Think of it as telling Terraform: "forget this resource exists, but leave it running."

The engineer intends to run it against staging:

Intended:  staging state       → forget the old test database

But they're working in the wrong directory. They run it against production instead.

Actual:    production state    → forget the live payments database

Nothing gets deleted. The production database is still running in AWS. But Terraform has now forgotten it exists.

Now Terraform and reality disagree. The next terraform plan sees a database defined in the code but missing from the state file, so it assumes the database doesn't exist and proposes creating a new one.

If nobody catches it in the plan output, Terraform creates a second production database alongside the original: two databases running in production, neither fully managed, and a very expensive mess to untangle.

terraform state rm, terraform import, and terraform state mv make immediate changes to the state file with no confirmation prompt. Run them from the wrong directory, the wrong workspace, or with the wrong resource address and you change the wrong state in seconds.

Two Teams Manage the Same Resource

The networking team owns a security group that controls access to the payments database. When a new microservice needs database access, a payments engineer has two options: ask the networking team to add a new rule, or manage the security group themselves.

They choose the second option. The engineer imports the existing security group into the payments state file and adds a rule for Microservice C.
From that moment, both teams think they own the same security group.

The problem is that Terraform does exactly what each state file tells it to do. The networking state says the security group should allow A and B. The payments state says it should allow A, B, and Microservice C. Both can't be true at the same time.

When the payments team applies their state, Microservice C gets access. But later that night, the networking pipeline runs. Terraform reads the networking state, sees only A and B, and updates the security group to match. Microservice C's rule disappears silently.

No errors are seen and both pipelines pass, which is exactly what makes this so hard to debug. Terraform isn't broken, it's receiving conflicting instructions from two different state files and doing exactly what each one says.

This isn't something to be fixed with Terraform commands. It's an ownership decision that should have been made before anyone ran an import. If the payments team had submitted a pull request to the networking repository asking them to add the rule, one team would own the security group, one state file would manage it, and the conflict could never have happened.

Why State File Gets Treated Like a Production Database

The state file looks like bookkeeping: a record of what Terraform created. The reason teams treat it differently is that it often contains secrets.

The state file stores sensitive values in plaintext. Database passwords, API keys, connection strings – if those values were passed to a Terraform resource during an apply, they're now sitting in the state file. Even if you marked the variable as sensitive in your Terraform code, the value still lands in the state file. Terraform needs it there to compute diffs on future plans.

That means: whoever can read the state file can read your database password.

In large organizations, engineers typically don't have direct access to the production state bucket. Instead, Terraform runs through a CI/CD pipeline that assumes a dedicated IAM role with permission to read and write the state bucket and perform applies. Engineers interact with infrastructure through pull requests and plan output, not by touching the state bucket directly.

This separation reduces risk and creates an audit trail. Every state change is performed by the pipeline and logged, making it straightforward to trace what changed and when.

How Enterprise Teams Structure Their Terraform Repositories

When you join a large engineering organization, the first thing you notice is the number of repositories. You might expect one repository for all infrastructure, but what you find is dozens.

The structure maps directly to ownership. Each repository belongs to one team, and that team is responsible for everything in it. A typical layout looks like this:

The diagram shows two types of repositories. The first type belongs to the platform team and contains reusable modules: things like VPC configurations, database templates, and security group patterns. These repositories don't create production resources directly.

The second type belongs to individual product teams, such as the payments team or the auth team. These repositories call the platform modules and use them to build their actual infrastructure. A mistake in a product team repository affects only that team. A mistake in a shared platform module can affect every team that depends on it.

The key thing to understand here is that the platform team repositories don't create production resources. They create reusable modules that the product teams call when building their actual infrastructure.

That distinction matters because some repositories are used by one team, while others are shared by everyone.

A mistake in a product team's repository usually affects only that team. A mistake in a shared module can affect every team that depends on it.

The diagram illustrates why shared repositories carry more risk than product-specific ones. A bug in the payments-infra repository affects only the payments team. A bug in the terraform-aws-postgres module affects every team that uses it to provision databases. A bug in the terraform-policies repository affects every pipeline in the company. The wider the module is shared, the larger the blast radius when something goes wrong.

This is why experienced engineers pay close attention to shared modules and policy repositories.

If the payments team's infrastructure breaks, the problem is probably in the payments repository.

If five different teams start seeing the same issue at the same time, the shared modules and policy repositories become the first place to investigate.

How Teams Split State Files to Protect Each Other

A single state file managing everything, VPC, Kubernetes cluster, databases, monitoring, is fine when one person is running things, but quickly becomes a problem when multiple teams share it.

Three specific problems emerge.

1. Blast radius: If the networking configuration and the database configuration live in the same state file, a bad networking apply can accidentally affect database resources that had nothing to do with the change. Separate state files keep failures contained.

2. Deployment speed: Networking infrastructure might change a few times a year. Applications might deploy dozens of times a day. If they share a state file, teams end up waiting on each other's locks.

3. Ownership conflicts: When multiple teams share a state file, one team can change something the other team depends on without realizing it.

The solution is to split state along ownership boundaries. A structure that addresses all three problems looks like this:

The structure image above shows one state file per domain under a production folder.

• networking handles VPC, subnets, routing, and NAT gateways.

• identity handles IAM roles, policies, and service accounts.

• platform handles the Kubernetes cluster, node pools, and add-ons.

• database handles RDS instances, Redis clusters, and backups.

• security handles security groups, WAF rules, and certificates.

• monitoring handles Prometheus, Grafana, and alerting pipelines.

• payments handles payment service infrastructure.

production/
  networking/terraform.tfstate   → VPC, subnets, routing, NAT gateways
  identity/terraform.tfstate     → IAM roles, policies, service accounts
  platform/terraform.tfstate     → Kubernetes cluster, node pools, add-ons
  database/terraform.tfstate     → RDS instances, Redis clusters, backups
  security/terraform.tfstate     → Security groups, WAF rules, certificates
  monitoring/terraform.tfstate   → Prometheus, Grafana, alerting pipelines
  payments/terraform.tfstate     → Payment service infrastructure

This is one example, not a universal standard. Larger organizations often split further. The principle is the same: one owning team per state file, one pipeline, one blast radius.

The rule is simple: every resource belongs to one state file. If the networking team owns a security group, it stays in the networking state. Other teams can reference it as a data source, but they don't import it into their own state.
That is what prevents the ownership collision described in the first section.

Why Some Teams Prefer Directories Over Workspaces for Production

Terraform CLI workspaces let you manage multiple environments like dev, staging, and production from a single directory. Each workspace gets its own state file, but they all share the same .tf configuration files.

infra/
  main.tf          ← same code runs for ALL environments
  variables.tf

  terraform.tfstate.d/
    dev/
    staging/
    production/    ← separate state, same code

The workspace approach keeps all environments in one directory called infra. It contains a single main.tf file that runs for all environments. State is stored separately under terraform.tfstate.d with folders for dev, staging, and production, but all three share the same code.

You switch environments with terraform workspace select production, then apply.

The risk is that switching workspaces is a manual step. If the wrong workspace is active, changes meant for staging can end up in production.

Many teams prefer separate directories for long-lived environments:

environments/
  dev/
    main.tf      ← its own code path
    backend.tf   ← points to the dev state bucket
  staging/
    main.tf      ← its own code path
    backend.tf   ← points to the staging state bucket
  production/
    main.tf      ← its own code path
    backend.tf   ← points to the production state bucket

The directory approach gives each environment its own folder under environments. Dev, staging, and production each have their own main.tf with a separate code path, and their own backend.tf pointing to a different state bucket. The environments are completely separate from each other.

To apply against production, you have to be in the production directory. Each environment has its own state, backend, and execution path.

The tradeoff is duplication. Teams usually solve that with shared modules, so each environment directory contains only environment-specific configuration.

Workspaces are still useful for short-lived environments such as feature branches, preview deployments, and temporary test infrastructure.

How Teams Share Infrastructure Through Modules on GitHub

When 30 teams each need a PostgreSQL database, two things happen.

Without a shared standard, every team writes their own database configuration. Six months later, a security audit runs across all environments and finds that:

The diagram shows what a security audit found when four teams each wrote their own database configuration independently.

Team A set backup_retention_period = 0, meaning their database was never backed up. Team B set storage_encrypted = false, leaving data in plaintext. Team C passed an empty tags = {}, so there was no cost tracking. Team D set deletion_protection = false, leaving the database one accident away from permanent data loss.

Nobody skipped those things on purpose, there was just no shared standard.

With a shared module, the platform team writes a postgres module once. They encode every organizational requirement into it: encryption on, 7-day backups, monitoring alarms, required tags, deletion protection enabled. They publish it to a GitHub repository called terraform-aws-postgres.

Every team that needs a database now writes this:

module "payments_db" {
  source         = "git::ssh://github.company.com/platform/terraform-aws-postgres.git?ref=v2.1.0"
  name           = "payments"
  environment    = "production"
  instance_class = "db.m5.large"
}

Four inputs. Everything else is handled by the module.

Large organizations usually expose approved modules through an internal registry so engineers can discover and version them without browsing GitHub repositories. Instead of the full Git URL, the reference becomes:

module "payments_db" {
  source  = "app.terraform.io/mycompany/postgres/aws"
  version = "~> 2.1"
}

HCP Terraform and Terraform Enterprise both include a private registry that connects to GitHub, watches for version tags on module repositories, and publishes new versions automatically.

How Teams Version and Release Terraform Modules

The ?ref=v2.1.0 in a module source URL isn't decoration. At the scale of 40 teams sharing one module, it's the thing that prevents a well-intentioned change from becoming a company-wide incident.

Without version pinning, the payments team references the Postgres module from main meaning whatever the latest code is at any given moment. The module owners rename an output variable from db_endpoint to database_endpoint to match a new naming convention. The next time any team runs terraform init, they pull that change. Their configuration still references db_endpoint.

Plans break:

payments-infra                        → plan fails
analytics-infra                       → plan fails
auth-infra                            → plan fails
reporting-infra                       → plan fails

Version pinning prevents this. The payments team stays on v2.1.0. The module owners release v2.2.0 with the renamed output and write a changelog. Teams upgrade when they're ready, after testing in staging. Nobody's pipeline breaks without warning.

The versioning convention is called semantic versioning:

v2.1.1  → patch:  bug fix. Safe to upgrade. Nothing to change in your code.
v2.2.0  → minor:  new optional feature. Safe to upgrade. Nothing to change.
v3.0.0  → major:  breaking change. Read the changelog. Update your code first.

The table shows three version types. A patch version like v2.1.1 means a bug fix, safe to upgrade with nothing to change in your code. A minor version like v2.2.0 means a new optional feature, also safe to upgrade with nothing to change. A major version like v3.0.0 means a breaking change, so you need to read the changelog and update your code before upgrading.

How Teams Maintain Terraform Modules at Scale

Building a Terraform module takes an afternoon, bit maintaining it for two years is a different job entirely.

A networking engineer needs a VPC module. The platform team has one, but their backlog is full. So the engineer creates a slightly different version. Three months later, another team does the same. Then another. Now this exists:

terraform-aws-vpc           ← original, maintained by platform team
terraform-aws-vpc-v2        ← created by the app team, author unknown
terraform-aws-vpc-shared    ← no idea which environments use this
terraform-aws-vpc-prod      ← unclear if this was ever different from the original

No one created a module graveyard on purpose. It grew one "I'll just make a quick variation" at a time. Each variant has slightly different security settings, different tagging, different defaults. When a compliance audit requires all VPCs to enable flow logging, the team has to investigate four different modules to figure out which environments are compliant.

Teams that avoid this treat their modules like shared services: named owner, contributions through pull requests, breaking changes in major versions with a migration guide, and deprecated modules with a retirement date. A CODEOWNERS file routes every pull request to the right reviewer automatically.

Organizations that skip this end up with modules that nobody owns, nobody wants to touch, and nobody is sure can be safely removed.

How Teams Share Data Between State Files

Once infrastructure is split into separate state files, a practical problem surfaces: teams need information from each other's infrastructure. The platform team's Kubernetes state needs the VPC ID from the networking team's state. The database state needs subnet IDs. The payments state needs the database endpoint.

Two patterns exist for solving this.

Reading Another Team's State Outputs

The terraform_remote_state data source lets one state read the outputs of another. The networking team marks their VPC ID and subnet IDs as outputs. The database team reads those outputs and uses them to place databases in the right subnets.

Networking state
  └── outputs: vpc_id, private_subnet_ids
                          ↓
               Database state reads them
               └── places RDS in the right subnets

This works, but there's a limitation. Reading another team's state requires full read access to their entire state file, not just the outputs you want. State files contain database passwords and API keys in plaintext. More dependencies means more teams reading each other's secrets.

Looking Up Resources Directly From the Cloud

The alternative, and the one HashiCorp now recommends, is to look up resources through the cloud provider's API instead of reading another team's state:

data "aws_vpc" "main" {
  tags = {
    Name        = "production-vpc"
    Environment = "production"
  }
}

No cross-team state access needed, and each team's state stays isolated. The tradeoff is consistent tagging: the networking team has to tag their VPC in a way the database team can reliably search for, which forces teams to agree on naming conventions early.

Many teams use both. Remote state for a small number of trusted, tightly coupled dependencies. Cloud data sources for everything broader.

How Infrastructure Changes Actually Move to Production

In large organizations managing production Terraform at scale, changes don't come from someone's laptop. Applying directly from a local machine requires production cloud credentials sitting on that machine, a security risk and leaves no audit trail if something breaks.

Instead, production changes move through a pipeline. Every change goes through a pull request in GitHub, and the pipeline does the work:

Engineer opens a pull request
        ↓
Pipeline: terraform validate + fmt check
        ↓
Pipeline: security scan (Checkov, tfsec, or similar)
        ↓
Pipeline: terraform plan → posts the full output as a comment on the PR
        ↓
Reviewer reads the plan output (not just the code)
        ↓
Required reviewers approve (enforced by CODEOWNERS + branch protection)
        ↓
Merge triggers the apply pipeline
        ↓
Pipeline: acquires state lock → applies → releases lock → logs result

The diagram above shows eight steps in order. An engineer opens a pull request. The pipeline runs terraform validate and a format check. A security scan runs using Checkov, tfsec, or similar. The pipeline runs terraform plan and posts the output as a comment on the pull request. A reviewer reads the full plan output. Required reviewers approve, enforced by CODEOWNERS and branch protection rules. Merging triggers the apply pipeline. The pipeline acquires the state lock, applies the changes, releases the lock, and logs the result.

The part that surprises engineers when they first encounter this is that the reviewer isn't approving the code. They're approving the plan output and the list of exactly what will be created, changed, or destroyed in the cloud.

A code change can look completely harmless and produce a destructive plan. Changing one database parameter might force a resource replacement, meaning Terraform destroys the current database and creates a new one. Seeing this in the plan output before the PR merges:

# aws_db_instance.payments must be replaced
-/+ resource "aws_db_instance" "payments" {

The image above shows a plan output that aws_db_instance.payments must be replaced, meaning Terraform will destroy the existing database and create a new one, not update it in place.

Catching that before merge is the entire point of reviewing the plan. Not the code.

How CODEOWNERS Enforces Who Reviews What

Earlier, we talked about module ownership. A VPC module might belong to the platform team, while database infrastructure belongs to the database team.

The challenge is making sure changes are actually reviewed by the people who own them.

GitHub solves this with a feature called CODEOWNERS. It lets a repository define which team is responsible for which directories. When someone opens a pull request that touches those files, GitHub automatically requests reviews from the correct team.

For example, if an engineer modifies the PostgreSQL module, GitHub can automatically require approval from the platform team before the change can be merged.

Without CODEOWNERS, engineers have to remember who owns which parts of the infrastructure.

CODEOWNERS makes ownership explicit and automatically requests reviews from the right team.

How Teams Detect Infrastructure Drift

Drift is the diff between what Terraform says should exist and what actually exists in the cloud.

Here's the scenario that produces drift more reliably than anything else:

Monday 3:00 AM  Production database CPU spikes. Outage.
Monday 3:15 AM  Engineer resizes database in AWS console: db.m5.large → db.m5.4xlarge
Monday 3:20 AM  Incident resolved. Engineer goes to sleep.
Monday 3:21 AM  Terraform state file: still says db.m5.large

The incident is forgotten, the ticket is closed, and life moves on.

Three months later, a routine Terraform apply runs. Terraform sees db.m5.large in the configuration but finds db.m5.4xlarge running in AWS. From Terraform's perspective, the database is larger than it should be, so the plan proposes changing it back.

Nobody notices the change in the plan output. The apply goes through, the database is downsized, and users begin reporting slow queries. The team spends hours investigating before eventually tracing the issue back to a Terraform change that reverted the emergency fix from months earlier.

Teams that handle this well run scheduled terraform plan jobs against every production state. If terraform plan exits with code 2, differences were found and an alert fires. The team then decides whether to apply to restore declared state or update the configuration to match reality. Either way, the change is visible and deliberate. Invisible drift always gets worse.

How Teams Recover When State Goes Wrong

State is recoverable in almost every situation, as long as the team set things up correctly before the incident happened.

The teams that recover in twenty minutes instead of three days aren't the ones with the deepest Terraform expertise. They're the ones who prepared.

Step 1: Pull a Backup Before Touching Anything.

terraform state pull > backup-$(date +%Y%m%d-%H%M%S).json

This saves the current state to a local file. Whatever you try next, you have a starting point to return to.

Step 2: Run terraform plan and Look at What it Proposes.

If Terraform proposes destroying resources that still exist in the cloud, the state is behind reality. If it proposes creating resources that already exist, reality is ahead of the state. Either way, the plan output tells you which direction the mismatch runs.

Step 3: Restore from S3 Versioning if the State is Corrupted.

Every write to a versioned S3 bucket saves a new version automatically. If the state file is corrupted or wrong, list the previous versions, download the last known good one, and push it back:

# List previous versions
aws s3api list-object-versions \
  --bucket mycompany-terraform-state \
  --prefix production/database/terraform.tfstate

# Download a specific version
aws s3api get-object \
  --bucket mycompany-terraform-state \
  --key production/database/terraform.tfstate \
  --version-id "the-version-id-here" \
  recovered-state.json

# Push it back
terraform state push recovered-state.json

Run terraform plan after restoring to confirm it looks correct before running any apply.

Step 4: Clear a Stale Lock if the Pipeline is Blocked.

If a lock was never released after a failed apply, clear it:

terraform force-unlock LOCK_ID

Only do this after confirming no apply is actively running. Clearing a live lock corrupts the state.

Step 5: Re-import Resources That Fell Out of State.

If a resource exists in the cloud but Terraform no longer knows about it — because of an accidental terraform state rm — bring it back without recreating it:

terraform import aws_db_instance.payments db-ABCD1234EFGH5678

Run terraform plan after importing to confirm no unexpected changes are proposed.

Conclusion

Every practice in this article traces back to a specific problem teams ran into as Terraform usage grew.

State locking prevents engineers from overwriting each other's changes.
State splitting reduces blast radius. Module versioning prevents shared infrastructure from breaking unexpectedly. Drift detection catches changes made outside Terraform. CODEOWNERS ensures the right people review the right changes.

Different problems with different solutions. But they all point to the same underlying theme which is ownership.

As teams grow, many Terraform problems have less to do with infrastructure and more to do with ownership.

State collisions happen when multiple people can modify the same state.
Module sprawl happens when nobody is responsible for maintaining a shared standard.

Drift becomes dangerous when changes are made without anyone taking ownership of bringing Terraform and reality back into alignment. Even review bottlenecks often trace back to uncertainty about who should approve what.

Understanding this changes how you read an unfamiliar Terraform repository.

Dozens of small state files aren't necessarily over-engineering. They're often ownership boundaries. A CODEOWNERS file is not bureaucracy. It's an ownership map. A pipeline that posts plan output on a pull request isn't just automation, it's a review process built around infrastructure consequences rather than code.

The infrastructure matters. But as teams grow, ownership is what keeps the system understandable.

I write about DevOps engineering, production systems, and the things tutorials do not cover weekly. If this was useful, please join the newsletter.
If you enjoyed reading this, we can also connect on Linkedin.

But managing hundreds across multiple teams, products, and deployment environments is where things start to break down.

At first, repository management feels simple. A team creates a repository, pushes code, and starts building features.

But as the organization grows, repositories multiply, and new services appear. Teams expand. Deployment pipelines become more complex. Different security requirements emerge. Suddenly, nobody knows who owns what, branch policies differ from one repository to another, and onboarding new developers becomes increasingly difficult.

I've seen this happen repeatedly in growing engineering teams.

What starts as a clean Azure DevOps environment eventually becomes a collection of inconsistent repositories, duplicated configurations, bloated Git histories, and fragmented governance.

The good news is that Azure Repos provides everything needed to prevent this from happening.

The challenge isn't creating repositories. The challenge is creating a repository strategy that continues working as your engineering organization grows.

In this guide, you'll learn how to organize and maintain Azure Repositories at scale using ownership-driven structures, cross-repository governance, automation, and repository maintenance practices that support long-term growth.

Table of Contents

• Why Repository Organization Becomes a Scaling Problem

• Build an Ownership-Driven Repository Structure

• Choosing Between a Mono-Repo and Multi-Repo Strategy

  • When a Mono-Repo Makes Sense

  • When a Multi-Repo Strategy Works Better

• Separate Azure DevOps Projects by Business Boundaries

• Create Naming Standards Before Repository Growth Happens

• Implement Cross-Repository Policies Instead of Managing Repositories Individually

• Enforce Branch Policies Across Multiple Repositories

  • Which Branches Should Be Protected?

  • Protecting the Main Branch

  • Protecting Release Branches

  • Protecting Hotfix Branches

  • Applying Consistent Policies Across Repositories

  • Require Build Validation Before Code Reaches Production

  • Use Role-Based Access Control Instead of Individual Permissions

• Automate Repository Provisioning from Day One

  • Why Repository Templates Matter

  • Automating Repository Creation with Terraform

• Creating Repositories Through the Azure DevOps REST API

  • Automatically Creating CI/CD Pipelines

  • Automatically Applying Branch Policies

  • Example: Automating the Provisioning of a New TypeScript Service

  • Think of Repository Creation as Product Manufacturing

• Monitor Repository Health Before Performance Degrades

• Keep Repository Sizes Under Control

• Use Git LFS for Large Assets

  • Automate Git LFS Setup for New Repositories

  • scripts/setup-git-lfs.sh

  • Automate Git LFS Check in Azure Pipelines

• Clean Repositories Regularly

• Improve Developer Experience with Shallow Cloning

• Example Architecture for a Modern TypeScript Platform

• Final Thoughts

Why Repository Organization Becomes a Scaling Problem

Many teams underestimate repository management because they only see today's requirements.

A startup with one application may only have:

frontend
backend
database

Everything seems manageable.

Fast forward two years and the same company might have:

customer-portal-web
customer-portal-api
billing-service
notification-service
auth-service
analytics-service
mobile-api
shared-components
design-system
internal-tools

The problem is no longer writing code. It becomes managing code.

Without clear standards, organizations typically experience:

• Repositories with unclear ownership

• Different branching strategies

• Security permission sprawl

• Duplicate CI/CD configurations

• Slower developer onboarding

• Bloated Git repositories

• Inconsistent documentation

• Difficult compliance audits

Repository management is ultimately about reducing operational friction. Every repository should be easy to understand, easy to secure, easy to maintain, and easy to scale.

Build an Ownership-Driven Repository Structure

One of the biggest mistakes teams make is organizing repositories like folders.

Repositories shouldn't exist because a folder was needed. They should exist because ownership, deployment, security, or lifecycle boundaries require them.

When deciding whether a new repository should be created, ask:

• Who owns this code?

• How is it deployed?

• Who can access it?

• Does it version independently?

• Does it require different security controls?

If the answers differ significantly from another codebase, it likely deserves its own repository.

Think about repositories as business assets rather than technical containers.

Choosing Between a Mono-Repo and Multi-Repo Strategy

One of the first decisions you'll make is whether to store everything in one repository or split projects across multiple repositories.

There's no universal answer.

The correct choice depends on ownership and deployment requirements.

When a Mono-Repo Makes Sense

A mono-repository works well when the same team owns everything and components are tightly coupled.

Example:

company-platform/
│
├── frontend/
├── backend/
├── shared-ui/
├── docs/
└── infrastructure/

This structure simplifies:

• Dependency management

• Refactoring

• Shared tooling

• Coordinated releases

But as teams grow, mono-repositories often become difficult to govern because everyone shares the same repository boundaries.

When a Multi-Repo Strategy Works Better

Large organizations generally benefit from multiple repositories.

Consider a SaaS platform built with Node.js, TypeScript, and React. Instead of one massive repository, separate repositories may look like:

customer-portal-web
customer-portal-api
billing-service
notification-service
shared-ui-library
authentication-service

Each repository can then:

• Have its own release cycle

• Maintain separate permissions

• Deploy independently

• Scale independently

This approach aligns naturally with modern microservice architectures.

Separate Azure DevOps Projects by Business Boundaries

Many teams create repositories correctly but place everything inside a single Azure DevOps Project.

That works initially, but becomes problematic later.

Azure DevOps Projects should represent organizational boundaries.

For example:

Customer Platform
├── customer-web
├── customer-api
├── mobile-api

Internal Systems
├── hr-system
├── payroll-api

Developer Platform
├── shared-components
├── infrastructure-tools

This structure improves security management, reporting, compliance, repository governance, and team autonomy.

A project should represent a logical business domain rather than a random collection of repositories.

Create Naming Standards Before Repository Growth Happens

Naming conventions often feel unimportant...until you have 500 repositories.

Without naming standards, developers waste time searching for repositories and understanding ownership.

Bad examples:

backend
backend-v2
new-api
test-project
final-final-api

Good examples:

sales-order-service
sales-payment-api
customer-auth-service
platform-notification-service
marketing-website

A simple convention works well:

[domain]-[service]

For example:

billing-payment-service
billing-invoice-service
customer-auth-service

Immediately, everyone understands the business domain, the service purpose, and the repository ownership.

Good naming reduces confusion before confusion appears.

Implement Cross-Repository Policies Instead of Managing Repositories Individually

This is where many Azure DevOps environments begin to fail.

Imagine managing 100 repositories, 300 developers, and 20 teams. Would you manually configure branch policies for every repository?

Of course not.

Yet many organizations still do exactly that. And the result is inconsistency.

Some repositories require pull requests, while others allow direct commits. Some require successful builds, while others don't.

Over time, repository quality becomes impossible to enforce consistently.

The solution is cross-repository governance. Instead of treating repositories individually, treat policies as organizational standards.

Enforce Branch Policies Across Multiple Repositories

As engineering teams grow, maintaining consistent code quality becomes increasingly difficult.

A repository with five developers may survive without strict governance. But a repository ecosystem with hundreds of developers and dozens of services can't.

Without branch protection, developers can:

• Push directly to production branches

• Bypass code reviews

• Merge untested code

• Accidentally introduce breaking changes

• Deploy features without proper traceability

Azure DevOps branch policies help prevent these problems by enforcing organizational standards before code can be merged.

Rather than configuring policies individually for every repository, organizations should establish a standard branch strategy and apply protections consistently across repositories.

Which Branches Should Be Protected?

Not every branch requires the same level of protection.

Most teams focus on protecting branches that directly affect production releases or customer-facing environments.

A common strategy is protecting:

main
release/*
hotfix/*

Let's look at why each branch matters.

Protecting the Main Branch

The main branch typically represents the most stable version of your application.

For a Node.js and TypeScript application, code in main is often what gets deployed to production.

Example:

main
│
├── Latest production-ready code
├── Passed automated testing
└── Approved through code review

Because this branch directly impacts customers, developers should never push changes directly into it. Instead, all changes should flow through pull requests.

Recommended protections include:

• Require pull requests

• Require reviewer approval

• Require successful build validation

• Require linked work items

• Block force pushes

This ensures every change entering production has been reviewed and validated.

Protecting Release Branches

Release branches are often used to prepare production deployments.

Example:

release/v1.0
release/v1.1
release/v2.0

These branches usually contain code that's undergoing final testing before deployment.

Without protection, developers may accidentally introduce new features or untested changes while the release is being stabilized.

Recommended protections include:

release/*

• Require pull requests

• Require QA approval

• Require successful test execution

• Restrict direct commits

This keeps release branches stable and predictable.

Protecting Hotfix Branches

Hotfix branches are used when urgent production issues need immediate attention.

Examples include:

• Payment failures

• Authentication outages

• Security vulnerabilities

• Critical application bugs

Example:

hotfix/payment-timeout
hotfix/authentication-error

Because hotfixes are usually created under pressure, they're more likely to introduce mistakes.

Teams often want to deploy quickly and skip review processes. But that's precisely why protection matters.

Recommended protections include:

hotfix/*

• Require at least one reviewer

• Require automated testing

• Track changes through work items

• Restrict direct pushes

Even during emergencies, code quality standards should remain intact.

Applying Consistent Policies Across Repositories

Imagine an organization managing the following:

customer-portal-api
billing-service
notification-service
authentication-service
reporting-service

If every repository has different branch rules, developers become confused and governance becomes difficult.

Instead, the team should establish repository-wide standards:

main       → 2 reviewers + successful build
release/*  → QA approval + successful build
hotfix/*   → 1 reviewer + successful build

This creates predictable workflows regardless of which repository a developer is working in.

A developer moving from the billing service to the notification service already understands the merge process because the same rules apply everywhere.

By enforcing branch protections consistently across Azure Repositories, organizations reduce production incidents, improve code quality, strengthen security, and create a development workflow that scales alongside engineering growth.

Require Build Validation Before Code Reaches Production

Many bugs reach production because code is reviewed but never tested automatically. Build validation closes that gap.

For a TypeScript Node.js project, an Azure Pipeline might run:

trigger:
  - main

pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '20.x'

  - script: npm install

  - script: npm run lint

  - script: npm run test

  - script: npm run build

This example uses an Ubuntu build agent. For most TypeScript, Node.js, React, and Tailwind CSS projects, Ubuntu is usually enough because the app isn't tied to a specific operating system.

But teams can also test across multiple operating systems. If so, replace imageName: ubuntu-latest with its respective operating system.

Example:

strategy:
  matrix:
    linux:
      imageName: ubuntu-latest
    windows:
      imageName: windows-latest
    mac:
      imageName: macOS-latest

pool:
  vmImage: $(imageName)

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '20.x'

  - script: npm install
  - script: npm run lint
  - script: npm run test
  - script: npm run build

Use this when the project needs to confirm that builds and tests pass on Linux, Windows, and macOS.

For a normal web app, Ubuntu is fine. For desktop apps, CLI tools, cross-platform packages, or mobile-related builds, multi-OS testing is better.

This pipeline ensures that dependencies install correctly, linting passes, tests pass, and production builds succeed. All before the pull request is merged.

Developers stop arguing about code quality because the pipeline enforces it automatically.

Use Role-Based Access Control Instead of Individual Permissions

Managing individual user permissions becomes impossible at scale. Imagine manually configuring 500 developers.

Instead, create groups.

Example:

Frontend Developers
Backend Developers
DevOps Engineers
QA Team
Project Administrators

Then map those groups to Azure DevOps roles:

Frontend Developers → Contributor
QA Team → Reader
DevOps Engineers → Administrator

This creates consistency and significantly reduces administrative overhead.

When a developer joins or leaves a team, administrators only update group membership. Repository permissions remain unchanged.

Automate Repository Provisioning from Day One

As organizations grow, repository creation often becomes an overlooked source of technical debt.

In a small team, manually creating repositories may seem harmless. A developer creates a new repository through the Azure DevOps portal, adds a README file, configures a pipeline, and starts building features.

The problem emerges when this process is repeated hundreds of times across multiple teams.

One repository contains a README. Another does not.

One repository has branch protection rules. Another allows direct commits to production.

One repository includes a CI/CD pipeline. Another requires manual deployment.

Over time, every repository begins to look different. And this inconsistency creates operational overhead, security risks, and onboarding challenges.

The solution is to treat repository creation as an automated process rather than a manual task.

Instead of allowing developers to create repositories from scratch, organizations should establish a repository blueprint that automatically provisions repositories with predefined standards and configurations.

A newly created repository should automatically include:

README.md
CONTRIBUTING.md
CODEOWNERS
.gitignore
azure-pipelines.yml
docs/
src/
tests/

This ensures every project starts with the same foundation.

The goal is simple:

Every repository should be production-ready the moment it's created.

Why Repository Templates Matter

Imagine your organization has 150 repositories.

Without automation, every repository owner must remember to:

• Create documentation

• Configure branch policies

• Set up build pipelines

• Configure permissions

• Add security checks

• Establish folder structures

The likelihood of inconsistency becomes extremely high.

With repository templates, every new repository automatically inherits organizational standards.

For example, a Node.js and TypeScript repository template might look like:

customer-auth-service/
│
├── src/
│
├── tests/
│
├── docs/
│
├── README.md
│
├── CONTRIBUTING.md
│
├── .gitignore
│
├── package.json
│
├── tsconfig.json
│
└── azure-pipelines.yml

Developers can immediately begin working without spending time configuring project infrastructure.

Automating Repository Creation with Terraform

One of the most common approaches is using Terraform to provision Azure DevOps resources.

Instead of manually creating repositories through the Azure DevOps dashboard, teams define repository creation using Infrastructure as Code.

Example:

resource "azuredevops_project" "platform" {
    name = "Customer Platform" 
} 

resource "azuredevops_git_repository" "auth_service" {         project_id = azuredevops_project.platform.id 

    name = "customer-auth-service" 
    
    initialization { 
        init_type = "Clean" 
} 
    }

Let's break this down.

The first block creates an Azure DevOps Project named: Customer Platform

The second block automatically creates a Git repository called customer-auth-service. Running:

terraform apply

creates the repository without requiring anyone to use the Azure DevOps interface.

This approach becomes extremely valuable when managing dozens or hundreds of repositories.

Creating Repositories Through the Azure DevOps REST API

Terraform is excellent for infrastructure teams. But some organizations prefer using internal automation platforms.

Azure DevOps provides a REST API that allows repositories to be created programmatically.

Example:

curl -X POST \
https://dev.azure.com/{organization}/{project}/_apis/git/repositories?api-version=7.1 \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <PAT>" \
-d '{
  "name": "customer-auth-service"
}'

This request automatically creates a repository inside Azure DevOps.

Many organizations build internal portals where developers fill out a form:

Repository Name:
customer-auth-service

Project:
Customer Platform

Language:
TypeScript

Template:
Node.js API

The platform then calls Azure DevOps APIs behind the scenes and provisions everything automatically.

Automatically Creating CI/CD Pipelines

Repository creation shouldn't stop at source control. A repository without automation is incomplete.

For a TypeScript Node.js service, an Azure Pipeline template could automatically be included:

Trigger:

• main

pool: vmImage: ubuntu-latest

Steps:

• task: NodeTool@0 inputs: versionSpec: '20.x'

• script: npm install

• script: npm run lint

• script: npm run test

• script: npm run build

This pipeline automatically installs Node.js and dependencies, runs linting, executes tests, and builds the application.

Every new repository receives the same CI/CD standards with no manual setup required.

Automatically Applying Branch Policies

Repository automation should also include governance.

After a repository is created, automation can immediately configure:

• Pull request requirements

• Reviewer policies

• Build validation

• Merge restrictions

For example:

main branch 
│ 
├── Require 2 reviewers 
├── Require successful build 
├── Require linked work item 
└── Block direct commits

Instead of relying on developers to remember these settings, automation guarantees every repository follows organizational policies from day one.

Example: Automating the Provisioning of a New TypeScript Service

Imagine that a developer requests a new service called:

customer-notification-service

A provisioning workflow could automatically:

1. Create the repository.

2. Add README.md

3. Add TypeScript project structure.

4. Configure Azure Pipeline.

5. Apply branch protection rules.

6. Assign ownership groups.

7. Configure security permissions.

8. Register monitoring and deployment pipelines.

Within minutes, the repository is ready for development. There's no manual setup, forgotten configurations, or inconsistent standards.

Think of Repository Creation as Product Manufacturing

A useful way to think about repository provisioning is through manufacturing. A factory doesn't build every car from scratch. It follows a repeatable process.

Repository creation should work the same way.

Every repository should come off the production line with:

• Standardized structure

• Security controls

• CI/CD pipelines

• Documentation

• Governance policies

Automation ensures that the hundredth repository is just as well-configured as the first.

As organizations scale, this consistency becomes one of the most important factors in maintaining repository quality, reducing operational overhead, and enabling engineering teams to move faster without sacrificing governance.

Monitor Repository Health Before Performance Degrades

Repository health is often ignored until developers complain. By then, the repository is already bloated.

Azure Repos provides repository insights that help identify:

• Large repositories

• Large files

• Excessive commit activity

• Storage growth

Regular monitoring prevents performance issues before they impact developers.

Keep Repository Sizes Under Control

Azure Repos supports repositories up to 250 GB. That doesn't mean repositories should approach that size. Performance usually begins degrading long before then. So watch out for huge binaries, large media assets, generated files, and build artifacts.

Never store:

.zip
.rar
.iso
.exe
.mp4
.psd

inside source repositories.

Instead, use Azure Blob Storage, package registries, Git LFS, and Azure Artifacts.

Source control should store source code. Nothing more!

Use Git LFS for Large Assets

Sometimes large files are unavoidable.

A normal web app shouldn't store heavy files in Git. But some projects need them. For example, a design system may include Photoshop files. A media platform may include sample videos. A game project may include textures, audio, and 3D assets.

The problem is that Git was built for source code, not large binary files. When you commit a large file directly into Git, it becomes part of the repository history. Even if you delete the file later, the old version still stays in history unless you rewrite it.

That is how repositories become slow and heavy over time.

Git LFS, which means Git Large File Storage, solves this by storing large files outside the normal Git history. Your repository keeps a small pointer file, while the real large file is stored separately.

Example – track large design files:

git lfs install

git lfs track "*.psd"
git lfs track "*.fig"
git lfs track "*.mp4"
git lfs track "*.zip"

git add .gitattributes
git commit -m "Configure Git LFS"

The .gitattributes file will look like this:

*.psd filter=lfs diff=lfs merge=lfs -text
*.fig filter=lfs diff=lfs merge=lfs -text
*.mp4 filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text

This tells Git: “Whenever these file types are added, store them with Git LFS instead of normal Git history.”

Example – add a large file after configuring Git LFS:

git add assets/design/homepage.psd
git commit -m "Add homepage design source file"
git push

Now the large .psd file is handled by Git LFS.

Automate Git LFS Setup for New Repositories

At scale, you shouldn't rely on every developer to remember these commands manually. Create a setup script that runs when a new repository is created.

scripts/setup-git-lfs.sh

#!/bin/bash

git lfs install

git lfs track "*.psd"
git lfs track "*.fig"
git lfs track "*.mp4"
git lfs track "*.mov"
git lfs track "*.zip"
git lfs track "*.ai"

git add .gitattributes
git commit -m "Configure Git LFS for large assets"

Run it with:

bash scripts/setup-git-lfs.sh

Automate Git LFS Check in Azure Pipelines

You can also stop large files from entering the repository without Git LFS.

azure-pipelines.yml

trigger:
  - main

pool:
  vmImage: ubuntu-latest

steps:
  - checkout: self
    lfs: true

  - script: |
      echo "Checking for large files not tracked by Git LFS..."

      MAX_SIZE=10485760

      files=$(git ls-files)

      for file in $files; do
        if [ -f "$file" ]; then
          size=\((stat -c%s "\)file")

          if [ "\(size" -gt "\)MAX_SIZE" ]; then
            if ! git check-attr filter -- "$file" | grep -q "filter: lfs"; then
              echo "Large file not tracked by Git LFS: $file"
              exit 1
            fi
          fi
        fi
      done

      echo "Large file check passed."
    displayName: "Check large files use Git LFS"

This pipeline checks files larger than 10MB. If a large file isn't tracked by Git LFS, the build fails.

That's the automation you want at scale. It prevents repository bloat before it enters the codebase.

Clean Repositories Regularly

Repository maintenance isn't a one-time task. You should treat repositories like production systems and schedule periodic reviews.

Remove:

• Stale branches

• Unused pipelines

• Obsolete repositories

• Outdated documentation

If secrets or large files are accidentally committed, use:

git-filter-repo

to permanently remove them from history.

This modern approach is significantly faster and safer than older Git history rewriting tools.

Improve Developer Experience with Shallow Cloning

Large repositories often contain years of history. And most developers don't need all of it.

A shallow clone downloads only recent history.

Example:

git clone --depth 1 https://dev.azure.com/company/project/repository

Benefits include:

• Faster onboarding

• Faster cloning

• Reduced storage consumption

• Lower network usage

Small optimizations become significant when hundreds of developers interact with repositories daily.

Example Architecture for a Modern TypeScript Platform

Imagine a SaaS platform built using:

• TypeScript

• Node.js

• React

• Tailwind CSS

• Azure DevOps

A scalable repository structure could look like:

customer-portal-web
customer-portal-api
billing-service
notification-service
shared-ui-library
infrastructure-templates
developer-documentation

Each repository serves a distinct purpose, can deploy independently, and can scale independently.

Most importantly, ownership remains clear. That clarity is what makes large engineering organizations successful.

Final Thoughts

The biggest mistake organizations make with Azure Repositories is assuming repository management is simply a storage problem.

It isn't.

Repository management is an organizational problem. The repositories you create today determine how efficiently teams collaborate tomorrow.

A scalable Azure Repos strategy starts with ownership-driven structures, consistent naming conventions, centralized governance, automated repository provisioning, repository health monitoring, and disciplined maintenance practices.

The goal isn't to manage more repositories. The goal is to create a repository ecosystem that remains secure, maintainable, and performant regardless of how large your engineering organization becomes.

The earlier you establish these standards, the easier it becomes to scale Azure DevOps without accumulating the repository debt that slows so many growing development teams.

Six months later, you have fourteen microservices, each holding their own API keys, writing their own retry logic, and failing in their own unique ways.

Nobody knows how much you're spending on tokens or which service is hammering the rate limit. And when OpenAI goes down, everything goes down with it.

That scenario plays out across engineering teams every single day, and the root cause is almost always the same: moving fast with LLMs while skipping the infrastructure thinking that holds everything together at scale.

Fortunately, a well-established architectural pattern solves exactly these problems. If you already run Kubernetes, you're more than halfway to implementing it. That pattern is called the LLM Gateway Pattern, and this article walks you through what it is, why it matters, and how to put it into practice.

Table of Contents

• What Is the LLM Gateway Pattern?

  • How It Works

• The Problem Without a Gateway

• Deploying an LLM Gateway on Kubernetes

  • Storing API Keys Securely

  • Defining Routing Rules in a ConfigMap

  • Scaling the Gateway

  • Wiring Up Observability

• Features of an LLM Gateway

  • Multi-Provider Routing

  • Semantic Caching

  • Rate Limiting Per Consumer

  • Fallback and Failover

  • Token Usage Tracking

• Wrapping Up

What Is the LLM Gateway Pattern?

The LLM Gateway Pattern is an architectural approach where all LLM API traffic from your applications flows through a single, centralized proxy service before reaching any external provider. Think of it as the AI equivalent of an API gateway, except it's purpose-built for the unique challenges that come with language models: token budgets, streaming responses, model routing, semantic caching, and multi-provider fallback.

Instead of every service in your cluster talking directly to OpenAI or Anthropic, they all talk to one internal gateway. That gateway handles authentication, routing, rate limiting, logging, and failover. Your application services stay clean and focused on business logic, while the gateway takes on all the messy operational concerns of working with LLMs at scale.

The pattern itself is not new in concept. Engineers have used API gateways for years to manage REST traffic. What makes LLM gateways distinct is that they understand the specific shape of LLM requests, including token counts, model parameters, prompt structure, and streaming semantics.

How It Works

The core components of an LLM Gateway on Kubernetes are straightforward. Here is the high-level flow:

App Pods send requests to the gateway using a standard OpenAI-compatible API format. Because of this, most existing LLM client libraries work without modification — you just change the base URL to point at your internal gateway service.

The Gateway Service receives each incoming request, authenticates the caller, applies any configured rate limits, checks the cache, selects the appropriate upstream provider based on routing rules, and forwards the request. On the way back, it logs token usage and latency before returning the response to the caller.

ConfigMap holds the routing rules. Which model should handle requests tagged as fast? Which provider should the system fall back to if the primary one is unavailable? All of this lives in configuration, not code, so you can update routing behaviour without redeploying anything.

Secrets hold the actual API keys for each provider. The gateway is the only service in the cluster that needs access to them. Application pods never touch provider credentials directly.

Provider endpoints are the actual LLM APIs: OpenAI, Anthropic, a self-hosted vLLM instance running in your cluster, or any other provider that exposes an OpenAI-compatible interface.

The Problem Without a Gateway

To appreciate why this pattern matters, it helps to look at what happens when you skip it.

1. Scattered Secrets and No Central Control

Every service that calls an LLM needs an API key. In Kubernetes, this usually means creating a Secret per namespace or per deployment.

When that key rotates or gets compromised, you're hunting through dozens of manifests to update it. There's no single place to revoke access or audit who is calling what.

2. No Visibility into Cost or Usage

LLM APIs charge per token. Without a centralized layer collecting usage data, you have no reliable way to know which service is responsible for that spike in your monthly bill.

3. Provider Lock-in at the Application Level

When you hardcode https://api.openai.com into your service, switching to a different provider or routing certain requests to a cheaper model becomes a code change. You need to redeploy your application just to change which model handles a request type.

4. No Caching

Many LLM applications send semantically similar or identical prompts repeatedly. Without a shared caching layer, each one incurs full token costs and full latency. The savings from even basic caching can be significant.

All of these problems compound as your team grows and more services start calling LLMs. The gateway pattern cuts through all of them in one architectural decision.

Deploying an LLM Gateway on Kubernetes

There are several tools that can serve as an LLM gateway in a Kubernetes environment, including LiteLLM Proxy, Portkey, OpenRouter, and Envoy with custom filters.

For the rest of this walkthrough, we'll use LiteLLM Proxy. It ships with a Helm chart, supports over a hundred models across all major providers, and comes with a management UI that makes initial configuration straightforward.

Storing API Keys Securely

Start by creating a Kubernetes Secret that holds your provider API keys. Your gateway pods will consume these credentials as environment variables, which means no provider key ever needs to live inside your application containers:

apiVersion: v1
kind: Secret
metadata:
  name: llm-gateway-secrets
  namespace: ai-platform
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-..."
  ANTHROPIC_API_KEY: "sk-ant-..."

Defining Routing Rules in a ConfigMap

The routing configuration tells the gateway which models are available and how to reach each one. Keeping this in a ConfigMap means you can update your routing rules without touching a single line of application code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-gateway-config
  namespace: ai-platform
data:
  config.yaml: |
    model_list:
      - model_name: gpt-4o
        litellm_params:
          model: openai/gpt-4o
          api_key: os.environ/OPENAI_API_KEY
      - model_name: claude-sonnet
        litellm_params:
          model: anthropic/claude-sonnet-4-20250514
          api_key: os.environ/ANTHROPIC_API_KEY
      - model_name: fast
        litellm_params:
          model: openai/gpt-4o-mini
          api_key: os.environ/OPENAI_API_KEY

With this configuration in place, any application in your cluster can reach the gateway at http://llm-gateway.ai-platform.svc.cluster.local using the standard OpenAI client format, regardless of which actual provider sits behind it.

Scaling the Gateway

Because the gateway is stateless, horizontal scaling is straightforward. You can attach a HorizontalPodAutoscaler to scale based on CPU utilization or request rate:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-gateway-hpa
  namespace: ai-platform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-gateway
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Wiring Up Observability

A gateway you can't observe is a gateway you can't trust, so wiring up monitoring before you go to production is worth the extra hour it takes.

LiteLLM exposes a /metrics endpoint in Prometheus format. You can scrape it with a standard ServiceMonitor if you run the Prometheus Operator, or configure Prometheus directly to target the gateway service.

The metrics that matter most in day-to-day operations are token throughput per model, request latency percentiles, error rates per provider, and cache hit ratio.

Once Prometheus is collecting that data, you can build Grafana dashboards that show token spend broken down by caller, model, and time period. This gives engineering managers and finance teams the cost visibility they've been asking for, and it takes surprisingly little effort to set up once the metrics pipeline is in place.

If you run an OpenTelemetry collector in your cluster, you can also configure the gateway to emit trace spans for every LLM request. This lets you see the full latency breakdown from the moment a user action triggers a call in your application all the way through to the provider response. So when something is slow, you can tell immediately whether the bottleneck sits in your service, the gateway, or upstream with the provider.

Features of an LLM Gateway

Not all gateway implementations are equal, so as your needs grow, these are the core capabilities worth evaluating.

Multi-Provider Routing

A well-built gateway routes requests to different providers based on declarative, configurable rules that live entirely outside your application code. This means that changing a model never requires a redeployment.

Semantic Caching

Rather than only caching byte-for-byte identical prompts, a semantic cache uses embedding similarity to recognise when two different prompts are asking essentially the same thing. This can cut redundant API calls dramatically.

Rate Limiting Per Consumer

The gateway should let you set token budgets and request limits per team, per namespace, or per application, so no single runaway service can starve the rest of your cluster or drive up costs unchecked.

Fallback and Failover

When a primary provider fails or exceeds acceptable latency thresholds, the gateway should automatically retry against a configured fallback. This centralizes logic that is notoriously hard to get right inside individual services.

Token Usage Tracking

Every request should produce a detailed usage record capturing input tokens, output tokens, model, caller identity, and latency. This gives engineering managers the clear, actionable picture of AI spending they need.

Wrapping Up

The LLM Gateway Pattern solves a set of operational problems that every team building on language models at scale will eventually run into. Scattered secrets, invisible costs, inconsistent failure handling, and provider lock-in are all symptoms of the same underlying issue: infrastructure concerns leaking into services that shouldn't have to deal with them.

A centralized gateway on Kubernetes gives your application teams a stable, provider-agnostic interface while giving your platform team the visibility and controls they need to manage cost and reliability effectively. When a provider goes down in the middle of the night, your configured fallback kicks in automatically instead of someone waking up to a page.

Start with LiteLLM Proxy, wire up the Prometheus metrics, build a simple Grafana dashboard, and watch how quickly the pattern pays for itself. Once you have seen what centralized LLM traffic management looks like in practice, it becomes very hard to go back to doing it any other way.

Your production will keep using a managed S3 / Cloudflare R2 / Hetzner Object Storage, while every staging upload, download, and pre-signed URL goes to your own server for free.

Table of Contents

• 1. Why Self‑Host Object Storage on Staging?

• 2. The Architecture: Production vs. Staging

• 3. Prerequisites

• 4. Step 1 — DNS: Point Your Domains to the Staging Server

• 5. Step 2 — Run MinIO with Docker Compose

• 6. Step 3 — Expose MinIO over HTTPS with Traefik

• 7. Step 4 — Create the Bucket and Access Keys

• 8. Step 5 — Configure Your App to Use MinIO on Staging Only

• 9. Step 6 — Upload Files (3 Ways)

• 10. Step 7 — Generate Presigned URLs (PUT and GET)

• 11. Step 8 — Get Public URLs for Documents

• 12. Step 9 — Lock Down CORS, Lifecycle, and Security

• 13. Step 10 — Backups and Monitoring

• 14. Troubleshooting Cheat Sheet

• 15. Wrapping Up

1. Why Self‑Host Object Storage on Staging?

If your app handles documents — PDFs, profile pictures, application transcripts, recordings — every test upload your QA team makes costs real money on AWS S3, Cloudflare R2, or Hetzner Object Storage. The price isn't huge per file, but staging is where you:

• run automated end‑to‑end tests that upload thousands of dummy files,

• reset databases nightly (which leaves orphan objects behind),

• let developers experiment with broken code that re‑uploads the same files,

• and hold months of test data nobody ever deletes.

In production those costs are justified. Managed storage gives you replication, availability, and someone else's pager. In staging, those costs are pure waste.

MinIO is a free, open‑source, S3‑compatible object server. Same API, same SDKs, same presigned URLs, same mc/aws s3 CLIs — but running on your own VPS, billed at $0 per gigabyte. Point your staging app at MinIO, point your production app at S3/R2, and the only thing that changes is an environment variable.

The result: identical code paths in both environments, zero storage bill on staging, and a nice fallback if your cloud provider ever has an outage.

2. The Architecture: Production vs. Staging

In real-world applications, you usually don’t want your development or staging environment writing directly to production storage.

A common and cost-effective setup is:

• Production: managed cloud object storage

• Staging / Development: self-hosted S3-compatible storage

The good part is that your application code doesn't need to change.

As long as both services are S3-compatible, the same SDK and upload logic work everywhere. Only the environment variables differ.

High-Level Architecture

The above diagram illustrates how the same application can communicate with different storage providers depending on the deployment environment.

In the production environment, uploads are stored in a managed object storage service such as AWS S3, Cloudflare R2, or Hetzner Object Storage. These services handle durability, scalability, backups, and infrastructure management.

In the staging environment, uploads are directed to a self-hosted MinIO instance running inside Docker on a VPS. MinIO implements the S3 API, making it behave similarly to production storage while keeping costs low.

Because both storage systems are S3-compatible, the application uses the same upload logic in every environment. The only difference is the configuration provided through environment variables.

Why This Architecture Is Useful

This setup gives you:

• A cheap staging environment

• Production-like testing

• Zero storage vendor lock-in

• The ability to switch providers without rewriting application code

Because both environments speak the S3 protocol, your upload logic remains identical.

Example Environment Variables

Your application only reads environment variables like these:

S3_ENDPOINT=
S3_REGION=
S3_ACCESS_KEY=
S3_SECRET_KEY=
S3_BUCKET=

Switch the values, and the exact same application now uploads files to a different backend.

Production Storage Example

In production, you typically use managed object storage providers such as:

• AWS S3

• Cloudflare R2

• Hetzner Object Storage

Example:

S3_ENDPOINT=https://<region>.r2.cloudflarestorage.com

The benefits are that it's highly scalable, globally available, durable, has managed backups, and doesn't have infrastructure maintenance.

Staging Environment Example

For staging, a lightweight self-hosted MinIO container is often enough.

Next.js App
     ↓
MinIO Container (inside Docker on VPS)

Example domains:

This allows you to:

• Test uploads safely

• Avoid production storage costs

• Reproduce production-like behavior locally

3. Prerequisites

You'll need:

• A Linux VPS (Hetzner, DigitalOcean, Contabo, OVH — anything with a public IP).

• Two A records pointing at that IP (we'll register them next).

• Docker + Docker Compose v2.

• Traefik v2 in front, with Let's Encrypt configured (any reverse proxy works – the labels below are Traefik's flavor).

• Open ports 80 and 443 on the firewall for Let's Encrypt + HTTPS.

• ~10 GB free disk for the MinIO data volume to start.

If Docker isn't installed:

curl -fsSL https://get.docker.com | sh
sudo apt-get install -y docker-compose-plugin
docker --version && docker compose version

4. Step 1 — DNS: Point Your Domains to the Staging Server

In your DNS provider (Cloudflare, Route 53, Namecheap, and so on), create two A records pointing at your staging server's public IP:

minio-staging.domain.com           A    203.0.113.45
minio-console-staging.domain.com   A    203.0.113.45

If you use Cloudflare, set the proxy status to DNS only (gray cloud) for minio-staging.*. Cloudflare's free plan caps uploads at 100 MB, and you don't want it stripping S3 signing headers. The console subdomain can stay proxied if you want a WAF in front of it.

Wait a minute and verify:

dig +short minio-staging.domain.com
# 203.0.113.45

5. Step 2 — Run MinIO with Docker Compose

Add this service to your staging compose file (docker-compose.staging.yml). MinIO is just one container — the disk is mounted as a Docker volume so data survives upgrades.

# docker-compose.staging.yml
networks:
  proxy:
    external: true
    name: proxy
  internal:
    name: internal

volumes:
  minio-data:

services:
  minio:
    image: minio/minio:latest
    container_name: minio-staging
    restart: unless-stopped
    environment:
      - MINIO_ROOT_USER=${MINIO_ROOT_USER:-admin}
      - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-change-me-please}
      # Tell MinIO which public domain to sign URLs with
      - MINIO_SERVER_URL=https://minio-staging.domain.com
      - MINIO_BROWSER_REDIRECT_URL=https://minio-console-staging.domain.com
    command: server /data --console-address ":9001"
    volumes:
      - minio-data:/data
    networks:
      - proxy
      - internal
    ports:
      - "9000:9000"  # S3 API
      - "9001:9001"  # Web console
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

Two things deserve attention:

• MINIO_SERVER_URL is the secret sauce. Without it, MinIO signs presigned URLs using its internal hostname (http://minio:9000), which then fails verification when the browser hits the public domain. Set it to the exact HTTPS URL clients will use.

• MINIO_BROWSER_REDIRECT_URL does the same for the web console (login redirects, OIDC callbacks, and so on).

Bring it up:

docker compose -f docker-compose.staging.yml up -d minio
docker compose -f docker-compose.staging.yml logs -f minio

You should see API: http://... and Console: http://... lines.

6. Step 3 — Expose MinIO over HTTPS with Traefik

We don't expose ports 9000/9001 to the world directly — Traefik does that for us, terminating TLS with a free Let's Encrypt certificate.

Add these labels to the minio service:

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # ---- S3 API (port 9000) ----
      - "traefik.http.routers.minio-staging.rule=Host(`minio-staging.domain.com`)"
      - "traefik.http.routers.minio-staging.entrypoints=websecure"
      - "traefik.http.routers.minio-staging.tls.certresolver=letsencrypt"
      - "traefik.http.routers.minio-staging.service=minio-staging"
      - "traefik.http.services.minio-staging.loadbalancer.server.port=9000"

      # ---- Web Console (port 9001) ----
      - "traefik.http.routers.minio-console-staging.rule=Host(`minio-console-staging.domain.com`)"
      - "traefik.http.routers.minio-console-staging.entrypoints=websecure"
      - "traefik.http.routers.minio-console-staging.tls.certresolver=letsencrypt"
      - "traefik.http.routers.minio-console-staging.service=minio-console-staging"
      - "traefik.http.services.minio-console-staging.loadbalancer.server.port=9001"

You also need an entrypoint for :443 and a certificatesresolver named letsencrypt. Here's the minimum Traefik config (traefik.staging.yml):

api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: web
      email: admin@domain.com
      storage: /etc/traefik/acme.json

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

Restart and watch the cert get issued:

docker compose -f docker-compose.staging.yml up -d
docker compose -f docker-compose.staging.yml logs -f traefik | grep -i acme

Sanity check from your laptop:

curl -I https://minio-staging.domain.com/minio/health/live
# HTTP/2 200

You can now log in to the web console at https://minio-console-staging.domain.com with admin / change-me-please.

Important upload size tweak: if you're behind Cloudflare or NGINX in front of Traefik, raise the request body limit. Traefik itself has no default limit, but Cloudflare's free plan refuses anything over 100 MB. For self‑hosted edge proxies, set client_max_body_size 0; (NGINX) or the equivalent.

7. Step 4 — Create the Bucket and Access Keys

Anything that speaks S3 can talk to MinIO. The easiest tool is mc (the official MinIO client), shipped inside the same image.

7.1 Connect mc to your server

docker exec -it minio-staging \
  mc alias set local http://localhost:9000 admin change-me-please

7.2 Create a bucket

docker exec -it minio-staging mc mb local/domain-files-staging

7.3 Choose a bucket policy

You have three choices, so just pick based on what you store:

Set one:

# Private (recommended for documents)
docker exec -it minio-staging \
  mc anonymous set none local/domain-files-staging

# OR public read for static assets only:
docker exec -it minio-staging \
  mc anonymous set download local/domain-files-staging

7.4 Create a dedicated app user (don't use root keys!)

The admin account can wipe everything. Make a least‑privilege user for your app:

docker exec -it minio-staging mc admin user add local \
  domain-app a-long-random-secret-key

# Attach the built-in read/write policy, scoped to one bucket via JSON:
cat > /tmp/policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": [
        "arn:aws:s3:::domain-files-staging",
        "arn:aws:s3:::domain-files-staging/*"
      ]
    }
  ]
}
EOF

docker cp /tmp/policy.json minio-staging:/tmp/policy.json
docker exec -it minio-staging \
  mc admin policy create local domain-rw /tmp/policy.json
docker exec -it minio-staging \
  mc admin policy attach local domain-rw --user domain-app

Save those two values — they are your S3_ACCESS_KEY and S3_SECRET_KEY.

8. Step 5 — Configure Your App to Use MinIO on Staging Only

The trick to "MinIO in staging, real S3 in prod" is to use the same S3 client in your code and only swap the env vars.

Your staging.env (loaded by your staging compose stack):

# ---- Staging: self-hosted MinIO ----
STORAGE_ENABLED=true
S3_ENDPOINT=https://minio-staging.domain.com
S3_PUBLIC_ENDPOINT=https://minio-staging.domain.com
S3_BUCKET=domain-files-staging
S3_ACCESS_KEY=domain-app
S3_SECRET_KEY=a-long-random-secret-key
S3_REGION=us-east-1
S3_FORCE_PATH_STYLE=true

Your production.env:

# ---- Production: Cloudflare R2 ----
STORAGE_ENABLED=true
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
S3_PUBLIC_ENDPOINT=https://files.domain.com
S3_BUCKET=domain-files
S3_ACCESS_KEY=<r2-access-key>
S3_SECRET_KEY=<r2-secret-key>
S3_REGION=auto
S3_FORCE_PATH_STYLE=true

S3_FORCE_PATH_STYLE=true is critical for both MinIO and R2/Hetzner. Without it, the SDK tries https://bucket.minio-staging.domain.com (virtual‑host style), which won't resolve.

Now in your application code (Node.js example using AWS SDK v3):

// src/lib/s3.js
import { S3Client } from "@aws-sdk/client-s3";

export const s3 = new S3Client({
  endpoint: process.env.S3_ENDPOINT,
  region: process.env.S3_REGION,
  credentials: {
    accessKeyId: process.env.S3_ACCESS_KEY,
    secretAccessKey: process.env.S3_SECRET_KEY,
  },
  forcePathStyle: process.env.S3_FORCE_PATH_STYLE === "true",
});

export const BUCKET = process.env.S3_BUCKET;
export const PUBLIC_ENDPOINT = process.env.S3_PUBLIC_ENDPOINT;

The same s3 instance now talks to MinIO on staging and to R2 in production with no code change.

9. Step 6 — Upload Files (3 Ways)

9.1 From a server (best for trusted backends)

import { PutObjectCommand } from "@aws-sdk/client-s3";
import { s3, BUCKET } from "./lib/s3.js";
import { readFile } from "node:fs/promises";

export async function uploadDocument(localPath, key, contentType) {
  const Body = await readFile(localPath);
  await s3.send(new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    Body,
    ContentType: contentType,
    // Optional: per-object metadata, useful for audits
    Metadata: { uploadedBy: "system", env: process.env.NODE_ENV },
  }));
  return key;
}

9.2 With the mc CLI (good for one‑off uploads / migrations)

mc alias set staging https://minio-staging.domain.com domain-app a-long-random-secret-key
mc cp ./report.pdf staging/domain-files-staging/reports/2026/report.pdf
mc ls staging/domain-files-staging --recursive

9.3 Directly from the browser via a presigned PUT URL

The recommended pattern for user uploads is: the file goes from the browser to MinIO with zero bytes touching your API server.

We'll cover this in detail next.

10. Step 7 — Generate Presigned URLs (PUT and GET)

A presigned URL is a regular HTTPS URL with a time‑limited signature in the query string. Anyone with the URL can do exactly the action it was signed for (PUT this object, or GET that object) for the next N minutes — and nothing else.

This is what makes "users upload directly to storage" safe.

10.1 Presigned PUT (for uploads)

// src/lib/presign.js
import { PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { s3, BUCKET } from "./s3.js";
import { randomUUID } from "node:crypto";

export async function presignUpload({ filename, contentType, userId }) {
  const key = `users/\({userId}/\){randomUUID()}-${filename}`;
  const cmd = new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    ContentType: contentType,
  });
  const uploadUrl = await getSignedUrl(s3, cmd, { expiresIn: 60 * 5 }); // 5 min
  return { uploadUrl, key };
}

Wire it to your API:

// POST /api/uploads/presign
app.post("/api/uploads/presign", requireAuth, async (req, res) => {
  const { filename, contentType } = req.body;
  const result = await presignUpload({
    filename,
    contentType,
    userId: req.user.id,
  });
  res.json(result); // { uploadUrl, key }
});

The browser uploads straight to MinIO:

// In your frontend
async function uploadFile(file) {
  const { uploadUrl, key } = await fetch("/api/uploads/presign", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ filename: file.name, contentType: file.type }),
  }).then(r => r.json());

  await fetch(uploadUrl, {
    method: "PUT",
    headers: { "Content-Type": file.type },
    body: file,
  });

  // Persist `key` in your DB so you can retrieve it later
  await fetch("/api/documents", {
    method: "POST",
    body: JSON.stringify({ key, originalName: file.name }),
  });
}

The Content-Type you send during PUT must match the one you signed with, or MinIO will reject the request with SignatureDoesNotMatch. This catches everyone the first time.

10.2 Presigned GET (for downloads)

Same idea, but with GetObjectCommand:

export async function presignDownload(key, expiresIn = 60 * 10) {
  const cmd = new GetObjectCommand({ Bucket: BUCKET, Key: key });
  return getSignedUrl(s3, cmd, { expiresIn });
}

A typical "view document" endpoint:

app.get("/api/documents/:id/url", requireAuth, async (req, res) => {
  const doc = await db.documents.findById(req.params.id);
  if (!doc || !canUserSee(req.user, doc)) return res.sendStatus(403);
  const url = await presignDownload(doc.key, 600);
  res.json({ url });
});

The frontend just opens that URL — the file streams from MinIO directly to the user.

10.3 Why presigned URLs beat "proxy through the API"

11. Step 8 — Get Public URLs for Documents

Sometimes you want a permanent, unauthenticated URL — for example public profile pictures.

If the bucket policy allows anonymous reads (mc anonymous set download …), the public URL pattern is:

https://minio-staging.domain.com/<bucket>/<key>

So users/42/avatar.png becomes:

https://minio-staging.domain.com/domain-files-staging/users/42/avatar.png

In code:

export function publicUrl(key) {
  return `\({process.env.S3_PUBLIC_ENDPOINT}/\){BUCKET}/${key}`;
}

For private buckets (most documents), don't use public URLs at all — always go through presignDownload(key) so you can re‑check authorization on every request and expire links.

12. Step 9 — Lock Down CORS, Lifecycle, and Security

12.1 Allow your frontend origins (CORS)

Browser uploads need CORS rules on the bucket. Drop this JSON via mc:

cat > /tmp/cors.json <<'EOF'
{
  "CORSRules": [
    {
      "AllowedOrigins": [
        "https://crm-staging.domain.com",
        "http://localhost:3000"
      ],
      "AllowedMethods": ["GET", "PUT", "POST", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3000
    }
  ]
}
EOF

docker cp /tmp/cors.json minio-staging:/tmp/cors.json
docker exec -it minio-staging \
  mc cors set local/domain-files-staging /tmp/cors.json

12.2 Auto‑delete old test files (lifecycle)

Staging accumulates junk. Tell MinIO to expire anything older than 30 days:

docker exec -it minio-staging \
  mc ilm rule add --expire-days 30 local/domain-files-staging

12.3 Encrypt at rest

docker exec -it minio-staging \
  mc encrypt set sse-s3 local/domain-files-staging

12.4 Hard rules

• Never ship MINIO_ROOT_USER=admin / MINIO_ROOT_PASSWORD=admin123 to a server reachable from the internet. Generate strong values and store them in your secret manager.

• The root account should be used only by mc admin, never by your app. The app uses a scoped IAM user (Step 7.4).

• Keep the console subdomain behind an IP allow‑list or basic auth via Traefik middleware if it's truly public.

• Rotate the app access keys at least every 90 days.

13. Step 10 — Backups and Monitoring

13.1 Backups: mirror to a cheap cold bucket weekly

Set up a tiny cron job that uses mc mirror to push to Backblaze B2, R2, or another cheap S3 endpoint:

mc alias set b2 https://s3.us-east-005.backblazeb2.com \(B2_KEY \)B2_SECRET
mc mirror --overwrite --remove \
  staging/domain-files-staging \
  b2/domain-staging-backup

Even at $6/TB/month this is essentially free for staging volumes.

13.2 Monitoring with Prometheus

MinIO exposes Prometheus metrics out of the box at /minio/v2/metrics/cluster. Scrape with:

scrape_configs:
  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    scheme: https
    static_configs:
      - targets: ["minio-staging.domain.com"]

If you have Grafana, import dashboard ID 13502 for an instant overview (capacity, request rates, latency, error counts).

14. Troubleshooting Cheat Sheet

Useful diagnostics:

# Check MinIO health
curl -I https://minio-staging.domain.com/minio/health/live

# List all objects in a bucket
docker exec -it minio-staging mc ls --recursive local/domain-files-staging

# Tail MinIO logs
docker compose -f docker-compose.staging.yml logs -f minio

# Decode a presigned URL to see what it was signed for
echo "<paste url>" | tr '&' '\n'

15. Wrapping Up

Here's what you have now:

• A free, S3‑compatible object store running on your own staging server.

• Real HTTPS on a real domain (https://minio-staging.domain.com), thanks to Traefik + Let's Encrypt.

• A scoped, least‑privilege application user — root keys stay locked away.

• The same exact code paths in staging and production. Switching between MinIO / R2 / Hetzner / AWS S3 is a four‑variable change in the env file.

• Presigned PUT URLs so users upload straight to storage, bypassing your API.

• Presigned GET URLs so private documents are short‑lived and authorization‑gated.

• Lifecycle rules that nuke old test files automatically.

• Optional weekly mirror to a cold backup bucket.

Production keeps running on managed storage where the SLA matters. Staging now costs you exactly $0 per month per gigabyte uploaded — and you can finally stop telling QA to "delete the test files when you're done."

Further Reading

• MinIO Documentation

• AWS SDK v3 — getSignedUrl

• Traefik v2 Docker provider

• S3 bucket policy reference

If this guide saved your team a few dollars, share it with another team that's still uploading test PDFs to a $90/month S3 bucket. Happy shipping.

Startup environments make this worse. The pressure to ship fast, the small team sizes, and the absence of senior engineers to review your decisions means mistakes happen quietly until they become outages, data loss events, or security incidents that cost the company thousands of dollars and weeks of recovery time.

This article is a direct breakdown of the ten most costly DevOps mistakes engineers make early in their careers at startups. For each mistake, you will get the real-world scenario, the business impact, and the concrete fix you can apply immediately.

Whether you are setting up your first production environment or auditing an existing one, this guide will help you build systems that are reliable, secure, and aligned with what the business actually needs.

Table of Contents

• Who This Article Is For

• Why Startups Are a Different Environment

• Mistake 1: Deploying Without Understanding What You're Deploying

• Mistake 2: Using Production as a Development Environment

• Mistake 3: Hardcoding Secrets and Credentials

• Mistake 4: Overengineering for Problems You Don't Have Yet

• Mistake 5: No Observability Before Launch

• Mistake 6: Treating Security as a Final Step

• Mistake 7: Manual Deployments in Production

• Mistake 8: No Disaster Recovery Plan

• Mistake 9: No Documentation or Runbooks

• Mistake 10: Solving Technical Problems Without Understanding the Business

• The System Thinking Framework Every DevOps Engineer Needs

• Your Production Readiness Checklist

• Conclusion

Who This Article Is For

• Early-career DevOps and cloud engineers who are building or maintaining production infrastructure at a startup.

• Backend developers who have recently taken on DevOps responsibilities.

• Engineers joining a startup who want to understand what operational discipline actually looks like in a fast-moving environment.

You do not need to be an expert in any specific tool to follow this article. The focus is on decision-making patterns and operational discipline, not tool configuration.

Why Startups Are a Different Environment

Before getting into the mistakes, you have to understand why startups produce them in the first place.

In a large company, you typically have dedicated security engineers, an SRE team, a platform team, and multiple reviewers for every infrastructure change. In a startup, you mostly likely have one engineer responsible for all of that simultaneously.

This creates four specific pressure points:

1. Speed pressure. The business needs features shipped now. Operational discipline gets treated as optional because nobody is watching closely yet.

2. Budget constraints. Every infrastructure decision has a direct impact on company runway. Engineers optimize for the cheapest option rather than the most reliable one.

3. Absent guardrails. There is no senior engineer reviewing your Terraform plans. There is no security audit before launch. The absence of immediate consequences can make bad decisions feel like good ones.

4. Constantly changing requirements. The architecture you design today may need to support a completely different product in six months. None of these pressures are excuses for poor decisions. But understanding them helps you see why the following mistakes happen so consistently.

Mistake 1: Deploying Without Understanding What You're Deploying

The Scenario

A junior engineer is asked to deploy the company's Node.js API to AWS. They find a tutorial for Elastic Beanstalk, follow it, and it works. Two weeks later, traffic increases. They try to scale "the same way as in the tutorial." The application goes down. They cannot debug it because they never understood what the deployment was actually doing.

The Business Impact

When production breaks and the person who deployed the system cannot explain how it works, diagnosis takes hours instead of minutes. The longer the incident runs, the higher the cost in customer trust, team morale, and potentially direct revenue loss.

The Fix

Before you deploy anything to production, you should be able to answer these five questions in writing:

1. What compute type is running my code? (EC2, Lambda, Fargate, container?)

2. How does a new version replace the old one? (Rolling? Blue/green? All-at-once?)

3. Where does configuration and secrets come from? (SSM? Secrets Manager? Environment file?)

4. What downstream services depend on this? (Database connections? Other APIs? Cache?)

5. How do I roll back in under five minutes if this breaks?

If you cannot answer all five, do not deploy until you can. The tutorial that got it running is not the documentation for how it operates.

"It is better to spend two hours understanding a system before deploying it than two days debugging it after something breaks."

Personally, when learning a new technology, tool, or implementing something I have not worked with before, I usually focus on three core questions: What, Why, and How.

• The first question is: What is this technology or concept about?
  This helps me build a solid foundation by doing deep research, studying the official documentation, understanding the core principles, and sometimes even learning the history behind the tool or technology. I believe having a well-grounded understanding before implementation is very important.

• The second question is: Why do we need it?
  I try to understand the value the technology brings, why it should be implemented, what problem it solves, and how it benefits the team or organization. This helps me make informed technical decisions instead of just implementing tools without understanding their purpose.

• The third question is: How should it be implemented?
  There are usually multiple approaches to solving a problem or implementing a technology, so I focus on understanding the best and most practical approach based on the use case and expected outcome.

This structured approach has helped me learn new technologies quickly, adapt fast, and implement solutions effectively in real-world environments.

Mistake 2: Using Production as a Development Environment

The Scenario

To save time, an engineer tests a new deployment script directly in the production AWS account. They accidentally run a command that terminates the production database instance. Automated backups exist but were misconfigured. Six hours of customer data is unrecoverable.

This scenario happens more often than you would expect. The reasoning is always the same: "It will only take a minute."

The Business Impact

A single test-in-production incident can result in data loss, hours of downtime, and a customer communication crisis. In a startup, that can permanently damage the company's reputation before it has had the chance to build one.

The Fix

You need at minimum three separate environments and ideally three separate AWS accounts:

Using separate AWS accounts (not just separate VPCs) gives you account-level isolation. A permission error in the dev account cannot accidentally touch production infrastructure at the API level.

Infrastructure as Code (Terraform or CloudFormation) makes this affordable, you write the configuration once and apply it three times with different variable files.

# terraform/environments/prod/main.tf
module "app" {
  source      = "../../modules/app"
  environment = "production"
  instance_type = "t3.medium"
  db_instance_class = "db.t3.medium"
  multi_az          = true
}

# terraform/environments/staging/main.tf
module "app" {
  source      = "../../modules/app"
  environment = "staging"
  instance_type = "t3.small"
  db_instance_class = "db.t3.small"
  multi_az          = false
}

The module is the same. The environment-specific variables are different. Separate environments are not a luxury, they are the minimum operating standard for any team running real software.

Mistake 3: Hardcoding Secrets and Credentials

The Scenario

A new engineer joins a startup and clones the repository. Inside they find a .env file committed to Git containing the production database password, the Stripe secret key, and an AWS access key with admin permissions. The repository has been public for six months.

GitHub's automated secret scanning never triggered because the secrets were inside a .env file rather than raw in the code. The credentials had been valid and actively used for over six months.

The Business Impact

Automated scanners run by attackers find exposed credentials within minutes of them being pushed to a public repository. A single exposed AWS access key with admin permissions can result in:

• Crypto-mining workloads generating thousands of dollars in cloud bills overnight

• Complete exfiltration of customer data from every S3 bucket

• Privilege escalation: the attacker creates new admin users and locks you out of your own account

• AWS account suspension while the investigation runs

According to GitHub's annual security report, millions of secrets are exposed in public repositories every year. The average time to detect a compromised cloud credential is 197 days.

The Fix

Step 1: Never commit secrets to Git. Not temporarily. Not in a branch. Not in a private repository.

Step 2: Add .gitignore before you create the first file. Check in the .gitignore with the first line of code before any .env files exist.

# .gitignore
.env
.env.*
*.pem
*.key
secrets/

Step 3: Use AWS Secrets Manager or SSM Parameter Store for all production secrets. Your application reads secrets at runtime:

# Python example — fetch secret at runtime, never at build time
import boto3
import json
 
def get_secret(secret_name: str, region: str = "us-east-1") -> dict:
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])
 
# Usage
db_config = get_secret("prod/myapp/database")
DATABASE_URL = db_config["connection_string"]

Step 4: Scan your existing repositories immediately. You may already have a problem:

# Install trufflehog to scan for exposed secrets in your repo history
pip install trufflehog
 
# Scan the entire commit history of your repository
trufflehog git file://.
 
# Or scan a remote GitHub repo
trufflehog github --repo https://github.com/your-org/your-repo

Step 5: Add a pre-commit hook to prevent future accidents:

pip install pre-commit

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/awslabs/git-secrets
    rev: master
    hooks:
      - id: git-secrets
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets

pre-commit install
# Now the hook runs before every commit and blocks detected secrets

There is no recovery from a publicly exposed database password. The fix takes ten minutes upfront. The incident takes weeks.

Mistake 4: Overengineering for Problems You Don't Have Yet

The Scenario

A five-person startup with 200 users decides to build a microservices architecture on Kubernetes because "Netflix uses it." They spend three months setting up Kubernetes, Istio service mesh, ArgoCD, Vault, Prometheus, and Grafana. Their product has not shipped a new feature in three months. A competitor with a monolith on a single EC2 instance shipped twelve new features in the same period.

The Business Impact

Every layer of infrastructure you add is a layer that can break, a layer that requires expertise to operate, and a layer that slows down every future change. Kubernetes is the right answer for organizations with the scale and team size to operate it. For a five-person startup, it is an expensive distraction.

Premature complexity does not just cost engineering time. It costs the competitive advantage that speed provides in the early stage.

The Fix

Match your infrastructure to your actual stage:

The question to ask before every infrastructure decision is: "What specific, measurable problem does this solve today that my current setup cannot solve?"

Amazon, Netflix, and Uber did not start with microservices. They started with monoliths and extracted services only when the monolith became the actual bottleneck. You are not Netflix. You are solving the problems in front of you today.

Use managed services wherever possible, RDS instead of self-hosted Postgres, Fargate instead of self-managed Kubernetes, ElastiCache instead of self-hosted Redis. Managed services let your team focus on the product instead of the infrastructure.

Mistake 5: No Observability Before Launch

The Scenario

A startup's checkout flow breaks on a Friday evening. Users are abandoning their carts and the company is losing revenue. The DevOps engineer finds out 45 minutes later because a customer sent a direct message to the CEO on Twitter.

The engineer has no dashboards, no log aggregation, and no alerting. They SSH into the production server and scroll through raw log files. Two hours later, they find the issue: a database connection pool was exhausted by a memory leak introduced in that morning's deployment.

Business Impact

Without observability:

• You find out about production problems from users, not from your systems

• Incidents take 10x longer to resolve because diagnosis is guesswork

• You cannot tell whether a deployment improved or degraded performance

• You have no data for making better architecture decisions

The Fix

Implement the four golden signals before any service goes to production. These come from Google's Site Reliability Engineering book:

1. Latency: How long requests take to complete (p50, p95, p99)

2. Traffic: How many requests per second the system is handling

3. Errors: The rate of failed requests (5xx responses per minute)

4. Saturation: How close the system is to its limits (CPU, memory, connection pool)

Here is a minimal CloudWatch alarm setup using the AWS CLI:

# Alert when error rate exceeds 1% for 5 consecutive minutes

aws cloudwatch put-metric-alarm \
  --alarm-name "high-error-rate-production" \
  --alarm-description "Error rate exceeded 1% for 5 minutes" \
  --metric-name "5XXError" \
  --namespace "AWS/ApplicationELB" \
  --statistic "Average" \
  --period 60 \
  --evaluation-periods 5 \
  --threshold 0.01 \
  --comparison-operator "GreaterThanOrEqualToThreshold" \
  --alarm-actions "arn:aws:sns:us-east-1:123456789:pagerduty-production" \
  --dimensions Name=LoadBalancer,Value=app/my-alb/1234567890abcdef

Every application should also expose a /health endpoint that returns 200 OK when healthy:

# FastAPI example

from fastapi import FastAPI
from sqlalchemy import text
 
app = FastAPI()
 
@app.get("/health")
async def health_check():
    # Check database connectivity
    try:
        db.execute(text("SELECT 1"))
        db_status = "healthy"
    except Exception:
        db_status = "unhealthy"
 
    return {
        "status": "healthy" if db_status == "healthy" else "degraded",
        "database": db_status,
        "version": os.getenv("APP_VERSION", "unknown")
    }

Your load balancer checks this endpoint. Your uptime monitor checks it. You check it after every deployment.

You do not get to say a system is working unless you have data to prove it. "Nobody complained" is not the same as "nothing is broken."

Mistake 6: Treating Security as a Final Step

The Scenario

A startup rushes to launch their MVP. Security reviews are "planned for after launch." Six months later, a potential enterprise customer requires a security audit before signing a contract. The audit reveals:

• S3 buckets publicly accessible by default

• EC2 instances with port 22 open to 0.0.0.0/0

• IAM users with AdministratorAccess for the entire team

• No encryption on the database at rest

• JWT secrets hardcoded in environment variables The audit fails. The enterprise deal worth $120,000 annually is lost. Remediation takes four weeks of engineering time.

The Business Impact

Security debt is the most expensive technical debt you can accumulate. Unlike performance debt that degrades gradually, security vulnerabilities cause sudden, catastrophic events: data breaches, ransomware, account takeovers, and regulatory fines. At a startup, any one of these can end the company.

The Fix

Apply these six security controls before the first line of production code ships:

1. Principle of Least Privilege every IAM role gets only what it needs:

One of the most common security mistakes in AWS is granting roles more permissions than they need either out of convenience (s3:*) or uncertainty about what the service actually requires. This creates unnecessary risk: if a role is compromised, the attacker inherits every permission you granted.

The fix is simple: look at what your service actually does, then write a policy that allows exactly that.

If your app uploads and reads files from a specific S3 bucket, the policy should say exactly that:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-app-uploads/*"
    }
  ]
}

Notice the Resource is scoped to my-app-uploads/* not all S3 buckets. And the Action list covers only GetObject and PutObject not DeleteObject, not s3:*. If the service gets compromised, the attacker can read and write to that one bucket. That is it. The rest of your account is untouched.

2. Block all S3 public access by default:

AWS S3 buckets are private by default when created but that can be overridden at the bucket level, the object level, or through a bucket policy. Misconfigured S3 buckets are one of the most common causes of data breaches, and they are almost always accidental.

The safest approach is to enable the "Block Public Access" setting at the account level, which overrides all other settings and prevents any bucket from being made public even if someone tries:

aws s3api put-public-access-block \
  --bucket my-app-bucket \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Run this for every bucket you create. Better yet, enable it at the AWS account level so it applies automatically to all future buckets by default.

3. Never open SSH to the internet, use AWS Systems Manager Session Manager instead:

Port 22 open to 0.0.0.0/0 is an attack surface that exists on thousands of AWS instances right now. Brute-force bots scan the internet continuously looking for open SSH ports. Even with a strong key, the exposure is unnecessary because AWS provides a better alternative.

AWS Systems Manager Session Manager gives you full shell access to any EC2 instance without opening a single inbound port on the security group. There is no port to scan, no port to attack, and every session is logged automatically to CloudTrail:

# Start a session on an EC2 instance without port 22 open
aws ssm start-session --target i-0123456789abcdef0

To use Session Manager, the EC2 instance needs the SSM Agent installed (included by default on Amazon Linux 2 and Ubuntu 20.04+) and an IAM instance profile with the AmazonSSMManagedInstanceCore policy attached. Once that is set up, you can close port 22 on the security group entirely.

4. Enable MFA for all IAM users and enforce it via policy:

A leaked IAM username and password with no MFA is a fully compromised account. Multi-factor authentication is the single most effective control against credential theft, and it costs nothing to enable.

Enforce it through an IAM policy that denies all actions when MFA is not present, except the actions needed to set up MFA in the first place. This means even if a set of credentials is stolen, the attacker cannot do anything without the second factor.

The AWS documentation provides the Complete Deny Without MFA Policy, attach it to every IAM user or group in your account. This is a one-time setup that permanently raises your account's security baseline.

5. Enable CloudTrail in all regions:

Without CloudTrail, you have no record of who did what in your AWS account. If a credential is compromised, you cannot investigate what the attacker accessed. If an engineer accidentally deletes a resource, you cannot trace it. You are operating blind.

CloudTrail logs every AWS API call who made it, from which IP, at what time, and what the response was. Enable it across all regions so activity in regions you do not actively use is also captured:

aws cloudtrail create-trail \
  --name production-audit-trail \
  --s3-bucket-name my-cloudtrail-logs \
  --is-multi-region-trail \
  --enable-log-file-validation

The --enable-log-file-validation flag generates a digest file for each log that lets you verify the log has not been tampered with, this is important if you ever need to use these logs in a security investigation or compliance audit. Once this is running, every AssumeRole, every DeleteBucket, and every RunInstances call in your account is permanently recorded.

6. Run AWS Security Hub from day one:

Most teams only discover security misconfigurations after a breach or a compliance audit. Security Hub inverts this, it continuously scans your AWS environment against industry-standard frameworks (CIS AWS Foundations Benchmark, AWS Foundational Security Best Practices) and surfaces findings before they become incidents.

Enabling it takes a single command:

aws securityhub enable-security-hub

Within minutes, Security Hub gives your account a compliance score and a prioritized list of findings. A finding might tell you that a security group has port 22 open to the world, that an S3 bucket has logging disabled, or that root account credentials were recently used. Each finding includes the affected resource and a remediation guide.

Treat every Security Hub finding the same way you treat a production bug: assign it a priority, assign an owner, and close it. A finding sitting unaddressed for 30 days is a known vulnerability you chose to leave open.

Mistake 7: Manual Deployments in Production

The Scenario

A startup's deployment process is documented in a Notion page that is four months out of date. It involves SSH-ing into the server, running git pull, running npm install, and restarting the PM2 process. Different engineers do it slightly differently. One engineer, rushing a late-night release, skips npm install. The application starts crashing because a new dependency is missing.

The Business Impact

Manual deployment processes are inherently unreliable. Humans under pressure skip steps, perform steps in the wrong order, and remember procedures differently. Every manual step in a production deployment process is a scheduled incident waiting for the right moment of stress.

The Fix

If a deployment step is performed manually more than twice, it needs to be automated. Here is a minimal but complete GitHub Actions deployment workflow for an ECS Fargate service:

# .github/workflows/deploy.yml
name: Deploy to Production
 
on:
  push:
    branches:
      - main
 
permissions:
  id-token: write   # Required for OIDC authentication with AWS
  contents: read
 
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
 
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
          aws-region: us-east-1
 
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2
 
      - name: Build and push Docker image
        id: build
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t \(ECR_REGISTRY/my-app:\)IMAGE_TAG .
          docker push \(ECR_REGISTRY/my-app:\)IMAGE_TAG
          echo "image=\(ECR_REGISTRY/my-app:\)IMAGE_TAG" >> $GITHUB_OUTPUT
 
      - name: Deploy to Amazon ECS
        uses: aws-actions/amazon-ecs-deploy-task-definition@v1
        with:
          task-definition: task-definition.json
          service: my-app-service
          cluster: production
          wait-for-service-stability: true

Notice wait-for-service-stability: true. Without this, the workflow reports success the moment ECS accepts the new task definition before the containers are actually healthy. With it, the workflow fails if the new containers crash. You want to know immediately, not discover it from user reports thirty minutes later.

Mistake 8: No Disaster Recovery Plan

The Scenario

A startup's production database runs on a single RDS instance with no Multi-AZ configuration. Automated backups are enabled but have never been tested. The EBS volume backing the instance fails. AWS provisions a new instance from the last snapshot, which is 18 hours old. 18 hours of customer data is permanently lost.

The startup had no disaster recovery plan, no tested recovery procedure, and no communication template ready for customers.

The Business Impact

The question is not whether your infrastructure will fail. It will fail. Every database, every server, every availability zone experiences failures. The question is whether you have a tested plan for when it does.

Data loss of any magnitude is serious. For startups that handle financial data, healthcare data, or anything under GDPR, even partial data loss can trigger regulatory consequences.

The Fix

Define your RTO and RPO before you design anything:

• RTO (Recovery Time Objective): How long can the business survive without this system? A payment API might have an RTO of 15 minutes. An internal analytics dashboard might have an RTO of 4 hours.

• RPO (Recovery Point Objective): How much data loss is acceptable? Zero means real-time replication. One hour means hourly snapshots are sufficient. This directly determines your backup frequency and architecture.

Enable RDS Multi-AZ for all production databases:

# Terraform
resource "aws_db_instance" "production" {
  identifier        = "prod-postgres"
  engine            = "postgres"
  engine_version    = "15.4"
  instance_class    = "db.t3.medium"
  allocated_storage = 100
 
  # Multi-AZ: automatic failover to standby in a different AZ
  # No data loss. Automatic failover in ~60-120 seconds.
  multi_az = true
 
  # Encryption at rest — non-negotiable
  storage_encrypted = true
 
  # Automated backups with 7-day retention
  backup_retention_period = 7
  backup_window           = "03:00-04:00"
 
  # Enable deletion protection in production
  deletion_protection = true
 
  tags = {
    Environment = "production"
  }
}

Test your backups on a schedule. Create a monthly calendar event: "Restore production backup to staging and verify data integrity." An untested backup is not a backup, it is a hope.

# Restore a snapshot to a test instance and verify
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier recovery-test \
  --db-snapshot-identifier rds:prod-postgres-2025-01-15 \
  --db-instance-class db.t3.medium \
  --no-multi-az
 
# Connect and verify row counts
psql -h recovery-test.xxxx.rds.amazonaws.com -U admin -d mydb \
  -c "SELECT COUNT(*) FROM users; SELECT COUNT(*) FROM orders;"

For official guidance on RDS backup and restore, refer to the AWS RDS Backup and Restore documentation.

Mistake 9: No Documentation or Runbooks

The Scenario

The startup's most experienced DevOps engineer takes two weeks of vacation. On day three of their holiday, the staging environment goes down. Nobody else knows how it was built, the engineer set it up manually over six months with no documentation, no Terraform, no notes. The team spends four days trying to reconstruct the environment from memory and guesswork. The engineer gets messages on their vacation every day. When they return, they rebuild the environment in four hours.

The Business Impact

Undocumented infrastructure creates single points of failure not in your systems, but in your team. It makes onboarding new engineers take weeks instead of hours. It makes incident response depend on specific people being available. When that person leaves the company, the knowledge walks out with them.

The Fix

Documentation for an engineering team means three specific things:

1. Infrastructure as Code is the highest form of documentation. The Terraform that defines your infrastructure IS the documentation for what exists and how it is configured. If something is not in code, it should not exist in production.

2. A runbook for every operational task. A runbook is a step-by-step procedure written well enough that someone in their first week at the company can follow it during an incident:

# Runbook: Production Database Connection Exhaustion
 
## Symptoms
- Application logs: "too many connections" errors
- 500 error rate spike on database-dependent endpoints
- pg_stat_activity shows max connections reached
 
## Diagnosis
# Check current connection count
psql -h \(DB_HOST -U \)DB_USER -c "SELECT COUNT(*) FROM pg_stat_activity;"
 
# See connections by application
psql -h \(DB_HOST -U \)DB_USER \
  -c "SELECT application_name, COUNT(*) FROM pg_stat_activity GROUP BY 1 ORDER BY 2 DESC;"

## Resolution
1. Identify and restart the service causing the connection leak
2. If immediate relief needed: kill idle connections older than 10 minutes
3. Long-term: review connection pool settings in application config

## Escalation
If unresolved in 30 minutes: page the on-call backend engineer.

1. An architecture README in every repository. Every engineer who clones your repository should be able to understand what it does, how to run it locally, how to deploy it, and what it depends on without asking anyone.

Mistake 10: Solving Technical Problems Without Understanding the Business

The Scenario

A startup is experiencing slow page loads. A DevOps engineer decides to solve it by migrating to Kubernetes with horizontal pod auto-scaling. The migration takes six weeks. Page loads improve slightly. But 80% of the slowness was caused by unoptimized database queries that had nothing to do with the infrastructure layer. The six-week migration solved 20% of the problem.

The Business Impact

Technical solutions to misdiagnosed problems are extraordinarily expensive. Every hour spent building the wrong solution is an hour not spent on the right one. Infrastructure is a tool for delivering business outcomes not an end in itself.

The Fix

Before making any infrastructure decision, answer these four questions:

1. What is the actual, measured bottleneck? Instrument before you act. The bottleneck is almost never where you assumed it was.

2. What does success look like, and how will you measure it? "Pages are faster" is not measurable. "p95 page load time drops below 1.2 seconds" is measurable.

3. What is the full cost of this solution? Time to implement, ongoing operational burden, team learning curve. Is this cost justified by the measured impact?

4. Can a simpler solution solve 80% of the problem in 20% of the time?

Always profile and measure before you rebuild:

# Check slow queries in PostgreSQL before any infrastructure changes
psql -h \(DB_HOST -U \)DB_USER -d $DB_NAME -c "
SELECT
  query,
  calls,
  total_exec_time / calls AS avg_ms,
  rows / calls AS avg_rows
FROM pg_stat_statements
ORDER BY avg_ms DESC
LIMIT 10;
"

Nine times out of ten, slow applications have slow queries, missing indexes, or an N+1 query problem, none of which require a new infrastructure layer to fix.

The System Thinking Framework Every DevOps Engineer Needs

Most of the mistakes above share a common root cause: the engineer was thinking about one component in isolation instead of the full system.

A system thinker asks six questions before making any change in production:

This is not a checklist you run through slowly. It is a thinking habit that becomes automatic with practice. Senior engineers do not spend more time on deployments than junior engineers do, they spend their time on different things, and this is one of them.

Your Production Readiness Checklist

Use this checklist before any production system goes live. Mark each item as done, in progress, or not yet started.

Infrastructure

• Infrastructure is defined as code (Terraform or CloudFormation) and version-controlled in Git

• Separate dev, staging, and production environments exist with separate credentials

• All production changes go through an automated CI/CD pipeline, no manual SSH deployments

• You can rebuild the entire production environment from code in under two hours

Security

• No secrets, credentials, or API keys exist in any Git repository

• All production secrets are in Secrets Manager or SSM Parameter Store

• All IAM roles follow the principle of least privilege

• S3 buckets have public access blocked by default

• Port 22 is not open to 0.0.0.0/0 on any security group

• CloudTrail is enabled in all regions

• All IAM users have MFA enabled

• AWS Security Hub is enabled and findings are reviewed weekly

Observability

• Every service has a /health endpoint that monitoring checks continuously

• Alerts fire within five minutes of a production error rate spike

• Dashboards exist showing latency, error rate, and resource utilization

• Logs are centralized and searchable, not scattered across individual servers

Reliability

• Production database has Multi-AZ enabled

• Backup restoration has been tested in the last 30 days

• Written runbooks exist for the three most likely failure scenarios

• RTO and RPO requirements are documented and the architecture meets them

Documentation

• Every repository has a README explaining what it does and how to deploy it

• A new engineer could understand the production architecture from documentation alone

• No single engineer holds critical knowledge that lives only in their head

Conclusion

None of the mistakes in this article require rare misfortune to experience. They are the predictable result of decisions that feel reasonable under startup pressure but accumulate into real operational risk over time.

The good news is that every single one of them is preventable with the right awareness and the right habits applied early.

You do not need a perfect infrastructure from day one. You need a correct one: version-controlled, automated, observable, secure, and documented. Start with that foundation. Add complexity only when a specific, measured problem requires it. Always connect technical decisions to business outcomes.

The goal of DevOps in a startup is not to build impressive infrastructure. It is to build reliable systems that support product growth safely, efficiently, and sustainably and to make sure that when something does break, you can recover faster than anyone notices.

Want to Go Deeper?

If this article resonated with you, The Startup DevOps Field Guide covers these principles in full depth with complete infrastructure blueprints, security frameworks, CI/CD pipeline templates, and the end-to-end decision-making playbook for engineers building DevOps practices in startup environments from scratch.

It is written specifically for the engineer who wants to do this right from the beginning not the one rebuilding everything after the first major incident.

But it has always carried a cost that rarely gets discussed openly. That cost isn't just money, though a DynamoDB table with on-demand billing adds up across multiple teams and environments.

The real cost is complexity. Every new AWS environment needs both resources provisioned before Terraform can manage anything else. Every engineer who sets up their first Terraform backend has to understand why two completely different AWS services are responsible for what is logically one thing: storing and protecting state. And every incident involving a stuck lock has required someone to manually delete a record from DynamoDB to unblock the team.

In November 2024, AWS announced that S3 now supports native object locking for Terraform state files, meaning DynamoDB is no longer required for state locking. Terraform 1.10 added support for this feature, and it's now generally available.

In this tutorial, you'll learn:

• What S3 native locking is and how it works

• How to set it up from scratch if you're starting a new project

• How to migrate an existing S3 + DynamoDB setup to S3 native locking safely

• How to verify locking is working and handle edge cases

By the end, you'll have a simpler, cleaner Terraform backend with one fewer AWS resource to manage.

Table of Contents

• What Is Terraform State Locking?

• What Is S3 Native State Locking?

• How S3 Native Locking Compares to the S3 + DynamoDB Approach

• Prerequisites

• Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

  • Step 1: Create the S3 Bucket with Versioning and Encryption

  • Step 2: Configure the Terraform Backend with Native Locking

  • Step 3: Initialize and Verify

• Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

  • Step 1: Verify Your Current Setup

  • Step 2: Enable Object Lock on the Existing S3 Bucket

  • Step 3: Update the Terraform Backend Configuration

  • Step 4: Reinitialize Terraform

  • Step 5: Verify the Migration

  • Step 6: Clean Up the DynamoDB Table

• How to Verify That Locking Is Working

• How to Handle a Stuck Lock

• Rollback Plan: If Something Goes Wrong

• Security Best Practices for Your State Bucket

• Conclusion

• References

What is Terraform State Locking?

Before looking at the new approach, it helps to understand what state locking is solving.

Terraform stores everything it knows about your infrastructure in a state file – a JSON document that maps your configuration to real AWS resources. When you run terraform apply, Terraform reads this file, calculates the difference between the current state and your configuration, and makes the necessary changes.

The problem arises when two engineers or two CI/CD pipelines run and try to apply changes at the same time. If both read the state file simultaneously, calculate changes independently, and both try to write back, you get a race condition. The second write overwrites changes from the first, and your state is now out of sync with reality. This is a serious problem that can cause resources to be untracked, doubled, or destroyed unexpectedly.

State locking solves this by creating a lock when any operation starts that could modify state. If a lock already exists, Terraform refuses to proceed and reports who holds the lock and when it was acquired. Only one operation can hold the lock at a time. When the operation completes, the lock is released.

Terraform Run A                 State File / Lock                Terraform Run B
(User 1)                         (S3/DynamoDB)                   (User 2)

   |                                   |                            |
   |------- 1. Acquire Lock ---------->|                            |
   |                                   |                            |
   |<------ 2. Lock Granted -----------|                            |
   |                                   |                            |
   |                                   |------- 3. Acquire Lock --->|
   |            [PROCESSING]           |                            |
   |      (Modifying Infrastructure)   |<------ 4. Lock Denied -----|
   |                                   |        (Wait / Retry)      |
   |                                   |                            |
   |------- 5. Release Lock ---------->|                            |
   |                                   |                            |
   |           [COMPLETED]             |<------ 6. Lock Granted ----|
   |                                   |                            |
   |                                   |       [PROCESSING]         |
   |                                   | (Modifying Infrastructure) |              
   |                                   |                            |

What Is S3 Native State Locking?

Previously, Terraform's S3 backend used a DynamoDB table as the locking mechanism. When a lock was needed, Terraform wrote a record to DynamoDB with a LockID primary key. DynamoDB's conditional writes guaranteed that only one process could create that record, which is what made the locking atomic.

S3 native locking uses S3 Object Lock instead. S3 Object Lock is an S3 feature originally designed to enforce WORM (Write Once, Read Many) compliance for regulatory requirements. AWS extended this capability to support Terraform's state locking workflow.

When S3 native locking is enabled in your Terraform backend:

1. Terraform writes your state to an .tfstate object in S3 (as before)

2. To acquire a lock, Terraform uses S3's conditional write operations – specifically the if-none-match conditional header to create a lock file atomically

3. If the lock file already exists, S3 rejects the write, and Terraform reports that a lock is held

4. When the operation completes, Terraform deletes the lock file to release the lock.

The key difference from DynamoDB: the entire locking mechanism lives inside S3. No second service. No second set of IAM permissions. No second resource to provision.

Note: This feature requires Terraform version 1.10.0 or later and an S3 bucket with Object Lock enabled. Object Lock must be enabled at bucket creation time. You can't enable it on an existing bucket through the console or CLI. But there is a supported workaround for existing buckets, which we'll cover in Part 2.

How S3 Native Locking Compares to the S3 + DynamoDB Approach

The functional behavior from Terraform's perspective is identical. Locking works the same way. The lock information displayed when a lock is held has the same structure. The only difference is what happens under the hood.

Prerequisites

Before you start, make sure you have the following in place:

• Terraform 1.10.0 or later installed. Check your version:

terraform version

If you need to upgrade, follow the official upgrade guide.

• AWS CLI installed and configured with credentials that have permission to create and manage S3 buckets.

aws --version
aws sts get-caller-identity   # confirm you're authenticated

• IAM permissions to perform the following S3 actions:

  • s3:CreateBucket

  • s3:PutBucketVersioning

  • s3:PutBucketEncryption

  • s3:PutObjectLegalHold

  • s3:PutObjectRetention

  • s3:GetObject

  • s3:PutObject

  • s3:DeleteObject

  • s3:ListBucket

• For the migration path: access to your existing Terraform project and the S3 bucket and DynamoDB table currently in use.

Part 1: Fresh Setup – How to Configure S3 Native Locking from Scratch

Follow this section if you're starting a new Terraform project and want to use S3 native locking from the beginning.

Step 1: Create the S3 Bucket with Versioning and Encryption

Object Lock must be enabled at bucket creation time. You can't add it afterward through the standard console flow. Create the bucket using the AWS CLI with Object Lock enabled:

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

Note: For regions other than us-east-1, add the --create-bucket-configuration flag.

aws s3api create-bucket \
  --bucket your-project-terraform-state \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1 \
  --object-lock-enabled-for-bucket

Now enable versioning on the bucket. Versioning is required alongside Object Lock and allows Terraform to recover previous state versions if something goes wrong:

aws s3api put-bucket-versioning \
  --bucket your-project-terraform-state \
  --versioning-configuration Status=Enabled

Enable server-side encryption so your state files are encrypted at rest:

aws s3api put-bucket-encryption \
  --bucket your-project-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

Block all public access to the bucket. A Terraform state file contains resource IDs, IP addresses, and potentially sensitive values. It should never be publicly accessible:

aws s3api put-public-access-block \
  --bucket your-project-terraform-state \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Verify the bucket configuration:

# Confirm Object Lock is enabled
aws s3api get-object-lock-configuration \
  --bucket your-project-terraform-state
 
# Confirm versioning is enabled
aws s3api get-bucket-versioning \
  --bucket your-project-terraform-state
 
# Confirm encryption is configured
aws s3api get-bucket-encryption \
  --bucket your-project-terraform-state

Expected output for the Object Lock check:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Step 2: Configure the Terraform Backend with Native Locking

In your Terraform project, create or update your backend.tf file:

terraform {
  backend "s3" {
    bucket = "your-project-terraform-state"
    key    = "production/terraform.tfstate"
    region = "us-east-1"
 
    # Enable S3 native state locking
    # Requires Terraform 1.10.0+ and a bucket with Object Lock enabled
    use_lockfile = true
 
    # Encryption at rest
    encrypt = true
  }
}

The critical difference from the old configuration is the use_lockfile = true parameter. Notice what is absent: there's no dynamodb_table argument. No DynamoDB table. No second service.

Here's a direct comparison of the old and new configurations:

Old configuration (S3 + DynamoDB):

terraform {
  backend "s3" {
    bucket         = "your-project-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # this goes away
  }
}

New configuration (S3 native locking):

terraform {
  backend "s3" {
    bucket       = "your-project-terraform-state"
    key          = "production/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # this replaces dynamodb_table
  }
}

Step 3: Initialize and Verify

Run terraform init to initialize the backend:

terraform init

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
 
Terraform has been successfully initialized!

Run a plan to confirm everything is working end-to-end:

terraform plan

If locking is working, you'll see a brief pause while Terraform acquires the lock before the plan output appears. You'll also see the lock information if you look at the S3 bucket – a .tflock file will appear temporarily alongside your state file during the operation and disappear when it completes.

Part 2: Migration – How to Move from S3 + DynamoDB to S3 Native Locking

Follow this section if you have an existing Terraform setup using an S3 bucket and DynamoDB table for state locking, and you want to migrate to S3 native locking.

Important: Migration requires a maintenance window or at minimum a period where no Terraform operations are running. You're changing the backend configuration, which means all team members and CI/CD pipelines must stop running terraform plan or terraform apply during the migration. The migration itself takes under 10 minutes.

Step 1: Verify Your Current Setup

Before making any changes, document your existing backend configuration and confirm the state file is accessible:

# Confirm your state file is in S3
aws s3 ls s3://your-existing-bucket/path/to/terraform.tfstate
 
# Confirm the DynamoDB table exists
aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table \
  --query 'Table.TableStatus'

Check your current backend.tf and note the exact values:

# Your current backend.tf - note these values before changing anything
terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"       # note this
    key            = "path/to/terraform.tfstate"   # note this
    region         = "us-east-1"                   # note this
    encrypt        = true
    dynamodb_table = "your-dynamodb-lock-table"    # this will be removed
  }
}

Run one final plan to confirm the current state is clean and there are no unexpected changes pending:

terraform plan

If the plan shows no changes, you're in a safe state to proceed.

Step 2: Enable Object Lock on the Existing S3 Bucket

This is the most important step in the migration. Object Lock can't normally be enabled on an existing bucket. It's a setting that must be configured at creation time.

But AWS provides a way to enable Object Lock on an existing bucket through a support request or through a direct API call that's not exposed in the standard console UI. AWS has officially documented this path for the Terraform migration use case.

Run the following AWS CLI command to enable Object Lock on your existing bucket:

aws s3api put-object-lock-configuration \
  --bucket your-existing-bucket \
  --object-lock-configuration '{"ObjectLockEnabled": "Enabled"}'

Note: This command enables Object Lock in governance mode with no default retention, meaning it enables the locking capability without setting a default retention period on all objects. This is exactly what Terraform's native locking needs: the ability to create and delete lock files, not permanent object retention.

Verify Object Lock is now enabled:

aws s3api get-object-lock-configuration \
  --bucket your-existing-bucket

Expected output:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Also verify that versioning is already enabled (it should be if you are running a production Terraform setup):

aws s3api get-bucket-versioning \
  --bucket your-existing-bucket

Expected output:

{
    "Status": "Enabled"
}

If versioning isn't enabled, enable it before proceeding:

aws s3api put-bucket-versioning \
  --bucket your-existing-bucket \
  --versioning-configuration Status=Enabled

Step 3: Update the Terraform Backend Configuration

Update your backend.tf to remove the dynamodb_table argument and add use_lockfile = true:

terraform {
  backend "s3" {
    bucket = "your-existing-bucket"
    key    = "path/to/terraform.tfstate"
    region = "us-east-1"
    encrypt = true
 
    # Add this:
    use_lockfile = true
 
    # Remove this line entirely:
    # dynamodb_table = "your-dynamodb-lock-table"
  }
}

Your updated backend.tf should look like this:

terraform {
  backend "s3" {
    bucket       = "your-existing-bucket"
    key          = "path/to/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Step 4: Reinitialize Terraform

Run terraform init with the -reconfigure flag. This flag tells Terraform that the backend configuration has changed intentionally and to reinitialize without prompting you to copy state (the state is already in the same bucket):

terraform init -reconfigure

Expected output:

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
 
Terraform has been successfully initialized!

If you see an error here: The most common cause is that Object Lock wasn't successfully enabled on the bucket. Re-run the verification from Step 2 before proceeding.

Step 5: Verify the Migration

Run a plan to confirm Terraform is working correctly with the new backend configuration:

terraform plan

The plan should:

• Complete successfully

• Show the same result as the plan you ran in Step 1 (no changes, or the same changes as before)

• NOT mention DynamoDB anywhere in its output

To confirm that locking is actually using S3 instead of DynamoDB, open a second terminal and run a plan while the first one is running. You should see the second terminal output a lock error that mentions S3, not DynamoDB:

╷
│ Error: Error acquiring the state lock
│
│Error message: operation error S3: PutObject, https response       error StatusCode: 409,
│ RequestID: ..., api error Conflict: Object lock already exists for this key.
│
│ Lock Info:
│   ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
│   Path:      your-existing-bucket/path/to/terraform.tfstate.tflock
│   Operation: OperationTypePlan
│   Who:       user@hostname
│   Version:   1.10.0
│   Created:   2026-05-06 14:22:01 UTC
│   Info:
╵

The Path field shows .tfstate.tflock, a file in your S3 bucket, not a DynamoDB record. This confirms that locking is now handled entirely by S3.

Step 6: Clean Up the DynamoDB Table

Once you've confirmed the migration is working correctly and your team has run at least one successful plan and apply cycle using the new backend, you can remove the DynamoDB table.

Wait at least 24-48 hours before deleting the DynamoDB table if you have CI/CD pipelines or multiple team members. This gives time to catch any pipeline that wasn't updated with the new backend configuration.

When you're ready, delete the DynamoDB table:

aws dynamodb delete-table \
  --table-name your-dynamodb-lock-table

Confirm the deletion:

aws dynamodb describe-table \
  --table-name your-dynamodb-lock-table

Expected output:

An error occurred (ResourceNotFoundException) when calling the DescribeTable operation:
Requested resource not found

This error confirms that the table is gone. The migration is complete.

If you provisioned the DynamoDB table using Terraform (which is the recommended pattern), remove the resource from your Terraform configuration and run terraform apply to destroy it via Terraform rather than the CLI directly. This keeps your state clean:

# Remove this entire block from your Terraform configuration:
resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
 
  attribute {
    name = "LockID"
    type = "S"
  }
}

After removing the block, run:

terraform apply

Terraform will detect that the DynamoDB table resource has been removed from configuration and will destroy the table.

How to Verify That Locking Is Working

After completing either the fresh setup or the migration, use this procedure to independently verify that locking is functioning correctly.

Method 1: Observe the lock file during an operation

In one terminal, start a long-running plan against a configuration with many resources:

terraform plan

While it's running, in a second terminal, check for the lock file in S3:

aws s3 ls s3://your-bucket/path/to/ | grep tflock

You should see a file like:

2026-05-06 14:22:01        512 terraform.tfstate.tflock

After the plan completes, run the same command again. The .tflock file should be gone.

Method 2: Read the lock file contents

While a plan is running, download and read the lock file to see its contents:

aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/current.lock && cat /tmp/current.lock

Expected output (formatted for readability):

{
  "ID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "Operation": "OperationTypePlan",
  "Info": "",
  "Who": "tolani@dev-machine",
  "Version": "1.10.0",
  "Created": "2026-05-06T14:22:01.123456789Z",
  "Path": "your-bucket/path/to/terraform.tfstate"
}

This is the same lock information that Terraform displays when a lock is held. It's now a JSON file in S3 rather than a record in DynamoDB.

How to Handle a Stuck Lock

With the DynamoDB backend, resolving a stuck lock meant deleting a record from the DynamoDB table. With S3 native locking, it means deleting the .tflock file from S3.

A lock can get stuck if:

• A terraform apply or plan process was killed mid-execution

• A CI/CD pipeline runner crashed during a Terraform operation

• A network interruption prevented the lock release from completing

Here's how you can check for a stuck lock:

aws s3 ls s3://your-bucket/path/to/ | grep tflock

If a .tflock file exists and no Terraform operation is currently running, it is a stuck lock.

You can also read the lock to understand who held it:

aws s3 cp \
  s3://your-bucket/path/to/terraform.tfstate.tflock \
  /tmp/stuck.lock && cat /tmp/stuck.lock

This tells you who (Who field) was running the operation, what operation it was (Operation field), and when it was acquired (Created field).

And you can force-unlock using Terraform like this:

terraform force-unlock LOCK-ID

Replace LOCK-ID with the ID value from the lock file contents. For example:

terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

Terraform will confirm:

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may be still be in use. Only 'yes' will be accepted to confirm.
 
  Enter a value: yes
 
Terraform state has been successfully unlocked!

An alternative is to delete the lock file directly via CLI. If terraform force-unlock doesn't work (for example, because you are running in a CI environment without Terraform available), delete the lock file directly:

aws s3 rm s3://your-bucket/path/to/terraform.tfstate.tflock

Only delete the lock file if you are certain no Terraform operation is currently running. Deleting a lock that is actively held by a running operation will allow a second concurrent operation to start, which is exactly the race condition locking is designed to prevent.

Rollback Plan: If Something Goes Wrong

If you encounter problems after migrating, you can roll back to the S3 + DynamoDB setup with these steps.

Step 1: Stop all Terraform operations in your team and CI/CD pipelines.

Step 2: Recreate the DynamoDB table if you already deleted it:

aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

Step 3: Revert backend.tf to the previous configuration:

terraform {
  backend "s3" {
    bucket         = "your-existing-bucket"
    key            = "path/to/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # restored
    # Remove: use_lockfile = true
  }
}

Step 4: Reinitialize:

terraform init -reconfigure

Step 5: Verify:

terraform plan

The state file hasn't moved, so there's no data loss during a rollback. The only change is which locking mechanism Terraform uses.

Note: Object Lock being enabled on the S3 bucket doesn't prevent the rollback. Object Lock and DynamoDB locking can coexist, Object Lock simply adds a capability to the bucket. Using dynamodb_table in your backend config tells Terraform to use DynamoDB regardless of whether Object Lock is enabled on the bucket.

Security Best Practices for Your State Bucket

Migrating to S3 native locking is a good opportunity to review the overall security configuration of your state bucket. Here are the practices every production Terraform state bucket should implement:

Enable Versioning (Required)

Versioning is a hard requirement for S3 native locking to work safely. It ensures that if a state file is accidentally overwritten or corrupted, you can restore a previous version.

aws s3api put-bucket-versioning \
  --bucket your-state-bucket \
  --versioning-configuration Status=Enabled

Block All Public Access (Non-Negotiable)

Your state file contains resource ARNs, IP addresses, and may contain sensitive values passed through Terraform variables. It must never be publicly accessible.

aws s3api put-public-access-block \
  --bucket your-state-bucket \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Enable Server-Side Encryption

Always encrypt state files at rest. AES256 is the minimum. If your organization requires KMS key management:

aws s3api put-bucket-encryption \
  --bucket your-state-bucket \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id"
        },
        "BucketKeyEnabled": true
      }
    ]
  }'

Apply Least-Privilege IAM Permissions

The role or user that Terraform uses to access the state bucket should have only the permissions it needs. Here's a minimal IAM policy for S3 native locking:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TerraformStateAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-state-bucket",
        "arn:aws:s3:::your-state-bucket/*"
      ]
    },
    {
      "Sid": "TerraformStateLocking",
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectLegalHold",
        "s3:PutObjectLegalHold",
        "s3:GetObjectRetention",
        "s3:PutObjectRetention"
      ],
      "Resource": "arn:aws:s3:::your-state-bucket/*.tflock"
    }
  ]
}

Notice what is absent: there are no DynamoDB permissions. This is a cleaner, smaller permission set than the old approach required.

Enable Access Logging

Log all access to your state bucket in CloudTrail or S3 server access logs. This gives you an audit trail of every time state was read, written, or locked:

aws s3api put-bucket-logging \
  --bucket your-state-bucket \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "your-logging-bucket",
      "TargetPrefix": "terraform-state-access/"
    }
  }'

Conclusion

AWS S3 native state locking removes the need for a DynamoDB table from your Terraform backend setup. The result is simpler infrastructure, a smaller IAM permission surface, and one fewer service to provision, monitor, and pay for across every environment your team manages.

Here's a summary of what you accomplished:

• Understood what state locking is and why it's required for safe Terraform operations

• Compared S3 native locking to the existing S3 + DynamoDB approach

• Set up a fresh Terraform backend using S3 native locking with correct bucket configuration

• Migrated an existing backend from S3 + DynamoDB to S3 native locking safely

• Learned how to verify locking, handle stuck locks, and roll back if needed

• Applied security best practices to the state bucket

This pattern – using S3 native locking – is the recommended approach for all new Terraform projects on AWS going forward. If you're managing a large estate with multiple Terraform backends, consider automating the migration using a script or Terraform module that applies the pattern across all your state buckets.

If you are building or optimizing cloud infrastructure for a startup and want a complete reference for production-ready Terraform modules, CI/CD pipeline patterns, and infrastructure runbooks, check out The Startup DevOps Field Guide. It covers the full lifecycle of AWS infrastructure from initial setup to production reliability.

References

• HashiCorp - S3 Backend Configuration: use_lockfile

• HashiCorp: Terraform 1.10 Release Notes

• AWS Docs: S3 Object Lock Overview

• AWS Docs: PutObjectLockConfiguration API

• AWS Docs: S3 Conditional Writes

• HashiCorp: Backend State Locking

• HashiCorp: terraform force-unlock Command

• AWS Docs: Enabling S3 Versioning

• AWS Docs: S3 Server-Side Encryption

Everyone publishes the controls list. But nobody publishes the week-by-week engineering calendar you'll need to follow to make sure your ducks are in a row.

Here is the exact 90-day timeline — including the mistakes that add 60 days (and how to avoid them).

Table of Contents

1. What You'll Learn

2. Prerequisites

3. Weeks 1–2: The Scope Decision

4. Weeks 3–6: The 14 Controls That Must Be Active on Day 1

5. Weeks 7–10: The Evidence Collection Infrastructure

6. Weeks 11–14: Auditor Selection and Readiness Assessment

7. Weeks 15–18: The Observation Period

8. The 90-Day SOC2 Timeline at a Glance

9. What's Next

10. Resources

What You'll Learn

By the end of this guide, you'll know:

• How to scope your SOC2 boundary correctly — the decision that determines everything else

• The 14 controls that must be active on day 1 of your observation period

• How to build evidence collection infrastructure that runs automatically

• How to choose an auditor and run a readiness assessment

• What happens during the observation period and how to close gaps without restarting the clock

Let's dive in.

Prerequisites

Before following along, you should have:

Knowledge:

• Basic understanding of AWS services (EC2, RDS, S3, IAM, VPC)

• Familiarity with Terraform or another infrastructure as code tool

• Comfort reading GitHub Actions YAML workflows

• A general understanding of what SOC2 is — if you are starting from scratch, read the AICPA's SOC2 overview first

Tools and access:

• An AWS account with administrator access

• A GitHub organisation with admin rights

• Terraform installed (v1.0 or later)

• Python 3.8 or later (for the evidence collector Lambda)

• A compliance automation platform — Vanta or Drata — connected to your AWS account and GitHub organisation

Estimated time: 90 days end-to-end, with active engineering work of approximately 8–12 hours per week in the first six weeks, tapering to 2–4 hours per week during the observation period.

Weeks 1–2: The Scope Decision — What Is In and Out of Your SOC2 Boundary

What Most Teams Get Wrong

Most teams scope their SOC2 boundary too broadly. They include every AWS account, every service, every environment. This is a mistake — and here is exactly why.

A broader scope means more controls to implement, more evidence to collect, and more systems the auditor will examine.

Every system inside your boundary must satisfy all 14 controls. Including your development sandbox means your engineers' experimental environments must have GuardDuty enabled, CloudTrail logging, and branch-protected deployments. That adds weeks of work and months of evidence collection for systems that pose no risk to your customers.

A correctly bounded scope means you include only the systems that store, process, or transmit customer data — and you prove that everything else cannot reach those systems.

Bad scope (over-inclusive):

Entire AWS Organization
├── Production (in scope)
├── Staging (in scope)
├── Development (in scope)
├── Sandbox (in scope)
└── CI/CD (in scope)

Good scope (correctly bounded):

SOC2 Boundary
├── Production AWS Account (in scope)
├── Production EKS Cluster (in scope)
├── Production RDS (in scope)
└── Everything else (OUT of scope — proven by network segmentation)

The correctly bounded scope works because it draws the tightest defensible line around the systems that actually handle customer data. Everything outside that line is excluded — not by assumption, but by technical controls that prevent those systems from reaching anything inside the boundary.

The Scope Decision Framework

For every system in your infrastructure, ask these four questions:

Any system where the answer to even one question is yes belongs inside your boundary.

Network Segmentation — The Technical Proof That Your Boundary Holds

Network segmentation is the practice of dividing your infrastructure into isolated zones so that systems in one zone can't communicate with systems in another unless you explicitly allow it.

In the context of SOC2, it's the technical control that proves your out-of-scope systems genuinely can't reach your in-scope systems — not just by policy, but by infrastructure enforcement.

Without network segmentation, the SOC2 auditor can't trust that your boundary is real. A developer in your sandbox environment who can query your production database means the sandbox is effectively in scope, regardless of what your diagram says.

Here's the Terraform that implements network segmentation between your production and non-production environments. The network access control list (NACL) blocks all inbound traffic from the broader private IP range (10.0.0.0/8) into your in-scope production VPC, while the explicit aws_vpc_peering_connection comment documents the deliberate decision not to peer environments:

# This account has NO VPC peering to non-production environments.
# The absence of peering is itself the segmentation control.
# Do NOT add peering connections to this account without SOC2 scope review.

resource "aws_network_acl" "deny_non_production" {
  vpc_id = aws_vpc.production.id

  # Block all inbound traffic from non-production IP ranges
  ingress {
    rule_no    = 100
    action     = "deny"
    from_port  = 0
    to_port    = 0
    protocol   = "-1"
    cidr_block = "10.0.0.0/8"
  }

  # Allow legitimate inbound traffic (HTTPS from internet)
  ingress {
    rule_no    = 200
    action     = "allow"
    from_port  = 443
    to_port    = 443
    protocol   = "tcp"
    cidr_block = "0.0.0.0/0"
  }

  # Allow all outbound (tighten this per your architecture)
  egress {
    rule_no    = 100
    action     = "allow"
    from_port  = 0
    to_port    = 0
    protocol   = "-1"
    cidr_block = "0.0.0.0/0"
  }

  tags = {
    Name        = "production-nacl"
    Environment = "production"
    Purpose     = "SOC2 network segmentation"
  }
}

Verify the segmentation with this command after applying the Terraform:

# Confirm no VPC peering connections exist from production to non-production
aws ec2 describe-vpc-peering-connections \
  --filters Name=status-code,Values=active \
  --query 'VpcPeeringConnections[*].{ID:VpcPeeringConnectionId,Requester:RequesterVpcInfo.VpcId,Accepter:AccepterVpcInfo.VpcId}' \
  --output table

The Deliverable: Your SOC2 Boundary Diagram

At the end of weeks 1–2, you need a boundary diagram — a visual document that shows every in-scope system, every out-of-scope system, and the segmentation controls between them.

Here is what the diagram should contain:

Include every AWS service, every data flow arrow, and a label on the segmentation control. This diagram becomes your primary scope evidence and is typically the first thing an auditor asks for.

Weeks 3–6: The 14 Controls That Must Be Active on Day 1

These 14 controls must be implemented and actively collecting evidence from day 1 of your observation period. If you add any of them late, the observation period clock for that control restarts from the implementation date — not from day 1 of the audit period.

Think of the observation period as a surveillance camera recording your infrastructure. The auditor watches the footage later. If the camera was not on when a specific event occurred, that event has no record — and the SOC2 control for it has a gap.

Control 1: MFA Enforcement (CC6.6)

Multi-Factor Authentication (MFA) requires a user to verify their identity using two independent factors — something they know (a password) and something they have (a phone or hardware key). Without MFA, a stolen password is sufficient to access your production systems.

SOC2 CC6.6 requires that access to systems is restricted to authorized users. MFA is the technical control that makes "authorized" meaningful. Without it, any password compromise is a production access event.

To implement MFA, you can use AWS IAM Identity Center (formerly SSO) connected to your identity provider (Okta, Google Workspace, or Azure AD). MFA is then enforced at the identity provider level — any user without MFA enrolled can't authenticate, regardless of which AWS service they're trying to reach.

# IAM Identity Center configuration — MFA is enforced at the IdP level.
# No IAM user has direct console or CLI access.
# All access goes through SSO sessions (8-hour expiry by default).

resource "aws_ssoadmin_instance_access_control_attributes" "mfa" {
  instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]

  attribute {
    key = "email"
    value {
      source = ["$${path:email}"]
    }
  }
}

You can verify that no IAM users retain direct console access (which would bypass MFA):

# Any user listed here has direct console access bypassing SSO — investigate immediately
aws iam list-users \
  --query 'Users[?PasswordLastUsed!=`null`].[UserName,PasswordLastUsed]' \
  --output table

Control 2: Infrastructure as Code (CC8.1)

Infrastructure as Code (IaC) means defining your cloud infrastructure in version-controlled code files (Terraform, Pulumi, or AWS CDK) rather than creating resources manually through the AWS console. Every infrastructure change is proposed in a pull request, reviewed by a colleague, and applied through an automated pipeline.

SOC2 CC8.1 covers change management — the requirement that every change to your production environment is documented, reviewed, and approved. Manual console changes produce no audit trail. If an engineer opens the AWS console and creates a security group without going through Terraform, that change is invisible to your SOC2 auditor. IaC makes every change reviewable and traceable.

Now let's see how to implement IaC here. This GitHub Actions workflow applies Terraform only from the main branch, after a pull request has been reviewed and approved. The workflow creates an immutable record of every infrastructure change:

# .github/workflows/terraform-apply.yml
name: Terraform Apply (Production)
on:
  push:
    branches: [main]
    paths: ['terraform/**']

permissions:
  id-token: write   # Required for AWS OIDC authentication
  contents: read

jobs:
  apply:
    name: Apply Infrastructure Changes
    runs-on: ubuntu-latest
    environment: production  # Requires manual approval for production

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Configure AWS credentials (OIDC — no long-lived keys)
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/terraform-apply
          aws-region: us-east-1

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: "1.6.0"

      - name: Terraform Plan
        run: |
          terraform init
          terraform plan -out=tfplan -input=false

      - name: Terraform Apply
        run: terraform apply -input=false tfplan

SOC2 evidence this produces: A GitHub Actions run log for every infrastructure change, showing who triggered it (the pull request author), when it was applied, and what changed.

Control 3: CloudTrail Enabled (CC7.1)

AWS CloudTrail is a service that records every API call made in your AWS account — who called it, when, from which IP address, and whether it succeeded. Think of it as the complete audit log of everything that has ever happened in your AWS environment.

SOC2 CC7.1 requires monitoring for security events. CloudTrail is the foundational logging layer — without it, you can't detect unauthorized access, investigate incidents, or prove to an auditor that your controls were operating as intended. An auditor who can't see historical AWS API activity can't verify that your access controls were enforced during the observation period.

To implement it, you'll want to enable multi-region CloudTrail so that activity in every AWS region is captured, including global services like IAM. You can ship logs to an S3 bucket with Object Lock enabled (Control 3 in the evidence collection section covers this) so logs can't be modified or deleted:

# Enable CloudTrail with log file validation and multi-region coverage
aws cloudtrail create-trail \
  --name production-audit-trail \
  --s3-bucket-name your-cloudtrail-logs-bucket \
  --is-multi-region-trail \
  --enable-log-file-validation \
  --include-global-service-events

# Start the trail (creation alone does not start logging)
aws cloudtrail start-logging --name production-audit-trail

# Verify the trail is active and logging
aws cloudtrail get-trail-status --name production-audit-trail \
  --query '{IsLogging:IsLogging,LatestDeliveryTime:LatestDeliveryTime}'

Control 4: GuardDuty Enabled (CC7.2)

AWS GuardDuty is a threat detection service that analyses your CloudTrail logs, VPC Flow Logs, and DNS logs. It uses machine learning to identify suspicious behaviour — things like an EC2 instance communicating with a known malware server, an IAM user logging in from an unusual country, or unusual API call patterns that indicate credential theft.

SOC2 CC7.2 requires the use of detection tools to identify potential security events. GuardDuty is the monitoring layer that tells you when something anomalous is happening, not just what happened after the fact. Without it, you would only discover a compromise when the damage is done.

Here's the implementation:

# Enable GuardDuty — findings published every 15 minutes for active threats
aws guardduty create-detector \
  --enable \
  --finding-publishing-frequency FIFTEEN_MINUTES

# Verify GuardDuty is active
aws guardduty list-detectors --query 'DetectorIds' --output table

You can set up an EventBridge rule to route CRITICAL and HIGH severity GuardDuty findings to your incident response channel immediately. A finding sitting unreviewed for 90 days is a qualified SOC2 finding.

Control 5: VPC Flow Logs (CC6.1)

VPC Flow Logs capture information about the IP traffic flowing through your Virtual Private Cloud — every accepted and rejected connection, including source IP, destination IP, port, protocol, and whether the traffic was allowed or denied. They are the network-level audit trail that CloudTrail doesn't provide.

SOC2 CC6.1 requires logical access controls and monitoring. VPC Flow Logs let you verify that your network segmentation is actually working (traffic you denied is showing as rejected in the logs), detect unexpected communication between services, and investigate security events at the network layer.

# Create an IAM role for VPC Flow Logs to deliver to CloudWatch
aws iam create-role \
  --role-name vpc-flow-logs-role \
  --assume-role-policy-document '{
    "Version":"2012-10-17",
    "Statement":[{
      "Effect":"Allow",
      "Principal":{"Service":"vpc-flow-logs.amazonaws.com"},
      "Action":"sts:AssumeRole"
    }]
  }'

# Enable VPC Flow Logs for all traffic (ACCEPT and REJECT)
aws ec2 create-flow-logs \
  --resource-ids vpc-YOUR_PRODUCTION_VPC_ID \
  --resource-type VPC \
  --traffic-type ALL \
  --log-group-name /aws/vpc/flow-logs/production \
  --deliver-log-permission-arn arn:aws:iam::YOUR_ACCOUNT_ID:role/vpc-flow-logs-role

# Verify flow logs are active
aws ec2 describe-flow-logs \
  --filter Name=resource-id,Values=vpc-YOUR_PRODUCTION_VPC_ID \
  --query 'FlowLogs[*].{Status:FlowLogStatus,LogGroup:LogGroupName}'

Control 6: Secrets Manager (CC6.7)

Secrets management means storing credentials (database passwords, API keys, certificates, and other sensitive configuration values) in a dedicated, access-controlled service (like AWS Secrets Manager or HashiCorp Vault) rather than in .env files, GitHub repository secrets, or hardcoded in application code.

SOC2 CC6.7 requires protecting sensitive system components from unauthorized access. A secret stored in an .env file committed to a repository is accessible to every developer with repo access, every CI/CD runner, and every engineer who has ever cloned the repo — including those who have since left the company.

A Secrets Manager provides centralised storage, access logging, automatic rotation, and fine-grained IAM permissions so only specific services can retrieve specific secrets.

Let's look at the implementation — storing and rotating a secret:

# Store a database credential with automatic 90-day rotation
aws secretsmanager create-secret \
  --name production/postgresql/credentials \
  --description "Production PostgreSQL credentials — rotated every 90 days" \
  --secret-string '{
    "username": "app_user",
    "password": "REPLACE_WITH_STRONG_PASSWORD",
    "host": "your-rds-endpoint.us-east-1.rds.amazonaws.com",
    "port": 5432,
    "dbname": "production"
  }'

# Enable automatic rotation every 90 days
aws secretsmanager rotate-secret \
  --secret-id production/postgresql/credentials \
  --rotation-rules AutomaticallyAfterDays=90

How your application retrieves the secret at runtime (no hardcoded credentials):

# Good: secret retrieved at runtime from Secrets Manager
import boto3
import json

def get_db_credentials():
    client = boto3.client('secretsmanager', region_name='us-east-1')
    response = client.get_secret_value(SecretId='production/postgresql/credentials')
    return json.loads(response['SecretString'])

# Bad: secret hardcoded in application code or .env file
DB_PASSWORD = "my_database_password_123"  # Never do this

The access log in CloudTrail records every time a secret is retrieved, by which IAM role, at what time. That log is your SOC2 evidence that secrets access is controlled and auditable.

Control 7: EBS Encryption (CC6.1)

EBS (Elastic Block Store) encryption ensures that the persistent disks attached to your EC2 instances and used by your RDS databases are encrypted at rest using AES-256. If an AWS employee or an attacker gained physical access to the storage hardware, the data would be unreadable without the encryption key.

SOC2 CC6.1 requires protecting information assets from unauthorised access. Encryption at rest is the control that protects data in the event of physical storage compromise or an improperly decommissioned disk. Enabling it account-wide means every new EBS volume is encrypted automatically, including RDS storage, EKS node volumes, and EC2 instance root volumes.

# Enable EBS encryption by default for all new volumes in this region
aws ec2 enable-ebs-encryption-by-default

# Verify it is enabled
aws ec2 get-ebs-encryption-by-default \
  --query 'EbsEncryptionByDefault'
# Expected output: true

# Check existing volumes — any showing false need to be migrated
aws ec2 describe-volumes \
  --query 'Volumes[?Encrypted==`false`].[VolumeId,Size,VolumeType]' \
  --output table

Any existing unencrypted volumes must be snapshot-and-replaced. The process: create a snapshot of the unencrypted volume, create a new encrypted volume from the snapshot, and swap it into the instance.

Control 8: S3 Block Public Access (CC6.1)

Amazon S3 buckets can be configured to allow public access — meaning anyone on the internet can read their contents without authentication. Block Public Access is an account-level and bucket-level setting that prevents any bucket from being made public, regardless of the bucket's own policy.

A misconfigured S3 bucket is one of the most common causes of data breaches in cloud environments. Block Public Access at the account level means a developer can't accidentally expose a bucket containing customer data, even if they set the wrong bucket policy. It's a guardrail, not just a policy.

# Block public access at the AWS account level — applies to all buckets
aws s3control put-public-access-block \
  --account-id YOUR_ACCOUNT_ID \
  --public-access-block-configuration \
    BlockPublicAcls=true,\
    IgnorePublicAcls=true,\
    BlockPublicPolicy=true,\
    RestrictPublicBuckets=true

# Verify account-level setting is active
aws s3control get-public-access-block \
  --account-id YOUR_ACCOUNT_ID

# Scan for any buckets that have public access enabled (should be zero)
aws s3api list-buckets --query 'Buckets[*].Name' --output text | \
  tr '\t' '\n' | while read bucket; do
    result=\((aws s3api get-public-access-block --bucket "\)bucket" 2>/dev/null)
    if echo "$result" | grep -q '"BlockPublicAcls": false'; then
      echo "WARNING: $bucket has public access not fully blocked"
    fi
  done

Control 9: Branch Protection (CC8.1)

Branch protection is a GitHub setting that prevents engineers from pushing code directly to your main branch without going through a pull request that has been reviewed and approved by at least one other team member. It also requires your CI pipeline to pass before any code can be merged.

SOC2 CC8.1 requires change management — the requirement that every change to production systems is documented, reviewed, and approved. Without branch protection, an engineer can push directly to main, which deploys directly to production through your CI/CD pipeline, with no review and no audit trail. Branch protection is the technical enforcement of your change management policy.

The critical setting that most teams miss: the "Do not allow bypassing the above settings" option must be enabled. Without it, administrators can bypass branch protection — and a SOC2 auditor will flag this as a gap because it means your change management control can be circumvented.

# .github/settings.yml — enforces branch protection via code
# Requires the settings GitHub App: https://github.com/apps/settings

branches:
  - name: main
    protection:
      required_pull_request_reviews:
        required_approving_review_count: 1
        dismiss_stale_reviews: true
        require_code_owner_reviews: false
      required_status_checks:
        strict: true
        contexts:
          - "CI / test"
          - "Security / trivy-scan"
      enforce_admins: true         # Admins cannot bypass — this is critical
      restrictions: null           # No push restriction beyond the above
      allow_force_pushes: false
      allow_deletions: false

Here's how you can verify that branch protection is enforced and admins can't bypass it:

# Returns the branch protection rules including enforce_admins status
curl -H "Authorization: token YOUR_GITHUB_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/YOUR_REPO/branches/main/protection \
  | jq '{enforce_admins: .enforce_admins.enabled, required_reviews: .required_pull_request_reviews.required_approving_review_count}'

Control 10: Container Image Scanning (CC7.4)

Container image scanning analyses your Docker images before deployment to identify known security vulnerabilities (CVEs) in the operating system packages and application dependencies they contain.

Trivy is an open-source scanner that checks the base image (Ubuntu, Alpine, and so on), all installed OS packages, and language-specific dependencies (npm, pip, Go modules) against the National Vulnerability Database.

SOC2 CC7.4 requires monitoring and identifying vulnerabilities. Every container you deploy contains a base image with OS packages — and those packages regularly receive CVE disclosures. A critical CVE left unpatched for 90 days in a production container is a SOC2 finding. Automated scanning in CI means every image is checked before it can deploy.

# .github/workflows/security-scan.yml
name: Security Scan
on: [push, pull_request]

jobs:
  trivy-scan:
    name: Container Vulnerability Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build container image
        run: docker build -t app:${{ github.sha }} .

      - name: Scan image for vulnerabilities
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: app:${{ github.sha }}
          format: sarif
          output: trivy-results.sarif
          severity: CRITICAL,HIGH
          exit-code: 1          # Fail the pipeline on CRITICAL or HIGH findings

      - name: Upload results to GitHub Security tab
        uses: github/codeql-action/upload-sarif@v2
        if: always()            # Upload even if scan found issues
        with:
          sarif_file: trivy-results.sarif

The scanner looks for:

• CVEs in base image OS packages (for example, a critical OpenSSL vulnerability in your Ubuntu base)

• Vulnerable versions of application dependencies (a known RCE in an npm package your app uses)

• Misconfigurations in the Dockerfile itself (running as root, using latest tags)

Results appear in the GitHub Security tab for your repository, giving you a historical record of every scan — which is your SOC2 evidence.

Control 11: Incident Response Plan (CC9.2)

An incident response plan is a written, tested procedure that defines exactly what your team does when a security event occurs — from the moment an alert fires through to customer notification and post-incident review.

SOC2 CC9.2 requires that you have a documented process for responding to security events and that you've tested it. The auditor will ask for the written runbook and evidence that a tabletop exercise (a simulated incident walkthrough) has been conducted within the observation period.

Your incident response runbook must include:

1. Severity classification: Definitions of P1 (production down, customer data at risk), P2 (degraded service, potential risk), and P3 (minor issue, no customer impact) — and the response SLA for each.

2. Escalation path: Exactly who gets paged at each severity level, with contact details. Not "the on-call engineer" — specific names and a backup if the first person doesn't respond within 10 minutes.

3. First 15 minutes: The specific steps to take immediately — isolate the affected system, assess the scope, notify the incident channel, begin the timeline log.

4. Communication templates: Pre-written Slack messages, customer email templates, and regulatory notification templates (GDPR requires notification within 72 hours, HIPAA within 60 days).

5. Post-incident review: The blameless postmortem process, the 5-why root cause analysis template, and the action item tracking process.

Conduct a tabletop exercise at least once during your observation period: gather your engineering team for 45 minutes, simulate a realistic scenario (for example, "an AWS access key was committed to a public GitHub repo"), and walk through the runbook together. Document the meeting date, attendees, scenario, gaps found, and remediation actions. This document is your evidence.

Control 12: Access Reviews (CC6.3)

An access review is a quarterly audit of who has access to what in your production systems — AWS accounts, GitHub repositories, production databases, and every SaaS tool that touches customer data. You verify that every person on the list still works at the company and still needs the access their role grants them.

SOC2 CC6.3 requires that access is revoked when it's no longer needed. Former employees who retain access to production AWS accounts represent a genuine security risk and a definitive SOC2 finding.

In every access review I've conducted, at least 3–5 former employees or contractors still had active access they should not.

The quarterly access review checklist:

# 1. IAM users — list all with their last login date
aws iam generate-credential-report
aws iam get-credential-report --output text --query Content \
  | base64 --decode | cut -d',' -f1,5 | column -t -s ','

# 2. IAM roles — find roles that have not been used in 90+ days
aws iam get-account-authorization-details \
  --query 'RoleDetailList[*].{Role:RoleName,LastUsed:RoleLastUsed.LastUsedDate}' \
  --output table

# 3. Verify AWS SSO user list matches your current employee list
aws identitystore list-users \
  --identity-store-id YOUR_IDENTITY_STORE_ID \
  --query 'Users[*].{Name:DisplayName,Email:Emails[0].Value}' \
  --output table

Cross-reference the output against your current employee list in your HR system. Document every change made — access removed, permissions reduced, accounts disabled. The documented changes are the evidence that the review was conducted meaningfully, not just as a checkbox exercise.

Control 13: Backup Verification (CC9.5)

Backup verification is the process of actually restoring your backups to confirm they work — not just confirming that backups are being created. A backup that has never been tested doesn't exist from a recovery perspective.

SOC2 CC9.5 requires that recovery procedures are tested. If your production database is corrupted and you discover for the first time during the incident that your automated RDS snapshots can't be restored, you have both a disaster recovery failure and a SOC2 finding.

How to test your RDS backup:

# Step 1: Find your most recent production snapshot
aws rds describe-db-snapshots \
  --db-instance-identifier your-production-db \
  --query 'sort_by(DBSnapshots, &SnapshotCreateTime)[-1].DBSnapshotIdentifier' \
  --output text

# Step 2: Restore the snapshot to a test instance
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier backup-verification-test \
  --db-snapshot-identifier YOUR_SNAPSHOT_ID \
  --db-instance-class db.t3.medium \
  --no-publicly-accessible \
  --tags Key=Purpose,Value=backup-verification Key=Environment,Value=test

# Step 3: Wait for the restore to complete (typically 5–15 minutes)
aws rds wait db-instance-available \
  --db-instance-identifier backup-verification-test

# Step 4: Connect and verify data integrity (spot check key tables)
# Run this against the restored instance
psql -h RESTORED_INSTANCE_ENDPOINT -U your_user -d your_database \
  -c "SELECT COUNT(*) FROM users; SELECT MAX(created_at) FROM orders;"

# Step 5: Document the test result and delete the test instance
aws rds delete-db-instance \
  --db-instance-identifier backup-verification-test \
  --skip-final-snapshot

Document the test date, the snapshot used, the restore time, the data verification query results, and who conducted the test. Run this quarterly at minimum. This documentation is your SOC2 evidence for CC9.5.

Control 14: Change Management Log (CC8.1)

A change management log is the auditable record of every change made to your production environment — what changed, who approved it, and when it was applied.

SOC2 CC8.1 requires that changes to your production environment are authorized and documented. With IaC and GitOps in place, you already have two separate sources of immutable change history that together satisfy this control.

GitHub Pull Request history provides the record of every code and infrastructure change: who opened the PR, who reviewed and approved it, what the CI status was, and when it was merged. This is your change management log for application and infrastructure changes.

ArgoCD sync history provides the record of every deployment to your Kubernetes cluster: which application was synced, from which Git commit, at what time, and whether the sync succeeded.

To export the ArgoCD sync history as evidence:

# Export ArgoCD application sync history as JSON evidence
argocd app history YOUR_APP_NAME --output json > argocd-sync-history-$(date +%Y%m).json

# Upload to your SOC2 evidence bucket
aws s3 cp argocd-sync-history-$(date +%Y%m).json \
  s3://your-soc2-evidence-bucket/change-management/$(date +%Y/%m)/

# For each deployment, the evidence contains:
# - App name, deployed revision (Git commit SHA)
# - Deployment timestamp
# - Initiating user or automated sync
# - Success/failure status

Together, the GitHub PR history and the ArgoCD sync history give the auditor a complete, tamper-evident record of every change to your production environment during the observation period.

Weeks 7–10: The Evidence Collection Infrastructure

Evidence is the difference between passing and failing SOC2.

You might be wondering: what exactly is evidence? In SOC2 terms, evidence is the documentation that proves a specific control was operating correctly during a specific point in time within the observation period. A policy document says you will do something. Evidence proves you did it — and that you did it continuously, not just the week before the audit.

For example:

• For MFA enforcement (Control 1), evidence is a screenshot of your IAM Identity Center MFA settings taken at a specific date during the observation period, combined with an IAM credential report showing zero IAM users with console access.

• For GuardDuty (Control 4), evidence is the GuardDuty console screenshot showing active detectors, plus your documented response to any findings during the period.

• For access reviews (Control 12), evidence is the completed access review document with dates, names, and specific access changes made.

The challenge is collecting this evidence continuously across 3–12 months without spending hundreds of hours on manual work. The solution is automated evidence collection infrastructure.

The Evidence Bucket — Tamper-Proof Storage for Your Audit Evidence

The evidence bucket is an S3 bucket with Object Lock enabled in GOVERNANCE mode. Object Lock prevents any object from being deleted or modified for the retention period you specify — in this case, 365 days. This means once a piece of evidence is uploaded, it can't be altered, even by a user with administrator access (without explicitly overriding the lock, which itself creates an audit trail).

This tamper-evident property is what gives the auditor confidence that the evidence was not created or modified after the fact.

# terraform/soc2-evidence-bucket.tf

resource "aws_s3_bucket" "soc2_evidence" {
  bucket = "\({var.company_name}-soc2-evidence-\){var.environment}"
}

# Block all public access to the evidence bucket
resource "aws_s3_bucket_public_access_block" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Enable versioning so overwrites create new versions, not replacements
resource "aws_s3_bucket_versioning" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Object Lock in GOVERNANCE mode — objects cannot be deleted for 365 days
resource "aws_s3_bucket_object_lock_configuration" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id

  rule {
    default_retention {
      mode = "GOVERNANCE"
      days = 365
    }
  }
}

# Encrypt all evidence at rest
resource "aws_s3_bucket_server_side_encryption_configuration" "soc2_evidence" {
  bucket = aws_s3_bucket.soc2_evidence.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

The Daily Evidence Collector Lambda

This Lambda function runs automatically every day and exports the status of each critical control to a time-stamped JSON file in the evidence bucket. Over your 3–12 month observation period, it creates a daily record proving that your controls were active and operating.

The function checks seven controls automatically: CloudTrail status, GuardDuty status, VPC Flow Logs, S3 public access block, EBS encryption, MFA compliance, and GuardDuty finding count. Each daily snapshot is uploaded with Object Lock enabled so it can't be modified.

# lambda/evidence-collector/handler.py

import boto3
import json
from datetime import datetime, timedelta, timezone

def lambda_handler(event, context):
    """
    Daily SOC2 evidence collector.
    Runs at 00:00 UTC every day via EventBridge scheduler.
    Exports control status to S3 evidence bucket with Object Lock.
    """
    evidence = {
        'collection_timestamp': datetime.now(timezone.utc).isoformat(),
        'collection_date': datetime.now(timezone.utc).strftime('%Y-%m-%d'),
        'account_id': boto3.client('sts').get_caller_identity()['Account'],
        'controls': {}
    }

    # Control 3: CloudTrail status
    cloudtrail = boto3.client('cloudtrail')
    trails = cloudtrail.describe_trails(includeShadowTrails=False)['trailList']
    multi_region_trails = [t for t in trails if t.get('IsMultiRegionTrail')]
    evidence['controls']['cloudtrail'] = {
        'status': 'PASS' if multi_region_trails else 'FAIL',
        'detail': f"{len(multi_region_trails)} multi-region trail(s) active",
        'trails': [t['Name'] for t in multi_region_trails]
    }

    # Control 4: GuardDuty status
    guardduty = boto3.client('guardduty')
    detectors = guardduty.list_detectors()['DetectorIds']
    unresolved_critical = 0
    for detector_id in detectors:
        findings = guardduty.list_findings(
            DetectorId=detector_id,
            FindingCriteria={
                'Criterion': {
                    'severity': {'Gte': 7},  # HIGH and CRITICAL only
                    'service.archived': {'Eq': ['false']}
                }
            }
        )
        unresolved_critical += len(findings['FindingIds'])

    evidence['controls']['guardduty'] = {
        'status': 'PASS' if detectors else 'FAIL',
        'detail': f"{len(detectors)} detector(s) active, {unresolved_critical} unresolved HIGH/CRITICAL findings",
        'unresolved_high_critical': unresolved_critical
    }

    # Control 5: VPC Flow Logs
    ec2 = boto3.client('ec2')
    flow_logs = ec2.describe_flow_logs(
        Filters=[{'Name': 'resource-type', 'Values': ['VPC']},
                 {'Name': 'flow-log-status', 'Values': ['ACTIVE']}]
    )['FlowLogs']
    evidence['controls']['vpc_flow_logs'] = {
        'status': 'PASS' if flow_logs else 'FAIL',
        'detail': f"{len(flow_logs)} active VPC flow log(s)",
        'active_flow_logs': len(flow_logs)
    }

    # Control 7: EBS encryption by default
    ebs_encryption = ec2.get_ebs_encryption_by_default()['EbsEncryptionByDefault']
    evidence['controls']['ebs_encryption_by_default'] = {
        'status': 'PASS' if ebs_encryption else 'FAIL',
        'detail': 'EBS encryption by default is enabled' if ebs_encryption else 'EBS encryption by default is NOT enabled'
    }

    # Control 8: S3 Block Public Access (account level)
    s3control = boto3.client('s3control')
    account_id = boto3.client('sts').get_caller_identity()['Account']
    try:
        pab = s3control.get_public_access_block(AccountId=account_id)['PublicAccessBlockConfiguration']
        all_blocked = all([pab['BlockPublicAcls'], pab['IgnorePublicAcls'],
                           pab['BlockPublicPolicy'], pab['RestrictPublicBuckets']])
        evidence['controls']['s3_block_public_access'] = {
            'status': 'PASS' if all_blocked else 'FAIL',
            'detail': 'All four S3 Block Public Access settings enabled' if all_blocked else 'One or more S3 Block Public Access settings not enabled',
            'configuration': pab
        }
    except Exception as e:
        evidence['controls']['s3_block_public_access'] = {'status': 'FAIL', 'detail': str(e)}

    # Upload evidence to S3 with Object Lock
    s3 = boto3.client('s3')
    evidence_key = f"daily/{evidence['collection_date']}/control-status.json"
    lock_until = datetime.now(timezone.utc) + timedelta(days=365)

    s3.put_object(
        Bucket='YOUR_EVIDENCE_BUCKET_NAME',
        Key=evidence_key,
        Body=json.dumps(evidence, indent=2),
        ContentType='application/json',
        ObjectLockMode='GOVERNANCE',
        ObjectLockRetainUntilDate=lock_until
    )

    # Alert if any control fails
    failed_controls = [k for k, v in evidence['controls'].items() if v['status'] == 'FAIL']
    if failed_controls:
        sns = boto3.client('sns')
        sns.publish(
            TopicArn='YOUR_ALERT_TOPIC_ARN',
            Subject=f'SOC2 Control Failure Detected — {evidence["collection_date"]}',
            Message=f'The following controls failed their daily check:\n\n{json.dumps(failed_controls, indent=2)}'
        )

    return {
        'statusCode': 200,
        'controls_checked': len(evidence['controls']),
        'controls_failed': len(failed_controls),
        'evidence_location': f"s3://YOUR_EVIDENCE_BUCKET_NAME/{evidence_key}"
    }

The GitHub Actions Evidence Workflow

This workflow runs daily and captures evidence that can't be automated through AWS APIs — GitHub-level controls like branch protection status, recent pull request activity, and CI pipeline results. It exports these as JSON files to the same evidence bucket.

# .github/workflows/soc2-evidence.yml
name: SOC2 Evidence Collection
on:
  schedule:
    - cron: '0 1 * * *'   # 01:00 UTC daily (after the Lambda runs at 00:00)
  workflow_dispatch:        # Allow manual trigger when needed

permissions:
  contents: read

jobs:
  collect-github-evidence:
    name: Collect GitHub Control Evidence
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/evidence-collector
          aws-region: us-east-1

      - name: Collect branch protection status
        run: |
          DATE=$(date +%Y-%m-%d)
          mkdir -p evidence/github

          # Export branch protection rules for main
          curl -s -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
            "https://api.github.com/repos/${{ github.repository }}/branches/main/protection" \
            | jq '{
                date: "'$DATE'",
                enforce_admins: .enforce_admins.enabled,
                required_reviews: .required_pull_request_reviews.required_approving_review_count,
                required_status_checks: .required_status_checks.contexts,
                allow_force_pushes: .allow_force_pushes.enabled
              }' > evidence/github/branch-protection-$DATE.json

          echo "Branch protection evidence collected"
          cat evidence/github/branch-protection-$DATE.json

      - name: Upload evidence to S3
        run: |
          DATE=$(date +%Y-%m-%d)
          aws s3 sync evidence/ \
            s3://\({{ secrets.SOC2_EVIDENCE_BUCKET }}/daily/\)DATE/github/ \
            --no-progress
          echo "Evidence uploaded: s3://\({{ secrets.SOC2_EVIDENCE_BUCKET }}/daily/\)DATE/github/"

Weeks 11–14: Auditor Selection and Readiness Assessment

How to Choose a SOC2 Auditor

Selecting the right auditor is more consequential than most teams realize. SOC2 audits are conducted by CPA firms — specifically, firms licensed to issue SOC reports. The right firm has experience with cloud-native, SaaS companies your size. The wrong firm could apply enterprise audit frameworks to a seed-stage startup and generate findings based on controls that aren't appropriate to your context.

Here is what to look for and what to watch out for:

Experience matters more than brand

A large Big Four firm isn't necessarily better than a specialist boutique auditor for a 20-person SaaS company.

Ask specifically: "How many SOC2 audits have you completed in the last 12 months for SaaS companies between 10 and 50 employees?" You want a firm where this is common, not exceptional.

Verify familiarity with your compliance tool

If you're using Vanta or Drata, confirm that the auditor has experience with evidence produced by those platforms. Some auditors prefer to collect evidence directly and are unfamiliar with automated evidence exports. An auditor who doesn't trust your Vanta evidence will ask you to re-collect everything manually.

Understand what Type II actually costs

For a Series A SaaS company, expect \(15,000–\)30,000 for a SOC2 Type II audit with a 3-month observation period. A quote below \(10,000 often means the auditor is cutting corners on the review depth. A quote above \)50,000 for a small company typically means the firm is applying enterprise pricing to a startup engagement.

Get references from similar companies

Ask the auditor for two or three references from SaaS companies they've audited in the last year. Call those references and ask: did the auditor understand cloud infrastructure? Were the findings reasonable? How was the communication during the review?

Here's a summary table of some things to watch out for:

How to Run a Readiness Assessment (Mock Audit)

A readiness assessment is a self-conducted simulation of the real audit, run 2–4 weeks before you engage the auditor. Its purpose is to find and close gaps before the auditor finds them, because gaps found in a mock audit cost you a week of remediation time, while gaps found in the real audit cost you a conditional report and a re-review.

You can run the readiness assessment yourself or hire a consultant to run it. The consultant approach is more valuable because an independent reviewer will find gaps you have rationalised away.

The process:

1. Step 1: Work through every control in the checklist below and attempt to produce the evidence that an auditor would request.

2. Step 2: For every control where you can't produce clear, timestamped evidence: that's a gap. Document it.

3. Step 3: Prioritise gaps by type. Evidence gaps (missing evidence for an active control) require evidence collection infrastructure fixes. Control gaps (a control that isn't implemented) require engineering work.

4. Step 4: Close all gaps before engaging the real auditor.

The one thing most teams skip: Running the readiness assessment against their own evidence bucket. Pull a random day's evidence from the daily Lambda export and verify that it's complete, timestamped, and accurately reflects the control status on that day.

If the evidence file for December 14th shows GuardDuty as PASS but GuardDuty was actually disabled that day, the auditor will find the discrepancy in the AWS account history — and that's a qualified finding.

Weeks 15–18: The Observation Period

How the Auditor Observes Your Controls

The SOC2 auditor doesn't physically visit your office or sit inside your AWS console watching your infrastructure in real time. The audit is a remote, documentation-based process conducted entirely through evidence review.

Here is how it actually works:

First, the auditor provides a list of evidence requests — typically 80–150 items for a Type II audit. You upload the evidence to a shared portal (the auditor provides this — it is usually a secure document sharing platform). The auditor reviews the evidence, asks follow-up questions, and identifies gaps where evidence is missing or a control wasn't operating as described.

For automated controls like CloudTrail and GuardDuty, the evidence is your daily Lambda exports — the auditor spot-checks a sample of daily snapshots across the observation period to verify the controls were consistently active.

For manual controls like access reviews and backup tests, the evidence is the documents you produced when you ran those processes.

The practical implication: the auditor is trusting your evidence. This is why the Object Lock on your evidence bucket matters. It proves to the auditor that the evidence was generated at the time it claims to have been generated and hasn't been modified since.

What the Auditor Reviews Over the Observation Period

What Triggers a Finding

A SOC2 finding is the auditor's documented conclusion that a control wasn't operating effectively during the observation period. Findings range from observations (minor issues that don't affect the audit opinion) to qualified opinions (material failures that result in a qualified rather than unqualified report).

Understanding what triggers findings — and which ones restart the observation period — is critical for managing your audit timeline.

Control gaps occur when a required control isn't implemented or was disabled during the observation period. If you discover in month 2 that MFA wasn't enforced on one IAM user for the first three weeks, you must document the remediation and demonstrate the gap was closed.

Whether this restarts your observation period depends on how long the gap lasted and how the auditor assesses the risk — but a gap of less than 30 days that's immediately remediated and documented typically doesn't restart the clock.

Evidence gaps are more serious. If your daily Lambda evidence collector failed for two weeks and produced no evidence exports, you have a two-week window with no documented proof that your controls were operating. The auditor can't verify controls they can't see evidence for.

Evidence gaps almost always require extending the observation period because there's no way to retroactively produce evidence for a period that wasn't recorded.

Process failures occur when a manual control wasn't executed as documented. The most common is an access review that was skipped. Like control gaps, these can typically be remediated without restarting the clock if they're documented promptly and the remediation is clear.

Unpatched critical CVEs are a special case. If Trivy identifies a CRITICAL vulnerability in a production container and it remains unpatched for more than your documented remediation SLA (typically 30 days for critical, 90 days for high), this is a qualified finding that the auditor will note in the report.

How to Close Gaps Without Restarting the Clock

When you discover a gap during the observation period:

For control gaps:

1. Fix the control immediately — don't wait
2. Document the fix: screenshot, PR link, or CLI command output with timestamp
3. Note the gap date range in your audit log: "Control gap: 2024-03-10 to 2024-03-14 (4 days). Root cause: [X]. Remediated: [Y]. No customer data accessed during gap period."
4. Notify your auditor proactively — they will find it anyway; proactive disclosure is better than defensive explanation
5. The observation period doesn't restart if the gap was short-lived and promptly remediated

For evidence gaps:

1. Fix the evidence collection infrastructure immediately
2. Understand that you can't retroactively generate evidence for the gap period
3. The observation period for affected controls effectively restarts from the date evidence collection resumed
4. If the gap is early in your observation period, you may be able to extend the period rather than restart — discuss with your auditor

The pro tip: Set up a CloudWatch alarm that triggers if the evidence Lambda fails to deliver to S3 on schedule. A missing daily evidence file is caught within 24 hours, not discovered during the audit review.

The 90-Day SOC2 Timeline at a Glance

What's Next?

Start with Week 1. Define your SOC2 boundary. Apply the four-question framework to every system in your infrastructure. Draw the diagram in Excalidraw. Document the network segmentation controls.

Then implement the 14 controls in order, starting with MFA and CloudTrail — the two that most commonly fail audits when they're missing.

Then build your evidence collection infrastructure before the observation period starts. The automated Lambda and GitHub Actions workflow are the difference between a smooth audit and a 60-day extension.

One thing to remember: SOC2 is 20% controls, 30% evidence, and 50% continuous operation. Start early. Automate everything. Run a mock audit before you call the real one.

Resources

The following resources are referenced throughout this guide:

• AICPA SOC2 Overview — The official SOC2 documentation from the American Institute of CPAs, including the Trust Service Criteria

• Vanta — Compliance automation platform that connects to AWS and GitHub to automate evidence collection and track control status

• Drata — Alternative compliance automation platform with similar capabilities to Vanta

• Trivy by Aqua Security — Open-source container and filesystem vulnerability scanner used in Control 10

• Excalidraw — Free, open-source diagram tool for creating the SOC2 boundary diagram

• AWS IAM Identity Center documentation — Official AWS documentation for setting up SSO and MFA enforcement

• GitHub branch protection documentation — Official GitHub documentation for configuring branch protection rules

• ArgoCD documentation — Official ArgoCD documentation for GitOps deployment and sync history

Ayobami Adejumo is a senior platform engineer and FinOps specialist. He writes about SOC2 compliance engineering, Kubernetes cost optimization, and platform engineering.

And yet the applications go out, and nothing comes back.

This is one of the most frustrating experiences in tech. You're genuinely learning, genuinely putting in the time, and you have nothing to show for it in terms of results. You start to wonder if the market is too competitive, if you need one more certification, or if there's some hidden door everyone else found that you're missing.

The truth is simpler and more actionable than any of that: hiring managers can't see your YouTube watch history. They can see your GitHub. Most beginners optimize for learning. Hired candidates optimize for proof.

In this guide, you'll get an honest breakdown of the nine factors hiring managers actually evaluate when they look at a junior cloud or DevOps candidate and a concrete 90-day plan to address each one. By the end, you'll know exactly where you stand and exactly what to do next.

Table of Contents

• The Three Patterns That Keep Beginners Stuck

  • Pattern 1: The Tutorial Loop

  • Pattern 2: The Theory-Practice Gap

  • Pattern 3: Silent Learning

• What Hiring Managers Are Actually Evaluating

• Factor 1: Proof of Work (The Non-Negotiable)

  • The Three Projects That Cover Everything

• Factor 2: System-Level Thinking

• Factor 3: Software Engineering Fundamentals

• Factor 4: Communication Skills

• Factor 5: Consistency Over Intensity

• Factor 6: Networking and Visibility

• Factor 7: Ownership Mindset

• Factor 8: Business Awareness

• Factor 9: Learning Agility

• Your 90-Day Action Plan

• Honest Self-Assessment: Where Do You Stand?

• Conclusion

• References and Recommended Resources

The Three Patterns That Keep Beginners Stuck

Pattern 1: The Tutorial Loop

Week 1: You watch eight hours of Docker content. Week 2: You start an AWS course and get 70% through. Week 3: A Kubernetes series looks interesting, so you start that instead. Week 4: You open LinkedIn and wonder why you're not getting callbacks.

Watching tutorials feels like progress. It's comfortable, passive, and has no failure state. Nothing breaks. Nothing goes wrong.

The problem is that it produces nothing a hiring manager can evaluate. Courses and certifications tell an employer what you've been exposed to. Your GitHub tells them what you can actually do.

Pattern 2: The Theory-Practice Gap

You can explain CI/CD fluently. You've read the Kubernetes documentation. You understand the conceptual difference between a container and a virtual machine.

But you've never taken a simple application, containerized it, connected it to a pipeline, and deployed it to a cloud server with a real URL that someone can visit.

In an interview, "I understand how it works" and "I have built this and here is the link" are not equivalent answers. Hiring managers hear the first version from hundreds of candidates. The second version gets callbacks.

Pattern 3: Silent Learning

This one is perhaps the most painful pattern because the learning is real. You're putting in the work every day but nobody knows. No GitHub activity. No LinkedIn posts. No community presence. Just cold applications sent from job boards to ATS systems that filter you out before a human ever sees your name.

The hard truth: people get hired through people. A hiring manager who has seen your LinkedIn post about a problem you solved is significantly more likely to give your résumé serious attention than a stranger who applied through a portal.

What Hiring Managers Are Actually Evaluating

I've grouped the nine factors that follow into three buckets: Mindset, Execution, and Visibility. The order matters: mindset shapes how you execute, and execution is what powers visibility.

Let's go through each one.

Factor 1: Proof of Work (The Non-Negotiable)

If there's one thing to take from this entire article, it's this: no portfolio means no serious consideration. The most technically capable candidate in the applicant pool is invisible without proof of work.

This isn't about impressing anyone with complexity. It's about demonstrating that you can take a system from zero to deployed, documented, and working.

Here's the checklist every portfolio project should meet before you consider it done:

• It's deployed: there's a real URL you can share, not "it works on my machine"

• It has a CI/CD pipeline: code changes are automatically tested and deployed

• Infrastructure is defined as code: not manually clicked together in the AWS console

• It has monitoring and alerting: you know when it breaks before users tell you

• It's documented: a README explains what it does, how to run it, and how it works

• It's on GitHub publicly: with real commit history showing iterative work

If your project meets all six criteria, you have proof of work. If it meets four of six, you have a project in progress. Finish it before you start applying.

The Three Projects That Cover Everything

You don't need ten projects. You need two to three projects that together demonstrate the full range of DevOps skills.

Project 1 : The Full-Stack Deploy Pipeline

This is the foundational DevOps project every beginner should build first.

Take any simple web application – a Python Flask app, a Node.js API, or even a static site. Containerize it with Docker. Write a CI/CD pipeline that runs tests, builds the Docker image, and deploys to a cloud server automatically on every push to the main branch. You can also set up Nginx as a reverse proxy and add an uptime monitor (UptimeRobot has a free tier).

Tools: GitHub Actions, Docker, AWS EC2 or Render.com, Nginx.

Why it matters to a hiring manager: it proves you can automate a full deployment workflow end-to-end. The hiring manager can visit your URL, see it running, and inspect your pipeline history.

This single project puts you ahead of most applicants who only have course completion screenshots.

Project 2: Infrastructure as Code with Terraform

Write Terraform code that provisions a complete environment: a VPC, public and private subnets, an EC2 instance with properly scoped security group rules, and an S3 bucket for remote state. Destroy it and recreate it from scratch to prove the code actually works. Add a GitHub Actions workflow that runs terraform plan on pull requests and terraform apply on merge to main.

Tools: Terraform, AWS (or Azure/GCP), GitHub Actions.

Why it matters: Infrastructure as Code with Terraform is a required skill at almost every company running cloud infrastructure. Showing you can write, version-control, and automate Terraform demonstrates a core professional competency.

Project 3: Monitoring and Observability Stack

Deploy a monitoring stack using Docker Compose: Prometheus scraping metrics from your application and the host, Grafana dashboards showing CPU, memory, request rates, and error rates, and Alertmanager configured to send alerts to Slack or email when thresholds are crossed. Connect this to your Project 1 application so the pipeline deploys and the monitoring watches it.

Tools: Prometheus, Grafana, Alertmanager, Node Exporter, Docker Compose.

Why it matters: most beginner portfolios have zero observability work. This project immediately signals that you understand production engineering, not just deployment. Any senior DevOps engineer or SRE reviewing your application will notice it and it will set you apart.

Factor 2: System-Level Thinking

This is the mindset that separates a DevOps engineer from someone who just knows a collection of tools. System-level thinking means you can see the whole picture, not just the part you happen to be working on at any given moment.

Here's the mental test hiring managers are running throughout your interview: can you trace a user request from the moment they click a button to the moment they see a response, and explain what happens at every layer in between?

Here's the full journey of a web request, the map of modern infrastructure every DevOps engineer needs to understand:

Why does this matter in an interview? Consider two candidates answering the question: "Tell me about a time something broke in production."

Candidate A: "The website was down."

Candidate B: "The load balancer health checks were failing because the app containers were running out of memory due to a memory leak introduced in the previous deploy. We identified it via memory metrics in Grafana, rolled back, and added a memory limit to the container spec."

Same incident. Completely different answer. System-level thinking is what makes the difference.

Factor 3: Software Engineering Fundamentals

Many beginners rush to learn Kubernetes and Terraform before mastering the foundations that make those tools make sense. This creates a knowledge structure that looks impressive but has no solid base underneath it.

Here are the fundamentals that actually matter and what to do if you have a gap in any of them:

1. Linux and the Command Line

DevOps tools run on Linux. CI/CD jobs run in Linux containers. SSH is the front door to every server. If the terminal makes you uncomfortable, you're not ready for a production environment. This is not a preference, it's a prerequisite.

Start with daily Linux practice. The Linux Foundation's free introductory materials are a solid starting point. And here's a solid freeCodeCamp course on Linux basics.

2. Networking Fundamentals

DNS, TCP/IP, HTTP/HTTPS, load balancing, firewalls, VPCs, subnets these concepts appear in every cloud architecture. Without them, Terraform and Kubernetes are magic boxes. Study the request flow in Factor 2 above until you can draw it from memory without looking.

Here's a computer networking fundamentals course to get you started.

3. Scripting: Bash and Python

CI/CD pipelines are scripts. Automation is scripting. If you cannot write a Bash script that reads a config file, calls an API, and handles errors gracefully your automation ceiling is very low. Fix this by writing one small, useful script every week. Solve real problems with code.

Here's a helpful tutorial on shell scripting in Linux for beginners.

4. Git and Version Control

Not just git commit and git push. Branching strategies, pull requests, merge conflicts, rebasing, and tagging releases are all standard practice in professional DevOps teams. Use Git for everything including your personal learning notes. Practice branching workflows intentionally.

Here's a full book on all the Git basics (and some more advanced topics, too) you need to know.

5. Docker and Containers

Docker is the universal packaging format for modern software. Understanding layers, multi-stage builds, volumes, networking, and container security is the floor not the ceiling. Every project you build should be containerized. Write your Dockerfiles by hand instead of copying them.

Here's a course on Docker and Kubernetes to get you started,

Factor 4: Communication Skills

Technical skills set your ceiling. Communication skills determine how fast you reach it. This is the most consistently underestimated factor among beginner DevOps candidates.

Two candidates with identical technical ability will have very different career outcomes based on how clearly they communicate. Here's what that looks like in practice:

Architecture explanation: Can you describe how your project works to someone who has never seen it? Can you draw the architecture on a whiteboard and walk someone through your design decisions and the trade-offs you made?

Trade-off articulation: "I chose X over Y because..." is one of the most powerful phrases in a technical interview. It shows you understand that every decision has pros and cons and you made a conscious, reasoned choice rather than just copying a tutorial.

Written documentation: A README is your project's cover letter. A well-written README with clear setup instructions, an architecture diagram, and documented decisions demonstrates engineering maturity that most beginners don't show.

Here's a quick test: open your most recent project on GitHub and read the README as if you're a hiring manager seeing it for the first time. Does it answer these questions?

• What does this project do, and why did you build it?

• What does the architecture look like?

• How do I run this locally, and how do I deploy it?

• What decisions did you make, and why?

• What would you improve if you continued working on it?

If you answered "no" to more than two of those rewrite the README before applying anywhere. This single action will meaningfully improve your response rate.

Interview communication: Hiring managers assess communication throughout the entire interview not just your answers. Thinking out loud, structuring your responses, and admitting uncertainty honestly are all evaluated.

Factor 5: Consistency Over Intensity

Hiring managers are pattern recognition machines. They look at your GitHub contribution graph, your LinkedIn activity, and your learning trajectory and form an impression before reading a single word on your résumé.

A binge-learning approach, 10-hour weekends followed by weeks of nothing produces a GitHub graph that tells the wrong story. Thirty minutes of focused daily practice for six months beats a monthly 10-hour binge. At the six-month mark, the daily practitioner has 90 hours of focused work. The binge learner has 60 with significantly worse retention.

Here's how to build consistency in practice:

• Pick a time slot in your day that you will protect. Thirty minutes is enough to make progress.

• Define a four-week learning sprint with a specific goal, not "learn Terraform" but "build and deploy a VPC with Terraform and write the README."

• Keep a private learning journal: date, what you studied, what you built, what confused you.

• When the sprint ends, evaluate what you built and plan the next one.

What to avoid: declaring publicly on LinkedIn that you're "grinding DevOps full time" and then disappearing for six weeks. The absence is noticed. Only commit publicly to what you will actually sustain.

Factor 6: Networking and Visibility

This is the factor most beginners resist most, and the one that makes the biggest practical difference in time-to-hire.

Most DevOps jobs are filled through people referrals, community connections, LinkedIn conversations. A warm introduction from someone who has seen your work outweighs fifty cold applications every time.

Here are three ways to build visibility without it feeling performative:

Community Engagement

Join communities where DevOps engineers actually talk: AWS User Groups, local DevOps meetups, DevOps Discord servers, Reddit communities like r/devops and r/kubernetes. You don't need to be the expert. Ask specific questions, answer what you genuinely know, and show up consistently. After three to six months, people will recognize your name.

LinkedIn Content

Post once per week about something you learned, built, or got stuck on. Not marketing – documentation. A post that says "This week I configured Prometheus alerting for a Docker Compose stack. Here's what tripped me up and how I solved it" attracts recruiters, leads to conversations, and builds a searchable record of your growth over time.

Asking Good Questions in Public

When you get stuck and figure it out, write it up. Post the solution in the same community where you asked the question. Answer someone else's version of the same question later. You position yourself as a helpful, engaged learner, exactly who hiring managers want to hire.

Here's a concrete three-month visibility sprint to follow:

By month three, recruiters searching for "DevOps" in your location will encounter your activity. Some of the best entry-level DevOps opportunities come from exactly this kind of low-pressure visibility.

Factor 7: Ownership Mindset

This factor is less about personality type and more about observable behavior. Hiring managers are looking for evidence that you finish what you start not just that you start things.

Here's what the contrast looks like:

Ownership mindset has three components. First, finish things: a complete, simple project is worth ten times more than ten incomplete complex ones. Second, take responsibility without blame when something breaks: ownership means identifying the cause, fixing it, and adding monitoring so it doesn't happen again. Third, self-direct your learning you don't wait for someone to tell you what to learn next. You see a gap, identify how to close it, and close it. This is what "junior who can work independently" actually means in job descriptions.

Factor 8: Business Awareness

Technical skill gets you in the door. Business awareness keeps you there and accelerates your career.

The core question hiring managers are testing is: can you connect your technical decisions to cost, uptime, and user impact? Infrastructure decisions are business decisions. Cloud costs are typically the second-largest engineering expense at most companies after salaries. A misconfigured auto-scaling group or a forgotten large EC2 instance can burn thousands of dollars overnight.

Here are a few benchmark questions worth being able to answer comfortably:

• If your company has a 99.9% SLA, how many minutes of downtime per month is that? (About 43 minutes.)

• If you move workloads from on-demand EC2 instances to Reserved Instances, what's the approximate cost saving? (Around 40–60%.)

• If your CI/CD pipeline takes 45 minutes per build and you run 20 builds per day, how much developer wait time does that represent weekly?

Most junior candidates can't answer these fluently in an interview. Candidates who can stand out immediately not because the questions are hard, but because so few people bother to connect infrastructure and business.

The simple habit to build: whenever you describe a technical decision in your project documentation or in an interview, add the business dimension. "I configured auto-scaling" becomes "I configured auto-scaling to handle traffic spikes, which eliminated the cost of over-provisioning and reduced our estimated monthly cloud spend by approximately $X."

Factor 9: Learning Agility

Everyone claims to be a fast learner. It's the most overused phrase in technology job applications. Here's how to make it actually mean something.

Saying "I'm a fast learner" in an interview is table stakes. The question is whether you can prove it. Proof sounds like this: "I had never used GitHub Actions before. I needed a CI/CD pipeline for a project I was building. In 48 hours, I had a working pipeline that runs tests, builds a Docker image, and deploys to AWS."

What makes that credible: it names a specific tool, a specific timeframe, and a specific outcome. There is a GitHub repository with a commit history and a working pipeline that a hiring manager can actually look at.

Learning agility is not about knowing many tools shallowly. It's about picking up new tools quickly because you deeply understand the underlying concepts. Tool names change every few years. Concepts networking, automation, observability, reliability do not.

To build a concrete track record of learning agility: once a month, pick one tool you haven't used. Follow its quick-start guide. Build something small. Document what was difficult. Post about it. This is your learning agility portfolio visible, dated, and specific.

Your 90-Day Action Plan

Here is a concrete, sequential plan that takes you from where you are now to your first DevOps interview-ready state.

Month 1: Build Your Foundation

Focus entirely on Project 1 from the Proof of Work section. Build it completely. Deploy it. Get the live URL. Don't start Project 2 until Project 1 meets all six checklist criteria.

Alongside the build: 30 minutes of Linux and Bash scripting practice daily. This isn't optional, it's the foundation everything else runs on.

Month 2: Expand Your Execution and Start Your Visibility

Begin Project 2 (Terraform IaC). Write your first LinkedIn post, it doesn't need to be polished, it needs to be specific. Join one community and introduce yourself.

Month 3: Complete the Portfolio and Document Everything

Finish all three projects to full checklist standard. Polish every README. Add architecture diagrams. Optimize your GitHub profile, pin your three best repos, write a profile README that describes who you are and what you build, and add links to your live project URLs.

Month 4 Onward: Apply with Strategy

Don't start applying before month four. Apply with real proof of work in hand. Target five to ten quality applications per week rather than spraying a hundred. Include your GitHub and your best project's live URL in every application. For roles at companies where you have a community connection, reach out to that person before applying.

Track every application in a spreadsheet: company, role, date applied, status, outcome, notes. After thirty applications, you'll have enough data to see what's working and what isn't.

Here's the full 90-day breakdown:

Honest Self-Assessment: Where Do You Stand?

Go through each statement below. Be completely honest: this is for you, not anyone else.

Count your "no" answers. Each one is a specific, actionable gap, not a vague sense of being behind. That's the difference between this self-assessment and the anxious feeling of "I'm not ready yet." You're not behind. You just have a prioritized list of what to build next.

Conclusion

Here's what you know now that most beginners still don't:

The gap between you and a DevOps job isn't a gap in certifications, a gap in courses completed, or a gap in the number of tools you've heard about. It's a gap in proof of work, visibility, and the consistency with which you execute.

Hiring managers aren't looking for someone who has watched everything. They're looking for someone who has built something, documented it, deployed it, monitored it, and can clearly explain every decision they made along the way.

The path isn't secret. It's just work. Build two to three complete projects that meet the full checklist. Document everything. Show up consistently in communities and on LinkedIn. Apply with strategy. Iterate based on feedback.

If you want a production-grade reference to support your DevOps journey complete with real Terraform modules, CI/CD workflow templates, infrastructure runbooks, and platform engineering patterns used in real startup environments The Startup DevOps Field Guide was built for exactly this stage of your career.

The information gap between you and your first DevOps role is smaller than you think. The execution gap is where the work is. Start today.

References and Recommended Resources

• roadmap.sh/devops: The community-maintained DevOps learning roadmap. Use this to sequence what you learn next and avoid random jumps between topics.

• DORA State of DevOps Report: Free annual report on what DevOps practices actually improve software delivery performance. Gives you the vocabulary hiring managers speak.

• Linux Foundation - Introduction to Linux: Free introductory Linux course. If the terminal still makes you nervous, start here.

• The Phoenix Project: A business novel about DevOps transformation. Teaches core concepts through story. Gives you vocabulary for business-aware conversations.

• ExplainShell.com: Paste any command you find online and see exactly what every part does. Use this constantly while building your projects.

• GitHub - How to Write a Good README: Official GitHub guidance on repository documentation.

• Prometheus Documentation: Official docs for the monitoring tool used in Project 3.

• Terraform Getting Started - AWS: Official step-by-step guide for Project 2.

• GitHub Actions Documentation: Complete reference for building CI/CD pipelines in Project 1.

• freeCodeCamp - Learn Linux for Beginners: Comprehensive Linux guide available on freeCodeCamp.

Here's why: static credentials don't expire on their own. If they get leaked through a misconfigured workflow, a public fork, or a compromised repository, an attacker has persistent access to your AWS environment until you manually rotate them. And most teams don't rotate them often enough.

OpenID Connect (OIDC) solves this entirely. Instead of storing long-lived credentials, GitHub Actions requests a short-lived token directly from AWS every time your workflow runs. No secrets to rotate. No credentials to leak. No manual key management.

In this tutorial, you'll learn how to set up OIDC authentication between GitHub Actions and AWS from scratch. By the end, your workflows will authenticate to AWS securely without storing a single access key.

Table of Contents

• What Is OpenID Connect (OIDC)?

• How OIDC Works Between GitHub Actions and AWS

• Prerequisites

• Step 1: Create an IAM OIDC Identity Provider in AWS

  Step 2: Create an IAM Role with a Trust Policy

  Step 3: Attach Permissions to the IAM Role

  Step 4: Store the Role ARN as a GitHub Actions Variable

  Step 5: Configure Your GitHub Actions Workflow

  Step 6: Run and Verify Your Workflow

• Security Best Practices

• Troubleshooting Common Errors

• Conclusion

• References

What Is OpenID Connect (OIDC)?

OpenID Connect is an identity protocol built on top of OAuth 2.0. It allows systems to verify identity through tokens rather than shared secrets.

In the context of GitHub Actions and AWS:

• GitHub acts as the identity provider (IdP). It issues a signed JWT (JSON Web Token) for each workflow run.

• AWS acts as the service provider. It validates that token against GitHub's public keys and exchanges it for temporary AWS credentials. The credentials AWS returns are short-lived (valid for up to 1 hour by default) and scoped to exactly the IAM role you define. When the workflow ends, those credentials are gone.

This model is called federated identity. It's the same concept used when you "Sign in with Google" on a third-party website. The difference is that instead of a user signing in, your workflow is the one authenticating.

How OIDC Works Between GitHub Actions and AWS

Before writing a single line of YAML, it beneficial to understand the flow. This is my personal approach when implementing new technologies or concepts. Here's what happens every time your workflow runs:

The diagram illustrates a secure authentication flow between GitHub Actions and AWS using OpenID Connect (OIDC), eliminating the need to store long-lived AWS credentials in GitHub. Here's what happens step-by-step:

1. Initial Authentication Request

When your GitHub Actions workflow starts, the runner (the virtual machine executing your workflow) requests a JSON Web Token (JWT) from GitHub's OIDC provider located at https://token.actions.githubusercontent.com.

2. Token Issuance

GitHub's OIDC provider generates and signs a JWT containing important claims (metadata) about your workflow. These claims include details like which repository the workflow is running from, which branch triggered it, what environment it's running in, and other contextual information that proves the workflow's identity.

3. Token Validation

The GitHub Actions runner presents this signed JWT to AWS Security Token Service (STS). AWS STS validates the JWT's signature by checking it against GitHub's publicly available cryptographic keys, ensuring the token is authentic and hasn't been tampered with.

4. Trust Policy Verification

AWS STS checks the trust policy configured on your IAM Role. This trust policy specifies which GitHub repositories, branches, or environments are allowed to assume this role. If the claims in the JWT match your trust policy conditions, authentication succeeds.

5. Temporary Credentials Issued

Once validated, AWS STS returns temporary security credentials to the GitHub Actions runner. These credentials include an Access Key ID, Secret Access Key, and Session Token that are valid for a limited time (typically 1 hour by default, configurable up to 12 hours).

6. AWS API Access

The GitHub Actions runner uses these temporary credentials to authenticate API calls to your AWS resources such as pushing Docker images to ECR, updating ECS services, writing to S3 buckets, or invoking Lambda functions.

The key point: AWS never sees your GitHub credentials, and GitHub never sees your AWS credentials. The JWT is the only thing exchanged and it's signed, scoped, and short-lived.

Prerequisites

Before you start, make sure you have the following in place:

• An AWS account with IAM permissions to create identity providers and roles

• A GitHub repository (public or private) where your workflows will run

• Basic familiarity with GitHub Actions, knowing how to write a .yml workflow file

• Basic familiarity with AWS IAM roles, policies, and permissions

• The AWS CLI installed and configured (optional, but useful for verification). You don't need to be an AWS expert. Each step includes the exact console path and the configuration values you need.

Step 1: Create an IAM OIDC Identity Provider in AWS

The first thing you need to do is tell AWS to trust GitHub as an identity provider. This is a one-time setup per AWS account.

How to Do It in the AWS Console

1. Open the AWS IAM Console

2. In the left sidebar, click Identity providers

3. Click Add provider

4. For Provider type, select OpenID Connect

5. For Provider URL, enter:

https://token.actions.githubusercontent.com

6. For Audience, enter:

sts.amazonaws.com

7. Click Add provider

How to Do It with the AWS CLI

If you prefer the terminal, run this command:

aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \

Once created, you'll see token.actions.githubusercontent.com listed under Identity providers in your IAM console. This provider will be referenced in your IAM role's trust policy in the next step.

Step 2: Create an IAM Role with a Trust Policy

Now you need an IAM role that your GitHub Actions workflow will assume. The trust policy on this role controls which repositories and branches are allowed to request credentials.

How to Create the IAM Role in the AWS Console

1. Open the AWS IAM Console

2. In the left sidebar, click Roles

3. Click Create role

4. For Trusted entity type, select Web identity

5. For Identity Provider, choose: token.actions.githubusercontent.com which you created earlier.

6. For Audience, choose sts.amazonaws.com as well

7. For GitHub organisation, enter your GitHub username or organization name

8. For GitHub repository, enter your GitHub repository

9. For GitHub branch, enter your branch name (for example, main)

10. Click Next, then Next, give a name to the role and click create role

Note: Creating the IAM role using this approach already establishes the Trusted Entities using a trusted policy based on the step 4-9 above. You can verify this by clicking on the created role and navigating to Trust relationships.

How to Create the IAM Role with the AWS CLI

First, you'll need to create a trust policy document on your local machine: You can call it trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:YOUR_GITHUB_ORG/YOUR_REPO_NAME:*"
        }
      }
    }
  ]
}

Replace the following placeholders before saving:

How to Understand the sub Condition

The sub (subject) claim in the JWT tells AWS exactly where the request is coming from. The value repo:your-org/your-repo:* means any branch in that repository can assume this role.

You can tighten this further depending on your needs:

# Only the main branch
"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
 
# Only a specific GitHub Environment
"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:environment:production"

Scoping this correctly is one of the most important security decisions in this setup. Here's how to decide:

• Use ref:refs/heads/main if only your main/production branch should deploy to AWS. This is the most restrictive and secure option: feature branches can't accidentally (or maliciously) trigger deployments or modify production resources.

• Use environment:production if you're using GitHub Environments with protection rules (required reviewers, deployment gates). This lets you control deployments through GitHub's approval workflow while still restricting which workflows can access AWS.

• Use repo:your-org/your-repo:* (wildcard) only if you need any branch to deploy. for example, in development environments where every feature branch deploys to its own isolated stack. Never use this for production roles.

Run this command to create the role using your trust policy:

aws iam create-role \
  --role-name GitHubActionsOIDCRole \
  --assume-role-policy-document file://trust-policy.json \
  --description "Role assumed by GitHub Actions via OIDC"

Take note of the Role ARN in the output. It will look like this:

arn:aws:iam::YOUR_ACCOUNT_ID:role/GitHubActionsOIDCRole

You'll need this ARN in your workflow YAML in Step 4.

Step 3: Attach Permissions to the IAM Role

The IAM role can now authenticate, but it has no permissions yet. You need to attach a policy that defines what your workflow is actually allowed to do in AWS.

How to Apply the Principle of Least Privilege

Only grant the permissions your workflow genuinely needs. If your workflow deploys to S3, give it S3 permissions. If it pushes images to ECR, give it ECR permissions. Never attach AdministratorAccess to a CI/CD role.

Option 1: Attach an AWS managed policy (quick start):

aws iam attach-role-policy \
  --role-name GitHubActionsOIDCRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

Option 2: Create a custom policy scoped to a specific S3 bucket (recommended for production):

This approach is recommended for production because it limits the blast radius of a security incident. If your workflow credentials are ever compromised, a custom policy scoped to a specific bucket means an attacker can only affect that single bucket not every S3 bucket in your AWS account. It also prevents accidental misconfigurations in your workflow from impacting unrelated resources.

Create a file called s3-deploy-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

Then create and attach it:

aws iam create-policy \
  --policy-name GitHubActionsS3DeployPolicy \
  --policy-document file://s3-deploy-policy.json
 
aws iam attach-role-policy \
  --role-name GitHubActionsOIDCRole \
  --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/GitHubActionsS3DeployPolicy

Note: You can as well implement Step 3 via the console.

Reference: For a full list of available AWS IAM actions, see the AWS IAM actions reference.

Step 4: Store the Role ARN as a GitHub Actions Variable

Before you configure your workflow, you need to make the Role ARN available to it. You'll store it as a repository variable in GitHub, not a secret, because the ARN itself isn't sensitive data.

How to Add the Variable in Your Repository

First, open your GitHub repository and click Settings:

In the left sidebar, scroll down to Secrets and variables, then click Actions:

Then click the Variables tab (not Secrets). Click New repository variable – you can set the Name to:

AWS_ROLE_ARN

Set the Value to your Role ARN from Step 2, for example:

arn:aws:iam::YOUR_ACCOUNT_ID::role/GitHubActionsOIDCRole

Click Add variable:

You'll reference this variable in your workflow in the next step using ${{ vars.AWS_ROLE_ARN }}.

Step 5: Configure Your GitHub Actions Workflow

With AWS and GitHub fully configured, you now need to update your workflow to request an OIDC token and use it to authenticate.

How to Set the Required Workflow Permissions

Your workflow must declare id-token: write. Without this, GitHub won't issue an OIDC token to the runner.

permissions:
  id-token: write   # Required to request the OIDC JWT
  contents: read    # Required to checkout the repository

Important: If you set permissions at the job level, they override any top-level permissions. Make sure id-token: write is present at whichever level your AWS authentication step runs.

Full Workflow Example

Here's a complete workflow that authenticates to AWS using OIDC and deploys a static site to S3:

name: Deploy to AWS S3
 
on:
  push:
    branches:
      - main
 
permissions:
  id-token: write
  contents: read
 
jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
 
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          aws-region: us-east-2
 
      - name: Verify AWS identity
        run: aws sts get-caller-identity
 
      - name: Deploy to S3
        run: |
          aws s3 sync ./code s3://your-bucket-name

Replace the following before committing:

You can see the code sample in my GitHub Repo here.

Note: The aws-actions/configure-aws-credentials action handles the entire OIDC token exchange automatically. It requests the JWT from GitHub, calls sts:AssumeRoleWithWebIdentity, and exports the temporary credentials as environment variables for the rest of the job.

See the action's official documentation for all available options.

Step 6: Run and Verify Your Workflow

Push your workflow to the main branch and open the Actions tab in your repository to watch it run.

What a Successful Run Looks Like

The Configure AWS credentials via OIDC step should show:

Assuming role with OIDC: arn:aws:iam::YOUR_ACCOUNT_ID:role/GitHubActionsOIDCRole

The Verify AWS identity step (aws sts get-caller-identity) should return:

{
    "UserId": "AROA...:GitHubActions",
    "Account": "YOUR_ACCOUNT_ID",
    "Arn": "arn:aws:sts::YOUR_ACCOUNT_ID:assumed-role/GitHubActionsOIDCRole/GitHubActions"
}

If you see an assumed-role ARN in the output, OIDC is working correctly. Your workflow is now authenticating to AWS without a single stored credential.

Security Best Practices

Getting OIDC working is step one. Locking it down properly is step two.

Scope the sub Condition as Tightly as Possible

Don't use a wildcard like repo:your-org/*:* that allows any repository in your organization to assume the role. Scope it to the exact repository and branch that needs access.

"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"

Use GitHub Environments for Production Deployments

GitHub Environments let you add manual approval gates and restrict which branches can deploy. When combined with OIDC, you can scope your trust policy to only allow the production environment:

"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:environment:production"

Apply Least-Privilege Permissions to Every IAM Role

Never attach AdministratorAccess or PowerUserAccess to a role used by CI/CD. Define a custom policy with only the actions your workflow actually needs.

Create Separate IAM Roles Per Environment

A staging role and a production role should have different permission scopes. Your staging deployment role should never have write access to production resources.

Enable AWS CloudTrail

Every call made using the temporary credentials is logged in CloudTrail under the assumed role ARN. This gives you a full audit trail of exactly what your workflow did in AWS.

Reference: GitHub's official security hardening guide for OIDC: About security hardening with OpenID Connect

Troubleshooting Common Errors

Error: Not authorized to perform sts:AssumeRoleWithWebIdentity

This usually means the trust policy on your IAM role doesn't match the sub claim in the JWT.

Check the following:

• The sub condition exactly matches your repository path (it is case-sensitive)

• The aud condition is set to sts.amazonaws.com

• The Federated principal uses the correct AWS account ID

To inspect the actual token claims your workflow is receiving, add this debug step temporarily:

- name: Print OIDC token claims
  run: |
    TOKEN=\((curl -s -H "Authorization: Bearer \)ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
      "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r '.value')
    echo $TOKEN | cut -d '.' -f2 | base64 -d 2>/dev/null | jq .

Error: Could not load credentials from any providers

This almost always means id-token: write is missing from your workflow permissions. Double-check that you have:

permissions:
  id-token: write
  contents: read

Error: AccessDenied When Calling an AWS Service

Authentication succeeded but the IAM role doesn't have permission to perform the action your workflow is attempting. Check the permissions policy attached to your role and compare it against the specific action in the error message.

Conclusion

You've gone from storing static, long-lived AWS credentials in GitHub Secrets to a fully keyless authentication setup using OIDC. Here's what you accomplished:

• Registered GitHub as a trusted OIDC identity provider in AWS.

• Created an IAM role with a scoped trust policy tied to a specific repository.

• Attached least-privilege permissions to that role.

• Configured your GitHub Actions workflow to request and use short-lived AWS credentials.

• Verified the authentication flow end-to-end.

This pattern works across every AWS service from S3, ECS, Lambda, ECR, Secrets Manager, and more. The workflow example here uses S3, but you only need to swap out the permissions policy and the deployment commands to adapt it for any service.

If you want to go further, explore:

• Configuring OIDC for multiple cloud providers: Azure, GCP, and HashiCorp Vault.

• GitHub Environments and deployment protection rules: for multi-stage pipelines with approval gates.

• AWS IAM Access Analyzer: to validate and tighten your role policies automatically.

If you're building out your DevOps practice and want a complete, production-ready reference for infrastructure automation, CI/CD, and platform engineering, check out The Startup DevOps Field Guide. It covers the patterns, templates, and runbooks I've used across real AWS environments.

You can also connect with me on LinkedIn

References

• GitHub Docs: About security hardening with OpenID Connect

• GitHub Docs: Configuring OpenID Connect in Amazon Web Services

• AWS Docs: Creating OpenID Connect (OIDC) identity providers

• AWS Docs: AssumeRoleWithWebIdentity API Reference

• aws-actions/configure-aws-credentials - GitHub

• AWS IAM Actions Reference

• AWS CloudTrail User Guide

You’ll learn how to expose services on a custom domain with auto-renewing HTTPS, and implement a smart deployment strategy that detects changes and redeploys only the affected microservices. This helps avoid unnecessary full-stack redeploys. We'll also cover real production issues and the exact fixes for each one.

Table of Contents

• 1. What you'll build

• 2. Architecture

• 3. Server prerequisites

• 4. Traefik — the reverse proxy

• 5. Run Jenkins in Docker

• 6. Expose Jenkins on a domain via Traefik

• 7. First-time Jenkins setup

• 8. Add the GitHub credential

• 9. Create the pipeline job

• 10. The Jenkinsfile (deploy only what changed)

• 11. End-to-end test

• 12. Troubleshooting — every error we hit

• 13. Mental model: host vs. container

• 14. Daily operations cheat sheet

• 15. What I'd do differently next time

• Closing thoughts

1. What You'll Build

In this tutorial, you'll build a Jenkins instance running inside Docker on the same Linux server as your application stack.

Traefik will act as a reverse proxy in front of Jenkins, exposing it via a clean URL (https://jenkins.example.com) with auto-renewing Let's Encrypt certificates.

You'll also create a Jenkinsfile in your application repository that:

• Automatically triggers on every push to the staging branch,

• Detects which microservices changed in each commit,

• Pulls the latest code on the host machine,

• Rebuilds and restarts only the affected services.

On every push, only the relevant services are redeployed.

Prerequisites

Before jumping in, this guide assumes you’re already comfortable with a few core concepts and tools.

This isn't a beginner-level tutorial — we’ll be working directly with infrastructure, containers, and CI/CD pipelines.

You should be familiar with:

• Basic Linux commands (SSH, file system navigation, permissions)

• Docker fundamentals (images, containers, volumes, networks)

• Git workflows (clone, pull, branches)

• General idea of CI/CD pipelines

Tools and environment required:

• A Linux server (Ubuntu recommended)

• Docker Engine + Docker Compose (v2)

• A domain name (for Traefik + HTTPS)

• GitHub repository (for your backend project)

• Basic understanding of microservices architecture

If you’re comfortable with the above, you’re ready to follow along.

2. Architecture

Here's an overview of the architecture:

┌──────────────────────────── Linux server (Ubuntu) ────────────────────────────┐
│                                                                               │
│   /home/developer/projects/                                                  │
│       └── project-prod-configs/             ← infra repo (compose, Traefik) │
│              ├── docker-compose.staging.yml                                   │
│              ├── traefik.staging.yml                                          │
│              └── project-backend/          ← app repo (services, gateways) │
│                     ├── Jenkinsfile                                           │
│                     ├── docker-compose.staging.yml                            │
│                     └── apps/                                                 │
│                            ├── services/<name>/                               │
│                            ├── gateways/<name>/                               │
│                            └── core/<name>/                                   │
│                                                                               │
│   ┌─────────────────────── Docker network: proxy ──────────────────────┐      │
│   │  traefik (80, 443)                                                 │      │
│   │     │                                                              │      │
│   │     ├──► jenkins  (projects-jenkins-staging)                     │      │
│   │     │      ↳ /projects  ← bind-mount of the host project tree     │      │
│   │     │      ↳ /var/run/docker.sock ← controls host Docker           │      │
│   │     │                                                              │      │
│   │     └──► your services & gateways (built by the pipeline)          │      │
│   └────────────────────────────────────────────────────────────────────┘      │
│                                                                               │
└───────────────────────────────────────────────────────────────────────────────┘
            ▲
            │  webhook on push
            │
   GitHub: <org>/project-backend (branch: staging)

There are two key ideas here:

1. Jenkins runs in a container, but it controls the host's Docker by mounting /var/run/docker.sock. It also bind-mounts the project folder as /projects/..., so it can cd into the real code on the host and run docker compose there.

2. The Jenkinsfile lives inside the app repo, so the pipeline definition is versioned with the code. Jenkins simply points at it.

3. Server Prerequisites

Before we start configuring Jenkins or Traefik, we need to prepare the server properly.

In this step, we’ll:

• Create a dedicated Linux user for managing the project

• Install Docker and Docker Compose

• Set up the folder structure for our repositories

This ensures our CI/CD pipeline runs in a clean and predictable environment.

# Linux user that owns the project tree
sudo adduser developer

# Docker engine + Compose plugin
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker developer

# Sanity check Compose v2
docker compose version
# -> Docker Compose version v2.x.y

# Find where the Compose plugin binary lives — write it down, you'll need it
ls /usr/libexec/docker/cli-plugins/docker-compose
# (some distros use /usr/lib/docker/cli-plugins/docker-compose)

# Project layout
sudo mkdir -p /home/developer/project
sudo chown -R developer:developer /home/developer/project

# Clone both repos in the right place
cd /home/developer/projects
git clone https://github.com/<org>/projects-prod-configs.git
cd projects-prod-configs
git clone -b staging https://github.com/<org>/projects-backend.git

You should now have:

/home/developer/projects/projects-prod-configs/projects-backend

Memorize this path — your Jenkinsfile references it.

DNS

Point an A-record for your Jenkins subdomain to the server's public IP before the next steps so Let's Encrypt can validate via HTTP challenge:

jenkins.example.com   A   <server-public-ip>

4. Traefik — the Reverse Proxy

Traefik acts as the entry point to your entire system. Instead of exposing each service manually with ports, Traefik automatically:

• Routes traffic based on domain names

• Generates and renews HTTPS certificates using Let’s Encrypt

• Connects to Docker and detects services dynamically

In simple terms, Traefik lets you access services like:

https://jenkins.example.com
https://api.example.com

…without manually configuring NGINX or managing SSL certificates.

In this setup, Traefik watches Docker containers and routes traffic using labels we'll define later.

Traefik gives every container a real domain and a real cert with zero per-service config — you just add a few labels.

traefik.staging.yml (static config)

Put this at the root of your infra repo:

api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: web
      email: admin@example.com           # ← change me
      storage: /etc/traefik/acme.json

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false              # only containers with traefik.enable=true
    network: proxy
  file:
    directory: /etc/traefik/dynamic
    watch: true

log:
  level: INFO

accessLog: {}

The Traefik service in docker-compose.staging.yml

networks:
  proxy:
    name: proxy
    driver: bridge
  internal:
    name: internal
    driver: bridge

volumes:
  acme-data:
  traefik-logs:
  jenkins-data:

services:
  traefik:
    image: traefik:v2.11
    container_name: projects-traefik-staging
    restart: unless-stopped
    ports:
      - "80:80"        # HTTP (auto-redirects to HTTPS)
      - "443:443"      # HTTPS
      - "8080:8080"    # Traefik dashboard (internal only — protect via firewall)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.staging.yml:/etc/traefik/traefik.yml:ro
      - ./dynamic:/etc/traefik/dynamic:ro
      - acme-data:/etc/traefik           # persists Let's Encrypt certs
      - traefik-logs:/var/log/traefik
    networks:
      - proxy
    command:
      - '--api.insecure=false'
      - '--api.dashboard=true'
      - '--providers.docker=true'
      - '--providers.docker.exposedbydefault=false'
      - '--providers.docker.network=proxy'
      - '--entrypoints.web.address=:80'
      - '--entrypoints.websecure.address=:443'
      - '--entrypoints.web.http.redirections.entryPoint.to=websecure'
      - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge=true'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web'
      - '--certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:-admin@example.com}'
      - '--certificatesresolvers.letsencrypt.acme.storage=/etc/traefik/acme.json'
      - '--log.level=INFO'
      - '--accesslog=true'
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      # Traefik's own dashboard
      - "traefik.http.routers.traefik-dash.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.traefik-dash.entrypoints=websecure"
      - "traefik.http.routers.traefik-dash.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik-dash.service=api@internal"

Bring it up:

cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d traefik

Watch the logs the first time — Traefik will request a cert for the dashboard host as soon as DNS resolves.

docker logs -f projects-traefik-staging

Tip. While testing, switch ACME to staging endpoint (acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory) so you don't burn through Let's Encrypt's rate limits if you misconfigure DNS. Remove that flag before going live.

5. Run Jenkins in Docker

Add this Jenkins service to the same docker-compose.staging.yml. Every line matters (and the comments explain why).

  jenkins:
    image: jenkins/jenkins:lts
    container_name: projects-jenkins-staging
    restart: unless-stopped
    user: root                           # to use host docker.sock without UID juggling
    environment:
      - JAVA_OPTS=-Xmx1g -Xms512m -Duser.timezone=Asia/Dhaka
      - TZ=Asia/Dhaka                    # OS-level timezone inside container
      - JENKINS_OPTS=--prefix=/
    ports:
      - "3095:8080"                      # web UI (also reachable directly if needed)
      - "50000:50000"                    # inbound agent port
    volumes:
      - jenkins-data:/var/jenkins_home   # Jenkins config/jobs/secrets persistence
      - /var/run/docker.sock:/var/run/docker.sock                          # control host Docker
      - /usr/bin/docker:/usr/bin/docker                                     # docker CLI from host
      - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro  # docker compose plugin
      - /home/developer/projects:/projects                                # project tree
      - /etc/localtime:/etc/localtime:ro                                    # match host clock
      - /etc/timezone:/etc/timezone:ro
    networks:
      - proxy
      - internal
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:8080/login']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s
    deploy:
      resources:
        limits:
          memory: 1024M

Why user: root? It's the simplest way to share docker.sock and the project bind-mount without UID/GID gymnastics. If you prefer an unprivileged user, you'll need to set group: docker and align UIDs/perms on host folders — possible but out of scope here.

6. Expose Jenkins on a Domain via Traefik

This is the section many guides skip. We'll add labels to the Jenkins service so Traefik picks it up automatically. No editing of Traefik config required.

  jenkins:
    # ... everything above ...
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # 1) Router — match incoming Host
      - "traefik.http.routers.jenkins.rule=Host(`jenkins.example.com`)"
      - "traefik.http.routers.jenkins.entrypoints=websecure"
      - "traefik.http.routers.jenkins.tls.certresolver=letsencrypt"
      - "traefik.http.routers.jenkins.service=jenkins"

      # 2) Service — tell Traefik which container port is the app
      - "traefik.http.services.jenkins.loadbalancer.server.port=8080"

      # 3) Middleware — Jenkins needs X-Forwarded-Proto so it knows it's behind HTTPS
      - "traefik.http.middlewares.jenkins-headers.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.jenkins.middlewares=jenkins-headers"

What each line does:

Bring Jenkins up:

cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d jenkins

# Watch Traefik issue the certificate
docker logs -f projects-traefik-staging | grep -i acme

After 10–60 seconds you should be able to open https://jenkins.example.com and see Jenkins's setup wizard with a valid lock icon.

Inside Jenkins (after first login):

Manage Jenkins → System → Jenkins URL → set this to: https://jenkins.example.com/

This is important because Jenkins uses this base URL to generate:

• Webhook endpoints (for GitHub triggers)

• Links inside emails and build logs

If this isn't set correctly, GitHub webhooks may fail, and any links Jenkins generates will point to the wrong address (often localhost or internal IPs).

7. First-Time Jenkins Setup

If you're running Jenkins for the first time on this server, follow this section to complete the initial setup.

If you already have Jenkins configured, you can skip this section — but make sure the required plugins and settings match what we use later in this guide.

1. Open https://jenkins.example.com. Get the initial admin password:

   docker exec projects-jenkins-staging cat /var/jenkins_home/secrets/initialAdminPassword
   

2. Paste it, choose Install suggested plugins.

3. Create your admin user.

4. Manage Jenkins → Plugins → Available and install:

   • GitHub (and GitHub Branch Source)

   • Pipeline: GitHub

   • Credentials Binding (usually preinstalled)

That's all the plugins you need for the rest of this guide.

8. Add the GitHub Credential

Jenkins needs permission to access your GitHub repository.

This is done using a GitHub Personal Access Token (PAT), which acts like a password for secure API and Git operations.

We’ll store this token inside Jenkins as a credential so it can pull code during pipeline execution and authenticate securely without exposing secrets in code.

This single credential is used both for the SCM checkout and for the deploy-time git pull.

1. Create a Personal Access Token (classic) on GitHub with repo scope.

2. In Jenkins: Manage Jenkins → Credentials → System → Global → Add Credentials.

3. Fill in:

   • Kind: Username with password

   • Username: your GitHub username

   • Password: the token

   • ID: github_classic_token (the Jenkinsfile references this exact ID)

9. Create the Pipeline Job

Now that Jenkins has access to your repository, the next step is to define how deployments should run.

A pipeline job tells Jenkins:

• where your code lives,

• which branch to monitor,

• and how to execute your deployment process.

In Jenkins, create a new Pipeline job and connect it to your GitHub repository. Once this is set up, Jenkins will automatically trigger deployments whenever you push to the staging branch.

Start by creating a new job:

New Item → Pipeline → name it projects-staging → OK

Then configure the job:

• Under Build Triggers, enable:
  GitHub hook trigger for GITScm polling

• Under Pipeline:

  • Definition: Pipeline script from SCM

  • SCM: Git

  • Repository URL: https://github.com/<org>/projects-backend.git

  • Credentials: github_classic_token

  • Branch: */staging

  • Script Path: Jenkinsfile

Save the configuration.

At this point, Jenkins is fully connected to your repository and ready to run your deployment pipeline automatically.

10. The Jenkinsfile (Deploy Only What Changed)

Place this at the root of the app repo (projects-backend/Jenkinsfile), branch staging.

pipeline {
  agent any

  environment {
    PROJECT_PATH = "/projects/projects-prod-configs/projects-backend"
    COMPOSE_FILE = "docker-compose.staging.yml"
  }

  stages {

    stage('Checkout') {
      steps {
        checkout scm
        echo "Checkout completed for branch: ${env.BRANCH_NAME ?: 'staging'}"
      }
    }

    stage('Detect Changes') {
      steps {
        script {
          def changedFiles = sh(
            script: "git diff --name-only HEAD~1 HEAD",
            returnStdout: true
          ).trim()

          echo "Changed files:\n${changedFiles}"

          def services = [] as Set
          changedFiles.split('\n').each { file ->
            def svc  = file =~ /^apps\/services\/([a-z0-9-]+)\//
            def gw   = file =~ /^apps\/gateways\/([a-z0-9-]+)\//
            def core = file =~ /^apps\/core\/([a-z0-9-]+)\//
            if (svc)  { services << svc[0][1]  }
            if (gw)   { services << gw[0][1]   }
            if (core) { services << core[0][1] }
          }
          services = services.findAll { !it.endsWith('-e2e') }
          env.CHANGED_SERVICES = services.join(' ')

          echo "Services to deploy: ${env.CHANGED_SERVICES ?: '(none)'}"
        }
      }
    }

    stage('Deploy') {
      when { expression { return env.CHANGED_SERVICES?.trim() } }
      steps {
        withCredentials([usernamePassword(
          credentialsId: 'github_classic_token',
          usernameVariable: 'GIT_USER',
          passwordVariable: 'GIT_TOKEN'
        )]) {
          sh '''
            set -eu
            git config --global --add safe.directory "${PROJECT_PATH}"
            cd "${PROJECT_PATH}"
            git remote set-url origin "https://github.com/<org>/projects-backend.git"
            git -c credential.helper= \
                -c "credential.helper=!f() { echo username=\({GIT_USER}; echo password=\){GIT_TOKEN}; }; f" \
                pull origin staging
            docker compose -f "\({COMPOSE_FILE}" up -d --build \){CHANGED_SERVICES}
          '''
        }
        echo "Deployed: ${env.CHANGED_SERVICES}"
      }
    }

    stage('Skip Deployment') {
      when { expression { return !env.CHANGED_SERVICES?.trim() } }
      steps { echo "No service changes detected — nothing to deploy." }
    }
  }
}

Why each tricky line is there:

• git config --global --add safe.directory ... — git refuses to operate on a repo whose owner UID differs from the current user's. The repo on disk is owned by developer, but Git inside the container runs as root. This whitelists the path.

• git remote set-url origin "https://..." — flips the on-disk remote to HTTPS so the token can be used. (A PAT can't authenticate git@github.com: URLs — those use SSH.) Idempotent — safe to re-run.

• git -c credential.helper="!f() { echo username=...; echo password=...; }; f" — feeds the username/token to git for that one command without writing the token to disk and without exposing it on the process command line.

• ${CHANGED_SERVICES} is unquoted on purpose so multiple service names expand as separate args.

11. End-to-End Test

Before considering the setup complete, we need to verify that the entire pipeline works as expected.

This end-to-end test ensures that:

• GitHub webhooks are triggering Jenkins correctly,

• Jenkins can detect which services changed,

• and only the affected services are rebuilt and deployed.

In other words, this simulates a real production deployment.

Start by making a small change in your repository. For example, modify a file inside:

apps/gateways/student-apigw/

Then push the change to the staging branch.

Once pushed, Jenkins should automatically trigger via the webhook. If not, you can manually click Build Now.

Now open the build’s Console Output and verify the flow. You should see something like:

• Checkout completed for branch: staging

• Services to deploy: student-apigw

• git pull origin staging (successful)

• docker compose ... up -d --build student-apigw

• Deployed: student-apigw

If you see this sequence, your pipeline is working correctly.

If anything fails, don’t worry — jump to Section 12 where every common issue and its fix is documented.

12. Troubleshooting — Every Error We Hit

This section covers real issues we faced while setting up this pipeline — and more importantly, why each fix works. Understanding the “why” will help you debug similar problems in your own setup.

cd: can't cd to /projects/projects-prod-configs/projects-backend

Cause:
The Jenkinsfile runs cd $PROJECT_PATH, but inside the container that path doesn’t exist. This usually happens when:

• the project wasn’t cloned on the host, or

• the bind mount isn’t configured correctly.

Fix:

ls /home/developer/projects/projects-prod-configs/projects-backend
# If missing: git clone -b staging <url> there.

Confirm the bind mount:

docker inspect projects-jenkins-staging --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'

If missing, recreate the container:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:

Jenkins runs inside a container, but your code lives on the host. The bind mount connects them. Without it, Jenkins cannot access your project directory.

fatal: detected dubious ownership in repository

Cause:
Git blocks access when the repository owner differs from the current user.

• Repo owner: developer (host)

• Git runs as: root (inside container)

Fix:

git config --global --add safe.directory "${PROJECT_PATH}"

Why this works:

This explicitly tells Git that the directory is trusted, bypassing ownership mismatch security restrictions.

Host key verification failed / Could not read from remote repository

Cause:

The repository uses SSH (git@github.com:...), but:

• the container has no SSH keys

• no known_hosts file exists

Also, GitHub tokens cannot authenticate over SSH.

Fix (recommended):

git remote set-url origin "https://github.com/<org>/projects-backend.git"

Why this works:

HTTPS uses token-based authentication (PAT), which works inside containers without SSH configuration.

unknown shorthand flag: 'f' in -f ( docker compose)

Cause:
The Docker CLI exists, but the Docker Compose plugin is missing inside the container.

Fix:

volumes:
  - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro

Find your path if needed:

find /usr -name docker-compose -type f 2>/dev/null

Verify:

docker exec projects-jenkins-staging docker compose version

Why this works:

Docker Compose v2 is a CLI plugin. Mounting this directory makes the docker compose command available inside the container.

Wrong timezone in build timestamps and Jenkins UI

Fix: Set both env var and JVM flag, and bind-mount the host's clock files:

environment:
  - TZ=Asia/Dhaka
  - JAVA_OPTS=... -Duser.timezone=Asia/Dhaka
volumes:
  - /etc/localtime:/etc/localtime:ro
  - /etc/timezone:/etc/timezone:ro

You must recreate the container for env-var changes to take effect:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:
Jenkins runs on Java, which uses its own timezone separate from the OS.
By aligning OS timezone, JVM timezone, and host clock, you ensure consistent timestamps everywhere.

ERR_SOCKET_TIMEOUT (pnpm install fails)

Cause:

If you have multiple services building in parallel and each runs pnpm install with ~1500 packages, the network gets saturated and a timeout occurs.

Fixes:

a) Increase timeout + control concurrency

RUN pnpm install --frozen-lockfile --ignore-scripts 
--network-timeout 600000 
--network-concurrency 8

Why: Gives pnpm more time and reduces network overload.

b) Enable pnpm cache (BuildKit)

RUN --mount=type=cache,id=pnpm-store,target=/root/.local/share/pnpm/store 
pnpm install --frozen-lockfile --ignore-scripts

Why: Dependencies are cached and reused instead of downloading every time.

c) Avoid unnecessary rebuilds

docker compose -f \(COMPOSE_FILE build \)CHANGED_SERVICES docker compose -f \(COMPOSE_FILE up -d --no-build \)CHANGED_SERVICES

Why: Only changed services are rebuilt → less network load → fewer failures.

Container changes don’t apply after editing docker-compose.yml

Cause:

Docker compose up -d does not update running containers.

Fix:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:

This forces Docker to recreate the container with updated configuration (env, volumes, labels).

Traefik shows default certificate (no HTTPS)

Common causes:

DNS not pointing to server Port 80 blocked Wrong Docker network

Check:

dig +short jenkins.example.com docker logs projects-traefik-staging 2>&1 | grep -i acme

Why this works:

Let’s Encrypt uses HTTP-01 challenge, so it must reach your server via port 80. If DNS or networking is wrong, certificate issuance fails.

Jenkins: "Reverse proxy setup is broken"

Fix:

Set the Jenkins URL to https://jenkins.example.com/
Ensure header:

X-Forwarded-Proto: https

Why this works:

Jenkins needs to know it's behind HTTPS. Without this, it generates incorrect URLs (http instead of https), breaking redirects and webhooks.

13. Mental Model: Host vs. Container

Many setup mistakes come from confusing the host filesystem with the container filesystem. This table makes it explicit:

When debugging, always ask: "Inside which filesystem is this command running, and does the file/folder it's looking for exist there?"

14. Daily Operations Cheat Sheet

# Recreate Jenkins after changing compose
cd /home/developer/Projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

# Tail Jenkins logs
docker logs -f projects-jenkins-staging

# Open a shell inside the Jenkins container
docker exec -it projects-jenkins-staging bash

# From inside the container — sanity checks
docker compose version
ls /projects/projects-prod-configs/projects-backend
git -C /projects/projects-prod-configs/projects-backend remote -v

# Manually trigger the same deploy the pipeline does
cd /projects/projects-configs/projects-backend
git pull origin staging
docker compose -f docker-compose.staging.yml up -d --build student-apigw

# Inspect Traefik routing decisions
docker logs projects-traefik-staging 2>&1 | grep -i jenkins

# Check renewed certs
docker exec projects-traefik-staging cat /etc/traefik/acme.json | head -50

15. What I'd Do Differently Next Time

• Pre-build a base image with all node_modules baked in. With ~1500 packages × 15 services, every clean build re-downloads ~22k tarballs. A shared base cuts that 90%.

• Run a private npm proxy (Verdaccio / Nexus / GitHub Packages) on the same Docker network — eliminates flaky npmjs.org timeouts entirely.

• Per-service Jenkinsfile if your services drift apart in tooling. With one Jenkinsfile, every team contends for the same pipeline definition.

• Replace git diff HEAD~1 HEAD with git diff $(git merge-base HEAD origin/staging~1) HEAD so squash-merges and force-pushes don't accidentally skip services.

• Move secrets to a vault (HashiCorp Vault / AWS Secrets Manager / Doppler). PATs in Jenkins work, but rotation across many jobs is painful.

• Use Jenkins' Configuration-as-Code (JCasC) so the entire Jenkins setup (jobs, credentials definitions, plugins) is in git. Then a server rebuild is a one-command operation.

Closing Thoughts

The pipeline itself is just three stages: Checkout → Detect Changes → Deploy — but a real production setup is mostly about plumbing: reverse proxy, certificates, bind-mounts, credentials, timezones, build caches. None of these are exotic. Together they decide whether your Friday-afternoon deploy goes silently green or eats your weekend.

Follow sections 1–11 to get a working pipeline. Bookmark section 12 to keep it working.

Happy shipping.

In this article, you'll build a reusable GPU-optimized machine image using Packer, pre-loaded with NVIDIA drivers, CUDA Toolkit, NVIDIA Container Toolkit, DCGM, and system-level GPU tuning like persistence mode.

Table of Contents

• Prerequisites

• Project Setup

• Step 1: Install Packer

• Step 2: Set Up Project Directory

• Step 3: Install Packer's Plugins

• Step 4: Define Your Source

• Step 5: Writing the Build Template

• Step 6: Writing the GPU Provisioning Script

  • section 1: Pre-Installation (Kernel Headers)

  • Section 2: Installing NVIDIA's Apt Repository

  • Section 3: Pinning NVIDIA Drivers Version

  • Section 4: Installing the Driver

  • Section 5: CUDA Toolkit Installation

  • Section 6: Nvidia Container Toolkit

  • Section 7: Installing DCGM — Data Center GPU Manager

  • Section 8: Enabling Persistence Mode

  • Section 9: System Tuning for GPU Compute Workloads

• Step 7:Assembling and Running the Build

• Step 8: Test the Image and Verify the GPU Stack

• Conclusion

• References

Prerequisites

• HashiCorp Packer >= 1.9

• Google Compute Packer plugin (installed via packer init)

• Optionally, the AWS Packer plugin can be used for EC2 builds by adding an amazon-ebs source to node.pkr.hcl

• GCP project with Compute Engine API enabled (or AWS account with EC2 access)

• GCP authentication (gcloud auth application-default login) or AWS credentials

• Access to an NVIDIA GPU instance type (For example, A100, H100, L4 on GCP; p4d, p5, G6 on AWS)

Project Setup

Step 1: Install Packer

To get started, you'll install Packer with the steps below if you're on macOS (or you can follow the official documentation for Linux and Windows installation guides).

First, you'll install the official Packer formula from the terminal.

Install the HashiCorp tap, a repository of all Hashicorp packages.

$ brew tap hashicorp/tap

Now, install Packer with hashicorp/tap/packer.

$ brew install hashicorp/tap/packer

Step 2: Set Up Project Directory

With Packer installed, you'll create your project directory. For clean code and separation of concerns, your project directory should look like the below. Go ahead and create these files in your packer_demo folder using the command below:

mkdir -p packer_demo/script && touch packer_demo/{build.pkr.hcl,source.pkr.hcl,variable.pkr.hcl,local.pkr.hcl,plugins.pkr.hcl,values.pkrvars.hcl} packer_demo/script/base.sh

Your file directory should look like this:

packer_demo
├── build.pkr.hcl                 # Build pipeline — provisioner ordering
├── source.pkr.hcl                # GCP source definition (googlecompute)
├── variable.pkr.hcl              # Variable definitions with defaults
├── local.pkr.hcl                 # Local values
├── plugins.pkr.hcl                # Packer plugin requirements
├── values.pkrvars.hcl             # variable values (copy and customize)
├── script/
│   ├── base.sh                  # requirement script 

Step 3: Install Packer's Plugins

In your plugins.pkr.hcl file,, define your plugins in the packer block. The packer {} block contains Packer settings, including specifying a required plugin version. You'll find the required_plugins block in the Packer block, which specifies all the plugins required by the template to build your image. If you're on Azure or AWS, you can check for the latest plugin here.

packer {
  required_plugins {
    googlecompute = {
      source  = "github.com/hashicorp/googlecompute"
      version = "~> 1"
    }
  }
}

Then, initialize your Packer plugin with the command below:

packer init .

Step 4: Define Your Source

With your plugin initialized, you can now define your source block. The source block configures a specific builder plugin, which is then invoked by a build block. Source blocks contain your project ID, the zone where your machine will be created, the source_image_family (think of this as your base image, such as Debian, Ubuntu, and so on), and your source_image_project_id.

In GCP, each has an image project ID, such as "ubuntu-os-cloud" for Ubuntu. You'll set the machine type to a GPU machine type because you're building your base image for a GPU machine, so the machine on which it will be created needs to be able to run your commands.

source "googlecompute" "gpu-node" {
  project_id              = var.project_id
  zone                    = var.zone
  source_image_family     = var.image_family
  source_image_project_id = var.image_project_id
  ssh_username            = var.ssh_username
  machine_type            = var.machine_type



  image_name        = var.image_name
  image_description = var.image_description

  disk_size           = var.disk_size
  on_host_maintenance = "TERMINATE"

  tags = ["gpu-node"]

}

Setting on_host_maintenance = "TERMINATE" on Google Cloud Compute Engine ensures that a VM instance stops instead of live-migrating during infrastructure maintenance. This is important when using GPUs or specialized hardware that can't migrate, preventing data corruption.

You'll define all your variables in the variable.pkr.hcl file, and set the values in the values.pkrvars.hcl. Remember to always add your values.pkrvars.hcl file to Gitignore.

variable "image_name" {
  type        = string
  description = "The name of the resulting image"
}

variable "image_description" {
  type        = string
  description = "Description of the image"
}

variable "project_id" {
  type        = string
  description = "The GCP project ID where the image will be created"
}

variable "image_family" {
  type        = string
  description = "The image family to which the resulting image belongs"
}

variable "image_project_id" {
  type        = list(string)
  description = "The project ID(s) to search for the source image"
}

variable "zone" {
  type        = string
  description = "The GCP zone where the build instance will be created"
}

variable "ssh_username" {
  type        = string
  description = "The SSH username to use for connecting to the instance"
}
variable "machine_type" {
  type        = string
  description = "The machine type to use for the build instance"
}

variable "cuda_version" {
  type        = string
  description = "CUDA toolkit version"
  default     = "13.1"
}

variable "driver_version" {
  type        = string
  description = "NVIDIA driver version"
  default     = "590.48.01"
}

variable "disk_size" {
  type        = number
  description = "Boot disk size in GB"
  default     = 50
}

values.pkrvars.hcl

image_name        = "base-gpu-image-{{timestamp}}"
image_description = "Ubuntu 24.04 LTS with gpu drivers and health checks"
project_id        = "your gcp project id"
image_family      = "ubuntu-2404-lts-amd64"
image_project_id  = ["ubuntu-os-cloud"]
zone              = "us-central1-a"
ssh_username      = "packer"
machine_type      = "g2-standard-4"
disk_size        = 50
driver_version   = "590.48.01"
cuda_version      = "13.1" 

Step 5: Writing the Build Template

Create build.pkr.hcl. The build block creates a temporary instance, runs provisioners, and produces an image.

Provisioners in this template are organized as follows:

• First provisioner runs system updates and upgrades.

• Second provisioner reboots the instance (expect_disconnect = true).

• Third provisioner waits for the instance to come back (pause_before), then runs script/base.sh. This provisioner sets max_retries to handle transient SSH timeouts and pass environment variables for DRIVER_VERSION and CUDA_VERSION.

Lastly, you have the post-processor to tell you the image ID and completion status:

build {
  sources = ["source.googlecompute.gpu-node"]

  provisioner "shell" {
    inline = [
      "set -e",
      "sudo apt update",
      "sudo apt -y dist-upgrade"
    ]
  }

  provisioner "shell" {
    expect_disconnect = true
    inline            = ["sudo reboot"]
  }

  # Base: NVIDIA drivers, CUDA, DCGM
  provisioner "shell" {
    pause_before = "60s"
    script       = "script/base.sh"
    max_retries  = 2
    environment_vars = [
      "DRIVER_VERSION=${var.driver_version}",
      "CUDA_VERSION=${var.cuda_version}"
    ]
  }

  post-processor "shell-local" {
    inline = [
      "echo '=== Image Build Complete ==='",
      "echo 'Image ID: ${build.ID}'",
      "date"
    ]
  }
}

Step 6: Writing the GPU Provisioning Script

Now we'll go through the base script, and break down some parts of it.

Section 1: Pre-Installation (Kernel Headers)

Before installing NVIDIA drivers, the system needs kernel headers and build tools. The NVIDIA driver compiles a kernel module during installation via DKMS, so if the headers for your running kernel aren't present, the build will fail silently, and the driver won't load on boot.

log "Installing kernel headers and build tools..."
sudo apt-get install -qq -y \
  "linux-headers-$(uname -r)" \
  build-essential \
  dkms \
  curl \
  wget

Section 2: Installing NVIDIA's Apt Repository

This snippet downloads and installs NVIDIA’s official keyring package based on your OS Linux distribution, which adds the trusted signing keys needed for the system to verify CUDA packages.

log "Adding NVIDIA CUDA apt repository (${DISTRO})..."
wget -q "https://developer.download.nvidia.com/compute/cuda/repos/\({DISTRO}/\){ARCH}/cuda-keyring_1.1-1_all.deb" \
  -O /tmp/cuda-keyring.deb
sudo dpkg -i /tmp/cuda-keyring.deb
rm /tmp/cuda-keyring.deb
sudo apt-get update -qq

Section 3: Pinning NVIDIA Drivers Version

Pinning the NVIDIA driver to a specific version ensures that the system always installs and keeps using exactly that driver version, even when newer drivers appear in the repository.

NVIDIA drivers are tightly coupled with CUDA toolkit versions, Kernel versions, and container runtimes like Docker or NVIDIA Container Toolkit

A mismatch, such as the system auto‑upgrading to a newer driver, can cause CUDA to stop working, break GPU acceleration, or make the machine image inconsistent across deployments.

log "Pinning driver to version ${DRIVER_VERSION}..."
sudo apt-get install -qq -y "nvidia-driver-pinning-${DRIVER_VERSION}"

Section 4: Installing the Driver

The libnvidia-compute installs only the compute‑related user‑space libraries (CUDA driver components), while the nvidia-dkms-open; installs the open‑source NVIDIA kernel module, built locally via DKMS.

Together, these two packages give you a fully functional CUDA driver environment without any GUI or graphics dependencies.

Here, we're using NVIDIA’s compute‑only driver stack using the open‑source kernel modules, as it deliberately avoids installing any display-related components, which you don't need.

This method provides an installation module based on DKMS that's better aligned with Linux distros, as it's lightweight, and compute-focused.

log "Installing NVIDIA compute-only driver (open kernel modules)..."
sudo apt-get -V install -y \
  libnvidia-compute \
  nvidia-dkms-open

Section 5: CUDA Toolkit Installation

This part of the script installs the CUDA Toolkit for the specified version and then makes sure that CUDA’s executables and libraries are available system‑wide for every user and every shell session.

It adds CUDA binaries to PATH, so commands like nvcc, cuda-gdb, and cuda-memcheck work without specifying full paths. It also adds CUDA libraries to LD_LIBRARY_PATH, so applications can find CUDA’s shared libraries at runtime.

log "Installing CUDA Toolkit ${CUDA_VERSION}..."
sudo apt-get install -qq -y "cuda-toolkit-${CUDA_VERSION}"

# Persist CUDA paths for all users and sessions
cat <<'EOF' | sudo tee /etc/profile.d/cuda.sh
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}
EOF
echo "/usr/local/cuda/lib64" | sudo tee /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

Section 6: NVIDIA Container Toolkit

This block installs the NVIDIA Container Toolkit and configures it so that containers (Docker or containerd) can access the GPU safely and correctly. It’s a critical step for Kubernetes GPU nodes, Docker GPU workloads, and any system that needs GPU acceleration inside containers.

log "Installing NVIDIA Container Toolkit..."
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update -qq
sudo apt-get install -qq -y nvidia-container-toolkit

# Configure for containerd (primary Kubernetes runtime)
sudo nvidia-ctk runtime configure --runtime=containerd

# Configure for Docker if present on this image
if systemctl list-unit-files | grep -q "^docker.service"; then
  sudo nvidia-ctk runtime configure --runtime=docker
fi

Section 7: Installing DCGM (Data Center GPU Manager)

This section covers the installation and validation of NVIDIA DCGM (Data Center GPU Manager), which is NVIDIA’s official management and telemetry framework for data center GPUs.

It offers health monitoring and diagnostics, telemetry (including temperature, clocks, power, and utilization), error reporting, and integration with Kubernetes, Prometheus, and monitoring agents. Your GPU monitoring stack relies on this.

The script extracts the installed version and checks that it meets the minimum required version for NVIDIA driver 590+. Then it enforces the version requirement. This prevents a mismatch between the GPU driver and DCGM, which would break monitoring and health checks. It also enables fabric manager for NVLink/NVswitches, if you're on a Multi‑GPU topologies like A100/H100 DGX or multi‑GPU servers.

log "Installing DCGM..."
sudo apt-get install -qq -y datacenter-gpu-manager

DCGM_VER=\((dpkg -s datacenter-gpu-manager 2>/dev/null | awk '/^Version:/{print \)2}' | sed 's/^[0-9]*://')
DCGM_MAJOR=\((echo "\){DCGM_VER}" | cut -d. -f1)
DCGM_MINOR=\((echo "\){DCGM_VER}" | cut -d. -f2)
if [[ "\({DCGM_MAJOR}" -lt 4 ]] || { [[ "\){DCGM_MAJOR}" -eq 4 ]] && [[ "${DCGM_MINOR}" -lt 3 ]]; }; then
  error "DCGM ${DCGM_VER} is below the 4.3 minimum required for driver 590+. Check your CUDA repo."
fi
log "DCGM installed: ${DCGM_VER}"

sudo systemctl enable nvidia-dcgm
sudo systemctl start  nvidia-dcgm

# Fabric Manager — only needed for NVLink/NVSwitch GPUs (A100/H100 multi-GPU nodes)
if systemctl list-unit-files | grep -q "^nvidia-fabricmanager.service"; then
  log "Enabling nvidia-fabricmanager for NVLink GPUs..."
  sudo systemctl enable nvidia-fabricmanager
  sudo systemctl start  nvidia-fabricmanager
fi

Section 8: Enabling Persistence Mode

The NVIDIA driver normally unloads itself when the GPU is idle. When a new workload starts, the driver must reload, reinitialize the GPU, and set up memory mappings. This adds a delay of a few hundred milliseconds to several seconds, depending on the GPU and system.

Enabling nvidia‑persistenced keeps the NVIDIA driver loaded in memory even when no GPU workloads are running.

log "Enabling nvidia-persistenced..."
sudo systemctl enable nvidia-persistenced
sudo systemctl start  nvidia-persistenced

Section 9: System Tuning for GPU Compute Workloads

This block applies a set of system‑level performance and stability tunings that are standard for high‑performance GPU servers, Kubernetes GPU nodes, and ML/AI workloads.

Each line targets a specific bottleneck or instability pattern that appears in real GPU production environments.

• Swap and memory behavior: Disabling swap and setting vm.swappiness=0 prevents the kernel from pushing GPU‑bound processes into swap. GPU workloads are extremely sensitive to latency, and swapping can cause CUDA context resets and GPU driver timeouts.

• Hugepages for large memory allocations: Setting vm.nr_hugepages=2048 allocates a pool of hugepages, which reduces TLB pressure for large contiguous memory allocations.

  CUDA, NCCL, and deep‑learning frameworks frequently allocate large buffers, and hugepages reduce page‑table overhead, improving memory bandwidth and lowering latency for large tensor operations. This is especially useful on multi‑GPU servers.

• CPU frequency governor: Installing cpupower and forcing the CPU governor to performance ensures the CPU stays at maximum frequency instead of scaling down.

  GPU workloads often become CPU‑bound during Data preprocessing, Kernel launches, and NCCL communication. Keeping CPUs at full speed reduces jitter and improves throughput.

• NUMA and topology tools: Installing numactl, libnuma-dev, and hwloc provides tools for pinning processes to NUMA nodes, understanding CPU–GPU affinity, and optimizing multi‑GPU placement.

• Disabling irqbalance: Stopping and disabling irqbalance it lets the NVIDIA driver manage interrupt affinity. For GPU servers, irqbalance can incorrectly move GPU interrupts to suboptimal CPUs, causing higher latency and lower throughput.

log "Applying system tuning..."

# Disable swap (critical for Kubernetes scheduler and ML stability)
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab
echo "vm.swappiness=0"     | sudo tee /etc/sysctl.d/99-gpu-swappiness.conf

# Hugepages — reduces TLB pressure for large memory allocations
echo "vm.nr_hugepages=2048" | sudo tee /etc/sysctl.d/99-gpu-hugepages.conf

# CPU performance governor
sudo apt-get install -qq -y linux-tools-common "linux-tools-$(uname -r)" || true
sudo cpupower frequency-set -g performance || true

# NUMA and topology tools for GPU affinity tuning
sudo apt-get install -qq -y numactl libnuma-dev hwloc

# Disable irqbalance — let NVIDIA driver manage interrupt affinity
sudo systemctl disable irqbalance || true
sudo systemctl stop    irqbalance || true

# Apply all sysctl settings now
sudo sysctl --system

Full base.sh script here:

#!/bin/bash
set -euo pipefail

log()   { echo "[BASE] $1"; }
error() { echo "[BASE][ERROR] $1" >&2; exit 1; }

###############################################################
###############################################################
[[ -z "${DRIVER_VERSION:-}" ]] && error "DRIVER_VERSION is not set."
[[ -z "${CUDA_VERSION:-}"   ]] && error "CUDA_VERSION is not set."

log "DRIVER_VERSION : ${DRIVER_VERSION}"
log "CUDA_VERSION   : ${CUDA_VERSION}"

DISTRO=\((. /etc/os-release && echo "\){ID}${VERSION_ID}" | tr -d '.')
ARCH="x86_64"

export DEBIAN_FRONTEND=noninteractive

###############################################################
# 1. System update
###############################################################
log "Updating system packages..."
sudo apt-get update -qq
sudo apt-get upgrade -qq -y

###############################################################
# 2. Pre-installation — kernel headers
#    Source: https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/ubuntu.html
###############################################################
log "Installing kernel headers and build tools..."
sudo apt-get install -qq -y \
  "linux-headers-$(uname -r)" \
  build-essential \
  dkms \
  curl \
  wget

###############################################################
# 3. NVIDIA CUDA Network Repository
###############################################################
log "Adding NVIDIA CUDA apt repository (${DISTRO})..."
wget -q "https://developer.download.nvidia.com/compute/cuda/repos/\({DISTRO}/\){ARCH}/cuda-keyring_1.1-1_all.deb" \
  -O /tmp/cuda-keyring.deb
sudo dpkg -i /tmp/cuda-keyring.deb
rm /tmp/cuda-keyring.deb
sudo apt-get update -qq

###############################################################
# 4. Pin driver version BEFORE installation (590+ requirement)
###############################################################
log "Pinning driver to version ${DRIVER_VERSION}..."
sudo apt-get install -qq -y "nvidia-driver-pinning-${DRIVER_VERSION}"

###############################################################
# 5. Compute-only (headless) driver — Open Kernel Modules
#    Source: NVIDIA Driver Installation Guide — Compute-only System (Open Kernel Modules)
#
#    libnvidia-compute  = compute libraries only (no GL/Vulkan/display)
#    nvidia-dkms-open   = open-source kernel module built via DKMS
#
#    Open kernel modules are the NVIDIA-recommended choice for
#    Ampere, Hopper, and Blackwell data centre GPUs (A100, H100, etc.)
###############################################################
log "Installing NVIDIA compute-only driver (open kernel modules)..."
sudo apt-get -V install -y \
  libnvidia-compute \
  nvidia-dkms-open

###############################################################
# 6. CUDA Toolkit
###############################################################
log "Installing CUDA Toolkit ${CUDA_VERSION}..."
sudo apt-get install -qq -y "cuda-toolkit-${CUDA_VERSION}"

# Persist CUDA paths for all users and sessions
cat <<'EOF' | sudo tee /etc/profile.d/cuda.sh
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}
EOF
echo "/usr/local/cuda/lib64" | sudo tee /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

###############################################################
# 7. NVIDIA Container Toolkit
#    Required for GPU workloads in Docker / containerd / Kubernetes
###############################################################
log "Installing NVIDIA Container Toolkit..."
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update -qq
sudo apt-get install -qq -y nvidia-container-toolkit

# Configure for containerd (primary Kubernetes runtime)
sudo nvidia-ctk runtime configure --runtime=containerd

# Configure for Docker if present on this image
if systemctl list-unit-files | grep -q "^docker.service"; then
  sudo nvidia-ctk runtime configure --runtime=docker
fi

###############################################################
# 8. DCGM — DataCenter GPU Manager
###############################################################
log "Installing DCGM..."
sudo apt-get install -qq -y datacenter-gpu-manager
 
DCGM_VER=\((dpkg -s datacenter-gpu-manager 2>/dev/null | awk '/^Version:/{print \)2}' | sed 's/^[0-9]*://')
DCGM_MAJOR=\((echo "\){DCGM_VER}" | cut -d. -f1)
DCGM_MINOR=\((echo "\){DCGM_VER}" | cut -d. -f2)
if [[ "\({DCGM_MAJOR}" -lt 4 ]] || { [[ "\){DCGM_MAJOR}" -eq 4 ]] && [[ "${DCGM_MINOR}" -lt 3 ]]; }; then
  error "DCGM ${DCGM_VER} is below the 4.3 minimum required for driver 590+. Check your CUDA repo."
fi
log "DCGM installed: ${DCGM_VER}"

sudo systemctl enable nvidia-dcgm
sudo systemctl start  nvidia-dcgm

# Fabric Manager — only needed for NVLink/NVSwitch GPUs (A100/H100 multi-GPU nodes)
if systemctl list-unit-files | grep -q "^nvidia-fabricmanager.service"; then
  log "Enabling nvidia-fabricmanager for NVLink GPUs..."
  sudo systemctl enable nvidia-fabricmanager
  sudo systemctl start  nvidia-fabricmanager
fi

###############################################################
# 9. NVIDIA Persistence Daemon
#    Keeps the driver loaded between jobs — reduces cold-start
#    latency on the first CUDA call in each new workload
###############################################################
log "Enabling nvidia-persistenced..."
sudo systemctl enable nvidia-persistenced
sudo systemctl start  nvidia-persistenced

###############################################################
# 10. System tuning for GPU compute workloads
###############################################################
log "Applying system tuning..."

# Disable swap (critical for Kubernetes scheduler and ML stability)
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab
echo "vm.swappiness=0"     | sudo tee /etc/sysctl.d/99-gpu-swappiness.conf

# Hugepages — reduces TLB pressure for large memory allocations
echo "vm.nr_hugepages=2048" | sudo tee /etc/sysctl.d/99-gpu-hugepages.conf

# CPU performance governor
sudo apt-get install -qq -y linux-tools-common "linux-tools-$(uname -r)" || true
sudo cpupower frequency-set -g performance || true

# NUMA and topology tools for GPU affinity tuning
sudo apt-get install -qq -y numactl libnuma-dev hwloc

# Disable irqbalance — let NVIDIA driver manage interrupt affinity
sudo systemctl disable irqbalance || true
sudo systemctl stop    irqbalance || true

# Apply all sysctl settings now
sudo sysctl --system

###############################################################
# Done
###############################################################
log "============================================"
log "Base layer provisioning complete."
log "  OS      : ${DISTRO}"
log "  Driver  : ${DRIVER_VERSION} (open kernel modules, compute-only)"
log "  CUDA    : cuda-toolkit-${CUDA_VERSION}"
log "  DCGM    : ${DCGM_VER}"
log "============================================"

Step 7: Assembling and Running the Build

Validate the template first, then run the build. Validation catches syntax or variable errors early, so the build doesn’t start on a broken config.

packer validate -var-file=values.pkrvars.hcl .

If validation succeeds, you’ll see a short confirmation like The configuration is valid.. After that, start the build. You should expect the process to create a temporary VM, run your provisioners, and produce an image:

packer build -var-file=values.pkrvars.hcl .

The build typically takes 15–20 minutes, depending on network speed and package installs. Watch the Packer log for three key checkpoints:

• Instance creation — confirms the temporary VM was provisioned.

• Provisioner output — shows each script step (updates, reboot, script/base.sh) and any errors.

• Image creation — indicates the build finished and an image artifact was written.

If the build fails, copy the failing provisioner’s log lines and re-run the build after fixing the script or variables. For quick troubleshooting, re-run the failing provisioner locally on a matching test VM to iterate faster.

googlecompute.gpu-node: output will be in this color.

==> googlecompute.gpu-node: Checking image does not exist...
==> googlecompute.gpu-node: Creating temporary RSA SSH key for instance...
==> googlecompute.gpu-node: no persistent disk to create
==> googlecompute.gpu-node: Using image: ubuntu-2404-noble-amd64-v20260225
==> googlecompute.gpu-node: Creating instance...
==> googlecompute.gpu-node: Loading zone: us-central1-a
==> googlecompute.gpu-node: Loading machine type: g2-standard-4
==> googlecompute.gpu-node: Requesting instance creation...
==> googlecompute.gpu-node: Waiting for creation operation to complete...
==> googlecompute.gpu-node: Instance has been created!
==> googlecompute.gpu-node: Waiting for the instance to become running...
==> googlecompute.gpu-node: IP: 34.58.58.214
==> googlecompute.gpu-node: Using SSH communicator to connect: 34.58.58.214
==> googlecompute.gpu-node: Waiting for SSH to become available...
systemd-logind.service
==> googlecompute.gpu-node:  systemctl restart unattended-upgrades.service
==> googlecompute.gpu-node:
==> googlecompute.gpu-node: No containers need to be restarted.
==> googlecompute.gpu-node:
==> googlecompute.gpu-node: User sessions running outdated binaries:
==> googlecompute.gpu-node:  packer @ session #1: sshd[1535]
==> googlecompute.gpu-node:  packer @ user manager service: systemd[1540]
==> googlecompute.gpu-node: Pausing 1m0s before the next provisioner...
==> googlecompute.gpu-node: Provisioning with shell script: script/base.sh
==> googlecompute.gpu-node: [BASE] DRIVER_VERSION : 590.48.01
==> googlecompute.gpu-node: [BASE] CUDA_VERSION   : 13.1
==> googlecompute.gpu-node: [BASE] Updating system packages...
==> googlecompute.gpu-node: [BASE] Installing kernel headers and build tools...
==> googlecompute.gpu-node: [BASE] Installing CUDA Toolkit 13.1...
==> googlecompute.gpu-node: [BASE] Installing DCGM...
==> googlecompute.gpu-node: [BASE] Enabling nvidia-persistenced...
==> googlecompute.gpu-node: [BASE] Applying system tuning...
==> googlecompute.gpu-node: vm.swappiness=0
==> googlecompute.gpu-node: vm.nr_hugepages=2048
==> googlecompute.gpu-node: Setting cpu: 0
==> googlecompute.gpu-node: Error setting new values. Common errors:
==> googlecompute.gpu-node: [BASE] ============================================
==> googlecompute.gpu-node: [BASE] Base layer provisioning complete.
==> googlecompute.gpu-node: [BASE]   OS      : ubuntu2404
==> googlecompute.gpu-node: [BASE]   Driver  : 590.48.01 (open kernel modules, compute-only)
==> googlecompute.gpu-node: [BASE]   CUDA    : cuda-toolkit-13.1
==> googlecompute.gpu-node: [BASE]   DCGM    : 1:3.3.9
==> googlecompute.gpu-node: [BASE] ============================================
==> googlecompute.gpu-node: Deleting instance...
==> googlecompute.gpu-node: Instance has been deleted!
==> googlecompute.gpu-node: Creating image...
==> googlecompute.gpu-node: Deleting disk...
==> googlecompute.gpu-node: Disk has been deleted!
==> googlecompute.gpu-node: Running post-processor:  (type shell-local)
==> googlecompute.gpu-node (shell-local): Running local shell script: 
==> googlecompute.gpu-node (shell-local): === Image Build Complete ===
==> googlecompute.gpu-node (shell-local): Image ID: packer-69b6c2ee-883a-3602-7bb5-059f1ba27c8b
==> googlecompute.gpu-node (shell-local): Sun Mar 15 15:50:09 WAT 2026
Build 'googlecompute.gpu-node' finished after 17 minutes 55 seconds.

==> Wait completed after 17 minutes 55 seconds

==> Builds finished. The artifacts of successful builds are:
--> googlecompute.gpu-node: A disk image was created in the 'my_project-00000' project: base-gpu-image-1773585134

Step 8: Test the Image and Verify the GPU Stack

Confirm the image exists in the GCP Console: Compute → Storage → Images and locate your newly created OS image.

Create a test VM from the image:

gcloud compute instances create my-gpu-vm \
  --machine-type=g2-standard-4 \
  --accelerator=count=1,type=nvidia-l4 \
  --image=base-gpu-image-1772718104 \
  --image-project=YOUR_PROJECT_ID \
  --boot-disk-size=50GB \
  --maintenance-policy=TERMINATE \
  --restart-on-failure \
  --zone=us-central1-a

Created [https://www.googleapis.com/compute/v1/projects/my-project-000/zones/us-central1-a/instances/my-gpu-vm].
NAME       ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP    EXTERNAL_IP      STATUS
my-gpu-vm  us-central1-a  g2-standard-4               10.128.15.227  104.154.184.217  RUNNING

Once the instance is RUNNING, verify the NVIDIA driver and GPU are visible:

The nvidia-smi output confirms:

• Driver 590.48.01 loaded

• CUDA 13.1 available

• Persistence Mode is On

• The L4 GPU is detected with 23GB VRAM

• Zero ECC errors

• No running processes (clean idle state).

This is exactly what a healthy base image should look like. Notice Disp.A: Off? That confirms our compute-only driver choice is working — no display adapter is active.

Confirm the installed CUDA toolkit by running. nvcc --version. You can see that version 13.1 was installed as specified.

Let's confirm DCGM installation by running dcgmi discovery -l. Successful output indicates DCGM is running and communicating with the driver.

Conclusion

You now have a production‑grade, GPU‑optimized base image that includes the NVIDIA compute‑only driver built with open kernel modules, DCGM for monitoring, and the CUDA Toolkit. You also applied OS‑level tuning tailored to GPU compute workloads, providing a consistent, reproducible environment with no manual setup.

From here, you can extend the build by adding an application‑layer script to install frameworks such as PyTorch, TensorFlow, or vLLM, or create an instance template that uses this image to scale your GPU infrastructure.

The full Packer project includes additional scripts for training and inference workloads that you can use to extend your image.

References

• NVIDIA Driver Installation Guide (Ubuntu): https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/

• NVIDIA CUDA Toolkit Documentation: https://docs.nvidia.com/cuda/

• NVIDIA Container Toolkit Installation Guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

• NVIDIA DCGM Documentation: https://docs.nvidia.com/datacenter/dcgm/latest/index.html

• NVIDIA Persistence Daemon: https://docs.nvidia.com/deploy/driver-persistence/index.html

• HashiCorp Packer Documentation: https://developer.hashicorp.com/packer/docs

• Packer Google Compute Builder: https://developer.hashicorp.com/packer/integrations/hashicorp/googlecompute

This is a common experience when contributing to open source: you make a small change, open a pull request, and suddenly everything fails.

Not just one check, but multiple:

• Lint errors

• YAML validation issues

• Build failures

• Deployment failures

Even more confusing, you may see errors in parts of the codebase you didn’t modify.

In this article, you'll learn how to debug these issues step by step. The goal is not just to fix one pull request, but to understand how CI systems validate your changes.

This guide is based on a real debugging experience from contributing to an open source documentation project.

While this example comes from a documentation project, the debugging workflow applies to many repositories that use CI pipelines, linting tools, and automated builds.

Table of Contents:

• Understanding the CI Pipeline (What’s Actually Happening)

• How a CI Pipeline Processes Your Pull Request

• A Practical Debugging Workflow

  • Step 1: Fix Authentication and Permission Issues

  • Step 2: Run Lint Checks Locally

  • Step 3: Fix Common Markdown Lint Errors

  • Step 4: Fix YAML Inside Markdown Code Blocks

  • Step 5: Fix Build Errors After Lint Passes

  • Step 6: Debug Cascading CI Failures

  • Step 7: Handle Git Issues During CI Debugging

• Key Takeaways

• Conclusion

Prerequisites

To follow this guide, you should have:

• Basic familiarity with Git and pull requests

• A GitHub account

• Some exposure to CI/CD concepts (helpful but not required)

Understanding the CI Pipeline (What’s Actually Happening)

In many projects, you will see the term CI/CD, which stands for Continuous Integration and Continuous Deployment (or Delivery).

In this guide, we'll focus specifically on the CI part – that is, Continuous Integration. This refers to the automated checks that run when you push code or open a pull request. These checks validate your changes before they're merged into the main codebase.

CD (Continuous Deployment/Delivery), on the other hand, typically handles what happens after those checks pass, such as deploying the application.

Understanding this distinction is important because most of the issues we debug in this guide happen during the CI stage.

Most repositories run multiple automated checks when you open a pull request:

• Linting tools (for example, markdownlint, yamllint) enforce formatting rules

• Build systems (for example, mdBook) validate structure and generate output

• Deployment checks (for example, Netlify) ensure that the site can be built and served

• Merge controllers (for example, Tide) enforce approval policies

A key point to remember: CI systems validate the entire set of files in your commit, not just the lines you changed.

How a CI Pipeline Processes Your Pull Request

When you push code or open a pull request, the CI pipeline runs several checks in sequence.

Let’s visualize how these checks are connected in a typical CI pipeline.

Figure: A simplified CI pipeline showing how linting, build, and deployment checks are executed sequentially.

The above diagram shows a sequential CI pipeline with feedback loops, where failures at any stage return you to fix the issue before continuing.

Let’s break down what this diagram shows:

1. You start by pushing code or opening a pull request.

2. The CI pipeline begins running automated checks.

3. The first set of checks typically includes linting tools like markdownlint or yamllint.

   • If linting fails, the pipeline stops, and you must fix formatting issues before continuing.

   • If linting passes, the pipeline moves to the build step (for example, mdBook in documentation projects).

   • If the build fails, it usually means there is a structural issue, such as duplicate entries or invalid references.

4. After a successful build, deployment checks (such as Netlify previews) run.

   • If deployment fails, the issue is often related to configuration or build output.

5. If all steps pass, the pull request becomes ready for review.

A Practical Debugging Workflow

Step 1: Fix Authentication and Permission Issues

Before CI runs, your push can fail due to authentication errors.

Example error:

refusing to allow a Personal Access Token to create or update workflow

This happens because GitHub requires special permissions when your commit includes files under:

.github/workflows/

The solution is to regenerate your Personal Access Token (PAT) with:

• repo access

• workflow permission

Step 2: Run Lint Checks Locally

Relying only on CI feedback slows you down because you have to push changes and wait for the pipeline to run before seeing errors.

Running checks locally allows you to catch issues immediately before pushing your code.

In practice, you should do both:

• Run checks locally to catch errors early and reduce iteration time

• Use CI as the final validation to ensure your changes meet the repository’s standards

Think of local checks as your first line of defense, and CI as the final gate before your code is accepted.

Here's an example (Markdown linting):

npm install -g markdownlint-cli2
markdownlint-cli2 docs/**/*.md

Step 3: Fix Common Markdown Lint Errors

Here are some common issues you may encounter:

1. Non-descriptive links

Non-descriptive links like "here" don't give readers any context about where the link leads. This makes documentation harder to understand and less accessible, especially for users relying on screen readers.

Instead of writing:

[here](https://example.com)

Use descriptive text like:

[command help documentation](https://example.com)

2. Line length violations

Many projects enforce a maximum line length (often around 80 characters) to improve readability across different devices and editors.

If a line is too long, you can split it into multiple lines without changing the meaning.

To do this, break the line at natural points such as spaces between words or after punctuation. Avoid breaking words or disrupting the sentence structure.
For example:

This is a long sentence that should be split across multiple
lines to satisfy lint rules.

3. List indentation issues

List indentation errors occur when nested list items aren't aligned consistently. This can break formatting and cause linting errors.

To avoid this, just make sure you use consistent spacing (usually 2 spaces per level).

Example (incorrect):

- Item 1
 - Subitem

Correct version:

- Item 1
  - Subitem

Step 4: Fix YAML Inside Markdown Code Blocks

YAML has strict formatting rules, including proper indentation, key-value structure, and consistent spacing.

Even when YAML appears inside a markdown code block, tools like yamllint still validate its structure.

Example (incorrect):

metadata:
annotations:

Correct version:

metadata:
  annotations:
    capi.metal3.io/unhealthy: "true"

In the incorrect example, annotations is not properly nested under metadata, and no key-value pair is defined.

In the corrected version:

• annotations is properly indented under metadata

• a valid key-value pair is added (capi.metal3.io/unhealthy: "true")

This structure satisfies YAML’s requirement for proper hierarchy and formatting.

Step 5: Fix Build Errors After Lint Passes

Passing lint checks doesn't guarantee that your build will succeed.

This is because linting focuses on syntax and formatting, while the build process validates the structure and integrity of the entire project.

Build failures often occur due to issues such as:

• Duplicate entries in navigation files

• Missing or incorrectly referenced files

• Invalid configuration settings

Even if your syntax is correct, the build system ensures everything connects properly.

For example, in documentation projects using tools like mdBook, a duplicate entry in SUMMARY.md can cause the build to fail even when all files pass lint checks.

Step 6: Debug Cascading CI Failures

CI pipelines are layered. One failure can trigger multiple downstream failures.

For example, imagine a YAML indentation error:

YAML error → build fails → deploy fails → multiple checks fail

To fix this:

1. Identify the first failing step in the CI logs

2. Fix that issue

3. Re-run the pipeline

In this example, the YAML indentation error is the root cause. Once you fix the YAML formatting, the lint check passes, which allows the build to proceed and the deployment step to succeed.

This is why it is important to always fix the first failure in the pipeline rather than trying to address all errors at once.

Step 7: Handle Git Issues During CI Debugging

When working with updated branches, you may encounter:

• Diverged branches

• Rebase conflicts

• Push rejections

To resolve these issues, you typically need to update your branch using one of two approaches:

Option 1: Rebase (clean history)

git pull --rebase

Rebasing rewrites your commit history so your changes appear on top of the latest version of the branch.

Use carefully:

• Only rebase your own branches

• Avoid rebasing shared branches

Option 2: Merge (safer)

git pull --no-rebase

Merging preserves the full commit history and is safer when working with others, but it may introduce additional merge commits.

Pushing your changes safely

After updating your branch, you may need to push changes:

git push --force-with-lease

Avoid using:

git push --force

The --force option can overwrite the other contributors’ work. The --force-with-lease option is safer because it only pushes if the remote branch has not changed unexpectedly.

Key Takeaways

• CI validates your entire commit, not just the specific lines you changed

• Linting and build systems enforce different rules

• YAML inside markdown must be structurally correct

• Documentation builds can fail due to structural issues

• Running checks locally significantly reduces debugging time

Conclusion

Debugging a failing pull request isn't just about fixing syntax errors.

You also need to understand how different systems interact:

• Version control

• CI pipelines

• Linting tools

• Build processes

Once you understand how these systems work together, you can debug issues systematically instead of guessing.

The next time your pull request fails, you will know exactly where to start and how to fix it.

Debugging CI issues may feel overwhelming at first, but with a structured approach, you can turn failures into a clear path for improvement.

I did. I spun up an EC2 instance, followed along for an hour, and then forgot to shut it down. A week later I had a $34 bill for a machine running nothing.

That was the last time I practiced on someone else's infrastructure.

Everything in this guide runs on your laptop. No cloud account, no credit card, no bill at the end of the month. By the end, you'll be able to spin up a multi-server environment from scratch, configure it automatically with Ansible, serve a site you wrote yourself, and diagnose what breaks when you intentionally destroy it.

That last part is where the actual learning happens.

Prerequisites

Before you start, make sure you have:

• A laptop with at least 8GB of RAM (16GB is better)

• At least 20GB of free disk space

• Windows, macOS, or Linux operating system

• Administrator access to your computer

• Virtualization enabled in your BIOS/UEFI settings

• A stable internet connection for the initial downloads

Knowledge and comfort level:

• You should be comfortable using a terminal (running commands, changing directories, and editing small text files with whatever editor you like).

• Basic familiarity with concepts like “a server,” “SSH,” and “a port” helps, but you don't need prior experience with Docker, Kubernetes, Vagrant, or Ansible. This guide introduces them as you go.

If you can follow step-by-step instructions and read error output without panicking, you're ready.

Table of Contents

1. What is DevOps?

2. Why Build a Local Lab?

3. How to Set Up Docker

4. How to Set Up Kubernetes

5. How to Install kubectl

6. How to Set Up Vagrant

7. How to Install Ansible

8. How to Build Your First DevOps Project

9. How to Break Your Lab on Purpose

10. What You Can Now Do

What is DevOps?

DevOps is the practice of breaking down the wall between software development and IT operations teams.

Traditionally, developers write code and hand it off to operations teams to deploy and maintain. That handoff causes delays, misunderstandings, and outages. DevOps is what happens when both teams work together from the start.

The tools you'll install in this guide each solve a specific part of that process:

• Docker packages your application and everything it needs into a portable container that runs the same way on any machine.

• Kubernetes manages multiple containers at scale, handling restarts, networking, and load balancing automatically.

• Vagrant creates and manages virtual machine environments so your whole team always works on identical setups.

• Ansible automates repetitive configuration tasks across many servers without writing a script for each one.

Why Build a Local Lab?

A local lab gives you a safe place to break things, fix them, and learn from that process without any cost or risk.

Here's what you get with a local setup:

• Zero cost. No cloud bills, no surprise charges, and no credit card required.

• Works offline. Practice anywhere, even without internet after the initial setup.

• Full control. You manage every layer from the OS up to the application.

• Safe experimentation. Break things freely. Nothing here affects production.

• Fast feedback. No waiting for cloud resources to spin up. Everything runs on your machine.

The tradeoff is resource limits. Your laptop's CPU and RAM are the ceiling. You can't simulate large-scale deployments, and some cloud-native services like AWS Lambda or S3 have no direct local equivalent. But for learning core DevOps workflows, none of that matters.

How to Set Up Docker

Docker is the foundation of this lab. Every other tool in this guide either runs inside Docker containers or works alongside them.

How to Install Docker on Windows

First, enable virtualization in your BIOS:

1. Restart your computer and enter BIOS/UEFI setup. The key is usually F2, F10, Del, or Esc during boot.

2. Find the virtualization setting. It's usually listed as Intel VT-x, AMD-V, SVM, or Virtualization Technology.

3. Enable it, save your changes, and exit.

Then install Docker Desktop:

1. Download Docker Desktop from Docker's official website.

2. Run the installer and follow the prompts.

3. Enable WSL 2 (Windows Subsystem for Linux) when asked.

4. Restart your computer.

5. Open Docker Desktop from the Start menu and wait for the whale icon in the taskbar to stop animating.

Troubleshooting: If Docker fails to start, run this in PowerShell as Administrator to verify virtualization is active:

systeminfo | findstr "Hyper-V Requirements"

All items should show "Yes". If they don't, revisit your BIOS settings.

How to Install Docker on Mac

1. Download Docker Desktop for Mac from Docker's website.

2. Open the downloaded .dmg file and drag Docker to your Applications folder.

3. Open Docker from Applications.

4. Enter your password when prompted.

5. Wait for the whale icon in the menu bar to stop animating.

How to Install Docker on Linux

Run these commands in order:

# Update your package lists
sudo apt-get update

# Install prerequisites
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Add the Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Update and install Docker
sudo apt-get update
sudo apt-get install docker-ce

# Start and enable Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add your user to the docker group
sudo usermod -aG docker $USER

Log out and back in for the group change to take effect.

How to Test Docker

Run this command:

docker run hello-world

If you see "Hello from Docker!" then Docker is working correctly.

Docker is set up. Next, you'll install Kubernetes to manage containers at scale.

How to Set Up Kubernetes

Kubernetes manages containers at scale. For a local lab, you have four options. Here's how to choose:

If you're just starting out, use Minikube. It has the simplest setup and a visual dashboard that helps you understand what's happening inside the cluster.

If your laptop has 8GB RAM or less, use k3s. It runs lean and behaves closer to a real cluster than Minikube does.

Use kubeadm only if you want to understand how Kubernetes nodes join a cluster — it requires more manual steps and isn't beginner-friendly.

How to Install Minikube (Recommended for Beginners)

Minikube creates a single-node Kubernetes cluster on your laptop.

On Windows:

1. Download the Minikube installer from Minikube's GitHub releases page.

2. Run the .exe installer.

3. Open Command Prompt as Administrator and start Minikube:

minikube start --driver=docker

On Mac:

brew install minikube
minikube start --driver=docker

On Linux:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube-linux-amd64
sudo mv minikube-linux-amd64 /usr/local/bin/minikube
minikube start --driver=docker

Test your cluster:

minikube status
minikube dashboard

How to Install k3s (Recommended for Low-RAM Machines)

k3s is a lightweight version of Kubernetes that installs in under a minute. It runs lean and behaves like a real cluster — not a simplified demo version.

On Linux (and Mac via Multipass):

curl -sfL https://get.k3s.io | sh -

That single command installs k3s and runs it automatically in the background. Check that it is running:

sudo k3s kubectl get nodes

You should see one node with status Ready.

On Mac directly — k3s doesn't run natively on macOS. Use Multipass to spin up a lightweight Ubuntu VM first, then run the install command inside it.

On Windows — use WSL2 (Ubuntu), then run the install command inside your WSL2 terminal.

How to Install Kind (Kubernetes IN Docker)

Kind runs a full Kubernetes cluster inside Docker containers. It starts faster than Minikube and is useful if you want to run multiple clusters simultaneously.

# Mac or Linux
brew install kind

# Windows
choco install kind

Create a cluster:

kind create cluster --name my-local-lab

How to Install kubeadm (For Understanding Cluster Bootstrap)

kubeadm is the tool Kubernetes uses to initialize and join nodes in a real cluster. Use this when you want to understand what happens under the hood — not as your daily driver.

It requires at least two machines (or VMs). The setup is more involved than the options above. Follow the official kubeadm installation guide for your OS, then initialize your cluster:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

After init, join worker nodes using the command kubeadm prints at the end of the output.

How to Install kubectl

kubectl is the command-line tool you use to interact with any Kubernetes cluster.

On Windows:

Download kubectl.exe from Kubernetes' website and place it in a directory that is in your PATH. Or install with Chocolatey:

choco install kubernetes-cli

On Mac:

brew install kubectl

On Linux:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl

Test it:

kubectl get pods --all-namespaces

On a fresh cluster, you'll see system pods running in the kube-system namespace — things like coredns and storage-provisioner. That's the expected output. It means your cluster is up and kubectl can talk to it.

Kubernetes is running. Next is Vagrant. But before that, there's one important distinction worth making.

Docker vs Vagrant — they aren't the same thing

Docker creates containers: lightweight processes that share your operating system's kernel. Vagrant creates full virtual machines: isolated computers with their own OS running inside your laptop.

Containers are fast and small. VMs are heavier but behave exactly like real servers. You'll use both in this lab for different reasons.

How to Set Up Vagrant

Vagrant lets you create and manage reproducible virtual machine environments. It is ideal for simulating multi-server setups on a single laptop.

How to Install Vagrant on Windows

1. Download and install VirtualBox with default options.

2. Download and install Vagrant.

3. Restart your computer if prompted.

Note: VirtualBox and Hyper-V can't run at the same time on Windows. Check if Hyper-V is active:

systeminfo | findstr "Hyper-V"

If it's enabled, you have two options: switch to the Hyper-V Vagrant provider, or disable Hyper-V with:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All

Restart after disabling.

How to Install Vagrant on Mac and Linux

On Mac:

1. Download and install VirtualBox.

2. After installation, open System Preferences > Security & Privacy > General. You will see a message saying system software from Oracle was blocked. Click Allow and restart your Mac. Without this step, VirtualBox will not run.

3. Download and install Vagrant.

Note for Apple Silicon (M1/M2/M3) Macs: VirtualBox support on Apple Silicon is still limited. If you're on an M-series Mac, use UTM as your VM provider instead, or use Multipass which works natively on Apple Silicon.

On Linux:

1. Download and install VirtualBox.

2. Download and install Vagrant.

Verify both are installed:

vboxmanage --version
vagrant --version

How to Create Your First Vagrant Environment

Create a new directory for your project. Inside it, create a file named Vagrantfile with this content:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  # Create a private network between VMs
  config.vm.network "private_network", type: "dhcp"

  # Forward port 8080 on your laptop to port 80 on the VM
  config.vm.network "forwarded_port", guest: 80, host: 8080

  # Install Nginx when the VM starts
  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y nginx
    echo "Hello from Vagrant!" > /var/www/html/index.html
  SHELL
end

Start the VM:

vagrant up

Visit http://localhost:8080 in your browser. You should see "Hello from Vagrant!"

Troubleshooting SSH on Windows

If vagrant ssh fails, try:

vagrant ssh -- -v

Or connect manually:

ssh -i .vagrant/machines/default/virtualbox/private_key vagrant@127.0.0.1 -p 2222

How to Create a Local Vagrant Box Without Internet

Note: Most readers can skip this. Only do this if you want to work fully offline after the initial setup.

1. Download Ubuntu 20.04 LTS and save the .iso file locally.

2. Open VirtualBox and create a new VM: Name it ubuntu-devops, Type: Linux, Version: Ubuntu (64-bit).

3. Assign 2048MB RAM and a 20GB VDI disk.

4. Attach the .iso under Storage > Optical Drive.

5. Start the VM and complete the Ubuntu installation.

6. Once installed, shut down the VM and run:

VBoxManage list vms
vagrant package --base "ubuntu-devops" --output ubuntu2004.box
vagrant box add ubuntu2004 ubuntu2004.box

You now have a reusable local box that works without internet.

You can spin up virtual machines. Next is Ansible, which automates what goes inside them.

How to Install Ansible

Ansible automates configuration and software installation across multiple servers. Instead of SSH-ing into ten machines and running the same commands manually, you write a playbook once and Ansible handles the rest.

How to Install Ansible on Windows

Ansible doesn't run natively on Windows. You need to use it through WSL (Windows Subsystem for Linux).

1. Open PowerShell as Administrator and enable WSL:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

1. Restart your computer.

2. Install Ubuntu from the Microsoft Store.

3. Open Ubuntu and install Ansible:

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

How to Install Ansible on Mac

brew install ansible

How to Install Ansible on Linux

# Ubuntu/Debian
sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

# Red Hat/CentOS
sudo yum install ansible

How to Test Ansible

Create a file called hosts in your current directory:

[local]
localhost ansible_connection=local

Create a file called playbook.yml in the same directory:

---
- name: Test playbook
  hosts: local
  tasks:
    - name: Print a message
      debug:
        msg: "Ansible is working!"

Run the playbook, passing the local hosts file with -i:

ansible-playbook -i hosts playbook.yml

You should see the message "Ansible is working!" in the output.

Alright, all your tools are installed. Now you'll use them together to build something real.

How to Build Your First DevOps Project

You can find the entire code for this lab in this repo: https://github.com/Osomudeya/homelab-demo-article

Now you'll put these tools together in one project. Each tool will perform its actual job, and nothing is forced.

Before you start, create a fresh directory for this project. Don't run it inside the directory you used to test Vagrant earlier, as the Vagrantfile here is different and will conflict.

You'll be building a two-VM environment: one machine serves a web page you write yourself inside a Docker container, and the other runs a MariaDB database. Vagrant creates the machines and Ansible configures them. The page you see at the end is yours.

Step 1: Create the Project Directory

mkdir devops-lab-project && cd devops-lab-project

Step 2: Write Your Site Content

Create a file called index.html in the project directory. Write whatever you want on this page — it's what you'll see in your browser at the end:

<!DOCTYPE html>
<html>
  <head><title>My DevOps Lab</title></head>
  <body>
    <h1>My DevOps Lab</h1>
    <p>Provisioned by Vagrant. Configured by Ansible. Served by Docker.</p>
    <p>Built on a laptop. No cloud account needed.</p>
  </body>
</html>

Change the text to whatever you like. This is your page.

Step 3: Write the Vagrantfile

Create a file called Vagrantfile in the same directory:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  config.vm.define "web" do |web|
    web.vm.network "private_network", ip: "192.168.33.10"
    web.vm.network "forwarded_port", guest: 80, host: 8080
  end

  config.vm.define "db" do |db|
    db.vm.network "private_network", ip: "192.168.33.11"
  end
end

Step 4: Start the Virtual Machines

vagrant up

The first run downloads the ubuntu/focal64 box, which is around 500MB.

Expect this to take 10–30 minutes depending on your connection. Subsequent runs will be much faster since the box is cached locally.

Step 5: Create the Ansible Inventory

Create a file called inventory in the same directory:

[webservers]
192.168.33.10 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

[dbservers]
192.168.33.11 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/db/virtualbox/private_key

Ansible uses the Vagrant-generated private keys so it can SSH in as the vagrant user. Host key checking for this lab is turned off in ansible.cfg (next step), not in the inventory.

Step 6: Create the Ansible Config File

Before running the playbook, create a file called ansible.cfg in the same directory:

[defaults]
inventory = inventory
host_key_checking = False

The inventory line tells Ansible to use the inventory file in this folder by default. host_key_checking = False tells Ansible not to verify SSH host keys when connecting to your Vagrant VMs. Without it, Ansible will fail with a Host key verification failed error on first connection because the VM's key is not yet in your known_hosts file.

These settings are for a local lab only. Do not use host_key_checking = False for production systems.

Step 7: Create the Ansible Playbook

Create a file called playbook.yml:

---
- name: Configure web server
  hosts: webservers
  become: yes
  tasks:

    - name: Install Docker
      apt:
        name: docker.io
        state: present
        update_cache: yes

    - name: Start Docker service
      service:
        name: docker
        state: started
        enabled: yes

    # Create the directory that will hold your site content
    - name: Create web content directory
      file:
        path: /var/www/html
        state: directory
        mode: '0755'

    # This copies your index.html from your laptop into the VM
    - name: Copy site content to web server
      copy:
        src: index.html
        dest: /var/www/html/index.html

    # This mounts that file into the Nginx container so it serves your page
    # The -v flag connects /var/www/html on the VM to /usr/share/nginx/html inside the container
    - name: Run Nginx serving your content
      shell: |
        docker rm -f webapp 2>/dev/null || true
        docker run -d --name webapp --restart always -p 80:80 \
          -v /var/www/html:/usr/share/nginx/html:ro nginx

- name: Configure database server
  hosts: dbservers
  become: yes
  tasks:

    # Hash sum mismatch on .deb downloads is often stale lists, a flaky mirror, or apt pipelining
    # behind NAT; fresh indices + Pipeline-Depth 0 usually fixes it on lab VMs.
    - name: Disable apt HTTP pipelining (mirror/proxy hash mismatch workaround)
      copy:
        dest: /etc/apt/apt.conf.d/99disable-pipelining
        content: 'Acquire::http::Pipeline-Depth "0";'
        mode: "0644"

    - name: Clear apt package index cache
      shell: apt-get clean && rm -rf /var/lib/apt/lists/* /var/lib/apt/lists/auxfiles/*
      changed_when: true

    - name: Update apt cache after reset
      apt:
        update_cache: yes

    - name: Install MariaDB
      apt:
        name: mariadb-server
        state: present
        update_cache: no

    - name: Start MariaDB service
      service:
        name: mariadb
        state: started
        enabled: yes

Two lines worth paying attention to:

• src: index.html — Ansible looks for this file in the same directory as the playbook. That is the file you wrote in Step 2.

• -v /var/www/html:/usr/share/nginx/html:ro — this mounts the directory from the VM into the Nginx container. The :ro means read-only. Nginx serves whatever is in that folder.

Step 8: Run the Playbook

ansible-playbook -i inventory playbook.yml

You'll see task-by-task output as Ansible connects to each VM over SSH and configures it. A green ok or yellow changed next to each task means it worked. Red fatal means something failed.

Step 9: Verify the Setup

Open http://localhost:8080 in your browser. You should see the page you wrote in Step 2 served from inside a Docker container, running on a Vagrant VM, configured automatically by Ansible.

If you see the page, every tool in this lab is working together.

Step 9: Clean Up (Optional)

When you're done:

vagrant destroy -f

This shuts down and deletes both VMs. Your Vagrantfile, inventory, playbook.yml, and index.html stay on disk — run vagrant up followed by ansible-playbook -i inventory playbook.yml any time to bring it all back.

Now that you have a working lab, let's use it properly.

How to Break Your Lab on Purpose

Following these steps has gotten you a running lab. Breaking things teaches you how everything actually works.

Here are five things to break and what to look for when you do.

Break 1: Crash the Main Process Inside the Container (and Watch It Come Back)

Doing this just proves that something inside the container can die (like a real bug or OOM), Docker can restart the container because of --restart always, and your site can come back without re-running Ansible.

After vagrant ssh web, every docker command below runs on the web VM. So keep your browser on your laptop at http://localhost:8080 (Vagrant forwards your host port to the VM’s port 80).

Troubleshooting: If Your Lab Isn't Ready

From your project folder on the host (your laptop) – unless the step says to run it on the VM:

• You ran vagrant destroy -f. Run vagrant up, then ansible-playbook -i inventory playbook.yml.

• docker ps shows webapp but status is Exited. On the web VM, run sudo docker start webapp, then sudo docker ps again.

• There's no webapp row in docker ps -a. Re-run ansible-playbook -i inventory playbook.yml on the host.

If the playbook is already applied and webapp is Up, skip this section and start at step 1 under Steps (happy path) below. (Don't skip SSH or docker ps. You need the VM shell and a quick check before you run docker exec.)

Steps (happy path)

1. SSH into the web VM:

vagrant ssh web

1. Confirm webapp is Up:

   sudo docker ps
   

2. Break it on purpose: kill the container’s main process from inside (PID 1). That ends the container the same way a crashing app would, not the same as docker stop on the host:

sudo docker exec webapp sh -c 'sleep 5 && kill 1'

The sleep 5 gives you a moment to switch to the browser. Right after you run the command, open or refresh http://localhost:8080. You may catch a brief error or blank page while nothing is listening on port 80.

1. Watch Docker restart the container:

watch sudo docker ps -a

Within a few seconds you should see Exited (137) become Up again. (Press Ctrl+C to exit watch.)

5. Refresh the browser. You should see the same HTML as before, because the files live on the VM under /var/www/html and are bind-mounted into the container; restarting only replaced the Nginx process, not those files.

Why not docker stop or docker kill on the host for this demo?

Those commands go through Docker’s API. On many setups (including recent Docker), Docker treats them as you choosing to stop the container (hasBeenManuallyStopped), and --restart always may not bring the container back until you docker start it or similar.

Killing PID 1 from inside the container is treated more like an internal crash, so the restart policy you set in the playbook is the one you actually get to observe here.

Kubernetes analogy: A pod whose containers exit can be restarted by the kubelet; a pod you delete does not come back by itself.

What to observe (three separate checks):

1. Exit code: After kill 1, docker ps -a should show the container exited with code 137, meaning the main process was killed by a signal. That confirms the container really died, not that you ran docker stop on the host.

2. Restart delay vs browser: Watch how many seconds pass between Exited and Up in docker ps -a; that interval is Docker applying --restart always. That's separate from what you see in the browser: the browser only shows whether something is accepting connections on port 80 on the VM, so it may show an error or blank page during the gap even while Docker is about to restart the container.

3. Content after recovery: After status is Up again, refresh the page. You should see the same HTML as before. That shows your content lives on the VM disk (mounted into the container with -v), not inside a file that vanishes when the container process restarts. The process was replaced, not your index.html on the host path.

Break 2: Cause a Container Name Conflict

On a single Docker daemon (here, on your web VM), a container name is a unique label. Two running (or stopped) containers can't share the same name. Scripts and playbooks that always use docker run --name webapp without cleaning up first hit this error constantly and recognizing it saves time in real work.

Before you start: Ansible already created one container named webapp.
Stay on the web VM (for example still inside vagrant ssh web) so the commands below run where that container lives.

So now, try to start a second container and also call it webapp. The image is plain nginx here on purpose – the point is the name clash, not matching your site’s ports or volume mounts.

sudo docker run -d --name webapp nginx

What actually happens here is that Docker doesn't create a second container. It returns an error immediately. Your original webapp is unchanged.

This is because the name webapp is already registered to the existing container (the error shows that container’s ID). Docker refuses to reuse the name until the old container is removed or renamed.

Example error (your ID will differ):

docker: Error response from daemon: Conflict. The container name "/webapp" is already in use by container "2e48b81a311c4b71cdc1e25e0df75a22296845c7eb53aab82f9ae739fb6410ec". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

To fix it, free the name, then create webapp again the same way the playbook does (publish port 80, mount your HTML, restart policy):

sudo docker rm -f webapp
sudo docker run -d --name webapp --restart always -p 80:80 \
  -v /var/www/html:/usr/share/nginx/html:ro nginx

After that, your site should behave as before (refresh http://localhost:8080 from your laptop).

What to observe:

Read Docker’s Conflict message end to end. You should see that the name /webapp is already in use and a container ID pointing at the existing box. In production, that pattern means “something already claimed this name. Just remove it, rename it, or pick a different name before you run docker run again.”

Break 3: Make Ansible Fail to Reach a VM

Ansible separates “could not connect” from “connected, but a task broke.” The first is UNREACHABLE, the second is FAILED. Knowing which one you have tells you whether to fix network / SSH or playbook / packages / permissions.

On your laptop, in the project folder, edit inventory and change the web server address from 192.168.33.10 to an IP no VM uses, for example 192.168.33.99. Save the file.

[webservers]
192.168.33.99 ansible_user=vagrant ansible_ssh_private_key_file=.vagrant/machines/web/virtualbox/private_key

What you run (from the same project folder on the host):

ansible-playbook -i inventory playbook.yml

After this, Ansible tries to SSH to 192.168.33.99. Nothing on your lab network answers as that host (or SSH never succeeds), so Ansible never runs tasks on the web server. It stops that host with UNREACHABLE:

fatal: [192.168.33.99]: UNREACHABLE! => {"msg": "Failed to connect to the host via ssh"}

This is realistic because the same message shape appears when the IP is wrong, the VM isn't running, a firewall blocks port 22, or the network is misconfigured. The common thread is no working SSH session.

Now it's time to put it back: restore 192.168.33.10 in inventory and run ansible-playbook -i inventory playbook.yml again. The web play should reach the VM and complete (assuming your lab is up).

UNREACHABLE vs FAILED – what to observe:

• If Ansible prints UNREACHABLE, you should assume it never opened SSH on that host and never ran tasks there. Go ahead and fix the connection (IP, VM up, firewall, key path) before you debug playbook logic.

• If Ansible prints FAILED, you should assume SSH worked and a task returned an error. Read the task output for the real cause (package name, permissions, syntax), not the network first.

When you debug later, you should look at the keyword Ansible prints: UNREACHABLE points to reachability while FAILED points to task output and the first failed task under that host.

Break 4: Fill the VM's Disk

Databases and other services need free disk for logs, temp files, and data. When the filesystem is full or nearly full, a service may fail to start or fail at runtime. This break walks through the same diagnosis habit you would use on a real server: check space, then read systemd and journal output for the service.

All commands below run on the db VM after vagrant ssh db. MariaDB was installed there by your playbook.

What you do:

1. Open a shell on the db VM:

   vagrant ssh db
   

2. Allocate a large file full of zeros (here 1GB) to simulate something eating disk space:

   sudo dd if=/dev/zero of=/tmp/bigfile bs=1M count=1024
   
   df -h
   

   Use df -h to see how full the root filesystem (or relevant mount) is. Your Vagrant disk may be large enough that 1GB only raises usage. If MariaDB still starts, you still practiced the checks. To see a stronger effect, you can repeat with a larger count= only in a lab (never fill production disks on purpose without a plan).

3. Ask systemd to restart MariaDB and show status:

   sudo systemctl restart mariadb
   sudo systemctl status mariadb
   

   If the disk is critically full, restart may fail or the service may show failed or not running.

4. If something looks wrong, read recent logs for the MariaDB unit:

   sudo journalctl -u mariadb --no-pager | tail -20
   

   Errors often mention disk, space, read-only filesystem, or InnoDB being unable to write.

5. Clean up so your VM stays usable:

   sudo rm /tmp/bigfile
   

   Optionally run sudo systemctl restart mariadb again and confirm it is active (running).

What to observe:

• You should use df -h first to confirm whether the filesystem is actually tight. That avoids blaming the database when disk space is fine.

• You should read systemctl status mariadb to see whether systemd thinks the service is active, failed, or flapping.

• You should read journalctl -u mariadb when status is bad, so you can tie the failure to concrete errors from MariaDB or the kernel (often mentioning disk, space, or read-only filesystem). Space + status + logs is the same order you would use on a production server.

Break 5: Run Minikube Out of Resources

Kubernetes schedules pods onto nodes that have enough CPU and memory. If you ask for more than the cluster can place, some pods stay Pending and Events explain why (for example Insufficient cpu). That is not the same as a pod that starts and then crashes.

To do this, you'll need a local cluster (we're using Minikube in this guide) and kubectl on your laptop. This break doesn't use the Vagrant VMs. If you haven't installed Minikube yet, complete the "How to Set Up Kubernetes" section first, or skip this break until you do.

You'll run this on your Mac, Linux, or Windows terminal (host), not inside vagrant ssh. If you're still inside a VM, type exit until your prompt is back on the host.

What you do:

1. Check Minikube:

   minikube status
   

   If it's stopped, start it (Docker driver matches earlier sections):

   minikube start --driver=docker
   

2. Create a deployment with many replicas so your single Minikube node can't run them all at once:

   kubectl create deployment stress --image=nginx --replicas=20
   
   #watch pods start
   kubectl get pods -w
   

   Press Ctrl+C when you're done watching. Some pods may stay Pending while others are Running.

3. Pick one Pending pod name from kubectl get pods and inspect it:

   kubectl describe pod <pod-name>
   

   Under Events, look for FailedScheduling and a line similar to:

   Warning  FailedScheduling  0/1 nodes are available: 1 Insufficient cpu.
   

   You might see Insufficient memory instead, depending on your machine.

4. Fix the lab by scaling back so the cluster can catch up:

   kubectl scale deployment stress --replicas=2
   

   You can delete the deployment entirely when finished: kubectl delete deployment stress.

What to observe:

• You should see Pending pods stay unscheduled until capacity frees up. That means the scheduler hasn't placed them on any node yet, usually because the node is out of CPU or memory for that workload.

• You should read kubectl describe pod <pod-name> and scroll to Events. Messages like Insufficient cpu or Insufficient memory mean the cluster ran out of schedulable capacity, not that the container image image is corrupt.

• You should contrast that with a pod that reaches Running and then CrashLoopBackOff, which usually means the process inside the container keeps exiting. that is an application or config problem, not a “nowhere to run” problem.

What You Can Now Do

You didn't just install tools in this tutorial. You also used them.

You can now spin up two servers from a single file. You can write a playbook that installs software and deploys a container without touching either machine manually.

You can serve a page you wrote from inside a Docker container running on a Vagrant VM, and bring the whole thing back from scratch in one command.

You also broke it. You saw what a container conflict looks like, what Ansible prints when it can't reach a machine, what disk pressure does to a running service, and what a Kubernetes scheduler says when it runs out of resources. Those error messages aren't unfamiliar anymore.

That's the difference between someone who has read about DevOps and someone who has run it.

Here are four free projects you can run in this same lab to go further:

• DevOps Home-Lab 2026 — Build a multi-service app (frontend, API, PostgreSQL, Redis) end-to-end with Docker Compose, Kubernetes, Prometheus/Grafana monitoring, GitOps with ArgoCD, and Cloudflare for global exposure.

• KubeLab — Trigger real Kubernetes failure scenarios, pod crashes, OOMKills, node drains, cascading failures, and watch how the cluster responds using live metrics.

• K8s Secrets Lab — Build a full secret management pipeline from AWS Secrets Manager into your cluster, including rotation behavior and IRSA.

• DevOps Troubleshooting Toolkit — Structured debugging guides across Linux, containers, Kubernetes, cloud, databases, and observability with copy-paste commands for real incidents.

All free and open source: github.com/Osomudeya/List-Of-DevOps-Projects.

If you want to go deeper, you can find six full chapters covering Terraform, Ansible, monitoring, CI/CD, and a simulated three-VM production environment at Build Your Own DevOps Lab.

Google Cloud Axion is Google's own custom ARM-based chip, built to handle the demands of modern cloud workloads. The performance and cost numbers are striking: Google claims Axion delivers up to 60% better energy efficiency and up to 65% better price-performance compared to comparable x86 machines.

AWS has Graviton. Azure has Cobalt. ARM is no longer niche. It's the direction the entire cloud industry is moving.

But there's a problem that catches almost every team off guard when they start this transition: container architecture mismatch.

If you build a Docker image on your M-series Mac and push it to an x86 server, it crashes on startup with a cryptic exec format error.

The server isn't broken. It just can't read the compiled instructions inside your image. An ARM binary and an x86 binary are written in fundamentally different languages at the machine level. The CPU literally can't execute instructions it wasn't designed for.

We're going to solve this problem completely in this tutorial. You'll build a single Docker image tag that automatically serves the correct binary on both ARM and x86 machines — no separate pipelines, no separate tags. Then you'll provision Google Cloud ARM nodes in GKE and configure your Kubernetes deployment to route workloads precisely to those cost-efficient nodes.

Here's what you'll build, step by step:

• A Go HTTP server that reports the CPU architecture it's running on at runtime

• A multi-stage Dockerfile that cross-compiles for both linux/amd64 and linux/arm64 without slow QEMU emulation

• A multi-arch image in Google Artifact Registry that acts as a single entry point for any architecture

• A GKE cluster with two node pools: a standard x86 pool and an ARM Axion pool

• A Kubernetes Deployment that pins your workload exclusively to the ARM nodes

By the end, you'll hit a live endpoint and see the word arm64 staring back at you from a Google Cloud ARM node. Let's get into it.

Table of Contents

• Prerequisites

• Step 1: Set Up Your Google Cloud Project

• Step 2: Create the GKE Cluster

• Step 3: Write the Application

• Step 4: Enable Multi-Arch Builds with Docker Buildx

• Step 5: Write the Dockerfile

• Step 6: Build and Push the Multi-Arch Image

• Step 7: Add the Axion ARM Node Pool

• Step 8: Deploy the App to the ARM Node Pool

• Step 9: Verify the Deployment

• Step 10: Cost Savings and Tradeoffs

• Cleanup

• Conclusion

• Project File Structure

Prerequisites

Before you start, make sure you have the following ready:

• A Google Cloud project with billing enabled. If you don't have one, create it at console.cloud.google.com. The total cost to follow this tutorial is around $5–10.

• gcloud CLI installed and authenticated. Run gcloud auth login to sign in and gcloud config set project YOUR_PROJECT_ID to point it at your project.

• Docker Desktop version 19.03 or later. Docker Buildx (the tool we'll use for multi-arch builds) ships bundled with it.

• kubectl installed. This is the CLI for interacting with Kubernetes clusters.

• Basic familiarity with Docker (images, layers, Dockerfile) and Kubernetes (pods, deployments, services). You don't need to be an expert, but you should know what these things are.

Step 1: Set Up Your Google Cloud Project

Before writing a single line of application code, let's get the cloud infrastructure side ready. This is the foundation everything else will build on.

Enable the Required APIs

Google Cloud services are off by default in any new project. Run this command to turn on the three APIs we'll need:

gcloud services enable \
  artifactregistry.googleapis.com \
  container.googleapis.com \
  containeranalysis.googleapis.com

Here's what each one does:

• artifactregistry.googleapis.com — enables Artifact Registry, where we'll store our Docker images

• container.googleapis.com — enables Google Kubernetes Engine (GKE), where our cluster will run

• containeranalysis.googleapis.com — enables vulnerability scanning for images stored in Artifact Registry

Create a Docker Repository in Artifact Registry

Artifact Registry is Google Cloud's managed container image store — the place where our built images will live before being deployed to the cluster. Create a dedicated repository for this tutorial:

gcloud artifacts repositories create multi-arch-repo \
  --repository-format=docker \
  --location=us-central1 \
  --description="Multi-arch tutorial images"

Breaking down the flags:

• --repository-format=docker — tells Artifact Registry this repository stores Docker images (as opposed to npm packages, Maven artifacts, and so on)

• --location=us-central1 — the Google Cloud region where your images will be stored. Use a region that's close to where your cluster will run to minimize image pull latency. Run gcloud artifacts locations list to see all options.

• --description — a human-readable label for the repository, shown in the console.

Authenticate Docker to Push to Artifact Registry

Docker needs credentials before it can push images to Google Cloud. Run this command to wire up authentication automatically:

gcloud auth configure-docker us-central1-docker.pkg.dev

This adds a credential helper entry to your ~/.docker/config.json file. What that means in practice: any time Docker tries to push or pull from a URL under us-central1-docker.pkg.dev, it will automatically call gcloud to get a valid auth token. You won't need to run docker login manually.

Step 2: Create the GKE Cluster

With Artifact Registry ready to receive images, let's create the Kubernetes cluster. We'll start with a standard cluster using x86 nodes and add an ARM node pool later once we have an image to deploy.

gcloud container clusters create axion-tutorial-cluster \
  --zone=us-central1-a \
  --num-nodes=2 \
  --machine-type=e2-standard-2 \
  --workload-pool=PROJECT_ID.svc.id.goog

Replace PROJECT_ID with your actual Google Cloud project ID.

What each flag does:

• --zone=us-central1-a — creates a zonal cluster in a single availability zone. A regional cluster (using --region) would spread nodes across three zones for higher resilience, but for this tutorial a single zone keeps things simple and avoids capacity issues that can affect specific zones. If us-central1-a is unavailable, try us-central1-b.

• --num-nodes=2 — two x86 nodes in this zone. We need at least 2 to have enough capacity alongside our ARM node pool later.

• --machine-type=e2-standard-2 — the machine type for this default node pool. e2-standard-2 is a cost-effective x86 machine with 2 vCPUs and 8 GB of memory, good for general workloads.

• --workload-pool=PROJECT_ID.svc.id.goog — enables Workload Identity, which is Google's recommended way for pods to authenticate with Google Cloud APIs. It avoids the need to download and store service account key files inside your cluster.

This command takes a few minutes. While it runs, you can move on to writing the application. We'll come back to the cluster in Step 6.

Step 3: Write the Application

We need an application to containerize. We'll use Go for three specific reasons:

1. Go compiles into a single, statically-linked binary. There's no runtime to install, no interpreter — just the binary. This makes for extremely lean container images.

2. Go has first-class, built-in cross-compilation support. We can compile an ARM64 binary from an x86 Mac, or vice versa, by setting two environment variables. This will matter a lot when we get to the Dockerfile.

3. Go exposes the architecture the binary was compiled for via runtime.GOARCH. Our server will report this at runtime, giving us hard proof that the correct binary is running on the correct hardware.

Start by creating the project directories:

mkdir -p hello-axion/app hello-axion/k8s
cd hello-axion/app

Initialize the Go module from inside app/. This creates go.mod in the current directory:

go mod init hello-axion

go mod init is Go's built-in command for starting a new module. It writes a go.mod file that declares the module name (hello-axion) and the minimum Go version required. Every modern Go project needs this file — without it, the compiler doesn't know how to resolve packages.

Now create the application at app/main.go:

package main

import (
    "fmt"
    "net/http"
    "os"
    "runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
    hostname, _ := os.Hostname()
    fmt.Fprintf(w, "Hello from freeCodeCamp!\n")
    fmt.Fprintf(w, "Architecture : %s\n", runtime.GOARCH)
    fmt.Fprintf(w, "OS           : %s\n", runtime.GOOS)
    fmt.Fprintf(w, "Pod hostname : %s\n", hostname)
}

func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintln(w, "ok")
}

func main() {
    http.HandleFunc("/", handler)
    http.HandleFunc("/healthz", healthz)
    fmt.Println("Server starting on port 8080...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Fprintf(os.Stderr, "server error: %v\n", err)
        os.Exit(1)
    }
}

Verify both files were created:

ls -la

You should see go.mod and main.go listed.

Let's walk through what this code does:

• import "runtime" — imports Go's built-in runtime package, which exposes information about the Go runtime environment, including the CPU architecture.

• runtime.GOARCH — returns a string like "arm64" or "amd64" representing the architecture this binary was compiled for. When we deploy to an ARM node, this value will be arm64. This is the core of our proof.

• os.Hostname() — returns the pod's hostname, which Kubernetes sets to the pod name. This lets us see which specific pod responded when we test the app later.

• handler — the main HTTP handler, registered on the root path /. It writes the architecture, OS, and hostname to the response.

• healthz — a separate handler registered on /healthz. It returns HTTP 200 with the text ok. Kubernetes will use this endpoint to check whether the container is alive and ready to serve traffic — we'll wire this up in the deployment manifest later.

• http.ListenAndServe(":8080", nil) — starts the server on port 8080. If it fails to start (for example, if the port is already in use), it prints the error and exits with a non-zero code so Kubernetes knows something went wrong.

Step 4: Enable Multi-Arch Builds with Docker Buildx

Before we write the Dockerfile, we need to understand a fundamental constraint, because it directly shapes how the Dockerfile must be written.

Why Your Docker Images Are Architecture-Specific By Default

A CPU only understands instructions written for its specific Instruction Set Architecture (ISA). ARM64 and x86_64 are different ISAs — different vocabularies of machine-level operations. When you compile a Go program, the compiler translates your source code into binary instructions for exactly one ISA. That binary can't run on a different ISA.

When you build a Docker image the normal way (docker build), the binary inside that image is compiled for your local machine's ISA. If you're on an Apple Silicon Mac, you get an ARM64 binary. Push that image to an x86 server, and when Docker tries to execute the binary, the kernel rejects it:

standard_init_linux.go:228: exec user process caused: exec format error

That's the operating system saying: "This binary was written for a different processor. I have no idea what to do with it."

The Solution: A Single Image Tag That Serves Any Architecture

Docker solves this with a structure called a Manifest List (also called a multi-arch image index). Instead of one image, a Manifest List is a pointer table. It holds multiple image references — one per architecture — all under the same tag.

When a server pulls hello-axion:v1, here's what actually happens:

1. Docker contacts the registry and requests the manifest for hello-axion:v1

2. The registry returns the Manifest List, which looks like this internally:

{
  "manifests": [
    { "digest": "sha256:a1b2...", "platform": { "architecture": "amd64", "os": "linux" } },
    { "digest": "sha256:c3d4...", "platform": { "architecture": "arm64", "os": "linux" } }
  ]
}

1. Docker checks the current machine's architecture, finds the matching entry, and pulls only that specific image layer. The x86 image never downloads onto your ARM server, and vice versa.

One tag, two actual images. Completely transparent to your deployment manifests.

Set Up Docker Buildx

Docker Buildx is the CLI tool that builds these Manifest Lists. It's powered by the BuildKit engine and ships bundled with Docker Desktop. Run the following to create and activate a new builder instance:

docker buildx create --name multiarch-builder --use

• --name multiarch-builder — gives this builder a memorable name. You can have multiple builders. This command creates a new one named multiarch-builder.

• --use — immediately sets this new builder as the active one, so all future docker buildx build commands use it.

Now boot the builder and confirm it supports the platforms we need:

docker buildx inspect --bootstrap

• --bootstrap — starts the builder container if it isn't already running, and prints its full configuration.

You should see output like this:

Name:          multiarch-builder
Driver:        docker-container
Platforms:     linux/amd64, linux/arm64, linux/arm/v7, linux/386, ...

The Platforms line lists every architecture this builder can produce images for. As long as you see linux/amd64 and linux/arm64 in that list, you're ready to build for both x86 and ARM.

Step 5: Write the Dockerfile

Now we can write the Dockerfile. We'll use two techniques together: a multi-stage build to keep the final image tiny, and a cross-compilation trick to avoid slow CPU emulation.

Create app/Dockerfile with the following content:

# -----------------------------------------------------------
# Stage 1: Build
# -----------------------------------------------------------
# $BUILDPLATFORM = the machine running this build (your laptop)
# \(TARGETOS / \)TARGETARCH = the platform we are building FOR
# -----------------------------------------------------------
FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

ARG TARGETOS
ARG TARGETARCH

WORKDIR /app

COPY go.mod .
RUN go mod download

COPY main.go .

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

# -----------------------------------------------------------
# Stage 2: Runtime
# -----------------------------------------------------------

FROM alpine:latest

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

WORKDIR /app
COPY --from=builder /app/server .

EXPOSE 8080
CMD ["./server"]

There's a lot happening here. Let's go through it carefully.

Stage 1: The Builder

FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

This is the most important line in the file. \(BUILDPLATFORM is a special build argument that Docker Buildx automatically injects — it equals the platform of the machine running the build (your laptop). By pinning the builder stage to \)BUILDPLATFORM, the Go compiler always runs natively on your machine, not inside a CPU emulator. This is what makes multi-arch builds fast.

Without --platform=$BUILDPLATFORM, Buildx would have to use QEMU — a full CPU emulator — to run an ARM64 build environment on your x86 machine (or vice versa). QEMU works, but it's typically 5–10 times slower than native execution. For a project with many dependencies, that's the difference between a 2-minute build and a 20-minute build.

ARG TARGETOS and ARG TARGETARCH

These two lines declare that our Dockerfile expects build arguments named TARGETOS and TARGETARCH. Buildx injects these automatically based on the --platform flag you pass at build time. For a linux/arm64 target, TARGETOS will be linux and TARGETARCH will be arm64.

COPY go.mod . and RUN go mod download

We copy go.mod first, before copying the rest of the source code. Docker builds images layer by layer and caches each layer. By copying only the module file first, we create a cached layer for go mod download.

On future builds, as long as go.mod hasn't changed, Docker skips the download step entirely — even if the source code changed. This speeds up iterative development significantly.

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

This is the cross-compilation step. GOOS and GOARCH are Go's built-in cross-compilation environment variables. Setting them tells the Go compiler to produce a binary for a different target than the machine it's running on. We set them from the \(TARGETOS and \)TARGETARCH build args injected by Buildx.

The -ldflags="-w -s" flag strips the debug symbol table and the DWARF debugging information from the binary. This has no effect on runtime behavior but reduces the binary size by roughly 30%.

Stage 2: The Runtime Image

FROM alpine:latest

This starts a brand-new image from Alpine Linux — a minimal Linux distribution that weighs about 5 MB. Critically, alpine:latest is itself a multi-arch image, so Docker automatically selects the arm64 or amd64 Alpine variant depending on which platform this stage is built for.

Everything from Stage 1 — the Go toolchain, the source files, the intermediate object files — is discarded. The final image contains only Alpine Linux plus our binary. Compared to a naive single-stage Go image (~300 MB), this approach produces an image under 15 MB.

RUN addgroup -S appgroup && adduser -S appuser -G appgroup and USER appuser

These two lines create a non-root user and set it as the active user for the container. Running containers as root is a security risk — if an attacker exploits a vulnerability in your application, they gain root access inside the container. Running as a non-root user limits the blast radius.

COPY --from=builder /app/server .

This is how multi-stage builds work: the --from=builder flag tells Docker to copy files from the builder stage (Stage 1), not from your local disk. Only the compiled binary (server) makes it into the final image.

Step 6: Build and Push the Multi-Arch Image

With the application and Dockerfile in place, we can now build images for both architectures and push them to Artifact Registry — all in a single command.

From inside the app/ directory, run:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 \
  --push \
  .

Replace PROJECT_ID with your actual GCP project ID.

Here's what each part of this command does:

• docker buildx build — uses the Buildx CLI instead of the standard docker build. Buildx is required for multi-platform builds.

• --platform linux/amd64,linux/arm64 — instructs Buildx to build the image twice: once targeting x86 Intel/AMD machines, and once targeting ARM64. Both builds run in parallel. Because our Dockerfile uses the $BUILDPLATFORM cross-compilation trick, both builds run natively on your machine without QEMU emulation.

• -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 — the full image path in Artifact Registry. The format is always REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME:TAG.

• --push — multi-arch images can't be loaded into your local Docker daemon (which only understands single-architecture images). This flag tells Buildx to skip local storage and push the completed Manifest List — with both architecture variants — directly to the registry.

• . — the build context, the directory Docker scans for the Dockerfile and any files the build needs.

Watch the output as the build runs. You'll see BuildKit working on both platforms simultaneously:

 => [linux/amd64 builder 1/5] FROM golang:1.23-alpine
 => [linux/arm64 builder 1/5] FROM golang:1.23-alpine
 ...
 => pushing manifest for us-central1-docker.pkg.dev/.../hello-axion:v1

Verify the Multi-Arch Image in Artifact Registry

Once the push completes, navigate to GCP Console → Artifact Registry → Repositories → multi-arch-repo and click on hello-axion.

You won't see a single image — you'll see something labelled "Image Index". That's the Manifest List we created. Click into it, and you'll find two child images with separate digests, one for linux/amd64 and one for linux/arm64.

You can also inspect this from the command line:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

The output lists every manifest inside the image index. You'll see entries for linux/amd64 and linux/arm64 — those are our two real images. You'll also see two entries with Platform: unknown/unknown labelled as attestation-manifest. These are build provenance records that Docker Buildx automatically attaches to prove how and where the image was built (a supply chain security feature called SLSA attestation).

The two entries you care about are linux/amd64 and linux/arm64. Note the digest for the arm64 entry — we'll use it in the verification step to confirm the cluster pulled the right variant.

Step 7: Add the Axion ARM Node Pool

We have a universal image. Now we need somewhere to run it.

Recall the cluster we created in Step 2 — it's running e2-standard-2 x86 machines. We're going to add a second node pool running ARM machines. This is the key architectural move: a mixed-architecture cluster where different workloads can be routed to different hardware.

Choosing Your ARM Machine Type

Google Cloud currently offers two ARM-based machine series in GKE:

This tutorial uses t2a-standard-2 because it's widely available. The commands are identical for c4a-standard-2 — just swap the --machine-type value. If t2a-standard-2 isn't available in your zone, GKE will tell you immediately when you run the node pool creation command below, and you can try a neighbouring zone.

Create the ARM Node Pool

Add the ARM node pool to your existing cluster:

gcloud container node-pools create axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a \
  --machine-type=t2a-standard-2 \
  --num-nodes=2 \
  --node-labels=workload-type=arm-optimized

What each flag does:

• --cluster=axion-tutorial-cluster — the name of the cluster we created in Step 2. Node pools are always added to an existing cluster.

• --zone=us-central1-a — must match the zone you used when creating the cluster.

• --machine-type=t2a-standard-2 — GKE detects this is an ARM machine type and automatically provisions the nodes with an ARM-compatible version of Container-Optimized OS (COS). You don't need to configure anything special for ARM at the OS level.

• --num-nodes=2 — two ARM nodes in the zone, enough to schedule our 3-replica deployment alongside other cluster overhead.

• --node-labels=workload-type=arm-optimized — attaches a custom label to every node in this pool. We'll use this label in our deployment manifest to target these specific nodes. Using a descriptive custom label (rather than just relying on the automatic kubernetes.io/arch=arm64 label) is good practice in real clusters — it communicates the intent of the pool, not just its hardware.

This command takes a few minutes. Once it completes, let's confirm our cluster now has both node pools:

gcloud container clusters get-credentials axion-tutorial-cluster --zone=us-central1-a

kubectl get nodes --label-columns=kubernetes.io/arch

The get-credentials command configures kubectl to authenticate with your new cluster. The get nodes command then lists all nodes and adds a column showing the kubernetes.io/arch label.

You should see something like:

NAME                                    STATUS   ARCH    AGE
gke-...default-pool-abc...              Ready    amd64   15m
gke-...default-pool-def...              Ready    amd64   15m
gke-...axion-pool-jkl...                Ready    arm64   3m
gke-...axion-pool-mno...                Ready    arm64   3m

amd64 for the default x86 pool, arm64 for our new Axion pool. This kubernetes.io/arch label is applied automatically by GKE — you don't set it, it's derived from the hardware.

Step 8: Deploy the App to the ARM Node Pool

We have a multi-arch image and a mixed-architecture cluster. Here's something important to understand before writing the deployment manifest: Kubernetes doesn't know or care about image architecture by default.

If you applied a standard Deployment right now, the scheduler would look for any available node with enough CPU and memory and place pods there — potentially landing on x86 nodes instead of your ARM Axion nodes. The multi-arch Manifest List would handle this gracefully (the right binary would run regardless), but you'd lose the cost efficiency you provisioned Axion nodes for in the first place.

To guarantee that pods land on ARM nodes and only ARM nodes, we use a nodeSelector.

How nodeSelector Works

A nodeSelector is a set of key-value pairs in your pod spec. Before the Kubernetes scheduler places a pod, it checks every available node's labels. If a node doesn't have all the labels in the nodeSelector, the scheduler skips it — the pod will remain in Pending state rather than land on the wrong node.

This is a hard constraint, which is exactly what we want for cost optimization. Contrast this with Node Affinity's soft preference mode (preferredDuringSchedulingIgnoredDuringExecution), which says "try to use ARM, but fall back to x86 if needed." Soft preferences are useful for resilience, but they undermine the whole point of dedicated ARM pools. We want the hard constraint.

Write the Deployment Manifest

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-axion
  labels:
    app: hello-axion
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-axion
  template:
    metadata:
      labels:
        app: hello-axion
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64

      containers:
      - name: hello-axion
        image: us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
        resources:
          requests:
            cpu: "250m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

Replace PROJECT_ID with your project ID. Here's what the key sections do:

replicas: 3 — tells Kubernetes to keep three instances of this pod running at all times. If one crashes or a node goes down, the scheduler spins up a replacement. Three replicas also means one pod per ARM node in us-central1, which distributes load across availability zones.

selector.matchLabels and template.metadata.labels — these two blocks must match. The selector tells the Deployment which pods it "owns," and the template.metadata.labels is what those pods will be tagged with. If they don't match, Kubernetes won't be able to manage the pods.

nodeSelector: kubernetes.io/arch: arm64 — this is the pin. The Kubernetes scheduler filters out every node that doesn't carry this label before considering resource availability. Since GKE automatically applies kubernetes.io/arch=arm64 to all ARM nodes, our pods will schedule only onto the axion-pool nodes.

livenessProbe — periodically calls GET /healthz. If this check fails a certain number of times in a row (indicating the container has deadlocked or is otherwise unresponsive), Kubernetes restarts the container. initialDelaySeconds: 5 gives the server 5 seconds to start up before the first check.

readinessProbe — similar to the liveness probe, but with a different purpose. While the readiness probe is failing, Kubernetes removes the pod from the service's load balancer, so no traffic is sent to it. This is important during startup — the pod won't receive traffic until it signals it's ready.

resources.requests — reserves 250m (25% of a CPU core) and 64Mi of memory on the node for this pod. The scheduler uses these numbers to decide whether a node has enough room for the pod. Setting requests is required for sensible bin-packing. Without them, nodes can be silently overcommitted.

resources.limits — caps the container at 500m CPU and 128Mi memory. If the container exceeds these limits, Kubernetes throttles the CPU or kills the container (for memory). This prevents a single misbehaving pod from starving other workloads on the same node.

A Note on Taints and Tolerations

Once you're comfortable with nodeSelector, the next step in production clusters is adding a taint to your ARM node pool. A taint is a repellent — any pod without an explicit toleration for that taint is blocked from landing on the tainted node.

This means other workloads in your cluster can't accidentally consume your ARM capacity. You'd add the taint when creating the pool:

# Add --node-taints to the pool creation command:
--node-taints=workload-type=arm-optimized:NoSchedule

And a matching toleration in the pod spec:

tolerations:
- key: "workload-type"
  operator: "Equal"
  value: "arm-optimized"
  effect: "NoSchedule"

We're not doing this in the tutorial to keep things simple, but it's the pattern production multi-tenant clusters use to enforce hard separation between workload types.

Write the Service Manifest

We also need a Kubernetes Service to expose the pods over the network. Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-axion-svc
spec:
  selector:
    app: hello-axion
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

• selector: app: hello-axion — the Service discovers pods using labels. Any pod with app: hello-axion on it will be added to this Service's load balancer pool.

• port: 80 — the port the Service is reachable on from outside the cluster.

• targetPort: 8080 — the port on the pod that traffic gets forwarded to. Our Go server listens on port 8080, so this must match.

• type: LoadBalancer — tells GKE to provision an external Google Cloud load balancer and assign it a public IP. This is what makes the Service reachable from the internet.

Apply Both Manifests

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl apply reads each manifest file and creates or updates the resources described in it. If the resources don't exist yet, they're created. If they already exist, Kubernetes only applies the diff — it won't restart pods unnecessarily.

Watch the pods come up in real time:

kubectl get pods -w

The -w flag watches for changes and prints updates as they happen. You should see pods transition from Pending → ContainerCreating → Running. Once all three show Running, press Ctrl+C to stop watching.

Step 9: Verify the Deployment

Everything is running. Now we need evidence — not just that pods are up, but that they're on the right nodes and serving the right binary.

Confirm Pod Placement

kubectl get pods -o wide

The -o wide flag adds extra columns to the output, including the name of the node each pod was scheduled on. Look at the NODE column:

NAME                          READY   STATUS    NODE
hello-axion-7b8d9f-abc12      1/1     Running   gke-axion-tutorial-axion-pool-a-...
hello-axion-7b8d9f-def34      1/1     Running   gke-axion-tutorial-axion-pool-b-...
hello-axion-7b8d9f-ghi56      1/1     Running   gke-axion-tutorial-axion-pool-c-...

All three pods should show node names containing axion-pool. None should show default-pool.

Confirm the Nodes Are ARM

Take one of those node names and verify its architecture label:

kubectl get node NODE_NAME --show-labels | grep kubernetes.io/arch

Replace NODE_NAME with one of the node names from the previous command. You should see:

kubernetes.io/arch=arm64

That's the automatic label GKE applied when it provisioned the ARM hardware. Our nodeSelector matched on this label to pin the pods here.

Ask the Application Itself

This is the most satisfying verification step. Our Go server reports the architecture of the binary that's running. Let's ask it directly.

Use kubectl port-forward to create a secure tunnel from port 8080 on your local machine to port 8080 on the Deployment:

kubectl port-forward deployment/hello-axion 8080:8080

This command stays running in the foreground — open a second terminal window and run:

curl http://localhost:8080

You should see:

Hello from freeCodeCamp!
Architecture : arm64
OS           : linux
Pod hostname : hello-axion-7b8d9f-abc12

Architecture : arm64. That's our Go binary confirming that it was compiled for ARM64 and is executing on an ARM64 CPU. The single image tag we built does the right thing automatically.

The Bonus: See the Manifest List in Action

Want to see the multi-arch image indexing at work? Stop the port-forward, then run:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Replace PROJECT_ID with your actual Google Cloud project ID.

You'll see four entries in the manifest list. Two are real images — Platform: linux/amd64 and Platform: linux/arm64. The other two show Platform: unknown/unknown with an attestation-manifest annotation. These are build provenance records that Docker Buildx automatically attaches to every image — a supply chain security feature (SLSA attestation) that proves how and where the image was built.

You may notice that if you check the image digest recorded in a running pod:

kubectl get pod POD_NAME \
  -o jsonpath='{.status.containerStatuses[0].imageID}'

Replace POD_NAME with one of the pod names from earlier.

The digest returned matches the top-level manifest list digest, not the arm64-specific one. This is expected behaviour. Modern Kubernetes (using containerd) records the manifest list digest, not the resolved platform digest. The platform resolution already happened when the node pulled the correct image variant.

The definitive proof that the right binary is running is what you already have: the node labeled kubernetes.io/arch=arm64 and the application reporting Architecture: arm64.

Step 10: Cost Savings and Tradeoffs

The hands-on work is done. Let's talk about why any of this is worth the effort.

The Cost Math

At the time of writing, here's how ARM compares to equivalent x86 machines on Google Cloud (prices are approximate and change over time — check the official pricing page before making decisions):

That's a raw 25–30% reduction in compute cost per node. Factor in Google's published claim of up to 65% better price-performance for Axion on relevant workloads — meaning you may need fewer nodes to handle the same traffic — and the savings compound further.

Here's how that looks at scale, for a service running 20 nodes continuously for a year:

• 20 × n2-standard-4 × \(0.19 × 8,760 hours = \)33,288/year

• 20 × t2a-standard-4 × \(0.14 × 8,760 hours = \)24,528/year

That's roughly $8,760 saved annually on compute, before committed use discounts (which further widen the gap).

When ARM Is the Right Choice

ARM works best for:

• Stateless API servers and web applications — like the app we built. ARM excels at high-throughput, low-latency network workloads.

• Background workers and queue processors — long-running services that don't depend on x86-specific binaries.

• Microservices written in Go, Rust, or Python — these languages have excellent ARM64 support and are built cross-platform by default.

When to Proceed Carefully

• Native library dependencies — some older C libraries, proprietary SDKs, or compiled ML model-serving runtimes don't have ARM64 builds. Always audit your dependency tree before migrating.

• CI pipelines need ARM too — your automated tests should run on ARM, not just x86. An image that silently fails only on ARM is harder to debug than one that never claimed ARM support.

• Profile before optimizing — the cost savings are real, but measure your actual workload behavior on ARM before committing. Not every workload benefits equally.

Cleanup

When you're done, clean up to avoid ongoing charges:

# Remove the Kubernetes resources from the cluster
kubectl delete -f k8s/

# Delete the ARM node pool
gcloud container node-pools delete axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the cluster itself
gcloud container clusters delete axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the images from Artifact Registry (optional — storage costs are minimal)
gcloud artifacts docker images delete \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Conclusion

Let's recap what you built and why each part matters.

You started with a Go application, a Dockerfile, and a docker buildx build command that produced two images — one for x86, one for ARM64 — wrapped in a single Manifest List tag. Any server that pulls that tag gets the right binary automatically, without you maintaining separate pipelines or separate tags.

You provisioned a GKE cluster with two node pools running different CPU architectures, then used nodeSelector to make sure your ARM-optimized workload lands only on the ARM Axion nodes — not on x86 by accident. The result is a deployment that's both architecture-correct and cost-efficient.

The patterns you practiced here don't stop at this demo. The same Dockerfile technique works for any language with cross-compilation support. The same nodeSelector approach works for any workload you want to pin to ARM. As more teams migrate services to ARM over the coming years, having these skills will be a real asset.

Where to go from here:

• Add a GitHub Actions workflow that runs docker buildx build --platform linux/amd64,linux/arm64 on every push, automating this entire process in CI.

• Audit one of your existing stateless services for ARM compatibility and try migrating it.

• Explore Node Affinity as a softer alternative to nodeSelector for workloads that can run on either architecture but prefer ARM.

• Look into GKE Autopilot, which now supports ARM nodes and handles node pool management automatically.

Happy building.

Project File Structure

hello-axion/
├── app/
│   ├── main.go          — Go HTTP server
│   ├── go.mod           — Go module definition
│   └── Dockerfile       — Multi-stage Dockerfile
└── k8s/
    ├── deployment.yaml  — Deployment with nodeSelector and probes
    └── service.yaml     — LoadBalancer Service

All source files for this tutorial are available in the companion GitHub repository: https://github.com/Amiynarh/multi-arch-docker-gke-arm

And while many SaaS tools can monitor your site, they often charge high monthly fees for simple alerts.

My goal with this article is to help you stop paying those expensive fees by showing you a powerful, free, open-source alternative called Uptime Kuma.

In this guide, you'll learn how to use Docker to deploy Uptime Kuma safely on a local Ubuntu machine.

By the end of this tutorial, you'll have set up your own private server monitoring dashboard in less than 10 minutes and created an automated Discord alert to ping your phone if your website goes offline.

Table of Contents

• Prerequisites

• Step 1: Update Packages and Prepare the Firewall

• Step 2: Create the Docker Compose File

• Step 3: Start the Application

• Step 4: Access the Dashboard

• Step 5: Use Case – Monitor a Website and Send Discord Alerts

• Conclusion

Prerequisites

Before you start, make sure you have:

• An Ubuntu machine (like a local server, VM, or desktop).

• Docker and Docker Compose installed.

• Basic knowledge of the Linux terminal.

Step 1: Update Packages and Prepare the Firewall

First, you'll want to make sure your system has the newest updates. Then, you'll install the Uncomplicated Firewall (UFW) and open the network "door" (port) that Uptime Kuma uses for the dashboard. You'll also need to allow SSH so you don't lock yourself out.

Run these commands in your terminal:

1. Update your packages:

sudo apt update && sudo apt upgrade -y

1. Install the firewall:

sudo apt install ufw -y

1. Allow SSH and open port 3001:

sudo ufw allow ssh
sudo ufw allow 3001/tcp

1. Enable the firewall:

sudo ufw enable
sudo ufw reload

Step 2: Create the Docker Compose File

Using a docker-compose.yml file is the professional way to manage Docker containers. It keeps your setup organised in one single place.

To start, create a new folder for your project and enter it:

mkdir uptime-kuma && cd uptime-kuma

Then create the configuration file:

nano docker-compose.yml

Paste the following code into the editor:

services:
  uptime-kuma:
    image: louislam/uptime-kuma:2
    restart: unless-stopped
    volumes:
      - ./data:/app/data
    ports:
      - "3001:3001"

Note: The ./data:/app/data line is very important. It saves your database in a normal folder on your machine, making it easy to back up later.

Finally, save and exit: Press CTRL + X, then Y, then Enter.

Step 3: Start the Application

Now, tell Docker to read your file and start the monitoring service in the background.

docker compose up -d

How to verify: Docker will download the files. When it finishes, your terminal should print Started uptime-kuma.

Step 4: Access the Dashboard

To access the dashboard, first open your web browser and go to http://localhost:3001 (or your machine's local IP address).

When asked to choose the database, select SQLite. It's simple, fast, and requires no extra setup.

Then create an account and choose a secure admin username and password.

Step 5: Use Case – Monitor a Website and Send Discord Alerts

Now you'll put Uptime Kuma to work by monitoring a live website and setting up an alert. Just follow these steps:

1. Click Add New Monitor.

2. Set the Monitor Type to HTTP(s).

3. Give it a Friendly Name (e.g., "My Blog") and enter your website's URL.

Pro-Tip: How to Fix "Down" Errors (Bot Protection)

If your site uses strict security, it might block Uptime Kuma and say your site is "Down" with a 403 Forbidden error.

The Fix: Scroll down to Advanced, find the User Agent box, and paste this text to make Uptime Kuma look like a normal Chrome browser:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Add a Discord Alert

To get a message on your phone when your site goes down:

1. On the right side of the monitor screen, click Setup Notification.

2. Select Discord from the dropdown list.

3. Paste a Discord Webhook URL (you can create one in your Discord server settings under Integrations).

4. Click Test to receive a test ping, then click Save.

Conclusion

Congratulations! You just took control of your server health. By deploying Uptime Kuma, you replaced an expensive SaaS subscription with a powerful, free monitoring tool that alerts you the second a project goes offline.

Let’s connect! I am a developer and technical writer specialising in writing step-by-step guides and workflows. You can find my latest projects on my Technical Writing Portfolio or reach out to me directly on LinkedIn.