Here's why: static credentials don't expire on their own. If they get leaked through a misconfigured workflow, a public fork, or a compromised repository, an attacker has persistent access to your AWS environment until you manually rotate them. And most teams don't rotate them often enough.

OpenID Connect (OIDC) solves this entirely. Instead of storing long-lived credentials, GitHub Actions requests a short-lived token directly from AWS every time your workflow runs. No secrets to rotate. No credentials to leak. No manual key management.

In this tutorial, you'll learn how to set up OIDC authentication between GitHub Actions and AWS from scratch. By the end, your workflows will authenticate to AWS securely without storing a single access key.

Table of Contents

• What Is OpenID Connect (OIDC)?

• How OIDC Works Between GitHub Actions and AWS

• Prerequisites

• Step 1: Create an IAM OIDC Identity Provider in AWS

  Step 2: Create an IAM Role with a Trust Policy

  Step 3: Attach Permissions to the IAM Role

  Step 4: Store the Role ARN as a GitHub Actions Variable

  Step 5: Configure Your GitHub Actions Workflow

  Step 6: Run and Verify Your Workflow

• Security Best Practices

• Troubleshooting Common Errors

• Conclusion

• References

What Is OpenID Connect (OIDC)?

OpenID Connect is an identity protocol built on top of OAuth 2.0. It allows systems to verify identity through tokens rather than shared secrets.

In the context of GitHub Actions and AWS:

• GitHub acts as the identity provider (IdP). It issues a signed JWT (JSON Web Token) for each workflow run.

• AWS acts as the service provider. It validates that token against GitHub's public keys and exchanges it for temporary AWS credentials. The credentials AWS returns are short-lived (valid for up to 1 hour by default) and scoped to exactly the IAM role you define. When the workflow ends, those credentials are gone.

This model is called federated identity. It's the same concept used when you "Sign in with Google" on a third-party website. The difference is that instead of a user signing in, your workflow is the one authenticating.

How OIDC Works Between GitHub Actions and AWS

Before writing a single line of YAML, it beneficial to understand the flow. This is my personal approach when implementing new technologies or concepts. Here's what happens every time your workflow runs:

The diagram illustrates a secure authentication flow between GitHub Actions and AWS using OpenID Connect (OIDC), eliminating the need to store long-lived AWS credentials in GitHub. Here's what happens step-by-step:

1. Initial Authentication Request

When your GitHub Actions workflow starts, the runner (the virtual machine executing your workflow) requests a JSON Web Token (JWT) from GitHub's OIDC provider located at https://token.actions.githubusercontent.com.

2. Token Issuance

GitHub's OIDC provider generates and signs a JWT containing important claims (metadata) about your workflow. These claims include details like which repository the workflow is running from, which branch triggered it, what environment it's running in, and other contextual information that proves the workflow's identity.

3. Token Validation

The GitHub Actions runner presents this signed JWT to AWS Security Token Service (STS). AWS STS validates the JWT's signature by checking it against GitHub's publicly available cryptographic keys, ensuring the token is authentic and hasn't been tampered with.

4. Trust Policy Verification

AWS STS checks the trust policy configured on your IAM Role. This trust policy specifies which GitHub repositories, branches, or environments are allowed to assume this role. If the claims in the JWT match your trust policy conditions, authentication succeeds.

5. Temporary Credentials Issued

Once validated, AWS STS returns temporary security credentials to the GitHub Actions runner. These credentials include an Access Key ID, Secret Access Key, and Session Token that are valid for a limited time (typically 1 hour by default, configurable up to 12 hours).

6. AWS API Access

The GitHub Actions runner uses these temporary credentials to authenticate API calls to your AWS resources such as pushing Docker images to ECR, updating ECS services, writing to S3 buckets, or invoking Lambda functions.

The key point: AWS never sees your GitHub credentials, and GitHub never sees your AWS credentials. The JWT is the only thing exchanged and it's signed, scoped, and short-lived.

Prerequisites

Before you start, make sure you have the following in place:

• An AWS account with IAM permissions to create identity providers and roles

• A GitHub repository (public or private) where your workflows will run

• Basic familiarity with GitHub Actions, knowing how to write a .yml workflow file

• Basic familiarity with AWS IAM roles, policies, and permissions

• The AWS CLI installed and configured (optional, but useful for verification). You don't need to be an AWS expert. Each step includes the exact console path and the configuration values you need.

Step 1: Create an IAM OIDC Identity Provider in AWS

The first thing you need to do is tell AWS to trust GitHub as an identity provider. This is a one-time setup per AWS account.

How to Do It in the AWS Console

1. Open the AWS IAM Console

2. In the left sidebar, click Identity providers

3. Click Add provider

4. For Provider type, select OpenID Connect

5. For Provider URL, enter:

https://token.actions.githubusercontent.com

6. For Audience, enter:

sts.amazonaws.com

7. Click Add provider

How to Do It with the AWS CLI

If you prefer the terminal, run this command:

aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \

Once created, you'll see token.actions.githubusercontent.com listed under Identity providers in your IAM console. This provider will be referenced in your IAM role's trust policy in the next step.

Step 2: Create an IAM Role with a Trust Policy

Now you need an IAM role that your GitHub Actions workflow will assume. The trust policy on this role controls which repositories and branches are allowed to request credentials.

How to Create the IAM Role in the AWS Console

1. Open the AWS IAM Console

2. In the left sidebar, click Roles

3. Click Create role

4. For Trusted entity type, select Web identity

5. For Identity Provider, choose: token.actions.githubusercontent.com which you created earlier.

6. For Audience, choose sts.amazonaws.com as well

7. For GitHub organisation, enter your GitHub username or organization name

8. For GitHub repository, enter your GitHub repository

9. For GitHub branch, enter your branch name (for example, main)

10. Click Next, then Next, give a name to the role and click create role

Note: Creating the IAM role using this approach already establishes the Trusted Entities using a trusted policy based on the step 4-9 above. You can verify this by clicking on the created role and navigating to Trust relationships.

How to Create the IAM Role with the AWS CLI

First, you'll need to create a trust policy document on your local machine: You can call it trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:YOUR_GITHUB_ORG/YOUR_REPO_NAME:*"
        }
      }
    }
  ]
}

Replace the following placeholders before saving:

How to Understand the sub Condition

The sub (subject) claim in the JWT tells AWS exactly where the request is coming from. The value repo:your-org/your-repo:* means any branch in that repository can assume this role.

You can tighten this further depending on your needs:

# Only the main branch
"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
 
# Only a specific GitHub Environment
"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:environment:production"

Scoping this correctly is one of the most important security decisions in this setup. Here's how to decide:

• Use ref:refs/heads/main if only your main/production branch should deploy to AWS. This is the most restrictive and secure option: feature branches can't accidentally (or maliciously) trigger deployments or modify production resources.

• Use environment:production if you're using GitHub Environments with protection rules (required reviewers, deployment gates). This lets you control deployments through GitHub's approval workflow while still restricting which workflows can access AWS.

• Use repo:your-org/your-repo:* (wildcard) only if you need any branch to deploy. for example, in development environments where every feature branch deploys to its own isolated stack. Never use this for production roles.

Run this command to create the role using your trust policy:

aws iam create-role \
  --role-name GitHubActionsOIDCRole \
  --assume-role-policy-document file://trust-policy.json \
  --description "Role assumed by GitHub Actions via OIDC"

Take note of the Role ARN in the output. It will look like this:

arn:aws:iam::YOUR_ACCOUNT_ID:role/GitHubActionsOIDCRole

You'll need this ARN in your workflow YAML in Step 4.

Step 3: Attach Permissions to the IAM Role

The IAM role can now authenticate, but it has no permissions yet. You need to attach a policy that defines what your workflow is actually allowed to do in AWS.

How to Apply the Principle of Least Privilege

Only grant the permissions your workflow genuinely needs. If your workflow deploys to S3, give it S3 permissions. If it pushes images to ECR, give it ECR permissions. Never attach AdministratorAccess to a CI/CD role.

Option 1: Attach an AWS managed policy (quick start):

aws iam attach-role-policy \
  --role-name GitHubActionsOIDCRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

Option 2: Create a custom policy scoped to a specific S3 bucket (recommended for production):

This approach is recommended for production because it limits the blast radius of a security incident. If your workflow credentials are ever compromised, a custom policy scoped to a specific bucket means an attacker can only affect that single bucket not every S3 bucket in your AWS account. It also prevents accidental misconfigurations in your workflow from impacting unrelated resources.

Create a file called s3-deploy-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

Then create and attach it:

aws iam create-policy \
  --policy-name GitHubActionsS3DeployPolicy \
  --policy-document file://s3-deploy-policy.json
 
aws iam attach-role-policy \
  --role-name GitHubActionsOIDCRole \
  --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/GitHubActionsS3DeployPolicy

Note: You can as well implement Step 3 via the console.

Reference: For a full list of available AWS IAM actions, see the AWS IAM actions reference.

Step 4: Store the Role ARN as a GitHub Actions Variable

Before you configure your workflow, you need to make the Role ARN available to it. You'll store it as a repository variable in GitHub, not a secret, because the ARN itself isn't sensitive data.

How to Add the Variable in Your Repository

First, open your GitHub repository and click Settings:

In the left sidebar, scroll down to Secrets and variables, then click Actions:

Then click the Variables tab (not Secrets). Click New repository variable – you can set the Name to:

AWS_ROLE_ARN

Set the Value to your Role ARN from Step 2, for example:

arn:aws:iam::YOUR_ACCOUNT_ID::role/GitHubActionsOIDCRole

Click Add variable:

You'll reference this variable in your workflow in the next step using ${{ vars.AWS_ROLE_ARN }}.

Step 5: Configure Your GitHub Actions Workflow

With AWS and GitHub fully configured, you now need to update your workflow to request an OIDC token and use it to authenticate.

How to Set the Required Workflow Permissions

Your workflow must declare id-token: write. Without this, GitHub won't issue an OIDC token to the runner.

permissions:
  id-token: write   # Required to request the OIDC JWT
  contents: read    # Required to checkout the repository

Important: If you set permissions at the job level, they override any top-level permissions. Make sure id-token: write is present at whichever level your AWS authentication step runs.

Full Workflow Example

Here's a complete workflow that authenticates to AWS using OIDC and deploys a static site to S3:

name: Deploy to AWS S3
 
on:
  push:
    branches:
      - main
 
permissions:
  id-token: write
  contents: read
 
jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
 
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
 
      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          aws-region: us-east-2
 
      - name: Verify AWS identity
        run: aws sts get-caller-identity
 
      - name: Deploy to S3
        run: |
          aws s3 sync ./code s3://your-bucket-name

Replace the following before committing:

You can see the code sample in my GitHub Repo here.

Note: The aws-actions/configure-aws-credentials action handles the entire OIDC token exchange automatically. It requests the JWT from GitHub, calls sts:AssumeRoleWithWebIdentity, and exports the temporary credentials as environment variables for the rest of the job.

See the action's official documentation for all available options.

Step 6: Run and Verify Your Workflow

Push your workflow to the main branch and open the Actions tab in your repository to watch it run.

What a Successful Run Looks Like

The Configure AWS credentials via OIDC step should show:

Assuming role with OIDC: arn:aws:iam::YOUR_ACCOUNT_ID:role/GitHubActionsOIDCRole

The Verify AWS identity step (aws sts get-caller-identity) should return:

{
    "UserId": "AROA...:GitHubActions",
    "Account": "YOUR_ACCOUNT_ID",
    "Arn": "arn:aws:sts::YOUR_ACCOUNT_ID:assumed-role/GitHubActionsOIDCRole/GitHubActions"
}

If you see an assumed-role ARN in the output, OIDC is working correctly. Your workflow is now authenticating to AWS without a single stored credential.

Security Best Practices

Getting OIDC working is step one. Locking it down properly is step two.

Scope the sub Condition as Tightly as Possible

Don't use a wildcard like repo:your-org/*:* that allows any repository in your organization to assume the role. Scope it to the exact repository and branch that needs access.

"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"

Use GitHub Environments for Production Deployments

GitHub Environments let you add manual approval gates and restrict which branches can deploy. When combined with OIDC, you can scope your trust policy to only allow the production environment:

"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:environment:production"

Apply Least-Privilege Permissions to Every IAM Role

Never attach AdministratorAccess or PowerUserAccess to a role used by CI/CD. Define a custom policy with only the actions your workflow actually needs.

Create Separate IAM Roles Per Environment

A staging role and a production role should have different permission scopes. Your staging deployment role should never have write access to production resources.

Enable AWS CloudTrail

Every call made using the temporary credentials is logged in CloudTrail under the assumed role ARN. This gives you a full audit trail of exactly what your workflow did in AWS.

Reference: GitHub's official security hardening guide for OIDC: About security hardening with OpenID Connect

Troubleshooting Common Errors

Error: Not authorized to perform sts:AssumeRoleWithWebIdentity

This usually means the trust policy on your IAM role doesn't match the sub claim in the JWT.

Check the following:

• The sub condition exactly matches your repository path (it is case-sensitive)

• The aud condition is set to sts.amazonaws.com

• The Federated principal uses the correct AWS account ID

To inspect the actual token claims your workflow is receiving, add this debug step temporarily:

- name: Print OIDC token claims
  run: |
    TOKEN=\((curl -s -H "Authorization: Bearer \)ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
      "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r '.value')
    echo $TOKEN | cut -d '.' -f2 | base64 -d 2>/dev/null | jq .

Error: Could not load credentials from any providers

This almost always means id-token: write is missing from your workflow permissions. Double-check that you have:

permissions:
  id-token: write
  contents: read

Error: AccessDenied When Calling an AWS Service

Authentication succeeded but the IAM role doesn't have permission to perform the action your workflow is attempting. Check the permissions policy attached to your role and compare it against the specific action in the error message.

Conclusion

You've gone from storing static, long-lived AWS credentials in GitHub Secrets to a fully keyless authentication setup using OIDC. Here's what you accomplished:

• Registered GitHub as a trusted OIDC identity provider in AWS.

• Created an IAM role with a scoped trust policy tied to a specific repository.

• Attached least-privilege permissions to that role.

• Configured your GitHub Actions workflow to request and use short-lived AWS credentials.

• Verified the authentication flow end-to-end.

This pattern works across every AWS service from S3, ECS, Lambda, ECR, Secrets Manager, and more. The workflow example here uses S3, but you only need to swap out the permissions policy and the deployment commands to adapt it for any service.

If you want to go further, explore:

• Configuring OIDC for multiple cloud providers: Azure, GCP, and HashiCorp Vault.

• GitHub Environments and deployment protection rules: for multi-stage pipelines with approval gates.

• AWS IAM Access Analyzer: to validate and tighten your role policies automatically.

If you're building out your DevOps practice and want a complete, production-ready reference for infrastructure automation, CI/CD, and platform engineering, check out The Startup DevOps Field Guide. It covers the patterns, templates, and runbooks I've used across real AWS environments.

You can also connect with me on LinkedIn

References

• GitHub Docs: About security hardening with OpenID Connect

• GitHub Docs: Configuring OpenID Connect in Amazon Web Services

• AWS Docs: Creating OpenID Connect (OIDC) identity providers

• AWS Docs: AssumeRoleWithWebIdentity API Reference

• aws-actions/configure-aws-credentials - GitHub

• AWS IAM Actions Reference

• AWS CloudTrail User Guide

You’ll learn how to expose services on a custom domain with auto-renewing HTTPS, and implement a smart deployment strategy that detects changes and redeploys only the affected microservices. This helps avoid unnecessary full-stack redeploys. We'll also cover real production issues and the exact fixes for each one.

Table of Contents

• 1. What you'll build

• 2. Architecture

• 3. Server prerequisites

• 4. Traefik — the reverse proxy

• 5. Run Jenkins in Docker

• 6. Expose Jenkins on a domain via Traefik

• 7. First-time Jenkins setup

• 8. Add the GitHub credential

• 9. Create the pipeline job

• 10. The Jenkinsfile (deploy only what changed)

• 11. End-to-end test

• 12. Troubleshooting — every error we hit

• 13. Mental model: host vs. container

• 14. Daily operations cheat sheet

• 15. What I'd do differently next time

• Closing thoughts

1. What You'll Build

In this tutorial, you'll build a Jenkins instance running inside Docker on the same Linux server as your application stack.

Traefik will act as a reverse proxy in front of Jenkins, exposing it via a clean URL (https://jenkins.example.com) with auto-renewing Let's Encrypt certificates.

You'll also create a Jenkinsfile in your application repository that:

• Automatically triggers on every push to the staging branch,

• Detects which microservices changed in each commit,

• Pulls the latest code on the host machine,

• Rebuilds and restarts only the affected services.

On every push, only the relevant services are redeployed.

Prerequisites

Before jumping in, this guide assumes you’re already comfortable with a few core concepts and tools.

This isn't a beginner-level tutorial — we’ll be working directly with infrastructure, containers, and CI/CD pipelines.

You should be familiar with:

• Basic Linux commands (SSH, file system navigation, permissions)

• Docker fundamentals (images, containers, volumes, networks)

• Git workflows (clone, pull, branches)

• General idea of CI/CD pipelines

Tools and environment required:

• A Linux server (Ubuntu recommended)

• Docker Engine + Docker Compose (v2)

• A domain name (for Traefik + HTTPS)

• GitHub repository (for your backend project)

• Basic understanding of microservices architecture

If you’re comfortable with the above, you’re ready to follow along.

2. Architecture

Here's an overview of the architecture:

┌──────────────────────────── Linux server (Ubuntu) ────────────────────────────┐
│                                                                               │
│   /home/developer/projects/                                                  │
│       └── project-prod-configs/             ← infra repo (compose, Traefik) │
│              ├── docker-compose.staging.yml                                   │
│              ├── traefik.staging.yml                                          │
│              └── project-backend/          ← app repo (services, gateways) │
│                     ├── Jenkinsfile                                           │
│                     ├── docker-compose.staging.yml                            │
│                     └── apps/                                                 │
│                            ├── services/<name>/                               │
│                            ├── gateways/<name>/                               │
│                            └── core/<name>/                                   │
│                                                                               │
│   ┌─────────────────────── Docker network: proxy ──────────────────────┐      │
│   │  traefik (80, 443)                                                 │      │
│   │     │                                                              │      │
│   │     ├──► jenkins  (projects-jenkins-staging)                     │      │
│   │     │      ↳ /projects  ← bind-mount of the host project tree     │      │
│   │     │      ↳ /var/run/docker.sock ← controls host Docker           │      │
│   │     │                                                              │      │
│   │     └──► your services & gateways (built by the pipeline)          │      │
│   └────────────────────────────────────────────────────────────────────┘      │
│                                                                               │
└───────────────────────────────────────────────────────────────────────────────┘
            ▲
            │  webhook on push
            │
   GitHub: <org>/project-backend (branch: staging)

There are two key ideas here:

1. Jenkins runs in a container, but it controls the host's Docker by mounting /var/run/docker.sock. It also bind-mounts the project folder as /projects/..., so it can cd into the real code on the host and run docker compose there.

2. The Jenkinsfile lives inside the app repo, so the pipeline definition is versioned with the code. Jenkins simply points at it.

3. Server Prerequisites

Before we start configuring Jenkins or Traefik, we need to prepare the server properly.

In this step, we’ll:

• Create a dedicated Linux user for managing the project

• Install Docker and Docker Compose

• Set up the folder structure for our repositories

This ensures our CI/CD pipeline runs in a clean and predictable environment.

# Linux user that owns the project tree
sudo adduser developer

# Docker engine + Compose plugin
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker developer

# Sanity check Compose v2
docker compose version
# -> Docker Compose version v2.x.y

# Find where the Compose plugin binary lives — write it down, you'll need it
ls /usr/libexec/docker/cli-plugins/docker-compose
# (some distros use /usr/lib/docker/cli-plugins/docker-compose)

# Project layout
sudo mkdir -p /home/developer/project
sudo chown -R developer:developer /home/developer/project

# Clone both repos in the right place
cd /home/developer/projects
git clone https://github.com/<org>/projects-prod-configs.git
cd projects-prod-configs
git clone -b staging https://github.com/<org>/projects-backend.git

You should now have:

/home/developer/projects/projects-prod-configs/projects-backend

Memorize this path — your Jenkinsfile references it.

DNS

Point an A-record for your Jenkins subdomain to the server's public IP before the next steps so Let's Encrypt can validate via HTTP challenge:

jenkins.example.com   A   <server-public-ip>

4. Traefik — the Reverse Proxy

Traefik acts as the entry point to your entire system. Instead of exposing each service manually with ports, Traefik automatically:

• Routes traffic based on domain names

• Generates and renews HTTPS certificates using Let’s Encrypt

• Connects to Docker and detects services dynamically

In simple terms, Traefik lets you access services like:

https://jenkins.example.com
https://api.example.com

…without manually configuring NGINX or managing SSL certificates.

In this setup, Traefik watches Docker containers and routes traffic using labels we'll define later.

Traefik gives every container a real domain and a real cert with zero per-service config — you just add a few labels.

traefik.staging.yml (static config)

Put this at the root of your infra repo:

api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: web
      email: admin@example.com           # ← change me
      storage: /etc/traefik/acme.json

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false              # only containers with traefik.enable=true
    network: proxy
  file:
    directory: /etc/traefik/dynamic
    watch: true

log:
  level: INFO

accessLog: {}

The Traefik service in docker-compose.staging.yml

networks:
  proxy:
    name: proxy
    driver: bridge
  internal:
    name: internal
    driver: bridge

volumes:
  acme-data:
  traefik-logs:
  jenkins-data:

services:
  traefik:
    image: traefik:v2.11
    container_name: projects-traefik-staging
    restart: unless-stopped
    ports:
      - "80:80"        # HTTP (auto-redirects to HTTPS)
      - "443:443"      # HTTPS
      - "8080:8080"    # Traefik dashboard (internal only — protect via firewall)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.staging.yml:/etc/traefik/traefik.yml:ro
      - ./dynamic:/etc/traefik/dynamic:ro
      - acme-data:/etc/traefik           # persists Let's Encrypt certs
      - traefik-logs:/var/log/traefik
    networks:
      - proxy
    command:
      - '--api.insecure=false'
      - '--api.dashboard=true'
      - '--providers.docker=true'
      - '--providers.docker.exposedbydefault=false'
      - '--providers.docker.network=proxy'
      - '--entrypoints.web.address=:80'
      - '--entrypoints.websecure.address=:443'
      - '--entrypoints.web.http.redirections.entryPoint.to=websecure'
      - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge=true'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web'
      - '--certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:-admin@example.com}'
      - '--certificatesresolvers.letsencrypt.acme.storage=/etc/traefik/acme.json'
      - '--log.level=INFO'
      - '--accesslog=true'
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      # Traefik's own dashboard
      - "traefik.http.routers.traefik-dash.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.traefik-dash.entrypoints=websecure"
      - "traefik.http.routers.traefik-dash.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik-dash.service=api@internal"

Bring it up:

cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d traefik

Watch the logs the first time — Traefik will request a cert for the dashboard host as soon as DNS resolves.

docker logs -f projects-traefik-staging

Tip. While testing, switch ACME to staging endpoint (acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory) so you don't burn through Let's Encrypt's rate limits if you misconfigure DNS. Remove that flag before going live.

5. Run Jenkins in Docker

Add this Jenkins service to the same docker-compose.staging.yml. Every line matters (and the comments explain why).

  jenkins:
    image: jenkins/jenkins:lts
    container_name: projects-jenkins-staging
    restart: unless-stopped
    user: root                           # to use host docker.sock without UID juggling
    environment:
      - JAVA_OPTS=-Xmx1g -Xms512m -Duser.timezone=Asia/Dhaka
      - TZ=Asia/Dhaka                    # OS-level timezone inside container
      - JENKINS_OPTS=--prefix=/
    ports:
      - "3095:8080"                      # web UI (also reachable directly if needed)
      - "50000:50000"                    # inbound agent port
    volumes:
      - jenkins-data:/var/jenkins_home   # Jenkins config/jobs/secrets persistence
      - /var/run/docker.sock:/var/run/docker.sock                          # control host Docker
      - /usr/bin/docker:/usr/bin/docker                                     # docker CLI from host
      - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro  # docker compose plugin
      - /home/developer/projects:/projects                                # project tree
      - /etc/localtime:/etc/localtime:ro                                    # match host clock
      - /etc/timezone:/etc/timezone:ro
    networks:
      - proxy
      - internal
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:8080/login']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s
    deploy:
      resources:
        limits:
          memory: 1024M

Why user: root? It's the simplest way to share docker.sock and the project bind-mount without UID/GID gymnastics. If you prefer an unprivileged user, you'll need to set group: docker and align UIDs/perms on host folders — possible but out of scope here.

6. Expose Jenkins on a Domain via Traefik

This is the section many guides skip. We'll add labels to the Jenkins service so Traefik picks it up automatically. No editing of Traefik config required.

  jenkins:
    # ... everything above ...
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # 1) Router — match incoming Host
      - "traefik.http.routers.jenkins.rule=Host(`jenkins.example.com`)"
      - "traefik.http.routers.jenkins.entrypoints=websecure"
      - "traefik.http.routers.jenkins.tls.certresolver=letsencrypt"
      - "traefik.http.routers.jenkins.service=jenkins"

      # 2) Service — tell Traefik which container port is the app
      - "traefik.http.services.jenkins.loadbalancer.server.port=8080"

      # 3) Middleware — Jenkins needs X-Forwarded-Proto so it knows it's behind HTTPS
      - "traefik.http.middlewares.jenkins-headers.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.jenkins.middlewares=jenkins-headers"

What each line does:

Bring Jenkins up:

cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d jenkins

# Watch Traefik issue the certificate
docker logs -f projects-traefik-staging | grep -i acme

After 10–60 seconds you should be able to open https://jenkins.example.com and see Jenkins's setup wizard with a valid lock icon.

Inside Jenkins (after first login):

Manage Jenkins → System → Jenkins URL → set this to: https://jenkins.example.com/

This is important because Jenkins uses this base URL to generate:

• Webhook endpoints (for GitHub triggers)

• Links inside emails and build logs

If this isn't set correctly, GitHub webhooks may fail, and any links Jenkins generates will point to the wrong address (often localhost or internal IPs).

7. First-Time Jenkins Setup

If you're running Jenkins for the first time on this server, follow this section to complete the initial setup.

If you already have Jenkins configured, you can skip this section — but make sure the required plugins and settings match what we use later in this guide.

1. Open https://jenkins.example.com. Get the initial admin password:

   docker exec projects-jenkins-staging cat /var/jenkins_home/secrets/initialAdminPassword
   

2. Paste it, choose Install suggested plugins.

3. Create your admin user.

4. Manage Jenkins → Plugins → Available and install:

   • GitHub (and GitHub Branch Source)

   • Pipeline: GitHub

   • Credentials Binding (usually preinstalled)

That's all the plugins you need for the rest of this guide.

8. Add the GitHub Credential

Jenkins needs permission to access your GitHub repository.

This is done using a GitHub Personal Access Token (PAT), which acts like a password for secure API and Git operations.

We’ll store this token inside Jenkins as a credential so it can pull code during pipeline execution and authenticate securely without exposing secrets in code.

This single credential is used both for the SCM checkout and for the deploy-time git pull.

1. Create a Personal Access Token (classic) on GitHub with repo scope.

2. In Jenkins: Manage Jenkins → Credentials → System → Global → Add Credentials.

3. Fill in:

   • Kind: Username with password

   • Username: your GitHub username

   • Password: the token

   • ID: github_classic_token (the Jenkinsfile references this exact ID)

9. Create the Pipeline Job

Now that Jenkins has access to your repository, the next step is to define how deployments should run.

A pipeline job tells Jenkins:

• where your code lives,

• which branch to monitor,

• and how to execute your deployment process.

In Jenkins, create a new Pipeline job and connect it to your GitHub repository. Once this is set up, Jenkins will automatically trigger deployments whenever you push to the staging branch.

Start by creating a new job:

New Item → Pipeline → name it projects-staging → OK

Then configure the job:

• Under Build Triggers, enable:
  GitHub hook trigger for GITScm polling

• Under Pipeline:

  • Definition: Pipeline script from SCM

  • SCM: Git

  • Repository URL: https://github.com/<org>/projects-backend.git

  • Credentials: github_classic_token

  • Branch: */staging

  • Script Path: Jenkinsfile

Save the configuration.

At this point, Jenkins is fully connected to your repository and ready to run your deployment pipeline automatically.

10. The Jenkinsfile (Deploy Only What Changed)

Place this at the root of the app repo (projects-backend/Jenkinsfile), branch staging.

pipeline {
  agent any

  environment {
    PROJECT_PATH = "/projects/projects-prod-configs/projects-backend"
    COMPOSE_FILE = "docker-compose.staging.yml"
  }

  stages {

    stage('Checkout') {
      steps {
        checkout scm
        echo "Checkout completed for branch: ${env.BRANCH_NAME ?: 'staging'}"
      }
    }

    stage('Detect Changes') {
      steps {
        script {
          def changedFiles = sh(
            script: "git diff --name-only HEAD~1 HEAD",
            returnStdout: true
          ).trim()

          echo "Changed files:\n${changedFiles}"

          def services = [] as Set
          changedFiles.split('\n').each { file ->
            def svc  = file =~ /^apps\/services\/([a-z0-9-]+)\//
            def gw   = file =~ /^apps\/gateways\/([a-z0-9-]+)\//
            def core = file =~ /^apps\/core\/([a-z0-9-]+)\//
            if (svc)  { services << svc[0][1]  }
            if (gw)   { services << gw[0][1]   }
            if (core) { services << core[0][1] }
          }
          services = services.findAll { !it.endsWith('-e2e') }
          env.CHANGED_SERVICES = services.join(' ')

          echo "Services to deploy: ${env.CHANGED_SERVICES ?: '(none)'}"
        }
      }
    }

    stage('Deploy') {
      when { expression { return env.CHANGED_SERVICES?.trim() } }
      steps {
        withCredentials([usernamePassword(
          credentialsId: 'github_classic_token',
          usernameVariable: 'GIT_USER',
          passwordVariable: 'GIT_TOKEN'
        )]) {
          sh '''
            set -eu
            git config --global --add safe.directory "${PROJECT_PATH}"
            cd "${PROJECT_PATH}"
            git remote set-url origin "https://github.com/<org>/projects-backend.git"
            git -c credential.helper= \
                -c "credential.helper=!f() { echo username=\({GIT_USER}; echo password=\){GIT_TOKEN}; }; f" \
                pull origin staging
            docker compose -f "\({COMPOSE_FILE}" up -d --build \){CHANGED_SERVICES}
          '''
        }
        echo "Deployed: ${env.CHANGED_SERVICES}"
      }
    }

    stage('Skip Deployment') {
      when { expression { return !env.CHANGED_SERVICES?.trim() } }
      steps { echo "No service changes detected — nothing to deploy." }
    }
  }
}

Why each tricky line is there:

• git config --global --add safe.directory ... — git refuses to operate on a repo whose owner UID differs from the current user's. The repo on disk is owned by developer, but Git inside the container runs as root. This whitelists the path.

• git remote set-url origin "https://..." — flips the on-disk remote to HTTPS so the token can be used. (A PAT can't authenticate git@github.com: URLs — those use SSH.) Idempotent — safe to re-run.

• git -c credential.helper="!f() { echo username=...; echo password=...; }; f" — feeds the username/token to git for that one command without writing the token to disk and without exposing it on the process command line.

• ${CHANGED_SERVICES} is unquoted on purpose so multiple service names expand as separate args.

11. End-to-End Test

Before considering the setup complete, we need to verify that the entire pipeline works as expected.

This end-to-end test ensures that:

• GitHub webhooks are triggering Jenkins correctly,

• Jenkins can detect which services changed,

• and only the affected services are rebuilt and deployed.

In other words, this simulates a real production deployment.

Start by making a small change in your repository. For example, modify a file inside:

apps/gateways/student-apigw/

Then push the change to the staging branch.

Once pushed, Jenkins should automatically trigger via the webhook. If not, you can manually click Build Now.

Now open the build’s Console Output and verify the flow. You should see something like:

• Checkout completed for branch: staging

• Services to deploy: student-apigw

• git pull origin staging (successful)

• docker compose ... up -d --build student-apigw

• Deployed: student-apigw

If you see this sequence, your pipeline is working correctly.

If anything fails, don’t worry — jump to Section 12 where every common issue and its fix is documented.

12. Troubleshooting — Every Error We Hit

This section covers real issues we faced while setting up this pipeline — and more importantly, why each fix works. Understanding the “why” will help you debug similar problems in your own setup.

cd: can't cd to /projects/projects-prod-configs/projects-backend

Cause:
The Jenkinsfile runs cd $PROJECT_PATH, but inside the container that path doesn’t exist. This usually happens when:

• the project wasn’t cloned on the host, or

• the bind mount isn’t configured correctly.

Fix:

ls /home/developer/projects/projects-prod-configs/projects-backend
# If missing: git clone -b staging <url> there.

Confirm the bind mount:

docker inspect projects-jenkins-staging --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'

If missing, recreate the container:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:

Jenkins runs inside a container, but your code lives on the host. The bind mount connects them. Without it, Jenkins cannot access your project directory.

fatal: detected dubious ownership in repository

Cause:
Git blocks access when the repository owner differs from the current user.

• Repo owner: developer (host)

• Git runs as: root (inside container)

Fix:

git config --global --add safe.directory "${PROJECT_PATH}"

Why this works:

This explicitly tells Git that the directory is trusted, bypassing ownership mismatch security restrictions.

Host key verification failed / Could not read from remote repository

Cause:

The repository uses SSH (git@github.com:...), but:

• the container has no SSH keys

• no known_hosts file exists

Also, GitHub tokens cannot authenticate over SSH.

Fix (recommended):

git remote set-url origin "https://github.com/<org>/projects-backend.git"

Why this works:

HTTPS uses token-based authentication (PAT), which works inside containers without SSH configuration.

unknown shorthand flag: 'f' in -f ( docker compose)

Cause:
The Docker CLI exists, but the Docker Compose plugin is missing inside the container.

Fix:

volumes:
  - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro

Find your path if needed:

find /usr -name docker-compose -type f 2>/dev/null

Verify:

docker exec projects-jenkins-staging docker compose version

Why this works:

Docker Compose v2 is a CLI plugin. Mounting this directory makes the docker compose command available inside the container.

Wrong timezone in build timestamps and Jenkins UI

Fix: Set both env var and JVM flag, and bind-mount the host's clock files:

environment:
  - TZ=Asia/Dhaka
  - JAVA_OPTS=... -Duser.timezone=Asia/Dhaka
volumes:
  - /etc/localtime:/etc/localtime:ro
  - /etc/timezone:/etc/timezone:ro

You must recreate the container for env-var changes to take effect:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:
Jenkins runs on Java, which uses its own timezone separate from the OS.
By aligning OS timezone, JVM timezone, and host clock, you ensure consistent timestamps everywhere.

ERR_SOCKET_TIMEOUT (pnpm install fails)

Cause:

If you have multiple services building in parallel and each runs pnpm install with ~1500 packages, the network gets saturated and a timeout occurs.

Fixes:

a) Increase timeout + control concurrency

RUN pnpm install --frozen-lockfile --ignore-scripts 
--network-timeout 600000 
--network-concurrency 8

Why: Gives pnpm more time and reduces network overload.

b) Enable pnpm cache (BuildKit)

RUN --mount=type=cache,id=pnpm-store,target=/root/.local/share/pnpm/store 
pnpm install --frozen-lockfile --ignore-scripts

Why: Dependencies are cached and reused instead of downloading every time.

c) Avoid unnecessary rebuilds

docker compose -f \(COMPOSE_FILE build \)CHANGED_SERVICES docker compose -f \(COMPOSE_FILE up -d --no-build \)CHANGED_SERVICES

Why: Only changed services are rebuilt → less network load → fewer failures.

Container changes don’t apply after editing docker-compose.yml

Cause:

Docker compose up -d does not update running containers.

Fix:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:

This forces Docker to recreate the container with updated configuration (env, volumes, labels).

Traefik shows default certificate (no HTTPS)

Common causes:

DNS not pointing to server Port 80 blocked Wrong Docker network

Check:

dig +short jenkins.example.com docker logs projects-traefik-staging 2>&1 | grep -i acme

Why this works:

Let’s Encrypt uses HTTP-01 challenge, so it must reach your server via port 80. If DNS or networking is wrong, certificate issuance fails.

Jenkins: "Reverse proxy setup is broken"

Fix:

Set the Jenkins URL to https://jenkins.example.com/
Ensure header:

X-Forwarded-Proto: https

Why this works:

Jenkins needs to know it's behind HTTPS. Without this, it generates incorrect URLs (http instead of https), breaking redirects and webhooks.

13. Mental Model: Host vs. Container

Many setup mistakes come from confusing the host filesystem with the container filesystem. This table makes it explicit:

When debugging, always ask: "Inside which filesystem is this command running, and does the file/folder it's looking for exist there?"

14. Daily Operations Cheat Sheet

# Recreate Jenkins after changing compose
cd /home/developer/Projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

# Tail Jenkins logs
docker logs -f projects-jenkins-staging

# Open a shell inside the Jenkins container
docker exec -it projects-jenkins-staging bash

# From inside the container — sanity checks
docker compose version
ls /projects/projects-prod-configs/projects-backend
git -C /projects/projects-prod-configs/projects-backend remote -v

# Manually trigger the same deploy the pipeline does
cd /projects/projects-configs/projects-backend
git pull origin staging
docker compose -f docker-compose.staging.yml up -d --build student-apigw

# Inspect Traefik routing decisions
docker logs projects-traefik-staging 2>&1 | grep -i jenkins

# Check renewed certs
docker exec projects-traefik-staging cat /etc/traefik/acme.json | head -50

15. What I'd Do Differently Next Time

• Pre-build a base image with all node_modules baked in. With ~1500 packages × 15 services, every clean build re-downloads ~22k tarballs. A shared base cuts that 90%.

• Run a private npm proxy (Verdaccio / Nexus / GitHub Packages) on the same Docker network — eliminates flaky npmjs.org timeouts entirely.

• Per-service Jenkinsfile if your services drift apart in tooling. With one Jenkinsfile, every team contends for the same pipeline definition.

• Replace git diff HEAD~1 HEAD with git diff $(git merge-base HEAD origin/staging~1) HEAD so squash-merges and force-pushes don't accidentally skip services.

• Move secrets to a vault (HashiCorp Vault / AWS Secrets Manager / Doppler). PATs in Jenkins work, but rotation across many jobs is painful.

• Use Jenkins' Configuration-as-Code (JCasC) so the entire Jenkins setup (jobs, credentials definitions, plugins) is in git. Then a server rebuild is a one-command operation.

Closing Thoughts

The pipeline itself is just three stages: Checkout → Detect Changes → Deploy — but a real production setup is mostly about plumbing: reverse proxy, certificates, bind-mounts, credentials, timezones, build caches. None of these are exotic. Together they decide whether your Friday-afternoon deploy goes silently green or eats your weekend.

Follow sections 1–11 to get a working pipeline. Bookmark section 12 to keep it working.

Happy shipping.

This is a common experience when contributing to open source: you make a small change, open a pull request, and suddenly everything fails.

Not just one check, but multiple:

• Lint errors

• YAML validation issues

• Build failures

• Deployment failures

Even more confusing, you may see errors in parts of the codebase you didn’t modify.

In this article, you'll learn how to debug these issues step by step. The goal is not just to fix one pull request, but to understand how CI systems validate your changes.

This guide is based on a real debugging experience from contributing to an open source documentation project.

While this example comes from a documentation project, the debugging workflow applies to many repositories that use CI pipelines, linting tools, and automated builds.

Table of Contents:

• Understanding the CI Pipeline (What’s Actually Happening)

• How a CI Pipeline Processes Your Pull Request

• A Practical Debugging Workflow

  • Step 1: Fix Authentication and Permission Issues

  • Step 2: Run Lint Checks Locally

  • Step 3: Fix Common Markdown Lint Errors

  • Step 4: Fix YAML Inside Markdown Code Blocks

  • Step 5: Fix Build Errors After Lint Passes

  • Step 6: Debug Cascading CI Failures

  • Step 7: Handle Git Issues During CI Debugging

• Key Takeaways

• Conclusion

Prerequisites

To follow this guide, you should have:

• Basic familiarity with Git and pull requests

• A GitHub account

• Some exposure to CI/CD concepts (helpful but not required)

Understanding the CI Pipeline (What’s Actually Happening)

In many projects, you will see the term CI/CD, which stands for Continuous Integration and Continuous Deployment (or Delivery).

In this guide, we'll focus specifically on the CI part – that is, Continuous Integration. This refers to the automated checks that run when you push code or open a pull request. These checks validate your changes before they're merged into the main codebase.

CD (Continuous Deployment/Delivery), on the other hand, typically handles what happens after those checks pass, such as deploying the application.

Understanding this distinction is important because most of the issues we debug in this guide happen during the CI stage.

Most repositories run multiple automated checks when you open a pull request:

• Linting tools (for example, markdownlint, yamllint) enforce formatting rules

• Build systems (for example, mdBook) validate structure and generate output

• Deployment checks (for example, Netlify) ensure that the site can be built and served

• Merge controllers (for example, Tide) enforce approval policies

A key point to remember: CI systems validate the entire set of files in your commit, not just the lines you changed.

How a CI Pipeline Processes Your Pull Request

When you push code or open a pull request, the CI pipeline runs several checks in sequence.

Let’s visualize how these checks are connected in a typical CI pipeline.

Figure: A simplified CI pipeline showing how linting, build, and deployment checks are executed sequentially.

The above diagram shows a sequential CI pipeline with feedback loops, where failures at any stage return you to fix the issue before continuing.

Let’s break down what this diagram shows:

1. You start by pushing code or opening a pull request.

2. The CI pipeline begins running automated checks.

3. The first set of checks typically includes linting tools like markdownlint or yamllint.

   • If linting fails, the pipeline stops, and you must fix formatting issues before continuing.

   • If linting passes, the pipeline moves to the build step (for example, mdBook in documentation projects).

   • If the build fails, it usually means there is a structural issue, such as duplicate entries or invalid references.

4. After a successful build, deployment checks (such as Netlify previews) run.

   • If deployment fails, the issue is often related to configuration or build output.

5. If all steps pass, the pull request becomes ready for review.

A Practical Debugging Workflow

Step 1: Fix Authentication and Permission Issues

Before CI runs, your push can fail due to authentication errors.

Example error:

refusing to allow a Personal Access Token to create or update workflow

This happens because GitHub requires special permissions when your commit includes files under:

.github/workflows/

The solution is to regenerate your Personal Access Token (PAT) with:

• repo access

• workflow permission

Step 2: Run Lint Checks Locally

Relying only on CI feedback slows you down because you have to push changes and wait for the pipeline to run before seeing errors.

Running checks locally allows you to catch issues immediately before pushing your code.

In practice, you should do both:

• Run checks locally to catch errors early and reduce iteration time

• Use CI as the final validation to ensure your changes meet the repository’s standards

Think of local checks as your first line of defense, and CI as the final gate before your code is accepted.

Here's an example (Markdown linting):

npm install -g markdownlint-cli2
markdownlint-cli2 docs/**/*.md

Step 3: Fix Common Markdown Lint Errors

Here are some common issues you may encounter:

1. Non-descriptive links

Non-descriptive links like "here" don't give readers any context about where the link leads. This makes documentation harder to understand and less accessible, especially for users relying on screen readers.

Instead of writing:

[here](https://example.com)

Use descriptive text like:

[command help documentation](https://example.com)

2. Line length violations

Many projects enforce a maximum line length (often around 80 characters) to improve readability across different devices and editors.

If a line is too long, you can split it into multiple lines without changing the meaning.

To do this, break the line at natural points such as spaces between words or after punctuation. Avoid breaking words or disrupting the sentence structure.
For example:

This is a long sentence that should be split across multiple
lines to satisfy lint rules.

3. List indentation issues

List indentation errors occur when nested list items aren't aligned consistently. This can break formatting and cause linting errors.

To avoid this, just make sure you use consistent spacing (usually 2 spaces per level).

Example (incorrect):

- Item 1
 - Subitem

Correct version:

- Item 1
  - Subitem

Step 4: Fix YAML Inside Markdown Code Blocks

YAML has strict formatting rules, including proper indentation, key-value structure, and consistent spacing.

Even when YAML appears inside a markdown code block, tools like yamllint still validate its structure.

Example (incorrect):

metadata:
annotations:

Correct version:

metadata:
  annotations:
    capi.metal3.io/unhealthy: "true"

In the incorrect example, annotations is not properly nested under metadata, and no key-value pair is defined.

In the corrected version:

• annotations is properly indented under metadata

• a valid key-value pair is added (capi.metal3.io/unhealthy: "true")

This structure satisfies YAML’s requirement for proper hierarchy and formatting.

Step 5: Fix Build Errors After Lint Passes

Passing lint checks doesn't guarantee that your build will succeed.

This is because linting focuses on syntax and formatting, while the build process validates the structure and integrity of the entire project.

Build failures often occur due to issues such as:

• Duplicate entries in navigation files

• Missing or incorrectly referenced files

• Invalid configuration settings

Even if your syntax is correct, the build system ensures everything connects properly.

For example, in documentation projects using tools like mdBook, a duplicate entry in SUMMARY.md can cause the build to fail even when all files pass lint checks.

Step 6: Debug Cascading CI Failures

CI pipelines are layered. One failure can trigger multiple downstream failures.

For example, imagine a YAML indentation error:

YAML error → build fails → deploy fails → multiple checks fail

To fix this:

1. Identify the first failing step in the CI logs

2. Fix that issue

3. Re-run the pipeline

In this example, the YAML indentation error is the root cause. Once you fix the YAML formatting, the lint check passes, which allows the build to proceed and the deployment step to succeed.

This is why it is important to always fix the first failure in the pipeline rather than trying to address all errors at once.

Step 7: Handle Git Issues During CI Debugging

When working with updated branches, you may encounter:

• Diverged branches

• Rebase conflicts

• Push rejections

To resolve these issues, you typically need to update your branch using one of two approaches:

Option 1: Rebase (clean history)

git pull --rebase

Rebasing rewrites your commit history so your changes appear on top of the latest version of the branch.

Use carefully:

• Only rebase your own branches

• Avoid rebasing shared branches

Option 2: Merge (safer)

git pull --no-rebase

Merging preserves the full commit history and is safer when working with others, but it may introduce additional merge commits.

Pushing your changes safely

After updating your branch, you may need to push changes:

git push --force-with-lease

Avoid using:

git push --force

The --force option can overwrite the other contributors’ work. The --force-with-lease option is safer because it only pushes if the remote branch has not changed unexpectedly.

Key Takeaways

• CI validates your entire commit, not just the specific lines you changed

• Linting and build systems enforce different rules

• YAML inside markdown must be structurally correct

• Documentation builds can fail due to structural issues

• Running checks locally significantly reduces debugging time

Conclusion

Debugging a failing pull request isn't just about fixing syntax errors.

You also need to understand how different systems interact:

• Version control

• CI pipelines

• Linting tools

• Build processes

Once you understand how these systems work together, you can debug issues systematically instead of guessing.

The next time your pull request fails, you will know exactly where to start and how to fix it.

Debugging CI issues may feel overwhelming at first, but with a structured approach, you can turn failures into a clear path for improvement.

And somewhere in that chain, something often goes wrong: an incorrect API key, a missed signing step, a build that worked on one machine and failed on another.

The solution is a properly configured CI/CD pipeline that takes that entire chain out of human hands. And in this article, we're building exactly that using Codemagic.

What is Codemagic?

Codemagic is a dedicated CI/CD platform built from the ground up specifically for mobile applications.

Unlike general-purpose CI platforms, Codemagic understands Flutter natively. It ships with Flutter pre-installed on its build machines, has dedicated support for Apple code signing, and integrates directly with both the Google Play Store and App Store Connect. This means less configuration noise and more focus on what actually matters , which is your deployment logic.

The pipeline we'll be building covers three distinct stages across both Android and iOS:

• A pull request gate that blocks unverified code from reaching your base branch

• A staging pipeline that injects real environment config, builds signed artifacts, and ships them to testers via Firebase App Distribution and TestFlight

• A production pipeline that obfuscates builds, uploads crash symbols to Sentry, and submits directly to the Play Store and App Store Connect

Table of Contents

• Prerequisites

• Understanding Codemagic's YAML Approach

• Pipeline Architecture

• The Helper Scripts

• PR Quality Gate

• Android Pipeline

• iOS Pipeline

• Environment Variables and Secrets Reference

• End-to-End Flow

• Conclusion

Prerequisites

You'll need the following before starting:

• A Flutter app with functional Android and iOS builds

• A Codemagic account with your repository connected

• A Firebase project with App Distribution set up

• A Sentry project configured for your app

• A Google Play Console app with at least an internal track ready

• An Apple Developer account with App Store Connect access

• A Google Play service account with the necessary API permissions

• Familiarity with writing Bash scripts

Understanding Codemagic's YAML Approach

Codemagic offers a visual workflow editor for teams that prefer a GUI – but we're not using that here. The codemagic.yaml approach gives you version-controlled, reviewable, fully reproducible pipeline definitions that live right alongside your application code. Any change to the pipeline goes through the same PR process as any other change. That matters in a team environment.

The file lives at the root of your project:

your-flutter-app/
  codemagic.yaml     
  lib/
  android/
  ios/
  scripts/

Codemagic detects this file when a build is triggered and executes the appropriate workflow based on the rules you define. One file, multiple workflows, all environments – no duplication.

Pipeline Architecture

Before writing any YAML, it helps to define exactly what the pipeline needs to do. The use case here is a team with three protected branches: develop, staging, and production. Each branch represents a distinct stage in the release lifecycle, and the pipeline behaves differently depending on which branch triggered it.

Here is how the three environments map to pipeline behaviour:

PR into develop: When a developer raises a pull request targeting the develop branch, a quality gate workflow fires. It runs code formatting checks, static analysis, the full test suite, and enforces a minimum coverage threshold. The PR cannot be considered clean until all of these pass.

Push to develop or staging: When code lands on either of these branches, the platform-specific build pipelines trigger. They detect the target branch, inject the correct environment configuration (dev or staging API keys), build signed artifacts, and distribute them to the appropriate testing channels: Firebase App Distribution for Android, TestFlight for iOS.

Push to production: When code reaches the production branch, the pipelines switch into release mode. Builds are obfuscated, debug symbols are uploaded to Sentry for crash observability, and the final artifacts are submitted directly to the Play Store and App Store Connect.

Your project structure will look like this:

codemagic.yaml

scripts/
  generate_config.sh
  quality_checks.sh
  upload_symbols.sh

lib/
  core/
    env/
      env_ci.dart       
      env_ci.g.dart     

The Helper Scripts

Rather than cramming logic directly into YAML, this pipeline delegates its core operations to three Bash scripts that live in a scripts/ folder at the project root. This keeps the YAML readable and, crucially, means you can run the exact same logic on your local machine that CI runs – eliminating an entire class of "works on my machine" issues.

Make all three scripts executable before committing them:

chmod +x scripts/generate_config.sh
chmod +x scripts/quality_checks.sh
chmod +x scripts/upload_symbols.sh

generate_config.sh

Injecting secrets safely is one of the hardest CI/CD problems in mobile development. The strategy here avoids committing credentials entirely: a Dart file with placeholder values is committed to source control, and at build time the script replaces those placeholders with real values sourced from Codemagic's encrypted secret storage.

#!/usr/bin/env bash
set -euo pipefail

# Usage: ./scripts/generate_config.sh ENV_NAME BASE_URL ENCRYPTION_KEY
ENV_NAME=${1:-}
BASE_URL=${2:-}
ENCRYPTION_KEY=${3:-}

TEMPLATE="lib/core/env/env_ci.dart"
OUT="lib/core/env/env_ci.g.dart"

if [ -z "\(ENV_NAME" ] || [ -z "\)BASE_URL" ] || [ -z "$ENCRYPTION_KEY" ]; then
  echo "Usage: $0 <env-name> <base-url> <encryption-key>"
  exit 2
fi

sed -e "s|<<BASE_URL>>|$BASE_URL|g" \
    -e "s|<<ENCRYPTION_KEY>>|$ENCRYPTION_KEY|g" \
    -e "s|<<ENV_NAME>>|$ENV_NAME|g" \
    "\(TEMPLATE" > "\)OUT"

echo "✅ Generated config for $ENV_NAME"

How it works:

set -euo pipefail enforces strict failure behaviour. -e exits immediately on any failed command, -u exits on undefined variables, and -o pipefail catches failures anywhere in a pipeline – not just the last command. In CI, silent failures can produce broken builds that look like they succeeded. This line prevents that.

The script takes three positional arguments: the environment name (dev, staging, or production), the API base URL, and an encryption or API key. The ${1:-} syntax defaults to an empty string if an argument is missing, which the validation block then catches explicitly with a clear usage message and an exit code of 2 (the conventional code for incorrect usage).

At the heart of the script, sed performs three placeholder replacements in a single pass over the template file, writing the result to env_ci.g.dart. That generated file must be added to .gitignore. It only ever exists inside a running build or on a developer's local machine after they run the script manually.

The two Dart files involved have completely different roles:

env_ci.dart – committed to source control, contains only placeholders:

// lib/core/env/env_ci.dart
class EnvConfig {
  static const String baseUrl = '<<BASE_URL>>';
  static const String encryptionKey = '<<ENCRYPTION_KEY>>';
  static const String environment = '<<ENV_NAME>>';
}

env_ci.g.dart – generated at build time, contains real values, never committed:

// lib/core/env/env_ci.g.dart
// GENERATED FILE — DO NOT COMMIT
class EnvConfig {
  static const String baseUrl = 'https://staging.api.example.com';
  static const String encryptionKey = 'sk_live_xxxxx';
  static const String environment = 'staging';
}

Add the generated file to .gitignore:

# Generated environment config
lib/core/env/env_ci.g.dart

quality_checks.sh

This script defines what passing quality means for your codebase. Every check it runs is a gate: if any step fails, the script stops immediately and the build fails.

#!/usr/bin/env bash
set -euo pipefail

echo "🚀 Running quality checks"

dart format --output=none --set-exit-if-changed .
flutter analyze
flutter test --no-pub --coverage

if command -v dart_code_metrics >/dev/null 2>&1; then
  dart_code_metrics analyze lib --reporter=console || true
fi

echo "✅ Quality checks passed"

What each step does:

dart format --output=none --set-exit-if-changed .: checks that all Dart files are formatted correctly without modifying them. If any file doesn't match the formatter's output, the command exits with a non-zero code, failing the build. Formatting is non-negotiable here.

flutter analyze: runs Dart's static analyser across the entire project. It catches null safety violations, unused imports, missing awaits, dead code, and a wide range of structural issues before they reach a reviewer's eyes.

flutter test --no-pub --coverage: runs the full test suite and generates a coverage report at coverage/lcov.info. The --no-pub flag skips pub get since dependencies are already installed. The coverage file is used downstream to enforce a minimum threshold.

The dart_code_metrics block is deliberately optional and non-blocking (|| true). The tool may not be installed in every environment, and its findings are advisory rather than hard failures. You can remove the || true later to make it mandatory once your team has adopted the tool.

The final echo line only executes if every step above it passed , because set -e would have exited the script on any earlier failure. If you see it in the logs, the branch is clean.

upload_symbols.sh

When Flutter production builds are compiled with --obfuscate, stack traces in crash reports become unreadable. This script uploads the debug symbol files that Sentry needs to reverse that obfuscation and show readable crash reports.

#!/usr/bin/env bash
set -euo pipefail

RELEASE=${1:-}

[ -z "$RELEASE" ] && exit 2

if ! command -v sentry-cli >/dev/null 2>&1; then
  exit 0
fi

sentry-cli releases new "$RELEASE" || true
sentry-cli upload-dif build/symbols || true
sentry-cli releases finalize "$RELEASE" || true

echo "✅ Symbols uploaded for release $RELEASE"

How it works:

The script takes a single argument: a release identifier. In practice, this is always the short Git commit SHA, passed from the workflow as $(git rev-parse --short HEAD). This ties the uploaded symbols, the deployed build, and the crash reports in Sentry to the exact same commit , which is essential for production debugging.

If sentry-cli is not installed in the environment, the script exits with 0 rather than failing. This makes symbol uploads environment-aware: production machines install the CLI, development environments skip the step cleanly without breaking the build.

Each sentry-cli command uses || true for resilience. Symbol uploads should never block a deployment , if the upload encounters a transient issue, the build should still succeed and the symbols can be re-uploaded manually from the stored artifacts.

The three commands do the following in sequence: releases new registers the release version in Sentry, upload-dif sends the debug information files from build/symbols (generated by --split-debug-info), and releases finalize marks the release as deployed and ready to aggregate crash reports.

The codemagic.yaml Structure

A codemagic.yaml file is organized around workflows. Each workflow is an independent pipeline definition with its own trigger rules, environment configuration, build scripts, and publishing targets. Multiple workflows live inside the same file under a top-level workflows key.

The skeleton looks like this:

workflows:
  pr-quality-gate:
    # triggers on pull requests
    # runs quality checks only

  android-pipeline:
    # triggers on push to develop, staging, production
    # handles Android builds and distribution

  ios-pipeline:
    # triggers on push to develop, staging, production
    # handles iOS builds and distribution

Each workflow can define its own machine type, environment variables, triggering conditions, and step scripts. This is what makes a single codemagic.yaml powerful: you're not managing three separate files, but you still get complete isolation between pipeline stages.

PR Quality Gate

Every PR raised against develop must pass a quality gate before any merge is allowed. This workflow runs on Codemagic's Linux machines since it doesn't need to produce a signed artifact for any platform – it only needs to verify the code.

workflows:
  pr-quality-gate:
    name: PR Quality Gate
    max_build_duration: 30
    instance_type: linux_x2

    triggering:
      events:
        - pull_request
      branch_patterns:
        - pattern: develop
          include: true
          source: true

    environment:
      flutter: stable

    scripts:
      - name: Install dependencies
        script: flutter pub get

      - name: Run quality checks
        script: ./scripts/quality_checks.sh

      - name: Enforce coverage threshold
        script: |
          COVERAGE=\((lcov --summary coverage/lcov.info | grep lines | awk '{print \)2}' | sed 's/%//')
          if [ \((echo "\)COVERAGE < 70" | bc) -eq 1 ]; then
            echo "Test coverage is at ${COVERAGE}% — minimum required is 70%"
            exit 1
          fi
          echo "Coverage at ${COVERAGE}% — threshold met"

    publishing:
      email:
        recipients:
          - your-team@example.com
        notify:
          success: true
          failure: true

Let's walk through what each section is doing.

instance_type: linux_x2

Codemagic offers different machine types for different workloads. For a quality gate that only needs to run Dart tooling, a Linux machine is perfectly sufficient and significantly cheaper than a macOS instance. You reserve the macOS machines for builds that actually need Xcode.

triggering

This is how Codemagic decides when to run a workflow. The pull_request event fires whenever a PR is opened or updated. The branch_patterns block tells Codemagic to watch for PRs targeting develop specifically. The source: true flag means this pattern applies to the target branch of the PR, not the source branch – so any branch raising a PR into develop will trigger this workflow.

environment

Codemagic's Flutter-aware machines come with multiple Flutter versions available. Setting flutter: stable pins the workflow to the current stable channel without requiring any manual SDK installation step. This is one of the areas where Codemagic saves setup time compared to a general-purpose runner.

Quality checks script

The workflow delegates to quality_checks.sh rather than inlining commands. This keeps the YAML readable and ensures the exact same logic runs when a developer calls the script locally. The script handles formatting, analysis, and test execution internally.

Coverage enforcement

After the tests run, lcov parses the coverage report generated by flutter test --coverage and extracts the line coverage percentage. If it falls below 70%, the build fails with a clear message. This threshold is something your team should agree on , 70% is a reasonable starting point for most projects.

publishing

Codemagic has native email notification support built in. Rather than scripting echo statements into CI logs, you declare recipients directly in the workflow and Codemagic handles delivery. Both success and failure states are covered.

Android Pipeline

The Android workflow handles all three environments in a single workflow definition, using Codemagic's environment variable groups and conditional scripting to behave differently depending on which branch triggered the build.

  android-pipeline:
    name: Android Build & Release
    max_build_duration: 60
    instance_type: linux_x2

    triggering:
      events:
        - push
      branch_patterns:
        - pattern: develop
          include: true
        - pattern: staging
          include: true
        - pattern: production
          include: true

    environment:
      flutter: stable
      android_signing:
        - android_keystore
      groups:
        - staging_secrets
        - production_secrets
        - firebase_credentials
        - sentry_credentials

    scripts:
      - name: Install dependencies
        script: flutter pub get

      - name: Detect environment
        script: |
          BRANCH=$(git rev-parse --abbrev-ref HEAD)
          if [ "$BRANCH" = "develop" ]; then
            echo "ENV=dev" >> $CM_ENV
          elif [ "$BRANCH" = "staging" ]; then
            echo "ENV=staging" >> $CM_ENV
          else
            echo "ENV=production" >> $CM_ENV
          fi

      - name: Generate environment config
        script: |
          if [ "$ENV" = "dev" ]; then
            ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"
          elif [ "$ENV" = "staging" ]; then
            ./scripts/generate_config.sh staging "\(STAGING_BASE_URL" "\)STAGING_API_KEY"
          else
            ./scripts/generate_config.sh production "\(PROD_BASE_URL" "\)PROD_API_KEY"
          fi

      - name: Build Android artifact
        script: |
          if [ "$ENV" = "production" ]; then
            flutter build appbundle --release \
              --obfuscate \
              --split-debug-info=build/symbols
          else
            flutter build appbundle --release
          fi

      - name: Distribute to Firebase App Distribution
        script: |
          if [ "\(ENV" = "dev" ] || [ "\)ENV" = "staging" ]; then
            firebase appdistribution:distribute \
              build/app/outputs/bundle/release/app-release.aab \
              --app "$FIREBASE_ANDROID_APP_ID" \
              --groups "$FIREBASE_GROUPS" \
              --token "$FIREBASE_TOKEN"
          fi

      - name: Submit to Play Store
        script: |
          if [ "$ENV" = "production" ]; then
            echo "$GOOGLE_PLAY_SERVICE_ACCOUNT_JSON" > /tmp/service_account.json
            flutter pub global activate fastlane 2>/dev/null || true
            fastlane supply \
              --aab build/app/outputs/bundle/release/app-release.aab \
              --json_key /tmp/service_account.json \
              --package_name com.your.package \
              --track production
          fi

      - name: Upload Sentry symbols
        script: |
          if [ "$ENV" = "production" ]; then
            ./scripts/upload_symbols.sh $(git rev-parse --short HEAD)
          fi

    artifacts:
      - build/app/outputs/bundle/release/app-release.aab
      - build/symbols/**

    publishing:
      email:
        recipients:
          - your-team@example.com
        notify:
          success: true
          failure: true

Here is what each section is doing and why it's designed this way.

android_signing

This is one of Codemagic's most valuable features. Instead of manually decoding a Base64 keystore and writing it to disk inside a script, you upload your keystore file directly to Codemagic's encrypted key storage under Teams → Code signing identities → Android keystores. You give it a reference name – android_keystore in this case – and Codemagic handles decoding, placement, and key.properties generation automatically before your build scripts run.

This eliminates an entire category of signing-related build failures.

groups

Codemagic lets you organize secrets into named groups in the environment variables section of your team settings. Rather than declaring individual secrets inline, you reference groups. The groups used here are:

• staging_secrets: contains STAGING_BASE_URL and STAGING_API_KEY

• production_secrets: contains PROD_BASE_URL and PROD_API_KEY

• firebase_credentials: contains FIREBASE_TOKEN, FIREBASE_ANDROID_APP_ID, FIREBASE_GROUPS

• sentry_credentials: contains SENTRY_AUTH_TOKEN, SENTRY_ORG, SENTRY_PROJECT

Environment detection with $CM_ENV

Codemagic exposes a special file path via the $CM_ENV variable. Writing KEY=VALUE to this file makes that variable available to every subsequent script step in the same build. This is how the branch name gets translated into an environment label that the rest of the pipeline reads.

Build differentiation

Production builds use --obfuscate and --split-debug-info=build/symbols. Dev and staging builds skip both flags for faster compilation and readable local stack traces.

Firebase distribution

The Firebase CLI distributes dev and staging builds to testers. Because Codemagic's Linux machines come with Node.js available, you can install the Firebase CLI with npm install -g firebase-tools as a setup step if it is not already present, or invoke it via npx.

Play Store submission

Production app bundles go to the Play Store using Fastlane's supply command. The service account JSON is written to a temporary file from the environment variable and passed to Fastlane directly. Replace com.your.package with your actual application ID.

artifacts

The artifacts section tells Codemagic which files to preserve after the build completes. These files become downloadable from the Codemagic build dashboard. The debug symbols are captured here as well, which is useful for manual Sentry uploads if the automated step ever needs to be re-run.

iOS Pipeline

iOS on Codemagic is where the platform's advantage becomes most visible. Apple code signing on a general-purpose runner requires a multi-step keychain dance involving security commands, certificate imports, and provisioning profile placement. Codemagic handles all of that automatically through its native signing integration.

  ios-pipeline:
    name: iOS Build & Release
    max_build_duration: 90
    instance_type: mac_mini_m2

    triggering:
      events:
        - push
      branch_patterns:
        - pattern: develop
          include: true
        - pattern: staging
          include: true
        - pattern: production
          include: true

    environment:
      flutter: stable
      ios_signing:
        distribution_type: app_store
        bundle_identifier: com.your.bundle.id
      groups:
        - staging_secrets
        - production_secrets
        - app_store_credentials
        - sentry_credentials

    scripts:
      - name: Install dependencies
        script: flutter pub get

      - name: Install Fastlane dependencies
        script: |
          cd ios
          gem install bundler --user-install
          bundle install

      - name: Detect environment
        script: |
          BRANCH=$(git rev-parse --abbrev-ref HEAD)
          if [ "$BRANCH" = "develop" ]; then
            echo "ENV=dev" >> $CM_ENV
          elif [ "$BRANCH" = "staging" ]; then
            echo "ENV=staging" >> $CM_ENV
          else
            echo "ENV=production" >> $CM_ENV
          fi

      - name: Generate environment config
        script: |
          if [ "$ENV" = "dev" ]; then
            ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"
          elif [ "$ENV" = "staging" ]; then
            ./scripts/generate_config.sh staging "\(STAGING_BASE_URL" "\)STAGING_API_KEY"
          else
            ./scripts/generate_config.sh production "\(PROD_BASE_URL" "\)PROD_API_KEY"
          fi

      - name: Build iOS (dev — no signing)
        script: |
          if [ "$ENV" = "dev" ]; then
            flutter build ios --release --no-codesign
          fi

      - name: Build and ship to TestFlight (staging)
        script: |
          if [ "$ENV" = "staging" ]; then
            flutter build ipa --release \
              --export-options-plist=/Users/builder/export_options.plist
            cd ios && bundle exec fastlane beta
          fi

      - name: Build and release to App Store (production)
        script: |
          if [ "$ENV" = "production" ]; then
            flutter build ipa --release \
              --obfuscate \
              --split-debug-info=build/symbols \
              --export-options-plist=/Users/builder/export_options.plist
            cd ios && bundle exec fastlane release
          fi

      - name: Upload Sentry symbols
        script: |
          if [ "$ENV" = "production" ]; then
            ./scripts/upload_symbols.sh $(git rev-parse --short HEAD)
          fi

    artifacts:
      - build/ios/ipa/*.ipa
      - build/symbols/**
      - /tmp/xcodebuild_logs/*.log

    publishing:
      app_store_connect:
        api_key: $APP_STORE_CONNECT_PRIVATE_KEY
        key_id: $APP_STORE_CONNECT_KEY_IDENTIFIER
        issuer_id: $APP_STORE_CONNECT_ISSUER_ID
        submit_to_testflight: true
        submit_to_app_store: false
      email:
        recipients:
          - your-team@example.com
        notify:
          success: true
          failure: true

Here's what is different from the Android workflow and why.

mac_mini_m2

iOS builds require Xcode, which means they need macOS. Codemagic provides Apple Silicon Mac Mini instances. These are meaningfully faster than Intel-based runners for Flutter and Xcode workloads, and Codemagic provisions them on demand without any infrastructure management on your side.

ios_signing

This is the section that replaces the entire keychain setup sequence. You upload your distribution certificate and provisioning profile once to Codemagic's code signing identities under your team settings. The distribution_type: app_store tells Codemagic to use App Store distribution signing, and bundle_identifier ties it to your specific app. Before your scripts run, Codemagic installs the certificate and profile automatically on the build machine.

No security commands, no keychain creation, no Base64 decoding. It's handled internally.

flutter build ipa

On iOS, the build output is an .ipa file rather than an .aab. Flutter's flutter build ipa command produces this directly when provided with an export options plist. The plist tells Xcode how to sign and package the output. Codemagic generates this file automatically based on your ios_signing configuration and places it at /Users/builder/export_options.plist.

Fastlane lanes

Codemagic installs Fastlane via Bundler in the ios/ directory, then calls the appropriate lane based on the detected environment. The beta lane uploads to TestFlight, and the release lane submits to the App Store.

publishing.app_store_connect

Codemagic has a native App Store Connect publisher. Rather than scripting the upload manually, you declare your API credentials in the publishing block and Codemagic handles the submission. The submit_to_testflight: true flag means staging builds are automatically available to TestFlight testers after the build completes. For production, you would flip submit_to_app_store to true instead.

Xcode logs as artifacts

The line /tmp/xcodebuild_logs/*.log captures raw Xcode build logs as downloadable artifacts. When an iOS build fails and the error message in the Codemagic dashboard is not specific enough, these logs are where you find the real cause.

Environment Variables and Secrets Reference

All secrets are configured in Codemagic under Teams → Environment variables. Group them logically so they can be referenced cleanly in the YAML.

staging_secrets group

production_secrets group

firebase_credentials group

app_store_credentials group

sentry_credentials group

For Android code signing, upload your keystore directly under Teams → Code signing identities → Android keystores rather than storing it as an environment variable.

For iOS, upload your distribution certificate and provisioning profile under Teams → Code signing identities → iOS certificates.

End-to-End Flow

With the full codemagic.yaml in place, here is the complete picture of what happens across a typical release cycle.

A developer finishes a feature and raises a PR into develop. Codemagic detects the pull request event and triggers the pr-quality-gate workflow on a Linux machine. The quality checks script runs formatting, analysis, tests, and coverage threshold check. If anything fails, Codemagic marks the build as failed, sends the team an email, and the PR cannot be considered ready. The developer pushes a fix, Codemagic runs again, and only when everything passes does the PR move forward.

Once the PR merges into develop, both the android-pipeline and ios-pipeline trigger simultaneously. Each detects develop as the source branch, maps it to the dev environment, injects placeholder config, builds an unsigned release artifact, and ships it to Firebase App Distribution. Testers have an installable build within minutes of the merge completing.

When develop is merged into staging, the same two platform pipelines fire again. This time real secrets are injected , the staging API URL, the staging encryption key. Android builds are signed with the keystore Codemagic manages automatically. iOS builds go through Fastlane's beta lane to TestFlight. The Codemagic App Store Connect publisher handles the TestFlight upload natively. QA now has a properly signed, properly configured staging build to test against.

When staging is promoted to production, the pipelines enter release mode. Production secrets are injected. Android builds are obfuscated with debug symbols split into build/symbols. iOS builds go through flutter build ipa with obfuscation enabled. Both platform pipelines call upload_symbols.sh with the current commit SHA, linking the Sentry release to the exact code that shipped. The Android bundle goes to the Play Store via Fastlane. The iOS IPA is submitted to App Store Connect via Codemagic's native publisher. The team receives a success notification.

That's the full cycle. No terminal, no manual step, no shared Slack message saying "I think I deployed staging."

Conclusion

The pipeline we just built covers the full release lifecycle: automated quality enforcement, environment-aware config injection, platform-specific signed builds, tester distribution, crash observability, and store submission , all from a single codemagic.yaml file.

What Codemagic brings to this setup is a tighter integration with the mobile ecosystem specifically. The keystore management, native App Store Connect publisher, pre-installed Flutter toolchain, and Apple Silicon Mac instances aren't add-ons you configure , they're part of the platform's core. This translates into fewer steps to maintain, fewer failure surfaces, and a pipeline that's easier to reason about when something does go wrong.

The scripts in your scripts/ folder remain completely platform-agnostic. If your team ever needs to move pipelines, those scripts move with you unchanged. The YAML changes, but the logic doesn't.

What you have at the end of this setup is a release process your team can trust: one where "did it deploy?" is answered by a notification, not a question in Slack.

One of the major improvements has been a properly automated CI/CD pipeline flow that gives us seamless automation, continuous integration, and continuous deployment.

In this article, I'll break down how you can automate and build a production ready CI/CD pipeline for your Flutter application using GitHub Actions.

Note that there are other ways to do this, like with Codemagic (built specifically for Flutter apps – which I'll cover in a subsequent tutorial), but in this article we'll focus on GitHub Actions instead.

Table of Contents

1. The Typical Workflow

2. Prerequisites

3. Pipeline Architecture

4. Writing the Workflows

   • The Helper Scripts

     • generate_config.sh

     • quality_gate.sh

     • upload_symbols.sh

   • PR Quality Gate (pr_checks.yml)

   • Android CI/CD Pipeline (android.yml)

   • iOS CI/CD Pipeline (ios.yml)

5. Secrets and Configuration Reference

6. End-to-End Flow

7. Conclusion

The Typical Workflow

First, let's define the common approach to deploying production-ready Flutter apps.

The development team does their work on local, pushes to the repository for merge or review, and eventually runs flutter build apk or flutter build appbundle to generate the apk file. This then gets shared with the QA team manually, or deployed to Firebase app distribution for testing. If it's a production move, the app bundle is submitted to the Google Play store for review and then deployed.

This process is often fully manual with no automated checks, validation, or control over quality, speed, and seamlessness. Manually shipping a Flutter app starts out relatively simply, but can quickly and quietly turn into a liability. You run flutter build, switch configs, sign the build, upload it somewhere, and hope you didn’t mix up staging keys with production ones.

As teams grow and release updates more and more quickly, these manual steps become real risks. A skipped quality check, a missing keystore, or an incorrect base URL deployed to production can cost hours of debugging or worse – it can affect your users.

Automating this process fully involves some high level configuration and predefined scripting. It completely takes control of the deployment process from the moment the developer raised a PR into the common or base branch (for example, the develop branch).

This automated process takes care of everything that needs to be done – provided it has been predefined, properly scripted, and aligns with the use case of the team.

What we'll do here:

In this tutorial, we'll build a production-grade CI/CD pipeline for a Flutter app using GitHub Actions. The pipeline automates the entire lifecycle: pull-request quality checks, environment-specific configuration injection, Android and iOS builds, Firebase App Distribution for testers, Sentry symbol uploads, and final deployment to the Play Store and App Store.

By the end, every release – from a developer opening a PR to the final build landing in users' hands – will be fully automated, with no one touching a terminal.

Prerequisites

Before starting, you should have:

1. A Flutter app with working Android and iOS builds

2. Basic familiarity with GitHub Actions (workflows and jobs)

3. A Firebase project with App Distribution enabled

4. A Sentry project for error tracking

5. A Google Play Console app already created

6. An Apple Developer account with App Store Connect access

7. Fastlane configured for your iOS project

8. Basic Bash knowledge (I’ll explain the important parts)

Pipeline Architecture

In this guide, we'll be building a CI/CD pipeline with very precise instructions and use cases. These use cases determine the way your pipeline is built.

For this tutorial, we'll use this use case:

I want to automate the workflow on my development team based on the following criteria:

1. When a developer on the team raises a PR into the common working branch develop in most cases), a workflow is triggered to run quality checks on the code. It only allows the merge to happen if all checks (like tests coverage, quality checks, and static analysis) pass.

2. Code that's moving from the develop branch to the staging branch goes through another workflow that injects staging configurations/secret keys, does all the necessary checks, and distributes the application for testing on Firebase App Distribution for android as well as Testflight for iOS.

3. Code that's moving from the staging to the production branch goes through the production level workflow which involves apk secured signing, production configuration injection, running tests to ensure nothing breaks, Sentry analysis for monitoring, and submission to App Store Connect as well as Google Play Console.

These are our predefined conditions which help with the construction of our workflows.

Writing the Workflows

We'll split this pipeline into three GitHub Actions workflows.

We'll also be taking it a notch higher by creating three helper .sh scripts for a cleaner and more maintainable workflow.

In your project root, create two folders:

1. .github/

2. scripts.

The .github/ folder will hold the workflows we'll be creating for each use case, while the scripts/ folder will hold the helper scripts that we can easily call in our CLI or in the workflows directly.

After this, we'll create three workflow .yaml files:

1. pr_checks.yaml

2. android.yaml

3. ios.yaml

Also in the scripts folder, let's create three .sh files:

1. generate_config.sh

2. quality_checks.sh

3. upload_symbols.sh

.github/
  workflows/
    pr_checks.yml
    android.yml
    ios.yml

scripts/
  generate_config.sh
  quality_checks.sh
  upload_symbols.sh

This workflow architecture ensures that a push to develop automatically produces a tester build. Also, merging to production ships directly to the stores without manual commands or config changes.

The scripts live outside the YAML on purpose. This lets you run the same logic locally.

The Helper Scripts

The scripts form the backbone of the pipeline. Each one has a single responsibility and is reused across workflows.

Instead of cramming logic into YAML, we'll move it into reusable scripts. This keeps workflows clean and lets you run the same logic locally. Let's go through each one now.

Script #1: generate_config.sh

Injecting secrets safely is one of the hardest CI/CD problems in mobile apps.

The strategy:

• Commit a Dart template file with placeholders

• Replace placeholders at build time using secrets from GitHub Actions

• Never commit real credentials

#!/usr/bin/env bash
set -euo pipefail


ENV_NAME=${1:-}
BASE_URL=${2:-}
ENCRYPTION_KEY=${3:-}

TEMPLATE="lib/core/env/env_ci.dart"
OUT="lib/core/env/env_ci.g.dart"

if [ -z "\(ENV_NAME" ] || [ -z "\)BASE_URL" ] || [ -z "$ENCRYPTION_KEY" ]; then
  echo "Usage: $0 <env-name> <base-url> <encryption-key>"
  exit 2
fi

sed -e "s|<<BASE_URL>>|$BASE_URL|g" \
    -e "s|<<ENCRYPTION_KEY>>|$ENCRYPTION_KEY|g" \
    -e "s|<<ENV_NAME>>|$ENV_NAME|g" \
    "\(TEMPLATE" > "\)OUT"

echo "Generated config for $ENV_NAME"

This script is responsible for injecting environment-specific configuration into the Flutter app at build time, without ever committing secrets to source control.

Let’s walk through it carefully.

1. Shebang: Choosing the Shell

#!/usr/bin/env bash

This line tells the system to execute the script using Bash, regardless of where Bash is installed on the machine.

Using /usr/bin/env bash instead of /bin/bash makes the script more portable across local machines, GitHub Actions runners, and Docker containers.

2. Fail Fast, Fail Loud

set -euo pipefail

This is one of the most important lines in the script.

It enables three strict Bash modes:

• -e: Exit immediately if any command fails

• -u: Exit if an undefined variable is used

• -o pipefail: Fail if any command in a pipeline fails, not just the last one

This matters in CI because silent failures are dangerous, partial config generation can break production builds, and CI should stop immediately when something is wrong.

This line ensures that no broken config ever makes it into a build.

3. Reading Input Arguments

ENV_NAME=${1:-}
BASE_URL=${2:-}
ENCRYPTION_KEY=${3:-}

These lines read positional arguments passed to the script:

• $1: Environment name (dev, staging, production)

• $2: API base URL

• $3: Encryption or API key

The ${1:-} syntax means:

“If the argument is missing, default to an empty string instead of crashing.”

This works hand-in-hand with set -u , we control the failure explicitly instead of letting Bash explode unexpectedly.

4. Defining Input and Output Files

TEMPLATE="lib/core/env/env_ci.dart"
OUT="lib/core/env/env_ci.g.dart"

Here we define two files:

• Template file (env_ci.dart)

  • Contains placeholder values like <<BASE_URL>>

  • Safe to commit to Git

• Generated file (env_ci.g.dart)

  • Contains real environment values

  • Must be ignored by Git (.gitignore)

At the heart of this approach are two Dart files with very different responsibilities. They may look similar, but they play completely different roles in the system.

env.ci.dart:

// lib/core/env/env_ci.dart

class EnvConfig {
  static const String baseUrl = '<<BASE_URL>>';
  static const String encryptionKey = '<<ENCRYPTION_KEY>>';
  static const String environment = '<<ENV_NAME>>';
}

This file is safe, static, and version-controlled. It contains placeholders, not real values.

Some of its key characteristics are:

• Contains no real secrets

• Uses obvious placeholders (<<BASE_URL>>, etc.)

• Safe to commit to Git

• Reviewed like normal source code

• Serves as the single source of truth for required config fields

Think of this file as a contract:

“These are the configuration values the app expects at runtime.”

env.ci.g.dart:

This file is created at build time by generate_config.sh. After substitution, it looks like this:

// lib/core/env/env_ci.g.dart
// GENERATED FILE — DO NOT COMMIT

class EnvConfig {
  static const String baseUrl = 'https://staging.api.example.com';
  static const String encryptionKey = 'sk_live_xxxxx';
  static const String environment = 'staging';
}

Key characteristics:

• Contains real environment values

• Generated dynamically in CI

• Differs per environment (dev / staging / production)

• Must never be committed to source control

This file exists only on a developer’s machine (if generated locally), inside the CI runner during a build. Once the job finishes, it disappears.

.gitignore:

To guarantee the generated file never leaks, it must be ignored:

Why This Separation Is Critical

This design solves several hard problems at once.

Security:

• Secrets live only in GitHub Actions secrets

• They never appear in the repository

• They never appear in PRs

• They never appear in Git history

Environment Isolation:

Each environment gets its own generated config:

• develop: dev API

• staging: staging API

• production: production API

The same codebase behaves differently without branching logic in Dart.

Deterministic Builds:

Every build is fully reproducible, fully automated, and explicit about which environment it targets.

There are no “it worked locally” scenarios.

5. Validating Required Arguments

if [ -z "\(ENV_NAME" ] || [ -z "\)BASE_URL" ] || [ -z "$ENCRYPTION_KEY" ]; then
  echo "Usage: $0 <env-name> <base-url> <encryption-key>"
  exit 2
fi

This block enforces correct usage.

• -z checks whether a variable is empty

• If any required argument is missing:

  • A helpful usage message is printed

  • The script exits with a non-zero status code

• 0: success

• 1+: failure

• 2 conventionally means incorrect usage

In CI, this immediately fails the job and prevents an invalid build.

6. Injecting Environment Values

sed -e "s|<<BASE_URL>>|$BASE_URL|g" \
    -e "s|<<ENCRYPTION_KEY>>|$ENCRYPTION_KEY|g" \
    -e "s|<<ENV_NAME>>|$ENV_NAME|g" \
    "\(TEMPLATE" > "\)OUT"

This is the heart of the script.

What’s happening here:

1. sed performs stream editing: it reads text, transforms it, and outputs the result

2. Each -e flag defines a replacement rule:

   • Replace <<BASE_URL>> with the actual API URL

   • Replace <<ENCRYPTION_KEY>> with the real key

   • Replace <<ENV_NAME>> with the environment label

3. The transformed output is written to env_ci.g.dart

This entire operation happens at build time:

• No secrets are committed

• No secrets are logged

• No secrets persist beyond the CI run

7. Success Feedback

echo "Generated config for $ENV_NAME"

This line provides a clear success signal in CI logs.

It answers three important questions instantly:

• Did the script run?

• Did it finish successfully?

• Which environment was generated?

In long CI logs, these small confirmations matter.

Alright, now let's move on to the second script.

Script #2: quality_gate.sh

This script defines what “good code” means for your team.

#!/usr/bin/env bash
set -euo pipefail

echo "Running quality checks"

dart format --output=none --set-exit-if-changed .
flutter analyze
flutter test --no-pub --coverage

if command -v dart_code_metrics >/dev/null 2>&1; then
  dart_code_metrics analyze lib --reporter=console || true
fi

echo "Quality checks passed"

Lets break down this script bit by bit.

1. Start & End Log Markers

echo "Running quality checks"
...
echo "Quality checks passed"

These two lines act as visual boundaries in CI logs.

In large pipelines (especially when Android and iOS jobs run in parallel), logs can be very noisy. Clear markers:

• Help developers quickly find the quality phase

• Make debugging faster

• Confirm that the script completed successfully

The final success message only prints if everything above it passed, because set -e would have terminated the script earlier on failure.

So this line effectively means: All quality gates passed. Safe to proceed.

2. Running the Test Suite

flutter test --no-pub --coverage

This line executes your entire Flutter test suite.

Let’s break it down carefully.

1. flutter test

This runs unit tests, widget tests, and any test under the test/ directory. If any test fails, the command exits with a non-zero status code.

Because we enabled set -e earlier, that immediately stops the script and fails the CI job.

2. --coverage

This flag generates a coverage report at:

coverage/lcov.info

This file can later be uploaded to Codecov, used to enforce minimum coverage thresholds, and tracked over time for quality improvement.

Even if you’re not enforcing coverage yet, generating it now future-proofs your pipeline.

3. Optional Code Metrics

if command -v dart_code_metrics >/dev/null 2>&1; then
  dart_code_metrics analyze lib --reporter=console || true
fi

This block is intentionally designed to be optional and non-blocking.

Step 1 – Check If the Tool Exists:

command -v dart_code_metrics >/dev/null 2>&1

This checks whether dart_code_metrics is installed.

• If installed, proceed

• If not installed, skip silently

The redirection:

• >/dev/null hides normal output

• 2>&1 hides errors

This makes the script portable:

• Developers without the tool can still run the script

• CI can enforce it if configured

Step 2 – Run Metrics (Soft Enforcement):

dart_code_metrics analyze lib --reporter=console || true

This analyzes the lib/ directory and prints results in the console.

The important part is:

|| true

Because we enabled set -e, any failing command would normally stop the script.

Adding || true overrides that behavior:

• If metrics report issues,

• The script continues,

• CI does not fail.

Why design it this way? Because metrics are often gradual improvements, technical debt indicators, or advisory rather than blocking.

You can later remove || true to make metrics mandatory.

4. Final Success Message

echo "✅ Quality checks passed"

This line only executes if formatting passed, static analysis passed, and tests passed.

If you see this in CI logs, it means the branch has successfully cleared the quality gate. It’s your automated approval before deployment steps begin.

What This Script Guarantees

With this in place, every branch must satisfy:

• Clean formatting

• No analyzer errors

• Passing tests

• (Optional) Healthy metrics

That’s how you move from “We try to maintain quality” to “Quality is enforced automatically.”

Alright, on to the third script.

Script #3: upload_symbols.sh (Sentry)

This script is responsible for uploading obfuscation debug symbols to Sentry so production crashes remain readable.

#!/usr/bin/env bash
set -euo pipefail

RELEASE=${1:-}

[ -z "$RELEASE" ] && exit 2

if ! command -v sentry-cli >/dev/null 2>&1; then
  exit 0
fi

sentry-cli releases new "$RELEASE" || true

sentry-cli upload-dif build/symbols || true

sentry-cli releases finalize "$RELEASE" || true

echo "✅ Symbols uploaded for release $RELEASE"

Let's go through it step by step.

1. Reading the Release Identifier

RELEASE=${1:-}

This reads the first positional argument passed to the script.

When you call the script in CI, it typically looks like:

./scripts/upload_symbols.sh $(git rev-parse --short HEAD)

So $1 becomes the short Git commit SHA.

Using ${1:-} ensures:

• If no argument is passed, the variable becomes an empty string

• The script does not crash due to set -u

This release value ties the uploaded symbols, deployed build, and crash reports all to the exact same commit. This linkage is critical for production debugging.

2. Validating the Release Argument

[ -z "$RELEASE" ] && exit 2

This is a compact validation check.

• -z checks whether the string is empty

• If it is empty → exit with status code 2

Conventionally:

• 0 = success

• 1+ = failure

• 2 = incorrect usage

This prevents symbol uploads from running without a release identifier, which would break traceability in Sentry.

3. Checking If sentry-cli Exists

if ! command -v sentry-cli >/dev/null 2>&1; then
  exit 0
fi

This block checks whether the sentry-cli tool is available in the environment.

What’s happening:

• command -v sentry-cli checks if it exists

• >/dev/null 2>&1 suppresses all output

• ! negates the condition

So this reads as: "If sentry-cli is NOT installed, exit successfully."

Why exit with 0 instead of failing?

Because not every environment needs symbol uploads. Also, dev builds may not install Sentry, and you don’t want CI to fail just because Sentry isn’t configured.

This makes symbol uploading environment-aware and optional.

Production environments can install sentry-cli, while dev environments skip it cleanly.

4. Creating a New Release in Sentry

sentry-cli releases new "$RELEASE" || true

This tells Sentry: “A new release exists with this version identifier.”

Even if the release already exists, the script continues because of:

|| true

This prevents the build from failing if:

• The release was already created

• The command returns a non-critical error

The goal is resilience, not strict enforcement.

5. Uploading Debug Information Files (DIFs)

sentry-cli upload-dif build/symbols || true

This is the core step.

build/symbols is generated when you build Flutter with:

--obfuscate --split-debug-info=build/symbols

When you obfuscate Flutter builds:

• Method names are renamed

• Stack traces become unreadable

The symbol files allow Sentry to reverse-map obfuscated stack traces and show readable crash reports.

Without this step, production crashes look like:

a.b.c.d (Unknown Source)

With this step, you get:

AuthRepository.login()

Again, || true ensures the build doesn’t fail if:

• The directory doesn’t exist

• No symbols were generated

• Upload encounters a transient issue

Symbol uploads should not block deployment.

6. Finalizing the Release

sentry-cli releases finalize "$RELEASE" || true

This marks the release as complete in Sentry.

Finalizing signals:

• The release is deployed

• It can begin aggregating crash reports

• It’s ready for production monitoring

Like the previous steps, this is soft-failed with || true to keep CI robust.

What This Script Guarantees

When everything is configured correctly:

1. Production build is obfuscated

2. Debug symbols are generated

3. Symbols are uploaded to Sentry

4. Crashes map back to real source code

5. Release version matches commit SHA

That’s production-grade crash observability.

Now that we've gone through the three helper scripts we've created to optimize and enhance this process, lets now dive into the three workflow .yaml files we're going to create.

Workflow #1: PR_CHECKS.YML

This workflow will be designed to help ensure a certain standard is met once a PR is raised into a certain common or base branch. This will ensure that all quality checks in the incoming code pass before allowing any merge into the base branch.

This is basically a gate that verifies the quality of the code that's about to be merged into the base branch. If your pipeline allows unverified code into your base branch, then your CI becomes decorative, not protective.

Lets break down what's actually needed during every PR Check.

1. Dependency Integrity

For Flutter apps, where we manage dependencies with the pub get command, it's important to make sure that the integrity of all dependencies are confirmed – up to date as well as compatible.

Every PR should begin with:

flutter pub get

This ensures:

• pubspec.yaml is valid

• Dependency constraints are consistent

• Lockfiles are not broken

• The project is buildable in a clean environment

If this fails, the branch is not deployable.

2. Static Analysis

This ensures code quality and architecture integrity. Static analysis helps prevent common issues like forgotten await, dead code, null safety violations, async misuse, and so on.

Most production bugs aren't business logic errors – they're structural carelessness. Static analysis helps enforce consistency automatically, so code reviews focus on intent, not linting.

flutter analyze --fatal-infos --fatal-warnings

3. Formatting

This command ensures that your code is properly formatted based on your organization's coding standard and policies.

dart format --output=none --set-exit-if-changed .

4. Tests

This runs the unit, widget and business logic tests to ensure quality and avoid regression leaks, silent behavior changes and feature drift.

flutter test --coverage

5. Test Coverage Enforcement

Ideally, running tests is not enough. Your workflow should also enforce a minimum threshold:

if [ \((lcov --summary coverage/lcov.info | grep lines | awk '{print \)2}' | sed 's/%//') -lt 70 ]; then
  echo "Coverage too low"
  exit 1
fi

The command above ensures that a minimum test coverage of 70% is met, with this quality becomes measurable.

The five commands above must be checked (at least) for a quality gate to guarantee code quality, security, and integrity.

Now here is the full pr_checks.yml file:

name: PR Quality Gate

on:
  pull_request:
    branches: develop
    types: [opened, synchronize, reopened, ready_for_review]

jobs:
  pr-checks:
    name: Run quality checks on this pull request
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup Java
        uses: actions/setup-java@v1
        with:
          java-version: "12.x"

      - name: Setup Flutter
        uses: subosito/flutter-action@v1
        with:
          channel: "stable"

      - name: Install dependencies
        run: flutter pub get

      - name: Run quality checks
        run: ./scripts/quality_checks.sh

      - name: Notify Team (Success)
        if: success()
        run: |
          echo "PR Quality Checks PASSED"
          echo "PR: ${{ github.event.pull_request.html_url }}"
          echo "Branch: \({{ github.head_ref }} → \){{ github.base_ref }}"
          echo "By: @${{ github.actor }}"
          echo "Team notification: @foluwaseyi-dev @olabodegbolu"

      - name: Notify Team (Failure)
        if: failure()
        run: |
          echo "PR Quality Checks FAILED"
          echo "PR: ${{ github.event.pull_request.html_url }}"
          echo "Branch: \({{ github.head_ref }} → \){{ github.base_ref }}"
          echo "By: @${{ github.actor }}"
          echo "Please fix the issues before requesting review 🔧"
          echo "Team notification: @foluwaseyi-dev @olabodegbolu"

Every time a developer opens (or updates) a pull request targeting the develop branch, this workflow kicks in automatically. Think of it as a bouncer at the door: no code gets through without passing inspection first.

What Triggers it?

The workflow fires on four events: when a PR is opened, synchronized (new commits pushed), reopened, or marked ready_for_review. So drafts won't trigger it – only PRs that are actually ready to be looked at.

What Does it Actually Do?

It spins up a fresh Ubuntu machine and runs five steps in sequence:

1. Checkout: pulls down the branch's code

2. Setup Java 12: installs the JDK (likely a dependency for some tooling or build process)

3. Setup Flutter (stable channel): this is a Flutter project, so it grabs the stable Flutter SDK

4. Install dependencies: runs flutter pub get to pull all Dart/Flutter packages

5. Run quality checks: executes the helper shell script (./scripts/quality_checks.sh) that we created which runs linting, tests, formatting checks, or all of the above

The Notification Layer

After the checks run, the workflow reports the verdict and it's context-aware:

• If everything passes, it logs a success message with the PR URL, branch info, and the person who opened it

• If something fails, it logs a failure message and nudges the author to fix issues before requesting a review

Both outcomes tag @foluwaseyi-dev and @olabodegbolu – the two team members responsible for staying in the loop.

This workflow enforces a "fix it before you merge it" culture. No one can merge broken code into develop without the team knowing about it.

Workflow #2: Android.yml

It's a better practice to split your workflows based on platform. This helps you properly manage the instructions regarding each platform. This is the reason behind keeping the Android workflow separate.

Unlike PR _Checks, this workflow presumes that all checks for quality and standards have been done and the code that runs this workflow already meets the required standards.

Based on our predefined use case, let's create a workflow to handle test deployments when merged to develop or staging, and production level activities when merged to production.

name: Android Build & Release

on:
  push:
    branches:
      - develop
      - staging
      - production

jobs:
  android:
    runs-on: ubuntu-latest
    env:
      FLUTTER_VERSION: 'stable'

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '11'

      - name: Setup Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: ${{ env.FLUTTER_VERSION }}

      - name: Install dependencies
        run: flutter pub get

      - name: Determine environment
        id: env
        run: |
          echo "branch=\({GITHUB_REF##*/}" >> \)GITHUB_OUTPUT
          if [ "${GITHUB_REF##*/}" = "develop" ]; then
            echo "ENV=dev" >> $GITHUB_OUTPUT
          elif [ "${GITHUB_REF##*/}" = "staging" ]; then
            echo "ENV=staging" >> $GITHUB_OUTPUT
          else
            echo "ENV=production" >> $GITHUB_OUTPUT
          fi

      # Dev uses hardcoded values no secrets needed
      - name: Generate config (dev)
        if: steps.env.outputs.ENV == 'dev'
        run: ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"

      # Staging and production inject real secrets
      - name: Generate config (staging/production)
        if: steps.env.outputs.ENV != 'dev'
        run: |
          if [ "${{ steps.env.outputs.ENV }}" = "staging" ]; then
            ./scripts/generate_config.sh staging \
              "${{ secrets.STAGING_BASE_URL }}" \
              "${{ secrets.STAGING_API_KEY }}"
          else
            ./scripts/generate_config.sh production \
              "${{ secrets.PROD_BASE_URL }}" \
              "${{ secrets.PROD_API_KEY }}"
          fi

      # Keystore is only needed for signed builds (staging & production)
      - name: Restore Keystore
        if: steps.env.outputs.ENV != 'dev'
        run: |
          echo "${{ secrets.ANDROID_KEYSTORE_BASE64 }}" | base64 --decode > android/app/upload-keystore.jks

      # Production builds are obfuscated + split debug info for Play Store
      - name: Build artifact
        run: |
          if [ "${{ steps.env.outputs.ENV }}" = "production" ]; then
            flutter build appbundle --release \
              --obfuscate \
              --split-debug-info=build/symbols
          else
            flutter build appbundle --release
          fi

      # Dev and staging go to Firebase App Distribution for internal testing
      - name: Upload to Firebase App Distribution
        if: steps.env.outputs.ENV == 'dev' || steps.env.outputs.ENV == 'staging'
        env:
          FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
          FIREBASE_ANDROID_APP_ID: ${{ secrets.FIREBASE_ANDROID_APP_ID }}
          FIREBASE_GROUPS: ${{ secrets.FIREBASE_GROUPS }}
        run: |
          firebase appdistribution:distribute \
            build/app/outputs/bundle/release/app-release.aab \
            --app "$FIREBASE_ANDROID_APP_ID" \
            --groups "$FIREBASE_GROUPS" \
            --token "$FIREBASE_TOKEN"

      # Only production goes to the Play Store
      - name: Upload to Play Store
        if: steps.env.outputs.ENV == 'production'
        uses: r0adkll/upload-google-play@v1
        with:
          serviceAccountJsonPlainText: ${{ secrets.GOOGLE_PLAY_SERVICE_ACCOUNT_JSON }}
          packageName: com.your.package
          releaseFiles: build/app/outputs/bundle/release/app-release.aab
          track: production

      - name: Notify Team (Success)
        if: success()
        run: |
          echo "Android Build & Release PASSED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"

      - name: Notify Team (Failure)
        if: failure()
        run: |
          echo "Android Build & Release FAILED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"
          echo "Check the logs and fix the issue before retrying"

This workflow ensures that whenever code lands on the develop, staging or production branch, this action is triggered on a fresh Ubuntu machine.

This is triggered by a simple push to any of the tracked branches, no manual intervention needed.

Let's walk through it piece by piece.

1. The Setup Phase

Before any Flutter-specific work happens, the workflow lays the foundation:

1. Checkout: grabs the latest code from the branch that triggered the run (using the more modern actions/checkout@v3).

2. Java 11 via Temurin: this is an upgrade from the first workflow we created. Instead of a generic setup-java@v1, this uses the temurin distribution which is the Eclipse's open-source JDK build. It's the current industry standard for Android toolchains.

3. Flutter (stable): this pulls the stable Flutter SDK, version pinned via an environment variable (FLUTTER_VERSION: 'stable') defined at the job level.

4. Install dependencies: this ensures we run flutter pub get to pull all packages

2. Environment Detection

This is where it gets interesting. This workflow also checks and determines the environment which will help us define the next set of instructions to run.

This command reads the branch name from GITHUB REF and maps it to its environment label which we already created in one of our helper scripts.

• develop → ENV=dev

• staging → ENV=staging

• production → ENV=production

It strips the branch name from the full ref path using \({GITHUB_REF##*/}, then writes both the branch name and the resolved ENV value to \)GITHUB_OUTPUT, making them available as named outputs (steps.env.outputs.ENV) for every subsequent step.

This means the rest of the pipeline can branch its behaviour based on which environment it's building for, different API keys, different signing configs, different targets – whatever the app needs.

3. Config Injection

With the environment resolved, the next step is injecting the right configuration into the app. This is where the generate_config.sh script we built earlier gets called directly from the workflow.

For the dev environment, hardcoded placeholder values are used. No real secrets are needed, since this build is only meant for internal developer testing:

- name: Generate config (dev)
  if: steps.env.outputs.ENV == 'dev'
  run: ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"

For staging and production, however, real secrets are pulled from GitHub Actions secrets and passed directly into the script:

- name: Generate config (staging/production)
  if: steps.env.outputs.ENV != 'dev'
  run: |
    if [ "${{ steps.env.outputs.ENV }}" = "staging" ]; then
      ./scripts/generate_config.sh staging \
        "${{ secrets.STAGING_BASE_URL }}" \
        "${{ secrets.STAGING_API_KEY }}"
    else
      ./scripts/generate_config.sh production \
        "${{ secrets.PROD_BASE_URL }}" \
        "${{ secrets.PROD_API_KEY }}"
    fi

Notice that these two steps use an if condition to make them mutually exclusive. Only one will ever run per job. This keeps the pipeline clean: no complicated branching logic inside the script itself, just a clear decision at the workflow level.

4. Keystore Restoration

Android requires signed builds for distribution. The signing keystore file cannot be committed to the repository for obvious security reasons, so it's stored as a Base64-encoded GitHub secret and decoded at build time.

- name: Restore Keystore
  if: steps.env.outputs.ENV != 'dev'
  run: |
    echo "${{ secrets.ANDROID_KEYSTORE_BASE64 }}" | base64 --decode > android/app/upload-keystore.jks

This step is skipped entirely for the dev environment because dev builds are unsigned debug artifacts meant purely for internal testing on Firebase App Distribution. Only staging and production builds need to be properly signed.

To encode your keystore file as a Base64 string for storing in GitHub secrets, you have to run this locally:

base64 -i upload-keystore.jks | pbcopy

This copies the encoded string to your clipboard, which you can then paste directly into your GitHub repository secrets.

5. Building the Artifact

With the environment configured and the keystore in place, the workflow builds the app bundle:

- name: Build artifact
  run: |
    if [ "${{ steps.env.outputs.ENV }}" = "production" ]; then
      flutter build appbundle --release \
        --obfuscate \
        --split-debug-info=build/symbols
    else
      flutter build appbundle --release
    fi

There's a deliberate difference between how production and non-production builds are compiled.

For production:

• --obfuscate renames method and class names in the compiled output, making it significantly harder to reverse engineer the app

• --split-debug-info=build/symbols extracts the debug symbols into a separate directory at build/symbols

These symbols are what upload_symbols.sh later ships to Sentry, so obfuscated crash reports remain readable in your monitoring dashboard.

For dev and staging, neither flag is used. This keeps build times faster and makes local debugging easier since stack traces remain human-readable.

6. Distributing to Firebase App Distribution

Once the app bundle is built, dev and staging builds are uploaded to Firebase App Distribution so testers can install them immediately:

- name: Upload to Firebase App Distribution
  if: steps.env.outputs.ENV == 'dev' || steps.env.outputs.ENV == 'staging'
  env:
    FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
    FIREBASE_ANDROID_APP_ID: ${{ secrets.FIREBASE_ANDROID_APP_ID }}
    FIREBASE_GROUPS: ${{ secrets.FIREBASE_GROUPS }}
  run: |
    firebase appdistribution:distribute \
      build/app/outputs/bundle/release/app-release.aab \
      --app "$FIREBASE_ANDROID_APP_ID" \
      --groups "$FIREBASE_GROUPS" \
      --token "$FIREBASE_TOKEN"

Three secrets power this step:

• FIREBASE_TOKEN: the authentication token generated from firebase login:ci

• FIREBASE_ANDROID_APP_ID: the app identifier from the Firebase console

• FIREBASE_GROUPS: the tester group(s) that should receive the build notification

Once this step completes, every tester in the specified groups receives an email with a direct download link. No one needs to manually share an APK file over Slack or email.

7. Deploying to the Play Store

Production builds skip Firebase entirely and goes straight to the Google Play Store:

- name: Upload to Play Store
  if: steps.env.outputs.ENV == 'production'
  uses: r0adkll/upload-google-play@v1
  with:
    serviceAccountJsonPlainText: ${{ secrets.GOOGLE_PLAY_SERVICE_ACCOUNT_JSON }}
    packageName: com.your.package
    releaseFiles: build/app/outputs/bundle/release/app-release.aab
    track: production

This uses the r0adkll/upload-google-play GitHub Action, which handles the Google Play API interaction under the hood. The only requirements are:

• A Google Play service account with the correct permissions, stored as a JSON secret

• The correct package name matching what is registered in your Play Console

• The track set to production (you can also use internal, alpha, or beta depending on your release strategy)

Replace com.your.package with your actual application ID (the same one defined in your build.gradle file).

8. The Notification Layer

Just like the PR checks workflow, this workflow reports its outcome clearly:

- name: Notify Team (Success)
  if: success()
  run: |
    echo "Android Build & Release PASSED"
    echo "Environment: ${{ steps.env.outputs.ENV }}"
    echo "Branch: ${{ steps.env.outputs.branch }}"
    echo "By: @${{ github.actor }}"
    echo "Commit: ${{ github.sha }}"

- name: Notify Team (Failure)
  if: failure()
  run: |
    echo "Android Build & Release FAILED"
    echo "Environment: ${{ steps.env.outputs.ENV }}"
    echo "Branch: ${{ steps.env.outputs.branch }}"
    echo "By: @${{ github.actor }}"
    echo "Commit: ${{ github.sha }}"
    echo "Check the logs and fix the issue before retrying 🔧"

The success notification includes the environment, branch, actor, and shares everything needed to trace exactly what was deployed and who triggered it.

The failure notification includes the same context, with a clear call to action.

Workflow #3: iOS.yml

iOS CI/CD is more complex than Android by nature. This is because Apple's signing requirements involve certificates, provisioning profiles, and entitlements that all need to be in the right place before Xcode will produce a valid archive.

Fastlane helps us handles all of that complexity, and the workflow simply calls into it.

Here is the full ios.yml:

name: iOS Build & Release

on:
  push:
    branches:
      - develop
      - staging
      - production

jobs:
  ios:
    runs-on: macos-latest
    env:
      FLUTTER_VERSION: 'stable'

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: ${{ env.FLUTTER_VERSION }}

      - name: Install dependencies
        run: flutter pub get

      - name: Determine environment
        id: env
        run: |
          echo "branch=\({GITHUB_REF##*/}" >> \)GITHUB_OUTPUT
          if [ "${GITHUB_REF##*/}" = "develop" ]; then
            echo "ENV=dev" >> $GITHUB_OUTPUT
          elif [ "${GITHUB_REF##*/}" = "staging" ]; then
            echo "ENV=staging" >> $GITHUB_OUTPUT
          else
            echo "ENV=production" >> $GITHUB_OUTPUT
          fi

      - name: Generate config (dev)
        if: steps.env.outputs.ENV == 'dev'
        run: ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"

      - name: Generate config (staging/production)
        if: steps.env.outputs.ENV != 'dev'
        run: |
          if [ "${{ steps.env.outputs.ENV }}" = "staging" ]; then
            ./scripts/generate_config.sh staging \
              "${{ secrets.STAGING_BASE_URL }}" \
              "${{ secrets.STAGING_API_KEY }}"
          else
            ./scripts/generate_config.sh production \
              "${{ secrets.PROD_BASE_URL }}" \
              "${{ secrets.PROD_API_KEY }}"
          fi

      - name: Install Fastlane
        run: |
          cd ios
          gem install bundler
          bundle install

      - name: Import signing certificate
        if: steps.env.outputs.ENV != 'dev'
        run: |
          echo "${{ secrets.IOS_CERTIFICATE_BASE64 }}" | base64 --decode > ios/cert.p12
          security create-keychain -p "" build.keychain
          security import ios/cert.p12 -k build.keychain -P "${{ secrets.IOS_CERTIFICATE_PASSWORD }}" -T /usr/bin/codesign
          security list-keychains -s build.keychain
          security default-keychain -s build.keychain
          security unlock-keychain -p "" build.keychain
          security set-key-partition-list -S apple-tool:,apple: -s -k "" build.keychain

      - name: Install provisioning profile
        if: steps.env.outputs.ENV != 'dev'
        run: |
          echo "${{ secrets.IOS_PROVISIONING_PROFILE_BASE64 }}" | base64 --decode > profile.mobileprovision
          mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
          cp profile.mobileprovision ~/Library/MobileDevice/Provisioning\ Profiles/

      - name: Build iOS (dev)
        if: steps.env.outputs.ENV == 'dev'
        run: flutter build ios --release --no-codesign

      - name: Build & distribute to TestFlight (staging)
        if: steps.env.outputs.ENV == 'staging'
        env:
          APP_STORE_CONNECT_API_KEY_ID: ${{ secrets.APP_STORE_CONNECT_API_KEY_ID }}
          APP_STORE_CONNECT_API_ISSUER_ID: ${{ secrets.APP_STORE_CONNECT_API_ISSUER_ID }}
          APP_STORE_CONNECT_API_KEY_CONTENT: ${{ secrets.APP_STORE_CONNECT_API_KEY_CONTENT }}
        run: |
          cd ios
          bundle exec fastlane beta

      - name: Build & release to App Store (production)
        if: steps.env.outputs.ENV == 'production'
        env:
          APP_STORE_CONNECT_API_KEY_ID: ${{ secrets.APP_STORE_CONNECT_API_KEY_ID }}
          APP_STORE_CONNECT_API_ISSUER_ID: ${{ secrets.APP_STORE_CONNECT_API_ISSUER_ID }}
          APP_STORE_CONNECT_API_KEY_CONTENT: ${{ secrets.APP_STORE_CONNECT_API_KEY_CONTENT }}
        run: |
          cd ios
          bundle exec fastlane release

      - name: Upload Sentry symbols (production only)
        if: steps.env.outputs.ENV == 'production'
        env:
          SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
          SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
          SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}
        run: ./scripts/upload_symbols.sh $(git rev-parse --short HEAD)

      - name: Notify Team (Success)
        if: success()
        run: |
          echo "iOS Build & Release PASSED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"

      - name: Notify Team (Failure)
        if: failure()
        run: |
          echo "iOS Build & Release FAILED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"
          echo "Check the logs and fix the issue before retrying 🔧"

Let's walk through what is different about this workflow compared to that of android.

1. MacOS Runner

runs-on: macos-latest

This is the major difference.

iOS builds require Xcode, which only runs on macOS. GitHub Actions provides hosted macOS runners, but they are significantly more expensive in terms of compute minutes than Ubuntu runners. Just keep that in mind when thinking about build frequency.

No Java setup is needed here. Flutter on iOS compiles through Xcode directly, so the toolchain requirements are different.

2. Installing Fastlane

- name: Install Fastlane
  run: |
    cd ios
    gem install bundler
    bundle install

Fastlane is a Ruby-based automation tool that handles certificate management, building, and uploading to TestFlight and the App Store.

This step navigates into the ios/ directory and installs Fastlane along with all its dependencies as defined in the project's Gemfile.

Your ios/Gemfile should look something like this:

source "https://rubygems.org"

gem "fastlane"

And your ios/fastlane/Fastfile should define at minimum two lanes: one for staging (TestFlight) and one for production (App Store):

default_platform(:ios)

platform :ios do
  lane :beta do
    build_app(scheme: "Runner", export_method: "app-store")
    upload_to_testflight(skip_waiting_for_build_processing: true)
  end

  lane :release do
    build_app(scheme: "Runner", export_method: "app-store")
    upload_to_app_store(force: true, skip_screenshots: true, skip_metadata: true)
  end
end

3. Certificate and Provisioning Profile Setup

This is the step that trips most teams up the first time. Apple's code signing requires two things to be present on the machine:

1. The signing certificate (a .p12 file)

2. The provisioning profile

Both are stored as Base64-encoded GitHub secrets and restored at build time.

- name: Import signing certificate
  if: steps.env.outputs.ENV != 'dev'
  run: |
    echo "${{ secrets.IOS_CERTIFICATE_BASE64 }}" | base64 --decode > ios/cert.p12
    security create-keychain -p "" build.keychain
    security import ios/cert.p12 -k build.keychain -P "${{ secrets.IOS_CERTIFICATE_PASSWORD }}" -T /usr/bin/codesign
    security list-keychains -s build.keychain
    security default-keychain -s build.keychain
    security unlock-keychain -p "" build.keychain
    security set-key-partition-list -S apple-tool:,apple: -s -k "" build.keychain

Breaking this down step by step:

• Decodes the Base64 certificate and write it to cert.p12

• Creates a temporary keychain called build.keychain with an empty password

• Imports the certificate into that keychain, granting codesign access

• Sets it as the default keychain so Xcode finds it automatically

• Unlocks the keychain so it can be used non-interactively

• Sets partition list to allow access without repeated prompts

The provisioning profile step is simpler:

- name: Install provisioning profile
  if: steps.env.outputs.ENV != 'dev'
  run: |
    echo "${{ secrets.IOS_PROVISIONING_PROFILE_BASE64 }}" | base64 --decode > profile.mobileprovision
    mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
    cp profile.mobileprovision ~/Library/MobileDevice/Provisioning\ Profiles/

It decodes the profile and copies it into the exact directory where Xcode expects to find provisioning profiles on any macOS system.

To encode your certificate and profile locally, you can run these:

base64 -i Certificates.p12 | pbcopy   # for the certificate
base64 -i YourApp.mobileprovision | pbcopy   # for the provisioning profile

4. Building for Each Environment

Dev builds skip signing entirely. They're built without code signing just to verify the project compiles correctly on a clean machine:

- name: Build iOS (dev)
  if: steps.env.outputs.ENV == 'dev'
  run: flutter build ios --release --no-codesign

Staging builds go through Fastlane's beta lane, which builds and uploads to TestFlight. Production builds go through Fastlane's release lane, which submits directly to App Store Connect.

Both staging and production steps consume the same three App Store Connect API key secrets: the key ID, the issuer ID, and the key content itself.

Fastlane uses these to authenticate with Apple's API without requiring a manual Apple ID login.

5. Sentry Symbol Upload

On production iOS builds, the upload_symbols.sh script runs after the build completes, passing the current short commit SHA as the release identifier:

- name: Upload Sentry symbols (production only)
  if: steps.env.outputs.ENV == 'production'
  env:
    SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
    SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
    SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}
  run: ./scripts/upload_symbols.sh $(git rev-parse --short HEAD)

This is the same script explained earlier in the helper scripts section. It creates a Sentry release, uploads the debug information files, and finalizes the release. Any production crash from this point forward will map back to real, readable source code in your Sentry dashboard.

Secrets and Configuration Reference

For this entire pipeline to work, you need to configure the following secrets in your GitHub repository. Go to Settings → Secrets and variables → Actions → New repository secret to add each one.

Shared (used across environments):

Staging:

Production:

Android:

iOS:

None of these values should ever appear in your codebase. If any secret is accidentally committed, rotate it immediately.

End-to-End Flow

With all three workflows in place, here is exactly what happens from the moment a developer opens a pull request to the moment a user receives an update:

1. Developer Opens a PR into develop

The pr_checks.yml workflow fires. It runs formatting checks, static analysis, and the full test suite. If anything fails, the PR cannot be merged and the team is notified immediately. The developer fixes the issues and pushes again, which triggers a fresh run.

2. PR is Approved and Merged into develop

The android.yml and ios.yml workflows both fire on the push event. They detect the environment as dev, inject placeholder config, build unsigned artifacts, and upload them to Firebase App Distribution. Testers receive an email and can install the build on their devices within minutes – no one shared a file manually.

3. develop is Merged into staging

Both platform workflows fire again. This time the environment resolves to staging. Real secrets are injected, builds are properly signed, and the artifacts go to Firebase App Distribution (Android) and TestFlight (iOS). QA begins testing the staging build against the staging API.

4. staging is merged into production

Both workflows fire one final time. Production secrets are injected, builds are obfuscated and signed, debug symbols are uploaded to Sentry, and the final artifacts are submitted to the Google Play Store and App Store Connect. The release goes live on Apple and Google's review timelines with no further human intervention required.

From that first PR to a production submission, not a single command was run manually.

Conclusion

Building this pipeline is an upfront investment that pays off from the very first release cycle. What used to be a sequence of error-prone manual steps building locally, signing, uploading, switching configs, and hoping nothing was mixed up is now a fully automated, auditable, and repeatable process that runs the moment code moves between branches.

The architecture we built here does more than just automate builds. The PR quality gate enforces team standards consistently, so code review becomes a conversation about intent rather than a hunt for formatting issues. The environment-aware config injection eliminates an entire class of production incidents where staging keys made it into a live release. The Sentry symbol upload means your team can debug production crashes with full source visibility even from an obfuscated binary.

Every piece of this pipeline also runs locally. The helper scripts in the scripts/ folder are plain Bash so you can call them from your terminal the same way CI calls them. This eliminates the frustrating cycle of pushing a commit just to test a pipeline change.

As your team grows, this foundation scales with you. You can extend the pr_checks.yml to enforce code coverage thresholds, add a performance benchmarking job, or introduce a dedicated security scanning step. You can extend the platform workflows to support multiple flavors, multiple Firebase projects, or staged rollouts on the Play Store. The architecture stays the same – you're just adding new steps to an already working system.

This ensures that standards are met, code quality remains high, you have a proper team structure, clear process and automated post development activities are in place – and at the end of the day, you'll have an optimized engineering approach that will help your team in so many ways.

Here are the key sections in this course:

• Modern SDLC Explained

• CI/CD Concepts & Branching Strategies

• Jenkins Basics & Installation

• Jenkins Freestyle Jobs Deep Dive

• CI/CD Project | Dockerized Flask App

• Jenkins Pipelines (Build, Push, Deploy)

• Transition to Multibranch Pipelines

• Maven for DevOps

• DevSecOps Explained

• DevSecOps Mega Project

• Jenkins Shared Libraries

Watch the full course on the freeCodeCamp.org YouTube channel (17-hour watch).

We will go far beyond the usual “Hello World” examples and walk you through containerizing a complete full-stack JavaScript application (Node.js + Express backend, HTML/CSS/JS frontend, MongoDB database, and Mongo Express admin UI).

You’ll learn about networking multiple containers, orchestrating everything with Docker Compose, building and versioning your own images, persisting data with volumes, and securely pushing your Images to a private AWS ECR repository for sharing and production deployment.

By the end, you’ll be able to eliminate “it works on my machine” issues, confidently manage multi-service applications, deploy consistent environments anywhere, and integrate Docker into your daily workflow and CI/CD pipelines like a pro.

Since Docker is such a key skill for backend developers, we’ll start by covering its basic concepts.

Prerequisites

This technical handbook is designed for developers who have some practical, hands-on experience in full-stack development. You should be comfortable deploying applications and have a basic understanding of CI/CD pipelines.

While we’ll cover Docker from the ground up, this guide is not for absolute beginner developers. I assume you have real-world development experience and want to level up your workflow with Docker.

Finally, a basic familiarity with AWS and general deployment concepts will also be useful, though you don’t need to be an expert. This handbook is ideal for developers looking to enhance their production-grade skills and confidently integrate Docker into their projects.

Table of Contents:

1. What is a Container?

2. Docker vs Virtual Machines

3. Docker Installation

4. Basic Docker Commands

5. Practice with JavaScript

   • How to Pull the MongoDB Image

   • How to Pull the Mongo Express Image

   • Docker Network

6. How to Run the Mongo Container

7. How to Run the Mongo Express Container

8. How to Connect Node.js to MongoDB

9. How to Use Docker Compose

   • Why Use Docker Compose?

10. How to Build Our Own Docker Image

    • The Solution

    • Why Mongodb Works

    • Add Your App to Docker Compose

    • Start All Services

    • Verify Everything Works

    • What Changed and Why It Works

11. How to Manage Your Containers

12. How to Create a Private Docker Repository

    • Step 1: Get Your AWS Access Keys

    • Step 2: Check if AWS CLI is Installed

    • Step 3: Configure AWS CLI

    • Step 4: Test Your AWS Configuration

    • Step 5: Login to ECR (Docker Registry)

    • Understanding Image Naming in Docker Repositories

    • Step 6: Build, Tag, and Push Your Image

13. Assignment: Create and Push a New Version

    • Deploying Our Image

    • Why Must We Use the Full Image URL for ECR?

    • Deploy Your App Using Docker Compose

    • Sharing Our Private Docker Image

14. Docker Volumes

    • How Docker Volumes Work

    • Types of Docker Volumes

    • Example Docker Compose File Using Volumes

    • Start Your Application

15. Conclusion

What is a Container?

A container is a way to package an application together with everything it needs, including its dependencies, libraries, and configuration files.

Because containers are portable, they can be shared across teams and deployed on any machine without worrying about compatibility.

Where Do Containers Live?

Since containers are portable and can be shared across teams and systems, they need a place to live. That’s where container repositories come in – special storage locations for containers. Organizations can have private repositories for internal use, while public ones like Docker Hub let anyone browse and use shared containers.

If you visit the catalog page on Docker Hub, you will see a variety of container repositories, both official and community-made, from developers and teams like Redis, Jenkins, and many others.

In the past, when multiple developers worked on different projects, each had to manually install services on their own systems. Since different developers often use different operating systems like Linux, macOS, and Windows, the setup process was never the same. It took a lot of time, led to plenty of errors, and made setting up new environments a real headache, especially when you had to repeat it for multiple services.

Docker changed the game for developers and teams. Instead of manually installing every service and dependency, you can just run a single Docker command to start a container. Each container has its own isolated environment with everything it needs, so it runs the same on any machine, no matter if it’s Windows, macOS, or Linux. This makes collaboration smoother and eliminates all the bottlenecks that come from different setups, missing dependencies, or version mismatches.

In short, Docker is a platform that packages your app and its dependencies into a single, portable container, so it runs the same way everywhere.

Docker vs Virtual Machines

Docker and virtual machines (VMs) are both ways to run apps in a “virtual” environment, but they work differently. To understand the differences, it helps to know a bit about how computers run software.

A quick look at the layers:

• Kernel: This is the part of the operating system that talks to your computer’s hardware, like the CPU, memory, and disk. Think of it as the middleman between your apps and your computer.

• Application layer: This is where programs and apps run. It sits on top of the kernel and uses it to access hardware resources.

So, now let’s get into a bit more detail about Virtual Machines. A VM virtualizes the entire operating system, which means it comes with its own kernel and its own application layer. When you download a VM, you are basically getting a full OS inside your computer, often several gigabytes in size.

Because it has to boot its own OS, VMs start slowly. But VMs are very compatible, and can run on almost any host because they include everything they need.

Docker, on the other hand, only virtualizes the application layer, not the full OS. Containers share the host system’s kernel but include everything the app needs, dependencies, libraries, and configuration.

Docker images are small, often just a few megabytes. Containers start almost instantly because they don’t boot a full OS. A Docker container can run anywhere Docker is installed, no matter what operating system your computer uses.

In simple terms, to summarize:

• A VM is like running a whole computer inside your computer – big, heavy, and slow.

• A Docker container is like a self-contained app package – small, fast, and portable.

Here’s a quick comparison:

Docker Installation

Alright, now that you know what Docker is, let’s get it running on your own machine.

Docker works on Windows, macOS, and Linux, but each system has slightly different steps. The official Docker documentation has clear instructions for all operating systems under Docker Docs: Install Docker.

If you are more of a visual learner, this YouTube video walks you through installing Docker on Windows and Linux step by step: Watch here.

Here is a simple roadmap:

First, check your system requirements. Docker won’t run on every computer, so make sure your OS version is supported (the official docs have a checklist).

1. Windows and macOS users:

   • Newer systems: Download and install Docker Desktop. It’s the easiest way to get started.

   • Older systems: If your computer doesn’t support Docker Desktop (for example, missing Hyper-V or older OS versions), you can use Docker Toolbox. Toolbox installs Docker using a lightweight virtual machine, so you can still run containers even on older machines.

2. Linux users: You will usually install Docker through your package manager (apt for Ubuntu/Debian, yum for CentOS/Fedora, etc.). The official docs show the commands for your distro.

Then verify your installation: Open a terminal or command prompt and type:

docker --version

If you see the Docker version displayed, congratulations! Docker is ready to go.

Once Docker is installed, you’ll be ready to start running containers, pulling images, and experimenting with your apps in a safe, isolated environment.

Tip for beginners:

If you’re on an older machine and using Docker Toolbox, commands are mostly the same, but you will run them inside the Docker Quickstart Terminal, which sets up the virtual machine for you.

Basic Docker Commands

So far, we have been throwing around terms like images and containers, sometimes even interchangeably. But there is an important difference:

• Docker image: Think of an image as a blueprint or a package. It contains everything your app needs: the code, libraries, dependencies, and configuration, but it’s not running yet.

• Docker container: A container is a running instance of an image. When you start a container, Docker takes the image and runs it in its own isolated environment.

A helpful way to remember it is this: the image is the recipe, while the container is the cake**.** You can have one recipe (image) and make multiple cakes (containers) from it.

Important note: Docker Hub stores images, not containers. So when you pull something from Docker Hub, you’re downloading an image. For example:

docker pull redis

Here’s what you’ll see:

This command downloads the Redis image to your machine. Once the download is complete, you can see all the images you have locally with:

docker images

From there, you can start a container from an image whenever you need it:

docker run -d --name my-redis redis

This command starts a container, my-redis, from the redis image you just pulled.

• docker run tells Docker to start a new container from an image.

• -d stands for “detached mode.” It means the container runs in the background so you can keep using your terminal.

• --name my-redis gives your container a friendly name (my-redis) instead of letting Docker assign a random one. It makes it easier to manage later.

• redis is the image you are using to start the container.

To see all containers that are currently running, you can use:

docker ps

This will list containers with details like:

• Container ID

• Name

• Status (running or stopped)

• The image it’s running from

If you want to see all containers, even ones that aren’t running, you can add the -a flag:

docker ps -a

How to Specify a Version of an Image:

By default, Docker pulls the latest version of an image. But sometimes you might need a specific version. You can do this using a colon (:) followed by the version tag. For example:

docker pull redis:7.2
docker run -d --name my-redis redis:7.2

To know which versions are available, you can visit Docker Hub or check the image tags online. Also, running docker images on your machine will show you all downloaded images and their versions.

How to Stop, Start, and Remove a Container

If you want to stop a running container, run this:

docker stop my-redis

To start it again:

docker start my-redis

You can also remove a container if you no longer need it:

docker rm my-redis

How to Restart a Container

You can restart a container using its container ID (or name) if something crashes, needs a refresh, or you just want to apply changes.

For example:

docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS      NAMES
c002bed0ae9a   redis     "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   6379/tcp   my-redis

Restart it like this:

docker restart c002bed0ae9a

or by name:

docker restart my-redis

Other handy ways:

• Stop then start

    docker stop c002bed0ae9a
    docker start c002bed0ae9a
  

• Start with logs

    docker start c002bed0ae9a && docker logs -f c002bed0ae9a
  

How to Run Multiple Redis Containers and Understanding Ports

Right now, you have a Redis container running:

docker ps

It shows something like this:

CONTAINER ID   IMAGE     COMMAND                  STATUS          PORTS      NAMES
c002bed0ae9a   redis     "docker-entrypoint.s…"   Up 20 minutes   6379/tcp   my-redis

Notice the PORTS column: 6379/tcp. This means the container is running Redis on its internal port 6379. By default, this port is inside the container and is not automatically exposed to your computer (the host). Docker maps it only if you specify it.

Trying to Run Another Redis Container on the Same Port

If you try:

docker run -d --name my-redis2 redis:7.4.7-alpine

It will fail to map the host port 6379 because the first container is already using it. This is where port binding comes in.

What is Port Binding?

Port binding (also called port mapping) is the mechanism Docker uses to connect a port inside a container to a port on your host machine (your laptop/desktop/server).

Without port binding, any service running inside a container is completely isolated: it can listen on its internal ports (for example, Redis on 6379, a Node.js app on 3000, MongoDB on 27017), but nothing outside the container, including your browser, another app on your computer, or even another container on a different network, can reach it.

• Container Port: The port inside the container where the app is running (Redis defaults to 6379).

• Host Port: The port on your computer that you want to use to access that container.

Docker lets you map a container port to a different host port using the -p flag.

Running a Second Redis Container on a Different Host Port

docker run -d --name my-redis2 -p 6380:6379 redis:7.4.7-alpine

-p 6380:6379 maps host port 6380 to container port 6379.

• Now you can connect to Redis in the second container using localhost:6380.

• Inside the container, Redis still runs on port 6379.

Check both containers:

docker ps

Output will look like this:

CONTAINER ID   IMAGE     STATUS          PORTS             NAMES
c002bed0ae9a   redis     Up 20 minutes   6379/tcp          my-redis
d123abcd5678   redis     Up 1 minute     0.0.0.0:6380->6379/tcp   my-redis2

The first container is running internally on 6379 (host port not exposed), while the second container is mapped so host port 6380 forwards traffic to container port 6379.

Think of each container as a room with a phone line (container port).

• You want to call that room from the outside (host).

• You can’t use the same external phone line for two rooms at the same time.

• With port binding, you assign a different external line for each room, even if the internal phone number is the same.

Why Port Binding Exists

1. Avoid port conflicts on the host: Only one process on your computer can use a given port at a time. If you already have one Redis container using host port 6379, a second container cannot also bind to the same host port. Port binding lets you run many identical containers side-by-side by mapping each one to a different host port (6379 → 6380, 6381, etc.).

2. Access containerised services from your host: Your browser, Postman, MongoDB Compass, redis-cli, curl, etc., all run on the host. Without -p, they have no way to talk to services inside containers.

3. Selective exposure: You don’t have to expose every port a container uses. Only map the ports you actually need externally, keeping the rest private and secure.

It also gives you more flexibility in development and production. In development, you might map container 3000 to host 3000. But in production (for example, behind a reverse proxy), you might map container 3000 to host 80 or 443, or not expose it at all and let another container talk to it over Docker’s internal network.

How to Explore a Container

To explore a container, run:

docker exec -it my-redis2 /bin/sh

• docker exec runs a command in the container.

• -it interactive terminal (lets you type and see output).

• /bin/sh starts a shell inside the container.

Once inside, your prompt changes to something like:

/data #

Now you can list files, navigate directories, or run programs, all inside the container, without affecting your host machine.

docker run vs docker start

We have been using docker run and docker start throughout this article, but here’s why the difference is important:

• Avoid accidental duplicates: Using docker run every time creates a new container. If you just want to restart something you already set up, docker start is faster and safer.

• Maintain configuration: docker start preserves the container’s original settings, ports, volumes, and names so you don’t risk breaking anything by changing options.

• Work efficiently with multiple containers: When running multiple services or different versions of the same app, knowing when to run vs start helps you manage resources, avoid port conflicts, and keep your workflow smooth.

• Speed up your workflow: Starting existing containers is almost instant, while creating a new one takes slightly longer.

Bottom line docker run = create something new, while docker start = resume what you already have.

Practice with JavaScript

Now that we have covered the core Docker concepts, let’s put them into action. In this section, we’ll containerize a simple JavaScript project that consists of:

• A frontend: Built with HTML, CSS, and JavaScript

• A backend: A simple Node.js server (server.js)

• A database: A MongoDB instance pulled directly from Docker Hub

• A UI for MongoDB: Using Mongo Express to visualize and manage our database

This example demonstrates how Docker can manage multiple components of an application, including code, dependencies, and services in isolated, consistent environments.

You can pull the starter project from GitHub here.

Or clone it directly using your terminal:

git clone https://github.com/Oghenekparobo/docker_tut_js.git
cd docker_tut_js

This contains the basic HTML and JavaScript files along with the Node.js backend.

Next, we will prepare to set up our database. Head over to Docker Hub and type “mongo” in the search box. You will see the official MongoDB image published by Docker.

How to Pull the MongoDB Image

Now that you have explored the official MongoDB image on Docker Hub, let’s actually pull it into your local environment.

Open your terminal, navigate to your project directory (for example, docker_tut_js), and run:

docker pull mongo

This command tells Docker to download the latest version of the MongoDB image from Docker Hub.

You will see output similar to this:

Using default tag: latest
latest: Pulling from library/mongo
b8a35db46e38: Already exists 
a637dbfff7e5: Pull complete 
0c9047ace63c: Pull complete 
02cd4cf70021: Pull complete 
dfb5d357a025: Pull complete 
007bf0024f67: Pull complete 
67fd8af3998d: Pull complete 
d702312e8109: Pull complete 
Digest: sha256:7d1a1a613b41523172dc2b1b02c706bc56cee64144ccd6205b1b38703c85bf61
Status: Downloaded newer image for mongo:latest
docker.io/library/mongo:latest

Here’s what’s happening:

• “Using default tag: latest”: Docker pulls the most recent version of MongoDB since no specific version was provided.

• “Pulling from library/mongo”: It’s downloading from Docker’s official image library.

• “Pull complete”: Each line represents a layer of the image being successfully downloaded.

• “Downloaded newer image for mongo:latest”: Confirms that the MongoDB image is now stored locally on your system.

You can confirm that it’s available by running:

docker images

You should see mongo listed in the repository column.

How to Pull the Mongo Express Image

Now that the MongoDB image is ready, let’s pull the Mongo Express image.

Mongo Express is a lightweight web-based interface that lets you view and manage your MongoDB collections through a browser, similar to how phpMyAdmin works for MySQL.

Open your terminal (still in your project directory) and run:

docker pull mongo-express

You’ll see output similar to this:

Using default tag: latest
latest: Pulling from library/mongo-express
b8a35db46e38: Already exists
a637dbfff7e5: Pull complete
4e0e0977e9c3: Pull complete
02cd4cf70021: Pull complete
Digest: sha256:3d6dbac587ad91d0e2eab83f09a5b31a1c8f9d91a8825ddaa6c7453c25cb4812
Status: Downloaded newer image for mongo-express:latest
docker.io/library/mongo-express:latest

Here’s what this means:

• docker pull mongo-express downloads the official Mongo Express image from Docker Hub.

• Each “Pull complete” line represents a successfully downloaded layer of the image.

• mongo-express:latest confirms that the latest version is now stored locally.

To verify that both images are available, run:

docker images

You should see mongo and mongo-express listed in the output.

Now that both images are downloaded, the next step is to run the containers to make sure MongoDB is up and accessible, and then connect it to Mongo Express so we can manage it through the browser.

Before we do that, let’s briefly look at how these two containers will communicate.

Docker Network

When MongoDB and Mongo Express run in separate containers, they need a way to talk to each other. Docker handles this using something called a Docker Network, a virtual bridge that lets containers communicate securely without exposing internal ports to the outside world.

When you run containers in Docker, it automatically creates an isolated network for them. Think of it like a private space where your containers can talk to each other safely without exposing everything to the outside world.

For example, if our MongoDB container and Mongo Express container are on the same Docker network, they can communicate just by using their container names (like mongo or mongo-express). You don’t need to use localhost or port numbers, as Docker handles that part internally.

But anything outside the Docker network (like your host machine or a Node.js app) connects through the exposed ports.

So later, when we package our entire application, the Node.js backend, MongoDB, Mongo Express, and even the frontend (index.html) into Docker, all these containers will interact smoothly through the Docker network. The browser on your computer will then connect to your Node.js app using the host address and port we have exposed.

By default, Docker already provides a few built-in networks. You can see them by running:

docker network ls

You will get something like this:

NETWORK ID     NAME      DRIVER    SCOPE
712a7144f1a0   bridge    bridge    local
4ae27eedea5b   host      host      local
4806000201ce   none      null      local

These are automatically created by Docker. You don’t need to worry too much about them right now – we will just focus on creating our own custom network.

For our setup, we will create a separate network that both MongoDB and Mongo Express can share. Let’s call it mongo-network:

docker network create mongo-network

How to Run the Mongo Container

To make sure our MongoDB and Mongo Express containers can communicate, we need to run them inside the same Docker network. That’s why we created mongo-network earlier.

Let’s start with MongoDB. Remember, the docker run command is used to start a container from an image. In this case, we will run the official MongoDB image and attach it to our network.

We will also expose the default MongoDB port 27017 so it’s accessible from outside the container, and set up environment variables for the root username and password.

Here is the command:

docker run -p 27017:27017 -d \
  -e MONGO_INITDB_ROOT_USERNAME=admin \
  -e MONGO_INITDB_ROOT_PASSWORD=password \
  --name mongo \
  --network mongo-network \
  mongo

Here’s what each part does:

• -p 27017:27017 maps the container’s MongoDB port to your host machine.

• -d runs the container in detached mode (in the background).

• -e sets environment variables for the database’s root credentials.

• --name mongo gives the container a custom name for easier reference.

• --network mongo-network connects the container to the network we created.

Once it runs successfully, your MongoDB instance will be up and running inside the Docker network, ready for other containers like Mongo Express to connect to it.

After creating your MongoDB container, you can easily check if it’s running and healthy.

First, run docker ps to see all active containers. You should see your MongoDB container (mongo) listed with its port 27017 exposed. To get more details about what’s happening inside the container, you can check its logs using docker logs mongo or, if you prefer, by using the container ID (for example: docker logs 7abb38175ae28). The logs will show startup messages from MongoDB, and you should look for lines indicating that the database started successfully and is ready to accept connections.

This is a quick way to verify that everything is working correctly before connecting other services, like Mongo Express, to it.

docker ps

This will list all running containers. You should see your MongoDB container (mongo) with its port 27017 exposed.

docker logs mongo or the id of the container e.g docker logs 7abb38175ae283429354609866c8d97521f37b535c475ae448295f8fc0ed947f

This will show startup messages. Look for lines indicating MongoDB started successfully and is ready to accept connections.

How to Run the Mongo Express Container

Now that MongoDB is up and running, we can run Mongo Express, which is a web-based interface to manage and view your MongoDB databases. We will connect it to the same network (mongo-network) so it can communicate with MongoDB.

Here’s the command:

docker run -d \
  -e ME_CONFIG_MONGODB_ADMINUSERNAME=admin \
  -e ME_CONFIG_MONGODB_ADMINPASSWORD=password \
  -e ME_CONFIG_MONGODB_SERVER=mongo \
  --name mongo-express \
  --network mongo-network \
  -p 8081:8081 \
  mongo-express

Here’s what each part does:

• -d runs the container in detached mode (in the background).

• -e ME_CONFIG_MONGODB_ADMINUSERNAME=admin sets the MongoDB admin username for Mongo Express to use.

• -e ME_CONFIG_MONGODB_ADMINPASSWORD=password sets the corresponding MongoDB password.

• -e ME_CONFIG_MONGODB_SERVER=mongo tells Mongo Express which MongoDB server to connect to. Here we use the container name mongo because both containers are on the same network.

• --name mongo-express gives the container a friendly name for easier reference.

• --network mongo-network connects the container to the same Docker network as MongoDB so they can talk to each other.

• -p 8081:8081 exposes the Mongo Express web interface on port 8081 of your host machine.

• mongo-express the name of the Docker image we’re running.

Once the container is running, you can open your browser and visit http://localhost:8081 to access Mongo Express and interact with your MongoDB instance.

For more details about the available environment variables and options, you can check the official Docker Hub page for Mongo Express here.

Before opening your browser at http://localhost:8081, it’s a good idea to check if the Mongo Express container is running properly. You can do this by viewing its logs:

docker logs <container-id>
# or
docker logs mongo-express

You should see output similar to this:

Waiting for mongo:27017...
No custom config.js found, loading config.default.js
Welcome to mongo-express 1.0.2
------------------------
Mongo Express server listening at http://0.0.0.0:8081
Server is open to allow connections from anyone (0.0.0.0)
basicAuth credentials are "admin:pass", it is recommended you change this in your config.js!

This confirms that Mongo Express is up and running and ready to connect to your MongoDB instance.

Take note of the basicAuth credentials shown in the logs (admin:pass). If these credentials are present, you’ll need to use them when accessing Mongo Express from your browser. Later, you can change them in a custom config.js file for better security.

Once everything looks good in the logs, you can safely visit http://localhost:8081 to access the Mongo Express interface.

If your browser asks for a username and password when accessing Mongo Express, use the basicAuth credentials shown in the container logs:

Username: admin
Password: pass

These are the default credentials, and it’s strongly recommended to change them later in a custom config.js file for better security.

When you open Mongo Express, you will notice some default databases already created. For this project, we will create a new database called todos. Once it’s created, your Node.js application can connect to this database to store and retrieve data.

How to Connect Node.js to MongoDB

You already have MongoDB running inside a Docker container (mongo). The container exposes the default MongoDB port 27017 to the host, so any process on your laptop/desktop can reach it via localhost:27017.

Important: The Node.js app is outside Docker (it’s just a regular node server.js process you start from your terminal).

Because the app is external, we must use localhost (or 127.0.0.1) as the host name – not the container name mongo.

Once we later containerise the Node.js app and put it on the same Docker network, we’ll switch the host to mongo. For now, keep it localhost.

Node.js Backend

Here’s a version of our server.js using MongoDB:

const express = require("express");
const multer = require("multer");
const path = require("path");
const fs = require("fs");
const { MongoClient, ObjectId } = require("mongodb");

const app = express();
const PORT = 3000;

// Host = localhost  →  talks to the MongoDB container via the exposed port
// Port = 27017      →  default MongoDB port
// User / Pass       →  admin / password (the credentials you gave the container)
const mongoUrl = "mongodb://admin:password@localhost:27017";
const dbName = "todos";
let db;

MongoClient.connect(mongoUrl)
  .then((client) => {
    db = client.db(dbName);
    console.log("Connected to MongoDB →", dbName);
  })
  .catch((err) => console.error("MongoDB connection error:", err));

const uploadDir = path.join(__dirname, "uploads");
if (!fs.existsSync(uploadDir)) fs.mkdirSync(uploadDir);

const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, uploadDir),
  filename: (req, file, cb) => {
    const unique = Date.now() + "-" + Math.round(Math.random() * 1e9);
    cb(null, "photo-" + unique + path.extname(file.originalname));
  },
});
const upload = multer({ storage });

app.use(express.static(__dirname));
app.use("/uploads", express.static(uploadDir));
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.get("/todos", async (req, res) => {
  const todos = await db.collection("todos").find().toArray();
  res.json(todos);
});

app.post("/todos", upload.single("photo"), async (req, res) => {
  const text = req.body.text?.trim();
  if (!text) return res.status(400).json({ error: "Text required" });

  const todo = {
    text,
    image: req.file ? `/uploads/${req.file.filename}` : null,
    createdAt: new Date(),
  };

  const result = await db.collection("todos").insertOne(todo);
  todo._id = result.insertedId;
  res.json(todo);
});

// Start server
app.listen(PORT, () => {
  console.log(`Server → http://localhost:${PORT}`);
});

Frontend

index.html:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Todo + Image</title>
    <style>
      body {
        font-family: sans-serif;
        margin: 2rem;
        max-width: 800px;
      }
      .todo {
        border: 1px solid #ccc;
        padding: 1rem;
        margin-bottom: 1rem;
        border-radius: 8px;
      }
      .todo img {
        max-height: 150px;
        margin-top: 0.5rem;
      }
      .error {
        color: red;
      }
      input[type="text"] {
        width: 100%;
        padding: 0.5rem;
        margin-bottom: 0.5rem;
      }
      #preview {
        max-width: 300px;
        margin-top: 0.5rem;
        display: none;
      }
    </style>
  </head>
  <body>
    <h1>Todo List with Images</h1>

    <div id="addForm">
      <input type="text" id="textInput" placeholder="What needs to be done?" />
      <input type="file" id="imageInput" accept="image/*" />
      <img id="preview" alt="preview" />
      <button id="addBtn">Add Todo</button>
      <p id="status"></p>
    </div>

    <h2>Todos</h2>
    <div id="todos"></div>

    <script>
      const $ = document.querySelector.bind(document);

      const textInput = $("#textInput");
      const imageInput = $("#imageInput");
      const preview = $("#preview");
      const addBtn = $("#addBtn");
      const status = $("#status");
      const todosDiv = $("#todos");

      imageInput.addEventListener("change", () => {
        const file = imageInput.files[0];
        if (!file) {
          preview.style.display = "none";
          return;
        }
        const reader = new FileReader();
        reader.onload = (e) => {
          preview.src = e.target.result;
          preview.style.display = "block";
        };
        reader.readAsDataURL(file);
      });

      addBtn.addEventListener("click", async () => {
        const text = textInput.value.trim();
        if (!text) {
          status.textContent = "Please enter a todo text.";
          status.className = "error";
          return;
        }

        const form = new FormData();
        form.append("text", text);
        if (imageInput.files[0]) form.append("photo", imageInput.files[0]);

        try {
          const res = await fetch("/todos", { method: "POST", body: form });
          const data = await res.json();
          if (!res.ok) throw new Error(data.error || "failed");
          status.textContent = "Todo added!";
          status.className = "";
          textInput.value = "";
          imageInput.value = "";
          preview.style.display = "none";
          loadTodos(); // refresh list
        } catch (err) {
          status.textContent = "Error: " + err.message;
          status.className = "error";
        }
      });

      async function loadTodos() {
        const res = await fetch("/todos");
        const todos = await res.json();
        todosDiv.innerHTML = "";
        todos.forEach((t) => {
          const div = document.createElement("div");
          div.className = "todo";
          div.innerHTML = `<strong>${escapeHtml(t.text)}</strong>`;
          if (t.image) {
            div.innerHTML += `<br><img src="${t.image}" alt="todo image">`;
          }
          todosDiv.appendChild(div);
        });
      }

      function escapeHtml(s) {
        const div = document.createElement("div");
        div.textContent = s;
        return div.innerHTML;
      }

      loadTodos();
    </script>
  </body>
</html>

Now your Node.js app can connect to the MongoDB container running in Docker. Since the app is running outside Docker for now, it connects through localhost:27017 using the credentials you set (admin / password).

Once connected, your Node.js backend stores and retrieves todos directly from the todos database in MongoDB, replacing the in-memory array. Later, if you containerize the Node.js app and put it on the same Docker network as MongoDB, you can switch the host from localhost to the container name mongo. we are getting there

You can get the full backend and frontend code ready to run and tweak it for your setup here: GitHub repo.

How to Use Docker Compose

So we now have our Node.js app connected to MongoDB and Mongo Express, both running inside containers. We’ve created the network, started the containers, and everything is talking to each other perfectly.

But let’s be honest: typing out all those long docker run commands every time can get tedious. You probably want a simpler, cleaner way to spin everything up with just one command. That’s where Docker Compose comes in.

Docker Compose is a tool that lets you define and run multi-container applications with a single command. Instead of manually running multiple docker run commands, you describe your setup in a simple docker-compose.yml file, specifying each service (like your Node.js app, MongoDB, and Mongo Express), their configurations, environment variables, and shared networks.

Basically, it lets you manage multiple containers as one project, easy to start, stop, and maintain with a single file and a single command.

The standard naming convention is docker-compose.yml (or docker-compose.yaml. Both work, but .yml is more common).

Docker automatically detects it when you run:

docker compose up

So yeah, stick with docker-compose.yml for convention.

Now, to run the containers for MongoDB and Mongo Express, we can use the following two commands, respectively:

# MongoDB container
docker run -p 27017:27017 -d \
  -e MONGO_INITDB_ROOT_USERNAME=admin \
  -e MONGO_INITDB_ROOT_PASSWORD=password \
  --name mongo \
  --network mongo-network \
  mongo

# Mongo Express container
docker run -d \
  -e ME_CONFIG_MONGODB_ADMINUSERNAME=admin \
  -e ME_CONFIG_MONGODB_ADMINPASSWORD=password \
  -e ME_CONFIG_MONGODB_SERVER=mongo \
  --name mongo-express \
  --network mongo-network \
  -p 8081:8081 \
  mongo-express

Now, instead of typing these long commands every time, we will combine them and run everything at once using a Docker Compose file.

The docker-compose.yml file will be located at the root of our Node.js project.

Here’s how our docker-compose.yml file looks:

version: "3.8"

services:
  mongodb:
    image: mongo
    container_name: mongo
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: password

  mongo-express:
    image: mongo-express
    container_name: mongo-express
    ports:
      - "8081:8081"
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: admin
      ME_CONFIG_MONGODB_ADMINPASSWORD: password
      ME_CONFIG_MONGODB_SERVER: mongodb
    depends_on:
      - mongodb

Let’s break down what’s going on here:

• version: "3.8": This defines the Compose file version. Each version has slightly different syntax rules and features. Version 3.8 is modern and works with the latest Docker Engine.

• services:: All the containers we want to run are defined here. In our case, two services: mongodb and mongo-express.

MongoDB service:

• image: mongo pulls the official MongoDB image from Docker Hub.

• container_name: mongo gives the container a friendly name.

• ports: "27017:27017" exposes MongoDB’s default port to our host, so Node.js or other apps can connect.

• environment: sets up the root username and password for MongoDB.

Mongo Express service:

• image: mongo-express is the official Mongo Express image.

• container_name: mongo-express is a friendly name for easier reference.

• ports: "8081:8081" exposes Mongo Express web interface on host port 8081.

• environment: let’s Mongo Express know how to connect to MongoDB (username, password, host).

• depends_on: - mongodb ensures MongoDB starts first, so Mongo Express can connect immediately.

Why Use Docker Compose?

• Single command: Instead of running multiple long docker run commands, just run:

docker compose up -d

• Automatic networking: Compose creates a default network so services can communicate using their service names (mongodb In our case)

• Easier maintenance: You can stop, start, or rebuild all services with simple commands.

Before we run our new docker-compose.yml, it’s important to make sure no conflicting containers are running. Remember, we already had MongoDB and Mongo Express running from the previous docker run commands.

To avoid conflicts (like ports already in use), we should stop and remove any running containers first.

Here’s how:

# List all running containers
docker ps

# Stop a specific container (replace <container_name> with mongo or mongo-express)
docker stop mongo
docker stop mongo-express

# Remove the stopped containers
docker rm mongo
docker rm mongo-express

# Optional: stop and remove all running containers at once
docker stop $(docker ps -q)
docker rm $(docker ps -a -q)

• docker ps shows currently running containers.

• docker stop <name> stops a container gracefully.

• docker rm <name> removes the container from Docker.

• docker stop $(docker ps -q) stops all running containers.

• docker rm $(docker ps -a -q) removes all containers (running or stopped).

Once all previous containers are stopped and removed, we’re ready to run our Docker Compose setup safely without conflicts.

Now that all previous containers are stopped, we can start MongoDB and Mongo Express together using our docker-compose.yml file.

From the root of your Node.js project (where the docker-compose.yml file is located), run:

docker compose up -d

Here’s what this does:

• docker compose tells Docker to use Compose.

• up builds (if needed) and starts all the services defined in the Compose file.

• -d runs the containers in detached mode, meaning they run in the background.

After running this command, Docker will start both MongoDB and Mongo Express, connect them on the same internal network, and expose the ports we defined (27017 for MongoDB and 8081 for Mongo Express).

If everything worked correctly, after running:

docker compose up -d

You should see output similar to this:

[+] Running 3/3
 ✔ Network docker_tut_default  Created                                                                                               0.0s 
 ✔ Container mongo             Started                                                                                               0.6s 
 ✔ Container mongo-express     Started                                                                                               0.8s 
stephenjohnson@Oghenekparobo docker_tut %

What this means:

• Network docker_tut_default Created: Docker Compose automatically creates a network for your services so they can communicate with each other.

• Container mongo Started: Your MongoDB container is running.

• Container mongo-express Started: Your Mongo Express container is running.

You can confirm that the containers are running by using:

docker ps

This will list all active containers. You should see both mongo and mongo-express with their respective ports (27017 for MongoDB and 8081 for Mongo Express) exposed.

• To access Mongo Express, open your browser and go to http://localhost:8081 to interact with MongoDB through the web interface.

• To access MongoDB, your Node.js app can connect to MongoDB at localhost:27017 using the credentials you set in the Compose file.

Compared to running long docker run commands for each container, using Docker Compose is easier because:

• Starts multiple containers with one command.

• Automatically sets up networking between containers.

• Makes it easier to stop, remove, or rebuild containers later.

In short, Docker Compose simplifies and organizes everything, making it much easier to manage your development environment.

At this stage, it’s important to know that any data you add to MongoDB is temporary. If you stop or remove your containers and then start them again, you will notice that all your data is gone. This happens because data inside a container isn’t persistent by default.

Don’t worry, this is expected, and we’ll cover how to make data persistent later in the tutorial when we introduce Docker volumes. For now, just be aware that each time you restart your containers, MongoDB starts fresh with no previous data.

You can get a full sample, including the Dockerfile and the docker‑compose file, here.

How to Build Our Own Docker Image

Now that we have tested our Node.js application locally and seen it working perfectly with MongoDB and Mongo Express, the next step is preparing it for deployment.

Running the app directly on our machine works fine for development, but it’s not practical when we want to move it to another environment or server. By creating a Docker image, we can package the application together with all its dependencies, configuration, and environment setup into a single, portable unit. This image can then run anywhere Docker is installed, ensuring our app works the same way across development, testing, and production.

In short, building a Docker image is how we containerize our app and make it deployment-ready.

In order to containerize our Todo app, we need a Dockerfile. A Dockerfile is essentially a blueprint that tells Docker how to build an image for our application. It defines the base environment, copies our application code, installs dependencies, and specifies how the app should start. With this blueprint, Docker can create a consistent image that behaves the same way on any machine, making our Node.js app fully portable and ready for deployment.

In our Dockerfile, notice the capital D, which is the standard naming convention. Place this file in the root directory of your Node.js project. In simple projects like ours, our main app file (like server.js or index.js) is usually in the root too, along with package.json. Docker will use this file as a blueprint to build a container image of your application.

If your main app file is inside a subfolder, that’s fine too. Just make sure the Dockerfile’s COPY and CMD commands point to the correct location. The important thing is that the Dockerfile lives in the root so Docker knows where to start building your app.

Here’s how the contents of our Dockerfile look:

# Use full Node 18 (Debian-based)
FROM node:18

# Set environment variables
ENV MONGO_DB_USERNAME=admin \
    MONGO_DB_PASSWORD=password

# Set working directory
WORKDIR /home/app

# Copy package files
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy source code
COPY . .

# Expose port
EXPOSE 3000

# Start the app
CMD ["node", "server.js"]

Let’s see what’s going on here:

• FROM node:13-alpine is the base image for our container. It comes with Node.js installed and is very lightweight, keeping the image small.

• ENV MONGO_DB_USERNAME=admin \ MONGO_DB_PASSWORD=password sets environment variables inside the container so the Node.js app can connect to MongoDB.

• WORKDIR /home/app sets the working directory inside the container. All subsequent commands like COPY or RUN will run relative to this folder.

• COPY . . copies all files from your local project into the container’s working directory. This includes your server.js, package.json, and any other files needed to run the app.

• RUN npm install installs all the Node.js dependencies listed in package.json inside the container.

• EXPOSE 3000 tells Docker that the container will listen on port 3000, which is the port our Node.js app runs on.

• CMD ["node", "server.js"] defines the command that runs when the container starts, which launches our Node.js server.

By placing this Dockerfile in the root of your project, Docker knows exactly where to find your app’s files and dependencies. When we build the image, it packages everything inside a portable container that can run anywhere Docker is installed, making deployment straightforward and consistent.

Now that we have our Dockerfile ready, the next step is to build the Docker image for our Node.js app.

To build the image, open your terminal, make sure you are in the root directory of your project (where the Dockerfile is), and run:

docker build -t todo-app:1.0 .

• todo-app is the name of your image.

• :1.0 is the version tag (you can use any versioning scheme, like 1.0, v1, latest, etc.).

• . tells Docker to use the current folder (root of your project) as the build context.

After running:

docker build -t todo-app:1.0 .

Docker reads your Dockerfile, packages your Node.js app with all its dependencies, and creates a Docker image. You can confirm the image exists by running:

docker images

You should see output like this:

REPOSITORY      TAG       IMAGE ID       CREATED          SIZE
todo-app        1.0       d85dd4ed97f9   45 seconds ago   147MB
mongo           latest    1d659cebf5e9   2 weeks ago      894MB
mongo-express   latest    1133e12468c7   20 months ago    182MB

This shows that your todo-app image has been created successfully, alongside the images for MongoDB and Mongo Express.

Running Your Node.js App Container

Now that the image exists, the next step is to run a container from it. A container is basically a running instance of your image. To do this:

docker run todo-app:1.0

Here’s what this command does:

• docker run starts a new container from the image.

• todo-app:1.0 tells Docker which image to use (the one we just built).

Once this runs, your Node.js app will be live inside a container, separate from your local environment. You can open your browser at http://localhost:3000 and see your Todo app working just like it did locally.

To see all running containers, use:

docker ps

You’ll see something like:

CONTAINER ID   IMAGE           COMMAND         CREATED       STATUS       PORTS                  NAMES
d85dd4ed97f9   todo-app:1.0    "node server.js"  10s ago      Up 10s       0.0.0.0:3000->3000/tcp   awesome_todo

This confirms your container is running. If you ever need to stop it:

docker stop <container-id>

Troubleshooting Errors

We started facing some issues here: when you run docker run todo-app:1.0 You'll see an error like this:

Server → http://localhost:3000 
MongoDB connection error: MongoServerSelectionError: getaddrinfo ENOTFOUND mongodb
    at Topology.selectServer (/home/app/node_modules/mongodb/lib/sdam/topology.js:346:38)
    ...
    [cause

especially when you try to perform an operation like creating a todo list.

The error getaddrinfo ENOTFOUND mongodb tells us that your Node.js container can't find MongoDB. Even though MongoDB is running in another container, your app container is isolated and doesn't know how to reach it.

Why This Happens:

Remember in our server.js, we connect to MongoDB using:

const mongoUrl = "mongodb://admin:password@localhost:27017";

The problem is with localhost. When you run your app locally on your machine (not in Docker), localhost works perfectly because MongoDB is running on the same machine. But when your app runs inside a Docker container, localhost refers to the container itself, not your host machine or other containers.

Think of it like this:

• Running locally: Your app and MongoDB are like two people in the same room, localhost works

• Running in Docker: Each container is like a separate room, localhost only refers to that specific room

The Solution

We need to change the MongoDB connection URL to use the Docker service name instead of localhost. Update your server.js file:

const mongoUrl = "mongodb://admin:password@localhost:27017";

To this:

const mongoUrl = "mongodb://admin:password@mongodb:27017";

Here's the complete updated server.js:

const express = require("express");
const multer = require("multer");
const path = require("path");
const fs = require("fs");
const { MongoClient, ObjectId } = require("mongodb");

const app = express();
const PORT = 3000;

// Host = localhost  →  talks to the MongoDB container via the exposed port
// Port = 27017      →  default MongoDB port
// User / Pass       →  admin / password (the credentials you gave the container)
const mongoUrl = "mongodb://admin:password@mongodb:27017";
const dbName = "todos";
let db;

MongoClient.connect(mongoUrl)
  .then((client) => {
    db = client.db(dbName);
    console.log("Connected to MongoDB →", dbName);
  })
  .catch((err) => console.error("MongoDB connection error:", err));

const uploadDir = path.join(__dirname, "uploads");
if (!fs.existsSync(uploadDir)) fs.mkdirSync(uploadDir);

const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, uploadDir),
  filename: (req, file, cb) => {
    const unique = Date.now() + "-" + Math.round(Math.random() * 1e9);
    cb(null, "photo-" + unique + path.extname(file.originalname));
  },
});
const upload = multer({ storage });

app.use(express.static(__dirname));
app.use("/uploads", express.static(uploadDir));
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.get("/todos", async (req, res) => {
  const todos = await db.collection("todos").find().toArray();
  res.json(todos);
});

app.post("/todos", upload.single("photo"), async (req, res) => {
  const text = req.body.text?.trim();
  if (!text) return res.status(400).json({ error: "Text required" });

  const todo = {
    text,
    image: req.file ? `/uploads/${req.file.filename}` : null,
    createdAt: new Date(),
  };

  const result = await db.collection("todos").insertOne(todo);
  todo._id = result.insertedId;
  res.json(todo);
});

// Start server
app.listen(PORT, () => {
  console.log(`Server → http://localhost:${PORT}`);
});

Why mongodb Works

The hostname mongodb matches the service name we defined in our docker-compose.yml:

services:
  mongodb:    # ← This is the hostname other containers use
    image: mongo
    container_name: mongo
    ...

When containers run in the same Docker Compose network, Docker provides an internal DNS that resolves service names to the correct container IP addresses. So when your app tries to connect to mongodb:27017, Docker automatically routes it to the MongoDB container.

Rebuild Your Docker Image

Now that we have updated the code, we need to rebuild the Docker image to include this change:

docker build -t todo-app:1.0 .
``

You should see output confirming the build completed successfully:
```
[+] Building 8.1s (10/10) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 443B
 ...
 => => naming to docker.io/library/todo-app:1.0

Add Your App to Docker Compose

Now update your docker-compose.yml file to include the todo-app service:

version: "3.8"

services:
  mongodb:
    image: mongo
    container_name: mongo
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: password

  mongo-express:
    image: mongo-express
    container_name: mongo-express
    ports:
      - "8081:8081"
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: admin
      ME_CONFIG_MONGODB_ADMINPASSWORD: password
      ME_CONFIG_MONGODB_SERVER: mongodb
    depends_on:
      - mongodb

  todo-app:
    image: todo-app:1.0
    container_name: todo-app
    ports:
      - "3000:3000"
    depends_on:
      - mongodb

The todo-app service includes:

• image: todo-app:1.0 that uses the Docker image we just rebuilt

• container_name: todo-app that gives the container a friendly name

• ports: "3000:3000" that exposes the app on port 3000

• depends_on: mongodb that ensures MongoDB starts before the app

Start All Services

First, stop any running containers:

docker compose down

If you have port 3000 running in your local system, then stop it (that is, free up port 3000).

We were running the server locally before, but now that we’ve built a Docker image, the app runs inside a container, so it’s no longer dependent on the local machine’s environment.

node server.js
Server → http://localhost:3000

Now stop it with Ctrl + C in that terminal. That’s it.

Then start everything together:

docker compose up -d
```

You should see:
```
[+] Running 4/4
 ✔ Network docker_tut_default  Created
 ✔ Container mongo             Started
 ✔ Container mongo-express     Started
 ✔ Container todo-app          Started

Verify Everything Works

Check that all containers are running:

docker ps
```

Expected output:
```
CONTAINER ID   IMAGE           COMMAND                  CREATED          STATUS          PORTS                      NAMES
a1b2c3d4e5f6   todo-app:1.0    "node server.js"         30 seconds ago   Up 28 seconds   0.0.0.0:3000->3000/tcp     todo-app
3d7c797fde1d   mongo-express   "/sbin/tini -- /dock…"   30 seconds ago   Up 29 seconds   0.0.0.0:8081->8081/tcp     mongo-express
4511ade73c38   mongo           "docker-entrypoint.s…"   30 seconds ago   Up 29 seconds   0.0.0.0:27017->27017/tcp   mongo
```

## Test Your Application

Now let's verify everything works:

### 1. Access Your Todo App
Open your browser and go to:
```
http://localhost:3000
```

### 2. Create Some Todos
Add a few todo items to test the functionality. Try uploading images too!

### 3. Verify in Mongo Express
Open Mongo Express:
```
http://localhost:8081

Navigate to the todos database, then the todos collection. You should see all the todos you just created with their complete data.

What Changed and Why It Works

Before the fix:

• Connection string used localhost:27017 ❌

• Container looked for MongoDB on itself

• Connection failed with ENOTFOUND error

After the fix:

• Connection string uses mongodb:27017 ✅

• Docker's internal DNS resolves mongodb to the MongoDB container

• Connection succeeds and data flows properly

This is a crucial lesson in Docker networking: containers communicate using service names, not localhost. Docker Compose automatically creates a network where all services can find each other by name.

How to Manage Your Containers

Here’s a quick overview of how to manage your containers once you have them up and running. You’ll typically use these common commands:

Stop all services:

docker compose down

View logs from your app:

docker compose logs todo-app

View logs in real-time:

docker compose logs -f todo-app

Rebuild after code changes:

docker build -t todo-app:1.0 .
docker compose up -d --force-recreate todo-app

Your application is now fully containerized and production-ready. All three services work together seamlessly, and you can deploy this entire stack anywhere Docker is supported with just the docker-compose.yml file and your built image.

Get the full updated code here.

How to Create a Private Docker Repository

Now we want to store our custom Docker image in a private container registry (instead of our local machine only). This gives you three major advantages:

1. Controlled access – Only people or servers you explicitly authorize can pull (or push) the image. Your code and dependencies stay private and secure.

2. Reliable distribution – Anyone (or any server) with the correct AWS credentials can pull the exact same image from anywhere in the world, eliminating “it works on my machine” problems.

3. Versioning and lifecycle management – You can keep multiple tagged versions (1.0, 2.0, latest, and so on) and easily roll back if needed.

The first step is to create a private Docker repository, also known as a container registry. In this case, we will use AWS Elastic Container Registry (ECR). Amazon ECR is a fully managed container registry that makes it easy to store, manage, share, and deploy your container images and artifacts securely from anywhere.

Once you’re on the home page, just click on the Create button. Name the repository the same as your image, todo-app, and then click Create to finalize the setup.

Don’t worry about the extra options – this isn’t an AWS tutorial.

Note: In AWS ECR, each image has its own repository, where we store the different tagged versions of that image.

Now, to push our image into the private repository, we need to do two things. First, we have to log in to the private repo. This is necessary because you’ll need authenticate yourself before AWS allows you to push anything. In other words, when you push your local image to the repo, you’re basically saying, “Yes, I have access to this registry. Here are my credentials.”

In our case, since we’re using AWS ECR, we will authenticate through AWS instead of typing our username and password manually.

Step 1: Get Your AWS Access Keys

To locate your access keys in the AWS console, follow these steps:

1. Log in to the AWS Console at https://console.aws.amazon.com

2. Click your account name (top right corner) and go to Security Credentials

3. Scroll down to "Access keys" section

4. If you don't have an access key:

   • Click "Create access key"

   • Select "Command Line Interface (CLI)"

   • Check the confirmation box and click Next

   • Add a description (optional) and click "Create access key"

5. IMPORTANT: Copy both the access key ID (looks like: AKIAIOSFODNN7EXAMPLE) and the secret access key (looks like: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY). Save these immediately. The secret key is only shown once. If you lose it, you'll need to create a new key pair.

Alternatively, if someone else manages your AWS account, you’ll need to ask your AWS administrator for:

• An IAM user with ECR permissions

• The Access Key ID and Secret Access Key for that user

Step 2: Check if AWS CLI is installed

You can do this by running this:

aws --version

Step 3: Configure AWS CLI with your credentials

Here’s how you can do this:

aws configure
```

It will prompt you for 4 things:
```
AWS Access Key ID [None]: <paste your Access Key ID here>
AWS Secret Access Key [None]: <paste your Secret Access Key here>
Default region name [None]: eu-north-1 or any region of your choice
Default output format [None]: json

Just paste your keys when prompted, type eu-north-1 or any region of your choice for region, and json for format (or just press Enter for format).

Step 4: Test your AWS configuration

Now you’ll want to test your config to make sure everything is set up properly:

aws sts get-caller-identity

This should show your AWS account details if everything is configured correctly.

Step 5: Login to ECR (Docker Registry)

Now, login to ECR:

aws ecr get-login-password --region eu-north-1 | docker login --username AWS --password-stdin 244836489456.dkr.ecr.eu-north-1.amazonaws.com

You should see: "Login Succeeded".

Understanding Image Naming in Docker Repositories

Every Docker image has a name that tells Docker where to find or store it. For example, when you run:

docker pull mongo:4.2

Docker is actually pulling from:

docker.io/library/mongo:4.2

Here’s what’s happening:

• docker.io is the registry (in this case, Docker Hub)

• library is the default namespace for official images

• mongo is the repository name

• 4.2 is the image tag

If you build a local image like todo-app:1.0, that image exists only on your machine. Docker won’t know where to push it unless you include the full registry path.

For AWS ECR, the image name must include your ECR registry URL. For example:

docker tag todo-app:1.0 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0

Then you can push it with:

docker push 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0

Without that full path, Docker won’t know which remote repository you’re referring to. That’s why just todo-app:1.0 alone won’t work.

Step 6: Build, Tag, and Push your image

# Tag your local image with the full ECR path
docker tag todo-app:1.0 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0

# Now push it
docker push 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0

⚠️ Note: Be careful when tagging and pushing your image, as every ECR repository URL is tied to a specific AWS account and region.

For example, in this tutorial, we’re using:

244836489456.dkr.ecr.eu-north-1.amazonaws.com

But your own ECR URL will be different depending on your AWS account and the region you selected (like us-east-1, ap-south-1, and so on).

So before you run your docker tag or docker push commands, make sure to replace the registry URL and region with your own.

If you don’t, Docker will throw errors like “tag does not exist” or “repository not found.”

In short, stay calm, double-check your region, and always confirm the exact ECR URL shown in your AWS console before pushing.

If you successfully ran Step 6, you should see output similar to this in your terminal:

stephenjohnson@Oghenekparobo docker_tut % docker tag todo-app:1.0 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0
stephenjohnson@Oghenekparobo docker_tut % docker push 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0
The push refers to repository [244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app]
4f94b5cbe8ab: Pushed 
85ba7bf54231: Pushed 
4ea46a43fa07: Pushed 
dee30873f229: Pushed 
e78159dbd370: Pushed 
a358a725b813: Pushed 
cd8a6003174c: Pushed 
abb63e49e652: Pushed 
6cc65bdde70e: Pushed 
41a4e3939504: Pushed 
3520c50ae60e: Pushed 
75ba6634710f: Pushed 
1.0: digest: sha256:51f07267936fc94d9b677db8a760801e6c5fd4764f4bb2bd7b4dd150c756a39b size: 2842

This confirms your image was successfully pushed to your private AWS ECR repository.

You can now go to the AWS Management Console and then ECR, and you should see your todo-app image listed there, along with the tag 1.0.

At this point, your image is safely stored in AWS ECR and ready to be pulled or deployed anywhere that has access to your repository.

Assignment: Create and Push a New Version of Your App

Now that your first image (todo-app:1.0) has been successfully pushed to AWS ECR, it’s time to simulate a real-world workflow where developers make updates and release new versions of their applications.

Now, you’ll make a small change to your Node.js app, rebuild it, and push the updated version as todo-app:2.0.

Deploying Our Image

Now it’s time to deploy our image using Docker Compose.

Up to this point, we have been running our app using a local image:

image: todo-app:1.0

But now that your image lives inside AWS ECR, we need to replace that line with the full ECR image URI, because Docker must know exactly where to pull the image from.

Local image:

image: todo-app:1.0

Private repository image (ECR):

image: 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0

Docker cannot magically guess where “todo-app:1.0” is stored. If you don’t include the full registry URL, Docker will assume it’s looking at your local machine, not AWS.

Here is the clean, fixed, properly formatted docker-compose file that pulls your app from ECR:

version: "3.8"

services:
  my-app:
    image: 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0
    container_name: my-app
    ports:
      - "3000:3000"
    depends_on:
      - mongodb

  mongodb:
    image: mongo
    container_name: mongo
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: password

  mongo-express:
    image: mongo-express
    container_name: mongo-express
    ports:
      - "8081:8081"
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: admin
      ME_CONFIG_MONGODB_ADMINPASSWORD: password
      ME_CONFIG_MONGODB_SERVER: mongodb
    depends_on:
      - mongodb

Why “my-app” instead of “todo-app”?

In this case, we renamed it to avoid confusion between:

• our local “todo-app:1.0”

• our ECR “todo-app:1.0”

This keeps things clean, but you can rename it back if you want.

Why Must We Use the Full Image URL for ECR?

Other containers like mongo and mongo-express work like this:

image: mongo
image: mongo-express

Because Docker knows these are on Docker Hub.

But for a private repo like AWS ECR, Docker has no idea where “todo-app” is unless you give the full path:

AWS_ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/repository_name:tag

This tells Docker:

• which account

• which region

• which repo

• which version

Without this URL, Docker can’t pull the image.

Every time we want to pull from a private ECR repo, including using Docker Compose, we must be logged in.

Run this:

aws ecr get-login-password --region eu-north-1 | docker login --username AWS --password-stdin 244836489456.dkr.ecr.eu-north-1.amazonaws.com

If you’re not logged in, Docker Compose will throw:

❌ pull access denied
❌ repository does not exist
❌ no basic auth credentials

Deploy Your App Using Docker Compose

Before deploying, it’s best practice to stop and remove any existing containers to avoid port conflicts or orphaned containers:

# Stop all running containers in this project
docker-compose down --remove-orphans

# Optional: verify nothing is running
docker ps

This ensures that port 3000 and other mapped ports are free, preventing errors when starting new containers.

Once the environment is clean, deploy your stack:

docker-compose up -d

Docker Compose will:

1. Connect to AWS ECR – Authenticate and pull the todo-app:1.0 image from your private repository.

2. Start MongoDB – Launch the database container with your configured credentials.

3. Start Mongo Express – Launch the web-based MongoDB admin interface.

4. Start your Node.js app – Launch the my-app container, linked to MongoDB.

Check the running containers:

docker ps

You should see:

• mongo

• mongo-express

• my-app

If my-app fails to start, it’s usually because port 3000 is already in use. Ensure it’s free by stopping any process using it:

lsof -i :3000
kill -9 <PID>  # if a process is using it

Then rerun:

docker-compose up -d

To access your app:

• Node.js app: http://localhost:3000

• Mongo Express: http://localhost:8081

This workflow ensures a clean start and avoids common port or container conflicts.

Sharing our Private Docker Image

Once your Node.js app is pushed to AWS ECR, it’s safely stored in your private repository. But what if another developer, team member, or server needs to run that same image? Since it’s private, Docker cannot pull it automatically like public images (e.g., mongo or nginx). They need authenticated access.

Here’s how they can get and use your image:

1. Grant IAM Access

Your collaborator needs an AWS IAM user or role with permissions for ECR. At minimum, the policy should allow:

• ecr:GetAuthorizationToken

• ecr:BatchCheckLayerAvailability

• ecr:GetDownloadUrlForLayer

• ecr:BatchGetImage

You can create a dedicated IAM user for this and provide them an Access Key ID and a Secret Access Key.

2. Install and Configure AWS CLI

The collaborator must have the AWS CLI installed. Then they configure it with their credentials:

aws configure

They enter:

• Access Key ID

• Secret Access Key

• Default region (the same region where the ECR repo exists, for example, eu-north-1)

• Default output format (usually json)

3. Authenticate Docker with ECR

Before pulling the image, Docker must authenticate using the AWS credentials:

aws ecr get-login-password --region eu-north-1 | docker login --username AWS --password-stdin 244836489456.dkr.ecr.eu-north-1.amazonaws.com

If successful, Docker will respond with:

Login Succeeded

4. Pull the Image

Now the collaborator can pull the image using the full ECR URI, which includes your AWS account, region, repository name, and tag:

docker pull 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0

5. Run the Container

After pulling, they can run the container locally:

docker run -p 3000:3000 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0

Or include it in a Docker Compose file, replacing the image: field with the full ECR URI.

• Public images like mongo don’t require this because Docker Hub is open. Private ECR images require explicit authentication.

• Every pull from a private repository requires an active login**.** Docker cannot guess credentials.

• Using the full image URI ensures Docker knows exactly where to fetch the image.

This setup allows your team to share, deploy, or run your application anywhere, on local machines, staging servers, or production, while keeping your repository private and secure.

Docker Volumes

When running containers like MongoDB, all data created inside a container is ephemeral. If the container stops or is removed, all data inside it disappears. This is fine for testing, but not suitable for production.

To solve this, Docker provides volumes, which allow containers to store data outside the container, either on the host machine or in Docker-managed storage, so it survives container restarts, rebuilds, or removals.

How Docker Volumes Work

Think of Docker volumes as persistent folders for containers:

• Data written inside a volume remains safe, even if the container is removed.

• Containers can read/write to these volumes.

• Volumes are essential for databases, logs, file uploads, or any persistent data your application needs.

Types of Docker Volumes

Docker has three main types of volumes:

1. Named Volumes

Named volumes are user-defined volumes with a clear name, that are fully managed by Docker. You’d typically use them in production databases and for persistent data that containers can share.

Here’s an example:

volumes:
  mongo-data:

And in a service:

volumes:
  - mongo-data:/data/db

2. Bind Mounts

Blind mounts map a folder from your host machine into the container. They’re often used for development, live syncing files, logs, and uploaded files.

Here’s an example:

volumes:
  - ./uploads:/usr/src/app/uploads

3. Anonymous Volumes

These are volumes without a name. Docker just assigns them a random name. You’d use them for temporary data for testing (and they’re not commonly used in production).

Here’s an example:

volumes:
  - /data/tmp

Example Docker Compose File Using Volumes

Here’s a full docker-compose.yml file using the most common volume types for a Node.js + MongoDB + Mongo Express stack:

version: "3.8"

services:
  my-app:
    image: 244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0
    container_name: my-app
    ports:
      - "3000:3000"
    depends_on:
      - mongodb
    volumes:
      - ./uploads:/usr/src/app/uploads  # bind mount for file uploads

  mongodb:
    image: mongo
    container_name: mongo
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: password
    volumes:
      - mongo-data:/data/db  # named volume for persistent database storage

  mongo-express:
    image: mongo-express
    container_name: mongo-express
    ports:
      - "8081:8081"
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: admin
      ME_CONFIG_MONGODB_ADMINPASSWORD: password
      ME_CONFIG_MONGODB_SERVER: mongodb
    depends_on:
      - mongodb

volumes:
  mongo-data:  # named volume definition

How this code is working:

1. MongoDB Volume (mongo-data): This is a named volume. It stores all database files under /data/db inside the container. It survives container restarts, removals, or rebuilds.

2. Node.js Uploads (./uploads): This is a bind mount. It maps the uploads folder on your host to /usr/src/app/uploads inside the container. Any uploaded files are immediately visible on your host.

3. Anonymous Volume: These are not shown in this file because it’s rarely used in production. Temporary data storage is created automatically by Docker if a volume is defined without a name.

Visual Concept (Simplified):

Host Machine
├─ /project/uploads  ← bind mount, synced with container
├─ Docker Volumes
│  └─ mongo-data    ← named volume, persistent MongoDB data

Containers
├─ my-app
│  └─ /usr/src/app/uploads  ← sees host uploads folder
├─ mongodb
│  └─ /data/db             ← uses named volume mongo-data
├─ mongo-express

Takeaways

• Always use volumes for data you care about.

• Named volumes are best for databases in production.

• Bind mounts are best for development and live syncing.

• Anonymous volumes are rarely needed outside testing.

• Volumes separate container lifecycle from data lifecycle, which is a cornerstone of Docker best practices.

Start Your Application

Once your Docker Compose is configured with volumes, the next step is to start your application and make sure the volumes are working correctly. Here’s a simple step-by-step guide.

1. Start the Containers

Run:

docker-compose up -d

The -d flag runs the containers in detached mode (in the background).

Docker will:

• Pull your app image from AWS ECR (if you’re logged in)

• Start MongoDB with the named volume

• Start Mongo Express

• Start your Node.js app

2. Check Running Containers

To see if everything started correctly:

docker ps

You should see something like:

CONTAINER ID   IMAGE                                               STATUS          PORTS
2a2e120cc912   244836489456.dkr.ecr.eu-north-1.amazonaws.com/todo-app:1.0   Up 5s    0.0.0.0:3000->3000/tcp
f4d5a1ab1234   mongo                                               Up 5s          0.0.0.0:27017->27017/tcp
c3d5b2bc2345   mongo-express                                      Up 5s          0.0.0.0:8081->8081/tcp

3. Verify Volumes

List Docker volumes:

docker volume ls

You should see your named volume, for example mongo-data.

Inspect the volume:

docker volume inspect docker_tut_mongo-data

This will show where Docker stores your MongoDB data on the host, for example:

[
    {
        "Name": "mongo-data",
        "Driver": "local",
        "Mountpoint": "/var/lib/docker/volumes/mongo-data/_data",
        "Labels": {},
        "Scope": "local"
    }
]

Anything stored in /data/db inside MongoDB is actually saved here on your host.

4. Test Data Persistence

1. Connect to MongoDB or your app and add some data.

2. Stop and remove the container:

docker-compose down

3. Restart the app:

docker-compose up -d

4. Check your data again.

• Because MongoDB uses the named volume, your data is still there.

• This proves the volume is persistent.

5. Optional: Check Node.js Uploads (Bind Mount)

• If you uploaded a file through your app, check your project folder ./uploads.

• You should see the file appear on your host machine because bind mounts sync host and container directories.

Conclusion

Well done, you have made it to the end of this comprehensive Docker tutorial. From unraveling the basics of containers and images, to networking, Docker Compose, volumes, and even deploying to a private AWS ECR repository, you've built a fully containerized Node.js application stack that's production-ready and scalable. These are hands-on skills that will transform how you develop, collaborate, and deploy applications in real-world scenarios.

Thank you for sticking with it. Docker can feel overwhelming at first – those long commands, networking quirks, and persistent data challenges aren't trivial. But getting to this point? It means you've conquered a steep learning curve and reached new heights in your development journey. You're now equipped to eliminate "it works on my machine" headaches, streamline CI/CD pipelines, and level up as a backend or full-stack pro.

Keep experimenting: Tweak your todo-app, try multi-stage builds in your Dockerfile, or explore orchestration tools like Kubernetes next. The Docker ecosystem is vast, but with this foundation, you're ready to dive deeper. If you hit snags or have questions, the community on Docker Hub, Stack Overflow, or GitHub.

You can find the final code here: https://github.com/Oghenekparobo/docker_tut_js/tree/final

In this tutorial, we will cover:

1. What is SonarQube?

2. How SonarQube Improves Code Quality

3. Step-by-step Installation and Configuration

4. How to Run Your First Code Analysis

What is SonarQube?

SonarQube is an open-source tool that checks for code quality continuously. It analyzes code to find issues like duplication, bad practices, test coverage gaps, bugs, and vulnerabilities, giving detailed reports. It works with many programming languages like Java, C#, JavaScript, Python, TypeScript, and Kotlin.

You can add SonarQube to your CI/CD pipelines, IDEs, and version control systems like GitHub, GitLab, or Bitbucket. It provides detailed dashboards that show metrics, trends, and issues in your code.

You can use custom rules to enforce coding standards and reduce technical debt. SonarQube also supports code coverage analysis to help teams improve their tests. With the Quality Gate feature, teams can ensure only clean, maintainable code goes into production.

SonarQube offers both free and paid versions to suit any team size. Overall, it helps improve software quality and encourages good coding practices.

How Does SonarQube Improve Code Quality?

Here’s how SonarQube helps improve code quality:

1. Early bug detection: Identifies bugs before they reach production

2. Improved maintainability: Highlights code and design issues

3. Security insights: Identifies vulnerabilities and security risks

4. Code coverage: Integration with testing tools to monitor unit test coverage

5. Customizable rules: Allows teams to set coding standards and policies

6. Team collaboration: Ensures consistent code quality across development teams

Step-by-Step Installation and Configuration

Prerequisites:

Here are the prerequisites that you will need before installing SonarQube

1. Java Runtime Environment(JRE): Java 11 or above installed in your system.

2. System Requirements: 2GB RAM minimum (Recommended: 4GB+).

3. MacOS: You can use HomeBrew, which is the package manager for MacOS that simplifies the installation of software.

Below are the steps to install SonarQube in your local machine:

Download SonarQube

Download the software from sonarsource downloads and choose the Community Edition for open-source projects.

Extract and Configure

To install SonarQube, you need to run the below command to unzip the file:

unzip sonarqube-<version>.zip
cd sonarqube-<version>/bin/<your-OS-folder>

Start SonarQube

On Linux/Mac, you need to run the below command:

./sonar.sh start

On Windows, you need to run this one:

StartSonar.bat

Access SonarQube

To access SonarQube, you need to open browser and go to: http://localhost:9000

Enter the default credentials:

• Username: admin

• Password: admin (you’ll be prompted to change it)

The page will look similar to below:

Set Up SonarQube in Your Project

To set up SonarQube in your project, start by opening the Java project on your machine. In the project root, create a sonar-project.properties file.

Add the below key value pairs in the file:

sonar.projectKey=spring-myproject
sonar.projectName=My Project
sonar.projectVersion=1.0
sonar.sources=.
sonar.host.url=http://localhost:9000

How to Run Your First Code Analysis

Configure and Run SonarScanner

SonarScanner is the tool that actually sends your code to SonarQube for analysis. Below are the detailed steps to follow to use it:

Install SonarScanner:

On Windows/Linux, download the software from SonarSource and unzip it:

unzip sonar-scanner-cli-<version>.zip

On MacOS, run the below command:

>brew install sonar-scanner

For both Windows/Linux and MacOS, verify the install by running the below command:

>sonar-scanner -v

Configure SonarScanner

After installing SonarScanner, you’ll need to configure it by setting the SonarQube server URL and authentication token. Then go to your SonarQube profile (top-right corner > My Account > Security) and generate a token.

Provide a name for the token and click ‘Generate’:

In the sonar-project.properties file in your project, add ‘sonar.login’ property and save.

sonar.projectKey=test-project
sonar.projectName=Test Project
sonar.host.url=http://localhost:9000
sonar.login=<YOUR_TOKEN_HERE>

Run the Analysis

Once the SonarScanner is configured, you can start scanning your project.

In a terminal or command prompt, go to the root of your project (where sonar-project.properties is located).

Run the following command:

>sonar-scanner

SonarScanner will analyze your code and push the results to your local SonarQube server. Visit http://localhost:9000, and you’ll see your project listed on the dashboard.

To view the analysis report, go to http://localhost:9000/dashboard?id=java-sonar-demo:

If you go to the ‘Issues’ tab at top left corner, you can view different categories of Software Quality, Severity of the Issues, and various other attributes in your code.

Conclusion

Now you have installed and configured SonarQube and learned how to scan your code using SonarScanner. You can easily configure it in your projects for continuous code quality analysis.

This is a fantastic tool for keeping your code base clean and maintainable. As the next steps, you can consider adding test coverage reports, enforcing quality gates in your pipeline, and exploring SonarCloud for cloud-based analysis.

CI/CD is a set of practices and tools designed to automate and streamline the process of integrating code changes, testing them, and deploying them to production. By adopting CI/CD, your team can reduce manual errors, speed up release cycles, and ensure that your code is always in a deployable state.

In this tutorial, we’ll focus on a beginner-friendly approach to setting up a basic CI/CD pipeline using Bitbucket, a Linux server, and Python with Flask. Specifically, we’ll create an automated process that pulls the latest changes from a Bitbucket repository to your Linux server whenever there’s a push or merge to a specific branch.

This process will be powered by Bitbucket webhooks and a simple Flask-based Python server that listens for incoming webhook events and triggers the deployment.

It’s important to note that CI/CD is a vast and complex field, and this tutorial is designed to provide a foundational understanding rather than to be an exhaustive guide.

We’ll cover the basics of setting up a CI/CD pipeline using tools that are accessible to beginners. Just keep in mind that real-world CI/CD systems often involve more advanced tools and configurations, such as containerization, orchestration, and multi-stage testing environments.

By the end of this tutorial, you’ll have a working example of how to automate deployments using Bitbucket, Linux, and Python, which you can build upon as you grow more comfortable with CI/CD concepts.

Table of Contents:

1. Why is CI/CD Important?

2. Step 1: Set Up a Webhook in Bitbucket

3. Step 2: Set Up the Flask Listener on Your Linux Server

4. Step 3: Expose the Flask App (Optional)

5. Step 4: Test the Setup

6. Step 5: Security Considerations

7. Wrapping Up

Why is CI/CD Important?

CI/CD has become a cornerstone of modern software development for several reasons. First and foremost, it accelerates the development process. By automating repetitive tasks like testing and deployment, developers can focus more on writing code and less on manual processes. This leads to faster delivery of new features and bug fixes, which is especially important in competitive markets where speed can be a differentiator.

Another key benefit of CI/CD is reduced errors and improved reliability. Automated testing ensures that every code change is rigorously checked for issues before it’s integrated into the main codebase. This minimizes the risk of introducing bugs that could disrupt the application or require costly fixes later. Automated deployment pipelines also reduce the likelihood of human error during the release process, ensuring that deployments are consistent and predictable.

CI/CD also fosters better collaboration among team members. In traditional development workflows, integrating code changes from multiple developers can be a time-consuming and error-prone process. With CI/CD, code is integrated and tested frequently, often multiple times a day. This means that conflicts are detected and resolved early, and the codebase remains in a stable state. As a result, teams can work more efficiently and with greater confidence, even when multiple contributors are working on different parts of the project simultaneously.

Finally, CI/CD supports continuous improvement and innovation. By automating the deployment process, teams can release updates to production more frequently and with less risk. This enables them to gather feedback from users faster and iterate on their products more effectively.

What We’ll Cover in This Tutorial

In this tutorial, we’ll walk through the process of setting up a simple CI/CD pipeline that automates the deployment of code changes from a Bitbucket repository to a Linux server. Here’s what you’ll learn:

1. How to configure a Bitbucket repository to send webhook notifications whenever there’s a push or merge to a specific branch.

2. How to set up a Flask-based Python server on your Linux server to listen for incoming webhook events.

3. How to write a script that pulls the latest changes from the repository and deploys them to the server.

4. How to test and troubleshoot your automated deployment process.

By the end of this tutorial, you’ll have a working example of a basic CI/CD pipeline that you can customize and expand as needed. Let’s get started!

Step 1: Set Up a Webhook in Bitbucket

Before starting with the setup, let’s briefly explain what a webhook is and how it fits into our CI/CD process.

A webhook is a mechanism that allows one system to notify another system about an event in real-time. In the context of Bitbucket, a webhook can be configured to send an HTTP request (often a POST request with payload data) to a specified URL whenever a specific event occurs in your repository, such as a push to a branch or a pull request merge.

In our case, the webhook will notify our Flask-based Python server (running on your Linux server) whenever there’s a push or merge to a specific branch. This notification will trigger a script on the server to pull the latest changes from the repository and deploy them automatically. Essentially, the webhook acts as the bridge between Bitbucket and your server, enabling seamless automation of the deployment process.

Now that you understand the role of a webhook, let’s set one up in Bitbucket:

1. Log in to Bitbucket and navigate to your repository.

2. On the left-hand sidebar, click on Settings.

3. Under the Workflow section, find and click on Webhooks.

4. Click the Add webhook button.

5. Enter a name for your webhook (for example, "Automatic Pull").

6. In the URL field, provide the URL to your server where the webhook will send the request. If you’re running a Flask app locally, this would be something like http://your-server-ip/pull-repo. (For production environments, it’s highly recommended to use HTTPS to secure the communication between Bitbucket and your server.)

7. In the Triggers section, choose the events you want to listen to. For this example, we will select Push (and optionally, Pull Request Merged if you want to deploy after merges, too).

8. Save the webhook with a self-explanatory name so it’s easy to identify later.

Once the webhook is set up, Bitbucket will send a POST request to the specified URL every time the selected event occurs. In the next steps, we’ll set up a Flask server to handle these incoming requests and trigger the deployment process.

Here is what you should see when you setup up the Bitbucket webhook

Step 2: Set Up the Flask Listener on Your Linux Server

In the next step, you’ll set up a simple web server on your Linux machine that will listen for the webhook from Bitbucket. When it receives the notification, it will execute a git pull or a force pull (in case of local changes) to update the repository.

Install Flask:

To create the Flask application, first install Flask by running:

pip install flask

Create the Flask App:

Create a new Python script (for example, app_repo_pull.py) on your server and add the following code:

from flask import Flask
import subprocess

app = Flask(__name__)

@app.route('/pull-repo', methods=['POST'])
def pull_repo():
    try:
        # Fetch the latest changes from the remote repository
        subprocess.run(["git", "-C", "/path/to/your/repository", "fetch"], check=True)
        # Force reset the local branch to match the remote 'test' branch
        subprocess.run(["git", "-C", "/path/to/your/repository", "reset", "--hard", "origin/test"], check=True)  # Replace 'test' with your branch name
        return "Force pull successful", 200
    except subprocess.CalledProcessError:
        return "Failed to force pull the repository", 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Here’s what this code does:

• subprocess.run(["git", "-C", "/path/to/your/repository", "fetch"]): This command fetches the latest changes from the remote repository without affecting the local working directory.

• subprocess.run(["git", "-C", "/path/to/your/repository", "reset", "--hard", "origin/test"]): This command performs a hard reset, forcing the local repository to match the remote test branch. Replace test with the name of your branch.

Make sure to replace /path/to/your/repository with the actual path to your local Git repository.

Step 3: Expose the Flask App (Optional)

If you want the Flask app to be accessible from outside your server, you need to expose it publicly. For this, you can set up a reverse proxy with NGINX. Here's how to do that:

First, install NGINX if you don't have it already by running this command:

sudo apt-get install nginx

Next, you’ll need to configure NGINX to proxy requests to your Flask app. Open the NGINX configuration file:

sudo nano /etc/nginx/sites-available/default

Modify the configuration to include this block:

server {
    listen 80;
    server_name your-server-ip;

    location /pull-repo {
        proxy_pass http://localhost:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Now just reload NGINX to apply the changes:

sudo systemctl reload nginx

Step 4: Test the Setup

Now that everything is set up, go ahead and start the Flask app by executing this Python script:

python3 app_repo_pull.py

Now to test if everything is working:

1. Make a commit: Push a commit to the test branch in your Bitbucket repository. This action will trigger the webhook.

1. Webhook trigger: The webhook will send a POST request to your server. The Flask app will receive this request, perform a force pull from the test branch, and update the local repository.

2. Verify the pull: Check the log output of your Flask app or inspect the local repository to verify that the changes have been pulled and applied successfully.

Step 5: Security Considerations

When exposing a Flask app to the internet, securing your server and application is crucial to protect it from unauthorized access, data breaches, and attacks. Here are the key areas to focus on:

1. Use a Secure Server with Proper Firewall Rules

A secure server is one that is configured to minimize exposure to external threats. This involves using firewall rules, minimizing unnecessary services, and ensuring that only required ports are open for communication.

Example of a secure server setup:

• Minimal software: Only install the software you need (for example, Python, Flask, NGINX) and remove unnecessary services.

• Operating system updates: Ensure your server's operating system is up-to-date with the latest security patches.

• Firewall configuration: Use a firewall to control incoming and outgoing traffic and limit access to your server.

For example, a basic UFW (Uncomplicated Firewall) configuration on Ubuntu might look like this:

# Allow SSH (port 22) for remote access
sudo ufw allow ssh

# Allow HTTP (port 80) and HTTPS (port 443) for web traffic
sudo ufw allow http
sudo ufw allow https

# Enable the firewall
sudo ufw enable

# Check the status of the firewall
sudo ufw status

In this case:

• The firewall allows incoming SSH connections on port 22, HTTP on port 80, and HTTPS on port 443.

• Any unnecessary ports or services should be blocked by default to limit exposure to attacks.

Additional Firewall Rules:

• Limit access to webhook endpoint: Ideally, only allow traffic to the webhook endpoint from Bitbucket's IP addresses to prevent external access. You can set this up in your firewall or using your web server (for example, NGINX) by only accepting requests from Bitbucket's IP range.

• Deny all other incoming traffic: For any service that does not need to be exposed to the internet (for example, database ports), ensure those ports are blocked.

2. Add Authentication to the Flask App

Since your Flask app will be publicly accessible via the webhook URL, you should consider adding authentication to ensure only authorized users (such as Bitbucket's servers) can trigger the pull.

Basic Authentication Example:

You can use a simple token-based authentication to secure your webhook endpoint. Here’s an example of how to modify your Flask app to require an authentication token:

from flask import Flask, request, abort
import subprocess

app = Flask(__name__)

# Define a secret token for webhook verification
SECRET_TOKEN = 'your-secret-token'

@app.route('/pull-repo', methods=['POST'])
def pull_repo():
    # Check if the request contains the correct token
    token = request.headers.get('X-Hub-Signature')
    if token != SECRET_TOKEN:
        abort(403)  # Forbidden if the token is incorrect

    try:
        subprocess.run(["git", "-C", "/path/to/your/repository", "fetch"], check=True)
        subprocess.run(["git", "-C", "/path/to/your/repository", "reset", "--hard", "origin/test"], check=True)
        return "Force pull successful", 200
    except subprocess.CalledProcessError:
        return "Failed to force pull the repository", 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

How it works:

• The X-Hub-Signature is a custom header that you add to the request when setting up the webhook in Bitbucket.

• Only requests with the correct token will be allowed to trigger the pull. If the token is missing or incorrect, the request is rejected with a 403 Forbidden response.

You can also use more complex forms of authentication, such as OAuth or HMAC (Hash-based Message Authentication Code), but this simple token approach works for many cases.

3. Use HTTPS for Secure Communication

It’s crucial to encrypt the data transmitted between your Flask app and the Bitbucket webhook, as well as any sensitive data (such as tokens or passwords) being transmitted over the network. This ensures that attackers cannot intercept or modify the data.

Why HTTPS?

• Data encryption: HTTPS encrypts the communication, ensuring that sensitive data like your authentication token is not exposed to man-in-the-middle attacks.

• Trust and integrity: HTTPS helps ensure that the data received by your server hasn’t been tampered with.

Using Let’s Encrypt to Secure Your Flask App with SSL:

1. Install Certbot (the tool for obtaining Let’s Encrypt certificates):

sudo apt-get update
sudo apt-get install certbot python3-certbot-nginx

Obtain a free SSL certificate for your domain:

sudo certbot --nginx -d your-domain.com

• This command will automatically configure Nginx to use HTTPS with a free SSL certificate from Let’s Encrypt.

• Ensure HTTPS is used: Make sure that your Flask app or Nginx configuration forces all traffic to use HTTPS. You can do this by setting up a redirection rule in Nginx:

server {
    listen 80;
    server_name your-domain.com;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name your-domain.com;

    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;

    # Other Nginx configuration...
}

Automatic Renewal: Let’s Encrypt certificates are valid for 90 days, so it’s important to set up automatic renewal:

sudo certbot renew --dry-run

This command tests the renewal process to make sure everything is working.

4. Logging and Monitoring

Implement logging and monitoring for your Flask app to track any unauthorized attempts, errors, or unusual activity:

• Log requests: Log all incoming requests, including the IP address, request headers, and response status, so you can monitor for any suspicious activity.

• Use monitoring tools: Set up tools like Prometheus, Grafana, or New Relic to monitor server performance and app health.

Wrapping Up

In this tutorial, we explored how to set up a simple, beginner-friendly CI/CD pipeline that automates deployments using Bitbucket, a Linux server, and Python with Flask. Here’s a recap of what you’ve learned:

1. CI/CD Fundamentals: We discussed the basics of Continuous Integration (CI) and Continuous Delivery/Deployment (CD), which are essential practices for automating the integration, testing, and deployment of code. You learned how CI/CD helps speed up development, reduce errors, and improve collaboration among developers.

2. Setting Up Bitbucket Webhooks: You learned how to configure a Bitbucket webhook to notify your server whenever there’s a push or merge to a specific branch. This webhook serves as a trigger to initiate the deployment process automatically.

3. Creating a Flask-based Webhook Listener: We showed you how to set up a Flask app on your Linux server to listen for incoming webhook requests from Bitbucket. This Flask app receives the notifications and runs the necessary Git commands to pull and deploy the latest changes.

4. Automating the Deployment Process: Using Python and Flask, we automated the process of pulling changes from the Bitbucket repository and performing a force pull to ensure the latest code is deployed. You also learned how to configure the server to expose the Flask app and accept requests securely.

5. Security Considerations: We covered critical security steps to protect your deployment process:

   • Firewall Rules: We discussed configuring firewall rules to limit exposure and ensure only authorized traffic (from Bitbucket) can access your server.

   • Authentication: We added token-based authentication to ensure only authorized requests can trigger deployments.

   • HTTPS: We explained how to secure the communication between your server and Bitbucket using SSL certificates from Let's Encrypt.

   • Logging and Monitoring: Lastly, we recommended setting up logging and monitoring to keep track of any unusual activity or errors.

Next Steps

By the end of this tutorial, you now have a working example of an automated deployment pipeline. While this is a basic implementation, it serves as a foundation you can build on. As you grow more comfortable with CI/CD, you can explore advanced topics like:

• Multi-stage deployment pipelines

• Integration with containerization tools like Docker

• More complex testing and deployment strategies

• Use of orchestration tools like Kubernetes for scaling

CI/CD practices are continually evolving, and by mastering the basics, you’ve set yourself up for success as you expand your skills in this area. Happy automating and thank you for reading!

You can fork the code from here.

As projects grow and evolve, managing documentation across various formats—whether it’s markdown, HTML, or PDF for offline use—can become a time-consuming and error-prone task.

Pandoc is a powerful tool that allows you to convert documentation between formats seamlessly.

Still, even with Pandoc, manually converting files for every update can become a bottleneck in large projects or teams where documentation is frequently updated.

In this article, I’ll guide you through setting up shell scripts, using Makefiles, and integrating Pandoc into CI/CD pipelines to streamline your workflow and keep your documentation up-to-date with minimal effort.

Table of Contents

• Why Automate Documentation Conversion?

• How to Automate Pandoc using Shell Scripts

• Basic Shell Script for Pandoc Conversion

• Customizing the Script for Several Files

• Automating with Makefiles

• A Makefile for Pandoc Conversions

• Defining Dependencies in Makefiles

• Why Use Makefiles for Pandoc Automation?

• How to Integrate Pandoc with CI/CD Pipelines

• Example: Setting Up Pandoc with GitHub Actions

• Triggering on Documentation Updates

• Adapting for GitLab CI or Jenkins

• Advanced Automation Techniques

• Automating with Cron Jobs

• Ensuring Consistency with Docker

• Combining Tools for Efficiency

• Conclusion

Why Automate Documentation Conversion?

Manually converting documentation between formats in large projects can become a daunting task.

Whether you're generating HTML, PDF, or DOCX versions from the same source, repeating this process for every update leads to various challenges:

• Time-Consuming: Running manual commands to convert documentation every time you make a change eats into valuable development time when updates happen often.

• Prone to Errors: The manual process increases the likelihood of mistakes, such as using incorrect commands, missing steps, or generating outdated versions of your documentation. These inconsistencies can confuse both developers and end-users.

• Difficult to Scale: As projects grow in size, managing documents across different formats without automation can become unmanageable. Teams working in parallel may struggle to keep documentation synchronized, leading to mismatches between formats.

By automating the conversion process with tools like Pandoc, you can overcome these challenges and enjoy a range of benefits:

• Consistency: Automation ensures that all versions of your documentation are always up-to-date and accurate. No matter the number of formats you're generating, the process remains standardized.

• Efficiency: Automated workflows free up time by handling repetitive tasks in the background, allowing teams to focus on development rather than manually managing documentation updates.

• Scalability: With automation, it’s straightforward to scale documentation efforts as your project grows. Whether you're maintaining a single document or an entire library of resources, automation makes sure everything stays synchronized with minimal effort.

The next section explores how to automate the conversion process.

How to Automate Pandoc using Shell Scripts

By using shell scripts, you can streamline the process of running PanDoc commands, saving time and reducing the risk of errors associated with manual command execution.

Basic Shell Script for PanDoc Conversion

To get started, we’ll create a shell script that converts a single Markdown file into various formats.

For instance, here’s a script that converts input.md into HTML and PDF:

#!/bin/bash

# Convert input.md to HTML and PDF
pandoc input.md -o output.html
pandoc input.md -o output.pdf

echo "Conversion complete!"

In the above script:

• The #!/bin/bash line indicates that the script should be run using the Bash shell.

• The pandoc commands convert the input.md file into output.html and output.pdf.

• The echo command confirms that the conversion process is complete.

Customizing the Script for Several Files

If you want to convert all Markdown files in a directory, you can customize your script to process several files at once.

For instance, here’s a script that converts input.md into HTML and PDF:

#!/bin/bash

# Go through all the Markdown files present in the current directory
for file in *.md; do
  # Convert each Markdown file to PDF
  pandoc "$file" -o "${file%.md}.pdf"
done

echo "All conversions complete!"

In this script:

• The for loop iterates over each .md file in the current directory.

• The pandoc command converts each Markdown file to a PDF, preserving the original filename but changing the extension to .pdf with the ${file%.md}.pdf syntax.

• The final echo confirms that all conversions are complete.

Adding Error Handling and Complex Logic

To enhance your script's robustness, you can add error handling and extra logic.

For instance, perhaps you want to make sure that Pandoc is installed before proceeding, and handle cases where the input file may be missing.

The script will be as follows:

#!/bin/bash

# Check if Pandoc is installed
if ! command -v pandoc &> /dev/null; then
  echo "Pandoc is not installed. Please install it and try again."
  exit 1
fi

# Go through all the Markdown files present in the current directory
for file in *.md; do
  if [ -e "$file" ]; then
    # Convert each Markdown file to PDF
    pandoc "$file" -o "${file%.md}.pdf"
    echo "Converted $file to PDF."
  else
    echo "No Markdown files found."
  fi
done

echo "All conversions complete!"

In this enhanced script:

• The if ! command -v pandoc &> /dev/null; then line checks whether Pandoc is installed.

• The if [ -e "$file" ]; then statement checks if each Markdown file exists before attempting to convert it.

• An informative message is printed after each successful conversion, providing feedback on the process.

As your project grows in complexity, relying solely on shell scripts for automation can become difficult to manage when dealing with lots of files and frequent updates.

This is where Makefiles come in handy.

Automating with Makefiles

A Makefile is a special file that defines rules and commands for automating tasks in a project.

It’s widely used by developers to compile code, but developers can also leverage it for non-compilation tasks, such as converting documentation formats with Pandoc.

A Makefile for Pandoc Conversions

Here’s an example of how you can create a Makefile to automate Pandoc conversions:

all: html pdf

html:
    pandoc input.md -o output.html

pdf:
    pandoc input.md -o output.pdf

In this Makefile:

• all is the default target. When you run make without specifying a target, it will run the html and pdf targets sequentially.

• The html target runs a PanDoc command to convert input.md into output.html.

• The pdf target runs a PanDoc command to convert input.md into output.pdf.

To use this Makefile, run the following command in your terminal:

codemake

This will execute both the html and pdf targets, converting the Markdown file into both formats.

Defining Dependencies in Makefiles

One of the major strengths of Makefiles is their ability to handle dependencies.

In the context of documentation conversion, you can specify which files to update when you detect changes in the source file.

Let’s look at an example:

output.html: input.md
    pandoc input.md -o output.html

output.pdf: input.md
    pandoc input.md -o output.pdf

In this Makefile:

• The output.html and output.pdf files depend on input.md.

• If you run make output.html or make output.pdf, Pandoc will regenerate the corresponding file if you update input.md after the last time you created the output file.

This ensures that Pandoc converts the files that have changed, saving time when working on large documentation projects.

Why Use Makefiles for Pandoc Automation?

Makefiles offer several advantages for automating Pandoc conversions in larger projects:

• Efficiency: Makefiles rebuild what’s necessary, meaning you won’t waste time converting files that haven’t changed.

• Simplicity: Once set up, running make simplifies the conversion process to a single command, making it easier for teams to maintain consistent documentation workflows.

• Scalability: As your project grows, you can add more targets and dependencies to the Makefile, automating everything from documentation to more complex build processes.

Makefiles are an excellent option for automating documentation conversions, when combined with Pandoc for multi-format outputs.

They help ensure that your workflow is efficient and your documentation remains up-to-date.

As modern development practices increasingly rely on Continuous Integration and Continuous Deployment (CI/CD), automating routine tasks such as documentation generation becomes essential.

Integrating Pandoc into your CI/CD pipelines allows for seamless documentation conversion and updates, ensuring that your docs are always up-to-date without manual intervention.

By automating this process, you can focus on coding and building, while your pipeline handles the conversion and distribution of your project documentation. Let’s explore how to set up Pandoc in a CI/CD pipeline to streamline your documentation workflows.

How to Integrate Pandoc with CI/CD Pipelines

Automating documentation generation within your CI/CD pipeline brings significant benefits, including consistency, efficiency, and hands-off updates.

Whether you’re using GitHub Actions, GitLab CI, Jenkins, or another automation tool, integrating Pandoc ensures that documentation gets generated and distributed whenever changes occur.

This approach reduces the risk of outdated or inconsistent documentation.

Example: Setting Up Pandoc with GitHub Actions

Let’s walk through a clear example of how you can use GitHub Actions to automate the process of generating documentation with Pandoc.

Below is a basic workflow that converts a Markdown file (*.md) into a PDF whenever code gets pushed to the repository.

name: Generate Documentation

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      # Step 1: Checkout code from the repository
      - name: Checkout code
        uses: actions/checkout@v2

      # Step 2: Install Pandoc
      - name: Install Pandoc
        run: sudo apt-get install pandoc

      # Step 3: Convert Markdown to PDF
      - name: Convert Markdown to PDF
        run: pandoc input.md -o output.pdf

      # Step 4: Upload the generated PDF as an artifact
      - name: Upload Documentation
        uses: actions/upload-artifact@v2
        with:
          name: documentation
          path: output.pdf

Here’s a breakdown of each step:

• Checkout Code: The actions/checkout@v2 action retrieves the latest code from your repository.

• Install Pandoc: This step installs Pandoc on the runner (in this case, an Ubuntu machine) so that it can gets used for the documentation conversion.

• Convert Markdown to PDF: The PanDoc command converts the input.md file into output.pdf.

• Upload Documentation: This step saves the generated PDF as an artifact in the CI/CD pipeline, making it downloadable or accessible later

Triggering on Documentation Updates

You can configure your CI/CD pipeline to trigger when documentation updates occur, for example, when changes get pushed to a docs branch:

on:
  push:
    branches:
      - docs

This setup ensures that your documentation gets regenerated when changes are specifically made to the documentation branch, reducing unnecessary rebuilds.

Adapting for GitLab CI or Jenkins

While the above example uses GitHub Actions, the same principles work on other CI/CD systems like GitLab CI or Jenkins.

For instance, in GitLab CI, you could define a .gitlab-ci.yml file that follows similar steps to check out the code, install Pandoc, convert the files, and store the resulting documentation.

Integrating Pandoc into your CI/CD pipeline offers a reliable way to automate your documentation workflows, ensuring that your project docs are always current, accurate, and accessible.

Advanced Automation Techniques

When your project reaches a stage where documentation needs to be generated frequently or across multiple environments, it’s essential to extend your automation strategy to maintain consistency and reliability.

Automating with Cron Jobs

For projects where documentation updates occur frequently, but not always due to code pushes or commits, you can automate the process using cron jobs.

Cron allows you to schedule tasks at specific intervals, meaning your documentation can automatically get updated at set times without needing manual input.

For example, setting up a cron job to run every day at midnight to convert Markdown files to PDFs:

0 0 * * * /path/to/pandoc /path/to/input.md -o /path/to/output.pdf

With this cron job in place, Pandoc will automatically convert your Markdown files to PDFs at the specified time, ensuring that your documentation remains up-to-date without manual intervention.

Ensuring Consistency with Docker

When working in a team or across various systems, ensuring that Pandoc runs consistently in different environments can be challenging.

Docker provides a solution to this by allowing you to package Pandoc and all its dependencies in a container, ensuring that the same setup gets used everywhere.

This is useful in CI/CD pipelines, where different environments (like local machines, staging, and production servers) can have different configurations.

You can use the official Pandoc Docker image to simplify the setup process:

docker run --rm -v $(pwd):/data pandoc/core:latest input.md -o output.pdf

This command runs Pandoc inside a Docker container, using the latest version of Pandoc, and mounts your current directory to the container’s /data directory.

By doing this, you ensure that no matter where the pipeline runs, the conversion will work the same way every time.

Combining Tools for Efficiency

By combining tools like cron jobs and Docker, you can set up an advanced automation pipeline that ensures:

• Scheduled updates: Cron jobs trigger documentation generation at regular intervals.

• Consistency across environments: Docker ensures that Pandoc and its dependencies are the same on every machine.

• Reliability: Together, these tools help ensure that your documentation is always accurate, no matter where or when it’s generated.

These advanced techniques allow you to further streamline your documentation workflows, improving both efficiency and reliability as your project grows.

Conclusion

Automating documentation conversion with Pandoc not only saves time but also ensures consistency and scalability as your project grows.

Whether you’re managing small or large projects, these techniques enable you to focus on coding and building, knowing that your documentation is automatically up-to-date and ready for distribution.

By embracing automation, you streamline your development process and enhance collaboration across teams, ensuring that your documentation evolves as efficiently as your code.

Now it’s time to apply these tools to your projects and enjoy the benefits of a fully automated documentation pipeline.

Let’s stay in touch:

• Connect with me on LinkedIn. I post regularly, so following me is a great idea.

• Follow me on X

Feel free to reach out through the channels above if you have any questions. I will be happy to help.

One of the most efficient ways to document a database is through dbdocs and DBML - an open sourced Database Markup Language.

In this guide, I’ll show you how to create database documentation using these tools, step by step.

What is dbdocs?

dbdocs is a platform that generates database documentation from your schema, easily shareable via a link. Using DBML (Database Markup Language), you can create clear, shareable, and updatable documentation of your database structure.

Prerequisites

Before we begin, ensure you have the following:

• Basic knowledge of databases and SQL.

• A database schema to document (we’ll use a PostgreSQL example in this guide).

Step 1: Install DBML CLI and dbdocs

Start by installing the DBML CLI, which helps convert your database schema into a DBML format. You also need the dbdocs CLI to generate and publish your documentation.

npm install -g dbdocs

Step 2: Export Your Database Schema to DBML

If you’re working with an existing database, you can export the schema into DBML using the DBML CLI tool.

For PostgreSQL, run the following command:

$ dbdocs db2dbml postgres <connection-string> -o database.dbml

✔ Connecting to database <db-name>... done.
✔ Generating DBML... done.
✔ Wrote to database.dbml

This command will export your database schema and save it into a file called database.dbml.

Here’s an example of how a generated DBML file might look:

Table users {
  id int [pk, increment]
  username varchar(50) [not null]
  email varchar(100) [not null, unique]
  created_at timestamp [not null]
}

Table orders {
  id int [pk, increment]
  user_id int [not null, ref: > users.id]
  total decimal [not null]
  created_at timestamp [not null]
}

In this example:

• The users and orders tables are defined.

• Fields are annotated with types and constraints.

• The relationship between orders.user_id and users.id is established using ref.

Step 3: Edit and Add Notes to the DBML File

You may want to clean it up or add extra documentation like table descriptions and field descriptions to communicate with other members in the team.

Step 4: Generate Documentation with dbdocs

Once your DBML file is ready, the next step is to generate the documentation using dbdocs. First, you need to login to dbdocs:

dbdocs login

After logging in, publish the DBML file:

dbdocs build database.dbml

This command will generate a shareable documentation link that you can access via the dbdocs platform. You can also set access permissions and collaborate with your team.

This seamless workflow ensures that your documentation always reflects the latest state of your database.

Benefits of Using dbdocs with DBML

• Simplicity: The DBML syntax is simple and easy to learn, making it a perfect fit for teams.

• Automation: You can automate your database documentation updates as part of your CI/CD pipeline.

• Collaboration: Easily share documentation links with your team or stakeholders for easy access and discussion.

• Version Control: Use schema changelog to track database schema changes over time.

• Visualization: dbdocs provides a clean interface to visualize your database schema, relationships, and annotations. Try this demo to learn more.

Conclusion

In this tutorial, we explored how to export a database schema, customize it, and generate shareable documentation using dbdocs.

By incorporating this workflow into your development process, you’ll improve your team’s collaboration, enhance your project’s scalability, and ensure that everyone stays on the same page. Happy documenting!