<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
    <channel>
        
        <title>
            <![CDATA[ Traefik - freeCodeCamp.org ]]>
        </title>
        <description>
            <![CDATA[ Browse thousands of programming tutorials written by experts. Learn Web Development, Data Science, DevOps, Security, and get developer career advice. ]]>
        </description>
        <link>https://www.freecodecamp.org/news/</link>
        <image>
            <url>https://cdn.freecodecamp.org/universal/favicons/favicon.png</url>
            <title>
                <![CDATA[ Traefik - freeCodeCamp.org ]]>
            </title>
            <link>https://www.freecodecamp.org/news/</link>
        </image>
        <generator>Eleventy</generator>
        <lastBuildDate>Tue, 26 May 2026 04:43:32 +0000</lastBuildDate>
        <atom:link href="https://www.freecodecamp.org/news/tag/traefik/rss.xml" rel="self" type="application/rss+xml" />
        <ttl>60</ttl>
        
            <item>
                <title>
                    <![CDATA[ How I Built a Production-Ready CI/CD Pipeline for a Monorepo-Based Microservices System with Jenkins, Docker Compose, and Traefik ]]>
                </title>
                <description>
                    <![CDATA[ This tutorial is a complete, real-world guide to building a production-ready CI/CD pipeline using Jenkins, Docker Compose, and Traefik on a single Linux server. You’ll learn how to expose services on  ]]>
                </description>
                <link>https://www.freecodecamp.org/news/build-production-ready-ci-cd-pipeline-for-monorepo-based-microservices-system/</link>
                <guid isPermaLink="false">69ea60c8904b915438a58ca2</guid>
                
                    <category>
                        <![CDATA[ Jenkins ]]>
                    </category>
                
                    <category>
                        <![CDATA[ ci-cd ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Docker ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Traefik ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Devops ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Md Tarikul Islam ]]>
                </dc:creator>
                <pubDate>Thu, 23 Apr 2026 18:11:20 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/66cb39fcaa2a09f9a8d691c1/d59c62f5-e376-4f09-851f-83e437f9960a.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>This tutorial is a complete, real-world guide to building a production-ready CI/CD pipeline using Jenkins, Docker Compose, and Traefik on a single Linux server.</p>
<p>You’ll learn how to expose services on a custom domain with auto-renewing HTTPS, and implement a smart deployment strategy that detects changes and redeploys only the affected microservices. This helps avoid unnecessary full-stack redeploys. We'll also cover real production issues and the exact fixes for each one.</p>
<h2 id="heading-table-of-contents"><strong>Table of Contents</strong></h2>
<ul>
<li><p><a href="#heading-1-what-youll-build">1. What you'll build</a></p>
</li>
<li><p><a href="#heading-2-architecture">2. Architecture</a></p>
</li>
<li><p><a href="#heading-3-server-prerequisites">3. Server prerequisites</a></p>
</li>
<li><p><a href="#heading-4-traefik-the-reverse-proxy">4. Traefik — the reverse proxy</a></p>
</li>
<li><p><a href="#heading-5-run-jenkins-in-docker">5. Run Jenkins in Docker</a></p>
</li>
<li><p><a href="#heading-6-expose-jenkins-on-a-domain-via-traefik">6. Expose Jenkins on a domain via Traefik</a></p>
</li>
<li><p><a href="#heading-7-first-time-jenkins-setup">7. First-time Jenkins setup</a></p>
</li>
<li><p><a href="#heading-8-add-the-github-credential">8. Add the GitHub credential</a></p>
</li>
<li><p><a href="#heading-9-create-the-pipeline-job">9. Create the pipeline job</a></p>
</li>
<li><p><a href="#heading-10-the-jenkinsfile-deploy-only-what-changed">10. The Jenkinsfile (deploy only what changed)</a></p>
</li>
<li><p><a href="#heading-11-end-to-end-test">11. End-to-end test</a></p>
</li>
<li><p><a href="#heading-12-troubleshooting-every-error-we-hit">12. Troubleshooting — every error we hit</a></p>
</li>
<li><p><a href="#heading-13-mental-model-host-vs-container">13. Mental model: host vs. container</a></p>
</li>
<li><p><a href="#heading-14-daily-operations-cheat-sheet">14. Daily operations cheat sheet</a></p>
</li>
<li><p><a href="#heading-15-what-id-do-differently-next-time">15. What I'd do differently next time</a></p>
</li>
<li><p><a href="#heading-closing-thoughts">Closing thoughts</a></p>
</li>
</ul>
<h2 id="heading-1-what-youll-build">1. What You'll Build</h2>
<p>In this tutorial, you'll build a Jenkins instance running inside Docker on the same Linux server as your application stack.</p>
<p>Traefik will act as a reverse proxy in front of Jenkins, exposing it via a clean URL (<a href="https://jenkins.example.com"><code>https://jenkins.example.com</code></a>) with <strong>auto-renewing Let's Encrypt certificates</strong>.</p>
<p>You'll also create a Jenkinsfile in your application repository that:</p>
<ul>
<li><p>Automatically triggers on every push to the <code>staging</code> branch,</p>
</li>
<li><p>Detects which microservices changed in each commit,</p>
</li>
<li><p>Pulls the latest code on the host machine,</p>
</li>
<li><p>Rebuilds and restarts <strong>only the affected services</strong>.</p>
</li>
</ul>
<p>On every push, only the relevant services are redeployed.</p>
<h3 id="heading-prerequisites">Prerequisites</h3>
<p>Before jumping in, this guide assumes you’re already comfortable with a few core concepts and tools.</p>
<p>This isn't a beginner-level tutorial — we’ll be working directly with infrastructure, containers, and CI/CD pipelines.</p>
<p>You should be familiar with:</p>
<ul>
<li><p>Basic Linux commands (SSH, file system navigation, permissions)</p>
</li>
<li><p>Docker fundamentals (images, containers, volumes, networks)</p>
</li>
<li><p>Git workflows (clone, pull, branches)</p>
</li>
<li><p>General idea of CI/CD pipelines</p>
</li>
</ul>
<p>Tools and environment required:</p>
<ul>
<li><p>A Linux server (Ubuntu recommended)</p>
</li>
<li><p>Docker Engine + Docker Compose (v2)</p>
</li>
<li><p>A domain name (for Traefik + HTTPS)</p>
</li>
<li><p>GitHub repository (for your backend project)</p>
</li>
<li><p>Basic understanding of microservices architecture</p>
</li>
</ul>
<p>If you’re comfortable with the above, you’re ready to follow along.</p>
<h2 id="heading-2-architecture">2. Architecture</h2>
<p>Here's an overview of the architecture:</p>
<pre><code class="language-plaintext">┌──────────────────────────── Linux server (Ubuntu) ────────────────────────────┐
│                                                                               │
│   /home/developer/projects/                                                  │
│       └── project-prod-configs/             ← infra repo (compose, Traefik) │
│              ├── docker-compose.staging.yml                                   │
│              ├── traefik.staging.yml                                          │
│              └── project-backend/          ← app repo (services, gateways) │
│                     ├── Jenkinsfile                                           │
│                     ├── docker-compose.staging.yml                            │
│                     └── apps/                                                 │
│                            ├── services/&lt;name&gt;/                               │
│                            ├── gateways/&lt;name&gt;/                               │
│                            └── core/&lt;name&gt;/                                   │
│                                                                               │
│   ┌─────────────────────── Docker network: proxy ──────────────────────┐      │
│   │  traefik (80, 443)                                                 │      │
│   │     │                                                              │      │
│   │     ├──► jenkins  (projects-jenkins-staging)                     │      │
│   │     │      ↳ /projects  ← bind-mount of the host project tree     │      │
│   │     │      ↳ /var/run/docker.sock ← controls host Docker           │      │
│   │     │                                                              │      │
│   │     └──► your services &amp; gateways (built by the pipeline)          │      │
│   └────────────────────────────────────────────────────────────────────┘      │
│                                                                               │
└───────────────────────────────────────────────────────────────────────────────┘
            ▲
            │  webhook on push
            │
   GitHub: &lt;org&gt;/project-backend (branch: staging)
</code></pre>
<p>There are two key ideas here:</p>
<ol>
<li><p><strong>Jenkins runs in a container</strong>, but it controls the <strong>host's</strong> Docker by mounting <code>/var/run/docker.sock</code>. It also bind-mounts the project folder as <code>/projects/...</code>, so it can <code>cd</code> into the real code on the host and run <code>docker compose</code> there.</p>
</li>
<li><p>The <strong>Jenkinsfile lives inside the app repo</strong>, so the pipeline definition is versioned with the code. Jenkins simply points at it.</p>
</li>
</ol>
<h3 id="heading-3-server-prerequisites">3. Server Prerequisites</h3>
<p>Before we start configuring Jenkins or Traefik, we need to prepare the server properly.</p>
<p>In this step, we’ll:</p>
<ul>
<li><p>Create a dedicated Linux user for managing the project</p>
</li>
<li><p>Install Docker and Docker Compose</p>
</li>
<li><p>Set up the folder structure for our repositories</p>
</li>
</ul>
<p>This ensures our CI/CD pipeline runs in a clean and predictable environment.</p>
<pre><code class="language-bash"># Linux user that owns the project tree
sudo adduser developer

# Docker engine + Compose plugin
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker developer

# Sanity check Compose v2
docker compose version
# -&gt; Docker Compose version v2.x.y

# Find where the Compose plugin binary lives — write it down, you'll need it
ls /usr/libexec/docker/cli-plugins/docker-compose
# (some distros use /usr/lib/docker/cli-plugins/docker-compose)

# Project layout
sudo mkdir -p /home/developer/project
sudo chown -R developer:developer /home/developer/project

# Clone both repos in the right place
cd /home/developer/projects
git clone https://github.com/&lt;org&gt;/projects-prod-configs.git
cd projects-prod-configs
git clone -b staging https://github.com/&lt;org&gt;/projects-backend.git
</code></pre>
<p>You should now have:</p>
<pre><code class="language-plaintext">/home/developer/projects/projects-prod-configs/projects-backend
</code></pre>
<p>Memorize this path — your Jenkinsfile references it.</p>
<h3 id="heading-dns">DNS</h3>
<p>Point an A-record for your Jenkins subdomain to the server's public IP <strong>before</strong> the next steps so Let's Encrypt can validate via HTTP challenge:</p>
<pre><code class="language-plaintext">jenkins.example.com   A   &lt;server-public-ip&gt;
</code></pre>
<h2 id="heading-4-traefik-the-reverse-proxy">4. Traefik — the Reverse Proxy</h2>
<p>Traefik acts as the entry point to your entire system. Instead of exposing each service manually with ports, Traefik automatically:</p>
<ul>
<li><p>Routes traffic based on domain names</p>
</li>
<li><p>Generates and renews HTTPS certificates using Let’s Encrypt</p>
</li>
<li><p>Connects to Docker and detects services dynamically</p>
</li>
</ul>
<p>In simple terms, Traefik lets you access services like:</p>
<p><a href="https://jenkins.example.com">https://jenkins.example.com</a><br><a href="https://api.example.com">https://api.example.com</a></p>
<p>…without manually configuring NGINX or managing SSL certificates.</p>
<p>In this setup, Traefik watches Docker containers and routes traffic using labels we'll define later.</p>
<p>Traefik gives every container a real domain and a real cert with <strong>zero per-service config</strong> — you just add a few labels.</p>
<h3 id="heading-traefikstagingyml-static-config"><code>traefik.staging.yml</code> (static config)</h3>
<p>Put this at the root of your infra repo:</p>
<pre><code class="language-yaml">api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: web
      email: admin@example.com           # ← change me
      storage: /etc/traefik/acme.json

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false              # only containers with traefik.enable=true
    network: proxy
  file:
    directory: /etc/traefik/dynamic
    watch: true

log:
  level: INFO

accessLog: {}
</code></pre>
<h3 id="heading-the-traefik-service-in-docker-composestagingyml">The Traefik service in <code>docker-compose.staging.yml</code></h3>
<pre><code class="language-yaml">networks:
  proxy:
    name: proxy
    driver: bridge
  internal:
    name: internal
    driver: bridge

volumes:
  acme-data:
  traefik-logs:
  jenkins-data:

services:
  traefik:
    image: traefik:v2.11
    container_name: projects-traefik-staging
    restart: unless-stopped
    ports:
      - "80:80"        # HTTP (auto-redirects to HTTPS)
      - "443:443"      # HTTPS
      - "8080:8080"    # Traefik dashboard (internal only — protect via firewall)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.staging.yml:/etc/traefik/traefik.yml:ro
      - ./dynamic:/etc/traefik/dynamic:ro
      - acme-data:/etc/traefik           # persists Let's Encrypt certs
      - traefik-logs:/var/log/traefik
    networks:
      - proxy
    command:
      - '--api.insecure=false'
      - '--api.dashboard=true'
      - '--providers.docker=true'
      - '--providers.docker.exposedbydefault=false'
      - '--providers.docker.network=proxy'
      - '--entrypoints.web.address=:80'
      - '--entrypoints.websecure.address=:443'
      - '--entrypoints.web.http.redirections.entryPoint.to=websecure'
      - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge=true'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web'
      - '--certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:-admin@example.com}'
      - '--certificatesresolvers.letsencrypt.acme.storage=/etc/traefik/acme.json'
      - '--log.level=INFO'
      - '--accesslog=true'
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      # Traefik's own dashboard
      - "traefik.http.routers.traefik-dash.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.traefik-dash.entrypoints=websecure"
      - "traefik.http.routers.traefik-dash.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik-dash.service=api@internal"
</code></pre>
<p>Bring it up:</p>
<pre><code class="language-bash">cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d traefik
</code></pre>
<p>Watch the logs the first time — Traefik will request a cert for the dashboard host as soon as DNS resolves.</p>
<pre><code class="language-bash">docker logs -f projects-traefik-staging
</code></pre>
<p><strong>Tip.</strong> While testing, switch ACME to staging endpoint (<code>acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory</code>) so you don't burn through Let's Encrypt's rate limits if you misconfigure DNS. Remove that flag before going live.</p>
<h2 id="heading-5-run-jenkins-in-docker">5. Run Jenkins in Docker</h2>
<p>Add this Jenkins service to the same <code>docker-compose.staging.yml</code>. Every line matters (and the comments explain why).</p>
<pre><code class="language-yaml">  jenkins:
    image: jenkins/jenkins:lts
    container_name: projects-jenkins-staging
    restart: unless-stopped
    user: root                           # to use host docker.sock without UID juggling
    environment:
      - JAVA_OPTS=-Xmx1g -Xms512m -Duser.timezone=Asia/Dhaka
      - TZ=Asia/Dhaka                    # OS-level timezone inside container
      - JENKINS_OPTS=--prefix=/
    ports:
      - "3095:8080"                      # web UI (also reachable directly if needed)
      - "50000:50000"                    # inbound agent port
    volumes:
      - jenkins-data:/var/jenkins_home   # Jenkins config/jobs/secrets persistence
      - /var/run/docker.sock:/var/run/docker.sock                          # control host Docker
      - /usr/bin/docker:/usr/bin/docker                                     # docker CLI from host
      - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro  # docker compose plugin
      - /home/developer/projects:/projects                                # project tree
      - /etc/localtime:/etc/localtime:ro                                    # match host clock
      - /etc/timezone:/etc/timezone:ro
    networks:
      - proxy
      - internal
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:8080/login']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s
    deploy:
      resources:
        limits:
          memory: 1024M
</code></pre>
<p><strong>Why</strong> <code>user: root</code><strong>?</strong> It's the simplest way to share <code>docker.sock</code> and the project bind-mount without UID/GID gymnastics. If you prefer an unprivileged user, you'll need to set <code>group: docker</code> and align UIDs/perms on host folders — possible but out of scope here.</p>
<h2 id="heading-6-expose-jenkins-on-a-domain-via-traefik">6. Expose Jenkins on a Domain via Traefik</h2>
<p>This is the section many guides skip. We'll add <strong>labels</strong> to the Jenkins service so Traefik picks it up automatically. No editing of Traefik config required.</p>
<pre><code class="language-yaml">  jenkins:
    # ... everything above ...
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # 1) Router — match incoming Host
      - "traefik.http.routers.jenkins.rule=Host(`jenkins.example.com`)"
      - "traefik.http.routers.jenkins.entrypoints=websecure"
      - "traefik.http.routers.jenkins.tls.certresolver=letsencrypt"
      - "traefik.http.routers.jenkins.service=jenkins"

      # 2) Service — tell Traefik which container port is the app
      - "traefik.http.services.jenkins.loadbalancer.server.port=8080"

      # 3) Middleware — Jenkins needs X-Forwarded-Proto so it knows it's behind HTTPS
      - "traefik.http.middlewares.jenkins-headers.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.jenkins.middlewares=jenkins-headers"
</code></pre>
<p>What each line does:</p>
<table>
<thead>
<tr>
<th>Label</th>
<th>Purpose</th>
</tr>
</thead>
<tbody><tr>
<td><code>traefik.enable=true</code></td>
<td>Opts this container in (we set <code>exposedByDefault=false</code>).</td>
</tr>
<tr>
<td><code>traefik.docker.network=proxy</code></td>
<td>Tells Traefik which network to talk to Jenkins on (Jenkins is on both <code>proxy</code> and <code>internal</code>).</td>
</tr>
<tr>
<td><code>routers.jenkins.rule=Host(...)</code></td>
<td>Forwards only this hostname to Jenkins.</td>
</tr>
<tr>
<td><code>routers.jenkins.entrypoints=websecure</code></td>
<td>Listens only on 443. (HTTP redirect was set up in section 4.)</td>
</tr>
<tr>
<td><code>routers.jenkins.tls.certresolver=letsencrypt</code></td>
<td>Auto-issues + renews the cert.</td>
</tr>
<tr>
<td><code>services.jenkins.loadbalancer.server.port=8080</code></td>
<td>Jenkins listens on 8080 inside the container.</td>
</tr>
<tr>
<td><code>customrequestheaders.X-Forwarded-Proto=https</code></td>
<td>Without this, Jenkins generates <code>http://</code> URLs in webhooks/links and breaks.</td>
</tr>
</tbody></table>
<p>Bring Jenkins up:</p>
<pre><code class="language-bash">cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d jenkins

# Watch Traefik issue the certificate
docker logs -f projects-traefik-staging | grep -i acme
</code></pre>
<p>After 10–60 seconds you should be able to open <code>https://jenkins.example.com</code> and see Jenkins's setup wizard with a valid lock icon.</p>
<p>Inside Jenkins (after first login):</p>
<p>Manage Jenkins → System → Jenkins URL → set this to: <a href="https://jenkins.example.com/">https://jenkins.example.com/</a></p>
<p>This is important because Jenkins uses this base URL to generate:</p>
<ul>
<li><p>Webhook endpoints (for GitHub triggers)</p>
</li>
<li><p>Links inside emails and build logs</p>
</li>
</ul>
<p>If this isn't set correctly, GitHub webhooks may fail, and any links Jenkins generates will point to the wrong address (often localhost or internal IPs).</p>
<h2 id="heading-7-first-time-jenkins-setup">7. First-Time Jenkins Setup</h2>
<p>If you're running Jenkins for the first time on this server, follow this section to complete the initial setup.</p>
<p>If you already have Jenkins configured, you can skip this section — but make sure the required plugins and settings match what we use later in this guide.</p>
<ol>
<li><p>Open <code>https://jenkins.example.com</code>. Get the initial admin password:</p>
<pre><code class="language-bash">docker exec projects-jenkins-staging cat /var/jenkins_home/secrets/initialAdminPassword
</code></pre>
</li>
<li><p>Paste it, choose Install suggested plugins.</p>
</li>
<li><p>Create your admin user.</p>
</li>
<li><p>Manage Jenkins → Plugins → Available and install:</p>
<ul>
<li><p>GitHub (and GitHub Branch Source)</p>
</li>
<li><p>Pipeline: GitHub</p>
</li>
<li><p>Credentials Binding (usually preinstalled)</p>
</li>
</ul>
</li>
</ol>
<p>That's all the plugins you need for the rest of this guide.</p>
<h2 id="heading-8-add-the-github-credential">8. Add the GitHub Credential</h2>
<p>Jenkins needs permission to access your GitHub repository.</p>
<p>This is done using a GitHub Personal Access Token (PAT), which acts like a password for secure API and Git operations.</p>
<p>We’ll store this token inside Jenkins as a credential so it can pull code during pipeline execution and authenticate securely without exposing secrets in code.</p>
<p>This single credential is used both for the SCM checkout and for the deploy-time <code>git pull</code>.</p>
<ol>
<li><p>Create a Personal Access Token (classic) on GitHub with <code>repo</code> scope.</p>
</li>
<li><p>In Jenkins: Manage Jenkins → Credentials → System → Global → Add Credentials.</p>
</li>
<li><p>Fill in:</p>
<ul>
<li><p>Kind: Username with password</p>
</li>
<li><p>Username: your GitHub username</p>
</li>
<li><p>Password: the token</p>
</li>
<li><p><strong>ID:</strong> <code>github_classic_token</code> <em>(the Jenkinsfile references this exact ID)</em></p>
</li>
</ul>
</li>
</ol>
<h2 id="heading-9-create-the-pipeline-job">9. Create the Pipeline Job</h2>
<p>Now that Jenkins has access to your repository, the next step is to define how deployments should run.</p>
<p>A pipeline job tells Jenkins:</p>
<ul>
<li><p>where your code lives,</p>
</li>
<li><p>which branch to monitor,</p>
</li>
<li><p>and how to execute your deployment process.</p>
</li>
</ul>
<p>In Jenkins, create a new Pipeline job and connect it to your GitHub repository. Once this is set up, Jenkins will automatically trigger deployments whenever you push to the <code>staging</code> branch.</p>
<p>Start by creating a new job:</p>
<p>New Item → Pipeline → name it <code>projects-staging</code> → OK</p>
<p>Then configure the job:</p>
<ul>
<li><p>Under <strong>Build Triggers</strong>, enable:<br><strong>GitHub hook trigger for GITScm polling</strong></p>
</li>
<li><p>Under <strong>Pipeline</strong>:</p>
<ul>
<li><p>Definition: Pipeline script from SCM</p>
</li>
<li><p>SCM: Git</p>
</li>
<li><p>Repository URL: <code>https://github.com/&lt;org&gt;/projects-backend.git</code></p>
</li>
<li><p>Credentials: <code>github_classic_token</code></p>
</li>
<li><p>Branch: <code>*/staging</code></p>
</li>
<li><p>Script Path: <code>Jenkinsfile</code></p>
</li>
</ul>
</li>
</ul>
<p>Save the configuration.</p>
<p>At this point, Jenkins is fully connected to your repository and ready to run your deployment pipeline automatically.</p>
<h2 id="heading-10-the-jenkinsfile-deploy-only-what-changed">10. The Jenkinsfile (Deploy Only What Changed)</h2>
<p>Place this at the root of the <strong>app</strong> repo (<code>projects-backend/Jenkinsfile</code>), branch <code>staging</code>.</p>
<pre><code class="language-groovy">pipeline {
  agent any

  environment {
    PROJECT_PATH = "/projects/projects-prod-configs/projects-backend"
    COMPOSE_FILE = "docker-compose.staging.yml"
  }

  stages {

    stage('Checkout') {
      steps {
        checkout scm
        echo "Checkout completed for branch: ${env.BRANCH_NAME ?: 'staging'}"
      }
    }

    stage('Detect Changes') {
      steps {
        script {
          def changedFiles = sh(
            script: "git diff --name-only HEAD~1 HEAD",
            returnStdout: true
          ).trim()

          echo "Changed files:\n${changedFiles}"

          def services = [] as Set
          changedFiles.split('\n').each { file -&gt;
            def svc  = file =~ /^apps\/services\/([a-z0-9-]+)\//
            def gw   = file =~ /^apps\/gateways\/([a-z0-9-]+)\//
            def core = file =~ /^apps\/core\/([a-z0-9-]+)\//
            if (svc)  { services &lt;&lt; svc[0][1]  }
            if (gw)   { services &lt;&lt; gw[0][1]   }
            if (core) { services &lt;&lt; core[0][1] }
          }
          services = services.findAll { !it.endsWith('-e2e') }
          env.CHANGED_SERVICES = services.join(' ')

          echo "Services to deploy: ${env.CHANGED_SERVICES ?: '(none)'}"
        }
      }
    }

    stage('Deploy') {
      when { expression { return env.CHANGED_SERVICES?.trim() } }
      steps {
        withCredentials([usernamePassword(
          credentialsId: 'github_classic_token',
          usernameVariable: 'GIT_USER',
          passwordVariable: 'GIT_TOKEN'
        )]) {
          sh '''
            set -eu
            git config --global --add safe.directory "${PROJECT_PATH}"
            cd "${PROJECT_PATH}"
            git remote set-url origin "https://github.com/&lt;org&gt;/projects-backend.git"
            git -c credential.helper= \
                -c "credential.helper=!f() { echo username=\({GIT_USER}; echo password=\){GIT_TOKEN}; }; f" \
                pull origin staging
            docker compose -f "\({COMPOSE_FILE}" up -d --build \){CHANGED_SERVICES}
          '''
        }
        echo "Deployed: ${env.CHANGED_SERVICES}"
      }
    }

    stage('Skip Deployment') {
      when { expression { return !env.CHANGED_SERVICES?.trim() } }
      steps { echo "No service changes detected — nothing to deploy." }
    }
  }
}
</code></pre>
<p>Why each tricky line is there:</p>
<ul>
<li><p><code>git config --global --add safe.directory ...</code> — git refuses to operate on a repo whose owner UID differs from the current user's. The repo on disk is owned by <code>developer</code>, but Git inside the container runs as <code>root</code>. This whitelists the path.</p>
</li>
<li><p><code>git remote set-url origin "https://..."</code> — flips the on-disk remote to HTTPS so the <strong>token can be used</strong>. (A PAT can't authenticate <code>git@github.com:</code> URLs — those use SSH.) Idempotent — safe to re-run.</p>
</li>
<li><p><code>git -c credential.helper="!f() { echo username=...; echo password=...; }; f"</code> — feeds the username/token to git for that one command without writing the token to disk and without exposing it on the process command line.</p>
</li>
<li><p><code>${CHANGED_SERVICES}</code> is unquoted on purpose so multiple service names expand as separate args.</p>
</li>
</ul>
<h2 id="heading-11-end-to-end-test">11. End-to-End Test</h2>
<p>Before considering the setup complete, we need to verify that the entire pipeline works as expected.</p>
<p>This end-to-end test ensures that:</p>
<ul>
<li><p>GitHub webhooks are triggering Jenkins correctly,</p>
</li>
<li><p>Jenkins can detect which services changed,</p>
</li>
<li><p>and only the affected services are rebuilt and deployed.</p>
</li>
</ul>
<p>In other words, this simulates a real production deployment.</p>
<p>Start by making a small change in your repository. For example, modify a file inside:</p>
<p>apps/gateways/student-apigw/</p>
<p>Then push the change to the <code>staging</code> branch.</p>
<p>Once pushed, Jenkins should automatically trigger via the webhook. If not, you can manually click <strong>Build Now</strong>.</p>
<p>Now open the build’s <strong>Console Output</strong> and verify the flow. You should see something like:</p>
<ul>
<li><p>Checkout completed for branch: staging</p>
</li>
<li><p>Services to deploy: student-apigw</p>
</li>
<li><p>git pull origin staging (successful)</p>
</li>
<li><p>docker compose ... up -d --build student-apigw</p>
</li>
<li><p>Deployed: student-apigw</p>
</li>
</ul>
<p>If you see this sequence, your pipeline is working correctly.</p>
<p>If anything fails, don’t worry — jump to Section 12 where every common issue and its fix is documented.</p>
<h2 id="heading-12-troubleshooting-every-error-we-hit">12. Troubleshooting — Every Error We Hit</h2>
<p>This section covers real issues we faced while setting up this pipeline — and more importantly, <em>why each fix works</em>. Understanding the “why” will help you debug similar problems in your own setup.</p>
<h3 id="heading-cd-cant-cd-to-projectsprojects-prod-configsprojects-backend">cd: can't cd to /projects/projects-prod-configs/projects-backend</h3>
<p><strong>Cause:</strong><br>The Jenkinsfile runs <code>cd $PROJECT_PATH</code>, but inside the container that path doesn’t exist. This usually happens when:</p>
<ul>
<li><p>the project wasn’t cloned on the host, or</p>
</li>
<li><p>the bind mount isn’t configured correctly.</p>
</li>
</ul>
<p><strong>Fix:</strong></p>
<pre><code class="language-bash">ls /home/developer/projects/projects-prod-configs/projects-backend
# If missing: git clone -b staging &lt;url&gt; there.
</code></pre>
<p>Confirm the bind mount:</p>
<pre><code class="language-plaintext">docker inspect projects-jenkins-staging --format '{{range .Mounts}}{{.Source}} -&gt; {{.Destination}}{{println}}{{end}}'
</code></pre>
<p>If missing, recreate the container:</p>
<pre><code class="language-plaintext">docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins
</code></pre>
<p><strong>Why this works:</strong></p>
<p>Jenkins runs inside a container, but your code lives on the host. The bind mount connects them. Without it, Jenkins cannot access your project directory.</p>
<h3 id="heading-fatal-detected-dubious-ownership-in-repository">fatal: detected dubious ownership in repository</h3>
<p><strong>Cause:</strong><br>Git blocks access when the repository owner differs from the current user.</p>
<ul>
<li><p>Repo owner: <code>developer</code> (host)</p>
</li>
<li><p>Git runs as: <code>root</code> (inside container)</p>
</li>
</ul>
<p><strong>Fix:</strong></p>
<pre><code class="language-plaintext">git config --global --add safe.directory "${PROJECT_PATH}"
</code></pre>
<p><strong>Why this works:</strong></p>
<p>This explicitly tells Git that the directory is trusted, bypassing ownership mismatch security restrictions.</p>
<h3 id="heading-host-key-verification-failed-could-not-read-from-remote-repository"><code>Host key verification failed</code> / <code>Could not read from remote repository</code></h3>
<h4 id="heading-cause">Cause:</h4>
<p>The repository uses SSH (<code>git@github.com:...</code>), but:</p>
<ul>
<li><p>the container has no SSH keys</p>
</li>
<li><p>no known_hosts file exists</p>
</li>
</ul>
<p>Also, GitHub tokens cannot authenticate over SSH.</p>
<p><strong>Fix (recommended):</strong></p>
<pre><code class="language-plaintext">git remote set-url origin "https://github.com/&lt;org&gt;/projects-backend.git"
</code></pre>
<p><strong>Why this works:</strong></p>
<p>HTTPS uses token-based authentication (PAT), which works inside containers without SSH configuration.</p>
<h3 id="heading-unknown-shorthand-flag-f-in-f-docker-compose"><code>unknown shorthand flag: 'f' in -f</code> ( <code>docker compose</code>)</h3>
<p><strong>Cause:</strong><br>The Docker CLI exists, but the Docker Compose plugin is missing inside the container.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-plaintext">volumes:
  - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro
</code></pre>
<p>Find your path if needed:</p>
<pre><code class="language-plaintext">find /usr -name docker-compose -type f 2&gt;/dev/null
</code></pre>
<p>Verify:</p>
<pre><code class="language-plaintext">docker exec projects-jenkins-staging docker compose version
</code></pre>
<p><strong>Why this works:</strong></p>
<p>Docker Compose v2 is a CLI plugin. Mounting this directory makes the <code>docker compose</code> command available inside the container.</p>
<h3 id="heading-wrong-timezone-in-build-timestamps-and-jenkins-ui">Wrong timezone in build timestamps and Jenkins UI</h3>
<p><strong>Fix:</strong> Set both env var and JVM flag, and bind-mount the host's clock files:</p>
<pre><code class="language-yaml">environment:
  - TZ=Asia/Dhaka
  - JAVA_OPTS=... -Duser.timezone=Asia/Dhaka
volumes:
  - /etc/localtime:/etc/localtime:ro
  - /etc/timezone:/etc/timezone:ro
</code></pre>
<p>You <strong>must</strong> recreate the container for env-var changes to take effect:</p>
<pre><code class="language-bash">docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins
</code></pre>
<p><strong>Why this works:</strong><br>Jenkins runs on Java, which uses its own timezone separate from the OS.<br>By aligning OS timezone, JVM timezone, and host clock, you ensure consistent timestamps everywhere.</p>
<h3 id="heading-errsockettimeout-pnpm-install-fails">ERR_SOCKET_TIMEOUT (pnpm install fails)</h3>
<h4 id="heading-cause">Cause:</h4>
<p>If you have multiple services building in parallel and each runs pnpm install with ~1500 packages, the network gets saturated and a timeout occurs.</p>
<h4 id="heading-fixes">Fixes:</h4>
<p>a) Increase timeout + control concurrency</p>
<pre><code class="language-xml">RUN pnpm install --frozen-lockfile --ignore-scripts 
--network-timeout 600000 
--network-concurrency 8
</code></pre>
<p>Why: Gives pnpm more time and reduces network overload.</p>
<p>b) Enable pnpm cache (BuildKit)</p>
<pre><code class="language-xml">RUN --mount=type=cache,id=pnpm-store,target=/root/.local/share/pnpm/store 
pnpm install --frozen-lockfile --ignore-scripts
</code></pre>
<p>Why: Dependencies are cached and reused instead of downloading every time.</p>
<p>c) Avoid unnecessary rebuilds</p>
<pre><code class="language-xml">docker compose -f \(COMPOSE_FILE build \)CHANGED_SERVICES docker compose -f \(COMPOSE_FILE up -d --no-build \)CHANGED_SERVICES
</code></pre>
<p>Why: Only changed services are rebuilt → less network load → fewer failures.</p>
<h3 id="heading-container-changes-dont-apply-after-editing-docker-composeyml">Container changes don’t apply after editing docker-compose.yml</h3>
<h4 id="heading-cause">Cause:</h4>
<p>Docker compose up -d does not update running containers.</p>
<h4 id="heading-fix">Fix:</h4>
<pre><code class="language-xml">docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins
</code></pre>
<p><strong>Why this works:</strong></p>
<p>This forces Docker to recreate the container with updated configuration (env, volumes, labels).</p>
<h3 id="heading-traefik-shows-default-certificate-no-https">Traefik shows default certificate (no HTTPS)</h3>
<h4 id="heading-common-causes">Common causes:</h4>
<p>DNS not pointing to server Port 80 blocked Wrong Docker network</p>
<h4 id="heading-check">Check:</h4>
<pre><code class="language-xml">dig +short jenkins.example.com docker logs projects-traefik-staging 2&gt;&amp;1 | grep -i acme
</code></pre>
<p><strong>Why this works:</strong></p>
<p>Let’s Encrypt uses HTTP-01 challenge, so it must reach your server via port 80. If DNS or networking is wrong, certificate issuance fails.</p>
<h3 id="heading-jenkins-reverse-proxy-setup-is-broken">Jenkins: "Reverse proxy setup is broken"</h3>
<h4 id="heading-fix">Fix:</h4>
<p>Set the Jenkins URL to <a href="https://jenkins.example.com/">https://jenkins.example.com/</a><br>Ensure header:</p>
<pre><code class="language-xml">X-Forwarded-Proto: https
</code></pre>
<p><strong>Why this works:</strong></p>
<p>Jenkins needs to know it's behind HTTPS. Without this, it generates incorrect URLs (http instead of https), breaking redirects and webhooks.</p>
<h2 id="heading-13-mental-model-host-vs-container">13. Mental Model: Host vs. Container</h2>
<p>Many setup mistakes come from confusing the <strong>host</strong> filesystem with the <strong>container</strong> filesystem. This table makes it explicit:</p>
<table>
<thead>
<tr>
<th>Inside the Jenkins container</th>
<th>Comes from on the host</th>
</tr>
</thead>
<tbody><tr>
<td><code>/var/jenkins_home</code></td>
<td>docker volume <code>jenkins-data</code> (Jenkins config, jobs, secrets)</td>
</tr>
<tr>
<td><code>/projects/...</code></td>
<td><code>/home/developer/projects/...</code> (your project tree)</td>
</tr>
<tr>
<td><code>/usr/bin/docker</code></td>
<td>host's <code>/usr/bin/docker</code></td>
</tr>
<tr>
<td><code>/usr/libexec/docker/cli-plugins/docker-compose</code></td>
<td>host plugin (lets <code>docker compose</code> work)</td>
</tr>
<tr>
<td><code>/var/run/docker.sock</code></td>
<td>host Docker daemon (so builds happen on the host's engine)</td>
</tr>
<tr>
<td><code>/etc/localtime</code>, <code>/etc/timezone</code></td>
<td>host clock</td>
</tr>
<tr>
<td><code>~/.ssh</code></td>
<td><strong>nothing</strong> — that's why SSH-to-GitHub doesn't work without extra setup</td>
</tr>
</tbody></table>
<p>When debugging, always ask: <em>"Inside which filesystem is this command running, and does the file/folder it's looking for exist there?"</em></p>
<h2 id="heading-14-daily-operations-cheat-sheet">14. Daily Operations Cheat Sheet</h2>
<pre><code class="language-bash"># Recreate Jenkins after changing compose
cd /home/developer/Projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

# Tail Jenkins logs
docker logs -f projects-jenkins-staging

# Open a shell inside the Jenkins container
docker exec -it projects-jenkins-staging bash

# From inside the container — sanity checks
docker compose version
ls /projects/projects-prod-configs/projects-backend
git -C /projects/projects-prod-configs/projects-backend remote -v

# Manually trigger the same deploy the pipeline does
cd /projects/projects-configs/projects-backend
git pull origin staging
docker compose -f docker-compose.staging.yml up -d --build student-apigw

# Inspect Traefik routing decisions
docker logs projects-traefik-staging 2&gt;&amp;1 | grep -i jenkins

# Check renewed certs
docker exec projects-traefik-staging cat /etc/traefik/acme.json | head -50
</code></pre>
<h2 id="heading-15-what-id-do-differently-next-time">15. What I'd Do Differently Next Time</h2>
<ul>
<li><p><strong>Pre-build a base image</strong> with all node_modules baked in. With ~1500 packages × 15 services, every clean build re-downloads ~22k tarballs. A shared base cuts that 90%.</p>
</li>
<li><p><strong>Run a private npm proxy</strong> (Verdaccio / Nexus / GitHub Packages) on the same Docker network — eliminates flaky <code>npmjs.org</code> timeouts entirely.</p>
</li>
<li><p><strong>Per-service Jenkinsfile</strong> if your services drift apart in tooling. With one Jenkinsfile, every team contends for the same pipeline definition.</p>
</li>
<li><p><strong>Replace</strong> <code>git diff HEAD~1 HEAD</code> with <code>git diff $(git merge-base HEAD origin/staging~1) HEAD</code> so squash-merges and force-pushes don't accidentally skip services.</p>
</li>
<li><p><strong>Move secrets to a vault</strong> (HashiCorp Vault / AWS Secrets Manager / Doppler). PATs in Jenkins work, but rotation across many jobs is painful.</p>
</li>
<li><p><strong>Use Jenkins' Configuration-as-Code (JCasC)</strong> so the entire Jenkins setup (jobs, credentials definitions, plugins) is in git. Then a server rebuild is a one-command operation.</p>
</li>
</ul>
<h2 id="heading-closing-thoughts">Closing Thoughts</h2>
<p>The pipeline itself is just three stages: <strong>Checkout → Detect Changes → Deploy</strong> — but a real production setup is mostly about <strong>plumbing</strong>: reverse proxy, certificates, bind-mounts, credentials, timezones, build caches. None of these are exotic. Together they decide whether your Friday-afternoon deploy goes silently green or eats your weekend.</p>
<p>Follow sections 1–11 to get a working pipeline. Bookmark section 12 to keep it working.</p>
<p>Happy shipping.</p>
 ]]>
                </content:encoded>
            </item>
        
    </channel>
</rss>
