But the official WhatsApp Business Cloud API can be slow to onboard, template-restricted for proactive messages, and priced per conversation — which adds up fast at scale.

There's another path: you can run your own WhatsApp HTTP gateway on a small server, connect it to a workflow engine, and keep every message — inbound and outbound — inside infrastructure you control. No monthly conversation fees, no template approvals for routine replies, no third-party middleman holding your customer data.

In this tutorial, you'll build exactly that. By the end, you'll have a WhatsApp bot that:

• Receives every incoming message through a webhook

• Routes messages through an n8n workflow

• Replies automatically based on keywords, AI, or any API call you want

• Runs entirely on your own server, using two open-source tools

You'll use WAHA (WhatsApp HTTP API) as the gateway, and n8n as the workflow engine. Both run in Docker, both are free for self-hosting, and together they cover everything from a simple auto-reply to a full CRM integration.

Table of contents

• What You'll Learn

• Prerequisites

• A Note on Which WhatsApp Account to Use

• WAHA vs the official WhatsApp Business Cloud API

• Part 1: Understanding WAHA

• Part 2: Running WAHA with Docker

• Part 3: Starting a WhatsApp session

• Part 4: Running n8n

• Part 5: Creating the Webhook Trigger in n8n

• Part 6: Wiring WAHA to n8n

• Part 7: Building the first auto-reply

• Part 8: A Second Example — Proactive Booking Confirmations

• Part 9: Going to Production

• Common Pitfalls

• Where to Go Next

What You'll Learn

• How WAHA works under the hood and when to use it instead of the official Cloud API

• How to run WAHA and n8n side by side with Docker Compose

• How to scan the QR code and bind a WhatsApp account to your gateway

• How to connect WAHA's webhook to an n8n workflow

• How to build a keyword-based auto-reply bot

• How to send proactive confirmations from a separate workflow

• How to harden the setup for production (HTTPS, API keys, rate limits, Queue Mode)

Prerequisites

• A Linux server (any VPS works — 2 GB of RAM is enough for a small bot)

• Docker and Docker Compose installed

• A public hostname with DNS pointing at the server, or an ngrok tunnel for local testing

• A WhatsApp account you're willing to dedicate to the bot (more on that below)

• Basic familiarity with JSON and HTTP requests

You don't need prior n8n experience. If you can drag a box and wire it to another box, you can build the flow.

A Note on Which WhatsApp Account to Use

WAHA works by running an actual WhatsApp Web session inside a headless Chromium process. It logs in as a real account — the same way you would open web.whatsapp.com in your browser. Meta doesn't officially endorse this approach for commercial use at scale, and heavy volume from a single number can lead to a ban.

For that reason, use a dedicated number for the bot. Don't use your personal WhatsApp. Get a second SIM, eSIM, or a VoIP number that supports WhatsApp activation. Keep outbound volume reasonable, and you'll be fine for most small-business use cases.

If you plan to send thousands of marketing messages per day, switch to the official WhatsApp Business Cloud API — that's what it exists for. This tutorial is aimed at the middle ground: support bots, order updates, booking confirmations, and similar conversational flows where you need real-time control without enterprise pricing.

WAHA vs the official WhatsApp Business Cloud API

Before writing any code, it helps to understand when each option is the right fit.

Neither is strictly better. If you run a support team for a small business, WAHA is often the pragmatic choice. If you're a bank sending millions of transactional messages, you want the Cloud API. Many teams run both — WAHA for conversational support, Cloud API for bulk transactional traffic.

Part 1: Understanding WAHA

WAHA is an open-source project that wraps WhatsApp Web behind a clean REST API. You POST /api/sendText with a chat ID and a message, and WAHA sends it. You configure a webhook URL, and WAHA POSTs to that URL every time a message arrives.

Under the hood, WAHA spawns a Chromium instance, opens WhatsApp Web, and uses an engine (whatsapp-web.js, NOWEB, or GOWS) to automate the session. Your code doesn't see any of that complexity — you just see an HTTP API.

The project ships in two flavors:

• WAHA Core — free, MIT licensed, one active session per container, community support.

• WAHA Plus — commercial license, multi-session support, priority support, and access to advanced endpoints.

For most developers building a single bot, Core is enough. You can always upgrade later.

Official docs live at waha.devlike.pro. Keep that open in another tab — we'll reference specific endpoints as we go.

Part 2: Running WAHA with Docker

Create a fresh directory for the project:

mkdir whatsapp-bot && cd whatsapp-bot

Create a docker-compose.yml file:

services:
  waha:
    image: devlikeapro/waha:latest
    container_name: waha
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - WAHA_DASHBOARD_ENABLED=true
      - WAHA_DASHBOARD_USERNAME=admin
      - WAHA_DASHBOARD_PASSWORD=change-me-now
      - WHATSAPP_API_KEY=super-secret-key-change-me
      - WHATSAPP_DEFAULT_ENGINE=WEBJS
    volumes:
      - ./waha-sessions:/app/.sessions

A few things to notice:

• The dashboard username and password protect the web UI at http://your-server:3000. Always change the defaults before you expose the port publicly.

• WHATSAPP_API_KEY is the key every HTTP request to WAHA must include in the X-Api-Key header. Treat it like a database password.

• WHATSAPP_DEFAULT_ENGINE=WEBJS uses the mature whatsapp-web.js engine. WAHA also supports NOWEB and GOWS engines with different trade-offs — WEBJS is the safest default for a first deployment.

• The volume mount persists the session across restarts. Without it, every container rebuild forces you to scan the QR code again.

Start the container:

docker compose up -d
docker compose logs -f waha

Within about 20 seconds WAHA finishes booting. Visit http://your-server:3000 and log in with the dashboard credentials.

Part 3: Starting a WhatsApp session

WAHA calls each WhatsApp account a "session." You can have one session at a time on WAHA Core.

From the dashboard, click Start New Session and name it default. WAHA displays a QR code.

On your phone:

1. Open WhatsApp.

2. Tap the three-dot menu (Android) or Settings (iOS).

3. Tap Linked Devices → Link a Device.

4. Point the camera at the QR code on your screen.

Within a few seconds the dashboard shows WORKING status. Your session is live.

You can also do this over the API. Start the session (default is the session name, encoded in the URL path):

curl -X POST http://your-server:3000/api/sessions/default/start \
  -H "X-Api-Key: super-secret-key-change-me"

The call is idempotent — if the session is already running, nothing happens.

Fetch the QR as a PNG:

curl http://your-server:3000/api/default/auth/qr \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Accept: image/png" \
  --output qr.png

Scan and you're in.

Test that the session works by sending a message to yourself:

curl -X POST http://your-server:3000/api/sendText \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "session": "default",
    "chatId": "15555550123@c.us",
    "text": "Hello from WAHA!"
  }'

Replace 15555550123 with your own number (country code plus number, no +, no spaces, no dashes). The @c.us suffix marks it as an individual chat. Groups use @g.us.

If the message lands on your phone — congratulations. The gateway works.

Part 4: Running n8n

Add an n8n service to your docker-compose.yml alongside WAHA:

services:
  waha:
    # ... existing config

  n8n:
    image: n8nio/n8n:latest
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=n8n.example.com
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - WEBHOOK_URL=https://n8n.example.com/
      - GENERIC_TIMEZONE=UTC
    volumes:
      - ./n8n-data:/home/node/.n8n

Replace n8n.example.com with your real domain. For purely local testing, set:

- N8N_HOST=localhost
- N8N_PROTOCOL=http
- WEBHOOK_URL=http://localhost:5678/

If you want to test webhooks from your laptop without a server, run ngrok http 5678 in another terminal and use the ngrok HTTPS URL as WEBHOOK_URL. n8n uses WEBHOOK_URL to tell external services where to POST — get this wrong and your webhooks will 404.

Start the stack:

docker compose up -d

Visit http://your-server:5678. On the first visit, n8n walks you through creating an owner account (email and password). Every subsequent visit requires that login. For extra safety in production, put n8n behind a reverse proxy with an allow-list or an additional auth layer — we'll set that up later.

Part 5: Creating the Webhook Trigger in n8n

Click Create Workflow. You'll see an empty canvas.

Add a Webhook node and configure it:

• HTTP Method: POST

• Path: whatsapp (this becomes part of the URL)

• Response Mode: Respond Immediately

• Response Data: First Entry JSON

Click Listen for Test Event. n8n shows you two URLs: a test URL and a production URL. Copy the production URL. It looks like this:

https://n8n.example.com/webhook/whatsapp

Not webhook-test — that one only fires while the editor is open. You want webhook.

Part 6: Wiring WAHA to n8n

WAHA can POST to a webhook on every WhatsApp event. Tell it where to send those events.

In the WAHA dashboard, open your session and set the webhook URL. Or do it over the API:

curl -X PUT http://your-server:3000/api/sessions/default \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "webhooks": [
        {
          "url": "https://n8n.example.com/webhook/whatsapp",
          "events": ["message", "session.status"]
        }
      ]
    }
  }'

The message event fires on every inbound message. session.status fires when the session connects, disconnects, or reconnects — which is useful for alerting when your bot goes down.

Test it. From another phone, send a WhatsApp message to your bot's number. Head back to the n8n editor. Within a second or two the webhook node lights up with the event data.

The payload looks roughly like this:

{
  "event": "message",
  "session": "default",
  "payload": {
    "id": "false_15555550123@c.us_3EB0...",
    "from": "15555550123@c.us",
    "body": "Hello",
    "timestamp": 1713801234,
    "fromMe": false
  }
}

Everything you need is in payload: who sent it (from), what they said (body), and when (timestamp).

Part 7: Building the first auto-reply

A bot that only listens is boring. Let's make it answer.

You'll build a tiny keyword router: if the user sends hi or hello, the bot greets them. If they send price, it sends a pricing message. Anything else gets a fallback.

After the Webhook node, add a Switch node.

Configure the Switch node:

• Mode: Expression

• Value: {{ $json.payload.body.toLowerCase().trim() }}

• Add routing rules:

  • Rule 1: equals hi — output 0

  • Rule 2: equals hello — output 0

  • Rule 3: equals price — output 1

  • Fallback output: 2

After the Switch, add three HTTP Request nodes, one per output.

Configure each HTTP Request node identically, except for the body text:

• Method: POST

• URL: http://waha:3000/api/sendText (inside the Docker network you can reach WAHA by its service name. From outside use the full public URL)

• Send Headers: on

  • X-Api-Key: super-secret-key-change-me

  • Content-Type: application/json

• Send Body: on

  • Body Content Type: JSON

  • Specify Body: Using JSON

For the greeting node, the JSON body is:

{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "Hi! I'm the bot. Send 'price' to see pricing, or anything else for help."
}

For the pricing node:

{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "Our plans start at $49/month. Reply 'sales' to talk to a human."
}

For the fallback:

{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "I didn't catch that. Try 'hi' or 'price'."
}

The ={{ ... }} syntax is an n8n expression — at runtime it pulls values from earlier nodes.

Connect the Switch outputs to their matching HTTP Request nodes. Save the workflow. Click Activate in the top-right.

Send hi to your bot from any phone. It should reply within a second.

Congratulations — you have a WhatsApp bot running entirely on your own infrastructure.

Part 8: A Second Example — Proactive Booking Confirmations

Auto-reply is useful. Proactive outbound is where the value really compounds. Here's a second workflow that sends a booking confirmation whenever a new row lands in a database.

Create a second workflow in n8n. Use one of these triggers:

• Schedule Trigger — poll a database every minute for new rows

• Webhook Trigger — listen for a notification from your booking system

• Database Trigger (Postgres, MySQL, Supabase) — react to inserts in real time

For this example, use a Schedule Trigger set to every minute, followed by a Postgres Execute Query node that reads pending confirmations:

SELECT id, customer_phone, service_name, booking_time
FROM bookings
WHERE confirmation_sent = false
LIMIT 20;

After the Postgres node, add an HTTP Request node pointing to the same WAHA sendText endpoint you used earlier. The body:

{
  "session": "default",
  "chatId": "={{ $json.customer_phone }}@c.us",
  "text": "Hi! Your booking for {{ \(json.service_name }} on {{ \)json.booking_time }} is confirmed. Reply 'change' to reschedule."
}

Finally, add a second Postgres node that marks the booking as sent:

UPDATE bookings
SET confirmation_sent = true, confirmation_sent_at = NOW()
WHERE id = {{ $json.id }};

Activate the workflow. Every minute, n8n pulls pending bookings, sends a WhatsApp confirmation, and marks them done.

This pattern generalizes. Replace the SQL with a call to Shopify for order confirmations, Stripe for receipt messages, or Calendly for appointment reminders. The WhatsApp layer stays the same — only the source of truth changes.

Part 9: Going to Production

The setup above works, but it's not yet production-ready. Here's what to harden before you point real customers at it.

1. Put Everything Behind HTTPS

Never expose n8n or WAHA directly on plain HTTP. Put a reverse proxy in front. Caddy is the easiest choice because it handles Let's Encrypt automatically.

A minimal Caddyfile:

n8n.example.com {
    reverse_proxy n8n:5678
}

waha.example.com {
    reverse_proxy waha:3000
}

Run Caddy as another service in the same Docker Compose. TLS certificates are issued and renewed automatically.

2. Rotate the API Keys

Don't ship super-secret-key-change-me to production. Generate a real key:

openssl rand -hex 32

Put it in a .env file, reference it as ${WHATSAPP_API_KEY} in docker-compose.yml, and add .env to your .gitignore.

3. Rate-limit Outbound Messages

WhatsApp bans accounts that send too many messages too fast. A safe outbound rate for a fresh number is well under 20 messages per minute. For bursty replies, add an n8n Wait node between sends, or queue outgoing messages through a small custom function node that sleeps between requests.

4. Scale n8n with Queue Mode

By default, n8n runs everything in a single process. That's fine for low volume. For higher throughput, switch to Queue Mode:

• Add a Redis container.

• Run one n8n main container (the web UI and webhook receiver).

• Run one or more n8n-worker containers that pull jobs from the queue.

Queue Mode is documented at docs.n8n.io/hosting/scaling/queue-mode/. Setup adds two environment variables (EXECUTIONS_MODE=queue, QUEUE_BULL_REDIS_HOST=redis) and decouples incoming webhooks from workflow execution. The webhook responds in milliseconds while workers chew through the queue in the background.

5. Monitor the Session

WhatsApp Web sessions drop. The phone loses connection, WhatsApp rotates security tokens, or your server reboots. Catch those drops early.

Subscribe to the session.status webhook event in WAHA. When status becomes FAILED or STOPPED, route it to an n8n workflow that posts to Slack, sends an email, or pages you. The faster you know, the faster you recover.

For overall uptime, point something like Uptime Kuma at GET /api/sessions/default on WAHA. If WAHA reports WORKING, you're fine. Anything else triggers an alert.

6. Back Up the Sessions Volume

The waha-sessions directory contains the logged-in state. If you lose it, you have to scan the QR code again — possibly from a phone that's no longer handy. Back it up nightly. A simple cron job with tar and rclone to S3-compatible storage is plenty.

7. Add a Live-Agent Handoff

Not every conversation should stay with the bot. When a user types human — or when your intent classifier can't answer confidently — hand off to a real agent.

Chatwoot is a solid open-source option: it has a dedicated WhatsApp channel, agent inbox, team assignment, and conversation history. The handoff is an n8n branch that stops processing bot replies and forwards the message stream to Chatwoot's API.

Common Pitfalls

A few issues catch almost everyone on their first production deploy.

Webhooks Timing Out

WAHA gives your webhook a few seconds to respond. If your n8n workflow is slow (calling an LLM, hitting a remote API), the webhook times out and WAHA retries, potentially causing duplicate replies.

Fix: make the webhook return 200 immediately and offload the slow work. In n8n, set the Webhook node's Response Mode to Using Respond to Webhook Node, add a Respond to Webhook node as the first step with a 200 and empty body, then do the heavy lifting after that.

Duplicate Messages

WAHA delivers the same message event more than once in edge cases (phone comes back online, session reconnects). Store the payload.id somewhere — Redis, a database, or n8n's static data store — and drop any ID you've already processed.

Messages Arriving Out of Order

The webhook is async, and n8n may parallelize executions. If ordering matters — for example, in a multi-step conversation — key a queue by the sender's chatId and process each sender serially.

Sessions Disconnecting After a Phone Restart

Normal WhatsApp Web behavior. WAHA auto-reconnects, but occasionally the linked-devices list needs a manual refresh. If a session refuses to come back, stop the WAHA container, delete that session's folder under waha-sessions/, start the container again, and rescan the QR.

Your Number Gets Banned

The single biggest cause is rate: a new number blasting hundreds of messages an hour gets flagged fast. Warm up a number slowly — send a normal, human-like volume for the first week. Don't send to strangers unsolicited. Prefer inbound-driven replies over outbound pushes wherever you can.

The Wrong Chat ID Format

WhatsApp individual chats use <number>@c.us and groups use <groupId>@g.us. Don't include the + or spaces in the number. If WAHA returns a 404 when sending, the chat ID is almost always the problem.

Where to Go Next

You now have the foundation. The same two-service stack supports almost any bot you can imagine — you're only limited by what you can build in an n8n workflow.

Some natural next steps:

• Plug in AI replies: Add an OpenAI or Anthropic node after the Webhook, pass the user's message through it with a short system prompt, and send the response back through WAHA. Cap conversation length to prevent runaway token usage.

• Integrate a CRM: Look up the caller's chatId in HubSpot, Pipedrive, or your own database before deciding how to reply. Segment responses by customer tier.

• Send proactive notifications: Appointment reminders, shipping updates, payment receipts, abandoned-cart nudges. Keep the content transactional and expected — unsolicited marketing blasts are the fastest way to a ban.

• Log every conversation: Add a Postgres or Supabase node after the Webhook to persist messages for analytics and customer history. Your future self (and your support team) will thank you.

• Add media handling: WAHA exposes sendImage, sendFile, and sendVoice endpoints. Teach the bot to accept photos for support tickets, or send invoices as PDFs directly inside the chat.

The WhatsApp layer stays the same. Everything interesting happens upstream in the workflow.

If you want to see production examples of n8n and WAHA running at scale — or you need a similar automation built for your business — I'm the founder of Achiya Automation, where we ship WhatsApp, n8n, and Chatwoot integrations. You can find more at achiya-automation.com.

We just posted a new course on the freeCodeCamp.org YouTube channel, led by instructor and developer Estafania, that will help you leverage the power of automation to help with all your tasks.

Zapier is a no-code platform that allows you to connect and share information between the applications you use every day. The core philosophy is simple: "If this happens, then do that".

• The Trigger: This is the "If this happens" part. It's a specific event in one app (like receiving a new lead in a form).

• The Action: This is the "do that" part. This is the task Zapier performs automatically in another app (like sending a Slack notification or adding a row to a Google Sheet).

This four-hour course takes you from a complete beginner to an advanced user. You will start by setting up a free account and learning the basic building blocks of a "Zap". As you progress, you will dive into modern, AI-enhanced features.

And for people looking to bridge the gap between AI and development, the course concludes with a deep dive into Model Context Protocol (MCP). You will learn how to set up an MCP server to share information from your apps with AI clients like Visual Studio Code and the Gemini CLI. This allows you to interact with your Google Calendar or GitHub repositories directly through an AI interface.

Watch the full course on the freeCodeCamp.org YouTube channel (4-hour watch).

That work is deterministic. An agent can do it.

In this tutorial, you'll build a local SEO audit agent from scratch using Python, Browser Use, and the Claude API. The agent visits real pages in a visible browser window, extracts SEO signals using Claude, checks for broken links asynchronously, handles edge cases with a human-in-the-loop pause, and writes a structured report — all resumable if interrupted.

By the end, you'll have a working agent you can run against any list of URLs. It costs less than $0.01 per URL to run.

What You'll Build

A seven-module Python agent that:

• Reads a URL list from a CSV file

• Visits each URL in a real Chromium browser (not a headless scraper)

• Extracts title, meta description, H1s, and canonical tag via Claude API

• Checks for broken links asynchronously using httpx

• Detects edge cases (404s, login walls, redirects) and pauses for human input

• Writes results to report.json incrementally — safe to interrupt and resume

• Generates a plain-English report-summary.txt on completion

The full code is on GitHub at dannwaneri/seo-agent.

Prerequisites

• Python 3.11 or higher

• An Anthropic API key (get one at console.anthropic.com)

• Windows, macOS, or Linux

• Basic familiarity with Python and the command line

Table of Contents

1. Why Browser Use Instead of a Scraper

2. Project Structure

3. Setup

4. Module 1: State Management

5. Module 2: Browser Integration

6. Module 3: Claude Extraction Layer

7. Module 4: Broken Link Checker

8. Module 5: Human-in-the-Loop

9. Module 6: Report Writer

10. Module 7: The Main Loop

11. Running the Agent

12. Scheduling for Agency Use

13. What the Results Look Like

Why Browser Use Instead of a Scraper

The standard approach to SEO auditing is to fetch page HTML with requests and parse it with BeautifulSoup. That works on static pages. It breaks on JavaScript-rendered content, misses dynamically injected meta tags, and fails entirely on authenticated pages.

Browser Use (84,000+ GitHub stars, MIT license) takes a different approach. It controls a real Chromium browser, reads the DOM after JavaScript executes, and exposes the page through Playwright's accessibility tree. The agent sees what a human would see.

The practical difference: a requests-based scraper might miss a meta description injected by a React component. Browser Use won't.

The other difference worth naming: Browser Use reads pages semantically. A Playwright script breaks when a button's CSS class changes from btn-primary to button-main. Browser Use identifies it's still a "Submit" button and acts accordingly. The extraction logic lives in the Claude prompt, not in brittle CSS selectors.

Project Structure

seo-agent/
├── index.py          # Main audit loop
├── browser.py        # Browser Use / Playwright page driver
├── extractor.py      # Claude API extraction layer
├── linkchecker.py    # Async broken link checker
├── hitl.py           # Human-in-the-loop pause logic
├── reporter.py       # Report writer
├── state.py          # State persistence (resume on interrupt)
├── input.csv         # Your URL list
├── requirements.txt
├── .env.example
└── .gitignore

Setup

Create a project folder and install dependencies:

mkdir seo-agent && cd seo-agent
pip install browser-use anthropic playwright httpx
playwright install chromium

Create input.csv with your URLs:

url
https://example.com
https://example.com/about
https://example.com/contact

Create .env.example:

ANTHROPIC_API_KEY=your-key-here

Set your API key as an environment variable before running:

# macOS/Linux
export ANTHROPIC_API_KEY="sk-ant-..."

# Windows PowerShell
$env:ANTHROPIC_API_KEY = "sk-ant-..."

Create .gitignore:

state.json
report.json
report-summary.txt
.env
__pycache__/
*.pyc

Module 1: State Management

The agent needs to track which URLs it has already audited. If the run is interrupted — power cut, keyboard interrupt, network error — it should resume from where it stopped, not start over.

state.py handles this with a flat JSON file:

import json
import os

STATE_FILE = os.path.join(os.path.dirname(__file__), "state.json")

_DEFAULT_STATE = {"audited": [], "pending": [], "needs_human": []}


def load_state() -> dict:
    if not os.path.exists(STATE_FILE):
        save_state(_DEFAULT_STATE.copy())
    with open(STATE_FILE, encoding="utf-8") as f:
        return json.load(f)


def save_state(state: dict) -> None:
    with open(STATE_FILE, "w", encoding="utf-8") as f:
        json.dump(state, f, indent=2)


def is_audited(url: str) -> bool:
    return url in load_state()["audited"]


def mark_audited(url: str) -> None:
    state = load_state()
    if url not in state["audited"]:
        state["audited"].append(url)
    save_state(state)


def add_to_needs_human(url: str) -> None:
    state = load_state()
    if url not in state["needs_human"]:
        state["needs_human"].append(url)
    save_state(state)

The design is intentional: mark_audited() is called immediately after a URL is processed and written to the report. If the agent crashes mid-run, it loses at most one URL's work.

Module 2: Browser Integration

browser.py does the actual page navigation. It uses Playwright directly (which Browser Use installs as a dependency) to open a visible Chromium window, navigate to the URL, capture HTTP status and redirect information, and extract the raw SEO signals from the DOM.

The key design decisions:

Visible browser, not headless. Set headless=False so you can watch the agent work. This matters for the demo and for debugging.

Status capture via response listener. Playwright raises an exception on 4xx/5xx responses, but the on("response", ...) handler fires before the exception. We capture status there.

2-second delay between visits. Prevents triggering rate limiting or bot detection on agency client sites.

Here is the core navigation function:

import asyncio
import sys
import time
from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeout

TIMEOUT = 20_000  # 20 seconds


def fetch_page(url: str) -> dict:
    result = {
        "final_url": url,
        "status_code": None,
        "title": None,
        "meta_description": None,
        "h1s": [],
        "canonical": None,
        "raw_links": [],
    }

    first_status = {"code": None}

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()

        def on_response(response):
            if first_status["code"] is None:
                first_status["code"] = response.status

        page.on("response", on_response)

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=TIMEOUT)
            result["status_code"] = first_status["code"] or 200
            result["final_url"] = page.url

            # Extract SEO signals from DOM
            result["title"] = page.title() or None
            result["meta_description"] = page.evaluate(
                "() => { const m = document.querySelector('meta[name=\"description\"]'); "
                "return m ? m.getAttribute('content') : null; }"
            )
            result["h1s"] = page.evaluate(
                "() => Array.from(document.querySelectorAll('h1')).map(h => h.innerText.trim())"
            )
            result["canonical"] = page.evaluate(
                "() => { const c = document.querySelector('link[rel=\"canonical\"]'); "
                "return c ? c.getAttribute('href') : null; }"
            )
            result["raw_links"] = page.evaluate(
                "() => Array.from(document.querySelectorAll('a[href]'))"
                ".map(a => a.href).filter(Boolean).slice(0, 100)"
            )

        except PlaywrightTimeout:
            result["status_code"] = first_status["code"] or 408
        except Exception as exc:
            print(f"[browser] Error: {exc}", file=sys.stderr)
            result["status_code"] = first_status["code"]
        finally:
            browser.close()

    time.sleep(2)
    return result

A few things worth noting:

The raw_links cap at 100 is deliberate. DEV.to profile pages have hundreds of links — you don't need all of them for broken link detection.

The wait_until="domcontentloaded" setting is faster than networkidle and sufficient for meta tag extraction. JavaScript-rendered content needs the DOM to be ready, not all network requests to complete.

Module 3: Claude Extraction Layer

extractor.py takes the raw page snapshot from browser.py and calls Claude to produce a structured SEO audit result.

This is where most tutorials go wrong. They either write complex parsing logic in Python (fragile) or ask Claude for a free-form response and try to parse prose (unreliable). The right approach: give Claude a strict JSON schema and tell it to return nothing else.

The prompt engineering that makes this reliable:

import json
import os
import sys
from datetime import datetime, timezone
import anthropic

MODEL = "claude-sonnet-4-20250514"
client = anthropic.Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))


def _strip_fences(text: str) -> str:
    """Remove accidental markdown code fences from Claude's response."""
    text = text.strip()
    if text.startswith("```"):
        lines = text.splitlines()
        # Drop opening fence
        lines = lines[1:] if lines[0].startswith("```") else lines
        # Drop closing fence
        if lines and lines[-1].strip() == "```":
            lines = lines[:-1]
        text = "\n".join(lines).strip()
    return text


def extract(snapshot: dict) -> dict:
    if not os.environ.get("ANTHROPIC_API_KEY"):
        raise OSError("ANTHROPIC_API_KEY is not set.")

    prompt = f"""You are an SEO auditor. Analyze this page snapshot and return ONLY a JSON object.
No prose. No explanation. No markdown fences. Raw JSON only.

Page data:
- URL: {snapshot.get('final_url')}
- Status code: {snapshot.get('status_code')}
- Title: {snapshot.get('title')}
- Meta description: {snapshot.get('meta_description')}
- H1 tags: {snapshot.get('h1s')}
- Canonical: {snapshot.get('canonical')}

Return this exact schema:
{{
  "url": "string",
  "final_url": "string",
  "status_code": number,
  "title": {{"value": "string or null", "length": number, "status": "PASS or FAIL"}},
  "description": {{"value": "string or null", "length": number, "status": "PASS or FAIL"}},
  "h1": {{"count": number, "value": "string or null", "status": "PASS or FAIL"}},
  "canonical": {{"value": "string or null", "status": "PASS or FAIL"}},
  "flags": ["array of strings describing specific issues"],
  "human_review": false,
  "audited_at": "ISO timestamp"
}}

PASS/FAIL rules:
- title: FAIL if null or length > 60 characters
- description: FAIL if null or length > 160 characters  
- h1: FAIL if count is 0 (missing) or count > 1 (multiple)
- canonical: FAIL if null
- flags: list every failing field with a clear description
- audited_at: use current UTC time in ISO 8601 format"""

    response = client.messages.create(
        model=MODEL,
        max_tokens=1000,
        messages=[{"role": "user", "content": prompt}],
    )

    raw = response.content[0].text
    clean = _strip_fences(raw)

    try:
        return json.loads(clean)
    except json.JSONDecodeError as exc:
        print(f"[extractor] JSON parse error: {exc}", file=sys.stderr)
        return _error_result(snapshot, str(exc))


def _error_result(snapshot: dict, reason: str) -> dict:
    return {
        "url": snapshot.get("final_url", ""),
        "final_url": snapshot.get("final_url", ""),
        "status_code": snapshot.get("status_code"),
        "title": {"value": None, "length": 0, "status": "ERROR"},
        "description": {"value": None, "length": 0, "status": "ERROR"},
        "h1": {"count": 0, "value": None, "status": "ERROR"},
        "canonical": {"value": None, "status": "ERROR"},
        "flags": [f"Extraction error: {reason}"],
        "human_review": True,
        "audited_at": datetime.now(timezone.utc).isoformat(),
    }

Two things make this reliable in production:

First, _strip_fences() handles the case where Claude wraps its response in ```json fences despite being told not to. This happens occasionally with Sonnet and consistently breaks json.loads() if you don't handle it.

Second, the _error_result() fallback means the agent never crashes on a bad Claude response — it logs the error and marks the URL for human review, then continues to the next URL.

Cost: Claude Sonnet 4 is priced at \(3 per million input tokens and \)15 per million output tokens. A typical page snapshot is around 500 input tokens; the structured JSON response is around 300 output tokens. That works out to roughly \(0.006 per URL — about \)0.12 for a 20-URL audit.

Module 4: Broken Link Checker

linkchecker.py takes the raw_links list from the browser snapshot and checks same-domain links for broken status using async HEAD requests.

The design choices:

• Same-domain only. Checking every external link on a page would take minutes and isn't what agency clients need. Filter to links on the same domain as the page being audited.

• HEAD requests, not GET. Faster, lower bandwidth, sufficient for status code detection.

• Cap at 50 links. Pages like DEV.to article listings have hundreds of internal links. Checking all of them would dominate the runtime.

• Concurrent requests via asyncio. All links are checked in parallel, not sequentially.

import asyncio
import logging
from urllib.parse import urlparse
import httpx

CAP = 50
TIMEOUT = 5.0
logger = logging.getLogger(__name__)


def _same_domain(link: str, final_url: str) -> bool:
    if not link:
        return False
    lower = link.strip().lower()
    if lower.startswith(("#", "mailto:", "javascript:", "tel:", "data:")):
        return False
    try:
        page_host = urlparse(final_url).netloc.lower()
        parsed = urlparse(link)
        return parsed.scheme in ("http", "https") and parsed.netloc.lower() == page_host
    except Exception:
        return False


async def _check_link(client: httpx.AsyncClient, url: str) -> tuple[str, bool]:
    try:
        resp = await client.head(url, follow_redirects=True, timeout=TIMEOUT)
        return url, resp.status_code != 200
    except Exception:
        return url, True  # Timeout or connection error = broken


async def _run_checks(links: list[str]) -> list[str]:
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*[_check_link(client, url) for url in links])
    return [url for url, broken in results if broken]


def check_links(raw_links: list[str], final_url: str) -> dict:
    same_domain = [l for l in raw_links if _same_domain(l, final_url)]

    capped = len(same_domain) > CAP
    if capped:
        logger.warning("Page has %d same-domain links — capping at %d.", len(same_domain), CAP)
        same_domain = same_domain[:CAP]

    broken = asyncio.run(_run_checks(same_domain))

    return {
        "broken": broken,
        "count": len(broken),
        "status": "FAIL" if broken else "PASS",
        "capped": capped,
    }

Module 5: Human-in-the-Loop

This is the part most automation tutorials skip. What happens when the agent hits a login wall? A page that returns 403? A URL that redirects to a "Subscribe to continue reading" page?

Most scripts either crash or silently skip. Neither is acceptable in an agency context.

hitl.py handles this with two functions: one that detects whether a pause is needed, and one that handles the pause itself.

from state import add_to_needs_human

LOGIN_KEYWORDS = {"login", "sign in", "sign-in", "access denied", "log in", "unauthorized"}
REDIRECT_CODES = {301, 302, 307, 308}


def should_pause(snapshot: dict) -> bool:
    code = snapshot.get("status_code")

    # Navigation failed entirely
    if code is None:
        return True

    # Non-200, non-redirect
    if code != 200 and code not in REDIRECT_CODES:
        return True

    # Login wall detection
    title = (snapshot.get("title") or "").lower()
    h1s = [h.lower() for h in (snapshot.get("h1s") or [])]

    if any(kw in title for kw in LOGIN_KEYWORDS):
        return True
    if any(kw in h1 for kw in LOGIN_KEYWORDS for h1 in h1s):
        return True

    return False


def pause_reason(snapshot: dict) -> str:
    code = snapshot.get("status_code")
    if code is None:
        return "Navigation failed (None status)"
    if code != 200 and code not in REDIRECT_CODES:
        return f"Unexpected status code: {code}"
    return "Possible login wall detected"


def pause_and_prompt(url: str, reason: str) -> str:
    print(f"\n⚠️  HUMAN REVIEW NEEDED")
    print(f"   URL:    {url}")
    print(f"   Reason: {reason}")
    print(f"   Options: [s] skip  [r] retry  [q] quit\n")

    while True:
        choice = input("Your choice: ").strip().lower()
        if choice in ("s", "r", "q"):
            return {"s": "skip", "r": "retry", "q": "quit"}[choice]
        print("   Enter s, r, or q.")

The should_pause() function catches four cases: navigation failure, unexpected HTTP status, login keywords in the title, and login keywords in H1 tags. The login keyword check is what catches "Please sign in to continue" pages that return 200 but are effectively inaccessible.

In --auto mode (for scheduled runs), the main loop skips the pause_and_prompt() call and automatically handles these cases by logging the URL to needs_human[] in state and continuing.

Module 6: Report Writer

reporter.py writes results incrementally. This is important: results are written after each URL is audited, not batched at the end. If the run is interrupted, you don't lose completed work.

import json
import os
from datetime import datetime, timezone

REPORT_JSON = os.path.join(os.path.dirname(__file__), "report.json")
REPORT_TXT = os.path.join(os.path.dirname(__file__), "report-summary.txt")


def _load_report() -> list:
    if not os.path.exists(REPORT_JSON):
        return []
    with open(REPORT_JSON, encoding="utf-8") as f:
        return json.load(f)


def write_result(result: dict) -> None:
    """Append or update a result in report.json."""
    entries = _load_report()
    url = result.get("url", "")

    # Update existing entry if URL already present (handles retries)
    for i, entry in enumerate(entries):
        if entry.get("url") == url:
            entries[i] = result
            break
    else:
        entries.append(result)

    with open(REPORT_JSON, "w", encoding="utf-8") as f:
        json.dump(entries, f, indent=2, ensure_ascii=False)


def _is_overall_pass(result: dict) -> bool:
    fields = ["title", "description", "h1", "canonical"]
    for field in fields:
        if result.get(field, {}).get("status") not in ("PASS",):
            return False
    if result.get("broken_links", {}).get("status") == "FAIL":
        return False
    return True


def write_summary() -> None:
    entries = _load_report()
    passed = sum(1 for e in entries if _is_overall_pass(e))

    lines = []
    for entry in entries:
        overall = "PASS" if _is_overall_pass(entry) else "FAIL"
        failed_fields = [
            f for f in ["title", "description", "h1", "canonical", "broken_links"]
            if entry.get(f, {}).get("status") == "FAIL"
        ]
        suffix = f" [{', '.join(failed_fields)}]" if failed_fields else ""
        lines.append(f"{entry.get('url', 'unknown'):<60} | {overall}{suffix}")

    lines.append("")
    lines.append(f"{passed}/{len(entries)} URLs passed")

    with open(REPORT_TXT, "w", encoding="utf-8") as f:
        f.write("\n".join(lines))

The deduplication in write_result() handles retries cleanly. If a URL is retried after a human reviews a login wall and authenticates, the new result replaces the old one rather than creating a duplicate entry.

Module 7: The Main Loop

index.py wires everything together. It reads the URL list, loads state, skips already-audited URLs, and runs the audit loop.

import csv
import os
import sys
import time
import argparse

from state import load_state, is_audited, mark_audited, add_to_needs_human
from browser import fetch_page
from extractor import extract
from linkchecker import check_links
from hitl import should_pause, pause_reason, pause_and_prompt
from reporter import write_result, write_summary

INPUT_CSV = os.path.join(os.path.dirname(__file__), "input.csv")


def read_urls(path: str) -> list[str]:
    with open(path, newline="", encoding="utf-8") as f:
        return [row["url"].strip() for row in csv.DictReader(f) if row.get("url", "").strip()]


def run(auto: bool = False):
    if not os.environ.get("ANTHROPIC_API_KEY"):
        print("Error: ANTHROPIC_API_KEY environment variable is not set.")
        sys.exit(1)

    urls = read_urls(INPUT_CSV)
    pending = [u for u in urls if not is_audited(u)]

    print(f"Starting audit: {len(pending)} pending, {len(urls) - len(pending)} already done.\n")

    total = len(urls)

    try:
        for i, url in enumerate(pending, start=1):
            position = urls.index(url) + 1
            print(f"[{position}/{total}] {url}", end=" -> ", flush=True)

            # Browser navigation
            snapshot = fetch_page(url)

            # Human-in-the-loop check
            if should_pause(snapshot):
                reason = pause_reason(snapshot)

                if auto:
                    print(f"AUTO-SKIPPED ({reason})")
                    add_to_needs_human(url)
                    mark_audited(url)
                    continue

                action = pause_and_prompt(url, reason)
                if action == "quit":
                    print("Exiting.")
                    break
                elif action == "skip":
                    add_to_needs_human(url)
                    mark_audited(url)
                    continue
                # "retry" falls through to re-fetch below
                snapshot = fetch_page(url)

            # Claude extraction
            result = extract(snapshot)

            # Broken link check
            links = check_links(snapshot.get("raw_links", []), snapshot.get("final_url", url))
            result["broken_links"] = links

            # Write result immediately
            write_result(result)
            mark_audited(url)

            overall = "PASS" if all(
                result.get(f, {}).get("status") == "PASS"
                for f in ["title", "description", "h1", "canonical"]
            ) and links["status"] == "PASS" else "FAIL"

            print(overall)

    except KeyboardInterrupt:
        print("\n\nInterrupted. Progress saved. Re-run to continue.")
        return

    write_summary()
    passed = sum(
        1 for e in [r for r in []]
        if all(e.get(f, {}).get("status") == "PASS" for f in ["title", "description", "h1", "canonical"])
    )
    print(f"\nAudit complete. Report saved to report.json and report-summary.txt")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--auto", action="store_true", help="Auto-skip URLs requiring human review")
    args = parser.parse_args()
    run(auto=args.auto)

The KeyboardInterrupt handler is the resume mechanism. When you press Ctrl+C, the handler prints a message and exits cleanly. Because mark_audited() is called after write_result() for each URL, the next run skips everything already processed.

Running the Agent

Interactive mode (pauses on edge cases):

python index.py

Auto mode (skips edge cases, adds to needs_human[]):

python index.py --auto

When it runs, you'll see the browser window open for each URL and the terminal print progress:

Starting audit: 7 pending, 0 already done.

[1/7] https://example.com -> PASS
[2/7] https://example.com/about -> FAIL
[3/7] https://example.com/contact -> AUTO-SKIPPED (Unexpected status code: 404)
...
Audit complete. Report saved to report.json and report-summary.txt

To resume after an interruption:

python index.py --auto
# Starting audit: 4 pending, 3 already done.

Scheduling for Agency Use

For recurring weekly audits, create a batch file and schedule it with Windows Task Scheduler.

Create run-audit.bat:

@echo off
set ANTHROPIC_API_KEY=your-key-here
cd /d C:\Users\yourname\Desktop\seo-agent
python index.py --auto

In Windows Task Scheduler:

1. Create a new Basic Task

2. Set the trigger to Weekly, Monday at 7:00 AM

3. Set the action to "Start a program"

4. Browse to your run-audit.bat file

Check report-summary.txt on Monday morning. URLs in needs_human[] in state.json need manual review — login walls, paywalls, or pages that returned unexpected status codes.

For macOS/Linux, use cron:

# Run every Monday at 7am
0 7 * * 1 cd /path/to/seo-agent && ANTHROPIC_API_KEY=your-key python index.py --auto

What the Results Look Like

I ran this agent against seven of my own published pages across Hashnode, freeCodeCamp, and DEV.to. Every single one failed.

https://hashnode.com/@dannwaneri                    | FAIL [h1]
https://freecodecamp.org/news/claude-code-skill     | FAIL [description]
https://freecodecamp.org/news/stop-letting-ai-guess | FAIL [description]
https://freecodecamp.org/news/rag-system-handbook   | FAIL [title, description]
https://freecodecamp.org/news/author/dannwaneri     | FAIL [description]
https://dev.to/dannwaneri/gatekeeping-panic         | FAIL [title]
https://dev.to/dannwaneri/production-rag-system     | FAIL [title]

0/7 URLs passed

The freeCodeCamp description issues are partly platform-level — freeCodeCamp's template sometimes truncates or omits meta descriptions for article listing pages. The DEV.to title issues are mine. Article titles that work as headlines often exceed 60 characters in the <title> tag.

A note on the 60-character title rule: this is a display threshold, not a ranking penalty. Google indexes titles of any length. The 60-character guideline reflects approximately how many characters fit in a desktop SERP result before truncation. Titles over 60 characters often still rank — they just get cut off in search results, which can hurt click-through rate. The agent flags display risk, not a ranking violation.

Next Steps

The agent as built handles the core SEO audit workflow. Obvious extensions:

• Performance metrics — add a Lighthouse or PageSpeed Insights API call per URL

• Structured data validation — check for JSON-LD schema markup and validate it

• Email delivery — send report-summary.txt via SMTP after the run completes

• Multi-client support — separate input.csv files per client, separate report directories

The full code including all seven modules is at dannwaneri/seo-agent. Clone it, add your URLs, and run it.

If you found this useful, I write about practical AI agent setups for developers and agencies at DEV.to/@dannwaneri. The DEV.to companion piece covers the design decisions behind the agent — why HITL matters, why Browser Use over scrapers, and what the audit results mean for your own published content.

Get-ChildItem (also known as gci, ls, dir ) is a very powerful command. And one of its most iconic uses is to find/search for a file. It's more precise and more reliable than Windows Explorer. It even has better filtering options that show the results that are more relevant to you.

In this tutorial, you'll learn how to use gci and how to combine it with other commands so that it becomes an even more powerful tool. Remember to enable copy-pasting in Windows PowerShell, so it's easier for you to follow along. You can see how to enable it here.

What we'll cover:

1. Basic explanation of the Get-ChildItem command

   • Most used examples of searching by gci command

2. Setup for other more complex examples

3. When is the -Path option not needed?

4. Advanced Searching – Combining Get-ChildItem with the Where-Object Command

   • 4.1. How to search through only a particular directory

   • 4.2. How to search while excluding a particular directory

   • 4.3 Searching only 1 directory from many with exactly the same name

   • 4.4 Filter how deep (how many folders in) you want to search for the file

5. How to Search Through Hidden Files

6. How can you know all the properties that you can use as a filter?

   • How to retrieve only 1 desired property

7. I don't know the file’s name, but I know what's inside it. How do I find the file by its content?

8. I can't see the full path - how do I fix this?

9. Hard to read? Open the results in the text editor of your choice

10. Summary - the ultimate commands for searching and finding whatever you need

1. Basic Explanation of the Get-ChildItem Command

Let's take a look at the example searching script to understand how it works:

Get-ChildItem -Recurse -Path "C:\path to\your directory\" -Filter "*whatImLookingFor*"

Get-ChildItem (aliases: dir, ls, gci) lists the content of a folder or directory just like the Linux ls command does.

This command works by searching every single file and directory in the path specified. It shows you everything it found that matches the filter. It doesn't mean that this command doesn't look everywhere else – because it does.

So you specify the path that is the parent (folder), which means that every folder and file under it is its child. If you know some CSS and JavaScript, treat it the same way that these languages do.

If you don't use -Recurse or -Depth, then the command works only in your current directory (parent Depth level 0) and searches for its children inside that directory (children Depth level 0).

If you use -Recurse, then the gci will search for what you want on ALL LEVELS. But by using-Depth, you can specify how deep you want it to look for a file/folder.

To recurse means "to repeat an operation". So, -Recurse means that gci will repeat the search for your file or folder in every child element of the "Documents" directory, and every directory inside it, all levels deep.

All of these files and folders are children of your "Documents" folder. If you delete the folder, you delete everything inside it too.

-Filter filters the output of the command to only show what matches the filter (examples of how to use filter are further in the article).

-Path tells where the command should be looking for files (by using "C:\", for example, you're telling it to look at the very basis of your computer). If you want to search in certain directory it would look like this:

Get-ChildItem -Path "C:\path to\your directory\"

OR

Get-ChildItem -Path "~\Documents\path to\your directory\"

~\ here is a shorthand for "inside current user's folder" or "C:\Users\YourUsername".

Next, we can specify whether we'd like to look for a file or a folder, so we have fewer results to look at:

Get-ChildItem -Path C:\ -Recurse -Filter "*whatImLookingFor*" -File

Get-ChildItem -Path C:\ -Recurse -Filter "*whatImLookingFor*" -Directory

You might be wondering how you can stop the search if it takes too long. When you're using -Recurse, the output that you'll get might become quite overwhelming, especially if you didn't specify your command enough (more about that in step 3 and step 4). Luckily, you can stop any command in PowerShell after starting it with Ctrl + C OR Ctrl + Z OR Ctrl + X. All of them should work.

Most Used Examples of Searching by gci Command

Here are some handy examples of searching scripts that you can use:

Example #1: search for all executive files on your PC (remember that you can stop this command with one of shortcuts, like Ctrl + C):

Get-ChildItem -Path C:\ -Recurse -Filter "*.exe" -File

REMEMBER:
In order to paste commands into the PowerShell, you have to first enable it. Here's how.

This command will show you a very long list of executable files and their folders (as shown in the image below).

These lists might be so long that it's impossible to find anything in them. That's why you'll learn how to use more advanced techniques of filtering in step 4 to see fewer unnecessary results that don't fit your criteria.

Example #2: search for an executable file that has "notepad" in its name (or search for any program you need, basically):

Get-ChildItem -Path C:\ -Recurse -Filter "notepad*.exe" -File

One of the results will show you the location of the file you want:

In our case it's the C:\Windows\System32 folder.

You can mix it however you want! Thanks to that command, you don't have to remember much about your file and it will still work.

Get-ChildItem -Path C:\ -Recurse -Filter "n*pad*.*xe"

So what if you see some errors while scanning the whole system. Should you worry?

It's ok! Sometimes you might get lots of errors. They will most likely occur when a script scours the system folders/files. If you want to get rid of them, add -ErrorAction SilentlyContinue, like you see here:

Get-ChildItem -Path C:\ -Recurse -Filter "notepad*.exe" -File -ErrorAction SilentlyContinue

You can try it now ;)

2. Setup for Other More Complex Examples

Now, let's look at even more use cases for this command. But first, we'll create a space where I can show you examples.

First, create new folder inside your "Documents" folder. Let's call it "Items".

Inside it, create two text documents. Name one of them "Item 1- Green Bracelet" and the other "Item 2- Blue Bracelet" (Yes, make sure you write the first letter of each word in UPPER CASE).

Copy these files now.

Go one folder back (you can use the Ctrl + UpArrow shortcut ) and create another folder next to "Items" called "More items":

Paste the copied files inside the "More items" folder and change their names, so they have only lower case letters ("item 1- green bracelet" and "item 2- blue bracelet" ).

PRO TIP:
You can click once on a file with your mouse and then type the F2 key on your keyboard in order to change their names.

3. When is the -Path option not needed?

You don't have to specify the path every time. You can always just move to the desired directory with the cd (change directory) command.

This command will move you to your Documents folder:

cd ~\Documents\

Now, you should be able to see PowerShell pointing to your Documents folder on the left of the screen:

If you don't see this, then you can use double quotes " ", like in this command:

cd "~\Documents\"

Make sure that PowerShell is pointing to our desired folder. Now, the searching command looks like this without the -Path option:

Get-ChildItem -Recurse -Filter "*item*" -File

Pretty simple, right?

As you can see in the image above, we first moved to our desired directory, so later we could perform the search inside it without specifying the -Path option/parameter.

But the -Path option is very useful, either when you're creating a script or you want to search for something without moving away from the current directory:

Get-ChildItem -Path ~\Documents\ -Recurse -Filter "*item*" -File

Get-ChildItem -Path ~\Documents\ -Recurse -Filter "*item*" -Directory

Here's an example. I'm inside the System32 folder and I want to know whether the thing I'm looking for is inside the Documents folder without moving in there:

And it really is there!

From now on, because you already know what the -Path option is being used for, I won't be using it unless it's necessary.

4. Advanced Searching – Combining Get-ChildItem with the Where-Object Command

Sometimes you might have several folders named exactly the same, but they're in different places. You might want to exclude them based on their content, which folder they are in, or based on their-Depth level (see the graphic with the explanation about -Depth level in step 1). That's what we're going to cover in the next few points.

For this part of the tutorial, make sure you've gone through step 2 (but you can skip step 3 if you want).

4.1. Searching through only a particular directory

Let's say that we're now looking for the bracelets that we created in step 2. But, we want to see the results from only one folder. For that, we'll use case-sensitive search (-clike) to get only our preferred results. But -clike doesn't work with gci alone. We need to apply another filter with the Where-Object { } command:

Get-ChildItem -Path ~\Documents\ -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*Item*" }

OR (clearer version, without the -Path option):

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*Item*" }

Let's review what's going on here:

• Get-ChildItem -Recurse -Filter "*item*" searches for all files and folders with "item" in their name

• | – the "pipe" symbol is used to get the output of the previous command (the list of all files and folders filtered by gci) and send it to the next command (Where-Object is applying another filter to what is already filtered by gci).

• Where-Object { } is the command used for filtering the lists of objects. The filter is being specified inside the { } curly brackets.

• \(_ refers to all the separate objects. Treat it as "ForEachObjectFromList". And treat the whole sequence after the | as "FindObjectsFromList that have a name with 'Item' ".
  \)_ is very often used with Where-Object, but also with some other commands.

• .Name – we choose a Name property to get from every object.

• -clike finds a match that is 100% correct. All letters must be the exact same case as the phrase we specified. c stands for "case sensitive" and it checks every letter to see if it's upper case or lower case.

So, Where-Object { $_.Name -clike "*Item*" } is a filter that takes the Name parameter of every object from the list (created by gci) and checks with -clike if any Name has the word "Item" in it.

As you can see in the image below, now we'll get only the files with upper case names in our result:

IMPORTANT:
-like alone means that we're looking for a certain pattern, no matter what case the letters are. The c in -clike means that we look for the thing with exactly the same capitalization of the letters (both upper and lower case, hence the "c").

If you want to see the files without the upper case first letter, you can do that by changing "*Item*" from our current command to "*item*":

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*item*" }

Let's try it out!

4.2. How to search while excluding a particular directory

In step 4.1 we learned how to search only for files/folders with specific case-sensitive names in them. After applying only two changes to our previous code, we can exclude certain directories from our search.

Here's our starting command once again:

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -clike "*Item*" }

Change #1

In the example above, -clike shows only files/folders including specific phrase in their names. If we change it to -cnotlike, we'll exclude from the search all files/folders with that specific phrase in their name.

Now our code looks like this:

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.Name -cnotlike "*Item*" }

Change #2

After the first change, Where-Object { \(_.Name -cnotlike "*Item*" } only excludes the names, not full paths. In order to avoid that, we need to exclude an actual path to these files. We can do that by changing \)_.Name to $_.FullName, which checks for a certain phrase in the whole path to the file and in the file's name.

Now, your command should look like this:

Get-ChildItem -Recurse -Filter "*item*" |   
Where-Object { $_.FullName -cnotlike "*Item*" }

We excluded the "Items" folder from our search. You should now be able to see the files only from the "More items" directory. Try it out yourself!

What if you want to exclude the "More items" directory instead? Just change the phrase inside the filter to something like this:

Get-ChildItem -Recurse -Filter "*green*" -File |   
Where-Object { $_.FullName -cnotlike "*More*" }

We also changed the name of the file from "*item*" to "*green*" in our gci search (first line of code). That's why now we'll see only one bracelet in our result list:

The gci command has two filters applied. First, it searches for files with phrase "green" in their names. The second filter is the "Where-Object" command, which excludes anything that has the word "More" in its path. In our case, the "More items" folder got excluded.

We don't even need the case-sensitive filter in our case. The command will work the same when we exclude just a lowercase word "more". So let's change -cnotlike "*More*" to -notlike "*more*" and see if it's true:

Get-ChildItem -Recurse -Filter "*green*" -File |   
Where-Object { $_.FullName -notlike "*more*" }

As you can see, the result is the same! Despite different cases of the letters, we still got the right keyword. So, case-sensitive search isn't always needed – only when you want to be very specific.

Sometimes, being too specific might be bad and make your code not work as intended. To see what I mean, let's look at the example below. Let's apply case-sensitive search once again, but to our unchanged, lowercase keyword "more" and see if it still works:

Get-ChildItem -Recurse -Filter "*green*" -File |   
Where-Object { $_.FullName -cnotlike "*more*" }

Case-sensitive search doesn't filter out anything now, because it's too specific. Both the "Items" and "More items" folders omit the filter now.

FAQ:

If the Where-Object command is what actually filters the output for us, shouldn't we drop (delete) the -Filter option from gci?

No, we should still use the -Filter option, because it already separates around 99% of the possible files, so the Where-Object command has to work roughly only on 1% of the objects. It makes this part of the command AT LEAST 100 times faster (more often 100,000 times or even faster).

You can try using this command in -Path C:/ with and without the -Filter option. In my case, using the -Filter shortened the time needed for the whole sequence of commands to finish from 16 seconds to 8 seconds (first 7.99 seconds is used by gci, so that's why the time got shortened only by a half). That's what we call ✨optimization✨ :D

4.3 Searching only 1 directory from many with exactly the same name

We've learned how to search for a phrase anywhere inside the path of a file. But what if we want to search inside exactly the "More items" folder? For that, we'll use the -match filter (which works similarly to the -like filter).

Our phrase will also use "\", instead of "\". This is because "\" is the symbol for a folder, but alone in programming it also has some other features, which we don't want.

This command will look for a match for the "More items" folder in the path of every file from the list. Then, it will show you this file if it matches.

What if we want to check for two folders, one next to the other, simultaneously? Very easy! Just connect them with the sign for a folder "\". Here, the command will search inside the "More items" folder only if it's inside the "Documents" folder:

As you can see, we didn't use "More items", only "More". You can shorten that filter how you want. It will still be applied to the whole path. See the example below:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.FullName -match "s\\Mo*" }

Earlier, we used the not statement in -like filter to exclude certain files and directories. The same can be done with -notmatch:

Get-ChildItem -Recurse -Filter "*green*" -File | 
Where-Object { $_.FullName -notmatch "ents\\Ite*" }

Be aware that we're now excluding the "Items" folder from the search, not "More items".

And, with -cmatch we can apply the same case-sensitive filter as with -clike:

Get-ChildItem -Recurse -Filter "*green*" -File | 
Where-Object { $_.FullName -cmatch "green*" }

I hope you get the gist of it now.

4.4 Filter how deep (how many folders in) you want to search for the file

Sometimes you might have a very long path to some of your files. If you don't want to waste time searching every folder on your computer recursively, you can use -Depth option. It specifies how many folders to search inside your folder tree. I already showed you the picture of a folder tree in the beginning of this article, but you should take a look at it here once again.

So, how does the -Depth parameter work?

-Depth 0 means that our command will search only the current folder. It will show results of all children of Depth level 0. Those results are:
1 "child file" and 2 "child folders".

-Depth 1 searches the current folder and its child-folders. It will show the results of all children of Depth level 1. Those results are:
1 "child file", 2 "child folders", 2 "grandchild files" and 1 "grandchild folder".

-Depth 2 searches the current folder and its child and grandchild folders. It will show results of all children of Depth level 2. Those results are:
1 "child file", 2 "child folders", 2 "grandchild files", 1 "grandchild folder" and 1 "great grandchild file".

Let's see the difference between these two commands:

Get-ChildItem -Recurse -Filter "*item*" -Depth 0

Get-ChildItem -Recurse -Filter "*item*" -Depth 1

The first command will show you only the files and folders inside our current directory.
The second command will also search for them inside every folder found inside the current folder.

For the sake of practice, let's combine it with Where-Object to find the green bracelet:

Get-ChildItem -Recurse -Filter "*item*" -Depth 1 | Where-Object { $_.name -clike"*green*" }

I hope that this example showed you how easy it is to use multiple options ( -Depth, -Recurse) and filters (-Filter, Where-Object).

5. How to Search Through Hidden Files

Some files are not that easily accessible to the user. You can see some of the hidden files and folders in Windows Explorer (here's how). But sometimes it's easier to find what you need if you see only those hidden files. That's possible with PowerShell.

The options we're going to use for that are:

• -Force: show files otherwise not accessible by the user, such as hidden files.

• -Hidden: show only those hidden files and directories.

This example will search for hidden files in our user's folder:

gci -Path ~\ -Force -Hidden

Everything here is usually invisible to the typical user. But not for you now :D

The interesting thing is that there are more files not available to the user than the available ones. If you're brave enough, you can see them yourself (Remember! Ctrl + C stops the command!):

gci -Path ~\ -Force -Hidden -Recurse

6. How can you know all the properties that you can use as a filter?

Up until now, we'vce used some common properties, like Name and Fullname. But there are many others that you might want to access, like CreationTime (date of creating the file) or LastWriteTime (date of last edit of the file).

In this section, I'll first show you how to see all the possible properties. After that, you'll learn how to retrieve only the property you want for scripting purposes.

Go through step 2 above if you haven't already, because we're going to use the same files that we created before.

Move to the Documents folder in PowerShell.

I hope that this script looks familiar to you now. It searches for files with "item" in their names and checks if these names contain the word "green" (all lowercase letters):

Get-ChildItem -Recurse -Filter "*item*" | 
Where-Object { $_.Name -clike "*green*" }

We know that only one file should appear (if you don't trust me, just see for yourself). So, we're going to see every possible property we can use by appending (adding at the end) this fragment of code:
| Select-Object -Property *

Select-Object (alias: select) is used for selecting different types of properties. By using an option -Property we tell it to show both values and names of all the properties.

For example:

Name of property: FullName
Value of property: ~\Documents\More items\item 1- green bracelet.txt

The asterisk * at the end tells this command to show these names and values for every property possible.

The final version of this command looks like this:

Get-ChildItem -Recurse -Filter "*item*" | 
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -Property *

Try finding the FullName property in there :D

This command showed us all possible properties that we can use for that 1 file that it found. If there were more files fitting the filter, then every single one of them would have a similar list of properties. But for different types of files you will get different results.

How to retrieve only 1 desired property

You've already learned how to check for all possible properties. So, how do we use any of them? Just put one of them instead an asterisk * at the end of the command, like we put CreationTime in here:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -Property CreationTime

You can use any other property for the sake of this exercise, like LastWriteTime:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -Property LastWriteTime

What if you want to retrieve only the value of the property without its name (because you already know its name and it also messes up your script)? You can retrieve just the value, by changing the -Property to -ExpandProperty:

Get-ChildItem -Recurse -Filter "*green*" -File |
Where-Object { $_.Name -clike "*green*" } | 
Select-Object -ExpandProperty LastWriteTime

See the result:

7. I don't know the file’s name, but I know what's inside it. How do I find the file by its content?

Sometimes it's easier to find a file by searching it by its content. Or perhaps you have lots of similar files and you'd like to check them quickly without opening and closing them. I'll show you some techniques that will let you achieve that in no time.

This command will search every file on your system for the specified word or phrase (in our case, the phrase is "match"):

Get-ChildItem -Path C:\ -Recurse -File | 
Select-String -Pattern 'match' -List

Here's what's happening:

• Get-ChildItem -Path C:\ -Recurse -File: as you already know, this part searches for every file on your computer.

• | – passes the list of files to the next command. So, the next command will search for a certain phrase only in the files listed by gci.

• Select-String – "String" is a common word in programming used to describe a word/phrase/some text. So, we select the phrase that we want to search for. That phrase is specified by the -Pattern parameter (in our case it's "match").

• -List tells the command to show only the first found match in every file (great if you want to just see the list of all found files).

Here's an example output of our command:

Of course, you have quite a lot of files, and some images may also appear in your search (like .svg files that are basically text files that tell the system how to draw an icon). So, it's always best to specify what type of file you're searching for. Let's look for the phrase "red" inside .svg files:

Get-ChildItem -Filter "*.svg" -Recurse | 
Select-String -Pattern 'red' -List

On the other hand, some text documents will never appear in your search (for example .doc and .docx documents are encoded in such a way that they're impossible to decode without Word).

But in regular text files, you can search for phrases with an emphasis on big and small letters with the -CaseSensitive option. Here, we're going to search for the phrase "github" with only lowercase letters:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List -CaseSensitive

Other options that you'll often use with the Select-String command are:

• Select-String -AllMatch will show you all matches found in every searched file (instead of only 1 match found per file, like with -List).
  Select-String -Context 3 shows the three lines of text before and after the line in which the match is found.
  Select-String -Raw won't show you the paths, just the content of the files. This is great for automation and scripts. It's often combined with the -Context option.

Let's see some of these options in action:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -AllMatch -Context 3

Thanks to the -Context parameter, you can see a total of seven lines (three lines before and three lines after the match) in this file, one after another. This makes it easier to differentiate it from all the other matches found by -AllMatch that might be put in a very similar context.

If you ever feel like there's too much clutter on your screen, you can combine Select-String with Select-Object to get only the paths of the files with matched phrases.

The command below will search every .txt file on your computer for the phrase specified:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List

Let's add the Select-Object -Property Path filter at the end. Now, the command will only show the paths, so there's less clutter on your screen:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List | 
Select-Object -Property Path

Some of the paths are not fully visible. We'll fix that in the next step.

8. I can't see the full path - how do I fix this?

Let's format the results with the Format-Table -Wrap -AutoSize command. -Autosize allows the result to take the whole available space. -Wrap allows wrapping (continuing the text in the next line when it doesn't fit in the space available), which creates more space if it's needed.

Here's an example:

Get-ChildItem -Path C:\ -Filter "*.txt" -Recurse | 
Select-String -Pattern 'github' -List | 
Select -Property Path | 
Format-Table -Wrap -AutoSize

Now, you can see the whole paths (or any other results you need) even in PowerShell!

9. Hard to read? Open the results in the text editor of your choice

You can send the results of any script/command in two ways:

> ~\Documents\command_output.txt
AND
| Out-File ~\Documents\command_output.txt

Both of these will create a file inside your Documents folder, which you can later open in any program of your choice and edit.

Just add whichever solution you prefer to the end of your command, like here:

Get-ChildItem -Filter "*.txt" -Recurse | 
Select-String -Pattern 'match' -List | 
Select -Property Path | 
Out-File ~\Documents\command_output.txt

In the image below, first you'll see the same command, but without exporting the results to another file. The second command, at the bottom of the image, will export the results to the other file without showing them in PowerShell:

You'll see the results from second command after opening the file in any text editor:

But, what if you can't see the full path even in your text editor?

To address this, you can add | Format-Table -Wrap -AutoSize right before sending the results to the file:

Get-ChildItem -Path C:\ -Filter "*.txt" -Recurse | 
Select-String -Pattern 'match' -List | 
Select -Property Path | 
Format-Table -Wrap -AutoSize |
Out-File ~\Documents\command_output.txt

And open the file to see the whole path!

Just remember that you have to copy each line one by one. Where you see the arrows in the screenshot above is a "newline" character, which you have to delete. Only after doing that can you copy the whole path and paste it into Windows Explorer or into some script.

10. Summary: the Ultimate Commands for Searching and Finding Whatever You Need

Here you can download a free cheat sheet with explanations of the commands and examples in one place.

Most used commands:

• Case-sensitive search:

Get-ChildItem -Path C:\ -Recurse -Filter "*whatYouNeed*" |   
Where-Object { $_.Name -clike "*whatYouNeed*" } |   
Select-Object { $_.FullName } |
Format-Table -Wrap -AutoSize

• Alternatively, send the result to a file:

Get-ChildItem -Path C:\ -Recurse -Filter "*whatYouNeed*" |   
Where-Object { $_.Name -clike "*whatYouNeed*" } |   
Select-Object { $_.FullName } |
Format-Table -Wrap -AutoSize |
Out-File ~\Documents\command_output.txt

• Search by file's content:

Get-ChildItem -Path C:\ -Recurse | 
Select-String -Pattern 'what you remember' -AllMatch -Context 2 |
Format-Table -Wrap -AutoSize

• Alternatively, send the result to the file:

Get-ChildItem -Path C:\ -Recurse | 
Select-String -Pattern 'what you remember' -CaseSensitive -AllMatch -Context 2 |
Format-Table -Wrap -AutoSize |
Out-File ~\Documents\command_output.txt

These commands should work for anything you want to find. I hope you understand now how they function after reading through this tutorial ;)

Wrapping Up

If you want to learn more about these commands, I show you how to work with them in depth in my tutorial “Learn PowerShell commands like a Linux user”.

If what you found here helped you in any way, consider following me on my social media in order to help me reach further audience: Mastodon, LinkedIn.

You can also rate me on Github and support me on Ko-fi!

Thank you for any support you're able to give. Have a great day!

One of the major improvements has been a properly automated CI/CD pipeline flow that gives us seamless automation, continuous integration, and continuous deployment.

In this article, I'll break down how you can automate and build a production ready CI/CD pipeline for your Flutter application using GitHub Actions.

Note that there are other ways to do this, like with Codemagic (built specifically for Flutter apps – which I'll cover in a subsequent tutorial), but in this article we'll focus on GitHub Actions instead.

Table of Contents

1. The Typical Workflow

2. Prerequisites

3. Pipeline Architecture

4. Writing the Workflows

   • The Helper Scripts

     • generate_config.sh

     • quality_gate.sh

     • upload_symbols.sh

   • PR Quality Gate (pr_checks.yml)

   • Android CI/CD Pipeline (android.yml)

   • iOS CI/CD Pipeline (ios.yml)

5. Secrets and Configuration Reference

6. End-to-End Flow

7. Conclusion

The Typical Workflow

First, let's define the common approach to deploying production-ready Flutter apps.

The development team does their work on local, pushes to the repository for merge or review, and eventually runs flutter build apk or flutter build appbundle to generate the apk file. This then gets shared with the QA team manually, or deployed to Firebase app distribution for testing. If it's a production move, the app bundle is submitted to the Google Play store for review and then deployed.

This process is often fully manual with no automated checks, validation, or control over quality, speed, and seamlessness. Manually shipping a Flutter app starts out relatively simply, but can quickly and quietly turn into a liability. You run flutter build, switch configs, sign the build, upload it somewhere, and hope you didn’t mix up staging keys with production ones.

As teams grow and release updates more and more quickly, these manual steps become real risks. A skipped quality check, a missing keystore, or an incorrect base URL deployed to production can cost hours of debugging or worse – it can affect your users.

Automating this process fully involves some high level configuration and predefined scripting. It completely takes control of the deployment process from the moment the developer raised a PR into the common or base branch (for example, the develop branch).

This automated process takes care of everything that needs to be done – provided it has been predefined, properly scripted, and aligns with the use case of the team.

What we'll do here:

In this tutorial, we'll build a production-grade CI/CD pipeline for a Flutter app using GitHub Actions. The pipeline automates the entire lifecycle: pull-request quality checks, environment-specific configuration injection, Android and iOS builds, Firebase App Distribution for testers, Sentry symbol uploads, and final deployment to the Play Store and App Store.

By the end, every release – from a developer opening a PR to the final build landing in users' hands – will be fully automated, with no one touching a terminal.

Prerequisites

Before starting, you should have:

1. A Flutter app with working Android and iOS builds

2. Basic familiarity with GitHub Actions (workflows and jobs)

3. A Firebase project with App Distribution enabled

4. A Sentry project for error tracking

5. A Google Play Console app already created

6. An Apple Developer account with App Store Connect access

7. Fastlane configured for your iOS project

8. Basic Bash knowledge (I’ll explain the important parts)

Pipeline Architecture

In this guide, we'll be building a CI/CD pipeline with very precise instructions and use cases. These use cases determine the way your pipeline is built.

For this tutorial, we'll use this use case:

I want to automate the workflow on my development team based on the following criteria:

1. When a developer on the team raises a PR into the common working branch develop in most cases), a workflow is triggered to run quality checks on the code. It only allows the merge to happen if all checks (like tests coverage, quality checks, and static analysis) pass.

2. Code that's moving from the develop branch to the staging branch goes through another workflow that injects staging configurations/secret keys, does all the necessary checks, and distributes the application for testing on Firebase App Distribution for android as well as Testflight for iOS.

3. Code that's moving from the staging to the production branch goes through the production level workflow which involves apk secured signing, production configuration injection, running tests to ensure nothing breaks, Sentry analysis for monitoring, and submission to App Store Connect as well as Google Play Console.

These are our predefined conditions which help with the construction of our workflows.

Writing the Workflows

We'll split this pipeline into three GitHub Actions workflows.

We'll also be taking it a notch higher by creating three helper .sh scripts for a cleaner and more maintainable workflow.

In your project root, create two folders:

1. .github/

2. scripts.

The .github/ folder will hold the workflows we'll be creating for each use case, while the scripts/ folder will hold the helper scripts that we can easily call in our CLI or in the workflows directly.

After this, we'll create three workflow .yaml files:

1. pr_checks.yaml

2. android.yaml

3. ios.yaml

Also in the scripts folder, let's create three .sh files:

1. generate_config.sh

2. quality_checks.sh

3. upload_symbols.sh

.github/
  workflows/
    pr_checks.yml
    android.yml
    ios.yml

scripts/
  generate_config.sh
  quality_checks.sh
  upload_symbols.sh

This workflow architecture ensures that a push to develop automatically produces a tester build. Also, merging to production ships directly to the stores without manual commands or config changes.

The scripts live outside the YAML on purpose. This lets you run the same logic locally.

The Helper Scripts

The scripts form the backbone of the pipeline. Each one has a single responsibility and is reused across workflows.

Instead of cramming logic into YAML, we'll move it into reusable scripts. This keeps workflows clean and lets you run the same logic locally. Let's go through each one now.

Script #1: generate_config.sh

Injecting secrets safely is one of the hardest CI/CD problems in mobile apps.

The strategy:

• Commit a Dart template file with placeholders

• Replace placeholders at build time using secrets from GitHub Actions

• Never commit real credentials

#!/usr/bin/env bash
set -euo pipefail


ENV_NAME=${1:-}
BASE_URL=${2:-}
ENCRYPTION_KEY=${3:-}

TEMPLATE="lib/core/env/env_ci.dart"
OUT="lib/core/env/env_ci.g.dart"

if [ -z "\(ENV_NAME" ] || [ -z "\)BASE_URL" ] || [ -z "$ENCRYPTION_KEY" ]; then
  echo "Usage: $0 <env-name> <base-url> <encryption-key>"
  exit 2
fi

sed -e "s|<<BASE_URL>>|$BASE_URL|g" \
    -e "s|<<ENCRYPTION_KEY>>|$ENCRYPTION_KEY|g" \
    -e "s|<<ENV_NAME>>|$ENV_NAME|g" \
    "\(TEMPLATE" > "\)OUT"

echo "Generated config for $ENV_NAME"

This script is responsible for injecting environment-specific configuration into the Flutter app at build time, without ever committing secrets to source control.

Let’s walk through it carefully.

1. Shebang: Choosing the Shell

#!/usr/bin/env bash

This line tells the system to execute the script using Bash, regardless of where Bash is installed on the machine.

Using /usr/bin/env bash instead of /bin/bash makes the script more portable across local machines, GitHub Actions runners, and Docker containers.

2. Fail Fast, Fail Loud

set -euo pipefail

This is one of the most important lines in the script.

It enables three strict Bash modes:

• -e: Exit immediately if any command fails

• -u: Exit if an undefined variable is used

• -o pipefail: Fail if any command in a pipeline fails, not just the last one

This matters in CI because silent failures are dangerous, partial config generation can break production builds, and CI should stop immediately when something is wrong.

This line ensures that no broken config ever makes it into a build.

3. Reading Input Arguments

ENV_NAME=${1:-}
BASE_URL=${2:-}
ENCRYPTION_KEY=${3:-}

These lines read positional arguments passed to the script:

• $1: Environment name (dev, staging, production)

• $2: API base URL

• $3: Encryption or API key

The ${1:-} syntax means:

“If the argument is missing, default to an empty string instead of crashing.”

This works hand-in-hand with set -u , we control the failure explicitly instead of letting Bash explode unexpectedly.

4. Defining Input and Output Files

TEMPLATE="lib/core/env/env_ci.dart"
OUT="lib/core/env/env_ci.g.dart"

Here we define two files:

• Template file (env_ci.dart)

  • Contains placeholder values like <<BASE_URL>>

  • Safe to commit to Git

• Generated file (env_ci.g.dart)

  • Contains real environment values

  • Must be ignored by Git (.gitignore)

At the heart of this approach are two Dart files with very different responsibilities. They may look similar, but they play completely different roles in the system.

env.ci.dart:

// lib/core/env/env_ci.dart

class EnvConfig {
  static const String baseUrl = '<<BASE_URL>>';
  static const String encryptionKey = '<<ENCRYPTION_KEY>>';
  static const String environment = '<<ENV_NAME>>';
}

This file is safe, static, and version-controlled. It contains placeholders, not real values.

Some of its key characteristics are:

• Contains no real secrets

• Uses obvious placeholders (<<BASE_URL>>, etc.)

• Safe to commit to Git

• Reviewed like normal source code

• Serves as the single source of truth for required config fields

Think of this file as a contract:

“These are the configuration values the app expects at runtime.”

env.ci.g.dart:

This file is created at build time by generate_config.sh. After substitution, it looks like this:

// lib/core/env/env_ci.g.dart
// GENERATED FILE — DO NOT COMMIT

class EnvConfig {
  static const String baseUrl = 'https://staging.api.example.com';
  static const String encryptionKey = 'sk_live_xxxxx';
  static const String environment = 'staging';
}

Key characteristics:

• Contains real environment values

• Generated dynamically in CI

• Differs per environment (dev / staging / production)

• Must never be committed to source control

This file exists only on a developer’s machine (if generated locally), inside the CI runner during a build. Once the job finishes, it disappears.

.gitignore:

To guarantee the generated file never leaks, it must be ignored:

Why This Separation Is Critical

This design solves several hard problems at once.

Security:

• Secrets live only in GitHub Actions secrets

• They never appear in the repository

• They never appear in PRs

• They never appear in Git history

Environment Isolation:

Each environment gets its own generated config:

• develop: dev API

• staging: staging API

• production: production API

The same codebase behaves differently without branching logic in Dart.

Deterministic Builds:

Every build is fully reproducible, fully automated, and explicit about which environment it targets.

There are no “it worked locally” scenarios.

5. Validating Required Arguments

if [ -z "\(ENV_NAME" ] || [ -z "\)BASE_URL" ] || [ -z "$ENCRYPTION_KEY" ]; then
  echo "Usage: $0 <env-name> <base-url> <encryption-key>"
  exit 2
fi

This block enforces correct usage.

• -z checks whether a variable is empty

• If any required argument is missing:

  • A helpful usage message is printed

  • The script exits with a non-zero status code

• 0: success

• 1+: failure

• 2 conventionally means incorrect usage

In CI, this immediately fails the job and prevents an invalid build.

6. Injecting Environment Values

sed -e "s|<<BASE_URL>>|$BASE_URL|g" \
    -e "s|<<ENCRYPTION_KEY>>|$ENCRYPTION_KEY|g" \
    -e "s|<<ENV_NAME>>|$ENV_NAME|g" \
    "\(TEMPLATE" > "\)OUT"

This is the heart of the script.

What’s happening here:

1. sed performs stream editing: it reads text, transforms it, and outputs the result

2. Each -e flag defines a replacement rule:

   • Replace <<BASE_URL>> with the actual API URL

   • Replace <<ENCRYPTION_KEY>> with the real key

   • Replace <<ENV_NAME>> with the environment label

3. The transformed output is written to env_ci.g.dart

This entire operation happens at build time:

• No secrets are committed

• No secrets are logged

• No secrets persist beyond the CI run

7. Success Feedback

echo "Generated config for $ENV_NAME"

This line provides a clear success signal in CI logs.

It answers three important questions instantly:

• Did the script run?

• Did it finish successfully?

• Which environment was generated?

In long CI logs, these small confirmations matter.

Alright, now let's move on to the second script.

Script #2: quality_gate.sh

This script defines what “good code” means for your team.

#!/usr/bin/env bash
set -euo pipefail

echo "Running quality checks"

dart format --output=none --set-exit-if-changed .
flutter analyze
flutter test --no-pub --coverage

if command -v dart_code_metrics >/dev/null 2>&1; then
  dart_code_metrics analyze lib --reporter=console || true
fi

echo "Quality checks passed"

Lets break down this script bit by bit.

1. Start & End Log Markers

echo "Running quality checks"
...
echo "Quality checks passed"

These two lines act as visual boundaries in CI logs.

In large pipelines (especially when Android and iOS jobs run in parallel), logs can be very noisy. Clear markers:

• Help developers quickly find the quality phase

• Make debugging faster

• Confirm that the script completed successfully

The final success message only prints if everything above it passed, because set -e would have terminated the script earlier on failure.

So this line effectively means: All quality gates passed. Safe to proceed.

2. Running the Test Suite

flutter test --no-pub --coverage

This line executes your entire Flutter test suite.

Let’s break it down carefully.

1. flutter test

This runs unit tests, widget tests, and any test under the test/ directory. If any test fails, the command exits with a non-zero status code.

Because we enabled set -e earlier, that immediately stops the script and fails the CI job.

2. --coverage

This flag generates a coverage report at:

coverage/lcov.info

This file can later be uploaded to Codecov, used to enforce minimum coverage thresholds, and tracked over time for quality improvement.

Even if you’re not enforcing coverage yet, generating it now future-proofs your pipeline.

3. Optional Code Metrics

if command -v dart_code_metrics >/dev/null 2>&1; then
  dart_code_metrics analyze lib --reporter=console || true
fi

This block is intentionally designed to be optional and non-blocking.

Step 1 – Check If the Tool Exists:

command -v dart_code_metrics >/dev/null 2>&1

This checks whether dart_code_metrics is installed.

• If installed, proceed

• If not installed, skip silently

The redirection:

• >/dev/null hides normal output

• 2>&1 hides errors

This makes the script portable:

• Developers without the tool can still run the script

• CI can enforce it if configured

Step 2 – Run Metrics (Soft Enforcement):

dart_code_metrics analyze lib --reporter=console || true

This analyzes the lib/ directory and prints results in the console.

The important part is:

|| true

Because we enabled set -e, any failing command would normally stop the script.

Adding || true overrides that behavior:

• If metrics report issues,

• The script continues,

• CI does not fail.

Why design it this way? Because metrics are often gradual improvements, technical debt indicators, or advisory rather than blocking.

You can later remove || true to make metrics mandatory.

4. Final Success Message

echo "✅ Quality checks passed"

This line only executes if formatting passed, static analysis passed, and tests passed.

If you see this in CI logs, it means the branch has successfully cleared the quality gate. It’s your automated approval before deployment steps begin.

What This Script Guarantees

With this in place, every branch must satisfy:

• Clean formatting

• No analyzer errors

• Passing tests

• (Optional) Healthy metrics

That’s how you move from “We try to maintain quality” to “Quality is enforced automatically.”

Alright, on to the third script.

Script #3: upload_symbols.sh (Sentry)

This script is responsible for uploading obfuscation debug symbols to Sentry so production crashes remain readable.

#!/usr/bin/env bash
set -euo pipefail

RELEASE=${1:-}

[ -z "$RELEASE" ] && exit 2

if ! command -v sentry-cli >/dev/null 2>&1; then
  exit 0
fi

sentry-cli releases new "$RELEASE" || true

sentry-cli upload-dif build/symbols || true

sentry-cli releases finalize "$RELEASE" || true

echo "✅ Symbols uploaded for release $RELEASE"

Let's go through it step by step.

1. Reading the Release Identifier

RELEASE=${1:-}

This reads the first positional argument passed to the script.

When you call the script in CI, it typically looks like:

./scripts/upload_symbols.sh $(git rev-parse --short HEAD)

So $1 becomes the short Git commit SHA.

Using ${1:-} ensures:

• If no argument is passed, the variable becomes an empty string

• The script does not crash due to set -u

This release value ties the uploaded symbols, deployed build, and crash reports all to the exact same commit. This linkage is critical for production debugging.

2. Validating the Release Argument

[ -z "$RELEASE" ] && exit 2

This is a compact validation check.

• -z checks whether the string is empty

• If it is empty → exit with status code 2

Conventionally:

• 0 = success

• 1+ = failure

• 2 = incorrect usage

This prevents symbol uploads from running without a release identifier, which would break traceability in Sentry.

3. Checking If sentry-cli Exists

if ! command -v sentry-cli >/dev/null 2>&1; then
  exit 0
fi

This block checks whether the sentry-cli tool is available in the environment.

What’s happening:

• command -v sentry-cli checks if it exists

• >/dev/null 2>&1 suppresses all output

• ! negates the condition

So this reads as: "If sentry-cli is NOT installed, exit successfully."

Why exit with 0 instead of failing?

Because not every environment needs symbol uploads. Also, dev builds may not install Sentry, and you don’t want CI to fail just because Sentry isn’t configured.

This makes symbol uploading environment-aware and optional.

Production environments can install sentry-cli, while dev environments skip it cleanly.

4. Creating a New Release in Sentry

sentry-cli releases new "$RELEASE" || true

This tells Sentry: “A new release exists with this version identifier.”

Even if the release already exists, the script continues because of:

|| true

This prevents the build from failing if:

• The release was already created

• The command returns a non-critical error

The goal is resilience, not strict enforcement.

5. Uploading Debug Information Files (DIFs)

sentry-cli upload-dif build/symbols || true

This is the core step.

build/symbols is generated when you build Flutter with:

--obfuscate --split-debug-info=build/symbols

When you obfuscate Flutter builds:

• Method names are renamed

• Stack traces become unreadable

The symbol files allow Sentry to reverse-map obfuscated stack traces and show readable crash reports.

Without this step, production crashes look like:

a.b.c.d (Unknown Source)

With this step, you get:

AuthRepository.login()

Again, || true ensures the build doesn’t fail if:

• The directory doesn’t exist

• No symbols were generated

• Upload encounters a transient issue

Symbol uploads should not block deployment.

6. Finalizing the Release

sentry-cli releases finalize "$RELEASE" || true

This marks the release as complete in Sentry.

Finalizing signals:

• The release is deployed

• It can begin aggregating crash reports

• It’s ready for production monitoring

Like the previous steps, this is soft-failed with || true to keep CI robust.

What This Script Guarantees

When everything is configured correctly:

1. Production build is obfuscated

2. Debug symbols are generated

3. Symbols are uploaded to Sentry

4. Crashes map back to real source code

5. Release version matches commit SHA

That’s production-grade crash observability.

Now that we've gone through the three helper scripts we've created to optimize and enhance this process, lets now dive into the three workflow .yaml files we're going to create.

Workflow #1: PR_CHECKS.YML

This workflow will be designed to help ensure a certain standard is met once a PR is raised into a certain common or base branch. This will ensure that all quality checks in the incoming code pass before allowing any merge into the base branch.

This is basically a gate that verifies the quality of the code that's about to be merged into the base branch. If your pipeline allows unverified code into your base branch, then your CI becomes decorative, not protective.

Lets break down what's actually needed during every PR Check.

1. Dependency Integrity

For Flutter apps, where we manage dependencies with the pub get command, it's important to make sure that the integrity of all dependencies are confirmed – up to date as well as compatible.

Every PR should begin with:

flutter pub get

This ensures:

• pubspec.yaml is valid

• Dependency constraints are consistent

• Lockfiles are not broken

• The project is buildable in a clean environment

If this fails, the branch is not deployable.

2. Static Analysis

This ensures code quality and architecture integrity. Static analysis helps prevent common issues like forgotten await, dead code, null safety violations, async misuse, and so on.

Most production bugs aren't business logic errors – they're structural carelessness. Static analysis helps enforce consistency automatically, so code reviews focus on intent, not linting.

flutter analyze --fatal-infos --fatal-warnings

3. Formatting

This command ensures that your code is properly formatted based on your organization's coding standard and policies.

dart format --output=none --set-exit-if-changed .

4. Tests

This runs the unit, widget and business logic tests to ensure quality and avoid regression leaks, silent behavior changes and feature drift.

flutter test --coverage

5. Test Coverage Enforcement

Ideally, running tests is not enough. Your workflow should also enforce a minimum threshold:

if [ \((lcov --summary coverage/lcov.info | grep lines | awk '{print \)2}' | sed 's/%//') -lt 70 ]; then
  echo "Coverage too low"
  exit 1
fi

The command above ensures that a minimum test coverage of 70% is met, with this quality becomes measurable.

The five commands above must be checked (at least) for a quality gate to guarantee code quality, security, and integrity.

Now here is the full pr_checks.yml file:

name: PR Quality Gate

on:
  pull_request:
    branches: develop
    types: [opened, synchronize, reopened, ready_for_review]

jobs:
  pr-checks:
    name: Run quality checks on this pull request
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Setup Java
        uses: actions/setup-java@v1
        with:
          java-version: "12.x"

      - name: Setup Flutter
        uses: subosito/flutter-action@v1
        with:
          channel: "stable"

      - name: Install dependencies
        run: flutter pub get

      - name: Run quality checks
        run: ./scripts/quality_checks.sh

      - name: Notify Team (Success)
        if: success()
        run: |
          echo "PR Quality Checks PASSED"
          echo "PR: ${{ github.event.pull_request.html_url }}"
          echo "Branch: \({{ github.head_ref }} → \){{ github.base_ref }}"
          echo "By: @${{ github.actor }}"
          echo "Team notification: @foluwaseyi-dev @olabodegbolu"

      - name: Notify Team (Failure)
        if: failure()
        run: |
          echo "PR Quality Checks FAILED"
          echo "PR: ${{ github.event.pull_request.html_url }}"
          echo "Branch: \({{ github.head_ref }} → \){{ github.base_ref }}"
          echo "By: @${{ github.actor }}"
          echo "Please fix the issues before requesting review 🔧"
          echo "Team notification: @foluwaseyi-dev @olabodegbolu"

Every time a developer opens (or updates) a pull request targeting the develop branch, this workflow kicks in automatically. Think of it as a bouncer at the door: no code gets through without passing inspection first.

What Triggers it?

The workflow fires on four events: when a PR is opened, synchronized (new commits pushed), reopened, or marked ready_for_review. So drafts won't trigger it – only PRs that are actually ready to be looked at.

What Does it Actually Do?

It spins up a fresh Ubuntu machine and runs five steps in sequence:

1. Checkout: pulls down the branch's code

2. Setup Java 12: installs the JDK (likely a dependency for some tooling or build process)

3. Setup Flutter (stable channel): this is a Flutter project, so it grabs the stable Flutter SDK

4. Install dependencies: runs flutter pub get to pull all Dart/Flutter packages

5. Run quality checks: executes the helper shell script (./scripts/quality_checks.sh) that we created which runs linting, tests, formatting checks, or all of the above

The Notification Layer

After the checks run, the workflow reports the verdict and it's context-aware:

• If everything passes, it logs a success message with the PR URL, branch info, and the person who opened it

• If something fails, it logs a failure message and nudges the author to fix issues before requesting a review

Both outcomes tag @foluwaseyi-dev and @olabodegbolu – the two team members responsible for staying in the loop.

This workflow enforces a "fix it before you merge it" culture. No one can merge broken code into develop without the team knowing about it.

Workflow #2: Android.yml

It's a better practice to split your workflows based on platform. This helps you properly manage the instructions regarding each platform. This is the reason behind keeping the Android workflow separate.

Unlike PR _Checks, this workflow presumes that all checks for quality and standards have been done and the code that runs this workflow already meets the required standards.

Based on our predefined use case, let's create a workflow to handle test deployments when merged to develop or staging, and production level activities when merged to production.

name: Android Build & Release

on:
  push:
    branches:
      - develop
      - staging
      - production

jobs:
  android:
    runs-on: ubuntu-latest
    env:
      FLUTTER_VERSION: 'stable'

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '11'

      - name: Setup Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: ${{ env.FLUTTER_VERSION }}

      - name: Install dependencies
        run: flutter pub get

      - name: Determine environment
        id: env
        run: |
          echo "branch=\({GITHUB_REF##*/}" >> \)GITHUB_OUTPUT
          if [ "${GITHUB_REF##*/}" = "develop" ]; then
            echo "ENV=dev" >> $GITHUB_OUTPUT
          elif [ "${GITHUB_REF##*/}" = "staging" ]; then
            echo "ENV=staging" >> $GITHUB_OUTPUT
          else
            echo "ENV=production" >> $GITHUB_OUTPUT
          fi

      # Dev uses hardcoded values no secrets needed
      - name: Generate config (dev)
        if: steps.env.outputs.ENV == 'dev'
        run: ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"

      # Staging and production inject real secrets
      - name: Generate config (staging/production)
        if: steps.env.outputs.ENV != 'dev'
        run: |
          if [ "${{ steps.env.outputs.ENV }}" = "staging" ]; then
            ./scripts/generate_config.sh staging \
              "${{ secrets.STAGING_BASE_URL }}" \
              "${{ secrets.STAGING_API_KEY }}"
          else
            ./scripts/generate_config.sh production \
              "${{ secrets.PROD_BASE_URL }}" \
              "${{ secrets.PROD_API_KEY }}"
          fi

      # Keystore is only needed for signed builds (staging & production)
      - name: Restore Keystore
        if: steps.env.outputs.ENV != 'dev'
        run: |
          echo "${{ secrets.ANDROID_KEYSTORE_BASE64 }}" | base64 --decode > android/app/upload-keystore.jks

      # Production builds are obfuscated + split debug info for Play Store
      - name: Build artifact
        run: |
          if [ "${{ steps.env.outputs.ENV }}" = "production" ]; then
            flutter build appbundle --release \
              --obfuscate \
              --split-debug-info=build/symbols
          else
            flutter build appbundle --release
          fi

      # Dev and staging go to Firebase App Distribution for internal testing
      - name: Upload to Firebase App Distribution
        if: steps.env.outputs.ENV == 'dev' || steps.env.outputs.ENV == 'staging'
        env:
          FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
          FIREBASE_ANDROID_APP_ID: ${{ secrets.FIREBASE_ANDROID_APP_ID }}
          FIREBASE_GROUPS: ${{ secrets.FIREBASE_GROUPS }}
        run: |
          firebase appdistribution:distribute \
            build/app/outputs/bundle/release/app-release.aab \
            --app "$FIREBASE_ANDROID_APP_ID" \
            --groups "$FIREBASE_GROUPS" \
            --token "$FIREBASE_TOKEN"

      # Only production goes to the Play Store
      - name: Upload to Play Store
        if: steps.env.outputs.ENV == 'production'
        uses: r0adkll/upload-google-play@v1
        with:
          serviceAccountJsonPlainText: ${{ secrets.GOOGLE_PLAY_SERVICE_ACCOUNT_JSON }}
          packageName: com.your.package
          releaseFiles: build/app/outputs/bundle/release/app-release.aab
          track: production

      - name: Notify Team (Success)
        if: success()
        run: |
          echo "Android Build & Release PASSED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"

      - name: Notify Team (Failure)
        if: failure()
        run: |
          echo "Android Build & Release FAILED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"
          echo "Check the logs and fix the issue before retrying"

This workflow ensures that whenever code lands on the develop, staging or production branch, this action is triggered on a fresh Ubuntu machine.

This is triggered by a simple push to any of the tracked branches, no manual intervention needed.

Let's walk through it piece by piece.

1. The Setup Phase

Before any Flutter-specific work happens, the workflow lays the foundation:

1. Checkout: grabs the latest code from the branch that triggered the run (using the more modern actions/checkout@v3).

2. Java 11 via Temurin: this is an upgrade from the first workflow we created. Instead of a generic setup-java@v1, this uses the temurin distribution which is the Eclipse's open-source JDK build. It's the current industry standard for Android toolchains.

3. Flutter (stable): this pulls the stable Flutter SDK, version pinned via an environment variable (FLUTTER_VERSION: 'stable') defined at the job level.

4. Install dependencies: this ensures we run flutter pub get to pull all packages

2. Environment Detection

This is where it gets interesting. This workflow also checks and determines the environment which will help us define the next set of instructions to run.

This command reads the branch name from GITHUB REF and maps it to its environment label which we already created in one of our helper scripts.

• develop → ENV=dev

• staging → ENV=staging

• production → ENV=production

It strips the branch name from the full ref path using \({GITHUB_REF##*/}, then writes both the branch name and the resolved ENV value to \)GITHUB_OUTPUT, making them available as named outputs (steps.env.outputs.ENV) for every subsequent step.

This means the rest of the pipeline can branch its behaviour based on which environment it's building for, different API keys, different signing configs, different targets – whatever the app needs.

3. Config Injection

With the environment resolved, the next step is injecting the right configuration into the app. This is where the generate_config.sh script we built earlier gets called directly from the workflow.

For the dev environment, hardcoded placeholder values are used. No real secrets are needed, since this build is only meant for internal developer testing:

- name: Generate config (dev)
  if: steps.env.outputs.ENV == 'dev'
  run: ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"

For staging and production, however, real secrets are pulled from GitHub Actions secrets and passed directly into the script:

- name: Generate config (staging/production)
  if: steps.env.outputs.ENV != 'dev'
  run: |
    if [ "${{ steps.env.outputs.ENV }}" = "staging" ]; then
      ./scripts/generate_config.sh staging \
        "${{ secrets.STAGING_BASE_URL }}" \
        "${{ secrets.STAGING_API_KEY }}"
    else
      ./scripts/generate_config.sh production \
        "${{ secrets.PROD_BASE_URL }}" \
        "${{ secrets.PROD_API_KEY }}"
    fi

Notice that these two steps use an if condition to make them mutually exclusive. Only one will ever run per job. This keeps the pipeline clean: no complicated branching logic inside the script itself, just a clear decision at the workflow level.

4. Keystore Restoration

Android requires signed builds for distribution. The signing keystore file cannot be committed to the repository for obvious security reasons, so it's stored as a Base64-encoded GitHub secret and decoded at build time.

- name: Restore Keystore
  if: steps.env.outputs.ENV != 'dev'
  run: |
    echo "${{ secrets.ANDROID_KEYSTORE_BASE64 }}" | base64 --decode > android/app/upload-keystore.jks

This step is skipped entirely for the dev environment because dev builds are unsigned debug artifacts meant purely for internal testing on Firebase App Distribution. Only staging and production builds need to be properly signed.

To encode your keystore file as a Base64 string for storing in GitHub secrets, you have to run this locally:

base64 -i upload-keystore.jks | pbcopy

This copies the encoded string to your clipboard, which you can then paste directly into your GitHub repository secrets.

5. Building the Artifact

With the environment configured and the keystore in place, the workflow builds the app bundle:

- name: Build artifact
  run: |
    if [ "${{ steps.env.outputs.ENV }}" = "production" ]; then
      flutter build appbundle --release \
        --obfuscate \
        --split-debug-info=build/symbols
    else
      flutter build appbundle --release
    fi

There's a deliberate difference between how production and non-production builds are compiled.

For production:

• --obfuscate renames method and class names in the compiled output, making it significantly harder to reverse engineer the app

• --split-debug-info=build/symbols extracts the debug symbols into a separate directory at build/symbols

These symbols are what upload_symbols.sh later ships to Sentry, so obfuscated crash reports remain readable in your monitoring dashboard.

For dev and staging, neither flag is used. This keeps build times faster and makes local debugging easier since stack traces remain human-readable.

6. Distributing to Firebase App Distribution

Once the app bundle is built, dev and staging builds are uploaded to Firebase App Distribution so testers can install them immediately:

- name: Upload to Firebase App Distribution
  if: steps.env.outputs.ENV == 'dev' || steps.env.outputs.ENV == 'staging'
  env:
    FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
    FIREBASE_ANDROID_APP_ID: ${{ secrets.FIREBASE_ANDROID_APP_ID }}
    FIREBASE_GROUPS: ${{ secrets.FIREBASE_GROUPS }}
  run: |
    firebase appdistribution:distribute \
      build/app/outputs/bundle/release/app-release.aab \
      --app "$FIREBASE_ANDROID_APP_ID" \
      --groups "$FIREBASE_GROUPS" \
      --token "$FIREBASE_TOKEN"

Three secrets power this step:

• FIREBASE_TOKEN: the authentication token generated from firebase login:ci

• FIREBASE_ANDROID_APP_ID: the app identifier from the Firebase console

• FIREBASE_GROUPS: the tester group(s) that should receive the build notification

Once this step completes, every tester in the specified groups receives an email with a direct download link. No one needs to manually share an APK file over Slack or email.

7. Deploying to the Play Store

Production builds skip Firebase entirely and goes straight to the Google Play Store:

- name: Upload to Play Store
  if: steps.env.outputs.ENV == 'production'
  uses: r0adkll/upload-google-play@v1
  with:
    serviceAccountJsonPlainText: ${{ secrets.GOOGLE_PLAY_SERVICE_ACCOUNT_JSON }}
    packageName: com.your.package
    releaseFiles: build/app/outputs/bundle/release/app-release.aab
    track: production

This uses the r0adkll/upload-google-play GitHub Action, which handles the Google Play API interaction under the hood. The only requirements are:

• A Google Play service account with the correct permissions, stored as a JSON secret

• The correct package name matching what is registered in your Play Console

• The track set to production (you can also use internal, alpha, or beta depending on your release strategy)

Replace com.your.package with your actual application ID (the same one defined in your build.gradle file).

8. The Notification Layer

Just like the PR checks workflow, this workflow reports its outcome clearly:

- name: Notify Team (Success)
  if: success()
  run: |
    echo "Android Build & Release PASSED"
    echo "Environment: ${{ steps.env.outputs.ENV }}"
    echo "Branch: ${{ steps.env.outputs.branch }}"
    echo "By: @${{ github.actor }}"
    echo "Commit: ${{ github.sha }}"

- name: Notify Team (Failure)
  if: failure()
  run: |
    echo "Android Build & Release FAILED"
    echo "Environment: ${{ steps.env.outputs.ENV }}"
    echo "Branch: ${{ steps.env.outputs.branch }}"
    echo "By: @${{ github.actor }}"
    echo "Commit: ${{ github.sha }}"
    echo "Check the logs and fix the issue before retrying 🔧"

The success notification includes the environment, branch, actor, and shares everything needed to trace exactly what was deployed and who triggered it.

The failure notification includes the same context, with a clear call to action.

Workflow #3: iOS.yml

iOS CI/CD is more complex than Android by nature. This is because Apple's signing requirements involve certificates, provisioning profiles, and entitlements that all need to be in the right place before Xcode will produce a valid archive.

Fastlane helps us handles all of that complexity, and the workflow simply calls into it.

Here is the full ios.yml:

name: iOS Build & Release

on:
  push:
    branches:
      - develop
      - staging
      - production

jobs:
  ios:
    runs-on: macos-latest
    env:
      FLUTTER_VERSION: 'stable'

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: ${{ env.FLUTTER_VERSION }}

      - name: Install dependencies
        run: flutter pub get

      - name: Determine environment
        id: env
        run: |
          echo "branch=\({GITHUB_REF##*/}" >> \)GITHUB_OUTPUT
          if [ "${GITHUB_REF##*/}" = "develop" ]; then
            echo "ENV=dev" >> $GITHUB_OUTPUT
          elif [ "${GITHUB_REF##*/}" = "staging" ]; then
            echo "ENV=staging" >> $GITHUB_OUTPUT
          else
            echo "ENV=production" >> $GITHUB_OUTPUT
          fi

      - name: Generate config (dev)
        if: steps.env.outputs.ENV == 'dev'
        run: ./scripts/generate_config.sh dev "https://dev.api.example.com" "dev_dummy_key"

      - name: Generate config (staging/production)
        if: steps.env.outputs.ENV != 'dev'
        run: |
          if [ "${{ steps.env.outputs.ENV }}" = "staging" ]; then
            ./scripts/generate_config.sh staging \
              "${{ secrets.STAGING_BASE_URL }}" \
              "${{ secrets.STAGING_API_KEY }}"
          else
            ./scripts/generate_config.sh production \
              "${{ secrets.PROD_BASE_URL }}" \
              "${{ secrets.PROD_API_KEY }}"
          fi

      - name: Install Fastlane
        run: |
          cd ios
          gem install bundler
          bundle install

      - name: Import signing certificate
        if: steps.env.outputs.ENV != 'dev'
        run: |
          echo "${{ secrets.IOS_CERTIFICATE_BASE64 }}" | base64 --decode > ios/cert.p12
          security create-keychain -p "" build.keychain
          security import ios/cert.p12 -k build.keychain -P "${{ secrets.IOS_CERTIFICATE_PASSWORD }}" -T /usr/bin/codesign
          security list-keychains -s build.keychain
          security default-keychain -s build.keychain
          security unlock-keychain -p "" build.keychain
          security set-key-partition-list -S apple-tool:,apple: -s -k "" build.keychain

      - name: Install provisioning profile
        if: steps.env.outputs.ENV != 'dev'
        run: |
          echo "${{ secrets.IOS_PROVISIONING_PROFILE_BASE64 }}" | base64 --decode > profile.mobileprovision
          mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
          cp profile.mobileprovision ~/Library/MobileDevice/Provisioning\ Profiles/

      - name: Build iOS (dev)
        if: steps.env.outputs.ENV == 'dev'
        run: flutter build ios --release --no-codesign

      - name: Build & distribute to TestFlight (staging)
        if: steps.env.outputs.ENV == 'staging'
        env:
          APP_STORE_CONNECT_API_KEY_ID: ${{ secrets.APP_STORE_CONNECT_API_KEY_ID }}
          APP_STORE_CONNECT_API_ISSUER_ID: ${{ secrets.APP_STORE_CONNECT_API_ISSUER_ID }}
          APP_STORE_CONNECT_API_KEY_CONTENT: ${{ secrets.APP_STORE_CONNECT_API_KEY_CONTENT }}
        run: |
          cd ios
          bundle exec fastlane beta

      - name: Build & release to App Store (production)
        if: steps.env.outputs.ENV == 'production'
        env:
          APP_STORE_CONNECT_API_KEY_ID: ${{ secrets.APP_STORE_CONNECT_API_KEY_ID }}
          APP_STORE_CONNECT_API_ISSUER_ID: ${{ secrets.APP_STORE_CONNECT_API_ISSUER_ID }}
          APP_STORE_CONNECT_API_KEY_CONTENT: ${{ secrets.APP_STORE_CONNECT_API_KEY_CONTENT }}
        run: |
          cd ios
          bundle exec fastlane release

      - name: Upload Sentry symbols (production only)
        if: steps.env.outputs.ENV == 'production'
        env:
          SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
          SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
          SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}
        run: ./scripts/upload_symbols.sh $(git rev-parse --short HEAD)

      - name: Notify Team (Success)
        if: success()
        run: |
          echo "iOS Build & Release PASSED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"

      - name: Notify Team (Failure)
        if: failure()
        run: |
          echo "iOS Build & Release FAILED"
          echo "Environment: ${{ steps.env.outputs.ENV }}"
          echo "Branch: ${{ steps.env.outputs.branch }}"
          echo "By: @${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"
          echo "Check the logs and fix the issue before retrying 🔧"

Let's walk through what is different about this workflow compared to that of android.

1. MacOS Runner

runs-on: macos-latest

This is the major difference.

iOS builds require Xcode, which only runs on macOS. GitHub Actions provides hosted macOS runners, but they are significantly more expensive in terms of compute minutes than Ubuntu runners. Just keep that in mind when thinking about build frequency.

No Java setup is needed here. Flutter on iOS compiles through Xcode directly, so the toolchain requirements are different.

2. Installing Fastlane

- name: Install Fastlane
  run: |
    cd ios
    gem install bundler
    bundle install

Fastlane is a Ruby-based automation tool that handles certificate management, building, and uploading to TestFlight and the App Store.

This step navigates into the ios/ directory and installs Fastlane along with all its dependencies as defined in the project's Gemfile.

Your ios/Gemfile should look something like this:

source "https://rubygems.org"

gem "fastlane"

And your ios/fastlane/Fastfile should define at minimum two lanes: one for staging (TestFlight) and one for production (App Store):

default_platform(:ios)

platform :ios do
  lane :beta do
    build_app(scheme: "Runner", export_method: "app-store")
    upload_to_testflight(skip_waiting_for_build_processing: true)
  end

  lane :release do
    build_app(scheme: "Runner", export_method: "app-store")
    upload_to_app_store(force: true, skip_screenshots: true, skip_metadata: true)
  end
end

3. Certificate and Provisioning Profile Setup

This is the step that trips most teams up the first time. Apple's code signing requires two things to be present on the machine:

1. The signing certificate (a .p12 file)

2. The provisioning profile

Both are stored as Base64-encoded GitHub secrets and restored at build time.

- name: Import signing certificate
  if: steps.env.outputs.ENV != 'dev'
  run: |
    echo "${{ secrets.IOS_CERTIFICATE_BASE64 }}" | base64 --decode > ios/cert.p12
    security create-keychain -p "" build.keychain
    security import ios/cert.p12 -k build.keychain -P "${{ secrets.IOS_CERTIFICATE_PASSWORD }}" -T /usr/bin/codesign
    security list-keychains -s build.keychain
    security default-keychain -s build.keychain
    security unlock-keychain -p "" build.keychain
    security set-key-partition-list -S apple-tool:,apple: -s -k "" build.keychain

Breaking this down step by step:

• Decodes the Base64 certificate and write it to cert.p12

• Creates a temporary keychain called build.keychain with an empty password

• Imports the certificate into that keychain, granting codesign access

• Sets it as the default keychain so Xcode finds it automatically

• Unlocks the keychain so it can be used non-interactively

• Sets partition list to allow access without repeated prompts

The provisioning profile step is simpler:

- name: Install provisioning profile
  if: steps.env.outputs.ENV != 'dev'
  run: |
    echo "${{ secrets.IOS_PROVISIONING_PROFILE_BASE64 }}" | base64 --decode > profile.mobileprovision
    mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
    cp profile.mobileprovision ~/Library/MobileDevice/Provisioning\ Profiles/

It decodes the profile and copies it into the exact directory where Xcode expects to find provisioning profiles on any macOS system.

To encode your certificate and profile locally, you can run these:

base64 -i Certificates.p12 | pbcopy   # for the certificate
base64 -i YourApp.mobileprovision | pbcopy   # for the provisioning profile

4. Building for Each Environment

Dev builds skip signing entirely. They're built without code signing just to verify the project compiles correctly on a clean machine:

- name: Build iOS (dev)
  if: steps.env.outputs.ENV == 'dev'
  run: flutter build ios --release --no-codesign

Staging builds go through Fastlane's beta lane, which builds and uploads to TestFlight. Production builds go through Fastlane's release lane, which submits directly to App Store Connect.

Both staging and production steps consume the same three App Store Connect API key secrets: the key ID, the issuer ID, and the key content itself.

Fastlane uses these to authenticate with Apple's API without requiring a manual Apple ID login.

5. Sentry Symbol Upload

On production iOS builds, the upload_symbols.sh script runs after the build completes, passing the current short commit SHA as the release identifier:

- name: Upload Sentry symbols (production only)
  if: steps.env.outputs.ENV == 'production'
  env:
    SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
    SENTRY_ORG: ${{ secrets.SENTRY_ORG }}
    SENTRY_PROJECT: ${{ secrets.SENTRY_PROJECT }}
  run: ./scripts/upload_symbols.sh $(git rev-parse --short HEAD)

This is the same script explained earlier in the helper scripts section. It creates a Sentry release, uploads the debug information files, and finalizes the release. Any production crash from this point forward will map back to real, readable source code in your Sentry dashboard.

Secrets and Configuration Reference

For this entire pipeline to work, you need to configure the following secrets in your GitHub repository. Go to Settings → Secrets and variables → Actions → New repository secret to add each one.

Shared (used across environments):

Staging:

Production:

Android:

iOS:

None of these values should ever appear in your codebase. If any secret is accidentally committed, rotate it immediately.

End-to-End Flow

With all three workflows in place, here is exactly what happens from the moment a developer opens a pull request to the moment a user receives an update:

1. Developer Opens a PR into develop

The pr_checks.yml workflow fires. It runs formatting checks, static analysis, and the full test suite. If anything fails, the PR cannot be merged and the team is notified immediately. The developer fixes the issues and pushes again, which triggers a fresh run.

2. PR is Approved and Merged into develop

The android.yml and ios.yml workflows both fire on the push event. They detect the environment as dev, inject placeholder config, build unsigned artifacts, and upload them to Firebase App Distribution. Testers receive an email and can install the build on their devices within minutes – no one shared a file manually.

3. develop is Merged into staging

Both platform workflows fire again. This time the environment resolves to staging. Real secrets are injected, builds are properly signed, and the artifacts go to Firebase App Distribution (Android) and TestFlight (iOS). QA begins testing the staging build against the staging API.

4. staging is merged into production

Both workflows fire one final time. Production secrets are injected, builds are obfuscated and signed, debug symbols are uploaded to Sentry, and the final artifacts are submitted to the Google Play Store and App Store Connect. The release goes live on Apple and Google's review timelines with no further human intervention required.

From that first PR to a production submission, not a single command was run manually.

Conclusion

Building this pipeline is an upfront investment that pays off from the very first release cycle. What used to be a sequence of error-prone manual steps building locally, signing, uploading, switching configs, and hoping nothing was mixed up is now a fully automated, auditable, and repeatable process that runs the moment code moves between branches.

The architecture we built here does more than just automate builds. The PR quality gate enforces team standards consistently, so code review becomes a conversation about intent rather than a hunt for formatting issues. The environment-aware config injection eliminates an entire class of production incidents where staging keys made it into a live release. The Sentry symbol upload means your team can debug production crashes with full source visibility even from an obfuscated binary.

Every piece of this pipeline also runs locally. The helper scripts in the scripts/ folder are plain Bash so you can call them from your terminal the same way CI calls them. This eliminates the frustrating cycle of pushing a commit just to test a pipeline change.

As your team grows, this foundation scales with you. You can extend the pr_checks.yml to enforce code coverage thresholds, add a performance benchmarking job, or introduce a dedicated security scanning step. You can extend the platform workflows to support multiple flavors, multiple Firebase projects, or staged rollouts on the Play Store. The architecture stays the same – you're just adding new steps to an already working system.

This ensures that standards are met, code quality remains high, you have a proper team structure, clear process and automated post development activities are in place – and at the end of the day, you'll have an optimized engineering approach that will help your team in so many ways.

For my work on circular economy and battery recycling, I needed a way to query multiple databases at once without the manual fatigue.

In this tutorial, you'll build an automated research pipeline using n8n that reduces roughly six hours of manual literature review into a five-minute automated process.

This isn’t a “cool demo workflow.” It’s a production-minded pipeline with parallel collection, normalisation, deduplication, structured AI extraction, scoring, and practical error handling.

Table of Contents

1. Prerequisites

2. The Problem: Research Takes Too Long

3. The Tech Stack

4. The Project Structure: How to Think About an n8n Workflow Like Software

5. Stage 1: Centralised Configuration

6. Stage 2: Parallel API Collection (With Failure Isolation)

7. Stage 3: Normalisation and Deduplication (DOI-first, Title fallback)

8. Stage 4: AI-Powered Content Extraction (Strict JSON)

9. Stage 5: Scoring and Synthesis

10. [Beginner-Friendly Evals (Retrieval and Extraction QA)(#heading-beginnerfriendly-evals-retrieval-and-extraction-qa)

11. Key Learnings and Error Handling

12. Conclusion

Prerequisites

You don’t need to be a DevOps engineer to follow this, but you should have:

• Basic comfort with APIs and JSON (request/response payloads)

• Familiarity with spreadsheets (Google Sheets basics)

• Willingness to use a small amount of JavaScript inside n8n Function/Code nodes

Access to:

• An n8n instance (self-hosted or cloud)

• A Groq API key (or a compatible LLM provider)

• Optional API keys, depending on the databases you use

What you’ll build assumes:

• You’re extracting from metadata + abstracts (not downloading full PDFs).

• You can accept that some sources will occasionally rate-limit or return partial results (and your workflow will be designed to survive this).

The Problem: Research Takes Too Long

Manual research is often a bottleneck for innovation. Before building this automation, my workflow involved searching multiple academic databases, scanning abstracts, and manually extracting key findings. This process was not only slow but also prone to human error and inconsistent note-taking.

The goal of this automation is to provide a “full-stack research assistant” that handles the heavy lifting of collecting candidate papers, removing duplicates, extracting consistent fields, scoring relevance and quality, and delivering a curated daily or weekly report, so you can spend your time on high-level synthesis rather than repetitive collection.

The Tech Stack

This workflow leverages a combination of automation tooling, high-speed LLM inference, and academic metadata providers.

Notes: coverage varies by provider. Some APIs return abstracts reliably, while others may omit them. Your pipeline should treat missing abstracts as a normal case, not a failure.

The Project Structure: How to Think About an n8n Workflow Like Software

While n8n is a visual tool, it helps to design your workflow as modular stages to avoid the “spaghetti workflow” problem.

.
├── configuration/         # Keywords, thresholds, limits, date filters
├── collectors/            # Parallel HTTP request nodes (multiple sources)
├── processing/            # Normalization + deduplication code nodes
├── extraction/            # LLM extraction nodes (strict JSON)
├── scoring/               # Relevance + quality scoring + filtering
└── delivery/              # Google Sheets + email/HTML report

Design principle: each stage should produce a clean, predictable output shape that the next stage can rely on.

Stage 1: Centralised Configuration

Instead of hardcoding search parameters (keywords, min year, citation thresholds) across multiple nodes, use one configuration node to define workflow variables.

This matters for maintainability (change a value once, not in ten nodes), reusability (repurpose the entire pipeline by swapping one config object), and debuggability (log the config at the start of each run so you can reproduce results).

Use a Set node, or a Code node returning JSON like this:

{
  "keywords": "circular economy battery recycling remanufacturing",
  "min_year": 2020,
  "max_results_per_source": 10,
  "min_citations": 2,
  "relevance_threshold": 15,
  "batch_size": 10
}

Tip: keep numeric fields as numbers (not strings) to avoid scoring bugs later.

Stage 2: Parallel API Collection (With Failure Isolation)

Your workflow should query multiple sources simultaneously. In n8n, you can branch from your configuration node into multiple HTTP Request nodes, and then merge results later.

The production mindset here is simple: APIs fail. Rate limits happen. Providers return partial data. The key is to prevent one failing collector from crashing the whole run.

To implement this, on each HTTP Request node, enable Continue On Fail (or the equivalent “don’t stop workflow” behaviour). Then, in the normalisation stage, treat missing or failed outputs as empty arrays so downstream stages still run.

In practice, it also helps to set explicit timeouts and add a small retry policy (one to two retries) for transient failures. “Good” looks like this: if two out of five sources fail, you still produce a useful report from the remaining three, and you log which sources failed so you can investigate later.

Stage 3: Normalisation and Deduplication (DOI-first, Title fallback)

Each academic API returns different field names and shapes. One might use title, another display_name, another paper_title. Your next stage should normalise all inputs into one schema.

Target normalised schema

Here’s a simple baseline schema (expand later as needed):

{
  "title": "string",
  "abstract": "string|null",
  "doi": "string|null",
  "year": 2024,
  "citations": 12,
  "url": "string|null",
  "source": "Semantic Scholar|OpenAlex|arXiv|PubMed"
}

What deduping by DOI means (and what a DOI is)

A DOI (Digital Object Identifier) is a unique, persistent identifier assigned to many scholarly publications. If a paper has a DOI, that DOI functions like a stable ID: the same paper may appear in multiple databases with slightly different metadata, but the DOI should remain consistent.

So, deduping by DOI means: if two records share the same DOI, treat them as the same paper and keep only one.

When a DOI is missing (which is common for some preprints and some API responses), the fallback is to dedupe using a normalised title key, lowercased, trimmed, punctuation stripped, and whitespace collapsed. It’s not as perfect as DOI-based matching, but it’s a strong pragmatic backup.

What “normalise into a unified object” means (what’s happening in the code)

“Normalise into a unified object” simply means converting every provider’s raw response into the same predictable shape (the schema above). Once everything looks the same, downstream steps, such as deduplication, scoring, AI extraction, and storage, become straightforward because they don’t need provider-specific logic.

In the code below, that’s what the normalized object is: it maps Semantic Scholar’s fields (paper.title, paper.externalIds.DOI, paper.citationCount) into your standard fields (title, doi, citations, etc.). After that, the workflow generates a dedupe key (doi:... if DOI exists, otherwise title:...) and uses a Set to keep only the first occurrence.

Example n8n Code Node (Normalisation + Dedupe Pattern)

const itemsIn = $input.all();

const seen = new Set();
const results = [];

function titleKey(t) {
  return (t || "")
    .toLowerCase()
    .replace(/[\W_]+/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}

for (const item of itemsIn) {
  // Example: Semantic Scholar response shape
  const papers = item.json?.data || [];

  for (const paper of papers) {
    // "Normalize into a unified object":
    // take the provider-specific fields and map them into our standard schema.
    const normalized = {
      title: paper.title || null,
      abstract: paper.abstract || null,
      doi: paper.externalIds?.DOI || null,
      year: paper.year || null,
      citations: paper.citationCount || 0,
      url: paper.url || null,
      source: "Semantic Scholar",
    };

    if (!normalized.title) continue;

    // Dedupe key: DOI is strongest; title is fallback
    const key = normalized.doi
      ? `doi:${normalized.doi.toLowerCase()}`
      : `title:${titleKey(normalized.title)}`;

    if (seen.has(key)) continue;
    seen.add(key);

    results.push(normalized);
  }
}

return results.map(r => ({ json: r }));

Production-minded note: keep a field like source so you can debug where bad metadata is coming from later.

Stage 4: AI-Powered Content Extraction (Strict JSON)

Once you have a deduplicated list of papers, you can send each paper (or a small batch) to Groq for structured extraction.

Why structured output matters

If your LLM returns narrative text instead of JSON, misses fields, or emits malformed JSON, your workflow breaks downstream. In a production workflow, that’s not a rare edge case; it’s something you should expect and design around.

That’s why you’ll use strict schema prompting and validate responses downstream.

System prompt vs user prompt (and how to compose them)

A helpful way to think about prompts in production is:

• The system prompt defines the non-negotiable contract: output format, allowed keys, no commentary, and what to do in uncertain cases. This is where you say “return ONLY valid JSON” and “no extra keys.”

• The user prompt provides the variable data for this specific request: title, year, citations, abstract, and the exact schema you want filled.

Composing them this way keeps your workflow stable. The system prompt stays mostly constant (your formatting contract), while the user prompt changes per paper (your payload). It also makes debugging easier: if outputs start failing, you can adjust the system constraints without rewriting every payload template.

Suggested extraction schema

Extract only what you can support from abstract-level data:

• research_question

• methodology

• key_findings

• limitations

• notes (for missing abstract / ambiguity)

Example prompt (system + user)

System:

You are a research extraction engine. You must return ONLY valid JSON.
No markdown. No extra keys. No commentary.
If the abstract is missing or too vague, set fields to null and include a reason in "notes".

User:

Extract structured fields from this paper.

TITLE: {{title}}
YEAR: {{year}}
CITATIONS: {{citations}}
ABSTRACT: {{abstract}}

Return JSON with keys:
research_question (string|null)
methodology (string|null)
key_findings (array of strings)
limitations (array of strings)
notes (string)

Model settings: keep temperature low (around 0.2–0.3) and keep responses short and structured.

Batch processing to avoid timeouts

Instead of sending 50 papers at once, process them in batches (for example, 10). This reduces latency spikes, failure blast radius, and cost surprises. Smaller batches also make it easier to retry only the failing chunk rather than re-running everything.

Stage 5: Scoring and Synthesis

Not every retrieved paper is worth your time. Without scoring, your pipeline becomes a firehose: you’ve automated collection, but you still have to manually decide what to read. Scoring is what turns “a big list of results” into a shortlist you can trust.

I recommend computing two signals:

• Relevance: Is this actually about your research question?

• Quality/priority: If it’s relevant, is it worth reading first?

For relevance, keep it simple and explainable. Count keyword hits in the title and abstract (and optionally in extracted key_findings). Title matches should be weighted higher because titles are deliberately compact summaries. Abstract hits are useful too, but cap them so long abstracts don’t dominate the score.

For quality/priority, use lightweight metadata you already have. Recency is a strong signal in fast-moving areas, and citations can help, but they should be treated as a weak signal (and capped) so newer high-value papers aren’t unfairly penalised.

A solid first scoring model is: add a title bonus, add a capped abstract bonus, add a capped citations bonus, and add a small recency bonus for papers from the last two years. Then filter using the relevance_threshold results from Stage 1. The advantage of this approach is that it’s easy to debug and tune: you can always explain why a paper passed or failed.

Once you’ve filtered down to your “gold” set, synthesis becomes safer and more useful. Write one row per accepted paper to Google Sheets, then generate a daily/weekly HTML summary (for example, top 5 papers with 1–2 key findings each) and include links so you can verify quickly.

Beginner-Friendly Evals: Retrieval and Extraction QA

AI workflows regress silently. A prompt tweak, a model update, or an API schema change can break extraction without throwing an obvious error. Adding lightweight evals is the difference between “it worked last week” and “it’s reliable.”

The goal here isn’t to build a full evaluation framework. It’s to add small, cheap checks that catch the most common failure modes:

• Are collectors still returning results?

• Are we actually removing duplicates?

• Is the LLM returning valid JSON with the keys we require?

What it looks like in n8n (a concrete example)

A simple implementation is to add an “Assertions” Code node immediately after your extraction step, plus (optionally) another one after normalisation/deduplication.

At a high level, the workflow section looks like:

1. Collectors (parallel HTTP Request nodes)

2. Merge results

3. Normalise + dedupe (Code node)

4. Split in Batches (optional)

5. LLM extraction (Groq/OpenAI-compatible node)

6. Assertions (Code node)

7. If node (pass/fail)

8. Delivery (Sheets + email)

Example: Assertions code node after extraction

This code node assumes each item is a paper with:

• title, abstract in the normalised fields, and

• an extraction field (or whatever you name it) containing the LLM response as an object or JSON string.

Adapt the field name to match your actual node output, but the pattern is the same: parse, validate required keys, compute percentages, then decide whether to fail or warn.

const items = $input.all();

let total = items.length;
let withTitle = 0;
let withAbstract = 0;

let parseOk = 0;
let schemaOk = 0;

const requiredKeys = [
  "research_question",
  "methodology",
  "key_findings",
  "limitations",
  "notes",
];

const failures = [];

for (let i = 0; i < items.length; i++) {
  const p = items[i].json;

  if (p.title && String(p.title).trim().length > 0) withTitle++;
  if (p.abstract && String(p.abstract).trim().length > 0) withAbstract++;

  // Adjust this depending on where you store the model output:
  const raw = p.extraction ?? p.llm ?? p.model_output;

  let obj = null;
  try {
    obj = typeof raw === "string" ? JSON.parse(raw) : raw;
    parseOk++;
  } catch (e) {
    failures.push({ index: i, title: p.title || null, reason: "JSON parse failed" });
    continue;
  }

  const hasAllKeys = requiredKeys.every(k => Object.prototype.hasOwnProperty.call(obj, k));
  if (!hasAllKeys) {
    failures.push({ index: i, title: p.title || null, reason: "Missing required keys" });
    continue;
  }

  // Optional: ensure arrays are arrays
  const arraysOk = Array.isArray(obj.key_findings) && Array.isArray(obj.limitations);
  if (!arraysOk) {
    failures.push({ index: i, title: p.title || null, reason: "key_findings/limitations not arrays" });
    continue;
  }

  schemaOk++;
}

const pct = (n) => (total === 0 ? 0 : Math.round((n / total) * 100));

const report = {
  total_papers: total,
  pct_with_title: pct(withTitle),
  pct_with_abstract: pct(withAbstract),
  pct_extraction_json_parse_ok: pct(parseOk),
  pct_extraction_schema_ok: pct(schemaOk),
  failures_sample: failures.slice(0, 5),
};

// Decide pass/fail thresholds
const HARD_FAIL_PARSE_BELOW = 90;
const HARD_FAIL_SCHEMA_BELOW = 85;

const shouldFail =
  report.pct_extraction_json_parse_ok < HARD_FAIL_PARSE_BELOW ||
  report.pct_extraction_schema_ok < HARD_FAIL_SCHEMA_BELOW;

return [
  {
    json: {
      eval_report: report,
      shouldFail,
    },
  },
];

Then add an If node:

• If shouldFail is true, then route to an “Alert/Stop” branch (Slack/email/log) and optionally stop the workflow.

• If false, then continue to the delivery stage.

This is the automation equivalent of unit tests: small, cheap, and extremely effective. It also gives you a concrete paper trail when something changes upstream.

Key Learnings and Error Handling

Building this automation taught me that the best workflows are designed for failure.

First, error resilience is not optional. Never let one failing API crash the workflow. Use “Continue On Fail” on your HTTP nodes, merge partial results, and log which sources failed in your final report so you can debug without losing an entire run.

Second, batching is your friend. Process papers in batches (often 5–15) to reduce timeouts and cost spikes. Keep LLM payloads small and focused on what you actually need (metadata + abstract), and retry transient failures once rather than repeatedly hammering the model or API.

Third, structured prompting is what makes AI reliable in automation. A strict JSON schema is the difference between a workflow that runs unattended and one that breaks randomly. Keep temperature low, enforce the schema in the system prompt, and validate everything downstream with simple parse-and-assert checks.

Conclusion

A good research pipeline doesn’t just retrieve papers – it turns scattered results into a consistent, deduplicated, scored, and review-ready shortlist you can trust.

By treating your n8n workflow like software modular stages, strict contracts between steps, and lightweight eval checks, you can reduce hours of manual literature review into a fast, repeatable process that survives real-world API failures and model quirks.

If you build this with good defaults (failure isolation, batching, normalisation, strict JSON extraction, and simple scoring), you end up with something you can run daily or weekly and actually rely on without the manual fatigue.

About Me

I am Chidozie Managwu, an award-winning AI Product Architect and founder focused on helping global tech talent build real, production-ready skills. I contribute to global AI initiatives as a GAFAI Delegate and lead the AI Titans Network, a community for developers learning how to ship AI products.

My work has been recognised with the Global Tech Hero award and featured on platforms like HackerNoon.

Attack surfaces are broad, technology stacks are complex, and adversaries are quick to exploit weak points.

Against this backdrop, companies must decide how best to test their defences.

Two main approaches have emerged as leaders: human-led penetration testing services and automated testing platforms. Each has strengths and limitations. Choosing the right one depends on your security goals, risk tolerance, and budget.

At its core, penetration testing is about finding security holes before attackers do. But how you get there matters.

Human experts bring creativity and real-world insight, while automated platforms offer scale and speed.

This article explores both approaches and compares top providers to help you decide what’s better for your organization in 2026.

What we'll cover:

1. What Are Penetration Testing Services?

2. What Are Automated Penetration Testing Platforms?

3. Why the Debate Matters in 2026

   • Depth of Testing: Humans vs Machines

   • Speed and Frequency of Testing

   • Cost Considerations

   • Integration with Security Workflows

4. Real World Context: Top Providers in 2026

5. Compliance and Reporting

6. Which One Should You Choose in 2026?

7. Final Thoughts

What Are Penetration Testing Services?

Penetration testing services are engagements where cybersecurity professionals actively probe your systems to find vulnerabilities. These experts use a mix of tools, manual techniques, and real-world attack simulations to surface weaknesses that machines might miss.

These services may include scheduled tests, one-time assessments, and ongoing engagements. Many providers tailor their approach to the environment being tested, whether that’s a corporate network, web application, cloud infrastructure, or mobile ecosystem.

Human testers think like attackers, combining automated scans with logic and adaptability that machines cannot replicate on their own.

These engagements are typically measured in reports, debrief sessions, and clear remediation guidance. The human element is the defining factor. A skilled tester doesn’t just find flaws. They understand context, creative exploit paths, and business impact.

What Are Automated Penetration Testing Platforms?

Automated penetration testing platforms use software to scan, crawl, and test systems for vulnerabilities. These platforms run scheduled scans or continuous assessments with minimal human intervention. They aim to find flaws early and often, integrating with development pipelines or security operations centers.

Automation brings consistency, speed, and the ability to repeat tests frequently. Many modern platforms use machine learning to prioritize findings and reduce noise. Some offer automation rules that trigger scans based on changes in the environment or codebase.

In contrast to full manual services, platforms are best suited for ongoing baseline assessments and rapid feedback. They are often priced in subscription models and integrate with other tooling like bug tracking systems or SIEMs. While they can pinpoint known vulnerability patterns efficiently, automated tools are limited in creative attack paths and logic-based exploits.

Why the Debate Matters in 2026

In 2026, the cybersecurity landscape is both more advanced and more hazardous. Organizations operate hybrid clouds, microservices architectures, and complex supply chains.

Threat actors are using AI to scale attacks. In this environment, the question is not only about finding old vulnerabilities but anticipating novel attack methods.

With limited resources, security leaders must choose wisely. Do you invest heavily in services with human experts? Do you adopt automated platforms that test continuously?

Maybe a mix is best. To answer these questions, let’s explore how the two approaches compare across key criteria.

Depth of Testing: Humans vs Machines

Human-led penetration tests shine when deep context and logic are required. Expert testers can chain together multiple issues to compromise a system in ways automated tools don't anticipate. They explore paths, think creatively, and adapt in real time to the environment they encounter.

Automated platforms excel at breadth and repetition. They perform wide sweeps of systems quickly and can generate alerts on common vulnerability classes. They're particularly strong in repetitive tasks like scanning hundreds of endpoints or validating compliance controls.

But platforms often rely on predefined signatures and patterns. They perform poorly when an exploit requires intuition or lateral thinking.

In simple terms, human services dig deep while platforms dig wide.

Speed and Frequency of Testing

Automated platforms have a clear advantage in speed and frequency. They can run multiple scans in parallel, test after every code commit, and provide almost immediate feedback. This makes them ideal for DevOps pipelines and agile environments that change daily.

Penetration testing services, by design, occur on a schedule. A quarterly or annual test may be thorough, but it cannot match the cadence that automated tools provide.

Manual tests take time to plan, execute, and analyze. In fast-moving environments, this might leave gaps between testing windows.

For many organizations, automation fills these gaps, while manual testing provides periodic, deep insight.

Cost Considerations

Cost is always a factor. Automated platforms generally come with lower upfront costs compared to human-led engagements. Subscriptions scale with usage and provide continuous assessment for a predictable price. This makes them appealing to midsize companies or teams with limited budgets.

Penetration testing services, especially from reputable consultancies, command higher fees. These reflect labor costs, expertise, and the bespoke nature of the work.

However, the value gained is often more than just flaw detection: it’s expert interpretation, custom exploitation paths, and strategic guidance.

In cost-benefit terms, automated platforms provide the most value per dollar for baseline security, while services deliver high-value insight that can justify a higher cost.

Integration with Security Workflows

Automated platforms are built to integrate with broader security tooling. They often connect to continuous integration/continuous delivery (CI/CD) pipelines, vulnerability management platforms, and ticketing systems. This integration ensures that issues are communicated to the teams who need them most and tracked to resolution.

Penetration testing services can integrate into workflows too, but this usually requires additional coordination. Reports must be ingested into tracking systems and aligned with internal priorities. Some providers offer APIs and extended services that help bridge this gap, but the process typically takes more effort than with automated platforms.

Integration matters because security cannot operate in isolation. Automated platforms fit more naturally into modern DevSecOps workflows, while services provide episodic insights that must be planned and bridged into operations.

Real World Context: Top Providers in 2026

To illustrate how these approaches manifest in practice, consider a few leading options. Each provider offers different strengths in manual services or automated tooling.

One such provider is XBOW. XBOW is known for deep manual testing engagements, combining expert human testers with structured methodologies across network, application, and cloud environments. Their work emphasizes real-world attack simulations and strategic risk reporting.

Another well-known provider is Cobalt. Cobalt blends human expertise with platform-based management. Their Pentest as a Service (PtaaS) model connects testers to client environments through a platform that organizes findings, workflows, and communication. Clients can collaborate with testers, track issues in real time, and integrate results with other systems.

A different model comes from Synack. Synack uses a crowd of vetted testers who work with a secure testing platform. This hybrid model aims to combine the creativity of human testers with the scalability and tracking of automated systems. Clients benefit from diverse testing styles and coordinated reporting within a structured platform.

Each of these approaches has merit. Some lean more toward pure services, others toward platform-driven collaboration. Your choice should align with your security maturity and goals.

Compliance and Reporting

For regulated industries, compliance matters. Automated platforms often include reporting features that map directly to standards like PCI DSS, HIPAA, or ISO 27001. These reports can be generated on a regular cadence and integrated into audit evidence.

Penetration testing services provide compliance support too, but the reports are typically narrative and bespoke. The real value is in expert interpretation of compliance requirements and guidance on remediating complex findings.

In essence, automation provides structured, repeatable reporting, while services deliver customized insights that may carry more weight with auditors and internal stakeholders.

Which One Should You Choose in 2026?

There is no one-size-fits-all answer. Many organizations adopt both approaches. Automated platforms serve as the first line of defense by continuously scanning for known issues and tracking progress over time. Human-led services then provide a deeper second layer, uncovering complex issues and offering strategic guidance.

If your environment is highly dynamic, with frequent releases and evolving infrastructure, an automated platform is essential. If you operate in a high-risk sector where attackers are likely to craft bespoke exploits, human-led penetration testing services are indispensable.

Most mature security programs use both. Automation drives frequency and scale. Human services provide depth and insight. Together, they form a layered testing strategy that maximizes coverage and minimizes blind spots.

Final Thoughts

In 2026, cybersecurity testing is more sophisticated and essential than ever. Organizations must balance speed, depth, cost, and context when selecting between penetration testing services and automated platforms. While one is not inherently better than the other in all cases, understanding their differences and complementary strengths will help you build a robust security posture.

Automated platforms catch the routine and repetitive, giving continuous visibility into known risks. Human-led services uncover the hidden and unexpected, thinking beyond patterns to simulate real adversaries. For most teams, the future of testing lies in a hybrid approach that leverages both.

By aligning your security goals with the right mix of services and tools, you can stay ahead of threats now and in the years to come.

Hope you enjoyed this article. Learn more about me by visiting my website.

I, like many others, struggled with the installation process. And working from Linux, the Mac specific orientation added extra pitfalls. It wasn't always clear whether configuration and management should be done in the docs, the CLI, or the interface.

I found the UI unintuitive and it left me wondering if it wasn't just a dev placeholder. The color choice in particular was especially harsh. All the red tricked the eye and made white text appear green. It also made everything seem like an error message.

I couldn't make heads or tails of the organization and structure. Workspaces, agents, and sessions are all terms I'm familiar with and understand. But the way Open Claw implements them made no sense to me.

Open Claw started as a way to connect a chat tool to an AI. I did that eight months ago with n8n. It's literally only a few nodes. It was so easy that I didn't think anything of it. In my opinion, Open Claw isn’t actually all that special. There’s no part of it that stands out as unique, except for the approach. It’s the Flappy Bird of the agentic AI world.

So I set out to make my own. And within a few hours, I'd whipped up a simple working prototype vibe-coded with Python and connected to Open WebUI (OWUI).

But I wanted to see what prompt OWUI was sending the agent, exactly. Now, if I was actually a Python guy, I would have done some console output. But instead, I went for my favorite tool: n8n (a powerful low-code automation system). And that's where things got interesting.

About This Handbook

This handbook will introduce you to agentic AI creation using a hands-on approach and a starter project I created called Decapod.

Decapod is not a self-contained SaaS offering. There is no part of it that is black boxed and unavailable to hack on. Decapod is a collection of docker-compose.yml containers, scripts, AI agent prompts, and n8n workflows that work together to help give you a leg up on your path to building your own agentic AI empire.

Concepts and technologies you'll be introduced to and using:

• Agentic AI with tools and skills

• Docker containers with Docker Compose

• Open WebUI

• n8n

• S3 and MinIO

• Caddy

• Postgres

For a list of required skills, services, and tools, please check out the "Requirements and Processes" section.

Table of Contents

• Decapod - The DIYer's Dream Agent

• How Decapod Works

  • Core Engine

  • Supakitchen - Supabase on a Budget

  • Open WebUI - AI Chat With All the Bells and Whistles

• Requirements and Processes - Tools I Use and Recommend

  • The Checklist

• Assembling the Dream Team - Ikea Style

  • Accessing Your VPS With Cursor and SSH

  • Installing and Configuring the Docker Containers

• Configuration and Wiring

  • Initiate the Database

  • A Little MinIO

  • Adding the Workflows

  • Getting Started With n8n

  • Now, Get OWUI to Talk to Decapod

  • There Was Supposed to Be an Earth Shattering Kaboom

• The Ever-Present "Hello World"

• Into the Future!

  • A Work in Progress

  • Adding Your Own Skills - Limitless Potential

  • Future Plans

• Got Questions? Meet Captain Finn!

Decapod – The DIYer's Dream Agent

I'll be honest. I'd never even considered the security issues with Open Claw at first. But they're enormous! Let's open a giant hole in our server and give a fledgling alien intelligence root access and all of our API keys. What could possibly go wrong?

Decapod isn't a monolithic app. It's a collection of tools and n8n workflows that give you complete control over your agent and its tools. It's a framework to give citizen developers a leg up.

By switching to n8n, I accidentally solved a ton of issues and made a far superior (in my opinion) project:

• Double (or triple if you choose to host in a VPS) sandboxed security. My agent lives inside of n8n inside of a Docker container inside of a VPS.

• The agent never sees a single API key or even ever needs to know exactly how you're connecting services. Credentials are handled by n8n.

• Universal access – I prefer OWUI. But literally anything that can connect to a standard OpenAI API endpoint can connect to Decapod.

• Over 1,000 integrations – What n8n does best is connecting any API to any other API via drag-and-drop nodes. And there are more than 1,000 of them.

• No more sketchy skills – Decapod uses skills, but they have to actually be connected to n8n workflows and nodes to work.

More problems Decapod solves:

• Fewer tokens burned – Decapod maintains a clean boundary between what's best handled with code/logic and what's best handled by AI.

• No endless loops and hung jobs – Decapod uses a jobs and tasks system that the AI can manage. So if it sees that a task has failed, it can change tasks or suspend the job.

• HITL (Human In The Loop) – You can add a HITL sub-workflow before any AI skill to give them permission to proceed or not.

• An MVP you can trust – The core Decapod system is just an MVP. But it's built on exclusively mature, open source, enterprise ready solutions: n8n, Open WebUI, Docker, Caddy, Postgres, and MinIO.

How Decapod Works

Decapod is middleware that acts like an OpenAI API. But it intercepts the API call and does agent work with the real API.

The OpenAI API standard is the most widely used in the industry. Almost every tool, like Open WebUI, Zed, and Obsidian have ways to connect to the OpenAI standard. So those tools can also connect to Decapod.

Decapod itself can connect to any API and pass available models through to other tools. I strongly prefer and recommend OpenRouter. OpenRouter also uses the OpenAI standard, but lets you connect to hundreds of mainstream and indie models under the same pricing system. Decapod is configured to work with OR out of the box.

This is an image of the Decapod agent tool router – one of the key n8n workflows in Decapod.

Core Engine

Decapod consists of an agent with tools and skills. By tools, I mean the agentic tools that an AI can access to perform tasks as part of the API. And by skills, I'm referring to Anthropic's Agent Skills standard. It's the same skills standard used by Open Claw.

The Decapod agent has a limited, immutable set of tools for managing Decapod's state and job queue. One tool is used to call skills. Skills are dynamic and you can add as many as you like mid-flight.

Each skill consists of core instructions, followed by JSON specs. The agent builds a skill request based on the JSON and calls the use_skill tool to have it executed. Then Decapod calls a sub-workflow with a name that matches the skill and sends it the JSON.

One skill = one sub-workflow. JSON specs = sub-workflow's expected input.

When Decapod receives a user message, it passes it to the agent. If it's just a message, the agent responds. If it's a call to action, the agent picks a tool and gets to work.

Decapod loops through each job in the queue, handling the agent's tool calls and passing it back the results. When the agent is done, it concludes the job and stops sending tool calls. The final message is passed back to the user.

Supakitchen – Supabase on a Budget

I'm a huge fan of Supabase. It's all the fun of Firebase, except with data normalization. But I'm self-hosting Decapod because paying $20 per month for each of five or more services doesn't sit right with me.

As a mad scientist, I like to be able to try different tools without dealing with the freemium hoops. So I'm running Decapod on a Hetzner VPS with 8 gigs of RAM for about $18 per month. Those 8 gigs go really far in the self-hosted world, but Supabase is heavy.

What I really wanted was to give my agent file access and a database. I accomplished that with MinIO and Postgres. No real-time data, but my agent is async anyway. And agent authentication is done through n8n. So it's good enough.

But you do you! Decapod can work with any S3 compatible file store and any Postgres database. So if you want to use Supabase instead, go for it!

Open WebUI – AI Chat With All the Bells and Whistles

You can use chat tools, like Discord, Telegram, Slack, and others, to chat with your AI easily enough. But if you want multiple sessions or to use different models, it can be tricky.

The easiest tool to set up and work with, by far, is Telegram. You get chat, UI elements, and even embedded apps without having to host your own server, like you do with Discord. I once used it to create a HITL lead qualification tool in a few hours.

BUT! While Telegram and friends do get the job done, if you want a new session you have to create a new bot for each and every one. If you want to switch models, you need to add /slash commands. If you want context management, you have to handle that server side.

That's why I prefer Open WebUI. OWUI gives you everything you expect from all of the best mainstream AI offerings, but with a direct tap to the API.

• It works great on browser and mobile as a progressive web app (PWA).

• You can mod it with Python.

• It has many ways to manage and supply context, including nested projects/folders and RAG support.

• You can collaboratively work on notes with AI.

Those are a few of my favorite features, but there are so many more. Why reinvent the wheel when the absolute best solution already exists?

Requirements and Processes – Tools I Use and Recommend

Welcome to my lab-or-a-tory. We're out there on the fringes of agentic AI now. Doing weird experiments by stitching together pieces and parts. Let me show you how I work and tell you where you can and can't stray from my process.

Decapod is a finished MVP and should work right out of the box with minimal headache. But it doesn't have more than a few skills yet. So you'll need to build your own until it takes off. Fortunately, your Decapod agent can help.

The Checklist

Skills:

• ✅ A generalist's mindset, problem-solving skills, and a sense of adventure.

  • You don't have to be an expert at anything to install Decapod. I'm not, and I built it.

  • But you do have to be comfortable with many different technologies.

• ✅ The command line, Docker, and probably Node. Decapod is self hosted. So you'll need to get your hands a bit dirty.

• ✅ The ability to read and write a little JavaScript. This helps a lot with n8n code nodes to give it more utility.

• ✅ Familiarity with JSON and APIs. Everything in n8n is about passing JSON from node to node. And n8n is nothing if not a universal API connector.

Services:

• ✅ A domain name with DNS access.

  • This is critical for n8n to work properly due to CORS and security issues.

  • Also, the OWUI PWA doesn't work when hosted through an IP. It's just a web page at that point.

  • Plus, it's just better for security overall with https support.

  • If cost is an issue, you can get an all-digit domain name from gen.xyz for $0.99. Seems legit, but I haven't tried it myself.

• ✅ A dedicated VPS with SSH access. (SSH access should be standard for any VPS.)

  • You can technically host this on your own PC if you know it will be running 24/7. But using a VPS will give you peace of mind and avoid complicating your PC.

  • Big-name solutions like AWS and Google Cloud can wind up going off the rails and costing you big bucks if you don't know exactly what you're doing. Better to stick with less enterprise-oriented offerings. I've used the following:

    • Hetzner – My current personal favorite. Germany based. High quality and affordable pricing with a few American servers. Even more affordable with European servers.

    • Digital Ocean – US based. Can't go wrong. Decent prices. Many offerings. Almost exclusively American servers.

    • Webdock – Denmark based. The most affordable of the bunch.

• ✅ An OpenRouter account. OR provides a universal interface for hundreds of AI models. There's no freemium upsell, like with Hugging Face, but there is a percentage add on when you buy credits/tokens. I feel like it's worth the extra fee to be able to easily swap from Claude to Kimi to GPT to DeepSeek as I please without more keys, more accounts, and more wiring. But this is optional. You can plug Decapod right into Kimi or Gemini and just leave it there if you like.

Tools:

• ✅ Cursor, or similar. I love Cursor. It matches my hands-on style. If you're freestyling and dreaming something into creation as you build it, AI will always take the wrong path if you take your hands off the wheel. Cursor lets me be in charge and play director while the AI does the heavy lifting and saves me from hours of Googling and digging through 10-year-old questions on Stack Overflow. Especially with the command line stuff. I could not have knocked out Decapod in two weeks without it. But it couldn't have built Decapod at all without me.

• ✅ Another AI bestie to help you dream, plot, and plan. Cursor is great, but very utilitarian. I always have a session open with a running commentary about my work. I'm constantly feeding it context and leaning on it to get a fresh perspective and solve more esoteric issues, like debugging n8n flow problems, for example. I use Claude for absolutely everything. It has the most natural conversational flow, it's good at taking meta instructions regarding its behavior, and it always has an eye on accuracy – very reliable.

Assembling the Dream Team – Ikea Style

Here are the pieces and parts you'll find in your Dekkaplonkën Ikea flat pack (the GitHub repo).

1. Four Docker containers containing five services with docker-compose files. Just heat and serve.

• Infrastructure: Caddy for routing and SSL certificates for https security.

• Infrastructure: Postgres for all your data needs.

• MinIO: An S3 compatible file storage system.

• n8n: The ultimate automation tool.

• Open WebUI: The ultimate AI chat interface.

• SQL tables

  • A table for the decapod state.

  • A table for jobs, tasks, and tool chat history.

• S3 Files and Folders – Agent Templates

  • Four starter skills (two actually implemented in n8n).

  • Two instructional files, including the persona and skill definitions.

• n8n Workflows (6,889 lines of pure JSON)

  • API Middleware: The entry and exit point that manages the session and loops.

  • AI Tool Router: Executes your agent's tool requests.

  • Construct Message History: Injects instructions into your agent's chat history.

  • Get Job Queue: A one-off database call that gets active jobs ordered by priority and creation date (First In First Out).

  • Utility Workbench: A place for testing and managing your flows. Currently contains a Skill assembly jig.

  • Worker: Loops over job queues, talking to the agent and calling the tool router with its responses.

  • A write-file skill and a research-recipes skill.

  • A couple more placeholders. (Decapod is an MVP)

• Also

  • A Docker cheatsheet.

  • A script to generate agents from the template.

  • A destructive script to upload local agent files to your S3 account by overwriting existing files. Good for dev. Bad if you let your agent start modding their own instructions.

  • Scripts to start and stop all Docker containers at once.

Accessing Your VPS With Cursor and SSH

SSH is the standard way to access any server and has been forever. But working through a terminal can be slow and plodding. Fortunately, there's a better way.

Connect to the server with Cursor, VS Code, Antigravity, or whatever you use. This gives you:

• Multiple terminals to access the remote server.

• The ability to view localhost servers as if they were on your own machine via port forwarding.

• Drag and drop folder and file management.

• No more Nano, Vim, or Emacs (unless you want to).

• And the best part! Cursor can do all the remote file system work for you, including troubleshooting servers and containers, writing scripts for automating common tasks, and helping you hash out actionable plans.

• (Cursor can also connect to your Decapod!)

Every VPS provider will have their own way of managing SSH access. They usually make adding them part of the sign up process.

Generating and managing keys is a pretty well-paved path and I won't go over it. It's a good job for Cursor, if you need help.

However! I use Bitwarden for SSH key generation and management. They still need to be stored locally for tools on your computer to access. But it's nice to have them in a single secure location.

VS Code requires an extra plugin to access a remote server. Cursor comes with it preinstalled. Just click Connect via SSH, set up your connection, and you're good to go.

📝 Side note: I was on the paid plan when I started, I swear. I tend to switch services a lot as new models are released and I discover different tools and options. But I only ever pay for 2 or 3 at a time.

I got about halfway through this article when Cursor expired. But I'm trying the new Gemini 3 models and switched to Antigravity mid-flight rather than re-up cursor.

Installing and Configuring the Docker Containers

Finally! After a novella's worth of lead-up, we, at long last, get to the actual installation. That will be shared in the next article – have a good night! Just kidding, please put down the brick.

Once you've SSHed in to a VPS, a Raspberry Pi with Ubuntu, or a Virtual Machine, you're ready to get started. I'm going to assume you know how to install tools like Docker and Node on your system and not go into a lot of detail. Ask your friendly neighborhood AI for help if you get stuck.

💡 Important! If you haven't already, get your domain name and open up the DNS page. You'll want to redirect "A" records to your IP for each relevant service.

Start by cloning the Decapod repo.

git clone https://github.com/leetheguy/decapod.git

cd decapod and create your Docker network.

docker network create web

Now we're going to go into each of the four Docker folders, configure them, and fire them up, starting with infrastructure.

cd infrastructure cp .env.example .env

Alternatively, you can move the files to rename them or just click on the file in the UI and F2 to rename it. Whatever floats your goat 🐐.

Now edit the new .env file. You can get the data folder path by clicking on the infrastructure folder and Ctrl/Cmd+Alt+C. The rest is up to you. I used Bitwarden to generate a password here.

Next, copy the Caddyfile template into its own file.

cp caddy_config/Caddyfile.template caddy_config/Caddyfile

And start the Docker container with docker compose up -d.

Back out of infrastructure and into minio. Same again with the .env – copy and configure. Make sure the URLs match your domain.

Once more for n8n and then again for openwebui.

OWUI config comes from the infrastructure and minio .env files:

• S3_ACCESS_KEY_ID=minio_admin

• S3_SECRET_ACCESS_KEY=minio_password

• S3_BUCKET_NAME=decapod

• MINIO_ROOT_USER=minio_admin

• MINIO_ROOT_PASSWORD=minio_password

• POSTGRES_DB=postgres

• POSTGRES_USER=postgres

• POSTGRES_PASSWORD=postgres_password

📝 Note! OWUI may take a moment or two to start. Go grab some water and it should be up by the time you get back.

Configuration and Wiring

Roll up your sleeves! This is where we get up to our elbows in pieces and parts.

If everything went to plan, you should now have all five services up and running. You can confirm the containers are live with docker ps. You can check that they're actually properly connected by visiting s3, OWUI, and n8n.your-domain.com.

Create accounts for all three and sign in to each.

⚡️ Important! Get your n8n license key! It's free and gives you access to all community features. You'll be severely limited without it. Activate it under Usage and plan in the settings.

Initiate the Database

Decapod only needs two data tables. You can add them from the command line. But I like pgAdmin.

Connect to your Postgres database in the usual way. But you'll need your server's IP for the host name instead of postgres (which you use to connect services inside of the Docker network) since pgAdmin isn't in your Docker network.

You'll find your SQL files in components/pgsql_tables. Create a decapod database and add both of the SQL files to it. A default decapod_state table record will be automatically generated when running the SQL.

In pgAdmin:

• Open the decapod server.

• Create a decapod database by right-clicking on databases.

• Select the new database.

• Click the query tool button at the top of the explorer.

• Copy and paste the decapod_state table into the query and run it with F5.

• Clear the query, paste in job_queue, run it.

Or ask Cursor or an AI bestie for help if you want to go pure command line.

A Little MinIO

Next up, you'll be adding your agent's instructions and persona files to your private S3 service. Start by visiting your MinIO server and adding a decapod bucket.

In components/S3_structure/agents/, you'll find a template for your agents. (I have the intention of making Decapod a multi-agent tool in a future release.) The template is meant to be copied to a new agent of your choice. But if you choose something other than Decapod, you'll need to update the state table.

You can do it manually if you wish. Copy the folder to match the new agent's name and update the definitions/skills.yaml file to include all the skills you want your agent to have. The name and description should exactly match what's found at the top of each skill file.

Alternatively, I vibe coded a script to make it a little easier. It's in the scripts folder and you'll need to install the inquirer Node module to use it. Run cd scripts and create-agent.mjs to use it.

You also need to make sure that the files and folder structure in your MinIO match those in S3_structure. Start by creating a bucket called decapod in your drive. Then upload the files from S3_structure into your bucket.

But that's easier said than done because they're on a remote server. And if you used the visual interface, you'd have to download them to your local machine first. So I made another script – upload_S3_structure.sh.

That script is strictly meant for dev purposes. It's absolute and destructive. Just a heavy mallet. So if you want to surgically alter your MinIO, do not use it! Remember kids: mallets and brain surgery don't mix.

Once your agent files are in place, you can let your agents edit them, Open Claw style, or you can edit them yourself. But MinIO doesn't give you much of anything in the way of features for their UI.

For a better experience, I'd recommend S3Drive. When you go to sign up, look for the connect button towards the bottom to connect to your own MinIO endpoint.

S3Drive will let you edit your files in place after you've uploaded them. This is good for quick fixes or copying and pasting sections without a complete wipe.

Adding the Workflows

You'll find most of what makes Decapod Decapod in the components folder. And the heart of that is in n8n_workflows.

You can manually import those workflows one at a time and go over each one to make sure they're safe and sound. Or you can use the n8n CLI inside of the Docker container and save yourself some tedium.

These commands move the workflows to the Docker container, import them with the n8n CLI, and then remove them from the tmp directory.

docker cp ./components/n8n_workflows n8n:/tmp/workflows

docker exec -u node n8n n8n import:workflow --input=/tmp/workflows --separate
docker exec -u node n8n n8n import:workflow --input=/tmp/workflows/skills --separate

docker exec -u root n8n rm -rf /tmp/workflows

Now, you should see the 10 workflows in n8n. I'd recommend drag-and-dropping the main workflows to a dedicated decapod folder and the two skills to decapod/skills, just to keep things tidy. But they reference each other by id, so do what you want.

Getting Started With n8n

Now would be a good time to start exploring the workflows in your n8n UI Personal tab. If you sort them by name, the main file will be on top. Crack it open and see it's not too intense, and it's self-documented. Blue for notes, Green for sub-workflows, and Red for nodes that require your credentials.

I'd recommend reading the notes and thoroughly exploring the sub-workflows to help you understand Decapod. It's your tool now! Create credentials as you go.

Because we're using a Docker network, creating credentials and connecting your services to each other couldn't be easier.

The standard to connect all of your services is to reference them by name:port. Because the Postgres credential has its own port field, you can just set it to Postgres. Port should be 5432.

📝 Note! All credential details, like your container names, ports, and passwords, can be found in your docker-compose and .env files.

For MinIO:

• Endpoint: http://minio:9000

• Force Path Style: Enabled! Important for MinIO.

API Connections to OpenRouter:

• choose: Authentication -> Predefined Credential Type

• then: Credential Type -> OpenRouter

• Now just paste your API key from OpenRouter.

n8n – (meta access to your workflow):

• In a new tab, go to n8n Settings -> n8n API.

• Turn off expiration if you like.

• Copy your key.

• Paste it in the field.

• Base URL: http://n8n:5678/api/v1

Once you've created credentials, you can reuse them for every relevant node that uses the same credential. Just select it from the dropdown.

💡 Tip! It may help to remove the red sticky notes as you add credentials. And don't forget the skills! I didn't sticky note them at all.

As a final step, make sure your n8n workflows are published in the following order:

• construct message history

• get job queue

• hitl yes/no

• tool router

• worker

• middleware

• and the two skills

💡 Tip! Always make sure your n8n workflows are in a published state with a green dot before calling them. Otherwise, you'll be calling an outdated version.

Now, Get OWUI to Talk to Decapod

OWUI is built for teams, so you have admin settings and personal settings. You'll want to edit the admin settings by clicking on the profile circle in the lower-left-hand corner, then Admin Panel -> Settings -> Connections.

From there:

• Ollama API Disabled: Just keeping things tidy.

• Configure the OpenAI link by clicking on the gear and delete that too.

• Direct Connections: Enabled

• Cache Base Model List: Enabled Now add your Decapod connector with the plus button.

• URL: http://n8n:5678/webhook/v1/decapod (Click the cycle icon to confirm your connection.)

• Auth: none (it's all in the same Docker network, so it's fine for now. You can add a password for production.)

• Prefix ID: decapod (If you do decide to use OpenAI, Hugging Face, or whatever else, this will help distinguish the model hosts.)

That's it. Save and go to the Models tab. Decapod passes OpenRouter models straight through. So if you see hundreds of models, take a victory lap! That means that Decapod is working, live, accepting requests, and you've even properly done your certifications (at least for OpenRouter).

Now create a new chat session and pick a model. I like Claude Haiku 4.5. Fast, cheap, and good. Pick three. I did all of my Decapod dev with it in the saddle, so I know it works. And 3.5 million tokens towards testing iterations cost me \(4, so I know it's reasonable. Alternatively, Kimi K2.5 will likely work and would be even a little bit cheaper. I burned through 4.7 million tokens installing a Docker container in Open Claw with Kimi for about \)3.

Time to say hello to your little friend! Haiku is fast. So if it takes more than a few seconds to respond, something could be borked in your n8n flow. It happened to me as I was writing this article. I had some issues with both Postgres and MinIO.

💡 Tip: If the agent does get hung, it's easier to resend the message than stop and try again.

There Was Supposed to Be an Earth Shattering Kaboom

So, your agent really wants to talk to you, but all you have is a pulsating dot. It's likely that something got misconfigured in n8n.

You can debug n8n by going to the middleware workflow and selecting executions from the top tab bar. If there's an error on the left list, look for a message in the lower right.

This was when I had some database config issues and it couldn't find the state table.

Some sub-workflows may fail quietly. You can trace flow from the webhook entry point to the error. All successful nodes will light up green. The bad node will be red. Drill down, check executions, and repeat for each sub-workflow.

When you find the culprit – the actual bad node in the bad execution – select "copy to editor" in the upper-right-hand corner. That will freeze the workflow to that state. Open the node, fix the credential or whatever, and click Execute Step to see if it's fixed.

Remember: after every change, always always always publish your update. Otherwise, n8n won't actually use the latest fixed version of your workflow.

Once you've successfully debugged your Decapod, make sure that you clean out the loose unfinished jobs in the job_queue table with pgAdmin or whatever. Otherwise, your agent will try to complete each of them before finishing the next job.

The Ever-Present "Hello World"

OK! Now for the moment of truth. You got your agent to say hello back. That was the easy part because it didn't need to do any work or use any tools.

I set you up with two skills to put it to the test: write-file and research-recipes. The recipes skill connects your bot to a free recipe API (no key needed). It's not just pulling recipes out of training data.

Try this prompt: Would you please look up pizza recipes and save them to a file?

If all of your credentials are properly configured, you should get what you asked for. Open up MinIO or S3Drive and look in /agents/decapod/documents for the file.

Into the Future!

I know that was a lot! (At least it felt like a lot from my end.) I hope it wasn't too painful. And look at the bright side: you just got a crash course on some really powerful technology. And if you made it through, that's a major accomplishment! The hard part is behind you. Now comes the fun.

A Work in Progress

I'll be honest. I just wanted to get Decapod out fast to prove how doable a personal agent is while Open Claw is still hot. Anyone can build their own Agentic AI with little or no code. And you don't have to settle for painful UI and poor security. You can have it all.

But, as I've said, Decapod is still an MVP. Complete and functional, but feature light. And I was stressing about that a little bit. I wanted multiple agents and more skills for the early adopters.

Then it hit me. Duh! You already have everything you need with n8n.

You can add an n8n agent node, connect it to a model and an MCP server, and have a sub-agent ready to go in minutes. Then have your agent produce a skill sheet to contact the sub-agent.

Adding Your Own Skills – Limitless Potential

Let's create a dead simple n8n agent to search the web. Then we'll add that to Decapod as a new skill.

In this image I used the prompt:

Thank you so much! Next up, I want to give you web search access via a sub-agent. So your web search skill wouldn't directly search the web, but would instead call a simple agent to do the search for you.

Would you please create a web-search.md skill for your future self to use? The only required field should be prompt.

The agent's file folder is sandboxed by default, so the agent's skills/web-search.md is actually in the agent's private documents storage. I moved it to the actual skills folder and updated my agent's skills.yaml file with the new skill.

Now I'll create a new n8n skill workflow in decapod/skills/.

⚡️ Important! Your n8n skill workflow name must match the skill name exactly. So, web-search.md would be a workflow called web-search. Decapod uses the name to look for the skill so it can be hot loaded without a secondary router.

The n8n screenshot above was pretty much exactly the whole thing. Try rebuilding it yourself. I used chat input to make sure it was working with n8n's chat interface. And I used the Exa Web Search MCP as the search tool. I used Haiku as the model, but an even simpler model would have likely been just fine. OpenRouter has a number of free models with tool abilities that would probably do the trick.

Once you have the workflow operating properly, replace the chat node with a "When Executed by Another Workflow" node with a parameters object as input.

Next, open up the utility/workbench workflow. This tool will help you turn your web-search workflow into a skill. Work through each node in order, testing the node with "Execute step" button as you go. Doing so will create output data that the next node can use as input data.

1. get workflow id from name: Set name to "web-search".

2. deliver JSON arguments to skill: Set parameters object to { "prompt": "Can I please get a list of a variety of pizza recipes complete with links to their sources?" }; (or whatever matches your skill sheet)

3. call skill based on workflow id: Should be ready to execute.

If your output looks like that, your skill should be ready to go.

In this image I used the prompt: Alright! I think you're all set. Try doing a search for dessert pizza recipes.

If your agent gives you the following error, make sure that it knows it MUST create a job before it can call the use_skill tool. It should know that from the instructions, but pobody's nerfect. (I'll need to tighten that up.)

Hopefully that was also pretty painless and now your mind is exploding with possibilities like mine is. If you're unconcerned with safety or actively want to invoke Skynet, you can even give your agent a skill to create its own n8n skills with the Create a workflow node. But don't do that.

Future Plans

Here are a few more features I'd like to add:

• /slash commands – You shouldn't have to go into n8n or pgAdmin to see what your agent is doing and manage its job queue.

• Streaming responses – I'd like to see what my agent is doing as it's doing it, but streaming is a bit tricky and was beyond the MVP.

• Multiple states – With multiple states, you can run multiple agents simultaneously. Or you can have different agents/models for different sessions. For example, you can have a health and fitness session with one agent with its own context window, job queue, and skill set. And you can have another one to help you keep track of your coding education.

• It's a bug, not a feature – There are many places where the state and model are hard-coded throughout the app. I also started working on features that didn't pan out and left some dangling nodes. I'd like to clean up the app and actually implement those features.

If you've read this far and are totally all in, I'd love to hear feedback and suggestions for more features. I'd be fascinated to hear about how Decapod is being used. And I'm also happy to answer any questions.

Got Questions? Meet Captain Finn!

Decapod is the culmination of a year spent studying and learning all things AI and automation. It's also the result of 20 years in the world of coding and app development.

I'm currently starting a community for AI Enthusiasts, Automation Inventors, and Systems Thinkers. It will be led by Captain Finn, a retro-futuristic space captain who got stranded without his crew in our time and space. He used AI, automation, and systems thinking to keep the ship working, give himself someone to talk to, and to wake up to the smell of fresh coffee every morning.

And yes, Finn himself is an AI persona, operating from AI-automated systems, like Decapod, that he will be teaching people about.

My goal is to create a welcoming environment for my fellow mad scientists, dreamers, and citizen developers to learn and grow with help from the community and Captain Finn Feldspar himself. I plan to release weekly articles, more tutorials like this, and other tips and tricks.

Whether you want help with Decapod, learning automation, or just want to geek out about the power and future of AI — Captain Finn's Fleet has a place for you. Join here for free.

In this article, you’ll build a résumé screening system using pure Python, focusing on core programming concepts and the power of multiprocessing. You’ll create a custom system that automates the evaluation process by transforming unstructured résumé documents into a ranked leaderboard.

By the end of this guide, you will:

• Parse documents by extracting text from PDF and DOCX résumés using Python

• Extract information by identifying skills and keywords from résumé content

• Design a scoring algorithm using weighted logic to rank candidates objectively

• Build a web interface using Streamlit

• Deploy the application on Streamlit Cloud for public access

By following this tutorial, you’ll build a tool capable of processing hundreds of résumés in seconds.

Here’s the source code: GitHub Repository

Table of Contents

• Prerequisites

• Project Overview

• How the System Works

• System Architecture

• Project Structure

• Step 1: Set Up the Project

• Step 2: Build the Résumé Parser

• Step 3: Build the Keyword Extractor

• Step 4: Implement the Scoring Engine

• Step 5: Build the Web Interface

• Step 6: Test the System

• Step 7: Deploy the Application

• Conclusion

Prerequisites

To follow along with this tutorial, you should have:

• Basic knowledge of Python (functions, loops, dictionaries)

• Python 3.8 or higher installed

• Familiarity with installing packages using pip

• A code editor such as VS Code, PyCharm, or any editor you prefer

Project Overview

In this guide, you’ll develop a system that takes a folder of résumés and a Job Description (JD) as input. The system processes each résumé, extracts relevant information, and calculates a score based on how well the candidate matches the job requirements.

How the System Works

The project consists of four core components:

• Résumé Parser: Reads PDF and DOCX files and extracts text

• JD Parser: Analyses the job description to identify required skills

• Keyword Extractor: Matches résumé content against a skills taxonomy

• Scoring Engine: Ranks candidates using a weighted algorithm

Scoring Formula

Here’s the scoring formula we’ll use:

Total Score =
(Required Skills × 50%) +
(Preferred Skills × 25%) +
(Experience × 15%) +
(Keywords × 10%)

This approach ensures that essential skills carry more weight than secondary keywords.

How This Approach Helps Reduce Bias

This system evaluates résumés using predefined criteria instead of subjective judgment. Each résumé is scored based on the same set of required skills, preferred skills, experience indicators, and keywords.

Because all candidates are evaluated using the same weighted formula, personal factors such as writing style, formatting, or unconscious preferences don’t influence the ranking. The scoring logic focuses only on how closely a résumé matches the job requirements.

By normalising the evaluation process, the system promotes more consistent and objective screening, which helps reduce bias during the initial résumé review stage.

System Architecture

Input                    Processing                     Output
─────                    ──────────                     ──────

Résumés ──► Résumé Parser ──► Keyword Extractor ──┐
(PDF/DOCX)                                        │
                                                  ├──► Scoring Engine ──► Ranked Results
Job Description ──► JD Parser ────────────────────┘
(TXT/PDF)

The system follows a simple input–process–output flow.

Résumés and the job description are provided as inputs. The Résumé Parser extracts text from each résumé, while the JD Parser identifies required and preferred skills from the job description.

The extracted résumé text is then passed to the Keyword Extractor, which matches skills and keywords using a predefined taxonomy.

Finally, the Scoring Engine applies a weighted formula to calculate a score for each candidate and outputs a ranked list of résumés.

Project Structure

resume_screening_system/
├── app.py                    # Streamlit web interface
├── main.py                   # Command-line interface
├── parsers/
│   ├── resume_parser.py      # PDF/DOCX text extraction
│   └── jd_parser.py          # Job description parsing
├── extractors/
│   └── keyword_extractor.py  # Skills and experience extraction
├── matcher/
│   └── scorer.py             # Scoring algorithm
├── data/
│   ├── config.json           # Scoring weights
│   └── skills_taxonomy.json  # Skills database
└── requirements.txt          # Dependencies

The project is organised into clear, modular directories. Parsing logic, keyword extraction, and scoring are separated into their own folders, while configuration files and data are kept isolated. This structure keeps the codebase easy to navigate, maintain, and extend.

Step 1: Set Up the Project

Create the folder structure and set up a virtual environment:

mkdir resume_screening_system
cd resume_screening_system
mkdir parsers extractors matcher data input output
python -m venv venv

Then go ahead and activate the virtual environment:

# Windows
source venv/Scripts/activate

# macOS / Linux
source venv/bin/activate

Install the required dependencies like this:

pip install PyPDF2 python-docx streamlit pandas

Step 2: Build the Résumé Parser

The résumé parser handles different file formats by using a separate extraction method for each type.

For PDF files, the parser opens the document page by page and extracts text from each page using a PDF reader. The extracted text is combined into a single string for further processing.

For DOCX files, the parser reads each paragraph in the document and joins the paragraph text into one block. This ensures consistent text output regardless of the résumé format.

By combining all résumés into plain text, the parser allows components such as keyword extraction and scoring to work efficiently.

File: parsers/resume_parser.py

def _extract_pdf(self, file_path: Path) -> str:
    text = ""
    with open(file_path, "rb") as file:
        pdf_reader = PyPDF2.PdfReader(file)
        for page in pdf_reader.pages:
            page_text = page.extract_text()
            if page_text:
                text += page_text + "\\n"
    return text.strip()

def _extract_docx(self, file_path: Path) -> str:
    from docx import Document
    doc = Document(file_path)
    return "\\n".join(
        para.text for para in doc.paragraphs
    ).strip()

Step 3: Build the Keyword Extractor

This project uses a résumé dataset from Kaggle to ensure the logic works with real-world professional data. The keyword extractor identifies skills by scanning the résumé text.

The résumé text is first converted to lowercase so that matching is case-insensitive. A predefined skills taxonomy stores each skill along with its possible variations. The extractor checks the résumé text against these variations to find matches.

Word boundaries are used during matching to avoid partial matches, such as matching “Java” inside “JavaScript”. Matched skills are stored in a set to prevent duplicates.

This approach ensures consistent and controlled skill detection across all résumés.

File: extractors/keyword_extractor.py

def extract_skills(self, text: str) -> Set[str]:
    text_lower = text.lower()
    found_skills = set()

    for category, skills_dict in self.skills_taxonomy.items():
        for skill_name, variations in skills_dict.items():
            for variation in variations:
                # Prevent "Java" matching "JavaScript"
                pattern = r"\\b" + re.escape(variation) + r"\\b"
                if re.search(pattern, text_lower):
                    found_skills.add(skill_name)
                    break

    return found_skills

Step 4: Implement the Scoring Engine

To produce objective rankings, the system uses a weighted scoring formula.

Total Score =
(S_req × 0.50) +
(S_pref × 0.25) +
(E_exp × 0.15) +
(K_key × 0.10)

The scoring engine calculates a final score for each résumé using weighted values.

It counts how many required skills, preferred skills, experience indicators, and keywords appear in a résumé. Each count is multiplied by its assigned weight, with required skills contributing the most.

The weighted values are summed to produce a single score. Résumés are then sorted by this score to generate a ranked list of candidates.

Step 5: Build the Web Interface

Streamlit provides a simple web interface for interacting with the résumé screening system.

The text area allows users to input a job description, while the file uploader lets them upload multiple résumé files. When the button is clicked, Streamlit triggers the backend logic to parse résumés, extract data, and calculate scores.

The results are then displayed in the browser, allowing users to run the screening process without using the command line.

File: app.py

import streamlit as st

jd_text = st.text_area(
    "Paste the job description here:",
    height=300
)

uploaded_files = st.file_uploader(
    "Upload resume files:",
    type=["pdf", "docx", "txt"],
    accept_multiple_files=True
)

if st.button("Screen Resumes", type="primary"):
    st.success("Processing resumes...")

Run the application:

streamlit run app.py

The app will be available at http://localhost:8500.

Step 6: Test the System

Sample Job Description Input

Below is an example of a job description you can use to test the system:

We are looking for a Senior Python Developer with strong experience in backend development.

Required Skills:
- Python
- Django
- REST APIs
- SQL

Preferred Skills:
- PostgreSQL
- Docker
- AWS

Experience:
- 3+ years of professional Python development
- Experience building web applications

This input helps the system identify required skills, preferred skills, and experience keywords, which are then used by the scoring engine to rank résumés.

python main.py

Sample Output

============================================================
SCREENING RESULTS
============================================================
Rank #1: Alice Johnson | Score: 85.42/100 | Matched: python, django, postgresql
Rank #2: Carol Davis   | Score: 72.50/100 | Matched: python, django

Step 7: Deploy the Application

To make the system publicly accessible:

1. Push the code to GitHub

2. Go to share.streamlit.io

3. Select your app.py file

4. Deploy the application

Your app will be live at:

<https://your-app-name.streamlit.app>

Conclusion

In this tutorial, you’ve built a complete résumé screening system from scratch using Python. By combining text processing, structured scoring, and automation, this project demonstrates how manual résumé screening can be transformed into an efficient and objective workflow.

This system helps reduce bias, save time, and evaluate candidates more consistently. Happy coding!

But here's the thing: while Kafka itself is incredibly powerful, managing Kafka clusters is notoriously challenging. This isn't a flaw in Kafka – it's just the reality of distributed systems. The bigger your cluster grows, the more complex operations become.

The most painful aspect? Manual cluster management. It's tedious, error-prone, and doesn't scale. What starts as simple topic creation with a few brokers turns into hours of carefully orchestrating partition reassignments across dozens of machines. One typo in a JSON file at 3 AM can take down production.

Sound familiar? You're not alone.

In this guide, you'll learn how two tools can transform Kafka operations from a manual slog into a manageable process:

• Kafka UI – A modern web interface that replaces cryptic CLI commands with visual cluster management

• Cruise Control – LinkedIn's automation engine that handles cluster balancing and self-healing

We'll start by experiencing the pain of manual management firsthand, then see how these tools solve real-world operational challenges. You'll set up everything locally with Docker and by the end you’ll know exactly how to manage Kafka clusters without the headache.

What We’ll Cover:

• Prerequisites

• Setting Up Our Unmanaged Cluster

• Starting the Cluster & Verification

• Creating Topics: The Manual Way

• Kafka UI

  • Setting up Kafka UI

  • Drawbacks of Kafka UI

• Cruise Control

  • How Cruise Control Works

  • Setting Up Cruise Control

  • Cruise Control Configuration File

  • Creating the Imbalance

  • Attempting Manual Rebalancing

  • Rebalancing Using Cruise Control

• Conclusion

The Problem: Manual Kafka Management

Let’s dive right in. First, I'm going to show you what managing a Kafka cluster looks like without any tools – just you, the command line, and dozens of manual operations.

You’ll spin up a small cluster locally, create some topics, and simulate the kind of growth you'd see in a real production environment. By the end of this section, you'll understand exactly why teams spend thousands of engineering hours just keeping Kafka clusters running smoothly.

Fair warning: this is going to feel tedious but it’s ok – that’s the point.

Prerequisites

Before we dive in, make sure you have:

1. Docker Desktop installed and running

   • Mac and Windows users: https://www.docker.com/products/docker-desktop/

   • Linux users can install Docker Engine via their package manager

2. Basic Kafka knowledge. You should understand:

   • Topics: Categories for organizing messages

   • Partitions: How topics are divided for parallelism

   • Brokers: The Kafka servers that store data

   • Producers and Consumers: Applications that write to and read from Kafka

   • KRaft: Kafka consensus based discovery?

If these terms are new to you, here’s a great handbook about them. I’d also recommend reading Kafka's Introduction first.

3. System Requirements

   • At least 8GB Ram

   • 10GB Free Disk space

4. Some basic understanding of containers is good to have:

   • Docker

   • Images

   • Volumes

   • Networks

Setting Up Our Unmanaged Cluster

Let’s go ahead and build the cluster so that we can see the problems firsthand. We’ll use Docker to spin up three Kafka brokers running in KRaft mode (the modern, ZooKeeper-free approach).

Start by creating a file called docker-compose-basic.yml:

version: '3.8'

services:
  kafka-1:
    image: confluentinc/cp-kafka:7.6.0
    container_name: kafka-1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-1-data:/var/lib/kafka/data

  kafka-2:
    image: confluentinc/cp-kafka:7.6.0
    container_name: kafka-2
    ports:
      - "9093:9093"
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:29092,PLAINTEXT_HOST://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-2-data:/var/lib/kafka/data

  kafka-3:
    image: confluentinc/cp-kafka:7.6.0
    container_name: kafka-3
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:29092,PLAINTEXT_HOST://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-3-data:/var/lib/kafka/data

volumes:
  kafka-1-data:
  kafka-2-data:
  kafka-3-data:

In the above configuration file, we’re creating three Kafka brokers (kafka-1, kafka-2, kafka-3). Each one uses the confluentinc/cp-kafka:7.6.0 image and has its port opened (9092, 9093, 9094).

The environment variables are:

• KAFKA_NODE_ID – A unique identifier for each broker (1,2,3). No two brokers can have the same ID.

• KAFKA_PROCESS_ROLES: broker, controller – This tells Kafka to run in KRaft mode (without ZooKeeper). Each broker acts as both a data broker and a controller for cluster coordination.

• KAFKA_CONTROLLER_QUORUM_VOTERS – The membership list that tells each broker how to find the others. All three brokers must have the identical list: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093. This is how they discover each other and elect a leader.

• CLUSTER_ID – A unique identifier for the entire cluster. All brokers must use the exact same value or they won't recognize each other as part of the same cluster. The actual value (MkU3OEVBNTcwNTJENDM2Qk) doesn't matter as long as long as it is consistent across brokers. One important thing to note is that CLUSTER_ID must be a valid base64-encoded UUID per Kafka’s requirement.

• KAFKA_LISTENERS - Defines which network interfaces and ports Kafka listens on. We have three listeners:

  • PLAINTEXT://0.0.0.0:29092: For inter-broker communication (brokers talking to each other)

  • CONTROLLER://0.0.0.0:29093: For controller communication in KRaft mode

  • PLAINTEXT_HOST://0.0.0.0:9092 (varies per broker): For external connections from your machine

• KAFKA_ADVERTISED_LISTENERS – Tells clients (producers/consumers) how to connect to this broker. This is what gets returned when a client asks "where should I connect?" The PLAINTEXT_HOST://localhost:9092 part is what allows you to connect from your Mac.

Note: Listener configuration is critical. Incorrect settings will prevent clients from connecting even when brokers are running. These settings work for local Docker environments where Docker's internal DNS resolves broker names (kafka-1, kafka-2, kafka-3). For production, replace hostnames with actual IP addresses or FQDNs - (Fully Qualified Domain Name):

• KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2 – How many copies of consumer offset data to keep. We use 2 instead of 3 because with only three brokers, this prevents issues during rolling restarts. In production with more brokers, you'd use 3 or more.

• The Volumes – kafka-x-data:/var/lib/kafka/data creates persistent storage for each broker’s data. Without volumes you will lose your topics and messages if you stop or restart your containers. Volumes are assigned to each broker so they don’t accidentally share data.

Note: For a restart from scratch you need to delete the volumes using the following command. The -v flag removes volumes. Without it, old data persists even after down.

docker compose -f docker-compose-basic.yml down -v

If you're using the legacy docker-compose tool (V1), replace docker compose with docker-compose in all commands throughout this tutorial.

Ports

Three ports are used for any given broker. Their purposes are:

Starting the Cluster & Verification

Now that we have the basic docker configuration for Kafka, let’s run it and verify the results.

Run the following command in the same directory where you saved docker-compose-basic.yml:

docker compose -f docker-compose-basic.yml up -d

The -d flag runs the containers in detached mode (in the background), so you get your terminal back.

You should see output like this:

Using the following command, check if the containers running Kafka brokers are up:

docker ps

You should see three Kafka containers (kafka-1, kafka-2, kafka-3) with status “Up” – something like this:

Run the following command to verify that all three brokers are registered in the cluster:

docker exec -it kafka-1 kafka-broker-api-versions --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092

You should see API version information for all three brokers (IDs 1, 2, 3) without any connection errors.

Note that we’re using kafka-1:29092,kafka-2:29092,kafka-3:29092 here (the internal Docker addresses) instead of localhost:9092 because this command runs inside the kafka-1 container by virtue of docker exec -it kafka-1, where localhost only refers to that specific container.

If any of the above verification returns errors or doesn’t show expected result as shown in screenshots, you can run the following command to see logs and debug:

docker logs kafka-1

Creating Topics: The Manual Way

Now that we have a cluster running, let’s simulate a real production use case where different teams need Kafka topics for their applications – payments, logs, events, metrics notifications, you name it.

Let’s start by creating a topic for logs. The command to do this is:

docker exec -it kafka-1 kafka-topics \
  --create \
  --topic freecodecamp-logs \
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 \
  --partitions 12 \
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy

You’ll need to specify some command parameters, which are:

1. The exact broker address kafka-1:29092,kafka-2:29092,kafka-3:29092 (or the IP address of your servers in production)

2. The number of partitions – I have used 12 in the above command. Creating too few partitions creates bottlenecks, while creating too many adds overhead.

3. Retention policy – I have used 7 days (that is, 604800000 milliseconds)

4. Compression type

Manually managing these parameters and running the command a handful of times is okay – but what if you have to run this for every team in your enterprise? Each team will have different requirements. The grind of copy, paste, adjust becomes painful if you have 100+ topics and multiple clusters (dev, staging, prod).

Feel the pain yet? Well, let’s just go on for a minute and we’ll address this issue shortly. For now, if you run the above command you should see the “Created topic” message:

Note: We’re using kafka-1:29092,kafka-2:29092,kafka-3:29092 to reach Kafka brokers because we’re running the command inside of broker kafka-1 by running using docker exec.

Let's keep going. We’ll create more topics using the same command by changing the topic name and partitions. Copy, paste, update, and run the above commands a couple times. On my machine, I ran it 3 more times like below (you can choose to run couple more times with changed values – it won’t matter because concrete values are not important for this tutorial):

docker exec -it kafka-1 kafka-topics \
  --create \
  --topic freecodecamp-views \    
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 \
  --partitions 20 \
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy


docker exec -it kafka-1 kafka-topics \
  --create \
  --topic freecodecamp-analytics \
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 \
  --partitions 3 \ 
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy


docker exec -it kafka-1 kafka-topics \
  --create \
  --topic freecodecamp-articles \ 
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 \
  --partitions 5 \ 
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy

After creating the topics, let’s see all the ones you have now by running the following command:

docker exec -it kafka-1 kafka-topics \ --list \ --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092

You should see a list of topics like this:

Notice that you just get the list of topics but no meaningful information, like:

• How many partitions does each have?

• Which brokers are hosting them?

• Are they evenly distributed?

• What are their configurations?

Partition Information

Let’s try to get information about our partitions. For this tutorial, I have created 4 topics and a total of 40 partitions spread across three brokers. I want to see which broker has the most partitions.

In a well-managed cluster, you’d want them roughly evenly distributed. But how can we check that?

Maybe the describe command shown below can help. Let’s run it:

docker exec -it kafka-1 kafka-topics \
  --describe \
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092

It will return a wall of text, something like this:

So, we have partition information but:

• No summary or aggregation

• No visual representation

• It’s difficult to scan and compare

• It gets exponentially worse with more topics

Counting Leaders

The Leader field in the above screenshot tells you which broker is the leader for each partition. Leaders handle all read and write requests, so you want them evenly distributed or else some brokers will become overloaded.

Let’s try to count how many partitions each broker leads. To do that, run the following command:

docker exec -it kafka-1 kafka-topics \
  --describe \
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 | grep "Leader: 1" | wc -l

It will show something like this:

Per my topic creation, 14 is the count of partitions where broker 1 (Leader : 1) is the leader. You might see a different number depending on how many topics and how many partitions you have created.

You can repeat this command to see the count of partitions led by other brokers. To do so, just change Leader: 1 to Leader: 2 or Leader: 3.. I get 14, 12, 14:

That’s somewhat balanced, but you had to run the command multiple times, parse using grep and wc, and this is just 3 brokers. What if you had 100+? Also, what if you have to get the replicas’ information?

I could go on and on with the data we need and the commands to get that information. But the point I’m trying to make here is that sooner or later this becomes impossible to manage. Your team is going to need an army, and to be honest there isn’t much value in doing all of this manually.

So far, you’ve seen only simple operational commands, but the problems don’t stop there. In a real production environment there are more complex and challenging operations like:

• Consumer Lag Monitoring: When consumers fall behind, you need to track which partitions are lagging, which consumer instances own them, and where the lag is growing or shrinking. With CLI tools, you get raw numbers but no trends or context.

• Broker Failures: When a broker fails, you need to identify under-replicated partitions, trigger leader elections, and create partition reassignment JSON files manually. One mistake in that JSON can cause data loss.

• Cluster rebalancing: You’ll see that when you add new brokers, they sit empty until you manually redistribute partitions. Similarly for removing brokers, you need to move all their partitions first. These operations require calculating optimal placement and creating complex reassignment plans.

If you’re still with me, you’re probably thinking that there has to be a better way. Fortunately, there is – actually, there are a couple complimentary ways and we are going to talk about those next.

Kafka UI

Kafka UI is a modern, open-source web interface for managing Kafka clusters. It replaces the command line chaos we just experienced with a clean, visual dashboard.

Kafka UI provides the following features:

• Visual cluster Overview: see all brokers, topics, and partitions at a glance.

• Topic management: create, configure, and delete topics with a GUI

• Consumer group monitoring: track lags, offsets, and consumer health in real-time

• Message browsing: view actual messages in topics without command line tools

Without further ado, let’s set up Kafka UI.

Setting Up Kafka UI

To setup up Kafka UI, let’s modify our existing docker-compose-basic.yml like this:

version: '3.8'

services:
  kafka-1:
    image: confluentinc/cp-kafka:7.6.0
    container_name: kafka-1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-1-data:/var/lib/kafka/data

  kafka-2:
    image: confluentinc/cp-kafka:7.6.0
    container_name: kafka-2
    ports:
      - "9093:9093"
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:29092,PLAINTEXT_HOST://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-2-data:/var/lib/kafka/data

  kafka-3:
    image: confluentinc/cp-kafka:7.6.0
    container_name: kafka-3
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:29092,PLAINTEXT_HOST://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-3-data:/var/lib/kafka/data
# Adding kafka-UI service start
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8080:8080"
    environment:
      DYNAMIC_CONFIG_ENABLED: 'true'
      KAFKA_CLUSTERS_0_NAME: freecodecamp-cluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:29092,kafka-2:29092,kafka-3:29092
    depends_on:
      - kafka-1
      - kafka-2
      - kafka-3
# Adding kafka-UI service end
volumes:
  kafka-1-data:
  kafka-2-data:
  kafka-3-data:

The yaml file is pretty much the same as before except that we have added a new service called kafka-ui (for better clarity, I have added the changes in between start and end comments).

Key Configurations are:

• Port 8080 – You can access the UI at http://localhost:8080 from your machine.

• KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS – This environment variable tells Kafka UI where to connect your cluster (using internal Docker addresses).

• KAFKA_CLUSTERS_0_NAME – A friendly name for your cluster in the UI.

Let’s first clean up the old cluster while keeping the topic data intact. Go ahead and run the following command to do so:

docker compose -f docker-compose-basic.yml down

Note that we’re not using -v here, so volumes (topic data) will remain intact.

Wait for couple seconds and then run the following docker up command to bring up our cluster with Kafka UI:

docker compose -f docker-compose-basic.yml up -d

Now open a browser and visit http://localhost:8080/. You’ll see the UI like this:

You can click around and see all information about the cluster we have created, like:

• Your 3 brokers

• The topics you created earlier

• Partition counts

For comparison with manual commands, let's look at the Brokers tab. You can see the partition leader count for each broker at a glance – remember that we had to run multiple commands to get this information earlier. Beyond this, the UI provides many other useful metrics that would require separate command-line queries.

Remember the CLI commands we had to run to create topics? If you go to the Topics tab, you will notice that Topic management (creation, deletion, data cleanup and so on) are just a few button clicks.

Similarly, managing Consumers only requires a few button clicks.

After exploring the Kafka UI, you'll see how much easier it is to monitor your cluster compared to running individual CLI commands.

Drawbacks of Kafka UI

That said, Kafka UI does have some limitations:

• Automatic rebalancing: One or few brokers having more partitions that others, you must manually reassign them.

• Self-healing: If a broker fails, you have to manually create reassignment plans.

• Performance optimization: The UI can’t recommend intelligent partition placement.

• Alerts: The UI doesn’t warn you before problems happen.

For small clusters (3 - 10 brokers ), Kafka UI and some command execution might be enough. You’ll be able to see problems clearly and fix them when needed.

For large clusters, manual operations are still not scalable, so we need some kind of a complementary tool…and that tool is Cruise Control.

Cruise Control

Cruise Control is an automation engine for Kafka clusters. While Kafka UI gives you visibility and manual control, Cruise Control provides intelligent automation and self-healing. You can think of Kafka UI as a dashboard with manual controls and Cruise Control as an autopilot. In other words, they complement each other.

Let’s try to create some imbalance in our cluster and fix it manually. This will help you learn how to reason through why you need Cruise Control.

To keep things simple, let’s start from scratch. We will first delete all the Docker resources we have created so far by running the following command:

docker compose -f docker-compose-basic.yml down -v

Running docker-compose down -v will delete all the topics and messages we created so far, but don’t worry –we’ll create them again.

How Cruise Control Works

You can think of Cruise Control as a metric-monitoring and action-taking tool. Kafka brokers collect internal metrics (CPU, disk, network traffic, partition sizes), and a metric reporter running inside each broker sends these metrics to a Kafka topic.

Cruise Control then reads from that topic and analyzes the data. Based on that analysis, it proposes partition movements. We’ll see this in action shortly.

Setting Up Cruise Control

As of this writing, I couldn’t find a compatible Kafka and Cruise Control image that supports KRaft (Kafka Consensus Algorithm), so I decided to create Kafka and Cruise Control public images that will help with the tutorial. I don’t recommend using these images in production. For production usage, you should either wait for community to provide an image or create one of your own.

Change the docker-compose-basic.yml file to look like the below:

version: '3.8'

services:
  kafka-1:
    image: justramesh2000/kafka-apache-cc:3.8.1
    container_name: kafka-1
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      # Cruise Control Metrics Reporter
      KAFKA_METRIC_REPORTERS: 'com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter'
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_BOOTSTRAP_SERVERS: 'kafka-1:29092,kafka-2:29092,kafka-3:29092'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_AUTO_CREATE: 'true'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_NUM_PARTITIONS: '1'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_REPLICATION_FACTOR: '2'
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_KUBERNETES_MODE: 'false'
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_METRICS_REPORTING_INTERVAL_MS: '60000'
    volumes:
      - kafka-1-data:/var/lib/kafka/data

  kafka-2:
    image: justramesh2000/kafka-apache-cc:3.8.1
    container_name: kafka-2
    ports:
      - "9093:9093"
    environment:
      KAFKA_NODE_ID: 2
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:29092,PLAINTEXT_HOST://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      KAFKA_METRIC_REPORTERS: com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka-1:29092,kafka-2:29092,kafka-3:29092
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_KUBERNETES_MODE: 'false'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC: __CruiseControlMetrics
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_AUTO_CREATE: 'true'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_NUM_PARTITIONS: '1'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_REPLICATION_FACTOR: '2'
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_METRICS_REPORTING_INTERVAL_MS: '60000'
    volumes:
      - kafka-2-data:/var/lib/kafka/data

  kafka-3:
    image: justramesh2000/kafka-apache-cc:3.8.1
    container_name: kafka-3
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 3
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:29092,PLAINTEXT_HOST://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      KAFKA_METRIC_REPORTERS: com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka-1:29092,kafka-2:29092,kafka-3:29092
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_KUBERNETES_MODE: 'false'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC: __CruiseControlMetrics
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_AUTO_CREATE: 'true'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_NUM_PARTITIONS: '1'
      KAFKA_CRUISE_CONTROL_METRICS_TOPIC_REPLICATION_FACTOR: '2'
      KAFKA_CRUISE_CONTROL_METRICS_REPORTER_METRICS_REPORTING_INTERVAL_MS: '60000'
    volumes:
      - kafka-3-data:/var/lib/kafka/data
  # Adding kafka-UI service start
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8080:8080"
    environment:
      DYNAMIC_CONFIG_ENABLED: 'true'
      KAFKA_CLUSTERS_0_NAME: freecodecamp-cluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:29092,kafka-2:29092,kafka-3:29092
    depends_on:
      - kafka-1
      - kafka-2
      - kafka-3
    volumes:
      - ./config:/opt/cruise-control/config  
  # Adding kafka-UI service end
  # Adding cruise-control start
  cruise-control:
    image: justramesh2000/cruise-control-kraft:2.5.142
    container_name: cruise-control
    ports:
      - "9090:9090"
    volumes:
      - ./config/cruisecontrol.properties:/opt/cruise-control/config/cruisecontrol.properties
      - ./config/capacityJBOD.json:/opt/cruise-control/config/capacityJBOD.json:ro
      - ./config/log4j.properties:/opt/cruise-control/config/log4j.properties:ro
    depends_on:
      - kafka-1
      - kafka-2
      - kafka-3
   # Adding cruise-control end    
volumes:
  kafka-1-data:
  kafka-2-data:
  kafka-3-data:

You should have made the following changes to the file:

• Changed Kafka image from confluentinc/cp-kafka:7.6.0 to justramesh2000/kafka-apache-cc:3.8.1. The new image contains the Cruise Control metrics exporter which will export metrics data from Kafka brokers to be used by Cruise Control.

• Added the following environment variables:

  • KAFKA_METRIC_REPORTERS – This variable tells Kafka to load a plugin called the Cruise Control Metrics Reporter. It runs inside each Kafka broker process, and hooks into Kafka’s internal metrics system. This helps with data collection.

  • KAFKA_CRUISE_CONTROL_METRICS_REPORTER_BOOTSTRAP_SERVERS – This tells the Cruise Control Metrics Reporter where to send metrics to, meaning which Kafka brokers and which port.

  • KAFKA_CRUISE_CONTROL_METRICS_REPORTER_KUBERNETES_MODE – This disables specific Kubernetes behaviors (Pod name, id instead of Host). We are using Docker, so we don’t need K8s behaviors.

  • KAFKA_CRUISE_CONTROL_METRICS_TOPIC – Specifies the name of the topic where metrics will be published. Default is __CruiseControlMetrics but you can customize using this variable if you want to.

  • KAFKA_CRUISE_CONTROL_METRICS_TOPIC_AUTO_CREATE – Automatically creates a __CruiseControlMetrics topic if it doesn’t exist. Without this metric, the reporter will fail reporting until you manually create this topic.

  • KAFKA_CRUISE_CONTROL_METRICS_TOPIC_NUM_PARTITIONS – Defines the number of partitions for the topic __CruiseControlMetrics.

  • KAFKA_CRUISE_CONTROL_METRICS_TOPIC_REPLICATION_FACTOR – Tells Kafka how many copies of metrics data to keep. In our case, we’re keeping 2 copies of the data.

  • KAFKA_CRUISE_CONTROL_METRICS_REPORTER_METRICS_REPORTING_INTERVAL_MS – Tells Kafka how often to send metrics. We’re sending every minute.

• Added Cruise-control service using image justramesh2000/cruise-control-kraft:2.5.142. For clarity, I have kept this change between the start and end comments.

• Under cruise control, we’ve mounted three Cruise Control configurations files. We’ll talk about those files next.

Cruise Control Configuration File

To run Cruise Control, we need to provide several configuration files. Among the key pieces of information are:

• Where the Kafka cluster is located

• The capacity of each broker

Create a config directory and add the following files:

mkdir config

cruisecontrol.properties

This is Cruise Control’s main configuration file.

Save the following content as cruisecontrol.properties in the config directory:

# Kafka cluster. Tells how to connect to brokers
bootstrap.servers=kafka-1:29092,kafka-2:29092,kafka-3:29092

# Topic from which metrics are to be read
metric.reporter.topic=__CruiseControlMetrics

# Aggregated partition data
partition.metric.sample.store.topic=__KafkaCruiseControlPartitionMetricSamples

#Aggregated broker data
broker.metric.sample.store.topic=__KafkaCruiseControlModelTrainingSamples

# Enable broker failure detection for KRaft mode (no ZooKeeper)
kafka.broker.failure.detection.enable=true

# Capacity. Tells where the capacity file is 
capacity.config.file=config/capacityJBOD.json

# Goals. What to optimize for during cluster balancing. These are the riles for CC to abide to during rebalancing
default.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal

# hard goals. 
hard.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,\
com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal

# Webserver. For WebApi access
webserver.http.port=9090
webserver.http.address=0.0.0.0

# Execution
num.broker.metrics.windows=1
num.partition.metrics.windows=1

I’ve added in line comments to explain much of the above configuration, but I think the Goals need special attention. These are the rules that we as users have set for Cruise Control to abide by.

By defining goals, we tell Cruise Control to do the following:

• RackAwareGoal – Spread replicas across racks (or in our case, brokers)

• ReplicaCapacityGoal – Don't overload brokers with too many replicas

• DiskCapacityGoal – Don't fill up disk

• NetworkInboundCapacityGoal – Balance incoming network traffic

• NetworkOutboundCapacityGoal – Balance outgoing network traffic

• CpuCapacityGoal – Balance CPU usage

• ReplicaDistributionGoal – Evenly distribute replicas

• DiskUsageDistributionGoal – Ensure even disk usage across brokers

• LeaderReplicaDistributionGoal – Evenly distribute leader replicas

• LeaderBytesInDistributionGoal – Balance data flowing to leaders

Via Cruise Control configuration, you can define two types of goals: Default goals and Hard goals. Hard goals must be met. Default goals that aren’t part of the hard goals become soft goals. This means that Cruise Control will give its best effort to satisfy them but won’t reject a proposal if it can’t.

Here’s a little summary:

Cruise control collects metrics for a defined period (default: 5 minutes) and creates a monitoring window. The following settings control how many windows Cruise Control needs before it’s ready to generate proposals (shortly, we will see what proposals are):

• num.broker.metrics.windows=1: Wait for 1 monitoring window before generating proposals. Each window in Cruise Control is 5 minutes by default. This means that Cruise Control will be ready after 5 minutes. I’ve set this to 1 for quick testing. The recommendation is to use a large window in production to avoid false proposals from temporary spikes.

• num.partition.metrics.windows=1: Wait for 1 window of partition metrics. Same reasoning as above.

Capacity

This informs cruise control about the capacity (CPU, DISK) of each broker, which then helps it to make decisions. Using the below file, we’re telling Cruise Control the following:

• What are the brokerIds

• What is the disk path /var/lib/kafka/data and disk capacity (100000000 MB = 100 GB). This is used by DiskCapacityGoal that we set up in the above cruisecontrol.properties file.

• What is the CPU 100% (1 Core). Used by CpuCapacityGoal.

• What is the NW_IN Network Inbound Capacity (125,000 KB/s = 1 MB/s –Megabytes per second) = 1 Gbps – Giga bits per second). Used by NetworkInboundCapacityGoal.

• What is the NW_OUT Network Outbound Capacity (125,000 KB/s). Used by NetworkOutboundCapacityGoal

Save the following content as capacityJBOD.json in the config directory:

{
  "brokerCapacities":[
    {
      "brokerId": "1",
      "capacity": {
        "DISK": {"/var/lib/kafka/data": "100000000"},
        "CPU": "100",
        "NW_IN": "125000",
        "NW_OUT": "125000"
      }
    },
    {
      "brokerId": "2",
      "capacity": {
        "DISK": {"/var/lib/kafka/data": "100000000"},
        "CPU": "100",
        "NW_IN": "125000",
        "NW_OUT": "125000"
      }
    },
    {
      "brokerId": "3",
      "capacity": {
        "DISK": {"/var/lib/kafka/data": "100000000"},
        "CPU": "100",
        "NW_IN": "125000",
        "NW_OUT": "125000"
      }
    }
  ]
}

Logging

This is not important for Cruise Control to work properly, but it’ll help you debug if there are issues. Save the following content as log4j.properties in the config directory. When you execute commands to start Cruise Control and If you see unexpected behaviors like container exiting, you can use the docker logs command to see what happened.

# Root logger - INFO level, output to console
rootLogger.level=INFO
appenders=console

# Console output (for docker logs)
appender.console.type=Console
appender.console.name=STDOUT
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=[%d] %p %m (%c)%n

# Send root logger to console
rootLogger.appenderRef.console.ref=STDOUT

Now that we have all the configurations in place, let’s run the following command to start Kafka brokers with Kafka UI and Cruise Control:

docker compose -f docker-compose-basic.yml up -d

Using the following command, verify that the three Kafka brokers, Kafka UI, and Cruise Control containers are running:

docker ps

You should see something like this:

Now that we have Cruise Control up and running, let’s create some Imbalance and see how much better of an experience we get by using Cruise Control versus mitigating the imbalance manually.

Creating the Imbalance

An imbalance is a scenario where some brokers are handling more messages than others – and they may run into high disk usage or high IOPS.

To create the imbalance in our cluster, we’ll have to create a few topics and then produce messages unevenly. Now that you have Kafka UI running, you can create topics using that method or you can create topics using commands. I’m going to use the commands because it’ll be easier for you to reproduce my work (but I recommend UI for production operations because it prevents typos).

If you also decide to use commands, run the following command. Then using UI, verify that the topics have been created.

Note: You’ll find that the commands are different from previous commands. This is because, previously in our docker-compose-basic.yml file, we were using the confluentinc/cp-kafka:7.6.0 image for Kafka. But now we’re using the justramesh2000/kafka-apache-cc:3.8.1 image which is based off of the apache/kafka:3.8.1 image. For different images, the tools are located at different places, so the command needs to be adjusted to account for that.

docker exec -it kafka-1 bash -c '
/opt/kafka/bin/kafka-topics.sh --create \
  --topic freecodecamp-logs \
  --bootstrap-server kafka-1:29092 \
  --partitions 12 \
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy

/opt/kafka/bin/kafka-topics.sh --create \
  --topic freecodecamp-views \
  --bootstrap-server kafka-1:29092 \
  --partitions 20 \
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy

/opt/kafka/bin/kafka-topics.sh --create \
  --topic freecodecamp-analytics \
  --bootstrap-server kafka-1:29092 \
  --partitions 3 \
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy

/opt/kafka/bin/kafka-topics.sh --create \
  --topic freecodecamp-articles \
  --bootstrap-server kafka-1:29092 \
  --partitions 5 \
  --replication-factor 2 \
  --config retention.ms=604800000 \
  --config compression.type=snappy
'

Run the following command to produce uneven messages on different topics we created above.

Heavy Load on freecodecamp-logs:

docker exec -it kafka-1 bash -c "
for i in {1..100000}; do 
  echo '{\"log_id\":\"'\$i'\",\"level\":\"INFO\",\"message\":\"Log entry '\$i'\"}'
done | /opt/kafka/bin/kafka-console-producer.sh --topic freecodecamp-logs --bootstrap-server kafka-1:29092"

Heavy load on freecodecamp-views:

docker exec -it kafka-1 bash -c "
for i in {1..80000}; do 
  echo '{\"view_id\":\"'\$i'\",\"page\":\"/article/'\$((i % 100))'\",\"user\":\"user_'\$((i % 1000))'\"}'
done | /opt/kafka/bin/kafka-console-producer.sh --topic freecodecamp-views --bootstrap-server kafka-1:29092"

Moderate load on freecodecamp-analytics:

docker exec -it kafka-1 bash -c "
for i in {1..30000}; do 
  echo '{\"event\":\"page_view\",\"user\":\"user_'\$i'\"}'
done | /opt/kafka/bin/kafka-console-producer.sh --topic freecodecamp-analytics --bootstrap-server kafka-1:29092"

Now, produce a message with a fixed key to force all data into one Partition. This is a fast way to create strong disk imbalance. Run the following command:

docker exec -it kafka-1 bash -c "
for i in {1..300000}; do
  echo 'hotkey:{\"log_id\":'\$i',\"msg\":\"big payload\"}'
done | /opt/kafka/bin/kafka-console-producer.sh \
  --topic freecodecamp-logs \
  --bootstrap-server kafka-1:29092 \
  --property parse.key=true \
  --property key.separator=:"

After running the above commands, come back to the UI, refresh, and you will see a number of messages like this:

Now, go to brokers tab and see the imbalance in Disk Usage:

You should be able to see that Broker-2 has only about 47% of the data that Broker-1 has, and Broker-3 has about 11% more data than Broker-1. Broker-2 is significantly underutilized, while Broker-1 and Broker-3 hold most of the data.

Attempting Manual Rebalancing

Step 1: First, we need to find out which topic is heavy – meaning which one handles more data. My setup shows the freecodecamp-logs topic with 8MB of data:

Step 2: Let’s see where the heavy partitions are.

Click on freecodecamp-logs in Kafka UI and see the partition table. Look at the message count: partition 4 is bigger than the others. The table also gives information about replicas of partitions: partition 4 has replicas on Broker 1 and 3. Broker 2 doesn’t have partition 4 at all. This explains why Broker 2 was underutilized.

Step 3: To balance the cluster, we need to move partition 4 around.

We can move partition 4 to Broker 2. But before that, let’s do some math to be able to rationalize our decision. Note that the calculation doesn’t have to be precise – we just want a relative sense of data between brokers.

Current state:

• Broker 1: 4.55 MB

• Broker 2: 2.29 MB (underutilized)

• Broker 3: 5.11 MB (over-utilized)

Note that roughly the compressed data size for partition 4 is 2.25 MB (exact size is not critical).

If we move partition 4 from [1,3] to [2,3]:

• Broker 1: Loses partition 4, so 4.55 + 2.25 = ~2.3 MB

• Broker 2: Gains Partition 4, so 2.33 + 2.25 = ~4.58 MB

• Broker 3: Already has partition 4, so = 5.11 MB (no change)

The result is that Broker 1 becomes underutilized.

How about if we move partition 4 from [1,3] to [1,2]?

• Broker 1: Already has partition 4 = 4.55 MB (no change)

• Broker 2: Gains Partition 4, so 2.33 + 2.25 = ~4.58 MB

• Broker 3: Loses partition 4, so 5.11 + 2.25 = ~2.8 MB

Hmm, this still creates an imbalance (broker 3 becomes too light).

So basically, manual rebalancing requires complex calculations. Moving a single partition impacts disk usage, leader distribution, and network traffic across multiple brokers. One poorly planned move can create a new imbalance elsewhere.

But, let’s say you somehow landed on a perfect mathematical calculation and you’re ready to make the move to balance. We’ll assume that the perfect plan is to move Partition 4 from [1, 3] to [2, 3]. I know it’s not the perfect move but the point is to see the pain afterwards.

Step 4: it’s time to move the partition manually.

We need to tell Kafka to move partition 4's replicas from brokers [1,3] to brokers [2,3].

To do that, you need create a file called reassignment.json on your machine:

{
  "version": 1,
  "partitions": [
    {
      "topic": "freecodecamp-logs",
      "partition": 4,
      "replicas": [2, 3],
      "log_dirs": ["any", "any"]
    }
  ]
}

What this means:

• "partition": 4 – Target Partition

• "replicas": [2, 3] – New placement: brokers 2 and 3

• "log_dirs": ["any", "any"] – Let Kafka choose the disk directory

Save this file somewhere accessible.

Then run the following command to copy the JSON to the Kafka cluster:

docker cp reassignment.json kafka-1:/tmp/reassignment.json

This copies your local file into the kafka-1 container's /tmp directory.

Run following command to verify the file is there:

docker exec -it kafka-1 cat /tmp/reassignment.json

You should see your JSON file content.

Now run the actual reassignment command:

docker exec -it kafka-1 /opt/kafka/bin/kafka-reassign-partitions.sh \
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 \
  --reassignment-json-file /tmp/reassignment.json \
  --execute

You will get a message from Kafka that will tell you if Kafka has accepted the reassignment and started moving the data.

You can monitor the reassignment using the following command:

docker exec -it kafka-1 /opt/kafka/bin/kafka-reassign-partitions.sh \
  --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 \
  --reassignment-json-file /tmp/reassignment.json \
  --verify

I’m not going to run the manual reassignment because I want to keep the imbalance and show how Cruise Control can help reduce the manual steps. Next, let’s see how Cruise Control handles the same imbalance automatically.

Rebalancing Using Cruise Control

After creating the topic and messages, I have let Cruise Control run for a couple minutes. During that time, it collected metrics and trained its linear regression model. You can run the following command to verify if Cruise Control is running fine and it has data (following is a REST API call using curl):

curl http://localhost:9090/kafkacruisecontrol/state

You will get multiple JSON object outputs as part of the response. Each JSON object holds some information about the state of Cruise Control and the Kafka cluster. Let’s see each of these one at a time:

MonitorState: {
  state: RUNNING(20.000% trained),
  NumValidWindows: (1/1) (100.000%),
  NumValidPartitions: 105/105 (100.000%),
  flawedPartitions: 0
}

This tells about the state of monitoring based on data collected by Cruise Control:

• state: RUNNING(20.000% trained) – Cruise Control is actively collecting metrics from your Kafka cluster. Right now it has trained its model on 20% of the expected monitoring data.

• NumValidWindows: (1/1) (100%) – Cruise Control has collected 1 complete monitoring window out of 1 required (100% ready). Remember, we had set num.broker.metrics.windows=1 in the cruisecontrol.properties configuration file.

• NumValidPartitions: 105/105 (100%) – Cruise Control analyzed all 105 partitions and has metrics for all.

• flawedPartitions: 0 – None of the partitions have problematic or missing metrics.

ExecutorState: {state: NO_TASK_IN_PROGRESS}

The above response indicates the execution engine is idle – no partition moves or leadership changes are currently in progress. This makes sense since we haven't asked Cruise Control to do anything yet.

AnalyzerState: {
  isProposalReady: true,
  readyGoals: [
    NetworkInboundCapacityGoal,
    LeaderBytesInDistributionGoal,
    DiskCapacityGoal,
    ReplicaDistributionGoal,
    RackAwareGoal,
    NetworkOutboundCapacityGoal,
    CpuCapacityGoal,
    DiskUsageDistributionGoal,
    LeaderReplicaDistributionGoal,
    ReplicaCapacityGoal
  ]
}

AnalyzerState tells whether Cruise Control is ready to show a proposal or not. In this case it’s ready.

• isProposalReady: true – Cruise Control has calculated a potential rebalancing plan (a proposal) that satisfies the configured goals.

• readyGoals – These are the goals that are considered ready and valid for rebalancing. Examples:

  • DiskCapacityGoal: balance disk usage among brokers

  • ReplicaDistributionGoal: balance number of replicas per broker

  • RackAwareGoal: maintain replicas across racks for fault tolerance

  • LeaderBytesInDistributionGoal: balance network traffic from leaders

  • DiskUsageDistributionGoal: ensures partitions are spread to prevent skew

Note that these are the goals we had set earlier in the cruisecontrol.properties file.

AnomalyDetectorState: {
  selfHealingEnabled:[],
  selfHealingDisabled:[BROKER_FAILURE, DISK_FAILURE, GOAL_VIOLATION, METRIC_ANOMALY, TOPIC_ANOMALY, MAINTENANCE_EVENT],
  selfHealingEnabledRatio:{...},
  recentGoalViolations:[],
  recentBrokerFailures:[],
  recentMetricAnomalies:[],
  recentDiskFailures:[],
  recentTopicAnomalies:[],
  recentMaintenanceEvents:[],
  metrics:{...},
  ongoingSelfHealingAnomaly:None,
  balancednessScore:100.000
}

Anomaly detection shows information about any existing anomaly and healing properties.

• selfHealingEnabled: [] – Automatic self-healing is currently off. Cruise Control will not move partitions automatically in response to anomalies.

• selfHealingDisabled: [...] – Lists the anomaly types that are disabled for automatic self-healing, including broker failures, disk failures, and goal violations.

• recentGoalViolations: [] – No goals have been violated recently.

• balancednessScore: 100.000 – This is how balanced the cluster is according to Cruise Control’s hard goals. 100% means the cluster is perfectly balanced according to the metrics and hard goals currently active. This metric only cares about Hard Goals (Disk Capacity, CPU capacity) being violated – that’s why it shows 100% even though we know there are some disk usage imbalances in our cluster.

The Proposal

Via AnalyzerState information, Cruise Control told us that it has a proposal for the cluster. Let’s see what it is. We can fetch the proposal using the proposal end point:

curl -s "http://localhost:9090/kafkacruisecontrol/proposals?json=true"

The JSON response is quite large. Let's focus on the key parts that show our cluster's imbalance and how Cruise Control plans to fix it:

{
  "summary": {
    "numReplicaMovements": 13,    // CC wants to move 13 partition replicas
    "numLeaderMovements": 6,      // And reassign 6 partition leaders
    "onDemandBalancednessScoreBefore": 84.67,   // Current: 84.67% balanced
    "onDemandBalancednessScoreAfter": 89.76.    // After: 89.76% balanced
  },
  "goalSummary": [
    {
      "goal": "DiskUsageDistributionGoal",
      "status": "VIOLATED"
    },
    {
      "goal": "LeaderBytesInDistributionGoal",
      "status": "VIOLATED"
    }
  ]
}

Based on the calculations, Cruise Control thinks:

1. Moving 13 partition replicas will help. Note that manually we decided to move just 1 partition, that is partition 4.

2. Reassigning 6 partition leaders will help. Manually we didn’t account for any leadership reassignment.

3. DiskUsageDistributionGoal has been violated. We know that the disk usage is not distributed perfectly.

4. LeaderBytesInDistributionGoal has also been violated. We couldn’t find this out manually. Technically, you could find out but it would take a decent amount of manual calculations and would still be error-prone.

Note: While we're focusing on disk usage imbalance, Cruise Control optimizes for 10 different goals (disk, CPU, network, leaders, and so on). This holistic approach gives it a better chance of achieving true cluster balance versus balancing manually.

Executing the proposal

Let’s run the actual rebalancing using Cruise Control. The command is:

curl -X POST 'http://localhost:9090/kafkacruisecontrol/rebalance?dryrun=false&json=true'

Again, you’ll get a huge JSON file similar to the proposal.

You can track the status using following API call:

curl "http://localhost:9090/kafkacruisecontrol/user_tasks"

You will get something like this:

Note that the 4th item in the list is our rebalance API call and it’s complete. This was quick for our small Dev cluster, but in large clusters you may see status as InExecution.

Let’s look at the UI to see what is the state of Imbalance now that Cruise Control has completed its execution of the proposal. The UI shows the following for me:

Comparison

Before rebalancing:

• Broker 1: 4.52 MB, 69 partitions, 35 leaders

• Broker 2: 2.22 MB, 69 partitions, 35 leaders (underutilized)

• Broker 3: 5.05 MB, 72 partitions, 35 leaders (overutilized)

• Disk range: 2.83 MB (5.05 - 2.22)

After rebalancing:

• Broker 1: 4.66 MB, 69 partitions, 38 leaders

• Broker 2: 3.87 MB, 77 partitions, 31 leaders

• Broker 3: 4.87 MB, 64 partitions, 36 leaders

• Disk range: 1.00 MB (4.87 - 3.87)

Results:

• Disk usage balanced – Range reduced from 2.83 MB to 1.00 MB (64% improvement!)

• Replicas redistributed – Broker 2 gained 8 replicas, Broker 3 lost 8 replicas

• Leaders balanced – Changed from 35-35-35 to 38-31-36. Cruise Control prioritized balancing actual network traffic over leader count.

The cluster is now more balanced across all metrics. Congrats!

Conclusion

We covered a lot in this tutorial, so let’s take a step back and look at what we did.

You started by experiencing the reality of manual Kafka management – the endless CLI commands, the tedious calculations, the JSON files, and the potential for costly mistakes. If you felt frustrated during that section, that’s to be expected. That frustration is exactly what thousands of engineering teams deal with every day.

Then you were presented with two complementary tools:

1. Kafka UI gave you visibility. No more grepping through command outputs or manually counting partition leaders. Everything you need, broker health, topic configurations, consumer lag is right there in a clean web interface. For small teams and development environments, this alone is a game-changer.

2. Cruise Control gave you intelligence. It didn't just automate what you'd do manually – it also did a fundamentally better job. While you were focused on moving one partition (partition 4), Cruise Control analyzed all 105 partitions across 10 different optimization goals and proposed a comprehensive rebalancing plan. That's the difference between human effort and automated intelligence.

I want to call out that this tutorial used a simplified setup. For production, you’ll expect complex configurations like”

• Kafka and Cruise Control running on separate machines

• Larger monitoring window for Cruise Control

• Some self healing capabilities enabled

If there's one thing you take away from this article, let it be this: you should stop managing your Kafka cluster manually. You've seen there's a better way. Use it. Thanks for reading!

Whether it’s moving data between apps, triggering actions when something changes, or building smart systems that run on their own, automation can save hours every week.

The problem is that most automation platforms make you choose between flexibility and simplicity.

Tools like Zapier are easy to use but limited when you need customisation. Writing your own scripts in Python or JavaScript gives you full control but takes more time to build and maintain.

n8n changes that balance. It is an open-source workflow automation platform that provides both control and simplicity.

n8n lets you automate anything from simple tasks to complex systems using a visual interface. You can drag and connect nodes to create workflows or write code when needed. It’s built for technical teams who want freedom without losing ease of use.

In this article, we’ll learn how to build and deploy your own automation workflows using n8n. By the end, you’ll have a working automation server and the knowledge to create smart, self-running workflows for any use case.

Table of Contents

• What n8n Does

• n8n Is Open Source

• How to Get Started with n8n

• Building a n8n Workflow

• Running n8n in Production using Sevalla

• Where n8n Becomes Powerful

• AI-Driven Automations

• Conclusion

What n8n Does

n8n connects the apps and systems you already use.

Each connection is called a node, and every node performs an action. You can combine multiple nodes into a workflow that runs automatically.

For example, you could create a workflow where a new form submission in Typeform triggers a Slack message and stores the data in Google Sheets. You can then add logic to send an email only if certain conditions are met.

This approach allows anyone to build automation visually, yet it stays developer-friendly. You can use JavaScript or Python inside the workflow for custom logic, import npm packages, or connect to any API that doesn’t have a prebuilt node yet.

The platform supports over four hundred integrations out of the box, from GitHub and AWS to OpenAI and Telegram. This large library of ready-to-use nodes means you can connect most tools you use every day without needing to write any code at all.

n8n is Open Source

The open source nature of n8n is what makes it stand out.

Most automation tools like Zapier are closed systems that hide their inner workings. With n8n, the source code is publicly available. You can host it on your own server, modify it, and inspect how everything works.

This matters for both privacy and flexibility.

When you self-host n8n, your data never leaves your environment. This is especially useful for industries like finance, healthcare, and security where sensitive data must stay private. Teams can build automations without sending information through third-party servers.

Being open source also means you are never locked into one vendor. You can add your own nodes, extend the platform, or even contribute back to the community.

The fair-code license ensures that while the project stays sustainable for the developers who maintain it, it remains accessible to anyone who wants to use or modify it.

How to Get Started with n8n

Getting started with n8n takes only a few minutes. If you already have Node.js installed, you can launch it right from your terminal using the command:

npx n8n

This will start n8n locally and open the visual editor at http://localhost:5678.

You can also deploy n8n with Docker using a few simple commands. Docker is often the easiest option if you want a persistent setup where your data and workflows are saved automatically.

Once the editor is open, you’ll see an empty canvas where you can drag and drop nodes. For beginners, the best way to learn is by building small workflows.

Building a n8n Workflow

Let’s build a simple n8n workflow.

Step 1: After logging in, click on “Create Workflow” at the top. This will open a blank workspace. Give your workflow a name such as “RSS to Email”. You’ll be building a simple chain of steps, where one action leads to another.

Step 2: Every workflow in n8n starts with a trigger, which decides when the workflow should run. In this example, we’ll use the Schedule Trigger so the workflow runs once a day.

Click the plus icon to add a new node and search for “On a Schedule”. Select it and choose the option that says “Every Day”. You can set the exact time you want it to run, for example, every morning at 9am. This means that once your workflow is activated, n8n will automatically start it daily at that time.

Step 3: Now that the workflow knows when to run, it needs to know what to do. The next step is to fetch the latest articles from a blog’s RSS feed. Click the plus icon again to add another node and search for “RSS Read”.

In the URL field, type the link to a blog’s feed such as https://blog.cloudflare.com/rss/. Click “Execute Node” to test it. You should now see a list of recent blog posts with their titles, descriptions, and links. This confirms that the feed is working correctly.

Step 4: Sometimes you may not want all the items from the RSS feed. For instance, you might only want the top three posts. To do this, you can add a Function node between the RSS and email steps. In that node, enter a short JavaScript snippet like return items.slice(0, 3);. This will trim the list and only keep the first three results. You can also choose to skip this step if you want to send all the posts in the email.

Step 5: The next step is to send the RSS feed items to your email inbox. Add another node and search for “Email”. You can use your preferred email service such as Gmail or Outlook, or configure it manually using SMTP settings.

For Gmail, choose “Send an email”. For the settings, get your oauth keys from Google. In the subject field, write something like “Daily Blog Updates”. In the message field, you can include the data from the RSS feed using expressions such as {{ $json["title"] }} - {{ $json["link"] }}.

This will automatically replace those variables with the actual titles and links when the workflow runs. You can test the email by clicking “Execute Node” and checking your inbox.

Step 6: Once you have added all three nodes, Schedule Trigger, RSS Feed Read, and Email, you need to connect them in that order. The arrows show the flow of data.

Click “Execute Workflow” to test everything. If the setup is correct, you should receive an email with the latest blog posts. When you’re satisfied with the result, turn on the workflow by clicking the toggle switch in the top right corner. It will now run automatically every day without you having to open n8n again.

As you get comfortable, you can start chaining multiple services together, add conditional logic, or include custom code nodes for specific cases. The live execution view helps you see how data moves between nodes in real time.

Running n8n in Production using Sevalla

When you are ready to move beyond testing, n8n gives you two main options. You can self-host it using your own infrastructure or use their managed cloud version at n8n.io.

Self-hosting gives you full control and is usually preferred by technical teams who want to integrate with private APIs or keep sensitive data in-house.

You can choose any cloud provider, like AWS, DigitalOcean, or others to set up N8N. But I will be using Sevalla.

Sevalla is a PaaS provider designed for developers and dev teams shipping features and updates constantly in the most efficient way. It offers application hosting, database, object storage, and static site hosting for your projects.

I am using Sevalla for two reasons:

• Every platform will charge you for creating a cloud resource. Sevalla comes with a $50 credit for us to use, so we won’t incur any costs for this example.

• Sevalla has a template for n8n, so it simplifies the manual installation and setup for each resource you will need for installation.

Log in to Sevalla and click on Templates. You can see n8n as one of the templates.

Click on the “N8N” template. You will see the resources needed to provision the application. Click on “Deploy Template”.

You can see the resource being provisioned. Once the resources are provisioned, go to your n8n application and click on the current deployment.

Wait for a few minutes. Once the deployment is complete, you will see a green checkmark.

Click on “Visit app”. You will get a cloud url eg. https://n8n-9u6kc.sevalla.app/.

You now have a production-grade n8n server running on the cloud. You can use this to build your automations in your self hosted cloud environment.

Where n8n Becomes Powerful

Most users begin with simple automations. But n8n’s true power shows up when you start building complex, multi-step workflows. You can create sequences that involve APIs, data transformation, and logic-based decision making.

For example, a marketing team could build a system that monitors mentions on Twitter, classifies them with an AI model, adds potential leads to a CRM, and sends a Slack alert for high-priority mentions.

A developer could build a workflow that triggers deployment pipelines automatically when code is merged into a branch.

Because n8n supports both no-code and full-code modes, you never outgrow it. As your automations become more advanced, you can still use the same platform to handle them.

AI-Driven Automations

n8n is also built for the era of AI. It comes with native support for connecting large language models and tools like LangChain. This means you can build AI workflows that use your own data and logic.

Imagine setting up a workflow that reads new support tickets, summarizes them with an AI model, and routes them to the right team. Or one that takes blog posts, generates summaries, and posts them automatically to your social channels.

You can design these workflows visually while letting the AI handle the heavy lifting.

Because n8n allows you to control how and where AI models are called, it gives teams flexibility without sacrificing data security. You can integrate your own OpenAI key, run local models, or use third-party APIs in the same environment.

The real value of n8n lies in how it combines flexibility, transparency, and control. It doesn’t hide complexity from you but gives you tools to manage it better. You can start small with visual automation and grow into advanced logic and AI-driven workflows.

Because it’s open source, you never risk losing access to your automations. You can run it anywhere, connect it with anything, and inspect everything that happens under the hood. This level of freedom is rare among modern automation platforms.

For beginners, n8n is an opportunity to understand how automation works without needing to learn full-stack programming. For developers, it’s a scalable system that can power serious production workflows.

Conclusion

Automation is becoming an essential part of every technical process. The challenge is finding a tool that balances simplicity with power. n8n achieves that balance by being open source, extensible, and flexible enough for both no-code users and developers.

n8n is not just another automation app. It is a complete, open, and developer-friendly platform built to make automation accessible to everyone.

Hope you enjoyed this article. Find me on Linkedin or visit my website.

Now, if you’re thinking “writing test cases isn’t my job as a frontend or backend developer”, then you’re missing the point. That mindset holds you back.

At the very least, every engineer should understand Unit Testing and Integration Testing. Writing test cases isn’t rocket science, it’s as simple as English and feels very similar to writing JavaScript code.

That said, if you’ve ever tried setting up testing in a JavaScript application, you probably know how complicated and frustrating it can get.

The JavaScript ecosystem is massive, with endless libraries and frameworks. Things shift constantly, new tools replace old ones, and community standards evolve almost overnight. That’s exactly why I decided to write this article.

In it, we’ll explore a modern approach to JavaScript testing, covering practical patterns, workflows, and even how AI-assisted tools are changing the game.

Let’s dive in.

Table of Contents

• The Evolution of Testing

• The Core Layers of Testing

  • Unit testing

  • Integration testing

  • End-to-End testing

  • AI-Augmented testing

• Future of JavaScript Testing

• Conclusion

• Before We End

The Evolution of Testing

Software testing has been around for as long as software itself. According to IBM (2016), testing started right alongside the very first programs. After World War II, three computer scientists wrote what’s considered to be the first piece of software.

It ran on June 21, 1948, at the University of Manchester in England, performing mathematical calculations with basic machine code instructions.

Since then, testing methods and principles have continuously evolved. As software became more complex and development cycles got faster, the need for reliable and systematic testing grew stronger.

In the early days, the concept of the Testing Pyramid became popular. At the base, you had unit tests, in the middle integration tests, and at the very top a thin layer of end-to-end (E2E) tests. This approach worked well for simpler applications.

But as apps grew more dynamic and interconnected, the pyramid approach began to show its limits. That’s where the Testing Trophy model came in. Instead of overloading with unit tests, it puts greater emphasis on integration testing while still keeping E2E tests and unit tests in balance.

Now, with the rise of AI in QA, testing has entered a new phase. AI-driven tools don’t just run tests, they help generate, maintain, and even self-heal them. This shift is creating a future-ready testing framework designed to handle the complexity of modern software in 2025 and beyond.

The Core Layers of Testing

Testing is not just about finding bugs, but also ensuring reliability, scalability, and user satisfaction. Every testing strategy should cover four main layers:

Unit Testing

Unit testing is a method where you test individual components or units of software in isolation to make sure they work as expected. A unit can be a simple function, a React component, or even a utility module.

When building JavaScript apps, we usually create separate modules or components that later get combined. If any one of those small pieces is broken, the entire application can fail. That’s why unit tests are essential, they catch problems early and ensure reliability before integration.

In the JavaScript ecosystem, there are several tools you can use for writing unit tests:

• Vitest – a modern, fast, and developer-friendly testing framework built to work seamlessly with Vite projects.

• Jest – one of the most widely used testing frameworks, great for React apps among others.

For this section, we’ll focus on Vitest, because it’s lightweight, super-fast, and feels very natural for modern frontend development. Let’s write a test case for a small module.

Imagine we have a simple utility function that adds two numbers:

// sum.ts
export const sum = function (a: number, b: number) {
  return a + b;
};

Every test typically has 3 parts:

1. A description (string).

2. The code execution.

3. The assertion.

Now, let’s write a unit test for the above function using Vitest.

// sum.test.ts
import { describe, expect, it } from "vitest";
import { sum } from "./sum";

describe("sum function", () => {
  it("should return the sum of two numbers", () => { // 1. description
    const result = sum(2, 3); // 2. code execution
    expect(result).toBe(5);   // 3. assertion
  });

  // ... other test cases
});

// ... other describe blocks

Breaking it down:

• describe groups related test cases together. Here, we group everything about the sum function.

• it (or test) defines a single test case. In this example: “should return the sum of two numbers.”

• expect makes the actual assertion. It checks if the result from sum(2,3) equals 5.

When you run this test, Vitest will quickly execute it and show you whether the function passed or failed.

If the function works, you’ll see 1 passed in green. If it fails, the output will be red with details about what went wrong.

Integration Testing

Now that we’ve covered unit testing, let’s move one step up to integration testing. While unit tests focus on testing individual pieces in isolation, integration tests ensure those pieces work together as expected.

Think of it like assembling Lego blocks: each piece might work fine on its own, but when you connect them, something might not fit right. Integration testing helps you catch those issues early.

In simple terms, Integration testing checks how components and modules interact with each other.

Let’s say we have a React component that fetches user data from an API and displays it on the screen.
We’re no longer just testing one function – we’re testing how the component behaves when it calls an API, manages loading states, and renders data dynamically.

Here’s a simple example:

import { useEffect, useState } from "react";

const User = () => {
  const [users, setUsers] = useState<{ name: string; email: string }[]>([]);
  const [loading, setLoading] = useState(false);

  const fetchUsers = async () => {
    setLoading(true);
    try {
      const res = await fetch("https://api.escuelajs.co/api/v1/users");
      const data = await res.json();
      setUsers(data);
    } catch (e) {
      console.log(e);
    } finally {
      setLoading(false);
    }
  };

  useEffect(() => {
    fetchUsers();
  }, []);

  return (
    <>
      {loading ? (
        <h2>Loading...</h2>
      ) : (
        <div>
          {users.map((user, index) => (
            <p key={index}>
              {user.name}: {user.email}
            </p>
          ))}
        </div>
      )}
    </>
  );
};

export default User;

This component does a few things:

• Calls an external API when the component mounts.

• Sets a loading state while fetching data.

• Renders the fetched users on the screen once the data is ready.

Now, our job is to test the complete flow, from the API call to the rendered UI, using Vitest and React Testing Library.

Here’s what the test file looks like:

import { render, screen, waitFor } from "@testing-library/react";
import User from "../components/User";
import { describe, test, expect } from "vitest";

describe("User Component", () => {
  test("fetches and displays users successfully", async () => {
    render(<User />);

    // 1. Initially shows loading
    expect(screen.getByText("Loading...")).toBeInTheDocument();

    // 2. Wait for API response and UI update
    await waitFor(() => {
      expect(
        screen.getByText("Ajay Yadav: ajay.yadav@example.com")
      ).toBeInTheDocument();
      expect(
        screen.getByText("Jane Smith: jane.smith@example.com")
      ).toBeInTheDocument();
    });

    // 3. Loading should disappear
    expect(screen.queryByText("Loading...")).not.toBeInTheDocument();
  });
});

This test looks simple, but it covers the entire flow of our component. Let’s understand it step-by-step:

• Render the component: Render the <User /> component inside the test environment.

• Check the loading state: As soon as the component mounts, the “Loading…” text should appear, indicating that data is being fetched.

• Wait for the data to load: Since the API call is asynchronous, use waitFor() to wait until the users are fetched and displayed.

• Verify the data: Once the API resolves, check if the user names and emails are correctly rendered on the screen.

• Confirm loading disappears: Finally, ensure that the “Loading…” text is removed once the data is displayed, confirming a proper state update.

You can also test how your component behaves when the API fails. For example, you can mock the fetch() call to reject and then verify if an error message appears on the screen.

Vitest and React Testing Library make it easy to mock responses and simulate both success and failure cases, helping you ensure that your app handles real-world scenarios gracefully.

End-to-End Testing

Now that we’ve seen how integration testing ensures that different components work together, let’s move to the third layer, End-to-End (E2E) testing.

While unit and integration tests run in isolated or simulated environments, E2E tests mimic how real users interact with your app.

They open a browser and perform actions like clicking buttons, typing in fields, and verifying what appears on the screen, exactly like a real person would.

Think of E2E testing as putting your entire app on stage and watching if it performs flawlessly in front of the audience. In simple words, E2E testing verifies the full user journey from start to finish.

Let’s take a common example, a login flow. As a developer, you’ve probably built dozens of login forms, but how do you know if they truly work under real conditions? That’s where E2E testing comes in.

Using tools like Playwright or Cypress, you can perform effective E2E testing. Both Playwright and Cypress are powerful tools and are popular among developers.

We can simulate a real browser, fill out the login form, submit it, and confirm that the user is redirected to the dashboard. Here’s what a simple E2E test looks like using Playwright:

// tests/login.e2e.ts
import { test, expect } from "@playwright/test";

test("should login successfully", async ({ page }) => {
  // 1. Visit the login page
  await page.goto("http://localhost:3000/login");

  // 2. Fill in the form
  await page.fill('input[name="email"]', "user@example.com");
  await page.fill('input[name="password"]', "password123");

  // 3. Click login button
  await page.click('button[type="submit"]');

  // 4. Wait for navigation and verify success message or dashboard
  await expect(page).toHaveURL("http://localhost:3000/dashboard");
  await expect(page.getByText("Welcome back!")).toBeVisible();
});

Let’s understand what’s happening here step-by-step:

• Visit the page: The test opens your web app in a real browser. It navigates to http://localhost:3000/login.

• Simulate user input: Playwright fills in the email and password fields, just like a real user typing into the form.

• Perform actions: It clicks the login button, triggering all the same logic your frontend and backend would normally handle.

• Verify the outcome: Once the user logs in, check if the URL changes to /dashboard and whether a welcome message appears on the screen.

That’s it, you just automated your first user journey from login to dashboard. Both frameworks achieve the same goal, ensuring your app behaves correctly in a real browser, not just in isolated tests.

AI-Augmented Testing

As testing evolves, a new layer has emerged that is AI-Augmented QA. This isn’t just another tool in the developer’s toolkit. It’s a complete transformation in how software quality is managed.

Traditionally, testing has been a manual process. Engineers wrote, maintained, and updated test cases whenever the product changed. But with AI entering the scene, that manual burden is decreasing.

AI models can now analyze your codebase, understand logic, and generate relevant test cases almost instantly, covering edge cases you might never think of. Tools like GitHub Copilot and CodiumAI already assist in generating smart test suites, while continuously learning from your coding style and past patterns.

Beyond code suggestions, complete AI QA platforms are changing automation itself. For example, an AI QA agent like Bug0 can adjust to UI changes automatically. If a button label or DOM structure changes, its self-healing tests find elements visually instead of depending on fixed selectors.

It also produces real-time test reports with detailed logs and video recordings, helping developers pinpoint UI or data changes causing failures.

With CI/CD integrations like GitHub or GitLab, it can automatically start and validate test runs for every pull request, updating PR checks just like a human QA engineer would.

While AI-assisted testing is powerful, it’s not a full replacement for human judgment. Developers still play a vital role in the following ways:

• AI can generate test cases, but humans must decide what truly matters for business logic and user experience.

• Reviewing AI-generated tests to ensure they are relevant and to avoid false positives.

• Interpreting failures contextually means understanding whether a test failure indicates a real bug or an expected change.

• Maintaining ethical and data-safe workflows involves avoiding the exposure of sensitive data when using cloud-based AI tools.

When used responsibly, AI becomes a testing partner, automating the tedious tasks while leaving creative problem-solving, decision-making, and domain understanding to developers.

This shift marks the beginning of intelligent, autonomous QA. AI isn’t just automating repetitive testing, it’s transforming the process into a continuous, adaptive feedback loop, capable of predicting and resolving failures on its own.

In the coming years, expect testing to evolve into a collaborative process between human engineers and AI copilots, ensuring every release is not just faster, but smarter and more reliable than ever before.

Future of JavaScript Testing

JavaScript testing is changing faster than ever. A few years ago, developers had to deal with tons of testing libraries and confusing setups. Now, things are becoming much more unified, smarter, and easier to work with.

In the future, testing will move from being reactive to proactive. That means instead of catching bugs after they happen, tools will be smart enough to predict and prevent them before they appear.

With AI-powered test generation and real-time monitoring, every commit you make could be automatically checked for reliability and performance without you even running a command.

Frameworks like Vitest, Playwright, and React Testing Library will still be the core tools, but the real progress will come from how they integrate and learn.

We’ll also see tighter CI/CD integrations, where pipelines can automatically adjust based on your test coverage and code risk. Testing won’t feel like an extra step anymore, it’ll become a natural part of development, powered by both human logic and machine intelligence.

In short, the future of JavaScript testing is about speed, intelligence, and automation. A world where developers spend more time building and less time debugging.

Conclusion

Testing isn’t just about preventing bugs, it’s about building confidence. Confidence that your code works, your features scale, and your users have a seamless experience.

Whether it’s unit tests ensuring logic, integration tests validating flow, E2E tests simulating real behavior, or AI-enhanced automation managing it all. Testing is the silent force that makes great software possible.

As a developer, understanding how testing fits into your workflow is no longer optional. Rather, it’s a skill that sets you apart. The more you test, the better you code and the faster you ship with peace of mind.

So, the next time someone says writing tests isn’t your job, you’ll know the truth: Testing isn’t extra work. Instead, it’s part of writing better, more reliable software.

Before We End

I hope you found this article insightful. I’m Ajay Yadav, a software developer and content creator.

You can connect with me on:

• Twitter/X and LinkedIn, where I share insights to help you improve 0.01% each day.

• Check out my GitHub for more projects.

• I also run a YouTube Channel where I share content about careers, software engineering, and technical writing.

See you in the next article — until then, keep learning!

Performing repetitive tasks while testing APIs is stressful and time-wasting. For example, the process of retrieving, copying and pasting new authentication tokens for use in Postman is repetitive. You can simplify this process by using Postman scripts to store auth tokens and then reuse them without repeating the copy and paste steps.

To practice along with this guide, you should have:

• The Postman API client installed on your computer

• Experience in making API requests with Postman

• A backend application that uses JWT authentication and has its documentation in your Postman client

If you don’t have a backend application setup, I created one that you can clone from GitHub at orimdominic/freeCodeCamp-postman-api-jwt.

By the end of this article, you should be able to simplify the process of obtaining and reusing authentication tokens across your API requests. You should also have a practical understanding of some scripts necessary for use in other areas of software testing with Postman.

Table of Contents

• Table of Contents

• What are Postman Scripts?

• How to Simplify Your JWT Authentication Process

  • Authenticate to Get the Token

  • How to Save the Token in a Variable with a Postman Script

  • How to Use the Variable in a Request

• Next Steps

What are Postman Scripts?

Postman scripts are blocks of JavaScript code that you can write and run within the Postman API client to automate and enhance API testing workflows. You can use Postman scripts to add code to run before and after API requests. These scripts can be used to:

• Add logic and process data from API requests

• Write test assertions for API responses

• Run automated tests on API endpoints

You can find Postman scripts under the Scripts tab of an API request. Code written in the Pre-request tab runs before the request is made and code written in the Post-response tab runs after the response is made.

How to Simplify Your JWT Authentication Process

In summary, you will carry out the following steps to achieve the objective of this tutorial:

1. Authenticate to get the token

2. Save the token in a collection variable with Postman scripts

3. Use the variable in an API request

Authenticate to Get the Token

To get started, carry out the following steps:

1. Start your backend application and make sure it is running successfully.

2. Open up your Postman application and go to the API request for signing in to get a JWT.

3. Make an API request to the sign in endpoint and take note of the JSON response schema.

The highlighted part of the image above shows the JSON response from a successful sign in request. In the response schema, the auth token to be used for authorization is in the data.token field. You will use Postman scripts to store this token in a variable and then use the variable in the Authorization header of requests that require authorization.

How to Save the Token in a Variable with a Postman Script

In Postman, click on the Scripts tab next to the Body tab. If the Postman application window is small, you may need to click a dropdown to see it. Next, click on the Post-response tab. In the text area to the right, you will write the script to capture the auth token from the response and store it in a Postman variable. Copy the JavaScript code below and paste it into the text area.

if (pm.response.code == 200) {
    const token = pm.response.json().data.token
    pm.collectionVariables.set("auth_token", token)
}

Postman scripts use the pm identifier to access and modify information in the Postman environment. The script above uses pm to first ensure that the request was successful by checking if the response status code is 200.

Inside the conditional statement, pm.response.json().data.token is used to get the authentication token from the JSON response and store it in a collection variable called auth_token. If auth_token doesn’t exist already, it is created and its value is set to the value of token. If it exists already, its value is replaced.

To confirm that auth_token has been set, click on the name of the collection (labelled 1 in the screenshot above) and then click on the Variables tab (labelled 2 in the screenshot above). Next, instead of repeatedly copying the token and pasting it in the Authorization header of your requests, you will use auth_token in the Authorization header of your requests.

How to Use the Variable in a Request

Reference the collection variable in the Authorization header by surrounding it with double curly braces {{auth_token}}. When you make an API request, Postman will use the value referenced by {{auth_token}} as the Authorization header.

If another authentication request causes the value of auth_token to be updated, you no longer need to copy the new auth token. The script in the post-response tab will update the auth_token value and you can go on with making API requests smoothly. No need for repeatedly copying and pasting - Don’t Repeat Yourself (DRY).

Next Steps

In this tutorial, you have learnt how to use Postman scripts to set environment variables in Postman. You have also learnt how to eliminate the process of repeatedly copying and pasting auth tokens for use in API requests.

For guides on writing assertion tests for your APIs, check out the Test API Functionality and Performance in Postman guide by Postman.

But managing disk space can be a quite a challenge, as WSL uses virtual hard disks that do not automatically free up unused space.

This tutorial will guide you through the process of manually compacting your WSL virtual hard disks. We’ll automate this task using a PowerShell script, ensuring that your WSL environment remains efficient and clutter-free.

Reclaim Your Space

WSL uses a virtualization platform to install Linux distributions on your Windows system. Each distribution you add gets its own Virtual Hard Disk (VHD), which uses the ext4 file system (common in Linux). It’s saved on your Windows drive as an ext4.vhdx file.

Key issues here:

• Inefficient storage: by default, VHD files do not reclaim unused space. This means that when you delete a file in WSL, the associated disk space isn’t immediately freed up.

• Disk space consumption: due to that inefficient storage, the VHD files can grow large thanks to that accumulated data, especially if you’re a WSL heavy user.

• Need for maintenance: you may not know that you need to compact your VHD files in order to reclaim disk space.

If you notice that your free disk space is shrinking even after deleting files and apps, WSL might be the reason. This tutorial will help you keep your WSL and Windows environment running smoothly.

Table of Contents

• Part 1: How to Manually Compact Your Virtual Hard Disk

  • Prerequisites

  • Step 1: Verify your WSL version and status

  • Step 2: List all installed distributions verbosely

  • Step 3: Locate your linux Virtual Hard Drive (VHDX) path

  • Step 4: Shut down all WSL instances

  • Step 5: Compact the Linux virtual hard drive using DiskPart

  • Step 6: Restart WSL and verify

• Part 2: How to Make Your Life Easier with Automation

  • Prerequisites

  • Step 1: Find out installed WSL2 distributions

  • Step 2: Select a distro to compact

  • Step 3: Locate the ext4.vhdx File

  • Step 4: The confirmation prompt

  • Step 5: Shut Down WSL and compact

  • Step 6: Run a DiskPart script

Part 1: How to Manually Compact Your Virtual Hard Disk

Let's start by going through the process manually. This section will guide you through checking your WSL version and associated Linux distributions, finding VHD files, shutting down WSL, and compacting the virtual disk.

Prerequisites

• Windows 10 (20H1/2004+) or Windows 11 with WSL2 installed

• The PowerShell or Command Prompt running as Administrator (from the Windows menu, right click the icon and choose run as Administrator).

Step 1: Verify your WSL version and status

First, make sure you’re running on WSL version 2 (commonly referred as WSL2). The first version is outdated and WSL2 provides significant improvements. Open PowerShell (as Admin) or Command Prompt (as Admin) and run:

wsl -v

wsl --status

These commands display the WSL client version and whether your default distro is using WSL 2. Here’s the output of the wsl -v command:

And here’s the output of the wsl --status command.

Step 2: List all installed distributions verbosely

To see a detailed list of your WSL distributions (including which version each uses), run:

wsl.exe --list --verbose

Above you can see the output of the WSL --list --verbose command.

Look for your distro name (for example, “Ubuntu”) and note its WSL version. If it shows “Version 2”, you can proceed with compaction.

Step 3: Locate your linux Virtual Hard Drive (VHDX) path

Each WSL distro’s files live in a VHDX file on your Windows drive. To find the path for any Linux distribution, use this PowerShell snippet:

(Get-ChildItem `

-Path HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss `

| Where-Object { $_.GetValue("DistributionName") -eq 'YOUR_DISTRO_NAME' }

).GetValue("BasePath") + "\ext4.vhdx"

Where you replace YOUR_DISTRO_NAME with yours (Ubuntu, Debian, Kali-linux..). Here’s the output of the command shown above in PowerShell (the filepath has been anonymized):

This command reads the registry key for your linux distribution, then appends “\ext4.vhdx” to build the full file path.

Make sure you copy the whole line. We will need it in later stages.

Step 4: Shut down all WSL instances

Before you can compact any virtual drive, make sure WSL is completely shut down. In PowerShell or Command Prompt (still as Administrator), run:

wsl.exe --shutdown

Step 5: Compact the Linux virtual hard drive using DiskPart

You successfully gathered all the needed information (about your system, the available distros, and their VHDX filepath) to proceed with the main task. In this step, you actually proceed with the compaction.

1. Launch DiskPart in the same elevated (admin) shell:

diskpart

DiskPart will open in a new window. It's a Windows command-line tool for managing disk partitions. Be cautious when using it, as incorrect actions can cause serious data loss.

1. In the DiskPart prompt, select the VHDX file you found earlier. Replace the path as displayed below with your actual path (the line you copied before):

select vdisk file="C:\Users\username\AppData\path\to\ext4.vhdx"

The above is the output of the select vdisk command (some data has been anonymized).

1. Attach the virtual drive in read-only mode:

Compaction only needs to scan the empty blocks in the file, not write to the Linux filesystem inside. Read-only mode guarantees that DiskPart only inspect the blocks for zero‐trimming without any chance of damaging or altering your Linux filesystem.

attach vdisk readonly

You can see in the screenshot above that the virtual hard drive has been successfully attached.

1. Compact the disk:

This frees up the disk space by shrinking the physical size of the .vhdx file to match the actual used data inside.

compact vdisk

This operation might take a while. When you see the “DiskPart successfully compacted the virtual disk file” message, proceed with the next step. In the image below, the virtual hard drive has been successfully compacted.

1. Detach the virtual drive:

detach vdisk

There you go – the virtual hard drive has been successfully detached.

This command releases any locks on the virtual drive and effectively dismounts it. If you don't use this command, the file remains "in use," preventing WSL (or you) from accessing it until you reboot or manually force it closed.

6. Exit DiskPart:

exit

Step 6: Restart WSL and verify

Back in PowerShell or Command Prompt, you can relaunch your distro:

wsl -d YOUR_DISTRO_NAME

You can even try the Unix df -h command in your WSL prompt to check your new available disk spaces.

Congrats, you just achieved a maintenance task that can free up lots of gigabytes of storage over time. Now, it’s time to automate.

Part 2: How to Make Your Life Easier with Automation

Since it's often hard to remember exactly where your WSL distro is located and you probably won't use it very often, this PowerShell script will automate the entire process we covered in part 1. Here's a preview of the steps you'll follow:

• Detect installed WSL distributions.

• Select one (and handle the cases there are more than one).

• Locate the corresponding ext4.vhdx file.

• Shut down WSL and use DiskPart to compact the virtual disk.

Prerequisites

• Windows 10 (20H1/2004+) or Windows 11 with WSL 2 enabled.

• PowerShell or Command Prompt (as Administrator).

You’ll also need a code editor. The Windows notepad is enough for completing this task. You can also use an IDE (Integrated Development Environment) like VS Code or an ISE (Integrated Scripting Environment) like PowerShell ISE (included with Windows).

To test the script, download it from GitHub. Open an elevated PowerShell or Command Prompt and navigate to the script’s folder. With just the command below, you will be able to run it and free up some disk space:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\wsl_compactor.ps1

Step 1: Find out installed WSL2 distributions

One of the main challenges is to find the Linux distributions available on the host system. Let’s check the first block and see what’s it about:

$lxssKey = 'HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss'
\(distros = Get-ChildItem \)lxssKey | ForEach-Object {
    \(p = Get-ItemProperty \)_.PSPath
    [PSCustomObject]@{
        Name     = $p.DistributionName
        BasePath = $p.BasePath
    }
}

WSL records each distribution under this windows registry key:

HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss

Each subkey has two important values:

• DistributionName (for example, Ubuntu)

• BasePath (This is where the distribution files are stored. It’s the directory that contains the ext4.vhdx file.)

The script uses Get-ChildItem and Get-ItemProperty to enumerate these subkeys and build a list of available Linux distributions.

if ($distros.Count -eq 0) {
Throw-And-Exit "No WSL distros found in the registry."
}

If no distributions are found, the script terminates and prints this error message on the terminal: "No WSL distros found in the registry.”

Step 2: Select a distro to compact

Here, the process has two steps:

• If multiple distros are found, it displays all the distros with a numbered menu and prompts you to choose one:

if ($distros.Count -gt 1) {
    Write-Host "Multiple distros detected. Please choose one:`n"

    for (\(i = 0; \)i -lt \(distros.Count; \)i++) {
        Write-Host "[\((\)i+1)] \((\)distros[$i].Name)"
    }
    \(selected = \)distros[[int]$choice - 1]
}

The computed menu will look like this:

Multiple distros detected. Please choose one:

[1] Ubuntu 20.04

[2] Debian

[3] Alpine

• If only one distribution is found on the host system, the script selects it automatically:

else {
\(selected = \)distros[0]
}

When setting up a distribution, whether chosen manually by the user or selected automatically, the important information is the path to each distribution's virtual hard disk. This path is saved in two main variables: 'distro' (which identifies the specific distribution) and 'basePath' (which shows where its virtual disk is located).

\(distro = \)selected.Name
\(basePath = \)selected.BasePath

Write-Host "`nSelected distro: $distro" -ForegroundColor DarkYellow
Write-Host "BasePath: $basePath"

The lines above display an output that looks like this:

Selected distro: Ubuntu (or any other distro)
BasePath: C:\Users\<User_name>\AppData\Local\Packages\…

Like for all other steps, it’s important to consider the case of something going wrong, by throwing an error and exiting the program:

if (-not (Test-Path $basePath)) {
Throw-And-Exit "BasePath '$basePath' does not exist on disk."
}

Step 3: Locate the ext4.vhdx File

In the first step, we collected the information we need about the available distributions and where they are stored on the Windows system. By choosing an entry (either manually or automatically), we can find the correct file. Sometimes, the ext4 file is located between the base path and a LocalState folder. This script manages both situations. It builds the usual locations where the file can be found.

They look like this:

• $BasePath\ext4.vhdx

• $BasePath\LocalState\ext4.vhdx

This can translate into something like this on your system (option 1):

C:\Users\Alice\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc\ext4.vhdx

or like this (option 2):

C:\Users\Alice\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc\LocalState\ext4.vhdx

(You might find out that your WSL2 distro is located in some other directory than “Packages” – but don’t worry, your BasePath will match the correct folders).

The idea is to build the two possible path options:

$possible = @(
Join-Path $basePath 'ext4.vhdx'
Join-Path $basePath 'LocalState\ext4.vhdx'
)

And pick the first one that actually contains the file:

\(vhdx = \)possible | Where-Object { Test-Path $_ } | Select-Object -First 1

Again, we throw an error message if no suitable file is found:

if (-not $vhdx) {
Throw-And-Exit "No ext4.vhdx found under '$basePath'."
}

Step 4: The confirmation prompt

Disk management tools require caution and you need to understand the potential consequences of your actions. A confirmation prompt is always a good safeguard to prevent accidental data loss or unwanted system changes.

Before proceeding, the script shows you:

• a Distro name

• its BasePath

• the VHDX file path

Write-Host "`nAbout to compact this WSL distro:" -ForegroundColor Magenta
Write-Host " Distro : $distro"
Write-Host " BasePath : $basePath"
Write-Host " VHDX file: $vhdx`n"

It then prompts “Are you sure you want to proceed? (Y/N)”:

Write-Host "Are you sure you want to proceed? (Y/N) " -ForegroundColor DarkCyan -NoNewline

# Then read the response
$answer = Read-Host

You’re then prompted to Type Y (case-insensitive) to continue or anything else to cancel.

if ($answer.ToUpper() -ne 'Y') {
    Write-Warning "Operation canceled"
    exit
}

For the two steps above, I had to use a trick to print the question in color but a simple option (without colors) could be:

if ((Read-Host 'Are you sure you want to proceed? (Y/N)').ToUpper() -ne 'Y') { 
    Write-Warning 'Operation canceled'
    exit 
}

Step 5: Shut down WSL and compact

Before proceeding to the DiskPart utility, it’s important to stop all running WSL instances. Pass the shutdown command directly in PowerShell.

Write-Host "Shutting down WSL…" -ForegroundColor Cyan
wsl.exe –shutdown

A common mistake is to forget to launch PowerShell or the Command Prompt with Administrator rights. You can prevent this case with a message:

if ($LASTEXITCODE -ne 0) {
     Throw-And-Exit "Failed to shut down WSL (exit code $LASTEXITCODE). Are you running as Administrator?"
}

Step 6: Run a DiskPart script

Building the script:

The process is the same as in the manual part, but this time, we ‘inject’ the ready-to-go DiskPart commands into the script.

$dpScript = @"
select vdisk file="$vhdx"
attach vdisk readonly
compact vdisk
detach vdisk
exit
"@

Before launching, there are two steps you need to take:

1. The PowerShell script writes the lines above to a temporary file:

$tempFile = [IO.Path]::GetTempFileName()
Set-Content -LiteralPath \(tempFile -Value \)dpScript -Encoding ASCII

This is the equivalent to the commands passed in the manual part:

select vdisk file="C:\…\ext4.vhdx" # full path to the vdisk file

attach vdisk readonly

compact vdisk

detach vdisk

exit

1. Compacting can take a while, especially if you’ve never de-cluttered your virtual drive before. It’s wise to show a warning before proceeding:

Write-Host "Running DiskPart to compact the VHDX. Be patient, this might take a while..." -ForegroundColor Cyan

Invoke DiskPart:

# Run DiskPart with the script saved to the temporary file and process each output line as it arrives
diskpart /s $tempFile | ForEach-Object {
    # Grab any "NN percent" type message from the line
    if ($_ -match '(\d+)\s+percent') {
        # Only print when the percentage actually changes
        Write-Host "\((\)Matches[1])% completed"
    }
    else {
        # Just echo all over line-types, verbatim
        Write-Host $_
    }
}

Several points to note here:

• It runs diskpart /s $tempFile: DiskPart reads and executes commands from the temporary file into the PowerShell loop for on-the-fly processing.

• For a better user-experience: the snippet below does the trick of filtering out repeated status values by comparing \(pct with the sentinel \)lastPct, and only writing new lines when they differ.

How?

Before entering the loop, we initialize:

$lastPct with -1
$lastPct = -1 # We initiate a sentinel value

We are having a guaranteed “first” value that no real percent (0–100) will equal. That way, as soon as you see the first 0 percent, 10 percent, or whatever, it differs from -1.

Then:

if ($_ -match '(\d+)\s+percent') {
    # Print only when the percentage changes
    Write-Host "\((\)Matches[1])% completed"
}

This guarantees that on the very first percentage update (say “0 percent” or “10 percent”), \(pct –ne \)lastPct will be true, so it emits the first line. Afterwards, $lastPct holds the last real percentage, and it only prints again when a new, different progress percentage comes in.

The output looks more clean:

10% completed
20% completed
…

Otherwise, it’ll flood the screen with dozens of identical “20 percent completed” (for example) updates.

Of course we handle the case of other values (non percent lines) normally:

else {
    # non-percent lines: print verbatim
    if ($_ -match '\S') {
        Write-Host $_
    }
}

Once the process is done, don’t forget to clean up the tempfile.

Remove-Item $tempFile -ErrorAction SilentlyContinue

By the end of the process, you should see something like this

Leaving DiskPart...

Okay, that’s it for the scripting! If you collected all the snippets so far, just save them with a .ps1 file extension, or download the full example from this GitHub repository.

Usage:

You now have a complete understanding of what's happening. Ready to get started? In Windows, open the Command Prompt or PowerShell as an administrator.

• Navigate to the directory containing your .ps1 script.

• Execute the script with the following command, replacing <File_name_here> with your actual file name:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\<File_name_here>.ps1

The -NoProfile -ExecutionPolicy Bypass parameters launch PowerShell in a clean, unrestricted environment that ignores user-specific settings and allows script execution without security restrictions. Don’t worry, in this case, it’s okay to do that.

Wait…wait…wait...

Well bravo! You’ve just reclaimed all that unused space inside your WSL2 (almost) hands free!

Now, you can take it a step further by modifying this script to run completely automatically, without the confirmation prompt (step 4). You can also schedule it as a regular maintenance task using a task scheduler:

schtasks /create /tn "Schedule_name" /tr "powershell.exe -ExecutionPolicy Bypass -File C:\path\to\script.ps1" /sc monthly /d 15 /st 09:00

This is an example for a monthly execution where /d 15 means 15th of each month and /st 09:00 is a start time set at 9 am.

That's it! Remember, regular maintenance, whether you do it manually or automatically, is essential to prevent unnecessary disk space usage and ensure a smooth experience with WSL.

Thanks for reading!

You can find a list of my current projects on GitHub.

An embedded system typically goes through a simple but powerful cycle:

1. Sense – Gather information from the environment using sensors.

2. Process – Use software logic to decide what to do with the data.

3. Act – Trigger a response, like turning on a motor or lighting an LED.

Each project begins with a use case – a specific goal like brewing coffee or controlling a car’s fuel injection. From that, engineers define system requirements, which are split into:

• Hardware (for example, microcontrollers, sensors, actuators)

• Software (what we call embedded software)

This handbook focuses on the software side of embedded systems: how we write code to make embedded systems intelligent. Embedded software runs on resource-constrained devices like microcontrollers, which may have just a few kilobytes of memory. The software might need to be highly efficient, reliable, and often capable of working in real-time.

But embedded software isn't just about writing code – it’s also about understanding:

• How hardware works

• How to manage memory and power

• How to handle timing and communication

• How to build robust, fail-safe systems

While embedded systems development isn’t typically research-focused in most industry roles, it demands a broad skill set, from low-level programming to system-level design. What makes this field especially exciting is how it brings together diverse domains like machine learning, digital signal processing (DSP), and control systems, all of which can be applied directly in real-world devices.

In this article, I’ll give you:

• A high-level overview of what embedded software involves

• Key concepts every developer should know

• A tour of commonly used tools and frameworks

• Resources to help you learn and understand basics.

Whether you're just curious or planning a career in embedded systems, this guide is your launchpad.

Table of Contents

• HW Layer: Microcontroller

• Firmware Design and Tools

• Tools and Concepts for Embedded Development

• Bare Metal, RTOS, and Embedded Operating Systems

• Designing Drivers for Embedded Systems

• Security in Embedded Systems

• Debugging and Forensics in Embedded Systems

• Automation and Testing in Embedded Systems

• Where to Go from Here

This article offers a broad overview of embedded firmware development, but it doesn’t cover every aspect, particularly advanced software architecture frameworks or comprehensive lists of open source software and tools. Where appropriate, I have included external resources that were valuable in expanding my own understanding.

Prerequisites

You don’t need to be an expert to follow this guide, but some prior knowledge will help you get the most out of it:

• Basic C or C++ programming**:** Familiarity with functions, pointers, and memory concepts is helpful.

• Computer architecture fundamentals**:** Understanding what a CPU does, how memory works, and basic instruction execution will make embedded concepts clearer.

• Electronics basics (optional): Knowing how sensors, resistors, or microcontrollers interact at a circuit level is useful but not mandatory.

• Comfort with the command line**:** Especially for working with build systems, compilers, and flashing tools.

This guide is ideal for students, engineers, or hobbyists looking to deepen their understanding of how software interacts with hardware in real-world systems.

With that, let’s start from the ground up, hardware. Throughout this guide, most examples will reference ARM Cortex-M microcontrollers, as they are among the most commonly used in the embedded world.

HW Layer: Microcontroller

One of the most important knowledge blocks in embedded firmware development is understanding how a microcontroller (MCU) works and how it connects to sensors, actuators, and other microcontrollers.

If you’re familiar with basic computer architecture (like instruction sets and memory organization), that knowledge translates well to embedded systems. In fact, Computer System Organization, often taught in computer science and electrical engineering programs, is a great foundation for understanding microcontrollers.

What is a Microcontroller?

A microcontroller is a compact computing unit that includes:

• A CPU (Central Processing Unit or Microprocessor)

• Memory (Flash and RAM)

• Peripherals (for I/O, timers, communication, and so on)

In essence, it's a tiny computer-on-a-chip, optimized for specific control tasks like reading sensors or driving motors.

By contrast, a microprocessor is just the CPU. It requires external memory and peripherals to function. Microcontrollers are self-contained and better suited for embedded applications.

For example, this reference manual for the STM32F4 series (from STMicroelectronics) provides detailed documentation on not just the CPU but each peripheral’s functionality and the register map.

Instruction Set Architecture (ISA)

A microprocessor executes a series of instructions defined by its Instruction Set Architecture (ISA). ISA as defined by ARM is a part of the abstract model of a computer that defines how the CPU is controlled by the software. The ISA acts as an interface between the hardware and the software, specifying both what the processor is capable of doing as well as how it gets done.

For example:

• ARMv7 – used in ARM Cortex-M3.

• ARMv7E – used in Cortex-M4 and M7.

Many vendors (for example, STMicroelectronics, NXP, TI) manufacture MCUs that support ARM ISAs but include their own peripheral sets. Understanding the ISA is essential for low-level coding and interpreting assembly instructions.

This ARMv7-M architecture reference manual provides more details on v7 Architecture.

Memory in Microcontrollers

Most microcontrollers typically feature two types of memory:

• Flash – Stores your code and read-only data.

• RAM – Used during program execution to hold:

  • The heap (for dynamic memory)

  • The stack

  • The .data and .bss sections (initialized/uninitialized global/static variables)

Later sections have resources that go deeper into memory mapping and how these regions interact during runtime.

Clock and Power Management

Microcontrollers are digital logic devices built from:

• Combinatorial logic – Logic gates that evaluate outputs instantly

• Sequential logic – Relies on clocks to move through states

The clock tree distributes timing signals across the CPU and peripherals. MCUs often support multiple clock sources (internal RC, external crystal, PLL), and use prescalers to drive components at different frequencies.

For power-sensitive applications, MCUs offer multiple low-power modes:

• Sleep – CPU off, timers and peripherals are mostly active, memory is retained

• Deep Sleep – CPU off, most clocks off, memory is retained, wake-up is slower than sleep, power consumption is lower than Sleep

• Standby – CPU off, few interrupts are active, everything else is powered down, memory is not retained. Lowest power mode.

These modes reduce power consumption by turning off clocks and disabling unused peripherals. Designing the system to switch in and out of low-power states effectively is a core skill in embedded software development.

This article talks about Clock Trees and Oscillators for the ARM Cortex microcontrollers.

Interrupts

Interrupts let MCUs react to asynchronous events, like button presses or sensor signals.

An interrupt temporarily pauses normal code execution to run a dedicated handler. After it’s serviced, the CPU resumes its previous task. They are vital for:

• Fast event response

• Reduced polling

• Efficient power use (for example, waking from sleep)

Timers

Timers are built-in peripherals used to track time or generate events.

Common uses are:

• Implementing software delays

• Creating precise software timers

• Waking up from low-power modes

Mastering timers helps with real-time behavior and precise event scheduling.

Communication Protocols

Microcontrollers often need to talk to other devices via built-in communication peripherals:

• UART (Universal Asynchronous Receiver/Transmitter): Serial communication between two devices, great for logs and debugging.

• I²C (Inter-Integrated Circuit): Two wire protocol for talking to sensors and EEPROMs.

• SPI (Serial Peripheral Interface): High Speed, full-duplex protocol for devices like Flash or displays.

• USB (Universal Serial Bus): Complex but widely used for PCs, data acquisition and HID devices.

Here’s a figure showing multiple peripherals connected to a MCU:

DMA or Direct Memory Access is an important peripheral which can be used to transfer data to/from memory without CPU involvement. It improves performance and allows the CPU to perform other tasks or enter low power mode to reduce power consumption.

This article provides a good overview of the communication protocols I2C, UART and SPI.

We’ve now covered the essential building blocks of microcontroller hardware – from memory and clocks to interrupts and communication buses.

Next, we’ll explore the software principles and tools that bring these microcontrollers to life, including compilers, debuggers, and embedded development frameworks.

Firmware Design and Tools

Designing Embedded Software

Even though embedded systems operate under unique hardware constraints, software design principles are still crucial. Applying them thoughtfully becomes even more important when memory, CPU cycles, and responsiveness are limited.

Most Embedded firmware projects begin with a structured design approach:

1. Understand the problem statement

2. List assumptions

3. Define use cases

4. Define system and software requirements

5. Create high-level architecture

6. Drill down to detailed design and implementation

If you’re new to software design, check out my article on design principles.

Here’s a figure showing the five blocks of software design:

Using Design Patterns

Once you're designing individual components, design patterns help you write scalable and maintainable code. Here are some common patterns in embedded systems:

• Publisher-Subscriber (Observer) – Useful for decoupling event producers and consumers (for example, sensor data being broadcast to multiple modules).

• Singleton – Ensures only one instance of a module or resource manager exists (for example, for drivers or HAL layers).

• Adapter – Translates between incompatible interfaces (for example, wrapping platform-specific code into a portable application layer).

• State Machine – Represents system behavior as transitions between states (for example, Bluetooth states: IDLE → SCANNING → CONNECTING → CONNECTED → DISCONNECTED).

Design patterns often need to be adapted for memory and timing constraints, but the core concepts remain highly relevant.

There are lot of great resources on design patterns – here are a few that helped me:

1. Book: Head-first Design patterns - A great book to get understand the concept of design patterns

2. Book: Design Patterns: Elements of Reusable Object-Oriented Software

3. Course: Object-Oriented Programming and Design Patterns in C#

4. Article on HSM: Hierarchical State Machine Overview (Barr Group)

Programming Languages for Embedded Systems

While any language can theoretically be used if it compiles to machine code, in practice, three dominate the embedded world:

• C – The industry standard. Provides deterministic behavior and low-level access, making it ideal for memory and timing-sensitive code.

• C++ – Adds object-oriented features while maintaining control. Once considered risky in embedded due to synthesized code and overhead, it’s now widely adopted where systems benefit from abstraction and modularity.

• Rust – A memory-safe alternative gaining traction in safety-critical and open-source embedded development.

Languages like Python (via MicroPython or CircuitPython) are used in educational or prototyping contexts but are not suitable for production due to performance and memory overhead.

Some resources on programming languages that might be helpful to understand concepts:

1. The Embedded Rust Book

2. C Programming Language by K&R

3. Inside the C++ Object model – There are a lot of books and lectures on C++, but for embedded, understanding the object model benefits a lot.

Data Structures Matter

Embedded systems require careful data handling due to strict memory and timing constraints. Mastering core data structures is essential:

• Arrays – fixed-size data.

• Linked Lists – Common in software timers, queues.

• Stacks and Queues – Task scheduling, event management and data storage.

• Bitfields/Flags – Memory efficient state representation.

• Binary Trees – Used in routing tables or decision logic.

You'll often build event queues, circular buffers, or timer lists, all of which rely on these foundational structures.

There are a lot of resources for understanding data structures, but I have found this one to be helpful for learning and practicing: GeeksForGeeks DSA Tutorial. And here’s a full course on DSA if you want to dive deeper.

Bit Manipulation: A Core Embedded Skill

Unlike general-purpose software, embedded systems often require low-level access to registers and require precise bit control:

• Setting and clearing individual bits

• Using bitwise operators like AND (&), OR (|), XOR (^)

• Bit masking and shifting (<<, >>)

Mastering bit hacks is essential for writing hardware drivers or manipulating control registers.

This resource provides a good number of examples for bit manipulation: Stanford Bit Hacks.

Tools and Concepts for Embedded Development

Cross Compilation

Embedded code is compiled on a host (like your PC) for a target architecture using cross-compilers.

To do this, you need:

• A compiler (for example, arm-none-eabi-gcc for ARM Cortex-M) that compiles high level language code into Assembly language instructions.

• A linker to layout and combine object files.

• A Makefile or build system to organize and automate compilation, linking and binary creation.

Here’s an example to compile a main.c to create a main.elf that can be flashed on the device:

arm-none-eabi-gcc main.c -o main.elf

A Makefile is a script used by the make build automation tool to compile and link programs to create a binary. It defines how to build your program from source files, manages compilation order based on dependencies and defines commands to complete the build.

For example, lets write a Makefile for building a project for an ARM Cortex-M4 target that has three source files: a main.c, utils.c, and sensor.c

CC = arm-none-eabi-gcc
CFLAGS = -c -mcpu=cortex-m4 -mthumb -Wall -O2
LDFLAGS = -mcpu=cortex-m4 -mthumb
TARGET = main.elf
OBJS = main.o utils.o sensor.o
SRC = main.c utils.c sensor.c

$(TARGET): $(OBJS)
    $(CC) $(OBJS) -o $(TARGET)

main.o: main.c
    $(CC) $(CFLAGS) main.c

utils.o: utils.c
    $(CC) $(CFLAGS) utils.c

sensor.o: sensor.c
    $(CC) $(CFLAGS) sensor.c

clean:
    rm -f *.o *.elf

In the above makefile, here’s a description of the flags:

• -mcpu=cortex-m4: Targets the ARM Cortex-M4 processor.

• -mthumb: Enables Thumb instruction set, which is used by ARM Cortex-M series.

• -Wall: Enables all common warnings.

• -O2: Optimization level 2 for balance between performance and code size.

Makefiles can seem intimidating, but they’re just scripts that define how to build your program from source. Once you understand the basics, they’re a huge productivity booster.

A linker script tells the linker (ld) how to organize the program in memory where to place code, data, stack, heap, and so on. It's crucial for embedded systems because you're working with limited memory and specific memory-mapped hardware.

Here’s an example of a simple linker script for a STM32F4 microcontroller:

/* STM32F4 Cortex‑M4 Simple Linker Script */

ENTRY(Reset_Handler)

/* Define memory regions based on STM32F4 datasheet */
MEMORY
{
  FLASH (rx) : ORIGIN = 0x08000000, LENGTH = 1024K
  RAM   (rwx): ORIGIN = 0x20000000, LENGTH = 128K
}

/* Section layout */
SECTIONS
{
  /* Interrupt vectors and code go into Flash */
  .isr_vector :
  {
    KEEP(*(.isr_vector))    /* Keep vector table (reset, etc.) */
  } > FLASH

  .text :
  {
    *(.text*)               /* All code */
    *(.rodata*)             /* Read-only data */
    . = ALIGN(4)
    _etext = .             /* End of code (used for data init) */
  } > FLASH

  /* Initialized data: load from Flash, run in RAM */
  .data : AT(_etext)
  {
    _sdata = .            /* Start of .data in RAM */
    *(.data*)
    . = ALIGN(4)
    _edata = .            /* End of .data */
  } > RAM

  /* Uninitialized data (zero-filled) */
  .bss :
  {
    _sbss = .
    *(.bss*)
    *(COMMON)
    . = ALIGN(4)
    _ebss = .
  } > RAM

  /* Define stack end (top of RAM) */
  _estack = ORIGIN(RAM) + LENGTH(RAM);
}

Descriptions of the above file:

• MEMORY: Defines your microcontroller’s memory layout – 1 MB Flash and 128 KB SRAM.

• ENTRY(Reset_Handler): Sets the reset handler as the program entry point.

• .isr_vector and **.**text: Code sections placed in Flash. .isr_vector must use KEEP() so it's not removed during linking.

• .data : AT(_etext): Loads initialized variables from Flash but places them in RAM.

• **.**bss: Zero-initialized data, allocated in RAM

• _estack: Defines the initial stack pointer using the end of RAM.

Here are some sources to understand Makefiles, cross-compilation, and Linkers. And just note that using Makefile in a project is the best way to learn and master Makefiles:

1. Makefiles:

   • GNU Make Manual

   • Makefile Tutorial

   • In Pyjama Makefile Article

2. Linker Scripts:

   • Interrupt Blog on Linker Scripts

   • Intro to Linker Files – Medium

Flashing the Binary

Once you’ve compiled your code into a binary file, the next step is to flash it into the target microcontroller’s non-volatile memory via SWD (Serial Wire Debug) or JTAG. Flashing tools like OpenOCD, ST-Link, J-Link, or vendor-specific utilities manage this process.

What Is Flashing?

Flashing is the process of writing a compiled firmware image (typically a .bin or .hex file) into the microcontroller’s Flash memory. This enables the embedded system to retain and run your code even after power is removed.

The flashing tool communicates with the microcontroller over SWD or JTAG to:

• Halt the MCU (if needed)

• Access the internal flash controller

• Erase the relevant flash sectors

• Write the binary data to specific memory addresses

• Verify that the data was written correctly

OpenOCD (Open On-Chip Debugger) is a powerful, open-source utility that facilitates debugging and flashing of ARM-based microcontrollers. It supports a wide variety of hardware interfaces and microcontroller families, including STM32.

OpenOCD provides:

• Flashing capabilities for .elf, .bin, and .hex files

• Debugging via GDB (GNU’s open source debugger) integration

• Support for multiple debug probes (J-Link, ST-Link, CMSIS-DAP)

• Scripting via configuration files for board-specific and target-specific setups

A simple command to flash a binary using OpenOCD might look like this:

bashCopyEditopenocd -f interface/stlink.cfg -f target/stm32f4x.cfg -c "program main.elf verify reset exit"

This tells OpenOCD to:

• Use the ST-Link interface

• Load the STM32F4 target configuration

• Program main.elf into flash

• Verify it was written correctly

• Reset the MCU

• Exit the session

For a detailed walkthrough, check out: OpenOCD Deep Dive – Kickstart Embedded

Bare Metal, RTOS, and Embedded Operating Systems

When writing embedded software, you can approach the problem in three main ways, each with its own trade-offs:

1. Bare-Metal Programming

2. Real-Time Operating Systems (RTOS) (like FreeRTOS, Zephyr)

3. Embedded Operating Systems (like Embedded Linux)

The best choice depends on your use case, application’s complexity, hardware constraints, and real-time needs.

Most Modern 32-bit microcontrollers (for example, STM32, NXP, Renesas) come with vendor-provided development tools that include:

• HAL (Hardware Abstraction Layer) libraries

• Startup code and linker scripts

• Peripheral drivers

• Sometimes even middleware like USB, BLE, or file system stacks

These tools (like STM32Cube Config Tools) simplify setup and peripheral configuration, helping you get started quickly, without needing to write low-level code manually.

Benefits of HALs:

• Rapid prototyping and development

• Clean, reusable APIs for peripherals

• Great for onboarding and small teams

Drawbacks:

• Code bloat – HALs support many edge cases and configurations, which can inflate your binary size

• Extra latency – HAL often inserts unnecessary layers that reduce performance.

For performance-critical systems, developers often replace HAL drivers with custom, low-level implementations.

Bare-Metal Programming

Bare-metal programming is the most direct and lightweight approach. There’s no OS, and your code runs directly on the hardware with full control.

Typical setup includes:

• Include the correct header files, especially MCU and peripheral-specific headers provided by the vendor’s HAL (Hardware Abstraction Layer).

• Implement a main() function with an infinite loop (while(1))

• Perform all hardware initialization before entering the loop

• Use Interrupts to handle asynchronous events.

• Continuously check and control inputs/outputs inside the loop

This assumes your toolchain provides startup code and memory setup from the vendor.

#include "MCU_Header.h"

int main(void) {
    /* Initialize the MCU and the peripherals */
    init_clock();
    init_peripherals();

    /* runs in a loop forever */
    while (1) {
        // Task 1 : Read sensor data
        read_sensor(); 
        // Task 2 : Update the actuator based on the sensor data
        update_actuator(); 
    }
}

How does it run?

When the device powers on or resets, the startup code provided by the vendor is executed first. This code:

• Initializes the reset vector

• Copies initialized data from Flash to RAM

• Zeros out the .bss section (for uninitialized global/static variables)

• Calls your main() function

After calling main(), the system enters an infinite loop where your logic runs. The only other context switch occurs when an interrupt is triggered, briefly diverting control to an Interrupt Service Routine (ISR), after which it returns to the main loop.

When to use it:

• Simpler applications (for example, blinking LEDs, reading sensors)

• Ultra-low-power or ultra-low-latency needs

• When every byte of Flash and RAM matters

Pros:

• Minimal memory usage

• Maximum control

• Great for learning

Cons:

• No built-in task management or scheduling

• Can become hard to maintain for complex systems

This resource provides good details and example on Bare Metal Programming. For more details, this book is great as well: ARM Baremetal Ebook.

Real-Time Operating Systems (RTOS)

A Real-Time Operating System (like FreeRTOS, Zephyr) adds lightweight multitasking capabilities to your embedded application. It allows you to split your software into independent tasks that run concurrently and communicate through queues, semaphores, or message passing.

RTOS kernels often support different scheduling strategies like:

• Rate Monotonic Scheduling (RMS) – Tasks with shorter periods get higher priority

• Earliest Deadline First (EDF) – Tasks are prioritized based on impending deadlines

Example use cases:

• A drone where sensor data, motor control, and telemetry need to run in parallel

• A medical device where timing is critical for safety

• Rockets

Typical RTOS features:

• Task scheduling

• Timers

• Inter-task communication

• Interrupt handling integration

• Power management

Pros:

• Modular code structure with tasks

• Easier to scale as complexity grows

• Deterministic execution (when configured correctly)

Cons:

• Slightly higher memory footprint than bare-metal

• Learning curve for scheduling and priority tuning

RTOS Scheduling techniques are interesting – this part of the docs talks about Zephyr scheduling.

Embedded Operating Systems

Sometimes an embedded system is powerful enough to run a full-fledged OS like Embedded Linux, Android Things, or Windows IoT Core. This is common on devices with a display, networking stack, or file system.

It’s best used when the system requires multitasking, user interfaces, file systems, or network stacks, and when there’s plenty of processing power (for example, ARM Cortex-A).

Think of:

• Smart home hubs

• Automotive infotainment

• Industrial gateways

This table provides a high level methodology for choosing the right type of OS based on your application:

To understand OS fundamentals, this is a great book: Operating System Concepts and this is a great course: UC Berkeley: CS162.

So far, we’ve looked at how embedded applications are structured, whether using bare-metal loops, RTOS multitasking, or full operating systems. But regardless of which execution model you choose, your software ultimately needs to interact with the hardware.

This is where driver development comes in. Drivers form the crucial link between your code and the peripherals it controls, whether it's reading temperature, blinking an LED, or transmitting data over SPI. Let’s take a closer look at how to design robust, portable drivers for embedded systems.

Designing Drivers for Embedded Systems

When working with embedded software, one of the most practical and common tasks you’ll encounter is driver development.

A driver is a piece of software that enables the microcontroller (MCU) to interface with a hardware peripheral. This could be a temperature sensor, a motor controller, a display, or even a wireless module.

Drivers act as a bridge between your hardware and the application logic. They abstract away the raw register-level programming so that higher-level code can use clear function calls like read_temperature() or start_motor().

What Goes Into a Driver?

A typical embedded driver will include:

• Configuration – Setting up the peripheral with initial parameters (for example, baud rate for UART)

• Initialization – Preparing the peripheral for use, including enabling clocks and interrupts

• Calibration (if needed) – Adjusting the peripheral based on specific environment or use case

• Register Access – Reading from and writing to hardware registers (if applicable)

• Power Management – Enabling/disabling the peripheral to save power or putting the peripheral into a low power mode

• Interrupt Management – Handling asynchronous events triggered by the peripheral

Here’s a simplified view of a sensor driver API:

void sensor_init(void);
void sensor_calibrate(void);
float sensor_read_temperature(void);
void sensor_sleep(void);
void sensor_write(uint8_t reg, uint8_t value); // Assumption : 8 bit register address and 8 bit data value

The actual implementation might involve:

• Register definitions from the peripheral’s datasheet

• Bit manipulations for control and status registers

• Interrupt Service Routines (ISRs)

• Timing and delay management

Platform Abstraction: Why It Matters

One of the most important principles in driver design is decoupling the application from the platform. This makes your code easier to:

• Port to different MCUs

• Adapt for similar hardware (for example, different sensor models)

• Test across simulated or real environments

Platform-Agnostic Design Example (in C++) :

Let’s say you're writing a driver for a temperature sensor:

// Abstracts the HW platform on which the sensor driver is being written
class TemperatureSensorPlatform {
public:
    void i2cInit(void);
    void i2cWrite(uint8_t reg, uint8_t value);
    uint8_t i2cRead(uint8_t reg);
};

// Creates a generic Temperature sensor driver interface
class TemperatureSensor {
public:
    virtual void init() = 0;
    virtual float read() = 0;
    virtual void sleep() = 0;
};

You can implement this interface differently for a specific type of temperature sensor and also add the platform support for the HW platform you are writing the driver on for example STM32.

class TempSensorTMP117 : public TemperatureSensor {
public:

    TempSensorTMP117(TemperatureSensorPlatform platform) : 
    _platform(platform)
    TemperatureSensor()
    {}

    void init() override {
        // TMP117-specific register configuration
    }

    float read() override {
        // Read ADC value and convert
        return 25.4f;
    }

    void sleep() override {
        // Put sensor in low-power mode
    }
private:
    TemperatureSensorPlatform _platform; // Implements the I2C driver for STM32
};

Your application code now depends on the TemperatureSensor interface and Temperature Sensor Platform passed in the constructor making it portable and testable across temperature sensors and HW platforms.

One of my previous articles provides details on how to interface a sensor and how to design a driver for it.

Designing robust and modular drivers helps your firmware interact seamlessly with hardware, but in today’s connected world, that’s only part of the challenge. As embedded devices increasingly communicate with other systems, security becomes just as critical as functionality.

Now that we’ve covered how to interface with hardware, let’s explore how to protect those systems from unauthorized access, tampering, and data breaches.

Security in Embedded Systems

Security is often overlooked in embedded development but it shouldn’t be. Embedded systems are increasingly connected to networks, cloud services, or other devices, which makes them vulnerable to attacks like unauthorized access, firmware tampering, or data leaks.

Even simple devices like smart plugs or fitness trackers can be exploited if their firmware is insecure.

Key Security Practices

• Secure Boot: Ensure the firmware is cryptographically signed and verified before execution. This prevents unauthorized firmware from running.

• Firmware Update Integrity: Use encrypted or signed updates, especially for Over-the-Air (OTA) upgrades. Unprotected updates can be a major attack vector.

• Lock Debug Interfaces: After flashing the final firmware, disable or lock access to JTAG, SWD, or UART debug ports to prevent reverse engineering.

• Minimal Exposure: Disable unused peripherals (for example, Bluetooth, USB, network interfaces) and avoid exposing debug info (like UART prints) in production.

• Watchdog Timers: While not security features per se, watchdogs help ensure system recovery in the event of unexpected software behavior – which could result from attacks or bugs.

Security should be layered, as no single mechanism is sufficient on its own. Build security into every stage of the development process, from boot to communication to update handling.

Whether you're designing a consumer product or an industrial controller, proactive security practices are essential for protecting user data, system reliability, and device reputation.

This resource provides a good understanding of Embedded Systems Security: BlackBerry QNX: Embedded System Security Guide

Debugging and Forensics in Embedded Systems

Debugging embedded systems is one of the most challenging and fascinating aspects of development. Unlike in desktop or web applications, bugs in embedded systems often manifest as unexpected hardware behavior rather than error messages.

For example, suppose your code is supposed to blink an LED once per second:

• If the LED stays on, your delay code might be broken.

• If it blinks erratically, you might have a timing bug.

• If it doesn’t blink at all, you might never be reaching that part of your code or the hardware might not be configured correctly.

Why Debugging is Critical

Embedded systems directly control real-world hardware, often in critical or safety-sensitive environments. A small bug can lead to large consequences.

Historical Note: During the Apollo 11 moon landing, the onboard computer started throwing alarms due to a task overflow. The system restarted and was able to recover itself and allowing the mission to continue safely.

Debugging and post-mortem analysis (forensics) are essential skills for embedded developers.

Common Debugging Tools and Techniques

1. Print Statements (UART Logging)

The simplest and most common method. They send debug messages over a serial connection (UART).

You can use printf() or similar to track variable values, function entries/exits, and system state

• Pros: Easy to implement

• Cons: Can affect timing – not usable if UART is unavailable or disabled

2. Trace Variables

In systems without output peripherals (like UART), you can use trace flags, setting bits in a global variable to indicate code progress.

uint32_t trace_flags = 0;

void init_sensor() 
{
    trace_flags |= (1 << 0); // Bit 0: sensor init started
    // ...
    trace_flags |= (1 << 1); // Bit 1: sensor init complete
}

You can then examine trace_flags in memory to track execution flow, even post-mortem. The trace flags can be printed out or dumped via lldb or gdb.

3. Hardware Debugging: JTAG, SWD, and Debuggers

Modern microcontrollers (like ARM Cortex-Ms) support hardware debugging interfaces such as:

• JTAG (Joint Test Action Group)

• SWD (Serial Wire Debug)

These allow a debugger to:

• Pause execution

• Set breakpoints

• Inspect and modify memory

• Single-step through code

ARM CoreSight is a debug and trace architecture developed by ARM for its processor cores (like Cortex-M, Cortex-A, Cortex-R). It provides a set of hardware modules built into ARM-based chips that allow developers to:

• Debug the system while it's running (non-intrusively)

• Trace code execution, memory accesses, and peripheral activity

• Analyze system performance and find hard-to-catch bugs

In short: CoreSight lets you look inside your embedded system while it's alive and working, without halting it unnecessarily.

Why CoreSight Exists

Traditional debugging tools (like breakpoints or single-stepping with JTAG) are often intrusive (they pause the system), limited (can't capture what happened right before a crash), or not suitable for real-time systems.

CoreSight solves these by enabling real-time tracing and non-intrusive observation of what's happening inside the chip.

Popular Debug Tools:

• ST-Link – HW from STMicrocontrollers

• J-Link – Universal debugger supporting a wide range of MCUs

• OpenOCD – Open-source interface for hardware debugging

• GDB / LLDB – Command-line debuggers used alongside the above

Single-stepping is most effective when compiler optimizations are off. With optimization, code might be reordered, inlined, or even eliminated.

4. Using Map and Disassembly Files

When debugging complex issues, especially crashes or memory overflows, you'll need to go deeper.

Map Files show the layout of functions and variables in memory (Flash and RAM). They help you locate:

• Stack overflows

• Unexpected memory usage

• Function addresses

Disassembly Files let you see the machine code generated from your source. This is critical when:

• Code is heavily optimized

• You’re diagnosing instruction-level failures

• You’re working without source code (e.g., binary-only drivers)

This resource provides a good overview on Map files, linkers and ELF format: Tenouk’s ELF/Map/Linker Guide

Common Bug: Buffer Overflows

Buffer overflows are one of the most frequent (and dangerous) issues in embedded systems. They happen when data is written past the end of an allocated array, overwriting nearby memory and causing unpredictable behavior.

Symptoms:

• Code crashes mysteriously

• Data appears to “corrupt itself”

• Variables change value without explanation

You can learn more in my article on Debugging Buffer Overflows, which walks through ways to debug a buffer overflow and build robust buffer code.

Embedded Forensics

Sometimes, a device fails in the field, where you can’t attach a debugger. That’s where forensics comes in:

• Use watchdog timers to reset the system and log failure info

• Save crash signatures to non-volatile memory (for example, EEPROM, Flash)

• Implement assert handlers that log file names, line numbers, or fault types

These techniques help you reconstruct what went wrong after the device has rebooted or been recovered.

You can learn more here: Debugging Techniques for Embedded Systems – Medium.

Debugging and forensics are invaluable when something goes wrong – but a robust system should aim to catch issues before they reach deployment.

That’s where automated testing becomes essential. With embedded software increasingly powering critical applications, the ability to run consistent, repeatable tests across hardware configurations saves time, improves reliability, and enables faster development cycles.

Next, let’s explore how embedded testing works, the challenges unique to hardware, and how automation frameworks help streamline validation.

Automation and Testing in Embedded Systems

Like all other areas of software engineering, testing is essential in embedded systems. But testing embedded software comes with its own set of challenges, mainly because it interacts with hardware.

Manual testing can be time-consuming and resource-intensive, especially when tests need to be repeated for multiple firmware versions or configurations. That’s where automated testing becomes invaluable.

Why Automated Testing?

Automated testing helps:

• Catch regressions early

• Test edge cases consistently

• Reduce human error

• Scale testing across versions and hardware setups

But automating tests for embedded systems isn’t just writing test cases – it’s about setting up an infrastructure that connects your code to the physical hardware under test.

Test Architecture: Host + DUT

Most embedded test setups involve two components:

• Host: Your development PC or CI test controller, which sends test commands and receives data.

• DUT (Device Under Test): The microcontroller board or embedded system running the firmware.

These two communicate over a physical link, commonly USB, UART, or FTDI, which carries commands and test data between them.

Diagram (suggested structure)

You could visualize this as:

Key Components of Embedded Test Automation

1. File Management

Many automated tests rely on CSV or JSON files to define:

• Input configurations

• Expected outputs

• Test parameters

Python makes it easy to:

• Read input vectors from CSVs

• Write logs or pass/fail results

• Parse structured data

2. Data Communication

Maintaining a stable and reliable link between the Host and DUT is critical. This includes:

• Opening and managing UART or USB connections (for example, with pyserial)

• Framing test commands using opcodes or simple protocols

• Handling timeouts, retries, and error recovery

Example (Python with PySerial):

import serial

ser = serial.Serial('/dev/ttyUSB0', 115200) #set Baud rate
ser.write(b'\x01')  # Send opcode for "start test"
response = ser.read(64)  # Read 64 bytes of response

3. Automation Manager (DUT-side)

A lightweight software agent runs on the embedded device. Its responsibilities:

• Parse incoming commands

• Trigger specific test routines

• Send response data back to the host

This is often implemented using a switch-case structure in C or C++:

void automation_manager(uint8_t opcode) {
    switch(opcode) {
        case 0x01: run_sensor_test(); break;
        case 0x02: run_motor_test(); break;
        default: break;
    }
}

4. Automation Manager (Host-side)

This is the control center of your test workflow:

• Sends test commands and parameters to the DUT

• Waits for and logs results

• Compares responses to expected output

• Handles communication retries or failures

Often written in Python using:

• pyserial for communication

• pandas for file/data processing

• unittest or pytest for test structure

Tips for Effective Automation

• Use unique opcodes for each test command to avoid ambiguity

• Implement timeout handling to avoid hanging scripts

• Log everything, responses, errors, test timestamps

• Use versioned test input files to track changes over time

• Include self-tests on the DUT to validate hardware state before running full tests

Automated testing in embedded systems is not just about running scripts, it's about building a bridge between your host PC and your device, managing the flow of commands and data, and ensuring tests are consistent, repeatable, and reliable.

While this requires effort to set up, the payoff is huge: confidence in your firmware, faster development cycles, and reduced risk of bugs making it into production.

Where to Go from Here

Building your Embedded Project

After exploring the theory and tooling of embedded systems, it's time to apply what you've learned. This section walks you through the steps to create your own embedded system – from concept to code and deployment.

Use the checklist below to guide your first project, whether you're prototyping a sensor device or automating a simple process.

Project Setup Checklist:

1. Define the Goal

   • What task does the system perform?

   • Identify inputs (for example, temperature sensor) and outputs (for example, relay or LED).

2. Requirements Gathering

   • Functional: What features must it support?

   • Non-functional: Memory limits, real-time behavior, power constraints.

   • Any security or safety-critical elements?

3. Choose Your Hardware

   • Microcontroller (for example, STM32F4)

   • Sensors and actuators

   • Communication interfaces (UART, I2C, SPI, and so on)

4. Software Architecture

   • Bare-metal, RTOS, or embedded OS?

   • Driver abstraction: will you use HAL or custom low-level code?

   • Organize code into layers: application logic, drivers, hardware init.

5. Toolchain Setup

   • Install GCC toolchain (for example, arm-none-eabi-gcc)

   • Configure Makefile and linker script

   • Set up debugger and flashing tools (for example, OpenOCD, ST-Link)

6. Firmware Implementation

   • Initialize peripherals

   • Implement control logic inside main() or tasks

   • Use interrupts or timers for responsiveness

7. Flashing and Initial Tests

   • Use OpenOCD or ST-Link to flash the binary

   • Test peripheral behavior and debug with UART or GDB

8. Debug and Profile

   • Use JTAG/SWD, CoreSight, and trace logs

   • Check memory layout with map/disassembly files

   • Identify bottlenecks and edge cases

9. Security Hardening

   • Disable debug interfaces post-flash

   • Add firmware signing and secure boot

   • Minimize surface area: disable unused features

10. Testing and Automation

• Connect Host to DUT via UART/USB

• Use Python + PySerial to send test vectors

• Log, compare, and report test outcomes

Embedded firmware development is a deep and rewarding field where software meets the hardware. Whether you're controlling an LED, reading from a sensor, or orchestrating multiple tasks in real time, the embedded stack teaches you how hardware, software, timing, and efficiency all come together.

Summary:

In this guide, we walked through the essential building blocks at a high level:

• What embedded systems are, and how they sense → process → act

• How microcontrollers work, from memory layout to interrupts and protocols

• How to design robust, scalable embedded software with clean architecture

• When to choose bare-metal, RTOS, or full OS solutions

• How to build drivers, write modular code, and interface with peripherals

• Tools for debugging, tracing, and analyzing system behavior

• Strategies for automating embedded testing using Python and host-device communication

• And finally, why security matters, especially in a connected world

Whether you're preparing for embedded job interviews, building your own IoT projects, or just exploring how software drives real-world systems, this article gives you a launchpad for deeper learning.