The order service handles the checkout, saves the order to its database, and then publishes an order.created event to a message queue so every downstream system can react independently.

This is a common and reasonable design, but it has a reliability problem that's easy to miss until something goes wrong in production.

When a customer places an order and the payment goes through, the application needs to do two things: save the order to the database and publish the event to the queue. These are two separate writes to two separate systems, and there's no way to make them share a single atomic transaction. If the process crashes, the network hiccups, or a deployment rolls out between the two writes, one side commits and the other does not. The order sits confirmed on the customer's screen while the warehouse has no idea it exists.

The transactional outbox pattern is the standard solution to this problem. In this article, we'll build it from scratch in Node.js, using PostgreSQL for the order service database, SQS for the queue, and DynamoDB as the fulfillment service's database. For local development, we'll use floci, a free open-source AWS emulator that runs all three with a single Docker container.

What We'll Cover

• Prerequisites

• The Problem with Two Writes

• The Outbox Pattern

• What We'll Build

• Project Setup

• Database Schema

• The Request Handler

• The Relay Worker

• The Consumer

• Running the Whole Thing

• Going to Production

• Conclusion

Prerequisites

To follow along, you should be comfortable with:

• Node.js and async/await

• Database transactions (BEGIN, COMMIT, ROLLBACK)

• The general concept of a message queue

You don't need prior experience with AWS, SQS, or DynamoDB. We'll be running everything locally.

You will need Node.js 20 or later and Docker installed on your machine.

The Problem with Two Writes

The order service scenario from the intro is one place this problem appears, but the same pattern comes up in many other contexts.

A user registers and the app inserts their account record, then sends a message to trigger the welcome email and the onboarding workflow. A file is uploaded and the API writes the metadata to the database, then publishes a message to kick off a processing worker for virus scanning or thumbnail generation. A payment webhook arrives, the handler records it in the database, then notifies downstream services that the payment is confirmed.

In every case, the application needs two writes to succeed together: one to the database and one to a queue or external system. If the second one is lost, the first one has no way of knowing.

If you want a deeper look at what database transactions actually guarantee and where they stop helping, see Beyond Happy Path Engineering: Databases.

The naïve implementation looks straightforward:

await db.query('INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2)', [customerId, amountCents]);
await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify({ customerId, amountCents }) }));

The database write happens first, then the queue write. Under normal conditions this works fine. The problem is what happens when something goes wrong between the two.

If the process crashes, runs out of memory, or gets killed mid-deployment after the database write but before sqs.send is called, the order record exists in the database but no event is ever published. The warehouse, email service, and fraud checker never find out the order happened. From the customer's perspective the order went through. From every downstream system's perspective it doesn't exist.

The failure can also go the other way. If sqs.send succeeds but the database write is later rolled back due to a constraint violation or an error in a subsequent step, you've published an event for an order that doesn't actually exist. A consumer acting on that event may try to fulfill an order with no corresponding record, or charge a customer for something that was never saved.

There's also a timing window even when both writes eventually succeed. Between the database commit and the successful sqs.send, a consumer that queries the database after receiving the event may not find the order yet, depending on transaction isolation and replication lag. These are two separate systems with no shared transaction boundary, and no amount of careful sequencing fully closes the gap.

These aren't edge cases that only happen under extraordinary circumstances. Deploys restart processes mid-request. Out-of-memory kills happen without warning. Networks drop connections at any point. Any of these can interrupt the two-write sequence, and the result is a system that's silently inconsistent with no error logged and no alert fired.

A variation I've seen a few times that looks safer but is actually worse is wrapping both operations in a database transaction:

// PLEASE DO NOT EVER DO THIS
const client = await pool.connect();
await client.query('BEGIN');
await client.query('INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2)', [customerId, amountCents]);
await sqs.send(new SendMessageCommand({ QueueUrl: QUEUE_URL, MessageBody: JSON.stringify({ customerId, amountCents }) }));
await client.query('COMMIT');

The intent is to make the two writes feel like a unit, but a database transaction has no authority over SQS. The transaction can only roll back database operations. If sqs.send succeeds and then COMMIT fails, the message is already in the queue and can't be taken back. If the process crashes after COMMIT but before the function returns, the transaction committed and the message was sent, but the caller may retry, potentially inserting a duplicate order.

Beyond the correctness problems, this pattern holds an open database connection and any row locks for the entire duration of the SQS network call. SQS is normally fast, but under load, retries, or a degraded queue, that call can take seconds. Every other request trying to read or write the same rows has to wait. In a busy application, this is a reliable way to exhaust the connection pool and bring down unrelated parts of the service.

The Outbox Pattern

The core idea is to stop treating the queue publish as a second write that happens after the database write, and instead make it part of the same database transaction.

Rather than calling sqs.send directly, the application inserts a row into an outbox table in the same transaction as the business record. A separate relay process reads the outbox table and publishes the messages to SQS. On the other end, a consumer receives the messages and writes to its own data store. In our case that is a fulfillment service writing to DynamoDB, completely separate from the order service's PostgreSQL database.

If the transaction rolls back for any reason, the outbox row disappears with it. There's no orphaned message in the queue because the message was never sent. If the application crashes after committing but before the relay runs, the outbox row is still there with status='pending', and the relay will pick it up on its next iteration.

The only guarantee the pattern relies on is the one the database already provides: atomicity within a single transaction.

The relay worker is responsible for the eventual delivery guarantee. It runs on an interval, selects pending rows, publishes them to SQS, and marks them as sent only after SQS confirms receipt. If the relay crashes mid-run, it will reprocess the same rows on the next iteration, which means SQS may receive some messages more than once.

That's why the consumer needs to be idempotent: it must handle receiving the same message twice without creating duplicate fulfillment records. We'll cover how to implement that when we build the consumer.

This separation of concerns is what makes the pattern practical. The request handler commits one atomic database transaction and returns. The relay handles the network call to SQS asynchronously, at its own pace, with its own retry logic, without holding database connections open or blocking request handling. The consumer is fully decoupled from the order service and owns its own data store.

The diagram below illustrates the flow:

What We'll Build

Now let's see the whole thing in practice. We'll implement a simple order placement API. When a customer sends a request to place an order, the order service saves it to PostgreSQL and inserts a row into the outbox table, all in one atomic transaction. A relay worker wakes up periodically, reads the pending outbox rows, and publishes each one as a message to SQS. A separate fulfillment service receives those messages from the queue and creates fulfillment records in DynamoDB.

By the end, you'll have an HTTP endpoint you can call, and you'll be able to verify that placing an order triggers the creation of a fulfillment record in a completely separate database, owned by a completely separate service, without either service ever talking to the other directly.

You can find the complete working code at github.com/gkoos/article-outbox.

Project Setup

Before you can run any code, you need to get floci running so you have local instances of PostgreSQL, SQS, and DynamoDB. You'll also need Node.js 20 or later and Docker installed.

Start by cloning the repository and installing dependencies:

git clone https://github.com/gkoos/article-outbox
cd article-outbox
npm install

Next, start floci. This command pulls the latest floci image and starts a Docker container that exposes a local AWS API endpoint (make sure Docker is running):

npm run floci:start

On Linux and macOS, this just works. On Windows with Docker Desktop, you need to edit the floci:start script in your package.json to change the Docker socket mount from /var/run/docker.sock to //var/run/docker.sock.

The floci container is now listening on port 4566 and can spin up RDS (PostgreSQL), SQS, and DynamoDB instances on demand.

Now provision the AWS resources with a single setup command:

npm run setup

This script creates an RDS PostgreSQL database instance, an SQS queue named orders, and a DynamoDB table named fulfillments. It waits for RDS to become available and then writes a .env file with the correct connection details. The environment variables PG_PORT, SQS_QUEUE_URL, and DYNAMODB_TABLE_NAME now point to the local emulated services.

Finally, create the PostgreSQL tables:

npm run migrate

This creates the orders table and the outbox table in PostgreSQL. You now have a fully functional local environment ready to build against.

Database Schema

The two tables are simple. orders holds the business records: each order has a customer ID, an amount in cents, and a timestamp. The outbox table is the heart of the pattern: it's where the application writes the event that needs to be published.

CREATE TABLE orders (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  customer_id TEXT NOT NULL,
  amount_cents INTEGER NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  created_at TIMESTAMPTZ DEFAULT now(),
  sent_at TIMESTAMPTZ
);

CREATE INDEX ON outbox (status, created_at) WHERE status = 'pending';

The orders table needs nothing special. The outbox table stores the event metadata: what type of event it is (event_type), what data it contains (payload as JSON), and whether it has been sent yet (status).

The status starts as pending. When the relay publishes it to SQS, it will mark it as sent and record the timestamp. The index on (status, created_at) WHERE status = 'pending' lets the relay quickly find the next batch of unsent events without scanning the entire table.

The Request Handler

This is where the pattern starts. The request handler receives an HTTP POST, inserts an order into the database, inserts a corresponding row into the outbox table, and commits everything in a single atomic transaction. The key insight is that neither write succeeds unless both succeed.

const client = await pool.connect();
try {
  await client.query('BEGIN');

  // Insert the order record
  const { rows } = await client.query(
    'INSERT INTO orders (customer_id, amount_cents) VALUES ($1, $2) RETURNING *',
    [customerId, amountCents]
  );
  const order = rows[0];

  // Insert the outbox record in the same transaction
  await client.query(
    `INSERT INTO outbox (event_type, payload)
     VALUES ($1, $2)`,
    ['order.created', JSON.stringify({ orderId: order.id, customerId: order.customer_id, amountCents: order.amount_cents, createdAt: order.created_at })],
  );

  await client.query('COMMIT');
  res.status(201).json(order);
} catch (err) {
  await client.query('ROLLBACK');
  next(err);
} finally {
  client.release();
}

The handler gets customerId and amountCents from the request body, starts an explicit transaction with BEGIN, and inserts the order. Then it inserts an outbox row with the order data as the payload.

Everything commits atomically. If anything fails, everything rolls back and the client gets an error. If the process crashes between the commit and the response, the client won't get a 201, but the order and the outbox row are still safely committed to the database and the relay will eventually pick it up. The handler doesn't call SQS at all. That is the relay's job.

The Relay Worker

The relay worker is a separate process that polls the outbox table every second and publishes pending rows to SQS. It runs independently of the HTTP server and has no shared state with it.

async function relay() {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    const { rows } = await client.query(`
      SELECT *
      FROM outbox
      WHERE status = 'pending'
      ORDER BY created_at
      LIMIT 10
      FOR UPDATE SKIP LOCKED -- prevents multiple relays from processing the same rows
    `);

    for (const row of rows) {
      await sqsClient.send(new SendMessageCommand({
        QueueUrl: QUEUE_URL,
        MessageBody: JSON.stringify(row.payload),
        MessageAttributes: {
          EventType: { DataType: 'String', StringValue: row.event_type },
        },
      }));

      await client.query(
        `UPDATE outbox SET status = 'sent', sent_at = now() WHERE id = $1`,
        [row.id],
      );
    }

    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    console.error('Relay error:', err.message);
  } finally {
    client.release();
  }
}

setInterval(relay, 1000);

FOR UPDATE SKIP LOCKED is the key to running multiple relay instances safely: when a relay picks up a batch of rows, it locks them. Any other relay instance trying to select the same rows will skip them and move to the next available ones, so you never get two relays publishing the same message from the same run.

The relay marks each row as sent only after sqsClient.send returns. If the relay crashes after sending to SQS but before updating the row, the row stays pending and the relay will resend it on the next iteration.

Note that the UPDATE happens inside the same transaction as the SELECT FOR UPDATE, so if the relay crashes mid-batch, the entire batch rolls back and all rows in it will be retried, including any that were already successfully sent to SQS.

The at-least-once delivery guarantee applies at the batch level, not the individual row level. You can read about this problem in Beyond Happy Path Engineering: the Network: when a response is lost, the caller can't know whether the operation succeeded, so it retries, and the receiver may see the same request twice. This means the consumer may see the same message more than once, which is why idempotency matters on the consumer side.

The Consumer

The consumer is a completely separate service. It knows nothing about the order service's PostgreSQL database. Its only input is the SQS queue, and its only output is the DynamoDB fulfillments table. This is the point of the pattern: the two services are decoupled by the queue, and each owns its own data store.

As we saw earlier, because SQS delivers at least once (meaning a message might be delivered more than once), the consumer must be idempotent. The PutItem call uses a ConditionExpression that makes the write a no-op if a fulfillment record for that order already exists, so redelivered messages are handled safely.

async function consume() {
  const { Messages } = await sqsClient.send(new ReceiveMessageCommand({
    QueueUrl:              QUEUE_URL,
    WaitTimeSeconds:       20,   // long-poll: wait up to 20s for messages
    MaxNumberOfMessages:   10,
    MessageAttributeNames: ['All'],
  }));

  for (const msg of Messages ?? []) {
    const event = JSON.parse(msg.Body);

    try {
      await dynamoClient.send(new PutItemCommand({
        TableName: 'fulfillments',
        Item: {
          orderId:     { S: event.orderId },
          customerId:  { S: event.customerId },
          amountCents: { N: String(event.amountCents) },
          status:      { S: 'received' },
          createdAt:   { S: new Date().toISOString() },
        },
        ConditionExpression: 'attribute_not_exists(orderId)', // idempotency check
      }));
    } catch (err) {
      if (err.name !== 'ConditionalCheckFailedException') throw err;
      // already processed, safe to continue
    }

    // delete the message only after the write succeeds (or was already done)
    await sqsClient.send(new DeleteMessageCommand({
      QueueUrl:      QUEUE_URL,
      ReceiptHandle: msg.ReceiptHandle,
    }));
  }
}

ConditionExpression: 'attribute_not_exists(orderId)' tells DynamoDB to reject the write if a record with that orderId already exists. When that happens, DynamoDB throws a ConditionalCheckFailedException. The consumer catches that specific error and ignores it, then deletes the message from the queue and moves on. Any other error is rethrown and the message stays in the queue to be retried.

The DeleteMessage call happens after the DynamoDB write, not before. If the process crashes between the write and the delete, SQS will redeliver the message and the condition check will handle it. If the process crashes before the write, the message stays in the queue and will be processed normally on the next delivery.

Running the Whole Thing

With floci running and the resources provisioned, open three terminal tabs and start each process:

node src/server.js    # the order API on port 3000
node src/relay.js     # the outbox relay
node src/consumer.js  # the fulfillment consumer

Now place an order:

curl -X POST localhost:3000/orders \
  -H 'Content-Type: application/json' \
  -d '{"customerId":"c1","amountCents":4999}'

You should get back a 201 with the new order record:

{"id":"1768d35b-083d-45f1-adb5-4063d8d7fcab","customer_id":"c1","amount_cents":4999,"created_at":"2026-07-30T20:27:10.628Z"}

Within a second the relay will pick up the outbox row and publish it to SQS. The consumer will receive the message and write a fulfillment record to DynamoDB. The repo includes a convenience script to verify this:

npm run check

You should see a fulfillment record with the orderId from the order you just placed:

{
  orderId: 'c335640e-bc4a-47e4-afed-484c95fbd6d3',
  customerId: 'c1',
  amountCents: '4999',
  status: 'received',
  createdAt: '2026-07-30T19:02:54.929Z'
}

Going to Production

Because the local setup uses floci to emulate AWS, switching to real AWS requires no code changes at all. The AWS SDK reads the endpoint from AWS_ENDPOINT_URL in the environment. In production, you simply don't set that variable and the SDK talks to real AWS using the credentials and region from the standard environment variables (AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, or an IAM role if you are running on EC2 or ECS).

Running multiple relay instances is safe out of the box because of FOR UPDATE SKIP LOCKED. You can scale the relay horizontally and each instance will pick up a different set of rows without duplicating messages.

One thing worth adding before going to production is handling permanent failures in the relay. Right now the relay only uses pending and sent. You should add a failed status and a retry counter: after a row has failed N times, mark it failed and stop retrying it. Then configure a dead-letter queue on the orders SQS queue as well, so that messages the consumer can't process after the maximum number of retries land somewhere you can inspect rather than disappearing silently.

For high-throughput systems where polling latency matters, change data capture (CDC) is a common alternative to the polling relay. Tools like Debezium read directly from the PostgreSQL write-ahead log and publish changes to Kafka or SQS without any polling delay. The outbox table and the consumer stay exactly the same, only the relay is replaced.

This is a bigger operational commitment than a polling worker, so polling is the right starting point for most systems.

Conclusion

The dual-write problem is easy to overlook because the naïve implementation works correctly most of the time. It only fails in the gaps between two separate system writes, and those gaps only become visible when something goes wrong at exactly the wrong moment. By the time you notice it in production, data is already inconsistent and there is no clean way to recover.

The transactional outbox pattern closes that gap at the database level. The outbox row is part of the same atomic commit as the business record, so the two are always in sync. The relay handles the network call to SQS independently, with its own retry logic, without touching the request lifecycle. The consumer handles at-least-once delivery with a single condition check on the write.

Each piece is simple on its own, and together they give you reliable, decoupled event delivery without distributed transactions.

An operator extends the same pattern to resources Kubernetes doesn't know about natively, letting you manage custom, often external, systems the same declarative way you manage everything else in the cluster.

This guide is divided into four parts: what an operator actually is, the anatomy of one, building one from scratch, and preparing it for production.

Table of Contents

• Part 1: Introduction

  • What is an Operator?

  • Operator vs Controller vs CRD

  • Why Not Just a Helm Chart, a CronJob, or a Script?

• Part 2: Anatomy of an Operator

  • Custom Resource

  • Watching for Change

  • Manager

  • Reconciliation Loop

• Part 3: Building the Operator

  • Setup

  • Mock Provider

  • Defining the VirtualMachine CRD

  • Reconciler

  • Failure Handling & Retries

  • Finalizer

  • Predicate

  • Owned Resources

  • Cross-Resource Reconciliation

  • RBAC

• Part 4: Production & Deployment

  • Packaging & Deployment

  • Performance & Resilience

  • Security

  • Observability

• Next steps

Part 1: Introduction

What is an Operator?

Kubernetes works by comparing the state we describe against actual state. A controller acts to close the gap, whether that's the Deployment controller replacing a pod we killed or scaling down the ones we no longer want.

This loop of observe, compare, and act is called reconciliation. It means looking up a resource's desired and actual state, deciding what to do next, and recomputing that decision fresh on every run regardless of what changed.

That's what makes the loop resilient: it never has to trust that it saw every event, only that it gets called again.

An operator applies that exact same loop to a resource Kubernetes doesn't understand natively. We define a Custom Resource, describe our domain's desired state in it, and write a controller that knows how to reconcile that domain.

That's the whole concept. Everything else in this guide (informers, workqueues, finalizers, status conditions) exists to make that one loop reliable for a resource type Kubernetes only knows about because we defined it.

Operator vs Controller vs CRD

These three terms are often used interchangeably, but they describe three different layers of the same system.

• CRD (CustomResourceDefinition): a schema we register with the Kubernetes API server to teach it about a new resource type. On its own, a CRD does nothing. It only gives the API server a shape to store, validate, and serve.

• Controller: any piece of software running a reconciliation loop against a resource type, from the built-in Deployment controller to a custom controller reconciling a PostgresCluster.

• Operator: a controller, or a small set of controllers, that targets a custom resource and encodes enough domain-specific knowledge to manage its full lifecycle without a human: provisioning, upgrades, failure recovery, and so on.

Every operator is a controller, but not every controller is an operator. A CRD without a controller behind it is just a schema that nothing acts on.

Why Not Just a Helm Chart, a CronJob, or a Script?

A Helm chart renders a set of values into YAML and applies it once. It has no way to keep watching afterward. if a resource it created is deleted or drifts, Helm has no idea until we run helm upgrade again by hand.

A CronJob gives us a loop back, at the cost of granularity, staleness up to one interval, no state carried between runs, and no way for one CronJob to react to a status change another one made.

And a one-off script only acts when triggered, manually or by a CI pipeline, and does nothing about drift in between runs. It's also rarely written with retries and idempotency as first-class concerns.

Why an operator wins here:

An operator is event-driven and continuous. The API server notifies it the instant a custom resource is created, updated, or deleted, and it keeps reconciling for that resource's entire lifetime, not just at apply time. That matters most for state that takes time to converge, can fail partway through, and can drift after it's first created.

This comes at a cost, though. An operator is a long-running process with its own RBAC (Role-Based Access Control), failure modes, and observability surface. This is more to build and operate than a chart or a script.

If the problem really is rendering some YAML once, a Helm chart is the right tool. An operator earns its keep when the problem is keeping something continuously correct, which is what the rest of this guide builds toward.

Part 2: Anatomy of an Operator

Next, we'll look at the pieces that make that loop actually work: the Custom Resource itself, the machinery that notices when something changed, and the manager that runs it all, before going into the reconciliation loop in detail.

Custom Resource

Before a controller can reconcile anything, the API server needs to know the shape of what it's storing. Registering a CRD teaches it that shape.

Every Custom Resource carries the same identity fields every Kubernetes object already has (kind, name, namespace, labels, and so on), plus two fields that are entirely ours to define: a spec and a status. That split isn't a style choice. It maps directly to the reconciliation loop.

• Spec is desired state. Whoever creates or edits the resource writes it, and the controller only ever reads it.

• Status is observed state. It's written only by the controller, to record what it found and what it did.

A client that writes to status directly is working around the controller instead of through it, which is why status is usually served as its own subresource with separate permissions.

The last piece is registration. The API server and any client talking to it need a shared, agreed-upon way to encode and decode our type, so we register it once against a scheme before anything can use it. Without that, our type is just a definition nobody can serve. With it, the API server can store and serve it exactly the way it serves Pods or Deployments.

We'll see exactly what that registration looks like when we build one for real in Part 3.

Watching for Change

A reconciler doesn't poll the API server in a loop asking "did anything change yet?" Three pieces work together to avoid that.

An informer opens a long-lived watch against the API server and keeps a local, in-memory cache of every object of a given type, updating it as add, update, and delete events arrive.

A lister reads from that cache instead of the API server, so a reconciler checking "does this Resource already exist?" costs a local map lookup, not a network call.

A workqueue sits between the informer and the reconciler. When the informer sees a change, it doesn't call the reconciler directly. Instead, it enqueues a key, namespace, and name, not the object itself. Workers pull keys off the queue and reconcile them, and the queue deduplicates and rate-limits on our behalf, so ten rapid updates to the same object collapse into one pending item instead of ten redundant reconciles.

This is also why a reconciler receives a key and not an object. By the time a worker picks the key off the queue, the object may have changed again, so the reconciler always looks up the current state itself rather than trusting whatever triggered it.

Manager

The manager is the process that owns all of this: the shared cache the informers populate, the client the reconciler uses to read and write objects, and the health, readiness, and metrics endpoints the rest of the cluster uses to know the controller is alive. Every reconciler we register runs inside one manager.

If we run more than one replica of the same controller for availability, we don't want both replicas reconciling the same object at once and racing each other.

The manager coordinates this through leader election. Replicas compete for a lease, exactly one holds it and actively reconciles, and the rest sit idle until the leader stops renewing it.

We'll come back to this in practice in Part 4. For now it's enough to know the manager is what makes it possible.

Reconciliation Loop

This is the part that matters most. Once we understand this loop well, most of what an operator does is a variation on it.

A reconciler's entry point is called with just a namespace and a name, nothing else. No spec, status, or diff. The reconciler has to fetch the object itself, compare its spec against what it can observe of the actual state, and decide what to do. That constraint is deliberate, and it's the reason for everything below.

Idempotency

Because the reconciler only ever gets a key, and because it can be called any number of times for the same object (in a row, out of order, or after a long gap), it has to produce the same end result no matter how many times it runs. A reconciler that blindly calls create every time it runs breaks the moment it runs twice, since the second call fails against an object that already exists.

The fix is to always check current state before acting: create only if missing, update only if different, and delete only if it shouldn't exist.

Event-driven reconciliation

A reconcile is triggered by a watch event on the resource being reconciled, and by convention also on anything it owns or otherwise depends on. On top of that, most controllers set a periodic resync so the loop also runs on a schedule even with no watch event at all, which matters once state can drift for reasons a watch would never catch.

Requeues

Sometimes a single pass through reconcile can't finish the job, becuase the work it's waiting on is still in progress elsewhere. A reconciler can ask to be called again after a delay without treating this as a failure. This is how it polls something that takes time to converge, rather than blocking inside a single call.

Error handling

Returning an error does something similar: it requeues, but with exponential backoff instead of a fixed delay. So a persistently failing reconcile doesn't hammer whatever it's failing against.

It's worth distinguishing errors that are worth retrying (like a timeout or lock conflict) from ones that aren't (like a spec that will never be valid, which should be surfaced as a status condition instead of retried forever).

Drift correction

Put all of the above together and the loop is self-healing by construction. Because reconcile recomputes the full diff every time rather than reacting to what specifically changed, it doesn't matter whether the drift came from someone running kubectl edit, another controller, or the underlying system the resource represents changing state on its own. The next reconcile, whether triggered by a watch event or a resync, sees the same gap either way and closes it the same way.

Part 3: Building the Operator

Everything so far has been building toward this. We now know what an operator is, how the terms around it relate, and what pieces a reconciliation loop is made of.

Now we'll put all of it to use and build VMOperator. It's an operator that manages a VirtualMachine Custom Resource backed by a mock cloud provider, a small HTTP service we'll also write that stands in for a real one.

apiVersion: compute.example.com/v1
kind: VirtualMachine
spec:
  image: ubuntu-22.04
  cpu: 2
  memory: 4Gi
status:
  phase: Running
  id: vm-123

Say we want to represent a virtual machine in a Kubernetes-native way, kubectl apply a YAML file, and get a VM – all without touching a cloud console or a separate CLI.

That's the motivation behind VMOperator: something that lives entirely outside Kubernetes becomes just another object the cluster's own tooling (kubectl, RBAC, GitOps pipelines) already knows how to work with.

Setup

We'll need a local cluster, and kind is the easiest way to get one:

kind create cluster --name vmoperator

Beyond that, we'll be using Go in this part for the operator, kubectl pointed at the new cluster, and Python with Flask for the mock provider below.

pip install flask

Mock Provider

Before we write controller code, we need something for it to control. The mock provider is a small HTTP service with three endpoints:

• POST /vms to create one

• GET /vms/{id} to check on it

• DELETE /vms/{id} to remove it

backed by nothing more than a dict in memory.

Every VM it creates starts in Provisioning and flips to Running a few seconds later on its own, which is enough to force our reconciler to actually poll instead of assuming success.

We're writing this one in Python rather than Go. It has nothing to do with the operator's code, as this is only for mock purposes.

import random
import string
import threading
import time

from flask import Flask, jsonify, request

app = Flask(__name__)
vms = {}  # in-memory store, keyed by VM id

def provision(vm):
    time.sleep(5)  # simulate provisioning taking time
    vm["phase"] = "Running"

@app.post("/vms")
def create_vm():
    body = request.get_json()
    vm_id = "vm-" + "".join(random.choices(string.digits, k=6))
    vm = {"id": vm_id, "image": body["image"], "phase": "Provisioning"}
    vms[vm_id] = vm

    threading.Thread(target=provision, args=(vm,), daemon=True).start()  # flips to Running in the background

    return jsonify(vm)

@app.get("/vms/<vm_id>")
def get_vm(vm_id):
    vm = vms.get(vm_id)
    if vm is None:
        return "", 404
    return jsonify(vm)

@app.delete("/vms/<vm_id>")
def delete_vm(vm_id):
    vms.pop(vm_id, None)
    return "", 204

if __name__ == "__main__":
    app.run(port=8080, threaded=True)

We'll run this as its own process, alongside the cluster, listening on the port the operator will be configured to call. Nothing about it knows Kubernetes exists, which is the point: it's standing in for a real cloud API.

Defining the VirtualMachine CRD

With something to control, we can define what we're controlling. The VirtualMachine type follows exactly the spec and status split from Part 2:

type VirtualMachineSpec struct {
	Image  string `json:"image"`
	CPU    int    `json:"cpu"`
	Memory string `json:"memory"`
}

type VirtualMachineStatus struct {
	ID    string `json:"id,omitempty"`    // provider-assigned id, empty until first provisioned
	Phase string `json:"phase,omitempty"` // mirrors the provider's lifecycle phase
}

type VirtualMachine struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   VirtualMachineSpec   `json:"spec,omitempty"`
	Status VirtualMachineStatus `json:"status,omitempty"`
}

type VirtualMachineList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []VirtualMachine `json:"items"`
}

TypeMeta carries kind and apiVersion, the same two fields on every Kubernetes object, built-in or custom, that say what this thing is.

ListMeta is its counterpart for list types, resourceVersion and continue for pagination (instead of name/namespace). This is why VirtualMachineList embeds ListMeta next to its TypeMeta while VirtualMachine itself embeds ObjectMeta.

Every type we register needs to satisfy runtime.Object, which means implementing DeepCopyObject. This is normally generated for us, but since we're doing this by hand, here's what that generated code actually looks like for VirtualMachine. The rest follow the same mechanical pattern:

func (in *VirtualMachine) DeepCopyObject() runtime.Object {
	out := VirtualMachine{
		TypeMeta:   in.TypeMeta,
		ObjectMeta: *in.ObjectMeta.DeepCopy(), // ObjectMeta already knows how to copy itself
		Spec:       in.Spec,                   // no pointers or slices in Spec, a plain copy is safe
		Status:     in.Status,
	}
	return &out
}

And the CRD manifest that teaches the API server about it:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: virtualmachines.compute.example.com
spec:
  group: compute.example.com
  scope: Namespaced
  names:
    kind: VirtualMachine
    listKind: VirtualMachineList
    plural: virtualmachines
    singular: virtualmachine
    shortNames: [vm] # lets us type `kubectl get vm` instead of the full plural
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {} # splits status into its own subresource, see Part 2
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [image, cpu, memory]
              properties:
                image: { type: string }
                cpu: { type: integer }
                memory: { type: string }
            status:
              type: object
              properties:
                phase: { type: string }
                id: { type: string }

The subresources.status line matters. It's what makes status a separate subresource with its own update path. This is exactly the boundary we talked about in Part 2 between what a client can write and what only the controller can.

The names block is also what kubectl resolves against, kubectl get virtualmachines works because plural says so. shortNames is why kubectl get vm works too, the same way kubectl get po works for Pods.

Reconciler

The reconciler's job is small on paper, look at a VirtualMachine, and make sure a matching VM exists in the provider and its status reflects reality. We wrap the provider's HTTP API behind a small client so the reconciler itself stays readable:

func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var vm computev1.VirtualMachine
	if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err) // object was deleted, nothing left to do
	}

	if vm.Status.ID == "" {
		// no VM yet, this is the first time we've seen this object
		created, err := r.Provider.Create(ctx, vm.Spec.Image)
		if err != nil {
			return ctrl.Result{}, err
		}

		vm.Status.ID = created.ID
		vm.Status.Phase = created.Phase
		if err := r.Status().Update(ctx, &vm); err != nil {
			return ctrl.Result{}, err
		}

		return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // check back shortly instead of blocking here
	}

	// VM already exists, poll the provider for whatever it knows right now
	current, err := r.Provider.Get(ctx, vm.Status.ID)
	if err != nil {
		return ctrl.Result{}, err
	}

	vm.Status.Phase = current.Phase
	if err := r.Status().Update(ctx, &vm); err != nil {
		return ctrl.Result{}, err
	}

	if current.Phase != "Running" {
		return ctrl.Result{RequeueAfter: 2 * time.Second}, nil // still provisioning, keep polling
	}

	return ctrl.Result{}, nil
}

Two things are worth calling out. First, this is only reachable at all because we've registered a watch on VirtualMachine. The API server tells us the moment one is created or edited, which is what triggers the first call.

Second, every branch ends by writing to vm.Status, mapping whatever the provider told us onto the resource. Kubernetes never talks to the provider directly. The only way anyone finds out a VM is running is because our reconciler wrote it into status.

Failure Handling & Retries

Notice the reconciler above never retries anything itself. When r.Provider.Create or r.Provider.Get fails (like because of a network blip or the mock provider not being up yet), it just returns the error. That's deliberate. Returning an error is how we ask controller-runtime to requeue with exponential backoff on our behalf. This means we don't need to hand-roll a retry loop, and a persistently unreachable provider doesn't get flooded with retries.

The one thing worth being careful about is treating every failure the same way. A timeout talking to the provider is worth retrying. A VirtualMachine whose spec.image the provider will never accept is not. Retrying that forever just produces a busy loop that never succeeds.

We'll leave surfacing that distinction through status conditions to the exercises. The reconciler above only has one failure mode to worry about, since the mock provider never rejects a request outright.

Finalizer

If we delete a VirtualMachine right now, Kubernetes removes the object and we're left with an orphaned VM the provider still thinks is running. A finalizer closes that gap: it's a string on the object that tells Kubernetes "don't actually delete this until I say so."

const vmFinalizer = "compute.example.com/vm-cleanup"

func (r *VirtualMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var vm computev1.VirtualMachine
	if err := r.Get(ctx, req.NamespacedName, &vm); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	if !vm.DeletionTimestamp.IsZero() {
		// being deleted, deprovision through the provider before letting it go
		if controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
			if vm.Status.ID != "" {
				if err := r.Provider.Delete(ctx, vm.Status.ID); err != nil {
					return ctrl.Result{}, err
				}
			}
			controllerutil.RemoveFinalizer(&vm, vmFinalizer) // safe to let the delete proceed now
			return ctrl.Result{}, r.Update(ctx, &vm)
		}
		return ctrl.Result{}, nil
	}

	if !controllerutil.ContainsFinalizer(&vm, vmFinalizer) {
		controllerutil.AddFinalizer(&vm, vmFinalizer) // register before we ever provision anything
		if err := r.Update(ctx, &vm); err != nil {
			return ctrl.Result{}, err
		}
	}

	// ... provisioning logic from before
	return ctrl.Result{}, nil
}

A kubectl delete on a VirtualMachine with our finalizer present doesn't remove it. Rather, it sets deletionTimestamp and waits.

Our reconciler sees that on the next call, deprovisions the VM through the provider, and only then removes the finalizer. At this point Kubernetes finally deletes the object. If there's no finalizer, there's no guarantee that cleanup ever runs.

Predicate

There's a subtle bug already sitting in the reconciler above. Every time it calls r.Status().Update, that write is itself a change to the object. This triggers our own watch, which calls reconcile again.

Left alone, this doesn't spin forever, since we're recomputing the same status until it settles. But it's still wasted work reconciling in response to writes we made ourselves.

A predicate filters those events that actually enqueue a reconcile, before our code ever runs:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). // drop status only events
		Complete(r)
}

generation only increments when spec changes. Status updates don't touch it. GenerationChangedPredicate uses that to drop events where nothing but status moved, so our own writes stop retriggering us. Then we're back to reconciling only when something meaningful changed, or when we explicitly ask to be requeued.

Owned Resources

A VirtualMachine being Running somewhere isn't very useful on its own, so let's make the operator also create a Secret holding the VM's connection details in-cluster:

func (r *VirtualMachineReconciler) reconcileConnectionSecret(ctx context.Context, vm *computev1.VirtualMachine) error {
	secret := &corev1.Secret{
		ObjectMeta: metav1.ObjectMeta{
			Name:      vm.Name + "-connection",
			Namespace: vm.Namespace,
		},
		StringData: map[string]string{"id": vm.Status.ID},
	}

	if err := controllerutil.SetControllerReference(vm, secret, r.Scheme); err != nil {
		return err // ties the Secret's lifecycle to this VirtualMachine
	}

	return r.Patch(ctx, secret, client.Apply, client.ForceOwnership, client.FieldOwner("vmoperator")) // create or update, either way
}

SetControllerReference is what makes this an owned resource, it stamps an owner reference onto the Secret pointing back at the VirtualMachine.

Two things fall out of that for free. Deleting the VirtualMachine now cascades, Kubernetes garbage collects the Secret automatically, and there's no finalizer needed since it's an in-cluster object, not an external one.

And if we add Owns(&corev1.Secret{}) alongside For(&computev1.VirtualMachine{}) in SetupWithManager, an edit or deletion of the Secret itself re-triggers reconciliation of its owning VirtualMachine. So if someone deletes it by hand, we notice and recreate it.

The same pattern, SetControllerReference call, and Owns() registration creates a second owned resource: a Service fronting the VM in-cluster. That's two different resource kinds owned by one VirtualMachine, which is all multiple owned resources means in practice. There's nothing more to it than calling the same pattern twice for different types.

Cross-Resource Reconciliation

Every VirtualMachine so far talks to one hardcoded provider endpoint. Real deployments need that to be configurable, and it's rarely a one-off: fifty VirtualMachines in the same AWS account share the same endpoint and credentials, and a hundred more might live in Azure instead.

We could put an endpoint field directly on VirtualMachineSpec, but rotating a credential or fixing a typo would then mean editing every VirtualMachine that uses it, one at a time. Pulling that into its own object lets many VirtualMachines reference it by name instead, so a single edit propagates to all of them.

Now let's add a second, small CRD:

type ProviderConfigSpec struct {
	Endpoint string `json:"endpoint"`
}

And a providerRef field on VirtualMachineSpec pointing at one by name. The interesting part isn't the new type. It's what happens when a ProviderConfig changes.

A VirtualMachine doesn't watch ProviderConfig directly, and there's no owner reference between them, so a plain Owns() won't do it. Instead, we watch the type and map each event onto every VirtualMachine that references it:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
		Owns(&corev1.Secret{}).
		Owns(&corev1.Service{}).
		Watches(
			&computev1.ProviderConfig{}, // not owned, so Owns() won't catch its changes
			handler.EnqueueRequestsFromMapFunc(r.findVirtualMachinesForProviderConfig),
		).
		Complete(r)
}

func (r *VirtualMachineReconciler) findVirtualMachinesForProviderConfig(ctx context.Context, obj client.Object) []reconcile.Request {
	var vms computev1.VirtualMachineList
	if err := r.List(ctx, &vms, client.InNamespace(obj.GetNamespace())); err != nil {
		return nil
	}

	var requests []reconcile.Request
	for _, vm := range vms.Items {
		if vm.Spec.ProviderRef == obj.GetName() { // only re-enqueue VMs that actually reference this config
			requests = append(requests, reconcile.Request{NamespacedName: client.ObjectKeyFromObject(&vm)})
		}
	}
	return requests
}

This is cross-resource reconciliation: one resource's change causing a different resource type entirely to reconcile, connected only by a field value rather than ownership.

It's also where the theme of this whole project comes back around. ProviderConfig is what would hold real credentials and a real endpoint for AWS, Azure, or GCP in a production version of this operator. The mock provider is standing in for exactly that boundary.

RBAC

None of the above works without permission to act on it. The manifest just has to list what we actually touch: VirtualMachine and ProviderConfig objects, the VirtualMachine status subresource separately, and the Secret/Service objects we create:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vmoperator-manager-role
rules:
  - apiGroups: ['compute.example.com']
    resources: ['virtualmachines', 'providerconfigs']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
  - apiGroups: ['compute.example.com']
    resources: ['virtualmachines/status'] # separate rule, it's a separate subresource
    verbs: ['get', 'update', 'patch']
  - apiGroups: ['']
    resources: ['secrets', 'services']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vmoperator-manager-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vmoperator-manager-role
subjects:
  - kind: ServiceAccount
    name: vmoperator-controller-manager
    namespace: vmoperator-system

Note: VMOperator manages a resource entirely outside the cluster through a hand-rolled HTTP client, but it's not a novel pattern. Crossplane, AWS Controllers for Kubernetes, Cluster API, and cert-manager all reconcile external or non-Kubernetes state through CRDs the same way. These resources are worth reading once this pattern feels familiar.

Another note: we're keeping VMOperator's scope narrow on purpose. Resizing a running VM, stopping and restarting one, taking snapshots, and supporting more than one real provider behind ProviderConfig are all natural extensions of what's here, and a reasonable next step once the core loop feels solid.

Part 4: Production & Deployment

Now that VMOperator works, let's see how to package and deploy it and improve it for production.

Packaging & Deployment

Everything so far has run as a binary on our own machine. go run against whatever cluster kubectl happens to be pointed at.

A Deployment needs an image instead, so the operator gets a multi-stage Dockerfile: one stage to compile it, and a second, much smaller image to actually run it:

FROM golang:1.26 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /vmoperator ./cmd/manager

FROM gcr.io/distroless/static-debian12
COPY --from=build /vmoperator /vmoperator
USER 65532:65532 # nonroot, matches the security context on the Deployment below
ENTRYPOINT ["/vmoperator"]

The build stage has the full Go toolchain and every source file, none of which need to ship. The final image only has the compiled binary, which covers most of what a container security context later in this part would otherwise have to ask for. There's no shell to get a foothold in even before runAsNonRoot is set.

docker build -t registry.example.com/vmoperator:v0.1.0 .
docker push registry.example.com/vmoperator:v0.1.0

That image is what the Deployment manifest under config/manager/ actually references. With it pushed somewhere the cluster can pull from, the rest of the manifests can go on: the CRDs, RBAC, the operator's Deployment, and a Deployment and Service for the mock provider. So it's no longer something we run as a side process on our own machine either:

kubectl apply -f config/crd/
kubectl apply -f config/rbac/
kubectl apply -f config/manager/

Installing the CRDs before anything else matters. Otherwise the operator's Deployment will crash-loop if it starts and immediately (it tries to watch a resource type the API server has never heard of).

Schema changes are the part that hand-written manifests make us feel directly. Adding a field to VirtualMachineSpec is harmless, but existing objects just don't have it set. Renaming or restructuring one isn't: every stored VirtualMachine was serialized against the old shape.

The CRD's versions list is built for exactly this, as more than one version can be served at once. One is marked storage to say which shape objects are actually persisted as, and a conversion webhook translates between them when a client asks for a version that isn't the stored one.

We don't need this for VMOperator today, since v1 is the only version that's ever existed. But it's why the versions field was a list and not a single value from the very first manifest we wrote.

None of the above replaces a person running kubectl apply by hand forever. A CI pipeline that builds the operator's image, pushes it, and applies the manifests on merge to main is the natural next step. This is ordinary CI/CD, nothing operator-specific about it once the manifests themselves are in Git.

Performance & Resilience

By default, a controller only processes one reconcile at a time. That's fine while we're the only ones testing it, but with hundreds of VirtualMachine objects it means that most of them sit in the workqueue waiting their turn even though nothing about reconciling one blocks reconciling another.

MaxConcurrentReconciles raises that:

func (r *VirtualMachineReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&computev1.VirtualMachine{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
		Owns(&corev1.Secret{}).
		Owns(&corev1.Service{}).
		Watches(&computev1.ProviderConfig{}, handler.EnqueueRequestsFromMapFunc(r.findVirtualMachinesForProviderConfig)).
		WithOptions(controller.Options{MaxConcurrentReconciles: 5}). // five VMs in flight instead of one
		Complete(r)
}

Caching only helps one side of this reconciler. Reading vm back from r.Get is already fast and local, as informers keep that in memory. But r.Provider.Get is a real HTTP round trip every single time, and there's no cache in front of it.

That asymmetry is worth sitting with, because it's the same one from Part 2: in-cluster reads are cheap because Kubernetes built the caching layer for us, and external reads are exactly as expensive as whatever's on the other end of the wire. We could add a short-lived cache in front of the provider client, but it comes with a real cost: a cached Running for a VM that just failed is a lie our status will repeat until the cache expires.

The provider not having a cache in front of it also means nothing is stopping us from hammering it. A burst of reconciles, say every VirtualMachine getting touched at once after a cluster restart, turns into a burst of HTTP calls with no coordination between them. Wrapping the client in a rate limiter caps that independently of whatever backoff the workqueue is already doing on failures:

type Client struct {
	baseURL string
	http    *http.Client
	limiter *rate.Limiter // shared across every reconcile using this client
}

func (c *Client) Create(ctx context.Context, image string) (*VM, error) {
	if err := c.limiter.Wait(ctx); err != nil {
		return nil, err
	}
	// ... existing HTTP call
}

Leader election is the other half of running more than one replica safely. We turned this down to a concept in Part 2, but in practice it's two fields on the manager:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
	LeaderElection:   true,
	LeaderElectionID: "vmoperator-leader",
})

With this set, every replica starts up, but only the one holding the lease actually reconciles. The rest sit ready to take over the moment it doesn't renew in time.

Concurrency also surfaces a race we glossed over in Part 3. Say the reconciler calls r.Provider.Create, the provider creates the VM and returns its id, and then the process crashes before r.Status().Update ever runs. vm.Status.ID is still empty, so the next reconcile sees an object with no VM yet and calls Create again. Now the provider has two VMs for one VirtualMachine.

Nothing about MaxConcurrentReconciles or leader election prevents this. It's a gap in the create step itself, and it only shows up once something can fail between the external call and the write that records it.

Closing it for real means the provider needs to accept an idempotency key, generated once and stored on the object before the first Create call, so a retried create recognizes that it already happened instead of making a second VM.

Security

The ClusterRole from Part 3 works, but it's broader than it needs to be. It grants every verb on secrets and services cluster-wide, when the operator only ever touches the ones it owns.

A tighter version scopes to a single namespace with Role/RoleBinding instead of ClusterRole/ClusterRoleBinding wherever VMOperator is only expected to run in one, and it drops verbs we never call. We never list or watch arbitrary Secrets outside our own, only the ones we create. This is also the RBAC the manifests applied in the previous section were referring to.

Credentials are the other gap. ProviderConfig currently holds a plaintext endpoint, and a real provider needs an API key alongside it, which has no business sitting in a CRD spec anyone with read access to the object can see. It belongs in a Secret, referenced by name instead of embedded:

type ProviderConfigSpec struct {
	Endpoint  string                      `json:"endpoint"`
	SecretRef corev1.LocalObjectReference `json:"secretRef"` // Secret holding the provider's API key
}

The reconciler resolves SecretRef at the point it builds the provider client, reads the key out of the Secret's data, and never logs it or writes it back to anything with wider read access, including the VirtualMachine's own status.

The last piece is the operator's own pod. A container security context that runs as a non-root user sets a read-only root filesystem, drops Linux capabilities it doesn't need, and shrinks what's possible if the binary itself is ever compromised. This is standard practice for any workload, not something specific to operators.

If we'd added an admission webhook anywhere in this guide, its certificates would belong here too. We didn't need one for VMOperator, so we'll leave that as a pointer rather than something to configure.

Observability

The manager exposes a Prometheus endpoint without us writing anything for it. Workqueue depth, reconcile duration, and reconcile error counts are already there per controller.

What isn't there automatically is anything about the provider, so we add a metric the same way any Go service would:

var providerCallDuration = prometheus.NewHistogramVec(
	prometheus.HistogramOpts{
		Name: "vmoperator_provider_call_duration_seconds",
		Help: "Duration of calls to the VM provider, by operation",
	},
	[]string{"operation"},
)

func init() {
	metrics.Registry.MustRegister(providerCallDuration) // shares the manager's existing /metrics endpoint
}

Wrapping each provider call with a timer around this turns "is the provider slow" from a question we'd have to guess at into one we can graph.

Logging benefits from the same instinct. log.FromContext(ctx) inside Reconcile already carries the VirtualMachine's name and namespace on every line if we set that up once in SetupWithManager. Adding vm.Status.ID to that logger right after it's set means every subsequent log line for that reconcile also carries the provider's own identifier for the VM. That one field is what makes it possible to grep a mock provider log and an operator log for the same request and find both sides of the same failure.

Next Steps

In this guide, you learned what a Kubernetes operator is, how to build one from scratch, and how to prepare it for production. You also learned about finalizers, predicates, owned resources, and cross-resource reconciliation along the way.

None of this is specific to managing VMs. The next operator, whatever it manages, is the same shape.

You can also review the resources below to keep learning:

• K8s custom resources

• client-go

• controller-runtime

• Docker docs

By the end, you'll have a scalable payment architecture that can be reused across multiple business domains.

Table of Contents

• Introduction

• Why Use a Dedicated Payment Service?

• Architecture Overview

  • Payment State Machine

• Prerequisites

• PayPal Concepts You Need to Know

  • Sandbox vs Live

  • Orders API Flow (What We Use)

  • Environment Variables

• Project Structure

• Step 1 — Create the Payment Service

• Step 2 — Define the gRPC Contract

• Step 3 — Implement the PayPal Service

• Step 4 — Build the Payment Flow (Create → Approve → Capture)

  • 4a. Create Payment

  • 4b. User Approves on PayPal

  • 4c. Capture Payment

• Step 5 — Connect Domain Services via gRPC

  • Domain Service Business Logic Example

• Step 6 — Add the API Gateway Layer

• Step 7 — Publish Payment Events with RabbitMQ

  • Two Paths to Mark an Order as Paid

• Step 8 — Database Schema and Migrations

  • Production Migration Gotcha

• Step 9 — Local Development Setup (Docker)

  • Environment Variables (.env)

  • Docker Compose (Local)

  • Start Services

  • Verify Health

  • Test Payment Flow

• Step 10 — Production Deployment

  • PayPal Live Credentials

  • Production .env

  • Docker Compose (Production)

  • Deploy Commands

  • Verify Production

  • Frontend Domain in Production

• Step 11 — Health Checks and Monitoring

• Complete Request Flow (Real Example)

• Coupon Support (Optional)

• PayPal Webhooks (Optional but Recommended)

• Testing Checklist

• Wrapping Up

• Further Reading

Introduction

Payment logic doesn't belong inside every microservice. When you scatter PayPal API calls across user-service, order-service, and billing-service, you end up with:

• Duplicated PayPal credentials and SDK code

• Inconsistent error handling and idempotency

• Hard-to-audit payment records

• Painful environment switching (sandbox to live)

The solution is a dedicated payment microservice that owns all PayPal interactions. Other services call it over gRPC, and payment outcomes are broadcast over RabbitMQ so domain services can update their own data.

This guide walks you through that pattern using a real-world stack:

Why Use a Dedicated Payment Service?

A dedicated payment service centralizes all payment-related responsibilities in one place. Instead of every microservice communicating directly with PayPal, they simply request payment operations from the payment service.

This service manages PayPal authentication, order creation, payment captures, wallet updates, ledger records, and webhook processing. Meanwhile, domain services remain focused on business logic such as student applications or subscriptions.

Domain services only need to know:

1. How much to charge

2. Who is paying

3. What business entity the payment is for (referenceId)

4. Where to redirect the user after payment (returnUrl / cancelUrl)

They do not need PayPal credentials.

Architecture Overview

Users initiate payments from the Frontend, and requests are routed through the API Gateway to the Students Service. The service uses gRPC to communicate with the Payment Service, which handles all interactions with PayPal.

Once the payment is completed, the Payment Service publishes an event to RabbitMQ, enabling the Students Service to update the payment status asynchronously.

┌────────────────────────────────────────────────────────────┐
│                     PRESENTATION LAYER                     │
├────────────────────────────────────────────────────────────┤
│ Frontend (React)                                           │
└───────────────────────┬────────────────────────────────────┘
                        │ HTTP
                        ▼

┌────────────────────────────────────────────────────────────┐
│                       GATEWAY LAYER                        │
├────────────────────────────────────────────────────────────┤
│ student-apigw                                               │
└───────────────────────┬────────────────────────────────────┘
                        │ gRPC
                        ▼

┌────────────────────────────────────────────────────────────┐
│                       DOMAIN LAYER                         │
├────────────────────────────────────────────────────────────┤
│ students-service                                            │
└───────────────────────┬────────────────────────────────────┘
                        │ gRPC
                        ▼

┌────────────────────────────────────────────────────────────┐
│                      PAYMENT LAYER                         │
├────────────────────────────────────────────────────────────┤
│ payment-service                                             │
│                                                            │
│ • Create Payment                                           │
│ • Capture Payment                                          │
│ • Wallet Management                                        │
│ • Ledger                                                   │
│ • Webhooks                                                 │
│ • Event Publishing                                         │
└──────────────┬───────────────────────┬─────────────────────┘
               │                       │
               │ REST                  │ RabbitMQ
               ▼                       ▼

      ┌───────────────┐      ┌────────────────────┐
      │    PayPal     │      │   payment_events   │
      │   Checkout    │      │       Queue        │
      └───────────────┘      └─────────┬──────────┘
                                       │
                                       ▼

                           ┌────────────────────┐
                           │ students-service   │
                           │ Event Consumer     │
                           └────────────────────┘

Payment State Machine

A payment state machine represents the lifecycle of a payment, tracking its progress from creation to completion (or failure). Each state reflects the current status of the payment, making it easier to monitor, retry, and prevent invalid operations.

NOT_STARTED → EXECUTING → SUCCESS
                      └→ FAILED

• NOT_STARTED — order record created in DB

• EXECUTING — PayPal order created, waiting for user approval

• SUCCESS — funds captured, ledger updated, event published

• FAILED — capture failed or user cancelled

Prerequisites

Before you start, make sure you have:

• Node.js 18+

• Docker and Docker Compose

• NestJS basics

• A PayPal Developer account

• Basic understanding of gRPC and message queues

PayPal Concepts You Need to Know

Before integrating PayPal, it's helpful to understand a few core concepts. PayPal provides separate environments for development and production, along with an order-based payment workflow that your application follows.

Sandbox vs Live

Always develop in sandbox. Switch to live only in production.

Orders API Flow (What We Use)

PayPal's Orders v2 API follows three steps:

1. Create Order: your backend creates an order with amount and return URLs

2. Approve: user is redirected to PayPal and approves payment

3. Capture: your backend captures the approved funds

This is different from the older Payments REST API. Orders v2 is the recommended approach for new integrations.

Environment Variables

The PayPal service reads its configuration from environment variables. This keeps sensitive credentials out of your source code and makes it easy to switch between sandbox and production environments.

PAYPAL_CLIENT_ID=your_client_id
PAYPAL_CLIENT_SECRET=your_client_secret
PAYPAL_API_BASE=https://api-m.sandbox.paypal.com   # or https://api-m.paypal.com for live

Never commit real credentials to Git. Use .env files and Docker environment injection.

Project Structure

apps/
├── core/
│   └── payment-service/          # Owns all PayPal logic
│       ├── src/
│       │   ├── app/payment/
│       │   │   ├── paypal/paypal.service.ts
│       │   │   ├── payment.service.ts
│       │   │   ├── payment.grpc.controller.ts
│       │   │   ├── payment.http.controller.ts
│       │   │   └── events/payment-events.publisher.ts
│       │   ├── migrations/       # DB schema
│       │   └── routes/health.routes.ts
│       └── Dockerfile
├── services/
│   └── students-service/         # Domain service example
│       └── src/app/payment/
│           ├── payment-client.service.ts      # gRPC client
│           ├── application-payment.service.ts # business logic
│           └── payment-events.consumer.ts     # RabbitMQ listener
└── gateways/
    └── student-apigw/            # HTTP API for frontend
libs/
└── shared/dto/src/lib/payment/
    └── payment.proto             # Shared gRPC contract

Step 1 — Create the Payment Service

The payment service runs two servers in one process

The payment service exposes both an HTTP server and a gRPC server in the same NestJS application. The HTTP server handles health checks, webhooks, and external requests, while the gRPC server accepts internal requests from other microservices.

// apps/core/payment-service/src/main.ts

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // Health route (outside /api prefix)
  app.use('/health', healthRouter);

  // gRPC microservice
  app.connectMicroservice<MicroserviceOptions>({
    transport: Transport.GRPC,
    options: {
      package: 'payment',
      protoPath: join(process.cwd(), 'libs/shared/dto/src/lib/payment/payment.proto'),
      url: `0.0.0.0:${process.env.GRPC_PORT || '50061'}`,
    },
  });

  app.setGlobalPrefix('api');
  await app.startAllMicroservices();
  await app.listen(process.env.PORT || 3003);
}

During startup, NestJS initializes both servers, allowing external clients and internal services to communicate through the appropriate protocol.

Key design choice: HTTP is for external/webhook traffic. gRPC is for fast, typed internal calls between services.

Step 2 — Define the gRPC Contract

Next, you'll create a shared .proto file so all services speak the same language:

A gRPC contract defines the API shared between microservices. Using a .proto file ensures that every service communicates with the payment service using the same request and response structure, regardless of the programming language.

// libs/shared/dto/src/lib/payment/payment.proto

syntax = "proto3";
package payment;

service PaymentService {
  rpc CreatePayment(CreatePaymentRequest) returns (CreatePaymentResponse) {}
  rpc CapturePayment(CapturePaymentRequest) returns (CapturePaymentResponse) {}
  rpc GetPaymentStatus(GetPaymentStatusRequest) returns (GetPaymentStatusResponse) {}
  rpc ListPayments(ListPaymentsRequest) returns (ListPaymentsResponse) {}
}

message CreatePaymentRequest {
  string checkout_id = 1;
  string payment_order_id = 2;
  string domain = 3;           // e.g. "application", "subscription"
  string reference_id = 4;     // business entity ID
  string payer_id = 5;
  string amount = 6;
  string currency = 7;
  string buyer_email = 8;
  string seller_account = 9;
  string payment_category = 10;
  string return_url = 11;      // PayPal redirect on success
  string cancel_url = 12;      // PayPal redirect on cancel
  string idempotency_key = 13;
  string metadata = 14;
  string description = 15;
}

message CreatePaymentResponse {
  int32 status = 1;
  string message = 2;
  string payment_order_id = 3;
  string paypal_order_id = 4;
  string approve_url = 5;      // Redirect user here
  string payment_order_status = 6;
}

The domain + reference_id pair lets one payment service handle payments for applications, subscriptions, university fees, and more without coupling to any single business model.

Step 3 — Implement the PayPal Service

Now, you'll create a dedicated PayPalService that wraps the PayPal REST API.

Instead of calling the PayPal API throughout the application, we encapsulate all PayPal communication inside a dedicated service. This keeps authentication, order creation, and payment capture logic centralized and easier to maintain.

// apps/core/payment-service/src/app/payment/paypal/paypal.service.ts

@Injectable()
export class PayPalService {
  private accessToken: string | null = null;
  private tokenExpiresAt = 0;

  private get apiBase(): string {
    return this.configService.get('PAYPAL_API_BASE')
      || 'https://api-m.sandbox.paypal.com';
  }

  // Step 1: Get OAuth access token (cached until expiry)
  private async getAccessToken(): Promise<string> {
    const now = Date.now();
    if (this.accessToken && now < this.tokenExpiresAt) {
      return this.accessToken;
    }

    const response = await axios.post(
      `${this.apiBase}/v1/oauth2/token`,
      'grant_type=client_credentials',
      {
        auth: {
          username: this.configService.get('PAYPAL_CLIENT_ID'),
          password: this.configService.get('PAYPAL_CLIENT_SECRET'),
        },
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
      }
    );

    this.accessToken = response.data.access_token;
    this.tokenExpiresAt = now + (response.data.expires_in - 60) * 1000;
    return this.accessToken;
  }

  // Step 2: Create PayPal checkout order
  async createOrder(input: PayPalCreateOrderInput) {
    const token = await this.getAccessToken();

    const response = await axios.post(
      `${this.apiBase}/v2/checkout/orders`,
      {
        intent: 'CAPTURE',
        purchase_units: [{
          custom_id: input.paymentOrderId,
          description: input.description,
          amount: {
            currency_code: input.currency,
            value: input.amount,
          },
        }],
        application_context: {
          return_url: input.returnUrl,
          cancel_url: input.cancelUrl,
          brand_name: 'YourApp',
          user_action: 'PAY_NOW',
        },
      },
      {
        headers: {
          Authorization: `Bearer ${token}`,
          'PayPal-Request-Id': input.idempotencyKey,
        },
      }
    );

    const paypalOrderId = response.data.id;
    const approveUrl = response.data.links
      ?.find((l) => l.rel === 'approve')?.href;

    return { paypalOrderId, approveUrl };
  }

  // Step 3: Capture approved order
  async captureOrder(paypalOrderId: string) {
    const token = await this.getAccessToken();

    const response = await axios.post(
      `${this.apiBase}/v2/checkout/orders/${paypalOrderId}/capture`,
      {},
      { headers: { Authorization: `Bearer ${token}` } }
    );

    const capture = response.data.purchase_units?.[0]?.payments?.captures?.[0];
    return { status: response.data.status, captureId: capture?.id || '' };
  }
}

On startup, log configuration (with masked secrets) so you can verify sandbox vs live at a glance:

PayPal configuration check:
  PAYPAL_API_BASE: https://api-m.paypal.com
  PAYPAL_CLIENT_ID: AQb2...aq1M (80 chars)
  credentialsPresent: true
  environment: live

Notice that the access token is cached until it expires. This avoids requesting a new OAuth token for every payment, improving performance and reducing unnecessary API calls.

Step 4 — Build the Payment Flow (Create, Approve, Capture)

Create Payment

PaymentService.createPayment() does the following:

1. Checks idempotency key and returns an existing order if one is already created

2. Creates a payment_events checkout record

3. Creates a payment_orders row with status NOT_STARTED

4. Calls PayPalService.createOrder()

5. Updates order status to EXECUTING

6. Returns approveUrl to the caller

async createPayment(input: CreatePaymentPayload) {
  // Idempotency: prevent duplicate charges
  const existing = await this.paymentOrderModel.findOne({
    where: { idempotencyKey: input.idempotencyKey },
  });
  if (existing) return this.buildCreateResponse(existing);

  const order = await this.paymentOrderModel.create({
    paymentOrderId: input.paymentOrderId,
    amount: input.amount,
    currency: input.currency,
    paymentOrderStatus: PaymentOrderStatus.NOT_STARTED,
    domain: input.domain,
    referenceId: input.referenceId,
    // ...
  });

  const paypalOrder = await this.paypalService.createOrder({
    paymentOrderId: order.paymentOrderId,
    amount: input.amount,
    currency: input.currency,
    returnUrl: input.returnUrl,
    cancelUrl: input.cancelUrl,
    idempotencyKey: input.idempotencyKey,
  });

  await order.update({
    paymentOrderStatus: PaymentOrderStatus.EXECUTING,
    paypalOrderId: paypalOrder.paypalOrderId,
  });

  return {
    approveUrl: paypalOrder.approveUrl,
    paypalOrderId: paypalOrder.paypalOrderId,
    paymentOrderStatus: PaymentOrderStatus.EXECUTING,
  };
}

User Approves on PayPal

The frontend redirects the user to approveUrl. PayPal handles authentication and approval, then redirects back to your returnUrl.

Capture Payment

After approval, call capturePayment() with either paymentOrderId or paypalOrderId:

async capturePayment(paymentOrderId?: string, paypalOrderId?: string) {
  const order = await this.findOrder(paymentOrderId, paypalOrderId);

  if (order.paymentOrderStatus === PaymentOrderStatus.SUCCESS) {
    return this.buildCaptureResponse(order); // already captured
  }

  const capture = await this.paypalService.captureOrder(order.paypalOrderId);

  if (capture.status !== 'COMPLETED') {
    throw new Error(`PayPal capture status: ${capture.status}`);
  }

  await this.finalizeSuccessfulPayment(order, capture.captureId);
  return this.buildCaptureResponse(order);
}

finalizeSuccessfulPayment() runs in a database transaction:

1. Updates order status to SUCCESS

2. Updates seller wallet balance

3. Creates ledger entries (audit trail)

4. Mark scheckout event as done

5. Publishes a payment.{domain}.completed event to RabbitMQ

Step 5 — Connect Domain Services via gRPC

Domain services (like students-service) never talk to PayPal directly. They use a gRPC client:

The Students Service communicates with the Payment Service through a gRPC client. Rather than calling the PayPal API directly, it invokes strongly typed remote procedures exposed by the payment service.

// apps/services/students-service/src/app/payment/payment.module.ts

ClientsModule.registerAsync([{
  name: 'PAYMENT_SERVICE',
  useFactory: () => ({
    transport: Transport.GRPC,
    options: {
      package: 'payment',
      protoPath: 'libs/shared/dto/src/lib/payment/payment.proto',
      url: process.env.PAYMENT_SERVICE_URL || 'payment-service:50061',
    },
  }),
}])

// payment-client.service.ts

@Injectable()
export class PaymentClientService implements OnModuleInit {
  private paymentService: PaymentGrpcService;

  constructor(@Inject('PAYMENT_SERVICE') private client: ClientGrpc) {}

  onModuleInit() {
    this.paymentService = this.client.getService('PaymentService');
  }

  async createPayment(data: CreatePaymentRequest) {
    return firstValueFrom(this.paymentService.CreatePayment(data));
  }

  async capturePayment(data: { payment_order_id?: string; paypal_order_id?: string }) {
    return firstValueFrom(this.paymentService.CapturePayment(data));
  }
}

Domain Service Business Logic Example:

This example shows how a domain service prepares business-specific data before delegating payment processing to the Payment Service.

// application-payment.service.ts

async initiateTuitionPayment(applicationId: number, options: { domain: string }) {
  const application = await this.applicationModel.findByPk(applicationId);

  // Build PayPal return URLs from frontend domain
  const frontendBase = options.domain; // e.g. https://crm.yourapp.com
  const returnUrl = `${frontendBase}/payment/successful?applicationId=${application.applicationId}`;
  const cancelUrl = `${frontendBase}/payment/failure?applicationId=${application.applicationId}`;

  const result = await this.paymentClient.createPayment({
    checkout_id: `checkout-app-${application.id}`,
    payment_order_id: uuidv4(),
    domain: 'application',
    reference_id: String(application.id),
    payer_id: application.studentId,
    amount: finalAmount.toFixed(2),
    currency: 'USD',
    buyer_email: buyerEmail,
    seller_account: `university-${application.universityId}`,
    payment_category: 'tuition_deposit',
    return_url: returnUrl,
    cancel_url: cancelUrl,
    idempotency_key: `app-${application.id}-tuition-${uuidv4()}`,
  });

  return {
    approveUrl: result.approve_url,
    paypalOrderId: result.paypal_order_id,
    paymentOrderId: result.payment_order_id,
  };
}

The domain service remains responsible for business rules, while the payment service handles the payment workflow itself.

Important: The frontend must send its own origin as domain so return URLs point to the correct environment (localhost in dev, production URL in prod).

Step 6 — Add the API Gateway Layer

The API gateway exposes HTTP endpoints to the frontend and forwards to domain services:

POST /applications/:id/pay/applicationfee
Body: { "domain": "https://crm.yourapp.com", "couponCode": "SAVE10" }

// student-apigw → students-service (gRPC) → payment-service (gRPC) → PayPal

Gateway responsibilities:

• Authentication (JWT)

• Request validation

• No PayPal credentials

Capture the endpoint after the PayPal redirect:

POST /applications/:id/pay/applicationfee/capture
Body: { "paypalOrderId": "PAYPAL_ORDER_ID_FROM_URL" }

Step 7 — Publish Payment Events with RabbitMQ

RabbitMQ enables asynchronous communication between services. Instead of waiting for every service to finish processing after a payment succeeds, the payment service simply publishes an event and lets interested services handle it independently.

After a successful capture, the payment service publishes an event:

// payment-events.publisher.ts

async publishCompleted(event: PaymentCompletedEvent) {
  const pattern = `payment.${event.domain}.completed`; // e.g. payment.application.completed
  this.eventsClient.emit(pattern, { ...event, eventId: uuidv4() });
}

Each domain service subscribes to payment events that are relevant to its business domain. For example, the Students Service listens for payment.application.completed events so it can mark student applications as paid.

// payment-events.consumer.ts (students-service)

@EventPattern('payment.application.completed')
async handlePaymentCompleted(@Payload() data: PaymentCompletedPayload) {
  await this.applicationPaymentService.handlePaymentCompletedEvent(data);
  // Marks application as PAID, records payment history
}

This decouples payment completion from domain updates. Even if students-service is temporarily down, you can replay events from the queue.

Two Paths to Mark an Order as Paid

Using both (with idempotency) gives you reliability: the sync path gives immediate UX feedback. The async path is a safety net.

Step 8 — Database Schema and Migrations

The payment service maintains its own database schema. Each table has a specific responsibility, allowing payment records, financial transactions, and webhook processing to remain isolated from other business services.

Production Migration Gotcha

In production Docker images, migration .ts files are not available unless you compile them to JavaScript and copy them into the image:

# Dockerfile — compile migrations for production
RUN pnpm exec tsc --project apps/core/payment-service/tsconfig.migrations.json
COPY --from=builder /app/dist/apps/core/payment-service/migrations ./migrations

Without this, you'll see Executed 0 migrations in logs and no tables will be created.

Create the database user before first deploy:

CREATE USER payment_user WITH PASSWORD 'payment_pass';
CREATE DATABASE payment_db;
GRANT ALL PRIVILEGES ON DATABASE payment_db TO payment_user;

Step 9 — Local Development Setup (Docker)

Environment Variables (.env)

PAYPAL_CLIENT_ID=your_sandbox_client_id
PAYPAL_CLIENT_SECRET=your_sandbox_client_secret
PAYPAL_API_BASE=https://api-m.sandbox.paypal.com

In this section, we'll configure the payment service for local development using Docker Compose. This setup provides a complete environment for testing payments without deploying to production.

Docker Compose (local)

The following configuration starts the payment service together with its required dependencies, including PostgreSQL and RabbitMQ.

payment-service:
  build:
    dockerfile: apps/core/payment-service/Dockerfile.dev
  ports:
    - '3003:3003'    # HTTP
    - '50061:50061'  # gRPC
  environment:
    - PAYPAL_API_BASE=https://api-m.sandbox.paypal.com
    - PAYPAL_CLIENT_ID=${PAYPAL_CLIENT_ID}
    - PAYPAL_CLIENT_SECRET=${PAYPAL_CLIENT_SECRET}
    - DB_HOST=postgres
    - DB_NAME=payment_db
    - DB_USER=payment_user
    - DB_PASSWORD=payment_pass
    - RABBITMQ_URL=amqp://rabbitmq:5672

students-service:
  environment:
    - PAYMENT_SERVICE_URL=payment-service:50061
  depends_on:
    payment-service:
      condition: service_healthy

Start Services

Once the configuration is complete, start the containers and verify that every service is running correctly before testing the payment flow.

docker compose up -d payment-service students-service student-apigw

Verify Health

curl http://localhost:3003/health
# {"status":"healthy","service":"payment-service",...}

Test Payment Flow

1. Call POST /applications/:id/pay/applicationfee with { "domain": "http://localhost:3000" }

2. Open the returned approveUrl in a browser

3. Log in with a PayPal sandbox buyer account

4. After approval, call POST /applications/:id/pay/applicationfee/capture with the paypalOrderId

5. Confirm application status is PAID

Step 10 — Production Deployment

After verifying everything locally, the next step is deploying the payment service to production. The main differences are using PayPal Live credentials, production environment variables, and production-ready Docker images.

PayPal Live Credentials

1. Go to PayPal Developer Dashboard → Live apps

2. Create a Live REST API app

3. Copy Client ID and Secret

Production .env (on Server — Never Commit)

PAYPAL_CLIENT_ID=your_live_client_id
PAYPAL_CLIENT_SECRET=your_live_secret
PAYPAL_API_BASE=https://api-m.paypal.com

Docker Compose (Production)

payment-service:
  build:
    dockerfile: apps/core/payment-service/Dockerfile
  environment:
    - NODE_ENV=production
    - PAYPAL_API_BASE=${PAYPAL_API_BASE:-https://api-m.paypal.com}
    - PAYPAL_CLIENT_ID=${PAYPAL_CLIENT_ID}
    - PAYPAL_CLIENT_SECRET=${PAYPAL_CLIENT_SECRET}
    - DB_HOST=${DB_HOST}
    - DB_NAME=payment_db
    - DB_USER=payment_user
    - DB_PASSWORD=payment_pass
    - RABBITMQ_URL=amqp://${RABBITMQ_USER}:${RABBITMQ_PASS}@rabbitmq:5672
  labels:
    - 'traefik.http.routers.payment.rule=Host(`payment-service.yourapp.com`)'

students-service:
  environment:
    - PAYMENT_SERVICE_URL=payment-service:50061
  depends_on:
    payment-service:
      condition: service_healthy

Deploy Commands

docker compose -f docker-compose.prod.yml build --no-cache payment-service
docker compose -f docker-compose.prod.yml up -d payment-service students-service

Verify Production

curl https://payment-service.yourapp.com/health

docker logs -f apply-goal-payment-service
# Look for:
#   environment: live
#   Found 8 pending migrations
#   Executed 8 migrations

Frontend Domain in Production

The frontend must send the production CRM URL when initiating payment:

{ "domain": "https://crm.yourapp.com" }

Not localhost. This controls where PayPal redirects after payment.

Step 11 — Health Checks and Monitoring

Health checks allow orchestration tools such as Docker and Traefik to verify that the payment service is running correctly. Monitoring these endpoints helps detect failures early and improves application reliability.

// GET /health
{ "status": "healthy", "service": "payment-service", "timestamp": "...", "version": "1.0.0" }

Used by:

• Docker HEALTHCHECK

• Traefik load balancer

• Uptime monitoring

PayPal credential check runs on startup via PayPalService.logConfiguration().

Complete Request Flow (Real Example)

Scenario: Student pays tuition fee for university application.

1. Frontend
   POST /applications/42/pay/applicationfee
   Body: { "domain": "https://crm.yourapp.com" }
        │
        ▼
2. student-apigw (HTTP → gRPC)
   InitiateApplicationTuitionPayment(applicationId: 42)
        │
        ▼
3. students-service
   - Validates application not already paid
   - Resolves tuition amount
   - Optionally validates coupon via payment-service gRPC
   - Builds returnUrl / cancelUrl from domain
   - Calls payment-service CreatePayment (gRPC)
        │
        ▼
4. payment-service
   - Creates payment_orders record (EXECUTING)
   - Calls PayPal POST /v2/checkout/orders
   - Returns approveUrl
        │
        ▼
5. Frontend redirects user to approveUrl (PayPal checkout)
        │
        ▼
6. User approves → PayPal redirects to returnUrl
        │
        ▼
7. Frontend
   POST /applications/42/pay/applicationfee/capture
   Body: { "paypalOrderId": "PAYPAL_ORDER_ID" }
        │
        ▼
8. payment-service
   - POST /v2/checkout/orders/{id}/capture
   - Updates order → SUCCESS
   - Updates wallet + ledger
   - Publishes payment.application.completed → RabbitMQ
        │
        ▼
9. students-service (event consumer)
   - Marks application paymentStatus = PAID
   - Records payment in application_payments table

Coupon Support (Optional)

Before creating a PayPal order, validate a coupon via gRPC:

const validation = await this.paymentClient.validateCoupon({
  code: 'SAVE20',
  universityId: application.universityId,
  originalAmount: 500,
  paymentType: 'application_fee',
});

const finalAmount = validation.data.finalAmount;

// If coupon covers 100% — skip PayPal entirely
if (finalAmount <= 0) {
  await this.markApplicationPaid(applicationId, { amount: 0, source: 'coupon' });
  return { paymentOrderStatus: 'COMPLETED' };
}

Coupon logic lives in payment-service so discount rules are centralized.

PayPal Webhooks (Optional but Recommended)

Register a webhook URL in the PayPal dashboard:

https://payment-service.yourapp.com/api/v1/payments/webhooks/paypal

The payment service handles:

Webhook events are deduplicated via processed_webhooks table to prevent double-processing.

Testing Checklist

• [ ] GET /health returns 200

• [ ] PayPal logs show credentialsPresent: true

• [ ] Database tables exist after startup (payment_orders, payment_events, etc.)

• [ ] Create payment returns valid approveUrl

• [ ] Sandbox buyer can complete checkout

• [ ] Capture returns payment_order_status: SUCCESS

• [ ] Application marked as PAID in domain service

• [ ] RabbitMQ event payment.application.completed is consumed

• [ ] Duplicate capture is handled gracefully (idempotent)

• [ ] Coupon 100% discount skips PayPal

• [ ] Production uses https://api-m.paypal.com (live)

Wrapping Up

Integrating PayPal in a microservice architecture comes down to a few principles:

1. One payment service owns all PayPal API calls

2. gRPC connects domain services to the payment service internally

3. RabbitMQ broadcasts payment outcomes so domain services stay decoupled

4. Idempotency keys prevent duplicate charges

5. Environment variables switch between sandbox and live — no code changes

6. Migrations must be compiled for production Docker images

7. Frontend sends domain so return URLs work in every environment

This pattern scales: add a new payment type (subscription, agency fee, university service fee) by sending a different domain and payment_category — no changes to PayPal integration code.

Further Reading

• PayPal Orders API v2 Documentation

• PayPal Sandbox Testing Guide

• NestJS Microservices (gRPC)

• NestJS RabbitMQ Transport

• Umzug Database Migrations

Maybe it's a different Node version, maybe an environment variable is missing, or maybe a system dependency doesn't match. You spend an hour debugging something that was never actually a code problem.

Docker fixes this at the root. With Docker, you stop shipping just code. The Node version, dependencies, and config all travel inside the container. Your laptop, a CI server, a production VM — it behaves the same on all of them. No more environment surprises.

In this tutorial, we'll go through all this step by step: a multi-stage Dockerfile, using Docker Compose with PostgreSQL for local development, and a GitHub Actions workflow that pushes a fresh image to Docker Hub on every merge to main.

The complete code for this tutorial is available on GitHub.

Table of Contents

1. Prerequisites

2. The Sample Application

3. Writing the Dockerfile

4. The .dockerignore File

5. The .gitignore File

6. Build and Test the Image Locally

7. Docker Compose for Local Development

8. Automate the Build with GitHub Actions

9. Deploying the Image

10. Wrapping Up

Prerequisites

• Node.js 18+

• Docker Desktop, which you can download at docs.docker.com/get-docker. Windows users need WSL 2 before Docker starts. Open PowerShell as Administrator and run wsl --install. After the restart, Docker Desktop will install without issues.

• A GitHub account

• A Docker Hub account (free at hub.docker.com)

• Some Express.js experience helps, but isn't required

The Sample Application

We're building a task management API with Express and PostgreSQL. Keep in mind the app is just a vehicle to teach you how this works. The Dockerfile and pipeline we set up here work the same way for any Node.js project.

Create the project:

mkdir nodejs-docker-cicd && cd nodejs-docker-cicd
npm init -y
npm install express pg dotenv
npm install --save-dev nodemon

Create src/index.js:

const express = require('express');
const { Pool } = require('pg');
require('dotenv').config();

const app = express();
app.use(express.json());

const pool = new Pool({
  host: process.env.DB_HOST,
  port: process.env.DB_PORT,
  database: process.env.DB_NAME,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
});

// Create table on startup
pool.query(`
  CREATE TABLE IF NOT EXISTS tasks (
    id SERIAL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    completed BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT NOW()
  )
`).catch(console.error);

// Health check — required for Docker HEALTHCHECK and load balancers
app.get('/health', (req, res) => {
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

app.get('/tasks', async (req, res) => {
  try {
    const result = await pool.query('SELECT * FROM tasks ORDER BY created_at DESC');
    res.json(result.rows);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.post('/tasks', async (req, res) => {
  const { title } = req.body;
  if (!title) return res.status(400).json({ error: 'Title is required' });
  try {
    const result = await pool.query(
      'INSERT INTO tasks (title) VALUES ($1) RETURNING *',
      [title]
    );
    res.status(201).json(result.rows[0]);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.patch('/tasks/:id', async (req, res) => {
  const { id } = req.params;
  const { completed } = req.body;
  try {
    const result = await pool.query(
      'UPDATE tasks SET completed = $1 WHERE id = $2 RETURNING *',
      [completed, id]
    );
    if (result.rows.length === 0) return res.status(404).json({ error: 'Task not found' });
    res.json(result.rows[0]);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Server running on port ${PORT}`));

Open package.json and update the "scripts" section:

"scripts": {
  "start": "node src/index.js",
  "dev": "nodemon src/index.js"
}

npm start runs the app directly with Node. npm run dev uses nodemon so the server restarts automatically when you edit a file.

For running without Docker, create a .env file:

DB_HOST=localhost
DB_PORT=5432
DB_NAME=tasksdb
DB_USER=postgres
DB_PASSWORD=yourpassword
PORT=3000

Notice that all database credentials come from environment variables rather than being hardcoded. Swap the variables, and the same image runs against your local database or a production one — no code changes needed. The /health endpoint is what Docker pings to know the app is actually handling requests.

Writing the Dockerfile

Before touching the Dockerfile, there are two terms you'll keep seeing. An image is a packaged, immutable version of your app — Node runtime, code, dependencies, everything together in one artifact. A container is a running instance of that image. One image, many containers, any machine.

Here's the Dockerfile we'll use:

# ── Stage 1: Install dependencies ──────────────────────────────────────────
FROM node:18-alpine AS builder

WORKDIR /app

# Copy package files first — Docker caches this layer separately.
# If you only change src code (not package.json), Docker skips npm ci on rebuild.
COPY package*.json ./
RUN npm ci

COPY . .


# ── Stage 2: Production image ───────────────────────────────────────────────
FROM node:18-alpine AS production

# Create a non-root user — running as root inside a container is a security risk
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodeuser -u 1001

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production

# Copy only the source code from the builder stage (not node_modules or dev files)
COPY --from=builder /app/src ./src

RUN chown -R nodeuser:nodejs /app
USER nodeuser

EXPOSE 3000

# Docker will ping /health every 30s. If it fails 3 times, the container is marked unhealthy.
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1

CMD ["node", "src/index.js"]

This is a multi-stage build. The first stage (builder) installs everything, including dev dependencies. The second stage (production) starts fresh and only copies what the app needs to run. Nodemon, test frameworks, and anything else dev-only never make it into the final image.

The size difference is real. A node:18 Debian image is over 950MB. Switch to node:18-alpine and cut out the dev dependencies, and the final image lands around 150–200MB instead. A smaller image means faster pushes and faster deploys.

npm ci instead of npm install is a deliberate choice for CI/CD. It reads exact versions from package-lock.json and fails hard if the lockfile doesn't match package.json. Every build on every machine installs the exact same versions — no surprises from a dependency that quietly updated overnight.

The nodeuser account exists because containers run as root by default. That's fine until something goes wrong. A non-root user means that an attacker who gets inside the container can't just do whatever they want.

The .dockerignore File

Create .dockerignore before building:

node_modules
npm-debug.log
.env
.git
.gitignore
README.md
Dockerfile
.dockerignore

The node_modules exclusion is the critical one. Your local modules were compiled for your operating system — macOS or Windows binaries won't work inside a Linux container. Excluding them means Docker installs fresh modules during the build, compiled for the correct platform. Without this exclusion, you'd either copy broken binaries into the image or waste time uploading hundreds of megabytes to the build context.

Never put .env in an image. Passwords, API keys, anything sensitive — those go in at runtime as environment variables, never inside the image itself.

The .gitignore File

One more thing before the first commit: a .gitignore. You don't want node_modules or .env tracked:

node_modules/
.env
.env.local
npm-debug.log*
logs/
.DS_Store
Thumbs.db
.vscode/
.idea/
dist/
build/

Build and Test the Image Locally

Open Docker Desktop first and give it a moment. On Windows, you'll see a whale icon in the taskbar that animates while the engine is starting up. Once it goes still, you're good to run Docker commands. If you try to run Docker before the engine is up, you'll hit this:

ERROR: Error response from daemon: Docker Desktop is unable to start

If that happens, quit Docker Desktop. Open PowerShell as Administrator, run wsl --update, and restart. Then go to Control Panel → Programs → Turn Windows features on or off. Both Hyper-V and Virtual Machine Platform need to be checked. After the restart, Docker Desktop should come up fine.

It's worth knowing about this error too:

docker : The term 'docker' is not recognized as the name of a cmdlet, function,
script file, or operable program.

This means that Docker Desktop isn't running or isn't installed. Open it from the Start menu and wait.

Run the build:

docker build -t nodejs-docker-cicd:latest .

The first time takes roughly 30 seconds since Docker has to pull node:18-alpine from the internet. Once that's cached, subsequent builds are much quicker. Both stages will scroll by:

[+] Building 33.1s (17/17) FINISHED
 => [builder 1/5] FROM docker.io/library/node:18-alpine       20.9s
 => [builder 4/5] RUN npm ci                                   3.5s
 => [production 5/7] RUN npm ci --only=production              3.2s
 => [production 7/7] RUN chown -R nodeuser:nodejs /app         3.2s
 => exporting to image                                         1.5s
 => => naming to docker.io/library/nodejs-docker-cicd:latest     0.0s

When you see (17/17) FINISHED the image is built. Check the size:

docker images nodejs-docker-cicd

IMAGE                     ID             DISK USAGE   CONTENT SIZE
nodejs-docker-cicd:latest   c9eed311d999        198MB         47.5MB

CONTENT SIZE (47.5MB) is the compressed size that gets pushed to Docker Hub. DISK USAGE (198MB) is what it takes up on disk locally. Compare that to a node:18 Debian image at 950MB+, and you can see why the Alpine base and multi-stage approach matter.

On subsequent builds, Docker reuses cached layers. Edit only your source files without touching package.json and the npm ci step gets skipped completely. That 33-second first build becomes 3 seconds.

Docker Compose for Local Development

The app needs a database. Setting up PostgreSQL locally means every developer who clones the repo has to do it, too. Docker Compose handles this: one file defines both services, and one command starts them.

Create docker-compose.yml:

services:
  app:
    build:
      context: .
      target: production
    ports:
      - '3000:3000'
    environment:
      DB_HOST: postgres
      DB_PORT: 5432
      DB_NAME: tasksdb
      DB_USER: postgres
      DB_PASSWORD: postgres
      PORT: 3000
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: tasksdb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - '5432:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres']
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:

A few things worth pointing out. DB_HOST is set to postgres. That's the service name, not localhost. Containers on the same Docker network reach each other by service name. Put localhost there and the app tries to connect to itself.

depends_on with condition: service_healthy holds the app back until Postgres actually passes its health check. Skip this and the app starts, tries to connect to a database that isn't ready yet, and crashes. The health check pings pg_isready every 5 seconds. Once it gets a green response, the app container starts.

The named volume postgres_data keeps your data alive between restarts. Run docker compose down and the data is still there next time. Add --volumes to wipe it clean.

Start both services:

docker compose up --build

You'll see PostgreSQL initialize and then the app start. Once you see Server running on port 3000 in the logs, the stack is up.

Open a second terminal to test — leave the compose logs running in the first one.

Linux/macOS:

curl -X POST http://localhost:3000/tasks \
  -H "Content-Type: application/json" \
  -d '{"title": "Learn Docker"}'

curl http://localhost:3000/tasks

curl http://localhost:3000/health

Windows PowerShell: Typing curl in PowerShell runs Invoke-WebRequest, not actual curl. Run curl.exe instead. For JSON bodies, write to a file first:

'{"title": "Learn Docker"}' | Set-Content body.json
curl.exe -X POST http://localhost:3000/tasks -H "Content-Type: application/json" --data `@body.json

curl.exe http://localhost:3000/tasks

curl.exe http://localhost:3000/health

The backtick before @body.json is necessary. PowerShell would otherwise try to interpret @ as a splatting operator rather than passing it to curl as a filename prefix.

You should see responses like these:

# POST /tasks
{"id":1,"title":"Learn Docker","completed":false,"created_at":"2026-07-09T22:21:17.073Z"}

# GET /tasks
[{"id":1,"title":"Learn Docker","completed":false,"created_at":"2026-07-09T22:21:17.073Z"}]

# GET /health
{"status":"ok","timestamp":"2026-07-09T22:11:44.700Z"}

The task hit PostgreSQL in one container and came back through the app. Ctrl+C in the compose terminal stops both.

Automate the Build with GitHub Actions

The image works locally, so it's time to stop doing this by hand.

Step 1: Create a Docker Hub Access Token

Go to hub.docker.com and then Account Settings → Security → New Access Token. Set permission to Read & Write, as read-only breaks the push. The token appears once, so copy it before closing the page.

Security warning: Don't paste this token into a chat, email, or commit. If you expose it by accident, delete it immediately, then make a new one.

Step 2: Add Secrets to Your GitHub Repository

Head to Settings → Secrets and variables → Actions in your repo and add:

• DOCKERHUB_USERNAME — your Docker Hub username

• DOCKERHUB_TOKEN — paste the token here, nowhere else

If you ran into Error: Username and password required, the secrets either aren't saved yet or the names are typed wrong. Both are case-sensitive.

A Node 20 deprecation warning in the logs is normal. It comes from the Docker actions internally, not your code.

Step 3: Create the Workflow File

Create .github/workflows/docker-publish.yml:

name: Build and Push Docker Image

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  IMAGE_NAME: ${{ secrets.DOCKERHUB_USERNAME }}/nodejs-docker-cicd

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to Docker Hub
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=sha-
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          target: production
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

The login step has if: github.event_name != 'pull_request'. This skips authentication on pull requests. PRs from forks don't have access to your secrets, so trying to log in would just fail. The build still runs on PRs to validate your Dockerfile, but the image isn't pushed.

The metadata action generates two tags on every merge to main: latest and a short commit SHA like sha-a1b2c3d. The SHA tag is what makes rollbacks practical. If latest breaks in production, you can pull any previous sha- tag and you're back to a known-good state in seconds.

The cache-from/cache-to: type=gha lines store Docker's layer cache in GitHub Actions' built-in cache. The first run builds everything from scratch. After that, unchanged layers are pulled from cache rather than rebuilt. On a typical Node.js app this brings build time from 2–3 minutes down to under 30 seconds.

Push and Watch it Run

git add .
git commit -m "Add Docker configuration and GitHub Actions workflow"
git push origin main

Go to your repo's Actions tab. You'll see the workflow running in real time. Each step turns green as it completes:

✅ Checkout code
✅ Set up Docker Buildx
✅ Log in to Docker Hub
✅ Extract metadata
✅ Build and push

Green across the board means your image is live on Docker Hub — two tags, latest and a commit SHA like sha-a1b2c3d. Every push to main from here builds and ships automatically.

Deploying the Image

With your image on Docker Hub, you can deploy it to any infrastructure:

Any VPS or server:

docker pull yourusername/nodejs-docker-cicd:latest
docker run -d -p 3000:3000 \
  -e DB_HOST=your-db-host \
  -e DB_NAME=tasksdb \
  -e DB_USER=postgres \
  -e DB_PASSWORD=yourpassword \
  yourusername/nodejs-docker-cicd:latest

Railway — Connect your Docker Hub image in the Railway dashboard and it deploys on the next push.

Fly.io — Run fly launch pointing at your Dockerfile and Fly handles the rest.

Render — Paste your Docker Hub image URL into the Render service settings.

Each push to main runs the workflow. New image goes to Docker Hub, platform picks it up — that's your deployment handled.

Wrapping Up

What started as a local Node.js app now runs in a container. You get the same behavior on any machine, real PostgreSQL in development, and a pipeline that builds and ships to Docker Hub without you doing anything after the push.

The multi-stage build keeps the image lean — dev tools stay out, non-root user, health check baked in. Compose gets the full stack up with one command for anyone who clones the repo. The SHA tag on every GitHub Actions build means rolling back is just a matter of pulling an older tag.

These same patterns (multi-stage builds, Compose for local development, automated image publishing) are used across the industry for production Node.js deployments. Pick up these patterns once and they follow you to every project.

From here, you can extend the pipeline: drop a test step in before the build, or add multi-platform support if you're targeting ARM. Once Docker Compose starts feeling limiting in production, that's usually when Kubernetes enters the picture.

The ability to scale applications instantly and ship software reliably is an important skill. Containerization is at the heart of modern development.

This hands-on, structured course is designed to take you from absolute scratch to becoming job-ready. Taught by instructor Eissa from DolfinEd, who brings over 25 years of industry experience and 21 years of teaching expertise, this course breaks down complex concepts into simple, actionable skills.

This is a complete, step-by-step practical course that covers everything you need to master Docker:

• Foundations: Understand the shift from legacy physical servers to virtual machines and containers.

• Core Skills: Master Docker files, image creation, and how to manage repositories using Docker Hub.

• Networking & Storage: Learn the gold standards for managing container networking, storage, and volumes.

• Orchestration: Move beyond basic containers by learning how to deploy multi-container applications with Docker Compose and get an introduction to Docker Swarm.

• Real-World Application: Put your skills to the test with structured quizzes, module assignments, and real-world projects that mirror professional environments.

Watch the full course now and start your journey to becoming a Docker expert (7-hour watch):

Your production will keep using a managed S3 / Cloudflare R2 / Hetzner Object Storage, while every staging upload, download, and pre-signed URL goes to your own server for free.

Table of Contents

• 1. Why Self‑Host Object Storage on Staging?

• 2. The Architecture: Production vs. Staging

• 3. Prerequisites

• 4. Step 1 — DNS: Point Your Domains to the Staging Server

• 5. Step 2 — Run MinIO with Docker Compose

• 6. Step 3 — Expose MinIO over HTTPS with Traefik

• 7. Step 4 — Create the Bucket and Access Keys

• 8. Step 5 — Configure Your App to Use MinIO on Staging Only

• 9. Step 6 — Upload Files (3 Ways)

• 10. Step 7 — Generate Presigned URLs (PUT and GET)

• 11. Step 8 — Get Public URLs for Documents

• 12. Step 9 — Lock Down CORS, Lifecycle, and Security

• 13. Step 10 — Backups and Monitoring

• 14. Troubleshooting Cheat Sheet

• 15. Wrapping Up

1. Why Self‑Host Object Storage on Staging?

If your app handles documents — PDFs, profile pictures, application transcripts, recordings — every test upload your QA team makes costs real money on AWS S3, Cloudflare R2, or Hetzner Object Storage. The price isn't huge per file, but staging is where you:

• run automated end‑to‑end tests that upload thousands of dummy files,

• reset databases nightly (which leaves orphan objects behind),

• let developers experiment with broken code that re‑uploads the same files,

• and hold months of test data nobody ever deletes.

In production those costs are justified. Managed storage gives you replication, availability, and someone else's pager. In staging, those costs are pure waste.

MinIO is a free, open‑source, S3‑compatible object server. Same API, same SDKs, same presigned URLs, same mc/aws s3 CLIs — but running on your own VPS, billed at $0 per gigabyte. Point your staging app at MinIO, point your production app at S3/R2, and the only thing that changes is an environment variable.

The result: identical code paths in both environments, zero storage bill on staging, and a nice fallback if your cloud provider ever has an outage.

2. The Architecture: Production vs. Staging

In real-world applications, you usually don’t want your development or staging environment writing directly to production storage.

A common and cost-effective setup is:

• Production: managed cloud object storage

• Staging / Development: self-hosted S3-compatible storage

The good part is that your application code doesn't need to change.

As long as both services are S3-compatible, the same SDK and upload logic work everywhere. Only the environment variables differ.

High-Level Architecture

The above diagram illustrates how the same application can communicate with different storage providers depending on the deployment environment.

In the production environment, uploads are stored in a managed object storage service such as AWS S3, Cloudflare R2, or Hetzner Object Storage. These services handle durability, scalability, backups, and infrastructure management.

In the staging environment, uploads are directed to a self-hosted MinIO instance running inside Docker on a VPS. MinIO implements the S3 API, making it behave similarly to production storage while keeping costs low.

Because both storage systems are S3-compatible, the application uses the same upload logic in every environment. The only difference is the configuration provided through environment variables.

Why This Architecture Is Useful

This setup gives you:

• A cheap staging environment

• Production-like testing

• Zero storage vendor lock-in

• The ability to switch providers without rewriting application code

Because both environments speak the S3 protocol, your upload logic remains identical.

Example Environment Variables

Your application only reads environment variables like these:

S3_ENDPOINT=
S3_REGION=
S3_ACCESS_KEY=
S3_SECRET_KEY=
S3_BUCKET=

Switch the values, and the exact same application now uploads files to a different backend.

Production Storage Example

In production, you typically use managed object storage providers such as:

• AWS S3

• Cloudflare R2

• Hetzner Object Storage

Example:

S3_ENDPOINT=https://<region>.r2.cloudflarestorage.com

The benefits are that it's highly scalable, globally available, durable, has managed backups, and doesn't have infrastructure maintenance.

Staging Environment Example

For staging, a lightweight self-hosted MinIO container is often enough.

Next.js App
     ↓
MinIO Container (inside Docker on VPS)

Example domains:

This allows you to:

• Test uploads safely

• Avoid production storage costs

• Reproduce production-like behavior locally

3. Prerequisites

You'll need:

• A Linux VPS (Hetzner, DigitalOcean, Contabo, OVH — anything with a public IP).

• Two A records pointing at that IP (we'll register them next).

• Docker + Docker Compose v2.

• Traefik v2 in front, with Let's Encrypt configured (any reverse proxy works – the labels below are Traefik's flavor).

• Open ports 80 and 443 on the firewall for Let's Encrypt + HTTPS.

• ~10 GB free disk for the MinIO data volume to start.

If Docker isn't installed:

curl -fsSL https://get.docker.com | sh
sudo apt-get install -y docker-compose-plugin
docker --version && docker compose version

4. Step 1 — DNS: Point Your Domains to the Staging Server

In your DNS provider (Cloudflare, Route 53, Namecheap, and so on), create two A records pointing at your staging server's public IP:

minio-staging.domain.com           A    203.0.113.45
minio-console-staging.domain.com   A    203.0.113.45

If you use Cloudflare, set the proxy status to DNS only (gray cloud) for minio-staging.*. Cloudflare's free plan caps uploads at 100 MB, and you don't want it stripping S3 signing headers. The console subdomain can stay proxied if you want a WAF in front of it.

Wait a minute and verify:

dig +short minio-staging.domain.com
# 203.0.113.45

5. Step 2 — Run MinIO with Docker Compose

Add this service to your staging compose file (docker-compose.staging.yml). MinIO is just one container — the disk is mounted as a Docker volume so data survives upgrades.

# docker-compose.staging.yml
networks:
  proxy:
    external: true
    name: proxy
  internal:
    name: internal

volumes:
  minio-data:

services:
  minio:
    image: minio/minio:latest
    container_name: minio-staging
    restart: unless-stopped
    environment:
      - MINIO_ROOT_USER=${MINIO_ROOT_USER:-admin}
      - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-change-me-please}
      # Tell MinIO which public domain to sign URLs with
      - MINIO_SERVER_URL=https://minio-staging.domain.com
      - MINIO_BROWSER_REDIRECT_URL=https://minio-console-staging.domain.com
    command: server /data --console-address ":9001"
    volumes:
      - minio-data:/data
    networks:
      - proxy
      - internal
    ports:
      - "9000:9000"  # S3 API
      - "9001:9001"  # Web console
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

Two things deserve attention:

• MINIO_SERVER_URL is the secret sauce. Without it, MinIO signs presigned URLs using its internal hostname (http://minio:9000), which then fails verification when the browser hits the public domain. Set it to the exact HTTPS URL clients will use.

• MINIO_BROWSER_REDIRECT_URL does the same for the web console (login redirects, OIDC callbacks, and so on).

Bring it up:

docker compose -f docker-compose.staging.yml up -d minio
docker compose -f docker-compose.staging.yml logs -f minio

You should see API: http://... and Console: http://... lines.

6. Step 3 — Expose MinIO over HTTPS with Traefik

We don't expose ports 9000/9001 to the world directly — Traefik does that for us, terminating TLS with a free Let's Encrypt certificate.

Add these labels to the minio service:

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # ---- S3 API (port 9000) ----
      - "traefik.http.routers.minio-staging.rule=Host(`minio-staging.domain.com`)"
      - "traefik.http.routers.minio-staging.entrypoints=websecure"
      - "traefik.http.routers.minio-staging.tls.certresolver=letsencrypt"
      - "traefik.http.routers.minio-staging.service=minio-staging"
      - "traefik.http.services.minio-staging.loadbalancer.server.port=9000"

      # ---- Web Console (port 9001) ----
      - "traefik.http.routers.minio-console-staging.rule=Host(`minio-console-staging.domain.com`)"
      - "traefik.http.routers.minio-console-staging.entrypoints=websecure"
      - "traefik.http.routers.minio-console-staging.tls.certresolver=letsencrypt"
      - "traefik.http.routers.minio-console-staging.service=minio-console-staging"
      - "traefik.http.services.minio-console-staging.loadbalancer.server.port=9001"

You also need an entrypoint for :443 and a certificatesresolver named letsencrypt. Here's the minimum Traefik config (traefik.staging.yml):

api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: web
      email: admin@domain.com
      storage: /etc/traefik/acme.json

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

Restart and watch the cert get issued:

docker compose -f docker-compose.staging.yml up -d
docker compose -f docker-compose.staging.yml logs -f traefik | grep -i acme

Sanity check from your laptop:

curl -I https://minio-staging.domain.com/minio/health/live
# HTTP/2 200

You can now log in to the web console at https://minio-console-staging.domain.com with admin / change-me-please.

Important upload size tweak: if you're behind Cloudflare or NGINX in front of Traefik, raise the request body limit. Traefik itself has no default limit, but Cloudflare's free plan refuses anything over 100 MB. For self‑hosted edge proxies, set client_max_body_size 0; (NGINX) or the equivalent.

7. Step 4 — Create the Bucket and Access Keys

Anything that speaks S3 can talk to MinIO. The easiest tool is mc (the official MinIO client), shipped inside the same image.

7.1 Connect mc to your server

docker exec -it minio-staging \
  mc alias set local http://localhost:9000 admin change-me-please

7.2 Create a bucket

docker exec -it minio-staging mc mb local/domain-files-staging

7.3 Choose a bucket policy

You have three choices, so just pick based on what you store:

Set one:

# Private (recommended for documents)
docker exec -it minio-staging \
  mc anonymous set none local/domain-files-staging

# OR public read for static assets only:
docker exec -it minio-staging \
  mc anonymous set download local/domain-files-staging

7.4 Create a dedicated app user (don't use root keys!)

The admin account can wipe everything. Make a least‑privilege user for your app:

docker exec -it minio-staging mc admin user add local \
  domain-app a-long-random-secret-key

# Attach the built-in read/write policy, scoped to one bucket via JSON:
cat > /tmp/policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": [
        "arn:aws:s3:::domain-files-staging",
        "arn:aws:s3:::domain-files-staging/*"
      ]
    }
  ]
}
EOF

docker cp /tmp/policy.json minio-staging:/tmp/policy.json
docker exec -it minio-staging \
  mc admin policy create local domain-rw /tmp/policy.json
docker exec -it minio-staging \
  mc admin policy attach local domain-rw --user domain-app

Save those two values — they are your S3_ACCESS_KEY and S3_SECRET_KEY.

8. Step 5 — Configure Your App to Use MinIO on Staging Only

The trick to "MinIO in staging, real S3 in prod" is to use the same S3 client in your code and only swap the env vars.

Your staging.env (loaded by your staging compose stack):

# ---- Staging: self-hosted MinIO ----
STORAGE_ENABLED=true
S3_ENDPOINT=https://minio-staging.domain.com
S3_PUBLIC_ENDPOINT=https://minio-staging.domain.com
S3_BUCKET=domain-files-staging
S3_ACCESS_KEY=domain-app
S3_SECRET_KEY=a-long-random-secret-key
S3_REGION=us-east-1
S3_FORCE_PATH_STYLE=true

Your production.env:

# ---- Production: Cloudflare R2 ----
STORAGE_ENABLED=true
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
S3_PUBLIC_ENDPOINT=https://files.domain.com
S3_BUCKET=domain-files
S3_ACCESS_KEY=<r2-access-key>
S3_SECRET_KEY=<r2-secret-key>
S3_REGION=auto
S3_FORCE_PATH_STYLE=true

S3_FORCE_PATH_STYLE=true is critical for both MinIO and R2/Hetzner. Without it, the SDK tries https://bucket.minio-staging.domain.com (virtual‑host style), which won't resolve.

Now in your application code (Node.js example using AWS SDK v3):

// src/lib/s3.js
import { S3Client } from "@aws-sdk/client-s3";

export const s3 = new S3Client({
  endpoint: process.env.S3_ENDPOINT,
  region: process.env.S3_REGION,
  credentials: {
    accessKeyId: process.env.S3_ACCESS_KEY,
    secretAccessKey: process.env.S3_SECRET_KEY,
  },
  forcePathStyle: process.env.S3_FORCE_PATH_STYLE === "true",
});

export const BUCKET = process.env.S3_BUCKET;
export const PUBLIC_ENDPOINT = process.env.S3_PUBLIC_ENDPOINT;

The same s3 instance now talks to MinIO on staging and to R2 in production with no code change.

9. Step 6 — Upload Files (3 Ways)

9.1 From a server (best for trusted backends)

import { PutObjectCommand } from "@aws-sdk/client-s3";
import { s3, BUCKET } from "./lib/s3.js";
import { readFile } from "node:fs/promises";

export async function uploadDocument(localPath, key, contentType) {
  const Body = await readFile(localPath);
  await s3.send(new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    Body,
    ContentType: contentType,
    // Optional: per-object metadata, useful for audits
    Metadata: { uploadedBy: "system", env: process.env.NODE_ENV },
  }));
  return key;
}

9.2 With the mc CLI (good for one‑off uploads / migrations)

mc alias set staging https://minio-staging.domain.com domain-app a-long-random-secret-key
mc cp ./report.pdf staging/domain-files-staging/reports/2026/report.pdf
mc ls staging/domain-files-staging --recursive

9.3 Directly from the browser via a presigned PUT URL

The recommended pattern for user uploads is: the file goes from the browser to MinIO with zero bytes touching your API server.

We'll cover this in detail next.

10. Step 7 — Generate Presigned URLs (PUT and GET)

A presigned URL is a regular HTTPS URL with a time‑limited signature in the query string. Anyone with the URL can do exactly the action it was signed for (PUT this object, or GET that object) for the next N minutes — and nothing else.

This is what makes "users upload directly to storage" safe.

10.1 Presigned PUT (for uploads)

// src/lib/presign.js
import { PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { s3, BUCKET } from "./s3.js";
import { randomUUID } from "node:crypto";

export async function presignUpload({ filename, contentType, userId }) {
  const key = `users/\({userId}/\){randomUUID()}-${filename}`;
  const cmd = new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    ContentType: contentType,
  });
  const uploadUrl = await getSignedUrl(s3, cmd, { expiresIn: 60 * 5 }); // 5 min
  return { uploadUrl, key };
}

Wire it to your API:

// POST /api/uploads/presign
app.post("/api/uploads/presign", requireAuth, async (req, res) => {
  const { filename, contentType } = req.body;
  const result = await presignUpload({
    filename,
    contentType,
    userId: req.user.id,
  });
  res.json(result); // { uploadUrl, key }
});

The browser uploads straight to MinIO:

// In your frontend
async function uploadFile(file) {
  const { uploadUrl, key } = await fetch("/api/uploads/presign", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ filename: file.name, contentType: file.type }),
  }).then(r => r.json());

  await fetch(uploadUrl, {
    method: "PUT",
    headers: { "Content-Type": file.type },
    body: file,
  });

  // Persist `key` in your DB so you can retrieve it later
  await fetch("/api/documents", {
    method: "POST",
    body: JSON.stringify({ key, originalName: file.name }),
  });
}

The Content-Type you send during PUT must match the one you signed with, or MinIO will reject the request with SignatureDoesNotMatch. This catches everyone the first time.

10.2 Presigned GET (for downloads)

Same idea, but with GetObjectCommand:

export async function presignDownload(key, expiresIn = 60 * 10) {
  const cmd = new GetObjectCommand({ Bucket: BUCKET, Key: key });
  return getSignedUrl(s3, cmd, { expiresIn });
}

A typical "view document" endpoint:

app.get("/api/documents/:id/url", requireAuth, async (req, res) => {
  const doc = await db.documents.findById(req.params.id);
  if (!doc || !canUserSee(req.user, doc)) return res.sendStatus(403);
  const url = await presignDownload(doc.key, 600);
  res.json({ url });
});

The frontend just opens that URL — the file streams from MinIO directly to the user.

10.3 Why presigned URLs beat "proxy through the API"

11. Step 8 — Get Public URLs for Documents

Sometimes you want a permanent, unauthenticated URL — for example public profile pictures.

If the bucket policy allows anonymous reads (mc anonymous set download …), the public URL pattern is:

https://minio-staging.domain.com/<bucket>/<key>

So users/42/avatar.png becomes:

https://minio-staging.domain.com/domain-files-staging/users/42/avatar.png

In code:

export function publicUrl(key) {
  return `\({process.env.S3_PUBLIC_ENDPOINT}/\){BUCKET}/${key}`;
}

For private buckets (most documents), don't use public URLs at all — always go through presignDownload(key) so you can re‑check authorization on every request and expire links.

12. Step 9 — Lock Down CORS, Lifecycle, and Security

12.1 Allow your frontend origins (CORS)

Browser uploads need CORS rules on the bucket. Drop this JSON via mc:

cat > /tmp/cors.json <<'EOF'
{
  "CORSRules": [
    {
      "AllowedOrigins": [
        "https://crm-staging.domain.com",
        "http://localhost:3000"
      ],
      "AllowedMethods": ["GET", "PUT", "POST", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3000
    }
  ]
}
EOF

docker cp /tmp/cors.json minio-staging:/tmp/cors.json
docker exec -it minio-staging \
  mc cors set local/domain-files-staging /tmp/cors.json

12.2 Auto‑delete old test files (lifecycle)

Staging accumulates junk. Tell MinIO to expire anything older than 30 days:

docker exec -it minio-staging \
  mc ilm rule add --expire-days 30 local/domain-files-staging

12.3 Encrypt at rest

docker exec -it minio-staging \
  mc encrypt set sse-s3 local/domain-files-staging

12.4 Hard rules

• Never ship MINIO_ROOT_USER=admin / MINIO_ROOT_PASSWORD=admin123 to a server reachable from the internet. Generate strong values and store them in your secret manager.

• The root account should be used only by mc admin, never by your app. The app uses a scoped IAM user (Step 7.4).

• Keep the console subdomain behind an IP allow‑list or basic auth via Traefik middleware if it's truly public.

• Rotate the app access keys at least every 90 days.

13. Step 10 — Backups and Monitoring

13.1 Backups: mirror to a cheap cold bucket weekly

Set up a tiny cron job that uses mc mirror to push to Backblaze B2, R2, or another cheap S3 endpoint:

mc alias set b2 https://s3.us-east-005.backblazeb2.com \(B2_KEY \)B2_SECRET
mc mirror --overwrite --remove \
  staging/domain-files-staging \
  b2/domain-staging-backup

Even at $6/TB/month this is essentially free for staging volumes.

13.2 Monitoring with Prometheus

MinIO exposes Prometheus metrics out of the box at /minio/v2/metrics/cluster. Scrape with:

scrape_configs:
  - job_name: minio
    metrics_path: /minio/v2/metrics/cluster
    scheme: https
    static_configs:
      - targets: ["minio-staging.domain.com"]

If you have Grafana, import dashboard ID 13502 for an instant overview (capacity, request rates, latency, error counts).

14. Troubleshooting Cheat Sheet

Useful diagnostics:

# Check MinIO health
curl -I https://minio-staging.domain.com/minio/health/live

# List all objects in a bucket
docker exec -it minio-staging mc ls --recursive local/domain-files-staging

# Tail MinIO logs
docker compose -f docker-compose.staging.yml logs -f minio

# Decode a presigned URL to see what it was signed for
echo "<paste url>" | tr '&' '\n'

15. Wrapping Up

Here's what you have now:

• A free, S3‑compatible object store running on your own staging server.

• Real HTTPS on a real domain (https://minio-staging.domain.com), thanks to Traefik + Let's Encrypt.

• A scoped, least‑privilege application user — root keys stay locked away.

• The same exact code paths in staging and production. Switching between MinIO / R2 / Hetzner / AWS S3 is a four‑variable change in the env file.

• Presigned PUT URLs so users upload straight to storage, bypassing your API.

• Presigned GET URLs so private documents are short‑lived and authorization‑gated.

• Lifecycle rules that nuke old test files automatically.

• Optional weekly mirror to a cold backup bucket.

Production keeps running on managed storage where the SLA matters. Staging now costs you exactly $0 per month per gigabyte uploaded — and you can finally stop telling QA to "delete the test files when you're done."

Further Reading

• MinIO Documentation

• AWS SDK v3 — getSignedUrl

• Traefik v2 Docker provider

• S3 bucket policy reference

If this guide saved your team a few dollars, share it with another team that's still uploading test PDFs to a $90/month S3 bucket. Happy shipping.

But the official WhatsApp Business Cloud API can be slow to onboard, template-restricted for proactive messages, and priced per conversation — which adds up fast at scale.

There's another path: you can run your own WhatsApp HTTP gateway on a small server, connect it to a workflow engine, and keep every message — inbound and outbound — inside infrastructure you control. No monthly conversation fees, no template approvals for routine replies, no third-party middleman holding your customer data.

In this tutorial, you'll build exactly that. By the end, you'll have a WhatsApp bot that:

• Receives every incoming message through a webhook

• Routes messages through an n8n workflow

• Replies automatically based on keywords, AI, or any API call you want

• Runs entirely on your own server, using two open-source tools

You'll use WAHA (WhatsApp HTTP API) as the gateway, and n8n as the workflow engine. Both run in Docker, both are free for self-hosting, and together they cover everything from a simple auto-reply to a full CRM integration.

Table of contents

• What You'll Learn

• Prerequisites

• A Note on Which WhatsApp Account to Use

• WAHA vs the official WhatsApp Business Cloud API

• Part 1: Understanding WAHA

• Part 2: Running WAHA with Docker

• Part 3: Starting a WhatsApp session

• Part 4: Running n8n

• Part 5: Creating the Webhook Trigger in n8n

• Part 6: Wiring WAHA to n8n

• Part 7: Building the first auto-reply

• Part 8: A Second Example — Proactive Booking Confirmations

• Part 9: Going to Production

• Common Pitfalls

• Where to Go Next

What You'll Learn

• How WAHA works under the hood and when to use it instead of the official Cloud API

• How to run WAHA and n8n side by side with Docker Compose

• How to scan the QR code and bind a WhatsApp account to your gateway

• How to connect WAHA's webhook to an n8n workflow

• How to build a keyword-based auto-reply bot

• How to send proactive confirmations from a separate workflow

• How to harden the setup for production (HTTPS, API keys, rate limits, Queue Mode)

Prerequisites

• A Linux server (any VPS works — 2 GB of RAM is enough for a small bot)

• Docker and Docker Compose installed

• A public hostname with DNS pointing at the server, or an ngrok tunnel for local testing

• A WhatsApp account you're willing to dedicate to the bot (more on that below)

• Basic familiarity with JSON and HTTP requests

You don't need prior n8n experience. If you can drag a box and wire it to another box, you can build the flow.

A Note on Which WhatsApp Account to Use

WAHA works by running an actual WhatsApp Web session inside a headless Chromium process. It logs in as a real account — the same way you would open web.whatsapp.com in your browser. Meta doesn't officially endorse this approach for commercial use at scale, and heavy volume from a single number can lead to a ban.

For that reason, use a dedicated number for the bot. Don't use your personal WhatsApp. Get a second SIM, eSIM, or a VoIP number that supports WhatsApp activation. Keep outbound volume reasonable, and you'll be fine for most small-business use cases.

If you plan to send thousands of marketing messages per day, switch to the official WhatsApp Business Cloud API — that's what it exists for. This tutorial is aimed at the middle ground: support bots, order updates, booking confirmations, and similar conversational flows where you need real-time control without enterprise pricing.

WAHA vs the official WhatsApp Business Cloud API

Before writing any code, it helps to understand when each option is the right fit.

Neither is strictly better. If you run a support team for a small business, WAHA is often the pragmatic choice. If you're a bank sending millions of transactional messages, you want the Cloud API. Many teams run both — WAHA for conversational support, Cloud API for bulk transactional traffic.

Part 1: Understanding WAHA

WAHA is an open-source project that wraps WhatsApp Web behind a clean REST API. You POST /api/sendText with a chat ID and a message, and WAHA sends it. You configure a webhook URL, and WAHA POSTs to that URL every time a message arrives.

Under the hood, WAHA spawns a Chromium instance, opens WhatsApp Web, and uses an engine (whatsapp-web.js, NOWEB, or GOWS) to automate the session. Your code doesn't see any of that complexity — you just see an HTTP API.

The project ships in two flavors:

• WAHA Core — free, MIT licensed, one active session per container, community support.

• WAHA Plus — commercial license, multi-session support, priority support, and access to advanced endpoints.

For most developers building a single bot, Core is enough. You can always upgrade later.

Official docs live at waha.devlike.pro. Keep that open in another tab — we'll reference specific endpoints as we go.

Part 2: Running WAHA with Docker

Create a fresh directory for the project:

mkdir whatsapp-bot && cd whatsapp-bot

Create a docker-compose.yml file:

services:
  waha:
    image: devlikeapro/waha:latest
    container_name: waha
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - WAHA_DASHBOARD_ENABLED=true
      - WAHA_DASHBOARD_USERNAME=admin
      - WAHA_DASHBOARD_PASSWORD=change-me-now
      - WHATSAPP_API_KEY=super-secret-key-change-me
      - WHATSAPP_DEFAULT_ENGINE=WEBJS
    volumes:
      - ./waha-sessions:/app/.sessions

A few things to notice:

• The dashboard username and password protect the web UI at http://your-server:3000. Always change the defaults before you expose the port publicly.

• WHATSAPP_API_KEY is the key every HTTP request to WAHA must include in the X-Api-Key header. Treat it like a database password.

• WHATSAPP_DEFAULT_ENGINE=WEBJS uses the mature whatsapp-web.js engine. WAHA also supports NOWEB and GOWS engines with different trade-offs — WEBJS is the safest default for a first deployment.

• The volume mount persists the session across restarts. Without it, every container rebuild forces you to scan the QR code again.

Start the container:

docker compose up -d
docker compose logs -f waha

Within about 20 seconds WAHA finishes booting. Visit http://your-server:3000 and log in with the dashboard credentials.

Part 3: Starting a WhatsApp session

WAHA calls each WhatsApp account a "session." You can have one session at a time on WAHA Core.

From the dashboard, click Start New Session and name it default. WAHA displays a QR code.

On your phone:

1. Open WhatsApp.

2. Tap the three-dot menu (Android) or Settings (iOS).

3. Tap Linked Devices → Link a Device.

4. Point the camera at the QR code on your screen.

Within a few seconds the dashboard shows WORKING status. Your session is live.

You can also do this over the API. Start the session (default is the session name, encoded in the URL path):

curl -X POST http://your-server:3000/api/sessions/default/start \
  -H "X-Api-Key: super-secret-key-change-me"

The call is idempotent — if the session is already running, nothing happens.

Fetch the QR as a PNG:

curl http://your-server:3000/api/default/auth/qr \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Accept: image/png" \
  --output qr.png

Scan and you're in.

Test that the session works by sending a message to yourself:

curl -X POST http://your-server:3000/api/sendText \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "session": "default",
    "chatId": "15555550123@c.us",
    "text": "Hello from WAHA!"
  }'

Replace 15555550123 with your own number (country code plus number, no +, no spaces, no dashes). The @c.us suffix marks it as an individual chat. Groups use @g.us.

If the message lands on your phone — congratulations. The gateway works.

Part 4: Running n8n

Add an n8n service to your docker-compose.yml alongside WAHA:

services:
  waha:
    # ... existing config

  n8n:
    image: n8nio/n8n:latest
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=n8n.example.com
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - WEBHOOK_URL=https://n8n.example.com/
      - GENERIC_TIMEZONE=UTC
    volumes:
      - ./n8n-data:/home/node/.n8n

Replace n8n.example.com with your real domain. For purely local testing, set:

- N8N_HOST=localhost
- N8N_PROTOCOL=http
- WEBHOOK_URL=http://localhost:5678/

If you want to test webhooks from your laptop without a server, run ngrok http 5678 in another terminal and use the ngrok HTTPS URL as WEBHOOK_URL. n8n uses WEBHOOK_URL to tell external services where to POST — get this wrong and your webhooks will 404.

Start the stack:

docker compose up -d

Visit http://your-server:5678. On the first visit, n8n walks you through creating an owner account (email and password). Every subsequent visit requires that login. For extra safety in production, put n8n behind a reverse proxy with an allow-list or an additional auth layer — we'll set that up later.

Part 5: Creating the Webhook Trigger in n8n

Click Create Workflow. You'll see an empty canvas.

Add a Webhook node and configure it:

• HTTP Method: POST

• Path: whatsapp (this becomes part of the URL)

• Response Mode: Respond Immediately

• Response Data: First Entry JSON

Click Listen for Test Event. n8n shows you two URLs: a test URL and a production URL. Copy the production URL. It looks like this:

https://n8n.example.com/webhook/whatsapp

Not webhook-test — that one only fires while the editor is open. You want webhook.

Part 6: Wiring WAHA to n8n

WAHA can POST to a webhook on every WhatsApp event. Tell it where to send those events.

In the WAHA dashboard, open your session and set the webhook URL. Or do it over the API:

curl -X PUT http://your-server:3000/api/sessions/default \
  -H "X-Api-Key: super-secret-key-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "webhooks": [
        {
          "url": "https://n8n.example.com/webhook/whatsapp",
          "events": ["message", "session.status"]
        }
      ]
    }
  }'

The message event fires on every inbound message. session.status fires when the session connects, disconnects, or reconnects — which is useful for alerting when your bot goes down.

Test it. From another phone, send a WhatsApp message to your bot's number. Head back to the n8n editor. Within a second or two the webhook node lights up with the event data.

The payload looks roughly like this:

{
  "event": "message",
  "session": "default",
  "payload": {
    "id": "false_15555550123@c.us_3EB0...",
    "from": "15555550123@c.us",
    "body": "Hello",
    "timestamp": 1713801234,
    "fromMe": false
  }
}

Everything you need is in payload: who sent it (from), what they said (body), and when (timestamp).

Part 7: Building the first auto-reply

A bot that only listens is boring. Let's make it answer.

You'll build a tiny keyword router: if the user sends hi or hello, the bot greets them. If they send price, it sends a pricing message. Anything else gets a fallback.

After the Webhook node, add a Switch node.

Configure the Switch node:

• Mode: Expression

• Value: {{ $json.payload.body.toLowerCase().trim() }}

• Add routing rules:

  • Rule 1: equals hi — output 0

  • Rule 2: equals hello — output 0

  • Rule 3: equals price — output 1

  • Fallback output: 2

After the Switch, add three HTTP Request nodes, one per output.

Configure each HTTP Request node identically, except for the body text:

• Method: POST

• URL: http://waha:3000/api/sendText (inside the Docker network you can reach WAHA by its service name. From outside use the full public URL)

• Send Headers: on

  • X-Api-Key: super-secret-key-change-me

  • Content-Type: application/json

• Send Body: on

  • Body Content Type: JSON

  • Specify Body: Using JSON

For the greeting node, the JSON body is:

{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "Hi! I'm the bot. Send 'price' to see pricing, or anything else for help."
}

For the pricing node:

{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "Our plans start at $49/month. Reply 'sales' to talk to a human."
}

For the fallback:

{
  "session": "default",
  "chatId": "={{ $('Webhook').item.json.payload.from }}",
  "text": "I didn't catch that. Try 'hi' or 'price'."
}

The ={{ ... }} syntax is an n8n expression — at runtime it pulls values from earlier nodes.

Connect the Switch outputs to their matching HTTP Request nodes. Save the workflow. Click Activate in the top-right.

Send hi to your bot from any phone. It should reply within a second.

Congratulations — you have a WhatsApp bot running entirely on your own infrastructure.

Part 8: A Second Example — Proactive Booking Confirmations

Auto-reply is useful. Proactive outbound is where the value really compounds. Here's a second workflow that sends a booking confirmation whenever a new row lands in a database.

Create a second workflow in n8n. Use one of these triggers:

• Schedule Trigger — poll a database every minute for new rows

• Webhook Trigger — listen for a notification from your booking system

• Database Trigger (Postgres, MySQL, Supabase) — react to inserts in real time

For this example, use a Schedule Trigger set to every minute, followed by a Postgres Execute Query node that reads pending confirmations:

SELECT id, customer_phone, service_name, booking_time
FROM bookings
WHERE confirmation_sent = false
LIMIT 20;

After the Postgres node, add an HTTP Request node pointing to the same WAHA sendText endpoint you used earlier. The body:

{
  "session": "default",
  "chatId": "={{ $json.customer_phone }}@c.us",
  "text": "Hi! Your booking for {{ \(json.service_name }} on {{ \)json.booking_time }} is confirmed. Reply 'change' to reschedule."
}

Finally, add a second Postgres node that marks the booking as sent:

UPDATE bookings
SET confirmation_sent = true, confirmation_sent_at = NOW()
WHERE id = {{ $json.id }};

Activate the workflow. Every minute, n8n pulls pending bookings, sends a WhatsApp confirmation, and marks them done.

This pattern generalizes. Replace the SQL with a call to Shopify for order confirmations, Stripe for receipt messages, or Calendly for appointment reminders. The WhatsApp layer stays the same — only the source of truth changes.

Part 9: Going to Production

The setup above works, but it's not yet production-ready. Here's what to harden before you point real customers at it.

1. Put Everything Behind HTTPS

Never expose n8n or WAHA directly on plain HTTP. Put a reverse proxy in front. Caddy is the easiest choice because it handles Let's Encrypt automatically.

A minimal Caddyfile:

n8n.example.com {
    reverse_proxy n8n:5678
}

waha.example.com {
    reverse_proxy waha:3000
}

Run Caddy as another service in the same Docker Compose. TLS certificates are issued and renewed automatically.

2. Rotate the API Keys

Don't ship super-secret-key-change-me to production. Generate a real key:

openssl rand -hex 32

Put it in a .env file, reference it as ${WHATSAPP_API_KEY} in docker-compose.yml, and add .env to your .gitignore.

3. Rate-limit Outbound Messages

WhatsApp bans accounts that send too many messages too fast. A safe outbound rate for a fresh number is well under 20 messages per minute. For bursty replies, add an n8n Wait node between sends, or queue outgoing messages through a small custom function node that sleeps between requests.

4. Scale n8n with Queue Mode

By default, n8n runs everything in a single process. That's fine for low volume. For higher throughput, switch to Queue Mode:

• Add a Redis container.

• Run one n8n main container (the web UI and webhook receiver).

• Run one or more n8n-worker containers that pull jobs from the queue.

Queue Mode is documented at docs.n8n.io/hosting/scaling/queue-mode/. Setup adds two environment variables (EXECUTIONS_MODE=queue, QUEUE_BULL_REDIS_HOST=redis) and decouples incoming webhooks from workflow execution. The webhook responds in milliseconds while workers chew through the queue in the background.

5. Monitor the Session

WhatsApp Web sessions drop. The phone loses connection, WhatsApp rotates security tokens, or your server reboots. Catch those drops early.

Subscribe to the session.status webhook event in WAHA. When status becomes FAILED or STOPPED, route it to an n8n workflow that posts to Slack, sends an email, or pages you. The faster you know, the faster you recover.

For overall uptime, point something like Uptime Kuma at GET /api/sessions/default on WAHA. If WAHA reports WORKING, you're fine. Anything else triggers an alert.

6. Back Up the Sessions Volume

The waha-sessions directory contains the logged-in state. If you lose it, you have to scan the QR code again — possibly from a phone that's no longer handy. Back it up nightly. A simple cron job with tar and rclone to S3-compatible storage is plenty.

7. Add a Live-Agent Handoff

Not every conversation should stay with the bot. When a user types human — or when your intent classifier can't answer confidently — hand off to a real agent.

Chatwoot is a solid open-source option: it has a dedicated WhatsApp channel, agent inbox, team assignment, and conversation history. The handoff is an n8n branch that stops processing bot replies and forwards the message stream to Chatwoot's API.

Common Pitfalls

A few issues catch almost everyone on their first production deploy.

Webhooks Timing Out

WAHA gives your webhook a few seconds to respond. If your n8n workflow is slow (calling an LLM, hitting a remote API), the webhook times out and WAHA retries, potentially causing duplicate replies.

Fix: make the webhook return 200 immediately and offload the slow work. In n8n, set the Webhook node's Response Mode to Using Respond to Webhook Node, add a Respond to Webhook node as the first step with a 200 and empty body, then do the heavy lifting after that.

Duplicate Messages

WAHA delivers the same message event more than once in edge cases (phone comes back online, session reconnects). Store the payload.id somewhere — Redis, a database, or n8n's static data store — and drop any ID you've already processed.

Messages Arriving Out of Order

The webhook is async, and n8n may parallelize executions. If ordering matters — for example, in a multi-step conversation — key a queue by the sender's chatId and process each sender serially.

Sessions Disconnecting After a Phone Restart

Normal WhatsApp Web behavior. WAHA auto-reconnects, but occasionally the linked-devices list needs a manual refresh. If a session refuses to come back, stop the WAHA container, delete that session's folder under waha-sessions/, start the container again, and rescan the QR.

Your Number Gets Banned

The single biggest cause is rate: a new number blasting hundreds of messages an hour gets flagged fast. Warm up a number slowly — send a normal, human-like volume for the first week. Don't send to strangers unsolicited. Prefer inbound-driven replies over outbound pushes wherever you can.

The Wrong Chat ID Format

WhatsApp individual chats use <number>@c.us and groups use <groupId>@g.us. Don't include the + or spaces in the number. If WAHA returns a 404 when sending, the chat ID is almost always the problem.

Where to Go Next

You now have the foundation. The same two-service stack supports almost any bot you can imagine — you're only limited by what you can build in an n8n workflow.

Some natural next steps:

• Plug in AI replies: Add an OpenAI or Anthropic node after the Webhook, pass the user's message through it with a short system prompt, and send the response back through WAHA. Cap conversation length to prevent runaway token usage.

• Integrate a CRM: Look up the caller's chatId in HubSpot, Pipedrive, or your own database before deciding how to reply. Segment responses by customer tier.

• Send proactive notifications: Appointment reminders, shipping updates, payment receipts, abandoned-cart nudges. Keep the content transactional and expected — unsolicited marketing blasts are the fastest way to a ban.

• Log every conversation: Add a Postgres or Supabase node after the Webhook to persist messages for analytics and customer history. Your future self (and your support team) will thank you.

• Add media handling: WAHA exposes sendImage, sendFile, and sendVoice endpoints. Teach the bot to accept photos for support tickets, or send invoices as PDFs directly inside the chat.

The WhatsApp layer stays the same. Everything interesting happens upstream in the workflow.

If you want to see production examples of n8n and WAHA running at scale — or you need a similar automation built for your business — I'm the founder of Achiya Automation, where we ship WhatsApp, n8n, and Chatwoot integrations. You can find more at achiya-automation.com.

So how do you bundle up your application so that it can run the same way in every local environment? That’s where Docker comes in.

For beginners, Docker isn't always a very easy concept to grasp. But once you get it, I promise that it’s very interesting. So interesting that you’ll want to dockerize every application you lay your hands on.

For this article, a Go application will be our case study. The fundamental concept of containerization as explained here is transferable, so don’t worry too much about how dockerizing applications in another language will look like.

We’ll go through the basics of dockerizing a Go app with just Docker, images and containers, setting up multiple containers in one application with Docker Compose, and the constituent of a Docker Compose file.

By the end of this article, you'll have a basic understanding of what Docker is, what an image or container is, and how to orchestrate multiple, dependent containers with Docker Compose.

What We'll Cover:

1. Prerequisites

2. What is Docker?

3. How to Install Docker

4. What is a Dockerfile?

5. What is Docker Compose?

6. The app Container

7. The database Container

8. The phpMyAdmin Container

9. Running Everything Together

10. Wrapping Up

Prerequisites

You don't need any prior knowledge of Docker to follow this tutorial. This article is written with a beginner POV in mind, so it's okay if the concept is new to you.

In order to be fully engaged and understand the Go coding examples used here, it'll be helpful if you have basic knowledge of Golang. If you already understand how to set up a Go application on your local computer, you're good to go. If not, you can check this article on how to get started coding in Go.

What is Docker?

Imagine that you have a box. In that box, you put your code and everything that it needs to run. That is, the programming language it uses and any other external packages you need to install.

If someone needs your application, you can just hand them the box. You can also hand this box to as many people as you want. They don’t need to install the language or any other thing on their computer because everything they need is already inside the box. So, when they run the application, what they're actually doing is running an instance of that box.

The app is running within the box which is the standard environment. This means for everyone who got the box and “opened it”, the application is going to run the exact same way.

With the help of Docker, apps can run under the same conditions across different systems, and you avoid the problem of “it works on my machine”.

In technical Docker terms, this box is called an image and the running instance is called a container.

An image is a lightweight, standalone, executable package that includes everything needed to run a piece of software. That is, code, runtime, libraries, system tools, and even the operating system.

A container is simply a runnable instance of an image. This represents the execution environment for a specific application.

If all this seems to abstract, don’t worry. We’ll get our hands dirty in a little bit.

How to Install Docker

In order to install Docker, we're going to install Docker Desktop which comes bundled up with the Docker Engine. Docker Destop is a GUI for managing containers, and you'll see how useful it is in subsequent sections.

At the time of writing, I'm using WSL (Windows Sub-system for Linux). If you're doing the same, you'll need to take that into consideration before installing because Docker requires different installation prerequisites and steps for different operating systems.

To install Docker Desktop on WSL,

1. Download and install the windows .exe file

2. Start Docker Desktop from the Start Menu and navigate to settings

3. Select Use WSL 2 based engine from the General tab

4. Click on apply.

That’s it for the WSL installation. If you are running another operating system, the official docs have a list of installation options for you.

What is a Dockerfile?

In order to build your box in the first place, Docker needs to follow a couple of outlined steps. It needs to know the dependencies, the run time, and it also needs to have the source code. All these steps we list in a Dockerfile.

Before we get down to cracking anything, let’s create a working directory and navigate into it.

mkdir go_book_api && cd go_book_api

To intialise the Go module in your application, run the following command:

go mod init go_book_api

This creates a go.mod file to keep track of your project dependencies. In the root of the project, create a cmd directory, and a main.go file in it. This will serve as the entry point of your application. In the main.go file, you can have a simple print statement:

// cmd/main.go
package main

import "fmt"

func main() {
	fmt.Println("Look at me gooo!")
}

Now, go ahead and create a file in the root of your project and call it Dockerfile. This file has no extensions, but your system automatically knows that it's a file for Docker commands.

Go ahead and paste the following in that file, and then we'll go through each of them one by one:

# base image
FROM golang:1.24

# define the working directory
WORKDIR /app

# copy the go.mod and go.sum so that the packages to be installed
# are known in the container. ./ here is the WORKDIR, /app
COPY go.mod ./

# command to install modules
RUN go mod download

# copy source code into working dir
COPY . .

# build
RUN CGO_ENABLED=0 GOOS=linux go build -o /docker-gs-ping ./cmd/main.go

# run the compiled binary when the container starts
CMD ["/docker-gs-ping"]

Most Dockerfiles begin with a base image, which is specified by the FROM keyword. A base image is a foundational template that provides minimal operating system environment, libraries, or dependencies required to build and run an application within a container.

In this case, your base image is golang:1.24 . Your base image could have been an operating system like Linux. In that case. when you ship your code to someone who isn’t running a Linux operating system, they wouldn’t have to worry because they will be running the application in an environment that already has a minimal Linux OS. In the same light, someone who doesn’t have Go installed locally can run your application.

To figure out what base image to use when setting up your Dockerfile, you can always peruse the official Docker Hub repository for published images. For this case, you can check out base images that are officially published by Golang here.

The next step is to define a working directory. Inside your box, you have a filesystem that is almost identical to the ones you’d see on a Linux system. You have folders like /app, /bin , /usr , and /var , and so on. The working directory you've defined in this case is /app, and it's done with the WORKDIR command.

After setting a working directory, you want to copy the go.mod and go.sum file into it, so that Docker knows what dependencies to add into your box.

The COPY command in Docker takes at least two arguments: the source directory(ies), and then the destination directory. In this case, you want to copy go.mod and go.sum into the working directory of your box, /app.

In the box, you'll run a command that downloads and installs all the modules defined in the go.mod file. To run a command in Docker environment, use RUN and then the command, which is go mod download in this case.

The next step is to copy any source code you have into the working directory.

At this point, you have the dependencies and the source code. The last step is to build the Go application into a single executable file which can be run inside your environment (inside the container).

Within the container, you’ll have a compiled binary at /docker-gs-ping, which is as a result of the compilation of the code in your main.go file. The last step is a RUN command that just tells Docker to run the executable binary after building it. It’s a way of saying “once the container starts running, execute this binary file”.

With these steps, Docker will build an image (a box per our analogy) that you can run. To build the image, you can run this command in your terminal:

docker build -t go_book_api .

The docker build command tells Docker to build an image based on the steps in the Dockerfile. -t is the flag for a tag, and this helps you refer to the image later when running the container.

To accompany your tag, you'll provide a name to the image which is go_book_api in this case. The . at the end is important because it tells Docker where the Dockerfile in question is, and the files that you need to copy into your image.

This is what the building looks like in my IDE:

If you check the Images tab on Docker Compose, you'll see that an image is built:

You can host this image on a public image repository platform like Docker Hub, and share it with your friends. They can pull your image, set it up, and run your application even if they don’t have Go installed. All they need to do is get the container running.

If you click on the little play button to the far-right, you can spin up an instance of the image (a container).

You can give a descriptive name to the container (Docker will generate a random one if you don’t), and click on the Run button. Once the container starts running, you're redirected to its log page.

Your container is up and running! You can see that this is a running instance of your application.

What is Docker Compose?

If you were building a simple Go application that needed no external dependencies, the above set-up would be more than sufficient.

In our example here, the application is supposed to be for a book API, so you’d expect that we'd have some service like a database and a database administrator client like phpMyAdmin to visualize or tables.

To set all this up in one file would be a little complicated using just Docker. This is because Docker doesn't allow you to have one base image for Go, another base image for a database, and so on, in one file.

You could use the base image of a small operating system, and then run commands to manually install these other services as dependencies, but this method makes your application hard to maintain and scale. This method isn't advisable because if one dependency crashes, the whole application will collapse instantly.

To remedy this situation, Docker compose allows you to have multiple containers for your application that are connected together. Docker compose handles running the containers in the right order, allows one container to use a folder from another container, or even keep its data in another container – and so on.

Our previous analogy of boxes is the same, except with Docker Compose, we don’t necessarily have only one box anymore:

The point of Docker Compose is to help you orchestrate multiple images needed to run your application. You can think of it as connecting several boxes together.

Following the explanation from before, your application would be running in the Go book api container, the book data we'll create with your application would be stored in the mysql container which is the database, and you can visualize your database with phpMyadmin, which is in the phpMyadmin container.

To see this technically, create a docker-compose.yml file in the root of the project. The name of this file is important, and Docker Compose only accepts filenames such as compose.yml , docker-compose.yml , or docker-compose.yaml. The file extension hints that the commands are written in yaml which is a language mostly used for file configurations.

services:
  app:
    depends_on:
      - database
    build: 
      context: .
    container_name: go_book_api
    hostname: go_book_api
    networks:
      - go_book_api_net
    ports:
      - 8080:8080
    env_file:
      - .env
    
  database:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_PASSWORD: ${DB_PASSWORD}
      MYSQL_USER: ${DB_USER}
    volumes:
      - mysql-go:/var/lib/mysql
    ports:
      - 3356:3306
    networks:
      - go_book_api_net

  phpmyadmin:
    image: phpmyadmin
    restart: always
    ports:
      - 9000:80
    environment:
      PMA_HOST: database
      PMA_ARBITRARY: 1
    depends_on:
      - database
    networks:
      - go_book_api_net

volumes:
  mysql-go:

networks:
  go_book_api_net:
    driver: bridge

At the root level of the docker-compose file, you have services . These are all the containers that are your application needs to run, and in the context of Docker Compose, they're each regarded as a service.

The app Container

 app:
    depends_on:
      - database
    build: 
      context: .
    container_name: go_book_api
    hostname: go_book_api
    networks:
      - go_book_api_net
    ports:
      - 8080:8080
    env_file:
      - .env

The very first container is the app container, which is your Go application. Under the app container, you'll need to define a few parameters that this container also needs to run.

The depends_on attribute controls the start-up and shut-down order of services within a container. This ensures that if container A depends on container B to start, the container B should be started first so that container A can use it. In this case, the database container must be started before the app container. Note that this doesn't mean app will always wait for the database to be ready.

The next attribute which is build tells Docker Compose to build the Docker image from the local project. Since the Dockerfile for your application is in the root of your app, you'll specify the root path with the context attribute as . .

To give a specific name to your container, you'll use container_name. hostname is what other containers will use for communication.

Recall that the point of Docker Compose is to have multiple containers communicating with each other. They do this with the help of networks. So you'll create another attribute, networks, and give it a name, go_book_api_net . To every other container that you want to associate with this app, you're going to specify the same network.

The next attribute is ports . Your application is an API, which means it's running on a backend Go server. To access the API, you'll need to map a local port to a port on the container. You're mapping port 8080 on your computer to port 8080 in the container.

The env_file attribute just tells Docker Compose where to read environment variables from. In this case, you can create a .env file in the root of your project to store important variables that your container will need.

The database Container

  database:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_PASSWORD: ${DB_PASSWORD}
      MYSQL_USER: ${DB_USER}
    volumes:
      - mysql-go:/var/lib/mysql
    ports:
      - 3356:3306
    networks:
      - go_book_api_net

The second container is the database container. Note, that you can give whatever name you choose to your listed services, but giving your containers descriptive names is always a good convention to follow.

For your Go application database, you'll be working with a MySQL database in this case. Your application needs MySQL to run, so you must set it up as one of the services.

Remember that to build a container, you need a base image. Your base image in this case is mysql:8.0 , as you've specified with the image property above. When trying to set up this container, Docker Compose knows to build your database container from this already existing official image.

If you’ve set up a database locally before, you know that configuration is a step you can’t skip. Every database you create needs a user, a password, and the database name. You can set these variables up in the environment property. Instead of hardcoding these values, you can set them up in a .env file, and reference the environmental variables as you've done here.

Database servers usually listen on specific ports for incoming connections, whether the database is running locally or remotely. Just as you specified for your app container, you can set a port for your database and map it to a corresponding port in the container. If you want to access the database locally, you'd do that on port 3356, and all requests are forwarded to port 3306 in the database container.

Once your containers go functional and your application starts running, creating, and storing data in the database, you’ll realise that every time you stop and then restart your containers, you lose the data stored in the database.

To avoid this, you'll need to store your data outside the container. That way, you won't lose the contents of your database every time you stop running your containers.

This is what volumes are for. You can allocate a specific location outside the database container to store all that content. For your volume in this case, the storage location you specified is mysql-go:/var/lib/mysql .

Just as you set the network in your app container above to go_book_api_net, you'll specify the same network for this database container. Since you want the containers to communicate with each other, it makes sense that they're within the same network.

The phpMyAdmin Container

The last container or last service you need (but that is optional) to configure in this case is the phpMyAdmin container. I find it easier having a database client because it lets me easily see the structure and content of my database.

 phpmyadmin:
    image: phpmyadmin
    restart: always
    ports:
      - 9000:80
    environment:
      PMA_HOST: database
      PMA_ARBITRARY: 1
    depends_on:
      - database
    networks:
      - go_book_api_net

The process is almost the same as the previous containers you've configured. You'll start by pulling the official phpmyadmin image from Docker so that your container is built on it.

The restart option here is just so that if you stop and restart the container, phpMyAdmin automatically reloads again.

On the host machine, which is your local environment, you can have access to this service via port 9000 and it maps to port 80 in the container.

As for the environment , PMA_HOST tells phpMyAdmin to connect to a host called database (which is your database container). This works because both containers are on the same network, as you can see in the networks attribute. PMA_ARBITRARY is used so that if you decide to connect to another host (say, you set up a another database in future and still wish to connect via phpMyAdmin), you can do that via the UI.

Your database client depends on the database container, and so you need to specify that in depends_on:

volumes:
  mysql-go:

networks:
  go_book_api_net:
    driver: bridge

The final section of your Docker Compose file is where you declared named values for the volume and network you've used in setting up your containers.

For the volumes, you'll declare a value called mysql-go. To the container where you want to attach this volume, you'll assign a specific storage location. You can see this in use in the database container.

 volumes:
      - mysql-go:/var/lib/mysql

The same concept follows for the network. You have a named network called go_book_api_net that every container within this same network can use. The driver option is used here to specify the network type, and bridge is used for private internal networks.

Running Everything Together

Before Docker Compose, you had one Dockerfile that built a single container for your Go application. With Docker Compose, You’re gonna be building three containers (your application container, the database, and phpMyAdmin), and orchestrating them to work together as one single application.

You can push all this to a platform like GitHub, and someone can clone, start, and run the application without having any of these services (MySQL or PhpMyAdmin) installed locally on their computer. But they do need to have Docker installed.

To build your containers all together, you can use the command docker compose build:

If you check your Docker Compose UI again, we see that a new image has been built, and it corresponds to the app service

To start running the containers, you can use the command docker compose up:

If you navigate to the container tab of Docker Compose, you can see that your containers are up and running:

The main app service, go_book_api, isn’t running because when you run your image, your binary runs and exits almost immediately.

In your main.go, let’s rewrite the code to set up a minimal HTTP handler function that listens on port 8080:

// cmd/main.go
package main

import (
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		_, _ = w.Write([]byte("ok"))
	})

	log.Println("listening on :8080")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatal(err)
	}
}

If you’re new to Go, don’t let the code above bother you too much. All it does it set up a health endpoint with an associated handler function that listens on a port (8080 in this case) and prints “ok”.

In your Dockerfile, let’s add a command to execute the created binary when the container starts:

# run the compiled binary when the container starts
CMD ["/docker-gs-ping"]

After adding this, you'll need to rebuild the containers and start them again. You can see that all containers are running now:

If you click on the go_book_api container, you can see that your server is running on port 8080 as configured:

Since your app is running on port 8080 and you have a /health endpoint set up for it, you can actually visit that endpoint in a browser to see the output “ok”.

Also, if you click on the exposed phpmyadmin port, you can access the database client locally on port 9000. Based on the environment variables set up in the .env file, you can log in.

Another interesting thing to look for on Docker desktop is volumes. There is a volumes tab where you can see your configured mysql-go volume.

You can always open these volumes/containers on the docker GUI, go through the files and logs, experiment with putting one container down and seeing how the others respond, and so on.

After this entire setup, what do you notice? You didn’t have to install Go, MySQL, or phpMyAdmin locally. You only used officially published base images to orchestrate a full application. That's the magic of Docker.

Wrapping Up

Docker can be very abstract at the beginning, but understanding the fundamental purpose behind it makes everything much clearer.

In this article, you've learned what Docker is, how to containerize a basic Go application, and how to manage multiple containers with Docker Compose.

If you have trouble wrapping your head around why or how the Dockerfile is set up in the order that it is, my advice is not to get too stuck figuring it out on your own. As a Docker beginner, I realised that it’s easier if you imagine it as creating a recipe. If you try to build an image and it fails, you know there’s a step that you’re skipping.

The official docker documentation has amazing resources if you want to understand Docker further than this tutorial. I encourage you to do so because this article only scratches the surface of the amazing things you can achieve with containerization.

When you run five agents in a swarm, each spawning its own tool calls and producing its own output, "read the log" stops being a strategy.

I built Claude Forge as an adversarial multi-agent coding framework on top of Claude Code. A typical run spawns a planner, an implementer, a reviewer, and a fixer. They evaluate each other's work and loop back when quality checks fail.

But when something went wrong, I had timestamps and text dumps but no way to see which agent was responsible, how long it actually took, or where the tokens went.

Jaeger fixed that. This article covers setting up Jaeger v2 with Docker, wiring it into a multi-agent system through OpenTelemetry, and what I learned along the way.

Table of Contents

• What Is Distributed Tracing?

• Why Jaeger v2?

• Prerequisites

• Installing Docker on Debian

• Setting Up Jaeger v2

• Setting Up Claude Forge Tracing

• Understanding the Span Model

• Instrumenting a Multi-Agent Swarm

• Viewing Traces in the Jaeger UI

• Lessons from the Trenches

• Environment Variable Reference

• Wrapping Up

What Is Distributed Tracing?

Distributed tracing tracks a single operation as it moves through multiple services. A span is one unit of work with a start time, end time, and key-value attributes. Spans nest into parent-child trees. One tree per operation is one trace.

Microservices people already know this pattern: follow an HTTP request from the gateway through auth, the database, and the cache. Same idea works for multi-agent AI. Follow one swarm invocation from the orchestrator through each subagent and its tool calls.

OpenTelemetry (OTel) is the standard. It gives you SDKs for creating spans and shipping them over OTLP. Jaeger receives that data and renders it as a searchable timeline.

Why Jaeger v2?

Jaeger started at Uber and graduated as a CNCF project in 2019. v1 hit end of life in December 2025. v2 is the current release, built on the OpenTelemetry Collector framework. Single binary: collector, query service, and UI. It speaks OTLP natively on port 4317 (gRPC) and 4318 (HTTP). There's no separate collector needed for local work.

One important difference from v1: configuration moved from CLI flags and environment variables to a YAML file. The old -e SPAN_STORAGE_TYPE=badger env vars are silently ignored in v2. The container starts fine but falls back to in-memory storage. I lost two days of traces before noticing. More on the correct setup below.

Prerequisites

• Docker installed and running.

• Claude Code installed.

• Python 3.8+ for the tracing hook.

• Claude Forge or another multi-agent system to instrument.

Installing Docker on Debian

Skip this if you already have Docker. macOS and Windows users can use Docker Desktop. On Debian:

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/debian \
  \((. /etc/os-release && echo "\)VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
newgrp docker

Ubuntu users: replace both linux/debian URLs with linux/ubuntu.

Setting Up Jaeger v2

Basic Run

For quick testing with no persistence:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/jaeger:2.17.0

Port 16686 is the UI. Port 4317 is OTLP/gRPC ingestion. Port 4318 is OTLP/HTTP. Remove the container and your traces are gone.

Persistent Storage with Badger

v2 reads configuration from a YAML file, not environment variables. Save this as ~/.local/share/jaeger/config.yaml:

service:
  extensions: [jaeger_storage, jaeger_query, healthcheckv2]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]
extensions:
  healthcheckv2:
    use_v2: true
    http: { endpoint: 0.0.0.0:13133 }
  jaeger_query:
    storage: { traces: main_store }
  jaeger_storage:
    backends:
      main_store:
        badger:
          directories: { keys: /badger/key, values: /badger/data }
          ephemeral: false
          ttl: { spans: 720h }
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch:
exporters:
  jaeger_storage_exporter:
    trace_storage: main_store

The Jaeger container runs as UID 10001. Docker named volumes default to root ownership. Without fixing permissions first, the container crash-loops with mkdir /badger/key: permission denied.

Pre-create the volume and fix ownership:

docker volume create jaeger-data

docker run --rm \
  -v jaeger-data:/badger \
  alpine sh -c "mkdir -p /badger/data /badger/key && chown -R 10001:10001 /badger"

Then run Jaeger with the config mounted in:

docker run -d --name jaeger \
  --restart unless-stopped \
  -v ~/.local/share/jaeger/config.yaml:/etc/jaeger/config.yaml:ro \
  -v jaeger-data:/badger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/jaeger:2.17.0 \
  --config /etc/jaeger/config.yaml

Verify persistence by running docker restart jaeger and confirming a previously recorded trace is still there. Hit http://localhost:16686 and you should see the UI.

Setting Up Claude Forge Tracing

Installing Claude Forge

Install it through the Claude Code plugin marketplace:

/plugin marketplace add hatmanstack/claude-forge
/plugin install forge@claude-forge
/reload-plugins

The install opens a TUI to confirm scope and settings. After reload, commands use the forge: prefix (for example, /forge:pipeline).

You can also clone the repo from GitHub.

Installing the Tracing Hook

From your target project directory, run the install script. For plugin installs:

cd your-project
forge-trace                # if you set up the alias from the README
# or, without the alias:
bash "$(find ~/.claude -path '*/forge*' -name install-tracing.sh 2>/dev/null | head -1)"

For clone installs:

cd your-project
bash /path/to/claude-forge/bin/install-tracing.sh

The script builds a dedicated venv at ~/.local/share/claude-forge/venv (prefers uv, falls back to python3 -m venv), installs the OpenTelemetry packages, copies the hook into place, merges hook entries into .claude/settings.local.json, and self-tests against the OTLP endpoint.

Pass --no-settings to skip the settings merge, or --uninstall to tear everything down.

Opting In

Add to your shell init and restart your terminal:

export CLAUDE_FORGE_TRACING=1

Restart Claude Code, run /pipeline, then check http://localhost:16686 for the claude-forge service.

Understanding the Span Model

Here's what the hierarchy looks like for a typical swarm run:

session: "implement login form with OAuth"        <- root span
├── subagent:planner
│   ├── tool:Write  (Phase-0.md)                  <- mutation spans (on by default)
│   ├── tool:Write  (Phase-1.md)
│   └── subagent_result:planner                   <- duration, token counts, output
├── subagent:implementer
│   ├── tool:Edit   (src/auth.ts)
│   ├── tool:Bash   (npm test)
│   ├── tool:Write  (src/oauth.ts)
│   └── subagent_result:implementer
├── subagent:reviewer
│   └── subagent_result:reviewer
└── session_complete                              <- session totals

The root span's name comes from the first line of your prompt. Find traces by what you asked for, not by a UUID.

Subagents get an anchor span on start and a result span on completion. The result carries duration, token counts, prompt, and output.

Three Tiers of Detail

Not all inner tool calls are equally interesting. Write, Edit, MultiEdit, and Bash are mutational: small in number, high signal. They tell you what actually changed. Read, Glob, Grep, and WebFetch are navigation: lots of them, mostly noise.

Tracing captures mutations by default. That middle ground turned out to be the right one. Before this change, you either saw nothing inside subagents or you saw 200+ spans per run.

Span Attributes

On session_complete: session.tokens.input, session.tokens.output, session.tokens.total, session.tokens.turns, session.duration_ms, user.prompt (first 2KB).

On subagent_result: agent.description, agent.prompt, agent.output, agent.duration_ms, agent.is_error, agent.tokens.input, agent.tokens.output.

On tool:*: tool.name, tool.input, tool.output, tool.duration_ms, tool.is_error.

Instrumenting a Multi-Agent Swarm

Hook Architecture

Claude Code has lifecycle hooks that fire scripts on specific events. Four matter here:

1. UserPromptSubmit (create the root span),

2. PreToolUse (start a span),

3. PostToolUse (end it with results), and

4. Stop (finalize the trace). Each hook gets a JSON payload on stdin and runs as a subprocess.

Sending Spans with OpenTelemetry

Here's some minimal Python to get a span into Jaeger:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "my-agent-system"})
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("agent-tracer")

with tracer.start_as_current_span("my-agent-task") as span:
    span.set_attribute("agent.name", "planner")
    span.set_attribute("agent.tokens.input", 1500)
    span.set_attribute("agent.tokens.output", 800)

Refresh localhost:16686, pick your service, click "Find Traces."

Correlating Pre and Post Events

You need to match each PreToolUse to its PostToolUse. Agent-type tool calls didn't include a tool_use_id in the payload, so I hashed the tool name and input instead. Pre and Post carry identical tool_input, so the hashes line up.

import hashlib, json

def correlation_key(tool_name: str, tool_input: dict) -> str:
    content = json.dumps({"tool": tool_name, "input": tool_input}, sort_keys=True)
    return hashlib.sha1(content.encode()).hexdigest()[:16]

State Across Invocations

Every hook call is a separate process. No shared memory. So I wrote span context to JSON files on Pre and read them back on Post:

/tmp/claude-forge-tracing/<session_id>/
├── _root.json              # trace ID, root span context
├── _session_start_ns.json  # timestamp for duration calculation
├── subagent_<hash>.json    # per-subagent span context
└── tool_<hash>.json        # per-tool span context

File names get sanitized against path traversal. _safe_name() strips everything outside [A-Za-z0-9._-] and falls back to a SHA1 slug.

Flushing Without Blocking

try:
    provider.force_flush(timeout_millis=1000)
except Exception:
    pass  # Never block the swarm

I tried 2000ms first and the swarm felt slow. 100ms lost spans on cold TLS connections. 1000ms worked. If Jaeger is down, the swarm keeps running regardless.

Viewing Traces in the Jaeger UI

Open http://localhost:16686. Pick claude-forge from the service dropdown. Click "Find Traces."

The trace search filters by operation name, tags, and time range. Since session spans take their name from your prompt, searching "login form" pulls up the runs where you asked for one.

The timeline view is where I spend most of my time. Every span is a horizontal bar, nested by parent-child relationships. I can see the planner took 12 seconds, the implementer 45, the reviewer 8. Click any bar to see token counts, prompts, outputs, error status.

Trace comparison puts two runs side by side. This is good for figuring out why one run succeeded and another did not.

Lessons from the Trenches

One trace per swarm, not per subagent: My first version wiped the root span's state file on every Stop event, so each subagent started a new trace. I changed Stop to mark a timestamp while preserving the root.

Use descriptions, not type names: Subagents all report their type as general-purpose. The description field is where the actual role lives.

Token attribution needs per-agent transcripts: Claude Code writes subagent transcripts to ~/.claude/projects/<project>/<session>/subagents/agent-*.jsonl. Match them via agent-*.meta.json.

Parse boolean env vars explicitly: bool("0") in Python is True. Use an allowlist: {"1", "true", "yes", "on"}.

Environment Variable Reference

Wrapping Up

Without visibility into the process, you're being inefficient with tokens and your time. Multi-agent swarms cost real money on every run. When an agent fails and retries, or when a reviewer rejects work that was close, you're paying for that blind.

Tracing gives you the map. You find out where the failure modes are. You find out which agents burn tokens going nowhere. A 45-second implementer run might have been 10 seconds with a better planner prompt. But you would never know that without seeing the breakdown.

Get observability in early. Jaeger and OpenTelemetry make it cheap to set up. Once you can see where things go wrong you can actually fix them.

Claude Forge tracing is on the main branch.

You’ll learn how to expose services on a custom domain with auto-renewing HTTPS, and implement a smart deployment strategy that detects changes and redeploys only the affected microservices. This helps avoid unnecessary full-stack redeploys. We'll also cover real production issues and the exact fixes for each one.

Table of Contents

• 1. What you'll build

• 2. Architecture

• 3. Server prerequisites

• 4. Traefik — the reverse proxy

• 5. Run Jenkins in Docker

• 6. Expose Jenkins on a domain via Traefik

• 7. First-time Jenkins setup

• 8. Add the GitHub credential

• 9. Create the pipeline job

• 10. The Jenkinsfile (deploy only what changed)

• 11. End-to-end test

• 12. Troubleshooting — every error we hit

• 13. Mental model: host vs. container

• 14. Daily operations cheat sheet

• 15. What I'd do differently next time

• Closing thoughts

1. What You'll Build

In this tutorial, you'll build a Jenkins instance running inside Docker on the same Linux server as your application stack.

Traefik will act as a reverse proxy in front of Jenkins, exposing it via a clean URL (https://jenkins.example.com) with auto-renewing Let's Encrypt certificates.

You'll also create a Jenkinsfile in your application repository that:

• Automatically triggers on every push to the staging branch,

• Detects which microservices changed in each commit,

• Pulls the latest code on the host machine,

• Rebuilds and restarts only the affected services.

On every push, only the relevant services are redeployed.

Prerequisites

Before jumping in, this guide assumes you’re already comfortable with a few core concepts and tools.

This isn't a beginner-level tutorial — we’ll be working directly with infrastructure, containers, and CI/CD pipelines.

You should be familiar with:

• Basic Linux commands (SSH, file system navigation, permissions)

• Docker fundamentals (images, containers, volumes, networks)

• Git workflows (clone, pull, branches)

• General idea of CI/CD pipelines

Tools and environment required:

• A Linux server (Ubuntu recommended)

• Docker Engine + Docker Compose (v2)

• A domain name (for Traefik + HTTPS)

• GitHub repository (for your backend project)

• Basic understanding of microservices architecture

If you’re comfortable with the above, you’re ready to follow along.

2. Architecture

Here's an overview of the architecture:

┌──────────────────────────── Linux server (Ubuntu) ────────────────────────────┐
│                                                                               │
│   /home/developer/projects/                                                  │
│       └── project-prod-configs/             ← infra repo (compose, Traefik) │
│              ├── docker-compose.staging.yml                                   │
│              ├── traefik.staging.yml                                          │
│              └── project-backend/          ← app repo (services, gateways) │
│                     ├── Jenkinsfile                                           │
│                     ├── docker-compose.staging.yml                            │
│                     └── apps/                                                 │
│                            ├── services/<name>/                               │
│                            ├── gateways/<name>/                               │
│                            └── core/<name>/                                   │
│                                                                               │
│   ┌─────────────────────── Docker network: proxy ──────────────────────┐      │
│   │  traefik (80, 443)                                                 │      │
│   │     │                                                              │      │
│   │     ├──► jenkins  (projects-jenkins-staging)                     │      │
│   │     │      ↳ /projects  ← bind-mount of the host project tree     │      │
│   │     │      ↳ /var/run/docker.sock ← controls host Docker           │      │
│   │     │                                                              │      │
│   │     └──► your services & gateways (built by the pipeline)          │      │
│   └────────────────────────────────────────────────────────────────────┘      │
│                                                                               │
└───────────────────────────────────────────────────────────────────────────────┘
            ▲
            │  webhook on push
            │
   GitHub: <org>/project-backend (branch: staging)

There are two key ideas here:

1. Jenkins runs in a container, but it controls the host's Docker by mounting /var/run/docker.sock. It also bind-mounts the project folder as /projects/..., so it can cd into the real code on the host and run docker compose there.

2. The Jenkinsfile lives inside the app repo, so the pipeline definition is versioned with the code. Jenkins simply points at it.

3. Server Prerequisites

Before we start configuring Jenkins or Traefik, we need to prepare the server properly.

In this step, we’ll:

• Create a dedicated Linux user for managing the project

• Install Docker and Docker Compose

• Set up the folder structure for our repositories

This ensures our CI/CD pipeline runs in a clean and predictable environment.

# Linux user that owns the project tree
sudo adduser developer

# Docker engine + Compose plugin
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker developer

# Sanity check Compose v2
docker compose version
# -> Docker Compose version v2.x.y

# Find where the Compose plugin binary lives — write it down, you'll need it
ls /usr/libexec/docker/cli-plugins/docker-compose
# (some distros use /usr/lib/docker/cli-plugins/docker-compose)

# Project layout
sudo mkdir -p /home/developer/project
sudo chown -R developer:developer /home/developer/project

# Clone both repos in the right place
cd /home/developer/projects
git clone https://github.com/<org>/projects-prod-configs.git
cd projects-prod-configs
git clone -b staging https://github.com/<org>/projects-backend.git

You should now have:

/home/developer/projects/projects-prod-configs/projects-backend

Memorize this path — your Jenkinsfile references it.

DNS

Point an A-record for your Jenkins subdomain to the server's public IP before the next steps so Let's Encrypt can validate via HTTP challenge:

jenkins.example.com   A   <server-public-ip>

4. Traefik — the Reverse Proxy

Traefik acts as the entry point to your entire system. Instead of exposing each service manually with ports, Traefik automatically:

• Routes traffic based on domain names

• Generates and renews HTTPS certificates using Let’s Encrypt

• Connects to Docker and detects services dynamically

In simple terms, Traefik lets you access services like:

https://jenkins.example.com
https://api.example.com

…without manually configuring NGINX or managing SSL certificates.

In this setup, Traefik watches Docker containers and routes traffic using labels we'll define later.

Traefik gives every container a real domain and a real cert with zero per-service config — you just add a few labels.

traefik.staging.yml (static config)

Put this at the root of your infra repo:

api:
  dashboard: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      httpChallenge:
        entryPoint: web
      email: admin@example.com           # ← change me
      storage: /etc/traefik/acme.json

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false              # only containers with traefik.enable=true
    network: proxy
  file:
    directory: /etc/traefik/dynamic
    watch: true

log:
  level: INFO

accessLog: {}

The Traefik service in docker-compose.staging.yml

networks:
  proxy:
    name: proxy
    driver: bridge
  internal:
    name: internal
    driver: bridge

volumes:
  acme-data:
  traefik-logs:
  jenkins-data:

services:
  traefik:
    image: traefik:v2.11
    container_name: projects-traefik-staging
    restart: unless-stopped
    ports:
      - "80:80"        # HTTP (auto-redirects to HTTPS)
      - "443:443"      # HTTPS
      - "8080:8080"    # Traefik dashboard (internal only — protect via firewall)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.staging.yml:/etc/traefik/traefik.yml:ro
      - ./dynamic:/etc/traefik/dynamic:ro
      - acme-data:/etc/traefik           # persists Let's Encrypt certs
      - traefik-logs:/var/log/traefik
    networks:
      - proxy
    command:
      - '--api.insecure=false'
      - '--api.dashboard=true'
      - '--providers.docker=true'
      - '--providers.docker.exposedbydefault=false'
      - '--providers.docker.network=proxy'
      - '--entrypoints.web.address=:80'
      - '--entrypoints.websecure.address=:443'
      - '--entrypoints.web.http.redirections.entryPoint.to=websecure'
      - '--entrypoints.web.http.redirections.entryPoint.scheme=https'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge=true'
      - '--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web'
      - '--certificatesresolvers.letsencrypt.acme.email=${ACME_EMAIL:-admin@example.com}'
      - '--certificatesresolvers.letsencrypt.acme.storage=/etc/traefik/acme.json'
      - '--log.level=INFO'
      - '--accesslog=true'
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      # Traefik's own dashboard
      - "traefik.http.routers.traefik-dash.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.traefik-dash.entrypoints=websecure"
      - "traefik.http.routers.traefik-dash.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik-dash.service=api@internal"

Bring it up:

cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d traefik

Watch the logs the first time — Traefik will request a cert for the dashboard host as soon as DNS resolves.

docker logs -f projects-traefik-staging

Tip. While testing, switch ACME to staging endpoint (acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory) so you don't burn through Let's Encrypt's rate limits if you misconfigure DNS. Remove that flag before going live.

5. Run Jenkins in Docker

Add this Jenkins service to the same docker-compose.staging.yml. Every line matters (and the comments explain why).

  jenkins:
    image: jenkins/jenkins:lts
    container_name: projects-jenkins-staging
    restart: unless-stopped
    user: root                           # to use host docker.sock without UID juggling
    environment:
      - JAVA_OPTS=-Xmx1g -Xms512m -Duser.timezone=Asia/Dhaka
      - TZ=Asia/Dhaka                    # OS-level timezone inside container
      - JENKINS_OPTS=--prefix=/
    ports:
      - "3095:8080"                      # web UI (also reachable directly if needed)
      - "50000:50000"                    # inbound agent port
    volumes:
      - jenkins-data:/var/jenkins_home   # Jenkins config/jobs/secrets persistence
      - /var/run/docker.sock:/var/run/docker.sock                          # control host Docker
      - /usr/bin/docker:/usr/bin/docker                                     # docker CLI from host
      - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro  # docker compose plugin
      - /home/developer/projects:/projects                                # project tree
      - /etc/localtime:/etc/localtime:ro                                    # match host clock
      - /etc/timezone:/etc/timezone:ro
    networks:
      - proxy
      - internal
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:8080/login']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s
    deploy:
      resources:
        limits:
          memory: 1024M

Why user: root? It's the simplest way to share docker.sock and the project bind-mount without UID/GID gymnastics. If you prefer an unprivileged user, you'll need to set group: docker and align UIDs/perms on host folders — possible but out of scope here.

6. Expose Jenkins on a Domain via Traefik

This is the section many guides skip. We'll add labels to the Jenkins service so Traefik picks it up automatically. No editing of Traefik config required.

  jenkins:
    # ... everything above ...
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"

      # 1) Router — match incoming Host
      - "traefik.http.routers.jenkins.rule=Host(`jenkins.example.com`)"
      - "traefik.http.routers.jenkins.entrypoints=websecure"
      - "traefik.http.routers.jenkins.tls.certresolver=letsencrypt"
      - "traefik.http.routers.jenkins.service=jenkins"

      # 2) Service — tell Traefik which container port is the app
      - "traefik.http.services.jenkins.loadbalancer.server.port=8080"

      # 3) Middleware — Jenkins needs X-Forwarded-Proto so it knows it's behind HTTPS
      - "traefik.http.middlewares.jenkins-headers.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.jenkins.middlewares=jenkins-headers"

What each line does:

Bring Jenkins up:

cd /home/developer/projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d jenkins

# Watch Traefik issue the certificate
docker logs -f projects-traefik-staging | grep -i acme

After 10–60 seconds you should be able to open https://jenkins.example.com and see Jenkins's setup wizard with a valid lock icon.

Inside Jenkins (after first login):

Manage Jenkins → System → Jenkins URL → set this to: https://jenkins.example.com/

This is important because Jenkins uses this base URL to generate:

• Webhook endpoints (for GitHub triggers)

• Links inside emails and build logs

If this isn't set correctly, GitHub webhooks may fail, and any links Jenkins generates will point to the wrong address (often localhost or internal IPs).

7. First-Time Jenkins Setup

If you're running Jenkins for the first time on this server, follow this section to complete the initial setup.

If you already have Jenkins configured, you can skip this section — but make sure the required plugins and settings match what we use later in this guide.

1. Open https://jenkins.example.com. Get the initial admin password:

   docker exec projects-jenkins-staging cat /var/jenkins_home/secrets/initialAdminPassword
   

2. Paste it, choose Install suggested plugins.

3. Create your admin user.

4. Manage Jenkins → Plugins → Available and install:

   • GitHub (and GitHub Branch Source)

   • Pipeline: GitHub

   • Credentials Binding (usually preinstalled)

That's all the plugins you need for the rest of this guide.

8. Add the GitHub Credential

Jenkins needs permission to access your GitHub repository.

This is done using a GitHub Personal Access Token (PAT), which acts like a password for secure API and Git operations.

We’ll store this token inside Jenkins as a credential so it can pull code during pipeline execution and authenticate securely without exposing secrets in code.

This single credential is used both for the SCM checkout and for the deploy-time git pull.

1. Create a Personal Access Token (classic) on GitHub with repo scope.

2. In Jenkins: Manage Jenkins → Credentials → System → Global → Add Credentials.

3. Fill in:

   • Kind: Username with password

   • Username: your GitHub username

   • Password: the token

   • ID: github_classic_token (the Jenkinsfile references this exact ID)

9. Create the Pipeline Job

Now that Jenkins has access to your repository, the next step is to define how deployments should run.

A pipeline job tells Jenkins:

• where your code lives,

• which branch to monitor,

• and how to execute your deployment process.

In Jenkins, create a new Pipeline job and connect it to your GitHub repository. Once this is set up, Jenkins will automatically trigger deployments whenever you push to the staging branch.

Start by creating a new job:

New Item → Pipeline → name it projects-staging → OK

Then configure the job:

• Under Build Triggers, enable:
  GitHub hook trigger for GITScm polling

• Under Pipeline:

  • Definition: Pipeline script from SCM

  • SCM: Git

  • Repository URL: https://github.com/<org>/projects-backend.git

  • Credentials: github_classic_token

  • Branch: */staging

  • Script Path: Jenkinsfile

Save the configuration.

At this point, Jenkins is fully connected to your repository and ready to run your deployment pipeline automatically.

10. The Jenkinsfile (Deploy Only What Changed)

Place this at the root of the app repo (projects-backend/Jenkinsfile), branch staging.

pipeline {
  agent any

  environment {
    PROJECT_PATH = "/projects/projects-prod-configs/projects-backend"
    COMPOSE_FILE = "docker-compose.staging.yml"
  }

  stages {

    stage('Checkout') {
      steps {
        checkout scm
        echo "Checkout completed for branch: ${env.BRANCH_NAME ?: 'staging'}"
      }
    }

    stage('Detect Changes') {
      steps {
        script {
          def changedFiles = sh(
            script: "git diff --name-only HEAD~1 HEAD",
            returnStdout: true
          ).trim()

          echo "Changed files:\n${changedFiles}"

          def services = [] as Set
          changedFiles.split('\n').each { file ->
            def svc  = file =~ /^apps\/services\/([a-z0-9-]+)\//
            def gw   = file =~ /^apps\/gateways\/([a-z0-9-]+)\//
            def core = file =~ /^apps\/core\/([a-z0-9-]+)\//
            if (svc)  { services << svc[0][1]  }
            if (gw)   { services << gw[0][1]   }
            if (core) { services << core[0][1] }
          }
          services = services.findAll { !it.endsWith('-e2e') }
          env.CHANGED_SERVICES = services.join(' ')

          echo "Services to deploy: ${env.CHANGED_SERVICES ?: '(none)'}"
        }
      }
    }

    stage('Deploy') {
      when { expression { return env.CHANGED_SERVICES?.trim() } }
      steps {
        withCredentials([usernamePassword(
          credentialsId: 'github_classic_token',
          usernameVariable: 'GIT_USER',
          passwordVariable: 'GIT_TOKEN'
        )]) {
          sh '''
            set -eu
            git config --global --add safe.directory "${PROJECT_PATH}"
            cd "${PROJECT_PATH}"
            git remote set-url origin "https://github.com/<org>/projects-backend.git"
            git -c credential.helper= \
                -c "credential.helper=!f() { echo username=\({GIT_USER}; echo password=\){GIT_TOKEN}; }; f" \
                pull origin staging
            docker compose -f "\({COMPOSE_FILE}" up -d --build \){CHANGED_SERVICES}
          '''
        }
        echo "Deployed: ${env.CHANGED_SERVICES}"
      }
    }

    stage('Skip Deployment') {
      when { expression { return !env.CHANGED_SERVICES?.trim() } }
      steps { echo "No service changes detected — nothing to deploy." }
    }
  }
}

Why each tricky line is there:

• git config --global --add safe.directory ... — git refuses to operate on a repo whose owner UID differs from the current user's. The repo on disk is owned by developer, but Git inside the container runs as root. This whitelists the path.

• git remote set-url origin "https://..." — flips the on-disk remote to HTTPS so the token can be used. (A PAT can't authenticate git@github.com: URLs — those use SSH.) Idempotent — safe to re-run.

• git -c credential.helper="!f() { echo username=...; echo password=...; }; f" — feeds the username/token to git for that one command without writing the token to disk and without exposing it on the process command line.

• ${CHANGED_SERVICES} is unquoted on purpose so multiple service names expand as separate args.

11. End-to-End Test

Before considering the setup complete, we need to verify that the entire pipeline works as expected.

This end-to-end test ensures that:

• GitHub webhooks are triggering Jenkins correctly,

• Jenkins can detect which services changed,

• and only the affected services are rebuilt and deployed.

In other words, this simulates a real production deployment.

Start by making a small change in your repository. For example, modify a file inside:

apps/gateways/student-apigw/

Then push the change to the staging branch.

Once pushed, Jenkins should automatically trigger via the webhook. If not, you can manually click Build Now.

Now open the build’s Console Output and verify the flow. You should see something like:

• Checkout completed for branch: staging

• Services to deploy: student-apigw

• git pull origin staging (successful)

• docker compose ... up -d --build student-apigw

• Deployed: student-apigw

If you see this sequence, your pipeline is working correctly.

If anything fails, don’t worry — jump to Section 12 where every common issue and its fix is documented.

12. Troubleshooting — Every Error We Hit

This section covers real issues we faced while setting up this pipeline — and more importantly, why each fix works. Understanding the “why” will help you debug similar problems in your own setup.

cd: can't cd to /projects/projects-prod-configs/projects-backend

Cause:
The Jenkinsfile runs cd $PROJECT_PATH, but inside the container that path doesn’t exist. This usually happens when:

• the project wasn’t cloned on the host, or

• the bind mount isn’t configured correctly.

Fix:

ls /home/developer/projects/projects-prod-configs/projects-backend
# If missing: git clone -b staging <url> there.

Confirm the bind mount:

docker inspect projects-jenkins-staging --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'

If missing, recreate the container:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:

Jenkins runs inside a container, but your code lives on the host. The bind mount connects them. Without it, Jenkins cannot access your project directory.

fatal: detected dubious ownership in repository

Cause:
Git blocks access when the repository owner differs from the current user.

• Repo owner: developer (host)

• Git runs as: root (inside container)

Fix:

git config --global --add safe.directory "${PROJECT_PATH}"

Why this works:

This explicitly tells Git that the directory is trusted, bypassing ownership mismatch security restrictions.

Host key verification failed / Could not read from remote repository

Cause:

The repository uses SSH (git@github.com:...), but:

• the container has no SSH keys

• no known_hosts file exists

Also, GitHub tokens cannot authenticate over SSH.

Fix (recommended):

git remote set-url origin "https://github.com/<org>/projects-backend.git"

Why this works:

HTTPS uses token-based authentication (PAT), which works inside containers without SSH configuration.

unknown shorthand flag: 'f' in -f ( docker compose)

Cause:
The Docker CLI exists, but the Docker Compose plugin is missing inside the container.

Fix:

volumes:
  - /usr/libexec/docker/cli-plugins:/usr/libexec/docker/cli-plugins:ro

Find your path if needed:

find /usr -name docker-compose -type f 2>/dev/null

Verify:

docker exec projects-jenkins-staging docker compose version

Why this works:

Docker Compose v2 is a CLI plugin. Mounting this directory makes the docker compose command available inside the container.

Wrong timezone in build timestamps and Jenkins UI

Fix: Set both env var and JVM flag, and bind-mount the host's clock files:

environment:
  - TZ=Asia/Dhaka
  - JAVA_OPTS=... -Duser.timezone=Asia/Dhaka
volumes:
  - /etc/localtime:/etc/localtime:ro
  - /etc/timezone:/etc/timezone:ro

You must recreate the container for env-var changes to take effect:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:
Jenkins runs on Java, which uses its own timezone separate from the OS.
By aligning OS timezone, JVM timezone, and host clock, you ensure consistent timestamps everywhere.

ERR_SOCKET_TIMEOUT (pnpm install fails)

Cause:

If you have multiple services building in parallel and each runs pnpm install with ~1500 packages, the network gets saturated and a timeout occurs.

Fixes:

a) Increase timeout + control concurrency

RUN pnpm install --frozen-lockfile --ignore-scripts 
--network-timeout 600000 
--network-concurrency 8

Why: Gives pnpm more time and reduces network overload.

b) Enable pnpm cache (BuildKit)

RUN --mount=type=cache,id=pnpm-store,target=/root/.local/share/pnpm/store 
pnpm install --frozen-lockfile --ignore-scripts

Why: Dependencies are cached and reused instead of downloading every time.

c) Avoid unnecessary rebuilds

docker compose -f \(COMPOSE_FILE build \)CHANGED_SERVICES docker compose -f \(COMPOSE_FILE up -d --no-build \)CHANGED_SERVICES

Why: Only changed services are rebuilt → less network load → fewer failures.

Container changes don’t apply after editing docker-compose.yml

Cause:

Docker compose up -d does not update running containers.

Fix:

docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

Why this works:

This forces Docker to recreate the container with updated configuration (env, volumes, labels).

Traefik shows default certificate (no HTTPS)

Common causes:

DNS not pointing to server Port 80 blocked Wrong Docker network

Check:

dig +short jenkins.example.com docker logs projects-traefik-staging 2>&1 | grep -i acme

Why this works:

Let’s Encrypt uses HTTP-01 challenge, so it must reach your server via port 80. If DNS or networking is wrong, certificate issuance fails.

Jenkins: "Reverse proxy setup is broken"

Fix:

Set the Jenkins URL to https://jenkins.example.com/
Ensure header:

X-Forwarded-Proto: https

Why this works:

Jenkins needs to know it's behind HTTPS. Without this, it generates incorrect URLs (http instead of https), breaking redirects and webhooks.

13. Mental Model: Host vs. Container

Many setup mistakes come from confusing the host filesystem with the container filesystem. This table makes it explicit:

When debugging, always ask: "Inside which filesystem is this command running, and does the file/folder it's looking for exist there?"

14. Daily Operations Cheat Sheet

# Recreate Jenkins after changing compose
cd /home/developer/Projects/projects-prod-configs
docker compose -f docker-compose.staging.yml up -d --force-recreate jenkins

# Tail Jenkins logs
docker logs -f projects-jenkins-staging

# Open a shell inside the Jenkins container
docker exec -it projects-jenkins-staging bash

# From inside the container — sanity checks
docker compose version
ls /projects/projects-prod-configs/projects-backend
git -C /projects/projects-prod-configs/projects-backend remote -v

# Manually trigger the same deploy the pipeline does
cd /projects/projects-configs/projects-backend
git pull origin staging
docker compose -f docker-compose.staging.yml up -d --build student-apigw

# Inspect Traefik routing decisions
docker logs projects-traefik-staging 2>&1 | grep -i jenkins

# Check renewed certs
docker exec projects-traefik-staging cat /etc/traefik/acme.json | head -50

15. What I'd Do Differently Next Time

• Pre-build a base image with all node_modules baked in. With ~1500 packages × 15 services, every clean build re-downloads ~22k tarballs. A shared base cuts that 90%.

• Run a private npm proxy (Verdaccio / Nexus / GitHub Packages) on the same Docker network — eliminates flaky npmjs.org timeouts entirely.

• Per-service Jenkinsfile if your services drift apart in tooling. With one Jenkinsfile, every team contends for the same pipeline definition.

• Replace git diff HEAD~1 HEAD with git diff $(git merge-base HEAD origin/staging~1) HEAD so squash-merges and force-pushes don't accidentally skip services.

• Move secrets to a vault (HashiCorp Vault / AWS Secrets Manager / Doppler). PATs in Jenkins work, but rotation across many jobs is painful.

• Use Jenkins' Configuration-as-Code (JCasC) so the entire Jenkins setup (jobs, credentials definitions, plugins) is in git. Then a server rebuild is a one-command operation.

Closing Thoughts

The pipeline itself is just three stages: Checkout → Detect Changes → Deploy — but a real production setup is mostly about plumbing: reverse proxy, certificates, bind-mounts, credentials, timezones, build caches. None of these are exotic. Together they decide whether your Friday-afternoon deploy goes silently green or eats your weekend.

Follow sections 1–11 to get a working pipeline. Bookmark section 12 to keep it working.

Happy shipping.

Using ASP.NET 10 and C#, you can build independent REST APIs for services like patients, appointments, and authentication, each with its own database and deployment lifecycle.

Combined with API gateways, JWT-based security, observability, and containerization, this approach ensures reliable, maintainable, and production-ready healthcare systems.

In this tutorial, you’ll learn how to design and build a microservices-based healthcare portal using ASP.NET 10 and C#. We’ll cover how to structure services, implement REST APIs, secure endpoints, enable service communication, and deploy using modern containerization practices.

By the end, you’ll have a clear understanding of how to create scalable, secure, and production-ready healthcare systems.

Table of Contents

• Prerequisites

• Overview

• Why Use Microservices for Healthcare Portals?

• High-Level Architecture

• Designing REST APIs for Healthcare Services

• How to Build a Microservice with ASP.NET 10

• Database per Service Pattern

• Service Communication

• API Gateway Implementation

• Implementing Security in Healthcare APIs

• Observability and Logging

• Containerization with Docker

• Deployment Strategies

• Best Practices (With Examples)

• When NOT to Use Microservices

• Future Enhancements

• Conclusion

Prerequisites

Before getting started, you should be familiar with:

• C# and ASP.NET Core fundamentals

• REST API concepts (HTTP methods, routing, status codes)

• Basic understanding of microservices architecture

Tools required:

• .NET 10 SDK

• Visual Studio or VS Code

• Postman or Swagger

• Docker (optional but recommended)

Overview

Healthcare portals power critical workflows such as patient registration, appointment scheduling, electronic health records (EHR), billing, and telemedicine. These systems must handle sensitive data, high availability requirements, and frequent updates.

Traditionally, many healthcare applications were built as monolithic systems. While simple to start with, monoliths quickly become difficult to scale, maintain, and secure. A single failure can impact the entire system, and even small changes require redeploying the entire application.

Microservices architecture addresses these challenges by breaking the application into smaller, independent services. Each service is responsible for a specific domain, such as patient management or appointment scheduling, and can be developed, deployed, and scaled independently.

In this article, you'll learn how to design and implement a microservices-based healthcare REST API using ASP.NET 10 and C#. We'll walk through architecture design, service implementation, communication patterns, security, observability, and deployment strategies.

Why Use Microservices for Healthcare Portals?

Healthcare systems are inherently complex. They involve multiple domains such as patient records, appointments, billing, authentication and authorization. A microservices approach allows each of these domains to be handled independently. There are many benefits to this approach such as:

• Scalability: Scale only the services under heavy load (for example, appointments during peak hours)

• Fault isolation: Failure in one service does not crash the entire system

• Faster deployment: Teams can deploy updates independently

• Improved security: Sensitive services can have stricter access controls

For example, a patient service can handle personal data, while a billing service manages transactions, each with different security policies.

High-Level Architecture

A typical healthcare microservices architecture includes API Gateway (central entry point), microservices (Patient, Appointment, Auth), database per Service and service Communication Layer.

The request flow starts with the client sending a request. Then the API Gateway routes the request and the target microservice processes it. Then a response is returned. This separation ensures modularity and maintainability.

Designing REST APIs for Healthcare Services

Designing REST APIs in a microservices architecture requires clear, consistent naming conventions so that endpoints are intuitive, predictable, and easy to consume by clients and other services.

Naming Conventions

REST APIs are resource-oriented, meaning URLs should represent entities (nouns), not actions (verbs). Each resource corresponds to a domain object in your system, such as patients, appointments, or billing records.

Key principles:

• Use plural nouns for resources (for example, /patients, /appointments)

• Avoid verbs in URLs (don't use /getPatients)

• Use hierarchical structure for relationships (for example, /patients/{id}/appointments)

• Keep naming consistent across all services

These conventions improve API readability, developer experience, and maintainability across teams

Example: Patient API Endpoints

The following endpoints represent standard CRUD (Create, Read, Update, Delete) operations for managing patients:

GET    /api/patients        // Retrieve all patients
GET    /api/patients/{id}   // Retrieve a specific patient
POST   /api/patients        // Create a new patient
PUT    /api/patients/{id}   // Update an existing patient
DELETE /api/patients/{id}   // Delete a patient

Each HTTP method defines the type of operation being performed:

• GET: Fetch data (read-only)

• POST: Create new resources

• PUT: Update existing resources

• DELETE: Remove resources

These operations follow REST standards, ensuring consistency across services and making APIs easier to integrate with frontend apps, mobile clients, or third-party healthcare systems

Best Practices for Designing Healthcare REST APIs

Designing REST APIs for healthcare systems requires more than standard conventions. It demands careful consideration of performance, data sensitivity, and interoperability.

1. Use proper HTTP methods

Ensure each endpoint uses the correct HTTP verb (GET, POST, PUT, DELETE) to clearly communicate its purpose. This improves API predictability and aligns with REST standards used across healthcare platforms.

2. Return meaningful status codes

Use appropriate HTTP status codes to indicate the result of a request. For example:

• 200 OK for successful retrieval

• 201 Created for successful resource creation

• 400 Bad Request for validation errors

• 404 Not Found when a resource doesn’t exist
  Clear status codes help clients handle responses correctly.

3. Implement pagination for large datasets

Healthcare systems often deal with large volumes of data (for example, patient records, appointment logs). Use pagination to limit response size:

GET /api/patients?page=1&pageSize=20

This improves performance and reduces server load.

4. Use API versioning

Version your APIs to avoid breaking existing clients when making changes:

/api/v1/patients

This is especially important in healthcare, where integrations with external systems must remain stable over time.

5. Validate and sanitize input data

Always validate incoming data to prevent errors and ensure data integrity. For example, enforce required fields like patient name, date of birth, and contact details.

6. Protect sensitive data

Avoid exposing sensitive patient information unnecessarily. Use filtering, masking, or field-level access control where needed to comply with healthcare data regulations.

7. Ensure consistent response structure

Return responses in a standard format (for example, including data, status, and message fields). This makes APIs easier to consume and debug across multiple services.

How to Build a Microservice with ASP.NET 10

Let’s implement a simple Patient Service.

Step 1: Create Project

In this step, we'll create a new ASP.NET Web API project that will serve as our Patient microservice. This project provides the foundation for defining endpoints, handling HTTP requests, and structuring our service independently from other parts of the system.

dotnet new webapi -n PatientService
cd PatientService

Step 2: Define Model

Next, we'll define a simple data model representing a patient. Models define the structure of the data your API will send and receive, and they typically map to database entities in real-world applications.

public class Patient
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
}

Step 3: Create Controller

Here, we're creating a controller to handle incoming HTTP requests. Controllers define API endpoints and contain the logic for processing requests, interacting with data, and returning responses to clients.

[ApiController]
[Route("api/patients")]
public class PatientController : ControllerBase
{
    private static List<Patient> patients = new();

    [HttpGet]
    public IActionResult GetPatients()
    {
        return Ok(patients);
    }

    [HttpPost]
    public IActionResult AddPatient(Patient patient)
    {
        patients.Add(patient);
        return CreatedAtAction(nameof(GetPatients), patient);
    }
}

Database per Service Pattern

Each microservice should manage its own database to ensure loose coupling and independent operation. This allows services to evolve, scale, and be deployed without affecting others. It also improves data isolation and aligns with the core principles of microservices architecture.

Here's an example with Entity Framework Core:

public class PatientDbContext : DbContext
{
    public PatientDbContext(DbContextOptions<PatientDbContext> options)
        : base(options) { }

    public DbSet<Patient> Patients { get; set; }
}

This matters because it avoids cross-service dependencies, enables independent scaling, and improves data security, making microservices more efficient and secure.

Service Communication

Microservices communicate with each other to share data and coordinate workflows across the system. This communication can be handled through synchronous requests or asynchronous messaging, depending on the use case.

Choosing the right approach helps ensure scalability, reliability, and responsiveness in distributed systems

1. Synchronous Communication (HTTP)

var response = await httpClient.GetAsync("http://appointment-service/api/appointments");

2. Asynchronous Communication (Messaging)

Using message brokers like RabbitMQ:

• Services publish events

• Other services consume them

Example:

When a patient registers, an event triggers an appointment service.

API Gateway Implementation

An API Gateway acts as the central entry point for all client requests in a microservices architecture. It handles routing, authentication, and request aggregation, simplifying how clients interact with multiple services. This layer helps improve security, scalability, and overall system management.

Here's an example (Ocelot configuration):

{
  "Routes": [
    {
      "DownstreamPathTemplate": "/api/patients",
      "UpstreamPathTemplate": "/patients",
      "DownstreamHostAndPorts": [
        { "Host": "localhost", "Port": 5001 }
      ]
    }
  ]
}

Benefits include centralized routing, authentication handling, and rate limiting

Implementing Security in Healthcare APIs

Security is critical in healthcare systems due to the sensitive nature of patient data. APIs must enforce strong authentication, authorization, and data protection mechanisms. Proper security ensures compliance, prevents unauthorized access, and safeguards user trust.

1. JWT Authentication

builder.Services.AddAuthentication("Bearer")
    .AddJwtBearer(options =>
    {
        options.Authority = "https://auth-server";
        options.Audience = "healthcare-api";
    });

JWT (JSON Web Token) authentication is used to verify the identity of users accessing the API.

The authentication scheme ("Bearer") tells the API to expect a token in the Authorization header: Authorization: Bearer <token>

Authority represents the trusted authentication server (identity provider) that issues tokens.

And audience ensures that the token is intended specifically for this API.

When a request is made, the API:

1. Extracts the JWT from the request header

2. Validates its signature using the authority

3. Checks claims like expiration and audience

4. Grants access only if the token is valid

This ensures that only authenticated users can access healthcare services.

2. Role-Based Authorization

[Authorize(Roles = "Doctor")]
public IActionResult GetSensitiveData()
{
    return Ok();
}

Role-based authorization restricts access based on user roles.

• The [Authorize] attribute enforces that only authenticated users can access the endpoint.

• The Roles = "Doctor" condition ensures that only users with the Doctor role can access this resource.

When a user sends a request:

1. Their JWT token is validated

2. The system checks the role claim inside the token

3. Access is granted only if the required role matches

This is critical in healthcare systems where doctors access medical records, admins manage system data, and patients access only their own information.

3. Secure Secrets Management

var connectionString = Environment.GetEnvironmentVariable("DB_CONNECTION");

Sensitive configuration data such as database connection strings should never be hardcoded in the application.

Environment.GetEnvironmentVariable() retrieves secrets securely from the environment. These values are typically stored in:

• Environment variables

• Secret managers (Azure Key Vault, AWS Secrets Manager)

• Container orchestration platforms

Benefits:

• Prevents exposure of credentials in source code

• Supports secure deployments across environments

• Simplifies secret rotation without code changes

4. Enforce HTTPS

app.UseHttpsRedirection();

HTTPS ensures that all communication between the client and server is encrypted.

UseHttpsRedirection() automatically redirects HTTP requests to HTTPS. This protects sensitive healthcare data (such as patient records and credentials) from Man-in-the-Middle attacks, data interception, and unauthorized access.

In healthcare systems, encryption is essential for compliance with data protection standards and regulations.

Together, these security mechanisms provide multiple layers of protection:

• Authentication verifies identity

• Authorization controls access

• Secrets management protects credentials

• HTTPS secures data in transit

This layered approach is essential for safeguarding sensitive healthcare data and ensuring compliance with industry standards.

Observability and Logging

Observability enables you to monitor system health, diagnose issues, and understand how services interact in real time. By implementing logging, metrics, and tracing, teams can quickly identify failures and performance bottlenecks. This is essential for maintaining reliability in distributed systems.

Here's a basic logging example:

_logger.LogInformation("Fetching patients");

This line writes an informational log entry whenever the patient data is being retrieved. The _logger instance is part of ASP.NET’s built-in logging framework and is typically injected into the class through dependency injection.

Logging at this level helps developers trace normal application behavior and understand when specific operations occur, which is especially useful during debugging and monitoring in production environments.

Application Insights Integration

builder.Services.AddApplicationInsightsTelemetry();

This configuration enables integration with Application Insights, a cloud-based monitoring service. By adding this line, the application automatically collects telemetry data such as request rates, response times, failure rates, and dependency calls. This allows teams to monitor the health of the application in real time and quickly identify performance bottlenecks or failures across distributed microservices.

Custom Metrics

var telemetryClient = new TelemetryClient();
telemetryClient.TrackMetric("PatientsFetched", 1);

Here, a TelemetryClient instance is used to send custom metrics to the monitoring system. The TrackMetric method records a numerical value – in this case, tracking how many times patients are fetched.

Custom metrics like this help measure business-specific operations and provide deeper insight into how the system is being used beyond standard performance metrics.

Health Checks

app.MapHealthChecks("/health");

This line exposes a health check endpoint at /health that external systems can use to verify whether the service is running correctly. When this endpoint is called, it returns the status of the application and any configured dependencies, such as databases or external services.

Health checks are commonly used by load balancers, container orchestrators, and monitoring tools to automatically detect failures and restart or reroute traffic if needed.

Together, logging, telemetry, custom metrics, and health checks provide a complete observability strategy. They allow teams to understand system behavior, detect issues early, and maintain reliability across distributed healthcare services where uptime and performance are critical.

Containerization with Docker

Containerization allows microservices to run in isolated and consistent environments across development and production. Using Docker, you can package applications with all dependencies, ensuring portability and easier deployment. This approach simplifies scaling and infrastructure management.

The following Dockerfile shows a minimal setup for packaging the Patient Service into a container image:

FROM mcr.microsoft.com/dotnet/aspnet:10.0
WORKDIR /app
COPY . .
ENTRYPOINT ["dotnet", "PatientService.dll"]

This Dockerfile defines how the Patient Service is packaged into a container image so it can run consistently across different environments.

The FROM instruction specifies the base image, which in this case is the official ASP.NET runtime image for .NET 10. This image includes all the necessary runtime components required to execute the application, so you don’t need to install .NET separately inside the container.

The WORKDIR /app line sets the working directory inside the container. All subsequent commands will run relative to this directory, helping organize application files in a predictable structure.

The COPY . . instruction copies all files from the current project directory on your machine into the container’s working directory. This includes the compiled application binaries and any required resources.

Finally, the ENTRYPOINT defines the command that runs when the container starts. In this case, it launches the PatientService application using the .NET runtime.

Together, these steps package the microservice into a portable unit that can be deployed consistently across development, staging, and production environments. This ensures that the application behaves the same regardless of where it is deployed, which is a key advantage of containerization in microservices architectures.

Deployment Strategies

Deploying microservices requires strategies that minimize downtime and reduce risk during updates.

Techniques like rolling updates, canary releases, and blue-green deployments help ensure smooth transitions. These approaches improve system stability and user experience during releases.

Key Strategies

Deploying microservices requires strategies that minimize downtime, reduce risk, and ensure system stability – especially in healthcare systems where availability and data integrity are critical.

1. Rolling Updates

Rolling updates deploy changes gradually by updating instances of a service one at a time instead of all at once. As new versions are deployed, old instances are terminated in phases, ensuring that the system remains available throughout the process.

This approach works well for stateless services and is commonly used in container orchestration platforms. It allows continuous availability while still enabling safe deployment of new features.

Rolling updates are best used when:

• You want zero downtime deployments

• Backward compatibility between versions is maintained

• Changes are relatively low risk

2. Canary Deployments

Canary deployments release a new version of a service to a small subset of users before rolling it out to everyone. This allows teams to monitor the behavior of the new version in a real-world environment with limited exposure.

If issues are detected, the deployment can be rolled back quickly without affecting the majority of users.

Canary deployments are ideal when:

• Releasing high-risk or complex features

• Testing performance under real traffic

• Gradually validating new functionality

3. Blue-Green Deployments

Blue-green deployment involves maintaining two identical environments: one running the current version (blue) and one running the new version (green). Traffic is switched from blue to green once the new version is fully tested and ready.

If something goes wrong, traffic can be immediately switched back to the previous version.

This strategy is particularly useful when:

• You need instant rollback capability

• System stability is critical

• Downtime must be completely avoided

Choosing the Right Strategy for Healthcare Microservices

In a healthcare portal, where reliability and patient data integrity are essential, blue-green deployments are often the safest choice. They allow full validation of the new version before exposing it to users and provide immediate rollback in case of failure.

But rolling updates are also commonly used for routine updates where backward compatibility is ensured, while canary deployments are useful when introducing new features like AI diagnostics or analytics modules.

Example: Blue-Green Deployment with Containers

Let’s walk through a simple conceptual example using containers.

Assume you have two environments:

• Blue (current version) running PatientService v1

• Green (new version) running PatientService v2

First, you deploy the new version (v2) alongside the existing one without affecting users.

Then you run tests and verify that the new version behaves correctly.

After that, you update the load balancer or API gateway to route traffic from blue to green. Then you monitor the system for errors or performance issues.

If everything is stable, you keep green as the active environment. If not, switch traffic back to blue instantly.

In a real-world setup, this traffic switching is typically handled by:

• API Gateways

• Load balancers

• Kubernetes services

This approach ensures that users experience no downtime while giving teams full control over deployment risk.

In practice, many production systems combine these strategies – for example, starting with a canary release and then completing deployment with a rolling update – to balance risk and efficiency.

Best Practices (With Examples)

Designing reliable microservices for healthcare systems requires applying proven patterns that improve stability, maintainability, and resilience. Below are some key best practices with practical examples.

1. Use API Versioning

API versioning ensures backward compatibility when your service evolves. In healthcare systems, where integrations with external systems (labs, insurance, EHR) are common, breaking changes can cause serious issues.

Here's an example:

[Route("api/v1/patients")]

This route attribute defines the base URL for the API and explicitly includes a version identifier (v1). By embedding the version in the route, the service can support multiple versions of the same API simultaneously. This allows existing clients to continue using older versions while newer versions are introduced without breaking compatibility.

You can later introduce a new version:

[Route("api/v2/patients")]

This represents a newer version of the same API with potentially updated functionality or structure. By separating versions at the routing level, developers can evolve the API safely while giving clients time to migrate.

This approach is especially important in healthcare systems where external integrations must remain stable over long periods.

This allows safe rollout of new features, support for legacy clients and gradual migration between versions.

2. Implement Retry Policies

Network calls between microservices can fail due to transient issues such as timeouts or temporary service unavailability. Retry policies help automatically recover from such failures.

Here's an example (using Polly):

services.AddHttpClient("api")
    .AddTransientHttpErrorPolicy(p => p.RetryAsync(3));

This code configures an HTTP client with a retry policy using Polly, a .NET resilience and transient-fault-handling library. Polly allows developers to define policies such as retries, circuit breakers, and timeouts for handling unreliable network calls.

The AddTransientHttpErrorPolicy method applies a retry strategy for temporary failures such as network timeouts or server errors. The RetryAsync(3) configuration means that if a request fails due to a transient issue, it will automatically be retried up to three times before returning an error.

This improves system reliability by handling temporary issues without requiring manual intervention.

This configuration retries failed requests up to three times before failing.

You can also add exponential backoff:

.AddTransientHttpErrorPolicy(p =>
    p.WaitAndRetryAsync(3, retryAttempt =>
        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))));

This configuration enhances the retry mechanism by introducing exponential backoff. Instead of retrying immediately, the system waits progressively longer between each retry attempt.

Exponential backoff means:

• The first retry waits for 2¹ seconds

• The second retry waits for 2² seconds

• The third retry waits for 2³ seconds

This approach reduces pressure on failing services and avoids overwhelming them with repeated requests. It's particularly useful in distributed systems where temporary failures are common and services need time to recover.

This helps in improving reliability, reducing temporary failures and avoiding manual retries.

3. Enforce Input Validation

Validating incoming data is critical, especially in healthcare systems where incorrect data can lead to serious consequences.

Here's an example:

if (string.IsNullOrEmpty(patient.Name))
    return BadRequest("Name is required");

This is a simple manual validation check that ensures the Name field is provided before processing the request. If the value is missing or empty, the API immediately returns a BadRequest response, preventing invalid data from entering the system.

A better approach is using data annotations:

public class Patient
{
    public int Id { get; set; }

    [Required]
    public string Name { get; set; }
}

This example uses data annotations to enforce validation rules at the model level. The [Required] attribute ensures that the Name property must be provided when a request is made. ASP.NET automatically validates the model during request processing and returns an error response if validation fails.

This approach is more scalable and maintainable than manual checks, especially in larger applications.

This ensures clean and valid data, reduced runtime errors, and better API usability.

4. Use Circuit Breaker Pattern

The circuit breaker pattern prevents cascading failures when a dependent service is down or slow.

For example, if the Appointment Service is unavailable, repeated calls from the Patient Service can overload the system. A circuit breaker stops these calls temporarily.

Here's an example (again using Polly):

services.AddHttpClient("api")
    .AddTransientHttpErrorPolicy(p =>
        p.CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)));

This means:

• After 5 consecutive failures, the circuit opens

• No further requests are sent for 30 seconds

• System gets time to recover

This helps in protecting system stability, preventing resource exhaustion, and improving overall resilience.

These practices ensure your microservices are backward-compatible (versioning), resilient (retry + circuit breaker), and reliable (validation).

In healthcare systems, where uptime and data integrity are critical, applying these patterns is essential.

This code configures a circuit breaker policy using Polly to protect the system from repeated failures when calling external services.

The CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)) configuration means that if five consecutive requests fail, the circuit will open and block further requests for 30 seconds. During this time, the system will not attempt to call the failing service, allowing it time to recover.

After the break period, the circuit enters a half-open state where a limited number of requests are allowed to test if the service has recovered. If successful, normal operation resumes. Otherwise, the circuit opens again.

This pattern prevents cascading failures, reduces unnecessary load on failing services, and improves overall system resilience.

These examples demonstrate how small design decisions (like versioning, retries, validation, and fault handling) can significantly improve the reliability and maintainability of microservices, especially in healthcare systems where failures can have serious consequences.

When NOT to Use Microservices

Microservices are powerful, but they're not a universal solution. In many cases, adopting microservices too early can introduce unnecessary complexity instead of solving real problems.

Before choosing this architecture, it’s important to understand when a simpler approach—such as a monolith—is more appropriate.

1. When the Application Is Small

If your application has limited functionality (for example, a basic patient registration system or internal tool), splitting it into multiple services adds unnecessary overhead.

A monolithic architecture allows you to develop faster with less setup, debug issues more easily, and avoid managing multiple deployments.

Example: A simple clinic portal with only patient registration and appointment booking doesn't require separate services for each feature.

2. When the Team Size Is Limited

When the team size is limited, microservices can become challenging. Managing multiple codebases, handling service communication, and dealing with deployments and monitoring can slow down development, making it tough for small teams to handle the complexity.

Example: A team of 2–3 developers may spend more time managing infrastructure than building features if microservices are used prematurely.

3. When Deployment Complexity Outweighs Benefits

Microservices introduce operational complexity, including API gateways, service discovery, container orchestration (for example, Kubernetes), and monitoring and logging across services.

If your application doesn't require independent scaling or frequent deployments, this complexity may not be justified.

Example: If all components of your system scale together and are updated at the same time, a monolith is often more efficient.

4. When Domain Boundaries Aren't Clear

Microservices rely on well-defined service boundaries. If your domain isn't clearly understood, splitting into services too early can lead to tight coupling between services, frequent cross-service changes, and poorly designed APIs.

In such cases, starting with a monolith and refactoring later is a better approach.

5. When You Lack DevOps and Observability Maturity

Microservices require strong DevOps practices, including CI/CD pipelines, centralized logging, distributed tracing and monitoring & alerting. Without these, debugging issues becomes extremely difficult.

Future Enhancements

Healthcare systems are evolving rapidly, and microservices architectures can adapt to support new capabilities. Future improvements may include:

1.Event-Driven Architecture

Adopting an event-driven approach allows services to communicate asynchronously through events rather than direct requests. This improves scalability, responsiveness, and fault tolerance, making it easier to handle high volumes of patient data and real-time updates across multiple services.

2. AI-Powered Diagnostics

Integrating AI and machine learning can enhance diagnostic capabilities by analyzing patient data, detecting patterns, and providing predictive insights. This can improve clinical decision-making and streamline workflows within the healthcare portal.

3.Integration with FHIR Standards

Supporting FHIR (Fast Healthcare Interoperability Resources) standards enables seamless data exchange between different healthcare systems, labs, and third-party applications. Standardized APIs ensure better interoperability, compliance, and easier integration with external platforms.

4.Real-Time Analytics

Real-time analytics allows healthcare providers to monitor patient data, system performance, and operational metrics continuously. This supports proactive decision-making, early detection of anomalies, and improved overall quality of care.

Conclusion

Microservices-based REST API development provides a powerful foundation for building scalable and secure healthcare portals. By breaking applications into independent services, teams can achieve better scalability, faster deployments, and improved fault isolation.

However, adopting microservices is not just a technical shift—it is an architectural and operational commitment. Developers should start small, identify clear service boundaries, and gradually evolve their systems.

As your application grows, focus on strengthening security, improving observability, and automating deployments. These practices will ensure your healthcare platform remains reliable, compliant, and ready to scale in a cloud-native world.

The next step is to build your first microservice, deploy it using containers, and incrementally expand your system into a fully distributed healthcare platform.

But the trade-off isn't just merely renting the outside infrastructure. It also includes proprietary abstraction lock-in, and an operational and security surface area built on top of vendor capabilities.

In this article, you'll set up a batch ingestion layer on an open-source data lake stack where you own every component.

The focus is deliberately narrow. We'll get the ingestion layer up and running end-to-end. Then we'll build on foundations that allow future extension: analytics, governance, and stream processing without locking you into any single tool for those layers. We'll also review documented integration failures along the way: misconfigured catalogs, partition values written as NULL, and Python version mismatches.

By the end, you'll have:

• A working single-node data lake running on Docker (compose), built on RustFS (object storage), Apache Iceberg (table format), and Project Nessie (catalog).

• A batch pipeline orchestrated with Apache Airflow, executing PySpark jobs that write versioned, partitioned Iceberg tables.

• A real-world ingestion pattern, an external web scraper decoupled from Airflow via Redis, writing raw data to object storage with a lightweight signal table.

• A view of what this stack is and isn't, and what you'd add to take it toward production.

A word on scope: this covers the E in ELT: getting data in. Transformation (dbt, Spark SQL) and analytics (Trino, Superset) are a natural next layer, but are outside the scope of this article. What you build here is the foundation they'd sit on.

What We'll Cover:

• The Ingestion Problem

• Stack

• System Overview

• Quick Start

• Running the Pipelines

• Setup

  • RustFS

  • Nessie

  • Spark

  • Apache Airflow

  • Scrapredis

  • Scrapworker

• Path Forward

  • Extending Capabilities

  • Adding Layers

The Ingestion Problem

The structure of a stack/solution is easier to understand with a use case. A high-level goal is to ingest financial data from external market APIs for trend analysis. You'll focus specifically on setting up ingestion of such data into the warehouse for further analytics.

The data is ingested via a web crawler with a specific rate limit per endpoint. In Batch processing, time-based partitioning is effective for processing by downstream pipelines. It also favors cleaner data retention.

The crawler runs as an external process, decoupled from Airflow via a Redis job queue. This keeps rate limiting and crawl lifecycle outside the orchestration layer, with each component failing and recovering independently.

During ingestion, the priority is data landing with high reliability due to the lack of idempotency in crawl jobs.

Stack

• RustFS: An S3-compatible object store written in Rust

• Project Nessie: Transactional catalog for Apache Iceberg tables

• Apache Spark: Distributed compute engine

• Apache Airflow: Job scheduling and orchestration

• Jupyter Notebook (optional): Ad-hoc Spark queries against Iceberg tables, not covered in this article

• Scrapredis: Job queue for the web crawler

• Scrapworker: Web crawler and ingestion worker

This setup was tested on a 4-core x86/AMD CPU, 16GB RAM, 60GB disk GCP VM running Debian GNU/Linux 11 (Bullseye). Docker with Compose v2 is required. The setup should work on any comparable Linux environment with similar or better specs.

System Overview

The crawler runs as an external process, decoupled from Airflow via a Redis job queue. Airflow pushes a job specification to the queue containing the endpoint, query params, and target path. The crawler picks it up, executes the crawl, and writes raw results directly to object storage.

This separation keeps rate limiting and crawl lifecycle concerns outside the orchestration layer, and isolates failure modes.

A crawl failure is harder to recover since crawl jobs lack idempotency. Pipeline failures after the crawl stage are independently retryable without re-triggering a crawl.

Quick Start

First, initialize the project:

# Clone the repository
git clone https://github.com/ps-mir/data-platform

# Create the shared Docker network
docker network create data-platform

# Create host directories, set permissions, and download Spark JARs
chmod +x init.sh && ./init.sh

Start services in this order (shutdown in reverse):

1. RustFS

cd rustfs && docker compose up -d

1. Nessie

cd nessie && docker compose up -d

1. Spark — requires a build on first run

cd spark && docker compose build && docker compose up -d

1. Scrapredis

cd scrapredis && docker compose up -d

1. Airflow — requires a build on first run

cd airflow-docker && docker compose build && docker compose up -d

Create the Nessie namespaces once after Nessie is up:

curl -X POST http://localhost:19120/iceberg/v1/main/namespaces \
  -H "Content-Type: application/json" \
  -d '{"namespace": ["default"]}'

curl -X POST http://localhost:19120/iceberg/v1/main/namespaces \
  -H "Content-Type: application/json" \
  -d '{"namespace": ["scraper"]}'

Scrapworker runs on the host directly (it's not dockerized). It requires Python >=3.14:

cd scrapworker
pip install -e .
CONFIG_PATH=./config/config.local.yaml RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin python -m scrapworker

Scrapworker must be running before activating scraper_pipeline_v1 in Airflow. Without it, the pipeline will push jobs to the queue with no worker to pick them up and hang indefinitely in wait_for_completion.

Trino is also present in setup but not tested for integration with Nessie yet.

Running the Pipelines

With the stack running, the next step is to activate the pipelines in Airflow. All DAGs are paused at creation by default. The four pipelines build on each other in complexity. Working through them in order is the fastest way to confirm that each layer of the stack is wired correctly before moving to the next.

All four pipelines are loaded but paused by default. Unpause each one in the Airflow UI before triggering.

Let's go over each pipeline:

spark_static_data_v1_skeleton: Hello DAG

This is a minimal DAG with no Spark, just a Python task that prints a message. If it goes green, Airflow's scheduler and worker are healthy. [2026-04-09 22:00:01] INFO - Task operator:<Task(_PythonDecoratedOperator): say_hello>

spark_static_data_v2_submit: Spark Submit

This submits a PySpark job via SparkSubmitOperator that writes a static dataset to an Iceberg table. No partitioning, every run overwrites the previous content.

In Nessie catalog it appears as:

Type: ICEBERG_TABLE
Metadata Location:s3://warehouse/default/static_data_e7e43123-95a7-44d2-b6d5-67c9c7aa4321/metadata/00000-08a5a2db-6f12-4f21-b2a9-de3d9123fbd3.metadata.json

spark_partitioned_data_v1: Spark Partitioned

This extends step2 with time-based partitioning. Partition values are derived from the scheduled slot time, so every run writes to its own (ds, hr, min) partition without touching previous ones.

Example file path in RustFS: warehouse/default/static_data_partitioned_b172c66f-722b-44f3-bbee-069355753ff6/data/ds=2026-03-28/hr=23/min=15/00000-4-7a196a47-2ac0-4023-af68-ca10487fccb2-0-00001.parquet

scraper_pipeline_v1: Scraper Pipeline

This is the full ingestion flow. Airflow pushes a job to Scrapredis, Scrapworker calls the Binance API and writes raw results to RustFS, then Airflow publishes a signal row to the Nessie catalog.

Every run fetches: https://api.binance.com/api/v3/trades?symbol=BTCUSDT&limit=10

Setup

This is a single-node development setup using Docker Compose. It's built on a well-structured base config that can be extended to production with targeted changes.

• A production deployment would require HA configuration, persistent volume management, and security hardening for each component.

• Images are pinned to specific versions to avoid silent breakage between pulls.

• All containers share a common external Docker network named data-platform, which allows services to communicate using container names as hostnames.

• An init.sh script creates the required local dirs inside the data folder and also creates the Docker network.

RustFS

RustFS is the object storage layer in this stack. Nessie's REST catalog mode has a hard dependency on an S3-compatible endpoint. Running it against a local filesystem fails the Nessie healthcheck at startup and causes catalog initialization to error out. The REST catalog is the recommended mode for new setups because it enables credential vending and multi-engine coordination.

MinIO was the natural choice for self-hosted S3-compatible storage, but it shifted to a more restrictive license. RustFS is the open-source alternative, written in Rust and backed by local disk.

At write time, Spark pushes Parquet files directly to RustFS via S3FileIO. Nessie commits the table metadata alongside, so data and catalog state land together or not at all. This is Apache Iceberg's core guarantee: atomic commits across both data files and metadata.

For production or cloud deployments, managed object storage services like AWS S3, Google Cloud Storage, or Azure Blob Storage are the natural next step. Self-hosted alternatives at scale include SeaweedFS, Ceph/RGW, and Garage.

Notes:

• Bucket creation: A rustfs-init sidecar using amazon/aws-cli runs after RustFS passes its healthcheck and creates the s3://warehouse bucket automatically. You don't create the bucket manually.

• Permissions: RustFS runs as uid=10001 inside the container. The host directories (data/rustfs/data and data/rustfs/applogs) must be owned by that uid before the container starts, or it will fail silently. init.sh handles this with sudo chown -R 10001:10001.

• Image pinning: The compose file pins to rustfs/rustfs:1.0.0-alpha.85-glibc. Before upgrading, verify the uid hasn't changed: docker run --rm --entrypoint id rustfs/rustfs:<new-tag>. If it has, re-run init.sh or re-chown manually.

• Spark writes: Spark writes data files directly to RustFS via S3FileIO. Nessie only manages catalog metadata, it doesn't proxy data. The two interact at commit time, not at write time.

Nessie

The catalog tracks the list of tables in the warehouse, along with their data files and schema. Without it, it's hard for Spark to agree on what's in the warehouse.

Hive Metastore offers a Thrift-based API and has been the catalog standard for years. It provides transaction semantics on metadata updates through its backing database, but those transactions stop at the catalog layer. Data files underneath aren't part of the same commit, and there's no cross-table history beyond what the database retains.

Apache Iceberg closes the data and metadata gap with atomic table commits. Nessie builds on that and goes further: it treats the catalog like a Git repository. Every table write is a commit. You can branch, tag, and roll back across multiple tables atomically.

Spark reads and writes table metadata through Nessie's Iceberg REST endpoint. Catalog state is persisted to Postgres, so it survives container restarts.

Namespace bootstrap

Unlike Hive Metastore, Nessie doesn't auto-create namespaces. Attempting to write a table to a namespace that doesn't exist fails after data has already been written to RustFS, leaving orphaned files with no catalog entry. Namespaces are structural metadata and belong in a one-time bootstrap step, not in a pipeline.

Nessie manages the Iceberg catalog metadata under s3://warehouse/. Iceberg table data lands under paths derived from the namespace, for example, s3://warehouse/default/ for the default namespace.

S3 Credential Configuration Issue

Nessie's S3 credential fields don't accept plain strings (likely for security reasons). They require a secret URI in the form urn:nessie-secret:quarkus:<name> even for local credentials.

Additionally, the SCREAMING_SNAKE_CASE environment variable convention is ambiguous for Quarkus property names containing hyphens. The property is silently ignored, and the default (which fails) is used instead. The working approach is dot-notation keys passed directly in the compose environment block, which Quarkus reads without conversion:

nessie.catalog.service.s3.default-options.access-key: "urn:nessie-secret:quarkus:nessie.catalog.secrets.access-key"
nessie.catalog.secrets.access-key.name: rustfsadmin
nessie.catalog.secrets.access-key.secret: rustfsadmin

Nessie health check

Once the RustFS settings are corrected, Nessie's health check URL(http://localhost:9090/q/health) should return the following response:

{
    "status": "UP",
    "checks": [
        {
            "name": "MongoDB connection health check",
            "status": "UP"
        },
        {
            "name": "Warehouses Object Stores",
            "status": "UP",
            "data": {
                "warehouse.warehouse.status": "UP"
            }
        },
        {
            "name": "Database connections health check",
            "status": "UP",
            "data": {
                "<default>": "UP"
            }
        }
    ]
}

The MongoDB connection health check appears in the response even though this stack doesn't use MongoDB. It's a Quarkus built-in probe registered automatically regardless of store type. With JDBC configured, MongoDB is never connected and the UP report is just a placeholder response.

Catalog endpoint vs Management

Nessie exposes two separate APIs. The Iceberg REST catalog is at /iceberg. This is what Spark and Trino connect to. The Nessie management API is at /api/v2, which is for branch operations, commit history, and table inspection. They aren't interchangeable.

# Iceberg REST API
http://localhost:19120/iceberg/v1/main/namespaces
http://localhost:19120/iceberg/v1/config

# Nessie management API
http://localhost:19120/api/v2/config

Notes:

• path-style-access: true is required for any non-AWS S3 endpoint. region is a dummy value required by the AWS SDK internally.

• Nessie's internal port 9000 is remapped to 9090 on the host to avoid conflict with RustFS which occupies 9000 and 9001.

Forward path

Nessie is a stateless REST service, so scaling reads can be done with LB with no coordination between nodes. Durability comes entirely from backend store.

Spark

As a distributed compute engine, Apache Spark is a reliable and stable choice for long-running jobs. In the current setup, it executes PySpark jobs submitted by Airflow, reads and writes Iceberg tables via the Nessie REST catalog, and writes data files directly to RustFS using S3FileIO. Spark runs in standalone mode with a single master and worker, configured via spark-defaults.conf.

Two JARs are required and must be placed in data/spark/jars/ before starting:

• iceberg-spark-runtime-3.5_2.12: Iceberg integration for Spark: SparkCatalog, DataFrameWriterV2, SQL extensions, and all table format logic.

• iceberg-aws-bundle: AWS SDK v2 and Iceberg's S3FileIO, the storage transport layer for writing data files to RustFS. The Spark base image ships only Hadoop AWS (SDK v1). This bundle provides the SDK v2 classes that S3FileIO requires.

Spark uses a custom Dockerfile to install Python 3.12. Build the image before first use:

cd spark
docker compose build
docker compose up -d

The PySpark jobs are covered in the Airflow section, where we walk through each DAG and its corresponding Spark script as part of the pipeline.

Before submitting any Spark job that writes an Iceberg table, the target namespace must exist in Nessie. Nessie doesn't auto-create namespaces, unlike Hive Metastore. Attempting to write to a missing namespace fails after data has already been written to RustFS, leaving orphaned files with no catalog entry.

Create the default namespace once before running any pipeline:

# Nessie should be up and running at this point
curl -X POST http://localhost:19120/iceberg/v1/main/namespaces \
  -H "Content-Type: application/json" \
  -d '{"namespace": ["default"]}'
{
  "namespace" : [ "default" ],
  "properties" : { }
}

Verify:

curl http://localhost:19120/iceberg/v1/main/namespaces

Catalog Mismatch: Tables Missing Across Query Engines

If tables written by Spark aren't visible in Trino, the likely cause is a catalog mismatch. Spark configured with NessieCatalog and Trino using the Iceberg REST catalog maintain separate metadata views — they don't share table state. Both engines must point at the same catalog endpoint: http://nessie:19120/iceberg.

Notes:

• Worker memory: The worker is configured with SPARK_WORKER_MEMORY: 8g. Spark's default is 1g is enough to register but not enough to run a job without queuing. Tune this based on available host memory.

• Remote signing: remote-signing-enabled: false Nessie's REST catalog supports credential vending via IAM/STS, but since that integration isn't present here, remote signing is disabled explicitly to avoid request failures.

• Config changes need full restart: Docker file-level bind mounts cache the inode at container start. Editing spark-defaults.conf won't take effect until Spark and the Airflow worker are restarted. In client mode, the Airflow worker is the Spark driver (the process that reads the config on job submission) and must be restarted too.

• Jupyter Notebook: A Jupyter instance with PySpark is included in the stack for ad-hoc queries against Iceberg tables. It connects to the same Spark cluster and Nessie catalog, so any table written by a pipeline is immediately queryable.

⚠️ Warning: The Spark worker and Airflow worker (the driver) must run the same Python minor version. PySpark enforces this at runtime and fails immediately if they diverge. The Spark image in this stack uses a custom Dockerfile to install Python 3.12, matching Airflow's base image. If you upgrade either, verify that the versions stay aligned.

Apache Airflow

Airflow makes it easier to author, schedule and monitor workflows. In this case, it handles the ingestion for batch processing, but it can be extended to use cases like stream processing.

The Airflow components resemble more closely the DAG processor Airflow Architecture from the official docs.

Key aspects:

• The DAG Processor continuously parses DAG files and serializes them to the Metadata DB.

• The Scheduler reads from there, detects when a DAG run is due, creates task instances, and pushes them to the CeleryExecutor (via Redis queue).

• The Celery worker picks up a task and executes it. In the case of a SparkSubmitOperator, the worker process becomes the Spark driver, submitting the job to the Spark cluster.

• Executors run on the Spark worker, write Parquet files directly to RustFS, and commit the table metadata to Nessie. Airflow records the task outcome back in the Metadata DB.

Airflow uses a custom Dockerfile to install Java 17 and additional providers. Build the image before first use:

cd airflow-docker
docker compose build
docker compose up -d

Pipelines

Pipelines need to be created inside airflow-docker/dags folder for dag processor to pick up load the pipeline DAG in metadata DB. Four pipeline examples are provided with varying complexity.

1. step1_hello_dag.py: single-task DAG with no dependencies, just a Python function that prints a message.

2. step2_spark_submit.py: submits a PySpark job via SparkSubmitOperator. The job writes a static dataset to an Iceberg table via the Nessie catalog.

3. step3_spark_partitioned.py: extends step 2 with time-based partitioning. The scheduled slot time is passed to the PySpark script.

   • Time-based partition values are derived from data_interval_start for idempotency (Backfill, Reruns).

4. scraper_pipeline: a real-world ingestion pipeline. Coordinates with the external task executor scrapworker via the Redis queue scrapredis.

   • Both scrapredis and scrapworker must be up and running for this pipeline to work.

Deploy Mode and Driver Config

The initial SparkSubmitOperator configuration used deploy_mode="cluster", which runs the driver on the Spark cluster rather than the submitting machine. This fails immediately on Spark standalone clusters with a hard error:

Cluster deploy mode is currently not supported for python applications on standalone clusters.

Cluster mode for Python is only available on YARN and Kubernetes. The fix is deploy_mode="client", but this shifts the problem: in client mode, the driver runs on the Airflow worker container, which means the worker needs everything the Spark containers have.

Overall, three changes are required in the Airflow worker:

• The Iceberg and Nessie JARs at /opt/spark/user-jars/

• spark-defaults.conf with catalog, extension, and JAR config

• SPARK_CONF_DIR=/opt/spark/conf, without this, pip-installed PySpark's spark-submit silently ignores the mounted conf file and runs with no catalog config

The fix was adding all three to x-airflow-common in airflow-docker/docker-compose.yaml so every Airflow service inherits them:

environment:
  SPARK_CONF_DIR: /opt/spark/conf

volumes:
  - ../data/spark/jars:/opt/spark/user-jars:ro
  - ../spark/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro

Partition Values Written as NULL

When the third pipeline (Spark Partitioned) ran for the first time, the data landed correctly in RustFS, but querying the Iceberg partitions metadata showed:

+------------------+----------+
|         partition|file_count|
+------------------+----------+
|{NULL, NULL, NULL}|         2|
+------------------+----------+

The original script used Spark's DataSource V1 API:

df.write.format("iceberg").mode("overwrite").saveAsTable(table)

The script used Spark's V1 DataFrame write API with format("iceberg"), which loads an isolated table reference and bypasses Iceberg's catalog write path. As a result, Iceberg committed the data files to storage but wrote NULL partition values into the manifest metadata.

The fix is in Iceberg's native DataFrameWriterV2 API:

df.writeTo(table).overwritePartitions()

This routes through Iceberg's native write path, evaluates partition transforms from the real column values (ds, hr, min), and registers them correctly in the manifest. overwritePartitions() overwrites only the partitions present in the DataFrame. A rerun with the same scheduled time produces the same values and atomically replaces that partition, leaving all others untouched.

⚠️ Existing NULL-partition manifest entries aren't retroactively corrected by subsequent V2 writes. For a brand-new table containing only bad data, DROP TABLE and rewrite is the simplest recovery.

Scrapredis

Scrapredis is a dedicated Redis instance that sits between Airflow and Scrapworker as a job queue. It's separate from Airflow's internal Redis, which exists solely for CeleryExecutor task dispatch. The separation means the crawler's job queue can be managed, scaled, or replaced without touching Airflow's internals.

The pattern generalises beyond scraping. Any external process that needs its own lifecycle, resource profile, or rate limiting can be wired the same way: Airflow pushes a job, the external worker pops it, and Airflow polls for the result.

The scraper pipeline follows this round-trip:

1. Airflow pushes the job payload to the queue:

QUEUE_KEY = "scrapworker:jobs"
client.lpush(QUEUE_KEY, json.dumps(payload))

1. Scrapworker blocks on the queue and pops the next job:

while True:
    _, payload = client.blpop(redis_cfg["queue_key"])

1. Once the crawl finishes, Scrapworker writes the outcome and s3_path back to Redis:

client.set(status_key, json.dumps({"status": "finished", "worker_id": worker_id, "s3_path": job["s3_path"]}), ex=TERMINAL_TTL)

1. The wait_for_completion task polls for that status key. On success, publish_nessie_signal picks up the s3_path and writes the signal row to Nessie.

Scrapworker

Scrapworker is a Python app that uses the Scrapy crawl framework to crawl all pages of the request. It's decoupled from Airflow due to URL/client specific rate limit semantics. For simplicity, consider it a type of external worker that receives and executes requests from Airflow.

It's responsible for downloading and writing content to object storage (RustFS). The Nessie catalog update is decoupled and kept in a separate Airflow pipeline task.

Fixed Signal Table

Scrapworker writes raw JSON to RustFS rather than writing scraped data directly as Iceberg columns. The pipeline then publishes a single lightweight signal row to a Nessie-managed Iceberg table.

The signal schema is fixed and minimal (run_id, endpoint, s3_path, ds, hr, min, published_at). It never changes, regardless of what's being scraped.

Mirroring the scraped payload as Iceberg columns would force Scrapworker to own schema evolution across different endpoints. This isn't an ideal place for schema ownership. Instead, schema ownership sits downstream:

Scrapworker  →  raw files in RustFS  +  signal row in Iceberg (from Pipeline)
Airflow job  →  reads raw via s3_path, applies schema, writes structured Iceberg table

The downstream job knows the domain, knows the schema, and is the right place to handle type casting, nulls, and partition layout. Scrapworker stays generic and thin — the same code handles any endpoint without modification.

Why Signal Publish is a Separate Airflow Task

Scrapworker writes to RustFS and sets status: finished in Redis with the s3_path. A separate Airflow task reads that status and publishes the signal row to Nessie. The two writes are intentionally decoupled.

If scrapworker published to Nessie directly after writing to RustFS, the two writes would share a failure mode. A Nessie failure after a successful RustFS write would leave data stranded with no signal and no clean recovery path. The only option would be a re-crawl which lacks idempotency.

With the decoupled approach, each failure is isolated. A Nessie failure triggers an Airflow retry of the signal publish task only, no re-scrape, no duplicate crawl. RustFS and Nessie failures are independently recoverable.

Notes:

• Raw scraped files are written directly to s3://warehouse/raw/, entirely outside Nessie's management. Nothing in the Iceberg layer touches this path.

• The scrapworker signal table lives in a dedicated scraper namespace. Create it once before scrapworker runs for the first time.

curl -X POST http://localhost:19120/iceberg/v1/main/namespaces \
  -H "Content-Type: application/json" \
  -d '{"namespace": ["scraper"]}'

Path Forward

The stack we've built here is a working ingestion layer. It lands data reliably, tracks it in a versioned catalog, and gives you a foundation to build on. Two directions are worth considering from here.

Extending Capabilities

These are improvements to what's already in the stack, making it more robust without adding new components.

Ingestion reliability: Scrapworker currently handles failures by setting status: failed in Redis, which requires Airflow to re-trigger the full pipeline. Adding client-side rate limiting and per-endpoint retry logic with backoff would make crawl jobs more self-healing, so that a failed page fetch can retry independently without surfacing to Airflow at all.

Config validation: A misconfigured endpoint schema in config.yaml fails silently at runtime, often deep into a crawl. A validate_config() call at startup would catch missing required fields like offset_param or response_map before any job runs. This becomes more important as more endpoints are added.

Observability: Airflow alerting and SLA monitoring give early warning when pipelines miss their schedule or tasks take longer than expected. The signal table is useful here too. A lightweight monitor that checks for expected signal rows within a time window is a simple SLA check that works without external tooling.

Adding Layers

These are new capabilities that build on the ingestion foundation.

Transform layer: The raw Iceberg tables written by the ingestion layer are the input for a transform step. dbt or Spark SQL can read from raw, apply schema, clean types, and write structured tables to a separate namespace. This is the L in ELT and the natural next step once ingestion is stable.

Analytics: Trino is already in the stack and partially integrated. Connecting it fully to Nessie enables SQL queries across all Iceberg tables. Adding Superset on top gives a visualisation layer without requiring any changes to the ingestion pipeline.

Broader source onboarding: The current stack handles one ingestion pattern: a scheduled Airflow pipeline triggering an external HTTP crawler. The same foundation supports pull-based sources like databases using CDC, and push-based sources like event streams via Kafka. The Iceberg tables and Nessie catalog serve as the landing zone regardless of how data arrives.

Governance: Iceberg and Nessie provide the foundations, covering snapshots, schema evolution, commit history, and time travel. The governance layer on top requires deliberate additions: access control, data quality checks, lineage tracking, and schema enforcement. None of these require replacing what's here, as they sit on top of it.

That frustration made the problem feel very clear to me: it was not about having fewer clothes. It was about having better organization, better visibility, and better guidance when making outfit decisions.

So I built a fashion web app that helps users organize their wardrobe, get outfit suggestions, evaluate shopping decisions, and improve recommendations over time using feedback.

In this article, I’ll walk through what the app does, how I built it, the decisions I made along the way, and the challenges that shaped the final result.

Table of Contents

• Table of Contents

• What the App Does

• Why I Built It

• Tech Stack

• Product Walkthrough (What Users See)

• How I Built It

• Challenges I Faced

• What I Learned

• What I Want to Improve Next

• Future Improvements

• Conclusion

What the App Does

At a high level, the app combines six core capabilities:

1. Wardrobe management

2. Outfit recommendations

3. Shopping suggestions

4. Discard recommendations

5. Feedback and usage tracking

6. Secure multi-user accounts

Users can upload clothing items, explore suggested outfits, and mark recommendations as helpful or not helpful. They can also rate outfits and track whether items are worn, kept, or discarded.

That feedback becomes structured data for improving future recommendation quality.

Why I Built It

I wanted to create something that felt personal and actually useful. A lot of fashion apps look polished, but they do not always help with everyday decisions. My goal was to build something that could make wardrobe management easier and outfit selection less overwhelming. The app needed to do three things well:

• store each user’s wardrobe data

• personalize recommendations

• learn from user feedback over time .

That feedback loop mattered to me because it makes the app feel more alive instead of static.

Tech Stack

Here are the tools I used to built the app:

• Frontend: React + Vite

• Backend: FastAPI

• Database: SQLite (local development)

• Background jobs: Celery + Redis

• Authentication: JWT (access + refresh token flow)

• Deployment support: Docker and GitHub Codespaces

This ended up giving me a pretty modular setup, which helped a lot as features started increasing: fast frontend iteration, clean API boundaries, and room to evolve recommendations separately from UI.

Product Walkthrough (What Users See)

1. Onboarding and Account Setup

To start using the app, a user needs to register, verify their email, and complete some profile basics.

Each account is isolated, so wardrobe history and recommendations stay user-specific.

In this onboarding screen above, you can see account creation, email verification, and profile fields for body shape, height, weight, and style preferences.

2. Wardrobe Upload

Users can upload clothing images .

Image analysis labels each item and makes it searchable for recommendations. The wardrobe upload form shows image analysis results with category, dominant color, secondary color, and pattern details listed.

3. Outfit Recommendations

Users can request recommendations, then rate outputs.

Above you can see the outfit recommendation dashboard that shows ranked outfit cards with feedback and rating actions. Recommendations are ranked by a weighted scoring model.

4. Shopping and Discard Assistants

The app evaluates new items against existing wardrobe data and flags low-value wardrobe items that may be worth removing.

You can see the recommendation scores, written reasons (not just a binary decision), and styling guidance for each item above. It also features a "how to style it" incase the user still wants to keep the item.

How I Built It

1. Frontend Setup (React + Vite)

I used React + Vite because I wanted fast iteration and a clean component structure.

The frontend is split into feature areas like onboarding, wardrobe management, outfits, shopping, and discarded-item suggestions. I also keep API calls in a service layer so the UI components stay focused on rendering and interaction.

The snippet below is a simplified example of the API service pattern used in the app. It is not meant to be copy-pasted as-is, but it shows the same structure the frontend uses when talking to the backend.

Example API client pattern:

export async function getOutfitRecommendations(userId, params = {}) {
  const query = new URLSearchParams(params).toString();
  const url = `/users/\({userId}/outfits/recommend\){query ? `?${query}` : ""}`;

  const response = await fetch(url, {
    headers: {
      Authorization: `Bearer ${localStorage.getItem("access_token")}`,
    },
  });

  if (!response.ok) {
    throw new Error("Failed to fetch outfit recommendations");
  }

  return response.json();
}

Here's what's happening in that snippet:

• URLSearchParams builds optional query strings like occasion, season, or limit.

• The request path is user-scoped, which keeps each user’s recommendations isolated.

• The Authorization header sends the access token so the backend can verify the session.

• The response is checked before parsing so the UI can surface a useful error if the request fails.

This pattern kept the frontend simple and reusable as the number of API calls grew.

2. Backend Architecture with FastAPI

The backend is organized around clear route groups:

• auth routes for register, login, refresh, logout, and sessions

• user analysis routes

• wardrobe CRUD routes

• recommendation routes for outfits, shopping, and discard analysis

• feedback routes for ratings and helpfulness signals

One of the most important design choices was enforcing ownership checks on user-scoped resources. That prevented one user from accessing another user’s wardrobe or feedback data.

The backend snippet below is another simplified example from the app’s route layer. It shows the request validation and orchestration logic, while the actual scoring work stays in the recommendation service.

@app.get("/users/{user_id}/outfits/recommend")
def recommend_outfits(user_id: int, occasion: str | None = None, season: str | None = None, limit: int = 10):
    user = get_user_or_404(user_id)
    wardrobe_items = get_user_wardrobe(user_id)

    if len(wardrobe_items) < 2:
        raise HTTPException(status_code=400, detail="Not enough wardrobe items")

    recommendations = outfit_generator.generate_outfit_recommendations(
        wardrobe_items=wardrobe_items,
        body_shape=user.body_shape,
        undertone=user.undertone,
        occasion=occasion,
        season=season,
        top_k=limit,
    )

    return {"user_id": user_id, "recommendations": recommendations}

Here's how to read that code:

• get_user_or_404 loads the profile data needed for personalization.

• get_user_wardrobe fetches only the current user’s items.

• The minimum wardrobe check prevents the recommendation logic from running on incomplete data.

• generate_outfit_recommendations handles the scoring logic separately, which keeps the route handler small and easier to test.

• The response returns the results in a shape the frontend can consume directly.

That separation helped keep the API layer readable while the recommendation logic stayed isolated in its own service.

3. Recommendation Logic

I intentionally started with deterministic rules before introducing heavy ML. That made behavior easier to debug and explain.

The outfit recommender scores combinations using weighted signals:

$$\text{outfit score} = 0.4 \cdot \text{color harmony} + 0.4 \cdot \text{body-shape fit} + 0.2 \cdot \text{undertone fit}$$

The snippet below is a simplified example from the recommendation engine. It shows how the app combines multiple signals into a single score:

def score_outfit(combo, user_context):
    color_score = color_harmony.score(combo)
    shape_score = body_shape_rules.score(combo, user_context.body_shape)
    undertone_score = undertone_rules.score(combo, user_context.undertone)

    total = 0.4 * color_score + 0.4 * shape_score + 0.2 * undertone_score
    return round(total, 3)

The logic behind this approach is straightforward:

• color harmony helps the outfit feel visually coherent

• body-shape scoring helps the outfit feel flattering

• undertone scoring helps the colors work better with the user’s profile

I used a similar structure for discard recommendations and shopping suggestions, but with different factors and thresholds.

4. Authentication and Secure Multi-user Design

Security was one of the most important parts of this build.

I implemented:

• short-lived access tokens

• refresh tokens with JTI tracking

• token rotation on refresh

• session revocation (single session and all sessions)

• email verification and password reset flows

The snippet below is a simplified example of the refresh-token lifecycle used in the app. It shows the important control points rather than every helper function:

def refresh_access_token(refresh_token: str):
    payload = decode_jwt(refresh_token)
    jti = payload["jti"]

    token_record = db.get_refresh_token(jti)
    if not token_record or token_record.revoked:
        raise AuthError("Invalid refresh token")

    new_refresh, new_jti = issue_refresh_token(payload["sub"])
    token_record.revoked = True
    token_record.replaced_by_jti = new_jti

    new_access = issue_access_token(payload["sub"])
    return {"access_token": new_access, "refresh_token": new_refresh}

What this code is doing:

• It decodes the refresh token and looks up its JTI in the database.

• It rejects reused or revoked sessions, which helps prevent replay attacks.

• It rotates the refresh token instead of reusing it.

• It issues a fresh access token so the session stays valid without forcing the user to log in again.

This design made multi-device sessions safer and gave me server-side control over logout behavior.

5. Background Jobs for Long-running Operations

Image analysis can be expensive, especially when the app needs to classify clothing, analyze colors, and estimate body-shape-related signals. To keep the request path responsive, I added Celery + Redis support for background tasks.

That gave the app two modes:

• synchronous processing for simpler local development

• queued processing for heavier or slower jobs

That tradeoff mattered because it let me keep the developer experience simple without blocking the app during more expensive work.

6. Data Model and Feedback Capture

A recommendation system only improves if it captures the right signals.

So I added dedicated feedback tables for:

• outfit ratings (1-5 + optional comments)

• recommendation helpful/unhelpful feedback

• item usage actions (worn/kept/discarded)

Here is the shape of one of those models:

class RecommendationFeedback(Base):
    __tablename__ = "recommendation_feedback"

    id = Column(Integer, primary_key=True)
    user_id = Column(Integer, ForeignKey("users.id"), nullable=False)
    recommendation_type = Column(String(50), nullable=False)
    recommendation_id = Column(Integer, nullable=False)
    helpful = Column(Boolean, nullable=False)
    created_at = Column(DateTime, default=datetime.utcnow)

How to read this model:

• user_id ties feedback to the person who gave it.

• recommendation_type tells me whether the feedback belongs to outfits, shopping, or discard suggestions.

• recommendation_id identifies the exact recommendation.

• helpful stores the user’s direct response.

• created_at makes it possible to analyze feedback trends over time.

This part of the system gives the app a real learning foundation, even though the feedback-to-model-update loop is still a future improvement.

Challenges I Faced

This was the section that taught me the most.

1. Image-heavy endpoints were slower than I wanted

The analyze and wardrobe upload flows were doing a lot of work at once: image validation, classification, color extraction, storage, and database writes.

At first, that made the request flow feel heavier than it should have.

What I changed:

• I bounded concurrent image jobs so the app wouldn't try to do too much at once.

• I separated slower jobs into background processing where possible.

• I used load-test results to confirm which endpoints were actually expensive.

The practical effect was that heavy image requests stopped competing with each other so aggressively. Instead of letting many expensive tasks pile up inside the same request cycle, I limited the active work and pushed slower operations into the queue when needed.

Why this fixed it:

• Bounding concurrency prevented the system from overloading CPU-bound tasks.

• Moving expensive work into async jobs kept the main request/response cycle more responsive.

• Load testing gave me evidence instead of guesswork, so I could tune the system based on real performance behavior.

In other words, I didn't just “optimize” the endpoint in theory. I changed the execution model so expensive analysis could not block every other request behind it.

2. JWT sessions needed real server-side control

A basic JWT setup is easy to get working, but it becomes less useful if you cannot revoke sessions or manage multiple devices cleanly.

What I changed:

• I stored refresh tokens in the database.

• I tracked token JTI values.

• I rotated refresh tokens when users refreshed their session.

• I added endpoints for logging out a single session or all sessions.

The important shift here was moving from “token exists, therefore session is valid” to “token exists, matches the database record, and has not been revoked or replaced.” That gave the server the authority to invalidate old sessions immediately.

Why this fixed it:

• Server-side token tracking made revocation possible.

• Rotation reduced the chance of token reuse.

• Session management became visible to the user, which made the app feel more trustworthy.

This is what made logout-all and multi-device management work in a real way instead of just being cosmetic UI actions.

3. User data isolation had to be explicit

Because this is a multi-user app, I had to be careful that one account could never accidentally see another account’s wardrobe data.

What I changed:

• I added ownership checks to user-scoped routes.

• I kept all wardrobe and feedback queries filtered by user_id.

• I used encrypted image storage instead of exposing raw paths.

In practice, this meant every route had to ask the same question: “Does this user own the resource they are trying to access?” If the answer was no, the request stopped immediately.

Why this fixed it:

• Ownership checks made data access rules explicit.

• User-filtered queries prevented accidental cross-account reads.

• Encrypted storage improved privacy and reduced the risk of exposing image data directly.

That combination is what kept wardrobe data, feedback history, and images separated correctly across accounts.

4. Docker made the project easier to share, but only after the stack was organized

The app includes the frontend, backend, Redis, Celery worker, and Celery Beat, so the first challenge was making the setup feel reproducible instead of fragile.

What I changed:

• I defined the stack in Docker Compose.

• I documented the required environment variables.

• I kept the dev stack aligned with how the app runs in practice.

This removed a lot of setup ambiguity. Instead of asking someone to manually figure out how the frontend, backend, Redis, and workers fit together, I made the stack describe itself.

Why this fixed it:

• Docker let contributors start the project with fewer manual steps.

• Clear environment configuration reduced setup mistakes.

• Matching the stack to the architecture made the app easier to understand and test.

That was important because the app depends on several moving parts, and the simplest way to make the project approachable was to make startup behavior predictable.

What I Learned

This project taught me a few important lessons:

• Small features become much more valuable when they work together.

• Feedback data is one of the strongest signals for improving recommendations.

• Clean data modeling matters a lot when multiple users are involved.

• Docker and clear setup instructions make a project much easier for other people to try.

I also learned that a project does not need to be huge to be useful. A focused app that solves one problem well can still feel meaningful.

What I Want to Improve Next

My roadmap from here:

1. Integrate feedback directly into ranking updates

2. Add visual analytics for recommendation quality trends

3. Improve mobile UX parity

4. Deploy with persistent cloud storage and production database defaults

5. Provide a public demo mode for easier evaluation

Future Improvements

There are still a few things I would like to add later:

• a more advanced recommendation engine

• visual analytics for user feedback

• better mobile support

• live deployment with persistent cloud storage

• a public demo mode for easier testing

Conclusion

This project began as a personal frustration and turned into a full web application with authentication, wardrobe storage, recommendation logic, and feedback infrastructure.

The most rewarding part was seeing how practical software decisions, not just flashy UI, can help people make everyday choices faster.

If you want to explore or run the project, check out the repo. You can try the flows and share feedback. I would especially love input on recommendation quality, UX clarity, and what features would make this genuinely useful in daily life.

Google Cloud Axion is Google's own custom ARM-based chip, built to handle the demands of modern cloud workloads. The performance and cost numbers are striking: Google claims Axion delivers up to 60% better energy efficiency and up to 65% better price-performance compared to comparable x86 machines.

AWS has Graviton. Azure has Cobalt. ARM is no longer niche. It's the direction the entire cloud industry is moving.

But there's a problem that catches almost every team off guard when they start this transition: container architecture mismatch.

If you build a Docker image on your M-series Mac and push it to an x86 server, it crashes on startup with a cryptic exec format error.

The server isn't broken. It just can't read the compiled instructions inside your image. An ARM binary and an x86 binary are written in fundamentally different languages at the machine level. The CPU literally can't execute instructions it wasn't designed for.

We're going to solve this problem completely in this tutorial. You'll build a single Docker image tag that automatically serves the correct binary on both ARM and x86 machines — no separate pipelines, no separate tags. Then you'll provision Google Cloud ARM nodes in GKE and configure your Kubernetes deployment to route workloads precisely to those cost-efficient nodes.

Here's what you'll build, step by step:

• A Go HTTP server that reports the CPU architecture it's running on at runtime

• A multi-stage Dockerfile that cross-compiles for both linux/amd64 and linux/arm64 without slow QEMU emulation

• A multi-arch image in Google Artifact Registry that acts as a single entry point for any architecture

• A GKE cluster with two node pools: a standard x86 pool and an ARM Axion pool

• A Kubernetes Deployment that pins your workload exclusively to the ARM nodes

By the end, you'll hit a live endpoint and see the word arm64 staring back at you from a Google Cloud ARM node. Let's get into it.

Table of Contents

• Prerequisites

• Step 1: Set Up Your Google Cloud Project

• Step 2: Create the GKE Cluster

• Step 3: Write the Application

• Step 4: Enable Multi-Arch Builds with Docker Buildx

• Step 5: Write the Dockerfile

• Step 6: Build and Push the Multi-Arch Image

• Step 7: Add the Axion ARM Node Pool

• Step 8: Deploy the App to the ARM Node Pool

• Step 9: Verify the Deployment

• Step 10: Cost Savings and Tradeoffs

• Cleanup

• Conclusion

• Project File Structure

Prerequisites

Before you start, make sure you have the following ready:

• A Google Cloud project with billing enabled. If you don't have one, create it at console.cloud.google.com. The total cost to follow this tutorial is around $5–10.

• gcloud CLI installed and authenticated. Run gcloud auth login to sign in and gcloud config set project YOUR_PROJECT_ID to point it at your project.

• Docker Desktop version 19.03 or later. Docker Buildx (the tool we'll use for multi-arch builds) ships bundled with it.

• kubectl installed. This is the CLI for interacting with Kubernetes clusters.

• Basic familiarity with Docker (images, layers, Dockerfile) and Kubernetes (pods, deployments, services). You don't need to be an expert, but you should know what these things are.

Step 1: Set Up Your Google Cloud Project

Before writing a single line of application code, let's get the cloud infrastructure side ready. This is the foundation everything else will build on.

Enable the Required APIs

Google Cloud services are off by default in any new project. Run this command to turn on the three APIs we'll need:

gcloud services enable \
  artifactregistry.googleapis.com \
  container.googleapis.com \
  containeranalysis.googleapis.com

Here's what each one does:

• artifactregistry.googleapis.com — enables Artifact Registry, where we'll store our Docker images

• container.googleapis.com — enables Google Kubernetes Engine (GKE), where our cluster will run

• containeranalysis.googleapis.com — enables vulnerability scanning for images stored in Artifact Registry

Create a Docker Repository in Artifact Registry

Artifact Registry is Google Cloud's managed container image store — the place where our built images will live before being deployed to the cluster. Create a dedicated repository for this tutorial:

gcloud artifacts repositories create multi-arch-repo \
  --repository-format=docker \
  --location=us-central1 \
  --description="Multi-arch tutorial images"

Breaking down the flags:

• --repository-format=docker — tells Artifact Registry this repository stores Docker images (as opposed to npm packages, Maven artifacts, and so on)

• --location=us-central1 — the Google Cloud region where your images will be stored. Use a region that's close to where your cluster will run to minimize image pull latency. Run gcloud artifacts locations list to see all options.

• --description — a human-readable label for the repository, shown in the console.

Authenticate Docker to Push to Artifact Registry

Docker needs credentials before it can push images to Google Cloud. Run this command to wire up authentication automatically:

gcloud auth configure-docker us-central1-docker.pkg.dev

This adds a credential helper entry to your ~/.docker/config.json file. What that means in practice: any time Docker tries to push or pull from a URL under us-central1-docker.pkg.dev, it will automatically call gcloud to get a valid auth token. You won't need to run docker login manually.

Step 2: Create the GKE Cluster

With Artifact Registry ready to receive images, let's create the Kubernetes cluster. We'll start with a standard cluster using x86 nodes and add an ARM node pool later once we have an image to deploy.

gcloud container clusters create axion-tutorial-cluster \
  --zone=us-central1-a \
  --num-nodes=2 \
  --machine-type=e2-standard-2 \
  --workload-pool=PROJECT_ID.svc.id.goog

Replace PROJECT_ID with your actual Google Cloud project ID.

What each flag does:

• --zone=us-central1-a — creates a zonal cluster in a single availability zone. A regional cluster (using --region) would spread nodes across three zones for higher resilience, but for this tutorial a single zone keeps things simple and avoids capacity issues that can affect specific zones. If us-central1-a is unavailable, try us-central1-b.

• --num-nodes=2 — two x86 nodes in this zone. We need at least 2 to have enough capacity alongside our ARM node pool later.

• --machine-type=e2-standard-2 — the machine type for this default node pool. e2-standard-2 is a cost-effective x86 machine with 2 vCPUs and 8 GB of memory, good for general workloads.

• --workload-pool=PROJECT_ID.svc.id.goog — enables Workload Identity, which is Google's recommended way for pods to authenticate with Google Cloud APIs. It avoids the need to download and store service account key files inside your cluster.

This command takes a few minutes. While it runs, you can move on to writing the application. We'll come back to the cluster in Step 6.

Step 3: Write the Application

We need an application to containerize. We'll use Go for three specific reasons:

1. Go compiles into a single, statically-linked binary. There's no runtime to install, no interpreter — just the binary. This makes for extremely lean container images.

2. Go has first-class, built-in cross-compilation support. We can compile an ARM64 binary from an x86 Mac, or vice versa, by setting two environment variables. This will matter a lot when we get to the Dockerfile.

3. Go exposes the architecture the binary was compiled for via runtime.GOARCH. Our server will report this at runtime, giving us hard proof that the correct binary is running on the correct hardware.

Start by creating the project directories:

mkdir -p hello-axion/app hello-axion/k8s
cd hello-axion/app

Initialize the Go module from inside app/. This creates go.mod in the current directory:

go mod init hello-axion

go mod init is Go's built-in command for starting a new module. It writes a go.mod file that declares the module name (hello-axion) and the minimum Go version required. Every modern Go project needs this file — without it, the compiler doesn't know how to resolve packages.

Now create the application at app/main.go:

package main

import (
    "fmt"
    "net/http"
    "os"
    "runtime"
)

func handler(w http.ResponseWriter, r *http.Request) {
    hostname, _ := os.Hostname()
    fmt.Fprintf(w, "Hello from freeCodeCamp!\n")
    fmt.Fprintf(w, "Architecture : %s\n", runtime.GOARCH)
    fmt.Fprintf(w, "OS           : %s\n", runtime.GOOS)
    fmt.Fprintf(w, "Pod hostname : %s\n", hostname)
}

func healthz(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprintln(w, "ok")
}

func main() {
    http.HandleFunc("/", handler)
    http.HandleFunc("/healthz", healthz)
    fmt.Println("Server starting on port 8080...")
    if err := http.ListenAndServe(":8080", nil); err != nil {
        fmt.Fprintf(os.Stderr, "server error: %v\n", err)
        os.Exit(1)
    }
}

Verify both files were created:

ls -la

You should see go.mod and main.go listed.

Let's walk through what this code does:

• import "runtime" — imports Go's built-in runtime package, which exposes information about the Go runtime environment, including the CPU architecture.

• runtime.GOARCH — returns a string like "arm64" or "amd64" representing the architecture this binary was compiled for. When we deploy to an ARM node, this value will be arm64. This is the core of our proof.

• os.Hostname() — returns the pod's hostname, which Kubernetes sets to the pod name. This lets us see which specific pod responded when we test the app later.

• handler — the main HTTP handler, registered on the root path /. It writes the architecture, OS, and hostname to the response.

• healthz — a separate handler registered on /healthz. It returns HTTP 200 with the text ok. Kubernetes will use this endpoint to check whether the container is alive and ready to serve traffic — we'll wire this up in the deployment manifest later.

• http.ListenAndServe(":8080", nil) — starts the server on port 8080. If it fails to start (for example, if the port is already in use), it prints the error and exits with a non-zero code so Kubernetes knows something went wrong.

Step 4: Enable Multi-Arch Builds with Docker Buildx

Before we write the Dockerfile, we need to understand a fundamental constraint, because it directly shapes how the Dockerfile must be written.

Why Your Docker Images Are Architecture-Specific By Default

A CPU only understands instructions written for its specific Instruction Set Architecture (ISA). ARM64 and x86_64 are different ISAs — different vocabularies of machine-level operations. When you compile a Go program, the compiler translates your source code into binary instructions for exactly one ISA. That binary can't run on a different ISA.

When you build a Docker image the normal way (docker build), the binary inside that image is compiled for your local machine's ISA. If you're on an Apple Silicon Mac, you get an ARM64 binary. Push that image to an x86 server, and when Docker tries to execute the binary, the kernel rejects it:

standard_init_linux.go:228: exec user process caused: exec format error

That's the operating system saying: "This binary was written for a different processor. I have no idea what to do with it."

The Solution: A Single Image Tag That Serves Any Architecture

Docker solves this with a structure called a Manifest List (also called a multi-arch image index). Instead of one image, a Manifest List is a pointer table. It holds multiple image references — one per architecture — all under the same tag.

When a server pulls hello-axion:v1, here's what actually happens:

1. Docker contacts the registry and requests the manifest for hello-axion:v1

2. The registry returns the Manifest List, which looks like this internally:

{
  "manifests": [
    { "digest": "sha256:a1b2...", "platform": { "architecture": "amd64", "os": "linux" } },
    { "digest": "sha256:c3d4...", "platform": { "architecture": "arm64", "os": "linux" } }
  ]
}

1. Docker checks the current machine's architecture, finds the matching entry, and pulls only that specific image layer. The x86 image never downloads onto your ARM server, and vice versa.

One tag, two actual images. Completely transparent to your deployment manifests.

Set Up Docker Buildx

Docker Buildx is the CLI tool that builds these Manifest Lists. It's powered by the BuildKit engine and ships bundled with Docker Desktop. Run the following to create and activate a new builder instance:

docker buildx create --name multiarch-builder --use

• --name multiarch-builder — gives this builder a memorable name. You can have multiple builders. This command creates a new one named multiarch-builder.

• --use — immediately sets this new builder as the active one, so all future docker buildx build commands use it.

Now boot the builder and confirm it supports the platforms we need:

docker buildx inspect --bootstrap

• --bootstrap — starts the builder container if it isn't already running, and prints its full configuration.

You should see output like this:

Name:          multiarch-builder
Driver:        docker-container
Platforms:     linux/amd64, linux/arm64, linux/arm/v7, linux/386, ...

The Platforms line lists every architecture this builder can produce images for. As long as you see linux/amd64 and linux/arm64 in that list, you're ready to build for both x86 and ARM.

Step 5: Write the Dockerfile

Now we can write the Dockerfile. We'll use two techniques together: a multi-stage build to keep the final image tiny, and a cross-compilation trick to avoid slow CPU emulation.

Create app/Dockerfile with the following content:

# -----------------------------------------------------------
# Stage 1: Build
# -----------------------------------------------------------
# $BUILDPLATFORM = the machine running this build (your laptop)
# \(TARGETOS / \)TARGETARCH = the platform we are building FOR
# -----------------------------------------------------------
FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

ARG TARGETOS
ARG TARGETARCH

WORKDIR /app

COPY go.mod .
RUN go mod download

COPY main.go .

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

# -----------------------------------------------------------
# Stage 2: Runtime
# -----------------------------------------------------------

FROM alpine:latest

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

WORKDIR /app
COPY --from=builder /app/server .

EXPOSE 8080
CMD ["./server"]

There's a lot happening here. Let's go through it carefully.

Stage 1: The Builder

FROM --platform=$BUILDPLATFORM golang:1.23-alpine AS builder

This is the most important line in the file. \(BUILDPLATFORM is a special build argument that Docker Buildx automatically injects — it equals the platform of the machine running the build (your laptop). By pinning the builder stage to \)BUILDPLATFORM, the Go compiler always runs natively on your machine, not inside a CPU emulator. This is what makes multi-arch builds fast.

Without --platform=$BUILDPLATFORM, Buildx would have to use QEMU — a full CPU emulator — to run an ARM64 build environment on your x86 machine (or vice versa). QEMU works, but it's typically 5–10 times slower than native execution. For a project with many dependencies, that's the difference between a 2-minute build and a 20-minute build.

ARG TARGETOS and ARG TARGETARCH

These two lines declare that our Dockerfile expects build arguments named TARGETOS and TARGETARCH. Buildx injects these automatically based on the --platform flag you pass at build time. For a linux/arm64 target, TARGETOS will be linux and TARGETARCH will be arm64.

COPY go.mod . and RUN go mod download

We copy go.mod first, before copying the rest of the source code. Docker builds images layer by layer and caches each layer. By copying only the module file first, we create a cached layer for go mod download.

On future builds, as long as go.mod hasn't changed, Docker skips the download step entirely — even if the source code changed. This speeds up iterative development significantly.

RUN GOOS=\(TARGETOS GOARCH=\)TARGETARCH go build -ldflags="-w -s" -o server main.go

This is the cross-compilation step. GOOS and GOARCH are Go's built-in cross-compilation environment variables. Setting them tells the Go compiler to produce a binary for a different target than the machine it's running on. We set them from the \(TARGETOS and \)TARGETARCH build args injected by Buildx.

The -ldflags="-w -s" flag strips the debug symbol table and the DWARF debugging information from the binary. This has no effect on runtime behavior but reduces the binary size by roughly 30%.

Stage 2: The Runtime Image

FROM alpine:latest

This starts a brand-new image from Alpine Linux — a minimal Linux distribution that weighs about 5 MB. Critically, alpine:latest is itself a multi-arch image, so Docker automatically selects the arm64 or amd64 Alpine variant depending on which platform this stage is built for.

Everything from Stage 1 — the Go toolchain, the source files, the intermediate object files — is discarded. The final image contains only Alpine Linux plus our binary. Compared to a naive single-stage Go image (~300 MB), this approach produces an image under 15 MB.

RUN addgroup -S appgroup && adduser -S appuser -G appgroup and USER appuser

These two lines create a non-root user and set it as the active user for the container. Running containers as root is a security risk — if an attacker exploits a vulnerability in your application, they gain root access inside the container. Running as a non-root user limits the blast radius.

COPY --from=builder /app/server .

This is how multi-stage builds work: the --from=builder flag tells Docker to copy files from the builder stage (Stage 1), not from your local disk. Only the compiled binary (server) makes it into the final image.

Step 6: Build and Push the Multi-Arch Image

With the application and Dockerfile in place, we can now build images for both architectures and push them to Artifact Registry — all in a single command.

From inside the app/ directory, run:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 \
  --push \
  .

Replace PROJECT_ID with your actual GCP project ID.

Here's what each part of this command does:

• docker buildx build — uses the Buildx CLI instead of the standard docker build. Buildx is required for multi-platform builds.

• --platform linux/amd64,linux/arm64 — instructs Buildx to build the image twice: once targeting x86 Intel/AMD machines, and once targeting ARM64. Both builds run in parallel. Because our Dockerfile uses the $BUILDPLATFORM cross-compilation trick, both builds run natively on your machine without QEMU emulation.

• -t us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1 — the full image path in Artifact Registry. The format is always REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME:TAG.

• --push — multi-arch images can't be loaded into your local Docker daemon (which only understands single-architecture images). This flag tells Buildx to skip local storage and push the completed Manifest List — with both architecture variants — directly to the registry.

• . — the build context, the directory Docker scans for the Dockerfile and any files the build needs.

Watch the output as the build runs. You'll see BuildKit working on both platforms simultaneously:

 => [linux/amd64 builder 1/5] FROM golang:1.23-alpine
 => [linux/arm64 builder 1/5] FROM golang:1.23-alpine
 ...
 => pushing manifest for us-central1-docker.pkg.dev/.../hello-axion:v1

Verify the Multi-Arch Image in Artifact Registry

Once the push completes, navigate to GCP Console → Artifact Registry → Repositories → multi-arch-repo and click on hello-axion.

You won't see a single image — you'll see something labelled "Image Index". That's the Manifest List we created. Click into it, and you'll find two child images with separate digests, one for linux/amd64 and one for linux/arm64.

You can also inspect this from the command line:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

The output lists every manifest inside the image index. You'll see entries for linux/amd64 and linux/arm64 — those are our two real images. You'll also see two entries with Platform: unknown/unknown labelled as attestation-manifest. These are build provenance records that Docker Buildx automatically attaches to prove how and where the image was built (a supply chain security feature called SLSA attestation).

The two entries you care about are linux/amd64 and linux/arm64. Note the digest for the arm64 entry — we'll use it in the verification step to confirm the cluster pulled the right variant.

Step 7: Add the Axion ARM Node Pool

We have a universal image. Now we need somewhere to run it.

Recall the cluster we created in Step 2 — it's running e2-standard-2 x86 machines. We're going to add a second node pool running ARM machines. This is the key architectural move: a mixed-architecture cluster where different workloads can be routed to different hardware.

Choosing Your ARM Machine Type

Google Cloud currently offers two ARM-based machine series in GKE:

This tutorial uses t2a-standard-2 because it's widely available. The commands are identical for c4a-standard-2 — just swap the --machine-type value. If t2a-standard-2 isn't available in your zone, GKE will tell you immediately when you run the node pool creation command below, and you can try a neighbouring zone.

Create the ARM Node Pool

Add the ARM node pool to your existing cluster:

gcloud container node-pools create axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a \
  --machine-type=t2a-standard-2 \
  --num-nodes=2 \
  --node-labels=workload-type=arm-optimized

What each flag does:

• --cluster=axion-tutorial-cluster — the name of the cluster we created in Step 2. Node pools are always added to an existing cluster.

• --zone=us-central1-a — must match the zone you used when creating the cluster.

• --machine-type=t2a-standard-2 — GKE detects this is an ARM machine type and automatically provisions the nodes with an ARM-compatible version of Container-Optimized OS (COS). You don't need to configure anything special for ARM at the OS level.

• --num-nodes=2 — two ARM nodes in the zone, enough to schedule our 3-replica deployment alongside other cluster overhead.

• --node-labels=workload-type=arm-optimized — attaches a custom label to every node in this pool. We'll use this label in our deployment manifest to target these specific nodes. Using a descriptive custom label (rather than just relying on the automatic kubernetes.io/arch=arm64 label) is good practice in real clusters — it communicates the intent of the pool, not just its hardware.

This command takes a few minutes. Once it completes, let's confirm our cluster now has both node pools:

gcloud container clusters get-credentials axion-tutorial-cluster --zone=us-central1-a

kubectl get nodes --label-columns=kubernetes.io/arch

The get-credentials command configures kubectl to authenticate with your new cluster. The get nodes command then lists all nodes and adds a column showing the kubernetes.io/arch label.

You should see something like:

NAME                                    STATUS   ARCH    AGE
gke-...default-pool-abc...              Ready    amd64   15m
gke-...default-pool-def...              Ready    amd64   15m
gke-...axion-pool-jkl...                Ready    arm64   3m
gke-...axion-pool-mno...                Ready    arm64   3m

amd64 for the default x86 pool, arm64 for our new Axion pool. This kubernetes.io/arch label is applied automatically by GKE — you don't set it, it's derived from the hardware.

Step 8: Deploy the App to the ARM Node Pool

We have a multi-arch image and a mixed-architecture cluster. Here's something important to understand before writing the deployment manifest: Kubernetes doesn't know or care about image architecture by default.

If you applied a standard Deployment right now, the scheduler would look for any available node with enough CPU and memory and place pods there — potentially landing on x86 nodes instead of your ARM Axion nodes. The multi-arch Manifest List would handle this gracefully (the right binary would run regardless), but you'd lose the cost efficiency you provisioned Axion nodes for in the first place.

To guarantee that pods land on ARM nodes and only ARM nodes, we use a nodeSelector.

How nodeSelector Works

A nodeSelector is a set of key-value pairs in your pod spec. Before the Kubernetes scheduler places a pod, it checks every available node's labels. If a node doesn't have all the labels in the nodeSelector, the scheduler skips it — the pod will remain in Pending state rather than land on the wrong node.

This is a hard constraint, which is exactly what we want for cost optimization. Contrast this with Node Affinity's soft preference mode (preferredDuringSchedulingIgnoredDuringExecution), which says "try to use ARM, but fall back to x86 if needed." Soft preferences are useful for resilience, but they undermine the whole point of dedicated ARM pools. We want the hard constraint.

Write the Deployment Manifest

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-axion
  labels:
    app: hello-axion
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-axion
  template:
    metadata:
      labels:
        app: hello-axion
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64

      containers:
      - name: hello-axion
        image: us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 5
        resources:
          requests:
            cpu: "250m"
            memory: "64Mi"
          limits:
            cpu: "500m"
            memory: "128Mi"

Replace PROJECT_ID with your project ID. Here's what the key sections do:

replicas: 3 — tells Kubernetes to keep three instances of this pod running at all times. If one crashes or a node goes down, the scheduler spins up a replacement. Three replicas also means one pod per ARM node in us-central1, which distributes load across availability zones.

selector.matchLabels and template.metadata.labels — these two blocks must match. The selector tells the Deployment which pods it "owns," and the template.metadata.labels is what those pods will be tagged with. If they don't match, Kubernetes won't be able to manage the pods.

nodeSelector: kubernetes.io/arch: arm64 — this is the pin. The Kubernetes scheduler filters out every node that doesn't carry this label before considering resource availability. Since GKE automatically applies kubernetes.io/arch=arm64 to all ARM nodes, our pods will schedule only onto the axion-pool nodes.

livenessProbe — periodically calls GET /healthz. If this check fails a certain number of times in a row (indicating the container has deadlocked or is otherwise unresponsive), Kubernetes restarts the container. initialDelaySeconds: 5 gives the server 5 seconds to start up before the first check.

readinessProbe — similar to the liveness probe, but with a different purpose. While the readiness probe is failing, Kubernetes removes the pod from the service's load balancer, so no traffic is sent to it. This is important during startup — the pod won't receive traffic until it signals it's ready.

resources.requests — reserves 250m (25% of a CPU core) and 64Mi of memory on the node for this pod. The scheduler uses these numbers to decide whether a node has enough room for the pod. Setting requests is required for sensible bin-packing. Without them, nodes can be silently overcommitted.

resources.limits — caps the container at 500m CPU and 128Mi memory. If the container exceeds these limits, Kubernetes throttles the CPU or kills the container (for memory). This prevents a single misbehaving pod from starving other workloads on the same node.

A Note on Taints and Tolerations

Once you're comfortable with nodeSelector, the next step in production clusters is adding a taint to your ARM node pool. A taint is a repellent — any pod without an explicit toleration for that taint is blocked from landing on the tainted node.

This means other workloads in your cluster can't accidentally consume your ARM capacity. You'd add the taint when creating the pool:

# Add --node-taints to the pool creation command:
--node-taints=workload-type=arm-optimized:NoSchedule

And a matching toleration in the pod spec:

tolerations:
- key: "workload-type"
  operator: "Equal"
  value: "arm-optimized"
  effect: "NoSchedule"

We're not doing this in the tutorial to keep things simple, but it's the pattern production multi-tenant clusters use to enforce hard separation between workload types.

Write the Service Manifest

We also need a Kubernetes Service to expose the pods over the network. Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-axion-svc
spec:
  selector:
    app: hello-axion
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

• selector: app: hello-axion — the Service discovers pods using labels. Any pod with app: hello-axion on it will be added to this Service's load balancer pool.

• port: 80 — the port the Service is reachable on from outside the cluster.

• targetPort: 8080 — the port on the pod that traffic gets forwarded to. Our Go server listens on port 8080, so this must match.

• type: LoadBalancer — tells GKE to provision an external Google Cloud load balancer and assign it a public IP. This is what makes the Service reachable from the internet.

Apply Both Manifests

kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

kubectl apply reads each manifest file and creates or updates the resources described in it. If the resources don't exist yet, they're created. If they already exist, Kubernetes only applies the diff — it won't restart pods unnecessarily.

Watch the pods come up in real time:

kubectl get pods -w

The -w flag watches for changes and prints updates as they happen. You should see pods transition from Pending → ContainerCreating → Running. Once all three show Running, press Ctrl+C to stop watching.

Step 9: Verify the Deployment

Everything is running. Now we need evidence — not just that pods are up, but that they're on the right nodes and serving the right binary.

Confirm Pod Placement

kubectl get pods -o wide

The -o wide flag adds extra columns to the output, including the name of the node each pod was scheduled on. Look at the NODE column:

NAME                          READY   STATUS    NODE
hello-axion-7b8d9f-abc12      1/1     Running   gke-axion-tutorial-axion-pool-a-...
hello-axion-7b8d9f-def34      1/1     Running   gke-axion-tutorial-axion-pool-b-...
hello-axion-7b8d9f-ghi56      1/1     Running   gke-axion-tutorial-axion-pool-c-...

All three pods should show node names containing axion-pool. None should show default-pool.

Confirm the Nodes Are ARM

Take one of those node names and verify its architecture label:

kubectl get node NODE_NAME --show-labels | grep kubernetes.io/arch

Replace NODE_NAME with one of the node names from the previous command. You should see:

kubernetes.io/arch=arm64

That's the automatic label GKE applied when it provisioned the ARM hardware. Our nodeSelector matched on this label to pin the pods here.

Ask the Application Itself

This is the most satisfying verification step. Our Go server reports the architecture of the binary that's running. Let's ask it directly.

Use kubectl port-forward to create a secure tunnel from port 8080 on your local machine to port 8080 on the Deployment:

kubectl port-forward deployment/hello-axion 8080:8080

This command stays running in the foreground — open a second terminal window and run:

curl http://localhost:8080

You should see:

Hello from freeCodeCamp!
Architecture : arm64
OS           : linux
Pod hostname : hello-axion-7b8d9f-abc12

Architecture : arm64. That's our Go binary confirming that it was compiled for ARM64 and is executing on an ARM64 CPU. The single image tag we built does the right thing automatically.

The Bonus: See the Manifest List in Action

Want to see the multi-arch image indexing at work? Stop the port-forward, then run:

docker buildx imagetools inspect \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Replace PROJECT_ID with your actual Google Cloud project ID.

You'll see four entries in the manifest list. Two are real images — Platform: linux/amd64 and Platform: linux/arm64. The other two show Platform: unknown/unknown with an attestation-manifest annotation. These are build provenance records that Docker Buildx automatically attaches to every image — a supply chain security feature (SLSA attestation) that proves how and where the image was built.

You may notice that if you check the image digest recorded in a running pod:

kubectl get pod POD_NAME \
  -o jsonpath='{.status.containerStatuses[0].imageID}'

Replace POD_NAME with one of the pod names from earlier.

The digest returned matches the top-level manifest list digest, not the arm64-specific one. This is expected behaviour. Modern Kubernetes (using containerd) records the manifest list digest, not the resolved platform digest. The platform resolution already happened when the node pulled the correct image variant.

The definitive proof that the right binary is running is what you already have: the node labeled kubernetes.io/arch=arm64 and the application reporting Architecture: arm64.

Step 10: Cost Savings and Tradeoffs

The hands-on work is done. Let's talk about why any of this is worth the effort.

The Cost Math

At the time of writing, here's how ARM compares to equivalent x86 machines on Google Cloud (prices are approximate and change over time — check the official pricing page before making decisions):

That's a raw 25–30% reduction in compute cost per node. Factor in Google's published claim of up to 65% better price-performance for Axion on relevant workloads — meaning you may need fewer nodes to handle the same traffic — and the savings compound further.

Here's how that looks at scale, for a service running 20 nodes continuously for a year:

• 20 × n2-standard-4 × \(0.19 × 8,760 hours = \)33,288/year

• 20 × t2a-standard-4 × \(0.14 × 8,760 hours = \)24,528/year

That's roughly $8,760 saved annually on compute, before committed use discounts (which further widen the gap).

When ARM Is the Right Choice

ARM works best for:

• Stateless API servers and web applications — like the app we built. ARM excels at high-throughput, low-latency network workloads.

• Background workers and queue processors — long-running services that don't depend on x86-specific binaries.

• Microservices written in Go, Rust, or Python — these languages have excellent ARM64 support and are built cross-platform by default.

When to Proceed Carefully

• Native library dependencies — some older C libraries, proprietary SDKs, or compiled ML model-serving runtimes don't have ARM64 builds. Always audit your dependency tree before migrating.

• CI pipelines need ARM too — your automated tests should run on ARM, not just x86. An image that silently fails only on ARM is harder to debug than one that never claimed ARM support.

• Profile before optimizing — the cost savings are real, but measure your actual workload behavior on ARM before committing. Not every workload benefits equally.

Cleanup

When you're done, clean up to avoid ongoing charges:

# Remove the Kubernetes resources from the cluster
kubectl delete -f k8s/

# Delete the ARM node pool
gcloud container node-pools delete axion-pool \
  --cluster=axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the cluster itself
gcloud container clusters delete axion-tutorial-cluster \
  --zone=us-central1-a

# Delete the images from Artifact Registry (optional — storage costs are minimal)
gcloud artifacts docker images delete \
  us-central1-docker.pkg.dev/PROJECT_ID/multi-arch-repo/hello-axion:v1

Conclusion

Let's recap what you built and why each part matters.

You started with a Go application, a Dockerfile, and a docker buildx build command that produced two images — one for x86, one for ARM64 — wrapped in a single Manifest List tag. Any server that pulls that tag gets the right binary automatically, without you maintaining separate pipelines or separate tags.

You provisioned a GKE cluster with two node pools running different CPU architectures, then used nodeSelector to make sure your ARM-optimized workload lands only on the ARM Axion nodes — not on x86 by accident. The result is a deployment that's both architecture-correct and cost-efficient.

The patterns you practiced here don't stop at this demo. The same Dockerfile technique works for any language with cross-compilation support. The same nodeSelector approach works for any workload you want to pin to ARM. As more teams migrate services to ARM over the coming years, having these skills will be a real asset.

Where to go from here:

• Add a GitHub Actions workflow that runs docker buildx build --platform linux/amd64,linux/arm64 on every push, automating this entire process in CI.

• Audit one of your existing stateless services for ARM compatibility and try migrating it.

• Explore Node Affinity as a softer alternative to nodeSelector for workloads that can run on either architecture but prefer ARM.

• Look into GKE Autopilot, which now supports ARM nodes and handles node pool management automatically.

Happy building.

Project File Structure

hello-axion/
├── app/
│   ├── main.go          — Go HTTP server
│   ├── go.mod           — Go module definition
│   └── Dockerfile       — Multi-stage Dockerfile
└── k8s/
    ├── deployment.yaml  — Deployment with nodeSelector and probes
    └── service.yaml     — LoadBalancer Service

All source files for this tutorial are available in the companion GitHub repository: https://github.com/Amiynarh/multi-arch-docker-gke-arm

And while many SaaS tools can monitor your site, they often charge high monthly fees for simple alerts.

My goal with this article is to help you stop paying those expensive fees by showing you a powerful, free, open-source alternative called Uptime Kuma.

In this guide, you'll learn how to use Docker to deploy Uptime Kuma safely on a local Ubuntu machine.

By the end of this tutorial, you'll have set up your own private server monitoring dashboard in less than 10 minutes and created an automated Discord alert to ping your phone if your website goes offline.

Table of Contents

• Prerequisites

• Step 1: Update Packages and Prepare the Firewall

• Step 2: Create the Docker Compose File

• Step 3: Start the Application

• Step 4: Access the Dashboard

• Step 5: Use Case – Monitor a Website and Send Discord Alerts

• Conclusion

Prerequisites

Before you start, make sure you have:

• An Ubuntu machine (like a local server, VM, or desktop).

• Docker and Docker Compose installed.

• Basic knowledge of the Linux terminal.

Step 1: Update Packages and Prepare the Firewall

First, you'll want to make sure your system has the newest updates. Then, you'll install the Uncomplicated Firewall (UFW) and open the network "door" (port) that Uptime Kuma uses for the dashboard. You'll also need to allow SSH so you don't lock yourself out.

Run these commands in your terminal:

1. Update your packages:

sudo apt update && sudo apt upgrade -y

1. Install the firewall:

sudo apt install ufw -y

1. Allow SSH and open port 3001:

sudo ufw allow ssh
sudo ufw allow 3001/tcp

1. Enable the firewall:

sudo ufw enable
sudo ufw reload

Step 2: Create the Docker Compose File

Using a docker-compose.yml file is the professional way to manage Docker containers. It keeps your setup organised in one single place.

To start, create a new folder for your project and enter it:

mkdir uptime-kuma && cd uptime-kuma

Then create the configuration file:

nano docker-compose.yml

Paste the following code into the editor:

services:
  uptime-kuma:
    image: louislam/uptime-kuma:2
    restart: unless-stopped
    volumes:
      - ./data:/app/data
    ports:
      - "3001:3001"

Note: The ./data:/app/data line is very important. It saves your database in a normal folder on your machine, making it easy to back up later.

Finally, save and exit: Press CTRL + X, then Y, then Enter.

Step 3: Start the Application

Now, tell Docker to read your file and start the monitoring service in the background.

docker compose up -d

How to verify: Docker will download the files. When it finishes, your terminal should print Started uptime-kuma.

Step 4: Access the Dashboard

To access the dashboard, first open your web browser and go to http://localhost:3001 (or your machine's local IP address).

When asked to choose the database, select SQLite. It's simple, fast, and requires no extra setup.

Then create an account and choose a secure admin username and password.

Step 5: Use Case – Monitor a Website and Send Discord Alerts

Now you'll put Uptime Kuma to work by monitoring a live website and setting up an alert. Just follow these steps:

1. Click Add New Monitor.

2. Set the Monitor Type to HTTP(s).

3. Give it a Friendly Name (e.g., "My Blog") and enter your website's URL.

Pro-Tip: How to Fix "Down" Errors (Bot Protection)

If your site uses strict security, it might block Uptime Kuma and say your site is "Down" with a 403 Forbidden error.

The Fix: Scroll down to Advanced, find the User Agent box, and paste this text to make Uptime Kuma look like a normal Chrome browser:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Add a Discord Alert

To get a message on your phone when your site goes down:

1. On the right side of the monitor screen, click Setup Notification.

2. Select Discord from the dropdown list.

3. Paste a Discord Webhook URL (you can create one in your Discord server settings under Integrations).

4. Click Test to receive a test ping, then click Save.

Conclusion

Congratulations! You just took control of your server health. By deploying Uptime Kuma, you replaced an expensive SaaS subscription with a powerful, free monitoring tool that alerts you the second a project goes offline.

Let’s connect! I am a developer and technical writer specialising in writing step-by-step guides and workflows. You can find my latest projects on my Technical Writing Portfolio or reach out to me directly on LinkedIn.